Aura

The aura.fi.muni.cz server is available to FI staff and PhD students for longer term, more demanding or GPU computations. For study or research purposes, FI staff can request access to others via unix@fi.muni.cz.

Hardware configuration

The Aura server is built on the Asus RS720A-E11-RS24U platform in the following configuration:

• Two 64-core AMD EPYC 7713 2.0 GHz processors ( 128 physical cores and 256 threads in total).
• 2 TiB DDR4 RAM 3200 MHz
• 10 Gbps Ethernet connection
• 2 SATA SSDs with 960 GB capacity in RAID 1
• 2 6 TB NVMe drives in RAID 1
• 2 NVIDIA A100 80 GB PCIe GPU cards with NVLink

See also the blog post introducing this server.

How compute servers work

Run long running processes (an hour or more) at a reduced priority (in the range 10-19, 19 being the lowest), for example nice ./your_program or nice -n 15 ./your_program.

To change the priority of an already running process, you can use the command renice command, but beware that a process may be running on multiple threads and changing the priority for one process may change the priority of only one thread. For example, you can get a listing of all the threads of your processes, including the priority, as follows:

ps x -Lo pgid,pid,tid,user,nice,tty,time,args

You can run short-term processes or interactive debugging of your programs with normal priority.

If your process does not adhere to the priority constraint and uses a large amount of computing power, all your processes will be set to the lowest priority 19 to prevent other users from being constrained. Repeated or more serious violations of this rule may result in temporary disabling of your faculty account.

Memory limitations using systemd

The upper limit of memory usage on the system can be determined using the command below. When this limit is exceeded, the OOM mechanism will be triggered to attempt to terminate the appropriate process.

systemctl show -p MemoryMax user-$UID.slice

However, you can create your own systemd scope in which a more stringent (lower) limit on usable memory can be set:

systemd-run --user --scope -p MemoryMax=9G program

The program can also be a command line (e.g. bash). The memory limit will be applied to it and all its children together. This is different from the ulimit mechanism, where the limit applies to each process separately.

Monitoring both the created scope and the user scope can be useful:

# monitoring of the memory and CPU usage of your processes
systemd-cgtop /user.slice/user-$UID.slice

Resource constraints using ulimit

Resource limiting commands:

# limit available resources
help ulimit
# cap the size of virtual memory to 20000 kB
ulimit -v 20000
# cap the amount of total CPU time to 3600 seconds
ulimit -t 3600
# cap the number of concurrently running threads/processes
ulimit -u 100

The above commands limit the resources of the shell and all its children to the specified values. These cannot be rolled back; another separate shell will need to be run to restore the environment without the limits set. Note, however, that the resources set by ulimit apply to each process separately. Thus, if you set the limit to 20 MB of memory and run 10 processes in such an environment, they may allocate a total of 200 MB of memory. If you just want to limit the total memory to 20 MB, use systemd-run.

Specific software

If you need to install libraries or tools for your work, you have several options (besides local compilation):

• if they are part of the distribution ( dnf search software-name), you can ask the maintainer to install them,
• you can make a module,
• if it is a Python package, you can ask the maintainer to install it in a module python3. You can also install it locally using pip/pip3 install --user. If you use virtualenv, conda etc, we recommend installing the environment into /var/tmp/login (see below for file lifetimes).

Disk capacities

For temporary data that should be quickly available locally, two directories are available on the Aura server.

• The directory /tmp is of type tmpfs. Due to its location in RAM, access is very fast, but the data is not persistent between server reboots and the capacity is very small.
• The directory /var/tmp is located on a fast NVMe RAID 1 volume.

The advantage of using them, especially for I/O-intensive computations, is also a lower load on the network and also on the server with the home and data storage.

To use this space, store your data in a directory with your login. Data that is not accessed (according to atime) is automatically deleted, for /tmp when it is a few days old, for /var/tmpwhen it is a few months old (see /etc/tmpfiles.d/tmp.conf for exact settings). Disk quotas do not apply here; however, be considerate of others in your use of space.

GPU calculations

The Aura server has two GPU cards, namely the NVIDIA A100 80 GB PCIe.

Please note: Aura is the first CVT-managed server that is also equipped with GPU cards, so it is possible that things will not run smoothly from the start. So if you have any comments or advice, feel free to write to us.

GPU computations on Aura are currently not restricted in any way by the system, so it is important to be respectful of others.

The GPU cards are partitioned using MIG (Multi-Instance GPU) technology, which makes it possible to have multiple non-interacting virtual GPUs (instances).

Before starting the computation, we have to choose a free MIG instance. Using the listing from nvidia-smi we can find out which instance IDs exist and on which the computation is running.

In the example below, we see that there is 1 MIG instance on GPU0 and 3 MIG instances on GPU1. Looking at Processes:, we see that the calculations are running on one of the instances on GPU0 (GI ID 0). On GPU1, the computations are running on instance 3 and hence the instances with GI IDs 2 and 4 are free. So we choose GPU1 with GI CI 4 for our computations, which in our case has MIG Dev 2.

[user@aura ~]$ nvidia-smi
Mon Mar 21 15:34:36 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100 80G...  Off  | 00000000:21:00.0 Off |                   On |
| N/A   43C    P0    65W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80G...  Off  | 00000000:61:00.0 Off |                   On |
| N/A   41C    P0    70W / 300W |     45MiB / 81920MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+----------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
|      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
|                  |                      |        ECC|                       |
|==================+======================+===========+=======================|
|  0    0   0   0  |   2406MiB / 81069MiB | 98      0 |  7   0    5    1    1 |
|                  |      5MiB / 13107... |           |                       |
+------------------+----------------------+-----------+-----------------------+
|  1    2   0   0  |     19MiB / 40192MiB | 42      0 |  3   0    2    0    0 |
|                  |      0MiB / 65535MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+
|  1    3   0   1  |   2080MiB / 19968MiB | 28      0 |  2   0    1    0    0 |
|                  |      2MiB / 32767MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+
|  1    4   0   2  |     13MiB / 19968MiB | 28      0 |  2   0    1    0    0 |
|                  |      0MiB / 32767MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0    0    0     235532      C   python3                          2403MiB |
|    1    3    0     235740      C   python3                          2061MiB |
+-----------------------------------------------------------------------------+

Now we will use the command nvidia-smi -L to find out the UUID. We want GPU 1 with MIG Dev 2, so MIG-64f11db9-b10b-5dd9-97d1-3c46450b9388:

[user@aura ~]$ nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-309d72fd-b4f8-d6e8-6a66-e3f2253e8540)
  MIG 7g.80gb     Device  0: (UUID: MIG-a5459e6a-b26d-5985-874c-528458a7728b)
GPU 1: NVIDIA A100 80GB PCIe (UUID: GPU-04712e69-7356-4de5-f983-84083131460e)
  MIG 3g.40gb     Device  0: (UUID: MIG-4f7fbfb7-a8a2-553d-875a-d9d56baf97a5)
  MIG 2g.20gb     Device  1: (UUID: MIG-bad562c5-744a-5742-be1a-c735195c52d0)
  MIG 2g.20gb     Device  2: (UUID: MIG-64f11db9-b10b-5dd9-97d1-3c46450b9388)

Now we set the environment variable CUDA_VISIBLE_DEVICES and we can compute.

[user@aura ~]$ export CUDA_VISIBLE_DEVICES=MIG-64f11db9-b10b-5dd9-97d1-3c46450b9388
[user@aura ~]$ python main.py

We can monitor our computation using either the command nvidia-smi or the graphical tool nvitop (we recommend to have a larger terminal window than 80x24). Due to the use of MIG, the monitoring tools are not able to show the GPU usage ( N/A), but only the memory usage.

If the existing configuration of MIG instances is not suitable for you, it can be changed by agreement at unix@fi.muni.cz if circumstances (other running computations) reasonably allow it.