GPU computing servers: Arachne and Aura

The serveraura.fi.muni.cz is available to FI staff, PhD students and contract staff for longer-term, more demanding or GPU-based computations. The newer and more powerful server arachne.fi.muni.cz is available primarily to FI PhD students and also to FI staff . For study or research purposes, PhD students or FI staff may request to grant access to other individuals. However, where possible, please prioritise use of the Nymf GPU, then the AuraGPU, and only then the Arachne GPU (see also the overview of GPUs at FI). Access to the Aura and Arachne servers is only possible from the MU network.

Hardware configuration

TheArachne server is built on the Asus ESC8000A-E13P platform with the following configuration:

• two 96-core AMD EPYC 9655 2.6 GHz processors (a total of 192 physical cores and 384 threads).
• 2.25 TiB DDR5 RAM at 2800 MHz
• 10 Gbps Ethernet connection
• 2 x 960 GB NVMe drives in RAID 1
• 8 NVMe drives with a capacity of 15.36 TB configured in RAID 10
• 4 NVIDIA H200 141 GB PCIe GPUs with NVLink
• Red Hat Enterprise Linux operating system

TheAura server is built on the Asus RS720A-E11-RS24U platform with the following configuration:

• two 64-core AMD EPYC 7713 2.0 GHz processors (a total of 128 physical cores and 256 threads).
• 2 TiB DDR4 RAM at 3200 MHz
• 10 Gbps Ethernet connection
• 2 x 960 GB SATA SSDs in RAID 1
• 2 NVMe drives, each with a capacity of 6 TB, configured in RAID 1
• 2 NVIDIA A100 80 GB PCIe GPUs with NVLink
• Red Hat Enterprise Linux operating system

See also the blog post introducing this server.

How to work on the computing servers

We recommend that you also familiarise yourself with the general information on running computations.

Run long-running processes (one hour or more) with a reduced priority (in the range 10–19, where 19 is the lowest), for example nice ./your_program ornice -n 15 ./your_program.

To change the priority of a process that is already running, you can use the command renice, but bear in mind that a process may run in multiple threads, and changing the priority of a single process may only change the priority of one thread. You can obtain a list of all threads in your processes, including their priorities, as follows, for example:

ps x -Lo pgid,pid,tid,user,nice,tty,time,args

You can run short-term processes or perform interactive debugging of your programmes at normal priority.

If your process does not adhere to the priority limits and consumes a large amount of computing power, in order to prevent restrictions on other users, all your processes will be set to the lowest priority of 19. In the event of repeated or more serious breaches of this rule, your faculty account may be temporarily suspended.

For long-running calculations, save your intermediate results once a day. This will prevent significant loss of work in the event of a system outage (whether planned or unplanned).

Limits via systemd/cgroups

The upper limit on memory usage in the system can be checked using the command below. Once this limit is exceeded, the OOM mechanism is triggered, which will attempt to terminate a suitable process.

systemctl show -p MemoryMax user-$UID.slice

However, you can create your own systemd scope, within which you can set, for example, a stricter limit on usable memory, a limit on the number of CPU cores (100% = 1 core) or other limits (using cgroups):

systemd-run --user --scope -p MemoryMax=9G -p CPUQuota=400% program

The programme can also be a command-line tool (e.g. `bash`). The limits set by cgroups will apply to it and all its descendants collectively. This is both a difference and an advantage compared to the `ulimit` mechanism, where the limits apply to each process individually.

It may be useful to monitor both the created scope and the user scope:

# monitoring of the memory and CPU usage of your processes
systemd-cgtop /user.slice/user-$UID.slice

Resource limits usingulimit

Commands for limiting resource usage:

# limit available resources
help ulimit
# cap the size of virtual memory to 20000 kB
ulimit -v 20000
# cap the amount of total CPU time to 3600 seconds
ulimit -t 3600
# cap the number of concurrently running threads/processes
ulimit -u 100

The commands listed above will limit the resources of the shell and all its descendants to the specified values. These cannot be increased again; to restore the environment without the set limits, you will need to run another separate shell. Be aware, however, that resources set using `ulimit ` apply to each process separately. Therefore, if you set a memory limit of 20 MB and run 10 processes in such an environment, they may collectively allocate 200 MB of memory. If you simply wish to limit the total memory to 20 MB, use `systemd-run`.

Specific software

If you need to install additional libraries or tools for your work, you have several options (apart from compiling locally):

• if they are part of the distribution (dnf search software-name), you can ask the administrator to install them,
• you can create a module,
• if it is a Python package, you can ask the administrator to install it into thepython3 module. You can also install it locally using pip/pip3 install --user. When usingvirtualenv, conda etc., we recommend installing the environment into/var/tmp/login (note the file lifespan – see below).

Disk space

For temporary data that needs to be quickly accessible locally, two directories are available on the computing servers:

• The /tmp is of the tmpfs type. As it is located in RAM, access is very fast; however, data is not persistent between server reboots and the capacity is very small. Do not store calculation results here, so as not to cause system-wide problems should it become full.
• The directory /var/tmp is located on a fast RAID volume built on NVMe.

An advantage of using these, particularly for I/O-intensive computations, is the reduced load on the network and on the servers hosting the storage athome anddata.

If you wish to use this space, please store your data in the directory corresponding to your login. Data that is not accessed (as peratime) is automatically deleted; for/tmp this occurs after a few days, and for/var/tmp after a few months (you can find the exact settings in the file /etc/tmpfiles.d/tmp.conf). Disk quotas do not apply here; however, please be considerate of others when using storage space.

If access to data does not need to be fast and is not intensive over the long term, or if you wish to store work results and free up space on/var/tmp, you may also consider the storage servicedata, where we can temporarily increase the quota, even significantly.

GPU computations

The Aura server is equipped with two NVIDIA A100 80 GB PCIe GPU cards, whilst the Arachne server has four NVIDIA H200 141 GB PCIe GPU cards.

If you have any suggestions regarding the functionality or how to work with the GPUs on Aura/Arachne, we’d be happy to hear about them at.

GPU computations on Aura/Arachne are not currently subject to any system-wide restrictions, and it is important to be considerate of others.

Choosing a graphics card

Unlike in the past, running multiple computations concurrently on a single GPU is no longer a major issue. If the GPU card is partitioned using MIG (Multi-Instance GPU) technology, it is possible to have several virtual GPUs (instances) that do not interfere with one another.

Before starting a calculation, we must set the environment variable CUDA_VISIBLE_DEVICES appropriately. To select a suitable value, you can use information from nvidia-smi or the tool nvisel:

[user@aura ~]$ nvidia-smi
Thu Jan 18 10:48:06 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe          On  | 00000000:21:00.0 Off |                   On |
| N/A   59C    P0             236W / 300W |  35141MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80GB PCIe          On  | 00000000:61:00.0 Off |                    0 |
| N/A   43C    P0              44W / 300W |      4MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| MIG devices:                                                                          |
+------------------+--------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                   Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                     BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                |        ECC|                       |
|==================+================================+===========+=======================|
|  0    3   0   0  |              25MiB / 19968MiB  | 28      0 |  2   0    1    0    0 |
|                  |               0MiB / 32767MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
|  0    4   0   1  |           17577MiB / 19968MiB  | 28      0 |  2   0    1    0    0 |
|                  |               2MiB / 32767MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
|  0    6   0   2  |           17514MiB / 19968MiB  | 14      0 |  1   0    1    0    0 |
|                  |               2MiB / 32767MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
|  0   11   0   3  |              12MiB /  9728MiB  | 14      0 |  1   0    1    1    1 |
|                  |               0MiB / 16383MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
|  0   12   0   4  |              12MiB /  9728MiB  | 14      0 |  1   0    0    0    0 |
|                  |               0MiB / 16383MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0    4    0     902451      C   ./computation0                            17544MiB |
|    0    6    0     902453      C   ./computation1                            17494MiB |
+---------------------------------------------------------------------------------------+

Here, in the first table, we can see a list of GPU cards and, most importantly, in the last column, information on whether MIG is enabled (indicated by the words ‘Enabled ’ and ‘Disabled ’). In this example, the GPU0 card is split using MIG, whilst the second card, GPU1, is not split.

In the second table, we can see the individual GPU instances with their allocated resources. If none of the cards is split, the ‘MIG devices: ’ table will not be displayed.

In the final table, we can see the computations currently running. In the example, computations are running on the partitioned GPU0 card in partitions 4 and 6. We can therefore choose between the unpartitioned GPU1, which has 80 GB of memory, or the partitioned GPU0, which has available instances with GI 3 (20 GB), 11 (10 GB) and 12 (10 GB).

If we have chosen the unpartitioned card, we set `CUDA_VISIBLE_DEVICES ` to the GPU card index (CUDA_VISIBLE_DEVICES=1). If we have chosen a partition with GI 11 (MIG Dev 3) from the partitioned GPU0 card, we need to find out its UUID. To do this, usenvidia-smi -L.

[user@aura ~]$ nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-309d72fd-b4f8-d6e8-6a66-e3f2253e8540)
  MIG 2g.20gb     Device  0: (UUID: MIG-ee0daf5f-9543-5e3f-8157-308a15c318b4)
  MIG 2g.20gb     Device  1: (UUID: MIG-fbb89bfe-6460-508c-ab51-9b961def7e01)
  MIG 1g.20gb     Device  2: (UUID: MIG-102d7a8b-5941-5275-be02-72ff5819ead4)
  MIG 1g.10gb     Device  3: (UUID: MIG-c4dc2f6b-2c55-566d-8738-fa8176580fda)
  MIG 1g.10gb     Device  4: (UUID: MIG-cd46e799-21e5-54d8-b751-f4a3afb52a46)
GPU 1: NVIDIA A100 80GB PCIe (UUID: GPU-04712e69-7356-4de5-f983-84083131460e)

Set the variable `CUDA_VISIBLE_DEVICES ` to:
CUDA_VISIBLE_DEVICES=MIG-c4dc2f6b-2c55-566d-8738-fa8176580fda

Monitoring computations

You can monitor your computation either using the command `nvidia-smi ` or `gpu-stat `, or via the interactive graphical tools `nvitop ` or `nvtop ` (we recommend using a terminal window larger than 80x24). The monitoring tools cannot display GPU utilisation (Util) for split cards and instead display `N/A`.

Changing GPU allocation

If the existing configuration of MIG instances is not suitable for you, it can be changed by arrangement at unix@fi.muni.cz , provided that circumstances (other running computations) reasonably permit it.

CUDA andnvcc

If you wish to usenvcc (or related tools), you can find it at/usr/local/cuda-$VER/bin/nvcc. Alternatively, you can install a different version via Podman.

GPU card reservations

In justified cases (e.g. for a workshop or presentation), it is possible to request via unix@fi.muni.cz the reservation of an entire GPU card for a specific user or group of users.

Reservations must be arranged well in advance due to technical constraints – a reservation prevents new jobs from being launched on the GPU card in question, although calculations already in progress may continue.

Container support – Podman

The Podman tool is also available on the computing servers for computations; it provides the same functionality as Docker. Thanks to the translation layerpodman-docker, you can also use the commanddocker.

Each user is assigned a range ofsubuid and subgid based on theirUID and can therefore use rootless containers. The range is 100,000 in size and starts fromUID*100000. By default, containers are placed in/var/tmp/containers/xlogin. No quotas are currently applied to this volume, so please keep an eye on your files, and if you no longer need the storage, please delete your containers properly.

Podman and GPU

Unlike when running a container normally, you must specify which GPU partition you wish to use. To set the selected GPU partition, instead of using `UUID `, enter the partition in the format `GPU:Device`, where `GPU ` and `Device ` are the numbers from the output of `nvidia-smi -L`. To use GPU 0 and partition 4, add the parameter `--device nvidia.com/gpu=0:4`.

Podman and directory sharing

To map a directory, you can use the `--volume, -v HOST-DIR:CONTAINER-DIR:OPTIONS` switch (see the documentation).

To set permissions correctly, it is advisable to use the `--userns=keep-id:uid=1000,gid=1000` option, which maps the current user’s permissions to the user with ID `1000 ` within the container.

Mapping the current working directory to the container might look as follows:

--userns=keep-id:uid=1000,gid=1000 -v "$PWD":"$PWD":Z -w "$PWD"

• --userns=keep-id:uid=1000,gid=1000 - setting user permissions
• -v "$PWD":"$PWD":Z - mapping `$PWD ` to `Z ` for SELinux private relabelling
• -w "$PWD" - setting the container’s home directory to$PWD instead/

Example

If you wish to use JupyterLab with TensorFlow and GPU support, you can find, for example, the `gpu-jupyter ` image onDocker Hub (this is a large image, so the first launch may take some time). On the server, simply run:

podman run --rm --security-opt label=disable -p 127.0.0.1:11000:8888 \
-v "${PWD}":/home/jovyan/work --device nvidia.com/gpu=0:4 \
cschranz/gpu-jupyter:v1.5_cuda-11.6_ubuntu-20.04_python-only

Please note that containers mapped in this way are only accessible locally from the server. For wider accessibility, you will need to set up an SSH tunnel, for example (ssh -L 8888:localhost:11000 aura). The port can also be mapped using-p 11000:8888 – in which case it will be accessible from the FI network.

The section-v "${PWD}":/home/jovyan/work maps the current working directory inside the container to/home/jovyan/work. JupyterLab uses this directory as its working directory.

We can then verify this functionality, for example, using this code, which returns the number of available GPUs (the expected output is 1):

import tensorflow as tf
print("Number of available GPUs: ", len(tf.config.list_physical_devices('gpu')))

Subsequent runs are faster, as the downloaded image is not automatically deleted upon termination. To clear up any remnants (unused images, containers, etc.), you can use the command `podman system prune -a`.