Machine-translated page for increased accessibility for English questioners.
Aura
The
aura.fi.muni.cz server is available to FI staff and PhD students for longer term, more demanding or GPU computations. For study or research purposes, FI staff can request access to others via
unixUALQbMBSN@fiJ17-8E7i0.muniKumW8dZ4W.cz. Access to the Aura server is only possible from the MU network.
Hardware configuration
The
Aura server is built on the
Asus RS720A-E11-RS24U platform in the following configuration:
- two 64-core AMD EPYC 7713 2.0 GHz processors ( 128 physical cores and 256 threads in total).
- 2 TiB DDR4 RAM 3200 MHz
- 10 Gbps Ethernet connection
- 2 SATA SSDs with 960 GB capacity in RAID 1
- 2 6 TB NVMe drives in RAID 1
- 2 NVIDIA A100 80 GB PCIe GPU cards with NVLink
- Red Hat Enterprise Linux operating system
See also the blog post introducing this server.
How to work on compute servers
We recommend you also familiarize yourself with general information about running compute.
Run long-running processes (an hour or more) at a
reduced priority (in the range 10-19, 19 being the lowest), for example
nice ./your_program or
nice -n 15 ./your_program.
To change the priority of an already running process, you can use the command
renice command, but beware that a process may be running on multiple threads and changing the priority for one process may change the priority of only one thread. For example, you can get a listing of all the threads of your processes, including priority, as follows:
ps x -Lo pgid,pid,tid,user,nice,tty,time,args
You can run short-term processes or interactive debugging of your programs with normal priority.
If your process does not adhere to the priority constraint and uses a large amount of computing power, all your processes will be set to the lowest priority 19 to prevent other users from being constrained. Repeated or more serious violations of this rule may result in temporary disabling of your faculty account.
Memory limitations using systemd
The upper limit of memory usage on the system can be determined using the command below. When this limit is exceeded, the OOM mechanism will be triggered to attempt to terminate the appropriate process.
systemctl show -p MemoryMax user-$UID.slice
However, you can create your own systemd scope in which a more stringent (lower) limit on usable memory can be set:
systemd-run --user --scope -p MemoryMax=9G program
The program can also be a command line (e.g.
bash). The memory limit will be applied to it and all its children together. This is different from the
ulimit mechanism, where the limit applies to each process separately.
Monitoring both the created scope and the user scope can be useful:
# monitoring of the memory and CPU usage of your processes
systemd-cgtop /user.slice/user-$UID.slice
Resource constraints using
ulimit
Resource limiting commands:
# limit available resources
help ulimit
# cap the size of virtual memory to 20000 kB
ulimit -v 20000
# cap the amount of total CPU time to 3600 seconds
ulimit -t 3600
# cap the number of concurrently running threads/processes
ulimit -u 100
The above commands limit the resources of
the shell and all its children to the specified values. These cannot be rolled back; another separate shell will need to be run to restore the environment without the limits set. Note, however, that the
resources set by
ulimit apply to each process separately. Thus, if you set the limit to 20 MB of memory and run 10 processes in such an environment, they may allocate a
total of 200 MB of memory. If you just want to limit the total memory to 20 MB, use
systemd-run.
Specific software
If you need to install libraries or tools for your work, you have several options (besides local compilation):
- if they are part of the distribution (
dnf search software-name), you can ask the maintainer to install them, - you can make a module,
- if it is a Python package, you can ask the maintainer to install it in a module
python3. You can also install it locally usingpip/pip3 install --user. If you usevirtualenv, condaetc, we recommend installing the environment into/var/tmp/login(see below for file lifetimes).
Disk capacities
For temporary data that should be quickly available locally, two directories are available on the Aura server.
- The directory
/tmpis of type tmpfs. Due to its location in RAM, access is very fast, but the data is not persistent between server reboots and the capacity is very small. - The directory
/var/tmpis located on a fast NVMe RAID 1 volume.
The advantage of using them, especially for I/O-intensive computations, is also lower network and server load with the
home and
data storage.
To use this space, store your data in a directory with your login. Data that is not accessed (according to
atime) is automatically deleted, for
/tmp when it is a few days old, for
/var/tmpwhen it is a few months old (see
/etc/tmpfiles.d/tmp.conf for exact settings). Disk quotas do not apply here; however, be considerate of others in your use of space.
GPU calculations
The Aura server has two GPU cards, namely the NVIDIA A100 80 GB PCIe.
Please note: Aura is the first CVT-managed server that is also equipped with GPU cards, so it is possible that things will not run smoothly from the start. So if you have any comments or advice, feel free to write to us.
GPU computations on Aura are currently not restricted in any way by the system, so it is important to be respectful of others.
The GPU cards are partitioned using MIG (Multi-Instance GPU) technology, which makes it possible to have multiple non-interacting virtual GPUs (instances).
Before starting the computation, we have to choose a free MIG instance. Using the listing from
nvidia-smi we can find out which instance IDs exist and on which the computation is running.
In the example below, we see that there is 1 MIG instance on GPU0 and 3 MIG instances on GPU1. Looking at
Processes:, we see that the calculations are running on one of the instances on GPU0 (GI ID 0). On GPU1, the computations are running on instance 3 and hence the instances with GI IDs 2 and 4 are free. So we choose GPU1 with GI CI 4 for our computations, which in our case has MIG Dev 2.
[user@aura ~]$ nvidia-smi
Mon Mar 21 15:34:36 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100 80G... Off | 00000000:21:00.0 Off | On |
| N/A 43C P0 65W / 300W | 0MiB / 81920MiB | N/A Default |
| | | Enabled |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A100 80G... Off | 00000000:61:00.0 Off | On |
| N/A 41C P0 70W / 300W | 45MiB / 81920MiB | N/A Default |
| | | Enabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| MIG devices: |
+------------------+----------------------+-----------+-----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG|
| | | ECC| |
|==================+======================+===========+=======================|
| 0 0 0 0 | 2406MiB / 81069MiB | 98 0 | 7 0 5 1 1 |
| | 5MiB / 13107... | | |
+------------------+----------------------+-----------+-----------------------+
| 1 2 0 0 | 19MiB / 40192MiB | 42 0 | 3 0 2 0 0 |
| | 0MiB / 65535MiB | | |
+------------------+----------------------+-----------+-----------------------+
| 1 3 0 1 | 2080MiB / 19968MiB | 28 0 | 2 0 1 0 0 |
| | 2MiB / 32767MiB | | |
+------------------+----------------------+-----------+-----------------------+
| 1 4 0 2 | 13MiB / 19968MiB | 28 0 | 2 0 1 0 0 |
| | 0MiB / 32767MiB | | |
+------------------+----------------------+-----------+-----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 0 0 235532 C python3 2403MiB |
| 1 3 0 235740 C python3 2061MiB |
+-----------------------------------------------------------------------------+
Now we will use the command
nvidia-smi -L to find out the UUID. We want GPU 1 with MIG Dev 2, so
MIG-64f11db9-b10b-5dd9-97d1-3c46450b9388:
[user@aura ~]$ nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-309d72fd-b4f8-d6e8-6a66-e3f2253e8540)
MIG 7g.80gb Device 0: (UUID: MIG-a5459e6a-b26d-5985-874c-528458a7728b)
GPU 1: NVIDIA A100 80GB PCIe (UUID: GPU-04712e69-7356-4de5-f983-84083131460e)
MIG 3g.40gb Device 0: (UUID: MIG-4f7fbfb7-a8a2-553d-875a-d9d56baf97a5)
MIG 2g.20gb Device 1: (UUID: MIG-bad562c5-744a-5742-be1a-c735195c52d0)
MIG 2g.20gb Device 2: (UUID: MIG-64f11db9-b10b-5dd9-97d1-3c46450b9388)
Now we set the environment variable
CUDA_VISIBLE_DEVICES and we can compute.
[user@aura ~]$ export CUDA_VISIBLE_DEVICES=MIG-64f11db9-b10b-5dd9-97d1-3c46450b9388
[user@aura ~]$ python main.py
We can monitor our computation using either the command
nvidia-smi or the graphical tool
nvitop (it is recommended to have a larger terminal window than 80x24). Due to the use of MIG, the monitoring tools are not able to display the GPU usage (
N/A), but only the memory usage.
If the existing configuration of MIG instances is not suitable for you, it can be changed by agreement at
unixbtSyvsuAK@fibJba2r4ef.muniGNRiX8tk_.cz if circumstances (other running computations) reasonably allow it.