You are here: Home ATLAS-BFG Running on NEMO

Running on NEMO

Running on NEMO

It is also possible to submit jobs to the NEMO cluster. The jobs will not run directly on the NEMO machines, but will be redirected to virtual machines, which provide (almost) the same setup which you are used to from the normal ATLAS-BFG worker nodes (more details below).

Submitting jobs

To submit Jobs to NEMO with slurm, use the partition nemo_vm_<group-id> using the sbatch option -p or --partition with argument nemo_vm_<group-id>. The <group-id> is either atlher, atljak, or atlsch. For example if you belong to the atlsch group:

sbatch -p nemo_vm_atlsch <script>


Submitting jobs to this partition will result in the startup of virtual machines according to the needs of the queue. The VMs are 4-core machines with a total memory of 22 GB. Whenever there are already VMs running with empty slots at the time of submission, your jobs will land there. Machines are usually drained after 24 hours, or after they are idle for ~15 minutes.

The startup time is about 2 minutes. If squeue gives you the reason PENDING (Resources), this can be due to the fact that there is a queue on NEMO. You can check the usage here: (Fig. 2: Current load). Also keep in mind that the mechanism which watches over the VMs starting and suspending will only start 10 machines per minute, so please be patient.

Accessing data

Data can be used only from the BeeGFS-based directory /work, which is mounted also on the BFG user interfaces (uis). Access to lustre (/storage) is not given from the virtual machines (at least not in a reliable way).

This has one additional feature: You can only use it with so-called workspaces. These have an expiration date since creation, with an initial maximum duration of 100 days which can be extended a maximum of 99 times.

Create a workspace with this command:

ws_allocate name 5

which creates a directory /work/ws/atlas/<userid>-name which is valid for 5 days until it expires and is removed. You can get an overview of your workspaces with ws_list.

A detailed documention can be found here.