Slurm: Difference between revisions

From Leo's Notes
This page was last edited on 19 June 2020, at 16:26.
← Older editNewer edit →
VisualWikitext
m Added categories
Line 1: Line 1:


== Cheat Sheet ==
==Cheat Sheet==
{| class="wikitable"
{| class="wikitable"
! width="60%" | Command
! width="60%" |Command
! Description
!Description
|-
|-
| {{code|squeue}}  
|{{code|squeue}}
| Displays jobs in the queue.  
|Displays jobs in the queue.
|-
|-
| {{code|sbatch [--test-only] job.sh}}  
|{{code|sbatch [--test-only] job.sh}}
| Submit a job
|Submit a job
|-
|-
| {{code|sinfo}}  
|{{code|sinfo}}
| Show the status of nodes.
|Show the status of nodes.
|-
|-
| {{code|scancel}}  
|{{code|scancel}}
| Cancel a job.
|Cancel a job.
|-
|-
| {{code|scontrol show jobid -dd jobid}}  
|{{code|scontrol show jobid -dd jobid}}
| Displays a particular job by ID
|Displays a particular job by ID
|-
|-
| {{code|1=scontrol update jobid=<jobid> TimeLimit=20-00:00:00}}
|{{code|1=scontrol update jobid=<jobid> TimeLimit=20-00:00:00}}
| Modifies an attribute of a pending job. Eg. priorities, time limits, etc.
|Modifies an attribute of a pending job. Eg. priorities, time limits, etc.
|-
|-
| {{code|sacctmgr show qos}}  
|{{code|sacctmgr show qos}}
| Show partition preemption settings and submission limits
|Show partition preemption settings and submission limits
|-
|-
| {{code|1=sacctmgr show assoc format=cluster,user,qos}}
|{{code|1=sacctmgr show assoc format=cluster,user,qos}}
| Shows user and QOS associations
|Shows user and QOS associations
|-
|-
| {{code|1=sacctmgr modify user where name=username set qos=normal}}
|{{code|1=sacctmgr modify user where name=username set qos=normal}}
| Set a user to the normal QOS. You can pass multiple QOS as a comma separated list.
|Set a user to the normal QOS. You can pass multiple QOS as a comma separated list.
|}
|}


You can define a custom squeue format by exporting a {{code|SQUEUE_FORMAT}} variable. Eg. {{code|1=export SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M %.6D %.20R %q"}}.
You can define a custom squeue format by exporting a {{code|SQUEUE_FORMAT}} variable. Eg. {{code|1=export SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M %.6D %.20R %q"}}.


== Installation ==
==Installation==
Installation on a RHEL/CentOS system involves building the RPM packages and then installing them.  The full instructions can be found at https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#installing-rpms
Installation on a RHEL/CentOS system involves building the RPM packages and then installing them.  The full instructions can be found at https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#installing-rpms


Line 46: Line 46:
}}
}}


=== Configuration ===
===Configuration===
The main Slurm configuration file docs:  
The main Slurm configuration file docs:  
https://slurm.schedmd.com/slurm.conf.html
https://slurm.schedmd.com/slurm.conf.html
Line 53: Line 53:
See: https://slurm.schedmd.com/gres.conf.html
See: https://slurm.schedmd.com/gres.conf.html


==== Nodes ====
====Nodes====
The cluster name is defined with {{code|1=ClusterName=name}}.
The cluster name is defined with {{code|1=ClusterName=name}}.


Line 64: Line 64:
}}
}}


==== Partitions ====
====Partitions====
A Slurm partition defines a group of nodes that a job can run on with additional attributes such as maximum allowed CPU time, the job priority, which users have access, etc.  You may have multiple partitions defined for a set of nodes.  An example use case for having multiple partitions target the same set of nodes would be to create a priority queue that allows jobs run before other jobs in another partition witht a normal priority.
A Slurm partition defines a group of nodes that a job can run on with additional attributes such as maximum allowed CPU time, the job priority, which users have access, etc.  You may have multiple partitions defined for a set of nodes.  An example use case for having multiple partitions target the same set of nodes would be to create a priority queue that allows jobs run before other jobs in another partition witht a normal priority.


Line 74: Line 74:
}}
}}


==== Resource Selector Algorithm ====
====Resource Selector Algorithm====
The {{code|SelectType}} defines which resource selector algorithm the scheduler will use.  
The {{code|SelectType}} defines which resource selector algorithm the scheduler will use.  
* Multiple jobs per node by allocating each job with individual "consumable resources" such as CPU cores using {{code|select/cons_res}} or {{code|select/cons_tres}}
 
* Single jobs per node by allocating whole nodes using {{code|select/linear}}, or by specifying {{code|1=OverSubscribe=Exclusive}}
*Multiple jobs per node by allocating each job with individual "consumable resources" such as CPU cores using {{code|select/cons_res}} or {{code|select/cons_tres}}
*Single jobs per node by allocating whole nodes using {{code|select/linear}}, or by specifying {{code|1=OverSubscribe=Exclusive}}


The consumable resource that the algorithm should count is defined with {{code|SelectTypeParameters}}.
The consumable resource that the algorithm should count is defined with {{code|SelectTypeParameters}}.
Line 87: Line 88:
}}
}}


==== CGroups ====
====CGroups====
Slurm supports cgroups which allows the control of resources a job has access to. This is useful to limit the amount of memory, CPU, swap, or devices that a job can access.
Slurm supports cgroups which allows the control of resources a job has access to. This is useful to limit the amount of memory, CPU, swap, or devices that a job can access.


Line 122: Line 123:
}}
}}


==== Example Config ====
====Example Config====
Example configuration on a single-node instance:
Example configuration on a single-node instance:
{{highlight|lang=text|style=max-height:350px;overflow:scroll;|code=
{{highlight|lang=text|style=max-height:350px;overflow:scroll;|code=
Line 170: Line 171:
}}
}}


== Usage ==
==Usage==


=== Job Submission ===
===Job Submission===
Use the {{code|sbatch job.sh}} command to submit a job. A job can be a simple shell script, but can also include job parameters with the addition of {{code|#SBATCH}} directives.
Use the {{code|sbatch job.sh}} command to submit a job. A job can be a simple shell script, but can also include job parameters with the addition of {{code|#SBATCH}} directives.


Line 203: Line 204:
Any job output is placed by default in the current working directory as {{code|slurm-JOB_ID.out}}.
Any job output is placed by default in the current working directory as {{code|slurm-JOB_ID.out}}.


=== Job Information ===
===Job Information===
All jobs can be listed using {{code|squeue [-u username] [-t RUNNING{{!}}PENDING{{!}}COMPLETED] [-p partition] }}
All jobs can be listed using {{code|squeue [-u username] [-t RUNNING{{!}}PENDING{{!}}COMPLETED] [-p partition] }}


The reason column specifies the reason for a job's state.
The reason column specifies the reason for a job's state.
* If there is no reason, the scheduler hasn't attended to your submission yet.
 
* Resources means your job is waiting for an appropriate compute node to open.
*If there is no reason, the scheduler hasn't attended to your submission yet.
* Priority indicates your priority is lower relative to others being scheduled.
*Resources means your job is waiting for an appropriate compute node to open.
*Priority indicates your priority is lower relative to others being scheduled.


A job's configured parameters can be displayed by running {{code|scontrol show jobid -dd jobid}} and can be updated with {{code|scontrol update}}.
A job's configured parameters can be displayed by running {{code|scontrol show jobid -dd jobid}} and can be updated with {{code|scontrol update}}.
Line 215: Line 217:
See: https://www.rc.fas.harvard.edu/resources/faq/fix_pending_job/
See: https://www.rc.fas.harvard.edu/resources/faq/fix_pending_job/


=== Modifying a Job ===
===Modifying a Job===
Any parameters of a job can be edited. First, determine what values need to be updated for a particular job:
Any parameters of a job can be edited. First, determine what values need to be updated for a particular job:


Line 233: Line 235:
}}
}}


== Administration ==
==Administration==
=== Database Configuration ===
===Database Configuration===
Show the current accounting configuration with {{code|sacctmgr}} show configuration
Show the current accounting configuration with {{code|sacctmgr}} show configuration


Line 244: Line 246:
}}
}}


=== Accounts ===
===Account Management===
A Slurm Account (henceforth simply as account) is like a UNIX group and consists of one or many users. Accounts can be nested in a hierarchical manner. A user must belong to at least one DefaultAccount.
A Slurm Account (henceforth simply as account) is like a UNIX group and consists of one or many users. Accounts can be nested in a hierarchical manner. A user must belong to at least one <code>DefaultAccount</code>.


When either adding or modifying an account, the following {{code|sacctmgr}} options are available:
When either adding or modifying an account, the following {{code|sacctmgr}} options are available:


* '''Cluster=''' Only add this account to these clusters. The account is added to all defined clusters by default.
*'''Cluster=''' Only add this account to these clusters. The account is added to all defined clusters by default.
* '''Description=''' Description of the account. (Default is account name)
*'''Description=''' Description of the account. (Default is account name)
* '''Name=''' Name of account. Note the name must be unique and can not represent different bank accounts at different points in the account hierarchy
*'''Name=''' Name of account. Note the name must be unique and can not represent different bank accounts at different points in the account hierarchy
* '''Organization=''' Organization of the account. (Default is parent account unless parent account is root then organization is set to the account name.)
*'''Organization=''' Organization of the account. (Default is parent account unless parent account is root then organization is set to the account name.)
* '''Parent=''' Make this account a child of this other account (already added).
*'''Parent=''' Make this account a child of this other account (already added).


{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
Line 265: Line 267:
When a user belonging to multiple accounts submit a job, they may specify which account the job belongs to with {{code|sbatch -A account}}.
When a user belonging to multiple accounts submit a job, they may specify which account the job belongs to with {{code|sbatch -A account}}.


=== Users ===
===User Management===
{{highlight|lang=terminal|code=
{| class="wikitable"
sacctmgr create user name=xxx DefaultAccount=yyy}}
|+
 
!Task
Add a user to an account
!Command
{{highlight|lang=terminal|code=
|-
|To create a new user, the username and the default account are required
|{{highlight|lang=terminal|code=
# sacctmgr create user name=xxx DefaultAccount=yyy}}
|-
|Set an user to an account
|{{highlight|lang=terminal|code=
# sacctmgr add user xxx Account=zzzz}}
# sacctmgr add user xxx Account=zzzz}}
 
|-
Change the default account
|Change the default account
{{highlight|lang=terminal|code=
|{{highlight|lang=terminal|code=
# sacctmgr add user xxx DefaultAccount=zzzz}}
# sacctmgr add user xxx DefaultAccount=zzzz}}
 
|-
Remove an account  
|Remove an account
{{highlight|lang=terminal|code=
|{{highlight|lang=terminal|code=
# sacctmgr remove user where default=test}}
# sacctmgr remove user where default=test}}
|-
|List users
|{{highlight|lang=terminal|code=
# sacctmgr show user
# sacctmgr show user <username>


The fairshare and other settings for the non-default account may be configured:
## Show more information with -s
{{highlight|lang=terminal|code=
# sacctmgr show user -s
# sacctmgr show user -s <username>}}
|-
|The fairshare and other settings for the non-default account may be configured:
|{{highlight|lang=terminal|code=
# sacctmgr modify user where name=xxx account=zzzz set fairshare=0}}
# sacctmgr modify user where name=xxx account=zzzz set fairshare=0}}
|}
User parameters that can be modified or set:


List users by:
*'''Account=''' Account(s) to add user to (see also DefaultAccount).
{{highlight|lang=terminal|code=
*'''AdminLevel=''' This field is used to allow a user to add accounting privileges to this user. Valid options are:
# sacctmgr show user
**None
# sacctmgr show user -s
**Operator: can add, modify, and remove any database object (user, account, etc), and add other operators. On a SlurmDBD served slurmctld these users can:
# sacctmgr show account -s xxx}}
**View information that is blocked to regular uses by a PrivateData flag (see slurm.conf).
 
**Create/Alter/Delete Reservations
* Account= Account(s) to add user to (see also DefaultAccount).
**Admin: These users have the same level of privileges as an operator in the database. They can also alter anything on a served slurmctld as if they were the slurm user or root.
* '''AdminLevel=''' This field is used to allow a user to add accounting privileges to this user. Valid options are:
*'''Cluster=''' Only add to accounts on these clusters (default is all clusters)
** None
*'''DefaultAccount=''' Default account for the user, used when no account is specified when a job is submitted. (Required on creation)
** Operator: can add, modify, and remove any database object (user, account, etc), and add other operators. On a SlurmDBD served slurmctld these users can:
*'''DefaultWCKey=''' Default WCkey for the user, used when no WCkey is specified when a job is submitted. (Only used when tracking WCkey.)
** View information that is blocked to regular uses by a PrivateData flag (see slurm.conf).
*'''Name=''' User name
** Create/Alter/Delete Reservations
*'''Partition=''' Name of Slurm partition this association applies to.
** Admin: These users have the same level of privileges as an operator in the database. They can also alter anything on a served slurmctld as if they were the slurm user or root.
* '''Cluster=''' Only add to accounts on these clusters (default is all clusters)
* '''DefaultAccount=''' Default account for the user, used when no account is specified when a job is submitted. (Required on creation)
* '''DefaultWCKey=''' Default WCkey for the user, used when no WCkey is specified when a job is submitted. (Only used when tracking WCkey.)
* '''Name=''' User name
* '''Partition=''' Name of Slurm partition this association applies to.




=== Limits ===
===Limits===
Trackable RESources (TRES) https://slurm.schedmd.com/tres.html
Trackable RESources (TRES) https://slurm.schedmd.com/tres.html


Enable in slurm.conf
Enable in <code>slurm.conf</code>
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
AccountingStorageEnforce=limits}}
AccountingStorageEnforce=limits}}
Line 317: Line 330:




 
===QOS===
=== QOS ===
QOS defines different job classes with different priorities and resource limits.  A priority with the greater number has higher priority.
QOS defines different job classes with different priorities and resource limits.  A priority with the greater number has higher priority.


Line 339: Line 351:




== Tasks ==
==Tasks==
=== Adding nodes to Slurm ===
===Adding nodes to Slurm===
# Ensure {{code|munge}} and {{code|slurm}} users have the same UID on both login node and new node.
# Munge from login node to new node: {{code|munge -n | ssh new-node unmunge}}
# Add the node to {{code|slurm.conf}} on the login node, then redistribute the file to all other nodes
# Restart {{code|slurmd}} on all nodes
# Restart {{code|slurmctld}} on the login node
# Ensure all nodes are visible with {{code|sinfo -lN}}.


=== Cron-like Jobs ===
#Ensure {{code|munge}} and {{code|slurm}} users have the same UID on both login node and new node.
#Munge from login node to new node: {{code|munge -n | ssh new-node unmunge}}
#Add the node to {{code|slurm.conf}} on the login node, then redistribute the file to all other nodes
#Restart {{code|slurmd}} on all nodes
#Restart {{code|slurmctld}} on the login node
#Ensure all nodes are visible with {{code|sinfo -lN}}.
 
===Cron-like Jobs===
A job that runs and re-schedules itself at a later time is possible. Example from University of Chicago RCC: https://rcc.uchicago.edu/docs/running-jobs/cron/index.html#cron-jobs
A job that runs and re-schedules itself at a later time is possible. Example from University of Chicago RCC: https://rcc.uchicago.edu/docs/running-jobs/cron/index.html#cron-jobs


Line 385: Line 398:
}}
}}


== Troubleshooting ==
==Troubleshooting==
Issues can be troubleshooted by looking at the logs at {{code|/var/log/slurmd.log}}.
Issues can be troubleshooted by looking at the logs at {{code|/var/log/slurmd.log}}.


=== Missing Cgroup namespace 'freezer' ===
===Missing Cgroup namespace 'freezer'===
{{highlight|lang=text|code=
{{highlight|lang=text|code=
[2019-10-30T12:45:32.578] error: cgroup namespace 'freezer' not mounted. aborting
[2019-10-30T12:45:32.578] error: cgroup namespace 'freezer' not mounted. aborting
Line 403: Line 416:
}}
}}


=== Fix a downed node ===
===Fix a downed node===
If a node is reporting as down from {{code|sinfo}} even though slurmd and munge are running, the node might require a manual update to the idle state. For example, I saw the following.
If a node is reporting as down from {{code|sinfo}} even though slurmd and munge are running, the node might require a manual update to the idle state. For example, I saw the following.
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
Line 430: Line 443:
}}
}}


== See Also ==
==See Also==
* https://www.rc.fas.harvard.edu/resources/documentation/convenient-slurm-commands/
 
* https://rcc.uchicago.edu/docs/using-midway/index.html
*https://www.rc.fas.harvard.edu/resources/documentation/convenient-slurm-commands/
* https://www.dkrz.de/up/systems/mistral/running-jobs/slurm-introduction
*https://rcc.uchicago.edu/docs/using-midway/index.html
* https://wiki.fysik.dtu.dk/niflheim/Slurm_accounting
*https://www.dkrz.de/up/systems/mistral/running-jobs/slurm-introduction
*https://wiki.fysik.dtu.dk/niflheim/Slurm_accounting


Installing Slurm on a Raspberry Pi cluster:
Installing Slurm on a Raspberry Pi cluster:
* https://medium.com/@glmdev/building-a-raspberry-pi-cluster-784f0df9afbd
* https://medium.com/@glmdev/building-a-raspberry-pi-cluster-aaa8d1f3d2ca
* https://medium.com/@glmdev/building-a-raspberry-pi-cluster-f5f2446702e8


{{Navbox Linux}}[[Category:Linux]]
*https://medium.com/@glmdev/building-a-raspberry-pi-cluster-784f0df9afbd
*https://medium.com/@glmdev/building-a-raspberry-pi-cluster-aaa8d1f3d2ca
*https://medium.com/@glmdev/building-a-raspberry-pi-cluster-f5f2446702e8
 
{{Navbox Linux}}
[[Category:Linux]]
[[Category:LinuxUtilities]]
[[Category:LinuxUtilities]]

Revision as of 16:26, 19 June 2020

Cheat Sheet

Command Description
squeue Displays jobs in the queue.
sbatch [--test-only] job.sh Submit a job
sinfo Show the status of nodes.
scancel Cancel a job.
scontrol show jobid -dd jobid Displays a particular job by ID
scontrol update jobid=<jobid> TimeLimit=20-00:00:00 Modifies an attribute of a pending job. Eg. priorities, time limits, etc.
sacctmgr show qos Show partition preemption settings and submission limits
sacctmgr show assoc format=cluster,user,qos Shows user and QOS associations
sacctmgr modify user where name=username set qos=normal Set a user to the normal QOS. You can pass multiple QOS as a comma separated list.

You can define a custom squeue format by exporting a SQUEUE_FORMAT variable. Eg. export SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M %.6D %.20R %q".

Installation

Installation on a RHEL/CentOS system involves building the RPM packages and then installing them. The full instructions can be found at https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#installing-rpms

Obtain the latest Slurm packages at https://www.schedmd.com/downloads.php, then install prerequsites and build the RPM package: 

# yum install rpm-build gcc openssl openssl-devel libssh2-devel pam-devel numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel gtk2-devel libssh2-devel libibmad libibumad perl-Switch perl-ExtUtils-MakeMaker
# yum install munge-devel munge-libs mariadb-server mariadb-devel man2html
# export VER=19.05.3-2
# rpmbuild -ta slurm-$VER.tar.bz2

Configuration

The main Slurm configuration file docs: https://slurm.schedmd.com/slurm.conf.html

Custom resources such as GPUs are defined in gres.conf. See: https://slurm.schedmd.com/gres.conf.html

Nodes

The cluster name is defined with ClusterName=name.

Nodes are defined near the end of the file. Format is: 

NodeName=node01 NodeAddr=<ip addr node01> CPUs=4 State=UNKNOWN
NodeName=node02 NodeAddr=<ip addr node02> CPUs=4 State=UNKNOWN
NodeName=node03 NodeAddr=<ip addr node03> CPUs=4 State=UNKNOWN
NodeName=node04 NodeAddr=<ip addr node04> CPUs=4 State=UNKNOWN

Partitions

A Slurm partition defines a group of nodes that a job can run on with additional attributes such as maximum allowed CPU time, the job priority, which users have access, etc. You may have multiple partitions defined for a set of nodes. An example use case for having multiple partitions target the same set of nodes would be to create a priority queue that allows jobs run before other jobs in another partition witht a normal priority.

When jobs are submitted to a certain partition, the scheduler will schedule the job to a node defined in the partition. Jobs that do not specify a partition will use the default partition.

The default partition can be defined with: 

PartitionName=cluster-name Nodes=nodes[0-5] Default=Yes

Resource Selector Algorithm

The SelectType defines which resource selector algorithm the scheduler will use.

  • Multiple jobs per node by allocating each job with individual "consumable resources" such as CPU cores using select/cons_res or select/cons_tres
  • Single jobs per node by allocating whole nodes using select/linear, or by specifying OverSubscribe=Exclusive

The consumable resource that the algorithm should count is defined with SelectTypeParameters.

For example, to allow jobs to be scheduled to nodes based on number of available CPU cores on each node: 

SelectType=select/cons_res
SelectTypeParameters=CR_Core

CGroups

Slurm supports cgroups which allows the control of resources a job has access to. This is useful to limit the amount of memory, CPU, swap, or devices that a job can access.

CGroups configs are loaded from /etc/slurm/cgroup.conf. If running an older version of Slurm on a newer system, you may need to configure the cgroup path from /cgroup to /sys/fs/cgroup.

If you don't care for cgroups, you may leave it disabled.

Example cgroup config: 

CgroupMountpoint="/sys/fs/cgroup"
CgroupAutomount=yes
CgroupReleaseAgentDir="/etc/slurm-llnl/cgroup"
AllowedDevicesFile="/etc/slurm-llnl/cgroup_allowed_devices_file.conf"
ConstrainCores=no
TaskAffinity=no
ConstrainRAMSpace=yes
ConstrainSwapSpace=no
ConstrainDevices=no
AllowedRamSpace=100
AllowedSwapSpace=0
MaxRAMPercent=100
MaxSwapPercent=100
MinRAMSpace=30

The whitelist device list: 

/dev/null
/dev/urandom
/dev/zero
/dev/sda*
/dev/cpu/*/*
/dev/pts/*
/any-other-mounts

Example Config

Example configuration on a single-node instance: 

ControlMachine=dragen
ControlAddr=dragen
AuthType=auth/munge
CryptoType=crypto/munge
JobRequeue=0
MaxJobCount=500000
MpiDefault=none
ReturnToService=0
SallocDefaultCommand="srun -n1 -N1 --mem-per-cpu=0 --pty --preserve-env --mpi=none $SHELL"
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
DefMemPerCPU=1024
FastSchedule=1
MaxArraySize=65535
SchedulerType=sched/backfill
SelectType=select/linear
PriorityType=priority/multifactor #basic means strict fifo
PriorityDecayHalfLife=7-0
PriorityFavorSmall=YES
PriorityWeightAge=1000
PriorityWeightFairshare=100000
ClusterName=dragen
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurmd.log
NodeName=dragen NodeAddr=dragen Sockets=2 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=256000
PartitionName=defq Default=YES MinNodes=1 DefaultTime=5-00:00:00 AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 DisableRootJobs=NO RootOnly=NO Hidden=NO Shared=NO GraceTime=0 Pren

Usage

Job Submission

Use the sbatch job.sh command to submit a job. A job can be a simple shell script, but can also include job parameters with the addition of #SBATCH directives.

For example: 

#!/bin/bash
#SBATCH --job-name=slurm-job
#SBATCH --workdir=/home/leo/some-project
#SBATCH --error=error.txt
#SBATCH --output=stdout.txt
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=12
#SBATCH --time=99-00:00:00
#SBATCH --nodes=1
#SBATCH --exclude=bad-node[00-05]

echo "Hello. Start at `date`"
sleep 30
echo "Hello. Done at `date`"

Submit the job by running sbatch job.sh and then view its status with squeue. 

# squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                27      defq  test.sh     root  R       0:01      1 node01

Any job output is placed by default in the current working directory as slurm-JOB_ID.out.

Job Information

All jobs can be listed using squeue [-u username] [-t RUNNING|PENDING|COMPLETED] [-p partition]

The reason column specifies the reason for a job's state.

  • If there is no reason, the scheduler hasn't attended to your submission yet.
  • Resources means your job is waiting for an appropriate compute node to open.
  • Priority indicates your priority is lower relative to others being scheduled.

A job's configured parameters can be displayed by running scontrol show jobid -dd jobid and can be updated with scontrol update.

See: https://www.rc.fas.harvard.edu/resources/faq/fix_pending_job/

Modifying a Job

Any parameters of a job can be edited. First, determine what values need to be updated for a particular job: 

# scontrol show jobid -dd <jobid>

To update a specific value (such as TimeLimit), run: 

# scontrol update jobid=<jobid> TimeLimit=20-00:00:00

To update the TimeLimit to 20 days for all jobs by a particular user: 

# squeue | grep username | awk '{print $1}' | while read i ; do scontrol update jobid=$i TimeLimit=20-00:00:00 ; done

Administration

Database Configuration

Show the current accounting configuration with sacctmgr show configuration

Associations are accounting records that links a specific user (by username, cluster, account, partition) to some attribute relating to their account. 

# sacctmgr show associations
# sacctmgr show associations format=account,user,fairshare,QOS,GrpTRES,GrpTRESRunMin

Account Management

A Slurm Account (henceforth simply as account) is like a UNIX group and consists of one or many users. Accounts can be nested in a hierarchical manner. A user must belong to at least one DefaultAccount.

When either adding or modifying an account, the following sacctmgr options are available:

  • Cluster= Only add this account to these clusters. The account is added to all defined clusters by default.
  • Description= Description of the account. (Default is account name)
  • Name= Name of account. Note the name must be unique and can not represent different bank accounts at different points in the account hierarchy
  • Organization= Organization of the account. (Default is parent account unless parent account is root then organization is set to the account name.)
  • Parent= Make this account a child of this other account (already added).
# sacctmgr add account dtu Description="DTU departments" Organization=dtu
# sacctmgr add account fysik Description="Physics department" Organization=fysik parent=dtu
# sacctmgr add account deptx Description="X department" Organization=deptx parent=dtu

# sacctmgr show account
# sacctmgr show account -s   # Show also associations in the accounts

When a user belonging to multiple accounts submit a job, they may specify which account the job belongs to with sbatch -A account.

User Management

Task Command
To create a new user, the username and the default account are required 
# sacctmgr create user name=xxx DefaultAccount=yyy
Set an user to an account 
# sacctmgr add user xxx Account=zzzz
Change the default account 
# sacctmgr add user xxx DefaultAccount=zzzz
Remove an account 
# sacctmgr remove user where default=test
List users 
# sacctmgr show user
# sacctmgr show user <username>

## Show more information with -s
# sacctmgr show user -s
# sacctmgr show user -s <username>
The fairshare and other settings for the non-default account may be configured: 
# sacctmgr modify user where name=xxx account=zzzz set fairshare=0

User parameters that can be modified or set:

  • Account= Account(s) to add user to (see also DefaultAccount).
  • AdminLevel= This field is used to allow a user to add accounting privileges to this user. Valid options are:
    • None
    • Operator: can add, modify, and remove any database object (user, account, etc), and add other operators. On a SlurmDBD served slurmctld these users can:
    • View information that is blocked to regular uses by a PrivateData flag (see slurm.conf).
    • Create/Alter/Delete Reservations
    • Admin: These users have the same level of privileges as an operator in the database. They can also alter anything on a served slurmctld as if they were the slurm user or root.
  • Cluster= Only add to accounts on these clusters (default is all clusters)
  • DefaultAccount= Default account for the user, used when no account is specified when a job is submitted. (Required on creation)
  • DefaultWCKey= Default WCkey for the user, used when no WCkey is specified when a job is submitted. (Only used when tracking WCkey.)
  • Name= User name
  • Partition= Name of Slurm partition this association applies to.


Limits

Trackable RESources (TRES) https://slurm.schedmd.com/tres.html

Enable in slurm.conf 

AccountingStorageEnforce=limits

Impose limits 

# sacctmgr modify user xxx set GrpTRES=CPU=1000 GrpTRESRunMin=CPU=2000000


QOS

QOS defines different job classes with different priorities and resource limits. A priority with the greater number has higher priority. 

# sacctmgr add qos high priority=10 MaxTRESPerUser=CPU=256

# sacctmgr show qos
# sacctmgr show qos format=name
# sacctmgr --noheader show qos format=name

# sacctmgr -i modify user where name=XXXX set QOS=normal,high
# sacctmgr -i modify user where name=XXXX set QOS+=high

## User's default QOS can be set
# sacctmgr -i modify user where name=XXXX set DefaultQOS=normal

Users must submit jobs to non-default QOS by specifying the qos in sbatch: 

# sbatch --qos=high ...


Tasks

Adding nodes to Slurm

  1. Ensure munge and slurm users have the same UID on both login node and new node.
  2. Munge from login node to new node: munge -n
  3. Add the node to slurm.conf on the login node, then redistribute the file to all other nodes
  4. Restart slurmd on all nodes
  5. Restart slurmctld on the login node
  6. Ensure all nodes are visible with sinfo -lN.

Cron-like Jobs

A job that runs and re-schedules itself at a later time is possible. Example from University of Chicago RCC: https://rcc.uchicago.edu/docs/running-jobs/cron/index.html#cron-jobs

This requires a script or program that prints the next timestamp for the job to run from a given cronjob schedule.

Slurm accepts the following for the --begin option: 

--begin=16:00
   --begin=now+1hour
   --begin=now+60           (seconds by default)
   --begin=12/26            (Next December 26)
   --begin=2020-01-20T12:34:00

An example submission would look like: 

#!/bin/bash

#SBATCH --time=00:05:00
#SBATCH --output=cron.log
#SBATCH --open-mode=append
#SBATCH --account=cron-account
#SBATCH --partition=cron
#SBATCH --qos=cron

# Specify a valid Cron string for the schedule. This specifies that
# the Cron job run once per day at 5:15a.
SCHEDULE='15 5 * * *'

# Here is an example of a simple command that prints the host name and
# the date and time.
echo "Hello on $(hostname) at $(date)."

# This schedules the next run.
sbatch --quiet --begin=$(next-cron-time "$SCHEDULE") cron.sbatch

Troubleshooting

Issues can be troubleshooted by looking at the logs at /var/log/slurmd.log.

Missing Cgroup namespace 'freezer'

[2019-10-30T12:45:32.578] error: cgroup namespace 'freezer' not mounted. aborting
[2019-10-30T12:45:32.578] error: unable to create freezer cgroup namespace
[2019-10-30T12:45:32.578] error: Couldn't load specified plugin name for proctrack/cgroup: Plugin init() callback failed
[2019-10-30T12:45:32.578] error: cannot create proctrack context for proctrack/cgroup
[2019-10-30T12:45:32.578] error: slurmd initialization failed

You need to define the cgroup path location in the cgroup.conf configuration file: 

CgroupMountpoint=/sys/fs/cgroup
CgroupAutomount=yes

Fix a downed node

If a node is reporting as down from sinfo even though slurmd and munge are running, the node might require a manual update to the idle state. For example, I saw the following. 

# sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite      1   down node01

Ensure that the node is reachable using the ping command: 

# scontrol ping
Slurmctld(primary/backup) at node01/(NULL) are UP/DOWN

Then, update the status with scontrol update: 

# scontrol
scontrol: update NodeName=node01 State=RESUME

If the node is functional, the state should return to idle and should begin accepting new jobs. 

# sinfo -a
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite      1   idle node01

See Also

Installing Slurm on a Raspberry Pi cluster:

Various Linux Notes
Linux Tools and Utilites
Package Management
Linux Distributions
Networking
Containers

Recent Changes

Recent Changes under Linux

Recent Changes under Guides

Ad-Free + Tracker-Free

I respect your privacy. This site does not serve ads, use trackers/analytics, or load resources from 3rd party websites.

Cookies may be created by MediaWiki but will contain no user identifiable information.