View source for Slurm

Slurm is a open source job scheduler that runs on Linux and is typically used in high performance computing environments.

==Usage Cheat Sheet==
{| class="wikitable"
! width="60%" |Command
!Description
|-
|{{code|squeue}}
|Displays jobs in the queue.
|-
|{{code|sbatch [--test-only] job.sh}}
|Submit a job
|-
|{{code|sinfo}}
|Show the status of nodes.
|-
|{{code|scancel}}
|Cancel a job.
|-
|{{code|scontrol show jobid -dd jobid}}
|Displays a particular job by ID
|-
|{{code|1=scontrol update jobid=<jobid> TimeLimit=20-00:00:00}}
|Modifies an attribute of a pending job. Eg. priorities, time limits, etc.
|-
|{{code|sacctmgr show qos}}
|Show partition preemption settings and submission limits
|-
|{{code|1=sacctmgr show assoc format=cluster,user,qos}}
|Shows user and QOS associations
|-
|{{code|1=sacctmgr modify user where name=username set qos=normal}}
|Set a user to the normal QOS. You can pass multiple QOS as a comma separated list.
|}

You can define a custom squeue format by exporting a {{code|SQUEUE_FORMAT}} variable. Eg. {{code|1=export SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M %.6D %.20R %q"}}.

==Installation==
Installation on a RHEL/CentOS system involves building the RPM packages and then installing them.  The full instructions can be found at https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#installing-rpms

Obtain the latest Slurm packages at https://www.schedmd.com/downloads.php, then install prerequsites and build the RPM package:
{{highlight|lang=terminal|code=
# yum install rpm-build gcc openssl openssl-devel libssh2-devel pam-devel numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel gtk2-devel libssh2-devel libibmad libibumad perl-Switch perl-ExtUtils-MakeMaker
# yum install munge-devel munge-libs mariadb-server mariadb-devel man2html
# export VER=19.05.3-2
# rpmbuild -ta slurm-$VER.tar.bz2
}}

===Configuration===
The main Slurm configuration file docs: 

*https://slurm.schedmd.com/slurm.conf.html

Custom resources such as GPUs are defined in {{code|gres.conf}}. See: https://slurm.schedmd.com/gres.conf.html

====Nodes====
The cluster name is defined with {{code|1=ClusterName=name}}.

Nodes are defined near the end of the file. Format is:
{{highlight|lang=text|code=
NodeName=node01 NodeAddr=<ip addr node01> CPUs=4 State=UNKNOWN
NodeName=node02 NodeAddr=<ip addr node02> CPUs=4 State=UNKNOWN
NodeName=node03 NodeAddr=<ip addr node03> CPUs=4 State=UNKNOWN
NodeName=node04 NodeAddr=<ip addr node04> CPUs=4 State=UNKNOWN
}}

====Partitions====
A Slurm partition defines a group of nodes that a job can run on with additional attributes such as maximum allowed CPU time, the job priority, which users have access, etc.  You may have multiple partitions defined for a set of nodes.  An example use case for having multiple partitions target the same set of nodes would be to create a priority queue that allows jobs run before other jobs in another partition witht a normal priority.

When jobs are submitted to a certain partition, the scheduler will schedule the job to a node defined in the partition.  Jobs that do not specify a partition will use the default partition.

The default partition can be defined with:
{{highlight|lang=text|code=
PartitionName=cluster-name Nodes=nodes[0-5] Default=Yes
}}

====Resource Selector Algorithm====
The {{code|SelectType}} defines which resource selector algorithm the scheduler will use. 

*Multiple jobs per node by allocating each job with individual "consumable resources" such as CPU cores using {{code|select/cons_res}} or {{code|select/cons_tres}}
*Single jobs per node by allocating whole nodes using {{code|select/linear}}, or by specifying {{code|1=OverSubscribe=Exclusive}}

The consumable resource that the algorithm should count is defined with {{code|SelectTypeParameters}}.

For example, to allow jobs to be scheduled to nodes based on number of available CPU cores on each node:
{{highlight|lang=text|code=
SelectType=select/cons_res
SelectTypeParameters=CR_Core
}}

====CGroups====
Slurm supports cgroups which allows the control of resources a job has access to. This is useful to limit the amount of memory, CPU, swap, or devices that a job can access.

CGroups configs are loaded from {{code|/etc/slurm/cgroup.conf}}. If running an older version of Slurm on a newer system, you may need to configure the cgroup path from {{code|/cgroup}} to {{code|/sys/fs/cgroup}}. 

If you don't care for cgroups, you may leave it disabled.

Example cgroup config:
{{highlight|lang=text|code=
CgroupMountpoint="/sys/fs/cgroup"
CgroupAutomount=yes
CgroupReleaseAgentDir="/etc/slurm-llnl/cgroup"
AllowedDevicesFile="/etc/slurm-llnl/cgroup_allowed_devices_file.conf"
ConstrainCores=no
TaskAffinity=no
ConstrainRAMSpace=yes
ConstrainSwapSpace=no
ConstrainDevices=no
AllowedRamSpace=100
AllowedSwapSpace=0
MaxRAMPercent=100
MaxSwapPercent=100
MinRAMSpace=30
}}

The whitelist device list:
{{highlight|lang=text|code=/dev/null
/dev/urandom
/dev/zero
/dev/sda*
/dev/cpu/*/*
/dev/pts/*
/any-other-mounts
}}

====Example Config====
Example configuration on a single-node instance:
{{highlight|lang=text|style=max-height:350px;overflow:scroll;|code=
ControlMachine=dragen
ControlAddr=dragen
AuthType=auth/munge
CryptoType=crypto/munge
JobRequeue=0
MaxJobCount=500000
MpiDefault=none
ReturnToService=0
SallocDefaultCommand="srun -n1 -N1 --mem-per-cpu=0 --pty --preserve-env --mpi=none $SHELL"
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
DefMemPerCPU=1024
FastSchedule=1
MaxArraySize=65535
SchedulerType=sched/backfill
SelectType=select/linear
PriorityType=priority/multifactor #basic means strict fifo
PriorityDecayHalfLife=7-0
PriorityFavorSmall=YES
PriorityWeightAge=1000
PriorityWeightFairshare=100000
ClusterName=dragen
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurmd.log
NodeName=dragen NodeAddr=dragen Sockets=2 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=256000
PartitionName=defq Default=YES MinNodes=1 DefaultTime=5-00:00:00 AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 DisableRootJobs=NO RootOnly=NO Hidden=NO Shared=NO GraceTime=0 Pren
}}

==Usage==

===Submitting a Job (with demo)===
Use the {{code|sbatch job.sh}} command to submit a job. A job can be a simple shell script, but can also include job parameters with the addition of {{code|#SBATCH}} directives.

For example:
{{highlight|lang=bash|code=
#!/bin/bash
#SBATCH --job-name=demo-slurm-job
#SBATCH --workdir=/home/leo/some-project
#SBATCH --error=error.txt
#SBATCH --output=stdout.txt
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=0-01:00:00
#SBATCH --nodes=1
## #SBATCH --exclude=bad-node[00-05]

echo "Hello. Start at `date`"
sleep 30
echo "Hello. Done at `date`"

}}

Submit the job by running {{code|sbatch job.sh}} and then view its status with {{code|squeue}}.
{{highlight|lang=terminal|code=
# squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                27      defq  test.sh     root  R       0:01      1 node01
}}

Any job output is placed by default in the current working directory as {{code|slurm-JOB_ID.out}}.

===Retrieving Job Information===
All jobs can be listed using {{code|squeue [-u username] [-t RUNNING{{!}}PENDING{{!}}COMPLETED] [-p partition] }}

The reason column specifies the reason for a job's state.

*If there is no reason, the scheduler hasn't attended to your submission yet.
*Resources means your job is waiting for an appropriate compute node to open.
*Priority indicates your priority is lower relative to others being scheduled.

A job's configured parameters can be displayed by running {{code|scontrol show jobid -dd jobid}} and can be updated with {{code|scontrol update}}.

See: https://www.rc.fas.harvard.edu/resources/faq/fix_pending_job/

===Modifying a Job===
Any parameters of a job can be edited. First, determine what values need to be updated for a particular job:

{{highlight|lang=terminal|code=
# scontrol show jobid -dd <jobid>
}}

To update a specific value (such as {{code|TimeLimit}}), run:

{{highlight|lang=terminal|code=
# scontrol update jobid=<jobid> TimeLimit=20-00:00:00
}}

To update the {{code|TimeLimit}} to 20 days for all jobs by a particular user:
{{highlight|lang=terminal|code=
# squeue {{!}} grep username {{!}} awk '{print $1}' {{!}} while read i ; do scontrol update jobid=$i TimeLimit=20-00:00:00 ; done
}}

==Administration==
===Database Configuration===
Show the current accounting configuration with {{code|sacctmgr}} show configuration

Associations are accounting records that links a specific user (by username, cluster, account, partition) to some attribute relating to their account.

{{highlight|lang=terminal|code=
# sacctmgr show associations
# sacctmgr show associations format=account,user,fairshare,QOS,GrpTRES,GrpTRESRunMin
}}

===Account Management===
A Slurm Account (henceforth simply as account) is like a UNIX group and consists of one or many users. Accounts can be nested in a hierarchical manner. A user must belong to at least one <code>DefaultAccount</code>.

When either adding or modifying an account, the following {{code|sacctmgr}} options are available:

*'''Cluster=''' Only add this account to these clusters. The account is added to all defined clusters by default.
*'''Description=''' Description of the account. (Default is account name)
*'''Name=''' Name of account. Note the name must be unique and can not represent different bank accounts at different points in the account hierarchy
*'''Organization=''' Organization of the account. (Default is parent account unless parent account is root then organization is set to the account name.)
*'''Parent=''' Make this account a child of this other account (already added).

{{highlight|lang=terminal|code=
# sacctmgr add account dtu Description="DTU departments" Organization=dtu
# sacctmgr add account fysik Description="Physics department" Organization=fysik parent=dtu
# sacctmgr add account deptx Description="X department" Organization=deptx parent=dtu

# sacctmgr show account
# sacctmgr show account -s   # Show also associations in the accounts}}

When a user belonging to multiple accounts submit a job, they may specify which account the job belongs to with {{code|sbatch -A account}}.

===User Management===
{| class="wikitable"
|+
!Task
!Command
|-
|To create a new user, the username and the default account are required
|{{highlight|lang=terminal|code=
# sacctmgr create user name=xxx DefaultAccount=yyy}}
|-
|Set an user to an account
|{{highlight|lang=terminal|code=
# sacctmgr add user xxx Account=zzzz}}
|-
|Change the default account
|{{highlight|lang=terminal|code=
# sacctmgr add user xxx DefaultAccount=zzzz}}
|-
|Remove an account
|{{highlight|lang=terminal|code=
# sacctmgr remove user where default=test}}
|-
|List users
|{{highlight|lang=terminal|code=
# sacctmgr show user
# sacctmgr show user <username>

## Show more information with -s
# sacctmgr show user -s
# sacctmgr show user -s <username>}}
|-
|The fairshare and other settings for the non-default account may be configured:
|{{highlight|lang=terminal|code=
# sacctmgr modify user where name=xxx account=zzzz set fairshare=0}}
|}
User parameters that can be modified or set:

*'''Account=''' Account(s) to add user to (see also DefaultAccount).
*'''AdminLevel=''' This field is used to allow a user to add accounting privileges to this user. Valid options are:
**None
**Operator: can add, modify, and remove any database object (user, account, etc), and add other operators. On a SlurmDBD served slurmctld these users can:
**View information that is blocked to regular uses by a PrivateData flag (see slurm.conf).
**Create/Alter/Delete Reservations
**Admin: These users have the same level of privileges as an operator in the database. They can also alter anything on a served slurmctld as if they were the slurm user or root.
*'''Cluster=''' Only add to accounts on these clusters (default is all clusters)
*'''DefaultAccount=''' Default account for the user, used when no account is specified when a job is submitted. (Required on creation)
*'''DefaultWCKey=''' Default WCkey for the user, used when no WCkey is specified when a job is submitted. (Only used when tracking WCkey.)
*'''Name=''' User name
*'''Partition=''' Name of Slurm partition this association applies to.


===Limits===
Trackable RESources (TRES) https://slurm.schedmd.com/tres.html

Enable in <code>slurm.conf</code>
{{highlight|lang=terminal|code=
AccountingStorageEnforce=limits}}

Impose limits
{{highlight|lang=terminal|code=
# sacctmgr modify user xxx set GrpTRES=CPU=1000 GrpTRESRunMin=CPU=2000000}}


===QOS===
QOS defines different job classes with different priorities and resource limits.  A priority with the greater number has higher priority.

{{highlight|lang=terminal|code=
# sacctmgr add qos high priority=10 MaxTRESPerUser=CPU=256

# sacctmgr show qos
# sacctmgr show qos format=name
# sacctmgr --noheader show qos format=name

# sacctmgr -i modify user where name=XXXX set QOS=normal,high
# sacctmgr -i modify user where name=XXXX set QOS+=high

## User's default QOS can be set
# sacctmgr -i modify user where name=XXXX set DefaultQOS=normal}}

Users must submit jobs to non-default QOS by specifying the qos in sbatch:
{{highlight|lang=terminal|code=
# sbatch --qos=high ...}}


==Tasks==
===Adding nodes to Slurm===

#Ensure {{code|munge}} and {{code|slurm}} users have the same UID on both login node and new node.
#Munge from login node to new node: {{code|munge -n | ssh new-node unmunge}}
#Add the node to {{code|slurm.conf}} on the login node, then redistribute the file to all other nodes
#Restart {{code|slurmd}} on all nodes
#Restart {{code|slurmctld}} on the login node
#Ensure all nodes are visible with {{code|sinfo -lN}}.

===Cron-like Jobs===
A job that runs and re-schedules itself at a later time is possible. Example from University of Chicago RCC: https://rcc.uchicago.edu/docs/running-jobs/cron/index.html#cron-jobs

This requires a script or program that prints the next timestamp for the job to run from a given cronjob schedule.

Slurm accepts the following for the --begin option:
{{highlight|lang=text|code=
   --begin=16:00
   --begin=now+1hour
   --begin=now+60           (seconds by default)
   --begin=12/26            (Next December 26)
   --begin=2020-01-20T12:34:00
}}

An example submission would look like:
{{highlight|lang=text|code=
#!/bin/bash

#SBATCH --time=00:05:00
#SBATCH --output=cron.log
#SBATCH --open-mode=append
#SBATCH --account=cron-account
#SBATCH --partition=cron
#SBATCH --qos=cron

# Specify a valid Cron string for the schedule. This specifies that
# the Cron job run once per day at 5:15a.
SCHEDULE='15 5 * * *'

# Here is an example of a simple command that prints the host name and
# the date and time.
echo "Hello on $(hostname) at $(date)."

# This schedules the next run.
sbatch --quiet --begin=$(next-cron-time "$SCHEDULE") cron.sbatch
}}

==Troubleshooting==
Issues can be troubleshooted by looking at the logs at {{code|/var/log/slurmd.log}}.

===Missing Cgroup namespace 'freezer'===
{{highlight|lang=text|code=
[2019-10-30T12:45:32.578] error: cgroup namespace 'freezer' not mounted. aborting
[2019-10-30T12:45:32.578] error: unable to create freezer cgroup namespace
[2019-10-30T12:45:32.578] error: Couldn't load specified plugin name for proctrack/cgroup: Plugin init() callback failed
[2019-10-30T12:45:32.578] error: cannot create proctrack context for proctrack/cgroup
[2019-10-30T12:45:32.578] error: slurmd initialization failed
}}

You need to define the cgroup path location in the {{code|cgroup.conf}} configuration file:
{{highlight|lang=text|code=
CgroupMountpoint=/sys/fs/cgroup
CgroupAutomount=yes
}}

===Fix a downed node===
If a node is reporting as down from {{code|sinfo}} even though slurmd and munge are running, the node might require a manual update to the idle state. For example, I saw the following.
{{highlight|lang=terminal|code=
# sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite      1   down node01
}}

Ensure that the node is reachable using the ping command:
{{highlight|lang=terminal|code=
# scontrol ping
Slurmctld(primary/backup) at node01/(NULL) are UP/DOWN
}}

Then, update the status with {{code|scontrol update}}:
{{highlight|lang=terminal|code=
# scontrol
scontrol: update NodeName=node01 State=RESUME
}}

If the node is functional, the state should return to {{code|idle}} and should begin accepting new jobs.
{{highlight|lang=terminal|code=
# sinfo -a
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite      1   idle node01
}}

==See Also==

*https://www.rc.fas.harvard.edu/resources/documentation/convenient-slurm-commands/
*https://rcc.uchicago.edu/docs/using-midway/index.html
*https://www.dkrz.de/up/systems/mistral/running-jobs/slurm-introduction
*https://wiki.fysik.dtu.dk/niflheim/Slurm_accounting
*https://lost-contact.mit.edu/afs/pdc.kth.se/roots/ilse/v0.7/pdc/vol/slurm/14.11.5/build/slurm-14.11.5/doc/html/accounting.shtml
*Yale has some good notes on Slurm too: [https://docs.ycrc.yale.edu/clusters-at-yale/job-scheduling/resource-usage/#seff https://docs.ycrc.yale.edu/clusters-at-yale/job-scheduling/resource-usage/]

Installing Slurm on a Raspberry Pi cluster:

*https://medium.com/@glmdev/building-a-raspberry-pi-cluster-784f0df9afbd
*https://medium.com/@glmdev/building-a-raspberry-pi-cluster-aaa8d1f3d2ca
*https://medium.com/@glmdev/building-a-raspberry-pi-cluster-f5f2446702e8

{{Navbox Linux}}
[[Category:Linux]]
[[Category:LinuxUtilities]]