Slurm: Difference between revisions

From Leo's Notes
This page was last edited on 8 January 2020, at 21:58.
← Older editNewer edit →
VisualWikitext
mNo edit summary
Line 98: Line 98:
See: https://www.rc.fas.harvard.edu/resources/faq/fix_pending_job/
See: https://www.rc.fas.harvard.edu/resources/faq/fix_pending_job/


== Modifying a Job ==
=== Modifying a Job ===
Any parameters of a job can be edited. First, determine what values need to be updated for a particular job:
Any parameters of a job can be edited. First, determine what values need to be updated for a particular job:


Revision as of 21:58, 8 January 2020

Installation

See:

Install prerequsites, then build the RPM. 

# yum install rpm-build gcc openssl openssl-devel libssh2-devel pam-devel numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel gtk2-devel libssh2-devel libibmad libibumad perl-Switch perl-ExtUtils-MakeMaker
# yum install munge-devel munge-libs mariadb-server mariadb-devel man2html
# export VER=19.05.3-2
# rpmbuild -ta slurm-$VER.tar.bz2

The latest Slurm package can be downloaded from https://www.schedmd.com/downloads.php

Configuration

The main slurm configuration file docs: https://slurm.schedmd.com/slurm.conf.html

Custom resources such as GPUs are defined in gres.conf. See: https://slurm.schedmd.com/gres.conf.html

Cgroups configs are loaded from /etc/slurm/cgroup.conf. If running an older version of Slurm on a newer system, you may need to configure the cgroup path from /cgroup to /sys/fs/cgroup.

Example configuration: 

ControlMachine=dragen
ControlAddr=dragen
AuthType=auth/munge
CryptoType=crypto/munge
JobRequeue=0
MaxJobCount=500000
MpiDefault=none
ReturnToService=0
SallocDefaultCommand="srun -n1 -N1 --mem-per-cpu=0 --pty --preserve-env --mpi=none $SHELL"
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
DefMemPerCPU=1024
FastSchedule=1
MaxArraySize=65535
SchedulerType=sched/backfill
SelectType=select/linear
PriorityType=priority/multifactor #basic means strict fifo
PriorityDecayHalfLife=7-0
PriorityFavorSmall=YES
PriorityWeightAge=1000
PriorityWeightFairshare=100000
ClusterName=dragen
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurmd.log
NodeName=dragen NodeAddr=dragen Sockets=2 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=256000
PartitionName=defq Default=YES MinNodes=1 DefaultTime=5-00:00:00 AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 DisableRootJobs=NO RootOnly=NO Hidden=NO Shared=NO GraceTime=0 Pren


Quick Usage

Start Slurmd and Slurmctld.

  • squeue to see jobs on the queue.
  • sbatch [--test-only] job.sh to submit a job.
  • sinfo to show the status of nodes.
  • scancel to cancel a job.
  • scontrol show jobid -dd jobid to see a particular job by ID

Output from jobs will be placed on the current working directory as slurm-JOB_ID.out.

Job Submission

Use the sbatch job.sh command to submit a job. A job looks like a shell script with the addition of #SBATCH definitions that outline certain parameters.

Job Information

All jobs can be listed using squeue [-u username] [-t RUNNING|PENDING|COMPLETED] [-p partition]

The reason column specifies the reason for a job's state.

  • If there is no reason, the scheduler hasn't attended to your submission yet.
  • Resources means your job is waiting for an appropriate compute node to open.
  • Priority indicates your priority is lower relative to others being scheduled.

A job's configured parameters can be displayed by running scontrol show jobid -dd jobid and can be updated with scontrol update.

See: https://www.rc.fas.harvard.edu/resources/faq/fix_pending_job/

Modifying a Job

Any parameters of a job can be edited. First, determine what values need to be updated for a particular job: 

# scontrol show jobid -dd <jobid>

To update a specific value (such as TimeLimit), run: 

# scontrol update jobid=<jobid> TimeLimit=20-00:00:00

To update the TimeLimit to 20 days for all jobs by a particular user: 

# squeue | grep username | awk '{print $1}' | while read i ; do scontrol update jobid=$i TimeLimit=20-00:00:00 ; done

Troubleshooting

Issues can be troubleshooted by looking at the logs at /var/log/slurmd.log. 

[2019-10-30T12:45:32.578] error: cgroup namespace 'freezer' not mounted. aborting
[2019-10-30T12:45:32.578] error: unable to create freezer cgroup namespace
[2019-10-30T12:45:32.578] error: Couldn't load specified plugin name for proctrack/cgroup: Plugin init() callback failed
[2019-10-30T12:45:32.578] error: cannot create proctrack context for proctrack/cgroup
[2019-10-30T12:45:32.578] error: slurmd initialization failed

You need to define the cgroup path location in the cgroup.conf configuration file: 

CgroupMountpoint=/sys/fs/cgroup
CgroupAutomount=yes

See Also