Slurm - Revision history

Leo: /* priority/multifactor */

2023-08-31T17:13:58Z

priority/multifactor

← Older revision		Revision as of 17:13, 31 August 2023
Line 690:		Line 690:
	Weights 1 1000 100000		Weights 1 1000 100000
	\| lang = terminal		\| lang = terminal
	}}You can find the current fairshare factors for all users in your system with this nifty command, ~~derivied~~ from [https://researchcomputing.princeton.edu/priority Princeton's RC] documentation.		}}You can find the current fairshare factors for all users in your system with this nifty command, derived from [https://researchcomputing.princeton.edu/priority Princeton's RC] documentation (though this version supports long usernames).
	{{Highlight		{{Highlight
	\| code = $ join -1 4 -2 2 -o 2.7,1.4,2.1 \		\| code = $ join -1 1 -2 2 -o 2.7,1.1,2.1 \
	<(squeue {{!}} sort ~~-k 4~~) \		<( SQUEUE_FORMAT="%32u" squeue {{!}} sort {{!}} uniq ) \
	<(sshare -a {{!}} awk 'NF>=7' {{!}} grep -v class {{!}} sort -k 2) \		<(sshare -a --parsable {{!}} tr '{{!}}' ' ' {{!}} awk 'NF>=7' {{!}} grep -v class {{!}} sort -k 2) \
	{{!}} sort -r {{!}} uniq {{!}} cat -n {{!}} column -t		{{!}} sort -r {{!}} uniq {{!}} cat -n {{!}} column -t
	# Fairshare Username Account		# Fairshare Username Account

Leo: /* Tasks */

2023-03-23T18:04:23Z

Tasks

← Older revision		Revision as of 18:04, 23 March 2023
Line 69:		Line 69:
	\|-		\|-
	\|{{code\|1=scontrol update jobid=<jobid> TimeLimit=20-00:00:00}}		\|{{code\|1=scontrol update jobid=<jobid> TimeLimit=20-00:00:00}}
	\|~~Change~~ the time limit of an existing job.		\|Extending or change the time limit of an existing job.
	You can use scontrol updatejobid=$jobid to modify other editable attributes of a ~~pending~~ job.		You can use <code>scontrol updatejobid=$jobid</code> to modify other editable attributes of a job (as reported by something like <code>scontrol show jobid -dd $jobid</code>).
	\|-		\|-
	\|<code>scontrol update nodename=$nodelist state=fail reason="fail"</code>		\|<code>scontrol update nodename=$nodelist state=fail reason="fail"</code>

Leo: /* Prolog error */

2023-01-30T23:02:24Z

Prolog error

@@ Line 1,045: / Line 1,045: @@
 | lang = text
 }}
-You can find 'REQUEST_LAUNCH' in the [https://github.com/SchedMD/slurm/blob/master/src/slurmd/slurmd/req.c#L6131 req.c source code] to see that it's timing out waiting for another thread to run <code>rpc_prolog</code>. Perhaps there's a bug resulting in a race condition of some sort and the call's getting lost?
+You can find 'REQUEST_LAUNCH' in the [https://github.com/SchedMD/slurm/blob/master/src/slurmd/slurmd/req.c#L6131 req.c source code] to see that it's timing out waiting for another thread to run <code>rpc_prolog</code>.
 ==See Also==

Leo: /* Prolog error */

2023-01-30T22:45:20Z

Prolog error

@@ Line 954: / Line 954: @@
 | lang = text
 }}
-The prolog script we have on the system is extremely simple:
+The prolog script we have on the system is extremely simple and is likely not the cause (return code's OK, otherwise it would fail with something like <code>error: [job 17671841] prolog failed status=1:0</code>)
 {{Highlight
 | code = ## cat /etc/slurm/prologue.sh
@@ Line 990: / Line 990: @@
 mc51: [2023-01-28T13:01:24.446] error: Waiting for JobId=17580438 REQUEST_LAUNCH_PROLOG notification failed, giving up after 20 sec
 | lang = terminal
 }}
 ==See Also==

Leo at 20:36, 30 January 2023

2023-01-30T20:36:21Z

@@ Line 939: / Line 939: @@
 | code = # scontrol update nodename=$node state=down reason=hung_job
 # scontrol update nodename=$node state=resume
 | lang = terminal
 }}

Leo at 17:37, 7 November 2022

2022-11-07T17:37:35Z

← Older revision		Revision as of 17:37, 7 November 2022
Line 71:		Line 71:
	\|Change the time limit of an existing job.		\|Change the time limit of an existing job.
	You can use scontrol updatejobid=$jobid to modify other editable attributes of a pending job.		You can use scontrol updatejobid=$jobid to modify other editable attributes of a pending job.
			\|-
			\|<code>scontrol update nodename=$nodelist state=fail reason="fail"</code>
			\|Sets one or more nodes into the 'FAIL' state to prevent scheduling. Useful if some hardware on the node has failed.
	\|-		\|-
	\|<code>scontrol reboot reason="reboot" $nodelist</code>		\|<code>scontrol reboot reason="reboot" $nodelist</code>

Leo: /* System-related commands */

2022-11-07T17:34:46Z

System-related commands

← Older revision		Revision as of 17:34, 7 November 2022
Line 69:		Line 69:
	\|-		\|-
	\|{{code\|1=scontrol update jobid=<jobid> TimeLimit=20-00:00:00}}		\|{{code\|1=scontrol update jobid=<jobid> TimeLimit=20-00:00:00}}
	\|~~Modifies~~ an ~~attribute~~ of a pending job. ~~Eg. priorities, time limits, etc.~~		\|Change the time limit of an existing job.
			You can use scontrol updatejobid=$jobid to modify other editable attributes of a pending job.
			\|-
			\|<code>scontrol reboot reason="reboot" $nodelist</code>
			\|Reboots one or more nodes when they become idle
	\|-		\|-
	\|{{code\|sacctmgr show qos}}		\|{{code\|sacctmgr show qos}}

Leo: /* Quality of Service (QOS) */

2022-02-23T00:51:19Z

Quality of Service (QOS)

← Older revision		Revision as of 00:51, 23 February 2022
Line 258:		Line 258:

	=== Submitting a job ===		=== Submitting a job ===
	All work has to be submitted to the scheduler as a job. A job is a unit of work that the scheduler will orchestrate onto one of the compute nodes in the cluster. Depending on the cluster you're working on and how it's configured, a job typically need to specify their resource requirements (memory, CPU, GPU, nodes, licenses, etc.) and its job configuration (where to log to, who to email, where the working directory is, etc).		All work has to be submitted to the scheduler as a job. A job is a unit of work that the scheduler will orchestrate onto one of the compute nodes in the cluster. There are two types of jobs: interactive and batch jobs.

	There are two types of jobs: interactive and batch jobs.

	# Interactive jobs gives you an interactive shell to do your work in when the job is scheduled and is similar to just connecting to a host via SSH and running your work manually. This is typically useful if you need to try out a new program or need to debug a new workflow and haven't automated the tasks into a shell script.		# Interactive jobs gives you an interactive shell to do your work in when the job is scheduled and is similar to just connecting to a host via SSH and running your work manually. This is typically useful if you need to try out a new program or need to debug a new workflow and haven't automated the tasks into a shell script.
	# A batch job is a job that runs without user intervention by the system. Basically, you 'batch' a set of commands together into a shell script and have Slurm run the job for you when the cluster is available.		# A batch job is a job that runs without user intervention by the system. Basically, you 'batch' a set of commands together into a shell script and have Slurm run the job for you when the cluster is available.
			Before we jump into starting a job, we need to first understand how to specify a job's resource requirements (memory, CPU, GPU, nodes, licenses, etc.) and its job configuration (where to log to, who to email, where the working directory is, etc).

			==== Job configuration and resource requests ====
			When using the sbatch/srun/salloc commands to run a job, you will need to specify the job's configuration and resource requirements. Common flags you'll encounter are given in the table below. For interactive jobs, stick these options to the srun/salloc command. For batch jobs, add these to your job script with the <code>#SBATCH</code> directive.
			{\| class="wikitable"
			!Flag
			!Example
			!Description
			\|-
			\|<code>--time</code>
			\|<code>--time=96:00:00</code>
			\|The maximum duration of your job
			\|-
			\|<code>--partition / -p</code>
			\|<code>-p bigmem</code>
			\|The partition your job will use. Use <code>sinfo</code> to see what partitions cluster has. If not given, you will use your cluster's default partition.
			\|-
			\|<code>--account</code>
			\|<code>-a default</code>
			\|The account your job should run under. Check with your HPC staff. This is likely set for you by default.
			\|-
			\|<code>--qos</code>
			\|
			\|The QOS you want your job to run under. Check with your HPC staff. This is likely set for you by default.
			\|-
			\|<code>--nodes</code>
			\|<code>--nodes=1</code>
			\|The number of nodes your job will require
			\|-
			\|<code>--mem=<mem></code>
			\|<code>--mem=48gb</code>
			\|The amount of memory per node that your job need
			\|-
			\|<code>--ntasks=<tasks>, -n</code>
			\|<code>-n 1</code>
			\|The total number of tasks your job requires
			\|-
			\|<code>--nodes=<min>[-<max>], -N</code>
			\|<code>-N 1-2</code>
			\|The minimum (and optionally maximum) number of nodes your job should be given.
			\|-
			\|<code>--mem-per-cpu=</code>
			\|
			\|The amount of memory per cpu your job requires
			\|-
			\|<code>--ntasks-per-node=#</code>
			\|
			\|Number of tasks per node.
			\|-
			\|<code>--gres gpu:#</code>
			\|<code>--gres gpu:1</code>
			\|The number of GPUs per node you need in you job
			\|-
			\|<code>--exclusive</code>
			\|
			\|Job must be exclusive on the given nodes.
			\|-
			\|<code>--constraint=<feature></code>
			\|<code>--constraint=intel&gpu</code>
			\|Constrain your job to nodes supporting a particular feature set. You can find the available features your cluster has by running <code>sinfo -o "%20N %10c %10m %25f %10G"</code> and then look under the <code>AVAIL_FEATURES</code> column.
			\|}

	==== Interactive jobs ====		==== Interactive jobs ====
Line 270:		Line 329:
	There are two ways to start an interactive session:		There are two ways to start an interactive session:

	# <code>srun --pty bash</code> along with any of the ~~usual <code>sbatch</code> options~~. This command will create a 'step 0' and drop you in it (don't worry if that makes no sense). As a result, you won't be able to launch subsequent steps with additional resource allocations.		# <code>srun --pty bash</code> along with any job options (such as the ones listed above). This command will create a 'step 0' and drop you in it (don't worry if that makes no sense). As a result, you won't be able to launch subsequent steps with additional resource allocations.
	# <code>salloc</code> along with any ~~of the usual <code>sbatch</code>~~ options. This is typically the preferred option and a requirement if you need to spawn additional steps in your interactive job. If the <code>LaunchParameters=use_interactive_step</code> option is set, <code>salloc</code> will drop you on the first allocated node rather than on a node you ran <code>salloc</code> on.		# <code>salloc</code> along with any job options. This is typically the preferred option and a requirement if you need to spawn additional steps in your interactive job. If the <code>LaunchParameters=use_interactive_step</code> option is set, <code>salloc</code> will drop you on the first allocated node rather than on a node you ran <code>salloc</code> on.

	==== Batch jobs ====		==== Batch jobs ====
	Submitting a batch job is done by first writing a shell script with your commands. Next, you will need to specify any additional job parameters by using the <code>#SBATCH</code> ~~directives~~. While they appear to be comments, the Slurm scheduler will read these directives and add the specified arguments to your job accordingly. You may also specify these by passing the appropriate arguments to <code>sbatch</code> at the time of submission, but specifying the parameters within the job script is less error prone and makes it clear what resources are required.		Submitting a batch job is done by first writing a shell script with your commands. Next, you will need to specify any additional job parameters by using the <code>#SBATCH</code> directive. While they appear to be comments, the Slurm scheduler will read these directives and add the specified arguments to your job accordingly. You may also specify these by passing the appropriate arguments to <code>sbatch</code> at the time of submission, but specifying the parameters within the job script is less error prone and makes it clear what resources are required.

	An example job script is given below. Notice how we can specify the job name, resource using the <code>#SBATCH</code> directive and the appropriate sbatch arguments. If you want to see what other options are available, run <code>man sbatch</code>.{{highlight		An example job script is given below. Notice how we can specify the job name, resource using the <code>#SBATCH</code> directive and the appropriate sbatch arguments. If you want to see what other options are available, run <code>man sbatch</code>.{{highlight
Line 485:		Line 544:
	!Description		!Description
	\|-		\|-
	\|Job		\| rowspan="4" \|Job
	\|'''MaxCPUMinsPerJob'''		\|'''MaxCPUMinsPerJob'''
	\|Maximum amount of CPU * wall time any one job can have		\|Maximum amount of CPU * wall time any one job can have
	\|-		\|-
	\|
	\|'''MaxCpusPerJob'''		\|'''MaxCpusPerJob'''
	\|Maximum number of CPU's any one job can be allocated.		\|Maximum number of CPU's any one job can be allocated.
	\|-		\|-
	\|
	\|'''MaxNodesPerJob'''		\|'''MaxNodesPerJob'''
	\|Maximum number of nodes any one job can be allocated		\|Maximum number of nodes any one job can be allocated
	\|-		\|-
	\|
	\|'''MaxWallDurationPerJob'''		\|'''MaxWallDurationPerJob'''
	\|Wall clock limit for any jobs running with this QOS.		\|Wall clock limit for any jobs running with this QOS.
	\|-		\|-
	\|User		\| rowspan="4" \|User
	\|'''MaxCpusPerUser'''		\|'''MaxCpusPerUser'''
	\|Maximum number of CPU's any user with this QOS can be allocated.		\|Maximum number of CPU's any user with this QOS can be allocated.
	\|-		\|-
	\|
	\|'''MaxJobsPerUser'''		\|'''MaxJobsPerUser'''
	\|Maximum number of jobs a user can have pending or running		\|Maximum number of jobs a user can have pending or running
	\|-		\|-
	\|
	\|'''MaxNodesPerUser'''		\|'''MaxNodesPerUser'''
	\|Maximum number of nodes that can be allocated to any user		\|Maximum number of nodes that can be allocated to any user
	\|-		\|-
	\|
	\|'''MaxSubmitJobsPerUser'''		\|'''MaxSubmitJobsPerUser'''
	\|Maximum number of jobs that can be in the system		\|Maximum number of jobs that can be in the system

Leo: /* QOS */

2022-02-23T00:26:46Z

QOS

@@ Line 474: / Line 474: @@
 Impose limits{{highlight|lang=terminal|code=
 # sacctmgr modify user xxx set GrpTRES=CPU=1000 GrpTRESRunMin=CPU=2000000}}<br />
-===QOS===
+===Quality of Service (QOS)===
-QOS defines different job classes with different priority values (a factor taken into account by the priority plugin) and enforces resource restrictions.
+The quality of service levels allow for an administrator to specify additional factors used to determine a job's priority, whether the jobs will preempt other jobs, and any additional resource restrictions such as the total CPU time and number of jobs that a particular user/group/account can have.
 Job limits are outlined here: https://lost-contact.mit.edu/afs/pdc.kth.se/roots/ilse/v0.7/pdc/vol/slurm/2.3.1-1/src/slurm-2.3.1/doc/html/qos.shtml
@@ Line 832: / Line 872: @@
 PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
 defq*        up   infinite      1   idle node01
 }}

Leo: /* Submitting a job */

2022-02-22T21:27:22Z

Submitting a job

← Older revision		Revision as of 21:27, 22 February 2022
Line 258:		Line 258:

	=== Submitting a job ===		=== Submitting a job ===
	All work has to be submitted to the scheduler as a job. ~~There are two types~~ of ~~jobs: interactive~~ and ~~batch jobs~~.		All work has to be submitted to the scheduler as a job. A job is a unit of work that the scheduler will orchestrate onto one of the compute nodes in the cluster. Depending on the cluster you're working on and how it's configured, a job typically need to specify their resource requirements (memory, CPU, GPU, nodes, licenses, etc.) and its job configuration (where to log to, who to email, where the working directory is, etc).

	~~Interactive~~ jobs ~~gives you an~~ interactive ~~shell to do your work in when the job is scheduled~~ and is similar to just connecting to a host via SSH and running your work manually. This is typically useful if you need to try out a new program or need to debug a new workflow and haven't automated the tasks into a shell script.		There are two types of jobs: interactive and batch jobs.

	A batch job is a job that runs without user intervention by the system. Basically, you 'batch' a set of commands together into a shell script and have Slurm run the job for you when the cluster is available.		# Interactive jobs gives you an interactive shell to do your work in when the job is scheduled and is similar to just connecting to a host via SSH and running your work manually. This is typically useful if you need to try out a new program or need to debug a new workflow and haven't automated the tasks into a shell script.
			# A batch job is a job that runs without user intervention by the system. Basically, you 'batch' a set of commands together into a shell script and have Slurm run the job for you when the cluster is available.

	==== ~~An example batch~~ job ====		==== Interactive jobs ====
			Interactive jobs let you use the compute resources through an interactive shell. The scheduler will allocate the requested resources and drop you in an interactive session on one of the compute nodes in the cluster. Interactive jobs only do work when the user runs a command and only end when the user explicitly exits from the session or when it hits its time limit. As a result, interactive jobs are typically frowned upon by HPC admins since these jobs tend to tie up resources and become unused.

			There are two ways to start an interactive session:

			# <code>srun --pty bash</code> along with any of the usual <code>sbatch</code> options. This command will create a 'step 0' and drop you in it (don't worry if that makes no sense). As a result, you won't be able to launch subsequent steps with additional resource allocations.
			# <code>salloc</code> along with any of the usual <code>sbatch</code> options. This is typically the preferred option and a requirement if you need to spawn additional steps in your interactive job. If the <code>LaunchParameters=use_interactive_step</code> option is set, <code>salloc</code> will drop you on the first allocated node rather than on a node you ran <code>salloc</code> on.

			==== Batch jobs ====
	Submitting a batch job is done by first writing a shell script with your commands. Next, you will need to specify any additional job parameters by using the <code>#SBATCH</code> directives. While they appear to be comments, the Slurm scheduler will read these directives and add the specified arguments to your job accordingly. You may also specify these by passing the appropriate arguments to <code>sbatch</code> at the time of submission, but specifying the parameters within the job script is less error prone and makes it clear what resources are required.		Submitting a batch job is done by first writing a shell script with your commands. Next, you will need to specify any additional job parameters by using the <code>#SBATCH</code> directives. While they appear to be comments, the Slurm scheduler will read these directives and add the specified arguments to your job accordingly. You may also specify these by passing the appropriate arguments to <code>sbatch</code> at the time of submission, but specifying the parameters within the job script is less error prone and makes it clear what resources are required.