LSF - Revision history

Leo: Added HPC category

2021-12-30T05:15:23Z

Added HPC category

← Older revision		Revision as of 05:15, 30 December 2021
Line 235:		Line 235:
	[[Category:Linux]]		[[Category:Linux]]
	[[Category:LinuxUtilities]]		[[Category:LinuxUtilities]]
			[[Category:HPC]]

Leo: /* Turn bhist output into metrics */

2020-10-23T23:45:29Z

Turn bhist output into metrics

← Older revision		Revision as of 23:45, 23 October 2020
Line 149:		Line 149:
	Here's a craptastic script that's a work in progress that I hacked together to parse out the output from <code>bhist -b -d -w -n X -u all</code>. Where <code>-b</code> for brief, <code>-d</code> for only finished jobs, <code>-w</code> for wide format, which doesn't do what I think it does (it's still bounded by a fixed column width??), <code>-n</code> for the number of logs to parse, and <code>-u</code> for all users.		Here's a craptastic script that's a work in progress that I hacked together to parse out the output from <code>bhist -b -d -w -n X -u all</code>. Where <code>-b</code> for brief, <code>-d</code> for only finished jobs, <code>-w</code> for wide format, which doesn't do what I think it does (it's still bounded by a fixed column width??), <code>-n</code> for the number of logs to parse, and <code>-u</code> for all users.
	{{Highlight		{{Highlight
	\| code = ~~<nowiki>~~#!/bin/bash		\| code = #!/bin/bash

	cat /tmp/jobs \		cat /tmp/jobs \
	\| sed ':a;N;$!ba;s/\n //g' \		<nowiki>\| </nowiki> sed ':a;N;$!ba;s/\n //g' \
	\| awk '		<nowiki>\| </nowiki> awk '
	BEGIN {		BEGIN {

Line 223:		Line 223:
	printf("Pending: " $1 " running: " $3 " total: " $7 "\n\n")		printf("Pending: " $1 " running: " $3 " total: " $7 "\n\n")
	}		}
	~~</nowiki>~~
	\| lang = bash		\| lang = bash
	}}		}}

Leo: /* Turn bhist output into metrics */

2020-10-23T23:44:55Z

Turn bhist output into metrics

← Older revision		Revision as of 23:44, 23 October 2020
Line 149:		Line 149:
	Here's a craptastic script that's a work in progress that I hacked together to parse out the output from <code>bhist -b -d -w -n X -u all</code>. Where <code>-b</code> for brief, <code>-d</code> for only finished jobs, <code>-w</code> for wide format, which doesn't do what I think it does (it's still bounded by a fixed column width??), <code>-n</code> for the number of logs to parse, and <code>-u</code> for all users.		Here's a craptastic script that's a work in progress that I hacked together to parse out the output from <code>bhist -b -d -w -n X -u all</code>. Where <code>-b</code> for brief, <code>-d</code> for only finished jobs, <code>-w</code> for wide format, which doesn't do what I think it does (it's still bounded by a fixed column width??), <code>-n</code> for the number of logs to parse, and <code>-u</code> for all users.
	{{Highlight		{{Highlight
	\| code = #!/bin/bash		\| code = <nowiki>#!/bin/bash

	cat /tmp/jobs \		cat /tmp/jobs \
Line 223:		Line 223:
	printf("Pending: " $1 " running: " $3 " total: " $7 "\n\n")		printf("Pending: " $1 " running: " $3 " total: " $7 "\n\n")
	}		}
			</nowiki>
	\| lang = bash		\| lang = bash
	}}		}}

Leo: /* Metrics */

2020-10-23T23:44:15Z

Metrics

@@ Line 144: / Line 144: @@
          done
 </nowiki>
 }}

Leo: /* Tasks */

2020-06-22T18:32:41Z

Tasks

← Older revision		Revision as of 18:32, 22 June 2020
Line 1:		Line 1:
	Platform Load Sharing Facility (or simply LSF) is a job scheduler and workload manager developed by IBM. It is similar to [[Slurm]].		Platform Load Sharing Facility (or simply LSF) is a job scheduler and workload manager developed by IBM. It is similar to [[Slurm]].

	== Usage ==		==Usage==
	=== Tasks ===		===Tasks===
	{\| class="wikitable"		{\| class="wikitable"
	! Command		!Command
	! Description		!Description
	\|-		\|-
	\| {{code\|bjobs -u all -a}}		\|{{code\|bjobs -u all -a}}
	\| Shows all jobs of all users		\|Shows all jobs of all users
	\|-		\|-
	\| {{code\|bjobs ~~101 102}}~~		\|<code>bjobs -p -u all -a</code>
	\| Shows jobs with ~~job_ID 101 and 102~~		\|Shows all jobs with pending reason
	\|-		\|-
	\| {{code\|~~bsub~~}}		\|{{code\|bjobs 101 102}}
	\| ~~Submits a batch job~~		\|Shows jobs with job_ID 101 and 102
	\|-		\|-
	\| {{code\|~~bhosts~~}}		\|{{code\|bsub}}
	\| ~~Shows all host status~~		\|Submits a batch job
	\|-		\|-
	\| {{code\|~~badmin~~}}		\|{{code\|bhosts}}
	\| ~~LSF administration shell~~		\|Shows all host status
	\|-		\|-
	\| {{code\|bhist -la -u username}}		\|{{code\|badmin}}
	\| Shows previous jobs by username		\|LSF administration shell
			\|-
			\|{{code\|bhist -la -u username}}
			\|Shows previous jobs by username
	\|}		\|}

	=== Admin ===		===Admin===
	Use the {{code\|lsadmin}} and {{code\|badmin}} commands to do most admin related things.		Use the {{code\|lsadmin}} and {{code\|badmin}} commands to do most admin related things.

	==== Startup ====		====Startup====
	Use {{code\|lsfstartup}}.		Use {{code\|lsfstartup}}.

Line 47:		Line 50:
	}}		}}

	==== Restart ====		====Restart====
	Use {{code\|lsfrestart}}.		Use {{code\|lsfrestart}}.

Line 64:		Line 67:
	}}		}}

	==== Shutdown ====		====Shutdown====
	Use {{code\|lsfshutdown}} to prevent users from submitting jobs.		Use {{code\|lsfshutdown}} to prevent users from submitting jobs.

Line 75:		Line 78:


	== Tasks ==		==Tasks==
	=== Extend Job Run Time ===		===Extend Job Run Time===
	If a job was created with too little wall time (eg. {{code\|bsub -W}}), you will see the following output when listing jobs using {{code\|bjobs -WL, -WF, or -WP}}:		If a job was created with too little wall time (eg. {{code\|bsub -W}}), you will see the following output when listing jobs using {{code\|bjobs -WL, -WF, or -WP}}:

Line 89:		Line 92:

	The {{code\|TIME_LEFT}} column shows the time left in hours and minutes. The state is one of:		The {{code\|TIME_LEFT}} column shows the time left in hours and minutes. The state is one of:
	* {{code\|E}}: The job has an estimated run time that has not been exceeded.
	* {{code\|L}}: The job has a hard run time limit specified but either has no estimated run time or the estimated run time is more than the hard run time limit.		*{{code\|E}}: The job has an estimated run time that has not been exceeded.
	* {{code\|X}}: The job has exceeded its estimated run time and the time displayed is the time remaining until the job reaches its hard run time limit.		*{{code\|L}}: The job has a hard run time limit specified but either has no estimated run time or the estimated run time is more than the hard run time limit.
	* {{code\|-}}: A dash indicates that the job has no estimated run time and no run limit, or that it has exceeded its run time but does not have a hard limit and therefore runs until completion.		*{{code\|X}}: The job has exceeded its estimated run time and the time displayed is the time remaining until the job reaches its hard run time limit.
			*{{code\|-}}: A dash indicates that the job has no estimated run time and no run limit, or that it has exceeded its run time but does not have a hard limit and therefore runs until completion.

	A job's run time limit can be adjusted using {{code\|bmod -W HH:MM Job_ID}} or removed using {{code\|bmod -Wn Job_ID}} .		A job's run time limit can be adjusted using {{code\|bmod -W HH:MM Job_ID}} or removed using {{code\|bmod -Wn Job_ID}} .

	== Administration ==		==Administration==
	=== Logs ===		===Logs===
	LSF events and accounting logs are stored in {{code\|/usr/share/lsf/work/hostname/logdir}}. Logs can grow quite large and can fill the system drive if left unchecked.		LSF events and accounting logs are stored in {{code\|/usr/share/lsf/work/hostname/logdir}}. Logs can grow quite large and can fill the system drive if left unchecked.

Line 103:		Line 107:

	See:		See:
	* https://tin6150.github.io/psg/3rdParty/lsf4_userGuide/07-manage-lsbatch.html

	=== Metrics to InfluxDB ===		*https://tin6150.github.io/psg/3rdParty/lsf4_userGuide/07-manage-lsbatch.html

			===Metrics to InfluxDB===
	Here's a quick and dirty script to dump data into InfluxDB that captures CPU allocation by user/partition/status. This should allow graphing of node usage as CPU cores allocated per node and by user.		Here's a quick and dirty script to dump data into InfluxDB that captures CPU allocation by user/partition/status. This should allow graphing of node usage as CPU cores allocated per node and by user.

Line 141:		Line 146:
	}}		}}

	== See Also ==		==See Also==
	* [https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.2/lsf_command_ref/bjobs.1.html IBM Documentation: bjobs reference]
	* [https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.2/lsf_command_ref/lsadmin.8.html IBM Documentation: lsadmin reference]		*[https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.2/lsf_command_ref/bjobs.1.html IBM Documentation: bjobs reference]
	* https://bmi.cchmc.org/resources/software/lsf-examples		*[https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.2/lsf_command_ref/lsadmin.8.html IBM Documentation: lsadmin reference]
			*https://bmi.cchmc.org/resources/software/lsf-examples

	{{Navbox Linux}}[[Category:Linux]]		{{Navbox Linux}}
			[[Category:Linux]]
	[[Category:LinuxUtilities]]		[[Category:LinuxUtilities]]

Leo: Added categories

2020-06-15T00:20:03Z

Added categories

← Older revision		Revision as of 00:20, 15 June 2020
Line 145:		Line 145:
	* [https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.2/lsf_command_ref/lsadmin.8.html IBM Documentation: lsadmin reference]		* [https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.2/lsf_command_ref/lsadmin.8.html IBM Documentation: lsadmin reference]
	* https://bmi.cchmc.org/resources/software/lsf-examples		* https://bmi.cchmc.org/resources/software/lsf-examples

			{{Navbox Linux}}[[Category:Linux]]
			[[Category:LinuxUtilities]]

Leo: /* Restart */

2020-04-14T17:37:50Z

Restart

← Older revision		Revision as of 17:37, 14 April 2020
Line 51:		Line 51:

	If some hosts are in a 'closed' state, you may need to restart LIM and RES on the host.		If some hosts are in a 'closed' state, you may need to restart LIM and RES on the host.
			{{highlight\|lang=terminal\|code=
			node01# lsadmin resrestart
			node01# lsadmin limrestart
			}}

	~~Fix~~ '~~closed~~' ~~status by running on crick~~:
			You can also restart on all nodes with 'all':
	{{highlight\|lang=terminal\|code=		{{highlight\|lang=terminal\|code=
	## restart the 'lsf' service and also lim and res.		## restart the 'lsf' service and also lim and res.

Leo at 19:45, 31 March 2020

2020-03-31T19:45:39Z

← Older revision		Revision as of 19:45, 31 March 2020
Line 21:		Line 21:
	\| {{code\|badmin}}		\| {{code\|badmin}}
	\| LSF administration shell		\| LSF administration shell
			\|-
			\| {{code\|bhist -la -u username}}
			\| Shows previous jobs by username
	\|}		\|}

Leo: /* Administration */

2020-02-07T22:39:09Z

Administration

← Older revision		Revision as of 22:39, 7 February 2020
Line 97:		Line 97:
	* https://tin6150.github.io/psg/3rdParty/lsf4_userGuide/07-manage-lsbatch.html		* https://tin6150.github.io/psg/3rdParty/lsf4_userGuide/07-manage-lsbatch.html

			=== Metrics to InfluxDB ===
			Here's a quick and dirty script to dump data into InfluxDB that captures CPU allocation by user/partition/status. This should allow graphing of node usage as CPU cores allocated per node and by user.

			{{highlight\|lang=bash\|code=
			<nowiki>
			#!/bin/bash

			bjobs -u all -a \
			\| tail -n+2 \
			\| awk '
			{
			# node
			split($6, z, "*");

			# No "*" means 1 core
			if ("" ~ z[2]) {
			node=z[1];
			x[$2, $3, $4, node] += 1
			} else {
			# Number of cpus given
			node=z[2];
			x[$2, $3, $4, node] += z[1]
			}
			}
			END {
			for (i in x) {
			split(i, y, SUBSEP);
			print "lsf,username="y[1]",status="y[2]",queue="y[3]",node="y[4]" value="x[y[1], y[2], y[3], y[4]]
			}
			}' \
			\| while read i ; do
			echo "curl -X POST 'http://influxdb/write?db=lsf' --data-binary '$i `date +%s"000000000"`'" \| sh
			done
			</nowiki>
			}}

	== See Also ==		== See Also ==

Leo at 00:04, 30 January 2020

2020-01-30T00:04:11Z

← Older revision		Revision as of 00:04, 30 January 2020
Line 87:		Line 87:

	A job's run time limit can be adjusted using {{code\|bmod -W HH:MM Job_ID}} or removed using {{code\|bmod -Wn Job_ID}} .		A job's run time limit can be adjusted using {{code\|bmod -W HH:MM Job_ID}} or removed using {{code\|bmod -Wn Job_ID}} .

			== Administration ==
			=== Logs ===
			LSF events and accounting logs are stored in {{code\|/usr/share/lsf/work/hostname/logdir}}. Logs can grow quite large and can fill the system drive if left unchecked.

			Delete old {{code\|lsb.events.X}} files.

			See:
			* https://tin6150.github.io/psg/3rdParty/lsf4_userGuide/07-manage-lsbatch.html


	== See Also ==		== See Also ==