Open OnDemand - Revision history

Leo: Reverted edits by 174.241.28.194 (talk) to last revision by Leo

2022-02-07T02:39:19Z

Reverted edits by 174.241.28.194 (talk) to last revision by Leo

← Older revision		Revision as of 02:39, 7 February 2022
Line 228:		Line 228:
	A workaround would be to increase the client header request size from the default 8k to something higher. To do this:		A workaround would be to increase the client header request size from the default 8k to something higher. To do this:

	#		# Add to <code>/etc/httpd/conf.d/ood-portal.conf</code> within the VirtualHost tags this: <code>LimitRequestFieldSize 65536</code>.
	# Create an additional config file at <code>/var/lib/ondemand-nginx/config/apps/sys/large-header.conf</code> with <code>large_client_header_buffers 8 64k;</code>.		# Create an additional config file at <code>/var/lib/ondemand-nginx/config/apps/sys/large-header.conf</code> with <code>large_client_header_buffers 8 64k;</code>.

174.241.28.194: /* 400 - Bad Request */

2022-02-06T21:46:28Z

400 - Bad Request

← Older revision		Revision as of 21:46, 6 February 2022
Line 228:		Line 228:
	A workaround would be to increase the client header request size from the default 8k to something higher. To do this:		A workaround would be to increase the client header request size from the default 8k to something higher. To do this:

	# ~~Add to <code>/etc/httpd/conf.d/ood-portal.conf</code> within the VirtualHost tags this: <code>LimitRequestFieldSize 65536</code>.~~		#
	# Create an additional config file at <code>/var/lib/ondemand-nginx/config/apps/sys/large-header.conf</code> with <code>large_client_header_buffers 8 64k;</code>.		# Create an additional config file at <code>/var/lib/ondemand-nginx/config/apps/sys/large-header.conf</code> with <code>large_client_header_buffers 8 64k;</code>.

Leo: Added categories and navbox

2021-12-30T01:13:21Z

Added categories and navbox

← Older revision		Revision as of 01:13, 30 December 2021
Line 251:		Line 251:

	*Documentation: https://osc.github.io/ood-documentation/latest/index.html		*Documentation: https://osc.github.io/ood-documentation/latest/index.html

			{{Navbox Linux}}
			[[Category:Linux]]
			[[Category:HPC]]

Leo: /* Troubleshooting */

2021-11-15T22:35:17Z

Troubleshooting

← Older revision		Revision as of 22:35, 15 November 2021
Line 158:		Line 158:
	For more information, review the docs at https://osc.github.io/ood-documentation-test/init-2.1/authentication.html		For more information, review the docs at https://osc.github.io/ood-documentation-test/init-2.1/authentication.html

	=== Plain old PAM ===		===Plain old PAM===
	To have Apache handle authentication using PAM, define the following lines in <code>ood_porta.yml</code>:		To have Apache handle authentication using PAM, define the following lines in <code>ood_porta.yml</code>:
	{{Highlight		{{Highlight
Line 221:		Line 221:
	*<code>/var/www/ood/apps/dev/$username/gateway</code> (symlinked to user's home directory <code>~/ondemand/dev</code>)		*<code>/var/www/ood/apps/dev/$username/gateway</code> (symlinked to user's home directory <code>~/ondemand/dev</code>)

	<br />		=== 400 - Bad Request ===
			Apache and the nginx instance running the PUN supports a 8k maximum client header request. Exceeding this limit will result in a <code>400 - Bad Request</code> error being returned by Apache to the client. A likely cause for exceeding this limit is with excessive amounts of cookies.

			If there are a large number of OpenID state cookies, then this error is likely a symptom of another issue. For more information on this issue, see: https://github.com/zmartzone/mod_auth_openidc/wiki/Cookies

			A workaround would be to increase the client header request size from the default 8k to something higher. To do this:

			# Add to <code>/etc/httpd/conf.d/ood-portal.conf</code> within the VirtualHost tags this: <code>LimitRequestFieldSize 65536</code>.
			# Create an additional config file at <code>/var/lib/ondemand-nginx/config/apps/sys/large-header.conf</code> with <code>large_client_header_buffers 8 64k;</code>.

	==Implementations==		==Implementations==

Leo: /* Authentication */

2021-11-15T21:35:00Z

Authentication

← Older revision		Revision as of 21:35, 15 November 2021
Line 27:		Line 27:
	===Remote Desktop===		===Remote Desktop===

	==== Setup ====		====Setup====
	Clone one of the existing <code>bc_desktop</code> apps and edit <code>form.yml</code>. Set the cluster value to a cluster you have configured in OnDemand (hint: what you have in the <code>clusters.d</code> directory). You may want to add/remove the CPU, memory, partition fields as desired. The default bc_desktop app includes startup scripts for various desktop managers but I hardcoded this to always use XFCE for my use-case.		Clone one of the existing <code>bc_desktop</code> apps and edit <code>form.yml</code>. Set the cluster value to a cluster you have configured in OnDemand (hint: what you have in the <code>clusters.d</code> directory). You may want to add/remove the CPU, memory, partition fields as desired. The default bc_desktop app includes startup scripts for various desktop managers but I hardcoded this to always use XFCE for my use-case.
	On the compute nodes that are intended to run the VNC sessions, you install the desktop environment (XFCE in my case), TurboVNC, and websockify. On a CentOS 8 machine, this was accomplished manually with:		On the compute nodes that are intended to run the VNC sessions, you install the desktop environment (XFCE in my case), TurboVNC, and websockify. On a CentOS 8 machine, this was accomplished manually with:
Line 50:		Line 50:
	If everything works correctly, requesting for a desktop will submit a job via the scheduler which launches Xvnc and proxies it via websockify to OnDemand which then renders it via noVNC.		If everything works correctly, requesting for a desktop will submit a job via the scheduler which launches Xvnc and proxies it via websockify to OnDemand which then renders it via noVNC.

	==== VirtualGL ====		====VirtualGL====
	The setup that was described above will render everything via software rendering (mesa), which you can confirm with <code>glxinfo</code>. For nodes that have dedicated GPUs, it might be desirable to have the desktop take advantage of hardware acceleration. This can be accomplished using VirtualGL which will draw the user's application on a local X11 server using hardware acceleration and then copies the rendered output to the user's virtual VNC display.		The setup that was described above will render everything via software rendering (mesa), which you can confirm with <code>glxinfo</code>. For nodes that have dedicated GPUs, it might be desirable to have the desktop take advantage of hardware acceleration. This can be accomplished using VirtualGL which will draw the user's application on a local X11 server using hardware acceleration and then copies the rendered output to the user's virtual VNC display.

Line 57:		Line 57:
	To accomplish this, I had to:		To accomplish this, I had to:

	# On Slurm, setup cgroups and a gres for GPUs		#On Slurm, setup cgroups and a gres for GPUs
	# Make all but the first GPU a gres. This will permit all users to see the first GPU and any other GPUs requested by the job		#Make all but the first GPU a gres. This will permit all users to see the first GPU and any other GPUs requested by the job
	# Start X11 on the compute nodes on each GPU. Each Xorg server should target a specific GPU (done so via specific Screen/Device sections in <code>xorg.conf</code>)		#Start X11 on the compute nodes on each GPU. Each Xorg server should target a specific GPU (done so via specific Screen/Device sections in <code>xorg.conf</code>)
	# Change the script that starts the display manager (like XFCE) so that it loads the xauth cookies for the X11 server corresponding to the assigned GPU and run the display manager with <code>vglrun</code> with the appropriate <code>VGL_DISPLAY</code> environment set.		#Change the script that starts the display manager (like XFCE) so that it loads the xauth cookies for the X11 server corresponding to the assigned GPU and run the display manager with <code>vglrun</code> with the appropriate <code>VGL_DISPLAY</code> environment set.

	<br />		<br />

	==== Troubleshooting ====		====Troubleshooting====
	Here are some issues I ran into. In hindsight, some of them are trivial but hopefully it helps someone.		Here are some issues I ran into. In hindsight, some of them are trivial but hopefully it helps someone.

	===== No cluster set =====		=====No cluster set=====
	Out of the box, trying to make a desktop results in:{{Quote\|quote=The cluster was never set. Either set it in form.yml.erb with `cluster` or `form.cluster` or set `cluster` in submit.yml.erb.		Out of the box, trying to make a desktop results in:{{Quote\|quote=The cluster was never set. Either set it in form.yml.erb with `cluster` or `form.cluster` or set `cluster` in submit.yml.erb.

Line 154:		Line 154:

	==Authentication==		==Authentication==
			Open OnDemand supports a number of authentication methods including plain old system authentication (based on PAM), LDAP/AD (with Dex), and federated authentication services (such as CAS, Shibboleth, OpenID).

			For more information, review the docs at https://osc.github.io/ood-documentation-test/init-2.1/authentication.html

			=== Plain old PAM ===
			To have Apache handle authentication using PAM, define the following lines in <code>ood_porta.yml</code>:
			{{Highlight
			\| code = auth:
			- 'AuthType Basic'
			- 'AuthName "Open OnDemand"'
			- 'AuthBasicProvider PAM'
			- 'AuthPAMService ood'
			- 'Require valid-user'
			\| lang = terminal
			}}
			When users attempt to access the OnDemand instance, they will authenticate using their web browser using basic authentication. When using this method, the user cannot log out unless they clear their browser of the basic authentication credentials.

	===OpenID===		===OpenID===

Leo: VirtualGL on remote desktops

2021-02-11T19:21:42Z

VirtualGL on remote desktops

← Older revision		Revision as of 19:21, 11 February 2021
Line 8:		Line 8:
	It may help to look at other apps that are available at:		It may help to look at other apps that are available at:

	* https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html		*https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html
	* https://code.osu.edu/ondemand		*https://code.osu.edu/ondemand

	====Deploying an app====		====Deploying an app====
Line 26:		Line 26:

	===Remote Desktop===		===Remote Desktop===
	Out of the box, trying to make a desktop results in:
	{{Quote\|quote=The cluster was never set. Either set it in form.yml.erb with `cluster` or `form.cluster` or set `cluster` in submit.yml.erb.		==== Setup ====
			Clone one of the existing <code>bc_desktop</code> apps and edit <code>form.yml</code>. Set the cluster value to a cluster you have configured in OnDemand (hint: what you have in the <code>clusters.d</code> directory). You may want to add/remove the CPU, memory, partition fields as desired. The default bc_desktop app includes startup scripts for various desktop managers but I hardcoded this to always use XFCE for my use-case.
			On the compute nodes that are intended to run the VNC sessions, you install the desktop environment (XFCE in my case), TurboVNC, and websockify. On a CentOS 8 machine, this was accomplished manually with:

			{{Highlight
			\| code = yum install -y nc wget python3-devel python3-pip;

			# Websockify
			pip3 install websockify;

			# TurboVNC
			wget https://turbovnc.org/pmwiki/uploads/Downloads/TurboVNC.repo;
			mv TurboVNC.repo /etc/yum.repos.d/;
			yum install -y turbovnc ;

			# Desktop DM
			yum -y install epel-release;
			yum -y group install "Xfce" "base-x"
			\| lang = terminal
			}}

			If everything works correctly, requesting for a desktop will submit a job via the scheduler which launches Xvnc and proxies it via websockify to OnDemand which then renders it via noVNC.

			==== VirtualGL ====
			The setup that was described above will render everything via software rendering (mesa), which you can confirm with <code>glxinfo</code>. For nodes that have dedicated GPUs, it might be desirable to have the desktop take advantage of hardware acceleration. This can be accomplished using VirtualGL which will draw the user's application on a local X11 server using hardware acceleration and then copies the rendered output to the user's virtual VNC display.

			I currently have nodes set up so that users may request for 0 or 1 GPUs. If no GPUs are requested, the desktop is rendered using a GPU that is shared with other users on the node. If 1 GPU is requested, the desktop is rendered on a dedicated GPU for that user.

			To accomplish this, I had to:

			# On Slurm, setup cgroups and a gres for GPUs
			# Make all but the first GPU a gres. This will permit all users to see the first GPU and any other GPUs requested by the job
			# Start X11 on the compute nodes on each GPU. Each Xorg server should target a specific GPU (done so via specific Screen/Device sections in <code>xorg.conf</code>)
			# Change the script that starts the display manager (like XFCE) so that it loads the xauth cookies for the X11 server corresponding to the assigned GPU and run the display manager with <code>vglrun</code> with the appropriate <code>VGL_DISPLAY</code> environment set.

			<br />

			==== Troubleshooting ====
			Here are some issues I ran into. In hindsight, some of them are trivial but hopefully it helps someone.

			===== No cluster set =====
			Out of the box, trying to make a desktop results in:{{Quote\|quote=The cluster was never set. Either set it in form.yml.erb with `cluster` or `form.cluster` or set `cluster` in submit.yml.erb.

	* The Desktop session data for this session can be accessed under the staged root directory.\|sign=\|source=}}		* The Desktop session data for this session can be accessed under the staged root directory.\|sign=\|source=}}
	You need to tweak the form to work in your environment. You need to edit either define the cluster value in <code>/var/www/ood/apps/sys/bc_desktop/form.yml</code> or launch your own interactive app. Like the Jupyter notebooks, you may want to offer the user a list of partitions that they can run their VNC session on. Do so by editing <code>form.yml</code> or converting it to a <code>.erb</code> so that you can run system commands to obtain all the slurm partitions to list.		You need to tweak the form to work in your environment. You need to edit either define the cluster value in <code>/var/www/ood/apps/sys/bc_desktop/form.yml</code> or launch your own interactive app.

	====bc_vnc_resolution is not showing====		=====bc_vnc_resolution is not showing=====
	If your <code>bc_vnc_resolution</code> form field isn't showing, it's most likely that you have ENABLE_NATIVE_VNC set which lets the VNC client to resize the desktop.		If your <code>bc_vnc_resolution</code> form field isn't showing, it's most likely that you have <code>ENABLE_NATIVE_VNC</code> set which lets the VNC client to resize the desktop.

	====VNC: Failed to connect to server====		=====VNC: Failed to connect to server=====
	Launching the VNC session opens noVNC. However, I got "Failed to connect to server". Error from apache logs show:		Launching the VNC session opens noVNC. However, I got "Failed to connect to server". Error from apache logs show:
	{{Highlight		{{Highlight
Line 45:		Line 86:
	which is weird because c2 is actually listening on port 5901 not 32316. The job output however clearly shows the websockify binary not being found. Fixing this fixed the VNC connection issue.		which is weird because c2 is actually listening on port 5901 not 32316. The job output however clearly shows the websockify binary not being found. Fixing this fixed the VNC connection issue.

	'''Solution''': Ensure websockify is found. If it's installed in a nonstandard location, tweak your clusters.d yaml file so that the batch_connect/vnc has a script_wrapper that exports the websockify path as WEBSOCKIFY_CMD.		'''Solution''': Ensure websockify is found. If it's installed in a nonstandard location, tweak your <code>clusters.d</code> yaml file so that the batch_connect/vnc has a script_wrapper that exports the websockify path as <code>WEBSOCKIFY_CMD</code>.
	{{Highlight		{{Highlight
	\| code = batch_connect:		\| code = batch_connect:
Line 63:		Line 104:
	}}		}}

	====No Home Environment Error====		=====No Home Environment Error=====
	{{Highlight		{{Highlight
	\| code = Setting VNC password...		\| code = Setting VNC password...

Leo at 00:40, 11 February 2021

2021-02-11T00:40:52Z

← Older revision		Revision as of 00:40, 11 February 2021
Line 6:		Line 6:
	See: https://osc.github.io/ood-documentation/latest/app-development/enabling-development-mode.html#enable-in-ondemand-v1-6		See: https://osc.github.io/ood-documentation/latest/app-development/enabling-development-mode.html#enable-in-ondemand-v1-6

	It may help to look at other apps that are available at: https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html		It may help to look at other apps that are available at:

			* https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html
			* https://code.osu.edu/ondemand

	====Deploying an app====		====Deploying an app====
Line 114:		Line 117:
	To get Open OnDemand to authenticate against the Azure AD service that the University of Calgary is using:		To get Open OnDemand to authenticate against the Azure AD service that the University of Calgary is using:

	# Obtain the OIDC Identity Provider URL, client ID, and client secret.		#Obtain the OIDC Identity Provider URL, client ID, and client secret.
	# Install the <code>mod_auth_openidc</code> package.		#Install the <code>mod_auth_openidc</code> package.
	# Setup <code>ood_portal.yml</code> for oidc authentication:		#Setup <code>ood_portal.yml</code> for oidc authentication:
	{{Highlight		{{Highlight
	\| code = servername: "ood.example.com"		\| code = servername: "ood.example.com"

Leo: Authentication with OpenID

2021-02-01T22:03:38Z

Authentication with OpenID

@@ Line 109: / Line 109: @@
 '''Solution:''' This only worked after setting <code>OOD_SHELL_ORIGIN_CHECK='off'</code>
-== Authentication ==
+==Authentication==
-=== OpenID ===
+===OpenID===
-See:
+To get Open OnDemand to authenticate against the Azure AD service that the University of Calgary is using:
-* https://osc.github.io/ood-documentation/latest/authentication/oidc.html
+# Obtain the OIDC Identity Provider URL, client ID, and client secret.
-* https://github.com/zmartzone/mod_auth_openidc
+# Install the <code>mod_auth_openidc</code> package.
 ==Troubleshooting==

Leo at 00:27, 30 January 2021

2021-01-30T00:27:26Z

← Older revision		Revision as of 00:27, 30 January 2021
Line 108:		Line 108:

	'''Solution:''' This only worked after setting <code>OOD_SHELL_ORIGIN_CHECK='off'</code>		'''Solution:''' This only worked after setting <code>OOD_SHELL_ORIGIN_CHECK='off'</code>

			== Authentication ==

			=== OpenID ===
			See:

			* https://osc.github.io/ood-documentation/latest/authentication/oidc.html
			* https://github.com/zmartzone/mod_auth_openidc

	==Troubleshooting==		==Troubleshooting==
	Logs are stored at:		Logs are stored at:

Leo at 00:25, 30 January 2021

2021-01-30T00:25:21Z

← Older revision		Revision as of 00:25, 30 January 2021
Line 1:		Line 1:
	Open OnDemand is a open source project by the Ohio Supercomputer Center that provides a web portal for HPC users. It is designed as a platform allowing system administrators to add additional modules or 'apps'. Users can use this platform to launch interactive jobs or VNC/SSH sessions, view their job statuses, interact with their files. It supports a variety of authentication mechanisms including federated authentication (OpenID, CAS, Shiboleth) or with an the system's underlying PAM (ldap, password file, etc.).		Open OnDemand is a open source project by the Ohio Supercomputer Center that provides a web portal for HPC users. It is designed as a platform allowing system administrators to add additional modules or 'apps'. Users can use this platform to launch interactive jobs or VNC/SSH sessions, view their job statuses, interact with their files. It supports a variety of authentication mechanisms including federated authentication (OpenID, CAS, Shiboleth) or with an the system's underlying PAM (ldap, password file, etc.).<br />

	<br />
	==Interactive Apps==		==Interactive Apps==
	===App Development===		===App Development===
Line 140:		Line 138:
	<br />		<br />

	== See Also ==		==See Also==

	* Documentation: https://osc.github.io/ood-documentation/latest/index.html		*Documentation: https://osc.github.io/ood-documentation/latest/index.html