Open OnDemand

From Leo's Notes
Revision as of 21:31, 21 January 2021 by Leo (talk | contribs)
This page was last edited on 21 January 2021, at 21:31.

Open OnDemand is a open source project by the Ohio Supercomputer Center that provides a web portal for HPC users. It is designed as a platform allowing system administrators to add additional modules or 'apps'. Users can use this platform to launch interactive jobs or VNC/SSH sessions, view their job statuses, interact with their files. It supports a variety of authentication mechanisms including federated authentication (OpenID, CAS, Shiboleth) or with an the system's underlying PAM (ldap, password file, etc.).

Open OnDemand is in use by:

  • https://arc-ts.umich.edu/greatlakes/user-guide/#document-3
  • Yale: https://docs.ycrc.yale.edu/clusters-at-yale/access/ood/

Tasks

Jupyter Notebooks

Running a Jupyter Notebook interactive app is somewhat involved.

Enable App Development on your account (see #App Development).

Copy an existing app and modify it to your own needs. Some examples can be found at https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html

Job Composer App

To install it, see:

The actual Job Composer app is at https://github.com/OSC/ondemand/tree/master/apps/myjobs

Issues

For some reason, when trying to create a new job or new template, I get "The change you wanted was rejected.". PUN logs show FATAL "ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):".

This was only 'fixed' by disabling CSRF by injecting Rails.application.config.action_controller.forgery_protection_origin_check = false into config/initializers/new_framework_defaults_5_2.rb. This is most likely caused by either Traefik (likely) or nginx doing the reverse proxy somehow causing the CSRF check to fail due to a difference in base_url. See this issue: https://github.com/rails/rails/issues/22965.

App Development

On Open OnDemand 1.8, you need to create /var/www/ood/apps/dev/lleung, then symlink /var/www/ood/apps/dev/lleung/gateway to /home/lleung/openondemand/dev. Restart the PUN web server. A 'Develop' section should appear in the navbar.

See: https://osc.github.io/ood-documentation/latest/app-development/enabling-development-mode.html#enable-in-ondemand-v1-6

Troubleshooting

Logs are stored at:

  • /var/log/ondemand-nginx/
  • /var/log/httpd/

Apps are stored at:

  • /var/www/ood/apps/sys
  • /var/www/ood/apps/dev/$username/gateway (symlinked to user's home directory /ondemand/dev)

SSH App: Failed to establish a websocket connection.

After setting up OnDemand, I had a hard time getting the SSH app to connect. I kept on getting: 

Failed to establish a websocket connection. Be sure you are using a browser that supports websocket connections.

Debugging the socket revealed that it was getting a 401 error from PUN. At first, I thought the reverse proxy (traefik 1.7) wasn't forwarding the authentication headers, but this was a red herring. Something with the PUN application was throwing this 401 error. I verified that the node is able to SSH and that the default hostname is correct in /etc/ood/config/apps/shell/env. I even tried connecting to a specific host to no avail. I then set OOD_SSH_WRAPPER=/test.sh with test.sh just dumping the environment to a tmp file which showed me that it wasn't even reaching the point of calling the SSH wrapper. OOD 1.8 also requires setting the OOD_SSHHOST_ALLOWLIST, but that didn't help.

Solution: This only worked after setting OOD_SHELL_ORIGIN_CHECK='off'.

Other notes