Leo's Notes - User contributions [en]

Self Signed SSL Certificates

2022-02-06T21:51:59Z

174.241.28.194: /* See Also */

Self signed SSL certificates can be generated using [[OpenSSL]].

== Steps ==
=== Private Key ===
Generate a new private key. The output will be a Privacy-Enhanced Mail (PEM) format.
{{highlight|lang=terminal|code=
## Generate a new private key
# openssl genrsa -out server.key 4096
}}

=== Certificate Signing Request ===
Generate the certificate signing request.
{{highlight|lang=terminal|code=
# openssl req -new -sha256 \
-key server.key \
-subj "/C=CA/ST=Alberta/O=Steamr/CN=steamr.com" \
-reqexts SAN \
-extensions SAN \
-config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:x.steamr.com,DNS:y.steamr.com")) \
-out server.csr
}}

The {{code|-subj}} value contains:
{| class="wikitable"
! Field !! Description !! Example
|-
| C || Country - The 2 letter International Standards Organization ISO abbreviation for your country || CA
|-
| ST || State - The state or province of your organization || Alberta
|-
| O || Organization - The legal name of the organization || Steamr Corp.
|-
| CN || Common Name - The fully qualified domain name for the certificate || steamr.com
|}

Additionally, the {{code|[SAN]}} section passed in through the {{code|-config}} flag is used to define any SANs the certificate should have and can be removed if unneeded.

You can verify your CSR using:
{{highlight|lang=terminal|code=
# openssl req -noout -text -in server.csr
}}

After you have verified your CSR, you can either continue to self sign with the next step, or submit it to a CA to purchase a SSL certificate.

=== Signing ===
Sign your certificate signing request.
{{highlight|lang=terminal|code=
# openssl x509 \
-req \
-days 730 \
-extfile <(printf "subjectAltName=DNS:x.steamr.com,DNS:y.steamr.com") \
-signkey server.key \
-in server.csr \
-out server.crt
}}
The {{code|-days}} value defines the number of days the certificate will be valid for from the time of signing. If not given, a default of 30 days will be used.

Any SANs will also need to be passed through using the {{code|-extfile}} flag.

You can verify the signed certificate by running:
{{highlight|lang=terminal|code=
# openssl x509 -noout -text -in server.crt
}}

=== Renewing ===
If the certificate has expired, you can 'renew' it by first regenerating a certificate signing request (CSR):

{{highlight|lang=terminal|code=
# openssl x509 -x509toreq -in expired_certificate.crt -out new_csr.csr -signkey domain_private.key
}}

Then signing the CSR with the private key to produce a new certificate:
{{highlight|lang=terminal|code=
# openssl x509 -req -days 3650 -in new_csr.csr -signkey domain_private.key -out new_certificate.crt
}}

Replace the expired certificate with the new certificate and reload any services that are using the certificate for it to be applied.

== Trusting Your Self Signed SSL Certificates ==
The key and certificate generated with the steps above will not be signed by a trusted certificate authority and therefore cannot be verified by your browser, resulting in a security warning. To get around this, you can add your certificate directly into your browser or operating system's list of trusted certificate authorities.

== See Also ==
* [[OpenSSL]-most-common-openssl-commands.html

}}

OpenSSL

2022-02-06T21:47:55Z

174.241.28.194: /* Verification */

OpenSSL allows you to create, modify, and view certificates and private keys. You can use it to generate [[Self Signed SSL Certificates]].

==General OpenSSL Commands==

=== Generate Keys and Certificate Signing Requests ===
These commands allow you to generate CSRs, Certificates, Private Keys and do other miscellaneous tasks.
{| class="wikitable"
!Task
!Command
|-
|Generate a new private key
|{{highlight|lang=terminal|code=
# openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key
}}Or just a 4096 bit RSA:{{highlight|lang=terminal|code=
# openssl genrsa -out server.key 4096
}}
|-
|Generate a new private key and Certificate Signing Request
|{{highlight|lang=terminal|code=
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt
}}
|-
|Generate a certificate signing request with an existing private key
|{{highlight|lang=terminal|code=
# openssl req -out CSR.csr -key privateKey.key -new
}}
|-
|Generate a certificate signing request based on an existing certificate
|{{highlight|lang=terminal|code=
# openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey privateKey.key
}}
|-
|Generate a certificate signing request for various SANs.
|{{highlight|lang=terminal|code=
# openssl req -new -sha256 \
-key server.key \
-subj "/C=CA/ST=Alberta/O=Steamr/CN=steamr.com" \
-reqexts SAN \
-extensions SAN \
-config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:x.steamr.com,DNS:y.steamr.com")) \
-out server.csr
}}
|-
|Remove a passphrase from a private key
|{{highlight|lang=terminal|code=
# openssl rsa -in privateKey.pem -out newPrivateKey.pem
}}
|}

=== Verification ===
{| class
!Task
!Command
|-
|View a PEM certificate
|{{highlight|lang=terminal|code=
# openssl x509 -in certificate.crt -text -noout
}}
|-
|Check a PKCS#12 file (.pfx or .p12)
|{{highlight|lang=terminal|code=
# openssl pkcs12 -info -in keyStore.p12
}}
|-
|View a certificate signing request
|{{highlight|lang=terminal|code=
# openssl req -text -noout -verify -in CSR.csr
}}
|-
|Check a private key
|{{highlight|lang=terminal|code=
# openssl rsa -in privateKey.key -check
}}
|-
|Check if your certificate matches a key
|{{highlight|lang=terminal|code=
# openssl x509 -noout -modulus -in certificate.crt | openssl md5
# openssl rsa -noout -modulus -iopenssl md5
}}
|}

==Checking & Verifying==
If you are receiving an error that the private doesn't match the certificate or that a certificate that you installed to a site is not trusted, try one of these commands. It might be a good idea to use an online SSL checker service as well to determine what issues that are with the installed certificate.

The public key should match your private key and must be used in your certificate and certificate signing request. You can verify this by running:
{{highlight|lang=terminal|code=
# openssl x509 -noout -modulus -in certificate.crt | openssl md5
# openssl rsa -noout -modulus -in privateKey.key | openssl md5
# openssl req -noout -modulus -in CSR.csr | openssl md5
}}

===Remote Server Certificate===
All the certificates (including Intermediates) should be displayed when using {{code|s_client}}.

SNI allows virtual hosting of multiple domains on the same IP address. It is a mechanism that allows the web server to know which domain is being accessed in order to use the proper certificate when establishing a connection to the client. For websites that have been assigned a dedicated IP address, using SNI is not required.

If the server is not using SNI:
{{highlight|lang=terminal|code=
# openssl s_client -showcerts -connect www.paypal.com:443
}}

If the server is using SNI, you will need to provide the hostname:
{{highlight|lang=terminal|code=
# openssl s_client -showcerts -servername www.paypal.com -connect www.paypal.com:443
}}

The x509 certificate can be retrieved in either case by piping the output through {{code| openssl x509 -text}}.
{{highlight|lang=terminal|code=
# openssl s_client -showcerts -connect www.paypal.com:443 | openssl x509 -text
# openssl s_client -showcerts -servername www.paypal.com -connect www.paypal.com:443 | openssl x509 -text
}}

==Certificate and Key Conversion==
These commands allow you to convert certificates and keys to different formats to make them compatible with specific types of servers or software. For example, you can convert a normal PEM file that would work with Apache to a PFX (PKCS#12) file and use it with Tomcat or IIS. Use our SSL Converter to convert certificates without messing with OpenSSL.

Convert a DER file (.crt .cer .der) to PEM
{{highlight|lang=terminal|code=
# openssl x509 -inform der -in certificate.cer -out certificate.pem
}}

Convert a PEM file to DER
{{highlight|lang=terminal|code=
# openssl x509 -outform der -in certificate.pem -out certificate.der
}}

Convert a PKCS#12 file (.pfx .p12) containing a private key and certificates to PEM
{{highlight|lang=terminal|code=
# openssl pkcs12 -in keyStore.pfx -out keyStore.pem -nodes
}}

You can add {{code|-nocerts}} to only output the private key or add {{code|-nokeys}} to only output the certificates.

Convert a PEM certificate file and a private key to PKCS#12 (.pfx .p12)
{{highlight|lang=terminal|code=
# openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt
}}

==See Also==

*https://www.sslshopper.com/article-most-common-openssl-commands.html
*[[Self Signed SSL Certificates]]

{{Navbox Linux}}
[[Category:Linux]]
[[Category:LinuxUtilities]]
[[Category:Certificates]]
{{Navbox Certificates}}

Open OnDemand

2022-02-06T21:46:28Z

174.241.28.194: /* 400 - Bad Request */

Open OnDemand is a open source project by the Ohio Supercomputer Center that provides a web portal for HPC users. It is designed as a platform allowing system administrators to add additional modules or 'apps'. Users can use this platform to launch interactive jobs or VNC/SSH sessions, view their job statuses, interact with their files. It supports a variety of authentication mechanisms including federated authentication (OpenID, CAS, Shiboleth) or with an the system's underlying PAM (ldap, password file, etc.).<br />
==Interactive Apps==
===App Development===
On Open OnDemand 1.8, you can only enable a specific user access to development mode by creating a directory <code>/var/www/ood/apps/dev/$username</code>, then symlinking <code>/var/www/ood/apps/dev/$username/gateway</code> to <code>/home/$username/openondemand/dev</code>. Restart the PUN web server. A 'Develop' section should appear in the navbar. This will only work for <code>$username</code>.

See: https://osc.github.io/ood-documentation/latest/app-development/enabling-development-mode.html#enable-in-ondemand-v1-6

It may help to look at other apps that are available at:

*https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html
*https://code.osu.edu/ondemand

====Deploying an app====
Once you have done developing an app, deploy it by moving it to <code>/var/www/ood/apps/sys</code>.

===Jupyter Notebooks===
Running a Jupyter Notebook interactive app requires tweaking the example OSC Jupyter Interactive App (https://github.com/OSC/bc_example_jupyter) to work in your environment. Before you get started, you should be able to submit jobs via Open OnDemand. If this isn't working, fix your configuration before proceeding with interactive apps.

The easiest way to get this working is to enable App Development on an account and then testing from this account.

#Enable App Development on your account
#Clone the OSC Jupyter app to <code>~/ondemand/dev/bc_example_jupyter</code>
#Tweak the <code>form.yml</code> (or <code>form.yml.erb</code>). Add or remove fields as required by you.
#Tweak <code>submit.yml.erb</code> to work in your environment.
#Try launching a Jupyter notebook and fix any issues as you go.

===Remote Desktop===

====Setup====
Clone one of the existing <code>bc_desktop</code> apps and edit <code>form.yml</code>. Set the cluster value to a cluster you have configured in OnDemand (hint: what you have in the <code>clusters.d</code> directory). You may want to add/remove the CPU, memory, partition fields as desired. The default bc_desktop app includes startup scripts for various desktop managers but I hardcoded this to always use XFCE for my use-case.
On the compute nodes that are intended to run the VNC sessions, you install the desktop environment (XFCE in my case), TurboVNC, and websockify. On a CentOS 8 machine, this was accomplished manually with:

{{Highlight
| code = yum install -y nc wget python3-devel python3-pip;

# Websockify
pip3 install websockify;

# TurboVNC
wget https://turbovnc.org/pmwiki/uploads/Downloads/TurboVNC.repo;
mv TurboVNC.repo /etc/yum.repos.d/;
yum install -y turbovnc ;

# Desktop DM
yum -y install epel-release;
yum -y group install "Xfce" "base-x"
| lang = terminal
}}

If everything works correctly, requesting for a desktop will submit a job via the scheduler which launches Xvnc and proxies it via websockify to OnDemand which then renders it via noVNC.

====VirtualGL====
The setup that was described above will render everything via software rendering (mesa), which you can confirm with <code>glxinfo</code>. For nodes that have dedicated GPUs, it might be desirable to have the desktop take advantage of hardware acceleration. This can be accomplished using VirtualGL which will draw the user's application on a local X11 server using hardware acceleration and then copies the rendered output to the user's virtual VNC display.

I currently have nodes set up so that users may request for 0 or 1 GPUs. If no GPUs are requested, the desktop is rendered using a GPU that is shared with other users on the node. If 1 GPU is requested, the desktop is rendered on a dedicated GPU for that user.

To accomplish this, I had to:

#On Slurm, setup cgroups and a gres for GPUs
#Make all but the first GPU a gres. This will permit all users to see the first GPU and any other GPUs requested by the job
#Start X11 on the compute nodes on each GPU. Each Xorg server should target a specific GPU (done so via specific Screen/Device sections in <code>xorg.conf</code>)
#Change the script that starts the display manager (like XFCE) so that it loads the xauth cookies for the X11 server corresponding to the assigned GPU and run the display manager with <code>vglrun</code> with the appropriate <code>VGL_DISPLAY</code> environment set.

<br />

====Troubleshooting====
Here are some issues I ran into. In hindsight, some of them are trivial but hopefully it helps someone.

=====No cluster set=====
Out of the box, trying to make a desktop results in:{{Quote|quote=The cluster was never set. Either set it in form.yml.erb with `cluster` or `form.cluster` or set `cluster` in submit.yml.erb.

* The Desktop session data for this session can be accessed under the staged root directory.|sign=|source=}}
You need to tweak the form to work in your environment. You need to edit either define the cluster value in <code>/var/www/ood/apps/sys/bc_desktop/form.yml</code> or launch your own interactive app.

=====bc_vnc_resolution is not showing=====
If your <code>bc_vnc_resolution</code> form field isn't showing, it's most likely that you have <code>ENABLE_NATIVE_VNC</code> set which lets the VNC client to resize the desktop.

=====VNC: Failed to connect to server=====
Launching the VNC session opens noVNC. However, I got "Failed to connect to server". Error from apache logs show:
{{Highlight
| code = ==> httpd/error.log <==
[Fri Jan 22 03:12:49.324498 2021] [proxy:error] [pid 5386:tid 140231151974144] (111)Connection refused: AH00957: WS: attempt to connect to 172.28.0.5:32316 (*) failed
[Fri Jan 22 03:12:49.324540 2021] [proxy_wstunnel:error] [pid 5386:tid 140231151974144] [client 136.159.79.128:54416] AH02452: failed to make connection to backend: c2
| lang = text
}}
which is weird because c2 is actually listening on port 5901 not 32316. The job output however clearly shows the websockify binary not being found. Fixing this fixed the VNC connection issue.

'''Solution''': Ensure websockify is found. If it's installed in a nonstandard location, tweak your <code>clusters.d</code> yaml file so that the batch_connect/vnc has a script_wrapper that exports the websockify path as <code>WEBSOCKIFY_CMD</code>.
{{Highlight
| code = batch_connect:
basic:
script_wrapper: {{!}}
%s
set_host: "host=$(hostname -A {{!}} awk '{print $1}')"
vnc:
# websockify by pip3 is in /usr/local/bin
script_wrapper: {{!}}
module purge
export PATH="/opt/TurboVNC/bin:$PATH"
export WEBSOCKIFY_CMD="/usr/local/bin/websockify"
%s
set_host: "host=$(hostname -A {{!}} awk '{print $1}')"
| lang = yaml
}}

=====No Home Environment Error=====
{{Highlight
| code = Setting VNC password...
Error: no HOME environment variable
Starting VNC server...
vncserver: The HOME environment variable is not set.
vncserver: The HOME environment variable is not set.
vncserver: The HOME environment variable is not set.
vncserver: The HOME environment variable is not set.
vncserver: The HOME environment variable is not set.
| lang = text
}}
Slurm job output showed this. It turns out, the nodes I've set up didn't have the user account so it didn't know what the user's home directory is.

'''Solution''': Ensure your system has the user account.

===Job Composer===
To install it, see:

*https://osc.github.io/ood-documentation/latest/applications/job-composer.html
*https://osc.github.io/ood-documentation/latest/customization.html#custom-job-composer-templates

The actual Job Composer app is at https://github.com/OSC/ondemand/tree/master/apps/myjobs

====CSRF Fail, resulting in 'The change you wanted was rejected'====
For some reason, when trying to create a new job or new template, I get "The change you wanted was rejected.". PUN logs show <code>FATAL "ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):</code>".

This was only 'fixed' by disabling CSRF by injecting <code>Rails.application.config.action_controller.forgery_protection_origin_check = false</code> into <code>config/initializers/new_framework_defaults_5_2.rb</code>. This is most likely caused by either Traefik (likely) or nginx doing the reverse proxy somehow causing the CSRF check to fail due to a difference in base_url. In fact, this issue prevails across all the other interactive apps and may require the same fix as well.

See also, this issue: https://github.com/rails/rails/issues/22965.

===Shell / Terminal Access===
The Shell App basically just runs a web SSH client from the Open OnDemand server. You limit which hosts it can connectt to by tweaking the env file:
{{Highlight
| code = DEFAULT_SSHHOST="arc.ucalgary.ca"
OOD_SSHHOST_ALLOWLIST="arc.ucalgary.ca"
| lang = text
}}

====Failed to establish a websocket connection.====
After setting up OnDemand, I had a hard time getting the SSH app to connect. I kept on getting:
{{Highlight
| code = Failed to establish a websocket connection. Be sure you are using a browser that supports websocket connections.
| lang = text
}}
Debugging the socket revealed that it was getting a 401 error from PUN. At first, I thought the reverse proxy (traefik 1.7) wasn't forwarding the authentication headers, but this was a red herring. Something with the PUN application was throwing this 401 error. I verified that the node is able to SSH and that the default hostname is correct in <code>/etc/ood/config/apps/shell/env</code>. I even tried connecting to a specific host to no avail. I then set <code>OOD_SSH_WRAPPER=/test.sh</code> with test.sh just dumping the environment to a tmp file which showed me that it wasn't even reaching the point of calling the SSH wrapper. OOD 1.8 also requires setting the <code>OOD_SSHHOST_ALLOWLIST</code>, but that didn't help.

'''Solution:''' This only worked after setting <code>OOD_SHELL_ORIGIN_CHECK='off'</code>

==Authentication==
Open OnDemand supports a number of authentication methods including plain old system authentication (based on PAM), LDAP/AD (with Dex), and federated authentication services (such as CAS, Shibboleth, OpenID).

For more information, review the docs at https://osc.github.io/ood-documentation-test/init-2.1/authentication.html

===Plain old PAM===
To have Apache handle authentication using PAM, define the following lines in <code>ood_porta.yml</code>:
{{Highlight
| code = auth:
- 'AuthType Basic'
- 'AuthName "Open OnDemand"'
- 'AuthBasicProvider PAM'
- 'AuthPAMService ood'
- 'Require valid-user'
| lang = terminal
}}
When users attempt to access the OnDemand instance, they will authenticate using their web browser using basic authentication. When using this method, the user cannot log out unless they clear their browser of the basic authentication credentials.

===OpenID===
To get Open OnDemand to authenticate against the Azure AD service that the University of Calgary is using:

#Obtain the OIDC Identity Provider URL, client ID, and client secret.
#Install the <code>mod_auth_openidc</code> package.
#Setup <code>ood_portal.yml</code> for oidc authentication:
{{Highlight
| code = servername: "ood.example.com"

# Use OIDC logout
logout_redirect: "/oidc/?logout=https%3A%2F%2Food.example.com"

# Authentication via MS results in 'user@ucalgary.ca' as the username
user_map_cmd: "/opt/ood/ood_auth_map/bin/ood_auth_map.regex --regex='^(.+)@ucalgary.ca' "

oidc_uri: "/oidc/"
oidc_provider_metadata_url: "https://login.microsoftonline.com/** Provided by MS **/v2.0/.well-known/openid-configuration"
oidc_client_id: "** Provided by MS **"
oidc_client_secret: "** Provided by MS **"
oidc_remote_user_claim: "preferred_username"
oidc_scope: "openid profile email"
oidc_session_inactivity_timeout: 28800
oidc_session_max_duration: 28800
oidc_state_max_number_of_cookies: "10 true"
oidc_settings:
OIDCPassIDTokenAs: "serialized"
OIDCPassRefreshToken: "On"
OIDCPassClaimsAs: "environment"
OIDCStripCookies: "mod_auth_openidc_session mod_auth_openidc_session_chunks mod_auth_openidc_session_0 mod_auth_openidc_session_1"
| lang = text
}}

If your Open OnDemand system is behind a reverse proxy doing SSL termination, you will still want the <code>OIDCRedirectURI</code> to be using the <code>https://</code> URL. However, the ood-portal-generator only uses the https protocol if the SSL certificates are defined in <code>ood_portal.yml</code> (which we don't want to do with an external SSL terminator). As a work around, my docker <code>entrypoint.sh</code> script will replace http with https before starting apache.

See Also:

*https://osc.github.io/ood-documentation/latest/authentication/oidc.html
*https://github.com/zmartzone/mod_auth_openidc

==Troubleshooting==
Logs are stored at:

*<code>/var/log/ondemand-nginx/</code>
*<code>/var/log/httpd/</code>

Apps are stored at:

*<code>/var/www/ood/apps/sys</code>
*<code>/var/www/ood/apps/dev/$username/gateway</code> (symlinked to user's home directory <code>~/ondemand/dev</code>)

=== 400 - Bad Request ===
Apache and the nginx instance running the PUN supports a 8k maximum client header request. Exceeding this limit will result in a <code>400 - Bad Request</code> error being returned by Apache to the client. A likely cause for exceeding this limit is with excessive amounts of cookies.

If there are a large number of OpenID state cookies, then this error is likely a symptom of another issue. For more information on this issue, see: https://github.com/zmartzone/mod_auth_openidc/wiki/Cookies

A workaround would be to increase the client header request size from the default 8k to something higher. To do this:

#
# Create an additional config file at <code>/var/lib/ondemand-nginx/config/apps/sys/large-header.conf</code> with <code>large_client_header_buffers 8 64k;</code>.

==Implementations==

*https://arc-ts.umich.edu/open-ondemand/ University of Michigan
*https://www.hpc.caltech.edu/documentation/open-ondemand Caltech
*https://www.ou.edu/oscer/support/openondemand#start University of Oklahoma
*https://its.unc.edu/research-computing/ondemand/ North Carolina
*https://chpc.utah.edu/presentations/images-and-pdfs/IntroOOD20s.pdf
*https://rc-docs.northeastern.edu/en/latest/using-ood/introduction.html North Eastern (pam auth)
*https://wiki.ncsa.illinois.edu/display/ISL20/Getting+started+with+HAL+OnDemand (pam)
*https://www.sherlock.stanford.edu/docs/user-guide/ondemand/ Stanford
*https://hcc.unl.edu/documents/kickstart_2020/HCC_OOD.pdf Nebraska
*https://dashboard.hpc.unimelb.edu.au/web_environments/ (pam, https://spartan-ood.hpc.unimelb.edu.au/pun/sys/dashboard/batch_connect/sessions)
*https://wiki.hpc.odu.edu/en/open-ondemand Old Dominion (sso, https://ondemand.wahab.hpc.odu.edu/)
*https://docs.ycrc.yale.edu/clusters-at-yale/access/ood/ Yale

<br />

==See Also==

*Documentation: https://osc.github.io/ood-documentation/latest/index.html

{{Navbox Linux}}
[[Category:Linux]]
[[Category:HPC]]