SSH

Secure Shell (SSH) is a method to securely communicate between computers. This page will discuss SSH under the context of the OpenSSH server.

Theory & Cryptography

SSH supports different key exchange algorithms, ciphers, and message authentication codes (MACs). The server and client negotiates and chooses a set of these algorithms that are supported by both and then proceeds with the key exchange.

Key Exchange

SSH handles key exchange using one of two ways: Diffie-Hellman and Elliptic Curve Diffie-Hellman. A shared secret is generated between the client and server for every session which will be used as the key, thereby providing forward secrecy.

Diffie-Hellman works with a multiplicative group of integers modulo a prime. Its security is based on the hardness of the discrete logarithm problem.

Elliptic Curve Diffie-Hellman works with elliptic curves over finite fields. Its security is based on the hardness of the elliptic curve discrete logarithm problem.

Authentication

With a shared secret derived from Diffie-Hellman, the client and server need to prove to each other they are who they are.

The server proves its identity to the client by signing the shared secret (the key) using a public key algorithm.

The client proves its identity to the server by using:

1. Password authentication
2. Public Key Authentication (same as the server's)

Symmetric Ciphers

With a shared key in place, data can be encrypted and decrypted between the client and server using symmetric ciphers.

Message Authentication Codes

Message authentication codes provide integrity. An authenticated encryption cipher mode already provides integrity, so extra MACs are not used. MACs are calculated and attached to every message if CTR is used.

There are multiple ways to combine ciphers and MACs:

• Encrypt-then-MAC: encrypt the message, then attach the MAC of the ciphertext.
• MAC-then-encrypt: attach the MAC of the plaintext, then encrypt everything.
• Encrypt-and-MAC: encrypt the message, then attach the MAC of the plaintext.

MAC-then-encrypt and Encrypt-and-MAC are prone to leak information because attackers can control an aspect of the system and perform timing attacks (eg: verifying a message vs. decrypting a message for Encrypt-and-MAC).

Configuration

Server-side Configuration

Here are a few things you should do on the server side configuration to secure OpenSSH.

Disable Weak Key Exchange Protocols

SSH supports a few protocols containing different factors:

1. ECDH curve choice
2. Bit size of the DH modulus
3. The hash function.

To provide good security:

• Avoid NIST curves - they leak data and cannot be trusted.
• Bit sizes smaller than 1024 does not provide sufficient security
• Do not use SHA1

This means, your sshd_config should contain:

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

And your ssh_config should contain:

Host *
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

Prime numbers smaller than 2000 bits should also be removed from the /etc/ssh/moduli file (the 5th column).

Disable Weak Server Authentication

There are 4 public key algorithms for server authentication.

1. DSA with SHA1
2. ECDSA with SHA256, SHA384 or SHA512 depending on key size
3. Ed25519 with SHA512
4. RSA with SHA1

For security:

1. Do not use SSH v1
2. Avoid DSA which requires exactly 1024 bits.
3. Avoid NIST (ECDSA uses it)
4. DSA and ECDSA requires randomness for its signatures. If random numbers are bad, it is possible to recover the secret key.

Using SHA1 here is not an issue if the value being signed is using something stronger (eg SHA2).

Your sshd_config should contain:

Protocol 2
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Disable Weak Symmetric Ciphers

For security:

1. Do not use DES and RC4 ciphers
2. At least 128 bits
3. Do not use Blowfish and Cast128 block ciphers since they use 64 bit block sizes.
4. Authenticated Encryption ciphers should be used so MAC isn't required. But CTR with Encrypt then MAC is also fine.

Your sshd_config should contain:

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

Your ssh_config should contain:

Host *
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

Disable Weak MACs

For security:

1. Do not use MD5 or SHA1 hashing algorithms
2. Encrypt-then-MAC only
3. Tag and key size must be at least 128 bits

Your sshd_config should contain:

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

PermitRootLogin

The PermitRootLogin option specifies whether root can log in. It must be one of:

1. yes to allow root login with password or public key authentication.
2. no to disallow root login.
3. prohibit-password to only allow public key authentication for login. This is the default. without-password is a deprecated alias.
4. force-commands-only to only allow public key authentication for commands.

Key Generation

There are 4 types of keys that can be generated for SSHv2 using the ssh-keygen utility: DSA, RSA, ECDSA, Ed25519. Use ssh-keygen -A to generate all key types.

Don't use DSA (mathematically broken and has since been disabled by default) or ECDSA (possibly cracked?) since they are both inferior. Ed25519 is probably strongest and fastest but not widely supported. RSA is a good alternative that is widely supported.

Usage example:

## Generate all host keys.
# ssh-keygen -A
## Alternatively, each one individually:
# ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
# ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key

Client-side Configuration

SSH client configuration

You can customize SSH's default behavior using a SSH configuration file located at ~/.ssh/config. This is useful when a remote server requires a specific username, key, or port number as it makes it easier for you to set and forget these settings. An example config file might look something like the following:

Host dev
    HostName dev.example.com
    Port 2222
    User leo

With the host defined, connecting to dev.example.com can be done with just ssh dev since all the options are read from the config file.

Use Case: GitHub

If you ever have more than one set of SSH private keys, defining a custom host is virtually required in order to make use of services such as GitHub or GitLab where pushing git changes requires key based SSH authentication.

Host github.com
    User git
    Port 22
    IdentityFile ~/.ssh/github_rsa

With that defined, ssh github.com will connect you as git using the private key github_rsa.

Client key generation

A public/private key pair is required if you wish to set up key-based authentication. If you just want to use plain old SSH using password-based authentication, this isn't required.

A client's public and private key pair should be stored in the ~/.ssh directory. This directory should not be world readable. In fact, the permissions on the ~/.ssh directory is unsafe, none of the keys or authorized_keys file will be used. By default, the SSH client will read the private key from ~/.ssh/id_rsa, ~/.ssh/id_dsa, ~/.ssh/id_ecdsa, or ~/.ssh/id_ed25519 unless otherwise specified using the -i option.

SSH keys can be protected with a passphrase. The private key will be encrypted using this passphrase which is great in the event that someone copies your private keys when you left your console unlocked by accident. This would however require that you unlock the key with this passphrase anytime that you need to use it which may be inconvenient though this can be made easier with SSH agents.

To generate a new SSH key:

# Generates the private/public key pair (~/.ssh/id_rsa, ~/.ssh/id_rsa.pub)
# Use empty pass phrase if you do not want to enter a password for the key.
ssh-keygen -t rsa

# set the permissions of the .ssh directory to 700.
chmod 700 ~/.ssh

# set the permissions of the keys so no one else can read them.
chmod 600 ~/.ssh/*

Changing a SSH key passphrase

After a key is created, if you wish to add or change the passphrase of a key, use ssh-keygen -p.

## Pass '-f ~/.ssh/id_rsa' to avoid being prompted.
$ ssh-keygen -p

Key-based authentication

Once you have set up a public/private key pair, you can use the public key to authenticate to your account by setting up key-based authentication. Any public key that is placed within the ~/.ssh/authorized_keys file will be authorized to log in to your account. There are some options which you can set in the authorized_keys file to limit certain options (such as specific commands, or disabling features like port forwarding).

Security Warning

Having an insecure private key is a security risk. You may wish to secure it with a password and then unlock it using a keyring / SSH agent. More on this in the next section.

Next, copy the public key to the remote host you wish to log in. You can either do this the complicated way, or use ssh-copy-id:

$ cat ~/.ssh/id_rsa.pub | ssh username@remote_machine "cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
## or
$ ssh-copy-id username@remote_machine

Either command will append the contents of your public key into the remote machine's authorized_keys file and then sets permission. ssh-copy-id goes a step further by ensuring that the key doesn't already exist before adding it to the file. With this complete, you should now be able to authenticate using the private key.

$ ssh username@remote_machine
user@remote_machine$

SSH Agents

Using a private key without a passphrase may be a security risk since a compromised key will allow anyone access to all systems using this key. One layer of security is to set a passphrase on the key. While this defeats the point of logging in to a system without a password, this can be worked around using SSH agents. SSH agents will unlock your private key and store it in memory for subsequent use. You can also configure SSH agent to 'forget' or timeout your key after a certain period of time.

A helper program called Keychain available at https://funtoo.org/Keychain makes it easy to use SSH agent. Load it in your .bashrc file to automatically start a SSH agent on first log in and to unlock any SSH keys that you wish to use.

# Attempt to load the keychain to unlock 'id_rsa' if on an interactive shell.
if [ "$PS1" ]; then
    eval `keychain --eval --agents ssh id_rsa`
fi

Quick usage for keychain:

## To unlock a key for 60 minutes
$ keychain --timeout 60 ~/.ssh/id_rsa

## To clear any keys
$ keychain --clear

## To list the public key of unlocked SSH keys
$ keychain -L

## To list the RSA fingerprint of unlocked SSH keys
$ keychain -l

## Kill all agents
$ keychain -k all

Tunnels

You can create a tunnel through your SSH connection to access services that may be behind a firewall or NAT. There are 3 types of tunnels which will be covered below.

Some other tips:

• It may be be desirable to not start a shell on the remote server. This can be done with the ssh -N option.
• When the tunnel cannot be set up for some reason (such as if the port is in use), you may want the SSH command to exit instead of continuing (as is the default behavior). This can be done with the ssh -o ExitOnForwardFailure=yes option.

Local tunnel

Local tunnels allow access a specific port on the host you are connected to. Local tunnels allow you to access a remote port or server locally. This is useful when the server is behind a firewall and you want to access a specific service that is not exposed (such as a web portal or status page, or some other service bound to 127.0.0.1 on the server). You can create a local tunnel with the -L local_port:remote_address:remote_port.

Local tunnels can also access any host via the server. This makes the server act as an intermediate host or proxy and can be useful to access a host that is behind a firewall or within a secure network, for example.

Remote or reverse tunnel

See Also: Reverse SSH Tunnel

Remote tunnels allow access a specific port on the client machine from the server. This remote tunnel allows access from the remote host to a local port. This is useful when the client is behind a NAT or firewall and you want to SSH from the server back to the client or access some service on the client from the server.

You can create a remote tunnel using -R remote_port:local_address:local_port. For example:

client$ ssh -R 22222:localhost:22 user@remote-server

## On another session on remote-server, you may now SSH back to the client:
remote-server$ ssh -p 22222 user@127.0.0.1

Dynamic tunnel

Dynamic tunnels will allow connections to resources via the SSH server as a proxy. This is useful when you want to access resources via the server, such as an iDRAC on a server-only subnet.

You can create a dynamic tunnel using -D local_port. Use 127.0.0.1:local_port as a SOCKS5 proxy.

Tips & Tricks with SSH

Common SSH options

The SSH client requires each individual SSH option to be specified by a -o. You can have any number of options when connecting. For convenience, you may want to put these options inside your ~/.ssh/config file. These options also apply to other SSH tools like scp.

Listed below are some common SSH connect options and any aliases that I've set on my shell.

Description	SSH command
SSH without host key checking. (aliased as sshnc)	ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no
SSH without public key authentication. (aliased as sshnk)	ssh -o PubkeyAuthentication=no
SSH using public key authentication. (aliased as sshok)	ssh -o PreferredAuthentications=publickey
SSH to a specific port	ssh -o Port=2222
Exit if port forwarding fails rather than continue without the port forwarding. Useful if this is executed as a service where the failure requires a retry later on.	ssh -o ExitOnForwardFailure=yes

SSH client and host configs

You may customize your SSH client using the ~/.ssh/config configuration file. The options before the first 'host' stanza applies to all hosts that you connect to using the SSH client. Options that are defined after a 'host' stanza only applies to its specific host.

My SSH client config typically looks like the following.

ForwardAgent yes
ConnectTimeout 300
HashKnownHosts yes

Host xyz
    Hostname full.host.name.com  (or 1.2.3.4)
    User my-username
    IdentitiesOnly yes
    IdentityFile ~/.ssh/id_x_ed25519

To make my private SSH keys 'pass through' to remote hosts, I always enable the SSH agent with the ForwardAgent yes option. You should disable this if connecting to an untrusted machine.

As an added security measure, I also hash my known hosts to prevent someone from seeing what hosts I've connected to previously using the HashKnownHosts yes option.

SSH Connection Sharing

If you have multiple SSH connections to a server, you may want to enable ControlMaster and ControlPath so that subsequent connections shares the existing TCP connection. This will cause SSH to multiplex multiple sessions through the single connection reducing the need to authenticate again. Keep in mind that the first connection must remain open for the other sessions to work.

Host x
    ...
    ControlMaster auto
    ControlPath ~/.ssh/master-%r@%h:%p

This feature is nice for scripts that rely heavily on executing commands on a remote server.

Proxy Hosts

You can use a host as an intermediate server transparently. This feature is akin to the user manually running 'ssh user@intermediate' upon login. However, using this option will also make SSH tunnels be automatically forwarded through these intermediate servers.

For example, to connect to a remote host via an intermediate host, use the -J option followed by one or more intermediate hosts in the order they should be connected to:

ssh -J user@intermediate user@finaldestination
ssh -J user@intermediate1,user@intermediate2 user@finaldestination

Alternatively, you may specify the intermediate hosts by setting the ProxyJump option in your SSH client configuration. For example, you could define the following in your ~/.ssh/config file:

Host finaldestination
    ProxyJump user@intermediate

With this configuration defined, running ssh user@finaldestination will automatically have the SSH client connect to user@intermediate first.

SSH tunnels with proxy hosts

SSH tunnels will be routed through all intermediate jump hosts automatically. As a result, you can do some interesting things such as exposing a port of a host that's behind multiple intermediate hosts.

For example, if we had a chain of hosts like this: [client] -> [hostA] -> [hostB] -> [hostC] and wanted to create a SSH tunnel from client to hostC, we can do something like this:

$ ssh user@hostB -J root@hostA:2222 -L 4444:hostC:22

Proxying by host name

Because SSH supports wildcard hostname configurations, you can define a rule so that hosts ending with an arbitrary hostname, such as -via-$hostname, is routed through a jump host by the name $hostname.

More information at https://jloh.co/posts/dynamic-ssh-jump-hosts/, but the gist of it is to define:

Host *-via-dc1
    ProxyCommand ssh you@jump-01.example.com nc $(echo %h | sed 's/-via-dc1$//') %p

Copying Public Key

Use the ssh-copy-id command to automatically append to a remote host's .ssh/authorized_keys file.

$ ssh-copy-id -i ~/.ssh/id_x_ed25519.pub x

Alternatively, use

$ cat ~/.ssh/id_x_ed25519.pub

Custom MOTD for specific users or groups

The /etc/ssh/sshd_config allows conditional options applied to specific users or groups (or many other criteria) using the Match directive.

To show a specific user a custom login banner:

Match User leo
        Banner /etc/motd-leo

This applies to groups as well:

Match Group awesome-group
        Banner /etc/motd-group

Reload the SSH daemon to apply.

Limit SSH connection to only a specific command

You can specify a specific command be executed when a particular public key is used for authentication. Generate a new public/private keypair and then add it to your authorized_keys file. Specify the command="" as well as any other options/restrictions when using this key.

command="/home/lleung/send_emails_from_arc.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1y... lleung@alpha

Tar Pipe

You can send files via SSH using tar:

# tar -czvf - /dir | ssh root@server "cat > ~/dir.tar.gz"

Troubleshooting

SSH Breaks during system update

On Fedora 31, when executing a system upgrade, the SSH server stops functioning and fails to start. Journald shows the following messages from OpenSSH:

Jun 06 22:42:23 bnas systemd[1]: Starting OpenSSH server daemon...
Jun 06 22:42:23 bnas sshd[18240]: command-line: line 0: Bad configuration option: CASignatureAlgorithms
Jun 06 22:42:23 bnas systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Jun 06 22:42:23 bnas systemd[1]: sshd.service: Failed with result 'exit-code'.
Jun 06 22:42:23 bnas systemd[1]: Failed to start OpenSSH server daemon.
Jun 06 22:43:05 bnas systemd[1]: sshd.service: Service RestartSec=42s expired, scheduling restart.
Jun 06 22:43:05 bnas systemd[1]: sshd.service: Scheduled restart job, restart counter is at 20.
Jun 06 22:43:05 bnas systemd[1]: Stopped OpenSSH server daemon.

This apparently is a known issue by Red Hat (see https://access.redhat.com/solutions/4587501). This can be an issue if the system being updated has many packages and/or is very slow.

Perhaps going forward, update OpenSSH separately to ensure it doesn't break.

Reading a server's key fingerprint

To get the SSH fingerprint of a remote host, use the ssh-keyscan and ssh-keygen utilities:

ssh-keyscan $1 > $$.tmp 2> /dev/null 
ssh-keygen -E md5 -lf $$.tmp
rm $$.tmp

For example, a server would produce:

2048 MD5:76:12:f7:27:59:6d:97:e7:32:db:18:10:4e:df:9a:61 csa (RSA)
256 MD5:bf:ae:71:b2:c2:d0:4d:03:f4:fe:93:12:6f:b7:36:ae csa (ECDSA)
256 MD5:10:8d:0d:d6:16:f4:42:11:1c:c5:06:1c:16:e0:76:c9 csa (ED25519)

Generate a public key from a private key

To generate a public key using a private key:

$ ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

See Also

• https://stribika.github.io/2015/01/04/secure-secure-shell.html