OpenSSH/Cookbook/Public Key Authentication

 

Authentication keys can improve efficiency, if done properly. As a bonus advantage, the passphrase and private key never leave the client. Key-based authentication is generally recommended for outward facing systems so password authentication can be turned off.

Key-based authentication

OpenSSH can use public key cryptography for authentication. In public key cryptography, encryption and decryption are asymmetric. The keys are used in pairs, a public key to encrypt and a hidden key to decrypt. ssh-keygen(1) can make RSA, Ed25519, or ECDSA keys for authenticating. Technically, DSA keys can still be made, being exactly 1024 bits in size, but they are no longer recommended and should be avoided. RSA keys are allowed to vary from 1024 bits on up. However, there is limited benefit after 2048 bits and elliptic curve algorithms are preferable. Ed25519 keys have a fixed length. ECDSA can be one of 256, 384 or 521 bits. Shorter keys are faster, but less secure. Longer keys are much slower to work with but provide better protection, up to a point.

The key files can be named anything, so it is possible to have many keys each for different services or tasks. The comment field at the end of the public key can be useful in helping to keep the keys sorted, if you have many of them or use them infrequently.

With key-based authentication, a copy of the public key is stored on the server and the private key is kept on the client. When the client first contacts the server, the server responds by using the client's public key to encrypt a random number and return that encrypted random number as a challenge to the client. The client uses the matching private key to decrypt the challenge and extract the random number. The client then makes an MD5 hash of the session ID and the random number from the challenge and returns that hash to the server. The server then makes its own hash of the session ID and the random number and compares that to the hash returned by the client. If there is a match, the login is allowed. If there is not a match, then the next public key on the server is tried until either a match is found or all the keys have been tried.

If an agent is used on the client side to manage authentication, the process is similar. It only differs in that ssh(1) passes the challenge off to the agent which then calculates the response and passes it back to ssh(1) which then passes the agent's response back to the server.

Basics of Public Key Authentication

A matching pair of keys is needed for public key authentication. ssh-keygen(1) is used to make the key pair. From that pair the public key must be properly stored on the remote host and the private key stored safely on the client. The public key is added to the designated authorized_keys file for the remote user account. Once the keys have been prepared they can be used again and again without needing to do anything else.

There are six steps in preparation for key-based authentication:

1) If either the authorized_keys file or .ssh directory do not exist on either the client machine or the remote machine, create them and set the permissions correctly:

 
$ mkdir ~/.ssh/
$ touch ~/.ssh/authorized_keys
$ chmod 0700 ~/.ssh/
$ chmod 0600 ~/.ssh/authorized_keys

2) Create a key pair. The example here creates a Ed25519 key pair in the directory ~/.ssh. The option -t assigns the key type and the option -f assigns the key file a name. It is good to give keys descriptive file names, especially if larger numbers are managed. As a result of the line below, the public key will be seen named mykey_rsa.pub and and the private key will be called mykey_rsa. Be sure to enter a sound passphrase to encrypt the private key using 128-bit AES.

 
$ ssh-keygen -t ed25519 -f ~/.ssh/mykey_ed25519

Ed25519 keys have a fixed length. For RSA and ECDSA keys, the -b option sets the number of bits used.

Since 6.5 a new private key format is available using a bcrypt(3) key derivative function (KDF) to better protect keys at rest. This new format is always used for Ed25519 keys, and sometime in the future will be the default for all keys. But for right now it may be requested when generating or saving existing keys of other types via the -o option in ssh-keygen(1).

 
$ ssh-keygen -o -b 4096 -t rsa -f ~/.ssh/mykey_rsa

Details of the new format are found in the source code in the file PROTOCOL.key.

3) Transfer both the public and private keys to the client machine if they are not already there from the step above. It is usually best to store both the public and private keys in the directory ~/.ssh/ together.

4) Transfer only the public key to remote machine.

 
$ scp ~/.ssh/mykey_rsa.pub fred@remotehost.example.org:.

5) Log in to the remote machine and add the new public key to the authorized_keys file, whether in ~/.ssh/authorized_keys, the default, or somewhere else as designated by the server's configuration.

 
$ cat mykey_rsa.pub >> ~/.ssh/authorized_keys
$ nano -w ~/.ssh/authorized_keys

Any editor that does not wrap long lines can be used. In the example above, nano(1) is started with the -w option to prevent wrapping of long lines. The same can also be accomplished permanently by editing nanorc(5) However it is done, the key must be in the file whole and unbroken, on a single line.

Alternately, use 'ssh-copy-id to replace steps #4 and #5 if your system has it.

6) Test the keys

While still logged in, start another SSH session in a new window and authenticate to the remote machine from the client using the private key.

 
$ ssh -i ~/.ssh/mykey_rsa -l fred remotehost.example.org

The option -i tells ssh(1) which private key to try.

➥ Troubleshooting of Key-based Authentication:

If the server refuses to accept the key and fails over to the next authentication method (eg: "Server refused our key"), then there are several possible mistakes to look for on the server side.

A common error, if a key doesn't work, is that the file permissions are wrong. The authorized key file must be owned by the user in question and not be group writable. Nor may the key file's directory be group or world writable.

 
$ chmod u=rwx,g=rx,o= ~/.ssh
$ chmod u=rw,g=,o=  ~/.ssh/authorized_keys

Another mistake that can happen is if the key is broken by line breaks or has other white space in the middle. That can be fixed by joining up the lines and removing the spaces or by recopying the key more carefully.

And, though it should go without saying, the halves of the key pair need to match. The public key on the server needs to match the private key held on the client. If the public key is lost, then a new one can be generated with the -y option, but not the other way around. If the private key is lost, then the public key should be erased as it is no longer of any use. If many keys are in use for an account, it might be a good idea to add comments to them. On the client, it can be a good idea to know which server the key is for, either through the file name itself or through the comment field. A comment can be added using the -C option.

 
$ ssh-keygen -b 4096 -t rsa -f ~/.ssh/mykey_rsa -C "web server mirror"

On the server, it can be important to annotate which client they key is from if there is more than one public key there in an account. There the comment can be added to the authorized key file on the server in the last column if a comment does not already exist. Again, the format of the authorized keys file is given in the manual page for sshd(8) in the section "AUTHORIZED_KEYS FILE FORMAT". If the keys are not labeled they can be hard to match, which might or might not be what you want.

Associating Keys Permanently with a Server

The Host directive in ssh_config(5) can apply specific settings to a target host. That includes using specific keys with certain hosts. So by changing ~/.ssh/config it is possible to assign keys to be tried automatically whenever making a connection to that host. Here, all that's needed is to type ssh web1 to connect with the key for that server.

 
Host web1
	Hostname 198.51.100.32
	IdentitiesOnly yes
	IdentityFile /home/fred/.ssh/web_key_rsa

Below ~/.ssh/config uses different keys for server versus server.example.org, regardless whether they resolve to the same machine. This is possible because the host name argument given to ssh(1) is not converted to a canonicalized host name before matching.

 
Host server
	IdentityFile /home/fred/.ssh/key_a_rsa

Host server.example.org
	IdentityFile /home/fred/.ssh/key_b_rsa

The configuration file parser is first-match. So in this example the shorter name is tried first. Of course less ambiguous shortcuts can be made instead.

Encrypted Home Directories

When using encrypted home directories the keys must be stored in an unencrypted directory somewhere else and sshd(8) needs to be configured appropriately to find the keys in that special location. Sometimes it is also necessary to add a script or call a program from /etc/ssh/sshrc immediately after authentication to decrypt the home directory.

One symptom of having an encrypted home directory is that key-based authentication only works when you are already logged into the same account, but fails when trying to make the first connection and log in for the first time.

Here is one method for solving the access problem. Each user is given a subdirectory under /etc/ssh/keys/ which they can then use for storing their authorized_keys file. This is set in the server's configuration file /etc/ssh/sshd_config

 
AuthorizedKeysFile      /etc/ssh/keys/%u/authorized_keys

Setting a special location for the keys opens up more possibilities as to how the keys can be managed and multiple key file locations can be specified if they are separated by whitespace. The user does not have to have write permissions for the authorized_keys file. Only read permissoin is needed to be able to log in. But if the user is allowed to add, remove, or change their keys, then they will need write access to the file to do that.

Passwordless Login

One way of allowing passwordless logins is to follow the steps above, but do not enter a passphrase. A better solution is to have a passphrase and work with an authentication agent in conjunction with a single-purpose key. Note that using keys that lack a passphrase is very risky, so the key files should be very well protected and kept track of. That includes that they only be used as single-purpose keys as described below. Timely key rotation becomes especially important. In general, it is not a good idea to make a key without a passphrase.

Requiring Both Keys and a Password

While users should have strong passphrases for their keys, there is no way to enforce or verify that. Indeed, the passphrase for any given key never leaves the client machine so it is nothing that the server can have any influence over. Starting with OpenSSH 6.2, it is possible for the server to require multiple authentication methods for login using the AuthenticationMethods directive.

 
AuthenticationMethods publickey,password

This example requires that users first authenticate using a key and only queries for a password if that succeeds. So with that configuration, it is not possible to get to the system password prompt without first authenticating with a valid key. Changing the order of the arguments changes the order of the authentication methods.

Requiring Two or More Keys

Since OpenSSH 6.8, the server now remembers which public keys have been used for authentication and refuses to accept previously-used keys. This allows a set up requiring that users authenticate using two different public keys, maybe one in the file system and the other in a hardware token.

 
AuthenticationMethods publickey,publickey

The AuthenticationMethods directive, whether for keys or passwords, can also be set on the server under a Match directive to apply only to certain groups or situations.

Requiring Certain Key Types

Also since OpenSSH 6.8, the PubkeyAcceptedKeyTypes directive can specify that certain key types are accepted. Those not in the comma-separated pattern list are not allowed.

 
PubkeyAcceptedKeyTypes ssh-ed25519*,ssh-rsa*,ecdsa-sha2*

Either the actual key types or a pattern can be in the list. Spaces are not allowed in the pattern list. The exact list of supported key types can be found by the -Q option on the client.

 
$ ssh -Q key

For host-based authentication, it is the HostbasedAcceptedKeyTypes directive that determines which key types are allowed for authentication.

Key-based Authentication Using an Agent

When an authentication agent, such as ssh-agent(1), is going to be used, it should generally be started at the beginning of a session and used to launch the login session or X-session so that the environment variables pointing to the agent and its unix-domain socket are passed to each subsequent shell and process. Many desktop distros do this automatically upon login or startup.

Starting an agent entails setting a pair of environment variables: ever us

• SSH_AGENT_PID : the process id of the agent
• SSH_AUTH_SOCK : the filename and full path to the unix-domain socket

The various SSH and SFTP clients find these variables automatically and use them to contact the agent and try when authentication is needed. However, it is mainly SSH_AUTH_SOCK which is only ever used. If the shell or desktop session was launched using ssh-agent(1), then these variables are already set and available. If not, then it is necessary to either set the variables manually inside each shell or for each application in order to use the agent or else to point to the agent's socket using the directive IdentityAgent in the client's configuration file.

Once an agent is available, a private key needs to be loaded before it can be used. Once in the agent it can then be used many times. The private keys are loaded into an agent with ssh-add(1).

 
$ ssh-add /home/fred/.ssh/mykey_ed25519
Identity added: /home/fred/.ssh/mykey_ed25519 (/home/fred/.ssh/mykey_ed25519)

Keys stay in the agent as long as it is running, unless specified otherwise with the -t option when starting the agent or when actually loading the key using the -t option with ssh-add(1). That will set a timeout interval, after which the key will be purged from the agent.

 
$ ssh-add -t 1h30m /home/fred/.ssh/mykey_ed25519
Identity added: /home/fred/.ssh/mykey_ed25519 (/home/fred/.ssh/mykey_ed25519)
Lifetime set to 5400 seconds

By default ssh-add(1) uses the agent connected via the socket named in the environment variable SSH_AUTH_SOCK, if it is set. Currently, that is its only possibility. However, for ssh(1) an alternative to using the environment variable is the client configuration directive IdentityAgent which tells the SSH clients which socket to use to communicate with the agent. If both the environment variable and the configuration directive are available at the same time, then the value in IdentityAgent takes precedence over what's in the environment variable. IdentityAgent can also be set to none to prevent the connection from trying to use any agent at all.

The client configuration directive AddKeysToAgent can also be useful in getting keys into an agent as needed. When set, it automatically loads a key into a running agent the first time the key is called for if it is not already loaded.

The option -l will list the fingerprints of all of the identities in the agent.

 
$ ssh-add -l                                    
256 SHA256:77mfUupj364g1WQ+O8NM1ELj0G1QRx/pHtvzvDvDlOk mykey_ed25519 (ED25519)
3072 SHA256:7unq90B/XjrRbucm/fqTOJu0I1vPygVkN9FgzsJdXbk myotherkey_rsa (RSA)

It is also possible to remove individual identities from the agent using -d which will remove them one at a time by name, but only if the name is given. Without the name of a private key, it will fail silently. Using -D will remove all of them at once without needing to specify any by name.

Agent Forwarding

In agent forwarding, intermediate machines forward challenges and responses back and forth between the client and the final destination. This eliminates the need for passwords or keys on any of these intermediate machines. In other words, an advantage of agent forwarding is that the private key itself is not needed on any remote machine, thus hindering unwanted access to it. Another advantage is that the actual agent to which the user has authenticated does not go anywhere and is thus less susceptible to analysis.

On the client side it is disabled by default and so it must be enabled explicitly. Put the following line in ssh_config(5) to enable agent forwarding for a particular server:

 
Host gateway.example.org
	ForwardAgent yes

The default configuration files for the server enable authentication agent forwarding, so to use it, nothing needs to be done there.

One risk with agents is that they can be re-used to tailgate in if the permissions allow it. Keys cannot be copied this way, but authentication is possible when there are incorrect permissions. Note that disabling agent forwarding does not improve security unless users are also denied shell access, as they can always install their own forwarders.

The risks of agent forwarding can be mitigated by confirming each use of a key by adding the -c option when adding the key to the agent. This requires the SSH_ASKPASS variable be set and available to the agent process, but will generate a prompt on the host running the agent upon each use of the key by a remote system. So if passing through one or more intermediate hosts, it is usually better to instead have the SSH client use stdio forwarding with -W or -J.

Somewhat Safer SSH Agent Forwarding

The best way to pass through one or more intermediate hosts is to use the ProxyJump option instead of SSH agent forwarding and thereby not risk exposing any private keys. If SSH agent forwarding must be used, then in the interest of following the principle of least privilege it would be advisable to only forward an agent containing the minimum necessary number of keys. There are several ways to solve that. One partial solution is to make a one-off, ephemeral agent to hold just the one key or keys needed for the task at hand. Another partial solution would be to set up a user-accessible service at the operating system level and then use ssh_config for the rest.

A rather portable way to automatically launch an ephemeral agent unique to each session is to craft a special shell script function to launch a single-use agent. It can also be written to require confirmation for each requested signature. The following example function is based on a blog post by Vincent Bernat on SSH agent forwarding:

fssh() {
    (
        # Ensure we don't use any existing agent.
        unset SSH_AUTH_SOCK
        unset SSH_AGENT_PID
    
        # Spawn a new, empty agent while setting
        # the related environment variables
        eval $(ssh-agent)
        [ -n "$SSH_AUTH_SOCK" ] || exit 1

        # On exit, kill the agent and unset the variables.
        trap 'eval $(ssh-agent -k)' EXIT

        # Invoke the SSH client with agent forwarding enabled
        # and automatically use the confirmation mode
        # when adding the private key.
        ssh -o AddKeysToAgent=confirm \
            -o ForwardAgent=yes \
            "$@"
    )
}

Using that function the SSH client will be launched with a unique, ephemeral supporting key agent by invoking the function while pointing it at a specific key using the IdentityFile or -i option. When the session is finished the function ends too and the trap will be triggered and it will kill the SSH agent pointed to in the SSH_AGENT_PID environment variable.

Another way is to rely on the client's configuration file for some of the settings. Methods that rely more on ssh_config still require an independent method to launch an ephemeral agent. That is because the OpenSSH client is already running by the time it reads the configuration file and is thus not affected by any changes to environment variables caused by the configuration file and it is through the environment variables that contain information about the agent. However, if the path to the UNIX-domain socket used to communicate with the authentication agent is decided in advance then the IdentityAgent option can point to it once the one-off agent is actually launched. The following uses a specific agent's predefined socket when connecting to two particular domains:

Host *.wikimedia.org *.wmflabs.org
        User fred
        IdentitiesOnly yes
        IdentityFile %d/.ssh/id_cloud_01
        IdentityAgent /run/user/%i/ssh-cloud-01.socket
        ForwardAgent yes
        AddKeysToAgent yes

For that configuration, the agent must already be up and running prior to starting the SSH client. The socket should be in a directory which is inaccessible to any other accounts. On some systems the %i token might come in handy when settinfg the IdentityAgent option inside the configuration file. Again, be careful when forwarding agents with which keys are in the forwarded agent.

However, in the interests of privacy and security in general, agent forwarding is to be avoided. The configuration directive ProxyJump is the best alternative and, on older systems, host traversal using ProxyCommand with netcat are preferable. See the section on Proxies and Jump Hosts for how those methods are used.

Single-purpose Keys

Tailored single-purpose keys can eliminate use of remote root logins for many administrative activities. A finely tailored sudoers is needed along with a user account. When done right, it gives just enough access to get the job done, following the security principle of Least Privilege. Single-purpose keys are accompanied by use of either the ForceCommand directive in sshd_config(5) or the command="..." directive inside the authorized_keys file. Further, neither the authorized_keys file nor the home directory should be writable. When building a single-purpose key, the -v option can show exactly what is being passed to the server so that sudoers can be set up correctly. This is especially needed when setting up encrypted remote backups using rsync(1), tar(1), mysqldump(1), and so on so that they only access designated parts of the file system. For example, here is what ssh -v shows from one particular usage of rsync(1):

 
$ rsync --server -vlHogDtprz --delete --delete-after \
	--ignore-errors . /org/backup

Key-based Authentication, with Limitations on Activity

The authorized_keys file can force a particular program to run whenever the key is used. This is useful for automating various tasks. The following launches firefox automatically and exits the connection when it is finished.

command="/usr/bin/firefox" ssh-rsa AAAAB3NzaC1yc2EAAA...OFy5Lwc8Lo+Jk=

The following further restricts the connection to coming from a single domain.

command="/usr/bin/firefox",from="*.example.net" ssh-rsa AAAAB3...FLoJk=

The manual page sshd(8) has the full list of options for the authorized_keys file.

The following key will only echo some text and then exit, unless used non-interactively with the -N option. No matter what the user tries, it will echo the text unless the -N option disables running the remote program, allowing the connection to stay open.

command="/bin/echo do-not-send-commands" ssh-rsa AAAAB3...99ptc=

This is very useful for keys that are only used for tunnels.

Read-only Access to Keys

In some cases it is necessary to prevent the users from changing their own authentication keys. This can be done by putting the key file in an external directory where the user has read-only access to the key file and no write permissions to either the file or the directory. The AuthorizedKeysFile directive sets where sshd(8) looks for the keys and can point to the secured location for the keys instead of the default location.

The default location for keys on most systems is usually ~/.ssh/authorized_keys. A good alternate location could be to create the directory /etc/ssh/authorized_keys and store the selected users' key files there. The change can be made to apply to only a group of users by putting it under a Match directive.

 
Match Group sftpusers
	AuthorizedKeysFile /etc/ssh/authorized_keys/%u

This works even within a chrooted environment.

 
Match Group sftpusers
	ChrootDirectory /home
	ForceCommand internal-sftp -d %u
	AuthorizedKeysFile /etc/ssh/authorized_keys/%u

Of course it could be set to affect all users by putting the directive in the main part of the server configuration file.

Mark Public Keys as Revoked

Keys that have been revoked can be stored in /etc/ssh/revoked_keys, a file specified in sshd_config(5) using the directive RevokedKeys, so that sshd(8) will prevent attempts to log in with them. No warning or error on the client side will be given if a revoked key is tried. Authentication will simply progress to the next key or method.

The revoked keys file should contain a list of public keys, one per line, that have been revoked and can no longer be used to connect to the server. The key cannot contain any extras, such as login options or it will be ignored. If one of the revoked keys is tried during a login attempt, the server will simply ignore it and move on to the next authentication method. An entry will be made in the logs of the attempt, including the key's fingerprint. See the section on logging for a little more on that.

 
RevokedKeys /etc/ssh/revoked_keys

The RevokedKeys configuration directive is not set in sshd_config(5) by default. It must be set explicitly if it is to be used.

Key Revocation Lists

A compact, binary form of representing revoked keys and certificates is available as a Key Revocation List (KRL). KRLs are generated with ssh-keygen(1) and can be created fresh.

 
$ ssh-keygen -kf  /etc/ssh/revoked_keys -z 1 ~/.ssh/old_key_rsa.pub

Or an existing one can updated.

 
$ ssh-keygen -ukf /etc/ssh/revoked_keys -z 2 ~/.ssh/old_key_dsa.pub

It is possible to test if a specific key or certificate is in the revocation list.

 
$ ssh-keygen -Qf  /etc/ssh/revoked_keys ~/.ssh/old_key_rsa.pub

Only public keys and certificates will be loaded into the KRL. Corrupt or broken keys will not be loaded and will produce an error message if tried. Like with the regular RevokedKeys list, the public key destined for the KRL cannot contain any extras like login options or it will produce an error when an attempt is made to load it into the KRL or search the KRL for it.

Verify a Host Key by Fingerprint

The first time connecting to a remote host, the key itself should be verified. Usually this is done by comparing the fingerprint which is metadata about the key rather than trying to compare the whole key itself.

 
$ ssh -l fred server.example.org
The authenticity of host 'server.example.org (192.0.32.10)' can't be established.
ECDSA key fingerprint is SHA256:LPFiMYrrCYQVsVUPzjOHv+ZjyxCHlVYJMBVFerVCP7k.
Are you sure you want to continue connecting (yes/no)?

That examples shows the key as a the usual base64 encoded SHA256 checksum. The fingerprint can also be displayed as an MD5 hash in hexadecimal instead by passing the client's FingerprintHash configuration directive as a runtime argument or setting it in ssh_config. But the default in new versions is SHA256 in base64.

 
$ ssh -o FingerprintHash=md5 host.example.org
The authenticity of host 'host.example.org (192.0.32.203)' can't be established.
RSA key fingerprint is MD5:10:4a:ec:d2:f1:38:f7:ea:0a:a0:0f:17:57:ea:a6:16.
Are you sure you want to continue connecting (yes/no)?

In OpenSSH 6.7 and earlier, the client showed fingerprints as a hexadecimal MD5 checksum instead a of the base64-encoded SHA256 checksum currently used:

 
$ ssh -l fred server.example.org
The authenticity of host 'server.example.org (192.0.32.10)' can't be established. 
RSA key fingerprint is 4a:11:ef:d3:f2:48:f8:ea:1a:a2:0d:17:57:ea:a6:16. 
Are you sure you want to continue connecting (yes/no)?

Another way of comparing keys is to use the ASCII art visual host key.

Downloading keys

Usually a host’s key is displayed for review the first time the SSH client tries to connect. But the remote host’s public keys can also be fetched on demand at any time using ssh-keyscan(1):

 
$ ssh-keyscan host.example.org  
# host.example.org SSH-2.0-OpenSSH_7.2
host.example.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLC2PpBnFrbXh2YoK030Y5JdglqCWfozNiSMjsbWQt1QS09TcINqWK1aLOsNLByBE2WBymtLJEppiUVOFFPze+I=
# host.example.org SSH-2.0-OpenSSH_7.2
host.example.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iViojCZkcpdLju7/3+OaxKs/11TAU4SuvIPTvVYvQO32o4KOdw54fQmd8f4qUWU59EUks9VQNdqf1uT1LXZN+3zXU51mCwzMzIsJuEH0nXECtUrlpEOMlhqYh5UVkOvm0pqx1jbBV0QaTyDBOhvZsNmzp2o8ZKRSLCt9kMsEgzJmexM0Ho7v3/zHeHSD7elP7TKOJOATwqi4f6R5nNWaR6v/oNdGDtFYJnQfKUn2pdD30VtOKgUl2Wz9xDNMKrIkiM8Vsg8ly35WEuFQ1xLKjVlWSS6Frl5wLqmU1oIgowwWv+3kJS2/CRlopECy726oBgKzNoYfDOBAAbahSK8R
# host.example.org SSH-2.0-OpenSSH_7.2
host.example.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDDOmBOknpyJ61Qnaeq2s+pHOH6rdMn09iREz2A/yO2m

Once a key is acquired, its fingerprint can be shown using ssh-keygen(1). This can be done directly with a pipe.

 
$ ssh-keyscan host.example.org | ssh-keygen -lf - 
# host.example.org SSH-2.0-OpenSSH_7.2
# host.example.org SSH-2.0-OpenSSH_7.2
# host.example.org SSH-2.0-OpenSSH_7.2
256 SHA256:sxh5i6KjXZd8c34mVTBfWk6/q5cC6BzR6Qxep5nBMVo host.example.org (ECDSA)
3072 SHA256:hlPei3IXhkZmo+GBLamiiIaWbeGZMqeTXg15R42yCC0 host.example.org (RSA)
256 SHA256:ZmS+IoHh31CmQZ4NJjv3z58Pfa0zMaOgxu8yAcpuwuw host.example.org (ED25519)

If there is more than one public key type is available from the server on the port polled, then ssh-keyscan(1) will fetch them. If there is more than one key fed via stdin or a file, then ssh-keygen(1) will process them in order. Prior to OpenSSH 7.2 manual fingerprinting was a two step process, the key was read to a file and then processed for its fingerprint.

 
$ ssh-keyscan -t ed25519 host.example.org > key.pub
# host.example.org SSH-2.0-OpenSSH_6.8
$ ssh-keygen -lf key.pub                       
256 SHA256:ZmS+IoHh31CmQZ4NJjv3z58Pfa0zMaOgxu8yAcpuwuw host.example.org (ED25519)

Note that some output from ssh-keyscan(1) is sent to stderr instead of stdout.

A hash, or fingerprint, can be generated manually with awk(1), sed(1) and xxd(1), on systems where they are found.

 
$ awk '{print $2}' key.pub | base64 -d | md5sum -b | sed 's/../&:/g; s/: .*$//'
$ awk '{print $2}' key.pub | base64 -d | sha256sum -b | sed 's/ .*$//' | xxd -r -p | base64

It is possible to find all hosts from a file which have new or different keys from those in known_hosts, if the host names are in clear text and not stored as hashes.

 
$ ssh-keyscan -t rsa,ecdsa -f ssh_hosts | \
  sort -u - ~/.ssh/known_hosts | \
  diff ~/.ssh/known_hosts -

ASCII Art Visual Host Key

An ASCII art representation of the key can be displayed along with the SHA256 base64 fingerprint:

 
$ ssh-keygen -lvf key 
256 SHA256:BClQBFAGuz55+tgHM1aazI8FUo8eJiwmMcqg2U3UgWU www.example.org (ED25519)
+--[ED25519 256]--+
|o+=*++Eo         |
|+o .+.o.         |
|B=.oo.  .        |
|*B.=.o .         |
|= B *   S        |
|. .@ .           |
| +..B            |
|  *. o           |
| o.o.            |
+----[SHA256]-----+

In OpenSSH 6.7 and earlier the fingerprint is in MD5 hexadecimal form.

 
$ ssh-keygen -lvf key 
2048 37:af:05:99:e7:fb:86:6c:98:ee:14:a6:30:06:bc:f0 www.example.net (RSA) 
+--[ RSA 2048]----+ 
|          o      | 
|         o .     | 
|        o o      | 
|       o +       | 
|   .  . S        | 
|    E ..         | 
|  .o.* ..        | 
|  .*=.+o         | 
|  ..==+.         | 
+-----------------+

More on Verifying SSH Keys

Keys on the client or the server can be verified against known good keys by comparing the base64-encoded SHA256 fingerprints.

Verifying Client Key Pairs

The comparison of key pairs held on the client machine needs to be done in several steps using ssh-keygen(1). First, a new public key is generated to stdout from the known private key and used to make a fingerprint. Next, the fingerprint of the unknown public key is generated.

 
$ ssh-keygen -y -f my_key_a_rsa | ssh-keygen -l -f - ; \
  ssh-keygen -l -f my_key_b_rsa.pub

The result is a base64-encoded SHA256 checksum for each key with the one fingerprint displayed right below the other. So, the comparison can be done visually with ease. Older versions don't support reading from stdin so an intermediate file will be needed then. Even older versions will only show an MD5 checksum for each key. Either way, automation with a shell script is simple enough to accomplish but outside the scope of this book.

However, since public keys are disposable, the easy way in such situations on the client machine is to just rename or erase the old, problematic, public key and replace it with a new one generated from the existing private key.

 
$ ssh-keygen -y -f ~/.ssh/my_key_rsa

If -e and -m are needed for conversion to a specific key format, then use them as well. The same goes for -C for adding a comment. Any options modifying or restricting the public key will have to be prepended manually after the file is generated. See the earlier section above on generating new keys for more explanation.

Verifying Server Keys

Reliable verification of a server's host key must be done when first connecting. It can be necessary to contact the system administrator who can provide it out of band so as to know the fingerprint in advance and have it ready to verify the first connection.

Here is an example of the server's RSA key being read and its fingerprint shown as SHA256 base64:

 
$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub     
3072 SHA256:hlPei3IXhkZmo+GBLamiiIaWbeGZMqeTXg15R42yCC0 root@server.example.net (RSA)

And here the corresponding ECDSA key is read, but shown as an MD5 hexadecimal hash:

 
$ ssh-keygen -E md5 -lf /etc/ssh/ssh_host_ecdsa_key.pub
256 MD5:ed:d2:34:b4:93:fd:0e:eb:08:ee:b3:c4:b3:4f:28:e4 root@server.example.net (ECDSA)

Prior to 6.8, the fingerprint was expressed as an MD5 hexadecimal hash:

 
$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub 
2048 MD5:e4:a0:f4:19:46:d7:a4:cc:be:ea:9b:65:a7:62:db:2c root@server.example.net (RSA)

It is also possible to use ssh-keyscan(1) to get keys from an active SSH server. However, the fingerprints still needs to be verified out of band.

Warning: Remote Host Identification Has Changed!

If the server's key does not match what has been recorded in either the system's or the local user's authorized_keys files, then the SSH client will issue a warning. Two main reasons for the warning exist. One is if the server's keys haves been replaced, maybe because the server's operating system was reinstalled without backing up the old keys. Another is when the system administrator has phased out deprecated or compromised keys. If there is time to plan the migration, new keys can just be added to the server and the clients use the UpdateHostKeys option to accept them if the old keys match. Yet another situation is when the connection is made to the wrong machine, one such case would be the during an attempted man in the middle attack or when contacting a server that changes addresses because of dynamic address allocation.

In cases where the key has changed there is only one thing to do: contact the system administrator and verify the key. The system administrator may be you yourself in some cases. If so, was OpenSSH-server recently reinstalled, or was the machine restored from an old backup?

The new key fingerprint can be sent out by some method where it is possible to verify the integrity and origin of the message, for example via PGP-signed e-mail. Then go over to the client machine where you got the error and remove the old key from ~/.ssh/known_hosts

 
$ ssh-keygen -R host.example.org

Then try logging in, but enter the password if and only if the key fingerprint matches what you got on the server console. If the key fingerprint matches, then go through with the login process and the key will be automatically added. If the key fingerprint does not match, stop immediately and it would be a good idea to get on the phone, a real phone not a computer phone, to the remote machine's system administrator or the network administrator.

Multiple Keys for a Host, Multiple Hosts for a Key in known_hosts

It is possible to assign multiple host names or ip addresses to the same key using pattern matching in known_hosts. That goes for both those global keys in /etc/ssh/ssh_known_hosts and those local user-specific key lists in ~/.ssh/known_hosts. These should provide you with an up-to-date set to add to the known hosts file. Otherwise, you must verify the keys by hand.

server1,server2,server3 ssh-rsa AAAAB097y0yiblo97gvl...jhvlhjgluibp7y807t08mmniKjug...==

You can use globbing to a limited extent in either /etc/ssh/ssh_known_hosts or ~/.ssh/known_hosts.

172.19.40.* ssh-rsa AAAAB097y0yiblo97gvl...jhvlhjgluibp7y807t08mmniKjug...==

However, to get the effect you want, multiple keys for the same address, make multiple entries in either /etc/ssh/ssh_known_hosts or ~/.ssh/known_hosts for each key.

server1 ssh-rsa AAAAB097y0yiblo97gvljh...vlhjgluibp7y807t08mmniKjug...==
server1 ssh-rsa AAAAB0liuouibl kuhlhlu...qerf1dcw16twc61c6cw1ryer4t...==
server1 ssh-rsa AAAAB568ijh68uhg63wedx...aq14rdfcvbhu865rfgbvcfrt65...==

To get a pool of servers to share a pool of keys, each server-key combination must be added manually to the known_hosts file.

 
server1 ssh-rsa AAAAB097y0yiblo97gvljh...07t8mmniKjug...==
server1 ssh-rsa AAAAB0liuouibl kuhlhlu...qerfw1ryer4t...==
server1 ssh-rsa AAAAB568ijh68uhg63wedx...aq14rvcfrt65...==

server2 ssh-rsa AAAAB097y0yiblo97gvljh...07t8mmniKjug...==
server2 ssh-rsa AAAAB0liuouibl kuhlhlu...qerfw1ryer4t...==
server2 ssh-rsa AAAAB568ijh68uhg63wedx...aq14rvcfrt65...==

This will fetch the RSA key for server1 and put the it key in a file (e.g. z.key) and give a fingerprint for the key in that file:

 
$ ssh-keyscan -t rsa server1 
$ ssh-keyscan -t rsa server1 > z.key 
$ ssh-keygen -l -f z.key

Then the file's contents can be copied into known_hosts.

Another way of Dealing with Dynamic (roaming) IP Addresses

It is possible to manually point to the right key using HostKeyAlias either as part of ssh_config(5) or as a runtime parameter. Here the key for machine Foobar is used to connect to host 192.168.11.15

 
$ ssh -o StrictHostKeyChecking=accept-new \
      -o HostKeyAlias=foobar   192.168.11.15

This is useful when DHCP is not configured to try to keep the same addresses for the same machines over time or when using certain stdio forwarding methods to pass through intermediate hosts.

Hostkey Update and Rotation in known_hosts

A protocol extension to rotate weak public keys out of known_hosts has been in OpenSSH from version 6.8 and later. With it the server is able to inform the client of all its host keys and update known_hosts with new ones when at least one trusted key already known. This method still requires the private keys be available to the server so that proofs can be completed. In ssh_config(5), the directive UpdateHostKeys specifies whether the client should accept updates of additional host keys from the server after authentication is completed and add them to known_hosts. A server can offer multiple keys of the same type for a period before removing the deprecated key from those offered, thus allowing an automated option for rotating keys as well as for upgrading from weaker algorithms to stronger ones.

 

References