I recently designed and implemented a personal OpenSSH infrastructure that leverages OpenSSH’s certificate-based authentication and its support for FIDO2-backed keys to build a robust and secure solution for accessing local and remote machines both at home and on the go. This post serves as a documentation and rough guide to its design for myself and posterity.
Considerations and goals
Afters tens of years of administrating multiple different systems I generally
know my way around SSH but the default experience is quite lacking in a couple
of spots. TOFU is not
really ergonomic, and managing multiple keys on multiple devices can quickly
get confusing. Worse still if a key was compromised, since suddenly you need to
make sure that the key is revoked for all possible systems it could be used
for. I’m sure I’m not the only one having stumbled upon an old
authorized_keys file and going: “That key was still allowed here!?”
So after all these years I’ve built up a bit of an advanced list of requirements:
- User authentication must be bound to a physical device, one that I can optionally take with me.
- User authentication must require additional knowledge (a PIN, for example), physical access to the device itself is not sufficient.
- All relevant key material resides on the physical device; private keys must never leave it.
- The infrastructure must be reasonably robust: if I lose a device, I must still be able to log in with another.
- I must be able to provision new devices without having to transfer their public key material to all existing hosts.
- Similarly, I must be able to trust hosts without trusting each of their specific public keys.
- If a device, user, or host is compromised, its revocation must be quick, painless, and global.
- Auditing must be available. I need a record of each time a user accessed a system, and which device was used in the process.
Thankfully, all of this is possible using
YubiKeys and OpenSSH. I already had three of
the former, but because their really old firmware does not support ed25519
keys I invested in three new ones. With three keys available, I don’t have to
worry about losing one.
YubiKeys also tick a lot of the rest of the boxes: they allow resident keys, work well with OpenSSH, and can be configured to require a PIN when used with it. For the rest of the requirements, a couple of OpenSSH features come into play: host and user certificate signing via a certificate authority, key revocation lists, and its auditing functionality through syslog. We’ll have a look at all these features separately.
Resident keys
YubiKeys may store SSH keys within its PIV, PGP, or FIDO2
modules. As the page explains, there’s pros
and cons for each of these. First off, PGP is out because I simply don’t like
it. The PIV module is a decent choice, but I prefer OpenSSH’s built-in FIDO
support instead of relying on a
PKCS#11 library. Hence we’ll be
generating ed25519-sk keys on each YubiKey. These can later be imported with
OpenSSH, and are also visible as a passkey on the Yubico
Authenticator.
Certificates, and the Certificate Authority
SSH allows signing of both
user and host keys to create user and host certificates. This is done using
something called a certificate authority (or CA). The CA itself is simply
another SSH key pair, the public part of which can be marked as trusted for an
entire system. This way, any certificate presented to a system that was signed
by a trusted CA is itself trusted and may be used for authentication or
verification. Once in place, a CA saves you from ever again having to add your
keys to the authorized_keys file or needing to
TOFU a host key.
A certificate itself has a serial number, an identifier, may be limited to a certain time frame (called its validity) and may contain additional restrictions. For user certificates, one may restrict usage of a certificate for a certain set of users. For host certificates, one may restrict usage for a certain set of hostnames. These sets are called the principals. User certificates may even contain additional restrictions such as a forced command, or an allowlist of source network addresses.
Note that whilst both the user and host certificates may have validities, the CA itself does not have one. Obviously one may still rotate the key after a specific time period, but there is no mechanism in OpenSSH that makes this mandatory.
Since the CA is itself an SSH key pair, we can also generate and store it on a
YubiKey. However, because ssh-keygen(1)
can only download key material for the CA via a
PKCS#11 library, we need to use the
PIV backend instead of the FIDO one.
Later on, host certificates will be uploaded to the specific host’s /etc/ssh
directory, whilst user certificates will be stored using the YubiKey’s
largeBlobs
facility. This way, the entire chain (private key handle, public key, and
certificate) can be imported onto a new client with the YubiKey alone.
Key revocation lists
Key revocation lists (or KRLs) may be used to centralize key and certificate revocation that would otherwise be split across multiple different locations.
Within each KRL we may specify revoked keys and certificates using multiple different formats. For a user key, for example, it is enough knowing its fingerprint hash (as often shown in log output), whilst for user certificates it may be enough specifying just a single number: its serial number. Of course certificates may also be revoked by their identifier or key hashes. If worst comes to worst, the entire CA may also be revoked in one of these, immediately locking out every key that was ever signed by it.
Auditing
Using sshd_config(5)’s
LogLevel directive, OpenSSH
will automatically log relevant actions to the system’s default logs. When using
user keys, these will log their fingerprints…
Accepted key ED25519 SHA256:[..] found at /home/user/.ssh/authorized_keys:1
… and when using user certificates, their identities, serial, and signing CA:
Accepted certificate ID "example_id" (serial 25) signed by ED25519 CA SHA256:[..] via /etc/ssh/authorized_ca_keys
Setup
Now that we know what we want and how the basics work, it’s time to set it all up.
Generating an SSH key on the YubiKey
The following will use
ssh-keygen(1)
generate an ed25519-sk key that resides on the YubiKey’s FIDO backend. The
label given with application= will later be visible in the Yubico
Authenticator and appended to the output of ssh-keygen -K when retrieving the
resident key on other machines.
ssh-keygen -t ed25519-sk -O resident -O verify-required -O application=ssh:login -O user=wolf
For example, we may label the key ssh:login to disambiguate it from a
potential future signing key. OpenSSH also allows us to pass a user for
additional disambiguation. Since the key is bound to the specific YubiKey
anyway, there’s no immediate need to also name the key in the application
label - the user certificates will have clear labels instead.
Additionally, list -O verify-required to require user presence. With this, the
YubiKey’s FIDO backend will first require a PIN, and then a press for each
login.
Generating the CA
The certificate authority can be generated using
yubico-piv-tool(1)
or directly in the Yubico
Authenticator. For the latter,
we navigate to “Certificates”, choose a slot (9c usually contains material
related to digital signatures), then click “Generate key”.
We generate an ed25519 key and self-signed signature with the following
example subject:
CN=Our new SSH CA,O=example.org
The signature’s subject is informational only, and is not used in operations by OpenSSH. The YubiKey will later show it in its certificate overview.
Next, we export the CA’s public key and convert it to a format that openssh
understands:
$ ykman piv keys export 9c ca-pub.pem
$ ssh-keygen -i -m PKCS8 -f ca-pub.pem > ca.pub
The ca.pub file may now be used for signing with
ssh-keygen(1).
Signing user and host keys
Signing is done using ssh-keygen(1) and
the YKCS11 backend
to access the certificate authority stored in the PIV module. Note that once (or
ideally before) a certificate expires, the following processes will have to be
repeated.
Signing a host key
To sign a host key, we retrieve the relevant host key from /etc/ssh/ either
manually or by using ssh-keyscan(1) and
invoke ssh-keygen(1) as follows:
ssh-keygen -D /usr/lib64/libykcs11.so -s ca.pub -h -I identity -n host.example.org -V +10w ssh_host_ed25519_key.pub
The principals listed here with -n will restrict which hostnames are
considered valid by the certificate. The identity is used to identify the
certificate.
Next, we transfer the generated certificate back to the host’s /etc/ssh/
directory and make sure that
/etc/ssh/sshd_config contains the
HostCertificate option
pointing to that certificate. Sadly sshd(8)
does not load any certificates by default.
Signing a user key
To sign a user key, we retrieve the relevant key from the YubiKey with
ssh-keygen -K and invoke ssh-keygen(1)
as follows:
ssh-keygen -D /usr/lib64/libykcs11.so -s ca.pub -I user -n user,toor -V +10w -O verify-required id_ed25519_sk.pub
The principals listed here will restrict which user accounts are available for the certificate. The identity is used to identify the certificate and will be logged by the system for each login.
Then, we store the generated certificate on the YubiKey’s
largeBlobs
facility using
fido2-token(1).
The name given by the -n flag is purely informational and merely identifies
this specific large blob. It needs to be given again when retrieving the
certificate later:
fido2-token -S -b -n ssh:login id_ed25519_sk-cert.pub /dev/hidraw7
Trusting the CA
Systems need to be made aware of the CA in order to trust certificates issued by it. There’s a couple of different ways to do this, depending on whether you have root access or not. If you have root access, you may trust the CA globally, otherwise you can trust it on a per-user basis.
Trusting user certificates signed by the CA
To have a host trust the CA globally for user certificates, we place the
contents of ca.pub into the file referenced by
TrustedUserCAKeys in
the host’s global sshd_config(5).
Without further configuration, this will allow authentication as any user listed
in the user certificate’s principals list. To have more fine-grained control,
use the
AuthorizedPrincipalsFile
directive.
Importantly, note that certificates that lack a principals list will not be
permitted for authentication when using TrustedUserCAKeys.
To trust issued user certificate only for a specific user, we place the contents
of ca.pub into the user’s
~/.ssh/authorized_keys
file along with the cert-authority marker, a principal definition, and an
optional identifier:
cert-authority,principals="user" ssh-ed25519 [..] Our new SSH CA
In this case, the principals= definition requires a presented certificate to
contain at least one of the listed principals in order for it to be accepted.
Trusting host certificates signed by the CA
To have a host trust the CA globally for host certificates, we place the
contents of ca.pub into the file referenced by
GlobalKnownHostsFile
in the host’s global ssh_config(5)
along with the @cert-authority marker, a principal definition, and an optional
identifier:
@cert-authority * ssh-ed25519 [..] Our new SSH CA
In this case, the principals definition (a wildcard * above) determines which
hostnames the certificate authority is accepted for. The above definition trusts
the authority for all possible hostnames. For more information on the format of
known_hosts, refer to
sshd(8).
To trust issued host certificates only for a specific user, we place the above
into the
~/.ssh/known_hosts
file instead.
Revocation
If a certificate authority’s private key has been lost, stolen, or otherwise
compromised, it is possible to revoke it globally using KRLs or locally through
the known_hosts
and
authorized_keys
file. For a centralized system where you have root access to the relevant hosts,
we would generally recommend KRLs.
Revoking trust for user certificates
To revoke keys or user certificates for user authentication globally, we point
the RevokedKeys option in
the host’s global sshd_config(5) to
a file containing a KRL. Note that the file needs to exist, otherwise OpenSSH
will deny all access. Therefore, we quickly generate an empty KRL with
ssh-keygen(1):
cd /etc/ssh/
ssh-keygen -kf revoked_keys
The only way to “revoke” keys or certificates locally is to remove the offending
item from
~/.ssh/authorized_keys.
Revoking trust for host certificates
To revoke host certificates globally, we point the
RevokedHostKeys option
in the host’s global ssh_config(5) to a
file containing a KRL. As before, the file needs to exist.
Locally, in the user’s
~/.ssh/known_hosts
file, we simply remove the compromised certificate, or replace the
@cert-authority marker with @revoked.
Usage
Finally, on a client that wishes to use SSH via a physical key, first import the
resident keys using ssh-keygen -K:
cd ~/.ssh
ssh-keygen -K
ssh-keygen(1) will use the stored label
and user (if applicable) to name the imported keys. For example, with a label of
ssh:login and a user user, the key will be named
id_ed25519_sk_rk_login_user. Of course one may choose to keep this file, but
since ssh(1) will only use resident keys named
id_ed25518_sk by default, it might make sense to rename it.
Next, retrieve the certificate using
fido2-token(1)
and save it to a given file:
fido2-token -G -b -n ssh:login id_ed25519_sk-cert.pub /dev/hidraw7
As before, the file name matters: by default,
ssh(1) will read a key’s certificate from a
file named -cert.pub.
With these files in place under ~/.ssh, we are all set. If we use the default
file names, no further configuration in ~/.ssh/config is needed.
A note on distribution
With this setup, especially if you choose to trust the CA globally, usual key distribution and trust is simplified:
- If we introduce a new YubiKey, we only need to sign the certificate and put it on the YubiKey. All hosts will allow the login automatically.
- If we introduce a new host, only its host key needs to be signed and the certificate uploaded to it. All other hosts will then be able to securely connect without any other changes.
- In most cases, the local
~/.ssh/authorized_keysfile can be left empty or removed entirely. - The local
~/.ssh/known_hostsfile will only contain hosts external to the CA.
However, using certificates does introduce a small overhead, too:
- If a certificate expires, we need to reissue and redistribute it. In the case of user certificates, we need to upload it to the YubiKey. In the case of host certificates, we need to upload it to the host.
- If the CA changes, we need to reissue all certificates and trust the new CA on all machines.
- Unless certificates are valid forever, we need to keep track of expiry not to lock ourselves out.
Also, KRLs need to be distributed: if a certificate or key is compromised, we
need to generate the relevant host or user KRL, and upload it to all hosts. With
the global setup mentioned above, however, there are only two files to keep
track of, which may be managed centrally on one host, and then copied to each
other host via scp(1).
Alternatively, one may elect to generate the KRLs, sign them with
signify(1) and put them in a location
accessible to all hosts. The hosts themselves may then periodically download,
verify, and replace the current KRLs.