Deploying SSH Host Keys with Digital Ocean Droplets

When deploying a VM in a remote location one might end up in an impossible situation: a VM base image doesn't contain SSH host keys as otherwise the image couldn't be shared between multiple hosts. But if SSH host keys are auto-generated during the first boot one doesn't know their fingerprints and thus they can't be trusted on first connect. This article describes how to securely deploy SSH host keys when deploying a Digital Ocean Droplet (i.e. a VM).

Problem Description

Pre-build disk images usually don't contain any SSH host keys. At least they shouldn't contain any host keys if they are used for multiple hosts, but sometimes people get this wrong, e.g. Hetzner (a web hoster) in 2015, Cisco (a network equipment company) also in 2015 and many others.

One common approach is to generate the SSH host keys during first boot.

The problem is then: how can we trust this new machine during the first SSH login? We don't know the fingerprint of any SSH host key because they are newly generated, remotely. Thus, we can't really trust it that way.

One approach is to just accept any first encountered host key and store its fingerprint for later verifications (trust on first use). This is better then ignoring the problem completely but this still makes you vulnerable against man-in-the-middle attacks.

Generating a new disk image with new host keys for each new host and transferring it to a cloud provider doesn't scale.

A Solution for Digital Ocean Droplets

DigitalOcean (like some other cloud providers) uses cloud-init for the first-boot configuration. And cloud-init also supports supplying SSH host keys via the so called cloud-config. Thus, we can supply a cloud-config (with locally generated host keys) via the user-data parameter when creating a new VM via the DigitalOcean API.

SSH host key problem solved. Almost, since it turns out that the user-data cloud-config is served over a link-local address for the lifetime of the VM. Since this turns the SSH host keys into world-readable information (to processes running on that VM) we have to consider these cloud-init SSH host keys as ephemeral and have to replace them with another set (of locally generated SSH host keys) as part of our deployment.

See also an example Ansible playbook of mine that creates a DigitalOcean Droplet and automates the SSH host key deployment in the described way.

In that way, man-in-the-middle attacks can be successfully averted (and detected) during deployment as the DigitalOcean API is called over HTTPS (where the server's certificate is signed by a trusted CA) and the first ephemeral SSH host key is thus transferred over that secure connection. The second and final SSH host key is transferred over SSH where the (ephemeral) host key can be authenticated by it's known fingerprint.