Deploying Full Disk Encryption in the Cloud with Ansible

Deploying a new virtual machine 'in the cloud' is simple enough and fast. As long as one doesn't need to setup an encrypted root filesystem. This article shows how to automate the setup of a Fedora system with an encrypted root filesystem remotely - 'in the cloud' - on e.g. a DigitalOcean, Hetzner, Linode, etc. VM - with Ansible and a special Fedora rescue system that runs purely from an initramfs archive.

Motivation

Having an encrypted root filesystem gives you data at rest protection. At the end of the day, your virtual disk device inside your virtual machine (VM) is just a file on a storage solution where disks are getting replaced and reallocated for other things. After - say - hardware replacements perhaps such a disk even ends up on eBay without being properly deleted and thus some random person now has access to your data. With your root filesystem being encrypted you are protected against such scenarios.

If you think that such a scenario is far fetched, see also for example this 2019 computer magazine article (in German) which describes in detail how some SSDs with very sensitive personal data ended up on eBay.

The Problem

Cloud providers such as DigitalOcean, Linode or Hetzner provide a set of pre-built disk images for different operating systems. Users also have the option to add custom images to their account, usually. When deploying a new virtual machine (e.g. a Droplet when using DigitalOcean) the selected disk image is extracted to the new VM guest, the image has to run some first-boot initialization script which fetches the network configuration, creates sshd host keys and stuff like that.

This process is very fast because the image is pre-built and just needs to be copied while the first-boot initialization doesn't have much to do. Compare this with a traditional automated Linux install method such as Kickstart where RPMs are installed one after the other. This takes much longer.

The disadvantage is a loss in flexibility. Especially, this makes setting up the root filesystem on a LUKS encrypted device challenging.

Non-Solutions

A naive approach to a remote system with an encrypted root filesystem is to locally install a Linux distribution with disk encryption enabled and upload the resulting image. The problem with this is that if you want to deploy multiple VMs all then have the same LUKS encryption key. Certainly you can't openly share this image with others and you would upload a new image for each new VM deployment. Obviously this is tedious and doesn't scale.

After the VM is deployed with an unencrypted root filesystem, encrypting it on the fly isn't possible, because the root filesystem is in use.

A Solution

For some time Linux has the interesting feature to boot a new kernel image from a running Linux system. That means the currently running kernel can be replaced with a new one using the kexec system call.

A standard part of the Linux boot process is to extract an early boot root filesystem image into a ram disk, execute an init process from it that basically mounts the real root filesystem and switches to it. Of course, the final root filesystem switch is optional.

In combination these features, i.e. kexec and an early-boot environment, can be used to bootstrap a fresh Linux installation from scratch - from a running system.

The basic idea here is to kexec a new kernel with an initramfs image that contains a small Linux system, i.e. one that includes enough software to provide network connectivity including SSH and to drive the Fedora install process.

Recently, I implemented this approach, i.e. I thus wrote a script that creates such initramfs images. That means it installs a minimal Fedora system into an initramfs.

I also created an Ansible playbook that drives the whole process. That means it downloads the initramfs image to the VM, creates VM specific configuration, kexecs into the rescue system and finally installs the LUKS-encrypted target system from there. It even supports creating a DigitalOcean Droplet, first.

Without considering the VM creation, the playbook consists of 30 steps or so. After the playbook finishes, the target VM has an encrypted root filesystem with a fresh Fedora system on top of it.

So far, I successfully used this method and playbook on DigitalOcean, Linode and Hetzner VMs and even on some old-school bare-metal root-server (rented from Manitu).