Now that I have my NixOS Pi 4 up and running and doing a couple of useful things, I thought I’d see what else I could do with it. So far it’s just running DHCP, DNS and Tailscale, so it’s got a bit of spare horsepower. I also wrote a fan control daemon for the Argon ONE case because I had some different requirements from the one that Argon40 themselves ship.

I decided to try out running Firecracker VMs on it, as I was aware Firecracker gained Pi 4 support some time ago and I’d been meaning to give it a go. I’d also recently read about the Ignite tool from Weaveworks that lets you run VMs based on Docker container images without having to repack or rebuild them yourself. I expected that the quoted 125ms VM startup time was unlikely to be met on this hardware, but I figured it probably would still work.

As I’m running NixOS, most of this was set up inside configuration.nix. I added firecracker, ignite, git, and binutils-unwrapped to environment.systemPackages, and also added a line to set virtualisation.containerd.enable to true. Ignite relies on git and the strings command from binutils, which I think should be explicit dependencies in the package, but I’ll come back to that later.

A quick rebuild later, I had all the tools in place to be able to try and run a VM:

sudo ignite run weaveworks/ignite-ubuntu \
    --cpus 1 \
    --memory 512MB \
    --ssh \
    --name hello-world

This didn’t give me a lot of output, but ultimately failed. Rerunning it with --log-level debug gave me something like this:

DEBU[0001] Writing "/var/lib/firecracker/vm/57b9e0bf6cf0228a/runtime.containerd.resolv.conf" with new hash: "fbe74b53a9d2380c8212bed5097146735021280bbc84d9e176139552a999fd25", old hash: ""
FATA[0001] failed to start container for VM "57b9e0bf6cf0228a": failed to mount /tmp/containerd-mount920509976: invalid argument

And digging into the journal gave me:

Oct 26 20:13:12 nixpi kernel: overlayfs: upper fs does not support RENAME_WHITEOUT.
Oct 26 20:13:12 nixpi kernel: overlayfs: upper fs missing required features.

Because I apparently enjoy making obtuse computing problems for myself to solve, I’m running this Pi on ZFS in order to learn more about how to operate it. It seems that ZFS doesn’t support something that overlayfs requires. I mentioned I had this problem on IRC and Graham suggested a zvol. This is not something I’d encountered before, but it’s a way of creating a block device that lives in your ZFS pool, which you can then format with whichever filesystem you like.

It turns out after all I’m not the first person to try and work around this, as this GitHub comment in the Ignite repository proves, and I ended up using a very similar approach to the one described there.

sudo systemctl stop containerd.service
sudo zfs create -s -V 15G rpool/containerd
sudo mkfs.ext4 -b 4096 -L containerd /dev/zvol/rpool/containerd
sudo mkdir /tmp/migrate
sudo mount /dev/zvol/rpool/containerd /tmp/migrate
sudo cp -r /var/lib/containerd/* /tmp/migrate
sudo umount /tmp/migrate
sudo mount /dev/zvol/rpool/containerd /var/lib/containerd
sudo systemctl start containerd.service

Running it again worked fine this time:

I’ll make the new filesystem permanent by adding it into my hardware-configuration.nix using something like the below. It turns out that using rpool/containerd doesn’t work on boot, instead I need to use the device it’s exposed as (/dev/zd0), although a more robust way to do it would be to find its symlink in /dev/disk/by-uuid.

  fileSystems."/var/lib/containerd" =
    { device = "/dev/zvol/rpool/containerd";
      fsType = "ext4";
    };

I’m pretty certain that this would have worked out of the box on a more conventional setup, but by doing it this way at I suppose I learned something. I don’t have any specific plans for this, but I’m going to see if it will handle things like the Unifi controller that I’d normally use Docker for.