Installing NixOS on Kimsufi machines

I’ve been using a few EC2 and DigitalOcean machines for various small projects recently, but I was getting a little bored of the cost of all the little machines I’d ended up setting up for different purposes. I decided to consolidate them all into one hardware machine upon which I am going to run some containers (probably LXD, which I anticipate will be a future blog post). I opted to get a Kimsufi machine to see if I can persuade it to run NixOS. That way, the things I do which are Nix-native can live in NixOS containers, and all the other things that live on Ubuntu or similar can run inside LXD. There’s the theory. As it’s an experiment, I got one of the cheaper machines with a single disk to start with.

Installing a deeply non-standard OS on a machine to which you have no console access is fun. I complicated my life further by deciding to use ZFS to learn more about that too. A couple of hours of research led me to two separate web pages that I decided to collide together:

Oops

The major mistake I made through this process was assuming that the Kimsufi box would be happy enough booting using UEFI. Not so! I did all the setup and installed everything, and upon rebooting, my machine fell off the planet. It was unhappy to the point where it wouldn’t boot the rescue system any more and support had to intervene. Redoing it all assuming BIOS worked.

How-to

Set your machine to netboot into the rescue system using the Kimsufi control panel, and reboot it. You’ll get a password for the root user via email. The rescue system starts the machine up using a Linux distro provided over a read-only NFS, and a home directory which is in a tmpfs in memory. We’ll use this space to install the Nix tools later on. We’re logged in as root which gives us a fair amount of flexibility. The machine’s own hard disk is not mounted by default–we’ll do this after partitioning.

The first steps are to log in via SSH, and flatten the disk, then set up the partition layout.

Partitioning and formatting

This layout is mostly inspired by the first post I mentioned above. It’s a 2TB disk and the machine has 8GB RAM, so I created a 512MB partition for /boot, then most of the disk is taken up by the ZFS partition, and the final 8GB is swap. I’m told it is theoretically possible to have /boot inside the ZFS pool but it can be hard to make it work reliably with grub, so I’m going to keep that one as ext4.

I’m going to present all these commands as if it were a script, but I did it all interactively.

wipefs -a /dev/sda

# This sets a variable called DISK to save some typing later
# It should be set to the file starting with ata in the directory
export DISK=/dev/disk/by-id/ata*

parted /dev/sda -- mklabel msdos
parted /dev/sda -- mkpart primary 1MiB 512MiB # /dev/sda1 is /boot
parted /dev/sda -- mkpart primary 512MiB -8GiB # This is the ZFS partition
parted /dev/sda -- mkpart primary linux-swap -8GiB 100% # Swap

mkfs.ext4 -L boot $DISK-part1 # Format /boot

# Create the zpool
# There are lots of options here: see the wiki page above

zpool create -f -O mountpoint=none rpool $DISK-part2

# Create some datasets inside the pool
# I'm going to add another later for LXD but this is enough for now

zfs create -o mountpoint=legacy rpool/root
zfs create -o mountpoint=legacy rpool/root/nixos
zfs create -o mountpoint=legacy rpool/home

mount -t zfs rpool/root/nixos /mnt
mkdir /mnt/home
mount -t zfs rpool/home /mnt/home

mkdir /mnt/boot
mount $DISK-part1 /mnt/boot

Now the partition layout is complete. We can move on to the NixOS parts.

Getting Nix

There are some prerequisites needed before we can use Nix. These are largely from the first post above again.

# Create a user for the Nix daemon
groupadd -g 30000 nixbld
useradd -u 30000 -g nixbld -G nixbld nixbld

useradd -m setupuser

# Set this to something you will remember, you'll need it in a moment
passwd setupuser 

sed s/root/setupuser/ /etc/sudoers -i
su setupuser -s /bin/bash

 # Install Nix
 # -L is required because the script is now a redirect
curl -L https://nixos.org/nix/install | sh

# This next line makes the Nix tools available in our shell
source $HOME/.nix-profile/etc/profile.d/nix.sh

nix-channel --add https://nixos.org/channels/nixos-21.05 nixpkgs
nix-channel --update

We need to take a minor detour here as nixos-generate-config is not installed by default.

nix-shell -p nixos-generate-config
sudo `which nixos-generate-config` --root /mnt

This will place the usual configuration in /etc/nixos/configuration.nix and /etc/nixos/hardware-configuration.nix.

hardware-configuration.nix

I didn’t change very much in here, I only added these two lines:

boot.initrd.supportedFilesystems = [ "zfs" ];
boot.supportedFilesystems = [ "zfs" ];

configuration.nix

I changed more in here. I’m not going to paste the entire config in here, but I will call out the key parts that were needed for this to work.


# This machine is very insistent on BIOS boot, so configure that
boot.loader.grub.enable = true;
boot.loader.grub.version = 2;
boot.loader.grub.device = "/dev/sda"; # or "nodev" for efi only

networking.hostName = "metis";

# Make a random 8-character hex string for this. ZFS requires it.
networking.hostId = "13371337";

# Check what the rescue system's IP, route and optionally nameservers are
# (or just use 8.8.8.8 for DNS if you like)

networking.usePredictableInterfaceNames = false;
networking.interfaces.eth0.ipv4.addresses = [{
    address = "192.0.2.1";
    prefixLength = 24;
}];

networking.defaultGateway = "192.0.2.254";
networking.nameservers = [ "8.8.8.8" ];

services.openssh.enable = true;

# You can enable this once you've proved the machine boots
networking.firewall.enable = false;

users.users.michael = {
    # Usual user config goes here. Be sure to include your SSH public key
}

Installing NixOS

I again had to take a detour here, because something is a little broken in NixOS 21.05 at the moment, resulting in errors like this:

getting attributes of path /nix/store/dbri2d4r470fc6nrh95qa8bwcj54wh1q-zfs-kernel-2.0.5-5.10.52: No such file or directory

I have no idea what causes this, and perhaps it’s fixed by the time you read this, but it is documented in this GitHub issue. The workaround described at that comment allowed me to continue, with this fairly contorted pair of commands:

sudo PATH="$PATH" NIX_PATH="$NIX_PATH" `which nix-build` '<nixpkgs/nixos>' -A config.system.build.toplevel -I nixos-config=/mnt/etc/nixos/configuration.nix
sudo PATH="$PATH" NIX_PATH="$NIX_PATH" `which nixos-install` --root /mnt

If all goes well, you should be asked to set a root password. You may encounter quite a lot of errors about locales being missing–I ignored these as they didn’t seem to matter.

At this point, you should have a NixOS system ready to go on the machine’s internal disk. You’ll need to use the Kimsufi control panel to set the machine back to boot from its own hard disk, using the menu option marked Netboot.

Now, all you can do is reboot and cross your fingers. It worked for me, although it’s worth noting that it took a good couple of minutes for the machine to come up and respond to SSH. You can connect as the user you set up in configuration.nix.

You will probably encounter an error when you connect because the SSH host key will have changed. You can fix this with ssh-keygen -R 192.0.2.1 (replacing the IP with the one for your machine), and then try again. Hopefully you’ve now got a new NixOS machine up and running and ready to do whatever you need it to do.

Thanks

Big thanks to Graham Christensen of Nix specialists Determinate Systems for helping me out throughout this process.