I have been wanting to find a more permanent home for the DNS server I built before which was on a Pi 4 running Raspbian. As is evident by the number of posts I have on here about installing Nix on every computing device I can get my hands on, I wanted to put NixOS on it. Previously I’d not been keen doing that on an SD card because I still have a perception that Pis running Linux can chew them up pretty quickly, although I suppose reinstalling them from a reproducible OS would be less painful were this to happen.

As it happens, last week a vendor sent me an email saying that they had a sale on the Argon ONE m.2 case, which would accept an m.2 SATA SSD. This would give me a bit more space and probably stand up to more use than the SD card would, so I got one.

Requirements

I could quite easily have just dded the SD card image onto the drive and booted that and called it a day, but I wanted to make life difficult for myself really wanted to use ZFS. I also wanted to use the AArch64 architecture instead of the armv7 that I believe Raspbian had been using.

The first thing I did before getting started was boot Raspbian, update it, and then use it to update the firmware to a version that properly supports USB boot, as my Pi 4 was old enough not to have that out of the box.

sudo rpi-eeprom-update # Read the output of this before proceeding
sudo rpi-eeprom-update -d -a # Actually do the update
sudo raspi-config # Go to Advanced Options and enable USB boot

Installation process

I did consider trying to install it from the existing Raspbian OS that was on the SD card already, in a similar fashion to how I did so using the Kimsufi recovery OS, but as I believe it was running a different architecture I wasn’t sure how that would work.

I grabbed the latest NixOS 21.05 new_kernel image I could find on Hydra. This ended up being one built in July–images since then appear to have had some build failures due to something related to ZFS. I didn’t entirely pay attention here and ended up writing the compressed image to the SD card and trying to boot that–this predictably failed, and I re-read the instructions and uncompressed it first before trying again.

Booting this results in it resizing to fit the SD card, and then drops you into a terminal. Normally here you’d probably create a configuration.nix and do nixos-rebuild to make the system you want, but I wanted to install to my SSD so I had a little more work to do.

The procedure I followed was remarkably similar to that I used for the Kimsufi box. The SSD even presented it as /dev/sda, the same as on that machine. I’ll reproduce the commands here, as there are a couple of differences.

Partitioning

wipefs -a /dev/sda

# This sets a variable called DISK to save some typing later
# It should be set to the file starting with ata in the directory
export DISK=/dev/disk/by-id/ata*

parted /dev/sda -- mklabel msdos
parted /dev/sda -- mkpart primary fat32 0MiB 512MiB # /dev/sda1 is /boot
parted /dev/sda -- mkpart primary 512MiB -4GiB # This is the ZFS partition
parted /dev/sda -- mkpart primary linux-swap -4GiB 100% # Swap. This is optional

mkfs.vfat -F32 $DISK-part1 # Format /boot

# Create the zpool
zpool create -f -O mountpoint=none rpool $DISK-part2

# Create some datasets inside the pool
zfs create -o mountpoint=legacy rpool/root
zfs create -o mountpoint=legacy rpool/root/nixos
zfs create -o mountpoint=legacy rpool/home

mount -t zfs rpool/root/nixos /mnt
mkdir /mnt/home
mount -t zfs rpool/home /mnt/home

mkdir /mnt/boot
mount $DISK-part1 /mnt/boot

Nix install

The installation process after this is a little cleaner than the one in the recovery system from previously.

nixos-generate-config --root /mnt

At this point, I can edit the files in /mnt/etc/nixos. However, as the SD image I’m using is from July, it still has the same bug in it that I encountered last time, where the nixos-install process crashes out with errors similar to

getting attributes of path /nix/store/dbri2d4r470fc6nrh95qa8bwcj54wh1q-zfs-kernel-2.0.5-5.10.52: No such file or directory

This has been fixed in a new version of Nix, but it wasn’t incorporated in the build I was using. I again used the workaround from this GitHub issue to allow me to proceed, although as it was a native NixOS machine I didn’t have quite so much to type as before:

nix-build '<nixpkgs/nixos>' -A config.system.build.toplevel -I nixos-config=/mnt/etc/nixos/configuration.nix
sudo nixos-install --root /mnt

Firmware

The Raspberry Pi and/or U-Boot need some firmware files to be present in the first filesystem on the disk. The SD installer image keeps this in a partition at /dev/mmcblk0p1 which is not actually mounted normally. I opted to mount the equivalent of that partition at /boot on my install, so it contains the generated initrd. However, NixOS doesn’t currently provide a neat way of copying these files into place.

In the end, I just did something like:

mkdir /firmware
mount /dev/mmcblk0p1 /firmware
cp /firmware/* /mnt/boot

It took me a bit of effort to try and work out what partition type to use here, and then a little more to work out how to get the firmware on there. It’s not ideal, but you only need to do it once.

Gotchas

There were a couple of things worth noting that I found out as I was doing this.

If the Pi’s firmware couldn’t work out what to do with the USB disk, it would fall back to booting off the SD card. This is fine in theory, but sometimes it would boot all the way past stage 2 and just at the point where you’d expect it to start a console the screen would go black, and then the monitor would switch off. The Pi wasn’t visible on the network at this point either so the only thing I could do was pull the power and give it a few moments before trying again. It normally worked from a cold boot.

It took me a little while to work out what the first partition should look like. I had a few occasions where the machine would complain Unable to read partition as FAT. In the end, I needed to make it FAT32, start right at the beginning of the disk instead of 1MiB in like I had done before, and also flag it as type 0c, which is W95 FAT32 (LBA) according to fdisk. I’m not sure which were the important parts of this, but it works now.

Finally, some NixOS-on-RPi documentation suggests using the boot.tmpOnTmpfs option to put /tmp into a ramdisk. I found that doing that and asking Nix to install a few things at once made it run out of space very quickly and fail the build, so I turned that off. I could imagine that if I were to be installing on an SD card and not doing too much work I would consider enabling it again.

Configuration files

These are here in case they’re useful to anyone.

My hardware-configuration.nix is pretty much the default, but there are some additions to include zfs in the supported filesystems.

hardware-configuration.nix

{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "xhci_pci" "usbhid" "uas" "usb_storage" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];
  boot.initrd.supportedFilesystems = [ "zfs" ];
  boot.supportedFilesystems = [ "zfs" ];

  fileSystems."/" =
    { device = "rpool/root/nixos";
      fsType = "zfs";
    };

  fileSystems."/home" =
    { device = "rpool/home";
      fsType = "zfs";
    };

  fileSystems."/boot" =
    { device = "/dev/sda1";
      fsType = "vfat";
    };


  swapDevices = [ ];

  powerManagement.cpuFreqGovernor = lib.mkDefault "ondemand";
  # high-resolution display
  hardware.video.hidpi.enable = lib.mkDefault true;
}

configuration.nix

As an experiment, I’m running XFCE on this, just to see how it performs. You can comment all of that out.

In this, I added an overlay, which you can see in the imports section. You can do this using the commands:

sudo nix-channel --add https://github.com/NixOS/nixos-hardware/archive/master.tar.gz nixos-hardware
sudo nix-channel --update

However, the only thing it provided for me was the option to do hardware acceleration for X (the option called hardware.raspberry-pi."4".fkms-3d.enable). This didn’t work out for me, and X just gave me a black screen. I intend to dig into this further and hopefully manage to reenable it in the future.

{ config, pkgs, ... }:

{
  imports =
    [
      <nixos-hardware/raspberry-pi/4>
      ./hardware-configuration.nix
    ];

  boot = {
    kernelPackages = pkgs.linuxPackages_rpi4;
    # tmpOnTmpfs = true; # See note
    kernelParams = [
      "8250.nr_uarts=1"
      "console=ttyAMA0,115200"
      "console=tty1"
      "cma=128M"
    ];
  };

  boot.loader.raspberryPi = {
    enable = true;
    version = 4;
  };
  boot.loader.grub.enable = false;
  boot.loader.generic-extlinux-compatible.enable = true;

  hardware.enableRedistributableFirmware = true;
  nixpkgs.config = {
    allowUnfree = true;
  };

  networking.hostName = "nixpi"; 
  networking.hostId = "3bc6efa8"; # For ZFS

  networking.networkmanager.enable = true;

  # Set your time zone.
  time.timeZone = "Europe/London";

  networking.useDHCP = false;
  networking.interfaces.eth0.useDHCP = true;
  networking.interfaces.wlan0.useDHCP = true;

  # Select internationalisation properties.
  i18n.defaultLocale = "en_GB.UTF-8";
  console = {
    font = "Lat2-Terminus16";
    keyMap = "uk";
  };

  services.xserver = {
    enable = true;
    displayManager.lightdm.enable = true;
    desktopManager.xfce.enable = true;
  };

  # Configure keymap in X11
  services.xserver.layout = "gb";
  services.xserver.xkbOptions = "eurosign:e";

  # Enable sound.
  sound.enable = true;
  hardware.pulseaudio.enable = true;

  # Enabling this caused X to have a black screen on boot
  # hardware.raspberry-pi."4".fkms-3d.enable = true;

  users.users.michael = {
     isNormalUser = true;
     extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
  };

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  environment.systemPackages = with pkgs; [
    vim
  ];

  # Enable the OpenSSH daemon.
  services.openssh.enable = true;
  system.stateVersion = "21.05";
}