Using disko to generate raspberry pi sd card images

[skaphi]

Is it possible to generate SD card images and deploy to a Raspberry Pi using disko to configure the sd image partitions? When I tried, it got errors because the SD card image generator (nixos/modules/installer/sd-card/sd-image-aarch64-new-kernel-no-zfs-installer.nix) tries to use ext4 as root file system, and therefore it does not want to compile. I think this would be really nice to use the same disko configuration format for generating SD images as you use for installing x86 systems.

The main problem I want to solve is to be able to use the btrfs file system to use impermanence, while also being able to generate the sd-card image from scratch.

[Mic92]

I think with our disko's make-disk-image it should be possible. The main part that needs to be replicated is installing firmware/bootloader, as you can see here: https://github.com/NixOS/nixpkgs/blob/eb9dbcd4e116ecb69dde3040102f6a7bfa8b1aab/nixos/modules/installer/sd-card/sd-image-raspberrypi.nix#L18 or here: https://github.com/NixOS/nixpkgs/blob/eb9dbcd4e116ecb69dde3040102f6a7bfa8b1aab/nixos/modules/installer/sd-card/sd-image-aarch64.nix#L22

If you don't have an arm builder at your disposal it should be also be possible to use boot.binfmt.emulatedSystems on a x86_64 machine to run the image generation.
I believe the default boot process for rpis is still to have a fat32 partition?
The alternative would be something like towboot, which might give you more flexibility.

[Nebucatnetzer]

@MatthewCroughan maybe you could post your solution here as you mentioned today in Rapperswil that you have a working setup for a Raspberry Pi?

[MatthewCroughan]

SD card images with Disko are possible, but not great unless you have an aarch64-linux machine to build the image on, due to double-emulation by default. If you're expecting to build an image for an aarch64 machine, from an x86_64 laptop using binfmt, you will end up emulating qemu itself, since the qemu that is used will be built for aarch64, since that is inherited from the nixosConfiguration that you're trying to build for. That can be solved with the recommendations here #698.

Another reason why disko isn't great for this, is the default requirement of native-emulation. It is quite difficult to wire things through, such that Disko will spawn an image-building VM that is itself a cross-compiled system. This makes building for riscv64 difficult, for example, as only cross-compilation is easy at the time of writing for riscv64. It requires a lot of manual and intensive overriding of values inside of disko's modules, and I'm not sure what the general solution could/would be yet. I have to think about it more.

On the other hand, systemd-repart works great, though its limitations are that you have to post/pre-process the image by overrideAttrs, since systemd-repart doesn't support MBR, or adjusting GPT table length, which is needed in order to create partitions that begin <1024 sectors, which is where most SoCs want u-boot to be flashed to, whereas disko gives you more control and flexibility here via preCreate/postCreate hooks, etc. systemd-repart also doesn't let you play with multi-disk FS'

In addition to all of that, disko randomly runs out of memory, and fails when building disk images, and I've never been able to figure out why. If you try again and again, eventually the derivation will succeed and build normally, though it's something that has always baffled me. This issue is documented here #769

If you're on an x86_64 host, and using binfmt to emulate aarch64, and want bcachefs on root, then the below disko config will work for you, though it will break if you're using aarch64 natively everywhere due to the unresolved architectural issues surrounding system architecture that I'm alluding to. Note the manual setting of the disko.imageBuilder.qemu option.

{ pkgs, lib, config, ... }:
let
  configTxt = pkgs.writeText "config.txt" ''
    [pi4]
    kernel=u-boot-rpi4.bin
    enable_gic=1

    # Otherwise the resolution will be weird in most cases, compared to
    # what the pi3 firmware does by default.
    disable_overscan=1

    # Supported in newer board revisions
    arm_boost=1

    [cm4]
    # Enable host mode on the 2711 built-in XHCI USB controller.
    # This line should be removed if the legacy DWC2 controller is required
    # (e.g. for USB device mode) or if USB support is not required.
    otg_mode=1

    [all]
    # Boot in 64-bit mode.
    arm_64bit=1

    # U-Boot needs this to work, regardless of whether UART is actually used or not.
    # Look in arch/arm/mach-bcm283x/Kconfig in the U-Boot tree to see if this is still
    # a requirement in the future.
    enable_uart=1

    # Prevent the firmware from smashing the framebuffer setup done by the mainline kernel
    # when attempting to show low-voltage or overtemperature warnings.
    avoid_warnings=1
  '';
in
{
  boot.postBootCommands = ''
    # On the first boot, resize the disk
    if [ -f /disko-first-boot ]; then
      set -euo pipefail
      set -x
      # Figure out device names for the boot device and root filesystem.
      rootPart=$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE /)
      bootDevice=$(lsblk -npo PKNAME $rootPart)
      partNum=$(lsblk -npo MAJ:MIN $rootPart | ${pkgs.gawk}/bin/awk -F: '{print $2}')

      # Resize the root partition and the filesystem to fit the disk
      echo ",+," | sfdisk -N$partNum --no-reread $bootDevice
      ${pkgs.parted}/bin/partprobe
      ${pkgs.bcachefs-tools}/bin/bcachefs device resize $rootPart

      # Prevents this from running on later boots.
      rm -f /disko-first-boot
    fi
  '';
  disko = {
    memSize = 6144;
    imageBuilder.qemu = (import pkgs.path { system = "x86_64-linux"; }).qemu + "/bin/qemu-system-aarch64 -M virt -cpu cortex-a57";
    imageBuilder.kernelPackages = pkgs.linuxPackages_latest;
    imageBuilder.extraPostVM = ''
      ${pkgs.zstd}/bin/zstd --compress $out/*raw
      rm $out/*raw
    '';
    devices = {
      disk = {
        disk1 = {
          imageSize = "20G";
          type = "disk";
          device = "/dev/mmcblk0";
          postCreateHook = ''
            lsblk
            sgdisk -A 1:set:2 /dev/vda
          '';
          content = {
            type = "gpt";
            partitions = {
              firmware = {
                size = "30M";
                priority = 1;
                type = "0700";
                content = {
                  type = "filesystem";
                  format = "vfat";
                  mountpoint = "/firmware";
                  postMountHook = toString (pkgs.writeScript "postMountHook.sh" ''
                    (cd ${pkgs.raspberrypifw}/share/raspberrypi/boot && cp bootcode.bin fixup*.dat start*.elf *.dtb /mnt/firmware/)
                    cp ${pkgs.ubootRaspberryPi4_64bit}/u-boot.bin /mnt/firmware/u-boot-rpi4.bin
                    cp ${configTxt} /mnt/firmware/config.txt
                  '');
                };
              };
              boot = {
                size = "1G";
                type = "EF00";
                content = {
                  type = "filesystem";
                  format = "vfat";
                  mountpoint = "/boot";
                };
              };
              root = {
                name = "root";
                size = "100%";
                content = {
                  type = "filesystem";
                  extraArgs = [ "--compression=zstd" ];
                  format = "bcachefs";
                  mountpoint = "/";
                  postMountHook = toString (pkgs.writeScript "postMountHook.sh" ''
                    touch /mnt/disko-first-boot
                  '');
                };
              };
            };
          };
        };
      };
    };
  };
}

[MatthewCroughan]

Oh, and another problem with disko is that you have to set the imageSize to an arbitrary incorrect value since there is no way of automatically setting it based on the resulting image size. I attempted to solve that here, but it's not good enough, as there are overheads that are unaccounted for by nix path-info #465

[Mic92]

Can we start disko vms in the host architecture and just call the bootloader installation with emulation (binfmt)?

[MatthewCroughan]

@Mic92 I suggested this a while ago

https://app.element.io/#/room/#disko:nixos.org/$--jOkPviGJQ3OsbXf1EMDOEdm0aNkjjdz220ZNbqJGs

instead, vmTools should execute a VM for the hostPlatform, with binfmt enabled, and run all the scripts for arm64, But vmTools is so minimal that putting binfmt in there wouldn't re-use the nixpkgs machinery (boot.binfmt.emulatedSystems)

So if you want to do it, I think it's a great idea, I just don't know how to hook binfmt up from scratch in a minimal VM like that. boot.binfmt.emulatedSystems is magic to me :D

[saviosg]

Builds using the custom QEMU setting would take hours on my machine, so I added binfmt support in #943, and now it takes only a few minutes.

[yajo]

Did any of you finally make it to build the SD image? How? I'm struggling with that now... 😅

[MatthewCroughan]

@yajo I'm using systemd-repart to do image building, and avoiding disko for this use case nowadays. I'm using disko for interactive machines.

[yajo]

Got any code samples? Do you completely disable disko in those machines or reuse its definitions for systemd-repart? Thanks.