RC

Homelab

By Richard Crowley

I bought a bag of Raspberry Pi boards when the Raspberry Pi 5 came out and I’m itching to do something with them. In lieu of knowing what something is, let’s PXEboot them.

I’ve got a Raspberry Pi Zero 2 W board for my PXEboot server, three Raspberry Pi 5 boards for my homelab servers, a clicky-clacky keyboard, a Radio Shack worth of cables, dongles, and power supplies, and an existing Ubiquiti network to coopt to the cause.

Raspberry Pi OS Lite on a microSD card

We have to setup one Raspberry Pi board manually in order to setup all future Raspberry Pi boards automatically. Start on a computer with a microSD card slot:

  1. Download the latest Raspberry Pi OS Lite image
  2. unxz 2024-10-22-raspios-bookworm-arm64-lite.img.xz
  3. Insert a sacrificial microSD card
  4. sudo dd if=2024-10-22-raspios-bookworm-arm64-lite.img.xz of=/dev/mmcblk0 status=progress && sync (/dev/mmcblk0 being the top-level device as seen in lsblk, not any partition on it)
  5. Remove the microSD card

Configure all the homelab servers to network boot

Very annoyingly, before you can PXEboot a Raspberry Pi, you have to boot it with another microSD card (like the one you just made!) or USB drive to change its BOOT_ORDER configuration. They come out of the bag with BOOT_ORDER=0xf461 which tries the microSD card (0x1) first, then NVMe drives (0x6), then the USB drive (0x4), and finally repeats (0xf). Yes, you read it in reverse. (See the documentation for more.)

Reconfigure every Raspberry Pi 5 homelab server thus:

  1. Insert the microSD card with Raspberry Pi OS Lite on it
  2. Boot it up
  3. Run sudo -E rpi-eeprom-config --edit to change BOOT_ORDER to 0xf241 (microSD, USB, PXE, repeat). (If you’re going to install an M.2 hat or something then you’ll want to leave 0x6 in the list for your NVMe drive.)

You don’t need to tell them anything more as the DHCP server changes above will guide them to the PXEboot server.

If you’re on the pedantic side, repeat the section above to get a truly pristine Raspberry Pi OS Lite microSD card before moving on.

PXEboot server: Raspberry Pi Zero 2W

Now insert the microSD card into the Raspberry Pi that’s to become the PXEboot server. In my case, it’s a Raspberry Pi Zero 2 W. Boot it up with a keyboard and monitor plugged in. It’s going to ask you some very Eurocentric questions about your keyboard layout and have you create a non-root user before dropping you to a Linux login prompt where you can login as said non-root user.

If you’re on a Raspberry Pi Zero 2 W like me, you must sudo raspi-config to configure WiFi before going any further. If you’re using a heartier Raspberry Pi, you might be connected via Ethernet and ifconfig will confirm you’re already online. (The WiFi configuration is under System Options, Wireless LAN, by the way.)

Now it’s time to configure the PXEboot server. You could make this a bigger project by making this your DHCP server, too, but I’m not.

Set the PXEboot server’s hostname:

  1. sudo hostname pxeboot
  2. hostname | sudo tee /etc/hostname
  3. hostname | sudo xargs -I_ sed -i s/raspberrypi/_/ /etc/hosts

Enable the SSH server, which you could use to make the rest of this guide a lot more copy-and-paste:

  1. sudo touch /boot/firmware/ssh
  2. sudo systemctl start sshswitch

Install the software we need: sudo apt-get update && sudo apt-get -y install dnsmasq nfs-kernel-server

Create /etc/dnsmasq.d/pxeboot.conf as follows:

enable-tftp
pxe-service=0,"Raspberry Pi"
tftp-root=/tftpboot

Now we need to populate /tftpboot and the root filesystem it’s going to reference. For this, we’ll need the Raspberry Pi OS Lite image again, untouched by booting this Raspberry Pi.

Download the image and create the intermediate directory we’ll need:

  1. sudo curl -o /var/raspios.img.xz https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-10-28/2024-10-22-raspios-bookworm-arm64-lite.img.xz
  2. sudo unxz -v /var/raspios.img.xz
  3. sudo mkdir -v /mnt/raspios

Mount the first partition from the image and copy its contents to /tftpboot:

  1. sudo mount -o loop,offset=$((512 * $(fdisk -l -o Device,Start /var/raspios.img | awk '/img1/ {print $2}'))) -v /var/raspios.img /mnt/raspios
  2. sudo cp -a -v /mnt/raspios /tftpboot
  3. sudo umount -v /mnt/raspios

Add a sentinel file to make SSH available during PXEboot: sudo touch /tftpboot/ssh

Mount the second partition from the image and copy its contents to /var/raspios:

  1. sudo mount -o loop,offset=$((512 * $(fdisk -l -o Device,Start /var/raspios.img | awk '/img2/ {print $2}'))) -v /var/raspios.img /mnt/raspios
  2. sudo cp -a -v /mnt/raspios /var/raspios
  3. sudo umount -v /mnt/raspios

Propagate the keyboard layout and user account from the PXEboot server to every homelab server:

  1. sudo cp -v /etc/default/keyboard /var/raspios/etc/default/keyboard
  2. sudo cp -v /etc/{group,gshadow,passwd,shadow}{,-} /var/raspios/etc
  3. sudo mv -v /var/raspios/home/pi /var/raspios/home/$USER
  4. sudo rm -v /var/raspios/etc/systemd/system/multi-user.target.wants/userconfig.service

Tidy up the PXEboot process for every homelab server:

  1. sudo rm -v /var/raspios/etc/init.d/resize2fs_once
  2. sudo ln -f -s -v /lib/systemd/system/getty@.service /var/raspios/etc/systemd/system/getty.target.wants/getty@tty1.service
  3. sudo sed -i s/^pi/%sudo/ /var/raspios/etc/sudoers.d/010_pi-nopasswd

Move the image itself into the soon-to-be root filesystem so we can install it onto the microSD cards in Raspberry Pis that PXEboot from here: sudo mv -v /var/raspios.img /var/raspios/var

Arrange for that image to be installed onto the microSD card when a homelab server PXEboots. If that image boots as-is, it’s going to start all the way back with asking you about keyboard layouts, so this program also includes all the homelab server setup we can dream up.

/var/raspios/etc/systemd/system/homelab.service:

[Install]
WantedBy=multi-user.target

[Service]
ExecStart=/usr/local/sbin/homelab
Type=simple

[Unit]
After=network-online.target
Description=Homelab server setup

/var/raspios/usr/local/sbin/homelab:

#!/bin/sh

set -e -x

dd if=/var/raspios.img of=/dev/mmcblk0
mkdir -p -v /mnt/raspios

mount -v /dev/mmcblk0p1 /mnt/raspios
touch /mnt/raspios/ssh
umount -v /mnt/raspios

mount -v /dev/mmcblk0p2 /mnt/raspios
mount --bind -v /dev /mnt/raspios/dev
mount --bind -v /proc /mnt/raspios/proc
mount --bind -v /sys /mnt/raspios/sys
cp -v /etc/default/keyboard /mnt/raspios/etc/default/keyboard
for FILENAME in group gshadow passwd shadow
do cp -v /etc/$FILENAME /etc/$FILENAME- /mnt/raspios/etc
done
mv -v /mnt/raspios/home/pi /mnt/raspios/home/$(id -n -u 1000)
rm -f -v /mnt/raspios/etc/systemd/system/multi-user.target.wants/userconfig.service
mkdir -p -v /mnt/raspios/etc/systemd/system/getty.target.wants
ln -f -s -v /lib/systemd/system/getty@.service /mnt/raspios/etc/systemd/system/getty.target.wants/getty@tty1.service
sed -i s/^pi/%sudo/ /mnt/raspios/etc/sudoers.d/010_pi-nopasswd

# TODO: Custom setup goes here. Anything beyond simple filesystem manipulation
# must be prefixed with `chroot /mnt/raspios` to run in the target filesystem.

umount -v /mnt/raspios/sys
umount -v /mnt/raspios/proc
umount -v /mnt/raspios/dev
umount -v /mnt/raspios

reboot now

Don’t forget to make the setup program executable: chmod +x /var/raspios/usr/local/sbin/homelab

This root filesystem is going to be served via NFS, which requires /tftpboot/cmdline.txt to read as follows (using the PXEboot Raspberry Pi’s IPv4 address per ifconfig):

console=serial0,115200 console=tty1 ip=dhcp nfsroot=192.168.1.81:/var/raspios,vers=3 root=/dev/nfs rootwait rw

/tftpboot on the PXEboot server becomes /boot/firmware on homelab servers via NFS. Replace the whole contents of /var/raspios/etc/fstab with this:

proc /proc proc defaults 0 0
192.168.1.81:/tftpboot /boot/firmware nfs defaults,vers=3 0 0

Speaking of NFS, we need to serve all of this up via NFS. Edit /etc/exports to add the following:

/tftpboot *(rw,sync,no_subtree_check,no_root_squash)
/var/raspios *(rw,sync,no_subtree_check,no_root_squash)

Restart running services to pick up all of these configuration changes: sudo systemctl restart dnsmasq nfs-kernel-server rpcbind

PXEboot is open for business!

Network

There are many of ways to serve DHCP option 66 but this one is mine. I already had Ubiquiti gear so it was natural to keep using it.

You need to create a new VLAN to use for PXEboot because you don’t want just any old device that joins your network to potentially PXEboot. That’s nearly certain to break something eventually. In your Ubiquiti console under Settings / Networks:

  1. Click New Virtual Network
  2. Give it a name, etc. and leave it in IPv4 mode
  3. Click Manual
  4. Next to DHCP Service Management, click Show Options
  5. Select Network Boot and TFTP Server, giving the IP address of your PXEboot server for both
  6. Click Add

I configured VLAN 2 to direct network boot and TFTP to 192.168.1.81, the DHCP-assigned IPv4 address of the PXEboot server I just setup. I set the filename field to helloworld and have no idea what it’s for, though I see it printed into the console when booting homelab servers.

The ports you’re going to plug your homelab servers into need to be assigned to the VLAN you just created because by the time the DHCP server is communicating with an Ethernet device on the network, it’s too late to make the VLAN decision. In the Ubiquiti console, click Ports and assign the appropriate ports.

Homelab servers: Raspberry Pi 5

Plug each homelab server Raspberry Pi into one of the Ethernet ports you assigned to the PXEboot VLAN, insert a non-bootable microSD card, and power it up.

Each one will come up with Raspberry Pi OS Lite, your keyboard layout and user account, and SSH running. The filesystem on the microSD cards will survive reboots.

If you do want one to PXEboot from scratch again, simply make the microSD card unbootable and reboot the Raspberry Pi.