Installing ubuntu 20.04 with root on encrypted ZFS mirror and UEFI boot
My home machine has been running a pair of disks in a soft raid1 for 7 years now. And then the other day one disk in the mirror finally started to crumble. There was a reason to reinstall the system from scratch and start using the encryption that 7 years ago was not involved. While googling the status of LUKS over mdadm configuration, I came across an article comparing zfs vs mdadm/ext4 performance. I then found another article testing the performance of encrypted disks using LUKS and zfs. According to both articles the performance of zfs is quite good and I decided to give it a try.
I decided to write my own article since I use UEFI for booting (the previous articles used legacy booting) and it has been 3 years since the last one and I thought that a well tested up-to-date tutorial could be useful for the community.
My installation was mostly guided by these articles:
Ubuntu 20.04 Root on ZFS
Installing UEFI ZFS Root on Ubuntu 20.04 with Native Encryption
I will describe the installation in a virtualbox virtual machine. The installation on a real machine is no different.
So I created a virtual machine with a couple of 20gb disks and 8gb of memory. I booted with ubuntu-20.04.1-desktop-amd64.iso, clicked on try ubuntu and started the terminal. In the terminal, I went straight to root, since all the commands used require root privileges. The first thing I did was to define some variables:
export DISK1=/dev/disk/by-id/ata-VBOX_HARDDISK_VBad5107ca-df268eef export DISK2=dev/disk/by-id/ata-VBOX_HARDDISK_VBaf134a71-943e2d11 export HOSTNAME=ubuntu-zfs-vm export USERNAME=toor
Now you can partition the disks:
sgdisk --zap-all $DISK1 sgdisk --zap-all $DISK2 sgdisk -n1:1M:+256M -t1:EF00 -c1:EFI $DISK1 sgdisk -n1:1M:+256M -t1:EF00 -c1:EFI $DISK2 sgdisk -n2:0:+1024M -t2:be00 -c2:Boot $DISK1 sgdisk -n2:0:+1024M -t2:be00 -c2:Boot $DISK2 sgdisk -n3:0:0 -t3:bf00 -c3:Ubuntu $DISK1 sgdisk -n3:0:0 -t3:bf00 -c3:Ubuntu $DISK2
I will use UEFI to boot, so I need to create a disk with type EF00 formatted to vfat:
mkfs.msdos -F 32 -n EFI ${DISK1}-part1
mkfs.msdos -F 32 -n EFI ${DISK2}-part1
Now it’s time to create zfs. I will use grub, which supports booting from zfs, although not with all options. So creating a boot partition requires explicitly specifying only what grub understands:
zpool create -f -o cachefile=/etc/zfs/zpool.cache -o ashift=12 \
-o autotrim=on -d -o feature@async_destroy=enabled
-o feature@bookmarks=enabled -o feature@embedded_data=enabled
-o feature@empty_bpobj=enabled -o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled
-o feature@filesystem_limits=enabled -o feature@hole_birth=enabled
-o feature@large_blocks=enabled -o feature@lz4_compress=enabled
-o feature@spacemap_histogram=enabled -O acltype=posixacl -O canmount=off \
-O compression=lz4 -O devices=off -O normalization=formD -O relatime=on
-O xattr=sa -O mountpoint=/boot -R /mnt \
bpool mirror ${DISK1}-part2 ${DISK2}-part2
The boot partition is created and we can now create the root partition (since we are using encryption we will need to type in the password):
zpool create -f -o ashift=12 -o autotrim=on -O encryption=aes-256-gcm \
-O keyylocation=prompt -O keyformat=passphrase -O acltype=posixacl \
-O canmount=off -O compression=lz4 -O dnodesize=auto
-O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/
-R /mnt rpool mirror ${DISK1}-part3 ${DISK2}-part3/
You can create datasets. I decided to keep it to a minimum:
zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zfs create -o canmount=off -o mountpoint=none bpool/BOOT
UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null \
|tr -dc 'a-z0-9' | cut -c-6)
zfs create -o mountpoint=/ -o com.ubuntu.zsys:bootfs=yes \
-o com.ubuntu.zsys:last-used=$(date +%s) \
rpool/ROOT/ubuntu_$UUID
zfs create -o mountpoint=/boot bpool/BOOT/ubuntu_$UUID
zfs create -o canmount=off -o mountpoint=/ rpool/USERDATA
zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID \
-o canmount=on -o mountpoint=/home/$USERNAME \
rpool/USERDATA/${USERNAME}_$UUID
To install the system, let’s use debootstrap:
apt-get install -y debootstrap debootstrap focal /mnt
Copy the missing components to the new filesystem:
echo $HOSTNAME >/mnt/etc/hostname sed '/cdrom/d' /etc/apt/sources.list > /mnt/etc/apt/sources.list sed "s/ubuntu/$HOSTNAME/" /etc/hosts > /mnt/etc/hosts cp /etc/netplan/*.yaml /mnt/etc/netplan/
And mount the pseudofs you need to continue the installation:
mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys
Enter the chroot environment:
chroot /mnt /usr/bin/env DISK1=$DISK1 DISK2=$DISK2 USERNAME=$USERNAME \
/bin/bash -login
Update the indexes of the binary packages and set the locale:
apt-get update locale-gen --purge "en_US.UTF-8" update-locale LANG=en_US.UTF-8 LANGUAGE=en_US dpkg-reconfigure --frontend noninteractive locales
Set the desired time zone:
dpkg-reconfigure tzdata
Mount the EFI partition. Usually they mount it under /boot/efi, but in our case we have 2 partitions and there is a problem with the order in which the disks are mounted. I decided to mount the disk in a different hierarchy and use a symlink:
mkdir /run/efi1
mount $DISK1-part1 /run/efi1
ln -s /run/efi1 /boot/efi
echo /dev/disk/by-uuid/$(blkid -s UUID -o value \
${DISK1}-part1) /run/efi1 vfat defaults 0 0 >> /etc/fstab
echo /dev/disk/by-uuid/$(blkid -s UUID -o value \
${DISK2}-part1) /run/efi2 vfat defaults 0 0 >> /etc/fstab
Install all other needed packages:
apt-get install -y grub-efi-amd64 grub-efi-amd64-signed linux-image-generic \
shim-signed zfs-initramfs zsys ubuntu-minimal network-manager
Because of the regression, we have to add the kernel parameter init_on_alloc=0:
sed -ie 's/\(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*\)/\1 init_on_alloc=0/' \
/etc/default/grub
I prefer to have a small swap:
zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off \
-o logbias=throughput -o sync=always -o primarycache=metadata \
-o secondarycache=none rpool/swap
mkswap -f /dev/zvol/rpool/swap
echo "/dev/zvol/rpool/swap none swap defaults 0 0". >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume
Add a user:
adduser $USERNAME find /etc/skel/ -type f|xargs cp -t /home/$USERNAME chown -R $USERNAME:$USERNAME /home/$USERNAME usermod -a -G adm,cdrom,dip,plugdev,sudo $USERNAME
Update inird and grub and get out of the chroot environment:
update-initramfs -c -k all
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
-bootloader-id=ubuntu --recheck --no-floppy
exit
Unmount what was mounted in the chroot environment and reboot:
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a
reboot
Since everything happens in virtualbox I have to mention that with UEFI enabled the virtualbox refuses to boot from the optical drive. So at this point I remove the disk from the virtual drive and enable UEFI boot.
Unless something unforeseen happens you’ll see a grub menu. But don’t rush to press enter! Instead boot into recovery mode, because when importing the zfs root pool an error will occur, caused by the fact that the last time the pool was used on a machine with a different name. The fix for this is simple:
zpool import -f rpool exit
After that you will be asked for your password to access the disk and your boot will continue up to the point where you will be asked to use the emergency console (because we are in recovery mode) or press ctrl-d to boot normally. Press ctrl-d. After a few seconds you will be able to login using the previously created user. This is not the end of our misadventures. Take a look at the /boot directory and you will see that it is empty. The boot pool has not been imported either. Let’s fix this:
zpool import bpool/
The final touch is to mark both EFI partitions as needing to be updated when grub changes:
dpkg-reconfigure grub-efi-amd64
Now the installation is completely finished and you can reboot and use the default grub menu item. You will see a black screen in the kernel options because the default is quiet. The disk access password will have to be entered blindly after a few seconds of booting. You can remove quiet from the parameters or put the plymouth package in.
All commands above can be downloaded in a single script which needs the variables DISK1, DISK2, HOSTNAME and USERNAME to work.
If you are looking for: the best courier service in Moscow