Use LVM to system-upgrade a Fedora Linux server with minimal downtime

Has your Fedora Linux server been End of Life for months because you can’t seem to schedule the hours of downtime required to upgrade it? There are ways to shorten that downtime to just the few minutes required for a reboot. You can do this utilizing LVM and VM technologies, all provided by Fedora.

Most users find it simple to upgrade from one Fedora Linux release to the next with the standard process. However, special cases can be handled using existing Fedora VM and LVM capabilities. This article shows one way to upgrade a Fedora Linux server using DNF while using Logical Volume Management (LVM) to keep a bootable backup in case of problems. How to run the upgrade concurrently in a virtual machine to minimize downtime will also be demonstrated. This example upgrades a Fedora Linux 33 virtual machine host to Fedora Linux 35.

The process shown here is admittedly more complex than the exceptionally easy standard upgrade process. Thus, you should have a strong grasp of how LVM and virtual machines work before attempting. Without proper skill and care, you could lose data and/or be forced to reinstall your system! If you don’t have essential Fedora Linux administration skills, it is highly recommended you stick to the supported upgrade methods only.

Prerequisite skills for this method include:

Prepare the system

This example assumes you already have installed:

• libvirt
• qemu-kvm
• partclone
• python3-dnf-plugin-system-upgrade

You must have enough memory available (2G recommended but I have succeeded with 1G) to create an additional virtual machine to run the upgrade. Your system can continue to operate while you prepare for and download the upgrade, and while the upgrade runs.

Before you start, ensure your existing system is fully updated.

$ sudo dnf upgrade --refresh

Since this is a server, rebooting may be more involved than a single user system. If the kernel was updated, or critical running packages were updated, you need to reboot after notifying users as needed.

Check that your root filesystem is mounted via LVM.

$ df /
Filesystem               1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_julie-f33  25671908  14349844  10089344  59% /

$ sudo lvs
LV   VG       Attr        LSize 
f31  vg_julie -wi-a-----  15.00g
f33  vg_julie -wi-ao----  25.00g
root vg_julie -wi-ao---- 300.00g
swap vg_julie -wi-a-----   4.00g

If you used the defaults when you installed Fedora Linux, you may find the root filesystem is mounted on a LV named root. The name of your volume group will likely be different. Look at the total size of the root volume. In the example, the root filesystem is named f33 and is 25G in size. In case you were wondering, the LV named root is a Btrfs filesystem with a subvolume named home which may at some point have a subvolume for the root filesystem as well.

Next, ensure you have enough free space in LVM.

$ sudo vgs
VG       #PV #LV #SN Attr   VSize   VFree
vg_julie   1   4   0 wz--n- 464.78g 130.78g

This system has enough free space to allocate a 25G logical volume for the upgraded Fedora Linux 35 root. If you don’t, LVM management is beyond the scope of this article, but you can review a few suggestions in a previous article.

Note for Btrfs users

The root filesystem must be copied to a logical volume to boot in a virtual machine. For a Btrfs root, you could use Btrfs send to copy a Btrfs snapshot of a root subvolume to a new Btrfs formatted LV, but you must boot from the new system to copy back the upgraded system because of selinux updates. This cryptic summary probably needs another article.

Clone the root filesystem

First, allocate a new LV for the upgraded system. Make sure to use the correct name for your system’s volume group (VG). In this example it’s vg_julie. Also, make sure you allocate the same size or more (if you want to expand later… not addressed here).

$ sudo lvcreate -L25G -n f35 vg_julie
Logical volume "f35" created.

We are going to run the snapshot in a VM with networking disabled. So first, we download the upgrade packages:

$ sudo dnf system-upgrade download --releasever=35

Do not reboot yet – the actual upgrade will take place in a virtual machine!

Next, make a snapshot of your current root filesystem. This example creates a snapshot volume named f33_s.

$ sync
$ sudo lvcreate -s -L1G -n f33_s vg_julie/f33
Using default stripesize 64.00 KiB.
Logical volume "f33_s" created.

Limitations of background upgrading

It is important to realize that once a snapshot is created, data logged to the root file system will not carry over to the upgraded file system. In terms of Logging, for instance, logs that are written to the host system during the period of updating will be available on the backup LV after the upgraded LV is booted on bare metal, but not in the new LV’s logging directory. This method does not address merging them. This method works best, therefore, where critical systems write to their own LVs, rather than a sub-directory on the root filesystem. For instance, if this is a mail server, you may wish to ensure /var/spool/mail is mounted separately from the root filesystem. Personally, I symlink /var/spool/mail to /home/mailspool, keeping all mailboxes on one seperate LV filesystem, unchanged by updates to the root filesystem. Use mailq before the snapsnot to ensure no mail is queued or quarantined. You may wish to stop mail transfer service during the upgrade, or run mailq again to ensure nothing is queued before rebooting into upgraded system.

Similarly, if this Fedora Linux server runs a postgresql database server, /var/lib/pgsql should probably be on its own filesystem.

This process underscores why it is best-practice for admins to move service data to its own LV filesystem.

Final Preparation

Take note of your current system, especially:

• Current Kernel Version
• What version you are currently running, and what version is your target
• Current root disk and/or UUID/Label
• How your swap is setup

For the latter two, you can find them in your /etc/fstab. Now might be a good time to make a quick backup of that file.

Create the upgrade VM

The snapshot can now be copied to the new LV. Make sure you have the destination correct when you substitute your own volume names. If you are not careful you could delete data irrevocably. Also, make sure you are copying from the snapshot of your root, not your live root. This example has an ext4 root filesystem. Change to use your actual root filesystem type. You could also change the root filesystem type at this step by using rsync, but that will be a future article.

$ sudo partclone.ext4 -b -s /dev/vg_julie/f33_s -o /dev/vg_julie/f35 
Partclone v0.3.17 http://partclone.org
Starting to back up device(/dev/vg_julie/f33_s) to device(/dev/vg_julie/f35)
Elapsed: 00:00:01, Remaining: 00:00:00, Completed: 100.00%
Total Time: 00:00:01, 100.00% completed!
done!
File system:  EXTFS
Device size:   26.8 GB = 6553600 Blocks
Space in use:  16.9 GB = 4136996 Blocks
Free Space:     9.9 GB = 2416604 Blocks
Block size:   4096 Byte
Elapsed: 00:07:32, Remaining: 00:00:00, Completed: 100.00%, Rate:   2.25GB/min,
current block:    6514688, total block:    6553600, Complete: 100.00%
Total Time: 00:07:32, Ave. Rate:    2.2GB/min, 100.00% completed!
Syncing... OK!
Partclone successfully cloned the device (/dev/vg_julie/f33_s) to the device (/dev/vg_julie/f35)
Cloned successfully.  

Give the new filesystem copy a unique UUID and label. This is not strictly necessary, but given that UUIDs are supposed to be unique, avoid future confusion by tagging the new filesystem. Here is how this is done for an ext4 root filesystem:

$ sudo e2fsck -f /dev/vg_julie/f35
$ sudo tune2fs -U random /dev/vg_julie/f35
$ sudo e2label /dev/vg_julie/f35 F35

Note: For Btrfs, the filesystem must be mounted to change UUID and/or LABEL.

Now remove the snapshot volume which is no longer needed:

$ sudo lvremove vg_julie/f33_s
Do you really want to remove active logical volume vg_julie/f33_s? [y/n]: y
Logical volume "f33_s" successfully removed

You may wish to make a snapshot of /home at this point if you have it mounted separately (use Clone the root filesystem for steps). Sometimes, upgraded applications make changes that are incompatible with a much older Fedora Linux version. If you need to revert, edit the /etc/fstab file on the old root filesystem to mount the snapshot on /home. Remember that when the snapshot is full, it will disappear! You may also wish to make a normal backup of /home (and other filesystems for database and mail) for good measure.

Configuring the VM to use the new root

Mount the new LV and backup your existing GRUB settings:

$ sudo mkdir /mnt/f35
$ sudo mount /dev/vg_julie/f35 /mnt/f35
$ sudo mkdir /mnt/f35/f33

Our previous article copied /boot/grub2/grub.cfg to a backup, but Fedora Linux now uses BLS – the Boot Loader System. Grub entries are in /boot/loader/entries and you generally don’t need to touch grub.cfg

Edit /mnt/f35/etc/default/grub and change the default root LV activated at boot:

GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_julie/f35 rd.lvm.lv=vg_julie/swap rhgb quiet"

Copy /mnt/f35/etc/fstab to /mnt/f35/etc/fstab.f33 as a backup. Edit /mnt/f35/etc/fstab to comment out any filesystems that will not be available to the virtual machine. Only the / block device will be available. Change the root filesystem to use the new UUID or LABEL. E.g.

LABEL=F35 / ext4 defaults 1 1

The upgrade process will expect a /boot directory. If /boot is not in your root filesystem, copy it into the new LV:

$ sudo rsync -ravHXx /boot/ /mnt/f35/boot
... shows you files copied

You will need to disable systemd services in the new LV that could interfere with host. For instance, a VPN like cjdns or openvpn:

$ sudo systemctl --root=/mnt/f35 disable cjdns
$ sudo systemctl --root=/mnt/f35 disable openvpn-client@client
$ sudo systemctl --root=/mnt/f35 disable libvirtd
...

Remember (write down!) what you disabled, because you will need to enable them again before booting the upgraded system on bare metal! Unmount the new root filesystem:

$ sudo umount /mnt/f35

Perform the upgrade in a VM

The VM disk will need a virtio driver, so make a new version of initramfs.

$ sudo dracut --add-drivers virtio_blk /tmp/initramfs-5.14.9-100.fc33.x86_64.img

If you converted the root filesystem to a new type, you may also need to add a new filesystem driver or module. For example, after converting ext4 to Btrfs, you would need ‐‐add btrfs ‐‐add-drivers virtio_blk.

Using libvirtd is out of scope for this already long article, because as admin of a virtual host, you should know to create a virtual machine using virsh or virt-manager or your favorite front end. Create a virtual machine with a single disk volume mapped to the new root filesystem LV (/dev/vg_julie/f35 in the example). Use Direct Kernel Boot with the same initramfs name you created with dracut:

Kernel: /boot/vmlinuz-5.14.9-100.fc33.x86_64
Initramfs: /tmp/initramfs-5.14.9-100.fc33.x86_64.img
Args: root=LABEL=F35 ro

Now, start the virtual machine, and login to its console as root. Check that nothing else needs to be disabled. If all is well, reboot the VM with $ sudo dnf system-upgrade reboot on the VM console (not on your bare metal console – that would not be a total disaster, but you’ll be down while the upgrade runs). You should see the upgrade proceeding on the VM console after the VM reboots. Meanwhile, your host system continues on as usual.

If, for some reason, sudo dnf system-upgrade reboot does not work, and network is available in the VM, you can call sudo dnf system-upgrade in the VM, and it will work fairly quickly, as everything is already downloaded.

… time passes as your virtual host hums along and the VM upgrades … get some coffee and a donut while your current host continues to operate as usual.

When the upgrade is finished, the VM will reboot again. Log in as root on the VM console and check things out. (Note: if you are using a GUI, look for “Send Key” to get a root console.) You will still be running the old kernel (f33 in this example), because you are using Direct Kernel Boot for the VM. If anything went wrong, you can destroy the VM, and start over at taking a snapshot of the current root filesystem.

If it looks ok, run sudo systemctl poweroff to shutdown the VM.

Prepare the VM host to boot the upgraded root filesystem

The initramfs generated inside the VM does not include the drivers you need for bare metal. In addition to using virtio_blk instead of an actual disk controller driver, a VM host probably needs LVM and RAID drivers. In the interest of simplicity, we will generate an initramfs with all drivers and modules. This creates a large image, around 100 Mbyte, but there is no drawback other than disk space used on a typically limited /boot filesystem. Experts: it is possible to use lsinitrd and dracut options ‐‐add and ‐‐add-drivers instead to create a smaller image if you know what is needed.

Copy the new boot loader entry and kernel files to the real /boot.

Change the kernel version to the one just installed by the system-upgrade. You can run dracut without ‐‐force initially to ensure it is overwriting what you expect.

$ sudo mount /dev/vg_julie/f35 /mnt/f35 # make sure your VM is truly not running
$ sudo mount -t proc /proc /mnt/f35/proc
$ sudo chroot /mnt/f35
# ls -l /boot/*fc35*
-rw-r--r--. 1 root root   236665 Sep 30 08:10 /boot/config-5.14.9-300.fc35.x86_64
-rw-------. 1 root root 35571851 Oct  6 15:15 /boot/initramfs-5.14.9-300.fc35.x86_64.img
lrwxrwxrwx. 1 root root       46 Oct  6 15:15 /boot/symvers-5.14.9-300.fc35.x86_64.gz -> /lib/modules/5.14.9-300.fc35.x86_64/symvers.gz
-rw-------. 1 root root  5849998 Sep 30 08:10 /boot/System.map-5.14.9-300.fc35.x86_64
-rwxr-xr-x. 1 root root 11032912 Sep 30 08:10 /boot/vmlinuz-5.14.9-300.fc35.x86_64
# dracut -N --kver=5.14.9-300.fc35.x86_64 --force
# exit
$ sudo cp -p /mnt/f35/boot/*-5.14.9-300.fc35* /boot
$ sudo cp -p /mnt/f35/boot/loader/entries/*-5.14.9-300.fc35.x86_64.conf /boot/loader/entries

If you mount /boot separately (very likely), then you need to edit the loader entry copied from the VM to remove /boot from the kernel and initrd. Edit /boot/loader/entries/*-5.14.9-300.fc35.x86_64.conf and change:

linux /boot/vmlinuz-5.14.9-300.fc35.x86_64
initrd /boot/initramfs-5.14.9-300.fc35.x86_64.img

... to ...

linux /vmlinuz-5.14.9-300.fc35.x86_64
initrd /initramfs-5.14.9-300.fc35.x86_64.img 

Re-enable services you disabled for the VM. If you didn’t write them down, check ~/.bash_history or the equivalent for your shell.

$ sudo systemctl --root=/mnt/f35 enable cjdns
$ sudo systemctl --root=/mnt/f35 enable openvpn-client@client
$ sudo systemctl --root=/mnt/f35 enable libvirtd

Edit fstab in your new VM (/mnt/f35/etc/fstab) to include all the filesystems you need, but commented out to run in a VM.

IF (and only if) /boot is separately mounted on your host (shows in df), rename and recreate /boot in the upgraded root – this will be a handy backup. (Otherwise, it will get mounted over and the files hidden.)

$ sudo mv /mnt/f35/boot /mnt/f35/bootVM
$ sudo mkdir /mnt/f35/boot

Finally, reboot into the upgraded system:

$ sudo systemctl reboot
... system reboots
$ df / /f33
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_julie-f35  25671908 15948668   8490520  66% /
/dev/mapper/vg_julie-f33  15350728 14540288     64256 100% /f33

Check that required services are running. If there are any insurmountable problems, you can reboot back into Fedora Linux 33!

Finally, if you are running well, remove the VM you used to upgrade with, so you don’t corrupt your host system by accidentally starting it while you are using the LV on the host system. Make sure you don’t delete the LV in the process.