Use LVM to Upgrade Fedora

Most users find it simple to upgrade from one Fedora release to the next with the standard process. However, there are inevitably many special cases that Fedora can also handle. This article shows one way to upgrade using DNF along with Logical Volume Management (LVM) to keep a bootable backup in case of problems. This example upgrades a Fedora 26 system to Fedora 28.

The process shown here is more complex than the standard upgrade process. You should have a strong grasp of how LVM works before you use this process. Without proper skill and care, you could lose data and/or be forced to reinstall your system! If you don’t know what you’re doing, it is highly recommended you stick to the supported upgrade methods only.

Preparing the system

Before you start, ensure your existing system is fully updated.

$ sudo dnf update
$ sudo systemctl reboot # or GUI equivalent

Check that your root filesystem is mounted via LVM.

$ df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_sdg-f26 20511312 14879816 4566536 77% /

$ sudo lvs
LV        VG           Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
f22       vg_sdg -wi-ao---- 15.00g
f24_64    vg_sdg -wi-ao---- 20.00g
f26       vg_sdg -wi-ao---- 20.00g
home      vg_sdg -wi-ao---- 100.00g
mockcache vg_sdg -wi-ao---- 10.00g
swap      vg_sdg -wi-ao---- 4.00g
test      vg_sdg -wi-a----- 1.00g
vg_vm     vg_sdg -wi-ao---- 20.00g

If you used the defaults when you installed Fedora, you may find the root filesystem is mounted on a LV named root. The name of your volume group will likely be different. Look at the total size of the root volume. In the example, the root filesystem is named f26 and is 20G in size.

Next, ensure you have enough free space in LVM.

$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
vg_sdg 1 8 0 wz--n- 232.39g 42.39g

This system has enough free space to allocate a 20G logical volume for the upgraded Fedora 28 root. If you used the default install, there will be no free space in LVM. Managing LVM in general is beyond the scope of this article, but here are some possibilities:

1. /home on its own LV, and lots of free space in /home

You can logout of the GUI and switch to a text console, logging in as root. Then you can unmount /home, and use lvreduce -r to resize and reallocate the /home LV. You might also boot from a Live image (so as not to use /home) and use the gparted GUI utility.

2. Most of the LVM space allocated to a root LV, with lots of free space in the filesystem

You can boot from a Live image and use the gparted GUI utility to reduce the root LV. Consider moving /home to its own filesystem at this point, but that is beyond the scope of this article.

3. Most of the file systems are full, but you have LVs you no longer need

You can delete the unneeded LVs, freeing space in the volume group for this operation.

Creating a backup

First, allocate a new LV for the upgraded system. Make sure to use the correct name for your system’s volume group (VG). In this example it’s vg_sdg.

$ sudo lvcreate -L20G -n f28 vg_sdg
Logical volume "f28" created.

Next, make a snapshot of your current root filesystem. This example creates a snapshot volume named f26_s.

$ sync
$ sudo lvcreate -s -L1G -n f26_s vg_sdg/f26
Using default stripesize 64.00 KiB.
Logical volume "f26_s" created.

The snapshot can now be copied to the new LV. Make sure you have the destination correct when you substitute your own volume names. If you are not careful you could delete data irrevocably. Also, make sure you are copying from the snapshot of your root, not your live root.

$ sudo dd if=/dev/vg_sdg/f26_s of=/dev/vg_sdg/f28 bs=256k
81920+0 records in
81920+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 149.179 s, 144 MB/s

Give the new filesystem copy a unique UUID. This is not strictly necessary, but UUIDs are supposed to be unique, so this avoids future confusion. Here is how for an ext4 root filesystem:

$ sudo e2fsck -f /dev/vg_sdg/f28
$ sudo tune2fs -U random /dev/vg_sdg/f28

Then remove the snapshot volume which is no longer needed:

$ sudo lvremove vg_sdg/f26_s
Do you really want to remove active logical volume vg_sdg/f26_s? [y/n]: y
Logical volume "f26_s" successfully removed

You may wish to make a snapshot of /home at this point if you have it mounted separately. Sometimes, upgraded applications make changes that are incompatible with a much older Fedora version. If needed, edit the /etc/fstab file on the old root filesystem to mount the snapshot on /home. Remember that when the snapshot is full, it will disappear! You may also wish to make a normal backup of /home for good measure.

Configuring to use the new root

First, mount the new LV and backup your existing GRUB settings:

$ sudo mkdir /mnt/f28
$ sudo mount /dev/vg_sdg/f28 /mnt/f28
$ sudo mkdir /mnt/f28/f26
$ cd /boot/grub2
$ sudo cp -p grub.cfg grub.cfg.old

Edit grub.conf and add this before the first menuentry, unless you already have it:

menuentry 'Old boot menu' {
        configfile /grub2/grub.cfg.old
}

Edit grub.conf and change the default menuentry to activate and mount the new root filesystem. Change this line:

linux16 /vmlinuz-4.16.11-100.fc26.x86_64 root=/dev/mapper/vg_sdg-f26 ro rd.lvm.lv=vg_sdg/f26 rd.lvm.lv=vg_sdg/swap rhgb quiet LANG=en_US.UTF-8

So that it reads like this. Remember to use the correct names for your system’s VG and LV entries!

linux16 /vmlinuz-4.16.11-100.fc26.x86_64 root=/dev/mapper/vg_sdg-f28 ro rd.lvm.lv=vg_sdg/f28 rd.lvm.lv=vg_sdg/swap rhgb quiet LANG=en_US.UTF-8

Edit /mnt/f28/etc/default/grub and change the default root LV activated at boot:

GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_sdg/f28 rd.lvm.lv=vg_sdg/swap rhgb quiet"

Edit /mnt/f28/etc/fstab to change the mounting of the root filesystem from the old volume:

/dev/mapper/vg_sdg-f26 / ext4 defaults 1 1

to the new one:

/dev/mapper/vg_sdg-f28 / ext4 defaults 1 1

Then, add a read-only mount of the old system for reference purposes:

/dev/mapper/vg_sdg-f26 /f26 ext4 ro,nodev,noexec 0 0

If your root filesystem is mounted by UUID, you will need to change this. Here is how if your root is an ext4 filesystem:

$ sudo e2label /dev/vg_sdg/f28 F28

Now edit /mnt/f28/etc/fstab to use the label. Change the mount line for the root file system so it reads like this:

LABEL=F28 / ext4 defaults 1 1

Rebooting and upgrading

Reboot, and your system will be using the new root filesystem. It is still Fedora 26, but a copy with a new LV name, and ready for dnf system-upgrade! If anything goes wrong, use the Old Boot Menu to boot back into your working system, which this procedure avoids touching.

$ sudo systemctl reboot # or GUI equivalent
...
$ df / /f26
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_sdg-f28 20511312 14903196 4543156 77% /
/dev/mapper/vg_sdg-f26 20511312 14866412 4579940 77% /f26

You may wish to verify that using the Old Boot Menu does indeed get you back to having root mounted on the old root filesystem.

Now follow the instructions at this wiki page. If anything goes wrong with the system upgrade, you have a working system to boot back into.

Future ideas

The steps to create a new LV and copy a snapshot of root to it could be automated with a generic script.  It needs only the name of the new LV, since the size and device of the existing root are easy to determine. For example, one would be able to type this command:

$ sudo copyfs / f28

Supplying the mount-point to copy makes it clearer what is happening, and copying other mount-points like /home could be useful.

For System Administrators Using Software

18 Comments

  1. Pierre

    Your article is very interesting, I learned about the snapshot logical volume which is demystified now ! Thanks

  2. I have my Fedora as LVM, so this comes just in time. Thanks!

  3. Some last minute changes didn’t get in. The most important is that changing the UUID needs to happen while the filesystem is unmounted. Insert after the step using dd:

    Give the new filesystem copy a unique UUID. This is not strictly necessary, but UUIDs are supposed to be unique, so let’s avoid confusing ourselves in the future. Here is how for an ext4 root filesystem:

    $ sudo e2fsck -f /dev/vg_sdg/f28
    $ sudo tune2fs -U random /dev/vg_sdg/f28

  4. Supersonic Tumbleweed

    Thanks for this excellent write up – this is why my experience with Fedora is great so far. It gives me a sense of community because I just was exploring and learning about lvm, moving my install between disks and there you are, explaining the tough parts without me even asking 😀
    Great stuff, have a nice day!

  5. Nice article! I wish there was a better general understanding of LVM and its benefits among the desktop users. Just two notes from me:
    1) This would be a lot easier and nicer if Fedora was using LVM Thin Provisioning by default.
    2) Using XFS instead of ext4 would also make things nicer/faster because XFS comes with the two amazing tools — xfsdump and xfsrestore — that can be used to backup a file system or, like in this use case, clone a filesystem to another device in a lot more efficient way than using dd.

    • Why does Vratislav prefer thinpools?

      Would LVM thinpool be easier?

      When I made the snapshot in the article, the snapshot is actually writable, and we could have allocated plenty of space for the snapshot and used it directly for the upgraded system. The snapshot initially points to the “origin” volume, but data is gradually copied to the snapshot via a copy on write process as origin or destination is written to.

      So why didn’t I do that? First, the COW (copy on write) data is shuffled on the disk, making access less efficient. Second, the COW table is kept in memory, and makes activating the snapshot at startup slow. Third, while the first snapshot is reasonably efficient, additional snapshots become much slower, as origin data is copied to all snapshots. Fourth, you can’t take a snapshot of a snapshot – so backups would be more difficult.

      That last is not really an issue. Sometimes, I rename the origin, and make the old Fedora system the snapshot. The old system is not active – except when booting from the old system when something goes wrong. Eventually, the old snapshot can be deleted.

      So how does thinpool help? With standard snapshots, each snapshot has its own dedicated storage pool, which you allocate when you create the snapshot. If that pool is exhausted – the snapshot effectively goes away, and all reads return an error.

      A thinpool creates a single storage pool, which is shared between many logical volumes. First, any number of snapshots can be created without affecting performance – at most one COW is required for any update. Second, snapshots of snapshots work just fine.

      With the root volume in a thinpool, we could just create a snapshot of f26 named f28, point grub at it, and we are done – with no copy required!

      So why don’t I use a thinpool for my root fs?

      First, there is still in reality a copy required – it is just done gradually via the copy on write process. All volumes in a thinpool perform like a single standard snapshot.

      Second, when storage in the thinpool is exhaused, all your volumes disappear! Well, hopefully that should be amended to “all volumes are frozen in a read-only mode” by this time. In either case, it is much more catastrophic than the temporary “instant backup” of a standard snapshot. You Never, Never, Never want your thinpool to run out of space – and any write, can consume that space!

      In contrast, using a standard logical volume (not in a thinpool) performs essentially like a partition. Like a partition, standard LVs are very forgiving of admin errors, as the data is contained in a few large segments, and the segment map is not updated frequently – there will be many copies of the metadata in your backups.

      • That’s quite a nice summary. I have been using LVM ThinP for my root and home volumes for years now without a single problem. I actually even use it for my backup volume on a backup HDD (creating snapshots so that I can go back in history). The performance has always been great and the flexibility is much greater than with non-ThinP LVM. The only downside is that one has to check one one or two numbers from time to time. Which I do anyway because I check my free space regularly.

        And it’s fair to mention that the Fedora installer leaves space in the VG for the thin pool’s data or metadata segments to grow on demand. So it’s not like one just runs into the state with “frozen” volumes easily.

        But maybe I’m incorrectly thinking about Fedora Workstation as “a developer workstation” (as used to be stated somewhere, IIRC) which is not the case these days.

    • Why does Vratislav prefer XFS?

      The upgrade process presented doesn’t really care what filesystem you use for your root fs. So this is a somewhat of a digression. XFS is a fine filesystem, and performance for large files is especially impressive (hence it is popular for video storage). So why don’t I use it for my root fs?

      First, XFS does not support shrinking the filesystem – other than by copying to a new smaller filesystem.

      Second, root fs has mostly small files.

      Third, ext4 performs better than XFS on simple flash media (i.e. a thumb drive or SD card). It is in many cases convenient to introduce people to Fedora™ by installing to such media when they are not ready to replace or overwrite their hard drive with Windows™ installed.

      2a. Why did I use dd instead of rsync or dump/restore to copy the root fs?

      First, the root fs in the example was 77% full. Using an image copy like dd is actually pretty efficient in that case. You can also use rsync, when the fs is mostly empty, or when you want to change to a different filesystem.

      Second, using rsync -ravhX would complicate the example, because you would then have to format the new filesystem and mount the snapshot and new filesystem. Even with (xfs)dump/restore, you have to format and mount the destination filesystem.

      • Let’s not hijack this comments thread for a file systems battle. There are various benchmarks and aspects that one has to take into account when comparing file systems (typical workload, physical device, number of CPUs,…). Having fresh data/results for a typical HW these days (multi-core, SSD/NVMe based,…) and the current version of XFS could actually prove you wrong. Simple claims are never correct.

        In my comment I was only referring to this particular use case where the xfsdump and xfsrestore tools would do an amazing work. They are not like neither rsync nor dd. They actually make use of the knowledge of the file system metadata to do the dump/restore (and thus ‘clone’ when combined) very efficiently. And one can also do delta backups easily. I’m not aware of similar tools existing for ext4, but if there are some, I’d be happy to learn about them.

        • I agree. I mostly wanted to point out that dd is file system agnostic, and reasonably efficient when the filesystem is mostly full. Any more efficient method of cloning the filesystem is going to be very filesystem specific, and hence not ideal for my article.

        • FYI, ext4 has dump and restore, which efficiently copy only the used blocks of an ext4 filesystem. xfsdump and xfsrestore were named after these venerable utilities.

          Here is an example using dump/restore:
          https://stackoverflow.com/questions/37488629/how-to-use-dump-and-restore-to-clone-a-linux-os-drive

          Using either dump/restore or xfsdump/xfsrestore would complicate the cloning instructions, as both require formatting and mounting the destination filesystem – which is the main reason I didn’t go that route for the article.

          I’m more and more seeing a need for an “fsclone” utility that knows about LVM and snapshots (there is no need for an fsclone with thinpool, of course), knows how to most efficiently clone some filesystem types, punts to using ‘dd’ for those it doesn’t know about, and handles formatting and mounting the destination when using (xfs)dump/restore or rsync. It might even use dd for nearly full filesystems, as there is a point at which dd is actually more efficient on any filesystem.

          • Heh. As usual, someone has already done most of it. The utility we want seems to be ‘partclone’ – and it is already in Fedora. It knows how to efficiently clone a variety of filesystems to/from an image backup or directly between block devices. I can use that in my fsclone utility, which will additionally know about LVM!

            • One of the supported filesystems is nilfs, which seems similar to reiserfs in concept. It is much more modest in scope than btrfs, being a simply a filesystem, and not trying to cross over into block device management as well. Hence it is well suited for use with LVM. I like keeping the management tools separate like that. LVM for block device management, your favorite filesystem on logical volumes.

              What log structured filesystems like reiserfs, nilfs, btrfs offer is continuous checkpointing. Instead of reusing the same reserved checkpoint area (log) over and over, the entire filesystem is structured as a giant log. Each point of consistency is a checkpoint. Checkpoints are garbage collected to recycle space, and selected checkpoints can be marked to become snapshots, and are not collected.

              nilfs is part of Fedora, and modest enough that I will certainly start giving it a try in virtual machines, and perhaps more. It could be a potential life saver (losing a lot of time is losing part of your life) as a development filesystem. Have you ever typed rm * .o and got the dreaded “.o: not found” ?

          • Here is an example of running partclone instead of dd to clone the filesystem:

            [root@melissa ~]# partclone.ext4 -b -s /dev/vg_melissa/f27_s -o /dev/vg_melissa/f28
            Partclone v0.3.11 http://partclone.org
            Starting to back up device(/dev/vg_melissa/f27_s) to device(/dev/vg_melissa/f28)
            Elapsed: 00:00:01, Remaining: 00:00:00, Completed: 100.00%
            Total Time: 00:00:01, 100.00% completed!
            done!
            File system: EXTFS
            Device size: 16.1 GB = 3932160 Blocks
            Space in use: 10.5 GB = 2563963 Blocks
            Free Space: 5.6 GB = 1368197 Blocks
            Block size: 4096 Byte
            Elapsed: 00:03:42, Remaining: 00:00:00, Completed: 100.00%, Rate: 2.84GB/min,
            current block: 3932160, total block: 3932160, Complete: 100.00%
            Total Time: 00:03:42, Ave. Rate: 2.8GB/min, 100.00% completed!
            Syncing… OK!
            Partclone successfully cloned the device (/dev/vg_melissa/f27_s) to the device (/dev/vg_melissa/f28)
            Cloned successfully.
            [root@melissa ~]# blkid /dev/vg_melissa/f27
            /dev/vg_melissa/f27: LABEL=”F27″ UUID=”335c6a7e-a1a7-48a2-90b0-a44bd699c664″ TYPE=”ext4″
            [root@melissa ~]# blkid /dev/vg_melissa/f28
            /dev/vg_melissa/f28: LABEL=”F27″ UUID=”335c6a7e-a1a7-48a2-90b0-a44bd699c664″ TYPE=”ext4″

            Notice that it also does not change the UUID – which is probably what you want for backup purposes when the filesystem is mounted by UUID.

          • Nice! I had no idea that the dump/restore tools were available in the old UNIX systems. And now that I see ‘partclone’ I realize it’s what ‘CloneZilla’ has been using for ages! So much wisdom available, one just has to search sometimes. 🙂

  6. I just realized an important omission. If you have an EFI install, then your grub.cfg will be under /boot/efi/EFI/fedora instead of /boot/grub2. I’ll have to get an EFI system installed to see what path to use for configfile in that case.

  7. Bobin

    Extraordinary, Awesome, superb

Leave a Reply

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions

%d bloggers like this: