Fedora Workstation 34 feature focus: Btrfs transparent compression

Photo by Patrick Lindenberg on Unsplash

The release of Fedora 34 grows ever closer, and with that, some fun new features! A previous feature focus talked about some changes coming to GNOME version 40. This article is going to go a little further under the hood and talk about data compression and transparent compression in btrfs. A term like that may sound scary at first, but less technical users need not be wary. This change is simple to grasp, and will help many Workstation users in several key areas.

What is transparent compression exactly?

Transparent compression is complex, but at its core it is simple to understand: it makes files take up less space. It is somewhat like a compressed tar file or ZIP file. Transparent compression will dynamically optimize your file system’s bits and bytes into a smaller, reversible format. This has many benefits that will be discussed in more depth later on, however, at its core, it makes files smaller. This may leave most computer users with a question: “I can’t just read ZIP files. You need to decompress them. Am I going to need to constantly decompress things when I access them?”. That is where the “transparent” part of this whole concept comes in.

Transparent compression makes a file smaller, but the final version is indistinguishable from the original by the human viewer. If you have ever worked with Audio, Video, or Photography you have probably heard of the terms “lossless” and “lossy”. Think of transparent compression like a lossless compressed PNG file. You want the image to look exactly like the original. Small enough to be streamed over the web but still readable by a human. Transparent compression works similarly. Your file system will look and behave the same way as before (no ZIP files everywhere, no major speed reductions). Everything will look, feel, and behave the same. However, in the background it is taking up much less disk space. This is because BTRFS will dynamically compress and decompress your files for you. It’s “Transparent” because even with all this going on, you won’t notice the difference.

You can learn more about transparent compression at https://btrfs.wiki.kernel.org/index.php/Compression

Transparent compression sounds cool, but also too good to be true…

I would be lying if I said transparent compression doesn’t slow some things down. It adds extra CPU cycles to pretty much any I/O operation, and can affect performance in certain scenarios. However, Fedora is using the extremely efficient zstd:1 algorithm. Several tests show that relative to the other benefits, the downsides are negligible (as I mentioned in my explanation before). Better disk space usage is the greatest benefit. You may also receive reduction of write amplification (can increase the lifespan of SSDs), and enhanced read/write performance.

Btrfs transparent compression is extremely performant, and chances are you won’t even notice a difference when it’s there.

I’m convinced! How do I get this working?

In fresh installations of Fedora 34 and its corresponding beta, it should be enabled by default. However, it is also straightforward to enable before and after an upgrade from Fedora 33. You can even enable it in Fedora 33, if you aren’t ready to upgrade just yet.

  1. (Optional) Backup any important data. The process itself is completely safe, but human error isn’t.
  2. To truly begin you will be editing your fstab. This file tells your computer what file systems exist where, and how they should be handled. You need to be cautious here, but only a few small changes will be made so don’t be intimidated. On an installation of Fedora 33 with the default Btrfs layout the /etc/fstab file will probably look something like this:

<strong>$ $EDITOR /etc/fstab</strong>
UUID=1234 /                       btrfs   subvol=root     0 0
UUID=1234 /boot                   ext4    defaults        1 2
UUID=1234         /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=1234 /home                   btrfs   subvol=home     0 0

NOTE: While this guide builds around the standard partition layout, you may be an advanced enough user to partition things yourself. If so, you are probably also advanced enough to extrapolate the info given here onto your existing system. However, comments on this article are always open for any questions.

Disregard the /boot and /boot/efi directories as they aren’t (currently) compressed. You will be adding the argument compress=zstd:1. This tells the computer that it should transparently compress any newly written files if they benefit from it. Add this option in the fourth column, which currently only contains the subvol option for both /home and /:


UUID=1234 /                       btrfs   subvol=root,compress=zstd:1     0 0
UUID=1234 /boot                   ext4    defaults        1 2
UUID=1234         /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=1234 /home                   btrfs   subvol=home,compress=zstd:1     0 0

Once complete, simply save and exit (on the default nano editor this is CTRL-X, SHIFT-Y, then ENTER).

3. Now that fstab has been edited, tell the computer to read it again. After this, it will make all the changes required:

$ sudo mount -o remount / /home/

Once you’ve done this, you officially have transparent compression enabled for all newly written files!

Recommended: Retroactively compress old files

Chances are you already have many files on your computer. While the previous configuration will compress all newly written files, those old files will not benefit. I recommend taking this next (but optional) step to receive the full benefits of transparent compression.

  1. (Optional) Clean out any data you don’t need (empty trash etc.). This will speed things up. However, it’s not required.
  2. Time to compress your data. One simple command can do this, but its form is dependent on your system. Fedora Workstation (and any other desktop spins using the DNF package manager) should use:
$ sudo btrfs filesystem defrag -czstd -rv / /home/

Fedora Silverblue users should use:

$ sudo btrfs filesystem defrag -czstd -rv / /var/home/ 

Silverblue users may take note of the immutability of some parts of the file system as described here as well as this Bugzilla entry.

NOTE: You may receive several warnings that say something like “Cannot compress permission denied.”. This is because some files, on Silverblue systems especially, the user cannot easily modify. This is a tiny subset of files. They will most likely compress on their own, in time, as the system upgrades.

Compression can take anywhere from a few minutes to an hour depending on how much data you have. Luckily, since all new writes are compressed, you can continue working while this process completes. Just remember it may partially slow down your work at hand and/or the process itself depending on your hardware.

Once this command completes you are officially fully compressed!

How much file space is used, how big are my files

Due to the nature of transparent compression, utilities like du will only report exact, uncompressed, files space usage. This is not the actual space they take up on the disk. The compsize utility is the best way to see how much space your files are actually taking up on disk. An example of a compsize command is:

$ sudo compsize -x / /home/ 

This example provides exact information on how the two locations, / and /home/ are currently, transparently, compressed. If not installed, this utility is available in the Fedora Linux repository.

Conclusion:

Transparent compression is a small but powerful change. It should benefit everyone from developers to sysadmin, from writers to artists, from hobbyists to gamers. It is one among many of the changes in Fedora 34. These changes will allow us to take further advantage of our hardware, and of the powerful Fedora Linux operating system. I have only just touched the surface here. I encourage those of you with interest to begin at the Fedora Project Wiki and Btrfs Wiki to learn more!

Fedora Project community Using Software

51 Comments

  1. Mattias Bengtsson

    Is btrfs compression really lossy? That sounds both very complicated (in that the kennel would have to have an understanding of the actual contents of files to be able to choose a proper lossy compression algorithm for each file type) as well as quite unexpected.

    Are you certain that this is the case and that it’s not just normal lossless compression?

  2. Mattias Bengtsson

    According to the Wikipedia page on zstd it’s a lossless algorithm.

    I found at least one reddit post from a user doing photography that was concerned about btrfs compression being lossy.

    • Bernard

      If you uncompress a zip file, you get exactly the same original file. That can’t be more lossless…

  3. annoyingcommenter

    This article has a few technical errors:

    somewhat like a tar
    tar files don’t compress any data

    Think of transparent compression like a lossy compressed video file.
    I find this analogy fails to really capture what the “transparent” part really means but the use of “lossy” here I feel is really off, because zstd compression is actually lossless and one of the major reasons you can even use it on your entire file system automatically is that it won’t lose any data.
    I think a better explanation would be: all of your files look normal, not zips, but with the use of transparent compression when data is saved to your drive, it gets automatically compressed. So everything on your drive is stored as a ‘zip’, but you operating system decompresses everything for you automatically.

    • I will clarify that I mean compressed tar files.

      I realize that this analogy was perhaps unclear, and have changed the article and left a note regarding it in the comments. Thanks for pointing it out!

      • annoyingcommenter

        Yes! Very nicely fixed with a very accurate explanation that should also be not too hard to grasp even for less technically enclined users, good job. Hope to see more work like this from you here in the future.

  4. Aaaaa

    The only problem I see is that I think some programs and I’m thinking of steam for example may think the drive is full when it isn’t and refuse to use that extra space. Some desktop environments may give constants warnings about lack of disk space. That’s the only thing actually keeping me of implementing this change on my installed systems. I’m not sure if that would be the case anyway.

  5. scareduser

    Think of transparent compression like a lossy compressed video file.

    I’d be very scared to put my filesystem through a lossy compressor. Did you mean lossless?

    • I apologize for not making things clearer, and have changed the article to better reflect what I initially intended. BTRFS transparent compression does not use a lossy compression algorithm. It uses the lossless algorithm zstd. What was intended as a high-level generalization was taken literally due to a lack of clarification on my part.

  6. Note: There was initially some confusion about a simile used in this article. I apologize for not making things clearer, and have changed the article to better reflect what I initially intended. BTRFS transparent compression does not use a lossy compression algorithm. It uses the lossless algorithm zstd. What was intended as a high-level generalization was taken literally due to a lack of clarification on my part.

  7. Cornel

    Good article!

    Do you know how heavy it is on CPU when writing huge files like for example .iso’s

  8. Andre Gompel

    How to allow compression on automounted BTRFS partition (for example a flash drive) ?

    • Just make sure to pass the

      compress

      option when mounting. More specifically

      compress=zstd:1

      .

    • Matti Pulkkinen

      I was actually curious about this myself a while back. At first it seemed like it would be really complicated, but it turned out to be very easy. You just use the command “btrfs property set compression zstd” to enable compression for that drive, where is of course the mountpoint of the flash drive, such as /run/media/user/flashdrive. The property seems to get saved in the filesystem itself, as it carries over even to different computers. Unfortunately this method doesn’t let you set the compression level for zstd, which means the default level is used, but I have not found this to be an issue.

      You can check whether the property got set correctly with the command “btrfs property get .

  9. Scott Trakker

    How much is the data compressed? A few percent? Is there an average number?

    • The files that are actually compressed are usually around 30-40% their original size, but the total end size really depends on your system. I only have around 80% compression ratio, but I also have lots of games and virtual machines that cannot be beneficially compressed and therefore are skipped by BTRFS. I have seen as low as 40%. It is a dissatisfying answer, but you probably won’t know until you try.

      • Scott Trakker

        So it is in the order of tens of percent! That’s really nice and more than expected!
        Great feature! Thanks for the article!

  10. laolux

    I enabled compression and ran the defrag command to compress all my data. Worked great until … I ran out of disk space! This surprised me, as compression should reduce disk usage.
    Solution was simple though. I had a bunch of subvols which were snapshots at different points in time. Those snapshots were of course not touched by defrag and contained the old, uncompressed data. Together with the new and compressed data my disk was running out of space. So I deleted my old snapshots and defrag could continue. Now I am enjoying the transparent compression of btrfs.

    • Dutchy

      Don’t run defrag on volumes with snapshots or reflinked files. Those linked blocks will be unlinked leading to lots of duplication with high potential for running out of space.
      Many many kernels ago defragmentation was snapshot aware, but that feature is unlikely to make a return anytime soon.
      The autodefrag mount option should still be OK I believe.

      • laolux

        Well, but I had to to get the benefits of compression.
        And there is a way to get the reflinks back:

        rmlint --types="duplicates" --config=sh:handler=clone [paths...]

        Taken from the btrfs wiki on deduplication, https://btrfs.wiki.kernel.org/index.php/Deduplication
        So we can defrag, just need to be a bit careful about what we are doing.

  11. zilexa

    The information is not complete and not entirely correct..
    The “compress” argument you add to fstab will let BTRFS check if a file can be compressed. This is a heuristic check and doesn’t work that we’ll: Most files that can be compressed are actually skipped.

    By using “compress-force=zstd:1” you force BTRFS to send all files through the zstd algorithm which does its very own proper high speed check and will compress much more files. It will still skip files that are already compressed by nature (zip, MP3, MP4 etc) or cannot be compressed. .

    Also, lzo is a faster algorithm than zstd:1. The compression ratio is not as great, but for your OS/system SSD it might make more sense to use lzo.
    For any other SSD, zstd:3 is recommended as it has much higher compression while speed is only slightly affected.
    For HDDs, zstd:5 or higher since this will actually increase speed since less data has to be written to disk.
    For a backup HDD I use zstd:8, above that (it goes all the way up to 15), compression ratio doesn’t change much while CPU load does increase.

  12. Erkki Ruohtula

    Compressed file system is unfortunately an idea that is nowadays far less useful than it used to be, back when disk sizes were counted in Mb. In a way, it came too late to Linux. Two reasons:
    1: Disk are big nowadays. Really running out of space happens more rarely, and adding space is cheap.
    2 (and actually the more relevant reason): Practically all truly space-consuming file formats, like video, audio and image files already employ compression. Compressing an already compressed file does not make it any smaller (you may save a tiny bit from compressing uncompressed header data, but that is insignificant). Even many office document formats like ODT and DOCX files are already compressed. The .git directory contents (which are a space-consuming item for developers) also employ compression.

    • Dutchy

      Yes, it is true we have lots of space nowadays and that most heavy stuff is already compressed.
      But file system level compression can still mean quite a difference.
      For instance on VMs, one can save gigabytes of space there. (My desktop saves about 5GB through btrfs compression. )
      Next is audio mixing, for pcm/wav it generally means a 50% size reduction, even more with lots of silence, and the new audacity project format doesn’t appear to use compression yet, but why bother zipping it since btrfs handles it automatically.
      Lots of software can now be much simpler because the file system will take care of the wasted empty space.
      Also, even though storage is cheap, cheap storage is not fast, so speeding up reading from HDDs via compression can lead to some nice performance boosts.

      • Erkki Ruohtula

        I agree VM images benefit, mainly because they often contain representations of unused space. But in my experience, the general-purpose compression algorithms such as used in btrfs do not perform so well for uncompressed audio data, the effect I see is more like just 20% off. This probably depends on the kind of sound, and how much noise it contains, so your files may compress better.

        • Dutchy

          Yes, for uncompressed music it probably won’t make a difference at all.
          But for generic audio, especially multi-track audio, that has the potential for moments of total silence it really can make a pretty big difference. pcm/wav means 1536 kbps at 48 kHz which results in a ~700 MB file per hour. With multiple tracks per recording the size quickly explodes. There still are recording applications that don’t convert into a sanely compressed formats on the fly.
          I reckon this is a bit of an edge case, but at the end of the day there are lots of these little edge cases and it really can add up.
          For VMs, it is not about the empty sectors, those usually aren’t written to disk and the underlying btrfs volume should have nodatacow set for those files so no compression on that level. It really is about all those little system files inside the VMs where a few gigabytes can be saved per install. At least that is what compsize is telling me.
          (Of course also base disk and reflinks and other trickery could also be used to reduce overall storage usage for largely identical VMs.)

  13. MrHeffersmith

    Really cool but two questions:

    What I find weird is that the file system lists only the uncompressed size. That means that e.g. my 1TB SSD will eventually show up as 1TB+x when I get close to filling it up. Compressed it is under 1TB but uncompressed it would show up as larger. Does this not cause confusion? It for sure will for the user. What is e.g. the system monitor saying? It’s nice that it is transparent but I think it should still show the actual used up disk space and not some hypothetical.
    You mentioned that this “it is also straightforward to enable before and after an upgrade from Fedora 33” does this mean that when upgrading to F34, one is just asked about this and just has to accept or is the process the same (e.g. editing fstab and refragment manually)?

    Thank you for these great articles!

    • You will need to run the same process after upgrading to F34 as well. Fedora generally doesn’t like automating filesystem changes, which is part of why I wrote this article.

      The size measure thing is a little more complex. It really depends on the tool, and they can be a little inconsistent. I encourage you to use

      compsize

      for the most accurate results.

    • Erkki Ruohtula

      I understand the reported size and available size from tools like “du” do not take compression into account. So the available size shown is the “real” size, not a hypothetical. Taking compression into account would be infeasible, because not all files compress well, and a btrfs file system can actually contain a mix of compressed and uncompressed files, and the compressed files may even be compressed with different algorithms (depending on the setting in force when they were written). See the discussion about compression on btrfs wiki https://btrfs.wiki.kernel.org/index.php/Compression
      (I have so far not yet used btrfs on Fedora, but I have played with it on an OpenSUSE VM).

  14. tokariew

    Btrfs have problems with detecting what files can be compressed.
    For me compress-force=zstd:1 get better compression ratio than compress=zstd:3 for whole secondary disk. This will not force all files to be compressed, but zstd will decide which file to compress, and which file to leave uncompressed.

  15. I enabled this on my system a week back. If anybody’s curious, here’s my compsize for my home folder:

    Processed 1949123 files, 2096018 regular extents (2104182 refs), 1025558 inline.
    Type Perc Disk Usage Uncompressed Referenced
    TOTAL 76% 285G 372G 371G
    none 100% 245G 245G 244G
    zstd 31% 40G 126G 126G
    prealloc 100% 72K 72K 188K

    I interpret this as 24% savings in storage space — nearly 100GB. No perceivable slowdown. Very cool.

  16. Mary Biggs

    I’ve been using ext4 on multiple machines for quite a long time….without problems.
    Can someone explain what the downsides are for ext4 compared with btrfs?
    If there are few (or no) downsides, why would conversion to btrfs help me?

    • Simon Fink

      I would not directly call it downsides, using ext4 as a filesystem is still fine. It is a mature and stable filesystem. However, BTRFS has some shiny new features which ext4 hasn’t. Transparent compression is one of them. Advantage is, that the same set of files will most likely take less disk space on BTRFS compared to ext4. Another pretty cool feature coming with BTRFS is Copy-On-Write, which lets you create snapshots and incremental backups in a very efficient way (time and disk space wise). See this article (https://fedoramagazine.org/btrfs-snapshots-backup-incremental) and the official documentation (https://btrfs.wiki.kernel.org/index.php/Main_Page) of BTRFS for further details.

  17. I wouldn’t call it downsides to ext4. It is perfectly fine. BTRFS just offers many more features. I couldn’t go into all of them here, but Copy on Write, data check summing, easy snapshots, and transparent compression are just a few.

    These might be good places to start:
    https://fedoramagazine.org/btrfs-coming-to-fedora-33/
    https://fedoramagazine.org/choose-between-btrfs-and-lvm-ext4/

  18. Great article! However, one thing I don’t quite understand is this: as compression is transparent, tools like du will show the uncompressed size of files, i.e. a higher usage than actually true. Won’t this also stop programs from writing to the file system once they find it’s seemingly full? And will be shown if the uncompressed size exceeds the actual disk size? Will the disk use be like 110 % ?

    • David

      I would think that the transparent compression would give you more space, so du/df should show your disk/partition as being larger that it physically is.

  19. Lukas

    I think compressing /var is not a good idea in general. I simply compressed /usr and /home and that is just fine.

  20. Alex

    How it deals with parts of downloading torrents?

  21. Anon Ymous the 5th

    Compression is what compression does as Forest Compression once said in the international compression summit. In about 1990 compression was cool, but nowadays for the vast majority of use cases compression is less important than stability and reliability. My nvme drive is the size and thickness of a dime. It holds 500gb of data. The upcoming takeover of usb4 will also make drives much easier to deal with and work with. Thankfully, the days of paying about 3000 for a computer that could hardly fit a song library are done. Compression is not a pressing issue for must use cases. In sharp contrast, stability and reliability will always be a priority for most users.

  22. Alex

    How to switch this feature off properly when I have got fresh installed Workstation and some files was already compressed during the installation and the first launch?

    Is there any way to disable it before installation from the Everything iso for example?

    • Alex

      Nice! There is no such way!

      “To disable compression perform the following command:

      btrfs property set /usr compression “”

      Unfortunately, at the time of writing it is not possible to force decompression of a directory (or files), there is no defragment command to do this. If you really need to do this, create a script which reads and writes all files but be careful.”

      https://lukas.zapletalovi.com/2021/02/use-btrfs-compression-in-fedora-33.html

  23. mark

    I like FEDORA. I have used it since FEDORA 6. However I do not like this change. So, how do I disable this compression feature permanently? So that it never ever even in future releases turns on without my direct consent?
    I do not need the space saving. Disk space is cheap.

  24. Xiaoliang

    Has this been tested with luks encryption? I mean is it still quick enough and good to ssd lifespan?

  25. Dave

    i have tested it on my system:

    compsize -x / /home/

    Processed 550906 files, 424624 regular extents (466220 refs), 229669 inline.
    Type Perc Disk Usage Uncompressed Referenced
    TOTAL 97% 419G 430G 434G
    none 100% 412G 412G 414G
    zstd 38% 6.9G 17G 20G
    prealloc 100% 7.0M 7.0M 5.0M

    if i read the above lines correctly, i saved 15G (~3%)

    • Yes, that’s not very much savings. Did you do the “retroactively compress old files” step? On my

      /home

      , I get 77%, so about 30% compression. You can see that most of your files are under “none”, which means that either they were written before compression was turned on (and you didn’t do the retroactive step) or that Btrfs decided that they’re not compressible and bailed (like if they are mostly media files which are already compressed).

  26. mutant

    I see all files in my btrfs on LUKS shows 0,0 % compression.
    100% of these files are copied in after creating and mounting the btrfs on LUKS having “compress=zlib” enabled in /etc/fstab.
    Is this due to the underlying LUKS?

  27. Damian Ivanov

    What I think would be ingteresting if Fedora had more secure boot signed kernels with optional features like Btrfs online deduplication. I really would like to test this…

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions