Building RHEL and RHEL UBI images with mkosi

Photo by Saad Salim on Unsplash

Mkosi is a lightweight tool to build images from distribution packages. This article describes how to use mkosi to build images with packages from RHEL (Red Hat Enterprise Linux) and RHEL UBI. RHEL Universal Base Image is a subset of RHEL that is freely available without a subscription.

Mkosi features

Mkosi supports a few output formats, but the most important one is Discoverable Disk Images (DDIs). The same DDI can be used to boot a container, as a virtual machine, copied to a USB stick and used to boot a real machine, and finally copied from the USB stick to the disk to boot from it. The image has a standarized layout and metadata that describes its purpose.

Mkosi relies on other tools to do most of the work: systemd-repart to create partitions on a disk image, mkfs.btrfs/mkfs.ext4/mkfs.xfs/… to write the file systems, and dnf/apt/pacman/zypper to download and unpack packages.

Mkosi supports a range of distributions: Debian and Ubuntu, Arch Linux, OpenSUSE, and of course Fedora, CentOS Stream and derivatives, and now RHEL UBI and RHEL since the last release. Because the actual “heavy lifting” is done by other tools, mkosi can do cross builds. This is where one distro is used to build images for various other distros. The only requirement is that the appropriate tools are installed on the host. Fedora has native packages for apt, pacman, and zypper, so it provides a good base to use mkosi to build any other distribution.

There are some nifty features: images can be created by an unprivileged user, or in a container without device files, in particular access to loopback devices. It can also launch those images as VMs (using qemu) without privileges.

The configuration is declarative and very easy to create. systemd-repart is used to create disk partitions, and repart.d configuration files are used to define how this should be done.

For more details see two talks by Daan DeMeyer at the All Systems Go conference: systemd-repart: Building Discoverable Disk Images and mkosi: Building Bespoke Operating System Images.

Project goal

One goal for Mkosi is to allow the testing of a software project against various distributions. It will create an image for a distribution (using packages from that distribution) and then compile and install the software project into that image, inserting additional files that are not part of a package. But the first stage, the creation of an image from packages, is useful on its own. This is what we will show first.

We1 recently added support for RHEL and RHEL UBI. Let’s start with RHEL UBI, just building an image out of distro packages.

Please note that the examples below require mkosi 19, and will not work with earlier versions.

A basic RHEL UBI image with a shell

$ mkdir -p mkosi.cache
$ mkosi \
  -d rhel-ubi \
  -t directory \
  -p bash,coreutils,util-linux,systemd,rpm \

The commands above specify the distribution ‘rhel-ubi’, the output format ‘directory’, and request that packages bash, coreutils, …, rpm be installed. rpm isn’t normally needed inside of the image, but here it will be useful for introspection. We also enable automatic login as the root user.

Before the build is started, we create the cache directory mkosi.cache. When a cache directory is present Mkosi uses it automatically to persist downloaded rpms. This will make subsequent invocations on the same package set much faster.

We can then boot this image as a container using systemd-nspawn:

$ sudo mkosi \
  -d rhel-ubi \
  -t directory \
Detected virtualization systemd-nspawn.
Detected architecture x86-64.
Detected first boot.

Red Hat Enterprise Linux 9.2 (Plow)
[ OK ] Created slice Slice /system/getty.
[ OK ] Created slice Slice /system/modprobe.
[ OK ] Created slice User and Session Slice.
[ OK ] Started User Login Management.
[ OK ] Reached target Multi-User System.

Red Hat Enterprise Linux 9.2 (Plow)
Kernel 6.5.6-300.fc39.x86_64 on an x86_64

image login: root (automatic login)

[root@image ~]# rpm -q rpm systemd

As mentioned earlier, the image can be used to boot a VM. In this setup, it is not possible — our image doesn’t have a kernel. In fact, RHEL UBI doesn’t provide a kernel at all, so we can’t use it to boot (in a VM or on bare metal).

Creating an image

I also promised an image, but so far we only have a directory. Let’s actually create an image:

$ mkosi \
  -d rhel-ubi \
  -t disk \
  -p bash,coreutils,util-linux,systemd,rpm \

This produces image.raw, an disk image with a GPT partition table, and a single root partition (for the native architecture).

$ sudo systemd-dissect image.raw
Name: image.raw
Size: 301.0M
Sec. Size: 512
Arch.: x86-64

Image UUID: dcbd6499-409e-4b62-b251-e0dd15e446d5
OS Release: NAME=Red Hat Enterprise Linux
VERSION=9.2 (Plow)
PRETTY_NAME=Red Hat Enterprise Linux 9.2 (Plow)
REDHAT_BUGZILLA_PRODUCT=Red Hat Enterprise Linux 9

Use As: ✗ bootable system for UEFI
        ✓ bootable system for container
        ✗ portable service
        ✗ initrd
        ✗ sysext extension for system
        ✗ sysext extension for initrd
        ✗ sysext extension for portable service

rw root 1236e211-4729-4561-a6fc-9ef8f18b828f root-x86-64 xfs x86-64 no yes /dev/loop0p1 1

OK, we have an image, the image has some content from RHEL UBI packages. How do we add our own stuff on top?

Extending an image with your own files

There are a few ways to extend the image, including compiling something from scratch. But first let’s do something easier and inject a provided file system into the image:

$ mkdir -p mkosi.extra/srv/www/content
$ cat >mkosi.extra/srv/www/content/index.html <<'EOF'
<h1>Hello, World!</h1>

The image will now have /srv/www/content/index.html.

This method is used to inject additional configuration or simple programs.

Building from source

Now let’s do the full monty and build something from sources. For example, a trivial meson project with a single C file:

$ cat >hello.c <<'EOF'
#include <stdio.h>

int main(int argc, char **argv) {
    char buf[1024];

    FILE *f = fopen("/srv/www/content/index.html", "re");
    size_t n = fread(buf, 1, sizeof buf, f);

    fwrite(buf, 1, n, stdout);
    return 0;

$ cat > <<'EOF'
project('hello', 'c')
executable('hello', 'hello.c',
           install: true)
$ cat > <<'EOF'
set -ex

mkosi-as-caller rm -rf "$BUILDDIR/build"
mkosi-as-caller meson setup "$BUILDDIR/build" "$SRCDIR"
mkosi-as-caller meson compile -C "$BUILDDIR/build"
meson install -C "$BUILDDIR/build" --no-rebuild
$ chmod +x

To summarize: we have some source code (hello.c), a build system configuration (, and a glue script ( that is to be invoked by mkosi. For a “real” project, we would have the same elements, just more complex.

The script requires some explanation. Mkosi uses user namespaces when creating the image. This allows the package managers (e.g. dnf) to install files owned by different users even though it is invoked by a normal unprivileged user. We are using mkosi-as-caller to switch back to the calling user to do the compilation. This way, the files created during compilation under $BUILDDIR will be owned by the calling user.

Now let’s build the image with our program. Compared to the previous invocation, we need additional packages: meson, gcc. Since we now have a build script, mkosi will execute two build stages: first an build image is built, and the build script is invoked in it, and the installation artifacts are stashed in a temporary directory, then a final image is built, and the installation artifacts are injected. (Mkosi sets $DESTDIR, and meson install uses $DESTDIR automatically, so the right things happen without us having to specify things explicitly.)

$ mkosi \
  -d rhel-ubi \
  -t disk \
  -p bash,coreutils,util-linux,systemd,rpm \
  --autologin \
  --build-package=meson,gcc \
  --build-dir=mkosi.builddir \ \

At this point we have the image image.raw with a custom payload. We can start our freshly minted executable as a shell command:

$ sudo mkosi -d rhel-ubi -t directory shell hello
<h1>Hello, World!</h1>

Obtaining a developer subscription for RHEL

RHEL UBI is intended for use primarily as a base layer for container builds. It has a limited set of packages available (about 1500). Let’s now switch to a full RHEL installation.

The easiest way to get access to RHEL is with a developer license. It provides an entitlement to register 16 physical or virtual nodes running RHEL, with self-service support.

First, create an account. After that, head over to management and make sure “Simple content access for Red Hat Subscription Management” is enabled. Then, create a new activation key with “Red Hat Developer Subscription for Individuals” selected. Make note of the Organization ID that is shown. We’ll refer to the key name and organization ID as $KEY_NAME and $ORGANIZATION_ID below.

Now we are ready to consume RHEL content:

$ sudo dnf install subscription-manager
$ sudo subscription-manager register \
  --org $ORGANIZATION_ID --activationkey $KEY_NAME

Building an image using RHEL

In previous examples, we specified all configuration through parameter switches. This is nice for quick development, but can become unwieldy. RHEL is a serious distribution, so let’s use a configuration file instead:

$ cat >mkosi.conf <<'EOF'




Let’s first check if everything is kosher:

$ mkosi summary

And now let’s build the image (err, directory):

$ mkosi build
$ mkosi qemu
Welcome to Red Hat Enterprise Linux 9.2 (Plow)!

[ OK ] Created slice Slice /system/modprobe.
[ OK ] Reached target Initrd Root Device.
[ OK ] Reached target Initrd /usr File System.
[ OK ] Reached target Local Integrity Protected Volumes.
[ OK ] Reached target Local File Systems.
[ OK ] Reached target Path Units.
[ OK ] Reached target Remote Encrypted Volumes.
[ OK ] Reached target Remote Verity Protected Volumes.
[ OK ] Reached target Slice Units.
[ OK ] Reached target Swaps.
[ OK ] Listening on Journal Socket.
[ OK ] Listening on udev Control Socket.
[ OK ] Listening on udev Kernel Socket.
Red Hat Enterprise Linux 9.2 (Plow)
Kernel 5.14.0-284.30.1.el9_2.x86_64 on an x86_64

localhost login: root (automatic login)
[root@localhost ~]#

Yes, we built the “image” as a directory with a file system tree, and booted it as a virtual machine.

In the booted virtual machine, findmnt / shows that the root file systems is virtiofs. This is a virtual file system that exposes a directory from the host to the guest. We could build a more traditional image with a partition table and file systems inside of a file, but a directory+virtiofs is quick and nicer for development.

The image that we just booted is not registered. To allow updates to be downloaded from inside of the image, we would have to add yum, subscription-manager, and NetworkManager to the package list, and before we download any updates, call subscription-manager in the same way as above. Once we do that, we have about 4500 packages at our disposal in the basic repositories, and a few dozen additional repositories with more specialized packages.


And that’s all I have for today. If you have questions, find us on Matrix at or on the systemd mailing list.

  1. Daan DeMeyer, Lukáš Nykrýn, Michal Sekletár, Zbigiew Jędrzejewski-Szmek ↩︎

Fedora Project community


  1. You Really Don't Want to Know

    This is the best tech article I have seen here … or anywhere else, come to that.

  2. Peter Boy

    Very interesting! When I looked into Mkosi several years ago, I got the impression that it might have lost its momentum. But that is clearly not (or no longer) the case.

    I would be very happy if you could contribute a (small or not so small) step-by-step guide to the nspawn section of our Fedora Server Edition documentation, especially about creating foreign distributions (from Fedora’s point of view) nspawn-containers and probably VMs. Would be really great!

  3. Klaas

    How does this compare to osbuild ?

    • Zbigniew Jędrzejewski-Szmek

      This is a complex issue. Some obvious differences:
      – mkosi is tool to be invoked locally, and is essentially a glorified python script. OsBuild is a beast: it a frontend, a backend, a composer, and a worker fleet.
      – both mkosi and OsBuild allow selection of packages, so they seem to be quite similar in this regard. In mkosi, the list of “packages” is passed to the tool, so actually the arguments can be package names, file paths, or nevra strings, or virtual provides or “rpm rich dependencies”, groups, i.e. anything that is supported by the underlying package manager. OsBuild seems to be less flexible here.
      – OsBuild has custom configuration language for most things: packages, groups, firewall customizations, service enablement, etc. OTOH, mkosi doesn’t have any understanding of those things: the user would generally be expected to just provide the right configuration file themselves. For example, drop in a config file snippet in mkosi.extra directory, and then mkosi will push it to the image. This is actually nice, because it’s more flexible, and the user doesn’t have to learn yet-another method to configure things. Of course, this works nicely for systems which are designed to be configured in this way. All systemd components support drop-in files and very simple text config, so this works out nicely. But for some other software this might not be the right approach.
      – OsBuild has native understanding of containers and is designed to work in this model. Mkosi is more traditional: some local config results in a disk image or a tarball, that’s it.
      – Mkosi now does most things without requiring any privileges. I’m not sure about OsBuild.
      – With mkosi, it’s fairly easy to customize the output very very precisely. For example, we build initrd images via mkosi.
      – Mkosi has native understanding of dm-verity and signatures, and will produce images with signatures. In general, it is intended to make use of systemd features to the maximum possible extent.

      With mkosi the focus is on local workflows and flexibility and experimentation with systemd features, while with OsBuild the focus is on reliable builds in a cloud environment. A lot of this is just philosophy and both systems are flexible, so it’s possible that with enough determination, both would allow many different things to be done.

      It’d be great if somebody who has experience with mkosi, osbuild, and kiwi, would create an in-depth comparison.

  4. Dirk Gottschalk

    Hello. Really nice article.

    I tried mkosi on my system and everytime I try to run it as regular user I get a “permission denied” error. Do I have to add a group membership or something like that?

    My Environment: Fedora Workstation 39, login with FreeIPA 4.11

    • Zbigniew Jędrzejewski-Szmek

      Older mkosi versions (<=14) required root privileges, but Fedora 39 has version 19. Newer versions are mostly supposed to run as an unprivileged user. The only operation that requires root privileges is mkosi shell or mkosi boot, because that uses systemd-nspawn.

      It’d help if you showed the exact package version, the command that is failing, and the error message.

      • Dirk Gottschalk

        I am using mkosi version 19.

        I get this error message:
        mkosi \
        -d rhel-ubi \
        -t directory \
        -p bash,coreutils,util-linux,systemd,rpm \
        Traceback (most recent call last):
        File “/usr/bin/mkosi”, line 8, in
        File “/usr/lib64/python3.12/”, line 81, in inner
        return func(*args, **kwds)
        File “/usr/lib/python3.12/site-packages/mkosi/”, line 52, in main
        run_verb(args, images)
        File “/usr/lib/python3.12/site-packages/mkosi/”, line 2801, in run_verb
        if not needs_build(args, config) and args.verb != Verb.clean:
        File “/usr/lib/python3.12/site-packages/mkosi/”, line 2582, in needs_build
        (args.force > 0 or not (config.output_dir_or_cwd() / config.output_with_compression).exists())
        File “/usr/lib64/python3.12/”, line 861, in exists
        File “/usr/lib64/python3.12/”, line 841, in stat
        return os.stat(self, follow_symlinks=follow_symlinks)
        PermissionError: [Errno 13] Permission denied: ‘/home/dgottschalk/osi/image’

        • Zbigniew Jędrzejewski-Szmek

          PermissionError: [Errno 13] Permission denied: ‘/home/dgottschalk/osi/image’

          It looks like it fails when trying to access this file. You probably have a left-over files owned by root (or some other user).

  5. Hey Zbigniew, great walkthrough on building RHEL and RHEL UBI images with mkosi! Love the flexibility mkosi offers, especially for cross builds. Excited to see the impact on software testing across different distros!

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions