Since Boltron is out and the team is working hard on Fedora 27 modular server, we could start talking more about what it all means. One of the aspects our team is interested in is utilizing modules in container images. If the world of Modularity is still new to you, don’t worry, we have you covered! I advise you to read the Boltron announcement and dive deep into Fedora Modularity website. As for containers, they keep being great at packaging applications with its dependencies and basic runtime environment in a single “box” which is easy to operate.
You might be wondering why anyone would include modules inside container images. There are some very practical reasons that make modules a good fit for containers.
1. You know what you are getting.
With traditional distribution, it’s hard to predict the version of software being installed. Let’s do an example. Do you know what version of nodejs you’ll get when you install it inside
container image?
$ dnf install nodejs
It may be version 6, it may be version 8, who knows? You could check koji, bodhi or dist-git to find out, but none of those would give you the answer. Only running the command would tell.
So how will modules help here? Let’s proceed to the next point.
2. You can pick the version you want.
Modules have a concept of a stream. Streams are defined by packagers where they clearly state what the compatibility status of the stream is. Some may be strict which will require all updates within the stream to be API and ABI compatible, others may just provide latest upstream version with no guarantees. The strict streams are usually tied to upstream major releases. When installing a module, you can select a stream to be installed. In the example below, I chose stream 8 for nodejs, which is tied to upstream version 8.
$ dnf install @nodejs:8 ====================================================== Package Arch Version ====================================================== Installing group packages: nodejs x86_64 1:8.0.0-1.module_42d8f2a0
Since the stream guarantees compatibility, updating nodejs package within that stream ensures that you will always be getting version 8.
This will also solve one of the problems we have when building container images in Fedora. There is a guideline that you should not set version label, because no one knows what version of the package is going to be installed. We concluded that it’s better to ignore the label rather than setting it wrong. With modules, we’ll select a precise stream and we’ll be sure to get a version we picked.
3. Compatibility under control.
Let’s get back to the first point.
What if your software stack works only with NodeJS 6 and the command
would install version 8. That would be a problem, right?
With streams, you will be able to select one which guarantees that you will always get an API/ABI compatible packages within that stream. If you need stability, pick a stream with strict compatibility guarantees — no surprises when installing packages anymore. So predictable, such happiness!
4. A module may come optimized for the container use case.
Back in the day we defined an install profile named “container”. Install profile contains a set of packages which are meant to be installed. The “container” install profile should contain a list of RPM packages which are meant to be installed into containers.
So in dnf, all you need to do is just:
$ dnf install @nodejs:8/container
and you should be getting the right packages. No need to spend time figuring out what those should be. And obviously, you can always pick the ones you really need.
5. The installation is simple and your muscle memory works.
Modules don’t need a new package manager. You can easily install them using dnf. Optionally, if you want better experience, you can use the functionality supporting modularity which was added into dnf.
6. In the end, you are getting ordinary RPMs.
Modularity is not a new packaging technology. When you’re installing modules, in reality, you are getting RPMs. So, don’t fear that you need to learn a new packaging format.
7. Automation!
This is my favorite one. Once the Factory 2 team reaches their goals, we’ll have a complete build pipeline in-place. This means that to build an updated container-image, all you need to do, is to update an RPM spec file, commit the changes and push them into Fedora dist-git. You heard me.
The pipeline should then trigger the build of the RPM, which should trigger a build of the associated modules, which should trigger a build of container images which utilize the modules. On top of it, every step will be gated by CI.
We’re not quite there yet, but we’re right on track.
With this kind of automation, rebuilding container images for sake of fixing CVEs should be much easier. Benefit for users is to get updated container images within hours.
8. Rebuilds that work.
Imagine this: there is a new CVE with a fancy name. Fedora rebuilds all container images to pick up the fix from a base image based on the automation work described in previous point. And boom! A bunch packages were updated which turned into breaking functionality of the particular containers. That’s not we wanted, right?
This scenario should not happen with modules. As I explained when talking about streams, when you select a stream with strict compatibility guarantees, you should be getting functional, compatible packages forever.
9. The simplicity.
If you ever played with Software Collections, or even tried to create a container image based on an existing collection, you might got to a point when it was too complicated:
- RPMs installed into a separate tree in
/opt
- the need to
scl enable
a collection
- and for sure, a bunch of hacks
Our goal is to make those pain points history. Why? Modules are installed directly in your root filesystem, you don’t need dedicated tooling to use the modules and… no hacks!
Oh, but your requirement is to have three database versions installed in parallel — no problem. Just get a container image for every version and you are good to go. Modularity doesn’t provide a solution for parallel installation, it provides multiple versions for a single package.
Are you so hyped for modules as I am?
Tomasz
I… don’t care? Being Fedora, I expect latest and greatest. I also expect rest of software in distribution working with the version being shipped – that what distributions are for, and we have dependencies in RPM for ensuring that.
That’s what I’m getting within released version of Fedora. If there is an update breaking API/ABI, then this is packaging error and will be rolled back.
API/ABI breaks can only occur between Fedora versions.
Tomas Tomecek
It’s good for you that you don’t care, but there is plenty of people who care about specific versions: if their application has hard dependency on a specific version, for example it’s written in python 2, and the installation would bring python 3, that would be a real problem and the application would not work.
dephekt
Sometimes you need a specific or different version of a piece of software which may include a dependency for a library with a hard version requirement that isn’t available in your distribution’s release channel. Usually this would end up with unresolvable dependencies and you’d have to manually get the libraries you need and install them in a way that doesn’t break other apps which were depending on the newer version offered by the package manager.
This sort of situation gets really messy fast and developers have to deal with it all the time. Entire separate projects were made just to help Ruby and Python developers keep their environments separated for similar reasons.
V
Hello
tips for fedora modularity server guide
the first steps are missing ( sudo dnf install docker , sudo systemctl enable docker , sudo systemctl start docker ) and all commands until the #bash need root or sudo
now i am testing the docker inside a KVM VM . nested VMs …. cool 🙂
First Noob questions
who is maintaining these docker images – fedora team ?
how we can list the available versions of a module-server ? example . i want a specific php version . how can i list all the available php servers
and
if there is not included my desired module-server is there any way to create and install it ? for example i want to install redis server . i saw that there is not any redis server available . what we are doing in these situations ?
Tomas Tomecek
If the images come from registry.fedoraproject.org registry, they are being maintained by Fedora maintainers. We had a set of images prepared at our organization in Docker Hub because we didn’t manage to build them inside Fedora infrastructure in time for Boltron.
Unfortunately, there is no php module yet:
https://src.fedoraproject.org/browse/projects/modules/php
We have this container image, meanwhile:
https://src.fedoraproject.org/container/php
If you wish to have a php module with different versions, I advise you to open a bug for php RPM component and ask the maintainer to provide such module — it’s trivial to utilize it in a container image afterwards. The same process applies for redis.
You can browse module sources over here (and see which were already created — it doesn’t necessarily mean they were also built and are being kept up to date): https://src.fedoraproject.org/projects/modules/*
Unfortunately, there is no simple interface to browse available built modules — we hope that bodhi will be able to do that in future.
V
Thank you very much Tomas for the detailed answer !
from your answer i understood that there are still some limitations in modular-server but it can improved in time while this will be more popular or if we demand specific additions via redhat bug report system
personal i would like had an option to create my specific modular-server and upload it somewhere for future personal use or to help others using it (helping others is always a good motivation to spent time on something 🙂 )
I am reading now the guide to create a module for local use from https://docs.pagure.org/modularity/development/building-modules/developing.html and https://fedoraproject.org/wiki/Module:Guidelines and i already have some headaches 🙂
Sébastien Wilmet
So basically two different streams of the same package cannot be installed in parallel in the same container. Is that correct?
What the article doesn’t explain is whether a stream is independent of the Fedora version. For example will the @nodejs:8 stream be the same in F27 and F28? Are they built from the same *.spec file? Or does the packaging effort needs to be duplicated for different Fedora versions, plus duplicated a second time for the different streams?
If we see the big picture here, you are still basically doing a lot of RPM packaging, downstream. The packaging effort is still duplicated in each Linux distro, that’s a big waste of human resources for Free Software IMHO. What looks to me much more useful are systems like Flatpak: a flatpak is packaged (ideally) by the upstream developer, and it is completely independent of any distro, to build it it doesn’t rely on RPM or DEB packages, it is something separate, upstream.
Another very interesting thing developed in GNOME is BuildStream [1] (but still in development). With BuildStream, the build instructions (for example what would be present in a *.spec file) are described in a declarative format that is independent of the output format, the two are decoupled. So with the same build instructions, it is for example possible to create a Flatpak, a RPM or a container, a VM, a bootable iso, etc.
So in short, it would be nice to have a system like Flatpak, but for servers (because Flatpak is for desktop apps).
[1] https://wiki.gnome.org/Projects/BuildStream/
Tomas Tomecek
That’s correct.
The stream is completely independent from the Fedora version. Basically, the maintainer of the module needs to decide which stream is suppose to be included in specific Fedora version.
Yes, they are. Essentially, the stream is a branch in dist-git.