Koschei is a continuous integration service for RPM packages. It helps developers fix bugs as fast as possible. It tracks package dependency changes in Rawhide, the bleeding-edge, development version of Fedora. Packages whose dependencies change too much are rebuilt. Koschei logs these rebuilds, displaying dependency changes and current state.

It tries to detect packages that fail to scratch-build from source (FTBFS) in Rawhide. (A scratch build is a short-term, temporary build Fedora doesn’t ship to users.) To do this, it builds packages from the latest available source in Koji, the Fedora build system. However, you don’t need a special Koji client for this purpose.

We talked with Red Hat software engineer Mikolaj Izdebski to learn more about Koschei.

What problem was this project created to solve?

At the beginning, Koschei was created to address the problems my team had with package maintenance. In 2012, we had only two people maintaining over 300 packages in Fedora and Red Hat Enterprise Linux 7. These packages were strongly connected; everything depended on everything else. Each time we changed something, there was a high risk of introducing a bug.

To address this problem, I came up with a proof of concept: a script to rebuild a few packages every night. When we came back in the morning after a day of work, we would have an email report if something went wrong or not. It worked well, we showed it to other teams and they liked it, so I figured this project could become a matter of general interest. That’s how it started.

How has the project evolved over time?

In 2012, the idea and a proof of concept came about — a very simple script.

In 2013, we expanded to other teams.

In 2014, we started a rewrite from scratch and redesigned the architecture.

In 2015, it was pushed to production, and became an officially supported service in Fedora infrastructure.

In 2016, we added support for tracking multiple Fedora releases.

Over time, we overcame a problem with hardware resources. Koschei is rebuilding a lot of packages, so we decided to use Koji because it has a lot of spare computing power that is normally unused. We didn’t want to introduce a new solution. We wanted to use an existing system.

What can your project do now?

As of now, the way it works is you go to the web interface, add your package to a package database, and subscribe to Koschei notifications. It is currently supported in Fedora only. Users of Koschei are expected to watch notifications, or go the web interface and look at something in red if they feel like fixing some bugs.

Since it is a very early testing tool, the bugs Koschei discovers are supposed to be fixed by developers before the packages reach QE. While it is not a useful tool for QE themselves, it does contribute to software quality overall.

Koschei serves several groups of people. One is package maintainers in Fedora and RHEL, to notify them of bugs in their packages so they can fix them. The others are upstream projects (in cooperation with Fedora) who can test upstream releases before they happen. For example, when a new release candidate comes out, Fedora maintainers can update their package to upstream release candidates, test this latest version via Koschei in Fedora on a large set of packages, and provide feedback to the upstream.

Koschei can work with any RPM-based distribution. Whether you’re involved in CentOS or openSUSE, you’re good to go. Even systems like Mageia could theoretically use Koschei, but if they don’t, they can still benefit from it. I get requests from other distros to add packages to Koschei. Why? When a package doesn’t build in Mageia, for example, Mageia developers would go to Koschei to see if it fails on Fedora too. If it does, it’s a common bug between these two distros and we can work together to fix them. If it works in Fedora, it’s a bug specific to them.

What’s next for the project?

We’d like to extend the usage of Koschei in Fedora. Currently, only one-third of Fedora’s packages are enabled, mostly due to lack of awareness about Koschei. The goal is to eventually enable all packages by default. People now have to explicitly subscribe for notifications, but we would like for every Fedora package maintainer to be notified about their package by default.

Last but very useful and more long term idea: allowing testing of scratch builds. Currently, to test anything in Koschei, people have to make a real build and in some cases, it’s not suitable. Some use cases for this are testing new releases (manually, or automatically via upstream release monitoring), testing breaking changes, etc.

If somebody reading could help you with one problem, what would it be?

Spread the word! Set up packages and tell others to do so. Tell success stories. There is no better reward for a developer than hearing how their product was useful, or how it saved time.

I hope you can join the group of teams who successfully use Koschei, including Eclipse, PHP, or Java. If you have any questions, ask them via email to mizdebsk@redhat.com. For adding packages or products, contact Michael Simacek or Mikolaj Izdebski.

Here are some additional references regarding Koschei: