Day in the life of a Fedora Packager

Ever wondered what it’s like being involved with the Fedora Project? There are many different roles and types of people that help make Fedora what it is. One common form of contributing is packaging. This is when someone takes software, “packages” it in the RPM format, and publishes the RPM to the Fedora repositories. There’s some steps along the way to being a packager. In this article, Fedora packager James Hogarth, responsible for ownCloud, Certbot (formerly LetsEncrypt), and more, details a day in the life of what it’s like being a Fedora Packager.

Packaging in Fedora

I’m often asked what role a packager plays and why it can take so long for an upstream release to appear as a package in Fedora or EPEL. Isn’t it just taking an existing tarball from some FOSS project and making an RPM out of it? In this article, we’ll step through what happens on an ownCloud update and the journey it takes to arrive in Fedora and EPEL.

Upstream monitoring

Fedora uses Anitya to track upstream releases. You can find Anitya in action at release-monitoring.org. At the time of writing, Fedora monitoring tracks a little over 10,000 packages.

When Anitya detects a new version upstream, a bug is logged on Bugzilla to alert the maintainer that there is a new version to check. This is the very start of our journey.

The first steps

In the case of ownCloud specifically, Composer is used to track PHP dependencies. Comparing the old and new

files gives a quick initial check of any PHP dependency changes we should be aware of.

Other projects will have different ways of providing the dependency requirements, even if it’s just a README file.

If it’s a minor update with little or no third-party changes that could affect bundling, we have the easiest path to follow. A quick version bump in the spec file and a test build will be enough.

$ rpmdev-bumpspec -n 9.0.3 -c "new release 9.0.3 upstream" owncloud.spec
$ spectool -g owncloud.spec
$ fedpkg srpm
$ mock -r fedora-rawhide-x86_64 ./owncloud-9.0.3-1.src.rpm

If there’s minimal changes (so patches still apply), there’s a good chance this will just work and

will successfully build the package. If it does, then testing the builds will be the next activity.

Bump in the road

If there are sufficient changes that the existing patches no longer cleanly apply, this will trip up the build.

has the capability to only run a given stage of the rpmbuild. In this case, we’re interested in the

stage, as patches need updating. From this we can then see how the patch has failed and what needs to be updated:

$ mock -r fedora-rawhide-x86_64 --short-circuit=prep ./owncloud-9.0.3-1.src.rpm
$ cd /var/lib/mock/fedora-rawhide-x86_64/root/builddir/build/BUILD/owncloud

Generally the quickest route I have found from here is to create a temporary git repo, apply patches or manually make changes, and then

to generate new patches that will apply clean.

Bringing these back to the original spec and generating a new SRPM puts us back on track fairly swiftly. Once the mock build is successful, we can test the builds.

Encountering “road work”

In Fedora (and EPEL as well), we have a strong preference to unbundling where possible. Bundling is when you include other software dependencies with your package. An example might be a web server that “bundles” in a version of OpenSSL with it, to handle HTTPS. Until recently, bundling required a specific Unbundling Exception from the Fedora Packaging Committee if there was a reason unbundling couldn’t happen.

Even with the removal of the strict requirement, it’s still preferred to unbundle. If a security issue happens with a dependency (think of Heartbleed with OpenSSL), the update will automatically be picked up by all applications using it, rather than that application needing its own update. It also ensures consistent behavior between applications that use a particular library.

When comparing the

for a PHP package, it reveals a change in the dependency requirements. Most often, this is a version bump due to a bug fix or a security issue. The first thing we need to check is what’s already in Fedora and/or EPEL. Due to our release monitoring outlined above, it’s likely a newer version of the library is already packaged. However, sometimes a package falls behind. This could be for a number of reasons, from the maintainer lacking time or a failure to build the new version for some reason.

In the event the updated dependency isn’t satisfied already, then Bugzilla and pkgdb need to be checked to see if an update is in progress. If not, then a bug needs to be logged to contact the other maintainer. If they are having issues, it can be nice to offer them a hand.

Occasionally, it’s noticed that a package has fallen by the wayside. We are, after all, volunteers. On occasion, life gets busy and the ability to carry out maintenance is compromised. In that situation, we have a specific process to allow the Fedora Engineering Steering Committee, FESCo, to pass the package on to someone else interested in it.

Once that new dependency is updated, it can be added to the build environment! We’re through the road work and back on our path to patches as a “bump in the road”.

Looking out for sinkholes

The biggest delay comes at major version updates. These will almost always involve rebasing patches, or possibly dropping them. This can happen if upstream merges or fixes an issue a patch dealt with. It can also include not just new versions of a library, but often recently-included libraries.

If there’s major updates of the dependencies, then some liaising and communication with other maintainers may be in order. You will need to see what an update might affect and then work with them to work in everyone’s best interest. This may mean holding back on your own package for a bit while another is updated for compatibility with the new library version.

If there’s a new library needed that wasn’t there before, then this could involve a new package in Fedora if it doesn’t already exist. This will need a package review. For a new package, it can take some time to get through, depending on the interest of the packages to other Fedora packagers in the community.

Once any new packages have passed review, are present in the Fedora repositories, and any existing dependencies are updated, we can finally get to a valid build and perform testing.

Testing the build

Depending on the complexity of the package, the amount of time and effort needed to test will vary.

In the case of some packages (such as

, formally known as

, or

) there’s a test suite that gets run as part of the

in the build. It’s relatively simple to test on an Amazon Web Services EC2 instance or on a personal server to verify behavior.

In the case of ownCloud, there’s a fair bit more of testing required. This is due to the various forms of components involved, testing as an upgrade, and testing as a fresh installation.

Upstream has test suites, but they are not on the release tarball. I haven’t had time to investigate if they could be incorporated in the rpm spec

. What we do

in the spec for is if there’s any new bundled libraries (as opposed to just bumped versions) and that we can automatically load all the PHP libraries correctly.

The bulk of the testing before pushing to repositories is manual. At the time of writing, my present test matrix for the upcoming ownCloud 9.0.2 packages consists of:

Distribution	Web Server	Database
Fedora Rawhide	Apache httpd	MariaDB
Fedora Rawhide	Apache httpd	PostgreSQL
Fedora Rawhide	Nginx	MariaDB
Fedora Rawhide	Nginx	PostgreSQL
Fedora 24	Apache httpd	MariaDB
Fedora 24	Apache httpd	PostgreSQL
Fedora 24	Nginx	MariaDB
Fedora 24	Nginx	PostgreSQL
Fedora 23	Apache httpd	MariaDB
Fedora 23	Apache httpd	PostgreSQL
Fedora 23	Nginx	MariaDB
Fedora 23	Nginx	PostgreSQL
CentOS 7	Apache httpd	MariaDB
CentOS 7	Apache httpd	PostgreSQL
CentOS 7	Nginx	MariaDB
CentOS 7	Nginx	PostgreSQL

I’m making the assumption I won’t be building for Fedora 22, based on expected timings for various bits. The 8.2 updates were tested on Fedora 22 as well.

Automating the testing

I’m sure you can picture what a burden that would be to manually install each of these environments to carry out testing. This is where using configuration management technologies such as Ansible against KVM guests come into play.

Making good use of the virt-install tools, it’s simple to script the generation of virtual machines (VMs) to test against. With the BTRFS reflink copying, this is almost instant and takes little disk space.

Next comes the Ansible playbooks. This makes use of a dynamic Ansible inventory Python script to lookup what VMs are running and assign them roles. The web server and database choices are based on the Ansible groups that the guest is in automatically. To easily switch between building, the Fedora repositories, and the local

builds, I have variables that can be set to enable the testing repos and a local repo.

 

Creating a fresh install of the new ownCloud package to test becomes as simple as running the Ansible playbook.

$ ./gen_oc_vms.sh
# wait for VMs to be accessible, this can be tested by checking ./libvirt_inv.py --list
ansible-playbook -i ./libvirt_inv.py -e "localrepo=true testingrepo=true" site.yml
# wait for playbook to complete
$ ./libvirt_inv.py --list | json_reformat  | awk -F':' '$1 ~ /ansible_host/ {print $2}' | sed 's/[", ]//g' | while read ocip; do xdg-open "http://${ocip}/owncloud/"; done

At this point, we’ll have the web interfaces of each install open and can log in and check functionality  of each. This might include SMB external storage or access to the ownCloud AppStore. If that is successful, the next step is testing an upgrade:

$ ./kill_oc_vms.sh
$ ./gen_oc_vms.sh
# wait for VMs to be accessible, this can be tested by checking ./libvirt_inv.py --list then install current OwnCloud
ansible-playbook -i ./libvirt_inv.py site.yml
# wait for playbook to complete
$ ./libvirt_inv.py --list | json_reformat  | awk -F':' '$1 ~ /ansible_host/ {print $2}' | sed 's/[", ]//g' | while read ocip; do xdg-open "http://${ocip}/owncloud/"; done
# Do some basic configuration such as install some apps and verify basic access is good
$ ansible-playbook -i ./libvirt_inv.py -e "localrepo=true testingrepo=true" site.yml
# Verify the "owncloud needs to be upgraded" page is shown, carry out the upgrade and verify it worked correctly and apps can be re-enabled etc

Issuing the updates

Once all this is complete and we’re happy with the build, the changes to the spec file are pushed to the Fedora git repository. The Fedora build system, koji, only builds from this trusted location when building for the official Fedora repos. Once built, an update is created in Bodhi. Bodhi is used to manage the built package going to the testing repositories, and then after a week or enough positive karma in testing, to the stable repositories.

The full packager experience

As you can see, it can sometimes be a complex journey from an initial upstream release to the official stable Fedora repositories. We do our best to ensure a well-tested package is built even before it hits the Rawhide and testing repositories.

I hope this provides some insight into our activities in Fedora as packagers and why it sometimes takes longer to see that update pending in the

lists than may be expected or hoped for on occasion.