Fedora Moves Towards Forgejo

The decision to move to Forgejo as the new git forge has reached a new stage. Fedora Linux’s leadership has decided to choose Forgejo as their preferred git forge replacement. There are numerous factors involved in this decision and this article will discuss them, present some background on the process, and invite one more chance for feedback before the formal Fedora Council vote.

Gist of the situation

We’ve known for a long time that the Fedora Project needs a new git forge solution. The software we currently use, Pagure, has served us well. Sadly, it never took off in the wider world. So we had to maintain the Pagure git forge ourselves and use it to build the Fedora Linux releases at the same time. A few years ago, we considered GitLab, and had a lot of discussion… which ultimately didn’t go anywhere. Out of that we got a clear message. It’s important that this crucial part of our infrastructure be free and open source software.

At the Fedora Council’s annual face-to-face meeting in February we discussed a large list of options. By the end of the day, we crossed off all but two: GitLab Community Edition and Forgejo. We also determined that no hosting providers can meet our unique needs; we’ll have to self-host. We then asked the Advanced Reconnaissance Crew (ARC, a subteam of Fedora Infrastructure) to investigate these in more detail. They were particularly asked to look at 1) any show-stopper missing features and 2) maintenance effort and cost.

ARC has presented their initial results (about which Akashdeep goes into detail below). The quick summary: either option could work, and both have some gaps we’ll need to fill. Several things tilt the scales one direction. GitLab CE is “open core”, with some features only in the non-open-source “premium” version (including some we really want). This is both an immediate practical concern and a long-term one. Open core software tends, over time, to get less open (no matter how good the initial intentions). In addition, the Infrastructure team is more comfortable with the Forgejo codebase and the language it is written in.

So, what’s next? The Council, currently, has a clear preference for Forgejo. This is a big decision and we don’t want it to feel rushed. Therefore, we’re opening this up one last time to everyone’s comments. We have created a discussion thread referencing this article to submit your feedback to. You can find it using the #git-forge-future tag. After two weeks, we’ll take our formal vote — and then get on with the work!

– Matthew

How we got here

The earliest attempts to move over to an alternative git forge solution started as early as January 2020. At that time the development work on Pagure became increasingly inconsistent. This resulted in the maintenance costs soaring due to the unmaintained state of the development. Shortly after that, in March 2020, the decision was made to move over to GitLab after evaluating over 300 user stories and a SaaS offering was planned for the Fedora Project. While the development of new features for Pagure slowed down, an exemplary group of contributors stepped up to ensure that the current state of Pagure was maintained well. After a detailed AMA session with folks from GitLab in September 2020, a SaaS offering was announced for Fedora Project in October 2021, and a bunch of sub-projects and SIGs started moving their workflows over to the Fedora Project namespace of GitLab.

While the case looked mostly positive for the sub-projects and SIGs like Fedora Websites and Apps team, there were concerns regarding the scalability in terms of contributor counts and suitability in terms of the Dist Git purposes. This was specifically brought up in the Fedora Council issue ticket discussion and realized at a much later stage in function which led to the initiative request of Dist Git to GitLab from Fedora Infrastructure in February 2023. Fedora Infrastructure worked on an application that helps migrate repositories from Pagure to GitLab through November 2023. By December 2023, the evaluation of Git Forges had become a priority for Fedora Council and a concurrent investigation that explored migrating Dist Git functionality in a forge-agnostic way to another git forge was completed by March 2024.

The Investigation, a.k.a The Prequel!

After the Git Forge alternatives were shortlisted during the Fedora Council F2F Hackfest in February 2024, the Community Linux Engineering (CLE) team (previously called the Community Platform Engineering (CPE) team) were tasked with the investigation. The Advanced Reconnaissance Crew (ARC) sub-team was to report the positives and negatives of each git forge option, while working with various stakeholders across the Fedora Project community to capture the projects requirements for our dist-git. The initiative was led by Tomas Hrcka and Akashdeep Dhar, and Akashdeep also represented the Fedora Councils needs to the team. They began collecting these requirements in the form of user stories in March 2024 to use as the baseline of the investigation. The initiative was then joined by David Kirwan and Ryan Lerch who were able to deploy an instance of both Forgejo and GitLab CE in the CommuniShift app, and the team began validating that each user story worked in each git forge instance. These findings were reported in the official documentation.

The availability of these instances really helped onboard more contributors into the user stories validation efforts. This meant that the focus was not only on the packagers and release engineering workflow but also covered those associated with subprojects/SIGs like Design, Documentation, Accessibility, Websites & Apps, Mindshare etc.

Throughout the investigation, conversations on Fedora Discussion and on the Fedora Project mailing list were happening, making sure Fedoras use cases were being addressed, and the ARC team had presentations during the Fedora Linux 40 Release Party as well as Tomas Hrcka’s talk during the Flock To Fedora 2024 to give this huge change maximum exposure.

Once the investigation completed, Fedora Council was slated to make the final decision on which forge the project would choose to migrate to.

And now the work really begins

Like any other initiative, or large project or piece of work, the plan is never perfect, and this particular effort was quite literally years in the making. The next step for all the parties involved is to have some retrospectives to reflect on how this all went. There are plenty of learnings to be had for sure, and we hope to have a better understanding of how to drive such major decisions in a much better way.

With the investigation on the git forge solutions now wrapped up, we plan to work with Community Linux Engineering (CLE) to direct the migration of Dist Git assets and the conversion of team workflows to the deployed Forgejo Server instance once the decision is formally (and finally!) announced in the coming weeks by Fedora Council.

Pagure Dist Git is connected with a variety of ecosystem services. These include, among others:

• Bodhi
• Fedora CI
• COPR
• Fedpkg
• The New Hotness
• Fedora Notifications
• Pagure
• Toddlers
• Fedora Messaging
• Monitor Gating
• Release Engineering Scripts
• Fedora Packagers
• Packit

Please do not hesitate to reach out to the Fedora Infrastructure team to provide your support. 

Utmost attempts will be made to ensure that the git forge replacement has feature parity with the systems it is attempting to replace (i.e. Pagure Dist Git and Bugzilla). However, it is important to understand that this is a complex undertaking. Most workflows related to various subteams, subprojects and SIGs are likely to remain the same. In some cases we might have to re-imagine the workflows to fit the change (best case scenario). In other cases it may be necessary to deprecate workflows that are no longer reasonably supportable (worst case scenario).

Folks are requested to have an understanding of the user stories that were taken into consideration while comparing the git forge alternatives. Please collaborate with the Fedora Infrastructure team to integrate your specific workflows into the Dist Git Replacement platform.

We look forward to bringing this change to Fedora collaboratively with you in 2025!