In February, I gave a talk at DevConf in the Czech Republic about — background on where it came from, what problems it’s trying to solve, what we are actually doing, and why we think those things address the problems. You can watch the video on YouTube, but there are a few reasons you might want to read this instead. First, videos are hard to skim or search. Second, time moves on and this contains a few updates (which I have marked with Update, so if that’s all you care about they will be easy to find). And finally, I made some embarrassing typos on the slides, which I have fixed. You can see those slides on my web site, and what follows is basically each one with commentary. It ends up rather long, so this is part one, with the second part and third part to follow, and the panel discussion and Q&A after that.

The Video

Fedora is Awesome

It’s important to start with this. I’ve heard some worry recently that is based on the premise that Fedora is currently broken, has no users, or otherwise somehow sucks. Of course, as Fedora users and developers, we know that this is not the case, so this concern has a shadowing worry behind it: that this is an outsider initiative done in a ham-fisted way, planning to change everything without really knowing what we have.

I hope I can put that to rest. This effort comes from within Fedora, and it really does start with a recognition of our strength.

We have very positive brand. When I go to a conference and talk about Fedora, obviously there are some complaints about specific things, but overall, people are happy with us. We have a very strong user and developer community — people are using Fedora in production in the real world, sometimes in amazing and crazy ways (for large-scale web hosting, as a platform for very high-stakes rapid stock trading, as the desktop for a not-small law firm, as the basis for the most popular CS course at Harvard…).

We’ve had ten successful years of this — as you can see in the video, I’ve got the t-shirt, and I expect many of you reading this have it too — and Fedora 20 is our best release yet, with a very positive press response, and great feedback from our uses. It came out very quickly on the heels of F19, but is very solid and basically just better across the board. (If you are on F19, or still on an older release, I encourage you to update!)

So, if we’re so great, why are we doing anything at all? Why can’t we just keep going along in the same way?


Meanwhile, out in the world….


Even if we go with the rule that 90% of everything is crap, 10 million repos is still a large number, and much larger than the 14,000-some packages in Fedora. There’s just no way we could possibly capture all of that, and it is a lot of open source developer energy happening in a space that we’re not playing in.

Also: the base OS has developed to the point where it has become uninteresting. Now, we do a lot of things in Fedora which are really on top of what might be considered a bare base OS, but basically the thing we are doing as our primary activity in Fedora is considered boring. I was at a large cloud conference a while ago, and almost nobody was using Fedora, and so I asked people why they chose the distribution they are building their stuff on, and why they didn’t choose Fedora. Almost universally, the response wasn’t “What I am using is great!” — it was “Oh, I don’t care. I just picked this, and that’s what I’m using and it’s fine.” That layer just isn’t exciting. Even if there really are a lot of interesting things going on, people who are trying to actually do things with the distribution don’t attach much importance to them.

And that’s kind of a change. It used to be that we could easily pack a room at a conference just by mentioning Fedora — now, you have to talk about Docker. The excitement about open source development is still there, but it’s gone somewhere else.

In turn, this leads to a shift in the balance between the effort to get software into a distribution and the reward of doing so. It used to be that if you had open source software, and you could convince the distros to get your get your software into a distro, that’s how you knew that you had arrived. So if you could get into Red Hat Linux or Slackware or whatever, you were really something, not just a little project off to the side. That’s really not the case anymore.

A lot of developers just don’t see that as important. Maybe it does still have value, but it’s not seen that way, so the effort people are willing to put into getting their software into a distro is much lower. (This is all still part of the question, by the way, not an answer — but we have some answers coming up.)

For the last two points, skip forward with me to the next slide….

In which I recommend another talk

Is distribution-level package management obsolete?

At FOSDEM, Donnie Berkholz of Gentoo gave a great talk entitled “Is distribution-level package management obsolete?”

Update At the time I gave the talk, the audio on the file wasn’t right, but that has been (mostly) fixed, so you can watch it now.

I actually do recommend taking an interlude here and watching the whole talk. In summary, it goes into depth on where distros are today and where they need to go, by looking at the question

How are actual software developers in the real world doing things, and how could distros better cater to them? How are users doing things, and how could we cater to them?

Which, obviously, is in line with some of the observations I’m making here, and if I haven’t convinced you, maybe Donnie can. If you value data over handwaving, this may help.

Donnie also talks about the issue of higher-level software moving at a different cadence than the base, which is also a concern I’d like Fedora to address — but which I didn’t really cover in the rest of my talk. Also: continuous integration, continuous delivery, integration with language level package management, and of course Docker gets name-checked. These are all important ideas feeding into as well.

Anyway, jumping back to the point from the former slide. One of the things which we often wring our hands about in Fedora is an apparent decline in our overall popularity. We look at the Google Trends, or the yum updates connection numbers, and the lines are going doowwwwwnnnn… are we doing something wrong? Has Ubuntu stolen all of our users?

Well, actually all of the major distributions that work basically in the way Fedora does are on the decline. Slackware peaked before Fedora; openSUSE and Fedora seem to have peaked in terms of the buzz/popularity measure around 2006 or 2007. But Ubuntu has the same peak, just a bit later in 2009. If we count the years from now… that’s a long trend of decline for all of us. Ubuntu is still very popular, of course, but, they’re not cool. None of us are cool anymore.

We want to be cool. How can we do that?

Update I forgot to mention Gentoo in the talk. Don’t feel left out, Gentoo friends. You are also no longer cool.

And also

Phones! Tablets! Android! ChromeOS! / Cloud! EC2! OpenStack! GCE! / Containers! Docker! CoreOS!

The world has changed, and there are all of these things which are different in the computing landscape from the way they they were 10 years ago. To continue to be relevant, we need to react to that in some way.

What exactly is “”?

A word for a bit of planning and direction-setting — instead of running ahead without looking around.

Basically, it is a big umbrella term that is full of handwaving. But, it means that we are trying to plan our direction — looking at where we are going, rather than just walking ahead and hoping the place we end up is useful to someone.

There is no secret back-room agenda to all of this. We all want Fedora to succeed. I work for Red Hat, and since Red Hat wants Fedora to succeed, I and other Fedora community members employed by Red Hat have been supported in working on this planning process. That’s pretty much all there is to that.

While some of us have been working on for a while now (including more and more people as we go from initial ideas to implementation), the whole process is still very open to more ideas and contributions. We want to hear things that you see out in the world that aren’t reflected what I’ve been talking about, and things that we aren’t doing that could solve some of these problems. I’d love to hear about all of those.

Update Feel free to reply here, or anywhere. I’ll respond directly, and I’ll also gather up any big question into a followup as the final post in this series.

Another thing: we are talking about some big changes, and it can’t all be immediate. This isn’t a situation where if we don’t flip everything all over by Fedora 21, it has failed. It’s going to take a while, and we’re working on planning out a roadmap for this.

Update The Fedora 21 schedule should be approved by the time you read this; that will give some solid ideas about the immediate change we can expect to see, and what parts might happen later.

I’m really big into the idea of incremental improvement. As long as we have a plan and are going forward and making some difference each release, that’s good. I know this sometimes makes people working on new ideas frustrated — for example, removing sendmail from the default install. I’m happy to get sendmail out of there, and I think it doesn’t belong for a lot of cases, but if it takes us two or three releases to remove it, eh, it’s okay. We haven’t failed. It’s a lot better than never starting the process and leaving things as they are when they should change, and it’s also better than dumping changes onto our users before they’re ready. There is a balance, and having a plan helps us find it.

Fedora’s Mission

The Fedora Project's mission is to lead the advancement of free and open source software and content as a collaborative community.

So, this is Fedora’s mission. I asked if anyone knew that it was, and very few hands came up.

It is very broad. The original Fedora Project, after the merge of and the Red Hat Linux Project, was tasked with making an open source Linux distribution, and that was basically it.

And so, after a while, there was some soul-searching — Fedora loves soul-searching… it’s part of who we are. Even four or five years ago, people felt that we were doing a pretty good job at that straightforward distro part, and started to ask if that’s all we should be doing, and so this is the mission that was developed out of that.

It’s… kind of broad, and maybe a little too broad, but it is very ambitious. The question is now: if that’s our mission — and there hasn’t been any talk of changing it — are we actually positioned to execute on it? Do the things we do actually move us toward the target we say we are aiming for?

So that’s part of what is: to look at our mission and decide what more we need to do to make it happen.

Additive, not Restrictive

We do a pretty good job of making the base distro. What can we do outside of that? What can we improve?

Most everything we are talking about in is additive. We’re not saying that you can’t do certain things in Fedora anymore, but we are saying that we may need to add some things to make the goals of the mission come true. We do somethings very well already — what else can we do.

Of course, we also need to look at how we can make what we are doing well even better, possibly by changing some of our focus and certainly by improving our processes. Because really, even though many people think the base OS is now boring, it’s far from done, and there is a lot of innovation going on at that level as well. CoreOS is an example, and the rpm-ostree project (Update: now Fedora Atomic Initiative) is an example of something that we might want to look at for future Fedora.

And of course there are a lot of process automation improvements that could happen at our core level as well.

More Questions!

Here I throw out a bunch more questions — there are a lot of them, and they all feed into the “why” of

First: it happened, as part of that previous round of soul-searching, that we decided that the default Fedora offering would be a desktop OS. That seemed like a good idea, and there’s still nothing inherently wrong with it.

But, Fedora has also traditionally had a large sysadmin audience. A lot of people installing RHEL on their servers, and they have Fedora on their desktop — and people with production Fedora servers. Increasingly, with the default offering being the desktop, those people often increasingly felt that their role was to periodically shout “No, wait! You’re breaking things! Stop it!” So, if you are a sysadmin who is active in Fedora, you might have started to feel that your job is to keep the desktop developers from breaking Fedora yet again. That ends up being a negative cycle with no winners — and an unhappy, fragmented user community.

So, one of the direct things we want to solve in is giving a positive voice for people using Fedora in server roles, or RHEL in server roles and with a concern about Fedora’s direction. We don’t want people to feel like the only job available is to be the stop energy; we want to focus all of that into a positive direction where we’re building something together.

It’s also valuable to think about and talk about our relationships with our downstream distributions.  We’re part of an ecosystem, we have some really big stakeholders, and we benefit a lot from doing things that align with their needs. When we do things that are useful for RHEL and for CentOS, that feeds back, and Red Hat is happy to pay for conferences and development on things, and that’s all pretty cool.

Also, those downstream distributions are seeing the same changes in the world, and if we can change in a way that is helpful to them, we all win. If we don’t include our stakeholders in the conversation, they’ll need to change independently, and we really all lose out. So increasing the transparency of the overall extended family of distributions seems like a good thing to do (but that doesn’t limit Fedora — or — to just the narrower scope that downstream might find important, and in fact we know that success requires diversity and adoption beyond that).

There’s the general trend towards base being considered boring again; I don’t have anything more to add at this point.

Related to that, though, is the difficulty of building things on top of Fedora. Fedora moves so fast and is kind of ill-defined, so basically the only way to participate in Fedora is to make sure your stuff is polished so that it will fit into the Fedora distribution as it stands today. No one has really been able to successfully make software which goes on top of Fedora and keeps up. Can we make that easier in some way? I think that would be nice. Again, Docker fits in nicely here, so there’s another Docker namedrop.

Next, do we have the barriers to entry in the wrong place? For example, our package review process really really makes you go over a very tall wall to get into Fedora, but once you’re over that wall, unless someone happens to notice, you can do any of horrible thing to your package (and even if someone notices, there isn’t a clear process for fixing it). So we have many packages in Fedora where initially, when the packages went through that first polishing phase, they were pretty good, but then the upstream updated or the packager changed something, and now they’re completely out of compliance with our guidelines. We don’t even have a good measure of what is out of compliance — we don’t look at this stuff much, except Fedora QA does a good job of making sure it installs every six months. (And again, we look at new packages when they come in.)

So, maybe we’re putting that in the wrong place. Maybe we could do a better job by letting people put their packages in to some level in an easier way, and then after they’re included in the project look at improving them. (Update This is actually in progress in the Enviroment and Stacks Working Group now, in the “Fedora Playground” repository. Cool!) And maybe also, we could focus more on packages where it really matters that they’re packaged well, and decide that with others, well, eh, that’s probably going to be crap forever and we’ll do what we can to make it as useful as possible to people.

Finally, this question of how we can move fast while still keeping a handle on the direction we’re going — balancing innovation with change management. That’s always a struggle for Linux distributions, but maybe with Fedora.nextwe can make it easier.

Watch for Part II Next Week

So! Those are some of the questions… the background Next week, some of the proposed answers. I’ll go over some of the work in progress (and other work planned for the future), and connect it back to these questions.

Also, as mentioned above, let’s continue this conversation in comments and replies, and in addition to responding, I’ll distill that into a Q&A post at the end of this series.