Heroes of Fedora QA: Fedora 21

With Fedora 21 out the door and into the wild, I’ve finally had time to gather stats on who contributed to the Fedora QA efforts. With each milestone release (and usually each quarter), QA likes to give a big shout out to those who made things possible. Fedora 21 was a departure from past releases and gave the whole of the Fedora community a lot of new processes to create and improve. Instead of one single release product, we tested and released 3 products – Workstation, Server and Cloud. Each of these required some additional testing which QA hadn’t had to do for previous releases.

The stats here were gathered from Bodhi, Bugzilla and the Fedora Wiki using the “fedora-qa” stats scripts as well as relval. Our current tooling doesn’t allow me to easily break these stats down by individual product, so these numbers are cumulative across all three. I have statistics for the entire Fedora 21 lifecycle, so I’ll be breaking this post into 3 parts. This first installment will be stats from the Release Validation efforts – which is entering test results into the Fedora Wiki. But before we get to that – here are some highlights!

Highlights

  • Throughout the Fedora 21 cycle, 2,524 testcases were run by roughly 30 testers
  • We had 5,143 total Bodhi comments from over 175 testers
  • 3,470 new bug reports from over 500 people for F21

Those are some pretty impressive numbers – great work and thanks to all who were involved!

Wiki Stats

This is a slightly trimmed list of people who submitted release validation testing results. There are several different matrices of test cases which get tested and they get run for every Milestone release (Alpha, Beta and Final). Testing at each release helps to show how thoroughly tested each of area is.

Alpha

Testers: 23
Comments: 463
Unique referenced bugs: 29

Name Reports submitted Referenced bugs1
pwhalen 124 1003122 1112387 1136990 892178 (4)
satellit 82 1036306 1103496 1121409 1136570 1136581 1136985 1136994 1139015 1141414 (9)
michalv 48
roshi 38 1134524 (1)
adamwill 26 1099299 1101341 1134524 1140867 1142512 1142516 1142517 1142520 1142521 1142522 892178 (11)
pschindl 24 1143972 (1)
kparal 23 1142862 1144040 (2)
jskladan 22
nonamedotc 15
wutao85 9
spstarr 7
hguemar 6
mattdm 6
sgallagh 6 1143988 1143990 (2)
lnie 6
robatino 5
jdulaney 4
passthejoe 3
jreznik 3 1003122 (1)
wolnei 2
b10n1k 2
konradr 1
jbwillia 1

Over 450 results were submitted for Alpha, which is great. Keep in mind, this doesn’t include all the rawhide validation testing we’d been doing until Alpha was branched. Pwhalen, Satellit and Michalv tore the matrix up with over 250 combined results submitted! Now let’s move on to Beta (keep an eye out for a trend here).

Beta

Testers: 30
Reports: 889
Unique referenced bugs: 38

 

Name Reports submitted Referenced bugs1
pwhalen 135 892178 (1)
adamwill 133 1142520 1142521 1142522 1145777 1145783 1148087 1150147 1155026 1155301 1155304 1155329 1156198 1156354 1156380 1156614 892178 (16)
satellit 112 1036306 1103496 1136581 1141414 1145281 1148436 1149782 1158533 1169440 (9)
roshi 105 1148087 1156198 1156603 (3)
kparal 88 1038413 1155964 1156008 1156073 1156354 1156380 (6)
pschindl 80 1158533 1158975 (2)
dmossor 31 1135024 (1)
sgallagh 27 1156378 1156593 (2)
yselkowitz 26
jskladan 25
jsedlak 20
nonamedotc 16
lnie 15
robatino 15
kushal 12
kevin 10
mattdm 6
jreznik 6 705086 (2)
optak 6
tflink 4
vehrlich 4 1156499 (1)
eischmann 3
szpak 3
wolnei 1
konradr 1
pbrobinson 1
lbrabec 1 1099299 (1)
cmurf 1
greylocks 1
outsh 1

Again, you can see Pwhalen doing some mad work throughout. Satellit comes in third after being edged out by Adamw by a mere 21 results. Between Alpha and Beta several new testers joined the cause and the reported results exploded from 463 to 889! Thanks to everyone for great testing during Beta!

Final

Testers: 29
Reports: 1172
Unique referenced bugs: 17

 

Name Reports submitted Referenced bugs1
pwhalen 223 705086 892178 (2)
roshi 148 1169085 (1)
satellit 141 1135720 1136581 1169440 (3)
kparal 86 1169332 1170153 1170499 1170623 (4)
amrithajij 79
lnie 71
mmarusic 70 892178 (1)
jskladan 61 1162215 (1)
dmossor 54 1169085 892178 (2)
adamwill 53 1142521 1142522 1170062 892178 (4)
pschindl 49 1167959 (1)
mccann2 20
robatino 20
kushal 17
sgallagh 16 (1)
spmadden 12 1169085 (1)
jsedlak 10
outsh 8
mattdm 7
nonamedotc 7
jreznik 4 1170535 (1)
cmurf 4
amritha 3
tflink 2
optak 2
jdulaney 2
sdharane 1
lbrabec 1
jstodola 1

Again total reports shot up between Beta and Final. While the number of testers remained about the same, the amount of testing steadily increased as we got closer and closer to release. Very similar to how things went with F20. Did you spot the trend in those three tables? Pwhalen and Satellit dominated for an entire release. Pwhalen submitted 482 test results and Satellit submitted 335 results – about 30% of all testcases reported.

Awesome work! Looking forward to what kinds of numbers you post for F22 :p The next installment will be on updates testing through Bodhi.

Footnotes

1 – This is a list of bug reports linked to the wiki results. They don’t have to be reported by that concrete person.

Fedora Project community

3 Comments

  1. Don Cosner

    Fedora 21 is a very impressive release! Nice to see so many committed to quality, it really shows in this release. My thanks to all those that made this possible.

  2. Adam

    Impressive. I’m curious… how can you maintain such a relatively low number of test cases for a whole Linux distribution and produce that excellent outcome?
    Some mobile apps are subject to half that number of test cases and they’re just that, mobile apps, not a whole OS.

    Thanks!

    • Mike Ruckman

      @Adam – those numbers are a rough approximation of the testing that goes into a release. The “Release Validation” matrices are filled out by the QA team and doesn’t reflect any testing that happened outside the validation process. Later posts regarding karma [0] (testing updated packages) and bugzilla activity [1] can give a better overall picture to the testing that goes into a release.

      We also have test days [2], which generate a lot of good feedback and bug reports. Also, running through the matrices is still all a manual process. Testing a mobile app or a website is likely automated, which makes it easier to run more test cases during testing. Does that answer your question?

      [0] https://fedoramag.wpengine.com/?p=6634
      [1] https://fedoramag.wpengine.com/heroes-of-fedora-qa-fedora-21-part-3/
      [2] https://fedoraproject.org/wiki/QA/Test_Days

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions