With Fedora 21 out the door and into the wild, I’ve finally had time to gather stats on who contributed to the Fedora QA efforts. With each milestone release (and usually each quarter), QA likes to give a big shout out to those who made things possible. Fedora 21 was a departure from past releases and gave the whole of the Fedora community a lot of new processes to create and improve. Instead of one single release product, we tested and released 3 products – Workstation, Server and Cloud. Each of these required some additional testing which QA hadn’t had to do for previous releases.
The stats here were gathered from Bodhi, Bugzilla and the Fedora Wiki using the “fedora-qa” stats scripts as well as relval. Our current tooling doesn’t allow me to easily break these stats down by individual product, so these numbers are cumulative across all three. I have statistics for the entire Fedora 21 lifecycle, so I’ll be breaking this post into 3 parts. This first installment will be stats from the Release Validation efforts – which is entering test results into the Fedora Wiki. But before we get to that – here are some highlights!
Highlights
- Throughout the Fedora 21 cycle, 2,524 testcases were run by roughly 30 testers
- We had 5,143 total Bodhi comments from over 175 testers
- 3,470 new bug reports from over 500 people for F21
Those are some pretty impressive numbers – great work and thanks to all who were involved!
Wiki Stats
This is a slightly trimmed list of people who submitted release validation testing results. There are several different matrices of test cases which get tested and they get run for every Milestone release (Alpha, Beta and Final). Testing at each release helps to show how thoroughly tested each of area is.
Alpha
Testers: 23
Comments: 463
Unique referenced bugs: 29
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| pwhalen | 124 | 1003122 1112387 1136990 892178 (4) |
| satellit | 82 | 1036306 1103496 1121409 1136570 1136581 1136985 1136994 1139015 1141414 (9) |
| michalv | 48 | |
| roshi | 38 | 1134524 (1) |
| adamwill | 26 | 1099299 1101341 1134524 1140867 1142512 1142516 1142517 1142520 1142521 1142522 892178 (11) |
| pschindl | 24 | 1143972 (1) |
| kparal | 23 | 1142862 1144040 (2) |
| jskladan | 22 | |
| nonamedotc | 15 | |
| wutao85 | 9 | |
| spstarr | 7 | |
| hguemar | 6 | |
| mattdm | 6 | |
| sgallagh | 6 | 1143988 1143990 (2) |
| lnie | 6 | |
| robatino | 5 | |
| jdulaney | 4 | |
| passthejoe | 3 | |
| jreznik | 3 | 1003122 (1) |
| wolnei | 2 | |
| b10n1k | 2 | |
| konradr | 1 | |
| jbwillia | 1 |
Over 450 results were submitted for Alpha, which is great. Keep in mind, this doesn’t include all the rawhide validation testing we’d been doing until Alpha was branched. Pwhalen, Satellit and Michalv tore the matrix up with over 250 combined results submitted! Now let’s move on to Beta (keep an eye out for a trend here).
Beta
Testers: 30
Reports: 889
Unique referenced bugs: 38
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| pwhalen | 135 | 892178 (1) |
| adamwill | 133 | 1142520 1142521 1142522 1145777 1145783 1148087 1150147 1155026 1155301 1155304 1155329 1156198 1156354 1156380 1156614 892178 (16) |
| satellit | 112 | 1036306 1103496 1136581 1141414 1145281 1148436 1149782 1158533 1169440 (9) |
| roshi | 105 | 1148087 1156198 1156603 (3) |
| kparal | 88 | 1038413 1155964 1156008 1156073 1156354 1156380 (6) |
| pschindl | 80 | 1158533 1158975 (2) |
| dmossor | 31 | 1135024 (1) |
| sgallagh | 27 | 1156378 1156593 (2) |
| yselkowitz | 26 | |
| jskladan | 25 | |
| jsedlak | 20 | |
| nonamedotc | 16 | |
| lnie | 15 | |
| robatino | 15 | |
| kushal | 12 | |
| kevin | 10 | |
| mattdm | 6 | |
| jreznik | 6 | 705086 (2) |
| optak | 6 | |
| tflink | 4 | |
| vehrlich | 4 | 1156499 (1) |
| eischmann | 3 | |
| szpak | 3 | |
| wolnei | 1 | |
| konradr | 1 | |
| pbrobinson | 1 | |
| lbrabec | 1 | 1099299 (1) |
| cmurf | 1 | |
| greylocks | 1 | |
| outsh | 1 |
Again, you can see Pwhalen doing some mad work throughout. Satellit comes in third after being edged out by Adamw by a mere 21 results. Between Alpha and Beta several new testers joined the cause and the reported results exploded from 463 to 889! Thanks to everyone for great testing during Beta!
Final
Testers: 29
Reports: 1172
Unique referenced bugs: 17
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| pwhalen | 223 | 705086 892178 (2) |
| roshi | 148 | 1169085 (1) |
| satellit | 141 | 1135720 1136581 1169440 (3) |
| kparal | 86 | 1169332 1170153 1170499 1170623 (4) |
| amrithajij | 79 | |
| lnie | 71 | |
| mmarusic | 70 | 892178 (1) |
| jskladan | 61 | 1162215 (1) |
| dmossor | 54 | 1169085 892178 (2) |
| adamwill | 53 | 1142521 1142522 1170062 892178 (4) |
| pschindl | 49 | 1167959 (1) |
| mccann2 | 20 | |
| robatino | 20 | |
| kushal | 17 | |
| sgallagh | 16 | (1) |
| spmadden | 12 | 1169085 (1) |
| jsedlak | 10 | |
| outsh | 8 | |
| mattdm | 7 | |
| nonamedotc | 7 | |
| jreznik | 4 | 1170535 (1) |
| cmurf | 4 | |
| amritha | 3 | |
| tflink | 2 | |
| optak | 2 | |
| jdulaney | 2 | |
| sdharane | 1 | |
| lbrabec | 1 | |
| jstodola | 1 |
Again total reports shot up between Beta and Final. While the number of testers remained about the same, the amount of testing steadily increased as we got closer and closer to release. Very similar to how things went with F20. Did you spot the trend in those three tables? Pwhalen and Satellit dominated for an entire release. Pwhalen submitted 482 test results and Satellit submitted 335 results – about 30% of all testcases reported.
Awesome work! Looking forward to what kinds of numbers you post for F22 :p The next installment will be on updates testing through Bodhi.
Footnotes
1 – This is a list of bug reports linked to the wiki results. They don’t have to be reported by that concrete person.
Don Cosner
Fedora 21 is a very impressive release! Nice to see so many committed to quality, it really shows in this release. My thanks to all those that made this possible.
Adam
Impressive. I’m curious… how can you maintain such a relatively low number of test cases for a whole Linux distribution and produce that excellent outcome?
Some mobile apps are subject to half that number of test cases and they’re just that, mobile apps, not a whole OS.
Thanks!
Mike Ruckman
@Adam – those numbers are a rough approximation of the testing that goes into a release. The “Release Validation” matrices are filled out by the QA team and doesn’t reflect any testing that happened outside the validation process. Later posts regarding karma [0] (testing updated packages) and bugzilla activity [1] can give a better overall picture to the testing that goes into a release.
We also have test days [2], which generate a lot of good feedback and bug reports. Also, running through the matrices is still all a manual process. Testing a mobile app or a website is likely automated, which makes it easier to run more test cases during testing. Does that answer your question?
[0] https://fedoramag.wpengine.com/?p=6634
[1] https://fedoramag.wpengine.com/heroes-of-fedora-qa-fedora-21-part-3/
[2] https://fedoraproject.org/wiki/QA/Test_Days