can you invite me, @sumantrom and @kparal when this gets discussed, so we can co-ordinate plans? thanks!

I've scheduled this for Tuesday, February 28, at 10:00 AM EST (15:00 UTC). (The meeting on Tuesday, February 21 is canceled.)

Join links:

• https://bluejeans.com/395383051/
• https://www.bluejeans.com/numbers

GNOME Music has been a problematic app in the past

Will try out with the testing as much as i can but has less time now as i have a part time day job usually on Tuesdays and Wednesdays

We discussed this at yesterday's working group call, and so far the results seem inconclusive. We'll have a series of test days next week, which will hopefully help, but as it stands we could end up in the same situation as F36 and F37.

I think that it's difficult for the workstation working group to propose policy changes, since they are likely to have consequences beyond our own area. I get the sense that we'd like to eliminate a certain class of issue from being a blocker, but exactly what is unclear.

Do we want to rehash https://pagure.io/fedora-workstation/issue/304 ?

Proposed starting point: we could suggest to QA that the default application functionality criterion should be interpreted somewhat less strictly. It's a deliberately hazy suggestion because "somewhat" leaves a lot of room for interpretation, and we'd need to figure out what the impact would really be via the usual course of the blocker review process, but hopefully we can have fewer blockers over more minor issues without affecting how we handle more major issues.

Maybe that will suffice. Maybe it won't and we'll need to consider more significant changes like modifying the criterion or exempting entire applications from the process. Probably wouldn't hurt to try.

we could suggest to QA that the default application functionality criterion should be interpreted somewhat less strictly.

I'd support that.

Looking at the requirements again, it strikes me that, while we have a blanket basic functionality requirement for apps, there doesn't seem to be an equivalent requirement for the system as a whole. Are we in a situation where our requirements for apps are higher than they are for system functionality?

I wonder if we could replace the basic functionality requirement with a broader rule that applies to everything on the install media and says something to the effect of:

Issues are considered blockers when they are:

1. Severe: a component or app crashes, or a feature completely fails to work, and
2. Important: the feature is critical to users using the system, or
3. Common: likely to be experienced by a large number of users

That's significantly stricter than the current criterion. Seems OK to be. If we find that quality of releases declines, we can always change our minds and revert back to what it was before, after all.

Perhaps add:

1. Embarrassing: bug could cause a Fedora release to be widely perceived as lower-quality

to give a little more wiggle room during blocker review.

That's significantly stricter than the current criterion.

Ha, I was hoping for the opposite!

The main difference is that it removes the "basic functionality" part of the defnition ("the app must at least be broadly capable of its most basic expected operations"). If a feature is essential to an app's purpose (say, creating albums in Photos) but isn't viewed to be critical to the operation of the system as a whole, then it wouldn't be considered a blocker.

What's nice about this is that it takes the importance of the app into account. If something fails in Files/Settings/Software then we might well consider it to be critical to the system as a whole. Whereas, if something fails in a less essential app, we might ignore it.

I should note that, in proposing this, I don't want to suggest that general app quality is not a concern. It totally is, and I don't think we should be including low quality apps. It's just that general app quality is not the responsibility of the final release criteria.

Ha, I was hoping for the opposite!

Ah, I used very confusing wording. I meant the criterion is more strict in the sense that fewer bugs would trigger the criterion, and so quality requirements are more lax.

Looking at the requirements again, it strikes me that, while we have a blanket basic functionality requirement for apps, there doesn't seem to be an equivalent requirement for the system as a whole. Are we in a situation where our requirements for apps are higher than they are for system functionality?

So, the reason we have a very generic requirement for apps is (IIRC) that desktop team - back when we wrote the criteria - specifically wanted us to ensure that any apps shipped by default are in a basically working state, but it is a huge amount of work to codify "basically working" for each app individually (and it would be ongoing work, because over time the set of included apps changes, and the features of each included app also change). We're not going to do that. So we have to go with something more generic.

Nobody has ever asked for a similar requirement for "the system" (however you want to define that...), and we (QA) don't think it's obviously required, so we don't have one.

If desktop team doesn't think it's important any more that all pre-installed apps basically work, we can of course remove that criterion. (Note we already did this for KDE: the requirement specifically does not apply to all pre-installed apps on KDE, but only a relatively short list of specific apps.) But every time we have this discussion, when we ask that question, you say "no no, we do still want that". I don't really see a different way we can have that requirement, unless someone wants to write up a huge list of specific requirements for each pre-installed app.

On @aday 's proposal: well, as mentioned, the KDE team specifically doesn't want to block on "basic functionality" for every pre-installed app, so I suspect they wouldn't want to block on that list of things for every pre-installed app either. What does "everything on the install media" mean? Does it apply to everything in /usr/bin in all blocking installer environments and installed systems? Really? That's a lot of stuff. What does "critical to users using the system" mean exactly? Which users? Using it how? If I'm a user with a very, very specific use case almost nobody else has and there's a bug which breaks that, is it covered? Do we really want to block on every "common" bug, no matter how important? I, uh, suspect the answer is no.

If we were to implement that, I suspect it would turn out to be much less strict (in the sense of "allowing fewer blockers") than you think. Especially with the "embarrassing" addition. The four items combined look, to me, to allow for a huge amount of criteria judo, to the extent that you could call almost anything a blocker under them. I mean, just for instance, you can certainly argue that the currently-proposed nautilus/gtk bug is "important", "common", and "embarrassing".

Note we actually do already have a catch-all requirement:

https://fedoraproject.org/wiki/Fedora_38_Beta_Release_Criteria#Beta_Blocker_Bugs

Note that that defines three categories of "blocker bug", of which "Bug relates to an unmet Beta Release Requirement" (i.e. the release criteria) is only one. Another is "A bug in a Critical Path package that: Cannot be fixed with a future stable update [and] Has a severity rating of high or greater and no reasonable workaround (see definition of severity and priority)". It seems this isn't really widely known about, but it is there, and we use it occasionally where appropriate.

Thanks for your comments, @adamwill !

I think it goes without saying that I/we are not experts on the QA process and are mainly trying to come up with ideas. If there are alternative suggestions or approaches that would resolve the current issue, then that would be great.

It seems to me that, what we are primarily looking for is a relaxation of the app basic functionality criterion. The question is, what does relaxation mean? My personal take might be:

• Preinstalled apps should generally work - they should run, and you should be able to use them in a superficial way without encountering multiple severe issues, but:
• Bugs related to individual app features shouldn't block the release, provided those features aren't essential or commonly used.

Where essential means special classes of activity that are considered essential for a desktop system: file management, viewing common file types, text editing, running commands from the terminal, browsing the web.

Defining "common" is always going to be somewhat vague, but you could maybe say: is it likely to be experienced by more than 30% of users? Or, is it a feature that we know a lot of people use?

Obviously that's just my take on this - we'd need to see what other members of the WG think.

I guess the point I'm trying to make is that that is pretty much what the criteria say already. I'm not sure that rewriting the criteria is going to move us forward.

It does seem interesting that every time we try a new formulation, to me, it seems entirely plausible that all the 'questionable' bugs would've been blockers under it anyway. Note you say "multiple severe issues" - well, that definitely seems to cover the bugs we were talking about in recent cycles (Contacts and Calendar definitely had "multiple severe issues", to the point where you couldn't practically use them for their intended purpose at all). The nautilus/gtk bug that @mclasen has concerns about making a blocker is in "file management", which you identify as "essential", and "moving/copying things around" sure seems like a commonly used and essential function to me.

It seems like whenever we try to think about this in the abstract, the rules we come up with turn out to cover the bugs that we were questioning the blockeriness of. Or to put it another way: nobody seems to want to come out and say "we are OK with shipping quite broken applications".

you say "multiple severe issues" - well, that definitely seems to cover the bugs we were talking about in recent cycles

It sounds like we have different interpretations of "severe" here. :)

The nautilus/gtk bug that @mclasen has concerns about making a blocker is in "file management", which you identify as "essential", and "moving/copying things around" sure seems like a commonly used and essential function to me.

This one is a bit borderline indeed. If this was the regular copy/paste function then I'd agree that it should definitely be a blocker. However, I've anecdotally heard that people don't really use "copy to" and "move to" - if that holds true then I'd say it's not a blocker. (Still, a bug that ought to be fixed.)

It seems like whenever we try to think about this in the abstract, the rules we come up with turn out to cover the bugs that we were questioning the blockeriness of.

It does seem that there are differing interpretations of the terms being used. However, we have the whole English language at our disposal - we ought to be able to write a criteria which meets the working group's expectations!

Perhaps the best way forward is therefore for the working group to indicate which of the previous proposed blockers meet our expectations for blockers - that's currently in https://etherpad.opensuse.org/p/workstation-blockers . That will provide us with a concrete shared reference point on which we can base updates to the criteria.

nobody seems to want to come out and say "we are OK with shipping quite broken applications"

I think the working group has explicitly said that it wants to keep some apps despite known quality issues, and we're attempting to be more permissive about the blockers. Is that not accepting the current state?

Whether we agree whether these apps should be included or not is another question, of course...

I've set myself a goal to try to resolve this situation. Once F38 is out, I'd like to come up with some proposals and let's see if we can agree on some of it. I'm happy to work with you on adjusting the wording until all parties are content.

while we have a blanket basic functionality requirement for apps, there doesn't seem to be an equivalent requirement for the system as a whole. Are we in a situation where our requirements for apps are higher than they are for system functionality?

Just a quick note here. If by "system" you mean the graphical desktop environment, I don't think that's accurate. We have criteria for keyboard layouts, system panels, keyring, user switching, session management, login screen, and we're also in a process of discussing window management. It's not a blanket "basic functionality" (it would be much harder to estimate what it means), but they are specific parts of the DE. We have more criteria regarding the system overall (not just DE). It's true that very often a broken desktop functionality prevents something else important from working, so it's easy to mark it as a "transitive blocker" without having explicit criteria for it. But we do try to add specific ones when needed (like now with fullscreen games problem).

F38 happened, and we seemed to avoid the last-minute scramble that we had in previous releases. Are there any followup actions that we need to take, or can we close this issue?

I like closing issues.

I think we did better for two reasons: (a) we became slightly stricter about interpreting existing blocker criteria, i.e. we required a slightly higher bar to designate an issue as a blocker, and (b) we got lucky and had fewer bugs than usual.

I'd say we did two out of the three things suggested in the OP (and it was a good idea not to do the third): we dropped the RC test week and we did do earlier testing of all the GNOME apps so we could decide on blocker status nice and early. I don't think the "ignore previous release blockers" policy is necessarily a great idea. So, makes sense to me to close the ticket: we made some improvements here and they seem to have worked out. We can always continue to improve things going forward, of course.

(b) we got lucky and had fewer bugs than usual.

Hehe. One of the factors, I think, was that I either created or updated over 40 bugs related to the GNOME test week (test day 1, test day 2). If was sufficiently early in the cycle, the most important ones got fixed. But it also took me almost 2 weeks full time, and I can't really promise that I can do that again in the next cycle.

Thanks for all your work on this, @adamwill and @kparal! Let's close this ticket and keep an eye on things for F39.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/workstation/tickets/issues/359

Please continue any further discussion there.