There are some things that make things slowish to get all the way to end users, but it should be possible to do this with the current infrastructure. It would require maintainers to make sure QA and releng folks know to test and move the update along faster than normal, and depending on mirrors it could take a while to sync out to particular mirrors, but it's definitely possible to do.

But I guess you want to make this a requirement / always the case? In that case, it would be a FESCo issue (Fedora Engineering Steering Comittee) to change the updates policy/process. You could file a ticket with them here:
https://pagure.io/fesco/issues/ but they likely will require discussion on the devel list, so you could just post there with more details of what you would like to see happen and why.

If FESCo makes this a policy, we could look at adjusting infrastructure, but without that there's nothing for us to do here.

There are some things that make things slowish to get all the way to en=
d users, but it should be possible to do this with the current infrastruc=
ture. It would require maintainers to make sure QA and releng folks know =
to test and move the update along faster than normal, and depending on mi=
rrors it could take a while to sync out to particular mirrors, but it's d=
efinitely possible to do.

That is good to know.

But I guess you want to make this a requirement / always the case? In t=
hat case, it would be a FESCo issue (Fedora Engineering Steering Comittee=
) to change the updates policy/process. You could file a ticket with them=
here:=20
https://pagure.io/fesco/issues/ but they likely will require discussion=
on the devel list, so you could just post there with more details of wha=
t you would like to see happen and why.=20

I do want to make this a requirement, but in the meantime, speeding up th=
e process would be greatly appreciated, especially for Firefox and Chromi=
um updates. That said, if you are blocked on FESCo, I understand.

It isn't about being blocked by FESCo, it is a whole set of things. Infrastructure does not 'build' packages.. it just provides the infrastructure to get the builds. Infrastructure does not push packages out to mirrors (which is where consumer systems get the updates from), those are pulled by the volunteer mirrors.

In the case you outlined in the opening, say Firefox or LibreOffice has a security fix. Fedora Infrastructure does run a service which tries to do this but it is up to the site to put updates in a form they can be 'found' and it requires the packager to want those notifications. That fix has to be found by the volunteer packager who maintains the package. The packager then have to get the source code, apply any other patches, work out compile problems and then put it into the build system (this is the second place Infrastructure would be involved.)

After it gets throught the build system it goes into the updates channel. Updates need to then be tested since more than one 'pushed too quickly for security' has nuked systems. It will then get put into the next compose of the system into updates-testing. At this point the various mirrors start pulling it onto their sites and the mirror system shows a change which can be gotten by users. It usually takes 12 hours for an update push to get out to most of the mirrors.

The changes that FESCO would have to make is one that packagers are expected to make updates faster (though FESCO has no 'power' to enforce that). It would also have that the release engineering group to make a 'fast-track' compose set which would be a repository added to installs. At that point the change would need to get added to fedora-release package so that it can be added. Fedora Infrastructure would also make changes to the mirrormanager for this 'fast-track' to be seen by its tools.

I think it is a worthy project to work on but it is going to need driving through multiple points (FESCo and getting developers to buy into it, and working out the changes needed in various tools from bodhi, koji, and various composers to do.)

It isn't about being blocked by FESCo, it is a whole set of things. Inf=
rastructure does not 'build' packages.. it just provides the infrastructu=
re to get the builds. Infrastructure does not push packages out to mirror=
s (which is where consumer systems get the updates from), those are pulle=
d by the volunteer mirrors.=20
=20
In the case you outlined in the opening, say Firefox or LibreOffice has=
a security fix. Fedora Infrastructure does run a service which tries to =
do this but it is up to the site to put updates in a form they can be 'fo=
und' and it requires the packager to want those notifications. That fix h=
as to be found by the volunteer packager who maintains the package. The p=
ackager then have to get the source code, apply any other patches, work o=
ut compile problems and then put it into the build system (this is the se=
cond place Infrastructure would be involved.)
=20
After it gets throught the build system it goes into the updates chann=
el. Updates need to then be tested since more than one 'pushed too quickl=
y for security' has nuked systems. It will then get put into the next com=
pose of the system into updates-testing. At this point the various mirro=
rs start pulling it onto their sites and the mirror system shows a change=
which can be gotten by users. It usually takes 12 hours for an update pu=
sh to get out to most of the mirrors.=20

Would it be possible for mirrors to poll faster? DNF has a mechanism
for invalidating metadata.

As far as updates testing, I believe that we can make a fast-path in
the following cases:

• No changes to install scripts.
• High-quality upstream (Firefox and LibreOffice both qualify)
• Upstream releases signed source code (almost certainly true of Firefox)=
  =2E

In those cases, I believe we should be able to fully automate the
package build and release process. As soon as upstream releases the
source tarballs, Fedora infrastructure would automatically download
the source code, check the signature, and produce a new source RPM.
This would then be used to build a new binary RPM, and the binary
RPMs would be pushed as soon as they are built. This is what QubesOS
does, although they use signed Git commits instead of source tarballs.
This does skip much of the QA, but I believe we can count on Mozilla
to do sufficient QA before release.

Personally, I believe we should stop relying on mirrors, and instead
use a CDN like Cloudflare or Fastly that supports rapid invalidation
of caches. That would allow us to ensure that there is one valid
version at any given time, and the metadata hashes could enforce this.

Sincerely,

Demi

Would it be possible for mirrors to poll faster? DNF has a mechanism
for invalidating metadata.

There are 2 problems with that.

1) Mirrors are voluntary sites at various universities and other places which are mainly looking to do what they can with limited resources.
2) There are only 4 master mirror sites to get those updates from. Rsync is a IO intensive routine which needs to check all the files in both trees to see what changed. The more mirrors checking more often, the more the galloping herd problem happens on the master mirrors and none of them get any updates.

As far as updates testing, I believe that we can make a fast-path in
the following cases:

No changes to install scripts.
High-quality upstream (Firefox and LibreOffice both qualify)
Upstream releases signed source code (almost certainly true of Firefox)=
=2E

In those cases, I believe we should be able to fully automate the
package build and release process. As soon as upstream releases the

For libreoffice it might be possible but Firefox ships some codecs which Fedora has to legally strip for patent reasons and same with some other upstreams. That requires developer time to look over what has been added and removed.

Packages in the end are not maintained by infrastructure. We provide the place where those bits are built and where the source code is stored.. but each package would need its maintainer to approve that it uses an auto-build method. It isn't something Infrastructure can or would want to force. [I also expect it would be unforceable because there are enough maintainers who are contrarian by nature that even if they agreed that it was a good idea would not want to go along with it for their package.]

source tarballs, Fedora infrastructure would automatically download
the source code, check the signature, and produce a new source RPM.
This would then be used to build a new binary RPM, and the binary
RPMs would be pushed as soon as they are built. This is what QubesOS
does, although they use signed Git commits instead of source tarballs.
This does skip much of the QA, but I believe we can count on Mozilla
to do sufficient QA before release.
Personally, I believe we should stop relying on mirrors, and instead
use a CDN like Cloudflare or Fastly that supports rapid invalidation
of caches. That would allow us to ensure that there is one valid
version at any given time, and the metadata hashes could enforce this.

CDN's cost money. You pay for every bit you change in the CDN, for how far it is duplicated and how much traffic is pulled out of it. It also takes time to 'push' content into the CDN with a daily compose taking anywhere from 48 hours to a week on some of the CDN's. We have tried other methods to feed the CDN's with changes over time but you end up with cache-invalidation problems with the client and master disagreeing while the CDN agreeing with neither of them.

A CDN solution for just fast updates might be possible but it would require careful engineering of tooling (which Infrastructure would help with) and of engineers (which is a political problem outside of what Infrastructure can do). That is an entire project which needs work from stem to stern versus a problem ticket. I think getting a feel of the community in the list(s?) as you did is a good place to start but it will take perseverance :persevere: .

Sincerely,
Demi