In Koji rpmbuild timeout is currently set to 48 hours. Technically it is possible to increase it, but I don't know whether we want to do that.

In 3rd part koji, we are routing the power hungry build to our more powerful builders.

in koji-hub.conf policy
channel =
[...]
source /chromium- :: use powerbuilder

There is a need to have powerbuilder machine for all arches...
Hope it helps.

In Koji rpmbuild timeout is currently set to 48 hours. Technically it is possible to increase it, but I don't know whether we want to do that.

I understand your point, but we will need something, as we don't want to leave user with a browser with known security vulnerabilities.

In 3rd part koji, we are routing the power hungry build to our more powerful builders.
in koji-hub.conf policy
channel =
[...]
source /chromium- :: use powerbuilder
There is a need to have powerbuilder machine for all arches...
Hope it helps.

That's a nice idea Nicolas. I think that it's possible to find these machines and other components could benefit from this as well. From my own ones, this would help me with WebKitGTK builds and I can imagine the Qt/KDE guys would like that too.

@pbrobinson do have some aarch64 hardware that could be used for this?
@sharkcz and the same question for s390x and ppc64le, but I don't have problems with these arches anymore.

I think the problem is in finding a balance between number of (v)CPUs, memory size, number of parallel tasks inside the build and overcommitting the host's resources. If the VM or the host start swapping, it will easily become too slow. If the host is too slow on its own (slow CPU, low performant I/O), then only increasing the timeout can help. Thus more information is needed for a proper decision.

So, we will be migrating to some new builders soonish, perhaps that will help. The buildvm's should have similar stats, but the cpus will be newer and possibly faster.

We do have some buildhw aarch64 boxes we could try putting them on, I don't know if that would help or not.

It seems odd to me that aarch64 is taking so much longer than x86, that points to some part of it's build that is... non optimized? Can you identify the part it spends so much time in and see if we can improve the tools there?

OK, another failure on F30 aarch64:

https://koji.fedoraproject.org/koji/getfile?taskID=37826028&volume=DEFAULT&name=build.log&offset=-4000

Would it be possible to increase the timeout by just one hour for example as a short term solution?

So, we will be migrating to some new builders soonish, perhaps that will help. The buildvm's should have similar stats, but the cpus will be newer and possibly faster.
We do have some buildhw aarch64 boxes we could try putting them on, I don't know if that would help or not.

These are good news, Kevin!

It seems odd to me that aarch64 is taking so much longer than x86, that points to some part of it's build that is... non optimized? Can you identify the part it spends so much time in and see if we can improve the tools there?

It's because on aarch64 it builds some extra stuff (at least when I compare the number of ninja edges). I can look what's the extra stuff that's needed to build.

We have upped the timeout from 48h to 54hours.

Thank you Kevin!

Looks like thats getting them to complete?

Shall we close this now and persue the other paths seperately (us: deploy new builders, you: investigate the extra build stuff)?

Feel free to re-open if you want us to do anything additionally here.