I think what's happening here is that https://github.com/coreos/rpm-ostree/pull/3193 landed, so rpm-ostree started grabbing a shared lock by default.

Now here x86_64 is finishing first, and regenerating the summary tries to get an exclusive lock: https://github.com/ostreedev/ostree/blob/267ca93da28e00adb3f961d208821bba4644a2d4/src/libostree/ostree-repo.c#L6125

So...it's not clear to me that we should really require an exclusive lock for regenerating the summary. The summary feels like it should be locked on its own, independently of the repo objects and refs.

Alternatively, pungi could change to regenerate the summary after all architectures have finished.

:thumbsup: We should probably do both those things.

Additionally, it might make sense to bump the lock timeout just in general for that repo. The default is 30 seconds, which might not be enough for a compose to write a full OSTree commit over NFS. (To clarify, the timeout doesn't need to cover the time of a whole compose, only the final bit where we commit to the target repo.)

https://github.com/ostreedev/ostree/blob/267ca93da28e00adb3f961d208821bba4644a2d4/man/ostree.repo-config.xml#L231

@walters what's your thoughts on this? Do you need more info, is there an upstream tracker?

Do we know if this is affecting other ostree users (e.g. SB) or just iot?

AFAICS, regenerating the summary is an option in pungi today:
https://pagure.io/pungi/blob/master/f/pungi/ostree/tree.py#_153

I am not seeing anything in pungi-fedora which enables it.

Short term, I think I'd recommend changing pungi to regenerate the summary after all architectures have finished.

Alternatively, we could make the rpm-ostree change to do locking opt-in.

PR in https://github.com/coreos/rpm-ostree/pull/3253

https://github.com/ostreedev/ostree/pull/2493

Did some looking into this.

• iot has a separate pungi repo where they do set update_summary to True.
• They also have a separate OSTree repo than the unified repo we use for Fedora CoreOS and SIlverblue (and used to use for Atomic Host/Workstation).
• They compose into the /mnt/fedora_koji/koji/compose/iot/repo/ whereas SB composes into /mnt/fedora_koji/koji/compose/ostree/repo/ (the unified repo).
• For the unified OSTree repo the commits are then synced over to the prod repo (/mnt/fedora_koji/koji/ostree/repo/) (including summary file update) using new-updates-sync.
• There is no prod/compose repo split for iot OSTree repo and no use of new-updates-sync, hence why they set update_summary to True.

This explains why iot is hitting it and SB isn't.

Ahhhh. I had done a git grep update_summary in pungi-fedora not realizing there was a separate pungi-iot.

Ok, thanks for digging into this! So it is currently IoT specific. But, I think it turned up a real issue and we should pursue those above patches.

These are fixed in the f35/36 composes 20211211.0 with ostree-2021.6-3.fc35