fedora-infrastructure

#12778 Fedora-CI `dist-git-trigger` jobs fail to trigger
Closed: Will Not/Can Not fix by lecris. Opened by lecris.

Closed: Will Not/Can Not fix
Reopen Issue

Describe what you would like us to do:

Fedora-CI dist-git-trigger jobs randomly fail to be triggered. These are triggered by RabbitMQ events https://github.com/fedora-ci/dist-git-trigger/blob/master/Jenkinsfile#L18-L34:
- topic: org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete
- queue: osci-pipelines-queue-11

Emphasis on the random part of it though. I will try my best to look through datagrepper to see if I can find some example events, but a few failing bodhi updates were:
- https://bodhi.fedoraproject.org/updates/FEDORA-2025-1d49305143
- https://bodhi.fedoraproject.org/updates/FEDORA-2025-b1eed46c99
- https://bodhi.fedoraproject.org/updates/FEDORA-2025-1b15972c16
- https://bodhi.fedoraproject.org/updates/FEDORA-2025-2c0b8bd637

When do you need this to be done by? (2025/09/08)

We can manually trigger the tests that are gating and are reported to us, but otherwise we don't know the impact of this issue.

Navigating the datagrepper I did find that the event was there and in a similar state for both a successful and a failing bodhi update

$ http get https://apps.fedoraproject.org/datagrepper/v2/search \
  delta==432000 \
  category==bodhi \
  topic=="org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete" \
  package==pacemaker

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, high-gain, ops

Could this be the 6.16.3 kernel networking bug? (as discussed on matrix)?

From what I was shared, it probably is not that

root@osci-jenkins-1-586454cbbf-7hj9t:/# uname -a
Linux osci-jenkins-1-586454cbbf-7hj9t 5.10.240-238.959.amzn2.x86_64 #1 SMP Thu Aug 7 19:21:39 UTC 2025 x86_64 GNU/Linux

It is seemingly an issue between that host and the rabbitmq server, but can't tell what it is. It seems that the setup of those host needs to be adapted to keep more logs to find out what it is.

Can put this in the backlog unless there are some ideas on what logs to check or listen to.

Yeah, I have not seen any other reports of problems with rabbitmq consumers... and there are a number of them.

I'm not sure how to debug this much more on our end. Open to ideas.

Managed to get admin access to the jenkins and confirmed that RabbitMQ messages are received, they are just not handled afterwards. Can close this now, and will investigate more on the issue on that end.

Metadata Update from @lecris:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)
Metadata
None

Needs investigation high-gain ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.