fedora-infrastructure

#10380 openQA message publishing to fedora-messaging staging doesn't work
Closed: Invalid by adamwill. Opened by adamwill.

Closed: Invalid
Reopen Issue

Since 2021-11-15, openQA has not been able to publish messages to the staging fedora-messaging instance. Currently it shows errors like this:

Nov 26 22:10:05 openqa-lab01.iad2.fedoraproject.org openqa-webui-daemon[2334134]: [error] Publishing org.fedoraproject.stg.ci.productmd-compose.test.complete failed: SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

the errors only started showing up after an openQA upgrade today which made it log errors better, but the last time the staging instance actually successfully published a message seems to have been this one at 1637011964 (Mon, 15 Nov 2021 21:32:44 GMT).

This doesn't seem to correspond to any change I've made to the instance. I did upgrade it to Fedora 35 recently - but that was earlier, on 2021-11-02 . I updated to a new openQA build - but that was today. There was no dnf transaction on 2021-11-15 - the closest transactions were on 2021-11-10 and 2021-11-17. I did tweak the ansible plays a bit, but again, not on 2021-11-15 - I only made possibly-relevant changes on 2021-11-19.

So, I'm thinking maybe something changed in the stg instance of messaging itself, but not really sure what.

openQA is configured to use amqps://openqa.stg:@rabbitmq.stg.fedoraproject.org/%2Fpubsub as the URL, with params ?amq.topic&cacertfile=/etc/fedora-messaging/stg-cacert.pem&certfile=/etc/pki/fedora-messaging/openqa.stg-cert.pem&keyfile=/etc/pki/fedora-messaging/openqa.stg-key.pem. This config has not changed for a long time. The openQA certfile and keyfile are provided from the secret store, I think.

I think this was due to messaging-bridges (The openshift app that bridges fedora-messaging to fedmsg and vice versa).

I fired a build of it and it's back online now...

Nope, still failing the same way. I don't think it'd be due to any bridge as openQA publishes directly to fedora-messaging (AMQP).

Huh. Looks like this was caused by a change in upstream openQA after all (they made the URL construction for the AMQP query more correct, which unfortunately broke us because we were relying on the old way being incorrect to hack in additional query parameters...)

That doesn't explain the timeframe inconsistency, though. I think there may be some other problem affecting the publishing of openqa. messages, in fact, though ci. messages are going through. I'll try and figure that out after I figure out what to do about the URL construction problem.

Metadata Update from @adamwill:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.