As far as @kevin and I knew.. loopabull had nothing to do with the CI pipeline. Did this change and what would we look for?

I was wrong, loopabull is not involved. This is the service:
https://pagure.io/ci-resultsdb-listener

Ticket https://pagure.io/fedora-ci/general/issue/66 speaks about PR flags, so that's done via loopabull, more specifically in:
- https://pagure.io/Fedora-Infra/loopabull-tasks/blob/master/f/playbooks/org.centos.prod.ci.pipeline.allpackages-pr.package.running.yml
and
- https://pagure.io/Fedora-Infra/loopabull-tasks/blob/master/f/playbooks/org.centos.prod.ci.pipeline.allpackages-pr.complete.yml

But as you can already see from these links, neither one is watching the .error topic

So is this working? Or we still need to dig more?

I'm still seeing problems. From my perspective -- as an ignorant end user with no clue of what greenwave, loopabull, or any of those terms are -- the issue is that between [1] and [2] the test status changed from "tests failed" to "test results missing". I am still seeing "missing" on [3] and [4], but those are only 2 and 3 hours old respectively, and in the past it has taken up to four hours to actually get results - so I can't be sure if the problem still exists, or if it's just the usual slowness from before.

[1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-bfff3112b5
[2] https://bodhi.fedoraproject.org/updates/FEDORA-2019-8d97c8b9e6

[3] https://bodhi.fedoraproject.org/updates/FEDORA-2019-8dce0aaefc
[4] https://bodhi.fedoraproject.org/updates/FEDORA-2019-4670e94e9b

@pingou yeah sorry for the confusion with loopabull, this is about the regular builds ... so ci-resultsdb-updater ....

I deleted the confusing comments and stand corrected

The links from the description are still relevant though ... the last resultsdb results were created 4 days ago +-

the last resultsdb results were created 4 days ago +-

Hm that sounds like a different problem then, let me check this

Ok rdbsync had failed to restart after the last reboot.

I've restarted the service and it seems messages are back in resultsdb: https://taskotron.fedoraproject.org/resultsdb/results?testcases=org.centos.prod.ci.pipeline.allpackages-build.complete :)

@pingou is there a way we could monitor these services? We are about to setup prometheus in upstream to monitor some critical parts of the infra ...

@mvadkert I use this script https://pagure.io/fedora-ci/monitor-gating to monitor the entire pipeline, it's a start but not yet enough.

This is definitely something we'll want to improve/fix with automation and proper reporting.