So, this is the queue coreos sends messages to and robosign is supposed to sign and process...

so, something is breaking down in signing?

Hmm - any more information about the messages that are stuck in the queue? When were they sent?

Our existing processes that pass messages back and forth to robosignatory seem happy at the moment.

All I see in the UI is that the queue is (slowly) growing (we're at 114 queued messages atm) and that the queue is bound to this topic: org.fedoraproject.*.coreos.build.request.*

Hmm. Maybe an artifact of the datacenter move? If there are no consumers of the queue then maybe we can just delete it. Our processes seem to be happy right now.

cc @jlebon for any more input.

I do see one failure earlier in the day today where signing timed out. It's this message:

https://apps.fedoraproject.org/datagrepper/id?id=2020-870d3d5a-d914-4478-a9a1-4ea527b47415&is_raw=true&size=extra-large

And it got no reply from RoboSignatory:

https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.coreos.build.request.artifacts-sign.finished&delta=36000

Anyway, otherwise apart from that build run, the pipeline successfully had artifacts signed by RoboSignatory on Saturday. I started a new run now to see where we stand this (EST) morning.

Could this somehow be a different coreos queue from the one that we're using now seemingly successfully? If so, then yes you can delete it. If not, then I guess we can delete the messages, though we should figure out why RoboSignatory didn't process them.

I started a new run now to see where we stand this (EST) morning.

Yup, looks like RoboSignatory is not responding now.

Request sent: https://apps.fedoraproject.org/datagrepper/id?id=2020-e56ef46d-9ba7-48c7-8611-122185095e6e&is_raw=true&size=extra-large.

Chatted with @kevin and @pingou in IRC about this. It seems like those messages accumulated during the DC move. We've deleted almost all the message in there, but will keep this open for a week or so to check in with the queue and make sure that it looks healthy.

Re. today's signing failure, RoboSignatory is busy signing perl packages. Waiting on that to finish before starting the new FCOS releases.

Cross-linking: https://pagure.io/robosignatory/issue/3

ok, so there's 19 messages in the queue as of now.

So, we still have something filling the queue thats not getting poped off. :(

Hmm, OK I was looking over the infra ansible, and I think I have an idea on what's going on. I think the simple answer is that there is nothing actually consuming the messages from the coreos queue:
- RoboSignatory uses the robosignatory queue and has its routing keys correctly set up: https://pagure.io/fedora-infra/ansible/blob/master/f/roles/robosignatory/templates/robosignatory.toml.j2#_13
- coreos-ostree-importer uses the coreos-ostree-importer queue and has its routing keys correctly set up: https://pagure.io/fedora-infra/ansible/blob/master/f/playbooks/openshift-apps/coreos-ostree-importer.yml#_18
- The coreos queue just picks up all coreos.build.request.* messages: https://pagure.io/fedora-infra/ansible/blob/master/f/roles/rabbitmq_cluster/tasks/apps.yml#_6 -- BUT there's nothing actually reading from that queue!

So... I think we should just delete the coreos queue. It was originally created by me as part of https://pagure.io/fedora-infrastructure/issue/8227: https://pagure.io/fedora-infra/ansible/c/42335b73708e0741166e74c7a59dc452c6dbf672?branch=master.

But I think at the time I was just confused about how RabbitMQ/AMQP works. I think I thought that you needed to send messages to a specific queue, rather than messages being routed to queues associated with a specific consuming application.

Anyway, if others agree with the diagnosis here, I can send a revert of that patch:

From 9ec28c68107dfc0ef223ecaf8cac3e3193bf04e9 Mon Sep 17 00:00:00 2001
From: Jonathan Lebon <jonathan@jlebon.com>
Date: Fri, 3 Jul 2020 10:42:52 -0400
Subject: [PATCH] Revert "rabbitmq: add coreos queue"
This reverts commit 42335b73708e0741166e74c7a59dc452c6dbf672.
We only need the `coreos` user, so that we can publish messages. We
don't need the `coreos` queue.
---
 roles/rabbitmq_cluster/tasks/apps.yml | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/roles/rabbitmq_cluster/tasks/apps.yml b/roles/rabbitmq_cluster/tasks/apps.yml
index 7e5b9f4c2..120e46629 100644
--- a/roles/rabbitmq_cluster/tasks/apps.yml
+++ b/roles/rabbitmq_cluster/tasks/apps.yml
@@ -3,15 +3,12 @@
 #
-- name: CoreOS queue
+- name: CoreOS user
   run_once: true
   include_role:
-    name: rabbit/queue
+    name: rabbit/user
   vars:
     username: coreos{{ env_suffix }}
-    queue_name: coreos{{ env_suffix }}
-    routing_keys:
-      - "org.fedoraproject.*.coreos.build.request.*"
 - name: CentOS CI
--
2.26.2

+1. We will need to also manually delete that queue, but we can do that when this is pushed...

Seems reasonable to me :thumbsup:

https://pagure.io/fedora-infra/ansible/pull-request/160

PR has been merged and queue has been deleted from the rabbitmq server.

Thanks! :)