Yes, I can have a look, do I have access to the VMs already?

Yes, I can have a look, do I have access to the VMs already?

Yes you need to add the iad2 bastion host to your ssh config file (https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html#ssh-configuration)

Thanks. I can't sudo to root on that host, I'm using the prod OTP token.

Also, on the docs file, it should probably be bastion-iad01.fedoraproject.org in the ProxyCommand.

Good spot @abompard I'll update the docs. Thanks

Thanks. I can't sudo to root on that host, I'm using the prod OTP token.

Yeah, it was busted. ;(

I have it working with otp now. yubikeys are still not working. Will that be enough for you?

Sure, OTP is fine, I'll have a look. Thanks.

Joining a node to a cluster is actually a manual operation, that is described here in the docs. I had to run the following commands on rabbitmq02 and rabbitmq03 to get them to join the cluster:

[root@rabbitmq02 ~][PROD-IAD2]# rabbitmqctl stop_app
Stopping rabbit application on node 'rabbit@rabbitmq02.iad2.fedoraproject.org'
[root@rabbitmq02 ~][PROD-IAD2]# rabbitmqctl reset
Resetting node 'rabbit@rabbitmq02.iad2.fedoraproject.org'
[root@rabbitmq02 ~][PROD-IAD2]# rabbitmqctl join_cluster rabbit@rabbitmq01.iad2.fedoraproject.org
Clustering node 'rabbit@rabbitmq02.iad2.fedoraproject.org' with 'rabbit@rabbitmq01.iad2.fedoraproject.org'
[root@rabbitmq02 ~][PROD-IAD2]# rabbitmqctl start_app
Starting node 'rabbit@rabbitmq02.iad2.fedoraproject.org'

I'm currently having trouble connecting rabbitmq03, it says it can't contact rabbitmq02 on the epmd port. I will look more into it tomorrow.

Ah, sorry, I had started writing the comment before you changed the metadata.

Is that something we could add to the playbook?

RAah!

One additional question: could we federate the iad2 cluster to the phx2 cluster? Or is doing that over a wan link crazy? If we could it might help migration... but just a thought...

OK apparently I only needed to restart rabbitmq on node 02 and it all fell into place :-)

The cluster seems to be up now:

[root@rabbitmq01 ~][PROD-IAD2]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@rabbitmq01.iad2.fedoraproject.org'
[{nodes,[{disc,['rabbit@rabbitmq01.iad2.fedoraproject.org',
                'rabbit@rabbitmq02.iad2.fedoraproject.org',
                'rabbit@rabbitmq03.iad2.fedoraproject.org']}]},
 {running_nodes,['rabbit@rabbitmq02.iad2.fedoraproject.org',
                 'rabbit@rabbitmq03.iad2.fedoraproject.org',
                 'rabbit@rabbitmq01.iad2.fedoraproject.org']},
 {cluster_name,<<"rabbit@rabbitmq01.iad2.fedoraproject.org">>},
 {partitions,[]},
 {alarms,[{'rabbit@rabbitmq02.iad2.fedoraproject.org',[]},
          {'rabbit@rabbitmq03.iad2.fedoraproject.org',[]},
          {'rabbit@rabbitmq01.iad2.fedoraproject.org',[]}]}]

Federating over WAN seems possible (federating, not clustering), what do you want to do exactly? Forward messages in phx2 to iad2? Or the other way around? (can't be both, there would be loops)

I'm not sure it's necessary to add join_cluster to the playbook: rabbitmq will try to contact other cluster members on startup to setup the cluster. Only this time, they were probably not networked together yet, as the log shows:

=WARNING REPORT==== 19-May-2020::16:19:49 ===
Could not auto-cluster with rabbit@rabbitmq01.phx2.fedoraproject.org: {badrpc,
                                                                       nodedown}
=WARNING REPORT==== 19-May-2020::16:19:49 ===
Could not auto-cluster with rabbit@rabbitmq02.phx2.fedoraproject.org: {badrpc,
                                                                       nodedown}
=WARNING REPORT==== 19-May-2020::16:19:49 ===
Could not auto-cluster with rabbit@rabbitmq03.phx2.fedoraproject.org: {badrpc,
                                                                       nodedown}
=WARNING REPORT==== 19-May-2020::16:19:49 ===
Could not find any node for auto-clustering from: ['rabbit@rabbitmq01.phx2.fedoraproject.org',
                                                   'rabbit@rabbitmq02.phx2.fedoraproject.org',
                                                   'rabbit@rabbitmq03.phx2.fedoraproject.org']
Starting blank node...

However, once a blank node has started, it has to be reset to join a cluster, and this should probably remain a manual operation as it destroys all data on that node. Now that the networking is setup, destroying a node and rebuilding it from scratch should end up in the new node re-joining the cluster.

Oh, my bad, the logs says phx2 in the cluster members. So the nodes were probably started with the phx2 configuration file, that's why they couldn't setup the cluster on startup.

Thanks for fixing this up!

I was wondering about federating so we could have phx2 send a copy of messages to iad2 while we were moving, then switch it at the end. However, I think now perhaps it's just easier/safer to let things be down when we move and bring it all up right in iad2.