It doesn't respond on ssh, ICMP seems to work. Could sssd be down?

Do you think we could get a trigger for hard rebooting these machines?

rebooted.

Yes, let me try and come up with a way to get you access.

ok.

I set up things so sysadmin-copr has access to cloud-noc-os01.rdu-cc.fedoraproject.org.

Thats our machine in rdu-cc that has access to the private mgmt network there that all the servers are on.

The power9 box is at 172.23.1.185 from there. You can use ipmitool on that server or ssh tunnel via it to get to the web interface.

From there, I made a copr user. The login info on that is in your homdir on batcave01. Can you (securely!) share it with anyone else on the copr-team that needs it?
Note that this user is an admin user there, so be careful. You should be able to power cycle it easily from there.

If needed we could set up access to the other machines there you use... but in practice I don't think they have been as unstable as the power9 box.

Awesome, thank you @kevin!

Follow-up: #11476.