fedora-infrastructure

#12955 Planned Outage - rdu2 to rdu3 datacenter move - 2025-12-08 15:00 UTC
Closed: Fixed by kevin. Opened by kevin.

Closed: Fixed
Reopen Issue

Planned Outage - rdu2 to rdu3 datacenter move - 2025-12-08 15:00 UTC

There will be an outage starting at 2025-12-08 15:00 UTC,
which will last approximately 24 hours.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:

date -d '2025-12-08 15:00 UTC'

Reason for outage:

We will be powering off hardware in our rdu2 datacenter, it will be deracked and moved to our rdu3 datacenter, reracked, and reconfigured for the new network.

Affected Services:

The following servers will be moved:

vmhost-p09-copr01.rdu-cc.fedoraproject.org
vmhost-x86-copr01.rdu-cc.fedoraproject.org
vmhost-x86-copr02.rdu-cc.fedoraproject.org
vmhost-x86-copr03.rdu-cc.fedoraproject.org
vmhost-x86-copr04.rdu-cc.fedoraproject.org
storinator01.rdu-cc.fedoraproject.org
vmhost-x86-cc01.rdu-cc.fedoraproject.org
vmhost-x86-cc02.rdu-cc.fedoraproject.org
vmhost-x86-cc03.rdu-cc.fedoraproject.org
retrace03.rdu-cc.fedoraproject.org

This will affect the following services:

retrace/abrt/faf will be down and not accepting user reports
smtp-auth-cc-rdu01 will be down and not accepting emails
download-cc-rdu01 will be down, use another mirror
proxy03/proxy14 will be down, but removed from dns, so no impact.

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/12955

Please join #admin:fedoraproject.org / #noc:fedoraproject.org on matrix.
or add comments to this ticket for more information or feedback.

Updated status for this outage may be available at
https://www.fedorastatus.org/

Metadata Update from @james:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble

To update: Things are moving slower than planned due to weather at the new datacenter location. Some folks had to have extra time to get in/home and unloading was slower than expected.

The servers are all at the new datacenter now, all racked and are being cabled this afternoon.
Hopefully we will get to mgmt today and can start (re)configuring things tomorrow...

You want me to update https://status.fedoraproject.org/ ... as the provided outage just ended.

I updated the status text and extended the outage by a day.

Thanks.

Next status update:

All the machines are moved and racked and cabled.
Due to a dhcp mistake on my part I cannot yet reach mgmt for all the hosts. :(

4 of them I can reach, the other 7 I cannot yet.

retrace03 I can reach mgmt on, but it's 10G interfaces show no link yet so I cannot bring it back up.

Tomorrow morning I expect folks will be back at the DC and can power cycle those other hosts so they come up on mgmt.
Then we can sort the network interfaces.

Priority list:

  • bring retrace03 back online and reconfiged for new net/ips
  • bring vmhost-x86-iso02 up and reinstall with rhel10 and then bring it's guests back up.
  • ditto for vmhost-x86-iso04 (and download-rdu-cc01 thats on it)
  • next will be reinstalling copr machines and bringing them back online

ok, sorry for the lack of updates here.

In addition to the weather there were some network issues to work through.

Current status:

retrace03 was brought back up yesterday. That machine still has broken rails, so likely there will be a short outage to re-rack it once the replacements arive.

download-cc-rdu01 should now be back up as 'download-iso01'. There is a cname from the old name to the new one.

smtp-auth-cc-rdu01 is back up as smtp-auth-iso01, however there's still some missing data that I need to extract from the old one (which is on a machine I cannot reach yet, see below).

vmhost-p09-copr01 is back up and should be working.

3 of the x86 copr hypervisors still need 10G connections sorted. That should happen tomorrow morning.
1 of the x86 copr hypervisors is ready to be reprovisioned. Hopefully will do that tomorrow.
vmhost-x86-cc01 needs it's 10G ports sorted out (former home of smtp-auth-cc-rdu01).

I am going to close the outage now and track additional work in the dc move ticket. #12818

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

medium-gain medium-trouble outage

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.