fedora-infrastructure

#12786 Copr outage - build queue slow down (or halt)
Closed: Fixed by praiskup. Opened by praiskup.

Closed: Fixed
Reopen Issue

The copr-backend is running out of storage. I'm not sure why we didn't receive an early Nagios alert, or if we did, I missed it.

Currently, we only have about 200 GB of free space on the RAID. It seems like there is a lot of traffic in @xsnrg's space, so I've temporarily paused builds there:

I've also limited the number of concurrent workers to 25% to see if the overnight cleanup helps. We probably need to expand the disk space, but I'll need help from my teammates for that.

In the meantime, we've also realized that we are under a heavy load of AI scrapers on the dist-git side. The cgit server was hit with a DDoS attack, even though we're running Anubis, so we've disabled cgit again.

Hi, builders are slow for a few days, thank you for the detailed explanations and the actions underway to remedy this unexpected situation.

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, high-trouble, ops

Up and running again. A script for cleaning temporary source RPM results had failed, consuming about 5 TB. With so little storage left, I/O wait times (especially for writes) became extreme.

The task queue grew significantly overnight, so it will take a few hours to fully catch up. However, overall throughput should be much better than it has been over the past week or two.

Sorry for inconveniences.

Metadata Update from @praiskup:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None

high-gain high-trouble ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.