fedora-infrastructure

#8425 ppc64le mirror problems
Closed: Fixed by kevin. Opened by praiskup.

Closed: Fixed
Reopen Issue

Similar issues for fedora-{30,31}-ppc64le chrotos in copr build

Copr repository                                  45 kB/s | 3.3 kB     00:00    
fedora                                           21 kB/s | 5.2 kB     00:00    
updates                                         3.0 kB/s | 8.1 kB     00:02    
Error: Failed to download metadata for repo 'updates': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

From dnf.log

2019-11-28T13:39:07Z DEBUG repo: downloading from remote: updates
2019-11-28T13:39:08Z DEBUG error: Downloading successful, but checksum doesn't match. Calculated: 5250a5854c7bb39a47322cea877af2e9f8ee3256693d18d914659550a7863f5bfc8a292aa3a34f573a33f5fc6a9491cfcd1830f957ff9723775726dd21444010(sha512) 5250a5854c7bb39a47322cea877af2e9f8ee3256693d18d914659550a7863f5bfc8a292aa3a34f573a33f5fc6a9491cfcd1830f957ff9723775726dd21444010(sha512)  Expected: 0cd5f940e593f21d9cde392d0350af941922997939feeabb32b34b916e1ec42335dede820bccf86d6147f8bdf401685de2d0e8134365dee24016ed96be5e6b0c(sha512)

The mirrorlist servers are not being restarted on many of the proxies and are therefore running with old data. There is a fix in ansible (KillMode=none) for the service files, but they have not been yet applied. Until someone with the right privileges runs the playbook against the proxies and reloads systemd and restarts the mirrorlist servers there might be old data served. The root cause is that systemd is partially killing the container cleanup process if KillMode=none is not specified.

Smooge applied this and restarted things a few hours ago.

So, should be fixed. If not, please re-open...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Metadata Update from @praiskup:
- Issue status updated to: Open (was: Closed)

Happened again:
https://copr-be.cloud.fedoraproject.org/results/@copr/copr-dev:pr:1152/fedora-31-ppc64le/01126556-copr-frontend/builder-live.log.gz

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: mirrorlists

Is this only happening with ppc64le?

Any news here @praiskup ?

Is this only happening with ppc64le?

no, I've seen i386 as well

Any news here @praiskup ?

no, it's fine now

Metadata Update from @praiskup:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

I think this happened again when our mirrorlist containers where not restarting correctly a few days ago.

According to upstream Podman we are not using the recommend systemd service files to start and stop Podman as described here:
https://github.com/containers/libpod/issues/3696#issuecomment-564461876

I will try to get those changes in Fedora's mirrorlist service files.

https://copr-be.cloud.fedoraproject.org/results/@copr/copr-dev:pr:1173/fedora-rawhide-ppc64le/01137403-copr-frontend/builder-live.log.gz happened again

Metadata Update from @praiskup:
- Issue status updated to: Open (was: Closed)

proxy04, proxy101 and proxy110 are running on outdated data for a few days and need restarting

The mirrorlist containers on those proxies to be precise.

I restarted proxy04.

proxy101 and 110 I reinstalled with fedora 31, and they don't seem to work right with the current deployment. However, they are only internal and not used by anything external anyhow.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

proxy101 and 110 I reinstalled with fedora 31, and they don't seem to work right with the current deployment. However, they are only internal and not used by anything external anyhow.

I have the ansible commits ready for Fedora 31. Let me know if I should push them.

Can you make them conditional so f29 proxies are still ok? If so, sure! If not, best wait for next week when people are back from PTO.

Can you make them conditional so f29 proxies are still ok?

The changes I prepared are already conditional to only change systems based on Fedora 31. I am dropping the container setup and I am now running the mirrorlist-server directly from systemd. As it is now a single binary without any additional dependencies it does not need to run in a container. This should remove all the problems we are seeing running it as a rootless container via systemd during container stop/removal. As the package is only available on Fedora 31 I had to make it conditional.

If so, sure! If not, best wait for next week when people are back from PTO.

That was my plan. I just saw that you mentioned that proxy101 and proxy110 are already switched to Fedora 31 and as I already prepared the changes I wanted to offer a fix. Not sure how important it is. I will wait until next week unless you prefer me to push them earlier.

If it's not going to bother the existing f29 ones, go ahead and push it. It would be nice to fix the proxy101/110 case (even tho they don't get any traffic)

If it's not going to bother the existing f29 ones, go ahead and push it. It would be nice to fix the proxy101/110 case (even tho they don't get any traffic)

Pushed. Please let me know if something with my changes is not working as expected.
Metadata
None

mirrorlists

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.