fedora-infrastructure

#10257 EPEL 8 repository checksum mismatch
Closed: Fixed with Explanation by kmittman. Opened by kmittman.

Closed: Fixed with Explanation
Reopen Issue

Unable to use EPEL 8 repository due to stale CDN metadata files

$ sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
Failed to set locale, defaulting to C.UTF-8
Extra Packages for Enterprise Linux Modular 8 - x86_64                                               361 kB/s | 956 kB     00:02    
Extra Packages for Enterprise Linux 8 - x86_64                                                       983  B/s | 4.7 kB     00:04    
Errors during downloading metadata for repository 'epel':
  - Downloading successful, but checksum doesn't match. 
Calculated: 9719bd3e7d8f3a217103b14eb122758d3d3a086a6a46eeeab953017a4a5907f9e96079b7faec9a7f836571713ce25d14e94abd8b7709a8717b78b8292221a621(sha512) 
9719bd3e7d8f3a217103b14eb122758d3d3a086a6a46eeeab953017a4a5907f9e96079b7faec9a7f836571713ce25d14e94abd8b7709a8717b78b8292221a621(sha512)  
Expected: d7866fd1604ef749ced5df995dfa050d491a4cf642c7b81797d5da38e6e80829758956deaec02405ab2dbcecff8cd6dcdb13b5559198101fd965daa1958e8500(sha512) 
be6e8f5d423e9a31310dd5a6450097c2a1253efeffd54ebcae0ded443e878feb851304ae77130f16ac021b27850c5ee83444f4e6650cad926684beb7648fc471(sha512) 
Error: Failed to download metadata for repo 'epel': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Reproduces for both RHEL 8 and CentOS 8 systems

Must be a certain out-of-sync mirror, as I see that for example this matches: https://mirror.arizona.edu/fedora-epel/8/Everything/x86_64/

$ curl -s https://mirror.arizona.edu/fedora-epel/8/Everything/x86_64/repodata/repomd.xml -o - | sha256sum 
be92bcd5964e80dccb00765e8e545e8844b337eecfc3bc6578d5d1c54542e2e9  -

$ curl -s https://mirror.arizona.edu/fedora-epel/8/Everything/x86_64/repodata/repomd.xml | grep "primary\.xml"
    <location href="repodata/35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5-primary.xml.gz"/>

curl -s https://mirror.arizona.edu/fedora-epel/8/Everything/x86_64/repodata/35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5-primary.xml.gz -o - | sha256sum 
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  -

And editing the .repo file to use this mirror works.

Is there is an easy way to check which servers are being checked with the default mirrorlist= ?

Some more digging

$ list=$(sudo dnf --verbose install dkms 2>&1 | grep -o "http.*repomd.xml")
$ for repomd in $list; do repodata=$(dirname "$repomd"); primary=$(curl -s "$repomd" | grep -o "/.*primary\.xml.gz"); curl -s "${repodata}${primary}" -o - | sha256sum | sed "s|\-|${repodata}|"; done
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://mirror.arizona.edu/fedora-epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  https://mirror.arizona.edu/fedora-epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  https://mirror.atl.genesisadaptive.com/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://mirror.atl.genesisadaptive.com/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  https://download-ib01.fedoraproject.org/pub/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://download-ib01.fedoraproject.org/pub/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://mirror.prgmr.com/pub/epel//8/Everything/x86_64/repodata
2323be22c91ad9cb29c97ebd5acf6b58dec80e6241a2054d93e54eb3e56072f2  https://d2lzkl7pfhq30w.cloudfront.net/pub/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  https://download-cc-rdu01.fedoraproject.org/pub/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://download-cc-rdu01.fedoraproject.org/pub/epel//8/Everything/x86_64/repodata
2323be22c91ad9cb29c97ebd5acf6b58dec80e6241a2054d93e54eb3e56072f2  https://dl.fedoraproject.org/pub/epel//8/Everything/x86_64/repodata
2323be22c91ad9cb29c97ebd5acf6b58dec80e6241a2054d93e54eb3e56072f2  http://dl.fedoraproject.org/pub/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://mirror.dst.ca/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  https://mirror.dst.ca/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://mirrors.upr.edu/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  https://mirrors.upr.edu/epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  https://mirror.12z.eu/pub/linux/fedora-epel//8/Everything/x86_64/repodata
35471cc8396068401f36b4e171dd15de9dca6e0f3ef836d2fd6c0b2bf2ba2cb5  http://mirror.12z.eu/pub/linux/fedora-epel//8/Everything/x86_64/repodata

Do you still see the error? It looks like it is working now.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

It should have been fixed about an hour or so after it started. Although you may have had to do a 'yum clean all' until the cache timed out.

The issue/problem was:

  • We added epel9-next/epel9-next-testing to the sync script that syncs updates repos.
  • playbooks were not run, so the change was not actually live.
  • playbooks were run to update something else and this change became live.
  • There were no epel9-next/epel9-next-testing repos to sync, so the sync script failed every run
  • I pointed them to a tree of empty repos we had there for epel releases.
  • repos synced, but because this tree of repos had one dir with no repodata and another dir that was a duplicated repodata, it caused mirrormanager to update epel8/epel8-testing somehow, making the checksums wrong for everyone.
  • I pushed a new epel8/epel8-testing updates push which updated those dirs and mirrormanager and got new checksums that were right pushed out.

So, we need to:

  • ALWAYS remember to run playbooks after merging PR's. You don't want to leave things unpushed / tested.

  • Fix the empty epel dirs tree to actually be correct.

Hope it is fixed, lets wait and see for the next sync.

Also, I fixed the empty-epel-repos and empty-epel-next-repos for future use.

Thank you for the RCCA, it's working now and our CI/CD is unblocked.

Metadata Update from @kmittman:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
High
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.