In the last 2 days our container images building with centos stream 9 base image (quay.io/centos/centos:stream9) started failing due to mismatch in package versions or missing packages. The error reproduces often (not always), assumed to be due to mirrors which are not up-to-date. Example logs:
podman run -it --pull always quay.io/centos/centos:stream9 bash Trying to pull quay.io/centos/centos:stream9... Getting image source signatures Copying blob b5ddbc70c93e skipped: already exists Copying config ab9123964b done | Writing manifest to image destination bash-5.1# dnf install -y --enablerepo=crb openssl openssl-devel postgresql nmstate-devel sqlite gcc genisoimage git libvirt-client libvirt-devel java make && dnf clean all CentOS Stream 9 - BaseOS 8.9 MB/s | 8.8 MB 00:00 CentOS Stream 9 - AppStream 11 MB/s | 24 MB 00:02 CentOS Stream 9 - CRB 6.8 MB/s | 7.2 MB 00:01 CentOS Stream 9 - Extras packages 1.5 kB/s | 19 kB 00:12 Error: Problem 1: cannot install the best candidate for the job - nothing provides nmstate-libs(x86-64) = 2.2.46-1.el9 needed by nmstate-devel-2.2.46-1.el9.x86_64 from crb Problem 2: cannot install the best candidate for the job - nothing provides libvirt-libs = 10.10.0-13.el9 needed by libvirt-devel-10.10.0-13.el9.x86_64 from crb Problem 3: package openssl-devel-1:3.5.0-4.el9.x86_64 from appstream requires openssl-libs(x86-64) = 1:3.5.0-4.el9, but none of the providers can be installed - cannot install both openssl-libs-1:3.5.1-1.el9.x86_64 from baseos and openssl-libs-1:3.5.0-4.el9.x86_64 from @System - cannot install both openssl-libs-1:3.5.1-1.el9.x86_64 from baseos and openssl-libs-1:3.5.0-4.el9.x86_64 from baseos - package openssl-1:3.5.1-1.el9.x86_64 from baseos requires openssl-libs(x86-64) = 1:3.5.1-1.el9, but none of the providers can be installed - cannot install the best candidate for the job (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
I am attaching the /var/log/dnf.librepo.log for the example above.
I can confirm this and I see that proxy04.fedoraproject.org and proxy12.fedoraproject.org are returning old checksums. Not sure if the mirrorlist processes need to be restarted or if the cache files are not updated.
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: dc-move, high-gain, medium-trouble, ops
Ah yeah, they were unreachable so we took them out of the mirrorlist_proxy group, but forgot to add them back now that they're reachable again.
mirrorlist_proxy
Fixed by https://pagure.io/fedora-infra/ansible/c/4272c8aa773495e9eda5c5c0b0dfd2752e7ea8b2, waiting for the next update-mirrorlist-cache cron.
Metadata Update from @abompard: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)