releng / failed-composes

#5620 Fedora-Rawhide-20231111.n.0 DOOMED
Closed by jnsamyak. Opened by releng.

Closed
Reopen Issue

pungi.global.log

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Kinoite, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 108884732. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/Kinoite/ostree_installer-3/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Onyx, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 108884845. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/Onyx/ostree_installer-4/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Sericea, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 108884958. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/Sericea/ostree_installer-5/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch aarch64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 108885079. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64/Silverblue/ostree_installer-6/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch ppc64le) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 108885224. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/ppc64le/Silverblue/ostree_installer-7/runroot.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887571. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Spins-SoaS.x86_64.log for more details.

Variant: Spins, subvariant: SoaS task failed. Pinging maintainers: @aperezbios @chimosky 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887547. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Labs-Scientific_KDE.x86_64.log for more details.

Variant: Labs, subvariant: Scientific_KDE task failed. Pinging maintainers: @scitech 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887530. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Labs-Astronomy_KDE.x86_64.log for more details.

Variant: Labs, subvariant: Astronomy_KDE task failed. Pinging maintainers: @lupinix 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Comp_Neuro) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887531. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Labs-Comp_Neuro.x86_64.log for more details.

Variant: Labs, subvariant: Comp_Neuro task failed. Pinging maintainers: @ankursinha @bt0dotninja @dan1mal @neuro-sig 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887567. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Spins-Xfce.x86_64.log for more details.

Variant: Spins, subvariant: Xfce task failed. Pinging maintainers: @maxamillion @kevin @nonamedotc 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887583. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Spins-Mate.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887577. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Spins-LXDE.x86_64.log for more details.

Variant: Spins, subvariant: LXDE task failed. Pinging maintainers: @cwickert @mtasaka 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887558. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Labs-Jam_KDE.x86_64.log for more details.

Variant: Labs, subvariant: Jam_KDE task failed. Pinging maintainers: @q5sys 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Sway) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887592. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Spins-Sway.x86_64.log for more details.

Variant: Spins, subvariant: Sway task failed. Pinging maintainers: @anthr76 @jkonecny @fale @alebastr 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant i3) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887588. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64-x86_64/livemedia-Spins-i3.aarch64-x86_64.log for more details.

Variant: Spins, subvariant: i3 task failed. Pinging maintainers: @x3mboy @jflory7 @defolos @odilhao @nasirhm 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Cinnamon) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887576. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Spins-Cinnamon.x86_64.log for more details.

Variant: Spins, subvariant: Cinnamon task failed. Pinging maintainers: @grinnz 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887541. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Labs-Design_suite.x86_64.log for more details.

Variant: Labs, subvariant: Design_suite task failed. Pinging maintainers: @luya 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Budgie) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887587. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Spins-Budgie.x86_64.log for more details.

Variant: Spins, subvariant: Budgie task failed. Pinging maintainers: @joshstrobl 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887559. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Labs-Robotics.x86_64.log for more details.

Variant: Labs, subvariant: Robotics task failed. Pinging maintainers: @x3mboy @robotics-sig 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887585. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64-x86_64/livemedia-Spins-LXQt.aarch64-x86_64.log for more details.

Variant: Spins, subvariant: LXQt task failed. Pinging maintainers: @lupinix @zsun

  • No Task ID, look at log statement
[OSBUILD         ] [ERROR   ] [FAIL] Osbuild (variant Workstation, arch *) failed, but going on anyway.
[OSBUILD         ] [ERROR   ] OSBuild: task 108887526 failed: see /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/global/osbuild/Workstation-1-watch-task.log for details

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Games) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 108887551. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/livemedia-Labs-Games.x86_64.log for more details.

Variant: Labs, subvariant: Games task failed. Pinging maintainers: @dagostinelli @richardanaya 

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 108887618. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.

Variant: Labs, subvariant: Scientific task failed. Pinging maintainers: @scitech 

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant KDE) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 108887650. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64/imagebuild-Spins-KDE-raw-xz.aarch64.log for more details.

Variant: Spins, subvariant: KDE task failed. Pinging maintainers: @rdieter @ngompa @marcdeop 

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 108887660. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64/imagebuild-Spins-Xfce-raw-xz.aarch64.log for more details.

Variant: Spins, subvariant: Xfce task failed. Pinging maintainers: @maxamillion @kevin @nonamedotc 

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant Phosh) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 108887665. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64/imagebuild-Spins-Phosh-raw-xz.aarch64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 108887671. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64/imagebuild-Spins-LXQt-raw-xz.aarch64.log for more details.

Variant: Spins, subvariant: LXQt task failed. Pinging maintainers: @lupinix @zsun 

[ERROR   ] Compose run failed: Live media task failed: 108887566. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/logs/aarch64-x86_64/livemedia-Spins-KDE.aarch64-x86_64.log for more details.

Compose Total time: 4:35:32
Compose phase INIT time: 0:00:20.
Compose phase PKGSET time: 0:27:52.
Compose phase WEAVER: FAILED.
Compose phase BUILDINSTALL time: 1:36:58.
Compose phase GATHER time: 1:39:06.
Compose phase OSTREE time: 0:30:58.
Compose phase OSTREE_INSTALLER time: 1:50:59.
Compose phase CREATEREPO time: 0:07:07.
Compose phase CREATEISO time: 0:10:00.
Compose phase REPOCLOSURE time: 0:01:33.
Compose phase IMAGE_BUILD: FAILED.
Compose phase LIVE_MEDIA: FAILED.
Compose phase OSBUILD time: 0:11:28.

possibly this is https://bugzilla.redhat.com/show_bug.cgi?id=2248915 ?

going to fire a new compose with updated glibc.

CC: @fweimer

ok, so no... glibc failed gating (both -20 and -21) so this shouldn't be at all related to them...

There seem to be multiple problems. There is this 502 error (HTTP?):

2023-11-11 06:39:02,301: Performing post-installation setup tasks
2023-11-11 06:40:19,326: running /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/work/x86_64/Kinoite/lorax_templates/ostree-based-installer/lorax-configure-repo.tmpl
2023-11-11 06:40:19,350: running /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/work/x86_64/Kinoite/lorax_templates/ostree-based-installer/lorax-embed-repo.tmpl
2023-11-11 06:40:37,670: command output:
error: Server returned HTTP 502
2023-11-11 06:40:37,670: command returned failure (1)
2023-11-11 06:40:37,670: template command error in /mnt/koji/compose/rawhide/Fedora-Rawhide-20231111.n.0/work/x86_64/Kinoite/lorax_templates/ostree-based-installer/lorax-embed-repo.tmpl:
2023-11-11 06:40:37,670:   runcmd ostree --repo=/var/tmp/lorax/lorax.pv52_blf/installtree/ostree/repo pull --mirror fedora fedora/rawhide/x86_64/kinoite
2023-11-11 06:40:37,673:   subprocess.CalledProcessError: Command '['ostree', '--repo=/var/tmp/lorax/lorax.pv52_blf/installtree/ostree/repo', 'pull', '--mirror', 'fedora', 'fedora/rawhide/x86_64/kinoite']' returned non-zero exit status 1.

https://kojipkgs.fedoraproject.org//work/tasks/4732/108884732/runroot.log

But there is also this:

DEBUG util.py:446:  2023-11-11 08:15:15,518: Traceback (most recent call last):
DEBUG util.py:446:  2023-11-11 08:15:15,519:   File "/usr/sbin/anaconda", line 191, in <module>
DEBUG util.py:446:  2023-11-11 08:15:15,519:     from pyanaconda.core import util, constants, path
DEBUG util.py:446:  2023-11-11 08:15:15,519:   File "/usr/lib64/python3.12/site-packages/pyanaconda/core/util.py", line 39, in <module>
DEBUG util.py:446:  2023-11-11 08:15:15,519:     from pyanaconda.core.configuration.anaconda import conf
DEBUG util.py:446:  2023-11-11 08:15:15,519:   File "/usr/lib64/python3.12/site-packages/pyanaconda/core/configuration/anaconda.py", line 29, in <module>
DEBUG util.py:446:  2023-11-11 08:15:15,519:     from pyanaconda.core.configuration.storage_constraints import StorageConstraints
DEBUG util.py:446:  2023-11-11 08:15:15,519:   File "/usr/lib64/python3.12/site-packages/pyanaconda/core/configuration/storage_constraints.py", line 22, in <module>
DEBUG util.py:446:  2023-11-11 08:15:15,520:     from pyanaconda.core.storage import DEVICE_TYPE_LVM, DEVICE_TYPE_MD, DEVICE_TYPE_PARTITION, \
DEBUG util.py:446:  2023-11-11 08:15:15,520:   File "/usr/lib64/python3.12/site-packages/pyanaconda/core/storage.py", line 22, in <module>
DEBUG util.py:446:  2023-11-11 08:15:15,520:     from blivet import udev
DEBUG util.py:446:  2023-11-11 08:15:15,520:   File "/usr/lib/python3.12/site-packages/blivet/udev.py", line 40, in <module>
DEBUG util.py:446:  2023-11-11 08:15:15,520:     global_udev = pyudev.Context()
DEBUG util.py:446:  2023-11-11 08:15:15,520:                   ^^^^^^^^^^^^^^^^
DEBUG util.py:446:  2023-11-11 08:15:15,520:   File "/usr/lib/python3.12/site-packages/pyudev/core.py", line 59, in __init__
DEBUG util.py:446:  2023-11-11 08:15:15,521:     self._libudev = load_ctypes_library("udev", SIGNATURES, ERROR_CHECKERS)
DEBUG util.py:446:  2023-11-11 08:15:15,521:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG util.py:446:  2023-11-11 08:15:15,521:   File "/usr/lib/python3.12/site-packages/pyudev/_ctypeslib/utils.py", line 51, in load_ctypes_library
DEBUG util.py:446:  2023-11-11 08:15:15,521:     raise ImportError("No library named %s" % name)
DEBUG util.py:446:  2023-11-11 08:15:15,521: ImportError: No library named udev

https://kojipkgs.fedoraproject.org//work/tasks/7606/108887606/root.log

And this could indeed be cause by the ldconfig issue.

The 502's are another issue. :(

If you look at the failed lives there's a bunch of:

https://kojipkgs.fedoraproject.org//work/tasks/7605/108887605/anaconda-packaging.log

But all the composes are using glibc -19... 

INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Installed: glibc-langpack-en-2.38.9000-19.fc40.x86_64 1699427890 d8a6be47fe7016ad7dface701e73a35c675db3d437a0457e7ee1b9acbe4debe0
INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Installed: glibc-common-2.38.9000-19.fc40.x86_64 1699427890 4cccac35cf305a5127853cd4b73535833a3a59b27a21aa3ff2f4fb01272af34b
INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Installed: grub2-common-1:2.06-104.fc40.noarch 1697021632 d2621df494be59cac175e13947c9313cfc87c8369e5c7ac66297a76711c12a31
INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Installed: bash-5.2.15-5.fc39.x86_64 1689777198 4aaaf44111f7fa8a5b7f3f2e217dcad1efa34047461b024b2acbca34037d65b3
INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Configuring (running scriptlet for): bash-5.2.15-5.fc39.x86_64 1689777198 4aaaf44111f7fa8a5b7f3f2e217dcad1efa34047461b024b2acbca34037d65b3
INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Installed: dbus-common-1:1.14.10-1.fc40.noarch 1693764812 ab09201812f377000593ecdf302be6c48dc2235b03ea5a7e125a67e4228333df
INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Configuring (running scriptlet for): dbus-common-1:1.14.10-1.fc40.noarch 1693764812 ab09201812f377000593ecdf302be6c48dc2235b03ea5a7e125a67e4228333df
INFO:dnf.rpm:error: failed to exec scriptlet interpreter /bin/sh: No such file or directory
warning: %post(dbus-common-1:1.14.10-1.fc40.noarch) scriptlet failed, exit status 127
ERROR:dnf.rpm:Error in POSTIN scriptlet in rpm package dbus-common
ERROR:anaconda.modules.payloads.payload.dnf.transaction_progress:Error in POSTIN scriptlet in rpm package dbus-common
INFO:dnf.rpm:error: failed to exec scriptlet interpreter /bin/sh: No such file or directory
error: %prein(xml-common-0.6.3-62.fc40.noarch) scriptlet failed, exit status 127

So, I see -18 started having the issue and... that update initially passed gating, was pushed to stable, then later failed?
-19 passed and was pushed stable.

I'm not sure if it's really the ldconfig issue because ld.so looks at /usr/lib64 directly if something can't be found in the cache. So I expect this to matter only in very obscure situations.

However, looking at https://kojipkgs.fedoraproject.org//work/tasks/7605/108887605/anaconda-packaging.log, I don't see anything installing glibc. The scriptlet for dbus-common seems to run way too early, which means that dbus-common is installed too early. I wonder if there's some new dependency loop. In the previous run, https://kojipkgs.fedoraproject.org//work/tasks/906/108840906/anaconda-packaging.log, the ordering was more reasonable, with bash being installed after glibc, and most packages before that are noarch.

Yeah.

Also there's an extra 44 pacakages: 

< DDEBUG:dnf:timer: depsolve: 993 ms
< INFO:anaconda.modules.payloads.payload.dnf.dnf_manager:The software selection has been resolved (1922 packages selected).
---
> DDEBUG:dnf:timer: depsolve: 1086 ms
> INFO:anaconda.modules.payloads.payload.dnf.dnf_manager:The software selection has been resolved (1966 packages selected).

I poked around a bit more but I can't seem to find it. :( Can look more tomorrow I guess...

I do have a sdiff of the last working workstation live install order vs the failing one... will attach.

zzzz

Possibly tighter diff from openQA last success to first failure:
diff

Note - the apparently 'missing' kernel in the last pass is spurious, it's because that was a test of the kernel, so anaconda didn't need to download it (given how we do this in the tests). The image actually included the same kernel as the first fail. Ditto the apparently 'missing' webkitgtk packages in the first fail - that was a test of webkitgtk itself, so those packages didn't need to be downloaded. I don't think webkitgtk is relevant to the bug.
Edited by adamwill

The problem seems to be condor providing libfmt. I have untagged it and there's an openqa test in progress to confirm.

In the mean while I have filed https://bugzilla.redhat.com/show_bug.cgi?id=2249305 about that.

Metadata Update from @jnsamyak:
- Issue status updated to: Closed (was: Open)
Metadata
None

DOOMED Fedora Rawhide

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.