releng / failed-composes

#5086 Fedora-Rawhide-20230621.n.0 FINISHED_INCOMPLETE
Closed by jnsamyak. Opened by releng.

Closed
Reopen Issue

pungi.global.log

[ERROR   ] [FAIL] Ostree (variant Sericea, arch ppc64le) failed, but going on anyway.
[ERROR   ] Runroot task failed: 102410779. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/ppc64le/Sericea/ostree-6/runroot.log for more details.

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch ppc64le) failed, but going on anyway.
[ERROR   ] Runroot task failed: 102410782. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/ppc64le/Silverblue/ostree-9/runroot.log for more details.

[ERROR   ] [FAIL] Ostree (variant Kinoite, arch ppc64le) failed, but going on anyway.
[ERROR   ] Runroot task failed: 102410784. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/ppc64le/Kinoite/ostree-2/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Kinoite, arch aarch64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 102412137. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/aarch64/Kinoite/ostree_installer-1/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Onyx, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 102412142. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/x86_64/Onyx/ostree_installer-4/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Sericea, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 102412152. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/x86_64/Sericea/ostree_installer-5/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 102412159. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/x86_64/Silverblue/ostree_installer-8/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Kinoite, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 102412167. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/x86_64/Kinoite/ostree_installer-3/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Kinoite, arch ppc64le) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 102412135. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/ppc64le/Kinoite/ostree_installer-2/runroot.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Server, arch *, subvariant Server_KVM) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 102414534. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/aarch64-ppc64le-s390x-x86_64/imagebuild-Server-Server_KVM-qcow2.aarch64-ppc64le-s390x-x86_64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant KDE) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 102414550. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/aarch64/imagebuild-Spins-KDE-raw-xz.aarch64.log for more details.

Variant: Spins, subvariant: KDE task failed. Pinging maintainers: @rdieter @ngompa @marcdeop 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 102414423. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/x86_64/livemedia-Labs-Security.x86_64.log for more details.

Variant: Labs, subvariant: Security task failed. Pinging maintainers: @jsimon @fab @q5sys 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 102414453. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/x86_64/livemedia-Spins-Mate.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 102414454. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/aarch64-x86_64/livemedia-Spins-LXQt.aarch64-x86_64.log for more details.

Variant: Spins, subvariant: LXQt task failed. Pinging maintainers: @zsun @lupinix 

[ERROR   ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: qcow2-raw-xz, arches: aarch64-ppc64le-s390x-x86_64, variant: Cloud, subvariant: Cloud_Base) (task id: 102414414)

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 102414577. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20230621.n.0/logs/aarch64/imagebuild-Spins-LXQt-raw-xz.aarch64.log for more details.

Variant: Spins, subvariant: LXQt task failed. Pinging maintainers: @zsun @lupinix

Compose Total time: 4:02:39
Compose phase INIT time: 0:00:25.
Compose phase PKGSET time: 0:19:51.
Compose phase WEAVER: FAILED.
Compose phase BUILDINSTALL time: 1:50:41.
Compose phase GATHER time: 1:56:59.
Compose phase OSTREE time: 0:30:31.
Compose phase OSTREE_INSTALLER time: 1:12:57.
Compose phase CREATEREPO time: 0:07:19.
Compose phase REPOCLOSURE time: 0:01:34.
Compose phase CREATEISO time: 0:08:54.
Compose phase IMAGE_BUILD time: 1:33:45.
Compose phase LIVE_MEDIA time: 1:09:01.
Compose phase IMAGE_CHECKSUM time: 0:03:11.
Compose phase TEST time: 0:00:01.

So, we have a few categories of stuff here:

  • A lot of ppc64le failures are because Firefox build is currently disabled on that arch
  • The other ostree installer image failures are all due to the mysterious 502 error issue we've had for a while
  • Two 'image' tasks - which both ran on ppc64le hosts, not sure if that's significant - failed with "ApplianceError: ImageFactory functions not available"
  • Security spin failed due to a dep issue: nothing provides perl-Net-Whois-IP needed by dnsenum-1.2.4.2-25.fc38.noarch
  • MATE_Compiz failed due to a conflict: file /usr/share/help/pt/mate-search-tool/index.docbook conflicts between attempted installs of mate-dictionary-1.26.1-1.fc39.x86_64 and mate-utils-common-1.26.1-1.fc39.noarch
  • LxQt failed due to a dep issue: nothing provides libprocps.so.8()(64bit) needed by lxqt-session-1.2.0-2.fc38.x86_64
  • Cloud_Base failed on s390x due to a problem creating a btrfs fs while doing a full trim (I'm gonna bet this is an endian booboo, it looks a lot like one):
    07:47:22,757 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:stderr[6]: Invalid mapping for 1081344-1097728, got 17592186044416-17592190238720
    07:47:22,757 WARNING org.fedoraproject.Anaconda.Modules.Storage:Couldn't map the block 1081344
    07:47:22,757 WARNING org.fedoraproject.Anaconda.Modules.Storage:ERROR: cannot read chunk root
    07:47:22,757 WARNING org.fedoraproject.Anaconda.Modules.Storage:ERROR: open ctree failed

It looks like the "ImageFactory functions not available" error comes from https://pagure.io/koji/pull-request/1346# , and implies something is somehow missing on the builder. The logged error message suggests "ImageFactory features require the following dependencies: pykickstart, imagefactory, oz and possibly python-hashlib", the actual check we're failing is this:

try:
    from imgfac.BuildDispatcher import BuildDispatcher
    from imgfac.Builder import Builder
    from imgfac.PluginManager import PluginManager
    from imgfac.ReservationManager import ReservationManager
    plugin_mgr = PluginManager('/etc/imagefactory/plugins.d')
    plugin_mgr.load()
    from imgfac.ApplicationConfiguration import ApplicationConfiguration
    from imgfac.PersistentImageManager import PersistentImageManager
    from imgfac.BaseImage import BaseImage
    from imgfac.TargetImage import TargetImage
    # NOTE: import below requires Factory 1.1.7 or higher
    from imgfac.FactoryUtils import qemu_convert_cmd
    ozif_enabled = True
except ImportError:  # pragma: no cover
    ozif_enabled = False

The affected builder is buildvm-ppc64le-16.iad2.fedoraproject.org , it'd be good if someone can check it and maybe re-run the role...
Edited by adamwill 
[sharkcz@buildvm-ppc64le-16 ~][PROD-IAD2]$ python3
Python 3.11.3 (main, May 24 2023, 00:00:00) [GCC 13.1.1 20230511 (Red Hat 13.1.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from imgfac.BuildDispatcher import BuildDispatcher
>>> from imgfac.Builder import Builder
>>> from imgfac.PluginManager import PluginManager
>>> from imgfac.ReservationManager import ReservationManager
>>> plugin_mgr = PluginManager('/etc/imagefactory/plugins.d')
>>> plugin_mgr.load()
>>> from imgfac.ApplicationConfiguration import ApplicationConfiguration
>>> from imgfac.PersistentImageManager import PersistentImageManager
>>> from imgfac.BaseImage import BaseImage
>>> from imgfac.TargetImage import TargetImage
>>> from imgfac.FactoryUtils import qemu_convert_cmd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/site-packages/imgfac/FactoryUtils.py", line 8, in <module>
    import guestfs
  File "/usr/lib64/python3.11/site-packages/guestfs.py", line 73, in <module>
    import libguestfsmod
ImportError: libyajl.so.2: cannot open shared object file: No such file or directory

https://bugzilla.redhat.com/show_bug.cgi?id=2216033 for the lxqt problem. I fixed the Security spin problem. Will look at MATE_Compiz and file the btrfs endian booboo upstream I guess.

so the buildvm-ppc64le-16 builder is broken, yajl is installed per rpm/dnf, but its files are missing, similar to what we saw last week

[sharkcz@buildvm-ppc64le-16 ~][PROD-IAD2]$ rpm -V yajl
missing     /usr/bin/json_reformat
missing     /usr/bin/json_verify
...

@kevin ^^
Edited by sharkcz

https://bugzilla.redhat.com/show_bug.cgi?id=2216460 for MATE, https://bugzilla.redhat.com/show_bug.cgi?id=2214522 already exists for the btrfs-progs issue , there is a fix upstream, our package just needs updating.

I've reinstalled that package, but there's other things missing.

I think I am going to reblance the ppc64le builders, reduce their number and reinstall them all...

there is much more missing as rpm -Va shows :-(
Edited by sharkcz

Yep. https://pagure.io/fedora-infrastructure/issue/11378 will try and get it done today if the fires stay low.

thanks, not sure what's going on there, it's the same problem we had a ~week ago with another builder ...

Metadata Update from @jnsamyak:
- Issue status updated to: Closed (was: Open)
Metadata
None

FINISHED_INCOMPLETE Fedora Rawhide

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.