fedora-infrastructure

#11220 ODCS is not processing composes
Closed: Upstream by kevin. Opened by sgallagh.

Closed: Upstream
Reopen Issue

NOTE

If your issue is for security or deals with sensitive info please
mark it as private using the checkbox below.

Describe what you would like us to do:

% odcs --watch create-raw-config --compose-type=production --label=Alpha-0.`date "+%s"` eln eln
Waiting for command create-raw-config on compose 27502 to finish.
{
    "arches": "x86_64",
    "base_module_br_name": null,
    "base_module_br_stream": null,
    "base_module_br_stream_version_gte": null,
    "base_module_br_stream_version_lte": null,
    "builds": null,
    "compose_type": "production",
    "flags": [],
    "id": 27502,
    "koji_event": 109123661,
    "koji_task_id": null,
    "label": "Alpha-0.1680528260",
    "lookaside_repos": "",
    "modular_koji_tags": null,
    "module_defaults_url": null,
    "modules": null,
    "multilib_arches": "",
    "multilib_method": 0,
    "owner": "sgallagh",
    "packages": null,
    "parent_pungi_compose_ids": null,
    "pungi_compose_id": null,
    "pungi_config_dump": null,
    "removed_by": null,
    "respin_of": null,
    "result_repo": "https://odcs.fedoraproject.org/composes/odcs-27502/compose/Temporary",
    "result_repofile": "",
    "results": [
        "repository"
    ],
    "scratch_build_tasks": null,
    "scratch_modules": null,
    "sigkeys": "",
    "source": "eln#eln",
    "source_type": 5,
    "state": 4,
    "state_name": "failed",
    "state_reason": "Error while generating compose: 'eln'",
    "target_dir": "default",
    "time_done": null,
    "time_removed": null,
    "time_started": "2023-04-03T13:24:20Z",
    "time_submitted": "2023-04-03T13:24:20Z",
    "time_to_expire": "2023-04-04T13:24:21Z",
    "toplevel_url": "https://odcs.fedoraproject.org/composes/odcs-27502"
}

Additionally, when I log into odcs-backend-releng01.iad2.fedoraproject.org, I see the following in the journal:

Apr 03 13:23:06 odcs-backend-releng01.iad2.fedoraproject.org sh[833084]: celery multi v5.2.6 (dawn-chorus)
Apr 03 13:23:06 odcs-backend-releng01.iad2.fedoraproject.org sh[833084]: > Starting nodes...
Apr 03 13:23:06 odcs-backend-releng01.iad2.fedoraproject.org sh[833084]:         > worker@odcs-backend-releng01.iad2.fedoraproject.org: OK
Apr 03 13:23:06 odcs-backend-releng01.iad2.fedoraproject.org systemd[1]: Started odcs-celery-backend.service - ODCS Celery backend service.
Apr 03 13:23:07 odcs-backend-releng01.iad2.fedoraproject.org /usr/lib/python3.10/site-packages/celery/__main__.py[833089]: Can't generate OpenAPI specification because of: No module named 'apispec'
                                                                                                                           Traceback (most recent call last):
                                                                                                                             File "/usr/lib/python3.10/site-packages/odcs/server/views.py", line 61, in <module>
                                                                                                                               from . import openapi
                                                                                                                             File "/usr/lib/python3.10/site-packages/odcs/server/openapi.py", line 1, in <module>
                                                                                                                               from apispec import APISpec
                                                                                                                           ModuleNotFoundError: No module named 'apispec'

When do you need this to be done by? (YYYY/MM/DD)

Immediately, it is blocking composes of ELN.

It looks like the odcs packages were 'updated' in Fedora 36 on Build Date : Thu 02 Mar 2023 03:12:19 UTC and then upgraded on Install Date: Tue 28 Mar 2023 22:25:18 UTC during the mass update.

I went and looked and I didn't quickly find any python module (even the openapi ones) which had a file or directory named apispec in any Fedora release. [I found some php ones and golang but nothing in python.] In Fedora 38 there are a set of modules called openapi_core

python3-openapi-core+django-0.16.5-2.fc38.noarch.rpm
python3-openapi-core+falcon-0.16.5-2.fc38.noarch.rpm
python3-openapi-core+flask-0.16.5-2.fc38.noarch.rpm
python3-openapi-core+requests-0.16.5-2.fc38.noarch.rpm
python3-openapi-core+starlette-0.16.5-2.fc38.noarch.rpm
python3-openapi-core-0.16.5-2.fc38.noarch.rpm
python3-openapi-schema-validator-0.4.3-1.fc38.noarch.rpm
python3-openapi-spec-validator-0.5.5-1.fc38.noarch.rpm
python3-sphinxcontrib-openapi-0.7.0-7.fc38.noarch.rpm

However when I tried to install odcs in a Fedora 38 VM, none of them were pulled in by the dnf transaction.

Mar 28 is the last time we had a compose kick off, so that tracks.

Did one of ODCS's dependencies grow a new dep on apispec?

https://pagure.io/odcs/c/4f92d82ef72d1758c50839e1464203cf1f0b9e30

is what changed the code which seems to be triggered.

So the previous working version of odcs was 0.4.0 but the only version dnf finds is the much older version 0.3.6 which was shipped in F36. My suggestion would be to find the 0.4.0 in koji and compare or reinstall with what is installed.

@jkaluza Can you help here at all?

I downgraded to 0.4.0, but that did not seem to help. Same error.

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble

I just hacked it to get it to not use apisepc (which is supposed to be optional from what I can see)

and it's running now.

Can you try composing?

We of course need to sort the problem out/get it fixed.

Compose 27505 has started successfully.

It's accepting requests now, but immediately failing them without doing any work.

This is probably related to the 0.4.0 -> 0.5.0 upgrade I asked @kevin to do.

Yeah, downgraded back to 0.4.x and it's processing again.

This is failing again in the same manner.

The playbook is constructed to not be idempotent.... it specifies 'latest' for packages. someone ran the playbook and it re-upgraded to the broken newer version. ;(

I have downgraded it again.

This seems to be tracked at https://pagure.io/odcs/issue/615 upstream.

Closing this in favor of that.

Metadata Update from @kevin:
- Issue close_status updated to: Upstream
- Issue status updated to: Closed (was: Open)
Metadata
None

high-gain medium-trouble

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.