#11239 Playbook manual/oci-registry-prune.yml failing to run to completion
Closed: Duplicate by kevin. Opened by dkirwan.

Describe what you would like us to do:


While attempting to follow SOP: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/registry/ seems a playbook needs updating: manual/oci-registry-prune.yml

When do you need this to be done by? (YYYY/MM/DD)


Preventing us running manual pruning jobs on the image registry, so sooner is good.

Task is failing at two locations:

TASK [Find and Delete 30 days old OCI images] ************************************************************************************************************************************************************************************************
Friday 14 April 2023  10:17:11 +0000 (0:00:00.260)       0:00:00.260 **********
Friday 14 April 2023  10:17:11 +0000 (0:00:00.259)       0:00:00.259 **********
skipping: [oci-candidate-registry01.stg.iad2.fedoraproject.org]
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: 'NoneType' object is not subscriptable
fatal: [oci-candidate-registry01.iad2.fedoraproject.org -> compose-x86-01.iad2.fedoraproject.org]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>\n  File \"<stdin>\", line 99, in _ansiballz_main\n  File \"<stdin>\", line 47, in invoke_module\n  File \"<frozen runpy>\", line 226, in run_module\n  File \"<frozen runpy>\", line 98, in _run_module_code\n  File \"<frozen runpy>\", line 88, in _run_code\n  File \"/tmp/ansible_delete_old_oci_images_payload_vap0j3jb/ansible_delete_old_oci_images_payload.zip/ansible/modules/delete_old_oci_images.py\", line 166, in <module>\n  File \"/tmp/ansible_delete_old_oci_images_payload_vap0j3jb/ansible_delete_old_oci_images_payload.zip/ansible/modules/delete_old_oci_images.py\", line 144, in main\nTypeError: 'NoneType' object is not subscriptable\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

and

TASK [Run registry garbage collection to reclaim disk space (stg)] ***************************************************************************************************************************************************************************
Friday 14 April 2023  10:17:43 +0000 (0:00:00.119)       0:00:32.301 **********
Friday 14 April 2023  10:17:43 +0000 (0:00:00.119)       0:00:32.301 **********
fatal: [oci-candidate-registry01.stg.iad2.fedoraproject.org]: FAILED! => {"changed": true, "cmd": ["registry", "garbage-collect", "/etc/docker-distribution/registry/config.yml"], "delta": "0:00:00.017506", "end": "2023-04-14 10:17:44.805204", "msg": "non-zero return code", "rc": 1, "start": "2023-04-14 10:17:44.787698", "stderr": "failed to garbage collect: failed to mark: filesystem: Path not found: /docker/registry/v2/repositories", "stderr_lines": ["failed to garbage collect: failed to mark: filesystem: Path not found: /docker/registry/v2/repositories"], "stdout": "", "stdout_lines": []}

Yep. This was noted in https://pagure.io/fedora-infrastructure/issue/11195 already. ;)

I'd love someone to dig in and figure out what broke in our ansible library there...

Metadata Update from @kevin:
- Issue close_status updated to: Duplicate
- Issue status updated to: Closed (was: Open)

Metadata