fedora-infra / toddlers

#377 cleaning_packager_groups toddler on staging is filling logs with error
Opened by zlopez. Modified

Open
Close Issue

Today I tried to resolve why log for OpenShift staging worker is being filled so quickly and most of the lines were generated by cleaning_packager_groups toddler with the following error:

2025-11-11 14:46:47,021 - [ERROR toddlers.base] Toddler 'clean_packagers_groups' failed to process message id: c2f8ec0c-5e53-49d3-a0d9-13ab41985032 -- putting it back in the queue
Traceback (most recent call last):
  File "/opt/app-root/src/toddlers/base.py", line 74, in __call__
    return self.process(self._config, message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/src/toddlers/plugins/cleaning_packager_groups.py", line 221, in process
    with self._ipa_session:
  File "/opt/app-root/lib64/python3.11/site-packages/ipalib/__init__.py", line 996, in __enter__
    conn.connect()
  File "/opt/app-root/lib64/python3.11/site-packages/ipalib/backend.py", line 69, in connect
    conn = self.create_connection(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/ipalib/rpc.py", line 938, in create_connection
    command([], {})
  File "/opt/app-root/lib64/python3.11/site-packages/ipalib/rpc.py", line 1150, in _call
    return self.__request(name, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/ipalib/rpc.py", line 1144, in __request
    raise error_class(**kw)
ipalib.errors.CCacheError: Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2598845123): No credentials cache found�
2025-11-11 14:46:47,022 - [WARNING fedora_messaging.twisted.consumer] Returning message id c2f8ec0c-5e53-49d3-a0d9-13ab41985032 to the queue

What is strange that the message was processed in the end. I scaled the toddler to 0 on staging till this error is resolved.

Thanks!
It's strange that the toddler could not authenticate. Fortunately on production it's working well.
I will add a rule to try to connect 5-10 and then stop trying, to limit the amount of logs if it happens next time.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.