Describe what you would like us to do:

I noticed that the cloud-image-uploader in Crashlooping. After looking at it more closely it seems that the problem is happening in azure-image-tester container
[2025-12-03 13:44:35,617 fedora_messaging.twisted.consumer ERROR] Received unexpected exception from consumer Consumer(queue=cloud-image-uploader-test-azure, callback=<fedora_image_tester.azure.Consumer object at 0x7f1cf75f4590>)
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/fedora_messaging/twisted/consumer.py", line 217, in _read_one
    yield d
  File "/usr/lib/python3.13/site-packages/twisted/python/threadpool.py", line 269, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/lib/python3.13/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ctx, func, *args, **kw
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/srv/fedora-image-tester/venv/lib64/python3.13/site-packages/fedora_image_tester/azure.py", line 70, in __call__
    delete_after = datetime.now(timezone.utc) + timedelta(hours="8")
                                                ~~~~~~~~~^^^^^^^^^^^
TypeError: unsupported type for timedelta hours component: str
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/fedora_messaging/twisted/consumer.py", line 217, in _read_one
    yield d
  File "/usr/lib/python3.13/site-packages/twisted/python/threadpool.py", line 269, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/lib/python3.13/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ctx, func, *args, **kw
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/srv/fedora-image-tester/venv/lib64/python3.13/site-packages/fedora_image_tester/azure.py", line 70, in __call__
    delete_after = datetime.now(timezone.utc) + timedelta(hours="8")
                                                ~~~~~~~~~^^^^^^^^^^^
TypeError: unsupported type for timedelta hours component: str
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib64/python3.13/logging/handlers.py", line 1088, in emit
    smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
  File "/usr/lib64/python3.13/smtplib.py", line 264, in __init__
    (code, msg) = self.connect(host, port)
                  ~~~~~~~~~~~~^^^^^^^^^^^^
  File "/usr/lib64/python3.13/smtplib.py", line 350, in connect
    self.sock = self._get_socket(host, port, self.timeout)
                ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/smtplib.py", line 321, in _get_socket
    return socket.create_connection((host, port), timeout,
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                                    self.source_address)
                                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/socket.py", line 840, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
               ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/socket.py", line 977, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known
Call stack:
  File "/srv/fedora-image-tester/venv/bin/fedora-messaging", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.13/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.13/site-packages/fedora_messaging/cli.py", line 125, in consume
    _consume(exchange, queue_name, routing_key, callback, app_name)
  File "/usr/lib/python3.13/site-packages/fedora_messaging/cli.py", line 167, in _consume
    reactor.run()
  File "/usr/lib/python3.13/site-packages/twisted/internet/asyncioreactor.py", line 253, in run
    self._asyncioEventloop.run_forever()
  File "/usr/lib64/python3.13/asyncio/base_events.py", line 683, in run_forever
    self._run_once()
  File "/usr/lib64/python3.13/asyncio/base_events.py", line 2050, in _run_once
    handle._run()
  File "/usr/lib64/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.13/site-packages/twisted/internet/asyncioreactor.py", line 269, in _onTimer
    self.runUntilCurrent()
  File "/usr/lib/python3.13/site-packages/twisted/internet/base.py", line 1105, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 926, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 994, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1088, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1963, in _gotResultInlineCallbacks
    _inlineCallbacks(r, gen, status, context)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks
    result = context.run(
  File "/usr/lib/python3.13/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
  File "/usr/lib/python3.13/site-packages/fedora_messaging/twisted/consumer.py", line 236, in _read_one
    _std_log.exception("Received unexpected exception from consumer %r", self)
Message: 'Received unexpected exception from consumer %r'
Arguments: (Consumer(queue=cloud-image-uploader-test-azure, callback=<fedora_image_tester.azure.Consumer object at 0x7f1cf75f4590>),)
[2025-12-03 13:44:35,628 fedora_messaging.cli ERROR] Unexpected error occurred in consumer Consumer(queue=cloud-image-uploader-test-azure, callback=<fedora_image_tester.azure.Consumer object at 0x7f1cf75f4590>): <twisted.python.failure.Failure builtins.TypeError: unsupported type for timedelta hours component: str>
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks
    result = context.run(
        cast(Failure, result).throwExceptionIntoGenerator, gen
    )
  File "/usr/lib/python3.13/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/fedora_messaging/twisted/consumer.py", line 178, in _read
    yield self._read_one(queue_object)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/usr/lib/python3.13/site-packages/fedora_messaging/twisted/consumer.py", line 239, in _read_one
    raise e
  File "/usr/lib/python3.13/site-packages/fedora_messaging/twisted/consumer.py", line 217, in _read_one
    yield d
  File "/usr/lib/python3.13/site-packages/twisted/python/threadpool.py", line 269, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/lib/python3.13/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ctx, func, *args, **kw
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/srv/fedora-image-tester/venv/lib64/python3.13/site-packages/fedora_image_tester/azure.py", line 70, in __call__
    delete_after = datetime.now(timezone.utc) + timedelta(hours="8")
                                                ~~~~~~~~~^^^^^^^^^^^
TypeError: unsupported type for timedelta hours component: str
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib64/python3.13/logging/handlers.py", line 1088, in emit
    smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
  File "/usr/lib64/python3.13/smtplib.py", line 264, in __init__
    (code, msg) = self.connect(host, port)
                  ~~~~~~~~~~~~^^^^^^^^^^^^
  File "/usr/lib64/python3.13/smtplib.py", line 350, in connect
    self.sock = self._get_socket(host, port, self.timeout)
                ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/smtplib.py", line 321, in _get_socket
    return socket.create_connection((host, port), timeout,
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                                    self.source_address)
                                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/socket.py", line 840, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
               ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/socket.py", line 977, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known
Call stack:
  File "/srv/fedora-image-tester/venv/bin/fedora-messaging", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.13/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.13/site-packages/fedora_messaging/cli.py", line 125, in consume
    _consume(exchange, queue_name, routing_key, callback, app_name)
  File "/usr/lib/python3.13/site-packages/fedora_messaging/cli.py", line 167, in _consume
    reactor.run()
  File "/usr/lib/python3.13/site-packages/twisted/internet/asyncioreactor.py", line 253, in run
    self._asyncioEventloop.run_forever()
  File "/usr/lib64/python3.13/asyncio/base_events.py", line 683, in run_forever
    self._run_once()
  File "/usr/lib64/python3.13/asyncio/base_events.py", line 2050, in _run_once
    handle._run()
  File "/usr/lib64/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.13/site-packages/twisted/internet/asyncioreactor.py", line 269, in _onTimer
    self.runUntilCurrent()
  File "/usr/lib/python3.13/site-packages/twisted/internet/base.py", line 1105, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 926, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 994, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1088, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1963, in _gotResultInlineCallbacks
    _inlineCallbacks(r, gen, status, context)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 2108, in _inlineCallbacks
    status.deferred.errback()
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 926, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 994, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1088, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1963, in _gotResultInlineCallbacks
    _inlineCallbacks(r, gen, status, context)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 2108, in _inlineCallbacks
    status.deferred.errback()
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 926, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 994, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1088, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/usr/lib/python3.13/site-packages/fedora_messaging/twisted/consumer.py", line 317, in _read_loop_errback
    self.result.errback(failure)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 926, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 994, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3.13/site-packages/twisted/internet/defer.py", line 1088, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/usr/lib/python3.13/site-packages/fedora_messaging/cli.py", line 333, in errback
    _log.error("Unexpected error occurred in consumer %r: %r", consumer, failure)
Message: 'Unexpected error occurred in consumer %r: %r'
Arguments: (Consumer(queue=cloud-image-uploader-test-azure, callback=<fedora_image_tester.azure.Consumer object at 0x7f1cf75f4590>), <twisted.python.failure.Failure builtins.TypeError: unsupported type for timedelta hours component: str>)
When do you need this to be done by? (YYYY/MM/DD)

Not sure how urgent this is
zlopez commented Dec 3, 2025
@jcline Could you check this out?
jcline commented Dec 3, 2025
Whoops, that was my bad. Fixed it.
I used to also get emails for error-level logs, but it looks like the email hostname is no longer valid. The config uses the Python logging SMTPHandler and the mailhost is configured as "bastion". What should I change it to?
Metadata Update from @jcline:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Dec 3, 2025
kevin commented Dec 3, 2025
I think openshift stopped using any search domains, so it should now be: bastion.rdu3.fedoraproject.org
Metadata

Assignee
None
Tags
Blocking
None
Depending on
None
Priority
Normal
fedora-infrastructure

#12958 cloud-image-uploader pod crashlooping Closed: Fixed Dec 3, 2025 by jcline. Opened Dec 3, 2025 by zlopez.

Describe what you would like us to do:

When do you need this to be done by? (YYYY/MM/DD)

Metadata

cloud dev medium-gain medium-trouble

#12958 cloud-image-uploader pod crashlooping

Closed: Fixed Dec 3, 2025 by jcline. Opened Dec 3, 2025 by zlopez.
