koji-fedmsg-plugin

#16 plugin errors on load on fedora 35
Opened by kevin. Modified

Open
Close Issue

I've updated koji01.stg to f35, but this plugin fails on load: 

 2021-12-06 23:24:47,709 [WARNING] m=None u=None p=16672 r=?:? koji: Setting resource limit: RLIMIT_AS = [10737418240, -1]
 2021-12-06 23:24:47,887 [ERROR] m=None u=None p=16672 r=?:? koji.plugin: Loading plugin fedmsg-koji-plugin failed
 2021-12-06 23:24:47,890 [ERROR] m=None u=None p=16672 r=?:? koji.plugins: Traceback (most recent call last):
   File "/usr/share/koji-hub/kojixmlrpc.py", line 510, in load_plugins
     tracker.load(name)
   File "/usr/lib/python3.10/site-packages/koji/plugin.py", line 103, in load
     plugin_spec.loader.exec_module(plugin)
   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
   File "/usr/lib/koji-hub-plugins/fedmsg-koji-plugin.py", line 18, in <module>
     import fedora_messaging.api
   File "/usr/lib/python3.10/site-packages/fedora_messaging/api.py", line 7, in <module>
     from twisted.internet import reactor, defer
   File "/usr/lib/python3.10/site-packages/twisted/internet/reactor.py", line 38, in <module>
     from twisted.internet import default
   File "/usr/lib/python3.10/site-packages/twisted/internet/default.py", line 55, in <module>
     install = _getInstallFunction(platform)
   File "/usr/lib/python3.10/site-packages/twisted/internet/default.py", line 43, in _getInstallFunction
     from twisted.internet.epollreactor import install
   File "/usr/lib/python3.10/site-packages/twisted/internet/epollreactor.py", line 19, in <module>  
     from twisted.internet import posixbase
   File "/usr/lib/python3.10/site-packages/twisted/internet/posixbase.py", line 20, in <module>   
     from twisted.internet import error, udp, tcp
   File "/usr/lib/python3.10/site-packages/twisted/internet/tcp.py", line 37, in <module>
     from twisted.internet._newtls import (
   File "/usr/lib/python3.10/site-packages/twisted/internet/_newtls.py", line 19, in <module>
     from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
   File "/usr/lib/python3.10/site-packages/twisted/protocols/tls.py", line 66, in <module>
     from twisted.internet._sslverify import _setAcceptableProtocols
   File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 1834, in <module>
     defaultCiphers = OpenSSLAcceptableCiphers.fromOpenSSLCipherString( 
   File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 1811, in fromOpenSSLCipherString
     _expandCipherString(
   File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 1759, in _expandCipherString
     return tuple(OpenSSLCipher(cipher) for cipher in ciphers)
   File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 1759, in <genexpr>
     return tuple(OpenSSLCipher(cipher) for cipher in ciphers)
 TypeError: OpenSSLCipher() takes no arguments

Oddly, it looks like something in twisted / fedora-messaging? at a loss.

You can edit /etc/koji-hub/hub.cfg and re-add fedmsg-koji-plugin to the Plugins line to duplicate it.

CC: @abompard @pingou

Oh, this is blocking my upgrading to f35 in prod (which I only have this week for before I am on PTO). :(

Looks to be coming from twisted

The code seems to still be there in "trunk": https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/_sslverify.py#L1758-L1761

Hmm, can't reproduce it on the koji.stg host. I'm just restarting apache, is there something else to do to reproduce it?
It may be because there's no space left on /mnt/fedora_koji, what can I do to clean it up a bit?

I've cleaned up space, re-enabled the plugin and restarted it.

It should be erroring away now.

Still looking into it. It's really weird, some incompatbility between the attrs package and the way koji sets up plugins maybe. I can reproduce it by writing this dummy koji plugin:

import attr
@attr.s
class TestClass:
    foo = attr.ib()
TestClass("debug")
Edited by abompard

I tried downgrading the attrs package but no change.

So...

[07:43:25] nirik: is it just me, or the rights seems a bit off?
[07:43:28] --w----r-T. 1 root root 9004 Aug 26 2020 /usr/lib/koji-hub-plugins/fedmsg-koji-plugin.py
[07:43:49] (that's on koji01.stg)
[07:45:40] mode: 644
[07:46:23] yep
[07:46:31] should be mode: 0644
[07:46:36] or mode: "644"

So, I did a chmod 0644 on it and restarted and... it seems like it's working now?

Ah, it's disabled, so the perms had nothing to do with it. :)

It seems to be a much deeper problem, some references that are similar to the problem we're having:

  • https://github.com/GrahamDumpleton/mod_wsgi/issues/729
  • https://github.com/poljar/weechat-matrix/issues/293

I don't know what's going on, I'll investigate further.

Darknao suggests the following workaround:
- patching twisted to use @attr.define() instead of @attr.s
- using a mod_wsgi version built on Python 3.9. It does not seem to be available in Fedora so we'd have to build that package.

The second workaround seems more error-proof, I can build that package if we want to go this route, until a proper fix is discovered by someone somewhere.

I've opened a bugzilla ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2030621

Miro is looking at the issue, which seems to be a python bug indeed. In the meantime I have built mod_wsgi for python3.9, if we need a fix more urgently:

https://abompard.fedorapeople.org/misc/python3.9-mod_wsgi/

I have confirmed that it works on my F35 VM. Should I build it in koji's infra tag and deploy it on koji01.stg?

Huh, how does that interact with the rest of the stack thats running python 3.10 tho?

I'm not sure I understand how it works, but I guess if it does, sure, go ahead and build it and update stg.

Looks like there's a fix for python3.10 proposed. Hopefully it will go out in a stable build soon and I can update.

Stg is running the scratch build
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.