I hit the same thing with the python bodhi-client module today as well:

('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

There's no max workers log messages or anything, but I see from collectd that this seems to have happened when we were around 2500 connections (which was the limit set).
Also, we are in general processing more connections now, so things are larger.

I don't know if it is causing this, but I have increased the limit to 3200 (based on 300 * cpus).

If you all could let me know if you see it again now that would be helpful. (and what time it was).

Thank you - I did some more work yesterday and this error happened only once more - so the issue is not entirely gone, though it appears (?) to be better. I'll watch out for it more today.

Can you tell around when that was? I'd like to look at logs...

No, sorry ... I didn't note the time. And it hasn't happened again today, so I'd say it's "fixed".

And of course a few seconds after I hit the button on the last comment I got this (so at about 19:49 UTC):

Could not execute build: The connection to Pagure failed while getting a list of branches from Pagure. The error was: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

A smattering of similar issues just now (21:42 - 21:43 UTC):

Could not execute build: The connection to BODHI failed while trying to get the active release branches. The error was: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Similar issues at 21:55 UTC (from koji watch-task and uploading to the lookaside cache).

It may not be possible, but if you can run with 'GIT_CURL_VERBOSE=1' on the fedpkg/git calls it might help out (but I realize thats a big dump of data each time).

Same issues here, except I got error 35. I didn't know about this ticket though so I'll keep it in mind for next time!

ok. I think I might be the problem on src... it was hitting a limit because I increased the limit but only did a reload a while back. It needed a restart.

I restarted it now so it's at the higher limit. (It was 16 * 30 but was supposed to be 100 * 30)

Can you let me know it you see it again? That would not explain the bodhi part though yet.

I'm seeing this now.

Can anyone share any recent occurances of this?

Note that this is NOT the 502 on long running connections thing with koji. Thats different.

At least I haven't seen this recently.

Thanks. ;(

Not finding anything in logs sadly.

Would you be able to add debugging to your connections to indicate at least which proxy / ip it connected to?

Anyone still seeing this? It seems like it's happening before it gets to our proxies (at least I can't find any logs with anything that looks like this).

It would be helpful to know what proxy folks are hitting this on, might be network at that location somehow.

I haven't done much since the mass rebuild happened. With that being said, it does fail once in a while but nothing catastrophic like before.

Just happened again a few seconds ago:

('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Though I'm not sure why this is errno 104, and previously I saw errno 35.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/infra/tickets/issues/12975

Please continue any further discussion there.