#12477 Bodhi API timeouts
Closed: Fixed with Explanation by kevin. Opened by petersen.

In the last couple of weeks I have been seeing Bodhi API timeouts occurring multiple times for a period of time.

I hope someone could look into it and find out what is going on.

I am primarily using the Releases endpoint (for example it is used by fbrnch and also dl-fedora, etc). I think I need to make them more robust for this, but this hasn't really been a problem at all until now.

Funny thing is it seems to depend on the calling client:
curl seems okay, but not /usr/bin/bodhi or /usr/bin/bodhi-cli.

Is it rejecting connections in some circumstances?

$ bodhi releases info FEDORA-42
<no response>

I mentioned these timeouts a few times on matrix including #devel


Thanks for asking in matrix about cookies etc.

I think fbrnch is accessing the bodhi API directly without any authentication or cookies.
But I think it is using http/1.1 and not http/2 like curl

However it also happens with the python bodhi client: does that use cookies or auth?

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, bodhi, medium-gain

The bodhi cli can auth with oauth... but if you are seeing it with curl, the auth angle isn't very likely.

When you get timeouts could you get curl -v output? that would tell us what proxy it was connecting to and possibly provide some hints.

I think there were some recent infra OS upgrades? I am wondering if it could relate to that.

It may not just be Bodhi indeed: right now download is timing out for me too (via Haskell).
edit: also pagure.io right now

Just to be clear curl always seems to work, but not python/haskell apparently it seems.

Here is a simple reproducer for me with python:

import requests
print(requests.get("https://bodhi.fedoraproject.org/releases/F42").json())

which always hangs.

Compare with: curl https://bodhi.fedoraproject.org/releases/F42

Or is this some kind of client filtering?

Both fedpkg request-branch and fbrnch request-branches are also timing out for me currently. edit: finally fedpkg completed (after 3 min).

Can you try and identify what proxy you are hitting when this happens?

either from ss -an output or tcpdump or there might be some way in requests.

I am really thinking it is somehow a proxy misbehaving as we are not seeing this issue very widely. ;(

Oh, you could also try just overriding the proxy in /etc/hosts:

38.145.60.20 bodhi.fedoraproject.org

does doing that work?

We also fixed a mailman query that was saturating I/O on the database server.

Are you still seeing this issue now?

Hopefully things are better/solved. Please re-open if you still are seeing anything.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Metadata