Mailing list moderation of the freeipa-users list, https://lists.fedoraproject.org/admin/lists/freeipa-users.lists.fedorahosted.org/held_messages , consistently fails with a 503 Error, Backend fetch.
Most moderation involves discarding spam and even on failure somtimes the number to moderate goes down even with a failure, but I'm concerned about losing messages to the list that need to be approved.
This could be related to the datacenter move and will clear itself up.
It is a slow boil but the longer it takes to address the worse it will become. It will just happen slowly assuming the current rate of spam continues.
I was able to open the URL without error, could you collaborate what exactly doesn't work?
Metadata Update from @zlopez: - Issue priority set to: Waiting on Reporter (was: Needs Review) - Issue tagged with: lists
The URL works but if you try to moderate, say select the first 10 and discard it times out after some time (one minute?) and nothing is apparently done.
I tried approving one of the held messages and I saw it show on the list but it still appeared in the moderation list. Then it went away, I think? I'm worried that messages are going to disappear.
I can confirm that, I got 503 when trying to reject one of the spams. I will need to investigate the mailman logs to see what is happening.
Metadata Update from @zlopez: - Issue assigned to zlopez - Issue priority set to: Waiting on Assignee (was: Waiting on Reporter) - Issue tagged with: Needs investigation, medium-gain, ops
When I tried to debug the issue I restarted the mailman3 service and the moderation works now. Not sure what happened. @rcritten Could you check if it's working for you? Feel free to close this ticket and in case it stops working again reopen it.
I tried and got a 503.
You are right, I tried it again now and it throws 503, but I was able to reject 3 mails during testing. I will try to investigate more.
It's strange, I enabled the debug log again and the moderation works. I disabled it and it still works. I assume it will stop working after some time, but it's really hard to debug
I'm not able to reproduce it anymore, I rejected almost all the spam e-mails in held messages and still didn't trigger the issue :/
It's working for me now as well. Ok I guess we can close this? My list gets a fair amount of spam. If it gets stuck again I'll re-open.
Let me close it for now as insufficient data and feel free to re-open when the issue happens again.
Metadata Update from @zlopez: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
FWIW now the entire service is down.
$ curl https://lists.fedoraproject.org/admin/lists/freeipa-users.lists.fedorahosted.org/held_messages <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>503 Service Unavailable</title> </head><body> <h1>Service Unavailable</h1> <p>The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.</p> </body></html>
You probably hit https://pagure.io/fedora-infrastructure/issue/12651, which takes some time to restart.