I tried to reproduce this on pkgs01 directly and when running the cron job manually it will not produce this error :/

But looking at the error itself it looks like instead of evaluating the /srv/git/rpms/*.git in for cycle, it works with it as a string. I would recommend using ls /srv/git/rpms/*.git instead.

But I don't understand what is the difference between running cron job and running the same in root shell.

Usually the environment is different. It can also be selinux-related as I think cron jobs run under a different context.

Added some prints yesterday and here are some results: https://pagure.io/releng/pull-request/12851#

It seems like the cron job would be duplictated, first run fails, second run passes.
No idea why.

Added some more debug prints here, please merge: https://pagure.io/releng/pull-request/12854

Oh wow... I see whats happening. The one thats failing is the staging one. It's for some reason not showing that it's coming from staging, but if you look in the email headers you can see it's coming from pkgs01.stg.

And that explains the problem too as I didn't copy the /srv/git/ data over from iad2 to rdu3 staging. :(

I guess we should copy git from prod to stg?

Good catch!
Do we even want it to run on staging? Before it was not running there.

If we want to have it running on staging, we would need to adjust the script to output staging retired packages to staging lookaside.
I see it exists: https://src.stg.fedoraproject.org/lookaside/
For the moment, if run on both prod and stg, the content on prod lookaside would be replaced by whichever script finishes later, that would not be good.

Here PR removing all the added prints: https://pagure.io/releng/pull-request/12854#request_diff

Well, are there any things that might consume it in staging that we might want to test? If so, we could run it there... if not, not much point.

I am copying /srv/git from pkgs01 to pkgs01.stg now... so it should have content soon.

Not sure if there's too much of a point to have it running on stg, although it would not be much work to make it possible.

@humaton Do we want to run the get_retired_packages.sh on staging?

Cool, it actually didn't need any adjustments: https://src.stg.fedoraproject.org/lookaside/

I don't see any cron-job logs from pkgs01, seems that the issue is solved.

Hurray. Thanks for tracking this down...