Would it be a viable option to send the e-mail to some mailing list of your choice?

Is there any way we could use something like Zabbix to get alerting for these errors? One challenge is that the pool of people who are both capable of working on this and informed enough to work on these challenges is very low. We should consider more ways to get added visibility into the localization stack in Fedora Infrastructure.

Also, if Zabbix could help with making a Matrix bot that alerts in a specific Matrix room when there are issues, e.g. #localization:fedoraproject.org, that would be very helpful too.

zabbix is open to check for everyone, but these are logs from cron jobs and we don't want to have them saved them in zabbix.

ok then, what would be the required steps to have those logs on zabbix?
if cron jobs are not suitable, what would be?

Well, I am not sure. zabbix is a monitoring system, so it runs a check and acts on the output of that check.

I suppose if the logs were stored somewhere it could run a check over them and alert if errors were found.

@gwmngilfen any ideas here?

So, yes, Zabbix could help here.
- The cron jobs could be updated to send a result to Zabbix (pass/fail, or even more complex data) using the Zabbix agent or zabbix_sender
- We can set Zabbix up to act on that, and send alerts to a Matrix room
- The setup for that is a little fiddly, but it can be Ansibilised once we're happy with it.
- The main challenge would be not reporting that event in #noc, if we're happy to have it reporting in both then this gets a lot simpler
- However, looking at the length of that output, I don't think we want to be storing it in the Zabbix DB, so you'd want the check to make it clear who/what needs to act on it.

I can help with the Zabbix-side stuff for sure - but the first step would be modifying the cron jobs to output something useful to Zabbix. Then we either:
- Put that output in a file and tell the Zabbix agent to read it, or
- Add the Zabbix call to the end of the cronjob - but that would mean ensuring that it does run, with error handling, etc

I lean to the first option - lets have the cron dump output to /var/log or similar, and and then we write a new script that checks both the output and the timestamp - if the file is too old, or has bad content, we send a warning.

How does that feel?

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/infra/tickets/issues/13030

Please continue any further discussion there.