I'll note that ipa01 seems fixed, but 02/03 aren't. :(

I didn't noticed 02/03 are still having issues.

Those failing cron jobs are complaining about space, so it's not an issue with cronjob itself, we just need to clear some space.

So we have 13 G available on ipa01, but the whole db has 16 G. So it can't back it up. I assume it got bloated during the spam :/

For ipa02 17 G vs 20 G free, so it should have enough space to create backup. Same for ipa03. It's strange that all of them are failing then.

I remember @gwmngilfen saying something about out of inode error on ipa machines, so that could be it.

It really needs more space, I tried to do backup manually and it consumed all available space during that.

Hm, the backups are being kept in /var/ipa/backups and taking up space. I thought those are moved to backup01 and then deleted.

And I see only ipa01 on backup01:/fedora_backups/, but all of them are already there.

It's keeping the latest 7 backups in cron https://pagure.io/fedora-infra/ansible/blob/main/f/roles/ipa/server/files/data-only-backup.sh, so we either give it more space or lower the number of kept backups on the machine.

Yeah, we keep the last 7 and we back that up to backup01 (on ipa01 only, because all of the cluster servers should be the same due to replication).

I guess we could keep less there, but it seemed handy to be able to easily restore.

Note that stage users were not being deleted correctly, so perhaps after thats fixed things will work better...

They are deleted now, but the db size is still the same. I tried to cleanup the tombstones, but it didn't changed the db size. From what I found we probably have to do re-indexing as well, but that needs to be done on stopped db.

TL;DR;: The backups are working again and there should be enough space to keep 7 of them.

With the help from LDAP folks I was able to reduce the size of LDAP db to 8G from 16G.

The problem why the size of the DB was so big was the replication_changelog. We still had some replication entries for ipaXX.aid2, which it couldn't reach and for some reason ipa01.rdu3 was there twice. This caused the replication to fail and the changelog to grow incrementally till the migration, the users created by spammers were just the last drop.

So I cleared the RUVs (Replication Update Vectors) and that helped me to finally clean the db manually.

Cool. Great detective work!

Shall we close this?

This ticket was originally opened for adding these checks to zabbix monitoring, backups were just something that we resolved during that. I don't think the checks are in place, so let's keep this open.

Oh, indeed. ok.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/infra/tickets/issues/12992

Please continue any further discussion there.