It's been running only for 2 hours now, not 8, sorry. But still.. I'll try to check how long it eventually took. Edit: it finished after 02:29:40

Hmm, okay. I'll try to have a look if this can be optimized. If I understand correctly, we should collect all rpms to be deleted and then run a single repoquery for all those packages. Alternatively, we could probably read rpm headers to get that information about the srpms it comes from.

Perhaps, I haven't had a closer look, yet.

I was experimenting with get_srpm_cmd = ["rpm", "-qp", "--queryformat" ,"%{SOURCERPM}", rpm] and it saves around 1/4 of time (EDIT: it saves 1/4 time on the test suite but in the test suite there are various calls related to tests so the real-usecase might prove to be faster). It, however, assumes the srpm will be placed in the same directory as rpm. The current solution assumes it too but could be probably tweaked to be more generic.

By any chance, could you compare runtime of the current version and this one:

https://pagure.io/prunerepo/c/f15f18e0c78f724262921246493e0117c9c81263?branch=faster-srpm-delete
https://copr.fedorainfracloud.org/coprs/clime/prunerepo-faster-srpm-delete/

on some large repo?

Do you have access to some larger repository to try this on? Possibly with --dry-run. It would be awesome to try if the tweak helps.

The best would be to run old prunerepo and new prunerepo on the same repo and then diff the resulting repos if they differ. I can try myself but it would help to have a tarball of actual repo from copr.

I'm not sure it is correct thing to do. Mock actually doesn't use one rpmbuild command (-ba) to generate both source RPM and binary RPM. It first generates source RPM, then it is installed, and then only binary RPMs are re-built:
https://github.com/rpm-software-management/mock/blob/2ae77b8393174c78286197673c283e5a40b025fe/mock/py/mockbuild/backend.py#L688

With all the experiments I've seen people are doing, the SOURCERPM header might lie. And I'm not sure if that is always available.

Why do not we consult the repodata for the list of source RPMs there? The other
benefit would be that we wouldn't have to actually read all the deleted RPMs (I/O).

Slightly related is the issue #9. If we analyzed the metadata only, we wouldn't have to
touch the RPM files at all IMO.

Hello, I tried e.g.:

https://download.copr.fedorainfracloud.org/results/decathorpe/sequoia/fedora-rawhide-x86_64/01885758-rust-buffered-reader/

and rpms there correctly the source rpm so it seems this would work. It seems to be generated in rpm code from NVR read from spec file and attaching .src.rpm extension to it..

Slightly related is the issue #9. If we analyzed the metadata only, we wouldn't have to
touch the RPM files at all IMO.

But metadata are being analyzed at the moment by dnf repoquery command. The command is quite slow and it's being invoked way too often.

and rpms there correctly the source rpm so it seems this would work.

I think it should work in most cases, but it is not 100% guaranteed.

But metadata are being analyzed at the moment by dnf repoquery command. The command is quite slow and it's being invoked way too often.

I think that the main query should be done only once per one repository, not for each source.RPM.

and rpms there correctly the source rpm so it seems this would work.

I think it should work in most cases, but it is not 100% guaranteed.

But metadata are being analyzed at the moment by dnf repoquery command. The command is quite slow and it's being invoked way too often.

I think that the main query should be done only once per one repository, not for each source.RPM.

The problem is that we would need to get srpms for all deleted rpms and I am not sure if there is a way to do that. Maybe there is but there is then one more problem that one would need to only delete srpm in the same build directory where the associated rpm was deleted and not in any other (i.e. not srpms of the same NVRs but in different build directories). The logic for deleting srpms is tricky (maybe there is a better way but i revisited this already a few times and never really figured out anything better).

we would need to only delete srpm in the same build directory

Speaking of that, the current code doesn't seem to take care of this carefully. It seems like it blindly compare NVR only, and doesn't expect that the very same NVR could exist anywhere else in the repository?

I did a quick research and it should be fairly easy to find corresponding source RPMs from RPM metadata.

Btw., note that we seem to have broken repositories by prunerepo, check the errors in:

./find-srpm-to-rpm-pairs.py https://copr-be.cloud.fedoraproject.org/results/%40python/python3.10/fedora-rawhide-x86_64 >/dev/null

It seems to be the reason why we have sometimes broken repo in Copr (missing source RPM, even though it shouldn't be missing).

FTR, 27 hours (and counting...) here:

[copr@copr-be ~][PROD]$ ps -p 3970435 -o etime,cmd
    ELAPSED CMD
 1-01:01:16 /usr/bin/python3 -s /usr/bin/prunerepo --verbose --days 7 --nocreaterepo /var/lib/copr/public_html/results/odubaj/autoconf-2.70/fedora-rawhide-x86_64
[copr@copr-be ~][PROD]$ pstree 3970435 -p -c -a -t
prunerepo,3970435 -s /usr/bin/prunerepo --verbose --days 7 --nocreaterepo /var/lib/copr/public_html/results/odubaj/autoconf-2.70/fedora-rawhide-x86_64
  └─dnf,39950 /usr/bin/dnf repoquery --repofrompath=prunerepo_query,/var/lib/copr/public_html/results/odubaj/autoconf-2.70/fedora-rawhide-x86_64 --repo=prunerepo_query --refresh --queryformat=%{location} --quiet --setopt=skip_if_unavailable=False --srpm texlive-exceltex-20200327-24.fc34~bootstrap.noarch

.. seems to be cycling for each binary RPM.

we would need to only delete srpm in the same build directory

Speaking of that, the current code doesn't seem to take care of this carefully. It seems like it blindly compare NVR only, and doesn't expect that the very same NVR could exist anywhere else in the repository?

There is this line here: https://pagure.io/prunerepo/blob/87387647a1b256d7c2d85fb38478064301bbe994/f/prunerepo#_119

Btw., note that we seem to have broken repositories by prunerepo, check the errors in:
./find-srpm-to-rpm-pairs.py https://copr-be.cloud.fedoraproject.org/results/%40python/python3.10/fedora-rawhide-x86_64 >/dev/null

Yes, I can see some srpms missing (sometimes the error messages are due to subpackages but not in every case), e.g. 01810709-libstoragemgmt/ or 01827148-python-markupsafe

Maybe it's some kind of synchronization problem? It would be good to figure out how it is possible that it can happen but I think that using the find-srpm-to-rpm-pairs.py to find the pairs is a good idea.

There is this line here: https://pagure.io/prunerepo/blob/87387647a1b256d7c2d85fb38478064301bbe994/f/prunerepo#_119

Indeed.

Maybe it's some kind of synchronization problem? It would be good to figure out how it is possible that it can happen

Yep, weird.

but I think that using the find-srpm-to-rpm-pairs.py to find the pairs is a good idea.

Good!

Can we use praiskup's code in find-srpm-to-rpm-pairs.py? it's serious for us that prunerepo takes ages for large repositories. It might seem that you don't like something about the code in find-srpm-to-rpm-pairs.py. Is it possible for me to do something to solve this issue?

Can we use praiskup's code in find-srpm-to-rpm-pairs.py? it's serious for us that prunerepo takes ages for large repositories. It might seem that you don't like something about the code in find-srpm-to-rpm-pairs.py. Is it possible for me to do something to solve this issue?

Sorry, i didn't have time to look in depth at the code. I think the code can be used but I am struggling to find time for this.

Commit 3cf711f5 fixes this issue

This issue has been migrated to GitHub: https://github.com/fedora-copr/prunerepo/issues/7
Don't forget to subscribe for notifications about new comments.