From aa7b0c7c6a3616cde997ad15f6313be574178537 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:15:01 +0000 Subject: [PATCH 1/11] ref: This is part of releng repo refactoring -> need_rebuild.py Signed-off-by: Anton Medvedev --- diff --git a/modules/release_guide/pages/sop_mass_rebuild.adoc b/modules/release_guide/pages/sop_mass_rebuild.adoc index 74eb7ba..6109f27 100644 --- a/modules/release_guide/pages/sop_mass_rebuild.adoc +++ b/modules/release_guide/pages/sop_mass_rebuild.adoc @@ -134,7 +134,7 @@ find-failures.py:: * mass-tag.py * {blank} + -need-rebuild.py:: +need_rebuild.py:: ** buildtag ** target ** updates @@ -194,7 +194,7 @@ ____ The community has a very high interest in the status of rebuilds and many maintainers will want to know if their build failed right away. The -`find-failures.py` and `need-rebuild.py` scripts are designed to update +`find_failures.py` and `need_rebuild.py` scripts are designed to update publicly available URLs for stakeholders to monitor. ===== @@ -229,7 +229,7 @@ $ while true; do ./find_failures.py > f{rawhide}-failures.html && cp f{rawhide}- . {blank} + Set up the Site for Packages that Need Rebuilt:: - The `need-rebuild.py` script discovers packages that have not yet been + The `need_rebuild.py` script discovers packages that have not yet been rebuilt and generates an html file listing them sorted by package owner. This gives external stakeholders a rough idea of how much work is remaining in the mass rebuild. @@ -237,7 +237,8 @@ Set up the Site for Packages that Need Rebuilt:: [source,subs="attributes+"] .... -$ while true; do ./need-rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhide}-need-rebuild.html /mnt/koji/mass-rebuild/f{rawhide}-need-rebuild.html; sleep 600; done +$ cd releng_repo/scripts_new/release-process/mass-rebuilds/ +$ while true; do ./need_rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhide}-need-rebuild.html /mnt/koji/mass-rebuild/f{rawhide}-need-rebuild.html; sleep 600; done .... ===== From 2437e750fe2dc448b3448962044accbe58f9cc50 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:05 +0000 Subject: [PATCH 2/11] This is part of releng repo refactoring -> mass_tag.py --- diff --git a/modules/release_guide/pages/sop_mass_rebuild.adoc b/modules/release_guide/pages/sop_mass_rebuild.adoc index 6109f27..757a896 100644 --- a/modules/release_guide/pages/sop_mass_rebuild.adoc +++ b/modules/release_guide/pages/sop_mass_rebuild.adoc @@ -131,7 +131,7 @@ find-failures.py:: ** buildtag ** desttag ** epoch -* mass-tag.py +* mass_tag.py * {blank} + need_rebuild.py:: @@ -245,7 +245,7 @@ $ while true; do ./need_rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhid === Post Mass Rebuild Tasks Once the mass rebuild script completes, and all the pending builds have -finished, the builds will need to be tagged. The `mass-tag.py` script +finished, the builds will need to be tagged. The `mass_tag.py` script will accomplish this task. The script will: * Discover completed builds @@ -257,15 +257,15 @@ email) ===== . Clone or checkout the latest copy of the https://pagure.io/releng[releng git repository] -. Run the `mass-tag.py` script (requires koji kerberos authentication) +. Run the `mass_tag.py` script (requires koji kerberos authentication) . Remove the koji target for the rebuild: $ koji remove-target f{rawhide}-rebuild + ____ [source,subs="attributes+"] .... -$ cd path/to/releng_repo/scripts -$ ./mass-tag.py --source f{rawhide}-rebuild --target f{rawhide} +$ cd path/to/releng_repo/scripts_new/release-process/mass-rebuilds/ +$ ./mass_tag.py --source f{rawhide}-rebuild --target f{rawhide} .... ____ . Send the final notification to the From 27385771a7507ed4b133e697724a3e4bc7b1d13e Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:05 +0000 Subject: [PATCH 3/11] This is part of releng repo refactoring -> get_retired_packages.sh --- diff --git a/modules/release_guide/pages/sop_final_freeze.adoc b/modules/release_guide/pages/sop_final_freeze.adoc index 7bd8b8c..d57bfb8 100644 --- a/modules/release_guide/pages/sop_final_freeze.adoc +++ b/modules/release_guide/pages/sop_final_freeze.adoc @@ -5,12 +5,12 @@ include::_partials/attributes.adoc[] == Make sure all retired packages are blocked in koji Before the freeze we have to make sure all retired packages have been blocked in koji. -To do this, [run the script](https://pagure.io/releng/blob/main/f/scripts/get_retired_packages.sh) that collects all the retired packages +To do this, [run the script](https://pagure.io/releng/blob/main/f/scripts_new/packages/distgit/get_retired_packages.sh) that collects all the retired packages and outputs them into json files in the [lookaside cache](https://src.fedoraproject.org/lookaside). .... $ ssh pkgs01.fedoraproject.org -$ cd releng/scripts && bash get_retired_packages.sh +$ cd releng/scripts_new/packages/distgit/get_retired_packages.sh && bash get_retired_packages.sh .... This script will run about 20 minutes. Once finished, run toddler 'koji_block_retired' to block all retired packages in koji. From a14ba32325c849adb40cdfdd0f86024dc3e4b254 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:49 +0000 Subject: [PATCH 4/11] This is part of releng repo refactoring -> find_failures.py Signed-off-by: Anton Medvedev # Conflicts: # modules/release_guide/pages/sop_mass_rebuild.adoc --- diff --git a/modules/release_guide/pages/sop_mass_rebuild.adoc b/modules/release_guide/pages/sop_mass_rebuild.adoc index 757a896..f1c3867 100644 --- a/modules/release_guide/pages/sop_mass_rebuild.adoc +++ b/modules/release_guide/pages/sop_mass_rebuild.adoc @@ -127,7 +127,7 @@ mass-rebuild.py:: ** target * {blank} + -find-failures.py:: +find_failures.py:: ** buildtag ** desttag ** epoch @@ -213,7 +213,13 @@ $ tmux .... ____ . Clone or checkout the latest copy of the -https://pagure.io/releng[releng git repository] +https://pagure.io/releng[releng git repository] and move to location that contains `find_failures.py` script. ++ +____ +.... +$ cd releng/scripts_new/release-process/mass-rebuilds/ +.... +____ . {blank} + Set Up the Rebuild Failures Notification Web Site:: From 226a99422dfb226d7a93035408c3491c94173c55 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:51 +0000 Subject: [PATCH 5/11] This is part of releng repo refactoring -> fedora_torrent_ini.py --- diff --git a/modules/release_guide/pages/sop_torrent_releases.adoc b/modules/release_guide/pages/sop_torrent_releases.adoc index 63d32f7..ea1bfb8 100644 --- a/modules/release_guide/pages/sop_torrent_releases.adoc +++ b/modules/release_guide/pages/sop_torrent_releases.adoc @@ -86,18 +86,18 @@ generally 14:00 UTC on Tuesday. ==== ____ -. Grab `fedora-torrent-init.py` from releng scripts and change it to +. Grab `fedora_torrent_ini.py` from releng scripts under `scripts_new/release_process_torrents/` and change it to executable + .... cd ~ -wget https://pagure.io/releng/raw/main/f/scripts/fedora-torrent-ini.py -chmod 755 ~/fedora-torrent-ini.py +wget https://pagure.io/releng/raw/main/f/scripts_new/release-process/torrents/fedora_torrent_ini.py +chmod 755 ~/fedora_torrent_ini.py .... . Run the following command from `/srv/torrent/new/fedora/` + .... -~/fedora-torrent-ini.py _ > _.ini +~/fedora_torrent_ini.py _ > _.ini .... . Copy all the torrents to `/srv/web/torrents/` + From 8698119e6859bc02eda14e50a490f26929af7c36 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:51 +0000 Subject: [PATCH 6/11] This is part of releng repo refactoring -> distgit_branch_unused.py --- diff --git a/modules/howtos/pages/remove_branch_distgit.adoc b/modules/howtos/pages/remove_branch_distgit.adoc index 2afe767..85e832c 100644 --- a/modules/howtos/pages/remove_branch_distgit.adoc +++ b/modules/howtos/pages/remove_branch_distgit.adoc @@ -40,7 +40,7 @@ cd clone && git checkout . Run the script: + ---- -python ../releng/scripts/distgit-branch-unused.py +python ../releng/scripts_new/packages/distgit/distgit_branch_unused.py ---- + (If needed, see the ``--help`` of the script for more information) diff --git a/modules/releng_misc_guide/pages/sop_remote_dist_git_branches.adoc b/modules/releng_misc_guide/pages/sop_remote_dist_git_branches.adoc index 2afe767..85e832c 100644 --- a/modules/releng_misc_guide/pages/sop_remote_dist_git_branches.adoc +++ b/modules/releng_misc_guide/pages/sop_remote_dist_git_branches.adoc @@ -40,7 +40,7 @@ cd clone && git checkout . Run the script: + ---- -python ../releng/scripts/distgit-branch-unused.py +python ../releng/scripts_new/packages/distgit/distgit_branch_unused.py ---- + (If needed, see the ``--help`` of the script for more information) From b5927e0650ddca1ec20856c022afec655900cac8 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:51 +0000 Subject: [PATCH 7/11] This is part of releng repo refactoring -> clean_amis.py --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 4cfcd26..6739668 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -67,7 +67,6 @@ *** xref:releng_misc_guide:sop_request_an_automation_user.adoc[Request an Automation User] *** xref:releng_misc_guide:sop_process_dist_git_requests.adoc[Process `fedora-scm-requests` Tickets] *** xref:releng_misc_guide:sop_signing_builds.adoc[Sign the Packages] -*** xref:releng_misc_guide:sop_clean_amis.adoc[Clean AMIs] *** xref:releng_misc_guide:sop_pushing_updates.adoc[Push Updates] *** xref:releng_misc_guide:sop_sigul_client_setup.adoc[Sigul Client Setup] *** xref:releng_misc_guide:sop_troubleshooting.adoc[General Troubleshooting Guide] diff --git a/modules/releng_misc_guide/pages/index.adoc b/modules/releng_misc_guide/pages/index.adoc index 5ec08a0..06ba739 100644 --- a/modules/releng_misc_guide/pages/index.adoc +++ b/modules/releng_misc_guide/pages/index.adoc @@ -22,7 +22,6 @@ Here are some of the miscellaneous tasks performed by the Fedora Release Enginee * link:sop_requesting_task_automation_users[Requesting Automation Users] * link:sop_process_dist_git_requests[Process fedora-scm-requests tickets] * link:sop_signing_builds[Sign the packages] -* link:sop_clean_amis[Clean AMIs Process] * link:sop_pushing_updates[Pushing Updates] * link:sop_package_blocking[Package Blocking] * link:sop_sigul_client_setup[Sigul Client Setup] diff --git a/modules/releng_misc_guide/pages/sop_clean_amis.adoc b/modules/releng_misc_guide/pages/sop_clean_amis.adoc deleted file mode 100644 index af17010..0000000 --- a/modules/releng_misc_guide/pages/sop_clean_amis.adoc +++ /dev/null @@ -1,59 +0,0 @@ -= Clean AMIs Process - -== Description - -The Fedora AMIs are uploaded on a daily basis to Amazon Web Services. -Over time the number of AMIs piles up and have to be removed manually. -Manual removal comes with it's own set of issues where missing to delete -the AMIs is a viable issue. - -The goal of the script is to automate the process and continue regular -removal of the AMIs. The report of the script is pushed to a -https://pagure.io/ami-purge-report[Pagure repo] - -== Action - -There is a script in the https://pagure.io/releng[Fedora RelEng repo] -named `clean-amis.py` under the `scripts` directory. - -The script runs as a cron job within the Fedora Infrastructure to delete -the old AMIs. The permission of the selected AMIs are changed to -private. This is to make sure that if someone from the community raises -an issue we have the option to get the AMI back to public. After 10 -days, if no complaints are raised the AMIs are deleted permanently. - -The complete process can be divided in couple of parts: - -* Fetching the data from datagrepper. Based on the [.title-ref]#--days# -param, the script starts fetching the fedmsg messages from datagrepper -for the specified timeframe i.e. for lasts [.title-ref]#n# days, where -[.title-ref]#n# is the value of [.title-ref]#--days# param. The queried -fedmsg topic [.title-ref]#fedimg.image.upload#. -* Selection of the AMIs: After the AMIs are parsed from datagrepper. The -AMIs are filtered to remove Beta, Two-week Atomic Host and GA released -AMIs. Composes with [.title-ref]#compose_type# set to -[.title-ref]#nightly# are picked up for deletion. Composes which contain -date in the [.title-ref]#compose label# are also picked up for deletion. -GA composes also have the compose_type set to production. So to -distinguish then we filter them if the compose_label have date in them. -The GA composes dont have date whereas they have the version in format -of X.Y -* Updated permissions of AMIs The permissions of the selected AMIs are -changed to private. -* Deletion of AMIs After 10 days, the private AMIs are deleted. - -In order to change the permissions of the AMIs use the command given -below, add [.title-ref]#--dry-run# argument test the command works. -Adding [.title-ref]#--dry-run# argument will print the AMIs to console. - -.... -AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --change-perms --days 7 --permswaitperiod 5 -.... - -In order to delete the AMIs whose launch permissions have been removed, -add [.title-ref]#--dry-run# argument test the command works. Adding -[.title-ref]#--dry-run# argument will print the AMIs to console. - -.... -AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --delete --days 17 --deletewaitperiod 10 -.... From a1d6be3a126732af3acf45260665279eb072e771 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:51 +0000 Subject: [PATCH 8/11] This is part of releng repo refactoring -> block_retired.py --- diff --git a/modules/releng_misc_guide/pages/sop_package_blocking.adoc b/modules/releng_misc_guide/pages/sop_package_blocking.adoc index 0af424f..6f52b52 100644 --- a/modules/releng_misc_guide/pages/sop_package_blocking.adoc +++ b/modules/releng_misc_guide/pages/sop_package_blocking.adoc @@ -15,3 +15,6 @@ the oldest tag will make it unavailable also in upstream tags. The blocking of retired packages is done by the https://pagure.io/releng/blob/master/f/scripts/block_retired.py[block_retired.py] script as part of the daily Rawhide and Branched composes. + +This script is outdated and now removed from releng repository, instead of it this proccess is done automatically by toddler plugin `koji_block_retired.py` which is reacting on message when package gets retired. + From fdc2dd7f209d195ca1c7275eb3776a7eee374227 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:18:51 +0000 Subject: [PATCH 9/11] This is part of releng repo refactoring -> orphaned-packages-proccess/ --- diff --git a/modules/release_guide/pages/sop_mass_branching.adoc b/modules/release_guide/pages/sop_mass_branching.adoc index 3eacba4..9d02c7f 100644 --- a/modules/release_guide/pages/sop_mass_branching.adoc +++ b/modules/release_guide/pages/sop_mass_branching.adoc @@ -146,7 +146,7 @@ correct this in the initial branch commit for Fedora 44, then remove this note. == Releng Changes -As part of the mass branching process, changes were made to the `find_unblocked_orphans.py` script to align with the latest Fedora release versions. +As part of the mass branching process, changes were made to the `find_unblocked_orphans.py` script to align with the latest Fedora release versions. The script can be found under `scripts_new/packages/orphaned/orphaned-packages-process/` directory in `releng` repository. === Key Changes - Updated `koji_tag` to reference the new Fedora version: diff --git a/modules/release_guide/pages/sop_retire_orphaned_packages.adoc b/modules/release_guide/pages/sop_retire_orphaned_packages.adoc index 01891c4..f337ff3 100644 --- a/modules/release_guide/pages/sop_retire_orphaned_packages.adoc +++ b/modules/release_guide/pages/sop_retire_orphaned_packages.adoc @@ -28,7 +28,7 @@ process. It should be run on a machine that has `koji` and complete. `find_unblocked_orphans.py` is available in the -https://pagure.io/releng[Release Engineering git repository] +https://pagure.io/releng[Release Engineering git repository] under `scripts_new/packages/orphaned/orphaned-packages-process/` directory. === Announcing Packages to be retired From 60a30562e2c05ee38e02414dbefa732541d18cd8 Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 12:42:41 +0000 Subject: [PATCH 10/11] This is part of releng repo refactoring -> critpath.py --- diff --git a/modules/releng_misc_guide/pages/sop_update_critpath.adoc b/modules/releng_misc_guide/pages/sop_update_critpath.adoc index 6ffc39a..1a4f45b 100644 --- a/modules/releng_misc_guide/pages/sop_update_critpath.adoc +++ b/modules/releng_misc_guide/pages/sop_update_critpath.adoc @@ -20,7 +20,7 @@ steps to update it manually are documented here in case of need. == Action . The release engineering script for updating critpath lives in the -https://pagure.io/releng[releng git repository]. Check this out. +https://pagure.io/releng[releng git repository] under `scripts_new/quality-assurance/critpath/` path. Check this out. . Run the script. If run with `all`, it will generate the info for all current releases, and name the output files as Bodhi expects. Otherwise you can specify the release to generate info for, and the From a4ba4b80ac71dbdaee519cf31675ae480f64046f Mon Sep 17 00:00:00 2001 From: Anton Medvedev Date: Oct 02 2025 13:27:35 +0000 Subject: [PATCH 11/11] This is part of releng repo refactoring -> failed_composes_cleanup.py Signed-off-by: Anton Medvedev --- diff --git a/modules/release_guide/pages/sop_post-release_cleanup.adoc b/modules/release_guide/pages/sop_post-release_cleanup.adoc index b432e1e..dddeaaa 100644 --- a/modules/release_guide/pages/sop_post-release_cleanup.adoc +++ b/modules/release_guide/pages/sop_post-release_cleanup.adoc @@ -71,7 +71,7 @@ Address and close any open issues related to failed composes to maintain the hea [discrete] === Steps -. Use the script available in the Fedora RelEng repository: https://pagure.io/releng/blob/main/f/scripts/misc/failed_composes_cleanup.py +. Use the script available in the Fedora RelEng repository: https://pagure.io/releng/blob/main/f/scripts_new/release-process/post-release/failed_composes_cleanup.py + [source,subs="attributes+"] ....