Mass Rebuild of Modules

Description

Periodically we do mass rebuilds of modules in rawhide during the development cycle. This SOP will outline the steps necessary to do this.

Assumptions

This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed updates.

Considerations

  • The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.

  • Check in on scripts frequently to avoid a long stalled command from adding significant delays in completing the rebuild.

Actions

Preparatory Steps

The following steps should be completed after the mass rebuild of packages is done.

  1. Update Scripts

The mass rebuild depends on two main scripts from the releng git repository. Each one requires some changes in variables for each new mass rebuild cycle.

  • mass-rebuild-modules.py

    • rebuildid

  • massrebuildsinfo.py

    • module_mass_rebuild_epoch

    • module_mass_rebuild_platform

Change the following items:

  • the rebuildid to match the release for which you are mass rebuilding modules as per in massrebuildsinfo.py

  • module_mass_rebuild_epoch mostly will be the epoch of mass rebuild of packages

  • module_mass_rebuild_platform should be the rawhide module platform

Starting the Mass Rebuild of Modules

The mass-rebuild-modules.py script takes care of:

  • Discovering available available modules from PDC

  • Find the module info from mbs and check if a module build is submitted after the epoch date

  • Checking out modules from dist-git

  • Switching to appropriate stream

  • Find modulemd file

  • Use libmodulemd to determine if this module stream applies to this platform version

  • If needs rebuilding, committing the change

  • Push the commit

  • Submitting the build request through mbs

  1. Connect to the mass-rebuild Machine

    $ ssh compose-branched01.iad2.fedoraproject.org

  2. Start a terminal multiplexer

    $ tmux

  3. Clone or checkout the latest copy of the releng git repository.

  4. Run the mass-rebuild-modules.py script from releng/scripts

    $ cd path/to/releng_repo/scripts
$ ./mass-rebuild-modules.py <path_to_token_file> build --wait 2>&1 | tee ~/massbuildmodules.out

Note

The token file should be located in infra’s private ansible repo, or ask infra to get it to you using this process.

Note

The build option is really important to pay attention, since the mass branching of modules will also use the same script, just changing the option to branch and module_mass_branching_platform in massrebuildsinfo.py

Post Mass Rebuild Tasks

Once the module mass rebuild is done, send an email to the devel-announce@ list

  1. Send the final notification to the devel-announce@lists.fedoraproject.org list