Mass Rebuild

Description

Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this.

Assumptions

This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates.

This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change.

Considerations

• The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.

• Check in on scripts frequently to avoid a long stalled command from adding significant delays in completing the rebuild.

• Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis.

Actions

Preparatory Steps

The following steps may be completed in the weeks leading up to the scheduled mass rebuild.

1. Create the Mass Rebuild Pagure Issue

   Create an issue on the Release Engineering issues page that points at the schedule for the current release.

   See the Fedora 27 mass rebuild issue example.

2. Set up the Mass Rebuild Wiki Page

   The mass rebuild wiki page should answer the following questions for maintainers:

   • Why the mass rebuild is happening

   • How to opt out of the mass rebuild

   Note

   See the Fedora 26 Wiki example.

3. Send out the Mass Rebuild Notice

   Send out the same information posted on the wiki to the devel-announce@lists.fedoraproject.org mailing list.

   Note

   See the Fedora 26 e-mail example.

4. Create a Tag to Contain the Mass Rebuild

   Mass rebuilds require their own tag to contain all related builds. The example assumes we are doing a rebuild for Fedora 26.

   $ koji add-tag f26-rebuild --parent f26
   

5. Request Package Auto-Signing for New Mass-Rebuild Tag

   File a ticket with Fedora Infrastructure requesting the new mass-rebuild tag be enabled for package auto-signing.

6. Create the Koji Target for the Mass Rebuild

   Using the same f26-rebuild tag created in the previous example:

   $ koji add-target f26-rebuild f26-build
   

   Note

   koji add-target target-name buildroot-tag destination-tag describes the syntax format above. If the destination-tag is not specified then it will be the same as the target-name.

7. Update Scripts

   The mass rebuild depends on four main scripts from the releng git repository. Each one requires some changes in variables for each new mass rebuild cycle.

   • mass-rebuild.py

     • buildtag

     • targets

     • epoch

     • comment

     • target

   • find-failures.py

     • buildtag

     • desttag

     • epoch

   • mass-tag.py

   • need-rebuild.py

     • buildtag

     • target

     • updates

     • epoch

Change the following items:

• the build tag, holding tag, and target tag should be updated to reflect the Fedora release you’re building for

• the epoch should be updated to the point at which all features that the mass rebuild is for have landed in the build system (and a newRepo task completed with those features)

• the comment which is inserted into spec changelogs

Starting the Mass Rebuild

The mass-rebuild.py script takes care of:

• Discovering available packages in koji

• Trimming out packages which have already been rebuilt

• Checking out packages from git

• Bumping the spec file

• Committing the change

• git tagging the change

• Submitting the build request to Koji

1. Connect to the mass-rebuild Machine

   $ ssh branched-composer.phx2.fedoraproject.org
   

2. Start a terminal multiplexer

   $ tmux
   

3. Clone or checkout the latest copy of the releng git repository.

4. Run the mass-rebuild.py script from releng/scripts

   $ cd path/to/releng_repo/scripts
   $ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
   

Monitoring Mass Rebuilds

The community has a very high interest in the status of rebuilds and many maintainers will want to know if their build failed right away. The find-failures.py and need-rebuild.py scripts are designed to update publicly available URLs for stakeholders to monitor.

1. Connect to a Compose Machine

   $ ssh compose-x86-02.phx2.fedoraproject.org
   

2. Start a terminal multiplexer

   $ tmux
   

3. Clone or checkout the latest copy of the releng git repository

4. Set Up the Rebuild Failures Notification Web Site

   The find_failures.py script discovers attempted builds that have failed. It lists those failed builds and sorts them by package owner.

   $ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
   

5. Start a second pane in the terminal emulator

6. Set up the Site for Packages that Need Rebuilt

   The need-rebuild.py script discovers packages that have not yet been rebuilt and generates an html file listing them sorted by package owner. This gives external stakeholders a rough idea of how much work is remaining in the mass rebuild.

   $ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
   

Post Mass Rebuild Tasks

Once the mass rebuild script completes, and all the pending builds have finished, the builds will need to be tagged. The mass-tag.py script will accomplish this task. The script will:

• Discover completed builds

• Trim out builds that are older than the latest build for a given package

• Tag remaining builds into their final destination (without generating email)

1. Clone or checkout the latest copy of the releng git repository

2. Run the mass-tag.py script (requires koji kerberos authentication)

   $ cd path/to/releng_repo/scripts
   $ ./mass-tag.py --source f36-rebuild --target f36
   

3. Send the final notification to the devel-announce@lists.fedoraproject.org list

   The contents should look something like this example email.