How to manage chroots

This article explains how to enable new chroots when a new Fedora is released and also how to disable them once the particular Fedora version is not supported anymore.

Commands overview

Chroots can be easily managed with these few commands.

copr-frontend create-chroot <name>
copr-frontend alter-chroot --action activate <name>
copr-frontend alter-chroot --action deactivate <name>
copr-frontend branch-fedora <new-branched-version>
copr-frontend rawhide-to-release <rawhide-chroot> <newly-created-chroot>
copr-frontend chroots-template [--template PATH]
copr-frontend eol-lifeless-rolling-chroots

However, enablement process upon Fedora branching and also chroot deactivation when Fedora reaches it’s EOL phase, are not that simple. That’s why the rest of this article explains the certain use-cases rather than atomic actions.

Branching process

Projects can be configured to follow Fedora branching. That means, that once a chroot for new Fedora release is enabled, it should be automatically turned-on for such projects. Moreover, builds from Rawhide should be forked into this new chroot.

So before Fedora branching happens (for exmaple to version 31), you want to run the following command on Copr Frontend, under the copr-fe user:

$ copr-frontend branch-fedora 31

This command takes a long time — even tens of minutes. And then processing of all the actions on the backend side will take about a day or so! Be prepared to this.

This command creates fedora-31-* chroots from the corresponding fedora-rawhide-* chroots, and it also copies (duplicates/forks) the latest successful rawhide package builds into the new chroots.

This could be done also manually for each supported architecture like:

copr-frontend create-chroot fedora-31-x86_64 --deactivated
copr-frontend rawhide-to-release fedora-rawhide-x86_64 fedora-31-x86_64

From the manual steps you can see that the new chroots are deactivated at the beginning.

It’s important to do the branch-fedora action before the branching in Fedora happens because the builds with bumped %dist did not happen yet – and the copied packages in the new (Fedora 31) chroots will have the old dist tag (.fc31, not .fc32).

Copr uses Mock for building packages, so you should check if the mock configs are already available and in which version of the mock-core-configs package they were added. This package needs to be on the Copr builders.

Now activate the new chroots (ASAP after all the builds were copied, you can check the FE statistics page if the Actions peak is processed):

copr-frontend alter-chroot --action activate fedora-31-x86_64 fedora-31-i386 \
    fedora-31-ppc64le fedora-31-aarch64 fedora-31-armhfp fedora-31-s390x

The sooner this is done the better. Since the branch-fedora action have been executed, there have been a time window when users kept building new builds into the “being forked” Rawhide chroots, and those new builds will not be copied into the branched chroots. The longer the activation takes, the more inconsistent the branched chroot is.

Note that you don’t have to care about the new (F31) official Fedora compose creation time at all, the mock-core-configs package (and deps, like distribution-gpg-keys) is prepared so both the Rawhide and the new branched chroot (31) will work during the tranisition period (both before and after the branched chroot has it’s first own mirroed compose, see the rel-eng thread when this was tested).

When everything is done, send an information email to a mailing list.

EOL deactivation process

Warning

Before deactivating chroots, please make sure other services such as Fedora Review Service doesn’t depend on them.

When some Fedora version reaches the end of its release cycle and is marked as EOL, you can safely disable its chroots. Though we want to keep the chroots enabled for a short period of time (few weeks) even after EOL, so our users can comfortably deal with it. It can be done like this

fv=34
copr-frontend alter-chroot --action eol fedora-$fv-x86_64 fedora-$fv-i386 \
    fedora-$fv-ppc64le fedora-$fv-aarch64 fedora-$fv-armhfp fedora-$fv-s390x

After running such command, no data are going to be removed. All repositories for the chroot are preserved. It is just disabled and users can’t build new packages in it anymore.

When it is done, send an information email to a mailing list. See the the disable chroots template.

Rawhide (and other rolling) chroots EOL

Run copr-frontend eol-lifeless-rolling-chroots to mark existing rolling Copr chroots for the future removal/deactivation — if they appear lifeless. You might want to run this daily in the copr-frontend-optional cron-job file. The logic in this command checks that no build happened in particular rolling chroot for a long time, so likely no work is being done there, and the old built packages are _likely_ non-installable anyway (as the rolling distro moves forward with dependencies, but no dependency resolution is being done with RPMs). If such a chroot is marked EOL, this command applies the same notification policy/process as with the EOL deactivation process so users can keep the chroot alive (either by prolonging the chroot, or triggering a new build).

Managing chroot comments

Some of the available Mock chroots deserve a special care in documentation, e.g. that epel-8-* chroots are nowadays built against Red Hat Enterprise Linux 8, not CentOS 8 (which is EOL). There’s an administrator command for this:

$ copr-frontend comment-chroot --chroot epel-8-x86_64 --comment '<strong>Built against RHEL 8!</strong>'

Note that HTML text is supported!

This was though a single-chroot command. There’s a better option for those Copr instances that contain dozens of chroots:

$ copr-frontend chroots-template [--template /etc/copr/chroots.conf]

This file reads the template file in the following format (a Python file defining the config dictionary):

config = {}
config["emulated"] = "This is an emulated chroot"
config["rules"] = [{
    "match": "fedora-rawhide-i386",
    "comment": "This is soon to be removed",
}, {
    "match": ["fedora-32", "fedora-33"],
    "comment": "<strong>Currently EOL</strong>, on your own risk",
},
{
    "match": ["aarch64", "ppc64le"],
    "match_type", "arch",
    "comment_append": "{{ emulated }}",
}]

When (manually) executed, the command recursively iterates across all the active Mock chroots, and applies the specified rules (only comment or comment_append currently) when the chroot matches the rules (match and match_type statements).

Mailing lists

After adding or disabling chroots on the production instance, an information email about the action should be sent to copr-devel@lists.fedorahosted.org . When doing both actions at the same time, describing it in one email is sufficient.