[archived] Testing modular content in Fedora

Interactive Testing

If the automated tests are failing, or if a new module is being built for the first time, it is useful to be able to check modules interactively before they make their way into an official compose. (For details on the regular testing of modules and their contents, see the Automated Testing section below)

The interactive testing guide below assumes that fedpkg module-build is succeeding - if that isn’t the case, then the module and package build logs should provide details as to what is going wrong.

Resolving dependency issues

The most common problems that are encountered despite a successful build and passing integration tests are dependency resolution issues when attempting to install a module. The resolving-dependency-issues section provides more guidance on dealing with such situations.

Boltron testing images

The original Fedora 26 Boltron prototype lives on as a set of test container images, which build atop the Fedora modular composes, but include some additional utililties to help out with testing modules prior to their release in Fedora.

Two variants of the image are provided, a boltron-27 image with module definitions from the Fedora 27 Modular Server release, and a boltron-bikeshed with the latest module definitions from the module build service that have passed their self-tests in Taskotron:

$ docker run --rm -it jamesantill/boltron-27 bash
$ docker run --rm -it jamesantill/boltron-bikeshed bash

See the README in the image source repository for more information on the available utilities.

Enabling access to unreleased modules

Module releases may take a day or more to become available in the default image, so the Boltron image provides a helper script to download built modules directly from the Module Build Service and incorporate them into the currently running image. For example:

# /LOCAL.sh postgresql:9.6:20171018083530

This will download the binary artifacts for that particular build of the PostreSQL 9.6 stream, set up a local repository for them (including the module metadata), enable that repository, and then install the module with its default profile.

A non-default install profile can be requested by appending the profile name to the build identifer, separated by a /:

# /LOCAL.sh postgresql:9.6:20171018083530/client

Finding module build identifiers

The list of currently build modules and their identifiers is available at http://modularity.fedorainfracloud.org/modularity/mbs/.

Automated Testing

As described in the Continuous Integration page on the Fedora wiki, Fedora is currently working to improve the level of automated test coverage for the entire distribution.

To provide feedback to package and module maintainers in a timely fashion, tests should be generally be executed at the first level where all the pieces necessary to run them are available.

Per-package tests

Independently of the modular build process, individual packages may define their own automated tests, both as part of the %check section in their RPM spec file, as well as in the tests/ subdirectory of their respective dist-git RPM repositories

Per-module tests in dist-git

Modules may also define their own individual automated tests, in the tests/ subdirectory of their respective dist-git module repositories

The Meta Test Family project is used to define test cases for whole modules and their resulting RPMs and container images.

When a module is built in Fedora’s infrastructure and MTF tests are available in its repository, they will be run afterwards by Taskotron.

Modular compose tests

Finally, a base set of automated tests check that the core set of modules defined in Fedora’s system profile can all be installed and updated.

These tests are written as Ansible playbooks, and can be found in the compose-tests repository. In addition to checking that the default set of available modules works, these tests also ensure that the actual system profile defined in the Pungi configuration matches the expected one defined in the F27 content tracking repository.

These tests are executed in the CentOS CI infrastructure:

Issues specific to the Fedora Modular release are tracked using the fedora-modular-release component in Bugzilla.

Modular DNF testing

In addition to automated testing of the overall modular compose, there is also lower level integration testing of DNF’s module management support: https://github.com/rpm-software-management/ci-dnf-stack/tree/modularity

If a module or modular compose test failure appears to be related to an error in DNF, then it should be reported against the dnf component of the Fedora project in Bugzilla and then linked from the Modular DNF tracker BZ.