Edit

Defining modules (using modulemd)

To have your module build, you need to write a modulemd file which is the definition of your module including the components, dependencies, API, and more. This page describes all the steps a module developer needs to do to define a module.

Examples

To have an idea about the final result, please have a look at some existing modules in the Fedora dist-git:

The final module definition will be similar to the following example. Please note that the example might not represent an actual buildable module - dependencies or package names have changed - but it shows the overall structure with all the fields this guide will be going through.

Prerequisites - tooling

Fedmod is a tool providing basic operation that significantly simplify the process of creating a new module. Please install it before we start:

$ sudo dnf copr enable @modularity/fedmod
$ sudo dnf install fedmod
$ fedmod fetch-metadata

Defining the module

Let’s start with the following template and fill it in as we go.

document: modulemd
version: 1
data:
    summary: ...
    description: ...
    license:
        module:
            - license-name
    dependencies:
        buildrequires:
            module-name: stream-name
            ...
        requires:
            module-name: stream-name
            ...
    references:
        community: http://example.com
        documentation: http://example.com
        tracker: http://example.com
    profiles:
        example:
            rpms:
                - package-name
                - ...
        ...
    api:
        rpms:
            - package-name
            - ...
    components:
        rpms:
            package-name:
                rationale: ...
                ref: ...
                buildorder: ...
            ...

Step 1: Deciding what the module is

The first step is to identify a package (or a set of packages) that form the core of the module. For example, to create an nginx module for the NGINX web server, the nginx package will be the core.

Based on this, fill in the summary, description, license, and references fields. Also add the main package(s) to components. Please note that the components.rpms field expects SRPM package names.

document: modulemd
version: 1
data:
    summary: A high performance web server and reverse proxy server
    description: >-
        Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3
        and IMAP protocols, with a strong focus on high concurrency,
        performance and low memory usage.
    license:
        module:
            - MIT
    references:
        community: http://example.com
        documentation: http://example.com
        tracker: http://example.com
    components:
        rpms:
            nginx:
                rationale: The main package of this module.
                ref: f27

Step 2: Dependencies

Every module needs to specify its runtime and build dependencies. These are modular dependencies (usually at least the platform module), and RPM dependencies (everything else that is not provided by the modular dependencies).

This is where the Fedmod tool becomes handy. To list all dependencies of a given package (nginx in this case), run:

$ fedmod resolve-deps nginx

This will return a very long list of packages. Many of these are included in the platform module. Let’s get all the missing dependencies that are not in platform by running:

$ fedmod resolve-deps -m platform nginx

This looks much better, right? At the time of writing this document, fedmod returned the following four packages:

gperftools-libs
nginx-mimetypes
nginx
nginx-filesystem

Fedmod can also tell if a certain package has been already added to an existing module. To find whether the gperftools-libs is already somewhere, run:

$ fedmod where-is-package gperftools-libs

At the time of writing, the gperftools-libs package is already in one module, the 389-ds module which is an LDAP server. In this case, it doesn’t make sense to use the 389-ds module as a dependency of nginx. There are two ways how to resolve this problem:

  1. Work with a maintainer of the 389-ds module and get the gperftools-libs package separated into another, shared module. Maybe it could be added to the platform module.
  2. Bundle the package in your module. Please note that this might make the nginx module conflict with the 389-ds module. But this is the way we go in this guide.

As mentioned before, the modulemd file expect components to be listed as SRPM packages. Fedmod can give you an SRPM package name for a given RPM. Let’s try it for the nginx-mimetypes package by running:

$ fedmod srpm-of-rpm nginx-mimetypes

By doing this for all the packages, the result was:

gperftools-libs  -> gperftools
nginx-mimetypes  -> mailcap
nginx            -> nginx
nginx-filesystem -> nginx

The nginx package is already present in the components.rpms field. We just need to add the mailcap and gperftools packages in the components.rpms field and the platform module as a runtime dependency.

With this, we have resolved the runtime dependencies. The next step would be resolving all the build dependencies for this module. However, not all build dependencies might have been modularized. As a workaround, we use a bootstrap module as the only build dependency right now.

After doing these changes, our modulemd will look as follows:

document: modulemd
version: 1
data:
    summary: A high performance web server and reverse proxy server
    description: >-
        Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3
        and IMAP protocols, with a strong focus on high concurrency,
        performance and low memory usage.
    license:
        module:
            - MIT
    dependencies:
        buildrequires:
            bootstrap: f27
        requires:
            platform: f27
    references:
        community: http://example.com
        documentation: http://example.com
        tracker: http://example.com
    components:
        rpms:
            nginx:
                rationale: The main package of this module.
                ref: f27
            mailcap:
                rationale: Provides a dependency: nginx-mimetypes.
                ref: f27
            gperftools:
                rationale: Provides a dependency: gperftools-libs.
                ref: f27

What do you support

As modules contain the main packages, along with their dependencies, it is useful to distinguish between these two. The module maintainer will probably offer an API/ABI stability on the main packages, but the dependencies might be considered an implementation detail. To communicate this clearly, add all the main packages to the api field in your modulemd. In this case, it could be the nginx, nginx-filesystem, and the nginx-mimetypes packages.

document: modulemd
version: 1
data:
    summary: A high performance web server and reverse proxy server
    description: >-
        Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3
        and IMAP protocols, with a strong focus on high concurrency,
        performance and low memory usage.
    license:
        module:
            - MIT
    dependencies:
        buildrequires:
            bootstrap: f27
        requires:
            platform: f27
    references:
        community: http://example.com
        documentation: http://example.com
        tracker: http://example.com
    api:
        rpms:
            - nginx
            - nginx-filesystem
            - nginx-mimetypes
    components:
        rpms:
            nginx:
                rationale: The main package of this module.
                ref: f27
            mailcap:
                rationale: Provides a dependency: nginx-mimetypes.
                ref: f27
            gperftools:
                rationale: Provides a dependency: gperftools-libs.
                ref: f27

Pre-defined installation profiles

To help users with installing this module, a set of installation profiles can and at least one should be defined for the module.

Profiles are very nicely described in the modulemd specification. In short, they are lists of packages to be installed on a system.

For example, a database module might define two installation profiles, client and server, to help the user install the right package for each use case.

Some profile names, such as default, are reserved. The default profile is installed by default if user doesn’t specify any other profile. In this case, our module will only contain this default profile.

document: modulemd
version: 1
data:
    summary: A high performance web server and reverse proxy server
    description: >-
        Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3
        and IMAP protocols, with a strong focus on high concurrency,
        performance and low memory usage.
    license:
        module:
            - MIT
    dependencies:
        buildrequires:
            bootstrap: f27
        requires:
            platform: f27
    references:
        community: http://example.com
        documentation: http://example.com
        tracker: http://example.com
    profiles:
        default:
            rpms:
                - nginx
    api:
        rpms:
            - nginx
            - nginx-filesystem
            - nginx-mimetypes
    components:
        rpms:
            nginx:
                rationale: The main package of this module.
                ref: f27
            mailcap:
                rationale: Provides a dependency: nginx-mimetypes.
                ref: f27
            gperftools:
                rationale: Provides a dependency: gperftools-libs.
                ref: f27

Validating the modulemd syntax

Once the modulemd file is finished, it is a good idea to check if there any errors in the yaml syntax. The check_modulemd program checks modulemd files for errors. You need to install some packages to use this:

  • python2-aexpect - dependency for python-avocado
  • python2-avocado - avocado testing framework
  • python2-modulemd - Module metadata manipulation library
  • python-enchant - spell checker library (needed only for check_modulemd.py)
  • hunspell-en-US - English dictionary (needed only for check_modulemd.py)

Then run
./run-checkmmd.sh /home/karsten/Modularity/modules/vim/vim.yaml

and check the output for errors:

Running: avocado run ./check_modulemd.py --mux-inject 'run:modulemd:/home/karsten/Modularity/modules/vim/vim.yaml'
JOB ID     : 51581372fec0086a50d9be947ea06873b33dd0e5
JOB LOG    : /home/karsten/avocado/job-results/job-2017-01-19T11.28-5158137/job.log
TESTS      : 11
 (01/11) ./check_modulemd.py:ModulemdTest.test_debugdump: PASS (0.16 s)
 (02/11) ./check_modulemd.py:ModulemdTest.test_api: PASS (0.15 s)
 (03/11) ./check_modulemd.py:ModulemdTest.test_components: PASS (0.16 s)
 (04/11) ./check_modulemd.py:ModulemdTest.test_dependencies: WARN (0.02 s)
 (05/11) ./check_modulemd.py:ModulemdTest.test_description: PASS (0.16 s)
 (06/11) ./check_modulemd.py:ModulemdTest.test_description_spelling: PASS (0.16 s)
 (07/11) ./check_modulemd.py:ModulemdTest.test_summary: PASS (0.16 s)
 (08/11) ./check_modulemd.py:ModulemdTest.test_summary_spelling: WARN (0.02 s)
 (09/11) ./check_modulemd.py:ModulemdTest.test_rationales: ERROR (0.04 s)
 (10/11) ./check_modulemd.py:ModulemdTest.test_rationales_spelling: PASS (0.16 s)
 (11/11) ./check_modulemd.py:ModulemdTest.test_component_availability: WARN (0.02 s)
RESULTS    : PASS 7 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 3 | INTERRUPT 0
TESTS TIME : 1.20 s

So this isn’t quite right yet, lets have a look at the logfile mentioned in the output.

grep -i error /home/karsten/avocado/job-results/job-2017-01-19T11.28-5158137
....
TestError: Rationale for component RPM generic-release should end with a period: build dependency

It seems that rationales need to end with a period. Change all those lines so that they look like this:

vim:
    rationale: Provides API for this module.
    ref: f25
    buildorder: 10
generic-release:
    rationale: build dependency.
    ref: f25
gpm:
    rationale: build dependency.
    ref: f25
perl:
    rationale: build dependency.
    ref: f25
perl-Carp:
    rationale: build dependency.
    ref: f25
perl-Exporter:
    rationale: build dependency.
    ref: f25

Another run of check_modulemd.py completes without errors.