Raw config composes

ODCS can also be used to generate composes defined by the Pungi configuration file stored in an external git repository. There is special compose source type called raw_config which is used for these composes.

This document describes how to configure ODCS server to generate raw_config composes.

High level overview

Raw config composes are submitted by the ODCS client in the raw_config_name#commit_hash format. For example fedora-30-nightly#master.

The ODCS server uses raw_config_name (in our case fedora-30-nightly) to find matching record in the raw_config_urls option stored in its main configuration file.

The raw_config_urls defines the git repository, branch and config file name. This is used to fetch the Pungi configuration and ensures that Raw config configuration can be stored only on trusted git repositories.

The raw_config_urls option can also define extra options influencing the generated compose (for example schema_override).

ODCS then fetches the git repository associated with the raw_config and wraps the main Pungi configuration file using the raw_config_wrapper.conf configuration file. This configuration file defines deployment related Pungi options like Koji profile or for example runroot methods. This ensures these options cannot be changed by the ODCS user.

Before executing Pungi, final Pungi configuration file is validated using the pungi-config-validate tool. It is possible to make the validation more strict by overriding default JSON schema used by Pungi. This can be done on global level using the raw_config_schema_override ODCS configuration option or per raw_config_name using the schema_override option in the raw_config_urls. It is also possible to combine these two options.

After the validation is done, ODCS executes Pungi and generates the compose.

Symlinks pointing to the generated compose are stored in the directories named according to compose type - for example nightly, ci, test or production.

ODCS server configuration

Preparing the raw_config_wrapper.conf

The /etc/odcs/raw_config_wrapper.conf is simple Pungi configuration script. It must always start with from raw_config import * to include the real raw_config configuration file.

Any further options are then used to override the real raw_config configuration.

This config file is pre-processed using Jinja2 templates. The compose Jinja2 variable can be used in this config file. This variable contains ODCS Compose JSON representation representing the current Compose.

Configuration file example:

from raw_config import *

# Override the koji_profile
koji_profile = 'odcs_stg'

# Allow overriding pkgset_koji_builds from ODCS client.
{%- if compose["builds"] %}
    pkgset_koji_builds = [
    {%- for build in compose["builds"].split(" ") %}
        '{{ build }}',
    {%- endfor %}
    ]
{%- endif %}

Adding the raw_config_urls record

The raw_config_urls in the ODCS configuration is a dict with the name of raw_config as a key and another dict as value. The dict used as value defines the raw_config and can have following keys:

• url - URL to git repository. It is later passed to git clone.

• commit [optional] - If set, it is not possible to specify commit using the ODCS client and only commit hash (or branch name) defined by this option will be used.

• path [optional] - If set, defines the path within the git repository where the configuration files are stored. This is useful in case when there are multiple independent Pungi configuration trees stored in the single git repository.

• config_filename - Name of the main Pungi configuration file.

• schema_override [optional] - If set, defines the path to JSON schema override file to be used when validating the main Pungi configuration file.

• pungi_timeout - [optional] - If set, defines the timeout in seconds in which the compose must be finished, otherwise the compose is marked as failed.

• raw_config_wrapper - [optional] - If set, defines the full path to custom raw_config_wrapper.conf file which is used by this Raw config compose.

There’s a parallel config option raw_config_credentials for configuring credentials to use with git commands to download the configuration. If Pungi is configured to use odcs-credential-helper, these credentials will be available to it too. This option should also be a dict with the same name of raw config as a key. The values should be another dict.

The key in this nested mapping is a regular expression matching URL of the repo, and the values should contain password and optionally username (unless it’s provided in the url).

Make sure that the regular expressions for URLs do not overlap, since it’s undefined which one would be used.

For example:

RAW_CONFIG_URLS = {
    "releng_fedora": {
        "url": "https://pagure.io/pungi-fedora.git",
        "config_filename": "fedora.conf",
    }
}

RAW_CONFIG_CREDENTIALS = {
    "releng_fedora": {
        "^https://gitlab.com/": {
            "username": "oauth1",
            "password": "this-is-secret"
        }
    }
}

Enabling pungi-config-validate

By default, the pungi-config-validate script is not executed for raw_config composes. It is however recommended to enable it, otherwise it is not possible to use the schema_override options.

To enable it, set the pungi_config_validate ODCS option to "pungi-config-validate" (Or to full path to the pungi-config-validate script).

Preparing the schema_override JSON file

Raw Pungi configuration files can be used to execute any command on the ODCS backend which might be a security issue in case the people editing the configuration files cannot be trusted. It is also for example possible to generate composes with external files coming from untrusted repositories.

It is therefore possible to handle cases like this using the extended JSON schema validation which will allow only certain values for certain options.

This is possible by creating global schema_override.json file and setting it using the raw_config_schema_override ODCS option. It is also possible to specify this for each raw_config using the schema_override option in the raw_config_urls ODCS option.

The schema_override.json format is the same as the one used by Pungi for the default JSON schema. The default schema can be obtained by running pungi-config-validate --dump-schema.

The schema_override.json is merged with this default JSON schema and overrides its values. For example, to allow only koji pkgset_source, the schema_override.json would look like this:

{
    "properties": {
        "pkgset_source": {
            "enum": ["koji"]
        }
    }
}

Allowing users/groups to generate raw_config composes

This is done by setting the raw_config source_type in the allowed_clients as well as particular raw_config_keys in the ODCS option like this:

allowed_clients = {
    "some_username": {
        "source_types": ["raw_config", ...],
        "raw_config_keys": ["releng_fedora", ...]
    }
}

If raw_config_keys is not specified, the user/group is able to generate any raw_config compose.

Regenerating expired raw_config compose

When regenerating expired raw_config compose using the HTTP REST API, there is a difference between the composes with production compose_type and other compose_types.

For production compose_type, ODCS stores the pungi_config_dump and regeneration such compose will result in the very same configuration file to be used. The regenerated compose should therefore be identical to original compose.

For other compose_types, the generated compose only uses the same the same source and koji_event to point to original Raw config configuration and Koji tags. Depending on the Raw config compose configuration, this can mean that the resulting compose is not 100% identical with the original one. The input packages should always be identical thanks to the same koji_event, but the configuration can differ for example in case when it points to other external git repositories and their commits using branch name. This branch name can be resolved to very different commit hash when such compose is regenerated in the future.

In both cases, the raw_config_wrapper.conf is processed normally and can influence the resulting regenerated compose.