Fedora RelEng Workflow Automation

The Fedora RelEng Workflow Automation is a means to allow RelEng to define a pattern by which Release Engineering work is automated in an uniform fashion. The automation technology of choice is ansible and the “workflow engine” is powered by loopabull, which is an event loop that allows us to pass the information contained within a fedmsg and insert it into ansible playbooks. This will effectively create an event driven workflow that can take action conditionally based on the contents of arbitrary fedmsg data.

Background on the topic can be found in the Release Engineering Automation Workflow Engine Change proposal, as well as in the releng-automation pagure repository.

RelEng Workflow Automation Architecture

By using fedmsg as the source of information feeding the event loop, we will configure loopabull to listen for specific fedmsg topics which will correspond with ansible playbooks. When one of the appropriate fedmsg topics is encountered across the message bus, it’s message payload is then injected into the corresponding playbook as an extra set of variables. A member of the Fedora Release Engineering Team can at that point use this as a means to perform whatever arbitrary action or series of actions they can otherwise perform with ansible (including what we can enable via custom modules) based on the input of the message payload.

The general overview of the architecture is below as well as a description of how it works:

             +------------+
             |   fedmsg   |
             |            |
             +---+--------+
                 | ^
                 | |
                 | |
                 | |
                 | |
                 | |
                 V |
+------------------+-----------------+
|                                    |
|      Release Engineering           |
|      Workflow Automation Engine    |
|                                    |
|      - RabbitMQ                    |
|      - fedmsg-rabbitmq-serializer  |
|      - loopabull                   |
|                                    |
+----------------+-------------------+
                 |
                 |
                 |
                 |
                 V
     +-----------------------+
     |                       |
     | composer/bodhi/etc    |
     |                       |
     +-----------------------+

The flow of data will begin with an event somewhere in the Fedora Infrastructure that sends a fedmsg across the message bus, then the messages will be taken in and serialized in to a rabbitmq worker queue using fedmsg-rabbitmq-serializer. Then loopabull will be listening to the rabbitmq worker queue for tasks to come in. Once a message is recieved, it is processed and once it is either no-op’d or a corresponding ansible playbook is run to completion, the message will be ack’d and cleared from the worker queue. This will allow for us to scale loopabull instances independently from the message queue as well as ensure that work is not lost because of a downed or busy loopabull instance. Also, as a point of note, the loopabull service instances will be scaled using systemd unit templates.

Once a playbook has been triggered, it will run tasks on remote systems on behalf of a loopabull automation user. These users can be privileged if need be, however the scope of their privilege is based on the purpose they serve. These user accounts are provisioned by the Fedora Infrastructure Team based on the requirements of the RelEng Task Automation User Request Standard Operating Procedure (SOP) document and tasks are subject to code and security audit.

Fedora Lib RelEng

Fedora Lib RelEng (flr), is a library and set of command line tools to expose the library that aims to provide re-usable code for common tasks that need to be done in Release Engineering. Combining this set of command line tools when necessary with the Release Engineering Automation pipeline allows for easy separation of permissions and responsibilities via sudo permissions on remote hosts. This is explained in more detail on the project’s pagure page.