Fedora Layered Image Build System¶
The Fedora Layered Image Build System aims to provide an new type of official binary artifact produced by Fedora. Currently, we produce two main types of artifacts: RPMs, and images. The RPMs are created in Koji from specfiles in dist-git. The images come in different formats, but have in common creation in Koji from kickstart files — this includes the official Fedora Docker Base Image. This change introduces a new type of image, a Docker Layered Image, which is created from a Dockerfile and builds on top of that base image.
Layered Image Build Service Architecture¶
+------------------------------+
| Future Items to Integrate |
+------------------------------+
| +--------------------------+ |
| |PDC Integration | |
| +--------------------------+ |
| |New Hotness | |
| +--------------------------+ |
| |Other??? | |
| +--------------------------+ |
| Magical Future |
| |
+------------------------------+
+------------------+
|Fedora |
|Users/Contributors|
+--------+---------+
^
| +----------------+
| | Fedora |
+-------+-----------+ | Layered Image |
| Fedora Production | | Maintainers |
| Docker Registry | | |
+-------------------+ +-------+--------+
^ |
| V
| +-------------------+--+
+-----------+------------+ | |
| RelEng Automation | | +------------------+ |
| (Release) | | |Dockerfile | |
+-----------+------------+ | +------------------+ |
^ | |App "init" scripts| |
| | +------------------+ |
| | |Docs | |
+---------+------------+ | +------------------+ |
| Taskotron | | DistGit |
| Automated QA | | |
+---------+-------+----+ +-----------+----------+
^ ^ |
| | |
| | |
| | |
| +-------------------+ |
| | |
[docker images] | |
| | |
| [fedmsg] |
+---------------+-----------+ | |
| | | +---------------+
| +----------------------+ | | |
| |Intermediate Docker | | +------+-------------------+ |
| |Images for OSBS Builds| | | | |
| +----------------------+ | | +----------------------+ | |
| |Layered Images | | | |containerbuild plugin | | |
| +----------------------+ | | +----------------------+ | |
| |docker|distribution | | | |rpmbuilds | | |
| +----------------------+ | | +----------------------+ | |
| Registry | | koji | |
| | | | |
+-----+----+----------------+ +-----+---+----+-----------+ |
| ^ | ^ ^ |
| | | | | |
| | +---------------------------+ | | |
| | | [osbs-client api] | | |
| | | +---------------------------+ | |
| | | | | |
| | | | | |
V | V | | |
+--+----+---+---+---+ | V
| | +-------------------+---+
| +--------------+ | |fedpkg container-build +
| |OpenShift v3 | | +-----------------------+
| +--------------+ |
| |Atomic Reactor| |
| +--------------+ |
| OSBS |
| |
+-------------------+
[--------------------- Robosignatory ------------------------------------]
[ Every time an image is tagged or changes names, Robosignatory signs it ]
[ ]
[ NOTE: It's injection points in the diagram are ommitted for brevity ]
[------------------------------------------------------------------------]
Layered Image Build System Components¶
The main aspects of the Layered Image Build System are:
Koji
koji-containerbuild plugin
OpenShift Origin v3
Atomic Reactor
osbs-client tools
A docker registry
docker-distribution
Taskotron
fedmsg
RelEng Automation
The build system is setup such that Fedora Layered Image maintainers will submit
a build to Koji via the fedpkg container-build command a containers
namespace within DistGit. This will trigger the build to be scheduled in
OpenShift via osbs-client tooling, this will create a custom
OpenShift Build which will use the pre-made buildroot Docker image that we
have created. The Atomic Reactor (atomic-reactor) utility will run within
the buildroot and prep the build container where the actual build action will
execute, it will also maintain uploading the Content Generator metadata back
to Koji and upload the built image to the candidate docker registry. This
will run on a host with iptables rules restricting access to the docker bridge,
this is how we will further limit the access of the buildroot to the outside
world verifying that all sources of information come from Fedora.
Completed layered image builds are hosted in a candidate docker registry which is then used to pull the image and perform tests with Taskotron. The taskotron tests are triggered by a fedmsg message that is emitted from Koji once the build is complete. Once the test is complete, taskotron will send fedmsg which is then caught by the RelEng Automation Engine that will run the Automatic Release tasks in order to push the layered image into a stable docker registry in the production space for end users to consume.
Note that every time the layered image tagged to a new registry, ultimately changing it’s name, Robosignatory will automatically sign the new image. This will also occur as part of the Automated Release process as the image will be tagged from the candidate docker registry into the production docker registry in order to “graduate” the image to stable.
Koji¶
Koji is the Fedora Build System.
koji-containerbuild plugin¶
The koji-containerbuild plugin integrates Koji and OSBS so that builds can be scheduled by koji and integrated into the build system with imports of metadata, logs, build data, and build artifacts.
OpenShift Origin v3¶
OpenShift Origin v3 is an open source Container Platform, built on top of kubernetes and Docker. This provides many aspects of the system needed including a build pipeline for Docker images with custom build profiles, image streams, and triggers based on events within the system.
Atomic Reactor¶
Atomic Reactor is an utility which allows for the building of containers from within other containers providing hooks to trigger automation of builds as well as plugins providing automatic integration many other utilities and services.
osbs-client tools¶
osbs-client tools allow for users to access the build functionality of OpenShift Origin v3 using a simple set of command line utilities.
docker-registry¶
A docker-registry is a stateless, highly scalable server side application that stores and lets you distribute Docker images.
There are many different implementations of docker-registries, two main ones are supported by the Fedora Layered Image Build System.
docker-distribution¶
The docker-distribution registry is considered the Docker upstream “v2 registry” such that it was used by upstream to implement the new version 2 specification of the docker-registry.
Fedora Production Registry¶
Implementation details of this are still unknown at the time of this writing and will be updated at a later date. For the current status and implementation notes please visit the FedoraDockerRegistry page.
Taskotron¶
Taskotron is an automated task execution framework, written on top of buildbot that currently executes many Fedora automated QA tasks and we will be adding the Layered Image automated QA tasks. The tests themselves will be held in DistGit and maintained by the Layered Image maintainers.
RelEng Automation¶
RelEng Automation is an ongoing effort to automate as much of the RelEng process as possible by using Ansible and being driven by fedmsg via Loopabull to execute Ansible Playbooks based on fedmsg events.
Robosignatory¶
Robosignatory is a fedmsg consumer that automatically signs artifacts and will be used to automatically sign docker layered images for verification by client tools as well as end users.
Future Integrations¶
In the future various other components of the Fedora Infrastructure will likely be incorporated.
PDC¶
PDC is Fedora’s implementation of Product Definition Center which allows Fedora to maintain a database of each Compose and all of it’s contents in a way that can be queried and used to make decisions in a programatic way.
The New Hotness¶
The New Hotness is a fedmsg consumer that listens to release-monitoring.org and files bugzilla bugs in response (to notify packagers that they can update their packages).