Building Images in Koji

Image Building

Koji is capable of building many different types of images or appliances. They broadly fall into a few categories: LiveMedias, Disk Images, and Containers. All types of images end up with a Name-Version-Release just like an RPM build. What you need to provide Koji to perform a build varies by category, and the specifics will be explained below.

For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the Koji project website dictates.

Kickstart First

No matter which type of image you want to build, you need to feed Koji a kickstart file, so we touch on how to create them and how to get them to Koji first. Once you have a kickstart to try out, look over the kickstart-specific caveats for each image type in the later sections.

Kickstart Reference

If you’re new to automated installations, you should read the Anaconda Kickstart Guide first.

Getting your Kickstart to Koji

For scratch builds, you can get away with just using the --kickstart parameter, which accepts a path (relative or absolute) to your kickstart file on disk. For non-scratch builds though, the kickstart file needs to live in a remote Source Control Manager (SCM) such as git or Subversion. To access that, you need to pass the --ksurl parameter, which accepts a wacky string in this form:

git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7

Here we pointed to a repository hosted at git://git.fedorahosted.org/git/spin-kickstarts.git. Note the ? in there, this delimiter separates the host URL with a directory structure to look in for the kickstart file. In other words, if you were to clone the repository, you should expect to see these (sub)directories laid out below the root of the clone. The important thing to realize is the behavior of --kickstart changes if --ksurl is specified. Instead of indicating a path to a kickstart file, you just indicate a filename. The file itself is not specified in the fragment of the string between ? and # that is passed to --ksurl, Koji still uses the --kickstart option for that. After the directory structure comes a #, which indicates the start of the commit ID. This is the commit Koji will reset the repository to after cloning. If you created a git repository which had just the spec file template in it, your SCM URL would have the ? and # right next to each other.

If we passed --ksurl the string above, and gave --kickstart "server-ec2.ks", then Koji would do the following:

  • Consult hub SCM policy build_rpm. Name of the policy could change in the future as it has misleading name now. Policy could check if the given SCM is allowed for this method, etc. In case it deny the build, it ends here.

  • Clone the repository locally

  • Call git reset --hard 68c40eb7

  • Search in the “fedora22” subdirectory of the clone for a file called server-ec2.ks, and pass that to Anaconda

  • Perform an automated install

Building LiveMedias

Let’s describe building LiveMedias and the mechanics behind it. We also still suport older method LiveCD which is deprecated now.

Getting Started

What you need before proceeding:

  • a name for your LiveMedia, and a version number

  • a Koji build target

  • what architectures you want

  • a flattened kickstart file

  • you may also need yum repositories generated by release engineering if you require signed packages be installed into the LiveMedia

  • the permission for building these in Koji - in default installation permission is not needed, but in production environments there will be probably some permission or at least policy in place

The Koji command that creates a LiveMedia is called spin-livemedia. It calls out to livemedia-creator from lorax project in a mock chroot to construct the LiveMedia. To run a LiveMedia build, use the spin-livemedia command like so:

$ koji spin-livemedia --release 20200922.n.0  \
                      --ksurl 'git+https://pagure.io/fedora-kickstarts.git?#b4956c05028a088c641d0ce6e1a31b6d8b20176f' \
                      --install-tree-url 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200922.n.0/compose/Everything/$basearch/os' \
                      --repo 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200922.n.0/compose/Everything/$basearch/os' \
                      --can-fail ppc64le,aarch64 \
                      Fedora-Workstation-Live 33 f33 x86_64,ppc64le,aarch64 fedora-live-workstation.ks

In this example (real task is here) a LiveMedia will be created with the N-V-R of Fedora-Workstation-Live-33-20200922.n.0. If --release was not included an incrementing value would be computed by Koji instead. spin-livemedia takes a minimum of 4 arguments:

Name

the name of the image, without versioning information. Examples could be Fedora-Live or Fedora-Workstation-Live.

Version

an arbitrary version string. For Fedora images, this usually matches the release version, such as 31, 32, or 33.

Build Target

just like RPM builds Koji must be told which target to use. This controls what tag the image will be tagged into, and what packages are available to install into the image.

Kickstart File

this is a recipe file that performs drives the construction of the image. Note, that you’re can either upload your kickstart or just point to some SCM (--ksurl). In our example fedora repo contains fedora-live-workstation.ks kickstart file.

Use --help to discover more options to spin-livemedia. You can override the Release field with --release, or --repo to override what yum repo is used to provide packages to install to the image. Note that regardless of what repo you use, it must contain RPMs built by Koji, otherwise you will get an error. (unless you are building a scratch image). --can-fail options says which architectures are optional and can fail without failing the whole build.

Mechanics

The way LiveMedias are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies from the livemedia-build package group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livemedia-creator command is executed using the mock('--chroot', ...) method.

Note

This process runs as root. This produces the desired image which is uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.

Caveats for LiveMedias

There are some known caveats with using the spin-livemedia command that users should be aware of.

%include macros in the kickstart

A word of caution about kickstart files and the %include macro. livemedia-creator is smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the fedora-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user’s local disk.

Package Groups in the Kickstart File

Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don’t since Koji’s comps.xml is based on the “groups” set up from the CLI. livemedia-creator’s behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are a couple possible workarounds:

  • do not use package groups in the kickstart file and just specify a huge list of packages

  • use --repo and specify a repo that does have a comps.xml that defines the groups it uses

Only Include RPMs Built in Koji

The image building tasks will fail if your image tries to include a package that was not built in your build system. This is because the package does not have any origin information stored in Koji’s database. The repos defined in the kickstart will automatically be overridden with the repo of the build tag for the build target, unless you use the --repo option. Since only packages you have built (or include from an external repo) should be there, you should never have this problem unless you use --repo.

No Signed or Debuginfo RPMs in Koji’s Build Tags

If you need signed RPMs or debuginfo RPMs, you will run into trouble because Koji does not keep those in its build tag repos. The only work around for this is to create a repo yourself that includes these RPMs and then use --repo. This will force the image to take RPMs from that repo. Remember, the task will fail if Koji detects RPMs were installed that were not built in the build system.

%post Section in Kickstart

While livemedia-creator does support building on SELinux disabled hosts, you can run into denials when booting if you create and use new files in the %post section of your kickstart file. If you do, you should either set the labels appropriately at the end of the %post section, or instigate an autorelabel at boot time.

Troubleshooting

If your build fails, you will be notified on the command line. In the output will be a URL to the Koji UI, visit that and click on the red subtask link. From that page review root.log and livemedia-out.log for errors. Often errors are caused by packages being missing, or malformed kickstart files. The log files are at the bottom of the page. If problem occurs later during installation livemedia-creator will also upload a lot of other logs like anaconda’s, etc. If you’re stuck, contact Release Engineering.

Build System Preparation

This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so here if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how livemedia-creator works. This section only covers preparation for LiveMedia builds.

Follow this procedure step by step to get things prepared they way they need to be.

  1. koji add-host-to-channel <your-host> livemedia

    add a builder to the livemedia channel

  2. koji grant-permission livemedia <user>

    grant the permission to build an image type to a user. This step is optional since admins have all permissions.

  3. You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveMedias with by default. (the alternative is to use the --repo option, more on that later)

  4. koji add-group <build-tag> livemedia-build add the livemedia-build group

  5. koji add-group-pkg <build-tag> livemedia-build <pkg> ...

    add packages to the livemedia-build group. These package lists vary has packages and dependencies change. As of September, 2020 for Fedora 33 the needed packages for each image type are:

    • bash, coreutils, glibc-all-langpacks, lorax-lmc-novirt, selinux-policy-targeted, shadow-utils, util-linux

Building Disk Images

Disk images are files that represent virtual disks. They have a partition table and filesystems on them, and are available in a variety of formats: qcow2, vmdk, ova, Hyper-V, raw, “base” container images, and more.

Getting Started

What you need:

  • a name for your image, and a version number

  • what architectures you want

  • a Koji build target

  • kickstart file

  • installation tree

  • you may also need yum repositories generated by Rel-Eng if you require signed packages be installed into the image

The Koji command to build a disk image is called image-build. The image-build command uses ImageFactory and Oz to start a VM guest and perform an automated Anaconda installation. Here is a (lengthy) example for building a disk image.

$ koji image-build --repo 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' --kickstart fedora-server.ks --scratch --distro Fedora-22 --format qcow2 fedora-server-kvm 22 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' x86_64

This example builds a scratch qcow2 disk image using packages from an additional yum repository. Without this option the yum repo to populate the build root would be used instead. If this was the first image with the N-V of fedora-server-kvm-22, then the N-V-R would be fedora-server-kvm-22-1, because Koji uses an incrementing number for the release if you do not provide one. Like all Koji commands, use --help to see more options that are available.

For Docker, Koji only supports Base Images right now using a kickstart file as described above. In the future it will support layered images, but not before some Docker requirements are met, and Koji is maintaining a Registry of its own. This scoping effort is ongoing.

image-build takes a minimum of 5 positional arguments, and 2 options must be specified. They are reviewed in the list below, with the positional arguments first.

Name

the name of the image, without versioning information. Examples could be fedora-server or fedora-workstation.

Version

an arbitrary version string. For Fedora images, this usually matches the release version, such as 22 or 23.

Build Target

just like RPM builds Koji must be told which target to use. This controls what tag the image will be tagged into, and what packages are available to install into the image.

Installation Tree URL

this is a URL to a location you can install an operating system from. It is the same place you would direct a PXE-booted system to go. In 99% of cases this location is provided by Release Engineering. It should have an “isolinux” subdirectory and yum metadata somewhere within.

Architecture

only x86_64 or i386 is supported, and you can specify both on the command line. This will cause two subtasks to be run, allowing you to build for both arches in parallel. If either fail, the whole build will fail.

Kickstart File

this is a recipe file that performs drives the construction of the image. Pass in the path to a kickstart file, which must be flattened.

Kickstart URL

in a non-scratch build, you’ll need this too. For more details, see the Getting your Kickstart to Koji section.

Distro

a string that indicates what OS is being built. These always follow the convention of “Fedora-X”, where X is the release number.

Since this command can get very long, a configuration file can drive the task as well, using the --config option. It accepts a path to a configuration file written in the Python ConfigParser format (like a Windows .ini). The options are all named the same with one caveat, see below. Here’s what one could look like:

[image-build]
name = fedora-server-docker
version = 22
target = f22-candidate
install_tree = https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
arches = x86_64

format = qcow2,rhevm-ova,vsphere-ova
distro = Fedora-22
repo = https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
disk_size = 20

ksversion = DEVEL
kickstart = fedora-22-server-docker.ks
ksurl = git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7
specfile = git://git.fedorahosted.org/git/spin-kickstarts.git?spec_templates/fedora22#68c40eb7

A few notes on the syntax:

  • it allows for comments too, the lines start with a hash (#)

  • options on the command line that can be used multiple times can accept values here as comma-separated strings

  • options with a hyphen need to use an underscore instead. --disk-size for example would be disk_size in the config file

OVA Features

If you’re building OVAs, either for RHEVM or vSphere, you can specify OVA options with a special section in the configuration file. It looks something like this:

[ova-options]
vsphere_product_version=22
rhevm_description=Fedora Cloud 22
vsphere_product_vendor_name=Fedora Project
ovf_memory_mb=6144
rhevm_default_display_type=1
vsphere_product_name=Fedora Cloud 22
ovf_cpu_count=4
rhevm_os_descriptor=Fedora-22

or this:

[ova-options]
vsphere_ova_format = vagrant-virtualbox
rhevm_ova_format = vagrant-libvirt
vagrant_sync_directory = /home/vagrant/sync

The second one is actually the secret sauce for generating an image for use in Vagrant. At this time, you would need rename the image file extension from .ova to .box, but otherwise this should work fine.

Kickstart Preparation

Kickstarts for the image-build command have some specific requirements which are covered in this section.

Required Kickstart Arguments

Anaconda of course requires many commands to be defined in the kickstart file. If you’re starting from scratch you should review the reference linked above, or use an existing kickstart file in the spin-kickstarts git repo. It is critically important that the installation be completely automated, if Anaconda has to prompt for input for any reason, the build will fail because you cannot send input to the guest. Some of the kickstart commands are optional to Anaconda, but are required in Koji for your build to succeed. Here’s the list and the reasons why.

zerombr

You must tell Anaconda to wipe out the MBR in the virtual block device, if you don’t Anaconda will ask you.

clearpart –all –initlabel

Anaconda has to be told to wipe out all data on the virtual block device we install on otherwise it will ask for confirmation to do so. Since it is blank anyway this is harmless.

reboot

When the installation completes, the guest is rebooted. ImageFactory is specifically looking for this behavior to conclude the installation completed. Anaconda’s default behavior is to wait for a key press to reboot the system, but this is impossible from outside of Koji.

locking the root account

You have to lock the root account (rootpw –lock) or create a non-root user (user), otherwise Anaconda will prompt for one.

Do not use the url command

The repo commands are overridden by Koji to point to internal Koji repos, or what you specified on the command line with --repo, it does not override the url command if you provided it. Anaconda has a behavior where it will prefer packages from the repositories given with the url command over those with the repo command, and this is generally not what you want. If Koji sees an RPM was installed that was not built in the system, it will fail the build.

Troubleshooting

If your disk image build fails, follow the link in the command line output that takes you to the task page in the Koji web UI. Click on the failed createImage subtask in red. On that page review the screenshot.ppm file if it was provided, or oz.log. Most failures are from Anaconda rejecting a malformed kickstart file, which will be indicated in the screenshot. Your installation must be completely automatic, there can be no interactivity at all, otherwise Anaconda will sit there indefinitely until Koji (actually ImageFactory) kills the task.

It is very easy to write a kickstart file with bugs or that results in a system that does not boot. This section will present a series of questions to ask yourself and examples to help diagnose where the problem lies. Once you know that, it should be easier to understand what you can do to inspect further.

There are 4 steps in the process:

  1. create a guest

  2. perform an automated installation in the guest

  3. boot the guest and extract the list of installed RPMs

  4. upload and archive the disk image of the guest

Is it a problem with guest creation?

There have been unusual cases where libvirt, ImageFactory, or Oz was misconfigured and guests could not be started properly. A misconfiguration with Puppet or whatever Fedora Infrastructure is up to can cause this. So far the errors have been clear in the task output, look either in the results string or oz.log. The bad news is that in this case you really can only inform Rel-Eng about the issue and wait for a resolution. The good news is these cases are very rare.

Did the installation fail?

The Anaconda installation can fail for many reasons: missing packages, network problems, or syntax errors in %post. Tasks will also fail if Anaconda prompts for input for any reason. If Koji detects a lack of disk activity in the guest for more than 5 minutes, it will fail the build and tear down the guest. Looking in oz.log may have the answer: dracut, anaconda, and yum logs are all printed there.

These sorts of failures often have a screenshot taken and saved with the task output called screenshot.ppm. Viewing this will usually tell you what Anaconda is complaining about if the installer detected an issue or prompted for input. The string in the results output that says “No disk activity in 300 seconds, failing.” This almost always means Anaconda hit an issue and either gave up or waited.

If Anaconda claims it is missing packages, confirm they exist in the repos you are using with --repo, if you are using that option. If you are not, confirm the builds you expect are in the tag inheritance for the target you are running. This is a lot like checking whether an RPM will build against the right libraries, except we’re building an image instead.

If you get the rare Anaconda dialog box that says something like “An unexpected error occurred”, try using the text command in kickstart, which will have Anaconda boot in text mode. Sometimes the Python traceback (or whatever the error condition is) will be printed there. I have also seen cases where text-mode yields a black screen, but booting in graphical mode (the default) does produce a useful dialog box. Issues like this stem from syntax errors in the kickstart file, or bugs in pykickstart itself. If you think it is a pykickstart bug, then someone in Rel-Eng needs to update pykickstart on the builders.

Did the guest boot?

Koji waits 5 minutes for a guest to boot in this step. It unfortunately does not give a lot of insight to why a guest may not boot, so these are a tougher class of issues to work through. You can usually answer this question by looking in results string. If you see “Timed out waiting for guest to boot”, then this is your problem. You can also confirm this in oz.log.

For now, the best way to investigate an issue like this is to drive a guest installation locally using something like Gnome’s Virtual Machine Manager (VMM). The steps to perform are:

  • Select a Network Install

  • For the Operating System Install URL use the same one you gave to Koji. It will be something like https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/

  • Set the Kickstart URL to where your kickstart file is. You may need to make it available over http.

  • Bump the memory to 2048M for good measure

  • Launch the guest and let it complete installation

  • Open a VNC session and watch what happens when the guest attempts to boot.

If the console is not providing enough information, we have to get more creative. Anaconda supports starting an SSH daemon while the installation is happening with the sshpw command in kickstart. Set that and comment out the reboot command. This will let the installation complete locally and wait for a keystroke to reboot the guest. At this point you should be able to ssh in and inspect the environment to figure out what is going on. You should also consider making use of the --log option to %post so that output from the script is saved somewhere.

Another option would be to scp logs and other files off of the guest as part of the %post script.

Other Guest Misconfigurations

If the guest boots but you’re having problems accessing it I’d suggest following same procedure as when the guest fails to boot. This could be a result of firewall misconfigurations or SSH not being available for some reason. Usually in this case the build is succeeding in Koji, but there’s something still fundamentally broken in the image. If the issue is something you can investigate while the guest is online (you can log in), then I’d suggest importing it locally using the libvirt.xml and the disk image provided in Koji’s task output.

You can also do investigative work in an offline mode by mounting the image locally or using something like libguestfs to poke around without starting the guest. The fast, dirty way to do it is by mounting it. This can often pollute your guest environment. Here’s how to do it:

  • Download the image from Koji

  • If the image format is not raw, you have to convert it first with qemu-img. Something like:

    $ qemu-img convert -O raw <image-file> <output-file></pre>

  • Now mount it up using loopback devices. (as root) If your image has multiple partitions in it, you may need to pass in a different mapped loopback device like loop0p2. Whichever one you think is the root partition or has the issue you’re trying to fix.

    # kpartx -av <raw-image>
# mount -o loop /dev/mapper/loop0p1 /mnt/my_directory

Hopefully at this point you figure out the issue. To tear down the image you’ll run commands as root like so:

# umount /mnt/my_directory
# dmsetup remove loop0p1
# losetup -d /dev/loop0

Again, if you used different loopback devices, substitute those in to the dmsetup and losetup commands.

Build System Preparation

Follow this guide if you’re a Koji admin and would like to enable image building or want to set up some testing before enabling the integration.

When moving to ImageFactory to do image builds Koji lost the ability to easily reproduce the build environment for images the way we do for RPMs using Mock. This section will document how to set up an image builder for Koji. This is a lengthy task, it will take first-timers about a week to have a useful instance, and it is painful because it requires a bare metal system be provisioned since ImageFactory provisions VMs to build the image. There is a significant performance penalty for using nested virtualization.

Follow the steps below to set up your builder.

Note

You do not have to stand up a complete Koji instance to test the way Koji builds images. However, if you want to test image builds with an accurate representation of how Koji does it, or you want to test code changes related to image builds in Koji, you should follow all the steps below.

ImageFactory/Oz Preparation

  1. Provision a system with at least 4G of memory with the current release of RHEL 6 or later. Sometimes builders lag behind a month or two before taking in updates, but this will still get you pretty close to where you want to be. For Fedora, use the latest release.

  2. Install the following packages to the builder. #. oz #. imagefactory #. imagefactory-plugins-TinMan #. imagefactory-plugins-vSphere #. imagefactory-plugins-ovfcommon #. imagefactory-plugins-docker #. imagefactory-plugins #. imagefactory-plugins-OVA #. imagefactory-plugins-RHEVM #. python-psphere => 0.5 #. VMDKStream => 0.2 #. pykickstart

  3. Edit /etc/kojid/kojid.conf, and set an second value, eg: 7200 for oz_install_timeout. It’s a timeout waiting guest installing. Default value is 0, that means oz will use its default value. Since oz-0.16.0, it can be configured in /etc/oz/oz.cfg as install in [timeouts] section.

  4. Edit /etc/oz/oz.cfg, and set the memory value in the [libvirt] section to at least 2048. Set safe_generation under [icicle] to yes.

  5. Run: mkdir -p ~root/.psphere/templates, and then copy the following code into ~root/.psphere/config.yaml. Do not worry about the server, username, and password credentials; they are not used anywhere.

    general:
  server: 10.16.120.224
  username: Administrator
  password: whatever
  template_dir: ~/.psphere/templates/
logging:
  destination: ~/.psphere/psphere.log
  level: DEBUG # DEBUG, INFO, etc

Start up the services and ImageFactory/Oz should be ready to go. You should read more about how to use Oz and how to use ImageFactory. If you want to try calling ImageFactory as if from a Koji Builder (but not set up a whole Koji instance), you can use the code below to emulate that. If you want to test the Koji integration with a full Koji instance, proceed to the next section instead.

#!/usr/bin/python -tt

import logging
import os.path
import random
import sys

from imgfac.BuildDispatcher import BuildDispatcher
from imgfac.PluginManager import PluginManager
from imgfac.ReservationManager import ReservationManager
plugin_mgr = PluginManager('/etc/imagefactory/plugins.d')
plugin_mgr.load()
from imgfac.ApplicationConfiguration import ApplicationConfiguration

# logging
handler = logging.StreamHandler(sys.stdout)
tlog = logging.getLogger()
tlog.setLevel(logging.DEBUG)
tlog.addHandler(handler)

# configuration
ks = open('oztest.ks').read()
workdir = '/tmp/koji/test'
config =  {
    #Oz specific
    'oz_data_dir': os.path.join(workdir, 'oz_data'),
    'oz_screenshot_dir': os.path.join(workdir, 'oz_screenshots'),
    #IF specific
    'imgdir': os.path.join(workdir, 'scratch_images'),
    'tmpdir': os.path.join(workdir, 'oz-tmp'),
    'verbose' : True,
    'timeout': 3600,
    'output': 'log',
    'raw': False,
    'debug': True,
    'image_manager': 'file',
    'plugins': '/etc/imagefactory/plugins.d',
    'tdl_require_root_pw': False,
    'image_manager_args': {
        'storage_path': os.path.join(workdir, 'output_image')},
}
random.seed() # necessary to ensure a unique mac address
rm = ReservationManager()
rm._listen_port = random.randint(rm.MIN_PORT, rm.MAX_PORT)
ApplicationConfiguration(configuration=config)
params = {'install_script': ks}
template = """<template>
    <name>test-appliance</name>
        <os>
            <name>Fedora</name>
            <version>22</version>
            <arch>x86_64</arch>
            <install type='url'>
                <url>https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/</url>
            </install>
            <icicle>
                <extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>
            </icicle>
        </os>
    <description>test-appliance OS</description>
    <disk>
        <size>16G</size>
    </disk>
</template>
"""
bd = BuildDispatcher()

# build the image
base = bd.builder_for_base_image(template, parameters=params)
base.base_thread.join()
tlog.removeHandler(handler)

This script is run as root with no arguments. It uses oztest.ks in your local directory as the kickstart you want to try to use. The URL in the XML template is where the RPM packages will be installed from, and what the guest will be booted with.

Koji Preparation

  1. Install Koji if you need it. This will easily be the most time-consuming part of the process; my first time took 3 days to get it working properly. Follow the guide closely, and go with the SSL authentication method. SSL is a lot easier to set up locally. You will need to install every Koji component (except koji-vmd) on the same system. Proceed to the next step after you’ve had a successful Kojira repository generated.

  2. At this point, you have a system that should be ready to build images. We just have to do some Koji configuration so that your instance is pulling content from Koji. Replace the base tag names with whatever fits your conventions.

    1. Add tags, tag inheritance, new pkg (with pkg owner), new external repo, and regen the repo

      koji add-tag fedora22
koji add-tag jay-fedora22
koji add-tag jay-fedora22-override --parent jay-1-fedora22
koji add-tag jay-fedora22-build --arches x86_64 --parent jay-fedora22-override
koji add-tag jay-fedora22-candidate --parent jay-fedora22
koji add-tag-inheritance --priority 40 jay-fedora22-build fedora22
koji add-pkg --owner kojiadmin jay-fedora22 fedora-server-ec2 fedora-server-kvm
koji add-external-repo -t fedora::20 fedora22 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'
koji add-target jay-fedora22-candidate jay-fedora22-build
koji regen-repo jay-fedora22-build

    2. Grab a kickstart file from an image task in Koji that relates to what you want to test.

    3. Finally, kick off a build!

      koji image-build fedora-server-ec2 22 --distro Fedora-22 jay-fedora22-candidate --kickstart fedora-server-starter-ec2.ks 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'

Building Appliances

This section is here for the sake of legacy. Unless you are trying to build ARM images, you should use the image-build command described in the previous section.

Note

The spin-appliance command, described herein, is deprecated.

Getting Started

Here’s what you need before proceeding:

  • a name for your Appliance, and a version number

  • a Koji build target

  • what architectures you want

  • a flattened kickstart file

  • you may also need yum repositories generated by release engineering if you require signed packages be installed into the appliance

  • the appliance permission in Koji

The Koji command that creates an Appliance is called spin-appliance. It calls out to appliance-creator in a mock chroot to construct the Appliance. To run an Appliance build, use the spin-appliance command like so:

$ koji spin-appliance --release 4 fedora-workstation 23 f23-build fedora-workstation.ks

In this example an Appliance will be created with the N-V-R of fedora-workstation-23-4. If --release was not included an incrementing value would be computed by Koji instead. spin-appliance takes a minimum of 5 arguments:

Name

the name of the image, without versioning information. Examples could be fedora or fedora-workstation.

Version

an arbitrary version string. For Fedora images, this usually matches the release version, such as 21, 22, or 23.

Build Target

just like RPM builds Koji must be told which target to use. This controls what tag the image will be tagged into, and what packages are available to install into the image.

Architecture

only arm, x86_64, or i386 are supported

Kickstart File

this is a recipe file that performs drives the construction of the image.

Use --help to discover more options to spin-appliance. You can override the Release field with --release, or --repo to override what yum repo is used to provide packages to install to the image. Note that regardless of what repo you use, it must contain RPMs built by Koji, otherwise you will get an error. (unless you are building a scratch image)

Mechanics

The way Appliances are created in Koji is the same as Building LiveMedias.

Caveats for Appliances

Caveats for using spin-appliance are the same as using spin-livemedia to Caveats for LiveMedias.

Troubleshooting

If your build fails, you will be notified on the command line. In the output will be a URL to the Koji UI, visit that and click on the red subtask link. From that page review root.log and appliance.log for errors. Often errors are caused by packages being missing, or malformed kickstart files. The log files are at the bottom of the page. If you’re stuck, contact Release Engineering.

Build System Preparation

This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so here if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how appliance-creator works. This section only covers preparation for Appliance builds.

Follow this procedure step by step to get things prepared they way they need to be.

  1. Add a builder to the appliance channel
    koji add-host-to-channel <your-host> appliance

  2. Grant the permission to build an appliance to a user. This step is optional since admins have all permissions.

    koji grant-permission appliance <user>

  3. You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the Appliances with by default. (the alternative is to use the --repo option, more on that later)

  4. Add the appliance-build group

    koji add-group <build-tag> appliance-build``

  5. Add packages to the appliance-build group. These package lists vary has packages and dependencies change. As of October, 2015 for Fedora 24 the needed packages for appliances:

    • appliance-tools, bash, coreutils, grub, parted, perl, policycoreutils, selinux-policy, shadow-utils, sssd-client

    koji add-group-pkg <build-tag> appliance-build <pkg> ...