=======================
Building Images in Koji
=======================

Image Building
==============

Koji is capable of building many different types of images or appliances. They
broadly fall into a few categories: LiveMedias, Disk Images, and Containers. All
types of images end up with a Name-Version-Release just like an RPM build.  What
you need to provide Koji to perform a build varies by category, and the
specifics will be explained below.

For additional questions and information, ask around in ``#koji`` on FreeNode
IRC and sign up to the right mailing lists as the `Koji project website`_
dictates.

Kickstart First
===============

No matter which type of image you want to build, you need to feed Koji a
`kickstart`_ file, so we touch on how to create them and how to get them to Koji
first. Once you have a kickstart to try out, look over the kickstart-specific
caveats for each image type in the later sections.

Kickstart Reference
-------------------

If you're new to automated installations, you should read the `Anaconda
Kickstart Guide`_ first.

Getting your Kickstart to Koji
------------------------------

For scratch builds, you can get away with just using the ``--kickstart``
parameter, which accepts a path (relative or absolute) to your kickstart file
on disk. For non-scratch builds though, the kickstart file needs to live in a
remote Source Control Manager (SCM) such as git or Subversion. To access that,
you need to pass the ``--ksurl`` parameter, which accepts a wacky string in
this form:

::

    git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7

Here we pointed to a repository hosted at
``git://git.fedorahosted.org/git/spin-kickstarts.git``. Note the ``?`` in
there, this delimiter separates the host URL with a directory structure to
look in for the kickstart file. In other words, if you were to clone the
repository, you should expect to see these (sub)directories laid out below the
root of the clone. The important thing to realize is the behavior of
``--kickstart`` changes if ``--ksurl`` is specified. Instead of indicating a
path to a kickstart file, you just indicate a filename. The file itself is not
specified in the fragment of the string between ``?`` and ``#`` that is passed
to ``--ksurl``, Koji still uses the ``--kickstart`` option for that. After the
directory structure comes a ``#``, which indicates the start of the commit ID.
This is the commit Koji will reset the repository to after cloning. If you
created a git repository which had just the spec file template in it, your SCM
URL would have the ``?`` and ``#`` right next to each other.

If we passed ``--ksurl`` the string above, and gave
``--kickstart "server-ec2.ks"``, then Koji would do the following:

* Clone the repository locally
* Call ``git reset --hard 68c40eb7``
* Search in the "fedora22" subdirectory of the clone for a file called
  ``server-ec2.ks``, and pass that to Anaconda
* Perform an automated install

.. _building-livemedia:

Building LiveMedias
===================

Let's describe building LiveMedias and the mechanics behind it. We also still
suport older method LiveCD which is deprecated now.

Getting Started
---------------

What you need before proceeding:

* a name for your LiveMedia, and a version number
* a Koji build target
* what architectures you want
* a flattened kickstart file
* you may also need yum repositories generated by release engineering if you
  require signed packages be installed into the LiveMedia
* the permission for building these in Koji - in default installation permission
  is not needed, but in production environments there will be probably some
  permission or at least policy in place

The Koji command that creates a LiveMedia is called ``spin-livemedia``. It calls
out to `livemedia-creator`_ from `lorax`_ project in a mock chroot to construct
the LiveMedia. To run a LiveMedia build, use the ``spin-livemedia`` command like
so:

::

    $ koji spin-livemedia --release 20200922.n.0  \
                          --ksurl 'git+https://pagure.io/fedora-kickstarts.git?#b4956c05028a088c641d0ce6e1a31b6d8b20176f' \
                          --install-tree-url 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200922.n.0/compose/Everything/$basearch/os' \
                          --repo 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200922.n.0/compose/Everything/$basearch/os' \
                          --can-fail ppc64le,aarch64 \
                          Fedora-Workstation-Live 33 f33 x86_64,ppc64le,aarch64 fedora-live-workstation.ks

In this example (real task is `here
<https://koji.fedoraproject.org/koji/taskinfo?taskID=52013359>`_) a LiveMedia will
be created with the N-V-R of ``Fedora-Workstation-Live-33-20200922.n.0``. If
``--release`` was not included an incrementing value would be computed by Koji
instead. spin-livemedia takes a minimum of 4 arguments:

Name
    the name of the image, without versioning information. Examples could be
    ``Fedora-Live`` or ``Fedora-Workstation-Live``.

Version
    an arbitrary version string. For Fedora images, this usually matches the
    release version, such as 31, 32, or 33.

Build Target
    just like RPM builds Koji must be told which target to use. This controls
    what tag the image will be tagged into, and what packages are available to
    install into the image.

Kickstart File
    this is a recipe file that performs drives the construction of the image.
    Note, that you're can either upload your kickstart or just point to some SCM
    (``--ksurl``).  In our example fedora repo contains
    ``fedora-live-workstation.ks`` kickstart file.

Use ``--help`` to discover more options to ``spin-livemedia``. You can override
the Release field with ``--release``, or ``--repo`` to override what yum repo is
used to provide packages to install to the image. Note that regardless of what
repo you use, it must contain RPMs built by Koji, otherwise you will get an
error. (unless you are building a scratch image). ``--can-fail`` options says
which architectures are optional and can fail without failing the whole build.

Mechanics
---------

The way LiveMedias are created in Koji is fairly straightforward. A chroot is
initialized and populated with the packages and their dependencies from the
``livemedia-build`` package group. Next, the kickstart file is copied into it if
it was provided from local storage. If not, it is checked out into it from an
SCM. It is then modified to use the repo associated with the build tag for the
target specified in the command unless the ``--repo`` option was given. Both the
original and the modified kickstart files are saved as part of the output for
the task for later review. A `livemedia-creator`_ command is executed using the
``mock('--chroot', ...)`` method.

.. note::
    This process runs as root. This produces the desired image which is uploaded
    to ``/mnt/koji/images/<image>/$imageID`` if it is not a scratch image.

.. _caveats-for-livemedias:

Caveats for LiveMedias
----------------------

There are some known caveats with using the spin-livemedia command that users
should be aware of.

%include macros in the kickstart
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A word of caution about kickstart files and the ``%include`` macro.
`livemedia-creator`_ is smart enough to search the current directory of the
submitted kickstart file if it has ``%include`` macros. If the kickstart
specified to koji is from local storage, only that kickstart file will be copied
into the chroot, and this creates a problem if it has ``%include`` macros,
because the other kickstart files it needs will be inaccessible. This issue is
not present when the kickstart file is retrieved from a remote SCM (such as the
fedora-kickstarts git repo), because the entire repository is checked out.
Presumably it will include any other kickstart files the specified one is
including in the same directory. A workaround for the issue would be to use
``ksflatten`` (from pykickstart) on kickstart files with ``%include`` macros
that are going to be submitted to koji from the user's local disk.

Package Groups in the Kickstart File
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Package Groups in the kickstart file cause a problem if the Koji repos do not
define them, which they most likely don't since Koji's comps.xml is based on the
"groups" set up from the CLI. `livemedia-creator`_'s behavior is to ignore
package groups that are not defined in the repo it is using, so this can be
troublesome when creating the image since packages could be left out. There are
a couple possible workarounds:

* do not use package groups in the kickstart file and just specify a huge list
  of packages
* use ``--repo`` and specify a repo that does have a comps.xml that defines
  the groups it uses

Only Include RPMs Built in Koji
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The image building tasks will fail if your image tries to include a package
that was not built in your build system. This is because the package does not
have any origin information stored in Koji's database. The repos defined in the
kickstart will automatically be overridden with the repo of the build tag for
the build target, unless you use the ``--repo`` option. Since only packages
you have built (or include from an external repo) should be there, you should
never have this problem unless you use ``--repo``.

No Signed or Debuginfo RPMs in Koji's Build Tags
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you need signed RPMs or debuginfo RPMs, you will run into trouble because
Koji does not keep those in its build tag repos. The only work around for this
is to create a repo yourself that includes these RPMs and then use ``--repo``.
This will force the image to take RPMs from that repo. Remember, the task will
fail if Koji detects RPMs were installed that were not built in the build
system.

%post Section in Kickstart
^^^^^^^^^^^^^^^^^^^^^^^^^^

While `livemedia-creator`_ does support building on SELinux disabled hosts, you
can run into denials when booting if you create and use new files in the
``%post`` section of your kickstart file. If you do, you should either set the
labels appropriately at the end of the ``%post`` section, or instigate an
autorelabel at boot time.

Troubleshooting
---------------

If your build fails, you will be notified on the command line. In the output
will be a URL to the Koji UI, visit that and click on the red subtask link.
From that page review ``root.log`` and ``livemedia-out.log`` for errors. Often
errors are caused by packages being missing, or malformed kickstart files. The
log files are at the bottom of the page. If problem occurs later during
installation `livemedia-creator`_ will also upload a lot of other logs like
anaconda's, etc. If you're stuck, contact Release Engineering.

Build System Preparation
------------------------

This section assumes you have know-how required to install and configure a new
instance of Koji, and that you have already done so. You can learn how to do so
:doc:`here <server_howto>` if you need to. Please ensure you are using the
latest version of the software and that your database schema is updated as
well. You should also have some familiarity with how `livemedia-creator`_ works.
This section only covers preparation for LiveMedia builds.

Follow this procedure step by step to get things prepared they way they need to
be.

#.  ``koji add-host-to-channel <your-host> livemedia``
        add a builder to the livemedia channel
#.  ``koji grant-permission livemedia <user>``
        grant the permission to build an image type to a user. This step is
        optional since admins have all permissions.
#.  You will need a tag and target to build the images from. The yum repo
    generated for the build tag of the target is what Koji will use to populate
    the LiveMedias with by default. (the alternative is to use the ``--repo``
    option, more on that later)
#.  ``koji add-group <build-tag> livemedia-build``
    add the livemedia-build group
#.  ``koji add-group-pkg <build-tag> livemedia-build <pkg> ...``
        add packages to the livemedia-build group. These package lists vary has
        packages and dependencies change. As of September, 2020 for Fedora 33 the
        needed packages for each image type are:

        * bash, coreutils, glibc-all-langpacks, lorax-lmc-novirt,
          selinux-policy-targeted, shadow-utils, util-linux


Building Disk Images
====================

Disk images are files that represent virtual disks. They have a partition table
and filesystems on them, and are available in a variety of formats: qcow2,
vmdk, ova, Hyper-V, raw, "base" container images, and more.

Getting Started
---------------

What you need:

* a name for your image, and a version number
* what architectures you want
* a Koji build target
* kickstart file
* installation tree
* you may also need yum repositories generated by Rel-Eng if you require signed
  packages be installed into the image

The Koji command to build a disk image is called ``image-build``. The
``image-build`` command uses `ImageFactory`_ and `Oz`_ to start a VM guest and
perform an automated Anaconda installation. Here is a (lengthy) example for
building a disk image.

::

    $ koji image-build --repo 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' --kickstart fedora-server.ks --scratch --distro Fedora-22 --format qcow2 fedora-server-kvm 22 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' x86_64

This example builds a scratch qcow2 disk image using packages from an
additional yum repository. Without this option the yum repo to populate the
build root would be used instead. If this was the first image with the N-V of
fedora-server-kvm-22, then the N-V-R would be fedora-server-kvm-22-1, because
Koji uses an incrementing number for the release if you do not provide one.
Like all Koji commands, use ``--help`` to see more options that are available.

For Docker, Koji only supports Base Images right now using a kickstart file as
described above. In the future it will support layered images, but not before
some Docker requirements are met, and Koji is maintaining a Registry of its
own. This scoping effort is ongoing.

``image-build`` takes a minimum of 5 positional arguments, and 2 options must
be specified. They are reviewed in the list below, with the positional
arguments first.

Name
    the name of the image, without versioning information. Examples could be
    ``fedora-server`` or ``fedora-workstation``.

Version
    an arbitrary version string. For Fedora images, this usually matches the
    release version, such as 22 or 23.

Build Target
    just like RPM builds Koji must be told which target to use. This controls
    what tag the image will be tagged into, and what packages are available to
    install into the image.

Installation Tree URL
    this is a URL to a location you can install an operating system from. It is
    the same place you would direct a PXE-booted system to go. In 99% of cases
    this location is provided by Release Engineering. It should have an
    "isolinux" subdirectory and yum metadata somewhere within.

Architecture
    only x86_64 or i386 is supported, and you can specify both on the command
    line. This will cause two subtasks to be run, allowing you to build for
    both arches in parallel. If either fail, the whole build will fail.

Kickstart File
    this is a recipe file that performs drives the construction of the image.
    Pass in the path to a kickstart file, which must be flattened.

Kickstart URL
    in a non-scratch build, you'll need this too. For more details, see the
    Getting your Kickstart to Koji section.

Distro
    a string that indicates what OS is being built. These always follow the
    convention of "Fedora-X", where X is the release number.

Since this command can get very long, a configuration file can drive the task
as well, using the ``--config`` option. It accepts a path to a configuration
file written in the Python ConfigParser format (like a Windows .ini). The
options are all named the same with one caveat, see below. Here's what one
could look like:

::

    [image-build]
    name = fedora-server-docker
    version = 22
    target = f22-candidate
    install_tree = https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
    arches = x86_64

    format = qcow2,rhevm-ova,vsphere-ova
    distro = Fedora-22
    repo = https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
    disk_size = 20

    ksversion = DEVEL
    kickstart = fedora-22-server-docker.ks
    ksurl = git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7
    specfile = git://git.fedorahosted.org/git/spin-kickstarts.git?spec_templates/fedora22#68c40eb7

A few notes on the syntax:

* it allows for comments too, the lines start with a hash (#)
* options on the command line that can be used multiple times can accept
  values here as comma-separated strings
* options with a hyphen need to use an underscore instead. ``--disk-size`` for
  example would be ``disk_size`` in the config file

OVA Features
------------

If you're building OVAs, either for RHEVM or vSphere, you can specify OVA
options with a special section in the configuration file. It looks something
like this:

::

    [ova-options]
    vsphere_product_version=22
    rhevm_description=Fedora Cloud 22
    vsphere_product_vendor_name=Fedora Project
    ovf_memory_mb=6144
    rhevm_default_display_type=1
    vsphere_product_name=Fedora Cloud 22
    ovf_cpu_count=4
    rhevm_os_descriptor=Fedora-22

or this:

::

    [ova-options]
    vsphere_ova_format = vagrant-virtualbox
    rhevm_ova_format = vagrant-libvirt
    vagrant_sync_directory = /home/vagrant/sync

The second one is actually the secret sauce for generating an image for use in
Vagrant. At this time, you would need rename the image file extension from .ova
to .box, but otherwise this should work fine.

Kickstart Preparation
---------------------

Kickstarts for the image-build command have some specific requirements which
are covered in this section.

Required Kickstart Arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Anaconda of course requires many commands to be defined in the kickstart file.
If you're starting from scratch you should review the reference linked above,
or use an existing kickstart file in the spin-kickstarts git repo. It is
critically important that the installation be completely automated, if
Anaconda has to prompt for input for any reason, the build will fail because
you cannot send input to the guest. Some of the kickstart commands are optional
to Anaconda, but are required in Koji for your build to succeed. Here's the
list and the reasons why.

zerombr
    You must tell Anaconda to wipe out the MBR in the virtual block device, if
    you don't Anaconda will ask you.

clearpart --all --initlabel
    Anaconda has to be told to wipe out all data on the virtual block device we
    install on otherwise it will ask for confirmation to do so. Since it is
    blank anyway this is harmless.

reboot
    When the installation completes, the guest is rebooted. `ImageFactory`_ is
    specifically looking for this behavior to conclude the installation
    completed. Anaconda's default behavior is to wait for a key press to reboot
    the system, but this is impossible from outside of Koji.

locking the root account
    You have to lock the root account (rootpw --lock) or create a non-root user
    (user), otherwise Anaconda will prompt for one.

Do not use the url command
    The repo commands are overridden by Koji to point to internal Koji repos,
    or what you specified on the command line with ``--repo``, it does not
    override the url command if you provided it. Anaconda has a behavior where
    it will prefer packages from the repositories given with the url command
    over those with the repo command, and this is generally not what you want.
    If Koji sees an RPM was installed that was not built in the system, it will
    fail the build.

Recommended Kickstart Arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Often you want a ``%post`` section in your kickstart to perform
post-installation configuration steps. Review that section of the reference
and note that you can specify ``--log`` and ``--interpreter``. Both of these
are recommended (but not required) to assist with the development and debugging
process. Here are some other recommendations:

* You probably want the network to use dhcp, sshd to be started, and port 22
  opened in the firewall to allow access as well.
* If you're building an image that will be shipped with a product, SELinux
  should be enabled.
* Images that will be used in cloud deployments like OpenStack or EC2 should
  have ``cloud-init`` in the package list.
* It is discouraged to have root passwords in plaintext in your kickstart file.
* If your %post section is written in bash, consider setting -x.
* For images that have multiple partitions, use the ``--asprimary`` option for
  the part command that defines the root file system. This will ensure it is
  the first partition on the image, which is a requirement in some cloud
  environments like EC2.

Troubleshooting
---------------

If your disk image build fails, follow the link in the command line output
that takes you to the task page in the Koji web UI. Click on the failed
createImage subtask in red. On that page review the screenshot.ppm file if it
was provided, or oz.log. Most failures are from Anaconda rejecting a malformed
kickstart file, which will be indicated in the screenshot. Your installation
must be completely automatic, there can be no interactivity at all, otherwise
Anaconda will sit there indefinitely until Koji (actually ImageFactory) kills
the task.

It is very easy to write a kickstart file with bugs or that results in a system
that does not boot. This section will present a series of questions to ask
yourself and examples to help diagnose where the problem lies. Once you know
that, it should be easier to understand what you can do to inspect further.

There are 4 steps in the process:

#.  create a guest
#.  perform an automated installation in the guest
#.  boot the guest and extract the list of installed RPMs
#.  upload and archive the disk image of the guest

Is it a problem with guest creation?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There have been unusual cases where libvirt, ImageFactory, or Oz was
misconfigured and guests could not be started properly. A misconfiguration with
Puppet or whatever Fedora Infrastructure is up to can cause this. So far the
errors have been clear in the task output, look either in the results string
or oz.log. The bad news is that in this case you really can only inform Rel-Eng
about the issue and wait for a resolution. The good news is these cases are
very rare.

Did the installation fail?
^^^^^^^^^^^^^^^^^^^^^^^^^^

The Anaconda installation can fail for many reasons: missing packages, network
problems, or syntax errors in %post. Tasks will also fail if Anaconda prompts
for input for any reason. If Koji detects a lack of disk activity in the guest
for more than 5 minutes, it will fail the build and tear down the guest.
Looking in oz.log may have the answer: dracut, anaconda, and yum logs are all
printed there.

These sorts of failures often have a screenshot taken and saved with the task
output called screenshot.ppm. Viewing this will usually tell you what Anaconda
is complaining about if the installer detected an issue or prompted for input.
The string in the results output that says "No disk activity in 300 seconds,
failing." This almost always means Anaconda hit an issue and either gave up or
waited.

If Anaconda claims it is missing packages, confirm they exist in the repos you
are using with ``--repo``, if you are using that option. If you are not,
confirm the builds you expect are in the tag inheritance for the target you are
running. This is a lot like checking whether an RPM will build against the
right libraries, except we're building an image instead.

If you get the rare Anaconda dialog box that says something like "An unexpected
error occurred", try using the ``text`` command in kickstart, which will have
Anaconda boot in text mode. Sometimes the Python traceback (or whatever the
error condition is) will be printed there. I have also seen cases where
text-mode yields a black screen, but booting in graphical mode (the default)
does produce a useful dialog box. Issues like this stem from syntax errors in
the kickstart file, or bugs in pykickstart itself. If you think it is a
pykickstart bug, then someone in Rel-Eng needs to update pykickstart on the
builders.

Did the guest boot?
^^^^^^^^^^^^^^^^^^^

Koji waits 5 minutes for a guest to boot in this step. It unfortunately does
not give a lot of insight to why a guest may not boot, so these are a tougher
class of issues to work through. You can usually answer this question by
looking in results string. If you see "Timed out waiting for guest to boot",
then this is your problem. You can also confirm this in oz.log.

For now, the best way to investigate an issue like this is to drive a guest
installation locally using something like Gnome's Virtual Machine Manager
(VMM). The steps to perform are:

* Select a Network Install
* For the Operating System Install URL use the same one you gave to Koji. It
  will be something like
  https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/
* Set the Kickstart URL to where your kickstart file is. You may need to make
  it available over http.
* Bump the memory to 2048M for good measure
* Launch the guest and let it complete installation
* Open a VNC session and watch what happens when the guest attempts to boot.

If the console is not providing enough information, we have to get more
creative. Anaconda supports starting an SSH daemon while the installation is
happening with the sshpw command in kickstart. Set that and comment out the
reboot command. This will let the installation complete locally and wait for a
keystroke to reboot the guest. At this point you should be able to ssh in and
inspect the environment to figure out what is going on. You should also
consider making use of the ``--log`` option to %post so that output from the
script is saved somewhere.

Another option would be to scp logs and other files off of the guest as part of
the ``%post`` script.

Other Guest Misconfigurations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If the guest boots but you're having problems accessing it I'd suggest
following same procedure as when the guest fails to boot. This could be a
result of firewall misconfigurations or SSH not being available for some
reason. Usually in this case the build is succeeding in Koji, but there's
something still fundamentally broken in the image. If the issue is something
you can investigate while the guest is online (you can log in), then I'd
suggest importing it locally using the libvirt.xml and the disk image provided
in Koji's task output.

You can also do investigative work in an offline mode by mounting the image
locally or using something like libguestfs to poke around without starting the
guest. The fast, dirty way to do it is by mounting it. This can often pollute
your guest environment. Here's how to do it:

* Download the image from Koji
* If the image format is not raw, you have to convert it first with qemu-img.
  Something like:
  ::

        $ qemu-img convert -O raw <image-file> <output-file></pre>

* Now mount it up using loopback devices. (as root) If your image has multiple
  partitions in it, you may need to pass in a different mapped loopback device
  like ``loop0p2``. Whichever one you think is the root partition or has the
  issue you're trying to fix.
  ::

        # kpartx -av <raw-image>
        # mount -o loop /dev/mapper/loop0p1 /mnt/my_directory

Hopefully at this point you figure out the issue. To tear down the image you'll
run commands as root like so:

::

    # umount /mnt/my_directory
    # dmsetup remove loop0p1
    # losetup -d /dev/loop0

Again, if you used different loopback devices, substitute those in to the
dmsetup and losetup commands.

Build System Preparation
------------------------

Follow this guide if you're a Koji admin and would like to enable image
building or want to set up some testing before enabling the integration.

When moving to ImageFactory to do image builds Koji lost the ability to easily
reproduce the build environment for images the way we do for RPMs using Mock.
This section will document how to set up an image builder for Koji. This is a
lengthy task, it will take first-timers about a week to have a useful instance,
and it is painful because it requires a bare metal system be provisioned since
`ImageFactory`_ provisions VMs to build the image. There is a significant
performance penalty for using nested virtualization.

Follow the steps below to set up your builder. 

.. note::
    You do not have to stand up a complete Koji instance to test the way Koji
    builds images. However, if you want to test image builds with an accurate
    representation of how Koji does it, or you want to test code changes
    related to image builds in Koji, you should follow all the steps below.

ImageFactory/Oz Preparation
^^^^^^^^^^^^^^^^^^^^^^^^^^^

#.  Provision a system with at least 4G of memory with the current release of
    RHEL 6 or later. Sometimes builders lag behind a month or two before taking
    in updates, but this will still get you pretty close to where you want to
    be. For Fedora, use the latest release.
#.  Install the following packages to the builder.
    #.  oz
    #.  imagefactory
    #.  imagefactory-plugins-TinMan
    #.  imagefactory-plugins-vSphere
    #.  imagefactory-plugins-ovfcommon
    #.  imagefactory-plugins-docker
    #.  imagefactory-plugins
    #.  imagefactory-plugins-OVA
    #.  imagefactory-plugins-RHEVM
    #.  python-psphere => 0.5
    #.  VMDKStream => 0.2
    #.  pykickstart
#.  Edit ``/etc/kojid/kojid.conf``, and set an second value, eg: 7200 for
    ``oz_install_timeout``. It's a timeout waiting guest installing. Default
    value is 0, that means oz will use its default value. Since ``oz-0.16.0``,
    it can be configured in ``/etc/oz/oz.cfg`` as ``install`` in ``[timeouts]``
    section.
#.  Edit ``/etc/oz/oz.cfg``, and set the memory value in the ``[libvirt]``
    section to at least 2048. Set ``safe_generation`` under ``[icicle]`` to yes.
#.  Run: ``mkdir -p ~root/.psphere/templates``, and then copy the following
    code into ``~root/.psphere/config.yaml``. Do not worry about the server,
    username, and password credentials; they are not used anywhere.

    ::

        general:
          server: 10.16.120.224
          username: Administrator
          password: whatever
          template_dir: ~/.psphere/templates/
        logging:
          destination: ~/.psphere/psphere.log
          level: DEBUG # DEBUG, INFO, etc

Start up the services and ImageFactory/Oz should be ready to go. You should
read more about `how to use Oz`_ and `how to use ImageFactory`_. If you want
to try calling `ImageFactory`_ as if from a Koji Builder (but not set up a
whole Koji instance), you can use the code below to emulate that. If you want
to test the Koji integration with a full Koji instance, proceed to the next
section instead.

::

    #!/usr/bin/python -tt

    import logging
    import os.path
    import random
    import sys

    from imgfac.BuildDispatcher import BuildDispatcher
    from imgfac.PluginManager import PluginManager
    from imgfac.ReservationManager import ReservationManager
    plugin_mgr = PluginManager('/etc/imagefactory/plugins.d')
    plugin_mgr.load()
    from imgfac.ApplicationConfiguration import ApplicationConfiguration

    # logging
    handler = logging.StreamHandler(sys.stdout)
    tlog = logging.getLogger()
    tlog.setLevel(logging.DEBUG)
    tlog.addHandler(handler)

    # configuration
    ks = open('oztest.ks').read()
    workdir = '/tmp/koji/test'
    config =  {
        #Oz specific
        'oz_data_dir': os.path.join(workdir, 'oz_data'),
        'oz_screenshot_dir': os.path.join(workdir, 'oz_screenshots'),
        #IF specific
        'imgdir': os.path.join(workdir, 'scratch_images'),
        'tmpdir': os.path.join(workdir, 'oz-tmp'),
        'verbose' : True,
        'timeout': 3600,
        'output': 'log',
        'raw': False,
        'debug': True,
        'image_manager': 'file',
        'plugins': '/etc/imagefactory/plugins.d',
        'tdl_require_root_pw': False,
        'image_manager_args': {
            'storage_path': os.path.join(workdir, 'output_image')},
    }
    random.seed() # necessary to ensure a unique mac address
    rm = ReservationManager()
    rm._listen_port = random.randint(rm.MIN_PORT, rm.MAX_PORT)
    ApplicationConfiguration(configuration=config)
    params = {'install_script': ks}
    template = """<template>
        <name>test-appliance</name>
            <os>
                <name>Fedora</name>
                <version>22</version>
                <arch>x86_64</arch>
                <install type='url'>
                    <url>https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/</url>
                </install>
                <icicle>
                    <extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>
                </icicle>
            </os>
        <description>test-appliance OS</description>
        <disk>
            <size>16G</size>
        </disk>
    </template>
    """
    bd = BuildDispatcher()

    # build the image
    base = bd.builder_for_base_image(template, parameters=params)
    base.base_thread.join()
    tlog.removeHandler(handler)

This script is run as root with no arguments. It uses ``oztest.ks`` in your
local directory as the kickstart you want to try to use. The URL in the XML
template is where the RPM packages will be installed from, and what the guest
will be booted with.

Koji Preparation
^^^^^^^^^^^^^^^^

#.  :doc:`Install Koji <server_howto>` if you need it. This will easily be the
    most time-consuming part of the process; my first time took 3 days to get
    it working properly. Follow the guide closely, and go with the SSL
    authentication method. SSL is a lot easier to set up locally. You will need
    to install every Koji component (except koji-vmd) on the same system.
    Proceed to the next step after you've had a successful Kojira repository
    generated.
#.  At this point, you have a system that should be ready to build images. We
    just have to do some Koji configuration so that your instance is pulling
    content from Koji. Replace the base tag names with whatever fits your
    conventions.

        #.  Add tags, tag inheritance, new pkg (with pkg owner), new external repo, and regen the repo

            ::

                koji add-tag fedora22
                koji add-tag jay-fedora22
                koji add-tag jay-fedora22-override --parent jay-1-fedora22
                koji add-tag jay-fedora22-build --arches x86_64 --parent jay-fedora22-override
                koji add-tag jay-fedora22-candidate --parent jay-fedora22
                koji add-tag-inheritance --priority 40 jay-fedora22-build fedora22
                koji add-pkg --owner kojiadmin jay-fedora22 fedora-server-ec2 fedora-server-kvm
                koji add-external-repo -t fedora::20 fedora22 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'
                koji add-target jay-fedora22-candidate jay-fedora22-build
                koji regen-repo jay-fedora22-build
        #.  Grab a kickstart file from an image task in Koji that relates to what you want to test.
        #.  Finally, kick off a build!

            ::

                koji image-build fedora-server-ec2 22 --distro Fedora-22 jay-fedora22-candidate --kickstart fedora-server-starter-ec2.ks 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'


Building Appliances
===================

This section is here for the sake of legacy. Unless you are trying to build
ARM images, you should use the image-build command described in the previous
section.

.. note::
    The spin-appliance command, described herein, is deprecated.

Getting Started
---------------

Here's what you need before proceeding:

* a name for your Appliance, and a version number
* a Koji build target
* what architectures you want
* a flattened kickstart file
* you may also need yum repositories generated by release engineering if you
  require signed packages be installed into the appliance
* the appliance permission in Koji

The Koji command that creates an Appliance is called ``spin-appliance``. It
calls out to appliance-creator in a mock chroot to construct the Appliance.
To run an Appliance build, use the spin-appliance command like so:

::

    $ koji spin-appliance --release 4 fedora-workstation 23 f23-build fedora-workstation.ks

In this example an Appliance will be created with the N-V-R of
``fedora-workstation-23-4``. If ``--release`` was not included an incrementing
value would be computed by Koji instead. spin-appliance takes a minimum of 5
arguments:

Name
    the name of the image, without versioning information. Examples could be
    ``fedora`` or ``fedora-workstation``.

Version
    an arbitrary version string. For Fedora images, this usually matches the
    release version, such as 21, 22, or 23.

Build Target
    just like RPM builds Koji must be told which target to use. This controls
    what tag the image will be tagged into, and what packages are available to
    install into the image.

Architecture
    only arm, x86_64, or i386 are supported

Kickstart File
    this is a recipe file that performs drives the construction of the image.

Use ``--help`` to discover more options to spin-appliance. You can override the
Release field with ``--release``, or ``--repo`` to override what yum repo is
used to provide packages to install to the image. Note that regardless of what
repo you use, it must contain RPMs built by Koji, otherwise you will get an
error. (unless you are building a scratch image)

Mechanics
---------

The way Appliances are created in Koji is the same as :ref:`building-livemedia`.

Caveats for Appliances
----------------------

Caveats for using ``spin-appliance`` are the same as using ``spin-livemedia`` to
:ref:`caveats-for-livemedias`.

Troubleshooting
---------------

If your build fails, you will be notified on the command line. In the output
will be a URL to the Koji UI, visit that and click on the red subtask link.
From that page review ``root.log`` and ``appliance.log`` for errors. Often
errors are caused by packages being missing, or malformed kickstart files. The
log files are at the bottom of the page. If you're stuck, contact Release
Engineering.

Build System Preparation
------------------------

This section assumes you have know-how required to install and configure a new
instance of Koji, and that you have already done so. You can learn how to do so
:doc:`here <server_howto>` if you need to. Please ensure you are using the
latest version of the software and that your database schema is updated as
well. You should also have some familiarity with how `appliance-creator`_
works. This section only covers preparation for Appliance builds.

Follow this procedure step by step to get things prepared they way they need
to be.

#.  Add a builder to the appliance channel
        ::

            koji add-host-to-channel <your-host> appliance

#.  Grant the permission to build an appliance to a user. This step is optional
    since admins have all permissions.

        ::

            koji grant-permission appliance <user>

#.  You will need a tag and target to build the images from. The yum repo
    generated for the build tag of the target is what Koji will use to populate
    the Appliances with by default. (the alternative is to use the ``--repo``
    option, more on that later)

#.  Add the appliance-build group

        ::

            koji add-group <build-tag> appliance-build``

#.  Add packages to the appliance-build group. These package lists vary has
    packages and dependencies change. As of October, 2015 for Fedora 24 the
    needed packages for appliances:

        * appliance-tools, bash, coreutils, grub, parted, perl, policycoreutils,
          selinux-policy, shadow-utils, sssd-client

        ::

            koji add-group-pkg <build-tag> appliance-build <pkg> ...

.. _Koji project website: https://fedorahosted.org/koji/wiki
.. _kickstart:
    https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst
.. _Anaconda Kickstart Guide:
    https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst
.. _lorax: https://github.com/weldr/lorax
.. _livemedia-creator: https://weldr.io/lorax/livemedia-creator.html
.. _ImageFactory: http://imgfac.org/
.. _Oz: https://github.com/clalancette/oz
.. _how to use Oz: https://github.com/clalancette/oz/wiki
.. _how to use ImageFactory: http://imgfac.org/documentation/
.. _appliance-creator: https://fedoraproject.org/wiki/Features/ApplianceTools