From 06bba56d49f3b948f0476cdf538f4441c7e355d1 Mon Sep 17 00:00:00 2001
From: Paul W. Frields <stickster@gmail.com>
Date: Oct 28 2019 15:38:26 +0000
Subject: Add history, background, requirements


---

diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc
index fd6c7ec..47f8c3e 100644
--- a/modules/ROOT/nav.adoc
+++ b/modules/ROOT/nav.adoc
@@ -42,5 +42,5 @@
 * xref:references.adoc[References]
 * xref:archive.adoc[Archive]
 ** xref:architecture/consuming/dnf-behavior.adoc[DNF Behavior]
-
-
+* xref:history.adoc[History and Background]
+** xref:requirements.adoc[Requirements and Use Cases]
diff --git a/modules/ROOT/pages/history.adoc b/modules/ROOT/pages/history.adoc
new file mode 100644
index 0000000..df70203
--- /dev/null
+++ b/modules/ROOT/pages/history.adoc
@@ -0,0 +1,117 @@
+= Modularity: History and Background
+
+Fedora Modularity is a new technology that tries to solve important
+and complex user problems. From time to time other solutions are
+suggested to solve these problems in different ways. Often those
+solutions fail to address one or more intended use cases. These pages
+to enumerate these cases in detail so as to serve as a common
+reference point for the ongoing discussions.
+
+It is important to note these are goals. There are numerous places
+where the implementation of Modularity at the time of this writing is
+not yet fully adherent to them.
+
+== It’s all about the apps!
+
+Very few people install a Linux distribution for its own
+sake. Ultimately, the goal is to “scratch a particular itch” that the
+user is experiencing. The solutions may take many forms, but
+ultimately this user wants to deploy some software that solves a
+problem for them.
+
+This leads us to a classic problem that Linux distributions have faced:
+the “Too Fast/Too Slow” problem. Linux distributions are traditionally
+quite monolithic. The package collections they ship are generally
+self-consistent, providing generally whatever the latest stable major
+release of the software at the time of the distribution release. As the
+release ages, it will receive bugfixes and enhancements, but usually
+will remain on the same major version.
+
+This is excellent for the maintainers of the distribution, because it
+allows them to test that everything works together as a cohesive whole.
+It means that there’s one authoritative version to align to.
+
+Users, on the other hand, are most concerned about solving their
+problem. It matters less to them that the distribution is cohesive and
+more that the tools they need are available to them.
+
+The “Too Fast/Too Slow” problem is basically this: users want a solid,
+stable, reliable, _unchanging_ system. They want it to stay that way for
+the life of their application. However, they also want their application
+to run using the set of dependencies it was designed for. If that
+doesn’t happen to be the same version (newer or older) as the one
+selected for the monolithic distribution, the user will now have to
+resort to alternative means to get up and running. This may be as simple
+as bundling a dependency or as drastic as selecting an entirely
+different distribution that better fits their specific need.
+
+== A little background
+
+One of the precursors to Fedora Modularity was
+https://www.softwarecollections.org/[Software Collections] (SCLs). This
+was a first try at solving the Too Fast/Too Slow” problem in the
+Fedora/Red Hat ecosystem. provides two basic advantages: _Parallel
+Availability_ and _Parallel Installability_.
+
+_Parallel Availability_ means that more than one major release of a
+popular software project is available for installation. For example, the
+“Developer Toolset” SCLs provide access to newer versions of GCC and its
+related toolchain for building software. There are Python and Ruby SCLs
+that provide assorted runtimes for those languages and so on.
+
+_Parallel Installability_ means that more than one major release of a
+software project can be installed on the same userspace.
+
+A few years back, the Product Management team inside Red Hat performed a
+large-scale survey of customers and potential customers about the user
+experience of Red Hat Enterprise Linux. In particular, they asked about
+their level of satisfaction with the software available from the
+enterprise distribution and their opinion on these Software Collections.
+
+Perhaps unsurprisingly, the overwhelming majority of respondents were
+thrilled to have supported versions of software beyond what had shipped
+with the base operating system. What the survey team did come away with
+that was an epiphany was that the respondents generally did not care
+about the parallel installability of the SCLs. For the most part, they
+maintained individual userspaces (using bare metal, traditional
+virtualization or containers) for each of the applications they cared
+about.
+
+The most common problem reported for Software Collections was that using
+them required changes to the applications they wanted to run. SCLs
+install to a separate filesystem location from more traditional RPMs and
+applications that rely on them need to know where to look for them. (In
+SCL parlance, this is called “activating” the collection.)
+
+The consequence of this relocation on disk is that users were unable to
+take existing applications (either FOSS or proprietary) and simply use
+them. Instead, they had to modify the projects to first activate the
+collections. This was a consistent pain point.
+
+Given this feedback, Red Hat came to the conclusion that parallel
+installability, while nice to have, was not a critical user requirement.
+Instead, the focus would be on the parallel _availability_. By dropping
+this requirement, it became possible to create a solution that allowed
+the different versions to be swapped in and take over the standard
+locations on the disk.
+
+== Meanwhile in Fedora
+
+Of course, it’s not just Red Hat — people in Fedora are also concerned
+with solving this Too Fast / Too Slow problem for our users. Efforts
+around this kicked off in seriousness with the
+https://fedoramagazine.org/fedora-present-and-future-a-fedora-next-2014-update-part-ii-whats-happening/[Fedora.next
+initiative] and Fedora Project Leader Matthew Miller’s
+“link:https://lwn.net/Articles/563395/[Rings]” talk at the first Flock
+conference in 2013.
+
+This led to the proposal and approval by the Fedora Council of the
+link:https://fedoraproject.org/wiki/Objectives/Fedora_Modularization%2C_Prototype_Phase[Modularity
+Prototype Fedora Objective] and its follow-up
+link:https://fedoraproject.org/wiki/Objectives/Fedora_Modularization_%E2%80%94_The_Release[Modularity
+Release Fedora Objective].
+
+== Requirements and use cases
+
+For more information on requirements and use cases, read on to the
+xref:requirements.adoc[Modularity requirements and use cases] page.
diff --git a/modules/ROOT/pages/requirements.adoc b/modules/ROOT/pages/requirements.adoc
new file mode 100644
index 0000000..7130ff4
--- /dev/null
+++ b/modules/ROOT/pages/requirements.adoc
@@ -0,0 +1,311 @@
+= Modularity: Requirements and use cases
+
+If you haven't already read it, you should check out the first part of this essay on the xref:history.adoc[history and background of modularity].
+
+== Critical use cases for consumers
+
+First and foremost, our primary driving goal is to make it easy for our
+users to understand and interact with alternative software versions. In
+any instance where choosing between the packager experience and the user
+experience is in conflict, we elect to improve things for the user.
+
+=== Standard Locations
+
+In order to make deployment of users’ applications simpler, we need to
+make sure that software can be installed into the common, expected
+locations on the system. This includes (but is not limited to):
+
+* Libraries must be installed to `/usr/lib[64]`.
+* Headers must be installed to `/usr/include`.
+* Executables must be installed to a location in the default system
+`$PATH`
+* Other -devel functionality such as pkgconfig files must be installed
+in their standard lookup locations.
+* Installed services may own a well-known DBUS address.
+* Services may own the appropriate standard TCP/UDP ports or local
+socket paths.
+
+_Requirement_: Installation must occur in the same locations as
+traditional RPM software delivery.
+
+=== Don’t break the app!
+
+It is very common for Fedora to update to the latest major version of
+packages at each new semiannual release. This ensures that Fedora
+remains at the leading edge of software development, but it can wreak
+havoc on anyone trying to maintain a deployment on Fedora. If they are
+running an app that is built for PostgreSQL 9.6 and Fedora switches to
+carrying PostgreSQL 10 in the next major release, upgrading to that
+release may break their app (possibly in ways undetectable by the
+upgrade process).
+
+However, staying on an old version of Fedora forever has its own
+problems. Not least of these is the problem of security updates: Once a
+release has been out for about 13 months, it stops receiving errata.
+Moreover, new releases of the Fedora platform may have other useful
+enhancements (better security defaults, increased performance thanks to
+compiler improvements, etc.).
+
+_Requirement_: We need to allow users to “lock” themselves onto certain
+dependencies as long as the packager is maintaining them. These
+dependencies must continue to receive updates.
+
+_Requirement_: There must be appropriate and helpful UX for dealing with
+when those dependencies go EOL.
+
+=== Support the developers
+
+Developers often want to build their applications using the
+latest-and-greatest version of their dependencies. However, that may not
+have been released until after the most recent Fedora release. In
+non-Modular Fedora, that means waiting up to six months to be able to
+work on it there.
+
+_Requirement_: It must be possible to gain access to newer software than
+was available at the Fedora release GA.
+
+Additionally, Dev/Ops people are rapidly switching to a new paradigm of
+development and deployment (containers) to solve the above issue.
+However, most containers today are retrieved from public repositories.
+The public repositories are generally user-managed and have not been
+verified and validated for security.
+
+_Requirement_: Provide a mechanism for building *trusted* container base
+and application images with content alternatives.
+
+=== Keep it updated
+
+It’s not enough that other versions of software are available to
+install. They also need to be kept up to date with bug fixes and
+security updates. In non-Modular Fedora, users had the ability to force
+DNF to lock to a specific RPM NEVRA, but they wouldn’t get updates from
+it.
+
+_Requirement_: Alternative software must receive be able to recieve and
+apply updates.
+
+=== Make it discoverable
+
+Having alternative versions available is important but not sufficient.
+It is also necessary for users to be able to locate these alternatives.
+Some of our early explorations into this area failed this ease-of-use
+test because they require the user to have knowledge of external sites
+and then to search those sites for what they think they want.
+
+_Requirement_: Users must be able to discover what alternative software
+versions are available with tools that are shipped with the OS by
+default. Ideally, these should be the same tools that they are already
+comfortable with.
+
+=== Don’t break existing package management workflows
+
+Users are slow to adapt to changes in the way they need to behave.
+Requiring them to learn a new set of commands to interact with their
+system will likely result in frustration and possibly exodus to other
+distributions.
+
+_Requirement_: It must remain possible to continue to operate with only
+the package management commands used in traditional Fedora. We may
+provide additional commands to support new functionality, but we must
+not break the previous workflow.
+
+_Requirement_: Existing automation tools such as anaconda’s kickstart
+and Ansible must continue to work.
+
+== Critical use-cases for packagers
+
+=== Dependencies
+
+Because very little software today is wholly self-contained, it must be
+possible for Modules to depend on each other.
+
+_Requirement_: There must be a mechanism for packagers to explicitly
+list dependencies on other software, including alternative versions.
+This mechanism must support both build-time and run-time dependencies.
+
+=== Alternative dependencies
+
+Some software is very restrictive about which dependencies it can work
+with. Other software may work with several different major releases of a
+dependency. For example, a user may ship two Ruby-based web
+applications, one which is capable of running on Ruby 2.5 and the other
+that can run on either Ruby 2.5 or Ruby 2.6. In non-modular Fedora, only
+one version of Ruby would be available. If the system version was 2.5,
+then both applications could run fine. But if in the next release of
+Fedora the Ruby 2.6 release becomes the system copy, one of those
+applications will have to be dropped (or patched) to work with it.
+
+_Requirement_: It must be possible to build software that can be run
+against multiple versions of its dependencies.
+
+_Requirement_: The packaging process for creating software that supports
+multiple versions of their dependencies must not be significantly more
+difficult than packaging for a single dependency.
+
+As more and more things become modules, there is concern that such
+things will grow into an unbounded matrix. For this, we need to
+establish policies on when the use of alternative dependencies is
+preferable or when it is better to constrain it to a single version or
+small set.
+
+_Requirement_: Packaging guidelines need to provide advice on when to
+use multiple alternative dependencies or to select a single one.
+
+=== Managing private dependencies
+
+When a person decides that they want Fedora to carry a particular
+package and decides to do the work to accomplish this, it is not
+uncommon to discover that the package they care about has additional
+dependencies that are not yet packaged in Fedora. Traditionally, this
+has meant that the packager has needed to package up those dependencies
+and then continue to maintain them for anyone who may be using them for
+other purposes. This can sometimes be a significant investment in time
+and energy, all to support a package they don’t necessarily care about
+except for how it supports the primary package.
+
+==== Build-time Dependencies
+
+Sometimes, a package is needed only to build the software and is not
+required at run-time. In such cases, Modularity should offer the ability
+to keep those build-time dependencies entirely private and not exposed
+to the Fedora Package Collection at large.
+
+_Requirement_: Build-time only dependencies for an alternative version
+may be excluded from the installable output artifacts. These excluded
+artifacts may be preserved by the build-system for other purposes.
+
+_Requirement_: All sources used for generating alternative versions,
+regardless of final visibility, must be available to the community for
+purposes of modification and reproducibility.
+
+==== Defining the public API
+
+Similarly, there are times when an application the packager cares about
+depends on another package that is required at runtime, but sufficiently
+complex that the packager would not want to maintain it for general use.
+(For example, an application that links to a complicated library but
+only uses a few functions.)
+
+In this case, we want there to be a standard mechanism for the packager
+to be able to indicate that some of the output artifacts are not
+supported for use outside this module. If they are needed by others,
+they should package it themselves and/or help maintain it in a shared
+place.
+
+_Requirement_: Packagers must be able to encode whether their output
+artifacts are intended for use by other projects or if they are
+effectively private to the alternative version. Packagers must also have
+a way of finding this information out so they understand what they can
+and cannot rely on as a dependency.
+
+=== Use-case-based installation
+
+Since the earliest days of Linux, the “package” has been the fundamental
+unit of installable software. If you want to have some functionality on
+the system, you need to learn the name of the individual packages that
+provide that functionality (not all of which are named obviously). As we
+build modules, one of the goals is to try to focus installation around
+use-cases rather than around upstream projects. A big piece of this is
+that we want to have a way to install a subset of a module that supports
+specific use-cases. A common example being “server” and “client” cases.
+
+_Requirement_: It must be possible to install a subset of artifacts from
+an alternative version. These installation groups should be easily
+discoverable.
+
+_Recommendation_: Installation groups should be named based on the
+use-case they are intended to solve. This will provide a better user
+experience.
+
+=== Lifecycle isolation
+
+Another of the major issues faced by Fedora is maintaining a release
+schedule when all of the components within it follow vastly differing
+schedules. There are two main aspects to this problem:
+
+* A major version of a popular piece of software is released just after
+a Fedora release, so it doesn’t land in Fedora for six months.
+* Some software does frequent major revisions (Django, Node.js, etc.)
+and swapping them out every six months for the latest one means that
+dependent projects are constantly needing to adapt to the new breakage
+or find alternative mechanisms for retaining the older, working version
+* Some software does not handle multiple-version upgrades (Nextcloud,
+for example). Attempting to go from version 15 to verison 19 requires
+first upgrading through 16, 17, and 18.
+
+_Requirement_: It must be possible for new alternative versions of
+software to become available to the Fedora Package Collection between
+release dates.
+
+_Requirement_: It must be possible for alternative versions of software
+to go end-of-life during a Fedora release. This does not mean that the
+software must disappear from the repositories, merely that an assertion
+exists somewhere that after a certain date, the package will not receive
+updates.
+
+_Requirement_: For alternative versions whose lifecycle will continue
+through at least part of the next Fedora release, it must be possible to
+upgrade from one release to the next and remain with the
+fully-compatible version.
+
+=== Third-party additions
+
+Some third-party add-on repositories (particularly EPEL) have been
+limited in the past by relying on the system copies of packages in the
+base distribution of the release. In the particular case of EPEL, little
+can be done to upgrade these system copies. In order to be able to
+package much of the available FOSS software out there, it may be
+necessary to override some of the content shipped in the base system
+with packages known to work properly.
+
+_Requirement_: It must be possible for third party repositories to
+create alternative versions that override base distribution content at
+the user’s explicit choice.
+
+_Requirement_: It must be possible for third party repositories to
+create alternative versions of software that exist in the base
+distribution.
+
+=== Reduce duplication in packaging work
+
+There is plenty of software out in the wild that maintains compatibility
+over time and is therefore useful to carry in multiple releases of
+Fedora. With traditional packaging, this means carrying and building
+separate branches of the packages for each release of Fedora. In the
+case of software “stacks” which are tightly bound, this means also
+manually building each of its dependencies in each release of Fedora.
+
+_Requirement_: It must be possible to build multiple component software
+packages in the same build process.
+
+_Requirement_: It must be possible for the packager to specify the order
+in which packages must be built (and to indicate which ones can be built
+in parallel).
+
+_Requirement_: It must be possible to be build for all supported
+platforms using the same specification and with a single build command.
+
+== Non-Goals
+
+=== Parallel installability
+
+As mentioned in the Background section, the goals of Modularity are
+specifically to *not* implement parallel-installability. We recommend
+instead that users should rely on other mechanisms such as
+virtualization or containerization to accomplish this. If
+parallel-installation is unavoidable, then Modularity is not the correct
+tool for this job.
+
+=== Arbitrary stream switching
+
+Module streams are intended to be compatible update streams. That means
+they must follow the same rules regarding RPM package-level updates
+within the stream. By definition, two streams of the same module exist
+because upgrades (or downgrades or cross-grades…) are not capable of
+being done in a safe, automated fashion.
+
+That does not mean that stream switching should be impossible, but it
+does mean that we will not build any tools intended to handle such
+switching in a generic manner. Stream switches should be handled on a
+module-by-module basis and detailed instructions and/or tools written
+for each such case.

