SIMP 6.6.0 planning for EL8 modularity + ISO distribution

EL8 AppStream/modularity

Starting with EL8, many RPMs are distributed in Application Streams (based on DNF modularity). This is not optional—a basic EL8 installation may require packages from both BaseOS and AppStream, and EPEL distributes some software in the new epel-modular repository.

New Repository metadata

Repositories with modules must provide an additional layer of modular (modulemd) metadata.

• The modulemd manifest describes every module stream and their RPMs across the entire repository.

• Modular repositories can only be built with a modularity-aware repository tool, like createrepo_c.

• The old createrepo command cannot build modular repositories—it is unaware of (and will destroy) modular metadata. DNF will be unable to install modular RPMs in a repository built by createrepo.

• Modular repositories may include a mix of standard RPMs and modular RPMs.

New RPM & DNF behaviours

• DNF must be used to install modular RPMs; yum does not support modularity

• Each modular RPM is built with a special ModularityLabel tag.

• DNF will not install an RPM with a ModularityTag unless it can be associated with modulemd metadata from an available repository.

AppStream Lifecycle

• Each RHEL AppStream (module stream) has its own lifecycle, which is usually not the full 10-year lifecycle of the Base OS: https://access.redhat.com/support/policy/updates/rhel8-app-streams-life-cycle.

  Open

  

• This includes most default streams.

• Some streams retire within two years (the first RHEL stream retires in April 2021).

SIMP ISO Build

Requirements

1. The SIMP ISO MUST be self-contained and capable of installing and boot-strapping a fully-functional OS and SIMP server on an isolated (offline) network.

2. Packages MUST install correctly from the SIMP ISO and repositories mirrored from the SIMP ISO

3. On systems with packages installed from the SIMP ISO/mirrors that are later configured to use those packages' original source repositories/mirrors:

   1. All ISO-distributed packages MUST upgrade correctly (including modular packages)

   2. All ISO-distributed package metadata MUST integrate seamlessly with the upstream (including modules and streams)

4. The ISO build process SHOULD continue to only include the packages it needs

Caveats

1. If necessary, the SIMP 6.6.0 ISO MAY mirror the entire base OS’s DVD AppStream/ repository if it would avoid delays to the SIMP 6.6.0 build process.

2. However, the ISO MUST NOT mirror entire third-party modular repositories (like epel-modular) just to provide a few packages it requires from single streams (e.g., 389-directory-server:stable)

3. “SIMP RPMs” (packaged by the SIMP project, included in the tarball) will NOT be distributed in module streams for the SIMP 6.6.0 release (and probably never).

Constraints

1. SIMP 6.6.0 requires RPMs from at least one module stream from epel-modular: 389-directory-server:stable, in order to support 389ds.

Requirements for SIMP ISO build + EL8 modularity

The following (derived from the information above) should enable us to build SIMP ISOs that distribute modular RPMs:

1. The ISO build process MUST distribute modular RPMs in “slim” subsets of their original module streams.

1. In order to maintain seamless interoperability, each “slim” modular stream subset MUST use the same name + stream + version + context + architecture (N:S:V:C:A) as their upstream sources

2. If feasible, the “slimming” process SHOULD be generalized enough to apply to packages from external sources like epel-modular AND the base OS’s DVD AppStream/ repository.

Alternatives considered were to include the entire modular repository (too big) or not ship modular dependencies (ISO + download server would no be longer self-contained).

2. The old createrepo command MUST NEVER be run on a modular repository

This applies to both the ISO’s build process and post-installation local tooling on SIMP systems.

1. TODO The ISO’s modular repositories MUST be built with modularity-aware tools, like createrepo_c or createrepo_mod from modulemd-tools.

2. Local SIMP tooling MUST be changed to NEVER use createrepo on modular repositories.

3. Modular repos' modulemd metadata MUST remain intact when mirrored or rebuilt.

1. Modular repos MUST NOT be mirrored or rebuilt without it

2. The modulemd metadata will always be necessary in order to install these repos' modular packages—and once lost it cannot be recreated from their RPMs (unlike standard RPMs).

3. TODO Local SIMP tooling that mirrors modular repositories MUST preserve modulemd metadata (e.g., dnf reposync --download-metadata)

The “Slim” subset repos are of particular concern, since SIMP tooling is guaranteed to encounter them.