DNF modularity notes

Owned by Chris Tessmer
Last updated: May 10, 2021
15 min read

 

Questions

Assuming at least some of packages we need to ship with SIMP already belong to modular repository streams, how can/should we handle:

Building SIMP RPM distributions (pkglist.txt + packages.yaml, repositories, ISOs, etc.,)

80% Confidence it’s Possible — but it will be complicated and hacky

  • The current methods of downloading external packages AND pruning OS ISOs down to a minimal set of packages will both result in broken modular repositories.

    • This could be solved by recreating the RPM’s module streams with exactly the same NSVCA values as the source repository—but practically every step of that will be hacky and require unsupported tooling or several workarounds.

  • The repo manipulation methods (createrepo, repoclosure) aren’t module-aware

Current methods of building/sourcing RPM packages

A SIMP RPM distribution is a self-contained collection of RPM packages:

  • SIMP packages are built by the SIMP team, from the repos under src/ in Puppetfile.pinned.

  • Minimal Base OS packages are filtered from a base OS’s installation media (usually a DVD ISO), down to the files DVD/<os.release.major>-pkglist.txt.

  • External packages are RPMs downloaded based on the data in yum_data/packages.yaml, and come from additional sources (e.g., EPEL, Puppet, base OS extras, various vendors).

  • (Optionally): Custom packages placed manually into either the directory yum_data/packages/ (which locates a yum source like External packages) or yum_data/packages/aux_packages/ (included but not checked).

These packages are distributed by the SIMP ISO. Together, these RPMs MUST self-resolve (enough) to install a basic SIMP system on a network-isolated host.

Current process to build and distribute SIMP ISO release

Here’s a quick summary of the SIMP 6.5.0 build process to illustrate where the problems with modularity will arise:

  1. Build a DVD overlay directory (and tarball) containing the SIMP packages.

  2. Download External packages (rake build:yum:sync) into yum_data/packages/

  3. Unpack the base OS installation media into an ISO staging directory.

    1. Prune the unpacked ISO RPMs down to the Minimal Base OS packages.

  4. Merge the DVD overlay contents & External packages on top of the files already in the ISO staging directory.

    1. Create new yum repositories using createrepo

    2. Verify that all ISO staging directory’s RPMs can self-resolve by running repoclosure (rake pkg:repoclosure).

  5. Build a SIMP distribution ISO from the contents of the ISO staging directory

  6. Host yum mirrors containing subsets of External package repositories (like EPEL) on the SIMP download service (ex: https://download.simp-project.com/SIMP/yum/releases/6.5.0-1/el/7/x86_64/epel/).

Pain points with modularity + the current process

  1. The DVD overlay directory (and tarball) should not be affected, because it won’t contain modular RPMs (the SIMP project doesn’t create them).

  2. Downloading External packages (rake build:yum:sync) into yum_data/packages/ has modularity problems:

    1. : You can’t simply [download the modular RPMs you want → remove the modular RPMS you don’t need → collect various modular RPMs together in a directory] and run createrepo to re-host them in a new repository. You need the correct metadata (called modulemd metadata) for all downloaded modular RPMs' streams, and special new commands, like createrepo_c.

      1. This metadata can only be obtained from the modular RPMs' source repo.

    2. There isn’t a “roll-your-own” solution to create arbitrary modular repositories yet The supported DNF repository tooling is mostly meant to mirror existing repositories.

      1. There are community tools (dir2module, repo2module), but they are incomplete, unsupported, and buggy. Some of these tools have been rolled into later versions of createrepo_c, but not the version on EL8 (and AFAIK, they may still be buggy).

      2. Pulp is the closest tool, because it supports some flavors of modular subsets

  3. Unpack the base OS installation media into an ISO staging directory runs into problems with pruning:

    1. : SIMP-9644: Determine how to safely prune to pkglist with modularityClosed
      The naive pruning strategy of “rm RPM is it’s not in a *pkglist.txt file” may work with very specific RPMs, but it will still need to the original modularity metadata and additional information to ensure the correct modules are used

  4. Merging the DVD overlay

    1. : createrepo_c is needed to create a useable modular repository.
      However, it requires quite a few things before it can work:

      • The correct modulemd data (see format specs) for modules.yaml from the original repository.

      • Some way of generating the modulemd YAML data for each module.

      • The ability to add all the modules' modulemd data into a single modules.yaml file for the entire repository.

      • The correct tools to create/merge the repo and the module metadata (createrepo_mod or the *_c commands it runs)

    2. : yum-utils provides a CLI compatibility layer with the newer DNF sub-commands (including repoclosure), but may require specific arguments (documented further below)

      • dnf repoclosure is sort of module-aware (bz#1547041), and may need extra logic to module enable non-default streams that need to be considered while depsolving.

    3. : To generate the modulemd YAML data for each module and combine it into a single modulemd.yaml file for createrepo_c to consume, we will need to roll either:

      1. build logic to do it in one shot

      2. build logic to glue together a process that combines the buggy community tools from modulemd-tools.

  5. Build a SIMP distribution ISO from the contents of the ISO staging directory

  6. Host yum mirrors containing subsets of External package repositories (like EPEL) on the SIMP download serv

Creating repos with useable modularity streams

We can re-package specific packages into slimmer versions of their source modules and then use mergerepo_c (or creatrepo_mod) to collect them into a slimmed-down AppStream/ repo (based on pkglist.txt) or SIMP/ (based on packages.yml). However, there are several issues that make this complicated:

tools like dir2module and repo2module[note] to re-package specific packages into slimmer versions of their source modules and then use mergerepo_c (or creatrepo_mod) to collect them into a slimmed-down AppStream/ repo (based on pkglist.txt) or SIMP/ (based on packages.yml). However, there are several issues that make this complicated:

  • RPM packages from repo modules contain a ModularityLabel header that is unique to the module’s name + stream + version + context + architecture (NSVCA). DNF must install these RPMs from modules (with modular metadata)—and will refuse to install them as ursine packages.

    • dir2module and repo2modules require a complete NSVCA string generating module metadata for a directory of RPMs.

    • The dir2module script provided by the EPEL8 RPM modulemd-tools-0.7-1.el8.noarch does not create a default (or any) profile for the module it creates. It is effectively useless. repo2module is better (which is still missing the Arch in NSVCA).

  • A repackaged (slim) module’s name and stream and context and architecture must match the values in the source repo’s metadata for the original module.

    • This is required so the packages maintain continuity with the complete upstream repo (e.g., receiving update from the complete AppStream repo, epel-modular, etc)

    • The version number must evaluate to more than the earlier (module versions) and less than the version of later modules in the full source repository. This number (like stream and context) is an arbitrary string set by the build platform, so we have to get it

    • For practical purposes, you need to mirror the repo’s metadata YAML at the same time as you retrieve the packages—it might be updated later, even if the packages you see hosted there are the same

  • The ModularityLabel header is string unique to a module builds' NSVC for a particular platform. The header data in RPMs packaged by RHEL/CentOS build system looks useful because a string it’s in NSVC format, however this data is actually arbitrary and cannot be relied upon to provide accurate NSVC data for the module.

  • The only canonical source for a module’s correct NSVCA/P data is the source repo’s metadata (generally under repodata/{XXXXX}-modules.yaml.gz, and defined by repodata/repomd.xml under <data type="modules">`)

  • I’ve tested a repacked “slim” repo alongside a repo with the full module w/identical NSVCA details, and it successfully resolved its metadata/packages with the full module.

    • This should also behave correctly with updated modules, but I haven’t been able to stage that yet.

 

Testing ISO RPMs' completeness with a repoclosure

Recommendation: Use dnf repoclosure (see below for arguments)

To run a repoclosure against ISO-bound/based repos that are staged on the local filesystem:

dnf repoclosure --disablerepo=\* \ --repofrompath=base,/mnt/BaseOS \ --repofrompath=test1,/opt/dnf_reposync_from_iso \ --arch x86_64 \ --check base \ --check tes1

Notes:

  • Make sure there is a --repofrompath= for each local repository directory that should be considered during resolution.

  • Using --disablerepo=\* --repofrompath=, the repoclosure doesn’t require purpose-built *.repo files and doesn’t have to run inside a chroot/container/mock.

  • The --arch x86_64 may be needed to keep the closure focused on the ISO’s target arch.

    • As released, the EL 8.3 ISO’s AppStream/ repo doesn’t close without it.

 

Impacts to the ISO installation

Recommendation: A minimum, change the createrepo . line in auto.cfg on systems installing modular repositories

This is effectively impossible to test until we have a SIMP ISO with modular repositories.

 

Impacts to unpack_dvd & self-hosted DNF repositories (for kickstarts, etc)

How should unpack_dvd handle repos that do/may contain modules?

If these recommendations are implemented, then a site’s unpack_dvd tool should only need to copy/reposync the files on the ISO.

This has been tested by mounting an ISO as a loopback device and syncing everything (including module metadata) with dnf reposync --download-metadata […] .

 

 

How does SELinux support this?

Where do the policies for the packages in various module streams come from?

Modularity CLI examples

Simple dnf module commands

View all modules and streams

dnf module list --all

View a module’s available streams

# dnf module list --available nodejs Waiting for process with pid 79896 to finish. Last metadata expiration check: 0:00:01 ago on Fri 12 Mar 2021 02:03:02 AM UTC. CentOS Linux 8 - AppStream Name Stream Profiles Summary nodejs 10 [d] common [d], development, minimal, s2i Javascript runtime nodejs 12 common [d], development, minimal, s2i Javascript runtime nodejs 14 common [d], development, minimal, s2i Javascript runtime Extra Packages for Enterprise Linux Modular 8 - x86_64 Name Stream Profiles Summary nodejs 13 default, development, minimal Javascript runtime Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

View all default streams

The 21 and 11 are the first two field widths. These may vary depending on the length of the stream names.

A stab at at more automation-friendly formatting

View packages in a module

 

View modules/profiles that provide a package

Example:

 

View the packages in each profile (for a specific stream)

 

 

 

Find all modular RPM files that under a directory and print their module headers

This identifies modular RPMs that would need to have their module streams re-created when distributed independently from their source repos:

The %{ModularityLabel} header is in (module)name:stream:version:context:arch (N:S:V:C:A) format (on RedHat/CentOS/Fedora-built packages), so the result looks like this:

However, it isn’t possible to re-create specific module/streams based on the RPM’s ModularityLabel headers.

  1. A stream’s full modulemd metadata is only found in the source repo’s *metadata.yaml.gz

  2. The content of ModularityLabel is unique to a build, but its contents cannot be trusted.

Note that the repo2module tool just creates a default profile called everything.

 

Identify all unique modules/streams from a collection of RPM files

 

Mirror the contents of a DNF repository, preserving all modules and package groups. 

The current state of CentOS 8 createrepo_c + modulemd-tools (EPEL8) allow us to:

(Works from CentOS 8.3 and CentOS 7.8, requires packages dnf and dnf-plugins-core )

An example of this mirroring a mounted CentOS 8.3 ISO’s AppStream repository:

dnf reposync should probably be the only kind of modular-capable mirroring that on-site tools like unpack_dvd should use.

DNF repoclosure

The ISO build process (really the tar build process) runs repoclosure to make sure the packages on the ISO will be self-contained.

Clean DNF install --downloadonly with all deps

Make sure you enable/disable the repos to match what you intend for resolution; --config is an option, too.

Notes:

  • An an --enablerepo= for each repo to consider during resolution.

  • Pointing to an empty --installroot= will cause dnf install --downloadonly to download EVERY dependency, including the packages for the baseos.

  • In this case, --setopt=install_weak_deps=False may be useful to ignore weak RPM dependencies and cut down on the download size.

     

Create initial repo with ursine modules

  • Create a modular repo from packages that already have a common module header
    WARNING The approach of taking N:S:V:C:A from the RPM headers below is a.) incomplete and b.) cannot be relied upon to be accurate or present—the ModularityLabel header can contain any String.

    DO NOT PRODUCTIZE Use another means to obtain N:V:S:C:A data; preferably from the repo itself (the data is sourced from the source repository’s {XXXXX}-modules.yaml.gz file)

 

Documentation

Modularity direction

In early 2020, modularity development moved from Fedora’s Modularity WG to RedHat’s internal DNF team. Fedora development and governance was very open and public, which made it relatively easy to hunt down clarifying information about otherwise obscure details in pagure or the WG’s meeting logs.

Since RedHat took over, it has become very difficult to find up-to-date documentation or discussions, even (especially?) in BZ.

  • Why Modularity? An explanation and list of links.

  • RHEL 9 and modularity (18 June 2020, Fedora devel@ mailing list)

    • Discussion with official RedHat rep, after taking modularity from Fedora and moving its development to an internal DNF team.

    • There’s also a side-discussion in the middle about the “inevitability” of moving the modular “namespacing” (N:S?) into RPM’s headers/tags (like ModularityLabel), where several of Fedora’s former modular WG members chime in with some educational

Projects

More esoteric things that may come in handy in the future: