Page Comparison

Questions

Assuming at least some of packages we need to ship with SIMP already belong to modular repository streams, how can/should we handle:

building SIMP distributions (pkglist.txt + packages.yaml, ISO, etc.,)

Current methods of building/sourcing RPM packages

SIMP is distributed as a self-contained collection of RPM packages:

• SIMP packages are built by the SIMP team, from the repos under src/ in Puppetfile.pinned.

• Minimal Base OS packages are filtered from a base OS’s installation media (usually a DVD ISO), down to the files DVD/<os.release.major>-pkglist.txt.

• External packages are RPMs downloaded based on the data in yum_data/packages.yaml, and come from additional sources (e.g., EPEL, Puppet, base OS extras, various vendors).

• (Optionally): Custom packages placed manually into the packages/ directory and treated like External packages.

Together, these RPMs MUST self-resolve (enough) to install a basic SIMP system on a network-isolated host.

Current process to build SIMP distributions

1. Build a DVD overlay directory (and tarball) containing the SIMP and External packages.

2. Unpack the base OS installation media into an ISO staging directory.

3. Prune the unpacked ISO RPMs down to the Minimal Base OS packages.

4. Merge the DVD overlay contents over the files already in the ISO staging directory.

5. Verify that all ISO staging directory’s RPMs can self-resolve by running repoclosure.

6. Build a SIMP distribution ISO from the contents of the ISO staging directory.

Potential problems with DNF modules

• Filtering an OS ISO down to its pkglist + adding external packages in packages.yaml

  • both

We can use tools like dir2module and repo2module to re-package specific modular packages into slimmer versions of their own modules and then use mergerepo_c

Questions

Assuming at least some of packages we need to ship with SIMP already belong to modular repository streams, how can/should we handle:

Building SIMP RPM distributions (pkglist.txt + packages.yaml, repositories, ISOs, etc.,)

Current methods of building/sourcing RPM packages

A SIMP RPM distribution is a self-contained collection of RPM packages:

• SIMP packages are built by the SIMP team, from the repos under src/ in Puppetfile.pinned.

• Minimal Base OS packages are filtered from a base OS’s installation media (usually a DVD ISO), down to the files DVD/<os.release.major>-pkglist.txt.

• External packages are RPMs downloaded based on the data in yum_data/packages.yaml, and come from additional sources (e.g., EPEL, Puppet, base OS extras, various vendors).

• (Optionally): Custom packages placed manually into either the directory yum_data/packages/ (which locates a yum source like External packages) or yum_data/packages/aux_packages/ (included but not checked).

These packages are distributed by the SIMP ISO. Together, these RPMs MUST self-resolve (enough) to install a basic SIMP system on a network-isolated host.

Current process to build and distribute SIMP ISO release

Here’s a quick summary of the build process to point out where the problems are

1. Build a DVD overlay directory (and tarball) containing the SIMP packages.

2. 1️⃣ Download External packages (rake build:yum:sync) into yum_data/packages/

3. Unpack the base OS installation media into an ISO staging directory.

   1. 1️⃣ Prune the unpacked ISO RPMs down to the Minimal Base OS packages.

4. Merge the DVD overlay contents & External packages on top of the files already in the ISO staging directory.

   1. 2️⃣ Create new yum repositories using createrepo

   2. 3️⃣ Verify that all ISO staging directory’s RPMs can self-resolve by running repoclosure (rake pkg:repoclosure).

5. Build a SIMP distribution ISO from the contents of the ISO staging directory

6. 1️⃣ Host yum mirrors containing subsets of External package repositories (like EPEL) on the SIMP download service (ex: https://download.simp-project.com/SIMP/yum/releases/6.5.0-1/el/7/x86_64/epel/).

Pain points with modularity + the current process

• 1️⃣: You can’t simply [download the modular RPMs you want/remove the modular RPMS you don’t need/throw a bunch of modular RPMs together] and run createrepo to re-host them. You also need the correct metadata for all downloaded modular RPMs' streams, and special new commands.

  • AFAICT, this metadata can only be obtained from the source repo.

  • There isn’t a complete “roll-your-own” solution to create modular repositories yet. Most of the tooling is meant to mirror existing repositories. There are community tools, but they are incomplete, unsupported, and buggy.

• 2️⃣: createrepo_c is needed to create a useable modular repository.
  However, we need quite a few things before that:

  • ⚠4️⃣ We need to get the correct modulemd data for modules.yaml from the original repository.

  • 5️⃣ We need some way of generating the modulemd YAML data for each module.

  • 6️⃣ We need to be able to add all the the modules' modulemd data into a single modules.yaml file for the entire repository.

  • ⚠7️⃣Finally we need to run the correct commands to create/merge the repo and the module metadata (createrepo_mod or the commands it runs)

• 3️⃣: yum-utilsprovides a CLI compatibility layer with the newer DNF sub-commands (including repoclosure), but may require specific arguments (documented further below)

  • dnf repoclosureis sort of module-aware (bz#1547041), and may need extra logic to module enable non-default streams that need to be considered while depsolving.
    

Creating repos with useable modularity streams

⚠8️⃣ We can use tools like dir2module and repo2module[1] to re-package specific packages into slimmer versions of their source modules and then use mergerepo_c (or creatrepo_mod) to collect them into a slimmed-down AppStream/ repo (based on pkglist.txt) or SIMP/ (based on packages.yml). However, there are several issues that make this complicated:

• RPM packages from repo modules contain a ModularityLabel header that is unique to the module’s name + stream + version + context + architecture (NVSCA). DNF must install these RPMs from modules (with modular metadata)—and will refuse to install them as ursine packages.

  • dir2module

  requires

  • and repo2modules require a complete NVSCA string generating module metadata for a directory of RPMs.

  

  • 5️⃣ The dir2module script provided by the EPEL8 RPM modulemd-tools-0.7-1.el8.noarch does not create a default (or any) profile for the module it creates

  .

  • The master in the GitHub repo at ; it is effectively useless; use repo2module instead (which is still missing the Arch in NVCSA).

• At a minimum, a repackaged (slim) module’s name and stream must match its source repo’s module metadata.

  • This is required so the packages maintain continuity with the complete upstream repo (e.g., receiving update from the complete AppStream repo, epel-modular, etc)

  • The module’s versionnumber must evaluate to more than the versions of earlier modules and less than the version of later modules in the full source repository.

• The ModularityLabel header is string unique to a module builds' NVSCfor a particular platform. The header data in RPMs packaged by RHEL/CentOS build system is currently in NVSC format, however this data is arbitrary and cannot be relied uponto provide accurate NVSC data for the module.

• The only canonical source for a module’s correct NVSCA/P data is the source repo’s metadata (generally under repodata/{XXXXX}-modules.yaml.gz, and defined by repodata/repomd.xml under <data type="modules">`)

...

...

• Can we repackage a “thinned-out” module with only select packages?

  • probably:

    • EL8 RPM prereqs: createrepo_c (appstream) + modulemd-tools (epel)

    • Process:

      1. Create initial repo with ursine modules

         Code Block
         mkdir -p $NEW_REPO_DIR/Packages/ursine cp "${URSINE_PACKAGE_FILES[@]}" "$NEW_REPO_DIR/Packages/ursine/" cd "$NEW_REPO_DIR" createrepo_c .

      2. Create a modular repo from packages that already have a common module header
         

         Status
         colour Yellow title WARNING

         The approach of taking NSVCA from the RPM headers is incomplete and cannot be relied upon to be accurate or available—the ModularityLabel header can contain any String

         

         Status
         colour Red title DO NOT PRODUCTIZE

         Use another means of obtaining this data; preferably from the repo itself (the data is sourced from the source repository’s {XXXXX}-modules.yaml.gz file)
         

         Code Block

         # Get the ModularityLabel from the RPMs ### WARNING: All RPMs in the module must have a SINGLE and IDENTICAL ModularityLabel find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \ | sort -u ### WARNING: the ModularityLabel headers in RPMs build by EL and EPEL #### are (currently) in N:S:V:C format by convention, but in ##### reality this string is arbitrary and cannot be relied upon ##### to reflect the actual source module's metadata. MODULE_HEADER=nodejs:10:8020020200707141642:6a468ee4

...