Page Comparison

...

...

1. The DVD overlay directory (and tarball)should not be affected, because it won’t contain modular RPMs (the SIMP project doesn’t create them).

2. Downloading External packages (rake build:yum:sync) into yum_data/packages/ has modularity problems:

   1. 1️⃣: You can’t simply [download the modular RPMs you want → remove the modular RPMS you don’t need → collect various modular RPMs together in a directory] and run createrepo to re-host them in a new repository. You need the correct metadata (called modulemd metadata) for all downloaded modular RPMs' streams, and special new commands, like createrepo_c.

      1. This metadata can only be obtained from the modular RPMs' source repo.

   2. There isn’t a “roll-your-own” solution to create arbitrary modular repositories yet The supported DNF repository tooling is mostly meant to mirror existing repositories.

      1. There are community tools (dir2module, repo2module), but they are incomplete, unsupported, and buggy. Some of these tools have been rolled into later versions of createrepo_c, but not the version on EL8 (and AFAIK, they may still be buggy).

      2. Pulp is the closest tool, because it supports some flavors of modular subsets

3. Unpack the base OS installation media into an ISO staging directory runs into problems with pruning:

   1. 2️⃣: Jira LegacyserverSystem JIRAcolumnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolutionserverId45e63be1-2551-37a6-b050-86a38172f71dkey: https://simp-project.atlassian.net/browse/SIMP-9644
      The naive pruning strategy of “rm RPM is it’s not in a *pkglist.txt file” may work with very specific RPMs, but it will still need to the original modularity metadata and additional information to ensure the correct modules are used

4. Merging the DVD overlay

   1. 3️⃣: createrepo_c is needed to create a useable modular repository.
      However, it requires quite a few things before it can work:

      • ⚠➡5️⃣ The correct modulemd data (see format specs) for modules.yaml from the original repository.

      • ➡5️⃣ Some way of generating the modulemd YAML data for each module.

      • ➡5️⃣ The ability to add all the modules' modulemd data into a single modules.yaml file for the entire repository.

      • ⚠➡5️⃣ The correct tools to create/merge the repo and the module metadata (createrepo_mod or the *_c commands it runs)

   2. 4️⃣: yum-utilsprovides a CLI compatibility layer with the newer DNF sub-commands (including repoclosure), but may require specific arguments (documented further below)

      • dnf repoclosureis sort of module-aware (bz#1547041), and may need extra logic to module enable non-default streams that need to be considered while depsolving.

   3. ➡6️⃣: To generate the modulemd YAML data for each module and combine it into a single modulemd.yaml file for createrepo_c to consume, we will need to roll either:

      1. build logic to do it in one shot

      2. build logic to glue together a process that combines the buggy community tools from modulemd-tools.

5. Build a SIMP distribution ISO from the contents of the ISO staging directory

6. Host yum mirrors containing subsets of External package repositories (like EPEL) on the SIMP download serv
   

Creating repos with useable modularity streams

⚠5️⃣ We can re-package specific packages into slimmer versions of their source modules and then use mergerepo_c (or creatrepo_mod) to collect them into a slimmed-down AppStream/ repo (based on pkglist.txt) or SIMP/ (based on packages.yml). However, there are several issues that make this complicated:

tools like dir2module and repo2module[note] to re-package specific packages into slimmer versions of their source modules and then use mergerepo_c (or creatrepo_mod) to collect them into a slimmed-down AppStream/ repo (based on pkglist.txt) or SIMP/ (based on packages.yml). However, there are several issues that make this complicated:

• RPM packages from repo modules contain a ModularityLabel header that is unique to the module’s name + stream + version + context + architecture (NSVCA). DNF must install these RPMs from modules (with modular metadata)—and will refuse to install them as ursine packages.

  • dir2module and repo2modules require a complete NSVCA string generating module metadata for a directory of RPMs.

  • 6️⃣ The dir2module script provided by the EPEL8 RPM modulemd-tools-0.7-1.el8.noarch does not create a default (or any) profile for the module it creates. It is effectively useless. repo2module is better (which is still missing the Arch in NSVCA).

• A repackaged (slim) module’s name and stream and context and architecture must match the values in the source repo’s metadata for the original module.

  • This is required so the packages maintain continuity with the complete upstream repo (e.g., receiving update from the complete AppStream repo, epel-modular, etc)

  • The versionnumber must evaluate to more than the earlier (module versions) and less than the version of later modules in the full source repository. This number (like stream and context) is an arbitrary string set by the build platform, so we have to get it

  • For practical purposes, you need to mirror the repo’s metadata YAML at the same time as you retrieve the packages—it might be updated later, even if the packages you see hosted there are the same

• The ModularityLabel header is string unique to a module builds' NSVCfor a particular platform. The header data in RPMs packaged by RHEL/CentOS build system looks useful because a string it’s in NSVC format, however this data is actually arbitrary and cannot be relied uponto provide accurate NSVC data for the module.

• The only canonical source for a module’s correct NSVCA/P data is the source repo’s metadata (generally under repodata/{XXXXX}-modules.yaml.gz, and defined by repodata/repomd.xml under <data type="modules">`)

• 🎉 I’ve tested a repacked “slim” repo alongside a repo with the full module w/identical NSVCA details, and it successfully resolved its metadata/packages with the full module.

  • This should also behave correctly with updated modules, but I haven’t been able to stage that yet.

...

...

(Works from CentOS 8.3 and CentOS 7.8, requires packages dnf and dnf-pluginplugins-core )

An example of this mirroring a mounted CentOS 8.3 ISO’s AppStream repository:

...

• Create a modular repo from packages that already have a common module header
  

  Status
  colour Yellow title WARNING

  The approach of taking N:S:V:C:A from the RPM headers below is a.) incomplete and b.) cannot be relied upon to be accurate or present—the ModularityLabel header can contain any String.

  

  Status
  colour Red title DO NOT PRODUCTIZE

  Use another means to obtain N:V:S:C:A data; preferably from the repo itself (the data is sourced from the source repository’s {XXXXX}-modules.yaml.gz file)
  

  Code Block

  # Get the ModularityLabel from the RPMs ########################################################## #### UPDATE: DO NOT USE OR PRODUCTIZE THIS TECHNIQUE ##### ########################################################## # Notes: # - All RPMs in the module must have a SINGLE and IDENTICAL ModularityLabel # - The only thing required of this String is that it is unique to RPMs # from other modules (and different versions/contexts of this module) find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \ | sort -u ### WARNING: the ModularityLabel headers in RPMs build by EL and EPEL #### are (currently) in N:S:V:C format by convention, but in ##### reality this string is arbitrary and cannot be relied upon ##### to reflect the actual source module's metadata. ##### The actual NVSCA/P data can *ONLY* be obtained from the ##### original repo's metadata MODULE_HEADER=nodejs:10:8020020200707141642:6a468ee4

...