Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Table of Contents
maxLevel3

...

Assuming at least some of packages we need to ship with SIMP already belong to repo modulesmodular repository streams, how can/should we handle:

building

...

SIMP distributions (pkglist.txt + packages.yaml, ISO, etc.,)

During the process of building a SIMP ISO, we use two files

📄 DVD/<os.release.major>-pkglist.txt — a list of packages to keep from OS’s base installation media (ex:, CentOS-8.3.2011-x86_64-dvd1.iso). Anything not listed in this file will be pruned from the base OS installation packages before

📄 yum_data/packages.yaml — a manifest of additional packages needed to support a self-contained SIMP ISO installation. These are extra packages not found on the base OS’s installation media (or are not the correction version).

An example entry from the CentOS 7 packages.yaml file:

jq: :rpm_name: jq-1.6-2.el7.x86_64.rpm :source: http://mirrors.lug.mtu.edu/epel/7/x86_64/Packages/j/jq-1.6-2.el7.x86_64.rpm
Code Block
languageyaml
Note

80% Confidence it’s Possible — but it will be complicated and hacky

  • The current methods of downloading external packages AND pruning OS ISOs down to a minimal set packages will both result in broken modular repositories.

    • This could be solved by recreating the RPM’s module streams with exactly the same NVSCA values as the source repository, but practically every step of that will require unsupported tooling or a hacky workaround.

  • The current methods (particularly repoclosure) rely on older yum-specific tooling that isn’t module-aware

    • There is dnf repoclosure, which is sort of module-aware (bz#1547041)

Current methods of building/sourcing RPM packages

SIMP is distributed as a self-contained collection of RPM packages:

  • SIMP packages are built by the SIMP team, from the repos under src/ in Puppetfile.pinned.

  • Minimal Base OS packages are filtered from a base OS’s installation media (usually a DVD ISO), down to the files DVD/<os.release.major>-pkglist.txt.

  • External packages are RPMs downloaded based on the data in yum_data/packages.yaml, and come from additional sources (e.g., EPEL, Puppet, base OS extras, various vendors).

  • (Optionally): Custom packages placed manually into the packages/ directory and treated like External packages.

Together, these RPMs MUST self-resolve (enough) to install a basic SIMP system on a network-isolated host.

Current process to build SIMP distributions

  1. Build a DVD overlay directory (and tarball) containing the SIMP and External packages.

  2. Unpack the base OS installation media into an ISO staging directory.

  3. Prune the unpacked ISO RPMs down to the Minimal Base OS packages.

  4. Merge the DVD overlay contents over the files already in the ISO staging directory.

  5. Verify that all ISO staging directory’s RPMs can self-resolve by running repoclosure.

  6. Build a SIMP distribution ISO from the contents of the ISO staging directory.

Potential problems with DNF modules

...

...

    • both

We can use tools like dir2module and repo2module to re-package specific modular packages into slimmer versions of their own modules and then use mergerepo_c to collect them into a slimmed-down AppStream/ repo (based on pkglist.txt) or SIMP/ (based on packages.yml). However, there are several issues that make this complicated:

...

  • RPM packages from repo modules contain a ModularityLabel header that is unique to the module’s name + stream + version + context + architecture (NVSCA). DNF must install these RPMs from modules (with modular metadata)—and will refuse to install them as ursine packages.

  • dir2module requires a complete NVSCA string generating module metadata for a directory of RPMs.

  • (warning) The dir2module script provided by the EPEL8 RPM modulemd-tools-0.7-1.el8.noarch does not create a default (or any) profile for the module it creates.

    • The master in the GitHub repo at

  • At a minimum, a repackaged (slim) module’s name and stream must match its source repo’s module metadata.

    • This is required so the packages maintain continuity with the complete upstream repo (e.g., receiving update from the complete AppStream repo, epel-modular, etc)

    • The module’s versionnumber must evaluate to more than the versions of earlier modules and less than the version of later modules in the full source repository.

  • The ModularityLabel header is string unique to a module builds' NVSCfor a particular platform. The header data in RPMs packaged by RHEL/CentOS build system is currently in NVSC format, however this data is arbitrary and cannot be relied uponto provide accurate NVSC data for the module.

  • The only canonical source for a module’s correct NVSCA/P data is the source repo’s metadata (generally under repodata/{XXXXX}-modules.yaml.gz, and defined by repodata/repomd.xml under <data type="modules">`)

...

Notes:

...

  • Can we repackage a “thinned-out” module with only select packages?

    • probably:

      • EL8 RPM prereqs: createrepo_c (appstream) + modulemd-tools (epel)

      • Process:

        1. Create initial repo with ursine modules

          Code Block
          mkdir -p $NEW_REPO_DIR/Packages/ursine
          cp "${URSINE_PACKAGE_FILES[@]}" "$NEW_REPO_DIR/Packages/ursine/"
          cd "$NEW_REPO_DIR"
          createrepo_c .
        2. Create a modular repo from packages that already have a common module header
          (warning)

          Status
          colourYellow
          titleWARNING
          The approach of taking NSVCA from the RPM headers is incomplete and cannot be relied upon to be accurate or available—the ModularityLabel header can contain any String

          (warning)

          Status
          colourRed
          titleDO NOT PRODUCTIZE
          Use another means of obtaining this data; preferably from the repo itself (the data is sourced from the source repository’s {XXXXX}-modules.yaml.gz file)

          Code Block
          # Get the ModularityLabel from the RPMs
          ### WARNING: All RPMs in the module must have a SINGLE and IDENTICAL ModularityLabel
          find "$DIR_WITH_RPMS" -name \*.rpm  \
            -exec rpm -qp {} --qf '%{ModularityLabel}\n' \;  \
            | sort -u
          
          ### WARNING: the ModularityLabel headers in RPMs build by EL and EPEL 
          ####         are (currently) in N:S:V:C format by convention, but in 
          #####        reality this string is arbitrary and cannot be relied upon
          #####        to reflect the actual source module's metadata.
          MODULE_HEADER=nodejs:10:8020020200707141642:6a468ee4
          
          

...