Page Comparison

...

...

Here’s a quick summary of the SIMP 6.5.0 build process to point out illustrate where the problems arewith modularity will arise:

1. Build a DVD overlay directory (and tarball) containing the SIMP packages.

2. ☑ 1️⃣ Download External packages (rake build:yum:sync) into yum_data/packages/

3. Unpack the base OS installation media into an ISO staging directory.

   1. ☑ 1️⃣ Prune the unpacked ISO RPMs down to the Minimal Base OS packages.

4. Merge the DVD overlay contents & External packages on top of the files already in the ISO staging directory.

   1. ☑ 2️⃣ Create new yum repositories using createrepo

   2. ☑ 3️⃣ Verify that all ISO staging directory’s RPMs can self-resolve by running repoclosure (rake pkg:repoclosure).

5. Build a SIMP distribution ISO from the contents of the ISO staging directory

6. ☑ 1️⃣ Host yum mirrors containing subsets of External package repositories (like EPEL) on the SIMP download service (ex: https://download.simp-project.com/SIMP/yum/releases/6.5.0-1/el/7/x86_64/epel/).

...

• 1️⃣: You can’t simply [download the modular RPMs you want/remove the modular RPMS you don’t need/throw a bunch of modular RPMs together] and run createrepo to re-host them. You also need the correct metadata (called modulemd metadata) for all downloaded modular RPMs' streams, and special new commands.

  • AFAICT, this metadata can only be obtained from the source repo.

  • There isn’t a complete “roll-your-own” solution to create modular repositories yet. Most of the tooling is meant to mirror existing repositories. There are community tools (dir2module, repo2module), but they are incomplete, unsupported, and buggy.

• 2️⃣: createrepo_c is needed to create a useable modular repository.
  However, we need it requires quite a few things before thatit can work:

  • ⚠4️⃣ The correct modulemd data (see format specs) for modules.yaml from the original repository.

  • ☑ 5️⃣ Some way of generating the modulemd YAML data for each module.

  • ☑ 6️⃣ The ability to add all the modules' modulemd data into a single modules.yaml file for the entire repository.

  • ⚠7️⃣The correct tools to create/merge the repo and the module metadata (createrepo_mod or the *_c commands it runs)

• 3️⃣: yum-utilsprovides a CLI compatibility layer with the newer DNF sub-commands (including repoclosure), but may require specific arguments (documented further below)

  • dnf repoclosureis sort of module-aware (bz#1547041), and may need extra logic to module enable non-default streams that need to be considered while depsolving.

Creating repos with useable modularity streams

...

• 5️⃣6️⃣ To generate the modulemd YAML data for each module and combine it into a single modulemd.yaml file for createrepo_c to consume, we will need to roll either:

  1. build logic to do it in one shot

  2. build logic to glue together a process that combines the buggy community tools from modulemd-tools.
     

Creating repos with useable modularity streams

⚠8️⃣ We can use tools like dir2module and repo2module[note] to re-package specific packages into slimmer versions of their source modules and then use mergerepo_c (or creatrepo_mod) to collect them into a slimmed-down AppStream/ repo (based on pkglist.txt) or SIMP/ (based on packages.yml). However, there are several issues that make this complicated:

• RPM packages from repo modules contain a ModularityLabel header that is unique to the module’s name + stream + version + context + architecture (NSVCA). DNF must install these RPMs from modules (with modular metadata)—and will refuse to install them as ursine packages.

  • dir2module and repo2modules require a complete NSVCA string generating module metadata for a directory of RPMs.

  • 5️⃣ The dir2module script provided by the EPEL8 RPM modulemd-tools-0.7-1.el8.noarch does not create a default (or any) profile for the module it creates; it is effectively useless; use repo2module instead (which is still missing the Arch in NSVCA).

• A repackaged (slim) module’s name and stream and context and architecture must match the values in the source repo’s metadata for the original module.

  • This is required so the packages maintain continuity with the complete upstream repo (e.g., receiving update from the complete AppStream repo, epel-modular, etc)

  • The versionnumber must evaluate to more than the earlier (module versions) and less than the version of later modules in the full source repository. This number (like stream and context) is an arbitrary string set by the build platform, so we have to get it

  • For practical purposes, you need to mirror the repo’s metadata YAML at the same time as you retrieve the packages—it might be updated later, even if the packages you see hosted there are the same

• The ModularityLabel header is string unique to a module builds' NSVCfor a particular platform. The header data in RPMs packaged by RHEL/CentOS build system looks useful because a string it’s in NSVC format, however this data is actually arbitrary and cannot be relied uponto provide accurate NSVC data for the module.

• The only canonical source for a module’s correct NSVCA/P data is the source repo’s metadata (generally under repodata/{XXXXX}-modules.yaml.gz, and defined by repodata/repomd.xml under <data type="modules">`)

• 🎉 I’ve tested a repacked “slim” repo alongside a repo with the full module w/identical NSVCA details, and it successfully resolved its metadata/packages with the full module.

  • This should also behave correctly with updated modules, but I haven’t been able to stage that yet.

...

...

# dnf module list --available nodejs

Waiting for process with pid 79896 to finish.
Last metadata expiration check: 0:00:01 ago on Fri 12 Mar 2021 02:03:02 AM UTC.
CentOS Linux 8 - AppStream
Name                   Stream                 Profiles                                             Summary
nodejs                 10 [d]                 common [d], development, minimal, s2i                Javascript runtime
nodejs                 12                     common [d], development, minimal, s2i                Javascript runtime
nodejs                 14                     common [d], development, minimal, s2i                Javascript runtime

Extra Packages for Enterprise Linux Modular 8 - x86_64
Name                   Stream                 Profiles                                             Summary
nodejs                 13                     default, development, minimal                        Javascript runtime

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

...

With nicer formatting:

# dnf module repoquery nodejs

View modules/profiles that provide a package

dnf module provides 389-ds-base

Example:

# dnf module provides 389-ds-base
Last metadata expiration check: 1:20:30 ago on Wed Mar 24 18:09:28 2021.
389-ds-base-1.4.3.17-1.module_el8+10764+2b5f8656.x86_64
Module   : 389-directory-server:stable:820201201092549:9edba152:x86_64
Profiles : default legacy minimal
Repo     : epel-modular
Summary  : 389 Directory Server

389-ds-base-1.4.3.8-6.module_el8.3.0+604+ab7bf9cc.x86_64
Module   : 389-ds:1.4:8030020201222185615:618f7055:x86_64
Profiles :
Repo     : appstream
Summary  : 389 Directory Server (base)

389-ds-base-1.4.4.9-1.module_el8+10763+39cf6b48.x86_64
Module   : 389-directory-server:testing:820201201092622:9edba152:x86_64
Profiles : default legacy minimal
Repo     : epel-modular
Summary  : 389 Directory Server

389-ds-base-2.0.1-1.module_el8+10522+e95198da.x86_64
Module   : 389-directory-server:next:820201104083723:9edba152:x86_64
Profiles : default minimal
Repo     : epel-modular
Summary  : 389 Directory Server

View the packages in each profile (for a specific stream)

]# dnf module info --profile  nodejs:12
Last metadata expiration check: 2:59:10 ago on Thu 11 Mar 2021 11:02:32 PM UTC.
Name        : nodejs:12:8030020210304194546:30b713e6:x86_64
common      : nodejs
  list \
  | egrep -v '^(Extra|Name|Hint:|CentOS|Last)' \
  | sed -e '/^$/d' \
  | awk '{printf "%-20s %s\n", $1, $2}' \
  | sort
  
389-directory-server next
389-directory-server stable
389-directory-server testing
389-ds               1.4 
ant                  1.10
avocado              82lts
avocado              latest
cobbler              3
container-tools      1.0
container-tools      2.0
container-tools      rhel8
dwm                  latest
freeradius           3.0
gimp                 2.8
go-toolset           rhel8
httpd           : npm development : nodejs 2.4
idm            : nodejs-devel      DL1
idm                  client
inkscape  : npm minimal     : nodejs s2i  0.92.3
javapackages-runtime 201801
jmc    : nodejs             :rhel8
nodejs-nodemonlibselinux-python    2.8
libuv              : npm 

...

Find all modular RPM files that under a directory and print their module headers

This could identify dnf modules that would need to be re-created when shipping modular RPMs independently from their base repos:

find "$DIR_WITH_RPMS" -name \*.rpm \
  -exec rpm -qp {} --qf '%{NVRA}   %{ModularityLabel}\n' \; \
  | grep -v '(none)' \
  | tee modular_rpms.txt

The %{ModularityLabel} macro is returned in (module)name:stream:version:context:arch (NSVCA) format , so the result looks like this:

389-ds-base-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64   389-ds:1.4:8030020200831174107:618f7055
389-ds-base-develepel8-buildroot
llvm-toolset         rhel8
  

View packages in a module

dnf module repoquery nodejs

View modules/profiles that provide a package

dnf module provides 389-ds-base

Example:

# dnf module provides 389-ds-base
Last metadata expiration check: 1:20:30 ago on Wed Mar 24 18:09:28 2021.
389-ds-base-1.4.3.817-51.module_el8.3.0+47310764+536825482b5f8656.x86_64
Module   : 389-ds:1.4:8030020200831174107:618f7055directory-server:stable:820201201092549:9edba152:x86_64
Profiles : default legacy minimal
Repo     : epel-modular
Summary  : 389 Directory Server

389-ds-base-legacy-tools-1.4.3.8-56.module_el8.3.0+473604+53682548ab7bf9cc.x86_64
Module   : 389-ds:1.4:80300202008311741078030020201222185615:618f7055:x86_64
Profiles :
Repo     : appstream
Summary  : 389 Directory Server (base)

389-ds-base-libs-1.4.34.89-51.module_el8.3.0+47310763+5368254839cf6b48.x86_64
Module   : 389-ds:1.4:8030020200831174107:618f7055-directory-server:testing:820201201092622:9edba152:x86_64
Profiles : default legacy minimal
Repo     : epel-modular
Summary  : 389 Directory Server

389-ds-base-snmp-12.40.3.81-51.module_el8.3.0+47310522+53682548e95198da.x86_64
Module   : 389-ds:1.4:8030020200831174107:618f7055
HdrHistogram-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch   jmc:rhel8:8020020200731165725:21dc74c6
HdrHistogram-javadoc-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch   jmc:rhel8:8020020200731165725:21dc74c6
Judy-1.0.5-18.module_el8.1.0+217+4d875839.x86_64   mariadb:10.3:8010020191115015915:cdc1202b
ant-1.10.5-1.module_el8.0.0+47+197dca37.noarch   ant:1.10:8000020190624202340:f7e686af
ant-lib-1.10.5-1.module_el8.0.0+47+197dca37.noarch   ant:1.10:8000020190624202340:f7e686af
aopalliance-1.0-17.module_el8.0.0+39+6a9b6e22.noarch   maven:3.5:8000020190624140656:f7e686af
aopalliance-1.0-20.module_el8.3.0+568+0c23fd64.noarch   maven:3.6:8030020201104064112:a623df05

It may be possible to re-create specific module/streams based on the RPM’s unique headers. However, there will need to be a second data source, because the RPM ModularityLabel has significant limitations:

Note that the repo2module tool just creates a default profile called everything.

...

We can’t rely on RPMs' ModularityLabel string to be in NSVC format, either!

...

Identify all unique modules/streams from a collection of RPM files

find "$DIR_WITH_RPMS" -name \*.rpm \
  -exec rpm -qp {} --qf 'directory-server:next:820201104083723:9edba152:x86_64
Profiles : default minimal
Repo     : epel-modular
Summary  : 389 Directory Server

View the packages in each profile (for a specific stream)

]# dnf module info --profile  nodejs:12
Last metadata expiration check: 2:59:10 ago on Thu 11 Mar 2021 11:02:32 PM UTC.
Name        : nodejs:12:8030020210304194546:30b713e6:x86_64
common      : nodejs
            : npm
development : nodejs
            : nodejs-devel
            : npm
minimal     : nodejs
s2i         : nodejs
            : nodejs-nodemon
            : npm

Find all modular RPM files that under a directory and print their module headers

This identifies modular RPMs that would need to have their module streams re-created when distributed independently from their source repos:

find "$DIR_WITH_RPMS" -name \*.rpm \
  -exec rpm -qp {} --qf '%{NVRA}   %{ModularityLabel}\n' \; \
  | grep -v '^(none)' | sort | uniq -c \
  | sort -nk1,1 \
  | tee unique_rpm_module_streamsmodular_rpms.txt

...

Mirror the contents of a DNF repository, preserving all modules and package groups. 

The current state of CentOS 8 createrepo_c + modulemd-tools (EPEL8) allow us to:

(Works from CentOS 8.3 and CentOS 7.8, requires packages dnf and dnf-plugin-core )

An example of this mirroring a mounted CentOS 8.3 ISO’s AppStream repository:

PATH_TO_LOCAL_MIRROR=/path/to/Appstream
PATH_TO_SOURCE_REPO=/mnt/AppStream

dnf reposync \
  --download-metadata --downloadcomps \
  --download-path "$PATH_TO_LOCAL_MIRROR" \
  --repofrompath iso,"$PATH_TO_SOURCE_REPO" \
  --repoid iso

# Useful EL8-only options: --remote-time --norepopath

dnf reposync should probably be the only kind of modular-capable mirroring that on-site tools like unpack_dvd should use.

DNF repoclosure

The ISO build process (really the tar build process) runs repoclosure to make sure the packages on the ISO will be self-contained.

dnf repoclosure --repofrompath iso,"$PWD" --repo appstream --repo baseos

Clean DNF install --downloadonly with all deps

Make sure you enable/disable the repos to match what you intend for resolution; --config is an option, too.

...

...

The %{ModularityLabel} header is in (module)name:stream:version:context:arch (N:S:V:C:A) format (on RedHat/CentOS/Fedora-built packages), so the result looks like this:

389-ds-base-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64   389-ds:1.4:8030020200831174107:618f7055
389-ds-base-devel-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64   389-ds:1.4:8030020200831174107:618f7055
389-ds-base-legacy-tools-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64   389-ds:1.4:8030020200831174107:618f7055
389-ds-base-libs-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64   389-ds:1.4:8030020200831174107:618f7055
389-ds-base-snmp-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64   389-ds:1.4:8030020200831174107:618f7055
HdrHistogram-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch   jmc:rhel8:8020020200731165725:21dc74c6
HdrHistogram-javadoc-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch   jmc:rhel8:8020020200731165725:21dc74c6
Judy-1.0.5-18.module_el8.1.0+217+4d875839.x86_64   mariadb:10.3:8010020191115015915:cdc1202b
ant-1.10.5-1.module_el8.0.0+47+197dca37.noarch   ant:1.10:8000020190624202340:f7e686af
ant-lib-1.10.5-1.module_el8.0.0+47+197dca37.noarch   ant:1.10:8000020190624202340:f7e686af
aopalliance-1.0-17.module_el8.0.0+39+6a9b6e22.noarch   maven:3.5:8000020190624140656:f7e686af
aopalliance-1.0-20.module_el8.3.0+568+0c23fd64.noarch   maven:3.6:8030020201104064112:a623df05

However, it isn’t possible to re-create specific module/streams based on the RPM’s ModularityLabel headers.

1. A stream’s full modulemd metadata is only found in the source repo’s *metadata.yaml.gz

2. The content of ModularityLabel is unique to a build, but its contents cannot be trusted.

Note that the repo2module tool just creates a default profile called everything.

Identify all unique modules/streams from a collection of RPM files

find "$DIR_WITH_RPMS" -name \*.rpm \
  -exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \
  | grep -v '^(none)' | sort | uniq -c \
  | sort -nk1,1 \
  | tee unique_rpm_module_streams.txt

Mirror the contents of a DNF repository, preserving all modules and package groups. 

The current state of CentOS 8 createrepo_c + modulemd-tools (EPEL8) allow us to:

(Works from CentOS 8.3 and CentOS 7.8, requires packages dnf and dnf-plugin-core )

An example of this mirroring a mounted CentOS 8.3 ISO’s AppStream repository:

PATH_TO_LOCAL_MIRROR=/path/to/Appstream
PATH_TO_SOURCE_REPO=/mnt/AppStream

dnf reposync \
  --download-metadata --downloadcomps \
  --download-path "$PATH_TO_LOCAL_MIRROR" \
  --repofrompath iso,"$PATH_TO_SOURCE_REPO" \
  --repoid iso

# Useful EL8-only options: --remote-time --norepopath

dnf reposync should probably be the only kind of modular-capable mirroring that on-site tools like unpack_dvd should use.

DNF repoclosure

The ISO build process (really the tar build process) runs repoclosure to make sure the packages on the ISO will be self-contained.

dnf repoclosure --repofrompath iso,"$PWD" --repo appstream --repo baseos

Clean DNF install --downloadonly with all deps

Make sure you enable/disable the repos to match what you intend for resolution; --config is an option, too.

DNF_DLONLY_TARGET_PACKAGE=httpd
DNF_DLONLY_INSTALL_ROOT=/root/fake-install-dir
DNF_DLONLY_PACKAGE_DIR=/root/downloadonly

dnf install --downloadonly \
  --setopt=install_weak_deps=False \
  --installroot="$DNF_DLONLY_INSTALL_ROOT" \
  --downloaddir="$DNF_DLONLY_PACKAGE_DIR" \
  --disablerepo=\* \
  --enablerepo=baseos \
  --enablerepo=appstream \
  --releasever=8 \
  "$DNF_DLONLY_TARGET_PACKAGE"

...

• Create a modular repo from packages that already have a common module header
  

  Status
  colour Yellow title WARNING

  The approach of taking NSVCA N:S:V:C:A from the RPM headers below is a.) incomplete and b.) cannot be relied upon to be accurate or available—the present—the ModularityLabel header can contain any String.

  

  Status
  colour Red title DO NOT PRODUCTIZE

  Use another means of obtaining this to obtain N:V:S:C:A data; preferably from the repo itself (the data is sourced from the source repository’s {XXXXX}-modules.yaml.gz file)
  

  Code Block

  # Get the ModularityLabel from the RPMs ########################################################## #### UPDATE: DO NOT USE OR PRODUCTIZE THIS TECHNIQUE ##### ########################################################## # Notes: # - All RPMs in the module must have a SINGLE and IDENTICAL ModularityLabel # - The only thing required of this String is that it is unique to RPMs # from other modules (and different versions/contexts of this module) find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \ | sort -u ### WARNING: the ModularityLabel headers in RPMs build by EL and EPEL #### are (currently) in N:S:V:C format by convention, but in ##### reality this string is arbitrary and cannot be relied upon ##### to reflect the actual source module's metadata. ##### The actual NVSCA/P data can *ONLY* be obtained from the ##### original repo's metadata MODULE_HEADER=nodejs:10:8020020200707141642:6a468ee4

...

• metadata MODULE_HEADER=nodejs:10:8020020200707141642:6a468ee4

Documentation

• https://docs.fedoraproject.org/en-US/modularity/

• https://dnf.readthedocs.io/en/latest/

• https://dnf-plugins-core.readthedocs.io/en/latest/

• https://sgallagh.wordpress.com/2019/08/14/sausage-factory-modules-fake-it-till-you-make-it/

  • Notes on building modules independently of Fedora’s hosted build system

• https://pagure.io/modularity/issue/141

  > * AGREED: Profiles cannot be removed during the lifetime of the stream in order to avoid breaking scripts. Options may exist for exceptional cases on an individual basis. (contyk, 15:41:06)

  https://docspagure.fedoraproject.org/en-US/modularity/

• https://dnf.readthedocs.io/en/latest/

• https://dnf-plugins-core.readthedocs.io/en/latest/

• https://sgallagh.wordpress.com/2019/08/14/sausage-factory-modules-fake-it-till-you-make-it/

  • Notes on building modules independently of Fedora’s hosted build system

• io/modularity/issue/141

  • > * AGREED: Profiles cannot be removed during the lifetime of the stream in order to avoid breaking scripts. Options may exist for exceptional cases on an individual basis. (contyk, 15:41:06)

Modularity direction

In early 2020, modularity development moved from Fedora’s Modularity WG to RedHat’s internal DNF team. Fedora development and governance was very open and public, which made it relatively easy to hunt down clarifying information about otherwise obscure details in pagure or the WG’s meeting logs.

Since RedHat took over, it has become very difficult to find up-to-date documentation or discussions, even (especially?) in BZ.

• Fedora Council and the future of Modularity (March 2020 blog post)

  • > “ Since then, responsibility for Modularity development in Red Hat moved to a new team. This presents a good opportunity to reset opinions and start anew. As the new team gets their arms around Modularity, you can expect an updated Objective proposal that will improve Modularity for Fedora. “

• RHEL 9 and modularity (18 June 2020, Fedora devel@ mailing list)

  • Discussion with official RedHat rep, after taking modularity from Fedora and moving its development to an internal DNF team.

  • There’s also a side-discussion in the middle about the “inevitability” of moving the modular “namespacing” (N:S?) into RPM’s headers/tags (like ModularityLabel), where several of Fedora’s former modular WG members chime in with some educational

    • Bonus: Stephen Gallagher on containers vs flatpak
      
      

More esoteric things that may come in handy in the future:

...