Page Comparison

Table of Contents

maxLevel	3

...

Note

80% Confidence it’s Possible — but it will be complicated and hacky

The current methods of downloading external packages AND pruning OS ISOs down to a minimal set packages will both result in broken modular repositories.
- This could be solved by recreating the RPM’s module streams with exactly the same NVSCA values as the source repository—but practically every step of that will be hacky and require unsupported tooling or several workarounds.
The repo manipulation methods (createrepo, repoclosure) aren’t module-aware

...

RPM packages from repo modules contain a ModularityLabel header that is unique to the module’s name + stream + version + context + architecture (NVSCA). DNF must install these RPMs from modules (with modular metadata)—and will refuse to install them as ursine packages.
- dir2module and repo2modules require a complete NVSCA string generating module metadata for a directory of RPMs.
- 5️⃣ The dir2module script provided by the EPEL8 RPM modulemd-tools-0.7-1.el8.noarch does not create a default (or any) profile for the module it creates; it is effectively useless; use repo2module instead (which is still missing the Arch in NVCSA).
A repackaged (slim) module’s name and stream and context and architecture must match the values in the source repo’s metadata for the original module.
- This is required so the packages maintain continuity with the complete upstream repo (e.g., receiving update from the complete AppStream repo, epel-modular, etc)
- The versionnumber must evaluate to more than the earlier (module versions) and less than the version of later modules in the full source repository. This number (like stream and context) is an arbitrary string set by the build platform, so we have to get it
- For practical purposes, you need to mirror the repo’s metadata YAML at the same time as you retrieve the packages—it might be updated later, even if the packages you see hosted there are the same
The ModularityLabel header is string unique to a module builds' NVSCfor a particular platform. The header data in RPMs packaged by RHEL/CentOS build system looks useful because a string it’s in NVSC format, however this data is actually arbitrary and cannot be relied uponto provide accurate NVSC data for the module.
The only canonical source for a module’s correct NVSCA/P data is the source repo’s metadata (generally under repodata/{XXXXX}-modules.yaml.gz, and defined by repodata/repomd.xml under <data type="modules">`)
🎉 I’ve tested a repacked “slim” repo alongside a repo with the full module w/identical NVSCA details, and it successfully resolved its metadata/packages with the full module.
- This should also behave correctly with updated modules, but I haven’t been able to stage that yet.

...

Tip

Recommendation: The unpack_dvd tool should assume that all ISO DNF repositories are complete enough to mirror.

Prerequisites:

Any necessary dnf module surgery MUST be completed during the tarball/ISO’s build process (described above)
Any “slimmed” dnf modules SHOULD be upgrade-compatible with an unpacked repository than contains the complete modules

If these recommendations are implemented, then a site’s unpack_dvd tool should only need to copy/reposync the files on the ISO.

This has been tested by mounting an ISO as a loopback device and syncing everything (including module metadata) with dnf reposync --download-metadata […] .

On a stock EL8 server, dnf reposync requires the dnf-plugins-core package
On a stock EL7 server, dnf reposync requires the dnf and dnf-plugins-core packages

How does SELinux support this?

Where do the policies for the packages in various module streams come from?

Info

Notes:

So far, the selinux-*policy RPMs have handled whatever I’ve tried
Ancient policy wikis at (last updated late 2018):
- SELinux Policy: https://fedoraproject.org/wiki/SELinuxModularityPackagingGuidance#Packaging_SELinux_Policy
- Design docs: https://fedoraproject.org/wiki/SELinuxModularityDesign

Notes from research

Can we repackage a “thinned-out” module with only select packages?

probably:

...

EL8 RPM prereqs: createrepo_c (appstream) + modulemd-tools (epel)

Process:

...

Create initial repo with ursine modules

Code Block
mkdir -p $NEW_REPO_DIR/Packages/ursine cp "${URSINE_PACKAGE_FILES[@]}" "$NEW_REPO_DIR/Packages/ursine/" cd "$NEW_REPO_DIR" createrepo_c .

Create a modular repo from packages that already have a common module header

Status

colour	Yellow
title	WARNING

The approach of taking NSVCA from the RPM headers is incomplete and cannot be relied upon to be accurate or available—the ModularityLabel header can contain any String
Status
colour Red
title DO NOT PRODUCTIZE
Use another means of obtaining this data; preferably from the repo itself (the data is sourced from the source repository’s {XXXXX}-modules.yaml.gz file)
...
Modularity CLI examples
Simple dnf module commands
View all modules
Code Block
dnf module list --all
View a module’s available streams
Code Block
breakoutMode wide
# dnf module list --available nodejs Waiting for process with pid 79896 to finish. Last metadata expiration check: 0:00:01 ago on Fri 12 Mar 2021 02:03:02 AM UTC. CentOS Linux 8 - AppStream Name Stream Profiles Summary nodejs 10 [d] common [d], development, minimal, s2i
...

...
Javascript
...
runtime
...
nodejs
...

...

...

...

...

...

...

...

...
12
...

...
Modularity CLI examples
Simple dnf module commands
View all modules
Code Block
dnf module list --all
View a module’s available streams
Code Block
breakoutMode wide
# dnf module list --available nodejs Waiting for process with pid 79896 to finish. Last metadata expiration check: 0:00:01 ago on Fri 12 Mar 2021 02:03:02 AM UTC. CentOS Linux 8 - AppStream Name common [d], development, minimal, s2i Javascript runtime nodejs Stream14 Profiles common [d], development, minimal, s2i Javascript runtime Extra Packages for Enterprise Linux Modular 8 - x86_64 Name Summary nodejs Stream 10 [d] Profiles common [d], development, minimal, s2i Javascript runtime nodejs Summary nodejs 12 13 common [d], development, minimal, s2i default, development, minimal Javascript runtime nodejs 14 Javascript runtime Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
View packages in a module
Code Block
dnf module repoquery nodejs
Info
It looks dnf module repoquery only reports on the default stream, regardless of the module-spec specified (tested on EL8.3)
View modules/profiles that provide a package
Code Block
dnf module common [d], development, minimal, s2i Javascript runtime Extra Packages for Enterprise Linux Modular 8 - x86_64 provides nodejs
Info
This is one of the few (two?) module commands that reports which DNF Repo hosts the module
View the packages in each profile (for a specific stream)
Code Block
language s
]# dnf module info --profile nodejs:12 Last metadata expiration check: 2:59:10 ago on Thu 11 Mar 2021 11:02:32 PM UTC. Name : Stream nodejs:12:8030020210304194546:30b713e6:x86_64 common : nodejs Profiles : npm development : nodejs : nodejs-devel : npm minimal Summary: nodejs s2i : nodejs 13 : nodejs-nodemon default,: development, minimal Javascript runtime Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
View packages in a module
Code Block
dnf module repoquery nodejs
Info
It looks dnf module repoquery only reports on the default stream, regardless of the module-spec specified (tested on EL8.3)
View modules/profiles that provide a package
Code Block
dnf module provides nodejs
Info
This is one of the few (two?) module commands that reports which DNF Repo hosts the module
View the packages in each profile (for a specific stream)
Code Block
language s
]# dnf module info --profile nodejs:12 Last metadata expiration check: 2:59:10 ago on Thu 11 Mar 2021 11:02:32 PM UTC. Name : nodejs:12:8030020210304194546:30b713e6:x86_64 common : nodejs : npm development : nodejs : nodejs-devel : npm minimal : nodejs s2i : nodejs : nodejs-nodemon : npm
...
Find all modular RPM files that under a directory and print their module headers
This could identify dnf modules that would need to be re-created when shipping modular RPMs independently from their base repos:
Code Block
language bash
find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{NVRA} %{ModularityLabel}\n' \; \ | grep -v '(none)' \ | tee modular_rpms.txt
The %{ModularityLabel} macro is returned in (module)name:stream:version:context:arch (NSVCA) format , so the result looks like this:
Code Block
389-ds-base-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-devel-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-legacy-tools-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-libs-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-snmp-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 HdrHistogram-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch jmc:rhel8:8020020200731165725:21dc74c6 HdrHistogram-javadoc-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch jmc:rhel8:8020020200731165725:21dc74c6 Judy-1.0.5-18.module_el8.1.0+217+4d875839.x86_64 mariadb:10.3:8010020191115015915:cdc1202b ant-1.10.5-1.module_el8.0.0+47+197dca37.noarch ant:1.10:8000020190624202340:f7e686af ant-lib-1.10.5-1.module_el8.0.0+47+197dca37.noarch ant:1.10:8000020190624202340:f7e686af aopalliance-1.0-17.module_el8.0.0+39+6a9b6e22.noarch maven:3.5:8000020190624140656:f7e686af aopalliance-1.0-20.module_el8.3.0+568+0c23fd64.noarch maven:3.6:8030020201104064112:a623df05
It may be possible to re-create specific module/streams based on the RPM’s unique headers. However, there will need to be a second data source, because the RPM ModularityLabel has significant limitations:
Note
By convention, RPMs' ModularityLabel string is in NVSC format, which doesn’t include profile information. Any module streams constructed solely on the RPM header would lose all the original streams' profiles.
Note that the repo2module tool just creates a default profile called everything.
...
We can’t rely on RPMs' ModularityLabel string to be in NVSC format, either!
...
Identify all unique modules/streams from a collection of RPM files
Code Block
find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \ | grep -v '^(none)' | sort | uniq -c \ | sort -nk1,1 \ | tee unique_rpm_module_streams.txt
...
Mirror the contents of a DNF repository, preserving all modules and package groups.
The current state of CentOS 8 createrepo_c + modulemd-tools (EPEL8) allow us to:
(Works from CentOS 8.3 and CentOS 7.8, requires packages dnf and dnf-plugin-core )
An example of this mirroring a mounted CentOS 8.3 ISO’s AppStream repository:
Code Block
language bash
PATH_TO_LOCAL_MIRROR=/path/to/Appstream PATH_TO_SOURCE_REPO=/mnt/AppStream dnf reposync \ --download-metadata --downloadcomps \ --download-path "$PATH_TO_LOCAL_MIRROR" \ --repofrompath iso,"$PATH_TO_SOURCE_REPO" \ --repoid iso # Useful EL8-only options: --remote-time --norepopath
dnf reposync should probably be the only kind of modular-capable mirroring that on-site tools like unpack_dvd should use.
DNF repoclosure
The ISO build process (really the tar build process) runs repoclosure to make sure the packages on the ISO will be self-contained.
Code Block
dnf repoclosure --repofrompath iso,"$PWD" --repo appstream --repo baseos
Info
Judging by the bug report at https://bugzilla.redhat.com/show_bug.cgi?id=1547041, dnf repoclosure is not module-aware, in the sense that its resolver does not consider packages in all available modules/streams.
The current behavior only resolves module packages using default or enabled streams & profiles.
Clean DNF install --downloadonly with all deps
Make sure you enable/disable the repos to match what you intend for resolution; --config is an option, too.
Code Block
language bash
DNF_DLONLY_TARGET_PACKAGE=httpd DNF_DLONLY_INSTALL_ROOT=/root/fake-install-dir DNF_DLONLY_PACKAGE_DIR=/root/downloadonly dnf install --downloadonly \ --setopt=install_weak_deps=False \ --installroot="$DNF_DLONLY_INSTALL_ROOT" \ --downloaddir="$DNF_DLONLY_PACKAGE_DIR" \ --disablerepo=\* \ --enablerepo=baseos \ --enablerepo=appstream \ --releasever=8 \ "$DNF_DLONLY_TARGET_PACKAGE"
Notes:
...
An an --enablerepo= for each repo to consider during resolution.
...
Pointing to an empty --installroot= will cause dnf install --downloadonly to download EVERY dependency, including the packages for the baseos.
...
npm
Anchor
module-headers
module-headers
Find all modular RPM files that under a directory and print their module headers
This could identify dnf modules that would need to be re-created when shipping modular RPMs independently from their base repos:
Code Block
language bash
find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{NVRA} %{ModularityLabel}\n' \; \ | grep -v '(none)' \ | tee modular_rpms.txt
The %{ModularityLabel} macro is returned in (module)name:stream:version:context:arch (NSVCA) format , so the result looks like this:
Code Block
389-ds-base-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-devel-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-legacy-tools-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-libs-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-snmp-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 HdrHistogram-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch jmc:rhel8:8020020200731165725:21dc74c6 HdrHistogram-javadoc-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch jmc:rhel8:8020020200731165725:21dc74c6 Judy-1.0.5-18.module_el8.1.0+217+4d875839.x86_64 mariadb:10.3:8010020191115015915:cdc1202b ant-1.10.5-1.module_el8.0.0+47+197dca37.noarch ant:1.10:8000020190624202340:f7e686af ant-lib-1.10.5-1.module_el8.0.0+47+197dca37.noarch ant:1.10:8000020190624202340:f7e686af aopalliance-1.0-17.module_el8.0.0+39+6a9b6e22.noarch maven:3.5:8000020190624140656:f7e686af aopalliance-1.0-20.module_el8.3.0+568+0c23fd64.noarch maven:3.6:8030020201104064112:a623df05
It may be possible to re-create specific module/streams based on the RPM’s unique headers. However, there will need to be a second data source, because the RPM ModularityLabel has significant limitations:
Note
By convention, RPMs' ModularityLabel string is in NVSC format, which doesn’t include profile information. Any module streams constructed solely on the RPM header would lose all the original streams' profiles.
Note that the repo2module tool just creates a default profile called everything.
Note
We can’t rely on RPMs' ModularityLabel string to be in NVSC format, either!
“The ModularityLabel can be any string at all. In Fedora, we have a convention to use name:stream:version:context to indicate from which build the RPM originally came from, but this is not to be relied upon. It may change at any time and it also may not be accurately reflective of the module in which it currently resides, due to component-reuse in the Module Build System.”

— https://sgallagh.wordpress.com/2019/08/14/sausage-factory-modules-fake-it-till-you-make-it/
Identify all unique modules/streams from a collection of RPM files
Code Block
find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \ | grep -v '^(none)' | sort | uniq -c \ | sort -nk1,1 \ | tee unique_rpm_module_streams.txt
Anchor
reposync
reposync
Mirror the contents of a DNF repository, preserving all modules and package groups.
The current state of CentOS 8 createrepo_c + modulemd-tools (EPEL8) allow us to:
(Works from CentOS 8.3 and CentOS 7.8, requires packages dnf and dnf-plugin-core )
An example of this mirroring a mounted CentOS 8.3 ISO’s AppStream repository:
Code Block
language bash
PATH_TO_LOCAL_MIRROR=/path/to/Appstream PATH_TO_SOURCE_REPO=/mnt/AppStream dnf reposync \ --download-metadata --downloadcomps \ --download-path "$PATH_TO_LOCAL_MIRROR" \ --repofrompath iso,"$PATH_TO_SOURCE_REPO" \ --repoid iso # Useful EL8-only options: --remote-time --norepopath
dnf reposync should probably be the only kind of modular-capable mirroring that on-site tools like unpack_dvd should use.
DNF repoclosure
The ISO build process (really the tar build process) runs repoclosure to make sure the packages on the ISO will be self-contained.
Code Block
dnf repoclosure --repofrompath iso,"$PWD" --repo appstream --repo baseos
Info
Judging by the bug report at https://bugzilla.redhat.com/show_bug.cgi?id=1547041, dnf repoclosure is not module-aware, in the sense that its resolver does not consider packages in all available modules/streams.
The current behavior only resolves module packages using default or enabled streams & profiles.
Clean DNF install --downloadonly with all deps
Make sure you enable/disable the repos to match what you intend for resolution; --config is an option, too.
Code Block
language bash
DNF_DLONLY_TARGET_PACKAGE=httpd DNF_DLONLY_INSTALL_ROOT=/root/fake-install-dir DNF_DLONLY_PACKAGE_DIR=/root/downloadonly dnf install --downloadonly \ --setopt=install_weak_deps=False \ --installroot="$DNF_DLONLY_INSTALL_ROOT" \ --downloaddir="$DNF_DLONLY_PACKAGE_DIR" \ --disablerepo=\* \ --enablerepo=baseos \ --enablerepo=appstream \ --releasever=8 \ "$DNF_DLONLY_TARGET_PACKAGE"
Notes:
An an --enablerepo= for each repo to consider during resolution.
Pointing to an empty --installroot= will cause dnf install --downloadonly to download EVERY dependency, including the packages for the baseos.
In this case, --setopt=install_weak_deps=False may be useful to ignore weak RPM dependencies and cut down on the download size.
Create initial repo with ursine modules
Code Block
mkdir -p $NEW_REPO_DIR/Packages/ursine cp "${URSINE_PACKAGE_FILES[@]}" "$NEW_REPO_DIR/Packages/ursine/" cd "$NEW_REPO_DIR" createrepo_c .
Create a modular repo from packages that already have a common module header
Status
colour Yellow
title WARNING
The approach of taking NSVCA from the RPM headers below is a.) incomplete and b.) cannot be relied upon to be accurate or available—the ModularityLabel header can contain any String
Status
colour Red
title DO NOT PRODUCTIZE
Use another means of obtaining this data; preferably from the repo itself (the data is sourced from the source repository’s {XXXXX}-modules.yaml.gz file)
Code Block
# Get the ModularityLabel from the RPMs ########################################################## #### UPDATE: DO NOT USE OR PRODUCTIZE THIS TECHNIQUE ##### ########################################################## # Notes: # - All RPMs in the module must have a SINGLE and IDENTICAL ModularityLabel # - The only thing required of this String is that it is unique to RPMs # from other modules (and different versions/contexts of this module) find "$DIR_WITH_RPMS" -name \*.rpm \ -exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \ | sort -u ### WARNING: the ModularityLabel headers in RPMs build by EL and EPEL #### are (currently) in N:S:V:C format by convention, but in ##### reality this string is arbitrary and cannot be relied upon ##### to reflect the actual source module's metadata. ##### The actual NVSCA/P data can *ONLY* be obtained from the ##### original repo's metadata MODULE_HEADER=nodejs:10:8020020200707141642:6a468ee4
Documentation
https://docs.fedoraproject.org/en-US/modularity/
https://dnf.readthedocs.io/en/latest/
https://dnf-plugins-core.readthedocs.io/en/latest/
https://sgallagh.wordpress.com/2019/08/14/sausage-factory-modules-fake-it-till-you-make-it/
Notes on building modules independently of Fedora’s hosted build system
https://pagure.io/modularity/issue/141
> * AGREED: Profiles cannot be removed during the lifetime of the stream in order to avoid breaking scripts. Options may exist for exceptional cases on an individual basis. (contyk, 15:41:06)

Versions Compared

Old Version 16

New Version 17

Key

How does SELinux support this?

Notes from research

Modularity CLI examples

Simple dnf module commands

View all modules

View a module’s available streams

Modularity CLI examples

Simple dnf module commands

View all modules

View a module’s available streams

View packages in a module

View modules/profiles that provide a package

View the packages in each profile (for a specific stream)

View packages in a module

View modules/profiles that provide a package

View the packages in each profile (for a specific stream)

Find all modular RPM files that under a directory and print their module headers

Identify all unique modules/streams from a collection of RPM files

Mirror the contents of a DNF repository, preserving all modules and package groups.

DNF repoclosure

Clean DNF install --downloadonly with all deps

Find all modular RPM files that under a directory and print their module headers

Identify all unique modules/streams from a collection of RPM files

Mirror the contents of a DNF repository, preserving all modules and package groups.

DNF repoclosure

Clean DNF install --downloadonly with all deps

Create initial repo with ursine modules

Documentation