Questions
Assuming at least some of the packages involved belong to repo modules, how can/should we handle:
pkglist + packages.yaml => building tarball/ISO
Can the pkg/tar/ISO build process still create a usable ISO as it currently does, or will something have to change?
filtering an OS ISO down to its pkglist
Possible (but involved)
We can probably use dir2module to re-package just the modules we need and mergerepo_c to build them into a slimmed-down AppStream based on pkglist.txt (or a SIMP/ repo for packages.yml)
Details at the end of the RPM module header examples
adding external packages in packages.yaml
running repoclosure
Currently failing:
Currently, an AppStream repo mirrored with dnf reposync fails dnf repoclosure rather badly, while the real AppStream repo only has two unresolved deps (both ursine).
If it can be done: How?
If not: What alternatives are there?
Impacts to unpack_dvd & kickstart repositories
How should unpack_dvd handle repos that do/may contain modules?
Recommendation:
Assuming we can limit any module editing to the tarball/ISO build process (above), unpack_dvd should just mount the ISO as a loopback device and sync everything with dnf reposync --download-metadata … .
How does SELinux support this?
Where do the policies for the packages in various module streams come from?
Notes:
So far, the selinux-*policy RPMs have handled whatever I’ve tried
Ancient policy wiki at https://fedoraproject.org/wiki/SELinuxModularityDesign (probably outdated)
Supporting questions
Can we repackage a “thinned-out” module with only select packages?
probably:
EL8 RPM prereqs: createrepo_c (appstream) + modulemd-tools (epel)
Process:
# Create initial repo with ursine modules mkdir -p $NEW_REPO_DIR/Packages/ursine cp "${URSINE_PACKAGE_FILES[@]}" "$NEW_REPO_DIR/Packages/ursine/" cd "$NEW_REPO_DIR" createrepo_c . #
Modularity CLI examples
Simple dnf module commands
View all modules
dnf module list --all
View a module’s available streams
# dnf module list --available nodejs Waiting for process with pid 79896 to finish. Last metadata expiration check: 0:00:01 ago on Fri 12 Mar 2021 02:03:02 AM UTC. CentOS Linux 8 - AppStream Name Stream Profiles Summary nodejs 10 [d] common [d], development, minimal, s2i Javascript runtime nodejs 12 common [d], development, minimal, s2i Javascript runtime nodejs 14 common [d], development, minimal, s2i Javascript runtime Extra Packages for Enterprise Linux Modular 8 - x86_64 Name Stream Profiles Summary nodejs 13 default, development, minimal Javascript runtime Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
View packages in a module
dnf module repoquery nodejs
It looks dnf module repoquery only reports on the default stream, regardless of the module-spec specified (tested on EL8.3)
View modules/profiles that provide a package
dnf module provides nodejs
This is one of the few (two?) module commands that reports which DNF Repo hosts the module
View the packages in each profile (for a specific stream)
]# dnf module info --profile nodejs:12
Last metadata expiration check: 2:59:10 ago on Thu 11 Mar 2021 11:02:32 PM UTC.
Name : nodejs:12:8030020210304194546:30b713e6:x86_64
common : nodejs
: npm
development : nodejs
: nodejs-devel
: npm
minimal : nodejs
s2i : nodejs
: nodejs-nodemon
: npm
Find all modular RPM files that under a directory and print their module headers
This could identify dnf modules that would need to be re-created when shipping modular RPMs independently from their base repos:
find "$DIR_WITH_RPMS" -name \*.rpm \
-exec rpm -qp {} --qf '%{NVRA} %{ModularityLabel}\n' \; \
| grep -v '(none)' \
| tee modular_rpms.txt
The %{ModularityLabel} macro is returned in (module)name:stream:version:context:arch (NSVCA) format , so the result looks like this:
389-ds-base-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-devel-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-legacy-tools-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-libs-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 389-ds-base-snmp-1.4.3.8-5.module_el8.3.0+473+53682548.x86_64 389-ds:1.4:8030020200831174107:618f7055 HdrHistogram-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch jmc:rhel8:8020020200731165725:21dc74c6 HdrHistogram-javadoc-2.1.11-2.module_el8.2.0+460+6583c1d0.noarch jmc:rhel8:8020020200731165725:21dc74c6 Judy-1.0.5-18.module_el8.1.0+217+4d875839.x86_64 mariadb:10.3:8010020191115015915:cdc1202b ant-1.10.5-1.module_el8.0.0+47+197dca37.noarch ant:1.10:8000020190624202340:f7e686af ant-lib-1.10.5-1.module_el8.0.0+47+197dca37.noarch ant:1.10:8000020190624202340:f7e686af aopalliance-1.0-17.module_el8.0.0+39+6a9b6e22.noarch maven:3.5:8000020190624140656:f7e686af aopalliance-1.0-20.module_el8.3.0+568+0c23fd64.noarch maven:3.6:8030020201104064112:a623df05
It may be possible to re-create specific module/streams based on the RPM’s unique headers. However, there will need to be a second data source, because the RPM ModularityLabel has significant limitations:
By convention, RPMs' ModularityLabel string is in NVSC format, which doesn’t include profile information. Any module streams constructed solely on the RPM header would lose all the original streams' profiles.
Note that the repo2module tool just creates a default profile called everything.
We can’t rely on RPMs' ModularityLabel string to be in NVSC format, either!
“The ModularityLabel can be any string at all. In Fedora, we have a convention to use name:stream:version:context to indicate from which build the RPM originally came from, but this is not to be relied upon. It may change at any time and it also may not be accurately reflective of the module in which it currently resides, due to component-reuse in the Module Build System.”
— https://sgallagh.wordpress.com/2019/08/14/sausage-factory-modules-fake-it-till-you-make-it/
Identify all unique modules/streams from a collection of RPM files
find "$DIR_WITH_RPMS" -name \*.rpm \
-exec rpm -qp {} --qf '%{ModularityLabel}\n' \; \
| grep -v '^(none)' | sort | uniq -c \
| sort -nk1,1 \
| tee unique_rpm_module_streams.txt
Mirror the contents of a DNF repository, preserving all modules and package groups.
The current state of CentOS 8 createrepo_c + modulemd-tools (EPEL8) allow us to:
(Works from CentOS 8.3 and CentOS 7.8, requires packages dnf and dnf-plugin-core )
An example of this mirroring a mounted CentOS 8.3 ISO’s AppStream repository:
PATH_TO_LOCAL_MIRROR=/path/to/Appstream PATH_TO_SOURCE_REPO=/mnt/AppStream dnf reposync \ --download-metadata --downloadcomps \ --download-path "$PATH_TO_LOCAL_MIRROR" \ --repofrompath iso,"$PATH_TO_SOURCE_REPO" \ --repoid iso # Useful EL8-only options: --remote-time --norepopath
dnf reposync should probably be the only kind of modular-capable mirroring that on-site tools like unpack_dvd should use.
DNF repoclosure
The ISO build process (really the tar build process) runs repoclosure to make sure the packages on the ISO will be self-contained.
dnf repoclosure --repofrompath iso,"$PWD" --repo appstream --repo baseos
Judging by the bug report at https://bugzilla.redhat.com/show_bug.cgi?id=1547041, dnf repoclosure is not module-aware, in the sense that its resolver does not consider packages in all available modules/streams.
The current behavior only resolves module packages using default or enabled streams & profiles.
Clean DNF install --downloadonly with all deps, without needing mock
Make sure you enable/disable the repos to match what you intend for resolution; --config is an option, too.
DNF_DLONLY_TARGET_PACKAGE=httpd DNF_DLONLY_INSTALL_ROOT=/root/fake-install-dir DNF_DLONLY_PACKAGE_DIR=/root/downloadonly dnf install --downloadonly \ --setopt=install_weak_deps=False \ --installroot="$DNF_DLONLY_INSTALL_ROOT" \ --downloaddir="$DNF_DLONLY_PACKAGE_DIR" \ --enablerepo=appstream \ --disablerepo=test \ --releasever=8 \ "$DNF_DLONLY_TARGET_PACKAGE"
Pointing to an empty --installroot= will cause dnf install --downloadonly to download EVERY dependency, including the packages for the baseos.
In this case, --setopt=install_weak_deps=False may be useful to ignore weak RPM dependencies and cut down on the download size.
Documentation
https://sgallagh.wordpress.com/2019/08/14/sausage-factory-modules-fake-it-till-you-make-it/
Notes on building modules independently of Fedora’s hosted build system
https://pagure.io/modularity/issue/141
> * AGREED: Profiles cannot be removed during the lifetime of the stream in order to avoid breaking scripts. Options may exist for exceptional cases on an individual basis. (contyk, 15:41:06)