Page Comparison

...

Implementing slim modular repos

...

...

...

1. Status
   colour Purple title TODO

   Most BaseOS EL8 modules have a default stream; use that the default stream if it exists

2. Status
   colour Purple title TODO

   We could also If there is only a single stream, default to the onlystream.
   This is hacky, but it will work for EL8.3—Base OS (i.e., AppStream) modules without a default stream are currently very rare, and at the moment all of them have a single stream:

   Code Block

   # dnf module --disablerepo=\* --enablerepo=appstream list | grep -v '\[d\]' CentOS Linux 8 - AppStream Name Stream Profiles Summary Summary 389-ds 1.4 389-ds 389 Directory Server (base) 1.4 389 Directory Server (base) libselinux-python 2.8 common Python 2 bindings for libselinux libselinux-python 2.8 common Python 2 bindings for libselinux mod_auth_openidc 2.3 Apache module suporting OpenID Connect authentication mod_auth_openidc 2.3 Apache module suporting OpenID Connect authentication parfait 0.5 common Parfait Module parfait 0.5 common Parfait Module pki-core 10.6 PKI Core module for PKI 10.6 or later pki-core 10.6 PKIpki-deps Core module for PKI 10.6 or later 10.6 PKI Dependencies module for PKI 10.6 or later pki-deps 10.6 PKI Dependencies module for PKI 10.6 or later

   

3. 

   Status
   colour Blue title NOT IN 6.6.0

   Status
   colour Red title unsolved

   This leaves open a potential edge-case: if in the future, we require an RPM from a Base OS modules without a default stream but ships with multiple streams (again, current population: 0), it will fail and there is no way to hint

   • Status
     colour Blue title NOT IN 6

   .6.0

   Status
   colour Red title unsolved

   This leaves open the future edge-case where we include RPMs that will fail when we need an RPM from one of the Base OS modules above that have multiple streams but don’t have a default stream (current population: 0).

   • Status
     colour Blue title NOT IN 6.6.0

     We should probably have a way of formally declaring N:S for *pkglist.txt Base OS RPMs in the future. Some possibilities:

     • A separate *pkglist.modularity.txt file

     • N:S-declaring directives in the comments of *pkglist.txt

     • Could this be combined with packages.yaml? (not without a major rewrite)

Unsolved problems

What are the “Fetch RPM” flow differences between Base OS (prune_packages) & External (yumdownloader) packages?

When running yumdownloader, how can we get the RPM’s source repo’s modulemd data?

This is easy to do by hand for an individual package, but I’m not sure how to automate it yet. Here are some options:

1. Option 1: see if yumdownloader can be convinced to display it, like --urls (haven’t found it yet)

2. Option 2: walk up the dir tree until we find metadata (hacky, expensive)

3. Option 3: find/define the DNF cache and somehow fish out the modulemd data for that specific package

...

1. • .6.0

     We should probably have a way of formally declaring N:S for *pkglist.txt Base OS RPMs in the future. Some possibilities:

     • A separate *pkglist.modularity.txt file

     • N:S-declaring directives in the comments of *pkglist.txt

     • Could this be combined with packages.yaml? (not without a major rewrite)

Unsolved problems

What are the “Fetch RPM” flow differences between Base OS (prune_packages) & External (yumdownloader) packages?

https://simp-project.atlassian.net/browse/SIMP-9643

How can we know the URL/path to an RPM’s source repo’s repomd.xml file?

https://simp-project.atlassian.net/browse/SIMP-9644

This is simple enough to do by hand for an individual package, but I’m not sure how to automate it yet. Here are some ideas:

1. Option 1: see if yumdownloader can be convinced to display the repo root’s URL, like --urls does with the RPM

   • (I haven’t found an option that does this)

2. Option 2: walk up the dir tree until we find metadata

   • (hacky, expensive)

3. Option 3: (somehow) find/define the DNF cache that was used to download the RPM and (somehow) fish out the modulemd data that was used for that specific package

4. Status
   colour Green title Current favorite

   Option 4: Do everything the other way around:

   1. Before getting packages, get each representative repo’s repomd.xml file first, then use it to find the xxx-packages.yaml.gz

   2. read the modulemd data from the packages.yaml file

   3. filter the modulemd data down to just the streams and packages you need

   4. then run yumdownloader to acquire those exact packages

Separate yumdownloader runs may result in RPMs for the same N:S having different N:S:V:C:A

Different RPMs could be sourced from different versions (V:) of the same module stream if yumdownloader pulls them from different repo mirrors that are out of sync with each other. Using the heuristic of a “slim” module stream per unique N:S:V:C:A , this would result in multiple module streams instead of one.

This is a rare edge case that V: is specifically intended to catch, and it seems correct to fail instead of rebuild building a modular “mirrored” stream subset using RPMs from a different (stream) versions. However, I can’t really demonstrate that the potential impact of this scenario is worth prioritizing its implementation.

The strongest impacts I came up with so far rely on the fact that there’s a good chance that, between two stream versions, the combined set of downloaded RPMs won’t match be a precise subset of either stream exactly. But unless your the mirrors were really out of sync, this probably wouldn’t matter much. The stream version is a snapshot in time of all the modulemd (modular) metadata for the stream; it —it doesn’t actually affect the RPM’s resolutionits RPMs' resolutions.

(I honestly don’t know many details of how/when V: is used other than “highest wins”, but it might lead to weird edge cases. :)

1. There’s a (staggeringly) remote change that the newer stream version dropped package(s) or one of its packages has a new dependency

2. The slim repo will use one N:S:V or the other, but neither upstream precisely matches its RPMs. After re-integrating with the full upstream repo or mirror, DNF might miss an update by deciding it already know the stream version resolve using the wrong stream version for some of the packages, to the wrong versions, miss updates

3. There may be other reasons to do with inter-modular dependencies.

4. TL;DR: Not sure if failing is the best way forward—input welcome.

For the time being, I am treating this as a don’t-have-to-solve problem

Are there conditions where streams don’t provide C:A information when packages are noarch?

...