Network-isolated EL8 agents kickstarted from SIMP 6.5.0-Alpha fail on dnf updates (404 for RPM-GPG-KEY-CentOS-Official)
This was tested on an isolated network with an simp-packer-built EL7 puppetserver built from the latest SIMP 6.5.0-(pre)-Alpha ISO
The EL8 ISOs no longer distribute their packages' GPG signing key. The simp::yum::repo::local_os_updates class is currently hard-coded to look for the gpgkey in that location, and fails if it 404s (see details below)
On the EL8 agent, /etc/yum..repos.d/os_updates contains the reference to RPM-GPG-KEY-CentOS-Official:
Regarding the EL8 agents' /etc/yum..repos.d/os_updates failing reference to gpgkey=https://puppet.simp.test/yum/CentOS/8/x86_64/RPM-GPG-KEY-CentOS-Official:
The CentOS 8 ISO (in this case, CentOS-8.2.2004-x86_64-dvd1.iso) does not include the GPG key, as all previous CentOS releases have done. As a result, the RPM-GPG-KEY-CentOS-Official is missing, and all attempts to yum install packages from the os_updates repo on a newly-kickstarted EL8 system will fail with 404 errors while trying to download the gpgkey.
SIMP 6.5.0 (pre-)Alpha currently does not provide an easy way out of this predicament for newly kickstarted EL8 nodes.
The yum repo's gpgkey is managed by the class simp::yum::repo::local_os_updates, which automatically determines the URL to download the os_update's gpgkey. It makes several assumptions that do not work for EL8:
It assumes the GPG key(s) will exist on the yum server at a consistent path and using a particular file name convention (after running the unpack_dvd script to extract them from the OS ISO).
Problem 1: CentOS-8.2.2004-x86_64-dvd1.iso no longer provides the GPG keys on the ISO, so unpack_dvd won't provide them.
Problem 2: There is currently no way to specify an alternate URL to replace the automatically-generated URLs
The class will generate a redundate gpgkey entry for each (identical) yum server passed into the $servers parameter. Additional gpgkey URLs can be provided with the $extra_gpgkey_urls parameter.
Problem 3: The implementation assumed that multiple gpgkey entries were intended for redundancy, but in fact they are intended for repos that contain packages signed by various keys. If any gpgkey URL fails to download, the yum install will also fail when gpgcheck=1.
Fortunately, simp-gpgkeys already ships with the correct CentOS 8 GPG key. In this case, the 6.5.0 pre-Alpha puppetmaster serves the it from the URL http://puppet.simp.test/yum/SIMP/GPGKEYS/RPM-GPG-KEY-CentOS-8.
Problem 4: However, due to the previously-described problems, there is currently no way to configure simp::yum::repo::local_os_updates to use the correct URL instead of the automatically-generated (and bogus for EL8) gpgkey URL, so all os_upgrade yum installs on a freshly PXE-booted EL8 agent will unavoidably fail with 404 errors.