CentOS8 ISOs no longer distribute GPG keys; cause EL8 dnf failures


Network-isolated EL8 agents kickstarted from SIMP 6.5.0-Alpha fail on dnf updates (404 for RPM-GPG-KEY-CentOS-Official)

Example failure:

This was tested on an isolated network with an simp-packer-built EL7 puppetserver built from the latest SIMP 6.5.0-(pre)-Alpha ISO


The EL8 ISOs no longer distribute their packages' GPG signing key. The simp::yum::repo::local_os_updates class is currently hard-coded to look for the gpgkey in that location, and fails if it 404s (see details below)

Acceptance Criteria



Chris Tessmer
October 13, 2020, 4:22 PM

On the EL8 agent, /etc/yum..repos.d/os_updates contains the reference to RPM-GPG-KEY-CentOS-Official:

Chris Tessmer
October 13, 2020, 4:27 PM

https://www.centos.org/keys/RPM-GPG-KEY-CentOS-Official is identical to simp/gpgkeys's RPM-GPG-KEY-CentOS-8 file

Chris Tessmer
October 13, 2020, 7:32 PM

Regarding the EL8 agents' /etc/yum..repos.d/os_updates failing reference to gpgkey=https://puppet.simp.test/yum/CentOS/8/x86_64/RPM-GPG-KEY-CentOS-Official:

The CentOS 8 ISO (in this case, CentOS-8.2.2004-x86_64-dvd1.iso) does not include the GPG key, as all previous CentOS releases have done. As a result, the RPM-GPG-KEY-CentOS-Official is missing, and all attempts to yum install packages from the os_updates repo on a newly-kickstarted EL8 system will fail with 404 errors while trying to download the gpgkey.

Chris Tessmer
October 13, 2020, 8:02 PM

SIMP 6.5.0 (pre-)Alpha currently does not provide an easy way out of this predicament for newly kickstarted EL8 nodes.

The yum repo's gpgkey is managed by the class simp::yum::repo::local_os_updates, which automatically determines the URL to download the os_update's gpgkey. It makes several assumptions that do not work for EL8:

  1. It assumes the GPG key(s) will exist on the yum server at a consistent path and using a particular file name convention (after running the unpack_dvd script to extract them from the OS ISO).

    • Problem 1: CentOS-8.2.2004-x86_64-dvd1.iso no longer provides the GPG keys on the ISO, so unpack_dvd won't provide them.

    • Problem 2: There is currently no way to specify an alternate URL to replace the automatically-generated URLs

  2. The class will generate a redundate gpgkey entry for each (identical) yum server passed into the $servers parameter. Additional gpgkey URLs can be provided with the $extra_gpgkey_urls parameter.

    • Problem 3: The implementation assumed that multiple gpgkey entries were intended for redundancy, but in fact they are intended for repos that contain packages signed by various keys. If any gpgkey URL fails to download, the yum install will also fail when gpgcheck=1.

Fortunately, simp-gpgkeys already ships with the correct CentOS 8 GPG key. In this case, the 6.5.0 pre-Alpha puppetmaster serves the it from the URL http://puppet.simp.test/yum/SIMP/GPGKEYS/RPM-GPG-KEY-CentOS-8.

  • Problem 4: However, due to the previously-described problems, there is currently no way to configure simp::yum::repo::local_os_updates to use the correct URL instead of the automatically-generated (and bogus for EL8) gpgkey URL, so all os_upgrade yum installs on a freshly PXE-booted EL8 agent will unavoidably fail with 404 errors.

Epic Link


Story Points




Chris Tessmer