When testing upgrades from SIMP 6.4.0 to SIMP 6.5.0-Alpha, I encountered an unexpected issue in which the puppet agent and yum update cron jobs were not disabled and the update to Puppet 6 was made with a puppet agent cron run. This happened because I had already unpacked the SIMP 6.5.0-Alpha DVD and remade the yum cache.
The general upgrade docs should tell users to disable the puppet agent and yum update cron jobs prior to upgrading and then re-enable them afterwards. They should also have explicit instructions on what to set in Hieradata if you need to pin the version of Puppet to 5.x.
simp::yum::schedule::enable: false will disable the yum cron job
Can temporarily (4 hours?) disable puppet agent from running with puppet agent --disable. Might be a FAQ or HOWTO related to this.
An oddity that I saw in an EL6 upgrade was that after the yum update of the puppetserver and puppetdb RPMs, the previously running puppetdb and puppetserver services could both in dead states:
puppetdb is dead but pid file exists
puppetserver dead but subsys locked
Occurred in 2 out of the 3 times I did the upgrade.
It would be helpful if the instructions told the users to start the puppetdb service, wait for the puppetdb service to accept connections, start the puppetserver service, and then wait for the puppetserver service to accept connections. Otherwise, the next puppet agent run they attempt will necessarily fail (with ugly messages that may not make sense to users who are new to Puppet).
Decided that documenting puppet agent disable/enable was sufficient for the unexpected update issue.
WRT to pinning puppet application versions, can pin puppetserver and puppet-agent versions with hieradata. Cannot do so for puppetdb version. Unclear if we should bother and document this since it will be a pain to pin the puppetdb version. Maybe we let the sysadmins remove the upgraded versions from the yum repos on their own.
WRT to EL6, not really supporting EL6 SIMP server. So, may not be worth documenting this.