At some point between 08 July 2020 and 15 July 2020, Puppet 5 spec tests started failing for previously working commits:
Example failure from https://travis-ci.org/github/simp/pupmod-simp-compliance_markup/jobs/710437770#L544:
6.17.0 (also released on 2020-07-14) and the recently-released 6.18.0 are also broken. Tests pass on 6.16.0.
Attaching a working (6.16.0) and a broken (6.18.0) compliance_report.yaml for reference.
I'm narrowing down where the problem is.
At https://github.com/simp/pupmod-simp-compliance_markup/blob/7761d7d9425bad606b0cd4b880cbc8ac18f2727d/lib/puppetx/simp/compliance_mapper.rb#L431 on a working version of puppet, @profile_list looks like
but on a broken version of puppet, it looks like
It appears that setting the compliance profiles via the global $compliance_profile is what broke. Example here: https://github.com/simp/pupmod-simp-compliance_markup/blob/7761d7d9425bad606b0cd4b880cbc8ac18f2727d/spec/classes/init_spec.rb#L139
For reasons that I have yet to determine, the HIERA_compliance_data_dir environment variable is getting lost, which is breaking classes/init_spec.rb. I'm currently testing a refactor of the tests to avoid using the environment variable.