...
| Component | Required Tests | Test | EL7 | EL6 | OL7 | OL6 | RH7 | RH6 | Status | Notes | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | simp-core | Verify Puppet File matches release versions | This must be completed before any other tests are done
| ||||||||||||||||||||||||||||
| 2 | Puppet 5/hiera 5 | 1 | For all core components, including simp-core:
| SIMP-5529 |
| ||||||||||||||||||||||||||
| 3 | Puppet 5/Hiera 5 | 2 | For all core components: verify full matrix acceptance tests pass. |
|  | | |  | |
| |||||||||||||||||||||
| 4 | simp-core | Verify puppetfile and dependencies.yaml by building ISO | | | | | | |
| ||||||||||||||||||||||
| 5 | simp-packer | 4 | Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS | | | | | | |
| |||||||||||||||||||||
| 6 | Verify installation from RPM. This simp-core test also verifies
| | | | | | | This is a normal release testing activity. In simp-core run using the tar file generated during ISO build set BEAKER_release_tarball=<location of the tarball for el7>
set BEAKER_release_tarball=<location of the tarball for el6>
| |||||||||||||||||||||||
| 7 | simp-core | Verify installation from RPM in FIPS This test also verifies
| | | | | | | This is a normal release testing activity. For all tests set BEAKER_fips=yes set BEAKER_release_tarball=<location of the tarball for el7>
set BEAKER_release_tarball=<location of the tarball for el6>
| ||||||||||||||||||||||
| 8 | installation | Verify installation from RPM for PE | | | | |  | | RedHat built on AWS? simp cli was updated make sure you test out simp cli
| ||||||||||||||||||||||
| 9 | tlog, mate, x2go, ima | 5 | Verify packages.yaml files have all dependencies for new modules by installing these from a system built by an ISO with no internet connectivity. | | | | | | | RedHat built on AWS?
| |||||||||||||||||||||
| 10 | tpm2 | 5 | Verify packages.yaml files have all dependencies by installing these from a system built by an ISO with no internet connectivity. | | | | | | | RedHat on AWS? These modules don't work in 6 or on OracleLinux. I think testing in CentOS 7 is good enough.
| |||||||||||||||||||||
| 11 | compliance_markup | Verify turning on compliance enforcement works. Review compliance report. | | | This addresses STIG updates and Compliance Markup modules changes. Should we run an oscap scan as well?
| ||||||||||||||||||||||||||
| 12 | simp-core | Verify Create and verify Changelog.rst | 13 | simp-doc, | '
| ||||||||||||||||||||||||||
| 13 | simp-doc, | 'Upgrading SIMP' has an entry for the latest version and is correct | See 6.3 Planning page in Jira
| ||||||||||||||||||||||||||||
| 14 | simp-doc, tpm2, ima | Review TPM documentation to make sure it is up to date | TPM and IMA were separated out for this release.
| ||||||||||||||||||||||||||||
| 15 | simp-doc, tlog | Review Session Auditing and links in read the docs to make sure TLOG documentation is correct. | TLOG was added for this release
| ||||||||||||||||||||||||||||
| 16 | simp-doc, x2go, mate | Review How to manage Workstation entry for updates for mate and x2go | X2go, Mate were added for this release
| ||||||||||||||||||||||||||||
| 17 | simp-doc, hiera 5 and puppet 5 upgrade | Review docs to make sure the hiera data directory has been updated to reflect changes. And that the version of puppet currently supported has been updated to puppet 5. | hiera and puppet updated to version 5.
| ||||||||||||||||||||||||||||
| 18 | simp-integration_test | SIMP server upgrade from RPM using the upgrade instructions. Document any steps that are needed outside the ordinary upgrade. Then, if there are additional steps, have someone else use the docs to do the upgrade to verify. |
| |
| |
| |
| See simp-integration_test for automated upgrade test. this should be done for servers in both FIPS and NOFIPS mode. A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting. (This is
The fix is procedural: upgrade steps (and fixes) in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure Pay attention to Auditd upgrade because it is a major version upgrade. | |||||||||||||||||||||
| 19 | PE | Upgrade PE SIMP server from 6.2 puppet 4 to puppet 5 (PE 2018.1) and simp 6.3 |
| | | | Not sure what OS it is most important to test this on.
| ||||||||||||||||||||||||
| 20 | RPMs and Released Modules | Or "drink your own home brew", which ever you prefer.
| |||||||||||||||||||||||||||||
| 21 | Final RPM testing | RPMs have been uploaded to packagecloud | Run the simp-core install_from_rpm test and versify that it passes. This will pull the packages from package.io and do an install. | | | | | in simp-core run
and then do it all again with BEAKER_fips=yes After modules have
| 22 | Puppet Forge Testing |
| ||||||||||||||||||||
| 22 | Puppet Forge Testing | After modules have been tag with an annotated tag | run the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge | | | |
| in simp-core run
and then do it all again with BEAKER_fips=yes |
Tests - Regression/Integration Testing
...
|
Tests - Regression/Integration Testing
The tests in this section are regression tests to ensure unmodified capabilities still function. These tests should use the packer boxes created in the New/Changed Feature tests, were applicable. Note that most of these test should eventually be automated by simp-packer, simp-integration_test, or simp-core tests.
| Test | EL6 | EL7 | Notes | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Verify non-standard BIOS boot options from the ISO:
| (SIMP, FIPS and Encryption are all tested by Packer.)
| |||||||||||||||
| 2 | Verify all boot Options in UEFI mode from ISO:
|
| |||||||||||||||
| 3 | PXE Boot Testing BIOS |
| |||||||||||||||
| 4 | PXE Boot Testing UEFI |
| |||||||||||||||
| 5 | Bootstrap simp-lite scenario |
| |||||||||||||||
| 6 | Verify simp-lite operations:
| 78 | Bootstrap run-once scenario
| Bootstrap poss scenario |
| ||||||||||||
| 7 | Bootstrap poss scenario |
| |||||||||||||||
| 8 | Bootstrap run-once scenario |
| |||||||||||||||
| 9 | Verify ability to switch FIPS from on to off | Switching from off to on is known to have problems because of ciphers.
| |||||||||||||||
| 10 | Verify ability to turn on and off auditing and selinux |
| |||||||||||||||
| 11 | General Review of documentation:
| ||||||||||||||||
| 12 | simp-utils: executables that are not tested otherwise work as advertised
| The following have acceptance tests that simply need to be executed:
|
...