6.3.0 Testing Matrix

Versions for gitlab and travis that release 6.3 modules will be tested with:

Release	Puppet	Ruby	Notes
PE 2017.2	4.10.*	2.1.9	SIMP 6.1.0, SIMP 6.2.0 also tested on this version
PE 2017.3.6	5.3.6	2.4.4
PE 2018.1	5.5.6	2.4.4

OS Versions For Testing:

CentOS 7.5
CentOS 6.10
RedHat 7.5
RedHat 6.10
OracleLinux 7.5
OracleLinux 6.10

- Means this version needs to be tested

- This version does not need to be tested.

- Testing probably sufficient from other versions.

- Testing has failed.

- Testing completed and Passed.

blank - This test is not OS specific

	Component	Required Tests	Test	Status	Notes
1	simp-core		Verify Puppet File matches release versions		This must be complete before any other tests are done
2	Puppet 5/hiera 5	1	For all core modules: Verify that gitlab-ci.yml is running correct testing matrix (see above) and that puppet 5 tests are no longer "allow failure" . Make sure BEAKER_PUPPET_COLLECTION: 'puppet5' is set for puppet 5 tests.		gitlab-ci.yml sample gitlab-ci.yml This addresses Puppet 5 part of 6.3
3	Puppet 5/Hiera 5	2	For all core modules: verify full matrix acceptance tests pass.		The test by default will run only default. This addresses puppet 5/hiera 5 update part of 6.3. Need to run beaker:suites[ALL,ALL] for pupmod-simp-* modules. Perhaps we could create and entry in gitlab-cm.yml called "Release" That will run this.
4	simp-core		verify puppetfile and dependencies.yaml by building ISO		Should result in ISO for download for Centos. Just verify it can be built for Redhat Genral release testing.
5	simp-packer	4	Verify installtion from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted		Should result in vagrant 6 vagrant boxes for testsing. General Release
6	installation		verify installation from RPM		General release. This test is accomplished by simp-core acceptance test install_from_rpm for CentOS 6&7. Can we expand the node set to OEL 6&7?
7	installation		verify installation from RPM in FIPS		General release. This test is accomplished by simp-core acceptance test install_from_rpm for CentOS 6&7. Can we expand the node set to OEL 6&7?
8	installation		verify installation from RPM for PE		Redhat built on AWS? simp cli was updated make sure you test out simp cli
9	tlog, mate, x2go, ima	5	Verify dependencies.yaml has all dependencies for new modules by installing these from a system built by an iso with no internet connectivity.		Redhat built on AWS?
10	tpm2	5	Verify dependencies.yaml has all dependencies by installing these from a system built by an iso with no internet connectivity.		Redhat on AWS? These modules don't work in 6 or on OracleLinux. I think testing in CentOS 7 is good enough.
11	compliance_markup		Verify turning on compliance enforcement works. review compliance report.		This addresses stig updates and Compliance Markup modules changes.
12	simp-core		Verify Changelog.rst
13	simp-doc,		'Upgrading SIMP' has an entry for the latest version and is correct		See 6.3 Planning page in JIRA
14	simp-doc, tpm2, ima		Review TPM documentation to make sure it is up to date		TPM and IMA were seperated out for this release.
15	simp-doc, tlog		Review Session Auditing and links in read the docs to make sure TLOG documentation is correct.		TLOG was added for this release
16	simp-doc, x2go, mate		Review How to manage Workstation entry for updates for mate and x2go		X2go, Mate were added for this release
17	simp-doc, hiera 5 and puppet 5 upgrade		Review docs to make sure the hiera data directory has been updated to reflect changes. And that the version of puppet currently supported has been updated to puppet 5.		hiera and puppet updated to version 5.
18	ISO		SIMP server upgrade from RPM using the upgrade instructions.	SIMP-5383 - Getting issue details... STATUS	A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting. (This is SIMP-5021 - Getting issue details... STATUS + 6.1.0) The fix is procedural: upgrade steps (and fixes) in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure Pay attention to Auditd upgrade because it is a major version upgrade.
19	PE		Upgrade PE SIMP server from 6.2 puppet 4 to puppet 5 (PE 2018.1) and simp 6.3		Not sure what OS it is most important to test this on.
20	ISO		Dogfooding

Regression/Integration Testing (most of these test should eventually be tested by simp-packer or simp-core tests. Use the packer boxes created to do testing were applicable.)

	Test	Automated	Notes
1	Verify non-standard BIOS boot options from the iso: choose own partitions minimum installation	N	(SIMP, FIPS and Encryption are all tested by Packer.)
2	Verify all boot Options in UEFI mode from iso SIMP, FIPS, Encrypted, choose partitions,minimun	N
3	PXE Boot Testing BIOS
4	PXE Boot Testing UEFI
5	Bootstrap simp-lite scenario
6	Verify simp-lite operations: login operations (PAM, LDAP, local user) NFS operations (home directory) logging operations (rsyslog) auditing operations
7	Bootstrap poss scenario
8	Bootstrap run-once scenario
9	Verify ability to switch FIPS from on to off		Switching from off to on is know to have problems because of ciphers.
10	Verify ability to turn on and off auditing and selinux
11	On SIMP server: Verify ldap operations are working: On puppet server and on a client admin1 can log in admin1 can sudo su - root user1 can not log in to puppet server		Users are created by packer already, just test they work. admin1, user1
12	General Review of documentation: Getting Started User Guide HOW TOs FAQS
13	simp-utils: executables that are not tested otherwise work as advertised `unpack_dvd` `gen_ldap_update` `updaterepos`