Page Comparison

This must be complete before any other tests are done

For all core components, including simp-core:

• This addresses puppet 5/hiera 5 update part of 6.3. 
• 'beaker:suites', by default will run only the 'default' suite and any suites that have added themselves to the 'default' suite via their metadata.yml files.   So, we Nneed to run beaker:suites[ALL,ALL] for pupmod-simp-* modules with BEAKER_PUPPET_COLLECTION=puppet5 and with and without BEAKER_FIPS=yes.

				Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5539	This must be completed before any other tests are done
2	Puppet 5/hiera 5	1	For all core components, including simp-core: Verify a .gitlab-ci.yml exists and the project is configured in gitlab Verify that .gitlab-ci.yml is running correct testing matrix (see above) and that puppet 5 tests are no longer "allow failure" . For puppet modules and simp-core tests, make sure BEAKER_PUPPET_COLLECTION: 'puppet5' is set for puppet 5 tests.

• This addresses Puppet 5 part of 6.3
• Sample .gitlab-ci.yml
• Some components have acceptance tests that are not being executed in GitLab because no corresponding project exists in GitLab and/or the project is missing a .gitlab-ci.yml file.  simp-utils is one such project.

				SIMP-5529 Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5585	This addresses Puppet 5 part of 6.3 Sample .gitlab-ci.yml Some components have acceptance tests that are not being executed in GitLab because no corresponding project exists in GitLab and/or the project is missing a .gitlab-ci.yml file.  simp-utils is one such project.
3	Puppet 5/Hiera 5	2	For all core components: verify full matrix acceptance tests pass.

• This is a normal release testing activity.
• The dependencies.yaml has to be manually reviewed each release to make sure it has the appropriate list of dependencies for non-SIMP-owned modules, appropriate other dependendencies for  any modules that require it, and appropriate obsoletes and RPM release qualifiers.
• Should result in ISO for CentOS. 
• Just verify it can be built for RedHat (AWS?)

server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5537	This addresses puppet 5/hiera 5 update part of 6.3.  'beaker:suites', by default will run only the 'default' suite and any suites that have added themselves to the 'default' suite via their metadata.yml files.   So, we Nneed to run beaker:suites[ALL,ALL] for pupmod-simp-* modules with BEAKER_PUPPET_COLLECTION=puppet5 and with and without BEAKER_FIPS=yes.
4	simp-core		Verify puppetfile and dependencies.yaml by building ISO

simp-core

Verify installation from RPM.

This simp-core test also verifies

• rsyslog forwarding
• ldap logins
• local logins

The dependencies.yaml has to be manually reviewed each release to make sure it has the appropriate list of dependencies for non-SIMP-owned modules, appropriate other dependendencies for  any modules that require it, and appropriate obsoletes and RPM release qualifiers. Should result in ISO for CentOS.  Just verify it can be built for RedHat (AWS?)
5	simp-packer	4	Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS

In simp-core run using the tar file generated during ISO build

set BEAKER_release_tarball=<location of the tarball  for el7>

• beaker:suites[install_from_tar]
• SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

set BEAKER_release_tarball=<location of the tarball  for el6>

• beaker:suites[install_from_tar,el6_server]
• SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]

Verify installation from RPM in FIPS

This test also verifies

• rsyslog forwarding
• ldap logins
• local logins

Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6
6	simp-core		Verify installation from RPM. This simp-core test also verifies rsyslog forwarding ldap logins local logins

in FIPS This test also verifies rsyslog forwarding ldap logins local logins

RedHat built on AWS?

simp cli was updated make sure you test out simp cli

RedHat built on AWS?

This addresses STIG updates and Compliance Markup modules changes.

Should we run an oscap scan as well?

	Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5598	This is a normal release testing activity. For all tests set BEAKER_fips=yes set BEAKER_release_tarball=<location of the tarball  for el7> beaker:suites[install_from_tar] SIMP_BEAKER_OS=oel beaker:suites[install_from_tar] set BEAKER_release_tarball=<location of the tarball  for el6> beaker:suites[install_from_tar,el6_server] SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]
8	installation		Verify installation from RPM for PE							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5535	RedHat built on AWS? simp cli was updated make sure you test out simp cli
9	tlog, mate, x2go, ima	5	Verify packages.yaml files have all dependencies for new modules by installing these from a system built by an ISO with no internet connectivity.							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5549	RedHat built on AWS?
10	tpm2	5	Verify packages.yaml files have all dependencies by installing these from a system built by an ISO with no internet connectivity.							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-

See simp-integration_test for automated upgrade test.

this should be done for servers in both FIPS and NOFIPS mode.

86a38172f71d key SIMP-5550	RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.
11	compliance_markup		Verify turning on compliance enforcement works.  Review compliance report.							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5601	This addresses STIG updates and Compliance Markup modules changes. Should we run an oscap scan as well?
12	simp-core		Create and verify Changelog.rst							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5578
13	simp-doc,		'Upgrading SIMP' has an entry for the latest version and is correct							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5543	See 6.3 Planning page in Jira
14	simp-doc, tpm2, ima		Review TPM documentation to make sure it is up to date							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5224	TPM and IMA were separated out for this release.
15	simp-doc, tlog		Review Session Auditing and links  in read the docs to make sure TLOG documentation is correct.							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5224	TLOG was added for this release
16	simp-doc, x2go, mate		Review How to manage Workstation entry for updates for mate and x2go							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5224	X2go, Mate  were added for this release
17	simp-doc, hiera 5 and puppet 5 upgrade		Review docs to make sure the hiera data directory has been updated to reflect changes.  And that the version of puppet currently supported has been updated to puppet 5.							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5224	hiera and puppet updated to version 5.
18	simp-integration_test		SIMP server upgrade from RPM using the upgrade instructions.  Document any steps that are needed outside the ordinary upgrade.  Then, if there are additional steps, have someone else use the docs to do the upgrade to verify.							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5383	See simp-integration_test for automated upgrade test. this should be done for servers in both FIPS and NOFIPS mode. A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting.   (This is  Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5021  + 6.1.0) The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure  Pay attention to Auditd upgrade because it is a major version upgrade.
19	PE		Upgrade PE SIMP server from 6.2 puppet 4 to puppet 5  (PE 2018.1) and simp 6.3							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5536	Not sure what OS it is most important to test this on.
20	RPMs and Released Modules		Dogfooding							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5577	Or "drink your own home brew", which ever you prefer. CentOS 7: SIMP server and SIMP client CentOS 6: SIMP client only
21	Final RPM testing	RPMs have been uploaded to packagecloud	Run the simp-core install_from_rpm test and versify that it passes.   This will pull the packages from package.io and do an install.							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5534	in simp-core run SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm] beaker:suites[install_from_rpm] beaker:suites[install_from_rpm,el6_server] SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm,el6_server] and then do it all again with BEAKER_fips=yes
22	Puppet Forge Testing	After modules have been tag with an annotated tag	run the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge							Jira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45e63be1-2551-37a6-b050-86a38172f71d key SIMP-5538	in simp-core run beaker:suites[install_from_core_module] SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module] beaker:suites[install_from_core_module,el6_server] SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module,el6_server] and then do it all again with BEAKER_fips=yes

The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure 

Pay attention to Auditd upgrade because it is a major version upgrade.

Upgrade PE SIMP server from 6.2

puppet 4 to puppet 5  (PE 2018.1) and simp 6.3

Dogfooding

Or "drink your own home brew", which ever you prefer.

CentOS 7: SIMP server and SIMP client

CentOS 6: SIMP client only

in simp-core run

• SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm]
• beaker:suites[install_from_rpm]
• beaker:suites[install_from_rpm,el6_server]
• SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm,el6_server]

and then do it all again with BEAKER_fips=yes

in simp-core run

• beaker:suites[install_from_core_module]
• SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module]
• beaker:suites[install_from_core_module,el6_server]
• SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module,el6_server]

and then do it all again with BEAKER_fips=yes

Verify non-standard BIOS boot options from the ISO:

•   choose own partitions
•   minimum installation

Verify all boot Options in UEFI mode from ISO:

• FIPS
• no-FIPS
• FIPS+Encrypted
• FIPS+choose own partitions
• minimum

Verify simp-lite operations:

• login operations (PAM, LDAP, local user)
• NFS operations (home directory)
• logging operations (rsyslog)
• auditing operations