Test Matrix

Owned by Jeanne Greulich
Last updated: Nov 16, 2018 by Liz Nemsick
5 min read

Table of Contents

Puppet Versions Testing

Puppet version for GitLab and TravisCI

Puppet modules will be tested in GitLab and TravisCI with the following versions:

ReleasePuppetRubyNotes
PE 2017.24.10.*2.1.9

SIMP 6.1.0, SIMP 6.2.0 also tested on this version

  • Modules may have progressed past this and that is OK. There is no longer a guarantee that 4.10.* will be supported
PE 2018.1.45.5.62.4.4

5.5.72.4.4Version to be included in SIMP 6.3.0 ISOs

Puppet version for simp-core (integration) and SIMP ISO testing

ReleasePuppetRubyNotes
SIMP 6.3.05.5.72.4.4Version to be included in SIMP 6.3.0 ISOs

OS Versions For Testing

  • SIMP acceptance tests will be executed with the following OS versions:
    • CentOS 7.5
    • CentOS 6.10
    • RedHat 7.5
    • RedHat 6.10
    • OracleLinux 7.5
    • OracleLinux 6.10
  • SIMP-6.3.0 ISOs for CentOS 7.5 and CentOS 6.10 will be built and tested.
  • If licenses are available, SIMP-6.3.0 ISOs for RedHat 7.5 and RedHat 6.10 will be built and tested.


Tests - New/Changed Features

The tests in this section focus on new features or major, SIMP-wide changes.

Legend:

(lightbulb) - Means this version needs to be tested

(grey lightbulb) - This version does not need to be tested.

(question) - Testing probably sufficient from other versions.

(error) - Testing has failed.

(tick) - Testing completed and Passed.

blank - This test is not OS specific

greenthese test are automated



Component

Required

Tests

TestEL7EL6OL7OL6RH7RH6StatusNotes
1simp-core
Verify Puppet File matches release versions

(grey lightbulb)(grey lightbulb)

 SIMP-5539 - Getting issue details... STATUS

This must be completed before any other tests are done


2Puppet 5/hiera 51

For all core components, including simp-core:

  • Verify a .gitlab-ci.yml exists and the project is configured in gitlab
  • Verify that .gitlab-ci.yml is running correct testing matrix (see above) and that puppet 5 tests are no longer "allow failure" .
  • For puppet modules and simp-core tests, make sure BEAKER_PUPPET_COLLECTION: 'puppet5' is set for puppet 5 tests.


(grey lightbulb)(grey lightbulb)

SIMP-5529 SIMP-5585 - Getting issue details... STATUS
  • This addresses Puppet 5 part of 6.3
  • Sample .gitlab-ci.yml
  • Some components have acceptance tests that are not being executed in GitLab because no corresponding project exists in GitLab and/or the project is missing a .gitlab-ci.yml file.  simp-utils is one such project.



3Puppet 5/Hiera 52For all core components: verify full matrix acceptance tests pass.

(lightbulb)


(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb) SIMP-5537 - Getting issue details... STATUS
  • This addresses puppet 5/hiera 5 update part of 6.3. 
  • 'beaker:suites', by default will run only the 'default' suite and any suites that have added themselves to the 'default' suite via their metadata.yml files.   So, we Nneed to run beaker:suites[ALL,ALL] for pupmod-simp-* modules with BEAKER_PUPPET_COLLECTION=puppet5 and with and without BEAKER_FIPS=yes.


4simp-core
Verify puppetfile and dependencies.yaml by building ISO(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb) SIMP-5540 - Getting issue details... STATUS
  • This is a normal release testing activity.
  • The dependencies.yaml has to be manually reviewed each release to make sure it has the appropriate list of dependencies for non-SIMP-owned modules, appropriate other dependendencies for  any modules that require it, and appropriate obsoletes and RPM release qualifiers.
  • Should result in ISO for CentOS. 
  • Just verify it can be built for RedHat (AWS?)


5simp-packer4Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb) SIMP-5548 - Getting issue details... STATUS
  • This is a normal release testing activity.
  • Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6


6

simp-core


Verify installation from RPM.

This simp-core test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins



(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb) SIMP-5598 - Getting issue details... STATUS

This is a normal release testing activity.

In simp-core run using the tar file generated during ISO build

set BEAKER_release_tarball=<location of the tarball  for el7>

  • beaker:suites[install_from_tar]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

set BEAKER_release_tarball=<location of the tarball  for el6>

  • beaker:suites[install_from_tar,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]


 
7simp-core

Verify installation from RPM in FIPS

This test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins
(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb) SIMP-5598 - Getting issue details... STATUS

This is a normal release testing activity.

For all tests set BEAKER_fips=yes

set BEAKER_release_tarball=<location of the tarball  for el7>

  • beaker:suites[install_from_tar]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

set BEAKER_release_tarball=<location of the tarball  for el6>

  • beaker:suites[install_from_tar,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]



8installation
Verify installation from RPM for PE(lightbulb)(lightbulb)(lightbulb)(lightbulb)(question)(question) SIMP-5535 - Getting issue details... STATUS

RedHat built on AWS?

simp cli was updated make sure you test out simp cli



9tlog, mate, x2go, ima5Verify packages.yaml files have all dependencies for new modules by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb) SIMP-5549 - Getting issue details... STATUS

RedHat built on AWS?




10tpm25Verify packages.yaml files have all dependencies by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(question)(grey lightbulb) SIMP-5550 - Getting issue details... STATUS

RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.



11compliance_markup
Verify turning on compliance enforcement works.  Review compliance report.(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)

 SIMP-5601 - Getting issue details... STATUS

This addresses STIG updates and Compliance Markup modules changes.

Should we run an oscap scan as well?


12simp-core
Create and verify Changelog.rst

(grey lightbulb)(grey lightbulb)

 SIMP-5578 - Getting issue details... STATUS
13simp-doc,
'Upgrading SIMP' has an entry for the latest version and is correct

(grey lightbulb)(grey lightbulb)

 SIMP-5543 - Getting issue details... STATUS

See 6.3 Planning page in Jira


14simp-doc, tpm2, ima
Review TPM documentation to make sure it is up to date

(grey lightbulb)(grey lightbulb)

 SIMP-5224 - Getting issue details... STATUS

TPM and IMA were separated out for this release.


15simp-doc, tlog
Review Session Auditing and links  in read the docs to make sure TLOG documentation is correct.

(grey lightbulb)(grey lightbulb)

 SIMP-5224 - Getting issue details... STATUS

TLOG was added for this release


16simp-doc, x2go, mate
Review How to manage Workstation entry for updates for mate and x2go

(grey lightbulb)(grey lightbulb)

 SIMP-5224 - Getting issue details... STATUS

X2go, Mate  were added for this release


17simp-doc, hiera 5 and puppet 5 upgrade
Review docs to make sure the hiera data directory has been updated to reflect changes.  And that the version of puppet currently supported has been updated to puppet 5. 

(grey lightbulb)(grey lightbulb)

 SIMP-5224 - Getting issue details... STATUS

hiera and puppet updated to version 5.


18simp-integration_test
SIMP server upgrade from RPM using the upgrade instructions.  Document any steps that are needed outside the ordinary upgrade.  Then, if there are additional steps, have someone else use the docs to do the upgrade to verify.

(lightbulb)

(lightbulb)

(lightbulb)


(lightbulb)

(question)


(question) SIMP-5383 - Getting issue details... STATUS

See simp-integration_test for automated upgrade test.

this should be done for servers in both FIPS and NOFIPS mode.

A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting.   (This is  SIMP-5021 - Getting issue details... STATUS  + 6.1.0)

The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure 

Pay attention to Auditd upgrade because it is a major version upgrade.


19PE

Upgrade PE SIMP server from 6.2

puppet 4 to puppet 5  (PE 2018.1) and simp 6.3

(lightbulb)


(lightbulb)(lightbulb)(lightbulb)

 SIMP-5536 - Getting issue details... STATUS

Not sure what OS it is most important to test this on.


20RPMs and Released Modules

Dogfooding



(grey lightbulb)(grey lightbulb)

 SIMP-5577 - Getting issue details... STATUS

Or "drink your own home brew", which ever you prefer.

  • CentOS 7: SIMP server and SIMP client
  • CentOS 6: SIMP client only


21Final RPM testingRPMs have been uploaded to packagecloudRun the simp-core install_from_rpm test and versify that it passes.   This will pull the packages from package.io and do an install.(lightbulb)(lightbulb)(lightbulb)(lightbulb)

 SIMP-5534 - Getting issue details... STATUS


in simp-core run

  • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm]
  • beaker:suites[install_from_rpm]
  • beaker:suites[install_from_rpm,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm,el6_server]

and then do it all again with BEAKER_fips=yes


22Puppet Forge TestingAfter modules have been tag with an annotated tagrun the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge(lightbulb)(lightbulb)(lightbulb)

(lightbulb)




 SIMP-5538 - Getting issue details... STATUS

in simp-core run

  • beaker:suites[install_from_core_module]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module]
  • beaker:suites[install_from_core_module,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module,el6_server]

and then do it all again with BEAKER_fips=yes



Tests - Regression/Integration Testing

The tests in this section are regression tests to ensure unmodified capabilities still function.  These tests should use the packer boxes created in the New/Changed Feature tests, were applicable.  Note that most of these test should eventually be automated by simp-packer, simp-integration_test, or simp-core tests.


TestEL6EL7StatusNotes
1

Verify non-standard BIOS boot options from the ISO:

  •   choose own partitions
  •   minimum installation



SIMP-5551 - Getting issue details... STATUS

(SIMP, FIPS and Encryption are all tested by Packer.)
2

Verify all boot Options in UEFI mode from ISO:

  • FIPS
  • no-FIPS
  • FIPS+Encrypted
  • FIPS+choose own partitions
  • minimum


 SIMP-5587 - Getting issue details... STATUS
3PXE Boot Testing BIOS

 SIMP-5579 - Getting issue details... STATUS
4PXE Boot Testing UEFI

 SIMP-5580 - Getting issue details... STATUS
5Bootstrap simp-lite scenario

 SIMP-5588 - Getting issue details... STATUS
6

Verify simp-lite operations:

  • login operations (PAM, LDAP, local user)
  • NFS operations (home directory)
  • logging operations (rsyslog)
  • auditing operations


 SIMP-5586 - Getting issue details... STATUS
7Bootstrap poss scenario

 SIMP-5588 - Getting issue details... STATUS
8Bootstrap run-once scenario

 SIMP-5588 - Getting issue details... STATUS
9Verify ability to switch FIPS from on to off

SIMP-5605 - Getting issue details... STATUS

Switching from off to on is known to have problems because of ciphers.
10Verify ability to turn on and off auditing and selinux

 SIMP-5605 - Getting issue details... STATUS
11

General Review of documentation:

  • Getting Started
  • User Guide
  • HOW TOs
  • FAQS




12simp-utils:

executables that are not tested otherwise work as advertised

  • unpack_dvd


SIMP-5553 - Getting issue details... STATUS

The following have acceptance tests that simply need to be executed:

  • gen_ldap_update
  • updaterepos