Versions Compared

Version Old Version 38 New Version Current
Changes made by

Liz Nemsick

Liz Nemsick

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

greenthese test are automated



TestEL6EL7Notes1

Verify non-standard BIOS boot options from the ISO:

  •   choose own partitions
  •   minimum installation
(SIMP, FIPS and Encryption are all tested by Packer.)columns



5585

(lightbulb)

key,simp-packer

Verify installation from RPM.

This simp-core test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins
key5598151718integration_test

(lightbulb)

(question)

5383See key,summary,

(lightbulb)

Or "drink your own home brew", which ever you prefer.

  • CentOS 7: SIMP server and SIMP client
  • CentOS 6: SIMP client only

in simp-core run

  • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm]
  • beaker:suites[install_from_rpm]
  • beaker:suites[install_from_rpm,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm,el6_server]

and then do it all again with BEAKER_fips=yes

run the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge

Component

Required

Tests

TestEL7EL6OL7OL6RH7RH6StatusNotes
1simp-core
Verify Puppet File matches release versions

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5539

This must be completed before any other tests are done


2Puppet 5/hiera 51

For all core components, including simp-core:

  • Verify a .gitlab-ci.yml exists and the project is configured in gitlab
  • Verify that .gitlab-ci.yml is running correct testing matrix (see above) and that puppet 5 tests are no longer "allow failure" .
  • For puppet modules and simp-core tests, make sure BEAKER_PUPPET_COLLECTION: 'puppet5' is set for puppet 5 tests.


(grey lightbulb)(grey lightbulb)

SIMP-5529This addresses
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5585
  • This addresses Puppet 5 part of 6.3
  • Sample .gitlab-ci.yml
  • Some components have acceptance tests that are not being executed in GitLab because no corresponding project exists in GitLab and/or the project is missing a .gitlab-ci.yml file.  simp-utils is one such project.
Jira Legacy
serverSystem JIRA
3Puppet 5/Hiera 52For all core components: verify full matrix acceptance tests pass.

(lightbulb)


(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
3Puppet
5537
  • This addresses puppet 5/
Hiera 5		2For all core components: verify full matrix acceptance tests pass.(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)
  • This addresses puppet 5/hiera 5 update part of 6.3. 
  • 'beaker:suites'hiera 5 update part of 6.3. 
  • 'beaker:suites', by default will run only the 'default' suite and any suites that have added themselves to the 'default' suite via their metadata.yml files.   So, we Nneed to run beaker:suites[ALL,ALL] for pupmod-simp-* modules with BEAKER_PUPPET_COLLECTION=puppet5 and with and without BEAKER_FIPS=yes.


4
Jira Legacy
serverSystem JIRA
columnssimp-core
Verify puppetfile and dependencies.yaml by building ISO(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5537		4simp-coreVerify puppetfile and dependencies.yaml by building ISO(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
5540
  • This is a normal release testing activity.
  • The dependencies.yaml has to be manually reviewed each release to make sure it has the appropriate list of dependencies for non-SIMP-owned modules, appropriate other dependendencies for  any modules that require it, and appropriate obsoletes and RPM release qualifiers.
  • Should result in ISO for CentOS. 
  • Just verify it can be built for RedHat (AWS?)
jira


5
serverSystem JIRA
columns4Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5540		5simp-packer4Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS
5548
  • This is a normal release testing activity.
  • Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6


6

simp-core


Verify installation from RPM.

This simp-core test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins



(lightbulb)(lightbulb)(grey lightbulb)(lightbulb)(grey lightbulb)(lightbulb)(lightbulb)(lightbulb)
  • This is a normal release testing activity.
  • Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6
Jira Legacy
server
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5548		6

simp-core

(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)

This is a normal release testing activity.

In simp-core run using the tar file generated during ISO build

5598

This is a normal release testing activity.

In simp-core run using the tar file generated during ISO build

set BEAKER_release_tarball=<location of the tarball  for el7>

  • beaker:suites[install_from_tar]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

set BEAKER_release_tarball=<location of the tarball  for el6>

  • beaker:suites[install_from_tar,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]
jira


 
serverSystem JIRA
columns
7simp-core

Verify installation from RPM in FIPS

This test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins
(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5598
 		7simp-core

Verify installation from RPM in FIPS

This test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins
(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)

This is a normal release testing activity.

For all tests set BEAKER_fips=yes

set BEAKER_release_This is a normal release testing activity.

For all tests set BEAKER_fips=yes

set BEAKER_release_tarball=<location of the tarball  for el7>

  • beaker:suites[install_from_tar]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

set BEAKER_release_tarball=<location of the tarball  for el6>

  • beaker:suites[install_from_tar,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]
jira



8
serverSystem JIRA
installation
Verify installation from RPM for PE(lightbulb)(lightbulb)(lightbulb)(lightbulb)(question)(question)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
8installationVerify installation from RPM for PE(lightbulb)(lightbulb)(lightbulb)(lightbulb)(question)(question)
5535

RedHat built on AWS?

simp cli was updated make sure you test out simp cli



9tlog, mate, x2go, ima5Verify packages.yaml files have all dependencies for new modules by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)RedHat built on AWS?
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5549

RedHat built on AWS?




10tpm25Verify packages.yaml files have all dependencies by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(question)(grey lightbulb)

RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.

jira
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5550

RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.



11compliance_markup
Verify turning on compliance enforcement works.  Review compliance report.(lightbulb)(lightbulb)

This addresses STIG updates and Compliance Markup modules changes.

Should we run an oscap scan as well?

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5601

This addresses STIG updates and Compliance Markup modules changes.

Should we run an oscap scan as well?


12simp-core
Verify Create and verify Changelog.rst13simp-doc,'Upgrading SIMP' has an entry for the latest version and is correct

See 6.3 Planning page in Jira

Jira Legacy
serverSystem JIRA


(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5543
5578

1413simp-doc, tpm2, imaReview TPM documentation to make sure it is up to date

TPM and IMA were separated out for this release.


'Upgrading SIMP' has an entry for the latest version and is correct

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5224
5543

See 6.3 Planning page in Jira


14simp-doc, tpm2, tlogima
Review Session Auditing and links  in read the docs TPM documentation to make sure TLOG documentation is correct.TLOG was added for this releaseit is up to date

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5224
16

TPM and IMA were separated out for this release.


15simp-doc, x2go, matetlog
Review How to manage Workstation entry for updates for mate and x2goX2go, Mate  were added for this releaseSession Auditing and links  in read the docs to make sure TLOG documentation is correct.

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5224

TLOG was added for this release


16simp-doc, hiera 5 and puppet 5 upgrade, x2go, mate
Review docs to make sure the hiera data directory has been updated to reflect changes.  And that the version of puppet currently supported has been updated to puppet 5. hiera and puppet updated to version 5.How to manage Workstation entry for updates for mate and x2go

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5224

X2go, Mate  were added for this release


17simp-SIMP server upgrade from RPM using the upgrade instructions.  Document any steps that are needed outside the ordinary upgrade.  Then, if there are additional steps, have someone else use the docs to do the upgrade to verify.

(lightbulb)

(lightbulb)(lightbulb)(question)
Jira Legacy
doc, hiera 5 and puppet 5 upgrade
Review docs to make sure the hiera data directory has been updated to reflect changes.  And that the version of puppet currently supported has been updated to puppet 5. 

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5224

hiera and puppet updated to version 5.


18simp-integration_test for automated upgrade test.

this should be done for servers in both FIPS and NOFIPS mode.

A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting.   (This is 
Jira Legacy
serverSystem JIRA
columns
SIMP server upgrade from RPM using the upgrade instructions.  Document any steps that are needed outside the ordinary upgrade.  Then, if there are additional steps, have someone else use the docs to do the upgrade to verify.

(lightbulb)

(lightbulb)

(lightbulb)


(lightbulb)

(question)


(question)
Jira Legacy
serverSystem JIRA
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5021 + 6.1.0)

The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure 

Pay attention to Auditd upgrade because it is a major version upgrade.

19PE

Upgrade PE SIMP server from 6.2

puppet 4 to puppet 5  (PE 2018.1) and simp 6.3

(lightbulb)(lightbulb)(lightbulb)Not sure what OS it is most important to test this on.20RPMs and Released Modules

Dogfooding

5383

See simp-integration_test for automated upgrade test.

this should be done for servers in both FIPS and NOFIPS mode.

A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting.   (This is 

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5021
 + 6.1.0)

The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure 

Pay attention to Auditd upgrade because it is a major version upgrade.


19PE

Upgrade PE SIMP server from 6.2

puppet 4 to puppet 5  (PE 2018.1) and simp 6.3

(lightbulb)


(lightbulb)(lightbulb)(lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5577		21Final RPM testingRPMs have been uploaded to packagecloudRun the simp-core install_from_rpm test and versify that it passes.   This will pull the packages from package.io and do an install.(lightbulb)(lightbulb)(lightbulb)(lightbulb)22Puppet Forge TestingAfter modules have been tag with an annotated tag
5536

Not sure what OS it is most important to test this on.


20RPMs and Released Modules

Dogfooding



(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5577

Or "drink your own home brew", which ever you prefer.

  • CentOS 7: SIMP server and SIMP client
  • CentOS 6: SIMP client only


21Final RPM testingRPMs have been uploaded to packagecloudRun the simp-core install_from_rpm test and versify that it passes.   This will pull the packages from package.io and do an install.(lightbulb)(lightbulb)(lightbulb)(lightbulb)

in simp-core run

  • beaker:suites[install_from_core_module]


    • Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5534


    in simp-core run

    • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_modulerpm]
    • beaker:suites[install_from_rpm]
    • beaker:suites[install_from_core_modulerpm,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm,el6_server]

    and then do it all again with BEAKER_

    core_module,el6_server]

    and then do it all again with BEAKER_fips=yes

    Tests - Regression/Integration Testing

    The tests in this section are regression tests to ensure unmodified capabilities still function.  These tests should use the packer boxes created in the New/Changed Feature tests, were applicable.  Note that most of these test should eventually be automated by simp-packer, simp-integration_test, or simp-core tests.

    fips=yes


    22Puppet Forge TestingAfter modules have been tag with an annotated tagrun the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge(lightbulb)(lightbulb)(lightbulb)

    (lightbulb)




    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5538

    in simp-core run

    • beaker:suites[install_from_core_module]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module]
    • beaker:suites[install_from_core_module,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module,el6_server]

    and then do it all again with BEAKER_fips=yes



    Tests - Regression/Integration Testing

    The tests in this section are regression tests to ensure unmodified capabilities still function.  These tests should use the packer boxes created in the New/Changed Feature tests, were applicable.  Note that most of these test should eventually be automated by simp-packer, simp-integration_test, or simp-core tests.


    		TestEL6EL7StatusNotes
    1

    Verify non-standard BIOS boot options from the ISO:

    •   choose own partitions
    •   minimum installation



    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5551

    		(SIMP, FIPS and Encryption are all tested by Packer.)
    2

    Verify all boot Options in UEFI mode from ISO:

    • FIPS
    • no-FIPS
    • FIPS+Encrypted
    • FIPS+choose own partitions
    • minimum


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5587

    3PXE Boot Testing BIOS

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    55512

    Verify all boot Options in UEFI mode from ISO:

  • FIPS
  • no-FIPS
  • FIPS+Encrypted
  • FIPS+choose own partitions
    • minimum
    5579

    4PXE Boot Testing UEFI

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    5587PXE Boot Testing BIOS
    5580
    3

    5Bootstrap simp-lite scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    55794PXE Boot Testing UEFI
    5588

    6

    Verify simp-lite operations:

    • login operations (PAM, LDAP, local user)
    • NFS operations (home directory)
    • logging operations (rsyslog)
    • auditing operations


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    5580
    5586
    5

    7Bootstrap
    simp-lite
    poss scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5588
    67Bootstrap poss scenario8Bootstrap run-once scenario

    8

    Verify simp-lite operations:

    • login operations (PAM, LDAP, local user)
    • NFS operations (home directory)
    • logging operations (rsyslog)
    • auditing operations
    Bootstrap run-once scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5588

    9Verify ability to switch FIPS from on to
    offSwitching from
    off
    to on is known to have problems because of ciphers.


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-

    86a38172f71dkeySIMP-5605

    86a38172f71d
    keySIMP-5605

    		Switching from off to on is known to have problems because of ciphers.
    10Verify ability to turn on and off auditing and selinux

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5605

    11

    General Review of documentation:

    • Getting Started
    • User Guide
    • HOW TOs
    • FAQS




    12simp-utils:

    executables that are not tested otherwise work as advertised

    • unpack_dvd


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5553

    The following have acceptance tests that simply need to be executed:

    • gen_ldap_update
    • updaterepos