Versions Compared

Version Old Version 39 New Version Current
Changes made by

Liz Nemsick

Liz Nemsick

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

columns



5585

(lightbulb)

key,summary


key,simp-packerserver

Verify installation from RPM.

This simp-core test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins
key5598

Component

Required

Tests

TestEL7EL6OL7OL6RH7RH6StatusNotes
1simp-core
Verify Puppet File matches release versions

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5539

This must be completed before any other tests are done


2Puppet 5/hiera 51

For all core components, including simp-core:

  • Verify a .gitlab-ci.yml exists and the project is configured in gitlab
  • Verify that .gitlab-ci.yml is running correct testing matrix (see above) and that puppet 5 tests are no longer "allow failure" .
  • For puppet modules and simp-core tests, make sure BEAKER_PUPPET_COLLECTION: 'puppet5' is set for puppet 5 tests.


(grey lightbulb)(grey lightbulb)

SIMP-5529This addresses Puppet
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5585
  • This addresses Puppet 5 part of 6.3
  • Sample .gitlab-ci.yml
  • Some components have acceptance tests that are not being executed in GitLab because no corresponding project exists in GitLab and/or the project is missing a .gitlab-ci.yml file.  simp-utils is one such project.
Jira Legacy
serverSystem JIRA
3Puppet 5/Hiera 52For all core components: verify full matrix acceptance tests pass.

(lightbulb)


(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
3Puppet
5537
  • This addresses puppet 5/
Hiera 5		2For all core components: verify full matrix acceptance tests pass.(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)
  • This addresses puppet 5/hiera 5 update part of 6.3. 
  • 'beaker:suites'hiera 5 update part of 6.3. 
  • 'beaker:suites', by default will run only the 'default' suite and any suites that have added themselves to the 'default' suite via their metadata.yml files.   So, we Nneed to run beaker:suites[ALL,ALL] for pupmod-simp-* modules with BEAKER_PUPPET_COLLECTION=puppet5 and with and without BEAKER_FIPS=yes.
Jira Legacy
serverSystem JIRA
columns
4simp-core
Verify puppetfile and dependencies.yaml by building ISO(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5537		4simp-coreVerify puppetfile and dependencies.yaml by building ISO(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
5540
  • This is a normal release testing activity.
  • The dependencies.yaml has to be manually reviewed each release to make sure it has the appropriate list of dependencies for non-SIMP-owned modules, appropriate other dependendencies for  any modules that require it, and appropriate obsoletes and RPM release qualifiers.
  • Should result in ISO for CentOS. 
  • Just verify it can be built for RedHat (AWS?)
jira


5
serverSystem JIRA
columns4Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5540		5simp-packer4
5548
  • This is a normal release testing activity.
  • Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6


6

simp-core


Verify installation from

ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS

RPM.

This simp-core test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins



(lightbulb)(lightbulb)(grey lightbulb)(lightbulb)(grey lightbulb)(lightbulb)(lightbulb)(lightbulb)
  • This is a normal release testing activity.
  • Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6
Jira Legacy
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5548		6

simp-core

(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)

This is a normal release testing activity.

In simp-core run using the tar file generated during ISO build

set
5598

This is a normal release testing activity.

In simp-core run using the tar file generated during ISO build

set BEAKER_release_tarball=<location of the tarball  for el7>

  • beaker:suites[install_from_tar]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

set BEAKER_release_tarball=<location of the tarball  for el6>

  • beaker:suites[install_from_tar,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]
jira


 
serverSystem JIRA
columns
7simp-core

Verify installation from RPM in FIPS

This test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins
(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5598
 		7simp-core

Verify installation from RPM in FIPS

This test also verifies

  • rsyslog forwarding
  • ldap logins
  • local logins
(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)

This is a normal release testing activity.

For all tests set BEAKER_fips=yes

set BEAKER_release_This is a normal release testing activity.

For all tests set BEAKER_fips=yes

set BEAKER_release_tarball=<location of the tarball  for el7>

  • beaker:suites[install_from_tar]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

set BEAKER_release_tarball=<location of the tarball  for el6>

  • beaker:suites[install_from_tar,el6_server]
  • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]
jira



8
serverSystem JIRA
installation
Verify installation from RPM for PE(lightbulb)(lightbulb)(lightbulb)(lightbulb)(question)(question)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
8installationVerify installation from RPM for PE(lightbulb)(lightbulb)(lightbulb)(lightbulb)(question)(question)
5535

RedHat built on AWS?

simp cli was updated make sure you test out simp cli

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5535



9tlog, mate, x2go, ima9tlog, mate, x2go, ima5Verify packages.yaml files have all dependencies for new modules by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)

RedHat built on AWS?

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5549

RedHat built on AWS?




10tpm25Verify packages.yaml files have all dependencies by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(question)(grey lightbulb)

RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.

Jira Legacy
server
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5550

RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.



11compliance_markup
Verify turning on compliance enforcement works.  Review compliance report.(lightbulb)(lightbulb)

This addresses STIG updates and Compliance Markup modules changes.

Should we run an oscap scan as well?

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5601

This addresses STIG updates and Compliance Markup modules changes.

Should we run an oscap scan as well?


12simp-core
Verify Create and verify Changelog.rst13simp-doc,'Upgrading SIMP' has an entry for the latest version and is correct

See 6.3 Planning page in Jira

Jira LegacyserverSystem JIRA

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5543
5578

1413simp-doc, tpm2, imaReview TPM documentation to make sure it is up to date

TPM and IMA were separated out for this release.

jira
'Upgrading SIMP' has an entry for the latest version and is correct

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
522415
5543

See 6.3 Planning page in Jira


14simp-doc, tpm2, tlogima
Review Session Auditing and links  in read the docs TPM documentation to make sure TLOG documentation is correct.TLOG was added for this releaseit is up to date

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5224
16

TPM and IMA were separated out for this release.


15simp-doc, x2go, matetlog
Review How to manage Workstation entry for updates for mate and x2goX2go, Mate  were added for this releaseSession Auditing and links  in read the docs to make sure TLOG documentation is correct.

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5224
17

TLOG was added for this release


16simp-doc, hiera 5 and puppet 5 upgrade, x2go, mate
Review docs to make sure the hiera data directory has been updated to reflect changes.  And that the version of puppet currently supported has been updated to puppet 5. hiera and puppet updated to version 5.How to manage Workstation entry for updates for mate and x2go

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5224
18

X2go, Mate  were added for this release


17simp-integration_testSIMP server upgrade from RPM using the upgrade instructions.  Document any steps that are needed outside the ordinary upgrade.  Then, if there are additional steps, have someone else use the docs to do the upgrade to verify.

(lightbulb)

(lightbulb)

(lightbulb)

(lightbulb)

(question)

(question) Jira Legacydoc, hiera 5 and puppet 5 upgrade
Review docs to make sure the hiera data directory has been updated to reflect changes.  And that the version of puppet currently supported has been updated to puppet 5. 

(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5383See
5224

hiera and puppet updated to version 5.


18simp-integration_test for automated upgrade test.

this should be done for servers in both FIPS and NOFIPS mode.

A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting.   (This is  Jira LegacyserverSystem JIRAcolumnskey,summary,type
SIMP server upgrade from RPM using the upgrade instructions.  Document any steps that are needed outside the ordinary upgrade.  Then, if there are additional steps, have someone else use the docs to do the upgrade to verify.

(lightbulb)

(lightbulb)

(lightbulb)


(lightbulb)

(question)


(question)

Upgrade PE SIMP server from 6.2

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
5021 + 6.1.0)

The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure 

Pay attention to Auditd upgrade because it is a major version upgrade.

19PE
5383

See simp-integration_test for automated upgrade test.

this should be done for servers in both FIPS and NOFIPS mode.

A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting.   (This is 

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5021
 + 6.1.0)

The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure 

Pay attention to Auditd upgrade because it is a major version upgrade.


19PE

Upgrade PE SIMP server from 6.2

puppet 4 to puppet 5  (PE 2018.1) and simp 6.3

(lightbulb)


(lightbulb)(lightbulb)(lightbulb)

Not sure what OS it is most important to test this on.

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5536

Not sure what OS it is most important to test this on.


20RPMs and Released Modules

Dogfooding



Or "drink your own home brew", which ever you prefer.

  • CentOS 7: SIMP server and SIMP client
  • CentOS 6: SIMP client only
(grey lightbulb)(grey lightbulb)

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5577
21

Or "drink your own home brew", which ever you prefer.

  • CentOS 7: SIMP server and SIMP client
  • CentOS 6: SIMP client only


21Final RPM testingRPMs have been uploaded to packagecloudRun the simp-core install_from_rpm test and versify that it passes.   This will pull the packages from package.io and do an install.(lightbulb)(lightbulb)(lightbulb)(lightbulb)

in simp-core run

  • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm]
  • beaker:suites[install_from_rpm]
    • beaker:suites[install_from_rpm,el6_server]

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5534


    in simp-core run

    • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm]
    • beaker:suites[install_from_rpm]
    • beaker:suites[install_from_rpm,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm,el6_server]

    and then do it all again with BEAKER_fips=yes


    22Puppet Forge TestingAfter modules have been tag with an annotated tagrun the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge(lightbulb)(lightbulb)(lightbulb)

    (lightbulb)

    in simp-core run

  • beaker:suites[install_from_core_module]
    • SIMP_BEAKER_OS=oel




    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5538

    in simp-core run

    • beaker:suites[install_from_core_module]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module]
    • beaker:suites[install_from_core_module,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module,el6_server]

    and then do it all again with BEAKER_fips=yes


    ...

    The tests in this section are regression tests to ensure unmodified capabilities still function.  These tests should use the packer boxes created in the New/Changed Feature tests, were applicable.  Note that most of these test should eventually be automated by simp-packer, simp-integration_test, or simp-core tests.


    		Test
    EL6EL7Notes1

    Verify non-standard BIOS boot options from the ISO:

    •   choose own partitions
    •   minimum installation
    (SIMP, FIPS and Encryption are all tested by Packer.)
    EL6EL7StatusNotes
    1

    Verify non-standard BIOS boot options from the ISO:

    •   choose own partitions
    •   minimum installation



    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5551

    		(SIMP, FIPS and Encryption are all tested by Packer.)
    2

    Verify all boot Options in UEFI mode from ISO:

    • FIPS
    • no-FIPS
    • FIPS+Encrypted
    • FIPS+choose own partitions
    • minimum


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5587

    3PXE Boot Testing BIOS

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    5551

    Verify all boot Options in UEFI mode from ISO:

  • FIPS
  • no-FIPS
  • FIPS+Encrypted
  • FIPS+choose own partitions
    • minimum
    5579
    2

    4PXE Boot Testing UEFI

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5580

    5Bootstrap simp-lite scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    55873PXE Boot Testing BIOS
    5588

    6

    Verify simp-lite operations:

    • login operations (PAM, LDAP, local user)
    • NFS operations (home directory)
    • logging operations (rsyslog)
    • auditing operations


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    5579PXE Boot Testing UEFI
    5586
    4

    7Bootstrap poss scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-
    5580
    5588
    5

    8Bootstrap
    simp
    run-
    lite
    once scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter
    ,priority,status,resolutionserverId45e63be1-2551-37a6-b050-86a38172f71dkeySIMP-55886

    Verify simp-lite operations:

    • login operations (PAM, LDAP, local user)
    • NFS operations (home directory)
    • logging operations (rsyslog)
    • auditing operations
    7Bootstrap poss scenario8Bootstrap run-once scenario
    ,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5588

    9Verify ability to switch FIPS from
    on to offSwitching from off to on is known to have problems because of ciphers.
    on to off

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5605

    		Switching from off to on is known to have problems because of ciphers.
    10Verify ability to turn on and off auditing and selinux

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5605

    11

    General Review of documentation:

    • Getting Started
    • User Guide
    • HOW TOs
    • FAQS




    12simp-utils:

    executables that are not tested otherwise work as advertised

    • unpack_dvd

    The following have acceptance tests that simply need to be executed:

    • gen_ldap_update
    • updaterepos


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5553

    The following have acceptance tests that simply need to be executed:

    • gen_ldap_update
    • updaterepos