Versions Compared

Old Version 41

changes.mady.by.user Judith Johnson

Saved on

compared with

New Version Current

changes.mady.by.user Liz Nemsick

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

key,summary,type,created

Component

Required

Tests

TestEL7EL6OL7OL6RH7RH6StatusNotes
1simp-core
Verify Puppet File matches release versions

(grey lightbulb)(grey lightbulb)

This must be completed before any other tests are done

Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5539

This must be completed before any other tests are done


2Puppet 5/hiera 51

For all core components, including simp-core:

  • Verify a .gitlab-ci.yml exists and the project is configured in gitlab
  • Verify that .gitlab-ci.yml is running correct testing matrix (see above) and that puppet 5 tests are no longer "allow failure" .
  • For puppet modules and simp-core tests, make sure BEAKER_PUPPET_COLLECTION: 'puppet5' is set for puppet 5 tests.


(grey lightbulb)(grey lightbulb)

SIMP-5529
  • This addresses Puppet 5 part of 6.3
  • Sample .gitlab-ci.yml
  • Some components have acceptance tests that are not being executed in GitLab because no corresponding project exists in GitLab and/or the project is missing a .gitlab-ci.yml file.  simp-utils is one such project.
Jira Legacy
serverSystem JIRA
columns
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5585
3
  • This addresses Puppet 5
/Hiera 52
  • part of 6.3
  • Sample .gitlab-ci.yml
  • Some components have acceptance tests that are not being executed in GitLab because no corresponding project exists in GitLab and/or the project is missing a .gitlab-ci.yml file.  simp-utils is one such project.



3Puppet 5/Hiera 52For all core components: verify full matrix acceptance tests pass.

(lightbulb)


(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)This addresses
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-5537
  • This addresses puppet 5/hiera 5 update part of 6.3. 
  • 'beaker:suites', by default will run only the 'default' suite and any suites that have added themselves to the 'default' suite via their metadata.yml files.   So, we Nneed to run beaker:suites[ALL,ALL] for pupmod-simp-* modules with BEAKER_PUPPET_COLLECTION=puppet5 and with and without BEAKER_FIPS=yes.
jira


4serverSystem JIRAcolumnssimp-core
Verify puppetfile and dependencies.yaml by building ISO(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
55374simp-coreVerify puppetfile and dependencies.yaml by building ISO(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
5540
  • This is a normal release testing activity.
  • The dependencies.yaml has to be manually reviewed each release to make sure it has the appropriate list of dependencies for non-SIMP-owned modules, appropriate other dependendencies for  any modules that require it, and appropriate obsoletes and RPM release qualifiers.
  • Should result in ISO for CentOS. 
  • Just verify it can be built for RedHat (AWS?)
jira


5serverSystem JIRAsimp-packer4Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
Jira Legacy
serverSystem JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId45e63be1-2551-37a6-b050-86a38172f71d
keySIMP-
55405simp-packer4Verify installation from ISO passes using simp-packer on ISO, both FIPS and no-FIPS, encrypted+FIPS(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)
  • This is a normal release testing activity.
    • Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6
    5548
    • This is a normal release testing activity.
    • Should result in 6 vagrant boxes for testing, 3 for CentOS 7 and 3 for CentOS 6


    6

    simp-core


    Verify installation from RPM.

    This simp-core test also verifies

    • rsyslog forwarding
    • ldap logins
    • local logins



    (lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)
    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee
    ,reporter,priority,status,resolutionserverId45e63be1-2551-37a6-b050-86a38172f71dkeySIMP-55486

    simp-core

    Verify installation from RPM.

    This simp-core test also verifies

    • rsyslog forwarding
    • ldap logins
    • local logins
    (lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)
    ,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5598

    This is a normal release testing activity.

    In simp-core run using the tar file generated during ISO build

    set BEAKER_release_tarball=<location of the tarball  for el7>

    • beaker:suites[install_from_tar]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

    set BEAKER_release_tarball=<location of the tarball  for el6>

    • beaker:suites[install_from_tar,el6_server]SIMP

    set BEAKER_

    BEAKER

    release_

    OS=oel

    tarball=<location of the tarball  for el6>

    • beaker:suites[install_from_tar,el6_server]
    Jira LegacyserverSystem JIRAcolumnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolutionserverId45e63be1-2551-37a6-b050-86a38172f71dkeySIMP-5598
    • server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]


     7simp-core

    Verify installation from RPM in FIPS

    This test also verifies

    • rsyslog forwarding
    • ldap logins
    • local logins
    (lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)
    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5598

    This is a normal release testing activity.

    For all tests set BEAKER_fips=yes

    set BEAKER_release_tarball=<location of the tarball  for el7>

  • beaker:suites[install_from_tar]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar]

    set BEAKER_fips=yes

    set BEAKER_release_tarball=<location of the tarball  for el6>el7>

    • beaker:suites[install_from_tar,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]
    Jira LegacyserverSystem JIRAcolumnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolutionserverId45e63be1-2551-37a6-b050-86a38172f71dkeySIMP-5598
    • ]

    set BEAKER_release_tarball=<location of the tarball  for el6>

    • beaker:suites[install_from_tar,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_tar,el6_server]



    8installation
    Verify installation from RPM for PE(lightbulb)(lightbulb)(lightbulb)(lightbulb)(question)(question)

    RedHat built on AWS?

    simp cli was updated make sure you test out simp cli(question)(question)
    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5535

    RedHat built on AWS?

    simp cli was updated make sure you test out simp cli



    9tlog, mate, x2go, ima5Verify packages.yaml files have all dependencies for new modules by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)RedHat built on AWS?
    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5549

    RedHat built on AWS?




    10tpm25Verify packages.yaml files have all dependencies by installing these from a system built by an ISO with no internet connectivity.(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(question)(grey lightbulb)RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.
    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5550

    RedHat on AWS? These modules don't work in 6 or on OracleLinux.  I think testing in CentOS 7 is good enough.



    11compliance_markup
    Verify turning on compliance enforcement works.  Review compliance report.(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)

    This addresses STIG updates and Compliance Markup modules changes.

    Should we run an oscap scan as well?

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    key
    SIMP-5601
    SIMP-5601

    This addresses STIG updates and Compliance Markup modules changes.

    Should we run an oscap scan as well?


    12simp-core
    Create and verify Changelog.rst

    (grey lightbulb)(grey lightbulb)

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5578

    13simp-doc,
    'Upgrading SIMP' has an entry for the latest version and is correct

    (grey lightbulb)(grey lightbulb)See 6.3 Planning page in Jira

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5543

    See 6.3 Planning page in Jira


    14simp-doc, tpm2, ima
    Review TPM documentation to make sure it is up to date

    (grey lightbulb)(grey lightbulb)TPM and IMA were separated out for this release.

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5224

    TPM and IMA were separated out for this release.


    15simp-doc, tlog
    Review Session Auditing and links  in read the docs to make sure TLOG documentation is correct.

    (grey lightbulb)(grey lightbulb)TLOG was added for this release

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5224

    TLOG was added for this release


    16simp-doc, x2go, mate
    Review How to manage Workstation entry for updates for mate and x2go

    (grey lightbulb)(grey lightbulb)X2go, Mate  were added for this release

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5224

    X2go, Mate  were added for this release


    17simp-doc, hiera 5 and puppet 5 upgrade
    Review docs to make sure the hiera data directory has been updated to reflect changes.  And that the version of puppet currently supported has been updated to puppet 5. 

    (grey lightbulb)(grey lightbulb)hiera and puppet updated to version 5.

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5224

    hiera and puppet updated to version 5.


    18simp-integration_test
    SIMP server upgrade from RPM using the upgrade instructions.  Document any steps that are needed outside the ordinary upgrade.  Then, if there are additional steps, have someone else use the docs to do the upgrade to verify.

    (lightbulb)

    (lightbulb)

    (lightbulb)


    (lightbulb)

    (question)


    (question)
    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5383

    See simp-integration_test for automated upgrade test.

    this should be done for servers in both FIPS and NOFIPS mode.

    A straight `yum update` following the General Upgrade instructions for RPM-based SIMP installations will kill puppetserver and prevent it from restarting.   (This is 

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5021
     + 6.1.0)

    The fix is procedural: upgrade steps (and fixes)  in "Upgrading from SIMP-6.1.0 to SIMP-6.2.0" procedure 

    Pay attention to Auditd upgrade because it is a major version upgrade.


    19PE

    Upgrade PE SIMP server from 6.2

    puppet 4 to puppet 5  (PE 2018.1) and simp 6.3

    (lightbulb)


    (lightbulb)(lightbulb)(lightbulb)Not sure what OS it is most important to test this on.

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5536
    keySIMP-5536

    Not sure what OS it is most important to test this on.


    20RPMs and Released Modules

    Dogfooding



    (grey lightbulb)(grey lightbulb)

    Or "drink your own home brew", which ever you prefer.

  • CentOS 7: SIMP server and SIMP client
    • CentOS 6: SIMP client only

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5577
    keySIMP-5577

    Or "drink your own home brew", which ever you prefer.

    • CentOS 7: SIMP server and SIMP client
    • CentOS 6: SIMP client only


    21Final RPM testingRPMs have been uploaded to packagecloudRun the simp-core install_from_rpm test and versify that it passes.   This will pull the packages from package.io and do an install.(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5534


    in simp-core run

    • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm]
    • beaker:suites[install_from_rpm]
    • beaker:suites[install_from_rpm,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_rpm,el6_server]
    and then do it all again with BEAKER_fips=yes
    • _server]

    and then do it all again with BEAKER_fips=yes


    22Puppet Forge TestingAfter modules have been tag with an annotated tagrun the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge(lightbulb)(lightbulb)(lightbulb)

    (lightbulb)




    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-553422Puppet Forge TestingAfter modules have been tag with an annotated tagrun the simp-core test install_from_core_module test to test the modules pushed to Puppet Forge(lightbulb)(lightbulb)(lightbulb)(lightbulb)
    keySIMP-5538

    in simp-core run

    • beaker:suites[install_from_core_module]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module]
    • beaker:suites[install_from_core_module,el6_server]
    • SIMP_BEAKER_OS=oel beaker:suites[install_from_core_module,el6_server]

    and then do it all again with BEAKER_fips=yes

    Jira LegacyserverSystem JIRAcolumnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolutionserverId45e63be1-2551-37a6-b050-86a38172f71dkeySIMP-5538
    • ,el6_server]

    and then do it all again with BEAKER_fips=yes



    Tests - Regression/Integration Testing

    The tests in this section are regression tests to ensure unmodified capabilities still function.  These tests should use the packer boxes created in the New/Changed Feature tests, were applicable.  Note that most of these test should eventually be automated by simp-packer, simp-integration_test, or simp-core tests.


    		TestEL6EL7StatusNotes
    1

    Verify non-standard BIOS boot options from the ISO:

    •   choose own partitions
    •   minimum installation
    (SIMP, FIPS and Encryption are all tested by Packer.)



    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5551

    		(SIMP, FIPS and Encryption are all tested by Packer.)
    2

    Verify all boot Options in UEFI mode from ISO:

    • FIPS
    • no-FIPS
    • FIPS+Encrypted
    • FIPS+choose own partitions
    • minimum


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5587

    3PXE Boot Testing BIOS

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5579

    4PXE Boot Testing UEFI

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5580

    5Bootstrap simp-lite scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5588

    6

    Verify simp-lite operations:

    • login operations (PAM, LDAP, local user)
    • NFS operations (home directory)
    • logging operations (rsyslog)
    • auditing operations


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5586

    7Bootstrap poss scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5588

    8Bootstrap run-once scenario

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5588

    9Verify ability to switch FIPS
    from on to offSwitching
    from
    off to
    on
    is known to have problems because of ciphers.
    to off

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5605

    		Switching from off to on is known to have problems because of ciphers.
    10Verify ability to turn on and off auditing and selinux

    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5605

    11

    General Review of documentation:

    • Getting Started
    • User Guide
    • HOW TOs
    • FAQS




    12simp-utils:

    executables that are not tested otherwise work as advertised

    • unpack_dvd

    The following have acceptance tests that simply need to be executed:

  • gen_ldap_update
    • updaterepos


    Jira Legacy
    serverSystem JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId45e63be1-2551-37a6-b050-86a38172f71d
    keySIMP-5553

    The following have acceptance tests that simply need to be executed:

    • gen_ldap_update
    • updaterepos