Automate acceptance testing for SIMP ISOs

Description

As SIMP grows, it becomes harder to adequately test SIMP releases using humans:

  • Humans must build ISOs and then stage test environments.

    • This results in inconsistent builds ( rake build:auto helps a lot, but doesn't prevent inconsistencies in the local repos)

    • The completeness of the test environments rely on the expertise of the tester

    • It takes far too many man-hours to stage the environments for multiple test scenarios from scratch

  • Humans must stage and run their own tests

  • Our testing methodology is very human-driven (inconsistent), and we miss things (especially after last-minute fixes)

    • We don't test enough scenarios

    • Close to release, it's impossible to for humans to rebuild and adequately regression test enough scenarios after new fixes are applied

So, We need an automated way to test ISO releases that can:

  • Install an ISO

  • Test various SIMP ISO boot types (e.g., simp, linux-min)

  • Test various SIMP ISO boot options (e.g., simp_disk_crypt, fips=0)

  • Run simp config on the console

  • Run through various SIMP installation configurations (e.g., LDAP, FIPS, upgrades)

  • Run through various SIMP client configurations

  • Kickstarts

  • Major client/server applications

Our approach is (for each SIMP release ISO/candidate):

1. Build/obtain the SIMP ISO
2. Use packer to automatically build test-ready SIMP VMs

  • (This also validates the installation procedure)
    3. Using Beaker to automate integration tests.
    4. Pipeline these steps

simp-packer

simp-packer builds SIMP Vagrant boxes from SIMP ISOs. This the current roadmap for simp-packer Vagrant boxes:

simp-integration_tests

simp-integration_tests uses beaker-rspec to run automated integration test suites for various SIMP scenarios

Acceptance Criteria

None

Story Points

None

Components

Priority

High

Epic Name

Automate SIMP ISO tests
Configure