Utilize simp-metadata as the build tool for SIMP

Background

Our goal is to utilize simp-metadata as the data structure and build tool for SIMP to allow for quicker, automated builds, as well as provide easier access to build information for developers and customers.

Relevant data

Currently our build system is a very manual, limited, and time consuming process. While the current method works for one-time builds for SIMP releases, there is no easy way to automate expand to achieve our goals of building nightlies, as well as expanding our deliverables to other methods such as AMI and VHD images. Simp-core relies on a lot of manual steps that can be tedious and time consuming. Further, the steps needed to determine “what version of X was in SIMP Y?” or other build data questions are more than what should be needed. Simp-metadata is a powerful tool that can already perform a lot of useful tasks including, but not limited to:

• Update versions of components in a release
• Add or remove components from releases
• Compare information between two SIMP releases
• Show information (version, url, ref, tag, etc) for a component, with release specific info
• Build, download, and create RPMs, Tarballs, ISOs, and Puppetfiles

With the ability to build a SIMP release, or individual components, with a single command, we can automate builds and start pushing nightly Tarballs, ISOs, and other deliverables into testing. This will allow us to release quicker, and more often, and discover bugs before the official release testing. On top of the benefits to our current deliverables (Puppetfile, Tarball, ISO, yum repo), it will allow us to add VHD, AMI, control repos, and other deliverables to our pipeline, without the addition of extra manual interaction.

The simp-metadata tool (delivered as an RPM built from rubygem-simp-metadata) is useful for both in house developers, as well as external customers and contributors. It utilizes a set of YAML files in a repository named 'simp-metadata'. The tool, by default, uses the upstream simp/simp-metadata, but can be used with a fork, or a locally cloned repo, making testing and development easy.

For development purposes, assuming you have already cloned simp-metadata to your own personal Github, you can copy X release to a new Y release, update component versions and remove/add components for Y release, and test the builds. This can be done locally on your machine, or in a container if you choose to use Docker. This allows for easy development and testing without the need for extra branches, or changes to the SIMP owned repository. Nightly YAML files are created automatically, making nightly builds easy to generate, and with all "official" builds (testable nightlies, pre-release builds, and final builds) being built in a Docker container on the download server, all files will be available for download as soon as the pipeline is complete (All files would be placed into dev/unstable folders until verifying that we are ready to release them).

The gitlab pipeline will contain environment checks, RPM check/build/sign/verify, tarball check/build, deliverable prep, deliverable build, test, and deploy stages. We would be able to add any deliverable builds and automated tests directly to the pipeline if and when needed. The manual RPM verify stage ensures that there is still manual interaction (for now) when RPMs are built and signed, before utilizing them in the actual build. 

simp-metadata would also be added to all releases of SIMP, allowing customers to utilize the tool. This will be useful if they want to know what changes were made between releases to see if they want to update, or (in future command additions) provide us with information for support. 

Options considered

Action items

• Full Test Cycle on an ISO built using simp-metadata
• Deprecate simp-core build tools
• Update open-source rubygem-simp-metadata to add build tools and updates
• Build Gitlab pipeline for builds

Outcome