6.6.0 SIMP RELENG Summary (2021-10-28)
- 1 Overview
- 2 RELENG Ecosystem
- 2.1 GitHub Actions
- 2.1.1 SIMP Puppet modules
- 2.1.2 SIMP RubyGems
- 2.1.3 Other SIMP assets
- 2.1.4 Notable edge cases
- 2.2 TODO GitLab CI
- 2.3 TODO SIMP RELENG Tools
- 2.3.1 TODO Rake tasks
- 2.3.2 TODO Puppetsync
- 2.3.3 TODO bolt-pulp3
- 2.3.4 TODO releng-misc Bolt project
- 2.1 GitHub Actions
- 3 TODO Lifecycles
Overview
This document provides an overview of the current SIMP RELENG ecosystem as of 10/28/2021 as it supports the upcoming 6.6.0 release.
RELENG Ecosystem
SIMP CE code is managed by repositories under the GitHub http://github.com/simp organization. Contributors fork repositories and contribute changes using Pull Requests (PRs). Each contribution’s PR encapsulates its own reviews, CI/CD workflows, and any discussion or patches until it is approved and merged into the upstream repository.
GitHub Actions
GitHub Actions (GHA) automates the CI/CD for most repositories that are included in/support the SIMP super-release. This generally includes:
On PRs and merges:
Running automated checks, linting, and tests
On PRs:
Triggering GLCI pipelines
(merging will auto-trigger GLCI as the new commits are mirrored)
On annotated Tags:
Creating a GitHub release
Building/publishing packages (e.g., Puppet modules, Ruby Gems) to upstream repositories
Building/signing/attaching RPMs, either a.) as a follow-on to a tag-driven GitHub Release, OR b.) from a manual GHA trigger.
RPMs are attached to the relevant GitHub release as downloadable assets
Each repository’s GHA workflow configurations are kept under .github/workflows/
.
Common workflows are managed by Puppetsync as standardized assets and should not be edited directly. Puppetsync-managed workflows identify themselves in their header comments.
Some repositories have their own unique workflows. These files are maintained directly and are not managed by Puppetsync. Puppetsync-managed workflows identify themselves in their header comments.
SIMP Puppet modules
GHA workflows are identical across all SIMP Puppet modules, managed by Puppetsync (as type pupmod
).
SIMP RubyGems
Most GHA workflows for RubyGems are managed by Puppetsync (as type rubygem
)
RubyGem workflows may have several variations, particularly the tag-to-release workflows: depending on the gems, it may need to publish to rubygems.org, build an RPM, or just release to GitHub.
Unique workflows are managed per-repository.
Other SIMP assets
Other SIMP assets (e.g., simp-utils, pkg-r10k) may share common GHA workflows managed by Puppetsync (as type simp-unknown
) and/or unique workflows that are managed per-repository.
Notable edge cases
Using GHA workflows to build RPMs for non-SIMP forked repo mirrors
Forked repo mirrors don’t use SIMP GHA workflows, so new tags to don’t trigger relases and RPM builds.
However, the release_rpms.yml workflow (a standardized asset in every repo that builds its own RPMs) is designed to work as a manual action that can release/build RPMs for any other SIMP repository (when provided a token with sufficient scope).
simp-core
simp-core currently has no GHA workflows.
As a result, PRs and merges do not automatically trigger GLCI pipelines for acceptance/integration tests.
TODO GitLab CI
TODO SIMP RELENG Tools
TODO Rake tasks
TODO Puppetsync
TODO bolt-pulp3
TODO releng-misc Bolt project
releng::download_assets_from_puppetfile
TODO Lifecycles
el8 modularity
TODO RPM lifecycle
GitHub (UNSTABLE-2) → UNSTABLE → STABLE
TODO RPMs, GPG keys, and Yum repositories
TODO Building and Signing RPMs
Updating the Download Server Repositories
Updating experimental repositories with a SIMP release’s GitHub RPMs
Running the following Bolt plan (from releng-misc) will download RPMs from GitHub releases and stage them local yum repositories:
bolt plan run releng::createrepos_from_puppetfile_assets \
--inventory inventory--github-only.yaml
The plan:
Determines what to download by referencing
Puppetfile.pinned
in simp-core’smaster
branch:A
:tag
's corresponding GitHub releaseThe HEAD-most tagged GitHub release along a repo’s tracking
:branch
NOTE: The plan looks a maximum of 30 commits back to find the HEAD-most release
Downloads all assets from each repo’s GitHub release into a
_release_assets/
directoryCreates four subdirectories containing yum repositories and hard-linked RPMs:
_release_assets/el8/
_release_assets/el7/
_release_assets/el8.src/
_release_assets/el7.src/
These directories can be transferred to the SIMP download server and used during development.
Uploading SIMP RPMs to the download server
Publishing to the download server is currently manual process.
Use rsync -n
to check what will change before uploading RPMs:
rsync -avzn --delete _release_assets/el8/ \
enterprise-download.simp-project.com:/data/community-download/simp/yum/experimenta
l/6/el/8Server/x86_64/simp/
After confirming the changes, run the command again without the -n
Updating a super-release’s dep repositories with bolt-pulp3
Currently only the repo_packages.yaml file for EL8 is maintained. It has recently been moved to simp-core. Read the project’s README and`bolt plan show` for instructions.
Manually triggering GHA RPM builds
The release_rpms.yml workflow will release/build/sign/attach RPMs for a component’s GitHub release. It is automatically triggered by its repo's tag_deploy*.yml
workflow, following its GitHub release. The workflow file is a standardized asset, and is present in every SIMP component repo that builds its own RPMs.
RPMs are built by running the rake task pkg:single
inside the SIMP build container
This is done to ensure that RPMs are built using the overrides in simp-core’s
build/rpm/dependencies.yaml
file.
RPMs are signed within a centos8 SIMP build container, using a GPG key that is only available to workflows running under the simp/ GitHub org.
Manually triggering GHA RPM builds
The release_rpms.yml workflow can be triggered manually from the GitHub web interface and API. This can be necessary when the target is a forked repo mirror without its own tag_deploy_*.yml
workflows, or an error prevented GHA from building RPMs during a component’s tagged release.
The workflow has many configurable options when triggered manually. Notably, it can be configured to release, build, and attach signed RPMs to tagged commites in any other repository under the simp/ org.
The manual trigger has the following requirements:
A GitHub token with sufficient scope
A target GitHub release, or at the very least, a tag commit (the workflow can optionally create a release from a pre-existing taga)
The repo must be able to build RPMs from simp-core, using
rake pkg:single
To manually trigger an RPM build:
From a SIMP component repo’s GitHub actions UI, select the RELENG: Build + attach RPMs to GitHub Release action
Click the “Run workflow” drop-down in the upper right
Fill out the relevant inputs, including an API token (and optionally, a different target repo)
Click “Run workflow”