Public SIMP yum repository structure and usage

Add your comments directly to the page. Include links to any relevant research, data, or feedback.

This decision log is to go over the proposed changes to simp-simp and the simp yum repos below:

Trevor Vaughan, Chris Tessmer, Dylan Cochran, Kendall Moore, Judith Johnson

Status
IN PROGRESS
Impact

HIGH

Driver
Approver
Contributors
Informed
Due date
Outcome

Background

Currently, the pupmod-simp-simp module hard-codes its internet repos to point to SIMP's PackageCloud repositories.

The PackageCloud repos contain a mix of latest, stable, and unstable RPMs.  This has caused issues in the past, with users getting a mix of something like 6.2.0-0 and 6.3-BETA RPMs.  We want to to restructure these repos to separate packages intended for managing specific releases and development.

Relevant data

  • SIMP Users (site admins) must be able to use a repo where packages are pinned to a specific SIMP super-release (e.g., 6.3.0-0)
  • SIMP Users (site admins) may want access to a stable repo that is pinned to the latest SIMP X.Y super-release (e.g., 6.3, which is identical to 6.3.3-0, if that is the latest release)
  • SIMP Users (site admins) may want access to a best-effort "rolling" repo with the latest SIMP X.Y super-release + additional "stable**" RPM updates (e.g., 6.3.0-0 + updates to pupmod-simp-auditd)
  • SIMP Developers, CI systems, and nightly builds must have access to packages intended for an upcoming release (e.g., the next release after 6.3.3, the upcoming 7.0.0)



** = The criteria for what "stable" means in a Rolling release is TBD.



Options considered


Option 1: Initial proposalOption 2: After 21 Feb 2019 REC DiscussionOption 3: After 07 Mar 2019 REC Discussion
Description

yum repo structure:

./
└── simp/
     └── yum/
          ├── 6/
          ├── 6.3/
          ├── 6.3.0/
          ├── 6.3.1/
          ├── 6.4/
          ├── 6.4.0/
          └── unstable/

  • Info for the structure:
    • 6: Rolling update of the latest stable 6 release, currently 6.3.1, eventually possibly 6.3.2, then 6.4.0, and so on, up until SIMP 7
      • Also contains all previous RPM versions for all previous SIMP 6 releases
    • 6.3: Rolling Latest 6.3 release, in this case 6.3.1
      • Also contains all previous RPM versions for all previous SIMP 6.3 releases
    • 6.3.1: Hard lock on specific components for 6.3.1
      • Only contains RPM versions for 6.3.1
    • Unstable: latest RPMs, not recommended
      • Contains all RPMs for previous builds as well as current Alpha/Beta/RC components.

Internet_Repos Update:

The update to simp-simp would be to fix the issues caused by PackageCloud updates. This would add a new variable for simp minor version, called simp_repo_version, and update the URLs from PackageCloud to the SIMP download server. There is currently a simp_version function that grabs the full SIMP version and EL Version (something like 6.3.0-0.el7). The new variable would use something like $simp_repo_version => $simp_version.split('.')[0..1].join('.') to grab '6.3' by default. Then this would be changeable in hiera to 6 or 6.3.1, etc. 

As with the yum repo structure, if a user keeps the default version, they will get all patches for SIMP 6.3, but never update to 6.4, unless they change the variable. If they change it to 6.3.1, they'll be locked at 6.3.1 and never get updated, if they change it to 6, they'll get every minor and patch update to SIMP 6.

Yum Install Update:

These same principles would apply to the SIMP yum install method. Repo files would be added the same way, and the user would be able to run yum install simp -v <simp_repo_version> with the same effects as above occurring with the install (6 would install latest SIMP 6 release, 6.3 would install latest version of SIMP 6.3, 6.3.1 would install 6.3.1 specifically). 

Proposed Changes

yum repos:

  • Create structure as listed above, and ensure all files are accessible, and all major/minor folders are matched with the latest patch release for each
  • Ensure archives are cleared out and anything pointing to them is updated
  • Create auto-script to ensure that major and minor folders are updated as new folders are created (6 and 6.3 get updated when 6.3.2 gets added, etc)

simp-simp:

  • Remove simp_internet_repos and simp_internet_dependencies manifests
    • purge simp_release_slug variable (seems unused other than above manifests)
  • Create simp_repo manifest
    • Create simp_repo_version variable from simp_version... $simp_repo_version => $simp_version.split('.')[0..1].join('.')
    • Ensure variable can be set via hiera
  • Update simp_platform RPM to include all repos and be able to install based on passed variable



Modeled at: https://download.simp-project.com/pub/simp-test/simp/yum/simp/

./
├── releases/       # <-- stable SIMP super-releases
│   ├── 6.1.0-0/
│   │   ├── EL6/
│   │   │   ├── core/
│   │   │   ├── dependencies/
│   │   │   └── GPGKEYS/
│   │   ├── EL7/...
│   │   └── SRPM/
│   │   └── repodata/
│   ├── 6.2.0-0/...
│   ├── 6.3.0-0/...
│   ├── 6.3.1-0/...
│   └── 6.4.0-0/...
|
├── rolling/        #  <-- RPMs for latest 
│   │               # of each SIMP X.Y
│   │               # super-release

│   ├── 6.0/
│   │   ├── EL6/
│   │   ├── EL7/
│   │   └── SRPM/
│   ├── 6.1/...
│   ├── 6.2/...
│   ├── 6.3/...     # (for this example) │   │               # identical to
│   │               # 
`releases/6.3.1-0`
│   └── 6.4/...
└── unstable/
    ├── EL6/
    │   ├── core/
    │   ├── dependencies/
    │   └── GPGKEYS/
    ├── EL7/...
    └── SRPM/




Modeled at: https://download.simp-project.com/pub/simp-test/simp/yum/simp/

./
├── releases/       # <-- stable SIMP super-releases
│   ├── 6.1.0-0/
│   │   ├── EL6/
│   │   │   ├── core/
│   │   │   ├── dependencies/
│   │   │   └── GPGKEYS/
│   │   ├── EL7/...
│   │   └── SRPM/
│   │   └── repodata/
│   ├── 6.2.0-0/...
│   ├── 6.3.0-0/
│   │   ├── EL6/
│   │   ├── EL7/
│   │   └── SRPM/
│   ├── 6.3.1-0  # <-- latest SIMP 6.3.X super-release
│   ├── 6.3/     # <-- identical to `releases/6.3.1-0`
│   └── 6.4.0-0/..
├── rolling/     #  <-- "stable" RPMs assumed to be in next 6.3.z
│   │            #      (The repo is expected to be "stable**")
│   │            #      (** = testing criteria TBD)

│   ├── 6.0/
│   │   ├── EL6/
│   │   ├── EL7/
│   │   └── SRPM/
│   ├── 6.1/...
│   ├── 6.2/...
│   ├── 6.3/...     # <-- releases/6.3.1-0, plus a change to
|   |               #     pupmod-simp-auditd

│   └── 6.4/...
└── unstable/
   ── 6.4/    # <-- separated by development work
   |    │      #     (used by developers & nightly builds)
   |    ├── EL6/
   |    │   ├── core/
   |    │   ├── dependencies/
   |    │   └── GPGKEYS/
   |    ├── EL7/...
   |    └── SRPM/
   ├── 6/...
   ── 7/...

  • In contrast to option 2, the rolling/ directories may also include "stable" RPMs that have been updated since the latest X.Y release
Pros and cons

(info) Initial proposal

(plus)

(minus)

(plus) Provides stable releases repositories for each super-release 

(plus) Provides "stable" rolling repos to get the latest super-release in each X.Y family

(minus) The unstable repository still lumps everything together into a single bucket, which may not be useful for developers working parallel releases

(plus) Provides stable releases repositories for each super-release 

(plus) Provides rolling repos to get the latest X.Y super-release with additional "stable" RPM updates

(plus)  The unstable repository supports parallel development branches and nightly builds

(minus)  The criteria to determine if an RPM is "stable" enough to add to the rolling repo isn't clear yet

(minus) The rolling repo assumes the next super-release will be a .z update

Estimated cost
MEH
WAT

¯\_(ツ)_/¯

Action items

Older action items (for option 1)

  • Decide on final name of variable (proposed simp_repo_version)
  • Figure out yum options for passing the simp_repo_variable to the repo file (for yum installs)

Outcome