/
How to restructure the RPM installation sequence to be more r10k/Code Manager friendly

How to restructure the RPM installation sequence to be more r10k/Code Manager friendly

Apr 01, 2019

Add your comments directly to the page. Include links to any relevant research, data, or feedback.

Status

Decided

Impact

HIGH

Driver

@Trevor Vaughan 

Approver

 

Contributors

@Judith Johnson @Liz Nemsick @Chris Tessmer @Jeanne Greulich @Brandon Riden (Unlicensed) @Dylan Cochran (Unlicensed) @Nick Markowski @Nick Miller @Kendall Moore @Adam Yohrling (Unlicensed) @Steven Pritchard

Informed

 

Due date

Jan 25, 2019

Outcome

Option 1 will be used unless significant issues are discovered during implementation.

These decisions have been updated

Portions of this decision have been updated or obsoleted by later decisions:

 

Background

The current RPM method, using an rsync-based simp-adapter, for installing the SIMP puppet modules began as a reasonable solution. However, r10k and the PE Code Manager have solidified as the de-facto method for installing puppet modules onto systems.  As such, while we have attempted to be careful, we have found that our current installation method is lacking in sustainability with an r10k workflow and it does not encourage users to migrate to a more DevOps-friendly workflow in the future.

The goal of this decision is to determine a path forward for release in 6.4.0.

Please note that it is understood that the proposed solutions may be complex. The complexity is generally present to give users a seamless path towards an operational system.

Please ensure that feedback is constructive in nature and provides a proposed solution to the perceived issues.

Relevant data

The new system should meet the following requirements:

  1. Does not assume that any infrastructure besides the initial SIMP server exists

    1. This includes the Internet

  2. Supports validation of installed components for auditing purposes

    1. This is the original reason that RPMs were rolled

  3. Is able to be understandably migrated to an r10k system as the user desires

  4. Does not destroy existing environments

    1. This can be achieved using the deployment option for purge_levels in the configuration: https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#deployment

  5. Cannot run "unnecessary services"

    1. Discussion may dictate that new services are necessary but each service that we add is an additional level of documentation, justification, and maintenance that is added to the stack.

Options considered

Option 1 - RPMs with Local r10k

Description

This model works around the concept that users should always be able to revert to any given module that has been installed by RPM at any time and that the entire workflow should be based around a standard Puppetfile. The impact to the current system is minimal in that only the simp-adapter will really need to change. All other components can use the same installation methods and hooks as they always have.

Module Installation

The idea is that a module installation will use the following procedure:

  1. RPM installation to /usr/share/simp/modules/<author>/<modulename>

    1. This will always contain the latest version of the module by that author

  2. Trigger of simp-adapter that will cause the following actions to be performed:

    1. Create /usr/share/simp/git if it does not exist

      1. This is our "local git repo" for lack of other options

    2. Create /usr/share/simp/git/puppet_modules if it does not exist

      1. This is the housing of all puppet modules that get installed

    3. Create a bare git repository at /usr/share/simp/git/puppet_modules/<name from metadata.json> if it does not exist

    4. Update the master branch of the repository to be the contents of the RPM

    5. Add a git tag to the repository that matches the version number in the module's metadata.json file

    6. Run an r10k deploy on the simp environment (or configured environment of your choice) if configured to do so

      1. This would use the simp-vendored-r10k packages since no other ones are guaranteed to actually exist on the system.

      2. PE installations would be configured to use puppet code deploy by default

  3. Upon RPM removal, the git repository will remain so that environments do not break.

Example:

Module installation example
[root@host ~]# yum install pupmod-simp-issue [root@host ~]# ls /usr/share/simp/modules/simp/issue/ CHANGELOG Gemfile LICENSE README.md Rakefile files manifests metadata.json spec [root@host ~]# ls /usr/share/simp/git/puppet_modules/pupmod-simp-issue/ HEAD branches config description hooks info objects refs [root@host ~]# git clone /usr/share/simp/git/puppet_modules/pupmod-simp-issue Cloning into 'pupmod-simp-issue'... done. [root@host ~]# ls pupmod-simp-issue/ CHANGELOG Gemfile LICENSE README.md Rakefile files manifests metadata.json spec [root@host ~]# cd pupmod-simp-issue/ [root@host pupmod-simp-issue]# git log commit e4d4e475cea8c2be590b6ac158fc998cc8dee293 Author: SIMP Adapter <simp_adapter@localhost> Date: Fri Jan 11 22:08:33 2019 +0000 Import of 0.0.4 [root@host pupmod-simp-issue]# git tag 0.0.4

 

Control Repository Installation

This is a bit more difficult since control repositories tend to be quite personal to the user and require more live malleability.

The proposal for a local control repository follows the general convention of having "hands off" materials in /usr and variable materials in /var. In this case, a directory at /var/simp/git would correlate with the /usr/share/simp/git space for items that are not to be modified (not saying that they can't be modified, just that they should not be).

The SIMP control repository would be housed at /usr/share/simp/git/control_repository and the simp RPM itself would take responsibility for updating this in a manner similar to that of the puppet modules and simp-adapter. The major difference here is that utmost care must be taken to not surprise the user with new environments or updates.

The exact workflow for how this will work will probably need to be based on usability experiments once we get past the general idea outlined in this section.

  • Tags prefaces with simp- will be ours to manage authoritatively and will be documented as such

  • The production and simp environments will be created if they do not exist and will be matched to each other out of the box

    • Unsure if we should attempt clean updates to the environments or if we should always make the user perform the update via git

  • Like modules, tags will be made each time SIMP is updated and will contain a clean copy of the release content

Example:

Module installation example
[root@host ~]# yum install simp-6.4.0-0 ... Installed: simp.x86_64 0:6.4.0.el7 [root@host ~]# ls /var/simp/git/control_repository/ HEAD branches config description hooks info objects refs [root@host ~]# git clone /var/simp/git/control_repository -b production Cloning into 'control_repository'... done. [root@host ~]# cd control_repository [root@host control_repository]# ls LICENSE Puppetfile README.md data environment.conf hiera.yaml manifests [root@host control_repository]# cat Puppetfile # Ideally this can simply be changed to the address of the git server # (even the Internet servers) if desired module_source = '/usr/share/simp/git' ... mod 'simp-simp', :git => "#{module_source}/pupmod-simp-simp", :tag => '4.6.0' mod 'simp-simplib', :git => "#{module_source}/pupmod-simp-simplib", :tag => '3.11.1' ... [root@host control_repository]# git tag simp-6.4.0-0 [root@host control_repository]# git branch -v --all * production 0cfacbc SIMP 6.4.0 remotes/origin/production 0cfacbc SIMP 6.4.0 remotes/origin/simp 0cfacbc SIMP 6.4.0

Pros

Allows users to move everything to a git server, by simply bundling up the repos and importing them to their system of choice

 The transition to git and r10k is transparent to users who simply follow the latest SIMP releases

Follows current best practice in terms of control repository deployment

Gives users a 'soft' approach to getting used to advanced system management

Cons

Is relatively complex (though less mysterious than the current method since we are using community-understood mechanisms)

A lot of precise and tested documentation is going to need to be provided for working with this workflow. This is probably the case no matter what we do if we move to r10k since it now requires some sort of git-based workflow.

Action items

@Chris TessmerFeb 8, 2019 Flesh out and re-upload Option 2

Outcome

After a month waiting for input and various side discussions, we'll move forward with attempting Option 1. There was a partially completed Option 2 that we can assess once it is provided.

 

Comments