SIMP-3479 - Getting issue details... STATUS

Managing Multiple Puppet environments (without SIMP)

Control repositories

Outside of SIMP, Git-based control repositories have become the dominant technology to manage and deploy Puppet code.

At a high-level, a control repository workflow involves:

• A Puppetfile that defines a collection of Puppet modules and where to get them
• A (Git) Control repository, where each branch defines a complete Puppet environment (which includes a Puppetfile)
• A tool (r10k or PE Code Manager) to deploy Puppet environments from the control repository branches
• (Advanced, but preferred) Repository webhooks that automatically trigger environment deployments on the Puppet master(s) whenever a branch is updated.

The upshot is that a control repository branch defines everything needed to recreate a specific Puppet environment. 

Puppet environments as code

Defining everything needed to create a Puppet environment enables Puppet Environments to be handled like code:

• Adding or removing a git branch in the control repository will add or remove that environment on the Puppet master(s) 
  • (using r10k/Code Manager + webhook triggers)
• The Puppet environment change control process can be handled as merge/pull requests 
• Developers can test, diff, and review Puppet environment changes before promoting them into important environments
• As control repo branches are updated, CI pipelines can automatically trigger tests and master-side deployments

Multiple Puppet environments

Because 1 control repo branch = 1 Puppet environment, managing multiple Puppet environments is as simple as using git:

• Fork an existing branch to create a new temporary testing (canary) environment
  • After the canary branch is validated, merge it back into the original 
• Write access to important permanent environments can be controlled by protecting branches.

Scaling Puppet with Control Repository-based workflows

• r10k, Code Manager make it easy to deploy consistent Puppet environments across Puppet architectures that involve multiple masters
• Bolt makes it possible to manage remote "agentless" hosts by checking out an environment's modules (using bolt puppetfile install) before running bolt apply

SIMP + Multiple Puppet environments

Limitations and conflicts

It has been technically possible to use multiple Puppet environments and control repositories since SIMP 5.0 (SIMP 5.0-6.3).  However, in practice this has been cumbersome and easy to get wrong.  Getting it wrong risks accidentally leaking or losing site data (such as passgen secrets and PKI files).  Getting it right requires an in-depth knowledge of Puppet environments and SIMP's "extra" environments.  Even when handled well, there are still significant limitations—for instance, it still won't be feasible to host redundant (compile) masters behind a VIP unless site administrators arrange some method to keep the SIMP "extra" environment data in sync across each server.

The recent 6.4.0 release (SIMP 6.4) updated SIMP to safely support Puppetfile deployments (e.g., r10k puppetfile install from a Puppet environment directory) in a single, permanent Puppet environment (i.e., production).

However, there are still two major problems when using SIMP 6.4.0 with multiple Puppet environments and control repositories:

Issue 1: SIMP automation alters files under the Puppet and SIMP environment directories 

1. SIMP 5.0-6.3 RPM updates and simp_rpm_helper could modify existing files Puppet and SIMP environment directories
2. SIMP 5.0-6.3 User-initiated tools like simp config could modify existing files Puppet and SIMP environment directories
3. SIMP 6.4 PARTIALLY FIXED The user-initiated tool simp environment new can ensure that there is a corresponding secondary and writable environments
   1. This is a safety improvement from earlier releases, because it will only alter environment files when the user initiates it
   2. However, the simp environment tool is only partially implemented, and the rest cannot be implemented safely.
   3. As things stand, this means SIMP 6.4.0:
      1. Safely supports SIMP extra data with the Local (Puppetfile-only) deployment scenario with a single environment (production)
      2. Does not safely support SIMP extra data with the Control Repository deployment scenarios (without additional conventions and limitations)

Issue 2: SIMP expects there to be separate Secondary and Writable assets to complement each Puppet environment

SIMP has always assumed that there will be two SIMP-specific "environment" data directories to complement each Puppet environment directory

1. SIMP (ALL) "Secondary" data/assets under /var/simp/environments/$environment/.  
   These are files too sensitive and/or too large to check into git in plaintext, maintained by site admins.
   1. "Site Files" modules, like pki_files (keydist) and krb3_files. 
   2. The FakeCA support script, including the FakeCA's private key
   3. RSync directories, used by the simp::server::rsync_shares profile to serve files for various modules.
      
      
2. SIMP (ALL) "Writable" data/assets under under /opt/puppetlabs/server/data/puppetserver/simp/environments/$environment/
These are secrets, read by certain Puppet functions while compiling catalogs (and automatically generated if they are missing).
   There are only two Puppet functions in SIMP that use the writeable environment directory:
   1. simplib::passgen()
      • a popular function, used to generate secret passwords
      • Used in 14 SIMP modules—usually (but not always) exposed as a parameter default
      • Secrets read by this function can also be generated by the simp passgen CLI tool
   2. ssh::autokey()—this function is unused by the stock SIMP modules, but is provided to allow local admins to automatically set up SSH keys as needed.

SIMP 6.4.0: Deploy single environment from a Puppetfile

SIMP 6.4.0 addressed many of tool and RPM-related problems by making Puppetfile-based module deployments the preferred method to deploy modules.

• Puppetfile-based deployment tools like r10k or Code Manager and now preferred to deploy SIMP's Puppet modules
  • To prevent conflicts with these tools, SIMP no longer interferes with any files under the Puppet or SIMP environment directories, unless directed to by the user.
• SIMP module RPMs now install tagged versions into local git repositories, for use in local Puppetfiles
  • The simp puppetfile tool was introduced to automatically generate Puppetfiles based on the current module RPMs.
• The simp environment new tool was introduced to help users ensure that corresponding SIMP and Puppet environments are created.

Improvements focused on making it simple to use Puppetfiles in a single environment, with multiple 

made it simple to deploy environments, but they didn't make them safe.  

During the development of SIMP 6.4.0, it became apparent that the simp environment workflow would have problems down the road:

• Making a new SIMP writable and secondary environment for ever Puppet environment was arbitrary 
  • The dependence on $environment made it necessary to adopt workarounds like `simp environment new [--copy|--link]` when environments used the same (or similar) resources
  • Linked 
• It would be too easy to permanently lose data from linked environments with the proposed simp environment rm command

Proposed SIMP 6.5+

Legacy 6.4 assets in SIMP 6.5+