Page Comparison

...

Backend logical structure

...

To facilitate implementations of this tree, key and folder names are restricted to sequences of alphanumeric, ‘.', ‘_', and ‘-’, and '/' characters, with where '/' is used as the path separator. Furthermore, when specifying the path to a key in an access operationfolder or key specification contains a path, the path cannot contain relative path subsequences (e.g., '/./' or '/../').

...

• Each backend has its own configuration.

• All backends Each backend configuration block must specify simpkv plugin type (e.g., ‘file’, ‘ldap’) and a user-provided instance identifier.

  • A plugin is a backend interface written in Ruby and conforming to an explicit plugin API and expected behavior. It is the logic that actually affects the keystore operation when that actually affects the keystore operation when a simpkv::* function is called during a Puppet catalog compilation. For the ‘ldap' plugin, this will be the software that modifies key/value pairs stored in an LDAP server.

  • The same plugin can be used for multiple backend instances.

  • The combination of plugin type and instance identifier uniquely identifies a backend instance.

• Each backend configuration block may specify additional, plugin-specific configuration (such as LDAP server URL and port, TLS configuration,…).

...

Internally, simpkv constructs a plugin object for each unique backend, and uses the plugin object to interface with it corresponding backend. When a simpkv::* function is called, an internal adapter calls the plugin’s corresponding API method with normalized arguments to affect the operation. The adapter then (de)normalizes the results of the operation and reports them back to the calling simpkv::* function.

For example, for a simpkv::put operation using a LDAP plugin, the sequence of operations is notionally as follows:

...

Then, for a simpkv::get operation using a LDAP plugin, the sequence of operations is notionally as follows:

...

Value normalization

One of the normalizations done by the simpkv adapter involves the value and optional, user-provided metadata associated with a key. In a simpkv::put operation, the simpkv adapter serializes a key’s value and optional metadata into a single JSON string and then sends that to the plugin for storage in the backend. Then, after a key’s information has been retrieved by a plugin during a simpkv::get or simpkv::list operation, the simpkv adapter deserializes each JSON string back into the key’s value and metadata objects before serving the results back to the calling function. This encoding of a key’s value an metadata into a single string with a known, parsable format is intended to simplify backend operations.

...

*‘Basic value’ ¹ ‘Basic value’ refers to a string, boolean, or numeric value.

**² 'Complex value’ value’ refers to an array or hash constructed from basic values.

³ simpkv currently only supports metadata hashes comprised of basic values.

***⁴ simpkv currently provides limited support for binary data.

• simpkv attempts to detect when the value is Puppet Binary type, transforms it into Base64 and records the transformation with ‘encoding' and 'original_encoding' attributes in the JSON. It then uses those attributes to properly deserialize back to the binary on a retrieval operation.

• simpkv does does not support binary data in arrays, hashes, or the metadata.

...

• metadata

...

• .

LDAP Directory Information Tree design

...

• There must be one LDAP backend DIT for all SIMP application data.

  • This is distinct from the DIT containing user accounts data.

  • Data to be stored must include simpkv data.

  • Data to be stored may in the future include other application data, (e.g., IP firewall data).

• The simpkv data must be a subtree of the DIT.

• The simpkv subtree must support partitioning the data into LDAP backend instances.

• The simpkv subtree must allow storage of per-LDAP-backend-instance global and environment-specific key/value entries.

  • Entries may be stored in subtrees within the LDAP instance subtree.

  • Each key/value entry must be a leaf node in the LDAP instance subtree.

  • The DistinguishedName Distinguished Name (DN) to each key/value entry throughout the entire DIT must be unique.

• The JSON value of the key/value entry must be stored in some form in the key/value entry.

  • The key/value entry may have an a single attribute containing the JSON-encoded value.

  • The key/value entry may have multiple attributes that map to the value’s JSON attributes.

• The tree must support efficient simpkv::get, simpkv::exists, and simpkv::list operations.

  • Folder and/or key objects may store data in attributes to leverage LDAP search capabilities.

  • The simpkv LDAP plugin should nothave to retrieve the entire tree or subtree in order to fulfill any of these operations.

• Any custom schema attributeType or objectClass will be specified with an Object Identifier (OID) below the official SIMP Object Identifier (OID).

...

• Use Organizations or Organizational Units to represent folders in a key path and other grouping (e.g., environments).

• Create a custom schema element with key name and value attributes to represent a key/value entry.

• Construct the DN for a key/value node using each part of the key path as a Relative relative DN (RDN) with the key name followed by sequence of RDNs where each one represents a folder in the key’s path.

So, for a key path production/app1/key1 the key/value pair could be found at the DN simpkvKey=key1,ou=app1,ou=production,ou=environments,<root DN for the backend instance>, where simpkvKey is an attribute of a simpkvEntry LDAP object used to store the key/value pair. Visually, this subtree in the DIT would look something like the following:

...

Unfortunately, there is a nuance in 389DS that complicates that simple mapping:

389DS instances treat DNs as case invariant strings.

So, the key paths production/app1/key1 and production/App1/Key1 both resolve to the same DN inside of 389DS, even though from simpkv’s perspective, they were intended to be distinct. This unexpected collision in the backend needs to be addressed either by simpkv or within the DIT itself.

...

The simplest design option enforces DN case invariance by requiring all the values of all attributes used in a DN for a key/value pair to be lowercase. In other words, change the experimental simpkv API to only allow lowercase letters, numerals, and ‘.', ‘_’, and ‘-’, and '/' characters for all key names, folder names, and plugin instance identifiers. Then, because each key’s DN is unique and case invariant, the simple mapping scheme described in 'Design Considerations’ can be used.

...

Below is an example of the Option 1 DIT in which simpkvEntry is a custom LDAP object class with simpkvKey and simpkvJsonValue attributes holding the key and value, respectively:

...

The second design option enforces DN case invariance without impacting the existing simpkv API. Its simpkv subtree has the same essential layout as that of Option 1, including the use of the ‘instances’, 'globals', and ‘environments’ grouping “folders”. However, in this design the

• The LDAP plugin transforms any problematic attributes that are to be used in a DN for a key/value pair to an encoded representation (e.g., hexadecimal, Base 64) . For example, with a hexadecimal transformation, all backend instance identifiers, key names, and folder names would be represented in hex, minus the ‘0x’ or ‘0X’ preface

...

• . (The Puppet environment does not require transformation, as Puppet environment names must be lowercase.) So, key paths production/app1/key1 and production/App1/Key1 would be mapped to
  simpkvHexId=61707031,simpkvHexId=6b657931,ou=production,ou=environments,...
  and
  simpkvHexId=41707031,simpkvHexId=4b657931,ou=production,ou=environments,...
  respectively, where simpkvHexId is an attribute of both

...

• an LDAP object used to represent backend identifiers/folders and

...

• an LDAP object used to store the key/value pair.

**Puppet environment names are not allowed to include uppercase letters.

...

• Each node with an encoded identifier

...

• RDN

...

• includes an attribute with the raw identifier.

...

• This additional information is necessary in order to Although this means a little more data must be stored in the DIT, this extra information will support external searches of the LDAP tree using the raw backend instance identifiers, key names, and folder names.

• Some ‘OrganizationalUnits’ in Option 1 would now be represented by a custom object that had encoded and raw identifier attributes.

• The custom class for the key/value nodes would have encoded and raw key attributes.

Below is an example of the DIT in which

• simpkvFolder is a custom LDAP object class with . In other words, users can search the LDAP tree without being forced to mimic the transformations done in simpkv::* functions.

Below is an example of the Option 2 DIT in which

• simpkvFolder is a custom LDAP object class with simpkvHexId and simpkvId attributes holding the transformed backend identifier/folder and raw identifier/folder, respectively

• simpkvEntry is a custom LDAP object class with simpkvHexId, simpkvId and simpkvJsonValue attributes holding the transformed key, raw key and JSON-formatted value, respectively.

Image Added

Recommendation

Schema and OID Subtree Design

...

The OID subtree will support multiple applications ('puppet', ‘snmp’).

...

Multiple plugin instances, possibly from different versions of the same module, can be instantiated at the same time in the puppetserver.

• Plugin implementations must be thread safe.

• Plugin implementations mreventing cross-puppet-environment contamination is essential

Plugins are written in Ruby and implement the simpkv plugin API. Pluginshave strict restrictions on their implementations in order to prevent cross-environment contamination, when different versions of the plugins are simultaneously loaded into the puppetserver during concurrent catalog compilations for nodes in different Puppet environmentsOption 1 is the recommended solution for the following reasons:

• It yields a DIT that is simple to understand and navigate.

• An API change is not unexpected for simp/simpkv, since it is still experimental (version < 1.0.0) and not enabled by default.

• SIMP can help users with the transition to lowercase key names for any existing simpkv key paths or simplib::passgen password names (whether using legacy mode or simpkv mode).

  • Any SIMP-provided module that uses simplib::passgen can be modified to ensure the password names are downcased.

  • The simplib::passgen function that uses simpkv can be modified to downcase existing password names that have any uppercase letters and then to emit a warning.

  • The script SIMP will provide to import any existing simpkv key entries or simplib::passgen passwords into a simpkv LDAP backend can check for uppercase letters in the destination key paths and either skip the import of the problematic entries, or convert to lowercase and warn the user of the conversion. Then, it would be up to the user to make any adjustments to the corresponding manifests.

OID Subtree Design and Custom LDAP Schema

Either option for the LDAP DIT for SIMP data requires at least one custom LDAP object class. The LDAP object class, in turn, must be specified by a unique OID. This section proposes a SIMP OID subtree design to support LDAP OIDs and then uses the OIDs in schemas for the two DIT options discussed above.

SIMP OID Subtree

SIMP has an officially registered OID, 1.3.6.1.4.1.47012, under which all OIDs for Puppet, SNMP, etc should reside. Once an OID is in use, its definition is not supposed to change. In other words, an OID can be deprecated, but not removed or reassigned a different name. So, the OID tree must be designed to allow future expansion.

Below is the proposed SIMP OID subtree showing the parent OIDs for attributes and class objects needed for the SIMP DIT.

...

LDAP Custom Schema

simpkv DIT Option 1

The proposed custom schema for the simpkv DIT option 1 is shown below. It has a custom object class, simpkvEntry, that is comprised of two custom attributes, simpkvKey and simpkvJsonValue.

• simpkvKey is a case-invariant string for the key (excluding path)

  • This is used as the final RDN of the DN for a key/value node.

• simpkvJsonValue is a case-sensitive string for the JSON-formatted value.

  • In the future, we could write a custom syntax validator for this attribute.

################################################################################
#
dn: cn=schema
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.1
  NAME 'simpkvKey'
  DESC 'key'
  SUP name
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.2
  NAME 'simpkvJsonValue'
  DESC 'JSON-formatted value'
  EQUALITY caseExactMatch
  SUBSTR caseExactSubstringsMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
objectClasses: (
  1.3.6.1.4.1.47012.1.1.1.1.2.1
  NAME 'simpkvEntry'
  DESC 'simpkv entry'
  SUP top
  STRUCTURAL
  MUST ( simpkvKey $ simpkvJsonValue )
  X-ORIGIN 'SIMP simpkv'
  )

The corresponding SIMP OID subtree is as follows:

...

simpkv DIT Option 2

The proposed custom schema for the simpkv DIT option 2 is shown below. It has two custom object classes and three custom attributes.

• Classes:

  • simpkvFolder is an object class for a node representing a backend identifier or folder.

  • simpkvEntry is an object class for a key/value node.

• Attributes:

  • simpkvHexId is an attribute that is a case-invariant, hex-encoded string for the backend identifier, folder or key (excluding path)

    • This is used as the final RDN of the DN for a node.

    • In the future, we could write a custom syntax validator for this attribute.

  • simpkvId is an attribute that is the raw, case-sensitive string for a backend identifier, folder or key (excluding path)

  • simpkvJsonValue is an attribute that is a case-sensitive string for a JSON-formatted value in a key/value node.

    • In the future, we could write a custom syntax validator for this attribute.

################################################################################
#
dn: cn=schema
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.1
  NAME 'simpkvHexId'
  DESC 'hex-encoded backend instance, folder, or key name'
  SUP name
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.2
  NAME 'simpkvId'
  DESC 'backend instance, key or folder name'
  EQUALITY caseExactMatch
  SUBSTR caseExactSubstringsMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.3
  NAME 'simpkvJsonValue'
  DESC 'JSON-formatted value'
  EQUALITY caseExactMatch
  SUBSTR caseExactSubstringsMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
objectClasses: (
  1.3.6.1.4.1.47012.1.1.1.1.2.1
  NAME 'simpkvEntry'
  DESC 'simpkv entry'
  SUP top
  STRUCTURAL
  MUST ( simpkvHexId $ simpkvId $ simpkvJsonValue )
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
objectClasses: (
  1.3.6.1.4.1.47012.1.1.1.1.2.2
  NAME 'simpkvFolder'
  DESC 'simpkv folder in which simpKvHexId represents the relative folder name in hex in the DN'
  SUP top
  STRUCTURAL
  MUST ( simpkvHexId $ simpkvId )
  X-ORIGIN 'SIMP simpkv'
  )

The corresponding SIMP OID subtree is as follows:

...

Technologies for Plugin Implementation

Requirements

• Plugins are written in Ruby and implement the simpkv plugin API.

• Plugins must be multi-thread safe.

• Plugins must be written to provide Puppet-environment isolation when executed on the puppetserver.

• Manifests that use simpkv::* functions must be able to be compiled with puppet agent, puppet apply or Bolt commands. This means the plugin code will run in JRuby in the puppetserver, run in the Ruby installed with puppet-agent, or run using the Bolt user’s Ruby into which the puppet gem is installed.

Options Considered

Recommendation

Option 1 without the auto-discovery mechanism is recommended for the following reasons:

• Options 2 and 3 require additional packaging in order to work on isolated networks for Bolt users. So, if you are going to require a Bolt user to install a package, anyways, might as well be an existing vendor package.

• The auto-discovery mechanism can be added after the initial implementation, because it is not required for the LDAP plugin to function.