Page Comparison

...

Function Name	Description
`simpkv::delete`	Deletes a key from a backend.
`simpkv::deletetree`	Deletes an entire folder from a backend.
`simpkv::exists`	Returns whether a key or key folder exists in a backend.
`simpkv::get`	Retrieves the a key’s value and any metadata stored for a key user-provided metadata from a backend.
`simpkv::list`	Returns a listing of all keys and sub-folders in a folder in a backend The list operation does not recurse through any sub-folders. Only information about the specified key folder is returned.
`simpkv::put`	Sets the a key’s value and optional, user-provided metadata for a key in a backend.

Backend logical structure

...

To facilitate implementations of this tree, key and folder names are restricted to sequences of alphanumeric, ‘.', ‘_', and ‘-’, and '/' characters, with where '/' is used as the path separator. Furthermore, when specifying the path to a key in an access operationfolder or key specification contains a path, the path cannot contain relative path subsequences (e.g., '/./' or '/../').

...

Each backend has its own configuration.
All backends Each backend configuration block must specify simpkv plugin type (e.g., ‘file’, ‘ldap’) and a user-provided instance identifier.
- A plugin is a backend interface written in Ruby and conforming to an explicit plugin API and expected behavior. It is the logic that actually affects the keystore operation when a simpkvthat actually affects the keystore operation when a simpkv::* function is called during a Puppet catalog compilation. For the ‘ldap' plugin, this will be the software that modifies key/value pairs stored in an LDAP server.
- The same plugin can be used for multiple backend instances.
- The combination of plugin type and instance identifier uniquely identifies a backend instance.
Each backend configuration block may specify additional, plugin-specific configuration (such as LDAP server URL and port, TLS configuration,…).

...

Internally, simpkv constructs a plugin object for each unique backend, and uses the plugin object to interface with it corresponding backend. When a simpkv::* function is called, an internal adapter calls the plugin’s corresponding API method with normalized arguments to affect the operation. The adapter then (de)normalizes the results of the operation and reports them back to the calling simpkv::* function.

For example, for a simpkv::put operation using a LDAP plugin, the sequence of operations is notionally as follows:

...

Then, for a simpkv::get operation using a LDAP plugin, the sequence of operations is notionally as follows:

...

Value normalization

One of the normalizations done by the simpkv adapter involves the value and optional, user-provided metadata associated with a key. In a simpkv::put operation, the simpkv adapter serializes a key’s value and optional metadata into a single JSON string and then sends that to the plugin for storage in the backend. Then, after a key’s information has been retrieved by a plugin during a simpkv::get or simpkv::list operation, the simpkv adapter deserializes each JSON string back into the key’s value and metadata objects before serving the results back to the calling function. This encoding of a key’s value an metadata into a single string with a known, parsable format is intended to simplify backend operations.

...

Value Type	Serialization Example
Basic value* ¹without metadata	`{"value":"the value","metadata":{}}` `{"value":10,"metadata":{}}`
Basic value with user-provided metadata****³	`{"value":true,"metadata":{"optional":"user","extra":"data"}}`
Complex value** ² with basic sub-elements with no user-provided metadata	`{"value":[1,2,3],"metadata":{}}`
Binary value*** ⁴ transformed by simpkv with no user-provided metadata	`{"value":"<Base64 string>","encoding":"base64","original_encoding":"ASCII-8BIT","metadata":{}"}`

*‘Basic value’ ¹‘Basic value’ refers to a string, boolean, or numeric value.

**² 'Complex value’ value’ refers to an array or hash constructed from basic values.

³ simpkv currently only supports metadata hashes comprised of basic values.

***⁴ simpkv currently provides limited support for binary data.

simpkv attempts to detect when the value is Puppet Binary type, transforms it into Base64 and records the transformation with ‘encoding' and 'original_encoding' attributes in the JSON. It then uses those attributes to properly deserialize back to the binary on a retrieval operation.
simpkv does does not support binary data in arrays, hashes, or the metadata.

...

metadata

...

.

LDAP Directory Information Tree design

...

There must be one LDAP backend DIT for all SIMP application data.
- This is distinct from the DIT containing user accounts data.
- Data to be stored must include simpkv data.
- Data to be stored may in the future include other application data, (e.g., IP firewall data).
The simpkv data must be a subtree of the DIT.
The simpkv subtree must support partitioning the data into LDAP backend instances.
The simpkv subtree must allow storage of per-LDAP-backend-instance global and environment-specific key/value entries.
- Entries may be stored in subtrees within the LDAP instance subtree.
- Each key/value entry must be a leaf node in the LDAP instance subtree.
- The DistinguishedName Distinguished Name (DN) to each key/value entry throughout the entire DIT must be unique.
The JSON value of the key/value entry must be stored in some form in the key/value entry.
- The key/value entry may have an a single attribute containing the JSON-encoded value.
- The key/value entry may have multiple attributes that map to the value’s JSON attributes.
The tree must support efficient simpkv::get, simpkv::exists, and simpkv::list operations.
- Folder and/or key objects may store data in attributes to leverage LDAP search capabilities.
- The simpkv LDAP plugin should nothave to retrieve the entire tree or subtree in order to fulfill any of these operations.
Any custom schema attributeType or objectClass will be specified with an Object Identifier (OID) below the official SIMP Object Identifier (OID).

...

Use Organizations or Organizational Units to represent folders in a key path and other grouping (e.g., environments).
Create a custom schema element with key name and value attributes to represent a key/value entry.
Construct the DN for the key as a Relative a key/value node using each part of the key path as a relative DN (RDN) with the key name followed by sequence of RDNs where each one represents a folder in the key’s path.

So, for a key path production/app1/key1 the key/value pair could be found at the DN simpkvKey=key1,ou=app1,ou=production,ou=environments,<root DN for the backend instance>, where simpkvKey is an attribute of a simpkvEntry LDAP object used to store the key/value pair. Visually, this subtree in the DIT would look something like the following:

...

Unfortunately, there is a nuance in 389DS that complicates that simple mapping:

389DS instances treat DNs as case invariant strings.

So, the key paths production/app1/key1 and production/App1/Key1 both resolve to the same DN inside of 389DS, even though from simpkv’s perspective, they were intended to be distinct. This unexpected collision in the backend needs to be addressed either by simpkv or within the DIT itself.

...

The simplest design option enforces DN case invariance by requiring all the values of all attributes used in a DN for a key/value pair to be lowercase. In other words, change the experimental simpkv API to only allow lowercase letters, numerals, and ‘.', ‘_’, and ‘-’, and '/' characters for all key names, folder names, and plugin instance identifiers. Then, because each key’s DN is unique and case invariant, the simple mapping scheme described in 'Design Considerations’ can be used.

...

Below is an example of the Option 1 DIT in which simpkvEntry is a custom LDAP object class with simpkvKey and simpkvJsonValue attributes holding the key and value, respectively:

...

The second design option enforces DN case invariance without impacting the existing simpkv API. Its simpkv subtree has the same essential layout as that of Option 1, including the use of the ‘instances’, 'globals', and ‘environments’ grouping “folders”. However, in this design the

The LDAP plugin transforms any problematic attributes that are to be used in a DN for a key/value pair to an encoded representation (e.g., hexadecimal, Base 64) . For example, with a hexadecimal transformation, all backend instance identifiers, key names, and folder names would be represented in hex, minus the ‘0x’ or ‘0X’ preface

...

. (The Puppet environment does not require transformation, as Puppet environment names must be lowercase.) So, key paths production/app1/key1 and production/App1/Key1 would be mapped to
simpkvHexId=61707031,simpkvHexId=6b657931,ou=production,ou=environments,...
and
simpkvHexId=41707031,simpkvHexId=4b657931,ou=production,ou=environments,...
respectively, where simpkvHexId is an attribute of both

...

an LDAP object used to represent backend identifiers/folders and

...

an LDAP object used to store the key/value pair.

**Puppet environment names are not allowed to include uppercase letters.

...

Each node with an encoded identifier

...

RDN

...

includes an attribute with the raw identifier.

...

This additional information is necessary in order to Although this means a little more data must be stored in the DIT, this extra information will support external searches of the LDAP tree using the raw backend instance identifiers, key names, and folder names.
Some ‘OrganizationalUnits’ in Option 1 would now be represented by a custom object that had encoded and raw identifier attributes.
The custom class for the key/value nodes would have encoded and raw key attributes.

Below is an example of the DIT in which

simpkvFolder is a custom LDAP object class with simpkvHexId and simpkvId attributes holding . In other words, users can search the LDAP tree without being forced to mimic the transformations done in simpkv::* functions.

Below is an example of the Option 2 DIT in which

simpkvFolder is a custom LDAP object class with simpkvHexId and simpkvId attributes holding the transformed backend identifier/folder and raw identifier/folder, respectively
simpkvEntry is a custom LDAP object class with simpkvHexId, simpkvId and simpkvJsonValue attributes holding the transformed key, raw key and JSON-formatted value, respectively.

Image Modified

Recommendation

...

It yields a DIT that is simple to understand and navigate.
An API change is not unexpected for simp/simpkv, since it is still experimental (version < 1.0.0) and not used enabled by default.
SIMP can help users with the transition to lowercase key names for any existing simpkv key paths or simplib::passgen password names (whether using legacy mode or simpkv mode).
- Any SIMP-provided modules module that uses simplib::passgen can be modified to ensure the password names are downcased.
- The simplib::passgen function that uses simpkv can be modified to downcase existing password names that have any uppercase letters and then to emit a warning.
- In the The script SIMP will provide to import any existing simpkv key entries or simplib::passgen passwords into an a simpkv LDAP simpkv backend , there can be a check for uppercase letters in the destination key paths . The script can and either skip the import of the problematic entries, or convert to lowercase and warn the user of the conversion. Then, it would be up to the user to make any adjustments to their the corresponding manifests.

OID Subtree Design and Custom LDAP Schema

...

Below is the proposed SIMP OID subtree showing the parent OIDs for attributes and class objects needed for the SIMP DIT.

...

LDAP Custom Schema

simpkv DIT Option 1

The proposed custom schema for the simpkv DIT option 1 is shown below. It has a custom object class, simpkvEntry, that is comprised of two custom attributes, simpkvKey and simpkvJsonValue.

simpkvKey is a case-invariant string that for the key . (excluding path)
- This is used as the final RDN of the DN for a key/value node.
simpkvJsonValue is a case-sensitive string to for the JSON-formatted value.
- In the future, we could write a custom syntax validator for this attribute.

Code Block

################################################################################
#
dn: cn=schema
#
################################################################################
#
attributeTypes: (
 
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.1
  NAME 'simpkvKey'
  DESC 'key'
  SUP name
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.2
  NAME 'simpkvJsonValue'
  DESC 'JSON-formatted value'
  EQUALITY caseExactMatch
  SUBSTR caseExactSubstringsMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
objectClasses: (
  1.3.6.1.4.1.47012.1.1.1.1.2.1
  NAME 'simpkvEntry'
  DESC 'simpkv entry'
  SUP top
  STRUCTURAL
  MUST ( simpkvKey $ simpkvJsonValue )
  X-ORIGIN 'SIMP simpkv'
  )

The corresponding SIMP OID subtree is as follows:

...

simpkv DIT Option 2

The proposed custom schema for the simpkv DIT option 2 is shown below. It has two custom object classes and three custom attributes.

Classes:
- simpkvFolder is an object class for a node representing a backend identifier or folder.
- simpkvEntry is an object class for a key/value node.
Attributes:
- simpkvHexId is an attribute that is a case-invariant, hex-encoded string for the backend identifier, folder or key (excluding path)
  - This is used as the final RDN of the DN for a node.
  - In the future, we could write a custom syntax validator for this attribute.
- simpkvId is an attribute that is the raw, case-sensitive string for a backend identifier, folder or key (excluding path)
- simpkvJsonValue is an attribute that is a case-sensitive string for a JSON-formatted value in a key/value node.
  - In the future, we could write a custom syntax validator for this attribute.

Code Block

################################################################################
#
dn: cn=schema
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.1
  NAME 'simpkvHexId'
  DESC 'hex-encoded backend instance, folder, or key name'
  SUP name
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.2
  NAME 'simpkvId'
  DESC 'backend instance, key or folder name'
  EQUALITY caseExactMatch
  SUBSTR caseExactSubstringsMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
attributeTypes: (
  1.3.6.1.4.1.47012.1.1.1.1.1.3
  NAME 'simpkvJsonValue'
  DESC 'JSON-formatted value'
  EQUALITY caseExactMatch
  SUBSTR caseExactSubstringsMatch
  SYNTAX 1.3.6.1.54.1.470121466.1115.1121.1.1.1.1
  NAME 'simpkvKey'
  DESC 'key'
  SUP name15
  SINGLE-VALUE
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
attributeTypesobjectClasses: (
  1.3.6.1.54.1.47012.1.1.1.1.2.1.2
  NAME 'simpkvJsonValuesimpkvEntry'
  DESC 'JSON-formattedsimpkv valueentry'
  SUP top
EQUALITY caseExactMatch STRUCTURAL
 SUBSTR caseExactSubstringsMatchMUST ( simpkvHexId SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
  SINGLE-VALUE$ simpkvId $ simpkvJsonValue )
  X-ORIGIN 'SIMP simpkv'
  )
#
################################################################################
#
objectClasses: (
  1.3.6.1.54.1.47012.1.1.1.1.2.1
  NAME 'simpkvEntry'
  DESC 'simpkv entry2
  NAME 'simpkvFolder'
  DESC 'simpkv folder in which simpKvHexId represents the relative folder name in hex in the DN'
  SUP top
  STRUCTURAL
  MUST ( simpkvKeysimpkvHexId $ simpkvJsonValuesimpkvId )
  X-ORIGIN 'SIMP simpkv'
  )

...

The corresponding SIMP OID subtree is as follows:

...

Technologies for Plugin Implementation

...

Plugins are written in Ruby and implement the simpkv plugin API.
Plugins must be multi-thread safe.
Plugins must be written to provide Puppet-environment isolation when executed on the puppetserver.
Manifests that use simpkv::* functions must be able to be compiled with puppet agent, puppet apply or Bolt commands. This means the plugin code will run in JRuby in the puppetserver, run in the Ruby installed with puppet-agent, or run using the Bolt user’s Ruby into which the puppet gem is installed.

...

Option	PROs	CONs
Tools provided by openldap-utils RPM	Existing, signed, vendor RPM. Package will already be installed on host operating as the simpkv LDAP server. Supports ldapi interface, which is faster than ldap/ldaps, while still being secure.	Requires openldap-utils RPM to be installed on host executing Bolt compiles. To take advantage of ldapi either have to educate user on when ldapi should be configured OR create internal auto-ldapi-detection logic to use the ldapi interface when it is available <--> complexity.
net-ldap Ruby gem	User can install gem without sysadmin support, when not on isolated network.	Requires gem RPM packaging for use on isolated networks (e.g., simp-vendored-net-ldap RPM) Requires gem installation into the puppetserver Does not support ldapi .
Support both tools provided by openldap-utils and net-ldap Ruby gem, using whichever it discovers is available	More installation flexibility when not on isolated networks.	Increased code+test complexity. Still has gem packaging issues on isolated systems for Bolt users. User still needs to know when ldapi can be used, unless auto-discovery mechanism is built.

...

Recommendation

Option 1 without the auto-discovery mechanism is recommended for the following reasons:

Options 2 and 3 require additional packaging in order to work on isolated networks for Bolt users. So, if you are going to require a Bolt user to install a package, anyways, might as well be an existing vendor package.
The auto-discovery mechanism can be added after the initial implementation, because it is not required for the LDAP plugin to function.

Versions Compared

Old Version 40

New Version Current

Key

Backend logical structure

Value normalization

LDAP Directory Information Tree design

Recommendation

OID Subtree Design and Custom LDAP Schema

LDAP Custom Schema

simpkv DIT Option 1

simpkv DIT Option 2

Technologies for Plugin Implementation

Recommendation