Table of Content

• Small helpers
• Data types, resources and more
  • Data types
  • Resource type
    • Usage of resource types without a title
  • Resource/Class reference
    • Class reference
    • Accessing value through reference
  • Specials for resource attributes
    • exec Resource
• Variables
  • Assigning data
  • Assigning from array
  • Assigning from hash
  • Naming convention
    • Reserved variable names
  • Types
    • Hash
• Execute object definition direct from agent
  • Single objects definitions
  • Multiple objects definitions
  • Adding additional puppet modules to module list for local usage
• Output during puppet runs
  • Using notify
  • Old and also maybe outdated
• Queering hieradata locally on puppet server for testing
  • Usage
  • Sample
• Tagging
  • Resources
  • Classes
  • Using tags
• Relationships and Ordering
  • for Resources
    • Chaining arrows
  • for Classes
    • Contain
    • Require
    • Stages
  • Automatic relationships
  • Dependency cycles
• Collectors
  • Collector Samples
  • Exported resource collectors
• Complex Resource syntax
  • Create resource if it does not exists
  • Manipulate an existing resource with append or overwrite attributes
  • Accessing other arrtibute values for resource
  • Add attributes with collector
• Create file from multiple templates
• Deploying full dir with files out of puppet
• BuildIn puppet functions
  • map
  • flatten
  • filter
  • reduce
  • slice
  • reverse each
  • dig
  • then
  • lest
• Full samples
  • Two items from a hiera hash inside of an array to a array with hashes
  • Two items from a hiera hash inside of an array to a has
  • Get one item from each hash and sum them up
  • Calculate occurrence of strings in array
  • Add item to array even it could be empty
• ERB validation
• Documenting modules with puppet string
• Puppet Server
  • Unclear server errors
    • uninitialized constant Concurrent

Small helpers

Data types, resources and more

Data types

• Abstract data types: If you’re using data types to match or restrict values and need more flexibility than what the core data types (such as String or Array) allow, you can use one of the abstract data types to construct a data type that suits your needs and matches the values you want.
• Arrays: Arrays are ordered lists of values. Resource attributes which accept multiple values (including the relationship metaparameters) generally expect those values in an array. Many functions also take arrays, including the iteration functions.
• Binary: A Binary object represents a sequence of bytes and it can be created from a String in Base64 format, a verbatim String, or an Array containing byte values. A Binary can also be created from a Hash containing the value to convert to a Binary.
• Booleans: Booleans are one-bit values, representing true or false. The condition of an if statement expects an expression that resolves to a boolean value. All of Puppet’s comparison operators resolve to boolean values, as do many functions.
• Data type syntax: Each value in the Puppet language has a data type, like “string.” There is also a set of values whose data type is “data type.” These values represent the other data types. For example, the value String represents the data type of strings. The value that represents the data type of these values is Type.
• Default: Puppet’s default value acts like a keyword in a few specific usages. Less commonly, it can also be used as a value.
• Error data type: An Error object contains a non-empty message. It can also contain additional context about why the error occurred.
• Hashes: Hashes map keys to values, maintaining the order of the entries according to insertion order.
• Numbers: Numbers in the Puppet language are normal integers and floating point numbers.
• Regular expressions: A regular expression (sometimes shortened to “regex” or “regexp”) is a pattern that can match some set of strings, and optionally capture parts of those strings for further use.
• Resource and class references: Resource references identify a specific Puppet resource by its type and title. Several attributes, such as the relationship metaparameters, require resource references.
• Resource types: Resource types are a special family of data types that behave differently from other data types. They are subtypes of the fairly abstract Resource data type. Resource references are a useful subset of this data type family.
• Sensitive: Sensitive types in the Puppet language are strings marked as sensitive. The value is displayed in plain text in the catalog and manifest, but is redacted from logs and reports. Because the value is maintained as plain text, use it only as an aid to ensure that sensitive values are not inadvertently disclosed.
• Strings: Strings are unstructured text fragments of any length. They’re a common and useful data type.
• Time-related data types: A Timespan defines the length of a duration of time, and a Timestamp defines a point in time. For example, “two hours” is a duration that can be represented as a Timespan, while “three o’clock in the afternoon UTC on 8 November, 2018” is a point in time that can be represented as a Timestamp. Both types can use nanosecond values if it is available on the platform.
• Undef: Puppet’s undef value is roughly equivalent to nil in Ruby. It represents the absence of a value. If the strict_variables setting isn’t enabled, variables which have never been declared have a value of undef.

Resource type

Resource types are a special family of data types that behave differently from other data types. They are subtypes of the fairly abstract Resource data type. Resource references are a useful subset of this data type family.

In the Puppet language, there are never any values whose data type is one of these resource types. That is, you can never create an expression where $my_value =~ Resource evaluates to true. For example, a resource declaration - an expression whose value you might expect would be a resource - executes a side effect and then produces a resource reference as its value. A resource reference is a data type in this family of data types, rather than a value that has one of these data types.

In almost all situations, if one of these resource type data types is involved, it makes more sense to treat it as a special language keyword than to treat it as part of a hierarchy of data types. It does have a place in that hierarchy, it’s just complicated, and you don’t need to know it to do things in the Puppet language.

For that reason, the information on this page is provided for the sake of technical completeness, but learning it isn’t critical to your ability to use Puppet successfully.

Puppet automatically creates new known data type values for every resource type it knows about, including custom resource types and defined types.

These one-off data types share the name of the resource type they correspond to, with the first letter of every namespace segment capitalized. For example, the file type creates a data type called File.

Additionally, there is a parent Resource data type. All of these one-off data types are more-specific subtypes of Resource.

Usage of resource types without a title

A resource data type can be used in the following places:

• The resource type slot of a resource declaration.
• The resource type slot of a resource default statement.

Resouce data types (written with first upper in case latters) are no resources any more, these are resource references

For example:

# A resource declaration using a resource data type:
File { "/etc/ntp.conf":
  mode  => "0644",
  owner => "root",
  group => "root",
}

# Equivalent to the above:
Resource["file"] { "/etc/ntp.conf":
  mode  => "0644",
  owner => "root",
  group => "root",
}

# A resource default:
File {
  mode  => "0644",
  owner => "root",
  group => "root",
}

If a resource data type includes a title, it acts as a resource reference, which are useful in several places.

Resource/Class reference

As you may have read above, you can manipulate attributes by using resource references, but such references allow you much more things.

There are several things what you can do, but lets focos on the most common one with the following syntax:

[A-Z][a-z]\[<title1>,<titel2>,<titelN>\]
[A-Z][a-z]\[<title1>]{<attribute_key_value>}
[A-Z][a-z]\[<title1>][<attribute_key>]

As you can see, it also allows you to specify mulitble resource titles, to apply the same on all of them.

Guess you have seen that arleady in other situations, like file { ['file1,'file2','fileN']: and so on. So this is qutie familiar.

Class reference

Class references, are often used if you need to ensure that a full class has to be appliyed before another resource kicks in. For example:

class setup_backup {
    ... super epic puppet code, we are not allowed to show it here ;) ...
}

class server_installation {
  service { 'run_backup':
    ensure  => enabled,
    running => true,
    ...
    require => Class['setup_backup'],
  }
}

If you have multible manifests and classes in one module, just get the class name und set each first letter (including the letters after ::) to upper case. Sample:

class setup_backup::install

would be:

Class[Setup_backup::Install]

Accessing value through reference

As it says, references also allow you to access attributes of another resource.

So if you do not want to define a default using a reference and a variable would be also overkill, you could use a resoruce refernce to share the same attibute value between resources.

class server_installation {
  service { 'run_backup':
    ensure  => enabled,
    running => true,
    ...
    require => Class['setup_backup'],
  }

  service { 'check_backup':
    ensure  => Service['run_backup']['ensure'],
    running => Service['run_backup']['running'],
    ...
    require => Class['setup_backup'],
  }
}

Specials for resource attributes

Things we found about resource attributes which are not 100% documented

exec Resource

creates creates only executes the command of the exec resource, if the file configured as vaule for creates does not exists.

What was discovered is, that this is not only true for files. This also works with (hard/soft) links and with directories

Just think think about that a second (then continue to read ;)

files, directories and links share the same resource type in puppet, so the assumtion is that this is the connection, why this works.

Variables

Puppet allows a given variable to be assigned a value only one time within a given scope. This is a little different from most programming languages. You cannot change the value of a variable, but you can assign a different value to the same variable name in a new scope. The only exception is, if you are in a loop and initialize there the variable.

If your variable can contain a real value or also undef you can use the variable type Optional. This allows you to define it + accept undef. Syntax: Optional[Boolean] $remove_unused_kernel = undef,

Assigning data

$variable_name = "your content"

That was easy right ;)

Now lets do it in a more fancy way and lets assume we have more then one variable where we want to assign some data.

Assigning from array

To assign multiple variables from an array, you must specify an equal number of variables and values. If the number of variables and values do not match, the operation fails. You can also use nested arrays.

[$a, $b, $c] = [1,2,3]         # $a = 1, $b = 2, $c = 3
[$a, [$b, $c]] = [1,[2,3]]     # $a = 1, $b = 2, $c = 3
[$a, $b] = [1, [2]]            # $a = 1, $b = [2]
[$a, [$b]] = [1, [2]]          # $a = 1, $b = 2

Assigning from hash

You can include extra key-value pairs in the hash, but all variables to the left of the operator must have a corresponding key in the hash

[$a, $b] = {a => 10, b => 20}  # $a = 10, $b = 20
[$a, $c] = {a => 5, b => 10, c => 15, d => 22}  # $a = 5, $c = 15

Naming convention

Variable names are case-sensitive and must begin with a dollar sign ($). Most variable names must start with a lowercase letter or an underscore. The exception is regex capture variables, which are named with only numbers. Variable names can include:

• Uppercase and lowercase letters
• Numbers
• Underscores (_). If the first character is an underscore, access that variable only from its own local scope.

Qualified variable names are prefixed with the name of their scope and the double colon (::) namespace separator. For example, the $vhostdir variable from the apache::params class would be $apache::params::vhostdir.

Optionally, the name of the very first namespace can be empty, representing the top namespace. The main reason to namespace this way is to indicate to anyone reading your code that you’re accessing a top-scope variable, such as $::is_virtual. You can also use a regular expression for variable names. Short variable names match the following regular expression:

\A\$[a-z0-9_][a-zA-Z0-9_]*\Z

Qualified variable names match the following regular expression:

\A\$([a-z][a-z0-9_]*)?(::[a-z][a-z0-9_]*)*::[a-z0-9_][a-zA-Z0-9_]*\Z

Reserved variable names

Types

For details on each data type, have a look at the specification documentation.

Hash

When hashes are merged (using the addition (+) operator), the keys in the constructed hash have the same order as in the original hashes, with the left hash keys ordered first, followed by any keys that appeared only in the hash on the right side of the merge. Where a key exists in both hashes, the merged hash uses the value of the key in the hash to the right of the addition operator (+). For example:

$values = {'a' => 'a', 'b' => 'b'}
$overrides = {'a' => 'overridden'}
$result = $values + $overrides
notice($result)
-> {'a' => 'overridden', 'b' => 'b'}

Hieradata

Hieradata is a vairable store for puppet and uses as .yaml files as a base. All the standard yaml functions are accepted and even some additions which puppet is able to use.

Interpolation

You can extend the content of variables by using the interpolation syntax which looks like the following:

<variable_name>: "%{variablename or interpolation function}"

One positive thing about interpolation is that it will be performed before the catalog build startes and hands over directly to the build process the final result.

Interpolate variables

Most commonly used are variables from the top-scope of puppet, meaning variables which are part of the following hashes:

• facts
• trusted
• server_facts

As puppet allows you to access these variables in certon ways like as hash facts['os'] and as top-scope ${os} you might want to stick with the hash call, as in hiera you have to care unknown scopes as it can overwrite the data you expected.

Just for record, a unknown top-scope vairable in hiera would look like this %{::os}

Sample for interpolate variables

Lets assume you have several times your domain inside your hiera data and dont want to rewrite it the whole time

apache_main_domain: "%{facts.networking.domain}"
apache_subdomain_1: "git_repos.%{facts.networking.domain}"
apache_subdomain_2: "myip.%{facts.networking.domain}"
mail_server: "mail.%{facts.networking.domain}"
ssh_server: "ssh.%{facts.networking.domain}"
db_server: "database.%{facts.networking.domain}"

So puppet will get during the catalog already the final content for the variables and will replace %{facts.networking.domain} with the correct data.

If you ask your self why it is written like this facts.networking.doamin and not like this facts['networking']['doamin'].

Easy to explain, if you would wirte it like the typical hash call you have to mask the quoatings which just takes longer and just combinding the items with . is also good to read. At least for me ;)

Interpolation functions

Interpolation functions can not be use in the hiera.yaml file which configures the hiera backend, but it still allows variable interpolation

Execute object or definition direct from agent

Single objects or definitions

puppet apply -e '< DEFINITION >'

example:

$ puppet apply -e 'service { 'samba': enable => false, ensure => 'stopped', }' --debug
# or
$ puppet apply -e 'notice(regsubst(regsubst(['test1','test2','test3'],"test","works"),"$",".test").join(", "))
Notice: Scope(Class[main]): works1.test, works2.test, works3.test
Notice: Compiled catalog for <mydevicename> in environment production in 0.04 seconds
Notice: Applied catalog in 0.04 seconds

Multiple objects/definitions

An easy way to do that, is to just create an puppet file (e.g. mytestfile.pp) containing all the information like:

service { 'rsyslog':
  ensure => 'running',
}

package { 'git':
  ensure => installed,
}

package { 'git-lfs':
  ensure  => installed,
  require => Package['git'],
}

#...

You can just cat the file inside of the puppet apply like this:

$ puppet apply -e "$(cat ./mytestfile.pp)"
# or
$ puppet apply ./mytestfile.pp

and puppet will just perform the actions.

Adding additional puppet modules to module list for local usage

If you are running puppet apply <filename> you might find out one day, that you need some supporter modules e.g. stdlib. To get this module into your local module list, you can act like this:

$ sudo puppet module install puppetlabs-stdlib
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└── puppetlabs-stdlib (v8.1.0)

After it finished, perform the following command, to see where your puppet agent expects the modules. If you have not added a single module yet, it will look like below, otherwise you will get the list of modules.

$ puppet module list
/opt/puppetlabs/puppet/modules (no modules installed)

Now there are several ways how to keep the dirs /etc/puppetlabs/code/environments/production/modules and /opt/puppetlabs/puppet/modules in sync.

For example you could create a symlink between the dirs, or create a systemd path to act on changes and many other this would work. But, you can also use the parameter --target-dir for puppet module install. With this parameter, you can just specify, where the module should get installed.

If you have a module already installed beneath /opt/puppetlabs/puppet/modules and want to install it again on a different place, puppet will give you an error, saying it is already installed

$ sudo puppet module install puppetlabs-stdlib --target-dir /opt/puppetlabs/puppet/modules`

If you execute now again the module list command, you will get it displayed, where it is + the version

$puppet module list
/opt/puppetlabs/puppet/modules
└── puppetlabs-stdlib (v8.1.0)

Output during puppet runs

Using notify

notify{'this will be shown duing every puppet run':}

By adding loglevel notify is able to act in different levels

Supported levels are:

• emerg
• alert
• crit
• err
• warning
• notice
• info
• verbose
• debug

Old and also maybe outdated

notice("try to run this script with -v and -d to see difference between log levels")
notice("function documentation is available here: http://docs.puppetlabs.com/references/latest/function.html")
notice("--------------------------------------------------------------------------")

debug("this is debug. visible only with -d or --debug")
info("this is info. visible only with -v or --verbose or -d or --debug")
alert("this is alert. always visible")
crit("this is crit. always visible")
emerg("this is emerg. always visible")
err("this is err. always visible")
warning("and this is warning. always visible")
notice("this is notice. always visible")
fail("this is fail. always visible. fail will break execution process")

Queering hieradata locally on puppet server for testing

Usage

$ puppet lookup ${variableyoulookfor}  --node ${nodeyouwanttotarget} --environment ${environment}

Sample

$ puppet lookup clustername  --node myserver.sons-of-sparda.at --environment legendary

Tagging

Tags can be applied on resources as well on classes.

Resources

To tag a resource you can use the attribute tag during the call of a resource or while defining it

user { 'root':
  ensure           => present,
  password_max_age => '60',
  password         => $newrootpwd,
  tag              => [ 'rootpasswordresetonly' ],
}

Classes

For tagging a class use the function tag inside of a class definition

class role::public_web {
  tag 'mirror1', 'mirror2'

  apache::vhost {'docs.puppetlabs.com':
    port => 80,
  }
  ssh::allowgroup {'www-data': }
  @@nagios::website {'docs.puppetlabs.com': }
}

Or defined type

define role::public_web {
  apache::vhost {$name:
    port => 80,
  }
  ssh::allowgroup {'www-data': }
  @@nagios::website {'docs.puppetlabs.com': }
}

role::public_web { 'docs.puppetlabs.com':
  tag => ['mirror1','mirror2'],
}

Using tags

To use tags you can either specify them in the puppet.conf or you can just add the parameter --tags [tag] to the puppet command

$ puppet agent -t --tags mirror1

You can also combine more tags at the same time like --tags [tag1,tag2,..]

$ puppet agent -t --tags mirror1,rootpasswordresetonly

Relationships and Ordering

for Resources

Set the value of any relationship meta parameter to either a resource reference or an array of references that point to one or more target resources:

• before: Applies a resource before the target resource.
• require: Applies a resource after the target resource.
• notify: Applies a resource before the target resource. The target resource refreshes if the notifying resource changes.
• subscribe: Applies a resource after the target resource. The subscribing resource refreshes if the target resource changes.

If two resources need to happen in order, you can either put a before attribute in the prior one or a require attribute in the subsequent one; either approach creates the same relationship. The same is true of notify and subscribe.

The two examples below create the same ordering relationship, ensuring that the openssh-server package is managed before the sshd_config file:

package { 'openssh-server':
  ensure => present,
  before => File['/etc/ssh/sshd_config'],
}

file { '/etc/ssh/sshd_config':
  ensure  => file,
  mode    => '0600',
  source  => 'puppet:///modules/sshd/sshd_config',
  require => Package['openssh-server'],
}

The two examples below create the same notifying relationship, so that if Puppet changes the sshd_config file, it sends a notification to the sshd service:

file { '/etc/ssh/sshd_config':
  ensure => file,
  mode   => '0600',
  source => 'puppet:///modules/sshd/sshd_config',
  notify => Service['sshd'],
}

service { 'sshd':
  ensure    => running,
  enable    => true,
  subscribe => File['/etc/ssh/sshd_config'],
}

Because an array of resource references can contain resources of differing types, these two examples also create the same ordering relationship. In both examples, Puppet manages the openssh-server package and the sshd_config file before it manages the sshd service.

service { 'sshd':
  ensure  => running,
  require => [
    Package['openssh-server'],
    File['/etc/ssh/sshd_config'],
  ],
}

package { 'openssh-server':
  ensure => present,
  before => Service['sshd'],
}

file { '/etc/ssh/sshd_config':
  ensure => file,
  mode   => '0600',
  source => 'puppet:///modules/sshd/sshd_config',
  before => Service['sshd'],
}

Chaining arrows

You can create relationships between resources or groups of resources using the -> and ~> operators.

The ordering arrow is a hyphen and a greater-than sign (->). It applies the resource on the left before the resource on the right.

The notifying arrow is a tilde and a greater-than sign (~>). It applies the resource on the left first. If the left-hand resource changes, the right-hand resource refreshes.

In this example, Puppet applies configuration to the ntp.conf file resource and notifies the ntpd service resource if there are any changes.

File['/etc/ntp.conf'] ~> Service['ntpd']

Note: When possible, use relationship meta parameters, not chaining arrows. Meta parameters are more explicit and easier to maintain. See the Puppet language style guide for information on when and how to use chaining arrows.

The chaining arrows accept the following kinds of operands on either side of the arrow:

• Resource references, including multi-resource references.
• Arrays of resource references.
• Resource declarations.
• Resource collectors.

You can link operands to apply a series of relationships and notifications. In this example, Puppet applies configuration to the package, notifies the file resource if there are changes, and then, if there are resulting changes to the file resource, Puppet notifies the service resource:

Package['ntp'] -> File['/etc/ntp.conf'] ~> Service['ntpd']

Resource declarations can be chained. That means you can use chaining arrows to make Puppet apply a section of code in the order that it is written. This example applies configuration to the package, the file, and the service, in that order, with each related resource notifying the next of any changes:

# first:
package { 'openssh-server':
  ensure => present,
} # and then:
-> file { '/etc/ssh/sshd_config':
  ensure => file,
  mode   => '0600',
  source => 'puppet:///modules/sshd/sshd_config',
} # and then:
~> service { 'sshd':
  ensure => running,
  enable => true,
}

Collectors can also be chained, so you can create relationships between many resources at one time. This example applies all apt repository resources before applying any package resources, which protects any packages that rely on custom repositories:

Aptrepo <| |> -> Package <| |>

Both chaining arrows have a reversed form (<- and <~). As implied by their shape, these forms operate in reverse, causing the resource on their right to be applied before the resource on their left. Avoid these reversed forms, as they are confusing and difficult to notice.

Cautions when chaining resource collectors

Chains can create dependency cycles Chained collectors can cause huge dependency cycles; be careful when using them. They can also be dangerous when used with virtual resources, which are implicitly realized by collectors.

Chains can break Although you can usually chain many resources or collectors together (File['one'] -> File['two'] -> File['three']), the chain can break if it includes a collector whose search expression doesn’t match any resources.

Implicit properties aren’t searchable Collectors can search only on attributes present in the manifests; they cannot see properties that are automatically set or are read from the target system. For example, the chain Aptrepo <| |> -> Package <| provider == apt |>, creates only relationships with packages whose provider attribute is explicitly set to apt in the manifests. It would not affect packages that didn’t specify a provider but use apt because it’s the operating system’s default provider.

for Classes

Unlike with resources, Puppet does not automatically contain classes when they are declared inside another class (by using the include function or resource-like declaration). But in certain situations, having classes contain other classes can be useful, especially in larger modules where you want to improve code readability by moving chunks of implementation into separate files.

You can declare a class in any number of places in the code, allowing classes to announce their needs without worrying about whether other code also needs the same classes at the same time. Puppet includes the declared class only one time, regardless of how many times it’s declared (that is, the include function is idempotent). Usually, this is fine, and code shouldn’t attempt to strictly contain the class. However, there are ways to explicitly set more strict containment relationships of contained classes when it is called for.

When you’re deciding whether to set up explicit containment relationships for declared classes, follow these guidelines:

• include: When you need to declare a class and nothing in it is required for the enforcement of the current class you’re working on, use the include function. It ensures that the named class is included. It sets no ordering relationships. Use include as your default choice for declaring classes. Use the other functions only if they meet specific criteria.
• require: When resources from another class should be enforced before the current class you’re working on can be enforced properly, use the require function to declare classes. It ensures that the named class is included. It sets relationships so that everything in the named class is enforced before the current class.
• contain: When you are writing a class in which users should be able to set relationships, use the contain function to declare classes. It ensures that the named class is included. It sets relationships so that any relationships specified on the current class also apply to the class you’re containing.
• stage: Allows you to place classes into run stages, which creates a rough ordering

Contain

Use the contain function to declare that a class is contained. This is useful for when you’re writing a class in which other users should be able to express relationships. Any classes contained in your class will have containment relationships with any other classes that declare your class. The contain function uses include-like behavior, containing a class within a surrounding class.

For example, suppose you have three classes that an app package (myapp::install), creating its configuration file (myapp::config), and managing its service (myapp::service). Using the contain function explicitly tells Puppet that the internal classes should be contained within the class that declares them. The contain function works like include, but also adds class relationships that ensure that relationships made on the parent class also propagate inside, just like they do with resources.

class myapp {
  # Using the contain function ensures that relationships on myapp also apply to these classes
  contain myapp::install
  contain myapp::config
  contain myapp::service

  Class['myapp::install'] -> Class['myapp::config'] ~> Class['myapp::service']
}

Although it may be tempting to use contain everywhere, it’s better to use include unless there’s an explicit reason why it won’t work.

Require

The require function is useful when the class you’re writing needs another class to be successfully enforced before it can be enforced properly.

For example, suppose you’re writing classes to install two apps, both of which are distributed by the apt package manager, for which you’ve created a class called apt. Both classes require that apt be properly managed before they can each proceed. Instead of using include, which won’t ensure apt’s resources are managed before it installs each app, use require.

class myapp::install {
  # works just like include, but also creates a relationship
  # Class['apt'] -> Class['myapp::install']
  require apt
  package { 'myapp':
    ensure => present,
  }
}

class my_other_app::install {
  require apt
  package { 'my_other_app':
    ensure => present,
  }
}

Stages

Puppet is aware of run stages, so what does that mean for you. A class inside of puppet run/catalog is assigned by default to the run stage main. If you have for example a class, where you know that this needs always to happen first or always to happen at the end, you can create custom stages to solve this.

To assign a class to a different stage, you must:

• Declare the new stage as a stage resource
• Declare an order relationship between the new stage and the main stage.
• Use the resource-like syntax to declare the class, and set the stage meta parameter to the name of the desired stage.

Important: This meta parameter can only be used on classes and only when declaring them with the resource-like syntax. It cannot be used on normal resources or on classes declared with include.

Also note that new stages are not useful unless you also declare their order in relation to the default main stage.

stage { 'pre':
  before => Stage['main'],
}

class { 'apt-updates':
  stage => 'pre',
}

Automatic relationships

Certain resource types can have automatic relationships with other resources, using auto require, auto notify, auto before, or auto subscribe. This creates an ordering relationship without you explicitly stating one.

Puppet establishes automatic relationships between types and resources when it applies a catalog. It searches the catalog for any resources that match certain rules and processes them in the correct order, sending refresh events if necessary. If any explicit relationship, such as those created by chaining arrows, conflicts with an automatic relationship, the explicit relationship take precedence.

Dependency cycles

If two or more resources require each other in a loop, Puppet compiles the catalog but won’t be able to apply it. Puppet logs an error like the following, and attempts to help identify the cycle:

err: Could not apply complete catalog: Found 1 dependency cycle:
(<RESOURCE> => <OTHER RESOURCE> => <RESOURCE>)
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz

To locate the directory containing the graph files, run puppet agent --configprint graphdir. Related information: Containment

Collectors

Resource collectors select a group of resources by searching the attributes of each resource in the catalog, even resources which haven’t yet been declared at the time the collector is written. Collectors realize virtual resources, are used in chaining statements, and override resource attributes. Collectors have an irregular syntax that enables them to function as a statement and a value.

The general form of a resource collector is:

• A capitalized resource type name. This cannot be Class, and there is no way to collect classes.
• <| An opening angle bracket (less-than sign) and pipe character.
• Optionally, a search expression.
• |> A pipe character and closing angle bracket (greater-than sign)

Note: Exported resource collectors have a slightly different syntax; see below.

Collectors can search only on attributes that are present in the manifests, and cannot read the state of the target system. For example, the collector Package <| provider == apt |> collects only packages whose provider attribute is explicitly set to apt in the manifests. It does not match packages that would default to the apt provider based on the state of the target system.

A collector with an empty search expression matches every resource of the specified resource type.

Collector Samples

Collect a single user resource whose title is luke:

User <| title == 'luke' |>

Collect any user resource whose list of supplemental groups includes admin:

User <| groups == 'admin' |>

Collect any file resource whose list of supplemental require includes /etc and not /etc/.git:

File <| require == '/etc' and require != '/etc/.git' |>

Collect any service resource whose attribute ensure is set to running or set to true:

Service <| ensure == running or ensure == true |>

Creates an order relationship with several package resources:

Aptrepo['custom_packages'] -> Package <| tag == 'custom' |>

Exported resource collectors

An exported resource collector uses a modified syntax that realizes exported resources and imports resources published by other nodes.

To use exported resource collectors, enable catalog storage and searching (storeconfigs). See Exported resources for more details. To enable exported resources, follow the installation instructions and Puppet configuration instructions in the PuppetDB docs.

Like normal collectors, use exported resource collectors with attribute blocks and chaining statements.

Note: The search for exported resources also searches the catalog being compiled, to avoid having to perform an additional run before finding them in the store of exported resources.

Exported resource collectors are identical to collectors, except that their angle brackets are doubled.

Nagios_service <<| |>> # realize all exported nagios_service resources

The general form of an exported resource collector is:

• The resource type name, capitalized.
• <<| Two opening angle brackets (less-than signs) and a pipe character.
• Optionally, a search expression.
• |>> A pipe character and two closing angle brackets (greater-than signs).

Complex Resource syntax

Create resource if it does not exists

Sometimes you ran into the problem, that you might will create a resources more then one time (e.g. inside of loops and you can not avoid it, due to some reason).

Other bad example would be, if you don’t know if the resource is already handled some where else, this would be still a possible way for you, but better get into your code ;)

There is are two functions inside of puppet_stdlib which can help you on that ensure_resource and ensure_resources

These two function can detect if a resource is already defined in the catalog and only adds it, if it is not getting loaded.

ensure_resource('<resource_type>', '<resource_name>', {'hash' => 'with normal attributes of resource'})
ensure_resource('<resource_type>', ['<list_of_resource_names_item1>','<list_of_resource_names_item2>', '...'], {'hash' => 'with normal attributes of resource'})

ensure_resources('<resource_type>', {'<resource_name1>' => { 'hash' => 'with normal attributes of resource' }, '<resource_name2>' => { 'hash' => 'with normal attributes of resource' }}, => 'present')

This maybe looks hard to read first, but lets do a sample, to make it easier

Sample for ensure_resource(s)

Creates a user

ensure_resource( 'user', 'my_user', { 'ensure' => 'present'} )

Creates a file

ensure_resource( 'file', '/home/my_user/temp/puppet/test1', { 'ensure' => 'present', 'require' => User['my_user']} )

Installs a package

ensure_resource('package', 'vim', {'ensure' => 'installed', 'require' => File['/home/my_user/temp/puppet/test1'] })

Accessing other arrtibute values for resource

You can use a resource reference to access the values of a resource’s attributes. To access a value, use square brackets and the name of an attribute (as a string). This works much like accessing hash values.

<resource_type> { <resource_name1>:
  <attribute1> => <attribute1_vallue>,
  <attribute2> => <attribute1_vallue>,
  <attribute3> => <attribute1_vallue>,
  ...
}

<resource_type> { <resource_name1>:
  <attribute1> => <attribute1_vallue>,
  <attribute2> => <resource_type>[<resource_name2>][<attribute2>],
  <attribute2> => <resource_type>[<resource_name3>][<attribute3>],
  ...
}

The resource whose values you’re accessing must exist.

Like referencing variables, attribute access depends on evaluation order, Puppet must evaluate the resource you’re accessing before you try to access it. If it hasn’t been evaluated yet, Puppet raises an evaluation error.

You can only access attributes that are valid for that resource type. If you try to access a nonexistent attribute, Puppet raises an evaluation error

Puppet can read the values of only those attributes that are explicitly set in the resource’s declaration. It can’t read the values of properties that would have to be read from the target system. It also can’t read the values of attributes that default to some predictable value. For example, in the code below, you wouldn’t be able to access the value of the path attribute, even though it defaults to the resource’s title.

Like with hash access, the value of an attribute whose value was never set is undef.

Sample

file { "/etc/ssh/sshd_config":
  ensure => file,
  mode   => "0644",
  owner  => "root",
}

file { "/etc/ssh/ssh_config":
  ensure => file,
  mode   => File["/etc/first.conf"]["mode"],
  owner  => File["/etc/first.conf"]["owner"],
}

Manipulate an existing resource with append or overwrite attributes

You can add/modify attributes of an existing resource by using the following syntax:

<resource_type>['<resource_name>'] {
  < hash of attributes to add/modify >
}

IMPORTANT This can only be done inside of the same class and can not be done from the outside

Sample

file {'/etc/passwd':
  ensure => file,
}

File['/etc/passwd'] {
  owner => 'root',
  group => 'root',
  mode  => '0640',
}

Add attributes with collector

By using a collector you are also able to add/append/amend attributes to resources.

IMPORTANT

• Using the collector can overwrite other attributes which you have already specified, regardless of class inheritance.
• It can affect large numbers or resources at one time.
• It implicitly realizes any virtual resources the collector matches.
• Because it ignores class inheritance, it can override the same attribute more than one time, which results in an evaluation order race where the last override wins.

For resource attributes that accept multiple values in an array, such as the relationship meta parameters, you can add to the existing values instead of replacing them by using the “plusignment” (+>) keyword instead of the usual hash rocket (=>).

class base::linux {
  file {'/etc/passwd':
    ensure => file,
  }
  ...}

include base::linux

File <| tag == 'base::linux' |> {
 mode  => '0640',
 owner => 'root',
 group => 'root',
}

Create file from multiple templates

If you don’t want to use the puppet module concat for some reason but still want to create one file which is concated out of multiple templates, you can just do it like this.

It is not really documented, but still, it works fine.

file { '/path/to/file':
  ensure  => file,
  *       => $some_other_attributes,
  content => template('modulename/template1.erb','modulename/template2.erb',...,'modulename/templateN.erb')
}

Deploying full dir with files out of puppet

Puppet is able to deploy a full dir path including files which are static

1. You need to define the directory

In the source attribute you are specifying than the path from where puppet should pick the dirs/files to deploy

file { '/etc/bind':
  ensure  => directory,  # so make this a directory
  recurse => true,       # enable recursive directory management
  purge   => true,       # purge all unmanaged junk
  force   => true,       # also purge subdirs and links etc.
  mode    => '0644',     # this mode will also apply to files from the source directory
  owner   => 'root',
  group   => 'root',     # puppet will automatically set +x for directories
  source  => 'puppet:///modules/puppet_bind/master/pri',
}

2. You can add more files into that dir by using normal puppet file resources, as puppet knows them, they will not get removed
3. If you need a directory beneath your recursively deployed dir, you can respecify the new one as a non recursive one.

This has the benefit, that puppet will not remove file/dirs which are unmanaged by puppet and stored in that dir

file { "/etc/bind/named.d":
  ensure => directory,
  owner  => 'root',
  group  => 'root',
  mode   => '0755',
}

BuildIn puppet functions

map

Applies a lambda to every value in a data structure and returns an array containing the results.

This function takes two mandatory arguments, in this order:

1. An array, hash, or other iterate able object that the function will iterate over.
2. A lambda, which the function calls for each element in the first argument. It can request one or two parameters.

$transformed_data = $data.map |$parameter| { <PUPPET CODE BLOCK> }

or

$transformed_data = map($data) |$parameter| { <PUPPET CODE BLOCK> }

By setting the map e.g. on an array you can interact with each single row then.

Lets do a small sample first, the variable test_small contains numbers from 1 to 5 and we want to add inside of the item new content:

$test_small = [ '1', '2', '3', '4', '5' ]
$test2 = $test_small.map | $value | { "content before real value : ${value}" }

# results into > test2 = ['content before real value : 1', 'content before real value : 2', 'content before real value : 3', 'content before real value : 4', 'content before real value : 5']

Lets use the variable test1 which contains a array and each item in there is a hash. In the sample below, the map, takes care of the line, so that the data is provided to the <PUPPET CODE BLOCK>

$test1 = [
  { 'name' => '1_1', 'b2' => '2', 'c3' => '3', 'd4' => '4', 'e5' => '5' },
  { 'name' => '2_1', 'g2' => 'h', 'i3' => '3', 'j4' => '4', 'k5' => '5' },
  { 'name' => '2_1', 'm2' => 'h', 'n3' => '3', 'o4' => '4', 'p5' => '5' }
]

$test1_mods = $test1.map | $stables | {
  $stables.filter | $valuefilter | {
    $valuefilter[0] == 'name'
    }.map | $key, $value | { $value }
}.flatten

# results into > 'test1_mods = [1_1, 2_1, 2_1]'

In the sample above, we have used more built-in functions, which can be found below

• flatten
• filter

flatten

Returns a flat Array produced from its possibly deeply nested given arguments.

One or more arguments of any data type can be given to this function. The result is always a flat array representation where any nested arrays are recursively flattened.

flatten(['a', ['b', ['c']]])
# Would return: ['a','b','c']

$hsh = { a => 1, b => 2}

# -- without conversion
$hsh.flatten()
# Would return [{a => 1, b => 2}]

# -- with conversion
$hsh.convert_to(Array).flatten()
# Would return [a,1,b,2]

flatten(Array($hsh))
# Would also return [a,1,b,2]

Output sample with our array from below:

$output = map($rehotehose) | $mreh1 | {
    map($mreh1) | $key, $value | {
      flatten($key, $value)
    } }
notify{$output:}
# Would return:
[[["host_share", "myserver0"], ["ip_share", "172.20.91.42"], ["port_share", "22001"], ["id_share", "ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8"], ["ensure_share", "present"]], [["host_share", "myserver1"], ["ip_share", "172.20.91.42"], ["port_share", "22001"], ["id_share", "ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8"], ["ensure_share", "present"]], [["host_share", "myserver2"], ["ip_share", "172.20.91.42"], ["port_share", "22001"], ["id_share", "ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8"], ["ensure_share", "present"]]]

With flatten, you can also add additional values into your result, like this:

$output = flatten(map($rehotehose) | $mreh1 | {
    map($mreh1) | $key, $value | {
      flatten($key, $value)
    }
  }
, [ 'SHINY1', 'addedVALUE1' ], [ 'SHINY1', 'MYsecondValue2' ], [ 'SHINY3', 'LetsDOANOTHER1' ] )
notify{$output:}
# Would return:
["host_share", "myserver0", "ip_share", "172.20.91.42", "port_share", "22001", "id_share", "ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8", "ensure_share", "present", "host_share", "myserver1", "ip_share", "172.20.91.42", "port_share", "22001", "id_share", "ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8", "ensure_share", "present", "host_share", "myserver2", "ip_share", "172.20.91.42", "port_share", "22001", "id_share", "ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8", "ensure_share", "present", "SHINY1", "addedVALUE1", "SHINY1", "MYsecondValue2", "SHINY3", "LetsDOANOTHER1"]

filter

Applies a lambda to every value in a data structure and returns an array or hash containing any elements for which the lambda evaluates to true. This function takes two mandatory arguments, in this order:

1. An array, hash, or other iterate able object that the function will iterate over.
2. A lambda, which the function calls for each element in the first argument. It can request one or two parameters.

$filtered_data = $data.filter |$parameter| { <PUPPET CODE BLOCK> }

or

$filtered_data = filter($data) |$parameter| { <PUPPET CODE BLOCK> }

# For the array $data, return an array containing the values that end with "berry"
$data = ["orange", "blueberry", "raspberry"]
$filtered_data = $data.filter |$items| { $items =~ /berry$/ }
# $filtered_data = [blueberry, raspberry]

# For the hash $data, return a hash containing all values of keys that end with "berry"
$data = { "orange" => 0, "blueberry" => 1, "raspberry" => 2 }
$filtered_data = $data.filter |$items| { $items[0] =~ /berry$/ }
# $filtered_data = {blueberry => 1, raspberry => 2}

Output sample with our array from below:

$output = flatten(map($rehotehose) | $mreh1 | {
    map($mreh1) | $key, $value | {
      flatten($key, $value)
    }
  }
).filter | $v | {
  $v =~ /.*-.*-.*-.*-.*-.*/ or $v == 'present' or $v == 'absent'
}
notify{$output:}
# Would return:
["ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8", "present", "ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8", "present", "ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8", "present"]

reduce

Applies a lambda to every value in a data structure from the first argument, carrying over the returned value of each iteration, and returns the result of the lambda’s final iteration. This lets you create a new value or data structure by combining values from the first argument’s data structure. This function takes two mandatory arguments, in this order:

1. An array, hash, or other iterate able object that the function will iterate over.
2. A lambda, which the function calls for each element in the first argument. It takes two mandatory parameters:
   1. A memo value that is overwritten after each iteration with the iteration’s result.
   2. A second value that is overwritten after each iteration with the next value in the function’s first argument.

$data.reduce |$memo, $value| { ... }

or

reduce($data) |$memo, $value| { ... }

You can also pass an optional “start memo” value as an argument, such as start below:

$data.reduce(start) |$memo, $value| { ... }

or

reduce($data, start) |$memo, $value| { ... }

When the first argument ($data in the above example) is an array, Puppet passes each of the data structure’s values in turn to the lambda’s parameters. When the first argument is a hash, Puppet converts each of the hash’s values to an array in the form [key, value].

If you pass a start memo value, Puppet executes the lambda with the provided memo value and the data structure’s first value. Otherwise, Puppet passes the structure’s first two values to the lambda.

Puppet calls the lambda for each of the data structure’s remaining values. For each call, it passes the result of the previous call as the first parameter ($memo in the above examples) and the next value from the data structure as the second parameter ($value).

Lets assume you have a hash and you want to create a new hash, which has parts of the old one and also data from a different source. This is our data source (e.g. hiera):

users:
  dante:
    id:  '1000'
    pwd: 'ENC[asdfasdf...]'
  vergil:
    id:  '1001'
    pwd: 'ENC[qwerqwer...]'

paths:
  home_users_prefix: '/home'

$paths = lookup('paths')
$users_with_home = map(lookup('users')) | $_user_name, $_user_conf | { $key }.reduce( {} ) | $memo, $value | {
  $memo + { $value => "${paths['home_users_prefix']}/${value}" }
}

notice($users_with_home)

# Would return:
{ 'dante' => '/home/dante', 'vergil' => '/home/vergil' }

So what now has happened. First we used map to iterate through the hash. In each iteration, we hand over only the $key which is in our case the username. With reduce we start another iteration, but we are able to remember the state before the last iteration.

The command reduce( {} ) means, that the start parameter for the variable $memo is already an empty hash.

Now we are just iterating through and recreate each time a new hash which we combine with the memorized hash from before and here we go, we have a new hash, which contains data from two different hashes.

For samples more, please view puppet internal functions

slice

Slices an array or hash into pieces of a given size. This function takes two mandatory arguments: the first should be an array or hash, and the second specifies the number of elements to include in each slice. When the first argument is a hash, each key value pair is counted as one. For example, a slice size of 2 will produce an array of two arrays with key, and value.

$a.slice($n) |$x| { ... }
slice($a) |$x| { ... }

$a.slice(2) |$entry|          { notice "first ${$entry[0]}, second ${$entry[1]}" }
$a.slice(2) |$first, $second| { notice "first ${first}, second ${second}" }

The function produces a concatenated result of the slices.

slice([1,2,3,4,5,6], 2) # produces [[1,2], [3,4], [5,6]]
slice(Integer[1,6], 2)  # produces [[1,2], [3,4], [5,6]]
slice(4,2)              # produces [[0,1], [2,3]]
slice('hello',2)        # produces [[h, e], [l, l], [o]]

reverse each

reverse_each allows you to iterate in reversely. Lets start with a small sample:

[1, 2, 3, 4].reverse_each.convert_to(Array)

The output would will look then like this: [ 4, 3, 2, 1 ]

This can be very helpful if you for example need to generate based on a list of vlans you have some reverse DNS zone files.

$reverse_arr = []
$vlanconf = [ '10.02.03', '10.02.30', '10.02.33' ]
$vlanconf.each | $vl_data| {
  $reverse_arr = concat($reverse_arr, $join(split($v1,'\.').convert_to(Array).reverse_each.convert_to(Array),'.'))
}

This would result in the output: [ '03.02.10', '30.02.10', '33.02.10' ].

So what we did there is, that we iterated over our array in $vlanconf.

Each item which we are getting, we are moving it to split and convert it to array, because on that one we can perform the reverse_each.convert_to(Array) and join the output with .

dig

dig allows you to go fast through complex hashes/arrays.

$my_hash_nested = {
    'a' => '0',
    'b' => '1',
    'c' => '2',
    'e' => [
      {
          'e1' => [
            '13',
            '37'
          ],
          'e2' => [
            '42',
            '64'
          ]
      },
      {
          'no_e1' => [ 'no_13', 'no_37'],
          'no_e2' => [ 'no_42', 'no_64' ]
      }
    ],
    'f' => '4',
}

notify{"${my_hash_nested.dig('e',0,'e1',1)}":}

This results into the below output.

Notice: 37
Notice: /Stage[main]/Dig_test/Notify[37]/message: defined 'message' as '37'

The big addvantage of the dig function compared to a direct access using $my_hash_nested['e'][0]['e1'][1] is that dig will not fail the catalog build if any part of the path is undef, it will rather return an undef value and still allow the agent’s to perform the run.

then

then allows you to act on detected values and will let you action with them as long as they are not undef.

$my_hash = {
  'a' => '1',
  'b' => '2',
  'c' => '3',
  'e' => '4',
  'f' => '5',
}

[ 'a', 'b', 'c', 'd', 'e', 'f' ].each | $_search_item | {
  notify{"data for ${_search_item}: ${my_hash[$_search_item].then | $found_value | {"found it:: ${found_value}}":}
}

As long as then get a value balck it will, work on it, in our case add additional strings to the result. But if it gets an undef as for the value of d it returns also undev back and shows now result:

Notice: data for a: found it: 1
Notice: /Stage[main]/Then_test/Notify[data for a: found it: 1]/message: defined 'message' as 'data for a: found it: 1'
Notice: data for b: found it: 2
Notice: /Stage[main]/Then_test/Notify[data for b: found it: 2]/message: defined 'message' as 'data for b: found it: 2'
Notice: data for c: found it: 3
Notice: /Stage[main]/Then_test/Notify[data for c: found it: 3]/message: defined 'message' as 'data for c: found it: 3'
Notice: data for d:
Notice: /Stage[main]/Then_test/Notify[data for d: ]/message: defined 'message' as 'data for d: '
Notice: data for e: found it: 4
Notice: /Stage[main]/Then_test/Notify[data for e: found it: 4]/message: defined 'message' as 'data for e: found it: 4'
Notice: data for f: found it: 5

lest

lest is the other way arround to then, it will return you the given found value, but allow you to act on undef.

$my_hash = {
  'a' => '1',
  'b' => '2',
  'c' => '3',
  'e' => '4',
  'f' => '5',
}

[ 'a', 'b', 'c', 'd', 'e', 'f' ].each | $_search_item | {
  notify{"data for ${_search_item}: ${my_hash[$_search_item].lest || {"missing data"}}":}
}

Here we have now the other way arround, you see that as long lest gets values returned, it does not touches them and hand them directly back. But if it receives an undef value, it starts to act on it and replaces it in our test with the string missing data:

Notice: data for a: 1
Notice: /Stage[main]/Lest_test/Notify[data for a: 1]/message: defined 'message' as 'data for a: 1'
Notice: data for b: 2
Notice: /Stage[main]/Lest_test/Notify[data for b: 2]/message: defined 'message' as 'data for b: 2'
Notice: data for c: 3
Notice: /Stage[main]/Lest_test/Notify[data for c: 3]/message: defined 'message' as 'data for c: 3'
Notice: data for d: missing data
Notice: /Stage[main]/Lest_test/Notify[data for d: missing data]/message: defined 'message' as 'data for d: missing data'
Notice: data for e: 4
Notice: /Stage[main]/Lest_test/Notify[data for e: 4]/message: defined 'message' as 'data for e: 4'
Notice: data for f: 5
Notice: /Stage[main]/Lest_test/Notify[data for f: 5]/message: defined 'message' as 'data for f: 5'

Full samples

Two items from a hiera hash inside of an array to a array with hashes

How does it look like in hiera the array:

myvariable:
  - host_share:   'myserver0'
    ip_share:     '172.20.91.42'
    port_share:   '22001'
    id_share:     'ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8'
    ensure_share: 'present'
  - host_share:   'myserver1'
    ip_share:     '172.20.91.42'
    port_share:   '22001'
    id_share:     'ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8'
    ensure_share: 'present'
  - host_share:   'myserver2'
    ip_share:     '172.20.91.42'
    port_share:   '22001'
    id_share:     'ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8'
    ensure_share: 'present'

$finetestarr_hash = $myvariable.map | $hosts | {
$hosts.filter | $valuefilter | {
    $valuefilter[0] == 'id_share' or $valuefilter[0] == 'ensure_share'
  }.map | $key, $value | { $value }
}.map | $key, $value| { { $value[0] => $value[1] }}

The output will be:

$ puppet agent -e "$(cat ./puppetfile)"
iNotice: [{ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}, {ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}, {ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}]
Notice: /Stage[main]/Main/Notify[{ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}, {ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}, {ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}]/message: defined 'message' as '[{ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}, {ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}, {ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}]'
Notice: Applied catalog in 0.03 seconds

So our result will be a has which looks like:

$fintestarr_hash => [
  {'ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8' => 'present'},
  {'ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8' => 'present'},
  {'ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8' => 'present'},
]

Two items from a hiera hash inside of an array to a hash

How does it look like in hiera the array:

myvariable:
  - host_share:   'myserver0'
    ip_share:     '172.20.91.42'
    port_share:   '22001'
    id_share:     'ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8'
    ensure_share: 'present'
  - host_share:   'myserver1'
    ip_share:     '172.20.91.42'
    port_share:   '22001'
    id_share:     'ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8'
    ensure_share: 'present'
  - host_share:   'myserver2'
    ip_share:     '172.20.91.42'
    port_share:   '22001'
    id_share:     'ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8'
    ensure_share: 'present'

$myvariable = [ { 'host_share' => 'myserver0', 'ip_share' => '172.20.91.42', 'port_share' => '22001', 'id_share' => 'ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8', 'ensure_share' => 'present' } , { 'host_share' => 'myserver1', 'ip_share' => '172.20.91.42', 'port_share' => '22001', 'id_share' => 'ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8', 'ensure_share' => 'present' }, { 'host_share' => 'myserver2', 'ip_share' => '172.20.91.42', 'port_share' => '22001', 'id_share' => 'ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8', 'ensure_share' => 'present' } ]

$testhas = flatten(map($myvariable) | $mreh1 | {
    map($mreh1) | $key, $value | {
      flatten($key, $value)
    }
  }
).filter | $v | {
  $v =~ /.*-.*-.*-.*-.*-.*/ or $v == 'present' or $v == 'absent'
}

$fintesthash = $testhas.slice(2).reduce( {} ) | Hash $key, Array $value | {
  $key + $value
}

notify{"${fintesthash}":}

or to do it like above, where we filter on keys:

$fintesthash = $myvariable.map | $hosts | {
$hosts.filter | $valuefilter | {
    $valuefilter[0] == 'id_share' or $valuefilter[0] == 'ensure_share'
  }.map | $key, $value | { $value }
}.slice(2).reduce( {} ) | Hash $key, Array $value | {
  $key + $value
}

notify{"${fintesthash}":}

The output will be:

$ puppet agent -e "$(cat ./puppetfile)"
iNotice: {ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present, ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present, ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}
Notice: /Stage[main]/Main/Notify[{ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present, ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present, ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}]/message: defined 'message' as '{ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present, ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present, ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8 => present}'
Notice: Applied catalog in 0.03 seconds

So our result will be a has which looks like:

$fintesthash => {
  'ASDF01-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8' => 'present',
  'ASDF11-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8' => 'present',
  'ASDF21-ASDF2-ASDF3-ASDF4-ASDF5-ASDF6-ASDF7-ASDF8' => 'present',
}

Get one item from each hash and sum them up

The idea behind that that is, to check if the sum of the same item from different hashes is exceeding a max value

How does it look like in hiera the array:

queue_definition:
  mq_number1:
    enable:    'enabled'
    msg_limit: '50'
  mq_number2:
    enable:    'enabled'
    msg_limit: '80'

The below puppet code will return you into the variable sum_queue_msg_limit the sum of the items.

It is also able to ignore undefined items.

In the variable mq_default_msg_limit the default is defined

$sum_queue_msg_limit=reduce(
  filter(
    $queue_definition.map |$key, $value| {
      if $value['msg_limit'] != undef {
        $value['msg_limit']
      } else {
        $mq_default_msg_limit
      }
    }
  ) |$filter_number| {
    $filter_number != undef
  }
) |$memo, $part| {
  $memo + $part
}

What is the code doing in detail.

It starts with a map to be able to access the inner hash. On this outcome, a filter gets applied, to remove undefined items.

If the filter would not be performed, you would get an array like that [,10,,5] if the item would be not defined in some hashes.

Next is to reduce the array which we got from the filter, which allows us to sum up the found values.

Why do we have a if condition in the map block, when we are using the filter afterwards.

It is in there to show you, how you could place a default value if the item was not found.

After the code, you can work with the variable sum_queue_msg_limit as you are used to e.g. using it in an if or case

Calculate occurrence of strings in array

# in oneline
['a', 'b', 'c', 'c', 'd', 'b'].merge | $hsh, $v | { { $v => $hsh[$v].lest || { 0 } + 1 } }

# better readable
['a', 'b', 'c', 'c', 'd', 'b'].merge | $hsh, $v | {
  {
    $v => $hsh[$v].lest || { 0 } + 1
  }
}

This will result in { a => 1, b => 2, c => 2, d => 1 }

Add item to array even it could be empty

If you want to combine two arrays or you want to add a single item, but potentially them variable is empty, it could be that the build of the catalog fails, as it can not deal with e.g. empty requires. To sort out things like this, there is a small handy trick for that.

Normally appending to an array is done by using << or +. Of course, they are faster to write but bring some disadvantages.

For example, if you want to combine two arrays, you will get something like that:

$test1 = ['1','2']
$test2 = ['5','6']

$result1 = $test1 << $test2
Notice: [1, 2, [5, 6]]

$result2 = $test1 + $test2
Notice: [1, 2, 5, 6]

With + we can see that it also flatted out the result, with << it really adds the array as third item into the original one. Lets do the same and assume that $test2 is empty like this: []

$test1 = ['1','2']
$test2 = []

$result1 = $test1 << $test2
Notice: [1, 2, []]

$result2 = $test1 + $test2
Notice: [1, 2]

Again on the + operation, it flatted out the result, and on << operation, it again added the array, but it is empty.

Now lets see how it behaves if we want to add a string with the same operators.

$test1 = ['1','2']
$test2 = '5'

# just do one of both, other wiese you will get a duplicate declaration issue
$result1 = $test1 << $test2
Notice: [1, 2, 5]

$result2 = $test1 + $test2
Notice: [1, 2, 5]

Both are acting in the same way and just add a new item to the array.

What happens now if the it is an empty string

$test1 = ['1','2']
$test2 = ''

# just do one of both, other wiese you will get a duplicate declaration issue
$result1 = $test1 << $test2
Notice: [1, 2, ]

$result2 = $test1 + $test2
Notice: [1, 2, ]

Surprise, the same as above, but now the + operation did not performed a flatting on the result.

What does that mean, if you are not sure what data type the variable will have it is hard to full fill all requirements by just using + and << operators.

What you could do, is to use this short function calls, they will always return a full single dimensional array, does not matter if you append an array (empty or not) to another array, or a string (empty or not).

$test1 = ['1','2']
$test2 = ''

$result = flatten($test1, $test2).filter | $ft | { $ft =~ /./ }
Notice: [1, 2]

$test1 = ['1','2']
$test2 = ['']

$result = flatten($test1, $test2).filter | $ft | { $ft =~ /./ }
Notice: [1, 2]

$test1 = ['1','2']
$test2 = ['5','6']

$result = flatten($test1, $test2).filter | $ft | { $ft =~ /./ }
Notice: [1, 2, 5, 6]

The + operation does not only work for Arrays, it is also working fine with hashs (only tested right now with simple ones) e.g.:

$asdf = {'dev01' => 'num1' , 'dov01' => 'num2'}
$qwer = {'dev91' => 'num9' , 'dov91' => 'num99'}
$asdfqwer = $asdf + $qwer
Notice: {dev01 => num1, dov01 => num2, dev91 => num9, dov91 => num99}

ERB validation

To validate your ERB template, pipe the output from the ERB command into ruby:

$ erb -P -x -T '-' example.erb | ruby -c

The -P switch ignores lines that start with ‘%’, the -x switch outputs the template’s Ruby script, and -T '-' sets the trim mode to be consistent with Puppet’s behavior. This output gets piped into Ruby’s syntax checker (-c).

If you need to validate many templates quickly, you can implement this command as a shell function in your shell’s login script, such as .bashrc, .zshrc, or .profile:

validate_erb() {
      erb -P -x -T '-' $1 | ruby -c
}

You can then run validate_erb example.erb to validate an ERB template.

Documenting modules with puppet string

online docu

For debian it is to less to install it with apt even the package exists (puppet-strings) it has to be installed over gem

$ /opt/puppetlabs/puppet/bin/gem install puppet-strings

Puppet Server

Unclear server errors

What do I mean with unclear server errors. Easy to say, I mean with that, errors which do not tell you right ahead in the service logs where the issue is coming from and only point you into direction. For example with the uninitialized constant Concurrent error, it shows only that there is an issue with the JRuby instance while loading it.

uninitialized constant Concurrent

These errors are most of the time pointing back to the PDK (Puppet Development Kit). In version 2.6.1 ther was an issue with the third party module puppet-lint-top_scope_facts-check which caused issues inside the module code base, there it was an issue with determing the ocrect PDK version for the rake tests. Also had this issue with PDk:3.0.1/puppetserver:8.4.0. No planed rake test have been added except for the default one from upstream/puppetlaps puppet-modules.

How did it looked like in the logs, it started normal as alwyas:

2024-01-18T21:26:28.380+01:00 INFO  [async-dispatch-2] [p.t.s.s.scheduler-service] Initializing Scheduler Service
2024-01-18T21:26:28.403+01:00 INFO  [async-dispatch-2] [o.q.i.StdSchedulerFactory] Using default implementation for ThreadExecutor
2024-01-18T21:26:28.412+01:00 INFO  [async-dispatch-2] [o.q.c.SchedulerSignalerImpl] Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2024-01-18T21:26:28.413+01:00 INFO  [async-dispatch-2] [o.q.c.QuartzScheduler] Quartz Scheduler v.2.3.2 created.
2024-01-18T21:26:28.413+01:00 INFO  [async-dispatch-2] [o.q.s.RAMJobStore] RAMJobStore initialized.
2024-01-18T21:26:28.414+01:00 INFO  [async-dispatch-2] [o.q.c.QuartzScheduler] Scheduler meta-data: Quartz Scheduler (v2.3.2) '85c32857-6191-41e9-9699-fa46f795797f' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2024-01-18T21:26:28.414+01:00 INFO  [async-dispatch-2] [o.q.i.StdSchedulerFactory] Quartz scheduler '85c32857-6191-41e9-9699-fa46f795797f' initialized from an externally provided properties instance.
2024-01-18T21:26:28.414+01:00 INFO  [async-dispatch-2] [o.q.i.StdSchedulerFactory] Quartz scheduler version: 2.3.2
2024-01-18T21:26:28.414+01:00 INFO  [async-dispatch-2] [o.q.c.QuartzScheduler] Scheduler 85c32857-6191-41e9-9699-fa46f795797f_$_NON_CLUSTERED started.
2024-01-18T21:26:28.416+01:00 INFO  [async-dispatch-2] [p.t.s.w.jetty10-service] Initializing web server(s).
2024-01-18T21:26:28.439+01:00 INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for service 'puppet-profiler', version 8.4.0
2024-01-18T21:26:28.441+01:00 INFO  [async-dispatch-2] [p.s.j.jruby-puppet-service] Initializing the JRuby service
2024-01-18T21:26:28.449+01:00 INFO  [async-dispatch-2] [p.s.j.jruby-pool-manager-service] Initializing the JRuby service
2024-01-18T21:26:28.454+01:00 INFO  [async-dispatch-2] [p.s.j.jruby-puppet-service] JRuby version info: jruby 9.4.3.0 (3.1.4) 2023-06-07 3086960792 OpenJDK 64-Bit Server VM 17.0.10-ea+6-Debian-1 on 17.0.10-ea+6-Debian-1 +jit [x86_64-linux]
2024-01-18T21:26:28.459+01:00 INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-internal] Creating JRubyInstance with id 1.
2024-01-18T21:26:28.466+01:00 INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for service 'jruby-metrics', version 8.4.0
2024-01-18T21:26:32.743+01:00 ERROR [clojure-agent-send-pool-0] [p.t.internal] shutdown-on-error triggered because of exception!

But then log lines like this showed up:

2024-01-23T00:00:17.923+01:00 ERROR [clojure-agent-send-pool-0] [p.t.internal] shutdown-on-error triggered because of exception!
java.lang.IllegalStateException: There was a problem adding a JRubyInstance to the pool.
        at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__34143$add_instance__34148$fn__34152.invoke(jruby_agents.clj:58)
        at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__34143$add_instance__34148.invoke(jruby_agents.clj:47)
        at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__34170$prime_pool_BANG___34175$fn__34179.invoke(jruby_agents.clj:76)
        at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__34170$prime_pool_BANG___34175.invoke(jruby_agents.clj:61)
        at puppetlabs.services.jruby_pool_manager.impl.instance_pool$fn__34732$fn__34733.invoke(instance_pool.clj:16)
        at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:403)
        at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:378)
        at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:388)
        at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:378)
        at puppetlabs.trapperkeeper.internal$fn__14886$shutdown_service__14891$fn$reify__14893$service_fnk__5336__auto___positional$reify__14898.shutdown_on_error(internal.clj:448)
        at puppetlabs.trapperkeeper.internal$fn__14833$G__14812__14841.invoke(internal.clj:411)
        at puppetlabs.trapperkeeper.internal$fn__14833$G__14811__14850.invoke(internal.clj:411)
        at clojure.core$partial$fn__5908.invoke(core.clj:2642)
        at clojure.core$partial$fn__5908.invoke(core.clj:2641)
        at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__34117$send_agent__34122$fn__34123$agent_fn__34124.invoke(jruby_agents.clj:41)
        at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2050)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.lang.Agent$Action.doRun(Agent.java:114)
        at clojure.lang.Agent$Action.run(Agent.java:163)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.jruby.embed.EvalFailedException: (NameError) uninitialized constant Concurrent::RubyThreadLocalVar
Did you mean?  Concurrent::RubyThreadPoolExecutor
        ...

Have a look if puppet craeted a file simimlart to this /tmp/clojure-[0-9]+.edn.

In there you will find some details about that error as well, specially the files which caused the issue.

Sample:

{:clojure.main/message
 "Execution error (NameError) at org.jruby.RubyModule/const_missing (org/jruby/RubyModule.java:4332).\n(NameError) uninitialized constant Concurrent::RubyThreadLocalVar\nDid you mean?  Concurrent::RubyThreadPoolExecutor\n",
 :clojure.main/triage
 {:clojure.error/class org.jruby.exceptions.NameError,
  :clojure.error/line 4332,
  :clojure.error/cause
  "(NameError) uninitialized constant Concurrent::RubyThreadLocalVar\nDid you mean?  Concurrent::RubyThreadPoolExecutor",
  :clojure.error/symbol org.jruby.RubyModule/const_missing,
  :clojure.error/source "org/jruby/RubyModule.java",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type java.lang.IllegalStateException,
    :message "There was a problem adding a JRubyInstance to the pool.",
    :at
    [puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__34143$add_instance__34148$fn__34152
     invoke
     "jruby_agents.clj"
     58]}
   {:type org.jruby.embed.EvalFailedException,
    :message
    "(NameError) uninitialized constant Concurrent::RubyThreadLocalVar\nDid you mean?  Concurrent::RubyThreadPoolExecutor",
    :at
    [org.jruby.embed.internal.EmbedEvalUnitImpl
     run
     "EmbedEvalUnitImpl.java"
     134]}
   {:type org.jruby.exceptions.NameError,
    :message
    "(NameError) uninitialized constant Concurrent::RubyThreadLocalVar\nDid you mean?  Concurrent::RubyThreadPoolExecutor",
    :at
    [org.jruby.RubyModule
     const_missing
     "org/jruby/RubyModule.java"
     4332]}],
  :trace
  [[org.jruby.RubyModule
    const_missing
    "org/jruby/RubyModule.java"
    4332]
   [RUBY
    <main>
    "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/thread_local.rb"
    7]
   [org.jruby.RubyKernel require "org/jruby/RubyKernel.java" 1071]
   [org.jruby.RubyKernel

In this case, all the affected files where part of the puppet-agent package.

$ dpkg -S /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/thread_local.rb
puppet-agent: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/thread_local.rb

I just gave it a try and performed a reinstall

$ apt install --reinstall puppet-agent

and it was able to fix the above mentioned files, which looks like that during the upgrade of the agent an issue occured and caused that problem.