Johnny ( Chianing ) Wang : Puppet Study Note - More about Resource, Manifest, Classes, Modules and Catalog

Resource

Puppet code is composed primarily of resource declarations which describes the "state of the system" such as a certain user or file should exist, or a package should be installed.
Resources are the fundamental "building blocks" used to model system state in Puppet.
They describe the "desired end state" of unique elements managed by Puppet on the system.
Everything that Puppet manages is expressed as a "resource".
Puppet uses a "declarative language" to define our configuration items (resources).
"Being declarative" creates an important distinction between Puppet and many other configuration tools.
A declarative language makes statements about the state of our configuration, for example, it declares that a package should be installed or a service should be started.

Namever

PK(Primary Key) for resource
Namever can be thought of as the resource's PK since most resources that need to be managed have unique identifiers.
If we don't specified, its value will default to the title (name) of the resource.
Although the "Title" and "Namever" are commonly the same, they serve two different purposes in Puppet.

The "Title" is used to reference the resource in the Puppet Catalog.
The "Namever" indicates the system's name for the resource.

Here is an example for the "Namever" is not the same as a resource's "Title". The Title of that resource is ntp and its Namever is httpd. This resource can be referenced as ntp, but the package under management is httpd.

custom_package { 'ntp':
name => 'httpd',
}

Provider

Providers implement the procedure used to manage resource.
Multiple providers can be implemented for a single type, allowing the same resource to be applied on different OS.
Puppet includes one or more providers for each of its native types to cross variety of Unix, Linux and even Windows platforms.
Although Puppet will automatically select an appropriate default provider, we can override the default with the provider attribute. (For example, package resources on Red Hat systems default to the yum provider, but we can specify provider => gem to install Ruby libraries with the gem command.)

Properties

Puppet's Resource Abstraction Layer (RAL) provides a clear separation (or integration) between types and providers. Then "Properties" are the key to this separation.

eg: ensure is a special property that models the existence of a resource. Until we implement ensure, resource cannot be created or destroyed.

package { 'ntp':
ensure => present,
}

Most and Starting point of resource implementation - By asking the following questions

Can I discover the state of this attribute ?
Can I update the state of this attribute ?"

If the answer to both of those questions is yes, then that attribute should be implemented as a "property.
In general, if the answer to one or both of these questions is no, then the characteristic should not be a property.

eg: The ensure property is special because it's used to create and destroy resource.You can set this property up on your resource type just by calling the ensurable in your type definition:

Puppet::Type.newtype(:database) do
    ensurable
    ...
  end

  Puppet::Type.newtype(:database) do
    ensurable
    newproperty(:owner) do
    desc "The owner of the database."
    ...
    end
  end

  newproperty(:owner) do
    validate do |value|
    unless value =~ /^\w+/
raise ArgumentError, "%s is not a valid user name" % value
    end
    end
  end

Parameter

Parameter vs Property

Parameters supply additional information to providers, which is used to manage its properties. In contrast with properties, parameters are not discovered from the system and cannot be created or updated.
Parameters allow us to specify additional context or the ability to override a provider's default behaviour.
Both properties and parameters will become the resource attributes available when declaring a resource of the new type.
The difference between the two is subtle but important:

Properties(屬性)should map "more or less directly to something measurable" on the target system.

For example, the UID and GID of a user account would be properties, since their current state can be queried or changed.
In practical terms, setting a value for a property causes a method to be called on the provider.

Parameters(參數)change how Puppet manages a resource, but "do NOT necessarily map directly to something measurable".

Parameters are defined essentially exactly the same as properties; the only difference between them is that parameters never result in methods being called on "providers".

RAL - Resource Abstraction Layer

Puppet handles the how by knowing how different platforms and OS manage certain "types" of resources. Each type has a number of "providers" - OS.
The full list of typs can be found on Puppetlabs. (https://docs.puppetlabs.com/references/stable/type.html)
A provider contains the how of managing packages using a particular package management tool, eg: RedHat - yum, Ubuntu apt-get. Here are the basic transaction.

FACTOR - A tool to return information about Agent, including what OS it is running.
Puppet then chooses the appropriate package "Provider" for that OS.
And use "Provider" to check if a specific package is installed.

if not --> Puppet install it.
if yes --> Puppet do nothing

---idempotency---vase state to take action

eg. install vim base on return fact then choose appropriate package provider for OS to check vim package installed. (RHEL-yum, Ubuntu->apttude, Solaris->pkg)
example of idempotency

If the package is not installed, Puppet will install it.
If the package is already install, Puppet does nothing.

Finally Agent will report back to Master of it's success or failure in applying the resource.

Fact and Facter

Key=>value pair: of system resource for each node.

Puppet gathers facts about each of it's nodes with a tool called facter.
It gathers information that is useful for system configuration such as OS name, hostname, IP Address, SSH key ... etc
These facts are gathered when the agent runs.
The facts are then sent to the Puppet master, and automatically created as variables available to Puppet at top scope.

Customize facts --- yes !to perform our configurations.

Command to list all the facts

# facter

These facts are made available as variables that can be used in out Puppet configuration.
Facter also helps Puppet understand how to manage particular resource on an agent.

eg: if Facter tells Puppet that a host runs Ubuntu, then Puppet knows to use aptitude to install packages on that agent.

Puppet Engine - Puppet's Transaction Layer is it's engine.

A Puppet transaction encompasses the process of configuring each host, including these steps

Interpret and compile our configuration ( *.pp )
Communicate the compiled configuration ( *.pp ) to the Agent.
Apply the configuration to the Agent.
Report ( check Agent and communicate to Master ) the result of that application to the Master

Puppet then takes the resources and compiles them into a catalog for each agent. The catalog is sent to the host and applied by the Puppet agent.The results of this application are then sent back to the master in the form of a report. The transaction layer allows configuration to be created and applied repeatedly on the host.

--- idempotency --- depend on state to take action
eg: check repeatedly to make sure the configuration consistent

Puppet is not fully transactional, not logging, can't roll back. However, model transaction is a noop ( dry-run / no-operation ) mode. - allow to test the execution of changes without apply them.

Manifest

Puppet codes are called manifests. Manifests are composed of puppet code and their filenames use the .pp extension.
The default main manifest is the example we play in previous example site.pp.

eg: Every 30 minutes.

It sends facts about itself to the master, and pull a current catalog which is a compiled list of resources and their desired states that are relevant to the agent, determined by the main manifest.
The agent node will then attempt to make the appropriate changes to achieve its desired state.
This cycle will continue as long as the Puppet master is running and communicating with the agent node.

Classes

Code Blocks: a block of codes allow to reuse in manifest, similar concept with convention class in OO.
Collection of Resources
Easier to read.

  # /root/examples/modules1-ntp1.pp
  # declare
  # class class_name {

  class ntp {
case $operatingsystem {
centos, redhat: {
$service_name = 'ntpd'
$conf_file = 'ntp.conf.el'
}
debian, ubuntu: {
$service_name = 'ntp'
$conf_file = 'ntp.conf.debian'
}
}

package { 'ntp':
ensure => installed,
}
file { 'ntp.conf':
path => '/etc/ntp.conf',
ensure => file,
require => Package['ntp'],
source => "/root/examples/answers/${conf_file}"
}
service { 'ntp':
name => $service_name,
ensure => running,
enable => true,
subscribe => File['ntp.conf'],
}
}

A class declaration occurs when a class is called in a manifest.
A class declaration tells Puppet to evaluate the code within the class.
Class declarations come in two different flavours: normal and resource-like:

  normal:
  include ntp

  resource-like:
  class ntp {
      ...
  }

Modules

Module is a portable collection of manifest that contain resources, classes, definitions, files, and templates.
To help you split up your manifests into an easier to understand structure, Puppet uses modules and module auto loader.

A module is a collection of manifests and data, and the have a specific "directory structure".
Setting a global value for modulepath in puppet.conf
It's useful for organizing Puppet Code, because Puppet allow to split code into multiple manifests.

If a class is defined in a module, you can declare that class by name in any manifest.

eg: include class_name

Puppet will "automatically find and load" the manifest that contains the class definition.

eg: We can pile of modules with sophisticated Puppet code, and your site.pp manifest can look like this, as long as all the classes codes are in the same module directory.

# /etc/puppetlabs/puppet/manifests/site.pp
include ntp
include apache
include mysql
include mongodb
include build_essential

eg: tree command to check module with structure
[root@johnny ~/puppetcode/modules:master±]# tree review/
review/
├── files
│ └── shells
├── manifests
│ └── init.pp
├── templates
│ └── motd.erb
└── tests
└── init.pp

Catalog

A compiled list of resources and their desired states that are relevant to the agent
Manifest --> Compile --> Catalog --> Apply ( Query State -> Enforce Defined State ) --> Defined System State

When configuring a node, Puppet Agent uses a document called a "Catalog", which it downloads from a Puppet Master server.
The Catalog describes the "desired state for each resource" that should be managed, and may "specify dependency information for resource that should be managed in a certain order".
It's a collection of resources compiled from a set of manifests.
It's a composition of resources that are used to model a service or a system.
It's easily intro-spected to better understand how a system should be configured, and what dependencies might exist.

ref: https://docs.puppetlabs.com/learning/agent_master_basic.html
eg: it might be easier to describe via diagrams ( Agent vs Agent <-> Master )

Graph - vertices (Puppet Resource) with edges ( dependencies )

The data structure of the catalog is "graph".
Graphs are characterized as a collection of objects where some of the object pairs are interconnected.
The objects are referred to as vertices (Puppet Resource) and the links between pairs of those object are edges ( dependencies )

Setup in Configuration
[agent]
report = true
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
graph = true

You can use it when you run puppet apply
# puppet apply --graph site.pp

# puppet apply --graph site.pp
Notice: Compiled catalog for johnny.puppetlabs.vm in environment production in 0.03 seconds
Notice: Hello Puppet Agent: johnny.puppetlabs.vm
Notice: /Stage[main]/Main/Node[default]/Notify[Hello Puppet Agent: johnny.puppetlabs.vm]/message: defined 'message' as 'Hello Puppet Agent: johnny.puppetlabs.vm'
Notice: Finished catalog run in 0.46 seconds

Generate .dot file under graphdir in configuration and readable by OmniGraffle (OS X) or graphviz.

# ls /var/opt/lib/pe-puppet/state/graphs/ -lrth
total 12K
-rw-r--r-- 1 root root 1.6K Jan 26 06:37 resources.dot
-rw-r--r-- 1 root root 1.3K Jan 26 06:37 relationships.dot
-rw-r--r-- 1 root root 2.4K Jan 26 06:37 expanded_relationships.dot
# date
Mon Jan 26 06:37:49 UTC 2015

Noop

Dry-Run Mode, it's a way for Puppet to simulate manifests and report pending changes.
When noop mode is enabled ( noop flag ). Puppet queries each resource and reports differences between the system and it's desired state.
This is useful for "seeing what changes Puppet will make without actually executing the changes".
Basic Logic: (skip updating : record diff - desire vs observe ( state )

When running noop mode, it skips for updating the underlying system, and records differences between desired and observed state as event w/o making any modification to the system.

eg: here is standard help

# puppet agent --help

* --noop:
Use 'noop' mode where the daemon runs in a no-op or dry-run mode. This is useful for seeing what changes Puppet will make without actually executing the changes. (This is a Puppet setting, and can go in puppet.conf. Note the special 'no-' prefix for boolean settings on the command line.)

# puppet agent -t --noop

Reference:

http://puppetlabs.com/blog/pro-puppet-second-edition
https://docs.puppetlabs.com

http://www.bogotobogo.com/DevOps/Puppet/puppet_basics_of_code_terminology.php

PS: Share the quote I like. For pushing to your-self to the limitation.

"There are no two words in the English language that are more harmful than "good job"."

- (Whiplash), 2014

Johnny ( Chianing ) Wang

Sunday, January 25, 2015

Puppet Study Note - More about Resource, Manifest, Classes, Modules and Catalog