Advanced Puppet

Design
Craig Dunn, Puppet Camp Berlin 2014

Friday, 11 April 14
 Hello
Craig Dunn
Puppet user since 2008
Previously worked for Puppet Labs
Founder of Enviatics
IT automation engineer and trainer
Based in Spain but work anywhere
Friday, 11 April 14
 My talk

Avoiding pain
Writing good modules
Challenges of codebase design
Roles / Profiles

Friday, 11 April 14
 But first....
Everyone loves polls, right?

Friday, 11 April 14
 Thinking right

Business has requirements

We use technology to fulfill them
But its not that simple!

Friday, 11 April 14
 Business logic does not often
align with technology design

Friday, 11 April 14
 Business requirement
We have 3
applications we
need to deploy using
Puppet

Friday, 11 April 14
 Puppetize

Write 3 Puppet modules for 3 applications

Because that was the requirement
Was it?

Friday, 11 April 14
 Its not working

Friday, 11 April 14
 Lets suppose

Each application is a set of shared

components implemented different ways

Friday, 11 April 14
 The business view

Application X

Application Y Application Z

Friday, 11 April 14
 The technical reality

Application X

ApplicationA
Ypplication Z

Friday, 11 April 14
 Danger Signs
Resources being declared in two modules
You dont know where your
implementation fits
Lots of logic at a node level
Repetition and duplication
The if statement is your go-to-guy
Friday, 11 April 14
 Catastrophic Signs
if ($::hostname =~ /^host[0-3].*/) {
package { my-special-app:
ensure => installed,
}
}

Friday, 11 April 14
 Catastrophic Signs
if !defined(Package[httpd]) {
package { httpd:
ensure => installed,
}
}

Friday, 11 April 14
 Catastrophic Signs
ensure_resource(package,httpd,{ensure
=> installed})

Friday, 11 April 14
 Catastrophic Signs
ensure_resource(package,httpd,{ensure
=> installed})

if function_defined_with_params(["#{type}[#{item}]", params])
Puppet.debug("Resource #{type}[#{item}] not created because it already exists")
else
Puppet::Parser::Functions.function(:create_resources)
function_create_resources([type.capitalize, { item => params }])
end

Friday, 11 April 14
 World ending signs

You use parser=future in production

You arent regretting it yet
You then implemented order=manifest

Friday, 11 April 14
 An unhappy Puppet

Friday, 11 April 14
 Stop thinking about
what it looks like

Break everything down into components

Granularity is the key
Think about what it actually is

Friday, 11 April 14
 Writing modules

Granularity
Portability
Flexibility of implementation

Friday, 11 April 14
 class web { A bad module
$docroot=/var/www/sites
$listenaddr=10.0.1.10
$servername=myweb.foo.com

package { httpd:
ensure => installed,
}

package { php5:
ensure => installed,
require => Package[httpd],
}

package { mysql-server:
ensure => installed,
}

file { /etc/httpd/httpd.conf:
ensure => file,
content => template(web/httpd.conf.erb),
}
}

Friday, 11 April 14
 class web { A bad module
Granularity
$docroot=/var/www/sites
$listenaddr=10.0.1.10
$servername=myweb.foo.com

package { httpd:
ensure => installed,
}

package { php5:
ensure => installed,
require => Package[httpd],
}

package { mysql-server:
ensure => installed,
}

file { /etc/httpd/httpd.conf:
ensure => file,
content => template(web/httpd.conf.erb),
}
}

Friday, 11 April 14
 class web { A bad module
Granularity
$docroot=/var/www/sites
$listenaddr=10.0.1.10
$servername=myweb.foo.com

package { httpd:
ensure => installed,
} Portability
package { php5:
ensure => installed,
require => Package[httpd],
}

package { mysql-server:
ensure => installed,
}

file { /etc/httpd/httpd.conf:
ensure => file,
content => template(web/httpd.conf.erb),
}
}

Friday, 11 April 14
 class web { A bad module
Granularity
$docroot=/var/www/sites
$listenaddr=10.0.1.10
$servername=myweb.foo.com

package { httpd:
ensure => installed,
} Portability
package { php5:
ensure => installed,
require => Package[httpd],
}
Implementation
package { mysql-server:
ensure => installed,
}

file { /etc/httpd/httpd.conf:
ensure => file,
content => template(web/httpd.conf.erb),
}
}

Friday, 11 April 14
 Keep your modules
granular

Manage only resources in the scope of the

module
Small modules are ok!

Friday, 11 April 14
 A granular module
class apache {

$docroot=/var/www/sites
$listenaddr=10.0.1.10
$servername=myweb.foo.com

package { httpd:
ensure => installed,
}

file { /etc/httpd/httpd.conf:
ensure => file,
content => template(web/httpd.conf.erb),
}
}

Friday, 11 April 14
 A granular module
class apache {

$docroot=/var/www/sites
$listenaddr=10.0.1.10
Granularity
$servername=myweb.foo.com

package { httpd:
ensure => installed,
}

file { /etc/httpd/httpd.conf:
ensure => file,
content => template(web/httpd.conf.erb),
}
}

Friday, 11 April 14
 A granular module
class apache {

$docroot=/var/www/sites
$listenaddr=10.0.1.10
Granularity
$servername=myweb.foo.com

package { httpd:
ensure => installed, Portability
}

file { /etc/httpd/httpd.conf:
ensure => file,
content => template(web/httpd.conf.erb),
}
}

Friday, 11 April 14
 Sharing is good

Re-usable by others in your team

You can publish to the forge!
People will collaborate

Friday, 11 April 14
 Its all about sharing!

Friday, 11 April 14
 You are gonna share
your s**t arent you?

Friday, 11 April 14
 Sharing is not just
pull requests
Share your ideas in blog posts
What worked? What didnt?
Discuss and collaborate on mailing lists
IRC

Friday, 11 April 14
 Making sharing easier

Data separation (Hiera)

Allow the user flexibility of implementation

Friday, 11 April 14
 defaults params pattern

Use a parameterized class

Default from an inherited class
Allow the user to decide implementation

Friday, 11 April 14
 defaults pattern
class apache {
$packagename=httpd
$docroot=/var/www
$listenaddr=10.0.1.12
$servername=myweb.foo.com

package { $packagename:
ensure => installed,
}

.....

Friday, 11 April 14
 defaults pattern
class apache { In-module
$packagename=httpd private data
$docroot=/var/www
$listenaddr=10.0.1.12
$servername=myweb.foo.com
No way to override
package { $packagename:
ensure => installed,
}

.....

Friday, 11 April 14
 defaults
class apache (
pattern
$packagename=httpd,
$docroot=/var/www,
$listenaddr,
$servername,
) {

package { $packagename:
ensure => installed,
}

.....

Friday, 11 April 14
 defaults pattern
Override data on implementation

class { apache:
listenaddr => 10.0.1.2,
servername => foo.example.com,
}

Friday, 11 April 14
 defaults pattern
Overriding from Hiera

# /etc/puppet/hieradata/dev.yaml

---
apache::docroot: /var/dev/sites

Friday, 11 April 14
 defaults pattern
class apache::defaults {

$packagename=$::osfamily ? {
redhat => httpd,
debian => apache2,
default => httpd,
}
$docroot=/var/www
$listenaddr = $::ipaddress
$servername = $::fqdn
}
.....

Friday, 11 April 14
 defaults pattern
class apache::defaults {

$packagename=$::osfamily ? {
redhat => httpd,
debian => apache2,
Added logic to
default => httpd,
defaults
}
$docroot=/var/www
$listenaddr = $::ipaddress
$servername = $::fqdn
} Common defaults
.....

Friday, 11 April 14
 defaults pattern
class apache (
$packagename=$::apache::defaults::packagename,
$docroot=$::apache::defaults::docroot,
$listenaddr=$::apache::defaults::listenaddr,
$servername=$::apache::defaults::servername,
) inherits apache::defaults {

package { $::apache::packagename:
ensure => installed,
}

.....

Friday, 11 April 14
 defaults pattern
Granularity Smaller modules with smaller scope

Portability No hard coded data in modules

User can decided how / where to

Implementation override defaults without editing
the module

Friday, 11 April 14
 Component Modules

Friday, 11 April 14
 Now to build
something awesome
... but where?

Friday, 11 April 14
 Designing Puppet

Node Classification

Component Modules

Friday, 11 April 14
 Node-level logic
node basil {
class { apache:
version => latest,
}
class { motd: }
class { ssh: }
class { users:
default_shell => /bin/false,
}
Class[ssh] -> Class[users]
}

Friday, 11 April 14
 Node-level logic

node basil inherits base {

class { apache:
version => latest,
}
}

Friday, 11 April 14
 Node-level logic

Risks duplication and repetition

No guarantee of consistency
Pseudo nodes and inheritance trees will get
messy, fast.
TMI!

Friday, 11 April 14
 Thinking beyond the
module....

Puppet is a code base

You need an effective framework
Gluing everything together

Friday, 11 April 14
 Introducing Profiles

Friday, 11 April 14
 Profiles

Wrapper classes implementing component

modules
Define a logical technology stack
But just one!

Friday, 11 April 14
 Profiles
class profile::blog {

User <| group == webadmins |>

class { ::mysql::server: }
class { ::mysql::bindings:
php_enable => true,
}

class { ::wordpress:
install_dir => /var/www/wp,
}
}

Friday, 11 April 14
 Profiles

Component modules manage the resources

Profiles provide a layer of implementation

Friday, 11 April 14
 Profiles and
Components

Resources
Friday, 11 April 14
 Profiles and
Components

Components: Resource modelling

Resources
Friday, 11 April 14
 Profiles and
Components

Profiles : Implementation

Components: Resource modelling

Resources
Friday, 11 April 14
 Lessons learned

Granularity is good
Dont assume business logic will directly
translate to technology
Abstraction is awesome.... but thats
nothing new....

Friday, 11 April 14
 Abstraction is a core
principle of coding
Implementation is abstracted by methods
Methods abstracted by classes and modules
They are abstracted with libraries
Puppet is code!

Friday, 11 April 14
 Puppet is all about
abstraction
Providers are abstracted by types
Resources are abstracted by classes
Classes are abstracted by modules

Friday, 11 April 14
 Puppet is all about
abstraction
Providers are abstracted by types
Resources are abstracted by classes
Classes are abstracted by modules
Modules are abstracted by profiles

Friday, 11 April 14
 Focussing on
Abstraction
Weve turned business logic into a
technology stack
Can we translate that back into business
logic?
Why would we even want to do that?

Friday, 11 April 14
 Introducing roles

Translate to business logic

Identify the function of a server in human
terms
We never said business logic was a bad
thing

Friday, 11 April 14
 Configuration model
This is a acme server

include profiles::security
include profiles::users
include profiles::networking
include profiles::blog

Friday, 11 April 14
 Think about the users
Meet John, Susan and Bill.

Friday, 11 April 14
 John is a Sysadmin

Wants to ensure all servers have kernel

hardening, NTP and SSH Server installed
Wants to manage what packages, services,
files and other resources
Is responsible for maintaining all the
components of a any type of server

Friday, 11 April 14
 Susan is an application
specialist

Cares that a the node has Wordpress and

MySQL implemented properly
She probably doesnt care about how
sudoers is configured

Friday, 11 April 14
 Bill is an IT manager

Bill cares that the server is an ACME App

server
He probably doesnt understand what
sudoers is

Friday, 11 April 14
 What do they care
about?

John cares about modelling all resources

Susan cares about the technology stack
Bill cares about the business logic

Friday, 11 April 14
 In Puppet

Resource modelling is done in component

modules
The technology stack is defined in profiles
Where do we represent the business logic
for Bill?

Friday, 11 April 14
 Roles

Represent business logic, not technology

Define a set of technology stacks (profiles)
that make up the logical role
Allow the business to manage how the
infrastructure looks without defining
what it is

Friday, 11 April 14
 A node can only have
one role
A role can include as many profiles as
required to define itself
If a node requires two roles, it has by
definition become a new role

Friday, 11 April 14
 A node can only have
one role
A role can include as many profiles as
required to define itself
If a node requires two roles, it has by
definition become a new role
Something couldnt be a lion and a
kangaroo at the same time!

Friday, 11 April 14
 It would be a Lingaroo

Friday, 11 April 14
 Roles

One-to-one to nodes
One-to-many to profiles
Only implement profiles, thats it!

Friday, 11 April 14
 The Stack

Resources
Friday, 11 April 14
 The Stack

Components: Resource modelling

Resources
Friday, 11 April 14
 The Stack

Profiles : Implementation

Components: Resource modelling

Resources
Friday, 11 April 14
 The Stack
Roles : Business Logic

Profiles : Implementation

Components: Resource modelling

Resources
Friday, 11 April 14
 Role classes
This is a acme server

class role::acme {
include profiles::security
include profiles::users
include profiles::networking
include profiles::blog
}

Friday, 11 April 14
 Terminology

Profiles and Roles are Puppet modules

They are not special
Everything is a module

Friday, 11 April 14
 Classification

Assigning classes to a node

You can classify within Puppet code
(site.pp)
You can use an External Node Classifier
(ENC)

Friday, 11 April 14
 Classification
You can classify your nodes however you
want
Puppet Dashboard
Enterprise Console
Foreman
Site.pp
Custom script
Friday, 11 April 14
 Classification

node craig.puppetlabs.vm {
include roles::acme_app
}

Friday, 11 April 14
 Classification

Friday, 11 April 14
 Classification

With roles and profiles we just classify the

role to the node

Friday, 11 April 14
 The Stack
Roles : Business Logic

Profiles : Implementation

Components: Resource modelling

Resources

Friday, 11 April 14
 The Stack
Roles : Business Logic Classifier

Profiles : Implementation

Components: Resource modelling

Resources

Friday, 11 April 14
 Data Separation

If youre not using Hiera you are going

to break your data!

Friday, 11 April 14
 Data Separation

Use parameterized classes

Hiera data bindings
Component modules and profiles can look
up data from Hiera
Roles should NOT

Friday, 11 April 14
 Roles and Profiles
for DevOps
Full props to Laurent Bernaille from D2SI
Achieving Continuous Delivery and Devops
with Puppet
Puppet Camp Paris, 2014.

Friday, 11 April 14
 Roles and Profiles
for DevOps
Using roles and profiles makes it easier for
developers and ops to all collaborate on
Puppet
Developers write profiles for the their apps
Ops write profiles for their infrastructure
Roles encompass all of them

Friday, 11 April 14
 The Roles/Profiles
Stack
Classifier
Roles : Business Logic
Classification

Dev profiles Ops profiles

Hiera:
Data
Components: Resource modelling

Resources

Friday, 11 April 14
 Roles/Profiles/Devops
http://fr.slideshare.net/D2SI/d2-si-puppetcamp

Friday, 11 April 14
 Key benefits
Reduced node-level logic to a role.
Gain the ability to be flexible with
implementation
Business logic improves managability by
non-Puppet users
Edge cases are now easy to solve

Friday, 11 April 14
 Enough Preaching!

Friday, 11 April 14
 This is not the way to
design Puppet... Its a
way.

Friday, 11 April 14
 Can I implement this
design without roles?

Friday, 11 April 14
 Can I implement this
design without roles?

Yes.
You lose the layer of abstraction that
exposes business logic

Friday, 11 April 14
 Can my roles be
defined in my ENC?

Friday, 11 April 14
 Can my roles be
defined in my ENC?

Yes.
Keeping it in code makes it versionable

Friday, 11 April 14
 Cant I just use Hiera
to define profiles?

Friday, 11 April 14
 Cant I just use Hiera
to define profiles?

Technically yes.
You lose the flexibility to implement code
logic in profiles and it may become
restrictive

Friday, 11 April 14
 The fundamental
concepts....

Friday, 11 April 14
 The fundamental
concepts....

Abstraction, abstraction, abstraction

Friday, 11 April 14
 The fundamental
concepts....

Abstraction, abstraction, abstraction

Decoupling business logic, implementation
and resource modelling.

Friday, 11 April 14
 The fundamental
concepts....

Abstraction, abstraction, abstraction

Decoupling business logic, implementation
and resource modelling.
Separating data and code

Friday, 11 April 14
 The fundamental
concepts....

Abstraction, abstraction, abstraction

Decoupling business logic, implementation
and resource modelling.
Separating data and code
Reducing node-level complexity
Friday, 11 April 14
 Other resources

http://garylarizza.com/blog/2014/02/17/
puppet-workflow-part-2/
http://www.craigdunn.org/2012/05/239/

Friday, 11 April 14
 Danke. Questions?

Follow me at @crayfishX
Bug me on Freenode: crayfishx

In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beer
and helped me on my journey of discovery learning how to implement Puppet properly. R.I.P

Friday, 11 April 14