slides

space, →	next slide
←	previous slide
d	debug mode
## <ret>	go to slide #
c	table of contents (vi)
f	toggle footer
r	reload slides
z	toggle help (this)

loading presentation...

Securing and Extending Puppet for World Domination

Hi, I’m Richard Crowley

Equal opportunity technology hater
DevStructure’s operator and UNIX hacker

Configuration management
Cliff’s Notes

Your infrastructure is never an immutable black box. It is one step in a long iteration.

Infrastructure is code

You can reason about code in ways you can’t about a tarball or AMI.

Just as you don't have to think about capacitors to microwave your burrito, you don't have to think about the intermediate steps to know how you want your server to be.

Declare state,
not process

More precise. Less verbose.
The state of a server is easier to unambiguously describe.

Why manage configuration?

Talk about choosing the right tool for the right job. For example, scripting ./configure && make && make install in Puppet is a good sign you should build a package.

An interlude on
package management

Configuration management is the
centralized authority
to a package manager’s
local authority.

~/bin/doit5 may never work again. It's a good thing that the Puppet language is not itself Ruby but I'm not here to start a holy war. Limits in configuration management serve the same purpose as in a templating language: they enforce separation of concerns. In this case, the limits separate process from desired state.

Puppet, Chef,
and `~/bin/doit5`

Limits are good.
Puppet and Chef are idempotent by default.

Puppet and Chef work basically the same way from this altitude.

Architecture

Master knows best.
Agents phone home.
Hostname matching.
Resources.
Agents make it so.

Resources

Puppet and Chef again work the same way here. The single difference is in the granularity of dependency declarations. Puppet's are at the resource level. Chef's are at the cookbook (module) level.

The smallest unit of configuration.

package { "foo": ensure => "0.0.0" }

file { "/etc/foo.conf":
    content => template("foo.conf.erb"),
    owner   => "foo",
    mode    => "600",
    ensure  => file,
}

Be wary of using latest on packages you don't build in-house.

Classes and definitions

Compose resources in interesting ways.

class bar {
    exec { "apt-get update": }
    package { "bar":
        require => Exec["apt-get update"],
        ensure  => latest,
    }
    file { "/etc/bar.conf":
        content => template("bar.conf.erb"),
        ensure  => file,
    }
}

A basic configuration
for Puppet itself

The defaults are good except for a few places that break filesystem standards, which I choose to fix. pluginsync will be important later. You can customize the master and agents separately using [master] and [agent] sections, INI-style.

`/etc/puppet/puppet.conf`

[main]
    logdir=/var/log/puppet
    rundir=/var/run/puppet
    ssldir=$vardir/ssl
    pluginsync=true
    server=puppetmaster.example.com

This should exist on the master, too. There's a minor memory hit to running puppet agent as a daemon but that makes a huge difference to DevStructure users with sometimes just 256 MB of RAM. This cron calls bash, not sh to use $RANDOM which in effect prevents the thundering herd from taking down your Puppet master every half hour. This is a good time to point out the obvious - that not all your servers will always be the same - plan accordingly.

`/etc/cron.d/puppet`

PATH="/usr/sbin:/usr/bin:/sbin:/bin"

# Remove the line breaks.
*/30 * * * * root bash -c '
    sleep $(($RANDOM \% 1800));
    puppet agent --certname=$(cat
    /etc/puppet/certname)
    --no-daemonize --onetime'

(R)TFM

puppet --genconfig
http://bit.ly/puppet-config-ref

Hello, world!

Entire thing in /etc/puppet
Entrypoint is manifests/site.pp

Hello, `manifests/site.pp`!

import "nodes"

Exec {
    path => "/usr/sbin:/usr/bin:/sbin:/bin",
}

Node and class names may collide. "default" is special so I use "base" as my most general class name.

Hello, `manifests/nodes.pp`!

node default { include base }

node www inherits default { include www }

node 'staging.example.com' inherits www {}
node /\.www\.example\.com$/ inherits www {}

Hello, `base`!

`modules/base/manifests/init.pp`

class base {
    package {
        "dnsutils": ensure => latest;
        "psmisc": ensure => latest;
        "strace": ensure => latest;
        "sysstat": ensure => latest;
        "telnet": ensure => latest;
    }
}

Hello, `www`!

`modules/www/manifests/init.pp`

class www {
    package { "nginx":
        ensure => "0.7.65-1ubuntu2",
    }
}

Get yours

Security

SSL Cliff’s Notes

SSL handshake

Server certificate authenticates
server to client.
Clients may verify a server’s certificate against trusted certificate authority.
Client and server compute matching
secret keys.

Client certificates

Client certificate authenticates
client to server.
Not used by browsers.
Used by Puppet because you can’t lie here.

SSL in Puppet

/var/lib/puppet/ssl on agents.
puppet cert on master.
http://bit.ly/puppet-security-ref

A tour of secondary Puppet config files

`autosign.conf`

foo.example.com
*.www.example.com
*

A bad idea when there are untrusted clients.

`auth.conf`

path ~ ^/catalog/([^/]+)$
method find
allow $1

Safe by default. Easy to mistakenly open.
http://bit.ly/puppet-security-ref
http://bit.ly/puppet-rest-ref

`fileserver.conf`

[modulename]
    path /foo/bar/baz
    allow *

Usually certificates have to be signed.
http://bit.ly/puppet-fileserver-ref

Why to lie to Puppet

Gather compromising information about other servers

iptables rules.
Database passwords.
SSH private keys.
/etc/passwd entries.
AWS credentials.

How to lie to Puppet

Maybe they’re autosigning?

How to lie to Puppet

Sneak through regex `node` definitions.

[agent]
    certname=foo.www.example.com

How to lie to Puppet

Unmatched names fall back to `default`.

[agent]
    certname=adhadgsdhsfsdhxcb.example.com

How to lie to Puppet

`autosign` disabled? Try social engineering.

[agent]
    certname=admin0.example.com
    # certname=dev.example.com
    # certname=test.example.com
    # certname=corp.example.com

How to lie to Puppet

With a signed certificate, snoop for other catalogs.

export ssldir=/var/lib/puppet/ssl
export server=puppetmaster.example.com

curl --insecure \
    --cert $ssldir/certs/$CERTNAME.pem \
    --key $ssldir/private_keys/$CERTNAME.pem \
    --cacert $ssldir/ca/ca_crt.pem \
    https://$server:8140/$ENV/catalog/db1.example.com

If you do have a signed certificate, you can snoop around using it, too.

How to lie to Puppet

Snoop around an open file server,
no certificates needed.

export server=puppetmaster.example.com

curl --insecure \
    https://$server:8140/$ENV/file_content/$MODULE/$FILE

How to safely not care

Don’t autosign anything, ever.

How to safely not care

~~Don’t autosign anything, ever.~~
Understand and protect the network.
Remove special cases.

The network

Understand who can contact your
Puppet master.
Private networks may be shared.

`iptables`

iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

iptables -F

iptables -A INPUT -m conntrack \
    --ctstate RELATED,ESTABLISHED -j ACCEPT

iptables -A INPUT -i eth1 -p tcp \
    -s 10.47.0.0/16 --dport 8140 -j ACCEPT
iptables -A INPUT -i eth1 -p udp \
    -s 10.47.0.0/16 --dport 8140 -j ACCEPT

iptables -A INPUT -i lo -j ACCEPT

iptables -A INPUT -j DROP

`stunnel`

(Not specific to Puppet.)
If it’s on the Internet,
it’s either public or encrypted.
Example: use stunnel(8)
to make $NoSQL SSL-aware.

`stunnel` Upstart config

description "stunnel-redis-client"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /usr/bin/stunnel -f -c -d localhost:6379 \
    -r redis.example.com:6381

description "stunnel-redis-server"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /usr/bin/stunnel -f -d 6381 -r localhost:6382

Special cases
aren’t that special

Hostname-specific file paths can mitigate this risk. Leak example: malicious server grabbing the database config or secrets file.

Use templates

file { "/foo/bar/baz":
    source  => "puppet://foo/bar/baz",
    content => template("foo/bar/baz"),
    ensure  => file,
}

File and catalog serving use different ACLs.
Template content is included in the catalog so they inherit the catalog’s ACL.

Use only the `default` node

Mitigate the consequences of
successful certname speculation.
Use --config, --manifestdir, or --manifest to run different masters listening
on different interfaces/ports.

Extending Puppet

Maybe don’t

$extlookup_datadir = "/etc/puppet/extdata"
$extlookup_precedence = [
    "%{fqdn}", "%{domain}", "base"]

file { "/foo/bar/baz":
    content => extlookup("foobarbaz"),
    ensure  => file,
}

extlookup function can retrieve data from external CSV files.

Then again, maybe
extend Puppet

Where does
your code run?

External node classifier runs on master.
Plugins run on agents.

Example task

Authorize an SSH key unique to each host.
(This is by no means impossible in the Puppet language.)

External node classifier

Run your own code on the master.
Make decisions based on
more than just the hostname.

Configuring an external node classifier

[master]
    external_nodes=/usr/local/bin/classifier
    node_terminus=exec

Input

Hostname...
...which maps to facts.

/usr/local/bin/classifier foo.example.com

# Facts are available as YAML in
# $vardir/yaml/facts/foo.example.com.yaml

Facts?

Key value pairs describing
the server in question.

--- !ruby/object:Puppet::Node::Facts
  expiration: 2010-09-20 20:27:14.445807
  name: &id003 foo.example.com
  values:
    hardwaremodel: &id002 x86_64
    kernelrelease: 2.6.35.1-rscloud
    selinux: "false"
    sshrsakey: OH HAI

Lots more: facter | less

Output

---
classes:
  - base
  - www
environment: production
parameters:
  mail_server: mail.example.com

Classes, variables. No resources. YAML.

Example external node classifier

#!/bin/sh
set -e
TMP=$(mktemp -d "$1.XXXXXXXXXX")
ssh-keygen -q -f "$TMP/id_rsa" -b 2048 -N ""
zomg_post_to_the_api "$1" "$(cat "$TMP/id_rsa")"
cat <<EOF
---
classes:
  - ssh
parameters:
  public_key: $(cat "$TMP/id_rsa.pub")
EOF

Example Puppet class

class ssh {
    file {
        "/root/.ssh":
            mode   => "700",
            ensure => directory;
        "/root/.ssh/authorized_keys":
            content => "$public_key\n",
            ensure  => file;
    }
}

Puppet plugins

Because sometimes Puppet won’t let you.

The easy way

“It’s just Ruby.”

The orderly way

Gather the necessary data.
Build a Puppet catalog complete with dependency declarations.
Apply the catalog.

Plugin file structure

modules/ssh/
    manifests/
        init.pp
    lib/puppet/
        type/
            keygen.rb
        provider/keygen/
            posix.rb

Types versus providers

A package is a type.
apt is a provider of packages.
Types parse and normalize arguments.
Providers make it so.

Manifest

class ssh {
    keygen { "name-it-whatever": }
}

Type

require 'puppet/type'
Puppet::Type.newtype(:keygen) do
  @doc = "ssh-keygen example"
  newparam(:whatever, :namevar => true) do
    desc "Name it whatever."
  end
  ensurable do
    self.defaultvalues
    defaultto :present
  end
end

Provider

require 'openssl'
require 'puppet/resource'
require 'puppet/resource/catalog'

Puppet::Type.type(:keygen).
  provide(:posix) do

  desc "ssh-keygen example for POSIX"
  defaultfor :operatingsystem => :debian

  # Define exists?, create, and destroy.

end

  def exists?
    File.exists?(
      "/root/.ssh/authorized_keys")
  end

  def destroy
    Puppet.warning "No turning back."
    raise NotImplementedError
  end

  def create
    key = OpenSSL::PKey::RSA.generate(2048)
    zomg_post_to_the_api \
      Facter.value(:certname), key
    catalog = Puppet::Resource::Catalog.new
    catalog.create_resource(:file,
      :path => "/root/.ssh",
      :mode => "700",
      :ensure => :directory
    )
    catalog.create_resource(:file,
      :path => "/root/.ssh/authorized_keys",
      :content => "#{key.public_key}\n",
      :ensure => :file
    )
    catalog.apply
  end

Which is right?
Easy or orderly?

That’s an exercise for the reader.

Extending Puppet

(One more thing.)

Rack middleware

Puppet master is Rack application.
Rack allows pre- and post- processing.

`config.ru`

$0 = "master"
ARGV << "--rack"
ARGV << "--certname=#{
  File.read("/etc/puppet/certname").chomp}"

require 'puppet/application/master'

# TODO Middleware.

run Puppet::Application[:master].run

No-op middleware

require 'base64'
require 'json'
require 'rack/utils'
require 'yaml'
require 'zlib'

class StuckInTheMiddleWithYou

  def initialize(app)
    @app = app
  end

  def call(env)
    # TODO Preprocessing.
    status, headers, body = @app.call(env)
    # TODO Postprocessing.
    [status, headers, body]
  end

end

use StuckInTheMiddleWithYou

Preprocessing

    params = Rack::Utils.parse_query(env["QUERY_STRING"], "&")
    facts = case params["facts_format"]
    when "b64_zlib_yaml"
      YAML.load(Zlib::Inflate.inflate(Base64.decode64(
        Rack::Utils.unescape(params["facts"]))))
    end

    # Change facts.
    if Puppet::Node::Facts === facts
      facts.values["foo"] = "bar"
    end

    params["facts"] = case params["facts_format"]
    when "b64_zlib_yaml"
      Rack::Utils.escape(Base64.encode64(Zlib::Deflate.deflate(
        YAML.dump(facts), Zlib::BEST_COMPRESSION)))
    end if facts
    env["QUERY_STRING"] = Rack::Utils.build_query(params)
    env["REQUEST_URI"] =
      "#{env["PATH_INFO"]}?#{env["QUERY_STRING"]}"

Postprocessing

    object = case headers["Content-Type"]
    when /[\/-]pson$/ then JSON.parse(body.body.join)
    when /[\/-]yaml$/ then YAML.load(body.body.join)
    when "text/marshal" then Marshal.load(body.body.join)
    else body.body.join
    end

    # Change catalog.
    if Hash === object && "Catalog" == object["document_type"]
      object["data"]["resources"].unshift({
        "exported" => false,
        "title" => "apt-get update",
        "parameters" => {"path"=>"/usr/sbin:/usr/bin:/sbin:/bin"},
        "type" => "Exec",
      })
    end

    body = case headers["Content-Type"]
    when /[\/-]pson$/ then [JSON.generate(object)]
    when /[\/-]yaml$/ then [YAML.dump(object)]
    when "text/marshal" then [Marshal.dump(object)]
    else [object]
    end
    headers["Content-Length"] = Rack::Utils.bytesize(body.first)

Mention this is a last resort and that I've fixed bugs in Puppet itself via this method.

Rack middleware

Drink responsibly.
http://gist.github.com/602922

Thank you

richard@devstructure.com or @rcrowley
http://rcrowley.org/talks/strange-loop-2010
P.S. use DevStructure.

Securing and Extending Puppet for World Domination

Hi, I’m Richard Crowley

Configuration managementCliff’s Notes

Infrastructure is code

Declare state,not process

Why manage configuration?

An interlude onpackage management

Puppet, Chef,and ~/bin/doit5

Architecture

Resources

The smallest unit of configuration.

Classes and definitions

Compose resources in interesting ways.

A basic configurationfor Puppet itself

/etc/puppet/puppet.conf

/etc/cron.d/puppet

(R)TFM

Hello, world!

Hello, manifests/site.pp!

Hello, manifests/nodes.pp!

Hello, base!

modules/base/manifests/init.pp

Hello, www!

modules/www/manifests/init.pp

Get yours

Security

SSL Cliff’s Notes

SSL handshake

Client certificates

SSL in Puppet

A tour of secondary Puppet config files

autosign.conf

auth.conf

fileserver.conf

Why to lie to Puppet

Gather compromising information about other servers

How to lie to Puppet

How to lie to Puppet

How to lie to Puppet

Sneak through regex node definitions.

How to lie to Puppet

Unmatched names fall back to default.

How to lie to Puppet

autosign disabled? Try social engineering.

How to lie to Puppet

With a signed certificate, snoop for other catalogs.

How to lie to Puppet

Snoop around an open file server,no certificates needed.

How to safely not care

How to safely not care

The network

iptables

stunnel

stunnel Upstart config

Special casesaren’t that special

Use templates

Use only the default node

Extending Puppet

Maybe don’t

Then again, maybeextend Puppet

Where doesyour code run?

Example task

External node classifier

Configuring an external node classifier

Input

Facts?

Key value pairs describingthe server in question.

Output

Example external node classifier

Example Puppet class

Puppet plugins

The easy way

The orderly way

Plugin file structure

Types versus providers

Manifest

Type

Provider

Which is right?Easy or orderly?

Extending Puppet

Configuration management
Cliff’s Notes

Declare state,
not process

An interlude on
package management

Puppet, Chef,
and `~/bin/doit5`

A basic configuration
for Puppet itself

`/etc/puppet/puppet.conf`

`/etc/cron.d/puppet`

Hello, `manifests/site.pp`!

Hello, `manifests/nodes.pp`!

Hello, `base`!

`modules/base/manifests/init.pp`

Hello, `www`!

`modules/www/manifests/init.pp`

`autosign.conf`

`auth.conf`

`fileserver.conf`

Sneak through regex `node` definitions.

Unmatched names fall back to `default`.

`autosign` disabled? Try social engineering.

Snoop around an open file server,
no certificates needed.

`iptables`

`stunnel`

`stunnel` Upstart config

Special cases
aren’t that special

Use only the `default` node

Then again, maybe
extend Puppet

Where does
your code run?

Key value pairs describing
the server in question.

Which is right?
Easy or orderly?

`config.ru`