System Automation – Part 2 – Puppet

So now we have our systems up and running with Ubuntu 10.04 and next you want to make sure that some common configuration files are identical across your infrastructure. You also want to have specific components installed on specific servers, for instance you want your databases installed on your database servers, you want your applications installed on your app servers, and you want your web servers install on your web servers and you want it all consistent and automated, sounds crazy right?

Installing The Puppetmaster

Again, for this tutorial we assume a default install of Ubuntu 10.04 Lucid. The first thing we are going to want to do is install the debian packages from the puppetlabs repo, they are the latest and have the most features so add the following line to your /etc/apt/sources.list file:

sudo echo "deb http://apt.puppetlabs.com/ubuntu lucid main" >> /etc/apt/sources.list

Once this is added in Ubuntu you need to update your apt-cache:

sudo apt-get update

Next we are going to install the puppetlabs debian packages:

sudo apt-get install puppet puppetmaster

This installs the debian packages to get the core of puppet running; however we want to make sure we enable passenger because by default puppet runs WEBrick which is only suitable for small deployments of 10 servers so we want to make puppet use passenger and apache:

First we need to add the following lines to our /etc/puppet/puppet.conf file on the puppetmaster:

[puppetmasterd]
  ssl_client_header = SSL_CLIENT_S_DN
  ssl_client_verify_header = SSL_CLIENT_VERIFY

Then we need to install the passenger related packages, the great thing about Ubuntu is you don’t need to install any gems for this to work. One thing to note is that you will get an error when puppetmaster-passenger tries to install because it will attempt to start apache which will try and bind to port 8140 since WEBrick is already running on that port the installation will fail. So make sure to stop the puppetmaster service:

sudo /etc/init.d/puppetmaster stop

Disable it from starting in the future:

sudo sed -i s/yes/no/g /etc/default/puppetmaster

Now install packages:

sudo apt-get install apache2 libapache2-mod-passenger rails librack-ruby libmysql-ruby puppetmaster-passenger

You should now have a puppetmaster with passenger enabled for high performance. Finally I had to make a symbolic link in /etc/puppet/ssl to /var/lib/puppet/ssl for everything to work properly, I am investigating why this happened after I installed the puppetmaster-passenger package.

Puppet Manifest Directory Structure

Now that we have a puppetmaster we see that we have some files and directories created in /etc/puppet first we want to create a few files in the /etc/puppet/manifests directory the first is going to be the site.pp file:

# /etc/puppet/manifests/site.pp

import "nodes/*"

# The filebucket option allows for file backups to the server
filebucket { main: server => 'puppet' }

# Set global defaults - including backing up all files to the main filebucket and adds a global path
File { backup => main }
Exec { path => "/usr/bin:/usr/sbin/:/bin:/sbin" }

The main thing to note in this file is that we are going to import a directory called nodes and all of the .pp files in the nodes directory. Some how-to guides either define all of their nodes in the site.pp file or create a single nodes.pp file, this isn’t a very smart way to do things especially if you have a very large environment, the best approach is to decide upon a system by which to categorize your nodes. For instance if you run a single puppet manifest for dev, qa, staging, and prod you might want to categorize your nodes by environment if you have a large production environment it might make more sense to categorize your nodes by “type” for instance mongodb.pp would contain the host definitions for your mongodb servers, and hadoop.pp will contain the host definitions for your hadoop servers.

Obviously we want to create the nodes directory:

sudo mkdir /etc/puppet/manifests/nodes

Now we want to create an initial base definition this will include basic modules all of your Linux servers need such as sudo, resolv.conf etc. For this example we will call this /etc/puppet/manifests/nodes/default.pp and include a single module (which we haven’t created yet) called ntpd:

node "default" {
        include "ntpd"
}

Finally, lets create a simple node file for our mongodb servers, create a new file /etc/puppet/manifests/nodes/mongodb.pp and put in the following:

node /.*mongod.*/ inherits default {

}

This would make sure all of the mongod servers inherit the default puppet modules and allows you to define other modules (such as my mongodb puppet module) for the mongod hosts.

First Puppet Module

First I want to talk about puppet module directory structure, the basic structure looks like:

/etc/puppet/modules
                  ~/ntpd
                       ~/files
                       ~/manifests
                       ~/lib
                       ~/templates

For the sake of this blog I am not going to cover the directory structure in depth for our sample module we are simply going to use files, manifests, and template.

Next we are going to create our first puppet module. There are many ways of creating puppet modules you can have a single init.pp with a single class, you can have an init.pp with 10 classes, you can have an init.pp which simply calls classes defined in other class.pp files. Personally I like to make my puppet modules very simple for other people to use so except for extremely complicated modules I like to have an init.pp and a params.pp the init.pp does the heavy lifting while the params.pp is a simple file which other people can edit to fit their environment.

Lets start with the the /etc/puppet/modules/ntp/manifests/init.pp for our ntp module. We are going to break down our init.pp into three phases install, config and service.

class ntp::install {
    package{"ntp":
        ensure => latest
    }
}

class ntp::config {
    File{
        require => Class["ntp::install"],
        notify  => Class["ntp::service"],
        owner   => "root",
        group   => "root",
        mode    => 644
    }
 
    file{"/etc/ntp.conf":
            content => "template("ntp/ntp.conf");
    }
}
 
class ntp::service {
    service{"ntp":
        ensure  => running,
        enable  => true,
        require => Class["ntp::config"],
    }
}
 
class ntp {
    include ntp::install, ntp::config, ntp::service, ntp::params
}

As you can see we are leveraging the template system of puppet for the ntp.conf file, we are also setting a “run order” so to speak where basically the application gets installed, then the configuration is applied and finally the service is started.

Next we want to make an /etc/puppet/modules/ntp/manifests/params.pp file this file is actually only going to have one single variable and that is going to be the ntp servers.

class ntp::params {

     $ntpservers = $hostname ? {
         default   => [ntp1.internal.com, ntp2.internal.com, ntp3.internal.com]
     }
}

This allows us to define a list of internal NTP servers, which will get pulled into our ntp.conf template in the next step.

So lets look at the /etc/puppet/modules/ntp/templates/ntp.conf.erb file, this file is going to serve as the template for our ntp.conf file.

driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
<% scope.lookupvar('ntp::params::ntpservers').each do |var| -%>
server <%= var %>
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1

In our template we have a simple loop which will add a server line for each of the hosts we have defined in our params.pp file.

I will write a more advanced puppet module creation tutorial later but this should get you off and running. You might also want to check out Puppet Forge its a great place to learn how other people are writing puppet modules most (if not all) are hosted on github so you can branch them and then make necessary changes for your environment.

Setting Up The Puppet Clients

Now that we have our puppetmaster all setup we want to make sure that we setup our servers to speak with the puppetmaster, again we are going to want to add the puppetlabs Ubuntu repository to our /etc/apt/sources.list file on the client machine:

sudo echo "deb http://apt.puppetlabs.com/ubuntu lucid main" >> /etc/apt/sources.list

Next we want to install the puppet agent on the client machine:

sudo apt-get install puppet

And now we want to setup the initial communication between the puppetmaster and client, so on the client run:

sudo puppetd --test --waitforcert 60 --server puppetmaster.example.com

This is going to generate an SSL certificate and ask the puppetmaster to sign that certificate, that way we can be sure that the clients are authorized to communicate with the puppetmaster and we can ensure the communication is encrypted, –waitforcert 60 tells the puppet agent to wait 60 seconds for the puppetmaster to sign the SSL request. So on the puppetmaster we are going to want to sign that certificate:

sudo puppetca --sign client.example.com

That’s all there is too it, now your client is actively checking into the puppetmaster applying configuration changes and maintaining state.