Puppet Relationships & App Deployment

I wanted to write a blog post about something that I assume other people might be struggling with when deploying applications to EC2. Specifically how do I tell my deployment scripts (fabric in my case) to wait until puppet has finished before moving on to actually deploying my application.

Simply Sleep

Initially when I was tasked with writing single command application deployment tasks for EC2 I had to come up with a simple way to create an EC2 instance, have puppet apply its rules, and then deploy the actual code to the new instance. Simple right? Not exactly.

Basically our deployment process looked like this:

EC2 API: Create Instance & Run User-Data Script
Sleep: 300
Fabric: Deploy Application

Its actually much more complex with that since we leverage a Puppet ENC and Route 53 but for the point of this blog our deployment process was as simple as that. To be honest it worked 90% of the time. Sleeping 300 seconds proved to be long enough for Amazon to create the instance and for the puppet agent to apply the puppet manifest; however the 10% was really annoying to say the least.

Puppet Relationship Syntax

One of the first issues that plagued me long before EC2 was with packages and how puppet didn’t always run apt-get update before trying to install packages. Well this can cause problems because if puppet had put in a new /etc/apt/sources.list then apt won’t know where to go find packages. So first I setup the following rules using the new relationship syntax.

Apt::Key <| |> -> Exec["apt-update"]
Apt::Source <| |> -> Exec["apt-update"]
Exec["/usr/bin/apt-get update || true"] -> Package <| |>

This solved nearly all of my “first run” problems as I like to call them. So puppet would make sure all apt dependencies where applied before running apt-get update, and would wait until apt-get update was run before trying to install any packages.

Next, I needed to figure out a way to put in a file which my deployment could “look” for before trying to run the next execute statement, which in my case is to install and start the nodejs application. So I simply created the following file definition in my nodejs puppet module:

    file { "/home/appuser/finished":
        force => true,
        ensure => present,
        content => "",
        owner => "appuser",
        group => "appuser",
        require => Class["nodejs::install"]
    }

Next, I added a relationship that told puppet to wait to put down the /home/appuser/finished file until after all packages had been installed:

Package <| |> -> File["/home/appuser/finished"]

Finally I had to tell fabric to wait for this file to exist before running the next execute statement so I created the following fabric task:

@task
def get_aws_deployment_status():
    with settings(
        hide('running', 'stdout')
    ):
        env.warn_only = True
        status = sudo('ls -al /home/appuser/finished > /dev/null 2>&1; echo $?')
        return status

And then called that task inside of my application deployment task waiting for the “status” to be 0:

    status = 1
    runs = 0
    while status != 0:
        if runs > 60:
            print (red("PROBLEM: Deployment Failed"))
            sys.exit(1)
        status = execute(get_aws_deployment_status hosts=ip_returned_from_aws_api)
        status = status[ip_returned_from_aws_api]
        time.sleep(10)
        runs += 1
    execute(app.deploy, hosts=ip_returned_from_aws_api)

There might be a more elegant way to do this but it works perfectly fine for my purposes. Fabric first deploys the EC2 instance, waits for puppet to put down /home/appuser/finished file, and then executes the application specific deployment task. So far this has prevented 100% of our application deployment failures. I’m still learning a lot about how to complete full environment deployments in EC2 and hopefully one day I can get to http://puppetlabs.com/blog/rapid-scaling-with-auto-generated-amis-using-puppet/ until then this is working out quite well for us.