System Automation – Part 3 – Git

At this point you have to be asking your self if you should edit the puppet manifests on the puppetmaster, well the answer to that is “Heck No”. If you want your infrastructure to come to a screeching halt then by all means have at it but if you actually want to be able to track changes to your production systems and revert configuration changes then you should be leveraging a version control system. In the Linux world there are three main players in the version control world and they are cvs, subversion, and git. When I first started using git vs subversion I didn’t really like it because it much more complex then subversion, but now that I have been using it for a while I have really learned to love it.

Welcome To Git

So what exactly is Git? From the Git website “Git is a free & open source, distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Every Git clone is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server. Branching and merging are fast and easy to do.” Git was originally developed by Linus Torvalds for Linux kernel development.

There seems to be some confusion when I talk with people about github vs git. While github is based on git it is not git itself.

Installing A Git Server

I want to dive right into getting a git server setup, although you can start using github to get some hands on experience, unless you want to pay for an account or have your code accessible by everyone you probably want to setup a local git repository.

On an Ubuntu system there is only a single package that needs to be installed on both the git server and git client:

sudo apt-get install git-core

This will install all the necessary packages to get you up and running. Once you have the package installed its now time to setup the git user, this is the user that is going to actually own all of the repos and will also be the user that hands authentication to the repositories:

sudo useradd -m git

Now we want to setup ssh authentication directory structure for the git user:

sudo su - git
ssh-keygen -t rsa

This will create the directory .ssh with the right permissions (actually you don’t have to actually generate keys for this user you only need to create the .ssh directory but I figure might as well create keys in-case they are needed in the future).

Now we want to initialize our first repository, for the purpose of this automation guide, we are going to create a repo for puppet, since we want to be able to check out the puppet manifest and work on it on our local machine, and one day soon I’m going to write a blog post about how to unit test your puppet manifest with cucumber-puppet, jenkins, and vagrant so you defiantly want to make sure your manifest is checked into git (or some version control system):

sudo su - git
mkdir -p git/puppet.git
cd !$
git --bare init

This is going to initialize your first git repository.

Setting Up Your Development Machine

The first thing we need to do if you haven’t don it already is generate your ssh keys:

ssh-keygen -t rsa

This is going to create your ~/.ssh/id_rsa.pub and ~/.ssh/id_rsa files (your public and private keys) make sure that you set a password on your public key so other users can’t use it. Now we want to add our public key to the git user on the git server, since we didn’t actually set a password on the git user this is going to be a manual step:

scp ~/.ssh/id_rsa.pub git_server:/var/tmp
ssh git_server
sudo su - git
cat /var/tmp/id_rsa.pub >> ~/.ssh/authorized_keys

Make sure you can ssh git@git_server, if you followed the steps above you should be golden. Next make sure you have the git-core package installed on your Ubuntu development system:

sudo apt-get install git-core

Now we want to a basic setup of git, this will allow us to identify ourselves when we push code:

git config --global user.name "My Name"
git config --global user.email my.name@company.com

Now we want to check out our empty puppet git repository:

mkdir ~/git/
cd !$
git clone ssh://git@git_server/home/git/puppet.git

We will see a message that we have cloned an empty repository, that is ok for now because we are going to copy our manifest from the puppetmaster into the git repository in the next step.

Setting Up Your PuppetMaster

If you’ve been following this guide you have a puppetmaster already setup a basic module or two and your nodes definition. Well like I said before you really don’t want to be editing the manifest on the puppetmaster we want to checkout the manifest and edit the files on our own local machine.

Log into your puppetmaster and do the following:

sudo ssh-keygen -t rsa (make sure to set a password)
cd /etc/puppet
sudo git init
sudo git remote add origin ssh://git@git_server/home/git/puppet.git
git add *
git commit -m "initial commit" -a
git push origin master

Now you have your puppet manifest in source control and on your development machine you can simply do:

cd ~/git/puppet.git
git pull

Git Best Practices

While it sounds like a good idea to always commit to master (that was sarcasm if you couldn’t tell) it isn’t the best way to do things. Lets say you have a team of 3 people working on puppet manifests or even larger what if you allow your development team to actually create puppet modules (I know now I’m really talking some scary stuff, but don’t worry I’ll be blogging about how to use a unit test frame work to test your puppet modules and even leverage vagrant so you can quickly spin up a VM and test the manifest before you even commit changes to your branch).

Now that you have your working copy of master you are going to want to make a branch:

cd ~/git/puppet.git
git branch brianc
git checkout brianc

I used brianc as the name of my branch well because my name is Brian Carpio, so you probably want to name your branch something unique from other members of the team.

As you begin to work away on your puppet modules and you think they are ready to be shared with other members of the team you are going to want to do pull down changes, merge changes, fix any issues associated with your co-workers changes and your new puppet module then finally merge your changes back into master:

cd ~/git/puppet.git
git checkout master
git pull
git checkout brianc
git rebase master

If everything went well you should see something like this:

First, rewinding head to replay your work on top of it...
Fast-forwarded brianc to master.

If there was a problem you will get an error message and you need to fix it… so basically we have merged in other team members commits and now we want to merge our commits into master:

cd ~/git/puppet.git
git checkout master
git merge --no-ff brianc
git push

You’ll notice you want to use the –no-ff option so you can keep a list of the commits which took place on the master in the log. For this merge I simply deleted a file I created called temp_master, everything worked and I saw the following output:

Removing temp_master
Merge made by recursive.
 0 files changed, 0 insertions(+), 0 deletions(-)
 delete mode 100644 temp_master

Applying Updates To The Puppetmaster

So by simply pushing changes to the master branch doesn’t apply them to the puppetmaster. Before we actually pull down changes from the git master branch into the puppetmaster we want to make sure that we have tested the manifest, that is currently out of the scope of this post but stand by I am going to show you how you can run a test suite against the manifest and even leverage vagrant to spin up a virtual machine and apply the manifest before you ever pull down the changes to the puppetmaster.

Anyway, you will want to log into the puppetmaster and do the following to apply the changes you recently merged into the git master branch:

cd /etc/puppet
sudo git pull

This will pull down all of the changes. In my next blog post I will explain how to leverage Fabric as a remote administration / orchestration tool so you never have to actually ssh into any server to run day to day system administration tasks.