Be ready to scale

It makes sense to only use the resources you need to save on costs, and in my last post I talked about using only a single node for your entire stack, so for example your web server, web application and and database all running on one node.

Hopefully, sooner rather than later, your system will require to scale horizontally as you have exceeded all the resources available on a single node. To make the process as painless as possible you should use a configuration management tool that allows modularity. I personally prefer Puppet and it allows you to create modules with classes that can take parameters.

When setting up your single node you can create a module per piece of your system, so you could have a database module, and a web server module and a app server module etc.  You will probably also have modules for the software your system is build on, Nginx, PostgreSql for example. In order to distinguish the app stack modules I prefix them with the name of my system. So I will have

mysystem-db
mysystem-app
mysystem-web.

Each of these modules is essentially a class that can take parameters  such as database connection string's host names etc that then can be used in the module directly or via its template when generating configuration files. When you have one node most of the pertinent parameters will be pointing to the same host or localhost.  However once you want to scale out you can easily pass in the new database host for example.

Once you have these building block modules you can have different server roles that you puppet main site script can check and then apply the particular modules.  On my initial node I would have say a primary role that would just apply all the modules with parameters pointing to localhost.  Then one I want a separate database server for example I would have two roles, once webapp and one database. The webapp would apply the mysystem-app and mysystem-web and the database role would apply the mysystem-db.

The other advantage of having puppet being able to setup your single node from a vanilla OS is that you can recover from disaster much quicker than if you have set everything up manually.

You can see from my previous posts I prefer to use a puppetmasterless setup and have me push updates with git rather than the puppetmaster, handling it.  But this begs the question how does puppet know which node role to apply?  This is easily achieved using a few more advanced bits of puppet and a special module we build that discovers the role.  We seed the role with out bootstrap shell script which is passed as a parameter to it. I will reveal all in another post.

Feel free to leave comments or ask questions if anything it unclear.

Puppet with no strings attached

In my previous post I talked about configuring a single node on a virtual machine using Vagrant with a Shell provisioner. The shell script which is run on the first boot to a vanilla OS, will install puppet and git then clone your puppet repository onto the node and run apply on it.  I won't go into the ins and outs of Puppet as they cover it really well in there documentation.

The standard setup of puppet makes use of a puppet server which all your nodes (as puppet clients) connect to, however it requires a bit of setup, and has issues which are covered in this excellent blog post. The blog post explains how to set up puppet using an empty git repo which you then push to via git.  Git has hooks that you can run code at on certain events. A post receive hook on the node will run after you have pushed to it.  I do this setup of the empty repo and the post receive hook via my base puppet module. Below is an example of a puppet script, this is run as part of the initial puppet apply described in my last post.

class my-base {

    user { "git":
      ensure => "present",
      home => "/var/git",
    }

    file { "/etc/sudoers.d/100-git":
        owner => root,
            group => root,
            mode => 440,
        content => "git ALL=(ALL) NOPASSWD:ALL",
        require => User["git"]
    }

    file {
      "/var/git": ensure => directory, owner => git, 
      require => User["git"];
      "/var/git/.ssh": ensure => directory, owner => git, 
      require => [User["git"], File["/var/git"]];
      "/var/git/puppet": ensure => directory, owner => git, 
      require => [User["git"], File["/var/git"]];
    }


    ssh_authorized_key { "git":
      ensure => present,
      key => "YOURKEY",
      user => git,
      name => "git@yourdomain.com",
      target => "/var/git/.ssh/authorized_keys",
      type => rsa,
      require => File["/var/git/.ssh"],
    }

    package { "git":
      ensure => installed,
    }

    exec { "createPuppetRepo":
      cwd => "/var/git/puppet",
      user => "git",
      command => "/usr/bin/git init --bare",
      creates => "/var/git/puppet/HEAD",
      require => [File["/var/git/puppet"], Package["git"], User["git"]],
    }

    $hook_puppet = "#!/bin/sh
     git archive --format=tar HEAD | (cd /etc/puppet && sudo tar xf -)
    sudo sh -c \"puppet apply /etc/puppet/manifests/site.pp >> /var/log/puppet/puppet.log 2>&1\"
    "

    file { "/var/git/puppet/hooks/post-receive": 
        ensure => present,
        content => $hook_puppet,
        group => git,
        mode => 755,
        owner => git,
        require => Exec["createPuppetRepo"]
    }

}

After this is applied I can push puppet updates from my local puppet repo using the command below, which adds my vagrant vm node as a remote and then pushes my changes. You would also want to push the changes to your hosted repo so new nodes get the latest version.

dev-macbook # git remote add mynode ssh://git@mynode/var/git/puppet
dev-macbook # git push mynode master

Once this is received it will apply the new puppet config vi the hook we setup.  I also create similar push git repos for application code (Python service code), in the post receive for that I restart the Supervisor daemon which manages my service instances, I will cover Supervisor in a later post.

Now we have the mechanisms to manage a node using the two commands below.

dev-macbook # vagrant up
....Some time later after updating puppet......
dev-macbook # git push mynode master

This setup can be used in most places, including Amazon's EC2, where the initial shell script would be passed as the user-data, you could do this via a boto python script. Also Linode has stackscripts which would allow you to run this script to get the node provisioned ready to go.

You will probably quickly grow out of having one node, or one type of node, for example you may have a database node and a application server node, and a load balancer node etc.  You can still use the mechanism above to achieve this with just an addition to the Shell script and a special Puppet module.  I will cover this next time.

Vagrant, Puppet and git

Vagrant, Puppet and git are powerful tools on their own, used together and you can have a fully functioning configured virtual box or multiple boxes with one command, and you can push changes to boxes with another command.

This is the command to create a node,

dev-macbook # vagrant up

This launches a virtual machine based on a base box (i.e. Ubuntu 12) via virtual box, vagrant uses a VagrantFile to configure the box, things like port forwarding, host name etc, however I prefer to leave all the configuration to puppet (which we will see later).

One very useful thing that  the VagrantFile can configure is a provisioner, it has support for Puppet, Chef and good ole Shell provisioners.  What this boils down to is just after the machine is created (the first time it boots) it calls the provisioner. I use the Shell provisioner as I want to use the same  mechansim for bootstrapping a node whether its with vagrant or with a cloud provider, and shell scripts are the most widely supported.  Another reason is some cloud providers allow limited length of a boot-strap script.

The bash script does a couple of things, it installs git and puppet then it sets up a ssh key for root to enable a git clone of a puppet repository automatically.  Bitbucket is a good choice for hosting your repository as it allows free unlimited private repos, with git ssh access.  Once the puppet repo is cloned it calls puppet apply on the site.pp.  I based my script on this excellent blog post, the part under "The Code", it is focused on AWS ec2 but it also works for Vagrant.

Puppet then takes over and builds your node, and I get  puppet to do a few things that allow pushing puppet changes to the node (or many nodes).  I will cover this in my next post.