Feb 23 2015

Go Deployment

Category: Deployment,Golang,Organizationjgoulah @ 12:06 AM

Intro

In the spirit of the recent re-release of Deployinator, it seems timely to write a little bit about deployment. However, this is not a how-to on writing a Deployinator stack, but an overview of the high-level mechanics of getting a golang app deployed into production. Its worth mentioning that this methodology is just one of many ways you might go about deploying your app, but this is what has worked for me. These are the fundamental pieces that you’ll need to think through no matter what deploy tool you end up using.

Whats to Deploy?

When you run go build it produces a binary executable that can easily be shipped around. But there are actually several key pieces to getting this binary deployed into production. The code you have must be able to compile against the libraries you’re using, and if you are deploying a web based app, you’ll need to think about where your asset files are deployed and how they’re referenced from the go binary. The configuration of the application is also important in that you’ll want to be able to have the app run under different environments, such as development and production. Lastly, you’ll need to think about how to control the app, so that you can easily and consistently restart it when you deploy.

Building

Typically when doing production deployments there exists a build server where the application is compiled and the pieces are put in place for shipping to production. Certainly no production facing app should be getting deployed from a developers virtual machine, but the process for building the app should be identical. Building a go app is fairly straightforward but it can be a good idea to organize your steps and common commands in a Makefile. We’ll build our binary into the bin subdirectory:

build:
        go build -o ./bin/myapp myapp.go

This produces the binary that we’ll eventually ship.

Dependencies

This build process works fine in development but how do we know the external dependencies exist on our build machine? How can we maintain this app building process in an identical way across environments? Whereas in development we can just go get our dependencies, it may be haphazardness to maintain things this way on a production build machine, since you might have different go apps being built that use different versions of the same dependency. Or your developers may have pulled down different versions at different points in time. The current versions of github repositories that your code relies on also may introduce bugs that you want to gain confidence doesn’t impact your application. To maintain consistency in dependency versions, and to ensure these same dependencies exist in our build environment, we’ll use a tool called godep.

Assuming you have a proper go toolchain setup, you can get started with godep by getting it and running a single command:

go get github.com/tools/godep
# then in your project dir
godep save -r ./...

This should pull all of your dependencies into a Godeps directory as well as update your files to import this “frozen” copy of the dependencies you are using.

Since we won’t want to stay locked on old versions of software, we need a way to update our projects dependencies from time to time. This is something you’d probably want to add to your Makefile:

updep:
        go get -u -v -d $(import)
        godep update $(import)
        godep save -r ./...

Using this command we can update any of our dependencies like so:

make updep import=github.com/codahale/metrics

Configuration

A config file is a reasonable way to maintain different environments such as development and production. The gcfg library is a perfectly fine way to define ini style configs that map easily to go structs. It will support multiple environments and you can have a file defining your shared values and then define development and production differences in another set of files. Then you can merge the appropriate set of files based on your environment using ReadFileInto. The example below merges a secrets configuration file with the base configuration to show a simple example, but you can extend this technique to support multiple environments.

In this example we make sure the base config exists, based on a location relative to this file. We’re next looking for the secrets config, first in the app directory where we may store it locally for development, and then in an /etc directory for production where a configuration management system may place the file more securely. This way this sensitive file doesn’t have to be stored in the main repository, but can pass production values securely to the application. Note that means this file wouldn’t be deployed in the normal way, and the app will fallback to the secure values in /etc when it doesn’t find the local file. The full configuration example implementation can be found here but remember this is just a very basic example to show how to merge configs.

Initialization

When you deploy the app to production you’ll have to restart it in some way. The simplest way to do this is to write an init script. But you shouldn’t have to write an init script by hand these days. Check out pleaserun to generate a script for your system. Now you can stop, start, and restart your process whether you are using upstart, systemd, plain old init, or something else. You can go one step further and implement a graceful restart, but the implementation will vary depending on your needs.

Tying it Together

So now we have a way to deal with our dependencies and the build. We know how to control the app. The app knows how to find its configuration files and override base values with production settings. We can use the same runtime.Caller trick that we used to locate the config file to make it easier for the application to find its static files as well.

At this point there are only a few steps that need to happen on a deploy box:

  • Get the newest code
  • git fetch && git reset --hard
  • Build the binary
  • cd $basedir && make build
  • Rsync the code
  • rsync -va --delete --force \
    --include='static' --include='bin' --include='config*.cfg' --include='*/*' --exclude='*' \
    deploy.mydomain.com:/var/myorg/build/golang/src/github.mydomain.com/jgoulah/myapp/ /usr/myorg/myapp
  • Use your init script to restart the app, depending on the system
  • /etc/init.d/myapp restart

In this example the rsync command is run on an application box and pulling from our deploy host. In a large deploy we may dsh out to many app hosts to rsync our build over in a pull based fashion. The /var/myorg/build directory is where our make build command was run to produce our binaries. We sync the bin directory that contains the binaries, the static directory which has our html/js/css, and the config files. Lastly we restart our app. These steps can be coded into a Deployinator stack or with the deploy tool of your choice. But regardless of the tool we use, we will have a repeatable and dependable process with reliable dependencies that we can ship.

Tags: , , , , ,


Jul 20 2012

Development is Production Too

Category: Conferencesjgoulah @ 8:56 PM

I was at OSCON this week, and my friend Erik Kastner and I did a talk about development environments. Specifically what to avoid and how to keep environments consistent across development and production. As usual the slides are not fully explanatory without seeing the accompanying talk but here they are anyway:

Tags: , , , , ,


Sep 07 2009

Deploying A Catalyst App on Private Hosting

Category: Deploymentjgoulah @ 1:26 PM

Intro

In the last post I wrote about deploying Catalyst on shared hosting.  While shared hosting may seem attractive pricewise, you’ll quickly grow out of it, and it makes sense to move to hosting with some more control if you are serious about your website.   There are lots of choices when it comes to where you may want to host,  and if you are looking for virtual options, Slicehost, Linode, prgmr, or even Amazon EC2 are great choices.  Picking and setting these up are well beyond the scope of this article,  so I’m assuming you have a server with root privileges ready to go.

Setting Up Your Modules

As I probably sound like a broken record if you’ve read any of my other perl-related entries,  its a good idea to setup your modules using local::lib.  Check out the last article on how to set it up,  or the module docs do a fine job if you follow the bootstrap instructions.  I always create a new user to set this up as,  such as ‘perluser’  or similar,  this way your modules aren’t affected by upgrades as your own user.  We’ll show how to point to these modules shortly.

For now, you’ll want to also make sure  you have FCGI and FCGI::ProcManager installed as that user that you just setup along with the rest of your apps modules.

$ cpan FCGI
$ cpan FCGI::ProcManager

You may want to checkout your app under this user, so that you can just do

$ perl Makefile.PL
$ make installdeps

assuming you’ve kept your Makefile.PL updated, this should be all of the perl modules you need to run your application.

Setting Up Apache

You can setup apache using your package manager of choice.  For ease of this article I’ll assume you’re on Ubuntu, or Debian based system, and you can do something like

$ sudo apt-get install apache2.2-common
$ sudo apt-get install apache2-mpm-worker

And you also need the mod_fastcgi module.  You could download and compile it.  Or you could grab it from apt.   You’ll probably have to add the multiverse repository,  so update your /etc/apt/sources.listso that each line will end with

main restricted universe multiverse

Now you can

$ sudo apt-get update
$ sudo apt-get install libapache2-mod-fastcgi

Setup the VirtualHost and App Directory Structure

You need to put a VirtualHost so that Apache knows how to handle the requests to your domain

FastCgiExternalServer /opt/mysite.com/fake/prod -socket /opt/mysite.com/run/prod.socket

<VirtualHost *:80>
ServerName www.mysite.com

DocumentRoot /opt/mysite.com/app
Alias /static /opt/mysite.com/app/root/static/
Alias / /opt/mysite.com/fake/prod/
</VirtualHost>

This “/” alias ties your document root to the listening socket defined in the FastCgiExternalServer line. The “/static” alias make sure your static files are served by apache, instead of fastcgi.

This config also assumes some directory structure is setup,  which is really entirely up to you.  But here we’ll assume you have a directory located at /opt/mysite.com with a few directories under that called fake, run, and app.

$ sudo mkdir -p /opt/mysite.com/{fake,run,app}

The only directory you have to put anything in is app, which should contain your code.

Note, this is a very simplified layout.  In the real world I’d put the fake, run, and app dirs under a versioned directory,  which my active virtualhost would then point to.  I’ve talked briefly about this kind of deployment technique before at a high level and there is a great writeup here on the technical details on how to use multiple fastcgi servers to host your staging and production apps with seamless deployment.  Part of the beauty of using FastCGI is that you can run two copies of the app against the same socket so its easy to bring up instances pointing to different versions of your code, and deploy with zero downtime.

Launching FastCGI

The last piece of the puzzle is to have a launch script,  which makes sure that your app is listening on the socket.  So to keep it simple you would have a script called launch_mysite.sh that looks like this

#!/bin/sh

export PERL5OPT='-Mlib=/home/perl_module_user/perl5/lib/perl5'

/opt/mysite.com/app/script/mysite_fastcgi.pl \
-l /opt/mysite.com/run/prod.socket -e \
-p  /opt/mysite.com/run/prod.pid -n 15

The first line is telling the script to use the modules from the user we setup the local::lib to hold the modules, so make sure you change this to the correct location.  Then it starts up fastcgi to listen on your socket, and create a process pid,  and to spawn in this case 15 processes to handle requests. Go ahead and hit your domain, and it should show you your website.

Conclusion

We’ve gone over the basics on how to setup FastCGI using FastCgiExternalServer. You now have a lot of flexibility in how many processes are handling requests, the ability to run different copies of your app and flipping the switch, and pointing to which modules are run with your app. There are a lot of improvements you can now make to setup a very sane deployment process so that each version of code deployed can be its own standalone build and ensuring your production app has 100% uptime, but at this point its up to your imagination.

Tags: , , , , ,


Aug 29 2009

Deploying a Catalyst App on Shared Hosting

Category: Deploymentjgoulah @ 1:48 PM

Intro

People have long complained that one of the tricky things about perl is the deployment phase, much because of the intricacies of mod_perl and its unfriendliness towards shared environments. In fact, I would highly recommend FastCGI over mod_perl since it is quite easy to understand and configure. This post is going to focus on smaller shared hosting environments, and how easy it can be to quickly deploy a Catalyst web app.

Getting Started

Assumptions

There are some basic assumptions here. First we need a Linux webserver that has Apache installed and is loading up
fcgid. I believe you can also use the favored mod_fastcgi which I just pointed to above, but I have yet to test this on a shared host. These are binary compatible modules so in theory both work. But again I’ve only used mod_fastcgi for large non-shared hosted deployments. You’ll also need mod_rewrite which is now fairly common.

Installing Your Modules

The best way to install the modules your application depends on is using local::lib. I’ve talked about this before so there isn’t a lot of need to go over the process in detail again, but in a nutshell you can do

$ wget http://search.cpan.org/CPAN/authors/id/A/AP/APEIRON/local-lib-1.004006.tar.gz
$ tar xzvf local-lib-1.004006.tar.gz
$ cd local-lib-1.004006
$ perl Makefile.PL --bootstrap
$ make test && make install
$ echo 'eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)' >>~/.bashrc
$ source ~/.bashrc

Now you have an environment that you can install your modules into. By default this is localized to ~/perl5. The next step is to install your modules that the application requires. It is good practice to put these into your Makefile.PL so that you can easily install them in one shot. A very basic one would follow this template

use inc::Module::Install;

name 'MyApp';
all_from 'lib/MyApp.pm';

requires 'Catalyst::Runtime' => '5.80011';
requires 'Config::General';
# require other modules here

install_script glob('script/*.pl');
auto_install;
WriteAll;

Now its easy to do

$ export PERL_MM_USE_DEFAULT=1
$ perl Makefile.PL
$ make installdeps

The PERL_MM_USE_DEFAULT will configure things such that you don’t have to press enter at every question about a dependency. The make installdeps will install any missing modules, which in this case is going to be everything. You can upgrade the version numbers in the Makefile.PL “requires” lines if you want installdeps to grab the newer distributions as they are released to CPAN.

Configuring Your App

First thing we have to do is a minor edit to our fastcgi script, which is to tell it to use our local::lib. Since its not part of the environment we setup earlier in .bashrc we have to tell the fastcgi perl script where to find things. Below the “use warnings;” line add this

use lib "/home/myuser/perl5/lib/perl5";
use local::lib;

Make sure to change the path to the correct location of your perl5 modules directory.

The last thing is to make sure your app is located in the public directory root for your host. In my case I created a symbolic link from the public_html folder to my app.

$ cd && mv public_html public_html.old  # get rid of current root folder
$ ln -s ~/myapp ~/public_html

Then create a .htaccess file in that folder, which should reside beside all of your code

Options ExecCGI FollowSymLinks
Order allow,deny
Allow from all
AddHandler fcgid-script .pl

RewriteEngine On
RewriteCond %{REQUEST_URI} !^/?script/myapp_fastcgi.pl
RewriteRule ^(.*)$ script/myapp_fastcgi.pl/$1 [PT,L]
RewriteRule ^static/(.*)$ static/$1 [PT,L]

We’re just telling apache to turn CGI on, and make sure to execute our fastcgi perl script. Be sure to change the rewrite lines to point to your script (hint: change myapp to your app name).

Now, you should be able to hit your domain. Simple!

Conclusion

So we’ve seen that deploying perl can actually be fairly easy. There are of course some assumptions here, for example, to get the rewrite rules working you’ll need mod_rewrite.so, but this is fairly standard these days. Now you can deploy a perl app with the same ease as languages such as PHP, where it is pretty much plug and play. This should enable people to more easily compete with all the badly written blog, forum, and other generically useful software in the open source world.

Tags: , , ,


Mar 14 2009

Code Deployment Techniques

Category: Deploymentjgoulah @ 7:09 PM

Introduction

Getting code out to your target production machines is a bit of an art form that many seem to take for granted. A lot of shops that I talk to seem to think of the deployment process as an svn up on each node. Even if this works for you, it’s really not a great way to deploy code. The main reason is that it does not scale well. You lend yourself to having different versions of code on different machines at the same time. For example, if your load balancing algorithm made the logs on one machine fill up faster than another and you run out of disk space during the svn update on a node. The correct process should be such that different releases of code can by deployed nearly instantaneously, with old copies left around for rollback. If you just svn up your code has the potential to be out of sync a lot more often than if you deploy code with an rsync and provide your webserver symbolic links to the current version of code. This article will show how this process can work in a scalable way.

Like most of my articles, this is targeted towards shops that are running something along the lines of your typical LAMP stack. I’ve setup this deployment architecture under a PHP setup with Apache, and also a slightly more complicated Perl setup running ModFastCGI instances behind Apache. In one of my most complex deployments I’ve had the database updates go out simultaneously along with the code that worked with those changes, so there is a lot of flexibility in what you can do. I’ll be talking at a very high level on how to accomplish this.

Code Environments

Deployment is the process of getting the code from your development environment into a working production system. There are usually at least three environments: development, staging, and production. Some places may have a demo or test environment for their own specific purpose but we will focus on the main three. Your development environment is active when code is being written and tested by the developers. Typically the development environment points at a ‘dev’ database or if you are using something like MySQL Sandbox perhaps each developer has his own database to work against. Your staging environment should be as close of a replica to your production environment as possible.  I typically deploy staging on the exact same servers as production, but point it to a copy of the production database instead of the real thing.  Its ideal for this copy to be taken nightly with something like MySQL LVM snapshotting so you’re working on similar data.  When staging code is assured to be “bug free” it is then pushed to your live production environement.

Its a good idea to have your code respect these environments. The initialization for your application should be able to infer the environment and pull the appropriate config based on that so that you are pointing to the right DB and any other specifics for that particular environment. One way to do this is to use an environment variable and define that inside of your virtual host file. Since you’ll have a separate virtual host for staging and production your application can then look at the variable and know what configuration to grab.

The Build Process

To get your code from development to staging you have to undergo some form of a build process. Some build processes are more complicated than others. A simple build process may just be as trivial as doing an svn export on your repository to the location you wish to push that code from. More complex builds may want to build and bundle external libraries that the application needs to work. For example in a Perl deployment I’ve compiled the required modules on the build step of each release so that the build consists of the same version of modules that the code was written against. This was really important since CPAN is ever changing, otherwise its possible in a rollback situation that there would be incompatibilities. For the purposes of simplicity I’ll just assume you are deploying code that has everything setup and working on the server (such as PHP usually is).

The build should also be responsible for putting the code into a versioned directory. There are a few benefits here. One is that you can correspond a build of code to the repository revision that it correlates to. Second is that it makes rollback a cinch because multiple versions of the code can exist on the server at the same time. Apache knows which version is active because the virtual host for the active version of code is pointed to by a symbolic link that gets modified as part of the deploy procedure.

Generating a Version

I’ve explained that your build script should generate a version and it makes sense for this to come from your repository revision that is getting deployed. Here’s a short bash function that can generate a version from an SVN repository (also works with SVK which is an SVN wrapper)

function get_version {
  if [ -e .svn ]; then
    eval $1=$(svn info | grep ^Revision | perl -ne "/Revision: (\d+)/; printf('1.%06i', \$1);")
  else
    eval $1=$(svk info | grep "Mirrored From" | perl -ne "/Rev. (\d+)/; printf('1.%06i', \$1);")
  fi
}

and now the build script can do something like

cd /path/to/svn/checkout
svn up
get_version version
echo "this is the version: $version"

Exporting the Code

Now that you’ve generated a version you should export that version of code into the directory you are going to push from. For this example we’ll say that we store versions in /opt/test.com so the structure might look something like:

/opt/test.com
     {version1}
     {version2}
     {versionN}
     ...

Then the code would live under version1, version2, … versionN for each version of code deployed.

One way to do this is something like:

mkdir /opt/test.com/$version
svn export -q svn://${svn_host}/trunk ${site_dir}/${version}

Informing Apache of the Document Root

Apache needs to know where the new version of code is. Since we’ve decided to put it into an ever changing directory name (the build version) there are at least two methods for letting Apache know the document root of the code. One way is to dynamically generate a virtual host with the explicit versioned path from above (eg. /opt/test.com/0.00010). The document root here will always point to the version that its bundled with, and then on deployment, a symlink from apache’s conf.d directory is pointed to the active virtual host config file. Another way to do this is to have a more static virtual host file, in which the document root path points to a symlinked directory to the version which changes on deploy (eg. /opt/test.com/current-www such that current-www is a symlink to the appropriate version).

The Deploy Process

In the end the point of deployment is to get your code into production. The version of code should first be rolled into staging, and then when verified as good, into production. The beauty of this is that all that it takes to move code from staging to production is an update to a symbolic link. The current production symlink is wiped out, and pointed to the new version of code, the same one that we deployed for staging.

The deployment should at a minimum

  • rsync code to each web server
  • point the symbolic links to the new version of code
  • restart apache

The great thing is that the symbolic links are not repointed until all of the code already exists on each server (the rsync to all servers should happen first).   This gives a deploy that doesn’t leave servers out of sync with each other.  The deploy is instantaneous and verified working without trying to rsync or svn up over top of running production code.

Conclusion

I’ve gone over a robust way to do code deployments that can be fully automated with a little bit of scripting.  Its an easy way to get code out to production and also to ensure that the exact version of code that currently exists in staging is the version of code that goes out.   It gives us a very easy way to rollback if a major problem does happen to be found in production and to quickly see what version is currently deployed.  And it allows us to change versions of code simultaneously without the end user ever knowing.

Tags: ,