Jul 20 2012
I was at OSCON this week, and my friend Erik Kastner and I did a talk about development environments. Specifically what to avoid and how to keep environments consistent across development and production. As usual the slides are not fully explanatory without seeing the accompanying talk but here they are anyway:
Sep 07 2009
In the last post I wrote about deploying Catalyst on shared hosting. While shared hosting may seem attractive pricewise, you’ll quickly grow out of it, and it makes sense to move to hosting with some more control if you are serious about your website. There are lots of choices when it comes to where you may want to host, and if you are looking for virtual options, Slicehost, Linode, prgmr, or even Amazon EC2 are great choices. Picking and setting these up are well beyond the scope of this article, so I’m assuming you have a server with root privileges ready to go.
As I probably sound like a broken record if you’ve read any of my other perl-related entries, its a good idea to setup your modules using local::lib. Check out the last article on how to set it up, or the module docs do a fine job if you follow the bootstrap instructions. I always create a new user to set this up as, such as ‘perluser’ or similar, this way your modules aren’t affected by upgrades as your own user. We’ll show how to point to these modules shortly.
For now, you’ll want to also make sure you have FCGI and FCGI::ProcManager installed as that user that you just setup along with the rest of your apps modules.
$ cpan FCGI $ cpan FCGI::ProcManager
You may want to checkout your app under this user, so that you can just do
$ perl Makefile.PL $ make installdeps
assuming you’ve kept your Makefile.PL updated, this should be all of the perl modules you need to run your application.
You can setup apache using your package manager of choice. For ease of this article I’ll assume you’re on Ubuntu, or Debian based system, and you can do something like
$ sudo apt-get install apache2.2-common $ sudo apt-get install apache2-mpm-worker
And you also need the mod_fastcgi module. You could download and compile it. Or you could grab it from apt. You’ll probably have to add the multiverse repository, so update your /etc/apt/sources.listso that each line will end with
main restricted universe multiverse
Now you can
$ sudo apt-get update $ sudo apt-get install libapache2-mod-fastcgi
You need to put a VirtualHost so that Apache knows how to handle the requests to your domain
FastCgiExternalServer /opt/mysite.com/fake/prod -socket /opt/mysite.com/run/prod.socket <VirtualHost *:80> ServerName www.mysite.com DocumentRoot /opt/mysite.com/app Alias /static /opt/mysite.com/app/root/static/ Alias / /opt/mysite.com/fake/prod/ </VirtualHost>
This “/” alias ties your document root to the listening socket defined in the FastCgiExternalServer line. The “/static” alias make sure your static files are served by apache, instead of fastcgi.
This config also assumes some directory structure is setup, which is really entirely up to you. But here we’ll assume you have a directory located at /opt/mysite.com with a few directories under that called fake, run, and app.
$ sudo mkdir -p /opt/mysite.com/{fake,run,app}
The only directory you have to put anything in is app, which should contain your code.
Note, this is a very simplified layout. In the real world I’d put the fake, run, and app dirs under a versioned directory, which my active virtualhost would then point to. I’ve talked briefly about this kind of deployment technique before at a high level and there is a great writeup here on the technical details on how to use multiple fastcgi servers to host your staging and production apps with seamless deployment. Part of the beauty of using FastCGI is that you can run two copies of the app against the same socket so its easy to bring up instances pointing to different versions of your code, and deploy with zero downtime.
The last piece of the puzzle is to have a launch script, which makes sure that your app is listening on the socket. So to keep it simple you would have a script called launch_mysite.sh that looks like this
#!/bin/sh export PERL5OPT='-Mlib=/home/perl_module_user/perl5/lib/perl5' /opt/mysite.com/app/script/mysite_fastcgi.pl \ -l /opt/mysite.com/run/prod.socket -e \ -p /opt/mysite.com/run/prod.pid -n 15
The first line is telling the script to use the modules from the user we setup the local::lib to hold the modules, so make sure you change this to the correct location. Then it starts up fastcgi to listen on your socket, and create a process pid, and to spawn in this case 15 processes to handle requests. Go ahead and hit your domain, and it should show you your website.
We’ve gone over the basics on how to setup FastCGI using FastCgiExternalServer. You now have a lot of flexibility in how many processes are handling requests, the ability to run different copies of your app and flipping the switch, and pointing to which modules are run with your app. There are a lot of improvements you can now make to setup a very sane deployment process so that each version of code deployed can be its own standalone build and ensuring your production app has 100% uptime, but at this point its up to your imagination.
Aug 29 2009
People have long complained that one of the tricky things about perl is the deployment phase, much because of the intricacies of mod_perl and its unfriendliness towards shared environments. In fact, I would highly recommend FastCGI over mod_perl since it is quite easy to understand and configure. This post is going to focus on smaller shared hosting environments, and how easy it can be to quickly deploy a Catalyst web app.
There are some basic assumptions here. First we need a Linux webserver that has Apache installed and is loading up
fcgid. I believe you can also use the favored mod_fastcgi which I just pointed to above, but I have yet to test this on a shared host. These are binary compatible modules so in theory both work. But again I’ve only used mod_fastcgi for large non-shared hosted deployments. You’ll also need mod_rewrite which is now fairly common.
The best way to install the modules your application depends on is using local::lib. I’ve talked about this before so there isn’t a lot of need to go over the process in detail again, but in a nutshell you can do
$ wget http://search.cpan.org/CPAN/authors/id/A/AP/APEIRON/local-lib-1.004006.tar.gz $ tar xzvf local-lib-1.004006.tar.gz $ cd local-lib-1.004006 $ perl Makefile.PL --bootstrap $ make test && make install $ echo 'eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)' >>~/.bashrc $ source ~/.bashrc
Now you have an environment that you can install your modules into. By default this is localized to ~/perl5. The next step is to install your modules that the application requires. It is good practice to put these into your Makefile.PL so that you can easily install them in one shot. A very basic one would follow this template
use inc::Module::Install;
name 'MyApp';
all_from 'lib/MyApp.pm';
requires 'Catalyst::Runtime' => '5.80011';
requires 'Config::General';
# require other modules here
install_script glob('script/*.pl');
auto_install;
WriteAll;
Now its easy to do
$ export PERL_MM_USE_DEFAULT=1 $ perl Makefile.PL $ make installdeps
The PERL_MM_USE_DEFAULT will configure things such that you don’t have to press enter at every question about a dependency. The make installdeps will install any missing modules, which in this case is going to be everything. You can upgrade the version numbers in the Makefile.PL “requires” lines if you want installdeps to grab the newer distributions as they are released to CPAN.
First thing we have to do is a minor edit to our fastcgi script, which is to tell it to use our local::lib. Since its not part of the environment we setup earlier in .bashrc we have to tell the fastcgi perl script where to find things. Below the “use warnings;” line add this
use lib "/home/myuser/perl5/lib/perl5"; use local::lib;
Make sure to change the path to the correct location of your perl5 modules directory.
The last thing is to make sure your app is located in the public directory root for your host. In my case I created a symbolic link from the public_html folder to my app.
$ cd && mv public_html public_html.old # get rid of current root folder $ ln -s ~/myapp ~/public_html
Then create a .htaccess file in that folder, which should reside beside all of your code
Options ExecCGI FollowSymLinks
Order allow,deny
Allow from all
AddHandler fcgid-script .pl
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/?script/myapp_fastcgi.pl
RewriteRule ^(.*)$ script/myapp_fastcgi.pl/$1 [PT,L]
RewriteRule ^static/(.*)$ static/$1 [PT,L]
We’re just telling apache to turn CGI on, and make sure to execute our fastcgi perl script. Be sure to change the rewrite lines to point to your script (hint: change myapp to your app name).
Now, you should be able to hit your domain. Simple!
So we’ve seen that deploying perl can actually be fairly easy. There are of course some assumptions here, for example, to get the rewrite rules working you’ll need mod_rewrite.so, but this is fairly standard these days. Now you can deploy a perl app with the same ease as languages such as PHP, where it is pretty much plug and play. This should enable people to more easily compete with all the badly written blog, forum, and other generically useful software in the open source world.
Mar 14 2009
Getting code out to your target production machines is a bit of an art form that many seem to take for granted. A lot of shops that I talk to seem to think of the deployment process as an svn up on each node. Even if this works for you, it’s really not a great way to deploy code. The main reason is that it does not scale well. You lend yourself to having different versions of code on different machines at the same time. For example, if your load balancing algorithm made the logs on one machine fill up faster than another and you run out of disk space during the svn update on a node. The correct process should be such that different releases of code can by deployed nearly instantaneously, with old copies left around for rollback. If you just svn up your code has the potential to be out of sync a lot more often than if you deploy code with an rsync and provide your webserver symbolic links to the current version of code. This article will show how this process can work in a scalable way.
Like most of my articles, this is targeted towards shops that are running something along the lines of your typical LAMP stack. I’ve setup this deployment architecture under a PHP setup with Apache, and also a slightly more complicated Perl setup running ModFastCGI instances behind Apache. In one of my most complex deployments I’ve had the database updates go out simultaneously along with the code that worked with those changes, so there is a lot of flexibility in what you can do. I’ll be talking at a very high level on how to accomplish this.
Deployment is the process of getting the code from your development environment into a working production system. There are usually at least three environments: development, staging, and production. Some places may have a demo or test environment for their own specific purpose but we will focus on the main three. Your development environment is active when code is being written and tested by the developers. Typically the development environment points at a ‘dev’ database or if you are using something like MySQL Sandbox perhaps each developer has his own database to work against. Your staging environment should be as close of a replica to your production environment as possible. I typically deploy staging on the exact same servers as production, but point it to a copy of the production database instead of the real thing. Its ideal for this copy to be taken nightly with something like MySQL LVM snapshotting so you’re working on similar data. When staging code is assured to be “bug free” it is then pushed to your live production environement.
Its a good idea to have your code respect these environments. The initialization for your application should be able to infer the environment and pull the appropriate config based on that so that you are pointing to the right DB and any other specifics for that particular environment. One way to do this is to use an environment variable and define that inside of your virtual host file. Since you’ll have a separate virtual host for staging and production your application can then look at the variable and know what configuration to grab.
To get your code from development to staging you have to undergo some form of a build process. Some build processes are more complicated than others. A simple build process may just be as trivial as doing an svn export on your repository to the location you wish to push that code from. More complex builds may want to build and bundle external libraries that the application needs to work. For example in a Perl deployment I’ve compiled the required modules on the build step of each release so that the build consists of the same version of modules that the code was written against. This was really important since CPAN is ever changing, otherwise its possible in a rollback situation that there would be incompatibilities. For the purposes of simplicity I’ll just assume you are deploying code that has everything setup and working on the server (such as PHP usually is).
The build should also be responsible for putting the code into a versioned directory. There are a few benefits here. One is that you can correspond a build of code to the repository revision that it correlates to. Second is that it makes rollback a cinch because multiple versions of the code can exist on the server at the same time. Apache knows which version is active because the virtual host for the active version of code is pointed to by a symbolic link that gets modified as part of the deploy procedure.
I’ve explained that your build script should generate a version and it makes sense for this to come from your repository revision that is getting deployed. Here’s a short bash function that can generate a version from an SVN repository (also works with SVK which is an SVN wrapper)
function get_version {
if [ -e .svn ]; then
eval $1=$(svn info | grep ^Revision | perl -ne "/Revision: (\d+)/; printf('1.%06i', \$1);")
else
eval $1=$(svk info | grep "Mirrored From" | perl -ne "/Rev. (\d+)/; printf('1.%06i', \$1);")
fi
}
and now the build script can do something like
cd /path/to/svn/checkout svn up get_version version echo "this is the version: $version"
Now that you’ve generated a version you should export that version of code into the directory you are going to push from. For this example we’ll say that we store versions in /opt/test.com so the structure might look something like:
/opt/test.com
{version1}
{version2}
{versionN}
...
Then the code would live under version1, version2, … versionN for each version of code deployed.
One way to do this is something like:
mkdir /opt/test.com/$version
svn export -q svn://${svn_host}/trunk ${site_dir}/${version}
Apache needs to know where the new version of code is. Since we’ve decided to put it into an ever changing directory name (the build version) there are at least two methods for letting Apache know the document root of the code. One way is to dynamically generate a virtual host with the explicit versioned path from above (eg. /opt/test.com/0.00010). The document root here will always point to the version that its bundled with, and then on deployment, a symlink from apache’s conf.d directory is pointed to the active virtual host config file. Another way to do this is to have a more static virtual host file, in which the document root path points to a symlinked directory to the version which changes on deploy (eg. /opt/test.com/current-www such that current-www is a symlink to the appropriate version).
In the end the point of deployment is to get your code into production. The version of code should first be rolled into staging, and then when verified as good, into production. The beauty of this is that all that it takes to move code from staging to production is an update to a symbolic link. The current production symlink is wiped out, and pointed to the new version of code, the same one that we deployed for staging.
The deployment should at a minimum
The great thing is that the symbolic links are not repointed until all of the code already exists on each server (the rsync to all servers should happen first). This gives a deploy that doesn’t leave servers out of sync with each other. The deploy is instantaneous and verified working without trying to rsync or svn up over top of running production code.
I’ve gone over a robust way to do code deployments that can be fully automated with a little bit of scripting. Its an easy way to get code out to production and also to ensure that the exact version of code that currently exists in staging is the version of code that goes out. It gives us a very easy way to rollback if a major problem does happen to be found in production and to quickly see what version is currently deployed. And it allows us to change versions of code simultaneously without the end user ever knowing.
