Feb 23 2015

Go Deployment

Category: Deployment,Golang,Organizationjgoulah @ 12:06 AM


In the spirit of the recent re-release of Deployinator, it seems timely to write a little bit about deployment. However, this is not a how-to on writing a Deployinator stack, but an overview of the high-level mechanics of getting a golang app deployed into production. Its worth mentioning that this methodology is just one of many ways you might go about deploying your app, but this is what has worked for me. These are the fundamental pieces that you’ll need to think through no matter what deploy tool you end up using.

Whats to Deploy?

When you run go build it produces a binary executable that can easily be shipped around. But there are actually several key pieces to getting this binary deployed into production. The code you have must be able to compile against the libraries you’re using, and if you are deploying a web based app, you’ll need to think about where your asset files are deployed and how they’re referenced from the go binary. The configuration of the application is also important in that you’ll want to be able to have the app run under different environments, such as development and production. Lastly, you’ll need to think about how to control the app, so that you can easily and consistently restart it when you deploy.


Typically when doing production deployments there exists a build server where the application is compiled and the pieces are put in place for shipping to production. Certainly no production facing app should be getting deployed from a developers virtual machine, but the process for building the app should be identical. Building a go app is fairly straightforward but it can be a good idea to organize your steps and common commands in a Makefile. We’ll build our binary into the bin subdirectory:

        go build -o ./bin/myapp myapp.go

This produces the binary that we’ll eventually ship.


This build process works fine in development but how do we know the external dependencies exist on our build machine? How can we maintain this app building process in an identical way across environments? Whereas in development we can just go get our dependencies, it may be haphazardness to maintain things this way on a production build machine, since you might have different go apps being built that use different versions of the same dependency. Or your developers may have pulled down different versions at different points in time. The current versions of github repositories that your code relies on also may introduce bugs that you want to gain confidence doesn’t impact your application. To maintain consistency in dependency versions, and to ensure these same dependencies exist in our build environment, we’ll use a tool called godep.

Assuming you have a proper go toolchain setup, you can get started with godep by getting it and running a single command:

go get github.com/tools/godep
# then in your project dir
godep save -r ./...

This should pull all of your dependencies into a Godeps directory as well as update your files to import this “frozen” copy of the dependencies you are using.

Since we won’t want to stay locked on old versions of software, we need a way to update our projects dependencies from time to time. This is something you’d probably want to add to your Makefile:

        go get -u -v -d $(import)
        godep update $(import)
        godep save -r ./...

Using this command we can update any of our dependencies like so:

make updep import=github.com/codahale/metrics


A config file is a reasonable way to maintain different environments such as development and production. The gcfg library is a perfectly fine way to define ini style configs that map easily to go structs. It will support multiple environments and you can have a file defining your shared values and then define development and production differences in another set of files. Then you can merge the appropriate set of files based on your environment using ReadFileInto. The example below merges a secrets configuration file with the base configuration to show a simple example, but you can extend this technique to support multiple environments.

In this example we make sure the base config exists, based on a location relative to this file. We’re next looking for the secrets config, first in the app directory where we may store it locally for development, and then in an /etc directory for production where a configuration management system may place the file more securely. This way this sensitive file doesn’t have to be stored in the main repository, but can pass production values securely to the application. Note that means this file wouldn’t be deployed in the normal way, and the app will fallback to the secure values in /etc when it doesn’t find the local file. The full configuration example implementation can be found here but remember this is just a very basic example to show how to merge configs.


When you deploy the app to production you’ll have to restart it in some way. The simplest way to do this is to write an init script. But you shouldn’t have to write an init script by hand these days. Check out pleaserun to generate a script for your system. Now you can stop, start, and restart your process whether you are using upstart, systemd, plain old init, or something else. You can go one step further and implement a graceful restart, but the implementation will vary depending on your needs.

Tying it Together

So now we have a way to deal with our dependencies and the build. We know how to control the app. The app knows how to find its configuration files and override base values with production settings. We can use the same runtime.Caller trick that we used to locate the config file to make it easier for the application to find its static files as well.

At this point there are only a few steps that need to happen on a deploy box:

  • Get the newest code
  • git fetch && git reset --hard
  • Build the binary
  • cd $basedir && make build
  • Rsync the code
  • rsync -va --delete --force \
    --include='static' --include='bin' --include='config*.cfg' --include='*/*' --exclude='*' \
    deploy.mydomain.com:/var/myorg/build/golang/src/github.mydomain.com/jgoulah/myapp/ /usr/myorg/myapp
  • Use your init script to restart the app, depending on the system
  • /etc/init.d/myapp restart

In this example the rsync command is run on an application box and pulling from our deploy host. In a large deploy we may dsh out to many app hosts to rsync our build over in a pull based fashion. The /var/myorg/build directory is where our make build command was run to produce our binaries. We sync the bin directory that contains the binaries, the static directory which has our html/js/css, and the config files. Lastly we restart our app. These steps can be coded into a Deployinator stack or with the deploy tool of your choice. But regardless of the tool we use, we will have a repeatable and dependable process with reliable dependencies that we can ship.

Tags: , , , , ,

Oct 02 2010

Quick Tip: Dynamically Updating Screen Window Titles With The Current Server Hostname

Category: Organization,SSH,Systems @ 12:13 PM

I haven’t had a ton of time for blogging lately but figured this tip was good enough to throw out there for all the screen users. One way I like to organize servers that I’m ssh’d into is using screen windows. As you hopefully know you can use Ctrl-A c to create sub windows within screen. Then you can switch between them in several ways such as using Ctrl-A X where X is the window number, Ctrl-A n or Ctrl-A p for next and previous, and Ctrl-A “ to get a list of the windows for selection. You can move windows around with Ctrl-A : then type number X where X is the window you want to swap with. Finally, you can also name the windows with Ctrl-A A. So usually I ssh into a server, and manually change the window title to the server name I’m ssh’d into so that its easy to organize and remember where my windows are.

Turns out thats a lot of repetitive work that can easily be scripted. You can create a really simple script called ssh-to and place in your ~/bin or somewhere in your path


hostname=`basename $0`

# set screen title to short hostname
echo -n -e "\033k${hostname%%.*}\033\134"

# ssh to the server with any additional args
ssh -X $hostname $*

# set hostname back when session is done (-s on osx)
echo -n -e "\033k`hostname -s`\033\134"

Now, create symbolic links to the script with each server hostname that you use. This can be a little tedious but you only have to do it once and the benefit is that you can now tab complete ssh’ing into servers.

For example you would do something like

ln -s ssh-to myserver.com

Then anytime you type “myserver.com” the tab completion can fill it out, you’re ssh’d into the server (assuming you have keys setup you don’t have to type a password) and your screen window title is updated with the domain info trimmed off, in this case myserver. Now your screen windows update their own titles anytime you ssh into a new server!

Tags: , , ,

May 01 2010

Quick Tip: Gist From the Command Line

Category: Organizationjgoulah @ 3:56 PM

Many a time while working it is convenient to quickly show someone else code or a chunk of output from some command. The easiest way to do this is through a pastebin service. It’s also pretty much mandated on IRC to use a service like this and there are literally thousands out there to choose from.

The gist service on github is actually fairly convenient especially if you have a github account since it keeps an archive of all of your previous gists. It also does a good job formatting and lets you paste privately. We’ll use the App::Nopaste tool to paste a gist straight from the command line.

First off install the tool from cpan

$ cpan App::Nopaste

You can use this tool anonymously but if you want to keep an archive of your pastes you can simply setup your git credentials in your .gitconfig file. The token here is your API token which can be found under your account settings page.

        user = jgoulah
        token = 00000000000000000000000000000000

You also need either Git installed or Config::INI::Reader to allow the module to read your .gitconfig file.

Now to make this a bit easier to remember we can create an alias in our .bashrc file. In this case I’m specifying –private so that only people that I give this secure URL out to can see, and I’m also specifying to use the Gist service. The nopaste app supports a variety of other services that you can use but as of this writing Gist is the only one that supports the –private flag.

alias gist='nopaste --private --service Gist'

Now you can use the command to paste something such as a script from the command line and the gist URL is returned

$ gist somescript.sh

There you have it!

Tags: , , , , ,

Apr 19 2009

Make Browsing Code Easier with Ack and Ctags

Category: Organizationjgoulah @ 8:44 PM


When working with open source software, its essential to know how to navigate large code bases, perhaps unfamiliar and quite large. There are a few tools I use to do this that should be part of any developers arsenal, and they are: ack and ctags.

Ack can be thought of as a faster and more powerful grep. It searches recursively by default, ignores binary files and most version control files (think .svn), lets you specify file types on search, use perl regular expressions, and has easier to read output than grep.

Ctags is a tool that many are familiar with, and there are tons of articles about it already.  But I bring it up so I can show some quick starter scripts that you’d use to generate the tags for PHP or Perl scripts.  I’ll show a quick C++ and Java example too, since I use those from time to time.



There’s really not much to installing ctags.  You could download and compile the source, but just get it from your package management system.

If you’re using Debian or Ubuntu you can do:

sudo apt-get install exuberant-ctags

Similarly in CentOS and Redhat based distros:

sudo yum install ctags


Ctags basically indexes your code and creates a tag file that can then be used in your editor to literally jump around your code.  If you see a method call as you’re browsing code, you can jump to the definition of that method with one keystroke, and back to where you were.  Same thing for variables.  In a keystroke you can jump to see where its defined.   As you jump through code a stack is created,  and as you jump back you are just popping off that stack, also known as LIFO, by the way.

Generating the Tag Files

I have a few scripts to generate the ctags files depending on different codebases. I tend to use VI so I’m going to cover how to do it with that editor, but you can also use emacs.

For these examples I’m going to send the output to a file in my ~/.vim/tags/ directory, which I’ll later add to .vimrc. You could expand these scripts to fit your needs. For these examples they are pretty basic and hardcoded.


$ cat bin/ctags_php
cd ~/mysandbox/myphpproject
ctags -f ~/.vim/tags/myphpproject \
--langmap="php:+.inc" -h ".php.inc" -R \
--exclude='*.js' \
--exclude='*.sql' \
--totals=yes \
--tag-relative=yes \
--PHP-kinds=+cf-v \
--regex-PHP='/abstract\s+class\s+([^ ]+)/\1/c/' \
--regex-PHP='/interface\s+([^ ]+)/\1/c/' \
--regex-PHP='/(public\s+|static\s+|abstract\s+|protected\s+|private\s+)function\s+\&?\s*([^ (]+)/\2/f/'

and when you run it, you’ll see something like:

$ ctags_php
498 files, 66678 lines (2624 kB) scanned in 0.6 seconds (4604 kB/s)
1643 tags added to tag file
1643 tags sorted in 0.00 seconds


In Perl you can do things a bit smarter since you should have a Makefile.PL script to keep track of your dependencies. If so you can add:

ctags -f ~/.vim/tags/myperlcode --recurse --totals \
--exclude=blib \
--exclude='*~' \
--languages=Perl --langmap=Perl:+.t \

Then you should do:

perl Makefile.PL
make tags


You can do a very similar thing for C++ code

$ cd /path/to/code
$ ctags -f ~/.vim/tags/myc++code --tag-relative=yes --recurse --language-force=c++ *


Or say we want to tag the entire java library itself

$ ctags -f ~/.vim/tags/java -R --language-force=java /opt/java/src

Letting VI know about your files

After creating one or more tagfiles you should edit your ~/.vimrc file and add the location to your tag files and separate the entries by commas or spaces

set tags=~/.vim/tags/myphpproject,~/.vim/tags/myperlcode

Navigating Around the Code

In VI there are two easy commands to jump around.

To move to the definition of a method/variable, place the cursor over it and press

Ctrl + ]

And to jump back

Ctrl + t

If you try to jump to something and it isn’t found, its probably something in a library you’re using, so you’ll have to grab the source and tag those too.



There are a few ways to install ack listed on the ack homepage.   If you are familiar with CPAN you can install App::Ack, or if you want to use package management you can grab ack-grep on Ubuntu or ack on Redhat based distros.


The best thing I can really tell you is to read the ack help

$ ack --help

Ack takes a regular expression as the first argument and a directory to search as the second. Typically you want to search all (-a) files or in a case insensitive fashion (-i)

$ ack -ai 'searchstring' .

Or you can search specific file types

$ ack --perl  searchterm

And one really cool thing is though ack gives nice colorized output in a structured fashion, if you pipe it to another process it outputs like grep by default so that you can continue to pipe it

For example lets say I want to find the modules MooseX::Types is using

ack -a '^use.*;' ~/perl5/lib/perl5/MooseX/Types/

It gives something looking like (output truncated)

9:use warnings;
10:use strict;
12:use base 'Exporter';

9:use warnings;
10:use strict;
12:use MooseX::Types;
13:use Moose::Util::TypeConstraints ();
15:use namespace::clean -except => [qw( meta )];

If you pipe the output it looks more like grep output

ack -a '^use.*;' ~/perl5/lib/perl5/MooseX/Types/ | cat

Again the output is truncated but looks like

/home/jgoulah/perl5/lib/perl5/MooseX/Types/Base.pm:11:use MooseX::Types::Util             qw( filter_tags );
/home/jgoulah/perl5/lib/perl5/MooseX/Types/Base.pm:12:use Sub::Exporter                   qw( build_exporter );
/home/jgoulah/perl5/lib/perl5/MooseX/Types/Base.pm:13:use Moose::Util::TypeConstraints;
/home/jgoulah/perl5/lib/perl5/MooseX/Types/Base.pm:15:use namespace::clean -except => [qw( meta )];

Lets say for example reasons I wanted to find all the modules used in the code and make sure they are installed (using the ‘make install’ command in the module would be the more correct and easier way), you could do

$ ack -ah '^use\s[^\d].*;' ~/perl5/lib/perl5/MooseX/Types/ | \
  ack -v 'warnings|strict|base' | \
  perl -ne "m|use ((\w+:?:?)+)(.*)(;)|; print qq{\$1\n};" | \
  sort -u | xargs cpan

which cleans the output and gives the module list to cpan for installation and it lets me know I have everything installed and up to date

Carp::Clan is up to date (6.00).
Class::MOP is up to date (0.81).
Devel::PartialDump is up to date (0.07).
Moose is up to date (0.74).
Moose::Meta::TypeConstraint::Union is up to date (0.74).
Moose::Util::TypeConstraints is up to date (0.74).
MooseX::Meta::TypeConstraint::Structured is up to date (undef).
MooseX::Types is up to date (0.10).
MooseX::Types::Util is up to date (undef).
namespace::clean is up to date (0.11).
Scalar::Util is up to date (1.19).
Sub::Exporter is up to date (0.982).


These are pretty commonplace but great tools to know if you don’t already. Try to integrate them into your work flow and I think you’ll notice that it will speed you up quite a bit, especially when you are browsing through unfamiliar territory.

Tags: , , , , ,

download xiuxiu editor foto shambho shankara mp3 free download pikeno e menor perdicao download download lagu surat at taubat smarthru 4 download pl abbey road 60s drums download mac cell phone repair download download kara winter magic album download intel gma booster terbaru download mp3 tantowi yahya free download farhan ali qadri video naats free download internal medicine harrison download music ragheb alama sinin dhada songs download in ziddu download form 4852 from the irs website free invitation templates download and print download boundless by cynthia hand free la chatimi cantare mp3 download free download of deception point ebook download munni badnam hui free mp3 download jtx party like a rockstar download hide ip ng 1.40 download ooh la la goldfrapp download time after time quietdrive crime and punishment mp3 download sweety gippy mp3 download caminhos da liberdade download minions banana video download reflex flugsimulator download gratis naruto shippuden 208 pt download sniper elite wii download ntsc pirata do espaço download dublado download ptanks full version o justiceiro download pc rip microsoft hda driver download download admit card iti jharkhand 2012 bada 2.0 download wave 2 download gangster life gta apple safari 4 x download business intelligence download oracle download os x dvd installer download gta 8 vice city myegy szybcy i wściekli 5 2011 download download highschool of the dead color download shakira ft pitbull rabiosa zippy download lagu t ara zombie jelly car music download download driver sony vaio 32bit klezmer music to download free download of shreenathji bhajan aga bai arechya download serie v 3 temporada download download the jeremy kyle show download jogos fazenda gratis pc minimizer download para mu susana nothing at all download download um novo vencedor damares playback download command line mac os download paypal jar for android download zeljko vasic zanjisi kukovima marian keyes watermelon ebook download internet download manager flurry icon kruti dev free download windows download chief keef choppa go bang download msi 3.1 windows installer chessmaster free download with crack bravo hits 98 download download disrespect kirko bangz x plane 6 demo download la baby jonas brothers mp3 download netgear ga311 windows 7 download free download habib painter mp3 download tweetdeck desktop windows 7 ekhon ami mp3 download promethean the created download pdf how to download youtube videos to ipad amazon download games steam cypress hill download 2011 download treu nha hang xom 2 download kick out the epic mother wooh da kid stepped download x2 x men united download imaginasamba perfeição download download navigation for mobile zor ka jhatka song download mp3 download audi a4 owners manual download pretty little liars s01e05 avi download nitro circus season 1 download eega promotional song download the simulator 2012 demo x264 codec download mac download lições para toda vida legendado download mkisofs for ubuntu download star trek voyager scorpion lil b 855 download download pokemon black 2 jap rom spells aprilynne pike pdf download ita download ways of reading cursed crusader trainer download download darmowe gry dla dzieci download spyglass for android alaa wardi 7aram free download massive attack teardrop song download download settings for nokia x6 download account opening form obc download fl studio on a mac worth dying for download s bot download free silkroad 1 click downloader download 5 ishq ka sheen download download melodia que eu conheço stephen king novels download free 100 download psp games for free a escolha download dublado ptgui 9 mac download kz hack download gratis download driver yamaha psr 3000 download macroeconomics policy and practice download hp dv1000 sound driver download famous five movies download phim benh nhan nguoi anh download gta 1 for free download gangs of wasseypur movie free download panasonic sd jukebox software download falling up drake download trackmania sunrise extreme full version free download de simuladores de combate aereo download i bruise easily download pioneer dj software free download noah and the whale life goes on mp3 download recover deleted files software download sơ đồ kế toán download free desk phone ringtone fußball manager 12 download vollversion kostenlos even greater mp3 download planetshakers download snmp for windows 2000 free download amuse park game bejeweled 3 jar download sleeping at last quicksand download download ứng dụng cho nokia n8 coral player download luna download jill scott whenever you're around download fairy tail games for pc download photoshop 8 cs me myegy download yahoo latest version for free mp7 player free download mac download internet explorer 8 vista java hry download 128x160 download jason upton key of david download march of the wooden soldiers download barad toro be dast avordam mp3 download songs of aitraaz from songs pk com 6.72 f ai nightmare download download tiny toon adventures nes download i'm yours mp3 download 9 hours rom pipi player download vista deskjet d2360 software download download booster pack hack abhas ha mp3 song download mp3 songs download 3gp download cod 4 3rd person mod trey songz blind download free download hawaii 5 0 season 2 kenji free download fort minor download clubbed to death 2 quebrando regras 2 download portugues chamas da vingança 1984 download download sketchup 8 deutsch virtual router manager download xp download instalador chrome offline download soundgarden live to rise download canon mx410 printer driver download the harold song kesha mp3 garmin 255w download maps free download mise a jour mcafee download visio windows 7 64 bit download gen psp 3000 quasi amici film download gratis ita download plano de fuga rmvb legendado download alfonso loher name in the sky download mp3 zahra damariva alasan download j rock songs download bihar secretariat assistant admit card download project 64 64 bits download naruto chapter 589 cartoon download for ipad download tributo a bezerra da silva download roda a roda jogo cx one full download armin van buuren rapture download mp3 download nein mann video download tambor de funk download rota de fuga rmvb free download of shawty got moves mp3 download cm7 for droid 2 global download executive resume format amnesia game download mac crash bandicoot mutant island download pc ipod touch download pictures to computer download tu pirata soy yo chayanne download lg pc suite p990 harmor vst plugin download download abaqus 6.10 student edition i like cereal song download filmes alta resolucao download italian lessons download mp3 mass effect 3 download pirate download manager idm key download wh cs 2011 bau simulator download ita download efek suara unik download girl talk ultraviolet sound download bicara hati episod 4 annie khalid songs download download driver epson cx5500 free quake 3 download bots infinity blade 2 ipa download crack pk songs download list dream the game download free cydia download step by step motorola xoom rom download download let's go ricky luna remix zune manual download pdf download hit and run 2012 dvdrip download do sapo videos download do jogo cities xl 2011 download song socha na tha by alamgir ktechlab for ubuntu download download vara rece kamelia zippy download schenk mir dein herz gipsy kings volare download free welsh flag to download download afinador do cifra club 3 gatsu 9 ka download free mp3 angry birds android download download manager error the server returned an error download sdo x season 2 song download paint shop pro download turn to u justin bieber download original ruu for evo download maps to print boys over flowers download songs download free regular show episodes download yamaha psr 1500 styles satyamev jayate download song download mouse fix for windows xp download elliot in the morning kenapa tidak bisa download film h.p f4200 printer software download p square game over download mp3 carpet 3d max download download mw 3 1.07 patch the legend of zelda download snes rom download 9 temporada friends download on my freebox ne fonctionne pas download tito lopez the blues sharebeast download sweeney todd final scene papago x8.5 wince 5 download download monkey for rhino download na paz de jah download gratuito adobe reader 8 ngo accounting software download download onto mp3 from youtube somebody's me mp3 download enrique download ocarina of time 1.0 download 64 bit windows tax form 8379 download frisky tinie tempa download zippy download pro update psp go 6.60 download lagu chrisye gejolak cinta download themes for gw300 clr via c# richter download download 8195 the damned rar parayathe ariyathe malayalam song download autobiography of abraham lincoln download download phim hiep dao hoa hetaoni english download part 2 cenário de novela download mp3 sandra brown envy download pdf download sims 1 love bed sende ahasa wage mp3 download download voice changing application download feed us 2 free download open source library management system download alana grace black roses red download audi a3 manual download free irctc mobile application stand o food 3 apk download download dss dj effects visual basic software download download film khuda kay liye download intezar remix by falak sri rama rajyam mp3 download download dan seals one friend download virtual families mac free download gossip girl cecily von ziegesar download autoramas fale mal de mim gabin doo uap mp3 download free download love in this club mp3 download dead space 3 demo pc charmed download season 1 nhac chuong theo ten download angry birds season download free pc reign of hunters download download de pokemon flash download full screen theme wordpress download manager 6.05 crack download planta x zumbi download razor ramon entrance music download skype xperia 8 download lagu tercipta untukku ungu download shining inheritance ep 16 trial download microsoft project 2010 halo ce download key thermodynamics 6th cengel download vmware ova converter download download e.r legendado 1 temporada wilfred season one download download hivi mata hati download apun bola mp3 zmierzch księżyc w nowiu download peb cod tool download 1.5 web client get download speed ryback meat mp3 download music download on itunes download beenie man i'm okay vigilante 8 2nd offence download pc download movie 2012 in hindi latha tamil font free download word melhor impossivel download legendado download lloyds tsb bank statement download oki b6300 driver how to download spoutcraft free popup blocker download google chrome download tenth avenue north losing download sound intervention mw2 download jogo harry potter pc download pokemon blanc nds alda célia playback download download shwayze get you home download tower bloxx mobile game dani california official video download download jump out the gym download all killer no filler sum 41 ra one full movie download 2011 download rebelde só pro meu prazer download tees maar khan movie in avi format activex control download install download video setia band stasiun cinta download de temas nokia x2 00 download power geez 2005 computer games download com download benny and babloo songs soc pc camera driver download for xp manually download sophos virus definitions jackie chan adventures download links download dragostea se face in doi download terjemahan kitab al umm hp photosmart c3180 download scanner download ipcop for windows 7 nero 8 download windows 7 64 bit kick buttowski kick in genes download dewana 2013 mp3 download download 2 chainz birthday song free how to download correct video driver download film g 30s pki firing games download full version download free 3d motorbike racing download call of duty 4 zombies download smart mobile themes download crbl romanu n are noroc hotfiles download diggy simmons make you mine download bangla natok bhalobashi tai download kml from my maps download song if this charlie sheen shinda new album download download outcast 1 temporada keterlaluan the potters mp3 download download office 2007 upload download falling skies 2 temporada rmvb download sajan all songs oblivion mod manager download mac download toma o meu coração download pierce the veil caraphernelia mp3 download lagu jkt48 original download leaf by elle varner vandalism coming alive mp3 download download god of war betrayal 240x320 download amanda by zigi mp3 download apostila do trf hp 635 driver download windows xp download pro e student version download office 2007 turkish proofing tools download filme a era do nariz vermelho are you in download download lagu true worshippers jadi sepertimu warlords battlecry 3 download free full version apostilas calculo 1 download calof duti 2 download gratis