Oftentimes you need to know what MySQL is doing right now and furthermore if you are handling heavy traffic you probably have multiple instances of it running across many nodes. I’m going to start by showing how to take a tcpdump capture on one node, a few ways to analyze that, and then go into how to take a distributed capture across many nodes for aggregate analysis.
The first thing you need to do is to take a capture of the interesting packets. You can either do this on the MySQL server or on the hosts talking to it. According to this percona post this command is the best way to capture mysql traffic on the eth0 interface and write it into mycapture.cap for later analysis:
% tcpdump -i eth0 -w mycapture.cap -s 0 "port 3306 and tcp[1] & 7 == 2 and tcp[3] & 7 == 2" tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 47542 packets captured 47703 packets received by filter 60 packets dropped by kernel
The next step is to take a look at your captured data. One way to do this is with tshark, which is the command line part of wireshark. You can do yum install wireshark or similar to install it. Usually you want to do this on a different host than the one taking traffic since it can be memory and CPU intensive.
You can then use it to reconstruct the mysql packets like so:
% tshark -d tcp.port==3306,mysql -T fields -R mysql.query -e frame.time -e ip.src -e ip.dst -e mysql.query -r mycapture.cap
This will give you the time, source IP, destination IP, and query but this is still really raw output. Its a nice start but we can do better. Percona has released the Percona Toolkit which includes some really nice command line tools (including what used to be in Maatkit).
The one we’re interested in here is pt-query-digest
It has tons of options and you should read the documentation, but here’s a few I’ve used recently.
Lets say you want to get the top tables queried from your tcpdump
% tcpdump -r mycapture.cap -n -x -q -tttt | pt-query-digest --type tcpdump --group-by tables --order-by Query_time:cnt \ --report-format profile --limit 5 reading from file mycapture.cap, link-type EN10MB (Ethernet) # Profile # Rank Query ID Response time Calls R/Call Apdx V/M Item # ==== ======== ============= ===== ====== ==== ===== ==================== # 1 0x 0.3140 6.1% 674 0.0005 1.00 0.00 shard.images # 2 0x 0.8840 17.1% 499 0.0018 1.00 0.03 shard.activity # 3 0x 0.1575 3.1% 266 0.0006 1.00 0.00 shard.listing_images # 4 0x 0.1680 3.3% 265 0.0006 1.00 0.00 shard.connection_edges_reverse # 5 0x 0.0598 1.2% 254 0.0002 1.00 0.00 shard.listing_translations # MISC 0xMISC 3.5771 69.3% 3534 0.0010 NS 0.0 <86 ITEMS>
Note the tcpdump options I used this time, which the tool requires to work properly when passing –type tcpdump. I also grouped by tables (as opposed to full queries) and ordered by the count (the Calls column). It will stop at your –limit and group the rest into MISC so be aware of that.
You can remove the –order-by to sort by response time, which is the default sort order, or provide other attributes to sort on. We can also change the –report-format, for example to header:
% tcpdump -r mycapture.cap -n -x -q -tttt | pt-query-digest --type tcpdump --group-by tables --report-format header reading from file mycapture.cap, link-type EN10MB (Ethernet) # Overall: 5.49k total, 91 unique, 321.13 QPS, 0.30x concurrency _________ # Time range: 2012-01-08 15:52:05.814608 to 15:52:22.916873 # Attribute total min max avg 95% stddev median # ============ ======= ======= ======= ======= ======= ======= ======= # Exec time 5s 3us 114ms 939us 2ms 3ms 348us # Rows affecte 316 0 13 0.06 0.99 0.29 0 # Query size 3.64M 18 5.65k 694.98 1.09k 386.68 592.07 # Warning coun 0 0 0 0 0 0 0 # Boolean: # No index use 0% yes, 99% no
If you set the –report-format to query_report you will get gobs of verbose information that you can dive into and you can use the –filter option to do things like getting slow queries:
% tcpdump -r mycapture.cap -n -x -q -tttt | \
pt-query-digest --type tcpdump --filter '($event->{No_index_used} eq "Yes" || $event->{No_good_index_used} eq "Yes")'
Now that we’ve taken a look at capturing and analyzing packets from one host, its time to dive into looking at our results across the cluster. The main trick is that tcpdump provides no option to stop capturing – you have to explicitly kill it. Otherwise we’ll just use dsh to send our commands out. We’ll assume you have a user that can hop around in a password-less fashion using ssh keys – setting that up is well outside the scope of this article but there’s plenty of info out there on how to do that.
There’s a few ways you can let a process run on a “timeout” but I’m assuming we don’t have any script written or tools like bash timeout or the one distributed in coreutils available.
So we’re going off the premise that you will background the process and kill it after a sleep by grabbing its pid:
( /path/to/command with options ) & sleep 5 ; kill $!
Simple enough, except we’ll want to capture the output on each host, so we need to ssh the output back over to the target using a pipe to grab the stdout. This means that $! will return the pid of our ssh command instead of our tcpdump command. We end up having to do a little trick to kill the right process, since the capture won’t be readable if we kill ssh command that is writing the output. We’ll need to kill tcpdump and to do that we can look at the parent pid of the ssh process, ask pkill (similar to pgrep) for all of the processes that have this parent, and finally kill the oldest one, which ends up being our tcpdump process.
Then end result looks like this if I were to run it across two machines:
% dsh -c -m web1000,web1001 \
'sudo /usr/sbin/tcpdump -i eth0 -w - -s 0 -x -n -q -tttt "port 3306 and tcp[1] & 7 == 2 and tcp[3] & 7 == 2" | \
ssh dshhost "cat - > ~/captures/$(hostname -a).cap" & sleep 10 ; \
sudo pkill -o -P $(ps -ef | awk "\$2 ~ /\<$!\>/ { print \$3; }")'
So this issues a dsh to two of our hosts (you can make a dsh group with 100 or 1000 hosts though) and runs the command concurrently on each (-c). We issue our tcpdump on each target machine and send the output to stdout for ssh to then cat back to a directory on the source machine that issued the dsh. This way we have all of our captures in one directory with each file named with the target name of each host the tcpdump was run. The sleep is how long the dump is going to run for before we then kill off the tcpdump.
The last piece of the puzzle is to get these all into one file and we can use the mergecap tool for this, which is also part of wireshark:
% /usr/sbin/mergecap -F libpcap -w output.cap *.cap
And then we can analyze it like we did above.
http://www.mysqlperformanceblog.com/2011/04/18/how-to-use-tcpdump-on-very-busy-hosts
http://stackoverflow.com/questions/687948/timeout-a-command-in-bash-without-unnecessary-delay
http://www.xaprb.com/blog/2009/08/18/how-to-find-un-indexed-queries-in-mysql-without-using-the-log/
Just to clarify this command a bit more, particularly how the kill part works since that was the trickiest part for me to figure out.
When we run this
$ dsh -c -m web1000,web1001 \
'sudo /usr/sbin/tcpdump -i eth0 -w - -s 0 -x -n -q -tttt "port 3306 and tcp[1] & 7 == 2 and tcp[3] & 7 == 2" | \
ssh dshhost "cat - > ~/captures/$(hostname -a).cap" & sleep 10 ; \
sudo pkill -o -P $(ps -ef | awk "\$2 ~ /\<$!\>/ { print \$3; }")'
on the server the process list looks something like
user 12505 12504 0 03:12 ? 00:00:00 bash -c sudo /usr/sbin/tcpdump -i eth0 -w - -s 0 -x -n -q -tttt "port 3306 and tcp[1] & 7 == 2 and tcp[3] & 7 == 2" | ssh myhost.myserver.com "cat - > /home/etsy/captures/$(hostname -a).cap" & sleep 5 ; sudo pkill -o -P $(ps -ef | awk "\$2 ~ /\<$!\>/ { print \$3; }")
pcap 12506 12505 1 03:12 ? 00:00:00 /usr/sbin/tcpdump -i eth0 -w - -s 0 -x -n -q -tttt port 3306 and tcp[1] & 7 == 2 and tcp[3] & 7 == 2
user 12507 12505 0 03:12 ? 00:00:00 ssh myhost.myserver.com cat - > ~/captures/web1001.cap
So $! is going to return the pid of the ssh process, 12507. We use awk to find the process matching that, and then print the parent pid out, which is then passed to the -P arg of pkill. If you use pgrep to look at this without the -o you’d get a list of the children of 12505, which are 12506 and 12507. The oldest child is the tcpdump command and so adding -o kills that guy off.
So if we were only running the command on one host we could use something much simpler
ssh dbhost01 '(sudo /usr/sbin/tcpdump -i eth0 -w - -s 0 port 3306) & sleep 10; sudo kill $!' | cat - > output.cap]]>
I’ve recently been playing around with some iOS code, and while I find Xcode an excellent editor with a lot of great built-ins, I’m a vim user at heart. It’s the editor I’ve been using on a daily basis for years, so its tough to switch when I want to do iPhone development. Of course there are some things you just have to use Xcode for, but I’ve been able to tailor vim to do most of my dev work, as well as build and run the simulator on the command line. I couldn’t find a single cohesive tutorial on this, so these are the pieces I put together to make this work for me.
To start, there is a decent plugin called cocoa.vim to get some of the basics going. It attempts to do a lot of things but I’m only using it for a few specific things – the syntax highlighting, the method list, and the documentation browse functionality.
One caveat is the plugin is not very well maintained, given the last release is nearly 2 years ago. Because of this you’ll want to grab the version from my github:
git clone https://jgoulah@github.com/jgoulah/cocoa.vim.git
Install it by following the simple instructions in the README.
You’ll get the syntax highlighting by default, and the method listing is pretty straightforward. While you have a file open you can type :ListMethods and navigate the methods in the file. Of course you may want to map this to a key combo, here’s what I have in my .vimrc for this, but of course you can map to whatever you like:
map <leader>l :ListMethods
There’s another command that you can use called :CocoaDoc that will search the documentation for the keyword and open it in your browser. For example, you can type :CocoaDoc NSString to open the documentation for NSString in your browser. However OS X gives a warning every time, which is pretty annoying. You can disable this on the command line:
defaults write com.apple.LaunchServices LSQuarantine -bool NO
After you’ve run the command, restart your Finder (or reboot). Lastly, it would be annoying to type the keyword every time, so you can configure a mapping in your .vimrc so that it will try to launch the documentation for the word the cursor is on:
map <leader>d :exec("CocoaDoc ".expand("<cword>"))<CR>
One of the biggest things I miss when I’m in Xcode is the functionality you get from ctags. Of course you can right click on a function and “jump to definition”. Works fine, but feels clunky to me. I love being able to hop through code quickly with ctags commands. I’ve talked about this before, so I’m not going to give a full tutorial about ctags, but I’ll quickly go over how I made it work for my code.
First go ahead and grab the version from github that includes Objective-C support:
git clone https://github.com/mcormier/ctags-ObjC-5.8.1
Compile and install in the usual way. Now you can utilize it to generate yourself a tags file. Mine looks like this:
#!/bin/bashcd ~/wdir
for i in MyApp ios_frameworks ; do echo "tagging $i" pushd $i /usr/local/bin/ctags -f ~/.vimtags/$i -R \ --exclude='.git' \ --langmap=objc:.m.h \ --totals=yes \ --tag-relative=yes \ --regex-objc='/^[[:space:]]*[-+][[:space:]]*\([[:alpha:]]+[[:space:]]*\*?\)[[:space:]]*([[:alnum:]]+):[[:space:]]*\(/\1/m,method/' \ --regex-objc='/^[[:space:]]*[-+][[:space:]]*\([[:alpha:]]+[[:space:]]*\*?\)[[:space:]]*([[:alnum:]]+)[[:space:]]*\{/\1/m,method/' \ --regex-objc='/^[[:space:]]*[-+][[:space:]]*\([[:alpha:]]+[[:space:]]*\*?\)[[:space:]]*([[:alnum:]]+)[[:space:]]*\;/\1/m,method/' \ --regex-objc='/^[[:space:]]*\@property[[:space:]]+.*[[:space:]]+\*?(.*);$/\1/p,property/' \ --regex-objc='/^[[:space:]]*\@implementation[[:space:]]+(.*)$/\1/c,class/' \ --regex-objc='/^[[:space:]]*\@interface[[:space:]]+(.*)[[:space:]]+:.*{/\1/i,interface/' popd
done;Just a little explanation here, I’m going into my working directory and generating a tags file for both MyApp and ios_frameworks. ios_frameworks is just a symlink that points to /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.sdk/System/Library/Frameworks
If you’re interested in how I load up these tag files you can check out my .vimrc on github.
Another handy feature from I wanted to emulate was flipping between the header and source file. Luckily there is yet another vim plugin that will do this for you called a.vim.
Grab it off github and install it by dropping it in your ~/.vim/plugin directory:
git clone https://github.com/vim-scripts/a.vim.git cp a.vim/plugin/a.vim ~/.vim/plugin
There’s a tiny bit of configuration that you can put in .vim/ftplugin/objc.vim:
let g:alternateExtensions_m = "h" let g:alternateExtensions_h = "m" map <leader>s :A
This just tells it which files to use for source and header files and creates another shortcut for us to swap between the two.
So now that we’ve got most of the basic things that help us edit code, its time to build the code. You can use xcodebuild for this on the command line. If you type xcodebuild –usage you can see the variety of options. Here’s what worked for me to build my target app, noting that I setup everything in Xcode first and made sure it built there. After that you can just specify the target, sdk, and configuration. This will create a build under the Debug configuration for the iPhone simulator:
xcodebuild -target "MyApp Enterprise" -configuration Debug -project MyApp.xcodeproj -sdk iphonesimulator5.0
And now to run the app. Back to github there’s a nice little app called iphonesim. Download that and just build it with xcode:
git clone https://github.com/jhaynie/iphonesim.git open iphonesim.xcodeproj
Put the resulting binary somewhere in your PATH like /usr/bin.
Now we can launch the app we just built in the simulator:
iphonesim launch /Users/jgoulah/wdir/MyApp/build/Debug-iphonesimulator/MyAppEnterprise.app
Last thing you’ll want are logs. Another thing that is arguably more easily accessible in Xcode, but we actually can get the console log. If you’re using something like NSLog to debug this will grab all of that output. You’ll have to add a couple lines to your main.m source file:
NSString *logPath = @"/some/path/to/output.log"; freopen([logPath fileSystemRepresentation], "a", stderr);
And then you can run tail -f /some/path/to/output.log
This is how I use vim to compile iPhone apps. Your mileage may vary and there are many ways to do this, it just happens to be the path that worked well for me without having to pull my hair out too much. Xcode is a great editor and there are some things that you will have to use it for such as the debugging capabilities and running the app on the phone itself. Of course even these are possible with some work, so if anyone has tips feel free to leave a comment.
]]>I’ve been meaning to play around with backbone.js for a while, trying to piece together what can make a sensible framework for frontend code. There seem to be a few out there, another I’ve wanted to try is sproutcore. The truth is, I’m a backend coder, a database guy, a systems engineer… but I love all aspects of coding, and although I’m quite terrible at design and CSS, I’ve been getting more into Javascript lately. Its quite a nice language for server side development with node – small, simple, server based apps that work well with websockets. I’ve written a little bit about that in the past. Everything is moving to the web, and its nice to have a modern and easy to use interface to place in front of your tools.
The app is a simple playlist. You can search for songs, get a list of potential matches, and add the song to your playlist. You can cycle back and forth through the songs, pause them, and delete them from the list. Thats about it!
If you want to cut to the code you can find it here on github.
Part of the goals for this project were to get out of my comfort zone and learn something new. So I decided to make it a frontend only app. All HTML/Javascript and no backend services. Well, none that I write, it does some interaction with other services or it would be pretty boring. If you end up checking the project out, you’ll find the markup could use some help (patches welcome!). But the goal was just to get something functional. This isn’t the way I’d go about things if I were building an app to start a company with, its just a fun hack.
I didn’t know backbone at all, and it seems hard to find good examples that are more than “heres a text box on a page”. One of the better ones out there that is well documented is the todo list app. It turns out this app fit rather nicely with my idea of a playlist, so I decided to start off with that codebase and modify it from there. It also uses the handy LocalStorage driver, which implements an HTML5 specification for storing structured data on the client side.
I also needed a simple way to query songs, and since the Rdio API is Oauth based, which can get complicated (and probably more so in javascript taking security into account) I found a nice API from echonest that turns out to have a lot of nice functionality thats easy to use. This is what I’m using to query for the songs.
I am using the Rdio playback API to actually play the songs from the Rdio service, and that part is based largely on their hello web playback example. Their API does allow you to build playlists in their service, which is probably the more correct way to do this since you’d be able to access them in their web UI, but since that adds complexity thats not used in this version.
So check out playlister, let me know what you think. There’s a lot of room for improvement, especially with the markup and how some of the pieces fit together. All of the backbone code is in one file and that should be remedied. It could also be made to sync to a backend instead of localstorage and save the playlist into Rdio, and things like that. When you download it, just plug in your API keys into the js/token.js and you’re pretty much ready to go.
]]>We also open sourced the tool we use to deploy code over 30 times a day here on github
If you’re interested, here are the slides from our talk:
]]>Turns out thats a lot of repetitive work that can easily be scripted. You can create a really simple script called ssh-to and place in your ~/bin or somewhere in your path
#!/bin/bash
hostname=`basename $0`
# set screen title to short hostname
echo -n -e "\033k${hostname%%.*}\033\134"
# ssh to the server with any additional args
ssh -X $hostname $*
# set hostname back when session is done (-s on osx)
echo -n -e "\033k`hostname -s`\033\134"
Now, create symbolic links to the script with each server hostname that you use. This can be a little tedious but you only have to do it once and the benefit is that you can now tab complete ssh’ing into servers.
For example you would do something like
ln -s ssh-to myserver.com
Then anytime you type “myserver.com” the tab completion can fill it out, you’re ssh’d into the server (assuming you have keys setup you don’t have to type a password) and your screen window title is updated with the domain info trimmed off, in this case myserver. Now your screen windows update their own titles anytime you ssh into a new server!
]]>The gist service on github is actually fairly convenient especially if you have a github account since it keeps an archive of all of your previous gists. It also does a good job formatting and lets you paste privately. We’ll use the App::Nopaste tool to paste a gist straight from the command line.
First off install the tool from cpan
$ cpan App::Nopaste
You can use this tool anonymously but if you want to keep an archive of your pastes you can simply setup your git credentials in your .gitconfig file. The token here is your API token which can be found under your account settings page.
[github]
user = jgoulah
token = 00000000000000000000000000000000
You also need either Git installed or Config::INI::Reader to allow the module to read your .gitconfig file.
Now to make this a bit easier to remember we can create an alias in our .bashrc file. In this case I’m specifying –private so that only people that I give this secure URL out to can see, and I’m also specifying to use the Gist service. The nopaste app supports a variety of other services that you can use but as of this writing Gist is the only one that supports the –private flag.
alias gist='nopaste --private --service Gist'
Now you can use the command to paste something such as a script from the command line and the gist URL is returned
$ gist somescript.sh http://gist.github.com/df552bacbbbb8d70c089
There you have it!
]]>In this article I’m going to demonstrate how to use the Twitter Streaming API to send a stream of status updates for real time display in a web browser using Web Sockets. We’ll implement a backend server module that streams the events over a web socket using the node.websocket.js framework, which provides a simple realtime HTTP server that implements the web socket protocol. Node.websocket.js is built on top of Node.js which provides an Asynchronous API using callbacks. Node.js is a server side javascript library that enforces an event driven style of programming which allows you to develop non-blocking code easily. This in turn allows you to write simple servers that are very CPU and Memory efficient because you don’t have multiple threads taking up shared resources. Node.js is implemented using Google’s V8 javascript engine and also the CommonJS specification which is defining a standard library for server side javascript.
First things first, you need node.js, which you can get from github. Install in the usual way
$ git clone git://github.com/ry/node.git $ cd node $ ./configure $ make $ sudo make install
Then we’ll need node.websocket.js which you can also get from github
$ git clone git://github.com/Guille/node.websocket.js.git
This is basically an experimental implementation of the web socket API. You can create simple server side modules and then a client side implementation that opens up a web socket to the server in which you can then exchange data in both directions. There are a few examples included that you can take a look at including a simple echo server and a chat server. Just fire up the server like so
$ node runserver.js
and then open up one of the html files in the test/websocket/ directory in your browser. One catch though, you’ll need a current browser such as Google Chrome. I’d recommend it anyway, as browsing the web with it does run a bit smoother.
You’ll also need curl, which is pretty common on any linux box these days and can be installed through your package manager.
The way we’re going to implement the server is to use the curl command to pull from the twitter stream into a file. Twitter gives you a bunch of JSON objects back which you can then parse and display. We’ll use this file in a moment to send the data over the web socket to the browser. There is some documentation on what you can do with the API but we’ll keep it simple and search for any tweets with ‘nyc’ in them
$ curl -dtrack=nyc http://stream.twitter.com/1/statuses/filter.json -uUSERNAME:PASSWORD > sample.json
Now its time to write some server side javascript. In the node.websocket.js checkout there is a modules/ directory. We’ll create a new module called gardenhose.js. So the full path is node.websocket.js/modules/gardenhose.js
In a nutshell this is waiting for the client to establish a connection, and then it creates a child process that tails our file from the curl command above. Anytime the file is written to the “output” listener is invoked, which runs our callback to parse the JSON into objects that we can then use to send a string back to the client with some readable information from the stream.
Lets break it down just a bit more in case you are not familiar with Node. First we are requiring the system and filesystem modules from Node. Now, node.websocket.js basically just uses the node API to implement a server in the websocket.js file. It looks at the request header to see which module to instantiate and then invokes your onData() method when a client sends over data.
Therefore the onData method is the one we need to implement in our module above. We’ll use the process object in Node to create a child process that emits an event called “output” each time the child sends data to stdout. So the addListener call sets up a callback that will be invoked when our file receives more data from the twitter stream. That data comes in the form of JSON objects, one per line. So we split on the lines to create an array of JSON objects, and loop through them. Each time through the loop we’re sending this data back to the client, which is the web browser.
Just make sure the file you pass in is the correct path to the file you are outputting to from the curl command in the monitor_file() function. Then to run the node.websocket.js server you can just invoke it like so
$ node runserver.js
However if you are running the server from another host, you may want to listen on more than just the default of localhost
$ node runserver.js --host=0.0.0.0
The goal here is to get the realtime tweets pumping through our web browser so our client will just be a web page with a little bit of web socket javascript.
This is probably a bit more straightforward. We’re just implementing a few of the functions from the Websocket interface. First we’re instantiating the WebSocket class with the hostname and port that we’re running the server on. The gardenhose in the path is to tell our server that we want to run the gardenhose.js module that we wrote above. The onopen() function is invoked when the socket is opened, and we send over the word “start” which if you recall from above understands the client has connected and to start the child process that runs tail on our file. The onmessage() function is invoked anytime the server is sending data over the web socket, which is the information we want to show on the page, so we append it to the HTML of our hose div. If the server closes the socket then onclose() is invoked, and we display that on the page.
We’ve written a server side module using Node.js and the node.websocket.js framework that will send tweets over a web socket connection. We have also looked at the WebSocket API and learned how to implement some of the functions defined by its interface. Of course, you could use JQuery or your favorite javascript libraries to enhance how this looks to the user, but the basics are all here in how the communication of a real time display can work with web sockets.
Async I/O – http://en.wikipedia.org/wiki/Asynchronous_I/O
CommonJS – http://www.commonjs.org/ and http://wiki.commonjs.org/wiki/CommonJS
V8 – http://code.google.com/p/v8
Node.JS – http://nodejs.org
Node.Websocket.JS – http://github.com/guille/node.websocket.js
Twitter Streaming API (Gardenhose) – http://apiwiki.twitter.com/Streaming-API-Documentation
Websocket API – http://dev.w3.org/html5/websockets
]]>So grab gitosis
git clone git://eagain.net/gitosis
Apply this patch to set an environment variable for the user that is doing a push
diff --git a/gitosis/serve.py b/gitosis/serve.py
index d83b1d8..a6a49a2 100644
--- a/gitosis/serve.py
+++ b/gitosis/serve.py
@@ -200,6 +200,7 @@ class Main(app.App):
sys.exit(1)
main_log.debug('Serving %s', newcmd)
+ os.putenv('USER', user)
os.execvp('git', ['git', 'shell', '-c', newcmd])
main_log.error('Cannot execute git-shell.')
sys.exit(1)
Then install gitosis with that change
$ sudo ./setup.py install
Now, edit the /usr/local/bin/git-post-receive-email script to change the subject line to make more sense
--- /usr/local/bin/git-post-receive-email.old 2010-02-06 19:29:35.000000000 +0000
+++ /usr/local/bin/git-post-receive-email 2010-02-06 19:14:03.000000000 +0000
@@ -186,7 +186,7 @@
# Generate header
cat <<-EOF
To: $recipients
- Subject: ${emailprefix} $refname_type, $short_refname, ${change_type}d. $describe
+ Subject: ${emailprefix} push from ${USER}- $refname_type, $short_refname, ${change_type}d.
X-Git-Refname: $refname
X-Git-Reftype: $refname_type
X-Git-Oldrev: $oldrev
If you haven't setup email already make sure that you have the hook installed by making a symbolic link in /home/git/repositories/myrepo.git/hooks
ln -s /usr/local/bin/git-post-receive-email post-receive
And now your subject changes from this
[GIT] branch, master, updated. d61c9b93446a13093ef7f510a588c69a606ff762
to this
[GIT] push from jgoulah- branch, master, updated.
Now we know who is pushing code by glancing at the subject line. The username comes from the filenames in gitosis-admin/keydir/. Obviously you could style the subject line however you see fit.
]]>With more and more traffic pouring into websites, it has become necessary to come up with creative ways to parse and analyze large data sets. One of the popular ways to do that lately is using MapReduce which is a framework used across distributed systems to help make sense of large data sets. There are lots of implementations of the map/reduce framework but an easy way to get started is by using MongoDB. MongoDB is a scalable and high performant document oriented database. It has replication, sharding, and mapreduce all built in which makes it easy to scale horizontally.
For this article we’ll look a common use case of map/reduce which is to help analyze your apache logs. Since there is no set schema in a document oriented database, its a good fit for log files since its fairly easy to import arbitrary data. We’ll look at getting the data into a format that mongo can import, and writing a map/reduce algorithm to make sense of some of the data.
Mongo is easy to install with detailed documentation here. In a nutshell you can do
$ mkdir -p /data/db $ curl -O http://downloads.mongodb.org/osx/mongodb-osx-i386-latest.tgz $ tar xzf mongodb-osx-i386-latest.tgz
At this point its not a bad idea to put this directory somewhere like /opt and adding its bin directory to your path. That way instead of
./mongodb-xxxxxxx/bin/mongod &
You can just do
mongodb &
In any case, start up the daemon one of those two ways depending how you set it up.
Apache access files can vary in the information reported. The log format is easy to change with the LogFormat directive which is documented here. In any case these logs that I’m working with are not out of the box apache format. They look something like this
Jan 18 17:20:26 web4 logger: networks-www-v2 84.58.8.36 [18/Jan/2010:17:20:26 -0500] "GET /javascript/2010-01-07-15-46-41/97fec578b695157cbccf12bfd647dcfa.js HTTP/1.1" 200 33445 "http://www.channelfrederator.com/hangover/episode/HNG_20100101/cuddlesticks-cartoon-hangover-4" "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7" www.channelfrederator.com 35119
We want to take this raw log and convert it to a JSON structure for importing into mongo, which I wrote a simple perl script to iterate through the log and parse it into sensible fields using a regular expression
#!/usr/bin/perl
use strict;
use warnings;
my $logfile = "/logs/httpd/remote_www_access_log";
open(LOGFH, "$logfile");
foreach my $logentry (<LOGFH>) {
chomp($logentry); # remove the newline
$logentry =~ m/(\w+\s\w+\s\w+:\w+:\w+)\s #date
(\w+)\slogger:\s # server host
([^\s]+)\s # vhost logger
(?:unknown,\s)?(-|(?:\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3},?\s?)*)\s #ip
\[(.*)\]\s # date again
\"(.*?)\"\s # request
(\d+)\s #status
([\d-]+)\s # bytes sent
\"(.*?)\"\s # referrer
\"(.*?)\"\s # user agent
([\w.-]+)\s? # domain name
(\d+)? # time to server (ms)
/x;
print <<JSON;
{"date": "$5", "webserver": "$2", "logger": "$3", "ip": "$4", "request": "$6", "status": "$7", "bytes_sent": "$8", "referrer": "$9", "user_agent": "$10", "domain_name": "$11", "time_to_serve": "$12"}
JSON
}
Again my regular expression probably won’t quite work on your logs, though you may be able to take bits and pieces of what I’ve documented above to use for your script. On my logs that script outputs a bunch of lines that look like this
{"date": "18/Jan/2010:17:20:26 -0500", "webserver": "web4", "logger": "networks-www-v2", "ip": "84.58.8.36", "request": "GET /javascript/2010-01-07-15-46-41/97fec578b695157cbccf12bfd647dcfa.js HTTP/1.1", "status": "200", "bytes_sent": "33445", "referrer": "http://www.channelfrederator.com/hangover/episode/HNG_20100101/cuddlesticks-cartoon-hangover-4", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7", "domain_name": "www.channelfrederator.com", "time_to_serve": "35119"}
And we can then import that directly to MongoDB. The creates a collection called weblogs in the logs database, from the file that has the output of the above JSON generator script
$ mongoimport --type json -d logs -c weblogs --file weblogs.json
We can also take a look at them and verify they loaded by running the find command, which dumps out 10 rows by default
$ mongo > use logs; switched to db logs > db.weblogs.find()
So for this example what I am looking for is how many hits are going to each domain. My servers handle a bunch of different domain names and that is one thing I’m outputting into my logs that is easy to examine
The basic command line interface to MondoDB is a kind of javascript interpreter, and the mongo backend takes javascript implementations of the map and reduce functions. We can type these directly into the mongo console. The map function must emit a key/value pair and in this example it will output a ’1′ each time a domain is found
> map = "function() { emit(this.domain_name, {count: 1}); }"
And so basically what comes out of this is a key for each domain with a set of counts, something like
{"www.something.com", [{count: 1}, {count: 1}, {count: 1}, {count: 1}]}
This is send to the reduce function, which sums up all those counts for each domain
> reduce = "function(key, values) { var sum = 0; values.forEach(function(f) { sum += f.count; }); return {count: sum}; };"
Now that you’ve setup map and reduce functions, you can call mapreduce on the collection
> results = db.weblogs.mapReduce(map, reduce)
...
>results
{
"result" : "tmp.mr.mapreduce_1263861252_3",
"timeMillis" : 9034,
"counts" : {
"input" : 1159355,
"emit" : 1159355,
"output" : 92
},
"ok" : 1,
}
This gives us a bit of information about the map/reduce operation itself. We see that we inputted a set of data and emmitted once per item in that set. And we reduced down to 92 domain with a count for each. A result collection is given, and we can print it out
> db.tmp.mr.mapreduce_1263861252_3.find() > it
You can type the ‘it’ operator to page through multiple pages of data. Or you can print it all at once like so
> db.tmp.mr.mapreduce_1263861252_3.find().forEach( function(x) { print(tojson(x));});
{ "_id" : "barelydigital.com", "value" : { "count" : 342888 } }
{ "_id" : "barelypolitical.com", "value" : { "count" : 875217 } }
{ "_id" : "www.fastlanedaily.com", "value" : { "count" : 998360 } }
{ "_id" : "www.threadbanger.com", "value" : { "count" : 331937 } }
Nice, we have our data aggregated and the answer to our initial problem.
As usual CPAN comes to the rescue with a MongoDB driver that we can use to interface with our database in a scripted fashion. The mongo guys have also done a great job of supporting it in a bunch of other languages which makes it easy to interface with if you are using more than just perl
#!/bin/perl
use MongoDB;
use Data::Dumper;
use strict;
use warnings;
my $conn = MongoDB::Connection->new("host" => "localhost", "port" => 27017);
my $db = $conn->get_database("logs");
my $map = "function() { emit(this.domain_name, {count: 1}); }";
my $reduce = "function(key, values) { var sum = 0; values.forEach(function(f) { sum += f.count; }); return {count: sum}; }";
my $idx = Tie::IxHash->new(mapreduce => 'weblogs', 'map' => $map, reduce => $reduce);
my $result = $db->run_command($idx);
print Dumper($result);
my $res_coll = $result->{'result'};
print "result collection is $res_coll\n";
my $collection = $db->get_collection($res_coll);
my $cursor = $collection->query({ }, { limit => 10 });
while (my $object = $cursor->next) {
print Dumper($object);
}
The script is straightforward. Its just using the documented perl interface to make a run_command call to mongo, which passes the map and reduce javascript functions in. It then prints the results similar to how we did on the command line earlier.
We’ve gone over a very simple example of how MapReduce can work for you. There are lots of other ways you can put it to good use such as distributed sorting and searching, document clustering, and machine learning. We also took a look at MongoDB which has great uses for schema-less data. It makes it easy to scale since it has built in replication and sharding capabilities. Now you can put map/reduce to work on your logs and find a ton of information you couldn’t easily get before.
http://github.com/mongodb/mongo
http://www.mongodb.org/display/DOCS/MapReduce
http://apirate.posterous.com/visualizing-log-files-with-mongodb-mapreduce
http://search.cpan.org/~kristina/MongoDB-0.27/
http://kylebanker.com/blog/2009/12/mongodb-map-reduce-basics/
Amazon recently released a new service that makes it easier set up, operate, and scale a relational database in the cloud called Amazon Relational Database Service. The service, based on MySQL for now, has its pluses and minuses and you should decide whether it fits your needs. The advantages are that it has an automated backup system that lets you restore to any point within the last 5 minutes and also allows you to easily take a snapshot at any time. It is also very easy to “scale up” your box. This is more in the realm of vertical scaling but if you find you are hitting limits you can upgrade to a more powerful server with little to no effort. It also gives you monitoring via Amazon Cloudwatch and automatically patches your database during your self defined maintenance windows. The downfalls are that you don’t have access directly to the box itself, so you can’t ssh in. You also at this point cannot use replication for a master-slave style setup. Amazon promises to have more high availability options forthcoming. Since you can’t ssh in, you adjust mysql parameters via their db parameter group API. I’ll go over an example of this.
The first thing to do is install the RDS command line API, which you can grab from here. I’m not going over the details of setting this up. Its basically as simple as putting it into your path and Amazon has plenty of documentation on this.
Once you have the command line tools setup you can create a database like so
rds-create-db-instance \
mydb-instance \
--allocated-storage 20 \
--db-instance-class db.m1.small \
--engine MySQL5.1 \
--master-username masteruser \
--master-user-password mypass \
--db-name mydb --headers \
--preferred-maintenance-window 'Mon:06:15-Mon:10:15' \
--preferred-backup-window '10:15-12:15' \
--backup-retention-period 1 \
--availability-zone us-east-1c
This should be fairly self explanatory. I’m creating an instance called mydb-instance. The master (basically root) user is called masteruser with password mypass. It also creates an initial database called mydb. You can add more databases and permissions later. This also sets up the maintenance and backup windows which are required, and defined in UTC. The backup retention period is how long it holds on to my backups, which I’ve defined as 1 day. If you set this to 0 it will disable the automated backups entirely which is not advised.
The next thing to do is setup your security groups so that your EC2 (or your hosted servers) have access to your database. There is good documentation on this so I will go over a basic use case.
rds-authorize-db-security-group-ingress \
default \
--ec2-security-group-name webnode \
--ec2-security-group-owner-id XXXXXXXXXXXX
In the case above I’m creating a security group called default, that allows my ec2 security group webnode access. The group-owner-id parameter is your AWS account id.
You can find what your database DNS name is via the rds-describe-db-instances command.
rds-describe-db-instances
DBINSTANCE mydb-instance 2009-11-06T02:19:59.160Z db.m1.small mysql5.1 20 masteruser available mydb-instance.cvjb75qirgzk.us-east-1.rds.amazonaws.com 3306 us-east-1d 1
SECGROUP default active
PARAMGRP default.mysql5.1 in-sync
So we can see our hostname is mydb-instance.cvjb75qirgzk.us-east-1.rds.amazonaws.com
Now you can login to your instance in the usual way that you access mysql on the command line, setup your users and import your database in the usual way.
mysql -u masteruser -h mydb-instance.cvjb75qirgzk.us-east-1.rds.amazonaws.com -pmypass
As I mentioned earlier you don’t have access to ssh into the instance, so you need to use db parameter groups to tweak your configuration rather than editing the my.cnf file. You can’t see the mysql slow query log on the box but there is still a way to access it and I’ll go over that process.
Amazon won’t let you edit the default group, so the first thing to do is create a parameter group to define your custom parameters.
rds-create-db-parameter-group my-custom --description='My Custom DB Param Group' --engine=MySQL5.1
Then set the parameter to turn the query log on
rds-modify-db-parameter-group my-custom --parameters="name=slow_query_log, value=ON, method=immediate"
We’re still using the default configuration so you have to tell the instance to use your custom parameter group
rds-modify-db-instance mydb-instance --db-parameter-group-name=my-custom
The first time you apply a new custom group you have to reboot the instance, as pending-reboot here indicates
$ rds-describe-db-instances
DBINSTANCE mydb-instance 2009-11-06T02:19:59.160Z db.m1.small mysql5.1 20 masteruser available mydb-instance.cvjb75qirgzk.us-east-1.rds.amazonaws.com 3306 us-east-1d 1
SECGROUP default active
PARAMGRP my-custom pending-reboot
So we can reboot it immediately like so
$ rds-reboot-db-instance mydb-instance
When it comes back up it will show that its in-sync
$ rds-describe-db-instances
DBINSTANCE mydb-instance 2009-11-06T02:19:59.160Z db.m1.small mysql5.1 20 masteruser available mydb-instance.cvjb75qirgzk.us-east-1.rds.amazonaws.com 3306 us-east-1d 1
SECGROUP default active
PARAMGRP my-custom in-sync
We can login to the instance and see that our parameter was set correctly
mysql> show global variables like 'log_slow_queries'; +------------------+-------+ | Variable_name | Value | +------------------+-------+ | log_slow_queries | ON | +------------------+-------+ 1 row in set (0.00 sec)
Since you don’t have access to the filesystem, its logged to a table on the mysql database
mysql> use mysql; mysql> describe slow_log; +----------------+--------------+------+-----+-------------------+-----------------------------+ | Field | Type | Null | Key | Default | Extra | +----------------+--------------+------+-----+-------------------+-----------------------------+ | start_time | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP | | user_host | mediumtext | NO | | NULL | | | query_time | time | NO | | NULL | | | lock_time | time | NO | | NULL | | | rows_sent | int(11) | NO | | NULL | | | rows_examined | int(11) | NO | | NULL | | | db | varchar(512) | NO | | NULL | | | last_insert_id | int(11) | NO | | NULL | | | insert_id | int(11) | NO | | NULL | | | server_id | int(11) | NO | | NULL | | | sql_text | mediumtext | NO | | NULL | | +----------------+--------------+------+-----+-------------------+-----------------------------+ 11 rows in set (0.11 sec)
We may also want to set things like the slow query time, since the default of 10 is pretty high
$ rds-modify-db-parameter-group my-custom --parameters="name=long_query_time, value=3, method=immediate"
The rds-describe-events command keeps a log of what you’ve been doing
$ rds-describe-events db-instance 2009-12-12T17:44:19.546Z mydb-instance Updated to use a DBParameterGroup my-custom db-instance 2009-12-12T17:45:51.636Z mydb-instance Database instance shutdown db-instance 2009-12-12T17:46:09.380Z mydb-instance Database instance restarted db-parameter-group 2009-12-12T17:56:02.568Z my-custom Updated parameter long_query_time to 3 with apply method immediate
And again you can check mysql that your parameter was edited properly. Note how this time we didn’t have to reboot anything as our parameter group is already active on this instance
mysql> show global variables like 'long_query_time'; +-----------------+----------+ | Variable_name | Value | +-----------------+----------+ | long_query_time | 3.000000 | +-----------------+----------+ 1 row in set (0.00 sec)
In this article we went over some basics of Amazon RDS and why you may or may not want to use it. If you are just starting out its a really easy way to get a working mysql setup going. However if you are porting from an architecture with multiple slaves or other HA options this may not be for you just yet. We also went over some basic use cases on how to tweak parameters since there is no command line access to the box itself. There is command line access to mysql though, so you can use your favorite tools there.
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2935&categoryID=295
]]>