Jul 17 2016

Provisioning Docker Containers With Chef

I’ll be totally honest in saying even though the Dockerfile is the supported way to build docker images, I have not quite bought into building my containers using the format. I believe the main thing we are losing with them is a platform agnostic tool to build machine images, which is useful if you don’t always want or need to use the container format but want to keep the same configuration across different types of hosts or environments. Defining infrastructure in code so that we can use logic, templates, and reusable patterns gives us so many more opportunities towards writing intelligent provisioning processes that span across development and production. Given that, I’ve turned to Chef as a tool which can represent my configuration at both the application and machine level.

My original use case was to spin up a Docker container to host our development environment, and then bring this same configuration into production. I’ve recently inherited an environment that didn’t use the concept of containers or really have any reproducible method to spin up new hosts. And even though this is basically what docker does best, we weren’t quite ready to mix containers into production when we were still trying to solidify the ground that it stands on and make small, careful improvements. So the idea was to use chef to provision docker containers for development of the application but also for testing chef itself, and then use those same recipes to build EC2 instances to stand up alongside the current system. The great thing about this is that we are able to run any chef recipe on a container, and test changes before moving them into production.

This chef container provisioning recipe is pretty straightforward and can be run with chef solo (for example, to easily provision a development instance on a laptop):

This gives us a container with the home directory mounted as a volume with correct user permissions, running a role we have defined as Dev containing several recipes. These same exact recipes might be put in a Prod role which is used in production. It doesn’t really matter if production is a container, EC2 instance, or something else like a bare metal server. Using this pattern can lay the groundwork to creating platform agnostic images for our infrastructure that are identical in configuration.

Tags: , ,

Jul 09 2016

Approximate Times to Update an RDS Instance

Category: Cloud Computing,Databases,Systemsjgoulah @ 12:59 PM

Here’s a quick overview from an upgrade a couple months ago of an RDS instance type db.t2.medium to type db.r3.large. In addition to changing the instance type, we upgraded the disk from 64GB to 100GB, and applied a new parameter group. The disk increase by far took the longest amount of time. We clocked in at ever so slightly over an hour for the disk increase, while the instance upgrade only took a bit over 16 minutes. As you can see, AWS also does a failover during the upgrade, but we decided to stop writes to this database instead of risking any data corruption.

production01 6:26:46 AM Applying modification to database instance class
production01 6:33:02 AM Multi-AZ instance failover started
production01 6:33:07 AM DB instance restarted
replica01 6:33:20 AM Streaming replication has stopped.
production01 6:33:35 AM Multi-AZ instance failover completed
replica01 6:34:51 AM DB instance shutdown
replica01 6:35:08 AM DB instance restarted
replica01 6:38:50 AM Replication for the Read Replica resumed
production01 6:42:59 AM Finished applying modification to DB instance class
production01 6:43:02 AM Applying modification to allocated storage
production01 7:39:05 AM Finished applying modification to allocated storage

One thing Amazon could really work on here is the feedback loop. Getting no updates whatsoever other than hoping that the word “modifying” eventually goes away and says “available” is really poor in terms of an experience and keeping confidence the operation is going to be successful after an extremely long time. This is an exercise in patience because your only option is to continue to hit the reload button (nope, it doesn’t reload automatically) and hope that it has moved from yellow and is instead showing green and not red.

In any case, I’m mostly publishing this post because I found these type of run time estimates or actual numbers very hard to find. And they are certainly variable depending on a variety of factors, so this can really be used as just one data point. That being said, I’m hoping it can help someone else that is looking for estimated times when planning their RDS upgrade.

Tags: , , , , , , , ,

Mar 30 2016

Why You Should Point Staging to Your Production Database

Category: Databases,Deployment,Systemsjgoulah @ 9:56 AM

I have been thinking about this topic more recently as I’ve just started working with a new infrastructure, and one of the things that I noticed is the staging database is a separate copy of the production database. It also happens to be a very old copy that has drifted, certainly in data and possibly in schema. It was likely put in place to avoid the perceived problem of new code possibly affecting production data in some critical way, and having a safe and solid environment to examine changes before they do hit production. But what are we accomplishing if the environment we test those changes in doesn’t really provide a fully identical setup to production? When you are dealing with DDL and the data underneath that may not actually mimic production, you’re only gaining a sense of false confidence.

This still sounds crazy, you say. “Of course we shouldn’t point staging to production! What if there is some bug that deletes some important rows from our database? We’d want to catch that before it hits in production!” And to this I wonder why your development environment isn’t an adequate testbed for diagnosing this sort of behavior well before its gone to staging in the first place?

As I thought more about this, it started to sound like the concept of staging itself originated more from an issue of difficult to reproduce environments. If your development environment does not exactly mirror production in terms of its configuration and setup, then it makes sense that you would need some intermediary place that is “closer to production” to test those changes. Hence staging is a place to almost test your changes and almost get that confidence you need, but without the same data underneath. In the end, this gives us no more confidence than until it is run in production itself.

In many ways, the staging environment originates from the waterfall model, and the days of manual QA testing. And this is not to say QA testing has no place, just that staging is not the right environment to perform those types of meticulous and slow, possibly at times manual work within the given time constraints. Unless your development to production cycle is hours or days, then the functionality you’re verifying on staging is only the small incremental changes that you are pushing at that time. You’re more likely and hopefully doing end-to-end testing across the codebase, and this is giving the confidence to move code through the pipeline quickly (in addition to metrics at the other end).

What we really want to test in staging is that our changes show up, the site builds and renders properly, and that we can perform the actions that we expect against the data we expect. If we are worried about rogue deleting data because of bad code, there are additional safeguards that can be taken aside from separating the environment entirely. The first thing is simply to not delete data. This may be a non-trivial change for a large legacy codebase, but it is worth considering. Instead of using the DELETE statement, consider flagging that data as removed and respecting the flag. Additionally, rolling code in percentages, to the people at your company and then small portions of traffic is a great way to gain confidence without sacrificing the integrity of the real purpose of staging.

This is not “testing in production”. This is ensuring your development environment is similar enough to production that staging technically only exists as an intermediary place to test your build. This means staging is the next thing in line for whats running on production and we have enough confidence to test our changes against production data. It makes a lot of sense for it to be allowed to interact on that data rather than a copy that may have a drastically different representation than whats on production. Therefore, it is worth considering pointing your staging database to production.

Tags: , , ,

Mar 21 2016

Configuring Chef for Provisioning

Category: Cloud Computing,Configuration Managementjgoulah @ 10:23 PM

If you’re working with infrastructure its good practice to describe it using code so that it is reproducible and consistent across servers and development environments. I’ve used Chef for quite some time and feel it is a pretty natural way to represent the source of truth for your servers, the packages installed on them, and their configuration. Chef can also be used as a provisioning tool, to bring your servers to life configured exactly to your specifications. You can use it with services like AWS or tools like Docker.

I started out using chef local mode to test my provisioning recipes, but also wanted to get things working with chef-client running as a daemon. But because of ACL’s in place when Chef is run this way, you need to grant permissions to the right groups to make sure they can do things such as create nodes.

This is hinted at with an error that looks like:

This error: Net::HTTPServerException: 403 "Forbidden"

or if you go digging with debug mode on (chef-client -l debug), you might see something analogous to this buried in a ton of output:

[2016-03-21T10:13:05-04:00] DEBUG: ---- HTTP Response Body ----
[2016-03-21T10:13:05-04:00] DEBUG: {"error":["missing update permission"]}

The default Chef ACL’s don’t allow nodes’ API clients to modify other nodes, and so we have to create a group with such permissions that your provisioning node (the one that kicks off the new instance/machine to be provisioned) can create the machines’ nodes and clients. This is similarly explained in this slightly outdated post here but unfortunately the commands aren’t quite right, so here it is using the most current version of the tooling.

Setting up Permissions

First things first install the ACL gem (assuming you’re using chef development kit)

chef gem install knife-acl

We can then create a group to give access to the permissions we need:

knife group create provisioners

Now, if you’re setting up a new node to be your provisioner, you would create the client key and node object:

knife client create -d chefconf-provisioner > ~/.chef/chefconf-provisioner.pem
knife node create -d chefconf-provisioner

Or you may already have a client that you run chef-client from. Lets say that is called chefconf-provisioner as it is the client we created above, so we’ll go with that, but your client can be named anything. Note, its usually the hostname of the node you’re running from. Add your client to the group we just created like so:

knife group add client chefconf-provisioner provisioners

Chef server uses role-based access control (RBAC) to restrict access to objects—nodes, environments, roles, data bags, cookbooks, and so on. This ensures that only authorized user and/or chef-client requests to the Chef server are allowed.

In this case we need to grant read/create/update/grant/delete permissions for clients and nodes so that our provisioning node can create the new instance/machine:

for permission in read create update grant delete
  knife acl add group provisioners containers clients $permission 
for permission in read create update grant delete
  knife acl add group provisioners containers nodes $permission 

And now you should have the permissions to be able to provision new nodes using Chef!

Tags: , , , ,

Mar 07 2016

Running Strace in Docker

Category: Cloud Computing,Containers,Kerneljgoulah @ 10:40 PM

I’ve been reverse engineering a new application setup and seemed like an appropriate place to try out docker. Spinning up a lightweight and reproducible environment is the goal and containerization is a reasonably efficient way to accomplish that. As I was looking into a problem with getting some services running properly, with little debug output and sparse documentation, I reached for the old trusty strace to see what was going on. But what do you know, strace is disabled by default on Docker. Here is the error that I got:

strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Operation not permitted
strace: test_ptrace_setoptions_for_all: unexpected exit status 1

This is admittedly an error I hadn’t seen before, and google isn’t a ton of help on this one. As a newbie with docker, it would have been helpful to have a bit more detailed error message as to why such a common tool as strace isn’t working.

Luckily some IRC logs come to the rescue in my quest through WTFed’ness. I learned that the security around this feature has apparently evolved a bit over time, with apparmor being the older but still used security system, and secconf being only available on newer distros (and OSX running boot2docker). Confusing things further, some of the articles out there are referring to apparmor (which uses different methods for modifying its security policy).

If you are using secconf, there are a couple of things you can pass to docker run to loosen up this security policy. To allow strace specifically, you enable the system call that it relies upon to get its information (ptrace):

--cap-add SYS_PTRACE

This whole paradigm is in fact documented but none of my original searches turned up these pages. In addition to disabling ptrace, there are a slew of other system level commands that you may (or may not) need that aren’t on the docker whitelist of allowed system calls. The list of calls can be adjusted very granularly by feeding docker a json file defining your security options. Or if you are feeling up for it, you can re-enable all of them in one fell swoop with this option to docker run:

--security-opt seccomp:unconfined

Its definitely worth considering which system calls your container really needs access to though, and strace is one of those that is quite useful for debugging purposes. There will always be that balance between security and usability, and decisions to make on which direction to swing the pendulum. It’s nice to see that this kind of functionality is being supported by docker to give the container very granular access to system level calls, and it might be interesting to think about ways it could be highlighted to a surprised enduser.

Tags: , , , ,

Feb 23 2015

Go Deployment

Category: Deployment,Golang,Organizationjgoulah @ 12:06 AM


In the spirit of the recent re-release of Deployinator, it seems timely to write a little bit about deployment. However, this is not a how-to on writing a Deployinator stack, but an overview of the high-level mechanics of getting a golang app deployed into production. Its worth mentioning that this methodology is just one of many ways you might go about deploying your app, but this is what has worked for me. These are the fundamental pieces that you’ll need to think through no matter what deploy tool you end up using.

Whats to Deploy?

When you run go build it produces a binary executable that can easily be shipped around. But there are actually several key pieces to getting this binary deployed into production. The code you have must be able to compile against the libraries you’re using, and if you are deploying a web based app, you’ll need to think about where your asset files are deployed and how they’re referenced from the go binary. The configuration of the application is also important in that you’ll want to be able to have the app run under different environments, such as development and production. Lastly, you’ll need to think about how to control the app, so that you can easily and consistently restart it when you deploy.


Typically when doing production deployments there exists a build server where the application is compiled and the pieces are put in place for shipping to production. Certainly no production facing app should be getting deployed from a developers virtual machine, but the process for building the app should be identical. Building a go app is fairly straightforward but it can be a good idea to organize your steps and common commands in a Makefile. We’ll build our binary into the bin subdirectory:

        go build -o ./bin/myapp myapp.go

This produces the binary that we’ll eventually ship.


This build process works fine in development but how do we know the external dependencies exist on our build machine? How can we maintain this app building process in an identical way across environments? Whereas in development we can just go get our dependencies, it may be haphazardness to maintain things this way on a production build machine, since you might have different go apps being built that use different versions of the same dependency. Or your developers may have pulled down different versions at different points in time. The current versions of github repositories that your code relies on also may introduce bugs that you want to gain confidence doesn’t impact your application. To maintain consistency in dependency versions, and to ensure these same dependencies exist in our build environment, we’ll use a tool called godep.

Assuming you have a proper go toolchain setup, you can get started with godep by getting it and running a single command:

go get github.com/tools/godep
# then in your project dir
godep save -r ./...

This should pull all of your dependencies into a Godeps directory as well as update your files to import this “frozen” copy of the dependencies you are using.

Since we won’t want to stay locked on old versions of software, we need a way to update our projects dependencies from time to time. This is something you’d probably want to add to your Makefile:

        go get -u -v -d $(import)
        godep update $(import)
        godep save -r ./...

Using this command we can update any of our dependencies like so:

make updep import=github.com/codahale/metrics


A config file is a reasonable way to maintain different environments such as development and production. The gcfg library is a perfectly fine way to define ini style configs that map easily to go structs. It will support multiple environments and you can have a file defining your shared values and then define development and production differences in another set of files. Then you can merge the appropriate set of files based on your environment using ReadFileInto. The example below merges a secrets configuration file with the base configuration to show a simple example, but you can extend this technique to support multiple environments.

In this example we make sure the base config exists, based on a location relative to this file. We’re next looking for the secrets config, first in the app directory where we may store it locally for development, and then in an /etc directory for production where a configuration management system may place the file more securely. This way this sensitive file doesn’t have to be stored in the main repository, but can pass production values securely to the application. Note that means this file wouldn’t be deployed in the normal way, and the app will fallback to the secure values in /etc when it doesn’t find the local file. The full configuration example implementation can be found here but remember this is just a very basic example to show how to merge configs.


When you deploy the app to production you’ll have to restart it in some way. The simplest way to do this is to write an init script. But you shouldn’t have to write an init script by hand these days. Check out pleaserun to generate a script for your system. Now you can stop, start, and restart your process whether you are using upstart, systemd, plain old init, or something else. You can go one step further and implement a graceful restart, but the implementation will vary depending on your needs.

Tying it Together

So now we have a way to deal with our dependencies and the build. We know how to control the app. The app knows how to find its configuration files and override base values with production settings. We can use the same runtime.Caller trick that we used to locate the config file to make it easier for the application to find its static files as well.

At this point there are only a few steps that need to happen on a deploy box:

  • Get the newest code
  • git fetch && git reset --hard
  • Build the binary
  • cd $basedir && make build
  • Rsync the code
  • rsync -va --delete --force \
    --include='static' --include='bin' --include='config*.cfg' --include='*/*' --exclude='*' \
    deploy.mydomain.com:/var/myorg/build/golang/src/github.mydomain.com/jgoulah/myapp/ /usr/myorg/myapp
  • Use your init script to restart the app, depending on the system
  • /etc/init.d/myapp restart

In this example the rsync command is run on an application box and pulling from our deploy host. In a large deploy we may dsh out to many app hosts to rsync our build over in a pull based fashion. The /var/myorg/build directory is where our make build command was run to produce our binaries. We sync the bin directory that contains the binaries, the static directory which has our html/js/css, and the config files. Lastly we restart our app. These steps can be coded into a Deployinator stack or with the deploy tool of your choice. But regardless of the tool we use, we will have a repeatable and dependable process with reliable dependencies that we can ship.

Tags: , , , , ,

Jul 06 2014

Dynamically Update Hypervisor Guest Info in Chef With Rehai

Category: Cloud Computing,Configuration Managementjgoulah @ 11:31 AM


It turns out this little tool is long overdue, as simple of a concept as it is, but also easy to misunderstand the use cases for, ours at Etsy however was very targeted. Several years ago we were hammering out our internal cloud infrastructure, using KVM/QEMU based solution that you can read about over here. We were populating our virtual machine frontend using Chef Ohai data as the canonical source of our system information. There is an ohai plugin that gathers KVM virtualization data using virsh commands which are part of libvirt. It was a perfect way to capture information about which guests existed on a given system and other information about them.

Our problem

We were hitting a bottleneck in that our chef clients were setup to run about every ten minutes. But within that ten minutes it would be possible that a virtual machine would be added or deleted, and therefore it was difficult to keep our interface in sync. Imagine creating a new virtual machine but not being able to display data about it until you waited around ten minutes, and to make matters worse these clients run at a splay interval, which means they don’t run all at the same time. Therefore, we started on a simple script that would let us run ohai quickly without needing to do the full chef run. While our chef runs are relatively quick (usually < 1 minute) it would introduce problems if we try to run chef while the client is already running.

Going to open source

It was supposed to be released a while ago, but has taken some time for various reasons. It’s a shockingly tiny amount of code but there were some barriers to releasing it. The majority of the code was written by Mark Roddy but he’d turned it over to me to open source. I went through the normal chef contribution process, which at the time required opening a Jira ticket that you can read if you’re interested in some of the details. In short there were some questions about the use case but when I explained that we weren’t trying to re-invent graphite in some horrible way, we were able to agree there could be some real world use cases. That being said, it was not accepted into the core yet because this does introduce very small race conditions since chef uses a read-modify-save model of changing the attribute data. There is a proposal to fix this, which divides attributes into different levels in which automated updates can access them without causing this issue. However in the wild this has not actually posed an issue for us even with several hundred nodes running it.


If you’re interested in this tool, you can install it with ruby gems using the command:

% gem install rehai

If you’d like to see the source, head on over to github.

Tags: , , , , ,

Jun 22 2014

Stubbing AFNetworking Calls In XCode Unit Tests

Category: iOSjgoulah @ 11:58 AM


A unit test by definition of testing the smallest possible part of the application, should eliminate all dependencies from the systems its interacting with. This in turn will remove any unknown or variable outcomes from the tests, focusing on the isolating the specific code you’re interested in. In the case of networking calls, it is unideal for tests that deal with the (un)marshalling of data to depend on the network. Therefore a stubbing library is needed.

OHHTTPStubs to the Rescue!

Luckily there is a great library to do just what we need here – stubbing out some network calls – called OHHTTPStubs.

Because these tests are asynchronous, the network requests are sent in a separate thread or queue different from the thread used to execute the test case. So you’ll have to ensure that your unit tests wait for the requests to finish and their response arrived before performing the assertions in your test cases. Otherwise your code making the assertions will execute before the request had time to get its response.

OHHTTPStubs includes a custom AsyncSenTestCase subclass of SenTestCase that provides methods like waitForAsyncOperationWithTimeout: and notifyAsyncOperationDone. We can utilize these methods in our test cases.

A Networking Stub

Most people are using the AFNetworking library so this example will show a test using that.

In this case I’ve created a network wrapper that allows turning stubs on and off. The request here happens to be mocking XML data, but JSON follows the same pattern. This is an implementation of a class called GrillNet that has two functions:

The two functions are: stubsOn to activate the stubbed networks calls, and getGrillStatusWithURL which invokes a network request. If we call stubsOn before the getGrillStatusWithURL call is made, it will invoke the stub and return the mock data which I’ve generated and put into the file cyberq.xml.

Notice that getGrillStatusWithURL implements a success and failure callback. Thus I’ve created a protocol to respond to the two scenarios which asks for optional implementations of notifyDone and notifyDoneWithData:data, shown here:

Our tests can implement this protocol to assist with the notifying that our asynchronous operation is done by calling notifyAsyncOperationDone. The class GrillzTests subclasses AsyncSenTestCase and implements GrillNetActionProtocol.

And in this same class we can implement notifyDone and notifyDoneWithData:data. They will both call notifyAsyncOperationDone from AsyncSenTestCase but notifyDone is only called on failure when no data is returned, while notifyDoneWithData:data is invoked when the request is successful .

Therefore it is the job of notifyDoneWithData:data to do something with this data. In this case we are setting the values in a GrillStatus object that was used in the test assertion comparing the value in _gs above.







Tags: , , , , ,

Jun 11 2013

Capturing A/V Multimedia Streams From the Browser

Category: Coding,Real-time Webjgoulah @ 9:29 PM


I’ve recently been working on finishing up an application to help gather presentation feedback when demo’ing a talk for the first time in front of a live audience. The app will play back a video stream and overlay the audiences comments on top of the video in real time. However I haven’t open sourced it yet because there was a dependency on launching an external video application to do the recording, and have the user upload that to the server. I wanted the application to do “in browser” audio/video recording so that it was more seamless with the overall web application.

Turns out A/V recording is still pretty experimental on the web, but is slowly rolling out in major web browsers. The key component to understand when dealing with A/V capture is the getUserMedia API. The getUserMedia API is an in progress API to access audio and video streams. You can find a decent intro on this API here.

I was then wondering how to capture the streams recorded by the API and found this article which covers video only recording. The MediaStreamRecorder API is currently unimplemented in chrome, but there is a nice interface implemented in whammy.js that the author uses to demonstrate recording video. However, we also need sound, and it turns out based on the comments in that article that others are looking for that too.

There is another github project that I found called recorder.js which can help us record the audio stream. It uses the AudioContext interface to export the audio as a blob.

Combining these two ideas, I can capture separate audio and video streams, and so they must be combined. Theoretically this could be done with a javascript encoder, but I wasn’t sure if one exists that fits this use case or how much computing power that would take on the client side. Therefore I chose to encode the streams server side with avconv (aka. ffmpeg). When the recording is completed the client sends HTTP POST requests to the server with the content of each stream. The video is webm format the and audio is wav.

The command to do our transcoding is pretty simple. On the server side after you have received the files from our POST requests, you would do something like this, replacing the input files names with those from the recording:

avconv -i input.wav -i input.webm -acodec copy -vcodec copy output.mkv

And the file output.mkv that is produced will contain the video with your audio stream.


If you’re interested in an example implementation you can find my code here. My next steps are to see if I can dig deeper into WebRTC to send both streams to my server and transcode them on the fly, instead of consuming the resources on the client side to hold the audio and video streams in memory until the video is complete. If you have a better way of accomplishing A/V recording please let me know in the comments!

Tags: , , , ,

Apr 23 2013

Crossing the Production Barrier: Development at Scale

Category: Conferences,Databasesjgoulah @ 5:59 PM

Slides from my talk at Percona 2013


Next Page »

download xiuxiu editor foto shambho shankara mp3 free download pikeno e menor perdicao download download lagu surat at taubat smarthru 4 download pl abbey road 60s drums download mac cell phone repair download download kara winter magic album download intel gma booster terbaru download mp3 tantowi yahya free download farhan ali qadri video naats free download internal medicine harrison download music ragheb alama sinin dhada songs download in ziddu download form 4852 from the irs website free invitation templates download and print download boundless by cynthia hand free la chatimi cantare mp3 download free download of deception point ebook download munni badnam hui free mp3 download jtx party like a rockstar download hide ip ng 1.40 download ooh la la goldfrapp download time after time quietdrive crime and punishment mp3 download sweety gippy mp3 download caminhos da liberdade download minions banana video download reflex flugsimulator download gratis naruto shippuden 208 pt download sniper elite wii download ntsc pirata do espaço download dublado download ptanks full version o justiceiro download pc rip microsoft hda driver download download admit card iti jharkhand 2012 bada 2.0 download wave 2 download gangster life gta apple safari 4 x download business intelligence download oracle download os x dvd installer download gta 8 vice city myegy szybcy i wściekli 5 2011 download download highschool of the dead color download shakira ft pitbull rabiosa zippy download lagu t ara zombie jelly car music download download driver sony vaio 32bit klezmer music to download free download of shreenathji bhajan aga bai arechya download serie v 3 temporada download download the jeremy kyle show download jogos fazenda gratis pc minimizer download para mu susana nothing at all download download um novo vencedor damares playback download command line mac os download paypal jar for android download zeljko vasic zanjisi kukovima marian keyes watermelon ebook download internet download manager flurry icon kruti dev free download windows download chief keef choppa go bang download msi 3.1 windows installer chessmaster free download with crack bravo hits 98 download download disrespect kirko bangz x plane 6 demo download la baby jonas brothers mp3 download netgear ga311 windows 7 download free download habib painter mp3 download tweetdeck desktop windows 7 ekhon ami mp3 download promethean the created download pdf how to download youtube videos to ipad amazon download games steam cypress hill download 2011 download treu nha hang xom 2 download kick out the epic mother wooh da kid stepped download x2 x men united download imaginasamba perfeição download download navigation for mobile zor ka jhatka song download mp3 download audi a4 owners manual download pretty little liars s01e05 avi download nitro circus season 1 download eega promotional song download the simulator 2012 demo x264 codec download mac download lições para toda vida legendado download mkisofs for ubuntu download star trek voyager scorpion lil b 855 download download pokemon black 2 jap rom spells aprilynne pike pdf download ita download ways of reading cursed crusader trainer download download darmowe gry dla dzieci download spyglass for android alaa wardi 7aram free download massive attack teardrop song download download settings for nokia x6 download account opening form obc download fl studio on a mac worth dying for download s bot download free silkroad 1 click downloader download 5 ishq ka sheen download download melodia que eu conheço stephen king novels download free 100 download psp games for free a escolha download dublado ptgui 9 mac download kz hack download gratis download driver yamaha psr 3000 download macroeconomics policy and practice download hp dv1000 sound driver download famous five movies download phim benh nhan nguoi anh download gta 1 for free download gangs of wasseypur movie free download panasonic sd jukebox software download falling up drake download trackmania sunrise extreme full version free download de simuladores de combate aereo download i bruise easily download pioneer dj software free download noah and the whale life goes on mp3 download recover deleted files software download sơ đồ kế toán download free desk phone ringtone fußball manager 12 download vollversion kostenlos even greater mp3 download planetshakers download snmp for windows 2000 free download amuse park game bejeweled 3 jar download sleeping at last quicksand download download ứng dụng cho nokia n8 coral player download luna download jill scott whenever you're around download fairy tail games for pc download photoshop 8 cs me myegy download yahoo latest version for free mp7 player free download mac download internet explorer 8 vista java hry download 128x160 download jason upton key of david download march of the wooden soldiers download barad toro be dast avordam mp3 download songs of aitraaz from songs pk com 6.72 f ai nightmare download download tiny toon adventures nes download i'm yours mp3 download 9 hours rom pipi player download vista deskjet d2360 software download download booster pack hack abhas ha mp3 song download mp3 songs download 3gp download cod 4 3rd person mod trey songz blind download free download hawaii 5 0 season 2 kenji free download fort minor download clubbed to death 2 quebrando regras 2 download portugues chamas da vingança 1984 download download sketchup 8 deutsch virtual router manager download xp download instalador chrome offline download soundgarden live to rise download canon mx410 printer driver download the harold song kesha mp3 garmin 255w download maps free download mise a jour mcafee download visio windows 7 64 bit download gen psp 3000 quasi amici film download gratis ita download plano de fuga rmvb legendado download alfonso loher name in the sky download mp3 zahra damariva alasan download j rock songs download bihar secretariat assistant admit card download project 64 64 bits download naruto chapter 589 cartoon download for ipad download tributo a bezerra da silva download roda a roda jogo cx one full download armin van buuren rapture download mp3 download nein mann video download tambor de funk download rota de fuga rmvb free download of shawty got moves mp3 download cm7 for droid 2 global download executive resume format amnesia game download mac crash bandicoot mutant island download pc ipod touch download pictures to computer download tu pirata soy yo chayanne download lg pc suite p990 harmor vst plugin download download abaqus 6.10 student edition i like cereal song download filmes alta resolucao download italian lessons download mp3 mass effect 3 download pirate download manager idm key download wh cs 2011 bau simulator download ita download efek suara unik download girl talk ultraviolet sound download bicara hati episod 4 annie khalid songs download download driver epson cx5500 free quake 3 download bots infinity blade 2 ipa download crack pk songs download list dream the game download free cydia download step by step motorola xoom rom download download let's go ricky luna remix zune manual download pdf download hit and run 2012 dvdrip download do sapo videos download do jogo cities xl 2011 download song socha na tha by alamgir ktechlab for ubuntu download download vara rece kamelia zippy download schenk mir dein herz gipsy kings volare download free welsh flag to download download afinador do cifra club 3 gatsu 9 ka download free mp3 angry birds android download download manager error the server returned an error download sdo x season 2 song download paint shop pro download turn to u justin bieber download original ruu for evo download maps to print boys over flowers download songs download free regular show episodes download yamaha psr 1500 styles satyamev jayate download song download mouse fix for windows xp download elliot in the morning kenapa tidak bisa download film h.p f4200 printer software download p square game over download mp3 carpet 3d max download download mw 3 1.07 patch the legend of zelda download snes rom download 9 temporada friends download on my freebox ne fonctionne pas download tito lopez the blues sharebeast download sweeney todd final scene papago x8.5 wince 5 download download monkey for rhino download na paz de jah download gratuito adobe reader 8 ngo accounting software download download onto mp3 from youtube somebody's me mp3 download enrique download ocarina of time 1.0 download 64 bit windows tax form 8379 download frisky tinie tempa download zippy download pro update psp go 6.60 download lagu chrisye gejolak cinta download themes for gw300 clr via c# richter download download 8195 the damned rar parayathe ariyathe malayalam song download autobiography of abraham lincoln download download phim hiep dao hoa hetaoni english download part 2 cenário de novela download mp3 sandra brown envy download pdf download sims 1 love bed sende ahasa wage mp3 download download voice changing application download feed us 2 free download open source library management system download alana grace black roses red download audi a3 manual download free irctc mobile application stand o food 3 apk download download dss dj effects visual basic software download download film khuda kay liye download intezar remix by falak sri rama rajyam mp3 download download dan seals one friend download virtual families mac free download gossip girl cecily von ziegesar download autoramas fale mal de mim gabin doo uap mp3 download free download love in this club mp3 download dead space 3 demo pc charmed download season 1 nhac chuong theo ten download angry birds season download free pc reign of hunters download download de pokemon flash download full screen theme wordpress download manager 6.05 crack download planta x zumbi download razor ramon entrance music download skype xperia 8 download lagu tercipta untukku ungu download shining inheritance ep 16 trial download microsoft project 2010 halo ce download key thermodynamics 6th cengel download vmware ova converter download download e.r legendado 1 temporada wilfred season one download download hivi mata hati download apun bola mp3 zmierzch księżyc w nowiu download peb cod tool download 1.5 web client get download speed ryback meat mp3 download music download on itunes download beenie man i'm okay vigilante 8 2nd offence download pc download movie 2012 in hindi latha tamil font free download word melhor impossivel download legendado download lloyds tsb bank statement download oki b6300 driver how to download spoutcraft free popup blocker download google chrome download tenth avenue north losing download sound intervention mw2 download jogo harry potter pc download pokemon blanc nds alda célia playback download download shwayze get you home download tower bloxx mobile game dani california official video download download jump out the gym download all killer no filler sum 41 ra one full movie download 2011 download rebelde só pro meu prazer download tees maar khan movie in avi format activex control download install download video setia band stasiun cinta download de temas nokia x2 00 download power geez 2005 computer games download com download benny and babloo songs soc pc camera driver download for xp manually download sophos virus definitions jackie chan adventures download links download dragostea se face in doi download terjemahan kitab al umm hp photosmart c3180 download scanner download ipcop for windows 7 nero 8 download windows 7 64 bit kick buttowski kick in genes download dewana 2013 mp3 download download 2 chainz birthday song free how to download correct video driver download film g 30s pki firing games download full version download free 3d motorbike racing download call of duty 4 zombies download smart mobile themes download crbl romanu n are noroc hotfiles download diggy simmons make you mine download bangla natok bhalobashi tai download kml from my maps download song if this charlie sheen shinda new album download download outcast 1 temporada keterlaluan the potters mp3 download download office 2007 upload download falling skies 2 temporada rmvb download sajan all songs oblivion mod manager download mac download toma o meu coração download pierce the veil caraphernelia mp3 download lagu jkt48 original download leaf by elle varner vandalism coming alive mp3 download download god of war betrayal 240x320 download amanda by zigi mp3 download apostila do trf hp 635 driver download windows xp download pro e student version download office 2007 turkish proofing tools download filme a era do nariz vermelho are you in download download lagu true worshippers jadi sepertimu warlords battlecry 3 download free full version apostilas calculo 1 download calof duti 2 download gratis