Mar 30 2016

Why You Should Point Staging to Your Production Database

Category: Databases,Deployment,Systemsjgoulah @ 9:56 AM

I have been thinking about this topic more recently as I’ve just started working with a new infrastructure, and one of the things that I noticed is the staging database is a separate copy of the production database. It also happens to be a very old copy that has drifted, certainly in data and possibly in schema. It was likely put in place to avoid the perceived problem of new code possibly affecting production data in some critical way, and having a safe and solid environment to examine changes before they do hit production. But what are we accomplishing if the environment we test those changes in doesn’t really provide a fully identical setup to production? When you are dealing with DDL and the data underneath that may not actually mimic production, you’re only gaining a sense of false confidence.

This still sounds crazy, you say. “Of course we shouldn’t point staging to production! What if there is some bug that deletes some important rows from our database? We’d want to catch that before it hits in production!” And to this I wonder why your development environment isn’t an adequate testbed for diagnosing this sort of behavior well before its gone to staging in the first place?

As I thought more about this, it started to sound like the concept of staging itself originated more from an issue of difficult to reproduce environments. If your development environment does not exactly mirror production in terms of its configuration and setup, then it makes sense that you would need some intermediary place that is “closer to production” to test those changes. Hence staging is a place to almost test your changes and almost get that confidence you need, but without the same data underneath. In the end, this gives us no more confidence than until it is run in production itself.

In many ways, the staging environment originates from the waterfall model, and the days of manual QA testing. And this is not to say QA testing has no place, just that staging is not the right environment to perform those types of meticulous and slow, possibly at times manual work within the given time constraints. Unless your development to production cycle is hours or days, then the functionality you’re verifying on staging is only the small incremental changes that you are pushing at that time. You’re more likely and hopefully doing end-to-end testing across the codebase, and this is giving the confidence to move code through the pipeline quickly (in addition to metrics at the other end).

What we really want to test in staging is that our changes show up, the site builds and renders properly, and that we can perform the actions that we expect against the data we expect. If we are worried about rogue deleting data because of bad code, there are additional safeguards that can be taken aside from separating the environment entirely. The first thing is simply to not delete data. This may be a non-trivial change for a large legacy codebase, but it is worth considering. Instead of using the DELETE statement, consider flagging that data as removed and respecting the flag. Additionally, rolling code in percentages, to the people at your company and then small portions of traffic is a great way to gain confidence without sacrificing the integrity of the real purpose of staging.

This is not “testing in production”. This is ensuring your development environment is similar enough to production that staging technically only exists as an intermediary place to test your build. This means staging is the next thing in line for whats running on production and we have enough confidence to test our changes against production data. It makes a lot of sense for it to be allowed to interact on that data rather than a copy that may have a drastically different representation than whats on production. Therefore, it is worth considering pointing your staging database to production.

Tags: , , ,


Mar 21 2016

Configuring Chef for Provisioning

Category: Cloud Computing,Configuration Managementjgoulah @ 10:23 PM

If you’re working with infrastructure its good practice to describe it using code so that it is reproducible and consistent across servers and development environments. I’ve used Chef for quite some time and feel it is a pretty natural way to represent the source of truth for your servers, the packages installed on them, and their configuration. Chef can also be used as a provisioning tool, to bring your servers to life configured exactly to your specifications. You can use it with services like AWS or tools like Docker.

I started out using chef local mode to test my provisioning recipes, but also wanted to get things working with chef-client running as a daemon. But because of ACL’s in place when Chef is run this way, you need to grant permissions to the right groups to make sure they can do things such as create nodes.

This is hinted at with an error that looks like:

This error: Net::HTTPServerException: 403 "Forbidden"

or if you go digging with debug mode on (chef-client -l debug), you might see something analogous to this buried in a ton of output:

[2016-03-21T10:13:05-04:00] DEBUG: ---- HTTP Response Body ----
[2016-03-21T10:13:05-04:00] DEBUG: {"error":["missing update permission"]}

The default Chef ACL’s don’t allow nodes’ API clients to modify other nodes, and so we have to create a group with such permissions that your provisioning node (the one that kicks off the new instance/machine to be provisioned) can create the machines’ nodes and clients. This is similarly explained in this slightly outdated post here but unfortunately the commands aren’t quite right, so here it is using the most current version of the tooling.

Setting up Permissions

First things first install the ACL gem (assuming you’re using chef development kit)

chef gem install knife-acl

We can then create a group to give access to the permissions we need:

knife group create provisioners

Now, if you’re setting up a new node to be your provisioner, you would create the client key and node object:

knife client create -d chefconf-provisioner > ~/.chef/chefconf-provisioner.pem
knife node create -d chefconf-provisioner

Or you may already have a client that you run chef-client from. Lets say that is called chefconf-provisioner as it is the client we created above, so we’ll go with that, but your client can be named anything. Note, its usually the hostname of the node you’re running from. Add your client to the group we just created like so:

knife group add client chefconf-provisioner provisioners

Chef server uses role-based access control (RBAC) to restrict access to objects—nodes, environments, roles, data bags, cookbooks, and so on. This ensures that only authorized user and/or chef-client requests to the Chef server are allowed.

In this case we need to grant read/create/update/grant/delete permissions for clients and nodes so that our provisioning node can create the new instance/machine:

for permission in read create update grant delete
do
  knife acl add group provisioners containers clients $permission 
done
for permission in read create update grant delete
do
  knife acl add group provisioners containers nodes $permission 
done

And now you should have the permissions to be able to provision new nodes using Chef!

Tags: , , , ,


Mar 07 2016

Running Strace in Docker

Category: Cloud Computing,Containers,Kerneljgoulah @ 10:40 PM

I’ve been reverse engineering a new application setup and seemed like an appropriate place to try out docker. Spinning up a lightweight and reproducible environment is the goal and containerization is a reasonably efficient way to accomplish that. As I was looking into a problem with getting some services running properly, with little debug output and sparse documentation, I reached for the old trusty strace to see what was going on. But what do you know, strace is disabled by default on Docker. Here is the error that I got:

strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Operation not permitted
strace: test_ptrace_setoptions_for_all: unexpected exit status 1

This is admittedly an error I hadn’t seen before, and google isn’t a ton of help on this one. As a newbie with docker, it would have been helpful to have a bit more detailed error message as to why such a common tool as strace isn’t working.

Luckily some IRC logs come to the rescue in my quest through WTFed’ness. I learned that the security around this feature has apparently evolved a bit over time, with apparmor being the older but still used security system, and secconf being only available on newer distros (and OSX running boot2docker). Confusing things further, some of the articles out there are referring to apparmor (which uses different methods for modifying its security policy).

If you are using secconf, there are a couple of things you can pass to docker run to loosen up this security policy. To allow strace specifically, you enable the system call that it relies upon to get its information (ptrace):

--cap-add SYS_PTRACE

This whole paradigm is in fact documented but none of my original searches turned up these pages. In addition to disabling ptrace, there are a slew of other system level commands that you may (or may not) need that aren’t on the docker whitelist of allowed system calls. The list of calls can be adjusted very granularly by feeding docker a json file defining your security options. Or if you are feeling up for it, you can re-enable all of them in one fell swoop with this option to docker run:

--security-opt seccomp:unconfined

Its definitely worth considering which system calls your container really needs access to though, and strace is one of those that is quite useful for debugging purposes. There will always be that balance between security and usability, and decisions to make on which direction to swing the pendulum. It’s nice to see that this kind of functionality is being supported by docker to give the container very granular access to system level calls, and it might be interesting to think about ways it could be highlighted to a surprised enduser.

Tags: , , , ,


Feb 23 2015

Go Deployment

Category: Deployment,Golang,Organizationjgoulah @ 12:06 AM

Intro

In the spirit of the recent re-release of Deployinator, it seems timely to write a little bit about deployment. However, this is not a how-to on writing a Deployinator stack, but an overview of the high-level mechanics of getting a golang app deployed into production. Its worth mentioning that this methodology is just one of many ways you might go about deploying your app, but this is what has worked for me. These are the fundamental pieces that you’ll need to think through no matter what deploy tool you end up using.

Whats to Deploy?

When you run go build it produces a binary executable that can easily be shipped around. But there are actually several key pieces to getting this binary deployed into production. The code you have must be able to compile against the libraries you’re using, and if you are deploying a web based app, you’ll need to think about where your asset files are deployed and how they’re referenced from the go binary. The configuration of the application is also important in that you’ll want to be able to have the app run under different environments, such as development and production. Lastly, you’ll need to think about how to control the app, so that you can easily and consistently restart it when you deploy.

Building

Typically when doing production deployments there exists a build server where the application is compiled and the pieces are put in place for shipping to production. Certainly no production facing app should be getting deployed from a developers virtual machine, but the process for building the app should be identical. Building a go app is fairly straightforward but it can be a good idea to organize your steps and common commands in a Makefile. We’ll build our binary into the bin subdirectory:

build:
        go build -o ./bin/myapp myapp.go

This produces the binary that we’ll eventually ship.

Dependencies

This build process works fine in development but how do we know the external dependencies exist on our build machine? How can we maintain this app building process in an identical way across environments? Whereas in development we can just go get our dependencies, it may be haphazardness to maintain things this way on a production build machine, since you might have different go apps being built that use different versions of the same dependency. Or your developers may have pulled down different versions at different points in time. The current versions of github repositories that your code relies on also may introduce bugs that you want to gain confidence doesn’t impact your application. To maintain consistency in dependency versions, and to ensure these same dependencies exist in our build environment, we’ll use a tool called godep.

Assuming you have a proper go toolchain setup, you can get started with godep by getting it and running a single command:

go get github.com/tools/godep
# then in your project dir
godep save -r ./...

This should pull all of your dependencies into a Godeps directory as well as update your files to import this “frozen” copy of the dependencies you are using.

Since we won’t want to stay locked on old versions of software, we need a way to update our projects dependencies from time to time. This is something you’d probably want to add to your Makefile:

updep:
        go get -u -v -d $(import)
        godep update $(import)
        godep save -r ./...

Using this command we can update any of our dependencies like so:

make updep import=github.com/codahale/metrics

Configuration

A config file is a reasonable way to maintain different environments such as development and production. The gcfg library is a perfectly fine way to define ini style configs that map easily to go structs. It will support multiple environments and you can have a file defining your shared values and then define development and production differences in another set of files. Then you can merge the appropriate set of files based on your environment using ReadFileInto. The example below merges a secrets configuration file with the base configuration to show a simple example, but you can extend this technique to support multiple environments.

In this example we make sure the base config exists, based on a location relative to this file. We’re next looking for the secrets config, first in the app directory where we may store it locally for development, and then in an /etc directory for production where a configuration management system may place the file more securely. This way this sensitive file doesn’t have to be stored in the main repository, but can pass production values securely to the application. Note that means this file wouldn’t be deployed in the normal way, and the app will fallback to the secure values in /etc when it doesn’t find the local file. The full configuration example implementation can be found here but remember this is just a very basic example to show how to merge configs.

Initialization

When you deploy the app to production you’ll have to restart it in some way. The simplest way to do this is to write an init script. But you shouldn’t have to write an init script by hand these days. Check out pleaserun to generate a script for your system. Now you can stop, start, and restart your process whether you are using upstart, systemd, plain old init, or something else. You can go one step further and implement a graceful restart, but the implementation will vary depending on your needs.

Tying it Together

So now we have a way to deal with our dependencies and the build. We know how to control the app. The app knows how to find its configuration files and override base values with production settings. We can use the same runtime.Caller trick that we used to locate the config file to make it easier for the application to find its static files as well.

At this point there are only a few steps that need to happen on a deploy box:

  • Get the newest code
  • git fetch && git reset --hard
  • Build the binary
  • cd $basedir && make build
  • Rsync the code
  • rsync -va --delete --force \
    --include='static' --include='bin' --include='config*.cfg' --include='*/*' --exclude='*' \
    deploy.mydomain.com:/var/myorg/build/golang/src/github.mydomain.com/jgoulah/myapp/ /usr/myorg/myapp
  • Use your init script to restart the app, depending on the system
  • /etc/init.d/myapp restart

In this example the rsync command is run on an application box and pulling from our deploy host. In a large deploy we may dsh out to many app hosts to rsync our build over in a pull based fashion. The /var/myorg/build directory is where our make build command was run to produce our binaries. We sync the bin directory that contains the binaries, the static directory which has our html/js/css, and the config files. Lastly we restart our app. These steps can be coded into a Deployinator stack or with the deploy tool of your choice. But regardless of the tool we use, we will have a repeatable and dependable process with reliable dependencies that we can ship.

Tags: , , , , ,


Jul 06 2014

Dynamically Update Hypervisor Guest Info in Chef With Rehai

Category: Cloud Computing,Configuration Managementjgoulah @ 11:31 AM

Introduction

It turns out this little tool is long overdue, as simple of a concept as it is, but also easy to misunderstand the use cases for, ours at Etsy however was very targeted. Several years ago we were hammering out our internal cloud infrastructure, using KVM/QEMU based solution that you can read about over here. We were populating our virtual machine frontend using Chef Ohai data as the canonical source of our system information. There is an ohai plugin that gathers KVM virtualization data using virsh commands which are part of libvirt. It was a perfect way to capture information about which guests existed on a given system and other information about them.

Our problem

We were hitting a bottleneck in that our chef clients were setup to run about every ten minutes. But within that ten minutes it would be possible that a virtual machine would be added or deleted, and therefore it was difficult to keep our interface in sync. Imagine creating a new virtual machine but not being able to display data about it until you waited around ten minutes, and to make matters worse these clients run at a splay interval, which means they don’t run all at the same time. Therefore, we started on a simple script that would let us run ohai quickly without needing to do the full chef run. While our chef runs are relatively quick (usually < 1 minute) it would introduce problems if we try to run chef while the client is already running.

Going to open source

It was supposed to be released a while ago, but has taken some time for various reasons. It’s a shockingly tiny amount of code but there were some barriers to releasing it. The majority of the code was written by Mark Roddy but he’d turned it over to me to open source. I went through the normal chef contribution process, which at the time required opening a Jira ticket that you can read if you’re interested in some of the details. In short there were some questions about the use case but when I explained that we weren’t trying to re-invent graphite in some horrible way, we were able to agree there could be some real world use cases. That being said, it was not accepted into the core yet because this does introduce very small race conditions since chef uses a read-modify-save model of changing the attribute data. There is a proposal to fix this, which divides attributes into different levels in which automated updates can access them without causing this issue. However in the wild this has not actually posed an issue for us even with several hundred nodes running it.

Installation

If you’re interested in this tool, you can install it with ruby gems using the command:

% gem install rehai

If you’d like to see the source, head on over to github.

Tags: , , , , ,


Jun 22 2014

Stubbing AFNetworking Calls In XCode Unit Tests

Category: iOSjgoulah @ 11:58 AM

Intro

A unit test by definition of testing the smallest possible part of the application, should eliminate all dependencies from the systems its interacting with. This in turn will remove any unknown or variable outcomes from the tests, focusing on the isolating the specific code you’re interested in. In the case of networking calls, it is unideal for tests that deal with the (un)marshalling of data to depend on the network. Therefore a stubbing library is needed.

OHHTTPStubs to the Rescue!

Luckily there is a great library to do just what we need here – stubbing out some network calls – called OHHTTPStubs.

Because these tests are asynchronous, the network requests are sent in a separate thread or queue different from the thread used to execute the test case. So you’ll have to ensure that your unit tests wait for the requests to finish and their response arrived before performing the assertions in your test cases. Otherwise your code making the assertions will execute before the request had time to get its response.

OHHTTPStubs includes a custom AsyncSenTestCase subclass of SenTestCase that provides methods like waitForAsyncOperationWithTimeout: and notifyAsyncOperationDone. We can utilize these methods in our test cases.

A Networking Stub

Most people are using the AFNetworking library so this example will show a test using that.

In this case I’ve created a network wrapper that allows turning stubs on and off. The request here happens to be mocking XML data, but JSON follows the same pattern. This is an implementation of a class called GrillNet that has two functions:

The two functions are: stubsOn to activate the stubbed networks calls, and getGrillStatusWithURL which invokes a network request. If we call stubsOn before the getGrillStatusWithURL call is made, it will invoke the stub and return the mock data which I’ve generated and put into the file cyberq.xml.

Notice that getGrillStatusWithURL implements a success and failure callback. Thus I’ve created a protocol to respond to the two scenarios which asks for optional implementations of notifyDone and notifyDoneWithData:data, shown here:

Our tests can implement this protocol to assist with the notifying that our asynchronous operation is done by calling notifyAsyncOperationDone. The class GrillzTests subclasses AsyncSenTestCase and implements GrillNetActionProtocol.

And in this same class we can implement notifyDone and notifyDoneWithData:data. They will both call notifyAsyncOperationDone from AsyncSenTestCase but notifyDone is only called on failure when no data is returned, while notifyDoneWithData:data is invoked when the request is successful .

Therefore it is the job of notifyDoneWithData:data to do something with this data. In this case we are setting the values in a GrillStatus object that was used in the test assertion comparing the value in _gs above.

References

https://github.com/AliSoftware/OHHTTPStubs

https://github.com/AFNetworking/AFNetworking

https://github.com/AliSoftware/OHHTTPStubs/wiki/OHHTTPStubs-and-asynchronous-tests

http://andreamazz.github.io/blog/2014/01/10/network-stubs-with-ohhttpstubs/

https://github.com/AliSoftware/OHHTTPStubs/wiki/Usage-Examples

Tags: , , , , ,


Jun 11 2013

Capturing A/V Multimedia Streams From the Browser

Category: Coding,Real-time Webjgoulah @ 9:29 PM

Background

I’ve recently been working on finishing up an application to help gather presentation feedback when demo’ing a talk for the first time in front of a live audience. The app will play back a video stream and overlay the audiences comments on top of the video in real time. However I haven’t open sourced it yet because there was a dependency on launching an external video application to do the recording, and have the user upload that to the server. I wanted the application to do “in browser” audio/video recording so that it was more seamless with the overall web application.

Turns out A/V recording is still pretty experimental on the web, but is slowly rolling out in major web browsers. The key component to understand when dealing with A/V capture is the getUserMedia API. The getUserMedia API is an in progress API to access audio and video streams. You can find a decent intro on this API here.

I was then wondering how to capture the streams recorded by the API and found this article which covers video only recording. The MediaStreamRecorder API is currently unimplemented in chrome, but there is a nice interface implemented in whammy.js that the author uses to demonstrate recording video. However, we also need sound, and it turns out based on the comments in that article that others are looking for that too.

There is another github project that I found called recorder.js which can help us record the audio stream. It uses the AudioContext interface to export the audio as a blob.

Combining these two ideas, I can capture separate audio and video streams, and so they must be combined. Theoretically this could be done with a javascript encoder, but I wasn’t sure if one exists that fits this use case or how much computing power that would take on the client side. Therefore I chose to encode the streams server side with avconv (aka. ffmpeg). When the recording is completed the client sends HTTP POST requests to the server with the content of each stream. The video is webm format the and audio is wav.

The command to do our transcoding is pretty simple. On the server side after you have received the files from our POST requests, you would do something like this, replacing the input files names with those from the recording:

avconv -i input.wav -i input.webm -acodec copy -vcodec copy output.mkv

And the file output.mkv that is produced will contain the video with your audio stream.

Implementation

If you’re interested in an example implementation you can find my code here. My next steps are to see if I can dig deeper into WebRTC to send both streams to my server and transcode them on the fly, instead of consuming the resources on the client side to hold the audio and video streams in memory until the video is complete. If you have a better way of accomplishing A/V recording please let me know in the comments!

Tags: , , , ,


Apr 23 2013

Crossing the Production Barrier: Development at Scale

Category: Conferences,Databasesjgoulah @ 5:59 PM

Slides from my talk at Percona 2013

Tags:


Mar 25 2013

Ganglia Grill Graphs

Category: Graphsjgoulah @ 10:55 PM

One of my favorite things to do (other than stare at a computer all day) is to cook. I recently bought a new smoker grill, and quickly realized that I was running outside the entire day to tune the vents to keep the temperature in the range I wanted for a 12 hour slow cook. I did a little bit of research and turns out there are some decent devices that provide a fan to keep the temperature consistent. Even better, there was one equipped with a web server that can be accessed over wi-fi, called the CyberQ.

Its not cheap, but life is short, and it will maintain the temperature you set up to 475 degrees Fahrenheit. I already have Ganglia setup in my house for basic server monitoring (you run linux servers in your house too right?) and since the device connects to the home network I figured it would be easy enough to query and graph. Luckily the people that made this device were nice enough to provide an XML endpoint that is easy to parse for all the information you want. There are a few different endpoints but I request the config.xml version since it gives you the most information.

So I wrote a gmetric to query the box and throw the data into a few ganglia graphs. As of this writing I graph the fan speed, the pit temp, and the three food temps it can provide. There is a bunch of other data returned, but this was most important for me initially.

While this device would normally be used for low and slow cooks, I did a faster high temp grill cook just to try it out. I set the temp to the max of 475 and you can see it took a bit of time to bring the grill up to that temp, and the fan output to get it there. Once it got there, you can see it drop when I opened up the lid to put the food on. The rest of the fluctuation is due to opening the lid and flipping the food:

pit-temp

fan


and then here is the food, two steaks cook rather quickly at that high of a temperature:


food1

food-temp


Tags: , , , ,


Jan 21 2013

Building Your Own Cloud From Scratch

Category: Cloud Computing,Systemsjgoulah @ 8:50 PM

Intro

There are a lot of private cloud solutions out there with great things built into them already to complete a full cloud stack – networking, dashboards, storage, and a framework that puts all the pieces together, amongst other things. But there is also a decent amount of overhead to getting these frameworks setup, and maybe you want more flexibility over some of the components, or even just something a little more homegrown. What might a lightweight cloud machine bootstrapping process look like if it where implemented from scratch?

Getting Started

We can use libvirt and KVM/QEMU to put something reasonably robust together, start by installing those packages:

apt-get install qemu-kvm libvirt libvirt-bin virtinst virt-viewer

The next important thing is to setup a bridge for proper networking on this host. This will allow the guests to use the bridge to communicate on the same network. There should be a few articles out there that can help you set this up, but the basics are that you want your bridge assigned the IP that your eth0 interface previously had, and then add the eth0 interface to the bridge. In this example 192.168.1.101 is the IP of the host machine:

# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eth0 inet manual

auto br0
iface br0 inet static
  address 192.168.1.101
  netmask 255.255.255.0
  gateway 192.168.1.1
  network 192.168.1.0
  bridge_ports eth0
  
ifup br0

Building the Image

The first step is setting up a base template that you create your instances from. So grab an iso to start from, we’ll use debian, but this process works with any distro:

% wget http://cdimage.debian.org/debian-cd/6.0.6/amd64/iso-cd/debian-6.0.6-amd64-netinst.iso

And allocate a file on disk to the size you’d like your template to be. I created one here at 8GB, it can always be expanded later, so this should only need to be big enough to hold the initial base image that all instances will start from. Generally smaller is better because of the copy step when instances get created later.

% dd if=/dev/zero of=/var/lib/libvirt/images/debbase.img bs=1M count=8192

Now you can start the linux installation, noting the –graphics args for the ability to connect with VNC. Our installation target disk is the one we created above, debbase.img, and we are giving it 512M RAM and 1 CPU.

% virt-install --name=virt-base-deb --ram=512 --graphics vnc,listen=0.0.0.0  --network=bridge=br0 \
--accelerate --virt-type=kvm --vcpus=1 --cpuset=auto --cpu=host --disk /var/lib/libvirt/images/debbase.img \
--cdrom debian-6.0.6-amd64-netinst.iso

Once thats started up you can use VNC on your client machine to connect to this instance graphically and run through the normal install setup. There are plenty of clients out there but a decent one is Chicken of the VNC. Its also possible at this step that you’d create the image off a PXE boot or similar bootstrapping mechanism.

Extract the Partition

Here we take advantage of QEMU ability to load Linux kernels and init ramdisks directly, thereby circumventing bootloaders such as GRUB. It then can be launched with the physical partition containing the root filesystem as the virtual disk.

There are two steps to make this work. First you’ll need the vmlinuz and initrd files, and the easiest way to get those is to copy them from the base image we setup above:

% scp BASEIP:/boot/vmlinuz-2.6.32-5-amd64 /var/lib/libvirt/kernels/
% scp BASEIP:/boot/initrd.img-2.6.32-5-amd64 /var/lib/libvirt/kernels/

The next step is to extract the root partition from that same base image. We want to take a look at how those partitions are laid out so that we can get the right numbers to pass to the dd command.

% sfdisk -l -uS /var/lib/libvirt/images/debbase.img

Disk /var/lib/libvirt/images/debbase.img: 1044 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/var/lib/libvirt/images/debbase.img1   *      2048  15988735   15986688  83  Linux
/var/lib/libvirt/images/debbase.img2      15990782  16775167     784386   5  Extended
/var/lib/libvirt/images/debbase.img3             0         -          0   0  Empty
/var/lib/libvirt/images/debbase.img4             0         -          0   0  Empty
/var/lib/libvirt/images/debbase.img5      15990784  16775167     784384  82  Linux swap / Solaris

We are going to pull the first partition out, note how the numbers line up to the first line corresponding to debbase.img1 line. We start at sector 2048 and get 15986688 sectors of 512 bytes each:

% dd if=/var/lib/libvirt/images/debbase.img of=/var/lib/libvirt/debian-tmpl skip=2048 count=15986688 bs=512

Templatize the Image

Now we have a disk file that serves as our image template. There’s a few things we want to change directly on this template. Note that we are using a few all caps placeholders ending in -TMPL that we’ll replace later with sed. We can edit the templates files by mounting the disk:

% mkdir -p /tmp/newtmpl
% mount -t ext3 -o loop /var/lib/libvirt/debian-tmpl /tmp/newtmpl
% chroot /tmp/newtmpl

Note at this point we are chrooted and these commands are acting against our template disk file.

Clear out the old IPs tied to our NIC when the base image networking was setup:

% echo "" > /etc/udev/rules.d/70-persistent-net.rules

We’re going to put a placeholder for our hostname in /etc/hostname:

% echo "HOSTNAME-TMPL" > /etc/hostname

Set a nameserver template in /etc/resolv.conf:

% echo "nameserver NAMESERVER-TMPL" > /etc/resolv.conf 

In the file /etc/network/interfaces:


# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address ADDRESS-TMPL
netmask NETMASK-TMPL
gateway GATEWAY-TMPL

This will give us console access when we boot it. Make sure /etc/inittab has this line (usually just uncomment it):

T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100

Creating an Instance

Now we have all the pieces together to launch an instance from our image. This script will create the instance given the IP and hostname. It does no error checking for readability reasons, and is well commented so that you know whats going on:

#!/bin/bash

# read in ' ' from command line
virt_ip=$1
virt_host=$2

# build the fqdn based off the short host name
virt_fqdn=${virt_host}.linux.bogus

# fill in your network defaults
virt_gateway=192.168.1.1
virt_netmask=255.255.225.0
virt_nameserver=192.168.1.101

# how the disk/ram/cpu is sized
virt_disk=10G
virt_ram=512
virt_cpus=1

# random mac address
virt_mac=$(openssl rand -hex 6 | sed 's/\(..\)/\1:/g; s/.$//')

cp /var/lib/libvirt/images/debian-tmpl /var/lib/libvirt/images/${virt_host}-disk0

# optionally resize the disk
qemu-img resize /var/lib/libvirt/images/${virt_host}-disk0 ${virt_disk}
loopback=`losetup -f --show /var/lib/libvirt/images/${virt_host}-disk0`
fsck.ext3 -fy $loopback
resize2fs $loopback ${virt_disk}
losetup -d $loopback

mountbase=/tmp/${virt_host}
mkdir -p ${mountbase}
mount -o loop /var/lib/libvirt/images/${virt_host}-disk0 ${mountbase}

# replace our template vars
sed -i -e "s/ADDRESS-TMPL/$virt_ip/g" \
       -e "s/NETMASK-TMPL/$virt_netmask/g" \
       -e "s/GATEWAY-TMPL/$virt_gateway/g" \
       -e "s/HOSTNAME-TMPL/$virt_fqdn/g" \
       -e "s/NAMESERVER-TMPL/$virt_nameserver/g" \
  ${mountbase}/etc/network/interfaces \
  ${mountbase}/etc/resolv.conf \
  ${mountbase}/etc/hostname

# unmount and remove the tmp files
umount /tmp/${virt_host}
rm -rf /tmp/${virt_host}*

# run a file system check on the disk
fsck.ext3 -pv /var/lib/libvirt/images/${virt_host}-disk0

# specify the kernel and initrd (these we copied with scp earlier)
vmlinuz=/var/lib/libvirt/kernels/vmlinuz-2.6.32-5-amd64
initrd=/var/lib/libvirt/kernels/initrd.img-2.6.32-5-amd64

# install the new domain with our specified parameters for cpu/disk/memory/network
virt-install --name=$virt_host --ram=$virt_ram \
--disk=path=/var/lib/libvirt/images/${virt_host}-disk0,bus=virtio,cache=none \
--network=bridge=br0 --import --accelerate --vcpus=$virt_cpus --cpuset=auto --mac=${virt_mac} --noreboot --graphics=vnc \
--cpu=host --boot=kernel=$vmlinuz,initrd=$initrd,kernel_args="root=/dev/vda console=ttyS0 _device=eth0 \
_ip=${virt_ip} _hostname=${virt_fqdn} _gateway=${virt_gateway} _dns1=${virt_nameserver} _netmask=${virt_netmask}"

# start it up
virsh start $virt_host

assuming we named it buildserver, run the above like:

% buildserver 192.168.1.197 jgoulah

Conclusion

This is really just the first step, but now that you can bring a templated disk up you can decide a little more about how you’d like networking to work for your cloud. You can either continue to use static IP assignment as shown here, and use nsupdate to insert dns entries when new guests come up, or you can set things up such that the base image uses dhcp, and you can configure your dhcp server to update records in dns when clients come online. You may also want to bake your favorite config management system into the template so that you can bootstrap the nodes and maintain configurations on them. Have fun!

Tags: , , , , ,


Next Page »


download xiuxiu editor foto shambho shankara mp3 free download pikeno e menor perdicao download download lagu surat at taubat smarthru 4 download pl abbey road 60s drums download mac cell phone repair download download kara winter magic album download intel gma booster terbaru download mp3 tantowi yahya free download farhan ali qadri video naats free download internal medicine harrison download music ragheb alama sinin dhada songs download in ziddu download form 4852 from the irs website free invitation templates download and print download boundless by cynthia hand free la chatimi cantare mp3 download free download of deception point ebook download munni badnam hui free mp3 download jtx party like a rockstar download hide ip ng 1.40 download ooh la la goldfrapp download time after time quietdrive crime and punishment mp3 download sweety gippy mp3 download caminhos da liberdade download minions banana video download reflex flugsimulator download gratis naruto shippuden 208 pt download sniper elite wii download ntsc pirata do espaço download dublado download ptanks full version o justiceiro download pc rip microsoft hda driver download download admit card iti jharkhand 2012 bada 2.0 download wave 2 download gangster life gta apple safari 4 x download business intelligence download oracle download os x dvd installer download gta 8 vice city myegy szybcy i wściekli 5 2011 download download highschool of the dead color download shakira ft pitbull rabiosa zippy download lagu t ara zombie jelly car music download download driver sony vaio 32bit klezmer music to download free download of shreenathji bhajan aga bai arechya download serie v 3 temporada download download the jeremy kyle show download jogos fazenda gratis pc minimizer download para mu susana nothing at all download download um novo vencedor damares playback download command line mac os download paypal jar for android download zeljko vasic zanjisi kukovima marian keyes watermelon ebook download internet download manager flurry icon kruti dev free download windows download chief keef choppa go bang download msi 3.1 windows installer chessmaster free download with crack bravo hits 98 download download disrespect kirko bangz x plane 6 demo download la baby jonas brothers mp3 download netgear ga311 windows 7 download free download habib painter mp3 download tweetdeck desktop windows 7 ekhon ami mp3 download promethean the created download pdf how to download youtube videos to ipad amazon download games steam cypress hill download 2011 download treu nha hang xom 2 download kick out the epic mother wooh da kid stepped download x2 x men united download imaginasamba perfeição download download navigation for mobile zor ka jhatka song download mp3 download audi a4 owners manual download pretty little liars s01e05 avi download nitro circus season 1 download eega promotional song download the simulator 2012 demo x264 codec download mac download lições para toda vida legendado download mkisofs for ubuntu download star trek voyager scorpion lil b 855 download download pokemon black 2 jap rom spells aprilynne pike pdf download ita download ways of reading cursed crusader trainer download download darmowe gry dla dzieci download spyglass for android alaa wardi 7aram free download massive attack teardrop song download download settings for nokia x6 download account opening form obc download fl studio on a mac worth dying for download s bot download free silkroad 1 click downloader download 5 ishq ka sheen download download melodia que eu conheço stephen king novels download free 100 download psp games for free a escolha download dublado ptgui 9 mac download kz hack download gratis download driver yamaha psr 3000 download macroeconomics policy and practice download hp dv1000 sound driver download famous five movies download phim benh nhan nguoi anh download gta 1 for free download gangs of wasseypur movie free download panasonic sd jukebox software download falling up drake download trackmania sunrise extreme full version free download de simuladores de combate aereo download i bruise easily download pioneer dj software free download noah and the whale life goes on mp3 download recover deleted files software download sơ đồ kế toán download free desk phone ringtone fußball manager 12 download vollversion kostenlos even greater mp3 download planetshakers download snmp for windows 2000 free download amuse park game bejeweled 3 jar download sleeping at last quicksand download download ứng dụng cho nokia n8 coral player download luna download jill scott whenever you're around download fairy tail games for pc download photoshop 8 cs me myegy download yahoo latest version for free mp7 player free download mac download internet explorer 8 vista java hry download 128x160 download jason upton key of david download march of the wooden soldiers download barad toro be dast avordam mp3 download songs of aitraaz from songs pk com 6.72 f ai nightmare download download tiny toon adventures nes download i'm yours mp3 download 9 hours rom pipi player download vista deskjet d2360 software download download booster pack hack abhas ha mp3 song download mp3 songs download 3gp download cod 4 3rd person mod trey songz blind download free download hawaii 5 0 season 2 kenji free download fort minor download clubbed to death 2 quebrando regras 2 download portugues chamas da vingança 1984 download download sketchup 8 deutsch virtual router manager download xp download instalador chrome offline download soundgarden live to rise download canon mx410 printer driver download the harold song kesha mp3 garmin 255w download maps free download mise a jour mcafee download visio windows 7 64 bit download gen psp 3000 quasi amici film download gratis ita download plano de fuga rmvb legendado download alfonso loher name in the sky download mp3 zahra damariva alasan download j rock songs download bihar secretariat assistant admit card download project 64 64 bits download naruto chapter 589 cartoon download for ipad download tributo a bezerra da silva download roda a roda jogo cx one full download armin van buuren rapture download mp3 download nein mann video download tambor de funk download rota de fuga rmvb free download of shawty got moves mp3 download cm7 for droid 2 global download executive resume format amnesia game download mac crash bandicoot mutant island download pc ipod touch download pictures to computer download tu pirata soy yo chayanne download lg pc suite p990 harmor vst plugin download download abaqus 6.10 student edition i like cereal song download filmes alta resolucao download italian lessons download mp3 mass effect 3 download pirate download manager idm key download wh cs 2011 bau simulator download ita download efek suara unik download girl talk ultraviolet sound download bicara hati episod 4 annie khalid songs download download driver epson cx5500 free quake 3 download bots infinity blade 2 ipa download crack pk songs download list dream the game download free cydia download step by step motorola xoom rom download download let's go ricky luna remix zune manual download pdf download hit and run 2012 dvdrip download do sapo videos download do jogo cities xl 2011 download song socha na tha by alamgir ktechlab for ubuntu download download vara rece kamelia zippy download schenk mir dein herz gipsy kings volare download free welsh flag to download download afinador do cifra club 3 gatsu 9 ka download free mp3 angry birds android download download manager error the server returned an error download sdo x season 2 song download paint shop pro download turn to u justin bieber download original ruu for evo download maps to print boys over flowers download songs download free regular show episodes download yamaha psr 1500 styles satyamev jayate download song download mouse fix for windows xp download elliot in the morning kenapa tidak bisa download film h.p f4200 printer software download p square game over download mp3 carpet 3d max download download mw 3 1.07 patch the legend of zelda download snes rom download 9 temporada friends download on my freebox ne fonctionne pas download tito lopez the blues sharebeast download sweeney todd final scene papago x8.5 wince 5 download download monkey for rhino download na paz de jah download gratuito adobe reader 8 ngo accounting software download download onto mp3 from youtube somebody's me mp3 download enrique download ocarina of time 1.0 download 64 bit windows tax form 8379 download frisky tinie tempa download zippy download pro update psp go 6.60 download lagu chrisye gejolak cinta download themes for gw300 clr via c# richter download download 8195 the damned rar parayathe ariyathe malayalam song download autobiography of abraham lincoln download download phim hiep dao hoa hetaoni english download part 2 cenário de novela download mp3 sandra brown envy download pdf download sims 1 love bed sende ahasa wage mp3 download download voice changing application download feed us 2 free download open source library management system download alana grace black roses red download audi a3 manual download free irctc mobile application stand o food 3 apk download download dss dj effects visual basic software download download film khuda kay liye download intezar remix by falak sri rama rajyam mp3 download download dan seals one friend download virtual families mac free download gossip girl cecily von ziegesar download autoramas fale mal de mim gabin doo uap mp3 download free download love in this club mp3 download dead space 3 demo pc charmed download season 1 nhac chuong theo ten download angry birds season download free pc reign of hunters download download de pokemon flash download full screen theme wordpress download manager 6.05 crack download planta x zumbi download razor ramon entrance music download skype xperia 8 download lagu tercipta untukku ungu download shining inheritance ep 16 trial download microsoft project 2010 halo ce download key thermodynamics 6th cengel download vmware ova converter download download e.r legendado 1 temporada wilfred season one download download hivi mata hati download apun bola mp3 zmierzch księżyc w nowiu download peb cod tool download 1.5 web client get download speed ryback meat mp3 download music download on itunes download beenie man i'm okay vigilante 8 2nd offence download pc download movie 2012 in hindi latha tamil font free download word melhor impossivel download legendado download lloyds tsb bank statement download oki b6300 driver how to download spoutcraft free popup blocker download google chrome download tenth avenue north losing download sound intervention mw2 download jogo harry potter pc download pokemon blanc nds alda célia playback download download shwayze get you home download tower bloxx mobile game dani california official video download download jump out the gym download all killer no filler sum 41 ra one full movie download 2011 download rebelde só pro meu prazer download tees maar khan movie in avi format activex control download install download video setia band stasiun cinta download de temas nokia x2 00 download power geez 2005 computer games download com download benny and babloo songs soc pc camera driver download for xp manually download sophos virus definitions jackie chan adventures download links download dragostea se face in doi download terjemahan kitab al umm hp photosmart c3180 download scanner download ipcop for windows 7 nero 8 download windows 7 64 bit kick buttowski kick in genes download dewana 2013 mp3 download download 2 chainz birthday song free how to download correct video driver download film g 30s pki firing games download full version download free 3d motorbike racing download call of duty 4 zombies download smart mobile themes download crbl romanu n are noroc hotfiles download diggy simmons make you mine download bangla natok bhalobashi tai download kml from my maps download song if this charlie sheen shinda new album download download outcast 1 temporada keterlaluan the potters mp3 download download office 2007 upload download falling skies 2 temporada rmvb download sajan all songs oblivion mod manager download mac download toma o meu coração download pierce the veil caraphernelia mp3 download lagu jkt48 original download leaf by elle varner vandalism coming alive mp3 download download god of war betrayal 240x320 download amanda by zigi mp3 download apostila do trf hp 635 driver download windows xp download pro e student version download office 2007 turkish proofing tools download filme a era do nariz vermelho are you in download download lagu true worshippers jadi sepertimu warlords battlecry 3 download free full version apostilas calculo 1 download calof duti 2 download gratis