Approximate Times to Update an RDS Instance

Here’s a quick overview from an upgrade a couple months ago of an RDS instance type db.t2.medium to type db.r3.large. In addition to changing the instance type, we upgraded the disk from 64GB to 100GB, and applied a new parameter group. The disk increase by far took the longest amount of time. We clocked in at ever so slightly over an hour for the disk increase, while the instance upgrade only took a bit over 16 minutes. [Read More]

Why You Should Point Staging to Your Production Database

I have been thinking about this topic more recently as I’ve just started working with a new infrastructure, and one of the things that I noticed is the staging database is a separate copy of the production database. It also happens to be a very old copy that has drifted, certainly in data and possibly in schema. It was likely put in place to avoid the perceived problem of new code possibly affecting production data in some critical way, and having a safe and solid environment to examine changes before they do hit production. [Read More]

Building Your Own Cloud From Scratch

Intro There are a lot of private cloud solutions out there with great things built into them already to complete a full cloud stack – networking, dashboards, storage, and a framework that puts all the pieces together, amongst other things. But there is also a decent amount of overhead to getting these frameworks setup, and maybe you want more flexibility over some of the components, or even just something a little more homegrown. [Read More]

Ditching Vino for X11vnc

I’d been using gnome vino as a VNC server for years on my media computer. This way I can use touchpad to control it from my iPad. It works fine, but a little bit clunky and badly documented, plus its tied directly to gnome. The other day I woke up to a filled drive (~2TB) of vino errors. I killed it off and cleaned up the error log file and tried to start it up again. [Read More]

Quick Tip: Find the QEMU VM to Virtual Interface Mapping

The other day we were getting some messages on our network switch that a host was flapping between ports. It turns out we had two virtual hosts on different machines using the same MAC address (not good). We had the interface and MAC information, and wanted to find what vm domains mapped to these. Its easy if you know the domain name of the vm node, you can get back the associated information like so: [Read More]

Percona Live 2012 – The Etsy Shard Architecture

I attended Percona Conf in Santa Clara last week. It was a great 3 days of lots of people dropping expert MySQL knowledge. I learned a lot and met some great people. I gave a talk about the Etsy shard architecture, and had a lot of good feedback and questions. A lot of people do active/passive, but we run everything in active/active master-master. This helps us keep a warm buffer pool on both sides, and if one side goes out we only lose about half of the queries until its pulled and they start hashing to the other side. [Read More]

Distributed MySQL Sleuthing on the Wire

Intro Oftentimes you need to know what MySQL is doing right now and furthermore if you are handling heavy traffic you probably have multiple instances of it running across many nodes. I’m going to start by showing how to take a tcpdump capture on one node, a few ways to analyze that, and then go into how to take a distributed capture across many nodes for aggregate analysis. Taking the Capture The first thing you need to do is to take a capture of the interesting packets. [Read More]

Quick Tip: Dynamically Updating Screen Window Titles With The Current Server Hostname

I haven’t had a ton of time for blogging lately but figured this tip was good enough to throw out there for all the screen users. One way I like to organize servers that I’m ssh’d into is using screen windows. As you hopefully know you can use Ctrl-A c to create sub windows within screen. Then you can switch between them in several ways such as using Ctrl-A X where X is the window number, Ctrl-A n or Ctrl-A p for next and previous, and Ctrl-A “ to get a list of the windows for selection. [Read More]

Using Mongo and Map Reduce on Apache Access Logs

Introduction With more and more traffic pouring into websites, it has become necessary to come up with creative ways to parse and analyze large data sets. One of the popular ways to do that lately is using MapReduce which is a framework used across distributed systems to help make sense of large data sets. There are lots of implementations of the map/reduce framework but an easy way to get started is by using MongoDB. [Read More]

Investigating Data in Memcached

Intro Almost every company I can think of uses Memcached at some layer of their stack, however I haven’t until recently found a great way to take a snapshot of the keys in memory and their associated metadata such as expire time, LRU time, the value size and whether its been expired or flushed. The tool to do this is called peep Installing Peep The main thing about installing peep is that you have to compile memcached with debugging symbols. [Read More]