It turns out this little tool is long overdue, as simple of a concept as it is, but also easy to misunderstand the use cases for, ours at Etsy however was very targeted. Several years ago we were hammering out our internal cloud infrastructure, using KVM/QEMU based solution that you can read about over here. We were populating our virtual machine frontend using Chef Ohai data as the canonical source of our system information. There is an ohai plugin that gathers KVM virtualization data using virsh commands which are part of libvirt. It was a perfect way to capture information about which guests existed on a given system and other information about them.
We were hitting a bottleneck in that our chef clients were setup to run about every ten minutes. But within that ten minutes it would be possible that a virtual machine would be added or deleted, and therefore it was difficult to keep our interface in sync. Imagine creating a new virtual machine but not being able to display data about it until you waited around ten minutes, and to make matters worse these clients run at a splay interval, which means they don’t run all at the same time. Therefore, we started on a simple script that would let us run ohai quickly without needing to do the full chef run. While our chef runs are relatively quick (usually < 1 minute) it would introduce problems if we try to run chef while the client is already running.
Going to open source
It was supposed to be released a while ago, but has taken some time for various reasons. It’s a shockingly tiny amount of code but there were some barriers to releasing it. The majority of the code was written by Mark Roddy but he’d turned it over to me to open source. I went through the normal chef contribution process, which at the time required opening a Jira ticket that you can read if you’re interested in some of the details. In short there were some questions about the use case but when I explained that we weren’t trying to re-invent graphite in some horrible way, we were able to agree there could be some real world use cases. That being said, it was not accepted into the core yet because this does introduce very small race conditions since chef uses a read-modify-save model of changing the attribute data. There is a proposal to fix this, which divides attributes into different levels in which automated updates can access them without causing this issue. However in the wild this has not actually posed an issue for us even with several hundred nodes running it.
If you’re interested in this tool, you can install it with ruby gems using the command:
% gem install rehai
If you’d like to see the source, head on over to github.
I’d been using gnome vino as a VNC server for years on my media computer. This way I can use touchpad to control it from my iPad. It works fine, but a little bit clunky and badly documented, plus its tied directly to gnome. The other day I woke up to a filled drive (~2TB) of vino errors. I killed it off and cleaned up the error log file and tried to start it up again. No go this time due to some startup errors. This isn’t the first time I’ve fought with it, surely something better exists.
This led me to X11vnc. A bit of fresh air its fairly easy to setup and get going very quickly. I’ll first show how to do it manually and then present my open sourced chef cookbook.
First thing is to install it:
sudo apt-get install x11vnc
Setup a password file:
sudo x11vnc -storepasswd YOUR_PASS_HERE /etc/x11vnc.pass
Create an upstart config:
sudo touch /etc/init/x11vnc.conf
Open it and put this into it:
start on login-session-start
x11vnc -xkb -noxrecord -noxfixes -noxdamage -display :0 -rfbauth /etc/x11vnc.pass \
-auth /var/run/lightdm/root/:0 -forever -bg -o /var/log/x11vnc.log
and start it up:
sudo service x11vnc start
Simple, easy, and just works.
Naturally, I ported this to a chef cookbook which you can find here. I have to admit I’m not totally happy with it yet, mainly because it doesn’t restart x11vnc on config changes. In some cases such as our production servers, we actually prefer this so that we don’t accidentally roll out a broken config change and have an auto-restart bring everything to its knees (there are some ways to avoid this but thats another post). In any case on my home computer I prefer it to restart if I make changes, but I’m struggling to get upstart to stop the process correctly due to what seems to be some disassociation with its pid file. The other thing is the recipe currently assumes you are using Ubuntu or something similar, but can easily be extended. Hope this helps someone else out there!
I was at OSCON this week, and my friend Erik Kastner and I did a talk about development environments. Specifically what to avoid and how to keep environments consistent across development and production. As usual the slides are not fully explanatory without seeing the accompanying talk but here they are anyway:
If you are developing chef recipes it really helps to use the command line tool called shef. Shef is just a REPL to run chef in an interactive ruby session. If you haven’t ever tried it, you can find some nice instructions over here to get you going.
Shef gives an easy way to iterate on your recipes so that you can make small changes and see the effects. However, I found the include_recipe function would only load the recipe one time, and complain that its seen the recipe before on subsequent tries. I added a small patch that implemented a new function called load_recipe that will allow you to reimport the recipe. The problem is that once the recipe is loaded again, the resource list is reimported giving us the same set of resources twice.
You can see the list of resources that are loaded up like so
chef:recipe > puts run_context.resource_collection.all_resources
If you were to call load_recipe again the list would double, and the new code would be run second when calling run_chef
chef:recipe > puts run_context.resource_collection.all_resources
The trick is that you can clear this list with this command
run_context.resource_collection = Chef::ResourceCollection.new
So to use load_recipe you should call the above before it to clear the current list. This can be done in one line like so
run_context.resource_collection = Chef::ResourceCollection.new; load_recipe "php"
Hopefully I’ll be able to patch things to add a reload_recipe that overwrites the old resources so you don’t have to use this trick, but for now this will work to get quick iterations going.