Jun 11 2013

Capturing A/V Multimedia Streams From the Browser

Category: Coding,Real-time Webjgoulah @ 9:29 PM

Background

I’ve recently been working on finishing up an application to help gather presentation feedback when demo’ing a talk for the first time in front of a live audience. The app will play back a video stream and overlay the audiences comments on top of the video in real time. However I haven’t open sourced it yet because there was a dependency on launching an external video application to do the recording, and have the user upload that to the server. I wanted the application to do “in browser” audio/video recording so that it was more seamless with the overall web application.

Turns out A/V recording is still pretty experimental on the web, but is slowly rolling out in major web browsers. The key component to understand when dealing with A/V capture is the getUserMedia API. The getUserMedia API is an in progress API to access audio and video streams. You can find a decent intro on this API here.

I was then wondering how to capture the streams recorded by the API and found this article which covers video only recording. The MediaStreamRecorder API is currently unimplemented in chrome, but there is a nice interface implemented in whammy.js that the author uses to demonstrate recording video. However, we also need sound, and it turns out based on the comments in that article that others are looking for that too.

There is another github project that I found called recorder.js which can help us record the audio stream. It uses the AudioContext interface to export the audio as a blob.

Combining these two ideas, I can capture separate audio and video streams, and so they must be combined. Theoretically this could be done with a javascript encoder, but I wasn’t sure if one exists that fits this use case or how much computing power that would take on the client side. Therefore I chose to encode the streams server side with avconv (aka. ffmpeg). When the recording is completed the client sends HTTP POST requests to the server with the content of each stream. The video is webm format the and audio is wav.

The command to do our transcoding is pretty simple. On the server side after you have received the files from our POST requests, you would do something like this, replacing the input files names with those from the recording:

avconv -i input.wav -i input.webm -acodec copy -vcodec copy output.mkv

And the file output.mkv that is produced will contain the video with your audio stream.

Implementation

If you’re interested in an example implementation you can find my code here. My next steps are to see if I can dig deeper into WebRTC to send both streams to my server and transcode them on the fly, instead of consuming the resources on the client side to hold the audio and video streams in memory until the video is complete. If you have a better way of accomplishing A/V recording please let me know in the comments!

Tags: , , , ,


May 28 2012

Using the New “load_recipe” Chef Function with Shef

Category: Codingjgoulah @ 9:21 PM

If you are developing chef recipes it really helps to use the command line tool called shef. Shef is just a REPL to run chef in an interactive ruby session. If you haven’t ever tried it, you can find some nice instructions over here to get you going.

Shef gives an easy way to iterate on your recipes so that you can make small changes and see the effects. However, I found the include_recipe function would only load the recipe one time, and complain that its seen the recipe before on subsequent tries. I added a small patch that implemented a new function called load_recipe that will allow you to reimport the recipe. The problem is that once the recipe is loaded again, the resource list is reimported giving us the same set of resources twice.

You can see the list of resources that are loaded up like so

chef:recipe > puts run_context.resource_collection.all_resources
package[php]
package[php-common]

If you were to call load_recipe again the list would double, and the new code would be run second when calling run_chef

chef:recipe > puts run_context.resource_collection.all_resources
package[php]
package[php-common]
package[php]
package[php-common]

The trick is that you can clear this list with this command

run_context.resource_collection = Chef::ResourceCollection.new

So to use load_recipe you should call the above before it to clear the current list. This can be done in one line like so

run_context.resource_collection = Chef::ResourceCollection.new; load_recipe "php"

Hopefully I’ll be able to patch things to add a reload_recipe that overwrites the old resources so you don’t have to use this trick, but for now this will work to get quick iterations going.

Tags: , , , ,


Dec 01 2011

iOS Development with Vim and the Command Line

Category: Coding,iOSjgoulah @ 11:12 PM

Intro

I’ve recently been playing around with some iOS code, and while I find Xcode an excellent editor with a lot of great built-ins, I’m a vim user at heart. It’s the editor I’ve been using on a daily basis for years, so its tough to switch when I want to do iPhone development. Of course there are some things you just have to use Xcode for, but I’ve been able to tailor vim to do most of my dev work, as well as build and run the simulator on the command line. I couldn’t find a single cohesive tutorial on this, so these are the pieces I put together to make this work for me.

Editing

The Vim Cocoa Plugin

To start, there is a decent plugin called cocoa.vim to get some of the basics going. It attempts to do a lot of things but I’m only using it for a few specific things – the syntax highlighting, the method list, and the documentation browse functionality.

One caveat is the plugin is not very well maintained, given the last release is nearly 2 years ago. Because of this you’ll want to grab the version from my github:

git clone https://jgoulah@github.com/jgoulah/cocoa.vim.git

Install it by following the simple instructions in the README.

You’ll get the syntax highlighting by default, and the method listing is pretty straightforward. While you have a file open you can type :ListMethods and navigate the methods in the file. Of course you may want to map this to a key combo, here’s what I have in my .vimrc for this, but of course you can map to whatever you like:

map <leader>l :ListMethods

There’s another command that you can use called :CocoaDoc that will search the documentation for the keyword and open it in your browser. For example, you can type :CocoaDoc NSString to open the documentation for NSString in your browser. However OS X gives a warning every time, which is pretty annoying. You can disable this on the command line:

defaults write com.apple.LaunchServices LSQuarantine -bool NO

After you’ve run the command, restart your Finder (or reboot). Lastly, it would be annoying to type the keyword every time, so you can configure a mapping in your .vimrc so that it will try to launch the documentation for the word the cursor is on:

map <leader>d :exec("CocoaDoc ".expand("<cword>"))<CR>

Navigation

One of the biggest things I miss when I’m in Xcode is the functionality you get from ctags. Of course you can right click on a function and “jump to definition”. Works fine, but feels clunky to me. I love being able to hop through code quickly with ctags commands. I’ve talked about this before, so I’m not going to give a full tutorial about ctags, but I’ll quickly go over how I made it work for my code.

First go ahead and grab the version from github that includes Objective-C support:

git clone https://github.com/mcormier/ctags-ObjC-5.8.1

Compile and install in the usual way. Now you can utilize it to generate yourself a tags file. Mine looks like this:

#!/bin/bash
cd ~/wdir
for i in MyApp ios_frameworks ; do
echo "tagging $i"
pushd $i
/usr/local/bin/ctags -f ~/.vimtags/$i -R \
--exclude='.git' \
--langmap=objc:.m.h \
--totals=yes \
--tag-relative=yes \
--regex-objc='/^[[:space:]]*[-+][[:space:]]*\([[:alpha:]]+[[:space:]]*\*?\)[[:space:]]*([[:alnum:]]+):[[:space:]]*\(/\1/m,method/' \
--regex-objc='/^[[:space:]]*[-+][[:space:]]*\([[:alpha:]]+[[:space:]]*\*?\)[[:space:]]*([[:alnum:]]+)[[:space:]]*\{/\1/m,method/' \
--regex-objc='/^[[:space:]]*[-+][[:space:]]*\([[:alpha:]]+[[:space:]]*\*?\)[[:space:]]*([[:alnum:]]+)[[:space:]]*\;/\1/m,method/' \
--regex-objc='/^[[:space:]]*\@property[[:space:]]+.*[[:space:]]+\*?(.*);$/\1/p,property/' \
--regex-objc='/^[[:space:]]*\@implementation[[:space:]]+(.*)$/\1/c,class/' \
--regex-objc='/^[[:space:]]*\@interface[[:space:]]+(.*)[[:space:]]+:.*{/\1/i,interface/'
popd
done;
view raw gistfile1.sh hosted with ❤ by GitHub

Just a little explanation here, I’m going into my working directory and generating a tags file for both MyApp and ios_frameworks. ios_frameworks is just a symlink that points to /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.sdk/System/Library/Frameworks

If you’re interested in how I load up these tag files you can check out my .vimrc on github.

Another handy feature from I wanted to emulate was flipping between the header and source file. Luckily there is yet another vim plugin that will do this for you called a.vim.

Grab it off github and install it by dropping it in your ~/.vim/plugin directory:

git clone https://github.com/vim-scripts/a.vim.git
cp a.vim/plugin/a.vim  ~/.vim/plugin

There’s a tiny bit of configuration that you can put in .vim/ftplugin/objc.vim:

let g:alternateExtensions_m = "h"
let g:alternateExtensions_h = "m"
map <leader>s :A

This just tells it which files to use for source and header files and creates another shortcut for us to swap between the two.

Build and Run

So now that we’ve got most of the basic things that help us edit code, its time to build the code. You can use xcodebuild for this on the command line. If you type xcodebuild –usage you can see the variety of options. Here’s what worked for me to build my target app, noting that I setup everything in Xcode first and made sure it built there. After that you can just specify the target, sdk, and configuration. This will create a build under the Debug configuration for the iPhone simulator:

xcodebuild -target "MyApp Enterprise" -configuration Debug -project MyApp.xcodeproj -sdk iphonesimulator5.0

And now to run the app. Back to github there’s a nice little app called iphonesim. Download that and just build it with xcode:

git clone https://github.com/jhaynie/iphonesim.git
open iphonesim.xcodeproj

Put the resulting binary somewhere in your PATH like /usr/bin.

Now we can launch the app we just built in the simulator:

iphonesim launch /Users/jgoulah/wdir/MyApp/build/Debug-iphonesimulator/MyAppEnterprise.app

Last thing you’ll want are logs. Another thing that is arguably more easily accessible in Xcode, but we actually can get the console log. If you’re using something like NSLog to debug this will grab all of that output. You’ll have to add a couple lines to your main.m source file:

NSString *logPath = @"/some/path/to/output.log";
freopen([logPath fileSystemRepresentation], "a", stderr);

And then you can run tail -f /some/path/to/output.log

Conclusion

This is how I use vim to compile iPhone apps. Your mileage may vary and there are many ways to do this, it just happens to be the path that worked well for me without having to pull my hair out too much. Xcode is a great editor and there are some things that you will have to use it for such as the debugging capabilities and running the app on the phone itself. Of course even these are possible with some work, so if anyone has tips feel free to leave a comment.

Tags: , , , , ,