Feb 06 2010

Improve Git Emails

Category: Uncategorized,Version Controljgoulah @ 3:02 PM

This is just a quick tip on how to hack gitosis to allow you to have better subject lines in your emails. This assumes you are using gitosis, not gitolite which many people hosting their own git repositories are moving to because it supports per branch and granulated user permissions. If you are looking on a full article on how to install gitosis check out this post.

So grab gitosis

 git clone git://eagain.net/gitosis

Apply this patch to set an environment variable for the user that is doing a push

diff --git a/gitosis/serve.py b/gitosis/serve.py
index d83b1d8..a6a49a2 100644
--- a/gitosis/serve.py
+++ b/gitosis/serve.py
@@ -200,6 +200,7 @@ class Main(app.App):
         main_log.debug('Serving %s', newcmd)
+        os.putenv('USER', user)
         os.execvp('git', ['git', 'shell', '-c', newcmd])
         main_log.error('Cannot execute git-shell.')

Then install gitosis with that change

$ sudo ./setup.py install

Now, edit the /usr/local/bin/git-post-receive-email script to change the subject line to make more sense

--- /usr/local/bin/git-post-receive-email.old	2010-02-06 19:29:35.000000000 +0000
+++ /usr/local/bin/git-post-receive-email	2010-02-06 19:14:03.000000000 +0000
@@ -186,7 +186,7 @@
 	# Generate header
 	cat <<-EOF
 	To: $recipients
-	Subject: ${emailprefix} $refname_type, $short_refname, ${change_type}d. $describe
+	Subject: ${emailprefix} push from ${USER}- $refname_type, $short_refname, ${change_type}d.
 	X-Git-Refname: $refname
 	X-Git-Reftype: $refname_type
 	X-Git-Oldrev: $oldrev

If you haven't setup email already make sure that you have the hook installed by making a symbolic link in /home/git/repositories/myrepo.git/hooks

ln -s /usr/local/bin/git-post-receive-email post-receive

And now your subject changes from this

[GIT] branch, master, updated. d61c9b93446a13093ef7f510a588c69a606ff762

to this

[GIT] push from jgoulah- branch, master, updated.

Now we know who is pushing code by glancing at the subject line. The username comes from the filenames in gitosis-admin/keydir/. Obviously you could style the subject line however you see fit.

Tags: ,

Nov 01 2009

Migrating SVN to Git

Category: Version Controljgoulah @ 5:55 PM


SVN migrations to Git can vary in complexity, depending on how old the repository is and how many branches were created and merged, as well as if you use regular SVN or a facade such as SVK to do your branching and merging. If you have a fairly new repository and the standard setup of a trunk, branches, and tags directory, your job could be pretty easy. However if you’ve done a ton of branching and merging, or your repository follows a non-standard directory setup, or that setup changed over time, you could have a bit more work on your hands.

SVN to Git Conversion

Building Git

The very first thing to do is make sure you are running the newest git. And for this I recommend from source. So grab it and do the usual compile stuff

$ wget http://kernel.org/pub/software/scm/git/git-
$ tar xjf git- 
$ cd git-
$ ./configure
$ make

Now I’m not sure if this is because I am always using local::lib but I tend to get this error after a while of building. If you don’t get it you can obviously skip this step

Only one of PREFIX or INSTALL_BASE can be given

This is fixable by going into the git/perl directory and manually building the perl modules

$ cd perl/
$ perl Makefile.PL 

Then go back up a directory and finish the build

$ cd ..
$ make
$ sudo make install

Using git-svn

If you have made any merges using SVK then you should grab the git-svn from Sam Vilain’s git branch here. I had to download it because the git clone wasn’t working so grab it like so

$ wget http://download.github.com/samv-git-ccef8e0.tar.gz

Open it up, and copy the git-svn.perl script into your bin directory

$ tar xzvf samv-git-ccef8e0.tar.gz
$ cd samv-git-ccef8e0
$ cp git-svn.perl ~/bin/git-svn

Now you can git svn clone your repository with the git-svn command. The –prefix=svn/ is necessary because otherwise the tools can’t tell apart SVN revisions from imported ones. If you are using the standard trunk, branches, tags layout you’ll just put –stdlayout like we have below. However if you had something different you may have to pass the –trunk, –branches, and –tags in to identify what is what. For example if your repository structure was trunk/companydir and you branched that instead of trunk, you would probably want ‘–trunk=trunk/companydir –branches=branches‘. You can optionally provide an authors file, and the last option to our script is the repository URL.

$ git-svn clone --prefix=svn/ --stdlayout --authors-file=authors.txt http://example.com/svn

This can take a few minutes or as long as a few hours depending how big your repository is. When its done you should end up with a git checkout of your repository. If you look at your remote branches you’ll see that its ported our svn stuff with the svn prefix we used

$ git branch -r

Anyway at this point it doesn’t hurt to create a tarball of your project so you can get back to it later if you screw up.

Fixing the Git Repository

There are a few more scripts you need to grab to finish this off. We want to clone the git-svn-abandon repository for these.

$ git clone git://github.com/nothingmuch/git-svn-abandon.git 

Now go into that directory and copy the new scripts into your local bin directory, or wherever you feel is good for your purposes

$ cd git-svn-abandon
$ cp git-svn-abandon-* ~/bin/

You can now run git-svn-abandon-fix-refs, which will run through all the imported refs, recreating properly dated annotated tags, and makes branches out of everything else. It’ll also rename trunk to master.

$ git-svn-abandon-fix-refs

We can see that it took the things that our git branch -r command earlier listed and imported them into actual git branches, as well as putting us on the master branch which is analogous to the trunk from svn

$ git branch
* master

Grafting Your Branches

Depending how correct you want your repository, you could skip this step. However it could be a good idea to try to get your merge history correct. This requires a little bit of understanding of the git internals, so at a minimum you should read and understand this introduction. There is also a nice presentation of how git works here that you may consider watching.

So now you understand (you went over those links right?) that the way git figures out merges is such that it basically creates a new node on the graph that points to both the head of master and of the branch. And then it basically takes the file contents from the 3 points, the new merge base node (the ancestor), and each parent, deltafies them and applies the result to the ancestor. Obviously if they don’t apply cleanly a conflict is created and must be resolved. The previous revisions contents is no longer relevant at all, except as a part of history, and our new snapshot of the history is born. So this is basically what we have to fix.

This is a snapshot from gitk of my repository. We can see near the bottom that it branched properly, but even though the repository was merged in svn, it doesn’t appear to be merged in our migration to git.

The way to fix this is to edit the .git/info/grafts file. With my great photoshop skills I’m going to point out what goes in there

We need to take what the first arrow is pointing to, and tell git that its parents are its current parent (the third arrow) as well as the point the branch was merged in (the second arrow). So all we have to do is add the hash from each of those commits, put the top node first, then the second and third with a space in between each to the .git/info/grafts file.

After we add those to the .git/info/grafts file we can just reload gitk or gitx to see those changes that visually show the branch was merged

When you are done you can run git-svn-abandon-cleanup which cleans up SVK style merge commit messages and removes git-svn-id strings. Another important thing that happens is the grafts entries are incorporated into the filtered commits, so the extra merge metadata becomes clonable

$ git svn-abandon-cleanup

Publish Your Repository

If you followed the last article you’ve got a gitosis setup that we can use to store our code centrally. So depending what host this is on you can import it like so

$ git remote add origin git@localhost:repository.git
$ git push --all
$ git push --tags

And we’re done.


We’ve gone over how to migrate an SVN repository to Git and how to deal with some of the complications of how Git interprets our repository based on what was in SVN and how to go about fixing that. Hopefully we also learned a little bit about Git too in the process.


Tags: , , ,

Oct 31 2009

Setting up Gitosis

Category: Version Controljgoulah @ 4:33 PM


This article is part one of a two part series that covers setting up a hosting server using gitosis for your central repository, and in the next article, taking an existing SVN repository and running the appropriate scripts and commands necessary to migrate it into something git can work with.

So this article is how to setup and manage a git repository.There are some great services out there than can do this for you, but why pay money for something you can easily do for free? This article shows how to setup and manage a secure and private git repository that people can use as a central sharing point.

Setting Up Gitosis

Gitosis is a tool for hosting git repositories. Its common usage is for a central repository that other developers can push changes to for sharing.

First clone the gitosis repository and run the basic python install. You just need the python setuptools package

sudo apt-get install python-setuptools

And then you can easily install it

git clone git://eagain.net/gitosis.git
cd gitosis
sudo python setup.py install

Next you need to create a user that will own the repositories you want to manage. You can put its home directory wherever you want, but in this example we’ll put it in the standard /home location.

sudo adduser \
    --system \
    --shell /bin/sh \
    --gecos 'git version control' \
    --group \
    --disabled-password \
    --home /home/git \

Then you must create an ssh public key (or use your existing one) for your first repository user. We’ll use an init command to copy it to server and load it. If you don’t have a public key you can create one with ssh-keygen like so

ssh-keygen -t dsa

Then gitosis-init is for the first time only, loads up your users key, and goes like this

sudo -H -u git gitosis-init < ~/.ssh/id_dsa.pub

Here it doesn't hurt to make sure your post-update hook has execute permissions.

sudo chmod 755 /home/git/repositories/gitosis-admin.git/hooks/post-update

Now you can clone the gitosis-admin repository, which is used to manage our repository permissions.

git clone git@YOUR_SERVER_HOSTNAME:gitosis-admin.git
cd gitosis-admin

Now you can see you have a gitosis.conf file and a keydir directory

$ ls -l
total 8
-rw-r--r-- 1 jgoulah mygroup   83 2009-10-31 20:44 gitosis.conf
drwxr-xr-x 2 jgoulah mygroup 4096 2009-10-31 20:44 keydir

The gitosis.conf file holds group and permission information for your repositories, and the keydir folder holds your public keys.

If I look in there I see my public key was imported from our earlier gitosis-init command

$ ls -l keydir/
total 4
-rw-r--r-- 1 jgoulah mygroup 603 2009-10-31 20:44 jgoulah.pub

So open up gitosis.conf and you should already see you have an entry for the gitosis-admin repository that we just cloned. The gitosis-init command above setup the access for us. From now on we can just crack open gitosis.conf and edit the permissions, commit and push back to our central repository.

If I wanted to create a new project for a repository called pizza_maker it would look something like this.

[group myteam]
members = jgoulah
writable = pizza_maker

Don't forget the members section is the name of your public key file without the .pub at the end. If your key was named XYZ.pub then your member line would have XYC here.

git commit -a -m "Create new repo permissions for pizza_maker project"
git push

As a reminder the second part of this series will show an svn to git import. For now lets assume we are starting from scratch. We'd create our project like this

cd && mkdir pizza_maker
cd pizza_maker
git init
git remote add origin git@YOUR_SERVER_HOSTNAME:pizza_maker.git
git add *
git commit -m "some stuff"
git push origin master:refs/heads/master

The only other thing to know is if you want to grant another user access to your repository. All you have to do is add their public key to the keydir folder, and then give the user permissions by modifying gitosis.conf

cd gitosis-admin
cp ~/otherdude.pub keydir/
 [group myteam]
- members = jgoulah
+ members = jgoulah otherdude
  writable = pizza_maker

If you need to, you can also grant public access over the git:// protocol like so

sudo -u git git-daemon --base-path=/home/git/repositories/ --export-all

Then someone can clone like

git clone git://YOUR_SERVER_HOSTNAME/pizza_maker.git


This article showed how to setup gitosis, how to initialize your gitosis-admin repository, which is a unique concept in itself to use a repository to manage repositories, and it works rather well. We also went over how to create our own new git repository, and how to manage the access permissions through gitosis.conf. Part two of this series will explain how to port from your current SVN setup to a Git setup. This article was a prerequisite if you want to host your own private repository when you're converting from SVN to Git, and thats what we'll look at next time.

Tags: , , , , , ,

Jan 11 2009

Intelligent Version Control and Branching

Category: Version Controljgoulah @ 12:28 PM


Most shops these days seem to know that using a Version Control System is necessary for the organization of a large software project involving multiple developers.   It’s essential to allow each developer to work on their part of the project and commit the changes to a central repository, which the rest of the developers can then pull down into their working sandbox.  Its an effective way to develop that avoids overwriting of changes, and allows for easy rollback and history.  However, one problem I see over and over again is either lack of branching altogether,  or doing it in a manual way that drifts the branch further out of sync with trunk as time goes on, and then trying to manually merge this with the trunk code when the branch is done.   For those who don’t know, trunk is basically your mainline, in which code can be branched off of for features that won’t interfere with trunk until they are merged at some later point.  This allows for testing changes without bothering other developers, but still being able to keep a commit history.

The difficult thing about this branching process is that if you try to manually merge code that has been on a branch for some time, the continuous development on trunk will have brought the two to such different places that you end up trying to resolve a ton of conflicts by hand.  Conflicts happen when changes are made to the same file and same lines in that file by different people.  Trunk is eventually a different bunch of code than when the branch was started, so by the end of your branch you could be developing against what is essentially a different codebase.  Anyone that has tried to do merging manually knows how painful this process can be.

Luckily, tools exist to help us with the process.  This article is going to focus on two tools that can interact with an SVN repository to help us with intelligent branching.   The tools I’m going to review are SVK and Git, and I will go over each separately. In fact they should not be used simultaneously since Git doesn’t know anything about the metadata SVK expects to be in your SVN props. In other words do not attempt to create a branch with SVK and then merge with Git, it just won’t work. Git by the way, can be used standalone, without SVN,  but I’m going to show you how to use it with SVN because that seems to be the version control system of choice in most shops these days. SVK is just a wrapper around SVN, its not a standalone tool. The great thing about both of these is you don’t necessarily have to migrate all of your developers over at once.  They can continue day to day work flow while you harness the power of branching.  Eventually people will see the advantage and try it out too.  Under the covers you are still storing your code in an SVN repository.

Using SVK

Installing SVK from source is well beyond the scope of this article, so I’m assuming you have a package management system that will handle this installation for you.  Once you have SVK installed, using it is very straightforward for SVN users.  In fact, its mostly the same exact commands, with a few additions.  Like SVN, you can do ‘svk help <command>’  to get detailed help on a particular command.

Checking Out Your Repository

Checking out your repository is very similar to SVN, and is done with the ‘checkout’ command:

svk co {repo}

where {repo} is your repository url (eg. http://svn.yourdomain.com/svn/repo)

You’ll be prompted for a base URI to mirror from and its ok to press enter here:

New URI encountered: http://svn.yourdomain.com/svn/repo
Choose a base URI to mirror from (press enter to use the full URI): ENTER

You’ll be prompted for a depot path and its also ok to accept the default

Depot path: [//mirror/repo] ENTER

SVK is a decentralized version control system and thus needs to mirror your SVN repository locally. You’ll see a prompt like this and you’ll be ok to accept the default:

Synchronizing the mirror for the first time:
a : Retrieve all revisions (default)
h : Only the most recent revision
-count : At most 'count' recent revisions
revision : Start from the specified revision
a)ll, h)ead, -count, revision? [a] ENTER

This step can take quite a while, depending on how large your repository is. Just be patient and it will finish up eventually.

Working with SVK

From here SVK is very similar to SVN. You’ll have checked out the entire repository, so if your SVN is setup according to standard you will see a trunk, branches, and tags directory under your repository root folder.

SVK has the same command set with one caveat, that when you update you have to give the -s flag:

svk up -s

This allows SVK to sync the mirror with the repository before updating your code.

Branching With SVK

The point of this article is how to use these tools for branching, so lets create a branch. There are a few ways to do this but a nice way is to copy trunk to a branch on the mirror itself:

svk cp //mirror/{repo}/trunk //mirror/{repo}/branches/my_branch_name
cd {repo}/branches
svk up -s # pull the new branch back from the mirror

Now you can work, work, work, and commit as you please. Here is where SVK really shines. You can pull from the repository path you branched from at any time. In this case we branched from trunk, so we can pull trunks changes into our branch.

svk pull

SVK will track all of the metadata surrounding the change, and if there happens to be a conflict it is resolved at pull time. Now the trunk code is merged into your branch. You are in sync with trunk, in addition to your branches changes.

The last thing to know is how to push your branch back to trunk when you are done. It doesn’t hurt to do one last pull to make sure you are in sync with trunk, and then:

svk push -l

Now your branch has been merged back to trunk. That is it! Painless and easy. You’ve already resolved the conflicts at pull time which SVK has tracked, so the push is simple. SVK has done all the hard work for us.

Its a good habit to get rid of your branch once you are done:

rm -rf branches/branch_name
svk rm branches/branch_name
svk commit -m "deleted old branch" branches/branch_name

Using Git

Using Git with SVN is a similar process, and like SVK, Git is also a distributed system. In a nutshell this means we can have local branches and remote branches as well. Its important to understand this concept when using Git much more so than when using SVK.  Its probably a good idea to go over the SVN to Git tutorial to see how familiar SVN commands map to Git.

Checking Out Your Repository

With Git you essentially clone your SVN repository. Here I’m using a repository called testrepo located on the same host so I am using the file protocol. You can use whichever protocol you normally use to access your SVN repository.

git svn clone --prefix=svn/ file:///usr/local/svn/testrepo -T trunk -b branches -t tags

You don’t have to use the –prefix here, but it allows you to be able to distinguish between local branches and those representing remote svn branches easily, so I highly recommend it.

Viewing Local and Remote Branches

With Git its important to remember that we have both local and remote branches. Local branches only exist for our sandbox, while remote branch are shared out on the SVN repository for others to use.

You can view your local branches like so:

$ git branch
* master

So far we only have the default master branch. When git svn is finished importing all of the svn history it sets up a master branch for you to work with. The tip of that branch will be the latest commit in the svn repository (note this is not necessarily trunk). The star indicates the current branch that we are on.

Its not a bad idea to have master correspond to trunk, and to be sure we can run:

git reset --hard svn/trunk

You can also view your remote branches. These exist out on the SVN repository we’ve cloned:

$ git branch -r

So far I can see that a trunk exists, as well as a branch called testbug. But these are remote, so to work with them we need to pull them down locally. Right now I’m interested in trunk, so we’ll grab that:

$ git checkout -b trunk svn/trunk
Switched to a new branch "trunk"

Anytime you would like to update the code with changes from the repository, you can do it like so:

git svn rebase

Branching With Git

Now lets assume we want to create a new branch called foo. This command will actually create the new branch on the SVN repository.

$ git svn branch foo

Since the branch is still only remote, we need to get a local copy:

$ git checkout -b foo svn/foo

Now lets actually do something on the branch, so we can show how merge operations work. We’ll just create a simple file:

$ touch myfile
$ echo "stuff" > myfile
$ cat myfile

And commit that to the local repository:

$ git add myfile
$ git commit -m "my new file"

Note this only commits to the local repository and other users on the branch won’t see the change until we commit it to the remote repository. Its a good idea to do a dry run to make sure we are committing to the right place. We can use ‘-n’ for dry run:

$ git svn dcommit -n
Committing to file:///usr/local/svn/testrepo/branches/foo ...

This looks correct, its going to commit to branches/foo on our remote repository, so run the command without the -n option.

$ git svn dcommit

Now lets switch back to trunk so we can perform a merge.

$ git checkout trunk

Now that we are on trunk, our new file doesn’t exist, which makes sense since it was created on the branch. But we want to merge our branch foo back into trunk, which can be done like so:

$ git merge --no-ff foo

And again we want to commit that local change to the remote repository:

$ git svn dcommit

There’s one more important thing to note. Normally you don’t need –no-ff unless you are using git against svn like this. –no-ff is needed so you get a real merge commit. If you don’t have that and a fast-forward is possible you’ll end up with the commit from the svn branch on top of the history, in which case the branch that commit on top is part of will be used for committing.

Therefore its a good idea to add this option to the default merge options for anything maintained with git-svn. Open up the .git/config and add the following:

branch.foo.mergeoptions = --no-ff

This allows you to use git merge and not remember the –no-ff option.


We’ve seen how to use two distributed revision control systems to help us manage our remote SVN repositories. We know how to create a branch, and stay in sync with the development on trunk. We have also learned now to merge our changes from the branch back into trunk in a very painless and easy way.

Tags: , , , ,