The Gitorious Blog

The last few days have been terrible, for us and for you. The performance on gitorious.org has been really bad, and the site has been going up and down constantly. We’re really sorry about that.

We started receiving notifications that the server was unavailable about a week ago, and since that we’ve been trying to find out what was causing the outages and bad response times – which became worse after the weekend.

One of the first things we noticed was that out caching server was unable to cache any content at all, due to our Rails backend sending a Set-Coookie HTTP header on every request. Since this would tax the server really badly, we deployed a fix for this on Tuesday and saw some of the most requested content (single commit diffs) being cached again.

We were hoping that we had found the root cause and hoping for better performance as our cache got warm again. Not seeing the desired improvement in response times, we started suspecting there could be other issues causing the bad response times. Analyzing the server load, we found a really high number of Atom requests for pages that rendered slowly and suspected that the combination of polling RSS clients and missing cache support were slowing down the servers. We set up our cache server to force caching of any atom request for an hour, removing Set-Cookie headers in the process. When this didn’t work, we temporarily disabled atom requests entirely, which helped a little.

But the problems persisted. Last night we noticed that rendering a project page on gitorious.org was really slow. Easily over a minute for the most popular ones. Looking through the code, it turned out that we were effectively no longer caching events listed on that page in Memcache, resulting in a lot of database access. Once you’ve found the problem, the solution usually isn’t very far away – which was also the case here. We deployed Christian’s commit from last night this morning, and the servers are handling the load a lot better today.

The reason this happened, and the reason it took so long to find the problem, was that we merged a quite big feature into master a week ago, private repositories. Gitorious.org will not be offering private repositories, and there is a “feature switch” which is turned off for gitorious.org. Apparently there was still a place or two where performance was affected even if the feature was turned off, which is what happened over the last few days.

We’re not very proud of how we’ve kept you informed about the problems over the last few days. Heads-down, diagnosing the problem, trying different fixes and responding to support email, we neglected updating our status site and Identi.ca/Twitter accounts.

We did a bad job of keeping you informed, and we will make sure that you are kept fully in the loop if trouble should strike in the future.

[Edit: after publishing this post, we discovered a similar issue in the repository pages on Gitorious, we just deployed a fix for that].

We need to add some more disk to our servers, which means we’ll have some downtime starting at 14 CET today. We’ll update this post as we proceed.

Updates:

• 14:10: the physical servers are rebooting
• 14:40: the host was giving us a hard time, but the virtual machines are finally booting now
• 14:41: the file systems are being checked, currently at 2 percent.
• 14:44: 4 percent
• 14:47: 6,5%
• 15:04: 20
• 15:13: 35%
• 15:20: 50%
• 15:29: 66.6%
• 15:35: 70%
• 15:55: 75%
• 15:58: 80%
• 16:50: the check is done, we’re working on getting the servers online
• 16:52: there are some network issues, a network expert is looking into i
• 17:25: We’re back! Send an email to support@gitorious.org if you’re having problems.

A few months ago, Axis Communications, the world’s leading expert in network video, got in touch with us to discuss some features they were missing in their local Gitorious installation. We agreed that the Gitorious team would develop these features on a consulting basis for them, and today we’re proud to announce that these features are available for all users of Gitorious; be it on gitorious.org or on one of the hundreds of self-hosted Gitorious installations out there.

We are especially excited to announce these features since they demonstrate one of the greatest benefits of free software: that features developed for one user of the software is made available to all other users of the software. We are very grateful to Axis Communications for hiring us to develop these features, and we’re equally grateful that they decided to share the new features with other Gitorious users. Thanks!

Here’s a short summary of what’s new:

Graphical log view

Once you work with more than a single branch in Git, you’ll feel the need to visualize your repository’s history. There are various tools on the desktop that do this.

Starting today, you’ll find a link from the commit log for any Git branch or tag in Gitorious to a graphical representation of the log. Here’s a screenshot of how it looks for Gitorious itself:

Click on the commit message to jump to a specific commit, click on the commit SHA1 to view a graph starting from that commit.

The graph does not require any browser plugins to work, and should work on any fairly modern web browser. We developed two tools that work together to generate the graph view:

• capillary.rb generates a JSON data structure from Git’s log
• capillary.js  renders the JSON output from capillary.rb in the browser using Raphaël. If you look at the source, you’ll find that capillary.js will output a lot of other formats as well, including ASCII.

Annotations, aka. blame

Ever found yourself wondering who on earth wrote a specific line in a program file and why? In that case you may have used Git’s blame tool, which lets you see which commit last changed each line in a file. Blame is now available in Gitorious too, whenever you’re viewing a file/blob in a repository. Simply hit the “Blame” link from the panes to the right above the file, and you’ll see the same contents along with some more details:

To the left you’ll see a link to the commit that changed this line, along with the name of the author and when the change was made. Toggle between normal blob view, blame and history using the panes at the top right.

Display diffs between several refs

One of the very first features in Gitorious was the ability to view the diff of a single commit. Since then we have added support for viewing the diffs introduced by a merge request, and today we’ll let you view the diff between any two Git refs (SHA1/branch/tag) in a repository. The first place you’ll find this is next to the activity log entry when someone pushes to a repository on Gitorious:

Here we can see that the user Goatlord pushed some commits to a repository, and there’s a link to view all the changes introduced in this push – which brings you here:

All the changes between two commits, displayed either inline or side-by-side. You’ll find links to compare branches and tags to each other from the tree views in Gitorious; simply select which tag or branch to compare from the right sidebar.

How to upgrade your own server

If you’re running your own Gitorious server, you have probably read about our versioning scheme on the Gitorious wiki. Today’s features are in our latest version, version 2.1.0. In addition to the normal steps (run bundler, remove cached javascript and css files) you’ll need to load a few Git submodules – “git submodule init && git submodule update” should take care of this. If you run into trouble upgrading, the mailing list is full of helpful people.

Over to you

First of all: have fun using these features, we hope you’ll find them useful. If you fix any bugs, don’t hesitate to submit a merge request. The same goes if you have developed any features for your own Gitorious server that you want to share. If there are other features you’d like to see in Gitorious and you want help in developing these, please get in touch with us. There is no big corporation behind Gitorious, neither do we charge any license fees that could pay our bills. It’s up to Gitorious’ users to help us evolve.

Due to some changes in Gitorious’ network infrastructure, we need to reboot the servers today at 14.30 CET. This should be completed before 14:35 – we’ll post an update here once we’re done.

Edit: As of 16:08, we’re back up and running.