Updates:

• 14:10: the physical servers are rebooting
• 14:40: the host was giving us a hard time, but the virtual machines are finally booting now
• 14:41: the file systems are being checked, currently at 2 percent.
• 14:44: 4 percent
• 14:47: 6,5%
• 15:04: 20
• 15:13: 35%
• 15:20: 50%
• 15:29: 66.6%
• 15:35: 70%
• 15:55: 75%
• 15:58: 80%
• 16:50: the check is done, we’re working on getting the servers online
• 16:52: there are some network issues, a network expert is looking into i
• 17:25: We’re back! Send an email to support@gitorious.org if you’re having problems.

Some of you may know of the infamous merge request #115. This partially solves the problem at hand, but unfortunately not in a way that we can accept responsibility for. However, the discussion around this merge request spawned the idea of a fund-raiser, and that’s how we are going to do this.

How, what and when

Technically, this work will be based off of Rodrigo Rosas’ branch which replaces Gitorious’ authentication implementation with the Devise gem.

On top of this system we will implement role-based authorisation that extends to pushing and pulling git repositories, as well as scoping all user generated data on the web site (project information, repository activity, events of various kinds).

We will put as much authorization logic as possible in the generic layer already in place in Gitorious so that it can easily be utilized regardless of whether you are using database backed logins, LDAP or the upcoming Atlassian Crowd SSO support. This means you can use LDAP groups to manage access to Gitorious content.

We are ready to start this work early 2012, given proper funding. We have estimated this feature to roughly 4 weeks of work. Using our reduced hourly rate offered to Local Install customers, this comes out to $24000.

Private repositories will be developed and shipped as part of Gitorious mainline. It will not be offered as a free service on gitorious.org. This means we will ship some sort of configuration switch controlling whether or not this system is available in a given installation.

Is your company interested in using this feature? How about helping funding it? Get in touch, let us know the amount you would like to contribute, and if you have any specific features you would want us to account for. You can help fund this project anonymously if you so wish, but we recommend you allow us to tell the world what a great company you are for helping a free software project becoming even better!

Any contribution is appreciated. Email support@gitorious.org if you are interested. I will keep you guys updated on the donation progress here on the blog.

We are especially excited to announce these features since they demonstrate one of the greatest benefits of free software: that features developed for one user of the software is made available to all other users of the software. We are very grateful to Axis Communications for hiring us to develop these features, and we’re equally grateful that they decided to share the new features with other Gitorious users. Thanks!

Here’s a short summary of what’s new:

Graphical log view

Once you work with more than a single branch in Git, you’ll feel the need to visualize your repository’s history. There are various tools on the desktop that do this.

Starting today, you’ll find a link from the commit log for any Git branch or tag in Gitorious to a graphical representation of the log. Here’s a screenshot of how it looks for Gitorious itself:

Click on the commit message to jump to a specific commit, click on the commit SHA1 to view a graph starting from that commit.

The graph does not require any browser plugins to work, and should work on any fairly modern web browser. We developed two tools that work together to generate the graph view:

• capillary.rb generates a JSON data structure from Git’s log
• capillary.js  renders the JSON output from capillary.rb in the browser using Raphaël. If you look at the source, you’ll find that capillary.js will output a lot of other formats as well, including ASCII.

Annotations, aka. blame

Ever found yourself wondering who on earth wrote a specific line in a program file and why? In that case you may have used Git’s blame tool, which lets you see which commit last changed each line in a file. Blame is now available in Gitorious too, whenever you’re viewing a file/blob in a repository. Simply hit the “Blame” link from the panes to the right above the file, and you’ll see the same contents along with some more details:

To the left you’ll see a link to the commit that changed this line, along with the name of the author and when the change was made. Toggle between normal blob view, blame and history using the panes at the top right.

Display diffs between several refs

One of the very first features in Gitorious was the ability to view the diff of a single commit. Since then we have added support for viewing the diffs introduced by a merge request, and today we’ll let you view the diff between any two Git refs (SHA1/branch/tag) in a repository. The first place you’ll find this is next to the activity log entry when someone pushes to a repository on Gitorious:

Here we can see that the user Goatlord pushed some commits to a repository, and there’s a link to view all the changes introduced in this push – which brings you here:

All the changes between two commits, displayed either inline or side-by-side. You’ll find links to compare branches and tags to each other from the tree views in Gitorious; simply select which tag or branch to compare from the right sidebar.

How to upgrade your own server

If you’re running your own Gitorious server, you have probably read about our versioning scheme on the Gitorious wiki. Today’s features are in our latest version, version 2.1.0. In addition to the normal steps (run bundler, remove cached javascript and css files) you’ll need to load a few Git submodules – “git submodule init && git submodule update” should take care of this. If you run into trouble upgrading, the mailing list is full of helpful people.

Over to you

First of all: have fun using these features, we hope you’ll find them useful. If you fix any bugs, don’t hesitate to submit a merge request. The same goes if you have developed any features for your own Gitorious server that you want to share. If there are other features you’d like to see in Gitorious and you want help in developing these, please get in touch with us. There is no big corporation behind Gitorious, neither do we charge any license fees that could pay our bills. It’s up to Gitorious’ users to help us evolve.

Please devote a few minutes of your time to fill out the simple questionnaire; it’ll help the Git community understand your needs, what you like about Git (and what you don’t), and overall help us improve it.

The survey will be open from 5 September to 3 October 2011.

The results will be published at GitSurvey2011.

Edit: As of 16:08, we’re back up and running.

We will post updates via our identi.ca/Twitter accounts, and update this post once we’ve completed the upgrade. As always, you can check our status site to see the current status.

Edit: We’re back up.

As someone pointed out on the Gitorious mailing list, most free software licenses prohibit sublicensing, which made our terms of service incompatible with these licenses. We have been working with our attourney and the Free Software Foundation over the past two months to come up with a terms of service that both limits Gitorious’ liabilities in case of copyright infringement and is compatible with free software licenses.

We have just posted an updated version of the Terms of Service that the FSF finds acceptable, and still protects Gitorious sufficiently. We believe the changes we have made are to our users’ benefit, but as a gitorious.org user you should review these changes.

If you have any comments or questions about these changes, please comment on this post.

With today’s deployment, gitorious.org will never redirect you away from https (if it does, report a bug to support@gitorious.org). You will also be forced to use https as long as you’re logged in, and when posting forms (i.e. logging in).

For those of you who maintain your own Gitorious setups, this change is pretty straight forward. The new SSL feature is enabled by default, and can be controlled through the gitorious.yml setting use_ssl. When this setting is set to true, Gitorious will enforce SSL where appropriate. When it is false, Gitorious will actively ensure http.

We will follow up this change by adding HSTS shortly.

In other news, gitorious.org is also available on IPv6, thanks to our awesome hosting partner, Linpro. We’re still having some issues with the backend for git:// and http:// Git access, so for now they are IPv4 only. We are working to resolve this issue.

We have been having performance issues with our database over the last few months. The main bottleneck has been our events table, which keeps track of all kind of activity on gitorious.org. The events table even contained one row for each commit ever pushed to gitorious.org, information that is already available through Git itself. A few weeks ago, we pushed a change to Gitorious which means we no longer create an entry in the database for each commit, only the push that occurs – and the push event keeps track of which commits were included. The commit data is then fetched from the Git log.

This morning we raised the amount of memory allocated to our MySQL server, specifically the innodb_buffer_pool_size setting, and restarted the database server. To be on the safe side of things we reloaded the Passenger instances, to keep them from hanging on to dead database connections. Initially, things seemed to be okay, until the web app went down about 15 minutes later. We tracked this down to about 100 Passenger instances waiting on the global queue, and found the MySQL server busy inserting data into the sessions table. By now the familiar sound of face-meets-palm was getting louder: we have a sessions table? Apparently, we did. Having one and not knowing it means it kept quite a lot of data too. So we switched to cookie based sessions and restarted the server, and things seem to be running smoothly now.

The reason why the database server stopped responding – blaming the huge amounts of data on ourselves – is that the file system where MySQL keeps its data went full while the server was coming back up after the restart. We’ll be adding more disk there now.

Sorry about the downtime, it happened because we’re addressing the performance on gitorious.org. Changing things involve a risk of breaking them, which is what happened today.

It turned out that over 20% of the requests to the servers were requests for a commit (Commits#show in Rails-speak). You guys seem to use Gitorious to read code! The thing is that all Git activity is quite expensive, so these 20-something percent requests accounted for 35% of the server’s rendering time.

The chance of a Git commit identified by a SHA changing over time is fairly low, which means it’s a very good candidate for caching. The only issue is that Gitorious displays quite a lot of personalized content, depending on whether you’re logged in or not: the top part of the page displays your username and unread message count, for instance. So if we naïvely cached it, the second user requesting a commit would receive the page rendered for the previous user viewing the same page.

So we extracted the diff and comment parts of the commit page into two separate parts, loaded using javascript when the page is loaded. This way, the full page is cheap to render, and the git data are loaded afterwards, in a payload that will be the same for every user.

Now all we had to do is to make sure these pages are rendered from cache. We’ve been running varnish in front of Apache on gitorious.org for quite a long time. Varnish uses some quite simple rules to determine whether contents can be cached or not. For our new pages this is what we needed to do to make Varnish serve them from the cache was:

• Set a Cache-Control header that tells Varnish for how long the content can be cached
• Make sure we’re not sending Set-Cookie headers from the Rails app, since Varnish by default will not cache data containing cookies

We just deployed this new code this afternoon, and have been watching the load on the server over the last few hours. Before we deployed it, the server was using all its Passenger processes to the full. After deploying the change, 2/3 of the processes have been idling, making Gitorious.org a whole lot snappier.

We’ll be putting more work into the performance of the web app over the next few weeks, so expect more posts like these in the feature. And as always, let us know if you’re having any issues, and we’ll help as good as we can!