Private repositories

For a long time, people have asked that Gitorious support a fine-grained permissions system where repositories can be made private and access granted only to select users/groups. We have decided to develop this feature for Gitorious, and we need your help.

Some of you may know of the infamous merge request #115. This partially solves the problem at hand, but unfortunately not in a way that we can accept responsibility for. However, the discussion around this merge request spawned the idea of a fund-raiser, and that’s how we are going to do this.

How, what and when

Technically, this work will be based off of Rodrigo Rosas’ branch which replaces Gitorious’ authentication implementation with the Devise gem.

On top of this system we will implement role-based authorisation that extends to pushing and pulling git repositories, as well as scoping all user generated data on the web site (project information, repository activity, events of various kinds).

We will put as much authorization logic as possible in the generic layer already in place in Gitorious so that it can easily be utilized regardless of whether you are using database backed logins, LDAP or the upcoming Atlassian Crowd SSO support. This means you can use LDAP groups to manage access to Gitorious content.

We are ready to start this work early 2012, given proper funding. We have estimated this feature to roughly 4 weeks of work. Using our reduced hourly rate offered to Local Install customers, this comes out to $24000.

Private repositories will be developed and shipped as part of Gitorious mainline. It will not be offered as a free service on gitorious.org. This means we will ship some sort of configuration switch controlling whether or not this system is available in a given installation.

Is your company interested in using this feature? How about helping funding it? Get in touch, let us know the amount you would like to contribute, and if you have any specific features you would want us to account for. You can help fund this project anonymously if you so wish, but we recommend you allow us to tell the world what a great company you are for helping a free software project becoming even better!

Any contribution is appreciated. Email support@gitorious.org if you are interested. I will keep you guys updated on the donation progress here on the blog.

A little help from our friends

A few months ago, Axis Communications, the world’s leading expert in network video, got in touch with us to discuss some features they were missing in their local Gitorious installation. We agreed that the Gitorious team would develop these features on a consulting basis for them, and today we’re proud to announce that these features are available for all users of Gitorious; be it on gitorious.org or on one of the hundreds of self-hosted Gitorious installations out there.

We are especially excited to announce these features since they demonstrate one of the greatest benefits of free software: that features developed for one user of the software is made available to all other users of the software. We are very grateful to Axis Communications for hiring us to develop these features, and we’re equally grateful that they decided to share the new features with other Gitorious users. Thanks!

Here’s a short summary of what’s new:

Graphical log view

Once you work with more than a single branch in Git, you’ll feel the need to visualize your repository’s history. There are various tools on the desktop that do this.

Starting today, you’ll find a link from the commit log for any Git branch or tag in Gitorious to a graphical representation of the log. Here’s a screenshot of how it looks for Gitorious itself:

Click on the commit message to jump to a specific commit, click on the commit SHA1 to view a graph starting from that commit.

The graph does not require any browser plugins to work, and should work on any fairly modern web browser. We developed two tools that work together to generate the graph view:

  • capillary.rb generates a JSON data structure from Git’s log
  • capillary.js  renders the JSON output from capillary.rb in the browser using Raphaël. If you look at the source, you’ll find that capillary.js will output a lot of other formats as well, including ASCII.

Annotations, aka. blame

Ever found yourself wondering who on earth wrote a specific line in a program file and why? In that case you may have used Git’s blame tool, which lets you see which commit last changed each line in a file. Blame is now available in Gitorious too, whenever you’re viewing a file/blob in a repository. Simply hit the “Blame” link from the panes to the right above the file, and you’ll see the same contents along with some more details:

To the left you’ll see a link to the commit that changed this line, along with the name of the author and when the change was made. Toggle between normal blob view, blame and history using the panes at the top right.

Display diffs between several refs

One of the very first features in Gitorious was the ability to view the diff of a single commit. Since then we have added support for viewing the diffs introduced by a merge request, and today we’ll let you view the diff between any two Git refs (SHA1/branch/tag) in a repository. The first place you’ll find this is next to the activity log entry when someone pushes to a repository on Gitorious:

Here we can see that the user Goatlord pushed some commits to a repository, and there’s a link to view all the changes introduced in this push – which brings you here:

All the changes between two commits, displayed either inline or side-by-side. You’ll find links to compare branches and tags to each other from the tree views in Gitorious; simply select which tag or branch to compare from the right sidebar.

How to upgrade your own server

If you’re running your own Gitorious server, you have probably read about our versioning scheme on the Gitorious wiki. Today’s features are in our latest version, version 2.1.0. In addition to the normal steps (run bundler, remove cached javascript and css files) you’ll need to load a few Git submodules – “git submodule init && git submodule update” should take care of this. If you run into trouble upgrading, the mailing list is full of helpful people.

Over to you

First of all: have fun using these features, we hope you’ll find them useful. If you fix any bugs, don’t hesitate to submit a merge request. The same goes if you have developed any features for your own Gitorious server that you want to share. If there are other features you’d like to see in Gitorious and you want help in developing these, please get in touch with us. There is no big corporation behind Gitorious, neither do we charge any license fees that could pay our bills. It’s up to Gitorious’ users to help us evolve.

It’s time for the Git User’s Survey 2011!

The Git User’s Survey 2011 is now up!

Please devote a few minutes of your time to fill out the simple questionnaire; it’ll help the Git community understand your needs, what you like about Git (and what you don’t), and overall help us improve it.

The survey will be open from 5 September to 3 October 2011.

The results will be published at GitSurvey2011.

Emergency reboot today at 14.30 CET [done]

Due to some changes in Gitorious’ network infrastructure, we need to reboot the servers today at 14.30 CET. This should be completed before 14:35 – we’ll post an update here once we’re done.

Edit: As of 16:08, we’re back up and running.

Scheduled downtime this Wednesday

We need to add some more storage to our SAN infrastructure, which means our servers will be unavailable for approximately 1 hour Wednesday morning, July 27., from 8AM CET.

We will post updates via our identi.ca/Twitter accounts, and update this post once we’ve completed the upgrade. As always, you can check our status site to see the current status.

 

Edit: We’re back up.

Updates to our Terms of Service

When we originally wrote our terms of service, we wanted to make sure that gitorious.org could not get in trouble if our users published source code they were not allowed to publish; for instance if they didn’t have a license for the source code. Based on this, our terms of service stated that users submitting content to gitorious.org granted Gitorious “a world-wide, perpetual, royalty-free, and non-exclusive license(…)” to the content.

As someone pointed out on the Gitorious mailing list, most free software licenses prohibit sublicensing, which made our terms of service incompatible with these licenses. We have been working with our attourney and the Free Software Foundation over the past two months to come up with a terms of service that both limits Gitorious’ liabilities in case of copyright infringement and is compatible with free software licenses.

We have just posted an updated version of the Terms of Service that the FSF finds acceptable, and still protects Gitorious sufficiently. We believe the changes we have made are to our users’ benefit, but as a gitorious.org user you should review these changes.

If you have any comments or questions about these changes, please comment on this post.

Improved SSL support + IPv6

As of today, gitorious.org has vastly improved SSL support. You are now free to surf gitorious.org through https only, should you wish to do so. Previously, our SSL support has been restricted to a few select actions, and there has been redirects from https to http.

With today’s deployment, gitorious.org will never redirect you away from https (if it does, report a bug to support@gitorious.org). You will also be forced to use https as long as you’re logged in, and when posting forms (i.e. logging in).

For those of you who maintain your own Gitorious setups, this change is pretty straight forward. The new SSL feature is enabled by default, and can be controlled through the gitorious.yml setting use_ssl. When this setting is set to true, Gitorious will enforce SSL where appropriate. When it is false, Gitorious will actively ensure http.

We will follow up this change by adding HSTS shortly.

In other news, gitorious.org is also available on IPv6, thanks to our awesome hosting partner, Linpro. We’re still having some issues with the backend for git:// and http:// Git access, so for now they are IPv4 only. We are working to resolve this issue.

Today’s outage

Around 12:00 CET today, gitorious.org went down and was down for about 10 minutes. During this time, both the web site and git push/pull was unavailable. This is a short summary of what happened.

We have been having performance issues with our database over the last few months. The main bottleneck has been our events table, which keeps track of all kind of activity on gitorious.org. The events table even contained one row for each commit ever pushed to gitorious.org, information that is already available through Git itself. A few weeks ago, we pushed a change to Gitorious which means we no longer create an entry in the database for each commit, only the push that occurs – and the push event keeps track of which commits were included. The commit data is then fetched from the Git log.

This morning we raised the amount of memory allocated to our MySQL server, specifically the innodb_buffer_pool_size setting, and restarted the database server. To be on the safe side of things we reloaded the Passenger instances, to keep them from hanging on to dead database connections. Initially, things seemed to be okay, until the web app went down about 15 minutes later. We tracked this down to about 100 Passenger instances waiting on the global queue, and found the MySQL server busy inserting data into the sessions table. By now the familiar sound of face-meets-palm was getting louder: we have a sessions table? Apparently, we did. Having one and not knowing it means it kept quite a lot of data too. So we switched to cookie based sessions and restarted the server, and things seem to be running smoothly now.

The reason why the database server stopped responding – blaming the huge amounts of data on ourselves – is that the file system where MySQL keeps its data went full while the server was coming back up after the restart. We’ll be adding more disk there now.

Sorry about the downtime, it happened because we’re addressing the performance on gitorious.org. Changing things involve a risk of breaking them, which is what happened today.

Web performance, part 1

After experiencing some really heavy loads on the server last weekend (which caused some 503′s – sorry again) we pulled down some logs from the servers and applied some grep and sed magic on them. We were a little surprised by what we learned.

It turned out that over 20% of the requests to the servers were requests for a commit (Commits#show in Rails-speak). You guys seem to use Gitorious to read code! The thing is that all Git activity is quite expensive, so these 20-something percent requests accounted for 35% of the server’s rendering time.

The chance of a Git commit identified by a SHA changing over time is fairly low, which means it’s a very good candidate for caching. The only issue is that Gitorious displays quite a lot of personalized content, depending on whether you’re logged in or not: the top part of the page displays your username and unread message count, for instance. So if we naïvely cached it, the second user requesting a commit would receive the page rendered for the previous user viewing the same page.

So we extracted the diff and comment parts of the commit page into two separate parts, loaded using javascript when the page is loaded. This way, the full page is cheap to render, and the git data are loaded afterwards, in a payload that will be the same for every user.

Now all we had to do is to make sure these pages are rendered from cache. We’ve been running varnish in front of Apache on gitorious.org for quite a long time. Varnish uses some quite simple rules to determine whether contents can be cached or not. For our new pages this is what we needed to do to make Varnish serve them from the cache was:

  • Set a Cache-Control header that tells Varnish for how long the content can be cached
  • Make sure we’re not sending Set-Cookie headers from the Rails app, since Varnish by default will not cache data containing cookies

We just deployed this new code this afternoon, and have been watching the load on the server over the last few hours. Before we deployed it, the server was using all its Passenger processes to the full. After deploying the change, 2/3 of the processes have been idling, making Gitorious.org a whole lot snappier.

We’ll be putting more work into the performance of the web app over the next few weeks, so expect more posts like these in the feature. And as always, let us know if you’re having any issues, and we’ll help as good as we can!

We’re in business. Literally

As mentioned in a previous blog post, gitorious.com is getting some new content. We’re launching the new gitorious.com today, and we’re really excited about it.

Gitorious started out as Johan Sørensen scratching his own itch almost four years ago. After steadily getting more users and projects on board, Gitorious grew into a business in 2009 when our friends at Nokia helped us provide some great new features – and Shortcut AS took on the responsibility of running Gitorious as a business. Starting January 2011, gitorious.org and the development of Gitorious as a project is done by the separate entity Gitorious AS, employing two full time employees.

Today, we (as in Gitorious AS) are announcing some new commercial offerings we think will both facilitate the continued success of gitorious.org and help companies use Gitorious internally. Our new web site provides more information about our offerings, so please have a look!

Gitorious.org will continue being free for FOSS projects. This is also the main reason we have no current plans of offering support for private projects on gitorious.org.

Companies and organizations who want to support us in offering this service for free to the free software community should consider our subdomain offering, which is a way to sponsor gitorious.org. For companies using or considering using Gitorious internally, our local install offering includes the services required to make this as painless as possible. Please note that we will keep working on simplifying the installation process of Gitorious, like we did when we added the Bundler library to our software stack.

Finally, we’d like to thank you all for helping Gitorious become the success it has become. There are tens of thousands of users using Gitorious every day, and exciting new projects turn up on gitorious.org all the time. We’re thrilled to have the oppurtunity of improving the service and keep adding features that makes life easier for FOSS developers.

Follow

Get every new post delivered to your Inbox.