UI upgrade and Rails 3

A while ago we announced some UI upgrades we are working on, and people have been asking when these changes will land in master. The short answer is that it will be a little while still. Read on for the longer answer.

If you’ve been following along, I’m sure you have seen our new repository browser. Dolt (a stand-alone tool to browse source code, view Readme files and more) is more or less done.

During the past couple of weeks I have made a couple of attempts at integrating Dolt back into Gitorious. Unfortunately these attempts have not yet been as successful as I’d hoped: there have been yaks to shave along the way.

Gitorious is currently based on a fairly old version of Rails. Working with old software imposes some constraints on what version of various third-party libraries can be used, which again causes trouble when trying to also use newer software (Dolt uses EventMachine and related libraries). In an attempt to resolve some issues related to this, I finally got my hands dirty and upgraded our Rails version from 2.3.5 to 2.3.14 last week.

This week, we decided to go one further and complete the long pending upgrade to Rails 3. I’m happy to report that this process is coming along nicely enough that we’re confident we’ll complete the upgrade next week. In doing this, we gain access to newer features in Rails that will help us in several ways. Most importantly, we will be able to integrate the source code browser properly.

We know that many of you are waiting for the new UI, and apologize for the delay. I wanted to post this update to let you know that we’re working on it, even if we have taken a detour to get there. When it all lands, we can all look forward to a more usable, prettier, safer and faster Gitorious.

Have a nice weekend!

LDAP authorization lands in Gitorious mainline

Gitorious started with a developer scratching his own itch. The first commit was done more than five years ago, and every commit since then has been about developers implementing features we need in a software tool we use all the time.

Gitorious is different from most other Git frontends and collaboration tools out there by being free software. Anyone using Gitorious is free to make changes to Gitorious itself; scratch their own itch. Over the last five years, 65 committers have committed to Gitorious and had their commits merged into the Gitorious mainline repository.

Gitorious is central in the software development process of hundreds of organizations, ranging from the very small to the very large. And as these organizations discover features they need in Gitorious, they are free to implement such features.

A year ago we announced three great new features in Gitorious, sponsored by a company using Gitorious internally. A few months later, we asked for help from the community in implementing support for private repositories in Gitorious, a feature which landed in Gitorious mainline last February.

LDAP Authorization

A few months ago, we were contacted by a company using Gitorious internally. They use LDAP for authentication, and wanted to discuss whether it was possible to use their LDAP server for authorization as well as authentication. They were willing to sponsor the development required to make this happen, and today we’re proud to announce that Gitorious now supports using an LDAP backend for authorization.

This is how it works

Granting access to a group of users in Gitorious is easy: you just define a Team of Gitorious users and grant access to your repositories and projects to that team. For a local Gitorious installation you would typically add Gitorious Teams for developers, QA/testing, project management and operations.

However, most larger organizations already has such groups defined in their LDAP/Active Directory directory, so duplicating this effort seems pointless. Furthermore, updating the Gitorious Teams as people join the company, change jobs or leave the company is a lot of extra work.

With LDAP authorization enabled, Gitorious no longer keeps track of which users are part of which teams. As users are added to and removed from LDAP groups, they will automatically be granted access (or have their access revoked) to any projects and repositories allowing access to those groups. As a project or repository owner you still grant access to teams, but the actual members of those teams are managed by your LDAP directory.

Since Gitorious needs to maintain the relationships between projects/repositories and (LDAP) Teams, you still define Teams in Gitorious – but those teams have LDAP groups as members, not users. With LDAP authorization enabled, each Gitorious Team has one or more LDAP groups as members, and any Gitorious user who is member of any of those LDAP groups will be granted access to anything allowed by that Gitorious Team. Technically, we switch the Team implementation in Gitorious between either database-backed or LDAP-backed teams based on whether LDAP authorization is enabled or not.

A scenario

Let’s say Bill is a new employee at BigCorp inc, and has never logged into their Gitorious server before. His LDAP username is bill, and he is member of the developers group in the LDAP directory. That group has commit access to the utilities repository in the tools project on the Gitorious server. Here’s what he needs to do to start committing to that repository:

  • Visit the login page on the server
  • Enter his LDAP username and password in the login form. The Gitorious server will try to authenticate him using the provided credentials. Once this succeeds, a new user record is created in the Gitorious database
  • Bill is prompted to upload an SSH key to the server
  • Bill can start pushing code to the repository

There is no registration step, no groups to update, it all Just Works™.

When is this available?

This feature was merged into the master branch of the Gitorious mainline repository a few days ago. If your organization uses LDAP we encourage you to try it out, and report any issues you find on the Gitorious issue tracker. We’ll release this as part of Gitorious v2.4.0 as soon as any issues have been fixed.

To get started, have a look at the sample authentication.yml file shipping with Gitorious. You’ll probably want to add the following options to config/authentication.yml:

  • bind_user: (username/password): a username/password to use for binding while looking up LDAP groups and memberships. Note: Specifying a bind user/password will cause Gitorious to use authenticated bind, another feature that has been lacking in Gitorious for some time.
  • membership_attribute_name: the name of the attribute your LDAP server uses to list groups a given user is member of.
  • members_attribute_name: the name of the attribute your LDAP server uses to list users who are member of a group.
  • base_dn: The base DN for users in your LDAP directory
  • group_search_dn: The base DN for groups in your LDAP directory

And in config/gitorious.yml, you’ll need to add:

  • use_ldap_for_authorization: true. The use_ldap_for_authorization option will replace the built-in Team provider with the LDAP Team provider.

Migrating to LDAP authorization

It is currently not recommended to simply turn on this feature on your existing Gitorious server, as the data in your database needs to be updated from the internal Team backend to the LDAP Team backend. However, setting up a new Gitorious server to try this out is easier than ever. Simply creating a database and connecting your Gitorious server to your LDAP server will allow you to start using Gitorious immediately, as user accounts are automatically created on first login.

We will start working on a tool to help you migrate to LDAP authorization, and hope to have this ready by the time we craft a new version of Gitorious supporting this new feature. We’ll be documenting how to use this feature and make it available at the Gitorious documentation site, launching really soon.

Maintenance window Thursday at 13:00 CET [DONE]

We’ll need to take down the servers for ~15 minutes the coming Thursday (October 25th) starting at 1PM CET to do another shot at migrating from our old SAN infrastructure. We’re sorry about the unforeseen troubles during our last two attempts at the same, but we should be better prepared this time.

We’ll update this blog post and our status site as we proceed.

Updates

  • 13:04: We’re taking down the servers
  • 13:15: We’re bringing up the first server temporarily, will be rebooted again afterwards
  • 13:28: We’re back up

Short maintenance Monday October 15th at 12 CET [complete]

We need to restart our servers the coming Monday starting 12:00 CET to reconfigure the storage pools on the physical hosts. We expect the process to take no more than 15 minutes, and will update this blog post as we proceed.

Updates:

  • 12:04 CET: We’re taking down the servers
  • 12:09 CET: The servers are down, SAN is being reconfigured on the host OS
  • 12:13 CET: We’re bringing the servers back up again
  • 12:16 CET: We’ll need to take down one of the servers again to verify the partition tables.
  • 12:25 CET: We’re seeing some messages in the logs that we’re not comfortable with; investigating
  • 12:35 CET: We’re rolling back, should be back up in a few minutes
  • 12:39 CET: We’re back up.

Gitorious 2.3.1 released

We just tagged version 2.3.1 of Gitorious.

To upgrade your installation, follow these instructions.

The highlights from the changelog:

Gitorious CLI scripts, unicorn/nginx support, bug fixes

This release introduces command line scripts for interacting with a
production Gitorious environment. These scripts reside in
$GITORIOUS_ROOT/bin, and (contrary to what’s in script) all will:

* Assume a production environment (specify RAILS_ENV for overriding
this)
* Switch to the user/group specified in gitorious.yml – no need use
su/sudo
* Set up the correct environment (no need for bundle exec)
* Change directory to your Gitorious root
* Will work with exec; you do not need to spawn a shell script to set up
environment variables etc. This means you can use these scripts directly
from cron, upstart, sysvinit, systemd

We also added support for running Gitorious under Nginx by sending
X-Accel-Redirect headers instead of the X-Sendfile headers used by
Apache with mod_xsendfile. Look at the frontend_server setting in
gitorious.yml and doc/templates/nginx.conf.sample for how to set up
Nginx.

Other bug fixes and minor improvements:
* use `git gc –auto` from script/repo_housekeeping
* start using a replacement pagination library
* provide templates for Upstart configuration files in
doc/templates/upstart.
* patch Rails to not send empty Set-Cookie headers when running behind
Unicorn. This makes Gitorious play nicely with eg. Varnish
* make rake resque:work load the processors
* provide a sample Unicorn configuration file

Scheduled downtime the coming Friday [DONE]

After migrating gitorious.org to new hardware last Friday, we need to take down the servers again to complete the transition by disconnecting the old SAN infrastructure from the servers.

This means gitorious.org will be unavailable the coming Friday October 5th starting at 12:00 CET. We hope to be down no more than 15 minutes, but we may need to perform integrity checks on the file systems; in which case the downtime will take a little longer.

We will update this post as we proceed, and also post updates on our status site and Twitter account.

Updates:

  • 12:00 We’re taking down the servers
  • 12:15 We’re currently reconfiguring the host machines
  • 12:30 This takes a little longer than expected, sorry about that.
  • 12:58 We hit some trouble and will revert to the old setup. Should be back within another 10-15 minutes
  • 13:03 And we’re back up. Sorry about the delay

Simplified Gitorious installation and getgitorious.com

We’re hard at work improving Gitorious this fall. In addition to developing the actual software we’re also making it a lot easier to get started with Gitorious.

So, today we’re launching a proper home page for the open source Gitorious project at getgitorious.com. We’re also rolling out a simplified Gitorious installer which makes it a snap to set up Gitorious on a standard server environment.

getgitorious.com

Previously, new adopters of Gitorious have been forced to jump straight to the code and wiki at gitorious.org. As of now, the Gitorious project home will instead be getgitorious.com.

This new project page will let new adopters instantly find installation methods, prebuilt Gitorious VMs, links to documentation, FOSS hosting on gitorious.org et cetera. We’re going to update our own webpages to link to http://getgitorious.com and we encourage you to do the same.

As mentioned earlier we’re also working on a new, consolidated and updated Gitorious documentation, which will be made available at getgitorious.com/documentation as we complete sections of it. It will deprecate parts of the gitorious.org/gitorious wiki pages and provide more up to date instructions than all the current guides and blog posts floating around on the web.

New, simplified installation

One of the strengths of Gitorious is that it’s open source, fully transparent, and consists of parts which you can mix, match and tweak to fit your own organization or team needs.

Unfortunately, setting it all up manually is not trivial and new Gitorious adopters often struggle with it. We figured it should be much, much easier to get started if you just need something standard up and going, so we’ve created a simplified Gitorious installer for CentOS servers (Debian, Ubuntu and Fedora support coming later).

By using the installer you can have a fresh Gitorious installation up and running within half an hour. The new installation procedure boils down to the following four commands:

yum install git
git clone git://gitorious.org/gitorious/ce-installer.git
cd ce-installer 
./install

At getgitorious.com/installer you'll find both the text and video walkthrough, including an FAQ. Here's the video walkthrough of the installation process:

Video walkthrough (Vimeo)

Please let us know what you think!

As always we love feedback: what else do you feel we could do to make Gitorious even easier to get started with?

Short maintenance the coming Thursday [DONE]

As part of making Gitorious snappier, we’ll be migrating our servers the coming Thursday September 27th starting at 12:00 CET.

All services will be unavailable for a period of less than 15 minutes, and we will update this blog post and our status site as the upgrade proceeds.

Updates:

  • 12:02 The servers are going down
  • 12:08 The servers are being migrated
  • 12:12 The servers are coming back up
  • 12: 13 The servers are back up!

Update on the Gitorious UI refurbish

TL;DR: You can use Dolt now (scroll to the bottom of this post), and we’ll start integrating it in Gitorious mainline next week.

Some of you are anxiously awaiting the UI upgrades we announced in August, so I thought I’d provide a quick update.

As previously announced, I’m currently working on a new code browser, called Dolt. Because we’re redoing not only the UI part of the code browser, but essentially the whole thing, this has taken sore more time than expected.

Dolt is nearing completion of its initial version. It now supports syntax highlighted blobs, tree browsing, blame, file history and raw blobs. Oh, and markup rendering (e.g. for Readme files). The remaining pieces are things like a branch/tag selector, loading in commit data in the tree browser and other “minor” things.

Because I will be teaching a course for the rest of this work, I don’t expect Dolt to be “done” (as in phase 1 done) until next week. When that happens, I will start working on removing the existing code browsing code in Gitorious, and integrating Dolt in its place.

For those of you interested in taking a look, you can easily use Dolt locally now. All you need is to have Ruby installed.

# Install
gem install dolt

# Run on a git repository
dolt .

If you want your markdown-formatted readme-files to be rendered as HTML, you also want the redcarpet gem:

gem install redcarpet

Other markup formats, such as the ever awesome org-mode, are supported the same way:

  • .textile -- gem install RedCloth
  • .rdoc -- gem install rdoc -v 3.6.1
  • .org -- gem install org-ruby
  • .creole -- gem install creole
  • .mediawiki -- gem install wikicloth
  • .rst -- easy_install docutils
  • .asciidoc -- brew install asciidoc
  • .pod -- Pod::Simple::HTML
    comes with Perl >= 5.10. Lower versions should install Pod::Simple from CPAN.

See also the GitHub markup docs.

Simple backup and recovery with the ‘snapshot’ command

Backup, disaster recovery and migrations of Gitorious has been a somewhat manual process up till now. Though all you really need to do is to dump the mysql database and save the repository folders, setting up and performing those backup tasks has been an ad-hoc task for each installation. Furthermore any disaster recovery work has been quite the forensic process.

To address this we’ve added convenience commands for simple snapshots and restore of Gitorious installations. The snapshot/restore commands are available on the Gitorious mainline now, and are present if you update to the newly released v2.3.

Using the snapshot/restore commands

If you’re on a recent version of Gitorious you’ll find two new commands under the scripts directory: scripts/snapshot and scripts/restore. Each of them takes a single parameter: the path of the tar file you want your Gitorious installation backed up to, or restored from. Note that you'll need to launch the commands from the root directory of your Gitorious install (the directory where you find your Rakefile, Gemfile, config directory etc).

How to perform snapshots in older Gitorious versions

It's fairly straightforward to add these new snapshot|restore commands to an older Gitorious instance a well: you only need to clone the latest version of the Gitorious code, and copy the following files to the same locations in your current, older installation:

script/restore

script/snapshot  

lib/tasks/backup.rake

Assumptions and caveats

  • For disaster recovery, you'll first need to get a functional installation of Gitorious up and running, after which you can run the restore command to bring in your data again. Note that the snapshot tarball also includes the configuration files from your old Gitorious installation. They are not automatically copied into your new installation, but can be used to recreate your old config in your new Gitorious installation (useful if your previous Gitorious installation had custom LDAP integration, custom hooks etc).
  • Snapshots will not preserve any custom code or theming you may have added to your Gitorious installation: any such local modifications will need to be backed up and restored separately
  • Some operations in Gitorious are asynchronous, meaning they are performed by publishing/consuming messages on a queue. The snapshot/restore commands don't currently capture or restore the state of the queue, so any current work on the queue will be lost.
  • The snapshot/recovery commands assume that you have the time and disk-space to slurp down all your hosted repos into a local tarball. Sites with huge amounts of git repository data may therefore need more custom backup schemes.
  • The restore command assumes that no breaking changes have happened between the version you snapshot from, and the version you restore your data into. In the future, major Gitorious version jumps may necessitate a more manual restore procedure due to changes in configurations, db schema, folder structure etc.

Usage scenarios

The snapshot|restore commands are useful for multiple tasks beyond just standard backups.

Backup and disaster recovery

Run periodic backups, for instance via cron. Execute the snapshot command, for example like this script/snapshot /tmp/todays-snapshot.tar. Transfer the newly created tarball to another server, offsite location or something like Amazon S3.

If disaster strikes on your Gitorious installation, install a new instance of Gitorious, then run script/restore todays-snapshot.tar to restore your old state in the new installation.

Migrating

The same procedure is useful if you just need to move or clone your Gitorious state from one server to another. Snapshot the old one, copy the tarball file over to the other installation and restore there. Simple.

Upgrading

Different scenario, same procedure: if you need to upgrade from an older version of Gitorious and don't want to bother with carefully upgrading your Gitorious instance across multiple versions, you can simply snapshot the state of your old Gitorious site, install a fresh new Gitorious instance and just restore your state there.

Snapshots and rollbacks for development and testing

If you're a developer extending, customizing or developing new features for Gitorious, you can use the snapshot|restore commands to simply setting up and restoring multiple standard database/repository states in order to simplify testing.

Let us know if you hit any snags

We've been testing and using these features internally at Gitorious AS for a while so they should be reliable by now, but do let us know if you find any issues.

Follow

Get every new post delivered to your Inbox.

Join 811 other followers