Rails 3 upgrade

After a week of working on the Rails 3 upgrade, this is the current status: 3358 tests, 4712 assertions, 5 failures, 0 errors. That’s pretty close, and may I say, pretty exciting to us. Once the last 5 test failures have been fixed, there are still some aspects of the application that needs to be either verified manually, or added automatic tests for, but for now, things are looking good.

The upgrade has taken a little longer than initially expected, but good things are coming out of it, and the end is in sight. I’ll post a new update next week, at which point we should be back to working on the UI upgrades and the new repo browser.

Have a nice weekend!

Gitorious 2.3.2 released (small hotfix for installation/db setup issue)

We just tagged version 2.3.2 of Gitorious.

This is only a minor hotfix. The update resolves an intermittent issue in some Gitorious environments/database setups due to a database migration script from earlier this year. There is some discussion of the issue at the Gitorious Google Group.

From the version tag description:

Resolve issue with an older database migration.

An old migration (committed february 2012) caused intermittent issues
in some Gitorious installations depending on how the
environment/database is set up. This fix makes the db migration in
question play nice.

We’ve added a description of how to upgrade to this minor version in getgitorious.com/documentation (see Upgrades -> “Upgrading from 2.3.1 to 2.3.2″).

Short story: this is only an important change for you if you’re seeing the below message during setup or upgrade of your Gitorious rig:

==  MakeProjectMembershipPolymorphicOnSubject: migrating ======================

– add_column(:project_memberships, :content_type, :string)

   -> 0.0062s

rake aborted!

An error has occurred, all later migrations canceled

Mysql::Error: Table ‘gitorious_production.content_memberships’ doesn’t exist: SELECT * FROM `content_memberships`

Quick reboot of the servers Tuesday at 11AM CET [completed]

We’ll need to do a quick reboot of our servers to update their kernels tomorrow at 11AM CET. We expect it to take no more than 5 minutes, and will keep this post updated as we proceed.

Updates:

  • 11:05: We’re taking down the first server
  • 11:06: The servers are down
  • 11:12: The servers are back up again

UI upgrade and Rails 3

A while ago we announced some UI upgrades we are working on, and people have been asking when these changes will land in master. The short answer is that it will be a little while still. Read on for the longer answer.

If you’ve been following along, I’m sure you have seen our new repository browser. Dolt (a stand-alone tool to browse source code, view Readme files and more) is more or less done.

During the past couple of weeks I have made a couple of attempts at integrating Dolt back into Gitorious. Unfortunately these attempts have not yet been as successful as I’d hoped: there have been yaks to shave along the way.

Gitorious is currently based on a fairly old version of Rails. Working with old software imposes some constraints on what version of various third-party libraries can be used, which again causes trouble when trying to also use newer software (Dolt uses EventMachine and related libraries). In an attempt to resolve some issues related to this, I finally got my hands dirty and upgraded our Rails version from 2.3.5 to 2.3.14 last week.

This week, we decided to go one further and complete the long pending upgrade to Rails 3. I’m happy to report that this process is coming along nicely enough that we’re confident we’ll complete the upgrade next week. In doing this, we gain access to newer features in Rails that will help us in several ways. Most importantly, we will be able to integrate the source code browser properly.

We know that many of you are waiting for the new UI, and apologize for the delay. I wanted to post this update to let you know that we’re working on it, even if we have taken a detour to get there. When it all lands, we can all look forward to a more usable, prettier, safer and faster Gitorious.

Have a nice weekend!

LDAP authorization lands in Gitorious mainline

Gitorious started with a developer scratching his own itch. The first commit was done more than five years ago, and every commit since then has been about developers implementing features we need in a software tool we use all the time.

Gitorious is different from most other Git frontends and collaboration tools out there by being free software. Anyone using Gitorious is free to make changes to Gitorious itself; scratch their own itch. Over the last five years, 65 committers have committed to Gitorious and had their commits merged into the Gitorious mainline repository.

Gitorious is central in the software development process of hundreds of organizations, ranging from the very small to the very large. And as these organizations discover features they need in Gitorious, they are free to implement such features.

A year ago we announced three great new features in Gitorious, sponsored by a company using Gitorious internally. A few months later, we asked for help from the community in implementing support for private repositories in Gitorious, a feature which landed in Gitorious mainline last February.

LDAP Authorization

A few months ago, we were contacted by a company using Gitorious internally. They use LDAP for authentication, and wanted to discuss whether it was possible to use their LDAP server for authorization as well as authentication. They were willing to sponsor the development required to make this happen, and today we’re proud to announce that Gitorious now supports using an LDAP backend for authorization.

This is how it works

Granting access to a group of users in Gitorious is easy: you just define a Team of Gitorious users and grant access to your repositories and projects to that team. For a local Gitorious installation you would typically add Gitorious Teams for developers, QA/testing, project management and operations.

However, most larger organizations already has such groups defined in their LDAP/Active Directory directory, so duplicating this effort seems pointless. Furthermore, updating the Gitorious Teams as people join the company, change jobs or leave the company is a lot of extra work.

With LDAP authorization enabled, Gitorious no longer keeps track of which users are part of which teams. As users are added to and removed from LDAP groups, they will automatically be granted access (or have their access revoked) to any projects and repositories allowing access to those groups. As a project or repository owner you still grant access to teams, but the actual members of those teams are managed by your LDAP directory.

Since Gitorious needs to maintain the relationships between projects/repositories and (LDAP) Teams, you still define Teams in Gitorious – but those teams have LDAP groups as members, not users. With LDAP authorization enabled, each Gitorious Team has one or more LDAP groups as members, and any Gitorious user who is member of any of those LDAP groups will be granted access to anything allowed by that Gitorious Team. Technically, we switch the Team implementation in Gitorious between either database-backed or LDAP-backed teams based on whether LDAP authorization is enabled or not.

A scenario

Let’s say Bill is a new employee at BigCorp inc, and has never logged into their Gitorious server before. His LDAP username is bill, and he is member of the developers group in the LDAP directory. That group has commit access to the utilities repository in the tools project on the Gitorious server. Here’s what he needs to do to start committing to that repository:

  • Visit the login page on the server
  • Enter his LDAP username and password in the login form. The Gitorious server will try to authenticate him using the provided credentials. Once this succeeds, a new user record is created in the Gitorious database
  • Bill is prompted to upload an SSH key to the server
  • Bill can start pushing code to the repository

There is no registration step, no groups to update, it all Just Works™.

When is this available?

This feature was merged into the master branch of the Gitorious mainline repository a few days ago. If your organization uses LDAP we encourage you to try it out, and report any issues you find on the Gitorious issue tracker. We’ll release this as part of Gitorious v2.4.0 as soon as any issues have been fixed.

To get started, have a look at the sample authentication.yml file shipping with Gitorious. You’ll probably want to add the following options to config/authentication.yml:

  • bind_user: (username/password): a username/password to use for binding while looking up LDAP groups and memberships. Note: Specifying a bind user/password will cause Gitorious to use authenticated bind, another feature that has been lacking in Gitorious for some time.
  • membership_attribute_name: the name of the attribute your LDAP server uses to list groups a given user is member of.
  • members_attribute_name: the name of the attribute your LDAP server uses to list users who are member of a group.
  • base_dn: The base DN for users in your LDAP directory
  • group_search_dn: The base DN for groups in your LDAP directory

And in config/gitorious.yml, you’ll need to add:

  • use_ldap_for_authorization: true. The use_ldap_for_authorization option will replace the built-in Team provider with the LDAP Team provider.

Migrating to LDAP authorization

It is currently not recommended to simply turn on this feature on your existing Gitorious server, as the data in your database needs to be updated from the internal Team backend to the LDAP Team backend. However, setting up a new Gitorious server to try this out is easier than ever. Simply creating a database and connecting your Gitorious server to your LDAP server will allow you to start using Gitorious immediately, as user accounts are automatically created on first login.

We will start working on a tool to help you migrate to LDAP authorization, and hope to have this ready by the time we craft a new version of Gitorious supporting this new feature. We’ll be documenting how to use this feature and make it available at the Gitorious documentation site, launching really soon.

Maintenance window Thursday at 13:00 CET [DONE]

We’ll need to take down the servers for ~15 minutes the coming Thursday (October 25th) starting at 1PM CET to do another shot at migrating from our old SAN infrastructure. We’re sorry about the unforeseen troubles during our last two attempts at the same, but we should be better prepared this time.

We’ll update this blog post and our status site as we proceed.

Updates

  • 13:04: We’re taking down the servers
  • 13:15: We’re bringing up the first server temporarily, will be rebooted again afterwards
  • 13:28: We’re back up

Short maintenance Monday October 15th at 12 CET [complete]

We need to restart our servers the coming Monday starting 12:00 CET to reconfigure the storage pools on the physical hosts. We expect the process to take no more than 15 minutes, and will update this blog post as we proceed.

Updates:

  • 12:04 CET: We’re taking down the servers
  • 12:09 CET: The servers are down, SAN is being reconfigured on the host OS
  • 12:13 CET: We’re bringing the servers back up again
  • 12:16 CET: We’ll need to take down one of the servers again to verify the partition tables.
  • 12:25 CET: We’re seeing some messages in the logs that we’re not comfortable with; investigating
  • 12:35 CET: We’re rolling back, should be back up in a few minutes
  • 12:39 CET: We’re back up.

Gitorious 2.3.1 released

We just tagged version 2.3.1 of Gitorious.

To upgrade your installation, follow these instructions.

The highlights from the changelog:

Gitorious CLI scripts, unicorn/nginx support, bug fixes

This release introduces command line scripts for interacting with a
production Gitorious environment. These scripts reside in
$GITORIOUS_ROOT/bin, and (contrary to what’s in script) all will:

* Assume a production environment (specify RAILS_ENV for overriding
this)
* Switch to the user/group specified in gitorious.yml – no need use
su/sudo
* Set up the correct environment (no need for bundle exec)
* Change directory to your Gitorious root
* Will work with exec; you do not need to spawn a shell script to set up
environment variables etc. This means you can use these scripts directly
from cron, upstart, sysvinit, systemd

We also added support for running Gitorious under Nginx by sending
X-Accel-Redirect headers instead of the X-Sendfile headers used by
Apache with mod_xsendfile. Look at the frontend_server setting in
gitorious.yml and doc/templates/nginx.conf.sample for how to set up
Nginx.

Other bug fixes and minor improvements:
* use `git gc –auto` from script/repo_housekeeping
* start using a replacement pagination library
* provide templates for Upstart configuration files in
doc/templates/upstart.
* patch Rails to not send empty Set-Cookie headers when running behind
Unicorn. This makes Gitorious play nicely with eg. Varnish
* make rake resque:work load the processors
* provide a sample Unicorn configuration file

Scheduled downtime the coming Friday [DONE]

After migrating gitorious.org to new hardware last Friday, we need to take down the servers again to complete the transition by disconnecting the old SAN infrastructure from the servers.

This means gitorious.org will be unavailable the coming Friday October 5th starting at 12:00 CET. We hope to be down no more than 15 minutes, but we may need to perform integrity checks on the file systems; in which case the downtime will take a little longer.

We will update this post as we proceed, and also post updates on our status site and Twitter account.

Updates:

  • 12:00 We’re taking down the servers
  • 12:15 We’re currently reconfiguring the host machines
  • 12:30 This takes a little longer than expected, sorry about that.
  • 12:58 We hit some trouble and will revert to the old setup. Should be back within another 10-15 minutes
  • 13:03 And we’re back up. Sorry about the delay

Simplified Gitorious installation and getgitorious.com

We’re hard at work improving Gitorious this fall. In addition to developing the actual software we’re also making it a lot easier to get started with Gitorious.

So, today we’re launching a proper home page for the open source Gitorious project at getgitorious.com. We’re also rolling out a simplified Gitorious installer which makes it a snap to set up Gitorious on a standard server environment.

getgitorious.com

Previously, new adopters of Gitorious have been forced to jump straight to the code and wiki at gitorious.org. As of now, the Gitorious project home will instead be getgitorious.com.

This new project page will let new adopters instantly find installation methods, prebuilt Gitorious VMs, links to documentation, FOSS hosting on gitorious.org et cetera. We’re going to update our own webpages to link to http://getgitorious.com and we encourage you to do the same.

As mentioned earlier we’re also working on a new, consolidated and updated Gitorious documentation, which will be made available at getgitorious.com/documentation as we complete sections of it. It will deprecate parts of the gitorious.org/gitorious wiki pages and provide more up to date instructions than all the current guides and blog posts floating around on the web.

New, simplified installation

One of the strengths of Gitorious is that it’s open source, fully transparent, and consists of parts which you can mix, match and tweak to fit your own organization or team needs.

Unfortunately, setting it all up manually is not trivial and new Gitorious adopters often struggle with it. We figured it should be much, much easier to get started if you just need something standard up and going, so we’ve created a simplified Gitorious installer for CentOS servers (Debian, Ubuntu and Fedora support coming later).

By using the installer you can have a fresh Gitorious installation up and running within half an hour. The new installation procedure boils down to the following four commands:

yum install git
git clone git://gitorious.org/gitorious/ce-installer.git
cd ce-installer 
./install

At getgitorious.com/installer you’ll find both the text and video walkthrough, including an FAQ. Here’s the video walkthrough of the installation process:

Video walkthrough (Vimeo)

Please let us know what you think!

As always we love feedback: what else do you feel we could do to make Gitorious even easier to get started with?

Follow

Get every new post delivered to your Inbox.

Join 846 other followers