Announcing Gitorious 3.0

Today we are proud and happy to announce the official release of Gitorious 3.0. This is a major upgrade which ships with a refreshed UI, a lot of improvements, and new features. It has been a huge effort by both the original Gitorious developers, who started the work on version 3.0 a year ago, as well as the new team who finalized the remaining tasks.

For Gitorious 3 we changed the entire UI – every single view has been updated, and many of the views have also been structurally revised and improved. This means that a few hundred ERB templates in a 6-years old rails app have been updated. Huh, wait, we actually did that? Sounds crazy. Anyway – trying to summarize all the UI changes is not really feasible, so I will focus on the most significant changes, and urge you to explore the rest on gitorious.org.

New repository browser

Our brand new repository browser comes with a nicer UI, improved syntax highlighter, ability to select lines of code and friendlier navigation. We also improved the diff pages, which now include a list of commits and a summary of changed files, in addition to the revamped inline commenting system.

New repository browser

Repository browser

Diff page

Diff page

You can read more about the new repository browser in the blog post about first beta of Gitorious 3.0.

New Merge Request UI

Another improvement that we’re really excited about is the new Merge request UI:

Merge Request page

Merge Request page

New dashboard and public user profile pages

The dashboard gives you quick access to everything you’re watching. In addition to that we also added “Your activities” tab so you can easily see what you’ve been doing recently without the need to browse to your public profile page (which also lists your activities).

New dashboard

Dashboard

Public profile page

Public profile page

Previously, links to user settings were accessible from the dashboard page, which was a bit confusing so we moved it to the pulldown at the top of the page:

Settings

Settings

New settings page

Speaking about the settings page – it is now split into easily accessible tabs so you can quickly access various settings like account information and SSH keys.

Settings page

Settings page

Service hooks with built-in integrations

The service hooks feature, described in another post, was extended with the support for built-in integrations for external services. This feature was sponsored by the fine folks at sprint.ly  – a beautiful project management application that we really like – so they are the first officially supported service.

Sprint.ly integration

Sprint.ly integration

The sprint.ly integration can be used to update stories in sprint.ly via commit messages – for more information check out the official guide here.

An integration for Jira is in the works, and many will follow in the time that comes. We also welcome contributions for your own favorite services and/or ideas to additional integrations.

Upgraded internals

Gitorious 3.0 changed a lot on the surface but it is also worth mentioning that it is now using Ruby 1.9.3 and Ruby on Rails 3.2.15 under the hood. This opens doors to upgrading to the latest Rails 4.0.x in the near future without too much of a hassle. We have also updated all 3rd party gem dependencies to the latest versions, incorporating security upgrades and other important patches. The result is a vastly modernized base, which is safer, more performant and more stable than Gitorious 2.

What’s next?

We, as the new gitorious team, are very excited about this release. It was a great opportunity for us to learn the codebase and we now feel very comfortable when working on Gitorious. Ambitious as we are though; no matter how happy we are with 3.0, we are already looking forward to start work on the next major features and improvements.

Here are some things we are planning:

  • An issue tracker that integrates nicely with other Gitorious features
  • Improved Merge Requests – making it even simpler to contribute and review code
  • Improved “activity stream” with better notifications (replacing the inbox feature)
  • Further improvements to the repository browser
  • Continued internal refactorings towards a clean Gitorious Ruby API that people can use to build new tools on top of Gitorious

And more will come. Yes, we’re gonna be busy.

One of our main goals going forward is to be innovative. We believe a lot can be done in a slightly or even completely different way resulting in a better experience. We also want to frequently push new releases, as working many months to achieve huge milestones is no fun. So – stay tuned and watch this space!

If you have wishes, ideas, suggestions now it’s a great time to let us know about them :)

How to install or upgrade?

The CE installer has been updated to install Gitorious version 3.0, and we’ve also put together an upgrade guide here. If you have questions or problems please let us know on the mailing list or drop us a line to support@gitorious.org. If you found a bug please report it in the issue tracker.

Thank you

We would like to thank the original Gitorious team for their amazing work – it was a real pleasure to work together for this short but fruitful period of time and we will do our best to take the project to the next level.

We also want to thank our community for the support we get and we hope you will enjoy using Gitorious 3.0 on gitorious.org.

Local installs

If you are running Gitorious 2 locally supported by our team, we will get in touch with you in the near future to discuss upgrading. If you are running Gitorious at your company or organization on your own and would like professional support for upgrades, maintenance or custom features, don’t hesitate getting in touch at customers@gitorious.com.

Future of the “Inbox” feature

The “Inbox” feature was added to Gitorious about 4 years ago and it was meant to give our users an easy way of communication. While we know it is a useful feature for many of you, we are also aware that it hasn’t gotten enough love recently. We’d love to know more about how you use it and get your opinion for its future with the form below. If you have 10 seconds to spare, please answer the 2 questions and hit Submit. Thanks!

Post mortem on yesterday’s downtime

Gitorious 3 has been in the next branch of Gitorious’ mainline repository since last autumn. We started out building a new code browser for Gitorious which would perform better than our previous solution, with server-side syntax highlighting of any programming language out there. At that time we had fallen behind on versions of Ruby on Rails, the framework Gitorious is built with, and we realized that we would have to build Gitorious 3 for Rails version 3.

The story with Rails 3 is a long one, and we’ll save that story for another occasion. But for some parts of Gitorious, for example our routing system, the Rails 3 upgrade took a long time. But we were finally able to run Gitorious under Rails 3, using Ruby 1.9. After Ruby 1.8 reached end of life earlier this year, this is now an important reason to upgrade.

As we started discussing the aquisition by Powow AS, Christian, Thomas and I wanted to give the new team a clean slate, with a version of Gitorious with less technical debt and a vastly improved UI. As part of the agreement with Powow we agreed that the old team would take care of upgrading the servers to Gitorious v3 around the time of the acquisition. Last week we picked yesterday as the final date of the deployment, and started planning for the upgrade.

D-day

When we started upgrading the servers yesterday morning, most things went according to plan. We followed the upgrade guide we had set up, and after half an hour we were able to view gitorious.org running Gitorious v3. After getting all the components of Gitorious running, we opened gitorious.org to the public around noon. We were expecting some errors to occur, due to the sheer volume of users and repositories on gitorious.org.  Running an application in test and staging environments is not the same as running in a producton environment with over half a million users. One of the first issues we addressed was that rendering of a project page was ridiculously slow; a bug we deployed a fix for within 30 minutes after upgrading.

One of the surprises we were not expecting, however, was that my profile page on gitorious.org rendered the international characters in my name wrong. Digging deeper, we found that although the Rails app was configured to use latin1 encoding when communicating with the database, we were not getting latin1 back. We tried changing the database configuration to use utf8, but discovered that the actual data in the database was wrong. In order to not have inconsistent data in the database, with some data using different encodings in the same database we decided to take gitorious.org offline.

With the help of our friends at Redpill Linpro, we tried dumping a copy of the database to a file and then importing that into the MySQL database, using utf-8 for the database dump. Looking through that file, the encoding looked really strange. Where my middle name in UTF-8 should be 7 bytes (\x4d\xc3\xa5\x72\x6e\x65\x73), it was actually 9 bytes in the database. Once the import had completed, the data was in the same state as it was before the import – broken. Seeing 4 byte characters where we should be seeing 2 byte characters made us believe  that the data in the database had been encoded twice. Running a double recode of the data proved us right:

SELECT CONVERT(CAST(CONVERT(fullname USING latin1) AS BINARY) USING utf8) FROM users where id=2;
Marius Mårnes Mathiesen

Now all we had to do was to find a way to recode it twice in the file produced by mysqldump, which (as always) could be done with a Perl one-liner. We ran this oneliner on a SQL file for a single table (extracted from the entire database dump using sed, of course) and used that to recreate the users table in our database. Naturally that didn’t work, so we had to set up the character set related settings in the MySQL server and client, which worked. We now had a strategy for how to recreate the database with all UTF-8 data.

After dropping the database and importing it from the database dump, processed by the Perl oneliner, we finally got Gitorious running again around 7:15pm yesterday. We got all the services running again, and re-opened access to gitorious.org at 7:30pm CET.

Bottom line

We’re all really sorry about this downtime. This is not the kind of service you should expect from us, and we appreciate your patience while we worked to resolve this.

The good news is that we wouldn’t have been able to resolve this situation without help from the new team. Database configuration and character encodings are not of the old team’s strongest skills, while Marcin used tools I personally never knew existed to read the actual byte values from a several GB SQL dump. Further good news: gitorious.org is now running the best release of Gitorious ever, the infrastructure is in a much better state, and the new team has a lot of plans to make Gitorious even better.

On behalf of the old team: please don’t blame “the new guys”, we’re the ones who messed up this time.

And again, we’re really sorry about this.

Powow AS acquires Gitorious AS

With Gitorious 3 just about to be released, we’ve got some more news that we’re really excited about.

The current Gitorious team has been working full time on Gitorious since January 2011, and over the last two and a half years we’ve managed to improve the quality, stability and feature set in Gitorious – while running Gitorious AS as a profitable business. The company today has three full time employees who work exclusively with the Gitorious software, gitorious.org, and supporting all the organizations out there depending on Gitorious for their day-to-day work.

Looking back, we’ve accomplished most of the goals we set out with when we founded the company.

However: looking forward, we realize that we’re in the same situation as many startups: we don’t have the energy and motivation to take Gitorious to the next level. The company is at a good place, but we strongly feel that Gitorious needs fresh blood and new ideas.

It was very important to us that our community and commercial customers would not be left hanging, and we started looking for a new steward for both the project and our company.

It was crucial to us that the new owner of the Gitorious project (and company) shared our most important values:

  • free/open source software

  • amazing support and customer service

  • hacker culture

  • devops/continuous delivery

  • test-driven development

  • being profitable as a result of doing what we love, not a goal in itself

We found the perfect match. Powow is a Norwegian-Polish Ruby consulting company which consists of some of our biggest heroes in the Ruby community. They will be in charge of the Gitorious project and Gitorious AS the company going forward.

Powow was founded in 2008 as a small software consultancy in Norway. Since then the company has expanded and built a great team of highly skilled and experienced developers. While originally a consulting shop, the company had already started shifting towards more product-centric work when we got in touch with them.

We’ve already started the transition, and while working with the new team over the last weeks we’ve been very impressed with their skill-set and values, not to mention how quickly they were able to start shipping features. Also, with several members of the team being long time Linux/UNIX users, we’re confident the operations and support offerings of the business is in even better hands than in the past. And as you can see, the new Gitorious team is bigger than the current one.

This is good news for both the Gitorious community and our enterprise customers. The new team has some great plans for the future, which I’m sure they’re anxious to share with you over the coming weeks.

Say hello to the new team

team_piotr Piotr Solnica – co-founder of Powow, software developer with almost a decade of experience. Ruby and JavaScript pro. Clean code and TDD evangelist. Open source hacker and Ruby Object Mapper core developer. Currently romancing with functional programming paradigms.
team_marcin Marcin Kulik – passionate software developer breathing HTTP since 2001. Ruby professional. Fluent in many other programming languages. Open standards proponent and open-source contributor. Linux geek. The man behind ascii.io.
team_adam Adam Pohorecki – aspiring software craftsman and TDD evangelist. Founder of SCKRK. Organizer and facilitator of code retreats and unconferences. Inventor of Ping-Pong Pomodoro Pair-Programming and bogus – a next generation Ruby mocking library.
team_pawel Pawel Pierzchala – Software Craftsman from Kraków. Code Retreats organizer and coach. While not coding or reading white papers for next SCKRK meetup, busy watching spaghetti westerns.
Mariusz Mariusz Ciesla, designer and wannabe developer. Really bad at maintaining balance on a skateboard.
team_rolf Rolf Bjaanes, developer, coffee geek, and as chief happiness officer of the company he’ll mainly be working on the business side of things. In his own words, “every developer’s hunt for the best editor ends up with Vim, Emacs or a management position.”

Next steps for Gitorious the open source project

The current team is focusing on finishing the work on Gitorious 3. It will be deployed on gitorious.org for battle-hardening next week, and will be made generally available with installers and the works shortly thereafter.

Once Gitorious 3 is out, you’ll start seeing more of the new team on the mailing list, IRC channel and blog as the Powow team transitions into being main committers and owners of the project. Development will commence with the new crew on-board. The AGPL licensing and terms of the open source project remains unchanged.

The main change for the Gitorious community is that the project will get an infusion of capable, proven, and outgoing open source developers with fresh ideas and energy. And they are dying to start talking with you about the roadmap going forward!

Next steps for Gitorious AS the company

Our customers keep telling us that the support they get from us is their main reason for choosing and staying with us. We know that a key part in providing good support for customers is trust and personal relations.

First and foremost, we’ll make sure service and support stays consistent and responsive while we pass the torch to the Powow team. We will get in touch with all our enterprise customers over the next few days to introduce the new team and make sure all concerns are addressed. Christian, Thomas and I will be working with the new team for as long as it takes for them to become fully self-sufficient with all aspects of the business.

Also, once Gitorious 3 is ready and has been proven stable, we’ll schedule time for upgrading all our customers’ servers. We will be doing this work together with the new team to ensure a smooth upgrade process.

As of today, Gitorious AS is formally 100% owned by Powow AS.

A final word from the founders

As this is probably the last post we’ll be writing on this blog, Thomas, Christian and I would like to thank the community, our customers and other supporters for some really good times over the last few years. Gitorious wouldn’t be where it is today without you. We trust that you’re in the best of hands going forward.

Gitorious will be upgraded on Monday [completed]

The time has finally come to deploy Gitorious 3.0 to the gitorious.org servers. We are planning to do the upgrade the coming Monday August 26th, starting at 10:00 AM CET. We expect a smooth upgrade process, but gitorious.org may be unavailable for up to two hours while we perform the upgrade.

We will post an update to this blog post once we’ve completed the upgrade and ensured all services are working normally again. Once we have performed the upgrade we will be constantly watching the performance and fix any issues that may occur due to the upgrade. Don’t hesitate to get in touch with us at support@gitorious.org or comment on this post if you have any questions.

If you’re curious about the changes you’ll be seeing after the upgrade, we have put up a preview site at https://v3.gitorious.org/.

Updates

  • 12:04 CET: We’re almost there, but need a little more time before we open the gates. We estimate another hour of downtime
  • 13:00 CET: The site is public, but dead slow. We’re hard at work taking care of the performance
  • 14:05 CET: We’re back, but we have some encoding issues in our database. We will probably have to take the site offline later today to convert the data in the database.
  • 14:!5 CET: We just took gitorious.org offline again while converting our database.
  • 14:50 CET: We’re currently importing the database, hopefully with the correct character sets
  • 16:40 CET: The character set conversion in the database didn’t work as expected, we’re currently investigating
  • 18:30 CET: We seem to have been able to convert our database. Currently preparing a restore
  • 18:35 CET: The database import is currently running, we expect to be back in around an hour.
  • 19:17 CET: Aaand we’re back up! We will write a post mortem on what happened today tomorrow.

Gitorious 3.0 Beta 1

It’s been quite a journey, but we pulled through, and we are excited to finally present you with the first beta of Gitorious 3.0!

Take it for a spin

We set up a trial instance on v3.gitorious.org. You should especially check out the new repository pages. This instance is free for all, so please register an account and test at will. Just note that it will be removed once gitorious.org is upgraded to Gitorious 3. There are already some known issues, and I will be posting updates to this demo throughout next week.

For those of you who want to run Gitorious 3 yourself, there will eventually be a pre-built CentOS-based VM available on getgitorious.com. Unfortunately, we haven’t had time to fix that up yet, and as we are now going into vacation mode, it will be a few weeks.

Your best option for self-hosting Gitorious 3 at this point is to install Gitorious 2, e.g. by using the Community Edition, and follow the manual upgrade instructions found in the repository.

For people wanting to hack on Gitorious, there are instructions in the repo for CentOS and Ubuntu for how to get the development environment up and running. These take some shortcuts that are not recommended for production deployments, but will get everything up and running.

What’s new?

The big news in Gitorious 3 is the new and improved code browser. In addition to being a big improvement UI-wise, it supports many new small useful features. The code browser also provides quick access to browsing and viewing files, blame, log for individual paths and more.

Browsable URLs

The old

/project/repository/blobs/*

and

/project/repository/trees/*

URLs are gone. In their place we now have a unified way of browsing source code through

/project/repository/source/ref:path

, e.g.

/project/repository/source/master:Readme.org

Automatic rendering of README in your repository

Gitorious will render any README file it can render in your repository.

DoltReadmes

Not only will it render READMEs in the root of your repository, but from any directory inside your repository.

Vastly improved syntax highlighting

The syntax highlighting in Gitorious 3 is powered by Pygments, arguably the best syntax highlighting toolkit around. This means Gitorious now supports every one of Pygments’ long list of supported languages.

syntax_highlighting

Highlight code

When browsing files, you can mouse over lines to highlight them. If you want to share highlights with other people, click a line, copy the URL and share at will. You can also highlight a region by clicking a line, holding shift and clicking another one. Copy the URL and off it goes.

line-nums

Convenient branch/ref selector

The code browser has a branch selector in the upper right corner, which is always available for any action where different branches/refs has different content (i.e. when browsing code, but also the log etc). This allows you to select branches and tags, or enter any Git oid to view it.

branch-selector

Curl-able downloads!

Gitorious has used a polling mechanism to serve tarballs for some time. In Gitorious 3, this is gone, and we use simple GETs to fetch tarballs. For high-traffic deployments, getgitorious.com has information on how to configure nginx so this scales well.

Detached code browser

The code browser shipping with Gitorious 3 is also released as a stand-alone tool that you can use to visualize your repositories locally. You can even use it for a light-weight read-only Git repository hosting service on your server. The tool is called Dolt, and if you have Ruby installed, you can try it out in 5 minutes: gem install dolt. Run either in a git repository, or in a directory that contains many git repositories: `cd /my/git/repo && dolt .`

dolt

Web hooks

In Gitorious 3, you can select the “admin” menu on your repositories when logged in, and it will offer you to manage web hooks. These are URLs that Gitorious will send a POST request to everytime someone pushes to the repository. Postbin is a great service for debugging these things, and here’s a sample payload from Gitorious.

webhooks

Log in with either email or login

In the past, Gitorious would only allow logins with email address. If you’re using the database backed authentication, you can now log in with either your email address, or your login/screen name.

Rails 3.2 under the hood

Gitorious 3 is based on Rails 3, which brings vastly improved security, performance and provides a better base for evolving the software. Some of you may have seen that Rails 4 was just released, and Gitorious 3.1 or 3.2 will likely be moving to Rails 4. When that happens, Ruby 1.8.7 (and Ruby Enterprise Edition) support is gone (Rails 4 does not support it).

Backwards and forwards compatibility

Gitorious 3.0 supports Ruby 1.8.7, primarily because many Linux distros still ship this version. Ruby 1.8.7 support should not be expected to last for long. We recommend that you run Gitorious 3.0 on Ruby 1.9.3, and Ruby 2.0.0 support is just around the corner (one of our dependencies is causing a segfault, but we have verified that everything will work once an update is released for it).

Improved Ruby API

The low-level APIs in Gitorious have been refactored and restructured in several areas. This work is not yet complete, and will continue throughout the summer. This will eventually make it easier to perform power-user/admin tasks from the console. Gitorious 3 ships with “use cases” that can be run from the command line, and encapsulates everything required to carry out certain tasks. Some documentation is available.

Git data mirroring

We’re currently finishing up a new feature that allows Gitorious to mirror all git data to one or more mirrors. This will help in cases where you want a “ready to go” failover server for Gitorious. This feature does not ship with the first beta, but will be in the final version.

What remains

This is a beta, and as such, bugs should be expected. We would love your help and support in testing this, and we will soon roll out the beta on gitorious.org. Once we’ve gotten through the worst crop of bugs, Gitorious 3.0 stable will be tagged and released. When that happens, everyone will be strongly urged to upgrade, as we will not provide updates for Gitorious 2 much longer.

You will also notice that while most of the repository pages sport a shiny new UI, we have not completed this transition entirely. Changing the UI all around is a big task, and we will continue iterating on this until the old UI is gone. We’re sorry that the app will have two faces in this transitional period, but we believe the improvements we’ve made already makes it more than worthwhile.

What do you think?

Try it and out, and let us know what you think. We are very excited about this release, and hope you will like it as much as we do.

On behalf of the Gitorious team,
Christian (@cjno).

Maintenance May 21st

The server center where the gitorious.org servers are located is being reorganized. From 1AM CET,  Tuesday May 21st, all gitorious.org services will be down for approximately 2 hours while the servers are relocated. This time is  night time European time, and evening time in the Americas, but we apologize for any inconveniences caused by this maintenance window.

Gitorious is featured on the GitMinutes podcast

GitMinutes is a fairly new podcast about Git, with weekly episodes featuring interviews with people doing all kinds of things with Git. This podcast is a great way to keep track of what’s happening in the Git community.

I’m in this week’s episode, talking about Gitorious and Git infrastructure. I had a great time chatting with Thomas from GitMinutes, and you’ll find the current episode here.

An update about Gitorious v3.0

When we merged the Rails 3 branch into next back in January it was our intention that this would become Gitorious 3.0, with few user-visible features. Our plan was to ship 3.1 shortly after, including the new code browser we started working on last year.

The upgrade to rails 3 was done mainly to enable us to run the code browser asynchronously, and we have put a lot of effort into making it possible to run an asynchronous web server alongside the Gitorious Rails application. Despite these efforts we were never able to get the stability we need with this setup. To make matters worse, the speed benefits from running asynchronously haven’t been as big as we had hoped. Because of this we have decided to make some changes to our plans:

Gitorious 3 will include the new code browser

We felt that shipping a new major without any major user-facing changes doesn’t make any sense. Since the updated code browser is so close to being merged, we’ll wait with tagging the 3.0 version until the new code browser has been merged into the next branch. We feel it’s worth waiting for:

New Gitorious UI sketch - syntax highlighting

The code browser in Gitorious 3 will not be asynchronous

We will change the code browser so it no longer runs asynchronously; rather it will be a Rack application running inside Gitorious. The git repository access is still done using libgit2/rugged, which gives great speed and stability gains, and we will finally get proper syntax highlighting courtesy of Pygments.

We hope to tag Gitorious 3.0 before the end of April, and will deploy it to gitorious.org as soon as it’s been tagged. It will feature:

  • Rails 3.2
  • Partial new UI
  • Significantly improved repository browser (Dolt)
  • New syntax highlighting, along with support for vast numbers of new languages
  • Readme-rendering for repositories
  • A JSON/HTTP based API, more details soon!
  • Ruby 1.9 support

Shortly after 3.0 lands we will keep working on propagating the UI upgrade to other parts of the application.

Gitorious v2.4.12 is released (security update)

Three new vulnerabilities have been fixed for Ruby on Rails, on which Gitorious is built. Read the original Ruby on Rails sec-list announcements for further details.

The steps for upgrading are, as usual (from within the root gitorious clone/source directory):

git fetch --tags

git merge v2.4.12

git submodule update --init

bundle install

We advise all users running their own Gitorious servers to upgrade immediately. Note that the Gitorious Community Edition installer has also been updated to install v2.4.12 now.

Follow

Get every new post delivered to your Inbox.

Join 718 other followers