Announcing HTTP(S) pushing support

We’re happy to announce that Gitorious just added support for pushing over HTTP(S) protocol in addition to the existing pushing over SSH. This is especially good news for developers whose organizations’ network policies don’t allow for outgoing SSH connections and for people traveling (for the same, network related reason).

You can do the following now:

git clone https://gitorious.org/project/repository.git
cd repository
... make changes and commit ...
git push

You will be asked for your Gitorious username and password. You can use email instead of username if you prefer. It can be annoying to enter your credentials on every push and there are ways to automate this. You can use git’s built in credentials helper or you can add the following lines to .netrc file (_netrc on Windows) in your home directory:

machine gitorious.org
 login <username>
 password <password>

Because you save your password in plain text make sure .netrc has minimal permissions.

HTTP pushing is now enabled for all repositories on gitorious.org and it will be part of Gitorious 3.2 (soon to be released).

For you geeks out there: HTTP access to repositories is now handled by gitorious-http-backend, a HTTP server wrapping git-http-backend, adding authorization and concurrency on top of it. For more implementation details feel free to browse the source code.

Security vulnerability in bash

Yesterday it was announced that a critical vulnerability has been discovered in GNU Bash. By exploiting the vulnerability a user can execute arbitrary commands on the server. While these commands may not run with root privileges it’s still a dangerous attack vector.

There are several ways to exploit the vulnerability, one of them is especially dangerous to Gitorious. Gitorious provides git repository access over ssh using “git” user that by default has its shell set to bash. It is thus possible for a user with Gitorious account who uploaded public key via Gitorious web interface to execute malicious commands on the server.

We have patched gitorious.org yesterday to fix this vulnerability and we’re keeping an eye on git-over-ssh access on our servers. We’re contacting our Gitorious Enterprise Edition customers regarding this security issue. If you’re managing your own instance of Gitorious it is advised to upgrade bash package ASAP.

Golang patterns for serving on-demand, generated content

We’re using Go for some parts of the Gitorious project and one of the recently open sourced sub-projects written in Go is git-archive-daemon. It’s a scalable, high-performance HTTP API for serving archives of git repositories.

While implementing this tool we noticed several apparent patterns emerge which are not specific to git archive generation and can be applied to other use cases. For example on-demand image or sound generation, or any other kind of synchronous generation where a client waits on the connection for the response can benefit from these patterns.

This tutorial will go through these patterns, starting with the simple, naive implementation, gradually making it smarter and more powerful.

Note: All the code examples below lack proper error handling. This is on purpose, to make the examples shorter and easier to understand. Feel free to browse through the source of git-archive-daemon to see how we handle errors there.

Generating files on demand – the naive way

While the following techniques can be applied to any type of generated content let’s use file generation as an example.

We’ll be requesting a file by its “virtual filename” using query parameter filename. The simplest way of generating a file on demand and serving it could be this:

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
        filename := req.URL.Query().Get("filename")
        path := workHard(filename) // takes "long" time
        http.ServeFile(w, req, path)
    })
    log.Fatal(http.ListenAndServe(":5000", nil))
}

workHard function does the actual work and returns a path on disk to the generated file. It doesn’t matter how this function is implemented. We know it takes time though. ServeFile from net/http package sends the file back to the client. Looks simple and it works… until you get lots of simultaneous requests.

Note, that ListenAndServe starts a new goroutine for each incoming request. This is desired in most cases and it allows handling hundreds of thousands of simultaneous connections which is great when response generation is relatively light and fast. However, in this case where we’re dealing with CPU heavy or I/O heavy task it can easily lead to DOS-ing the server.

Pattern 1: Processing requests with a worker pool

In order to limit the amount of heavy work that is executed to fulfill the incoming requests we can use a pool of workers. By having a fixed size pool we can ensure that the amount of work happening at any given time doesn’t exceed a specified threshold.

Let’s introduce Request struct and a worker pool:

type Request struct {
    HttpRequest *http.Request
    ResponseWriter http.ResponseWriter
}

func WorkerPool(n int) chan *Request {
    requests := make(chan *Request)

    for i:=0; i<n; i++ {
        go Worker(requests)
    }

    return requests
}

func Worker(requests chan *Request) {
    for request := range requests {
        filename := request.HttpRequest.URL.Query().Get("filename")
        path := workHard(filename)
        http.ServeFile(request.ResponseWriter, request.HttpRequest, path)
    }
}

WorkerPool creates and returns a channel on which it receives the incoming requests to process. Every worker reads new requests from this channel, does the heavy lifting, and finally serves the resulting file back to the client.

Let’s create a Server struct that implements http.Handler interface and connect everything together in main func:

type Server struct {
    Requests chan *Request
}

func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    s.Requests <- &Request{HttpRequest: req, ResponseWriter: w}
}

func main() {
    requests := WorkerPool(5)
    server := &Server{Requests: requests}
    log.Fatal(http.ListenAndServe(":5000", server))
}

Request handling logic is trivial – it just sends every request over the channel and the real request processing happens in the worker pool.

But it doesn’t work. You get empty response body for every request. Why? There is a gotcha here. When ServeHTTP returns, ListenAndServe closes the network connection immediately, so we need to block ServeHTTP until we have response ready.

Let’s fix the code then. First, Request struct:

type Request struct {
    Filename string // changed
    ResultChan chan string // changed
}

Instead of having http.Request and http.ResponseWriter in our Request struct we have Filename, the only information from the request we need to do the job, and ResultChan, a channel from which we’ll read the job result (a path on disk) when it’s ready.

We also need to update ServeHTTP to receive the result from this channel:

func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    filename := req.URL.Query().Get("filename")
    request := &Request{Filename: filename, ResultChan: make(chan string)}
    requests <- request
    path := <-request.ResultChan // this blocks
    http.ServeFile(w, req, path)
}

The last thing to do is to update the worker to send the result over the ResultChan instead of directly sending the response to the client:

func Worker(requests chan *Request) {
    for request := range requests {
        path := workHard(request.Filename)
        request.ResultChan <- path // changed
    }
}

This is it. It works as expected now, properly sending the responses back to the clients while limiting the amount of work in progress.

Note that alternatively you can limit the number of requests using a tool like Haproxy. However doing this the way explained above is more flexible, as we’ll see in the next section.

Pattern 2: Request grouping

Let’s assume the following:

  • Some of the requests are exactly the same (they trigger the same work and response)
  • Server gets hundreds of requests per second
  • Generating a response takes 5+ seconds

With the current implementation we pass every new request onto the processing queue (requests channel), whether there is a response being currently generated for similar request or not. This is not ideal because we’re doing the same work multiple times. That is a waste of resources. Let’s fix that.

We’ll update our implementation to group similar requests together and send the response to all of the related clients immediately once the work is done.

To do that let’s first make a distinction between a “request” and a “job”. We introduce Job, a struct that holds a filename (a job payload), and the result. We also embed Job inside of a Request:

type Job struct {
    Filename string
    Result string
}

type Request {
    Job *Job // changed
    ResultChan chan string
}

Workers will now operate on Job structs, reading new jobs from the jobs channel and sending processed jobs to the results channel:

func WorkerPool(n int) (chan *Job, chan *Job) {
    jobs := make(chan *Job)
    results := make(chan *Job)

    for i:=0; i<n; i++ {
        go Worker(jobs, results)
    }

    return jobs, results
}

func Worker(jobs chan *Job, results chan *Job) {
    for job := range jobs {
        job.Result = workHard(job.Filename)
        results <- job
    }
}

Next, we need to group the requests by their jobs’ payload (Filename). Let’s have a “request queue” (a slice of Requests) for each unique filename and make sure we only schedule a single job for a queue.

This logic will live in a function named RequestMux as it effectively multiplexes requests over a pool of workers. Here is how it looks:

func RequestMux(jobs chan *Job, results chan *Job) chan *Request {
    requests := make(chan *Request)

    go func() {
        queues := make(map[string][]*Request)

        for {
            select {
            case request := <-requests:
                job := request.Job
                queues[job.Filename] = append(queues[job.Filename], request)

                if len(queues[job.Filename]) == 1 { // the one we appended is the first one
                    go func() {
                        jobs <- job
                    }()
                }

            case job := <-results:
                for _, request := range queues[job.Filename] {
                    request.ResultChan <- job.Result
                }

                delete(queues, job.Filename)
            }
        }
    }()

    return requests
}

Let’s focus on the goroutine that gets spawned in this function. It does the following things in a loop:

  • processes incoming request, adding it to a queue identified by job’s filename,
  • sends a job to a worker pool for every new created queue,
  • handles processed job, sending the result to each queued request’s ResultChan.

Finally it returns the requests channel so server can send new requests to it. We need to modify main function to reflect the changes we did to the worker pool and to wire up RequestMux:

func main() {
    jobs, results := WorkerPool(5) // changed
    requests := RequestMux(jobs, results) // added
    server := &Server{Requests: requests}
    log.Fatal(http.ListenAndServe(":5000", server))
}

It is a pretty powerful server now, but there is still one more optimization we can do.

Pattern 3: Caching job results

Request grouping prevents us from doing the same work in parallel when similar requests come in a relatively short period of time. However we still repeat the same work when we get similar requests sequentially rather than simultaneously.

Sounds like a job for a cache. Let’s update main function first for a change:

func main() {
    jobs, results := WorkerPool(5)
    jobs, results = Cache(jobs, results) // added
    requests := RequestMux(jobs, results)
    server := &Server{Requests: requests}
    log.Fatal(http.ListenAndServe(":5000", server))
}

Notice that we “hijack” jobs and results variables! We use a Cache function that gets original jobs and requests channels and returns a new pair, which we pass to RequestMux. This nicely shows how you can easily compose different pieces of code using channels in Go.

Example Cache function looks like this:

func Cache(upstreamJobs chan *Job, upstreamResults chan *Job) (chan *Job, chan *Job) {
    jobs := make(chan *Job)
    results := make(chan *Job)

    go func() {
        for {
            select {
            case job := <-jobs:
                cachedPath := "/cache/" + job.Filename
                if isCached(cachedPath) { // cache hit
                    job.Result = cachedPath
                    results <- job
                } else { // cache miss
                    upstreamJobs <- job
                }

            case job := <-upstreamResults:
                cachedPath := "/cache/" + job.Filename
                moveFile(job.Result, cachedPath)
                job.Result = cachedPath
                results <- job
            }
        }
    }()

    return jobs, results
}

The idea here is to receive new jobs from the jobs channel and check if there is a cached result. If it’s there we immediately pass the result on the results channel. Otherwise we send the job to the original jobs channel (locally named upstreamJobs). When the real job is processed we get it from the original results channel (locally named upstreamResults), cache the result and send it back over the new results channel.

Note that you can apply this technique to any type of work pipeline that involves input (jobs) and output (results) channels.

Conclusion

We hope you enjoyed this overview of the patterns we found very useful. Go is a great language for solving this type of problems and we’re very happy we have it now in our arsenal at Gitorious.

Announcing Gitorious 3.1

We’re happy to announce Gitorious 3.1 – a minor upgrade bringing several new features, many improvements and lots of bugfixes.

Detailed list of the changes can be found in our CHANGELOG, the highlights of this release are presented below.

New features and notable changes

Our commenting system got further improvements. On all diff views you can now open multiple inline comment forms, meaning you can write and preview the comments independently. Also, you won’t see any unnecessary page reloads when commenting on either commits, diffs or merge requests anymore. It’s worth to mention that displaying of the comments is now driven by React.js, which had proven to be a great fit for this use case.

Teams page now lists the teams in alphabetical order, project page lists the repositories in alphabetical order too. It makes finding teams or repositories much easier.

If you’re an admin in your company and you want to manage SSH keys used by your users to push then we have something for you. We added a new page to the admin panel (accessible at /admin path), which is a simple interface for adding and removing SSH keys of all the users in Gitorious.

Another new thing on the administration side is the ability to use external SMTP server for sending all Gitorious emails by creating config/smtp.yml file. See config/smtp.sample.yml for details.

One of the requested features was ability to automatically grant access to all of the repositories for all of the users in Gitorious installation. We’re happy to introduce this feature in the form of the “super group”. Super group is a special system group that contains all of the users, and it’s added to all of the projects, both existing and new ones. To enable it make sure you have enable_super_group: true in gitorious.yml config file.

To ease troubleshooting we have included more information in application’s log file (production.log). Now every logged request additionally includes timestamp, pid of the worker process and UUID identifier.

One thing that got lost in our previous release was the user’s ability to change its website/blog URL (which is displayed on his/her profile). It’s back and you can find it on your user settings page.

Last but not least, we made lots of cleanup and internal refactoring of the code making it easier for us to work with and easier for the community to contribute.

Stability improvements

This release brings lots of bugfixes related to the overall stability and security of Gitorious.

In this version Gitorious was upgraded to run on Ruby on Rails 3.2.19, the latest stable and secure version of Rails in 3.x series, which fixes many security issues found in recent months. Ruby version was upgraded from 1.9.3 to 2.0, giving noticeable performance boost to the application, making it feel snappier.

We fixed several XSS vulnerabilities, Wiki and README files rendering, and crashing of background job processors. Merge requests were improved too. Caching of merge request diffs has been fixed, merge request creation page now properly handles diverged branches, and the numbering of newly created merge requests has been made predictable and consistent.

That’s not all when it comes to fixes and improvements. You can read the comprehensive list in the CHANGELOG file.

Installing or upgrading

To install Gitorious 3.1 on your server please refer to the Community Edition installation instructions. If you’re interested in supported Gitorious Enterprise Edition get in touch via sales@gitorious.org. If you’re already a Gitorious customer we’ll discuss the upgrade with you soon.

To upgrade your existing Community Edition installation to 3.1 please refer to the Upgrading guide.

Announcing Gitorious 3.0

Today we are proud and happy to announce the official release of Gitorious 3.0. This is a major upgrade which ships with a refreshed UI, a lot of improvements, and new features. It has been a huge effort by both the original Gitorious developers, who started the work on version 3.0 a year ago, as well as the new team who finalized the remaining tasks.

For Gitorious 3 we changed the entire UI – every single view has been updated, and many of the views have also been structurally revised and improved. This means that a few hundred ERB templates in a 6-years old rails app have been updated. Huh, wait, we actually did that? Sounds crazy. Anyway – trying to summarize all the UI changes is not really feasible, so I will focus on the most significant changes, and urge you to explore the rest on gitorious.org.

New repository browser

Our brand new repository browser comes with a nicer UI, improved syntax highlighter, ability to select lines of code and friendlier navigation. We also improved the diff pages, which now include a list of commits and a summary of changed files, in addition to the revamped inline commenting system.

New repository browser

Repository browser

Diff page

Diff page

You can read more about the new repository browser in the blog post about first beta of Gitorious 3.0.

New Merge Request UI

Another improvement that we’re really excited about is the new Merge request UI:

Merge Request page

Merge Request page

New dashboard and public user profile pages

The dashboard gives you quick access to everything you’re watching. In addition to that we also added “Your activities” tab so you can easily see what you’ve been doing recently without the need to browse to your public profile page (which also lists your activities).

New dashboard

Dashboard

Public profile page

Public profile page

Previously, links to user settings were accessible from the dashboard page, which was a bit confusing so we moved it to the pulldown at the top of the page:

Settings

Settings

New settings page

Speaking about the settings page – it is now split into easily accessible tabs so you can quickly access various settings like account information and SSH keys.

Settings page

Settings page

Service hooks with built-in integrations

The service hooks feature, described in another post, was extended with the support for built-in integrations for external services. This feature was sponsored by the fine folks at sprint.ly  – a beautiful project management application that we really like – so they are the first officially supported service.

Sprint.ly integration

Sprint.ly integration

The sprint.ly integration can be used to update stories in sprint.ly via commit messages – for more information check out the official guide here.

An integration for Jira is in the works, and many will follow in the time that comes. We also welcome contributions for your own favorite services and/or ideas to additional integrations.

Upgraded internals

Gitorious 3.0 changed a lot on the surface but it is also worth mentioning that it is now using Ruby 1.9.3 and Ruby on Rails 3.2.15 under the hood. This opens doors to upgrading to the latest Rails 4.0.x in the near future without too much of a hassle. We have also updated all 3rd party gem dependencies to the latest versions, incorporating security upgrades and other important patches. The result is a vastly modernized base, which is safer, more performant and more stable than Gitorious 2.

What’s next?

We, as the new gitorious team, are very excited about this release. It was a great opportunity for us to learn the codebase and we now feel very comfortable when working on Gitorious. Ambitious as we are though; no matter how happy we are with 3.0, we are already looking forward to start work on the next major features and improvements.

Here are some things we are planning:

  • An issue tracker that integrates nicely with other Gitorious features
  • Improved Merge Requests – making it even simpler to contribute and review code
  • Improved “activity stream” with better notifications (replacing the inbox feature)
  • Further improvements to the repository browser
  • Continued internal refactorings towards a clean Gitorious Ruby API that people can use to build new tools on top of Gitorious

And more will come. Yes, we’re gonna be busy.

One of our main goals going forward is to be innovative. We believe a lot can be done in a slightly or even completely different way resulting in a better experience. We also want to frequently push new releases, as working many months to achieve huge milestones is no fun. So – stay tuned and watch this space!

If you have wishes, ideas, suggestions now it’s a great time to let us know about them :)

How to install or upgrade?

The CE installer has been updated to install Gitorious version 3.0, and we’ve also put together an upgrade guide here. If you have questions or problems please let us know on the mailing list or drop us a line to support@gitorious.org. If you found a bug please report it in the issue tracker.

Thank you

We would like to thank the original Gitorious team for their amazing work – it was a real pleasure to work together for this short but fruitful period of time and we will do our best to take the project to the next level.

We also want to thank our community for the support we get and we hope you will enjoy using Gitorious 3.0 on gitorious.org.

Local installs

If you are running Gitorious 2 locally supported by our team, we will get in touch with you in the near future to discuss upgrading. If you are running Gitorious at your company or organization on your own and would like professional support for upgrades, maintenance or custom features, don’t hesitate getting in touch at customers@gitorious.com.

Future of the “Inbox” feature

The “Inbox” feature was added to Gitorious about 4 years ago and it was meant to give our users an easy way of communication. While we know it is a useful feature for many of you, we are also aware that it hasn’t gotten enough love recently. We’d love to know more about how you use it and get your opinion for its future with the form below. If you have 10 seconds to spare, please answer the 2 questions and hit Submit. Thanks!

Post mortem on yesterday’s downtime

Gitorious 3 has been in the next branch of Gitorious’ mainline repository since last autumn. We started out building a new code browser for Gitorious which would perform better than our previous solution, with server-side syntax highlighting of any programming language out there. At that time we had fallen behind on versions of Ruby on Rails, the framework Gitorious is built with, and we realized that we would have to build Gitorious 3 for Rails version 3.

The story with Rails 3 is a long one, and we’ll save that story for another occasion. But for some parts of Gitorious, for example our routing system, the Rails 3 upgrade took a long time. But we were finally able to run Gitorious under Rails 3, using Ruby 1.9. After Ruby 1.8 reached end of life earlier this year, this is now an important reason to upgrade.

As we started discussing the aquisition by Powow AS, Christian, Thomas and I wanted to give the new team a clean slate, with a version of Gitorious with less technical debt and a vastly improved UI. As part of the agreement with Powow we agreed that the old team would take care of upgrading the servers to Gitorious v3 around the time of the acquisition. Last week we picked yesterday as the final date of the deployment, and started planning for the upgrade.

D-day

When we started upgrading the servers yesterday morning, most things went according to plan. We followed the upgrade guide we had set up, and after half an hour we were able to view gitorious.org running Gitorious v3. After getting all the components of Gitorious running, we opened gitorious.org to the public around noon. We were expecting some errors to occur, due to the sheer volume of users and repositories on gitorious.org.  Running an application in test and staging environments is not the same as running in a producton environment with over half a million users. One of the first issues we addressed was that rendering of a project page was ridiculously slow; a bug we deployed a fix for within 30 minutes after upgrading.

One of the surprises we were not expecting, however, was that my profile page on gitorious.org rendered the international characters in my name wrong. Digging deeper, we found that although the Rails app was configured to use latin1 encoding when communicating with the database, we were not getting latin1 back. We tried changing the database configuration to use utf8, but discovered that the actual data in the database was wrong. In order to not have inconsistent data in the database, with some data using different encodings in the same database we decided to take gitorious.org offline.

With the help of our friends at Redpill Linpro, we tried dumping a copy of the database to a file and then importing that into the MySQL database, using utf-8 for the database dump. Looking through that file, the encoding looked really strange. Where my middle name in UTF-8 should be 7 bytes (\x4d\xc3\xa5\x72\x6e\x65\x73), it was actually 9 bytes in the database. Once the import had completed, the data was in the same state as it was before the import – broken. Seeing 4 byte characters where we should be seeing 2 byte characters made us believe  that the data in the database had been encoded twice. Running a double recode of the data proved us right:

SELECT CONVERT(CAST(CONVERT(fullname USING latin1) AS BINARY) USING utf8) FROM users where id=2;
Marius Mårnes Mathiesen

Now all we had to do was to find a way to recode it twice in the file produced by mysqldump, which (as always) could be done with a Perl one-liner. We ran this oneliner on a SQL file for a single table (extracted from the entire database dump using sed, of course) and used that to recreate the users table in our database. Naturally that didn’t work, so we had to set up the character set related settings in the MySQL server and client, which worked. We now had a strategy for how to recreate the database with all UTF-8 data.

After dropping the database and importing it from the database dump, processed by the Perl oneliner, we finally got Gitorious running again around 7:15pm yesterday. We got all the services running again, and re-opened access to gitorious.org at 7:30pm CET.

Bottom line

We’re all really sorry about this downtime. This is not the kind of service you should expect from us, and we appreciate your patience while we worked to resolve this.

The good news is that we wouldn’t have been able to resolve this situation without help from the new team. Database configuration and character encodings are not of the old team’s strongest skills, while Marcin used tools I personally never knew existed to read the actual byte values from a several GB SQL dump. Further good news: gitorious.org is now running the best release of Gitorious ever, the infrastructure is in a much better state, and the new team has a lot of plans to make Gitorious even better.

On behalf of the old team: please don’t blame “the new guys”, we’re the ones who messed up this time.

And again, we’re really sorry about this.

Powow AS acquires Gitorious AS

With Gitorious 3 just about to be released, we’ve got some more news that we’re really excited about.

The current Gitorious team has been working full time on Gitorious since January 2011, and over the last two and a half years we’ve managed to improve the quality, stability and feature set in Gitorious – while running Gitorious AS as a profitable business. The company today has three full time employees who work exclusively with the Gitorious software, gitorious.org, and supporting all the organizations out there depending on Gitorious for their day-to-day work.

Looking back, we’ve accomplished most of the goals we set out with when we founded the company.

However: looking forward, we realize that we’re in the same situation as many startups: we don’t have the energy and motivation to take Gitorious to the next level. The company is at a good place, but we strongly feel that Gitorious needs fresh blood and new ideas.

It was very important to us that our community and commercial customers would not be left hanging, and we started looking for a new steward for both the project and our company.

It was crucial to us that the new owner of the Gitorious project (and company) shared our most important values:

  • free/open source software

  • amazing support and customer service

  • hacker culture

  • devops/continuous delivery

  • test-driven development

  • being profitable as a result of doing what we love, not a goal in itself

We found the perfect match. Powow is a Norwegian-Polish Ruby consulting company which consists of some of our biggest heroes in the Ruby community. They will be in charge of the Gitorious project and Gitorious AS the company going forward.

Powow was founded in 2008 as a small software consultancy in Norway. Since then the company has expanded and built a great team of highly skilled and experienced developers. While originally a consulting shop, the company had already started shifting towards more product-centric work when we got in touch with them.

We’ve already started the transition, and while working with the new team over the last weeks we’ve been very impressed with their skill-set and values, not to mention how quickly they were able to start shipping features. Also, with several members of the team being long time Linux/UNIX users, we’re confident the operations and support offerings of the business is in even better hands than in the past. And as you can see, the new Gitorious team is bigger than the current one.

This is good news for both the Gitorious community and our enterprise customers. The new team has some great plans for the future, which I’m sure they’re anxious to share with you over the coming weeks.

Say hello to the new team

team_piotr Piotr Solnica – co-founder of Powow, software developer with almost a decade of experience. Ruby and JavaScript pro. Clean code and TDD evangelist. Open source hacker and Ruby Object Mapper core developer. Currently romancing with functional programming paradigms.
team_marcin Marcin Kulik – passionate software developer breathing HTTP since 2001. Ruby professional. Fluent in many other programming languages. Open standards proponent and open-source contributor. Linux geek. The man behind ascii.io.
team_adam Adam Pohorecki – aspiring software craftsman and TDD evangelist. Founder of SCKRK. Organizer and facilitator of code retreats and unconferences. Inventor of Ping-Pong Pomodoro Pair-Programming and bogus – a next generation Ruby mocking library.
team_pawel Pawel Pierzchala – Software Craftsman from Kraków. Code Retreats organizer and coach. While not coding or reading white papers for next SCKRK meetup, busy watching spaghetti westerns.
Mariusz Mariusz Ciesla, designer and wannabe developer. Really bad at maintaining balance on a skateboard.
team_rolf Rolf Bjaanes, developer, coffee geek, and as chief happiness officer of the company he’ll mainly be working on the business side of things. In his own words, “every developer’s hunt for the best editor ends up with Vim, Emacs or a management position.”

Next steps for Gitorious the open source project

The current team is focusing on finishing the work on Gitorious 3. It will be deployed on gitorious.org for battle-hardening next week, and will be made generally available with installers and the works shortly thereafter.

Once Gitorious 3 is out, you’ll start seeing more of the new team on the mailing list, IRC channel and blog as the Powow team transitions into being main committers and owners of the project. Development will commence with the new crew on-board. The AGPL licensing and terms of the open source project remains unchanged.

The main change for the Gitorious community is that the project will get an infusion of capable, proven, and outgoing open source developers with fresh ideas and energy. And they are dying to start talking with you about the roadmap going forward!

Next steps for Gitorious AS the company

Our customers keep telling us that the support they get from us is their main reason for choosing and staying with us. We know that a key part in providing good support for customers is trust and personal relations.

First and foremost, we’ll make sure service and support stays consistent and responsive while we pass the torch to the Powow team. We will get in touch with all our enterprise customers over the next few days to introduce the new team and make sure all concerns are addressed. Christian, Thomas and I will be working with the new team for as long as it takes for them to become fully self-sufficient with all aspects of the business.

Also, once Gitorious 3 is ready and has been proven stable, we’ll schedule time for upgrading all our customers’ servers. We will be doing this work together with the new team to ensure a smooth upgrade process.

As of today, Gitorious AS is formally 100% owned by Powow AS.

A final word from the founders

As this is probably the last post we’ll be writing on this blog, Thomas, Christian and I would like to thank the community, our customers and other supporters for some really good times over the last few years. Gitorious wouldn’t be where it is today without you. We trust that you’re in the best of hands going forward.

Gitorious will be upgraded on Monday [completed]

The time has finally come to deploy Gitorious 3.0 to the gitorious.org servers. We are planning to do the upgrade the coming Monday August 26th, starting at 10:00 AM CET. We expect a smooth upgrade process, but gitorious.org may be unavailable for up to two hours while we perform the upgrade.

We will post an update to this blog post once we’ve completed the upgrade and ensured all services are working normally again. Once we have performed the upgrade we will be constantly watching the performance and fix any issues that may occur due to the upgrade. Don’t hesitate to get in touch with us at support@gitorious.org or comment on this post if you have any questions.

If you’re curious about the changes you’ll be seeing after the upgrade, we have put up a preview site at https://v3.gitorious.org/.

Updates

  • 12:04 CET: We’re almost there, but need a little more time before we open the gates. We estimate another hour of downtime
  • 13:00 CET: The site is public, but dead slow. We’re hard at work taking care of the performance
  • 14:05 CET: We’re back, but we have some encoding issues in our database. We will probably have to take the site offline later today to convert the data in the database.
  • 14:!5 CET: We just took gitorious.org offline again while converting our database.
  • 14:50 CET: We’re currently importing the database, hopefully with the correct character sets
  • 16:40 CET: The character set conversion in the database didn’t work as expected, we’re currently investigating
  • 18:30 CET: We seem to have been able to convert our database. Currently preparing a restore
  • 18:35 CET: The database import is currently running, we expect to be back in around an hour.
  • 19:17 CET: Aaand we’re back up! We will write a post mortem on what happened today tomorrow.

Gitorious 3.0 Beta 1

It’s been quite a journey, but we pulled through, and we are excited to finally present you with the first beta of Gitorious 3.0!

Take it for a spin

We set up a trial instance on v3.gitorious.org. You should especially check out the new repository pages. This instance is free for all, so please register an account and test at will. Just note that it will be removed once gitorious.org is upgraded to Gitorious 3. There are already some known issues, and I will be posting updates to this demo throughout next week.

For those of you who want to run Gitorious 3 yourself, there will eventually be a pre-built CentOS-based VM available on getgitorious.com. Unfortunately, we haven’t had time to fix that up yet, and as we are now going into vacation mode, it will be a few weeks.

Your best option for self-hosting Gitorious 3 at this point is to install Gitorious 2, e.g. by using the Community Edition, and follow the manual upgrade instructions found in the repository.

For people wanting to hack on Gitorious, there are instructions in the repo for CentOS and Ubuntu for how to get the development environment up and running. These take some shortcuts that are not recommended for production deployments, but will get everything up and running.

What’s new?

The big news in Gitorious 3 is the new and improved code browser. In addition to being a big improvement UI-wise, it supports many new small useful features. The code browser also provides quick access to browsing and viewing files, blame, log for individual paths and more.

Browsable URLs

The old

/project/repository/blobs/*

and

/project/repository/trees/*

URLs are gone. In their place we now have a unified way of browsing source code through

/project/repository/source/ref:path

, e.g.

/project/repository/source/master:Readme.org

Automatic rendering of README in your repository

Gitorious will render any README file it can render in your repository.

DoltReadmes

Not only will it render READMEs in the root of your repository, but from any directory inside your repository.

Vastly improved syntax highlighting

The syntax highlighting in Gitorious 3 is powered by Pygments, arguably the best syntax highlighting toolkit around. This means Gitorious now supports every one of Pygments’ long list of supported languages.

syntax_highlighting

Highlight code

When browsing files, you can mouse over lines to highlight them. If you want to share highlights with other people, click a line, copy the URL and share at will. You can also highlight a region by clicking a line, holding shift and clicking another one. Copy the URL and off it goes.

line-nums

Convenient branch/ref selector

The code browser has a branch selector in the upper right corner, which is always available for any action where different branches/refs has different content (i.e. when browsing code, but also the log etc). This allows you to select branches and tags, or enter any Git oid to view it.

branch-selector

Curl-able downloads!

Gitorious has used a polling mechanism to serve tarballs for some time. In Gitorious 3, this is gone, and we use simple GETs to fetch tarballs. For high-traffic deployments, getgitorious.com has information on how to configure nginx so this scales well.

Detached code browser

The code browser shipping with Gitorious 3 is also released as a stand-alone tool that you can use to visualize your repositories locally. You can even use it for a light-weight read-only Git repository hosting service on your server. The tool is called Dolt, and if you have Ruby installed, you can try it out in 5 minutes: gem install dolt. Run either in a git repository, or in a directory that contains many git repositories: `cd /my/git/repo && dolt .`

dolt

Web hooks

In Gitorious 3, you can select the “admin” menu on your repositories when logged in, and it will offer you to manage web hooks. These are URLs that Gitorious will send a POST request to everytime someone pushes to the repository. Postbin is a great service for debugging these things, and here’s a sample payload from Gitorious.

webhooks

Log in with either email or login

In the past, Gitorious would only allow logins with email address. If you’re using the database backed authentication, you can now log in with either your email address, or your login/screen name.

Rails 3.2 under the hood

Gitorious 3 is based on Rails 3, which brings vastly improved security, performance and provides a better base for evolving the software. Some of you may have seen that Rails 4 was just released, and Gitorious 3.1 or 3.2 will likely be moving to Rails 4. When that happens, Ruby 1.8.7 (and Ruby Enterprise Edition) support is gone (Rails 4 does not support it).

Backwards and forwards compatibility

Gitorious 3.0 supports Ruby 1.8.7, primarily because many Linux distros still ship this version. Ruby 1.8.7 support should not be expected to last for long. We recommend that you run Gitorious 3.0 on Ruby 1.9.3, and Ruby 2.0.0 support is just around the corner (one of our dependencies is causing a segfault, but we have verified that everything will work once an update is released for it).

Improved Ruby API

The low-level APIs in Gitorious have been refactored and restructured in several areas. This work is not yet complete, and will continue throughout the summer. This will eventually make it easier to perform power-user/admin tasks from the console. Gitorious 3 ships with “use cases” that can be run from the command line, and encapsulates everything required to carry out certain tasks. Some documentation is available.

Git data mirroring

We’re currently finishing up a new feature that allows Gitorious to mirror all git data to one or more mirrors. This will help in cases where you want a “ready to go” failover server for Gitorious. This feature does not ship with the first beta, but will be in the final version.

What remains

This is a beta, and as such, bugs should be expected. We would love your help and support in testing this, and we will soon roll out the beta on gitorious.org. Once we’ve gotten through the worst crop of bugs, Gitorious 3.0 stable will be tagged and released. When that happens, everyone will be strongly urged to upgrade, as we will not provide updates for Gitorious 2 much longer.

You will also notice that while most of the repository pages sport a shiny new UI, we have not completed this transition entirely. Changing the UI all around is a big task, and we will continue iterating on this until the old UI is gone. We’re sorry that the app will have two faces in this transitional period, but we believe the improvements we’ve made already makes it more than worthwhile.

What do you think?

Try it and out, and let us know what you think. We are very excited about this release, and hope you will like it as much as we do.

On behalf of the Gitorious team,
Christian (@cjno).

Follow

Get every new post delivered to your Inbox.

Join 845 other followers