Continuous Integration (CI) is a popular concept actively exploited by Agile software development teams. The idea, originally formulated by Martin Fowler in September 2000, is simple and at the same time very effective: on a daily basis, a pre-configured automata (CI tool) grabs the latest version of source code from the repository, compiles it, tests, and reports about any problems found. The report reaches all parties involved and triggers certain organizational and management actions. If, at any moment of time, anyone introduces a test-detectable defect into the repository, everybody gets notified on the next day.

At first sight, it appears you can accomplish most necessary CI tasks with a simple few lines of bash script, started every day by cron, for example (assume we’re using Subversion and Maven):

#!/bin/bashcd /home/ci/trunksvn updatemvn test

Then we can hook this script to crontab with the following two lines (assume it’s a Unix-like platform):

[email protected]* 0 * * * /home/ci/ci.sh 2>&1

Everything will work fine, until you start thinking about more advanced features. For example, it would be nice to have an ability to do the following:

• To send emails only in case of failure
• To email different people, depending on their role in the project
• To store protocols somewhere on the server in order to review them later
• To enable a Web UI, instead of a plain old SSH terminal
• To let someone start a new build at any time, not only at midnight
• To publish online certain HTML reports produced by Maven
• To enable something else except Subversion and Maven

Hosted Continuous Integration Requirements

You see, we need something more complex than just a small shell script. We need a real CI tool that can automate at least these operations. Such tools have been created during the last few years by different authors. Some of them are open source, others are commercial. Two years ago, my team started to experiment with continuous integration tools for our internal software projects (we do custom programming), and came up to the idea of creating a free hosted platform. We called it FaZend.com. Since we had already invested a lot of our time into installation, configuration, and maintenance of CI tools, it was easy to let other teams use our experience and server resources. Now I will explain the lessons learned.

During the last two years, we have tried three different products and concluded that Hudson is the best choice for our needs. FaZend.com provides a continuous integration service to public audience, so security and reliability are top requirements for us. Compared to the situation where the continuous integration server is used by a single isolated software team, our case is more sophisticated. Our servers build and test products that belong to third-parties, not only to us. At the moment of this writing there are 102 software products being built by our server.

Let me explain what our needs are, since they are not typical:

• Computer-driven configuration shall be enabled. We don’t use Web panel for configuration of CI tools, instead we integrate it with our existing user management server-side system. We need the CI tool to enable its configuration through some reliable and easy-to-understand mechanism.
• Access control shall be configurable. Since we have our own database of users and we manage their passwords and emails, we need to be able to inject this information into the CI tool.
• Seamless integration with common SCMs. We have different clients, they use Subversion, Mercurial, Git, and CVS. Maybe in the future we will meet something else, maybe IBM Rational ClearCase. The CI tool should be ready to accept these formats.
• Business logic shall be easily configurable. Almost every project has its own requirements and needs, from “I want a build triggered twice per night” to “After every successful build we should upload this directory by FTP to this server.”

Continuum Continuous Integration

We started in 2009 with Continuum. It’s an open source product being developed by the Apache Software Foundation, with a strong focus on Apache technologies. The biggest problem was access control configuration. In terms of user management, Continuum relies on Redback, which has two options for where to keep account information: database or LDAP. Thus, in order to inject our own user accounts details into Continuum we could either 1) make direct SQL updates to its database, or 2) install, configure, and maintain an LDAP server just for this purpose. We pursued the second option, since the idea of getting a direct SQL access to a third-party database looked very un-safe.

All other problems were related to the principles of information management implemented in Continuum. It keeps everything in an un-documented relational database, where it was almost impossible to add anything without breaking the internal rules of these data structures. In other words, the system looks like a very closed black-box, not oriented to integration with other products.

CruiseControl Continuous Integration

In November 2009 we decided to migrate to CruiseControl. The software was originally developed by ThoughtWorks, Inc. This system looked more open for us, since it keeps all (well, almost all) of its internal configuration data on disc, in XML files. Unfortunately, even this CI tool appeared to be rather closed, and we were not able to extend it for our needs. For example, there is no ability to define a custom schedule of build execution. You can’t build both after every commit and every night.

These were minor configuration problems, which were solved or we just let them live in the system. A much bigger problem was a lack of documentation for server-side formats and principles of organization of files. Sometimes they are managed in a very strange way (not in XML, to say the least).

Hudson Continuous Integration

In August 2010 we migrated to Hudson. This open source product is developed by Oracle and looks very lightweight and extensible. To be honest, there is still a complete absence of documentation for server-side components, but their structure and organization is easy to understand after just a few hours of experimenting. Everything is configurable through just a few XML files, and there are a lot of plug-ins written by contributing developers.

Editor’s Note: The primary developers of Hudson recently ported their own version of the project to GitHub and renamed it Jenkins.

This is a summary of our needs and software tools we have tested so far:

Hudson Comes Out on Top

At the moment we are happy with Hudson. There are many other tools in the market and each one of them is more suitable to a certain business case than the other. In our situation, with an unpredictable variety of SCM systems, build tools, and configuration requirements, Hudson proved to be the best choice.