raditionally, integration testing is something that is done right at the end of a project: Developers check-in their code and a build team is tasked with compiling everything and creating an installation program that can be deployed for acceptance testing. This is where the problems start, with broken builds, missing files, and a laborious manual process delaying release of the project to customers and squeezing the amount of time available for testing. Continuous integration (CI) attempts to resolve these issues by ensuring that the integration and build phase occurs every time source code changes.

CI is one of the practices of Extreme Programming (XP) and, as such, shares XP’s emphasis on open communication, short development cycles, and continuous feedback; but even if you’re not practicing XP, CI brings many benefits, including:

• The program is always integrated and functional, ready to be delivered to the customer—there’s no chance of the build cycle delaying the project.
• There’s no need for a separate build team?CI requires an automated build process, saving countless painful hours of work.
• CI provides constant feedback to the development team and highlights any integration issues as soon as they occur; locating and fixing the source of broken builds becomes trivial.

Daily Builds are for Wimps
Integrating and building software is a difficult and tedious task, and this is the main reason why many development shops leave it until the last minute, or do at most a daily build. In order to meet the demands of CI, you need an automated build process that is triggered by changes to source code. To accomplish this, the build system must be integrated with the source code repository so that whenever source code is checked in, a build is initiated. This is a significant departure from traditional build cycles.

Regardless of your stance on XP, it’s hard to argue the case against CI. It’s an extremely smart practice, with a huge payoff both for the development team and the business as a whole. XP and CI has been widely accepted and practiced in the Java world, but is still waiting to take hold in Microsoft shops; for .NET developers in particular, implementing CI is difficult and riddled with roadblocks. CI necessitates the use of build automation, but so far Microsoft has not taken the initiative to provide build tools that integrate directly with Visual Studio and Visual SourceSafe. The vacuum has been filled by a couple of open source projects, but they are currently difficult to set up and poorly documented. We believe this is a critical factor preventing development shops from taking up CI en masse.

Another factor is developer resistance: There are cultural changes required for CI to work in development shops using Microsoft technologies. For those .NET developers who are trying to decide whether it’s the right time to tackle an automated build process, we decided to give it a test run and report the major issues.

NAnt and Draco
NAnt is a build tool for .NET inspired by the already established Ant build tool for Java. NAnt allows you to create an XML build file, which shells out execution tasks to other applications. Another tool, Draco.NET, supports integration between NAnt and a source code repository (such as Visual SourceSafe), so that you can have NAnt execute a script automatically execute every time someone checks in code. It also e-mails the team a report each time a build executes, describing all the changes made and a list of errors in the case of a broken build.

Without these tools we wouldn’t have been able to set up a CI solution. That said, we had problems with both of them:

1. Creating a NAnt build script is a manual process and can be tricky; there are a few tools out there that can help (such as NAntPad), but if you’re using Visual Studio.NET, developers can add projects and files to the solution at any time, which necessitates an update to the NAnt script?a manual step that easily gets overlooked, resulting in broken builds. To solve this we initially tried to use the SlingShot NAnt Task from the NAntContrib project, which can be used to create a NAnt script from a Visual Studio.NET solution. Unfortunately, we were never able to get SlingShot working; the SlingShot documentation was pretty sparse so we moved on.
2. We turned instead to the Solution task, also from the NAntContrib project, but which generates a NAnt script at build time as the first step in your automated build. This also gave us problems, this time because our solution contained COM interop references, which the task was not resolving properly. We delved into the source and made a few modifications; it turned out that the code deals with COM interop references by prefixing the reference name with the hard coded string “Interop.” That probably works in the majority of cases, but not in ours; so, we modified the source to discover the assembly’s location by searching for the correct COM entry in the registry.
3. We initially thought that amending the source code would be straightforward?this is, after all, an open source project. However, we had to go to a great deal of trouble to get the NAnt source to build using Visual Studio.NET; we mistakenly assumed that because NAnt is a .NET project it would be Visual Studio-friendly. Our next problem was that NAnt wouldn’t compile solutions created using Visual Studio.NET 2003. There was nothing in the documentation about this, but in reading the newsgroups we were alerted to a setting change in the NAnt.exe.config file that solved the problem.
4. Draco.NET was relatively easy to set up, once the major problems with NAnt were solved. There were one or two minor issues: the tool occasionally misses a SourceSafe check in resulting in the wrong developer being blamed for a broken build. It’s also tricky to read through the report that Draco generates as the result of a build, but it’s nothing that a bit of XSLT can’t fix.

The biggest problem we faced was the lack of documentation?a not uncommon (or even unexpected) problem with open source tools. While NAnt has a brief guide to writing scripts, the majority of documentation is geared toward contributing to the project, not on how to use it. So we found plenty of blog and newsgroup postings from puzzled people who would love to implement CI, but who encountered problems and had to give up. Open source projects rely on feedback and fixes from users, but this is hindered when developers can’t get the projects working in the first place.

We’re not condemning these tools, but they are difficult to work with at the moment. They’re relatively new projects, and if they’re to improve, more developers need to start using them and contributing to the projects. However, what’s currently there could be improved dramatically through better documentation and some basic best practice guidance.

Who Broke the Build?
Another fundamental obstacle to the widespread adoption of CI is developer resistance. By this, we mean a series of things?from skepticism and fear to ingrained development practices?that need to change if CI is to be more widely adopted.

Seeing the benefits of CI may require a leap of faith. Those who’ve never worked in a large development team where long, tedious, error-prone build cycles are a fact of life may find it hard to see what CI brings to the table. Even those who do see the benefits of CI may need to change their mindset to make it work:

1. Using traditional development methods, you’ll generally check code out of the source repository when you begin a task, then check it in days or weeks later when you’re done. This approach will not work with CI?development iterations are too short, and code needs to be constantly integrated?you need to unit test and check-in every couple of hours at the very least, and the whole team checks everything in at the end of the day before they leave the office.
2. The biggest source of build failures will be due to developers not keeping their local source repository up to date; every time the team is notified of a successful build, it’s time to get the latest versions from the source repository. Frankly this is a pain at the moment, and again it’s a limitation of the tools?Visual Studio.NET’s SourceSafe integration needs to be overhauled and made dramatically faster.
3. Sooner or later, everyone breaks the build, and using automated CI, the whole team will know about it when it happens. Fear of embarrassment may be one reason that coders resist the CI change, but they should view it as an opportunity to quickly work out why a build was broken and to resolve the issues as a team. Another advantage is that the team will see the changes made by other team members?a valuable source of feedback. It also allows you to quickly track down the source of a broken build. The information could even be used to construct release notes.

Getting Past a Rough Patch
Of all the XP practices, CI should be the easiest to implement and provide the most tangible benefits, but it is difficult to set up right now. The problems with the tools can be solved with more and better documentation and support. Although open source encourages it, not all developers who use open source tools will be willing to invest a huge amount of time in understanding how they work.

While Microsoft is working on a new build tool (MSBuild) that will ship with Visual Studio.NET 2005, there’s no indication that they’re planning to do anything with the practice. The lack of an integrated CI tool in Visual Studio is another key reason that most Microsoft shops aren’t trying CI. Microsoft could take a lead here, applying its resources to an open source CI project, similar to how it released the Windows Installer XML (WiX) project under an open source CPL license on SourceForge.

Finally, a lot of developers will have to change their mindset to see the benefits of CI and recognize that a certain amount of effort up front will bring immeasurable benefits further down the line.