Continuous integration (CI) is the implementation of a process for continuously verifying whether a piece of software is working. CI provides developers with feedback about whether the piece of software compiles, passes all kinds of tests, builds into deliverables, and so on. It alerts the developer when changes he or she has just committed caused the build to fail, allowing the developer to correct errors while it’s still fairly painless. Integrating changes late in the software development process can be costly and it often leads to errors.
This article presents the essential practices for implementing CI in your software projects, using the process we employed at my company, TouK, as an example. We implemented CI using the following open source tools, which we gradually adopted over time:
- Source code management:
- Repository for caching external libraries and storing released software (definitive software library):
- CI server:
- Quality management platform:
You can see how they interact in Figure 1 below. There are alternative implementations of these tools and open source products should easily integrate into an ecosystem like this.
There are two software engineering practices that are essential to beginning CI: collective code ownership and automated builds. Intrinsic to the first practice is some kind of source code management. We use Subversion and Git. Generally, our teams are free to choose tools that suit them best. We have very few rules, but we do expect that code in the repository will always work and can always be built with minimum effort after checkout. This requires some kind of build automation. We use various building tools (Ant, Maven, Gradle, make, etc.) and they all work in our common environment. So, if you exchange any code via email or build your project in an IDE, these are the first things to change.
The Critical Role of the CI Server
When you start writing new code, you want your project to be stable. But how do you make sure the code in your repository works at all times? Simple, test it after every change. This is the main task of a CI server; every change in the repository triggers configured build jobs in the CI server. The CI server can be notified of a change or it can monitor a source code repository. How you configure the build jobs depends on your particular project. For any single project, we have several types of jobs and each serves a different purpose.
The following are the different CI server job types and their corresponding structure.
Quick-running jobs are triggered on each commit and they tell you within a few minutes after making the change whether the build is functioning correctly. Quick-running jobs update the working copy, compile the project and run unit tests.
Test jobs are triggered several times a day if any commits were made. You can chain them with “after commit jobs.” They will run all tests or subsets of tests, and they can take several hours for when running integration tests or automated frontend testing.
Daily jobs are triggered daily afterhours. They build the project from a fresh checkout, fetching all dependencies and running all tests. It monitors the environment (mainly repositories) to make sure it’s easy to start working on the project from scratch.
Quality monitoring jobs are triggered daily afterhours if any commits were made. They build the project and perform the additional tasks of static code analysis and tests coverage. Afterwards, they send the results to Sonar.
The CI server will notify you when something goes wrong. If the compilation fails, you will know it right away. It keeps a history of recent builds, so you can monitor the codebase — how stable it is, how long it takes to compile or test, when a bug was first introduced, and so on. Rapid feedback is the key to a good CI process. Maintain your builds so you can respond fast.
Managing The Continuous Integration Process… and Beyond
CI jobs are configured by the development teams who are responsible for them. But when a project is finished and the team disbands, we still want to make sure we can continue with changes and fixes at any time. This is when our DevOps team takes over, utilizing their own tools. They constantly monitor CI server instances for any problems, availability of executors, and length of job queues. On the development side, we optimize our environment to minimize the average length of the job queue and we optimize builds to run fast.
The next step in the evolution of your continuous integration implementation is continuous Delivery. You can employ the same set of tools to automate acceptance tests, deployments and releases. But that’s topic for another day.
