The Future of Continuous Integration
For all its benefits, Continuous Integration (CI) does have its share of flaws. For one, CI is somewhat reactionary when integration builds break because the code has already been committed to the mainline of the version control system.
Broken Builds
The main problem of reactionary CI is that you don’t learn that your full integration build has failed until the code is committed to the version control system and other developers are affected by it, which can decrease team velocity.
Current workarounds
In Software Configuration Management Patterns, Berczuk and Appleton describe the Private System Build as a pattern for preventing a broken build from occurring on the integration build machine. The private system build includes integrating changes from other developers on your project and running a successful integration build before you commit changes to your version control system. However, there are certain errors such as those that may be configuration-related or you may have simply forgotten to commit new files to the repository. Another workaround to preventing broken builds is performing Manual Integration Builds as James Shore covers in his blog. In this workaround, developers are put into a queue where only one person (or pair) integrates at a time. This was the original concept of CI with XP teams. This technique should be coupled with the private system build. While this approach can take some discipline and can be a little more difficult for distributed teams, it can prevent builds from being broken too long. However, you still have the same problem of having broken code (compile, test or otherwise) in the repository — even if it isn’t as long as an asynchronous build when using a CI server.
Possible Solutions
There is a new breed of tools entering the marketplace that hold some promise. While they probably don’t yet solve all the “problems” of CI, they are addressing some of CI’s flaws. There are servers that provide CI capabilities and can prevent “bad” code from entering the version control system. Borland’s Gauntlet (full disclosure: my company is a partner of Borland) and JetBrains’ TeamCity. Gauntlet uses a feature called “sandboxing” which creates a temporary repository branch for each integration build and promotes the code to the mainline of the repository only if the integration build is successful (including compilation, tests, inspections and so on).
What’s Next?
Don’t get me wrong, I use and evangelize CI regularly (even wrote a book on it) and I think its practices provide significant benefits…especially when compared to “big-bang” integration when teams wait until the end of a significant milestone (e.g. the day before they were supposed to release) to integrate all of their changes. However, like anything, things can be improved and we are continuing to see improvements in the evolution of the practice and I expect to see more in the future. As I eluded, there are other issues with CI and I will cover these in another blog entry…

April 30th, 2007 at 8:52 am
[…] Paul Duvall’s post The Future of Continuous Integration points out a weakness in how continuous integration is typically implemented: code must be checked in before it is tested by the continuous integration server. Although problems are picked up fast, there is still a time window where developers may be affected by broken code. This is exactly why we added personal builds to Pulse in version 1.2. Personal builds allow individual developers to submit their changes to Pulse for testing before committing to version control. Pulse does a build and test of the latest version with the developer’s changes applied, allowing to developer to check the full impact of their changes prior to affecting the shared code base. Apart from closing the time window, personal builds also allow other powerful usage patterns, such as using the distributed building features of Pulse to test across multiple platforms before checking in. […]
May 6th, 2007 at 10:46 am
Thanks for bringing this topic to our attention. As any other technology, CI is not a silver bullet and it is important to understand its limitations and how it can be improved.
I am curious on your thoughts on how CI scales. I think some of the basic assumptions of CI - that you have effective automated tests with large code coverage, that you can quickly solve any problems introduced in the mainline, or that the cost of breaking the mainline, even temporarily, is worth the risks - do not hold up for long as the size and complexity of the software being built and the number of developers involved grow to “enterprise levels”.
In your experience, what is the breaking point? Would you use “pure” CI in a build with hundreds of developers and millions of lines of code? If not, which changes you would recommend?
Thanks,
Nascif
May 7th, 2007 at 7:09 am
A follow-up to my previous question to make it more specific.
When I said “enterprise-level”, I was talking about having multiple layers of software spread accross different projects, and each project supported by a different team.
So while I can see how for each project CI would apply, I have doubts about how it applies across projects and specially team boundaries. Testing in particular becomes an issue; even if a layer has all the required automated unit testware, it becomes harder and harder to properly automate the integration of all the layers. And without that, how can you effectively have CI?
You risk a situation where the bottom layers push code to the mainline so that it is “visible” to the upper layers, allowing for the integration testing that only then can provide - but many times creating exactly the broken build scenarios that you described in this article.
Thanks again,
Nascif
June 16th, 2007 at 8:44 pm
[…] In a recent post of the future of Continuous Integration , I pondered how the broken build can affect your development team and offered potential solutions. My next topic is what many consider to be the bane of CI, Long-Running Builds. Again, I don’t think there’s a silver bullet answer to this conundrum. But, depending on how the problem manifests, there are some solutions. I spend quite a bit of time in chapter 4 discussing long-running builds, so this is a quick synopsis. […]
February 4th, 2008 at 10:47 pm
[…] Test Early ยป The Future of Continuous Integration […]