Integration points control product development.
—Dontar Oosterwal, The Lean Machine
Continuous Integration Abstract
Continuous integration is neither new, nor invented by Agile. However, SAFe is dependent on continuous integration, not only at the Team level, but also at the system level, which incorporates code from many teams on the Agile Release Train. In addition, for larger value streams, continuous integration practices must be extended to these largest systems via cross-train integration, where fast feedback is every bit as important, maybe even more so. Effective Continuous Integration (CI) practices help reduce the cost of delay, and substantially reduces overall program risk.
The Problem with Discontinuous Integration
CI is a well-established practice (though not always well-mastered) for Agile Teams. At the program level, however, it is often the case that system level CI tends to be deferred until the final stages of the release cycle, which adds substantial risk and delays to the program. To illustrate this, we’ll start by highlighting the problems of discontinuous integration. Imagine a small program with just three agile teams, working together on a common system, as illustrated in Figure 1.
Each team is in charge of a feature or component and is working in accordance with their understanding of what is needed in their code, and how that code contributes to the overall system.
- At point (1) in time, team A checks-in their code. This could be a short period of time in an Agile project, ranging from hours to days; longer in non-agile environments.
- At point (2), team B checks-in their code. Only now do the teams see the problems arising from the integration of the two, which in turn triggers rework in both areas. Worse, before they discovered these problems, team A was already working on the next item of work, building more code on top of the problematic area, and ultimately further increasing the amount of rework needed.
- Finally, team C checks-in their work at point (3), which triggers further cycles of rework, for all three teams.
This vicious rework cycle might be relatively short on a small program, but on a larger program (Agile release trains routinely have 5, 10 even 15 teams), distributed across geographical boundaries, this becomes a huge effort, one that ultimately may well determine the overall time to market for new capabilities. Effective, system-level CI is not optional in SAFe.
Enough complaining, let’s now look to a solution to this problem:
Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. 
As Figure 2 illustrates, Continuous integration is not simply a matter of frequent code check-ins with automated builds. Comprehensive CI tools follow the build by running unit tests and acceptance tests to verify code constancy, and then installing the system on a testing environment and running Build Verification Tests (BVT). The BVT could be the full regression suite, or only a subset intended to quickly provide a “smoke test” of the new, end-to-end system.
Finally, the integration results are made immediately available (automated notification of stakeholder) that highlights what is broken and what check-in created this problem. If this happens only nightly, there could be several check-ins, form different people, any one or multiple of which might have broken the build. Instead, if the infrastructure supports a BVT on every check-in, investigation and error correction becomes easier.
Frequency of CI
Team level features and components should have at least a daily integration cycle. At the system level, an equivalent response time is desirable, but perhaps initially, ARTs should focus on the capability to provide at least two integrations per iteration. Then, and only then do we know for sure that the system is sprinting, not just the teams. Teams will now find problems during their development process and quickly perform any needed rework, as part of the normal workflow. Problems are detected earlier; rework cycles are faster and shorter; time to market is improved.
The System Team is responsible for creating and maintaining the CI tools and environment and ensuring it is run continuously and its results are made public.
Value Stream Integration
When multiple trains in the same value stream contribute to a common solution, CI must also be implemented at this level (see Coordination). Value stream level solutions should be integrated at least twice per PI, and preferably every iteration.
Working in a Single Branch
When working at scale, a development team’s typical branching practices can introduce a serious impediment to system-level CI. Individual team or feature-based branches tend to delay the actual integration of the whole system, while creating a false sense of control and efficiency. While such branches, if short-lived, can be acceptable, in practice, teams often fail to sustain the discipline of branching-and-merging-back quickly. As a result, multiple development branches may exist and continuous integration is deferred until all those branches are finally collapsed into one. Therefore, having all teams operate in a single, program-level development branch enforces good CI discipline and enables fast system-wide integration and evaluation.
Figure 3 illustrates such an approach. Each developer checks out his code from the one branch, and checks it back in when finished, or at least daily.
If cross ART integration is required at the value stream level, all trains may operate out of a single value stream branch. Or in the case where the programs agile maturity, system design, or infrastructure makes that difficult, there can be separate program branches, which must then be integrated into the value stream branch, at least once per sprint.
Continuous Integration Culture
While most CI discussions revolve around tooling, the fact is that continuous integration is a cultural change, requiring responsibility at the individual, team and program levels. Three practices are critical for a successful CI culture
- Integrate often – The more frequently we integrate, the quicker we find problems. This will create faster investigation cycles and less rework.
- Make integration results visible – When the integration process breaks, it is essential that everybody knows about it, and about why and how it broke.
- Fixing broken builds is the top priority - If a broken build or failing tests are not treated as the top priority, continuous integration systems quickly fall out of sync, and are either ignored, or revert back to late, infrequent integration. This negates the gains from the CI practices and can create apathy for the effort. To stress this aspect, many teams use flashing lights or other indicators to indicate a broken build. Attention is then directed to fixing the problems before new code is added to the system, thereby accelerating learning.
Starting with Program Level CI
Agile development at scale puts pressure on the enterprises software development environment. Initially, many programs may not have the development infrastructure (and maybe the skills) needed to support effective, system-level CI. However, not having such a system in place should not be a barrier to Agile implementation, as a release train launched without a CI system should still be able to integrate fairly regularly, produce a system demo and attain many of the Agile benefits. Moreover, working towards ever-faster feedback and reduced delays at the program level is an excellent driver for the needed system CI initiatives.
Role of System Team(s) in Continuous Integration
Agile Release Trains typically have a System team that provides certain program-centralized functions to the teams on the train. Since system-level continuous integration is a system-level problem, the system team often plays an important role in building and maintaining the CI infrastructure. They may also be responsible for automation of the process, and may even need to be involved in helping debug broken builds. However, the primary responsibility for integrating new code and fixing broken builds is the responsibility of the individuals and teams who contributed the code, so the system team should not spend much of its time in that pursuit.
In addition, as we’ve indicated in this article, larger value streams that contain dependent Agile Release Trains often have additional levels of concern in the integration and validation of the full, value stream solution. See Coordination for more guidance on this topic.
Continuous integration is necessary to reduce delays associated with late, and problem-plagued, system integration, which slows time to market and provides a false sense of velocity to teams and programs. Effective CI systems allow all teams on the train, and all trains in a value stream, to understand how their work impacts others and ensure that they are working effectively to make that sure the system is sprinting, not just the teams.
While an initial lack of CI should not prevent a program from heading down the scaled Agile path; maximum velocity, quality and shortest possible time to market cannot be achieved until the essential CI systems, skills, and culture is well established.
 See Continuous Integration at http://martinfowler.com/articles/continuousIntegration.html
Last Modified 27, June 2014
© 2011-2014 Leffingwell, LLC. All rights reserved.