High-end Thoughtworks consultants Jez Humble and David Farley wrote the book on Continuous Integration/Continuous Delivery (CI/CD). The thought leadership espoused by Martin Fowler, Humble, and Farley around CI/CD provides customers with well-tested, frequent deployments. Dig deeper into Continuous Delivery’s principles, and, interestingly, very few engineering teams follow them. Is Continuous Delivery dead?
Part 2 of this series outlines different use cases, some of which lend themselves to Continuous Delivery (CD) and some that do not. The diagram below illustrates different classes of software and their suitability for CD. When CD is indicated, software for technical Github users, for example, it has an adaptable audience that appreciates rapid change. In general, CD is contraindicated when unexpected changes may cause either the loss of life or cause great expense. In the case of Enterprise software, users accustomed to executing a prescribed workflow waste valuable time and money attempting to absorb changes that may appear with little to no forewarning. For software with scads of enterprise users, it’s nearly impossible to provide features that everyone wants, even if everyone’s in the same business vertical. Consequently, some customers will be unhappy to see these additions.
The trickle-down effect in clothing fashion is a simile of the CD recommendations from experts versus the CD that teams actually practice. Haute Couture designers Dior, Chanel, et al. produce annual fashion lines affordable only to those on the economic ladder’s highest rungs. These high-end designers have very few customers, but they don’t need quantity when a small number of patrons pay astronomical prices. The general public may admire aspects of the fashion but won’t wear it until Kohl’s and Target start making practical imitations. The fashion knock-offs may incorporate just the originals’ themes or colors but omit the design’s unaffordable aspects.
The CI/CD experts tout a highfalutin approach that few but the most sophisticated and talented teams will adopt. Instead, the CI/CD’s conceptual framework is reinterpreted in a more approachable manner for the masses. Just like with couture, pragmatic CI/CD omits the fancy stitching and expensive fabrics. Continuous delivery of enterprise software requires an additional level of interpretation that the experts don’t prescribe.
Continuous Integration (CI) is the DevOps counterpart to Agile development – every development team worth its salt is practicing CI, however imperfectly. Virtually all of the experts’ pronouncements around CI are difficult to achieve but worthwhile and necessary. The Continuous Delivery (CD) side of CI/CD is murkier. Most practitioners agree on the importance of always-deployable software; there is considerable disagreement about the best way to accomplish this feat. Some of the experts’ advice crosses the line from aspirational to impractical.
A Tale of Two Workflows
A brief foray into Version Control Systems workflows provides important context before excavating the chasm between best practice recommendations and rubber-meets-the-road actual team behaviors.
At a high level, when a team of developers is each building different software features or fixes against the same code repository, they each need a place to work where they won’t interfere with one another. When a developer finishes a feature, it must be merged into a place where it becomes eligible for release and accessible to the other developers. Selecting a workflow for handling these hotfixes and new features is one of the most consequential decisions a software team will make.
Many software teams use Git as their Version Control System (VCS) and GitHub as the cloud service to host Git repositories. Git supports millions of workflows. Teams have the freedom to blaze their own path or use a workflow devised by someone else. One of the most popular workflows, GitFlow, was conceived by Vincent Driessen in 2010.
The Gitflow workflow relies on a complicated branching strategy that originates from Master, aka “the trunk.” A Develop branch is created from Master that becomes the point from which Feature and Release branches originate. When a Feature is complete, it is merged back into the Develop branch. When a Release branch is complete, it is merged into the Develop and Master branches. If a bug is discovered in a release that demands an immediate fix, a Hotfix branch is created from Master and, when complete, is merged to both Develop and Master. The Gitflow workflow is well suited for teams that must support multiple live software versions. This workflow is not generally associated with CD.
Many teams regard Gitflow as needlessly complex for their purposes and choose a lighter-weight workflow like the feature branch-based one below, aka Github flow.
Github flow has simpler feature branching directly off Master. When a developer needs feedback on changes, a Pull Request is used for online review and collaboration. When the team reaches closure, the new feature is merged into Master and deployed. The GitHub team uses the GitHub-Flow workflow and it is popular with development teams small and large. GitHub-Flow is similar to the recommendations of Humble, Farley, and Fowler, with some key differences.
What the Experts Recommend … and What Really Happens
Recommendation: Perform trunk-based development.
Developers may create very short-lived feature branches, but code is committed directly to the trunk. In theory, trunk-based development isn’t much different from the Feature-branch workflow. The one area of divergence is the longevity of feature branches. Most developers agree that having months-old feature branches are nothing but trouble because they get out of sync with the rest of the code; developers must take special care to merge from the trunk back into the long-running branch. However, most development teams are okay with keeping a feature branch going for the life of a normal feature’s development.
Recommendation: Commit code to the trunk a minimum of once a day.
Since features often require more than a day to develop, the experts recommend committing incomplete features and hiding them instead of maintaining long-running feature branches that become difficult to merge. Most teams commit code when a feature or fix is considered complete by peers after the developer issues a Pull Request. The “commit once a day” rule isn’t followed by most teams. Although the trunk-based aficionados argue that feature branch merge difficulties justify daily check-ins, many development teams have little problem merging relatively short-lived feature branches.
Recommendation: Hide incomplete features.
The experts recommend the branch by abstraction technique and feature toggles to safely hide incomplete features released to production. Most teams choose to keep features separated on a branch until they are ready for prime time instead of writing code to hide incomplete work.
Recommendation: Perform pair programming.
This tenet of Lean Programming combines two developers’ skills to increase quality and reduce completed work review-time. When two skilled developers pair on a coding problem, they often bypass a review process and commit to the trunk. Development teams recognize pairing benefits, but most use it selectively on especially thorny pieces of work. Although pairing may save money and time, it’s a hard sell to nontechnical upper management. Git’s wildly popular killer feature, Pull Requests, provides a structured environment for code review and back and forth discussion. Furthermore, Pull Requests enable less experienced developers to review others’ code without assuming sole responsibility, providing immense growth opportunities in technical oversight and critical feedback.
Adapting Continuous Enterprise Delivery
Although Enterprise customers may request bi-annual releases, they also wait impatiently for their requested features. There’s some middle-ground in Enterprise Continuous Delivery, but it requires reinterpretation of the CD tenets. A few rules of the road for Enterprise Continuous Delivery:
- Acknowledge that “continuous” does not mean multiple times a day for the Enterprise. Instead, consider a cadence of planned and preannounced frequent releases – a huge improvement over bi-annual or quarterly releases that pose their own problems.
- Release new features without breaking preexisting workflows. Use default values so new features needn’t be touched, even if they’re mandatory. Design the user interface to make features visible but not intrusive – sometimes, it’s necessary to insert new fields in pre-existing sections. More often, new functionality can be tucked into separate sections/tabs/pages.
- Provide easy, inline education, marketing, direct mail, and training sessions to encourage the adoption of new features. New features solve customers’ problems, and success is measured by feature usage.
- Workflow-breaking changes need a significant amount of lead time so that customers are informed and have time to prepare.
- Although most SaaS companies support just one production version, a trunk-based workflow (like GitHub-flow) may not provide the flexibility required for controlled releases to enterprise customers. A Gitflow-style workflow may be a more appropriate choice.
- Use Pull Requests as the default and pair programming with direct-to-trunk commits when appropriate.
Continuous Delivery isn’t dead. The ideas of Continuous Delivery may require some life support in hostile conditions, like the change-resistant Enterprise. A creative reinterpretation of the CD tenets yields a regular cadence of continuous delivery. Nimble companies strike a healthy balance between bi-annual releases and hourly releases by careful coordination between Engineering, Product Management, Marketing, and Customer Success.
