Nostalgia for simpler times often glosses over the rough edges of the past. We may complain about our COVID-19 isolation, but it’s nothing compared to our ancestors who lived through the Spanish Flu (1918-1920) without internet connectedness. The previous post in this series exposes the negatives of old-time, infrequent software releases while presenting the good and the bad of the current day Continuous Integration/Continuous Deployment (CI/CD) approach. No matter how technically sophisticated we become, alleviating customer pains without introducing ancillary frustrations remains a Sisyphean software challenge. Current CI/CD technology enables practitioners to release production software many times in a single day. However, the technology doesn’t address the question, Should we release software hundreds of times a day just because we can?
The answer is, “It depends.” Facebook and Netflix deploy production software hundreds or thousands of times a day. Typically, these are small, unobtrusive changes that don’t affect the robustness of the software. No, these frequent releases aren’t the sole province of gargantuan companies, although these rich, engineering-centric companies invest heavily in building rock-solid processes to safely and rapidly release software.
In some cases, the software is so intuitive that users easily absorb small changes. In other cases, if users aren’t paying for the software, like Google Hangouts, they’re probably willing to deal with a changed interface every other time they use it. When there’s more at stake – cost, safety, security – customers are less willing to suffer surprises.
Here’s an example where too-frequent releases are detrimental. Imagine a call center where hundreds of operators expertly glide through the fields and pages of their enterprise software. Subtle changes to the software cost their company money as operators spend precious time learning navigation changes, new fields, and altered functionality. This call center example isn’t an argument for yesteryear’s bi-yearly releases but suggests that a more controlled release cycle with change documentation is more appropriate for these users.
Another interesting use case is SpaceX, a company that produces mission-critical software and hardware. One might expect that the risk to human lives contraindicates rapid and frequent deployments. Furthermore, it’s easy to imagine SpaceX’s software release process mired in up-the-chain levels of approval and signoff. In reality, SpaceX Product Owners likely choose not to deploy as rapidly as Netflix because their use cases don’t demand it. However, SpaceX uses Agile practices and a thoughtfully developed CI/CD pipeline; the company is likely positioned to deploy as often as it wishes. SpaceX expends immense amounts of intellectual firepower on automated testing, ensuring that test coverage is complete and their practices provide safety nets to catch problems before they reach production.
The nuts and bolts of CI/CD focus on releasing well-tested, well-performing software. All the other necessities of successful software deployments, like communication, documentation, training, and support, are left as exercises for a company’s non-engineers. CI/CD done right gives engineering teams superpowers that the rest of the company must match. As Uncle Ben uttered to a young Peter Parker, “With great power comes great responsibility.” Applying Uncle Ben’s homily to software releases, the great power of CI/CD requires an equal emphasis on the non-technical aspects of releasing software – communication, training, documentation, and sound decision-making.
Although this isn’t a deep technical dive into CI/CD, it’s useful to take a shallow exploration of its contours. CI/CD is a 21st-century concept backed by current Version Control Systems (VCS) and automated deployment tools. The underlying idea is that any time an engineer commits code to a VCS, a series of events automatically occur:
- Execute a set of unit tests that verify the code fulfills its intended purpose
- Commit the code to the VCS
- Build the software
- Run some quick smoke tests for a sanity check
- Deploy the software to a production-like environment
- Execute acceptance, integration, functional, and non-functional tests
- Optionally, deploy software to production
If it seems fanciful that all of this can occur at the push of a button, there are four important points to absorb:
- An automated CI/CD pipeline isn’t for the faint of heart. Building automated testing and automated deployment processes is arduous and difficult. However, it greatly reduces the inevitable errors that humans introduce with manual processes.
- Humans are essential to this process. Regardless of a CI/CD pipeline’s sophistication, humans will always need to perform exploratory testing – poking and prodding at the system to see what happens if we do this?
- Although it busies up a CI/CD pipeline diagram, functional companies have multi-disciplinary teams that decide what/when to release and ensure that customer communications and education keep pace with deployments.
- The Product Owner must exercise enough situational awareness to match the deployment plan with the customer base’s unique needs. Having a “deploy anywhere” button is a powerful tool when exercised judiciously.
Jez Humble and David Farley wrote their 2010 foundational book, Continuous Deployment, espousing a trunk-based development discipline mandating daily commits to the trunk of the VCS repository. Conceptually, daily commits suggest these chunks of code might make their way into production. If a new feature requires more than a day to complete, does trunk-based development imply that half-finished work appears in production? Part Three of this Quality series explores the disconnect between the aspirational CI/CD experts and the pragmatism of many in the software-developing community.
