Continuous Delivery in an Agile World

The most important problem that we face as software professionals is this: If somebody thinks of a good idea, how do we deliver it to users as quickly as possible?

It was 2010. This was how Jez Humble and David Farley opened their book Continuous Delivery. Things were going to change fast. Continuous Delivery was measurable and demonstrable and organizations made radical changes to adopt this new methodology.

In this article, we'll look at what Continuous Delivery is, why it's a competitive advantage, and what the process looks like.

Contents

Objectives

1. Describe what Continuous Delivery is and what the process looks like
2. Identify some come antipatterns that organizations face during deployment when Continuous Delivery is not used
3. Explain the two pillars that make up the foundation for a solid continuous delivery process
4. Describe the key principles of continuous delivery

What is Continuous Delivery?

Continuous Delivery (CD) is the process of delivering software from the development phase to the hands of the users reliably, safely, and as quickly as possible. With software, I mean everything from source code to configurations, data, and the environment.

The focus of CD is on the deployment pipeline, which involves all steps in your application's build, deploy, test, and release process. Every single step must be automated. By doing so, the process of building, deploying, testing, and releasing will be visible and clear for everyone in the organization, enhancing a culture of support and collaboration among developers. Testing every change in an automated way will generate quick feedback for developers so they can identify and resolve problems immediately when the effort required is minimal. CD enables teams to deploy and release their software when they want and with no deployment pain.

If you're implementing CD right, you can release your software just by pushing a single button or running a single command. Unfortunately, in many organizations, this is not true and the release day is usually a day (or more) of pain. Some common antipatterns can be identified among organizations:

1. Software is deployed manually. Software releases tend to be a long manual process that needs to be executed by a team of engineers. A lot of things can go wrong in the process; and, if a step is not executed perfectly it will cause the whole process to fail. You can usually recognize this antipattern when there's a lot of documentation that needs to be followed to prepare a release, a manual test is needed to ensure everything is working, and/or the whole release process takes more than a couple of minutes. Many problems may arise from this antipattern. The deployment process is not repeatable and reliable, can't be tested, and is error-prone. The size of the documentation will increase over time, be hard to maintain, and become obsolete. But the most critical thing is that the only way to test your deployment process is to do it, which is dangerous in itself.
2. Software is deployed to a production-like environment only after the entire development cycle is complete. This usually happens when there are two teams: a dev team and an ops team. With this antipattern, software is considered done even if it has never been tested in the production environment, the deployment documentation was never tested, and operations people see the software for the first time on the release day. The first deployment in production is always the most troublesome. Plus, assumptions made by developers may be wrong, and with such a long release cycle there's no chance to fix them. This scenario ends with tens of emails, tickets, and "it's someone else's responsibility".
3. Configuration of the production environment is managed manually. This antipattern is present when any of these are true:
   • a needed change in the production environment is done manually and there's a record of the changes
   • to prepare the production environment takes a lot of time
   • it's not possible to roll back to a previous version.

Do you recognize any of these antipattern in the usual release process in your company? In this case, you need to bring this to everyone's attention and start addressing each of the antipatterns as you move to a more agile environment.

The software deployment process must be fully automated, used by everybody, and for every deployment. This will ensure that the deployment script will be tested over and over and that it will work reliably on release day.

Deployment to the production environment should be integrated into the development workflow to use Continuous Integration (CI) as a way of testing both software and deployment. This discipline ensures that when you're ready to release your software it will happen with little or no risk since it was tested over and over.

Configuration must be managed as code. Every aspect of your environments (specifically OS configuration, VM configuration, third-party elements, infrastructure configuration, and your application stack) must be checked into version control. You should be able to recreate your environments in an automated way.

Continuous Delivery Foundations

Speed is key here, since not delivering software has costs. You're missing an opportunity because you can't start learning until you release. That fancy feature you've designed has no value until your customers start using it. The quicker you can get feedback, the quicker your customers can validate your software. Decreasing the feedback loop is the first pilar.

The second pillar is quality. We want to deliver high-quality software. If you have a manual test suite that takes five minutes to set up and configure before even running, this is a clear signal that something is going wrong.

To achieve the two pillars you need automation and frequency. If the build, deploy, test, and release process is not automated it's not repeatable. Every time you go through the process, it will be different. Manual steps are error-prone, difficult to document, and impossible to review once done. If you don't automate you'll never have control over the release process. Releasing software shouldn't be an art but a boring and monotonous engineering process. If your releases are frequent the difference between them will be minimal. This will help to drastically reduce the risk associated with the release itself and it'll be easier to roll back. Releasing frequently provides faster feedback and feedback is essential for a good CD process.

Feedback Process

Every change to executable code, configuration, the host environment, and/or data must trigger the feedback process. So, each of those components must be kept under version control and every change must be tested.

When a change is made to the source code, the resulting binary must be built and tested in an automated fashion. This practice is known as Continuous Integration.

If your environments change, the differences must be captured as configuration information, and any change in the configuration should be tested. The same applies to the environment where the application is to be deployed. Also, if the structure of the data changes, this change must be tested.

What is the feedback process? It involves testing every change in a fully automated fashion, as far as possible. Practically it means that for every change in the codebase:

• The building of the source code must be successful
• Unit tests must pass
• Code quality requirements must be fulfilled
• Software acceptance tests must pass
• Software's nonfunctional tests must pass (security tests, availability tests, etc.)

Automation is the key to fast feedback. Manual processes dependent on people are error-prone and take longer. Moreover, they are boring and repetitive. Through automation, we can free people to work on interesting things and leave repetition to machines.

We can set up our CD pipeline to have two stages of testing:

1. The first one will be very fast, it may not cover the whole codebase but will address the critical paths so that if any test fails our application should never be released. It can run in a neutral environment.
2. The second stage, on the other hand, can be slower, even if some tests fail we may want to release anyway. It should run in a production-like environment.

This kind of setup will ensure resource optimization: the first stage can run on cheap hardware and if it fails the process stops; the second stage will run on more demanding hardware but can be parallelized.

Everyone who is part of software delivery should be involved in the feedback process and they should work together daily to improve software delivery. This kind of iterative process is essential to quickly improving the quality of the released software.

Key Principles

At this point, you may have an idea of the philosophy of CD and it can be framed in a series of principles.

Create a Repeatable, Reliable Process for Releasing Software

Releasing software should be easy because you've tested every step of the process several times. Repeatability and reliability come from automation and the use of version control, which will make releasing software as easy as pushing a button.

Automate Almost Everything

Of course, some things can't be automated. Demonstration of software to your users can't be automated. Approvals for compliance purposes can't be automated. Does anything else come to mind?

Many teams don't automate their release because it looks like a big effort; instead, they rely on manual steps. The effort is big but it's worth it and it'll pay off when you perform the 10th release (probably even after the 5th, but you got the point).

Keep Everything in Version Control

Everything needed to build, deploy, test, and release your application should be under version control. A new team member should be able to check out your repository, run a single command to build, and deploy the whole project.

If It Hurts, Do It More Frequently, and Bring the Pain Forward

Integration is usually a pain, so you should do it more often. Testing and releasing are painful. Even creating new documentation is difficult. You should put in place a philosophy where painful things are done early and often in the process.

Build Quality In

The earlier you detect defects, the cheaper they are to fix. This means that testing doesn't start after development and it's not just a team responsibility. Approaches like Test-Driven Development can help achieve this principle.

Done Means Released

Do you have user stories that are "done" and others that are "done done"? The definition of done must be clear and it ideally means released into production. This may not always be practical so we can use the following definition of done: a feature is done when it has been successfully tested in a production-like environment. You can imagine how it's not a one-person job to get a feature "done" but a team effort. Remember also that there's nothing like a "90%" done because it's impossible to estimate realistically that remaining percentage.

Everybody is Responsible for the Delivery Process

For small startups, this is easy to achieve, but in bigger organizations it may require a lot of work. Sometimes the reality is that departments work in "silos" and so they end up blaming one another over bugs. This is one of the pillars of the DevOps movement.

Continuous Improvement

After the first release of your production application, it's important that your process improves as more releases will follow. This can be done by holding retrospectives involving everyone in your organization to collect ideas and act upon them.

Impact of Continuous Delivery

Several studies have been done to evaluate the impact of Continuous Deliveries in organizations.

The State of DevOps reports says that teams that practice CD spend 44% more of their time on new work than teams that don't.

In 2013 Nicole Forsgren, Jez Humble, and Gene Kim started a journey among companies to understand processes and performances for software delivery. The results were published in the Accelerate book.

They tried to assess and measure the following capabilities:

• The use of version control for application code, system configuration, application configuration, and build configuration scripts
• Comprehensive test automation that is reliable, easy to fix, and runs regularly
• Deployment automation
• Continuous Integration
• Shifting left on security: bringing security -- and security teams -- in process with software delivery rather than as a downstream phase
• Using trunk-based development as opposed to long-lived feature branches
• Effective test data management

The result was amazing. When taken together, these capabilities have a strong positive impact on software delivery performance. They help to decrease deployment pain and team burnout. Moreover, teams that did well with CD also identified more strongly with the organization they worked for.

Another crucial question they wanted to answer was: Does CD increase quality? In particular, they focused on: the quality of applications, as perceived by those working on them, the time spent on rework, and the time spent working on defects identified by end-users. The analysis found that teams who applied the above key points were correlated with high software delivery performance. Such teams also spent the least amount of time on rework.

What Continuous Delivery Looks Like

This is an example of a deployment pipeline:

It starts with a developer committing code into the version control system and the Continuous Integration systems triggering the execution of a new instance of the pipeline.

The first step of the pipeline is called the commit stage. This stage aims to eliminate builds that are unfit for production and to provide feedback as quickly as possible since we want to invest the minimum amount of time on a version that is obviously broken. We also need to prevent context switches. So, the developer who commits new code waits for the results before moving on to the next task. Ideally, we would like our developers to wait for all tests to pass so that they can fix any problem immediately. In reality, this is not practical for the whole pipeline but can be achieved for the commit stage, so once the commit stage is completed developers are free to move to the next task.

There are a couple of things we want to do in this stage: build the code, run the unit tests, create and store the binaries in an artifact repository for later use, and perform code analysis.

If the first stage completes successfully, it means the unit tests are passing. Keep in mind that unit tests only test a developer's perspective of the solution to a problem. They don't really cover if the application does what it's supposed to from a user's perspective. If we want to test if our application is providing value to our users, we need another form of tests. These tests make up the second stage: the automated acceptance test stage.

This stage can be split into different test suites and parallelized. It also serves as a regression test suite, verifying that no bugs are introduced into existing behavior by new changes. It's important to consider the environments that your application will encounter in production when executing acceptance tests.

Once the automated acceptance test stage has been completed, we have a successful release candidate. At this point, the pipeline branches to enable capacity testing, exploratory testing, usability testing, and independent deployment to various environments. It's possible to automate operations in this stage to ensure an automated deployment to production, but for many organizations, some form of manual testing is desirable before release and it can be done in this stage so a human being will decide whether the release candidate should be promoted or not.

Practical Advice

Depending on the current status of your software delivery process, you may now feel a bit overwhelmed, but I want to share some practical advice that I hope you'll find useful for when you start your journey towards CD.

Take It One Step at a Time

Implementing a CD process is not a one-day job. It will take time and dedication and require many changes to your daily workflow, so my suggestion is: focus on one thing at a time.

You should start by first implementing Continuous Integration. This alone will provide an enormous productivity boost and increase the quality of your project.

Provide Actionable Error Messages

When a release is rejected, the error message(s) must clearly explain what went wrong and how to fix the situation. For example, avoid error messages like "unit tests failed" and instead provide the name of the failing test and a link to the log message.

Measure and Use Data

Your deployment pipeline can provide a lot of data that you should measure. Record the cycle time from committing to releasing and the time spent in every stage. This will help with evaluating the status of your process and identifying bottlenecks.

Include a Deployment Break Glass

It may be necessary to bypass your deployment pipelines from time to time. For example, a change could be taking too long or getting stuck in your pipeline. A break glass mechanism that bypasses your pipeline stages can allow engineers to quickly resolve an outage.

Conclusion

To summarize the biggest takeaways:

• Your goal is to deliver value to your users
• The release process should be quick and provide constant feedback
• Releasing software shouldn't be an art but a boring and monotonous engineering process
• If it's not automated you don't have control over it
• Fix a problem immediately when you find it
• The first step towards Continuous Delivery is Continuous Integration

What does your release process look like? Hit me up on Twitter and let me know. Cheers!