Faster CI Builds with Docker Layer Caching and BuildKit

This article takes a look at how to speed up your Docker-based builds on CircleCI, GitLab CI, and GitHub Actions with Docker layer Caching and BuildKit.

Docker Layer Caching
BuildKit
CI Environments
Single-stage Builds
- Compose
Multi-stage Builds
- Compose
Conclusion

Docker Layer Caching

Docker caches each layer as an image is built, and each layer will only be re-built if it or the layer above it has changed since the last build. So, you can significantly speed up builds with Docker cache. Let's take a look at a quick example.

Dockerfile:

# pull base image
FROM python:3.12.2-slim

# install netcat
RUN apt-get update && \
    apt-get -y install netcat && \
    apt-get clean

# set working directory
WORKDIR /usr/src/app

# install requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# add app
COPY . .

# run server
CMD gunicorn -b 0.0.0.0:5000 manage:app

You can find the full source code for this project in the docker-ci-cache repo on GitHub.

The first Docker build can take several minutes to complete, depending on your connection speed. Subsequent builds should only take a few seconds since the layers get cached after that first build:

[+] Building 0.4s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 37B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 35B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.12.2-slim                                                     0.3s
 => [internal] load build context                                                                                        0.0s
 => => transferring context: 555B                                                                                        0.0s
 => [1/7] FROM docker.io/library/python:3.12.2-slim@sha256:bdefda2b80c5b4d993ef83d2445d81b2b894bf627b62bd7b0f01244de2b6a  0.0s
 => CACHED [2/7] RUN apt-get update &&     apt-get -y install netcat &&     apt-get clean                                0.0s
 => CACHED [3/7] WORKDIR /usr/src/app                                                                                    0.0s
 => CACHED [4/7] COPY ./requirements.txt .                                                                               0.0s
 => CACHED [5/7] RUN pip install -r requirements.txt                                                                     0.0s
 => CACHED [6/7] COPY project .                                                                                          0.0s
 => CACHED [7/7] COPY manage.py .                                                                                        0.0s
 => exporting to image                                                                                                   0.0s
 => => exporting layers                                                                                                  0.0s
 => => writing image sha256:2b8b7c5a6d1b77d5bcd689ab265b0281ad531bd2e34729cff82285f5abdcb59f                             0.0s
 => => naming to docker.io/library/cache                                                                                 0.0s

Even if you make a change to the source code it should still only take a few seconds to build as the dependencies will not need to be downloaded. Only the last two layers have to be re-built, in other words:

 => [6/7] COPY project .
 => [7/7] COPY manage.py .

To avoid invalidating the cache:

Start your Dockerfile with commands that are less likely to change
Place commands that are more likely to change (like COPY . .) as late as possible
Add only the necessary files (use a .dockerignore file)

For more tips and best practices, check out the Docker Best Practices for Python Developers article.

BuildKit

If you're using a Docker version >= 19.03 you can use BuildKit, a container image builder, in place of the traditional image builder back-end inside the Docker engine. Without BuildKit, if an image doesn't exist on your local image registry, you would need to pull the remote images before building in order to take advantage of Docker layer caching.

Example:

$ docker pull mjhea0/docker-ci-cache:latest

$ docker build --tag mjhea0/docker-ci-cache:latest .

With BuildKit, you don't need to pull the remote images before building since it caches each build layer in your image registry. Then, when you build the image, each layer is downloaded as needed during the build.

To enable BuildKit, set the DOCKER_BUILDKIT environment variable to 1. Then, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

Example:

export DOCKER_BUILDKIT=1

# Build and cache image
$ docker build --tag mjhea0/docker-ci-cache:latest --build-arg BUILDKIT_INLINE_CACHE=1 .

# Build image from remote cache
$ docker build --cache-from mjhea0/docker-ci-cache:latest .

CI Environments

Since CI platforms provide a fresh environment for every build, you'll need to use a remote image registry as the source of the cache for BuildKit's layer caching.

Steps:

Log in to the image registry (like Docker Hub, Elastic Container Registry (ECR), and Quay, to name a few).

It's worth noting that both GitLab and GitHub have their own registries for use within your repositories (both public and private) on their platforms -- GitLab Container Registry and GitHub Packages, respectively.
Use Docker build's --cache-from option to use the existing image as the cache source.
Push the new image to the registry if the build is successful.

Let's look at how to do this on CircleCI, GitLab CI, and GitHub Actions, using both single and multi-stage Docker builds with and without Docker Compose. Each of the examples use Docker Hub as the image registry with REGISTRY_USER and REGISTRY_PASS set as variables in the CI builds in order to push to and pull from the registry.

Make sure to set REGISTRY_USER and REGISTRY_PASS as environment variables in the build environment:

CircleCI

GitLab CI

GitHub Actions

Single-stage Builds

CircleCI:

# _config-examples/single-stage/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2204:2024.01.1
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build from dockerfile
          command: |
            docker build \
              --cache-from $CACHE_IMAGE:latest \
              --tag $CACHE_IMAGE:latest \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Push to docker hub
          command: docker push $CACHE_IMAGE:latest

GitLab CI:

# _config-examples/single-stage/.gitlab-ci.yml

image: docker:stable
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --cache-from $CACHE_IMAGE:latest
        --tag $CACHE_IMAGE:latest
        --file ./Dockerfile
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
  after_script:
    - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v4
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build from dockerfile
        run: |
          docker build \
            --cache-from $CACHE_IMAGE:latest \
            --tag $CACHE_IMAGE:latest \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Push to docker hub
        run: docker push $CACHE_IMAGE:latest

Compose

If you're using Docker Compose, you can add the cache_from option to the compose file, which maps back to the docker build --cache-from <image> command when you run docker-compose build.

Example:

version: '3.8'

services:

  web:
    build:
      context: .
      cache_from:
        - mjhea0/docker-ci-cache:latest
    image: mjhea0/docker-ci-cache:latest

To take advantage of BuildKit, make sure you're using a version of Docker Compose >= 1.25.0. To enable BuildKit, set the DOCKER_BUILDKIT and COMPOSE_DOCKER_CLI_BUILD environment variables to 1. Then, again, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

CircleCI:

# _config-examples/single-stage/compose/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2204:2024.01.1
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build images
          command: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
      - run:
          name: Push to docker hub
          command: docker push $CACHE_IMAGE:latest

GitLab CI:

# _config-examples/single-stage/compose/.gitlab-ci.yml

image: docker/compose:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
  after_script:
    - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/compose/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v4
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build Docker images
        run: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
      - name: Push to docker hub
        run: docker push $CACHE_IMAGE:latest

Multi-stage Builds

With the multi-stage build pattern, you'll have to apply the same workflow (build, then push) for each intermediate stage since those images are discarded before the final image is created. The --target option can be used to build each stage of the multi-stage build separately.

Dockerfile.multi:

# base
FROM python:3.12.2 as base
COPY ./requirements.txt /
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt

# stage
FROM python:3.12.2-slim
RUN apt-get update && \
    apt-get -y install netcat && \
    apt-get clean
WORKDIR /usr/src/app
COPY --from=base /wheels /wheels
COPY --from=base requirements.txt .
RUN pip install --no-cache /wheels/*
COPY . /usr/src/app
CMD gunicorn -b 0.0.0.0:5000 manage:app

CircleCI:

# _config-examples/multi-stage/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2204:2024.01.1
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build base from dockerfile
          command: |
            docker build \
              --target base \
              --cache-from $CACHE_IMAGE:base \
              --tag $CACHE_IMAGE:base \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Build stage from dockerfile
          command: |
            docker build \
              --cache-from $CACHE_IMAGE:base \
              --cache-from $CACHE_IMAGE:stage \
              --tag $CACHE_IMAGE:stage \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Push base image to docker hub
          command: docker push $CACHE_IMAGE:base
      - run:
          name: Push stage image to docker hub
          command: docker push $CACHE_IMAGE:stage

GitLab CI:

# _config-examples/multi-stage/.gitlab-ci.yml

image: docker:stable
services:
  - docker:dind


variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --target base
        --cache-from $CACHE_IMAGE:base
        --tag $CACHE_IMAGE:base
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
    - docker build
        --cache-from $CACHE_IMAGE:base
        --cache-from $CACHE_IMAGE:stage
        --tag $CACHE_IMAGE:stage
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
  after_script:
    - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v4
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build base from dockerfile
        run: |
          docker build \
            --target base \
            --cache-from $CACHE_IMAGE:base \
            --tag $CACHE_IMAGE:base \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Build stage from dockerfile
        run: |
          docker build \
            --cache-from $CACHE_IMAGE:base \
            --cache-from $CACHE_IMAGE:stage \
            --tag $CACHE_IMAGE:stage \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Push base image to docker hub
        run: docker push $CACHE_IMAGE:base
      - name: Push stage image to docker hub
        run: docker push $CACHE_IMAGE:stage

Compose

Example compose file:

version: '3.8'

services:

  web:
    build:
      context: .
      cache_from:
        - mjhea0/docker-ci-cache:stage
    image: mjhea0/docker-ci-cache:stage

CircleCI:

# _config-examples/multi-stage/compose/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2204:2024.01.1
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build base from dockerfile
          command: |
            docker build \
              --target base \
              --cache-from $CACHE_IMAGE:base \
              --tag $CACHE_IMAGE:base \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Build Docker images
          command: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
      - run:
          name: Push base image to docker hub
          command: docker push $CACHE_IMAGE:base
      - run:
          name: Push stage image to docker hub
          command: docker push $CACHE_IMAGE:stage

GitLab CI:

# _config-examples/multi-stage/compose/.gitlab-ci.yml

image: docker/compose:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --target base
        --cache-from $CACHE_IMAGE:base
        --tag $CACHE_IMAGE:base
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
    - docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
  after_script:
    - docker push $CACHE_IMAGE:base
    - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/compose/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v4
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build base from dockerfile
        run: |
          docker build \
            --target base \
            --cache-from $CACHE_IMAGE:base \
            --tag $CACHE_IMAGE:base \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Build images
        run: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
      - name: Push base image to docker hub
        run: docker push $CACHE_IMAGE:base
      - name: Push stage image to docker hub
        run: docker push $CACHE_IMAGE:stage

Conclusion

The caching strategies outlined in this article should work well for single-stage builds and multi-stage builds with two or three stages.

Each stage added to a build step requires a new build and push along with the addition of the --cache-from options for each parent stage. Thus, each new stage will add more clutter, making the CI file increasingly more difficult to read. Fortunately, BuildKit supports multi-stage builds with Docker layer caching built using a single stage. Review the following articles for more info on such advanced BuildKit patterns:

Finally, it's important to note that while caching may speed up your CI builds, you should re-build your images without cache from time to time in order to download the latest OS patches and security updates. For more on this, review this thread.

The code can be found in the docker-ci-cache repo:

Cheers!

Featured Course

Scalable FastAPI Applications on AWS

In this course, you'll learn how to go from idea to scalable FastAPI application running on AWS infrastructure managed by Terraform.

Buy Now $30 View Course

Share this tutorial

Contents

Docker Layer Caching

BuildKit

CI Environments

Single-stage Builds

Compose

Multi-stage Builds

Compose

Conclusion

Scalable FastAPI Applications on AWS

Scalable FastAPI Applications on AWS

Recommended Tutorials