Deploying Vault and Consul

Last updated May 15th, 2020

Let's look at how to deploy Hashicorp's Vault and Consul to DigitalOcean with Docker Swarm.

This tutorial assumes that you have a basic working knowledge of using Vault and Consul to manage secrets. Please refer to the Managing Secrets with Vault and Consul blog post for more info.

Upon completion, you will be able to:

  1. Provision hosts on Digital Ocean with Docker Machine
  2. Configure a Docker Swarm cluster to run on Digital Ocean
  3. Run Vault and Consul on Docker Swarm

Main dependencies:

  • Docker v19.03.8
  • Docker-Compose v1.25.4
  • Docker-Machine v0.16.2
  • Vault v1.3.5
  • Consul v1.7.3

Contents

Consul

Create a new project directory:

$ mkdir vault-consul-swarm && cd vault-consul-swarm

Then, add a docker-compose.yml file to the project root:

version: '3.7'

services:

  server-bootstrap:
    image: consul:1.7.3
    ports:
      - 8500:8500
    command: "agent -server -bootstrap-expect 3 -ui -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"

  server:
    image: consul:1.7.3
    command: "agent -server -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
    deploy:
      replicas: 2
    depends_on:
      - server-bootstrap

  client:
    image: consul:1.7.3
    command: "agent -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
    deploy:
      replicas: 2
    depends_on:
      - server-bootstrap

networks:
  default:
    external: true
    name: core

This configuration should look familiar.

  1. Refer to the Compose File section of the Running Flask on Docker Swarm blog post for more info on using a compose file for Docker Swarm mode.
  2. Review the Consul with Containers guide for info on the above Consul config.

Docker Swarm

Sign up for a DigitalOcean account (if you don’t already have one), and then generate an access token so you can access the DigitalOcean API.

Add the token to your environment:

$ export DIGITAL_OCEAN_ACCESS_TOKEN=[your_digital_ocean_token]

Spin up three droplets:

$ for i in 1 2 3; do
    docker-machine create \
      --digitalocean-region "nyc1" \
      --driver digitalocean \
      --digitalocean-size "8gb" \
      --digitalocean-access-token $DIGITAL_OCEAN_ACCESS_TOKEN \
      node-$i;
done

Initialize Swarm mode on the first node, node-1:

$ docker-machine ssh node-1 -- docker swarm init --advertise-addr $(docker-machine ip node-1)

Use the join token from the output of the previous command to add the remaining two nodes to the Swarm as workers:

$ for i in 2 3; do
    docker-machine ssh node-$i -- docker swarm join --token YOUR_JOIN_TOKEN HOST:PORT;
done

For example:

for i in 2 3; do
    docker-machine ssh node-$i -- docker swarm join --token SWMTKN-1-18xrfgcgq7k6krqr7tvav3ydx5c5104y662lzh4pyct2t0ror3-e3ed1ggivhf8z15i40z6x55g5 67.205.165.166:2377;
done

You should see:

This node joined a swarm as a worker.
This node joined a swarm as a worker.

Point the Docker daemon at node-1, create an attachable overlay network (called core), and deploy the stack:

$ eval $(docker-machine env node-1)
$ docker network create -d overlay --attachable core
$ docker stack deploy --compose-file=docker-compose.yml secrets

List out the services in the stack:

$ docker stack ps -f "desired-state=running" secrets

You should see something similar to:

ID             NAME                         IMAGE          NODE     DESIRED STATE   CURRENT STATE
b5f5eycrhf3o   secrets_client.1             consul:1.7.3   node-1   Running         Running 7 seconds ago
zs7a5t8khcew   secrets_server.1             consul:1.7.3   node-2   Running         Running 9 seconds ago
qnhtlan6m0sp   secrets_server-bootstrap.1   consul:1.7.3   node-1   Running         Running 7 seconds ago
u61eycesmsl7   secrets_client.2             consul:1.7.3   node-2   Running         Running 9 seconds ago
vgpql8lfy5fi   secrets_server.2             consul:1.7.3   node-3   Running         Running 9 seconds ago

Grab the IP associated with node-1:

$ docker-machine ip node-1

Then, test out the Consul UI in your browser at http://YOUR_MACHINE_IP:8500/ui. There should be three running services and five nodes.

consul ui services

consul ui nodes

Vault

Add the vault service to docker-compose.yml:

vault:
  image: vault:1.3.5
  deploy:
    replicas: 1
  ports:
    - 8200:8200
  environment:
    - VAULT_ADDR=http://127.0.0.1:8200
    - VAULT_LOCAL_CONFIG={"backend":{"consul":{"address":"http://server-bootstrap:8500","path":"vault/"}},"listener":{"tcp":{"address":"0.0.0.0:8200","tls_disable":1}},"ui":true, "disable_mlock":true}
  command: server
  depends_on:
    - consul

Take note of the VAULT_LOCAL_CONFIG environment variable:

{
  "backend": {
    "consul": {
      "address": "http://server-bootstrap:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true,
  "disable_mlock": true
}

Review the Consul Backend section from the Managing Secrets with Vault and Consul blog post for more info. Also, setting disable_mlock to true is not recommended for production environments; however, it must be enabled since --cap-add is not available in Docker Swarm mode. See the following GitHub issues for details:

  1. --cap-add=IPC_LOCK unavailable in docker swarm
  2. Missing from Swarmmode --cap-add

Test

Re-deploy the stack:

$ docker stack deploy --compose-file=docker-compose.yml secrets

Wait a few seconds for the services to spin up, then check the status:

$ docker stack ps -f "desired-state=running" secrets

Again, you should see something similar to:

ID              NAME                         IMAGE           NODE      DESIRED STATE   CURRENT STATE
k5i2ru4kov26    secrets_vault.1              vault:1.3.5     node-3    Running         Running 7 seconds ago
b5f5eycrhf3o    secrets_client.1             consul:1.7.3    node-1    Running         Running 19 minutes ago
zs7a5t8khcew    secrets_server.1             consul:1.7.3    node-2    Running         Running 19 minutes ago
qnhtlan6m0sp    secrets_server-bootstrap.1   consul:1.7.3    node-1    Running         Running 19 minutes ago
u61eycesmsl7    secrets_client.2             consul:1.7.3    node-2    Running         Running 19 minutes ago
vgpql8lfy5fi    secrets_server.2             consul:1.7.3    node-3    Running         Running 19 minutes ago

Next, ensure Vault is listed on the "Services" section of the Consul UI:

consul ui services

You should now be able to interact with Vault via the CLI, HTTP API, and the UI. We'll be using the UI in this tutorial to unseal, log in, and interact with the secrets backend, but feel free to test things out on your own via the CLI or HTTP API.

Init, unseal, and log in:

vault init

Create a new secret:

vault add secret

Remove the nodes once done:

$ docker-machine rm node-1 node-2 node-3 -y

Automation Script

Finally, let's create a quick script to automate the deployment process:

  1. Provision three DigitalOcean droplets with Docker Machine
  2. Configure Docker Swarm mode
  3. Add nodes to the Swarm
  4. Deploy the stack

Add a new file called deploy.sh to the project root:

#!/bin/bash


echo "Spinning up three droplets..."

for i in 1 2 3; do
  docker-machine create \
    --digitalocean-region "nyc1" \
    --driver digitalocean \
    --digitalocean-size "8gb" \
    --digitalocean-access-token $DIGITAL_OCEAN_ACCESS_TOKEN \
    node-$i;
done


echo "Initializing Swarm mode..."

docker-machine ssh node-1 -- docker swarm init --advertise-addr $(docker-machine ip node-1)


echo "Adding the nodes to the Swarm..."

TOKEN=`docker-machine ssh node-1 docker swarm join-token worker | grep token | awk '{ print $5 }'`

for i in 2 3; do
  docker-machine ssh node-$i \
    -- docker swarm join --token ${TOKEN} $(docker-machine ip node-1):2377;
done


echo "Creating networking..."

eval $(docker-machine env node-1)
docker network create -d overlay --attachable core


echo "Deploying the stack..."

docker stack deploy --compose-file=docker-compose.yml secrets

Try it out!

$ sh deploy.sh

Bring down the droplets once done:

$ docker-machine rm node-1 node-2 node-3 -y

The code can be found in the vault-consul-swarm repo. Cheers!

Featured Course

Test-Driven Development with Django, Django REST Framework, and Docker

In this course, you'll learn how to set up a development environment with Docker in order to build and deploy a RESTful API powered by Python, Django, and Django REST Framework.

Featured Course

Test-Driven Development with Django, Django REST Framework, and Docker

In this course, you'll learn how to set up a development environment with Docker in order to build and deploy a RESTful API powered by Python, Django, and Django REST Framework.