Michael Herman
In this tutorial, we'll spin up a three-node Kubernetes cluster using Ubuntu 20.04 DigitalOcean droplets. We'll also look at how to automate the set up of a Kubernetes cluster with Python and Fabric.
Feel free to swap out DigitalOcean for a different cloud hosting provider or your own on-premise environment.
Dependencies:
- Docker v20.10
- Kubernetes v1.21
Contents
What is Fabric?
Fabric is a Python library used for automating routine shell commands over SSH, which we'll be using to automate the setup of a Kubernetes cluster.
Install:
$ pip install fabric==2.6.0
Verify the version:
$ fab --version
Fabric 2.6.0
Paramiko 2.8.0
Invoke 1.6.0
Test it out by adding the following code to a new file called fabfile.py:
from fabric import task
@task
def ping(ctx, output):
"""Sanity check"""
print("pong!")
print(f"hello {output}!")
Try it out:
$ fab ping --output="world"
pong!
hello world!
For more, review the official Fabric docs.
Droplets Setup
First, sign up for an account on DigitalOcean (if you don’t already have one), add a public SSH key to your account, and then generate an access token so you can access the DigitalOcean API.
Add the token to your environment:
$ export DIGITAL_OCEAN_ACCESS_TOKEN=<YOUR_DIGITAL_OCEAN_ACCESS_TOKEN>
Next, to interact with the API programmatically, install the python-digitalocean module:
$ pip install python-digitalocean==1.17.0
Now, let's create another task to spin up three droplets: One for the Kubernetes master and two for the workers. Update fabfile.py like so:
import os
from digitalocean import Droplet, Manager
from fabric import task
DIGITAL_OCEAN_ACCESS_TOKEN = os.getenv("DIGITAL_OCEAN_ACCESS_TOKEN")
# tasks
@task
def ping(ctx, output):
"""Sanity check"""
print("pong!")
print(f"hello {output}!")
@task
def create_droplets(ctx):
"""
Create three new DigitalOcean droplets -
node-1, node-2, node-3
"""
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
keys = manager.get_all_sshkeys()
for num in range(3):
node = f"node-{num + 1}"
droplet = Droplet(
token=DIGITAL_OCEAN_ACCESS_TOKEN,
name=node,
region="nyc3",
image="ubuntu-20-04-x64",
size="s-2vcpu-4gb",
tags=[node],
ssh_keys=keys,
)
droplet.create()
print(f"{node} has been created.")
Take note of the arguments passed to the Droplet class. Essentially, this will create three Ubuntu 20.04 droplets in the NYC3 region with 4 GB of memory each. It will also add all SSH keys to each droplet. You may want to update this to only include the SSH key that you created specifically for this project:
@task
def create_droplets(ctx):
"""
Create three new DigitalOcean droplets -
node-1, node-2, node-3
"""
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
# Get ALL SSH keys
all_keys = manager.get_all_sshkeys()
keys = []
for key in all_keys:
if key.name == "<ADD_YOUR_KEY_NAME_HERE>":
keys.append(key)
for num in range(3):
node = f"node-{num + 1}"
droplet = Droplet(
token=DIGITAL_OCEAN_ACCESS_TOKEN,
name=node,
region="nyc3",
image="ubuntu-20-04-x64",
size="s-2vcpu-4gb",
tags=[node],
ssh_keys=keys,
)
droplet.create()
print(f"{node} has been created.")
Create the droplets:
$ fab create-droplets
node-1 has been created.
node-2 has been created.
node-3 has been created.
Moving along, let's add a task that checks the status of each droplet, to ensure that each are up and ready to go before we start installing Docker and Kubernetes:
@task
def wait_for_droplets(ctx):
"""Wait for each droplet to be ready and active"""
for num in range(3):
node = f"node-{num + 1}"
while True:
status = get_droplet_status(node)
if status == "active":
print(f"{node} is ready.")
break
else:
print(f"{node} is not ready.")
time.sleep(1)
Add the get_droplet_status helper function:
def get_droplet_status(node):
"""Given a droplet's tag name, return the status of the droplet"""
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
droplet = manager.get_all_droplets(tag_name=node)
return droplet[0].status
Don't forget the import:
import time
Before we test, add another task to destroy the droplets:
@task
def destroy_droplets(ctx):
"""Destroy the droplets - node-1, node-2, node-3"""
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
for num in range(3):
node = f"node-{num + 1}"
droplets = manager.get_all_droplets(tag_name=node)
for droplet in droplets:
droplet.destroy()
print(f"{node} has been destroyed.")
Destroy the three droplets we just created:
$ fab destroy-droplets
node-1 has been destroyed.
node-2 has been destroyed.
node-3 has been destroyed.
Then, bring up three new droplets and verify that they are good to go:
$ fab create-droplets
node-1 has been created.
node-2 has been created.
node-3 has been created.
$ fab wait-for-droplets
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is ready.
node-2 is not ready.
node-2 is not ready.
node-2 is ready.
node-3 is ready.
Provision the Machines
The following tasks need to be run on each droplet...
Set Addresses
Start by adding a task to set the host addresses in the hosts environment variable:
@@task
def get_addresses(ctx, type):
"""Get IP address"""
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
if type == "master":
droplet = manager.get_all_droplets(tag_name="node-1")
print(droplet[0].ip_address)
hosts.append(droplet[0].ip_address)
elif type == "workers":
for num in range(2, 4):
node = f"node-{num}"
droplet = manager.get_all_droplets(tag_name=node)
print(droplet[0].ip_address)
hosts.append(droplet[0].ip_address)
elif type == "all":
for num in range(3):
node = f"node-{num + 1}"
droplet = manager.get_all_droplets(tag_name=node)
print(droplet[0].ip_address)
hosts.append(droplet[0].ip_address)
else:
print('The "type" should be either "master", "workers", or "all".')
print(f"Host addresses - {hosts}")
Define the following variables at the top, just below DIGITAL_OCEAN_ACCESS_TOKEN = os.getenv('DIGITAL_OCEAN_ACCESS_TOKEN'):
user = "root"
hosts = []
Run:
$ fab get-addresses --type=all
165.227.96.238
134.122.8.106
134.122.8.204
Host addresses - ['165.227.96.238', '134.122.8.106', '134.122.8.204']
With that, we can start installing the Docker and Kubernetes dependencies.
Install Dependencies
Install Docker along with-
- kubeadm - bootstraps a Kubernetes cluster
- kubelet - configures containers to run on a host
- kubectl - command line tool used managing a cluster
Add a task to install Docker to the fabfile:
@task
def install_docker(ctx):
"""Install Docker"""
print(f"Installing Docker on {ctx.host}")
ctx.sudo("apt-get update && apt-get install -qy docker.io")
ctx.run("docker --version")
ctx.sudo("systemctl enable docker.service")
Let's disable the swap file:
@task
def disable_selinux_swap(ctx):
"""
Disable SELinux so kubernetes can communicate with other hosts
Disable Swap https://github.com/kubernetes/kubernetes/issues/53533
"""
ctx.sudo('sed -i "/ swap / s/^/#/" /etc/fstab')
ctx.sudo("swapoff -a")
Install Kubernetes:
@task
def install_kubernetes(ctx):
"""Install Kubernetes"""
print(f"Installing Kubernetes on {ctx.host}")
ctx.sudo("apt-get update && apt-get install -y apt-transport-https")
ctx.sudo(
"curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -"
)
ctx.sudo(
'echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | \
tee -a /etc/apt/sources.list.d/kubernetes.list && apt-get update'
)
ctx.sudo(
"apt-get update && apt-get install -y kubelet=1.21.1-00 kubeadm=1.21.1-00 kubectl=1.21.1-00"
)
ctx.sudo("apt-mark hold kubelet kubeadm kubectl")
Instead of running each of these separately, create a main provision_machines task:
@task
def provision_machines(ctx):
for conn in get_connections(hosts):
install_docker(conn)
disable_selinux_swap(conn)
install_kubernetes(conn)
Add the get_connections helper function:
def get_connections(hosts):
for host in hosts:
yield Connection(
f"{user}@{host}",
)
Update the import:
from fabric import Connection, task
Run:
$ fab get-addresses --type=all provision-machines
This will take a few minutes to install the required packages.
Configure the Master Node
Init the Kubernetes cluster and deploy the flannel network:
@task
def configure_master(ctx):
"""
Init Kubernetes
Set up the Kubernetes Config
Deploy flannel network to the cluster
"""
ctx.sudo("kubeadm init")
ctx.sudo("mkdir -p $HOME/.kube")
ctx.sudo("cp -i /etc/kubernetes/admin.conf $HOME/.kube/config")
ctx.sudo("chown $(id -u):$(id -g) $HOME/.kube/config")
ctx.sudo(
"kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml"
)
Save the join token:
@task
def get_join_key(ctx):
sudo_command_res = ctx.sudo("kubeadm token create --print-join-command")
token = re.findall("^kubeadm.*$", str(sudo_command_res), re.MULTILINE)[0]
with open("join.txt", "w") as f:
with stdout_redirected(f):
print(token)
Add the following imports:
import re
import sys
from contextlib import contextmanager
Create the stdout_redirected context manager:
@contextmanager
def stdout_redirected(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Again, add a parent task to run these:
@task
def create_cluster(ctx):
for conn in get_connections(hosts):
configure_master(conn)
get_join_key(conn)
Run it:
$ fab get-addresses --type=master create-cluster
This will take a minute or two to run. Once done, the join token command should be outputted to the screen and saved to a join.txt file:
kubeadm join 165.227.96.238:6443 --token mvk32y.7z7i5x3viga4f4kn --discovery-token-ca-cert-hash sha256:f358dfc00ae7160fff3cb8fa3e3a3c8865f3c5b83c1f242fc9e51efe94108960
Configure the Worker Nodes
Using the saved join command from above, add a task to "join" the workers to the master:
@task
def configure_worker_node(ctx):
"""Join a worker to the cluster"""
with open("join.txt") as f:
join_command = f.readline()
for conn in get_connections(hosts):
conn.sudo(f"{join_command}")
Run this on the two worker nodes:
$ fab get-addresses --type=workers configure-worker-node
Sanity Check
Finally, to ensure the cluster is up and running, add a task to view the nodes:
@task
def get_nodes(ctx):
for conn in get_connections(hosts):
conn.sudo("kubectl get nodes")
Run:
$ fab get-addresses --type=master get-nodes
You should see something similar to:
NAME STATUS ROLES AGE VERSION
node-1 Ready control-plane,master 3m6s v1.21.1
node-2 Ready <none> 84s v1.21.1
node-3 Ready <none> 77s v1.21.1
Remove the droplets once done:
$ fab destroy-droplets
node-1 has been destroyed.
node-2 has been destroyed.
node-3 has been destroyed.
Automation Script
One last thing: Add a create.sh script to automate this full process:
#!/bin/bash
echo "Creating droplets..."
fab create-droplets
fab wait-for-droplets
sleep 20
echo "Provision the droplets..."
fab get-addresses --type=all provision-machines
echo "Configure the master..."
fab get-addresses --type=master create-cluster
echo "Configure the workers..."
fab get-addresses --type=workers configure-worker-node
sleep 20
echo "Running a sanity check..."
fab get-addresses --type=master get-nodes
Try it out:
$ sh create.sh
That's it!
You can find the scripts in the kubernetes-fabric repo on GitHub.