Michael Herman
In this tutorial, we'll spin up a three-node Kubernetes cluster using Ubuntu 20.04 DigitalOcean droplets. We'll also look at how to automate the setup of a Kubernetes cluster with Python and Fabric.
Feel free to swap out DigitalOcean for a different cloud hosting provider or your own on-premise environment.
Dependencies:
- Docker v19.03.8
- Kubernetes v1.18.3
Contents
What is Fabric?
Fabric is a Python library used for automating routine shell commands over SSH, which we'll be using to automate the setup of a Kubernetes cluster.
Install:
$ pip install fabric==2.5.0
Verify the version:
$ fab --version
Fabric 2.5.0
Paramiko 2.7.1
Invoke 1.4.1
Test it out by adding the following code to a new file called fabfile.py:
from fabric import task
@task
def ping(ctx, output):
""" Sanity check """
print(f'pong!')
print(f'hello {output}!')
Try it out:
$ fab ping --output="world"
pong!
hello world!
For more, review the official Fabric docs.
Droplets Setup
First, sign up for an account on DigitalOcean (if you don’t already have one), add a public SSH key to your account, and then generate an access token so you can access the DigitalOcean API.
Add the token to your environment:
$ export DIGITAL_OCEAN_ACCESS_TOKEN=<YOUR_DIGITAL_OCEAN_ACCESS_TOKEN>
Next, to interact with the API programmatically, install the python-digitalocean module:
$ pip install python-digitalocean==1.15.0
Now, let's create another task to spin up three droplets: one for the Kubernetes master and two for the workers. Update fabfile.py like so:
import os
from fabric import task
from digitalocean import Droplet, Manager
DIGITAL_OCEAN_ACCESS_TOKEN = os.getenv('DIGITAL_OCEAN_ACCESS_TOKEN')
# tasks
@task
def ping(ctx, output):
""" Sanity check """
print(f'pong!')
print(f'hello {output}!')
@task
def create_droplets(ctx):
"""
Create three new DigitalOcean droplets -
node-1, node-2, node-3
"""
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
keys = manager.get_all_sshkeys()
for num in range(3):
node = f'node-{num + 1}'
droplet = Droplet(
token=DIGITAL_OCEAN_ACCESS_TOKEN,
name=node,
region='nyc3',
image='ubuntu-20-04-x64',
size_slug='4gb',
tags=[node],
ssh_keys=keys,
)
droplet.create()
print(f'{node} has been created.')
Take note of the arguments passed to the Droplet class. Essentially, we're creating three Ubuntu 20.04 droplets in the NYC3 region with 4 GB of memory each. We're also adding all SSH keys to each droplet. You may want to update this to only include the SSH key that you created specifically for this project:
@task
def create_droplets(ctx):
"""
Create three new DigitalOcean droplets -
node-1, node-2, node-3
"""
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
# Get ALL SSH keys
all_keys = manager.get_all_sshkeys()
keys = []
for key in all_keys:
if key.name == '<ADD_YOUR_KEY_NAME_HERE>':
keys.append(key)
for num in range(3):
node = f'node-{num + 1}'
droplet = Droplet(
token=DIGITAL_OCEAN_ACCESS_TOKEN,
name=node,
region='nyc3',
image='ubuntu-20-04-x64',
size_slug='4gb',
tags=[node],
ssh_keys=keys,
)
droplet.create()
print(f'{node} has been created.')
Create the droplets:
$ fab create-droplets
node-1 has been created.
node-2 has been created.
node-3 has been created.
Moving along, let's add a task that checks the status of each droplet, to ensure that each are up and ready to go before we start installing Docker and Kubernetes:
@task
def wait_for_droplets(ctx):
""" Wait for each droplet to be ready and active """
for num in range(3):
node = f'node-{num + 1}'
while True:
status = get_droplet_status(node)
if status == 'active':
print(f'{node} is ready.')
break
else:
print(f'{node} is not ready.')
time.sleep(1)
Add the get_droplet_status helper function:
def get_droplet_status(node):
""" Given a droplet's tag name, return the status of the droplet """
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
droplet = manager.get_all_droplets(tag_name=node)
return droplet[0].status
Don't forget the import:
import time
Before we test, add another task to destroy the droplets:
@task
def destroy_droplets(ctx):
""" Destroy the droplets - node-1, node-2, node-3 """
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
for num in range(3):
node = f'node-{num + 1}'
droplets = manager.get_all_droplets(tag_name=node)
for droplet in droplets:
droplet.destroy()
print(f'{node} has been destroyed.')
Destroy the three droplets we just created:
$ fab destroy-droplets
node-1 has been destroyed.
node-2 has been destroyed.
node-3 has been destroyed.
Then, bring up three new droplets and verify that they are good to go:
$ fab create-droplets
node-1 has been created.
node-2 has been created.
node-3 has been created.
$ fab wait-for-droplets
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is ready.
node-2 is not ready.
node-2 is not ready.
node-2 is ready.
node-3 is ready.
Provision the Machines
The following tasks need to be run on each droplet...
Set Addresses
Start by adding a task to set the host addresses in the hosts environment variable:
@task
def get_addresses(ctx, type):
""" Get IP address """
manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
if type == 'master':
droplet = manager.get_all_droplets(tag_name='node-1')
print(droplet[0].ip_address)
hosts.append(droplet[0].ip_address)
elif type == 'workers':
for num in range(2, 4):
node = f'node-{num}'
droplet = manager.get_all_droplets(tag_name=node)
print(droplet[0].ip_address)
hosts.append(droplet[0].ip_address)
elif type == 'all':
for num in range(3):
node = f'node-{num + 1}'
droplet = manager.get_all_droplets(tag_name=node)
print(droplet[0].ip_address)
hosts.append(droplet[0].ip_address)
else:
print('The "type" should be either "master", "workers", or "all".')
print(f'Host addresses - {hosts}')
Define the following variables at the top, just below DIGITAL_OCEAN_ACCESS_TOKEN = os.getenv('DIGITAL_OCEAN_ACCESS_TOKEN'):
user = 'root'
hosts = []
Run:
$ fab get-addresses --type=all
165.227.103.30
159.65.182.113
165.227.222.71
Host addresses - ['165.227.103.30', '159.65.182.113', '165.227.222.71']
With that, we can start installing the Docker and Kubernetes dependencies.
Install Dependencies
Install Docker along with-
- kubeadm - bootstraps a Kubernetes cluster
- kubelet - configures containers to run on a host
- kubectl - command line tool used managing a cluster
Add a task to install Docker to the fabfile:
@task
def install_docker(ctx):
""" Install Docker """
print(f'Installing Docker on {ctx.host}')
ctx.sudo('apt-get update && apt-get install -qy docker.io')
ctx.run('docker --version')
ctx.sudo('systemctl enable docker.service')
Let's disable the swap file:
@task
def disable_selinux_swap(ctx):
"""
Disable SELinux so kubernetes can communicate with other hosts
Disable Swap https://github.com/kubernetes/kubernetes/issues/53533
"""
ctx.sudo('sed -i "/ swap / s/^/#/" /etc/fstab')
ctx.sudo('swapoff -a')
Install Kubernetes:
@task
def install_kubernetes(ctx):
""" Install Kubernetes """
print(f'Installing Kubernetes on {ctx.host}')
ctx.sudo('apt-get update && apt-get install -y apt-transport-https')
ctx.sudo('curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -')
ctx.sudo('echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | \
tee -a /etc/apt/sources.list.d/kubernetes.list && apt-get update')
ctx.sudo('apt-get update && apt-get install -y kubelet kubeadm kubectl')
Instead of running each of these separately, create a main provision_machines task:
@task
def provision_machines(ctx):
for conn in get_connections(hosts):
install_docker(conn)
disable_selinux_swap(conn)
install_kubernetes(conn)
Add the get_connections helper function:
def get_connections(hosts):
for host in hosts:
yield Connection(f"{user}@{host}")
Update the import:
from fabric import task, Connection
Run:
$ fab get-addresses --type=all provision-machines
This will take a few minutes to install the required packages.
Configure the Master Node
Init the Kubernetes cluster and deploy the flannel network:
@task
def configure_master(ctx):
"""
Init Kubernetes
Set up the Kubernetes Config
Deploy flannel network to the cluster
"""
ctx.sudo('kubeadm init')
ctx.sudo('mkdir -p $HOME/.kube')
ctx.sudo('cp -i /etc/kubernetes/admin.conf $HOME/.kube/config')
ctx.sudo('chown $(id -u):$(id -g) $HOME/.kube/config')
ctx.sudo('kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml')
Save the join token:
@task
def get_join_key(ctx):
sudo_command_res = ctx.sudo('kubeadm token create --print-join-command')
token = re.findall("^kubeadm.*$", str(sudo_command_res), re.MULTILINE)[0]
with open('join.txt', "w") as f:
with stdout_redirected(f):
print(token)
Add the following imports:
import re
import sys
from contextlib import contextmanager
Create the stdout_redirected context manager:
@contextmanager
def stdout_redirected(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Again, add a parent task to run these:
@task
def create_cluster(ctx):
for conn in get_connections(hosts):
configure_master(conn)
get_join_key(conn)
Run it:
$ fab get-addresses --type=master create-cluster
This will take a minute or two to run. Once done, the join token command should be outputted to the screen and saved in a join.txt file:
kubeadm join 165.227.103.30:6443 --token dabsh3.itdhdo45fxj65lrb --discovery-token-ca-cert-hash sha256:5af14ed1388b240e25fe2b3bbaa38752c6a23328516e47aedef501d4db4057af
Configure the Worker Nodes
Using the saved join command from above, add a task to "join" the workers to the master:
@task
def configure_worker_node(ctx):
""" Join a worker to the cluster """
with open('join.txt') as f:
join_command = f.readline()
for conn in get_connections(hosts):
conn.sudo(f'{join_command}')
Run this on the two worker nodes:
$ fab get-addresses --type=workers configure-worker-node
Sanity Check
Finally, to ensure the cluster is up and running, add a task to view the nodes:
@task
def get_nodes(ctx):
for conn in get_connections(hosts):
conn.sudo('kubectl get nodes')
Run:
$ fab get-addresses --type=master get-nodes
You should see something similar to:
NAME STATUS ROLES AGE VERSION
node-1 Ready master 44s v1.18.3
node-2 Ready <none> 30s v1.18.3
node-3 Ready <none> 26s v1.18.3
Remove the droplets once done:
$ fab destroy-droplets
node-1 has been destroyed.
node-2 has been destroyed.
node-3 has been destroyed.
Automation Script
One last thing: Add a create.sh script to automate this full process:
#!/bin/bash
echo "Creating droplets..."
fab create-droplets
fab wait-for-droplets
sleep 20
echo "Provision the droplets..."
fab get-addresses --type=all provision-machines
echo "Configure the master..."
fab get-addresses --type=master create-cluster
echo "Configure the workers..."
fab get-addresses --type=workers configure-worker-node
sleep 20
echo "Running a sanity check..."
fab get-addresses --type=master get-nodes
Try it out:
$ sh create.sh
That's it!
You can find the scripts in the kubernetes-fabric repo on GitHub.