Deploy Kubernetes

Setup Linux for Kubernetes

Since Kubernetes doesn’t support swap you must disable it on any node you plan on running kubernetes

swapoff -a cp /etc/fstab /etc/fstab.bck sed -e '/swap/s/^/#/g' -i /etc/fstab

If that doesn’t work try this command

sudo sed -i '/swap/d' /etc/fstab

The Kubernetes project doesn’t support the use of swap as referred to in KEP-2400.

Verify swap is turned off by running

sudo cat /etc/fstab | grep swap

You should see the # at the start of the line to ensure the swap disk will not be mounted on a reboot or you may just not see anything since no definition of swap exists in fstab file

Also run

sudo cat /proc/swaps

The return of the command should have nothing in the list

Disable firewall

sudo systemctl disable --now firewalld

While keeping firewalld enabled is supported with additional firewall rules, it does complicate and conflict with the built-in support with ip tables.

Verify firewall is disabled

sudo systemctl status firewalld --lines=0

The return of the command should have Active set to inactive OR you’ll get the Unit firewalld.service could not be found because its not installed

**** If command line becomes unresponsive, hit CTRL + C to fix terminal.

Disable selinux

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

While leaving SELinux in an enforcing state is ideal for security reasons, you are being instructed to disable SELinux in this lab to reduce any potential friction. With that said, if you ware setting up your own Kubernetes cluster, you should evaluate if you want to leave SELinux enabled, and be prepared to add any necessary changes to SELinux to support deviations that it may block.

Verify selinux is disabled

sudo getenforce

The output should say Permissive or at least not Enforcing

Install and enable containerd

# The below command will remove any conflicts
sudo yum remove -y containers-common

# Setup the Yum Repository to install containerd from
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo  https://download.docker.com/linux/centos/docker-ce.repo 

# install ContainerD
sudo yum install -y containerd.io

# Have ContainerD create the default config file
sudo containerd config default > /etc/containerd/config.toml

# Start and enable the ContainerD daemon
sudo systemctl enable --now containerd

You will notice that the URL provided for the repository is coming from Docker. This is because Docker the company manages the package installer for ContainerD. Otherwise, you would need to install ContainerD from the source code. This does NOT mean that we are using Docker as the runtime.

Run the following to setup the ipv4 forwarding and enabling iptables to see bridged traffic

sudo cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system
sudo yum install -y iproute-tc

In the above set of commands, you are sending config data to /etc/modules-load.d/k8s.conf and /etc/sysctl.d/k8s.conf, and then running modprobe and sysctl commands in order to enable kernel the features (overlay and br_netfilter) that are turned off by default on this distribution, but are needed by Kubernetes. Kubernetes uses these features to provide proxy functionality that allows pods to communicate to each other across the cluster.

Validate that ContainerD is running

sudo systemctl status containerd --lines=0

The output should have Active set to active

Ensure that br_netfilter is enabled

sudo lsmod | grep br_netfilter

The output should have content. If it wasn’t enabled the response would be blank

Ensure that overlay is enabled.

sudo lsmod | grep overlay

The output should have content. If it wasn’t enabled the response would be blank

Ensure that net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-ip6tables, and net.ipv4.ip_forward are enabled.

sudo sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

The output should have each entry set to 1. If it wasn’t enabled the response would be blank

The installation of containerd also results in the installation of runc, check that this is available in a system path

sudo which runc

Containerd is bundled with an unsupported debug administrative tool known as ctr, check that this is available in your path

sudo which ctr

Deploy using docker commands

First pull and image of ubuntu using docker. Treat the container as an instance of your application, but your application is embedded into an image. The proper usage would be creating a custom image, where you embed all your files, configurations, environment variables etc, into the image then in order to use your application, you just run the image with proper port settings or other dynamic variables, using docker run <your-image>

docker run --rm -it ubuntu bash

Running containers with --rm flag is good for those containers that you use for very short while just to accomplish something, e.g., compile your application inside a container, or just testing something and then you know it’s a short lived container and you tell your Docker daemon that once it’s done running, erase everything related to it and save the disk space.

Running containers with --it flag starts the container in the interactive mode that allows you to interact with /bin/bash of the container. That means now you will have bash session inside the container, so you can ls, mkdir, or do any bash command inside the container.

sudo docker run –name cnitest –net=none -d jonlangemak/web_server_1
835583cdf382520283c709b5a5ee866b9dccf4861672b95eccbc7b7688109b56

Deploy using ctr commands

Run the equivalent docker command of docker run –rm -it ubuntu bash through ctr.  Unlike docker which will automatically pull an image if it doesn’t exist, ctr requires this to be actioned manually

sudo ctr images pull docker.io/library/ubuntu:latest

Firstly, we’ll run a container: -t to create a tty, -d to detach, docker.io/library/ubuntu:latest for the container image and a name of quirky

sudo ctr run -t -d docker.io/library/ubuntu:latest quirky

We can verify that the container is running via ctr, note that runc is used the low level container runtime

sudo ctr container list

And we can see the task (process) that is running, this is the process on our running system for this container

sudo ctr task list

If we check our running processes we can see containerd running a container and a process of bash running within

sudo ps -ef --forest

And now we can attach, to the running container

sudo ctr task attach quirky

Whilst in the container, if we execute a ps we can see our running process of bash

ps

This container has no external network connectivity, if you try an apt update, this will fail

apt update

When you’ve finished using this container, you can exit

exit

As we attached to the running container task which was our bash shell and instructed bash to exit, our task will no longer exist, lets check, note, we can use ls instead of list as a shortcut

sudo ctr task ls

Although no tasks are running, our container will still exist

sudo ctr container list

We will now delete the container

sudo ctr container delete quirky

Deploy using nerdctl commands

Whilst ctr is useful, it’s not friendly to use like the docker-cli, let’s install nerdctl to provide a docker-cli like environment for containerd. Extract the nerdctl binary and place in /usr/local/bin

sudo tar -xzvf /resources/nerdctl-1.1.0-linux.tar.gz --directory /usr/local/bin nerdctl

Nerdctl requires CNI Plugins, the same ones used by Kubernetes CNI solutions. Unpack the plugins to /opt/cni/bin

tar -xzvf /resources/cni-plugins-linux-v1.2.0.tgz -C /opt/cni/bin

Before we run nerdctl, verify that the standard CNI configuration directory of /etc/cni/net.d is empty

sudo ls -altr /etc/cni/net.d

When nerdctl runs for the first time it will bootstrap a CNI network for it’s use. Let’s try this out, we’ll run a docker like command but we’ll substitute nerdctl for docker, we’ll use the following options –

run: to run a container
–rm: to cleanup up
-it: for interactive
ubuntu: the image we shall use
bash: the command we will run

sudo nerdctl run --rm -it ubuntu bash

Whilst in the container, we could run a command like we did when in the ctr container, for example

ps -ef

Let’s check our network connectivity by running an apt update

apt update

We will install iproute2 to provide the ip command

apt install -y iproute2

And we can also see that this container has an ip address

ip addr

Let’s exit the container

exit

When we executed nerdctl, it boostraped a CNI configuration file that can be viewed here

sudo cat /etc/cni/net.d/nerdctl-bridge.conflist

We can see the nerdctl0 bridge interface that relates to this in ip addr, notice the 10.4.0.1/24 subnet that relates to the entry in the configuration file

sudo ip addr

As Kubernetes makes use of the same directory to search for a CNI configuration file, let’s move that file out of the way

sudo mv /etc/cni/net.d/nerdctl-bridge.conflist /resources

At this stage, you have deployed the CRI needed to run containers on the nodes that will end up being part of the cluster

Ways to Monitor your container

Install htop using apt, htop is an interactive system monitor, process viewer and process manager designed for Unix systems

sudo apt install -y htop

htop will be installed to a system path, we can confirm this by executing

sudo which htop

As /usr/bin is in our system PATH by default, we can run htop just by issuing the command htop

sudo htop

With htop running, now is a good time to try tweaking the terminal. Click Settings and try changing the Terminal Font, Terminal Font Size and the Terminal Theme