In the upcoming blog posts, I will cover monitoring with Prometheus in a Kubernetes infrastructure. As I started blogging, I decided it would be useful to delve into the design of the k8s infrastructure.
This will allow you to approach these blogs better if you wish to reproduce them.
Therefore, I will start with the basics and explain step by step how to build a Kubernetes cluster with kubeadm easily.
I suggest we start with a relatively simple, if not basic, installation with a cluster including a Master (Control Plane) and two worker nodes. To do this, I will use a Debian distribution, with a rather minimalist sizing with 2 CPUs and 4GB of RAM each.
Install Packages
1. Login to your master server (Control Plane)
2. Create a configuration file containerd, which will contain the necessary packages for the container runtime :
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
3. Load the overlay and br_netfilter kernel modules:
sudo modprobe overlay
sudo modprobe br_netfilter
4. Set system configurations for Kubernetes networking:
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
5. Load sysctl settings from all available configuration files:
sudo sysctl --system
6. Update the package list and install the “containerd” package:
sudo apt-get update && sudo apt-get install -y containerd
7. Create the /etc/containerd directory if it doesn’t exist (we will use it to store the default configuration file for containerd ):
sudo mkdir -p /etc/containerd
8. Generate a default config file for containerd and save it to the newly created default file:
sudo containerd config default | sudo tee /etc/containerd/config.toml
9. Restart containerd to ensure new configuration file usage:
sudo systemctl restart containerd
10. Verify that containerd is running:
sudo systemctl status containerd
dbinla@dbisbx-master01:~$ sudo systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-01-09 10:28:12 UTC; 22s ago
Docs: https://containerd.io
Process: 2013 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 2014 (containerd)
Tasks: 8
Memory: 10.8M
CPU: 140ms
CGroup: /system.slice/containerd.service
└─2014 /usr/bin/containerd
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.551429182Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.551607403Z" level=info msg=serving... address=/run/containerd/containerd.sock
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.551954435Z" level=info msg="containerd successfully booted in 0.051594s"
Jan 09 10:28:12 dbisbx-master01 systemd[1]: Started containerd container runtime.
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.557282442Z" level=info msg="Start subscribing containerd event"
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.557379562Z" level=info msg="Start recovering state"
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.557588184Z" level=info msg="Start event monitor"
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.557619734Z" level=info msg="Start snapshots syncer"
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.557634754Z" level=info msg="Start cni network conf syncer"
Jan 09 10:28:12 dbisbx-master01 containerd[2014]: time="2023-01-09T10:28:12.557649394Z" level=info msg="Start streaming server"
dbinla@dbisbx-master01:~$11. Disable swap:
sudo swapoff -a
12. Update and install dependency packages:
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
13. Download and add the Google Cloud public signing key:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
Depending on the version of your distribution, you might have a warning about the deprecated command. Therefore you can type instead of:
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
14. Create a new file called “/etc/apt/sources.list.d/kubernetes.list” and add the Kubernetes apt repository to it:
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
15. Update package listings:
sudo apt-get update
16. Install specific versions of the kubelet, kubeadm, and kubectl packages :
sudo apt-get install -y kubelet=1.24.0-00 kubeadm=1.24.0-00 kubectl=1.24.0-0017. Mark the kubelet, kubeadm, and kubectl packages as “held” to prevent them from being automatically upgraded:
sudo apt-mark hold kubelet kubeadm kubectl
18. The installation of the packages must be performed on all servers. The entire part 1 must be repeated on the worker nodes.
Initialize the Cluster
1. This part had to be done only on the Master server (Control Plane)
Install specific versions of the kubelet, kubeadm, and kubectl packages. And then initialize the Kubernetes cluster with kubeadm, specifying the pod network CIDR and the Kubernetes version
sudo apt-get install -y kubelet=1.24.0-00 kubeadm=1.24.0-00 kubectl=1.24.0-00
sudo kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.24.0
dbinla@dbisbx-master01:~$ sudo kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.24.0
[init] Using Kubernetes version: v1.24.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [dbisbx-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.*.*.* 172.*.*.*]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [dbisbx-master01 localhost] and IPs [172.*.*.* 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [dbisbx-master01 localhost] and IPs [172.*.*.* 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 53.186020 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node dbisbx-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node dbisbx-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: vnv0kw.xmh*************
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.*.*.*:6443 --token vnv0kw.xmh************* \
--discovery-token-ca-cert-hash sha256:eaec80d623e107619c02f743a726d8e5f6******************************
dbinla@dbisbx-master01:~$2. Once the initialization is done, we have to set the kubectl access.
For this aim, let’s create the .kube folder in our user’s home directory if it doesn’t already exist.
We will then copy the admin.conf file from /etc/kubernetes to the .kube directory. Finally, we will change the owner of the copied file to the current user.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
3. Test the access to the cluster.
kubectl get nodes
dbinla@dbisbx-master01:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
dbisbx-master01 NotReady control-plane 02m v1.24.0
dbinla@dbisbx-master01:~$Install the Calico Network Add-On
1. On the control plane node, install Calico Networking by creating the necessary custom resource:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml
More information on configuration option are available here
dbinla@dbisbx-master01:~$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
dbinla@dbisbx-master01:~$2. Check the status of the control plane node:
kubectl get nodes -owide
dbinla@dbisbx-master01:~$ kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
dbisbx-master01 Ready control-plane 07m v1.24.0 172.*.*.* <none> Ubuntu 20.04.5 LTS 5.15.0-1027-aws containerd://1.5.9
Join the worker nodes to k8s cluster
In the chapter “Cluster Initialization”, we retrieved the command allowing us to add our workers to the cluster.
You can retrieve the command and run it on each worker.
In case you didn’t copy the command, you still have the possibility to display the join command by recreating the token that will allow you to join your nodes.
1. From the control plane server, execute the following command.
kubeadm token create --print-join-command
dbinla@dbisbx-master01:~$ kubeadm token create --print-join-command
kubeadm join 172.31.99.146:6443 --token u871a6.k*************** --discovery-token-ca-cert-hash sha256:eaec80d623e107619c02f743a726*********************************
2. On the first worker node, execute the previous printed command "kubeadm join <IP:PORT> ..." being root, or running as root user.
sudo kubeadm join 172.*.*.*:6443 --token u871a6.kbzwx2j6jztxyvg3 --discovery-token-ca-cert-hash sha256:eaec80d623e107619c02f743a726*********************************
dbinla@dbisbx-worker01:~$ sudo kubeadm join 172.*.*.*:6443 --token u871a6.kbzwx2j6jztxyvg3 --discovery-token-ca-cert-hash sha256:eaec80d623e107619c02f743a726*********************************
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
dbinla@dbisbx-worker01:~$All that remains is to check that the node has been added to the cluster; it may take a few minutes for the node to go to the ready status
kubectl get nodes -owide
dbinla@dbisbx-master01:~$ kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
dbisbx-master01 Ready control-plane 19m v1.24.0 172.31.99.146 <none> Ubuntu 20.04.5 LTS 5.15.0-1027-aws containerd://1.5.9
dbisbx-worker01 Ready <none> 05m v1.24.0 172.31.101.246 <none> Ubuntu 20.04.5 LTS 5.15.0-1027-aws containerd://1.5.9
dbisbx-worker02 Ready <none> 03m v1.24.0 172.31.96.172 <none> Ubuntu 20.04.5 LTS 5.15.0-1027-aws containerd://1.5.9Here we are; we could quickly build a small Kubernetes cluster in less than 20 minutes.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/NLA_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2023/03/KKE_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/ENB_web-min-scaled.jpg)