In this post I’ll share my experience with installing Cilium in Minikube on my Mac. It was not that hard but I faced an issue I had to troubleshoot. I’ll also show you how to create several Kubernetes clusters in Minikube in order to have one with Calico as CNI and another one with Cilium as CNI. This way I can switch from one to the other according to my needs.

Cilium is a Container Network Interface (CNI) based on eBPF (a recent linux kernel technology that offers better performance than the traditional iptables) which provides network connectivity between the pods as does Calico, Flannel, Weave,… In addition to CNI, Cilium provides also more advanced features in Networking, Observability and Security.

Calico Cluster in Minikube

Let’s start by creating our Calico cluster in Minikube with the following command:

enb@DBI-LT-ENB ~ % minikube start --nodes 2 --network-plugin=cni --cni=calico -p cluster-calico

With this command I’ll start a cluster with 2 nodes and use Calico as CNI. The name of this cluster is cluster-calico.

Cilium Cluster in Minikube

The installation instructions are provided on the Cilium website and include instructions for a cluster using Minikube: https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/

By adapting those instructions we can now create another cluster in Minikube as follows:

enb@DBI-LT-ENB ~ % minikube start --nodes 3 --network-plugin=cni --cni=false -p cluster-cilium

This cluster is named cluster-cilium and will have 3 nodes and will be ready to use the cilium.

Then we can install Cilium CLI and Cilium as per the instructions.

The issue

Unfortunately, the Cilium installation didn’t complete as expected, below is the output of the cilium install command:

enb@DBI-LT-ENB ~ % cilium install
Using Cilium version 1.12.1
Auto-detected cluster name: cluster-cilium
Auto-detected datapath mode: tunnel
Auto-detected kube-proxy has been installed
helm template --namespace kube-system cilium cilium/cilium --version 1.12.1 --set cluster.id=0,cluster.name=cluster-cilium,encryption.nodeEncryption=false,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
Storing helm values file in kube-system/cilium-cli-helm-values Secret
Created CA in secret cilium-ca
Generating certificates for Hubble...
Creating Service accounts...
Creating Cluster roles...
Creating ConfigMap for Cilium version 1.12.1...
Creating Agent DaemonSet...
Creating Operator Deployment...
Waiting for Cilium to be installed and ready...
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         1 errors, 3 warnings
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 3, Unavailable: 3/3
Containers:       cilium             Pending: 3
                  cilium-operator    Running: 1
Cluster Pods:     0/1 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.1@sha256:ea2db1ee21b88127b5c18a96ad155c25485d0815a667ef77c2b7c7f31cab601b: 3
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.1@sha256:93d5aaeda37d59e6c4325ff05030d7b48fabde6576478e3fdbfb9bb4a68ec4a1: 1
Errors:           cilium             cilium          3 pods of DaemonSet cilium are not ready
Warnings:         cilium             cilium-ccmms    pod is pending
                  cilium             cilium-hlwvq    pod is pending
                  cilium             cilium-vpjbq    pod is pending
↩️ Rolling back installation...

Error: Unable to install Cilium: timeout while waiting for status to become successful: context deadline exceeded

We can see that the cilium pod stay in the pending state and so the installation fails. This is confirmed with the following command that gives more information:

enb@DBI-LT-ENB ~ % k get po -n kube-system
NAME                                     READY   STATUS                  RESTARTS      AGE
cilium-operator-6f5c6f768d-zpgjl         1/1     Running                 0             7m5s
cilium-pw7bm                             0/1     Init:ImagePullBackOff   0             7m5s
cilium-qnf4j                             0/1     Init:ImagePullBackOff   0             7m5s
cilium-wzz2t                             0/1     Init:ImagePullBackOff   0             7m5s

The Events section of one of those pod (using kubectl describe pod cilium-pw7bm -n kube-system) gives the output below:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  7m21s                  default-scheduler  Successfully assigned kube-system/cilium-pw7bm to cluster-cilium-m03
  Normal   Pulling    2m44s (x3 over 7m20s)  kubelet            Pulling image "quay.io/cilium/cilium:v1.12.1@sha256:ea2db1ee21b88127b5c18a96ad155c25485d0815a667ef77c2b7c7f31cab601b"
  Warning  Failed     45s (x3 over 5m21s)    kubelet            Failed to pull image "quay.io/cilium/cilium:v1.12.1@sha256:ea2db1ee21b88127b5c18a96ad155c25485d0815a667ef77c2b7c7f31cab601b": rpc error: code = Unknown desc = context deadline exceeded
  Warning  Failed     45s (x3 over 5m21s)    kubelet            Error: ErrImagePull
  Normal   BackOff    7s (x5 over 5m21s)     kubelet            Back-off pulling image "quay.io/cilium/cilium:v1.12.1@sha256:ea2db1ee21b88127b5c18a96ad155c25485d0815a667ef77c2b7c7f31cab601b"
  Warning  Failed     7s (x5 over 5m21s)     kubelet            Error: ImagePullBackOff

The kubelet failed to pull the required image for this pod.

The solution

After some investigation I’ve found this post that seemed to be related to the issue above: https://github.com/kubernetes/minikube/issues/14789

The issue is that there is a timeout before the image is completely pulled. The workaround is to pull the image manually with the command below:

enb@DBI-LT-ENB ~ % minikube ssh docker pull "quay.io/cilium/cilium:v1.12.1@sha256:ea2db1ee21b88127b5c18a96ad155c25485d0815a667ef77c2b7c7f31cab601b" -p cluster-cilium

I deleted the 3 cilium pods and they came back with the image properly loaded and reached the Running state. I could then run cilium status in order to validate everything was in order:

enb@DBI-LT-ENB ~ % cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

DaemonSet         cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium             Running: 3
                  cilium-operator    Running: 1
Cluster Pods:     1/1 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.1@sha256:ea2db1ee21b88127b5c18a96ad155c25485d0815a667ef77c2b7c7f31cab601b: 3
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.1@sha256:93d5aaeda37d59e6c4325ff05030d7b48fabde6576478e3fdbfb9bb4a68ec4a1: 1

According to the last comments of the post about this issue less than 2 weeks ago, it should be fixed when minikube is using a different container runtime as containerd but I haven’t tried it yet.

Last tips

I’ve started this post by using 2 clusters in minikube, one for calico and one for cilium. If you want to check minikube status or install some addons, you need to specify on which cluster you want to do it using the -p flag. Each cluster is called a profile in minikube and you can check them as follows:

enb@DBI-LT-ENB ~ % minikube profile list
|----------------|-----------|---------|--------------|------|---------|---------|-------|--------|
|    Profile     | VM Driver | Runtime |      IP      | Port | Version | Status  | Nodes | Active |
|----------------|-----------|---------|--------------|------|---------|---------|-------|--------|
| cluster-calico | docker    | docker  | 192.168.58.2 | 8443 | v1.24.3 | Running |     2 |        |
| cluster-cilium | docker    | docker  | 192.168.49.2 | 8443 | v1.24.3 | Running |     3 |        |
|----------------|-----------|---------|--------------|------|---------|---------|-------|--------|

If you want to switch to your calico cluster, just enter the following command:

enb@DBI-LT-ENB ~ % minikube profile cluster-calico
minikube profile was successfully set to cluster-calico

And use minikube profile cluster-cilium in order to switch back to your cilium cluster.

That’s all folks, I hope this post will help you in your testing with minikube and cilium.

Post Views: 4,539
Thumbnail [60x60]
by
Benoît Entzmann