Gateway API is a relatively new and powerful extension to Kubernetes designed to improve how traffic routing and network policies are managed. It is a set of resources that provides more expressive and extensible tools for controlling network traffic. Unlike the traditional Ingress resource, which focuses primarily on HTTP and HTTPS routing, Gateway API offers a more modular approach, supporting multiple protocols and providing advanced capabilities that address the limitations of Ingress.

The Gateway API introduces concepts like Gateways, Routes, and GatewayClasses, which allow for a clearer separation of concerns, enabling both infrastructure administrators and application developers to work more efficiently. By supporting features such as traffic splitting, path-based routing, and more granular control over traffic policies, Gateway API offers a significant step forward in Kubernetes traffic management.

As organizations increasingly demand more advanced networking features and greater flexibility in their deployments, the Gateway API is positioned as a future-proof solution, providing a standardized and consistent way to manage traffic across a wide range of use cases. Whether you are dealing with complex microservices architectures, multi-tenant environments, or simply need more control over your traffic routing, Gateway API offers the tools to meet these needs effectively.

Ingress and Gateway API comparison

In Kubernetes, managing and directing traffic to applications is a critical aspect of deploying and maintaining services. Two primary methods for achieving this are the Ingress resource and the Gateway API. Both serve the purpose of handling inbound traffic, but they differ significantly in their design, functionality, and intended use cases. Understanding these differences is crucial for choosing the right approach for your specific needs.

Ingress

Ingress is one of the older and more established methods for managing external access to services within a Kubernetes cluster. It defines a set of rules for routing external HTTP and HTTPS traffic to backend services. An Ingress resource is used in conjunction with an Ingress Controller, which implements the routing rules defined in the Ingress resource.

Key features of ingress:

1. Simplicity and ease of use
   • Ingress provides a straightforward way to define routing rules using a single Kubernetes resource.
   • It is suitable for basic to moderately complex routing scenarios.
2. Support for HTTP and HTTPS
   • Ingress is primarily designed to handle HTTP and HTTPS traffic.
   • It allows for features like SSL/TLS termination, name-based virtual hosting, and URL path-based routing.
3. Integration with Ingress Controllers
   • Various Ingress Controllers such as NGINX, Traefik, and HAProxy can be used to implement the routing rules.
   • Each Ingress Controller may offer additional features and customizations, often managed via annotations.
4. Annotations for extended features
   • Ingress relies heavily on annotations to extend its functionality, which can lead to inconsistencies and complexity.
   • Different controllers may interpret annotations differently, affecting portability.
5. Common use cases:
   • Ideal for simple to moderately complex HTTP/HTTPS routing needs.
   • Suitable for scenarios where ease of use and quick setup are prioritized.

Gateway API

Gateway API is a more modern and flexible approach to traffic management in Kubernetes. It addresses some of the limitations of Ingress by providing a more powerful and extensible framework. Gateway API introduces a set of new resources designed to offer greater flexibility and control over traffic routing and management.

Key features of gateway API:

1. Modular and extensible design
   • Gateway API introduces multiple resources such as GatewayClass, Gateway, HTTPRoute, TCPRoute, and TLSRoute.
   • This modular design allows for more granular and precise control over traffic management.
2. Support for multiple protocols
   • In addition to HTTP and HTTPS, Gateway API supports other protocols like TCP and TLS.
   • This makes it suitable for a broader range of use cases beyond simple web traffic.
3. Separation of responsibilities
   • Gateway API separates the concerns of infrastructure management and application development.
   • Administrators can manage Gateways, while developers can define routing rules using Route resources.
4. Advanced features and flexibility
   • Gateway API supports advanced features such as weighted traffic splitting, traffic mirroring, and more complex routing rules.
   • It provides a more consistent and standardized way to define and manage these features.
5. Improved consistency and portability
   • Gateway API aims to reduce the reliance on annotations and provide a more consistent API surface.
   • This enhances portability across different environments and implementations.
6. Common use cases
   • Ideal for complex traffic management scenarios requiring advanced routing capabilities.
   • Suitable for multi-tenant environments and architectures with microservices requiring fine-grained control over traffic routing.

Example: Use Gateway API with Cilium

At dbi services, we follow the best practice of using Cilium as our Kubernetes CNI (Container Network Interface). If you are not familiar with Cilium you can follow these articles from my colleagues:

• Kubernetes Networking by Using Cilium – Beginner Level
• Kubernetes Networking by Using Cilium – Intermediate Level – Network Interfaces
• Kubernetes Networking by Using Cilium – Intermediate Level – Traditional Linux Routing
• Kubernetes Networking by Using Cilium – Advanced Level – eBPF Routing

Cilium not only provides powerful networking and security capabilities but also implements several Gateway API features, such as:

• GatewayClass and Gateway support: Allowing for flexible and scalable traffic management.
• L4 and L7 traffic filtering: Enabling more granular control over traffic routing.
• Native integration with HTTPRoute, GRPCRoute, and TLSRoute: Facilitating advanced protocol support and routing capabilities.

In this example, we will demonstrate how to install Cilium with Gateway API support and set up a simple HTTPRoute to route traffic to a backend service.

Other Gateway API solutions exist, so if you’re familiar with other network tools (Traefik, Istio, Contour, …) or need to implement TCPRoute and/or UDPRoute resources, you’ll find a list of implementations in the official Gateway API documentation.

Prerequisites

According to the Cilium’s documentation, we need to add the necessary CRDs then deploy the Cilium’s Helm chart with the right options:

• Add Gateway API CRDs

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

• Deploy cilium on your Kubernetes with Gateway API feature

# Cilium install using Helm:
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
    --version 1.16.1 \
    --namespace kube-system \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true

# Wait for Cilium to be healthy
cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 1
                       cilium-operator    Running: 1
Cluster Pods:          14/15 managed by Cilium

HTTPRoute example:

• Create YAML definition file for the HTTP server deployment and service (http-server.yaml):

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: testgw-backend
    example: http-routing
  name: testgw-backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testgw-backend
  template:
    metadata:
      labels:
        app: testgw-backend
    spec:
      containers:
      - env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: gcr.io/k8s-staging-ingressconformance/echoserver:v20221109-7ee2f3e
        imagePullPolicy: IfNotPresent
        name: testgw-backend
        resources:
          requests:
            cpu: 10m
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    example: http-routing
  name: testgw-svc
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 3000
  selector:
    app: testgw-backend
  type: ClusterIP

• Create YAML definition file for the Gateway (gateway.yaml):

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cilium-gateway
spec:
  gatewayClassName: cilium
# Depending on your Loadbalancer strategy, precise the IP address to use
  addresses:
  - type: IPAddress
    value: 172.22.14.96
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: web-gw
    port: 80
    protocol: HTTP

• Create YAML definition file for the HTTPRoute (httproute.yaml):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  labels:
    example: http-routing
  name: testgw-route
  namespace: default
spec:
  hostnames:
  - testgw.dbi-services.com
  parentRefs:
  - kind: Gateway
    name: cilium-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: testgw-svc
      port: 8080
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /get

• Apply the resources:

kubectl apply -f http-server.yaml
kubectl apply -f gateway.yaml 
kubectl apply -f httproute.yaml

• Check your Gateway API resources

kubectl get gateway
NAME             CLASS    ADDRESS         PROGRAMMED   AGE
cilium-gateway   cilium   172.22.14.96    True         9m31s

kubectl get httproute
NAME        HOSTNAMES             AGE
testgw-route   ["testgw.dbi-services.com"]   11m

• Test an HTTP request

curl http://testgw.dbi-services.com/get
{
 "path": "/get",
 "host": "testgw.dbi-services.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.9.1"
  ]
 },
 "namespace": "default",
 "ingress": "",
 "service": "",
 "pod": "testgw-backend-67d4f66b9b-jtj9s"
}

Well done ! Your Gateway API HTTPRouting is working ! 🎉

Conclusion

While both Ingress and Gateway API serve to manage external traffic in Kubernetes, they cater to different needs and use cases. Ingress is simpler and quicker to set up, making it ideal for straightforward HTTP/HTTPS routing scenarios. On the other hand, Gateway API offers a more powerful, flexible, and extensible framework suitable for advanced traffic management requirements, including support for multiple protocols and more complex routing configurations. Understanding these differences can help you make an informed decision on which approach to use for your Kubernetes deployments.

L’article Kubernetes Traffic Control: Ingress vs. Gateway API est apparu en premier sur dbi Blog.

If your Kubernetes cluster is multi-tenant this is going to be a requirement. Multi-tenant means the same Kubernetes cluster host several applications. Each of them is in their own namespace. However the product owners are different for each application. In this case you’ll need to isolate each application from each other. It would be very bad if from a pod of your application you could reach and connect to a pod of another application and the other way around. To provide this networking isolation we need to configure and deploy Network Policy. They are often the pet peeve of many Kubernetes Administrator that are not familiar with networking. If it is your case, this blog post is for you. I’ll give you all the basics to understand it completely.

Ingress and Egress

Kubernetes Network Policy is like a police officer in front of your pods. It allows or denies traffic entering or exiting the pod. This is the first concept to grasp, the traffic can flow in 2 directions: entering or exiting. In traditional networking we call it incoming and outgoing (sometimes it is inbound and outbound) traffic. In Kubernetes we are talking about ingress and egress but they are the same thing.

Ingress or egress traffic depends on the point of reference. We basically apply a Network Policy to a pod. Cilium calls it an endpoint but it is the same. For this pod, the direction of traffic is egress when it comes from the pod to the outside. An ingress traffic is for traffic coming from outside and entering into this pod. The drawing below illustrates that concept:

On the left, the police officer looks to the pod so she will stop and filter any traffic coming from that pod before it goes outside. On the right, he looks to the outside with his back to the pod. He will then filter and stop any traffic from outside before it enters that pod.

Labels

As you may know, pods get their IP Address from Cilium when they start. A pod by design can be deleted and recreated which means its IP Address will change. In traditional networking, we filter traffic with firewalls or Access Control List (ACL). This layer 3 filtering is based on the IP Address. In Kubernetes we then have to use another way than IP Addresses and so we use labels instead. We set a label to each pod and if pods use the same label then they belong to the same group. It is the case for the replicas of a front end or a back end pod that will share the same label. We apply a Network Policy to a group of pods based on this label. Let’s look at this on a drawing:

In our example we have 2 applications in 2 separate namespaces. Each application has 2 Front End (FE) and 2 Back End (BE). In the namespace app1, the FE have the same label “app: frontend1″ and the BE have the label “app:backend1″. We apply the same to the namespace app2.

Cilium Network Policy

Kubernetes provides an object called NetworkPolicy that is implemented by the Container Network Interface (CNI). Cilium is using its own object called CiliumNetworkPolicy that extends the capabilities of that standard Network Policy object. You can learn more about it from this Isovalent blog post.

A Network Policy is a yaml file where we configure to which pod labels it applies as well as the ingress and/or egress rules we apply to them. When we apply that yaml file in our cluster, we set the police officer in front of the pods door.

Kubernetes Network Policy ingress rule

Let’s look at an example and start with an ingress rule. We will use the simple CiliumNetworkPolicy below:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "ingress-backend-rule"
  namespace: app1
spec:
  endpointSelector:
    matchLabels:
      app: backend1
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend1

This CiliumNetworkPolicy applies to any pod in the namespace app1 that has the label “app: backend1“. Only one ingress rule is applied here where traffic from all pods with the label “app: frontend1” is allowed. Any other ingress traffic will be discarded. Below is the drawing showing it in a visual way:

When any traffic in this Kubernetes cluster reach the pods BE 1 or BE 2 in the namespace app1, the ingress rules of our Network Policy applies. Only traffic coming from the pods FE 1 or FE 2 in the namespace app1 will be allowed to enter because they have the authorized label.

Kubernetes Network Policy egress rule

Let’s continue with a simple egress rule. We will use the CiliumNetworkPolicy below:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "egress-frontend-rule"
  namespace: app1
spec:
  endpointSelector:
    matchLabels:
      app: frontend1
  egress:
  - toEndpoints:
    - matchLabels:
        app: backend1

This CiliumNetworkPolicy applies to any pod in the namespace app1 that has the label “app: frontend1“. Only one egress rule is applied here where traffic to pods with the label “app: backend1” is allowed. Any other egress traffic from there pods will be discarded. Below is the drawing showing it in a visual way:

When traffic from the pods FE 1 or FE 2 in the namespace app1 is going out, the egress rule of our Network Policy applies. Only traffic to pods BE 1 and BE 2 will be allowed as they have the authorized label.

Wrap up

We’ve covered the basics of Kubernetes Network Policy by explaining what are its key components: Ingress, egress, labels and the CiliumNetworkPolicy object. Then we’ve seen a simple example of ingress and egress rules and illustrated them with a drawing. These basics should get you started in the world of Kubernetes Network Policy.

With the simple ingress and egress rules above did we completely isolate the traffic flow between FE and BE in app1? The answer is no because from BE 1 or BE 2 I can still reach any pods in the cluster. There is only an ingress rule but no egress rule. The same, any traffic can reach FE 1 or FE 2 because there is only an egress rule but no ingress rule. I would then need to configure an ingress rule and an egress rule (the same as above) into the same CiliumNetworkPolicy. Congratulations, you’ve just entered into the complex world of Kubernetes Network Policy!

L’article Kubernetes Network Policy by using Cilium – Beginner Level est apparu en premier sur dbi Blog.

If you still want to know more, then you are in the right place. Fasten your seat belt because in this blog post we will dive deep into the eBPF routing. When using Cilium for networking in your Kubernetes cluster, you will automatically use eBPF for routing between pods. I’ll stick to the same 2 examples of routing between pods on the same node and on different nodes. You’ll then see the continuity on the same drawing that will hopefully help you understand this routing topic completely.

Discovering eBPF

eBPF Routing

In my high level blog and in the previous one about routing, I’ve talked about servants that are waiting at some network interfaces to help and direct you in the Kubernetes cluster. I’ve told you they were using a magical eBPF map to guide you through a secret passage toward your destination. Let’s now see what these servants are exactly in our Kubernetes cluster.

A servant is actually an eBPF program attached to a network interface at the kernel level. That is what eBPF allows you to do, modify the kernel dynamically. The advantage is that you can attach your own program (that is C code transformed to machine code) to some events and network interfaces in the kernel and trigger any action you have programmed. An action could be routing a packet or report and act on any interaction with the kernel. This is why eBPF fits nicely for observability because anything happening in a node interact with the kernel. It could be running a process, opening a file, routing a packet,… Regarding routing, it is very fast because it shortcuts the traditional Linux routing process. That is the reason why we couldn’t see all the routing steps in the previous post by using our traditional networking tools. We now need eBPF tools to inspect this routing in more details.

Cilium Agent and eBPF routing

You know from the previous posts that there is a Cilium agent pod on each node in the cluster. This agent takes care of everything about networking on that node and so also eBPF routing. It comes with some eBPF tools packed with it that will allow us to explore it more deeply.

Let’s now see what these servants really look like in our cluster. As a reminder and to gather all the information here, below are all our Cilium agents in our cluster:

$ kubectl get po -n kube-system -owide|grep cilium
cilium-9zh9s                                      1/1     Running   5 (65m ago)   113d   172.18.0.3    mycluster-control-plane   <none>           <none>
cilium-czffc                                      1/1     Running   5 (65m ago)   113d   172.18.0.4    mycluster-worker2         <none>           <none>
cilium-dprvh                                      1/1     Running   5 (65m ago)   113d   172.18.0.2    mycluster-worker          <none>           <none>
cilium-operator-6b865946df-24ljf                  1/1     Running   5 (65m ago)   113d   172.18.0.2    mycluster-worker          <none>           <none>

As before in this series, we will trace the routing from a pod on the node mycluster-worker and so we will need to interact with the Cilium agent of that node: cilium-dprvh. We use into that agent pod the tool bpftool to list all the network interfaces and the eBPF programs attached to them:

$ kubectl exec -it -n kube-system cilium-dprvh -- bpftool net show
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
xdp:

tc:
cilium_net(2) clsact/ingress cil_to_host-cilium_net id 1209
cilium_host(3) clsact/ingress cil_to_host-cilium_host id 1184
cilium_host(3) clsact/egress cil_from_host-cilium_host id 1197
cilium_vxlan(4) clsact/ingress cil_from_overlay-cilium_vxlan id 1154
cilium_vxlan(4) clsact/egress cil_to_overlay-cilium_vxlan id 1155
lxc_health(6) clsact/ingress cil_from_container-lxc_health id 1295
eth0(7) clsact/ingress cil_from_netdev-eth0 id 1223
lxc4a891387ff1a(9) clsact/ingress cil_from_container-lxc4a891387ff1a id 1285
lxc5b7b34955e61(11) clsact/ingress cil_from_container-lxc5b7b34955e61 id 1303
lxc73d2e1d7cf4(13) clsact/ingress cil_from_container-lxc73d2e1d7cf4 id 1294

Yes these programs are our servants! Armed with the knowledge of the network interfaces in a previous post you should recognize their names. The one that will be used at the starting point of our travel is lxc4a891387ff1a. In parenthesis you have the interface id 9 that is on the node side of this link to this container. The Cilium agent assigned to this interface the id 1285. The name of the program attached to this interface is called cil_from_container. From this name, you know for which direction the program will operate. Here it is for the traffic coming from the container to the node, in the other direction there is no processing. As the Cilium community edition is open source, you can read its code directly here. In the file bpf_lxc.c just search for the program name and you will see all the details. Yes, this is advanced but don’t worry, I did the hard work for you, so you just have to read along!

Before we move on to trace the eBPF routing, let’s update our drawing to show these eBPF servants:

Don’t be confused about the lxc network interface id. For example lxc4a891387ff1a(9) from the command output above. It means it is linked to id 9 which is on the node side, on the interface called lxc….@if8. Id 8 being the id inside of the container (you will see in it 8: eth0@if9).

Pod to pod routing on the same node

I don’t want to lose you with all the details so I’ll give you only the key information to follow along this eBPF routing. Yes, from here it is becoming more complicated but I’ll try to make it smooth for you to enjoy it anyway.

As we did in the previous post, let’s now see how eBPF routes the traffic thanks to its servants from 10.10.2.117 to 10.10.2.121. The Cilium agent gives information to the eBPF program and receives information from them. For exchanging information, it uses what is called maps (yes these are our magic maps!) in the eBPF terminology. There are several of them that are used for different purposes. These maps are stored in the folder /sys/fs/bpf/tc/globals into the Cilium agent pod and we can manually interact with them by using the bpftool. We can then trace the eBPF routing that way. When the packet leaves the container, it reaches the lxc interface on the node and the program cil_from_container is triggered to route that packet. The program sees the destination IP Address is in the same subnet as the source and will then just forward the traffic to the destination lxc network interface by using the map called cilium_lxc.

Below is how we can trace that eBPF routing:

$ kubectl exec -it -n kube-system cilium-dprvh -- bpftool map lookup pinned /sys/fs/bpf/tc/globals/cilium_lxc key hex 0a 0a 02 79 00 00 00 00  00 00 00 00 00 00 00 00 01 00 00 00
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
key:
0a 0a 02 79 00 00 00 00  00 00 00 00 00 00 00 00
01 00 00 00
value:
0b 00 00 00 00 00 33 06  00 00 00 00 00 00 00 00
ea c4 71 d6 4f a0 00 00  f6 87 b6 c3 a6 45 00 00
2c 0d 00 00 00 00 00 00  00 00 00 00 00 00 00 00

We have to give to the bpftool an hexadecimal string that it expects (the details of this format can be found in the source code). The destination IP Address we want to trace is 10.10.2.121 which is 0a 0a 02 79 in hexadecimal. This information is at the beginning of the string in our command.

In the output of this command you can see the key section, which is the hexadecimal string we have provided as well as the result in the value section.

In this result we get the MAC Address of the destination lxc interface on the pod side which is ea c4 71 d6 4f a0. We also get the MAC Address of the destination lxc interface on the node side which is f6 87 b6 c3 a6 45. Finally we get the lxc id of the destination pod which is 33 06. A little trick here is that this value is stored in reverse order (you can do a search about big endian and little endian if you want to know more on this) so we have to read 06 33 which converted to decimal gives 1587. This value is the endpoint ID used by Cilium to identify the pods. You can get this general information in the cluster with the following command:

$ kubectl get ciliumendpoint -n networking101
NAME                        ENDPOINT ID   IDENTITY ID   INGRESS ENFORCEMENT   EGRESS ENFORCEMENT   VISIBILITY POLICY   ENDPOINT STATE   IPV4          IPV6
busybox-c8bbbbb84-fmhwc     897           3372          <status disabled>     <status disabled>    <status disabled>   ready            10.10.1.164
busybox-c8bbbbb84-t6ggh     715           3372          <status disabled>     <status disabled>    <status disabled>   ready            10.10.2.117
netshoot-7d996d7884-fwt8z   1587          10388         <status disabled>     <status disabled>    <status disabled>   ready            10.10.2.121
netshoot-7d996d7884-gcxrm   3564          10388         <status disabled>     <status disabled>    <status disabled>   ready            10.10.1.155

So the destination pod is netshoot-7d996d7884-fwt8z and it is reached directly through its lxc interface. And that’s it! Of course the routing occurs much faster than me explaining it but more importantly it is faster than the traditional Linux routing.

Pod to pod routing on a different node

Let’s now check the eBPF routing from 10.10.2.117 to 10.10.1.155 which is then on a different node. The packet is also intercepted by the program cil_from_container but this time the destination IP Address belongs to another subnet so it is going to use another map called cilium_ipcache to route that packet.

As before, we have to provide an hexadecimal string in the expected format to trace the routing of this packet. Here, our destination IP Address is 10.10.1.155 so 0a 0a 01 9b in hexadecimal.

$ kubectl exec -it -n kube-system cilium-dprvh -- bpftool map lookup pinned /sys/fs/bpf/tc/globals/cilium_ipcache key hex 40 00 00 00 00 00 00 01 0a 0a 01 9b 00 00 00 00 00 00 00 00 00 00 00 00
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
key:
40 00 00 00 00 00 00 01  0a 0a 01 9b 00 00 00 00
00 00 00 00 00 00 00 00
value:
94 28 00 00 ac 12 00 04  00 00 00 00

In the value section we can see the destination IP Address of the node mycluster-worker2 which is ac 12 00 04 (172.18.0.4). From there, the packet is encapsulated and goes straight to the egress VXLAN interface to reach that node. Once there, it is now the eBPF programs of the Cilium agent pod on this node that will be used. Let’s check them as we did before for the other node:

$ kubectl exec -it -n kube-system cilium-czffc -- bpftool net show
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
xdp:

tc:
cilium_net(2) clsact/ingress cil_to_host-cilium_net id 1254
cilium_host(3) clsact/ingress cil_to_host-cilium_host id 1242
cilium_host(3) clsact/egress cil_from_host-cilium_host id 1246
cilium_vxlan(4) clsact/ingress cil_from_overlay-cilium_vxlan id 1163
cilium_vxlan(4) clsact/egress cil_to_overlay-cilium_vxlan id 1164
lxc_health(6) clsact/ingress cil_from_container-lxc_health id 1414
lxc0a97661d8043(8) clsact/ingress cil_from_container-lxc0a97661d8043 id 1387
eth0(9) clsact/ingress cil_from_netdev-eth0 id 1261
lxc174c023046ff(11) clsact/ingress cil_from_container-lxc174c023046ff id 1391
lxce84a702bb02c(13) clsact/ingress cil_from_container-lxce84a702bb02c id 1419

The output is similar to the other node. So, once the packet exits the VXLAN tunnel, it is caught by another program on the ingress VXLAN interface called cil_from_overlay-cilium_vxlan. This program sees the destination IP Address belongs to this node. It will then use the map cilium_lxc to forward the traffic to the lxc interface of the destination pod as we have seen before. Note that there is no programs in the list above called to_container so the packet in not processed further. We can then trace that eBPF routing part as before by using the hexadecimal value of our destination IP Address 10.10.1.155:

$ kubectl exec -it -n kube-system cilium-dprvh -- bpftool map lookup pinned /sys/fs/bpf/tc/globals/cilium_lxc key hex 0a 0a 01 9b 00 00 00 00  00 00 00 00 00 00 00 00 01 00 00 00
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
key:
0a 0a 01 9b 00 00 00 00  00 00 00 00 00 00 00 00
01 00 00 00
value:
0b 00 00 00 00 00 ec 0d  00 00 00 00 00 00 00 00
be 57 3d 54 40 f1 00 00  92 65 df 09 dd 28 00 00
2c 0d 00 00 00 00 00 00  00 00 00 00 00 00 00 00

The result gives us the following information:

• 92 65 df 09 dd 28: This is the MAC Address of the destination lxc interface on the node side
• be 57 3d 54 40 f1: This is the MAC Address of the destination lxc interface on the pod side
• ec 0d: We reverse it to 0d ec which we convert to 3564. This is the Cilium endpoint of the destination pod netshoot-7d996d7884-gcxrm as we have seen previously (you can check the output of the command above and find that pod).

As before, from here the packet is directly forwarded to this destination pod. This is how an eBPF routing plan comes up together!

Wrap up

Wow! Congratulations if you’ve reached that point! You are champions! You now have the complete and detailed picture of how the basic networking between pods is working with Cilium in a Kubernetes cluster. There is more to cover as I’ve mentioned network policies in a previous post but we didn’t talk about services, ingress or name resolution. Also in addition to these basic Kubernetes networking topics, Cilium provides a lot of other features that enrich what can be done in a cluster. So stay tuned!

L’article Kubernetes Networking by Using Cilium – Advanced Level – eBPF Routing est apparu en premier sur dbi Blog.

Below is the drawing of where we left off:

We will continue to use the same method, you are the packet that will travel from your apartment (pod) 10.10.2.117 on the top left to other pods in this Kubernetes cluster. But first, let’s take this opportunity to talk about namespace and enrich our drawing with a new analogy.

Routing between namespaces

A namespace is a logical group of objects that provide isolation in the cluster. However, by default, all pods can communicate together in a “vanilla” Kubernetes. Whatever they belong to the same namespace or not. So this isolation provided by namespace doesn’t mean the pods can’t communicate together. To allow or deny such communication, you will need to create network policies. That could be the topic for another blog post!

We can use the analogy of a namespace being the same floor number of all building of our cluster. All apartments on the same floor in each building will be logically grouped into the same namespace. This is what we can see below in our namespace called networking101:

$ kubectl get po -n networking101 -owide
NAME                        READY   STATUS    RESTARTS       AGE    IP            NODE                NOMINATED NODE   READINESS GATES
busybox-c8bbbbb84-fmhwc     1/1     Running   1 (125m ago)   4d1h   10.10.1.164   mycluster-worker2   <none>           <none>
busybox-c8bbbbb84-t6ggh     1/1     Running   1 (125m ago)   4d1h   10.10.2.117   mycluster-worker    <none>           <none>
netshoot-7d996d7884-fwt8z   1/1     Running   0              103m   10.10.2.121   mycluster-worker    <none>           <none>
netshoot-7d996d7884-gcxrm   1/1     Running   0              103m   10.10.1.155   mycluster-worker2   <none>           <none>

That’s our 4 apartments / pods on the same floor, grouped together in one namespace:

The routing process doesn’t care about the pod’s namespace, only its destination IP Address will be used. Let’s now see how we can go from the apartment 10.10.2.117 to the apartment 10.10.2.121 in the same building (node).

Pod to pod routing on the same node

From the pod 10.10.2.117, you’ve then decided to go to pay a visit to 10.10.2.121. You first look at the routing table in order to know how to reach this destination. But you can’t go out if you don’t also have the MAC Address of your destination. You need both destination information (IP Address and MAC Address) before you can start to travel. You then look at the ARP table to find out this information. The ARP table contains the known mapping of a MAC Address to an IP Address in your IP subnet. If it is not there, you send first a scout to knock at the door of each apartment in your community until you find the MAC Address of your destination. This is called the ARP request. When the scout comes back with that information, you write it into the ARP table. You thank the scout for his help and are now ready to start your travel by exiting the pod.

Let’s see how we can trace this in our source pod 10.10.2.117

$ kubectl exec -it -n networking101 busybox-c8bbbbb84-t6ggh -- ip route
default via 10.10.2.205 dev eth0
10.10.2.205 dev eth0 scope link

Very simple routing instruction! For every destination, you go through 10.10.2.205 by using your only network interface eth0 in the pod. You can see from the drawing above that 10.10.2.205 is the IP Address of the cilium_host. You then check your ARP table:

$ kubectl exec -it -n networking101 busybox-c8bbbbb84-t6ggh -- arp -a

The arp -a command list the content of the ARP table and we can see there is nothing in there.

A way to send a scout out is by using the arping tool toward our destination. You may have noticed that for my pods I’m using busybox and netshoot images. Both provide networking tools that are useful for troubleshooting:

$ kubectl exec -it -n networking101 busybox-c8bbbbb84-t6ggh -- arping 10.10.2.121
ARPING 10.10.2.121 from 10.10.2.117 eth0
Unicast reply from 10.10.2.121 [d6:21:74:eb:67:6b] 0.028ms
Unicast reply from 10.10.2.121 [d6:21:74:eb:67:6b] 0.092ms
Unicast reply from 10.10.2.121 [d6:21:74:eb:67:6b] 0.123ms
^CSent 3 probe(s) (1 broadcast(s))
Received 3 response(s) (0 request(s), 0 broadcast(s))

We now have the piece of information that was missing, the MAC address of our destination. We can then just check it is written into our ARP table of our source pod:

$ kubectl exec -it -n networking101 busybox-c8bbbbb84-t6ggh -- arp -a
? (10.10.2.205) at d6:21:74:eb:67:6b [ether]  on eth0

Here it is! However you may wonder why we don’t see here the IP Address of our destination 10.10.2.121 right? In traditional networking this is what you will see but here we are in a Kubernetes cluster and we are using Cilium that is taking care of the networking in it. Also we have seen above from the routing table of the source pod that for every destination we go to this cilium_host interface.

So the cilium_host on that node is attracting all the traffic even for communication between pods in the same IP subnet.

As a side note, below is a command where you can quickly display all the IP Addresses of the cilium_host and the nodes in your cluster in one shot:

$ kubectl get ciliumnodes
NAME                      CILIUMINTERNALIP   INTERNALIP   AGE
mycluster-control-plane   10.10.0.54         172.18.0.3   122d
mycluster-worker          10.10.2.205        172.18.0.2   122d
mycluster-worker2         10.10.1.55        172.18.0.4   122d

In traditional networking, doing L2 switching, the MAC Address of the destination is the one related to the destination IP Address. That is not the case here in Kubernetes networking. So which interface has the MAC Address d6:21:74:eb:67:6b ? Let’s respond to that question immediately:

$ sudo docker exec -it mycluster-worker ip a | grep -iB1 d6:21:74:eb:67:6b
9: lxc4a891387ff1a@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:21:74:eb:67:6b brd ff:ff:ff:ff:ff:ff link-netns cni-67a5da05-a221-ade5-08dc-64808339ad05

That is the LXC interface of the node as it is indeed our next step from the source pod to reach our destination. You’ve learned from my first post blog of this networking series that there is a servant waiting here at the LXC interface to direct us toward our destination.

From there, we don’t see much of the travel to the destination from the traditional Linux routing point of view. This is because the routing is done by the Cilium agent using eBPF. As the destination is in the same IP subnet as the source, the Cilium agent just switch it directly to the destination LXC interface and then reach the destination pod.

When the destination pod responds to the source, the same process occurs and for the sake of completeness let’s look at the routing table and ARP table in the destination pod:

$ kubectl exec -it -n networking101 netshoot-7d996d7884-fwt8z -- ip route
default via 10.10.2.205 dev eth0 mtu 1450
10.10.2.205 dev eth0 scope link

$ kubectl exec -it -n networking101 netshoot-7d996d7884-fwt8z -- arp -a
? (10.10.2.205) at 92:65:df:09:dd:28 [ether]  on eth0

$ sudo docker exec -it mycluster-worker ip a | grep -iB1 92:65:df:09:dd:28
13: lxce84a702bb02c@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:65:df:09:dd:28 brd ff:ff:ff:ff:ff:ff link-netns cni-d259ef79-a81c-eba6-1255-6e46b8d1c779

So from the traditional Linux routing point of view, everything goes to the cilium_host and the destination MAC address is the LXC interface of the node that is linked to our pod. This is exactly the same we have seen with our source pod.

Pod to pod routing on a different node

Let’s now have a look at how we could reach the pod 10.10.1.155 from the source pod 10.10.2.117 which is hosted in another node. The routing is the same at the beginning but when talking to the servant at the LXC interface, he sees that the destination IP Address doesn’t belong to the same IP subnet and so directs us to the cilium_host in the Lobby. From there we are routed to the cilium_vxlan interface to reach the node that host our destination pod.

Let’s now have a look at the routing table of the host:

$ sudo docker exec -it mycluster-worker ip route
default via 172.18.0.1 dev eth0
10.10.0.0/24 via 10.10.2.205 dev cilium_host proto kernel src 10.10.2.205 mtu 1450
10.10.1.0/24 via 10.10.2.205 dev cilium_host proto kernel src 10.10.2.205 mtu 1450
10.10.2.0/24 via 10.10.2.205 dev cilium_host proto kernel src 10.10.2.205
10.10.2.205 dev cilium_host proto kernel scope link
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2

We don’t see much here as the routing is using eBPF and is managed by the Cilium agent as we’ve seen before.

As a side note and to share everything with you, the output of the network interfaces as well as the ip route in the Cilium agent pod is identical to the one of the node. This is because at startup the Cilium agent provides these information to the node. You can check the Cilium agent with the following commands:

$ kubectl exec -it -n kube-system cilium-dprvh -- ip a
$ kubectl exec -it -n kube-system cilium-dprvh -- ip route

So you go through the VXLAN tunnel and you reach the node mycluster-worker2. Here is the routing table of this node:

$ sudo docker exec -it mycluster-worker2 ip route
default via 172.18.0.1 dev eth0
10.10.0.0/24 via 10.10.1.55 dev cilium_host proto kernel src 10.10.1.55 mtu 1450
10.10.1.0/24 via 10.10.1.55 dev cilium_host proto kernel src 10.10.1.55
10.10.1.55 dev cilium_host proto kernel scope link
10.10.2.0/24 via 10.10.1.55 dev cilium_host proto kernel src 10.10.1.55 mtu 1450
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.4

Again from the traditional Linux routing point of view there isn’t much to see, except that all the traffic for the pods subnet are going to the cilium_host that is managed by the Cilium agent. This is identical as what we’ve learned in the other node. When we reach the cilium_vxlan interface, a servant is waiting for us with his magical eBPF map and directs us through a secret passage to the LXC corridor interface of the top left pod where we can reach our destination.

Wrap up

We’ve explored all that can be seen in routing from the traditional Linux point of view by using the common networking tools.

Maybe you feel frustrated to not understand it completely because there are some gaps in this step-by-step packet routing? Cilium uses eBPF for routing the packets so it adds some complexity to the routing understanding. However it is much faster than the traditional Linux routing due to the secret passages opened by the eBPF servants.

If you want to know more about this, don’t miss my next blog post where I’ll dive deep into the meanders of eBPF routing. See you there!

L’article Kubernetes Networking by Using Cilium – Intermediate Level – Traditional Linux Routing est apparu en premier sur dbi Blog.

If you want to understand this networking in Kubernetes in more details, read on, this blog post is for you! I’ll consider you know the basics about Kubernetes and how to interact with it, otherwise you may find our training course on it very interesting (in English or in French)!

Diving into the IP Addresses configuration

Let’s start by checking our environment and update our picture with real information from our Kubernetes cluster:

$ kubectl get no -owide
NAME                      STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION      CONTAINER-RUNTIME
mycluster-control-plane   Ready    control-plane   113d   v1.27.3   172.18.0.3    <none>        Debian GNU/Linux 11 (bullseye)   5.15.0-94-generic   containerd://1.7.1
mycluster-worker          Ready    <none>          113d   v1.27.3   172.18.0.2    <none>        Debian GNU/Linux 11 (bullseye)   5.15.0-94-generic   containerd://1.7.1
mycluster-worker2         Ready    <none>          113d   v1.27.3   172.18.0.4    <none>        Debian GNU/Linux 11 (bullseye)   5.15.0-94-generic   containerd://1.7.1

$ kubectl get po -n networking101 -owide
NAME                        READY   STATUS    RESTARTS      AGE     IP            NODE                NOMINATED NODE   READINESS GATES
busybox-c8bbbbb84-fmhwc     1/1     Running   1 (24m ago)   3d23h   10.10.1.164   mycluster-worker2   <none>           <none>
busybox-c8bbbbb84-t6ggh     1/1     Running   1 (24m ago)   3d23h   10.10.2.117   mycluster-worker    <none>           <none>
netshoot-7d996d7884-fwt8z   1/1     Running   0             79s     10.10.2.121   mycluster-worker    <none>           <none>
netshoot-7d996d7884-gcxrm   1/1     Running   0             80s     10.10.1.155   mycluster-worker2   <none>           <none>

You can now see for real that the IP subnets of the pods are different than the one of the nodes. Also the IP subnet of pods on each node is different from each other. If you are not sure why, you are perfectly right because it is not so clear at this stage. So let’s clarify it by checking our Cilium configuration.

I’ve told you in my previous blog that there is one Cilium Agent per building. This Agent is a pod itself and he takes care about networking in the node. This is what they look like in our cluster:

$ kubectl get po -n kube-system -owide|grep cilium
cilium-9zh9s                                      1/1     Running   5 (65m ago)   113d   172.18.0.3    mycluster-control-plane   <none>           <none>
cilium-czffc                                      1/1     Running   5 (65m ago)   113d   172.18.0.4    mycluster-worker2         <none>           <none>
cilium-dprvh                                      1/1     Running   5 (65m ago)   113d   172.18.0.2    mycluster-worker          <none>           <none>
cilium-operator-6b865946df-24ljf                  1/1     Running   5 (65m ago)   113d   172.18.0.2    mycluster-worker          <none>           <none>

There is two things to notice here:

• The Cilium Agent is a Daemonset so that is how you make sure to always have one on each node of our cluster. As it is a pod, it also gets an IP Address… but wait a minute… this is the same IP Address as the node! Exactly! This is a special case for pods IP Address assignation, usually for system pods that need direct access to the node (host) network. If you look at the pods in the kube-system namespace, you’ll see most of them uses the node IP Address.
• The Cilium Operator pod is responsible for IP address management in the cluster and so it gives to each Cilium Agent its range to use.

Now you want to see which IP range is used by each node right? Let’s just check that Cilium Agent on each node as we have found their name above:

$ kubectl exec -it -n kube-system cilium-dprvh -- cilium debuginfo | grep IPAM
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
IPAM:                   IPv4: 5/254 allocated from 10.10.2.0/24,

$ kubectl exec -it -n kube-system cilium-czffc -- cilium debuginfo | grep IPAM
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
IPAM:                   IPv4: 5/254 allocated from 10.10.1.0/24,

You can now see the different IP subnet on each node. In my previous blog I told you that an IP Address belong to a group and it uses the subnet mask. This subnet mask is here /24 which means for the first node that any address starting with 10.10.2 belongs to the same group. For the second node it is 10.10.1 and so they are both in a separate group or IP subnet.

What now about checking the interfaces that are the doors of our drawing?

Diving into the interfaces configuration

Let’s explore our buildings and see what we could find out! We are going to start with our four pods:

$ kubectl get po -n networking101 -owide
NAME                        READY   STATUS    RESTARTS       AGE    IP            NODE                NOMINATED NODE   READINESS GATES
busybox-c8bbbbb84-fmhwc     1/1     Running   1 (125m ago)   4d1h   10.10.1.164   mycluster-worker2   <none>           <none>
busybox-c8bbbbb84-t6ggh     1/1     Running   1 (125m ago)   4d1h   10.10.2.117   mycluster-worker    <none>           <none>
netshoot-7d996d7884-fwt8z   1/1     Running   0              103m   10.10.2.121   mycluster-worker    <none>           <none>
netshoot-7d996d7884-gcxrm   1/1     Running   0              103m   10.10.1.155   mycluster-worker2   <none>           <none>

$ kubectl exec -it -n networking101 busybox-c8bbbbb84-t6ggh -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue qlen 1000
    link/ether 9e:80:70:0d:d9:37 brd ff:ff:ff:ff:ff:ff
    inet 10.10.2.117/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9c80:70ff:fe0d:d937/64 scope link
       valid_lft forever preferred_lft forever

$ kubectl exec -it -n networking101 netshoot-7d996d7884-fwt8z -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ea:c4:71:d6:4f:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.2.121/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::e8c4:71ff:fed6:4fa0/64 scope link
       valid_lft forever preferred_lft forever

$ kubectl exec -it -n networking101 netshoot-7d996d7884-gcxrm -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether be:57:3d:54:40:f1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.1.155/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::bc57:3dff:fe54:40f1/64 scope link
       valid_lft forever preferred_lft forever

$ kubectl exec -it -n networking101 busybox-c8bbbbb84-fmhwc -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue qlen 1000
    link/ether 2a:7f:05:a0:69:db brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.164/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::287f:5ff:fea0:69db/64 scope link
       valid_lft forever preferred_lft forever

You can see that each container has only one network interface in addition to its local loopback. The format is for example 8: eth0@if9 which means the interface in the container has the number 9 and is linked to its pair interface number 8 of the node it is hosted on. These are the 2 doors connected by a corridor in my drawing.

Then check the nodes network interfaces:

$ sudo docker exec -it mycluster-worker ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5e:84:64:22:90:7f brd ff:ff:ff:ff:ff:ff
3: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ca:7e:e1:cc:4e:74 brd ff:ff:ff:ff:ff:ff
    inet 10.10.2.205/32 scope global cilium_host
       valid_lft forever preferred_lft forever
4: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether f6:bf:81:9b:e2:c5 brd ff:ff:ff:ff:ff:ff
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::2/64 scope global nodad
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:2/64 scope link
       valid_lft forever preferred_lft forever
7: lxc_health@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8a:de:c1:2c:f5:83 brd ff:ff:ff:ff:ff:ff link-netnsid 1
9: lxc4a891387ff1a@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:21:74:eb:67:6b brd ff:ff:ff:ff:ff:ff link-netns cni-67a5da05-a221-ade5-08dc-64808339ad05
11: lxc5b7b34955e61@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:80:da:a5:17:74 brd ff:ff:ff:ff:ff:ff link-netns cni-0b438679-e5d3-d429-85c0-b6e3c8914250
13: lxc73d2e1d7cf4f@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:87:b6:c3:a6:45 brd ff:ff:ff:ff:ff:ff link-netns cni-f608f13c-1869-6134-3d6b-a0f76fd6d483

$ sudo docker exec -it mycluster-worker2 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:91:2b:31:1f:47 brd ff:ff:ff:ff:ff:ff
3: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether be:7f:e0:2b:d6:b1 brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.55/32 scope global cilium_host
       valid_lft forever preferred_lft forever
4: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether e6:c8:8d:5d:1e:2d brd ff:ff:ff:ff:ff:ff
6: lxc_health@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d2:cf:ec:4c:51:b6 brd ff:ff:ff:ff:ff:ff link-netnsid 1
8: lxcdc5fb9751595@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:b4:3a:e0:67:a3 brd ff:ff:ff:ff:ff:ff link-netns cni-c0d4bea2-92fd-03fb-ba61-3656864d8bd7
9: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.4/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:4/64 scope link
       valid_lft forever preferred_lft forever
11: lxc174c023046ff@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:7a:b9:6b:b3:c1 brd ff:ff:ff:ff:ff:ff link-netns cni-4172177b-df75-61a8-884c-f9d556165df2
13: lxce84a702bb02c@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:65:df:09:dd:28 brd ff:ff:ff:ff:ff:ff link-netns cni-d259ef79-a81c-eba6-1255-6e46b8d1c779

On each node there are several interfaces to notice. I’ll take the first node for example:

• eth0@if6: As our Kubernetes cluster is created with Kind, a node is actually a container (and this interface open a corridor to its pair interface on my laptop). If it feels like the movie Inception, well, it is a perfectly correct comparison! This interface is the main door of the building.
• lxc4a891387ff1a@if8: This is the pair interface number 8 that is linked to the left container above.
• lxc73d2e1d7cf4f@if12: This is the pair interface number 12 that is linked to the right container above.
• cilium_host@cilium_net: This is the circle interface in my drawing that allows the routing to/from other nodes in our cluster.
• cilium_vxlan: This is the rectangle in my drawing and is the tunnel interface that will transport you to/from the other nodes in our cluster.

Let’s now get the complete picture by updating our drawing with these information:

Wrap up

With this foundation knowledge, you now have all the key elements to understand the communication between pods on the same node or on different nodes. This is what we will look at in my next blog post. Stay tuned!

L’article Kubernetes Networking by Using Cilium – Intermediate Level – Network Interfaces est apparu en premier sur dbi Blog.

The best way to learn networking is by doing what is called “Follow the packet” or “The life of a packet”. Basically, you follow the packet from the sender to the receiver by stopping at each step of this journey. I did that a while ago with a communication from a Pod to another Pod by using Calico. This time I’ll use another Container Network Interface (CNI) called Cilium which is based on eBPF (understand fast and flexible routing) and comes with a lot of powerful features and tools. Let’s hit that packet road!

Traditional Networking without Kubernetes

We are going to start from the beginning. I’ll consider you know nothing about networking. Maybe you already know what an IP Address is? The IP Address is the numerical Address of a network interface in a computer. This is how your computer can connect for example to your Wi-Fi network and give you access to the Internet. If you are using a laptop, your Wi-Fi network interface has one IP Address. This network interface also has another address that is unique and burnt on it by the hardware provider. This address is called the Medium Access Control (MAC) Address.

The IP Address belongs to a group (the IP Subnet). To know to which group it belongs to, it uses something called the Subnet Mask. This mask when applied to the IP Address gives a result and this result is identical for every IP Address that belong to the same group. This is like your community.

Let’s use the drawing below to make analogies:

The house is a computer, a server or a virtual machine. It could be of different sizes according to its CPU and memory but let use the same one for simplicity sake. A house has a door that is your network interface. The serial number of that door is your MAC Address and the number on your house (which is usually nailed on the door) is your IP Address. It is only if you change your door that your serial number will change. However, your house number has been assigned to you by the architect of your community and could change if there is a reassignment or a change in the design.

The community 10th (using numbers from 10 to 19) in blue belong to the same group (same IP subnet) and the community 20th in green is another group. In each group there are five houses so there is space for the community to grow. In each community the door has a direct link to a fountain which represents a Switch. At the fountain, there is a sign for each path to indicate which door you can reach. Yes, the fountain doesn’t know the house number but only the serial number of the door. For humans it is not very practical so we use a map (called the ARP table) that gives the translation between the house number and the serial number of its door.

If you live in house 14 and want to visit house 15, you’ll use the path (there is only one and it is yours so no traffic jams!) to the fountain first and then look at the signs. You know from your map which serial number correspond to which house and so you use the path to house 15. In this star topology you always go to the fountain first and not directly to the house you want to visit because there is no direct path. A path inside your community represents a Layer 2 link. You can’t reach another community by using these paths.

Traveling between communities

What now if from your house 14, you want to pay a visit to house 24? This is another community which means the couple IP Address / Subnet mask for 14 doesn’t produce the same result as for 24. Indeed the 10th and the 20th communities are different. So you know the destination is another community and in that case you have to go first to your gatekeeper (but always through the fountain as we have seen). He is the default gateway of your community and he lives in house 11. The rule is to go to him for any destinations that is outside of your community.

Only he has a map (the Routing Table) to reach community 20th and know which road to take (This is called Layer 3 routing because you are traveling outside of your community). This map shows that to reach 24 in the 20th community you have to use another door. Wait a minute, if a door is a network interface, does it mean that the gatekeeper house has another door? Absolutely correct! The house 11 has another door with another number on it (101) and of course this door has another serial number (MAC Address).

By exiting this door you can now take the path to reach community 20th which has its own gatekeeper in house 21. The map (Routing Table) of this gatekeeper directs you to the proper door to reach your destination. This door gives you access to community 20th as your destination 24 belongs to it. The gatekeeper also give you the map (ARP table) so you can orientate yourself at the fountain. You can now walk on the path to the green fountain. From there, you just have to follow the sign and the path to house 24. When you want to get back home, you travel back using the same path in the opposite direction.

Networking in Kubernetes

Now you know the basics about networking, let’s see how it works in comparison in Kubernetes. Yes it is slightly more complicated but let’s go step by step and use the picture below to understand it better:

Instead of houses, we now have buildings. The networking between the buildings is still the same as the traditional one with a Switch/fountain in the middle. The entrance of the building has a door with the building number on it (its IP Address) that is part of a 1000th community. One building will represent one node of our Kubernetes cluster.

You know Kubernetes is a container orchestrator. A container is wrapped into a pod. For simplicity sake, let’s consider a pod has only one container and so both terms are here equivalent. This pod will be one appartement in our building and so is like a private part of it. The apartment could be of different size in the building as it could have 2, 3 or 4 bedrooms for example. This will be the CPU and memory capacities your container will need on the node. Some appartement are empty so the building still has some capacity. However in Kubernetes, pods are created and deleted as needed. So in our building that would mean sometimes a 2 bedrooms appartement is created and when not used anymore, it could be removed from the building. Then a 5 bedrooms appartement could be created next if the building has enough space for it. Imagine then that it is a LEGO building and inside it you can build and demolish apartments of various size according to your need! Isn’t it awesome?

In each building, the containers/pods have their own community (IP subnet). The function of a CNI in Kubernetes is basically to assign numbers (IP Addresses) to pods so they can communicate together. By default Cilium uses different community for each building. When an apartment is created, Cilium assigns a number to it. When an apartment is deleted an recreated, it will get another number so it is ephemeral. Here the blue community uses the 10th range and the green one uses the 20th. You can notice that the number ranges of the blue and green communities are different than the range of the buildings. Just for you to know, this design is called an Overlay network. There are other ones possible but that is a commonly used. It is a network of pods on top of the network of nodes.

Traveling between apartments in the same building

Now, you live in the apartment 12, how are you going to pay a visit to apartment 14? Like we did with the traditional networking example, you are the packet we are going to follow! Of course you leave the apartment (container) through its door (its network interface). The difference with our previous example is that you are now not out of the house, you are just out of your apartment but still inside of the building. You then follow a private corridor and reach another door (this is the LXC interface).

This door gives you access to a common space of the building where the routing and dispatching occurs. We call it the Cilium Lobby (the blue rectangle). This Lobby has been installed by the Cilium Agent of this building when Cilium was selected to provide communications in this Kubernetes cluster. There is one Cilium Agent per building and he is taking care of everything related to networking. In this Lobby there is a gatekeeper that doesn’t live in an apartment but waits at a deck in the Lobby. He has a team of servants that wait at different doors in the building to provide directions. This is because Cilium is using a magic routing map called eBPF that efficiently helps the travellers.

So when you reach the door at the end of the corridor, you show the servant who is waiting here you want to go to number 14. He finds a match in his magic eBPF map with number 14 and shows you directly the corridor door on the top right. You don’t have to go to the Lobby, he shows you a secret passage to go straight to it. You then open that door, follow the corridor and reach apartment number 14. You would go back to apartment number 12 following the same path and process but in the opposite direction.

So this dispatching is different from traditional switching and it is very fast thanks to the magic eBPF map!

Traveling between apartments in different buildings

Now from apartment 12 you want to pay a visit to the apartment number 22 which is in another building. The beginning of your travel is the same as previously, you exit your apartment, follow the corridor and ask the servant waiting here for directions. As the destination number 22 belongs to another community, he directs you to the Lobby this time. Here and as with traditional networking, you need the help of the gatekeeper in the Lobby. The gatekeeper looks at his map (the Routing table) for direction to number 22 and shows you the door number 11 to use (the cilium_host).

When you open that door, you see another door behind it: It is the blue triangle and it is called the VXLAN interface. This door open to a nice transparent tunnel that goes through the main door of the building. You are protected from the rain and can enjoy the landscape while walking to the other building. You even spot the outdoor fountain! When you reach the green building you exit the tunnel and go meet the servant that is waiting for you at the green triangle (VXLAN interface). You give him your destination, he finds a match in his magic eBPF map with number 22 and shows you a secret passage to the corridor door on the top left. From there you follow the corridor and reach your destination. As before, your way back will follow the same journey in the opposite direction.

This is Layer 3 routing as the destination community is different than yours. You can see it is a bit more complex in Kubernetes than in traditional routing.

Wrap up

I hope this helped you understand the difference between the traditional networking and Kubernetes networking and also that the latter is clearer now for you. If that is all you needed, then I’m glad you’ve read this blog post, I hope you’ve enjoyed it. If you now want to learn more about Kubernetes Networking, stay tuned because I’ll write an intermediate level where you’ll see what a building really looks like on a real cluster!

L’article Kubernetes Networking by Using Cilium – Beginner Level est apparu en premier sur dbi Blog.

However, over the past year there has been several very interesting new features being released. Smartly, Isovalent gives you the opportunity to play with each new feature in their excellent labs they have now put together in this original map quest. You follow the instructions to learn how the feature works and earn a badge (for the main features) if you solve the final challenge. Be careful it is very addictive, as soon as you earn your first badge you will then want to collect all of them!

Simplifying a Kubernetes cluster architecture

Cilium just became a Graduated CNCF project and I wanted to try to combine several of its features together to simplify our traditional Kubernetes cluster architecture for future projects.

If you are using a Kubernetes on-premise cluster that hosts applications reachable from outside of this cluster then you are using an Ingress Controller and a load balancer (probably MetalLB using Layer 2 configuration). This is the traditional architecture we deploy and with Cilium it is now possible to replace these two components by using a combination of the following features: Ingress Controller, LoadBalancer IPAM and L2 Service Announcement.

I’ve done my tests with Minikube as at this stage I just wanted to see how the combining configuration would work. I didn’t see yet some examples of those 3 features together so if you are also interested to optimize the architecture of your Kubernetes cluster with Cilium then read on!

Overview of the configuration

If you are like me, you like to have some visuals about what we are doing. So I start by giving the summary of my configuration that combine the 3 Cilium features (with their name in bold in the diagram below). The colored rectangle boxes show the important parameters of the objects and when they are of the same color, it is how the link is made between the objects.

I’ve used the application “testcontainers/helloworld” that I want to expose outside of my cluster. So from my laptop I should be able to reach it and this is my goal for this first stage of testing.

Each feature brings its customized object (CRD) and you can see in the diagram above the configuration of the object for each of the 3 features. The service object with the type LoadBalancer is automatically created when the feature Ingress Controller is enabled. We then just associate a deployment for our “testcontainers/helloworld” application to this service by using the selector/labels association.

Minikube configuration

Let’s start from the beginning, here is my Minikube configuration for testing these features. I’ve started a new Minikube cluster with 3 nodes, no cni and I like to create a separate profile called here cluster-cilium to separate that cluster from the other cluster tests I’m doing.

$ minikube start --nodes 3 --network-plugin=cni --cni=false --memory=4096 -p cluster-cilium

When my cluster is up and running, I install Cilium 1.14.2 and activate all the features and parameters I need for my testing. I also like to install Cilium in a separate namespace called here cilium:

$ kubectl create ns cilium
$ cilium install --version 1.14.2 --set kubeProxyReplacement=strict --set ingressController.enabled=true --set ingressController.loadbalancerMode=dedicated --set kubeProxyReplacement=true --set l2announcements.enabled=true --set l2announcements.leaseDuration="3s" --set l2announcements.leaseRenewDeadline="1s" --set l2announcements.leaseRetryPeriod="500ms" --set l2podAnnouncements.enabled=true --set l2podAnnouncements.interface="eth0" --set externalIPs.enabled=true -n cilium

Combined features configuration

When my cluster is ready, I can check the LoadBalancer service is automatically created with an http and https nodePort. For this example I’ve edited this service and removed the https port to avoid any confusion. The http nodePort 31814 was automatically created but that value could be changed. For my testing I’ll not use it:

$ kubectl get svc -n cilium
NAME             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.107.176.240   <pending>      80:31814/TCP   16h

You can see that this service doesn’t have any external IP address yet (the status shows pending). To provide one, we need to create an object CiliumLoadBalancerIPPool and define a cidr range of IP to use as external IP Address. Note that /30 (so 2 IP Addresses) is the minimum range you can use. It is not possible to use just 1 IP Address in a pool. We then set a serviceSelector to match the label blue (in this example) so each LoadBalancer service that have that label will take an IP Address from this pool if there is one available. If all IP Addresses are already taken then the EXTERNAL-IP field of the service will stay in the pending state. Below is the CiliumLoadBalancerIPPool configuration:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "pool-blue"
spec:
  cidrs:
  - cidr: "10.0.0.0/30"
  serviceSelector:
    matchLabels:
      color: blue

You’ll then need to add the blue label to the LoadBalancer service

$ kubectl edit svc -n cilium cilium-ingress-basic-ingress

Just add the label color: blue and you’ll then see the following:

$ kubectl get svc -n cilium
NAME             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.107.176.240   10.0.0.1      80:31814/TCP   16h

Now our service has an external IP Address that comes from the pool-blue we have defined previously.

We can create our deployment by applying the following yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: testciliuml2
  namespace: cilium
spec:
  replicas: 2
  selector:
    matchLabels:
      name: testcontainers
  template:
    metadata:
      labels:
        name: testcontainers
    spec:
      containers:
      - name: testciliuml2
        image: testcontainers/helloworld
        env:
        - name: DELAY_START_MSEC
          value: "2000"

You can see we use the label name: testcontainers for this deployment. We will then need to edit again our LoadBalancer service to associate it with that deployment label (you can also see it in the diagram above if you want to check):

  selector:
    name: testcontainers

Finally we can configure an ingress object to link it to our LoadBalancer service by using the yaml file below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-ingress
  namespace: cilium
spec:
  ingressClassName: cilium
  rules:
  - http:
      paths:
      - backend:
          service:
            name: cilium-ingress-basic-ingress
            port:
              number: 80
        path: /
        pathType: Prefix

Note that the ingressClassName field uses the value cilium. This instructs Kubernetes to use Cilium as the Ingress controller for this resource. We can check the state of our ingress:

 kubectl get ing -n cilium
NAME            CLASS    HOSTS   ADDRESS    PORTS   AGE
basic-ingress   cilium   *       10.0.0.1   80      30d

You can recognize the external IP Address of our LoadBalancer service.

Test of connection to the LoadBalancer service

To fully test the flow from our laptop through the ingress and down to the pod, we would need the external IP Address to be reachable from my laptop. In production you will not use this external IP Address directly but use a URL with a domain name that will be resolved by a DNS to this external IP Address.

Here I will just test the connection to the LoadBalancer service with kubectl port-forward to reach port 80 of this service as it would be by our ingress rule:

$ kubectl port-forward svc/cilium-ingress-basic-ingress -n cilium 8080:80 &

I can then reach my application from my laptop by using the URL http://localhost:8080

As a conclusion, I’ve found the configuration of all these features pretty easy to bound together. It is then very simple to provide a complete Kubernetes architecture with only Cilium features to reach my application from outside of this cluster.

The next step will be to test it in a real Kubernetes cluster with some failover scenarii but that all look very promising to me!

L’article Combining Powerful Cilium Features Together In Kubernetes est apparu en premier sur dbi Blog.

The basics

CoreDNS is then a core component of a Kubernetes cluster and is operating as a Pod running on one of the node of your cluster. This node could be a master or a worker node. There are 2 CoreDNS Pods by default hosted on 2 separate nodes. That provides redundancy and load balancing in the cluster for this name to IP function. CoreDNS is deployed through a deployments object and so it can be scaled out or in according to your needs. The CoreDNS pods are reached through a service that holds a virtual IP Address to load balance traffic between these Pods.

The CoreDNS Pods belong to the kube-system namespace and its configuration is stored in a configmaps called coredns in that same namespace. This default configuration is something like this:

$ kubectl get cm coredns -n kube-system -o yaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-09-26T15:40:53Z"
  name: coredns
  namespace: kube-system

We advise to use log as in the example above in order to record all the resolution CoreDNS will do in its logs. This will ease the understanding and the troubleshooting around CoreDNS.

The DNS traffic to an object name inside the Kubernetes cluster

CoreDNS acts as a DNS resolver for our Kubernetes cluster so it listen on UDP port 53 as expected for a DNS. If you need to capture traffic to and from a CoreDNS Pod I would recommend to scale in the deployments to 1 Pod to be sure that all the traffic will go through it.

Let’s have a look at an example of a Pod sending a DNS Request and capture that traffic with cilium to better understand it. We will use a netshoot Pod that contains several networking tools for us to play with (here we will just use nslookup):

$ kubectl run netshoot --image=nicolaka/netshoot -it -- bash
netshoot:~# nslookup myservice.mynamespace.svc.cluster.local
Server:         172.21.0.10
Address:        172.21.0.10#53

Name:    myservice.mynamespace.svc.cluster.local
Address: 172.21.0.100

From the netshoot Pod we have resolved the name of one of our service (myservice) in the cluster to its IP Address 172.21.0.100. The CoreDNS service has the IP Address of 172.21.0.10.

At the same time in another shell we captured that traffic with Cilium. To monitor this traffic we have to use the Cilium agent that is running on the same node as our CoreDNS Pod. The result is as follows:

$ kubectl exec -it -n cilium cilium-4zs7z -- cilium monitor|grep ":53 "
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
-> endpoint 907 flow 0x0 , identity 5812->21815 state new ifindex lxcb2c39c2042dc orig-ip 172.20.1.46: 172.20.1.46:51450 -> 172.20.3.34:53 udp
-> overlay flow 0x0 , identity 21815->5812 state reply ifindex cilium_vxlan orig-ip 0.0.0.0: 172.20.3.34:53 -> 172.20.1.46:51450 udp

Our netshoot Pod has the IP Address of 172.20.1.46 and our CoreDNS Pod has the IP Address of 172.20.3.34. We can see that the request goes to the CoreDNS Pod on port 53 and then it responds back to the netshoot Pod. Not that if that service name didn’t exist, the flow would be identical, CoreDNS will not try to resolve this DNS request further because CoreDNS has authority on the domain svc.cluster.local. So if it doesn’t know, then nobody does and it will not ask another DNS resolver. Also note that I could do nslookup myservice.mynamespace and CoreDNS will automatically try a few suffix he has authority on and try to resolve it.

To give you the complete picture, here are the CoreDNS logs associated with this DNS traffic:

$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns
...
[INFO] 172.20.1.46:36694 - 8284 "A IN myservice.mynamespace.svc.cluster.local. udp 60 false 512" NOERROR qr,aa,rd 118 0.000289493s
...

We can see the IP Address of our netshoot Pod, the name to be resolved as well as its result. NOERROR means that the resolution has been done successfully and at the end we also have the time this resolution took. If the resolution had failed you would have seen NXDOMAIN instead of NOERROR.

The DNS traffic to a name outside of the Kubernetes cluster

Now let’s see what is happening when a Pod tries to resolve a name on which CoreDNS doesn’t have the authority. It could be any of the URL in your ingress rules or the API server name of your cluster. Those names are supposed to be resolved from a DNS that is outside of the Kubernetes cluster before the traffic can enter into your cluster. Also you may have an architecture where a Pod tries to connect to an Oracle database that is outside of the Kubernetes cluster and so the name resolution of this database will be done by a DNS in your local network that is outside of the cluster. Let’s use our netshoot Pod again to learn more about it:

netshoot:~# nslookup myoracledatabase.mydomain.com
Server:         172.21.0.10
Address:        172.21.0.10#53

Name:   myoracledatabase.mydomain.com
Address: 10.0.0.10

The name has been resolved but from the Pod we don’t see much about this resolution process. The Pod only knows the CoreDNS service IP Address which is 172.21.0.10. If you look at the DNS configuration inside this netshoot Pod you’ll see the following:

netshoot:~# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.21.0.10
options ndots:5

You see the domains on which CoreDNS has authority on and the CoreDNS service IP Address. And that’s it! So let’s have a look as previously what cilium can see about this traffic:

$ kubectl exec -it -n cilium cilium-4zs7z -- cilium monitor|grep ":53 "
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
-> endpoint 907 flow 0x0 , identity 5812->21815 state new ifindex lxcb2c39c2042dc orig-ip 172.20.1.46: 172.20.1.46:52794 -> 172.20.3.34:53 udp
-> stack flow 0x8414f33b , identity 21815->world state new ifindex 0 orig-ip 0.0.0.0: 172.20.3.34:46263 -> 8.8.8.8:53 udp
-> endpoint 907 flow 0xd6863373 , identity world->21815 state reply ifindex lxcb2c39c2042dc orig-ip 8.8.8.8: 8.8.8.8:53 -> 172.20.3.34:46263 udp
-> overlay flow 0x0 , identity 21815->5812 state reply ifindex cilium_vxlan orig-ip 0.0.0.0: 172.20.3.34:53 -> 172.20.1.46:52794 udp

You can see that in this case as CoreDNS doesn’t have authority on mydomain.com it asks the DNS he knows to do the resolution for him. Here it is 8.8.8.8 that is a google public DNS for this example and to anonymize the real DNS IP Address in our local network. So the DNS traffic goes that way:

Netshoot Pod (172.20.1.46) -> CoreDNS Pod (172.20.3.34) -> Local DNS (8.8.8.8) and back on the same path. Note that this is what is seen by the Cilium agent in our Kubernetes cluster but that our local DNS may also asks other DNS to do some resolution for him (this is the standard DNS recursive mechanics).

In the CoreDNS logs you can see this name resolution has been successful and that the time it took was slightly longer than for a resolution inside of the cluster. This is because it had to go through the local DNS so there were more steps in the process.

$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns
...
[INFO] 172.20.1.46:52167 - 21729 "A IN myoracledatabase.mydomain.com. udp 60 false 512" NOERROR qr,aa,rd 118 0.001330359s
...

How CoreDNS knows which local DNS to ask?

That is a very good question! This comes from the coredns configmaps plugin forward . /etc/resolv.conf we have seen at the beginning of this blog. OK that looks like an answer given by ChatGPT but can you see what is inside this file and where does it come from? I didn’t try ChatGPT on it but I guess it may have a hard time with these two questions. I’ve promised to explore CoreDNS in depth in this blog so I have to provide you the answers! Let’s see how we can learn this. The Pods are designed to provide a function and so the container inside it uses an image built with only the minimum of tools required for this function. That means it doesn’t always contain a shell for example. That also reduce the attack surface for the naughty guys and gals out there. The CoreDNS image doesn’t contain a shell but this is how you could kind of have one anyway to explore files in its container:

$ kubectl debug -it coredns-d669857b7-d4wqb -n kube-system --image=nicolaka/netshoot --target=coredns

coredns-d669857b7-hknvf  ~ cat /proc/1/root/etc/resolv.conf
search localdomain.com
nameserver 8.8.8.8

With kubectl debug you can jump into the Pod by using another image of your choice but still have access to the content of the original CoreDNS container (crazy stuff isn’t it?). coredns of target=coredns is the name of the container into the pod coredns-d669857b7-d4wqb. You can then look at the content of /proc/1/root/etc/resolv.conf which is what we were looking for.

To tell you everything, the coredns configmaps is also stored as a file in this container with the path /proc/1/root/etc/coredns/Corefile.

That answers the first question. The second one was where he gets this resolv.conf file from? Well the response is easy when you know it: It just comes from the /etc/resolv.conf file of the node that hosts this CoreDNS Pod! I did the test to add a record into /etc/resolv.conf of the node and I’ve redeployed CoreDNS and sure enough, the added record was also into /etc/resolv.conf of the coredns container! Remember that the CoreDNS Pod can be hosted on any node of your cluster. So if you want to do this test, you can either force this Pod Scheduling to a specific node or change the /etc/resolv.conf file of all the nodes. Testing it in a lab with a few nodes should then be pretty easy.

Modify the configuration of the CoreDNS

So you know by now how CoreDNS works but is there some way you could control this name resolution from inside of the cluster? You may have an architecture where your Pods need to use an URL that is managed by the Ingress Controllers of your cluster. In that case the name resolution will not be done by the CoreDNS but by your local DNS as we have seen previously. In your organization these local DNS (there are usually two of them) may be managed by another team and you don’t have the control over them. If for some reason these local DNS become unreachable then the applications in your cluster will not work anymore. If you want the Pods in your cluster to independently resolve these URL names, then you could map them statically in your CoreDNS by using the hosts plugin.

You just edit the coredns configmaps and add the hosts plugin before the forward plugin (Those blocks or instructions are called plugin in the CoreDNS terminology) as shown below:

$ kubectl get cm coredns -n kube-system -o yaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        hosts {
           1.2.3.4 myoracledatabase.mydomain.com
           fallthrough
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-09-26T15:40:53Z"
  name: coredns
  namespace: kube-system

After a few seconds, if you try again a name resolution for myoracledatabase.mydomain.com from the netshoot Pod, you’ll see it gives the IP Address you have set in the hosts plugin.

netshoot:~# nslookup myoracledatabase.mydomain.com
Server:         172.21.0.10
Address:        172.21.0.10#53

Name:   myoracledatabase.mydomain.com
Address: 1.2.3.4

So CoreDNS is able to do the resolution itself for this URL and doesn’t need to ask for the local DNS configured in /etc/resolv.conf. The parameter fallthrough means that if he doesn’t have a static mapping in hosts, then it will forward the DNS request to the local DNS for resolution just as before.

So this change is dynamic (there is nothing to restart) and with the fallthrough, the resolution continues to be done by the local DNS for external names or URLs if there is no mapping. That way you can add at your own pace these mappings that will be done by CoreDNS. By doing this, the name resolution will be faster and you keep the control over it. Of course the drawback is that you’ll have to keep that mapping up to date when new applications, using ingress rules for example, will be hosted in your cluster.

Note that it is useless to insert the hosts plugin after the forward plugin because the later doesn’t have the fallthrough parameter and so the hosts mapping will never be reached.

Wrap up

I hope that by now you are pretty solid about CoreDNS! I believe that understanding the details of how a component works is a great advantage when it comes to troubleshoot issues around it. Go deep and keep learning!

L’article Kubernetes CoreDNS in depth est apparu en premier sur dbi Blog.

And learning sessions were organised; the day before was dedicated to workshops, provided on three different subjects

• GitOps with ArgoCD
• Platform Engineering with Backstage
• Kubernetes Basics

Jumping on the tramway, spending a bit of time on the Gurtenbahn, we were right on time to start our journey. All was well organised. At the time specified in their email communication, staff was waiting for all the attendees. Badges were alphabetically sorted, so we didn’t suffer from any waiting queues!

After wandering across the site, we took a seat in the main pavillon, where the welcome session and keynote were going to start.

Anais Urlichs, from Aqua Security, introduced the day by talking about, naturally, security and cloud native. Her title, “Back to the future in the space of cloud native security” definitely ended up with a picture of a sports car, which became popular after a 1985 movie She started to flashback five years ago and came back to some security events. Codecov in 2021, SolarWinds in 2020, and MGM Resorts early this month: here are some breaches publicly reported.

What do they have in common? They were undetected by the security measures in place, which obviously failed. But staying positive, she closed her session by pushing the public to look at open-source and cloud native solutions, mentioning Sigstore for signing digital artefacts, Kyverno for policies, and Tracee using eBPF for monitoring.

A short break, we moved on, and the next session we chose was from Kunal Kushwaha “Embarking on a Kubernetes Odyssey: A Tale of Building, Deploying, and Scaling”. It was very worthwhile; his session was easy to follow with always a hint of humour. He was embarking us on a K8s journey as he promised, using his virtual assistant, Alex, and was tagging some questions and quotes: “Why should I go cloud native?”, “Benefits of a multi-cloud approach”. The most interesting was maybe this one: “Understanding the hype cycle”. From the “initial hype” to the “plateau of productivity”, he explained each step we would be faced with.

As a huge fan of Cilium from Isovalent, I stayed there to listen to Marga Manterola. She has been appointed Director of Engineering. We, at dbi services, use Cilium in Kubernetes deployments, running in a kube-proxy free setup. Even if I’m familiar with the concept of eBPF, it is always a great pleasure and an opportunity to learn new functionalities around the use of eBPF in the Cilium stack.

A few slides after the start of her session, she announced the great news, following a GitHub PR opened in Oct 2022! Cilium is finally upgraded to Graduated Project at the CNCF. Unfortunately, the Cilium page does not seem to reflect this change. This marks the maturity of the project, proving its stability and wide adoption.

Security on Kubernetes, using Cilium, involves the following points

• network policies (facilitated by the online editor),
• transparent encryption across nodes (either powered by the Wireguard or IPSec protocols),
• observability using Hubble (either the UI, the CLI, or the metrics for Prometheus and Grafana), 
• and runtime security using Tetragon, Cilium’s sister project.

Annie Talvasto, during the afternoon, provided us with an interesting overview of an unfortunately popular subject. Working at VSHN, Annie’s goal was to show us “How Kubernetes Optimisation Can Combat Climate Change”. Ambitious topic, I naively thought fighting against global warming would simply require some kubectl scale sts mylittle_voracious_statefulset --replicas=0. I was wrong. She went through the principles of green software, location shifting (moving workload across different locations), time shifting (scaling up and down accordingly to the time of the day), and demand shaping. She displayed the website of the Branch Magazine. Based on the demand, the website is showing more or less components, from full monochrome-text-only, to a color-flavored-with-nice-pictures web site.

Informative, KEDA was also mentioned at the end of the talk. I would definitely look at this tool, as it provides much more functionalities for automatic scaling on Kubernetes. At least much more than the “traditional” HPA.

I’ve probably forgotten to tell you about the other sessions, but all were, even if not as technical as initially expected, very informative. This was the first year we joined this event, and we’ll definitely be back next year! See you soon, Swiss Cloud Native Day !

L’article Swiss Cloud Native day landed at the Gurten! est apparu en premier sur dbi Blog.

I attended for the first time the KCD (Kubernetes Community Days) in Zürich with my colleagues Jean-Philippe and Arjen.

I can say I was pretty eager to know more about topics around Kubernetes and eBPF stuff as Isovalent was well represented.
This event is a two day conference with 2 streams at some slots, first day is dedicated to technical workshops and day 2 (the one I attended) is dedicated to talk sessions.

It started very well in a beautiful place at Google Swiss headquarter.

As we were at the opening, we had some time to do networking and it was funny to find Sacha Dubois a Senior solution specialist from VMware with who I had worked to implement a Managed Kubernetes with Tanzu last year.

The event started exactly on time at 8:45 AM in a crowdy sold out main conference room with a funny welcome message with many remarkable numbers starting with 2014 which is the initial release date of Kubernetes and ended with the number 10^100 which represent “one Googol” to thanks Google for their support.

Cilium Mesh

The first talk was with Thomas Graf from Isovalent and gave us an overview of what can be done with Cilium Mesh to manage our infrastructure with a minimal effort by just only adding annotations to mention if a service should be reachable from another Kubernetes cluster with Cluster Mesh.

The innovation, here, is that our workloads and machines can be managed easily whether they are in the cloud, on-prem or on the edge by creating an universal networking layer.

State of Green Washing

Then Max Körbächer talked about concrete sustainability with power consumption when using cloud resources for your Kubernetes cluster, to resume the idea, the best way to reduce your carbon emission is to be able to calculate it and then start by reducing non required tasks like multiple checks in your pipeline that are not required, change to “green regions”, and so on…

Scale at another level

After that, I followed a talk at the extreme opposite from Ricardo Rocha a computing Engineer at the CERN who explain us how they manage to handle billions of data in a short time with Kubernetes and before his talk I was wondering what concrete applications can require that much nodes, now I know!

He also explain his other challenge to succeed on computing efficiently data gathered. The software that compute should be able to scale rapidly, but its image is about 24 GB!!!
This is where they start to use stargz snapshotter, this project leverage on the fact that pulling an image before the container can start take around 76% of the startup time and optimize the container start by rearranging image layers and allowing lazy pulling, in his demo we saw the pod starting in less than 7s compared to 1 minute for the standard image, very impressive!

Anime and Kubernetes

Before the lunch I went to a more lightweight talk around Kubernetes but nevertheless very creative. Annie Talvasto has one of her passions which is Japanese animation, she talked about the analogy you can find in Anime and Kubernetes or tech careers.
With for example Naruto Multicloning and Kubernetes replicaset or the different evolution of Goku (Dragon Ball Z) from a kid to a Super Saiyan compared to when you started you career and when you start gaining badge from technical certificates, I loved that!
Thanks Annie for keeping fun in our job!

During the lunch break, I had the opportunity to talk to Liz Rice and got her book “What is eBPF” signed.

In the afternoon, I joined Lena Fuhrimann’s talk about how to troubleshoot efficiently our Kubernetes cluster with their project livelint.

After that, I choose to ear Julius Volz, co-founder of Prometheus.io, talking about enhancement done is prometheus metrics for histograms with native histograms, never think all is carved in marble when you use open-source project, enhancement can be done everywhere.

Extend observability with eBPF

Before the last talk, I followed Raphaël Pinson’s talk an other Isovalent talented member on removing the bridge between Dev and Ops when talking about monitoring by bringing observability to a new level with eBPF.
A fact is, eBPF is not a new thing, it’s everywhere since 10 years now, it allows Linux Kernel to be programmable without any additional configuration at application layer.

You can find here the list of eBPF applications.

The conference ended with Liz Rice talk in the main room touting about eBPF possibilities and coupled with Kubernetes Tetragon CRDs to bring your security to another level, the less we can say is, if you consider security, think with eBPF capabilities.

Conclusion

This day was full of good talks and I hope you learned something from reading this resume.
I encourage you also to read Jean-Philippe’s blog here for the other talks of this event.

Thanks dbi services for giving the opportunity to make your employees growing up!

L’article Kubernetes Community Days 2023 in Zürich est apparu en premier sur dbi Blog.