You know that CoreDNS is one of the core component of Kubernetes. You may also know that the CoreDNS pod is up and running only after the Container Network Interface (CNI) has been installed in your Kubernetes cluster. Of course you know what CoreDNS is about, it converts names to IP Addresses for objects in the Kubernetes cluster. If you are interested to learn a bit more about how it works under the hood then keep reading. I’ll share with you some information that will help you understand CoreDNS in more depth. We will use Cilium as CNI to easily monitor DNS traffic in our cluster. Take a deep breath and let’s dive in.

The basics

CoreDNS is then a core component of a Kubernetes cluster and is operating as a Pod running on one of the node of your cluster. This node could be a master or a worker node. There are 2 CoreDNS Pods by default hosted on 2 separate nodes. That provides redundancy and load balancing in the cluster for this name to IP function. CoreDNS is deployed through a deployments object and so it can be scaled out or in according to your needs. The CoreDNS pods are reached through a service that holds a virtual IP Address to load balance traffic between these Pods.

The CoreDNS Pods belong to the kube-system namespace and its configuration is stored in a configmaps called coredns in that same namespace. This default configuration is something like this:

$ kubectl get cm coredns -n kube-system -o yaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-09-26T15:40:53Z"
  name: coredns
  namespace: kube-system

We advise to use log as in the example above in order to record all the resolution CoreDNS will do in its logs. This will ease the understanding and the troubleshooting around CoreDNS.

The DNS traffic to an object name inside the Kubernetes cluster

CoreDNS acts as a DNS resolver for our Kubernetes cluster so it listen on UDP port 53 as expected for a DNS. If you need to capture traffic to and from a CoreDNS Pod I would recommend to scale in the deployments to 1 Pod to be sure that all the traffic will go through it.

Let’s have a look at an example of a Pod sending a DNS Request and capture that traffic with cilium to better understand it. We will use a netshoot Pod that contains several networking tools for us to play with (here we will just use nslookup):

$ kubectl run netshoot --image=nicolaka/netshoot -it -- bash
netshoot:~# nslookup myservice.mynamespace.svc.cluster.local
Server:         172.21.0.10
Address:        172.21.0.10#53

Name:    myservice.mynamespace.svc.cluster.local
Address: 172.21.0.100

From the netshoot Pod we have resolved the name of one of our service (myservice) in the cluster to its IP Address 172.21.0.100. The CoreDNS service has the IP Address of 172.21.0.10.

At the same time in another shell we captured that traffic with Cilium. To monitor this traffic we have to use the Cilium agent that is running on the same node as our CoreDNS Pod. The result is as follows:

$ kubectl exec -it -n cilium cilium-4zs7z -- cilium monitor|grep ":53 "
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
-> endpoint 907 flow 0x0 , identity 5812->21815 state new ifindex lxcb2c39c2042dc orig-ip 172.20.1.46: 172.20.1.46:51450 -> 172.20.3.34:53 udp
-> overlay flow 0x0 , identity 21815->5812 state reply ifindex cilium_vxlan orig-ip 0.0.0.0: 172.20.3.34:53 -> 172.20.1.46:51450 udp

Our netshoot Pod has the IP Address of 172.20.1.46 and our CoreDNS Pod has the IP Address of 172.20.3.34. We can see that the request goes to the CoreDNS Pod on port 53 and then it responds back to the netshoot Pod. Not that if that service name didn’t exist, the flow would be identical, CoreDNS will not try to resolve this DNS request further because CoreDNS has authority on the domain svc.cluster.local. So if it doesn’t know, then nobody does and it will not ask another DNS resolver. Also note that I could do nslookup myservice.mynamespace and CoreDNS will automatically try a few suffix he has authority on and try to resolve it.

To give you the complete picture, here are the CoreDNS logs associated with this DNS traffic:

$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns
...
[INFO] 172.20.1.46:36694 - 8284 "A IN myservice.mynamespace.svc.cluster.local. udp 60 false 512" NOERROR qr,aa,rd 118 0.000289493s
...

We can see the IP Address of our netshoot Pod, the name to be resolved as well as its result. NOERROR means that the resolution has been done successfully and at the end we also have the time this resolution took. If the resolution had failed you would have seen NXDOMAIN instead of NOERROR.

The DNS traffic to a name outside of the Kubernetes cluster

Now let’s see what is happening when a Pod tries to resolve a name on which CoreDNS doesn’t have the authority. It could be any of the URL in your ingress rules or the API server name of your cluster. Those names are supposed to be resolved from a DNS that is outside of the Kubernetes cluster before the traffic can enter into your cluster. Also you may have an architecture where a Pod tries to connect to an Oracle database that is outside of the Kubernetes cluster and so the name resolution of this database will be done by a DNS in your local network that is outside of the cluster. Let’s use our netshoot Pod again to learn more about it:

netshoot:~# nslookup myoracledatabase.mydomain.com
Server:         172.21.0.10
Address:        172.21.0.10#53

Name:   myoracledatabase.mydomain.com
Address: 10.0.0.10

The name has been resolved but from the Pod we don’t see much about this resolution process. The Pod only knows the CoreDNS service IP Address which is 172.21.0.10. If you look at the DNS configuration inside this netshoot Pod you’ll see the following:

netshoot:~# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.21.0.10
options ndots:5

You see the domains on which CoreDNS has authority on and the CoreDNS service IP Address. And that’s it! So let’s have a look as previously what cilium can see about this traffic:

$ kubectl exec -it -n cilium cilium-4zs7z -- cilium monitor|grep ":53 "
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
-> endpoint 907 flow 0x0 , identity 5812->21815 state new ifindex lxcb2c39c2042dc orig-ip 172.20.1.46: 172.20.1.46:52794 -> 172.20.3.34:53 udp
-> stack flow 0x8414f33b , identity 21815->world state new ifindex 0 orig-ip 0.0.0.0: 172.20.3.34:46263 -> 8.8.8.8:53 udp
-> endpoint 907 flow 0xd6863373 , identity world->21815 state reply ifindex lxcb2c39c2042dc orig-ip 8.8.8.8: 8.8.8.8:53 -> 172.20.3.34:46263 udp
-> overlay flow 0x0 , identity 21815->5812 state reply ifindex cilium_vxlan orig-ip 0.0.0.0: 172.20.3.34:53 -> 172.20.1.46:52794 udp

You can see that in this case as CoreDNS doesn’t have authority on mydomain.com it asks the DNS he knows to do the resolution for him. Here it is 8.8.8.8 that is a google public DNS for this example and to anonymize the real DNS IP Address in our local network. So the DNS traffic goes that way:

Netshoot Pod (172.20.1.46) -> CoreDNS Pod (172.20.3.34) -> Local DNS (8.8.8.8) and back on the same path. Note that this is what is seen by the Cilium agent in our Kubernetes cluster but that our local DNS may also asks other DNS to do some resolution for him (this is the standard DNS recursive mechanics).

In the CoreDNS logs you can see this name resolution has been successful and that the time it took was slightly longer than for a resolution inside of the cluster. This is because it had to go through the local DNS so there were more steps in the process.

$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns
...
[INFO] 172.20.1.46:52167 - 21729 "A IN myoracledatabase.mydomain.com. udp 60 false 512" NOERROR qr,aa,rd 118 0.001330359s
...

How CoreDNS knows which local DNS to ask?

That is a very good question! This comes from the coredns configmaps plugin forward . /etc/resolv.conf we have seen at the beginning of this blog. OK that looks like an answer given by ChatGPT but can you see what is inside this file and where does it come from? I didn’t try ChatGPT on it but I guess it may have a hard time with these two questions. I’ve promised to explore CoreDNS in depth in this blog so I have to provide you the answers! Let’s see how we can learn this. The Pods are designed to provide a function and so the container inside it uses an image built with only the minimum of tools required for this function. That means it doesn’t always contain a shell for example. That also reduce the attack surface for the naughty guys and gals out there. The CoreDNS image doesn’t contain a shell but this is how you could kind of have one anyway to explore files in its container:

$ kubectl debug -it coredns-d669857b7-d4wqb -n kube-system --image=nicolaka/netshoot --target=coredns

coredns-d669857b7-hknvf  ~ cat /proc/1/root/etc/resolv.conf
search localdomain.com
nameserver 8.8.8.8

With kubectl debug you can jump into the Pod by using another image of your choice but still have access to the content of the original CoreDNS container (crazy stuff isn’t it?). coredns of target=coredns is the name of the container into the pod coredns-d669857b7-d4wqb. You can then look at the content of /proc/1/root/etc/resolv.conf which is what we were looking for.

To tell you everything, the coredns configmaps is also stored as a file in this container with the path /proc/1/root/etc/coredns/Corefile.

That answers the first question. The second one was where he gets this resolv.conf file from? Well the response is easy when you know it: It just comes from the /etc/resolv.conf file of the node that hosts this CoreDNS Pod! I did the test to add a record into /etc/resolv.conf of the node and I’ve redeployed CoreDNS and sure enough, the added record was also into /etc/resolv.conf of the coredns container! Remember that the CoreDNS Pod can be hosted on any node of your cluster. So if you want to do this test, you can either force this Pod Scheduling to a specific node or change the /etc/resolv.conf file of all the nodes. Testing it in a lab with a few nodes should then be pretty easy.

Modify the configuration of the CoreDNS

So you know by now how CoreDNS works but is there some way you could control this name resolution from inside of the cluster? You may have an architecture where your Pods need to use an URL that is managed by the Ingress Controllers of your cluster. In that case the name resolution will not be done by the CoreDNS but by your local DNS as we have seen previously. In your organization these local DNS (there are usually two of them) may be managed by another team and you don’t have the control over them. If for some reason these local DNS become unreachable then the applications in your cluster will not work anymore. If you want the Pods in your cluster to independently resolve these URL names, then you could map them statically in your CoreDNS by using the hosts plugin.

You just edit the coredns configmaps and add the hosts plugin before the forward plugin (Those blocks or instructions are called plugin in the CoreDNS terminology) as shown below:

$ kubectl get cm coredns -n kube-system -o yaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        hosts {
           1.2.3.4 myoracledatabase.mydomain.com
           fallthrough
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-09-26T15:40:53Z"
  name: coredns
  namespace: kube-system

After a few seconds, if you try again a name resolution for myoracledatabase.mydomain.com from the netshoot Pod, you’ll see it gives the IP Address you have set in the hosts plugin.

netshoot:~# nslookup myoracledatabase.mydomain.com
Server:         172.21.0.10
Address:        172.21.0.10#53

Name:   myoracledatabase.mydomain.com
Address: 1.2.3.4

So CoreDNS is able to do the resolution itself for this URL and doesn’t need to ask for the local DNS configured in /etc/resolv.conf. The parameter fallthrough means that if he doesn’t have a static mapping in hosts, then it will forward the DNS request to the local DNS for resolution just as before.

So this change is dynamic (there is nothing to restart) and with the fallthrough, the resolution continues to be done by the local DNS for external names or URLs if there is no mapping. That way you can add at your own pace these mappings that will be done by CoreDNS. By doing this, the name resolution will be faster and you keep the control over it. Of course the drawback is that you’ll have to keep that mapping up to date when new applications, using ingress rules for example, will be hosted in your cluster.

Note that it is useless to insert the hosts plugin after the forward plugin because the later doesn’t have the fallthrough parameter and so the hosts mapping will never be reached.

Wrap up

I hope that by now you are pretty solid about CoreDNS! I believe that understanding the details of how a component works is a great advantage when it comes to troubleshoot issues around it. Go deep and keep learning!

Post Views: 1,397
Thumbnail [60x60]
by
Benoît Entzmann