KubeCon / CloudNativeCon 2023 – Day 3

Today is the last day of conferences. It is unexpected but I get fitter and fitter as I walk between 4 and 5 kilometre each day from one session to another. The RAI Amsterdam Convention Centre is huge and when finally on the last day you have the map of it in the head and know all the shortcuts, it is almost time to leave!

If you’ve missed my blog from the 2 previous days have a look here: Day 1 and Day 2

Keynote sessions

As we are used to now, we start with a bunch of Keynote sessions. Those presentations are made from the top speakers, la crème de la crème (excuse my French)! I was also very interested to come at such event to see how the other public speakers are doing their presentations and get some tips and ideas I could use for my own. Yesterday I really appreciated the nice pictures in the presentation of Emily Fox. There were some new styles today I was interested in. I’ve experienced lately creating images using AI and I’ve seen that several persons in these conferences are doing the same to integrate in their presentations. Frederick Kautz for example did it and he chose a theme with cats. That supported very well his talk about Trust No System. He gave examples of consequences of too much trust or too little. The key is to have no implicit trust but reason about trust. Below is a selection of his slides:

I also had great interest in the talk of Giles Heron about enabling real-time media in Kubernetes. The open source Media Streaming Mesh project has been created for this purpose and he demonstrated the example of a soccer video using traditional streaming and streaming in Kubernetes. In Kubernetes it was much faster and you could see the player scored much earlier!

In summary there is a continuous effort to leverage Kubernetes and make it better. Its complexity is well known and it is a trade-off between simplicity and flexibility as summarised below from Aparna Subramanian‘s presentation:

And as shown below by Denise Schannon we all want to reach Nirvana with Kubernetes! I love that picture!

The Security track

If you’ve read my post on the last 2 days you now know that I’ve focused on the security presentations at those conferences. I have myself a fresh presentation about “Hacking Kubernetes for Beginners” so I’m keen to learn from the best in this area.

The first presentation I’ve attended to is about preventing cluster takeovers from Shopify. I’ve learned there is a state of Kubernetes security report and that for the last year, there has been 53% of detected Kubernetes security misconfigurations. Most of the default configurations of the components are not secure and can be exploited for attack. It is not only true for vanilla Kubernetes, they talked about a 2-years vulnerability in runC that has been used in Azure containers. It is now fixed but it is a good reminder that even Kubernetes services from a Cloud provider can be compromised. They then did a great demo of attack and defence. They used their open source tool kubeaudit to show how to fix default configuration issues and then close the door to attacks using them. Another good tool I’ll have to try!

Yesterday I unsuccessfully tried to attend a LIVE demo of hacking and defending a Kubernetes cluster but there were a doing it again today! Unfortunately I couldn’t attend today either because the room was already full again. I guess the attraction for such demo has been underestimated because there were bigger room at this convention centre.

I’m glad I’ve chosen the fantastic presentation of a quartet about containers scanner. The dynamic of the talk between them was perfectly orchestrated, I had in front of my eyes an example of a human living Kubernetes cluster! They did the show and the audience was responsive. They demonstrated how with a few tricks it is possible to completely confuse a container scanner and going from several vulnerabilities found, down to 0! With just a matter of deleting files, creating symbolic links and building a multi-stage image. They compared four container scanners and all were tricked! This presentation was a call for developing better security together as a community.

The scanning result with the default image:

The scanning result with a tweaked image:

The last presentation in this track I’ve attended to was about challenges with Pod Security Admission from VMware. There is some work to do before deploying a workload in a Kubernetes cluster which is to limit the capabilities of a Pod to the minimum it requires. It is easier to do that work beforehand than trying to reduce its capabilities once it is already deployed. They gave us a real world example on how to do such a design: Separate Pods in separate namespace and for each of them limit its capabilities to what is really required by the application. That design is much more secure than just giving all the privileges to all Pods of a workload.

The Kubernetes track

I’m glad I was able to secure a seat at Thomas Graf‘s (co-founder of Isovalent and Cilium) presentation about troubleshooting Kubernetes networking. The room was packed and some people were standing up in the back. Late comers were not allowed to enter anymore for security reasons. As we support Cilium for our customers and I’ve already used it, I was interested to know more. It was a nice recap with several use cases scenarii starting from the Grafana Dashboard down to the hubble command line to narrow down a network issue and fix it. It was a well-rounded presentation and demo around Network Policy and DNS issues. I loved that picture which is a nice recap of the network chain involved in troubleshooting:

The service mesh topic has also been very much around at these conferences so I decided to learn more about it with the session from IBM, Google, Solo and Microsoft. It was in the format of questions and answers. They talked mainly about sidecar and eBPF. I’ve re-learned that eBPF has some limitations (about mTLS,…). This is what I’ve already heard earlier so I’ll have to dig that topic (and yeah I’ve got now the signed book Learning eBPF from Liz Rice!). Because of those limitations, security of service mesh will not use it. eBPF is good for traffic and infrastructure knowledge as well as observability, but can’t be used for everything.

What is a better way to finish this marathon conference than attending a presentation from Liz Rice? She talked about Cilium networking for multicloud Kubernetes. I’ve learned more about this new feature Cilium Mesh that has been announced earlier. As usual Liz presentation was amazing with demo on Cilium favorite theme: Star wars! Cilium Mesh has been created to connect workload in multiple clusters and non-Kubernetes environments in public clouds and on-prem! She demonstrated for example that by just using one annotation in a yaml file, it was possible for a service to reach CiliumEndPoint in different clusters. And so load-balance between them. Amazing!

Wrap up

Finally to wrap this up, I’ve learned during those sessions something funny I didn’t realise yet. There is a real competition between 2 groups of people regarding how to pronounce kubectl. The old school says kube “c” “t” “l” or kube control but the trendy group says kube “cutl”! My colleague Chay Te is definitely trendy!

That was an amazing event and the organisation of everything was very high level. Kudos to everybody who has worked on preparing and operating those events! #KubeCon and #CloudNativeCon rock and I thank dbi services for giving me the opportunity to attend. I hope to be a speaker there one day!