KubeCon / CloudNativeCon 2023 – Day 1

Day 0 warm up!

We arrived yesterday with my colleague Arnaud Berbier in Amsterdam for the KubeCon / CloudNativeCon but also the first CiliumCon 2023. All those events are grouped together. Thank you dbi services for this very great opportunity!

We’ve carefully prepared our Schedule to make the most of it. There are a lot of sessions of various style and duration. Some are common to all and for others you’ll have to select a stream in concurrent sessions. That is the most difficult part because everything is interesting! One thing is certain: this is going to be intense every day!

Yesterday afternoon we already had the opportunity to meet the community and attend a few sessions to put us in the mood. I’ve followed a 2 hours presentation of Cilium from Isovalent. The room was crowded as people working from companies around the world have started using Cilium or are considering it. At dbi services we already support a customer that is using Cilium for several months so we have been exposed to some part of it. It was then for me an opportunity to learn more about Cilium features we didn’t play with yet as Tetragon, Gateway API or Service Load Balancing. Isovalent also announced a new feature I should learn more about this week: Cilium Mesh!

This pre-day conference – or day 0 if you prefer – ended up with a common session of lightening talks for one hour. This punchy format of 5 minutes introduces a topic, an idea or just something to share with the community and I’ve noticed a trend around power consumption. That topic is also a great concern for dbi services and my colleague Emmanuel Wagner brilliantly tackled Green Coding at our last internal dbi xChange.

We ended the day at the Hive Mind Mingle with Isovalent for a very nice aperitif!

Keynotes

Day 1 started very strongly as there were more than 10’000 people converging to the conference room this morning! This is the highest audience on record yet for CNCF conferences. Look at this:

We were warmed up with a smorgasboard of Keynote presentations, an update on CNCF , its projects and its partners as well. CNCF is now the larger open source community with a growing number of projects. CNCF announced new Cloud native and GitOps certifications.

A lot of focus again in the topics presented about the efficiency of Power consumption. The open source Kepler project from Red Hat has been launched to capture power usage metrics from Kubernetes cluster. Another project called KEDA Operator get current Carbon Intensity and limit scaling in a Kubernetes cluster when Carbon Intensity is high and allow maximum scaling when it is low.

We then got a glimpse at the update of some CNCF projects as Envoy Proxy, Flux, Fluentbit, Harbor, Linkerd, Open Policy Agent, Prometheus, TUF and Vitess.

Innovation for debugging + Service Mesh

I’ve attended a presentation about correlation from Red Hat where they have started to work on a new product called correl8r. When we get an alert in Kubernetes (OpenShit for Red Hat but the problem applies to any Kubernetes flavor), it takes a long time to gather signals (logs, metrics, traces, network events and K8s events) of every components involved in this alert. It is only after this collection that the analysis can start to solve the issue. This product will focus on gathering automatically all those signals for an alert and show their relations in a graph. The product is in its early stage but it will have an UI as well as a CLI. Here is an example of what this product can do:

That is definitely the kind of product that will help troubleshooting a Kubernetes cluster more efficiently.

Zahari Dichev shared his view about sidecars for Service Mesh and if eBPF can replace it. eBPF has limitations regarding layer 7 and it has difficulties to handle mTLS handshake amongst other things. In his view, sidecar do the job well for service mesh and are then meant to stay on the long haul. Here are some popular folklore against sidecars he talked about:

Security path

I’ve then focused on the security path of those conferences by starting with using OpenTelemetry for Application Security.

The application architecture evolved from monolithic with local vulnerabilities to cloud native with distributed vulnerabilities:

This implies that new vulnerabilities could be exploited by hackers and so the tools used to discover them should evolve as well to adapt to this new architecture.

Ron Vider did a compelling demonstration that used OpenTelemetry and Jaeger to detect a vulnerable variable in a Python program that could be exploited from outside of the cluster. OpenTelemetry (here with Jaeger) needed to be adapted to be able to do this as by default it is not developed for such purpose. Traditionally Bandit would be used to statically scan the code and detect this vulnerability. However the same result was found with OpenTelemetry that is more adapted to this cloud native architecture. Observability is then crucial for improving security in modern software development.

I’ve then reached the most interesting part of my day here (the second most interesting actually, the first one was receiving a signed copy of the book “Learning eBPF” from Liz Rice herself!): Hacking Kubernetes! I’ve presented that topic during our last dbi xChange. I’m a beginner in this area so I was very keen on learning from the experts in this field especially when they are present at those conferences! The demonstration of Patrycja Wegrzynowicz on hacking a weakly configured Kubernetes cluster was brilliant. She used the tool kubeletctl to scan the cluster, get root access into a Pod and hacked participant accounts who were asked to create one at the beginning of the presentation.

I’ve then attended an introduction to Cloud Native Capture The Flag (CTF) from the company ControlPlane. Andrew Martin who wrote the famous book “Hacking Kubernetes” and is the founder and CEO of this company suggests to us to learn by doing. The goal is to connect to a Kubernetes cluster and find the flags by exploiting vulnerabilities in this cluster. Tomorrow almost the whole day is planned for this activity, I’m still pondering it I will give it a try as I will miss all the other presentations that day.

I’ve finally listened Maya Levine about the Anatomy of a Cloud Security Breach. I’ve learned that attacker’s skills are on the raise in Cloud technology and they are targeting for example weak configured storage for Ransomware extortion. Their main goal is then making profit with Ransomware or Cryptomining but also disrupt services for political reasons. Supply chain attacks using malicious image distribution is also very much used with devastating effects. 87% of containers images have high or critical vulnerabilities so it is important to fix them before using those images.

Also 83% of containers are running as root so it is recommended to change that to limit the possibilities of an attacker owning one. Finally get visibility in real time of your system and apply the least privilege rule for any components to reduce the risks of exposure.

What a day! See you tomorrow!