On producer side, messages are load balanced across partition based on hashed key. On the consumer side, to ensure that each message is processed by only one consumer within the group, allowing for load balancing, fault tolerance, and efficient utilization of resources in a Kafka consumer application, there is the concept of consumer group. Creation of Consumer Group There is…
In this blog post, I will demonstrate the first few concepts to understand when you want to learn Apache Kafka. Documentation explains it well, but I find that concrete examples give a better feeling of the product and features. Message and Broker Kafka uses the message (also known as event or record) as the unit of data. Optionally, they can…
Enabling monitoring on Kafka with Prometheus/Grafana is not more difficult than others exporters I configured in my previous monitoring in Prometheus blogs. Let’s see this together. JMX Exporter vs Kafka Exporter As Kafka is a Java based application, it would have been possible to monitor it with JMX Exporter as I did for Apache Tomcat. But this time, I will…
I will add another example of what can be done with Ansible Driven Automation by interfacing it with Kafka. For those who wonder what Event-Driven Ansible is have a look at my previous blog about it. Additional Installation Steps In my previous blog, I already listed the steps to setup Event Driven Ansible (EDA). To connect to Kafka (I will…
What is Apache Kafka ? No, Kafka is not only the famous author (en.wikipedia.org/wiki/Franz_Kafka), it’s an open-source distributed pub-sub messaging system with powerful skills like scalability and fault tolerance. It’s also a stream processing platform (near real-time) for the streaming datasources. The design of Apache Kafka is strongly influenced by the commit logs. Apache Kafka was originally developed by Linkedin and was…
