{"id":32707,"date":"2024-04-19T14:52:21","date_gmt":"2024-04-19T12:52:21","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=32707"},"modified":"2024-09-06T14:00:17","modified_gmt":"2024-09-06T12:00:17","slug":"apache-kafka-and-ksqldb","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/apache-kafka-and-ksqldb\/","title":{"rendered":"Apache Kafka and ksqlDB"},"content":{"rendered":"\n

After two introduction blogs on Apache Kafka (ie. Apache Kafka Concepts by Example<\/a> and Apache Kafka Consumer Group<\/a>), it is time to discover the wide ecosystem around it. In this blog post, I will play with ksqlDB<\/a>, streams and tables.<\/p>\n\n\n\n

ksqlDB<\/h2>\n\n\n\n

ksqlDB is a server that takes benefit of an Apache Kafka infrastructure for real time data streaming. It can be used to capture events (via, for example, Kafka Connect), transform events, expose views (or tables).<\/p>\n\n\n\n

Starting ksqlDB server is easy. First, we need to set the bootstrap servers of our Kafka cluster in etc\/ksqldb\/ksql-server.properties<\/code> file:<\/p>\n\n\n

\nbootstrap.servers=localhost:29092\n<\/pre><\/div>\n\n\n
As it is used for a proof of concept, I use a one broker cluster (ie. without high availability). Of course, ksqlDB server supports HA.<\/p>\n\n\n\n
The command to start:<\/p>\n\n\n
\nbin\/ksql-server-start etc\/ksqldb\/ksql-server.properties\n<\/pre><\/div>\n\n\n
And once, it up and running, you should see this:<\/p>\n\n\n
\n[2024-04-19 11:43:25,485] INFO Waiting until monitored service is ready for metrics collection (io.confluent.support.metrics.BaseMetricsReporter:173)\n[2024-04-19 11:43:25,485] INFO Monitored service is now ready (io.confluent.support.metrics.BaseMetricsReporter:185)\n[2024-04-19 11:43:25,485] INFO Attempting to collect and submit metrics (io.confluent.support.metrics.BaseMetricsReporter:144)\n[2024-04-19 11:43:25,486] INFO ksqlDB API server listening on http:\/\/0.0.0.0:8088 (io.confluent.ksql.rest.server.KsqlRestApplication:385)\n\n                  ===========================================\n                  =       _              _ ____  ____       =\n                  =      | | _____  __ _| |  _ \\| __ )      =\n                  =      | |\/ \/ __|\/ _` | | | | |  _ \\      =\n                  =      |   &lt;\\__ \\ (_| | | |_| | |_) |     =\n                  =      |_|\\_\\___\/\\__, |_|____\/|____\/      =\n                  =                   |_|                   =\n                  =        The Database purpose-built       =\n                  =        for stream processing apps       =\n                  ===========================================\n\nCopyright 2017-2022 Confluent Inc.\n\nServer 0.29.0 listening on http:\/\/0.0.0.0:8088\n\nTo access the KSQL CLI, run:\nksql http:\/\/0.0.0.0:8088\n\n[2024-04-19 11:43:25,489] INFO Server up and running (io.confluent.ksql.rest.server.KsqlServerMain:153)\n[2024-04-19 11:47:04,248] INFO Successfully submitted metrics to Confluent via secure endpoint (io.confluent.support.metrics.submitters.ConfluentSubmitter:146)\n<\/pre><\/div>\n\n\n
ksqlDB Client<\/h2>\n\n\n\n
Now, we are ready to start the ksqlDB client:<\/p>\n\n\n
\nbin\/ksql http:\/\/localhost:8088\n<\/pre><\/div>\n\n\n
if all went well, you should receive the ksql prompt and see the server status as RUNNING<\/mark>:<\/p>\n\n\n
\n                  ===========================================\n                  =       _              _ ____  ____       =\n                  =      | | _____  __ _| |  _ \\| __ )      =\n                  =      | |\/ \/ __|\/ _` | | | | |  _ \\      =\n                  =      |   &lt;\\__ \\ (_| | | |_| | |_) |     =\n                  =      |_|\\_\\___\/\\__, |_|____\/|____\/      =\n                  =                   |_|                   =\n                  =        The Database purpose-built       =\n                  =        for stream processing apps       =\n                  ===========================================\n\nCopyright 2017-2022 Confluent Inc.\n\nCLI v0.29.0, Server v0.29.0 located at http:\/\/localhost:8088\nServer Status: RUNNING\n\nHaving trouble? Type 'help' (case-insensitive) for a rundown of how things work!\n\nksql&gt;\n<\/pre><\/div>\n\n\n
Stream<\/h2>\n\n\n\n
One advantage, and a drawback as well, is that a Kafka topic (refer to my previous blog post<\/a> if you don’t know what this is) can store anything and each message can have its own format (text or binary). Schema Registry<\/a> can enforce formatting rules, versioning and serialization\/de-serialization information (I will cover that in another blog). ksql also enforces the formatting when defining a stream. For example, I create a order stream with 4 fields:<\/p>\n\n\n\n