Todd M

Apache Kafka, renowned for its distributed streaming capabilities, relies on a well-defined set of APIs to facilitate communication between clients and brokers. Understanding the compatibility between Kafka clients and broker API versions is crucial for maintaining a stable and efficient streaming environment. In this blog post, we’ll delve into the realm of Kafka Broker API … Read more

To implement a multi tenant Kafka architecture, several requirements need to be addressed in order to increase your chances of success. In this post, we will list and describe four requirements in multi tenant Kafka architectures which can lead to one optional configuration benefit. The final benefit will only be interesting depending on your unique … Read more

Apache Kafka has become a cornerstone in data processing and streaming architectures, offering robust publish-subscribe capabilities for handling real-time data. It’s well-regarded for its high-throughput, durability, and scalability which is essential for modern applications that rely on fast, reliable data streaming. Yet, managing Kafka clusters and their associated infrastructure can be complex, necessitating tools that … Read more

How do Kafka administrators perform administrative and diagnostic collection actions of Kafka topics?  This post explores a Kafka topic admin tool called kafka-topics.sh.  This command-line tool included in Apache Kafka distributions.  There are other examples of both open source and 3rd party tools which can also be used for Kafka topic administrative tasks, but for … Read more

Kafka namespaces are not directly supported in Apache Kafka, but there are two ways to implement namespace-like capability in Kafka. In this Kafka namespaces tutorial, we’ll cover both examples, history, options, why you might need namespaces, and much more. Let’s go. A quick note on how this tutorial is configured. In the beginning, I am … Read more

Kafka quotas provide the ability to govern and control the broker resources used by Kafka clients. More broadly, with Kafka quotas, you can limit how the resources used on the entire Kafka cluster from Kafka clients. Kafka quotas are used for primarily two reasons: 1) prevent misbehaving client(s) from unintentionally or intentionally attempting to adverse … Read more

Apache Kafka includes a command-line tool named kafka-configs.sh used to obtain configuration values for various types of entities such as topics, clients, users, brokers, and loggers. But, using this tool to determine current configuration values at runtime can be more difficult to use than you might expect. This can be especially true if you want … Read more

Knowing the Kafka version you are using may not be as straightforward as you might think. For example, if you search for “kafka version” in your favorite search engine or chatbot, there are all kind of results. But, I took a look at many of the top results and became concerned because the answers provided … Read more

Kafka Access Control Lists (ACLs) are a way to secure your Kafka cluster by specifying which users or client applications have access to which Kafka resources (topics, clusters, etc.). The process of authorizing or refusing access to particular resources or functions within a software application is referred to as “authorization” in software. It is the … Read more

Although Apache Kafka is written in Java, there are Python Kafka clients available for use with Kafka.  In this tutorial, let’s go through examples of Kafka with Python Producer and Consumer clients.  Let’s consider this a “Getting Started” tutorial.  After completing this, you will be ready to proceed to more complex examples.  But we need … Read more

In this post, let’s answer the question of Kafka and Dead Letter Quest. But first, let’s start with an overview. A dead letter queue (DLQ) is a queue, or a topic in Kafka, used to hold messages which can not be processed successfully. The origin of DLQs are traditional messaging systems which were popular before … Read more

Kafka provides multiple authentication options.  In this tutorial, we will describe and show the authentication options and then configure and run a demo example of Kafka authentication. There are two primary goals of this tutorial: There are a few key subjects which must be considered when building a multi-tenant cluster, but it all starts with … Read more

How do Kafka administrators perform administrative and diagnostic collection actions of Kafka Consumer Groups?  This post explores a Kafka Groups operations admin tool called kafka-consumer-groups.sh.  This popular, command-line tool included in Apache Kafka distributions.  There are other examples of both open source and 3rd party tools not included with Apache Kafka which can also be … Read more

Why “Joinable” Streaming Test Data for Kafka? When creating streaming join applications in KStreams, ksqldb, Spark, Flink, etc. with source data in Kafka, it would be convenient to generate fake data with cross-topic relationships; i.e. a customer topic and an order topic with a value attribute of customer.id.  In this example, we might want to … Read more

Kafka and Kinesis are two popular streaming data platforms that enable real-time data processing. Kafka is an open-source distributed streaming platform that is used for building real-time data pipelines and streaming applications. It is designed to handle high-volume data streams and provides features such as fault-tolerance and scalability. Kinesis, on the other hand, is a … Read more

If you are considering Kafka Certification, this page describes what I did to pass the Confluent Certified Developer for Apache Kafka Certification exam.  You may see it shortened to “ccdak confluent certified developer for apache kafka tests“. Good luck and hopefully this page is helpful for you! There are many reasons why you may wish … Read more

After you start working with Kafka, you will soon find yourself asking the question, “how can I generate test data into my Kafka cluster?”  Well, I’m here to show you have many options for generating test data in Kafka.  In this post and demonstration video, we’ll cover a few of the ways you can generate … Read more

Kafka Producers are one of the options to publish data events (messages) to Kafka topics.  Kafka Producers are custom coded in a variety of languages through the use of Kafka client libraries.  The Kafka Producer API allows messages to be sent to Kafka topics asynchronously, so they are built for speed, but also Kafka Producers have the ability … Read more

In this Kafka Consumer tutorial, we’re going to demonstrate how to develop and run an example of Kafka Consumer in Scala, so you can gain the confidence to develop and deploy your own Kafka Consumer applications.  At the end of this Kafka Consumer tutorial, you’ll have both the source code and screencast of how to … Read more

Kafka Consumer Groups are the way to horizontally scale out event consumption from Kafka topics… with failover resiliency.  “With failover resiliency” you say!?  That sounds interesting.  Well, hold on, let’s leave out the resiliency part for now and just focus on scaling out.  We’ll come back to resiliency later. When designing for horizontal scale-out, let’s … Read more

Apache Kafka offers message delivery guarantees between producers and consumers.  For more background or information Kafka mechanics such as producers and consumers on this, please see Kafka Tutorial page.  Kafka delivery guarantees can be divided into three groups which include “at most once”, “at least once” and “exactly once”. Which option sounds the most appealing?  … Read more