How To Kafka Configuration with kafka-configs.sh [Tutorial with 4 Examples]

Kafka Configuration Tutorial

Apache Kafka includes a command-line tool named kafka-configs.sh used to obtain configuration values for various types of entities such as topics, clients, users, brokers, and loggers. But, using this tool to determine current configuration values at runtime can be more difficult to use than you might expect. This can be especially true if you want […]

Best Ways to Determine Apache Kafka Version [1 Right and 2 Wrong Ways]

How to Determine Kafka Version the Right Way

Knowing the Kafka version you are using may not be as straightforward as you might think. For example, if you search for “kafka version” in your favorite search engine or chatbot, there are all kind of results. But, I took a look at many of the top results and became concerned because the answers provided […]

Easy Kafka ACL (How To Implement Kafka Authorization)

Kafka Authorization Examples

Kafka Access Control Lists (ACLs) are a way to secure your Kafka cluster by specifying which users or client applications have access to which Kafka resources (topics, clusters, etc.). The process of authorizing or refusing access to particular resources or functions within a software application is referred to as “authorization” in software. It is the […]

Python Kafka in Two Minutes. Maybe Less.

Python Kafka Tutorial

Although Apache Kafka is written in Java, there are Python Kafka clients available for use with Kafka.  In this tutorial, let’s go through examples of Kafka with Python Producer and Consumer clients.  Let’s consider this a “Getting Started” tutorial.  After completing this, you will be ready to proceed to more complex examples.  But we need […]

Kafka and Dead Letter Queues Support? Yes and No

kafka-dead-letter-queues

In this post, let’s answer the question of Kafka and Dead Letter Quest. But first, let’s start with an overview. A dead letter queue (DLQ) is a queue, or a topic in Kafka, used to hold messages which can not be processed successfully. The origin of DLQs are traditional messaging systems which were popular before […]

Effortless Kafka Authentication Tutorial (with 5 Examples)

Kafka Authentication Options

Kafka provides multiple authentication options.  In this tutorial, we will describe and show the authentication options and then configure and run a demo example of Kafka authentication. There are two primary goals of this tutorial: There are a few key subjects which must be considered when building a multi-tenant cluster, but it all starts with […]

Kafka Consumer Groups with kafka-consumer-groups.sh

Kafka Consumer Groups Operation Examples

How do Kafka administrators perform administrative and diagnostic collection actions of Kafka Consumer Groups?  This post explores a Kafka Groups operations admin tool called kafka-consumer-groups.sh.  This popular, command-line tool included in Apache Kafka distributions.  There are other examples of both open source and 3rd party tools not included with Apache Kafka which can also be […]

How To Generate Kafka Streaming Join Test Data By Example

Kafka Joins Test Data

Why “Joinable” Streaming Test Data for Kafka? When creating streaming join applications in KStreams, ksqldb, Spark, Flink, etc. with source data in Kafka, it would be convenient to generate fake data with cross-topic relationships; i.e. a customer topic and an order topic with a value attribute of customer.id.  In this example, we might want to […]

Kafka Certification Tips for Developers

Kafka Certification Tips

If you are considering Kafka Certification, this page describes what I did to pass the Confluent Certified Developer for Apache Kafka Certification exam.  You may see it shortened to “ccdak confluent certified developer for apache kafka tests“. Good luck and hopefully this page is helpful for you! There are many reasons why you may wish […]

Kafka Test Data Generation Examples

Kafka Test Data Generation

After you start working with Kafka, you will soon find yourself asking the question, “how can I generate test data into my Kafka cluster?”  Well, I’m here to show you have many options for generating test data in Kafka.  In this post and demonstration video, we’ll cover a few of the ways you can generate […]

Kafka Producer in Scala

Kafka Producer Example

Kafka Producers are one of the options to publish data events (messages) to Kafka topics.  Kafka Producers are custom coded in a variety of languages through the use of Kafka client libraries.  The Kafka Producer API allows messages to be sent to Kafka topics asynchronously, so they are built for speed, but also Kafka Producers have the ability […]

Kafka Consumer in Scala

Kafka Consumer

In this Kafka Consumer tutorial, we’re going to demonstrate how to develop and run an example of Kafka Consumer in Scala, so you can gain the confidence to develop and deploy your own Kafka Consumer applications.  At the end of this Kafka Consumer tutorial, you’ll have both the source code and screencast of how to […]

Kafka Consumer Groups by Example

Kafka Consumer Group Example

Kafka Consumer Groups are the way to horizontally scale out event consumption from Kafka topics… with failover resiliency.  “With failover resiliency” you say!?  That sounds interesting.  Well, hold on, let’s leave out the resiliency part for now and just focus on scaling out.  We’ll come back to resiliency later. When designing for horizontal scale-out, let’s […]

Apache Kafka Architecture – Delivery Guarantees

Kafka architecture delivery guarantees

Apache Kafka offers message delivery guarantees between producers and consumers.  For more background or information Kafka mechanics such as producers and consumers on this, please see Kafka Tutorial page.  Kafka delivery guarantees can be divided into three groups which include “at most once”, “at least once” and “exactly once”. at most once which can lead to […]

Kafka vs Amazon Kinesis – How do they compare?

apache kafka

Apache Kafka vs. Amazon Kinesis The question of Kafka vs Kinesis often comes up.  Let’s start with Kinesis. *** Updated Spring 2020 *** Since this original post, AWS has released MSK.  I think this tells us everything we need to know about Kafka vs Kinesis.  Also, since the original post, Kinesis has been separated into […]