Kafka Consumer Groups with kafka-consumer-groups.sh

Kafka Consumer Groups Operation Examples

How do Kafka administrators perform administrative and diagnostic collection actions of Kafka Consumer Groups?  This post explores a Kafka Groups operations admin tool called kafka-consumer-groups.sh.  This popular, command-line tool included in Apache Kafka distributions.  There are other examples of both open source and 3rd party tools not included with Apache Kafka which can also be […]

How To Generate Kafka Streaming Join Test Data By Example

Kafka Joins Test Data

Why “Joinable” Streaming Test Data for Kafka? When creating streaming join applications in KStreams, ksqldb, Spark, Flink, etc. with source data in Kafka, it would be convenient to generate fake data with cross-topic relationships; i.e. a customer topic and an order topic with a value attribute of customer.id.  In this example, we might want to […]

Kafka Certification Tips for Developers

Kafka Certification Tips

If you are considering Kafka Certification, this page describes what I did to pass the Confluent Certified Developer for Apache Kafka Certification exam.  Good luck and hopefully this page is helpful for you! There are many reasons why you may wish to become Kafka certified. My number 1 recommendation to help you become Kafka Developer […]

Kafka Test Data Generation Examples

Kafka Test Data Generation

After you start working with Kafka, you will soon find yourself asking the question, “how can I generate test data into my Kafka cluster?”  Well, I’m here to show you have many options for generating test data in Kafka.  In this post and demonstration video, we’ll cover a few of the ways you can generate […]

Kafka Producer

Kafka Producer Example

Kafka Producers are one of the options to publish data events (messages) to Kafka topics.  Kafka Producers are custom coded in a variety of languages through the use of Kafka client libraries.  The Kafka Producer API allows messages to be sent to Kafka topics asynchronously, so they are built for speed, but also Kafka Producers have the ability […]

Kafka Consumer

Kafka Consumer

In this Kafka Consumer tutorial, we’re going to demonstrate how to develop and run an example of Kafka Consumer, so you can gain the confidence to develop and deploy your own Kafka Consumer applications.  At the end of this Kafka Consumer tutorial, you’ll have both the source code and screencast of how to run and […]

Kafka Consumer Groups by Example

Kafka Consumer Group Example

Kafka Consumer Groups are the way to horizontally scale out event consumption from Kafka topics… with failover resiliency.  “With failover resiliency” you say!?  That sounds interesting.  Well, hold on, let’s leave out the resiliency part for now and just focus on scaling out.  We’ll come back to resiliency later. When designing for horizontal scale-out, let’s […]

Apache Kafka Architecture – Delivery Guarantees

Kafka architecture delivery guarantees

Apache Kafka offers message delivery guarantees between producers and consumers.  For more background or information Kafka mechanics such as producers and consumers on this, please see Kafka Tutorial page.  Kafka delivery guarantees can be divided into three groups which include “at most once”, “at least once” and “exactly once”. at most once which can lead to […]

Kafka vs Amazon Kinesis – How do they compare?

apache kafka

Apache Kafka vs. Amazon Kinesis The question of Kafka vs Kinesis often comes up.  Let’s start with Kinesis. *** Updated Spring 2020 *** Since this original post, AWS has released MSK.  I think this tells us everything we need to know about Kafka vs Kinesis.  Also, since the original post, Kinesis has been separated into […]