How do Kafka administrators perform administrative and diagnostic collection actions of Kafka topics?
This post explores a Kafka topic admin tool called kafka-topics.sh. This command-line tool included in Apache Kafka distributions. There are other examples of both open source and 3rd party tools which can also be used for Kafka topic administrative tasks, but for this post, let’s focus on kafka-topics.sh. The concepts are what matter because they can easily be applied to other tools.
Table of Contents
- What is kafka-topics.sh?
- Why kafka-topics.sh?
- Where is kafka-topics.sh?
- Is there a Windows version ok kafka-topics.sh?
- Is there a Confluent version of kafka-topics.sh?
- Kafka Topic Operations Examples
- Kafka Topic Create Example
- Kafka List Topic(s) Example
- Kafka Delete Topic Example
- Kafka Alter Topic Example
- Kafka Describe Topic Example
- Conclusion
- Resources
What is kafka-topics.sh?
As briefly mentioned above, kafka-topics.sh is a command-line tool included with Apache Kafka used to manage Kafka topics. I will show exempls of using it in this post to create, list, describe, delete, and modify topics in a Kafka cluster. The tool is typically used by system administrators or developers who need to manage Kafka topics from the command line, but understanding it will also be helpful if you use a GUI tool as well.
As hopefully you already know by now, Kafka topics are fundamental to Kafka’s messaging system, so it’s critical to know the kafka topic options.
Why kafka-topics.sh?
kafka-topics.sh is a useful tool for managing Apache Kafka topics. It provides a way to perform common tasks related to Kafka topics.
Some reasons why to use kafka-topics.sh include:
- Creating new topics: kafka-topics.sh can be used to create new Kafka topics with specific configurations, such as the number of partitions and replication factor.
- Listing topics: kafka-topics.sh can be used to list all the topics that currently exist in a Kafka cluster.
- Describing topics: kafka-topics.sh can be used to display detailed information about a specific topic, such as the number of partitions, replication factor, and configuration settings.
- Modifying topics: kafka-topics.sh can be used to modify the configuration of an existing topic, such as changing the number of partitions or replication factor.
- Deleting topics: kafka-topics.sh can be used to delete a topic from a Kafka cluster.
Overall, kafka-topics.sh is a tool to help manage Kafka topics so use it is used in Kafka adminstration.
Where is kafka-topics.sh?
The kafka-topics.sh script is included with Apache Kafka and is typically located in the bin directory of the Kafka installation.
The exact location may vary depending on how Kafka was installed and configured on your system.
In a default installation of Kafka, the kafka-topics.sh script can be found in the bin directory of the Kafka installation folder. For example, if you installed Kafka in /usr/local/kafka, then you can find kafka-topics.sh in /usr/local/kafka/bin/kafka-topics.sh.
You can also run the which kafka-topics.sh command in the terminal to find the location of the script if it is in your system’s PATH.
Is there a Windows version ok kafka-topics.sh?
There is a Windows version of the kafka-topics.sh script, which is called kafka-topics.bat. It is included in the Apache Kafka distribution for Windows.
You can find the kafka-topics.bat script in the bin\windows directory of the Kafka installation folder. For example, if you installed Kafka in C:\kafka, then you can find kafka-topics.bat in C:\kafka\bin\windows\kafka-topics.bat.
The kafka-topics.bat script provides similar functionality to the kafka-topics.sh script on Linux and other Unix-based systems. You can use it to create, list, describe, modify, and delete Kafka topics on Windows.
Is there a Confluent version of kafka-topics.sh?
Yes and no. This has changed over time, but the future intention is becoming more clear.
In early 2023, Confluent announced the Confluent CLI at https://www.confluent.io/blog/announcing-the-source-available-confluent-cli/
Kafka Topic Operations Examples
1. Kafka Topic Operations Examples Environment Setup
To cover the various examples of Kafka topic administrative tasks, we need Kafka cluster. Shocking, I know. There are various ways we can do this, but let’s go with the most simple which for me, means running a Kafka cluster in docker using docker-compose.
If you want to follow along with examples below, here are the summarized requirements and steps.
2. Requirements
- Download and extract Apache Kafka so we have access to
kafka-topics.shcommand in the bin directory (Verification will be shown below). Any version will do for now. - Start Docker (if it is not already running)
- git clone https://github.com/supergloo/kafka-examples/
- cd kafka-examples/kafka-topic
- docker-compose -f kafka-topic-examples.yml up -d (wait 5-30 seconds for everything to come online)
Here’s terminal output from starting up the Kafka docker containers in step 5:
$ docker-compose -f kafka-topic-examples.yml up -d
Creating network "kafka-topic_default" with the default driver
Pulling kafka (wurstmeister/kafka:2.13-2.8.1)...
2.13-2.8.1: Pulling from wurstmeister/kafka
Digest: sha256:2d4bbf9cc83d9854d36582987da5f939fb9255fb128d18e3cf2c6ad825a32751
Status: Downloaded newer image for wurstmeister/kafka:2.13-2.8.1
Creating kafka-topic_zookeeper_1 ... done
Creating kafka-topic_kafka_1 ... done
$ pwd
/Users/toddmcg/dev/kafka-examples/kafka-topicAlso, I have downloaded and extracted Apache Kafka into my ~/dev directory in a different terminal window.
$ pwd
/Users/toddmcg/dev/kafka_2.13-2.8.1/binOk, at this point, we have two terminal windows with a running Kafka cluster in one and access to Kafka CLI bin directory in the other. We are ready to start running some examples.
We are going to run kafka-topics.sh from the second terminal window with a bootstrap-server setting to the Kafka cluster running in first terminal if this isn’t completely obvious. Again, there is more than one way to do this, but since I am the big shot blog author, I get to choose to do it this way.
Here’s a quick video of me showing the setup described above.
Special note: the examples below will use no authentication or authorization when connecting to the cluster. See Kafka authentication tutorial if interested in learning more about Kafka authentication options and examples and perhaps follow it up with the Kafka authorization examples.
Kafka Topic Create Example
Let’s start at the beginning and show how to create a Kafka topic:
$ ./kafka-topics.sh --create --topic first_topic --bootstrap-server localhost:9092 --partitions 9 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic first_topic.Ok, easy enough we created a topic named first_topic and were warned about Kafka naming.
A couple of notes on the options supplied in addition to –create.
The –partitions and –replication-factor arguments are optional. If not provided, the default values configured in your Kafka installation will be used. For example, the default value for partitions is 1 in this environment.
Wondering what your existing topic partitions or replication factor (RF) are set to? We’ll show the –describe option below which can help.
Also, a note on the simple Kafka cluster we are using in the examples. It’s just a single Kafka broker, so if we try to create a RF > 1, there will be an error such as this:
$ ./kafka-topics.sh --create --topic first_topic --bootstrap-server localhost:9092 --partitions 9 --replication-factor 2
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Replication factor: 2 larger than available brokers: 1.
[2023-06-29 08:41:21,522] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.
(kafka.admin.TopicCommand$)Kafka List Topic(s) Example
Now that we have a topic how do we list the existing Kafka topics? Easy, pass in the –list option.
$ ./kafka-topics.sh --list --bootstrap-server localhost:9092
first_topic
This doesn’t require any explanation, but at this point, it may be helpful to confirm you can get a list of all available options and brief description of each with the –help option
$ ./kafka-topics.sh --create --help
This tool helps to create, delete, describe, or change a topic.
Option Description
------ -----------
--alter Alter the number of partitions,
replica assignment, and/or
configuration for the topic.
...Kafka Delete Topic Example
Run the following command to delete a topic:
$ ./kafka-topics.sh --delete --topic first_topic --bootstrap-server localhost:9092
$ ./kafka-topics.sh --list --bootstrap-server localhost:9092
$Note: a commonly asked question when deleting a topic is that the topic and associated files are not immediately deleted after running the command. (The example above did immediately delete, so not applicable.)
Two things to note:
- It can take some time based on
file.delete.delay.msconfig setting to remove the topic files from the file system. This can be frustrating, for example, if your disk(s) are filled and you are looking to get space back quickly - If your topic is not deleting, then ensure
delete.topic.enableconfig setting is set to true. See Kafka Config tutorial for more on Kafka configuration.
Kafka Alter Topic Example
Use the –alter option to modify an existing Kafka topic.
For example, let’s create a new topic, bypassing the optional arguments, and then change the number of partitions
$ ./kafka-topics.sh --create --topic second_topic --bootstrap-server localhost:9092
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic second_topic.
$ ./kafka-topics.sh --describe --topic second_topic --bootstrap-server localhost:9092
Topic: second_topic TopicId: RDPCjdmtRrij_jpKVlZeQQ PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: second_topic Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
$ ./kafka-topics.sh --alter --topic second_topic --bootstrap-server localhost:9092 --partitions 3
$ ./kafka-topics.sh --describe --topic second_topic --bootstrap-server localhost:9092
Topic: second_topic TopicId: RDPCjdmtRrij_jpKVlZeQQ PartitionCount: 3 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: second_topic Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: second_topic Partition: 1 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: second_topic Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001
$If not obvious, we changed the partitions from 1 to 3 in the above example.
Note that altering the topic configurations, such as the number of partitions or replication factor, may have implications on the data distribution, performance, and availability of the topic. Make sure to understand the impact of these changes and plan accordingly before altering the topic configurations in a production environment.
Kafka Describe Topic Example
We saw use of it the previous example, but to view the settings of an existing topic, use the –describe option.
$ ./kafka-topics.sh --describe --topic second_topic --bootstrap-server localhost:9092
Topic: second_topic TopicId: RDPCjdmtRrij_jpKVlZeQQ PartitionCount: 3 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: second_topic Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: second_topic Partition: 1 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: second_topic Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001Conclusion
Hope you found this exploration of Kafka Topic administration with kafka-topics.sh helpful. If you have any suggestions for improvements or if you’d like to see any other examples or approaches using different tools, let me know in the comments below.
