Knowing the Kafka version you are using may not be as straightforward as you might think. For example, if you search for “kafka version” in your favorite search engine or chatbot, there are all kind of results.
But, I took a look at many of the top results and became concerned because the answers provided may lead you in the wrong direction.
Now, the results I saw may not, but it may be confusing to you.
So, I’m going to ensure you know.
This post on showing how to determine Kafka version should help you. If it doesn’t, let me know if you have questions or suggestions in comments below.
This post is especially focused on helping determine the Kafka cluster version when you are not responsible for running the Kafka cluster or using a managed Kafka service. In these cases, you may be attempting to develop or troubleshoot as a client of the Kafka cluster, but you don’t know the Kafka version being used by the cluster.
We’ll cover that and more.
Table of Contents
- Kafka Version Check Background
- The Wrong Way to Kafka Broker Version
- The Correct Way to Determine Kafka Broker Version
- Determine Kafka Client Version the Right Way
- Kafka Version Compatibility
Kafka Version Check Background
Kafka clients, that is producers and consumers, use a specific Kafka version.
Likewise, Kafka brokers will be running a particular version.
The client and broker do not have to be running the same version in most cases.
For example, a client may be running version 3 and the Kafka cluster brokers may be running version 2.
Any client that supports KIP-35 will be able to revert back to older request types or throw appropriate errors if the newer functionality is not available. See the Compatibility Matrix link below for more information.
You may see this functionality of not requiring the same version between client and cluster referred to as “message conversion”.
As you may expect, this message conversion transformation feature in Apache Kafka adds processing overhead in order for messages sent between Kafka clients and brokers to be understood by both parties.
It’s not really the focus of this particular post, but it is possible to monitor the overhead of the message conversion.
See
- MBean: kafka.server:type=BrokerTopicMetrics,name=ProduceMessageConversionsPerSec,topic=([-.\w]+)
- MBean: kafka.network:type=RequestMetrics,name=MessageConversionsTimeMs,request={Produce|Fetch}
The point is there is a cost of version compatibility conversion.
I just needed to be ensure we’re on the same page here before continuing. As you will see in the examples of checking Kafka versions next, there are two types of versions: the client version and the broker version.
If you receive the Kafka client version, but we’re thinking it’s the Kafka broker version, then that’s a one way ticket to TroubleTown. TroubleTown is not a real place by the way. I made it up. But you get the idea, right? Most people do not want to head to a town where there is known trouble. Most people want to avoid TroubleTown. Well, maybe you like to go to TroubleTown from time to time. I get it. Heck, it’s your world. This is your blog post. I’m just here to help you.
If you want to got to TroubleTown kid, you get there, you don’t let anyone stop you. I believe in you. I’m your number one fan in fact. Go get it.
The Wrong Way to Kafka Broker Version
The wrong way to determine Kafka cluster version from the client perspective can be seen in the following two examples.
~ $ ~/dev/kafka_2.13-2.8.1/bin/kafka-broker-api-versions.sh --version --bootstrap-server localhost:9092
2.8.1 (Commit:839b886f9b732b15)This is wrong because I ran this command from 2.8.1 client but I know my Kafka cluster is running 2.4.1
This first example is particularly deceiving for multiple reasons including:
- the name of the shell script itself “kafka-broker-api-versions” implies using it to get the version of the Kafka cluster
- use of the “bootstrap-server” arg implies we will determine the answer from a broker (or, is that just me?)
- the example passes in the “–version” argument. As we will see, the –version arg is a way to be confused in other examples.
Another cited example I found to check Kafka version was to use the –version argument for other Kafka CLI commands such as kafka-topics.sh
Again, there are couple of things to call out here and I’ve tried to make it obvious in the screenshot above.
One, I’m using a different Kafka client version. Two, I’m using the --version argument and passing in the bootstrap-server variable as well.
In this new example, you can see the version returned 3.3.2 is the same as the kafka-topics.sh CLI associated version.
This is exactly like the first example which returned 2.8.1 from the CLI version being used: ~/dev/kafka_2.13-2.8.1/bin/kafka-broker-api-versions.sh
The Correct Way to Determine Kafka Broker Version
If you are running your Kafka cluster, then you are probably not asking the question of what Kafka version. You know it already.
But, if you are Kafka application developer and want to know which version the Kafka cluster version you are using, then how do you do it?
We already saw in the previous “Wrong Way” section, how not to do it, so how do we do it?
Here’s where I need to level with you and tell you something you were not hoping to hear.
There isn’t an easy way.
Now, there’s a way, but I know it’s not going to be as easy you hoped.
But, you’re pretty tough right? You are determined to find solutions and you’ll get over it. Remember I’m your number one fan and pretty much believe in you.
The answer is to use the kafka-broker-api-versions.sh script, use the “bootstrap-server” argument, but do not use the –version argument.
The results will show a map of what is supported and what is not. Use the result to reverse engineer to what version of the Kafka broker is running by knowing what features are supported or not. Let’s use an example.
For example, if I run the following kafka-broker-api-versions.sh from Kafka distribution 2.8.1 to a Kafka Cluster running 2.4.1, I receive the following at the top of the results.
This shows what features are supported by the Kafka brokers. All of the functionality shown in this screenshot above are supported; i.e. “usable” value is equal to the number of features in the category.
Pretty simple, right?
But, if we change our focus to the bottom of the kafka-broker-api-versions.sh results, we’ll see something much different:
It’s here where we can begin to unravel what version of the Kafka broker we are using.
For example, the AlterClientQuotas line shows UNSUPPORTED. To move ahead, we can look to the source code of the particular broker version or the Javadoc of Kafka Admin interface to learn more:
As we can see from above Javadoc on alterClientQuotas, “This operation is supported by brokers with version 2.6.0 or higher”.
So, what does this tell us?
That’s right, it tells us the Kafka cluster version we are using is not 2.6 or above. We know it must be below 2.6.
At this point, this may be enough. Maybe we just needed to know if it was 2.6 or above. However, if we need more precision, we would need to continue to analyze what is supported in what version, but continuing to research the results.
With this approach you know the way to determine which version of Kafka broker is running when you do not have access to run commands on the broker itself or you are not responsible for the Kafka operation side of things.
I warned you, it’s not easy, but it’s the way it is.
You can see it noted the source code itself if you don’t believe me at https://github.com/apache/kafka/blob/3.3/core/src/main/scala/kafka/admin/BrokerApiVersionsCommand.scala#L103 which says
// org.apache.kafka.clients.admin.AdminClient doesn't currently expose a way to retrieve the supported api versions.
I believe there’s a JIRA to change this, but I can’t find it now. That’s ok though because I imagine there will be some security minded folks who think it’s a good idea to NOT expose the API version. I would bet it gets bogged down in debate.
What do you think? Do you think the API version should be exposed by AdminClient?
Actually, don’t answer because honestly, I really don’t care.
Hey, I hope this doesn’t sound too harsh because I’m here to help you for free in this post, but you want honesty when I do and don’t want to hear from you, right? Well, in this case, I don’t want to hear from you. I thought I did, but I changed my mind. When you’re a big time blog tutorial poster like myself, you can change your mind.
Listen, I want you to know I still like you.
But, I mean, common, if I’m being entirely honest, you’re not perfect, you know.
I know sometimes you want to express what’s on your mind, but you need to know when your audience wants to hear your opinion or not.
In this case, like I said, I don’t.
Rub some dirt on it and get over it.
Let’s move on.
Determine Kafka Client Version the Right Way
The answer is probably obvious by now, but just to be thorough, the right way to determine the Kafka client version is to use the –version flag as shown in the previous examples.
The implementation of this functionality has changed over time. For more background on the changes, check out the following Kafka JIRAs
https://issues.apache.org/jira/browse/KAFKA-2061
https://issues.apache.org/jira/browse/KAFKA-8131
I think these two JIRAs describe it all already, so I’m not going to spend time more time on determine Kafka client versions.
I hope this helps, but again, let me know if you have questions or better ideas. That I would like to hear from you.
Kafka Version Compatibility
In the beginning of this post, I mentioned the compatibility matrix. I see this following page often referenced when people are looking for answers on whether or not particular clients will work with Kafka cluster versions.
It’s the only page I know of
https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
If you know of a more convenient way to determine compatibility between Kafka clients and brokers, do let me know.
Thanks and have a great weekend!
