Kafka Monthly Digest: October 2024

This 81st edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in October 2024.

For last month’s digest, see Kafka Monthly Digest: September 2024.

Releases

There is 1 new release and 3 releases are in progress:

3.8.1

On October 29, Josep Prat released Kafka 3.8.1. This bugfix release contains 17 fixes including a few blocker issues and several dependency updates. For more details, you can check the release announcement on the blog and the release notes.

3.9.0

On October 10, Colin McCabe publish 3.9.0 RC2 but a number of issues (KAFKA-17788, KAFKA-17031, KAFKA-17794) were quickly found by the community. RC3 was then published on October 21 but an issue in the kafka-storage tool (KAFKA-17868) was found. There was an issue publishing RC4, so RC5 was published on October 27. The vote passed so 3.9.0 will be released in the next few days.

4.0.0

Progress on Kafka 4.0.0 continued. The next milestone is KIP freeze on November 20. You can find the release plan in the wiki.

3.7.2

On October 10, Matthias Sax volunteered to be the release manager for Kafka 3.7.2. Since 3.7.1, 15 Jiras have been fixed including several blocker issues and dependency updates to address CVEs.
 

Kafka Improvement Proposals

Last month, the community submitted 10 KIPs (KIP-1096 to KIP-1105). I'll highlight a few of them:

• KIP-1097: Error record reporter in Connect: When Connect fails processing a record, it relies on the configured errors.tolerance behavior to handle the failure. It can either skip the record and carry on, or fail the task and stop processing. Additionally Connect can log the failure and the record, and for sink connectors send the record to a dead letter queue topic. This KIP proposes making the error reporting logic pluggable so users can implement their own logic when a record is unprocessable.
   
• KIP-1102: Enable clients to rebootstrap based on timeout or error code: In Kafka 3.8.0 KIP-889 enabled Kafka clients to rebootstrap in case all the brokers in their current metadata are unavailable. However the conditions this mechanism can trigger are very limited. This KIP aims at making this mechanism more effective by enlarging the conditions that trigger rebootstrapping and enabling that feature by default.
   
• KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins: When doing joins in a Kafka Streams application, the foreign key extractor function currently only receives the record value as well as the key of other KTable or KStream. This KIP proposes adding the record key as well, so the foreign key can be derived from both the record key and value.

Community Releases

• strimzi-kafka-operator 0.44: Strimzi is a Kubernetes Operator for running Kafka. Strimzi now supports managing the offsets of KafkaConnector and KafkaMirrorMaker2 resources. Strimzi can now also perform automatic rebalancing, using Cruise Control, when clusters are scaled up or down.
   
• Debezium 3.0: Debezium is a Change Data Capture platform. This new major release is now based on Kafka 3.8.0 and requires Java 17. A new major version also means there are a few breaking changes, and a number of deprecated configurations and options have been removed, so check the release notes before upgrading. This release includes many new features and improvements in all connectors.
   
• Librdkafka 2.6: Librdkafka is a Kafka client in C/C++. This new release adds support for sending producer and consumer client metrics to brokers (KIP-714). The Admin client can now also trigger leader elections (KIP-460).

Blogs

I selected some interesting blog articles that were published last month:

• The Schema Registry API is not how you use schema with Kafka!
• Naming Kafka objects (III) – Kafka Connectors
• Preparing the Consumer Fetch: Kafka Producer and Consumer Internals, Part 3
• Revisiting the Outbox Pattern

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.