Kafka Monthly Digest: December 2023

December 22, 2023
Mickael Maison

This 71st edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in December 2023. We will also look back at some of the milestones that the Kafka project and community reached over the past year.

For last month’s digest, see Kafka Monthly Digest: November 2023.

Releases

There are 2 new releases 3.6.1 and 3.5.2, and 3.7.0 is still in progress:

3.7.0

The release process for Kafka 3.7.0 continued. Feature freeze happened on December 6 and the 3.7 branch was cut on December 12. The first release candidate should be ready in the next few days. The release date is still targeted for January. You can find the release plan in the wiki.

3.6.1

On December 7, I released Kafka 3.6.1. This bugfix release updates the ZooKeeper and Netty dependencies to address CVEs and also contain ~20 fixes. For more details, check the announcement on the blog and the release plan on the wiki.

3.5.2

On December 11, Luke Chen released Kafka 3.5.2. This bugfix release updates a few dependencies to address CVEs and also includes 39 fixes. You can find the announcement on the blog and the release plan on the wiki.

Kafka Improvement Proposals

Last month, the community submitted 4 KIPs (KIP-1010 to KIP-1013). I'll highlight a few of them:

Community Releases

  • kroxylicious 0.4.0: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. This first release includes a preview of an EnvelopeEncryption filter which provides transparent encryption at rest.
  • Debezium 2.5: Debezium is a Change Data Capture platform. The new release includes improvements in all connectors. It introduces a new connector for IBM Informix, and brings preview support for MariaDB with the goal to bring it in line with the MySQL connector. There are also a few breaking changes that are worth checking before you upgrade.
  • strimzi-kafka-operator 0.39: Strimzi is a Kubernetes Operator for running Kafka. This release adds support for Kafka 3.5.2 and 3.6.1. The StableConnectIdentities, KafkaNodePools and UnidirectionalTopicOperator feature gates are now GA and enabled by default. The example Grafana dashboards for Kafka Connect and MirrorMaker have also been updated. Finally the Strimzi project introduced a new component, strimzi-mqtt-bridge, which provides an MQTT bridge to Apache Kafka.

Blogs

I selected some interesting blog articles that were published last month:

Project Milestones in 2023

As the year concludes, let's look at some of the milestones Apache Kafka achieved in 2023.

Releases in 2023

The project followed its time-based release plan. Consequently, it released three minor versions: 3.4, 3.5 and 3.6 as well as five bugfix releases (3.3.2, 3.4.1, 3.5.1, 3.5.2 and 3.6.1). Figure 1 shows the timeline for these releases.

Figure 1. Releases in 2023.
Figure 1. Releases in 2023.

KIPs

In the past 12 months the community raised 116 KIPs, crossing the 1000th KIP mark in November. Figure 2 shows how this compares to previous years.

KIPs created over the past years
Figure 2. KIPs created over the past years.

Code and contributors

This year 59 unique contributors made more than 1100 commits. Figure 3 shows the size of the code base, in lines of code, for a few releases.

Lines of codes across Apache Kafka releases
Figure 3. Lines of code per Kafka release.

Committers and Project Management Committee members

In 2023, ten contributors were invited to become Committers:

  • Satish Duggana
  • Justine Olshan
  • Edoardo Comar
  • Walker Carlson
  • Stanislav Kozlovski
  • Lucas Bradstreet
  • Divij Vaidya
  • Greg Harris
  • Lucas Brutschy
  • Yash Mayya

Likewise, five Committers also joined the Apache Kafka PMC:

  • Chris Egerton
  • David Arthur
  • Justine Olshan
  • Satish Duggana
  • Divij Vaidya

In June, Jun Rao stepped down from his role as chair of the Apache Kafka PMC. I was elected by the PMC to become the new chair.

The current roster of Committers and PMC members is available on the Kafka website.

What's coming for Kafka in 2024

Last year I predicted that Kafka in 2023 would get support for Tiered Storage, have a migration process from ZooKeeper to KRaft and have a new consumer rebalance protocol. Tiered Storage finally landed in early access in 3.6.0, migrating from ZooKeeper to KRaft initially landed in 3.4.0 and was marked as production ready in 3.6.0. A lot of work has happened on the new consumer rebalance protocol but it's still not available.

I'll play the game again and here are my predictions for 2024 (this is my personal point of view and does not represent Apache Kafka nor my employer)

  • Kafka 4.0: KIP-500 which proposed replacing ZooKeeper was voted over 4 years ago. Changing the underlying metadata coordination layer is definitively not a simple task! At last it seems all the requirements to fully remove ZooKeeper are almost there and I expect Kafka 4.0 to release this year. It will be interesting to see how many lines of codes this will remove from the code base.
  • KIP-848: The Next Generation of the Consumer Rebalance Protocol: Looking at KAFKA-14048 over two thirds of the tasks are now completed so it's likely the new consumer rebalance protocol will be introduced in 2024.
  • Removal of Scala code: Kafka was initially mostly written in Scala. Over the years every time a new module was created, the community used Java. Scala now accounts for less than 20% of the code base. I expect Kafka to still have some Scala in the next few years but with the removal of ZooKeeper and work on-going to split the last Scala module, the percentage of Scala in the code base may drop into single digits in 2024.

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Last updated: January 2, 2024
Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.