Kafka Java Clients: Spring Kafka vs Spring Cloud Stream

The Java Kafka libraries in this guide all build on Apache's official kafka-clients library. What differs is how much polling, offset management, and recovery a framework handles, and whether the application needs basic message consumption or stateful stream processing with windows, joins, and recoverable aggregates.

This guide compares the Apache Kafka Java client, Spring Kafka, and Spring Cloud Stream, plus Kafka connectors for Akka and Pekko, focusing on consumer models, retries, transactions, and serialization. For Kafka Streams, see Kafka Streams in Java.

Quick decision

Kafka clients or Kafka Streams?

Use a messaging client when records trigger application work: send a notification, update an order, or call another service. Your application owns the business logic and any database state; the client or framework handles communication with Kafka.

Choose Kafka Streams when you need managed state for operations such as five-minute sales totals, joins between orders and payments, or running inventory counts. It runs as a library inside your application. You define a topology—a graph of processing steps—and Kafka Streams manages the processing tasks and state recovery.

Alpakka and Pekko Kafka connectors belong with the messaging clients. They connect Kafka to an Akka or Pekko Streams pipeline so downstream stages can limit incoming work with backpressure. They do not provide Kafka Streams' managed state stores or recovery for windowed aggregates.

Kafka Java clients compared

Capabilities checked against official documentation on 7 Sep 2026; runtime requirements depend on the release line you select.

A dead-letter topic (DLT) holds records routed away after processing fails.

All these options can use Schema Registry by adding serializer dependencies and registry configuration. For example, Confluent's Java serializers add Avro, Protobuf, and JSON Schema support.

Apache Kafka Java client

The official org.apache.kafka:kafka-clients artifact provides KafkaProducer, KafkaConsumer, and Admin. It gives your application direct access to Kafka configuration, partition assignment, offsets, and transactions.

Capabilities and limitations

• Consumer ownership: Your code runs the polling loop and coordinates processing with commits. KafkaConsumer is not thread-safe; moving work to an executor requires tracking completion before committing offsets. Long processing can exceed max.poll.interval.ms and trigger reassignment. See the consumer API.
• Recovery: Client retries handle communication failures. Retrying business logic, publishing failed records to a DLT, and deciding when to skip a record remain application responsibilities.
• Runtime: Kafka 4.0 raised the minimum Java version for clients and Streams to 11. Broker requirements are different, as the compatibility matrix shows; check the Java requirement for your client release separately.

When to choose: Use the Apache client for a service without Spring, a reusable integration library, or a consumer that needs custom polling and commit behavior. Choose Spring Kafka if you would otherwise build listener management and recovery infrastructure inside a Spring application.

Spring Kafka

Spring Kafka adds KafkaTemplate for sending and listener containers for consuming. With @KafkaListener, the framework handles polling and invokes your method with a record or batch.

Capabilities and limitations

• Recovery tools: DefaultErrorHandler configures retries, and DeadLetterPublishingRecoverer can route failed records to a DLT. Use ErrorHandlingDeserializer to pass deserialization failures to the container's error handler. Transactional containers use rollback processing. See exception handling.
• Retry trade-offs: Non-blocking retries move records through retry topics, allowing later records to proceed. They do not preserve the original processing order and cannot be combined with container transactions or batch listeners.
• Version alignment: The current quick tour specifies Java 17 as the minimum. In Spring Boot projects, use Boot's dependency management to select compatible Spring Kafka and Kafka client versions.

When to choose: Choose Spring Kafka for Spring services that need explicit control over listener concurrency, acknowledgment, and failure recovery. Its container configuration is usually a better fit than Spring Cloud Stream when Kafka-specific behavior is central to the application.

Spring Cloud Stream Kafka binder

Spring Cloud Stream connects application functions to messaging systems through adapters called binders. Its Kafka binder handles messaging; its Kafka Streams binder runs Kafka Streams topologies. They are separate dependencies with different programming and configuration models.

With spring-cloud-stream-binder-kafka, you define Supplier, Consumer, or Function beans and configure their destinations and consumer groups. The binder uses Spring Kafka underneath. See the functional programming model.

Capabilities and limitations

• Configuration: Bindings centralize topics, groups, concurrency, and message conversion. Kafka-specific behavior still needs Kafka-specific settings; changing binders does not make partitioning or delivery semantics interchangeable.
• Failures and transactions: The binder offers retry and dead-letter configuration. With the transactional binder, retry and recovery use the listener container's rollback handling instead of ordinary binder retries.
• Serialization: Decide whether Spring message conversion or Kafka's native serializers own the payload format. For an existing Schema Registry contract, use matching native serializers and deserializers rather than assuming JSON conversion produces the same bytes.

When to choose: Choose the Kafka binder when your team already organizes services around Spring Cloud Stream function bindings. Choose Spring Kafka when direct listener configuration is clearer than another configuration layer.

Alpakka Kafka and Apache Pekko Kafka Connector

Alpakka Kafka integrates with Akka Streams; Apache Pekko Kafka Connector provides the corresponding integration for Pekko Streams. Both offer Java APIs for composing Kafka records with asynchronous processing stages.

Capabilities and limitations

Committable sources let processing carry an offset through the pipeline and commit after work completes. Transactional APIs support Kafka-to-Kafka processing; Pekko documents this in its transaction guide. Application failures still need a recovery policy, including what happens to unfinished records when a stream stops.

Neither requires writing business logic as actors, but both introduce their stream runtime and dependencies.

When to choose: Use the connector matching an existing Akka or Pekko application. For a new pipeline where Apache-2.0 licensing is required, consider Pekko; current Alpakka Kafka releases use Business Source License 1.1. Adding an Akka or Pekko Streams runtime solely for a straightforward Kafka listener is usually unnecessary.

Stream processing in Java

Kafka Streams can run directly, in Spring Boot through Spring Kafka, through the Spring Cloud Stream Kafka Streams binder, or with the Quarkus extension. Kafka Streams in Java compares them and covers versions, state stores, and testing.

Consider Flink when the team wants streaming jobs managed by a dedicated processing runtime, particularly pipelines combining Kafka with other sources and sinks. Consider Spark Structured Streaming when the workload fits an existing Spark SQL or DataFrame pipeline. Both introduce a separate processing runtime to deploy and operate; Kafka Streams fits when the processing should run as part of a Java service.

Test Java Kafka clients with Kafma

Spring message conversion, native serializers, and type headers can change what a consumer receives. Retry and dead-letter settings determine what happens when processing fails.

Kafma is a desktop Kafka GUI client that runs alongside your Java application. Use it to inspect records from your producer or Kafka Streams output topic, then send test records to your running consumer.

Inspect serialized records, schemas, and Spring headers

Send a record from your application to a test topic, or open your Kafka Streams output topic in Kafma. Inspect the key, value, and headers. With Schema Registry configured, Kafma automatically decodes records using the standard Confluent schema-ID payload format. Check the schema ID and switch between Decoded and Raw to compare the readable value with its bytes.

For Spring JSON messages, compare __TypeId__ and related headers with your consumer's target type or type mappings. Missing headers are not necessarily an error: the deserializer can use a configured type and ignore headers. See Spring's serialization options.

Send test messages to your consumer

Keep your consumer running—for example, the service containing your @KafkaListener—and use Kafma's producer panel to send a valid record to its test topic. Then send one with a payload or type header that does not match your consumer configuration, followed by another valid record. Send all three records to the same partition.

To test invalid JSON, select String encoding for the value and send malformed JSON text.

Check your application logs and configured dead-letter topic. Is the failed record retried, routed to that topic, or stopping processing? Does the next valid record get processed? Compare the result with your error-handling configuration; a deserialization failure may occur before your listener runs.

For a stop-and-restart test with messages arriving, use Loop to send records at a fixed interval and observe how your consumer handles unfinished work.

Clone data for testing

Hand-written test data can miss older schema versions, missing headers, and tombstones with null values. Kafma Data Clone copies a selected message range to a test topic, with optional topic creation, schema registration, and field masking. Free clones one topic per run, within a cluster or between clusters; cloning several topics in one run requires Pro.

Use the copy to test your consumer's deserialization and message handling. Copied records receive new offsets and may use different partitions; consumer-group offsets are not copied.

Download Kafma and connect to the same test cluster as your Java application.

Frequently asked questions

Does Confluent provide its own Java Kafka client?

Confluent's Java client documentation uses Apache's KafkaProducer and KafkaConsumer. For standard Java messaging, you use org.apache.kafka:kafka-clients; Confluent adds components such as Schema Registry clients and Avro, Protobuf, and JSON Schema serializers. You do not need a separate Confluent producer/consumer API to connect to Confluent Cloud.

Is Reactor Kafka still maintained?

Spring announced the discontinuation of Reactor Kafka in May 2025. For a new Spring service, start with Spring Kafka. Teams maintaining Reactor Kafka applications should evaluate migrating to Spring Kafka and review processing completion, backpressure, and commits when changing APIs. Using WebFlux for HTTP endpoints does not by itself require a reactive Kafka client.

Which Java Kafka clients support the KIP-848 consumer group protocol?

Apache Kafka's Java consumer supports KIP-848, generally available with Kafka 4.0. Enable it with group.protocol=consumer on compatible clients and brokers. Spring Kafka supports it from version 4.0 onward.

For Spring Cloud Stream and Akka/Pekko connectors, check both the resolved Kafka client version and the framework's compatibility guidance. Depending on kafka-clients alone is not enough to establish that every wrapper feature works with the new protocol. Assignment and heartbeat settings also change; use Apache's upgrade guidance when enabling it.

Can producers and consumers use different Java Kafka libraries?

Yes. They must agree on key and value serialization, required headers, and Schema Registry framing. A Spring JSON consumer must resolve type headers to the intended Java class, or ignore them and use a configured target type.

Consumers sharing a group also need compatible group protocols and assignment behavior.

Conclusion

For most new Java services, start with Spring Kafka on Spring Boot, or the Apache Kafka Java client when you want direct control without a messaging framework. For windows, joins, or recoverable state, use Kafka Streams. Use Spring Cloud Stream when your application already uses function bindings, or an Akka/Pekko connector when it already uses the corresponding Streams runtime.

This guide is maintained by the team behind Kafma.