Kafka Streams in Java: Spring Boot and Spring Cloud Stream

Kafka Streams is a Java library that processes Kafka records inside your application, with no separate processing cluster. It handles aggregations, windows, and joins, and keeps their state in local stores that can be recovered from Kafka.

Use Kafka Streams directly, or through Spring Boot with Spring Kafka, the Spring Cloud Stream Kafka Streams binder, or the Quarkus extension. The three integrations wrap the same library but differ in configuration, lifecycle, serialization defaults, and error handling.

This guide compares these options, then covers Maven dependencies and version alignment, state store recovery, and testing with and without a broker. For plain producers and consumers, see Kafka Java clients.

Quick decision

Kafka Streams integrations compared

Capabilities verified against each release's source and documentation: 19 Oct 2026

A Serde pairs a serializer with a deserializer for a key or value type.

Every column gets Kafka Streams' state stores, windows, joins, and exactly-once processing. Which features are available, and how they behave, depends on the Kafka Streams version each one resolves and on its configuration.

The sections below use a running example: count orders per customer ID and keep the counts in a state store named order-counts.

Apache Kafka Streams in plain Java

The org.apache.kafka:kafka-streams library has two APIs. The DSL covers filtering, grouping, counting, windows, and joins; the Processor API lets you write custom processors that use state stores directly. One topology can mix both.

For the order count, you group the orders stream by key, count it into order-counts, and write the changes to an output topic. Your code then creates KafkaStreams from the topology and its Properties, starts it, and closes it on shutdown.

Capabilities and limitations

• Lifecycle: KafkaStreams runs its own stream threads. Register an uncaught exception handler to choose whether a failed thread is replaced or the client shuts down, and close the instance on shutdown so local state is flushed.
• Default failures: Deserialization and processing errors shut down the local KafkaStreams instance by default. Use the LogAndContinue handlers to skip failed records instead.
• Native DLQ: From Kafka 4.2, errors.dead.letter.queue.topic.name makes the built-in handlers also send failed records to a dead-letter topic. Custom handlers must implement DLQ support explicitly.
• Versions: You pick the Kafka Streams version directly, with no framework dependency management in between.

When to choose: Use plain Kafka Streams for services without Spring or Quarkus, or when you want to manage configuration and lifecycle yourself.

Kafka Streams with Spring Boot and Spring Kafka

Spring Kafka manages Kafka Streams through a StreamsBuilderFactoryBean. With kafka-streams on the classpath and @EnableKafkaStreams on a configuration class, Spring Boot builds its configuration from spring.kafka.streams.* properties.

For the order count, a @Bean method receives the shared StreamsBuilder, defines the stream, and returns it. The factory bean starts the topology with the application context and closes it on shutdown.

Capabilities and limitations

• Configuration: spring.kafka.streams.application-id defaults to spring.application.name. Boot-supported settings use spring.kafka.streams.*; other Kafka Streams properties go under spring.kafka.streams.properties.*.
• Several topologies: Each topology with its own lifecycle needs its own factory bean and application ID.
• Failed records: RecoveringDeserializationExceptionHandler can send records that fail deserialization to a dead-letter topic. Spring Kafka 4.1 adds matching handlers for processing and production errors, which can use Kafka's native DLQ.
• Helpers: KafkaStreamsInteractiveQueryService reads state stores, KafkaStreamsMicrometerListener registers metrics, and Spring Kafka 4.1 can switch to the streams rebalance protocol with setGroupProtocol().

When to choose: Use Spring Kafka for Kafka Streams in Spring Boot services. You do not need Spring Cloud Stream to run a topology in Spring Boot.

Spring Cloud Stream Kafka Streams binder

The spring-cloud-stream-binder-kafka-streams artifact binds functions to Kafka topics. For the order count, a Function<KStream<String, Order>, KStream<String, Long>> bean receives the input stream and returns the counts.

Binding properties map the function's -in-0 and -out-0 bindings to topics; see the programming model.

Each bound processor runs as its own Kafka Streams instance; composed functions such as foo|bar form one processor. Without an explicit ID, the binder uses spring.application.name, or generates one from the function bean name if that is not set.

spring.application.name suits a single processor only; with several, they would all share that ID. Set IDs explicitly in production, one per function, and give every replica of a processor the same ID.

Capabilities and limitations

• Serdes: The binder infers Serdes from the function's generic types. When no Kafka Serde matches, it falls back to Spring Kafka's JacksonJsonSerde, so configure Serdes explicitly for Avro or Protobuf topics.
• Deserialization errors: With deserializationExceptionHandler: sendToDlq, records that fail deserialization go to error.<input-topic>.<application-id> by default.
• Errors in your code: Wrap the logic in DltAwareProcessor to publish failures to a DLT, or in RecordRecoverableProcessor to handle them yourself; see error handling.
• DLT keys: DltAwareProcessor publishes through the regular Kafka binder, so configure the DLT binding's key serializer unless keys are byte[].
• Status: The Kafka Streams binder is maintained; version 5.0.3 shipped on 20 Aug 2026. The deprecation in Spring Cloud Stream 4.3.0 applies to the reactive Kafka binder.

When to choose: Use the binder when your services already use Spring Cloud Stream function bindings. It adds configuration on top of Kafka Streams, not more stream-processing features.

Quarkus Kafka Streams

With quarkus-kafka-streams, a CDI producer method returns the Topology and Quarkus starts Kafka Streams for it. The application ID defaults to quarkus.application.name, and kafka-streams.* properties pass straight through; see the Quarkus guide.

Capabilities and limitations

• Startup: The extension waits until every topic in quarkus.kafka-streams.topics exists before starting the topology. With quarkus-smallrye-health, it adds a readiness check for those topics and a liveness check for the Kafka Streams state.
• Development: Dev mode reloads the topology after code changes. Dev Services starts a Kafka broker in dev mode and tests when Docker is available and kafka.bootstrap.servers is not set.
• Native executables: Kafka Streams applications can build as native executables.
• Serdes: As in plain Kafka Streams, set Serdes in the topology code or as defaults through kafka-streams.*; ObjectMapperSerde covers JSON values.

When to choose: Use the extension for Quarkus services, especially when you want built-in health checks and a broker for local tests.

Kafka Streams Maven dependencies and version alignment

In Spring Boot and Quarkus, the framework's dependency management picks the Kafka Streams version. Check the resolved version before you rely on a feature from a newer Kafka release, such as the native DLQ or the streams rebalance protocol from 4.2.

To see the versions your build resolves, run mvn dependency:tree -Dincludes=org.apache.kafka or gradle dependencyInsight --dependency kafka-streams.

Spring Cloud 2025.1 supports Spring Boot 4.0 and, from 2025.1.2, Spring Boot 4.1; see the compatibility table. The binder is built against Kafka Streams 4.1.2, but Spring Boot's dependency management decides the version your application uses.

With spring-boot-starter-parent, override kafka.version to keep the Kafka artifacts aligned, including kafka-streams-test-utils. If you only import spring-boot-dependencies, override the relevant dependencyManagement entries instead.

Kafka Streams state stores and recovery

A state store holds data a stateful operation needs between records, such as each customer's count in order-counts. Each instance keeps the stores for the partitions it owns; by default, each store is backed by a changelog topic.

RocksDB and in-memory stores

DSL operations such as count() and aggregate() use persistent RocksDB stores by default, kept under state.dir (default ${java.io.tmpdir}/kafka-streams).

In-memory stores avoid disk I/O, but after a restart they are rebuilt from the changelog, or start empty if logging is disabled.

Set the store type for all DSL operations with dsl.store.suppliers.class, or for one operation with Materialized. RocksDB uses off-heap memory for its block cache and write buffers, so leave room beyond the JVM heap; see memory management.

Internal topics are named <application.id>-<name>-changelog and -repartition. Name stores explicitly, for example with Materialized.as("order-counts"), because generated names can change when the topology changes.

Since Kafka 4.1, setting ensure.explicit.internal.resource.naming=true stops the application from starting while any internal name is generated.

Querying state with Interactive Queries

Interactive Queries read a running instance's local stores, for example to return one customer's count from a REST endpoint. Each instance holds only the partitions it owns, so a key may live on another instance.

To reach that instance, set a unique application.server on each instance and look up the key's host with queryMetadataForKey(). The call to the other instance goes through an HTTP or RPC endpoint you provide.

Spring Kafka's KafkaStreamsInteractiveQueryService and the binder's InteractiveQueryService wrap the store and host lookups. The Quarkus guide shows the same pattern with REST endpoints.

While a store is migrating or restoring, lookups throw InvalidStateStoreException; both Spring helpers can retry them.

Restoration, standby replicas, and scaling

When a task moves to another instance, Kafka Streams replays the store's changelog before processing resumes, which takes time for large stores.

num.standby.replicas (default 0) keeps copies of each stateful task's stores on other instances, so a failover only replays the changelog records a standby has not caught up on.

With one standby per task, run at least two instances and plan for roughly twice the total state-store capacity.

When a new instance joins, the default classic protocol keeps tasks on caught-up instances and warms up state on the new one first, controlled by acceptable.recovery.lag and max.warmup.replicas.

Caught-up warmups take over at the next probing rebalance, every probing.rebalance.interval.ms (10 minutes by default). See Streams architecture.

In this single-input topology, five input partitions allow at most five active tasks; a sixth instance gets no active task but can host standby or warmup tasks.

With exactly_once_v2, an unclean shutdown can make restores longer: Kafka Streams 4.3 still wipes the affected stores and rebuilds them from the changelog.

Kafka 4.2 also added a broker-side rebalance protocol for Kafka Streams; see the FAQ for what it supports.

Kafka Streams testing

Test the topology without a broker first, then test what depends on Kafka itself against a broker.

Test a topology with TopologyTestDriver

TopologyTestDriver from kafka-streams-test-utils runs a topology in one process without a broker. You pipe records into TestInputTopic, read TestOutputTopic, and read state stores directly, for example to check that three orders leave a count of 3.

It also controls time: event-time punctuations follow record timestamps, and advanceWallClockTime() fires wall-clock punctuations, so time-based logic runs without waiting.

The driver processes records synchronously and commits after each input, so it does not reproduce production caching or commit timing.

In Spring or Quarkus, keep the stream definition in a method you can call from a plain unit test.

Integration tests with a Kafka broker

The test driver does not create internal topics, rebalance between instances, restore state from changelogs, or run Kafka transactions. Test those behaviors against a broker.

Schema Registry Serdes can run inside the driver, for example against Confluent's in-memory mock:// registry. Test the connection to your real registry and schema compatibility separately.

Spring Kafka's @EmbeddedKafka starts an in-process KRaft broker for tests (testing guide). Quarkus Dev Services starts a Kafka container in dev mode and tests. For plain Java, Testcontainers' Kafka module does the same.

A useful broker test runs two instances with the same application ID and records one customer's count, N. Stop the instance that owns that customer's partition, wait for the other to take over, send one more order, and check that the count becomes N+1.

When running both instances on the same host, give each a separate state.dir.

Inspect Kafka Streams topics with Kafma

Unit tests check the topology logic. A deployed application can still use the wrong Serde, restore slowly, or fill a dead-letter topic, and those problems show up in topics and group positions.

Kafma is a desktop Kafka IDE that runs beside your Java application. Use its Kafka console to send input, read output and dead-letter topics, and watch the application's group.

Send keyed input and check the counts

Use the producer panel to send orders with the customer ID as a String key, then open the output topic and watch that customer's count increase. Loop keeps orders arriving at a fixed interval, which the restart test below needs.

By default, counts can take up to 30 seconds to appear, and several orders may show up as one update, because Kafka Streams caches results until the next commit.

Counts written with Serdes.Long() are 8-byte values; select the Long decoder to read them. With Schema Registry configured, Kafma decodes Avro, Protobuf, and JSON Schema records automatically.

Watch the application during restarts

With the default classic protocol, a Kafka Streams application appears as a consumer group named after its application.id. Open Watch Group on it, then stop an instance or start a second one.

While a moved task restores its store, its committed positions stop advancing, and lag grows if orders keep arriving. Kafma reads the group's positions without joining it, so watching does not trigger a rebalance.

The consumer group UI shows each member's partitions as ownership moves.

Inspect and replay dead-letter records

Kafka Streams' native DLQ records add __streams.errors.* headers with the exception, stack trace, and source topic, partition, and offset. Spring Kafka's handlers add kafka_dlt-* headers instead. Expand a record in Kafma to read them.

For records that failed deserialization on the input topic, fix the cause, then use Replay to topic to load the record into the producer for that topic. Remove the error headers, check the key and value encodings, and produce it.

Download Kafma and connect to the same test cluster as your application.

Frequently asked questions

What is the difference between the Kafka binder and the Kafka Streams binder?

Both are Spring Cloud Stream binders for Kafka. spring-cloud-stream-binder-kafka passes individual messages to Consumer and Function beans and runs on Spring Kafka listener containers.

spring-cloud-stream-binder-kafka-streams passes KStream, KTable, or GlobalKTable objects to your functions and runs a Kafka Streams topology. For the message binder, see Kafka Java clients.

Do you need Spring Cloud Stream to use Kafka Streams with Spring Boot?

No. Spring Kafka runs Kafka Streams in Spring Boot with @EnableKafkaStreams and spring.kafka.streams.* properties. Spring Cloud Stream adds function bindings on top; add it when your services already use them.

Can Spring Kafka listeners and Kafka Streams run in the same application?

Yes. One Spring Boot application can use KafkaTemplate, @KafkaListener, and a Kafka Streams topology together. Give the topology an application.id that differs from your listener group IDs, because Kafka Streams uses it as its group ID.

What is the difference between KStream and KTable?

A KStream treats each record as a separate event, such as an order. A KTable treats each record as an update to the latest value for its key, such as a customer's current count; counting a KStream by key produces a KTable.

A GlobalKTable gives every instance a full copy of a table.

Is the Spring Cloud Stream Kafka Streams binder deprecated?

No. Spring Cloud Stream 5.0.3 shipped the Kafka Streams binder on 20 Aug 2026. The deprecation announced in Spring Cloud Stream 4.3.0 covers the reactive Kafka binder, after Reactor Kafka was discontinued.

Does Kafka Streams require a separate cluster?

It needs a Kafka cluster but no separate processing cluster. The library runs inside your application, and instances with the same application.id share the work by dividing the input partitions among them.

Can Kafka Streams use the new rebalance protocol?

Yes, when brokers and clients run Kafka 4.2 or later (see the version table). New 4.2 clusters support the streams rebalance protocol (KIP-1071) by default; upgraded clusters first need streams.version=1 enabled with kafka-features.sh.

Applications still default to classic and opt in with group.protocol=streams.

The application then registers as a streams group instead of a consumer group. Standby replicas become a group setting, streams.num.standby.replicas, set with kafka-configs.sh; the client's num.standby.replicas is ignored.

Static membership, warmup tasks, regex subscriptions, and online migration are not supported yet.

To migrate an existing application offline, stop every instance, wait until the group is empty, and restart with the new setting; only committed offsets carry over. Use 4.2.1 or later brokers for this, since 4.2.0 has a migration bug.

Conclusion

In Spring Boot, start with Spring Kafka and @EnableKafkaStreams; in Quarkus, use the extension; elsewhere, use Kafka Streams directly. Add the Spring Cloud Stream binder when your services already use its function bindings.

Whichever you choose, check the Kafka Streams version it resolves, name your state stores, and decide where failed records go. Test the topology with TopologyTestDriver first, then test restores and rebalances against a broker.

This guide is maintained by the team behind Kafma.