Library or separate engine
Libraries run inside your service. Flink and Spark are separate engines: you run a cluster or pay for a managed one.
Kafka Stream Processing
Pick your language or engine. Each guide compares the options side by side, checked against specific releases.
Kafka stream processing guides
- JavaPlain Kafka Streams vs Spring Boot vs Spring Cloud Stream vs Quarkus
- PythonFaust Streaming vs Quix Streams
- GoGoka vs Watermill vs go-streams vs Redpanda Connect
- Apache FlinkFlink vs Kafka Streams vs ksqlDB
- Apache SparkSpark Structured Streaming vs Kafka Streams
- Exactly-oncePlain clients vs Kafka Streams vs Flink vs Spark vs Kafka Connect
- Dead letter queuesSpring Kafka vs Spring Cloud Stream vs Kafka Connect vs Kafka Streams
How to choose a Kafka stream processing framework
Options differ in where they run, which languages they support, and what happens to their state after a restart.
Language
Kafka Streams runs only on the JVM. Python, Go, and .NET have their own libraries. Flink and Spark also have Python APIs.
Sources and sinks
Kafka Streams reads and writes topics in one cluster. Flink and Spark also connect to files, tables, and databases.
State after a restart
Most options restore state from Kafka topics or checkpoints. Watermill leaves it to you, and go-streams keeps windows in memory.
Windows and joins
Goka has no built-in windows, Faust no built-in joins, and most Kafka Streams joins need co-partitioned topics.
Exactly-once
Exactly-once output works in Kafka Streams, Flink, Quix, Faust, and Streamiz. Spark's Kafka sink and Goka are at-least-once.
How to verify stream processing results
Unit tests check the logic, not what reaches Kafka. On a test cluster, check six things.
Input and output records
Work out the expected result by hand, then read the output topic, with read_committed if it is exactly-once. Results can lag until the next commit or window close.
Keys
Most joins in Kafka Streams and Goka need a key on the same partition number in both topics, but default partitioners differ across clients. Compare partitions for the same key.
Schemas
Check that output decodes with the schema consumers expect. Serdes.Long() writes 8 raw bytes, and the Spring Cloud Stream binder can fall back to JSON on Avro topics.
Consumer positions
Kafka Streams commits to a group named after application.id, and Flink when checkpoints complete. Spark commits nothing, so copy its offsets into a group.
Restarts
Give each input a unique ID and restart mid-run. For one-to-one output, compare IDs: duplicates fit at-least-once, missing IDs don't. For aggregates, check each key's totals.
Bad records
Send a record the processor cannot read. Depending on the framework and settings, it reaches a dead letter topic, gets skipped, or stops the application.
Kafka Stream Processing FAQ
- What is Kafka stream processing?
- Handling records continuously as they arrive, instead of waiting for a complete batch. For example, a processor reads an orders topic, adds up each customer's orders every five minutes, and writes the totals to another topic. It has to keep each running total through restarts, place late orders in the right window, and recover from a crash without losing or double-counting orders.
- What is the difference between Kafka and Kafka Streams?
- Kafka stores events in topics and delivers them to consumers. Kafka Streams is a Java library, shipped with Apache Kafka, that runs inside your application, reads those topics, computes results, and writes them back to Kafka. It is one stream processing tool among several: Flink and Spark are separate engines that run on their own clusters.
- Is a plain Kafka consumer enough?
- For work on one record at a time, such as filtering, routing, or converting a format, yes. For aggregates, windows, and joins, you would have to build state and recovery yourself. To pick a client, see Kafka client libraries.
- What is the difference between stream and batch processing?
- A batch job runs over a finished set of data, such as yesterday's orders, and then stops. A stream processor keeps running and updates its results as records arrive. By default, Spark runs a streaming query as a series of small batch jobs, called micro-batches.
- Can I process Kafka streams without Java?
- Yes. Python has Quix Streams and Faust Streaming, Go has Goka, and .NET has Streamiz.Kafka.Net. PySpark and PyFlink let you write Spark and Flink jobs in Python, but they still need a cluster.
- Is ksqlDB still an option?
- For existing queries, yes. ksqlDB is a SQL engine built on Kafka Streams; it still ships with Confluent Platform, and Confluent Cloud fully supports existing applications. For new SQL pipelines on Confluent Cloud, Confluent recommends Flink. See Flink SQL vs ksqlDB.
Ready to try the Kafka IDE?
Whichever framework you choose, Kafma helps you check the records it reads and writes, and how far it has read.
Available for macOS, Windows, Linux