Spark vs Kafka Streams 2026: Which Should You Use?

Kafka stores and moves events; Apache Spark and Kafka Streams process them. Spark runs queries on a cluster through Structured Streaming, often still called Spark Streaming, and can also read and write files, tables, and databases. Kafka Streams is a Java library that runs inside your application and reads and writes topics in one Kafka cluster.

Three differences usually decide between them. Spark has a full Python API, while Kafka Streams runs only on the JVM. Spark processes records in micro-batches, small batch jobs that run one after another, unless a query fits its new Real-time Mode. Spark keeps its Kafka offsets in its own checkpoint, so consumer group tools cannot show how far behind it is.

The examples use five-minute order totals per customer from an orders topic, as in Flink vs Kafka Streams. The Spark behavior described here was tested with PySpark 4.3.0 against a Kafka 4.4.0 broker.

Quick decision

Spark vs Kafka Streams compared

Capabilities verified against each release's documentation and source: 6 Nov 2026

Deployment and operations

Kafka Streams runs inside your own Java service. To scale, you start more copies with the same application.id, and they split the partitions between them. State is backed up to Kafka topics, so nothing outside Kafka is needed.

A Spark query runs on a cluster. The driver plans each micro-batch and the executors run its tasks. Managed platforms such as Databricks, Amazon EMR, and Google Cloud Dataproc can run the cluster for you, and every query needs a checkpoint location on durable storage such as S3 or HDFS.

PySpark supports Structured Streaming, including the Kafka source and sink and query listeners, so a Python team can write the whole pipeline without Java. Kafka Streams has no Python API. For Python libraries that run without a cluster, see Kafka Streams in Python.

Micro-batches and Spark Real-time Mode

By default, Spark runs a query as a series of micro-batches. Each batch reads the new records from every partition, processes them, and writes the results. A trigger sets how often batches start; without one, the next batch starts as soon as the previous one finishes. Kafka Streams processes each record as it arrives. For aggregations such as the order totals, Kafka Streams caches updates and sends them downstream when the cache fills or on each commit: every 30 seconds by default, or 100 ms with exactly_once_v2.

Real-time Mode, added in Spark 4.1, removes the wait for the next batch. It keeps one long-running task per Kafka partition that processes records as they arrive, so the cluster needs at least one free core per partition. In Python, enable it with .trigger(realTime="5 minutes"). The duration sets how often progress is checkpointed, not the latency.

In a local test, a query that copied orders over 100 EUR to another topic took a median of 4 ms from order to output in Real-time Mode, and 558 ms as micro-batches. The test ran on a laptop with three Kafka partitions and five orders per second, with no explicit trigger for micro-batches. Latency was measured from each order's createdAt to the output record's Kafka timestamp.

Spark 4.3 added aggregations, including window(), and dropDuplicates to the stateless queries Real-time Mode already ran. In the same test, the order totals query took roughly 31 ms in Real-time Mode and 844 ms as micro-batches, estimated from each customer's latest order. Stream-stream joins, session windows, and PySpark's transformWithState are still not supported. The mode requires update output, reads only from Kafka, writes to Kafka, foreach, console, or memory sinks, and rejects minPartitions and maxOffsetsPerTrigger.

Aggregations need more cores, because shuffle tasks run alongside the Kafka readers: 10 Kafka partitions and 5 shuffle partitions need at least 15. Real-time Mode also requires state store checkpoint format v2, while micro-batch queries default to v1, so moving an existing query usually means a new checkpoint.

State, recovery, and scaling

Kafka Streams keeps each customer's running total in a local RocksDB store, backed up to a changelog topic. When an instance stops, another replays that topic to rebuild the state. Without repartitioning, the number of orders partitions caps the number of tasks; see restoration and scaling.

Spark keeps state in executor memory by default and writes each batch's changes to the checkpoint location. For millions of keys, the Spark docs recommend the RocksDB state store, which keeps state in native memory and on local disk to avoid long garbage collection pauses. After a restart, the query loads its state from the checkpoint and continues from the offsets saved there.

In micro-batch mode, Spark can split a busy Kafka partition across several tasks with minPartitions and cap each batch with maxOffsetsPerTrigger. Stateful steps are different: their partition count comes from spark.sql.shuffle.partitions when the query first starts and is saved in the checkpoint. Changing it means a new checkpoint, which starts the state from empty, so choose it before going live.

Event time, windows, and joins

Both place orders in windows by event time, the time on the order, not the wall clock. Spark windows by a column you choose, such as createdAt. Kafka Streams uses each record's Kafka timestamp by default; to window by a field such as createdAt, configure a TimestampExtractor.

They handle late orders differently. Kafka Streams accepts them until a grace period after the window's end. Spark tracks a watermark, the newest order time it has seen minus the delay you pass to withWatermark; it can drop orders older than the watermark and removes a window's state once the watermark passes the window's end.

The output mode decides when totals reach Kafka. In update mode, each micro-batch writes the windows it changed, so a customer's total is written again as orders arrive, as Kafka Streams does by default. In append mode, Spark writes each window once, after the watermark passes its end, so totals arrive at least the watermark delay late.

Kafka Streams joins usually need both topics partitioned the same way, with the same partition count and partitioner. Spark reshuffles records itself, so it can join any two streams. For a join between two streams, give both sides a watermark and a time limit on matches. Inner joins without them keep all past input in state, and outer joins require them.

Exactly-once and output latency

Exactly-once means each order is counted once in the output, even if the job fails and reprocesses some input. Kafka Streams does this with processing.guarantee=exactly_once_v2, which commits output and input offsets in one Kafka transaction, every 100 ms by default. Consumers that read with read_committed then see each result once.

Spark updates its state exactly once, but its Kafka sink is at-least-once in both micro-batches and Real-time Mode. If a batch fails after writing some results, Spark reruns it from the offsets in the checkpoint and writes those results again. Include the customer and window in each total, so consumers can keep the latest value for each pair and ignore repeats. Spark's Kafka source also reads with read_uncommitted by default, so set kafka.isolation.level=read_committed when the input comes from transactional producers. See exactly-once support by framework.

In micro-batch mode, Spark's output latency follows its batches. A total reaches Kafka while the micro-batch that processes the order runs, so with a 10-second trigger it can take more than 10 seconds. Real-time Mode writes each total as it changes.

Spark offsets and consumer lag

Where Spark keeps its Kafka offsets

Spark's Kafka source does not commit offsets to Kafka. Each query reads with its own generated group ID, starting with spark-kafka-source, and saves its progress in the checkpoint location instead. With the query below running, kafka-consumer-groups.sh --list showed no group for it at all.

Here is the order totals query in PySpark. The Kafka connector ships separately, so start Spark with --packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.3.0.

from pyspark.sql.functions import col, from_json, struct, sum as sum_, to_json, window

orders = (
    spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "orders")
    .option("startingOffsets", "earliest")
    .load()
    .select(from_json(col("value").cast("string"),
                      "customerId STRING, total DECIMAL(10, 2), createdAt TIMESTAMP").alias("order"))
    .select("order.*")
)

totals = (
    orders.withWatermark("createdAt", "1 minute")
    .groupBy(window("createdAt", "5 minutes"), "customerId")
    .agg(sum_("total").alias("totalAmount"))
)

query = (
    totals.select(col("customerId").alias("key"),
                  to_json(struct("customerId", "window", "totalAmount")).alias("value"))
    .writeStream.format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("topic", "customer-order-totals")
    .option("checkpointLocation", "s3a://my-bucket/checkpoints/order-totals")  # Saves state and Kafka offsets
    .outputMode("update")
    .queryName("order-totals")
    .trigger(processingTime="10 seconds")
    .start()
)

Before each micro-batch, Spark writes a file to offsets/ in the checkpoint, named after the batch number. Its last line holds the end offset of every partition, such as {"orders":{"0":203,"1":185,"2":0}}. When the batch finishes, Spark writes a file with the same number to commits/, and after a restart it reruns a batch that has no commit file.

startingOffsets applies only to a new checkpoint. After that, the query always resumes where its checkpoint says. To reprocess a topic, start the query with a new checkpoint location, which also starts its state from empty. Kafka Streams uses kafka-streams-application-reset instead: stop every instance, run the tool to rewind input offsets and delete internal topics, then clear each instance's local state.

Monitor lag from query progress

After each micro-batch, Spark reports a StreamingQueryProgress. For a Kafka source, its endOffset holds the offsets the batch read up to, and its metrics include minOffsetsBehindLatest, maxOffsetsBehindLatest, and avgOffsetsBehindLatest.

In micro-batch mode, these metrics do not measure lag against the current end of the log. Spark's Kafka source compares the batch's end offsets with the newest offsets Spark saw when it planned the batch. Without maxOffsetsPerTrigger, a batch reads everything available at that moment, so the metrics stay at 0. In the test, they read 0 while the query trailed the log by up to about 50 orders between 10-second batches.

To see lag against the end of the log, copy each batch's end offsets into a consumer group. Kafka tools then compute its lag the same way as for any other consumer.

Commit Spark progress to a consumer group

A StreamingQueryListener runs on the driver and is called after each micro-batch of every query in the session. This one picks the order-totals query by name and writes its end offsets to a consumer group used only for monitoring, with the admin client from confluent-kafka. Register it before calling start(). It assumes the query reads only from Kafka. In Real-time Mode it runs once per trigger duration, so the group updates that often and its lag can include records already processed.

import json

from confluent_kafka import ConsumerGroupTopicPartitions, TopicPartition
from confluent_kafka.admin import AdminClient
from pyspark.sql.streaming import StreamingQueryListener


class CommitOffsets(StreamingQueryListener):
    def __init__(self, bootstrap_servers, group_id, query_name):
        self.admin = AdminClient({"bootstrap.servers": bootstrap_servers})
        self.group_id = group_id
        self.query_name = query_name

    def onQueryStarted(self, event): pass
    def onQueryIdle(self, event): pass
    def onQueryTerminated(self, event): pass

    def onQueryProgress(self, event):
        if event.progress.name != self.query_name:
            return
        for source in event.progress.sources:
            end = json.loads(source.endOffset)
            partitions = [
                TopicPartition(topic, int(partition), offset)
                for topic, offsets in end.items()
                for partition, offset in offsets.items()
            ]
            request = ConsumerGroupTopicPartitions(self.group_id, partitions)
            for future in self.admin.alter_consumer_group_offsets([request]).values():
                future.result()


spark.streams.addListener(CommitOffsets("localhost:9092", "order-totals-spark", "order-totals"))

The group has no members, and its offsets are as of the last finished batch. Spark still restarts from its checkpoint, so these offsets are only for monitoring. Resetting the group does not change what the query reads.

Inspect Spark and Kafka Streams topics with Kafma

Kafma is a desktop Kafka IDE. Kafka Streams commits its offsets to a consumer group named after application.id, so Kafma shows its lag with no extra code; see Inspect Kafka Streams topics with Kafma. Spark doesn't commit offsets, so for a Spark query, open the group that the listener above writes to.

Watch a Spark query's progress

Open Watch Group on orders with the listener's group. Records above the divider are not yet in a finished batch, and records below it are processed. The lag chart rises between batches and drops when one finishes. If its low points keep rising, the query is falling behind.

Find the records a failing batch reads

When a micro-batch keeps failing, find the newest file in offsets/ that has no matching file in commits/. It holds where the failing batch ends, and the previous offsets file holds where it starts. The first batch has no previous offsets file; its start offsets are resolved from startingOffsets when the query initializes.

In the Kafka console, open orders, filter to one partition, set From to Offset, and enter that partition's start offset. The batch reads from there up to, but not including, the end offset.

Download Kafma and connect to the cluster your Spark queries read from.

Frequently asked questions

What is the difference between Spark and Kafka?

Kafka stores events durably in topics and delivers them to consumers. Spark is a processing engine: Structured Streaming reads those events, computes results such as windowed totals, and writes them to Kafka or other systems.

Is Spark Streaming the same as Structured Streaming?

No. Spark Streaming is the older DStream API, which the Spark docs call a legacy project that no longer gets updates. Structured Streaming is the current engine, built on DataFrames and SQL, and it is what this article compares. Many people still say Spark Streaming for both.

Can Spark replace Kafka Streams?

For most workloads, yes, if you can run a Spark cluster and accept micro-batch latency or Real-time Mode's limits. You lose exactly-once output to Kafka, and Kafka Streams state does not carry over, so plan to rebuild it by reprocessing the input.

Does Spark commit offsets to Kafka?

No. The Kafka source keeps offsets in the query's checkpoint and doesn't commit any to Kafka. To show lag in Kafka tools, copy each batch's end offsets into a consumer group from a StreamingQueryListener, as shown above.

Can I use Kafka Streams from Python?

No. Kafka Streams runs only on the JVM. Python teams use PySpark, PyFlink, or a Python library such as Quix Streams; see Kafka Streams in Python.

Which is faster, Spark or Kafka Streams?

For stateless steps, Kafka Streams and Spark Real-time Mode handle each record as it arrives, while micro-batches take about 100 ms at best. Aggregated results wait longer: Kafka Streams buffers them until a commit or a full cache flushes them, and Spark micro-batches write them while the next batch runs. Spark's release notes put Real-time Mode aggregations on the order of 100 ms. For throughput, compare both on your own workload with the same delivery guarantees.

Conclusion

Choose Kafka Streams when processing belongs in a JVM service whose topics are all in one cluster, or when you need exactly-once output to Kafka. Choose Spark for Python, for combining Kafka with files and tables, or when your team already runs Spark.

Either way, decide on the settings that are hard to change later: input partition counts for Kafka Streams, and spark.sql.shuffle.partitions and the checkpoint location for Spark. If you want Spark's lag in the same tools as your other consumers, add the listener before the first run.

This guide is maintained by the team behind Kafma.