[{"data":1,"prerenderedAt":2019},["ShallowReactive",2],{"blog-post-\u002Fblog\u002Fspark-vs-kafka-streams":3},{"id":4,"title":5,"body":6,"date":2007,"description":2008,"extension":2009,"meta":2010,"navigation":655,"ogImage":2011,"ogImageHeight":2011,"ogImageWidth":2011,"path":2012,"pinned":2013,"seo":2014,"sitemap":2011,"stem":2015,"tags":2016,"__hash__":2018},"blog\u002Fblog\u002Fspark-vs-kafka-streams.md","Spark vs Kafka Streams 2026: Which Should You Use?",{"type":7,"value":8,"toc":1979},"minimark",[9,35,38,51,56,141,145,151,291,296,302,305,313,317,322,334,341,362,365,369,379,382,395,399,488,500,507,518,521,525,536,551,554,558,562,573,579,1165,1180,1190,1194,1215,1227,1230,1234,1253,1814,1817,1821,1842,1846,1857,1866,1870,1882,1901,1908,1912,1916,1919,1923,1926,1930,1933,1937,1943,1947,1952,1956,1959,1963,1966,1972,1975],[10,11,12,13,20,21,26,27,31,32,34],"p",{},"Kafka stores and moves events; ",[14,15,19],"a",{"href":16,"rel":17},"https:\u002F\u002Fspark.apache.org\u002F",[18],"nofollow","Apache Spark"," and ",[14,22,25],{"href":23,"rel":24},"https:\u002F\u002Fkafka.apache.org\u002F44\u002Fstreams\u002Fcore-concepts\u002F",[18],"Kafka Streams"," process them. ",[28,29,30],"strong",{},"Spark"," runs queries on a cluster through Structured Streaming, often still called Spark Streaming, and can also read and write files, tables, and databases. ",[28,33,25],{}," is a Java library that runs inside your application and reads and writes topics in one Kafka cluster.",[10,36,37],{},"Three differences usually decide between them. Spark has a full Python API, while Kafka Streams runs only on the JVM. Spark processes records in micro-batches, small batch jobs that run one after another, unless a query fits its new Real-time Mode. Spark keeps its Kafka offsets in its own checkpoint, so consumer group tools cannot show how far behind it is.",[10,39,40,41,45,46,50],{},"The examples use five-minute order totals per customer from an ",[42,43,44],"code",{},"orders"," topic, as in ",[14,47,49],{"href":48},"\u002Fblog\u002Fflink-vs-kafka-streams","Flink vs Kafka Streams",". The Spark behavior described here was tested with PySpark 4.3.0 against a Kafka 4.4.0 broker.",[52,53,55],"h2",{"id":54},"quick-decision","Quick decision",[57,58,59,75],"table",{},[60,61,62],"thead",{},[63,64,65,69,72],"tr",{},[66,67,68],"th",{},"Situation",[66,70,71],{},"Best starting point",[66,73,74],{},"Main caveat",[76,77,78,91,103,116,128],"tbody",{},[63,79,80,86,88],{},[81,82,83],"td",{},[28,84,85],{},"JVM service whose inputs and outputs are all Kafka topics",[81,87,25],{},[81,89,90],{},"Parallelism depends on topic partition counts; all topics must be in one cluster.",[63,92,93,98,100],{},[81,94,95],{},[28,96,97],{},"Exactly-once output to Kafka",[81,99,25],{},[81,101,102],{},"Covers only topics in the same Kafka cluster.",[63,104,105,110,113],{},[81,106,107],{},[28,108,109],{},"Python team",[81,111,112],{},"Spark with PySpark",[81,114,115],{},"You run a Spark cluster or pay for a managed one.",[63,117,118,123,125],{},[81,119,120],{},[28,121,122],{},"Kafka data combined with files or tables, or backfills with the same code",[81,124,30],{},[81,126,127],{},"Micro-batches add latency, about 100 ms at best.",[63,129,130,135,138],{},[81,131,132],{},[28,133,134],{},"Filtering or routing within milliseconds",[81,136,137],{},"Kafka Streams, or Spark Real-time Mode",[81,139,140],{},"Real-time Mode holds a core for each task and does not support stream-stream joins.",[52,142,144],{"id":143},"spark-vs-kafka-streams-compared","Spark vs Kafka Streams compared",[10,146,147,150],{},[28,148,149],{},"Capabilities verified against each release's documentation and source:"," 6 Nov 2026",[57,152,153,175],{},[60,154,155],{},[63,156,157,160,167],{},[66,158,159],{},"Area",[66,161,162],{},[14,163,166],{"href":164,"rel":165},"https:\u002F\u002Fgithub.com\u002Fapache\u002Fkafka\u002Ftree\u002F4.4.0",[18],"Kafka Streams 4.4.0",[66,168,169,174],{},[14,170,173],{"href":171,"rel":172},"https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark\u002Ftree\u002Fv4.3.0",[18],"Spark 4.3.0"," Structured Streaming",[76,176,177,190,203,216,229,245,258,275],{},[63,178,179,184,187],{},[81,180,181],{},[28,182,183],{},"Runs as",[81,185,186],{},"Library inside your JVM application",[81,188,189],{},"Queries on a Spark cluster: standalone, YARN, or Kubernetes",[63,191,192,197,200],{},[81,193,194],{},[28,195,196],{},"APIs",[81,198,199],{},"DSL and Processor API in Java",[81,201,202],{},"DataFrame API and SQL, in Python, Scala, and Java",[63,204,205,210,213],{},[81,206,207],{},[28,208,209],{},"Processing",[81,211,212],{},"One record at a time",[81,214,215],{},"Micro-batches, or Real-time Mode for supported queries",[63,217,218,223,226],{},[81,219,220],{},[28,221,222],{},"State",[81,224,225],{},"Local RocksDB or in-memory stores, backed by changelog topics",[81,227,228],{},"Executor memory by default, or RocksDB, saved to a checkpoint location such as S3",[63,230,231,236,242],{},[81,232,233],{},[28,234,235],{},"Kafka offsets",[81,237,238,239],{},"Committed to the consumer group named after ",[42,240,241],{},"application.id",[81,243,244],{},"Saved in the checkpoint, never committed to Kafka",[63,246,247,252,255],{},[81,248,249],{},[28,250,251],{},"Scaling",[81,253,254],{},"More instances, limited by partition counts",[81,256,257],{},"More executors; state partitions fixed when the query first starts",[63,259,260,265,272],{},[81,261,262],{},[28,263,264],{},"Exactly-once to Kafka",[81,266,267,268,271],{},"Yes, with ",[42,269,270],{},"exactly_once_v2",", in one cluster",[81,273,274],{},"No; the Kafka sink is at-least-once",[63,276,277,282,285],{},[81,278,279],{},[28,280,281],{},"Sources and sinks",[81,283,284],{},"Kafka topics in one cluster",[81,286,287,288],{},"Kafka, files, and tables; any batch sink through ",[42,289,290],{},"foreachBatch",[292,293,295],"h3",{"id":294},"deployment-and-operations","Deployment and operations",[10,297,298,299,301],{},"Kafka Streams runs inside your own Java service. To scale, you start more copies with the same ",[42,300,241],{},", and they split the partitions between them. State is backed up to Kafka topics, so nothing outside Kafka is needed.",[10,303,304],{},"A Spark query runs on a cluster. The driver plans each micro-batch and the executors run its tasks. Managed platforms such as Databricks, Amazon EMR, and Google Cloud Dataproc can run the cluster for you, and every query needs a checkpoint location on durable storage such as S3 or HDFS.",[10,306,307,308,312],{},"PySpark supports Structured Streaming, including the Kafka source and sink and query listeners, so a Python team can write the whole pipeline without Java. Kafka Streams has no Python API. For Python libraries that run without a cluster, see ",[14,309,311],{"href":310},"\u002Fblog\u002Fkafka-streams-in-python","Kafka Streams in Python",".",[292,314,316],{"id":315},"micro-batches-and-spark-real-time-mode","Micro-batches and Spark Real-time Mode",[10,318,319,320,312],{},"By default, Spark runs a query as a series of micro-batches. Each batch reads the new records from every partition, processes them, and writes the results. A trigger sets how often batches start; without one, the next batch starts as soon as the previous one finishes. Kafka Streams processes each record as it arrives. For aggregations such as the order totals, Kafka Streams caches updates and sends them downstream when the cache fills or on each commit: every 30 seconds by default, or 100 ms with ",[42,321,270],{},[10,323,324,329,330,333],{},[14,325,328],{"href":326,"rel":327},"https:\u002F\u002Fspark.apache.org\u002Fdocs\u002F4.3.0\u002Fstreaming\u002Freal-time-mode.html",[18],"Real-time Mode",", added in Spark 4.1, removes the wait for the next batch. It keeps one long-running task per Kafka partition that processes records as they arrive, so the cluster needs at least one free core per partition. In Python, enable it with ",[42,331,332],{},".trigger(realTime=\"5 minutes\")",". The duration sets how often progress is checkpointed, not the latency.",[10,335,336,337,340],{},"In a local test, a query that copied orders over 100 EUR to another topic took a median of 4 ms from order to output in Real-time Mode, and 558 ms as micro-batches. The test ran on a laptop with three Kafka partitions and five orders per second, with no explicit trigger for micro-batches. Latency was measured from each order's ",[42,338,339],{},"createdAt"," to the output record's Kafka timestamp.",[10,342,343,344,347,348,351,352,355,356,20,359,312],{},"Spark 4.3 added aggregations, including ",[42,345,346],{},"window()",", and ",[42,349,350],{},"dropDuplicates"," to the stateless queries Real-time Mode already ran. In the same test, the order totals query took roughly 31 ms in Real-time Mode and 844 ms as micro-batches, estimated from each customer's latest order. Stream-stream joins, session windows, and PySpark's ",[42,353,354],{},"transformWithState"," are still not supported. The mode requires update output, reads only from Kafka, writes to Kafka, foreach, console, or memory sinks, and rejects ",[42,357,358],{},"minPartitions",[42,360,361],{},"maxOffsetsPerTrigger",[10,363,364],{},"Aggregations need more cores, because shuffle tasks run alongside the Kafka readers: 10 Kafka partitions and 5 shuffle partitions need at least 15. Real-time Mode also requires state store checkpoint format v2, while micro-batch queries default to v1, so moving an existing query usually means a new checkpoint.",[292,366,368],{"id":367},"state-recovery-and-scaling","State, recovery, and scaling",[10,370,371,372,374,375,312],{},"Kafka Streams keeps each customer's running total in a local RocksDB store, backed up to a changelog topic. When an instance stops, another replays that topic to rebuild the state. Without repartitioning, the number of ",[42,373,44],{}," partitions caps the number of tasks; see ",[14,376,378],{"href":377},"\u002Fblog\u002Fkafka-streams-in-java#restoration-standby-replicas-and-scaling","restoration and scaling",[10,380,381],{},"Spark keeps state in executor memory by default and writes each batch's changes to the checkpoint location. For millions of keys, the Spark docs recommend the RocksDB state store, which keeps state in native memory and on local disk to avoid long garbage collection pauses. After a restart, the query loads its state from the checkpoint and continues from the offsets saved there.",[10,383,384,385,387,388,390,391,394],{},"In micro-batch mode, Spark can split a busy Kafka partition across several tasks with ",[42,386,358],{}," and cap each batch with ",[42,389,361],{},". Stateful steps are different: their partition count comes from ",[42,392,393],{},"spark.sql.shuffle.partitions"," when the query first starts and is saved in the checkpoint. Changing it means a new checkpoint, which starts the state from empty, so choose it before going live.",[292,396,398],{"id":397},"event-time-windows-and-joins","Event time, windows, and joins",[57,400,401,415],{},[60,402,403],{},[63,404,405,408,411,413],{},[66,406,407],{},"Window type",[66,409,410],{},"Example",[66,412,25],{},[66,414,30],{},[76,416,417,435,454,472],{},[63,418,419,424,427,430],{},[81,420,421],{},[28,422,423],{},"Fixed, no overlap",[81,425,426],{},"10:00–10:05, 10:05–10:10",[81,428,429],{},"Tumbling",[81,431,432],{},[42,433,434],{},"window(createdAt, \"5 minutes\")",[63,436,437,442,445,448],{},[81,438,439],{},[28,440,441],{},"Fixed size, overlapping",[81,443,444],{},"The last 5 minutes, updated every minute",[81,446,447],{},"Hopping",[81,449,450,453],{},[42,451,452],{},"window(createdAt, \"5 minutes\", \"1 minute\")",", called sliding",[63,455,456,461,464,467],{},[81,457,458],{},[28,459,460],{},"Ends after inactivity",[81,462,463],{},"A customer's orders until 30 minutes pass with none",[81,465,466],{},"Session",[81,468,469],{},[42,470,471],{},"session_window(createdAt, \"30 minutes\")",[63,473,474,479,482,485],{},[81,475,476],{},[28,477,478],{},"Based on time between records",[81,480,481],{},"Orders within 5 minutes of each other",[81,483,484],{},"Sliding",[81,486,487],{},"No direct equivalent",[10,489,490,491,493,494,496,497,312],{},"Both place orders in windows by event time, the time on the order, not the wall clock. Spark windows by a column you choose, such as ",[42,492,339],{},". Kafka Streams uses each record's Kafka timestamp by default; to window by a field such as ",[42,495,339],{},", configure a ",[42,498,499],{},"TimestampExtractor",[10,501,502,503,506],{},"They handle late orders differently. Kafka Streams accepts them until a grace period after the window's end. Spark tracks a watermark, the newest order time it has seen minus the delay you pass to ",[42,504,505],{},"withWatermark","; it can drop orders older than the watermark and removes a window's state once the watermark passes the window's end.",[10,508,509,510,513,514,517],{},"The output mode decides when totals reach Kafka. In ",[28,511,512],{},"update mode",", each micro-batch writes the windows it changed, so a customer's total is written again as orders arrive, as Kafka Streams does by default. In ",[28,515,516],{},"append mode",", Spark writes each window once, after the watermark passes its end, so totals arrive at least the watermark delay late.",[10,519,520],{},"Kafka Streams joins usually need both topics partitioned the same way, with the same partition count and partitioner. Spark reshuffles records itself, so it can join any two streams. For a join between two streams, give both sides a watermark and a time limit on matches. Inner joins without them keep all past input in state, and outer joins require them.",[292,522,524],{"id":523},"exactly-once-and-output-latency","Exactly-once and output latency",[10,526,527,528,531,532,535],{},"Exactly-once means each order is counted once in the output, even if the job fails and reprocesses some input. Kafka Streams does this with ",[42,529,530],{},"processing.guarantee=exactly_once_v2",", which commits output and input offsets in one Kafka transaction, every 100 ms by default. Consumers that read with ",[42,533,534],{},"read_committed"," then see each result once.",[10,537,538,539,542,543,546,547,312],{},"Spark updates its state exactly once, but its Kafka sink is at-least-once in both micro-batches and Real-time Mode. If a batch fails after writing some results, Spark reruns it from the offsets in the checkpoint and writes those results again. Include the customer and window in each total, so consumers can keep the latest value for each pair and ignore repeats. Spark's Kafka source also reads with ",[42,540,541],{},"read_uncommitted"," by default, so set ",[42,544,545],{},"kafka.isolation.level=read_committed"," when the input comes from transactional producers. See ",[14,548,550],{"href":549},"\u002Fblog\u002Fkafka-transactions-exactly-once#exactly-once-support-in-kafka-streams-flink-spark-and-connect","exactly-once support by framework",[10,552,553],{},"In micro-batch mode, Spark's output latency follows its batches. A total reaches Kafka while the micro-batch that processes the order runs, so with a 10-second trigger it can take more than 10 seconds. Real-time Mode writes each total as it changes.",[52,555,557],{"id":556},"spark-offsets-and-consumer-lag","Spark offsets and consumer lag",[292,559,561],{"id":560},"where-spark-keeps-its-kafka-offsets","Where Spark keeps its Kafka offsets",[10,563,564,565,568,569,572],{},"Spark's Kafka source does not commit offsets to Kafka. Each query reads with its own generated group ID, starting with ",[42,566,567],{},"spark-kafka-source",", and saves its progress in the checkpoint location instead. With the query below running, ",[42,570,571],{},"kafka-consumer-groups.sh --list"," showed no group for it at all.",[10,574,575,576,312],{},"Here is the order totals query in PySpark. The Kafka connector ships separately, so start Spark with ",[42,577,578],{},"--packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.3.0",[580,581,586],"pre",{"className":582,"code":583,"language":584,"meta":585,"style":585},"language-python shiki shiki-themes kafma-dark kafma-light","from pyspark.sql.functions import col, from_json, struct, sum as sum_, to_json, window\n\norders = (\n    spark.readStream.format(\"kafka\")\n    .option(\"kafka.bootstrap.servers\", \"localhost:9092\")\n    .option(\"subscribe\", \"orders\")\n    .option(\"startingOffsets\", \"earliest\")\n    .load()\n    .select(from_json(col(\"value\").cast(\"string\"),\n                      \"customerId STRING, total DECIMAL(10, 2), createdAt TIMESTAMP\").alias(\"order\"))\n    .select(\"order.*\")\n)\n\ntotals = (\n    orders.withWatermark(\"createdAt\", \"1 minute\")\n    .groupBy(window(\"createdAt\", \"5 minutes\"), \"customerId\")\n    .agg(sum_(\"total\").alias(\"totalAmount\"))\n)\n\nquery = (\n    totals.select(col(\"customerId\").alias(\"key\"),\n                  to_json(struct(\"customerId\", \"window\", \"totalAmount\")).alias(\"value\"))\n    .writeStream.format(\"kafka\")\n    .option(\"kafka.bootstrap.servers\", \"localhost:9092\")\n    .option(\"topic\", \"customer-order-totals\")\n    .option(\"checkpointLocation\", \"s3a:\u002F\u002Fmy-bucket\u002Fcheckpoints\u002Forder-totals\")  # Saves state and Kafka offsets\n    .outputMode(\"update\")\n    .queryName(\"order-totals\")\n    .trigger(processingTime=\"10 seconds\")\n    .start()\n)\n","python","",[42,587,588,650,657,669,695,716,735,754,765,802,821,835,840,845,855,877,907,936,941,946,956,986,1022,1040,1057,1076,1100,1115,1130,1150,1160],{"__ignoreMap":585},[589,590,593,597,601,603,606,608,611,614,617,620,623,625,628,630,634,637,640,642,645,647],"span",{"class":591,"line":592},"line",1,[589,594,596],{"class":595},"srAoB","from",[589,598,600],{"class":599},"spVhG"," pyspark",[589,602,312],{"class":595},[589,604,605],{"class":599},"sql",[589,607,312],{"class":595},[589,609,610],{"class":599},"functions ",[589,612,613],{"class":595},"import",[589,615,616],{"class":599}," col",[589,618,619],{"class":595},",",[589,621,622],{"class":599}," from_json",[589,624,619],{"class":595},[589,626,627],{"class":599}," struct",[589,629,619],{"class":595},[589,631,633],{"class":632},"sS_Im"," sum",[589,635,636],{"class":595}," as",[589,638,639],{"class":599}," sum_",[589,641,619],{"class":595},[589,643,644],{"class":599}," to_json",[589,646,619],{"class":595},[589,648,649],{"class":599}," window\n",[589,651,653],{"class":591,"line":652},2,[589,654,656],{"emptyLinePlaceholder":655},true,"\n",[589,658,660,663,666],{"class":591,"line":659},3,[589,661,662],{"class":599},"orders ",[589,664,665],{"class":595},"=",[589,667,668],{"class":595}," (\n",[589,670,672,675,677,680,682,685,688,692],{"class":591,"line":671},4,[589,673,674],{"class":599},"    spark",[589,676,312],{"class":595},[589,678,679],{"class":599},"readStream",[589,681,312],{"class":595},[589,683,684],{"class":599},"format",[589,686,687],{"class":595},"(",[589,689,691],{"class":690},"sIsOz","\"kafka\"",[589,693,694],{"class":595},")\n",[589,696,698,701,704,706,709,711,714],{"class":591,"line":697},5,[589,699,700],{"class":595},"    .",[589,702,703],{"class":599},"option",[589,705,687],{"class":595},[589,707,708],{"class":690},"\"kafka.bootstrap.servers\"",[589,710,619],{"class":595},[589,712,713],{"class":690}," \"localhost:9092\"",[589,715,694],{"class":595},[589,717,719,721,723,725,728,730,733],{"class":591,"line":718},6,[589,720,700],{"class":595},[589,722,703],{"class":599},[589,724,687],{"class":595},[589,726,727],{"class":690},"\"subscribe\"",[589,729,619],{"class":595},[589,731,732],{"class":690}," \"orders\"",[589,734,694],{"class":595},[589,736,738,740,742,744,747,749,752],{"class":591,"line":737},7,[589,739,700],{"class":595},[589,741,703],{"class":599},[589,743,687],{"class":595},[589,745,746],{"class":690},"\"startingOffsets\"",[589,748,619],{"class":595},[589,750,751],{"class":690}," \"earliest\"",[589,753,694],{"class":595},[589,755,757,759,762],{"class":591,"line":756},8,[589,758,700],{"class":595},[589,760,761],{"class":599},"load",[589,763,764],{"class":595},"()\n",[589,766,768,770,773,775,778,780,783,785,788,791,794,796,799],{"class":591,"line":767},9,[589,769,700],{"class":595},[589,771,772],{"class":599},"select",[589,774,687],{"class":595},[589,776,777],{"class":599},"from_json",[589,779,687],{"class":595},[589,781,782],{"class":599},"col",[589,784,687],{"class":595},[589,786,787],{"class":690},"\"value\"",[589,789,790],{"class":595},").",[589,792,793],{"class":599},"cast",[589,795,687],{"class":595},[589,797,798],{"class":690},"\"string\"",[589,800,801],{"class":595},"),\n",[589,803,805,808,810,813,815,818],{"class":591,"line":804},10,[589,806,807],{"class":690},"                      \"customerId STRING, total DECIMAL(10, 2), createdAt TIMESTAMP\"",[589,809,790],{"class":595},[589,811,812],{"class":599},"alias",[589,814,687],{"class":595},[589,816,817],{"class":690},"\"order\"",[589,819,820],{"class":595},"))\n",[589,822,824,826,828,830,833],{"class":591,"line":823},11,[589,825,700],{"class":595},[589,827,772],{"class":599},[589,829,687],{"class":595},[589,831,832],{"class":690},"\"order.*\"",[589,834,694],{"class":595},[589,836,838],{"class":591,"line":837},12,[589,839,694],{"class":595},[589,841,843],{"class":591,"line":842},13,[589,844,656],{"emptyLinePlaceholder":655},[589,846,848,851,853],{"class":591,"line":847},14,[589,849,850],{"class":599},"totals ",[589,852,665],{"class":595},[589,854,668],{"class":595},[589,856,858,861,863,865,867,870,872,875],{"class":591,"line":857},15,[589,859,860],{"class":599},"    orders",[589,862,312],{"class":595},[589,864,505],{"class":599},[589,866,687],{"class":595},[589,868,869],{"class":690},"\"createdAt\"",[589,871,619],{"class":595},[589,873,874],{"class":690}," \"1 minute\"",[589,876,694],{"class":595},[589,878,880,882,885,887,890,892,894,896,899,902,905],{"class":591,"line":879},16,[589,881,700],{"class":595},[589,883,884],{"class":599},"groupBy",[589,886,687],{"class":595},[589,888,889],{"class":599},"window",[589,891,687],{"class":595},[589,893,869],{"class":690},[589,895,619],{"class":595},[589,897,898],{"class":690}," \"5 minutes\"",[589,900,901],{"class":595},"),",[589,903,904],{"class":690}," \"customerId\"",[589,906,694],{"class":595},[589,908,910,912,915,917,920,922,925,927,929,931,934],{"class":591,"line":909},17,[589,911,700],{"class":595},[589,913,914],{"class":599},"agg",[589,916,687],{"class":595},[589,918,919],{"class":599},"sum_",[589,921,687],{"class":595},[589,923,924],{"class":690},"\"total\"",[589,926,790],{"class":595},[589,928,812],{"class":599},[589,930,687],{"class":595},[589,932,933],{"class":690},"\"totalAmount\"",[589,935,820],{"class":595},[589,937,939],{"class":591,"line":938},18,[589,940,694],{"class":595},[589,942,944],{"class":591,"line":943},19,[589,945,656],{"emptyLinePlaceholder":655},[589,947,949,952,954],{"class":591,"line":948},20,[589,950,951],{"class":599},"query ",[589,953,665],{"class":595},[589,955,668],{"class":595},[589,957,959,962,964,966,968,970,972,975,977,979,981,984],{"class":591,"line":958},21,[589,960,961],{"class":599},"    totals",[589,963,312],{"class":595},[589,965,772],{"class":599},[589,967,687],{"class":595},[589,969,782],{"class":599},[589,971,687],{"class":595},[589,973,974],{"class":690},"\"customerId\"",[589,976,790],{"class":595},[589,978,812],{"class":599},[589,980,687],{"class":595},[589,982,983],{"class":690},"\"key\"",[589,985,801],{"class":595},[589,987,989,992,994,997,999,1001,1003,1006,1008,1011,1014,1016,1018,1020],{"class":591,"line":988},22,[589,990,991],{"class":599},"                  to_json",[589,993,687],{"class":595},[589,995,996],{"class":599},"struct",[589,998,687],{"class":595},[589,1000,974],{"class":690},[589,1002,619],{"class":595},[589,1004,1005],{"class":690}," \"window\"",[589,1007,619],{"class":595},[589,1009,1010],{"class":690}," \"totalAmount\"",[589,1012,1013],{"class":595},")).",[589,1015,812],{"class":599},[589,1017,687],{"class":595},[589,1019,787],{"class":690},[589,1021,820],{"class":595},[589,1023,1025,1027,1030,1032,1034,1036,1038],{"class":591,"line":1024},23,[589,1026,700],{"class":595},[589,1028,1029],{"class":599},"writeStream",[589,1031,312],{"class":595},[589,1033,684],{"class":599},[589,1035,687],{"class":595},[589,1037,691],{"class":690},[589,1039,694],{"class":595},[589,1041,1043,1045,1047,1049,1051,1053,1055],{"class":591,"line":1042},24,[589,1044,700],{"class":595},[589,1046,703],{"class":599},[589,1048,687],{"class":595},[589,1050,708],{"class":690},[589,1052,619],{"class":595},[589,1054,713],{"class":690},[589,1056,694],{"class":595},[589,1058,1060,1062,1064,1066,1069,1071,1074],{"class":591,"line":1059},25,[589,1061,700],{"class":595},[589,1063,703],{"class":599},[589,1065,687],{"class":595},[589,1067,1068],{"class":690},"\"topic\"",[589,1070,619],{"class":595},[589,1072,1073],{"class":690}," \"customer-order-totals\"",[589,1075,694],{"class":595},[589,1077,1079,1081,1083,1085,1088,1090,1093,1096],{"class":591,"line":1078},26,[589,1080,700],{"class":595},[589,1082,703],{"class":599},[589,1084,687],{"class":595},[589,1086,1087],{"class":690},"\"checkpointLocation\"",[589,1089,619],{"class":595},[589,1091,1092],{"class":690}," \"s3a:\u002F\u002Fmy-bucket\u002Fcheckpoints\u002Forder-totals\"",[589,1094,1095],{"class":595},")",[589,1097,1099],{"class":1098},"sqpaY","  # Saves state and Kafka offsets\n",[589,1101,1103,1105,1108,1110,1113],{"class":591,"line":1102},27,[589,1104,700],{"class":595},[589,1106,1107],{"class":599},"outputMode",[589,1109,687],{"class":595},[589,1111,1112],{"class":690},"\"update\"",[589,1114,694],{"class":595},[589,1116,1118,1120,1123,1125,1128],{"class":591,"line":1117},28,[589,1119,700],{"class":595},[589,1121,1122],{"class":599},"queryName",[589,1124,687],{"class":595},[589,1126,1127],{"class":690},"\"order-totals\"",[589,1129,694],{"class":595},[589,1131,1133,1135,1138,1140,1143,1145,1148],{"class":591,"line":1132},29,[589,1134,700],{"class":595},[589,1136,1137],{"class":599},"trigger",[589,1139,687],{"class":595},[589,1141,1142],{"class":599},"processingTime",[589,1144,665],{"class":595},[589,1146,1147],{"class":690},"\"10 seconds\"",[589,1149,694],{"class":595},[589,1151,1153,1155,1158],{"class":591,"line":1152},30,[589,1154,700],{"class":595},[589,1156,1157],{"class":599},"start",[589,1159,764],{"class":595},[589,1161,1163],{"class":591,"line":1162},31,[589,1164,694],{"class":595},[10,1166,1167,1168,1171,1172,1175,1176,1179],{},"Before each micro-batch, Spark writes a file to ",[42,1169,1170],{},"offsets\u002F"," in the checkpoint, named after the batch number. Its last line holds the end offset of every partition, such as ",[42,1173,1174],{},"{\"orders\":{\"0\":203,\"1\":185,\"2\":0}}",". When the batch finishes, Spark writes a file with the same number to ",[42,1177,1178],{},"commits\u002F",", and after a restart it reruns a batch that has no commit file.",[10,1181,1182,1185,1186,1189],{},[42,1183,1184],{},"startingOffsets"," applies only to a new checkpoint. After that, the query always resumes where its checkpoint says. To reprocess a topic, start the query with a new checkpoint location, which also starts its state from empty. Kafka Streams uses ",[42,1187,1188],{},"kafka-streams-application-reset"," instead: stop every instance, run the tool to rewind input offsets and delete internal topics, then clear each instance's local state.",[292,1191,1193],{"id":1192},"monitor-lag-from-query-progress","Monitor lag from query progress",[10,1195,1196,1197,1200,1201,1204,1205,1208,1209,347,1212,312],{},"After each micro-batch, Spark reports a ",[42,1198,1199],{},"StreamingQueryProgress",". For a Kafka source, its ",[42,1202,1203],{},"endOffset"," holds the offsets the batch read up to, and its metrics include ",[42,1206,1207],{},"minOffsetsBehindLatest",", ",[42,1210,1211],{},"maxOffsetsBehindLatest",[42,1213,1214],{},"avgOffsetsBehindLatest",[10,1216,1217,1218,1223,1224,1226],{},"In micro-batch mode, these metrics do not measure lag against the current end of the log. ",[14,1219,1222],{"href":1220,"rel":1221},"https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark\u002Fblob\u002Fv4.3.0\u002Fconnector\u002Fkafka-0-10-sql\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fsql\u002Fkafka010\u002FKafkaMicroBatchStream.scala#L512",[18],"Spark's Kafka source"," compares the batch's end offsets with the newest offsets Spark saw when it planned the batch. Without ",[42,1225,361],{},", a batch reads everything available at that moment, so the metrics stay at 0. In the test, they read 0 while the query trailed the log by up to about 50 orders between 10-second batches.",[10,1228,1229],{},"To see lag against the end of the log, copy each batch's end offsets into a consumer group. Kafka tools then compute its lag the same way as for any other consumer.",[292,1231,1233],{"id":1232},"commit-spark-progress-to-a-consumer-group","Commit Spark progress to a consumer group",[10,1235,1236,1237,1240,1241,1244,1245,1248,1249,1252],{},"A ",[42,1238,1239],{},"StreamingQueryListener"," runs on the driver and is called after each micro-batch of every query in the session. This one picks the ",[42,1242,1243],{},"order-totals"," query by name and writes its end offsets to a consumer group used only for monitoring, with the admin client from ",[42,1246,1247],{},"confluent-kafka",". Register it before calling ",[42,1250,1251],{},"start()",". It assumes the query reads only from Kafka. In Real-time Mode it runs once per trigger duration, so the group updates that often and its lag can include records already processed.",[580,1254,1256],{"className":582,"code":1255,"language":584,"meta":585,"style":585},"import json\n\nfrom confluent_kafka import ConsumerGroupTopicPartitions, TopicPartition\nfrom confluent_kafka.admin import AdminClient\nfrom pyspark.sql.streaming import StreamingQueryListener\n\n\nclass CommitOffsets(StreamingQueryListener):\n    def __init__(self, bootstrap_servers, group_id, query_name):\n        self.admin = AdminClient({\"bootstrap.servers\": bootstrap_servers})\n        self.group_id = group_id\n        self.query_name = query_name\n\n    def onQueryStarted(self, event): pass\n    def onQueryIdle(self, event): pass\n    def onQueryTerminated(self, event): pass\n\n    def onQueryProgress(self, event):\n        if event.progress.name != self.query_name:\n            return\n        for source in event.progress.sources:\n            end = json.loads(source.endOffset)\n            partitions = [\n                TopicPartition(topic, int(partition), offset)\n                for topic, offsets in end.items()\n                for partition, offset in offsets.items()\n            ]\n            request = ConsumerGroupTopicPartitions(self.group_id, partitions)\n            for future in self.admin.alter_consumer_group_offsets([request]).values():\n                future.result()\n\n\nspark.streams.addListener(CommitOffsets(\"localhost:9092\", \"order-totals-spark\", \"order-totals\"))\n",[42,1257,1258,1265,1269,1286,1303,1323,1327,1331,1346,1376,1404,1418,1432,1436,1458,1477,1496,1500,1517,1548,1553,1577,1603,1613,1640,1665,1688,1693,1718,1755,1767,1771,1776],{"__ignoreMap":585},[589,1259,1260,1262],{"class":591,"line":592},[589,1261,613],{"class":595},[589,1263,1264],{"class":599}," json\n",[589,1266,1267],{"class":591,"line":652},[589,1268,656],{"emptyLinePlaceholder":655},[589,1270,1271,1273,1276,1278,1281,1283],{"class":591,"line":659},[589,1272,596],{"class":595},[589,1274,1275],{"class":599}," confluent_kafka ",[589,1277,613],{"class":595},[589,1279,1280],{"class":599}," ConsumerGroupTopicPartitions",[589,1282,619],{"class":595},[589,1284,1285],{"class":599}," TopicPartition\n",[589,1287,1288,1290,1293,1295,1298,1300],{"class":591,"line":671},[589,1289,596],{"class":595},[589,1291,1292],{"class":599}," confluent_kafka",[589,1294,312],{"class":595},[589,1296,1297],{"class":599},"admin ",[589,1299,613],{"class":595},[589,1301,1302],{"class":599}," AdminClient\n",[589,1304,1305,1307,1309,1311,1313,1315,1318,1320],{"class":591,"line":697},[589,1306,596],{"class":595},[589,1308,600],{"class":599},[589,1310,312],{"class":595},[589,1312,605],{"class":599},[589,1314,312],{"class":595},[589,1316,1317],{"class":599},"streaming ",[589,1319,613],{"class":595},[589,1321,1322],{"class":599}," StreamingQueryListener\n",[589,1324,1325],{"class":591,"line":718},[589,1326,656],{"emptyLinePlaceholder":655},[589,1328,1329],{"class":591,"line":737},[589,1330,656],{"emptyLinePlaceholder":655},[589,1332,1333,1336,1339,1341,1343],{"class":591,"line":756},[589,1334,1335],{"class":595},"class",[589,1337,1338],{"class":632}," CommitOffsets",[589,1340,687],{"class":595},[589,1342,1239],{"class":599},[589,1344,1345],{"class":595},"):\n",[589,1347,1348,1351,1354,1356,1359,1361,1364,1366,1369,1371,1374],{"class":591,"line":767},[589,1349,1350],{"class":595},"    def",[589,1352,1353],{"class":632}," __init__",[589,1355,687],{"class":595},[589,1357,1358],{"class":599},"self",[589,1360,619],{"class":595},[589,1362,1363],{"class":599}," bootstrap_servers",[589,1365,619],{"class":595},[589,1367,1368],{"class":599}," group_id",[589,1370,619],{"class":595},[589,1372,1373],{"class":599}," query_name",[589,1375,1345],{"class":595},[589,1377,1378,1381,1383,1385,1387,1390,1393,1396,1399,1401],{"class":591,"line":804},[589,1379,1380],{"class":599},"        self",[589,1382,312],{"class":595},[589,1384,1297],{"class":599},[589,1386,665],{"class":595},[589,1388,1389],{"class":599}," AdminClient",[589,1391,1392],{"class":595},"({",[589,1394,1395],{"class":690},"\"bootstrap.servers\"",[589,1397,1398],{"class":595},":",[589,1400,1363],{"class":599},[589,1402,1403],{"class":595},"})\n",[589,1405,1406,1408,1410,1413,1415],{"class":591,"line":823},[589,1407,1380],{"class":599},[589,1409,312],{"class":595},[589,1411,1412],{"class":599},"group_id ",[589,1414,665],{"class":595},[589,1416,1417],{"class":599}," group_id\n",[589,1419,1420,1422,1424,1427,1429],{"class":591,"line":837},[589,1421,1380],{"class":599},[589,1423,312],{"class":595},[589,1425,1426],{"class":599},"query_name ",[589,1428,665],{"class":595},[589,1430,1431],{"class":599}," query_name\n",[589,1433,1434],{"class":591,"line":842},[589,1435,656],{"emptyLinePlaceholder":655},[589,1437,1438,1440,1443,1445,1447,1449,1452,1455],{"class":591,"line":847},[589,1439,1350],{"class":595},[589,1441,1442],{"class":632}," onQueryStarted",[589,1444,687],{"class":595},[589,1446,1358],{"class":599},[589,1448,619],{"class":595},[589,1450,1451],{"class":599}," event",[589,1453,1454],{"class":595},"):",[589,1456,1457],{"class":595}," pass\n",[589,1459,1460,1462,1465,1467,1469,1471,1473,1475],{"class":591,"line":857},[589,1461,1350],{"class":595},[589,1463,1464],{"class":632}," onQueryIdle",[589,1466,687],{"class":595},[589,1468,1358],{"class":599},[589,1470,619],{"class":595},[589,1472,1451],{"class":599},[589,1474,1454],{"class":595},[589,1476,1457],{"class":595},[589,1478,1479,1481,1484,1486,1488,1490,1492,1494],{"class":591,"line":879},[589,1480,1350],{"class":595},[589,1482,1483],{"class":632}," onQueryTerminated",[589,1485,687],{"class":595},[589,1487,1358],{"class":599},[589,1489,619],{"class":595},[589,1491,1451],{"class":599},[589,1493,1454],{"class":595},[589,1495,1457],{"class":595},[589,1497,1498],{"class":591,"line":909},[589,1499,656],{"emptyLinePlaceholder":655},[589,1501,1502,1504,1507,1509,1511,1513,1515],{"class":591,"line":938},[589,1503,1350],{"class":595},[589,1505,1506],{"class":632}," onQueryProgress",[589,1508,687],{"class":595},[589,1510,1358],{"class":599},[589,1512,619],{"class":595},[589,1514,1451],{"class":599},[589,1516,1345],{"class":595},[589,1518,1519,1522,1524,1526,1529,1531,1534,1537,1540,1542,1545],{"class":591,"line":943},[589,1520,1521],{"class":595},"        if",[589,1523,1451],{"class":599},[589,1525,312],{"class":595},[589,1527,1528],{"class":599},"progress",[589,1530,312],{"class":595},[589,1532,1533],{"class":599},"name ",[589,1535,1536],{"class":595},"!=",[589,1538,1539],{"class":599}," self",[589,1541,312],{"class":595},[589,1543,1544],{"class":599},"query_name",[589,1546,1547],{"class":595},":\n",[589,1549,1550],{"class":591,"line":948},[589,1551,1552],{"class":595},"            return\n",[589,1554,1555,1558,1561,1564,1566,1568,1570,1572,1575],{"class":591,"line":958},[589,1556,1557],{"class":595},"        for",[589,1559,1560],{"class":599}," source ",[589,1562,1563],{"class":595},"in",[589,1565,1451],{"class":599},[589,1567,312],{"class":595},[589,1569,1528],{"class":599},[589,1571,312],{"class":595},[589,1573,1574],{"class":599},"sources",[589,1576,1547],{"class":595},[589,1578,1579,1582,1584,1587,1589,1592,1594,1597,1599,1601],{"class":591,"line":988},[589,1580,1581],{"class":599},"            end ",[589,1583,665],{"class":595},[589,1585,1586],{"class":599}," json",[589,1588,312],{"class":595},[589,1590,1591],{"class":599},"loads",[589,1593,687],{"class":595},[589,1595,1596],{"class":599},"source",[589,1598,312],{"class":595},[589,1600,1203],{"class":599},[589,1602,694],{"class":595},[589,1604,1605,1608,1610],{"class":591,"line":1024},[589,1606,1607],{"class":599},"            partitions ",[589,1609,665],{"class":595},[589,1611,1612],{"class":595}," [\n",[589,1614,1615,1618,1620,1623,1625,1628,1630,1633,1635,1638],{"class":591,"line":1042},[589,1616,1617],{"class":599},"                TopicPartition",[589,1619,687],{"class":595},[589,1621,1622],{"class":599},"topic",[589,1624,619],{"class":595},[589,1626,1627],{"class":632}," int",[589,1629,687],{"class":595},[589,1631,1632],{"class":599},"partition",[589,1634,901],{"class":595},[589,1636,1637],{"class":599}," offset",[589,1639,694],{"class":595},[589,1641,1642,1645,1648,1650,1653,1655,1658,1660,1663],{"class":591,"line":1059},[589,1643,1644],{"class":595},"                for",[589,1646,1647],{"class":599}," topic",[589,1649,619],{"class":595},[589,1651,1652],{"class":599}," offsets ",[589,1654,1563],{"class":595},[589,1656,1657],{"class":599}," end",[589,1659,312],{"class":595},[589,1661,1662],{"class":599},"items",[589,1664,764],{"class":595},[589,1666,1667,1669,1672,1674,1677,1679,1682,1684,1686],{"class":591,"line":1078},[589,1668,1644],{"class":595},[589,1670,1671],{"class":599}," partition",[589,1673,619],{"class":595},[589,1675,1676],{"class":599}," offset ",[589,1678,1563],{"class":595},[589,1680,1681],{"class":599}," offsets",[589,1683,312],{"class":595},[589,1685,1662],{"class":599},[589,1687,764],{"class":595},[589,1689,1690],{"class":591,"line":1102},[589,1691,1692],{"class":595},"            ]\n",[589,1694,1695,1698,1700,1702,1704,1706,1708,1711,1713,1716],{"class":591,"line":1117},[589,1696,1697],{"class":599},"            request ",[589,1699,665],{"class":595},[589,1701,1280],{"class":599},[589,1703,687],{"class":595},[589,1705,1358],{"class":599},[589,1707,312],{"class":595},[589,1709,1710],{"class":599},"group_id",[589,1712,619],{"class":595},[589,1714,1715],{"class":599}," partitions",[589,1717,694],{"class":595},[589,1719,1720,1723,1726,1728,1730,1732,1735,1737,1740,1743,1746,1749,1752],{"class":591,"line":1132},[589,1721,1722],{"class":595},"            for",[589,1724,1725],{"class":599}," future ",[589,1727,1563],{"class":595},[589,1729,1539],{"class":599},[589,1731,312],{"class":595},[589,1733,1734],{"class":599},"admin",[589,1736,312],{"class":595},[589,1738,1739],{"class":599},"alter_consumer_group_offsets",[589,1741,1742],{"class":595},"([",[589,1744,1745],{"class":599},"request",[589,1747,1748],{"class":595},"]).",[589,1750,1751],{"class":599},"values",[589,1753,1754],{"class":595},"():\n",[589,1756,1757,1760,1762,1765],{"class":591,"line":1152},[589,1758,1759],{"class":599},"                future",[589,1761,312],{"class":595},[589,1763,1764],{"class":599},"result",[589,1766,764],{"class":595},[589,1768,1769],{"class":591,"line":1162},[589,1770,656],{"emptyLinePlaceholder":655},[589,1772,1774],{"class":591,"line":1773},32,[589,1775,656],{"emptyLinePlaceholder":655},[589,1777,1779,1782,1784,1787,1789,1792,1794,1797,1799,1802,1804,1807,1809,1812],{"class":591,"line":1778},33,[589,1780,1781],{"class":599},"spark",[589,1783,312],{"class":595},[589,1785,1786],{"class":599},"streams",[589,1788,312],{"class":595},[589,1790,1791],{"class":599},"addListener",[589,1793,687],{"class":595},[589,1795,1796],{"class":599},"CommitOffsets",[589,1798,687],{"class":595},[589,1800,1801],{"class":690},"\"localhost:9092\"",[589,1803,619],{"class":595},[589,1805,1806],{"class":690}," \"order-totals-spark\"",[589,1808,619],{"class":595},[589,1810,1811],{"class":690}," \"order-totals\"",[589,1813,820],{"class":595},[10,1815,1816],{},"The group has no members, and its offsets are as of the last finished batch. Spark still restarts from its checkpoint, so these offsets are only for monitoring. Resetting the group does not change what the query reads.",[52,1818,1820],{"id":1819},"inspect-spark-and-kafka-streams-topics-with-kafma","Inspect Spark and Kafka Streams topics with Kafma",[10,1822,1823,1824,1828,1829,1831,1832,1836,1837,1841],{},"Kafma is a desktop ",[14,1825,1827],{"href":1826},"\u002F","Kafka IDE",". Kafka Streams commits its offsets to a consumer group named after ",[42,1830,241],{},", so Kafma shows its lag with no extra code; see ",[14,1833,1835],{"href":1834},"\u002Fblog\u002Fkafka-streams-in-java#inspect-kafka-streams-topics-with-kafma","Inspect Kafka Streams topics with Kafma",". Spark doesn't commit offsets, so for a Spark query, open the group that the ",[14,1838,1840],{"href":1839},"#commit-spark-progress-to-a-consumer-group","listener above"," writes to.",[292,1843,1845],{"id":1844},"watch-a-spark-querys-progress","Watch a Spark query's progress",[10,1847,1848,1849,1853,1854,1856],{},"Open ",[14,1850,1852],{"href":1851},"\u002Fdocs\u002Fconsole\u002Fconsuming-messages#live-and-watch-group-modes","Watch Group"," on ",[42,1855,44],{}," with the listener's group. Records above the divider are not yet in a finished batch, and records below it are processed. The lag chart rises between batches and drops when one finishes. If its low points keep rising, the query is falling behind.",[10,1858,1859],{},[1860,1861],"img",{"alt":1862,"height":1863,"src":1864,"width":1865},"Kafma Watch Group on orders for order-totals-spark, a group with no active members whose lag rises and falls with each Spark micro-batch",1800,"https:\u002F\u002Fmedia.kafma.app\u002Fblog\u002Fspark-vs-kafka-streams\u002Fwatch-group-spark-query.png",3024,[292,1867,1869],{"id":1868},"find-the-records-a-failing-batch-reads","Find the records a failing batch reads",[10,1871,1872,1873,1875,1876,1878,1879,1881],{},"When a micro-batch keeps failing, find the newest file in ",[42,1874,1170],{}," that has no matching file in ",[42,1877,1178],{},". It holds where the failing batch ends, and the previous offsets file holds where it starts. The first batch has no previous offsets file; its start offsets are resolved from ",[42,1880,1184],{}," when the query initializes.",[10,1883,1884,1885,1889,1890,1892,1893,1896,1897,1900],{},"In the ",[14,1886,1888],{"href":1887},"\u002Ffeatures\u002Fkafka-console","Kafka console",", open ",[42,1891,44],{},", filter to one partition, set ",[28,1894,1895],{},"From"," to ",[28,1898,1899],{},"Offset",", and enter that partition's start offset. The batch reads from there up to, but not including, the end offset.",[10,1902,1903,1907],{},[14,1904,1906],{"href":1905},"\u002Fdownload","Download Kafma"," and connect to the cluster your Spark queries read from.",[52,1909,1911],{"id":1910},"frequently-asked-questions","Frequently asked questions",[292,1913,1915],{"id":1914},"what-is-the-difference-between-spark-and-kafka","What is the difference between Spark and Kafka?",[10,1917,1918],{},"Kafka stores events durably in topics and delivers them to consumers. Spark is a processing engine: Structured Streaming reads those events, computes results such as windowed totals, and writes them to Kafka or other systems.",[292,1920,1922],{"id":1921},"is-spark-streaming-the-same-as-structured-streaming","Is Spark Streaming the same as Structured Streaming?",[10,1924,1925],{},"No. Spark Streaming is the older DStream API, which the Spark docs call a legacy project that no longer gets updates. Structured Streaming is the current engine, built on DataFrames and SQL, and it is what this article compares. Many people still say Spark Streaming for both.",[292,1927,1929],{"id":1928},"can-spark-replace-kafka-streams","Can Spark replace Kafka Streams?",[10,1931,1932],{},"For most workloads, yes, if you can run a Spark cluster and accept micro-batch latency or Real-time Mode's limits. You lose exactly-once output to Kafka, and Kafka Streams state does not carry over, so plan to rebuild it by reprocessing the input.",[292,1934,1936],{"id":1935},"does-spark-commit-offsets-to-kafka","Does Spark commit offsets to Kafka?",[10,1938,1939,1940,1942],{},"No. The Kafka source keeps offsets in the query's checkpoint and doesn't commit any to Kafka. To show lag in Kafka tools, copy each batch's end offsets into a consumer group from a ",[42,1941,1239],{},", as shown above.",[292,1944,1946],{"id":1945},"can-i-use-kafka-streams-from-python","Can I use Kafka Streams from Python?",[10,1948,1949,1950,312],{},"No. Kafka Streams runs only on the JVM. Python teams use PySpark, PyFlink, or a Python library such as Quix Streams; see ",[14,1951,311],{"href":310},[292,1953,1955],{"id":1954},"which-is-faster-spark-or-kafka-streams","Which is faster, Spark or Kafka Streams?",[10,1957,1958],{},"For stateless steps, Kafka Streams and Spark Real-time Mode handle each record as it arrives, while micro-batches take about 100 ms at best. Aggregated results wait longer: Kafka Streams buffers them until a commit or a full cache flushes them, and Spark micro-batches write them while the next batch runs. Spark's release notes put Real-time Mode aggregations on the order of 100 ms. For throughput, compare both on your own workload with the same delivery guarantees.",[52,1960,1962],{"id":1961},"conclusion","Conclusion",[10,1964,1965],{},"Choose Kafka Streams when processing belongs in a JVM service whose topics are all in one cluster, or when you need exactly-once output to Kafka. Choose Spark for Python, for combining Kafka with files and tables, or when your team already runs Spark.",[10,1967,1968,1969,1971],{},"Either way, decide on the settings that are hard to change later: input partition counts for Kafka Streams, and ",[42,1970,393],{}," and the checkpoint location for Spark. If you want Spark's lag in the same tools as your other consumers, add the listener before the first run.",[10,1973,1974],{},"This guide is maintained by the team behind Kafma.",[1976,1977,1978],"style",{},"html pre.shiki code .srAoB, html code.shiki .srAoB{--shiki-default:#8A8F98;--shiki-light:#5B6069}html pre.shiki code .spVhG, html code.shiki .spVhG{--shiki-default:#E7E8EA;--shiki-light:#1A1C22}html pre.shiki code .sS_Im, html code.shiki .sS_Im{--shiki-default:#FFFFFF;--shiki-light:#0A0B10}html pre.shiki code .sIsOz, html code.shiki .sIsOz{--shiki-default:#83CFAE;--shiki-light:#1F7A52}html pre.shiki code .sqpaY, html code.shiki .sqpaY{--shiki-default:#787D86;--shiki-light:#6A6F79}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}",{"title":585,"searchDepth":659,"depth":659,"links":1980},[1981,1982,1989,1994,1998,2006],{"id":54,"depth":652,"text":55},{"id":143,"depth":652,"text":144,"children":1983},[1984,1985,1986,1987,1988],{"id":294,"depth":659,"text":295},{"id":315,"depth":659,"text":316},{"id":367,"depth":659,"text":368},{"id":397,"depth":659,"text":398},{"id":523,"depth":659,"text":524},{"id":556,"depth":652,"text":557,"children":1990},[1991,1992,1993],{"id":560,"depth":659,"text":561},{"id":1192,"depth":659,"text":1193},{"id":1232,"depth":659,"text":1233},{"id":1819,"depth":652,"text":1820,"children":1995},[1996,1997],{"id":1844,"depth":659,"text":1845},{"id":1868,"depth":659,"text":1869},{"id":1910,"depth":652,"text":1911,"children":1999},[2000,2001,2002,2003,2004,2005],{"id":1914,"depth":659,"text":1915},{"id":1921,"depth":659,"text":1922},{"id":1928,"depth":659,"text":1929},{"id":1935,"depth":659,"text":1936},{"id":1945,"depth":659,"text":1946},{"id":1954,"depth":659,"text":1955},{"id":1961,"depth":652,"text":1962},"2026-11-06","Spark vs Kafka Streams: Structured Streaming on a cluster or a library in your app. Compare Real-time Mode, Python, state, offsets and lag, and exactly-once.","md",{"draft":655},null,"\u002Fblog\u002Fspark-vs-kafka-streams",false,{"title":5,"description":2008},"blog\u002Fspark-vs-kafka-streams",[2017,19,25],"Kafka Stream Processing","i5KaU9WznkVV0w4Br2xW9pJDjIv8TW6WsJJy9B7z5aw",1793956251516]