[{"data":1,"prerenderedAt":1164},["ShallowReactive",2],{"blog-post-\u002Fblog\u002Fkafka-transactions-exactly-once":3},{"id":4,"title":5,"body":6,"date":1148,"description":1149,"extension":1150,"meta":1151,"navigation":1152,"ogImage":1153,"ogImageHeight":1153,"ogImageWidth":1153,"path":1154,"pinned":1155,"seo":1156,"sitemap":1157,"stem":1158,"tags":1159,"__hash__":1163},"blog\u002Fblog\u002Fkafka-transactions-exactly-once.md","Kafka Transactions, Idempotence, and Exactly-Once Semantics",{"type":7,"value":8,"toc":1121},"minimark",[9,18,21,40,45,48,142,151,155,160,163,181,185,207,218,224,233,237,250,256,259,265,269,272,278,434,437,440,447,461,467,470,475,616,619,659,662,666,670,676,686,692,696,702,709,717,726,729,736,740,747,806,809,820,868,872,875,1004,1019,1026,1030,1034,1037,1041,1047,1051,1061,1065,1071,1075,1081,1085,1095,1099,1105,1114,1117],[10,11,12,13,17],"p",{},"Exactly-once semantics in Kafka means each input record affects the output once, even after retries and restarts. It needs all three of the following together: the producer writes its output in transactions, the input offsets are saved together with the output, and the consumers that read the output use ",[14,15,16],"code",{},"read_committed",". If any one is missing, a retry or restart can still duplicate output.",[10,19,20],{},"The guarantee covers reading from and writing to topics in one Kafka cluster. Writes to a database or to another cluster are outside it and need their own protection against duplicates.",[10,22,23,24,28,29,28,32,35,36,39],{},"This article compares what each setup guarantees, from plain clients to ",[25,26,27],"strong",{},"Kafka Streams",", ",[25,30,31],{},"Flink",[25,33,34],{},"Spark",", and ",[25,37,38],{},"Kafka Connect",", and how client defaults differ. It then shows how to find and fix transactions that block consumers.",[41,42,44],"h2",{"id":43},"kafka-delivery-guarantees-by-setup","Kafka delivery guarantees by setup",[10,46,47],{},"A consumer commits offsets to record how far it has read. Whether it commits before or after processing decides what a crash does to the records in between.",[49,50,51,67],"table",{},[52,53,54],"thead",{},[55,56,57,61,64],"tr",{},[58,59,60],"th",{},"Setup",[58,62,63],{},"Guarantee",[58,65,66],{},"Main caveat",[68,69,70,82,93,104,121,132],"tbody",{},[55,71,72,76,79],{},[73,74,75],"td",{},"Consumer commits offsets before processing",[73,77,78],{},"At-most-once",[73,80,81],{},"A crash after the commit skips the records being processed",[55,83,84,87,90],{},[73,85,86],{},"Consumer commits offsets after processing",[73,88,89],{},"At-least-once",[73,91,92],{},"A crash before the commit processes some records again",[55,94,95,98,101],{},[73,96,97],{},"Idempotent producer, the Java default",[73,99,100],{},"No duplicates from producer retries",[73,102,103],{},"Restarts and application resends can still write duplicates",[55,105,106,112,115],{},[73,107,108,109,111],{},"Transactional producer and ",[14,110,16],{}," consumers",[73,113,114],{},"Writes to several partitions are committed or aborted together",[73,116,117,120],{},[14,118,119],{},"read_uncommitted"," consumers also see aborted records",[55,122,123,126,129],{},[73,124,125],{},"Consume, process, and produce in one transaction",[73,127,128],{},"Exactly-once from Kafka to Kafka",[73,130,131],{},"One cluster only. Database writes or HTTP calls made while processing can repeat.",[55,133,134,137,139],{},[73,135,136],{},"Consumer that writes to a database",[73,138,89],{},[73,140,141],{},"Needs idempotent writes, or offsets stored with the data",[10,143,144,145,150],{},"Stream processing frameworks build on the same features. See ",[146,147,149],"a",{"href":148},"#exactly-once-support-in-kafka-streams-flink-spark-and-connect","Exactly-once support in Kafka Streams, Flink, Spark, and Connect",".",[41,152,154],{"id":153},"what-kafka-transactions-cover","What Kafka transactions cover",[156,157,159],"h3",{"id":158},"kafka-idempotent-producer-and-enableidempotence","Kafka idempotent producer and enable.idempotence",[10,161,162],{},"The idempotent producer removes only the duplicates its own retries create. If a send times out and the producer retries a batch the broker already wrote, the broker drops the second copy. If your application sends the same order twice, or sends it again after a restart, Kafka stores it twice.",[10,164,165,166,169,170,173,174,177,178,180],{},"The Java client has enabled idempotence by default since Kafka 3.0. Settings such as ",[14,167,168],{},"acks=1"," or ",[14,171,172],{},"retries=0"," quietly turn it off, and the producer fails at startup if you combine one of them with ",[14,175,176],{},"enable.idempotence=true",". Set ",[14,179,176],{}," explicitly, so a later config change cannot turn it off unnoticed.",[156,182,184],{"id":183},"kafka-transactions-and-transactionalid","Kafka transactions and transactional.id",[10,186,187,188,191,192,195,196,199,200,203,204,206],{},"A transaction lets a producer write to several partitions, even in different topics, as one unit. For example, an order service writes an order to ",[14,189,190],{},"orders"," and a payment request to ",[14,193,194],{},"payments"," between ",[14,197,198],{},"beginTransaction"," and ",[14,201,202],{},"commitTransaction",". If it crashes before the commit, neither record ever reaches ",[14,205,16],{}," consumers.",[10,208,209,210,213,214,217],{},"A transactional producer needs a ",[14,211,212],{},"transactional.id",", a name that stays the same when the producer restarts, so Kafka can recognize it as the same producer. At startup, ",[14,215,216],{},"initTransactions"," aborts the transaction the previous instance left open, and stops that instance from writing if it is still alive. Stopping the old instance is called fencing.",[10,219,220,221,223],{},"Keep the ID the same across restarts. With a new ID on every start, nothing aborts the old transaction, so it blocks ",[14,222,16],{}," consumers until it times out.",[10,225,226,227,169,230,150],{},"Give each producer that runs at the same time its own ID, such as the service name plus an instance number. If two running producers share an ID, the newer one fences the older one, which then fails with ",[14,228,229],{},"InvalidProducerEpochException",[14,231,232],{},"ProducerFencedException",[156,234,236],{"id":235},"kafka-isolation-levels-and-read_committed","Kafka isolation levels and read_committed",[10,238,239,240,243,244,177,246,249],{},"A Java consumer reads records from aborted transactions by default, because ",[14,241,242],{},"isolation.level"," defaults to ",[14,245,119],{},[14,247,248],{},"isolation.level=read_committed"," on every consumer of transactional output so it sees only committed records.",[10,251,252,253,255],{},"A ",[14,254,16],{}," consumer reads only up to the last stable offset (LSO), which is where the oldest open transaction in the partition starts, or the end of the log when none is open. Records after that point wait until the transaction commits or aborts, including records from producers that do not use transactions.",[10,257,258],{},"When a transaction commits or aborts, Kafka writes a marker record into each partition the transaction wrote to. The marker takes up an offset like a normal record, but consumers never receive it. Markers and skipped aborted records leave gaps between offsets, so a gap alone does not mean data loss.",[10,260,261,262,150],{},"Markers can also leave a group's lag at 1 when nothing is left to read. If a transaction writes offsets 0 to 2, its commit marker takes offset 3 and the partition's end offset becomes 4. A consumer that commits 3, right after its last record, still shows lag 1. Spring Kafka commits this way unless you turn on its container property ",[14,263,264],{},"fixTxOffsets",[156,266,268],{"id":267},"idempotence-and-isolation-defaults-by-client","Idempotence and isolation defaults by client",[10,270,271],{},"Whether producers are idempotent and which isolation level consumers use by default depends on the client library, not the language. The three Python clients, for example, each combine these defaults differently, so switching libraries can change what your producers and consumers guarantee.",[10,273,274,277],{},[25,275,276],{},"Defaults verified against each client's source:"," 22 Oct 2026",[49,279,280,293],{},[52,281,282],{},[55,283,284,287,290],{},[58,285,286],{},"Client",[58,288,289],{},"Idempotent producer by default",[58,291,292],{},"Default consumer isolation",[68,294,295,311,326,341,358,373,388,404,419],{},[55,296,297,304,307],{},[73,298,299,303],{},[146,300,302],{"href":301},"\u002Fblog\u002Fkafka-java-clients#apache-kafka-java-client","Java client"," 4.3",[73,305,306],{},"Yes",[73,308,309],{},[14,310,119],{},[55,312,313,320,322],{},[73,314,315,319],{},[146,316,318],{"href":317},"\u002Fblog\u002Fkafka-python-clients#kafka-python","kafka-python"," 3.0",[73,321,306],{},[73,323,324],{},[14,325,119],{},[55,327,328,335,337],{},[73,329,330,334],{},[146,331,333],{"href":332},"\u002Fblog\u002Fkafka-golang-clients#franz-go","franz-go"," 1.22",[73,336,306],{},[73,338,339],{},[14,340,119],{},[55,342,343,351,354],{},[73,344,345,346,350],{},"librdkafka 2.16 and ",[146,347,349],{"href":348},"\u002Fkafka-clients","clients built on it",", such as confluent-kafka-python, confluent-kafka-go, and Confluent.Kafka",[73,352,353],{},"No",[73,355,356],{},[14,357,16],{},[55,359,360,367,369],{},[73,361,362,366],{},[146,363,365],{"href":364},"\u002Fblog\u002Fkafka-python-clients#aiokafka","aiokafka"," 0.14",[73,368,353],{},[73,370,371],{},[14,372,119],{},[55,374,375,382,384],{},[73,376,377,381],{},[146,378,380],{"href":379},"\u002Fblog\u002Fkafka-golang-clients#sarama","Sarama"," 1.61",[73,383,353],{},[73,385,386],{},[14,387,119],{},[55,389,390,397,400],{},[73,391,392,396],{},[146,393,395],{"href":394},"\u002Fblog\u002Fkafka-golang-clients#kafka-go","kafka-go"," 0.4",[73,398,399],{},"Not supported",[73,401,402],{},[14,403,119],{},[55,405,406,413,415],{},[73,407,408,412],{},[146,409,411],{"href":410},"\u002Fblog\u002Fkafka-javascript-clients#kafkajs","KafkaJS"," 2.2",[73,414,353],{},[73,416,417],{},[14,418,16],{},[55,420,421,428,430],{},[73,422,423,427],{},[146,424,426],{"href":425},"\u002Fblog\u002Fkafka-javascript-clients#platformatic-kafka","Platformatic Kafka"," 2.14",[73,429,353],{},[73,431,432],{},[14,433,16],{},[10,435,436],{},"Set idempotence and isolation explicitly in each client instead of relying on these defaults.",[156,438,128],{"id":439},"exactly-once-from-kafka-to-kafka",[10,441,442,443,446],{},"An application that reads from Kafka and writes its results back to Kafka gets exactly-once by committing its output and its input offsets in one transaction. Kafka Streams does this for you when you set ",[14,444,445],{},"processing.guarantee=exactly_once_v2",". Flink instead saves input offsets in its checkpoints and commits its Kafka transactions when a checkpoint completes, so the offsets it commits to the consumer group only show progress.",[10,448,449,450,453,454,457,458,150],{},"With the Java client, turn off ",[14,451,452],{},"enable.auto.commit",", so offsets are committed only with the transaction. Before each commit, call ",[14,455,456],{},"sendOffsetsToTransaction"," with each partition's next offset to read (the last processed offset plus 1) and ",[14,459,460],{},"consumer.groupMetadata()",[10,462,463,464,466],{},"If the application crashes before the commit, the transaction aborts. Its output never reaches ",[14,465,16],{}," consumers and its offsets are not committed, so the restarted application reads the same input again and writes the output once.",[41,468,149],{"id":469},"exactly-once-support-in-kafka-streams-flink-spark-and-connect",[10,471,472,277],{},[25,473,474],{},"Behavior verified against each release's documentation and source:",[49,476,477,496],{},[52,478,479],{},[55,480,481,484,487,490,493],{},[58,482,483],{},"Framework",[58,485,486],{},"How to enable",[58,488,489],{},"Output guarantee",[58,491,492],{},"Default input isolation",[58,494,495],{},"Main limit",[68,497,498,527,555,578,598],{},[55,499,500,506,510,513,518],{},[73,501,502],{},[146,503,505],{"href":504},"\u002Fblog\u002Fkafka-streams-in-java#apache-kafka-streams-in-plain-java","Kafka Streams 4.3.1",[73,507,508],{},[14,509,445],{},[73,511,512],{},"Exactly-once within one cluster",[73,514,515,517],{},[14,516,16],{}," when exactly-once is on",[73,519,520,521,199,524,150],{},"Output appears at each commit, every 100 ms by default. Kafka 4.0 removed ",[14,522,523],{},"exactly_once",[14,525,526],{},"exactly_once_beta",[55,528,529,535,545,548,552],{},[73,530,531],{},[146,532,534],{"href":533},"\u002Fblog\u002Fflink-vs-kafka-streams#exactly-once-and-output-latency","Flink 2.2.1 with connector 5.0.0",[73,536,537,538,541,542],{},"Checkpointing, and ",[14,539,540],{},"KafkaSink"," set to ",[14,543,544],{},"EXACTLY_ONCE",[73,546,547],{},"Exactly-once",[73,549,550],{},[14,551,119],{},[73,553,554],{},"Output appears when a checkpoint completes",[55,556,557,566,569,571,575],{},[73,558,559,565],{},[146,560,564],{"href":561,"rel":562},"https:\u002F\u002Fspark.apache.org\u002Fdocs\u002F4.2.0\u002Fstreaming\u002Fstructured-streaming-kafka-integration.html",[563],"nofollow","Spark 4.2.0"," Structured Streaming",[73,567,568],{},"Not available",[73,570,89],{},[73,572,573],{},[14,574,119],{},[73,576,577],{},"Retries and recovery can write duplicates",[55,579,580,583,589,592,595],{},[73,581,582],{},"Kafka Connect 4.3.1 source connectors",[73,584,585,588],{},[14,586,587],{},"exactly.once.source.support=enabled"," on every worker",[73,590,591],{},"Exactly-once, if the connector supports it",[73,593,594],{},"Not applicable",[73,596,597],{},"Distributed mode only",[55,599,600,603,606,609,613],{},[73,601,602],{},"Kafka Connect 4.3.1 sink connectors",[73,604,605],{},"No framework setting",[73,607,608],{},"At-least-once, unless the connector removes duplicates",[73,610,611],{},[14,612,119],{},[73,614,615],{},"Depends on how the connector writes to the target system",[10,617,618],{},"Flink, Spark, and Connect sinks see records from aborted transactions by default. When their input comes from a transactional producer, such as a Kafka Streams application with exactly-once, set the consumer's isolation level:",[620,621,622,640,650],"ul",{},[623,624,625,628,629,631,632,635,636,639],"li",{},[25,626,627],{},"Flink:"," ",[14,630,248],{}," in the ",[14,633,634],{},"KafkaSource"," properties, or ",[14,637,638],{},"properties.isolation.level"," in Flink SQL.",[623,641,642,645,646,649],{},[25,643,644],{},"Spark:"," the ",[14,647,648],{},"kafka.isolation.level=read_committed"," option on the Kafka source.",[623,651,652,628,655,658],{},[25,653,654],{},"Kafka Connect sinks:",[14,656,657],{},"consumer.override.isolation.level=read_committed"," in the connector config.",[10,660,661],{},"Spark's Kafka sink has no transactional mode, so a retried write or a restarted query can write the same row twice. Include a stable ID with each row, such as a business key or the input topic, partition, and offset, and deduplicate on it downstream.",[41,663,665],{"id":664},"when-kafka-transactions-get-stuck","When Kafka transactions get stuck",[156,667,669],{"id":668},"open-transactions-block-read_committed-consumers","Open transactions block read_committed consumers",[10,671,672,673,675],{},"When a transactional producer crashes or stops responding in the middle of a transaction, the transaction stays open. ",[14,674,16],{}," consumers of that partition then stop at its first record, while other producers keep writing behind it.",[10,677,678,679,681,682,685],{},"The transaction coordinator, the broker that tracks each transaction, aborts it when a producer with the same ",[14,680,212],{}," starts, or once ",[14,683,684],{},"transaction.timeout.ms"," has passed. The Java client's default timeout is 60 seconds, and brokers allow up to 15 minutes. Keep it as short as your longest transaction allows, because consumers stay blocked until then.",[10,687,688,689,150],{},"Flink is different. Its Kafka sink keeps each transaction open until the checkpoint completes, and across a restart. If the timeout passes first, Kafka aborts the transaction and its output is lost, so set the timeout above your longest checkpoint plus recovery time. See ",[146,690,691],{"href":533},"Flink vs Kafka Streams",[156,693,695],{"id":694},"see-blocked-partitions-in-kafma","See blocked partitions in Kafma",[10,697,698,699,701],{},"Kafma reads with ",[14,700,16],{},", so the Console shows the topic the way your consumers see it. New records stop appearing at the open transaction.",[10,703,704,705,708],{},"From Kafma 1.2.1, consumer lag counts to the end of the log, the same as ",[14,706,707],{},"kafka-consumer-groups.sh",", so a blocked group's lag keeps growing.",[10,710,711,712,716],{},"Once Kafma has seen a transaction block a partition for more than 10 seconds, ",[146,713,715],{"href":714},"\u002Fdocs\u002Fconsole\u002Fconsuming-messages#live-and-watch-group-modes","Watch Group"," shows a line with how many offsets it holds back. Hover over the line to see the offset where the transaction starts and how long Kafma has seen it blocking.",[10,718,719],{},[720,721],"img",{"alt":722,"height":723,"src":724,"width":725},"Kafma Watch Group on partition 0 of orders, where order-service is committed at offset 146, the start of an unfinished transaction",1520,"https:\u002F\u002Fmedia.kafma.app\u002Fblog\u002Fkafka-transactions-exactly-once\u002Fwatch-group-transaction-block.png",3024,[10,727,728],{},"In the group's details, the partition row shows the same label, and the group's lag keeps growing.",[10,730,731],{},[720,732],{"alt":733,"height":734,"src":735,"width":725},"Kafma consumer group details with partition 0 of orders blocked by an unfinished transaction at offset 146 while the group's lag keeps rising",1800,"https:\u002F\u002Fmedia.kafma.app\u002Fblog\u002Fkafka-transactions-exactly-once\u002Fconsumer-group-transaction-block.png",[156,737,739],{"id":738},"find-open-and-hanging-transactions","Find open and hanging transactions",[10,741,742,743,746],{},"To see which producer holds the open transaction, list the transactions that have run for more than a minute. Then describe the producers on the blocked partition. The ",[14,744,745],{},"CurrentTransactionStartOffset"," column shows where the open transaction starts, which is where the consumers stopped.",[748,749,754],"pre",{"className":750,"code":751,"language":752,"meta":753,"style":753},"language-shell shiki shiki-themes kafma-dark kafma-light","kafka-transactions.sh --bootstrap-server localhost:9092 list --duration-filter 60000\nkafka-transactions.sh --bootstrap-server localhost:9092 describe-producers --topic orders --partition 0\n","shell","",[14,755,756,782],{"__ignoreMap":753},[757,758,761,765,769,773,776,779],"span",{"class":759,"line":760},"line",1,[757,762,764],{"class":763},"sS_Im","kafka-transactions.sh",[757,766,768],{"class":767},"spVhG"," --bootstrap-server",[757,770,772],{"class":771},"sIsOz"," localhost:9092",[757,774,775],{"class":771}," list",[757,777,778],{"class":767}," --duration-filter",[757,780,781],{"class":771}," 60000\n",[757,783,785,787,789,791,794,797,800,803],{"class":759,"line":784},2,[757,786,764],{"class":763},[757,788,768],{"class":767},[757,790,772],{"class":771},[757,792,793],{"class":771}," describe-producers",[757,795,796],{"class":767}," --topic",[757,798,799],{"class":771}," orders",[757,801,802],{"class":767}," --partition",[757,804,805],{"class":771}," 0\n",[10,807,808],{},"A hanging transaction is an open transaction the coordinator no longer tracks, so it never times out and blocks consumers until you abort it. It is usually caused by a delayed producer retry that arrives after the coordinator has finished the transaction. Since Kafka 3.6, brokers reject such writes, so hanging transactions are rare.",[10,810,811,812,815,816,819],{},"On older brokers, or with ",[14,813,814],{},"transaction.partition.verification.enable=false",", run ",[14,817,818],{},"find-hanging"," for each broker. Abort only the transactions it reports, by their start offset.",[748,821,823],{"className":750,"code":822,"language":752,"meta":753,"style":753},"kafka-transactions.sh --bootstrap-server localhost:9092 find-hanging --broker-id 1\nkafka-transactions.sh --bootstrap-server localhost:9092 abort --topic orders --partition 0 --start-offset 1042\n",[14,824,825,842],{"__ignoreMap":753},[757,826,827,829,831,833,836,839],{"class":759,"line":760},[757,828,764],{"class":763},[757,830,768],{"class":767},[757,832,772],{"class":771},[757,834,835],{"class":771}," find-hanging",[757,837,838],{"class":767}," --broker-id",[757,840,841],{"class":771}," 1\n",[757,843,844,846,848,850,853,855,857,859,862,865],{"class":759,"line":784},[757,845,764],{"class":763},[757,847,768],{"class":767},[757,849,772],{"class":771},[757,851,852],{"class":771}," abort",[757,854,796],{"class":767},[757,856,799],{"class":771},[757,858,802],{"class":767},[757,860,861],{"class":771}," 0",[757,863,864],{"class":767}," --start-offset",[757,866,867],{"class":771}," 1042\n",[156,869,871],{"id":870},"producer-errors-fenced-timed-out-or-unauthorized","Producer errors: fenced, timed out, or unauthorized",[10,873,874],{},"These errors were checked against Kafka 4.3.1 with the Java client.",[49,876,877,890],{},[52,878,879],{},[55,880,881,884,887],{},[58,882,883],{},"Error",[58,885,886],{},"Cause",[58,888,889],{},"Fix",[68,891,892,917,930,947,964,977],{},[55,893,894,899,907],{},[73,895,896,898],{},[14,897,229],{}," on send",[73,900,901,902,904,905],{},"Another producer started with the same ",[14,903,212],{},", or the coordinator aborted the transaction after ",[14,906,684],{},[73,908,909,910,913,914,916],{},"Call ",[14,911,912],{},"abortTransaction()",". If it succeeds, begin a new transaction and resend every record from the aborted one. If it throws ",[14,915,232],{},", close the producer.",[55,918,919,923,927],{},[73,920,921],{},[14,922,232],{},[73,924,901,925],{},[14,926,212],{},[73,928,929],{},"Close this producer, and give each running instance its own ID",[55,931,932,938,944],{},[73,933,934,937],{},[14,935,936],{},"InvalidTxnStateException"," on commit",[73,939,940,941,943],{},"The transaction ran longer than ",[14,942,684],{},", so the coordinator aborted it",[73,945,946],{},"Close the producer and create a new one. Keep transactions shorter, or raise the timeout.",[55,948,949,952,961],{},[73,950,951],{},"Transaction timeout \"larger than the maximum value allowed by the broker\"",[73,953,954,956,957,960],{},[14,955,684],{}," is above the broker's ",[14,958,959],{},"transaction.max.timeout.ms",", 15 minutes by default",[73,962,963],{},"Lower the producer timeout or raise the broker limit",[55,965,966,971,974],{},[73,967,968],{},[14,969,970],{},"TransactionalIdAuthorizationException",[73,972,973],{},"ACLs are on, and the producer's principal lacks Write on its transactional ID",[73,975,976],{},"Grant Write on the TransactionalId resource",[55,978,979,988,995],{},[73,980,981,984,985],{},[14,982,983],{},"TimeoutException"," while awaiting ",[14,986,987],{},"InitProducerId",[73,989,990,991,994],{},"The broker cannot create the internal ",[14,992,993],{},"__transaction_state"," topic, for example on one broker with its default replication factor of 3",[73,996,997,998,199,1001],{},"On a one-broker cluster, set ",[14,999,1000],{},"transaction.state.log.replication.factor=1",[14,1002,1003],{},"transaction.state.log.min.isr=1",[10,1005,1006,1007,1011,1012,28,1015,1018],{},"To check in Kafma whether a producer may write to its transactional ID, open the ",[146,1008,1010],{"href":1009},"\u002Fdocs\u002Foperate\u002Faccess-control#simulate-access","ACL Simulator",", enter the producer's principal, and choose ",[25,1013,1014],{},"WRITE",[25,1016,1017],{},"Transactional ID",", and the ID. An application that sends offsets in its transactions also needs Read on its consumer group.",[10,1020,1021,1025],{},[146,1022,1024],{"href":1023},"\u002Fdownload","Download Kafma"," to see which partitions open transactions are blocking.",[41,1027,1029],{"id":1028},"frequently-asked-questions","Frequently asked questions",[156,1031,1033],{"id":1032},"does-kafka-support-exactly-once","Does Kafka support exactly-once?",[10,1035,1036],{},"Yes, between topics in one Kafka cluster. Kafka Streams and Flink use transactions to provide it, and plain clients get it by sending offsets in the transaction. Writes to databases and other systems need idempotent writes or their own transactions.",[156,1038,1040],{"id":1039},"what-is-the-difference-between-at-least-once-and-exactly-once-delivery","What is the difference between at-least-once and exactly-once delivery?",[10,1042,1043,1044,1046],{},"At-least-once means a record is never skipped but may be processed again after a failure. Exactly-once means each record affects the output once, which in Kafka needs transactions and ",[14,1045,16],{}," consumers. At-most-once is the opposite trade-off and may skip records.",[156,1048,1050],{"id":1049},"is-enableidempotence-true-by-default","Is enable.idempotence true by default?",[10,1052,1053,1054,1056,1057,150],{},"In the Java client, yes, since Kafka 3.0, unless a conflicting setting such as ",[14,1055,168],{}," turns it off. librdkafka, KafkaJS, Sarama, and aiokafka default to off. See ",[146,1058,1060],{"href":1059},"#idempotence-and-isolation-defaults-by-client","defaults by client",[156,1062,1064],{"id":1063},"what-is-a-transactional-id-in-kafka","What is a transactional ID in Kafka?",[10,1066,1067,1068,1070],{},"It is a name, set with ",[14,1069,212],{},", that identifies one transactional producer across restarts. Kafka uses it to abort transactions the previous instance left open and to fence old instances. Each producer running at the same time needs its own.",[156,1072,1074],{"id":1073},"do-transactions-slow-kafka-down","Do transactions slow Kafka down?",[10,1076,1077,1078,1080],{},"Somewhat. Each commit waits for the coordinator and writes a marker to every partition in the transaction, and ",[14,1079,16],{}," consumers see records only after the commit. Fewer, larger transactions cost less throughput but add latency.",[156,1082,1084],{"id":1083},"does-exactly-once-work-across-kafka-clusters","Does exactly-once work across Kafka clusters?",[10,1086,1087,1088,1091,1092,1094],{},"Not in one transaction, which covers a single cluster. From Kafka 3.5, MirrorMaker 2 can copy topics exactly once when it runs as a dedicated MirrorMaker cluster with ",[14,1089,1090],{},"exactly.once.source.support"," enabled for the target. Also set its source consumer to ",[14,1093,16],{},", or it copies aborted records too.",[41,1096,1098],{"id":1097},"conclusion","Conclusion",[10,1100,1101,1102,1104],{},"Exactly-once in Kafka needs a transactional producer, input offsets saved together with the output, and consumers that read with ",[14,1103,16],{},". Defaults differ by client and framework, so set these explicitly.",[10,1106,1107,1108,1110,1111,1113],{},"When consumers stop advancing, check for an open transaction first. Keep ",[14,1109,212],{}," stable across restarts and ",[14,1112,684],{}," as short as your transactions allow, to limit how long a crashed producer blocks consumers.",[10,1115,1116],{},"This guide is maintained by the team behind Kafma.",[1118,1119,1120],"style",{},"html pre.shiki code .sS_Im, html code.shiki .sS_Im{--shiki-default:#FFFFFF;--shiki-light:#0A0B10}html pre.shiki code .spVhG, html code.shiki .spVhG{--shiki-default:#E7E8EA;--shiki-light:#1A1C22}html pre.shiki code .sIsOz, html code.shiki .sIsOz{--shiki-default:#83CFAE;--shiki-light:#1F7A52}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}",{"title":753,"searchDepth":1122,"depth":1122,"links":1123},3,[1124,1125,1132,1133,1139,1147],{"id":43,"depth":784,"text":44},{"id":153,"depth":784,"text":154,"children":1126},[1127,1128,1129,1130,1131],{"id":158,"depth":1122,"text":159},{"id":183,"depth":1122,"text":184},{"id":235,"depth":1122,"text":236},{"id":267,"depth":1122,"text":268},{"id":439,"depth":1122,"text":128},{"id":469,"depth":784,"text":149},{"id":664,"depth":784,"text":665,"children":1134},[1135,1136,1137,1138],{"id":668,"depth":1122,"text":669},{"id":694,"depth":1122,"text":695},{"id":738,"depth":1122,"text":739},{"id":870,"depth":1122,"text":871},{"id":1028,"depth":784,"text":1029,"children":1140},[1141,1142,1143,1144,1145,1146],{"id":1032,"depth":1122,"text":1033},{"id":1039,"depth":1122,"text":1040},{"id":1049,"depth":1122,"text":1050},{"id":1063,"depth":1122,"text":1064},{"id":1073,"depth":1122,"text":1074},{"id":1083,"depth":1122,"text":1084},{"id":1097,"depth":784,"text":1098},"2026-10-22","What Kafka idempotence, transactions, and exactly-once semantics guarantee, where they stop, and how to fix transactions that block consumers.","md",{},true,null,"\u002Fblog\u002Fkafka-transactions-exactly-once",false,{"title":5,"description":1149},{"loc":1154},"blog\u002Fkafka-transactions-exactly-once",[1160,1161,1162],"Kafka Stream Processing","Kafka Transactions","Exactly-Once","yoexqbK9evu1Y_P4kyHjvkXJ4v-_Q6q8WzOQVVhBzI0",1792819406245]