Kafka Transactions, Idempotence, and Exactly-Once Semantics

Exactly-once semantics in Kafka means each input record affects the output once, even after retries and restarts. It needs all three of the following together: the producer writes its output in transactions, the input offsets are saved together with the output, and the consumers that read the output use read_committed. If any one is missing, a retry or restart can still duplicate output.

The guarantee covers reading from and writing to topics in one Kafka cluster. Writes to a database or to another cluster are outside it and need their own protection against duplicates.

This article compares what each setup guarantees, from plain clients to Kafka Streams, Flink, Spark, and Kafka Connect, and how client defaults differ. It then shows how to find and fix transactions that block consumers.

Kafka delivery guarantees by setup

A consumer commits offsets to record how far it has read. Whether it commits before or after processing decides what a crash does to the records in between.

Stream processing frameworks build on the same features. See Exactly-once support in Kafka Streams, Flink, Spark, and Connect.

What Kafka transactions cover

Kafka idempotent producer and enable.idempotence

The idempotent producer removes only the duplicates its own retries create. If a send times out and the producer retries a batch the broker already wrote, the broker drops the second copy. If your application sends the same order twice, or sends it again after a restart, Kafka stores it twice.

The Java client has enabled idempotence by default since Kafka 3.0. Settings such as acks=1 or retries=0 quietly turn it off, and the producer fails at startup if you combine one of them with enable.idempotence=true. Set enable.idempotence=true explicitly, so a later config change cannot turn it off unnoticed.

Kafka transactions and transactional.id

A transaction lets a producer write to several partitions, even in different topics, as one unit. For example, an order service writes an order to orders and a payment request to payments between beginTransaction and commitTransaction. If it crashes before the commit, neither record ever reaches read_committed consumers.

A transactional producer needs a transactional.id, a name that stays the same when the producer restarts, so Kafka can recognize it as the same producer. At startup, initTransactions aborts the transaction the previous instance left open, and stops that instance from writing if it is still alive. Stopping the old instance is called fencing.

Keep the ID the same across restarts. With a new ID on every start, nothing aborts the old transaction, so it blocks read_committed consumers until it times out.

Give each producer that runs at the same time its own ID, such as the service name plus an instance number. If two running producers share an ID, the newer one fences the older one, which then fails with InvalidProducerEpochException or ProducerFencedException.

Kafka isolation levels and read_committed

A Java consumer reads records from aborted transactions by default, because isolation.level defaults to read_uncommitted. Set isolation.level=read_committed on every consumer of transactional output so it sees only committed records.

A read_committed consumer reads only up to the last stable offset (LSO), which is where the oldest open transaction in the partition starts, or the end of the log when none is open. Records after that point wait until the transaction commits or aborts, including records from producers that do not use transactions.

When a transaction commits or aborts, Kafka writes a marker record into each partition the transaction wrote to. The marker takes up an offset like a normal record, but consumers never receive it. Markers and skipped aborted records leave gaps between offsets, so a gap alone does not mean data loss.

Markers can also leave a group's lag at 1 when nothing is left to read. If a transaction writes offsets 0 to 2, its commit marker takes offset 3 and the partition's end offset becomes 4. A consumer that commits 3, right after its last record, still shows lag 1. Spring Kafka commits this way unless you turn on its container property fixTxOffsets.

Idempotence and isolation defaults by client

Whether producers are idempotent and which isolation level consumers use by default depends on the client library, not the language. The three Python clients, for example, each combine these defaults differently, so switching libraries can change what your producers and consumers guarantee.

Defaults verified against each client's source: 22 Oct 2026

Set idempotence and isolation explicitly in each client instead of relying on these defaults.

Exactly-once from Kafka to Kafka

An application that reads from Kafka and writes its results back to Kafka gets exactly-once by committing its output and its input offsets in one transaction. Kafka Streams does this for you when you set processing.guarantee=exactly_once_v2. Flink instead saves input offsets in its checkpoints and commits its Kafka transactions when a checkpoint completes, so the offsets it commits to the consumer group only show progress.

With the Java client, turn off enable.auto.commit, so offsets are committed only with the transaction. Before each commit, call sendOffsetsToTransaction with each partition's next offset to read (the last processed offset plus 1) and consumer.groupMetadata().

If the application crashes before the commit, the transaction aborts. Its output never reaches read_committed consumers and its offsets are not committed, so the restarted application reads the same input again and writes the output once.

Exactly-once support in Kafka Streams, Flink, Spark, and Connect

Behavior verified against each release's documentation and source: 22 Oct 2026

Flink, Spark, and Connect sinks see records from aborted transactions by default. When their input comes from a transactional producer, such as a Kafka Streams application with exactly-once, set the consumer's isolation level:

• Flink: isolation.level=read_committed in the KafkaSource properties, or properties.isolation.level in Flink SQL.
• Spark: the kafka.isolation.level=read_committed option on the Kafka source.
• Kafka Connect sinks: consumer.override.isolation.level=read_committed in the connector config.

Spark's Kafka sink has no transactional mode, so a retried write or a restarted query can write the same row twice. Include a stable ID with each row, such as a business key or the input topic, partition, and offset, and deduplicate on it downstream.

When Kafka transactions get stuck

Open transactions block read_committed consumers

When a transactional producer crashes or stops responding in the middle of a transaction, the transaction stays open. read_committed consumers of that partition then stop at its first record, while other producers keep writing behind it.

The transaction coordinator, the broker that tracks each transaction, aborts it when a producer with the same transactional.id starts, or once transaction.timeout.ms has passed. The Java client's default timeout is 60 seconds, and brokers allow up to 15 minutes. Keep it as short as your longest transaction allows, because consumers stay blocked until then.

Flink is different. Its Kafka sink keeps each transaction open until the checkpoint completes, and across a restart. If the timeout passes first, Kafka aborts the transaction and its output is lost, so set the timeout above your longest checkpoint plus recovery time. See Flink vs Kafka Streams.

See blocked partitions in Kafma

Kafma reads with read_committed, so the Console shows the topic the way your consumers see it. New records stop appearing at the open transaction.

From Kafma 1.2.1, consumer lag counts to the end of the log, the same as kafka-consumer-groups.sh, so a blocked group's lag keeps growing.

Once Kafma has seen a transaction block a partition for more than 10 seconds, Watch Group shows a line with how many offsets it holds back. Hover over the line to see the offset where the transaction starts and how long Kafma has seen it blocking.

In the group's details, the partition row shows the same label, and the group's lag keeps growing.

Find open and hanging transactions

To see which producer holds the open transaction, list the transactions that have run for more than a minute. Then describe the producers on the blocked partition. The CurrentTransactionStartOffset column shows where the open transaction starts, which is where the consumers stopped.

kafka-transactions.sh --bootstrap-server localhost:9092 list --duration-filter 60000
kafka-transactions.sh --bootstrap-server localhost:9092 describe-producers --topic orders --partition 0

A hanging transaction is an open transaction the coordinator no longer tracks, so it never times out and blocks consumers until you abort it. It is usually caused by a delayed producer retry that arrives after the coordinator has finished the transaction. Since Kafka 3.6, brokers reject such writes, so hanging transactions are rare.

On older brokers, or with transaction.partition.verification.enable=false, run find-hanging for each broker. Abort only the transactions it reports, by their start offset.

kafka-transactions.sh --bootstrap-server localhost:9092 find-hanging --broker-id 1
kafka-transactions.sh --bootstrap-server localhost:9092 abort --topic orders --partition 0 --start-offset 1042

Producer errors: fenced, timed out, or unauthorized

These errors were checked against Kafka 4.3.1 with the Java client.

To check in Kafma whether a producer may write to its transactional ID, open the ACL Simulator, enter the producer's principal, and choose WRITE, Transactional ID, and the ID. An application that sends offsets in its transactions also needs Read on its consumer group.

Download Kafma to see which partitions open transactions are blocking.

Frequently asked questions

Does Kafka support exactly-once?

Yes, between topics in one Kafka cluster. Kafka Streams and Flink use transactions to provide it, and plain clients get it by sending offsets in the transaction. Writes to databases and other systems need idempotent writes or their own transactions.

What is the difference between at-least-once and exactly-once delivery?

At-least-once means a record is never skipped but may be processed again after a failure. Exactly-once means each record affects the output once, which in Kafka needs transactions and read_committed consumers. At-most-once is the opposite trade-off and may skip records.

Is enable.idempotence true by default?

In the Java client, yes, since Kafka 3.0, unless a conflicting setting such as acks=1 turns it off. librdkafka, KafkaJS, Sarama, and aiokafka default to off. See defaults by client.

What is a transactional ID in Kafka?

It is a name, set with transactional.id, that identifies one transactional producer across restarts. Kafka uses it to abort transactions the previous instance left open and to fence old instances. Each producer running at the same time needs its own.

Do transactions slow Kafka down?

Somewhat. Each commit waits for the coordinator and writes a marker to every partition in the transaction, and read_committed consumers see records only after the commit. Fewer, larger transactions cost less throughput but add latency.

Does exactly-once work across Kafka clusters?

Not in one transaction, which covers a single cluster. From Kafka 3.5, MirrorMaker 2 can copy topics exactly once when it runs as a dedicated MirrorMaker cluster with exactly.once.source.support enabled for the target. Also set its source consumer to read_committed, or it copies aborted records too.

Conclusion

Exactly-once in Kafka needs a transactional producer, input offsets saved together with the output, and consumers that read with read_committed. Defaults differ by client and framework, so set these explicitly.

When consumers stop advancing, check for an open transaction first. Keep transactional.id stable across restarts and transaction.timeout.ms as short as your transactions allow, to limit how long a crashed producer blocks consumers.

This guide is maintained by the team behind Kafma.