Exactly-Once Semantics Stop Where Kafka Stops

Kafka defines the goal as "Each message is processed once and only once". Read that word again. It says processed, not delivered, and exactly-once semantics live or die on that difference.

The trick is simple. Kafka makes the offset and the output commit together. That works because both sit in Kafka, and your database does not.

Key takeaways#

• The docs promise processing once, not delivery once.
• Three separate features share one name: the idempotent producer, transactions, and read_committed.
• The idempotent producer only drops retries within one producer session.
• KIP-98 admits a consumer may not see a whole transaction together.
• For an outside system, Kafka's own advice is to store the offset with the output.

What do exactly-once semantics actually promise?#

They promise that a read-process-write loop whose output is Kafka commits as one unit. The consumer offset lives in a Kafka topic, so one transaction can cover the offset and the new records at once. Either both land or neither does, and a replay after a crash sees no repeats.

Notice what exactly-once semantics need in order to hold. The offset and the output must live in one system, under one commit. Kafka can promise this because Kafka owns both halves.

Three features, one overloaded name#

Much of the confusion comes from treating one label as one feature. There are three. They fix different bugs, and you can turn each on by itself.

Feature	Stops	Scope	Config
Idempotent producer	Repeats from producer retries	One producer session, per partition	enable.idempotence=true
Transactions	Partial writes across topics and offsets	One read-process-write loop	transactional.id
read_committed	Reading aborted records	The consumer	isolation.level

I picked those columns because scope is the thing people get wrong. Each row is useful. None of them covers a write to something that is not Kafka.

Take the first row. KIP-98 gives each producer an ID and a rising sequence number per partition. The broker "will reject a produce request if its sequence number is not exactly one greater than the last committed message from that PID/TopicPartition pair".

So a retried send gets dropped instead of appended. That is a real fix. Still, read the scope column: one session. Restart the producer and the window starts again.

The third row hides the trap that catches most people. isolation.level defaults to read_uncommitted, and the docs spell out what that means: a consumer "will return all messages, even transactional messages which have been aborted".

Read that against the second row. You can configure the producer perfectly, commit and abort exactly as intended, and still hand aborted records to a consumer that never opted in. Exactly-once semantics are not one switch, and the consumer half is off by default.

Turning it on costs something too, which is rarely mentioned. In read_committed a consumer only reads up to the last stable offset, so records sitting behind an open transaction are withheld until it finishes. The docs are explicit that such consumers "will not be able to read up to the high watermark when there are in flight transactions".

That is a latency floor tied to your commit interval, not a bug. A long-running transaction stalls every reader behind it, so keep transactions short if consumers care about freshness.

Fencing, and what transactional.id is really for#

The second row hides a subtlety. A transactional.id is not a label for your logs, and picking it at random breaks the guarantee you turned it on for.

Think about a consumer that hangs long enough to be kicked out of its group. Kafka hands its partitions to someone else. Then the old process wakes up and tries to finish its work. Now two processes believe they own the same partitions.

Kafka calls the stale one a zombie, and the transactional.id is how it dies. Each time a producer calls initTransactions, Kafka bumps an epoch tied to that ID. Writes carrying the old epoch get refused.

That only works if the ID is stable across restarts and maps to the same partitions each time. A random ID per process fences nothing, because the zombie and its replacement look like two unrelated producers. Derive it from the partition assignment, not from a UUID or a hostname.

A worked example#

Take a consumer that reads payment events and writes one row per payment to Postgres. Batch size 500, and it commits offsets after each batch.

Now crash it after 300 rows, before the offset commit. On restart it resumes from the last committed offset, so it replays all 500 events. That writes 300 second rows.

So one ordinary restart made 300 duplicate payments.

Now turn on enable.idempotence and set a transactional.id. Run the same crash. You still get 300 duplicate rows.

Nothing failed, and that is the part worth sitting with. The transaction covered the offset and any Kafka output. Your Postgres write was never in it, because the transaction coordinator has no idea that connection exists.

Change one thing instead. Give each payment a stable key, and make the insert depend on it. Now the replay writes 0 extra rows, with no Kafka transaction at all.

The consumer caveat the KIP admits#

Even inside Kafka, the all-or-nothing story is softer than the pitch. KIP-98 is honest about it, and most write-ups skip the line.

The KIP says plainly that "we cannot guarantee that all the messages of a committed transaction will be consumed all together". It lists why. Log compaction can remove records, old segments age out, a consumer can seek anywhere, and it may read only some of the partitions a transaction touched.

So a transaction is atomic when you write it. On the read side you get "no aborted records" from read_committed, which is weaker and different. Build consumers that survive seeing half a transaction.

Why your database sits outside the line#

The reason is built in, not a missing feature. To make two systems agree, you need one commit that both obey. No such commit exists between a Kafka transaction and a BEGIN in Postgres.

Kafka's design docs name the classic answer and then turn it down. They describe adding "a two-phase commit between the storage of the consumer position and the storage of the consumers output".

Their reason for rejecting it is blunt: "many of the output systems a consumer might want to write to will not support a two-phase commit". A payment API will not join your transaction. Nor will a mail server.

What the docs recommend instead#

Here is the advice people miss, and it points away from transactions.

The design docs suggest handling this "more simply and generally by letting the consumer store its offset in the same place as its output". You move the offset into the target system. Then the write and the position update share that system's transaction.

In practice that means one Postgres table holding the payment row and the partition offset, written in one BEGIN and COMMIT. On restart you read your position from Postgres, not from Kafka. The atomicity is real, and Postgres provides it.

The cheaper option is to keep Kafka's offsets and make the write itself idempotent. A stable key plus a unique index turns a repeat into a no-op. Time-ordered keys work well here, for reasons we covered in UUID v4 versus v7 as a primary key, and the UUID generator makes either kind.

The one place this is easy#

Everything above assumes you leave Kafka. If you do not, the picture changes completely.

Kafka Streams is Kafka-to-Kafka by design. Its state stores are backed by Kafka topics, so a state update is just another write inside the same transaction. Set processing.guarantee=exactly_once_v2 and you get the strong version for roughly one line of config.

That is not a loophole. It is the shape the feature was built for, and it shows where the edge really lies. The question is never "does Kafka support exactly-once semantics", but "does my output live where the offset lives".

So the useful test is a boring one. List every write your consumer performs. Any write that is not a Kafka produce call sits outside the transaction, no matter what your config says.

The part that changed#

One line above is now dated, and this is where most articles are stale. "Will not support a two-phase commit" describes the output systems. Kafka itself used to be an awkward participant too.

KIP-939 changes Kafka's side. It adds transaction.two.phase.commit.enable, the producer methods prepareTransaction() and completeTransaction(PreparedTxnState), and an initTransactions(boolean keepPreparedTxn) overload that recovers without killing a prepared transaction. Operators can end a stuck one with forceTerminateTransaction.

Together these let an outside coordinator make the commit call, with Kafka as one voter. Turn it on and transactions stop expiring via transaction.max.timeout.ms, since a prepared transaction must survive until that coordinator decides.

Weigh that before adopting it. You are choosing transactions that never time out, which turns a self-healing failure into one that needs a human. So it pays off when you truly have an outside coordinator, and not otherwise.

The failure mode, and how to detect it#

The nasty part of a broken exactly-once semantics assumption is that nothing errors. Duplicates look just like real records, and they arrive down the same code path.

So detection has to be a check you run, not an alert you wait for. Pick the natural business key — payment ID, order ID, event ID. Count rows grouped by it and look for any group above one.

Run that on a schedule. The count is cheap and the answer is clear.

Watch restarts as your leading signal. Duplicates bunch up around rebalances, deploys and OOM kills. Line your duplicate count up against restart times before you blame the producer.

One more check is worth automating. If you believe a write is idempotent, prove it: replay a known batch into staging twice and diff the row counts. A constraint you assume and a constraint the database enforces are different things.

What at-least-once plus idempotency gets right#

The boring option deserves a fair hearing, because it is often the better build.

At-least-once with idempotent writes has fewer moving parts. No transaction coordinator, no transactional.id to keep stable, no fencing to reason about, no prepared state left dangling. The whole correctness argument is one unique index that a reader can check in seconds.

It also fails well. When something breaks you get a rejected insert, and a rejected insert is a dull failure.

Its cost is real too. You need a stable key for every event. That is easy for payments and awkward for a sensor reading with no natural identity. Where no key exists, Kafka transactions are the better tool, and the read-process-write loop is exactly what they were built for.

Where this advice is wrong#

Kafka Streams changes the maths. As covered above, a Kafka-to-Kafka topology gets the strong version almost free. My doubt is about crossing the edge, so it does not apply to work that never leaves Kafka.

I have implied duplicates always hurt. Many pipelines are already idempotent. A write that sets a value rather than adding to one, or an upsert into an analytics table, shrugs off a replay. Building dedup machinery around those wastes effort.

Offsets in your database cost something. You own the position now. That means your own monitoring, your own lag maths, and no help from the standard consumer group tools. On a small pipeline that can outweigh what you bought, so idempotent writes with normal offsets are usually the better first move.

The short version#

Exactly-once semantics mean a read-process-write loop commits as one unit when the output is Kafka, because the offset and the output share one system. Cross into a database or an API and you are back to at-least-once, whatever your producer config says. Kafka's own docs suggest storing the offset with the output, or you can make the write idempotent and stop counting deliveries. KIP-939 now lets Kafka join an outside two-phase commit, at the price of transactions that never expire.

Sources: Apache Kafka design documentation, message delivery semantics, KIP-98: Exactly Once Delivery and Transactional Messaging, KIP-939: Support Participation in 2PC