Narcis Miclaus

9 min read

Data & System Architecture, from the ground up Lesson 45 / 80

Exactly-once semantics in streams

What Kafka transactions actually provide, the source-sink coordination problem, the limits, and why exactly-once across services is hard.

Lesson 16, in Module 2, made the case that exactly-once delivery is impossible across an unreliable network, and that the architectural answer is at-least-once delivery plus idempotent processing. That lesson was about messaging in general. This one is about a specific shape of the problem inside streaming pipelines, where vendors do ship something they call “exactly-once,” and where that label means something narrower and more useful than the words suggest.

The promise of exactly-once is appealing: each input event affects each output exactly one time. Not zero, because the system is reliable. Not two, because the system is deduplicating. The reason it is hard is the same reason as before, and the reason it is sometimes deliverable in streaming is that streaming engines control more of the path than a generic messaging client does.

Why exactly-once is hard, in pictures

A stream-processing job has a shape. There is an input source, a processor, and an output sink. Every event passes through all three. Each one can fail. When a stage fails, the orchestration retries.

Without coordination, the retry is what causes duplicates. The processor reads an event from Kafka, computes a transformation, writes the result to Kafka or to a database, and acknowledges the source by advancing its consumer offset. Each of those steps is a separate operation. If the processor crashes between the output write and the offset advance, the next attempt re-reads the same event and writes the output a second time. At-least-once. The duplicates are real and you have no way to spot them after the fact unless your sink can deduplicate.

The opposite failure (advance the offset, then crash before writing the output) gives you at-most-once, also bad. The right behaviour is that the offset advance and the output write either both happen or neither happens. That is a transaction, distributed across the input source, the processor, and the output sink.

For most combinations of source and sink, that transaction does not exist. There is no off-the-shelf protocol that makes “advance Kafka offset and insert a row in Postgres” atomic. The two systems have separate notions of commit, separate failure semantics, no shared coordinator. That is the heart of the problem.

Kafka exactly-once semantics, what it actually covers

Kafka ships a feature called exactly-once semantics, abbreviated EOS. It is not a single switch. It is three cooperating mechanisms that together give you a transaction across Kafka topics.

Idempotent producer. Every producer instance has a unique producer ID, and every batch of records it sends carries a sequence number. The broker remembers, per partition, the latest sequence it has accepted from each producer. If a network blip causes the producer to retry a batch the broker already wrote, the broker recognises the duplicate sequence and returns the original ack instead of writing twice. The producer-side retry, which would normally be a duplicate source, becomes a no-op at the broker. This is enabled by setting enable.idempotence=true and is the floor that everything else builds on.

Transactions. A producer can open a transaction, write to multiple partitions across multiple topics, and commit. From the consumer’s perspective, the writes either all become visible together or none of them do. The implementation uses a transaction coordinator (one of the brokers) and two-phase-commit-style markers in the partition logs. The producer also has a transactional ID that survives restarts, which prevents two zombie instances of the same producer from both writing inside what they each think is the same transaction.

Read-committed isolation. Consumers can be configured with isolation.level=read_committed. They will skip any records that are part of an open or aborted transaction and only see records that belong to a committed transaction. Without this, consumers see everything written, including aborts, and the transaction’s atomicity is invisible.

Offset commit inside the transaction. This is the bit that closes the loop. When a Kafka Streams or transactional consumer-producer pipeline commits its work, the offset advance for the input topic is included as a write inside the same transaction as the output writes. Either the offsets and the outputs both commit, or both abort.

Put together, this gives you a strong guarantee, but only inside a specific boundary: input from Kafka, processing, output to Kafka, all within the same Kafka cluster (or, with newer features, federated clusters that share a transaction coordinator). Inside that boundary, no input record produces an output more than once.

sequenceDiagram
    participant Consumer
    participant Producer as Transactional producer
    participant TC as Transaction coordinator
    participant P1 as Output partition 1
    participant P2 as Output partition 2
    participant Off as Consumer offset topic
    Consumer->>Producer: read records (offset 100 to 105)
    Producer->>TC: beginTransaction
    Producer->>P1: write records
    Producer->>P2: write records
    Producer->>Off: send offsets (advance to 106)
    Producer->>TC: commitTransaction
    TC->>P1: write commit marker
    TC->>P2: write commit marker
    TC->>Off: write commit marker
    Note over P1,Off: read_committed consumers see all three together

Kafka Streams uses this machinery automatically when processing.guarantee=exactly_once_v2 is set. The runtime opens a transaction per task per commit interval, writes outputs and offsets inside it, and commits. As long as the topology stays inside Kafka, the guarantee holds.

Where it stops

The boundary, in plain terms, is “inputs and outputs that the Kafka transaction coordinator can include in a commit.” That is Kafka topics, plus the consumer offset topic. It is not Postgres. It is not S3. It is not an HTTP API. It is not Elasticsearch. It is not a metrics system. None of those participates in the transaction.

A Kafka Streams job that reads from a topic, does some processing, and writes to another topic: exactly-once is real and you can rely on it. A Kafka Streams job that reads from a topic and pushes results to a Postgres database via a sink connector: exactly-once is a partial fiction. The transaction commits the output to Kafka (the connector reads from a Kafka output topic), but the connector’s write into Postgres happens outside the Kafka transaction. If the connector crashes between writing to Postgres and committing its own offset, on restart it re-writes. Duplicates in Postgres.

The same thing happens for any external sink. The Kafka transaction stops at the edge of Kafka. Whatever lies beyond the edge has to handle duplicates on its own.

The fixes for external sinks

There are three options, in descending order of how often they actually work in practice.

Idempotent sinks. This is the right answer in almost every case. Design the sink so that re-writing the same output produces the same end-state. The patterns from lesson 16 apply directly. Upsert into Postgres on a unique key. POST to an HTTP API with an Idempotency-Key header that the receiver remembers. Write to S3 with a deterministic object key derived from the input, so that the second write either replaces the first or is rejected as a duplicate. The streaming engine’s exactly-once guarantee covers everything up to the sink, and the sink’s idempotency covers the rest. Together they give you end-to-end correctness.

The discipline is that the sink’s idempotency has to be designed in, not assumed. A naive Postgres INSERT is not idempotent. An INSERT ... ON CONFLICT (key) DO UPDATE is. A naive HTTP POST is not. A POST with a server-side dedup table keyed on Idempotency-Key is. The streaming engine cannot make a non-idempotent sink behave; you have to fix the sink.

Two-phase commit-style coordination. Some sinks support a write-then-commit protocol that the streaming engine can drive. Flink calls these “two-phase commit sinks.” The engine writes the output during the streaming transaction, holds the data in a pending state, and tells the sink to commit only after the engine’s checkpoint succeeds. If the engine fails, it tells the sink to abort. The sink has to support pending writes and explicit commit or abort calls, which most do not. JDBC and Kafka sinks have implementations. Most other sinks do not. When you have it, it works. When you do not, the option is closed and you fall back to idempotent writes.

At-least-once plus downstream dedup. The pragmatic option when neither of the above is feasible. The streaming engine is at-least-once. The sink accepts duplicates. A downstream batch step or a query-time aggregation deduplicates by an event ID. This is the same shape as the append-with-dedup pattern in lesson 38. It works, and it is operationally heavier than the idempotent-sink option, because every read or every batch step has to pay for the dedup. Reach for it when the sink really cannot be made idempotent.

What this means in practice

The honest summary is short. Exactly-once is a system-wide property, not a checkbox in a config file. Kafka EOS is excellent for the part of the system it covers, which is Kafka-to-Kafka. For anything else, the streaming engine gives you a strong guarantee up to the sink, and you, the engineer, are responsible for the sink’s behaviour.

The architectural rule that falls out of this is the same as Module 2’s rule, restated for streaming: design every sink to be idempotent, and treat exactly-once labels with skepticism. Identify the boundary the label applies to. Identify the sinks outside that boundary. Make those sinks idempotent on their own.

There are two specific traps worth flagging.

The first trap is enabling EOS without read-committed consumers. The producer side is transactional, the consumer side is not, and the consumer reads aborted writes alongside committed ones. The output looks fine in the topic but is wrong when consumed. EOS is the producer plus the consumer’s isolation level plus the offset commit, all together. Half a configuration is no configuration.

The second trap is assuming Kafka Connect sinks inherit EOS. They do not, by default. A connector that writes to a non-Kafka system is at-least-once at the sink edge unless the sink-specific connector implements two-phase commit and the destination supports it. Confluent’s documentation is explicit about which connectors are exactly-once-capable and under what configuration. Read that page before you promise the property to a stakeholder.

Where this leaves us

Module 6 has now covered, in turn, the engines, the topologies, the state, the time, and the delivery semantics of streaming. The next lesson moves to a different but adjacent problem: how state changes in your databases get into the streaming layer in the first place, without violating the consistency guarantees of either the database or the message bus. That is change data capture and the outbox pattern, and they are the bridge between the OLTP world the rest of the company writes code in and the streaming world this module has been describing.

The thread that runs through all of these lessons is the same one that ran through Module 2: distributed correctness is not free, and the patterns that make it tractable are a small set, applied consistently, until they stop being techniques and become how you build by default. Exactly-once is one of those patterns. The discipline is knowing exactly what your engine guarantees, exactly where the guarantee stops, and exactly which lines of code at the sink make up the difference.

Citations and further reading

  • Confluent, “Exactly-Once Semantics Are Possible: Here’s How Kafka Does It”, https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ (retrieved 2026-05-01). The original announcement, with careful scoping.
  • Apache Kafka documentation, “Transactions”, https://kafka.apache.org/documentation/#transactions (retrieved 2026-05-01).
  • Confluent, “Patterns for streaming microservices” (retrieved 2026-05-01). The streaming-microservices write-up that covers EOS scope, sink idempotency, and the two-phase-commit sink pattern.
  • Tyler Akidau, Slava Chernyak, Reuven Lax, “Streaming Systems” (O’Reilly, 2018). Chapters on consistency models in streaming and the role of side effects. Retrieval 2026-05-01.
  • Apache Flink documentation, “End-to-End Exactly Once Processing in Apache Flink with Apache Kafka”, https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka.html (retrieved 2026-05-01).
Search