Exactly-Once Is a Property of Your Consumer, Not Your Broker

Every few months someone asks me whether Kafka gives you exactly-once delivery. The honest answer is that it gives you something narrower than the phrase suggests, and that the useful guarantee lives in code you write rather than in a broker setting you enable.

I want to lay out what Kafka actually promises, where the promise stops, and the handful of patterns that make the gap survivable. This is the mental model I built running a real-time telemetry pipeline for construction and mining fleets, where machines pushed continuous data from sites with patchy connectivity and reprocessing was a daily fact rather than an edge case.

The three delivery semantics, and why one of them is mostly marketing

At-most-once. Commit the offset before you process the message. If the consumer dies in between, the message is gone and nobody will tell you. Occasionally this is what you want, for high-volume metrics where a gap is cheaper than a duplicate. Usually it is not.

At-least-once. Process the message, then commit the offset. If the consumer dies in between, the message is redelivered on restart. This is the default posture and the right one for almost everything. It guarantees you will see every message. It also guarantees that, given enough uptime, you will see some of them twice.

Exactly-once. Kafka does support this, and the support is real, but it is scoped more tightly than most people assume. The transactional producer, combined with a consumer reading at isolation.level=read_committed, gives you atomic read-process-write within Kafka. You consume from a topic, produce to another topic, and commit the consumed offsets as part of the same transaction:

producer.initTransactions();

while (running) {
    var records = consumer.poll(Duration.ofMillis(200));
    producer.beginTransaction();
    for (var record : records) {
        producer.send(transform(record));
    }
    producer.sendOffsetsToTransaction(offsetsOf(records), consumer.groupMetadata());
    producer.commitTransaction();
}

That is genuinely exactly-once. The offsets and the output records commit or roll back together, so there is no window where you have produced output but not recorded that you consumed the input.

Now put a database write, an HTTP call to a payment provider, or an email in the middle of that loop. The transaction cannot cover it. Kafka can roll back its own writes; it cannot un-send an email. The moment your side effects leave Kafka, exactly-once becomes a distributed transaction problem across systems that do not share a transaction coordinator, and you are back to at-least-once with extra steps.

So the position I have settled on, and the one I would defend in a design review:

Treat every consumer as at-least-once. Put the guarantee in your write path. Exactly-once is at-least-once plus an idempotent consumer.

Making the consumer idempotent

The word gets used loosely, so here is what it means mechanically.

Derive the key from the event, not from the consumer. If you generate a UUID when you process the message, a redelivery generates a different one and you have achieved nothing. The key has to be stable across redeliveries: a producer-supplied event ID, or a deterministic hash of the fields that identify the operation, such as tenant, entity, operation and version.

Keep a dedup record with a bounded lifetime. A unique constraint in Postgres or a SET NX in Redis both work. The TTL should be sized to your realistic replay window rather than kept forever. If you can replay seven days of a topic, seven days of dedup keys is the floor.

insert into processed_events (event_id, tenant_id, processed_at)
values ($1, $2, now())
on conflict (event_id) do nothing;
-- zero rows affected means we have seen this one; skip the side effect

Prefer naturally idempotent writes. set balance = 500 is idempotent. balance = balance + 100 is not. Where the domain lets you model the write as a statement of desired state rather than a delta, take it, because then redelivery is harmless without any dedup machinery at all.

Duplicates and disorder are different problems

This distinction gets collapsed constantly, and collapsing it produces bugs that only appear under load.

Deduplication stops you applying the same message twice. It does nothing about applying an older message after a newer one. Redelivery, consumer lag, a partition reassignment or a producer retry can all deliver events out of order relative to what your database already knows.

The fix is a conditional write. Carry a version or a source timestamp on the event and refuse to move state backwards:

update devices
   set status = $1,
       version = $2,
       updated_at = $3
 where device_id = $4
   and version < $2;

If the update affects zero rows, a newer version already landed and the message is safely dropped. This is optimistic concurrency, and it is the single most useful line of defence in an event-driven system. It costs one column.

Note that ordering is only ever guaranteed within a partition. If ordering matters for an entity, that entity has to key to a stable partition. Which brings us to the part people get wrong at scale.

Partitioning is the decision you cannot easily undo

Kafka guarantees order within a partition and nothing across partitions. So the partition key is not a performance knob, it is a correctness decision.

Key by the entity whose ordering you care about. Usually that is a tenant ID or a resource ID. Then watch for two failure modes.

Hot partitions. One large tenant produces most of your traffic, hashes to one partition, and that partition becomes the bottleneck for everyone sharing it. Mitigations are a composite key that spreads a large tenant across several partitions where per-entity ordering still holds, or routing the largest tenants to dedicated topics.

Repartitioning breaks ordering. Increasing the partition count rehashes keys, so an entity that used to land on partition 3 may now land on partition 7 while messages for it are still in flight on partition 3. There is no clean online fix. Size partitions with headroom you expect to need, because raising the number later is not free.

Rebalancing, and how a slow consumer takes down a group

Within a consumer group, each partition is owned by exactly one consumer. Add more consumers than partitions and the extras sit idle.

The failure I have seen most often is a rebalance storm. A consumer takes too long between poll() calls, exceeds max.poll.interval.ms, and the coordinator assumes it is dead. Its partitions are reassigned, which pauses processing group-wide, which pushes another consumer past the threshold, and the group spends its time rebalancing instead of consuming.

Three things help. Reduce max.poll.records so a single batch cannot take too long. Move slow work off the poll thread so heartbeats keep flowing. Use the cooperative sticky assignor so a rebalance does not stop every consumer in the group at once.

The metric to alert on is consumer lag, and specifically its rate of change rather than its absolute value. Lag of ten thousand that is falling is fine. Lag of five hundred that is climbing steadily is an outage forming.

Do not let one bad message block a partition

A message that always throws will be retried forever, and because ordering is per partition, it blocks every message behind it. On a partition shared by many tenants that is a head-of-line outage for all of them, caused by one malformed payload.

Retry with backoff, bound the attempts, then move the message to a dead letter topic carrying the original headers plus the failure reason and a stack trace. Then alert on the DLQ, because a dead letter queue nobody watches is just a slower way to lose data.

What I would tell someone starting out

Assume duplicates. Assume disorder. Put the guarantee in your own write path, where you can test it, rather than in a broker configuration you will misread under pressure.

Concretely, that is four things: a stable idempotency key derived from the event, a dedup record with a sensible TTL, a conditional write that refuses to move state backwards, and a dead letter path with an alert on it. None of that is exotic, and together it is worth more than any amount of configuration tuning.

The systems that survive are not the ones that never redeliver. They are the ones where redelivery does not matter.