Exactly-Once Processing Is a Myth: Why Duplicate Events Are Inevitable

Summary: Exactly-once processing is the idea that an event is handled once and only once, end to end. In practice it isn't achievable across an unreliable network, because a lost acknowledgment is indistinguishable from a failure. Production systems choose at-least-once delivery instead and make duplicate processing harmless. This blog explains why, using real duplicate-event failures from payment webhooks, SQS and Lambda.

The first time I saw the same webhook event processed twice, I assumed we had a bug.

It was a payment-related service. A customer had been notified twice about the same payment. Somewhere in our code, I figured, there was a loop that shouldn't be there, or a handler wired up twice, or a retry we'd forgotten about. Something we'd done. It took almost two months to understand that the duplicate wasn't a defect in our implementation. It was a property of the system we'd built.

In Part 1 of this series, I covered seven challenges that only show up in production. This post takes the third one — duplicate delivery — all the way down. Because once you accept that exactly-once processing isn't available to you, the question changes shape. You stop asking why am I getting duplicate events and start asking how should my system behave when duplicates arrive. That second question has an answer. The first one mostly doesn't.

At-Most-Once, At-Least-Once, Exactly-Once: The Three Delivery Guarantees

Every messaging system makes you pick one of three promises. They're worth stating precisely, because most of the confusion around exactly-once processing comes from people using the words loosely. The Kafka project's documentation on delivery semantics is the clearest primary source if you want the formal version.

At-most-once. The sender fires the message and moves on. If it arrives, it's processed once. If the network drops it, it's gone and nobody retries. You will never see a duplicate. You will sometimes lose events entirely.

At-least-once. The sender waits for an acknowledgment. If the acknowledgment doesn't arrive in time, it sends again. Nothing gets lost. But if the original message was processed and only the acknowledgment went missing, the retry produces a duplicate.

Exactly-once. Every message is processed once and only once, with no losses and no duplicates. This is what everybody wants. It is not something you can get end to end across a network, for reasons we'll get to in a moment.

That last row is the whole post, so let's take it apart.

Why Exactly-Once Delivery Breaks When an Acknowledgment Is Lost

Here's the scenario that ends the argument.

A payment provider sends you a PaymentSucceeded webhook. Your handler receives it, validates it, updates the order, queues a confirmation email. Everything works. You return 200 OK.

And the response never arrives. A connection resets, a load balancer times out, a route flaps — the reason doesn't matter. What matters is that the acknowledgment died on the way back.

Payment Provider                    Our Webhook Handler
         │                                     │
         │ ─────── PaymentSucceeded ────────►  │
         │                                     │
         │                                 processed
         │                              (order updated,
         │                               email queued)
         │                                     │
         │ ◄────── 200 OK ──────X              │
         │          (lost in transit)          │
         │                                     │
    no response                                │
         │                                     │
         │ ─────── PaymentSucceeded ────────►  │
         │            (retry)              processed again
         │                                     │

Now look at it from the provider's side. It sent an event and got nothing back. It cannot tell the difference between your service never received this, your service received it and crashed, and your service processed it perfectly and the response got lost. All three look identical from where it's standing. So it does the only sensible thing available to it: it retries.

Part 1 walked through three different failures that look identical to the caller. This is what happens when the second of those — a lost response after successful processing — meets a sender that retries. You get a duplicate, and neither side did anything wrong.

You can push the acknowledgment earlier. You can make it faster. You cannot make it certain, because there is no point at which both machines simultaneously know that the other one knows.

Debugging Duplicate SQS Messages: Why It Took Me Two Months

The theory above is clean. Finding it in production was not.

The service consumed webhook events asynchronously through SQS and Lambda. A webhook arrived, we acknowledged it, we pushed a message onto a queue, and a Lambda consumer picked it up and did the real work. Standard enough.

When the first duplicate surfaced, I had three suspects. The payment provider might have sent the event twice. SQS might have redelivered the message — which it will do if a consumer doesn't acknowledge within the visibility timeout. Or Lambda might have retried the invocation.

So I went looking. I pulled up the CloudWatch logs for the payment service, searched for the webhook event ID, compared the timestamps on each occurrence, and traced the event through the webhook handler, into SQS, and out to the Lambda consumer.

 Duplicate notification reported
                │
                ▼
     Search CloudWatch by event ID
                │
                ▼
        Compare timestamps
                │
      ┌─────────┼─────────┐
      ▼         ▼         ▼
   Webhook     SQS      Lambda
   handler   delivery  consumer
      └─────────┼─────────┘
                ▼
      Still can't tell which

And that's where it ended. We could not conclusively determine whether the duplicate originated with the webhook sender or with the queue. Both were plausible. Neither left a fingerprint clear enough to convict.

That's not a satisfying result, and for a while it didn't feel like a finding at all. The issue was random and hard to reproduce, so it stayed in the pile of things that were odd but not urgent. It only became a real problem as we onboarded more customers. The bugs team started reporting the same symptom more often, from different customers, in different flows.

That's what made it click. Not a smoking gun in a log file — the pattern across customers. One duplicate is an anomaly you can shrug at. The same duplicate arriving through four different code paths, from four different accounts, is telling you something about the architecture. We'd been treating a property of the system as a series of isolated bugs.

One Duplicate Event, Two Payment Notifications

The incident that forced the issue was straightforward on the surface. A customer received the same payment notification twice, complained, and the bugs team escalated it. We traced the customer and the payment event through CloudWatch and found the same webhook event processed more than once.

What made it interesting was where the damage actually landed.

The payment itself was never the problem. The duplicate event made the downstream notification fire twice. Even when the upstream provider behaves perfectly, duplicate delivery anywhere in the pipeline produces duplicate business actions unless the consumer is idempotent.

That distinction reframed how I thought about the whole class of problem. We had been asking whether the payment provider was misbehaving. Wrong question. The provider could be flawless and we'd still have shipped two emails, because the thing that ran twice wasn't the payment — it was our side effect.

It's also the kind of failure that keeps every dashboard green. No error rate moved. No health check failed. Nothing retried into a dead-letter queue. The system did exactly what it was told, twice, and the only signal was a customer noticing something in their inbox.

Why Production Systems Choose At-Least-Once Delivery on Purpose

My first instinct after that incident was to go looking for a system that wouldn't produce duplicates. What I found instead was that every tool already in our stack had deliberately chosen to produce them.

Razorpay retries webhook deliveries that don't return a 2xx — the behaviour is documented in their webhooks guide, and it's the reason our provider kept sending. SQS redelivers a message when the visibility timeout expires without an acknowledgment. Kafka consumers replay records after a failure or a rebalance.

None of these is a bug. They're all the same decision, made independently: given a choice between occasionally delivering an event twice and occasionally losing one, lose nothing.

That's the right call, and it isn't close. A duplicate PaymentSucceeded costs you two confirmation emails and an awkward support conversation. A lost PaymentSucceeded costs you a customer who paid and never got their order. One of those is embarrassing. The other is a refund, a support ticket, and a customer who doesn't come back.

Retries are a feature. The duplicates are the bill for it.

Does Kafka Have Exactly-Once Semantics? What It Actually Covers

This is where most discussions of exactly-once processing end up, and it's the first thing someone asks when you describe a duplicate problem. So it's worth being precise about what Kafka's guarantee covers.

Kafka's transactional model has three moving parts. Idempotent producers (enable.idempotence) prevent a producer's own retries from writing the same record twice. Transactional producers let you write to multiple partitions atomically. Consumers set to isolation.level=read_committed only read records from transactions that actually committed. Put together, these give you exactly-once semantics — Confluent's walkthrough of how it works is the best explanation of the mechanism.

The important word in that guarantee is boundary. It holds for Kafka-to-Kafka processing: read from a topic, transform, write back to a topic, commit offsets inside the transaction. That's a genuinely hard problem and Kafka solves it.

It stops at Kafka's edge. The moment your consumer writes a row to Postgres, calls a payment API, or sends an email, that side effect is outside the transaction. Kafka can guarantee the record was consumed exactly once. It cannot un-send your email.

Which leads to the test I'd actually apply, and the one that decided it for us:

Does your duplicate hurt inside the message system, or at the side effect? If it's the side effect, no delivery guarantee at any layer fixes it.

We were on SQS and Lambda. We didn't reach for transactional guarantees, and not because they'd have been hard to adopt — because they wouldn't have touched the thing that was hurting us. Our duplicate notification came from a side effect running twice. That's outside every boundary any broker can draw.

Does SQS FIFO Guarantee Exactly-Once Processing?

The same question came up on our side, and it's the one I'd have asked too: couldn't we just switch to a FIFO queue?

FIFO queues deduplicate at the queue level. You supply a MessageDeduplicationId, or let SQS hash the message body, and duplicates within a five-minute window are dropped. That's real, and it's useful. If your duplicates come from a producer retrying into the same queue, FIFO will handle a good share of them.

What it doesn't do is guarantee that a business operation happens once.

The same business event can reach your system by another path — a webhook and a reconciliation job, say, both reporting the same payment. The dedup window expires; a retry seven minutes later is a new message as far as the queue is concerned. And a consumer can crash after performing the side effect but before acknowledging, at which point the message comes back and the side effect runs again, with no duplicate message ever having existed. Google Pub/Sub's exactly-once delivery has the same shape and the same limit.

Queue-level deduplication is protection. It isn't correctness. It reduces how often you're asked the hard question; it doesn't answer it.

Message Ordering Is a Separate Problem From Deduplication

Worth separating out, because it gets tangled up with this and it's a different problem.

If PaymentSucceeded arrives before PaymentInitiated, deduplication does nothing for you. That event isn't a duplicate — it's simply early, and your handler is missing state it expected to exist. Ordering guarantees are handled at the messaging layer, and which mechanism you need depends on the workflow. Part 1 covers out-of-order events in more detail.

Different problem, different tools. Solving one doesn't help with the other.

How to Handle Duplicate Events: Idempotency at the Consumer

Once you accept that duplicate delivery can happen, the goal changes. You stop trying to guarantee that an event arrives only once, and start making sure that processing it twice has the same effect as processing it once.

For our payment webhooks, every event came with a stable event ID. Say the provider sends:

evt_123

The obvious approach is to check whether you've seen that ID before, and skip it if you have.

Event arrives
              │
              ▼
   Have we processed evt_123?
              │
          ┌───┴───┐
         Yes      No
          │       │
        Skip   Process
                  │
                  ▼
            Record evt_123

There's a subtle bug hiding in this version.

Imagine two consumers receive the same event at almost the same time:

 Consumer A                  Consumer B

  check evt_123               check evt_123
        │                           │
    not found                   not found
        │                           │
  send notification           send notification

Both checked before either had recorded anything. The duplicate check exists, and the side effect still happens twice.

So the important operation isn't checking whether an event exists. It's atomically claiming it, before doing any work rather than after:

INSERT evt_123, status = 'processing'
with UNIQUE(event_id)

Only one consumer can claim that ID. The other hits the uniqueness conflict, and what it does next depends on what the existing row says. If the status is done, the event has already been handled and the consumer can safely skip it. If the status is still processing, another consumer either holds it right now or died holding it, so the right move is to leave the message on the queue and let redelivery sort it out.

That status column matters more than it looks. Claim without it and you've traded a duplicate-processing problem for a lost-event problem: a consumer that claims evt_123 and then crashes before sending the notification leaves a row that tells every future attempt the work is finished. Nothing gets sent, and nothing reports an error. You'll also want a timeout on stale processing rows, or one dead consumer blocks that event forever.

The pattern ends up as:

Stable deduplication key → atomic claim → perform the side effect → mark it done.

Exactly how you implement the claim depends on your datastore. A unique constraint, a conditional write, a compare-and-set operation, or any other atomic primitive can all serve the same purpose. Stripe's idempotent requests documentation is the reference implementation most teams end up working from.

Where should deduplication happen?

There are two useful places.

       Webhook arrives
              │
              ▼
   ┌──────────────────────┐
   │  Ingestion layer     │  dedup here = efficiency
   │  drop obvious        │  fewer messages, less work
   │  repeats             │
   └──────────┬───────────┘
              │
              ▼
          SQS Queue
              │
              ▼
   ┌──────────────────────┐
   │  Consumer            │  idempotency here = correctness
   │  atomically claim    │
   │  dedup key           │
   └──────────┬───────────┘
              │
              ▼
      Perform side effect

Deduplicate at ingestion for efficiency. Be idempotent at the consumer for correctness.

Dropping duplicates at the webhook layer saves queue traffic and unnecessary processing, but it can't be your only protection, because a duplicate can be introduced after ingestion. The component performing the side effect has to protect itself.

The deduplication key isn't always an event ID

A provider-generated event ID is ideal, but real integrations aren't always that convenient. The key just needs to represent the logical event you're trying to process once.

This tends to be integration-specific rather than something you can read off an architecture diagram.

And sometimes the duplicate isn't caused by your queue or the provider at all. It's caused by how you acknowledge the webhook.

On one hosted-checkout integration, we started receiving the same webhook repeatedly, and it looked like the provider was sending duplicates for no reason. The actual problem was us. We weren't returning a successful response quickly enough, so from the provider's side delivery had failed and it retried.

The fix was to acknowledge the webhook immediately and move the real processing behind SQS:

     Provider
        │
        ▼
  Webhook endpoint
        │
        ├──► return 200 quickly
        │
        ▼
       SQS
        │
        ▼
     Consumer

That solved one reliability problem and introduced an interesting property: once the event was safely queued, SQS itself could redeliver it. The system needed both ideas at once. Acknowledge quickly so the producer doesn't retry unnecessarily, and process idempotently because retries and redelivery can still happen anyway.

As for the payment service I spent two months on, we never did find out where those duplicates came from. Eventually that stopped mattering. We stored the provider's event ID behind a uniqueness constraint, made the notification consumer claim an event before acting on it, and the duplicate reports stopped. Not because duplicate delivery went away, but because it could no longer produce a duplicate outcome. That's the distinction worth taking from all of this: the goal was never to stop duplicates arriving, only to stop them mattering.

What Comes Next

This article covered why duplicates are inevitable and why no delivery guarantee removes them. In the next article, we'll look at idempotency properly — how to design an operation so that processing the same event five times leaves the system in exactly the same state as processing it once, what to use as a key, where to store it, and how to make the claim atomic without turning your database into a bottleneck.

The shift that mattered most for me wasn't learning a pattern. It was giving up on a guarantee I was never going to get, and designing for the world as it actually behaves.

Duplicate events aren't an edge case you patch later. They're a design input, and the cost of treating them as one shows up as double charges and duplicate notifications months after launch.

We build event-driven backends on the assumption that every event can arrive twice. If your team is dealing with duplicate processing, webhook reliability, or queue consumers that were never designed to run more than once, that's the work our backend engineering team does.

And if you've hit a duplicate-event problem of your own — or you think I've got something wrong in here - I'd like to hear about it. You can find me on LinkedIn.

Frequently Asked Questions

Is exactly-once delivery possible?

Not end to end across a network. Some systems provide exactly-once processing within their own boundary, but the moment an acknowledgment can be lost, the sender cannot distinguish success from failure. It retries, and a duplicate arrives. The dropped-acknowledgment section above walks through why this can't be engineered away.

What is the difference between exactly-once and at-least-once processing?

At-least-once guarantees no event is lost, and accepts that some will be delivered more than once. Exactly-once claims no event is lost and none is duplicated. In practice you get at-least-once delivery plus idempotent processing, which produces an exactly-once outcome without an exactly-once guarantee.

What is delivery semantics: at-most-once, at-least-once, exactly-once?

They're the three promises a messaging system can make about how often a message reaches its consumer. At-most-once never duplicates but can lose events. At-least-once never loses events but can duplicate them. Exactly-once claims neither. The comparison table near the top of this post covers where you'll encounter each.

How does Kafka achieve exactly-once semantics?

Through idempotent producers, transactional producers, and consumers reading with isolation.level=read_committed. Together these give exactly-once processing within a Kafka-to-Kafka transactional boundary. As the Kafka section above explains, the guarantee ends where Kafka does — external side effects like database writes or API calls fall outside it.

Does SQS FIFO guarantee exactly-once processing?

FIFO queues deduplicate messages within a five-minute window using a deduplication ID. That prevents duplicate messages, not duplicate business operations. The same event can arrive by another path, or a consumer can crash after performing a side effect but before acknowledging. Queue-level deduplication is protection, not correctness.

Why do duplicate webhook events happen?

Usually because the sender didn't receive a fast enough success response and retried. That can mean your endpoint was slow, the acknowledgment was lost in transit, or the request genuinely failed. The sender can't tell which. As the debugging section above describes, acknowledging immediately and processing asynchronously removes a common cause.

What is idempotency?

An operation is idempotent when performing it multiple times produces the same result as performing it once. In event processing, it means using a stable key to recognise an event you've already handled and skipping the side effect. It's the subject of the next post in this series.