Idempotency in Distributed Systems: How to Safely Process the Same Event Twice

Summary: An operation is idempotent when running it five times leaves the system in the same state as running it once. In event-driven systems this is the practical answer to duplicate delivery, which no messaging guarantee removes. This blog covers choosing a deduplication key, claiming it atomically, recovering work from a consumer that died mid-processing, and monitoring the whole thing in production.

The obvious deduplication key for a shipping event is the order ID. One order, one key, no duplicate processing. It looks right.

Then IN_TRANSIT arrives and gets handled. Ten minutes later OUT_FOR_DELIVERY arrives, the consumer checks its records, finds that order ID already there, and drops the event as a duplicate. The shipment keeps moving. The tracking page does not.

Idempotency in distributed systems is full of decisions like that one, where the protection you added becomes the thing doing the damage. Part 2 of this series covered why duplicate events are unavoidable once anything in your pipeline retries. This post is the other half: what you build in response, and the handful of choices inside it that are easy to get wrong in both directions.

What Is Idempotency in Distributed Systems?

An operation is idempotent if applying it repeatedly produces the same result as applying it once. Setting an order's status to PAID is idempotent: run it five times and the order is still paid. Adding ₹500 to a balance is not: run it five times and the customer has ₹2,500 they should not have.

That difference is the entire design problem. Some of the work your consumers do is already safe to repeat. The rest has to be made safe.

Everything in the second half of that table needs protection. Everything in the first half does not, and sorting them before you build anything saves you machinery you do not need.

Which is the framing that matters: idempotency is not really about detecting duplicate messages. It is about controlling their effects, and the table above is where you work out which effects need controlling.

Why Event Deduplication Fails When Two Consumers Race

The obvious approach is a list of events you have already handled. An event arrives, you check whether its ID is in the list, and if it is not, you process it and add it.

        Event arrives
              │
              ▼
   Have we processed evt_123?
              │
          ┌───┴───┐
         Yes      No
          │       │
        Skip   Process
                  │
                  ▼
            Record evt_123

This works until two consumers pick up the same event at nearly the same moment.

   Consumer A                  Consumer B

  check evt_123               check evt_123
        │                           │
    not found                   not found
        │                           │
  send notification           send notification

Both ran the check before either recorded anything. The duplicate check is present, correct, and useless, because the window between reading and writing is wide enough for two workers to pass through it.

This is a check-then-act race, and it is the reason a duplicate-processing bug tends to appear under load and vanish when you try to reproduce it. One consumer never hits it. Several consumers and a busy queue will.

How to Claim an Event Atomically: Unique Constraints and Conditional Writes

The fix is to stop checking and start claiming. Rather than asking whether an event has been seen, try to take ownership of it in a single operation that only one worker can win, and let the database decide the winner.

In a relational database, a unique constraint does this for you. PostgreSQL's ON CONFLICT clause makes the losing write a no-op rather than an error:

CREATE TABLE processed_events (
    event_id   VARCHAR(255) PRIMARY KEY,
    status     VARCHAR(32)  NOT NULL,
    created_at TIMESTAMP    DEFAULT CURRENT_TIMESTAMP
);

INSERT INTO processed_events (event_id, status)
VALUES ('evt_123', 'PROCESSING')
ON CONFLICT DO NOTHING;

Exactly one insert succeeds. Every other consumer gets zero rows affected and knows, without a separate query, that someone else owns this event.

The same idea works outside SQL. DynamoDB gives you the equivalent through a conditional write, where the put only succeeds if the key does not already exist. Whichever store you use, the property you need is the same: a single operation that both tests and takes, with no gap in between for a second worker to slip through.

The Idempotent Consumer Problem: Claiming an Event Is Not Completing It

Here is the trap that the fix above creates.

A worker wins the claim, writes its row, and then stops before finishing the actual work. Nothing dramatic causes this. The Lambda hits its timeout, runs out of memory, or throws after the claim has already been committed. Lambda then stops that invocation. The claim row is left behind, and the work it was supposed to protect never finishes. The queue never got an acknowledgment, so it redelivers the event, which is correct behaviour. The next consumer looks up evt_123, finds a row, concludes the event is handled, and skips it.

  Worker A claims evt_123
            │
            ▼
      Worker A dies
            │
            ▼
   Queue redelivers evt_123
            │
            ▼
  Worker B sees the row, skips
            │
            ▼
   Refund never happens, silently

The protection against doing the work twice has become the reason it is never done at all, and that is the worst failure. A duplicate refund can be clawed back. A refund that silently never happened sits there until the customer complains.

So the record has to carry a state, not just an existence. PROCESSING, COMPLETED, FAILED. A consumer that hits a conflict then reads the existing row and decides:

• COMPLETED: the work is genuinely done, skip it and acknowledge the message.
• FAILED: it broke, this attempt can take it.
• PROCESSING: another consumer either holds it right now or died holding it.

That last case is the one teams skip, and it needs a rule rather than a guess. The practical approach is a timeout: a row stuck in PROCESSING past some age is treated as abandoned and made available again. Where the work already sits behind a queue, the SQS visibility timeout is doing a version of this for you at the message level, and the two timeouts need to agree with each other, or a message comes back while its row still looks live.

Without that rule, one dead worker blocks one event forever, and nothing reports an error.

How to Choose an Idempotency Key

There is no universal idempotency key. The key has to represent the business operation you want to happen exactly once, and that changes with every integration.

A payment provider that sends a stable event ID with every webhook makes this easy. Use their ID. It is generated once, travels with every retry of that event, and means precisely what you need it to mean. Stripe's idempotent requests documentation is the reference implementation most teams end up working from, whether or not they ever touch Stripe.

The interesting cases are the ones where the obvious key is wrong, and the shipping example this post opened with is the common shape of it. One order produces a series of legitimate events over its life, so a key built from the order alone makes every status after the first look like a repeat of it. The working key there is the order identity combined with the status, which gives each transition its own key and deduplicates it against itself rather than against its siblings.

Getting that right is a judgement call about the domain rather than a technical one, and it is the part of this that comes with having been wrong before.

A key that is too narrow lets the same work run twice. A key that is too broad throws away work that was genuinely new. The second failure is quieter, and it is the one that takes longer to find.

Across the integrations we have handled, the key has been different almost every time:

If you are hashing a payload because nothing stable is on offer, check what goes into the hash. Anything the sender varies between retries, a timestamp or an attempt counter, makes every retry look like a new event and the hash stops working.

Where to Store Deduplication Keys, and What They Cost to Keep

Idempotency is not free. You are adding at least one write and one read to every event, and a table that grows roughly in line with your event volume.

That table does get large, and the honest framing is that this becomes an ordinary database problem rather than an idempotency one. Retention, indexing and partitioning are the same tools you would reach for with any high-volume table. The one thing specific to this case is that the retention window has to outlive the longest retry window of anything that talks to you. Drop a key earlier than that and a late retry arrives to find no record of itself.

Weighed against one customer being charged twice, an extra write per event is not a close call. It is worth doing the arithmetic once at your own volume so the decision is made on a number rather than on instinct.

How to Monitor Duplicate Processing With AWS CloudWatch, SNS, and Slack

If your idempotency logic silently stopped working tonight, would anything other than a customer tell you? For most teams that have implemented the pattern, the honest answer is no, which is the difference between a pattern you have written and one you can trust.

For our services the answer runs through CloudWatch. Consumers log what they did with each event, including the claim outcome, so a duplicate that was correctly rejected is visible rather than invisible. A CloudWatch alarm watches the failure rate on the consumer. When it crosses the threshold the alarm publishes to SNS, and from SNS we push it into Slack through Amazon Q, so it lands in a channel someone is already watching rather than in an inbox nobody checks. From there it is the same path as the two-month debugging hunt in Part 2: search by event ID, compare timestamps, follow the event through the handler, the queue and the consumer.

Two things need instrumenting specifically, because a generic error rate catches neither:

• Conflicting claims. A count of claims rejected because the key already existed. This number going to zero is a signal, not a success.
• Rows aging in PROCESSING. The age of the oldest unfinished claim is the cheapest proxy available for work that stopped without telling anyone.

Neither needs a new tool. Both need someone to decide they deserve a metric.

Idempotency in REST APIs Is a Different Problem

Worth separating out, because the same word covers two different conversations and the search results blur them together.

In HTTP, idempotency is a property of the method. GET, PUT and DELETE are defined as idempotent, meaning repeating an identical request has the same effect on the server as making it once, while POST is not. That is a specification-level statement about method semantics, and it is why payment APIs that must accept repeated POST requests add an explicit idempotency key header instead of relying on the method to protect them.

The idempotency in this post sits one layer down, in the consumer that does the work after the request is accepted. An API can honour an idempotency key header perfectly and still send two emails if the thing performing the side effect was never protected. Getting the API contract right is a separate piece of work with failure modes of its own.

When One Operation Spans Several Systems: The Outbox Pattern

Everything above assumes the side effect lands in one place. Real workflows rarely do.

A single event might write a row to the database, publish a message to a queue, send an email through a third-party provider, and call an external API. Four systems, no shared transaction. Claim the event atomically, complete two of the four steps, and crash. The claim says the event is in progress. It does not say which half of the work happened.

Do not ask whether the request ran before. Ask which effects have already happened, and which are safe to repeat.

In practice that means tracking progress per effect rather than per event, so a retry can skip the email that already went out and still make the API call that did not. It also means a step that cannot be made idempotent by its own provider has to be made idempotent by you, usually by recording that you performed it before you perform the next one.

This is the point at which teams reach for the transactional outbox pattern, which makes the database write and the message publish part of one commit so the two cannot disagree. It solves a real version of this problem, and it is easier to adopt before you need it than during the incident that makes you want it.

What Idempotent Design Gets You in Production

None of the pieces here are clever. A key that means the right thing in your domain. A claim only one worker can win. A status that separates started from finished. A timeout that rescues the ones that started and never finished. A metric that speaks up when any of it stops working. The difficulty is in choosing them for a specific integration, which is why most of this post is about decisions rather than code.

What you end up with is not an exactly-once guarantee. It is one correct outcome, which is what anyone wanted from the guarantee in the first place.

If your team is dealing with duplicate processing, webhook reliability, or queue consumers that were never designed to run more than once, that is the work our backend engineering team does.

I would rather hear about the cases this does not cover than the ones it does, so if you have hit a duplicate-event problem of your own, or you think I have got something wrong here, connect with me on LinkedIn.

Frequently Asked Questions

What is idempotency?

An operation is idempotent when performing it multiple times produces the same result as performing it once. In event-driven systems it means using a stable key to recognise an event you have already handled, so that a duplicate delivery does not produce a duplicate side effect such as a second charge or a second email.

Can you give a real-life example of idempotency?

Marking an order as shipped is idempotent: process that event five times and the order is shipped, exactly as one run would leave it. Sending the customer the shipping email is not, because five runs mean five emails and none of them can be recalled. The same event often contains both, which is why protection belongs on the side effect rather than on the event. The "What Is Idempotency in Distributed Systems" section sorts common operations into those two groups.

What is an idempotency key?

The value your system uses to decide that two arrivals are the same piece of work. A provider-generated event ID is the easiest case. Where none exists, teams use an order identifier combined with a status, or a hash of the payload. The "How to Choose an Idempotency Key" section covers why a key that is too broad is more dangerous than one that is too narrow.

Why is checking whether an event was already processed not enough?

Because checking and recording are two separate steps, and two consumers can both complete the check before either records anything. Both then decide the event is new. The fix is to claim the event in a single atomic operation, using a unique constraint or a conditional write, so only one consumer can win.

What happens if a consumer crashes after claiming an event?

The claim record says the event is being handled, but no work was completed, so a redelivery gets skipped and the work is silently lost. This is why the record should track a state such as PROCESSING, COMPLETED or FAILED rather than mere existence, with a timeout that releases a claim left in progress too long.

Is idempotency the same as exactly-once delivery?

No. Exactly-once delivery is a guarantee that a message reaches a consumer exactly one time, and it is not achievable end to end across a network. Idempotency accepts that the message may arrive more than once and makes the repeat harmless, which produces one correct outcome without the guarantee.

How do you ensure idempotency in a payment API?

The key has to be attached to the operation before any money moves, and the claim on it has to be taken before the charge, not after. Where the caller drives the request, that key usually arrives as an idempotency key header. Where the work happens asynchronously behind a queue, it is derived from the event itself. Either way the record has to be written before the charge and kept long enough that a late retry still finds it.