Retrying a payment is easy. Retrying it safely is a storage problem wearing the costume of a network problem. Every failed call you retry is a chance to charge someone twice, and the failure is almost never in the happy path you tested.
The failure everyone ships first
A producer sends a payment event, the consumer charges a card, then the consumer crashes before it commits its offset. Kafka redelivers. You charge the card again. Nobody noticed in staging because staging doesn’t fail at the exact wrong moment.
The instinct is to add a try/catch and call it done. That doesn’t help: the
problem isn’t that the request failed, it’s that you can’t tell a new request
from a replayed one.
Idempotency is a storage problem
The fix is to give every operation a key the caller controls, and to record keys you’ve already completed. The record is the contract:
create table processed_operations (
idempotency_key text primary key,
operation_type text not null,
result jsonb not null,
created_at timestamptz not null default now()
);The primary key does the concurrency control for you. Two workers racing on the same key can’t both insert — one wins and returns the stored result, the other reads it. No locks, no coordination service, no distributed mutex to babysit.
A few rules that keep it honest:
- The key must come from the caller, not be generated inside the consumer.
- Insert the key in the same transaction as the side effect it guards.
- Store the result, so a retry can return the original answer verbatim.
- Never reuse a key for a different intent — that’s how you “successfully” skip a legitimate second charge.
The outbox closes the gap
Idempotent consumers still leave a gap on the write side: you commit a payment, then the process dies before publishing the event. The money moved; the downstream world never heard about it. The transactional outbox removes the gap by writing the event to the same database transaction as the state change.
@Transactional
public void capture(PaymentCommand cmd) {
var result = ledger.capture(cmd); // the side effect
outbox.enqueue("payment.captured", result); // same transaction
idempotency.remember(cmd.key(), result); // same transaction
}A separate relay polls the outbox and publishes to Kafka, marking rows sent once the broker acknowledges. Delivery becomes at least once, which is fine — your consumer is idempotent now, remember?
What actually holds up
- At-least-once everywhere, idempotent at the edges. Stop chasing exactly-once.
- Keys expire, but not too early. Retention must outlast your retry window.
- Watch for the replayed key with a different payload. That’s a bug upstream.
- Instrument the duplicates. A rising skip rate is a signal, not noise.
A retry that isn’t idempotent isn’t a retry. It’s a duplicate waiting for a quiet afternoon to surface.
None of this is clever. It’s a table, a key you trust, and the discipline to write the receipt before you move the money.