Idempotent integrations: retries, duplicate events and failed handoffs.
Most integration failures are not dramatic. An event arrives twice, a request times out after the destination has already accepted it, or a record is rejected and waits in a log nobody reads. An integration that stays correct under those conditions is designed around them from the start: every operation can be repeated safely, failures are classified, and each record leaves a trace.
Published
Assume every event can arrive more than once
Queues, webhooks and polling jobs commonly deliver at least once. Retries on the sending side, redelivery after a consumer crashes and manual replays all produce duplicates. Exactly-once delivery across two independent systems is not something to rely on, so the integration has to make repeated processing harmless instead.
That property is idempotency: processing the same event twice leaves the destination in the same state as processing it once.
Give each operation an idempotency key
Derive a key from something stable in the source, such as the source system's event identifier, or the record identifier combined with its version. Before writing, check whether that key has already been applied; after a successful write, record it. Where the destination supports idempotency keys or conditional writes, use them, because they close the gap between checking and writing.
- Prefer upserts on a stable business identifier over blind inserts.
- Keep processed keys for at least as long as duplicates can plausibly arrive, including manual replays.
- Do not derive the key from delivery metadata, such as a message identifier that changes on redelivery.
Treat a timeout as an unknown outcome
A timeout does not mean the request failed. The destination may have committed the change and lost the response. Retrying without idempotency is how duplicate orders and double updates happen. With a key in place the retry is safe; without one, the integration should look the record up in the destination before trying again.
Separate transient failures from records that need correction
Retrying is right for failures that time will fix and wrong for failures it will not. Rate limiting, temporary unavailability and network errors belong in the first group. A missing required field, an unknown identifier or a validation error from the destination belongs in the second, and retrying it only delays the moment someone sees it.
- Retry transient failures with exponential backoff and jitter, and cap both the number of attempts and the total time.
- Respect rate-limit responses and any retry interval the destination provides.
- Send permanent failures to an exception queue with the reason, a reference to the payload and an owner.
- Make replay from the exception queue a normal, documented action that passes through the same idempotency check.
Decide how late and conflicting updates are handled
When updates can arrive out of order, compare versions or source timestamps before writing, so an older change cannot overwrite a newer one. When two systems can change the same field, the rule for which one wins is a business decision about ownership. Write it down per field; an integration that picks a winner silently is creating policy nobody agreed to.
Keep a trace per record
Give each record a correlation identifier that follows it through every attempt. Record when it was received, each attempt and its result, and the final outcome. That trace answers the questions people actually ask: did this record arrive, where did it stop, and was it changed twice. For regulated data it is also the audit trail.
Alert on the age of the oldest unprocessed item and on the growth of the exception queue, not only on error counts. A queue that is quietly getting older is the failure that goes unnoticed longest.
Test the failure cases on purpose
Normal transfers prove little about an integration. The cases below are the ones that decide whether it can be trusted, and each can be reproduced in a test environment.
- Deliver the same event twice and confirm a single result.
- Simulate a timeout after the destination has committed the write, then retry.
- Send records the destination rejects and confirm they reach the exception queue with a usable reason.
- Deliver an older update after a newer one and confirm it does not overwrite.
- Correct a rejected record and replay it from the exception queue.
Where Landvex fits
Our system integration work builds these properties in from the first handoff and tests the failure cases before the scope grows. Read how we scope and deliver system integration.
Bring one concrete workflow
Describe the work as it happens today: who does it, how often, and what breaks. You will hear back from a founder.