Three Flags and a Race: Building a Convergence Engine on an Eventually-Consistent API

Distributed systems · Webhooks · Idempotency

A pre-order feature sounds like a checkbox. It is actually a distributed state machine that has to agree with a third-party system you do not control, over a network that lies to you, driven by events that arrive out of order and sometimes twice.

The problem

The user-facing ask is one sentence: when a product sells out, let customers pre-order it instead of seeing "sold out."

The implementation ask is very different. A merchant's catalog lives in a commerce platform we integrate with, not in our database. We can read it and mutate it over an Admin API, and the platform pushes us webhooks when things change. To put a single product variant into "pre-order," three separate pieces of remote state have to be true at once:

Flag	Pre-order	Buy now
Inventory policy	`CONTINUE` (oversell allowed)	`DENY`
Variant metafield	`"true"`	`"null"` / absent
Selling-plan group	attached	detached

Each flag is set by a different API mutation. There is no transaction across them. So the real state space isn't two states — it's eight, six of which are corrupt:

CONTINUE + metafield true + no selling plan → the store oversells with no pre-order terms attached. Money taken, nothing promised.
DENY + selling plan attached → storefront advertises pre-order, checkout refuses the sale.
metafield true + DENY → the pre-order banner renders on a variant nobody can buy.

Any of these is a revenue bug visible to a real shopper. And the flags drift constantly, because merchants edit inventory by hand, bulk-import CSVs, and delete things from the platform admin while our system is mid-write.

So the design goal stopped being "set three flags" and became: guarantee convergence. Whatever state the remote system is in, however it got there, the system must drive it back to one of the two legal states — and must be safe to run again, and again, on the same input.

The shape of the solution

Nothing here is a CRUD endpoint. The whole system is one reconciliation loop, deployed as three cooperating pieces:

A control plane (the merchant-facing service): enrollment, bulk enable/disable, teardown, and the config a merchant sees in the dashboard.
A convergence worker (a serverless function behind a task queue): consumes platform webhooks, computes desired state per variant, and mutates only what diverges.
A checkout guard (a platform-native function running at checkout time): the last line of defense, rejecting carts whose flags contradict each other.

The core of the worker is deliberately boring:

const desired = state.tracked && state.available <= threshold
  ? PRE_ORDER
  : BUY_NOW;

if (describe(state) === desired) return "nothing to change";
await applyState(variant, desired);

Two lines of policy, one guard, one write. Every hard-won lesson below is a story about why that if exists, or why state is harder to obtain than it looks.

Lesson 1 — Your own writes come back as events

The worker reacts to products/update webhooks. The worker also mutates products. You can see where this goes.

A single stock change fans out one webhook. That webhook carries the entire variants array, so we enqueue one task per enrolled variant. Each task writes flags. Each write makes the platform emit another products/update. Which fans out again.

1 event → N tasks → N webhooks → N² tasks → ...

The state stayed correct the whole time — the toggle is idempotent, so re-running it changes nothing. It just never stopped. And it burned the merchant's entire API rate-limit budget doing nothing.

The fix is a loop breaker: before enqueuing work, compare the flag already present in the webhook payload against the one the stock level implies. If they match, this event is almost certainly the echo of our own write — drop it.

const impliedPolicy = qty > threshold ? "DENY" : "CONTINUE";
if (payloadVariant.inventory_policy === impliedPolicy) {
  return skip("already converged");
}

With that in place, every repair produces exactly one follow-up event, and that event terminates in "nothing to change." The invariant we actually test for is: never more than two rounds.

Two things worth noting, because they're the interesting part:

The create topic is exempt from the guard. It fires once, and we never create products, so an echo is impossible — the guard could only cause a miss.
The guard is deliberately incomplete. Webhook payloads carry the inventory policy but not metafields. So a variant a human manually flipped to CONTINUE without our metafield looks "converged" and gets skipped. We accepted that: it self-heals on the next real stock transition. Choosing a known, bounded blind spot over an unbounded loop is a trade, and writing the trade down is part of shipping it.

Lesson 2 — Read-after-write is a lie

This one cost the most time and produced the most durable lesson.

Symptom: occasionally a variant would latch into a broken state and stay there. The worker's logs showed it reading the selling plan as attached, concluding nothing needed fixing, and exiting — while the platform admin clearly showed it detached.

The worker wasn't wrong. It was reading inside the platform's eventual-consistency window. A mutation had landed, the webhook for that mutation had already been delivered and processed, and the read API was still serving pre-mutation data. Fast webhook delivery plus lagging read replicas is a race, and the worker was losing it — then persisting its wrong conclusion as "converged."

The fix was a confirming re-read: read state, and if it implies "nothing to change," back off ~1.5s and read again before believing it. Cheap, and it only pays the cost on the path that was about to do nothing anyway.

But the fix isn't the lesson. The lesson is that I refused to ship it unverified. A race that reproduces "occasionally in production" is not a fixed bug, it's a coin flip you stopped watching. So I built a harness that reproduces it deterministically:

Force the variant into a fully converged pre-order state, then wait for it to settle.
Pre-compute the webhook payload and its HMAC signature — before touching anything.
Fire the detach mutation, and immediately POST the pre-signed webhook with zero API round trips in between.
Wait out the worker's full read → backoff → re-read → repair cycle.
Assert all three flags, and exit non-zero if still broken.

Step 2 is the whole trick. Signing the payload after the detach would add a network round trip and let the read replicas catch up — the race would quietly stop reproducing and I'd have concluded I'd fixed it. Pre-signing collapses the gap to microseconds and lands the worker's first read squarely inside the consistency window, every single run.

A race condition you cannot reproduce on demand is not understood. Building the repro is not overhead before the fix; it is the fix, because it's the only thing that can tell you the fix worked.

Lesson 3 — Ask whether the input means anything

A subtler bug. Some variants have inventory tracking turned off — the platform treats them as infinitely purchasable, and their quantity field is just 0, meaningless.

We read 0 as "out of stock" and dutifully put them into pre-order. That state was unrecoverable: the control plane's backfill scan filters on tracks-inventory, so it would never re-enroll them. The variant sat at CONTINUE + metafield true with no selling plan forever — the exact "oversell with no terms" corruption from the top of this post.

One condition:

const wantPreOrder = state.tracked && state.available <= threshold;

I found it by deleting a selling-plan group and watching which products failed to come back — which is a technique worth keeping. Destroy state and watch what fails to self-heal. Convergent systems hide their gaps beautifully during normal operation; the gaps only show when you force a rebuild.

The generalizable point: 0 and "unknown" are not the same value, and a numeric field is only meaningful when the flag that gives it meaning is also true. Read the flag.

Lesson 4 — Events are duplicated, so make the work claimable

Two independent problems, one shape.

Duplicate topics. A stock change from 1 to 0 emits two different webhook topics roughly 300ms apart, both describing the same transition. Handling both means double work; handling one means missing cases the other covers. We eventually collapsed to the product-level topics and deleted the variant-level subscriptions entirely — but only after proving with logs that the variant topics were being correctly ignored, then removing them from the store. Verify the code path is dead before you delete the thing feeding it.

Duplicate group reconciliation. Group-level webhooks arrive in bursts, each triggering a full diff of the group. The guard is a compare-and-swap on a timestamp in the shared datastore, inside a transaction, with a 60-second debounce window:

// only one worker instance wins the claim per window
const claimed = await txn(`${store}/reconcileAt`, prev =>
  (now - (prev ?? 0) < 60_000) ? ABORT : now
);
if (!claimed) return "another instance owns this window";

Serverless workers scale to N instances with no shared memory, so "have we already done this?" has to be answered by an atomic operation in the datastore. A transaction on a timestamp is the smallest thing that works.

What reconciliation actually does

Given a group webhook, diff our record of which variants should be on the plan against which variants actually are, then act on the difference:

In our list, missing from the plan → re-attach it.
On the plan, missing from our list → detach it.
In our list, no longer exists in the platform at all → drop it from our list.

That third case is the one people forget. Merchants delete products. Without a stale-entry sweep, your source of truth accumulates ghosts and every reconciliation pass wastes API calls resurrecting things that are gone.

Lesson 5 — Shard to bound the blast radius, not to fit a row count

The system originally put every pre-order product into one selling-plan group. That works fine for a store with 40 products. I load-tested it against a store I seeded with 20,000 products and 200,000 variants (roughly 100,000 of them out of stock), and the single-group model failed in two ways that had nothing to do with storage:

Every group event became a whole-catalog event. When the platform emits "this selling-plan group changed," the correct response is to reconcile the group's members. With one group, that meant every product in the store — one webhook, tens of thousands of products to verify.
Finding a product's group meant asking the platform. To answer "is this product already enrolled?", the old code fetched the group's product list over the API — a paginated read, on the hot webhook path, against a rate limit, inside the consistency window from Lesson 2.

How the sharding actually works

The platform allows many groups to share the same name, so the fix was to stop having one group and start having as many as needed. The allocation rule is fill-and-roll, not hash-based:

Look up the currently open group and its member count.
If count < LIMIT, it has room: add products until it's full.
When it fills, create a new group, mark it open, and chunk the remaining products into it — repeating until the backlog is empty.

Two maps in the datastore make that possible without ever asking the platform:

sellingPlanGroups/{groupId} → { count, products: { [id]: true } }
productGroups/{productId}   → groupId

The forward map answers "which group has room?" — that's the allocator's input. The reverse map answers "which group owns this product?" in a single indexed lookup, replacing the paginated API scan entirely.

Why cap the group at all

Worth being precise here, because it's the part that's easy to get wrong in the retelling: the per-group cap is a self-imposed number, not a documented platform limit. I went looking for a hard cap and the docs don't publish one. The cap exists for two engineering reasons instead:

It bounds reconciliation work. A group event now means "verify at most N products," regardless of catalog size. Worst-case work per webhook becomes a constant you choose, rather than a function of how big the merchant is.
It aligns with the rate-limit budget. The API throttles on a cost-point bucket that refills at a fixed rate. Sizing a shard to a batch the bucket can absorb means enrollment paces itself structurally instead of relying on retry-and-backoff to absorb overload. The enrollment batch size was retied to the same constant so a database batch and a full group are the same unit of work.

And the quiet win: fewer API calls on the hot path means fewer opportunities to read stale data. The sharding work shrank the surface area of the Lesson 2 race without setting out to.

An aside on trusting count APIs at scale

While verifying that 20,000-product seed, the platform's productsCount field confidently returned 10,000. Not an error — a silently ceilinged value, with a precision hint most callers never read. I only caught it because I verified by paginating the full collection and counting distinct items instead of trusting the aggregate.

If a number is load-bearing for a scaling decision, count the rows. Aggregate endpoints are allowed to approximate, and they rarely tell you they did.

Lesson 6 — Know which endpoint is polled

A quick one with an outsized payoff. The dashboard polled a config endpoint every 5 seconds while a merchant watched a bulk-enrollment progress bar. That endpoint had accreted two live third-party API calls for diagnostics nobody was reading 12 times a minute.

Split it in two:

The polled endpoint: two local reads, zero external calls, returns only the progress fields the spinner needs.
A separate diagnostics endpoint, fetched once: the expensive cross-checks, plus a computed inSync boolean comparing our record against live platform state.

An endpoint's cost is its latency multiplied by its call frequency, and frequency is set by a caller you may not be looking at. Also: shipping a drift-detection boolean to the dashboard meant support could answer "is this store healthy?" without a single engineer touching a log.

Lesson 7 — Test against the real thing, with real mutations

Hand-firing webhooks at your own service tests your handler. It does not test your system, because you've replaced the two hardest parts — what the platform actually sends, and when.

The suite that finally gave me confidence ran real mutations against a real test store, then asserted on real state:

Perform an actual mutation (place an order, edit inventory, delete a group, inject drift).
Wait out the convergence cycle.
Read the after-state back from the API and assert all three flags.
Pull the matching log window and assert on the path taken — how many rounds, which branch, and that it ended in "nothing to change."

Asserting on the log trace, not just the final state, is what catches the amplification loop. A system stuck in an infinite idempotent repair cycle has perfect final state and is completely broken.

The suite also became the place to write down expected weirdness. Group membership is product-level, so a variant that isn't enrolled legitimately reports "attached" when a sibling variant is. An absent metafield is indistinguishable from the literal string "null". Overselling to a negative quantity correctly stays in pre-order. Each of those looks exactly like a bug in a test report, and each of them is correct. Encoding "known non-bugs" in the verdict logic is how a suite stays trustworthy instead of becoming noise everyone learns to ignore.

What generalizes

Strip out the commerce vocabulary and this is a pattern you meet constantly — anything that syncs state into a system you don't own: a payment provider, a CRM, a cloud API, a search index.

Model desired state, not transitions. Compute what should be true, diff against reality, write only the difference. Transition-based logic breaks the moment an event is lost or replayed; convergent logic doesn't care.
Assume every event is a duplicate. Idempotency keeps state correct but doesn't keep work bounded. You need a loop breaker and an atomic claim.
Your own writes will page you. Any system that both reacts to and produces events on the same channel needs an echo guard before it needs anything else.
Read-after-write across a service boundary is not guaranteed. If a stale read causes you to persist "nothing to do," confirm with a second read before believing it.
A field is only meaningful if the flag enabling it is set. Zero is not the same as unknown.
Reconcile in three directions — missing, extra, and no-longer-exists. Everyone implements the first two.
Shard to bound per-event work, not just to fit data. Pick a partition size your rate limit and your worst-case reconciliation can both absorb, and keep a reverse index so lookups never leave your own datastore.
Don't trust aggregate count APIs for scaling decisions. They're allowed to approximate or ceiling silently. Enumerate and count.
Build the deterministic repro. For races, it's not preparation for the fix. It's the only proof the fix exists.

The feature is a checkbox in a dashboard. Behind it is a convergence engine, an echo guard, a distributed claim, a shard allocator, and a test suite that pokes a real store and reads the logs back. That gap — between how simple the requirement sounds and what correctness actually costs — is most of what the job is.