Sharding a Pre-Order Engine: Why One Big Group Doesn't Work

Architecture · Webhooks · Sharding

A webhook that tells you what is still there, instead of what changed, quietly turns every edit into a full-catalog diff. This is the architecture that came out of fixing that — and the race condition that made me reorder every write in the system.

What pre-order actually is

The ask is one sentence: when a product sells out, offer a pre-order instead of “sold out”, and go back to normal when it restocks. The catalog lives in a commerce platform we integrate with, not in our database, so “pre-order” is not a column we own. It is three separate pieces of remote state that all have to agree.

That last line matters more than it looks. Group membership is product-level and durable — set once when the merchant enrolls, untouched by every subsequent flip. The inventory policy and the metafield are variant-level and dynamic. Conflating those two layers is what let an early version flip non-enrolled sibling variants into pre-order alongside their enrolled siblings.

wantPreOrder = inventoryTracked && availableQty <= threshold

inventoryTracked is not defensive noise — a variant with untracked inventory has no meaningful stock level, so it must never flip at all.

Two systems, one contract

Three things move these flags and only one of them is us: our own writes, the platform's order and inventory events, and the merchant editing a product by hand in the platform's admin. That third one is fatal to the obvious design. Apply the flags at request time and call it done, and you are correct until the first time someone changes something without going through you — which is immediately.

So: two services, deployed to two different clouds, that never call each other. They coordinate through one shared store.

The system is one reconciler between desired state (ours) and observed state (theirs). Neither side trusts the other's view of the platform.

The fan-out is not decoration. A product can carry dozens of variants and each repair is two or three Admin API calls. Do that inline in a webhook handler and you blow the platform's delivery timeout, it retries, and a non-idempotent handler multiplies the damage. So the handler classifies the event, enqueues one task per variant, and returns immediately.

The webhook that broke one big group

Enrollment started as one selling-plan group per store, holding every enrolled product. Simple, and it worked until we subscribed to the group-update webhook.

We needed that subscription: a merchant can detach a product from the group by hand in the platform's admin, and we have to notice and repair it. But the payload has a shape that changes everything.

The group-update webhook tells you what is still in the group. It does not tell you what was removed.

So there is no delta to read. To find out what changed, you diff the payload's membership list against your own record of that group's membership. With one group holding the whole catalog, that means diffing fifteen thousand product IDs against fourteen thousand nine hundred and ninety-nine to discover the single product that left — on every group mutation, including the ones we caused ourselves.

The cost is not just CPU on a list comparison. Reconciling a 15,000-product group means paginated reads back from the platform to establish what membership actually is now, against a rate limit, inside a webhook handler with a delivery deadline. On a large store that reconcile timed out — which means the repair never ran, which means a merchant's hand-detached product silently stayed broken.

Sharding fixes the shape of the problem rather than the constant. Cap membership at fifty products per group and the diff is bounded by the shard, not the catalog. A group-update webhook now carries at most fifty IDs, and only the shard that was touched gets re-checked. Per-webhook work stops being a function of how big the merchant is.

Fifty is also roughly what the platform's rate-limit bucket absorbs in one batch, so enrollment paces itself structurally instead of leaning on retry-and-backoff. A shard whose remote group was deleted out from under us is self-healed — forgotten, with a fresh shard opened in its place.

The state that makes all of this navigable is small:

stores/<shop>/
├── config/
│   ├── allProductsEnabled          false      ← whole-store mode
│   ├── openShardId                 <group>    ← the shard accepting writes
│   ├── threshold                   0          ← flip at or below this qty
│   └── enrolledProducts/<product>/
│       ├── allVariantsEnabled      true
│       └── enrolledVariants/<id>   true       ← variant-level opt-in
├── productGroups/<product>         <group>    ← reverse index
├── sellingPlanGroups/<group>/
│   ├── count                       31
│   └── products/<product>          true
└── pendingGroupDeletes/<group>     true       ← two-phase teardown

One naming lesson worth passing on: that pointer was originally called sellingPlanGroupId, which reads like the group for the store rather than the shard currently accepting writes. More than one bug came from code that believed the name over the behaviour. It is openShardId now.

The race worth the whole post

Removing a product from the dashboard appeared to work perfectly. The platform confirmed the removal, all ten variants flipped to DENY, the metafield cleared. Then the product came back — re-attached to its group, every variant pre-order again.

When a write to an external system will echo back to you as an event, update your own source of truth first. Any convergence loop that reads state the write path hasn't finished updating will cheerfully undo it.

Why a realtime document store and not Postgres

Fair question, and the honest answer starts with the boring part: it was already provisioned, so it cost no infrastructure lead time. That is a real engineering reason and pretending otherwise would be revisionism. It also happened to fit, for three reasons that held up:

1. The two services live in different clouds. A shared relational database means one of them reaches cross-cloud into the other's network — peering or a public endpoint, credential distribution, an egress path to own and monitor. A store that is an authenticated HTTPS endpoint is equally close to both. For a database whose entire job is to be shared, that is an architectural property, not a convenience.
2. The request pattern is hostile to a connection pool. The convergence side is a stateless function scaling horizontally, plus a queue worker invoked once per variant: many short-lived instances, each wanting a connection, none living long enough to amortise it. The usual fix is another moving part — a pooler or a data proxy. There is no pool to exhaust here.
3. Every read is a known path, and the shape is a tree. Look at the state above: no joins, no aggregates, no reporting. Nested and sparse. In Postgres that is either a jsonb column — a document store with extra steps — or four tables and a migration for a schema that changed three times during the build. The one guarantee I actually needed was an atomic transaction on a single path, for the shard slot claim. That I got.

The cost is real and worth naming. There is no ad-hoc queryability: “which stores have shards over forty products” is a scan in application code. More pointedly, there is no way to express enrolled in one map XOR present in the other as a database constraint — and that is precisely the invariant the removal race violated. In Postgres it is a CHECK and the bug is unrepresentable. That is the trade I made. If enrollment ever needs reporting, it moves.

What generalizes

• Read the payload shape before you design the handler. An event that reports current state rather than a delta pushes the diff onto you — and the cost of that diff scales with whatever you let one partition hold.
• Shard to bound per-event work, not to fit data. The row count was never the problem. The reconcile that every event triggers was.
• Separate intent from convergence. A write path that declares desired state plus a loop that repairs observed state stays correct under edits you don't mediate.
• Order your writes by who is listening. If a side effect echoes back as an event, the source of truth has to land first.
• Two places to clear is a bug waiting to happen. The removal race was only possible because enrollment lived in two maps and unenrolling meant clearing both.