fix(square): do not resolve a record that belongs to another client

Two clients on one Square location double a day's tender in the window
between deploying and finishing the migration. Reproduced end to end:

  1. charge X carries the legacy key square/charge/P and belongs to
     client B's order
  2. client A's payout import resolves X through existing-id's legacy
     fallback and renames it into A's scope
  3. client B's next order import matches neither scheme, so it mints a
     second charge
  4. :sales-order/charges is cardinality-many and orders transact as
     plain maps, so nothing retracts the first

B's order ends up holding two charges for one payment — $200 of tender
for a $100 payment — and running the migration on that state produces
square/charge/BBB-LB-AAA-LA-P, the same double-scoped shape that already
doubled tender on five clients once during this work.

existing-id's legacy branch now declines any record already owned by a
different client, reading the owner attribute and, for charges that
predate :charge/client, the client of the referencing order. Declining is
also correct on its merits: the write then lands on this client's own
copy, which is what the scoped keys exist to create. The payout path also
writes :charge/client/:charge/location alongside the key, so a charge's
scope and its owner can no longer disagree.

The guard is transitional and gets deleted with the legacy branch it
protects, at rollout step 9.

Rollout resequenced for the decision to leave duplicate client records
active: no deactivation, no "which record survives" call, and the risk
window closed by pausing the importer across deploy + migrate rather than
by removing one of the two writers. Both records converge to independent
stable histories once every key carries its owner.

Also from review:

- migrate-all! now collision-checks the charge pass like the other three
  attributes instead of discovering a clash mid-run over 17M rows
- split-and-rekey-charges! logs progress every 200 batches; an
  interrupted 19M-order run left no trail
- unscoped-report's docstring no longer promises a zero its :no-owner
  column cannot reach; plan is named as the authoritative signal
- the rollout's pre-flight asked for a :collisions key plan never
  returns, so it silently passed on every database
- the multi-parent gate sampled (take 400000 (all-order-ids db)), which
  streams :aevt — ascending entity id — and so read the OLDEST 2% of
  orders: 2019-12-31 to 2021-06-03, before any of the contention it
  looks for. Now every order of the last year via the client+date index,
  5,159,787 on the restored copy, reading 0
- the report claimed same-client pairs get copied once batches split
  them. They do not, at any batch size; verified at batch-size 1 and now
  pinned by a test

30 tests, 72 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-16 08:09:53 -07:00
parent 57a84dae11
commit 10d0d01b82
6 changed files with 375 additions and 180 deletions

View File

@@ -230,6 +230,17 @@
shifts</strong> — the same figures as at the restore point. Refunds went from 50,986 to 51,986,
and all 1,000 of those came from the live Square import run afterwards, not from the renaming.
Had the fallback lookup been missing, each of these would have doubled instead.</p>
<p><strong>The fallback also has to refuse.</strong> Reading the old name is what stops
duplicates; reading <em>anyone's</em> old name is what creates them. Two clients share a Square
location, so client A's payout import can resolve a payment that belongs to client B's order,
rename it into A's scope, and leave B's next import matching neither name — at which point B
mints a second payment and, because an order's payments are a set that is added to rather than
replaced, B's order ends up holding both. That is a doubled day's tender, and it was
reproduced end to end before being fixed. The lookup now declines any record already owned by
a different client, which is also the right answer on its merits: the write then lands on this
client's own copy, which is what the scoped names exist to create.</p>
<p>This is transitional. Once no legacy names remain, the fallback and the refusal are deleted
together and the guarantee stops depending on either.</p>
</div>
<h3>3 · Give every order its own payment record</h3>
@@ -250,7 +261,7 @@
<div class="measure">
<p>Run over the whole database that was <strong>9,100,314 renamed and 200,027 copied</strong>,
and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across
400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it
every order of the last year. The record count rose by about 200,027 — the number of copies it
reported making, which is the check that it created what it meant to and nothing else.</p>
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
@@ -454,10 +465,9 @@
<table>
<thead><tr><th>Step</th><th>Result</th></tr></thead>
<tbody>
<tr><td>Deactivate the duplicate client at each shared location</td><td class="n">10 locations · shared locations remaining: <span class="good">0</span></td></tr>
<tr><td>Walk every order in the database</td><td class="n">19,040,785 orders</td></tr>
<tr><td>Give every order its own payment record</td><td class="n">9,100,314 re-keyed · 200,027 copied</td></tr>
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across 400,000 orders checked</span></td></tr>
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across every order of the last year — 5,159,787</span></td></tr>
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
@@ -465,7 +475,8 @@
</table>
</div>
<div class="measure">
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.</p>
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, close to the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)</p>
<p><strong>The measurement above was taken with the duplicate client records deactivated, and that is not how this will be deployed.</strong> Leaving both records live is the intended configuration — the re-key is what separates them — but it means the deactivation that made this measurement clean will not be there. The gap that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why <code>existing-id</code> now refuses to resolve a record belonging to another client.</p>
</div>
<div class="callout">
@@ -568,15 +579,20 @@
<p>The problem this work exists to solve is gone: no payment answers to two orders, so the
component relationship means what it says and deleting an order can no longer take another
order's money with it.</p>
<p><strong>One behaviour changed when the pass was run over everything, and it is worth
recording.</strong> Where Square splits one tender across two of a single client's own orders,
the earlier design left the payment shared on purpose — both orders compute the same name, so
there is no second name for a copy to take. That rule only holds for two orders processed in
the same batch. Run across nineteen million orders in batches of two thousand, such pairs
almost always fall in different batches and the second order now takes a copy. Inside the
ninety-day window this changed nothing measurable: the recompute after the database-wide pass
matched the one before it to the cent. Outside the window it has not been measured, and it
should be before this runs against production.</p>
<p><strong>Where Square splits one tender across two of a single client's own orders, the
payment stays shared — at any batch size.</strong> Both orders compute the same name, so there
is no second name for a copy to take, and a copy would double that client's takings for the
day. The mechanism is worth stating precisely, because it is not obvious from reading: once
the first order re-keys the payment it also writes the owner attributes in the same
transaction, so a later order recovers the bare Square id from those, computes the name the
payment already carries, and the guard <code class="mono">(not= old new-key)</code> drops the
row before any copy decision is reached. Verified by running the migration at a batch size of
one, which forces the two orders into separate batches: no copy is made.</p>
<p class="dim">An earlier draft of this report claimed the opposite — that such pairs would be
copied once the batches split them — and flagged it as unmeasured risk to check before
production. That was wrong, and it is recorded here rather than quietly deleted because it did
real damage: an independent reviewer cited this paragraph as evidence and raised a defect that
does not exist. A test now pins the behaviour at batch size one.</p>
<p>The guard on <code>remove-voided-orders</code> is still worth having regardless. It is
cheap, and it makes the safety a property of the deletion rather than of the migration having
been run first.</p>