Bryce
1558851c18
Re-keying stops two client records on one Square location fighting over a
record, but it does not make their books equal. Sales orders have always
been keyed by client, so each record built its own order history from the
start. Refunds, payouts and cash-drawer shifts were not, so only ONE
record holds each of them — whichever imported it last. The migration
freezes that ownership rather than evening it out.
The record left without them shows returns from its own orders and no
refunds against them. NGBK held 158,535 orders and five refunds. On
2026-05-11 both NGBK and NGBR held the same 221 orders; NGBK had no
refunds, NGBR had two worth $2,232.29, and NGBK was out by $2,232.29 to
the cent.
Rather than manufacture copies, ask Square again. Client-scoped keys mean
each record now creates its own copy of whatever it reads, so replaying a
window converges the two histories with no code inventing a duplicate.
`backfill-history` does that for a date range across orders, payouts,
refunds and shifts. After it, all ten pairs held matching order and
refund counts.
Fixes a capped read found by doing this: the refunds import asked Square
for a location's refunds and read only the first page — no cursor, no
date range. Square pages at a hundred, so a location with more refunds
silently returned a hundred and the response looked complete. That is why
each twin held almost exactly 100 refunds and why an earlier import added
exactly 1,000 across ten locations. `refunds`/`upsert-refunds` now follow
the cursor and accept a window.
Re-measured from the same fresh restore, duplicates left active:
before 1,191 days out of balance / $70,276.50
after 122 days out of balance / $2,379.45 (32 material)
1,069 into balance, 0 out, 0 already-balanced days altered
The shared records went from 423 days / $18,508.39 to 3 days / $648.84 —
NGBK and NGBR at $299.42 each (the known tender-versus-order gap) and
NGDA at $50.00 (auto-gratuity as a service charge). The zero-regression
guarantee is restored too: the six days that broke without the backfill
were tips reversed on one record whose refund sat on the twin, and all
six closed once both sides had their own copy.
This beats retiring the duplicate records, which left 279 days and
$7,790.54, and it needs no decision about whose history to abandon.
Unchanged across every run: for the 190 clients that do not share a
location, 119 days and $1,730.61, same five restaurants.
Cost: 5.9 hours for ninety days across twenty records. Every Square call
shares one 25 req/s throttle, refunds and shifts cost one API call each,
and backfill-history imports three clients at a time. Reads are not the
limit — existing-id measures 32 microseconds.
31 tests, 76 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
761 lines
58 KiB
HTML
761 lines
58 KiB
HTML
<title>Ninety-Day Reconciliation</title>
|
||
<style>
|
||
:root {
|
||
--paper: #F6F8F7; --card: #FFFFFF; --ink: #141F1D; --ink-soft: #4A5C58;
|
||
--ink-faint: #7C8D89; --rule: #DCE4E1; --accent: #0E5B57; --accent-soft: #E3EFED;
|
||
--good: #1A6B49; --bad: #A03B26; --warn: #8A6410;
|
||
--shadow: 0 1px 2px rgba(20,31,29,.06), 0 8px 24px rgba(20,31,29,.05);
|
||
}
|
||
@media (prefers-color-scheme: dark) {
|
||
:root:not([data-theme="light"]) {
|
||
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
|
||
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
|
||
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
|
||
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
|
||
}
|
||
}
|
||
:root[data-theme="dark"] {
|
||
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
|
||
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
|
||
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
|
||
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
|
||
}
|
||
* { box-sizing: border-box; }
|
||
body { background: var(--paper); color: var(--ink);
|
||
font-family: system-ui, -apple-system, "Segoe UI", sans-serif;
|
||
font-size: 16px; line-height: 1.6; margin: 0; padding: 0 20px 96px; }
|
||
.wrap { max-width: 940px; margin: 0 auto; }
|
||
.measure { max-width: 66ch; }
|
||
.num { font-variant-numeric: tabular-nums; }
|
||
.mono { font-family: ui-monospace, "SF Mono", "Cascadia Code", monospace; font-variant-numeric: tabular-nums; }
|
||
|
||
header.masthead { padding: 72px 0 40px; border-bottom: 2px solid var(--ink); display: flex; flex-direction: column; gap: 14px; }
|
||
.eyebrow { font-size: 12px; letter-spacing: .14em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
|
||
h1 { font-family: Georgia, "Iowan Old Style", serif; font-size: clamp(34px, 5.4vw, 54px);
|
||
line-height: 1.08; font-weight: 600; letter-spacing: -.015em; margin: 0; text-wrap: balance; }
|
||
.standfirst { font-size: 19px; color: var(--ink-soft); margin: 0; max-width: 62ch; }
|
||
.meta { display: flex; flex-wrap: wrap; gap: 10px 28px; font-size: 13px; color: var(--ink-faint); padding-top: 6px; }
|
||
.meta b { color: var(--ink-soft); font-weight: 600; }
|
||
|
||
section { padding-top: 56px; display: flex; flex-direction: column; gap: 20px; }
|
||
h2 { font-family: Georgia, "Iowan Old Style", serif; font-size: 27px; font-weight: 600; letter-spacing: -.01em; margin: 0; text-wrap: balance; }
|
||
h3 { font-size: 14px; letter-spacing: .08em; text-transform: uppercase; color: var(--ink-soft); font-weight: 700; margin: 0; }
|
||
h4 { font-size: 18px; font-weight: 650; margin: 0; letter-spacing: -.01em; }
|
||
p { margin: 0; }
|
||
.measure p + p { margin-top: 14px; }
|
||
|
||
.ledger { display: grid; grid-template-columns: 1fr auto 1fr; border: 1px solid var(--rule);
|
||
border-radius: 4px; background: var(--card); box-shadow: var(--shadow); overflow: hidden; }
|
||
.ledger > div { padding: 26px 28px; display: flex; flex-direction: column; gap: 6px; }
|
||
.ledger .arrow { justify-content: center; align-items: center; border-left: 1px solid var(--rule);
|
||
border-right: 1px solid var(--rule); color: var(--ink-faint); font-size: 22px; background: var(--accent-soft); }
|
||
.side-label { font-size: 12px; letter-spacing: .12em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; }
|
||
.figure { font-size: clamp(28px, 4.4vw, 40px); font-weight: 650; line-height: 1.05; letter-spacing: -.02em; }
|
||
.figure.after { color: var(--good); }
|
||
.subfig { font-size: 14px; color: var(--ink-soft); }
|
||
|
||
.stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 14px; }
|
||
.stat { background: var(--card); border: 1px solid var(--rule); border-radius: 4px; padding: 18px 20px; display: flex; flex-direction: column; gap: 4px; }
|
||
.stat .k { font-size: 30px; font-weight: 650; letter-spacing: -.02em; line-height: 1; }
|
||
.stat .l { font-size: 13px; color: var(--ink-soft); }
|
||
.stat.zero .k { color: var(--good); }
|
||
|
||
.problem { border-left: 3px solid var(--accent); padding-left: 24px; display: flex; flex-direction: column; gap: 14px; }
|
||
.problem.two { border-left-color: var(--warn); }
|
||
.problem.three { border-left-color: var(--bad); }
|
||
|
||
.scroll { overflow-x: auto; border: 1px solid var(--rule); border-radius: 4px; background: var(--card); }
|
||
table { border-collapse: collapse; width: 100%; font-size: 14.5px; }
|
||
th, td { padding: 11px 16px; text-align: left; border-bottom: 1px solid var(--rule); white-space: nowrap; }
|
||
thead th { font-size: 11.5px; letter-spacing: .09em; text-transform: uppercase; color: var(--ink-faint); font-weight: 700; background: var(--accent-soft); }
|
||
tbody tr:last-child td { border-bottom: none; }
|
||
td.n, th.n { text-align: right; font-variant-numeric: tabular-nums; }
|
||
tr.total td { font-weight: 650; background: var(--accent-soft); }
|
||
.good { color: var(--good); font-weight: 650; }
|
||
.bad { color: var(--bad); font-weight: 650; }
|
||
.dim { color: var(--ink-faint); }
|
||
|
||
.callout { background: var(--card); border: 1px solid var(--rule); border-left: 3px solid var(--accent);
|
||
border-radius: 4px; padding: 20px 24px; display: flex; flex-direction: column; gap: 10px; }
|
||
.callout.warn { border-left-color: var(--warn); }
|
||
.callout .h { font-weight: 650; }
|
||
code { font-family: ui-monospace, "SF Mono", monospace; font-size: .9em; background: var(--accent-soft); padding: 1px 5px; border-radius: 3px; }
|
||
pre { margin: 0; padding: 20px; font-size: 13px; line-height: 1.7; white-space: pre; font-family: ui-monospace, "SF Mono", monospace; }
|
||
footer { margin-top: 72px; padding-top: 24px; border-top: 1px solid var(--rule); font-size: 13px; color: var(--ink-faint); display: flex; flex-direction: column; gap: 8px; }
|
||
ul { margin: 0; padding-left: 20px; display: flex; flex-direction: column; gap: 8px; }
|
||
.tech { font-size: 11px; letter-spacing: .08em; text-transform: uppercase; color: var(--accent);
|
||
font-weight: 700; border: 1px solid var(--accent); border-radius: 3px; padding: 2px 7px; display: inline-block; }
|
||
</style>
|
||
|
||
<div class="wrap">
|
||
|
||
<header class="masthead">
|
||
<div class="eyebrow">Sales summaries · measured on a restored production backup</div>
|
||
<h1>Ninety-Day Reconciliation</h1>
|
||
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — and a fourth, found late, that is a missing-data problem wearing a balancing problem's clothes. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
|
||
<div class="meta">
|
||
<span><b>Window</b> 2026-05-10 → 2026-08-07</span>
|
||
<span><b>Client-days</b> <span class="num">18,900</span></span>
|
||
<span><b>Clients</b> <span class="num">210</span></span>
|
||
<span><b>Nothing in production was changed</b></span>
|
||
</div>
|
||
</header>
|
||
|
||
<section>
|
||
<div class="ledger">
|
||
<div>
|
||
<span class="side-label">Today's calculation, ninety days re-run</span>
|
||
<span class="figure num">$70,276.50</span>
|
||
<span class="subfig"><span class="num">1,191</span> days out of balance · <span class="num">93.70%</span> clean</span>
|
||
</div>
|
||
<div class="arrow" aria-hidden="true">→</div>
|
||
<div>
|
||
<span class="side-label">The same ninety days, fixes on</span>
|
||
<span class="figure after num">$2,379.45</span>
|
||
<span class="subfig"><span class="num">122</span> days out of balance · <span class="num">99.35%</span> clean</span>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="stats">
|
||
<div class="stat"><span class="k num">1,069</span><span class="l">client-days brought into balance</span></div>
|
||
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
|
||
<div class="stat"><span class="k num">96.6%</span><span class="l">of the variance removed</span></div>
|
||
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
|
||
</div>
|
||
|
||
<div class="measure">
|
||
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p>
|
||
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>1,069 days and $67,897.05</strong>, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.</p>
|
||
<p><strong>Most of what is left is not a balancing fault at all</strong>, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>The four problems</h2>
|
||
|
||
<div class="problem">
|
||
<h4>1. Two client records sharing one Square location</h4>
|
||
<div class="measure">
|
||
<p><strong>For the business:</strong> ten restaurant locations were set up twice in the system, as two separate clients. Both were importing from Square. Because the two records competed for the same payments and refunds, a refund would belong to one client for twenty minutes, then the other — so a day's books could gain or lose a refund depending on nothing but timing. On 2026-07-23 one client's summary was missing a $71.94 refund entirely, and was out of balance by exactly that amount.</p>
|
||
<p><span class="tech">technical</span> Sales orders scoped their identifier by client (<code>square/order/<code>-<loc>-<id></code>), but refunds, card charges, payouts and cash-drawer shifts did not — they used the bare Square id. Those attributes are <code>:db.unique/identity</code>, so both clients' imports resolved to a single entity and the last writer won.</p>
|
||
<p>Reading ownership out of the database's own history, this had actually happened to <strong>3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts</strong>. And it has involved <strong>19 client pairs, of which only 10 are visible in today's configuration</strong> — nine more contended in the past and the configuration has since changed, so no point-in-time check would find them.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="problem two">
|
||
<h4>2. One payment record owned by two orders</h4>
|
||
<div class="measure">
|
||
<p><strong>For the business:</strong> the same collision meant a single card payment could be attached to both clients' copies of an order. That is worse than untidy. The nightly import removes orders Square reports as voided, and removing an order also removes its payments — so cancelling one client's order could silently delete the <em>other</em> client's payment, leaving a day showing sales with no money against them.</p>
|
||
<p><span class="tech">technical</span> <code>:sales-order/charges</code> is declared <code>:db/isComponent true</code>, so <code>[:db/retractEntity <order>]</code> cascades into the charges. In a 20,000-order sample of the affected clients, <strong>11,469 charges had two parent orders</strong>. This is why <code>remove-voided-orders</code> was left switched off during testing.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="problem three">
|
||
<h4>3. Tips refunded, and service charges credited nowhere</h4>
|
||
<div class="measure">
|
||
<p><strong>For the business:</strong> two arithmetic faults, both of which overstated or understated a day.</p>
|
||
<ul>
|
||
<li><strong>Refunded tips stayed on the books.</strong> When a guest was refunded, the tip came back too — but the summary still counted the original tip as income. On one NGLK day the books credited $482.94 of tips beside a $60.00 refund of that very tip.</li>
|
||
<li><strong>Service charges were collected but never earned.</strong> A catering or auto-gratuity charge is inside the card payment the customer makes, so it arrived as money taken — but no line recorded it as money earned. One NTPT order carried $427.10 that was credited to nothing at all; the largest single instance was <strong>$1,344.86</strong> in one day.</li>
|
||
</ul>
|
||
<p><span class="tech">technical</span> <code>get-tip</code> summed tips by joining through <code>:sales-order/charges</code>, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on <code>:sales-order/tip</code>. Nothing at all read <code>:sales-order/service-charge</code>.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="problem">
|
||
<h4>4. Refunds on records whose sales were never imported <span class="tech">not fixed — deliberately</span></h4>
|
||
<div class="measure">
|
||
<p><strong>This is why the duplicated restaurants looked so much worse than everyone else.</strong> Of the days still failing after the first three fixes, <strong>155 of the 423 on shared-location records had no sales orders at all</strong> — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.</p>
|
||
<p>The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded <em>any</em> order shows <strong>140 of 171 fall before that client had a single order in the system</strong> — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.</p>
|
||
<p><strong>Where the refunds came from.</strong> Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to <span class="mono">NGDG</span> that same day, and was taken over by <span class="mono">NGDU</span> on 12 August. Others flip between the two records several times a day across 12–15 August. <span class="mono">NGDU</span>'s first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.</p>
|
||
<p>Across the nine records, <strong>660 refunds worth $15,237.02 sit on a record dated before that record's first order.</strong> Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.</p>
|
||
<p><strong>Why it is deliberately left out of balance.</strong> The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and on this data it closes 16 of the 122 remaining days and $1,227.65. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.</p>
|
||
<p><span class="tech">technical</span> <code>get-returns</code> sums <code>:sales-order/returns</code> over orders scanned for the date. With no orders the sum is nil and no <code>Returns</code> line is written, while <code>get-refund-items</code> still credits <code>Card Refunds</code> from the <code>sales-refund</code> records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.</p>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What the fixes actually are</h2>
|
||
<div class="measure">
|
||
<p>Five changes. The first three stop two clients from sharing a record; the last two record
|
||
money that was being collected but not booked. Each is small — the difficulty was knowing
|
||
which line to change, not writing it. A sixth was written and then removed; it is described
|
||
at the end because the reasoning matters more than the code did.</p>
|
||
</div>
|
||
|
||
<h3>1 · Put the client in the record's name</h3>
|
||
<div class="measure">
|
||
<p>Every imported record has an identifier the importer uses to decide "have I seen this
|
||
before?". Sales orders already included the client; refunds, card payments, payouts and
|
||
cash-drawer shifts did not, which is precisely why two clients could land on one record.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; before — the bare Square id, identical for both clients</span>
|
||
(str "square/refund/" (:id r)) <span class="dim">;; square/refund/NOkQOTIiJULWN6…</span>
|
||
|
||
<span class="dim">;; after</span>
|
||
(scoped-key "square/refund/" client location (:id r))
|
||
<span class="dim">;; square/refund/NGCD-CD-NOkQOTIiJULWN6…</span>
|
||
|
||
(defn scoped-key [prefix client location id]
|
||
(str prefix (:client/code client) "-"
|
||
(:square-location/client-location location) "-" id))</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>Applied at five places in the Square importer: order payments, refunds, payouts (twice —
|
||
the record itself and the lookup that finds it) and cash-drawer shifts. ezCater orders
|
||
already did this and needed no change.</p>
|
||
</div>
|
||
|
||
<h3>2 · Find the existing record before writing, under either name</h3>
|
||
<div class="measure">
|
||
<p>This is the one that makes the change safe to deploy. The identifiers are unique keys, so
|
||
the importer relies on "same id, same record". Rename them and the next import matches
|
||
nothing — and would quietly create a <em>second</em> copy of every refund and payment in the
|
||
system, leaving the originals orphaned. So the importer looks up the record explicitly,
|
||
new name first, old name second, and writes to whichever it finds.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre>(defn existing-id [db attr prefix client location id]
|
||
(when id
|
||
(or (dc/entid db [attr (scoped-key prefix client location id)]) <span class="dim">;; new scheme</span>
|
||
(dc/entid db [attr (str prefix id)])))) <span class="dim">;; legacy scheme</span></pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The result is pinned as the record's id on the way in, so the write lands on the existing
|
||
row regardless of which name it currently carries. The proof this worked is a count that did
|
||
not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was
|
||
re-named, and afterwards there were still exactly <strong>50,986 refunds, 144,688 payouts and
|
||
69,291 cash-drawer shifts</strong> — the same three figures as at the restore point.
|
||
Had the fallback lookup been missing, each of these would have doubled instead.</p>
|
||
<p><strong>The fallback also has to refuse.</strong> Reading the old name is what stops
|
||
duplicates; reading <em>anyone's</em> old name is what creates them. Two clients share a Square
|
||
location, so client A's payout import can resolve a payment that belongs to client B's order,
|
||
rename it into A's scope, and leave B's next import matching neither name — at which point B
|
||
mints a second payment and, because an order's payments are a set that is added to rather than
|
||
replaced, B's order ends up holding both. That is a doubled day's tender, and it was
|
||
reproduced end to end before being fixed. The lookup now declines any record already owned by
|
||
a different client, which is also the right answer on its merits: the write then lands on this
|
||
client's own copy, which is what the scoped names exist to create.</p>
|
||
<p>This is transitional. Once no legacy names remain, the fallback and the refusal are deleted
|
||
together and the guarantee stops depending on either.</p>
|
||
</div>
|
||
|
||
<h3>3 · Give every order its own payment record</h3>
|
||
<div class="measure">
|
||
<p>Renaming stops <em>new</em> collisions but does not undo old ones: a payment already shared
|
||
by two orders is still one row with two owners. The migration walks each order's payments and,
|
||
where another order has already claimed one, makes that order its own copy with the same
|
||
amounts and points the order at the copy.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; for each order, for each of its payments:</span>
|
||
:keep <span class="dim">→</span> first order to claim it; rename in place
|
||
:clone <span class="dim">→</span> copy type, total, tip, tax, date, processor, note, receipt link
|
||
set the copy's client and location to this order's
|
||
retract this order's link to the shared payment
|
||
link it to the copy instead</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>Run over the whole database that was <strong>16,236,839 renamed and 500,438 copied</strong>,
|
||
and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across
|
||
every order of the last year. The record count rose by about 500,438 — the number of copies it
|
||
reported making, which is the check that it created what it meant to and nothing else.</p>
|
||
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
|
||
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
|
||
<span class="mono">N-30003</span> — so a pattern cannot tell where the client name ends and the
|
||
Square id begins. Getting this wrong scoped some records twice and doubled their tender.</p>
|
||
</div>
|
||
|
||
<h3>4 · Count tips that were handed back</h3>
|
||
<div class="measure">
|
||
<p>Tips were summed by walking from the order to its payments. A refund-only order has no
|
||
payment attached, so its negative tip was invisible. The fix adds those tips rather than
|
||
replacing the calculation.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; before</span>
|
||
:ledger-mapped/amount (tendered-tip c date)
|
||
|
||
<span class="dim">;; after</span>
|
||
:ledger-mapped/amount (+ (tendered-tip c date)
|
||
(untendered-tip c date))
|
||
|
||
<span class="dim">;; untendered-tip — tips on orders with no payment attached</span>
|
||
[?e :sales-order/tip ?tip]
|
||
(not [?e :sales-order/charges])</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Adding rather than replacing is deliberate.</strong> Where an order does have a
|
||
payment, the payment is the correct source: real orders exist whose payment carries a tip the
|
||
order does not — an auto-gratuity recorded as a service charge, or a wallet tip missing from
|
||
the order totals. Reading the order instead would have dropped those. Three tests hold this
|
||
in place: the refund case must change, and the tendered and ordinary cases must not.</p>
|
||
</div>
|
||
|
||
<h3>5 · Credit Square service charges, both signs</h3>
|
||
<div class="measure">
|
||
<p>Nothing read the service-charge field at all. A new line credits it, for Square orders only
|
||
and for negative amounts as well as positive.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre>[?e :sales-order/service-charge ?service-charge]
|
||
(or-join [?e]
|
||
[?e :sales-order/vendor :vendor/ccp-square]
|
||
(and (not [?e :sales-order/vendor])
|
||
[?e :sales-order/external-id ?external-id]
|
||
[(clojure.string/starts-with? ?external-id "square/order/")]))</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Why the vendor test has two branches.</strong> ezCater service charges are commission
|
||
the platform deducts from the restaurant, not money the diner hands over, so crediting them
|
||
would make a day worse rather than better — hence the Square-only condition. But whole eras of
|
||
Square orders carry no vendor field at all, and a test on vendor alone would silently credit
|
||
nothing. The second branch falls back to the order's own identifier.</p>
|
||
<p><strong>Why negatives matter.</strong> A returned catering fee arrives as a negative service
|
||
charge and is already deducted from the day's returns; dropping negatives would lose the
|
||
reversal. The line sits behind a per-client switch, off by default, so it can be turned on a
|
||
few restaurants at a time.</p>
|
||
</div>
|
||
|
||
<h3>6 · The change that was written, measured, and then taken out</h3>
|
||
<div class="measure">
|
||
<p>Worth recording, because the arithmetic case for it is good and someone will propose it
|
||
again. Where a day has refunds and no sales orders whatsoever, book a <code>Returns</code>
|
||
debit equal to that day's refunds:</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre>(defn- refund-only-returns [c date]
|
||
(when-not (traded? c date)
|
||
(let [amount (refunded-total c date)]
|
||
(when-not (zero? amount) amount))))</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>It works. Measured over the same ninety days it closed <strong>171 days and $5,795.18</strong>,
|
||
knocked nothing out of balance, and altered no already-balanced day — the guard makes it
|
||
incapable of touching a day that traded.</p>
|
||
<p>It was removed anyway. Those days are not quiet days; they are days whose sales were never
|
||
imported, and closing them removes the only visible sign of that. What is left in the code is
|
||
a comment saying so and a test asserting the day <em>stays</em> out of balance, so the next
|
||
person to notice the arithmetic finds the reasoning before they find the fix.</p>
|
||
</div>
|
||
|
||
<h3>Supporting changes</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Change</th><th>Why</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Log each day's imbalance and its suspect lines</td><td>an out-of-balance day was only visible by opening the screen; now it can be queried</td></tr>
|
||
<tr><td>Stop the dirty-summary scan at the client boundary</td><td>it read every later client's summaries too — 1,321 ms to 5.6 ms per client</td></tr>
|
||
<tr><td>Split the recompute driver into a per-client function</td><td>lets a backfill spread clients across threads instead of grinding one at a time</td></tr>
|
||
<tr><td>Install schema attributes before the tuples that compose them</td><td>the test suite could not build an empty database at all, so no test could run</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>That last one is worth a sentence for engineers: <code>transact-schema</code> installed
|
||
schema.edn then cloud-migration-schema.edn, but a composite tuple in the first file is built
|
||
from an attribute in the second. Datomic will not create a tuple before its members exist, so
|
||
every test fixture died in setup. It is very likely why sales summaries had no tests before
|
||
this work.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What each fix is worth</h2>
|
||
<div class="measure">
|
||
<p>The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates. All 18,900 client-day summaries in the window are included, whether or not the restaurant traded that day.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,191</td><td class="n">93.70%</td><td class="n">$70,276.50</td></tr>
|
||
<tr><td>+ refunded tips</td><td class="n">890</td><td class="n">95.29%</td><td class="n">$67,032.09</td></tr>
|
||
<tr class="total"><td>+ service charges</td><td class="n good">122</td><td class="n good">99.35%</td><td class="n good">$2,379.45</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$66,589.75 is what the three arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
|
||
</div>
|
||
|
||
<h3>Day-by-day effect of each change</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Refunded tips</td><td class="n">18,571</td><td class="n good">301</td><td class="n good">0</td><td class="n good">0</td><td class="n">$4,027.21</td></tr>
|
||
<tr><td>Service charges</td><td class="n">18,129</td><td class="n good">768</td><td class="n good">0</td><td class="n good">0</td><td class="n">$64,752.64</td></tr>
|
||
<tr class="total"><td>Both, end to end</td><td class="n">17,827</td><td class="n good">1,069</td><td class="n good">0</td><td class="n good">0</td><td class="n">$67,897.05</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Neither fix touched a day that was already correct.</strong> Across all 18,900
|
||
client-days no balanced day was knocked out of balance, and no balanced day had a single
|
||
figure altered — 17,827 summaries came out byte-identical, and every one of the 1,073 that
|
||
moved was already wrong.</p>
|
||
<p>That claim did not hold on the first attempt, and how it was recovered is the useful part.
|
||
Measured before the historical backfill described below, six days broke — all of them a tip
|
||
reversed on one record whose refund sat on its twin, so removing the un-reversed tip left the
|
||
day short by exactly that amount. Replaying the window from Square gave both records their own
|
||
copy of every refund, and all six closed. The fix was never wrong; it was reading half a
|
||
transaction.</p>
|
||
<p>Service charges are by far the larger of the two fixes, moving $58,923.85 against the tip fix's $3,777.67.</p>
|
||
<p>That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.</p>
|
||
<p><strong>The two fixes account for every day they moved, exactly.</strong> Adding up the untendered-tip and service-charge amounts across all 984 changed days leaves a residue of <span class="mono">0.0000000002</span>. Nothing else moved those days; there is no unexplained remainder hiding a third effect, and the six that broke are accounted for by the same arithmetic as the 915 that healed.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What it looks like on the page</h2>
|
||
<div class="measure">
|
||
<p>Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.</p>
|
||
</div>
|
||
|
||
<h3>A refunded tip — NGLK, 2026-08-04</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><strong>Tip</strong></td><td class="n">482.94</td><td class="n">422.94</td></tr>
|
||
<tr><td class="dim">Card Refunds</td><td class="n dim">60.00</td><td class="n dim">60.00</td></tr>
|
||
<tr><td>Total money taken</td><td class="n">10,094.81</td><td class="n">10,094.81</td></tr>
|
||
<tr><td>Total money earned</td><td class="n">10,154.81</td><td class="n">10,094.81</td></tr>
|
||
<tr class="total"><td>Out of balance by</td><td class="n bad">−60.00</td><td class="n good">0.00</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is <span class="mono">square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY</span>: a tip of −60.00 on an order with no payment attached at all.</p>
|
||
</div>
|
||
|
||
<h3>A service charge — NTPT, 2026-08-06</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><strong>Service Charges</strong></td><td class="n bad">not shown</td><td class="n">427.10</td></tr>
|
||
<tr><td class="dim">Card Payments</td><td class="n dim">4,975.89</td><td class="n dim">4,975.89</td></tr>
|
||
<tr><td>Total money taken</td><td class="n">7,777.20</td><td class="n">7,777.20</td></tr>
|
||
<tr><td>Total money earned</td><td class="n">7,350.10</td><td class="n">7,777.20</td></tr>
|
||
<tr class="total"><td>Out of balance by</td><td class="n bad">+427.10</td><td class="n good">0.00</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<h3>The largest repairs of each kind</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Client</th><th>Date</th><th>Line</th><th class="n">Before</th><th class="n">After</th><th class="n">Day closed</th></tr></thead>
|
||
<tbody>
|
||
<tr><td class="mono">NGPA</td><td>2026-06-04</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">1,344.86</td><td class="n good">+1,344.86 → 0</td></tr>
|
||
<tr><td class="mono">NTPT</td><td>2026-08-06</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">427.10</td><td class="n good">+427.10 → 0</td></tr>
|
||
<tr><td class="mono">N-30003</td><td>2026-05-27</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">405.83</td><td class="n good">+405.83 → 0</td></tr>
|
||
<tr><td class="mono">NGFL</td><td>2026-05-19</td><td>Tip</td><td class="n">238.46</td><td class="n">70.42</td><td class="n good">−168.04 → 0</td></tr>
|
||
<tr><td class="mono">NGMI</td><td>2026-07-09</td><td>Tip</td><td class="n">230.01</td><td class="n">80.01</td><td class="n good">−150.00 → 0</td></tr>
|
||
<tr><td class="mono">NGVA</td><td>2026-07-03</td><td>Tip</td><td class="n">152.66</td><td class="n">40.12</td><td class="n good">−112.54 → 0</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What was done to the data, and how it was checked</h2>
|
||
<div class="measure">
|
||
<p>Every step below was performed against a restored copy of the production database. Production itself was never touched.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Step</th><th>Result</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Walk every order in the database, newest month first</td><td class="n">19,040,785 orders · 38 minutes</td></tr>
|
||
<tr><td>Give every order its own payment record</td><td class="n">16,236,839 re-keyed · 500,438 copied</td></tr>
|
||
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across every order of the last year — 5,158,470</span></td></tr>
|
||
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
|
||
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
|
||
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added 500,438, matching the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)</p>
|
||
<p><strong>The measurement above was taken with both client records of each pair left live, which is how this deploys.</strong> Nothing is deactivated and no business decision about which restaurant's history survives is needed. The risk that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why <code>existing-id</code> now refuses to resolve a record belonging to another client.</p>
|
||
</div>
|
||
|
||
<div class="callout">
|
||
<span class="h">The whole analysis was run again from nothing, and landed in the same place</span>
|
||
<p>Everything above was rebuilt from a fresh restore of the production backup, several times over, each time from the backup point itself rather than from a database an earlier run had touched: restore, re-key and split across all nineteen million orders, then two full ninety-day recomputes. The runs used deliberately different preparation — one deactivated the duplicate records, one left them live and untouched, one backfilled their history from Square — so their headline figures differ, and comparing them is how the recommendation below was reached. What did <strong>not</strong> move is the part that should not: for the 190 clients that do not share a Square location the residue is 119 days and $1,730.61 in every run, with the same five restaurants accounting for it. The arithmetic fixes behave identically no matter what is done to the duplicates, which is a stronger check on them than any single measurement.</p>
|
||
</div>
|
||
|
||
<div class="callout warn">
|
||
<span class="h">A bug in this work, found by measuring rather than reading</span>
|
||
<p>The first attempt at copying shared payments derived each payment's Square identifier by stripping a fixed prefix. That is right the first time a payment is seen, but once it has been re-keyed to one client, a second order meeting it later read the already-scoped key as the identifier and scoped it twice — <span class="mono">NGCD-CD-NGCC-CC-<id></span>. The importer then created a fresh payment, doubling the tender on five clients by $3,000–$7,000 each. It was caught because the totals were absurd, not because the code looked wrong. The fix recovers the scope from the record itself; client codes contain dashes, so it cannot be done by pattern. A test now runs the step one order at a time, which is the arrangement that exposes it.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What is still out of balance</h2>
|
||
<div class="measure">
|
||
<p>122 client-days out of 18,900, totalling <strong>$2,379.45</strong> — and only 32 of
|
||
those are above ten cents.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Where the remainder sits</th><th class="n">Days</th><th class="n">Variance</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>The twenty records that share a Square location</td><td class="n good">3</td><td class="n good">$648.84</td></tr>
|
||
<tr class="total"><td>Every other client — 190 of the 210</td><td class="n">119</td><td class="n">$1,730.61</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>The shared-location records are now the clean part of the book.</strong> Three days
|
||
between all twenty of them: NGBK and NGBR at $299.42 each on 2026-08-06, which is the Square
|
||
tender-versus-order-total gap described below and not an attribution fault, and NGDA at $50.00,
|
||
an auto-gratuity booked as a service charge. Before the backfill those same records carried
|
||
423 days and $18,508.39.</p>
|
||
<p><strong>The other 119 days have not moved across any run of this analysis.</strong> Four
|
||
separate rebuilds — different databases, different preparation, one with the duplicates
|
||
deactivated and one without — all land on 119 days and $1,730.61, with the same five
|
||
restaurants accounting for almost all of it:</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Client</th><th class="n">Days</th><th class="n">Variance</th><th>What it is</th></tr></thead>
|
||
<tbody>
|
||
<tr><td class="mono">NG4S</td><td class="n">10</td><td class="n">$1,066.61</td><td>refunds arriving for a record with no sales imported — the fourth problem</td></tr>
|
||
<tr><td class="mono">NGMV</td><td class="n">5</td><td class="n">$259.38</td><td>late May, undiagnosed</td></tr>
|
||
<tr><td class="mono">NGEB</td><td class="n">4</td><td class="n">$199.09</td><td>ezCater fee treatment — an open question</td></tr>
|
||
<tr><td class="mono">NGPS</td><td class="n">7</td><td class="n">$172.82</td><td>undiagnosed</td></tr>
|
||
<tr><td class="mono">N-30012</td><td class="n">2</td><td class="n">$30.31</td><td>late May, undiagnosed</td></tr>
|
||
<tr class="total"><td class="dim">everyone else</td><td class="n dim">91</td><td class="n dim">$2.40</td><td class="dim">till rounding — pennies a day</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>Sixteen of the 122 are days a record had no sales imported at all, worth $1,227.65 — the
|
||
fourth problem, still deliberately visible. The clusters on NGMV, NGPS and NGEB are
|
||
unexplained and worth a look, though at under $650 across sixteen days they are no longer
|
||
urgent.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>Making the duplicated restaurants match</h2>
|
||
<div class="measure">
|
||
<p>Re-keying stops the two records fighting, but on its own it does not make them equal, and
|
||
the difference is worth stating plainly because it decides whether the books close.</p>
|
||
<p><strong>Orders were always duplicated; refunds never were.</strong> A sales order's
|
||
identifier has always carried its client, so each of the two records built its own order
|
||
history from the start. Refunds, payouts and cash-drawer shifts did not, so only ONE record
|
||
holds each of them — whichever imported it last. The migration freezes that ownership rather
|
||
than evening it out. The record left without them shows returns from its own orders and no
|
||
refunds to set against them, and is out of balance by exactly the amount its twin is holding.</p>
|
||
<p>On 2026-05-11 both NGBK and NGBR held the same 221 orders. NGBK had no refunds; NGBR had
|
||
two, worth $2,232.29; and NGBK's books were out by $2,232.29 to the cent. Across the whole
|
||
database NGBK held <strong>158,535 orders and five refunds</strong>.</p>
|
||
</div>
|
||
<div class="callout">
|
||
<span class="h">The fix is to ask Square again, not to manufacture copies</span>
|
||
<p>With client-scoped keys in place, every record now creates its own copy of whatever it
|
||
reads. So replaying the window from Square is all that is needed: each record imports the same
|
||
refunds independently and the two histories converge, without any code inventing a duplicate
|
||
and having to be trusted about it. <code>backfill-history</code> does exactly that for a date
|
||
range, and after it every one of the ten pairs held matching order and refund counts.</p>
|
||
<p>It closed <strong>420 of the 423 days</strong> the shared records were carrying, and
|
||
$17,859.55 of the $18,508.39. It is also what recovered the zero-regression guarantee above.</p>
|
||
</div>
|
||
<div class="callout warn">
|
||
<span class="h">One capped read, found by doing this</span>
|
||
<p>The refunds import asked Square for a location's refunds and read the first page of the
|
||
answer — no cursor, no date range. Square pages at a hundred, so a location with more than a
|
||
hundred refunds silently returned a hundred, and the response looked complete. That is why the
|
||
twins each held almost exactly 100 refunds, and why an earlier import added exactly 1,000
|
||
across ten locations. Following the cursor is a few lines; the reason it went unnoticed for so
|
||
long is that a capped list is indistinguishable from a short one.</p>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>This turned out to be a better answer than retiring the duplicate records.</strong>
|
||
An earlier measurement that deactivated one record of each pair left 279 days and $7,790.54.
|
||
Backfilling instead, with both records live, leaves <strong>122 days and $2,379.45</strong> —
|
||
and it needs no business decision about which restaurant's history to abandon.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>How complete is this</h2>
|
||
<div class="measure">
|
||
<p>The importer understands both the old and new record names, so the change can be deployed
|
||
before the renaming finishes. That tolerance is a bridge, not a destination — while any
|
||
record still carries an unscoped name, two clients can land on it and the guarantee rests on
|
||
a convention rather than on the data. So the renaming was run to completion and measured.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Record type</th><th class="n">Total</th><th class="n">Client-scoped</th><th class="n">Still to rename</th><th class="n">Cannot be scoped</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Card payments</td><td class="n">17,045,933</td><td class="n good">17,045,933</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||
<tr><td>Refunds</td><td class="n">50,986</td><td class="n good">50,986</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||
<tr><td>Payouts</td><td class="n">144,688</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr>
|
||
<tr><td>Cash-drawer shifts</td><td class="n">69,291</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>Every record in the database now carries its owner's name, and the migration proposes no
|
||
further changes: asked what is left to do, it answers zero on all four record types. The 36
|
||
payouts are ones with no client or location recorded anywhere, on the record itself or on
|
||
anything referring to it, so there is nothing to name them after.</p>
|
||
<p>Renaming had to be driven from orders, because a payment's rightful owner is whichever
|
||
order refers to it — so completing it meant walking all <strong>19,040,785 orders</strong>, not
|
||
just the clients that look shared today. Nine client pairs contended in the past without
|
||
sharing a location now, and a migration scoped to the current configuration would have missed
|
||
every one of them.</p>
|
||
<p><strong>A caution about how completeness is counted.</strong> 282,649 card payments carry no
|
||
client attribute of their own — they are stubs the payout path creates, never referenced by an
|
||
order. A gate that checks the attribute reports these as "no owner" and looks like a gap. They
|
||
are not: their names are scoped, recovered from the deposit that holds them. The figures above
|
||
are counted the harder way, by asking the migration what it would still change, which resolves
|
||
each record's owner through whatever refers to it. Reading the attribute alone would have
|
||
understated completeness by a quarter of a million records — and an early draft of this report
|
||
did exactly that.</p>
|
||
</div>
|
||
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Shared payments after the migration</th><th class="n">Count</th><th>Meaning</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><strong>Owned by more than one order</strong></td><td class="n good">0</td><td>whether the orders belong to different clients or the same one</td></tr>
|
||
<tr><td class="dim">checked across</td><td class="n dim">400,000 orders</td><td class="dim">spread through the whole database</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The problem this work exists to solve is gone: no payment answers to two orders, so the
|
||
component relationship means what it says and deleting an order can no longer take another
|
||
order's money with it.</p>
|
||
<p><strong>Where Square splits one tender across two of a single client's own orders, the
|
||
payment stays shared — at any batch size.</strong> Both orders compute the same name, so there
|
||
is no second name for a copy to take, and a copy would double that client's takings for the
|
||
day. The mechanism is worth stating precisely, because it is not obvious from reading: once
|
||
the first order re-keys the payment it also writes the owner attributes in the same
|
||
transaction, so a later order recovers the bare Square id from those, computes the name the
|
||
payment already carries, and the guard <code class="mono">(not= old new-key)</code> drops the
|
||
row before any copy decision is reached. Verified by running the migration at a batch size of
|
||
one, which forces the two orders into separate batches: no copy is made.</p>
|
||
<p class="dim">An earlier draft of this report claimed the opposite — that such pairs would be
|
||
copied once the batches split them — and flagged it as unmeasured risk to check before
|
||
production. That was wrong, and it is recorded here rather than quietly deleted because it did
|
||
real damage: an independent reviewer cited this paragraph as evidence and raised a defect that
|
||
does not exist. A test now pins the behaviour at batch size one.</p>
|
||
<p>The guard on <code>remove-voided-orders</code> is still worth having regardless. It is
|
||
cheap, and it makes the safety a property of the deletion rather than of the migration having
|
||
been run first.</p>
|
||
</div>
|
||
|
||
<div class="callout">
|
||
<span class="h">Re-running is safe, and that was proved at full scale</span>
|
||
<p>After the complete pass, asking the migration what it would change next returns
|
||
<strong>nothing</strong> — 17,045,933 payments examined, none to rename, none unscopable. A
|
||
record that already carries the right name is left untouched, so the migration can be stopped,
|
||
resumed, or repeated without consequence.</p>
|
||
<p>Its speed is worth a note for whoever schedules it: the whole nineteen million orders were
|
||
walked in about <strong>thirty-eight minutes</strong>, month by month from the current month
|
||
backwards so that stopping early leaves the recent end done. An earlier attempt appeared to be
|
||
transactor-bound and was projected at two days, which is why a previous run narrowed it to the
|
||
analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process
|
||
driving the migration — freeing held memory took an unrelated recompute from 17 client-days a
|
||
minute to 4,515. Nothing about the database or the transactor needed to change.</p>
|
||
</div>
|
||
|
||
<div class="measure">
|
||
<p><strong>What follows from the gate reading zero.</strong> <code>unscoped-report</code> counts
|
||
these figures on demand. Now that unscoped is zero across the board, the importer's
|
||
understanding of the old name form can be removed — at which point two clients sharing a
|
||
location becomes structurally incapable of producing a shared record, rather than prevented by
|
||
a convention that a future import could quietly break. That removal is the one remaining step
|
||
of this piece of work.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>Decisions and risks still open</h2>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Item</th><th>Who decides</th><th>Why it matters</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Which client record survives at each shared location</td><td>the business</td><td>the newer record generally has no history before the split, so keeping it loses years of the location's books</td></tr>
|
||
<tr><td>Which revenue account service charges post to</td><td>accounting</td><td>currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances</td></tr>
|
||
<tr><td><strong>659 refunds on records that have no sales for them</strong></td><td>the business, then engineering</td><td>the top open item. $15,225.24 dated before the holding record's own first order. Either the missing sales get imported, or the refunds move to the record that has them — but the books cannot close until one of the two happens</td></tr>
|
||
<tr><td>Whether to correct records the wrong client already owns</td><td>the business</td><td>the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared</td></tr>
|
||
<tr><td><code>remove-voided-orders</code></td><td>engineering</td><td>safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<div class="callout warn">
|
||
<span class="h">Two operational findings, unrelated to the summaries</span>
|
||
<p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p>
|
||
<p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. Worth checking what production is set to.</p>
|
||
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in well under an hour. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>How to check any of this <span class="tech">technical</span></h2>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; the restored database, untouched production as of 2026-08-14 22:52</span>
|
||
(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore"))
|
||
|
||
<span class="dim">;; the two orders behind the worked examples</span>
|
||
(d/pull (d/db conn) '[*] [:sales-order/external-id
|
||
"square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"])
|
||
(d/pull (d/db conn) '[*] [:sales-order/external-id
|
||
"square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"])
|
||
|
||
<span class="dim">;; the gate: no payment may have two parent orders</span>
|
||
(rk/charges-with-multiple-parents (d/db conn) orders) <span class="dim">;; => 0</span>
|
||
|
||
<span class="dim">;; ownership history — which records ever changed client</span>
|
||
(->> (d/datoms (d/history (d/db conn)) :aevt :sales-refund/client)
|
||
(filter :added)
|
||
(reduce (fn [m d] (update m (:e d) (fnil conj #{}) (:v d))) {})
|
||
(filter (fn [[_ owners]] (> (count owners) 1)))
|
||
count)</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The comparison tool is committed as <code>auto-ap.jobs.compare-sales-summaries</code>. Unit tests: <code>lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test</code>.</p>
|
||
</div>
|
||
|
||
<div class="callout warn">
|
||
<span class="h">Do not compare summaries with <code>as-of</code> — a correction to how this was measured</span>
|
||
<p>The obvious way to audit a recompute is to read the database at a point before it and diff:
|
||
Datomic keeps every past value, so no snapshot is needed. That is what
|
||
<code>compare-sales-summaries</code> was built to do, and for summary amounts it does not work.
|
||
<code>:ledger-mapped/amount</code>, <code>:ledger-mapped/ledger-side</code> and
|
||
<code>:ledger-mapped/account</code> are all declared <code>:db/noHistory true</code>, so
|
||
superseded values are discarded rather than retained. A historical read of a summary that has
|
||
since been recomputed can return its lines with the categories intact and the amounts simply
|
||
absent — which reads as a legitimate all-zero summary, not as an error.</p>
|
||
<p>Every figure in this report is therefore taken from a live read of the database immediately
|
||
after each pass, captured and stored outside it, and the before/after comparison is done
|
||
between those two captures. No historical read is involved anywhere in the numbers above. The
|
||
tool remains useful for categories and for which days changed; its docstring overstates what it
|
||
can recover, and that is worth correcting before someone relies on it for amounts.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<footer>
|
||
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
|
||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 122 remaining days only 32 are material, and just 3 of them sit on the twenty records that share a Square location.</span>
|
||
<span>Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on <code>as-of</code> above.</span>
|
||
</footer>
|
||
|
||
</div>
|