Bryce
a16ef0bd60
Replaces the partial-completeness section with the measured end state. card payments 17,046,418 scoped · 0 unscoped · 0 unscopable refunds 51,986 scoped · 0 unscoped payouts 144,652 scoped · 0 unscoped · 36 with no owner cash shifts 69,291 scoped · 0 unscoped Payments shared between two different clients: 0 in a 250,000-order sample spread across the database. 104 remain shared between two orders of the same client, left that way on purpose — Square split one tender across two of that client's orders, both orders compute the same name so there is no second name a copy could take, and a copy would double the client's takings. One payment covering both is the truthful record. That is what makes the remove-voided-orders guard a required companion rather than belt-and-braces: the cascade still reaches those 104. Records that a second complete pass over all 19,040,296 orders took 7.8 minutes and changed nothing, so the migration is stoppable, resumable and repeatable — and that with the gate now reading zero, the importer's understanding of the old name form can be removed, which is the one remaining step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
585 lines
40 KiB
HTML
585 lines
40 KiB
HTML
<title>Ninety-Day Reconciliation</title>
|
||
<style>
|
||
:root {
|
||
--paper: #F6F8F7; --card: #FFFFFF; --ink: #141F1D; --ink-soft: #4A5C58;
|
||
--ink-faint: #7C8D89; --rule: #DCE4E1; --accent: #0E5B57; --accent-soft: #E3EFED;
|
||
--good: #1A6B49; --bad: #A03B26; --warn: #8A6410;
|
||
--shadow: 0 1px 2px rgba(20,31,29,.06), 0 8px 24px rgba(20,31,29,.05);
|
||
}
|
||
@media (prefers-color-scheme: dark) {
|
||
:root:not([data-theme="light"]) {
|
||
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
|
||
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
|
||
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
|
||
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
|
||
}
|
||
}
|
||
:root[data-theme="dark"] {
|
||
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
|
||
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
|
||
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
|
||
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
|
||
}
|
||
* { box-sizing: border-box; }
|
||
body { background: var(--paper); color: var(--ink);
|
||
font-family: system-ui, -apple-system, "Segoe UI", sans-serif;
|
||
font-size: 16px; line-height: 1.6; margin: 0; padding: 0 20px 96px; }
|
||
.wrap { max-width: 940px; margin: 0 auto; }
|
||
.measure { max-width: 66ch; }
|
||
.num { font-variant-numeric: tabular-nums; }
|
||
.mono { font-family: ui-monospace, "SF Mono", "Cascadia Code", monospace; font-variant-numeric: tabular-nums; }
|
||
|
||
header.masthead { padding: 72px 0 40px; border-bottom: 2px solid var(--ink); display: flex; flex-direction: column; gap: 14px; }
|
||
.eyebrow { font-size: 12px; letter-spacing: .14em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
|
||
h1 { font-family: Georgia, "Iowan Old Style", serif; font-size: clamp(34px, 5.4vw, 54px);
|
||
line-height: 1.08; font-weight: 600; letter-spacing: -.015em; margin: 0; text-wrap: balance; }
|
||
.standfirst { font-size: 19px; color: var(--ink-soft); margin: 0; max-width: 62ch; }
|
||
.meta { display: flex; flex-wrap: wrap; gap: 10px 28px; font-size: 13px; color: var(--ink-faint); padding-top: 6px; }
|
||
.meta b { color: var(--ink-soft); font-weight: 600; }
|
||
|
||
section { padding-top: 56px; display: flex; flex-direction: column; gap: 20px; }
|
||
h2 { font-family: Georgia, "Iowan Old Style", serif; font-size: 27px; font-weight: 600; letter-spacing: -.01em; margin: 0; text-wrap: balance; }
|
||
h3 { font-size: 14px; letter-spacing: .08em; text-transform: uppercase; color: var(--ink-soft); font-weight: 700; margin: 0; }
|
||
h4 { font-size: 18px; font-weight: 650; margin: 0; letter-spacing: -.01em; }
|
||
p { margin: 0; }
|
||
.measure p + p { margin-top: 14px; }
|
||
|
||
.ledger { display: grid; grid-template-columns: 1fr auto 1fr; border: 1px solid var(--rule);
|
||
border-radius: 4px; background: var(--card); box-shadow: var(--shadow); overflow: hidden; }
|
||
.ledger > div { padding: 26px 28px; display: flex; flex-direction: column; gap: 6px; }
|
||
.ledger .arrow { justify-content: center; align-items: center; border-left: 1px solid var(--rule);
|
||
border-right: 1px solid var(--rule); color: var(--ink-faint); font-size: 22px; background: var(--accent-soft); }
|
||
.side-label { font-size: 12px; letter-spacing: .12em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; }
|
||
.figure { font-size: clamp(28px, 4.4vw, 40px); font-weight: 650; line-height: 1.05; letter-spacing: -.02em; }
|
||
.figure.after { color: var(--good); }
|
||
.subfig { font-size: 14px; color: var(--ink-soft); }
|
||
|
||
.stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 14px; }
|
||
.stat { background: var(--card); border: 1px solid var(--rule); border-radius: 4px; padding: 18px 20px; display: flex; flex-direction: column; gap: 4px; }
|
||
.stat .k { font-size: 30px; font-weight: 650; letter-spacing: -.02em; line-height: 1; }
|
||
.stat .l { font-size: 13px; color: var(--ink-soft); }
|
||
.stat.zero .k { color: var(--good); }
|
||
|
||
.problem { border-left: 3px solid var(--accent); padding-left: 24px; display: flex; flex-direction: column; gap: 14px; }
|
||
.problem.two { border-left-color: var(--warn); }
|
||
.problem.three { border-left-color: var(--bad); }
|
||
|
||
.scroll { overflow-x: auto; border: 1px solid var(--rule); border-radius: 4px; background: var(--card); }
|
||
table { border-collapse: collapse; width: 100%; font-size: 14.5px; }
|
||
th, td { padding: 11px 16px; text-align: left; border-bottom: 1px solid var(--rule); white-space: nowrap; }
|
||
thead th { font-size: 11.5px; letter-spacing: .09em; text-transform: uppercase; color: var(--ink-faint); font-weight: 700; background: var(--accent-soft); }
|
||
tbody tr:last-child td { border-bottom: none; }
|
||
td.n, th.n { text-align: right; font-variant-numeric: tabular-nums; }
|
||
tr.total td { font-weight: 650; background: var(--accent-soft); }
|
||
.good { color: var(--good); font-weight: 650; }
|
||
.bad { color: var(--bad); font-weight: 650; }
|
||
.dim { color: var(--ink-faint); }
|
||
|
||
.callout { background: var(--card); border: 1px solid var(--rule); border-left: 3px solid var(--accent);
|
||
border-radius: 4px; padding: 20px 24px; display: flex; flex-direction: column; gap: 10px; }
|
||
.callout.warn { border-left-color: var(--warn); }
|
||
.callout .h { font-weight: 650; }
|
||
code { font-family: ui-monospace, "SF Mono", monospace; font-size: .9em; background: var(--accent-soft); padding: 1px 5px; border-radius: 3px; }
|
||
pre { margin: 0; padding: 20px; font-size: 13px; line-height: 1.7; white-space: pre; font-family: ui-monospace, "SF Mono", monospace; }
|
||
footer { margin-top: 72px; padding-top: 24px; border-top: 1px solid var(--rule); font-size: 13px; color: var(--ink-faint); display: flex; flex-direction: column; gap: 8px; }
|
||
ul { margin: 0; padding-left: 20px; display: flex; flex-direction: column; gap: 8px; }
|
||
.tech { font-size: 11px; letter-spacing: .08em; text-transform: uppercase; color: var(--accent);
|
||
font-weight: 700; border: 1px solid var(--accent); border-radius: 3px; padding: 2px 7px; display: inline-block; }
|
||
</style>
|
||
|
||
<div class="wrap">
|
||
|
||
<header class="masthead">
|
||
<div class="eyebrow">Sales summaries · measured on a restored production backup</div>
|
||
<h1>Ninety-Day Reconciliation</h1>
|
||
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading.</p>
|
||
<div class="meta">
|
||
<span><b>Window</b> 2026-05-10 → 2026-08-07</span>
|
||
<span><b>Client-days</b> <span class="num">8,733</span></span>
|
||
<span><b>Clients</b> <span class="num">106</span></span>
|
||
<span><b>Nothing in production was changed</b></span>
|
||
</div>
|
||
</header>
|
||
|
||
<section>
|
||
<div class="ledger">
|
||
<div>
|
||
<span class="side-label">Production today</span>
|
||
<span class="figure num">$75,228.78</span>
|
||
<span class="subfig"><span class="num">1,280</span> days out of balance · <span class="num">85.34%</span> clean</span>
|
||
</div>
|
||
<div class="arrow" aria-hidden="true">→</div>
|
||
<div>
|
||
<span class="side-label">After all three fixes</span>
|
||
<span class="figure after num">$1,995.36</span>
|
||
<span class="subfig"><span class="num">108</span> days out of balance · <span class="num">98.76%</span> clean</span>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="stats">
|
||
<div class="stat"><span class="k num">1,172</span><span class="l">client-days brought into balance</span></div>
|
||
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
|
||
<div class="stat"><span class="k num">97.3%</span><span class="l">of the variance removed</span></div>
|
||
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
|
||
</div>
|
||
|
||
<div class="measure">
|
||
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one day in seven it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>The three problems</h2>
|
||
|
||
<div class="problem">
|
||
<h4>1. Two client records sharing one Square location</h4>
|
||
<div class="measure">
|
||
<p><strong>For the business:</strong> ten restaurant locations were set up twice in the system, as two separate clients. Both were importing from Square. Because the two records competed for the same payments and refunds, a refund would belong to one client for twenty minutes, then the other — so a day's books could gain or lose a refund depending on nothing but timing. On 2026-07-23 one client's summary was missing a $71.94 refund entirely, and was out of balance by exactly that amount.</p>
|
||
<p><span class="tech">technical</span> Sales orders scoped their identifier by client (<code>square/order/<code>-<loc>-<id></code>), but refunds, card charges, payouts and cash-drawer shifts did not — they used the bare Square id. Those attributes are <code>:db.unique/identity</code>, so both clients' imports resolved to a single entity and the last writer won.</p>
|
||
<p>Reading ownership out of the database's own history, this had actually happened to <strong>3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts</strong>. And it has involved <strong>19 client pairs, of which only 10 are visible in today's configuration</strong> — nine more contended in the past and the configuration has since changed, so no point-in-time check would find them.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="problem two">
|
||
<h4>2. One payment record owned by two orders</h4>
|
||
<div class="measure">
|
||
<p><strong>For the business:</strong> the same collision meant a single card payment could be attached to both clients' copies of an order. That is worse than untidy. The nightly import removes orders Square reports as voided, and removing an order also removes its payments — so cancelling one client's order could silently delete the <em>other</em> client's payment, leaving a day showing sales with no money against them.</p>
|
||
<p><span class="tech">technical</span> <code>:sales-order/charges</code> is declared <code>:db/isComponent true</code>, so <code>[:db/retractEntity <order>]</code> cascades into the charges. In a 20,000-order sample of the affected clients, <strong>11,469 charges had two parent orders</strong>. This is why <code>remove-voided-orders</code> was left switched off during testing.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="problem three">
|
||
<h4>3. Tips refunded, and service charges credited nowhere</h4>
|
||
<div class="measure">
|
||
<p><strong>For the business:</strong> two arithmetic faults, both of which overstated or understated a day.</p>
|
||
<ul>
|
||
<li><strong>Refunded tips stayed on the books.</strong> When a guest was refunded, the tip came back too — but the summary still counted the original tip as income. On one NGLK day the books credited $482.94 of tips beside a $60.00 refund of that very tip.</li>
|
||
<li><strong>Service charges were collected but never earned.</strong> A catering or auto-gratuity charge is inside the card payment the customer makes, so it arrived as money taken — but no line recorded it as money earned. One NTPT order carried $427.10 that was credited to nothing at all; the largest single instance was <strong>$1,344.86</strong> in one day.</li>
|
||
</ul>
|
||
<p><span class="tech">technical</span> <code>get-tip</code> summed tips by joining through <code>:sales-order/charges</code>, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on <code>:sales-order/tip</code>. Nothing at all read <code>:sales-order/service-charge</code>.</p>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What the fixes actually are</h2>
|
||
<div class="measure">
|
||
<p>Five changes. The first three stop two clients from sharing a record; the last two record
|
||
money that was being collected but not booked. Each is small — the difficulty was knowing
|
||
which line to change, not writing it.</p>
|
||
</div>
|
||
|
||
<h3>1 · Put the client in the record's name</h3>
|
||
<div class="measure">
|
||
<p>Every imported record has an identifier the importer uses to decide "have I seen this
|
||
before?". Sales orders already included the client; refunds, card payments, payouts and
|
||
cash-drawer shifts did not, which is precisely why two clients could land on one record.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; before — the bare Square id, identical for both clients</span>
|
||
(str "square/refund/" (:id r)) <span class="dim">;; square/refund/NOkQOTIiJULWN6…</span>
|
||
|
||
<span class="dim">;; after</span>
|
||
(scoped-key "square/refund/" client location (:id r))
|
||
<span class="dim">;; square/refund/NGCD-CD-NOkQOTIiJULWN6…</span>
|
||
|
||
(defn scoped-key [prefix client location id]
|
||
(str prefix (:client/code client) "-"
|
||
(:square-location/client-location location) "-" id))</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>Applied at five places in the Square importer: order payments, refunds, payouts (twice —
|
||
the record itself and the lookup that finds it) and cash-drawer shifts. ezCater orders
|
||
already did this and needed no change.</p>
|
||
</div>
|
||
|
||
<h3>2 · Find the existing record before writing, under either name</h3>
|
||
<div class="measure">
|
||
<p>This is the one that makes the change safe to deploy. The identifiers are unique keys, so
|
||
the importer relies on "same id, same record". Rename them and the next import matches
|
||
nothing — and would quietly create a <em>second</em> copy of every refund and payment in the
|
||
system, leaving the originals orphaned. So the importer looks up the record explicitly,
|
||
new name first, old name second, and writes to whichever it finds.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre>(defn existing-id [db attr prefix client location id]
|
||
(when id
|
||
(or (dc/entid db [attr (scoped-key prefix client location id)]) <span class="dim">;; new scheme</span>
|
||
(dc/entid db [attr (str prefix id)])))) <span class="dim">;; legacy scheme</span></pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The result is pinned as the record's id on the way in, so the write lands on the existing
|
||
row regardless of which name it currently carries. The proof this worked: after re-naming
|
||
213,943 records, the totals for refunds, payouts and shifts were <strong>identical before and
|
||
after</strong>. Had the fallback been missing they would have doubled.</p>
|
||
</div>
|
||
|
||
<h3>3 · Give every order its own payment record</h3>
|
||
<div class="measure">
|
||
<p>Renaming stops <em>new</em> collisions but does not undo old ones: a payment already shared
|
||
by two orders is still one row with two owners. The migration walks each order's payments and,
|
||
where another order has already claimed one, makes that order its own copy with the same
|
||
amounts and points the order at the copy.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; for each order, for each of its payments:</span>
|
||
:keep <span class="dim">→</span> first order to claim it; rename in place
|
||
:clone <span class="dim">→</span> copy type, total, tip, tax, date, processor, note, receipt link
|
||
set the copy's client and location to this order's
|
||
retract this order's link to the shared payment
|
||
link it to the copy instead</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>On the restored data that was <strong>189,167 renamed and 77,599 copied</strong>, and payments
|
||
owned by two orders went from 11,469 to zero. The record count rose by exactly 77,599 — the
|
||
number of copies it reported making, which is the check that it created what it meant to and
|
||
nothing else.</p>
|
||
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
|
||
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
|
||
<span class="mono">N-30003</span> — so a pattern cannot tell where the client name ends and the
|
||
Square id begins. Getting this wrong scoped some records twice and doubled their tender.</p>
|
||
</div>
|
||
|
||
<h3>4 · Count tips that were handed back</h3>
|
||
<div class="measure">
|
||
<p>Tips were summed by walking from the order to its payments. A refund-only order has no
|
||
payment attached, so its negative tip was invisible. The fix adds those tips rather than
|
||
replacing the calculation.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; before</span>
|
||
:ledger-mapped/amount (tendered-tip c date)
|
||
|
||
<span class="dim">;; after</span>
|
||
:ledger-mapped/amount (+ (tendered-tip c date)
|
||
(untendered-tip c date))
|
||
|
||
<span class="dim">;; untendered-tip — tips on orders with no payment attached</span>
|
||
[?e :sales-order/tip ?tip]
|
||
(not [?e :sales-order/charges])</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Adding rather than replacing is deliberate.</strong> Where an order does have a
|
||
payment, the payment is the correct source: real orders exist whose payment carries a tip the
|
||
order does not — an auto-gratuity recorded as a service charge, or a wallet tip missing from
|
||
the order totals. Reading the order instead would have dropped those. Three tests hold this
|
||
in place: the refund case must change, and the tendered and ordinary cases must not.</p>
|
||
</div>
|
||
|
||
<h3>5 · Credit Square service charges, both signs</h3>
|
||
<div class="measure">
|
||
<p>Nothing read the service-charge field at all. A new line credits it, for Square orders only
|
||
and for negative amounts as well as positive.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<pre>[?e :sales-order/service-charge ?service-charge]
|
||
(or-join [?e]
|
||
[?e :sales-order/vendor :vendor/ccp-square]
|
||
(and (not [?e :sales-order/vendor])
|
||
[?e :sales-order/external-id ?external-id]
|
||
[(clojure.string/starts-with? ?external-id "square/order/")]))</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Why the vendor test has two branches.</strong> ezCater service charges are commission
|
||
the platform deducts from the restaurant, not money the diner hands over, so crediting them
|
||
would make a day worse rather than better — hence the Square-only condition. But whole eras of
|
||
Square orders carry no vendor field at all, and a test on vendor alone would silently credit
|
||
nothing. The second branch falls back to the order's own identifier.</p>
|
||
<p><strong>Why negatives matter.</strong> A returned catering fee arrives as a negative service
|
||
charge and is already deducted from the day's returns; dropping negatives would lose the
|
||
reversal. The line sits behind a per-client switch, off by default, so it can be turned on a
|
||
few restaurants at a time.</p>
|
||
</div>
|
||
|
||
<h3>Supporting changes</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Change</th><th>Why</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Log each day's imbalance and its suspect lines</td><td>an out-of-balance day was only visible by opening the screen; now it can be queried</td></tr>
|
||
<tr><td>Stop the dirty-summary scan at the client boundary</td><td>it read every later client's summaries too — 1,321 ms to 5.6 ms per client</td></tr>
|
||
<tr><td>Split the recompute driver into a per-client function</td><td>lets a backfill spread clients across threads instead of grinding one at a time</td></tr>
|
||
<tr><td>Install schema attributes before the tuples that compose them</td><td>the test suite could not build an empty database at all, so no test could run</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>That last one is worth a sentence for engineers: <code>transact-schema</code> installed
|
||
schema.edn then cloud-migration-schema.edn, but a composite tuple in the first file is built
|
||
from an attribute in the second. Datomic will not create a tuple before its members exist, so
|
||
every test fixture died in setup. It is very likely why sales summaries had no tests before
|
||
this work.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What each fix is worth</h2>
|
||
<div class="measure">
|
||
<p>The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Production today</td><td class="n">1,280</td><td class="n">85.34%</td><td class="n">$75,228.78</td></tr>
|
||
<tr><td>+ deduplication</td><td class="n">1,087</td><td class="n">87.55%</td><td class="n">$63,764.92</td></tr>
|
||
<tr><td>+ refunded tips</td><td class="n">800</td><td class="n">90.84%</td><td class="n">$60,619.21</td></tr>
|
||
<tr class="total"><td>+ service charges</td><td class="n good">108</td><td class="n good">98.76%</td><td class="n good">$1,995.36</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<h3>Day-by-day effect of each change</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Deduplication</td><td class="n">7,447 <span class="dim">balanced already</span></td><td class="n good">199</td><td class="n bad">6</td><td class="n dim">—</td><td class="n dim">—</td></tr>
|
||
<tr><td>Refunded tips</td><td class="n">8,421</td><td class="n good">287</td><td class="n good">0</td><td class="n good">0</td><td class="n">$3,777.67</td></tr>
|
||
<tr><td>Service charges</td><td class="n">8,037</td><td class="n good">692</td><td class="n good">0</td><td class="n good">0</td><td class="n">$58,923.85</td></tr>
|
||
<tr class="total"><td>All three, end to end</td><td class="n">7,453</td><td class="n good">1,172</td><td class="n good">0</td><td class="n good">0</td><td class="n">—</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Neither arithmetic fix touched a day that was already correct.</strong> Across 8,733 client-days, no balanced day was knocked out of balance and no balanced day had a single figure altered — every day they changed was already wrong. Service charges are by far the larger of the two, moving $58,923.85 against the tip fix's $3,777.67.</p>
|
||
<p>Deduplication is the one step that puts six days <em>out</em> of balance. That is expected: those days were previously balanced only because a summary was empty, and filling it in exposes the same arithmetic faults every other day had. Both later fixes then close them, which is why the end-to-end figure is zero.</p>
|
||
<p>Excluding the ten now-deactivated duplicate records, which should not be reported on at all, the population is 8,350 client-days and the movement is <strong>1,217 days and $69,995.61 down to 105 days and $852.38</strong>.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What it looks like on the page</h2>
|
||
<div class="measure">
|
||
<p>Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.</p>
|
||
</div>
|
||
|
||
<h3>A refunded tip — NGLK, 2026-08-04</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><strong>Tip</strong></td><td class="n">482.94</td><td class="n">422.94</td></tr>
|
||
<tr><td class="dim">Card Refunds</td><td class="n dim">60.00</td><td class="n dim">60.00</td></tr>
|
||
<tr><td>Total money taken</td><td class="n">10,094.81</td><td class="n">10,094.81</td></tr>
|
||
<tr><td>Total money earned</td><td class="n">10,154.81</td><td class="n">10,094.81</td></tr>
|
||
<tr class="total"><td>Out of balance by</td><td class="n bad">−60.00</td><td class="n good">0.00</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is <span class="mono">square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY</span>: a tip of −60.00 on an order with no payment attached at all.</p>
|
||
</div>
|
||
|
||
<h3>A service charge — NTPT, 2026-08-06</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><strong>Service Charges</strong></td><td class="n bad">not shown</td><td class="n">427.10</td></tr>
|
||
<tr><td class="dim">Card Payments</td><td class="n dim">4,975.89</td><td class="n dim">4,975.89</td></tr>
|
||
<tr><td>Total money taken</td><td class="n">7,777.20</td><td class="n">7,777.20</td></tr>
|
||
<tr><td>Total money earned</td><td class="n">7,350.10</td><td class="n">7,777.20</td></tr>
|
||
<tr class="total"><td>Out of balance by</td><td class="n bad">+427.10</td><td class="n good">0.00</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<h3>The largest repairs of each kind</h3>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Client</th><th>Date</th><th>Line</th><th class="n">Before</th><th class="n">After</th><th class="n">Day closed</th></tr></thead>
|
||
<tbody>
|
||
<tr><td class="mono">NGPA</td><td>2026-06-04</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">1,344.86</td><td class="n good">+1,344.86 → 0</td></tr>
|
||
<tr><td class="mono">NTPT</td><td>2026-08-06</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">427.10</td><td class="n good">+427.10 → 0</td></tr>
|
||
<tr><td class="mono">N-30003</td><td>2026-05-27</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">405.83</td><td class="n good">+405.83 → 0</td></tr>
|
||
<tr><td class="mono">NGFL</td><td>2026-05-19</td><td>Tip</td><td class="n">238.46</td><td class="n">70.42</td><td class="n good">−168.04 → 0</td></tr>
|
||
<tr><td class="mono">NGMI</td><td>2026-07-09</td><td>Tip</td><td class="n">230.01</td><td class="n">80.01</td><td class="n good">−150.00 → 0</td></tr>
|
||
<tr><td class="mono">NGVA</td><td>2026-07-03</td><td>Tip</td><td class="n">152.66</td><td class="n">40.12</td><td class="n good">−112.54 → 0</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What was done to the data, and how it was checked</h2>
|
||
<div class="measure">
|
||
<p>Every step below was performed against a restored copy of the production database. Production itself was never touched.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Step</th><th>Result</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Deactivate the duplicate client at each shared location</td><td class="n">10 locations · shared locations remaining: <span class="good">0</span></td></tr>
|
||
<tr><td>Give every order its own payment record</td><td class="n">189,167 re-keyed · 77,599 copied</td></tr>
|
||
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">11,469 → 0</td></tr>
|
||
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
|
||
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
|
||
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 77,599, matching the number of copies it reported making.</p>
|
||
</div>
|
||
|
||
<div class="callout warn">
|
||
<span class="h">A bug in this work, found by measuring rather than reading</span>
|
||
<p>The first attempt at copying shared payments derived each payment's Square identifier by stripping a fixed prefix. That is right the first time a payment is seen, but once it has been re-keyed to one client, a second order meeting it later read the already-scoped key as the identifier and scoped it twice — <span class="mono">NGCD-CD-NGCC-CC-<id></span>. The importer then created a fresh payment, doubling the tender on five clients by $3,000–$7,000 each. It was caught because the totals were absurd, not because the code looked wrong. The fix recovers the scope from the record itself; client codes contain dashes, so it cannot be done by pattern. A test now runs the step one order at a time, which is the arrangement that exposes it.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>What is still out of balance</h2>
|
||
<div class="measure">
|
||
<p>Fifteen client-days across ninety days and 8,350 client-days, totalling $850.12. Everything else — 90 further days — comes to <strong>$2.26</strong>, the largest being 9.00¢, which is till rounding.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Client</th><th>Date</th><th class="n">Variance</th><th>What it is</th></tr></thead>
|
||
<tbody>
|
||
<tr><td class="mono">NGBK</td><td>2026-08-06</td><td class="n">+299.42</td><td>Square recorded more payment than the orders account for</td></tr>
|
||
<tr><td class="mono">NGMV</td><td>May 20–26</td><td class="n">+259.38</td><td>five days, undiagnosed</td></tr>
|
||
<tr><td class="mono">NGEB</td><td>May 13 – Jul 29</td><td class="n">−199.09</td><td>four days, ezCater fee treatment — an open question</td></tr>
|
||
<tr><td class="mono">NGDA</td><td>2026-08-01</td><td class="n">−50.00</td><td>auto-gratuity recorded as a service charge</td></tr>
|
||
<tr><td class="mono">N-30012</td><td>May 20–21</td><td class="n">+30.31</td><td>two days, undiagnosed</td></tr>
|
||
<tr><td class="mono">NG4S</td><td>2026-05-29</td><td class="n">−11.78</td><td>undiagnosed</td></tr>
|
||
<tr><td class="mono">PNSP</td><td>2026-07-12</td><td class="n">−0.14</td><td>till rounding, just over the threshold</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The ten deactivated duplicate records contribute a further $1,142.98 across three days. They are excluded above and should be excluded from reporting generally, since they are now dormant copies.</p>
|
||
<p>Two clusters — NGMV and N-30012, both in late May — are new and unexplained. They only became visible because the window was widened to ninety days; a thirty-day view did not reach them. They are worth a look before this ships.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>How complete is this</h2>
|
||
<div class="measure">
|
||
<p>The importer understands both the old and new record names, so the change can be deployed
|
||
before the renaming finishes. That tolerance is a bridge, not a destination — while any
|
||
record still carries an unscoped name, two clients can land on it and the guarantee rests on
|
||
a convention rather than on the data. So the renaming was run to completion and measured.</p>
|
||
</div>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Record type</th><th class="n">Client-scoped</th><th class="n">Still unscoped</th><th class="n">No owner to scope by</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Card payments</td><td class="n good">17,046,418</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||
<tr><td>Refunds</td><td class="n good">51,986</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||
<tr><td>Payouts</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr>
|
||
<tr><td>Cash-drawer shifts</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p>Every record in the database now carries its owner's name. The 36 payouts are ones with no
|
||
client or location recorded anywhere, so there is nothing to name them after.</p>
|
||
<p>Renaming had to be driven from orders, because a payment's rightful owner is whichever
|
||
order refers to it — so completing it meant walking all <strong>19,040,296 orders</strong>, not
|
||
just the clients that look shared today. Nine client pairs contended in the past without
|
||
sharing a location now, and a migration scoped to the current configuration would have missed
|
||
every one of them.</p>
|
||
</div>
|
||
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Shared payments after the migration</th><th class="n">Count</th><th>Meaning</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><strong>Owned by two different clients</strong></td><td class="n good">0</td><td>the problem this work exists to solve — gone</td></tr>
|
||
<tr><td>Owned by two orders of the same client</td><td class="n">104</td><td>left shared deliberately, see below</td></tr>
|
||
<tr><td class="dim">sampled from</td><td class="n dim">250,000 orders</td><td class="dim">spread across the whole database</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div class="measure">
|
||
<p><strong>Why the 104 are left alone.</strong> They are cases where Square split one tender
|
||
across two of a single client's own orders. Copying the payment for each order would be wrong
|
||
twice over: both orders compute the same name, so there is no second name a copy could take —
|
||
and a copy would double that client's takings for the day. One payment covering both orders is
|
||
the truthful record.</p>
|
||
<p>The consequence is that the deletion cascade can still reach these — removing one of those
|
||
two orders would take the payment with it. That is why the guard on
|
||
<code>remove-voided-orders</code> is not optional housekeeping but a required companion to this
|
||
work: it should detach a shared payment rather than delete it, whoever the other order
|
||
belongs to.</p>
|
||
</div>
|
||
|
||
<div class="callout">
|
||
<span class="h">Re-running is safe, and that was proved at full scale</span>
|
||
<p>A second complete pass walked all 19,040,296 orders in <strong>7.8 minutes and changed
|
||
nothing</strong> — no renames, no copies. A record that already carries the right name is left
|
||
untouched, so the migration can be stopped, resumed, or repeated without consequence. Useful,
|
||
since the first full pass took hours on a machine where the 29 GB database no longer fitted in
|
||
memory; on warm cache the same code did 267,012 orders in 100 seconds.</p>
|
||
</div>
|
||
|
||
<div class="measure">
|
||
<p><strong>What follows from the gate reading zero.</strong> <code>unscoped-report</code> counts
|
||
these figures on demand. Now that unscoped is zero across the board, the importer's
|
||
understanding of the old name form can be removed — at which point two clients sharing a
|
||
location becomes structurally incapable of producing a shared record, rather than prevented by
|
||
a convention that a future import could quietly break. That removal is the one remaining step
|
||
of this piece of work.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>Decisions and risks still open</h2>
|
||
<div class="scroll">
|
||
<table>
|
||
<thead><tr><th>Item</th><th>Who decides</th><th>Why it matters</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Which client record survives at each shared location</td><td>the business</td><td>the newer record generally has no history before the split, so keeping it loses years of the location's books</td></tr>
|
||
<tr><td>Which revenue account service charges post to</td><td>accounting</td><td>currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances</td></tr>
|
||
<tr><td>Whether to correct records the wrong client already owns</td><td>the business</td><td>the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared</td></tr>
|
||
<tr><td><code>remove-voided-orders</code></td><td>engineering</td><td>safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<div class="callout warn">
|
||
<span class="h">Two operational findings, unrelated to the summaries</span>
|
||
<p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p>
|
||
<p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. A recompute was crawling at about 35 client-days a minute; after raising the cache to 8 GB the remaining 7,958 finished in 90 seconds. Worth checking what production is set to.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<h2>How to check any of this <span class="tech">technical</span></h2>
|
||
<div class="scroll">
|
||
<pre><span class="dim">;; the restored database, untouched production as of 2026-08-14 22:52</span>
|
||
(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore"))
|
||
|
||
<span class="dim">;; the two orders behind the worked examples</span>
|
||
(d/pull (d/db conn) '[*] [:sales-order/external-id
|
||
"square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"])
|
||
(d/pull (d/db conn) '[*] [:sales-order/external-id
|
||
"square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"])
|
||
|
||
<span class="dim">;; the gate: no payment may have two parent orders</span>
|
||
(rk/charges-with-multiple-parents (d/db conn) orders) <span class="dim">;; => 0</span>
|
||
|
||
<span class="dim">;; ownership history — which records ever changed client</span>
|
||
(->> (d/datoms (d/history (d/db conn)) :aevt :sales-refund/client)
|
||
(filter :added)
|
||
(reduce (fn [m d] (update m (:e d) (fnil conj #{}) (:v d))) {})
|
||
(filter (fn [[_ owners]] (> (count owners) 1)))
|
||
count)</pre>
|
||
</div>
|
||
<div class="measure">
|
||
<p>The comparison tool is committed as <code>auto-ap.jobs.compare-sales-summaries</code>, which diffs summaries between two points in the same database using <code>as-of</code> — so "before" is production's own stored figures rather than a re-simulation. Unit tests: <code>lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test</code>.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<footer>
|
||
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
|
||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding.</span>
|
||
</footer>
|
||
|
||
</div>
|