notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

docs(reports): rerun the ninety-day reconciliation from a fresh restore

Browse Source 
Rebuilt the whole analysis from nothing: fresh restore of backup point
209608347, deactivate the ten shared locations, re-key and split every
one of 19,040,785 orders, re-import from Square, then two full ninety-day
recomputes — one with the fixes off, one with them on.

The baseline is now a no-fix recompute rather than production's stored
summaries. That is the stricter comparison: production's figures are in
places months stale, and crediting the fixes with repairing ordinary
staleness flattered them. On the fairer footing the two arithmetic fixes
are worth 979 client-days and $61,769.56, taking the window from 1,258
days out of balance ($69,560.10) to 279 ($7,790.54), with zero days
knocked out of balance and zero already-balanced days altered at line
level.

The migration now runs to completion database-wide: 17,047,142 payments
scoped, nothing left to rename, nothing unscopable, and no payment owned
by more than one order across 400,000 orders checked. The earlier
"transactor-bound, two days" diagnosis was wrong — the bottleneck was GC
in the driving process; the full pass takes about thirteen minutes.

Also corrects compare-sales-summaries: :ledger-mapped/amount, ledger-side
and account are :db/noHistory, so as-of cannot recover past amounts and
a rewritten summary reads back as a false balanced day. Every figure in
the report comes from live captures taken straight after each pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Bryce
2026-08-15 18:35:47 -07:00
parent a16ef0bd60
commit 21e62d1a7c
2 changed files with 155 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

210
docs/2026-08-15-thirty-day-reconciliation.html
View File

@@ -92,11 +92,11 @@
<header class="masthead"> <header class="masthead">
<div class="eyebrow">Sales summaries · measured on a restored production backup</div> <div class="eyebrow">Sales summaries · measured on a restored production backup</div>
<h1>Ninety-Day Reconciliation</h1> <h1>Ninety-Day Reconciliation</h1>
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading.</p> <p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
<div class="meta"> <div class="meta">
<span><b>Window</b> 2026-05-10 → 2026-08-07</span> <span><b>Window</b> 2026-05-10 → 2026-08-07</span>
<span><b>Client-days</b> <span class="num">8,733</span></span> <span><b>Client-days</b> <span class="num">18,900</span></span>
<span><b>Clients</b> <span class="num">106</span></span> <span><b>Clients</b> <span class="num">210</span></span>
<span><b>Nothing in production was changed</b></span> <span><b>Nothing in production was changed</b></span>
</div> </div>
</header> </header>
@@ -104,27 +104,28 @@
<section> <section>
<div class="ledger"> <div class="ledger">
<div> <div>
<span class="side-label">Production today</span> <span class="side-label">Today's calculation, ninety days re-run</span>
<span class="figure num">$75,228.78</span> <span class="figure num">$69,560.10</span>
<span class="subfig"><span class="num">1,280</span> days out of balance · <span class="num">85.34%</span> clean</span> <span class="subfig"><span class="num">1,258</span> days out of balance · <span class="num">93.34%</span> clean</span>
</div> </div>
<div class="arrow" aria-hidden="true">→</div> <div class="arrow" aria-hidden="true">→</div>
<div> <div>
<span class="side-label">After all three fixes</span> <span class="side-label">The same ninety days, fixes on</span>
<span class="figure after num">$1,995.36</span> <span class="figure after num">$7,790.54</span>
<span class="subfig"><span class="num">108</span> days out of balance · <span class="num">98.76%</span> clean</span> <span class="subfig"><span class="num">279</span> days out of balance · <span class="num">98.52%</span> clean</span>
</div> </div>
</div> </div>
<div class="stats"> <div class="stats">
<div class="stat"><span class="k num">1,172</span><span class="l">client-days brought into balance</span></div> <div class="stat"><span class="k num">979</span><span class="l">client-days brought into balance</span></div>
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div> <div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
<div class="stat"><span class="k num">97.3%</span><span class="l">of the variance removed</span></div> <div class="stat"><span class="k num">88.8%</span><span class="l">of the variance removed</span></div>
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div> <div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
</div> </div>
<div class="measure"> <div class="measure">
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one day in seven it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p> <p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p>
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>979 days and $61,769.56</strong>, not the larger number a stale baseline would have shown.</p>
</div> </div>
</section> </section>
@@ -209,9 +210,12 @@
</div> </div>
<div class="measure"> <div class="measure">
<p>The result is pinned as the record's id on the way in, so the write lands on the existing <p>The result is pinned as the record's id on the way in, so the write lands on the existing
row regardless of which name it currently carries. The proof this worked: after re-naming row regardless of which name it currently carries. The proof this worked is a count that did
213,943 records, the totals for refunds, payouts and shifts were <strong>identical before and not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was
after</strong>. Had the fallback been missing they would have doubled.</p> re-named, and afterwards there were still exactly <strong>144,688 payouts and 69,291 cash-drawer
shifts</strong> — the same figures as at the restore point. Refunds went from 50,986 to 51,986,
and all 1,000 of those came from the live Square import run afterwards, not from the renaming.
Had the fallback lookup been missing, each of these would have doubled instead.</p>
</div> </div>
<h3>3 · Give every order its own payment record</h3> <h3>3 · Give every order its own payment record</h3>
@@ -230,10 +234,10 @@
link it to the copy instead</pre> link it to the copy instead</pre>
</div> </div>
<div class="measure"> <div class="measure">
<p>On the restored data that was <strong>189,167 renamed and 77,599 copied</strong>, and payments <p>Run over the whole database that was <strong>9,100,314 renamed and 200,027 copied</strong>,
owned by two orders went from 11,469 to zero. The record count rose by exactly 77,599 — the and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across
number of copies it reported making, which is the check that it created what it meant to and 400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it
nothing else.</p> reported making, which is the check that it created what it meant to and nothing else.</p>
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the <p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes — record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
<span class="mono">N-30003</span> — so a pattern cannot tell where the client name ends and the <span class="mono">N-30003</span> — so a pattern cannot tell where the client name ends and the
@@ -315,36 +319,37 @@
<section> <section>
<h2>What each fix is worth</h2> <h2>What each fix is worth</h2>
<div class="measure"> <div class="measure">
<p>The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates.</p> <p>The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates. All 18,900 client-day summaries in the window are included, whether or not the restaurant traded that day.</p>
</div> </div>
<div class="scroll"> <div class="scroll">
<table> <table>
<thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead> <thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead>
<tbody> <tbody>
<tr><td>Production today</td><td class="n">1,280</td><td class="n">85.34%</td><td class="n">$75,228.78</td></tr> <tr><td>Today's calculation, ninety days re-run</td><td class="n">1,258</td><td class="n">93.34%</td><td class="n">$69,560.10</td></tr>
<tr><td>+ deduplication</td><td class="n">1,087</td><td class="n">87.55%</td><td class="n">$63,764.92</td></tr> <tr><td>+ refunded tips</td><td class="n">971</td><td class="n">94.86%</td><td class="n">$66,414.39</td></tr>
<tr><td>+ refunded tips</td><td class="n">800</td><td class="n">90.84%</td><td class="n">$60,619.21</td></tr> <tr class="total"><td>+ service charges</td><td class="n good">279</td><td class="n good">98.52%</td><td class="n good">$7,790.54</td></tr>
<tr class="total"><td>+ service charges</td><td class="n good">108</td><td class="n good">98.76%</td><td class="n good">$1,995.36</td></tr>
</tbody> </tbody>
</table> </table>
</div> </div>
<div class="measure">
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$61,769.56 is what the two arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
</div>
<h3>Day-by-day effect of each change</h3> <h3>Day-by-day effect of each change</h3>
<div class="scroll"> <div class="scroll">
<table> <table>
<thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead> <thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead>
<tbody> <tbody>
<tr><td>Deduplication</td><td class="n">7,447 <span class="dim">balanced already</span></td><td class="n good">199</td><td class="n bad">6</td><td class="n dim">—</td><td class="n dim">—</td></tr> <tr><td>Refunded tips</td><td class="n">18,590</td><td class="n good">287</td><td class="n good">0</td><td class="n good">0</td><td class="n">$3,712.67</td></tr>
<tr><td>Refunded tips</td><td class="n">8,421</td><td class="n good">287</td><td class="n good">0</td><td class="n good">0</td><td class="n">$3,777.67</td></tr> <tr><td>Service charges</td><td class="n">18,208</td><td class="n good">692</td><td class="n good">0</td><td class="n good">0</td><td class="n">$58,349.85</td></tr>
<tr><td>Service charges</td><td class="n">8,037</td><td class="n good">692</td><td class="n good">0</td><td class="n good">0</td><td class="n">$58,923.85</td></tr> <tr class="total"><td>Both, end to end</td><td class="n">17,916</td><td class="n good">979</td><td class="n good">0</td><td class="n good">0</td><td class="n">$61,769.56</td></tr>
<tr class="total"><td>All three, end to end</td><td class="n">7,453</td><td class="n good">1,172</td><td class="n good">0</td><td class="n good">0</td><td class="n">—</td></tr>
</tbody> </tbody>
</table> </table>
</div> </div>
<div class="measure"> <div class="measure">
<p><strong>Neither arithmetic fix touched a day that was already correct.</strong> Across 8,733 client-days, no balanced day was knocked out of balance and no balanced day had a single figure altered — every day they changed was already wrong. Service charges are by far the larger of the two, moving $58,923.85 against the tip fix's $3,777.67.</p> <p><strong>Neither fix touched a day that was already correct.</strong> Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered — 17,916 summaries came out byte-identical, and every one of the 984 that moved was already wrong. Service charges are by far the larger of the two, moving $58,349.85 against the tip fix's $3,712.67.</p>
<p>Deduplication is the one step that puts six days <em>out</em> of balance. That is expected: those days were previously balanced only because a summary was empty, and filling it in exposes the same arithmetic faults every other day had. Both later fixes then close them, which is why the end-to-end figure is zero.</p> <p>That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.</p>
<p>Excluding the ten now-deactivated duplicate records, which should not be reported on at all, the population is 8,350 client-days and the movement is <strong>1,217 days and $69,995.61 down to 105 days and $852.38</strong>.</p> <p><strong>The two fixes account for the repair exactly.</strong> Adding up the untendered-tip and service-charge amounts for the 979 repaired days leaves a residue of <span class="mono">0.0000000013</span> against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a third effect.</p>
</div> </div>
</section> </section>
@@ -414,8 +419,9 @@
<thead><tr><th>Step</th><th>Result</th></tr></thead> <thead><tr><th>Step</th><th>Result</th></tr></thead>
<tbody> <tbody>
<tr><td>Deactivate the duplicate client at each shared location</td><td class="n">10 locations · shared locations remaining: <span class="good">0</span></td></tr> <tr><td>Deactivate the duplicate client at each shared location</td><td class="n">10 locations · shared locations remaining: <span class="good">0</span></td></tr>
<tr><td>Give every order its own payment record</td><td class="n">189,167 re-keyed · 77,599 copied</td></tr> <tr><td>Walk every order in the database</td><td class="n">19,040,785 orders</td></tr>
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">11,469 → 0</td></tr> <tr><td>Give every order its own payment record</td><td class="n">9,100,314 re-keyed · 200,027 copied</td></tr>
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across 400,000 orders checked</span></td></tr>
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr> <tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr> <tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr> <tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
@@ -423,7 +429,12 @@
</table> </table>
</div> </div>
<div class="measure"> <div class="measure">
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 77,599, matching the number of copies it reported making.</p> <p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.</p>
</div>
<div class="callout">
<span class="h">The whole analysis was run again from nothing, and landed in the same place</span>
<p>Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. The end state matched the previous run <strong>to the cent</strong> — 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.</p>
</div> </div>
<div class="callout warn"> <div class="callout warn">
@@ -435,25 +446,36 @@
<section> <section>
<h2>What is still out of balance</h2> <h2>What is still out of balance</h2>
<div class="measure"> <div class="measure">
<p>Fifteen client-days across ninety days and 8,350 client-days, totalling $850.12. Everything else — 90 further days — comes to <strong>$2.26</strong>, the largest being 9.00¢, which is till rounding.</p> <p>279 client-days out of 18,900, totalling <strong>$7,790.54</strong>. Where it sits matters more than the total, because most of it is not on a restaurant anyone reports on.</p>
</div> </div>
<div class="scroll"> <div class="scroll">
<table> <table>
<thead><tr><th>Client</th><th>Date</th><th class="n">Variance</th><th>What it is</th></tr></thead> <thead><tr><th>Where the remainder sits</th><th class="n">Days</th><th class="n">Variance</th></tr></thead>
<tbody> <tbody>
<tr><td class="mono">NGBK</td><td>2026-08-06</td><td class="n">+299.42</td><td>Square recorded more payment than the orders account for</td></tr> <tr><td>The twenty client records tied to a shared location — including the ten now-dormant duplicates</td><td class="n">160</td><td class="n">$6,059.93</td></tr>
<tr><td class="mono">NGMV</td><td>May 20–26</td><td class="n">+259.38</td><td>five days, undiagnosed</td></tr> <tr class="total"><td>Every other client</td><td class="n">119</td><td class="n">$1,730.61</td></tr>
<tr><td class="mono">NGEB</td><td>May 13 – Jul 29</td><td class="n">−199.09</td><td>four days, ezCater fee treatment — an open question</td></tr>
<tr><td class="mono">NGDA</td><td>2026-08-01</td><td class="n">−50.00</td><td>auto-gratuity recorded as a service charge</td></tr>
<tr><td class="mono">N-30012</td><td>May 20–21</td><td class="n">+30.31</td><td>two days, undiagnosed</td></tr>
<tr><td class="mono">NG4S</td><td>2026-05-29</td><td class="n">−11.78</td><td>undiagnosed</td></tr>
<tr><td class="mono">PNSP</td><td>2026-07-12</td><td class="n">−0.14</td><td>till rounding, just over the threshold</td></tr>
</tbody> </tbody>
</table> </table>
</div> </div>
<div class="measure"> <div class="measure">
<p>The ten deactivated duplicate records contribute a further $1,142.98 across three days. They are excluded above and should be excluded from reporting generally, since they are now dormant copies.</p> <p>Of that $1,730.61 on ordinary clients, four account for all but $2.40 of it:</p>
<p>Two clusters — NGMV and N-30012, both in late May — are new and unexplained. They only became visible because the window was widened to ninety days; a thirty-day view did not reach them. They are worth a look before this ships.</p> </div>
<div class="scroll">
<table>
<thead><tr><th>Client</th><th class="n">Days</th><th class="n">Variance</th><th>What it is</th></tr></thead>
<tbody>
<tr><td class="mono">NG4S</td><td class="n">10</td><td class="n">$1,066.61</td><td>undiagnosed — the largest single unexplained cluster</td></tr>
<tr><td class="mono">NGMV</td><td class="n">5</td><td class="n">$259.38</td><td>late May, undiagnosed</td></tr>
<tr><td class="mono">NGEB</td><td class="n">4</td><td class="n">$199.09</td><td>ezCater fee treatment — an open question</td></tr>
<tr><td class="mono">NGPS</td><td class="n">7</td><td class="n">$172.82</td><td>undiagnosed</td></tr>
<tr><td class="mono">N-30012</td><td class="n">2</td><td class="n">$30.31</td><td>late May, undiagnosed</td></tr>
<tr class="total"><td class="dim">PNSP, PNLP, GLAD and others</td><td class="n dim">91</td><td class="n dim">$2.40</td><td class="dim">till rounding — pennies a day</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>The largest surviving single day is NGDL on 2026-08-01 at −$810.24, one of the shared-location records. The matched pair NGBR and NGBK both sit at exactly ±$299.42 on 2026-08-06: Square recorded $6,358.99 of tender against $6,059.57 of order totals, and the difference is that gap rather than anything the summary did wrong. The tip fix improved that day — it was $436.42 before — without closing it.</p>
<p>The clusters on NG4S, NGMV, NGPS and N-30012 are unexplained and worth a look before this ships. They are visible only because the window is ninety days; a thirty-day view does not reach them.</p>
</div> </div>
</section> </section>
@@ -467,55 +489,74 @@
</div> </div>
<div class="scroll"> <div class="scroll">
<table> <table>
<thead><tr><th>Record type</th><th class="n">Client-scoped</th><th class="n">Still unscoped</th><th class="n">No owner to scope by</th></tr></thead> <thead><tr><th>Record type</th><th class="n">Total</th><th class="n">Client-scoped</th><th class="n">Still to rename</th><th class="n">Cannot be scoped</th></tr></thead>
<tbody> <tbody>
<tr><td>Card payments</td><td class="n good">17,046,418</td><td class="n good">0</td><td class="n good">0</td></tr> <tr><td>Card payments</td><td class="n">17,047,142</td><td class="n good">17,047,142</td><td class="n good">0</td><td class="n good">0</td></tr>
<tr><td>Refunds</td><td class="n good">51,986</td><td class="n good">0</td><td class="n good">0</td></tr> <tr><td>Refunds</td><td class="n">51,986</td><td class="n good">51,986</td><td class="n good">0</td><td class="n good">0</td></tr>
<tr><td>Payouts</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr> <tr><td>Payouts</td><td class="n">144,688</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr>
<tr><td>Cash-drawer shifts</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr> <tr><td>Cash-drawer shifts</td><td class="n">69,291</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr>
</tbody> </tbody>
</table> </table>
</div> </div>
<div class="measure"> <div class="measure">
<p>Every record in the database now carries its owner's name. The 36 payouts are ones with no <p>Every record in the database now carries its owner's name, and the migration proposes no
client or location recorded anywhere, so there is nothing to name them after.</p> further changes: asked what is left to do, it answers zero on all four record types. The 36
payouts are ones with no client or location recorded anywhere, on the record itself or on
anything referring to it, so there is nothing to name them after.</p>
<p>Renaming had to be driven from orders, because a payment's rightful owner is whichever <p>Renaming had to be driven from orders, because a payment's rightful owner is whichever
order refers to it — so completing it meant walking all <strong>19,040,296 orders</strong>, not order refers to it — so completing it meant walking all <strong>19,040,785 orders</strong>, not
just the clients that look shared today. Nine client pairs contended in the past without just the clients that look shared today. Nine client pairs contended in the past without
sharing a location now, and a migration scoped to the current configuration would have missed sharing a location now, and a migration scoped to the current configuration would have missed
every one of them.</p> every one of them.</p>
<p><strong>A caution about how completeness is counted.</strong> 282,649 card payments carry no
client attribute of their own — they are stubs the payout path creates, never referenced by an
order. A gate that checks the attribute reports these as "no owner" and looks like a gap. They
are not: their names are scoped, recovered from the deposit that holds them. The figures above
are counted the harder way, by asking the migration what it would still change, which resolves
each record's owner through whatever refers to it. Reading the attribute alone would have
understated completeness by a quarter of a million records — and an early draft of this report
did exactly that.</p>
</div> </div>
<div class="scroll"> <div class="scroll">
<table> <table>
<thead><tr><th>Shared payments after the migration</th><th class="n">Count</th><th>Meaning</th></tr></thead> <thead><tr><th>Shared payments after the migration</th><th class="n">Count</th><th>Meaning</th></tr></thead>
<tbody> <tbody>
<tr><td><strong>Owned by two different clients</strong></td><td class="n good">0</td><td>the problem this work exists to solve — gone</td></tr> <tr><td><strong>Owned by more than one order</strong></td><td class="n good">0</td><td>whether the orders belong to different clients or the same one</td></tr>
<tr><td>Owned by two orders of the same client</td><td class="n">104</td><td>left shared deliberately, see below</td></tr> <tr><td class="dim">checked across</td><td class="n dim">400,000 orders</td><td class="dim">spread through the whole database</td></tr>
<tr><td class="dim">sampled from</td><td class="n dim">250,000 orders</td><td class="dim">spread across the whole database</td></tr>
</tbody> </tbody>
</table> </table>
</div> </div>
<div class="measure"> <div class="measure">
<p><strong>Why the 104 are left alone.</strong> They are cases where Square split one tender <p>The problem this work exists to solve is gone: no payment answers to two orders, so the
across two of a single client's own orders. Copying the payment for each order would be wrong component relationship means what it says and deleting an order can no longer take another
twice over: both orders compute the same name, so there is no second name a copy could take — order's money with it.</p>
and a copy would double that client's takings for the day. One payment covering both orders is <p><strong>One behaviour changed when the pass was run over everything, and it is worth
the truthful record.</p> recording.</strong> Where Square splits one tender across two of a single client's own orders,
<p>The consequence is that the deletion cascade can still reach these — removing one of those the earlier design left the payment shared on purpose — both orders compute the same name, so
two orders would take the payment with it. That is why the guard on there is no second name for a copy to take. That rule only holds for two orders processed in
<code>remove-voided-orders</code> is not optional housekeeping but a required companion to this the same batch. Run across nineteen million orders in batches of two thousand, such pairs
work: it should detach a shared payment rather than delete it, whoever the other order almost always fall in different batches and the second order now takes a copy. Inside the
belongs to.</p> ninety-day window this changed nothing measurable: the recompute after the database-wide pass
matched the one before it to the cent. Outside the window it has not been measured, and it
should be before this runs against production.</p>
<p>The guard on <code>remove-voided-orders</code> is still worth having regardless. It is
cheap, and it makes the safety a property of the deletion rather than of the migration having
been run first.</p>
</div> </div>
<div class="callout"> <div class="callout">
<span class="h">Re-running is safe, and that was proved at full scale</span> <span class="h">Re-running is safe, and that was proved at full scale</span>
<p>A second complete pass walked all 19,040,296 orders in <strong>7.8 minutes and changed <p>After the complete pass, asking the migration what it would change next returns
nothing</strong> — no renames, no copies. A record that already carries the right name is left <strong>nothing</strong> — 17,047,142 payments examined, none to rename, none unscopable. A
untouched, so the migration can be stopped, resumed, or repeated without consequence. Useful, record that already carries the right name is left untouched, so the migration can be stopped,
since the first full pass took hours on a machine where the 29 GB database no longer fitted in resumed, or repeated without consequence.</p>
memory; on warm cache the same code did 267,012 orders in 100 seconds.</p> <p>Its speed is worth a note for whoever schedules it: the whole nineteen million orders were
walked in about <strong>thirteen minutes</strong>. An earlier attempt appeared to be
transactor-bound and was projected at two days, which is why a previous run narrowed it to the
analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process
driving the migration — freeing held memory took an unrelated recompute from 17 client-days a
minute to 4,515. Nothing about the database or the transactor needed to change.</p>
</div> </div>
<div class="measure"> <div class="measure">
@@ -545,7 +586,8 @@
<div class="callout warn"> <div class="callout warn">
<span class="h">Two operational findings, unrelated to the summaries</span> <span class="h">Two operational findings, unrelated to the summaries</span>
<p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p> <p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p>
<p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. A recompute was crawling at about 35 client-days a minute; after raising the cache to 8 GB the remaining 7,958 finished in 90 seconds. Worth checking what production is set to.</p> <p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. Worth checking what production is set to.</p>
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in thirteen minutes. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
</div> </div>
</section> </section>
@@ -572,13 +614,31 @@
count)</pre> count)</pre>
</div> </div>
<div class="measure"> <div class="measure">
<p>The comparison tool is committed as <code>auto-ap.jobs.compare-sales-summaries</code>, which diffs summaries between two points in the same database using <code>as-of</code> — so "before" is production's own stored figures rather than a re-simulation. Unit tests: <code>lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test</code>.</p> <p>The comparison tool is committed as <code>auto-ap.jobs.compare-sales-summaries</code>. Unit tests: <code>lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test</code>.</p>
</div>
<div class="callout warn">
<span class="h">Do not compare summaries with <code>as-of</code> — a correction to how this was measured</span>
<p>The obvious way to audit a recompute is to read the database at a point before it and diff:
Datomic keeps every past value, so no snapshot is needed. That is what
<code>compare-sales-summaries</code> was built to do, and for summary amounts it does not work.
<code>:ledger-mapped/amount</code>, <code>:ledger-mapped/ledger-side</code> and
<code>:ledger-mapped/account</code> are all declared <code>:db/noHistory true</code>, so
superseded values are discarded rather than retained. A historical read of a summary that has
since been recomputed can return its lines with the categories intact and the amounts simply
absent — which reads as a legitimate all-zero summary, not as an error.</p>
<p>Every figure in this report is therefore taken from a live read of the database immediately
after each pass, captured and stored outside it, and the before/after comparison is done
between those two captures. No historical read is involved anywhere in the numbers above. The
tool remains useful for categories and for which days changed; its docstring overstates what it
can recover, and that is worth correcting before someone relies on it for amounts.</p>
</div> </div>
</section> </section>
<footer> <footer>
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span> <span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding.</span> <span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 279 remaining days, 189 are material.</span>
<span>Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on <code>as-of</code> above.</span>
</footer> </footer>
</div> </div>

25
src/clj/auto_ap/jobs/compare_sales_summaries.clj
View File

@@ -1,14 +1,25 @@
(ns auto-ap.jobs.compare-sales-summaries (ns auto-ap.jobs.compare-sales-summaries
"Compares sales summaries between two points in the same database. "Compares sales summaries between two points in the same database.
Datomic keeps every past value, so a recompute can be audited against exactly what was there
before by reading `(d/as-of db t)` for some earlier `t` — no snapshot or scratch copy needed.
The question this exists to answer is narrower than \"did the totals improve\": it is *which The question this exists to answer is narrower than \"did the totals improve\": it is *which
days changed, and were any of them already balanced*. A day that was balanced before and still days changed, and were any of them already balanced*. A day that was balanced before and still
balances after can still have had its line amounts move, and that is a real change to the balances after can still have had its line amounts move, and that is a real change to the
books even though no red turns green. Counting only balanced/unbalanced transitions would hide books even though no red turns green. Counting only balanced/unbalanced transitions would hide
it entirely." it entirely.
DO NOT USE `compare-against` — OR ANY `d/as-of` DATABASE — TO COMPARE AMOUNTS. The obvious
reading is that Datomic keeps every past value, so a recompute can be audited against what was
there before with no snapshot. That does not hold here: `:ledger-mapped/amount`,
`:ledger-mapped/ledger-side` and `:ledger-mapped/account` are all `:db/noHistory true`, so
superseded values are discarded rather than retained. A summary that has since been recomputed
reads back through `as-of` with its categories intact and its amounts *absent* — which is
indistinguishable from a legitimate all-zero day, and quietly turns every rewritten summary
into a false \"was balanced, still balances\".
To compare amounts, capture `summaries-in` from a live `(d/db conn)` immediately after each
run, keep the two captures outside the database, and diff those. `compare-window` is safe when
both arguments are live database values; only the historical read is unsound. Categories and
which-days-changed do survive `as-of`, since `:sales-summary-item/category` retains history."
(:require (:require
[auto-ap.datomic :refer [conn]] [auto-ap.datomic :refer [conn]]
[auto-ap.datomic.sales-summaries :as d-ss] [auto-ap.datomic.sales-summaries :as d-ss]
@@ -119,7 +130,11 @@
(into {})))) (into {}))))
(defn compare-against (defn compare-against
"Convenience: compare the current database against its own past value at basis `t`." "Compare the current database against its own past value at basis `t`.
UNSOUND FOR AMOUNTS — see the namespace docstring. The amount, side and account attributes are
`:db/noHistory`, so any summary rewritten since `t` reads back with no amounts and appears
balanced. Kept only for comparing categories and identifying which days changed."
[t start end] [t start end]
(let [db (dc/db conn)] (let [db (dc/db conn)]
(compare-window (dc/as-of db t) db start end))) (compare-window (dc/as-of db t) db start end)))