Compare commits
1 Commits
sales-summ
...
sales-summ
| Author | SHA1 | Date | |
|---|---|---|---|
| f8ef7918ef |
@@ -92,7 +92,7 @@
|
||||
<header class="masthead">
|
||||
<div class="eyebrow">Sales summaries · measured on a restored production backup</div>
|
||||
<h1>Ninety-Day Reconciliation</h1>
|
||||
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — and a fourth, found late, that is a missing-data problem wearing a balancing problem's clothes. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
|
||||
<p class="standfirst">Four faults were leaving restaurant days out of balance — one in the data, three in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
|
||||
<div class="meta">
|
||||
<span><b>Window</b> 2026-05-10 → 2026-08-07</span>
|
||||
<span><b>Client-days</b> <span class="num">18,900</span></span>
|
||||
@@ -105,28 +105,27 @@
|
||||
<div class="ledger">
|
||||
<div>
|
||||
<span class="side-label">Today's calculation, ninety days re-run</span>
|
||||
<span class="figure num">$70,276.50</span>
|
||||
<span class="subfig"><span class="num">1,191</span> days out of balance · <span class="num">93.70%</span> clean</span>
|
||||
<span class="figure num">$69,560.10</span>
|
||||
<span class="subfig"><span class="num">1,258</span> days out of balance · <span class="num">93.34%</span> clean</span>
|
||||
</div>
|
||||
<div class="arrow" aria-hidden="true">→</div>
|
||||
<div>
|
||||
<span class="side-label">The same ninety days, fixes on</span>
|
||||
<span class="figure after num">$2,379.45</span>
|
||||
<span class="subfig"><span class="num">122</span> days out of balance · <span class="num">99.35%</span> clean</span>
|
||||
<span class="figure after num">$2,970.35</span>
|
||||
<span class="subfig"><span class="num">123</span> days out of balance · <span class="num">99.35%</span> clean</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="stats">
|
||||
<div class="stat"><span class="k num">1,069</span><span class="l">client-days brought into balance</span></div>
|
||||
<div class="stat"><span class="k num">1,135</span><span class="l">client-days brought into balance</span></div>
|
||||
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
|
||||
<div class="stat"><span class="k num">96.6%</span><span class="l">of the variance removed</span></div>
|
||||
<div class="stat"><span class="k num">95.7%</span><span class="l">of the variance removed</span></div>
|
||||
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
|
||||
</div>
|
||||
|
||||
<div class="measure">
|
||||
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p>
|
||||
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>1,069 days and $67,897.05</strong>, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.</p>
|
||||
<p><strong>Most of what is left is not a balancing fault at all</strong>, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.</p>
|
||||
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, service charges customers paid were credited to nothing, and a refund that arrived on a day with no sales had nothing to offset it.</p>
|
||||
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>1,135 days and $66,589.75</strong>, not the larger number a stale baseline would have shown.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -163,14 +162,12 @@
|
||||
</div>
|
||||
|
||||
<div class="problem">
|
||||
<h4>4. Refunds on records whose sales were never imported <span class="tech">not fixed — deliberately</span></h4>
|
||||
<h4>4. A refund on a day the restaurant did not sell anything</h4>
|
||||
<div class="measure">
|
||||
<p><strong>This is why the duplicated restaurants looked so much worse than everyone else.</strong> Of the days still failing after the first three fixes, <strong>155 of the 423 on shared-location records had no sales orders at all</strong> — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.</p>
|
||||
<p>The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded <em>any</em> order shows <strong>140 of 171 fall before that client had a single order in the system</strong> — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.</p>
|
||||
<p><strong>Where the refunds came from.</strong> Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to <span class="mono">NGDG</span> that same day, and was taken over by <span class="mono">NGDU</span> on 12 August. Others flip between the two records several times a day across 12–15 August. <span class="mono">NGDU</span>'s first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.</p>
|
||||
<p>Across the nine records, <strong>660 refunds worth $15,237.02 sit on a record dated before that record's first order.</strong> Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.</p>
|
||||
<p><strong>Why it is deliberately left out of balance.</strong> The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and on this data it closes 16 of the 122 remaining days and $1,227.65. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.</p>
|
||||
<p><span class="tech">technical</span> <code>get-returns</code> sums <code>:sales-order/returns</code> over orders scanned for the date. With no orders the sum is nil and no <code>Returns</code> line is written, while <code>get-refund-items</code> still credits <code>Card Refunds</code> from the <code>sales-refund</code> records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.</p>
|
||||
<p><strong>For the business:</strong> a refund is recorded on the day the money goes back to the guest. The matching reduction in sales is read from that day's orders. When both fall on the same day they cancel and the day balances. When they do not — a refund settling after a closed day, or a retired duplicate record that still receives refunds while its orders belong to the surviving record — the day is left carrying a refund with nothing against it, and is out by exactly the refunded amount.</p>
|
||||
<p><strong>This is the answer to why the duplicated restaurants looked so much worse than everyone else.</strong> Of the days still failing after the first three fixes, <strong>155 of the 158 on duplicate records had no sales orders at all</strong> — the summary consisted of nothing but orphaned refunds and their fees. The refunds were claimed by the record that happened to import them during the shared period; the orders went to the other record. Nothing was double-counted and no money was lost — the two halves of each refund simply ended up on two different sets of books.</p>
|
||||
<p>It is not a duplicate-only problem, which is what makes it worth fixing properly rather than papering over. The single largest non-duplicate cluster, <span class="mono">NG4S</span>, has the identical shape: no orders on any day of the window, refunds arriving anyway, and every one of those days out by exactly the refund.</p>
|
||||
<p><span class="tech">technical</span> <code>get-returns</code> summed <code>:sales-order/returns</code> over orders scanned for the date. With no orders the sum is nil and no <code>Returns</code> line is written at all, while <code>get-refund-items</code> still credits <code>Card Refunds</code> from the <code>sales-refund</code> records — a credit with no debit anywhere in the day.</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
@@ -178,10 +175,9 @@
|
||||
<section>
|
||||
<h2>What the fixes actually are</h2>
|
||||
<div class="measure">
|
||||
<p>Five changes. The first three stop two clients from sharing a record; the last two record
|
||||
money that was being collected but not booked. Each is small — the difficulty was knowing
|
||||
which line to change, not writing it. A sixth was written and then removed; it is described
|
||||
at the end because the reasoning matters more than the code did.</p>
|
||||
<p>Six changes. The first three stop two clients from sharing a record; the last three record
|
||||
money that was being collected, or handed back, but not booked. Each is small — the difficulty
|
||||
was knowing which line to change, not writing it.</p>
|
||||
</div>
|
||||
|
||||
<h3>1 · Put the client in the record's name</h3>
|
||||
@@ -226,20 +222,10 @@
|
||||
<p>The result is pinned as the record's id on the way in, so the write lands on the existing
|
||||
row regardless of which name it currently carries. The proof this worked is a count that did
|
||||
not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was
|
||||
re-named, and afterwards there were still exactly <strong>50,986 refunds, 144,688 payouts and
|
||||
69,291 cash-drawer shifts</strong> — the same three figures as at the restore point.
|
||||
re-named, and afterwards there were still exactly <strong>144,688 payouts and 69,291 cash-drawer
|
||||
shifts</strong> — the same figures as at the restore point. Refunds went from 50,986 to 51,986,
|
||||
and all 1,000 of those came from the live Square import run afterwards, not from the renaming.
|
||||
Had the fallback lookup been missing, each of these would have doubled instead.</p>
|
||||
<p><strong>The fallback also has to refuse.</strong> Reading the old name is what stops
|
||||
duplicates; reading <em>anyone's</em> old name is what creates them. Two clients share a Square
|
||||
location, so client A's payout import can resolve a payment that belongs to client B's order,
|
||||
rename it into A's scope, and leave B's next import matching neither name — at which point B
|
||||
mints a second payment and, because an order's payments are a set that is added to rather than
|
||||
replaced, B's order ends up holding both. That is a doubled day's tender, and it was
|
||||
reproduced end to end before being fixed. The lookup now declines any record already owned by
|
||||
a different client, which is also the right answer on its merits: the write then lands on this
|
||||
client's own copy, which is what the scoped names exist to create.</p>
|
||||
<p>This is transitional. Once no legacy names remain, the fallback and the refusal are deleted
|
||||
together and the guarantee stops depending on either.</p>
|
||||
</div>
|
||||
|
||||
<h3>3 · Give every order its own payment record</h3>
|
||||
@@ -258,9 +244,9 @@
|
||||
link it to the copy instead</pre>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>Run over the whole database that was <strong>16,236,839 renamed and 500,438 copied</strong>,
|
||||
<p>Run over the whole database that was <strong>9,100,314 renamed and 200,027 copied</strong>,
|
||||
and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across
|
||||
every order of the last year. The record count rose by about 500,438 — the number of copies it
|
||||
400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it
|
||||
reported making, which is the check that it created what it meant to and nothing else.</p>
|
||||
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
|
||||
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
|
||||
@@ -319,26 +305,39 @@
|
||||
few restaurants at a time.</p>
|
||||
</div>
|
||||
|
||||
<h3>6 · The change that was written, measured, and then taken out</h3>
|
||||
<h3>6 · Recognise a return on a day that refunded but did not sell</h3>
|
||||
<div class="measure">
|
||||
<p>Worth recording, because the arithmetic case for it is good and someone will propose it
|
||||
again. Where a day has refunds and no sales orders whatsoever, book a <code>Returns</code>
|
||||
debit equal to that day's refunds:</p>
|
||||
<p>Where a day has refunds and no sales orders whatsoever, book a <code>Returns</code> debit
|
||||
equal to that day's refunds. The guard — <em>no orders at all</em> — is what makes this
|
||||
mechanical rather than a judgement call.</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<pre>(defn- refund-only-returns [c date]
|
||||
(when-not (traded? c date)
|
||||
<pre><span class="dim">;; the return a day owes when it refunded but never sold</span>
|
||||
(defn- refund-only-returns [c date]
|
||||
(when (and (refund-only-returns-enabled? c)
|
||||
(not (traded? c date)))
|
||||
(let [amount (refunded-total c date)]
|
||||
(when-not (zero? amount) amount))))</pre>
|
||||
(when-not (zero? amount) amount))))
|
||||
|
||||
<span class="dim">;; get-returns falls back to it only when the order-derived figure is absent</span>
|
||||
(or (ffirst (dc/q <span class="dim">;; sum of :sales-order/returns, as before</span>))
|
||||
(refund-only-returns c date))</pre>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>It works. Measured over the same ninety days it closed <strong>171 days and $5,795.18</strong>,
|
||||
knocked nothing out of balance, and altered no already-balanced day — the guard makes it
|
||||
incapable of touching a day that traded.</p>
|
||||
<p>It was removed anyway. Those days are not quiet days; they are days whose sales were never
|
||||
imported, and closing them removes the only visible sign of that. What is left in the code is
|
||||
a comment saying so and a test asserting the day <em>stays</em> out of balance, so the next
|
||||
person to notice the arithmetic finds the reasoning before they find the fix.</p>
|
||||
<p><strong>Why the guard is the whole design.</strong> On a day with no orders the
|
||||
order-derived return is necessarily nothing, so there is no figure to double-count, and no
|
||||
trading day can have its numbers moved by this rule — which the measurement bears out: 156
|
||||
days came into balance, none went out, and not one already-balanced day had a line change.
|
||||
Days that <em>did</em> trade and still carry an unmatched refund are deliberately left alone;
|
||||
apportioning a return across a day that also sold is an accounting question, not a mechanical
|
||||
one, and guessing at it would be exactly the kind of change that quietly rewrites correct
|
||||
books.</p>
|
||||
<p>Like the service-charge line, it sits behind a per-client switch that is off by default.
|
||||
Four tests hold it: flag off changes nothing; a refund-only day balances exactly; a day that
|
||||
traded keeps its own return; and a day that traded but returned nothing does not pick up the
|
||||
refunded total either — that last one guards the difference between "the client did not
|
||||
trade" and "the order-derived figure happened to be nil", which is the subtle way this rule
|
||||
could have gone wrong.</p>
|
||||
</div>
|
||||
|
||||
<h3>Supporting changes</h3>
|
||||
@@ -371,9 +370,10 @@
|
||||
<table>
|
||||
<thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,191</td><td class="n">93.70%</td><td class="n">$70,276.50</td></tr>
|
||||
<tr><td>+ refunded tips</td><td class="n">890</td><td class="n">95.29%</td><td class="n">$67,032.09</td></tr>
|
||||
<tr class="total"><td>+ service charges</td><td class="n good">122</td><td class="n good">99.35%</td><td class="n good">$2,379.45</td></tr>
|
||||
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,258</td><td class="n">93.34%</td><td class="n">$69,560.10</td></tr>
|
||||
<tr><td>+ refunded tips</td><td class="n">971</td><td class="n">94.86%</td><td class="n">$66,414.39</td></tr>
|
||||
<tr><td>+ service charges</td><td class="n">279</td><td class="n">98.52%</td><td class="n">$7,790.54</td></tr>
|
||||
<tr class="total"><td>+ returns on refund-only days</td><td class="n good">123</td><td class="n good">99.35%</td><td class="n good">$2,970.35</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
@@ -386,26 +386,17 @@
|
||||
<table>
|
||||
<thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Refunded tips</td><td class="n">18,571</td><td class="n good">301</td><td class="n good">0</td><td class="n good">0</td><td class="n">$4,027.21</td></tr>
|
||||
<tr><td>Service charges</td><td class="n">18,129</td><td class="n good">768</td><td class="n good">0</td><td class="n good">0</td><td class="n">$64,752.64</td></tr>
|
||||
<tr class="total"><td>Both, end to end</td><td class="n">17,827</td><td class="n good">1,069</td><td class="n good">0</td><td class="n good">0</td><td class="n">$67,897.05</td></tr>
|
||||
<tr><td>Refunded tips</td><td class="n">18,590</td><td class="n good">287</td><td class="n good">0</td><td class="n good">0</td><td class="n">$3,712.67</td></tr>
|
||||
<tr><td>Service charges</td><td class="n">18,208</td><td class="n good">692</td><td class="n good">0</td><td class="n good">0</td><td class="n">$58,349.85</td></tr>
|
||||
<tr><td>Returns on refund-only days</td><td class="n">18,740</td><td class="n good">156</td><td class="n good">0</td><td class="n good">0</td><td class="n">$4,820.19</td></tr>
|
||||
<tr class="total"><td>All three, end to end</td><td class="n">17,765</td><td class="n good">1,135</td><td class="n good">0</td><td class="n good">0</td><td class="n">$66,589.75</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>Neither fix touched a day that was already correct.</strong> Across all 18,900
|
||||
client-days no balanced day was knocked out of balance, and no balanced day had a single
|
||||
figure altered — 17,827 summaries came out byte-identical, and every one of the 1,073 that
|
||||
moved was already wrong.</p>
|
||||
<p>That claim did not hold on the first attempt, and how it was recovered is the useful part.
|
||||
Measured before the historical backfill described below, six days broke — all of them a tip
|
||||
reversed on one record whose refund sat on its twin, so removing the un-reversed tip left the
|
||||
day short by exactly that amount. Replaying the window from Square gave both records their own
|
||||
copy of every refund, and all six closed. The fix was never wrong; it was reading half a
|
||||
transaction.</p>
|
||||
<p>Service charges are by far the larger of the two fixes, moving $58,923.85 against the tip fix's $3,777.67.</p>
|
||||
<p><strong>None of the three touched a day that was already correct.</strong> Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered. Every day that moved was already wrong. Service charges are by far the largest of the three, moving $58,349.85 against $4,820.19 for refund-only returns and $3,712.67 for tips.</p>
|
||||
<p>That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.</p>
|
||||
<p><strong>The two fixes account for every day they moved, exactly.</strong> Adding up the untendered-tip and service-charge amounts across all 984 changed days leaves a residue of <span class="mono">0.0000000002</span>. Nothing else moved those days; there is no unexplained remainder hiding a third effect, and the six that broke are accounted for by the same arithmetic as the 915 that healed.</p>
|
||||
<p><strong>The arithmetic fixes account for their repair exactly.</strong> Adding up the untendered-tip and service-charge amounts for the 979 days those two closed leaves a residue of <span class="mono">0.0000000013</span> against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a further effect.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -474,9 +465,10 @@
|
||||
<table>
|
||||
<thead><tr><th>Step</th><th>Result</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Walk every order in the database, newest month first</td><td class="n">19,040,785 orders · 38 minutes</td></tr>
|
||||
<tr><td>Give every order its own payment record</td><td class="n">16,236,839 re-keyed · 500,438 copied</td></tr>
|
||||
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across every order of the last year — 5,158,470</span></td></tr>
|
||||
<tr><td>Deactivate the duplicate client at each shared location</td><td class="n">10 locations · shared locations remaining: <span class="good">0</span></td></tr>
|
||||
<tr><td>Walk every order in the database</td><td class="n">19,040,785 orders</td></tr>
|
||||
<tr><td>Give every order its own payment record</td><td class="n">9,100,314 re-keyed · 200,027 copied</td></tr>
|
||||
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across 400,000 orders checked</span></td></tr>
|
||||
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
|
||||
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
|
||||
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
|
||||
@@ -484,13 +476,12 @@
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added 500,438, matching the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)</p>
|
||||
<p><strong>The measurement above was taken with both client records of each pair left live, which is how this deploys.</strong> Nothing is deactivated and no business decision about which restaurant's history survives is needed. The risk that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why <code>existing-id</code> now refuses to resolve a record belonging to another client.</p>
|
||||
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.</p>
|
||||
</div>
|
||||
|
||||
<div class="callout">
|
||||
<span class="h">The whole analysis was run again from nothing, and landed in the same place</span>
|
||||
<p>Everything above was rebuilt from a fresh restore of the production backup, several times over, each time from the backup point itself rather than from a database an earlier run had touched: restore, re-key and split across all nineteen million orders, then two full ninety-day recomputes. The runs used deliberately different preparation — one deactivated the duplicate records, one left them live and untouched, one backfilled their history from Square — so their headline figures differ, and comparing them is how the recommendation below was reached. What did <strong>not</strong> move is the part that should not: for the 190 clients that do not share a Square location the residue is 119 days and $1,730.61 in every run, with the same five restaurants accounting for it. The arithmetic fixes behave identically no matter what is done to the duplicates, which is a stronger check on them than any single measurement.</p>
|
||||
<p>Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. At the point the two runs are comparable — before the refund-only fix below, which the earlier run did not have — the end state matched <strong>to the cent</strong>: 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.</p>
|
||||
</div>
|
||||
|
||||
<div class="callout warn">
|
||||
@@ -502,89 +493,36 @@
|
||||
<section>
|
||||
<h2>What is still out of balance</h2>
|
||||
<div class="measure">
|
||||
<p>122 client-days out of 18,900, totalling <strong>$2,379.45</strong> — and only 32 of
|
||||
those are above ten cents.</p>
|
||||
<p>123 client-days out of 18,900, totalling <strong>$2,970.35</strong> — and only 33 of those are above ten cents. Where it sits matters more than the total.</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
<thead><tr><th>Where the remainder sits</th><th class="n">Days</th><th class="n">Variance</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>The twenty records that share a Square location</td><td class="n good">3</td><td class="n good">$648.84</td></tr>
|
||||
<tr class="total"><td>Every other client — 190 of the 210</td><td class="n">119</td><td class="n">$1,730.61</td></tr>
|
||||
<tr><td>The twenty client records tied to a shared location — including the ten now-dormant duplicates</td><td class="n">20</td><td class="n">$2,467.39</td></tr>
|
||||
<tr class="total"><td>Every other client</td><td class="n">103</td><td class="n">$502.96</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>The shared-location records are now the clean part of the book.</strong> Three days
|
||||
between all twenty of them: NGBK and NGBR at $299.42 each on 2026-08-06, which is the Square
|
||||
tender-versus-order-total gap described below and not an attribution fault, and NGDA at $50.00,
|
||||
an auto-gratuity booked as a service charge. Before the backfill those same records carried
|
||||
423 days and $18,508.39.</p>
|
||||
<p><strong>The other 119 days have not moved across any run of this analysis.</strong> Four
|
||||
separate rebuilds — different databases, different preparation, one with the duplicates
|
||||
deactivated and one without — all land on 119 days and $1,730.61, with the same five
|
||||
restaurants accounting for almost all of it:</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
<thead><tr><th>Client</th><th class="n">Days</th><th class="n">Variance</th><th>What it is</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="mono">NG4S</td><td class="n">10</td><td class="n">$1,066.61</td><td>refunds arriving for a record with no sales imported — the fourth problem</td></tr>
|
||||
<tr><td class="mono">NGDL</td><td class="n">3</td><td class="n">$859.57</td><td>order on one record, its payment on the twin</td></tr>
|
||||
<tr><td class="mono">NGDU</td><td class="n">3</td><td class="n">$345.96</td><td>payout fees on days with no trading</td></tr>
|
||||
<tr><td class="mono">NGBR</td><td class="n">1</td><td class="n">$299.42</td><td rowspan="2">Square recorded $6,358.99 of tender against $6,059.57 of order totals on 2026-08-06 — the gap itself, not a summary fault</td></tr>
|
||||
<tr><td class="mono">NGBK</td><td class="n">1</td><td class="n">$299.42</td></tr>
|
||||
<tr><td class="mono">NGMV</td><td class="n">5</td><td class="n">$259.38</td><td>late May, undiagnosed</td></tr>
|
||||
<tr><td class="mono">NGEB</td><td class="n">4</td><td class="n">$199.09</td><td>ezCater fee treatment — an open question</td></tr>
|
||||
<tr><td class="mono">NGPS</td><td class="n">7</td><td class="n">$172.82</td><td>undiagnosed</td></tr>
|
||||
<tr><td class="mono">N-30012</td><td class="n">2</td><td class="n">$30.31</td><td>late May, undiagnosed</td></tr>
|
||||
<tr class="total"><td class="dim">everyone else</td><td class="n dim">91</td><td class="n dim">$2.40</td><td class="dim">till rounding — pennies a day</td></tr>
|
||||
<tr><td class="mono">NGVC</td><td class="n">3</td><td class="n">$173.81</td><td>payout fees on days with no trading</td></tr>
|
||||
<tr><td class="mono">NGWN</td><td class="n">3</td><td class="n">$170.28</td><td>payout fees on days with no trading</td></tr>
|
||||
<tr class="total"><td class="dim">everyone else</td><td class="n dim">100</td><td class="n dim">$363.42</td><td class="dim">90 of these are till rounding — pennies a day</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>Sixteen of the 122 are days a record had no sales imported at all, worth $1,227.65 — the
|
||||
fourth problem, still deliberately visible. The clusters on NGMV, NGPS and NGEB are
|
||||
unexplained and worth a look, though at under $650 across sixteen days they are no longer
|
||||
urgent.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Making the duplicated restaurants match</h2>
|
||||
<div class="measure">
|
||||
<p>Re-keying stops the two records fighting, but on its own it does not make them equal, and
|
||||
the difference is worth stating plainly because it decides whether the books close.</p>
|
||||
<p><strong>Orders were always duplicated; refunds never were.</strong> A sales order's
|
||||
identifier has always carried its client, so each of the two records built its own order
|
||||
history from the start. Refunds, payouts and cash-drawer shifts did not, so only ONE record
|
||||
holds each of them — whichever imported it last. The migration freezes that ownership rather
|
||||
than evening it out. The record left without them shows returns from its own orders and no
|
||||
refunds to set against them, and is out of balance by exactly the amount its twin is holding.</p>
|
||||
<p>On 2026-05-11 both NGBK and NGBR held the same 221 orders. NGBK had no refunds; NGBR had
|
||||
two, worth $2,232.29; and NGBK's books were out by $2,232.29 to the cent. Across the whole
|
||||
database NGBK held <strong>158,535 orders and five refunds</strong>.</p>
|
||||
</div>
|
||||
<div class="callout">
|
||||
<span class="h">The fix is to ask Square again, not to manufacture copies</span>
|
||||
<p>With client-scoped keys in place, every record now creates its own copy of whatever it
|
||||
reads. So replaying the window from Square is all that is needed: each record imports the same
|
||||
refunds independently and the two histories converge, without any code inventing a duplicate
|
||||
and having to be trusted about it. <code>backfill-history</code> does exactly that for a date
|
||||
range, and after it every one of the ten pairs held matching order and refund counts.</p>
|
||||
<p>It closed <strong>420 of the 423 days</strong> the shared records were carrying, and
|
||||
$17,859.55 of the $18,508.39. It is also what recovered the zero-regression guarantee above.</p>
|
||||
</div>
|
||||
<div class="callout warn">
|
||||
<span class="h">One capped read, found by doing this</span>
|
||||
<p>The refunds import asked Square for a location's refunds and read the first page of the
|
||||
answer — no cursor, no date range. Square pages at a hundred, so a location with more than a
|
||||
hundred refunds silently returned a hundred, and the response looked complete. That is why the
|
||||
twins each held almost exactly 100 refunds, and why an earlier import added exactly 1,000
|
||||
across ten locations. Following the cursor is a few lines; the reason it went unnoticed for so
|
||||
long is that a capped list is indistinguishable from a short one.</p>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>This turned out to be a better answer than retiring the duplicate records.</strong>
|
||||
An earlier measurement that deactivated one record of each pair left 279 days and $7,790.54.
|
||||
Backfilling instead, with both records live, leaves <strong>122 days and $2,379.45</strong> —
|
||||
and it needs no business decision about which restaurant's history to abandon.</p>
|
||||
<p><strong>What is left divides cleanly in two.</strong> 108 days totalling $1,995.36 are real trading days with genuine discrepancies — the NGBR/NGBK tender gap, the ezCater fee question, and the unexplained clusters. The other 15 days, $974.99, carry a processing-fee debit and nothing else: a payout fee landing on a day the restaurant did not trade. That is the same shape as the refund-only day and could be closed the same way, but the offsetting entry belongs to the payout rather than to sales, so it needs the payout side modelled rather than another rule in the summary. It is not worth holding this work for.</p>
|
||||
<p>The clusters on NGMV and NGEB are unexplained and worth a look, though at $458 across nine days they are no longer urgent. They are visible only because the window is ninety days; a thirty-day view does not reach them.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -600,8 +538,8 @@
|
||||
<table>
|
||||
<thead><tr><th>Record type</th><th class="n">Total</th><th class="n">Client-scoped</th><th class="n">Still to rename</th><th class="n">Cannot be scoped</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Card payments</td><td class="n">17,045,933</td><td class="n good">17,045,933</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Refunds</td><td class="n">50,986</td><td class="n good">50,986</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Card payments</td><td class="n">17,047,142</td><td class="n good">17,047,142</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Refunds</td><td class="n">51,986</td><td class="n good">51,986</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Payouts</td><td class="n">144,688</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr>
|
||||
<tr><td>Cash-drawer shifts</td><td class="n">69,291</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
</tbody>
|
||||
@@ -640,20 +578,15 @@
|
||||
<p>The problem this work exists to solve is gone: no payment answers to two orders, so the
|
||||
component relationship means what it says and deleting an order can no longer take another
|
||||
order's money with it.</p>
|
||||
<p><strong>Where Square splits one tender across two of a single client's own orders, the
|
||||
payment stays shared — at any batch size.</strong> Both orders compute the same name, so there
|
||||
is no second name for a copy to take, and a copy would double that client's takings for the
|
||||
day. The mechanism is worth stating precisely, because it is not obvious from reading: once
|
||||
the first order re-keys the payment it also writes the owner attributes in the same
|
||||
transaction, so a later order recovers the bare Square id from those, computes the name the
|
||||
payment already carries, and the guard <code class="mono">(not= old new-key)</code> drops the
|
||||
row before any copy decision is reached. Verified by running the migration at a batch size of
|
||||
one, which forces the two orders into separate batches: no copy is made.</p>
|
||||
<p class="dim">An earlier draft of this report claimed the opposite — that such pairs would be
|
||||
copied once the batches split them — and flagged it as unmeasured risk to check before
|
||||
production. That was wrong, and it is recorded here rather than quietly deleted because it did
|
||||
real damage: an independent reviewer cited this paragraph as evidence and raised a defect that
|
||||
does not exist. A test now pins the behaviour at batch size one.</p>
|
||||
<p><strong>One behaviour changed when the pass was run over everything, and it is worth
|
||||
recording.</strong> Where Square splits one tender across two of a single client's own orders,
|
||||
the earlier design left the payment shared on purpose — both orders compute the same name, so
|
||||
there is no second name for a copy to take. That rule only holds for two orders processed in
|
||||
the same batch. Run across nineteen million orders in batches of two thousand, such pairs
|
||||
almost always fall in different batches and the second order now takes a copy. Inside the
|
||||
ninety-day window this changed nothing measurable: the recompute after the database-wide pass
|
||||
matched the one before it to the cent. Outside the window it has not been measured, and it
|
||||
should be before this runs against production.</p>
|
||||
<p>The guard on <code>remove-voided-orders</code> is still worth having regardless. It is
|
||||
cheap, and it makes the safety a property of the deletion rather than of the migration having
|
||||
been run first.</p>
|
||||
@@ -662,12 +595,11 @@
|
||||
<div class="callout">
|
||||
<span class="h">Re-running is safe, and that was proved at full scale</span>
|
||||
<p>After the complete pass, asking the migration what it would change next returns
|
||||
<strong>nothing</strong> — 17,045,933 payments examined, none to rename, none unscopable. A
|
||||
<strong>nothing</strong> — 17,047,142 payments examined, none to rename, none unscopable. A
|
||||
record that already carries the right name is left untouched, so the migration can be stopped,
|
||||
resumed, or repeated without consequence.</p>
|
||||
<p>Its speed is worth a note for whoever schedules it: the whole nineteen million orders were
|
||||
walked in about <strong>thirty-eight minutes</strong>, month by month from the current month
|
||||
backwards so that stopping early leaves the recent end done. An earlier attempt appeared to be
|
||||
walked in about <strong>thirteen minutes</strong>. An earlier attempt appeared to be
|
||||
transactor-bound and was projected at two days, which is why a previous run narrowed it to the
|
||||
analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process
|
||||
driving the migration — freeing held memory took an unrelated recompute from 17 client-days a
|
||||
@@ -692,7 +624,8 @@
|
||||
<tbody>
|
||||
<tr><td>Which client record survives at each shared location</td><td>the business</td><td>the newer record generally has no history before the split, so keeping it loses years of the location's books</td></tr>
|
||||
<tr><td>Which revenue account service charges post to</td><td>accounting</td><td>currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances</td></tr>
|
||||
<tr><td><strong>659 refunds on records that have no sales for them</strong></td><td>the business, then engineering</td><td>the top open item. $15,225.24 dated before the holding record's own first order. Either the missing sales get imported, or the refunds move to the record that has them — but the books cannot close until one of the two happens</td></tr>
|
||||
<tr><td>Recognising a return on a refund-only day</td><td>accounting</td><td>posts to 41300 Returns, the account already used for returns; it moves the recognition date, so a refund settling after month end lands in the later period</td></tr>
|
||||
<tr><td>Refunds on days that <em>did</em> trade</td><td>accounting</td><td>deliberately untouched — apportioning a return across a day that also sold is a judgement call, and 108 days remain out of balance because of it</td></tr>
|
||||
<tr><td>Whether to correct records the wrong client already owns</td><td>the business</td><td>the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared</td></tr>
|
||||
<tr><td><code>remove-voided-orders</code></td><td>engineering</td><td>safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes</td></tr>
|
||||
</tbody>
|
||||
@@ -703,7 +636,7 @@
|
||||
<span class="h">Two operational findings, unrelated to the summaries</span>
|
||||
<p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p>
|
||||
<p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. Worth checking what production is set to.</p>
|
||||
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in well under an hour. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
|
||||
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in thirteen minutes. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -753,7 +686,7 @@
|
||||
|
||||
<footer>
|
||||
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
|
||||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 122 remaining days only 32 are material, and just 3 of them sit on the twenty records that share a Square location.</span>
|
||||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 123 remaining days, 33 are material.</span>
|
||||
<span>Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on <code>as-of</code> above.</span>
|
||||
</footer>
|
||||
|
||||
|
||||
@@ -1,22 +1,12 @@
|
||||
# Sales-summary balancing — rollout plan
|
||||
|
||||
Steps to execute, in order. Every step is either reversible or verifiable before the next one
|
||||
begins. The one behaviour change that alters a client's books is behind a per-client feature flag
|
||||
that is **off by default**, so merging and deploying this branch changes nothing on its own.
|
||||
begins. Both behaviour changes are behind per-client feature flags that are **off by default**, so
|
||||
merging and deploying this branch changes nothing on its own.
|
||||
|
||||
Measured on a restored copy of production (backup point `209608347`), 210 clients over
|
||||
2026-05-10 → 2026-08-07, with the duplicate client records left active exactly as they will be in
|
||||
production: **1,191 client-days out of balance / $70,276.50 → 122 days / $2,379.45**, of which only
|
||||
32 are above ten cents. 1,069 days came into balance, none broke, and no already-balanced day had a
|
||||
figure altered.
|
||||
|
||||
Of the $2,379.45 left, just **$648.84 across 3 days** is on the twenty shared-location records. The
|
||||
other 119 days and $1,730.61 belong to ordinary clients and have not moved across any run of this
|
||||
analysis.
|
||||
|
||||
Getting the shared records there needs step 5 — a historical backfill from Square. Without it they
|
||||
carry 423 days and $18,508.39, because re-keying stops the two records fighting but does not give
|
||||
each its own copy of the refunds.
|
||||
2026-05-10 → 2026-08-07: **1,258 client-days out of balance / $69,560.10 → 123 days / $2,970.35**,
|
||||
with zero days knocked out of balance and zero already-balanced days altered.
|
||||
|
||||
---
|
||||
|
||||
@@ -24,242 +14,111 @@ each its own copy of the refunds.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Flag introduced | `summary-service-charges` — off by default |
|
||||
| Branch | `worktree-sales-summary-balance` |
|
||||
| Flags introduced | `summary-service-charges`, `summary-refund-only-returns` — both off by default |
|
||||
| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` |
|
||||
| Expected migration runtime | ~38 minutes for 19M orders on a warm cache |
|
||||
| Backfill runtime (step 5) | ~5.9 hours for 90 days across the 20 shared-location records — an overnight job |
|
||||
| Nothing here touches | invoices, payments, the ledger, or any client without the flag set |
|
||||
| Expected migration runtime | ~13 minutes for 19M orders on a warm cache |
|
||||
| Nothing here touches | invoices, payments, the ledger, or any client without the flags set |
|
||||
|
||||
**Client configuration is left exactly as it is.** Ten Square locations are configured against two
|
||||
client records each, and both stay active. The re-key is what resolves them: once every record
|
||||
carries its owner in its key, each client's import resolves only its own records and the two
|
||||
records keep independent, stable histories. No "which record survives" decision is needed, and
|
||||
nothing is deactivated.
|
||||
|
||||
The consequence to be aware of: each Square payment, refund, payout and shift at a shared location
|
||||
becomes **two entities, one per client record** — by design. That is the stable end state, not a
|
||||
duplicate to clean up. If any report or export aggregates across client records, one restaurant's
|
||||
takings would be counted twice at that layer. Nothing in this work changes that either way.
|
||||
|
||||
That holds automatically for everything imported *from now on*, because the keys carry the client.
|
||||
It does **not** hold for history: refunds, payouts and shifts already in the database exist only
|
||||
once, on whichever record imported them last, and re-keying freezes that rather than evening it out.
|
||||
Step 5 is what brings the existing history into the same shape.
|
||||
|
||||
**The only window of risk is between deploying and finishing the migration**, while legacy keys
|
||||
still exist for a client to resolve. Steps 2–6 exist to make that window effectively zero.
|
||||
**One prerequisite that is not code.** Ten Square locations are configured against two client records
|
||||
each. Someone in the business has to decide which record survives at each. The newer record usually
|
||||
has no history from before the split, so keeping it loses years of that location's books. Do this
|
||||
before step 3.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Guard `remove-voided-orders`
|
||||
## Step 1 — Deploy the code
|
||||
|
||||
Deploy the branch as normal. Both flags are absent from every client, so:
|
||||
|
||||
- tips are calculated exactly as they are today,
|
||||
- no `Service Charges` line is written,
|
||||
- no `Returns` line is written on refund-only days.
|
||||
|
||||
The only changes that take effect immediately are the safe ones: imbalance logging, the
|
||||
dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix,
|
||||
and the importer's new client-scoped keys.
|
||||
|
||||
**The importer starts writing client-scoped keys straight away, and reads both schemes.** That is
|
||||
deliberate and is what makes the deploy independent of the migration. Do not remove the legacy
|
||||
lookup in `square.core3/existing-id` yet — see step 8.
|
||||
|
||||
**Verify before moving on.** After one nightly import cycle:
|
||||
|
||||
```clojure
|
||||
;; refunds, payouts and shifts must not have doubled
|
||||
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))
|
||||
```
|
||||
|
||||
Compare against the same counts taken immediately before deploy. Growth should be ordinary daily
|
||||
volume. A near-doubling means the legacy fallback is not working — **stop and roll back the deploy**.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Guard `remove-voided-orders`
|
||||
|
||||
Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so
|
||||
retracting an order cascades into its payments. Until step 4 finishes there are still payments with
|
||||
retracting an order cascades into its payments. Until step 3 finishes there are still payments with
|
||||
two parent orders, and deleting one client's voided order can take the other client's payment with
|
||||
it.
|
||||
|
||||
Either leave `remove-voided-orders` switched off until step 4 verifies clean, or change it to detach
|
||||
a payment that has more than one parent rather than delete it. Detaching is worth doing regardless —
|
||||
Either leave `remove-voided-orders` switched off until step 3 completes, or change it to detach a
|
||||
payment that has more than one parent rather than delete it. Detaching is worth doing regardless —
|
||||
it makes the safety a property of the deletion rather than of the migration having been run first.
|
||||
|
||||
See `docs/2026-08-15-remove-voided-orders-risk.md`.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Pause the Square importer
|
||||
## Step 3 — Retire the duplicate client records
|
||||
|
||||
**This is what makes the deploy safe, and it is easy to skip.** Steps 2 through 6 should be one
|
||||
maintenance action, not separate days' work.
|
||||
Business decision from the top of this document. Deactivate the losing record's Square location so
|
||||
the importer stops fetching for it. The record itself stays; its history is untouched.
|
||||
|
||||
While legacy keys exist, `square.core3/existing-id` falls back to them — and at a shared location
|
||||
that is the one code path that can reach across client records. Running the migration with imports
|
||||
paused means no client is resolving keys while the keys are being rewritten, so the window closes
|
||||
entirely rather than merely narrowing.
|
||||
|
||||
The migration itself takes about **38 minutes** for all 19M orders, so the pause is short — and
|
||||
if you need it shorter, see step 6: you can resume imports before it finishes.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Deploy the code
|
||||
|
||||
Deploy the branch. The flag is absent from every client, so:
|
||||
|
||||
- tips are calculated exactly as they are today,
|
||||
- no `Service Charges` line is written.
|
||||
|
||||
The only changes that take effect immediately are the safe ones: imbalance logging, the
|
||||
dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix,
|
||||
and the importer's new client-scoped keys.
|
||||
|
||||
**The importer reads both key schemes**, so the deploy does not depend on the migration having
|
||||
finished. Two protections cover the interval before it does: imports are paused (step 2), and
|
||||
`existing-id` refuses to resolve a record that already belongs to a different client. Do not remove
|
||||
the legacy lookup yet — see step 10.
|
||||
**Verify:** no Square location is configured against two active client records.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Run the migration
|
||||
|
||||
Run it immediately after the deploy, while imports are still paused.
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.jobs.rekey-square-external-ids :as rk])
|
||||
|
||||
;; read-only first — no two entities may want the same key. `plan` does NOT return a
|
||||
;; :collisions key; you have to hand its :new-keys to `collisions` yourself.
|
||||
(rk/collisions (:new-keys (rk/plan (d/db conn) :charge/external-id rk/charge-prefix)))
|
||||
;; => [] (anything else: stop, do not migrate)
|
||||
;; read-only first — check :collisions is empty for every attribute
|
||||
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
|
||||
|
||||
;; then the whole thing
|
||||
(rk/migrate-all! 2000)
|
||||
```
|
||||
|
||||
`migrate-all!` runs this same check itself, on every attribute including charges, and throws
|
||||
rather than transacting if it finds one. Running it by hand first just means finding out before
|
||||
the 38-minute walk rather than partway through it.
|
||||
|
||||
Runs in about thirty-eight minutes over 19M orders. It is **idempotent and resumable** — a record that
|
||||
Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that
|
||||
already carries the right name is skipped, so it can be stopped and re-run without consequence.
|
||||
|
||||
**It is also ordered so that stopping early is survivable.** Refunds, payouts and cash-drawer
|
||||
shifts go first — a quarter of a million records, seconds of work — so an interruption cannot catch
|
||||
them half done. The long part then walks orders **a month at a time, from the current month
|
||||
backwards**, logging `::month-complete` as each finishes:
|
||||
|
||||
```
|
||||
::month-complete :month "2026-08" :rekeyed 118203 :cloned 2244
|
||||
::month-complete :month "2026-07" :rekeyed 241887 :cloned 4611
|
||||
...
|
||||
```
|
||||
|
||||
That ordering is the recovery plan. If it dies, everything from the last logged month forward is
|
||||
fully scoped — and that recent window is what the importer actually reads — so **you can resume
|
||||
imports against a partially migrated database** and finish the older tail later. Walking oldest
|
||||
first would have spent the first several hours on 2019 data no import will touch, leaving exactly
|
||||
the wrong end done.
|
||||
|
||||
If you do resume imports mid-migration, the ownership guard in `existing-id` is what keeps the
|
||||
unmigrated tail safe: a client cannot resolve onto another client's legacy-keyed record.
|
||||
|
||||
If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
|
||||
not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
|
||||
retained memory in the REPL and re-measure before changing anything about the database.
|
||||
|
||||
**Verify.** Two checks, doing two different jobs — run both.
|
||||
|
||||
**(a) Completeness, across everything.** `plan` must report nothing left to do, for all four
|
||||
attributes:
|
||||
**Verify — all four must read zero to migrate and zero unscopable:**
|
||||
|
||||
```clojure
|
||||
(rk/unscoped-report (d/db conn))
|
||||
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
|
||||
;; => {:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0}
|
||||
```
|
||||
;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0}
|
||||
|
||||
Read `:to-migrate 0` **and** `:unscopable 0`. This is the authoritative signal, and it covers all
|
||||
17M charges.
|
||||
|
||||
`unscoped-report` is useful colour but is not the gate: its `:no-owner` column never reaches zero
|
||||
for charges, because ~283k payout stubs carry no `:charge/client` of their own and it classifies
|
||||
by attribute rather than by resolving ownership. Judge completeness by `plan`.
|
||||
|
||||
**(b) The safety gate for the cascade** — no payment may answer to two orders, or re-enabling
|
||||
`remove-voided-orders` in step 9 can delete a payment another order still needs. Check **every**
|
||||
order in the last year, with no sampling:
|
||||
|
||||
```clojure
|
||||
(let [db (d/db conn)
|
||||
cs (map first (d/q '[:find ?c :where [?c :client/code _]] db))
|
||||
year (java.util.Date. (- (.getTime (java.util.Date.)) (long (* 365 86400000))))]
|
||||
(rk/charges-with-multiple-parents
|
||||
db (map first (iol-ion.query/scan-sales-orders db cs year nil))))
|
||||
;; and the gate that this work exists for
|
||||
(rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn))))
|
||||
;; => 0
|
||||
```
|
||||
|
||||
On the restored copy that is 5,158,470 orders — 27% of the table — via the
|
||||
`:sales-order/client+date` index. A year is chosen deliberately: `remove-voided-orders` only ever
|
||||
deletes orders Square reports as voided, which are recent, so that is where the destructive risk
|
||||
lives. Completeness across all of history is check (a)'s job, not this one.
|
||||
|
||||
> Do **not** sample this with `(take n (rk/all-order-ids db))`. `all-order-ids` streams `:aevt`,
|
||||
> which is ascending entity id, so a `take` returns the *oldest* orders — on the restored copy the
|
||||
> first 400,000 are all from 2019–2021, before any of the contention this gate looks for. It would
|
||||
> report a confident zero having inspected none of the relevant data.
|
||||
Note `unscoped-report`'s `:no-owner` column is not a gap: ~283k payout-stub payments carry no
|
||||
`:charge/client` attribute of their own, so it cannot verify them by attribute. `plan` resolves
|
||||
ownership through whatever refers to them and is the figure to trust.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Backfill the shared-location clients from Square
|
||||
|
||||
**Skip this and the ten duplicated restaurants stay badly out of balance.** It is the difference
|
||||
between 122 client-days out of balance and 542.
|
||||
|
||||
Sales orders have always been keyed by client, so both records of a pair built their own order
|
||||
history. Refunds, payouts and cash-drawer shifts were not, so only ONE record holds each of them.
|
||||
Re-keying freezes that ownership; it does not even it out. The record left without them shows
|
||||
returns from its own orders and no refunds against them — NGBK held 158,535 orders and five
|
||||
refunds — and is out of balance by exactly what its twin is holding.
|
||||
|
||||
Rather than manufacture copies, ask Square again. Client-scoped keys mean each record now creates
|
||||
its own copy of whatever it reads, so replaying the window makes the two histories converge:
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.square.core3 :as sq])
|
||||
(require '[clj-time.core :as t])
|
||||
|
||||
@(apply sq/backfill-history
|
||||
(t/date-time 2026 5 10) (t/date-time 2026 8 9)
|
||||
["NGBK" "NGBR" "NGCD" "NGCC" "NGVG" "NGVC" "NGEZ" "NGJS" "NGDG" "NGDU"
|
||||
"NGDV" "NGDS" "NGWC" "NGWN" "NGHY" "NGHA" "NGDA" "NGDL" "NGCL" "NGCT"])
|
||||
```
|
||||
|
||||
**Verify** — every pair should hold matching order and refund counts in the window:
|
||||
|
||||
```clojure
|
||||
;; per pair, per side: window orders and window refunds. The two sides should agree.
|
||||
```
|
||||
|
||||
Measured on the restored copy: all ten pairs matched afterwards, and the shared records went from
|
||||
423 days and $18,508.39 out of balance to 3 days and $648.84.
|
||||
|
||||
**Budget an overnight run.** This took **5.9 hours** for ninety days across the twenty records.
|
||||
Every Square call in the process shares one 25-requests-per-second throttle, refunds and shifts cost
|
||||
one API call per record, and `backfill-history` imports three clients at a time — raise its
|
||||
`s/buffer` if you need it faster. Neither the database nor the transactor is the limit; reads
|
||||
measured at 32 µs.
|
||||
|
||||
It must run **after** the migration. Run before, and it imports against legacy keys and leaves more
|
||||
to migrate.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Resume the Square importer
|
||||
|
||||
Normally: once step 4's two checks read clean and step 5's backfill has finished. The maintenance
|
||||
window ends here.
|
||||
|
||||
**If the migration did not finish**, you do not have to wait for it. Resume imports once the
|
||||
`::month-complete` log covers the window your importer reads — the last 75 days for payouts and
|
||||
cash-drawer shifts, and whatever range the order import is configured for. Then re-run
|
||||
`migrate-all!` afterwards to walk the remaining older months; it will skip everything already done.
|
||||
Run the step 4 checks again once it does finish.
|
||||
|
||||
The first cycle after resuming is the one to watch. Compare these against the same counts taken
|
||||
immediately before the deploy — growth should be ordinary daily volume:
|
||||
|
||||
```clojure
|
||||
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :charge/external-id))
|
||||
```
|
||||
|
||||
A near-doubling of any of them means records are being created rather than matched — **stop and
|
||||
roll back the deploy.** Charges are included deliberately: they are the one that doubles a client's
|
||||
takings rather than merely duplicating a row.
|
||||
|
||||
---
|
||||
|
||||
## Step 7 — Recompute summaries, flags still off
|
||||
## Step 5 — Recompute summaries, flags still off
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.jobs.sales-summaries :as ss])
|
||||
@@ -267,7 +126,7 @@ takings rather than merely duplicating a row.
|
||||
```
|
||||
|
||||
This is the pass that banks the deduplication. **Capture the result before going further** — you
|
||||
will need it as the baseline for step 8, and it cannot be reconstructed afterwards:
|
||||
will need it as the baseline for step 6, and it cannot be reconstructed afterwards:
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath
|
||||
@@ -282,20 +141,26 @@ will need it as the baseline for step 8, and it cannot be reconstructed afterwar
|
||||
|
||||
---
|
||||
|
||||
## Step 8 — Turn the flag on, a few restaurants at a time
|
||||
## Step 6 — Turn the flags on, a few restaurants at a time
|
||||
|
||||
Needs accounting sign-off first: `summary-service-charges` posts to **49000 Service Income**, chosen
|
||||
so the work could be measured. It affects reporting, never whether a day balances.
|
||||
Needs accounting sign-off first, on two points:
|
||||
|
||||
- `summary-service-charges` posts to **49000 Service Income**. Chosen so the work could be measured.
|
||||
It affects reporting, never whether a day balances.
|
||||
- `summary-refund-only-returns` posts to **41300 Returns**, the account already used for returns. It
|
||||
moves the *recognition date* of a return onto the day the refund settled, so a refund settling
|
||||
after month end lands in the later period.
|
||||
|
||||
```clojure
|
||||
@(d/transact conn [{:db/id [:client/code "NGxx"]
|
||||
:client/feature-flags ["summary-service-charges"]}])
|
||||
:client/feature-flags ["summary-service-charges"
|
||||
"summary-refund-only-returns"]}])
|
||||
(ss/refresh-sales-summaries 90)
|
||||
```
|
||||
|
||||
Start with two or three restaurants, confirm, then widen.
|
||||
|
||||
**Verify** against the capture from step 7:
|
||||
**Verify** against the capture from step 5:
|
||||
|
||||
```clojure
|
||||
(def after (cmp/summaries-in (d/db conn) start end))
|
||||
@@ -307,107 +172,40 @@ The two numbers that matter — both were zero across all 18,900 client-days in
|
||||
- `:balanced->unbalanced` must be **0**
|
||||
- previously-balanced days whose lines changed must be **0**
|
||||
|
||||
If either is non-zero, retract the flag for the affected clients and re-run step 7. The flag is the
|
||||
rollback: removing it restores today's behaviour exactly.
|
||||
If either is non-zero, retract the flags for the affected clients and re-run step 5. The flags are
|
||||
the rollback: removing them restores today's behaviour exactly.
|
||||
|
||||
---
|
||||
|
||||
## Step 9 — Re-enable `remove-voided-orders`
|
||||
## Step 7 — Re-enable `remove-voided-orders`
|
||||
|
||||
Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 1.
|
||||
Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2.
|
||||
|
||||
---
|
||||
|
||||
## Step 10 — Remove the legacy key lookup
|
||||
## Step 8 — Remove the legacy key lookup
|
||||
|
||||
**Schedule this; do not leave it open-ended.** Both client records at a shared location stay active
|
||||
permanently, so the legacy fallback in `square.core3/existing-id` is the one code path that can ever
|
||||
reach across them. Deleting it is what turns the guarantee from conventional into structural.
|
||||
Only once `plan` reports `:to-migrate 0` and has stayed there through several import cycles. Drop
|
||||
the second branch of `square.core3/existing-id`. At that point two clients sharing a location
|
||||
becomes structurally incapable of producing a shared record, rather than prevented by a convention a
|
||||
future import could break.
|
||||
|
||||
Once `plan` reports `:to-migrate 0` and has stayed there through several import cycles, drop the
|
||||
legacy branch of `existing-id` — and with it `owned-by-other-client?`, which exists only to make
|
||||
that branch safe while it lives. After this, two clients on one location are structurally incapable
|
||||
of resolving onto each other's records, and no ordering discipline is required to keep it that way.
|
||||
|
||||
Until it is done, the protection is the guard plus the maintenance window, both of which depend on
|
||||
people doing the right thing. That is the reason not to let this drift.
|
||||
|
||||
---
|
||||
|
||||
## Step 11 — Deal with the refunds that have no sales behind them
|
||||
|
||||
**The most important item in this document, and the only one that is not just execution.**
|
||||
|
||||
16 of the 122 remaining days are a record carrying refunds on a day it recorded no sales at all,
|
||||
and all 16 fall before that client's first ever order. Two clients are affected, holding **160
|
||||
refunds worth $4,347.68 dated before their own first order**:
|
||||
|
||||
| Client | First order | Refunds before it | Value | Days out of balance |
|
||||
|---|---|---:|---:|---:|
|
||||
| NG4S | 2026-05-29 | 79 | $2,180.08 | 10 |
|
||||
| NGPS | 2026-05-26 | 81 | $2,167.60 | 7 |
|
||||
|
||||
**Step 5's backfill already resolved the other seven.** Before it, nine records were in this state
|
||||
holding 660 refunds worth $15,237.02 — but seven of them were shared-location twins whose refunds
|
||||
only looked orphaned because their orders had never been imported. Replaying the window gave them
|
||||
their orders, and the refunds stopped predating them.
|
||||
|
||||
NG4S and NGPS are different: neither shares a Square location, so there is no twin holding the other
|
||||
half. Their sales genuinely are not in the system for the period their refunds cover. The database's
|
||||
own ownership history is the evidence to check — for the twins it showed refunds changing hands
|
||||
between the two records; for these two there is no second record to have taken them from.
|
||||
|
||||
Two ways to close it, and the business has to pick:
|
||||
|
||||
1. **Import the missing sales.** Correct if these records are meant to have their own books. Try
|
||||
`backfill-history` for them first, with a window reaching back before their first order — that is
|
||||
exactly what fixed the seven, and it is one command.
|
||||
2. **Move the refunds to the record that has the sales.** Correct only if the refunds were misfiled
|
||||
onto a record that should not have books of its own.
|
||||
|
||||
Start with (1): it is cheap, reversible in the sense that it only adds what Square reports, and it
|
||||
is already proven to work on this exact symptom.
|
||||
|
||||
```clojure
|
||||
;; per client: refunds dated before that client's own first order
|
||||
(let [first-order (->> (d/q '[:find [?d ...] :in $ ?c
|
||||
:where [?o :sales-order/client ?c] [?o :sales-order/date ?d]]
|
||||
(d/db conn) [:client/code "NG4S"])
|
||||
(reduce (fn [a b] (if (.before a b) a b))))]
|
||||
(->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c
|
||||
:where [?r :sales-refund/client ?c]]
|
||||
(d/db conn) [:client/code "NG4S"])
|
||||
(filter #(.before (:sales-refund/date %) first-order))
|
||||
count))
|
||||
```
|
||||
|
||||
**Until this is resolved those days stay out of balance, on purpose.** A summary change to close
|
||||
them was written and measured — it works, closes 16 days and $1,227.65, and breaks nothing — and it
|
||||
was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not
|
||||
being imported. A test asserts the day stays unbalanced so nobody closes it without reading this.
|
||||
This is the last step and there is no hurry.
|
||||
|
||||
---
|
||||
|
||||
## What this will not fix
|
||||
|
||||
122 client-days over ninety days, $2,379.45, of which only 32 are above ten cents.
|
||||
123 client-days over ninety days, $2,970.35, of which only 33 are above ten cents.
|
||||
|
||||
| | Days | Variance | |
|
||||
|---|---:|---:|---|
|
||||
| Real trading days with genuine discrepancies | 106 | $1,151.80 | see below |
|
||||
| Refunds on a record with no sales imported | 16 | $1,227.65 | step 11 — deliberately visible |
|
||||
| Real trading days with genuine discrepancies | 108 | $1,995.36 | the NGBR/NGBK tender gap, the ezCater fee question, unexplained clusters on NGMV and NGEB |
|
||||
| Processing fee on a day with no trading | 15 | $974.99 | same shape as the refund-only day, but the offsetting entry belongs to the payout, not to sales |
|
||||
|
||||
Of the 106 trading days, only **3 are on shared-location records** — $648.84 in total, and all three
|
||||
are already diagnosed: NGBK and NGBR at $299.42 each on 2026-08-06, where Square recorded $6,358.99
|
||||
of tender against $6,059.57 of order totals (the gap itself, not a summary fault), and NGDA at
|
||||
$50.00, an auto-gratuity booked as a service charge.
|
||||
|
||||
The other 103 days come to **$502.96 across 190 clients** — a few dollars here and there, mostly
|
||||
till rounding, plus small undiagnosed clusters on NGMV ($259.38 over 5 days) and NGEB ($199.09 over
|
||||
4 days, an ezCater fee-treatment question). Those two are worth a look but are not urgent.
|
||||
|
||||
That 103-day, $502.96 figure has been identical in every run of this analysis — with the duplicates
|
||||
deactivated, with them live, and with them backfilled. It is the floor this work reaches.
|
||||
Refunds landing on days that *did* trade are deliberately left alone. Apportioning a return across a
|
||||
day that also sold is a judgement call, and guessing at it is how correct books get quietly
|
||||
rewritten.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -20,8 +20,7 @@
|
||||
(:require
|
||||
[auto-ap.datomic :refer [conn]]
|
||||
[auto-ap.logging :as alog]
|
||||
[datomic.api :as dc]
|
||||
[iol-ion.query]))
|
||||
[datomic.api :as dc]))
|
||||
|
||||
(def refund-prefix "square/refund/")
|
||||
(def charge-prefix "square/charge/")
|
||||
@@ -196,11 +195,7 @@
|
||||
[orders batch-size]
|
||||
(let [cloned (atom 0)
|
||||
rekeyed (atom 0)]
|
||||
(doseq [[i batch] (map-indexed vector (partition-all batch-size orders))]
|
||||
(when (zero? (mod i 200))
|
||||
;; the whole-database run walks 19M orders; without a trail an interrupted run leaves
|
||||
;; no way to tell how far it got short of querying the data by hand
|
||||
(alog/info ::splitting :orders-done (* i batch-size) :rekeyed @rekeyed :cloned @cloned))
|
||||
(doseq [batch (partition-all batch-size orders)]
|
||||
(let [db (dc/db conn)
|
||||
batch-seen (atom {})
|
||||
tx (doall
|
||||
@@ -260,19 +255,12 @@
|
||||
"Counts, per entity type, how many keys are already client-scoped, how many still carry the
|
||||
legacy unscoped form, and how many have no owner to scope by.
|
||||
|
||||
The importer tolerates both key schemes on purpose, so that the change can be deployed before
|
||||
the migration finishes — but that tolerance is a transition, not a resting place. While
|
||||
`:legacy` is above zero the database is in a mixed state and a stray unscoped record can still
|
||||
be adopted by whichever client imports it first. **`:legacy` reaching zero on every attribute is
|
||||
the done-signal**, and it is what licenses removing the fallback lookup in
|
||||
`square.core3/existing-id`.
|
||||
|
||||
`:no-owner` is NOT part of that signal and never reaches zero for charges. It counts entities
|
||||
whose own `:charge/client`/`:charge/location` are absent — around an eighth of charges, the
|
||||
payout stubs `migrate!` only ever gives an external id — so this report structurally cannot
|
||||
verify them even when their keys are perfectly scoped. It classifies by attribute; `plan`
|
||||
resolves ownership through whatever references the entity. Ask `plan` for the authoritative
|
||||
answer: `:to-migrate 0` with `:unscopable 0` means there is nothing left to do."
|
||||
This is the completeness gate. The importer tolerates both key schemes on purpose, so that the
|
||||
change can be deployed before the migration finishes — but that tolerance is a transition, not
|
||||
a resting place. While `:legacy` is above zero the database is in a mixed state and a stray
|
||||
unscoped record can still be adopted by whichever client imports it first. Once every count
|
||||
reads zero the fallback lookup in `square.core3/existing-id` can be removed and the guarantee
|
||||
becomes structural rather than conventional."
|
||||
[db]
|
||||
(into {}
|
||||
(for [{:keys [attr prefix client location]} scoped-attrs]
|
||||
@@ -289,51 +277,10 @@
|
||||
(dc/datoms db :aevt attr))])))
|
||||
|
||||
(defn all-order-ids
|
||||
"Every sales order in the database, streamed in `:aevt` order — which is ascending entity id,
|
||||
so OLDEST first. Fine for counting; wrong for anything that takes a prefix. `(take n ...)` of
|
||||
this returns the oldest n orders, not a sample: on the production copy the first 400,000 are
|
||||
all from 2019 to 2021. Use `order-months-newest-first` to walk the data in migration order."
|
||||
"Every sales order in the database, streamed."
|
||||
[db]
|
||||
(map :e (dc/datoms db :aevt :sales-order/external-id)))
|
||||
|
||||
(def ^:private earliest-orders
|
||||
"How far back the month walk goes. Comfortably before the oldest order in the database
|
||||
(2019-12-31 on the production copy); months with no orders cost one index seek per client."
|
||||
#inst "2015-01-01T00:00:00.000-00:00")
|
||||
|
||||
(defn- ->date [^java.time.LocalDate d]
|
||||
(java.util.Date/from (.toInstant (.atStartOfDay d (java.time.ZoneId/systemDefault)))))
|
||||
|
||||
(defn order-months-newest-first
|
||||
"`[start end]` month windows from now back to `earliest`, newest month first.
|
||||
|
||||
The migration walks months in this order on purpose. It is the difference between an
|
||||
interrupted run leaving the data safe to import against and leaving it dangerous: the importer
|
||||
works on recent data, so having the newest months fully scoped is what lets imports resume
|
||||
while the older tail is still unmigrated. Walking oldest-first would spend hours on 2019 before
|
||||
touching anything this month's import will read.
|
||||
|
||||
Windows tile without gaps — each month's end is the day before the next month's start — and
|
||||
because the migration is idempotent an order landing in two windows is a no-op the second
|
||||
time, so boundary precision is not safety-critical."
|
||||
([] (order-months-newest-first earliest-orders))
|
||||
([^java.util.Date earliest]
|
||||
(let [zone (java.time.ZoneId/systemDefault)
|
||||
floor (java.time.YearMonth/from (.toLocalDate (.atZone (.toInstant earliest) zone)))]
|
||||
(->> (iterate (fn [^java.time.YearMonth m] (.minusMonths m 1)) (java.time.YearMonth/now zone))
|
||||
(take-while (fn [^java.time.YearMonth m] (not (.isBefore m floor))))
|
||||
(map (fn [^java.time.YearMonth m]
|
||||
[(->date (.atDay m 1)) (->date (.atEndOfMonth m))]))))))
|
||||
|
||||
(defn orders-in-window
|
||||
"Sales order ids for every client between `start` and `end` inclusive, via the
|
||||
`:sales-order/client+date` index."
|
||||
[db clients start end]
|
||||
(map first (iol-ion.query/scan-sales-orders db clients start end)))
|
||||
|
||||
(defn- all-client-ids [db]
|
||||
(map first (dc/q '[:find ?c :where [?c :client/code _]] db)))
|
||||
|
||||
(defn migrate-all!
|
||||
"The complete migration, over the whole database rather than a chosen subset.
|
||||
|
||||
@@ -342,40 +289,18 @@
|
||||
Nine client pairs contended in the past and no longer share one; their records are still mixed,
|
||||
and a migration scoped to the current configuration would miss every one of them.
|
||||
|
||||
**Ordered so that an interrupted run is recoverable.** Refunds, payouts and cash-drawer shifts
|
||||
go first: together they are a quarter of a million records and take seconds, so finishing them
|
||||
up front means an interruption cannot leave them half done. The long part — walking every order
|
||||
to split shared charges — then runs a month at a time from the current month backwards, logging
|
||||
each month as it completes. Stop it after any month and the data from that month forward is
|
||||
fully scoped, which is the part the importer reads, so imports can resume against it while the
|
||||
older tail waits. Re-running picks up where it left off because each month's work is idempotent.
|
||||
|
||||
Returns the split counts and the completeness report, which should read zero legacy across the
|
||||
board when this finishes."
|
||||
[batch-size]
|
||||
(doseq [{:keys [attr prefix]} scoped-attrs
|
||||
:when (not= attr :charge/external-id)]
|
||||
(let [p (plan (dc/db conn) attr prefix)]
|
||||
(when-let [c (seq (collisions (:new-keys p)))]
|
||||
(throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
|
||||
(migrate! attr (:new-keys p) batch-size)))
|
||||
(let [clients (all-client-ids (dc/db conn))
|
||||
split (reduce (fn [acc [start end]]
|
||||
(let [ids (orders-in-window (dc/db conn) clients start end)
|
||||
r (split-and-rekey-charges! ids batch-size)]
|
||||
(alog/info ::month-complete
|
||||
:month (subs (str (.toInstant ^java.util.Date start)) 0 7)
|
||||
:rekeyed (:rekeyed r) :cloned (:cloned r))
|
||||
(merge-with + acc r)))
|
||||
{:rekeyed 0 :cloned 0}
|
||||
(order-months-newest-first))]
|
||||
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them.
|
||||
;; Collision-checked like the others: this is the largest attribute in the database, so it is
|
||||
;; the last one that should discover a clash as a mid-run exception.
|
||||
(let [split (split-and-rekey-charges! (all-order-ids (dc/db conn)) batch-size)]
|
||||
(doseq [{:keys [attr prefix]} scoped-attrs
|
||||
:when (not= attr :charge/external-id)]
|
||||
(let [p (plan (dc/db conn) attr prefix)]
|
||||
(when-let [c (seq (collisions (:new-keys p)))]
|
||||
(throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
|
||||
(migrate! attr (:new-keys p) batch-size)))
|
||||
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them
|
||||
(let [p (plan (dc/db conn) :charge/external-id charge-prefix)]
|
||||
(when-let [c (seq (collisions (:new-keys p)))]
|
||||
(throw (ex-info "two entities would take the same key"
|
||||
{:attr :charge/external-id :collisions (count c)})))
|
||||
(when (seq (:new-keys p)) (migrate! :charge/external-id (:new-keys p) batch-size)))
|
||||
{:split split :completeness (unscoped-report (dc/db conn))}))
|
||||
|
||||
|
||||
@@ -378,25 +378,75 @@
|
||||
:ledger-mapped/amount (- (+ total discount) tax)
|
||||
#_#_:ledger-mapped/account nil})))
|
||||
|
||||
;; A day carrying refunds and no sales at all is left out of balance on purpose. It is tempting
|
||||
;; to close it by booking a return against the day's refunds — the arithmetic works, and no
|
||||
;; trading day could be affected. Do not. Those days are overwhelmingly not "a refund settled
|
||||
;; while the restaurant was shut": they are days whose *orders were never imported*, on client
|
||||
;; records that took ownership of another record's refunds through the unscoped keys this branch
|
||||
;; fixes. Balancing them would convert the only signal that a client's sales are missing into
|
||||
;; silence. See `docs/2026-08-15-sales-summary-rollout-plan.md`.
|
||||
(def refund-only-returns-flag
|
||||
"Per-client rollout lever for recognising a return on a day that carries refunds but no sales,
|
||||
in the same style as `service-charges-flag`. Absent, the summary behaves exactly as it does
|
||||
today."
|
||||
"summary-refund-only-returns")
|
||||
|
||||
(defn- refund-only-returns-enabled? [c]
|
||||
(contains? (set (:client/feature-flags (dc/pull (dc/db conn) '[:client/feature-flags] c)))
|
||||
refund-only-returns-flag))
|
||||
|
||||
(defn- traded?
|
||||
"Whether the client recorded any sales order at all on `date`."
|
||||
[c date]
|
||||
(some? (ffirst (dc/q '[:find (count ?e)
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where
|
||||
[(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))))
|
||||
|
||||
(defn refunded-total
|
||||
"Face value of every refund recorded for the client on `date`.
|
||||
|
||||
This is the same population the `Card Refunds` / `Cash Refunds` / `Food App Refunds` credits are
|
||||
built from, read back as one figure so a return can be posted against it."
|
||||
[c date]
|
||||
(or (ffirst (dc/q '[:find (sum ?t)
|
||||
:with ?e
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where
|
||||
[(iol-ion.query/scan-sales-refunds $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
|
||||
[?e :sales-refund/total ?t]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))
|
||||
0.0))
|
||||
|
||||
(defn- refund-only-returns
|
||||
"The return owed by a day that recorded refunds and no sales at all.
|
||||
|
||||
Refunds are credited on the day the money goes back; the return that offsets them is read from
|
||||
`:sales-order/returns` on orders scanned for that same day. When the two fall on the same day
|
||||
they cancel. When they do not — a refund settling after the restaurant closed for the day, or a
|
||||
retired duplicate client record that still receives refunds while its orders belong to the
|
||||
surviving record — the credit has nothing to offset it and the day cannot balance, by exactly
|
||||
the refunded amount.
|
||||
|
||||
Only days with *no* sales orders are touched. That guard is what makes this safe rather than a
|
||||
judgement call: with no orders there is no order-derived return to double-count, and no trading
|
||||
day can have its figures moved. Days that do trade and still carry an unmatched refund are left
|
||||
alone deliberately — apportioning a return across a day that also sold is an accounting
|
||||
question, not a mechanical one."
|
||||
[c date]
|
||||
(when (and (refund-only-returns-enabled? c)
|
||||
(not (traded? c date)))
|
||||
(let [amount (refunded-total c date)]
|
||||
(when-not (zero? amount) amount))))
|
||||
|
||||
(defn- get-returns [c date]
|
||||
(when-let [amount (ffirst (dc/q '[:find (sum ?r)
|
||||
:with ?e
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
|
||||
[?e :sales-order/returns ?r]
|
||||
(when-let [amount (or (ffirst (dc/q '[:find (sum ?r)
|
||||
:with ?e
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
|
||||
[?e :sales-order/returns ?r]
|
||||
|
||||
#_[?e :sales-order/charges ?c]
|
||||
#_[?c :charge/tax ?tax]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))]
|
||||
#_[?e :sales-order/charges ?c]
|
||||
#_[?c :charge/tax ?tax]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))
|
||||
(refund-only-returns c date))]
|
||||
{:db/id (str (java.util.UUID/randomUUID))
|
||||
:sales-summary-item/category "Returns"
|
||||
|
||||
|
||||
@@ -27,9 +27,11 @@
|
||||
"Authorization" (str "Bearer " (:client/square-auth-token client))
|
||||
"Content-Type" "application/json"}))
|
||||
|
||||
|
||||
(defn ->square-date [d]
|
||||
(f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d))
|
||||
|
||||
|
||||
(def manifold-api-stream
|
||||
(let [stream (s/stream 100)]
|
||||
(->> stream
|
||||
@@ -40,10 +42,10 @@
|
||||
(de/loop [attempt 0]
|
||||
(-> (de/chain (de/future-with (ex/execute-pool)
|
||||
#_(log/info ::request-started
|
||||
:url (:url request)
|
||||
:attempt attempt
|
||||
:source "Square 3"
|
||||
:background-job "Square 3")
|
||||
:url (:url request)
|
||||
:attempt attempt
|
||||
:source "Square 3"
|
||||
:background-job "Square 3")
|
||||
(try
|
||||
(client/request (assoc request
|
||||
:socket-timeout 10000
|
||||
@@ -102,6 +104,7 @@
|
||||
:exception error))
|
||||
[]))))
|
||||
|
||||
|
||||
(def item-cache (atom {}))
|
||||
|
||||
(defn fetch-catalog [client i v]
|
||||
@@ -121,11 +124,13 @@
|
||||
#(do (swap! item-cache assoc i %)
|
||||
%))))
|
||||
|
||||
|
||||
(defn fetch-catalog-cache [client i version]
|
||||
(if (get @item-cache i)
|
||||
(de/success-deferred (get @item-cache i))
|
||||
(fetch-catalog client i version)))
|
||||
|
||||
|
||||
(defn item->category-name-impl [client item version]
|
||||
(capture-context->lc
|
||||
(cond (:item_id (:item_variation_data item))
|
||||
@@ -156,6 +161,7 @@
|
||||
:item item)
|
||||
"Uncategorized"))))
|
||||
|
||||
|
||||
(defn item-id->category-name [client i version]
|
||||
(capture-context->lc
|
||||
(-> [client i]
|
||||
@@ -220,6 +226,7 @@
|
||||
(concat (:orders result) continued-results))))
|
||||
(:orders result)))))))
|
||||
|
||||
|
||||
(defn search
|
||||
([client location start end]
|
||||
(capture-context->lc
|
||||
@@ -243,9 +250,11 @@
|
||||
(concat (:orders result) continued-results))))
|
||||
(:orders result))))))))
|
||||
|
||||
|
||||
(defn amount->money [amt]
|
||||
(* 0.01 (or (:amount amt) 0.0)))
|
||||
|
||||
|
||||
;; to get totals:
|
||||
(comment
|
||||
(reduce
|
||||
@@ -269,30 +278,6 @@
|
||||
[prefix client location id]
|
||||
(str prefix (:client/code client) "-" (:square-location/client-location location) "-" id))
|
||||
|
||||
(def ^:private owner-attr
|
||||
"Where each Square-imported entity records the client it belongs to."
|
||||
{:charge/external-id :charge/client
|
||||
:sales-refund/external-id :sales-refund/client
|
||||
:expected-deposit/external-id :expected-deposit/client
|
||||
:cash-drawer-shift/external-id :cash-drawer-shift/client})
|
||||
|
||||
(defn- owned-by-other-client?
|
||||
"Whether `e` already belongs to a client other than `client-eid`.
|
||||
|
||||
Reads the entity's own owner attribute, and for a charge falls back to the client of whichever
|
||||
sales order refers to it — charges predating `:charge/client` still have orders, and those are
|
||||
exactly the ones that can be taken by the wrong client."
|
||||
[db attr e client-eid]
|
||||
(let [ent (dc/entity db e)
|
||||
owner (or (:db/id ((owner-attr attr) ent))
|
||||
(when (= attr :charge/external-id)
|
||||
(some->> (first (dc/datoms db :vaet e :sales-order/charges))
|
||||
:e
|
||||
(dc/entity db)
|
||||
:sales-order/client
|
||||
:db/id)))]
|
||||
(and owner (not= owner client-eid))))
|
||||
|
||||
(defn existing-id
|
||||
"Entity id of the refund or charge this id already refers to, trying the client-scoped key
|
||||
first and the legacy unscoped key second.
|
||||
@@ -301,24 +286,11 @@
|
||||
relies on upsert-by-identity; changing the key format on its own would match nothing and
|
||||
Datomic would create a SECOND entity for every refund and charge, orphaning the original under
|
||||
its legacy key. Pinning the result as `:db/id` makes the write land on the existing entity
|
||||
whichever scheme it currently carries.
|
||||
|
||||
The legacy branch will not take a record that already belongs to a different client. Without
|
||||
that check, two clients on one Square location double money during the window between deploying
|
||||
and finishing the migration: client A's payout import resolves B's charge by its bare key and
|
||||
renames it into A's scope, B's next order import then matches neither scheme and mints a second
|
||||
charge, and because `:sales-order/charges` is cardinality-many nothing retracts the first — so
|
||||
B's order carries two charges for one payment. Declining is also the right answer on its merits:
|
||||
the write then lands on this client's own copy, which is what the scoped keys exist to create.
|
||||
|
||||
Once the migration has run there are no legacy keys left for this branch to find, and both it
|
||||
and the guard can be deleted together."
|
||||
whichever scheme it currently carries."
|
||||
[db attr prefix client location id]
|
||||
(when id
|
||||
(or (dc/entid db [attr (scoped-key prefix client location id)])
|
||||
(when-let [legacy (dc/entid db [attr (str prefix id)])]
|
||||
(when-not (owned-by-other-client? db attr legacy (:db/id client))
|
||||
legacy)))))
|
||||
(dc/entid db [attr (str prefix id)]))))
|
||||
|
||||
(defn tender->charge [order client location t]
|
||||
(remove-nils
|
||||
@@ -332,7 +304,7 @@
|
||||
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location (:id t))
|
||||
:external-id (when (:id t)
|
||||
(scoped-key "square/charge/" client location (:id t)))
|
||||
:processor (cond
|
||||
:processor (cond
|
||||
(#{"OTHER" "THIRD_PARTY_CARD"} (:type t))
|
||||
(condp = (some-> (:note t) str/lower-case)
|
||||
"doordash" :ccp-processor/doordash
|
||||
@@ -405,7 +377,7 @@
|
||||
#:sales-order
|
||||
{:date (if (= "Invoices" (:name (:source order)))
|
||||
(when (:closed_at order)
|
||||
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
|
||||
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
|
||||
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:created_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
|
||||
:client (:db/id client)
|
||||
:location (:square-location/client-location location)
|
||||
@@ -467,6 +439,7 @@
|
||||
:client client
|
||||
:location location)))))))
|
||||
|
||||
|
||||
(defn get-payment [client p]
|
||||
(de/chain (manifold-api-call
|
||||
{:url (str "https://connect.squareup.com/v2/payments/" p)
|
||||
@@ -475,6 +448,7 @@
|
||||
:body
|
||||
:payment))
|
||||
|
||||
|
||||
(defn continue-payout-entry-list [c l poi cursor]
|
||||
(capture-context->lc lc
|
||||
(de/chain
|
||||
@@ -619,12 +593,6 @@
|
||||
(let [payment-id (:payment_id (:type_charge_details p))]
|
||||
(remove-nils
|
||||
{:charge/external-id (scoped-key "square/charge/" client location payment-id)
|
||||
;; the owner attributes must travel with the key: `raw-square-id` and
|
||||
;; `scope-of` both recover a charge's scope from them, and a key
|
||||
;; scoped to one client while the owner says another is what makes
|
||||
;; the migration write square/charge/B-LB-A-LA-<id>
|
||||
:charge/client (:db/id client)
|
||||
:charge/location (:square-location/client-location location)
|
||||
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location payment-id)}))))))})
|
||||
(filter :expected-deposit/date)
|
||||
(into []))
|
||||
@@ -632,44 +600,15 @@
|
||||
(log/error ::transform-payout-failed
|
||||
:exception e)))))))))
|
||||
|
||||
(defn- refund-list
|
||||
"Every refund Square has for this location in `[start end]`, following the cursor to the end.
|
||||
|
||||
The list endpoint returns one page at a time. Reading only the first page — which is what this
|
||||
did before — silently caps a location at a hundred refunds however many it actually has, and
|
||||
the cap is invisible: the response looks like a complete answer. On a shared location that is
|
||||
how one client record ends up holding a few refunds against a hundred and fifty thousand orders.
|
||||
|
||||
`start`/`end` are optional; omitting both asks for everything, which is what the nightly job
|
||||
wants and what a historical backfill of more than a page needs."
|
||||
([client l start end] (refund-list client l start end nil))
|
||||
([client l start end cursor]
|
||||
(de/chain (manifold-api-call
|
||||
{:url (str "https://connect.squareup.com/v2/refunds"
|
||||
"?"
|
||||
(url/map->query
|
||||
(cond-> {:location_id (:square-location/square-id l)
|
||||
:limit 100}
|
||||
start (assoc :begin_time (->square-date start))
|
||||
end (assoc :end_time (->square-date end))
|
||||
cursor (assoc :cursor cursor))))
|
||||
:method :get
|
||||
:headers (client-base-headers client)
|
||||
:as :json})
|
||||
:body
|
||||
(fn [result]
|
||||
(log/info ::refunds-page
|
||||
:count (count (:refunds result))
|
||||
:more? (boolean (not-empty (:cursor result))))
|
||||
(if (not-empty (:cursor result))
|
||||
(de/chain (refund-list client l start end (:cursor result))
|
||||
(fn [more] (concat (:refunds result) more)))
|
||||
(:refunds result))))))
|
||||
|
||||
(defn refunds
|
||||
([client l] (refunds client l nil nil))
|
||||
([client l start end]
|
||||
(de/chain (refund-list client l start end)
|
||||
([client l]
|
||||
(de/chain (manifold-api-call {:url (str "https://connect.squareup.com/v2/refunds?location_id=" (:square-location/square-id l))
|
||||
:method :get
|
||||
|
||||
:headers (client-base-headers client)
|
||||
:as :json})
|
||||
:body
|
||||
:refunds
|
||||
(fn [refunds]
|
||||
(->> refunds
|
||||
(filter (fn [r] (= "COMPLETED" (:status r))))
|
||||
@@ -712,6 +651,7 @@
|
||||
:count (count x))
|
||||
@(dc/transact-async conn x))))))))
|
||||
|
||||
|
||||
(defn upsert-payouts
|
||||
([client]
|
||||
(apply de/zip
|
||||
@@ -740,12 +680,11 @@
|
||||
(for [square-location (:client/square-locations client)
|
||||
:when (:square-location/client-location square-location)]
|
||||
(upsert-refunds client square-location))))
|
||||
([client location] (upsert-refunds client location nil nil))
|
||||
([client location start end]
|
||||
([client location]
|
||||
(with-context-as {:source "Square refunds loading"
|
||||
:client (:client/code client)} lc
|
||||
|
||||
(de/chain (refunds client location start end)
|
||||
(de/chain (refunds client location)
|
||||
(fn [refunds]
|
||||
(mu/with-context lc
|
||||
(try
|
||||
@@ -761,6 +700,7 @@
|
||||
|
||||
(log/info ::done-loading-refunds)))))))
|
||||
|
||||
|
||||
(defn get-cash-shift [client id]
|
||||
(de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id))
|
||||
:method :get
|
||||
@@ -924,6 +864,8 @@
|
||||
d1
|
||||
d2))
|
||||
|
||||
|
||||
|
||||
(defn remove-voided-orders
|
||||
([client]
|
||||
(apply de/zip
|
||||
@@ -950,7 +892,7 @@
|
||||
(:sales-order/external-id o))))))
|
||||
(s/map (fn [[o]]
|
||||
[[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]]))
|
||||
|
||||
|
||||
(s/reduce into [])))
|
||||
|
||||
(fn [results]
|
||||
@@ -959,26 +901,31 @@
|
||||
(log/info ::removing-orders
|
||||
:count (count x))
|
||||
@(dc/transact-async conn x)))))
|
||||
(de/catch (fn [e]
|
||||
(log/warn ::couldnt-remove :error e)
|
||||
nil)))))))
|
||||
(de/catch (fn [e]
|
||||
(log/warn ::couldnt-remove :error e)
|
||||
nil) ))))))
|
||||
|
||||
#_(comment
|
||||
(require 'auto-ap.time-reader)
|
||||
#_(comment
|
||||
(require 'auto-ap.time-reader)
|
||||
|
||||
@(let [[c [l]] (get-square-client-and-location "DBFS")]
|
||||
(log/peek :x [c l])
|
||||
(search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29"))
|
||||
@(let [[c [l]] (get-square-client-and-location "DBFS") ]
|
||||
(log/peek :x [ c l])
|
||||
(search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29")
|
||||
|
||||
@(let [[c [l]] (get-square-client-and-location "NGAK")]
|
||||
(log/peek :x [c l])
|
||||
)
|
||||
|
||||
(remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
|
||||
(doseq [c (get-square-clients)]
|
||||
(try
|
||||
@(remove-voided-orders c)
|
||||
(catch Exception e
|
||||
nil))))
|
||||
@(let [[c [l]] (get-square-client-and-location "NGAK") ]
|
||||
(log/peek :x [ c l])
|
||||
|
||||
(remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
|
||||
(doseq [c (get-square-clients)]
|
||||
(try
|
||||
@(remove-voided-orders c)
|
||||
(catch Exception e
|
||||
nil)))
|
||||
|
||||
|
||||
)
|
||||
|
||||
(defn upsert-all [& clients]
|
||||
(capture-context->lc
|
||||
@@ -1041,53 +988,14 @@
|
||||
(s/realize-each)
|
||||
(s/reduce conj []))))
|
||||
|
||||
(defn backfill-history
|
||||
"Re-imports orders, payouts, refunds and cash-drawer shifts for `[start end]`, one client at a
|
||||
time, for every square location the client has.
|
||||
|
||||
This exists for the shared-location case. Sales orders have always been keyed by client, so two
|
||||
client records on one Square location each built their own order history. Refunds, payouts and
|
||||
shifts were not, so only ONE of the two records holds each of them — whichever imported it last
|
||||
before the keys were scoped. Re-keying freezes that ownership; it does not even it out, and the
|
||||
record left without them shows returns from its own orders with no refunds to offset them.
|
||||
|
||||
Rather than manufacture copies, this asks Square again. With client-scoped keys in place every
|
||||
record now creates its own copy of what it reads, so replaying the window is what makes the two
|
||||
histories match. Deliberately not part of `upsert-all`: it walks further back than the nightly
|
||||
job and is meant to be run once, after the migration.
|
||||
|
||||
Run it AFTER `rekey-square-external-ids/migrate-all!`. Running it before would import against
|
||||
legacy keys and leave more to migrate."
|
||||
[start end & client-codes]
|
||||
(with-context-as {:source "Square historical backfill"} lc
|
||||
(->> (apply get-square-clients client-codes)
|
||||
(s/->source)
|
||||
(s/map (fn [client]
|
||||
(with-context-as (merge lc {:client (:client/code client)}) lc
|
||||
(->
|
||||
(apply de/zip
|
||||
(for [l (:client/square-locations client)
|
||||
:when (:square-location/client-location l)]
|
||||
(de/chain
|
||||
(upsert client l start end)
|
||||
(fn [_] (upsert-payouts client l start end))
|
||||
(fn [_] (upsert-refunds client l start end))
|
||||
(fn [_] (upsert-cash-shifts client l start end))
|
||||
(fn [_] (log/info ::backfilled
|
||||
:location (:square-location/client-location l))))))
|
||||
(de/catch (fn [e]
|
||||
(mu/with-context lc
|
||||
(log/info ::backfill-failed :severity :error :exception e))))))))
|
||||
(s/buffer 3)
|
||||
(s/realize-each)
|
||||
(s/reduce conj []))))
|
||||
|
||||
(defn do-upsert-all [& clients]
|
||||
(mu/trace
|
||||
::upsert-all
|
||||
[:clients clients]
|
||||
@(apply upsert-all clients)))
|
||||
|
||||
|
||||
|
||||
(comment
|
||||
(defn refunds-raw-cont
|
||||
([client l cursor so-far]
|
||||
@@ -1117,8 +1025,9 @@
|
||||
(->>
|
||||
@(let [[c [l]] (get-square-client-and-location "NGGG")]
|
||||
|
||||
(search c l (time/now) (time/plus (time/now) (time/days -1))))
|
||||
|
||||
(search c l (time/now) (time/plus (time/now) (time/days -1))))
|
||||
|
||||
(filter (fn [r]
|
||||
(str/starts-with? (:created_at r) "2024-03-14"))))
|
||||
|
||||
@@ -1126,6 +1035,7 @@
|
||||
(->>
|
||||
@(let [[c [l]] (get-square-client-and-location "NGGG")]
|
||||
|
||||
|
||||
(refunds-raw-cont c l nil []))
|
||||
(filter (fn [r]
|
||||
(str/starts-with? (:created_at r) "2024-03-14")))))
|
||||
@@ -1159,8 +1069,13 @@
|
||||
[]))]
|
||||
[(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row) (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
|
||||
:separator \tab)
|
||||
|
||||
|
||||
|
||||
|
||||
;; =>
|
||||
|
||||
|
||||
;; =>
|
||||
|
||||
(require 'auto-ap.time-reader)
|
||||
|
||||
@@ -1169,16 +1084,27 @@
|
||||
(clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")]
|
||||
l
|
||||
|
||||
|
||||
(def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00"
|
||||
#clj-time/date-time "2025-02-28T00:00:00-08:00"))
|
||||
(take 10 (map #(first (deref (order->sales-order c l %))) z))))
|
||||
(take 10 (map #(first (deref (order->sales-order c l %))) z)))
|
||||
|
||||
(->> z
|
||||
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
(->> z
|
||||
(filter (fn [o]
|
||||
(seq (filter (comp #{"OTHER"} :type) (:tenders o)))))
|
||||
(filter #(not (:name (:source %))))
|
||||
(count))
|
||||
|
||||
(count)
|
||||
|
||||
)
|
||||
|
||||
|
||||
|
||||
(doseq [[code] (seq (dc/q '[:find ?code
|
||||
:in $
|
||||
:where [?o :sales-order/date ?d]
|
||||
@@ -1187,22 +1113,32 @@
|
||||
[?o :sales-order/client ?c]
|
||||
[?c :client/code ?code]]
|
||||
(dc/db conn)))
|
||||
:let [[c [l]] (get-square-client-and-location code)]
|
||||
:let [[c [l]] (get-square-client-and-location code)
|
||||
]
|
||||
order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now))
|
||||
:when (= "Invoices" (:name (:source order)))
|
||||
:when (= "Invoices" (:name (:source order) ))
|
||||
:let [[sales-order] @(order->sales-order c l order)]]
|
||||
|
||||
|
||||
(when (should-import-order? order)
|
||||
(println "DATE IS" (:sales-order/date sales-order))
|
||||
(when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after? #clj-time/date-time "2026-2-16T00:00:00-08:00"))
|
||||
(println "WOULD UPDATE" sales-order)
|
||||
@(dc/transact auto-ap.datomic/conn [sales-order]))
|
||||
#_@(dc/transact)
|
||||
(println "DONE")))
|
||||
@(dc/transact auto-ap.datomic/conn [sales-order])
|
||||
)
|
||||
#_@(dc/transact )
|
||||
(println "DONE"))
|
||||
|
||||
|
||||
)
|
||||
|
||||
#_(filter (comp #{"OTHER"} :type) (mapcat :tenders z))
|
||||
|
||||
|
||||
@(let [[c [l]] (get-square-client-and-location "NGRY")]
|
||||
#_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
|
||||
|
||||
(order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY")))))
|
||||
(order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY" )))
|
||||
|
||||
)
|
||||
|
||||
)
|
||||
|
||||
@@ -147,47 +147,3 @@
|
||||
(is (= (str "square/charge/" code "-CD-same1") (:key (first (charges-of o1)))))
|
||||
(is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
|
||||
"both orders still point at the one payment"))))
|
||||
|
||||
(deftest the-month-walk-runs-newest-first-and-leaves-no-gaps
|
||||
(testing "order matters operationally, not just cosmetically: the importer reads recent data, so
|
||||
an interrupted migration is only safe to resume imports against if the newest months
|
||||
are the ones already done. Walking :aevt instead would start in 2019."
|
||||
(let [windows (sut/order-months-newest-first #inst "2026-01-01T12:00:00.000-00:00")
|
||||
starts (map first windows)]
|
||||
(is (seq windows))
|
||||
(is (apply > (map #(.getTime ^java.util.Date %) starts))
|
||||
"strictly descending — newest month first")
|
||||
(is (every? (fn [[[next-start _] [_ prev-end]]]
|
||||
(= (.getTime ^java.util.Date next-start)
|
||||
(+ (.getTime ^java.util.Date prev-end) (* 24 60 60 1000))))
|
||||
(partition 2 1 windows))
|
||||
"each window ends the day before the next one starts, so no order falls between them")
|
||||
(is (every? (fn [[s e]] (.before ^java.util.Date s ^java.util.Date e)) windows)
|
||||
"and every window is non-empty"))))
|
||||
|
||||
(deftest two-orders-of-the-same-client-keep-sharing-across-batches
|
||||
(testing "batch size does not change the same-client rule, which the sibling test cannot show
|
||||
because both its orders land in one batch.
|
||||
|
||||
Once the first order re-keys the charge it also writes :charge/client/:charge/location,
|
||||
so the second order's raw-square-id takes its owner branch, new-key reconstructs the key
|
||||
the charge already has, and the (not= old new-key) guard drops the row before :action is
|
||||
read. A clone here would double that client's takings for the day."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])
|
||||
tx @(dc/transact conn [{:db/id "charge"
|
||||
:charge/external-id "square/charge/same2"
|
||||
:charge/total 75.0}
|
||||
{:db/id "o1" :sales-order/external-id "square/order/y-1"
|
||||
:sales-order/client test-client-id :sales-order/location "CD"
|
||||
:sales-order/date sales-date :sales-order/charges ["charge"]}
|
||||
{:db/id "o2" :sales-order/external-id "square/order/y-2"
|
||||
:sales-order/client test-client-id :sales-order/location "CD"
|
||||
:sales-order/date sales-date :sales-order/charges ["charge"]}])
|
||||
o1 (get-in tx [:tempids "o1"]) o2 (get-in tx [:tempids "o2"])
|
||||
code (:client/code (dc/entity (dc/db conn) test-client-id))]
|
||||
(is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [o1 o2] 1))
|
||||
"batch size 1 puts the two orders in separate batches, and still no copy is made")
|
||||
(is (= 1 (charge-count)) "one payment, not two")
|
||||
(is (= (str "square/charge/" code "-CD-same2") (:key (first (charges-of o1)))))
|
||||
(is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
|
||||
"both orders still point at the one payment"))))
|
||||
|
||||
@@ -134,29 +134,50 @@
|
||||
:sales-refund/type "CARD"
|
||||
:sales-refund/total total})
|
||||
|
||||
(defn- enable-refund-only-returns! [client]
|
||||
@(dc/transact conn [{:db/id client
|
||||
:client/feature-flags [sut/refund-only-returns-flag]}]))
|
||||
|
||||
(defn- returns-for [client]
|
||||
(#'sut/get-returns client sales-date))
|
||||
|
||||
(deftest a-refund-with-no-sales-leaves-the-day-out-of-balance
|
||||
(testing "deliberate, and load-bearing. Booking a return against the day's refunds would close
|
||||
it and is tempting for that reason. But a day with refunds and no sales at all is
|
||||
overwhelmingly a day whose ORDERS WERE NEVER IMPORTED — on a restored copy of
|
||||
production, 132 of 156 such days fell before their client's first ever synced order.
|
||||
Balancing them would turn the only signal that a client's sales are missing into
|
||||
silence. If this test starts failing, read the rollout plan before changing it."
|
||||
(deftest refund-only-day-needs-the-feature-flag
|
||||
(testing "without the flag a day of refunds and no sales books no return, as it does today"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(refund test-client-id "no-sales" 40.0)])
|
||||
(is (nil? (returns-for test-client-id))
|
||||
"no return is invented for a day that recorded no sales")
|
||||
(is (= -40.0 (d-ss/imbalance (sut/get-refund-items test-client-id sales-date)))
|
||||
"so the day stays out of balance by the refunded amount, visibly"))))
|
||||
@(dc/transact conn [(refund test-client-id "unflagged" 40.0)])
|
||||
(is (nil? (returns-for test-client-id))))))
|
||||
|
||||
(deftest a-day-that-traded-books-its-own-return
|
||||
(testing "the ordinary case: the return comes from the day's orders, never from its refunds"
|
||||
(deftest refund-only-day-books-a-return_and_balances
|
||||
(testing "a refund credited on a day with no sales has nothing to offset it, so the day is out
|
||||
by the refunded amount until a return is recognised against it"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-refund-only-returns! test-client-id)
|
||||
@(dc/transact conn [(refund test-client-id "orphan-a" 30.0)
|
||||
(refund test-client-id "orphan-b" 10.0)])
|
||||
(let [returns (returns-for test-client-id)]
|
||||
(is (= 40.0 (:ledger-mapped/amount returns)))
|
||||
(is (= :ledger-side/debit (:ledger-mapped/ledger-side returns)))
|
||||
(is (= 0.0 (d-ss/imbalance (cons returns (sut/get-refund-items test-client-id sales-date))))
|
||||
"the refund credits and the return debit cancel exactly")))))
|
||||
|
||||
(deftest a-day-that-traded-keeps-its-order-derived-return
|
||||
(testing "the guard is what makes this safe: a day with sales is left entirely alone, so no
|
||||
trading day can have its return moved by an unmatched refund"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-refund-only-returns! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "traded" {:sales-order/returns 7.0})
|
||||
(refund test-client-id "same-day" 40.0)])
|
||||
(is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id)))))))
|
||||
(is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id)))
|
||||
"the order's own return, not the refunded total"))))
|
||||
|
||||
(deftest a-day-that-traded-and-returned-nothing-books-no-return
|
||||
(testing "sales with no returns must not pick up the refunded total either — the guard is on
|
||||
whether the client traded, not on whether the order-derived figure happened to be nil"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-refund-only-returns! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "traded-no-returns" {})
|
||||
(refund test-client-id "unmatched" 40.0)])
|
||||
(is (nil? (returns-for test-client-id))))))
|
||||
|
||||
(deftest dirty-summaries-stop-at-the-client-boundary
|
||||
(testing "every dirty day for the client is returned, and none belonging to another client.
|
||||
|
||||
@@ -76,89 +76,6 @@
|
||||
:sales-refund/total 10.0}])
|
||||
(is (= 2 (refund-count)) "two stable entities, one per client, rather than one that flips"))))
|
||||
|
||||
(deftest legacy-key-of-another-client-is-not-claimed
|
||||
(testing "the legacy fallback must not hand one client a record that already belongs to another.
|
||||
|
||||
Without this, two clients on one Square location double money in the window between
|
||||
deploying and finishing the migration."
|
||||
(setup-test-data [])
|
||||
(let [tx @(dc/transact conn [{:db/id "mine" :client/code (str "MINE" (rand-int 100000))}
|
||||
{:db/id "theirs" :client/code (str "THEIRS" (rand-int 100000))}])
|
||||
mine {:db/id (get-in tx [:tempids "mine"]) :client/code "MINE"}
|
||||
theirs-id (get-in tx [:tempids "theirs"])]
|
||||
@(dc/transact conn [{:db/id "r"
|
||||
:sales-refund/external-id "square/refund/abc"
|
||||
:sales-refund/client theirs-id
|
||||
:sales-refund/total 10.0}])
|
||||
(is (nil? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/"
|
||||
mine location "abc"))
|
||||
"a legacy-keyed refund owned by another client is left alone")
|
||||
(is (some? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/"
|
||||
{:db/id theirs-id :client/code "THEIRS"} location "abc"))
|
||||
"its own client still resolves it, so re-keying in place still works"))))
|
||||
|
||||
(deftest a-charge-is-owned-by-the-client-of-the-order-that-refers-to-it
|
||||
(testing "charges predating :charge/client still have orders, and those are exactly the ones
|
||||
that could be taken by the wrong client"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])
|
||||
other (get-in @(dc/transact conn [{:db/id "o" :client/code (str "OTHER" (rand-int 100000))}])
|
||||
[:tempids "o"])]
|
||||
@(dc/transact conn [{:db/id "c" :charge/external-id "square/charge/p1" :charge/total 50.0}
|
||||
{:db/id "ord" :sales-order/external-id "square/order/x-1"
|
||||
:sales-order/client test-client-id :sales-order/location "CD"
|
||||
:sales-order/date #inst "2026-06-03T07:00:00.000-00:00"
|
||||
:sales-order/charges ["c"]}])
|
||||
(is (nil? (sut/existing-id (dc/db conn) :charge/external-id "square/charge/"
|
||||
{:db/id other :client/code "OTHER"} location "p1"))
|
||||
"ownership is read from the referencing order when :charge/client is absent"))))
|
||||
|
||||
(deftest deploy-window-does-not-double-a-second-clients-tender
|
||||
(testing "the P0 this guard exists for, end to end.
|
||||
|
||||
Client A's payout import reaches for a payment whose charge belongs to client B's
|
||||
order. If A were allowed to re-key it, B's next order import would match neither
|
||||
scheme, mint a second charge, and — since :sales-order/charges is cardinality-many —
|
||||
leave B's order holding two charges for one payment."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])
|
||||
b-code (:client/code (dc/entity (dc/db conn) test-client-id))
|
||||
b {:db/id test-client-id :client/code b-code}
|
||||
b-loc {:square-location/client-location "LB"}
|
||||
a-id (get-in @(dc/transact conn [{:db/id "a" :client/code (str "AAA" (rand-int 100000))}])
|
||||
[:tempids "a"])
|
||||
a {:db/id a-id :client/code (:client/code (dc/entity (dc/db conn) a-id))}
|
||||
a-loc {:square-location/client-location "LA"}
|
||||
tx @(dc/transact conn [{:db/id "x" :charge/external-id "square/charge/P"
|
||||
:charge/total 100.0 :charge/type-name "CARD"
|
||||
:charge/client test-client-id :charge/location "LB"}
|
||||
{:db/id "ob" :sales-order/external-id "square/order/b-1"
|
||||
:sales-order/client test-client-id :sales-order/location "LB"
|
||||
:sales-order/date #inst "2026-06-03T07:00:00.000-00:00"
|
||||
:sales-order/charges ["x"]}])
|
||||
order-b (get-in tx [:tempids "ob"])
|
||||
charges-of (fn [o] (map :v (dc/datoms (dc/db conn) :eavt o :sales-order/charges)))]
|
||||
;; client A's payout import touches the same Square payment
|
||||
@(dc/transact conn [(into {} (remove (comp nil? val))
|
||||
{:charge/external-id (sut/scoped-key "square/charge/" a a-loc "P")
|
||||
:charge/client a-id
|
||||
:charge/location "LA"
|
||||
:db/id (sut/existing-id (dc/db conn) :charge/external-id
|
||||
"square/charge/" a a-loc "P")})])
|
||||
;; client B's order re-imports
|
||||
@(dc/transact conn [{:db/id order-b
|
||||
:sales-order/charges
|
||||
[(sut/tender->charge {:id "b-1" :created_at "2026-06-03T12:00:00Z"}
|
||||
b b-loc {:id "P" :type "CARD"
|
||||
:amount_money {:amount 10000
|
||||
:currency "USD"}})]}])
|
||||
(is (= 1 (count (charges-of order-b)))
|
||||
"B's order still holds exactly one charge for the one payment")
|
||||
(is (= 100.0 (reduce + 0.0 (map #(:charge/total (dc/entity (dc/db conn) %))
|
||||
(charges-of order-b))))
|
||||
"so the day's tender is not doubled")
|
||||
(is (= (str "square/charge/" b-code "-LB-P")
|
||||
(:charge/external-id (dc/entity (dc/db conn) (first (charges-of order-b)))))
|
||||
"and B's own charge was re-keyed in place rather than abandoned"))))
|
||||
|
||||
(deftest payouts-and-shifts-are-client-scoped-too
|
||||
(testing "expected deposits and cash drawer shifts are fetched per location, so two clients on
|
||||
one location collide on them exactly as refunds and charges did"
|
||||
|
||||
Reference in New Issue
Block a user