Compare commits
1 Commits
worktree-s
...
sales-summ
| Author | SHA1 | Date | |
|---|---|---|---|
| f8ef7918ef |
@@ -1,393 +0,0 @@
|
||||
---
|
||||
title: Sales Summary Balance — validation against a restored production backup
|
||||
type: validation
|
||||
date: 2026-08-15
|
||||
status: measured
|
||||
---
|
||||
|
||||
# Sales Summary Balance — validation against a restored production backup
|
||||
|
||||
Measures the two calculation fixes (R1 reversed tips, R2 Square service charges) against real
|
||||
production data, and checks them against the predictions in
|
||||
[`docs/plans/2026-08-13-fix-sales-summary-balance-plan.md`](plans/2026-08-13-fix-sales-summary-balance-plan.md).
|
||||
|
||||
**Headline: the fixes behave as predicted.** R2 clears 307 client-days with zero regressions.
|
||||
After both fixes, the entire remaining dollar error outside the contended clients is $73.98,
|
||||
and all but six client-days of it is sub-10¢ register rounding.
|
||||
|
||||
---
|
||||
|
||||
## 1. What was measured against
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Database | `datomic:dev://localhost:4337/integreat-prod-restore` |
|
||||
| Restored from | `s3://integreat-backups/`, restore point **209608347** |
|
||||
| Basis-t | 209608347 |
|
||||
| Newest transaction | **2026-08-14 22:52 local** |
|
||||
| Clients | 210 (106 with orders in the window) |
|
||||
| Refunds / charges / orders | 50,986 / 16,545,495 / 19,039,575 |
|
||||
| Window | 2026-07-13 → 2026-08-14 |
|
||||
| Population | **3,342 client-days** |
|
||||
|
||||
The earlier restore in this directory was **stale** — the backup had written no root since
|
||||
2025-03-10 even though segments kept uploading daily through 2026-07-12. A fresh production
|
||||
backup on 2026-08-14 produced root 209608347, which is what this measures. The staleness was a
|
||||
production backup failure of roughly 17 months, and is worth treating as its own issue.
|
||||
|
||||
The plan's population was 2,314 client-days because it counted existing summary entities. This
|
||||
counts every client-day that actually has orders, which is the fuller denominator; percentages
|
||||
are comparable, absolute counts are not.
|
||||
|
||||
## 2. Method
|
||||
|
||||
Each client-day is composed exactly as `sales-summaries-v2` composes it — the same
|
||||
`get-sales`, `get-payment-items`, `get-refund-items`, `get-discounts`, `get-fees`, `get-tax`,
|
||||
`get-returns` — and its imbalance taken as `d-ss/imbalance`, i.e. debits minus credits. Three
|
||||
variants differ only in the tip and service-charge terms:
|
||||
|
||||
- **Baseline** — tips from tenders only, no service-charge credit. What production does today.
|
||||
- **R1** — tips additive: tendered tips plus tips on orders carrying no tender.
|
||||
- **R2** — R1 plus Square service charges, both signs.
|
||||
|
||||
Nothing was transacted. A day counts as off when `|imbalance| >= $0.005`, i.e. it does not
|
||||
round to zero.
|
||||
|
||||
## 3. Results
|
||||
|
||||
| Stage | Days off | Clean | Total \|Δ\| |
|
||||
|---|---|---|---|
|
||||
| Baseline (production today) | 569 | 82.97% | $32,921.40 |
|
||||
| + R1 additive tip | 493 | 85.25% | $32,437.67 |
|
||||
| **+ R2 Square service charges** | **186** | **94.43%** | **$5,166.49** |
|
||||
|
||||
The plan predicted 82.7% clean at baseline including duplicates. Measured: **82.97%**.
|
||||
|
||||
### Fixed / regressed — the plan's pre-merge gate
|
||||
|
||||
| Change | Fixed | Regressed |
|
||||
|---|---|---|
|
||||
| R1 vs baseline | 78 | **2** |
|
||||
| R2 vs R1 | 307 | **0** |
|
||||
|
||||
R2's gate — "days going balanced → unbalanced must be 0, or each explained" — **passes
|
||||
outright**.
|
||||
|
||||
R1's two regressions are both days that balanced only because two errors cancelled, exactly the
|
||||
mechanism the plan anticipated:
|
||||
|
||||
| Client | Date | Before | After | Untendered tip |
|
||||
|---|---|---|---|---|
|
||||
| NGCD | 2026-08-08 | $0.00 | +$11.80 | −$11.80 |
|
||||
| NGDG | 2026-08-05 | $0.00 | +$10.00 | −$10.00 |
|
||||
|
||||
Both clients are halves of duplicate Square-location pairs (NGCD/NGCC, NGDG/NGDU), so these are
|
||||
expected to resolve under Phase 0 rather than needing a change to R1.
|
||||
|
||||
## 4. Where the remaining error lives
|
||||
|
||||
Splitting the post-R2 residual by whether the client shares a Square location with another
|
||||
client:
|
||||
|
||||
| Group | Client-days | Days off | Total \|Δ\| |
|
||||
|---|---|---|---|
|
||||
| 10 duplicate pairs (20 clients) | 546 | 112 | **$5,092.51** |
|
||||
| Everyone else (96 clients) | 2,796 | 74 | **$73.98** |
|
||||
|
||||
**98.6% of the remaining dollar error sits in the contended clients** — which is Phase 0's to
|
||||
fix, not a calculation defect. This is the plan's central claim, and it holds.
|
||||
|
||||
### The $73.98 residual, in full
|
||||
|
||||
| Client | Days | Total \|Δ\| | |
|
||||
|---|---|---|---|
|
||||
| PNSP | 30 | $0.91 | register rounding |
|
||||
| PNLP | 26 | $0.77 | register rounding |
|
||||
| GLAD | 12 | $0.20 | register rounding |
|
||||
| NGEB | 2 | $45.00 | ezCater fee semantics (plan §15.4) |
|
||||
| NGPS | 1 | $9.62 | unexplained |
|
||||
| NGGB | 1 | $7.59 | partial final day |
|
||||
| NGSC | 1 | $5.02 | partial final day |
|
||||
| NGTY | 1 | $4.87 | partial final day |
|
||||
|
||||
Applying Phase 6's 10¢ materiality threshold leaves **6 material client-days**. The largest
|
||||
sub-threshold day is **9.00¢**, so 10¢ separates rounding noise from real variance with no day
|
||||
sitting near the boundary — the threshold is empirically justified, not arbitrary.
|
||||
|
||||
Three of the six material days — NGGB, NGSC, NGTY — all fall on **2026-08-14**, the day the
|
||||
backup was taken at 22:52, and no other date carries more than one material day. They are
|
||||
partial-day artifacts of the backup cut, not defects.
|
||||
|
||||
That leaves exactly the residual the plan predicted, to the cent:
|
||||
|
||||
- **NGEB 2026-08-10 −$25.00** and **NGEB 2026-07-29 −$20.00** — predicted
|
||||
- **NGPS 2026-08-12 +$9.62** — predicted, still unexplained
|
||||
|
||||
## 5. Confirmed independently: the dropped refund line
|
||||
|
||||
The plan reports NGCD 2026-07-23 missing a $71.94 refund line because the refund belonged to
|
||||
NGCC at recompute time. That client-day measures **+$71.94** out of balance here, and neither R1
|
||||
nor R2 moves it — correct, since it is a duplicate-client defect rather than a calculation one.
|
||||
|
||||
## 6. Amendment to Phase 1 — charges that cannot be scoped
|
||||
|
||||
The re-key pre-flight was run against the restore.
|
||||
|
||||
**Refunds — clean.** All 50,986 carry both a client code and a location; 0 unscopable, 0
|
||||
collisions. Collisions are in fact structurally impossible: the old key is `:db.unique/identity`
|
||||
and the new key is a deterministic function of that same id plus client and location, so two
|
||||
distinct entities cannot converge. The run confirms the reasoning.
|
||||
|
||||
**Charges — not clean.** Of 16,545,495 charges, **2,122,161 (12.8%) have neither
|
||||
`:charge/client` nor `:charge/location`**, so they cannot be scoped from the entity itself.
|
||||
A 5,000-entity sample profiles them as:
|
||||
|
||||
| | Count | Share |
|
||||
|---|---|---|
|
||||
| Referenced only by an expected-deposit, no total/type/date | 4,897 | 98% |
|
||||
| Referenced by a sales order, real tender records | 290 | 5.8% |
|
||||
| Orphaned — reachable from nothing | **0** | 0% |
|
||||
|
||||
These are stubs created by the payout path, which asserts `{:charge/external-id ...}` alone and
|
||||
lets unique-identity upsert bring a bare entity into existence.
|
||||
|
||||
**Consequence:** Phase 1 cannot read scoping off the charge. The migration must derive client and
|
||||
location from the referencing order or expected-deposit — roughly 123 K charges via orders and
|
||||
2.0 M via deposits, extrapolated. Nothing is unreachable, so the phase remains feasible, but the
|
||||
step as written in §4.2 would strand 12.8% of charges on the legacy scheme.
|
||||
|
||||
The payout call site in `square.core3` needs the same treatment: it constructs
|
||||
`(str "square/charge/" payment-id)` and would keep minting legacy-scheme stubs after the cutover.
|
||||
It has `client` and `location` in scope already, so the fix is local.
|
||||
|
||||
## 7. Not yet run
|
||||
|
||||
- **Phase 0** itself — deactivating the duplicate client per location, then re-measuring. The
|
||||
numbers above isolate the contended clients rather than deactivating them.
|
||||
- **Charges referenced by more than one order** (§3.3), the gate on any historical cleanup.
|
||||
- The account for service charges is **49000 Service Income**, chosen to make measurement
|
||||
possible. It does not affect balance at all — only whether a day can reach `accepted?` — so
|
||||
every number here is independent of that choice.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Part 2 — executed end to end on the restore
|
||||
|
||||
Part 1 measured what the fixes *would* do without writing anything. This part actually ran them:
|
||||
Phase 0, the re-key migration, a live Square import, and a full recompute through the real job.
|
||||
All windows re-cut to end **2026-08-13**, since the backup was taken mid-evening on the 14th and
|
||||
that partial day was polluting the comparison.
|
||||
|
||||
## What was executed, in order
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| Phase 0 — deactivate the newer twin at all 10 shared locations | 10 datoms retracted; no location shared by two clients any more |
|
||||
| Phase 1 — re-key refunds | 50,986 re-keyed in 1.9 s, 0 collisions, 0 unscopable |
|
||||
| Phase 1 — re-key contended clients' charges | 56,829 re-keyed, 0 collisions |
|
||||
| **Count-unchanged assertion** | refunds 50,986 → 50,986, charges 16,545,495 → 16,545,495 |
|
||||
| Live Square import, 10 surviving clients | all succeeded |
|
||||
| Recompute | 6,720 client-days through `sales-summaries-v2` |
|
||||
|
||||
The count-unchanged assertion is the one that matters: 107,815 entities were re-keyed and **not a
|
||||
single entity was created or destroyed**, which is what proves the expand step resolved legacy
|
||||
keys rather than duplicating everything under the new scheme.
|
||||
|
||||
## Deduplication works on an ongoing basis
|
||||
|
||||
- **The deactivated twins received nothing.** Across all 10, orders and refunds were byte-identical
|
||||
before and after a live import. Phase 0 holds.
|
||||
- **Refunds stopped alternating and became two stable copies.** 98 of NGCC's 125 refunds are now
|
||||
also held by NGCD as separate entities. Each client owns its own; neither can take the other's.
|
||||
This is the intended behaviour of client-scoped keys, and it is what makes the data
|
||||
reproducible — but note it double counts at the group level, which is why Phase 0 (one client
|
||||
per location) is the actual fix and the re-key is the safety net.
|
||||
|
||||
### The dropped refund line, fixed
|
||||
|
||||
NGCD 2026-07-23 was out of balance by exactly **+$71.94** with no refund line, because the refund
|
||||
belonged to NGCC at recompute time. After the full pipeline the summary carries a **"Card Refunds"
|
||||
line of $71.94** and the day's imbalance is **9.1e-13 — zero**.
|
||||
|
||||
## Results
|
||||
|
||||
Reportable population excludes the 10 deactivated twins, which the plan says to exclude from
|
||||
reporting.
|
||||
|
||||
| | Client-days | Days off | Clean | Total \|Δ\| |
|
||||
|---|---|---|---|---|
|
||||
| Before (production code, nothing applied) | 3,236 | 542 | 83.25% | $30,982.11 |
|
||||
| **After (everything applied)** | 3,030 | **71** | **97.66%** | **$405.79** |
|
||||
|
||||
A **98.7% reduction in dollar error**. With the 10¢ materiality threshold, **5 material days
|
||||
remain**:
|
||||
|
||||
| Client | Date | Δ | |
|
||||
|---|---|---|---|
|
||||
| NGBK | 2026-08-06 | +$299.42 | new — see below |
|
||||
| NGDA | 2026-08-01 | −$50.00 | the known auto-gratuity-as-service-charge order |
|
||||
| NGEB | 2026-08-10 | −$25.00 | ezCater fee semantics, predicted |
|
||||
| NGEB | 2026-07-29 | −$20.00 | ezCater fee semantics, predicted |
|
||||
| NGPS | 2026-08-12 | +$9.62 | predicted, still unexplained |
|
||||
|
||||
The three predicted residuals reproduced exactly. NGBK 08-06 is new and appeared only after the
|
||||
import: NGBK went from 5 refunds to 105, because it now holds its own copies of refunds that had
|
||||
been sitting under its twin NGBR. That is the double-counting consequence of two clean copies, and
|
||||
it is an argument for Phase 0 being the real fix rather than the re-key.
|
||||
|
||||
## Two defects found by executing rather than simulating
|
||||
|
||||
### 1. Re-keying a shared charge duplicates the tender
|
||||
|
||||
**Found by measurement, not by reading.** After the import, the contended clients' total imbalance
|
||||
was **$441,045**, roughly 100× worse than predicted. The cause:
|
||||
|
||||
1. 35,870 of 56,829 charges had **two parent orders**, one per twin.
|
||||
2. The migration's `scope-of` resolved such a charge through whichever referencing order it found
|
||||
first — sometimes the twin's — so it was re-keyed to, say, `square/charge/NGDU-DU-<id>`.
|
||||
3. On import, NGDG computed `square/charge/NGDG-DB-<id>`, did not find it, fell back to the legacy
|
||||
key — which the migration had already removed — and so **created a second charge**.
|
||||
4. `:sales-order/charges` is cardinality **many**, so the new charge was *added* beside the stale
|
||||
one. One order, two tenders for one payment. Tender roughly doubled.
|
||||
|
||||
Confirmed on a single order: `square/order/NGDG-DB-IKiIDdTK…` referencing both
|
||||
`square/charge/NGDU-DU-dEhvoH50…` and `square/charge/NGDG-DB-dEhvoH50…`.
|
||||
|
||||
This is exactly what plan §4.3 warned about — re-keying gives each charge one new key, it does not
|
||||
**split** a charge with two parents — and it was under-weighted when the migration was written.
|
||||
|
||||
**Before this migration ships it needs one of:**
|
||||
- split shared charges (clone per client) as part of the migration rather than re-keying in place, or
|
||||
- have the order upsert *reset* its charge refs rather than accumulate them, or
|
||||
- run the re-key only after the duplicate clients' orders are gone.
|
||||
|
||||
On the restore this was repaired by retracting 15,961 stale charge refs — for each order, keeping
|
||||
the charge scoped to its own client — after which no order had two charges for one payment id, and
|
||||
the contended clients' error fell from $441,045 to $1,492.
|
||||
|
||||
### 2. `dirty-sales-summaries` scans the whole index per client
|
||||
|
||||
```clojure
|
||||
(dc/index-pull db {:index :avet :start [:sales-summary/client+dirty [client-id true]]})
|
||||
```
|
||||
|
||||
There is no `:end`, and the client filter that follows is a lazy `filter` which does not stop the
|
||||
scan. So for every client the job walks every summary of every client sorting after it, pulling
|
||||
their items. It is O(n²) in the number of summaries.
|
||||
|
||||
Observed: the recompute ran at ~180 client-days/minute early and collapsed to ~3/minute as
|
||||
summaries accumulated — a full pass took over two hours for 6,720 days. Adding an `:end` bound at
|
||||
the client boundary should fix it.
|
||||
|
||||
## Method note
|
||||
|
||||
The 96 non-contended clients were measured from the **persisted summaries** the job wrote, and
|
||||
came to $56.37 across 69 days — identical to the in-memory prediction in Part 1, which validates
|
||||
the job path end to end. The 20 contended clients were re-measured in memory after the charge
|
||||
repair, because a second full pass through the job would have taken hours for the reason in defect
|
||||
2. The two methods agree exactly where both were run.
|
||||
|
||||
## Ongoing deduplication, with both clients active again
|
||||
|
||||
The re-key is meant to make contention impossible even if a location is configured on two clients
|
||||
again. To test that rather than assume it, all 10 twins were **re-activated** — every location
|
||||
shared once more — and the import re-run.
|
||||
|
||||
**Repeated imports are idempotent.** Three consecutive refund imports for both halves of the
|
||||
Concord pair, with both clients live on location `L43Z5GMW72VMG`:
|
||||
|
||||
| | run 0 | run 1 | run 2 | run 3 |
|
||||
|---|---|---|---|---|
|
||||
| NGCC | 127 | 127 | 127 | 127 |
|
||||
| NGCD | 1,419 | 1,419 | 1,419 | 1,419 |
|
||||
|
||||
Not one entity created or moved. Before the re-key this is the exact configuration that made a
|
||||
refund's owner flip every ~20 minutes.
|
||||
|
||||
The first import after the migration is the only one that grows anything, and it grows once: NGCD
|
||||
gained 100 refunds materialising its own copies, while NGCC — whose refunds already carried its
|
||||
own scoped key — gained only the 2 that were genuinely new. After that, steady state.
|
||||
|
||||
**State the restore was left in:** twins re-activated, i.e. Phase 0 deliberately undone for this
|
||||
test. Re-applying it is the ten-datom retraction recorded above.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Part 3 — all summaries updated, fixes compared to baseline
|
||||
|
||||
Every summary in the database was recomputed — **14,458 client-days**, 2024-04-01 through
|
||||
2026-08-14 — through `sales-summaries-v2`, with zero failures and zero left dirty. The
|
||||
comparison below is the **last 30 days, 2026-07-15 → 2026-08-13**.
|
||||
|
||||
Both arms are measured on the same, already-deduplicated data (Phase 0 applied, refunds and the
|
||||
contended clients' charges re-keyed, duplicate charge refs repaired), so this isolates what the
|
||||
**calculation fixes** are worth on top of the deduplication work.
|
||||
|
||||
## Making the full recompute possible
|
||||
|
||||
Two changes were needed before recomputing everything was practical:
|
||||
|
||||
1. **`dirty-sales-summaries` scanned to the end of the index** (fixed in
|
||||
`perf(sales-summaries)`): 1,321 ms → **5.6 ms** per client, a 237× improvement, identical
|
||||
results.
|
||||
2. **The driver was one serial `doseq`.** `refresh-client!` was split out of `sales-summaries-v2`
|
||||
so a client's work stands alone. Spread across threads, the remaining 5,463 client-days
|
||||
finished in about 90 seconds — the serial run was tracking at roughly nine more hours.
|
||||
|
||||
## Method
|
||||
|
||||
Baseline is derived rather than recomputed: R1 and R2 only ever *add credits*, so
|
||||
|
||||
```
|
||||
baseline imbalance = fixed imbalance + untendered tip + service charges
|
||||
```
|
||||
|
||||
This identity was checked against a full from-scratch baseline recomputation on 20 randomly
|
||||
sampled client-days and agreed on every one, to within a hundredth of a cent.
|
||||
|
||||
## Results — last 30 days, excluding the 10 deactivated twins
|
||||
|
||||
2,842 client-days across 96 clients.
|
||||
|
||||
| | Days off | Clean | Total \|Δ\| | Material (≥10¢) |
|
||||
|---|---|---|---|---|
|
||||
| **Baseline** (production calculation) | 398 | 86.00% | $22,527.40 | 336 |
|
||||
| **Fixed** (R1 + R2) | **67** | **97.64%** | **$405.66** | **5** |
|
||||
|
||||
**331 client-days fixed, 0 regressed.** Not one day that balanced under the baseline stopped
|
||||
balancing under the fixes, across the whole window.
|
||||
|
||||
Including the deactivated twins (3,040 client-days) the picture is the same shape: 432 → 70 days
|
||||
off, $25,622.98 → $1,548.64.
|
||||
|
||||
### What is left
|
||||
|
||||
| Client | Date | Δ | |
|
||||
|---|---|---|---|
|
||||
| NGBK | 2026-08-06 | +$299.42 | refunds now held by both twins — see below |
|
||||
| NGDA | 2026-08-01 | −$50.00 | auto-gratuity booked as a service charge |
|
||||
| NGEB | 2026-08-10 | −$25.00 | ezCater fee semantics, predicted by the plan |
|
||||
| NGEB | 2026-07-29 | −$20.00 | ezCater fee semantics, predicted by the plan |
|
||||
| NGPS | 2026-08-12 | +$9.62 | predicted by the plan, still unexplained |
|
||||
|
||||
Everything else — 62 client-days — totals **$1.62**, with the largest single day at **9.00¢**.
|
||||
The 10¢ materiality threshold cleanly separates register rounding from real variance, with
|
||||
nothing sitting near the boundary.
|
||||
|
||||
NGBK is the one genuinely new residual and it is a consequence of the re-key rather than the
|
||||
calculation: NGBK went from 5 refunds to 105 because it now holds its own copies of refunds that
|
||||
had been sitting under its twin NGBR. Two stable copies is the intended behaviour, but it double
|
||||
counts at the group level — which is the argument for Phase 0 being the real fix and the re-key
|
||||
being the safety net.
|
||||
|
||||
### Representative days the fixes repair outright
|
||||
|
||||
| Client | Date | Baseline → Fixed |
|
||||
|---|---|---|
|
||||
| NTPT | 2026-08-06 | $427.10 → $0.00 |
|
||||
| NGFO | 2026-07-16 | $275.23 → $0.00 |
|
||||
| NGRN | 2026-08-06 | $236.96 → $0.00 |
|
||||
| N-30008 | 2026-07-24 | $229.42 → $0.00 |
|
||||
| N-30003 | 2026-07-18 | $225.00 → $0.00 |
|
||||
@@ -92,7 +92,7 @@
|
||||
<header class="masthead">
|
||||
<div class="eyebrow">Sales summaries · measured on a restored production backup</div>
|
||||
<h1>Ninety-Day Reconciliation</h1>
|
||||
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
|
||||
<p class="standfirst">Four faults were leaving restaurant days out of balance — one in the data, three in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
|
||||
<div class="meta">
|
||||
<span><b>Window</b> 2026-05-10 → 2026-08-07</span>
|
||||
<span><b>Client-days</b> <span class="num">18,900</span></span>
|
||||
@@ -111,26 +111,26 @@
|
||||
<div class="arrow" aria-hidden="true">→</div>
|
||||
<div>
|
||||
<span class="side-label">The same ninety days, fixes on</span>
|
||||
<span class="figure after num">$7,790.54</span>
|
||||
<span class="subfig"><span class="num">279</span> days out of balance · <span class="num">98.52%</span> clean</span>
|
||||
<span class="figure after num">$2,970.35</span>
|
||||
<span class="subfig"><span class="num">123</span> days out of balance · <span class="num">99.35%</span> clean</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="stats">
|
||||
<div class="stat"><span class="k num">979</span><span class="l">client-days brought into balance</span></div>
|
||||
<div class="stat"><span class="k num">1,135</span><span class="l">client-days brought into balance</span></div>
|
||||
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
|
||||
<div class="stat"><span class="k num">88.8%</span><span class="l">of the variance removed</span></div>
|
||||
<div class="stat"><span class="k num">95.7%</span><span class="l">of the variance removed</span></div>
|
||||
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
|
||||
</div>
|
||||
|
||||
<div class="measure">
|
||||
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p>
|
||||
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>979 days and $61,769.56</strong>, not the larger number a stale baseline would have shown.</p>
|
||||
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, service charges customers paid were credited to nothing, and a refund that arrived on a day with no sales had nothing to offset it.</p>
|
||||
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>1,135 days and $66,589.75</strong>, not the larger number a stale baseline would have shown.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>The three problems</h2>
|
||||
<h2>The four problems</h2>
|
||||
|
||||
<div class="problem">
|
||||
<h4>1. Two client records sharing one Square location</h4>
|
||||
@@ -160,14 +160,24 @@
|
||||
<p><span class="tech">technical</span> <code>get-tip</code> summed tips by joining through <code>:sales-order/charges</code>, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on <code>:sales-order/tip</code>. Nothing at all read <code>:sales-order/service-charge</code>.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="problem">
|
||||
<h4>4. A refund on a day the restaurant did not sell anything</h4>
|
||||
<div class="measure">
|
||||
<p><strong>For the business:</strong> a refund is recorded on the day the money goes back to the guest. The matching reduction in sales is read from that day's orders. When both fall on the same day they cancel and the day balances. When they do not — a refund settling after a closed day, or a retired duplicate record that still receives refunds while its orders belong to the surviving record — the day is left carrying a refund with nothing against it, and is out by exactly the refunded amount.</p>
|
||||
<p><strong>This is the answer to why the duplicated restaurants looked so much worse than everyone else.</strong> Of the days still failing after the first three fixes, <strong>155 of the 158 on duplicate records had no sales orders at all</strong> — the summary consisted of nothing but orphaned refunds and their fees. The refunds were claimed by the record that happened to import them during the shared period; the orders went to the other record. Nothing was double-counted and no money was lost — the two halves of each refund simply ended up on two different sets of books.</p>
|
||||
<p>It is not a duplicate-only problem, which is what makes it worth fixing properly rather than papering over. The single largest non-duplicate cluster, <span class="mono">NG4S</span>, has the identical shape: no orders on any day of the window, refunds arriving anyway, and every one of those days out by exactly the refund.</p>
|
||||
<p><span class="tech">technical</span> <code>get-returns</code> summed <code>:sales-order/returns</code> over orders scanned for the date. With no orders the sum is nil and no <code>Returns</code> line is written at all, while <code>get-refund-items</code> still credits <code>Card Refunds</code> from the <code>sales-refund</code> records — a credit with no debit anywhere in the day.</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>What the fixes actually are</h2>
|
||||
<div class="measure">
|
||||
<p>Five changes. The first three stop two clients from sharing a record; the last two record
|
||||
money that was being collected but not booked. Each is small — the difficulty was knowing
|
||||
which line to change, not writing it.</p>
|
||||
<p>Six changes. The first three stop two clients from sharing a record; the last three record
|
||||
money that was being collected, or handed back, but not booked. Each is small — the difficulty
|
||||
was knowing which line to change, not writing it.</p>
|
||||
</div>
|
||||
|
||||
<h3>1 · Put the client in the record's name</h3>
|
||||
@@ -295,6 +305,41 @@
|
||||
few restaurants at a time.</p>
|
||||
</div>
|
||||
|
||||
<h3>6 · Recognise a return on a day that refunded but did not sell</h3>
|
||||
<div class="measure">
|
||||
<p>Where a day has refunds and no sales orders whatsoever, book a <code>Returns</code> debit
|
||||
equal to that day's refunds. The guard — <em>no orders at all</em> — is what makes this
|
||||
mechanical rather than a judgement call.</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<pre><span class="dim">;; the return a day owes when it refunded but never sold</span>
|
||||
(defn- refund-only-returns [c date]
|
||||
(when (and (refund-only-returns-enabled? c)
|
||||
(not (traded? c date)))
|
||||
(let [amount (refunded-total c date)]
|
||||
(when-not (zero? amount) amount))))
|
||||
|
||||
<span class="dim">;; get-returns falls back to it only when the order-derived figure is absent</span>
|
||||
(or (ffirst (dc/q <span class="dim">;; sum of :sales-order/returns, as before</span>))
|
||||
(refund-only-returns c date))</pre>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>Why the guard is the whole design.</strong> On a day with no orders the
|
||||
order-derived return is necessarily nothing, so there is no figure to double-count, and no
|
||||
trading day can have its numbers moved by this rule — which the measurement bears out: 156
|
||||
days came into balance, none went out, and not one already-balanced day had a line change.
|
||||
Days that <em>did</em> trade and still carry an unmatched refund are deliberately left alone;
|
||||
apportioning a return across a day that also sold is an accounting question, not a mechanical
|
||||
one, and guessing at it would be exactly the kind of change that quietly rewrites correct
|
||||
books.</p>
|
||||
<p>Like the service-charge line, it sits behind a per-client switch that is off by default.
|
||||
Four tests hold it: flag off changes nothing; a refund-only day balances exactly; a day that
|
||||
traded keeps its own return; and a day that traded but returned nothing does not pick up the
|
||||
refunded total either — that last one guards the difference between "the client did not
|
||||
trade" and "the order-derived figure happened to be nil", which is the subtle way this rule
|
||||
could have gone wrong.</p>
|
||||
</div>
|
||||
|
||||
<h3>Supporting changes</h3>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
@@ -327,12 +372,13 @@
|
||||
<tbody>
|
||||
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,258</td><td class="n">93.34%</td><td class="n">$69,560.10</td></tr>
|
||||
<tr><td>+ refunded tips</td><td class="n">971</td><td class="n">94.86%</td><td class="n">$66,414.39</td></tr>
|
||||
<tr class="total"><td>+ service charges</td><td class="n good">279</td><td class="n good">98.52%</td><td class="n good">$7,790.54</td></tr>
|
||||
<tr><td>+ service charges</td><td class="n">279</td><td class="n">98.52%</td><td class="n">$7,790.54</td></tr>
|
||||
<tr class="total"><td>+ returns on refund-only days</td><td class="n good">123</td><td class="n good">99.35%</td><td class="n good">$2,970.35</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$61,769.56 is what the two arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
|
||||
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$66,589.75 is what the three arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
|
||||
</div>
|
||||
|
||||
<h3>Day-by-day effect of each change</h3>
|
||||
@@ -342,14 +388,15 @@
|
||||
<tbody>
|
||||
<tr><td>Refunded tips</td><td class="n">18,590</td><td class="n good">287</td><td class="n good">0</td><td class="n good">0</td><td class="n">$3,712.67</td></tr>
|
||||
<tr><td>Service charges</td><td class="n">18,208</td><td class="n good">692</td><td class="n good">0</td><td class="n good">0</td><td class="n">$58,349.85</td></tr>
|
||||
<tr class="total"><td>Both, end to end</td><td class="n">17,916</td><td class="n good">979</td><td class="n good">0</td><td class="n good">0</td><td class="n">$61,769.56</td></tr>
|
||||
<tr><td>Returns on refund-only days</td><td class="n">18,740</td><td class="n good">156</td><td class="n good">0</td><td class="n good">0</td><td class="n">$4,820.19</td></tr>
|
||||
<tr class="total"><td>All three, end to end</td><td class="n">17,765</td><td class="n good">1,135</td><td class="n good">0</td><td class="n good">0</td><td class="n">$66,589.75</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>Neither fix touched a day that was already correct.</strong> Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered — 17,916 summaries came out byte-identical, and every one of the 984 that moved was already wrong. Service charges are by far the larger of the two, moving $58,349.85 against the tip fix's $3,712.67.</p>
|
||||
<p><strong>None of the three touched a day that was already correct.</strong> Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered. Every day that moved was already wrong. Service charges are by far the largest of the three, moving $58,349.85 against $4,820.19 for refund-only returns and $3,712.67 for tips.</p>
|
||||
<p>That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.</p>
|
||||
<p><strong>The two fixes account for the repair exactly.</strong> Adding up the untendered-tip and service-charge amounts for the 979 repaired days leaves a residue of <span class="mono">0.0000000013</span> against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a third effect.</p>
|
||||
<p><strong>The arithmetic fixes account for their repair exactly.</strong> Adding up the untendered-tip and service-charge amounts for the 979 days those two closed leaves a residue of <span class="mono">0.0000000013</span> against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a further effect.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -434,7 +481,7 @@
|
||||
|
||||
<div class="callout">
|
||||
<span class="h">The whole analysis was run again from nothing, and landed in the same place</span>
|
||||
<p>Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. The end state matched the previous run <strong>to the cent</strong> — 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.</p>
|
||||
<p>Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. At the point the two runs are comparable — before the refund-only fix below, which the earlier run did not have — the end state matched <strong>to the cent</strong>: 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.</p>
|
||||
</div>
|
||||
|
||||
<div class="callout warn">
|
||||
@@ -446,36 +493,36 @@
|
||||
<section>
|
||||
<h2>What is still out of balance</h2>
|
||||
<div class="measure">
|
||||
<p>279 client-days out of 18,900, totalling <strong>$7,790.54</strong>. Where it sits matters more than the total, because most of it is not on a restaurant anyone reports on.</p>
|
||||
<p>123 client-days out of 18,900, totalling <strong>$2,970.35</strong> — and only 33 of those are above ten cents. Where it sits matters more than the total.</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
<thead><tr><th>Where the remainder sits</th><th class="n">Days</th><th class="n">Variance</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>The twenty client records tied to a shared location — including the ten now-dormant duplicates</td><td class="n">160</td><td class="n">$6,059.93</td></tr>
|
||||
<tr class="total"><td>Every other client</td><td class="n">119</td><td class="n">$1,730.61</td></tr>
|
||||
<tr><td>The twenty client records tied to a shared location — including the ten now-dormant duplicates</td><td class="n">20</td><td class="n">$2,467.39</td></tr>
|
||||
<tr class="total"><td>Every other client</td><td class="n">103</td><td class="n">$502.96</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>Of that $1,730.61 on ordinary clients, four account for all but $2.40 of it:</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
<thead><tr><th>Client</th><th class="n">Days</th><th class="n">Variance</th><th>What it is</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="mono">NG4S</td><td class="n">10</td><td class="n">$1,066.61</td><td>undiagnosed — the largest single unexplained cluster</td></tr>
|
||||
<tr><td class="mono">NGDL</td><td class="n">3</td><td class="n">$859.57</td><td>order on one record, its payment on the twin</td></tr>
|
||||
<tr><td class="mono">NGDU</td><td class="n">3</td><td class="n">$345.96</td><td>payout fees on days with no trading</td></tr>
|
||||
<tr><td class="mono">NGBR</td><td class="n">1</td><td class="n">$299.42</td><td rowspan="2">Square recorded $6,358.99 of tender against $6,059.57 of order totals on 2026-08-06 — the gap itself, not a summary fault</td></tr>
|
||||
<tr><td class="mono">NGBK</td><td class="n">1</td><td class="n">$299.42</td></tr>
|
||||
<tr><td class="mono">NGMV</td><td class="n">5</td><td class="n">$259.38</td><td>late May, undiagnosed</td></tr>
|
||||
<tr><td class="mono">NGEB</td><td class="n">4</td><td class="n">$199.09</td><td>ezCater fee treatment — an open question</td></tr>
|
||||
<tr><td class="mono">NGPS</td><td class="n">7</td><td class="n">$172.82</td><td>undiagnosed</td></tr>
|
||||
<tr><td class="mono">N-30012</td><td class="n">2</td><td class="n">$30.31</td><td>late May, undiagnosed</td></tr>
|
||||
<tr class="total"><td class="dim">PNSP, PNLP, GLAD and others</td><td class="n dim">91</td><td class="n dim">$2.40</td><td class="dim">till rounding — pennies a day</td></tr>
|
||||
<tr><td class="mono">NGVC</td><td class="n">3</td><td class="n">$173.81</td><td>payout fees on days with no trading</td></tr>
|
||||
<tr><td class="mono">NGWN</td><td class="n">3</td><td class="n">$170.28</td><td>payout fees on days with no trading</td></tr>
|
||||
<tr class="total"><td class="dim">everyone else</td><td class="n dim">100</td><td class="n dim">$363.42</td><td class="dim">90 of these are till rounding — pennies a day</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>The largest surviving single day is NGDL on 2026-08-01 at −$810.24, one of the shared-location records. The matched pair NGBR and NGBK both sit at exactly ±$299.42 on 2026-08-06: Square recorded $6,358.99 of tender against $6,059.57 of order totals, and the difference is that gap rather than anything the summary did wrong. The tip fix improved that day — it was $436.42 before — without closing it.</p>
|
||||
<p>The clusters on NG4S, NGMV, NGPS and N-30012 are unexplained and worth a look before this ships. They are visible only because the window is ninety days; a thirty-day view does not reach them.</p>
|
||||
<p><strong>What is left divides cleanly in two.</strong> 108 days totalling $1,995.36 are real trading days with genuine discrepancies — the NGBR/NGBK tender gap, the ezCater fee question, and the unexplained clusters. The other 15 days, $974.99, carry a processing-fee debit and nothing else: a payout fee landing on a day the restaurant did not trade. That is the same shape as the refund-only day and could be closed the same way, but the offsetting entry belongs to the payout rather than to sales, so it needs the payout side modelled rather than another rule in the summary. It is not worth holding this work for.</p>
|
||||
<p>The clusters on NGMV and NGEB are unexplained and worth a look, though at $458 across nine days they are no longer urgent. They are visible only because the window is ninety days; a thirty-day view does not reach them.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -577,6 +624,8 @@
|
||||
<tbody>
|
||||
<tr><td>Which client record survives at each shared location</td><td>the business</td><td>the newer record generally has no history before the split, so keeping it loses years of the location's books</td></tr>
|
||||
<tr><td>Which revenue account service charges post to</td><td>accounting</td><td>currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances</td></tr>
|
||||
<tr><td>Recognising a return on a refund-only day</td><td>accounting</td><td>posts to 41300 Returns, the account already used for returns; it moves the recognition date, so a refund settling after month end lands in the later period</td></tr>
|
||||
<tr><td>Refunds on days that <em>did</em> trade</td><td>accounting</td><td>deliberately untouched — apportioning a return across a day that also sold is a judgement call, and 108 days remain out of balance because of it</td></tr>
|
||||
<tr><td>Whether to correct records the wrong client already owns</td><td>the business</td><td>the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared</td></tr>
|
||||
<tr><td><code>remove-voided-orders</code></td><td>engineering</td><td>safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes</td></tr>
|
||||
</tbody>
|
||||
@@ -637,7 +686,7 @@
|
||||
|
||||
<footer>
|
||||
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
|
||||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 279 remaining days, 189 are material.</span>
|
||||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 123 remaining days, 33 are material.</span>
|
||||
<span>Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on <code>as-of</code> above.</span>
|
||||
</footer>
|
||||
|
||||
217
docs/2026-08-15-sales-summary-rollout-plan.md
Normal file
217
docs/2026-08-15-sales-summary-rollout-plan.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Sales-summary balancing — rollout plan
|
||||
|
||||
Steps to execute, in order. Every step is either reversible or verifiable before the next one
|
||||
begins. Both behaviour changes are behind per-client feature flags that are **off by default**, so
|
||||
merging and deploying this branch changes nothing on its own.
|
||||
|
||||
Measured on a restored copy of production (backup point `209608347`), 210 clients over
|
||||
2026-05-10 → 2026-08-07: **1,258 client-days out of balance / $69,560.10 → 123 days / $2,970.35**,
|
||||
with zero days knocked out of balance and zero already-balanced days altered.
|
||||
|
||||
---
|
||||
|
||||
## Before you start
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Branch | `worktree-sales-summary-balance` |
|
||||
| Flags introduced | `summary-service-charges`, `summary-refund-only-returns` — both off by default |
|
||||
| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` |
|
||||
| Expected migration runtime | ~13 minutes for 19M orders on a warm cache |
|
||||
| Nothing here touches | invoices, payments, the ledger, or any client without the flags set |
|
||||
|
||||
**One prerequisite that is not code.** Ten Square locations are configured against two client records
|
||||
each. Someone in the business has to decide which record survives at each. The newer record usually
|
||||
has no history from before the split, so keeping it loses years of that location's books. Do this
|
||||
before step 3.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Deploy the code
|
||||
|
||||
Deploy the branch as normal. Both flags are absent from every client, so:
|
||||
|
||||
- tips are calculated exactly as they are today,
|
||||
- no `Service Charges` line is written,
|
||||
- no `Returns` line is written on refund-only days.
|
||||
|
||||
The only changes that take effect immediately are the safe ones: imbalance logging, the
|
||||
dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix,
|
||||
and the importer's new client-scoped keys.
|
||||
|
||||
**The importer starts writing client-scoped keys straight away, and reads both schemes.** That is
|
||||
deliberate and is what makes the deploy independent of the migration. Do not remove the legacy
|
||||
lookup in `square.core3/existing-id` yet — see step 8.
|
||||
|
||||
**Verify before moving on.** After one nightly import cycle:
|
||||
|
||||
```clojure
|
||||
;; refunds, payouts and shifts must not have doubled
|
||||
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))
|
||||
```
|
||||
|
||||
Compare against the same counts taken immediately before deploy. Growth should be ordinary daily
|
||||
volume. A near-doubling means the legacy fallback is not working — **stop and roll back the deploy**.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Guard `remove-voided-orders`
|
||||
|
||||
Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so
|
||||
retracting an order cascades into its payments. Until step 3 finishes there are still payments with
|
||||
two parent orders, and deleting one client's voided order can take the other client's payment with
|
||||
it.
|
||||
|
||||
Either leave `remove-voided-orders` switched off until step 3 completes, or change it to detach a
|
||||
payment that has more than one parent rather than delete it. Detaching is worth doing regardless —
|
||||
it makes the safety a property of the deletion rather than of the migration having been run first.
|
||||
|
||||
See `docs/2026-08-15-remove-voided-orders-risk.md`.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Retire the duplicate client records
|
||||
|
||||
Business decision from the top of this document. Deactivate the losing record's Square location so
|
||||
the importer stops fetching for it. The record itself stays; its history is untouched.
|
||||
|
||||
**Verify:** no Square location is configured against two active client records.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Run the migration
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.jobs.rekey-square-external-ids :as rk])
|
||||
|
||||
;; read-only first — check :collisions is empty for every attribute
|
||||
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
|
||||
|
||||
;; then the whole thing
|
||||
(rk/migrate-all! 2000)
|
||||
```
|
||||
|
||||
Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that
|
||||
already carries the right name is skipped, so it can be stopped and re-run without consequence.
|
||||
|
||||
If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
|
||||
not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
|
||||
retained memory in the REPL and re-measure before changing anything about the database.
|
||||
|
||||
**Verify — all four must read zero to migrate and zero unscopable:**
|
||||
|
||||
```clojure
|
||||
(rk/unscoped-report (d/db conn))
|
||||
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
|
||||
;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0}
|
||||
|
||||
;; and the gate that this work exists for
|
||||
(rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn))))
|
||||
;; => 0
|
||||
```
|
||||
|
||||
Note `unscoped-report`'s `:no-owner` column is not a gap: ~283k payout-stub payments carry no
|
||||
`:charge/client` attribute of their own, so it cannot verify them by attribute. `plan` resolves
|
||||
ownership through whatever refers to them and is the figure to trust.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Recompute summaries, flags still off
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.jobs.sales-summaries :as ss])
|
||||
(ss/refresh-sales-summaries 90)
|
||||
```
|
||||
|
||||
This is the pass that banks the deduplication. **Capture the result before going further** — you
|
||||
will need it as the baseline for step 6, and it cannot be reconstructed afterwards:
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath
|
||||
(def before (cmp/summaries-in (d/db conn) start end))
|
||||
(spit "before.edn" (pr-str before))
|
||||
```
|
||||
|
||||
> **Do not use `d/as-of` to compare summary amounts.** `:ledger-mapped/amount`, `ledger-side` and
|
||||
> `account` are `:db/noHistory`, so past values are discarded. A summary that has since been
|
||||
> recomputed reads back through `as-of` with its amounts *absent*, which looks like a legitimate
|
||||
> balanced day. Capture live, before and after, and diff the captures.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Turn the flags on, a few restaurants at a time
|
||||
|
||||
Needs accounting sign-off first, on two points:
|
||||
|
||||
- `summary-service-charges` posts to **49000 Service Income**. Chosen so the work could be measured.
|
||||
It affects reporting, never whether a day balances.
|
||||
- `summary-refund-only-returns` posts to **41300 Returns**, the account already used for returns. It
|
||||
moves the *recognition date* of a return onto the day the refund settled, so a refund settling
|
||||
after month end lands in the later period.
|
||||
|
||||
```clojure
|
||||
@(d/transact conn [{:db/id [:client/code "NGxx"]
|
||||
:client/feature-flags ["summary-service-charges"
|
||||
"summary-refund-only-returns"]}])
|
||||
(ss/refresh-sales-summaries 90)
|
||||
```
|
||||
|
||||
Start with two or three restaurants, confirm, then widen.
|
||||
|
||||
**Verify** against the capture from step 5:
|
||||
|
||||
```clojure
|
||||
(def after (cmp/summaries-in (d/db conn) start end))
|
||||
(cmp/compare-window ...) ; both arguments live database values, never as-of
|
||||
```
|
||||
|
||||
The two numbers that matter — both were zero across all 18,900 client-days in testing:
|
||||
|
||||
- `:balanced->unbalanced` must be **0**
|
||||
- previously-balanced days whose lines changed must be **0**
|
||||
|
||||
If either is non-zero, retract the flags for the affected clients and re-run step 5. The flags are
|
||||
the rollback: removing them restores today's behaviour exactly.
|
||||
|
||||
---
|
||||
|
||||
## Step 7 — Re-enable `remove-voided-orders`
|
||||
|
||||
Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2.
|
||||
|
||||
---
|
||||
|
||||
## Step 8 — Remove the legacy key lookup
|
||||
|
||||
Only once `plan` reports `:to-migrate 0` and has stayed there through several import cycles. Drop
|
||||
the second branch of `square.core3/existing-id`. At that point two clients sharing a location
|
||||
becomes structurally incapable of producing a shared record, rather than prevented by a convention a
|
||||
future import could break.
|
||||
|
||||
This is the last step and there is no hurry.
|
||||
|
||||
---
|
||||
|
||||
## What this will not fix
|
||||
|
||||
123 client-days over ninety days, $2,970.35, of which only 33 are above ten cents.
|
||||
|
||||
| | Days | Variance | |
|
||||
|---|---:|---:|---|
|
||||
| Real trading days with genuine discrepancies | 108 | $1,995.36 | the NGBR/NGBK tender gap, the ezCater fee question, unexplained clusters on NGMV and NGEB |
|
||||
| Processing fee on a day with no trading | 15 | $974.99 | same shape as the refund-only day, but the offsetting entry belongs to the payout, not to sales |
|
||||
|
||||
Refunds landing on days that *did* trade are deliberately left alone. Apportioning a return across a
|
||||
day that also sold is a judgement call, and guessing at it is how correct books get quietly
|
||||
rewritten.
|
||||
|
||||
---
|
||||
|
||||
## Two operational findings, unrelated to the summaries
|
||||
|
||||
- **The production backup had not written a restore point since 2025-03-10** — about seventeen
|
||||
months — although data files were still uploading daily. Worth an alert on restore-point age.
|
||||
- **The database server is sized for a much smaller dataset**: a 2 GB cache against 27 GB of data.
|
||||
Worth checking what production is set to.
|
||||
@@ -378,17 +378,75 @@
|
||||
:ledger-mapped/amount (- (+ total discount) tax)
|
||||
#_#_:ledger-mapped/account nil})))
|
||||
|
||||
(defn- get-returns [c date]
|
||||
(when-let [amount (ffirst (dc/q '[:find (sum ?r)
|
||||
:with ?e
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
|
||||
[?e :sales-order/returns ?r]
|
||||
(def refund-only-returns-flag
|
||||
"Per-client rollout lever for recognising a return on a day that carries refunds but no sales,
|
||||
in the same style as `service-charges-flag`. Absent, the summary behaves exactly as it does
|
||||
today."
|
||||
"summary-refund-only-returns")
|
||||
|
||||
#_[?e :sales-order/charges ?c]
|
||||
#_[?c :charge/tax ?tax]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))]
|
||||
(defn- refund-only-returns-enabled? [c]
|
||||
(contains? (set (:client/feature-flags (dc/pull (dc/db conn) '[:client/feature-flags] c)))
|
||||
refund-only-returns-flag))
|
||||
|
||||
(defn- traded?
|
||||
"Whether the client recorded any sales order at all on `date`."
|
||||
[c date]
|
||||
(some? (ffirst (dc/q '[:find (count ?e)
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where
|
||||
[(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))))
|
||||
|
||||
(defn refunded-total
|
||||
"Face value of every refund recorded for the client on `date`.
|
||||
|
||||
This is the same population the `Card Refunds` / `Cash Refunds` / `Food App Refunds` credits are
|
||||
built from, read back as one figure so a return can be posted against it."
|
||||
[c date]
|
||||
(or (ffirst (dc/q '[:find (sum ?t)
|
||||
:with ?e
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where
|
||||
[(iol-ion.query/scan-sales-refunds $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
|
||||
[?e :sales-refund/total ?t]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))
|
||||
0.0))
|
||||
|
||||
(defn- refund-only-returns
|
||||
"The return owed by a day that recorded refunds and no sales at all.
|
||||
|
||||
Refunds are credited on the day the money goes back; the return that offsets them is read from
|
||||
`:sales-order/returns` on orders scanned for that same day. When the two fall on the same day
|
||||
they cancel. When they do not — a refund settling after the restaurant closed for the day, or a
|
||||
retired duplicate client record that still receives refunds while its orders belong to the
|
||||
surviving record — the credit has nothing to offset it and the day cannot balance, by exactly
|
||||
the refunded amount.
|
||||
|
||||
Only days with *no* sales orders are touched. That guard is what makes this safe rather than a
|
||||
judgement call: with no orders there is no order-derived return to double-count, and no trading
|
||||
day can have its figures moved. Days that do trade and still carry an unmatched refund are left
|
||||
alone deliberately — apportioning a return across a day that also sold is an accounting
|
||||
question, not a mechanical one."
|
||||
[c date]
|
||||
(when (and (refund-only-returns-enabled? c)
|
||||
(not (traded? c date)))
|
||||
(let [amount (refunded-total c date)]
|
||||
(when-not (zero? amount) amount))))
|
||||
|
||||
(defn- get-returns [c date]
|
||||
(when-let [amount (or (ffirst (dc/q '[:find (sum ?r)
|
||||
:with ?e
|
||||
:in $ [?clients ?start-date ?end-date]
|
||||
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
|
||||
[?e :sales-order/returns ?r]
|
||||
|
||||
#_[?e :sales-order/charges ?c]
|
||||
#_[?c :charge/tax ?tax]]
|
||||
(dc/db conn)
|
||||
[[c] date date]))
|
||||
(refund-only-returns c date))]
|
||||
{:db/id (str (java.util.UUID/randomUUID))
|
||||
:sales-summary-item/category "Returns"
|
||||
|
||||
@@ -509,7 +567,6 @@
|
||||
@(dc/transact conn [[:upsert-sales-summary result]]))
|
||||
@(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
|
||||
|
||||
|
||||
(defn sales-summaries-v2
|
||||
"Recomputes every dirty summary, client by client."
|
||||
[]
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
(ns auto-ap.jobs.sales-summaries-test
|
||||
(:require
|
||||
[auto-ap.datomic :refer [conn]]
|
||||
[auto-ap.datomic.sales-summaries :as d-ss]
|
||||
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
|
||||
[auto-ap.jobs.sales-summaries :as sut]
|
||||
[clojure.test :refer [deftest is testing use-fixtures]]
|
||||
@@ -121,6 +122,63 @@
|
||||
:sales-order/service-charge -75.0})])
|
||||
(is (nil? (service-charges-for test-client-id))))))
|
||||
|
||||
(defn- refund
|
||||
"A card refund on `sales-date`. The client+date tuple is set explicitly because
|
||||
`scan-sales-refunds` walks that index rather than the plain attributes."
|
||||
[client id total]
|
||||
{:db/id (str "refund-" id)
|
||||
:sales-refund/external-id (str "square/refund/TEST-" id)
|
||||
:sales-refund/client client
|
||||
:sales-refund/date sales-date
|
||||
:sales-refund/client+date [client sales-date]
|
||||
:sales-refund/type "CARD"
|
||||
:sales-refund/total total})
|
||||
|
||||
(defn- enable-refund-only-returns! [client]
|
||||
@(dc/transact conn [{:db/id client
|
||||
:client/feature-flags [sut/refund-only-returns-flag]}]))
|
||||
|
||||
(defn- returns-for [client]
|
||||
(#'sut/get-returns client sales-date))
|
||||
|
||||
(deftest refund-only-day-needs-the-feature-flag
|
||||
(testing "without the flag a day of refunds and no sales books no return, as it does today"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(refund test-client-id "unflagged" 40.0)])
|
||||
(is (nil? (returns-for test-client-id))))))
|
||||
|
||||
(deftest refund-only-day-books-a-return_and_balances
|
||||
(testing "a refund credited on a day with no sales has nothing to offset it, so the day is out
|
||||
by the refunded amount until a return is recognised against it"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-refund-only-returns! test-client-id)
|
||||
@(dc/transact conn [(refund test-client-id "orphan-a" 30.0)
|
||||
(refund test-client-id "orphan-b" 10.0)])
|
||||
(let [returns (returns-for test-client-id)]
|
||||
(is (= 40.0 (:ledger-mapped/amount returns)))
|
||||
(is (= :ledger-side/debit (:ledger-mapped/ledger-side returns)))
|
||||
(is (= 0.0 (d-ss/imbalance (cons returns (sut/get-refund-items test-client-id sales-date))))
|
||||
"the refund credits and the return debit cancel exactly")))))
|
||||
|
||||
(deftest a-day-that-traded-keeps-its-order-derived-return
|
||||
(testing "the guard is what makes this safe: a day with sales is left entirely alone, so no
|
||||
trading day can have its return moved by an unmatched refund"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-refund-only-returns! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "traded" {:sales-order/returns 7.0})
|
||||
(refund test-client-id "same-day" 40.0)])
|
||||
(is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id)))
|
||||
"the order's own return, not the refunded total"))))
|
||||
|
||||
(deftest a-day-that-traded-and-returned-nothing-books-no-return
|
||||
(testing "sales with no returns must not pick up the refunded total either — the guard is on
|
||||
whether the client traded, not on whether the order-derived figure happened to be nil"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-refund-only-returns! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "traded-no-returns" {})
|
||||
(refund test-client-id "unmatched" 40.0)])
|
||||
(is (nil? (returns-for test-client-id))))))
|
||||
|
||||
(deftest dirty-summaries-stop-at-the-client-boundary
|
||||
(testing "every dirty day for the client is returned, and none belonging to another client.
|
||||
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
(ns auto-ap.jobs.compare-sales-summaries
|
||||
(ns auto-ap.tools.compare-sales-summaries
|
||||
"Compares sales summaries between two points in the same database.
|
||||
|
||||
A verification tool, not part of the running application: it lives on the test/dev classpath so
|
||||
nothing in production can depend on it. Load it from a REPL when auditing a recompute.
|
||||
|
||||
The question this exists to answer is narrower than \"did the totals improve\": it is *which
|
||||
days changed, and were any of them already balanced*. A day that was balanced before and still
|
||||
balances after can still have had its line amounts move, and that is a real change to the
|
||||
Reference in New Issue
Block a user