docs(sales-summaries): record the end-to-end run on the restore
Executes what Part 1 only measured: Phase 0, the re-key migration, a live Square import, and a full recompute through sales-summaries-v2, with every window re-cut to end 08-13 so the backup's partial final day stops skewing the numbers. Reportable error falls from 542 days off / $30,982.11 to 71 days off / $405.79, a 98.7% reduction, with 5 material days left — three of them the ones the plan predicted. 107,815 entities were re-keyed with refund and charge counts unchanged, which is what proves the expand step resolves legacy keys instead of duplicating under the new scheme. NGCD 2026-07-23, off by exactly $71.94 with no refund line, now carries the line and balances to 9.1e-13. Records two defects that only executing could surface: Re-keying a charge with two parent orders duplicates the tender. The migration scoped such charges through whichever order it saw first, so the other client's import found neither the scoped nor the legacy key and created a second charge, which cardinality-many then added beside the first. Contended clients' error was $441,045 until 15,961 stale refs were retracted. §4.3 warned about this and the migration under-weighted it; it must split shared charges before shipping. dirty-sales-summaries index-pulls from the client's start point with no :end and filters lazily, so it scans every later client's summaries too — O(n²), and the recompute degraded from ~180 to ~3 client-days per minute as summaries piled up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -168,3 +168,122 @@ It has `client` and `location` in scope already, so the fix is local.
|
||||
- The account for service charges is **49000 Service Income**, chosen to make measurement
|
||||
possible. It does not affect balance at all — only whether a day can reach `accepted?` — so
|
||||
every number here is independent of that choice.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Part 2 — executed end to end on the restore
|
||||
|
||||
Part 1 measured what the fixes *would* do without writing anything. This part actually ran them:
|
||||
Phase 0, the re-key migration, a live Square import, and a full recompute through the real job.
|
||||
All windows re-cut to end **2026-08-13**, since the backup was taken mid-evening on the 14th and
|
||||
that partial day was polluting the comparison.
|
||||
|
||||
## What was executed, in order
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| Phase 0 — deactivate the newer twin at all 10 shared locations | 10 datoms retracted; no location shared by two clients any more |
|
||||
| Phase 1 — re-key refunds | 50,986 re-keyed in 1.9 s, 0 collisions, 0 unscopable |
|
||||
| Phase 1 — re-key contended clients' charges | 56,829 re-keyed, 0 collisions |
|
||||
| **Count-unchanged assertion** | refunds 50,986 → 50,986, charges 16,545,495 → 16,545,495 |
|
||||
| Live Square import, 10 surviving clients | all succeeded |
|
||||
| Recompute | 6,720 client-days through `sales-summaries-v2` |
|
||||
|
||||
The count-unchanged assertion is the one that matters: 107,815 entities were re-keyed and **not a
|
||||
single entity was created or destroyed**, which is what proves the expand step resolved legacy
|
||||
keys rather than duplicating everything under the new scheme.
|
||||
|
||||
## Deduplication works on an ongoing basis
|
||||
|
||||
- **The deactivated twins received nothing.** Across all 10, orders and refunds were byte-identical
|
||||
before and after a live import. Phase 0 holds.
|
||||
- **Refunds stopped alternating and became two stable copies.** 98 of NGCC's 125 refunds are now
|
||||
also held by NGCD as separate entities. Each client owns its own; neither can take the other's.
|
||||
This is the intended behaviour of client-scoped keys, and it is what makes the data
|
||||
reproducible — but note it double counts at the group level, which is why Phase 0 (one client
|
||||
per location) is the actual fix and the re-key is the safety net.
|
||||
|
||||
### The dropped refund line, fixed
|
||||
|
||||
NGCD 2026-07-23 was out of balance by exactly **+$71.94** with no refund line, because the refund
|
||||
belonged to NGCC at recompute time. After the full pipeline the summary carries a **"Card Refunds"
|
||||
line of $71.94** and the day's imbalance is **9.1e-13 — zero**.
|
||||
|
||||
## Results
|
||||
|
||||
Reportable population excludes the 10 deactivated twins, which the plan says to exclude from
|
||||
reporting.
|
||||
|
||||
| | Client-days | Days off | Clean | Total \|Δ\| |
|
||||
|---|---|---|---|---|
|
||||
| Before (production code, nothing applied) | 3,236 | 542 | 83.25% | $30,982.11 |
|
||||
| **After (everything applied)** | 3,030 | **71** | **97.66%** | **$405.79** |
|
||||
|
||||
A **98.7% reduction in dollar error**. With the 10¢ materiality threshold, **5 material days
|
||||
remain**:
|
||||
|
||||
| Client | Date | Δ | |
|
||||
|---|---|---|---|
|
||||
| NGBK | 2026-08-06 | +$299.42 | new — see below |
|
||||
| NGDA | 2026-08-01 | −$50.00 | the known auto-gratuity-as-service-charge order |
|
||||
| NGEB | 2026-08-10 | −$25.00 | ezCater fee semantics, predicted |
|
||||
| NGEB | 2026-07-29 | −$20.00 | ezCater fee semantics, predicted |
|
||||
| NGPS | 2026-08-12 | +$9.62 | predicted, still unexplained |
|
||||
|
||||
The three predicted residuals reproduced exactly. NGBK 08-06 is new and appeared only after the
|
||||
import: NGBK went from 5 refunds to 105, because it now holds its own copies of refunds that had
|
||||
been sitting under its twin NGBR. That is the double-counting consequence of two clean copies, and
|
||||
it is an argument for Phase 0 being the real fix rather than the re-key.
|
||||
|
||||
## Two defects found by executing rather than simulating
|
||||
|
||||
### 1. Re-keying a shared charge duplicates the tender
|
||||
|
||||
**Found by measurement, not by reading.** After the import, the contended clients' total imbalance
|
||||
was **$441,045**, roughly 100× worse than predicted. The cause:
|
||||
|
||||
1. 35,870 of 56,829 charges had **two parent orders**, one per twin.
|
||||
2. The migration's `scope-of` resolved such a charge through whichever referencing order it found
|
||||
first — sometimes the twin's — so it was re-keyed to, say, `square/charge/NGDU-DU-<id>`.
|
||||
3. On import, NGDG computed `square/charge/NGDG-DB-<id>`, did not find it, fell back to the legacy
|
||||
key — which the migration had already removed — and so **created a second charge**.
|
||||
4. `:sales-order/charges` is cardinality **many**, so the new charge was *added* beside the stale
|
||||
one. One order, two tenders for one payment. Tender roughly doubled.
|
||||
|
||||
Confirmed on a single order: `square/order/NGDG-DB-IKiIDdTK…` referencing both
|
||||
`square/charge/NGDU-DU-dEhvoH50…` and `square/charge/NGDG-DB-dEhvoH50…`.
|
||||
|
||||
This is exactly what plan §4.3 warned about — re-keying gives each charge one new key, it does not
|
||||
**split** a charge with two parents — and it was under-weighted when the migration was written.
|
||||
|
||||
**Before this migration ships it needs one of:**
|
||||
- split shared charges (clone per client) as part of the migration rather than re-keying in place, or
|
||||
- have the order upsert *reset* its charge refs rather than accumulate them, or
|
||||
- run the re-key only after the duplicate clients' orders are gone.
|
||||
|
||||
On the restore this was repaired by retracting 15,961 stale charge refs — for each order, keeping
|
||||
the charge scoped to its own client — after which no order had two charges for one payment id, and
|
||||
the contended clients' error fell from $441,045 to $1,492.
|
||||
|
||||
### 2. `dirty-sales-summaries` scans the whole index per client
|
||||
|
||||
```clojure
|
||||
(dc/index-pull db {:index :avet :start [:sales-summary/client+dirty [client-id true]]})
|
||||
```
|
||||
|
||||
There is no `:end`, and the client filter that follows is a lazy `filter` which does not stop the
|
||||
scan. So for every client the job walks every summary of every client sorting after it, pulling
|
||||
their items. It is O(n²) in the number of summaries.
|
||||
|
||||
Observed: the recompute ran at ~180 client-days/minute early and collapsed to ~3/minute as
|
||||
summaries accumulated — a full pass took over two hours for 6,720 days. Adding an `:end` bound at
|
||||
the client boundary should fix it.
|
||||
|
||||
## Method note
|
||||
|
||||
The 96 non-contended clients were measured from the **persisted summaries** the job wrote, and
|
||||
came to $56.37 across 69 days — identical to the in-memory prediction in Part 1, which validates
|
||||
the job path end to end. The 20 contended clients were re-measured in memory after the charge
|
||||
repair, because a second full pass through the job would have taken hours for the reason in defect
|
||||
2. The two methods agree exactly where both were run.
|
||||
|
||||
Reference in New Issue
Block a user