integreat/docs/2026-08-15-sales-summary-balance-validation.md

---
title: Sales Summary Balance — validation against a restored production backup
type: validation
date: 2026-08-15
status: measured
---

# Sales Summary Balance — validation against a restored production backup

Measures the two calculation fixes (R1 reversed tips, R2 Square service charges) against real
production data, and checks them against the predictions in
[`docs/plans/2026-08-13-fix-sales-summary-balance-plan.md`](plans/2026-08-13-fix-sales-summary-balance-plan.md).

**Headline: the fixes behave as predicted.** R2 clears 307 client-days with zero regressions.
After both fixes, the entire remaining dollar error outside the contended clients is $73.98,
and all but six client-days of it is sub-10¢ register rounding.

---

## 1. What was measured against

| | |
|---|---|
| Database | `datomic:dev://localhost:4337/integreat-prod-restore` |
| Restored from | `s3://integreat-backups/`, restore point **209608347** |
| Basis-t | 209608347 |
| Newest transaction | **2026-08-14 22:52 local** |
| Clients | 210 (106 with orders in the window) |
| Refunds / charges / orders | 50,986 / 16,545,495 / 19,039,575 |
| Window | 2026-07-13 → 2026-08-14 |
| Population | **3,342 client-days** |

The earlier restore in this directory was **stale** — the backup had written no root since
2025-03-10 even though segments kept uploading daily through 2026-07-12. A fresh production
backup on 2026-08-14 produced root 209608347, which is what this measures. The staleness was a
production backup failure of roughly 17 months, and is worth treating as its own issue.

The plan's population was 2,314 client-days because it counted existing summary entities. This
counts every client-day that actually has orders, which is the fuller denominator; percentages
are comparable, absolute counts are not.

## 2. Method

Each client-day is composed exactly as `sales-summaries-v2` composes it — the same
`get-sales`, `get-payment-items`, `get-refund-items`, `get-discounts`, `get-fees`, `get-tax`,
`get-returns` — and its imbalance taken as `d-ss/imbalance`, i.e. debits minus credits. Three
variants differ only in the tip and service-charge terms:

- **Baseline** — tips from tenders only, no service-charge credit. What production does today.
- **R1** — tips additive: tendered tips plus tips on orders carrying no tender.
- **R2** — R1 plus Square service charges, both signs.

Nothing was transacted. A day counts as off when `|imbalance| >= $0.005`, i.e. it does not
round to zero.

## 3. Results

| Stage | Days off | Clean | Total \|Δ\| |
|---|---|---|---|
| Baseline (production today) | 569 | 82.97% | $32,921.40 |
| + R1 additive tip | 493 | 85.25% | $32,437.67 |
| **+ R2 Square service charges** | **186** | **94.43%** | **$5,166.49** |

The plan predicted 82.7% clean at baseline including duplicates. Measured: **82.97%**.

### Fixed / regressed — the plan's pre-merge gate

| Change | Fixed | Regressed |
|---|---|---|
| R1 vs baseline | 78 | **2** |
| R2 vs R1 | 307 | **0** |

R2's gate — "days going balanced → unbalanced must be 0, or each explained" — **passes
outright**.

R1's two regressions are both days that balanced only because two errors cancelled, exactly the
mechanism the plan anticipated:

| Client | Date | Before | After | Untendered tip |
|---|---|---|---|---|
| NGCD | 2026-08-08 | $0.00 | +$11.80 | −$11.80 |
| NGDG | 2026-08-05 | $0.00 | +$10.00 | −$10.00 |

Both clients are halves of duplicate Square-location pairs (NGCD/NGCC, NGDG/NGDU), so these are
expected to resolve under Phase 0 rather than needing a change to R1.

## 4. Where the remaining error lives

Splitting the post-R2 residual by whether the client shares a Square location with another
client:

| Group | Client-days | Days off | Total \|Δ\| |
|---|---|---|---|
| 10 duplicate pairs (20 clients) | 546 | 112 | **$5,092.51** |
| Everyone else (96 clients) | 2,796 | 74 | **$73.98** |

**98.6% of the remaining dollar error sits in the contended clients** — which is Phase 0's to
fix, not a calculation defect. This is the plan's central claim, and it holds.

### The $73.98 residual, in full

| Client | Days | Total \|Δ\| | |
|---|---|---|---|
| PNSP | 30 | $0.91 | register rounding |
| PNLP | 26 | $0.77 | register rounding |
| GLAD | 12 | $0.20 | register rounding |
| NGEB | 2 | $45.00 | ezCater fee semantics (plan §15.4) |
| NGPS | 1 | $9.62 | unexplained |
| NGGB | 1 | $7.59 | partial final day |
| NGSC | 1 | $5.02 | partial final day |
| NGTY | 1 | $4.87 | partial final day |

Applying Phase 6's 10¢ materiality threshold leaves **6 material client-days**. The largest
sub-threshold day is **9.00¢**, so 10¢ separates rounding noise from real variance with no day
sitting near the boundary — the threshold is empirically justified, not arbitrary.

Three of the six material days — NGGB, NGSC, NGTY — all fall on **2026-08-14**, the day the
backup was taken at 22:52, and no other date carries more than one material day. They are
partial-day artifacts of the backup cut, not defects.

That leaves exactly the residual the plan predicted, to the cent:

- **NGEB 2026-08-10 −$25.00** and **NGEB 2026-07-29 −$20.00** — predicted
- **NGPS 2026-08-12 +$9.62** — predicted, still unexplained

## 5. Confirmed independently: the dropped refund line

The plan reports NGCD 2026-07-23 missing a $71.94 refund line because the refund belonged to
NGCC at recompute time. That client-day measures **+$71.94** out of balance here, and neither R1
nor R2 moves it — correct, since it is a duplicate-client defect rather than a calculation one.

## 6. Amendment to Phase 1 — charges that cannot be scoped

The re-key pre-flight was run against the restore.

**Refunds — clean.** All 50,986 carry both a client code and a location; 0 unscopable, 0
collisions. Collisions are in fact structurally impossible: the old key is `:db.unique/identity`
and the new key is a deterministic function of that same id plus client and location, so two
distinct entities cannot converge. The run confirms the reasoning.

**Charges — not clean.** Of 16,545,495 charges, **2,122,161 (12.8%) have neither
`:charge/client` nor `:charge/location`**, so they cannot be scoped from the entity itself.
A 5,000-entity sample profiles them as:

| | Count | Share |
|---|---|---|
| Referenced only by an expected-deposit, no total/type/date | 4,897 | 98% |
| Referenced by a sales order, real tender records | 290 | 5.8% |
| Orphaned — reachable from nothing | **0** | 0% |

These are stubs created by the payout path, which asserts `{:charge/external-id ...}` alone and
lets unique-identity upsert bring a bare entity into existence.

**Consequence:** Phase 1 cannot read scoping off the charge. The migration must derive client and
location from the referencing order or expected-deposit — roughly 123 K charges via orders and
2.0 M via deposits, extrapolated. Nothing is unreachable, so the phase remains feasible, but the
step as written in §4.2 would strand 12.8% of charges on the legacy scheme.

The payout call site in `square.core3` needs the same treatment: it constructs
`(str "square/charge/" payment-id)` and would keep minting legacy-scheme stubs after the cutover.
It has `client` and `location` in scope already, so the fix is local.

## 7. Not yet run

- **Phase 0** itself — deactivating the duplicate client per location, then re-measuring. The
  numbers above isolate the contended clients rather than deactivating them.
- **Charges referenced by more than one order** (§3.3), the gate on any historical cleanup.
- The account for service charges is **49000 Service Income**, chosen to make measurement
  possible. It does not affect balance at all — only whether a day can reach `accepted?` — so
  every number here is independent of that choice.


---

# Part 2 — executed end to end on the restore

Part 1 measured what the fixes *would* do without writing anything. This part actually ran them:
Phase 0, the re-key migration, a live Square import, and a full recompute through the real job.
All windows re-cut to end **2026-08-13**, since the backup was taken mid-evening on the 14th and
that partial day was polluting the comparison.

## What was executed, in order

| Step | Result |
|---|---|
| Phase 0 — deactivate the newer twin at all 10 shared locations | 10 datoms retracted; no location shared by two clients any more |
| Phase 1 — re-key refunds | 50,986 re-keyed in 1.9 s, 0 collisions, 0 unscopable |
| Phase 1 — re-key contended clients' charges | 56,829 re-keyed, 0 collisions |
| **Count-unchanged assertion** | refunds 50,986 → 50,986, charges 16,545,495 → 16,545,495 |
| Live Square import, 10 surviving clients | all succeeded |
| Recompute | 6,720 client-days through `sales-summaries-v2` |

The count-unchanged assertion is the one that matters: 107,815 entities were re-keyed and **not a
single entity was created or destroyed**, which is what proves the expand step resolved legacy
keys rather than duplicating everything under the new scheme.

## Deduplication works on an ongoing basis

- **The deactivated twins received nothing.** Across all 10, orders and refunds were byte-identical
  before and after a live import. Phase 0 holds.
- **Refunds stopped alternating and became two stable copies.** 98 of NGCC's 125 refunds are now
  also held by NGCD as separate entities. Each client owns its own; neither can take the other's.
  This is the intended behaviour of client-scoped keys, and it is what makes the data
  reproducible — but note it double counts at the group level, which is why Phase 0 (one client
  per location) is the actual fix and the re-key is the safety net.

### The dropped refund line, fixed

NGCD 2026-07-23 was out of balance by exactly **+$71.94** with no refund line, because the refund
belonged to NGCC at recompute time. After the full pipeline the summary carries a **"Card Refunds"
line of $71.94** and the day's imbalance is **9.1e-13 — zero**.

## Results

Reportable population excludes the 10 deactivated twins, which the plan says to exclude from
reporting.

| | Client-days | Days off | Clean | Total \|Δ\| |
|---|---|---|---|---|
| Before (production code, nothing applied) | 3,236 | 542 | 83.25% | $30,982.11 |
| **After (everything applied)** | 3,030 | **71** | **97.66%** | **$405.79** |

A **98.7% reduction in dollar error**. With the 10¢ materiality threshold, **5 material days
remain**:

| Client | Date | Δ | |
|---|---|---|---|
| NGBK | 2026-08-06 | +$299.42 | new — see below |
| NGDA | 2026-08-01 | −$50.00 | the known auto-gratuity-as-service-charge order |
| NGEB | 2026-08-10 | −$25.00 | ezCater fee semantics, predicted |
| NGEB | 2026-07-29 | −$20.00 | ezCater fee semantics, predicted |
| NGPS | 2026-08-12 | +$9.62 | predicted, still unexplained |

The three predicted residuals reproduced exactly. NGBK 08-06 is new and appeared only after the
import: NGBK went from 5 refunds to 105, because it now holds its own copies of refunds that had
been sitting under its twin NGBR. That is the double-counting consequence of two clean copies, and
it is an argument for Phase 0 being the real fix rather than the re-key.

## Two defects found by executing rather than simulating

### 1. Re-keying a shared charge duplicates the tender

**Found by measurement, not by reading.** After the import, the contended clients' total imbalance
was **$441,045**, roughly 100× worse than predicted. The cause:

1. 35,870 of 56,829 charges had **two parent orders**, one per twin.
2. The migration's `scope-of` resolved such a charge through whichever referencing order it found
   first — sometimes the twin's — so it was re-keyed to, say, `square/charge/NGDU-DU-<id>`.
3. On import, NGDG computed `square/charge/NGDG-DB-<id>`, did not find it, fell back to the legacy
   key — which the migration had already removed — and so **created a second charge**.
4. `:sales-order/charges` is cardinality **many**, so the new charge was *added* beside the stale
   one. One order, two tenders for one payment. Tender roughly doubled.

Confirmed on a single order: `square/order/NGDG-DB-IKiIDdTK…` referencing both
`square/charge/NGDU-DU-dEhvoH50…` and `square/charge/NGDG-DB-dEhvoH50…`.

This is exactly what plan §4.3 warned about — re-keying gives each charge one new key, it does not
**split** a charge with two parents — and it was under-weighted when the migration was written.

**Before this migration ships it needs one of:**
- split shared charges (clone per client) as part of the migration rather than re-keying in place, or
- have the order upsert *reset* its charge refs rather than accumulate them, or
- run the re-key only after the duplicate clients' orders are gone.

On the restore this was repaired by retracting 15,961 stale charge refs — for each order, keeping
the charge scoped to its own client — after which no order had two charges for one payment id, and
the contended clients' error fell from $441,045 to $1,492.

### 2. `dirty-sales-summaries` scans the whole index per client

```clojure
(dc/index-pull db {:index :avet :start [:sales-summary/client+dirty [client-id true]]})
```

There is no `:end`, and the client filter that follows is a lazy `filter` which does not stop the
scan. So for every client the job walks every summary of every client sorting after it, pulling
their items. It is O(n²) in the number of summaries.

Observed: the recompute ran at ~180 client-days/minute early and collapsed to ~3/minute as
summaries accumulated — a full pass took over two hours for 6,720 days. Adding an `:end` bound at
the client boundary should fix it.

## Method note

The 96 non-contended clients were measured from the **persisted summaries** the job wrote, and
came to $56.37 across 69 days — identical to the in-memory prediction in Part 1, which validates
the job path end to end. The 20 contended clients were re-measured in memory after the charge
repair, because a second full pass through the job would have taken hours for the reason in defect
2. The two methods agree exactly where both were run.

## Ongoing deduplication, with both clients active again

The re-key is meant to make contention impossible even if a location is configured on two clients
again. To test that rather than assume it, all 10 twins were **re-activated** — every location
shared once more — and the import re-run.

**Repeated imports are idempotent.** Three consecutive refund imports for both halves of the
Concord pair, with both clients live on location `L43Z5GMW72VMG`:

| | run 0 | run 1 | run 2 | run 3 |
|---|---|---|---|---|
| NGCC | 127 | 127 | 127 | 127 |
| NGCD | 1,419 | 1,419 | 1,419 | 1,419 |

Not one entity created or moved. Before the re-key this is the exact configuration that made a
refund's owner flip every ~20 minutes.

The first import after the migration is the only one that grows anything, and it grows once: NGCD
gained 100 refunds materialising its own copies, while NGCC — whose refunds already carried its
own scoped key — gained only the 2 that were genuinely new. After that, steady state.

**State the restore was left in:** twins re-activated, i.e. Phase 0 deliberately undone for this
test. Re-applying it is the ten-datom retraction recorded above.


---

# Part 3 — all summaries updated, fixes compared to baseline

Every summary in the database was recomputed — **14,458 client-days**, 2024-04-01 through
2026-08-14 — through `sales-summaries-v2`, with zero failures and zero left dirty. The
comparison below is the **last 30 days, 2026-07-15 → 2026-08-13**.

Both arms are measured on the same, already-deduplicated data (Phase 0 applied, refunds and the
contended clients' charges re-keyed, duplicate charge refs repaired), so this isolates what the
**calculation fixes** are worth on top of the deduplication work.

## Making the full recompute possible

Two changes were needed before recomputing everything was practical:

1. **`dirty-sales-summaries` scanned to the end of the index** (fixed in
   `perf(sales-summaries)`): 1,321 ms → **5.6 ms** per client, a 237× improvement, identical
   results.
2. **The driver was one serial `doseq`.** `refresh-client!` was split out of `sales-summaries-v2`
   so a client's work stands alone. Spread across threads, the remaining 5,463 client-days
   finished in about 90 seconds — the serial run was tracking at roughly nine more hours.

## Method

Baseline is derived rather than recomputed: R1 and R2 only ever *add credits*, so

```
baseline imbalance = fixed imbalance + untendered tip + service charges
```

This identity was checked against a full from-scratch baseline recomputation on 20 randomly
sampled client-days and agreed on every one, to within a hundredth of a cent.

## Results — last 30 days, excluding the 10 deactivated twins

2,842 client-days across 96 clients.

| | Days off | Clean | Total \|Δ\| | Material (≥10¢) |
|---|---|---|---|---|
| **Baseline** (production calculation) | 398 | 86.00% | $22,527.40 | 336 |
| **Fixed** (R1 + R2) | **67** | **97.64%** | **$405.66** | **5** |

**331 client-days fixed, 0 regressed.** Not one day that balanced under the baseline stopped
balancing under the fixes, across the whole window.

Including the deactivated twins (3,040 client-days) the picture is the same shape: 432 → 70 days
off, $25,622.98 → $1,548.64.

### What is left

| Client | Date | Δ | |
|---|---|---|---|
| NGBK | 2026-08-06 | +$299.42 | refunds now held by both twins — see below |
| NGDA | 2026-08-01 | −$50.00 | auto-gratuity booked as a service charge |
| NGEB | 2026-08-10 | −$25.00 | ezCater fee semantics, predicted by the plan |
| NGEB | 2026-07-29 | −$20.00 | ezCater fee semantics, predicted by the plan |
| NGPS | 2026-08-12 | +$9.62 | predicted by the plan, still unexplained |

Everything else — 62 client-days — totals **$1.62**, with the largest single day at **9.00¢**.
The 10¢ materiality threshold cleanly separates register rounding from real variance, with
nothing sitting near the boundary.

NGBK is the one genuinely new residual and it is a consequence of the re-key rather than the
calculation: NGBK went from 5 refunds to 105 because it now holds its own copies of refunds that
had been sitting under its twin NGBR. Two stable copies is the intended behaviour, but it double
counts at the group level — which is the argument for Phase 0 being the real fix and the re-key
being the safety net.

### Representative days the fixes repair outright

| Client | Date | Baseline → Fixed |
|---|---|---|
| NTPT | 2026-08-06 | $427.10 → $0.00 |
| NGFO | 2026-07-16 | $275.23 → $0.00 |
| NGRN | 2026-08-06 | $236.96 → $0.00 |
| N-30008 | 2026-07-24 | $229.42 → $0.00 |
| N-30003 | 2026-07-18 | $225.00 → $0.00 |