integreat/docs/2026-08-15-sales-summary-balance-validation.md

title, type, date, status

title	type	date	status
Sales Summary Balance — validation against a restored production backup	validation	2026-08-15	measured

Sales Summary Balance — validation against a restored production backup

Measures the two calculation fixes (R1 reversed tips, R2 Square service charges) against real production data, and checks them against the predictions in docs/plans/2026-08-13-fix-sales-summary-balance-plan.md.

Headline: the fixes behave as predicted. R2 clears 307 client-days with zero regressions. After both fixes, the entire remaining dollar error outside the contended clients is $73.98, and all but six client-days of it is sub-10¢ register rounding.

---

1. What was measured against

Database	datomic:dev://localhost:4337/integreat-prod-restore
Restored from	s3://integreat-backups/, restore point 209608347
Basis-t	209608347
Newest transaction	2026-08-14 22:52 local
Clients	210 (106 with orders in the window)
Refunds / charges / orders	50,986 / 16,545,495 / 19,039,575
Window	2026-07-13 → 2026-08-14
Population	3,342 client-days

The earlier restore in this directory was stale — the backup had written no root since 2025-03-10 even though segments kept uploading daily through 2026-07-12. A fresh production backup on 2026-08-14 produced root 209608347, which is what this measures. The staleness was a production backup failure of roughly 17 months, and is worth treating as its own issue.

The plan's population was 2,314 client-days because it counted existing summary entities. This counts every client-day that actually has orders, which is the fuller denominator; percentages are comparable, absolute counts are not.

2. Method

Each client-day is composed exactly as sales-summaries-v2 composes it — the same get-sales, get-payment-items, get-refund-items, get-discounts, get-fees, get-tax, get-returns — and its imbalance taken as d-ss/imbalance, i.e. debits minus credits. Three variants differ only in the tip and service-charge terms:

• Baseline — tips from tenders only, no service-charge credit. What production does today.
• R1 — tips additive: tendered tips plus tips on orders carrying no tender.
• R2 — R1 plus Square service charges, both signs.

Nothing was transacted. A day counts as off when |imbalance| >= $0.005, i.e. it does not round to zero.

3. Results

Stage	Days off	Clean	Total |Δ|
Baseline (production today)	569	82.97%	$32,921.40
+ R1 additive tip	493	85.25%	$32,437.67
+ R2 Square service charges	186	94.43%	$5,166.49

The plan predicted 82.7% clean at baseline including duplicates. Measured: 82.97%.

Fixed / regressed — the plan's pre-merge gate

Change	Fixed	Regressed
R1 vs baseline	78	2
R2 vs R1	307	0

R2's gate — "days going balanced → unbalanced must be 0, or each explained" — passes outright.

R1's two regressions are both days that balanced only because two errors cancelled, exactly the mechanism the plan anticipated:

Client	Date	Before	After	Untendered tip
NGCD	2026-08-08	$0.00	+$11.80	−$11.80
NGDG	2026-08-05	$0.00	+$10.00	−$10.00

Both clients are halves of duplicate Square-location pairs (NGCD/NGCC, NGDG/NGDU), so these are expected to resolve under Phase 0 rather than needing a change to R1.

4. Where the remaining error lives

Splitting the post-R2 residual by whether the client shares a Square location with another client:

Group	Client-days	Days off	Total |Δ|
10 duplicate pairs (20 clients)	546	112	$5,092.51
Everyone else (96 clients)	2,796	74	$73.98

98.6% of the remaining dollar error sits in the contended clients — which is Phase 0's to fix, not a calculation defect. This is the plan's central claim, and it holds.

The $73.98 residual, in full

Client	Days	Total |Δ|
PNSP	30	$0.91	register rounding
PNLP	26	$0.77	register rounding
GLAD	12	$0.20	register rounding
NGEB	2	$45.00	ezCater fee semantics (plan §15.4)
NGPS	1	$9.62	unexplained
NGGB	1	$7.59	partial final day
NGSC	1	$5.02	partial final day
NGTY	1	$4.87	partial final day

Applying Phase 6's 10¢ materiality threshold leaves 6 material client-days. The largest sub-threshold day is 9.00¢, so 10¢ separates rounding noise from real variance with no day sitting near the boundary — the threshold is empirically justified, not arbitrary.

Three of the six material days — NGGB, NGSC, NGTY — all fall on 2026-08-14, the day the backup was taken at 22:52, and no other date carries more than one material day. They are partial-day artifacts of the backup cut, not defects.

That leaves exactly the residual the plan predicted, to the cent:

• NGEB 2026-08-10 −$25.00 and NGEB 2026-07-29 −$20.00 — predicted
• NGPS 2026-08-12 +$9.62 — predicted, still unexplained

5. Confirmed independently: the dropped refund line

The plan reports NGCD 2026-07-23 missing a $71.94 refund line because the refund belonged to NGCC at recompute time. That client-day measures +$71.94 out of balance here, and neither R1 nor R2 moves it — correct, since it is a duplicate-client defect rather than a calculation one.

6. Amendment to Phase 1 — charges that cannot be scoped

The re-key pre-flight was run against the restore.

Refunds — clean. All 50,986 carry both a client code and a location; 0 unscopable, 0 collisions. Collisions are in fact structurally impossible: the old key is :db.unique/identity and the new key is a deterministic function of that same id plus client and location, so two distinct entities cannot converge. The run confirms the reasoning.

Charges — not clean. Of 16,545,495 charges, 2,122,161 (12.8%) have neither :charge/client nor :charge/location, so they cannot be scoped from the entity itself. A 5,000-entity sample profiles them as:

	Count	Share
Referenced only by an expected-deposit, no total/type/date	4,897	98%
Referenced by a sales order, real tender records	290	5.8%
Orphaned — reachable from nothing	0	0%

These are stubs created by the payout path, which asserts {:charge/external-id ...} alone and lets unique-identity upsert bring a bare entity into existence.

Consequence: Phase 1 cannot read scoping off the charge. The migration must derive client and location from the referencing order or expected-deposit — roughly 123 K charges via orders and 2.0 M via deposits, extrapolated. Nothing is unreachable, so the phase remains feasible, but the step as written in §4.2 would strand 12.8% of charges on the legacy scheme.

The payout call site in square.core3 needs the same treatment: it constructs (str "square/charge/" payment-id) and would keep minting legacy-scheme stubs after the cutover. It has client and location in scope already, so the fix is local.

7. Not yet run

• Phase 0 itself — deactivating the duplicate client per location, then re-measuring. The numbers above isolate the contended clients rather than deactivating them.
• Charges referenced by more than one order (§3.3), the gate on any historical cleanup.
• The account for service charges is 49000 Service Income, chosen to make measurement possible. It does not affect balance at all — only whether a day can reach accepted? — so every number here is independent of that choice.

---

Part 2 — executed end to end on the restore

Part 1 measured what the fixes would do without writing anything. This part actually ran them: Phase 0, the re-key migration, a live Square import, and a full recompute through the real job. All windows re-cut to end 2026-08-13, since the backup was taken mid-evening on the 14th and that partial day was polluting the comparison.

What was executed, in order

Step	Result
Phase 0 — deactivate the newer twin at all 10 shared locations	10 datoms retracted; no location shared by two clients any more
Phase 1 — re-key refunds	50,986 re-keyed in 1.9 s, 0 collisions, 0 unscopable
Phase 1 — re-key contended clients' charges	56,829 re-keyed, 0 collisions
Count-unchanged assertion	refunds 50,986 → 50,986, charges 16,545,495 → 16,545,495
Live Square import, 10 surviving clients	all succeeded
Recompute	6,720 client-days through sales-summaries-v2

The count-unchanged assertion is the one that matters: 107,815 entities were re-keyed and not a single entity was created or destroyed, which is what proves the expand step resolved legacy keys rather than duplicating everything under the new scheme.

Deduplication works on an ongoing basis

• The deactivated twins received nothing. Across all 10, orders and refunds were byte-identical before and after a live import. Phase 0 holds.
• Refunds stopped alternating and became two stable copies. 98 of NGCC's 125 refunds are now also held by NGCD as separate entities. Each client owns its own; neither can take the other's. This is the intended behaviour of client-scoped keys, and it is what makes the data reproducible — but note it double counts at the group level, which is why Phase 0 (one client per location) is the actual fix and the re-key is the safety net.

The dropped refund line, fixed

NGCD 2026-07-23 was out of balance by exactly +$71.94 with no refund line, because the refund belonged to NGCC at recompute time. After the full pipeline the summary carries a "Card Refunds" line of $71.94 and the day's imbalance is 9.1e-13 — zero.

Results

Reportable population excludes the 10 deactivated twins, which the plan says to exclude from reporting.

	Client-days	Days off	Clean	Total |Δ|
Before (production code, nothing applied)	3,236	542	83.25%	$30,982.11
After (everything applied)	3,030	71	97.66%	$405.79

A 98.7% reduction in dollar error. With the 10¢ materiality threshold, 5 material days remain:

Client	Date	Δ
NGBK	2026-08-06	+$299.42	new — see below
NGDA	2026-08-01	−$50.00	the known auto-gratuity-as-service-charge order
NGEB	2026-08-10	−$25.00	ezCater fee semantics, predicted
NGEB	2026-07-29	−$20.00	ezCater fee semantics, predicted
NGPS	2026-08-12	+$9.62	predicted, still unexplained

The three predicted residuals reproduced exactly. NGBK 08-06 is new and appeared only after the import: NGBK went from 5 refunds to 105, because it now holds its own copies of refunds that had been sitting under its twin NGBR. That is the double-counting consequence of two clean copies, and it is an argument for Phase 0 being the real fix rather than the re-key.

Two defects found by executing rather than simulating

1. Re-keying a shared charge duplicates the tender

Found by measurement, not by reading. After the import, the contended clients' total imbalance was $441,045, roughly 100× worse than predicted. The cause:

1. 35,870 of 56,829 charges had two parent orders, one per twin.
2. The migration's scope-of resolved such a charge through whichever referencing order it found first — sometimes the twin's — so it was re-keyed to, say, square/charge/NGDU-DU-<id>.
3. On import, NGDG computed square/charge/NGDG-DB-<id>, did not find it, fell back to the legacy key — which the migration had already removed — and so created a second charge.
4. :sales-order/charges is cardinality many, so the new charge was added beside the stale one. One order, two tenders for one payment. Tender roughly doubled.

Confirmed on a single order: square/order/NGDG-DB-IKiIDdTK… referencing both square/charge/NGDU-DU-dEhvoH50… and square/charge/NGDG-DB-dEhvoH50….

This is exactly what plan §4.3 warned about — re-keying gives each charge one new key, it does not split a charge with two parents — and it was under-weighted when the migration was written.

Before this migration ships it needs one of:

• split shared charges (clone per client) as part of the migration rather than re-keying in place, or
• have the order upsert reset its charge refs rather than accumulate them, or
• run the re-key only after the duplicate clients' orders are gone.

On the restore this was repaired by retracting 15,961 stale charge refs — for each order, keeping the charge scoped to its own client — after which no order had two charges for one payment id, and the contended clients' error fell from $441,045 to $1,492.

2. dirty-sales-summaries scans the whole index per client

(dc/index-pull db {:index :avet :start [:sales-summary/client+dirty [client-id true]]})

There is no :end, and the client filter that follows is a lazy filter which does not stop the scan. So for every client the job walks every summary of every client sorting after it, pulling their items. It is O(n²) in the number of summaries.

Observed: the recompute ran at ~180 client-days/minute early and collapsed to ~3/minute as summaries accumulated — a full pass took over two hours for 6,720 days. Adding an :end bound at the client boundary should fix it.

Method note

The 96 non-contended clients were measured from the persisted summaries the job wrote, and came to $56.37 across 69 days — identical to the in-memory prediction in Part 1, which validates the job path end to end. The 20 contended clients were re-measured in memory after the charge repair, because a second full pass through the job would have taken hours for the reason in defect 2. The two methods agree exactly where both were run.

Ongoing deduplication, with both clients active again

The re-key is meant to make contention impossible even if a location is configured on two clients again. To test that rather than assume it, all 10 twins were re-activated — every location shared once more — and the import re-run.

Repeated imports are idempotent. Three consecutive refund imports for both halves of the Concord pair, with both clients live on location L43Z5GMW72VMG:

	run 0	run 1	run 2	run 3
NGCC	127	127	127	127
NGCD	1,419	1,419	1,419	1,419

Not one entity created or moved. Before the re-key this is the exact configuration that made a refund's owner flip every ~20 minutes.

The first import after the migration is the only one that grows anything, and it grows once: NGCD gained 100 refunds materialising its own copies, while NGCC — whose refunds already carried its own scoped key — gained only the 2 that were genuinely new. After that, steady state.

State the restore was left in: twins re-activated, i.e. Phase 0 deliberately undone for this test. Re-applying it is the ten-datom retraction recorded above.

---

Part 3 — all summaries updated, fixes compared to baseline

Every summary in the database was recomputed — 14,458 client-days, 2024-04-01 through 2026-08-14 — through sales-summaries-v2, with zero failures and zero left dirty. The comparison below is the last 30 days, 2026-07-15 → 2026-08-13.

Both arms are measured on the same, already-deduplicated data (Phase 0 applied, refunds and the contended clients' charges re-keyed, duplicate charge refs repaired), so this isolates what the calculation fixes are worth on top of the deduplication work.

Making the full recompute possible

Two changes were needed before recomputing everything was practical:

1. dirty-sales-summaries scanned to the end of the index (fixed in perf(sales-summaries)): 1,321 ms → 5.6 ms per client, a 237× improvement, identical results.
2. The driver was one serial doseq. refresh-client! was split out of sales-summaries-v2 so a client's work stands alone. Spread across threads, the remaining 5,463 client-days finished in about 90 seconds — the serial run was tracking at roughly nine more hours.

Method

Baseline is derived rather than recomputed: R1 and R2 only ever add credits, so

baseline imbalance = fixed imbalance + untendered tip + service charges

This identity was checked against a full from-scratch baseline recomputation on 20 randomly sampled client-days and agreed on every one, to within a hundredth of a cent.

Results — last 30 days, excluding the 10 deactivated twins

2,842 client-days across 96 clients.

	Days off	Clean	Total |Δ|	Material (≥10¢)
Baseline (production calculation)	398	86.00%	$22,527.40	336
Fixed (R1 + R2)	67	97.64%	$405.66	5

331 client-days fixed, 0 regressed. Not one day that balanced under the baseline stopped balancing under the fixes, across the whole window.

Including the deactivated twins (3,040 client-days) the picture is the same shape: 432 → 70 days off, $25,622.98 → $1,548.64.

What is left

Client	Date	Δ
NGBK	2026-08-06	+$299.42	refunds now held by both twins — see below
NGDA	2026-08-01	−$50.00	auto-gratuity booked as a service charge
NGEB	2026-08-10	−$25.00	ezCater fee semantics, predicted by the plan
NGEB	2026-07-29	−$20.00	ezCater fee semantics, predicted by the plan
NGPS	2026-08-12	+$9.62	predicted by the plan, still unexplained

Everything else — 62 client-days — totals $1.62, with the largest single day at 9.00¢. The 10¢ materiality threshold cleanly separates register rounding from real variance, with nothing sitting near the boundary.

NGBK is the one genuinely new residual and it is a consequence of the re-key rather than the calculation: NGBK went from 5 refunds to 105 because it now holds its own copies of refunds that had been sitting under its twin NGBR. Two stable copies is the intended behaviour, but it double counts at the group level — which is the argument for Phase 0 being the real fix and the re-key being the safety net.

Representative days the fixes repair outright

Client	Date	Baseline → Fixed
NTPT	2026-08-06	$427.10 → $0.00
NGFO	2026-07-16	$275.23 → $0.00
NGRN	2026-08-06	$236.96 → $0.00
N-30008	2026-07-24	$229.42 → $0.00
N-30003	2026-07-18	$225.00 → $0.00