A true baseline: the job re-run three times over ninety days — production's calculation, then each fix added — so every number below is measured output, not arithmetic.
+Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading.
Figures exclude the ten deactivated duplicate clients, which the plan says to exclude from reporting. Across the full 8,733 client-days including them the shape is the same: 1,087 → 108 days out of balance, $63,764.92 → $1,995.36.
+In one sentence: a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one day in seven it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.
The earlier thirty-day report derived its baseline arithmetically. This one does not. The job was run three separate times over the same ninety days, against the same data, writing real summaries each time:
-get-tip restored to its tendered-only form. This is exactly what production calculates today.get-tip changed.The underlying data — Phase 0, the re-key, the repaired charge references — is identical across all three, so what separates them is the calculation and nothing else. 18,900 summaries were written per run, 56,700 in total.
-The most recent week (08-08 → 08-14) is excluded deliberately. The backup was cut mid-evening on the 14th, and the days either side of that are partial, which distorted the earlier window.
+For the business: ten restaurant locations were set up twice in the system, as two separate clients. Both were importing from Square. Because the two records competed for the same payments and refunds, a refund would belong to one client for twenty minutes, then the other — so a day's books could gain or lose a refund depending on nothing but timing. On 2026-07-23 one client's summary was missing a $71.94 refund entirely, and was out of balance by exactly that amount.
+technical Sales orders scoped their identifier by client (square/order/<code>-<loc>-<id>), but refunds, card charges, payouts and cash-drawer shifts did not — they used the bare Square id. Those attributes are :db.unique/identity, so both clients' imports resolved to a single entity and the last writer won.
Reading ownership out of the database's own history, this had actually happened to 3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts. And it has involved 19 client pairs, of which only 10 are visible in today's configuration — nine more contended in the past and the configuration has since changed, so no point-in-time check would find them.
+For the business: the same collision meant a single card payment could be attached to both clients' copies of an order. That is worse than untidy. The nightly import removes orders Square reports as voided, and removing an order also removes its payments — so cancelling one client's order could silently delete the other client's payment, leaving a day showing sales with no money against them.
+technical :sales-order/charges is declared :db/isComponent true, so [:db/retractEntity <order>] cascades into the charges. In a 20,000-order sample of the affected clients, 11,469 charges had two parent orders. This is why remove-voided-orders was left switched off during testing.
For the business: two arithmetic faults, both of which overstated or understated a day.
+technical get-tip summed tips by joining through :sales-order/charges, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on :sales-order/tip. Nothing at all read :sales-order/service-charge.
Applied in sequence, each measured against the run before it, over 8,350 reportable client-days.
-The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates.
| Outcome | Client-days | Share | |
|---|---|---|---|
| Stage | Days out of balance | Clean | Total variance |
| Unchanged | 8,054 | 96.5% | |
| Out of balance → balanced | 272 | 3.3% | |
| Balanced → out of balance | 0 | — | |
| Balanced → balanced, numbers changed | 0 | — | |
| Still out of balance, but closer | 24 | 0.3% | |
| Days touched · dollars moved | 296 | $3,623.44 | |
| Production today | 1,280 | 85.34% | $75,228.78 |
| + deduplication | 1,087 | 87.55% | $63,764.92 |
| + refunded tips | 800 | 90.84% | $60,619.21 |
| + service charges | 108 | 98.76% | $1,995.36 |
| Outcome | Client-days | Share | |||
|---|---|---|---|---|---|
| Change | Unchanged | Into balance | Out of balance | Balanced days altered | Money moved |
| Unchanged | 7,701 | 92.2% | |||
| Out of balance → balanced | 647 | 7.7% | |||
| Balanced → out of balance | 0 | — | |||
| Balanced → balanced, numbers changed | 0 | — | |||
| Still out of balance, but closer | 2 | <0.1% | |||
| Days touched · dollars moved | 649 | $54,775.79 | |||
| Deduplication | 7,447 balanced already | 199 | 6 | — | — |
| Refunded tips | 8,421 | 287 | 0 | 0 | $3,777.67 |
| Service charges | 8,037 | 692 | 0 | 0 | $58,923.85 |
| All three, end to end | 7,453 | 1,172 | 0 | 0 | — |
| Outcome | Client-days |
|---|---|
| Unchanged | 7,749 |
| Out of balance → balanced | 979 |
| Balanced → out of balance | 0 |
| Balanced → balanced, numbers changed | 0 |
| Still out of balance, but closer | 5 |
| Days touched · dollars moved | 984 · $62,069.64 |
Across 8,733 client-days and 90 days of trading, not one day that balanced under production's calculation was altered by either fix: zero knocked out of balance, and zero whose amounts moved while staying balanced. Every day they touched was already out of balance. The thirty-day run found the same thing; this replicates it on a much larger sample.
+Neither arithmetic fix touched a day that was already correct. Across 8,733 client-days, no balanced day was knocked out of balance and no balanced day had a single figure altered — every day they changed was already wrong. Service charges are by far the larger of the two, moving $58,923.85 against the tip fix's $3,777.67.
+Deduplication is the one step that puts six days out of balance. That is expected: those days were previously balanced only because a summary was empty, and filling it in exposes the same arithmetic faults every other day had. Both later fixes then close them, which is why the end-to-end figure is zero.
+Excluding the ten now-deactivated duplicate records, which should not be reported on at all, the population is 8,350 client-days and the movement is 1,217 days and $69,995.61 down to 105 days and $852.38.
Both fixes add a single credit line. Nothing else in a summary moves — no sales figure, no - tender, no tax. That is why they can only ever help a day that was already short, and it is - visible in the line items.
+Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.
| Line | Before | After | Change |
|---|---|---|---|
| Line | Before | After | |
| Tip (credit) | 482.94 | 422.94 | −60.00 |
| Card Refunds (credit) | 60.00 | 60.00 | — |
| every other line | unchanged | — | |
| Total debits | 10,094.81 | 10,094.81 | — |
| Total credits | 10,154.81 | 10,094.81 | −60.00 |
| Imbalance | −60.00 | 0.00 | balanced |
| Tip | 482.94 | 422.94 | |
| Card Refunds | 60.00 | 60.00 | |
| Total money taken | 10,094.81 | 10,094.81 | |
| Total money earned | 10,154.81 | 10,094.81 | |
| Out of balance by | −60.00 | 0.00 | |
Read the two credit lines together and the story is complete: the day already carried a
- $60.00 card refund — the guest was given their money back, tip included — but the
- tip was still credited in full at $482.94, because the reversal lives on an order with no tender
- and get-tip only reached tips through tenders. The books claimed $60 of tip income
- that had been handed back. The corrected figure, $422.94, matches the refund exactly.
The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY: a tip of −60.00 on an order with no payment attached at all.
| Line | Before | After | Change |
|---|---|---|---|
| Line | Before | After | |
| Service Charges (credit) | absent | 427.10 | +427.10 |
| Card Payments (debit) | 4,975.89 | 4,975.89 | — |
| every other line | unchanged | — | |
| Total debits | 7,777.20 | 7,777.20 | — |
| Total credits | 7,350.10 | 7,777.20 | +427.10 |
| Imbalance | +427.10 | 0.00 | balanced |
| Service Charges | not shown | 427.10 | |
| Card Payments | 4,975.89 | 4,975.89 | |
| Total money taken | 7,777.20 | 7,777.20 | |
| Total money earned | 7,350.10 | 7,777.20 | |
| Out of balance by | +427.10 | 0.00 | |
| Client | Date | Line | Before | After | Day closed |
|---|---|---|---|---|---|
| NGPA | 2026-06-04 | Service Charges | not shown | 1,344.86 | +1,344.86 → 0 |
| NTPT | 2026-08-06 | Service Charges | not shown | 427.10 | +427.10 → 0 |
| N-30003 | 2026-05-27 | Service Charges | not shown | 405.83 | +405.83 → 0 |
| NGFL | 2026-05-19 | Tip | 238.46 | 70.42 | −168.04 → 0 |
| NGMI | 2026-07-09 | Tip | 230.01 | 80.01 | −150.00 → 0 |
| NGVA | 2026-07-03 | Tip | 152.66 | 40.12 | −112.54 → 0 |
The customer paid $427.10 of service charge as part of a $4,975.89 card tender, so the money
- arrived on the debit side. No line credited it, so the day showed $427.10 more collected than
- earned. Adding the credit closes it exactly, and the single order responsible is
- square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY, carrying a
- :sales-order/service-charge of 427.10 on a 3,198.78 order.
| Client | Date | Tip before | Tip after | Imbalance before | After |
|---|---|---|---|---|---|
| NGFL | 2026-05-19 | 238.46 | 70.42 | −168.04 | 0.00 |
| NGMI | 2026-07-09 | 230.01 | 80.01 | −150.00 | 0.00 |
| NGVA | 2026-07-03 | 152.66 | 40.12 | −112.54 | 0.00 |
| NGLK | 2026-08-04 | 482.94 | 422.94 | −60.00 | 0.00 |
| NGPA | 2026-06-15 | 383.70 | 331.19 | −52.51 | 0.00 |
| NGLK | 2026-06-09 | 518.88 | 467.69 | −51.19 | 0.00 |
Every one overstated tip income, and in every case the correction equals the imbalance to the - cent. NGFL was crediting $238.46 of tips on a day where $168.04 had been given back.
-| Client | Date | Service charges before | After | Imbalance before | After |
|---|---|---|---|---|---|
| NGPA | 2026-06-04 | absent | 1,344.86 | +1,344.86 | 0.00 |
| NTPT | 2026-08-06 | absent | 427.10 | +427.10 | 0.00 |
| N-30003 | 2026-05-27 | absent | 405.83 | +405.83 | 0.00 |
| NGPA | 2026-07-02 | absent | 348.45 | +348.45 | 0.00 |
| NGA1 | 2026-06-04 | absent | 296.35 | +296.35 | 0.00 |
NGPA 2026-06-04 is the single largest repair in the ninety days: $1,344.86 of service charges - collected from customers and credited to no revenue account at all.
-| NGNP, 2026-06-25 | Before | After | Change |
|---|---|---|---|
| Tip (credit) | 93.90 | 92.10 | −1.80 |
| Service Charges (credit) | absent | 301.40 | +301.40 |
| Imbalance | +299.60 | 0.00 | 301.40 − 1.80 |
The two corrections pull in opposite directions and still land on zero, which is a useful - check that they are independent and neither is compensating for the other.
+In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.
Fifteen client-days above the ten-cent materiality threshold, out of 8,350. Everything else — 90 client-days — totals $2.26.
+Every step below was performed against a restored copy of the production database. Production itself was never touched.
| Client | Date | Variance | Note |
|---|---|---|---|
| NGBK | 2026-08-06 | +299.42 | tender exceeds order totals by exactly this much |
| NGMV | 2026-05-24 | +100.87 | a five-day cluster in late May, new to this window and not yet diagnosed |
| NGMV | 2026-05-20 | +72.05 | |
| NGMV | 2026-05-21 | +57.64 | |
| NGMV | 2026-05-22 | +14.41 | |
| NGMV | 2026-05-26 | +14.41 | |
| NGEB | 2026-05-19 | −79.01 | ezCater fee semantics — the plan's open question §15.4 |
| NGEB | 2026-06-23 | −50.28 | |
| NGEB | 2026-05-13 | −49.80 | |
| NGEB | 2026-07-29 | −20.00 | |
| NGDA | 2026-08-01 | −50.00 | auto-gratuity booked as a service charge |
| N-30012 | 2026-05-20 | +17.32 | two consecutive days, undiagnosed |
| N-30012 | 2026-05-21 | +12.99 | |
| NG4S | 2026-05-29 | −11.78 | undiagnosed |
| PNSP | 2026-07-12 | −0.14 | register rounding, just over threshold |
Widening from thirty days to ninety surfaced two clusters the shorter window could not see: NGMV across five days in late May, and NGEB across four days spanning May to July. NGEB is the known ezCater fee-semantics question. NGMV and N-30012 are new and worth a look before this ships — they are the kind of thing only a longer baseline exposes.
-The ten deactivated twins, excluded above, contribute a further $1,142.98 across three days. They should not be reported on at all once one client per location is settled.
-Running the job 56,700 times surfaced two problems that a normal nightly pass would hide, both now fixed or documented.
-| Problem | Effect | Fix |
|---|---|---|
dirty-sales-summaries scanned the whole index | 1,321 ms → 5.6 ms per client | bound the scan at the client boundary |
| Transactor sized for a toy database | ~35/min → 5,300/min | object cache 2 GB → 8 GB, heap 4 GB → 16 GB |
The second is a deployment setting rather than a code change, but it is dramatic: with a 2 GB object cache against a 27 GB database, the final 7,958 client-days of a pass were crawling at about 35 a minute. After resizing they completed in 90 seconds. Worth checking what production's transactor is sized at.
-;; connect to the restored backup -(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore")) - -;; baseline mode: clear the flag, restore tendered-only tips -(doseq [[c _] clients] - @(d/transact conn [[:db/retract c :client/feature-flags "summary-service-charges"]])) -(alter-var-root #'ss/get-tip (constantly baseline-get-tip)) - -;; mark the window and run the real job, then capture -(doseq [[c _] clients] (ss/mark-dirty c w90-start w90-end)) -(pmap (fn [[c code]] (ss/refresh-client! c code)) pending) - -;; the single orders behind the worked examples -(d/pull (d/db conn) '[*] - [:sales-order/external-id - "square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"]) -(d/pull (d/db conn) '[*] - [:sales-order/external-id - "square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"])-
The unit tests covering both fixes and the re-key:
-lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test
-;; => 15 tests, 26 assertions, 0 failures
- Re-keying a charge that has two parent orders duplicates the tender, because the other client's import then matches neither key and creates a second charge which cardinality-many appends. The migration must split shared charges first. Separately, remove-voided-orders retracts orders, and 63% of the contended clients' charges have two parents, so a retraction there deletes the other client's payment.
Balancing was only half the problem. The other half is whether an entity can change hands - between two clients at all. Rather than reason about it, every entity type the importers - create with a unique external id was audited, and ownership was read out of the database's - own history.
-| Entity | Swapped | Of total | Key before |
|---|---|---|---|
| Expected deposits (Square payouts) | 4,069 | 144,688 | square/payout/<id> |
| Cash drawer shifts | 2,628 | 69,291 | square/cash-drawer-shift/<id> |
| Refunds | 3,387 | 51,990 | fixed earlier |
| Sales orders, ezCater orders | 0 | — | already scoped |
Both payout and cash-shift endpoints are filtered by location, so two clients configured on - one Square location import the same records and collide on a single entity — the identical - defect refunds and charges had, in two types nobody had looked at.
-Nine pairs no longer share a location in the current configuration, so no point-in-time - check would find them — yet their data is still mixed. The largest are NGMJ/NGSC with 1,546 - affected entities, NGAK/NGMH with 952, NGBW/NGWD with 780 and NGNP/NGVZ with 645. This is the - argument for scoping the keys rather than only deactivating what currently looks shared: - deactivation is a snapshot fix that rots the next time someone configures a location twice.
-Worth knowing alongside it: 105 of the clients share a single Square auth token, - so these importers all reach into one Square account. The blast radius of a future - misconfiguration is the estate, not a pair.
-| Step | Result |
|---|---|
| Re-key expected deposits | 144,652 re-keyed · 0 collisions · 36 unscopable |
| Re-key cash drawer shifts | 69,291 re-keyed · 0 collisions · 0 unscopable |
| Entity counts before vs after | unchanged on all four types |
| Legacy-scheme keys remaining | 0 |
| Live re-import, 66 of 102 clients, 90-day window | 0 entities created · 0 values rewritten |
| Ownership changes since the re-key | 0 deposits · 0 shifts · 0 refunds |
| Summaries recomputed and compared | 18,900 · 0 differences |
| Deactivate the duplicate client at each shared location | 10 locations · shared locations remaining: 0 |
| Give every order its own payment record | 189,167 re-keyed · 77,599 copied |
| Payments owned by two orders | 11,469 → 0 |
| Client-scope refunds, payouts and cash-drawer shifts | counts unchanged · 0 collisions |
| Live Square import afterwards | 0 orders with duplicated payment · 0 shared payments |
| Ownership changes after the change | 0 refunds · 0 payouts · 0 shifts |
The re-import is the important negative result. Every payout and shift the importer fetched - resolved to the client's own entity and wrote nothing — the re-key is transparent to the - importer, which is exactly what the legacy-key fallback is for. And recomputing all 18,900 - client-days afterwards reproduced the previous figures exactly — largest difference 0.000000, so re-keying 213,943 - entities moved no money at all.
-One caveat worth stating plainly: re-keying stops future contention, but it does not - retrospectively re-attribute records that were claimed by the wrong client while the - configuration was shared. Those stay where they were last written. Correcting them is a - separate exercise, and one to do only once the business decides which client owns each - location.
+The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 77,599, matching the number of copies it reported making.
+The first attempt at copying shared payments derived each payment's Square identifier by stripping a fixed prefix. That is right the first time a payment is seen, but once it has been re-keyed to one client, a second order meeting it later read the already-scoped key as the identifier and scoped it twice — NGCD-CD-NGCC-CC-<id>. The importer then created a fresh payment, doubling the tender on five clients by $3,000–$7,000 each. It was caught because the totals were absurd, not because the code looked wrong. The fix recovers the scope from the record itself; client codes contain dashes, so it cannot be done by pattern. A test now runs the step one order at a time, which is the arrangement that exposes it.
remove-voided-orders is dangerousThe schema says a charge belongs to its order:
+Fifteen client-days across ninety days and 8,350 client-days, totalling $850.12. Everything else — 90 further days — comes to $2.26, the largest being 9.00¢, which is till rounding.
{:db/ident :sales-order/charges, :db/isComponent true}
+ | Client | Date | Variance | What it is |
|---|---|---|---|
| NGBK | 2026-08-06 | +299.42 | Square recorded more payment than the orders account for |
| NGMV | May 20–26 | +259.38 | five days, undiagnosed |
| NGEB | May 13 – Jul 29 | −199.09 | four days, ezCater fee treatment — an open question |
| NGDA | 2026-08-01 | −50.00 | auto-gratuity recorded as a service charge |
| N-30012 | May 20–21 | +30.31 | two days, undiagnosed |
| NG4S | 2026-05-29 | −11.78 | undiagnosed |
| PNSP | 2026-07-12 | −0.14 | till rounding, just over the threshold |
:db/isComponent means Datomic treats charges as parts of the order rather
- than independent records. So [:db/retractEntity <order>] deletes the order
- and its charges. That is the documented behaviour, and normally it is what you want.
remove-voided-orders asks Square for the last ten days of orders, keeps the ones
- Square reports as voided, and retracts them. Correct on its own terms.
The problem is that where two clients shared a location, both clients' orders resolved to the - same charge entity, because charge keys carried no client:
+The ten deactivated duplicate records contribute a further $1,142.98 across three days. They are excluded above and should be excluded from reporting generally, since they are now dormant copies.
+Two clusters — NGMV and N-30012, both in late May — are new and unexplained. They only became visible because the window was widened to ninety days; a thirty-day view did not reach them. They are worth a look before this ships.
NGCD order ──┐ - ├──> charge 17592490524 ← one entity, two parents -NGCC order ──┘+
| Item | Who decides | Why it matters |
|---|---|---|
| Which client record survives at each shared location | the business | the newer record generally has no history before the split, so keeping it loses years of the location's books |
| Which revenue account service charges post to | accounting | currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances |
| Whether to correct records the wrong client already owns | the business | the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared |
remove-voided-orders | engineering | safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes |
The production backup had not written a restore point since 2025-03-10 — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.
+The database server was sized for a toy dataset: a 2 GB cache against 27 GB of data. A recompute was crawling at about 35 client-days a minute; after raising the cache to 8 GB the remaining 7,958 finished in 90 seconds. Worth checking what production is set to.
+;; the restored database, untouched production as of 2026-08-14 22:52 +(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore")) + +;; the two orders behind the worked examples +(d/pull (d/db conn) '[*] [:sales-order/external-id + "square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"]) +(d/pull (d/db conn) '[*] [:sales-order/external-id + "square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"]) + +;; the gate: no payment may have two parent orders +(rk/charges-with-multiple-parents (d/db conn) orders) ;; => 0 + +;; ownership history — which records ever changed client +(->> (d/datoms (d/history (d/db conn)) :aevt :sales-refund/client) + (filter :added) + (reduce (fn [m d] (update m (:e d) (fnil conj #{}) (:v d))) {}) + (filter (fn [[_ owners]] (> (count owners) 1))) + count)
Retract either order and Datomic deletes that shared charge. The other client's order survives - with its sales intact but its payment gone: the day silently goes out of balance and the tender - disappears from the current database value. Measured on the restore, - 35,870 of 56,829 charges (63%) in the contended clients' recent window have two - parent orders.
-The re-key stops new sharing but does not un-share those. Two remedies, either sufficient: - split them by re-importing the affected window now that keys are client-scoped, or guard the - retraction so it detaches a shared charge instead of deleting it. The guard is small and makes - the operation safe whatever shape the data is in.
+The comparison tool is committed as auto-ap.jobs.compare-sales-summaries, which diffs summaries between two points in the same database using as-of — so "before" is production's own stored figures rather than a re-simulation. Unit tests: lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test.