integreat/docs/2026-08-15-sales-summary-rollout-plan.md

Sales-summary balancing — rollout plan

Steps to execute, in order. Every step is either reversible or verifiable before the next one begins. The one behaviour change that alters a client's books is behind a per-client feature flag that is off by default, so merging and deploying this branch changes nothing on its own.

Measured on a restored copy of production (backup point 209608347), 210 clients over 2026-05-10 → 2026-08-07: 1,258 client-days out of balance / $69,560.10 → 279 days / $7,790.54, with zero days knocked out of balance and zero already-balanced days altered.

Of the 279 days left, 171 are not balancing faults — they are days where a client's sales were never imported while its refunds were. Step 9 is about those, and it is the most important thing in this document.

---

Before you start

Flag introduced	summary-service-charges — off by default
Migration to run once	auto-ap.jobs.rekey-square-external-ids/migrate-all!
Expected migration runtime	~13 minutes for 19M orders on a warm cache
Nothing here touches	invoices, payments, the ledger, or any client without the flag set

One prerequisite that is not code. Ten Square locations are configured against two client records each. Someone in the business has to decide which record survives at each. The newer record usually has no history from before the split, so keeping it loses years of that location's books. Do this before step 3.

---

Step 1 — Deploy the code

Deploy the branch as normal. The flag is absent from every client, so:

• tips are calculated exactly as they are today,
• no Service Charges line is written.

The only changes that take effect immediately are the safe ones: imbalance logging, the dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix, and the importer's new client-scoped keys.

The importer starts writing client-scoped keys straight away, and reads both schemes. That is deliberate and is what makes the deploy independent of the migration. Do not remove the legacy lookup in square.core3/existing-id yet — see step 8.

Verify before moving on. After one nightly import cycle:

;; refunds, payouts and shifts must not have doubled
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))

Compare against the same counts taken immediately before deploy. Growth should be ordinary daily volume. A near-doubling means the legacy fallback is not working — stop and roll back the deploy.

---

Step 2 — Guard remove-voided-orders

Do this before the migration, not after. :sales-order/charges is :db/isComponent true, so retracting an order cascades into its payments. Until step 3 finishes there are still payments with two parent orders, and deleting one client's voided order can take the other client's payment with it.

Either leave remove-voided-orders switched off until step 3 completes, or change it to detach a payment that has more than one parent rather than delete it. Detaching is worth doing regardless — it makes the safety a property of the deletion rather than of the migration having been run first.

See docs/2026-08-15-remove-voided-orders-risk.md.

---

Step 3 — Retire the duplicate client records

Business decision from the top of this document. Deactivate the losing record's Square location so the importer stops fetching for it. The record itself stays; its history is untouched.

Verify: no Square location is configured against two active client records.

---

Step 4 — Run the migration

(require '[auto-ap.jobs.rekey-square-external-ids :as rk])

;; read-only first — check :collisions is empty for every attribute
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)

;; then the whole thing
(rk/migrate-all! 2000)

Runs in about thirteen minutes over 19M orders. It is idempotent and resumable — a record that already carries the right name is skipped, so it can be stopped and re-run without consequence.

If it appears to crawl, the cause is almost certainly garbage collection in the process driving it, not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free retained memory in the REPL and re-measure before changing anything about the database.

Verify — all four must read zero to migrate and zero unscopable:

(rk/unscoped-report (d/db conn))
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0}

;; and the gate that this work exists for
(rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn))))
;; => 0

Note unscoped-report's :no-owner column is not a gap: ~283k payout-stub payments carry no :charge/client attribute of their own, so it cannot verify them by attribute. plan resolves ownership through whatever refers to them and is the figure to trust.

---

Step 5 — Recompute summaries, flags still off

(require '[auto-ap.jobs.sales-summaries :as ss])
(ss/refresh-sales-summaries 90)

This is the pass that banks the deduplication. Capture the result before going further — you will need it as the baseline for step 6, and it cannot be reconstructed afterwards:

(require '[auto-ap.tools.compare-sales-summaries :as cmp])   ; test/dev classpath
(def before (cmp/summaries-in (d/db conn) start end))
(spit "before.edn" (pr-str before))

Do not use d/as-of to compare summary amounts. :ledger-mapped/amount, ledger-side and account are :db/noHistory, so past values are discarded. A summary that has since been recomputed reads back through as-of with its amounts absent, which looks like a legitimate balanced day. Capture live, before and after, and diff the captures.

---

Step 6 — Turn the flag on, a few restaurants at a time

Needs accounting sign-off first: summary-service-charges posts to 49000 Service Income, chosen so the work could be measured. It affects reporting, never whether a day balances.

@(d/transact conn [{:db/id [:client/code "NGxx"]
                    :client/feature-flags ["summary-service-charges"]}])
(ss/refresh-sales-summaries 90)

Start with two or three restaurants, confirm, then widen.

Verify against the capture from step 5:

(def after (cmp/summaries-in (d/db conn) start end))
(cmp/compare-window ...)   ; both arguments live database values, never as-of

The two numbers that matter — both were zero across all 18,900 client-days in testing:

• :balanced->unbalanced must be 0
• previously-balanced days whose lines changed must be 0

If either is non-zero, retract the flag for the affected clients and re-run step 5. The flag is the rollback: removing it restores today's behaviour exactly.

---

Step 7 — Re-enable remove-voided-orders

Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2.

---

Step 8 — Remove the legacy key lookup

Only once plan reports :to-migrate 0 and has stayed there through several import cycles. Drop the second branch of square.core3/existing-id. At that point two clients sharing a location becomes structurally incapable of producing a shared record, rather than prevented by a convention a future import could break.

This is the last step and there is no hurry.

---

Step 9 — Deal with the refunds that have no sales behind them

The most important item in this document, and the only one that is not just execution.

156 of the 279 remaining days are a record carrying refunds on a day it recorded no sales at all. 132 of those fall before that client's first ever order. Nine records are affected, holding 659 refunds worth $15,225.24 dated before their own first order:

Client	First order	Refunds before it	Value
NGDL	2026-07-29	96	$2,751.50
NGWN	2026-07-30	95	$2,218.61
NGDU	2026-08-02	94	$2,188.44
NG4S	2026-05-29	78	$2,168.30
NGPS	2026-05-26	81	$2,167.60
NGCT	2026-08-01	98	$1,789.49
NGVC	2026-08-02	95	$1,594.17
NGDS	2026-01-14	22	$347.13

They did not make those refunds. The database's own ownership history shows a $35.35 refund dated 26 February belonging to NGDG that day and taken over by NGDU on 12 August, with others flipping between the two several times a day across 12–15 August. Unscoped refund keys meant whichever import ran last took ownership — fault 1, seen from the other end. Steps 1 and 4 stop it recurring; they do not undo it.

Note NG4S and NGPS are not among the ten shared locations, so this is wider than the duplicate records.

Two ways to close it, and the business has to pick:

1. Import the missing sales. Correct if these records are meant to have their own books. Their refunds are real and the matching orders exist in Square.
2. Move the refunds to the record that has the sales. Correct if the record is a duplicate that should never have had books of its own.

Before either, check whether the surviving record already holds the same Square refund — within the ninety-day window all 192 orphans did, but outside it payouts and shifts have no counterpart at all, so a blanket delete would destroy the only copy of some records.

;; per client: refunds dated before that client's own first order
(let [first-order (->> (d/q '[:find [?d ...] :in $ ?c
                              :where [?o :sales-order/client ?c] [?o :sales-order/date ?d]]
                            (d/db conn) [:client/code "NGDU"])
                       (reduce (fn [a b] (if (.before a b) a b))))]
  (->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c
              :where [?r :sales-refund/client ?c]]
            (d/db conn) [:client/code "NGDU"])
       (filter #(.before (:sales-refund/date %) first-order))
       count))

Until this is resolved those days stay out of balance, on purpose. A summary change to close them was written and measured — it works, closes 156 days and $4,820.19, and breaks nothing — and it was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not being imported. A test asserts the day stays unbalanced so nobody closes it without reading this.

---

What this will not fix

The 108 real trading days still out of balance, $1,995.36 — the NGBR/NGBK tender gap ($6,358.99 of tender against $6,059.57 of order totals on one day), the ezCater fee question, and unexplained clusters on NGMV and NGEB.

Plus 15 days / $974.99 where a processing fee lands on a day with no trading — the same shape as step 9 but from the payout side, so it needs the payout modelled rather than a rule in the summary.

---

Two operational findings, unrelated to the summaries

• The production backup had not written a restore point since 2025-03-10 — about seventeen months — although data files were still uploading daily. Worth an alert on restore-point age.
• The database server is sized for a much smaller dataset: a 2 GB cache against 27 GB of data. Worth checking what production is set to.