# Sales-summary balancing — rollout plan Steps to execute, in order. Every step is either reversible or verifiable before the next one begins. The one behaviour change that alters a client's books is behind a per-client feature flag that is **off by default**, so merging and deploying this branch changes nothing on its own. Measured on a restored copy of production (backup point `209608347`), 210 clients over 2026-05-10 → 2026-08-07: **1,258 client-days out of balance / $69,560.10 → 279 days / $7,790.54**, with zero days knocked out of balance and zero already-balanced days altered. Of the 279 days left, **171 are not balancing faults** — they are days where a client's sales were never imported while its refunds were. Step 9 is about those, and it is the most important thing in this document. --- ## Before you start | | | |---|---| | Flag introduced | `summary-service-charges` — off by default | | Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` | | Expected migration runtime | ~13 minutes for 19M orders on a warm cache | | Nothing here touches | invoices, payments, the ledger, or any client without the flag set | **One prerequisite that is not code.** Ten Square locations are configured against two client records each. Someone in the business has to decide which record survives at each. The newer record usually has no history from before the split, so keeping it loses years of that location's books. Do this before step 3. --- ## Step 1 — Deploy the code Deploy the branch as normal. The flag is absent from every client, so: - tips are calculated exactly as they are today, - no `Service Charges` line is written. The only changes that take effect immediately are the safe ones: imbalance logging, the dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix, and the importer's new client-scoped keys. **The importer starts writing client-scoped keys straight away, and reads both schemes.** That is deliberate and is what makes the deploy independent of the migration. Do not remove the legacy lookup in `square.core3/existing-id` yet — see step 8. **Verify before moving on.** After one nightly import cycle: ```clojure ;; refunds, payouts and shifts must not have doubled (count (d/datoms (d/db conn) :aevt :sales-refund/external-id)) (count (d/datoms (d/db conn) :aevt :expected-deposit/external-id)) (count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id)) ``` Compare against the same counts taken immediately before deploy. Growth should be ordinary daily volume. A near-doubling means the legacy fallback is not working — **stop and roll back the deploy**. --- ## Step 2 — Guard `remove-voided-orders` Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so retracting an order cascades into its payments. Until step 3 finishes there are still payments with two parent orders, and deleting one client's voided order can take the other client's payment with it. Either leave `remove-voided-orders` switched off until step 3 completes, or change it to detach a payment that has more than one parent rather than delete it. Detaching is worth doing regardless — it makes the safety a property of the deletion rather than of the migration having been run first. See `docs/2026-08-15-remove-voided-orders-risk.md`. --- ## Step 3 — Retire the duplicate client records Business decision from the top of this document. Deactivate the losing record's Square location so the importer stops fetching for it. The record itself stays; its history is untouched. **Verify:** no Square location is configured against two active client records. --- ## Step 4 — Run the migration ```clojure (require '[auto-ap.jobs.rekey-square-external-ids :as rk]) ;; read-only first — check :collisions is empty for every attribute (dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys) ;; then the whole thing (rk/migrate-all! 2000) ``` Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that already carries the right name is skipped, so it can be stopped and re-run without consequence. If it appears to crawl, the cause is almost certainly garbage collection in the process driving it, not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free retained memory in the REPL and re-measure before changing anything about the database. **Verify — all four must read zero to migrate and zero unscopable:** ```clojure (rk/unscoped-report (d/db conn)) (dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys) ;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0} ;; and the gate that this work exists for (rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn)))) ;; => 0 ``` Note `unscoped-report`'s `:no-owner` column is not a gap: ~283k payout-stub payments carry no `:charge/client` attribute of their own, so it cannot verify them by attribute. `plan` resolves ownership through whatever refers to them and is the figure to trust. --- ## Step 5 — Recompute summaries, flags still off ```clojure (require '[auto-ap.jobs.sales-summaries :as ss]) (ss/refresh-sales-summaries 90) ``` This is the pass that banks the deduplication. **Capture the result before going further** — you will need it as the baseline for step 6, and it cannot be reconstructed afterwards: ```clojure (require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath (def before (cmp/summaries-in (d/db conn) start end)) (spit "before.edn" (pr-str before)) ``` > **Do not use `d/as-of` to compare summary amounts.** `:ledger-mapped/amount`, `ledger-side` and > `account` are `:db/noHistory`, so past values are discarded. A summary that has since been > recomputed reads back through `as-of` with its amounts *absent*, which looks like a legitimate > balanced day. Capture live, before and after, and diff the captures. --- ## Step 6 — Turn the flag on, a few restaurants at a time Needs accounting sign-off first: `summary-service-charges` posts to **49000 Service Income**, chosen so the work could be measured. It affects reporting, never whether a day balances. ```clojure @(d/transact conn [{:db/id [:client/code "NGxx"] :client/feature-flags ["summary-service-charges"]}]) (ss/refresh-sales-summaries 90) ``` Start with two or three restaurants, confirm, then widen. **Verify** against the capture from step 5: ```clojure (def after (cmp/summaries-in (d/db conn) start end)) (cmp/compare-window ...) ; both arguments live database values, never as-of ``` The two numbers that matter — both were zero across all 18,900 client-days in testing: - `:balanced->unbalanced` must be **0** - previously-balanced days whose lines changed must be **0** If either is non-zero, retract the flag for the affected clients and re-run step 5. The flag is the rollback: removing it restores today's behaviour exactly. --- ## Step 7 — Re-enable `remove-voided-orders` Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2. --- ## Step 8 — Remove the legacy key lookup Only once `plan` reports `:to-migrate 0` and has stayed there through several import cycles. Drop the second branch of `square.core3/existing-id`. At that point two clients sharing a location becomes structurally incapable of producing a shared record, rather than prevented by a convention a future import could break. This is the last step and there is no hurry. --- ## Step 9 — Deal with the refunds that have no sales behind them **The most important item in this document, and the only one that is not just execution.** 156 of the 279 remaining days are a record carrying refunds on a day it recorded no sales at all. 132 of those fall *before that client's first ever order*. Nine records are affected, holding **659 refunds worth $15,225.24 dated before their own first order**: | Client | First order | Refunds before it | Value | |---|---|---:|---:| | NGDL | 2026-07-29 | 96 | $2,751.50 | | NGWN | 2026-07-30 | 95 | $2,218.61 | | NGDU | 2026-08-02 | 94 | $2,188.44 | | NG4S | 2026-05-29 | 78 | $2,168.30 | | NGPS | 2026-05-26 | 81 | $2,167.60 | | NGCT | 2026-08-01 | 98 | $1,789.49 | | NGVC | 2026-08-02 | 95 | $1,594.17 | | NGDS | 2026-01-14 | 22 | $347.13 | They did not make those refunds. The database's own ownership history shows a $35.35 refund dated 26 February belonging to `NGDG` that day and taken over by `NGDU` on 12 August, with others flipping between the two several times a day across 12–15 August. Unscoped refund keys meant whichever import ran last took ownership — fault 1, seen from the other end. Steps 1 and 4 stop it recurring; they do not undo it. Note `NG4S` and `NGPS` are **not** among the ten shared locations, so this is wider than the duplicate records. Two ways to close it, and the business has to pick: 1. **Import the missing sales.** Correct if these records are meant to have their own books. Their refunds are real and the matching orders exist in Square. 2. **Move the refunds to the record that has the sales.** Correct if the record is a duplicate that should never have had books of its own. Before either, check whether the surviving record already holds the same Square refund — within the ninety-day window all 192 orphans did, but **outside it payouts and shifts have no counterpart at all**, so a blanket delete would destroy the only copy of some records. ```clojure ;; per client: refunds dated before that client's own first order (let [first-order (->> (d/q '[:find [?d ...] :in $ ?c :where [?o :sales-order/client ?c] [?o :sales-order/date ?d]] (d/db conn) [:client/code "NGDU"]) (reduce (fn [a b] (if (.before a b) a b))))] (->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c :where [?r :sales-refund/client ?c]] (d/db conn) [:client/code "NGDU"]) (filter #(.before (:sales-refund/date %) first-order)) count)) ``` **Until this is resolved those days stay out of balance, on purpose.** A summary change to close them was written and measured — it works, closes 156 days and $4,820.19, and breaks nothing — and it was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not being imported. A test asserts the day stays unbalanced so nobody closes it without reading this. --- ## What this will not fix The 108 real trading days still out of balance, $1,995.36 — the NGBR/NGBK tender gap ($6,358.99 of tender against $6,059.57 of order totals on one day), the ezCater fee question, and unexplained clusters on NGMV and NGEB. Plus 15 days / $974.99 where a processing fee lands on a day with no trading — the same shape as step 9 but from the payout side, so it needs the payout modelled rather than a rule in the summary. --- ## Two operational findings, unrelated to the summaries - **The production backup had not written a restore point since 2025-03-10** — about seventeen months — although data files were still uploading daily. Worth an alert on restore-point age. - **The database server is sized for a much smaller dataset**: a 2 GB cache against 27 GB of data. Worth checking what production is set to.