integreat/docs/2026-08-15-sales-summary-rollout-plan.md

# Sales-summary balancing — rollout plan

Steps to execute, in order. Every step is either reversible or verifiable before the next one
begins. Both behaviour changes are behind per-client feature flags that are **off by default**, so
merging and deploying this branch changes nothing on its own.

Measured on a restored copy of production (backup point `209608347`), 210 clients over
2026-05-10 → 2026-08-07: **1,258 client-days out of balance / $69,560.10 → 123 days / $2,970.35**,
with zero days knocked out of balance and zero already-balanced days altered.

---

## Before you start

| | |
|---|---|
| Branch | `worktree-sales-summary-balance` |
| Flags introduced | `summary-service-charges`, `summary-refund-only-returns` — both off by default |
| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` |
| Expected migration runtime | ~13 minutes for 19M orders on a warm cache |
| Nothing here touches | invoices, payments, the ledger, or any client without the flags set |

**One prerequisite that is not code.** Ten Square locations are configured against two client records
each. Someone in the business has to decide which record survives at each. The newer record usually
has no history from before the split, so keeping it loses years of that location's books. Do this
before step 3.

---

## Step 1 — Deploy the code

Deploy the branch as normal. Both flags are absent from every client, so:

- tips are calculated exactly as they are today,
- no `Service Charges` line is written,
- no `Returns` line is written on refund-only days.

The only changes that take effect immediately are the safe ones: imbalance logging, the
dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix,
and the importer's new client-scoped keys.

**The importer starts writing client-scoped keys straight away, and reads both schemes.** That is
deliberate and is what makes the deploy independent of the migration. Do not remove the legacy
lookup in `square.core3/existing-id` yet — see step 8.

**Verify before moving on.** After one nightly import cycle:

```clojure
;; refunds, payouts and shifts must not have doubled
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))
```

Compare against the same counts taken immediately before deploy. Growth should be ordinary daily
volume. A near-doubling means the legacy fallback is not working — **stop and roll back the deploy**.

---

## Step 2 — Guard `remove-voided-orders`

Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so
retracting an order cascades into its payments. Until step 3 finishes there are still payments with
two parent orders, and deleting one client's voided order can take the other client's payment with
it.

Either leave `remove-voided-orders` switched off until step 3 completes, or change it to detach a
payment that has more than one parent rather than delete it. Detaching is worth doing regardless —
it makes the safety a property of the deletion rather than of the migration having been run first.

See `docs/2026-08-15-remove-voided-orders-risk.md`.

---

## Step 3 — Retire the duplicate client records

Business decision from the top of this document. Deactivate the losing record's Square location so
the importer stops fetching for it. The record itself stays; its history is untouched.

**Verify:** no Square location is configured against two active client records.

---

## Step 4 — Run the migration

```clojure
(require '[auto-ap.jobs.rekey-square-external-ids :as rk])

;; read-only first — check :collisions is empty for every attribute
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)

;; then the whole thing
(rk/migrate-all! 2000)
```

Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that
already carries the right name is skipped, so it can be stopped and re-run without consequence.

If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
retained memory in the REPL and re-measure before changing anything about the database.

**Verify — all four must read zero to migrate and zero unscopable:**

```clojure
(rk/unscoped-report (d/db conn))
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0}

;; and the gate that this work exists for
(rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn))))
;; => 0
```

Note `unscoped-report`'s `:no-owner` column is not a gap: ~283k payout-stub payments carry no
`:charge/client` attribute of their own, so it cannot verify them by attribute. `plan` resolves
ownership through whatever refers to them and is the figure to trust.

---

## Step 5 — Recompute summaries, flags still off

```clojure
(require '[auto-ap.jobs.sales-summaries :as ss])
(ss/refresh-sales-summaries 90)
```

This is the pass that banks the deduplication. **Capture the result before going further** — you
will need it as the baseline for step 6, and it cannot be reconstructed afterwards:

```clojure
(require '[auto-ap.tools.compare-sales-summaries :as cmp])   ; test/dev classpath
(def before (cmp/summaries-in (d/db conn) start end))
(spit "before.edn" (pr-str before))
```

> **Do not use `d/as-of` to compare summary amounts.** `:ledger-mapped/amount`, `ledger-side` and
> `account` are `:db/noHistory`, so past values are discarded. A summary that has since been
> recomputed reads back through `as-of` with its amounts *absent*, which looks like a legitimate
> balanced day. Capture live, before and after, and diff the captures.

---

## Step 6 — Turn the flags on, a few restaurants at a time

Needs accounting sign-off first, on two points:

- `summary-service-charges` posts to **49000 Service Income**. Chosen so the work could be measured.
  It affects reporting, never whether a day balances.
- `summary-refund-only-returns` posts to **41300 Returns**, the account already used for returns. It
  moves the *recognition date* of a return onto the day the refund settled, so a refund settling
  after month end lands in the later period.

```clojure
@(d/transact conn [{:db/id [:client/code "NGxx"]
                    :client/feature-flags ["summary-service-charges"
                                           "summary-refund-only-returns"]}])
(ss/refresh-sales-summaries 90)
```

Start with two or three restaurants, confirm, then widen.

**Verify** against the capture from step 5:

```clojure
(def after (cmp/summaries-in (d/db conn) start end))
(cmp/compare-window ...)   ; both arguments live database values, never as-of
```

The two numbers that matter — both were zero across all 18,900 client-days in testing:

- `:balanced->unbalanced` must be **0**
- previously-balanced days whose lines changed must be **0**

If either is non-zero, retract the flags for the affected clients and re-run step 5. The flags are
the rollback: removing them restores today's behaviour exactly.

---

## Step 7 — Re-enable `remove-voided-orders`

Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2.

---

## Step 8 — Remove the legacy key lookup

Only once `plan` reports `:to-migrate 0` and has stayed there through several import cycles. Drop
the second branch of `square.core3/existing-id`. At that point two clients sharing a location
becomes structurally incapable of producing a shared record, rather than prevented by a convention a
future import could break.

This is the last step and there is no hurry.

---

## What this will not fix

123 client-days over ninety days, $2,970.35, of which only 33 are above ten cents.

| | Days | Variance | |
|---|---:|---:|---|
| Real trading days with genuine discrepancies | 108 | $1,995.36 | the NGBR/NGBK tender gap, the ezCater fee question, unexplained clusters on NGMV and NGEB |
| Processing fee on a day with no trading | 15 | $974.99 | same shape as the refund-only day, but the offsetting entry belongs to the payout, not to sales |

Refunds landing on days that *did* trade are deliberately left alone. Apportioning a return across a
day that also sold is a judgement call, and guessing at it is how correct books get quietly
rewritten.

---

## Two operational findings, unrelated to the summaries

- **The production backup had not written a restore point since 2025-03-10** — about seventeen
  months — although data files were still uploading daily. Worth an alert on restore-point age.
- **The database server is sized for a much smaller dataset**: a 2 GB cache against 27 GB of data.
  Worth checking what production is set to.