notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: notid:worktree-sales-summary-balance
Branches Tags
notid:master notid:sales-summary-balancing-v2 notid:staging notid:sales-summary-balancing notid:worktree-sales-summary-balance notid:integreat-fix-report notid:feat/scheduled-sales-summary-refresh notid:integreat-send-email notid:integreat-execute-refactor notid:docs/ssr-rendering-modernization-plan notid:integreat-fix-other-dates notid:integreat-unify-charts notid:integreat-add-transaction-manual notid:integreat-critique-wizard notid:test-taste-redesign notid:test-plan-docs notid:feat/complete-sales-summaries notid:feat/sales-parquet-migration
...
pull from: notid:sales-summary-balancing-v2
Branches Tags
notid:sales-summary-balancing-v2 notid:staging notid:sales-summary-balancing notid:worktree-sales-summary-balance notid:integreat-fix-report notid:feat/scheduled-sales-summary-refresh notid:integreat-send-email notid:master notid:integreat-execute-refactor notid:docs/ssr-rendering-modernization-plan notid:integreat-fix-other-dates notid:integreat-unify-charts notid:integreat-add-transaction-manual notid:integreat-critique-wizard notid:test-taste-redesign notid:test-plan-docs notid:feat/complete-sales-summaries notid:feat/sales-parquet-migration

6 Commits
worktree-s ... sales-summ

Author SHA1 Message Date
Bryce
79a4b457b0 docs: bring the rollout plan's later steps up to date with the backfill 
The headline and the new backfill step were current, but step 11 and
"what this will not fix" still carried pre-backfill figures. The backfill
did not just change the totals — it changed step 11's population.

Before it, nine records held refunds dated before their own first order:
660 refunds worth $15,237.02. Seven of those were shared-location twins
whose refunds only looked orphaned because their orders had never been
imported; replaying the window gave them their orders and the refunds
stopped predating them.

Two are left, and they are a different case — neither shares a Square
location, so there is no twin holding the other half:

  NG4S  first order 2026-05-29  79 refunds  $2,180.08  10 days
  NGPS  first order 2026-05-26  81 refunds  $2,167.60   7 days

Step 11 now recommends trying backfill-history on them first, with a
window reaching back before their first order, since that is exactly what
resolved the other seven.

"What this will not fix" re-measured: 122 days / $2,379.45, of which 106
are real trading days ($1,151.80) and 16 are refunds with no sales
imported ($1,227.65). Only 3 of the trading days are on shared-location
records, all already diagnosed. The other 103 days and $502.96 have been
identical in every run of this analysis — deactivated, live, and
backfilled — and are the floor this work reaches.

Also adds the backfill's ~5.9 hour runtime to the up-front table, and
notes that "two entities per Square object" holds automatically for new
imports but needs step 5 for existing history.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 19:04:50 -07:00
Bryce
1558851c18 feat(square): backfill history so shared-location records converge 
Re-keying stops two client records on one Square location fighting over a
record, but it does not make their books equal. Sales orders have always
been keyed by client, so each record built its own order history from the
start. Refunds, payouts and cash-drawer shifts were not, so only ONE
record holds each of them — whichever imported it last. The migration
freezes that ownership rather than evening it out.

The record left without them shows returns from its own orders and no
refunds against them. NGBK held 158,535 orders and five refunds. On
2026-05-11 both NGBK and NGBR held the same 221 orders; NGBK had no
refunds, NGBR had two worth $2,232.29, and NGBK was out by $2,232.29 to
the cent.

Rather than manufacture copies, ask Square again. Client-scoped keys mean
each record now creates its own copy of whatever it reads, so replaying a
window converges the two histories with no code inventing a duplicate.
`backfill-history` does that for a date range across orders, payouts,
refunds and shifts. After it, all ten pairs held matching order and
refund counts.

Fixes a capped read found by doing this: the refunds import asked Square
for a location's refunds and read only the first page — no cursor, no
date range. Square pages at a hundred, so a location with more refunds
silently returned a hundred and the response looked complete. That is why
each twin held almost exactly 100 refunds and why an earlier import added
exactly 1,000 across ten locations. `refunds`/`upsert-refunds` now follow
the cursor and accept a window.

Re-measured from the same fresh restore, duplicates left active:

  before  1,191 days out of balance / $70,276.50
  after     122 days out of balance / $2,379.45   (32 material)
            1,069 into balance, 0 out, 0 already-balanced days altered

The shared records went from 423 days / $18,508.39 to 3 days / $648.84 —
NGBK and NGBR at $299.42 each (the known tender-versus-order gap) and
NGDA at $50.00 (auto-gratuity as a service charge). The zero-regression
guarantee is restored too: the six days that broke without the backfill
were tips reversed on one record whose refund sat on the twin, and all
six closed once both sides had their own copy.

This beats retiring the duplicate records, which left 279 days and
$7,790.54, and it needs no decision about whose history to abandon.

Unchanged across every run: for the 190 clients that do not share a
location, 119 days and $1,730.61, same five restaurants.

Cost: 5.9 hours for ninety days across twenty records. Every Square call
shares one 25 req/s throttle, refunds and shifts cost one API call each,
and backfill-history imports three clients at a time. Reads are not the
limit — existing-id measures 32 microseconds.

31 tests, 76 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 17:58:59 -07:00
Bryce
4a1817711d docs: re-measure from a fresh restore with the duplicates left active 
The report's figures were taken with one client record of each shared
pair deactivated and a live Square import run afterwards. That is no
longer how this deploys — both records stay live and the re-key is what
separates them — so the numbers described a configuration that will not
exist.

Re-run from scratch: fresh restore of backup point 209608347 (verified
back to 16,545,495 charges and legacy keys before starting), the new
month-wise migration, then two full ninety-day recomputes.

  before  1,451 days out of balance / $81,023.96
  after     542 days out of balance / $20,239.00
            915 into balance, 6 out, 17,916 summaries untouched

Two corrections to claims that no longer hold:

- "zero days knocked out of balance" is now six. All six are the same
  shape — the Tip line falls by a round amount and the day breaks by
  exactly that — and all six are on shared-location records. Each is a
  tip reversed on one record whose refund went to its twin: the fix makes
  a hidden mis-attribution visible rather than causing one.
- the migration takes ~38 minutes, not ~13. The month-wise walk adds
  per-month index overhead, and the earlier figure predated it.

What did NOT move is the part that should not. For the 190 clients that
do not share a Square location the residue is 119 days and $1,730.61 in
both runs, with the same five restaurants accounting for it (NG4S, NGMV,
NGEB, NGPS, N-30012). The arithmetic fixes behave identically whatever is
done to the duplicates, which is a stronger check than either run alone.

$18,508.39 of the $20,239.00 — 91% — sits on the twenty shared-location
records. The report now states plainly what retiring the duplicates would
be worth (~$12,000 of variance across ~260 client-days per ninety days),
while noting the comparison is not perfectly isolated because the earlier
run also included a live import.

Migration re-measured: 16,236,839 re-keyed, 500,438 cloned, plan reports
{:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0},
and the multi-parent gate reads 0 across all 5,158,470 orders of the last
year. Step 10's refunds re-measured too: 660 worth $15,237.02 dated
before their own record's first order, 140 of the 171 no-sales days
falling before it.

31 tests, 76 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 09:59:32 -07:00
Bryce
967e77e443 feat(square): walk the migration newest month first, so stopping early is safe 
migrate-all! drove the charge split from all-order-ids, which streams
:aevt — ascending entity id, so oldest first. On the production copy that
means the first several hours are spent on 2019 and 2020 data no import
will ever read, leaving the recent end (the part the importer actually
touches) for last. Interrupt it there and the data is unmigrated exactly
where it matters.

Now:

- refunds, payouts and cash-drawer shifts run first. Together they are
  ~266k records and take seconds, so an interruption cannot leave them
  half done.
- the long order walk then runs a month at a time from the current month
  backwards, logging ::month-complete with per-month counts.

Stop it after any month and everything from that month forward is fully
scoped, so imports can resume against a partially migrated database and
the older tail can be finished later — the re-run skips what is done.
While a tail remains unmigrated, existing-id's ownership guard is what
keeps it safe.

order-months-newest-first tiles [start end] windows with no gaps (each
month ends the day before the next begins) and is bounded below by a
constant comfortably older than the oldest order. Windows are walked via
the :sales-order/client+date index. Verified against the restored copy:
141 windows from 2026-08 back, August returning 241,126 orders and July
476,235, each dated inside its window.

all-order-ids keeps its old behaviour but its docstring now warns that a
prefix of it is the oldest orders, not a sample — the trap that made an
earlier verification gate read 2019-2021 data.

31 tests, 76 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 08:35:02 -07:00
Bryce
10d0d01b82 fix(square): do not resolve a record that belongs to another client 
Two clients on one Square location double a day's tender in the window
between deploying and finishing the migration. Reproduced end to end:

  1. charge X carries the legacy key square/charge/P and belongs to
     client B's order
  2. client A's payout import resolves X through existing-id's legacy
     fallback and renames it into A's scope
  3. client B's next order import matches neither scheme, so it mints a
     second charge
  4. :sales-order/charges is cardinality-many and orders transact as
     plain maps, so nothing retracts the first

B's order ends up holding two charges for one payment — $200 of tender
for a $100 payment — and running the migration on that state produces
square/charge/BBB-LB-AAA-LA-P, the same double-scoped shape that already
doubled tender on five clients once during this work.

existing-id's legacy branch now declines any record already owned by a
different client, reading the owner attribute and, for charges that
predate :charge/client, the client of the referencing order. Declining is
also correct on its merits: the write then lands on this client's own
copy, which is what the scoped keys exist to create. The payout path also
writes :charge/client/:charge/location alongside the key, so a charge's
scope and its owner can no longer disagree.

The guard is transitional and gets deleted with the legacy branch it
protects, at rollout step 9.

Rollout resequenced for the decision to leave duplicate client records
active: no deactivation, no "which record survives" call, and the risk
window closed by pausing the importer across deploy + migrate rather than
by removing one of the two writers. Both records converge to independent
stable histories once every key carries its owner.

Also from review:

- migrate-all! now collision-checks the charge pass like the other three
  attributes instead of discovering a clash mid-run over 17M rows
- split-and-rekey-charges! logs progress every 200 batches; an
  interrupted 19M-order run left no trail
- unscoped-report's docstring no longer promises a zero its :no-owner
  column cannot reach; plan is named as the authoritative signal
- the rollout's pre-flight asked for a :collisions key plan never
  returns, so it silently passed on every database
- the multi-parent gate sampled (take 400000 (all-order-ids db)), which
  streams :aevt — ascending entity id — and so read the OLDEST 2% of
  orders: 2019-12-31 to 2021-06-03, before any of the contention it
  looks for. Now every order of the last year via the client+date index,
  5,159,787 on the restored copy, reading 0
- the report claimed same-client pairs get copied once batches split
  them. They do not, at any batch size; verified at batch-size 1 and now
  pinned by a test

30 tests, 72 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 08:09:53 -07:00
Bryce
57a84dae11 fix(sales-summaries): stop days falling out of balance 
Three faults were leaving restaurant days out of balance — one in the
data, two in the arithmetic — plus a fourth that turned out to be a
missing-data problem and is deliberately left visible. Measured over ninety days on a restored
copy of production (210 clients, 18,900 client-days): 1,258 days out of
balance and $69,560.10 becomes 279 days and $7,790.54 — of which 171 are
not arithmetic faults at all, but days whose sales were never imported.

979 days repaired, none knocked out of balance, and not one
already-balanced day altered — verified line by line (category, side,
amount to the cent, account), not just on each day's bottom line.

THE DATA FAULT

Ten Square locations were configured against two client records each.
Sales orders scoped their identifier by client; refunds, card payments,
payouts and cash-drawer shifts used the bare Square id. Those attributes
are :db.unique/identity, so both clients' imports resolved to a single
entity and the last writer won — 3,387 refunds, 4,069 payouts and 2,628
cash-drawer shifts changed hands over time, across 19 client pairs of
which only 10 are visible in today's configuration.

Worse, one payment could belong to two orders. :sales-order/charges is
:db/isComponent, so removing a voided order cascaded into payments the
other client still needed.

Fixes: client-scope the four key schemes; look the record up under both
schemes so the change deploys before the migration finishes; and a
migration that gives every order its own payment. Run over the whole
database that is 19,040,785 orders walked, 9,100,314 payments re-keyed
and 200,027 copied, ending with 17,047,142 payments scoped, none left to
rename, none unscopable, and no payment owned by more than one order.
Idempotent and resumable; about thirteen minutes.

THE ARITHMETIC FAULTS

- Refunded tips stayed on the books. get-tip summed tips by joining
  through :sales-order/charges, so a return-only order — no tender to
  join through — contributed nothing while its reversal sat unread on
  :sales-order/tip. Additive, not substitutive: where an order does have
  a tender the tender is the correct source.

- Service charges were collected but never earned. Nothing read
  :sales-order/service-charge. Now credited for Square orders only, both
  signs, behind summary-service-charges.

The flag is off by default, so deploying this changes nothing until a
client is opted in. docs/2026-08-15-sales-summary-rollout-plan.md has the
steps.

WHAT IS DELIBERATELY NOT FIXED

156 of the 279 remaining days carry refunds on a record that recorded no
sales at all that day, and 132 of those fall before that client's first
ever order. The refunds are not theirs: ownership history shows a $35.35
refund dated 26 February belonging to NGDG that day and taken over by
NGDU on 12 August, flipping between the two several times a day. Across
nine records, 659 refunds worth $15,225.24 sit on a record dated before
its own first order — unscoped keys let whichever import ran last take
ownership.

A rule closing those days was written and measured (156 days, $4,820.19,
nothing broken) and then removed. An unbalanced day is the only visible
signal that a restaurant's sales are not being imported; balancing it
would remove the alarm and leave the fire. A comment and a test hold that
decision in place. Step 9 of the rollout plan is the real fix, and it
needs a business decision.

SUPPORTING

- Install schema attributes before the tuples that compose them. A tuple
  in schema.edn is built from an attribute in cloud-migration-schema.edn,
  so every test fixture died in setup — very likely why sales summaries
  had no tests before this.
- Log each day's imbalance and its suspect lines.
- Bound the dirty-summary scan to one client: 1,321 ms to 5.6 ms.
- compare-sales-summaries lives in test/clj as auto-ap.tools.* — it is a
  verification harness, not part of the running application. Its
  docstring now warns that d/as-of cannot be used to compare summary
  amounts: :ledger-mapped/amount, ledger-side and account are
  :db/noHistory, so a recomputed summary reads back with its amounts
  absent and looks like a legitimate balanced day.

26 tests, 62 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-15 20:17:58 -07:00
12 changed files with 2797 additions and 129 deletions
Expand all files Collapse all files

115
docs/2026-08-15-remove-voided-orders-risk.md Normal file
View File

@@ -0,0 +1,115 @@
---
title: remove-voided-orders can delete another client's payments
type: risk
date: 2026-08-15
status: open — decide before merging the re-key
---
# `remove-voided-orders` can delete another client's payments
Measured on the restored backup, 2026-08-15. This risk is **pre-existing** — nothing in the
sales-summary work created it — but it is live right now, and the re-key work touches the same
data, so it should be understood before merging.
## The mechanism, in four steps
**1. Charges are component entities of an order.**
```clojure
;; resources/schema.edn
{:db/ident :sales-order/charges
:db/valueType :db.type/ref
:db/isComponent true ;; <- this is the load-bearing bit
:db/cardinality :db.cardinality/many}
```
`:db/isComponent true` tells Datomic the charges *belong to* the order. It is what lets you
transact an order with its tenders nested inside, and it means the charges have no independent
existence as far as Datomic is concerned.
**2. `retractEntity` on a component parent deletes the children too.**
That is the documented behaviour of `:db/retractEntity`: it recursively retracts component
values. `square.core3/remove-voided-orders` ends with exactly that:
```clojure
(s/map (fn [[o]]
[[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]]))
```
It asks Square for the last 10 days of orders, keeps the ones that should *not* be imported —
voided and cancelled orders — and retracts any of those we already stored. That is correct and
desirable on its own: a voided order should not sit in the books.
**3. But one charge can be shared by two orders.**
When two clients are configured on the same Square location, both import the same Square data.
Order keys embed the client, so each client gets its own order entity. Charge keys did **not**
embed the client, and `:charge/external-id` is `:db.unique/identity`, so both clients' orders
resolved to *the same charge entity*:
```
NGCD order 17592395490523 ──┐
├──> charge 17592490524 ← one entity, two parents
NGCC order 17592395511722 ──┘
```
**4. So retracting one order deletes a charge the other order still points at.**
Datomic sees a component and removes it. The surviving order keeps its line items — its sales —
but its tender is gone. The day then shows revenue with no payment against it, the summary goes
out of balance, and the payment is gone from the current database value. (History retains it, so
it is recoverable by someone who knows to look, but nothing in the app will show it again.)
## How exposed are we
Measured over the 10 contended clients across 2026-07-13 → 08-14:
| | |
|---|---|
| Charges examined | 56,829 |
| **Referenced by more than one order** | **35,870 (63%)** |
So this is not a theoretical corner. Roughly two thirds of the charges in that population have
two parents, and any voided order among them takes a charge down with it.
The exposure window for *new* damage is the rolling 10 days `remove-voided-orders` searches, but
the shared charges themselves span the whole period the locations were double-configured.
## What changes after Phase 0 and the re-key, and what doesn't
- **Phase 0 (done on the restore)** stops new sharing: only one client per location imports now,
so no new order pairs form.
- **The re-key (done for refunds and the contended clients' charges)** makes sharing structurally
impossible going forward, because a charge key now contains the client code.
- **Neither retroactively splits the 35,870 charges that are already shared.** They still have two
parents. Until they are split, `remove-voided-orders` remains capable of deleting a payment
belonging to the other client.
This is why plan §3.3 forbids retracting anything — including any historical cleanup of the
duplicate clients' data — until a verification query shows zero charges with more than one parent.
## Options, roughly in order of preference
1. **Split the shared charges, then let removal run normally.** Re-import the affected window now
that keys are client-scoped, so each client creates its own charge entity. This reuses the
import path rather than hand-constructing component entities. Verify with a query for charges
having more than one referencing order; it must reach zero.
2. **Guard the retraction.** Before retracting an order, check whether any of its charges are
referenced by another order; detach those (retract the `:sales-order/charges` ref rather than
the charge) and retract the rest. Small, contained change, and it makes the operation safe
regardless of what shape the data is in — worth doing on its own merits even after a split.
3. **Do nothing and accept it.** Only defensible once every location has a single client *and*
the historical shared charges are gone. Not true today.
## What I did about it during the validation run
I ran the import on the restore with `remove-voided-orders` **skipped**, and ran the other steps
(`upsert-locations`, `upsert`, `upsert-payouts`, `upsert-refunds`) normally. That kept the
validation faithful to how the import behaves without risking silent payment loss in the data
the measurements were about to be taken from.
**Nothing in the production system has been changed.** This note is about a risk that already
exists there.

760
docs/2026-08-15-sales-summary-reconciliation.html Normal file
View File

@@ -0,0 +1,760 @@
<title>Ninety-Day Reconciliation</title>
<style>
:root {
--paper: #F6F8F7; --card: #FFFFFF; --ink: #141F1D; --ink-soft: #4A5C58;
--ink-faint: #7C8D89; --rule: #DCE4E1; --accent: #0E5B57; --accent-soft: #E3EFED;
--good: #1A6B49; --bad: #A03B26; --warn: #8A6410;
--shadow: 0 1px 2px rgba(20,31,29,.06), 0 8px 24px rgba(20,31,29,.05);
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
}
}
:root[data-theme="dark"] {
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
}
* { box-sizing: border-box; }
body { background: var(--paper); color: var(--ink);
font-family: system-ui, -apple-system, "Segoe UI", sans-serif;
font-size: 16px; line-height: 1.6; margin: 0; padding: 0 20px 96px; }
.wrap { max-width: 940px; margin: 0 auto; }
.measure { max-width: 66ch; }
.num { font-variant-numeric: tabular-nums; }
.mono { font-family: ui-monospace, "SF Mono", "Cascadia Code", monospace; font-variant-numeric: tabular-nums; }
header.masthead { padding: 72px 0 40px; border-bottom: 2px solid var(--ink); display: flex; flex-direction: column; gap: 14px; }
.eyebrow { font-size: 12px; letter-spacing: .14em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
h1 { font-family: Georgia, "Iowan Old Style", serif; font-size: clamp(34px, 5.4vw, 54px);
line-height: 1.08; font-weight: 600; letter-spacing: -.015em; margin: 0; text-wrap: balance; }
.standfirst { font-size: 19px; color: var(--ink-soft); margin: 0; max-width: 62ch; }
.meta { display: flex; flex-wrap: wrap; gap: 10px 28px; font-size: 13px; color: var(--ink-faint); padding-top: 6px; }
.meta b { color: var(--ink-soft); font-weight: 600; }
section { padding-top: 56px; display: flex; flex-direction: column; gap: 20px; }
h2 { font-family: Georgia, "Iowan Old Style", serif; font-size: 27px; font-weight: 600; letter-spacing: -.01em; margin: 0; text-wrap: balance; }
h3 { font-size: 14px; letter-spacing: .08em; text-transform: uppercase; color: var(--ink-soft); font-weight: 700; margin: 0; }
h4 { font-size: 18px; font-weight: 650; margin: 0; letter-spacing: -.01em; }
p { margin: 0; }
.measure p + p { margin-top: 14px; }
.ledger { display: grid; grid-template-columns: 1fr auto 1fr; border: 1px solid var(--rule);
border-radius: 4px; background: var(--card); box-shadow: var(--shadow); overflow: hidden; }
.ledger > div { padding: 26px 28px; display: flex; flex-direction: column; gap: 6px; }
.ledger .arrow { justify-content: center; align-items: center; border-left: 1px solid var(--rule);
border-right: 1px solid var(--rule); color: var(--ink-faint); font-size: 22px; background: var(--accent-soft); }
.side-label { font-size: 12px; letter-spacing: .12em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; }
.figure { font-size: clamp(28px, 4.4vw, 40px); font-weight: 650; line-height: 1.05; letter-spacing: -.02em; }
.figure.after { color: var(--good); }
.subfig { font-size: 14px; color: var(--ink-soft); }
.stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 14px; }
.stat { background: var(--card); border: 1px solid var(--rule); border-radius: 4px; padding: 18px 20px; display: flex; flex-direction: column; gap: 4px; }
.stat .k { font-size: 30px; font-weight: 650; letter-spacing: -.02em; line-height: 1; }
.stat .l { font-size: 13px; color: var(--ink-soft); }
.stat.zero .k { color: var(--good); }
.problem { border-left: 3px solid var(--accent); padding-left: 24px; display: flex; flex-direction: column; gap: 14px; }
.problem.two { border-left-color: var(--warn); }
.problem.three { border-left-color: var(--bad); }
.scroll { overflow-x: auto; border: 1px solid var(--rule); border-radius: 4px; background: var(--card); }
table { border-collapse: collapse; width: 100%; font-size: 14.5px; }
th, td { padding: 11px 16px; text-align: left; border-bottom: 1px solid var(--rule); white-space: nowrap; }
thead th { font-size: 11.5px; letter-spacing: .09em; text-transform: uppercase; color: var(--ink-faint); font-weight: 700; background: var(--accent-soft); }
tbody tr:last-child td { border-bottom: none; }
td.n, th.n { text-align: right; font-variant-numeric: tabular-nums; }
tr.total td { font-weight: 650; background: var(--accent-soft); }
.good { color: var(--good); font-weight: 650; }
.bad { color: var(--bad); font-weight: 650; }
.dim { color: var(--ink-faint); }
.callout { background: var(--card); border: 1px solid var(--rule); border-left: 3px solid var(--accent);
border-radius: 4px; padding: 20px 24px; display: flex; flex-direction: column; gap: 10px; }
.callout.warn { border-left-color: var(--warn); }
.callout .h { font-weight: 650; }
code { font-family: ui-monospace, "SF Mono", monospace; font-size: .9em; background: var(--accent-soft); padding: 1px 5px; border-radius: 3px; }
pre { margin: 0; padding: 20px; font-size: 13px; line-height: 1.7; white-space: pre; font-family: ui-monospace, "SF Mono", monospace; }
footer { margin-top: 72px; padding-top: 24px; border-top: 1px solid var(--rule); font-size: 13px; color: var(--ink-faint); display: flex; flex-direction: column; gap: 8px; }
ul { margin: 0; padding-left: 20px; display: flex; flex-direction: column; gap: 8px; }
.tech { font-size: 11px; letter-spacing: .08em; text-transform: uppercase; color: var(--accent);
font-weight: 700; border: 1px solid var(--accent); border-radius: 3px; padding: 2px 7px; display: inline-block; }
</style>
<div class="wrap">
<header class="masthead">
<div class="eyebrow">Sales summaries · measured on a restored production backup</div>
<h1>Ninety-Day Reconciliation</h1>
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — and a fourth, found late, that is a missing-data problem wearing a balancing problem's clothes. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
<div class="meta">
<span><b>Window</b> 2026-05-10 → 2026-08-07</span>
<span><b>Client-days</b> <span class="num">18,900</span></span>
<span><b>Clients</b> <span class="num">210</span></span>
<span><b>Nothing in production was changed</b></span>
</div>
</header>
<section>
<div class="ledger">
<div>
<span class="side-label">Today's calculation, ninety days re-run</span>
<span class="figure num">$70,276.50</span>
<span class="subfig"><span class="num">1,191</span> days out of balance · <span class="num">93.70%</span> clean</span>
</div>
<div class="arrow" aria-hidden="true">→</div>
<div>
<span class="side-label">The same ninety days, fixes on</span>
<span class="figure after num">$2,379.45</span>
<span class="subfig"><span class="num">122</span> days out of balance · <span class="num">99.35%</span> clean</span>
</div>
</div>
<div class="stats">
<div class="stat"><span class="k num">1,069</span><span class="l">client-days brought into balance</span></div>
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
<div class="stat"><span class="k num">96.6%</span><span class="l">of the variance removed</span></div>
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
</div>
<div class="measure">
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p>
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>1,069 days and $67,897.05</strong>, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.</p>
<p><strong>Most of what is left is not a balancing fault at all</strong>, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.</p>
</div>
</section>
<section>
<h2>The four problems</h2>
<div class="problem">
<h4>1. Two client records sharing one Square location</h4>
<div class="measure">
<p><strong>For the business:</strong> ten restaurant locations were set up twice in the system, as two separate clients. Both were importing from Square. Because the two records competed for the same payments and refunds, a refund would belong to one client for twenty minutes, then the other — so a day's books could gain or lose a refund depending on nothing but timing. On 2026-07-23 one client's summary was missing a $71.94 refund entirely, and was out of balance by exactly that amount.</p>
<p><span class="tech">technical</span> Sales orders scoped their identifier by client (<code>square/order/&lt;code&gt;-&lt;loc&gt;-&lt;id&gt;</code>), but refunds, card charges, payouts and cash-drawer shifts did not — they used the bare Square id. Those attributes are <code>:db.unique/identity</code>, so both clients' imports resolved to a single entity and the last writer won.</p>
<p>Reading ownership out of the database's own history, this had actually happened to <strong>3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts</strong>. And it has involved <strong>19 client pairs, of which only 10 are visible in today's configuration</strong> — nine more contended in the past and the configuration has since changed, so no point-in-time check would find them.</p>
</div>
</div>
<div class="problem two">
<h4>2. One payment record owned by two orders</h4>
<div class="measure">
<p><strong>For the business:</strong> the same collision meant a single card payment could be attached to both clients' copies of an order. That is worse than untidy. The nightly import removes orders Square reports as voided, and removing an order also removes its payments — so cancelling one client's order could silently delete the <em>other</em> client's payment, leaving a day showing sales with no money against them.</p>
<p><span class="tech">technical</span> <code>:sales-order/charges</code> is declared <code>:db/isComponent true</code>, so <code>[:db/retractEntity &lt;order&gt;]</code> cascades into the charges. In a 20,000-order sample of the affected clients, <strong>11,469 charges had two parent orders</strong>. This is why <code>remove-voided-orders</code> was left switched off during testing.</p>
</div>
</div>
<div class="problem three">
<h4>3. Tips refunded, and service charges credited nowhere</h4>
<div class="measure">
<p><strong>For the business:</strong> two arithmetic faults, both of which overstated or understated a day.</p>
<ul>
<li><strong>Refunded tips stayed on the books.</strong> When a guest was refunded, the tip came back too — but the summary still counted the original tip as income. On one NGLK day the books credited $482.94 of tips beside a $60.00 refund of that very tip.</li>
<li><strong>Service charges were collected but never earned.</strong> A catering or auto-gratuity charge is inside the card payment the customer makes, so it arrived as money taken — but no line recorded it as money earned. One NTPT order carried $427.10 that was credited to nothing at all; the largest single instance was <strong>$1,344.86</strong> in one day.</li>
</ul>
<p><span class="tech">technical</span> <code>get-tip</code> summed tips by joining through <code>:sales-order/charges</code>, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on <code>:sales-order/tip</code>. Nothing at all read <code>:sales-order/service-charge</code>.</p>
</div>
</div>
<div class="problem">
<h4>4. Refunds on records whose sales were never imported <span class="tech">not fixed — deliberately</span></h4>
<div class="measure">
<p><strong>This is why the duplicated restaurants looked so much worse than everyone else.</strong> Of the days still failing after the first three fixes, <strong>155 of the 423 on shared-location records had no sales orders at all</strong> — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.</p>
<p>The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded <em>any</em> order shows <strong>140 of 171 fall before that client had a single order in the system</strong> — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.</p>
<p><strong>Where the refunds came from.</strong> Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to <span class="mono">NGDG</span> that same day, and was taken over by <span class="mono">NGDU</span> on 12 August. Others flip between the two records several times a day across 12–15 August. <span class="mono">NGDU</span>'s first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.</p>
<p>Across the nine records, <strong>660 refunds worth $15,237.02 sit on a record dated before that record's first order.</strong> Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.</p>
<p><strong>Why it is deliberately left out of balance.</strong> The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and on this data it closes 16 of the 122 remaining days and $1,227.65. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.</p>
<p><span class="tech">technical</span> <code>get-returns</code> sums <code>:sales-order/returns</code> over orders scanned for the date. With no orders the sum is nil and no <code>Returns</code> line is written, while <code>get-refund-items</code> still credits <code>Card Refunds</code> from the <code>sales-refund</code> records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.</p>
</div>
</div>
</section>
<section>
<h2>What the fixes actually are</h2>
<div class="measure">
<p>Five changes. The first three stop two clients from sharing a record; the last two record
money that was being collected but not booked. Each is small — the difficulty was knowing
which line to change, not writing it. A sixth was written and then removed; it is described
at the end because the reasoning matters more than the code did.</p>
</div>
<h3>1 · Put the client in the record's name</h3>
<div class="measure">
<p>Every imported record has an identifier the importer uses to decide "have I seen this
before?". Sales orders already included the client; refunds, card payments, payouts and
cash-drawer shifts did not, which is precisely why two clients could land on one record.</p>
</div>
<div class="scroll">
<pre><span class="dim">;; before — the bare Square id, identical for both clients</span>
(str "square/refund/" (:id r)) <span class="dim">;; square/refund/NOkQOTIiJULWN6…</span>
<span class="dim">;; after</span>
(scoped-key "square/refund/" client location (:id r))
<span class="dim">;; square/refund/NGCD-CD-NOkQOTIiJULWN6…</span>
(defn scoped-key [prefix client location id]
(str prefix (:client/code client) "-"
(:square-location/client-location location) "-" id))</pre>
</div>
<div class="measure">
<p>Applied at five places in the Square importer: order payments, refunds, payouts (twice —
the record itself and the lookup that finds it) and cash-drawer shifts. ezCater orders
already did this and needed no change.</p>
</div>
<h3>2 · Find the existing record before writing, under either name</h3>
<div class="measure">
<p>This is the one that makes the change safe to deploy. The identifiers are unique keys, so
the importer relies on "same id, same record". Rename them and the next import matches
nothing — and would quietly create a <em>second</em> copy of every refund and payment in the
system, leaving the originals orphaned. So the importer looks up the record explicitly,
new name first, old name second, and writes to whichever it finds.</p>
</div>
<div class="scroll">
<pre>(defn existing-id [db attr prefix client location id]
(when id
(or (dc/entid db [attr (scoped-key prefix client location id)]) <span class="dim">;; new scheme</span>
(dc/entid db [attr (str prefix id)])))) <span class="dim">;; legacy scheme</span></pre>
</div>
<div class="measure">
<p>The result is pinned as the record's id on the way in, so the write lands on the existing
row regardless of which name it currently carries. The proof this worked is a count that did
not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was
re-named, and afterwards there were still exactly <strong>50,986 refunds, 144,688 payouts and
69,291 cash-drawer shifts</strong> — the same three figures as at the restore point.
Had the fallback lookup been missing, each of these would have doubled instead.</p>
<p><strong>The fallback also has to refuse.</strong> Reading the old name is what stops
duplicates; reading <em>anyone's</em> old name is what creates them. Two clients share a Square
location, so client A's payout import can resolve a payment that belongs to client B's order,
rename it into A's scope, and leave B's next import matching neither name — at which point B
mints a second payment and, because an order's payments are a set that is added to rather than
replaced, B's order ends up holding both. That is a doubled day's tender, and it was
reproduced end to end before being fixed. The lookup now declines any record already owned by
a different client, which is also the right answer on its merits: the write then lands on this
client's own copy, which is what the scoped names exist to create.</p>
<p>This is transitional. Once no legacy names remain, the fallback and the refusal are deleted
together and the guarantee stops depending on either.</p>
</div>
<h3>3 · Give every order its own payment record</h3>
<div class="measure">
<p>Renaming stops <em>new</em> collisions but does not undo old ones: a payment already shared
by two orders is still one row with two owners. The migration walks each order's payments and,
where another order has already claimed one, makes that order its own copy with the same
amounts and points the order at the copy.</p>
</div>
<div class="scroll">
<pre><span class="dim">;; for each order, for each of its payments:</span>
:keep <span class="dim">→</span> first order to claim it; rename in place
:clone <span class="dim">→</span> copy type, total, tip, tax, date, processor, note, receipt link
set the copy's client and location to this order's
retract this order's link to the shared payment
link it to the copy instead</pre>
</div>
<div class="measure">
<p>Run over the whole database that was <strong>16,236,839 renamed and 500,438 copied</strong>,
and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across
every order of the last year. The record count rose by about 500,438 — the number of copies it
reported making, which is the check that it created what it meant to and nothing else.</p>
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
<span class="mono">N-30003</span> — so a pattern cannot tell where the client name ends and the
Square id begins. Getting this wrong scoped some records twice and doubled their tender.</p>
</div>
<h3>4 · Count tips that were handed back</h3>
<div class="measure">
<p>Tips were summed by walking from the order to its payments. A refund-only order has no
payment attached, so its negative tip was invisible. The fix adds those tips rather than
replacing the calculation.</p>
</div>
<div class="scroll">
<pre><span class="dim">;; before</span>
:ledger-mapped/amount (tendered-tip c date)
<span class="dim">;; after</span>
:ledger-mapped/amount (+ (tendered-tip c date)
(untendered-tip c date))
<span class="dim">;; untendered-tip — tips on orders with no payment attached</span>
[?e :sales-order/tip ?tip]
(not [?e :sales-order/charges])</pre>
</div>
<div class="measure">
<p><strong>Adding rather than replacing is deliberate.</strong> Where an order does have a
payment, the payment is the correct source: real orders exist whose payment carries a tip the
order does not — an auto-gratuity recorded as a service charge, or a wallet tip missing from
the order totals. Reading the order instead would have dropped those. Three tests hold this
in place: the refund case must change, and the tendered and ordinary cases must not.</p>
</div>
<h3>5 · Credit Square service charges, both signs</h3>
<div class="measure">
<p>Nothing read the service-charge field at all. A new line credits it, for Square orders only
and for negative amounts as well as positive.</p>
</div>
<div class="scroll">
<pre>[?e :sales-order/service-charge ?service-charge]
(or-join [?e]
[?e :sales-order/vendor :vendor/ccp-square]
(and (not [?e :sales-order/vendor])
[?e :sales-order/external-id ?external-id]
[(clojure.string/starts-with? ?external-id "square/order/")]))</pre>
</div>
<div class="measure">
<p><strong>Why the vendor test has two branches.</strong> ezCater service charges are commission
the platform deducts from the restaurant, not money the diner hands over, so crediting them
would make a day worse rather than better — hence the Square-only condition. But whole eras of
Square orders carry no vendor field at all, and a test on vendor alone would silently credit
nothing. The second branch falls back to the order's own identifier.</p>
<p><strong>Why negatives matter.</strong> A returned catering fee arrives as a negative service
charge and is already deducted from the day's returns; dropping negatives would lose the
reversal. The line sits behind a per-client switch, off by default, so it can be turned on a
few restaurants at a time.</p>
</div>
<h3>6 · The change that was written, measured, and then taken out</h3>
<div class="measure">
<p>Worth recording, because the arithmetic case for it is good and someone will propose it
again. Where a day has refunds and no sales orders whatsoever, book a <code>Returns</code>
debit equal to that day's refunds:</p>
</div>
<div class="scroll">
<pre>(defn- refund-only-returns [c date]
(when-not (traded? c date)
(let [amount (refunded-total c date)]
(when-not (zero? amount) amount))))</pre>
</div>
<div class="measure">
<p>It works. Measured over the same ninety days it closed <strong>171 days and $5,795.18</strong>,
knocked nothing out of balance, and altered no already-balanced day — the guard makes it
incapable of touching a day that traded.</p>
<p>It was removed anyway. Those days are not quiet days; they are days whose sales were never
imported, and closing them removes the only visible sign of that. What is left in the code is
a comment saying so and a test asserting the day <em>stays</em> out of balance, so the next
person to notice the arithmetic finds the reasoning before they find the fix.</p>
</div>
<h3>Supporting changes</h3>
<div class="scroll">
<table>
<thead><tr><th>Change</th><th>Why</th></tr></thead>
<tbody>
<tr><td>Log each day's imbalance and its suspect lines</td><td>an out-of-balance day was only visible by opening the screen; now it can be queried</td></tr>
<tr><td>Stop the dirty-summary scan at the client boundary</td><td>it read every later client's summaries too — 1,321 ms to 5.6 ms per client</td></tr>
<tr><td>Split the recompute driver into a per-client function</td><td>lets a backfill spread clients across threads instead of grinding one at a time</td></tr>
<tr><td>Install schema attributes before the tuples that compose them</td><td>the test suite could not build an empty database at all, so no test could run</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>That last one is worth a sentence for engineers: <code>transact-schema</code> installed
schema.edn then cloud-migration-schema.edn, but a composite tuple in the first file is built
from an attribute in the second. Datomic will not create a tuple before its members exist, so
every test fixture died in setup. It is very likely why sales summaries had no tests before
this work.</p>
</div>
</section>
<section>
<h2>What each fix is worth</h2>
<div class="measure">
<p>The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates. All 18,900 client-day summaries in the window are included, whether or not the restaurant traded that day.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead>
<tbody>
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,191</td><td class="n">93.70%</td><td class="n">$70,276.50</td></tr>
<tr><td>+ refunded tips</td><td class="n">890</td><td class="n">95.29%</td><td class="n">$67,032.09</td></tr>
<tr class="total"><td>+ service charges</td><td class="n good">122</td><td class="n good">99.35%</td><td class="n good">$2,379.45</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$66,589.75 is what the three arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
</div>
<h3>Day-by-day effect of each change</h3>
<div class="scroll">
<table>
<thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead>
<tbody>
<tr><td>Refunded tips</td><td class="n">18,571</td><td class="n good">301</td><td class="n good">0</td><td class="n good">0</td><td class="n">$4,027.21</td></tr>
<tr><td>Service charges</td><td class="n">18,129</td><td class="n good">768</td><td class="n good">0</td><td class="n good">0</td><td class="n">$64,752.64</td></tr>
<tr class="total"><td>Both, end to end</td><td class="n">17,827</td><td class="n good">1,069</td><td class="n good">0</td><td class="n good">0</td><td class="n">$67,897.05</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p><strong>Neither fix touched a day that was already correct.</strong> Across all 18,900
client-days no balanced day was knocked out of balance, and no balanced day had a single
figure altered — 17,827 summaries came out byte-identical, and every one of the 1,073 that
moved was already wrong.</p>
<p>That claim did not hold on the first attempt, and how it was recovered is the useful part.
Measured before the historical backfill described below, six days broke — all of them a tip
reversed on one record whose refund sat on its twin, so removing the un-reversed tip left the
day short by exactly that amount. Replaying the window from Square gave both records their own
copy of every refund, and all six closed. The fix was never wrong; it was reading half a
transaction.</p>
<p>Service charges are by far the larger of the two fixes, moving $58,923.85 against the tip fix's $3,777.67.</p>
<p>That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.</p>
<p><strong>The two fixes account for every day they moved, exactly.</strong> Adding up the untendered-tip and service-charge amounts across all 984 changed days leaves a residue of <span class="mono">0.0000000002</span>. Nothing else moved those days; there is no unexplained remainder hiding a third effect, and the six that broke are accounted for by the same arithmetic as the 915 that healed.</p>
</div>
</section>
<section>
<h2>What it looks like on the page</h2>
<div class="measure">
<p>Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.</p>
</div>
<h3>A refunded tip — NGLK, 2026-08-04</h3>
<div class="scroll">
<table>
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
<tbody>
<tr><td><strong>Tip</strong></td><td class="n">482.94</td><td class="n">422.94</td></tr>
<tr><td class="dim">Card Refunds</td><td class="n dim">60.00</td><td class="n dim">60.00</td></tr>
<tr><td>Total money taken</td><td class="n">10,094.81</td><td class="n">10,094.81</td></tr>
<tr><td>Total money earned</td><td class="n">10,154.81</td><td class="n">10,094.81</td></tr>
<tr class="total"><td>Out of balance by</td><td class="n bad">−60.00</td><td class="n good">0.00</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is <span class="mono">square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY</span>: a tip of −60.00 on an order with no payment attached at all.</p>
</div>
<h3>A service charge — NTPT, 2026-08-06</h3>
<div class="scroll">
<table>
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
<tbody>
<tr><td><strong>Service Charges</strong></td><td class="n bad">not shown</td><td class="n">427.10</td></tr>
<tr><td class="dim">Card Payments</td><td class="n dim">4,975.89</td><td class="n dim">4,975.89</td></tr>
<tr><td>Total money taken</td><td class="n">7,777.20</td><td class="n">7,777.20</td></tr>
<tr><td>Total money earned</td><td class="n">7,350.10</td><td class="n">7,777.20</td></tr>
<tr class="total"><td>Out of balance by</td><td class="n bad">+427.10</td><td class="n good">0.00</td></tr>
</tbody>
</table>
</div>
<h3>The largest repairs of each kind</h3>
<div class="scroll">
<table>
<thead><tr><th>Client</th><th>Date</th><th>Line</th><th class="n">Before</th><th class="n">After</th><th class="n">Day closed</th></tr></thead>
<tbody>
<tr><td class="mono">NGPA</td><td>2026-06-04</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">1,344.86</td><td class="n good">+1,344.86 → 0</td></tr>
<tr><td class="mono">NTPT</td><td>2026-08-06</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">427.10</td><td class="n good">+427.10 → 0</td></tr>
<tr><td class="mono">N-30003</td><td>2026-05-27</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">405.83</td><td class="n good">+405.83 → 0</td></tr>
<tr><td class="mono">NGFL</td><td>2026-05-19</td><td>Tip</td><td class="n">238.46</td><td class="n">70.42</td><td class="n good">−168.04 → 0</td></tr>
<tr><td class="mono">NGMI</td><td>2026-07-09</td><td>Tip</td><td class="n">230.01</td><td class="n">80.01</td><td class="n good">−150.00 → 0</td></tr>
<tr><td class="mono">NGVA</td><td>2026-07-03</td><td>Tip</td><td class="n">152.66</td><td class="n">40.12</td><td class="n good">−112.54 → 0</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.</p>
</div>
</section>
<section>
<h2>What was done to the data, and how it was checked</h2>
<div class="measure">
<p>Every step below was performed against a restored copy of the production database. Production itself was never touched.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Step</th><th>Result</th></tr></thead>
<tbody>
<tr><td>Walk every order in the database, newest month first</td><td class="n">19,040,785 orders · 38 minutes</td></tr>
<tr><td>Give every order its own payment record</td><td class="n">16,236,839 re-keyed · 500,438 copied</td></tr>
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across every order of the last year — 5,158,470</span></td></tr>
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added 500,438, matching the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)</p>
<p><strong>The measurement above was taken with both client records of each pair left live, which is how this deploys.</strong> Nothing is deactivated and no business decision about which restaurant's history survives is needed. The risk that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why <code>existing-id</code> now refuses to resolve a record belonging to another client.</p>
</div>
<div class="callout">
<span class="h">The whole analysis was run again from nothing, and landed in the same place</span>
<p>Everything above was rebuilt from a fresh restore of the production backup, several times over, each time from the backup point itself rather than from a database an earlier run had touched: restore, re-key and split across all nineteen million orders, then two full ninety-day recomputes. The runs used deliberately different preparation — one deactivated the duplicate records, one left them live and untouched, one backfilled their history from Square — so their headline figures differ, and comparing them is how the recommendation below was reached. What did <strong>not</strong> move is the part that should not: for the 190 clients that do not share a Square location the residue is 119 days and $1,730.61 in every run, with the same five restaurants accounting for it. The arithmetic fixes behave identically no matter what is done to the duplicates, which is a stronger check on them than any single measurement.</p>
</div>
<div class="callout warn">
<span class="h">A bug in this work, found by measuring rather than reading</span>
<p>The first attempt at copying shared payments derived each payment's Square identifier by stripping a fixed prefix. That is right the first time a payment is seen, but once it has been re-keyed to one client, a second order meeting it later read the already-scoped key as the identifier and scoped it twice — <span class="mono">NGCD-CD-NGCC-CC-&lt;id&gt;</span>. The importer then created a fresh payment, doubling the tender on five clients by $3,000–$7,000 each. It was caught because the totals were absurd, not because the code looked wrong. The fix recovers the scope from the record itself; client codes contain dashes, so it cannot be done by pattern. A test now runs the step one order at a time, which is the arrangement that exposes it.</p>
</div>
</section>
<section>
<h2>What is still out of balance</h2>
<div class="measure">
<p>122 client-days out of 18,900, totalling <strong>$2,379.45</strong> — and only 32 of
those are above ten cents.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Where the remainder sits</th><th class="n">Days</th><th class="n">Variance</th></tr></thead>
<tbody>
<tr><td>The twenty records that share a Square location</td><td class="n good">3</td><td class="n good">$648.84</td></tr>
<tr class="total"><td>Every other client — 190 of the 210</td><td class="n">119</td><td class="n">$1,730.61</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p><strong>The shared-location records are now the clean part of the book.</strong> Three days
between all twenty of them: NGBK and NGBR at $299.42 each on 2026-08-06, which is the Square
tender-versus-order-total gap described below and not an attribution fault, and NGDA at $50.00,
an auto-gratuity booked as a service charge. Before the backfill those same records carried
423 days and $18,508.39.</p>
<p><strong>The other 119 days have not moved across any run of this analysis.</strong> Four
separate rebuilds — different databases, different preparation, one with the duplicates
deactivated and one without — all land on 119 days and $1,730.61, with the same five
restaurants accounting for almost all of it:</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Client</th><th class="n">Days</th><th class="n">Variance</th><th>What it is</th></tr></thead>
<tbody>
<tr><td class="mono">NG4S</td><td class="n">10</td><td class="n">$1,066.61</td><td>refunds arriving for a record with no sales imported — the fourth problem</td></tr>
<tr><td class="mono">NGMV</td><td class="n">5</td><td class="n">$259.38</td><td>late May, undiagnosed</td></tr>
<tr><td class="mono">NGEB</td><td class="n">4</td><td class="n">$199.09</td><td>ezCater fee treatment — an open question</td></tr>
<tr><td class="mono">NGPS</td><td class="n">7</td><td class="n">$172.82</td><td>undiagnosed</td></tr>
<tr><td class="mono">N-30012</td><td class="n">2</td><td class="n">$30.31</td><td>late May, undiagnosed</td></tr>
<tr class="total"><td class="dim">everyone else</td><td class="n dim">91</td><td class="n dim">$2.40</td><td class="dim">till rounding — pennies a day</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>Sixteen of the 122 are days a record had no sales imported at all, worth $1,227.65 — the
fourth problem, still deliberately visible. The clusters on NGMV, NGPS and NGEB are
unexplained and worth a look, though at under $650 across sixteen days they are no longer
urgent.</p>
</div>
</section>
<section>
<h2>Making the duplicated restaurants match</h2>
<div class="measure">
<p>Re-keying stops the two records fighting, but on its own it does not make them equal, and
the difference is worth stating plainly because it decides whether the books close.</p>
<p><strong>Orders were always duplicated; refunds never were.</strong> A sales order's
identifier has always carried its client, so each of the two records built its own order
history from the start. Refunds, payouts and cash-drawer shifts did not, so only ONE record
holds each of them — whichever imported it last. The migration freezes that ownership rather
than evening it out. The record left without them shows returns from its own orders and no
refunds to set against them, and is out of balance by exactly the amount its twin is holding.</p>
<p>On 2026-05-11 both NGBK and NGBR held the same 221 orders. NGBK had no refunds; NGBR had
two, worth $2,232.29; and NGBK's books were out by $2,232.29 to the cent. Across the whole
database NGBK held <strong>158,535 orders and five refunds</strong>.</p>
</div>
<div class="callout">
<span class="h">The fix is to ask Square again, not to manufacture copies</span>
<p>With client-scoped keys in place, every record now creates its own copy of whatever it
reads. So replaying the window from Square is all that is needed: each record imports the same
refunds independently and the two histories converge, without any code inventing a duplicate
and having to be trusted about it. <code>backfill-history</code> does exactly that for a date
range, and after it every one of the ten pairs held matching order and refund counts.</p>
<p>It closed <strong>420 of the 423 days</strong> the shared records were carrying, and
$17,859.55 of the $18,508.39. It is also what recovered the zero-regression guarantee above.</p>
</div>
<div class="callout warn">
<span class="h">One capped read, found by doing this</span>
<p>The refunds import asked Square for a location's refunds and read the first page of the
answer — no cursor, no date range. Square pages at a hundred, so a location with more than a
hundred refunds silently returned a hundred, and the response looked complete. That is why the
twins each held almost exactly 100 refunds, and why an earlier import added exactly 1,000
across ten locations. Following the cursor is a few lines; the reason it went unnoticed for so
long is that a capped list is indistinguishable from a short one.</p>
</div>
<div class="measure">
<p><strong>This turned out to be a better answer than retiring the duplicate records.</strong>
An earlier measurement that deactivated one record of each pair left 279 days and $7,790.54.
Backfilling instead, with both records live, leaves <strong>122 days and $2,379.45</strong> —
and it needs no business decision about which restaurant's history to abandon.</p>
</div>
</section>
<section>
<h2>How complete is this</h2>
<div class="measure">
<p>The importer understands both the old and new record names, so the change can be deployed
before the renaming finishes. That tolerance is a bridge, not a destination — while any
record still carries an unscoped name, two clients can land on it and the guarantee rests on
a convention rather than on the data. So the renaming was run to completion and measured.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Record type</th><th class="n">Total</th><th class="n">Client-scoped</th><th class="n">Still to rename</th><th class="n">Cannot be scoped</th></tr></thead>
<tbody>
<tr><td>Card payments</td><td class="n">17,045,933</td><td class="n good">17,045,933</td><td class="n good">0</td><td class="n good">0</td></tr>
<tr><td>Refunds</td><td class="n">50,986</td><td class="n good">50,986</td><td class="n good">0</td><td class="n good">0</td></tr>
<tr><td>Payouts</td><td class="n">144,688</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr>
<tr><td>Cash-drawer shifts</td><td class="n">69,291</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>Every record in the database now carries its owner's name, and the migration proposes no
further changes: asked what is left to do, it answers zero on all four record types. The 36
payouts are ones with no client or location recorded anywhere, on the record itself or on
anything referring to it, so there is nothing to name them after.</p>
<p>Renaming had to be driven from orders, because a payment's rightful owner is whichever
order refers to it — so completing it meant walking all <strong>19,040,785 orders</strong>, not
just the clients that look shared today. Nine client pairs contended in the past without
sharing a location now, and a migration scoped to the current configuration would have missed
every one of them.</p>
<p><strong>A caution about how completeness is counted.</strong> 282,649 card payments carry no
client attribute of their own — they are stubs the payout path creates, never referenced by an
order. A gate that checks the attribute reports these as "no owner" and looks like a gap. They
are not: their names are scoped, recovered from the deposit that holds them. The figures above
are counted the harder way, by asking the migration what it would still change, which resolves
each record's owner through whatever refers to it. Reading the attribute alone would have
understated completeness by a quarter of a million records — and an early draft of this report
did exactly that.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Shared payments after the migration</th><th class="n">Count</th><th>Meaning</th></tr></thead>
<tbody>
<tr><td><strong>Owned by more than one order</strong></td><td class="n good">0</td><td>whether the orders belong to different clients or the same one</td></tr>
<tr><td class="dim">checked across</td><td class="n dim">400,000 orders</td><td class="dim">spread through the whole database</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>The problem this work exists to solve is gone: no payment answers to two orders, so the
component relationship means what it says and deleting an order can no longer take another
order's money with it.</p>
<p><strong>Where Square splits one tender across two of a single client's own orders, the
payment stays shared — at any batch size.</strong> Both orders compute the same name, so there
is no second name for a copy to take, and a copy would double that client's takings for the
day. The mechanism is worth stating precisely, because it is not obvious from reading: once
the first order re-keys the payment it also writes the owner attributes in the same
transaction, so a later order recovers the bare Square id from those, computes the name the
payment already carries, and the guard <code class="mono">(not= old new-key)</code> drops the
row before any copy decision is reached. Verified by running the migration at a batch size of
one, which forces the two orders into separate batches: no copy is made.</p>
<p class="dim">An earlier draft of this report claimed the opposite — that such pairs would be
copied once the batches split them — and flagged it as unmeasured risk to check before
production. That was wrong, and it is recorded here rather than quietly deleted because it did
real damage: an independent reviewer cited this paragraph as evidence and raised a defect that
does not exist. A test now pins the behaviour at batch size one.</p>
<p>The guard on <code>remove-voided-orders</code> is still worth having regardless. It is
cheap, and it makes the safety a property of the deletion rather than of the migration having
been run first.</p>
</div>
<div class="callout">
<span class="h">Re-running is safe, and that was proved at full scale</span>
<p>After the complete pass, asking the migration what it would change next returns
<strong>nothing</strong> — 17,045,933 payments examined, none to rename, none unscopable. A
record that already carries the right name is left untouched, so the migration can be stopped,
resumed, or repeated without consequence.</p>
<p>Its speed is worth a note for whoever schedules it: the whole nineteen million orders were
walked in about <strong>thirty-eight minutes</strong>, month by month from the current month
backwards so that stopping early leaves the recent end done. An earlier attempt appeared to be
transactor-bound and was projected at two days, which is why a previous run narrowed it to the
analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process
driving the migration — freeing held memory took an unrelated recompute from 17 client-days a
minute to 4,515. Nothing about the database or the transactor needed to change.</p>
</div>
<div class="measure">
<p><strong>What follows from the gate reading zero.</strong> <code>unscoped-report</code> counts
these figures on demand. Now that unscoped is zero across the board, the importer's
understanding of the old name form can be removed — at which point two clients sharing a
location becomes structurally incapable of producing a shared record, rather than prevented by
a convention that a future import could quietly break. That removal is the one remaining step
of this piece of work.</p>
</div>
</section>
<section>
<h2>Decisions and risks still open</h2>
<div class="scroll">
<table>
<thead><tr><th>Item</th><th>Who decides</th><th>Why it matters</th></tr></thead>
<tbody>
<tr><td>Which client record survives at each shared location</td><td>the business</td><td>the newer record generally has no history before the split, so keeping it loses years of the location's books</td></tr>
<tr><td>Which revenue account service charges post to</td><td>accounting</td><td>currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances</td></tr>
<tr><td><strong>659 refunds on records that have no sales for them</strong></td><td>the business, then engineering</td><td>the top open item. $15,225.24 dated before the holding record's own first order. Either the missing sales get imported, or the refunds move to the record that has them — but the books cannot close until one of the two happens</td></tr>
<tr><td>Whether to correct records the wrong client already owns</td><td>the business</td><td>the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared</td></tr>
<tr><td><code>remove-voided-orders</code></td><td>engineering</td><td>safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes</td></tr>
</tbody>
</table>
</div>
<div class="callout warn">
<span class="h">Two operational findings, unrelated to the summaries</span>
<p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p>
<p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. Worth checking what production is set to.</p>
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in well under an hour. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
</div>
</section>
<section>
<h2>How to check any of this <span class="tech">technical</span></h2>
<div class="scroll">
<pre><span class="dim">;; the restored database, untouched production as of 2026-08-14 22:52</span>
(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore"))
<span class="dim">;; the two orders behind the worked examples</span>
(d/pull (d/db conn) '[*] [:sales-order/external-id
"square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"])
(d/pull (d/db conn) '[*] [:sales-order/external-id
"square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"])
<span class="dim">;; the gate: no payment may have two parent orders</span>
(rk/charges-with-multiple-parents (d/db conn) orders) <span class="dim">;; =&gt; 0</span>
<span class="dim">;; ownership history — which records ever changed client</span>
(->> (d/datoms (d/history (d/db conn)) :aevt :sales-refund/client)
(filter :added)
(reduce (fn [m d] (update m (:e d) (fnil conj #{}) (:v d))) {})
(filter (fn [[_ owners]] (&gt; (count owners) 1)))
count)</pre>
</div>
<div class="measure">
<p>The comparison tool is committed as <code>auto-ap.jobs.compare-sales-summaries</code>. Unit tests: <code>lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test</code>.</p>
</div>
<div class="callout warn">
<span class="h">Do not compare summaries with <code>as-of</code> — a correction to how this was measured</span>
<p>The obvious way to audit a recompute is to read the database at a point before it and diff:
Datomic keeps every past value, so no snapshot is needed. That is what
<code>compare-sales-summaries</code> was built to do, and for summary amounts it does not work.
<code>:ledger-mapped/amount</code>, <code>:ledger-mapped/ledger-side</code> and
<code>:ledger-mapped/account</code> are all declared <code>:db/noHistory true</code>, so
superseded values are discarded rather than retained. A historical read of a summary that has
since been recomputed can return its lines with the categories intact and the amounts simply
absent — which reads as a legitimate all-zero summary, not as an error.</p>
<p>Every figure in this report is therefore taken from a live read of the database immediately
after each pass, captured and stored outside it, and the before/after comparison is done
between those two captures. No historical read is involved anywhere in the numbers above. The
tool remains useful for categories and for which days changed; its docstring overstates what it
can recover, and that is worth correcting before someone relies on it for amounts.</p>
</div>
</section>
<footer>
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 122 remaining days only 32 are material, and just 3 of them sit on the twenty records that share a Square location.</span>
<span>Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on <code>as-of</code> above.</span>
</footer>
</div>

419
docs/2026-08-15-sales-summary-rollout-plan.md Normal file
View File

@@ -0,0 +1,419 @@
# Sales-summary balancing — rollout plan
Steps to execute, in order. Every step is either reversible or verifiable before the next one
begins. The one behaviour change that alters a client's books is behind a per-client feature flag
that is **off by default**, so merging and deploying this branch changes nothing on its own.
Measured on a restored copy of production (backup point `209608347`), 210 clients over
2026-05-10 → 2026-08-07, with the duplicate client records left active exactly as they will be in
production: **1,191 client-days out of balance / $70,276.50 → 122 days / $2,379.45**, of which only
32 are above ten cents. 1,069 days came into balance, none broke, and no already-balanced day had a
figure altered.
Of the $2,379.45 left, just **$648.84 across 3 days** is on the twenty shared-location records. The
other 119 days and $1,730.61 belong to ordinary clients and have not moved across any run of this
analysis.
Getting the shared records there needs step 5 — a historical backfill from Square. Without it they
carry 423 days and $18,508.39, because re-keying stops the two records fighting but does not give
each its own copy of the refunds.
---
## Before you start
| | |
|---|---|
| Flag introduced | `summary-service-charges` — off by default |
| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` |
| Expected migration runtime | ~38 minutes for 19M orders on a warm cache |
| Backfill runtime (step 5) | ~5.9 hours for 90 days across the 20 shared-location records — an overnight job |
| Nothing here touches | invoices, payments, the ledger, or any client without the flag set |
**Client configuration is left exactly as it is.** Ten Square locations are configured against two
client records each, and both stay active. The re-key is what resolves them: once every record
carries its owner in its key, each client's import resolves only its own records and the two
records keep independent, stable histories. No "which record survives" decision is needed, and
nothing is deactivated.
The consequence to be aware of: each Square payment, refund, payout and shift at a shared location
becomes **two entities, one per client record** — by design. That is the stable end state, not a
duplicate to clean up. If any report or export aggregates across client records, one restaurant's
takings would be counted twice at that layer. Nothing in this work changes that either way.
That holds automatically for everything imported *from now on*, because the keys carry the client.
It does **not** hold for history: refunds, payouts and shifts already in the database exist only
once, on whichever record imported them last, and re-keying freezes that rather than evening it out.
Step 5 is what brings the existing history into the same shape.
**The only window of risk is between deploying and finishing the migration**, while legacy keys
still exist for a client to resolve. Steps 2–6 exist to make that window effectively zero.
---
## Step 1 — Guard `remove-voided-orders`
Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so
retracting an order cascades into its payments. Until step 4 finishes there are still payments with
two parent orders, and deleting one client's voided order can take the other client's payment with
it.
Either leave `remove-voided-orders` switched off until step 4 verifies clean, or change it to detach
a payment that has more than one parent rather than delete it. Detaching is worth doing regardless —
it makes the safety a property of the deletion rather than of the migration having been run first.
See `docs/2026-08-15-remove-voided-orders-risk.md`.
---
## Step 2 — Pause the Square importer
**This is what makes the deploy safe, and it is easy to skip.** Steps 2 through 6 should be one
maintenance action, not separate days' work.
While legacy keys exist, `square.core3/existing-id` falls back to them — and at a shared location
that is the one code path that can reach across client records. Running the migration with imports
paused means no client is resolving keys while the keys are being rewritten, so the window closes
entirely rather than merely narrowing.
The migration itself takes about **38 minutes** for all 19M orders, so the pause is short — and
if you need it shorter, see step 6: you can resume imports before it finishes.
---
## Step 3 — Deploy the code
Deploy the branch. The flag is absent from every client, so:
- tips are calculated exactly as they are today,
- no `Service Charges` line is written.
The only changes that take effect immediately are the safe ones: imbalance logging, the
dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix,
and the importer's new client-scoped keys.
**The importer reads both key schemes**, so the deploy does not depend on the migration having
finished. Two protections cover the interval before it does: imports are paused (step 2), and
`existing-id` refuses to resolve a record that already belongs to a different client. Do not remove
the legacy lookup yet — see step 10.
---
## Step 4 — Run the migration
Run it immediately after the deploy, while imports are still paused.
```clojure
(require '[auto-ap.jobs.rekey-square-external-ids :as rk])
;; read-only first — no two entities may want the same key. `plan` does NOT return a
;; :collisions key; you have to hand its :new-keys to `collisions` yourself.
(rk/collisions (:new-keys (rk/plan (d/db conn) :charge/external-id rk/charge-prefix)))
;; => [] (anything else: stop, do not migrate)
;; then the whole thing
(rk/migrate-all! 2000)
```
`migrate-all!` runs this same check itself, on every attribute including charges, and throws
rather than transacting if it finds one. Running it by hand first just means finding out before
the 38-minute walk rather than partway through it.
Runs in about thirty-eight minutes over 19M orders. It is **idempotent and resumable** — a record that
already carries the right name is skipped, so it can be stopped and re-run without consequence.
**It is also ordered so that stopping early is survivable.** Refunds, payouts and cash-drawer
shifts go first — a quarter of a million records, seconds of work — so an interruption cannot catch
them half done. The long part then walks orders **a month at a time, from the current month
backwards**, logging `::month-complete` as each finishes:
```
::month-complete :month "2026-08" :rekeyed 118203 :cloned 2244
::month-complete :month "2026-07" :rekeyed 241887 :cloned 4611
...
```
That ordering is the recovery plan. If it dies, everything from the last logged month forward is
fully scoped — and that recent window is what the importer actually reads — so **you can resume
imports against a partially migrated database** and finish the older tail later. Walking oldest
first would have spent the first several hours on 2019 data no import will touch, leaving exactly
the wrong end done.
If you do resume imports mid-migration, the ownership guard in `existing-id` is what keeps the
unmigrated tail safe: a client cannot resolve onto another client's legacy-keyed record.
If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
retained memory in the REPL and re-measure before changing anything about the database.
**Verify.** Two checks, doing two different jobs — run both.
**(a) Completeness, across everything.** `plan` must report nothing left to do, for all four
attributes:
```clojure
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
;; => {:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0}
```
Read `:to-migrate 0` **and** `:unscopable 0`. This is the authoritative signal, and it covers all
17M charges.
`unscoped-report` is useful colour but is not the gate: its `:no-owner` column never reaches zero
for charges, because ~283k payout stubs carry no `:charge/client` of their own and it classifies
by attribute rather than by resolving ownership. Judge completeness by `plan`.
**(b) The safety gate for the cascade** — no payment may answer to two orders, or re-enabling
`remove-voided-orders` in step 9 can delete a payment another order still needs. Check **every**
order in the last year, with no sampling:
```clojure
(let [db (d/db conn)
cs (map first (d/q '[:find ?c :where [?c :client/code _]] db))
year (java.util.Date. (- (.getTime (java.util.Date.)) (long (* 365 86400000))))]
(rk/charges-with-multiple-parents
db (map first (iol-ion.query/scan-sales-orders db cs year nil))))
;; => 0
```
On the restored copy that is 5,158,470 orders — 27% of the table — via the
`:sales-order/client+date` index. A year is chosen deliberately: `remove-voided-orders` only ever
deletes orders Square reports as voided, which are recent, so that is where the destructive risk
lives. Completeness across all of history is check (a)'s job, not this one.
> Do **not** sample this with `(take n (rk/all-order-ids db))`. `all-order-ids` streams `:aevt`,
> which is ascending entity id, so a `take` returns the *oldest* orders — on the restored copy the
> first 400,000 are all from 2019–2021, before any of the contention this gate looks for. It would
> report a confident zero having inspected none of the relevant data.
---
## Step 5 — Backfill the shared-location clients from Square
**Skip this and the ten duplicated restaurants stay badly out of balance.** It is the difference
between 122 client-days out of balance and 542.
Sales orders have always been keyed by client, so both records of a pair built their own order
history. Refunds, payouts and cash-drawer shifts were not, so only ONE record holds each of them.
Re-keying freezes that ownership; it does not even it out. The record left without them shows
returns from its own orders and no refunds against them — NGBK held 158,535 orders and five
refunds — and is out of balance by exactly what its twin is holding.
Rather than manufacture copies, ask Square again. Client-scoped keys mean each record now creates
its own copy of whatever it reads, so replaying the window makes the two histories converge:
```clojure
(require '[auto-ap.square.core3 :as sq])
(require '[clj-time.core :as t])
@(apply sq/backfill-history
(t/date-time 2026 5 10) (t/date-time 2026 8 9)
["NGBK" "NGBR" "NGCD" "NGCC" "NGVG" "NGVC" "NGEZ" "NGJS" "NGDG" "NGDU"
"NGDV" "NGDS" "NGWC" "NGWN" "NGHY" "NGHA" "NGDA" "NGDL" "NGCL" "NGCT"])
```
**Verify** — every pair should hold matching order and refund counts in the window:
```clojure
;; per pair, per side: window orders and window refunds. The two sides should agree.
```
Measured on the restored copy: all ten pairs matched afterwards, and the shared records went from
423 days and $18,508.39 out of balance to 3 days and $648.84.
**Budget an overnight run.** This took **5.9 hours** for ninety days across the twenty records.
Every Square call in the process shares one 25-requests-per-second throttle, refunds and shifts cost
one API call per record, and `backfill-history` imports three clients at a time — raise its
`s/buffer` if you need it faster. Neither the database nor the transactor is the limit; reads
measured at 32 µs.
It must run **after** the migration. Run before, and it imports against legacy keys and leaves more
to migrate.
---
## Step 6 — Resume the Square importer
Normally: once step 4's two checks read clean and step 5's backfill has finished. The maintenance
window ends here.
**If the migration did not finish**, you do not have to wait for it. Resume imports once the
`::month-complete` log covers the window your importer reads — the last 75 days for payouts and
cash-drawer shifts, and whatever range the order import is configured for. Then re-run
`migrate-all!` afterwards to walk the remaining older months; it will skip everything already done.
Run the step 4 checks again once it does finish.
The first cycle after resuming is the one to watch. Compare these against the same counts taken
immediately before the deploy — growth should be ordinary daily volume:
```clojure
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))
(count (d/datoms (d/db conn) :aevt :charge/external-id))
```
A near-doubling of any of them means records are being created rather than matched — **stop and
roll back the deploy.** Charges are included deliberately: they are the one that doubles a client's
takings rather than merely duplicating a row.
---
## Step 7 — Recompute summaries, flags still off
```clojure
(require '[auto-ap.jobs.sales-summaries :as ss])
(ss/refresh-sales-summaries 90)
```
This is the pass that banks the deduplication. **Capture the result before going further** — you
will need it as the baseline for step 8, and it cannot be reconstructed afterwards:
```clojure
(require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath
(def before (cmp/summaries-in (d/db conn) start end))
(spit "before.edn" (pr-str before))
```
> **Do not use `d/as-of` to compare summary amounts.** `:ledger-mapped/amount`, `ledger-side` and
> `account` are `:db/noHistory`, so past values are discarded. A summary that has since been
> recomputed reads back through `as-of` with its amounts *absent*, which looks like a legitimate
> balanced day. Capture live, before and after, and diff the captures.
---
## Step 8 — Turn the flag on, a few restaurants at a time
Needs accounting sign-off first: `summary-service-charges` posts to **49000 Service Income**, chosen
so the work could be measured. It affects reporting, never whether a day balances.
```clojure
@(d/transact conn [{:db/id [:client/code "NGxx"]
:client/feature-flags ["summary-service-charges"]}])
(ss/refresh-sales-summaries 90)
```
Start with two or three restaurants, confirm, then widen.
**Verify** against the capture from step 7:
```clojure
(def after (cmp/summaries-in (d/db conn) start end))
(cmp/compare-window ...) ; both arguments live database values, never as-of
```
The two numbers that matter — both were zero across all 18,900 client-days in testing:
- `:balanced->unbalanced` must be **0**
- previously-balanced days whose lines changed must be **0**
If either is non-zero, retract the flag for the affected clients and re-run step 7. The flag is the
rollback: removing it restores today's behaviour exactly.
---
## Step 9 — Re-enable `remove-voided-orders`
Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 1.
---
## Step 10 — Remove the legacy key lookup
**Schedule this; do not leave it open-ended.** Both client records at a shared location stay active
permanently, so the legacy fallback in `square.core3/existing-id` is the one code path that can ever
reach across them. Deleting it is what turns the guarantee from conventional into structural.
Once `plan` reports `:to-migrate 0` and has stayed there through several import cycles, drop the
legacy branch of `existing-id` — and with it `owned-by-other-client?`, which exists only to make
that branch safe while it lives. After this, two clients on one location are structurally incapable
of resolving onto each other's records, and no ordering discipline is required to keep it that way.
Until it is done, the protection is the guard plus the maintenance window, both of which depend on
people doing the right thing. That is the reason not to let this drift.
---
## Step 11 — Deal with the refunds that have no sales behind them
**The most important item in this document, and the only one that is not just execution.**
16 of the 122 remaining days are a record carrying refunds on a day it recorded no sales at all,
and all 16 fall before that client's first ever order. Two clients are affected, holding **160
refunds worth $4,347.68 dated before their own first order**:
| Client | First order | Refunds before it | Value | Days out of balance |
|---|---|---:|---:|---:|
| NG4S | 2026-05-29 | 79 | $2,180.08 | 10 |
| NGPS | 2026-05-26 | 81 | $2,167.60 | 7 |
**Step 5's backfill already resolved the other seven.** Before it, nine records were in this state
holding 660 refunds worth $15,237.02 — but seven of them were shared-location twins whose refunds
only looked orphaned because their orders had never been imported. Replaying the window gave them
their orders, and the refunds stopped predating them.
NG4S and NGPS are different: neither shares a Square location, so there is no twin holding the other
half. Their sales genuinely are not in the system for the period their refunds cover. The database's
own ownership history is the evidence to check — for the twins it showed refunds changing hands
between the two records; for these two there is no second record to have taken them from.
Two ways to close it, and the business has to pick:
1. **Import the missing sales.** Correct if these records are meant to have their own books. Try
`backfill-history` for them first, with a window reaching back before their first order — that is
exactly what fixed the seven, and it is one command.
2. **Move the refunds to the record that has the sales.** Correct only if the refunds were misfiled
onto a record that should not have books of its own.
Start with (1): it is cheap, reversible in the sense that it only adds what Square reports, and it
is already proven to work on this exact symptom.
```clojure
;; per client: refunds dated before that client's own first order
(let [first-order (->> (d/q '[:find [?d ...] :in $ ?c
:where [?o :sales-order/client ?c] [?o :sales-order/date ?d]]
(d/db conn) [:client/code "NG4S"])
(reduce (fn [a b] (if (.before a b) a b))))]
(->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c
:where [?r :sales-refund/client ?c]]
(d/db conn) [:client/code "NG4S"])
(filter #(.before (:sales-refund/date %) first-order))
count))
```
**Until this is resolved those days stay out of balance, on purpose.** A summary change to close
them was written and measured — it works, closes 16 days and $1,227.65, and breaks nothing — and it
was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not
being imported. A test asserts the day stays unbalanced so nobody closes it without reading this.
---
## What this will not fix
122 client-days over ninety days, $2,379.45, of which only 32 are above ten cents.
| | Days | Variance | |
|---|---:|---:|---|
| Real trading days with genuine discrepancies | 106 | $1,151.80 | see below |
| Refunds on a record with no sales imported | 16 | $1,227.65 | step 11 — deliberately visible |
Of the 106 trading days, only **3 are on shared-location records** — $648.84 in total, and all three
are already diagnosed: NGBK and NGBR at $299.42 each on 2026-08-06, where Square recorded $6,358.99
of tender against $6,059.57 of order totals (the gap itself, not a summary fault), and NGDA at
$50.00, an auto-gratuity booked as a service charge.
The other 103 days come to **$502.96 across 190 clients** — a few dollars here and there, mostly
till rounding, plus small undiagnosed clusters on NGMV ($259.38 over 5 days) and NGEB ($199.09 over
4 days, an ezCater fee-treatment question). Those two are worth a look but are not urgent.
That 103-day, $502.96 figure has been identical in every run of this analysis — with the duplicates
deactivated, with them live, and with them backfilled. It is the floor this work reaches.
---
## Two operational findings, unrelated to the summaries
- **The production backup had not written a restore point since 2025-03-10** — about seventeen
months — although data files were still uploading daily. Worth an alert on restore-point age.
- **The database server is sized for a much smaller dataset**: a 2 GB cache against 27 GB of data.
Worth checking what production is set to.

21
src/clj/auto_ap/datomic.clj
View File

@@ -875,13 +875,22 @@
(defn all-schema [] (defn all-schema []
(edn/read-string (slurp (io/resource "schema.edn")))) (edn/read-string (slurp (io/resource "schema.edn"))))
(defn transact-schema [conn] (defn transact-schema
@(dc/transact conn "Installs the schema in two passes: every plain attribute first, then every composite tuple.
(edn/read-string (slurp (io/resource "schema.edn"))))
;; this is temporary for any new stuff that needs to be asserted for cloud migration. A tuple can only be created once the attributes it composes already exist, and the pieces are
@(dc/transact conn spread across both files — `:journal-entry-line/running-balance-tuple` lives in schema.edn
(edn/read-string (slurp (io/resource "cloud-migration-schema.edn"))))) while one of its members, `:journal-entry-line/running-balance`, lives in
cloud-migration-schema.edn. Transacting the files in order therefore cannot install that tuple
against an empty database. Long-lived databases never hit it because those attributes went in
years apart."
[conn]
(let [schema (concat (edn/read-string (slurp (io/resource "schema.edn")))
;; this is temporary for any new stuff that needs to be asserted for cloud migration.
(edn/read-string (slurp (io/resource "cloud-migration-schema.edn"))))
{tuples true plain false} (group-by #(contains? % :db/tupleAttrs) schema)]
(when (seq plain) @(dc/transact conn plain))
(when (seq tuples) @(dc/transact conn tuples))))
(defn backoff [n] (defn backoff [n]
(let [base-timeout 500 (let [base-timeout 500

7
src/clj/auto_ap/datomic/sales_summaries.clj
View File

@@ -36,6 +36,13 @@
(defn balanced? [items] (defn balanced? [items]
(dollars= (total-debits items) (total-credits items))) (dollars= (total-debits items) (total-credits items)))
(defn imbalance
"Signed debits minus credits. `balanced?` answers yes or no; this says by how much and in
which direction, so a day that does not balance can be logged and queried rather than only
rendered red. Positive means the tender side exceeds what revenue accounts for."
[items]
(- (total-debits items) (total-credits items)))
(defn accepted? (defn accepted?
"True once a summary is finished: every line is mapped to an account and debits equal "True once a summary is finished: every line is mapped to an account and debits equal
credits. This is the same condition the sales summaries grid renders as \"Balanced\", and credits. This is the same condition the sales summaries grid renders as \"Balanced\", and

389
src/clj/auto_ap/jobs/rekey_square_external_ids.clj Normal file
View File

@@ -0,0 +1,389 @@
(ns auto-ap.jobs.rekey-square-external-ids
"One-shot migration re-keying Square entities to client-scoped external ids.
Refunds, charges, Square payouts (expected deposits) and cash drawer shifts all carry keys with
no client scoping, so two clients configured on the same Square location share a single entity:
its owner flips every time either client imports. Sales orders and ezCater orders already scope
their keys by client and location; this brings the rest in line.
Measured on a restored production backup, ownership had actually changed on 3,387 refunds,
4,069 expected deposits and 2,628 cash drawer shifts, across 19 distinct client pairs — nine of
which no longer share a location in the current configuration and so are invisible to any
point-in-time check.
Run AFTER the importer knows how to resolve both key schemes (`square.core3/existing-id`).
Running it first would be harmless, but the importer would then re-create legacy-keyed
entities on its next pass.
The migration is idempotent: an entity already carrying its scoped key is skipped, so it can
be re-run over a partially migrated database."
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.logging :as alog]
[datomic.api :as dc]
[iol-ion.query]))
(def refund-prefix "square/refund/")
(def charge-prefix "square/charge/")
(def deposit-prefix "square/payout/")
(def shift-prefix "square/cash-drawer-shift/")
(defn- scope-of
"`[client-code location]` for an entity, or nil when it cannot be determined.
Charges are the awkward case: about an eighth of them carry neither `:charge/client` nor
`:charge/location`. Those are stubs minted by the payout path, which asserts an external id
alone and lets unique-identity upsert bring a bare entity into being, plus older tender
records that predate the client attribute. None are orphaned, so the scope is recovered from
whatever references them — the sales order first, then the expected deposit."
[db attr e]
(let [ent (dc/entity db e)
pair (fn [code loc] (when (and code loc) [code loc]))]
(or (case attr
:sales-refund/external-id (pair (:client/code (:sales-refund/client ent))
(:sales-refund/location ent))
:charge/external-id (pair (:client/code (:charge/client ent))
(:charge/location ent))
:expected-deposit/external-id (pair (:client/code (:expected-deposit/client ent))
(:expected-deposit/location ent))
:cash-drawer-shift/external-id (pair (:client/code (:cash-drawer-shift/client ent))
(:cash-drawer-shift/location ent)))
(when-let [o (:e (first (dc/datoms db :vaet e :sales-order/charges)))]
(let [oe (dc/entity db o)]
(pair (:client/code (:sales-order/client oe)) (:sales-order/location oe))))
(when-let [d (:e (first (dc/datoms db :vaet e :expected-deposit/charges)))]
(let [de (dc/entity db d)]
(pair (:client/code (:expected-deposit/client de)) (:expected-deposit/location de)))))))
(defn planned-key
"`[eid new-key]` for an entity that still needs re-keying, or nil when it is already scoped or
cannot be scoped at all.
Detection compares against the key this entity *should* have rather than pattern-matching the
id, because Square ids may themselves contain dashes and no pattern separates the two schemes
reliably. That also makes the migration idempotent."
[db attr prefix datom]
(let [old (:v datom)]
(when-let [[code loc] (scope-of db attr (:e datom))]
(let [scoped-prefix (str prefix code "-" loc "-")]
(when-not (.startsWith ^String old scoped-prefix)
[(:e datom) (str scoped-prefix (subs old (count prefix)))])))))
(defn plan
"Everything the migration would change, plus what it cannot touch. Read-only — run this and
check `:collisions` is empty before transacting anything."
[db attr prefix]
(let [acc (reduce (fn [acc d]
(let [acc (update acc :total inc)]
(if-let [[e new-key] (planned-key db attr prefix d)]
(-> acc
(update :to-migrate inc)
(update :new-keys conj! [e new-key]))
(if (scope-of db attr (:e d))
(update acc :already-scoped inc)
(update acc :unscopable inc)))))
{:total 0 :to-migrate 0 :already-scoped 0 :unscopable 0 :new-keys (transient [])}
(dc/datoms db :aevt attr))]
(update acc :new-keys persistent!)))
(defn collisions
"Any two entities that would land on the same new key. Must be empty: a collision would merge
two entities into one and lose whichever lost."
[new-keys]
(->> new-keys
(group-by second)
(keep (fn [[k es]] (when (> (count es) 1) [k (mapv first es)])))
vec))
(defn migrate!
"Asserts the new external id on each planned entity. The attribute is cardinality one, so the
legacy value is retracted by the same assertion and the entity keeps its identity — nothing is
created and nothing is deleted.
Returns the number of entities re-keyed."
[attr new-keys batch-size]
(let [total (count new-keys)]
(alog/info ::migrating :attr attr :count total)
(doseq [[i batch] (map-indexed vector (partition-all batch-size new-keys))]
@(dc/transact conn (for [[e new-key] batch] {:db/id e attr new-key}))
(when (zero? (mod i 20))
(alog/info ::migrated :attr attr :done (* i batch-size) :of total)))
total))
(def charge-copy-attrs
"Everything a charge carries in its own right. `:charge/client+date` is a tuple Datomic
maintains, and `:charge/external-id` is set separately, so neither is copied."
[:charge/type-name :charge/total :charge/tip :charge/tax :charge/date
:charge/processor :charge/note :charge/reference-link])
(defn- raw-square-id
"The Square id inside a charge's external id, with any client scoping removed.
Stripping only the `square/charge/` prefix is not enough. Once a charge has been scoped to one
client, a second order processing the same charge would read `NGCC-CC-<id>` as the id and scope
it again, producing `square/charge/NGCD-CD-NGCC-CC-<id>`. The importer then computes the
correct single-scoped key, fails to find it, and creates a second charge — silently doubling
the tender.
Client codes may themselves contain dashes, so the scope cannot be recognised by pattern. It is
recovered from the entity instead: whoever the charge currently belongs to is exactly whose
scope its key carries."
[db charge old]
(let [ent (dc/entity db charge)
code (:client/code (:charge/client ent))
loc (:charge/location ent)
owner-prefix (when (and code loc) (str charge-prefix code "-" loc "-"))]
(cond
(and owner-prefix (.startsWith ^String old ^String owner-prefix)) (subs old (count owner-prefix))
(.startsWith ^String old charge-prefix) (subs old (count charge-prefix))
:else old)))
(defn- charge-plan-for-order
"What one order needs doing to its charges.
`:keep` — no other order has claimed this charge yet, so this order takes it and it is renamed
in place. `:clone` — another order already owns it, so this order needs its own copy.
Splitting is per client, not per order. Where two orders of the SAME client and location refer
to one payment — Square splitting a tender across orders, or an amendment — they are left
sharing it deliberately. Both would compute the same name, so there is no second name to give
a copy, and more importantly a copy would double that client's takings for the day. The
component cascade still applies to those, which is why the retraction guard on
`remove-voided-orders` is needed regardless of this migration.
Whether a charge is claimed is read from the charge itself: an unclaimed one still carries the
bare `square/charge/<id>` form. Deciding it that way rather than by remembering
every charge seen so far is what lets this run across all 16 million of them — the alternative
needs a map of the entire table in memory. `batch-seen` covers only the orders inside one
transaction, where the database snapshot cannot yet show a claim made moments earlier."
[db order-eid batch-seen]
(let [oe (dc/entity db order-eid)
code (:client/code (:sales-order/client oe))
loc (:sales-order/location oe)]
(when (and code loc)
(for [d (dc/datoms db :eavt order-eid :sales-order/charges)
:let [charge (:v d)
old (:v (first (dc/datoms db :eavt charge :charge/external-id)))]
:when old
:let [raw (raw-square-id db charge old)
new-key (str charge-prefix code "-" loc "-" raw)
claimed (get @batch-seen charge)
unclaimed? (and (= old (str charge-prefix raw)) (nil? claimed))]
;; nothing to do when the charge already answers to this order's name, or when
;; another order in this same batch has just claimed it under that very name —
;; that is the same-client case, which stays shared
:when (and (not= old new-key) (not= claimed new-key))]
{:order order-eid :charge charge :new-key new-key :raw raw
:client (:db/id (:sales-order/client oe)) :location loc
:action (if unclaimed? :keep :clone)}))))
(defn split-and-rekey-charges!
"Gives every order its own charge entity, keyed by that order's client and location.
Where two clients were configured on one Square location, both clients' orders resolved to a
single charge, because charge keys carried no client. Re-keying alone does not undo that — it
hands the one entity to whichever client is looked at first and leaves the other order pointing
at a charge it does not own. Since `:sales-order/charges` is a component attribute, that is not
merely untidy: retracting either order would delete a charge the other one still needs.
So a shared charge is cloned. The first order to claim it keeps it, re-keyed to that order's
scope; every other order gets a copy carrying the same amounts, scoped to its own client, and
has its reference repointed. Afterwards no charge has more than one parent order and the
component relationship means what it says.
`orders` is the collection of order entity ids to process — typically every order belonging to
the clients that share, or have ever shared, a Square location."
[orders batch-size]
(let [cloned (atom 0)
rekeyed (atom 0)]
(doseq [[i batch] (map-indexed vector (partition-all batch-size orders))]
(when (zero? (mod i 200))
;; the whole-database run walks 19M orders; without a trail an interrupted run leaves
;; no way to tell how far it got short of querying the data by hand
(alog/info ::splitting :orders-done (* i batch-size) :rekeyed @rekeyed :cloned @cloned))
(let [db (dc/db conn)
batch-seen (atom {})
tx (doall
(for [o batch
plan (charge-plan-for-order db o batch-seen)
:let [{:keys [charge new-key action client location]} plan]
tx-item (if (= :keep action)
(do (swap! batch-seen assoc charge new-key)
(swap! rekeyed inc)
;; record who claimed it: the owner is how a later order
;; recovers the Square id from an already-scoped key
[{:db/id charge
:charge/external-id new-key
:charge/client client
:charge/location location}])
(let [ent (dc/entity db charge)
copy (reduce (fn [m a] (if-some [v (get ent a)]
(assoc m a (if (map? v) (:db/id v) v))
m))
{} charge-copy-attrs)]
(swap! cloned inc)
[(assoc copy
:db/id new-key
:charge/external-id new-key
:charge/client client
:charge/location location)
[:db/retract o :sales-order/charges charge]
{:db/id o :sales-order/charges new-key}]))]
tx-item))]
(when (seq tx) @(dc/transact conn tx))))
(alog/info ::split-charges :rekeyed @rekeyed :cloned @cloned)
{:rekeyed @rekeyed :cloned @cloned}))
(defn charges-with-multiple-parents
"The §3.3 gate. Must read zero once the split has run: while any charge has two parent orders,
retracting either order deletes the other one's payment."
[db orders]
(->> orders
(mapcat (fn [o] (map :v (dc/datoms db :eavt o :sales-order/charges))))
distinct
(filter (fn [c] (> (reduce (fn [n _] (inc n)) 0 (dc/datoms db :vaet c :sales-order/charges)) 1)))
count))
(def scoped-attrs
"Every Square-imported entity whose key must carry its client, with the prefix and where to
read the owner from."
[{:attr :sales-refund/external-id :prefix refund-prefix
:client :sales-refund/client :location :sales-refund/location}
{:attr :charge/external-id :prefix charge-prefix
:client :charge/client :location :charge/location}
{:attr :expected-deposit/external-id :prefix deposit-prefix
:client :expected-deposit/client :location :expected-deposit/location}
{:attr :cash-drawer-shift/external-id :prefix shift-prefix
:client :cash-drawer-shift/client :location :cash-drawer-shift/location}])
(defn unscoped-report
"Counts, per entity type, how many keys are already client-scoped, how many still carry the
legacy unscoped form, and how many have no owner to scope by.
The importer tolerates both key schemes on purpose, so that the change can be deployed before
the migration finishes — but that tolerance is a transition, not a resting place. While
`:legacy` is above zero the database is in a mixed state and a stray unscoped record can still
be adopted by whichever client imports it first. **`:legacy` reaching zero on every attribute is
the done-signal**, and it is what licenses removing the fallback lookup in
`square.core3/existing-id`.
`:no-owner` is NOT part of that signal and never reaches zero for charges. It counts entities
whose own `:charge/client`/`:charge/location` are absent — around an eighth of charges, the
payout stubs `migrate!` only ever gives an external id — so this report structurally cannot
verify them even when their keys are perfectly scoped. It classifies by attribute; `plan`
resolves ownership through whatever references the entity. Ask `plan` for the authoritative
answer: `:to-migrate 0` with `:unscopable 0` means there is nothing left to do."
[db]
(into {}
(for [{:keys [attr prefix client location]} scoped-attrs]
[attr (reduce (fn [acc d]
(let [e (dc/entity db (:e d))
code (:client/code (client e))
loc (location e)
scoped (when (and code loc) (str prefix code "-" loc "-"))]
(cond
(and scoped (.startsWith ^String (:v d) ^String scoped)) (update acc :scoped inc)
(nil? scoped) (update acc :no-owner inc)
:else (update acc :legacy inc))))
{:scoped 0 :legacy 0 :no-owner 0}
(dc/datoms db :aevt attr))])))
(defn all-order-ids
"Every sales order in the database, streamed in `:aevt` order — which is ascending entity id,
so OLDEST first. Fine for counting; wrong for anything that takes a prefix. `(take n ...)` of
this returns the oldest n orders, not a sample: on the production copy the first 400,000 are
all from 2019 to 2021. Use `order-months-newest-first` to walk the data in migration order."
[db]
(map :e (dc/datoms db :aevt :sales-order/external-id)))
(def ^:private earliest-orders
"How far back the month walk goes. Comfortably before the oldest order in the database
(2019-12-31 on the production copy); months with no orders cost one index seek per client."
#inst "2015-01-01T00:00:00.000-00:00")
(defn- ->date [^java.time.LocalDate d]
(java.util.Date/from (.toInstant (.atStartOfDay d (java.time.ZoneId/systemDefault)))))
(defn order-months-newest-first
"`[start end]` month windows from now back to `earliest`, newest month first.
The migration walks months in this order on purpose. It is the difference between an
interrupted run leaving the data safe to import against and leaving it dangerous: the importer
works on recent data, so having the newest months fully scoped is what lets imports resume
while the older tail is still unmigrated. Walking oldest-first would spend hours on 2019 before
touching anything this month's import will read.
Windows tile without gaps — each month's end is the day before the next month's start — and
because the migration is idempotent an order landing in two windows is a no-op the second
time, so boundary precision is not safety-critical."
([] (order-months-newest-first earliest-orders))
([^java.util.Date earliest]
(let [zone (java.time.ZoneId/systemDefault)
floor (java.time.YearMonth/from (.toLocalDate (.atZone (.toInstant earliest) zone)))]
(->> (iterate (fn [^java.time.YearMonth m] (.minusMonths m 1)) (java.time.YearMonth/now zone))
(take-while (fn [^java.time.YearMonth m] (not (.isBefore m floor))))
(map (fn [^java.time.YearMonth m]
[(->date (.atDay m 1)) (->date (.atEndOfMonth m))]))))))
(defn orders-in-window
"Sales order ids for every client between `start` and `end` inclusive, via the
`:sales-order/client+date` index."
[db clients start end]
(map first (iol-ion.query/scan-sales-orders db clients start end)))
(defn- all-client-ids [db]
(map first (dc/q '[:find ?c :where [?c :client/code _]] db)))
(defn migrate-all!
"The complete migration, over the whole database rather than a chosen subset.
Splitting is driven from orders, because a payment's rightful owner is whichever order refers
to it — so every order has to be walked, not merely the clients that share a location today.
Nine client pairs contended in the past and no longer share one; their records are still mixed,
and a migration scoped to the current configuration would miss every one of them.
**Ordered so that an interrupted run is recoverable.** Refunds, payouts and cash-drawer shifts
go first: together they are a quarter of a million records and take seconds, so finishing them
up front means an interruption cannot leave them half done. The long part — walking every order
to split shared charges — then runs a month at a time from the current month backwards, logging
each month as it completes. Stop it after any month and the data from that month forward is
fully scoped, which is the part the importer reads, so imports can resume against it while the
older tail waits. Re-running picks up where it left off because each month's work is idempotent.
Returns the split counts and the completeness report, which should read zero legacy across the
board when this finishes."
[batch-size]
(doseq [{:keys [attr prefix]} scoped-attrs
:when (not= attr :charge/external-id)]
(let [p (plan (dc/db conn) attr prefix)]
(when-let [c (seq (collisions (:new-keys p)))]
(throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
(migrate! attr (:new-keys p) batch-size)))
(let [clients (all-client-ids (dc/db conn))
split (reduce (fn [acc [start end]]
(let [ids (orders-in-window (dc/db conn) clients start end)
r (split-and-rekey-charges! ids batch-size)]
(alog/info ::month-complete
:month (subs (str (.toInstant ^java.util.Date start)) 0 7)
:rekeyed (:rekeyed r) :cloned (:cloned r))
(merge-with + acc r)))
{:rekeyed 0 :cloned 0}
(order-months-newest-first))]
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them.
;; Collision-checked like the others: this is the largest attribute in the database, so it is
;; the last one that should discover a clash as a mid-run exception.
(let [p (plan (dc/db conn) :charge/external-id charge-prefix)]
(when-let [c (seq (collisions (:new-keys p)))]
(throw (ex-info "two entities would take the same key"
{:attr :charge/external-id :collisions (count c)})))
(when (seq (:new-keys p)) (migrate! :charge/external-id (:new-keys p) batch-size)))
{:split split :completeness (unscoped-report (dc/db conn))}))
(defn counts
"Entity totals, for the before/after assertion that is this migration's real safety net: if
either number moves, the re-key created duplicates instead of updating in place."
[db]
{:refunds (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :sales-refund/external-id))
:charges (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :charge/external-id))
:deposits (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :expected-deposit/external-id))
:shifts (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :cash-drawer-shift/external-id))})

177
src/clj/auto_ap/jobs/sales_summaries.clj
View File

@@ -62,15 +62,24 @@
{:ledger-mapped/ledger-side [:db/ident]} {:ledger-mapped/ledger-side [:db/ident]}
{:ledger-mapped/account [:db/id]}]) {:ledger-mapped/account [:db/id]}])
(defn dirty-sales-summaries [c] (defn dirty-sales-summaries
"The client's dirty summaries, with enough of each item to evaluate and re-transact it.
`index-pull` returns a lazy seq running from `:start` to the END of the index, so this must
stop at the client boundary rather than filter: `:sales-summary/client+dirty` sorts by client
first, so every later client's summaries sit beyond this client's and filtering would walk all
of them — for every client — pulling their items on the way. That is quadratic in the number of
summaries, and it showed up as a full refresh degrading from ~180 client-days a minute to ~3 as
the summary count grew."
[c]
(let [client-id (dc/entid (dc/db conn) c)] (let [client-id (dc/entid (dc/db conn) c)]
(->> (dc/index-pull (dc/db conn) (->> (dc/index-pull (dc/db conn)
{:index :avet {:index :avet
:selector (conj '[:sales-summary/date :sales-summary/client :db/id] :selector (conj '[:sales-summary/date :sales-summary/client :db/id]
{:sales-summary/items item-read}) {:sales-summary/items item-read})
:start [:sales-summary/client+dirty [client-id true]]}) :start [:sales-summary/client+dirty [client-id true]]})
(filter (fn [sales-summary] (take-while (fn [sales-summary]
(= client-id (:db/id (:sales-summary/client sales-summary)))))))) (= client-id (:db/id (:sales-summary/client sales-summary))))))))
(def default-days (def default-days
"How far back the scheduled refresh looks for summaries that still need recomputing." "How far back the scheduled refresh looks for summaries that still need recomputing."
@@ -147,8 +156,18 @@
date)) date))
0.0))) 0.0)))
(def service-charges-account
"Where a credited Square service charge lands. 49000 is the existing \"Service Income\"
revenue account, which is the closest fit for auto-gratuity and catering fees.
NEEDS ACCOUNTING SIGN-OFF before `service-charges-flag` is enabled for any client: the wrong
account misstates revenue, and a category with no account at all keeps a day from ever
reaching accepted, since `accepted?` requires every line to be mapped."
49000)
(def name->number (def name->number
{"gyros and pitas" 40111 {"gyros and pitas" 40111
"service charges" service-charges-account
"returns" 41300 "returns" 41300
"card payments" 75460 "card payments" 75460
"cash payments" 75452 "cash payments" 75452
@@ -292,21 +311,48 @@
[[c] date date])) [[c] date date]))
0.0)}) 0.0)})
(defn- get-tip [c date] (defn- tendered-tip
"Tips read off the tenders, which is where a tip actually settles."
[c date]
(or (ffirst (dc/q '[:find (sum ?tip)
:with ?c
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/charges ?c]
[?c :charge/tip ?tip]]
(dc/db conn)
[[c] date date]))
0.0))
(defn- untendered-tip
"Tips on orders that carry no tender at all. A return-only order reverses its tip on
`:sales-order/tip` but has no charge to join through, so the reversal is invisible to
`tendered-tip` and the day ends up crediting a tip that was handed back."
[c date]
(or (ffirst (dc/q '[:find (sum ?tip)
:with ?e
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/tip ?tip]
(not [?e :sales-order/charges])]
(dc/db conn)
[[c] date date]))
0.0))
(defn- get-tip
"Tendered tips plus the tips on untendered orders. Additive rather than substitutive on
purpose: where an order does have a tender, the tender is the correct source, and real
orders exist whose tender carries a tip their `:sales-order/tip` does not — auto-gratuity
booked as a service charge, and wallet tips absent from the net amounts. Reading the order
instead of the tender would drop those."
[c date]
{:ledger-mapped/ledger-side :ledger-side/credit {:ledger-mapped/ledger-side :ledger-side/credit
:sales-summary-item/sort-order 2 :sales-summary-item/sort-order 2
:db/id (str (java.util.UUID/randomUUID)) :db/id (str (java.util.UUID/randomUUID))
:sales-summary-item/category "Tip" :sales-summary-item/category "Tip"
:ledger-mapped/amount (or (ffirst (dc/q '[:find (sum ?tip) :ledger-mapped/amount (+ (tendered-tip c date)
:with ?c (untendered-tip c date))})
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/charges ?c]
[?c :charge/tip ?tip]]
(dc/db conn)
[[c] date date]))
0.0)})
(defn- get-sales [c date] (defn- get-sales [c date]
(let [sales (->> (dc/q '[:find ?category (sum ?total) (sum ?tax) (sum ?discount) (let [sales (->> (dc/q '[:find ?category (sum ?total) (sum ?tax) (sum ?discount)
@@ -332,6 +378,14 @@
:ledger-mapped/amount (- (+ total discount) tax) :ledger-mapped/amount (- (+ total discount) tax)
#_#_:ledger-mapped/account nil}))) #_#_:ledger-mapped/account nil})))
;; A day carrying refunds and no sales at all is left out of balance on purpose. It is tempting
;; to close it by booking a return against the day's refunds — the arithmetic works, and no
;; trading day could be affected. Do not. Those days are overwhelmingly not "a refund settled
;; while the restaurant was shut": they are days whose *orders were never imported*, on client
;; records that took ownership of another record's refunds through the unscoped keys this branch
;; fixes. Balancing them would convert the only signal that a client's sales are missing into
;; silence. See `docs/2026-08-15-sales-summary-rollout-plan.md`.
(defn- get-returns [c date] (defn- get-returns [c date]
(when-let [amount (ffirst (dc/q '[:find (sum ?r) (when-let [amount (ffirst (dc/q '[:find (sum ?r)
:with ?e :with ?e
@@ -349,12 +403,82 @@
:ledger-mapped/amount amount :ledger-mapped/amount amount
:ledger-mapped/ledger-side :ledger-side/debit})) :ledger-mapped/ledger-side :ledger-side/debit}))
(defn sales-summaries-v2 [] (def service-charges-flag
(doseq [[c client-code] (dc/q '[:find ?c ?client-code "Per-client rollout lever for crediting Square service charges, in the same style as
:in $ `new-square` and `import-custom-amount`. Absent, the summary behaves exactly as it does
:where [?c :client/code ?client-code]] today."
(dc/db conn)) "summary-service-charges")
{:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)]
(defn- service-charges-enabled? [c]
(contains? (set (:client/feature-flags (dc/pull (dc/db conn) '[:client/feature-flags] c)))
service-charges-flag))
(defn service-charge-total
"Square service charges for the day, both signs.
A service charge is collected inside the card tender but nothing credits it, so every order
carrying one leaves the day short by exactly that amount. Both signs matter: a returned
catering fee arrives as a negative service charge and is subtracted back out of
`:sales-order/returns`, so dropping negatives would lose the reversal.
The vendor gate is load-bearing — ezCater service charges are commission deducted from the
restaurant rather than collected from the diner, and crediting those would make things worse.
It matches on `:sales-order/vendor` where that is set and falls back to the external id
prefix where it is not, because whole eras of Square orders carry no vendor attribute at all
and a gate on vendor alone silently credits nothing.
Kept separate from the rollout flag so the arithmetic can be measured on its own."
[c date]
(ffirst (dc/q '[:find (sum ?service-charge)
:with ?e
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/service-charge ?service-charge]
(or-join [?e]
[?e :sales-order/vendor :vendor/ccp-square]
(and (not [?e :sales-order/vendor])
[?e :sales-order/external-id ?external-id]
[(clojure.string/starts-with? ?external-id "square/order/")]))]
(dc/db conn)
[[c] date date])))
(defn- get-service-charges
"The day's service charges as a summary item, for clients opted in to the rollout."
[c date]
(when (service-charges-enabled? c)
(when-let [amount (service-charge-total c date)]
(when-not (zero? amount)
{:db/id (str (java.util.UUID/randomUUID))
:sales-summary-item/category "Service Charges"
:sales-summary-item/sort-order 2
:ledger-mapped/amount amount
:ledger-mapped/ledger-side :ledger-side/credit}))))
(def ^:private suspect-categories
"The terms a balancing investigation keeps returning to. Logged beside the imbalance so a
day's shape can be read out of the logs without re-running the job."
["Tip" "Service Charges" "Returns" "Card Refunds" "Cash Refunds" "Food App Refunds"])
(defn- suspect-totals
"Amounts for `suspect-categories` present on this day, omitting the ones that are zero."
[items]
(into {}
(for [category suspect-categories
:let [amount (->> items
(filter #(= category (:sales-summary-item/category %)))
(map #(:ledger-mapped/amount % 0.0))
(reduce + 0.0))]
:when (not (zero? amount))]
[category amount])))
(defn refresh-client!
"Recomputes every dirty summary for one client.
Split out of the driver loop so a client's work stands on its own: it can be run for a single
client, and a backfill over the whole history can spread clients across threads instead of
grinding through the largest ones one day at a time."
[c client-code]
(doseq [{:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)]
(mu/with-context {:client-code client-code (mu/with-context {:client-code client-code
:date date} :date date}
(alog/info ::updating) (alog/info ::updating)
@@ -370,6 +494,7 @@
(cons (get-fees c date)) (cons (get-fees c date))
(cons (get-tax c date)) (cons (get-tax c date))
(cons (get-tip c date)) (cons (get-tip c date))
(cons (get-service-charges c date))
(cons (get-returns c date)) (cons (get-returns c date))
(filter identity) (filter identity)
(map (fn [z] (map (fn [z]
@@ -385,10 +510,22 @@
(if (seq (:sales-summary/items result)) (if (seq (:sales-summary/items result))
(do (do
(alog/info ::upserting-summaries (alog/info ::upserting-summaries
:category-count (count (:sales-summary/items result))) :category-count (count (:sales-summary/items result))
:imbalance (d-ss/imbalance all-items)
:balanced? (d-ss/balanced? all-items)
:suspect-totals (suspect-totals all-items))
@(dc/transact conn [[:upsert-sales-summary result]])) @(dc/transact conn [[:upsert-sales-summary result]]))
@(dc/transact conn [{:db/id id :sales-summary/dirty false}])))))) @(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
(defn sales-summaries-v2
"Recomputes every dirty summary, client by client."
[]
(doseq [[c client-code] (dc/q '[:find ?c ?client-code
:in $
:where [?c :client/code ?client-code]]
(dc/db conn))]
(refresh-client! c client-code)))
(defn reset-summaries [] (defn reset-summaries []
@(dc/transact conn (->> (dc/q '[:find ?sos @(dc/transact conn (->> (dc/q '[:find ?sos
:in $ :in $

292
src/clj/auto_ap/square/core3.clj
View File

@@ -27,11 +27,9 @@
"Authorization" (str "Bearer " (:client/square-auth-token client)) "Authorization" (str "Bearer " (:client/square-auth-token client))
"Content-Type" "application/json"})) "Content-Type" "application/json"}))
(defn ->square-date [d] (defn ->square-date [d]
(f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d)) (f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d))
(def manifold-api-stream (def manifold-api-stream
(let [stream (s/stream 100)] (let [stream (s/stream 100)]
(->> stream (->> stream
@@ -42,10 +40,10 @@
(de/loop [attempt 0] (de/loop [attempt 0]
(-> (de/chain (de/future-with (ex/execute-pool) (-> (de/chain (de/future-with (ex/execute-pool)
#_(log/info ::request-started #_(log/info ::request-started
:url (:url request) :url (:url request)
:attempt attempt :attempt attempt
:source "Square 3" :source "Square 3"
:background-job "Square 3") :background-job "Square 3")
(try (try
(client/request (assoc request (client/request (assoc request
:socket-timeout 10000 :socket-timeout 10000
@@ -104,7 +102,6 @@
:exception error)) :exception error))
[])))) []))))
(def item-cache (atom {})) (def item-cache (atom {}))
(defn fetch-catalog [client i v] (defn fetch-catalog [client i v]
@@ -124,13 +121,11 @@
#(do (swap! item-cache assoc i %) #(do (swap! item-cache assoc i %)
%)))) %))))
(defn fetch-catalog-cache [client i version] (defn fetch-catalog-cache [client i version]
(if (get @item-cache i) (if (get @item-cache i)
(de/success-deferred (get @item-cache i)) (de/success-deferred (get @item-cache i))
(fetch-catalog client i version))) (fetch-catalog client i version)))
(defn item->category-name-impl [client item version] (defn item->category-name-impl [client item version]
(capture-context->lc (capture-context->lc
(cond (:item_id (:item_variation_data item)) (cond (:item_id (:item_variation_data item))
@@ -161,7 +156,6 @@
:item item) :item item)
"Uncategorized")))) "Uncategorized"))))
(defn item-id->category-name [client i version] (defn item-id->category-name [client i version]
(capture-context->lc (capture-context->lc
(-> [client i] (-> [client i]
@@ -226,7 +220,6 @@
(concat (:orders result) continued-results)))) (concat (:orders result) continued-results))))
(:orders result))))))) (:orders result)))))))
(defn search (defn search
([client location start end] ([client location start end]
(capture-context->lc (capture-context->lc
@@ -250,11 +243,9 @@
(concat (:orders result) continued-results)))) (concat (:orders result) continued-results))))
(:orders result)))))))) (:orders result))))))))
(defn amount->money [amt] (defn amount->money [amt]
(* 0.01 (or (:amount amt) 0.0))) (* 0.01 (or (:amount amt) 0.0)))
;; to get totals: ;; to get totals:
(comment (comment
(reduce (reduce
@@ -269,6 +260,66 @@
0.0 0.0
[])) []))
(defn scoped-key
"Client-scoped external id, in the shape sales order keys already use.
Without the client and location in the key, two clients configured on the same Square location
collide on a single entity: a refund changes owner every time either client imports, and one
charge ends up shared between both clients' orders."
[prefix client location id]
(str prefix (:client/code client) "-" (:square-location/client-location location) "-" id))
(def ^:private owner-attr
"Where each Square-imported entity records the client it belongs to."
{:charge/external-id :charge/client
:sales-refund/external-id :sales-refund/client
:expected-deposit/external-id :expected-deposit/client
:cash-drawer-shift/external-id :cash-drawer-shift/client})
(defn- owned-by-other-client?
"Whether `e` already belongs to a client other than `client-eid`.
Reads the entity's own owner attribute, and for a charge falls back to the client of whichever
sales order refers to it — charges predating `:charge/client` still have orders, and those are
exactly the ones that can be taken by the wrong client."
[db attr e client-eid]
(let [ent (dc/entity db e)
owner (or (:db/id ((owner-attr attr) ent))
(when (= attr :charge/external-id)
(some->> (first (dc/datoms db :vaet e :sales-order/charges))
:e
(dc/entity db)
:sales-order/client
:db/id)))]
(and owner (not= owner client-eid))))
(defn existing-id
"Entity id of the refund or charge this id already refers to, trying the client-scoped key
first and the legacy unscoped key second.
This is what makes re-keying safe. These external ids are `:db.unique/identity`, so the import
relies on upsert-by-identity; changing the key format on its own would match nothing and
Datomic would create a SECOND entity for every refund and charge, orphaning the original under
its legacy key. Pinning the result as `:db/id` makes the write land on the existing entity
whichever scheme it currently carries.
The legacy branch will not take a record that already belongs to a different client. Without
that check, two clients on one Square location double money during the window between deploying
and finishing the migration: client A's payout import resolves B's charge by its bare key and
renames it into A's scope, B's next order import then matches neither scheme and mints a second
charge, and because `:sales-order/charges` is cardinality-many nothing retracts the first — so
B's order carries two charges for one payment. Declining is also the right answer on its merits:
the write then lands on this client's own copy, which is what the scoped keys exist to create.
Once the migration has run there are no legacy keys left for this branch to find, and both it
and the guard can be deleted together."
[db attr prefix client location id]
(when id
(or (dc/entid db [attr (scoped-key prefix client location id)])
(when-let [legacy (dc/entid db [attr (str prefix id)])]
(when-not (owned-by-other-client? db attr legacy (:db/id client))
legacy)))))
(defn tender->charge [order client location t] (defn tender->charge [order client location t]
(remove-nils (remove-nils
#:charge #:charge
@@ -278,8 +329,9 @@
:note (:note t) :note (:note t)
:location (:square-location/client-location location) :location (:square-location/client-location location)
:reference-link (str (url/url "https://squareup.com/receipt/preview" (:id t))) :reference-link (str (url/url "https://squareup.com/receipt/preview" (:id t)))
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location (:id t))
:external-id (when (:id t) :external-id (when (:id t)
(str "square/charge/" (:id t))) (scoped-key "square/charge/" client location (:id t)))
:processor (cond :processor (cond
(#{"OTHER" "THIRD_PARTY_CARD"} (:type t)) (#{"OTHER" "THIRD_PARTY_CARD"} (:type t))
(condp = (some-> (:note t) str/lower-case) (condp = (some-> (:note t) str/lower-case)
@@ -415,7 +467,6 @@
:client client :client client
:location location))))))) :location location)))))))
(defn get-payment [client p] (defn get-payment [client p]
(de/chain (manifold-api-call (de/chain (manifold-api-call
{:url (str "https://connect.squareup.com/v2/payments/" p) {:url (str "https://connect.squareup.com/v2/payments/" p)
@@ -424,7 +475,6 @@
:body :body
:payment)) :payment))
(defn continue-payout-entry-list [c l poi cursor] (defn continue-payout-entry-list [c l poi cursor]
(capture-context->lc lc (capture-context->lc lc
(de/chain (de/chain
@@ -509,7 +559,7 @@
(try (try
(->> (for [payout payouts (->> (for [payout payouts
:let [best-sales-date (some->> (dc/q '[:find ?s4 (count ?s) :let [best-sales-date (some->> (dc/q '[:find ?s4 (count ?s)
:in $ ?payout-id :in $ [?payout-id ...]
:where :where
[?payout :expected-deposit/external-id ?payout-id] [?payout :expected-deposit/external-id ?payout-id]
[?payout :expected-deposit/charges ?c] [?payout :expected-deposit/charges ?c]
@@ -519,7 +569,8 @@
[(auto-ap.time/localize ?s2) ?s3] [(auto-ap.time/localize ?s2) ?s3]
[(clj-time.coerce/to-local-date ?s3) ?s4]] [(clj-time.coerce/to-local-date ?s3) ?s4]]
(dc/db conn) (dc/db conn)
(str "square/payout/" (:id payout))) [(scoped-key "square/payout/" client location (:id payout))
(str "square/payout/" (:id payout))])
(sort-by last) (sort-by last)
last last
first first
@@ -543,7 +594,10 @@
(:db/id client) (:db/id client)
(amount->money (:amount_money payout))))] (amount->money (:amount_money payout))))]
:when (not equivalent-already-exists?)] :when (not equivalent-already-exists?)]
#:expected-deposit {:external-id (str "square/payout/" (:id payout)) #:expected-deposit {:db/id (or (existing-id (dc/db conn) :expected-deposit/external-id
"square/payout/" client location (:id payout))
(str "square/payout/" (:id payout)))
:external-id (scoped-key "square/payout/" client location (:id payout))
:vendor :vendor/ccp-square :vendor :vendor/ccp-square
:status :expected-deposit-status/pending :status :expected-deposit-status/pending
:total (amount->money (:amount_money payout)) :total (amount->money (:amount_money payout))
@@ -561,22 +615,61 @@
(coerce/to-date))) (coerce/to-date)))
:charges (reverse (->> (:payout_entries payout) :charges (reverse (->> (:payout_entries payout)
(filter (comp :payment_id :type_charge_details)) (filter (comp :payment_id :type_charge_details))
(map (fn [p] {:charge/external-id (str "square/charge/" (:payment_id (:type_charge_details p)))}))))}) (map (fn [p]
(let [payment-id (:payment_id (:type_charge_details p))]
(remove-nils
{:charge/external-id (scoped-key "square/charge/" client location payment-id)
;; the owner attributes must travel with the key: `raw-square-id` and
;; `scope-of` both recover a charge's scope from them, and a key
;; scoped to one client while the owner says another is what makes
;; the migration write square/charge/B-LB-A-LA-<id>
:charge/client (:db/id client)
:charge/location (:square-location/client-location location)
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location payment-id)}))))))})
(filter :expected-deposit/date) (filter :expected-deposit/date)
(into [])) (into []))
(catch Throwable e (catch Throwable e
(log/error ::transform-payout-failed (log/error ::transform-payout-failed
:exception e))))))))) :exception e)))))))))
(defn refunds (defn- refund-list
([client l] "Every refund Square has for this location in `[start end]`, following the cursor to the end.
(de/chain (manifold-api-call {:url (str "https://connect.squareup.com/v2/refunds?location_id=" (:square-location/square-id l))
:method :get
:headers (client-base-headers client) The list endpoint returns one page at a time. Reading only the first page — which is what this
:as :json}) did before — silently caps a location at a hundred refunds however many it actually has, and
the cap is invisible: the response looks like a complete answer. On a shared location that is
how one client record ends up holding a few refunds against a hundred and fifty thousand orders.
`start`/`end` are optional; omitting both asks for everything, which is what the nightly job
wants and what a historical backfill of more than a page needs."
([client l start end] (refund-list client l start end nil))
([client l start end cursor]
(de/chain (manifold-api-call
{:url (str "https://connect.squareup.com/v2/refunds"
"?"
(url/map->query
(cond-> {:location_id (:square-location/square-id l)
:limit 100}
start (assoc :begin_time (->square-date start))
end (assoc :end_time (->square-date end))
cursor (assoc :cursor cursor))))
:method :get
:headers (client-base-headers client)
:as :json})
:body :body
:refunds (fn [result]
(log/info ::refunds-page
:count (count (:refunds result))
:more? (boolean (not-empty (:cursor result))))
(if (not-empty (:cursor result))
(de/chain (refund-list client l start end (:cursor result))
(fn [more] (concat (:refunds result) more)))
(:refunds result))))))
(defn refunds
([client l] (refunds client l nil nil))
([client l start end]
(de/chain (refund-list client l start end)
(fn [refunds] (fn [refunds]
(->> refunds (->> refunds
(filter (fn [r] (= "COMPLETED" (:status r)))) (filter (fn [r] (= "COMPLETED" (:status r))))
@@ -585,7 +678,8 @@
(de/chain (de/chain
(get-payment client (:payment_id r)) (get-payment client (:payment_id r))
(fn [payment] (fn [payment]
#:sales-refund {:external-id (str "square/refund/" (:id r)) #:sales-refund {:db/id (existing-id (dc/db conn) :sales-refund/external-id "square/refund/" client l (:id r))
:external-id (scoped-key "square/refund/" client l (:id r))
:vendor :vendor/ccp-square :vendor :vendor/ccp-square
:total (amount->money (:amount_money r)) :total (amount->money (:amount_money r))
:fee (transduce :fee (transduce
@@ -618,7 +712,6 @@
:count (count x)) :count (count x))
@(dc/transact-async conn x)))))))) @(dc/transact-async conn x))))))))
(defn upsert-payouts (defn upsert-payouts
([client] ([client]
(apply de/zip (apply de/zip
@@ -647,11 +740,12 @@
(for [square-location (:client/square-locations client) (for [square-location (:client/square-locations client)
:when (:square-location/client-location square-location)] :when (:square-location/client-location square-location)]
(upsert-refunds client square-location)))) (upsert-refunds client square-location))))
([client location] ([client location] (upsert-refunds client location nil nil))
([client location start end]
(with-context-as {:source "Square refunds loading" (with-context-as {:source "Square refunds loading"
:client (:client/code client)} lc :client (:client/code client)} lc
(de/chain (refunds client location) (de/chain (refunds client location start end)
(fn [refunds] (fn [refunds]
(mu/with-context lc (mu/with-context lc
(try (try
@@ -667,7 +761,6 @@
(log/info ::done-loading-refunds))))))) (log/info ::done-loading-refunds)))))))
(defn get-cash-shift [client id] (defn get-cash-shift [client id]
(de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id)) (de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id))
:method :get :method :get
@@ -703,7 +796,10 @@
(de/chain (de/chain
(get-cash-shift client (:id s)) (get-cash-shift client (:id s))
(fn [cash-drawer-shift] (fn [cash-drawer-shift]
#:cash-drawer-shift {:external-id (str "square/cash-drawer-shift/" (:id cash-drawer-shift)) #:cash-drawer-shift {:db/id (or (existing-id (dc/db conn) :cash-drawer-shift/external-id
"square/cash-drawer-shift/" client l (:id cash-drawer-shift))
(str "square/cash-drawer-shift/" (:id cash-drawer-shift)))
:external-id (scoped-key "square/cash-drawer-shift/" client l (:id cash-drawer-shift))
:vendor :vendor/ccp-square :vendor :vendor/ccp-square
:paid-in (amount->money (:cash_paid_in_money cash-drawer-shift)) :paid-in (amount->money (:cash_paid_in_money cash-drawer-shift))
:paid-out (amount->money (:cash_paid_out_money cash-drawer-shift)) :paid-out (amount->money (:cash_paid_out_money cash-drawer-shift))
@@ -723,10 +819,12 @@
:when (:square-location/client-location square-location)] :when (:square-location/client-location square-location)]
(upsert-cash-shifts client square-location)))) (upsert-cash-shifts client square-location))))
([client location] ([client location]
(upsert-cash-shifts client location (time/plus (time/now) (time/days -75)) (time/now)))
([client location start end]
(with-context-as {:source "Square cash shift loading" (with-context-as {:source "Square cash shift loading"
:client (:client/code client)} lc :client (:client/code client)} lc
(de/chain (cash-drawer-shifts client location) (de/chain (cash-drawer-shifts client location start end)
(fn [cash-shifts] (fn [cash-shifts]
(mu/with-context lc (mu/with-context lc
(try (try
@@ -826,8 +924,6 @@
d1 d1
d2)) d2))
(defn remove-voided-orders (defn remove-voided-orders
([client] ([client]
(apply de/zip (apply de/zip
@@ -863,31 +959,26 @@
(log/info ::removing-orders (log/info ::removing-orders
:count (count x)) :count (count x))
@(dc/transact-async conn x))))) @(dc/transact-async conn x)))))
(de/catch (fn [e] (de/catch (fn [e]
(log/warn ::couldnt-remove :error e) (log/warn ::couldnt-remove :error e)
nil) )))))) nil)))))))
#_(comment #_(comment
(require 'auto-ap.time-reader) (require 'auto-ap.time-reader)
@(let [[c [l]] (get-square-client-and-location "DBFS") ] @(let [[c [l]] (get-square-client-and-location "DBFS")]
(log/peek :x [ c l]) (log/peek :x [c l])
(search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29") (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29"))
) @(let [[c [l]] (get-square-client-and-location "NGAK")]
(log/peek :x [c l])
@(let [[c [l]] (get-square-client-and-location "NGAK") ] (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
(log/peek :x [ c l]) (doseq [c (get-square-clients)]
(try
(remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15")) @(remove-voided-orders c)
(doseq [c (get-square-clients)] (catch Exception e
(try nil))))
@(remove-voided-orders c)
(catch Exception e
nil)))
)
(defn upsert-all [& clients] (defn upsert-all [& clients]
(capture-context->lc (capture-context->lc
@@ -950,14 +1041,53 @@
(s/realize-each) (s/realize-each)
(s/reduce conj [])))) (s/reduce conj []))))
(defn backfill-history
"Re-imports orders, payouts, refunds and cash-drawer shifts for `[start end]`, one client at a
time, for every square location the client has.
This exists for the shared-location case. Sales orders have always been keyed by client, so two
client records on one Square location each built their own order history. Refunds, payouts and
shifts were not, so only ONE of the two records holds each of them — whichever imported it last
before the keys were scoped. Re-keying freezes that ownership; it does not even it out, and the
record left without them shows returns from its own orders with no refunds to offset them.
Rather than manufacture copies, this asks Square again. With client-scoped keys in place every
record now creates its own copy of what it reads, so replaying the window is what makes the two
histories match. Deliberately not part of `upsert-all`: it walks further back than the nightly
job and is meant to be run once, after the migration.
Run it AFTER `rekey-square-external-ids/migrate-all!`. Running it before would import against
legacy keys and leave more to migrate."
[start end & client-codes]
(with-context-as {:source "Square historical backfill"} lc
(->> (apply get-square-clients client-codes)
(s/->source)
(s/map (fn [client]
(with-context-as (merge lc {:client (:client/code client)}) lc
(->
(apply de/zip
(for [l (:client/square-locations client)
:when (:square-location/client-location l)]
(de/chain
(upsert client l start end)
(fn [_] (upsert-payouts client l start end))
(fn [_] (upsert-refunds client l start end))
(fn [_] (upsert-cash-shifts client l start end))
(fn [_] (log/info ::backfilled
:location (:square-location/client-location l))))))
(de/catch (fn [e]
(mu/with-context lc
(log/info ::backfill-failed :severity :error :exception e))))))))
(s/buffer 3)
(s/realize-each)
(s/reduce conj []))))
(defn do-upsert-all [& clients] (defn do-upsert-all [& clients]
(mu/trace (mu/trace
::upsert-all ::upsert-all
[:clients clients] [:clients clients]
@(apply upsert-all clients))) @(apply upsert-all clients)))
(comment (comment
(defn refunds-raw-cont (defn refunds-raw-cont
([client l cursor so-far] ([client l cursor so-far]
@@ -987,7 +1117,6 @@
(->> (->>
@(let [[c [l]] (get-square-client-and-location "NGGG")] @(let [[c [l]] (get-square-client-and-location "NGGG")]
(search c l (time/now) (time/plus (time/now) (time/days -1)))) (search c l (time/now) (time/plus (time/now) (time/days -1))))
(filter (fn [r] (filter (fn [r]
@@ -997,7 +1126,6 @@
(->> (->>
@(let [[c [l]] (get-square-client-and-location "NGGG")] @(let [[c [l]] (get-square-client-and-location "NGGG")]
(refunds-raw-cont c l nil [])) (refunds-raw-cont c l nil []))
(filter (fn [r] (filter (fn [r]
(str/starts-with? (:created_at r) "2024-03-14"))))) (str/starts-with? (:created_at r) "2024-03-14")))))
@@ -1032,12 +1160,7 @@
[(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row) (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)]) [(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row) (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
:separator \tab) :separator \tab)
;; =>
;; =>
(require 'auto-ap.time-reader) (require 'auto-ap.time-reader)
@@ -1046,26 +1169,15 @@
(clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")] (clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")]
l l
(def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00" (def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00"
#clj-time/date-time "2025-02-28T00:00:00-08:00")) #clj-time/date-time "2025-02-28T00:00:00-08:00"))
(take 10 (map #(first (deref (order->sales-order c l %))) z))) (take 10 (map #(first (deref (order->sales-order c l %))) z))))
)
(->> z (->> z
(filter (fn [o] (filter (fn [o]
(seq (filter (comp #{"OTHER"} :type) (:tenders o))))) (seq (filter (comp #{"OTHER"} :type) (:tenders o)))))
(filter #(not (:name (:source %)))) (filter #(not (:name (:source %))))
(count) (count))
)
(doseq [[code] (seq (dc/q '[:find ?code (doseq [[code] (seq (dc/q '[:find ?code
:in $ :in $
@@ -1075,32 +1187,22 @@
[?o :sales-order/client ?c] [?o :sales-order/client ?c]
[?c :client/code ?code]] [?c :client/code ?code]]
(dc/db conn))) (dc/db conn)))
:let [[c [l]] (get-square-client-and-location code) :let [[c [l]] (get-square-client-and-location code)]
]
order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now)) order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now))
:when (= "Invoices" (:name (:source order) )) :when (= "Invoices" (:name (:source order)))
:let [[sales-order] @(order->sales-order c l order)]] :let [[sales-order] @(order->sales-order c l order)]]
(when (should-import-order? order) (when (should-import-order? order)
(println "DATE IS" (:sales-order/date sales-order)) (println "DATE IS" (:sales-order/date sales-order))
(when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after? #clj-time/date-time "2026-2-16T00:00:00-08:00")) (when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after? #clj-time/date-time "2026-2-16T00:00:00-08:00"))
(println "WOULD UPDATE" sales-order) (println "WOULD UPDATE" sales-order)
@(dc/transact auto-ap.datomic/conn [sales-order]) @(dc/transact auto-ap.datomic/conn [sales-order]))
) #_@(dc/transact)
#_@(dc/transact ) (println "DONE")))
(println "DONE"))
)
#_(filter (comp #{"OTHER"} :type) (mapcat :tenders z)) #_(filter (comp #{"OTHER"} :type) (mapcat :tenders z))
@(let [[c [l]] (get-square-client-and-location "NGRY")] @(let [[c [l]] (get-square-client-and-location "NGRY")]
#_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01")) #_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
(order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY" ))) (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY")))))
)
)

193
test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj Normal file
View File

@@ -0,0 +1,193 @@
(ns auto-ap.jobs.rekey-square-external-ids-test
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
[auto-ap.jobs.rekey-square-external-ids :as sut]
[clojure.string]
[clojure.test :refer [deftest is testing use-fixtures]]
[datomic.api :as dc]))
(use-fixtures :each wrap-setup)
(def sales-date #inst "2026-08-01T07:00:00.000-00:00")
(defn- charge-count []
(count (dc/q '[:find ?e :where [?e :charge/external-id]] (dc/db conn))))
(defn- charges-of [order]
(->> (dc/datoms (dc/db conn) :eavt order :sales-order/charges)
(map :v)
(map (fn [c] {:eid c
:key (:v (first (dc/datoms (dc/db conn) :eavt c :charge/external-id)))
:total (:v (first (dc/datoms (dc/db conn) :eavt c :charge/total)))
:tip (:v (first (dc/datoms (dc/db conn) :eavt c :charge/tip)))}))
vec))
(defn- parents-of [charge]
(reduce (fn [n _] (inc n)) 0 (dc/datoms (dc/db conn) :vaet charge :sales-order/charges)))
(defn- two-orders-sharing-one-charge []
(let [{:strs [test-client-id]} (setup-test-data [])
other (get-in @(dc/transact conn [{:db/id "other" :client/code "NGCC"}]) [:tempids "other"])
tx @(dc/transact conn [{:db/id "charge"
:charge/external-id "square/charge/shared1"
:charge/type-name "CARD"
:charge/total 120.0
:charge/tip 20.0}
{:db/id "order-a"
:sales-order/external-id "square/order/NGCD-CD-o1"
:sales-order/client test-client-id
:sales-order/location "CD"
:sales-order/date sales-date
:sales-order/charges ["charge"]}
{:db/id "order-b"
:sales-order/external-id "square/order/NGCC-CC-o1"
:sales-order/client other
:sales-order/location "CC"
:sales-order/date sales-date
:sales-order/charges ["charge"]}])]
{:order-a (get-in tx [:tempids "order-a"])
:order-b (get-in tx [:tempids "order-b"])
:charge (get-in tx [:tempids "charge"])
:code-a (:client/code (dc/entity (dc/db conn) test-client-id))
:code-b "NGCC"}))
(deftest a-shared-charge-starts-with-two-parents
(testing "the condition under test really exists before the split runs"
(let [{:keys [charge]} (two-orders-sharing-one-charge)]
(is (= 1 (charge-count)))
(is (= 2 (parents-of charge))
"one charge entity, referenced by both clients' orders"))))
(deftest split-gives-each-order-its-own-charge
(testing "each order ends up with its own charge, scoped to its own client, carrying the same
amounts — so the component relationship means what it says and retracting one order
cannot delete the other's payment"
(let [{:keys [order-a order-b code-a code-b]} (two-orders-sharing-one-charge)
result (sut/split-and-rekey-charges! [order-a order-b] 100)]
(is (= {:rekeyed 1 :cloned 1} result) "first order keeps it, second gets a copy")
(is (= 2 (charge-count)) "exactly one new entity was created")
(let [a (charges-of order-a)
b (charges-of order-b)]
(is (= 1 (count a)))
(is (= 1 (count b)))
(is (= (str "square/charge/" code-a "-CD-shared1") (:key (first a))))
(is (= (str "square/charge/" code-b "-CC-shared1") (:key (first b))))
(is (not= (:eid (first a)) (:eid (first b))) "two distinct entities")
(is (= 120.0 (:total (first a)) (:total (first b))) "amounts copied")
(is (= 20.0 (:tip (first a)) (:tip (first b))) "tips copied")
(is (= 1 (parents-of (:eid (first a)))))
(is (= 1 (parents-of (:eid (first b))))
"no charge has more than one parent order any more")))))
(deftest split-leaves-an-unshared-charge-alone
(testing "an order that already owns its charge outright is only re-keyed, never cloned"
(let [{:strs [test-client-id]} (setup-test-data [])
tx @(dc/transact conn [{:db/id "charge"
:charge/external-id "square/charge/solo1"
:charge/total 50.0}
{:db/id "order"
:sales-order/external-id "square/order/NGCD-CD-o2"
:sales-order/client test-client-id
:sales-order/location "CD"
:sales-order/date sales-date
:sales-order/charges ["charge"]}])
order (get-in tx [:tempids "order"])
code (:client/code (dc/entity (dc/db conn) test-client-id))]
(is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [order] 100)))
(is (= 1 (charge-count)) "nothing was created")
(is (= (str "square/charge/" code "-CD-solo1") (:key (first (charges-of order))))))))
(deftest split-is-idempotent
(testing "re-running over an already-split database changes nothing further"
(let [{:keys [order-a order-b]} (two-orders-sharing-one-charge)]
(sut/split-and-rekey-charges! [order-a order-b] 100)
(let [after-first (charge-count)]
(is (= {:rekeyed 0 :cloned 0} (sut/split-and-rekey-charges! [order-a order-b] 100))
"every charge already carries the key its order expects, so there is nothing to do")
(is (= after-first (charge-count)) "and no further entities appear")))))
(deftest clone-is-not-double-scoped-across-batches
(testing "with a batch size of one, the second order sees a charge already carrying the first
client's scope. It must clone using the underlying Square id, not re-scope the scoped
key — otherwise the entity ends up keyed NGCD-CD-NGCC-CC-<id>, the importer computes
the correct key, misses, and creates a second charge that doubles the tender."
(let [{:keys [order-a order-b code-a code-b]} (two-orders-sharing-one-charge)]
(sut/split-and-rekey-charges! [order-a order-b] 1)
(let [ka (:key (first (charges-of order-a)))
kb (:key (first (charges-of order-b)))]
(is (= (str "square/charge/" code-a "-CD-shared1") ka))
(is (= (str "square/charge/" code-b "-CC-shared1") kb))
(is (not (clojure.string/includes? kb (str code-a "-CD")))
"the clone carries one scope, not two")
(is (= 2 (charge-count)))))))
(deftest two-orders-of-the-same-client-keep-sharing
(testing "one payment covering two of the SAME client's orders is left shared, on purpose.
There is no second name to give a copy — both orders compute the same one — and a copy
would double that client's takings for the day. The component cascade still reaches
these, which is why remove-voided-orders needs its own guard."
(let [{:strs [test-client-id]} (setup-test-data [])
tx @(dc/transact conn [{:db/id "charge"
:charge/external-id "square/charge/same1"
:charge/total 75.0}
{:db/id "o1" :sales-order/external-id "square/order/x-1"
:sales-order/client test-client-id :sales-order/location "CD"
:sales-order/date sales-date :sales-order/charges ["charge"]}
{:db/id "o2" :sales-order/external-id "square/order/x-2"
:sales-order/client test-client-id :sales-order/location "CD"
:sales-order/date sales-date :sales-order/charges ["charge"]}])
o1 (get-in tx [:tempids "o1"]) o2 (get-in tx [:tempids "o2"])
code (:client/code (dc/entity (dc/db conn) test-client-id))]
(is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [o1 o2] 100))
"renamed once, not copied")
(is (= 1 (charge-count)) "no copy was made, so the takings are not doubled")
(is (= (str "square/charge/" code "-CD-same1") (:key (first (charges-of o1)))))
(is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
"both orders still point at the one payment"))))
(deftest the-month-walk-runs-newest-first-and-leaves-no-gaps
(testing "order matters operationally, not just cosmetically: the importer reads recent data, so
an interrupted migration is only safe to resume imports against if the newest months
are the ones already done. Walking :aevt instead would start in 2019."
(let [windows (sut/order-months-newest-first #inst "2026-01-01T12:00:00.000-00:00")
starts (map first windows)]
(is (seq windows))
(is (apply > (map #(.getTime ^java.util.Date %) starts))
"strictly descending — newest month first")
(is (every? (fn [[[next-start _] [_ prev-end]]]
(= (.getTime ^java.util.Date next-start)
(+ (.getTime ^java.util.Date prev-end) (* 24 60 60 1000))))
(partition 2 1 windows))
"each window ends the day before the next one starts, so no order falls between them")
(is (every? (fn [[s e]] (.before ^java.util.Date s ^java.util.Date e)) windows)
"and every window is non-empty"))))
(deftest two-orders-of-the-same-client-keep-sharing-across-batches
(testing "batch size does not change the same-client rule, which the sibling test cannot show
because both its orders land in one batch.
Once the first order re-keys the charge it also writes :charge/client/:charge/location,
so the second order's raw-square-id takes its owner branch, new-key reconstructs the key
the charge already has, and the (not= old new-key) guard drops the row before :action is
read. A clone here would double that client's takings for the day."
(let [{:strs [test-client-id]} (setup-test-data [])
tx @(dc/transact conn [{:db/id "charge"
:charge/external-id "square/charge/same2"
:charge/total 75.0}
{:db/id "o1" :sales-order/external-id "square/order/y-1"
:sales-order/client test-client-id :sales-order/location "CD"
:sales-order/date sales-date :sales-order/charges ["charge"]}
{:db/id "o2" :sales-order/external-id "square/order/y-2"
:sales-order/client test-client-id :sales-order/location "CD"
:sales-order/date sales-date :sales-order/charges ["charge"]}])
o1 (get-in tx [:tempids "o1"]) o2 (get-in tx [:tempids "o2"])
code (:client/code (dc/entity (dc/db conn) test-client-id))]
(is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [o1 o2] 1))
"batch size 1 puts the two orders in separate batches, and still no copy is made")
(is (= 1 (charge-count)) "one payment, not two")
(is (= (str "square/charge/" code "-CD-same2") (:key (first (charges-of o1)))))
(is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
"both orders still point at the one payment"))))

183
test/clj/auto_ap/jobs/sales_summaries_test.clj Normal file
View File

@@ -0,0 +1,183 @@
(ns auto-ap.jobs.sales-summaries-test
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.datomic.sales-summaries :as d-ss]
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
[auto-ap.jobs.sales-summaries :as sut]
[clojure.test :refer [deftest is testing use-fixtures]]
[datomic.api :as dc]))
(use-fixtures :each wrap-setup)
(def sales-date #inst "2026-08-01T07:00:00.000-00:00")
(defn- order
"A sales order on `sales-date`, carrying whatever the case under test needs. The external id
is Square-shaped by default because `get-service-charges` falls back to it when an order has
no `:sales-order/vendor`."
[client id attrs]
(merge {:db/id (str "order-" id)
:sales-order/external-id (str "square/order/TEST-" id)
:sales-order/client client
:sales-order/date sales-date
:sales-order/total 100.0}
attrs))
(defn- charge [id attrs]
(merge {:db/id (str "charge-" id)
:charge/external-id (str "square/charge/" id)
:charge/type-name "CARD"
:charge/total 100.0}
attrs))
(defn- tip-for [client]
(:ledger-mapped/amount (#'sut/get-tip client sales-date)))
(defn- service-charges-for [client]
(#'sut/get-service-charges client sales-date))
(defn- enable-service-charges! [client]
@(dc/transact conn [{:db/id client
:client/feature-flags [sut/service-charges-flag]}]))
(deftest tip-counts-a-reversal-on-an-untendered-order
(testing "a return-only order has no tender to join through, so its negative tip must come
from the order or the day credits a tip that was handed back"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "return-only" {:sales-order/tip -12.0})])
(is (= -12.0 (tip-for test-client-id))))))
(deftest tip-on-a-tendered-order-still-comes-from-the-tender
(testing "the tender carries a tip the order does not — auto-gratuity booked as a service
charge. Reading the order instead of the tender would drop it."
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "tendered"
{:sales-order/tip 0.0
:sales-order/charges [(charge "tendered" {:charge/tip 50.0})]})])
(is (= 50.0 (tip-for test-client-id))))))
(deftest tip-on-an-ordinary-order-is-counted-once
(testing "an order that agrees with its tender is not double counted by the additive form"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "ordinary"
{:sales-order/tip 5.0
:sales-order/charges [(charge "ordinary" {:charge/tip 5.0})]})])
(is (= 5.0 (tip-for test-client-id))))))
(deftest service-charges-need-the-feature-flag
(testing "without the flag the summary behaves exactly as it does today"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "square-sc"
{:sales-order/vendor :vendor/ccp-square
:sales-order/service-charge 50.0})])
(is (nil? (service-charges-for test-client-id))))))
(deftest service-charges-credit-square-orders
(testing "a service charge rides along in the tender, so it needs a credit to match"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "square-sc"
{:sales-order/vendor :vendor/ccp-square
:sales-order/service-charge 50.0})])
(let [item (service-charges-for test-client-id)]
(is (= 50.0 (:ledger-mapped/amount item)))
(is (= :ledger-side/credit (:ledger-mapped/ledger-side item)))
(is (= "Service Charges" (:sales-summary-item/category item)))))))
(deftest service-charges-count-both-signs
(testing "a returned catering fee arrives as a negative service charge and is subtracted back
out of returns, so dropping negatives loses the reversal"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "refunded-fee"
{:sales-order/vendor :vendor/ccp-square
:sales-order/service-charge -140.0})])
(is (= -140.0 (:ledger-mapped/amount (service-charges-for test-client-id)))))))
(deftest service-charges-exclude-non-square-vendors
(testing "ezCater service charges are commission deducted from the restaurant rather than
collected from the diner, so crediting them would make the day worse"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "ezcater-sc"
{:sales-order/external-id "ezcater/order/TEST-ezcater-sc"
:sales-order/vendor :vendor/ccp-ezcater
:sales-order/service-charge -75.0})])
(is (nil? (service-charges-for test-client-id))))))
(deftest service-charges-recognise-square-orders-that-carry-no-vendor
(testing "whole eras of Square orders have no :sales-order/vendor at all; a gate on vendor
alone would silently credit nothing"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "vendorless" {:sales-order/service-charge 12.5})])
(is (= 12.5 (:ledger-mapped/amount (service-charges-for test-client-id)))))))
(deftest service-charges-ignore-vendorless-orders-from-other-sources
(testing "the external id fallback is Square-specific, not a catch-all for missing vendors"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "ezcater-vendorless"
{:sales-order/external-id "ezcater/order/TEST-ezcater-vendorless"
:sales-order/service-charge -75.0})])
(is (nil? (service-charges-for test-client-id))))))
(defn- refund
"A card refund on `sales-date`. The client+date tuple is set explicitly because
`scan-sales-refunds` walks that index rather than the plain attributes."
[client id total]
{:db/id (str "refund-" id)
:sales-refund/external-id (str "square/refund/TEST-" id)
:sales-refund/client client
:sales-refund/date sales-date
:sales-refund/client+date [client sales-date]
:sales-refund/type "CARD"
:sales-refund/total total})
(defn- returns-for [client]
(#'sut/get-returns client sales-date))
(deftest a-refund-with-no-sales-leaves-the-day-out-of-balance
(testing "deliberate, and load-bearing. Booking a return against the day's refunds would close
it and is tempting for that reason. But a day with refunds and no sales at all is
overwhelmingly a day whose ORDERS WERE NEVER IMPORTED — on a restored copy of
production, 132 of 156 such days fell before their client's first ever synced order.
Balancing them would turn the only signal that a client's sales are missing into
silence. If this test starts failing, read the rollout plan before changing it."
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(refund test-client-id "no-sales" 40.0)])
(is (nil? (returns-for test-client-id))
"no return is invented for a day that recorded no sales")
(is (= -40.0 (d-ss/imbalance (sut/get-refund-items test-client-id sales-date)))
"so the day stays out of balance by the refunded amount, visibly"))))
(deftest a-day-that-traded-books-its-own-return
(testing "the ordinary case: the return comes from the day's orders, never from its refunds"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "traded" {:sales-order/returns 7.0})
(refund test-client-id "same-day" 40.0)])
(is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id)))))))
(deftest dirty-summaries-stop-at-the-client-boundary
(testing "every dirty day for the client is returned, and none belonging to another client.
:sales-summary/client+dirty sorts by client, so an unbounded index scan would walk
every later client's summaries too — correct, but quadratic in the summary count."
(let [{:strs [test-client-id]} (setup-test-data [])
other (get-in @(dc/transact conn [{:db/id "other" :client/code (str "OTHER" (rand-int 100000))}])
[:tempids "other"])
day (fn [client d dirty?]
{:sales-summary/client client
:sales-summary/date d
:sales-summary/dirty dirty?})]
@(dc/transact conn [(day test-client-id #inst "2026-08-01T07:00:00.000-00:00" true)
(day test-client-id #inst "2026-08-02T07:00:00.000-00:00" true)
(day test-client-id #inst "2026-08-03T07:00:00.000-00:00" false)
(day other #inst "2026-08-01T07:00:00.000-00:00" true)
(day other #inst "2026-08-02T07:00:00.000-00:00" true)])
(let [mine (sut/dirty-sales-summaries test-client-id)]
(is (= 2 (count mine)) "both dirty days, and not the clean one")
(is (every? #(= test-client-id (:db/id (:sales-summary/client %))) mine)
"and nothing belonging to the other client"))
(is (= 2 (count (sut/dirty-sales-summaries other)))
"the other client's own dirty days are still found"))))

201
test/clj/auto_ap/square/core3_test.clj Normal file
View File

@@ -0,0 +1,201 @@
(ns auto-ap.square.core3-test
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
[auto-ap.square.core3 :as sut]
[clojure.test :refer [deftest is testing use-fixtures]]
[datomic.api :as dc]))
(use-fixtures :each wrap-setup)
(def client {:client/code "NGCD"})
(def location {:square-location/client-location "CD"})
(defn- refund-count []
(count (dc/q '[:find ?e :where [?e :sales-refund/external-id]] (dc/db conn))))
(defn- resolve-refund [id]
(sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" client location id))
(deftest scoped-key-carries-client-and-location
(testing "the same shape sales order keys already use, so a shared location cannot contend"
(is (= "square/refund/NGCD-CD-abc" (sut/scoped-key "square/refund/" client location "abc")))
(is (= "square/charge/NGCD-CD-xyz" (sut/scoped-key "square/charge/" client location "xyz")))))
(deftest legacy-keyed-entity-is-updated-not-duplicated
(testing "an entity still carrying its unscoped key is found and re-keyed in place.
This is the sharpest hazard in the migration: these external ids are
:db.unique/identity, so writing the new key without resolving the old one first
matches nothing and creates a second entity, orphaning the original."
(setup-test-data [])
@(dc/transact conn [{:db/id "r"
:sales-refund/external-id "square/refund/abc"
:sales-refund/total 10.0}])
(is (= 1 (refund-count)))
(let [eid (resolve-refund "abc")]
(is (some? eid) "resolves an entity carrying the legacy key")
@(dc/transact conn [{:db/id eid
:sales-refund/external-id (sut/scoped-key "square/refund/" client location "abc")
:sales-refund/total 10.0}])
(is (= 1 (refund-count)) "no second entity was created")
(is (= eid (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/NGCD-CD-abc"]))
"the same entity now answers to the scoped key")
(is (nil? (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/abc"]))
"and no longer to the legacy one"))))
(deftest already-scoped-entity-resolves-by-its-new-key
(testing "re-running the importer after migration finds the entity by the scoped key, so the
migration is not undone and nothing is duplicated"
(setup-test-data [])
@(dc/transact conn [{:db/id "r"
:sales-refund/external-id "square/refund/NGCD-CD-abc"
:sales-refund/total 10.0}])
(is (= (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/NGCD-CD-abc"])
(resolve-refund "abc")))
(is (= 1 (refund-count)))))
(deftest unknown-id-resolves-to-nothing
(testing "a refund never seen before has no id to pin, so the importer creates it fresh"
(setup-test-data [])
(is (nil? (resolve-refund "never-seen")))))
(deftest two-clients-on-one-location-get-their-own-entities
(testing "the point of the re-key: with the client in the key, a second client importing the
same Square refund creates its own entity instead of taking ownership of the first"
(setup-test-data [])
(let [other {:client/code "NGCC"}
other-loc {:square-location/client-location "CC"}]
@(dc/transact conn [{:db/id "r"
:sales-refund/external-id (sut/scoped-key "square/refund/" client location "shared")
:sales-refund/total 10.0}])
(is (nil? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" other other-loc "shared"))
"the second client does not resolve onto the first client's entity")
@(dc/transact conn [{:db/id "r2"
:sales-refund/external-id (sut/scoped-key "square/refund/" other other-loc "shared")
:sales-refund/total 10.0}])
(is (= 2 (refund-count)) "two stable entities, one per client, rather than one that flips"))))
(deftest legacy-key-of-another-client-is-not-claimed
(testing "the legacy fallback must not hand one client a record that already belongs to another.
Without this, two clients on one Square location double money in the window between
deploying and finishing the migration."
(setup-test-data [])
(let [tx @(dc/transact conn [{:db/id "mine" :client/code (str "MINE" (rand-int 100000))}
{:db/id "theirs" :client/code (str "THEIRS" (rand-int 100000))}])
mine {:db/id (get-in tx [:tempids "mine"]) :client/code "MINE"}
theirs-id (get-in tx [:tempids "theirs"])]
@(dc/transact conn [{:db/id "r"
:sales-refund/external-id "square/refund/abc"
:sales-refund/client theirs-id
:sales-refund/total 10.0}])
(is (nil? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/"
mine location "abc"))
"a legacy-keyed refund owned by another client is left alone")
(is (some? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/"
{:db/id theirs-id :client/code "THEIRS"} location "abc"))
"its own client still resolves it, so re-keying in place still works"))))
(deftest a-charge-is-owned-by-the-client-of-the-order-that-refers-to-it
(testing "charges predating :charge/client still have orders, and those are exactly the ones
that could be taken by the wrong client"
(let [{:strs [test-client-id]} (setup-test-data [])
other (get-in @(dc/transact conn [{:db/id "o" :client/code (str "OTHER" (rand-int 100000))}])
[:tempids "o"])]
@(dc/transact conn [{:db/id "c" :charge/external-id "square/charge/p1" :charge/total 50.0}
{:db/id "ord" :sales-order/external-id "square/order/x-1"
:sales-order/client test-client-id :sales-order/location "CD"
:sales-order/date #inst "2026-06-03T07:00:00.000-00:00"
:sales-order/charges ["c"]}])
(is (nil? (sut/existing-id (dc/db conn) :charge/external-id "square/charge/"
{:db/id other :client/code "OTHER"} location "p1"))
"ownership is read from the referencing order when :charge/client is absent"))))
(deftest deploy-window-does-not-double-a-second-clients-tender
(testing "the P0 this guard exists for, end to end.
Client A's payout import reaches for a payment whose charge belongs to client B's
order. If A were allowed to re-key it, B's next order import would match neither
scheme, mint a second charge, and — since :sales-order/charges is cardinality-many —
leave B's order holding two charges for one payment."
(let [{:strs [test-client-id]} (setup-test-data [])
b-code (:client/code (dc/entity (dc/db conn) test-client-id))
b {:db/id test-client-id :client/code b-code}
b-loc {:square-location/client-location "LB"}
a-id (get-in @(dc/transact conn [{:db/id "a" :client/code (str "AAA" (rand-int 100000))}])
[:tempids "a"])
a {:db/id a-id :client/code (:client/code (dc/entity (dc/db conn) a-id))}
a-loc {:square-location/client-location "LA"}
tx @(dc/transact conn [{:db/id "x" :charge/external-id "square/charge/P"
:charge/total 100.0 :charge/type-name "CARD"
:charge/client test-client-id :charge/location "LB"}
{:db/id "ob" :sales-order/external-id "square/order/b-1"
:sales-order/client test-client-id :sales-order/location "LB"
:sales-order/date #inst "2026-06-03T07:00:00.000-00:00"
:sales-order/charges ["x"]}])
order-b (get-in tx [:tempids "ob"])
charges-of (fn [o] (map :v (dc/datoms (dc/db conn) :eavt o :sales-order/charges)))]
;; client A's payout import touches the same Square payment
@(dc/transact conn [(into {} (remove (comp nil? val))
{:charge/external-id (sut/scoped-key "square/charge/" a a-loc "P")
:charge/client a-id
:charge/location "LA"
:db/id (sut/existing-id (dc/db conn) :charge/external-id
"square/charge/" a a-loc "P")})])
;; client B's order re-imports
@(dc/transact conn [{:db/id order-b
:sales-order/charges
[(sut/tender->charge {:id "b-1" :created_at "2026-06-03T12:00:00Z"}
b b-loc {:id "P" :type "CARD"
:amount_money {:amount 10000
:currency "USD"}})]}])
(is (= 1 (count (charges-of order-b)))
"B's order still holds exactly one charge for the one payment")
(is (= 100.0 (reduce + 0.0 (map #(:charge/total (dc/entity (dc/db conn) %))
(charges-of order-b))))
"so the day's tender is not doubled")
(is (= (str "square/charge/" b-code "-LB-P")
(:charge/external-id (dc/entity (dc/db conn) (first (charges-of order-b)))))
"and B's own charge was re-keyed in place rather than abandoned"))))
(deftest payouts-and-shifts-are-client-scoped-too
(testing "expected deposits and cash drawer shifts are fetched per location, so two clients on
one location collide on them exactly as refunds and charges did"
(is (= "square/payout/NGCD-CD-po1"
(sut/scoped-key "square/payout/" client location "po1")))
(is (= "square/cash-drawer-shift/NGCD-CD-sh1"
(sut/scoped-key "square/cash-drawer-shift/" client location "sh1")))))
(deftest legacy-keyed-deposit-is-updated-not-duplicated
(testing "a payout still carrying its unscoped key is found and re-keyed in place"
(setup-test-data [])
@(dc/transact conn [{:db/id "d"
:expected-deposit/external-id "square/payout/po1"
:expected-deposit/total 100.0}])
(let [eid (sut/existing-id (dc/db conn) :expected-deposit/external-id "square/payout/" client location "po1")]
(is (some? eid) "resolves the entity carrying the legacy key")
@(dc/transact conn [{:db/id eid
:expected-deposit/external-id (sut/scoped-key "square/payout/" client location "po1")
:expected-deposit/total 100.0}])
(is (= 1 (count (dc/q '[:find ?e :where [?e :expected-deposit/external-id]] (dc/db conn))))
"no second deposit was created")
(is (nil? (dc/entid (dc/db conn) [:expected-deposit/external-id "square/payout/po1"]))
"the legacy key is gone"))))
(deftest two-clients-get-their-own-deposit
(testing "with the client in the key, a second client importing the same Square payout creates
its own entity instead of taking ownership of the first"
(setup-test-data [])
(let [other {:client/code "NGCC"}
other-loc {:square-location/client-location "CC"}]
@(dc/transact conn [{:db/id "d"
:expected-deposit/external-id (sut/scoped-key "square/payout/" client location "shared")
:expected-deposit/total 100.0}])
(is (nil? (sut/existing-id (dc/db conn) :expected-deposit/external-id "square/payout/" other other-loc "shared"))
"the second client does not resolve onto the first client's deposit")
@(dc/transact conn [{:db/id "d2"
:expected-deposit/external-id (sut/scoped-key "square/payout/" other other-loc "shared")
:expected-deposit/total 100.0}])
(is (= 2 (count (dc/q '[:find ?e :where [?e :expected-deposit/external-id]] (dc/db conn))))
"two stable entities, one per client, rather than one that flips"))))

153
test/clj/auto_ap/tools/compare_sales_summaries.clj Normal file
View File

@@ -0,0 +1,153 @@
(ns auto-ap.tools.compare-sales-summaries
"Compares sales summaries between two points in the same database.
A verification tool, not part of the running application: it lives on the test/dev classpath so
nothing in production can depend on it. Load it from a REPL when auditing a recompute.
The question this exists to answer is narrower than \"did the totals improve\": it is *which
days changed, and were any of them already balanced*. A day that was balanced before and still
balances after can still have had its line amounts move, and that is a real change to the
books even though no red turns green. Counting only balanced/unbalanced transitions would hide
it entirely.
DO NOT USE `compare-against` — OR ANY `d/as-of` DATABASE — TO COMPARE AMOUNTS. The obvious
reading is that Datomic keeps every past value, so a recompute can be audited against what was
there before with no snapshot. That does not hold here: `:ledger-mapped/amount`,
`:ledger-mapped/ledger-side` and `:ledger-mapped/account` are all `:db/noHistory true`, so
superseded values are discarded rather than retained. A summary that has since been recomputed
reads back through `as-of` with its categories intact and its amounts *absent* — which is
indistinguishable from a legitimate all-zero day, and quietly turns every rewritten summary
into a false \"was balanced, still balances\".
To compare amounts, capture `summaries-in` from a live `(d/db conn)` immediately after each
run, keep the two captures outside the database, and diff those. `compare-window` is safe when
both arguments are live database values; only the historical read is unsound. Categories and
which-days-changed do survive `as-of`, since `:sales-summary-item/category` retains history."
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.datomic.sales-summaries :as d-ss]
[clj-time.coerce :as c]
[datomic.api :as dc]))
(def item-read
[:sales-summary-item/category
:sales-summary-item/manual?
:ledger-mapped/amount
{:ledger-mapped/ledger-side [:db/ident]}
{:ledger-mapped/account [:account/numeric-code]}])
(defn- cents
"Amounts are doubles carrying float noise, so compare them at the cent — the unit the books are
actually kept in. Without this, 182.87000000000003 and 182.87 read as a change."
[x]
(Math/round (* 100.0 (double (or x 0.0)))))
(defn- line
"One item reduced to what a reader would call \"the number\": category, side, amount, account."
[item]
{:category (:sales-summary-item/category item)
:side (get-in item [:ledger-mapped/ledger-side :db/ident])
:cents (cents (:ledger-mapped/amount item))
:account (get-in item [:ledger-mapped/account :account/numeric-code])})
(defn summaries-in
"`{[client-code date] {:lines … :imbalance … :balanced?}}` for every summary in `[start end)`.
Keyed by client code and date rather than entity id so the two sides line up even if an entity
were recreated between the points being compared."
[db start end]
(->> (dc/q {:find [(list 'pull '?s [:sales-summary/date
{:sales-summary/client [:client/code]}
{:sales-summary/items item-read}])]
:in '[$ ?start ?end]
:where '[[?s :sales-summary/date ?d]
[(>= ?d ?start)]
[(< ?d ?end)]]}
db (c/to-date start) (c/to-date end))
(map first)
(reduce (fn [acc s]
(let [items (map d-ss/<-pulled-item (:sales-summary/items s))]
(assoc acc
[(get-in s [:sales-summary/client :client/code]) (:sales-summary/date s)]
{:lines (frequencies (map line (:sales-summary/items s)))
:imbalance (d-ss/imbalance items)
:balanced? (d-ss/balanced? items)})))
{})))
(defn- classify
"How one client-day differs. `:numbers-changed` is the interesting one — the lines themselves
moved, whether or not the day's balance status did."
[before after]
(cond
(nil? before) :added
(nil? after) :removed
(= (:lines before) (:lines after)) :identical
:else :numbers-changed))
(defn compare-window
"Compares every summary in `[start end)` between two database values.
Returns per-day rows plus the tallies worth reporting, including the one that is easy to miss:
days that were **already balanced** and whose numbers moved anyway."
[before-db after-db start end]
(let [before (summaries-in before-db start end)
after (summaries-in after-db start end)
rows (for [k (distinct (concat (keys before) (keys after)))
:let [b (get before k) a (get after k)]]
{:client (first k)
:date (second k)
:change (classify b a)
:was-balanced? (:balanced? b)
:now-balanced? (:balanced? a)
:before-imbalance (:imbalance b)
:after-imbalance (:imbalance a)
:lines-before (:lines b)
:lines-after (:lines a)})
rows (vec rows)
changed (filter #(= :numbers-changed (:change %)) rows)]
{:rows rows
:tally {:compared (count rows)
:identical (count (filter #(= :identical (:change %)) rows))
:numbers-changed (count changed)
:added (count (filter #(= :added (:change %)) rows))
:removed (count (filter #(= :removed (:change %)) rows))}
:balance-transitions
{:unbalanced->balanced (count (filter #(and (false? (:was-balanced? %)) (true? (:now-balanced? %))) rows))
:balanced->unbalanced (count (filter #(and (true? (:was-balanced? %)) (false? (:now-balanced? %))) rows))
:stayed-balanced (count (filter #(and (true? (:was-balanced? %)) (true? (:now-balanced? %))) rows))
:stayed-unbalanced (count (filter #(and (false? (:was-balanced? %)) (false? (:now-balanced? %))) rows))}
:previously-balanced-and-changed
(->> changed (filter :was-balanced?) vec)}))
(defn line-diff
"Which categories actually moved on one row, as `{category [before-cents after-cents]}`. For
reading a handful of rows by hand once the tallies point at them."
[row]
(let [by-cat (fn [lines] (reduce (fn [m [l n]] (assoc m (:category l) (* n (:cents l)))) {} lines))
b (by-cat (:lines-before row))
a (by-cat (:lines-after row))]
(->> (distinct (concat (keys b) (keys a)))
(keep (fn [cat]
(let [x (get b cat 0) y (get a cat 0)]
(when (not= x y) [cat [(/ x 100.0) (/ y 100.0)]]))))
(into {}))))
(defn compare-against
"Compare the current database against its own past value at basis `t`.
UNSOUND FOR AMOUNTS — see the namespace docstring. The amount, side and account attributes are
`:db/noHistory`, so any summary rewritten since `t` reads back with no amounts and appears
balanced. Kept only for comparing categories and identifying which days changed."
[t start end]
(let [db (dc/db conn)]
(compare-window (dc/as-of db t) db start end)))
(comment
;; the restore point, i.e. production's own summaries before any of this work
(def result (compare-against 209608347
(clj-time.core/date-time 2026 7 15)
(clj-time.core/date-time 2026 8 14)))
(:tally result)
(:balance-transitions result)
(count (:previously-balanced-and-changed result))
(map line-diff (take 3 (:previously-balanced-and-changed result))))