4a1817711d45c7ce265b75269b84530f12e0b3a7
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 4a1817711d |
docs: re-measure from a fresh restore with the duplicates left active
The report's figures were taken with one client record of each shared
pair deactivated and a live Square import run afterwards. That is no
longer how this deploys — both records stay live and the re-key is what
separates them — so the numbers described a configuration that will not
exist.
Re-run from scratch: fresh restore of backup point 209608347 (verified
back to 16,545,495 charges and legacy keys before starting), the new
month-wise migration, then two full ninety-day recomputes.
before 1,451 days out of balance / $81,023.96
after 542 days out of balance / $20,239.00
915 into balance, 6 out, 17,916 summaries untouched
Two corrections to claims that no longer hold:
- "zero days knocked out of balance" is now six. All six are the same
shape — the Tip line falls by a round amount and the day breaks by
exactly that — and all six are on shared-location records. Each is a
tip reversed on one record whose refund went to its twin: the fix makes
a hidden mis-attribution visible rather than causing one.
- the migration takes ~38 minutes, not ~13. The month-wise walk adds
per-month index overhead, and the earlier figure predated it.
What did NOT move is the part that should not. For the 190 clients that
do not share a Square location the residue is 119 days and $1,730.61 in
both runs, with the same five restaurants accounting for it (NG4S, NGMV,
NGEB, NGPS, N-30012). The arithmetic fixes behave identically whatever is
done to the duplicates, which is a stronger check than either run alone.
$18,508.39 of the $20,239.00 — 91% — sits on the twenty shared-location
records. The report now states plainly what retiring the duplicates would
be worth (~$12,000 of variance across ~260 client-days per ninety days),
while noting the comparison is not perfectly isolated because the earlier
run also included a live import.
Migration re-measured: 16,236,839 re-keyed, 500,438 cloned, plan reports
{:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0},
and the multi-parent gate reads 0 across all 5,158,470 orders of the last
year. Step 10's refunds re-measured too: 660 worth $15,237.02 dated
before their own record's first order, 140 of the 171 no-sales days
falling before it.
31 tests, 76 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|||
| 967e77e443 |
feat(square): walk the migration newest month first, so stopping early is safe
migrate-all! drove the charge split from all-order-ids, which streams :aevt — ascending entity id, so oldest first. On the production copy that means the first several hours are spent on 2019 and 2020 data no import will ever read, leaving the recent end (the part the importer actually touches) for last. Interrupt it there and the data is unmigrated exactly where it matters. Now: - refunds, payouts and cash-drawer shifts run first. Together they are ~266k records and take seconds, so an interruption cannot leave them half done. - the long order walk then runs a month at a time from the current month backwards, logging ::month-complete with per-month counts. Stop it after any month and everything from that month forward is fully scoped, so imports can resume against a partially migrated database and the older tail can be finished later — the re-run skips what is done. While a tail remains unmigrated, existing-id's ownership guard is what keeps it safe. order-months-newest-first tiles [start end] windows with no gaps (each month ends the day before the next begins) and is bounded below by a constant comfortably older than the oldest order. Windows are walked via the :sales-order/client+date index. Verified against the restored copy: 141 windows from 2026-08 back, August returning 241,126 orders and July 476,235, each dated inside its window. all-order-ids keeps its old behaviour but its docstring now warns that a prefix of it is the oldest orders, not a sample — the trap that made an earlier verification gate read 2019-2021 data. 31 tests, 76 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| 10d0d01b82 |
fix(square): do not resolve a record that belongs to another client
Two clients on one Square location double a day's tender in the window
between deploying and finishing the migration. Reproduced end to end:
1. charge X carries the legacy key square/charge/P and belongs to
client B's order
2. client A's payout import resolves X through existing-id's legacy
fallback and renames it into A's scope
3. client B's next order import matches neither scheme, so it mints a
second charge
4. :sales-order/charges is cardinality-many and orders transact as
plain maps, so nothing retracts the first
B's order ends up holding two charges for one payment — $200 of tender
for a $100 payment — and running the migration on that state produces
square/charge/BBB-LB-AAA-LA-P, the same double-scoped shape that already
doubled tender on five clients once during this work.
existing-id's legacy branch now declines any record already owned by a
different client, reading the owner attribute and, for charges that
predate :charge/client, the client of the referencing order. Declining is
also correct on its merits: the write then lands on this client's own
copy, which is what the scoped keys exist to create. The payout path also
writes :charge/client/:charge/location alongside the key, so a charge's
scope and its owner can no longer disagree.
The guard is transitional and gets deleted with the legacy branch it
protects, at rollout step 9.
Rollout resequenced for the decision to leave duplicate client records
active: no deactivation, no "which record survives" call, and the risk
window closed by pausing the importer across deploy + migrate rather than
by removing one of the two writers. Both records converge to independent
stable histories once every key carries its owner.
Also from review:
- migrate-all! now collision-checks the charge pass like the other three
attributes instead of discovering a clash mid-run over 17M rows
- split-and-rekey-charges! logs progress every 200 batches; an
interrupted 19M-order run left no trail
- unscoped-report's docstring no longer promises a zero its :no-owner
column cannot reach; plan is named as the authoritative signal
- the rollout's pre-flight asked for a :collisions key plan never
returns, so it silently passed on every database
- the multi-parent gate sampled (take 400000 (all-order-ids db)), which
streams :aevt — ascending entity id — and so read the OLDEST 2% of
orders: 2019-12-31 to 2021-06-03, before any of the contention it
looks for. Now every order of the last year via the client+date index,
5,159,787 on the restored copy, reading 0
- the report claimed same-client pairs get copied once batches split
them. They do not, at any batch size; verified at batch-size 1 and now
pinned by a test
30 tests, 72 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|||
| 57a84dae11 |
fix(sales-summaries): stop days falling out of balance
Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — plus a fourth that turned out to be a missing-data problem and is deliberately left visible. Measured over ninety days on a restored copy of production (210 clients, 18,900 client-days): 1,258 days out of balance and $69,560.10 becomes 279 days and $7,790.54 — of which 171 are not arithmetic faults at all, but days whose sales were never imported. 979 days repaired, none knocked out of balance, and not one already-balanced day altered — verified line by line (category, side, amount to the cent, account), not just on each day's bottom line. THE DATA FAULT Ten Square locations were configured against two client records each. Sales orders scoped their identifier by client; refunds, card payments, payouts and cash-drawer shifts used the bare Square id. Those attributes are :db.unique/identity, so both clients' imports resolved to a single entity and the last writer won — 3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts changed hands over time, across 19 client pairs of which only 10 are visible in today's configuration. Worse, one payment could belong to two orders. :sales-order/charges is :db/isComponent, so removing a voided order cascaded into payments the other client still needed. Fixes: client-scope the four key schemes; look the record up under both schemes so the change deploys before the migration finishes; and a migration that gives every order its own payment. Run over the whole database that is 19,040,785 orders walked, 9,100,314 payments re-keyed and 200,027 copied, ending with 17,047,142 payments scoped, none left to rename, none unscopable, and no payment owned by more than one order. Idempotent and resumable; about thirteen minutes. THE ARITHMETIC FAULTS - Refunded tips stayed on the books. get-tip summed tips by joining through :sales-order/charges, so a return-only order — no tender to join through — contributed nothing while its reversal sat unread on :sales-order/tip. Additive, not substitutive: where an order does have a tender the tender is the correct source. - Service charges were collected but never earned. Nothing read :sales-order/service-charge. Now credited for Square orders only, both signs, behind summary-service-charges. The flag is off by default, so deploying this changes nothing until a client is opted in. docs/2026-08-15-sales-summary-rollout-plan.md has the steps. WHAT IS DELIBERATELY NOT FIXED 156 of the 279 remaining days carry refunds on a record that recorded no sales at all that day, and 132 of those fall before that client's first ever order. The refunds are not theirs: ownership history shows a $35.35 refund dated 26 February belonging to NGDG that day and taken over by NGDU on 12 August, flipping between the two several times a day. Across nine records, 659 refunds worth $15,225.24 sit on a record dated before its own first order — unscoped keys let whichever import ran last take ownership. A rule closing those days was written and measured (156 days, $4,820.19, nothing broken) and then removed. An unbalanced day is the only visible signal that a restaurant's sales are not being imported; balancing it would remove the alarm and leave the fire. A comment and a test hold that decision in place. Step 9 of the rollout plan is the real fix, and it needs a business decision. SUPPORTING - Install schema attributes before the tuples that compose them. A tuple in schema.edn is built from an attribute in cloud-migration-schema.edn, so every test fixture died in setup — very likely why sales summaries had no tests before this. - Log each day's imbalance and its suspect lines. - Bound the dirty-summary scan to one client: 1,321 ms to 5.6 ms. - compare-sales-summaries lives in test/clj as auto-ap.tools.* — it is a verification harness, not part of the running application. Its docstring now warns that d/as-of cannot be used to compare summary amounts: :ledger-mapped/amount, ledger-side and account are :db/noHistory, so a recomputed summary reads back with its amounts absent and looks like a legitimate balanced day. 26 tests, 62 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
