The headline and the new backfill step were current, but step 11 and
"what this will not fix" still carried pre-backfill figures. The backfill
did not just change the totals — it changed step 11's population.
Before it, nine records held refunds dated before their own first order:
660 refunds worth $15,237.02. Seven of those were shared-location twins
whose refunds only looked orphaned because their orders had never been
imported; replaying the window gave them their orders and the refunds
stopped predating them.
Two are left, and they are a different case — neither shares a Square
location, so there is no twin holding the other half:
NG4S first order 2026-05-29 79 refunds $2,180.08 10 days
NGPS first order 2026-05-26 81 refunds $2,167.60 7 days
Step 11 now recommends trying backfill-history on them first, with a
window reaching back before their first order, since that is exactly what
resolved the other seven.
"What this will not fix" re-measured: 122 days / $2,379.45, of which 106
are real trading days ($1,151.80) and 16 are refunds with no sales
imported ($1,227.65). Only 3 of the trading days are on shared-location
records, all already diagnosed. The other 103 days and $502.96 have been
identical in every run of this analysis — deactivated, live, and
backfilled — and are the floor this work reaches.
Also adds the backfill's ~5.9 hour runtime to the up-front table, and
notes that "two entities per Square object" holds automatically for new
imports but needs step 5 for existing history.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-keying stops two client records on one Square location fighting over a
record, but it does not make their books equal. Sales orders have always
been keyed by client, so each record built its own order history from the
start. Refunds, payouts and cash-drawer shifts were not, so only ONE
record holds each of them — whichever imported it last. The migration
freezes that ownership rather than evening it out.
The record left without them shows returns from its own orders and no
refunds against them. NGBK held 158,535 orders and five refunds. On
2026-05-11 both NGBK and NGBR held the same 221 orders; NGBK had no
refunds, NGBR had two worth $2,232.29, and NGBK was out by $2,232.29 to
the cent.
Rather than manufacture copies, ask Square again. Client-scoped keys mean
each record now creates its own copy of whatever it reads, so replaying a
window converges the two histories with no code inventing a duplicate.
`backfill-history` does that for a date range across orders, payouts,
refunds and shifts. After it, all ten pairs held matching order and
refund counts.
Fixes a capped read found by doing this: the refunds import asked Square
for a location's refunds and read only the first page — no cursor, no
date range. Square pages at a hundred, so a location with more refunds
silently returned a hundred and the response looked complete. That is why
each twin held almost exactly 100 refunds and why an earlier import added
exactly 1,000 across ten locations. `refunds`/`upsert-refunds` now follow
the cursor and accept a window.
Re-measured from the same fresh restore, duplicates left active:
before 1,191 days out of balance / $70,276.50
after 122 days out of balance / $2,379.45 (32 material)
1,069 into balance, 0 out, 0 already-balanced days altered
The shared records went from 423 days / $18,508.39 to 3 days / $648.84 —
NGBK and NGBR at $299.42 each (the known tender-versus-order gap) and
NGDA at $50.00 (auto-gratuity as a service charge). The zero-regression
guarantee is restored too: the six days that broke without the backfill
were tips reversed on one record whose refund sat on the twin, and all
six closed once both sides had their own copy.
This beats retiring the duplicate records, which left 279 days and
$7,790.54, and it needs no decision about whose history to abandon.
Unchanged across every run: for the 190 clients that do not share a
location, 119 days and $1,730.61, same five restaurants.
Cost: 5.9 hours for ninety days across twenty records. Every Square call
shares one 25 req/s throttle, refunds and shifts cost one API call each,
and backfill-history imports three clients at a time. Reads are not the
limit — existing-id measures 32 microseconds.
31 tests, 76 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report's figures were taken with one client record of each shared
pair deactivated and a live Square import run afterwards. That is no
longer how this deploys — both records stay live and the re-key is what
separates them — so the numbers described a configuration that will not
exist.
Re-run from scratch: fresh restore of backup point 209608347 (verified
back to 16,545,495 charges and legacy keys before starting), the new
month-wise migration, then two full ninety-day recomputes.
before 1,451 days out of balance / $81,023.96
after 542 days out of balance / $20,239.00
915 into balance, 6 out, 17,916 summaries untouched
Two corrections to claims that no longer hold:
- "zero days knocked out of balance" is now six. All six are the same
shape — the Tip line falls by a round amount and the day breaks by
exactly that — and all six are on shared-location records. Each is a
tip reversed on one record whose refund went to its twin: the fix makes
a hidden mis-attribution visible rather than causing one.
- the migration takes ~38 minutes, not ~13. The month-wise walk adds
per-month index overhead, and the earlier figure predated it.
What did NOT move is the part that should not. For the 190 clients that
do not share a Square location the residue is 119 days and $1,730.61 in
both runs, with the same five restaurants accounting for it (NG4S, NGMV,
NGEB, NGPS, N-30012). The arithmetic fixes behave identically whatever is
done to the duplicates, which is a stronger check than either run alone.
$18,508.39 of the $20,239.00 — 91% — sits on the twenty shared-location
records. The report now states plainly what retiring the duplicates would
be worth (~$12,000 of variance across ~260 client-days per ninety days),
while noting the comparison is not perfectly isolated because the earlier
run also included a live import.
Migration re-measured: 16,236,839 re-keyed, 500,438 cloned, plan reports
{:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0},
and the multi-parent gate reads 0 across all 5,158,470 orders of the last
year. Step 10's refunds re-measured too: 660 worth $15,237.02 dated
before their own record's first order, 140 of the 171 no-sales days
falling before it.
31 tests, 76 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
migrate-all! drove the charge split from all-order-ids, which streams
:aevt — ascending entity id, so oldest first. On the production copy that
means the first several hours are spent on 2019 and 2020 data no import
will ever read, leaving the recent end (the part the importer actually
touches) for last. Interrupt it there and the data is unmigrated exactly
where it matters.
Now:
- refunds, payouts and cash-drawer shifts run first. Together they are
~266k records and take seconds, so an interruption cannot leave them
half done.
- the long order walk then runs a month at a time from the current month
backwards, logging ::month-complete with per-month counts.
Stop it after any month and everything from that month forward is fully
scoped, so imports can resume against a partially migrated database and
the older tail can be finished later — the re-run skips what is done.
While a tail remains unmigrated, existing-id's ownership guard is what
keeps it safe.
order-months-newest-first tiles [start end] windows with no gaps (each
month ends the day before the next begins) and is bounded below by a
constant comfortably older than the oldest order. Windows are walked via
the :sales-order/client+date index. Verified against the restored copy:
141 windows from 2026-08 back, August returning 241,126 orders and July
476,235, each dated inside its window.
all-order-ids keeps its old behaviour but its docstring now warns that a
prefix of it is the oldest orders, not a sample — the trap that made an
earlier verification gate read 2019-2021 data.
31 tests, 76 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two clients on one Square location double a day's tender in the window
between deploying and finishing the migration. Reproduced end to end:
1. charge X carries the legacy key square/charge/P and belongs to
client B's order
2. client A's payout import resolves X through existing-id's legacy
fallback and renames it into A's scope
3. client B's next order import matches neither scheme, so it mints a
second charge
4. :sales-order/charges is cardinality-many and orders transact as
plain maps, so nothing retracts the first
B's order ends up holding two charges for one payment — $200 of tender
for a $100 payment — and running the migration on that state produces
square/charge/BBB-LB-AAA-LA-P, the same double-scoped shape that already
doubled tender on five clients once during this work.
existing-id's legacy branch now declines any record already owned by a
different client, reading the owner attribute and, for charges that
predate :charge/client, the client of the referencing order. Declining is
also correct on its merits: the write then lands on this client's own
copy, which is what the scoped keys exist to create. The payout path also
writes :charge/client/:charge/location alongside the key, so a charge's
scope and its owner can no longer disagree.
The guard is transitional and gets deleted with the legacy branch it
protects, at rollout step 9.
Rollout resequenced for the decision to leave duplicate client records
active: no deactivation, no "which record survives" call, and the risk
window closed by pausing the importer across deploy + migrate rather than
by removing one of the two writers. Both records converge to independent
stable histories once every key carries its owner.
Also from review:
- migrate-all! now collision-checks the charge pass like the other three
attributes instead of discovering a clash mid-run over 17M rows
- split-and-rekey-charges! logs progress every 200 batches; an
interrupted 19M-order run left no trail
- unscoped-report's docstring no longer promises a zero its :no-owner
column cannot reach; plan is named as the authoritative signal
- the rollout's pre-flight asked for a :collisions key plan never
returns, so it silently passed on every database
- the multi-parent gate sampled (take 400000 (all-order-ids db)), which
streams :aevt — ascending entity id — and so read the OLDEST 2% of
orders: 2019-12-31 to 2021-06-03, before any of the contention it
looks for. Now every order of the last year via the client+date index,
5,159,787 on the restored copy, reading 0
- the report claimed same-client pairs get copied once batches split
them. They do not, at any batch size; verified at batch-size 1 and now
pinned by a test
30 tests, 72 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three faults were leaving restaurant days out of balance — one in the
data, two in the arithmetic — plus a fourth that turned out to be a
missing-data problem and is deliberately left visible. Measured over ninety days on a restored
copy of production (210 clients, 18,900 client-days): 1,258 days out of
balance and $69,560.10 becomes 279 days and $7,790.54 — of which 171 are
not arithmetic faults at all, but days whose sales were never imported.
979 days repaired, none knocked out of balance, and not one
already-balanced day altered — verified line by line (category, side,
amount to the cent, account), not just on each day's bottom line.
THE DATA FAULT
Ten Square locations were configured against two client records each.
Sales orders scoped their identifier by client; refunds, card payments,
payouts and cash-drawer shifts used the bare Square id. Those attributes
are :db.unique/identity, so both clients' imports resolved to a single
entity and the last writer won — 3,387 refunds, 4,069 payouts and 2,628
cash-drawer shifts changed hands over time, across 19 client pairs of
which only 10 are visible in today's configuration.
Worse, one payment could belong to two orders. :sales-order/charges is
:db/isComponent, so removing a voided order cascaded into payments the
other client still needed.
Fixes: client-scope the four key schemes; look the record up under both
schemes so the change deploys before the migration finishes; and a
migration that gives every order its own payment. Run over the whole
database that is 19,040,785 orders walked, 9,100,314 payments re-keyed
and 200,027 copied, ending with 17,047,142 payments scoped, none left to
rename, none unscopable, and no payment owned by more than one order.
Idempotent and resumable; about thirteen minutes.
THE ARITHMETIC FAULTS
- Refunded tips stayed on the books. get-tip summed tips by joining
through :sales-order/charges, so a return-only order — no tender to
join through — contributed nothing while its reversal sat unread on
:sales-order/tip. Additive, not substitutive: where an order does have
a tender the tender is the correct source.
- Service charges were collected but never earned. Nothing read
:sales-order/service-charge. Now credited for Square orders only, both
signs, behind summary-service-charges.
The flag is off by default, so deploying this changes nothing until a
client is opted in. docs/2026-08-15-sales-summary-rollout-plan.md has the
steps.
WHAT IS DELIBERATELY NOT FIXED
156 of the 279 remaining days carry refunds on a record that recorded no
sales at all that day, and 132 of those fall before that client's first
ever order. The refunds are not theirs: ownership history shows a $35.35
refund dated 26 February belonging to NGDG that day and taken over by
NGDU on 12 August, flipping between the two several times a day. Across
nine records, 659 refunds worth $15,225.24 sit on a record dated before
its own first order — unscoped keys let whichever import ran last take
ownership.
A rule closing those days was written and measured (156 days, $4,820.19,
nothing broken) and then removed. An unbalanced day is the only visible
signal that a restaurant's sales are not being imported; balancing it
would remove the alarm and leave the fire. A comment and a test hold that
decision in place. Step 9 of the rollout plan is the real fix, and it
needs a business decision.
SUPPORTING
- Install schema attributes before the tuples that compose them. A tuple
in schema.edn is built from an attribute in cloud-migration-schema.edn,
so every test fixture died in setup — very likely why sales summaries
had no tests before this.
- Log each day's imbalance and its suspect lines.
- Bound the dirty-summary scan to one client: 1,321 ms to 5.6 ms.
- compare-sales-summaries lives in test/clj as auto-ap.tools.* — it is a
verification harness, not part of the running application. Its
docstring now warns that d/as-of cannot be used to compare summary
amounts: :ledger-mapped/amount, ledger-side and account are
:db/noHistory, so a recomputed summary reads back with its amounts
absent and looks like a legitimate balanced day.
26 tests, 62 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sales-summaries-v2 recomputes every dirty summary, but nothing set the dirty
flag on a schedule: mark-all-dirty was only ever called by hand from the
comment block, and the job was registered in neither server.clj's
INTEGREAT_JOB dispatch nor terraform/deploy.tf, so -main was dead code that
could never run in production. Summaries were only recalculated when someone
remembered to do it in the REPL, and POS data keeps arriving after a business
day closes, so a summary computed once on the day was routinely wrong and
stayed wrong.
Add a daily job that marks the trailing 7 days dirty and recomputes them,
leaving finished work alone. "Finished" is the condition the app already calls
Balanced -- debits equal credits and every line is mapped to an account. Since
that is derived rather than stored, a summary that later falls out of balance
is picked up again on the next run.
Extract the Balanced predicate into auto-ap.datomic.sales-summaries so the
grid's pill and the job share one definition, rather than a background job
requiring an SSR namespace. total-debits/total-credits resolve the ledger side
from either a plain keyword or the {:db/ident ...} map a pull returns, and
accepted? requires every item to declare a side: un-normalized pulled items
otherwise sum to 0.0 on both sides, read as balanced, and get skipped
silently and permanently.
Also fix sales-summaries-v2 destroying user-entered line items. It filtered
for :sales-summary-item/manual? to preserve them, but dirty-sales-summaries'
index-pull selector never fetched :sales-summary/items, so manual-items was
always empty. Because items is a component attribute upserted via
[:reset-rels ...], every recompute deleted the hand-entered lines -- often the
very lines that make a summary balance. Harmless while nothing ran on a
schedule; destructive the moment this does.
Register the job in the admin Background Jobs dropdown too, with a days
field: schedules are prod-only, so the admin page is the only way to run it
on staging.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Recategorizes 131 Sysco line-item descriptions per the client-reviewed
"Product Category Verification with codes" sheet: 96 existing rows re-coded
and 35 new rows appended (ids 1796-1830).
Root cause this addresses: get-line-account matches on exact description
string and silently defaults anything unmapped to 50000 Food Costs. Only 147
of the sheet's 321 reviewed rows coded the way the client expected. Note the
sheet's "change" column understates the work -- 20 of its 53 change rows are
no-ops (Paper -> Paper, confirming the gloves/liners/hairnets) and 3 were
already fixed in 38575aa5 / 7a0e256f, while 214,398 lines of movement come
from rows the client ticked as correct against a suggestion that already
differed from production.
Moves, replayed over all 1,022,732 DET lines in sysco-poller:
50000 -> 51500 Dry Goods 90,481 ln $5,215,011.82 105 clients
50000 -> 51450 Dressing & Sauce 58,191 ln $4,445,726.56 98
50000 -> 51400 Bread and Bun 37,705 ln $3,964,462.24 96
50000 -> 52000 Soft Beverage 36,244 ln $1,033,497.52 94
50000 -> 51200 Produce 6,519 ln $460,629.32 98
55000 -> 51500 Dry Goods 6,516 ln $259,750.60 97
50000 -> 74100 Cleaning Supplies 5,630 ln $205,233.19 98
55000 -> 74100 Cleaning Supplies 5,222 ln $125,946.32 99
50000 -> 51120 Chicken/Poultry 265 ln $42,482.50 8
50000 -> 51300 Dairy 36 ln $5,941.69 6
50000 -> 55000 Paperware 54 ln $1,751.22 18
54400 -> 51450 Dressing & Sauce 24 ln $1,413.26 1
total 246,887 ln $15,761,846.24
Only three source accounts are touched: 50000 and 55000 (the two silent
defaults) plus the single intended 54400 -> 51450 vinaigrette row. Nothing
else leaves a deliberately assigned account.
The 7 Misc Charges descriptions are deliberately left alone per Bryce,
including PICKLE CHIP KOSH 1/4 KK, which therefore stays at the 50000
default rather than moving to Produce as the sheet originally suggested.
Also corrects the PAPER & DISP fallback comment in sysco.clj: 440 of its 455
mapped descriptions point at 55000, not all 454. The 15 exceptions (foil
pans -> 51500, scour pads -> 74100) are mapped explicitly, so the
description map still wins ahead of the fallback. No logic changed.
Affects only clients with the code-sysco-items feature flag, and only at
import time -- already-imported invoices keep their existing splits.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Sysco importer codes each line item by exact description match against
resources/sysco_line_item_mapping.csv, silently defaulting to GL 50000 (Food
Costs) when the description is absent. Every new or renamed Sysco SKU
therefore leaks into Food Costs until someone hand-patches the CSV, which is
what 38575aa5 did for 34 descriptions.
Add a category-level fallback consulted after the description map and before
the 50000 default, enabled for PAPER & DISP only. The description mapping
still wins wherever it exists, so nothing already mapped changes.
PAPER & DISP is safe to generalize: all 454 mapped PAPER & DISP rows point at
55000, with no exceptions. Of the 852 distinct descriptions ever invoiced
under that category, only 3 resolved elsewhere, each because a row with a
different category shared the description and won the later-wins (into {}).
One of those, DESSERT CUP, was simply mis-categorized -- it is paper, and its
own lid (id 1782 LID DOME DESSERT CUP) was already 55000 -- so correct id 1772
to PAPER & DISP / 55000. The remaining two stay at 50000 on purpose, since
they are not paper: PAD SCRUB S-S 35 GRAM 1.25 OZ (SUPP & EQUIP) and TEST
STRIP SANITIZER QUAT (CHEMICAL/JANTRL).
Verified by replaying both changes over all 1,022,732 DET lines in the 56,010
CSVs under sysco-poller/: every resulting transition is 50000 -> 55000 (10,994
lines, $728,106.62). No line that already resolved to a non-default account
moved.
Note this only affects clients carrying the code-sysco-items feature flag, and
only on import -- already-imported invoices need a separate recode.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The statement no longer prints "Reel Produce" as text (only
orders@reelproduce.com), switched to MM/DD/YYYY dates, and moved the
invoice number into an "INV #..." transaction description, so no
template matched and the file fell through to the glimpse2 fallback.
Adds a QuickBooks-statement-style template (same shape as Suncrest /
Ocean Queen) keyed on reelproduce.com + Statement, placed after the
existing Reel Produce statement template so the old layout still wins.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rotation in 111eca41 updated the Plaid secret-key only in prod.edn,
leaving prod-background-worker.edn, prod-cloud-background-worker.edn, and
prod-cloud.edn on the old (now-invalidated) secret. Background worker jobs
loading those files failed with INVALID_API_KEYS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Sysco importer codes line items by exact-matching the item description
against resources/sysco_line_item_mapping.csv, falling back to GL 50000
(Food Costs) when no entry exists. On master, 8 of the paper-product
descriptions on recent invoices (e.g. BAG PAPER 250 CT, NAPKIN 2PLY INTR
FOLD 6.3X8.26, CONTAINER PAPER 4/110OZ NTG) were missing, so they
defaulted to 50000 instead of 55000 (Paper Costs).
Append the 34 curated mappings (Ids 1762-1795) covering these paper items
(-> 55000) plus the other new items from the same invoices, so they code
correctly on re-import.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Aligns debit/credit amounts to a right column with tabular-nums;
replaces the in-cell delta and balanced text with chip-style status
indicators; shortens the edit dialog and clarifies its totals/unbalanced
footer rows; gives manual line items a subtle accent so they're
distinguishable from auto-generated rows.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Completes the automatic sales summary pipeline end-to-end: the `sales-summaries-v2` job now calculates aggregate totals, preserves manual adjustments, and automatically posts balanced journal entries to the ledger.
## What Changed
**New Datomic transaction function** (`upsert-sales-summary-ledger`)
- Transforms detailed `sales-summary-item`s into aggregated `journal-entry` lines grouped by account and ledger side
- Handles the full upsert: posts a new journal entry for summaries with mapped accounts, or retracts the orphaned entry if items no longer qualify
**Enhanced `sales-summaries-v2` job**
- Calculates and stores 13 aggregate total attributes (card/cash/food-app/gift-card payments, refunds, fees, discounts, tax, tip, returns, unknown, net)
- Preserves manual items (`manual? true`) during recalculation — only auto-calculated items are replaced
**Ledger reconciliation**
- `reconcile-ledger` now queries for sales summaries missing journal entries and repairs them via `:upsert-sales-summary-ledger`, alongside existing invoice and transaction repairs
**Schema**
- Added 13 `total-*` attributes on `sales-summary` (all `db.type/double`, no history)
- Registered the new transaction function in `tx.clj` and `datomic.clj`
**Admin UI cleanup**
- Resolved "clientize" and HTMX `client-id` TODOs in the sales summaries admin page
- `new-summary-item` now correctly passes `client-id` via `hx-vals`
- Removed stale TODO comments and placeholder code
## Files Changed (8)
| File | Purpose |
|------|---------|
| `iol_ion/.../upsert_sales_summary_ledger.clj` | New Datomic tx function |
| `iol_ion/.../tx.clj` | Register new tx function |
| `resources/schema.edn` | 13 new `total-*` attributes |
| `src/.../datomic.clj` | Load new tx namespace |
| `src/.../jobs/sales_summaries.clj` | Aggregate totals + manual item preservation |
| `src/.../ledger.clj` | Sales summary repair in `reconcile-ledger` |
| `src/.../ssr/admin/sales_summaries.clj` | UI TODO cleanup |
| `docs/plans/...plan.md` | Implementation plan document |
Co-authored-by: Bryce <bryce@integreatconsult.com>
Reviewed-on: #5
Co-authored-by: Bryce <bryce@brycecovertoperations.com>
Co-committed-by: Bryce <bryce@brycecovertoperations.com>
- Add testing patterns section (Grid Page, Wizard, Permission Gates)
- Convert all behaviors to numbered tables with checkboxes
- Specify test strategy per behavior (Unit/Integration/UI)
- Group by feature area: Display, Filter, Sort, Pay Wizard, etc.
- Add cross-cutting Permissions and Lock Date tables
- Reduce from 496 to 403 lines while being more comprehensive
Add behavior documentation covering all SSR and legacy SPA pages:
- Testing strategy and type definitions (unit/integration/UI)
- Dashboard, Invoice, Payment, Transaction, Ledger pages
- Company/Settings, POS, Admin, Search, Auth pages
- Legacy SPA behavior docs (no UI tests until migrated)
- Edge cases, test data requirements, and dependencies per subsystem
Total: 3,600+ lines of behavior documentation to guide test authorship.
Validates existing template correctly parses multi-page invoice with:
- Invoice number 03882095
- Customer identifier NICK THE GREEK
- Account number 600 VISTA WAY
- Total of $946.24
