diff --git a/docs/2026-08-15-remove-voided-orders-risk.md b/docs/2026-08-15-remove-voided-orders-risk.md new file mode 100644 index 00000000..4c19716f --- /dev/null +++ b/docs/2026-08-15-remove-voided-orders-risk.md @@ -0,0 +1,115 @@ +--- +title: remove-voided-orders can delete another client's payments +type: risk +date: 2026-08-15 +status: open — decide before merging the re-key +--- + +# `remove-voided-orders` can delete another client's payments + +Measured on the restored backup, 2026-08-15. This risk is **pre-existing** — nothing in the +sales-summary work created it — but it is live right now, and the re-key work touches the same +data, so it should be understood before merging. + +## The mechanism, in four steps + +**1. Charges are component entities of an order.** + +```clojure +;; resources/schema.edn +{:db/ident :sales-order/charges + :db/valueType :db.type/ref + :db/isComponent true ;; <- this is the load-bearing bit + :db/cardinality :db.cardinality/many} +``` + +`:db/isComponent true` tells Datomic the charges *belong to* the order. It is what lets you +transact an order with its tenders nested inside, and it means the charges have no independent +existence as far as Datomic is concerned. + +**2. `retractEntity` on a component parent deletes the children too.** + +That is the documented behaviour of `:db/retractEntity`: it recursively retracts component +values. `square.core3/remove-voided-orders` ends with exactly that: + +```clojure +(s/map (fn [[o]] + [[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]])) +``` + +It asks Square for the last 10 days of orders, keeps the ones that should *not* be imported — +voided and cancelled orders — and retracts any of those we already stored. That is correct and +desirable on its own: a voided order should not sit in the books. + +**3. But one charge can be shared by two orders.** + +When two clients are configured on the same Square location, both import the same Square data. +Order keys embed the client, so each client gets its own order entity. Charge keys did **not** +embed the client, and `:charge/external-id` is `:db.unique/identity`, so both clients' orders +resolved to *the same charge entity*: + +``` +NGCD order 17592395490523 ──┐ + ├──> charge 17592490524 ← one entity, two parents +NGCC order 17592395511722 ──┘ +``` + +**4. So retracting one order deletes a charge the other order still points at.** + +Datomic sees a component and removes it. The surviving order keeps its line items — its sales — +but its tender is gone. The day then shows revenue with no payment against it, the summary goes +out of balance, and the payment is gone from the current database value. (History retains it, so +it is recoverable by someone who knows to look, but nothing in the app will show it again.) + +## How exposed are we + +Measured over the 10 contended clients across 2026-07-13 → 08-14: + +| | | +|---|---| +| Charges examined | 56,829 | +| **Referenced by more than one order** | **35,870 (63%)** | + +So this is not a theoretical corner. Roughly two thirds of the charges in that population have +two parents, and any voided order among them takes a charge down with it. + +The exposure window for *new* damage is the rolling 10 days `remove-voided-orders` searches, but +the shared charges themselves span the whole period the locations were double-configured. + +## What changes after Phase 0 and the re-key, and what doesn't + +- **Phase 0 (done on the restore)** stops new sharing: only one client per location imports now, + so no new order pairs form. +- **The re-key (done for refunds and the contended clients' charges)** makes sharing structurally + impossible going forward, because a charge key now contains the client code. +- **Neither retroactively splits the 35,870 charges that are already shared.** They still have two + parents. Until they are split, `remove-voided-orders` remains capable of deleting a payment + belonging to the other client. + +This is why plan §3.3 forbids retracting anything — including any historical cleanup of the +duplicate clients' data — until a verification query shows zero charges with more than one parent. + +## Options, roughly in order of preference + +1. **Split the shared charges, then let removal run normally.** Re-import the affected window now + that keys are client-scoped, so each client creates its own charge entity. This reuses the + import path rather than hand-constructing component entities. Verify with a query for charges + having more than one referencing order; it must reach zero. + +2. **Guard the retraction.** Before retracting an order, check whether any of its charges are + referenced by another order; detach those (retract the `:sales-order/charges` ref rather than + the charge) and retract the rest. Small, contained change, and it makes the operation safe + regardless of what shape the data is in — worth doing on its own merits even after a split. + +3. **Do nothing and accept it.** Only defensible once every location has a single client *and* + the historical shared charges are gone. Not true today. + +## What I did about it during the validation run + +I ran the import on the restore with `remove-voided-orders` **skipped**, and ran the other steps +(`upsert-locations`, `upsert`, `upsert-payouts`, `upsert-refunds`) normally. That kept the +validation faithful to how the import behaves without risking silent payment loss in the data +the measurements were about to be taken from. + +**Nothing in the production system has been changed.** This note is about a risk that already +exists there. diff --git a/docs/2026-08-15-sales-summary-reconciliation.html b/docs/2026-08-15-sales-summary-reconciliation.html new file mode 100644 index 00000000..2399a1de --- /dev/null +++ b/docs/2026-08-15-sales-summary-reconciliation.html @@ -0,0 +1,682 @@ +
Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — and a fourth, found late, that is a missing-data problem wearing a balancing problem's clothes. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.
+ +In one sentence: a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.
+How the two figures above were produced. Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth 979 days and $61,769.56, not the larger number a stale baseline would have shown.
+Most of what is left is not a balancing fault at all, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.
+For the business: ten restaurant locations were set up twice in the system, as two separate clients. Both were importing from Square. Because the two records competed for the same payments and refunds, a refund would belong to one client for twenty minutes, then the other — so a day's books could gain or lose a refund depending on nothing but timing. On 2026-07-23 one client's summary was missing a $71.94 refund entirely, and was out of balance by exactly that amount.
+technical Sales orders scoped their identifier by client (square/order/<code>-<loc>-<id>), but refunds, card charges, payouts and cash-drawer shifts did not — they used the bare Square id. Those attributes are :db.unique/identity, so both clients' imports resolved to a single entity and the last writer won.
Reading ownership out of the database's own history, this had actually happened to 3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts. And it has involved 19 client pairs, of which only 10 are visible in today's configuration — nine more contended in the past and the configuration has since changed, so no point-in-time check would find them.
+For the business: the same collision meant a single card payment could be attached to both clients' copies of an order. That is worse than untidy. The nightly import removes orders Square reports as voided, and removing an order also removes its payments — so cancelling one client's order could silently delete the other client's payment, leaving a day showing sales with no money against them.
+technical :sales-order/charges is declared :db/isComponent true, so [:db/retractEntity <order>] cascades into the charges. In a 20,000-order sample of the affected clients, 11,469 charges had two parent orders. This is why remove-voided-orders was left switched off during testing.
For the business: two arithmetic faults, both of which overstated or understated a day.
+technical get-tip summed tips by joining through :sales-order/charges, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on :sales-order/tip. Nothing at all read :sales-order/service-charge.
This is why the duplicated restaurants looked so much worse than everyone else. Of the days still failing after the first three fixes, 155 of the 158 on duplicate records had no sales orders at all — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.
+The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded any order shows 132 of 156 fall before that client had a single order in the system — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.
+Where the refunds came from. Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to NGDG that same day, and was taken over by NGDU on 12 August. Others flip between the two records several times a day across 12–15 August. NGDU's first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.
+Across the nine records, 659 refunds worth $15,225.24 sit on a record dated before that record's first order. Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.
+Why it is deliberately left out of balance. The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and it closes 156 of the 279 remaining days and $4,820.19. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.
+technical get-returns sums :sales-order/returns over orders scanned for the date. With no orders the sum is nil and no Returns line is written, while get-refund-items still credits Card Refunds from the sales-refund records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.
Five changes. The first three stop two clients from sharing a record; the last two record + money that was being collected but not booked. Each is small — the difficulty was knowing + which line to change, not writing it. A sixth was written and then removed; it is described + at the end because the reasoning matters more than the code did.
+Every imported record has an identifier the importer uses to decide "have I seen this + before?". Sales orders already included the client; refunds, card payments, payouts and + cash-drawer shifts did not, which is precisely why two clients could land on one record.
+;; before — the bare Square id, identical for both clients +(str "square/refund/" (:id r)) ;; square/refund/NOkQOTIiJULWN6… + +;; after +(scoped-key "square/refund/" client location (:id r)) +;; square/refund/NGCD-CD-NOkQOTIiJULWN6… + +(defn scoped-key [prefix client location id] + (str prefix (:client/code client) "-" + (:square-location/client-location location) "-" id))+
Applied at five places in the Square importer: order payments, refunds, payouts (twice — + the record itself and the lookup that finds it) and cash-drawer shifts. ezCater orders + already did this and needed no change.
+This is the one that makes the change safe to deploy. The identifiers are unique keys, so + the importer relies on "same id, same record". Rename them and the next import matches + nothing — and would quietly create a second copy of every refund and payment in the + system, leaving the originals orphaned. So the importer looks up the record explicitly, + new name first, old name second, and writes to whichever it finds.
+(defn existing-id [db attr prefix client location id] + (when id + (or (dc/entid db [attr (scoped-key prefix client location id)]) ;; new scheme + (dc/entid db [attr (str prefix id)])))) ;; legacy scheme+
The result is pinned as the record's id on the way in, so the write lands on the existing + row regardless of which name it currently carries. The proof this worked is a count that did + not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was + re-named, and afterwards there were still exactly 144,688 payouts and 69,291 cash-drawer + shifts — the same figures as at the restore point. Refunds went from 50,986 to 51,986, + and all 1,000 of those came from the live Square import run afterwards, not from the renaming. + Had the fallback lookup been missing, each of these would have doubled instead.
+Renaming stops new collisions but does not undo old ones: a payment already shared + by two orders is still one row with two owners. The migration walks each order's payments and, + where another order has already claimed one, makes that order its own copy with the same + amounts and points the order at the copy.
+;; for each order, for each of its payments: +:keep → first order to claim it; rename in place +:clone → copy type, total, tip, tax, date, processor, note, receipt link + set the copy's client and location to this order's + retract this order's link to the shared payment + link it to the copy instead+
Run over the whole database that was 9,100,314 renamed and 200,027 copied, + and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across + 400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it + reported making, which is the check that it created what it meant to and nothing else.
+One subtlety worth recording, because it bit us: the Square id has to be recovered from the + record's current owner rather than by trimming a fixed prefix. Client codes contain dashes — + N-30003 — so a pattern cannot tell where the client name ends and the + Square id begins. Getting this wrong scoped some records twice and doubled their tender.
+Tips were summed by walking from the order to its payments. A refund-only order has no + payment attached, so its negative tip was invisible. The fix adds those tips rather than + replacing the calculation.
+;; before +:ledger-mapped/amount (tendered-tip c date) + +;; after +:ledger-mapped/amount (+ (tendered-tip c date) + (untendered-tip c date)) + +;; untendered-tip — tips on orders with no payment attached +[?e :sales-order/tip ?tip] +(not [?e :sales-order/charges])+
Adding rather than replacing is deliberate. Where an order does have a + payment, the payment is the correct source: real orders exist whose payment carries a tip the + order does not — an auto-gratuity recorded as a service charge, or a wallet tip missing from + the order totals. Reading the order instead would have dropped those. Three tests hold this + in place: the refund case must change, and the tendered and ordinary cases must not.
+Nothing read the service-charge field at all. A new line credits it, for Square orders only + and for negative amounts as well as positive.
+[?e :sales-order/service-charge ?service-charge] +(or-join [?e] + [?e :sales-order/vendor :vendor/ccp-square] + (and (not [?e :sales-order/vendor]) + [?e :sales-order/external-id ?external-id] + [(clojure.string/starts-with? ?external-id "square/order/")]))+
Why the vendor test has two branches. ezCater service charges are commission + the platform deducts from the restaurant, not money the diner hands over, so crediting them + would make a day worse rather than better — hence the Square-only condition. But whole eras of + Square orders carry no vendor field at all, and a test on vendor alone would silently credit + nothing. The second branch falls back to the order's own identifier.
+Why negatives matter. A returned catering fee arrives as a negative service + charge and is already deducted from the day's returns; dropping negatives would lose the + reversal. The line sits behind a per-client switch, off by default, so it can be turned on a + few restaurants at a time.
+Worth recording, because the arithmetic case for it is good and someone will propose it
+ again. Where a day has refunds and no sales orders whatsoever, book a Returns
+ debit equal to that day's refunds:
(defn- refund-only-returns [c date] + (when-not (traded? c date) + (let [amount (refunded-total c date)] + (when-not (zero? amount) amount))))+
It works. Measured over the same ninety days it closed 156 days and $4,820.19, + knocked nothing out of balance, and altered no already-balanced day — the guard makes it + incapable of touching a day that traded.
+It was removed anyway. Those days are not quiet days; they are days whose sales were never + imported, and closing them removes the only visible sign of that. What is left in the code is + a comment saying so and a test asserting the day stays out of balance, so the next + person to notice the arithmetic finds the reasoning before they find the fix.
+| Change | Why |
|---|---|
| Log each day's imbalance and its suspect lines | an out-of-balance day was only visible by opening the screen; now it can be queried |
| Stop the dirty-summary scan at the client boundary | it read every later client's summaries too — 1,321 ms to 5.6 ms per client |
| Split the recompute driver into a per-client function | lets a backfill spread clients across threads instead of grinding one at a time |
| Install schema attributes before the tuples that compose them | the test suite could not build an empty database at all, so no test could run |
That last one is worth a sentence for engineers: transact-schema installed
+ schema.edn then cloud-migration-schema.edn, but a composite tuple in the first file is built
+ from an attribute in the second. Datomic will not create a tuple before its members exist, so
+ every test fixture died in setup. It is very likely why sales summaries had no tests before
+ this work.
The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates. All 18,900 client-day summaries in the window are included, whether or not the restaurant traded that day.
+| Stage | Days out of balance | Clean | Total variance |
|---|---|---|---|
| Today's calculation, ninety days re-run | 1,258 | 93.34% | $69,560.10 |
| + refunded tips | 971 | 94.86% | $66,414.39 |
| + service charges | 279 | 98.52% | $7,790.54 |
Deduplication is not a row in this table, and that is deliberate. Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that $66,589.75 is what the three arithmetic fixes are worth on their own, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.
+| Change | Unchanged | Into balance | Out of balance | Balanced days altered | Money moved |
|---|---|---|---|---|---|
| Refunded tips | 18,590 | 287 | 0 | 0 | $3,712.67 |
| Service charges | 18,208 | 692 | 0 | 0 | $58,349.85 |
| Both, end to end | 17,916 | 979 | 0 | 0 | $61,769.56 |
Neither fix touched a day that was already correct. Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered — 17,916 summaries came out byte-identical, and every one of the 984 that moved was already wrong. Service charges are by far the larger of the two, moving $58,349.85 against the tip fix's $3,712.67.
+That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.
+The two fixes account for the repair exactly. Adding up the untendered-tip and service-charge amounts for the 979 repaired days leaves a residue of 0.0000000013 against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a third effect.
+Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.
+| Line | Before | After |
|---|---|---|
| Tip | 482.94 | 422.94 |
| Card Refunds | 60.00 | 60.00 |
| Total money taken | 10,094.81 | 10,094.81 |
| Total money earned | 10,154.81 | 10,094.81 |
| Out of balance by | −60.00 | 0.00 |
The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY: a tip of −60.00 on an order with no payment attached at all.
+| Line | Before | After |
|---|---|---|
| Service Charges | not shown | 427.10 |
| Card Payments | 4,975.89 | 4,975.89 |
| Total money taken | 7,777.20 | 7,777.20 |
| Total money earned | 7,350.10 | 7,777.20 |
| Out of balance by | +427.10 | 0.00 |
| Client | Date | Line | Before | After | Day closed |
|---|---|---|---|---|---|
| NGPA | 2026-06-04 | Service Charges | not shown | 1,344.86 | +1,344.86 → 0 |
| NTPT | 2026-08-06 | Service Charges | not shown | 427.10 | +427.10 → 0 |
| N-30003 | 2026-05-27 | Service Charges | not shown | 405.83 | +405.83 → 0 |
| NGFL | 2026-05-19 | Tip | 238.46 | 70.42 | −168.04 → 0 |
| NGMI | 2026-07-09 | Tip | 230.01 | 80.01 | −150.00 → 0 |
| NGVA | 2026-07-03 | Tip | 152.66 | 40.12 | −112.54 → 0 |
In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.
+Every step below was performed against a restored copy of the production database. Production itself was never touched.
+| Step | Result |
|---|---|
| Deactivate the duplicate client at each shared location | 10 locations · shared locations remaining: 0 |
| Walk every order in the database | 19,040,785 orders |
| Give every order its own payment record | 9,100,314 re-keyed · 200,027 copied |
| Payments owned by two orders | 0 across 400,000 orders checked |
| Client-scope refunds, payouts and cash-drawer shifts | counts unchanged · 0 collisions |
| Live Square import afterwards | 0 orders with duplicated payment · 0 shared payments |
| Ownership changes after the change | 0 refunds · 0 payouts · 0 shifts |
The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.
+Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. At the point the two runs are comparable — before the refund-only fix below, which the earlier run did not have — the end state matched to the cent: 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.
+The first attempt at copying shared payments derived each payment's Square identifier by stripping a fixed prefix. That is right the first time a payment is seen, but once it has been re-keyed to one client, a second order meeting it later read the already-scoped key as the identifier and scoped it twice — NGCD-CD-NGCC-CC-<id>. The importer then created a fresh payment, doubling the tender on five clients by $3,000–$7,000 each. It was caught because the totals were absurd, not because the code looked wrong. The fix recovers the scope from the record itself; client codes contain dashes, so it cannot be done by pattern. A test now runs the step one order at a time, which is the arrangement that exposes it.
+279 client-days out of 18,900, totalling $7,790.54. Most of it is not a balancing fault, and the split is the point:
+| What the remainder actually is | Days | Variance |
|---|---|---|
| Sales never imported — refunds on a record with no orders at all that day | 156 | $4,988.53 |
| Processing fee on a day with no trading — the same shape, from the payout side | 15 | $974.99 |
| Real trading days with genuine discrepancies | 108 | $1,995.36 |
Two thirds of what is left is a data-completeness problem, not an arithmetic one, and it is being left visible on purpose. Of those 156 days, 132 fall before their client had recorded a single order — they are periods with no sales in the system at all. The nine records involved hold 659 refunds worth $15,225.24 dated before their own first order.
+The 15 fee-only days are the same story from the payout side: a processing fee landing on a day with no trading. Closing them needs the payout modelled, not another rule in the summary.
+| Of the 108 real trading days | Days | Variance | What it is |
|---|---|---|---|
| NGBR | 1 | $299.42 | Square recorded $6,358.99 of tender against $6,059.57 of order totals on 2026-08-06 — the gap itself, not a summary fault |
| NGBK | 1 | $299.42 | |
| NGDL | 1 | $810.24 | an order on one record whose payment is on the twin — the mirror image of the refund case |
| NGMV | 5 | $259.38 | late May, undiagnosed |
| NGEB | 4 | $199.09 | ezCater fee treatment — an open question |
| everyone else | 96 | $127.81 | mostly till rounding — pennies a day |
The clusters on NGMV and NGEB are unexplained and worth a look. They are visible only because the window is ninety days; a thirty-day view does not reach them.
+The importer understands both the old and new record names, so the change can be deployed + before the renaming finishes. That tolerance is a bridge, not a destination — while any + record still carries an unscoped name, two clients can land on it and the guarantee rests on + a convention rather than on the data. So the renaming was run to completion and measured.
+| Record type | Total | Client-scoped | Still to rename | Cannot be scoped |
|---|---|---|---|---|
| Card payments | 17,047,142 | 17,047,142 | 0 | 0 |
| Refunds | 51,986 | 51,986 | 0 | 0 |
| Payouts | 144,688 | 144,652 | 0 | 36 |
| Cash-drawer shifts | 69,291 | 69,291 | 0 | 0 |
Every record in the database now carries its owner's name, and the migration proposes no + further changes: asked what is left to do, it answers zero on all four record types. The 36 + payouts are ones with no client or location recorded anywhere, on the record itself or on + anything referring to it, so there is nothing to name them after.
+Renaming had to be driven from orders, because a payment's rightful owner is whichever + order refers to it — so completing it meant walking all 19,040,785 orders, not + just the clients that look shared today. Nine client pairs contended in the past without + sharing a location now, and a migration scoped to the current configuration would have missed + every one of them.
+A caution about how completeness is counted. 282,649 card payments carry no + client attribute of their own — they are stubs the payout path creates, never referenced by an + order. A gate that checks the attribute reports these as "no owner" and looks like a gap. They + are not: their names are scoped, recovered from the deposit that holds them. The figures above + are counted the harder way, by asking the migration what it would still change, which resolves + each record's owner through whatever refers to it. Reading the attribute alone would have + understated completeness by a quarter of a million records — and an early draft of this report + did exactly that.
+| Shared payments after the migration | Count | Meaning |
|---|---|---|
| Owned by more than one order | 0 | whether the orders belong to different clients or the same one |
| checked across | 400,000 orders | spread through the whole database |
The problem this work exists to solve is gone: no payment answers to two orders, so the + component relationship means what it says and deleting an order can no longer take another + order's money with it.
+One behaviour changed when the pass was run over everything, and it is worth + recording. Where Square splits one tender across two of a single client's own orders, + the earlier design left the payment shared on purpose — both orders compute the same name, so + there is no second name for a copy to take. That rule only holds for two orders processed in + the same batch. Run across nineteen million orders in batches of two thousand, such pairs + almost always fall in different batches and the second order now takes a copy. Inside the + ninety-day window this changed nothing measurable: the recompute after the database-wide pass + matched the one before it to the cent. Outside the window it has not been measured, and it + should be before this runs against production.
+The guard on remove-voided-orders is still worth having regardless. It is
+ cheap, and it makes the safety a property of the deletion rather than of the migration having
+ been run first.
After the complete pass, asking the migration what it would change next returns + nothing — 17,047,142 payments examined, none to rename, none unscopable. A + record that already carries the right name is left untouched, so the migration can be stopped, + resumed, or repeated without consequence.
+Its speed is worth a note for whoever schedules it: the whole nineteen million orders were + walked in about thirteen minutes. An earlier attempt appeared to be + transactor-bound and was projected at two days, which is why a previous run narrowed it to the + analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process + driving the migration — freeing held memory took an unrelated recompute from 17 client-days a + minute to 4,515. Nothing about the database or the transactor needed to change.
+What follows from the gate reading zero. unscoped-report counts
+ these figures on demand. Now that unscoped is zero across the board, the importer's
+ understanding of the old name form can be removed — at which point two clients sharing a
+ location becomes structurally incapable of producing a shared record, rather than prevented by
+ a convention that a future import could quietly break. That removal is the one remaining step
+ of this piece of work.
| Item | Who decides | Why it matters |
|---|---|---|
| Which client record survives at each shared location | the business | the newer record generally has no history before the split, so keeping it loses years of the location's books |
| Which revenue account service charges post to | accounting | currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances |
| 659 refunds on records that have no sales for them | the business, then engineering | the top open item. $15,225.24 dated before the holding record's own first order. Either the missing sales get imported, or the refunds move to the record that has them — but the books cannot close until one of the two happens |
| Whether to correct records the wrong client already owns | the business | the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared |
remove-voided-orders | engineering | safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes |
The production backup had not written a restore point since 2025-03-10 — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.
+The database server was sized for a toy dataset: a 2 GB cache against 27 GB of data. Worth checking what production is set to.
+Slowness here was misdiagnosed twice, in the same direction. Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in thirteen minutes. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.
+;; the restored database, untouched production as of 2026-08-14 22:52 +(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore")) + +;; the two orders behind the worked examples +(d/pull (d/db conn) '[*] [:sales-order/external-id + "square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"]) +(d/pull (d/db conn) '[*] [:sales-order/external-id + "square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"]) + +;; the gate: no payment may have two parent orders +(rk/charges-with-multiple-parents (d/db conn) orders) ;; => 0 + +;; ownership history — which records ever changed client +(->> (d/datoms (d/history (d/db conn)) :aevt :sales-refund/client) + (filter :added) + (reduce (fn [m d] (update m (:e d) (fnil conj #{}) (:v d))) {}) + (filter (fn [[_ owners]] (> (count owners) 1))) + count)+
The comparison tool is committed as auto-ap.jobs.compare-sales-summaries. Unit tests: lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test.
as-of — a correction to how this was measured
+ The obvious way to audit a recompute is to read the database at a point before it and diff:
+ Datomic keeps every past value, so no snapshot is needed. That is what
+ compare-sales-summaries was built to do, and for summary amounts it does not work.
+ :ledger-mapped/amount, :ledger-mapped/ledger-side and
+ :ledger-mapped/account are all declared :db/noHistory true, so
+ superseded values are discarded rather than retained. A historical read of a summary that has
+ since been recomputed can return its lines with the categories intact and the amounts simply
+ absent — which reads as a legitimate all-zero summary, not as an error.
Every figure in this report is therefore taken from a live read of the database immediately + after each pass, captured and stored outside it, and the before/after comparison is done + between those two captures. No historical read is involved anywhere in the numbers above. The + tool remains useful for categories and for which days changed; its docstring overstates what it + can recover, and that is worth correcting before someone relies on it for amounts.
+