Ninety-Day Reconciliation

In one sentence: a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.

How the two figures above were produced. Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth 1,069 days and $67,897.05, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.

Most of what is left is not a balancing fault at all, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.

Five changes. The first three stop two clients from sharing a record; the last two record money that was being collected but not booked. Each is small — the difficulty was knowing which line to change, not writing it. A sixth was written and then removed; it is described at the end because the reasoning matters more than the code did.

Every imported record has an identifier the importer uses to decide "have I seen this before?". Sales orders already included the client; refunds, card payments, payouts and cash-drawer shifts did not, which is precisely why two clients could land on one record.

Applied at five places in the Square importer: order payments, refunds, payouts (twice — the record itself and the lookup that finds it) and cash-drawer shifts. ezCater orders already did this and needed no change.

This is the one that makes the change safe to deploy. The identifiers are unique keys, so the importer relies on "same id, same record". Rename them and the next import matches nothing — and would quietly create a second copy of every refund and payment in the system, leaving the originals orphaned. So the importer looks up the record explicitly, new name first, old name second, and writes to whichever it finds.

The result is pinned as the record's id on the way in, so the write lands on the existing row regardless of which name it currently carries. The proof this worked is a count that did not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was re-named, and afterwards there were still exactly 50,986 refunds, 144,688 payouts and 69,291 cash-drawer shifts — the same three figures as at the restore point. Had the fallback lookup been missing, each of these would have doubled instead.

The fallback also has to refuse. Reading the old name is what stops duplicates; reading anyone's old name is what creates them. Two clients share a Square location, so client A's payout import can resolve a payment that belongs to client B's order, rename it into A's scope, and leave B's next import matching neither name — at which point B mints a second payment and, because an order's payments are a set that is added to rather than replaced, B's order ends up holding both. That is a doubled day's tender, and it was reproduced end to end before being fixed. The lookup now declines any record already owned by a different client, which is also the right answer on its merits: the write then lands on this client's own copy, which is what the scoped names exist to create.

This is transitional. Once no legacy names remain, the fallback and the refusal are deleted together and the guarantee stops depending on either.

Renaming stops new collisions but does not undo old ones: a payment already shared by two orders is still one row with two owners. The migration walks each order's payments and, where another order has already claimed one, makes that order its own copy with the same amounts and points the order at the copy.

Run over the whole database that was 16,236,839 renamed and 500,438 copied, and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across every order of the last year. The record count rose by about 500,438 — the number of copies it reported making, which is the check that it created what it meant to and nothing else.

One subtlety worth recording, because it bit us: the Square id has to be recovered from the record's current owner rather than by trimming a fixed prefix. Client codes contain dashes — N-30003 — so a pattern cannot tell where the client name ends and the Square id begins. Getting this wrong scoped some records twice and doubled their tender.

Tips were summed by walking from the order to its payments. A refund-only order has no payment attached, so its negative tip was invisible. The fix adds those tips rather than replacing the calculation.

Adding rather than replacing is deliberate. Where an order does have a payment, the payment is the correct source: real orders exist whose payment carries a tip the order does not — an auto-gratuity recorded as a service charge, or a wallet tip missing from the order totals. Reading the order instead would have dropped those. Three tests hold this in place: the refund case must change, and the tendered and ordinary cases must not.

Nothing read the service-charge field at all. A new line credits it, for Square orders only and for negative amounts as well as positive.

Why the vendor test has two branches. ezCater service charges are commission the platform deducts from the restaurant, not money the diner hands over, so crediting them would make a day worse rather than better — hence the Square-only condition. But whole eras of Square orders carry no vendor field at all, and a test on vendor alone would silently credit nothing. The second branch falls back to the order's own identifier.

Why negatives matter. A returned catering fee arrives as a negative service charge and is already deducted from the day's returns; dropping negatives would lose the reversal. The line sits behind a per-client switch, off by default, so it can be turned on a few restaurants at a time.

Worth recording, because the arithmetic case for it is good and someone will propose it again. Where a day has refunds and no sales orders whatsoever, book a Returns debit equal to that day's refunds:

It works. Measured over the same ninety days it closed 171 days and $5,795.18, knocked nothing out of balance, and altered no already-balanced day — the guard makes it incapable of touching a day that traded.

It was removed anyway. Those days are not quiet days; they are days whose sales were never imported, and closing them removes the only visible sign of that. What is left in the code is a comment saying so and a test asserting the day stays out of balance, so the next person to notice the arithmetic finds the reasoning before they find the fix.

That last one is worth a sentence for engineers: transact-schema installed schema.edn then cloud-migration-schema.edn, but a composite tuple in the first file is built from an attribute in the second. Datomic will not create a tuple before its members exist, so every test fixture died in setup. It is very likely why sales summaries had no tests before this work.

The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates. All 18,900 client-day summaries in the window are included, whether or not the restaurant traded that day.

Deduplication is not a row in this table, and that is deliberate. Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that $66,589.75 is what the three arithmetic fixes are worth on their own, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.

Neither fix touched a day that was already correct. Across all 18,900 client-days no balanced day was knocked out of balance, and no balanced day had a single figure altered — 17,827 summaries came out byte-identical, and every one of the 1,073 that moved was already wrong.

That claim did not hold on the first attempt, and how it was recovered is the useful part. Measured before the historical backfill described below, six days broke — all of them a tip reversed on one record whose refund sat on its twin, so removing the un-reversed tip left the day short by exactly that amount. Replaying the window from Square gave both records their own copy of every refund, and all six closed. The fix was never wrong; it was reading half a transaction.

Service charges are by far the larger of the two fixes, moving $58,923.85 against the tip fix's $3,777.67.

That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.

The two fixes account for every day they moved, exactly. Adding up the untendered-tip and service-charge amounts across all 984 changed days leaves a residue of 0.0000000002. Nothing else moved those days; there is no unexplained remainder hiding a third effect, and the six that broke are accounted for by the same arithmetic as the 915 that healed.

Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.

The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY: a tip of −60.00 on an order with no payment attached at all.

In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.

Every step below was performed against a restored copy of the production database. Production itself was never touched.

The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added 500,438, matching the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)

The measurement above was taken with both client records of each pair left live, which is how this deploys. Nothing is deactivated and no business decision about which restaurant's history survives is needed. The risk that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why existing-id now refuses to resolve a record belonging to another client.

122 client-days out of 18,900, totalling $2,379.45 — and only 32 of those are above ten cents.

The shared-location records are now the clean part of the book. Three days between all twenty of them: NGBK and NGBR at $299.42 each on 2026-08-06, which is the Square tender-versus-order-total gap described below and not an attribution fault, and NGDA at $50.00, an auto-gratuity booked as a service charge. Before the backfill those same records carried 423 days and $18,508.39.

The other 119 days have not moved across any run of this analysis. Four separate rebuilds — different databases, different preparation, one with the duplicates deactivated and one without — all land on 119 days and $1,730.61, with the same five restaurants accounting for almost all of it:

Sixteen of the 122 are days a record had no sales imported at all, worth $1,227.65 — the fourth problem, still deliberately visible. The clusters on NGMV, NGPS and NGEB are unexplained and worth a look, though at under $650 across sixteen days they are no longer urgent.

Re-keying stops the two records fighting, but on its own it does not make them equal, and the difference is worth stating plainly because it decides whether the books close.

Orders were always duplicated; refunds never were. A sales order's identifier has always carried its client, so each of the two records built its own order history from the start. Refunds, payouts and cash-drawer shifts did not, so only ONE record holds each of them — whichever imported it last. The migration freezes that ownership rather than evening it out. The record left without them shows returns from its own orders and no refunds to set against them, and is out of balance by exactly the amount its twin is holding.

On 2026-05-11 both NGBK and NGBR held the same 221 orders. NGBK had no refunds; NGBR had two, worth $2,232.29; and NGBK's books were out by $2,232.29 to the cent. Across the whole database NGBK held 158,535 orders and five refunds.

This turned out to be a better answer than retiring the duplicate records. An earlier measurement that deactivated one record of each pair left 279 days and $7,790.54. Backfilling instead, with both records live, leaves 122 days and $2,379.45 — and it needs no business decision about which restaurant's history to abandon.

The importer understands both the old and new record names, so the change can be deployed before the renaming finishes. That tolerance is a bridge, not a destination — while any record still carries an unscoped name, two clients can land on it and the guarantee rests on a convention rather than on the data. So the renaming was run to completion and measured.

Every record in the database now carries its owner's name, and the migration proposes no further changes: asked what is left to do, it answers zero on all four record types. The 36 payouts are ones with no client or location recorded anywhere, on the record itself or on anything referring to it, so there is nothing to name them after.

Renaming had to be driven from orders, because a payment's rightful owner is whichever order refers to it — so completing it meant walking all 19,040,785 orders, not just the clients that look shared today. Nine client pairs contended in the past without sharing a location now, and a migration scoped to the current configuration would have missed every one of them.

A caution about how completeness is counted. 282,649 card payments carry no client attribute of their own — they are stubs the payout path creates, never referenced by an order. A gate that checks the attribute reports these as "no owner" and looks like a gap. They are not: their names are scoped, recovered from the deposit that holds them. The figures above are counted the harder way, by asking the migration what it would still change, which resolves each record's owner through whatever refers to it. Reading the attribute alone would have understated completeness by a quarter of a million records — and an early draft of this report did exactly that.

The problem this work exists to solve is gone: no payment answers to two orders, so the component relationship means what it says and deleting an order can no longer take another order's money with it.

Where Square splits one tender across two of a single client's own orders, the payment stays shared — at any batch size. Both orders compute the same name, so there is no second name for a copy to take, and a copy would double that client's takings for the day. The mechanism is worth stating precisely, because it is not obvious from reading: once the first order re-keys the payment it also writes the owner attributes in the same transaction, so a later order recovers the bare Square id from those, computes the name the payment already carries, and the guard (not= old new-key) drops the row before any copy decision is reached. Verified by running the migration at a batch size of one, which forces the two orders into separate batches: no copy is made.

An earlier draft of this report claimed the opposite — that such pairs would be copied once the batches split them — and flagged it as unmeasured risk to check before production. That was wrong, and it is recorded here rather than quietly deleted because it did real damage: an independent reviewer cited this paragraph as evidence and raised a defect that does not exist. A test now pins the behaviour at batch size one.

The guard on remove-voided-orders is still worth having regardless. It is cheap, and it makes the safety a property of the deletion rather than of the migration having been run first.

What follows from the gate reading zero. unscoped-report counts these figures on demand. Now that unscoped is zero across the board, the importer's understanding of the old name form can be removed — at which point two clients sharing a location becomes structurally incapable of producing a shared record, rather than prevented by a convention that a future import could quietly break. That removal is the one remaining step of this piece of work.

The comparison tool is committed as auto-ap.jobs.compare-sales-summaries. Unit tests: lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test.