Compare commits
6 Commits
worktree-s
...
sales-summ
| Author | SHA1 | Date | |
|---|---|---|---|
| 79a4b457b0 | |||
| 1558851c18 | |||
| 4a1817711d | |||
| 967e77e443 | |||
| 10d0d01b82 | |||
| 57a84dae11 |
@@ -1,393 +0,0 @@
|
||||
---
|
||||
title: Sales Summary Balance — validation against a restored production backup
|
||||
type: validation
|
||||
date: 2026-08-15
|
||||
status: measured
|
||||
---
|
||||
|
||||
# Sales Summary Balance — validation against a restored production backup
|
||||
|
||||
Measures the two calculation fixes (R1 reversed tips, R2 Square service charges) against real
|
||||
production data, and checks them against the predictions in
|
||||
[`docs/plans/2026-08-13-fix-sales-summary-balance-plan.md`](plans/2026-08-13-fix-sales-summary-balance-plan.md).
|
||||
|
||||
**Headline: the fixes behave as predicted.** R2 clears 307 client-days with zero regressions.
|
||||
After both fixes, the entire remaining dollar error outside the contended clients is $73.98,
|
||||
and all but six client-days of it is sub-10¢ register rounding.
|
||||
|
||||
---
|
||||
|
||||
## 1. What was measured against
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Database | `datomic:dev://localhost:4337/integreat-prod-restore` |
|
||||
| Restored from | `s3://integreat-backups/`, restore point **209608347** |
|
||||
| Basis-t | 209608347 |
|
||||
| Newest transaction | **2026-08-14 22:52 local** |
|
||||
| Clients | 210 (106 with orders in the window) |
|
||||
| Refunds / charges / orders | 50,986 / 16,545,495 / 19,039,575 |
|
||||
| Window | 2026-07-13 → 2026-08-14 |
|
||||
| Population | **3,342 client-days** |
|
||||
|
||||
The earlier restore in this directory was **stale** — the backup had written no root since
|
||||
2025-03-10 even though segments kept uploading daily through 2026-07-12. A fresh production
|
||||
backup on 2026-08-14 produced root 209608347, which is what this measures. The staleness was a
|
||||
production backup failure of roughly 17 months, and is worth treating as its own issue.
|
||||
|
||||
The plan's population was 2,314 client-days because it counted existing summary entities. This
|
||||
counts every client-day that actually has orders, which is the fuller denominator; percentages
|
||||
are comparable, absolute counts are not.
|
||||
|
||||
## 2. Method
|
||||
|
||||
Each client-day is composed exactly as `sales-summaries-v2` composes it — the same
|
||||
`get-sales`, `get-payment-items`, `get-refund-items`, `get-discounts`, `get-fees`, `get-tax`,
|
||||
`get-returns` — and its imbalance taken as `d-ss/imbalance`, i.e. debits minus credits. Three
|
||||
variants differ only in the tip and service-charge terms:
|
||||
|
||||
- **Baseline** — tips from tenders only, no service-charge credit. What production does today.
|
||||
- **R1** — tips additive: tendered tips plus tips on orders carrying no tender.
|
||||
- **R2** — R1 plus Square service charges, both signs.
|
||||
|
||||
Nothing was transacted. A day counts as off when `|imbalance| >= $0.005`, i.e. it does not
|
||||
round to zero.
|
||||
|
||||
## 3. Results
|
||||
|
||||
| Stage | Days off | Clean | Total \|Δ\| |
|
||||
|---|---|---|---|
|
||||
| Baseline (production today) | 569 | 82.97% | $32,921.40 |
|
||||
| + R1 additive tip | 493 | 85.25% | $32,437.67 |
|
||||
| **+ R2 Square service charges** | **186** | **94.43%** | **$5,166.49** |
|
||||
|
||||
The plan predicted 82.7% clean at baseline including duplicates. Measured: **82.97%**.
|
||||
|
||||
### Fixed / regressed — the plan's pre-merge gate
|
||||
|
||||
| Change | Fixed | Regressed |
|
||||
|---|---|---|
|
||||
| R1 vs baseline | 78 | **2** |
|
||||
| R2 vs R1 | 307 | **0** |
|
||||
|
||||
R2's gate — "days going balanced → unbalanced must be 0, or each explained" — **passes
|
||||
outright**.
|
||||
|
||||
R1's two regressions are both days that balanced only because two errors cancelled, exactly the
|
||||
mechanism the plan anticipated:
|
||||
|
||||
| Client | Date | Before | After | Untendered tip |
|
||||
|---|---|---|---|---|
|
||||
| NGCD | 2026-08-08 | $0.00 | +$11.80 | −$11.80 |
|
||||
| NGDG | 2026-08-05 | $0.00 | +$10.00 | −$10.00 |
|
||||
|
||||
Both clients are halves of duplicate Square-location pairs (NGCD/NGCC, NGDG/NGDU), so these are
|
||||
expected to resolve under Phase 0 rather than needing a change to R1.
|
||||
|
||||
## 4. Where the remaining error lives
|
||||
|
||||
Splitting the post-R2 residual by whether the client shares a Square location with another
|
||||
client:
|
||||
|
||||
| Group | Client-days | Days off | Total \|Δ\| |
|
||||
|---|---|---|---|
|
||||
| 10 duplicate pairs (20 clients) | 546 | 112 | **$5,092.51** |
|
||||
| Everyone else (96 clients) | 2,796 | 74 | **$73.98** |
|
||||
|
||||
**98.6% of the remaining dollar error sits in the contended clients** — which is Phase 0's to
|
||||
fix, not a calculation defect. This is the plan's central claim, and it holds.
|
||||
|
||||
### The $73.98 residual, in full
|
||||
|
||||
| Client | Days | Total \|Δ\| | |
|
||||
|---|---|---|---|
|
||||
| PNSP | 30 | $0.91 | register rounding |
|
||||
| PNLP | 26 | $0.77 | register rounding |
|
||||
| GLAD | 12 | $0.20 | register rounding |
|
||||
| NGEB | 2 | $45.00 | ezCater fee semantics (plan §15.4) |
|
||||
| NGPS | 1 | $9.62 | unexplained |
|
||||
| NGGB | 1 | $7.59 | partial final day |
|
||||
| NGSC | 1 | $5.02 | partial final day |
|
||||
| NGTY | 1 | $4.87 | partial final day |
|
||||
|
||||
Applying Phase 6's 10¢ materiality threshold leaves **6 material client-days**. The largest
|
||||
sub-threshold day is **9.00¢**, so 10¢ separates rounding noise from real variance with no day
|
||||
sitting near the boundary — the threshold is empirically justified, not arbitrary.
|
||||
|
||||
Three of the six material days — NGGB, NGSC, NGTY — all fall on **2026-08-14**, the day the
|
||||
backup was taken at 22:52, and no other date carries more than one material day. They are
|
||||
partial-day artifacts of the backup cut, not defects.
|
||||
|
||||
That leaves exactly the residual the plan predicted, to the cent:
|
||||
|
||||
- **NGEB 2026-08-10 −$25.00** and **NGEB 2026-07-29 −$20.00** — predicted
|
||||
- **NGPS 2026-08-12 +$9.62** — predicted, still unexplained
|
||||
|
||||
## 5. Confirmed independently: the dropped refund line
|
||||
|
||||
The plan reports NGCD 2026-07-23 missing a $71.94 refund line because the refund belonged to
|
||||
NGCC at recompute time. That client-day measures **+$71.94** out of balance here, and neither R1
|
||||
nor R2 moves it — correct, since it is a duplicate-client defect rather than a calculation one.
|
||||
|
||||
## 6. Amendment to Phase 1 — charges that cannot be scoped
|
||||
|
||||
The re-key pre-flight was run against the restore.
|
||||
|
||||
**Refunds — clean.** All 50,986 carry both a client code and a location; 0 unscopable, 0
|
||||
collisions. Collisions are in fact structurally impossible: the old key is `:db.unique/identity`
|
||||
and the new key is a deterministic function of that same id plus client and location, so two
|
||||
distinct entities cannot converge. The run confirms the reasoning.
|
||||
|
||||
**Charges — not clean.** Of 16,545,495 charges, **2,122,161 (12.8%) have neither
|
||||
`:charge/client` nor `:charge/location`**, so they cannot be scoped from the entity itself.
|
||||
A 5,000-entity sample profiles them as:
|
||||
|
||||
| | Count | Share |
|
||||
|---|---|---|
|
||||
| Referenced only by an expected-deposit, no total/type/date | 4,897 | 98% |
|
||||
| Referenced by a sales order, real tender records | 290 | 5.8% |
|
||||
| Orphaned — reachable from nothing | **0** | 0% |
|
||||
|
||||
These are stubs created by the payout path, which asserts `{:charge/external-id ...}` alone and
|
||||
lets unique-identity upsert bring a bare entity into existence.
|
||||
|
||||
**Consequence:** Phase 1 cannot read scoping off the charge. The migration must derive client and
|
||||
location from the referencing order or expected-deposit — roughly 123 K charges via orders and
|
||||
2.0 M via deposits, extrapolated. Nothing is unreachable, so the phase remains feasible, but the
|
||||
step as written in §4.2 would strand 12.8% of charges on the legacy scheme.
|
||||
|
||||
The payout call site in `square.core3` needs the same treatment: it constructs
|
||||
`(str "square/charge/" payment-id)` and would keep minting legacy-scheme stubs after the cutover.
|
||||
It has `client` and `location` in scope already, so the fix is local.
|
||||
|
||||
## 7. Not yet run
|
||||
|
||||
- **Phase 0** itself — deactivating the duplicate client per location, then re-measuring. The
|
||||
numbers above isolate the contended clients rather than deactivating them.
|
||||
- **Charges referenced by more than one order** (§3.3), the gate on any historical cleanup.
|
||||
- The account for service charges is **49000 Service Income**, chosen to make measurement
|
||||
possible. It does not affect balance at all — only whether a day can reach `accepted?` — so
|
||||
every number here is independent of that choice.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Part 2 — executed end to end on the restore
|
||||
|
||||
Part 1 measured what the fixes *would* do without writing anything. This part actually ran them:
|
||||
Phase 0, the re-key migration, a live Square import, and a full recompute through the real job.
|
||||
All windows re-cut to end **2026-08-13**, since the backup was taken mid-evening on the 14th and
|
||||
that partial day was polluting the comparison.
|
||||
|
||||
## What was executed, in order
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| Phase 0 — deactivate the newer twin at all 10 shared locations | 10 datoms retracted; no location shared by two clients any more |
|
||||
| Phase 1 — re-key refunds | 50,986 re-keyed in 1.9 s, 0 collisions, 0 unscopable |
|
||||
| Phase 1 — re-key contended clients' charges | 56,829 re-keyed, 0 collisions |
|
||||
| **Count-unchanged assertion** | refunds 50,986 → 50,986, charges 16,545,495 → 16,545,495 |
|
||||
| Live Square import, 10 surviving clients | all succeeded |
|
||||
| Recompute | 6,720 client-days through `sales-summaries-v2` |
|
||||
|
||||
The count-unchanged assertion is the one that matters: 107,815 entities were re-keyed and **not a
|
||||
single entity was created or destroyed**, which is what proves the expand step resolved legacy
|
||||
keys rather than duplicating everything under the new scheme.
|
||||
|
||||
## Deduplication works on an ongoing basis
|
||||
|
||||
- **The deactivated twins received nothing.** Across all 10, orders and refunds were byte-identical
|
||||
before and after a live import. Phase 0 holds.
|
||||
- **Refunds stopped alternating and became two stable copies.** 98 of NGCC's 125 refunds are now
|
||||
also held by NGCD as separate entities. Each client owns its own; neither can take the other's.
|
||||
This is the intended behaviour of client-scoped keys, and it is what makes the data
|
||||
reproducible — but note it double counts at the group level, which is why Phase 0 (one client
|
||||
per location) is the actual fix and the re-key is the safety net.
|
||||
|
||||
### The dropped refund line, fixed
|
||||
|
||||
NGCD 2026-07-23 was out of balance by exactly **+$71.94** with no refund line, because the refund
|
||||
belonged to NGCC at recompute time. After the full pipeline the summary carries a **"Card Refunds"
|
||||
line of $71.94** and the day's imbalance is **9.1e-13 — zero**.
|
||||
|
||||
## Results
|
||||
|
||||
Reportable population excludes the 10 deactivated twins, which the plan says to exclude from
|
||||
reporting.
|
||||
|
||||
| | Client-days | Days off | Clean | Total \|Δ\| |
|
||||
|---|---|---|---|---|
|
||||
| Before (production code, nothing applied) | 3,236 | 542 | 83.25% | $30,982.11 |
|
||||
| **After (everything applied)** | 3,030 | **71** | **97.66%** | **$405.79** |
|
||||
|
||||
A **98.7% reduction in dollar error**. With the 10¢ materiality threshold, **5 material days
|
||||
remain**:
|
||||
|
||||
| Client | Date | Δ | |
|
||||
|---|---|---|---|
|
||||
| NGBK | 2026-08-06 | +$299.42 | new — see below |
|
||||
| NGDA | 2026-08-01 | −$50.00 | the known auto-gratuity-as-service-charge order |
|
||||
| NGEB | 2026-08-10 | −$25.00 | ezCater fee semantics, predicted |
|
||||
| NGEB | 2026-07-29 | −$20.00 | ezCater fee semantics, predicted |
|
||||
| NGPS | 2026-08-12 | +$9.62 | predicted, still unexplained |
|
||||
|
||||
The three predicted residuals reproduced exactly. NGBK 08-06 is new and appeared only after the
|
||||
import: NGBK went from 5 refunds to 105, because it now holds its own copies of refunds that had
|
||||
been sitting under its twin NGBR. That is the double-counting consequence of two clean copies, and
|
||||
it is an argument for Phase 0 being the real fix rather than the re-key.
|
||||
|
||||
## Two defects found by executing rather than simulating
|
||||
|
||||
### 1. Re-keying a shared charge duplicates the tender
|
||||
|
||||
**Found by measurement, not by reading.** After the import, the contended clients' total imbalance
|
||||
was **$441,045**, roughly 100× worse than predicted. The cause:
|
||||
|
||||
1. 35,870 of 56,829 charges had **two parent orders**, one per twin.
|
||||
2. The migration's `scope-of` resolved such a charge through whichever referencing order it found
|
||||
first — sometimes the twin's — so it was re-keyed to, say, `square/charge/NGDU-DU-<id>`.
|
||||
3. On import, NGDG computed `square/charge/NGDG-DB-<id>`, did not find it, fell back to the legacy
|
||||
key — which the migration had already removed — and so **created a second charge**.
|
||||
4. `:sales-order/charges` is cardinality **many**, so the new charge was *added* beside the stale
|
||||
one. One order, two tenders for one payment. Tender roughly doubled.
|
||||
|
||||
Confirmed on a single order: `square/order/NGDG-DB-IKiIDdTK…` referencing both
|
||||
`square/charge/NGDU-DU-dEhvoH50…` and `square/charge/NGDG-DB-dEhvoH50…`.
|
||||
|
||||
This is exactly what plan §4.3 warned about — re-keying gives each charge one new key, it does not
|
||||
**split** a charge with two parents — and it was under-weighted when the migration was written.
|
||||
|
||||
**Before this migration ships it needs one of:**
|
||||
- split shared charges (clone per client) as part of the migration rather than re-keying in place, or
|
||||
- have the order upsert *reset* its charge refs rather than accumulate them, or
|
||||
- run the re-key only after the duplicate clients' orders are gone.
|
||||
|
||||
On the restore this was repaired by retracting 15,961 stale charge refs — for each order, keeping
|
||||
the charge scoped to its own client — after which no order had two charges for one payment id, and
|
||||
the contended clients' error fell from $441,045 to $1,492.
|
||||
|
||||
### 2. `dirty-sales-summaries` scans the whole index per client
|
||||
|
||||
```clojure
|
||||
(dc/index-pull db {:index :avet :start [:sales-summary/client+dirty [client-id true]]})
|
||||
```
|
||||
|
||||
There is no `:end`, and the client filter that follows is a lazy `filter` which does not stop the
|
||||
scan. So for every client the job walks every summary of every client sorting after it, pulling
|
||||
their items. It is O(n²) in the number of summaries.
|
||||
|
||||
Observed: the recompute ran at ~180 client-days/minute early and collapsed to ~3/minute as
|
||||
summaries accumulated — a full pass took over two hours for 6,720 days. Adding an `:end` bound at
|
||||
the client boundary should fix it.
|
||||
|
||||
## Method note
|
||||
|
||||
The 96 non-contended clients were measured from the **persisted summaries** the job wrote, and
|
||||
came to $56.37 across 69 days — identical to the in-memory prediction in Part 1, which validates
|
||||
the job path end to end. The 20 contended clients were re-measured in memory after the charge
|
||||
repair, because a second full pass through the job would have taken hours for the reason in defect
|
||||
2. The two methods agree exactly where both were run.
|
||||
|
||||
## Ongoing deduplication, with both clients active again
|
||||
|
||||
The re-key is meant to make contention impossible even if a location is configured on two clients
|
||||
again. To test that rather than assume it, all 10 twins were **re-activated** — every location
|
||||
shared once more — and the import re-run.
|
||||
|
||||
**Repeated imports are idempotent.** Three consecutive refund imports for both halves of the
|
||||
Concord pair, with both clients live on location `L43Z5GMW72VMG`:
|
||||
|
||||
| | run 0 | run 1 | run 2 | run 3 |
|
||||
|---|---|---|---|---|
|
||||
| NGCC | 127 | 127 | 127 | 127 |
|
||||
| NGCD | 1,419 | 1,419 | 1,419 | 1,419 |
|
||||
|
||||
Not one entity created or moved. Before the re-key this is the exact configuration that made a
|
||||
refund's owner flip every ~20 minutes.
|
||||
|
||||
The first import after the migration is the only one that grows anything, and it grows once: NGCD
|
||||
gained 100 refunds materialising its own copies, while NGCC — whose refunds already carried its
|
||||
own scoped key — gained only the 2 that were genuinely new. After that, steady state.
|
||||
|
||||
**State the restore was left in:** twins re-activated, i.e. Phase 0 deliberately undone for this
|
||||
test. Re-applying it is the ten-datom retraction recorded above.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Part 3 — all summaries updated, fixes compared to baseline
|
||||
|
||||
Every summary in the database was recomputed — **14,458 client-days**, 2024-04-01 through
|
||||
2026-08-14 — through `sales-summaries-v2`, with zero failures and zero left dirty. The
|
||||
comparison below is the **last 30 days, 2026-07-15 → 2026-08-13**.
|
||||
|
||||
Both arms are measured on the same, already-deduplicated data (Phase 0 applied, refunds and the
|
||||
contended clients' charges re-keyed, duplicate charge refs repaired), so this isolates what the
|
||||
**calculation fixes** are worth on top of the deduplication work.
|
||||
|
||||
## Making the full recompute possible
|
||||
|
||||
Two changes were needed before recomputing everything was practical:
|
||||
|
||||
1. **`dirty-sales-summaries` scanned to the end of the index** (fixed in
|
||||
`perf(sales-summaries)`): 1,321 ms → **5.6 ms** per client, a 237× improvement, identical
|
||||
results.
|
||||
2. **The driver was one serial `doseq`.** `refresh-client!` was split out of `sales-summaries-v2`
|
||||
so a client's work stands alone. Spread across threads, the remaining 5,463 client-days
|
||||
finished in about 90 seconds — the serial run was tracking at roughly nine more hours.
|
||||
|
||||
## Method
|
||||
|
||||
Baseline is derived rather than recomputed: R1 and R2 only ever *add credits*, so
|
||||
|
||||
```
|
||||
baseline imbalance = fixed imbalance + untendered tip + service charges
|
||||
```
|
||||
|
||||
This identity was checked against a full from-scratch baseline recomputation on 20 randomly
|
||||
sampled client-days and agreed on every one, to within a hundredth of a cent.
|
||||
|
||||
## Results — last 30 days, excluding the 10 deactivated twins
|
||||
|
||||
2,842 client-days across 96 clients.
|
||||
|
||||
| | Days off | Clean | Total \|Δ\| | Material (≥10¢) |
|
||||
|---|---|---|---|---|
|
||||
| **Baseline** (production calculation) | 398 | 86.00% | $22,527.40 | 336 |
|
||||
| **Fixed** (R1 + R2) | **67** | **97.64%** | **$405.66** | **5** |
|
||||
|
||||
**331 client-days fixed, 0 regressed.** Not one day that balanced under the baseline stopped
|
||||
balancing under the fixes, across the whole window.
|
||||
|
||||
Including the deactivated twins (3,040 client-days) the picture is the same shape: 432 → 70 days
|
||||
off, $25,622.98 → $1,548.64.
|
||||
|
||||
### What is left
|
||||
|
||||
| Client | Date | Δ | |
|
||||
|---|---|---|---|
|
||||
| NGBK | 2026-08-06 | +$299.42 | refunds now held by both twins — see below |
|
||||
| NGDA | 2026-08-01 | −$50.00 | auto-gratuity booked as a service charge |
|
||||
| NGEB | 2026-08-10 | −$25.00 | ezCater fee semantics, predicted by the plan |
|
||||
| NGEB | 2026-07-29 | −$20.00 | ezCater fee semantics, predicted by the plan |
|
||||
| NGPS | 2026-08-12 | +$9.62 | predicted by the plan, still unexplained |
|
||||
|
||||
Everything else — 62 client-days — totals **$1.62**, with the largest single day at **9.00¢**.
|
||||
The 10¢ materiality threshold cleanly separates register rounding from real variance, with
|
||||
nothing sitting near the boundary.
|
||||
|
||||
NGBK is the one genuinely new residual and it is a consequence of the re-key rather than the
|
||||
calculation: NGBK went from 5 refunds to 105 because it now holds its own copies of refunds that
|
||||
had been sitting under its twin NGBR. Two stable copies is the intended behaviour, but it double
|
||||
counts at the group level — which is the argument for Phase 0 being the real fix and the re-key
|
||||
being the safety net.
|
||||
|
||||
### Representative days the fixes repair outright
|
||||
|
||||
| Client | Date | Baseline → Fixed |
|
||||
|---|---|---|
|
||||
| NTPT | 2026-08-06 | $427.10 → $0.00 |
|
||||
| NGFO | 2026-07-16 | $275.23 → $0.00 |
|
||||
| NGRN | 2026-08-06 | $236.96 → $0.00 |
|
||||
| N-30008 | 2026-07-24 | $229.42 → $0.00 |
|
||||
| N-30003 | 2026-07-18 | $225.00 → $0.00 |
|
||||
@@ -92,7 +92,7 @@
|
||||
<header class="masthead">
|
||||
<div class="eyebrow">Sales summaries · measured on a restored production backup</div>
|
||||
<h1>Ninety-Day Reconciliation</h1>
|
||||
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
|
||||
<p class="standfirst">Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — and a fourth, found late, that is a missing-data problem wearing a balancing problem's clothes. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
|
||||
<div class="meta">
|
||||
<span><b>Window</b> 2026-05-10 → 2026-08-07</span>
|
||||
<span><b>Client-days</b> <span class="num">18,900</span></span>
|
||||
@@ -105,32 +105,33 @@
|
||||
<div class="ledger">
|
||||
<div>
|
||||
<span class="side-label">Today's calculation, ninety days re-run</span>
|
||||
<span class="figure num">$69,560.10</span>
|
||||
<span class="subfig"><span class="num">1,258</span> days out of balance · <span class="num">93.34%</span> clean</span>
|
||||
<span class="figure num">$70,276.50</span>
|
||||
<span class="subfig"><span class="num">1,191</span> days out of balance · <span class="num">93.70%</span> clean</span>
|
||||
</div>
|
||||
<div class="arrow" aria-hidden="true">→</div>
|
||||
<div>
|
||||
<span class="side-label">The same ninety days, fixes on</span>
|
||||
<span class="figure after num">$7,790.54</span>
|
||||
<span class="subfig"><span class="num">279</span> days out of balance · <span class="num">98.52%</span> clean</span>
|
||||
<span class="figure after num">$2,379.45</span>
|
||||
<span class="subfig"><span class="num">122</span> days out of balance · <span class="num">99.35%</span> clean</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="stats">
|
||||
<div class="stat"><span class="k num">979</span><span class="l">client-days brought into balance</span></div>
|
||||
<div class="stat"><span class="k num">1,069</span><span class="l">client-days brought into balance</span></div>
|
||||
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
|
||||
<div class="stat"><span class="k num">88.8%</span><span class="l">of the variance removed</span></div>
|
||||
<div class="stat"><span class="k num">96.6%</span><span class="l">of the variance removed</span></div>
|
||||
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
|
||||
</div>
|
||||
|
||||
<div class="measure">
|
||||
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.</p>
|
||||
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>979 days and $61,769.56</strong>, not the larger number a stale baseline would have shown.</p>
|
||||
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>1,069 days and $67,897.05</strong>, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.</p>
|
||||
<p><strong>Most of what is left is not a balancing fault at all</strong>, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>The three problems</h2>
|
||||
<h2>The four problems</h2>
|
||||
|
||||
<div class="problem">
|
||||
<h4>1. Two client records sharing one Square location</h4>
|
||||
@@ -160,6 +161,18 @@
|
||||
<p><span class="tech">technical</span> <code>get-tip</code> summed tips by joining through <code>:sales-order/charges</code>, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on <code>:sales-order/tip</code>. Nothing at all read <code>:sales-order/service-charge</code>.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="problem">
|
||||
<h4>4. Refunds on records whose sales were never imported <span class="tech">not fixed — deliberately</span></h4>
|
||||
<div class="measure">
|
||||
<p><strong>This is why the duplicated restaurants looked so much worse than everyone else.</strong> Of the days still failing after the first three fixes, <strong>155 of the 423 on shared-location records had no sales orders at all</strong> — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.</p>
|
||||
<p>The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded <em>any</em> order shows <strong>140 of 171 fall before that client had a single order in the system</strong> — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.</p>
|
||||
<p><strong>Where the refunds came from.</strong> Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to <span class="mono">NGDG</span> that same day, and was taken over by <span class="mono">NGDU</span> on 12 August. Others flip between the two records several times a day across 12–15 August. <span class="mono">NGDU</span>'s first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.</p>
|
||||
<p>Across the nine records, <strong>660 refunds worth $15,237.02 sit on a record dated before that record's first order.</strong> Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.</p>
|
||||
<p><strong>Why it is deliberately left out of balance.</strong> The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and on this data it closes 16 of the 122 remaining days and $1,227.65. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.</p>
|
||||
<p><span class="tech">technical</span> <code>get-returns</code> sums <code>:sales-order/returns</code> over orders scanned for the date. With no orders the sum is nil and no <code>Returns</code> line is written, while <code>get-refund-items</code> still credits <code>Card Refunds</code> from the <code>sales-refund</code> records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
@@ -167,7 +180,8 @@
|
||||
<div class="measure">
|
||||
<p>Five changes. The first three stop two clients from sharing a record; the last two record
|
||||
money that was being collected but not booked. Each is small — the difficulty was knowing
|
||||
which line to change, not writing it.</p>
|
||||
which line to change, not writing it. A sixth was written and then removed; it is described
|
||||
at the end because the reasoning matters more than the code did.</p>
|
||||
</div>
|
||||
|
||||
<h3>1 · Put the client in the record's name</h3>
|
||||
@@ -212,10 +226,20 @@
|
||||
<p>The result is pinned as the record's id on the way in, so the write lands on the existing
|
||||
row regardless of which name it currently carries. The proof this worked is a count that did
|
||||
not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was
|
||||
re-named, and afterwards there were still exactly <strong>144,688 payouts and 69,291 cash-drawer
|
||||
shifts</strong> — the same figures as at the restore point. Refunds went from 50,986 to 51,986,
|
||||
and all 1,000 of those came from the live Square import run afterwards, not from the renaming.
|
||||
re-named, and afterwards there were still exactly <strong>50,986 refunds, 144,688 payouts and
|
||||
69,291 cash-drawer shifts</strong> — the same three figures as at the restore point.
|
||||
Had the fallback lookup been missing, each of these would have doubled instead.</p>
|
||||
<p><strong>The fallback also has to refuse.</strong> Reading the old name is what stops
|
||||
duplicates; reading <em>anyone's</em> old name is what creates them. Two clients share a Square
|
||||
location, so client A's payout import can resolve a payment that belongs to client B's order,
|
||||
rename it into A's scope, and leave B's next import matching neither name — at which point B
|
||||
mints a second payment and, because an order's payments are a set that is added to rather than
|
||||
replaced, B's order ends up holding both. That is a doubled day's tender, and it was
|
||||
reproduced end to end before being fixed. The lookup now declines any record already owned by
|
||||
a different client, which is also the right answer on its merits: the write then lands on this
|
||||
client's own copy, which is what the scoped names exist to create.</p>
|
||||
<p>This is transitional. Once no legacy names remain, the fallback and the refusal are deleted
|
||||
together and the guarantee stops depending on either.</p>
|
||||
</div>
|
||||
|
||||
<h3>3 · Give every order its own payment record</h3>
|
||||
@@ -234,9 +258,9 @@
|
||||
link it to the copy instead</pre>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>Run over the whole database that was <strong>9,100,314 renamed and 200,027 copied</strong>,
|
||||
<p>Run over the whole database that was <strong>16,236,839 renamed and 500,438 copied</strong>,
|
||||
and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across
|
||||
400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it
|
||||
every order of the last year. The record count rose by about 500,438 — the number of copies it
|
||||
reported making, which is the check that it created what it meant to and nothing else.</p>
|
||||
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
|
||||
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
|
||||
@@ -295,6 +319,28 @@
|
||||
few restaurants at a time.</p>
|
||||
</div>
|
||||
|
||||
<h3>6 · The change that was written, measured, and then taken out</h3>
|
||||
<div class="measure">
|
||||
<p>Worth recording, because the arithmetic case for it is good and someone will propose it
|
||||
again. Where a day has refunds and no sales orders whatsoever, book a <code>Returns</code>
|
||||
debit equal to that day's refunds:</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<pre>(defn- refund-only-returns [c date]
|
||||
(when-not (traded? c date)
|
||||
(let [amount (refunded-total c date)]
|
||||
(when-not (zero? amount) amount))))</pre>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>It works. Measured over the same ninety days it closed <strong>171 days and $5,795.18</strong>,
|
||||
knocked nothing out of balance, and altered no already-balanced day — the guard makes it
|
||||
incapable of touching a day that traded.</p>
|
||||
<p>It was removed anyway. Those days are not quiet days; they are days whose sales were never
|
||||
imported, and closing them removes the only visible sign of that. What is left in the code is
|
||||
a comment saying so and a test asserting the day <em>stays</em> out of balance, so the next
|
||||
person to notice the arithmetic finds the reasoning before they find the fix.</p>
|
||||
</div>
|
||||
|
||||
<h3>Supporting changes</h3>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
@@ -325,14 +371,14 @@
|
||||
<table>
|
||||
<thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,258</td><td class="n">93.34%</td><td class="n">$69,560.10</td></tr>
|
||||
<tr><td>+ refunded tips</td><td class="n">971</td><td class="n">94.86%</td><td class="n">$66,414.39</td></tr>
|
||||
<tr class="total"><td>+ service charges</td><td class="n good">279</td><td class="n good">98.52%</td><td class="n good">$7,790.54</td></tr>
|
||||
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,191</td><td class="n">93.70%</td><td class="n">$70,276.50</td></tr>
|
||||
<tr><td>+ refunded tips</td><td class="n">890</td><td class="n">95.29%</td><td class="n">$67,032.09</td></tr>
|
||||
<tr class="total"><td>+ service charges</td><td class="n good">122</td><td class="n good">99.35%</td><td class="n good">$2,379.45</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$61,769.56 is what the two arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
|
||||
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$66,589.75 is what the three arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
|
||||
</div>
|
||||
|
||||
<h3>Day-by-day effect of each change</h3>
|
||||
@@ -340,16 +386,26 @@
|
||||
<table>
|
||||
<thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Refunded tips</td><td class="n">18,590</td><td class="n good">287</td><td class="n good">0</td><td class="n good">0</td><td class="n">$3,712.67</td></tr>
|
||||
<tr><td>Service charges</td><td class="n">18,208</td><td class="n good">692</td><td class="n good">0</td><td class="n good">0</td><td class="n">$58,349.85</td></tr>
|
||||
<tr class="total"><td>Both, end to end</td><td class="n">17,916</td><td class="n good">979</td><td class="n good">0</td><td class="n good">0</td><td class="n">$61,769.56</td></tr>
|
||||
<tr><td>Refunded tips</td><td class="n">18,571</td><td class="n good">301</td><td class="n good">0</td><td class="n good">0</td><td class="n">$4,027.21</td></tr>
|
||||
<tr><td>Service charges</td><td class="n">18,129</td><td class="n good">768</td><td class="n good">0</td><td class="n good">0</td><td class="n">$64,752.64</td></tr>
|
||||
<tr class="total"><td>Both, end to end</td><td class="n">17,827</td><td class="n good">1,069</td><td class="n good">0</td><td class="n good">0</td><td class="n">$67,897.05</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>Neither fix touched a day that was already correct.</strong> Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered — 17,916 summaries came out byte-identical, and every one of the 984 that moved was already wrong. Service charges are by far the larger of the two, moving $58,349.85 against the tip fix's $3,712.67.</p>
|
||||
<p><strong>Neither fix touched a day that was already correct.</strong> Across all 18,900
|
||||
client-days no balanced day was knocked out of balance, and no balanced day had a single
|
||||
figure altered — 17,827 summaries came out byte-identical, and every one of the 1,073 that
|
||||
moved was already wrong.</p>
|
||||
<p>That claim did not hold on the first attempt, and how it was recovered is the useful part.
|
||||
Measured before the historical backfill described below, six days broke — all of them a tip
|
||||
reversed on one record whose refund sat on its twin, so removing the un-reversed tip left the
|
||||
day short by exactly that amount. Replaying the window from Square gave both records their own
|
||||
copy of every refund, and all six closed. The fix was never wrong; it was reading half a
|
||||
transaction.</p>
|
||||
<p>Service charges are by far the larger of the two fixes, moving $58,923.85 against the tip fix's $3,777.67.</p>
|
||||
<p>That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.</p>
|
||||
<p><strong>The two fixes account for the repair exactly.</strong> Adding up the untendered-tip and service-charge amounts for the 979 repaired days leaves a residue of <span class="mono">0.0000000013</span> against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a third effect.</p>
|
||||
<p><strong>The two fixes account for every day they moved, exactly.</strong> Adding up the untendered-tip and service-charge amounts across all 984 changed days leaves a residue of <span class="mono">0.0000000002</span>. Nothing else moved those days; there is no unexplained remainder hiding a third effect, and the six that broke are accounted for by the same arithmetic as the 915 that healed.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -418,10 +474,9 @@
|
||||
<table>
|
||||
<thead><tr><th>Step</th><th>Result</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Deactivate the duplicate client at each shared location</td><td class="n">10 locations · shared locations remaining: <span class="good">0</span></td></tr>
|
||||
<tr><td>Walk every order in the database</td><td class="n">19,040,785 orders</td></tr>
|
||||
<tr><td>Give every order its own payment record</td><td class="n">9,100,314 re-keyed · 200,027 copied</td></tr>
|
||||
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across 400,000 orders checked</span></td></tr>
|
||||
<tr><td>Walk every order in the database, newest month first</td><td class="n">19,040,785 orders · 38 minutes</td></tr>
|
||||
<tr><td>Give every order its own payment record</td><td class="n">16,236,839 re-keyed · 500,438 copied</td></tr>
|
||||
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across every order of the last year — 5,158,470</span></td></tr>
|
||||
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
|
||||
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
|
||||
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
|
||||
@@ -429,12 +484,13 @@
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.</p>
|
||||
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added 500,438, matching the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)</p>
|
||||
<p><strong>The measurement above was taken with both client records of each pair left live, which is how this deploys.</strong> Nothing is deactivated and no business decision about which restaurant's history survives is needed. The risk that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why <code>existing-id</code> now refuses to resolve a record belonging to another client.</p>
|
||||
</div>
|
||||
|
||||
<div class="callout">
|
||||
<span class="h">The whole analysis was run again from nothing, and landed in the same place</span>
|
||||
<p>Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. The end state matched the previous run <strong>to the cent</strong> — 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.</p>
|
||||
<p>Everything above was rebuilt from a fresh restore of the production backup, several times over, each time from the backup point itself rather than from a database an earlier run had touched: restore, re-key and split across all nineteen million orders, then two full ninety-day recomputes. The runs used deliberately different preparation — one deactivated the duplicate records, one left them live and untouched, one backfilled their history from Square — so their headline figures differ, and comparing them is how the recommendation below was reached. What did <strong>not</strong> move is the part that should not: for the 190 clients that do not share a Square location the residue is 119 days and $1,730.61 in every run, with the same five restaurants accounting for it. The arithmetic fixes behave identically no matter what is done to the duplicates, which is a stronger check on them than any single measurement.</p>
|
||||
</div>
|
||||
|
||||
<div class="callout warn">
|
||||
@@ -446,36 +502,89 @@
|
||||
<section>
|
||||
<h2>What is still out of balance</h2>
|
||||
<div class="measure">
|
||||
<p>279 client-days out of 18,900, totalling <strong>$7,790.54</strong>. Where it sits matters more than the total, because most of it is not on a restaurant anyone reports on.</p>
|
||||
<p>122 client-days out of 18,900, totalling <strong>$2,379.45</strong> — and only 32 of
|
||||
those are above ten cents.</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
<thead><tr><th>Where the remainder sits</th><th class="n">Days</th><th class="n">Variance</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>The twenty client records tied to a shared location — including the ten now-dormant duplicates</td><td class="n">160</td><td class="n">$6,059.93</td></tr>
|
||||
<tr class="total"><td>Every other client</td><td class="n">119</td><td class="n">$1,730.61</td></tr>
|
||||
<tr><td>The twenty records that share a Square location</td><td class="n good">3</td><td class="n good">$648.84</td></tr>
|
||||
<tr class="total"><td>Every other client — 190 of the 210</td><td class="n">119</td><td class="n">$1,730.61</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>Of that $1,730.61 on ordinary clients, four account for all but $2.40 of it:</p>
|
||||
<p><strong>The shared-location records are now the clean part of the book.</strong> Three days
|
||||
between all twenty of them: NGBK and NGBR at $299.42 each on 2026-08-06, which is the Square
|
||||
tender-versus-order-total gap described below and not an attribution fault, and NGDA at $50.00,
|
||||
an auto-gratuity booked as a service charge. Before the backfill those same records carried
|
||||
423 days and $18,508.39.</p>
|
||||
<p><strong>The other 119 days have not moved across any run of this analysis.</strong> Four
|
||||
separate rebuilds — different databases, different preparation, one with the duplicates
|
||||
deactivated and one without — all land on 119 days and $1,730.61, with the same five
|
||||
restaurants accounting for almost all of it:</p>
|
||||
</div>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
<thead><tr><th>Client</th><th class="n">Days</th><th class="n">Variance</th><th>What it is</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="mono">NG4S</td><td class="n">10</td><td class="n">$1,066.61</td><td>undiagnosed — the largest single unexplained cluster</td></tr>
|
||||
<tr><td class="mono">NG4S</td><td class="n">10</td><td class="n">$1,066.61</td><td>refunds arriving for a record with no sales imported — the fourth problem</td></tr>
|
||||
<tr><td class="mono">NGMV</td><td class="n">5</td><td class="n">$259.38</td><td>late May, undiagnosed</td></tr>
|
||||
<tr><td class="mono">NGEB</td><td class="n">4</td><td class="n">$199.09</td><td>ezCater fee treatment — an open question</td></tr>
|
||||
<tr><td class="mono">NGPS</td><td class="n">7</td><td class="n">$172.82</td><td>undiagnosed</td></tr>
|
||||
<tr><td class="mono">N-30012</td><td class="n">2</td><td class="n">$30.31</td><td>late May, undiagnosed</td></tr>
|
||||
<tr class="total"><td class="dim">PNSP, PNLP, GLAD and others</td><td class="n dim">91</td><td class="n dim">$2.40</td><td class="dim">till rounding — pennies a day</td></tr>
|
||||
<tr class="total"><td class="dim">everyone else</td><td class="n dim">91</td><td class="n dim">$2.40</td><td class="dim">till rounding — pennies a day</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p>The largest surviving single day is NGDL on 2026-08-01 at −$810.24, one of the shared-location records. The matched pair NGBR and NGBK both sit at exactly ±$299.42 on 2026-08-06: Square recorded $6,358.99 of tender against $6,059.57 of order totals, and the difference is that gap rather than anything the summary did wrong. The tip fix improved that day — it was $436.42 before — without closing it.</p>
|
||||
<p>The clusters on NG4S, NGMV, NGPS and N-30012 are unexplained and worth a look before this ships. They are visible only because the window is ninety days; a thirty-day view does not reach them.</p>
|
||||
<p>Sixteen of the 122 are days a record had no sales imported at all, worth $1,227.65 — the
|
||||
fourth problem, still deliberately visible. The clusters on NGMV, NGPS and NGEB are
|
||||
unexplained and worth a look, though at under $650 across sixteen days they are no longer
|
||||
urgent.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Making the duplicated restaurants match</h2>
|
||||
<div class="measure">
|
||||
<p>Re-keying stops the two records fighting, but on its own it does not make them equal, and
|
||||
the difference is worth stating plainly because it decides whether the books close.</p>
|
||||
<p><strong>Orders were always duplicated; refunds never were.</strong> A sales order's
|
||||
identifier has always carried its client, so each of the two records built its own order
|
||||
history from the start. Refunds, payouts and cash-drawer shifts did not, so only ONE record
|
||||
holds each of them — whichever imported it last. The migration freezes that ownership rather
|
||||
than evening it out. The record left without them shows returns from its own orders and no
|
||||
refunds to set against them, and is out of balance by exactly the amount its twin is holding.</p>
|
||||
<p>On 2026-05-11 both NGBK and NGBR held the same 221 orders. NGBK had no refunds; NGBR had
|
||||
two, worth $2,232.29; and NGBK's books were out by $2,232.29 to the cent. Across the whole
|
||||
database NGBK held <strong>158,535 orders and five refunds</strong>.</p>
|
||||
</div>
|
||||
<div class="callout">
|
||||
<span class="h">The fix is to ask Square again, not to manufacture copies</span>
|
||||
<p>With client-scoped keys in place, every record now creates its own copy of whatever it
|
||||
reads. So replaying the window from Square is all that is needed: each record imports the same
|
||||
refunds independently and the two histories converge, without any code inventing a duplicate
|
||||
and having to be trusted about it. <code>backfill-history</code> does exactly that for a date
|
||||
range, and after it every one of the ten pairs held matching order and refund counts.</p>
|
||||
<p>It closed <strong>420 of the 423 days</strong> the shared records were carrying, and
|
||||
$17,859.55 of the $18,508.39. It is also what recovered the zero-regression guarantee above.</p>
|
||||
</div>
|
||||
<div class="callout warn">
|
||||
<span class="h">One capped read, found by doing this</span>
|
||||
<p>The refunds import asked Square for a location's refunds and read the first page of the
|
||||
answer — no cursor, no date range. Square pages at a hundred, so a location with more than a
|
||||
hundred refunds silently returned a hundred, and the response looked complete. That is why the
|
||||
twins each held almost exactly 100 refunds, and why an earlier import added exactly 1,000
|
||||
across ten locations. Following the cursor is a few lines; the reason it went unnoticed for so
|
||||
long is that a capped list is indistinguishable from a short one.</p>
|
||||
</div>
|
||||
<div class="measure">
|
||||
<p><strong>This turned out to be a better answer than retiring the duplicate records.</strong>
|
||||
An earlier measurement that deactivated one record of each pair left 279 days and $7,790.54.
|
||||
Backfilling instead, with both records live, leaves <strong>122 days and $2,379.45</strong> —
|
||||
and it needs no business decision about which restaurant's history to abandon.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -491,8 +600,8 @@
|
||||
<table>
|
||||
<thead><tr><th>Record type</th><th class="n">Total</th><th class="n">Client-scoped</th><th class="n">Still to rename</th><th class="n">Cannot be scoped</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Card payments</td><td class="n">17,047,142</td><td class="n good">17,047,142</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Refunds</td><td class="n">51,986</td><td class="n good">51,986</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Card payments</td><td class="n">17,045,933</td><td class="n good">17,045,933</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Refunds</td><td class="n">50,986</td><td class="n good">50,986</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
<tr><td>Payouts</td><td class="n">144,688</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr>
|
||||
<tr><td>Cash-drawer shifts</td><td class="n">69,291</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr>
|
||||
</tbody>
|
||||
@@ -531,15 +640,20 @@
|
||||
<p>The problem this work exists to solve is gone: no payment answers to two orders, so the
|
||||
component relationship means what it says and deleting an order can no longer take another
|
||||
order's money with it.</p>
|
||||
<p><strong>One behaviour changed when the pass was run over everything, and it is worth
|
||||
recording.</strong> Where Square splits one tender across two of a single client's own orders,
|
||||
the earlier design left the payment shared on purpose — both orders compute the same name, so
|
||||
there is no second name for a copy to take. That rule only holds for two orders processed in
|
||||
the same batch. Run across nineteen million orders in batches of two thousand, such pairs
|
||||
almost always fall in different batches and the second order now takes a copy. Inside the
|
||||
ninety-day window this changed nothing measurable: the recompute after the database-wide pass
|
||||
matched the one before it to the cent. Outside the window it has not been measured, and it
|
||||
should be before this runs against production.</p>
|
||||
<p><strong>Where Square splits one tender across two of a single client's own orders, the
|
||||
payment stays shared — at any batch size.</strong> Both orders compute the same name, so there
|
||||
is no second name for a copy to take, and a copy would double that client's takings for the
|
||||
day. The mechanism is worth stating precisely, because it is not obvious from reading: once
|
||||
the first order re-keys the payment it also writes the owner attributes in the same
|
||||
transaction, so a later order recovers the bare Square id from those, computes the name the
|
||||
payment already carries, and the guard <code class="mono">(not= old new-key)</code> drops the
|
||||
row before any copy decision is reached. Verified by running the migration at a batch size of
|
||||
one, which forces the two orders into separate batches: no copy is made.</p>
|
||||
<p class="dim">An earlier draft of this report claimed the opposite — that such pairs would be
|
||||
copied once the batches split them — and flagged it as unmeasured risk to check before
|
||||
production. That was wrong, and it is recorded here rather than quietly deleted because it did
|
||||
real damage: an independent reviewer cited this paragraph as evidence and raised a defect that
|
||||
does not exist. A test now pins the behaviour at batch size one.</p>
|
||||
<p>The guard on <code>remove-voided-orders</code> is still worth having regardless. It is
|
||||
cheap, and it makes the safety a property of the deletion rather than of the migration having
|
||||
been run first.</p>
|
||||
@@ -548,11 +662,12 @@
|
||||
<div class="callout">
|
||||
<span class="h">Re-running is safe, and that was proved at full scale</span>
|
||||
<p>After the complete pass, asking the migration what it would change next returns
|
||||
<strong>nothing</strong> — 17,047,142 payments examined, none to rename, none unscopable. A
|
||||
<strong>nothing</strong> — 17,045,933 payments examined, none to rename, none unscopable. A
|
||||
record that already carries the right name is left untouched, so the migration can be stopped,
|
||||
resumed, or repeated without consequence.</p>
|
||||
<p>Its speed is worth a note for whoever schedules it: the whole nineteen million orders were
|
||||
walked in about <strong>thirteen minutes</strong>. An earlier attempt appeared to be
|
||||
walked in about <strong>thirty-eight minutes</strong>, month by month from the current month
|
||||
backwards so that stopping early leaves the recent end done. An earlier attempt appeared to be
|
||||
transactor-bound and was projected at two days, which is why a previous run narrowed it to the
|
||||
analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process
|
||||
driving the migration — freeing held memory took an unrelated recompute from 17 client-days a
|
||||
@@ -577,6 +692,7 @@
|
||||
<tbody>
|
||||
<tr><td>Which client record survives at each shared location</td><td>the business</td><td>the newer record generally has no history before the split, so keeping it loses years of the location's books</td></tr>
|
||||
<tr><td>Which revenue account service charges post to</td><td>accounting</td><td>currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances</td></tr>
|
||||
<tr><td><strong>659 refunds on records that have no sales for them</strong></td><td>the business, then engineering</td><td>the top open item. $15,225.24 dated before the holding record's own first order. Either the missing sales get imported, or the refunds move to the record that has them — but the books cannot close until one of the two happens</td></tr>
|
||||
<tr><td>Whether to correct records the wrong client already owns</td><td>the business</td><td>the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared</td></tr>
|
||||
<tr><td><code>remove-voided-orders</code></td><td>engineering</td><td>safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes</td></tr>
|
||||
</tbody>
|
||||
@@ -587,7 +703,7 @@
|
||||
<span class="h">Two operational findings, unrelated to the summaries</span>
|
||||
<p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p>
|
||||
<p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. Worth checking what production is set to.</p>
|
||||
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in thirteen minutes. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
|
||||
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in well under an hour. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -637,7 +753,7 @@
|
||||
|
||||
<footer>
|
||||
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
|
||||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 279 remaining days, 189 are material.</span>
|
||||
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 122 remaining days only 32 are material, and just 3 of them sit on the twenty records that share a Square location.</span>
|
||||
<span>Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on <code>as-of</code> above.</span>
|
||||
</footer>
|
||||
|
||||
419
docs/2026-08-15-sales-summary-rollout-plan.md
Normal file
419
docs/2026-08-15-sales-summary-rollout-plan.md
Normal file
@@ -0,0 +1,419 @@
|
||||
# Sales-summary balancing — rollout plan
|
||||
|
||||
Steps to execute, in order. Every step is either reversible or verifiable before the next one
|
||||
begins. The one behaviour change that alters a client's books is behind a per-client feature flag
|
||||
that is **off by default**, so merging and deploying this branch changes nothing on its own.
|
||||
|
||||
Measured on a restored copy of production (backup point `209608347`), 210 clients over
|
||||
2026-05-10 → 2026-08-07, with the duplicate client records left active exactly as they will be in
|
||||
production: **1,191 client-days out of balance / $70,276.50 → 122 days / $2,379.45**, of which only
|
||||
32 are above ten cents. 1,069 days came into balance, none broke, and no already-balanced day had a
|
||||
figure altered.
|
||||
|
||||
Of the $2,379.45 left, just **$648.84 across 3 days** is on the twenty shared-location records. The
|
||||
other 119 days and $1,730.61 belong to ordinary clients and have not moved across any run of this
|
||||
analysis.
|
||||
|
||||
Getting the shared records there needs step 5 — a historical backfill from Square. Without it they
|
||||
carry 423 days and $18,508.39, because re-keying stops the two records fighting but does not give
|
||||
each its own copy of the refunds.
|
||||
|
||||
---
|
||||
|
||||
## Before you start
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Flag introduced | `summary-service-charges` — off by default |
|
||||
| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` |
|
||||
| Expected migration runtime | ~38 minutes for 19M orders on a warm cache |
|
||||
| Backfill runtime (step 5) | ~5.9 hours for 90 days across the 20 shared-location records — an overnight job |
|
||||
| Nothing here touches | invoices, payments, the ledger, or any client without the flag set |
|
||||
|
||||
**Client configuration is left exactly as it is.** Ten Square locations are configured against two
|
||||
client records each, and both stay active. The re-key is what resolves them: once every record
|
||||
carries its owner in its key, each client's import resolves only its own records and the two
|
||||
records keep independent, stable histories. No "which record survives" decision is needed, and
|
||||
nothing is deactivated.
|
||||
|
||||
The consequence to be aware of: each Square payment, refund, payout and shift at a shared location
|
||||
becomes **two entities, one per client record** — by design. That is the stable end state, not a
|
||||
duplicate to clean up. If any report or export aggregates across client records, one restaurant's
|
||||
takings would be counted twice at that layer. Nothing in this work changes that either way.
|
||||
|
||||
That holds automatically for everything imported *from now on*, because the keys carry the client.
|
||||
It does **not** hold for history: refunds, payouts and shifts already in the database exist only
|
||||
once, on whichever record imported them last, and re-keying freezes that rather than evening it out.
|
||||
Step 5 is what brings the existing history into the same shape.
|
||||
|
||||
**The only window of risk is between deploying and finishing the migration**, while legacy keys
|
||||
still exist for a client to resolve. Steps 2–6 exist to make that window effectively zero.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Guard `remove-voided-orders`
|
||||
|
||||
Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so
|
||||
retracting an order cascades into its payments. Until step 4 finishes there are still payments with
|
||||
two parent orders, and deleting one client's voided order can take the other client's payment with
|
||||
it.
|
||||
|
||||
Either leave `remove-voided-orders` switched off until step 4 verifies clean, or change it to detach
|
||||
a payment that has more than one parent rather than delete it. Detaching is worth doing regardless —
|
||||
it makes the safety a property of the deletion rather than of the migration having been run first.
|
||||
|
||||
See `docs/2026-08-15-remove-voided-orders-risk.md`.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Pause the Square importer
|
||||
|
||||
**This is what makes the deploy safe, and it is easy to skip.** Steps 2 through 6 should be one
|
||||
maintenance action, not separate days' work.
|
||||
|
||||
While legacy keys exist, `square.core3/existing-id` falls back to them — and at a shared location
|
||||
that is the one code path that can reach across client records. Running the migration with imports
|
||||
paused means no client is resolving keys while the keys are being rewritten, so the window closes
|
||||
entirely rather than merely narrowing.
|
||||
|
||||
The migration itself takes about **38 minutes** for all 19M orders, so the pause is short — and
|
||||
if you need it shorter, see step 6: you can resume imports before it finishes.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Deploy the code
|
||||
|
||||
Deploy the branch. The flag is absent from every client, so:
|
||||
|
||||
- tips are calculated exactly as they are today,
|
||||
- no `Service Charges` line is written.
|
||||
|
||||
The only changes that take effect immediately are the safe ones: imbalance logging, the
|
||||
dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix,
|
||||
and the importer's new client-scoped keys.
|
||||
|
||||
**The importer reads both key schemes**, so the deploy does not depend on the migration having
|
||||
finished. Two protections cover the interval before it does: imports are paused (step 2), and
|
||||
`existing-id` refuses to resolve a record that already belongs to a different client. Do not remove
|
||||
the legacy lookup yet — see step 10.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Run the migration
|
||||
|
||||
Run it immediately after the deploy, while imports are still paused.
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.jobs.rekey-square-external-ids :as rk])
|
||||
|
||||
;; read-only first — no two entities may want the same key. `plan` does NOT return a
|
||||
;; :collisions key; you have to hand its :new-keys to `collisions` yourself.
|
||||
(rk/collisions (:new-keys (rk/plan (d/db conn) :charge/external-id rk/charge-prefix)))
|
||||
;; => [] (anything else: stop, do not migrate)
|
||||
|
||||
;; then the whole thing
|
||||
(rk/migrate-all! 2000)
|
||||
```
|
||||
|
||||
`migrate-all!` runs this same check itself, on every attribute including charges, and throws
|
||||
rather than transacting if it finds one. Running it by hand first just means finding out before
|
||||
the 38-minute walk rather than partway through it.
|
||||
|
||||
Runs in about thirty-eight minutes over 19M orders. It is **idempotent and resumable** — a record that
|
||||
already carries the right name is skipped, so it can be stopped and re-run without consequence.
|
||||
|
||||
**It is also ordered so that stopping early is survivable.** Refunds, payouts and cash-drawer
|
||||
shifts go first — a quarter of a million records, seconds of work — so an interruption cannot catch
|
||||
them half done. The long part then walks orders **a month at a time, from the current month
|
||||
backwards**, logging `::month-complete` as each finishes:
|
||||
|
||||
```
|
||||
::month-complete :month "2026-08" :rekeyed 118203 :cloned 2244
|
||||
::month-complete :month "2026-07" :rekeyed 241887 :cloned 4611
|
||||
...
|
||||
```
|
||||
|
||||
That ordering is the recovery plan. If it dies, everything from the last logged month forward is
|
||||
fully scoped — and that recent window is what the importer actually reads — so **you can resume
|
||||
imports against a partially migrated database** and finish the older tail later. Walking oldest
|
||||
first would have spent the first several hours on 2019 data no import will touch, leaving exactly
|
||||
the wrong end done.
|
||||
|
||||
If you do resume imports mid-migration, the ownership guard in `existing-id` is what keeps the
|
||||
unmigrated tail safe: a client cannot resolve onto another client's legacy-keyed record.
|
||||
|
||||
If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
|
||||
not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
|
||||
retained memory in the REPL and re-measure before changing anything about the database.
|
||||
|
||||
**Verify.** Two checks, doing two different jobs — run both.
|
||||
|
||||
**(a) Completeness, across everything.** `plan` must report nothing left to do, for all four
|
||||
attributes:
|
||||
|
||||
```clojure
|
||||
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
|
||||
;; => {:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0}
|
||||
```
|
||||
|
||||
Read `:to-migrate 0` **and** `:unscopable 0`. This is the authoritative signal, and it covers all
|
||||
17M charges.
|
||||
|
||||
`unscoped-report` is useful colour but is not the gate: its `:no-owner` column never reaches zero
|
||||
for charges, because ~283k payout stubs carry no `:charge/client` of their own and it classifies
|
||||
by attribute rather than by resolving ownership. Judge completeness by `plan`.
|
||||
|
||||
**(b) The safety gate for the cascade** — no payment may answer to two orders, or re-enabling
|
||||
`remove-voided-orders` in step 9 can delete a payment another order still needs. Check **every**
|
||||
order in the last year, with no sampling:
|
||||
|
||||
```clojure
|
||||
(let [db (d/db conn)
|
||||
cs (map first (d/q '[:find ?c :where [?c :client/code _]] db))
|
||||
year (java.util.Date. (- (.getTime (java.util.Date.)) (long (* 365 86400000))))]
|
||||
(rk/charges-with-multiple-parents
|
||||
db (map first (iol-ion.query/scan-sales-orders db cs year nil))))
|
||||
;; => 0
|
||||
```
|
||||
|
||||
On the restored copy that is 5,158,470 orders — 27% of the table — via the
|
||||
`:sales-order/client+date` index. A year is chosen deliberately: `remove-voided-orders` only ever
|
||||
deletes orders Square reports as voided, which are recent, so that is where the destructive risk
|
||||
lives. Completeness across all of history is check (a)'s job, not this one.
|
||||
|
||||
> Do **not** sample this with `(take n (rk/all-order-ids db))`. `all-order-ids` streams `:aevt`,
|
||||
> which is ascending entity id, so a `take` returns the *oldest* orders — on the restored copy the
|
||||
> first 400,000 are all from 2019–2021, before any of the contention this gate looks for. It would
|
||||
> report a confident zero having inspected none of the relevant data.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Backfill the shared-location clients from Square
|
||||
|
||||
**Skip this and the ten duplicated restaurants stay badly out of balance.** It is the difference
|
||||
between 122 client-days out of balance and 542.
|
||||
|
||||
Sales orders have always been keyed by client, so both records of a pair built their own order
|
||||
history. Refunds, payouts and cash-drawer shifts were not, so only ONE record holds each of them.
|
||||
Re-keying freezes that ownership; it does not even it out. The record left without them shows
|
||||
returns from its own orders and no refunds against them — NGBK held 158,535 orders and five
|
||||
refunds — and is out of balance by exactly what its twin is holding.
|
||||
|
||||
Rather than manufacture copies, ask Square again. Client-scoped keys mean each record now creates
|
||||
its own copy of whatever it reads, so replaying the window makes the two histories converge:
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.square.core3 :as sq])
|
||||
(require '[clj-time.core :as t])
|
||||
|
||||
@(apply sq/backfill-history
|
||||
(t/date-time 2026 5 10) (t/date-time 2026 8 9)
|
||||
["NGBK" "NGBR" "NGCD" "NGCC" "NGVG" "NGVC" "NGEZ" "NGJS" "NGDG" "NGDU"
|
||||
"NGDV" "NGDS" "NGWC" "NGWN" "NGHY" "NGHA" "NGDA" "NGDL" "NGCL" "NGCT"])
|
||||
```
|
||||
|
||||
**Verify** — every pair should hold matching order and refund counts in the window:
|
||||
|
||||
```clojure
|
||||
;; per pair, per side: window orders and window refunds. The two sides should agree.
|
||||
```
|
||||
|
||||
Measured on the restored copy: all ten pairs matched afterwards, and the shared records went from
|
||||
423 days and $18,508.39 out of balance to 3 days and $648.84.
|
||||
|
||||
**Budget an overnight run.** This took **5.9 hours** for ninety days across the twenty records.
|
||||
Every Square call in the process shares one 25-requests-per-second throttle, refunds and shifts cost
|
||||
one API call per record, and `backfill-history` imports three clients at a time — raise its
|
||||
`s/buffer` if you need it faster. Neither the database nor the transactor is the limit; reads
|
||||
measured at 32 µs.
|
||||
|
||||
It must run **after** the migration. Run before, and it imports against legacy keys and leaves more
|
||||
to migrate.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Resume the Square importer
|
||||
|
||||
Normally: once step 4's two checks read clean and step 5's backfill has finished. The maintenance
|
||||
window ends here.
|
||||
|
||||
**If the migration did not finish**, you do not have to wait for it. Resume imports once the
|
||||
`::month-complete` log covers the window your importer reads — the last 75 days for payouts and
|
||||
cash-drawer shifts, and whatever range the order import is configured for. Then re-run
|
||||
`migrate-all!` afterwards to walk the remaining older months; it will skip everything already done.
|
||||
Run the step 4 checks again once it does finish.
|
||||
|
||||
The first cycle after resuming is the one to watch. Compare these against the same counts taken
|
||||
immediately before the deploy — growth should be ordinary daily volume:
|
||||
|
||||
```clojure
|
||||
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))
|
||||
(count (d/datoms (d/db conn) :aevt :charge/external-id))
|
||||
```
|
||||
|
||||
A near-doubling of any of them means records are being created rather than matched — **stop and
|
||||
roll back the deploy.** Charges are included deliberately: they are the one that doubles a client's
|
||||
takings rather than merely duplicating a row.
|
||||
|
||||
---
|
||||
|
||||
## Step 7 — Recompute summaries, flags still off
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.jobs.sales-summaries :as ss])
|
||||
(ss/refresh-sales-summaries 90)
|
||||
```
|
||||
|
||||
This is the pass that banks the deduplication. **Capture the result before going further** — you
|
||||
will need it as the baseline for step 8, and it cannot be reconstructed afterwards:
|
||||
|
||||
```clojure
|
||||
(require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath
|
||||
(def before (cmp/summaries-in (d/db conn) start end))
|
||||
(spit "before.edn" (pr-str before))
|
||||
```
|
||||
|
||||
> **Do not use `d/as-of` to compare summary amounts.** `:ledger-mapped/amount`, `ledger-side` and
|
||||
> `account` are `:db/noHistory`, so past values are discarded. A summary that has since been
|
||||
> recomputed reads back through `as-of` with its amounts *absent*, which looks like a legitimate
|
||||
> balanced day. Capture live, before and after, and diff the captures.
|
||||
|
||||
---
|
||||
|
||||
## Step 8 — Turn the flag on, a few restaurants at a time
|
||||
|
||||
Needs accounting sign-off first: `summary-service-charges` posts to **49000 Service Income**, chosen
|
||||
so the work could be measured. It affects reporting, never whether a day balances.
|
||||
|
||||
```clojure
|
||||
@(d/transact conn [{:db/id [:client/code "NGxx"]
|
||||
:client/feature-flags ["summary-service-charges"]}])
|
||||
(ss/refresh-sales-summaries 90)
|
||||
```
|
||||
|
||||
Start with two or three restaurants, confirm, then widen.
|
||||
|
||||
**Verify** against the capture from step 7:
|
||||
|
||||
```clojure
|
||||
(def after (cmp/summaries-in (d/db conn) start end))
|
||||
(cmp/compare-window ...) ; both arguments live database values, never as-of
|
||||
```
|
||||
|
||||
The two numbers that matter — both were zero across all 18,900 client-days in testing:
|
||||
|
||||
- `:balanced->unbalanced` must be **0**
|
||||
- previously-balanced days whose lines changed must be **0**
|
||||
|
||||
If either is non-zero, retract the flag for the affected clients and re-run step 7. The flag is the
|
||||
rollback: removing it restores today's behaviour exactly.
|
||||
|
||||
---
|
||||
|
||||
## Step 9 — Re-enable `remove-voided-orders`
|
||||
|
||||
Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 1.
|
||||
|
||||
---
|
||||
|
||||
## Step 10 — Remove the legacy key lookup
|
||||
|
||||
**Schedule this; do not leave it open-ended.** Both client records at a shared location stay active
|
||||
permanently, so the legacy fallback in `square.core3/existing-id` is the one code path that can ever
|
||||
reach across them. Deleting it is what turns the guarantee from conventional into structural.
|
||||
|
||||
Once `plan` reports `:to-migrate 0` and has stayed there through several import cycles, drop the
|
||||
legacy branch of `existing-id` — and with it `owned-by-other-client?`, which exists only to make
|
||||
that branch safe while it lives. After this, two clients on one location are structurally incapable
|
||||
of resolving onto each other's records, and no ordering discipline is required to keep it that way.
|
||||
|
||||
Until it is done, the protection is the guard plus the maintenance window, both of which depend on
|
||||
people doing the right thing. That is the reason not to let this drift.
|
||||
|
||||
---
|
||||
|
||||
## Step 11 — Deal with the refunds that have no sales behind them
|
||||
|
||||
**The most important item in this document, and the only one that is not just execution.**
|
||||
|
||||
16 of the 122 remaining days are a record carrying refunds on a day it recorded no sales at all,
|
||||
and all 16 fall before that client's first ever order. Two clients are affected, holding **160
|
||||
refunds worth $4,347.68 dated before their own first order**:
|
||||
|
||||
| Client | First order | Refunds before it | Value | Days out of balance |
|
||||
|---|---|---:|---:|---:|
|
||||
| NG4S | 2026-05-29 | 79 | $2,180.08 | 10 |
|
||||
| NGPS | 2026-05-26 | 81 | $2,167.60 | 7 |
|
||||
|
||||
**Step 5's backfill already resolved the other seven.** Before it, nine records were in this state
|
||||
holding 660 refunds worth $15,237.02 — but seven of them were shared-location twins whose refunds
|
||||
only looked orphaned because their orders had never been imported. Replaying the window gave them
|
||||
their orders, and the refunds stopped predating them.
|
||||
|
||||
NG4S and NGPS are different: neither shares a Square location, so there is no twin holding the other
|
||||
half. Their sales genuinely are not in the system for the period their refunds cover. The database's
|
||||
own ownership history is the evidence to check — for the twins it showed refunds changing hands
|
||||
between the two records; for these two there is no second record to have taken them from.
|
||||
|
||||
Two ways to close it, and the business has to pick:
|
||||
|
||||
1. **Import the missing sales.** Correct if these records are meant to have their own books. Try
|
||||
`backfill-history` for them first, with a window reaching back before their first order — that is
|
||||
exactly what fixed the seven, and it is one command.
|
||||
2. **Move the refunds to the record that has the sales.** Correct only if the refunds were misfiled
|
||||
onto a record that should not have books of its own.
|
||||
|
||||
Start with (1): it is cheap, reversible in the sense that it only adds what Square reports, and it
|
||||
is already proven to work on this exact symptom.
|
||||
|
||||
```clojure
|
||||
;; per client: refunds dated before that client's own first order
|
||||
(let [first-order (->> (d/q '[:find [?d ...] :in $ ?c
|
||||
:where [?o :sales-order/client ?c] [?o :sales-order/date ?d]]
|
||||
(d/db conn) [:client/code "NG4S"])
|
||||
(reduce (fn [a b] (if (.before a b) a b))))]
|
||||
(->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c
|
||||
:where [?r :sales-refund/client ?c]]
|
||||
(d/db conn) [:client/code "NG4S"])
|
||||
(filter #(.before (:sales-refund/date %) first-order))
|
||||
count))
|
||||
```
|
||||
|
||||
**Until this is resolved those days stay out of balance, on purpose.** A summary change to close
|
||||
them was written and measured — it works, closes 16 days and $1,227.65, and breaks nothing — and it
|
||||
was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not
|
||||
being imported. A test asserts the day stays unbalanced so nobody closes it without reading this.
|
||||
|
||||
---
|
||||
|
||||
## What this will not fix
|
||||
|
||||
122 client-days over ninety days, $2,379.45, of which only 32 are above ten cents.
|
||||
|
||||
| | Days | Variance | |
|
||||
|---|---:|---:|---|
|
||||
| Real trading days with genuine discrepancies | 106 | $1,151.80 | see below |
|
||||
| Refunds on a record with no sales imported | 16 | $1,227.65 | step 11 — deliberately visible |
|
||||
|
||||
Of the 106 trading days, only **3 are on shared-location records** — $648.84 in total, and all three
|
||||
are already diagnosed: NGBK and NGBR at $299.42 each on 2026-08-06, where Square recorded $6,358.99
|
||||
of tender against $6,059.57 of order totals (the gap itself, not a summary fault), and NGDA at
|
||||
$50.00, an auto-gratuity booked as a service charge.
|
||||
|
||||
The other 103 days come to **$502.96 across 190 clients** — a few dollars here and there, mostly
|
||||
till rounding, plus small undiagnosed clusters on NGMV ($259.38 over 5 days) and NGEB ($199.09 over
|
||||
4 days, an ezCater fee-treatment question). Those two are worth a look but are not urgent.
|
||||
|
||||
That 103-day, $502.96 figure has been identical in every run of this analysis — with the duplicates
|
||||
deactivated, with them live, and with them backfilled. It is the floor this work reaches.
|
||||
|
||||
---
|
||||
|
||||
## Two operational findings, unrelated to the summaries
|
||||
|
||||
- **The production backup had not written a restore point since 2025-03-10** — about seventeen
|
||||
months — although data files were still uploading daily. Worth an alert on restore-point age.
|
||||
- **The database server is sized for a much smaller dataset**: a 2 GB cache against 27 GB of data.
|
||||
Worth checking what production is set to.
|
||||
@@ -20,7 +20,8 @@
|
||||
(:require
|
||||
[auto-ap.datomic :refer [conn]]
|
||||
[auto-ap.logging :as alog]
|
||||
[datomic.api :as dc]))
|
||||
[datomic.api :as dc]
|
||||
[iol-ion.query]))
|
||||
|
||||
(def refund-prefix "square/refund/")
|
||||
(def charge-prefix "square/charge/")
|
||||
@@ -195,7 +196,11 @@
|
||||
[orders batch-size]
|
||||
(let [cloned (atom 0)
|
||||
rekeyed (atom 0)]
|
||||
(doseq [batch (partition-all batch-size orders)]
|
||||
(doseq [[i batch] (map-indexed vector (partition-all batch-size orders))]
|
||||
(when (zero? (mod i 200))
|
||||
;; the whole-database run walks 19M orders; without a trail an interrupted run leaves
|
||||
;; no way to tell how far it got short of querying the data by hand
|
||||
(alog/info ::splitting :orders-done (* i batch-size) :rekeyed @rekeyed :cloned @cloned))
|
||||
(let [db (dc/db conn)
|
||||
batch-seen (atom {})
|
||||
tx (doall
|
||||
@@ -255,12 +260,19 @@
|
||||
"Counts, per entity type, how many keys are already client-scoped, how many still carry the
|
||||
legacy unscoped form, and how many have no owner to scope by.
|
||||
|
||||
This is the completeness gate. The importer tolerates both key schemes on purpose, so that the
|
||||
change can be deployed before the migration finishes — but that tolerance is a transition, not
|
||||
a resting place. While `:legacy` is above zero the database is in a mixed state and a stray
|
||||
unscoped record can still be adopted by whichever client imports it first. Once every count
|
||||
reads zero the fallback lookup in `square.core3/existing-id` can be removed and the guarantee
|
||||
becomes structural rather than conventional."
|
||||
The importer tolerates both key schemes on purpose, so that the change can be deployed before
|
||||
the migration finishes — but that tolerance is a transition, not a resting place. While
|
||||
`:legacy` is above zero the database is in a mixed state and a stray unscoped record can still
|
||||
be adopted by whichever client imports it first. **`:legacy` reaching zero on every attribute is
|
||||
the done-signal**, and it is what licenses removing the fallback lookup in
|
||||
`square.core3/existing-id`.
|
||||
|
||||
`:no-owner` is NOT part of that signal and never reaches zero for charges. It counts entities
|
||||
whose own `:charge/client`/`:charge/location` are absent — around an eighth of charges, the
|
||||
payout stubs `migrate!` only ever gives an external id — so this report structurally cannot
|
||||
verify them even when their keys are perfectly scoped. It classifies by attribute; `plan`
|
||||
resolves ownership through whatever references the entity. Ask `plan` for the authoritative
|
||||
answer: `:to-migrate 0` with `:unscopable 0` means there is nothing left to do."
|
||||
[db]
|
||||
(into {}
|
||||
(for [{:keys [attr prefix client location]} scoped-attrs]
|
||||
@@ -277,10 +289,51 @@
|
||||
(dc/datoms db :aevt attr))])))
|
||||
|
||||
(defn all-order-ids
|
||||
"Every sales order in the database, streamed."
|
||||
"Every sales order in the database, streamed in `:aevt` order — which is ascending entity id,
|
||||
so OLDEST first. Fine for counting; wrong for anything that takes a prefix. `(take n ...)` of
|
||||
this returns the oldest n orders, not a sample: on the production copy the first 400,000 are
|
||||
all from 2019 to 2021. Use `order-months-newest-first` to walk the data in migration order."
|
||||
[db]
|
||||
(map :e (dc/datoms db :aevt :sales-order/external-id)))
|
||||
|
||||
(def ^:private earliest-orders
|
||||
"How far back the month walk goes. Comfortably before the oldest order in the database
|
||||
(2019-12-31 on the production copy); months with no orders cost one index seek per client."
|
||||
#inst "2015-01-01T00:00:00.000-00:00")
|
||||
|
||||
(defn- ->date [^java.time.LocalDate d]
|
||||
(java.util.Date/from (.toInstant (.atStartOfDay d (java.time.ZoneId/systemDefault)))))
|
||||
|
||||
(defn order-months-newest-first
|
||||
"`[start end]` month windows from now back to `earliest`, newest month first.
|
||||
|
||||
The migration walks months in this order on purpose. It is the difference between an
|
||||
interrupted run leaving the data safe to import against and leaving it dangerous: the importer
|
||||
works on recent data, so having the newest months fully scoped is what lets imports resume
|
||||
while the older tail is still unmigrated. Walking oldest-first would spend hours on 2019 before
|
||||
touching anything this month's import will read.
|
||||
|
||||
Windows tile without gaps — each month's end is the day before the next month's start — and
|
||||
because the migration is idempotent an order landing in two windows is a no-op the second
|
||||
time, so boundary precision is not safety-critical."
|
||||
([] (order-months-newest-first earliest-orders))
|
||||
([^java.util.Date earliest]
|
||||
(let [zone (java.time.ZoneId/systemDefault)
|
||||
floor (java.time.YearMonth/from (.toLocalDate (.atZone (.toInstant earliest) zone)))]
|
||||
(->> (iterate (fn [^java.time.YearMonth m] (.minusMonths m 1)) (java.time.YearMonth/now zone))
|
||||
(take-while (fn [^java.time.YearMonth m] (not (.isBefore m floor))))
|
||||
(map (fn [^java.time.YearMonth m]
|
||||
[(->date (.atDay m 1)) (->date (.atEndOfMonth m))]))))))
|
||||
|
||||
(defn orders-in-window
|
||||
"Sales order ids for every client between `start` and `end` inclusive, via the
|
||||
`:sales-order/client+date` index."
|
||||
[db clients start end]
|
||||
(map first (iol-ion.query/scan-sales-orders db clients start end)))
|
||||
|
||||
(defn- all-client-ids [db]
|
||||
(map first (dc/q '[:find ?c :where [?c :client/code _]] db)))
|
||||
|
||||
(defn migrate-all!
|
||||
"The complete migration, over the whole database rather than a chosen subset.
|
||||
|
||||
@@ -289,18 +342,40 @@
|
||||
Nine client pairs contended in the past and no longer share one; their records are still mixed,
|
||||
and a migration scoped to the current configuration would miss every one of them.
|
||||
|
||||
**Ordered so that an interrupted run is recoverable.** Refunds, payouts and cash-drawer shifts
|
||||
go first: together they are a quarter of a million records and take seconds, so finishing them
|
||||
up front means an interruption cannot leave them half done. The long part — walking every order
|
||||
to split shared charges — then runs a month at a time from the current month backwards, logging
|
||||
each month as it completes. Stop it after any month and the data from that month forward is
|
||||
fully scoped, which is the part the importer reads, so imports can resume against it while the
|
||||
older tail waits. Re-running picks up where it left off because each month's work is idempotent.
|
||||
|
||||
Returns the split counts and the completeness report, which should read zero legacy across the
|
||||
board when this finishes."
|
||||
[batch-size]
|
||||
(let [split (split-and-rekey-charges! (all-order-ids (dc/db conn)) batch-size)]
|
||||
(doseq [{:keys [attr prefix]} scoped-attrs
|
||||
:when (not= attr :charge/external-id)]
|
||||
(let [p (plan (dc/db conn) attr prefix)]
|
||||
(when-let [c (seq (collisions (:new-keys p)))]
|
||||
(throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
|
||||
(migrate! attr (:new-keys p) batch-size)))
|
||||
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them
|
||||
(doseq [{:keys [attr prefix]} scoped-attrs
|
||||
:when (not= attr :charge/external-id)]
|
||||
(let [p (plan (dc/db conn) attr prefix)]
|
||||
(when-let [c (seq (collisions (:new-keys p)))]
|
||||
(throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
|
||||
(migrate! attr (:new-keys p) batch-size)))
|
||||
(let [clients (all-client-ids (dc/db conn))
|
||||
split (reduce (fn [acc [start end]]
|
||||
(let [ids (orders-in-window (dc/db conn) clients start end)
|
||||
r (split-and-rekey-charges! ids batch-size)]
|
||||
(alog/info ::month-complete
|
||||
:month (subs (str (.toInstant ^java.util.Date start)) 0 7)
|
||||
:rekeyed (:rekeyed r) :cloned (:cloned r))
|
||||
(merge-with + acc r)))
|
||||
{:rekeyed 0 :cloned 0}
|
||||
(order-months-newest-first))]
|
||||
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them.
|
||||
;; Collision-checked like the others: this is the largest attribute in the database, so it is
|
||||
;; the last one that should discover a clash as a mid-run exception.
|
||||
(let [p (plan (dc/db conn) :charge/external-id charge-prefix)]
|
||||
(when-let [c (seq (collisions (:new-keys p)))]
|
||||
(throw (ex-info "two entities would take the same key"
|
||||
{:attr :charge/external-id :collisions (count c)})))
|
||||
(when (seq (:new-keys p)) (migrate! :charge/external-id (:new-keys p) batch-size)))
|
||||
{:split split :completeness (unscoped-report (dc/db conn))}))
|
||||
|
||||
|
||||
@@ -378,6 +378,14 @@
|
||||
:ledger-mapped/amount (- (+ total discount) tax)
|
||||
#_#_:ledger-mapped/account nil})))
|
||||
|
||||
;; A day carrying refunds and no sales at all is left out of balance on purpose. It is tempting
|
||||
;; to close it by booking a return against the day's refunds — the arithmetic works, and no
|
||||
;; trading day could be affected. Do not. Those days are overwhelmingly not "a refund settled
|
||||
;; while the restaurant was shut": they are days whose *orders were never imported*, on client
|
||||
;; records that took ownership of another record's refunds through the unscoped keys this branch
|
||||
;; fixes. Balancing them would convert the only signal that a client's sales are missing into
|
||||
;; silence. See `docs/2026-08-15-sales-summary-rollout-plan.md`.
|
||||
|
||||
(defn- get-returns [c date]
|
||||
(when-let [amount (ffirst (dc/q '[:find (sum ?r)
|
||||
:with ?e
|
||||
@@ -509,7 +517,6 @@
|
||||
@(dc/transact conn [[:upsert-sales-summary result]]))
|
||||
@(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
|
||||
|
||||
|
||||
(defn sales-summaries-v2
|
||||
"Recomputes every dirty summary, client by client."
|
||||
[]
|
||||
|
||||
@@ -27,11 +27,9 @@
|
||||
"Authorization" (str "Bearer " (:client/square-auth-token client))
|
||||
"Content-Type" "application/json"}))
|
||||
|
||||
|
||||
(defn ->square-date [d]
|
||||
(f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d))
|
||||
|
||||
|
||||
(def manifold-api-stream
|
||||
(let [stream (s/stream 100)]
|
||||
(->> stream
|
||||
@@ -42,10 +40,10 @@
|
||||
(de/loop [attempt 0]
|
||||
(-> (de/chain (de/future-with (ex/execute-pool)
|
||||
#_(log/info ::request-started
|
||||
:url (:url request)
|
||||
:attempt attempt
|
||||
:source "Square 3"
|
||||
:background-job "Square 3")
|
||||
:url (:url request)
|
||||
:attempt attempt
|
||||
:source "Square 3"
|
||||
:background-job "Square 3")
|
||||
(try
|
||||
(client/request (assoc request
|
||||
:socket-timeout 10000
|
||||
@@ -104,7 +102,6 @@
|
||||
:exception error))
|
||||
[]))))
|
||||
|
||||
|
||||
(def item-cache (atom {}))
|
||||
|
||||
(defn fetch-catalog [client i v]
|
||||
@@ -124,13 +121,11 @@
|
||||
#(do (swap! item-cache assoc i %)
|
||||
%))))
|
||||
|
||||
|
||||
(defn fetch-catalog-cache [client i version]
|
||||
(if (get @item-cache i)
|
||||
(de/success-deferred (get @item-cache i))
|
||||
(fetch-catalog client i version)))
|
||||
|
||||
|
||||
(defn item->category-name-impl [client item version]
|
||||
(capture-context->lc
|
||||
(cond (:item_id (:item_variation_data item))
|
||||
@@ -161,7 +156,6 @@
|
||||
:item item)
|
||||
"Uncategorized"))))
|
||||
|
||||
|
||||
(defn item-id->category-name [client i version]
|
||||
(capture-context->lc
|
||||
(-> [client i]
|
||||
@@ -226,7 +220,6 @@
|
||||
(concat (:orders result) continued-results))))
|
||||
(:orders result)))))))
|
||||
|
||||
|
||||
(defn search
|
||||
([client location start end]
|
||||
(capture-context->lc
|
||||
@@ -250,11 +243,9 @@
|
||||
(concat (:orders result) continued-results))))
|
||||
(:orders result))))))))
|
||||
|
||||
|
||||
(defn amount->money [amt]
|
||||
(* 0.01 (or (:amount amt) 0.0)))
|
||||
|
||||
|
||||
;; to get totals:
|
||||
(comment
|
||||
(reduce
|
||||
@@ -278,6 +269,30 @@
|
||||
[prefix client location id]
|
||||
(str prefix (:client/code client) "-" (:square-location/client-location location) "-" id))
|
||||
|
||||
(def ^:private owner-attr
|
||||
"Where each Square-imported entity records the client it belongs to."
|
||||
{:charge/external-id :charge/client
|
||||
:sales-refund/external-id :sales-refund/client
|
||||
:expected-deposit/external-id :expected-deposit/client
|
||||
:cash-drawer-shift/external-id :cash-drawer-shift/client})
|
||||
|
||||
(defn- owned-by-other-client?
|
||||
"Whether `e` already belongs to a client other than `client-eid`.
|
||||
|
||||
Reads the entity's own owner attribute, and for a charge falls back to the client of whichever
|
||||
sales order refers to it — charges predating `:charge/client` still have orders, and those are
|
||||
exactly the ones that can be taken by the wrong client."
|
||||
[db attr e client-eid]
|
||||
(let [ent (dc/entity db e)
|
||||
owner (or (:db/id ((owner-attr attr) ent))
|
||||
(when (= attr :charge/external-id)
|
||||
(some->> (first (dc/datoms db :vaet e :sales-order/charges))
|
||||
:e
|
||||
(dc/entity db)
|
||||
:sales-order/client
|
||||
:db/id)))]
|
||||
(and owner (not= owner client-eid))))
|
||||
|
||||
(defn existing-id
|
||||
"Entity id of the refund or charge this id already refers to, trying the client-scoped key
|
||||
first and the legacy unscoped key second.
|
||||
@@ -286,11 +301,24 @@
|
||||
relies on upsert-by-identity; changing the key format on its own would match nothing and
|
||||
Datomic would create a SECOND entity for every refund and charge, orphaning the original under
|
||||
its legacy key. Pinning the result as `:db/id` makes the write land on the existing entity
|
||||
whichever scheme it currently carries."
|
||||
whichever scheme it currently carries.
|
||||
|
||||
The legacy branch will not take a record that already belongs to a different client. Without
|
||||
that check, two clients on one Square location double money during the window between deploying
|
||||
and finishing the migration: client A's payout import resolves B's charge by its bare key and
|
||||
renames it into A's scope, B's next order import then matches neither scheme and mints a second
|
||||
charge, and because `:sales-order/charges` is cardinality-many nothing retracts the first — so
|
||||
B's order carries two charges for one payment. Declining is also the right answer on its merits:
|
||||
the write then lands on this client's own copy, which is what the scoped keys exist to create.
|
||||
|
||||
Once the migration has run there are no legacy keys left for this branch to find, and both it
|
||||
and the guard can be deleted together."
|
||||
[db attr prefix client location id]
|
||||
(when id
|
||||
(or (dc/entid db [attr (scoped-key prefix client location id)])
|
||||
(dc/entid db [attr (str prefix id)]))))
|
||||
(when-let [legacy (dc/entid db [attr (str prefix id)])]
|
||||
(when-not (owned-by-other-client? db attr legacy (:db/id client))
|
||||
legacy)))))
|
||||
|
||||
(defn tender->charge [order client location t]
|
||||
(remove-nils
|
||||
@@ -304,7 +332,7 @@
|
||||
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location (:id t))
|
||||
:external-id (when (:id t)
|
||||
(scoped-key "square/charge/" client location (:id t)))
|
||||
:processor (cond
|
||||
:processor (cond
|
||||
(#{"OTHER" "THIRD_PARTY_CARD"} (:type t))
|
||||
(condp = (some-> (:note t) str/lower-case)
|
||||
"doordash" :ccp-processor/doordash
|
||||
@@ -377,7 +405,7 @@
|
||||
#:sales-order
|
||||
{:date (if (= "Invoices" (:name (:source order)))
|
||||
(when (:closed_at order)
|
||||
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
|
||||
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
|
||||
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:created_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
|
||||
:client (:db/id client)
|
||||
:location (:square-location/client-location location)
|
||||
@@ -439,7 +467,6 @@
|
||||
:client client
|
||||
:location location)))))))
|
||||
|
||||
|
||||
(defn get-payment [client p]
|
||||
(de/chain (manifold-api-call
|
||||
{:url (str "https://connect.squareup.com/v2/payments/" p)
|
||||
@@ -448,7 +475,6 @@
|
||||
:body
|
||||
:payment))
|
||||
|
||||
|
||||
(defn continue-payout-entry-list [c l poi cursor]
|
||||
(capture-context->lc lc
|
||||
(de/chain
|
||||
@@ -593,6 +619,12 @@
|
||||
(let [payment-id (:payment_id (:type_charge_details p))]
|
||||
(remove-nils
|
||||
{:charge/external-id (scoped-key "square/charge/" client location payment-id)
|
||||
;; the owner attributes must travel with the key: `raw-square-id` and
|
||||
;; `scope-of` both recover a charge's scope from them, and a key
|
||||
;; scoped to one client while the owner says another is what makes
|
||||
;; the migration write square/charge/B-LB-A-LA-<id>
|
||||
:charge/client (:db/id client)
|
||||
:charge/location (:square-location/client-location location)
|
||||
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location payment-id)}))))))})
|
||||
(filter :expected-deposit/date)
|
||||
(into []))
|
||||
@@ -600,15 +632,44 @@
|
||||
(log/error ::transform-payout-failed
|
||||
:exception e)))))))))
|
||||
|
||||
(defn refunds
|
||||
([client l]
|
||||
(de/chain (manifold-api-call {:url (str "https://connect.squareup.com/v2/refunds?location_id=" (:square-location/square-id l))
|
||||
:method :get
|
||||
(defn- refund-list
|
||||
"Every refund Square has for this location in `[start end]`, following the cursor to the end.
|
||||
|
||||
:headers (client-base-headers client)
|
||||
:as :json})
|
||||
The list endpoint returns one page at a time. Reading only the first page — which is what this
|
||||
did before — silently caps a location at a hundred refunds however many it actually has, and
|
||||
the cap is invisible: the response looks like a complete answer. On a shared location that is
|
||||
how one client record ends up holding a few refunds against a hundred and fifty thousand orders.
|
||||
|
||||
`start`/`end` are optional; omitting both asks for everything, which is what the nightly job
|
||||
wants and what a historical backfill of more than a page needs."
|
||||
([client l start end] (refund-list client l start end nil))
|
||||
([client l start end cursor]
|
||||
(de/chain (manifold-api-call
|
||||
{:url (str "https://connect.squareup.com/v2/refunds"
|
||||
"?"
|
||||
(url/map->query
|
||||
(cond-> {:location_id (:square-location/square-id l)
|
||||
:limit 100}
|
||||
start (assoc :begin_time (->square-date start))
|
||||
end (assoc :end_time (->square-date end))
|
||||
cursor (assoc :cursor cursor))))
|
||||
:method :get
|
||||
:headers (client-base-headers client)
|
||||
:as :json})
|
||||
:body
|
||||
:refunds
|
||||
(fn [result]
|
||||
(log/info ::refunds-page
|
||||
:count (count (:refunds result))
|
||||
:more? (boolean (not-empty (:cursor result))))
|
||||
(if (not-empty (:cursor result))
|
||||
(de/chain (refund-list client l start end (:cursor result))
|
||||
(fn [more] (concat (:refunds result) more)))
|
||||
(:refunds result))))))
|
||||
|
||||
(defn refunds
|
||||
([client l] (refunds client l nil nil))
|
||||
([client l start end]
|
||||
(de/chain (refund-list client l start end)
|
||||
(fn [refunds]
|
||||
(->> refunds
|
||||
(filter (fn [r] (= "COMPLETED" (:status r))))
|
||||
@@ -651,7 +712,6 @@
|
||||
:count (count x))
|
||||
@(dc/transact-async conn x))))))))
|
||||
|
||||
|
||||
(defn upsert-payouts
|
||||
([client]
|
||||
(apply de/zip
|
||||
@@ -680,11 +740,12 @@
|
||||
(for [square-location (:client/square-locations client)
|
||||
:when (:square-location/client-location square-location)]
|
||||
(upsert-refunds client square-location))))
|
||||
([client location]
|
||||
([client location] (upsert-refunds client location nil nil))
|
||||
([client location start end]
|
||||
(with-context-as {:source "Square refunds loading"
|
||||
:client (:client/code client)} lc
|
||||
|
||||
(de/chain (refunds client location)
|
||||
(de/chain (refunds client location start end)
|
||||
(fn [refunds]
|
||||
(mu/with-context lc
|
||||
(try
|
||||
@@ -700,7 +761,6 @@
|
||||
|
||||
(log/info ::done-loading-refunds)))))))
|
||||
|
||||
|
||||
(defn get-cash-shift [client id]
|
||||
(de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id))
|
||||
:method :get
|
||||
@@ -864,8 +924,6 @@
|
||||
d1
|
||||
d2))
|
||||
|
||||
|
||||
|
||||
(defn remove-voided-orders
|
||||
([client]
|
||||
(apply de/zip
|
||||
@@ -892,7 +950,7 @@
|
||||
(:sales-order/external-id o))))))
|
||||
(s/map (fn [[o]]
|
||||
[[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]]))
|
||||
|
||||
|
||||
(s/reduce into [])))
|
||||
|
||||
(fn [results]
|
||||
@@ -901,31 +959,26 @@
|
||||
(log/info ::removing-orders
|
||||
:count (count x))
|
||||
@(dc/transact-async conn x)))))
|
||||
(de/catch (fn [e]
|
||||
(log/warn ::couldnt-remove :error e)
|
||||
nil) ))))))
|
||||
(de/catch (fn [e]
|
||||
(log/warn ::couldnt-remove :error e)
|
||||
nil)))))))
|
||||
|
||||
#_(comment
|
||||
(require 'auto-ap.time-reader)
|
||||
#_(comment
|
||||
(require 'auto-ap.time-reader)
|
||||
|
||||
@(let [[c [l]] (get-square-client-and-location "DBFS") ]
|
||||
(log/peek :x [ c l])
|
||||
(search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29")
|
||||
@(let [[c [l]] (get-square-client-and-location "DBFS")]
|
||||
(log/peek :x [c l])
|
||||
(search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29"))
|
||||
|
||||
)
|
||||
@(let [[c [l]] (get-square-client-and-location "NGAK")]
|
||||
(log/peek :x [c l])
|
||||
|
||||
@(let [[c [l]] (get-square-client-and-location "NGAK") ]
|
||||
(log/peek :x [ c l])
|
||||
|
||||
(remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
|
||||
(doseq [c (get-square-clients)]
|
||||
(try
|
||||
@(remove-voided-orders c)
|
||||
(catch Exception e
|
||||
nil)))
|
||||
|
||||
|
||||
)
|
||||
(remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
|
||||
(doseq [c (get-square-clients)]
|
||||
(try
|
||||
@(remove-voided-orders c)
|
||||
(catch Exception e
|
||||
nil))))
|
||||
|
||||
(defn upsert-all [& clients]
|
||||
(capture-context->lc
|
||||
@@ -988,14 +1041,53 @@
|
||||
(s/realize-each)
|
||||
(s/reduce conj []))))
|
||||
|
||||
(defn backfill-history
|
||||
"Re-imports orders, payouts, refunds and cash-drawer shifts for `[start end]`, one client at a
|
||||
time, for every square location the client has.
|
||||
|
||||
This exists for the shared-location case. Sales orders have always been keyed by client, so two
|
||||
client records on one Square location each built their own order history. Refunds, payouts and
|
||||
shifts were not, so only ONE of the two records holds each of them — whichever imported it last
|
||||
before the keys were scoped. Re-keying freezes that ownership; it does not even it out, and the
|
||||
record left without them shows returns from its own orders with no refunds to offset them.
|
||||
|
||||
Rather than manufacture copies, this asks Square again. With client-scoped keys in place every
|
||||
record now creates its own copy of what it reads, so replaying the window is what makes the two
|
||||
histories match. Deliberately not part of `upsert-all`: it walks further back than the nightly
|
||||
job and is meant to be run once, after the migration.
|
||||
|
||||
Run it AFTER `rekey-square-external-ids/migrate-all!`. Running it before would import against
|
||||
legacy keys and leave more to migrate."
|
||||
[start end & client-codes]
|
||||
(with-context-as {:source "Square historical backfill"} lc
|
||||
(->> (apply get-square-clients client-codes)
|
||||
(s/->source)
|
||||
(s/map (fn [client]
|
||||
(with-context-as (merge lc {:client (:client/code client)}) lc
|
||||
(->
|
||||
(apply de/zip
|
||||
(for [l (:client/square-locations client)
|
||||
:when (:square-location/client-location l)]
|
||||
(de/chain
|
||||
(upsert client l start end)
|
||||
(fn [_] (upsert-payouts client l start end))
|
||||
(fn [_] (upsert-refunds client l start end))
|
||||
(fn [_] (upsert-cash-shifts client l start end))
|
||||
(fn [_] (log/info ::backfilled
|
||||
:location (:square-location/client-location l))))))
|
||||
(de/catch (fn [e]
|
||||
(mu/with-context lc
|
||||
(log/info ::backfill-failed :severity :error :exception e))))))))
|
||||
(s/buffer 3)
|
||||
(s/realize-each)
|
||||
(s/reduce conj []))))
|
||||
|
||||
(defn do-upsert-all [& clients]
|
||||
(mu/trace
|
||||
::upsert-all
|
||||
[:clients clients]
|
||||
@(apply upsert-all clients)))
|
||||
|
||||
|
||||
|
||||
(comment
|
||||
(defn refunds-raw-cont
|
||||
([client l cursor so-far]
|
||||
@@ -1025,9 +1117,8 @@
|
||||
(->>
|
||||
@(let [[c [l]] (get-square-client-and-location "NGGG")]
|
||||
|
||||
|
||||
(search c l (time/now) (time/plus (time/now) (time/days -1))))
|
||||
|
||||
|
||||
(filter (fn [r]
|
||||
(str/starts-with? (:created_at r) "2024-03-14"))))
|
||||
|
||||
@@ -1035,7 +1126,6 @@
|
||||
(->>
|
||||
@(let [[c [l]] (get-square-client-and-location "NGGG")]
|
||||
|
||||
|
||||
(refunds-raw-cont c l nil []))
|
||||
(filter (fn [r]
|
||||
(str/starts-with? (:created_at r) "2024-03-14")))))
|
||||
@@ -1069,13 +1159,8 @@
|
||||
[]))]
|
||||
[(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row) (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
|
||||
:separator \tab)
|
||||
|
||||
|
||||
|
||||
|
||||
;; =>
|
||||
|
||||
|
||||
;; =>
|
||||
|
||||
(require 'auto-ap.time-reader)
|
||||
|
||||
@@ -1084,27 +1169,16 @@
|
||||
(clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")]
|
||||
l
|
||||
|
||||
|
||||
(def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00"
|
||||
#clj-time/date-time "2025-02-28T00:00:00-08:00"))
|
||||
(take 10 (map #(first (deref (order->sales-order c l %))) z)))
|
||||
(take 10 (map #(first (deref (order->sales-order c l %))) z))))
|
||||
|
||||
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
(->> z
|
||||
(->> z
|
||||
(filter (fn [o]
|
||||
(seq (filter (comp #{"OTHER"} :type) (:tenders o)))))
|
||||
(filter #(not (:name (:source %))))
|
||||
(count)
|
||||
|
||||
)
|
||||
|
||||
|
||||
|
||||
(count))
|
||||
|
||||
(doseq [[code] (seq (dc/q '[:find ?code
|
||||
:in $
|
||||
:where [?o :sales-order/date ?d]
|
||||
@@ -1113,32 +1187,22 @@
|
||||
[?o :sales-order/client ?c]
|
||||
[?c :client/code ?code]]
|
||||
(dc/db conn)))
|
||||
:let [[c [l]] (get-square-client-and-location code)
|
||||
]
|
||||
:let [[c [l]] (get-square-client-and-location code)]
|
||||
order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now))
|
||||
:when (= "Invoices" (:name (:source order) ))
|
||||
:when (= "Invoices" (:name (:source order)))
|
||||
:let [[sales-order] @(order->sales-order c l order)]]
|
||||
|
||||
|
||||
(when (should-import-order? order)
|
||||
(println "DATE IS" (:sales-order/date sales-order))
|
||||
(when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after? #clj-time/date-time "2026-2-16T00:00:00-08:00"))
|
||||
(println "WOULD UPDATE" sales-order)
|
||||
@(dc/transact auto-ap.datomic/conn [sales-order])
|
||||
)
|
||||
#_@(dc/transact )
|
||||
(println "DONE"))
|
||||
|
||||
|
||||
)
|
||||
@(dc/transact auto-ap.datomic/conn [sales-order]))
|
||||
#_@(dc/transact)
|
||||
(println "DONE")))
|
||||
|
||||
#_(filter (comp #{"OTHER"} :type) (mapcat :tenders z))
|
||||
|
||||
|
||||
@(let [[c [l]] (get-square-client-and-location "NGRY")]
|
||||
#_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
|
||||
|
||||
(order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY" )))
|
||||
|
||||
)
|
||||
|
||||
)
|
||||
(order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY")))))
|
||||
|
||||
@@ -147,3 +147,47 @@
|
||||
(is (= (str "square/charge/" code "-CD-same1") (:key (first (charges-of o1)))))
|
||||
(is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
|
||||
"both orders still point at the one payment"))))
|
||||
|
||||
(deftest the-month-walk-runs-newest-first-and-leaves-no-gaps
|
||||
(testing "order matters operationally, not just cosmetically: the importer reads recent data, so
|
||||
an interrupted migration is only safe to resume imports against if the newest months
|
||||
are the ones already done. Walking :aevt instead would start in 2019."
|
||||
(let [windows (sut/order-months-newest-first #inst "2026-01-01T12:00:00.000-00:00")
|
||||
starts (map first windows)]
|
||||
(is (seq windows))
|
||||
(is (apply > (map #(.getTime ^java.util.Date %) starts))
|
||||
"strictly descending — newest month first")
|
||||
(is (every? (fn [[[next-start _] [_ prev-end]]]
|
||||
(= (.getTime ^java.util.Date next-start)
|
||||
(+ (.getTime ^java.util.Date prev-end) (* 24 60 60 1000))))
|
||||
(partition 2 1 windows))
|
||||
"each window ends the day before the next one starts, so no order falls between them")
|
||||
(is (every? (fn [[s e]] (.before ^java.util.Date s ^java.util.Date e)) windows)
|
||||
"and every window is non-empty"))))
|
||||
|
||||
(deftest two-orders-of-the-same-client-keep-sharing-across-batches
|
||||
(testing "batch size does not change the same-client rule, which the sibling test cannot show
|
||||
because both its orders land in one batch.
|
||||
|
||||
Once the first order re-keys the charge it also writes :charge/client/:charge/location,
|
||||
so the second order's raw-square-id takes its owner branch, new-key reconstructs the key
|
||||
the charge already has, and the (not= old new-key) guard drops the row before :action is
|
||||
read. A clone here would double that client's takings for the day."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])
|
||||
tx @(dc/transact conn [{:db/id "charge"
|
||||
:charge/external-id "square/charge/same2"
|
||||
:charge/total 75.0}
|
||||
{:db/id "o1" :sales-order/external-id "square/order/y-1"
|
||||
:sales-order/client test-client-id :sales-order/location "CD"
|
||||
:sales-order/date sales-date :sales-order/charges ["charge"]}
|
||||
{:db/id "o2" :sales-order/external-id "square/order/y-2"
|
||||
:sales-order/client test-client-id :sales-order/location "CD"
|
||||
:sales-order/date sales-date :sales-order/charges ["charge"]}])
|
||||
o1 (get-in tx [:tempids "o1"]) o2 (get-in tx [:tempids "o2"])
|
||||
code (:client/code (dc/entity (dc/db conn) test-client-id))]
|
||||
(is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [o1 o2] 1))
|
||||
"batch size 1 puts the two orders in separate batches, and still no copy is made")
|
||||
(is (= 1 (charge-count)) "one payment, not two")
|
||||
(is (= (str "square/charge/" code "-CD-same2") (:key (first (charges-of o1)))))
|
||||
(is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
|
||||
"both orders still point at the one payment"))))
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
(ns auto-ap.jobs.sales-summaries-test
|
||||
(:require
|
||||
[auto-ap.datomic :refer [conn]]
|
||||
[auto-ap.datomic.sales-summaries :as d-ss]
|
||||
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
|
||||
[auto-ap.jobs.sales-summaries :as sut]
|
||||
[clojure.test :refer [deftest is testing use-fixtures]]
|
||||
@@ -121,6 +122,42 @@
|
||||
:sales-order/service-charge -75.0})])
|
||||
(is (nil? (service-charges-for test-client-id))))))
|
||||
|
||||
(defn- refund
|
||||
"A card refund on `sales-date`. The client+date tuple is set explicitly because
|
||||
`scan-sales-refunds` walks that index rather than the plain attributes."
|
||||
[client id total]
|
||||
{:db/id (str "refund-" id)
|
||||
:sales-refund/external-id (str "square/refund/TEST-" id)
|
||||
:sales-refund/client client
|
||||
:sales-refund/date sales-date
|
||||
:sales-refund/client+date [client sales-date]
|
||||
:sales-refund/type "CARD"
|
||||
:sales-refund/total total})
|
||||
|
||||
(defn- returns-for [client]
|
||||
(#'sut/get-returns client sales-date))
|
||||
|
||||
(deftest a-refund-with-no-sales-leaves-the-day-out-of-balance
|
||||
(testing "deliberate, and load-bearing. Booking a return against the day's refunds would close
|
||||
it and is tempting for that reason. But a day with refunds and no sales at all is
|
||||
overwhelmingly a day whose ORDERS WERE NEVER IMPORTED — on a restored copy of
|
||||
production, 132 of 156 such days fell before their client's first ever synced order.
|
||||
Balancing them would turn the only signal that a client's sales are missing into
|
||||
silence. If this test starts failing, read the rollout plan before changing it."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(refund test-client-id "no-sales" 40.0)])
|
||||
(is (nil? (returns-for test-client-id))
|
||||
"no return is invented for a day that recorded no sales")
|
||||
(is (= -40.0 (d-ss/imbalance (sut/get-refund-items test-client-id sales-date)))
|
||||
"so the day stays out of balance by the refunded amount, visibly"))))
|
||||
|
||||
(deftest a-day-that-traded-books-its-own-return
|
||||
(testing "the ordinary case: the return comes from the day's orders, never from its refunds"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(order test-client-id "traded" {:sales-order/returns 7.0})
|
||||
(refund test-client-id "same-day" 40.0)])
|
||||
(is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id)))))))
|
||||
|
||||
(deftest dirty-summaries-stop-at-the-client-boundary
|
||||
(testing "every dirty day for the client is returned, and none belonging to another client.
|
||||
|
||||
|
||||
@@ -76,6 +76,89 @@
|
||||
:sales-refund/total 10.0}])
|
||||
(is (= 2 (refund-count)) "two stable entities, one per client, rather than one that flips"))))
|
||||
|
||||
(deftest legacy-key-of-another-client-is-not-claimed
|
||||
(testing "the legacy fallback must not hand one client a record that already belongs to another.
|
||||
|
||||
Without this, two clients on one Square location double money in the window between
|
||||
deploying and finishing the migration."
|
||||
(setup-test-data [])
|
||||
(let [tx @(dc/transact conn [{:db/id "mine" :client/code (str "MINE" (rand-int 100000))}
|
||||
{:db/id "theirs" :client/code (str "THEIRS" (rand-int 100000))}])
|
||||
mine {:db/id (get-in tx [:tempids "mine"]) :client/code "MINE"}
|
||||
theirs-id (get-in tx [:tempids "theirs"])]
|
||||
@(dc/transact conn [{:db/id "r"
|
||||
:sales-refund/external-id "square/refund/abc"
|
||||
:sales-refund/client theirs-id
|
||||
:sales-refund/total 10.0}])
|
||||
(is (nil? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/"
|
||||
mine location "abc"))
|
||||
"a legacy-keyed refund owned by another client is left alone")
|
||||
(is (some? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/"
|
||||
{:db/id theirs-id :client/code "THEIRS"} location "abc"))
|
||||
"its own client still resolves it, so re-keying in place still works"))))
|
||||
|
||||
(deftest a-charge-is-owned-by-the-client-of-the-order-that-refers-to-it
|
||||
(testing "charges predating :charge/client still have orders, and those are exactly the ones
|
||||
that could be taken by the wrong client"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])
|
||||
other (get-in @(dc/transact conn [{:db/id "o" :client/code (str "OTHER" (rand-int 100000))}])
|
||||
[:tempids "o"])]
|
||||
@(dc/transact conn [{:db/id "c" :charge/external-id "square/charge/p1" :charge/total 50.0}
|
||||
{:db/id "ord" :sales-order/external-id "square/order/x-1"
|
||||
:sales-order/client test-client-id :sales-order/location "CD"
|
||||
:sales-order/date #inst "2026-06-03T07:00:00.000-00:00"
|
||||
:sales-order/charges ["c"]}])
|
||||
(is (nil? (sut/existing-id (dc/db conn) :charge/external-id "square/charge/"
|
||||
{:db/id other :client/code "OTHER"} location "p1"))
|
||||
"ownership is read from the referencing order when :charge/client is absent"))))
|
||||
|
||||
(deftest deploy-window-does-not-double-a-second-clients-tender
|
||||
(testing "the P0 this guard exists for, end to end.
|
||||
|
||||
Client A's payout import reaches for a payment whose charge belongs to client B's
|
||||
order. If A were allowed to re-key it, B's next order import would match neither
|
||||
scheme, mint a second charge, and — since :sales-order/charges is cardinality-many —
|
||||
leave B's order holding two charges for one payment."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])
|
||||
b-code (:client/code (dc/entity (dc/db conn) test-client-id))
|
||||
b {:db/id test-client-id :client/code b-code}
|
||||
b-loc {:square-location/client-location "LB"}
|
||||
a-id (get-in @(dc/transact conn [{:db/id "a" :client/code (str "AAA" (rand-int 100000))}])
|
||||
[:tempids "a"])
|
||||
a {:db/id a-id :client/code (:client/code (dc/entity (dc/db conn) a-id))}
|
||||
a-loc {:square-location/client-location "LA"}
|
||||
tx @(dc/transact conn [{:db/id "x" :charge/external-id "square/charge/P"
|
||||
:charge/total 100.0 :charge/type-name "CARD"
|
||||
:charge/client test-client-id :charge/location "LB"}
|
||||
{:db/id "ob" :sales-order/external-id "square/order/b-1"
|
||||
:sales-order/client test-client-id :sales-order/location "LB"
|
||||
:sales-order/date #inst "2026-06-03T07:00:00.000-00:00"
|
||||
:sales-order/charges ["x"]}])
|
||||
order-b (get-in tx [:tempids "ob"])
|
||||
charges-of (fn [o] (map :v (dc/datoms (dc/db conn) :eavt o :sales-order/charges)))]
|
||||
;; client A's payout import touches the same Square payment
|
||||
@(dc/transact conn [(into {} (remove (comp nil? val))
|
||||
{:charge/external-id (sut/scoped-key "square/charge/" a a-loc "P")
|
||||
:charge/client a-id
|
||||
:charge/location "LA"
|
||||
:db/id (sut/existing-id (dc/db conn) :charge/external-id
|
||||
"square/charge/" a a-loc "P")})])
|
||||
;; client B's order re-imports
|
||||
@(dc/transact conn [{:db/id order-b
|
||||
:sales-order/charges
|
||||
[(sut/tender->charge {:id "b-1" :created_at "2026-06-03T12:00:00Z"}
|
||||
b b-loc {:id "P" :type "CARD"
|
||||
:amount_money {:amount 10000
|
||||
:currency "USD"}})]}])
|
||||
(is (= 1 (count (charges-of order-b)))
|
||||
"B's order still holds exactly one charge for the one payment")
|
||||
(is (= 100.0 (reduce + 0.0 (map #(:charge/total (dc/entity (dc/db conn) %))
|
||||
(charges-of order-b))))
|
||||
"so the day's tender is not doubled")
|
||||
(is (= (str "square/charge/" b-code "-LB-P")
|
||||
(:charge/external-id (dc/entity (dc/db conn) (first (charges-of order-b)))))
|
||||
"and B's own charge was re-keyed in place rather than abandoned"))))
|
||||
|
||||
(deftest payouts-and-shifts-are-client-scoped-too
|
||||
(testing "expected deposits and cash drawer shifts are fetched per location, so two clients on
|
||||
one location collide on them exactly as refunds and charges did"
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
(ns auto-ap.jobs.compare-sales-summaries
|
||||
(ns auto-ap.tools.compare-sales-summaries
|
||||
"Compares sales summaries between two points in the same database.
|
||||
|
||||
A verification tool, not part of the running application: it lives on the test/dev classpath so
|
||||
nothing in production can depend on it. Load it from a REPL when auditing a recompute.
|
||||
|
||||
The question this exists to answer is narrower than \"did the totals improve\": it is *which
|
||||
days changed, and were any of them already balanced*. A day that was balanced before and still
|
||||
balances after can still have had its line amounts move, and that is a real change to the
|
||||
Reference in New Issue
Block a user