notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

docs: bring the rollout plan's later steps up to date with the backfill

Browse Source 
The headline and the new backfill step were current, but step 11 and
"what this will not fix" still carried pre-backfill figures. The backfill
did not just change the totals — it changed step 11's population.

Before it, nine records held refunds dated before their own first order:
660 refunds worth $15,237.02. Seven of those were shared-location twins
whose refunds only looked orphaned because their orders had never been
imported; replaying the window gave them their orders and the refunds
stopped predating them.

Two are left, and they are a different case — neither shares a Square
location, so there is no twin holding the other half:

  NG4S  first order 2026-05-29  79 refunds  $2,180.08  10 days
  NGPS  first order 2026-05-26  81 refunds  $2,167.60   7 days

Step 11 now recommends trying backfill-history on them first, with a
window reaching back before their first order, since that is exactly what
resolved the other seven.

"What this will not fix" re-measured: 122 days / $2,379.45, of which 106
are real trading days ($1,151.80) and 16 are refunds with no sales
imported ($1,227.65). Only 3 of the trading days are on shared-location
records, all already diagnosed. The other 103 days and $502.96 have been
identical in every run of this analysis — deactivated, live, and
backfilled — and are the floor this work reaches.

Also adds the backfill's ~5.9 hour runtime to the up-front table, and
notes that "two entities per Square object" holds automatically for new
imports but needs step 5 for existing history.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Bryce
2026-08-16 19:04:50 -07:00
parent 1558851c18
commit 79a4b457b0
1 changed files with 48 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

85
docs/2026-08-15-sales-summary-rollout-plan.md
View File

@@ -27,6 +27,7 @@ each its own copy of the refunds.
| Flag introduced | `summary-service-charges` — off by default |
| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` |
| Expected migration runtime | ~38 minutes for 19M orders on a warm cache |
| Backfill runtime (step 5) | ~5.9 hours for 90 days across the 20 shared-location records — an overnight job |
| Nothing here touches | invoices, payments, the ledger, or any client without the flag set |
**Client configuration is left exactly as it is.** Ten Square locations are configured against two
@@ -40,6 +41,11 @@ becomes **two entities, one per client record** — by design. That is the stabl
duplicate to clean up. If any report or export aggregates across client records, one restaurant's
takings would be counted twice at that layer. Nothing in this work changes that either way.
That holds automatically for everything imported *from now on*, because the keys carry the client.
It does **not** hold for history: refunds, payouts and shifts already in the database exist only
once, on whichever record imported them last, and re-keying freezes that rather than evening it out.
Step 5 is what brings the existing history into the same shape.
**The only window of risk is between deploying and finishing the migration**, while legacy keys
still exist for a client to resolve. Steps 2–6 exist to make that window effectively zero.
@@ -332,56 +338,51 @@ people doing the right thing. That is the reason not to let this drift.
**The most important item in this document, and the only one that is not just execution.**
171 of the 542 remaining days are a record carrying refunds on a day it recorded no sales at all.
140 of those fall *before that client's first ever order*. Nine records are affected, holding
**660 refunds worth $15,237.02 dated before their own first order**:
16 of the 122 remaining days are a record carrying refunds on a day it recorded no sales at all,
and all 16 fall before that client's first ever order. Two clients are affected, holding **160
refunds worth $4,347.68 dated before their own first order**:
| Client | First order | Refunds before it | Value |
|---|---|---:|---:|
| NGDL | 2026-07-29 | 96 | $2,751.50 |
| NGWN | 2026-07-30 | 95 | $2,218.61 |
| NGDU | 2026-08-02 | 94 | $2,188.44 |
| NG4S | 2026-05-29 | 79 | $2,180.08 |
| NGPS | 2026-05-26 | 81 | $2,167.60 |
| NGCT | 2026-08-01 | 98 | $1,789.49 |
| NGVC | 2026-08-02 | 95 | $1,594.17 |
| NGDS | 2026-01-14 | 22 | $347.13 |
| Client | First order | Refunds before it | Value | Days out of balance |
|---|---|---:|---:|---:|
| NG4S | 2026-05-29 | 79 | $2,180.08 | 10 |
| NGPS | 2026-05-26 | 81 | $2,167.60 | 7 |
They did not make those refunds. The database's own ownership history shows a $35.35 refund dated
26 February belonging to `NGDG` that day and taken over by `NGDU` on 12 August, with others flipping
between the two several times a day across 12–15 August. Unscoped refund keys meant whichever import
ran last took ownership — fault 1, seen from the other end. Steps 1 and 4 stop it recurring; they do
not undo it.
**Step 5's backfill already resolved the other seven.** Before it, nine records were in this state
holding 660 refunds worth $15,237.02 — but seven of them were shared-location twins whose refunds
only looked orphaned because their orders had never been imported. Replaying the window gave them
their orders, and the refunds stopped predating them.
Note `NG4S` and `NGPS` are **not** among the ten shared locations, so this is wider than the
duplicate records.
NG4S and NGPS are different: neither shares a Square location, so there is no twin holding the other
half. Their sales genuinely are not in the system for the period their refunds cover. The database's
own ownership history is the evidence to check — for the twins it showed refunds changing hands
between the two records; for these two there is no second record to have taken them from.
Two ways to close it, and the business has to pick:
1. **Import the missing sales.** Correct if these records are meant to have their own books. Their
refunds are real and the matching orders exist in Square.
2. **Move the refunds to the record that has the sales.** Correct if the record is a duplicate that
should never have had books of its own.
1. **Import the missing sales.** Correct if these records are meant to have their own books. Try
`backfill-history` for them first, with a window reaching back before their first order — that is
exactly what fixed the seven, and it is one command.
2. **Move the refunds to the record that has the sales.** Correct only if the refunds were misfiled
onto a record that should not have books of its own.
Before either, check whether the surviving record already holds the same Square refund — within the
ninety-day window all 192 orphans did, but **outside it payouts and shifts have no counterpart at
all**, so a blanket delete would destroy the only copy of some records.
Start with (1): it is cheap, reversible in the sense that it only adds what Square reports, and it
is already proven to work on this exact symptom.
```clojure
;; per client: refunds dated before that client's own first order
(let [first-order (->> (d/q '[:find [?d ...] :in $ ?c
:where [?o :sales-order/client ?c] [?o :sales-order/date ?d]]
(d/db conn) [:client/code "NGDU"])
(d/db conn) [:client/code "NG4S"])
(reduce (fn [a b] (if (.before a b) a b))))]
(->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c
:where [?r :sales-refund/client ?c]]
(d/db conn) [:client/code "NGDU"])
(d/db conn) [:client/code "NG4S"])
(filter #(.before (:sales-refund/date %) first-order))
count))
```
**Until this is resolved those days stay out of balance, on purpose.** A summary change to close
them was written and measured — it works, closes 171 days and $5,795.18, and breaks nothing — and it
them was written and measured — it works, closes 16 days and $1,227.65, and breaks nothing — and it
was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not
being imported. A test asserts the day stays unbalanced so nobody closes it without reading this.
@@ -389,14 +390,24 @@ being imported. A test asserts the day stays unbalanced so nobody closes it with
## What this will not fix
The 371 real trading days still out of balance, $14,443.82 — of which $13,940.86 is on the twenty
shared-location records, where the two books disagree about who owns what. For every other client
it is 103 days and $502.96. Named causes: the NGBR/NGBK tender gap ($6,358.99 of
tender against $6,059.57 of order totals on one day), the ezCater fee question, and unexplained
clusters on NGMV and NGEB.
122 client-days over ninety days, $2,379.45, of which only 32 are above ten cents.
Plus 16 days / $1,227.65 where a processing fee lands on a day with no trading — the same shape as
step 11 but from the payout side, so it needs the payout modelled rather than a rule in the summary.
| | Days | Variance | |
|---|---:|---:|---|
| Real trading days with genuine discrepancies | 106 | $1,151.80 | see below |
| Refunds on a record with no sales imported | 16 | $1,227.65 | step 11 — deliberately visible |
Of the 106 trading days, only **3 are on shared-location records** — $648.84 in total, and all three
are already diagnosed: NGBK and NGBR at $299.42 each on 2026-08-06, where Square recorded $6,358.99
of tender against $6,059.57 of order totals (the gap itself, not a summary fault), and NGDA at
$50.00, an auto-gratuity booked as a service charge.
The other 103 days come to **$502.96 across 190 clients** — a few dollars here and there, mostly
till rounding, plus small undiagnosed clusters on NGMV ($259.38 over 5 days) and NGEB ($199.09 over
4 days, an ezCater fee-treatment question). Those two are worth a look but are not urgent.
That 103-day, $502.96 figure has been identical in every run of this analysis — with the duplicates
deactivated, with them live, and with them backfilled. It is the floor this work reaches.
---