feat(square): walk the migration newest month first, so stopping early is safe
migrate-all! drove the charge split from all-order-ids, which streams :aevt — ascending entity id, so oldest first. On the production copy that means the first several hours are spent on 2019 and 2020 data no import will ever read, leaving the recent end (the part the importer actually touches) for last. Interrupt it there and the data is unmigrated exactly where it matters. Now: - refunds, payouts and cash-drawer shifts run first. Together they are ~266k records and take seconds, so an interruption cannot leave them half done. - the long order walk then runs a month at a time from the current month backwards, logging ::month-complete with per-month counts. Stop it after any month and everything from that month forward is fully scoped, so imports can resume against a partially migrated database and the older tail can be finished later — the re-run skips what is done. While a tail remains unmigrated, existing-id's ownership guard is what keeps it safe. order-months-newest-first tiles [start end] windows with no gaps (each month ends the day before the next begins) and is bounded below by a constant comfortably older than the oldest order. Windows are walked via the :sales-order/client+date index. Verified against the restored copy: 141 windows from 2026-08 back, August returning 241,126 orders and July 476,235, each dated inside its window. all-order-ids keeps its old behaviour but its docstring now warns that a prefix of it is the oldest orders, not a sample — the trap that made an earlier verification gate read 2019-2021 data. 31 tests, 76 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -109,6 +109,26 @@ the 13-minute walk rather than partway through it.
|
||||
Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that
|
||||
already carries the right name is skipped, so it can be stopped and re-run without consequence.
|
||||
|
||||
**It is also ordered so that stopping early is survivable.** Refunds, payouts and cash-drawer
|
||||
shifts go first — a quarter of a million records, seconds of work — so an interruption cannot catch
|
||||
them half done. The long part then walks orders **a month at a time, from the current month
|
||||
backwards**, logging `::month-complete` as each finishes:
|
||||
|
||||
```
|
||||
::month-complete :month "2026-08" :rekeyed 118203 :cloned 2244
|
||||
::month-complete :month "2026-07" :rekeyed 241887 :cloned 4611
|
||||
...
|
||||
```
|
||||
|
||||
That ordering is the recovery plan. If it dies, everything from the last logged month forward is
|
||||
fully scoped — and that recent window is what the importer actually reads — so **you can resume
|
||||
imports against a partially migrated database** and finish the older tail later. Walking oldest
|
||||
first would have spent the first several hours on 2019 data no import will touch, leaving exactly
|
||||
the wrong end done.
|
||||
|
||||
If you do resume imports mid-migration, the ownership guard in `existing-id` is what keeps the
|
||||
unmigrated tail safe: a client cannot resolve onto another client's legacy-keyed record.
|
||||
|
||||
If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
|
||||
not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
|
||||
retained memory in the REPL and re-measure before changing anything about the database.
|
||||
@@ -157,7 +177,13 @@ lives. Completeness across all of history is check (a)'s job, not this one.
|
||||
|
||||
## Step 5 — Resume the Square importer
|
||||
|
||||
Only once step 4's two checks read clean. The maintenance window ends here.
|
||||
Normally: once step 4's two checks read clean. The maintenance window ends here.
|
||||
|
||||
**If the migration did not finish**, you do not have to wait for it. Resume imports once the
|
||||
`::month-complete` log covers the window your importer reads — the last 75 days for payouts and
|
||||
cash-drawer shifts, and whatever range the order import is configured for. Then re-run
|
||||
`migrate-all!` afterwards to walk the remaining older months; it will skip everything already done.
|
||||
Run the step 4 checks again once it does finish.
|
||||
|
||||
The first cycle after resuming is the one to watch. Compare these against the same counts taken
|
||||
immediately before the deploy — growth should be ordinary daily volume:
|
||||
|
||||
Reference in New Issue
Block a user