docs/2026-08-15-sales-summary-rollout-plan.md

@@ -109,6 +109,26 @@ the 13-minute walk rather than partway through it.
 Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that
 already carries the right name is skipped, so it can be stopped and re-run without consequence.
 
+**It is also ordered so that stopping early is survivable.** Refunds, payouts and cash-drawer
+shifts go first — a quarter of a million records, seconds of work — so an interruption cannot catch
+them half done. The long part then walks orders **a month at a time, from the current month
+backwards**, logging `::month-complete` as each finishes:
+
+```
+::month-complete :month "2026-08" :rekeyed 118203 :cloned 2244
+::month-complete :month "2026-07" :rekeyed 241887 :cloned 4611
+...
+```
+
+That ordering is the recovery plan. If it dies, everything from the last logged month forward is
+fully scoped — and that recent window is what the importer actually reads — so **you can resume
+imports against a partially migrated database** and finish the older tail later. Walking oldest
+first would have spent the first several hours on 2019 data no import will touch, leaving exactly
+the wrong end done.
+
+If you do resume imports mid-migration, the ownership guard in `existing-id` is what keeps the
+unmigrated tail safe: a client cannot resolve onto another client's legacy-keyed record.
+
 If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
 not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
 retained memory in the REPL and re-measure before changing anything about the database.
@@ -157,7 +177,13 @@ lives. Completeness across all of history is check (a)'s job, not this one.
 
 ## Step 5 — Resume the Square importer
 
-Only once step 4's two checks read clean. The maintenance window ends here.
+Normally: once step 4's two checks read clean. The maintenance window ends here.
+
+**If the migration did not finish**, you do not have to wait for it. Resume imports once the
+`::month-complete` log covers the window your importer reads — the last 75 days for payouts and
+cash-drawer shifts, and whatever range the order import is configured for. Then re-run
+`migrate-all!` afterwards to walk the remaining older months; it will skip everything already done.
+Run the step 4 checks again once it does finish.
 
 The first cycle after resuming is the one to watch. Compare these against the same counts taken
 immediately before the deploy — growth should be ordinary daily volume:

src/clj/auto_ap/jobs/rekey_square_external_ids.clj

@@ -20,7 +20,8 @@
   (:require
    [auto-ap.datomic :refer [conn]]
    [auto-ap.logging :as alog]
-   [datomic.api :as dc]))
+   [datomic.api :as dc]
+   [iol-ion.query]))
 
 (def refund-prefix "square/refund/")
 (def charge-prefix "square/charge/")
@@ -288,10 +289,51 @@
                         (dc/datoms db :aevt attr))])))
 
 (defn all-order-ids
-  "Every sales order in the database, streamed."
+  "Every sales order in the database, streamed in `:aevt` order — which is ascending entity id,
+  so OLDEST first. Fine for counting; wrong for anything that takes a prefix. `(take n ...)` of
+  this returns the oldest n orders, not a sample: on the production copy the first 400,000 are
+  all from 2019 to 2021. Use `order-months-newest-first` to walk the data in migration order."
   [db]
   (map :e (dc/datoms db :aevt :sales-order/external-id)))
 
+(def ^:private earliest-orders
+  "How far back the month walk goes. Comfortably before the oldest order in the database
+  (2019-12-31 on the production copy); months with no orders cost one index seek per client."
+  #inst "2015-01-01T00:00:00.000-00:00")
+
+(defn- ->date [^java.time.LocalDate d]
+  (java.util.Date/from (.toInstant (.atStartOfDay d (java.time.ZoneId/systemDefault)))))
+
+(defn order-months-newest-first
+  "`[start end]` month windows from now back to `earliest`, newest month first.
+
+  The migration walks months in this order on purpose. It is the difference between an
+  interrupted run leaving the data safe to import against and leaving it dangerous: the importer
+  works on recent data, so having the newest months fully scoped is what lets imports resume
+  while the older tail is still unmigrated. Walking oldest-first would spend hours on 2019 before
+  touching anything this month's import will read.
+
+  Windows tile without gaps — each month's end is the day before the next month's start — and
+  because the migration is idempotent an order landing in two windows is a no-op the second
+  time, so boundary precision is not safety-critical."
+  ([] (order-months-newest-first earliest-orders))
+  ([^java.util.Date earliest]
+   (let [zone (java.time.ZoneId/systemDefault)
+         floor (java.time.YearMonth/from (.toLocalDate (.atZone (.toInstant earliest) zone)))]
+     (->> (iterate (fn [^java.time.YearMonth m] (.minusMonths m 1)) (java.time.YearMonth/now zone))
+          (take-while (fn [^java.time.YearMonth m] (not (.isBefore m floor))))
+          (map (fn [^java.time.YearMonth m]
+                 [(->date (.atDay m 1)) (->date (.atEndOfMonth m))]))))))
+
+(defn orders-in-window
+  "Sales order ids for every client between `start` and `end` inclusive, via the
+  `:sales-order/client+date` index."
+  [db clients start end]
+  (map first (iol-ion.query/scan-sales-orders db clients start end)))
+
+(defn- all-client-ids [db]
+  (map first (dc/q '[:find ?c :where [?c :client/code _]] db)))
+
 (defn migrate-all!
   "The complete migration, over the whole database rather than a chosen subset.
 
@@ -300,16 +342,33 @@
   Nine client pairs contended in the past and no longer share one; their records are still mixed,
   and a migration scoped to the current configuration would miss every one of them.
 
+  **Ordered so that an interrupted run is recoverable.** Refunds, payouts and cash-drawer shifts
+  go first: together they are a quarter of a million records and take seconds, so finishing them
+  up front means an interruption cannot leave them half done. The long part — walking every order
+  to split shared charges — then runs a month at a time from the current month backwards, logging
+  each month as it completes. Stop it after any month and the data from that month forward is
+  fully scoped, which is the part the importer reads, so imports can resume against it while the
+  older tail waits. Re-running picks up where it left off because each month's work is idempotent.
+
   Returns the split counts and the completeness report, which should read zero legacy across the
   board when this finishes."
   [batch-size]
-  (let [split (split-and-rekey-charges! (all-order-ids (dc/db conn)) batch-size)]
   (doseq [{:keys [attr prefix]} scoped-attrs
           :when (not= attr :charge/external-id)]
     (let [p (plan (dc/db conn) attr prefix)]
       (when-let [c (seq (collisions (:new-keys p)))]
         (throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
       (migrate! attr (:new-keys p) batch-size)))
+  (let [clients (all-client-ids (dc/db conn))
+        split (reduce (fn [acc [start end]]
+                        (let [ids (orders-in-window (dc/db conn) clients start end)
+                              r (split-and-rekey-charges! ids batch-size)]
+                          (alog/info ::month-complete
+                                     :month (subs (str (.toInstant ^java.util.Date start)) 0 7)
+                                     :rekeyed (:rekeyed r) :cloned (:cloned r))
+                          (merge-with + acc r)))
+                      {:rekeyed 0 :cloned 0}
+                      (order-months-newest-first))]
     ;; charges no order refers to — payout stubs — are scoped from the deposit that holds them.
     ;; Collision-checked like the others: this is the largest attribute in the database, so it is
     ;; the last one that should discover a clash as a mid-run exception.

test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj

@@ -148,6 +148,23 @@
       (is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
           "both orders still point at the one payment"))))
 
+(deftest the-month-walk-runs-newest-first-and-leaves-no-gaps
+  (testing "order matters operationally, not just cosmetically: the importer reads recent data, so
+           an interrupted migration is only safe to resume imports against if the newest months
+           are the ones already done. Walking :aevt instead would start in 2019."
+    (let [windows (sut/order-months-newest-first #inst "2026-01-01T12:00:00.000-00:00")
+          starts (map first windows)]
+      (is (seq windows))
+      (is (apply > (map #(.getTime ^java.util.Date %) starts))
+          "strictly descending — newest month first")
+      (is (every? (fn [[[next-start _] [_ prev-end]]]
+                    (= (.getTime ^java.util.Date next-start)
+                       (+ (.getTime ^java.util.Date prev-end) (* 24 60 60 1000))))
+                  (partition 2 1 windows))
+          "each window ends the day before the next one starts, so no order falls between them")
+      (is (every? (fn [[s e]] (.before ^java.util.Date s ^java.util.Date e)) windows)
+          "and every window is non-empty"))))
+
 (deftest two-orders-of-the-same-client-keep-sharing-across-batches
   (testing "batch size does not change the same-client rule, which the sibling test cannot show
            because both its orders land in one batch.