From 95d0537c6993a9c729c63e9449cb8f7ba5f187bf Mon Sep 17 00:00:00 2001 From: Bryce Date: Wed, 12 Aug 2026 15:01:10 -0700 Subject: [PATCH 1/8] feat(sales-summaries): refresh on a schedule, skipping accepted days sales-summaries-v2 recomputes every dirty summary, but nothing set the dirty flag on a schedule: mark-all-dirty was only ever called by hand from the comment block, and the job was registered in neither server.clj's INTEGREAT_JOB dispatch nor terraform/deploy.tf, so -main was dead code that could never run in production. Summaries were only recalculated when someone remembered to do it in the REPL, and POS data keeps arriving after a business day closes, so a summary computed once on the day was routinely wrong and stayed wrong. Add a daily job that marks the trailing 7 days dirty and recomputes them, leaving finished work alone. "Finished" is the condition the app already calls Balanced -- debits equal credits and every line is mapped to an account. Since that is derived rather than stored, a summary that later falls out of balance is picked up again on the next run. Extract the Balanced predicate into auto-ap.datomic.sales-summaries so the grid's pill and the job share one definition, rather than a background job requiring an SSR namespace. total-debits/total-credits resolve the ledger side from either a plain keyword or the {:db/ident ...} map a pull returns, and accepted? requires every item to declare a side: un-normalized pulled items otherwise sum to 0.0 on both sides, read as balanced, and get skipped silently and permanently. Also fix sales-summaries-v2 destroying user-entered line items. It filtered for :sales-summary-item/manual? to preserve them, but dirty-sales-summaries' index-pull selector never fetched :sales-summary/items, so manual-items was always empty. Because items is a component attribute upserted via [:reset-rels ...], every recompute deleted the hand-entered lines -- often the very lines that make a summary balance. Harmless while nothing ran on a schedule; destructive the moment this does. Register the job in the admin Background Jobs dropdown too, with a days field: schedules are prod-only, so the admin page is the only way to run it on staging. Co-Authored-By: Claude Opus 5 --- src/clj/auto_ap/datomic/sales_summaries.clj | 55 ++++ src/clj/auto_ap/jobs/sales_summaries.clj | 308 +++++++++++------- src/clj/auto_ap/server.clj | 29 +- src/clj/auto_ap/ssr/admin/background_jobs.clj | 206 ++++++------ src/clj/auto_ap/ssr/pos/sales_summaries.clj | 91 +++--- terraform/deploy.tf | 14 + 6 files changed, 418 insertions(+), 285 deletions(-) create mode 100644 src/clj/auto_ap/datomic/sales_summaries.clj diff --git a/src/clj/auto_ap/datomic/sales_summaries.clj b/src/clj/auto_ap/datomic/sales_summaries.clj new file mode 100644 index 00000000..c8d2dd69 --- /dev/null +++ b/src/clj/auto_ap/datomic/sales_summaries.clj @@ -0,0 +1,55 @@ +(ns auto-ap.datomic.sales-summaries + (:require + [iol-ion.query :refer [dollars=]])) + +(defn- ledger-side + "The ledger side of an item as a keyword, whether it arrived as a plain keyword (a + transaction map, or a pull using `:xform iol-ion.query/ident`) or as the `{:db/ident ...}` + map a plain `pull` returns. Resolving both shapes here matters: items whose side does not + compare equal are counted on neither side, which would leave a summary looking balanced at + zero and therefore silently accepted." + [item] + (let [side (:ledger-mapped/ledger-side item)] + (if (map? side) (:db/ident side) side))) + +(defn- side-total [side items] + (->> items + (filter #(= side (ledger-side %))) + (map #(:ledger-mapped/amount % 0.0)) + (reduce + 0.0))) + +(defn total-debits [items] + (side-total :ledger-side/debit items)) + +(defn total-credits [items] + (side-total :ledger-side/credit items)) + +(defn fully-mapped? [items] + (every? :ledger-mapped/account items)) + +(defn fully-sided? + "Every item says which side of the ledger it belongs on. Guards `accepted?` against + reading a collection of sideless items as balanced at zero." + [items] + (every? #(#{:ledger-side/debit :ledger-side/credit} (ledger-side %)) items)) + +(defn balanced? [items] + (dollars= (total-debits items) (total-credits items))) + +(defn accepted? + "True once a summary is finished: every line is mapped to an account and debits equal + credits. This is the same condition the sales summaries grid renders as \"Balanced\", and + the condition the scheduled refresh treats as \"leave this alone\"." + [items] + (boolean (and (seq items) + (fully-mapped? items) + (fully-sided? items) + (balanced? items)))) + +(defn <-pulled-item + "Flattens the ref values on a pulled sales summary item back to the scalars a transaction + expects. `accepted?` reads either shape, so this is only needed on the write path." + [item] + (cond-> item + (map? (:ledger-mapped/ledger-side item)) (update :ledger-mapped/ledger-side :db/ident) + (map? (:ledger-mapped/account item)) (update :ledger-mapped/account :db/id))) diff --git a/src/clj/auto_ap/jobs/sales_summaries.clj b/src/clj/auto_ap/jobs/sales_summaries.clj index e759078b..52391a1b 100644 --- a/src/clj/auto_ap/jobs/sales_summaries.clj +++ b/src/clj/auto_ap/jobs/sales_summaries.clj @@ -1,5 +1,6 @@ (ns auto-ap.jobs.sales-summaries (:require [auto-ap.datomic :refer [conn]] + [auto-ap.datomic.sales-summaries :as d-ss] [auto-ap.jobs.core :refer [execute]] [auto-ap.logging :as alog] [auto-ap.time :as atime] @@ -8,6 +9,7 @@ [clj-time.periodic :as per] [clojure.string :as str] [com.brunobonacci.mulog :as mu] + [config.core :refer [env]] [datomic.api :as dc])) (defn mark-dirty [client start end] @@ -39,27 +41,100 @@ (dc/db conn) number))) - (defn delete-all [] @(dc/transact-async conn - (->> - (dc/q '[:find ?ss - :where [?ss :sales-summary/date]] - (dc/db conn)) - (map (fn [[ ss]] - [:db/retractEntity ss]))))) - + (->> + (dc/q '[:find ?ss + :where [?ss :sales-summary/date]] + (dc/db conn)) + (map (fn [[ss]] + [:db/retractEntity ss]))))) +(def item-read + "Enough of a summary item to both evaluate `d-ss/accepted?` and transact the item back + unchanged. `:db/id` matters: `:sales-summary/items` is a component attribute upserted via + `[:reset-rels ...]`, so an item re-transacted without its id is deleted and recreated." + '[:db/id + :sales-summary-item/category + :sales-summary-item/sort-order + :sales-summary-item/manual? + :ledger-mapped/amount + {:ledger-mapped/ledger-side [:db/ident]} + {:ledger-mapped/account [:db/id]}]) (defn dirty-sales-summaries [c] (let [client-id (dc/entid (dc/db conn) c)] (->> (dc/index-pull (dc/db conn) {:index :avet - :selector '[:sales-summary/date :sales-summary/client :db/id] + :selector (conj '[:sales-summary/date :sales-summary/client :db/id] + {:sales-summary/items item-read}) :start [:sales-summary/client+dirty [client-id true]]}) (filter (fn [sales-summary] (= client-id (:db/id (:sales-summary/client sales-summary)))))))) +(def default-days + "How far back the scheduled refresh looks for summaries that still need recomputing." + 7) + +(defn trailing-window + "`[start end)` covering the last `days` business days, ending with today. `end` is + exclusive, matching both `periodic-seq`'s 3-arity and the grid's date filters, so + `(trailing-window 7)` is day -6 through today inclusive." + [days] + [(.toDateMidnight (atime/localize (time/minus (time/now) (time/days (dec days))))) + (.toDateMidnight (atime/localize (time/plus (time/now) (time/days 1))))]) + +(defn accepted-client+dates + "Set of `[client-id date]` pairs in `[start end)` whose summary is already accepted, and + so should be left alone rather than re-marked. Accepted means balanced with every line + mapped to an account — the condition the grid renders as \"Balanced\"." + [db start end] + (->> (dc/q '[:find (pull ?ss selector) + :in $ ?start ?end selector + :where + [?ss :sales-summary/date ?d] + [(>= ?d ?start)] + [(< ?d ?end)]] + db + (c/to-date start) + (c/to-date end) + (conj '[:sales-summary/date {:sales-summary/client [:db/id]}] + {:sales-summary/items item-read})) + (map first) + (filter #(d-ss/accepted? (map d-ss/<-pulled-item (:sales-summary/items %)))) + (map (juxt (comp :db/id :sales-summary/client) :sales-summary/date)) + set)) + +(defn mark-stale-dirty + "Marks every client/day in the trailing `days` window dirty so `sales-summaries-v2` will + recompute it, skipping days whose summary is already accepted. Because accepted is derived + rather than stored, a summary that later falls out of balance is picked up again on the + next run. Returns the number of client/days marked." + [days] + (let [db (dc/db conn) + [start end] (trailing-window days) + accepted (accepted-client+dates db start end) + clients (map first (dc/q '[:find ?c + :in $ + :where [_ :sales-order/client ?c]] + db)) + dates (map c/to-date (per/periodic-seq start end (time/days 1))) + tx-data (for [client clients + date dates + :when (not (accepted [client date]))] + {:sales-summary/client client + :sales-summary/date date + :sales-summary/dirty true + :sales-summary/client+date [client date]})] + (alog/info ::marking-dirty + :days days + :client-count (count clients) + :accepted-count (count accepted) + :marked (count tx-data)) + (doseq [batch (partition-all 500 tx-data)] + @(dc/transact conn batch)) + (count tx-data))) + (defn- get-fee [c date] (- (or (ffirst (dc/q '[:find ?f :in $ ?client ?d @@ -99,53 +174,53 @@ "food app refunds" 41400}) (defn get-payment-items [c date] - (->> - (dc/q '[:find ?processor ?type-name (sum ?total) - :with ?c - :in $ [?clients ?start-date ?end-date] - :where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] - [?e :sales-order/charges ?c] - [?c :charge/type-name ?type-name] - (or-join [?c ?processor] - (and [?c :charge/processor ?p] - [?p :db/ident ?processor]) - (and - (not [?c :charge/processor]) - [(ground :ccp-processor/na) ?processor])) - [?c :charge/total ?total]] - (dc/db conn) - [[c] date date]) - (reduce - (fn [acc [processor type-name total]] - (update - acc - (cond (= type-name "CARD") - "Card Payments" - (= type-name "CASH") - "Cash Payments" - (#{"SQUARE_GIFT_CARD" "WALLET" "GIFT_CARD"} type-name) - "Gift Card Payments" - (#{:ccp-processor/toast - #_:ccp-processor/ezcater - #_:ccp-processor/koala - :ccp-processor/doordash - :ccp-processor/grubhub - :ccp-processor/uber-eats} processor) - "Food App Payments" - :else - "Unknown") - (fnil + 0.0) - total)) - {}) - (map (fn [[k v]] - {:db/id (str (java.util.UUID/randomUUID)) - :sales-summary-item/sort-order 0 - :sales-summary-item/category k - - :ledger-mapped/amount (if (= "Card Payments" k) - (- v (get-fee c date)) - v) - :ledger-mapped/ledger-side :ledger-side/debit})))) + (->> + (dc/q '[:find ?processor ?type-name (sum ?total) + :with ?c + :in $ [?clients ?start-date ?end-date] + :where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] + [?e :sales-order/charges ?c] + [?c :charge/type-name ?type-name] + (or-join [?c ?processor] + (and [?c :charge/processor ?p] + [?p :db/ident ?processor]) + (and + (not [?c :charge/processor]) + [(ground :ccp-processor/na) ?processor])) + [?c :charge/total ?total]] + (dc/db conn) + [[c] date date]) + (reduce + (fn [acc [processor type-name total]] + (update + acc + (cond (= type-name "CARD") + "Card Payments" + (= type-name "CASH") + "Cash Payments" + (#{"SQUARE_GIFT_CARD" "WALLET" "GIFT_CARD"} type-name) + "Gift Card Payments" + (#{:ccp-processor/toast + #_:ccp-processor/ezcater + #_:ccp-processor/koala + :ccp-processor/doordash + :ccp-processor/grubhub + :ccp-processor/uber-eats} processor) + "Food App Payments" + :else + "Unknown") + (fnil + 0.0) + total)) + {}) + (map (fn [[k v]] + {:db/id (str (java.util.UUID/randomUUID)) + :sales-summary-item/sort-order 0 + :sales-summary-item/category k + + :ledger-mapped/amount (if (= "Card Payments" k) + (- v (get-fee c date)) + v) + :ledger-mapped/ledger-side :ledger-side/debit})))) (defn get-discounts [c date] (when-let [discount (ffirst (dc/q '[:find (sum ?discount) @@ -162,7 +237,7 @@ :ledger-mapped/ledger-side :ledger-side/debit})) (defn get-refund-items [c date] - (->> + (->> (dc/q '[:find ?type-name (sum ?t) :with ?e :in $ [?clients ?start-date ?end-date] @@ -173,26 +248,24 @@ (dc/db conn) [[c] date date]) (reduce - (fn [acc [type-name total]] - (update - acc - (cond (= type-name "CARD") - "Card Refunds" - (= type-name "CASH") - "Cash Refunds" - :else - "Food App Refunds") - (fnil + 0.0) - total)) - {}) + (fn [acc [type-name total]] + (update + acc + (cond (= type-name "CARD") + "Card Refunds" + (= type-name "CASH") + "Cash Refunds" + :else + "Food App Refunds") + (fnil + 0.0) + total)) + {}) (map (fn [[k v]] - {:db/id (str (java.util.UUID/randomUUID)) - :sales-summary-item/sort-order 3 - :sales-summary-item/category k - :ledger-mapped/amount v - :ledger-mapped/ledger-side :ledger-side/credit})))) - - + {:db/id (str (java.util.UUID/randomUUID)) + :sales-summary-item/sort-order 3 + :sales-summary-item/category k + :ledger-mapped/amount v + :ledger-mapped/ledger-side :ledger-side/credit})))) (defn get-fees [c date] (when-let [fee (get-fee c date)] @@ -278,17 +351,18 @@ (defn sales-summaries-v2 [] (doseq [[c client-code] (dc/q '[:find ?c ?client-code - :in $ - :where [?c :client/code ?client-code]] - (dc/db conn)) - {:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)] + :in $ + :where [?c :client/code ?client-code]] + (dc/db conn)) + {:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)] (mu/with-context {:client-code client-code - :date date} - (alog/info ::updating) - (let [manual-items (->> existing-summary - :sales-summary/items - (filter :sales-summary-item/manual?)) - calculated-items (->> + :date date} + (alog/info ::updating) + (let [manual-items (->> existing-summary + :sales-summary/items + (filter :sales-summary-item/manual?) + (map d-ss/<-pulled-item)) + calculated-items (->> (get-sales c date) (concat (get-payment-items c date)) (concat (get-refund-items c date)) @@ -301,20 +375,19 @@ (map (fn [z] (assoc z :ledger-mapped/account (some-> z :sales-summary-item/category str/lower-case name->number lookup-account) :sales-summary-item/manual? false)))) - all-items (concat calculated-items manual-items) - result {:db/id id - :sales-summary/client c - :sales-summary/date date - :sales-summary/dirty false - :sales-summary/client+date [c date] - :sales-summary/items all-items}] - (if (seq (:sales-summary/items result)) - (do - (alog/info ::upserting-summaries - :category-count (count (:sales-summary/items result))) - @(dc/transact conn [[:upsert-sales-summary result]])) - @(dc/transact conn [{:db/id id :sales-summary/dirty false}])))))) - + all-items (concat calculated-items manual-items) + result {:db/id id + :sales-summary/client c + :sales-summary/date date + :sales-summary/dirty false + :sales-summary/client+date [c date] + :sales-summary/items all-items}] + (if (seq (:sales-summary/items result)) + (do + (alog/info ::upserting-summaries + :category-count (count (:sales-summary/items result))) + @(dc/transact conn [[:upsert-sales-summary result]])) + @(dc/transact conn [{:db/id id :sales-summary/dirty false}])))))) (defn reset-summaries [] @(dc/transact conn (->> (dc/q '[:find ?sos @@ -324,9 +397,6 @@ (map (fn [[sos]] [:db/retractEntity sos]))))) - - - (comment (auto-ap.datomic/transact-schema conn) @@ -336,26 +406,19 @@ (dirty-sales-summaries [:client/code "NGWH"]) - (apply mark-dirty [:client/code "NGWH"] (last-n-days 5)) (iol-ion.tx.upsert-sales-summary-ledger/summary->journal-entry (dc/db conn) 17592314245819) - (iol-ion.tx.upsert-sales-summary-ledger/upsert-sales-summary (dc/db conn) {:db/id 17592314241429}) - - (mark-all-dirty 5) (delete-all) - (sales-summaries-v2) - 1 - (dc/q '[:find (pull ?sos [* {:sales-summary/sales-items [*]}]) :in $ :where [?sos :sales-summary/client [:client/code "NGHW"]] @@ -386,15 +449,24 @@ @(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy} {:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}]) - (auto-ap.datomic/transact-schema conn) - - - - - ) + (auto-ap.datomic/transact-schema conn)) +(defn days-arg + "Trailing-window size from the job's `args`, e.g. `{:days 30}` set as a container override + from the admin Background Jobs page for an ad-hoc wider backfill. Values arrive as EDN but + may still be strings, so coerce defensively the way load-historical-sales does." + [args] + (let [days (:days args)] + (cond-> (or days default-days) + (string? days) (#(Long/parseLong %))))) +(defn refresh-sales-summaries + "Marks the trailing `days` window dirty, skipping accepted summaries, then recomputes + everything left dirty." + ([] (refresh-sales-summaries default-days)) + ([days] + (mark-stale-dirty days) + (sales-summaries-v2))) (defn -main [& _] - (execute "sales-summaries" sales-summaries-v2)) - \ No newline at end of file + (execute "sales-summaries" #(refresh-sales-summaries (days-arg (:args env))))) diff --git a/src/clj/auto_ap/server.clj b/src/clj/auto_ap/server.clj index c3143575..82cfcff1 100644 --- a/src/clj/auto_ap/server.clj +++ b/src/clj/auto_ap/server.clj @@ -13,6 +13,7 @@ [auto-ap.jobs.load-historical-sales :as job-load-historical-sales] [auto-ap.jobs.plaid :as job-plaid] [auto-ap.jobs.register-invoice-import :as job-register-invoice-import] + [auto-ap.jobs.sales-summaries :as job-sales-summaries] [auto-ap.jobs.square :as job-square] [auto-ap.jobs.sysco :as job-sysco] [auto-ap.jobs.vendor-usages :as job-vendor-usages] @@ -33,23 +34,22 @@ (.addShutdownHook (Runtime/getRuntime) (Thread. f))) - (defn gzip-handler [] (let [gz (GzipHandler.)] (doto gz (.setIncludedMethods (into-array ["GET" "POST" "PUT" "DELETE" "PATCH"])) - (.setIncludedMimeTypes (into-array ["text/css" - "text/*" - "text/plain" - "text/javascript" - "text/csv" - "text/html" - "text/html;charset=utf-8" - "application/javascript" - "application/csv" - "application/edn" - "application/json" - "image/svg+xml"])) + (.setIncludedMimeTypes (into-array ["text/css" + "text/*" + "text/plain" + "text/javascript" + "text/csv" + "text/html" + "text/html;charset=utf-8" + "application/javascript" + "application/csv" + "application/edn" + "application/json" + "image/svg+xml"])) (.setMinGzipSize 1024)) gz)) @@ -126,6 +126,9 @@ (= job "close-auto-invoices") (job-close-auto-invoices/-main) + (= job "sales-summaries") + (job-sales-summaries/-main) + (= job "ezcater-upsert") (job-ezcater-upsert/-main) diff --git a/src/clj/auto_ap/ssr/admin/background_jobs.clj b/src/clj/auto_ap/ssr/admin/background_jobs.clj index b5576aa1..9d6e65d7 100644 --- a/src/clj/auto_ap/ssr/admin/background_jobs.clj +++ b/src/clj/auto_ap/ssr/admin/background_jobs.clj @@ -28,14 +28,13 @@ (com.amazonaws.services.ecs.model AssignPublicIp))) (defn get-ecs-tasks [] - (->> - (concat (:task-arns (ecs/list-tasks :max-results 50)) (:task-arns (ecs/list-tasks :desired-status "STOPPED" :max-results 50))) - (ecs/describe-tasks :include [] :tasks) - :tasks - (map #(assoc % :task-definition (:task-definition (ecs/describe-task-definition :task-definition (:task-definition-arn %))))) - (sort-by :created-at) - reverse)) - + (->> + (concat (:task-arns (ecs/list-tasks :max-results 50)) (:task-arns (ecs/list-tasks :desired-status "STOPPED" :max-results 50))) + (ecs/describe-tasks :include [] :tasks) + :tasks + (map #(assoc % :task-definition (:task-definition (ecs/describe-task-definition :task-definition (:task-definition-arn %))))) + (sort-by :created-at) + reverse)) (defn is-background-job? "This function checks whether a given task is a background job. @@ -60,7 +59,7 @@ (defn job-exited-successfully? [task] (if (= 0 (->> task :containers - (filter (comp #{"integreat-app" } :name)) + (filter (comp #{"integreat-app"} :name)) (first) :exit-code)) true @@ -77,7 +76,7 @@ :succeeded :failed)) :name (task-definition->job-name (:task-definition task)) - :end-date (some-> (:stopped-at task) coerce/to-date-time (time/to-time-zone (time/time-zone-for-offset 0))) + :end-date (some-> (:stopped-at task) coerce/to-date-time (time/to-time-zone (time/time-zone-for-offset 0))) :start-date (some-> (:created-at task) coerce/to-date-time (time/to-time-zone (time/time-zone-for-offset 0)))}) (defn fetch-page [request] @@ -85,7 +84,7 @@ (filter is-background-job?) (map ecs-task->job))] [jobs (count jobs)])) -(def query-schema (mc/schema [:map ])) +(def query-schema (mc/schema [:map])) (def grid-page (helper/build {:id "job-table" @@ -107,8 +106,7 @@ :entity-name "Job" :query-schema query-schema :route :admin-job-table - :headers [ - {:key "start" + :headers [{:key "start" :name "Start" :render #(some-> % :start-date (atime/unparse-local atime/standard-time))} {:key "end" @@ -119,7 +117,7 @@ :render (fn [e] (when (and (:start-date e) (:end-date e)) - (str (time/in-minutes (time/interval + (str (time/in-minutes (time/interval (:start-date e) (:end-date e))) " minutes")))} {:key "name" @@ -150,16 +148,16 @@ :network-configuration {:aws-vpc-configuration {:subnets ["subnet-5e675761" "subnet-8519fde2" "subnet-89bab8d4"] :security-groups ["sg-004e5855310c453a3" "sg-02d167406b1082698"] :assign-public-ip AssignPublicIp/ENABLED}}} - args (assoc-in [:overrides :container-overrides ] [{:name "integreat-app" :environment [{:name "args" :value (pr-str args)}]}])))) + args (assoc-in [:overrides :container-overrides] [{:name "integreat-app" :environment [{:name "args" :value (pr-str args)}]}])))) (defn job-start [{:keys [form-params]}] (if (not (get (currently-running-jobs) (:name form-params))) (let [new-job (run-task - (-> (:name form-params) - (str/replace #"-" "_") - (str/replace #":" "") - (str "_" (:dd-env env))) - (dissoc form-params :name))] + (-> (:name form-params) + (str/replace #"-" "_") + (str/replace #":" "") + (str "_" (:dd-env env))) + (dissoc form-params :name))] {:message (str "task " (str new-job) " started.")}) (form-validation-error "This job is already running" :form-params form-params))) @@ -170,107 +168,109 @@ [(fc/with-field :ledger-url (com/validated-field {:label "Url" :errors (fc/field-errors)} - [:div.flex.place-items-center.gap-2 - [:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"] - (com/text-input {:placeholder "ledger-data.csv" - :name (fc/field-name) - :value (fc/field-value)} )]))] + [:div.flex.place-items-center.gap-2 + [:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"] + (com/text-input {:placeholder "ledger-data.csv" + :name (fc/field-name) + :value (fc/field-value)})]))] (= "register-invoice-import" name) - [ - (fc/with-field :invoice-url - (com/validated-field {:label "Url" - :errors (fc/field-errors)} - [:div.flex.place-items-center.gap-2 - [:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"] - (com/text-input {:placeholder "invoice-data.csv" - :name (fc/field-name) - :value (fc/field-value)} )]))] + [(fc/with-field :invoice-url + (com/validated-field {:label "Url" + :errors (fc/field-errors)} + [:div.flex.place-items-center.gap-2 + [:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"] + (com/text-input {:placeholder "invoice-data.csv" + :name (fc/field-name) + :value (fc/field-value)})]))] (= "load-historical-sales" name) - [ - (fc/with-field :client - (com/validated-field {:label "Client" - :errors (fc/field-errors)} - (com/typeahead {:name (fc/field-name) - :value (fc/field-value) - :placeholder "Search..." - :url (bidi/path-for ssr-routes/only-routes - :company-search)}))) - (fc/with-field :days + [(fc/with-field :client + (com/validated-field {:label "Client" + :errors (fc/field-errors)} + (com/typeahead {:name (fc/field-name) + :value (fc/field-value) + :placeholder "Search..." + :url (bidi/path-for ssr-routes/only-routes + :company-search)}))) + (fc/with-field :days (com/validated-field {:label "Days to load" :errors (fc/field-errors)} (com/text-input {:placeholder "60" :name (fc/field-name) - :value (fc/field-value)} )))] - :else nil)) + :value (fc/field-value)})))] + (= "sales-summaries" name) + [(fc/with-field :days + (com/validated-field {:label "Days to refresh" + :errors (fc/field-errors)} + (com/text-input {:placeholder "7" + :name (fc/field-name) + :value (fc/field-value)})))] + :else nil))) - - ) - -(defn subform [{{:keys [name]} :query-params }] +(defn subform [{{:keys [name]} :query-params}] (html-response - (fc/start-form {} nil - (subform* {:name name})))) + (fc/start-form {} nil + (subform* {:name name})))) (defn job-start-dialog [{:keys [form-errors form-params] :as request}] (fc/start-form (or form-params {}) form-errors - (modal-response - (com/modal ;; TODO we need a cleaner way to have forms that wrap the whole. In this cas - {} - [:form {:hx-post (bidi/path-for ssr-routes/only-routes :admin-job-start) - :class "h-full w-full"} - [:fieldset {:class "hx-disable h-full w-full"} - (com/modal-card {} - [:div.m-2 "New job"] - [:div.space-y-6 + (modal-response + (com/modal ;; TODO we need a cleaner way to have forms that wrap the whole. In this cas + {} + [:form {:hx-post (bidi/path-for ssr-routes/only-routes :admin-job-start) + :class "h-full w-full"} + [:fieldset {:class "hx-disable h-full w-full"} + (com/modal-card {} + [:div.m-2 "New job"] + [:div.space-y-6 - (fc/with-field :name - (com/validated-field {:label "Job" - :errors (fc/field-errors)} - (com/select {:name (fc/field-name) - :value (fc/field-value) - :class "w-64" - :options [["" ""] - ["yodlee2" "Yodlee Import"] - ["yodlee2-accounts" "Yodlee Account Import"] - ["intuit" "Intuit import"] - ["plaid" "Plaid import"] - ["bulk-journal-import" "Bulk Journal Import"] - ["square-import-job" "Square Import"] - ["register-invoice-import" "Register Invoice Import "] - ["ezcater-upsert" "Upsert recent ezcater orders"] - ["load-historical-sales" "Load Historical Square Sales"] - ["export-backup" "Export Backup"]] - :hx-get (bidi/path-for ssr-routes/only-routes - :admin-job-subform) - :hx-target "#sub-form" - :hx-swap "innerHTML"}))) + (fc/with-field :name + (com/validated-field {:label "Job" + :errors (fc/field-errors)} + (com/select {:name (fc/field-name) + :value (fc/field-value) + :class "w-64" + :options [["" ""] + ["yodlee2" "Yodlee Import"] + ["yodlee2-accounts" "Yodlee Account Import"] + ["intuit" "Intuit import"] + ["plaid" "Plaid import"] + ["bulk-journal-import" "Bulk Journal Import"] + ["square-import-job" "Square Import"] + ["register-invoice-import" "Register Invoice Import "] + ["ezcater-upsert" "Upsert recent ezcater orders"] + ["load-historical-sales" "Load Historical Square Sales"] + ["sales-summaries" "Refresh Sales Summaries"] + ["export-backup" "Export Backup"]] + :hx-get (bidi/path-for ssr-routes/only-routes + :admin-job-subform) + :hx-target "#sub-form" + :hx-swap "innerHTML"}))) - [:div#sub-form (subform* {:name (fc/with-field :name (fc/field-value))}) ]] - [:div - - (com/form-errors {:errors (:errors fc/*form-errors*)}) - (com/validated-save-button {:errors form-errors} "Run job")])]])))) + [:div#sub-form (subform* {:name (fc/with-field :name (fc/field-value))})]] + [:div + + (com/form-errors {:errors (:errors fc/*form-errors*)}) + (com/validated-save-button {:errors form-errors} "Run job")])]])))) (def form-schema (mc/schema [:map [:name [:string {:min 1}]] [:ledger-url {:optional true} [:string {:min 1}]] [:invoice-url {:optional true} [:string {:min 1}]] [:client {:optional true} entity-id] - [:days {:optional true} [:int {:min 1 :max 120}]] - ])) + [:days {:optional true} [:int {:min 1 :max 120}]]])) (def key->handler - (apply-middleware-to-all-handlers - (->> - {:admin-jobs (helper/page-route grid-page) - :admin-job-table (helper/table-route grid-page) - :admin-job-subform (-> subform (wrap-schema-enforce :query-schema [:map [:name {:optional true} [:maybe :string]]])) - :admin-job-start (-> job-start - (wrap-schema-enforce :form-schema form-schema) - (wrap-nested-form-params) - (wrap-form-4xx-2 job-start-dialog)) - :admin-job-start-dialog job-start-dialog}) - (fn [h] - (-> h - (wrap-admin) - (wrap-client-redirect-unauthenticated))))) + (apply-middleware-to-all-handlers + (->> + {:admin-jobs (helper/page-route grid-page) + :admin-job-table (helper/table-route grid-page) + :admin-job-subform (-> subform (wrap-schema-enforce :query-schema [:map [:name {:optional true} [:maybe :string]]])) + :admin-job-start (-> job-start + (wrap-schema-enforce :form-schema form-schema) + (wrap-nested-form-params) + (wrap-form-4xx-2 job-start-dialog)) + :admin-job-start-dialog job-start-dialog}) + (fn [h] + (-> h + (wrap-admin) + (wrap-client-redirect-unauthenticated))))) diff --git a/src/clj/auto_ap/ssr/pos/sales_summaries.clj b/src/clj/auto_ap/ssr/pos/sales_summaries.clj index 45988b07..c9d0bad3 100644 --- a/src/clj/auto_ap/ssr/pos/sales_summaries.clj +++ b/src/clj/auto_ap/ssr/pos/sales_summaries.clj @@ -4,6 +4,7 @@ :refer [apply-pagination apply-sort-3 conn merge-query pull-many query2]] [auto-ap.datomic.accounts :as d-accounts] + [auto-ap.datomic.sales-summaries :refer [total-credits total-debits]] [auto-ap.graphql.utils :refer [extract-client-ids]] [auto-ap.query-params :refer [wrap-copy-qp-pqp]] [auto-ap.client-routes :as client-routes] @@ -116,18 +117,6 @@ (defn sort-items [ss] (sort-by (juxt :ledger-mapped/ledger-side :sales-summary-item/sort-order :sales-summary-item/category) ss)) -(defn total-debits [items] - (->> items - (filter #(= :ledger-side/debit (:ledger-mapped/ledger-side %))) - (map #(:ledger-mapped/amount % 0.0)) - (reduce + 0.0))) - -(defn total-credits [items] - (->> items - (filter #(= :ledger-side/credit (:ledger-mapped/ledger-side %))) - (map #(:ledger-mapped/amount % 0.0)) - (reduce + 0.0))) - (defn truncate [s max-len] (if (> (count s) max-len) (str (subs s 0 (- max-len 3)) "...") @@ -158,13 +147,13 @@ [:span.text-sm account-name] (com/pill {:color :red} "Missing acct")) (com/a-icon-button {:class "p-1" - :hx-get (bidi/path-for ssr-routes/only-routes ::route/edit-item-account) - :hx-target "closest .account-cell" - :hx-swap "outerHTML" - :hx-vals (hx/json {:item-index (or (:item-index item) 0) - :client-id client-id - :current-account-id (or account-id "")})} - svg/pencil)])) + :hx-get (bidi/path-for ssr-routes/only-routes ::route/edit-item-account) + :hx-target "closest .account-cell" + :hx-swap "outerHTML" + :hx-vals (hx/json {:item-index (or (:item-index item) 0) + :client-id client-id + :current-account-id (or account-id "")})} + svg/pencil)])) (defn account-edit-cell [{:keys [field-name-prefix client-id current-account-id]}] (let [account-input-name (str field-name-prefix "[ledger-mapped/account]")] @@ -172,23 +161,23 @@ (account-typeahead* {:name account-input-name :value current-account-id :client-id client-id}) - [:div.flex.gap-1 - (com/a-icon-button {:class "p-1" - :hx-put (bidi/path-for ssr-routes/only-routes ::route/save-item-account) - :hx-target "closest .account-cell" - :hx-swap "outerHTML" - :hx-include "closest .account-cell" - :hx-vals (hx/json {:field-name-prefix field-name-prefix - :client-id client-id})} - svg/check) - (com/a-icon-button {:class "p-1" - :hx-get (bidi/path-for ssr-routes/only-routes ::route/cancel-item-account) - :hx-target "closest .account-cell" - :hx-swap "outerHTML" - :hx-vals (hx/json {:field-name-prefix field-name-prefix - :client-id client-id - :current-account-id (or current-account-id "")})} - svg/x)]])) + [:div.flex.gap-1 + (com/a-icon-button {:class "p-1" + :hx-put (bidi/path-for ssr-routes/only-routes ::route/save-item-account) + :hx-target "closest .account-cell" + :hx-swap "outerHTML" + :hx-include "closest .account-cell" + :hx-vals (hx/json {:field-name-prefix field-name-prefix + :client-id client-id})} + svg/check) + (com/a-icon-button {:class "p-1" + :hx-get (bidi/path-for ssr-routes/only-routes ::route/cancel-item-account) + :hx-target "closest .account-cell" + :hx-swap "outerHTML" + :hx-vals (hx/json {:field-name-prefix field-name-prefix + :client-id client-id + :current-account-id (or current-account-id "")})} + svg/x)]])) (def grid-page (helper/build {:id "entity-table" @@ -576,8 +565,8 @@ [:span.text-gray-500 (truncate (:sales-summary-item/category item) 30)] (account-display-cell {:item (assoc item :item-index actual-idx) :field-name-prefix (str "step-params[sales-summary/items][" actual-idx "]") - :client-id client-id}) - [:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]])) + :client-id client-id}) + [:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]])) [:div.h-6]))] [:div.mt-2.border-t.pt-1 (summary-total-display request) @@ -619,13 +608,13 @@ [:span.text-gray-500 (truncate (:sales-summary-item/category item) 30)] (account-display-cell {:item (assoc item :item-index actual-idx) :field-name-prefix (str "step-params[sales-summary/items][" actual-idx "]") - :client-id client-id}) - [:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]])) + :client-id client-id}) + [:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]])) [:div.h-6]))] [:div.mt-2.border-t.pt-1 (summary-total-display request) (unbalanced-display request)]]] - [:div.mt-4.border-t.pt-2 + [:div.mt-4.border-t.pt-2 (fc/with-field :sales-summary/items (com/data-grid-new-row {:colspan 2 :hx-get (bidi/path-for ssr-routes/only-routes ::route/new-summary-item) @@ -761,16 +750,16 @@ ::route/edit-wizard-navigate (-> mm/next-handler (mm/wrap-wizard edit-wizard) (mm/wrap-decode-multi-form-state)) - ::route/new-summary-item (-> (add-new-entity-handler [:step-params :sales-summary/items] - (fn render [cursor request] - (sales-summary-item-row* - {:value cursor - :client-id (:client-id (:query-params request))})) - (fn build-new-row [base _] - (assoc base :sales-summary-item/manual? true))) - (wrap-schema-enforce :query-schema [:map - [:client-id {:optional true} - [:maybe entity-id]]])) + ::route/new-summary-item (-> (add-new-entity-handler [:step-params :sales-summary/items] + (fn render [cursor request] + (sales-summary-item-row* + {:value cursor + :client-id (:client-id (:query-params request))})) + (fn build-new-row [base _] + (assoc base :sales-summary-item/manual? true))) + (wrap-schema-enforce :query-schema [:map + [:client-id {:optional true} + [:maybe entity-id]]])) ::route/edit-item-account (-> edit-item-account (wrap-schema-enforce :query-schema [:map [:item-index nat-int?] diff --git a/terraform/deploy.tf b/terraform/deploy.tf index 6e6f95c9..36cdf097 100644 --- a/terraform/deploy.tf +++ b/terraform/deploy.tf @@ -386,6 +386,20 @@ module "close_auto_invoices_job" { cpu = 512 } +module "sales_summaries_job" { + count = var.enable_schedules ? 1 : 0 + source = "./background-job/" + ecs_cluster = var.ecs_cluster + task_role_arn = var.task_role_arn + stage = var.stage + schedule = "rate(1 day)" + job_name = "sales-summaries" + execution_role_arn = var.execution_role_arn + use_schedule = true + memory = 4096 + cpu = 2048 +} + module "yodlee2_accounts_job" { count = var.enable_schedules ? 1 : 0 source = "./background-job/" -- 2.49.1 From 943bc188429680ea8b30c4600378902d6e91a45d Mon Sep 17 00:00:00 2001 From: Bryce Date: Fri, 14 Aug 2026 18:19:41 -0700 Subject: [PATCH 2/8] update --- terraform/prod.tfvars | 4 +- .../prod/terraform.tfstate | 228 ++++++++++++++++- .../prod/terraform.tfstate.backup | 231 +++++++----------- 3 files changed, 306 insertions(+), 157 deletions(-) diff --git a/terraform/prod.tfvars b/terraform/prod.tfvars index 13a153fd..e0869919 100644 --- a/terraform/prod.tfvars +++ b/terraform/prod.tfvars @@ -1,5 +1,5 @@ -aws_access_key_id="AKIAINHACMVQJ6NYD26A" -aws_secret_access_key="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" +aws_access_key_id="AKIAZ4TSKSJ27WXFCOWK" +aws_secret_access_key="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" domain="app.integreatconsult.com" invoice_address="invoices@mail.app.integreatconsult.com" base_url="https://app.integreatconsult.com" diff --git a/terraform/terraform.tfstate.d/prod/terraform.tfstate b/terraform/terraform.tfstate.d/prod/terraform.tfstate index 32be0e42..791e258f 100644 --- a/terraform/terraform.tfstate.d/prod/terraform.tfstate +++ b/terraform/terraform.tfstate.d/prod/terraform.tfstate @@ -1,7 +1,7 @@ { "version": 4, - "terraform_version": "1.9.2", - "serial": 718, + "terraform_version": "1.15.1", + "serial": 722, "lineage": "9b630886-8cee-a57d-c7a2-4f19f13f9c51", "outputs": { "aws_access_key_id": { @@ -115,7 +115,8 @@ "usage_operation": "RunInstances", "virtualization_type": "hvm" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -133,7 +134,8 @@ "id": "679918342773", "user_id": "AIDAJPUJFTOKO4IRADMV4" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -181,7 +183,8 @@ ], "version": "2012-10-17" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -246,6 +249,7 @@ } ] ], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -311,6 +315,7 @@ } ] ], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -346,6 +351,7 @@ "type": "gp2" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwLCJ1cGRhdGUiOjMwMDAwMDAwMDAwMH19", "dependencies": [ "aws_instance.solr_ec2", @@ -445,6 +451,7 @@ "wait_for_steady_state": true }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_ecs_task_definition.integreat_app", @@ -535,6 +542,7 @@ "wait_for_steady_state": true }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_ecs_task_definition.solr", @@ -580,6 +588,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -636,6 +645,7 @@ ] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "aws_efs_file_system.solr_storage" @@ -681,6 +691,7 @@ "throughput_mode": "bursting" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -719,6 +730,7 @@ } ] ], + "identity_schema_version": 0, "dependencies": [ "aws_iam_user.app_user" ] @@ -744,7 +756,8 @@ "tags_all": {}, "unique_id": "AIDAINFBWI2I7A3TKPGW2" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -762,6 +775,7 @@ "user": "integreat-prod" }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_iam_user.app_user" ] @@ -908,6 +922,7 @@ ] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "data.aws_ami.amazon_linux_2023" @@ -1014,6 +1029,7 @@ "zone_id": "Z35SXDOTRQ7X7K" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19" } ] @@ -1063,6 +1079,7 @@ } }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_lb.integreat_app" @@ -1106,6 +1123,7 @@ } }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_acm_certificate.cert", @@ -1173,6 +1191,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "aws_acm_certificate.cert", @@ -1242,6 +1261,7 @@ "vpc_id": "vpc-b5b7d6ce" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1320,6 +1340,7 @@ "website_endpoint": "data.prod.app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1391,6 +1412,7 @@ "website_endpoint": null }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "data.aws_caller_identity.current" @@ -1489,6 +1511,7 @@ "website_endpoint": "app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1560,6 +1583,7 @@ "website_endpoint": null }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjM2MDAwMDAwMDAwMDAsInJlYWQiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19" } ] @@ -1591,6 +1615,7 @@ "topic": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_s3_bucket.invoices", "aws_sqs_queue.integreat-mail", @@ -1613,6 +1638,7 @@ "policy": "{\"Statement\":[{\"Action\":\"s3:*\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam::679918342773:role/http-proxy\",\"arn:aws:iam::679918342773:role/datomic-ddb\"]},\"Resource\":[\"arn:aws:s3:::toast.prod.app.integreatconsult.com/*\",\"arn:aws:s3:::toast.prod.app.integreatconsult.com\"],\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "aws_s3_bucket.toast_bucket", @@ -1638,6 +1664,7 @@ "service_id": "srv-ren22oppkwwryqqr" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "aws_instance.solr_ec2", @@ -1685,6 +1712,7 @@ "type": "DNS_HTTP" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1727,6 +1755,7 @@ "type": "DNS_HTTP" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1769,6 +1798,7 @@ "type": "DNS_HTTP" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1810,6 +1840,7 @@ "workmail_action": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_s3_bucket.invoices", "aws_ses_receipt_rule_set.main", @@ -1831,7 +1862,8 @@ "id": "default-rule-set", "rule_set_name": "default-rule-set" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -1868,6 +1900,7 @@ "visibility_timeout_seconds": 30 }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1905,6 +1938,7 @@ "visibility_timeout_seconds": 30 }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_s3_bucket.invoices", "data.aws_caller_identity.current" @@ -1945,6 +1979,7 @@ "visibility_timeout_seconds": 30 }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1968,6 +2003,7 @@ "volume_id": "vol-0069283d41ff6c010" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_ebs_volume.solr_ec2_storage", @@ -2014,6 +2050,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2043,6 +2080,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2107,6 +2145,7 @@ "target_id": "close-auto-invoices" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.close_auto_invoices_job.aws_cloudwatch_event_rule.schedule", @@ -2152,6 +2191,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2181,6 +2221,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2245,6 +2286,7 @@ "target_id": "import-uploaded-invoices" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.import_uploaded_invoices_job.aws_cloudwatch_event_rule.schedule", @@ -2290,6 +2332,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2319,6 +2362,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2383,6 +2427,7 @@ "target_id": "insight-outcome-recommendation" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.insight_outcome_recommendation_job.aws_cloudwatch_event_rule.schedule", @@ -2428,6 +2473,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2457,6 +2503,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2521,6 +2568,7 @@ "target_id": "intuit" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.intuit_job.aws_cloudwatch_event_rule.schedule", @@ -2566,6 +2614,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2607,6 +2656,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2636,6 +2686,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2700,6 +2751,7 @@ "target_id": "ntg" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.ntg_job.aws_cloudwatch_event_rule.schedule", @@ -2745,6 +2797,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2774,6 +2827,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2838,6 +2892,7 @@ "target_id": "plaid" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.plaid_job.aws_cloudwatch_event_rule.schedule", @@ -2883,6 +2938,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2912,6 +2968,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2976,6 +3033,7 @@ "target_id": "reconcile-ledger" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.reconcile_ledger_job.aws_cloudwatch_event_rule.schedule", @@ -3021,6 +3079,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3062,6 +3121,148 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, + "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" + } + ] + }, + { + "module": "module.sales_summaries_job[0]", + "mode": "managed", + "type": "aws_cloudwatch_event_rule", + "name": "schedule", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "index_key": 0, + "schema_version": 0, + "attributes": { + "arn": "arn:aws:events:us-east-1:679918342773:rule/sales-summaries-schedule-prod", + "description": "", + "event_bus_name": "default", + "event_pattern": null, + "id": "sales-summaries-schedule-prod", + "is_enabled": true, + "name": "sales-summaries-schedule-prod", + "name_prefix": "", + "role_arn": "", + "schedule_expression": "rate(1 day)", + "tags": null, + "tags_all": {} + }, + "sensitive_attributes": [], + "identity_schema_version": 0, + "private": "bnVsbA==" + } + ] + }, + { + "module": "module.sales_summaries_job[0]", + "mode": "managed", + "type": "aws_cloudwatch_event_target", + "name": "job_target", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "index_key": 0, + "schema_version": 1, + "attributes": { + "arn": "arn:aws:ecs:us-east-1:679918342773:cluster/default", + "batch_target": [], + "dead_letter_config": [], + "ecs_target": [ + { + "capacity_provider_strategy": [], + "enable_ecs_managed_tags": false, + "enable_execute_command": false, + "group": "", + "launch_type": "FARGATE", + "network_configuration": [ + { + "assign_public_ip": true, + "security_groups": [ + "sg-004e5855310c453a3", + "sg-02d167406b1082698" + ], + "subnets": [ + "subnet-5e675761", + "subnet-8519fde2", + "subnet-89bab8d4" + ] + } + ], + "ordered_placement_strategy": [], + "placement_constraint": [], + "platform_version": "", + "propagate_tags": "TASK_DEFINITION", + "tags": null, + "task_count": 1, + "task_definition_arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/sales_summaries_prod:1" + } + ], + "event_bus_name": "default", + "http_target": [], + "id": "sales-summaries-schedule-prod-sales-summaries", + "input": "", + "input_path": "", + "input_transformer": [], + "kinesis_target": [], + "redshift_target": [], + "retry_policy": [], + "role_arn": "arn:aws:iam::679918342773:role/service-role/Amazon_EventBridge_Invoke_ECS_1758992733", + "rule": "sales-summaries-schedule-prod", + "run_command_targets": [], + "sqs_target": [], + "target_id": "sales-summaries" + }, + "sensitive_attributes": [], + "identity_schema_version": 0, + "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", + "dependencies": [ + "module.sales_summaries_job.aws_cloudwatch_event_rule.schedule", + "module.sales_summaries_job.aws_ecs_task_definition.background_taskdef" + ] + } + ] + }, + { + "module": "module.sales_summaries_job[0]", + "mode": "managed", + "type": "aws_ecs_task_definition", + "name": "background_taskdef", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "schema_version": 1, + "attributes": { + "arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/sales_summaries_prod:1", + "arn_without_revision": "arn:aws:ecs:us-east-1:679918342773:task-definition/sales_summaries_prod", + "container_definitions": "[{\"cpu\":0,\"dockerLabels\":{\"com.datadoghq.tags.env\":\"prod\",\"com.datadoghq.tags.service\":\"sales-summaries\"},\"environment\":[{\"name\":\"DD_CONTAINER_ENV_AS_TAGS\",\"value\":\"{\\\"INTEGREAT_JOB\\\":\\\"background_job\\\"}\"},{\"name\":\"DD_ENV\",\"value\":\"prod\"},{\"name\":\"DD_SERVICE\",\"value\":\"sales-summaries\"},{\"name\":\"INTEGREAT_JOB\",\"value\":\"sales-summaries\"},{\"name\":\"config\",\"value\":\"/usr/local/config/prod-background-worker.edn\"}],\"essential\":true,\"image\":\"679918342773.dkr.ecr.us-east-1.amazonaws.com/integreat-cloud:prod\",\"logConfiguration\":{\"logDriver\":\"awslogs\",\"options\":{\"awslogs-group\":\"/ecs/integreat-app-prod\",\"awslogs-region\":\"us-east-1\",\"awslogs-stream-prefix\":\"ecs\"}},\"mountPoints\":[],\"name\":\"integreat-app\",\"portMappings\":[{\"containerPort\":9000,\"hostPort\":9000,\"protocol\":\"tcp\"},{\"containerPort\":9090,\"hostPort\":9090,\"protocol\":\"tcp\"}],\"systemControls\":[],\"volumesFrom\":[]},{\"cpu\":0,\"environment\":[{\"name\":\"DD_API_KEY\",\"value\":\"ce10d932c47b358e81081ae67bd8c112\"},{\"name\":\"ECS_FARGATE\",\"value\":\"true\"}],\"essential\":true,\"image\":\"public.ecr.aws/datadog/agent:latest\",\"mountPoints\":[],\"name\":\"datadog-agent\",\"portMappings\":[],\"systemControls\":[],\"volumesFrom\":[]}]", + "cpu": "2048", + "ephemeral_storage": [], + "execution_role_arn": "arn:aws:iam::679918342773:role/ecsTaskExecutionRole", + "family": "sales_summaries_prod", + "id": "sales_summaries_prod", + "inference_accelerator": [], + "ipc_mode": "", + "memory": "4096", + "network_mode": "awsvpc", + "pid_mode": "", + "placement_constraints": [], + "proxy_configuration": [], + "requires_compatibilities": [ + "FARGATE" + ], + "revision": 1, + "runtime_platform": [], + "skip_destroy": false, + "tags": null, + "tags_all": {}, + "task_role_arn": "arn:aws:iam::679918342773:role/datomic-ddb", + "volume": [] + }, + "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3091,6 +3292,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3155,6 +3357,7 @@ "target_id": "square-import-job" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.square_import_job.aws_cloudwatch_event_rule.schedule", @@ -3200,6 +3403,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3229,6 +3433,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3293,6 +3498,7 @@ "target_id": "sysco" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.sysco_job.aws_cloudwatch_event_rule.schedule", @@ -3338,6 +3544,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3367,6 +3574,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3431,6 +3639,7 @@ "target_id": "vendor-usages" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.vendor_usages_job.aws_cloudwatch_event_rule.schedule", @@ -3476,6 +3685,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3517,6 +3727,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3546,6 +3757,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3610,6 +3822,7 @@ "target_id": "yodlee2" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.yodlee2_job.aws_cloudwatch_event_rule.schedule", @@ -3655,6 +3868,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] diff --git a/terraform/terraform.tfstate.d/prod/terraform.tfstate.backup b/terraform/terraform.tfstate.d/prod/terraform.tfstate.backup index 8139110e..d392d1c9 100644 --- a/terraform/terraform.tfstate.d/prod/terraform.tfstate.backup +++ b/terraform/terraform.tfstate.d/prod/terraform.tfstate.backup @@ -1,7 +1,7 @@ { "version": 4, - "terraform_version": "1.9.2", - "serial": 714, + "terraform_version": "1.15.1", + "serial": 718, "lineage": "9b630886-8cee-a57d-c7a2-4f19f13f9c51", "outputs": { "aws_access_key_id": { @@ -115,7 +115,8 @@ "usage_operation": "RunInstances", "virtualization_type": "hvm" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -133,7 +134,8 @@ "id": "679918342773", "user_id": "AIDAJPUJFTOKO4IRADMV4" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -181,7 +183,8 @@ ], "version": "2012-10-17" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -246,6 +249,7 @@ } ] ], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -311,6 +315,7 @@ } ] ], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -346,6 +351,7 @@ "type": "gp2" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwLCJ1cGRhdGUiOjMwMDAwMDAwMDAwMH19", "dependencies": [ "aws_instance.solr_ec2", @@ -435,7 +441,7 @@ ], "tags": {}, "tags_all": {}, - "task_definition": "arn:aws:ecs:us-east-1:679918342773:task-definition/integreat_app_prod:837", + "task_definition": "arn:aws:ecs:us-east-1:679918342773:task-definition/integreat_app_prod:841", "timeouts": { "create": null, "delete": null, @@ -445,6 +451,7 @@ "wait_for_steady_state": true }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_ecs_task_definition.integreat_app", @@ -535,6 +542,7 @@ "wait_for_steady_state": true }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_ecs_task_definition.solr", @@ -580,6 +588,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -636,6 +645,7 @@ ] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "aws_efs_file_system.solr_storage" @@ -667,9 +677,9 @@ "provisioned_throughput_in_mibps": 0, "size_in_bytes": [ { - "value": 1432420352, + "value": 1434062848, "value_in_ia": 0, - "value_in_standard": 1432420352 + "value_in_standard": 1434062848 } ], "tags": { @@ -681,6 +691,7 @@ "throughput_mode": "bursting" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -719,6 +730,7 @@ } ] ], + "identity_schema_version": 0, "dependencies": [ "aws_iam_user.app_user" ] @@ -744,7 +756,8 @@ "tags_all": {}, "unique_id": "AIDAINFBWI2I7A3TKPGW2" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -762,6 +775,7 @@ "user": "integreat-prod" }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_iam_user.app_user" ] @@ -908,6 +922,7 @@ ] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "data.aws_ami.amazon_linux_2023" @@ -1014,6 +1029,7 @@ "zone_id": "Z35SXDOTRQ7X7K" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19" } ] @@ -1063,6 +1079,7 @@ } }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_lb.integreat_app" @@ -1106,6 +1123,7 @@ } }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_acm_certificate.cert", @@ -1173,6 +1191,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "aws_acm_certificate.cert", @@ -1242,6 +1261,7 @@ "vpc_id": "vpc-b5b7d6ce" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1320,6 +1340,7 @@ "website_endpoint": "data.prod.app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1391,6 +1412,7 @@ "website_endpoint": null }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "data.aws_caller_identity.current" @@ -1489,6 +1511,7 @@ "website_endpoint": "app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1560,6 +1583,7 @@ "website_endpoint": null }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjM2MDAwMDAwMDAwMDAsInJlYWQiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19" } ] @@ -1591,6 +1615,7 @@ "topic": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_s3_bucket.invoices", "aws_sqs_queue.integreat-mail", @@ -1613,6 +1638,7 @@ "policy": "{\"Statement\":[{\"Action\":\"s3:*\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam::679918342773:role/http-proxy\",\"arn:aws:iam::679918342773:role/datomic-ddb\"]},\"Resource\":[\"arn:aws:s3:::toast.prod.app.integreatconsult.com/*\",\"arn:aws:s3:::toast.prod.app.integreatconsult.com\"],\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "aws_s3_bucket.toast_bucket", @@ -1638,6 +1664,7 @@ "service_id": "srv-ren22oppkwwryqqr" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==", "dependencies": [ "aws_instance.solr_ec2", @@ -1685,6 +1712,7 @@ "type": "DNS_HTTP" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1727,6 +1755,7 @@ "type": "DNS_HTTP" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1769,6 +1798,7 @@ "type": "DNS_HTTP" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1810,6 +1840,7 @@ "workmail_action": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_s3_bucket.invoices", "aws_ses_receipt_rule_set.main", @@ -1831,7 +1862,8 @@ "id": "default-rule-set", "rule_set_name": "default-rule-set" }, - "sensitive_attributes": [] + "sensitive_attributes": [], + "identity_schema_version": 0 } ] }, @@ -1868,6 +1900,7 @@ "visibility_timeout_seconds": 30 }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1905,6 +1938,7 @@ "visibility_timeout_seconds": 30 }, "sensitive_attributes": [], + "identity_schema_version": 0, "dependencies": [ "aws_s3_bucket.invoices", "data.aws_caller_identity.current" @@ -1945,6 +1979,7 @@ "visibility_timeout_seconds": 30 }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -1968,6 +2003,7 @@ "volume_id": "vol-0069283d41ff6c010" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwfX0=", "dependencies": [ "aws_ebs_volume.solr_ec2_storage", @@ -2014,6 +2050,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2043,6 +2080,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2107,6 +2145,7 @@ "target_id": "close-auto-invoices" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.close_auto_invoices_job.aws_cloudwatch_event_rule.schedule", @@ -2152,144 +2191,7 @@ "volume": [] }, "sensitive_attributes": [], - "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" - } - ] - }, - { - "module": "module.current_balance_cache[0]", - "mode": "managed", - "type": "aws_cloudwatch_event_rule", - "name": "schedule", - "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", - "instances": [ - { - "index_key": 0, - "schema_version": 0, - "attributes": { - "arn": "arn:aws:events:us-east-1:679918342773:rule/current-balance-cache-schedule-prod", - "description": "", - "event_bus_name": "default", - "event_pattern": null, - "id": "current-balance-cache-schedule-prod", - "is_enabled": true, - "name": "current-balance-cache-schedule-prod", - "name_prefix": "", - "role_arn": "", - "schedule_expression": "rate(30 minutes)", - "tags": {}, - "tags_all": {} - }, - "sensitive_attributes": [], - "private": "bnVsbA==" - } - ] - }, - { - "module": "module.current_balance_cache[0]", - "mode": "managed", - "type": "aws_cloudwatch_event_target", - "name": "job_target", - "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", - "instances": [ - { - "index_key": 0, - "schema_version": 1, - "attributes": { - "arn": "arn:aws:ecs:us-east-1:679918342773:cluster/default", - "batch_target": [], - "dead_letter_config": [], - "ecs_target": [ - { - "capacity_provider_strategy": [], - "enable_ecs_managed_tags": false, - "enable_execute_command": false, - "group": "", - "launch_type": "FARGATE", - "network_configuration": [ - { - "assign_public_ip": true, - "security_groups": [ - "sg-004e5855310c453a3", - "sg-02d167406b1082698" - ], - "subnets": [ - "subnet-5e675761", - "subnet-8519fde2", - "subnet-89bab8d4" - ] - } - ], - "ordered_placement_strategy": [], - "placement_constraint": [], - "platform_version": "", - "propagate_tags": "TASK_DEFINITION", - "tags": {}, - "task_count": 1, - "task_definition_arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/current_balance_cache_prod:3" - } - ], - "event_bus_name": "default", - "http_target": [], - "id": "current-balance-cache-schedule-prod-current-balance-cache", - "input": "", - "input_path": "", - "input_transformer": [], - "kinesis_target": [], - "redshift_target": [], - "retry_policy": [], - "role_arn": "arn:aws:iam::679918342773:role/service-role/Amazon_EventBridge_Invoke_ECS_1758992733", - "rule": "current-balance-cache-schedule-prod", - "run_command_targets": [], - "sqs_target": [], - "target_id": "current-balance-cache" - }, - "sensitive_attributes": [], - "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", - "dependencies": [ - "module.current_balance_cache.aws_cloudwatch_event_rule.schedule", - "module.current_balance_cache.aws_ecs_task_definition.background_taskdef" - ] - } - ] - }, - { - "module": "module.current_balance_cache[0]", - "mode": "managed", - "type": "aws_ecs_task_definition", - "name": "background_taskdef", - "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", - "instances": [ - { - "schema_version": 1, - "attributes": { - "arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/current_balance_cache_prod:3", - "arn_without_revision": "arn:aws:ecs:us-east-1:679918342773:task-definition/current_balance_cache_prod", - "container_definitions": "[{\"cpu\":0,\"dockerLabels\":{\"com.datadoghq.tags.env\":\"prod\",\"com.datadoghq.tags.service\":\"current-balance-cache\"},\"environment\":[{\"name\":\"DD_CONTAINER_ENV_AS_TAGS\",\"value\":\"{\\\"INTEGREAT_JOB\\\":\\\"background_job\\\"}\"},{\"name\":\"DD_ENV\",\"value\":\"prod\"},{\"name\":\"DD_SERVICE\",\"value\":\"current-balance-cache\"},{\"name\":\"INTEGREAT_JOB\",\"value\":\"current-balance-cache\"},{\"name\":\"config\",\"value\":\"/usr/local/config/prod-background-worker.edn\"}],\"essential\":true,\"image\":\"679918342773.dkr.ecr.us-east-1.amazonaws.com/integreat-cloud:prod\",\"logConfiguration\":{\"logDriver\":\"awslogs\",\"options\":{\"awslogs-group\":\"/ecs/integreat-app-prod\",\"awslogs-region\":\"us-east-1\",\"awslogs-stream-prefix\":\"ecs\"}},\"mountPoints\":[],\"name\":\"integreat-app\",\"portMappings\":[{\"containerPort\":9000,\"hostPort\":9000,\"protocol\":\"tcp\"},{\"containerPort\":9090,\"hostPort\":9090,\"protocol\":\"tcp\"}],\"systemControls\":[],\"volumesFrom\":[]},{\"cpu\":0,\"environment\":[{\"name\":\"DD_API_KEY\",\"value\":\"ce10d932c47b358e81081ae67bd8c112\"},{\"name\":\"ECS_FARGATE\",\"value\":\"true\"}],\"essential\":true,\"image\":\"public.ecr.aws/datadog/agent:latest\",\"mountPoints\":[],\"name\":\"datadog-agent\",\"portMappings\":[],\"systemControls\":[],\"volumesFrom\":[]}]", - "cpu": "512", - "ephemeral_storage": [], - "execution_role_arn": "arn:aws:iam::679918342773:role/ecsTaskExecutionRole", - "family": "current_balance_cache_prod", - "id": "current_balance_cache_prod", - "inference_accelerator": [], - "ipc_mode": "", - "memory": "2048", - "network_mode": "awsvpc", - "pid_mode": "", - "placement_constraints": [], - "proxy_configuration": [], - "requires_compatibilities": [ - "FARGATE" - ], - "revision": 3, - "runtime_platform": [], - "skip_destroy": false, - "tags": {}, - "tags_all": {}, - "task_role_arn": "arn:aws:iam::679918342773:role/datomic-ddb", - "volume": [] - }, - "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2319,6 +2221,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2383,6 +2286,7 @@ "target_id": "import-uploaded-invoices" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.import_uploaded_invoices_job.aws_cloudwatch_event_rule.schedule", @@ -2428,6 +2332,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2457,6 +2362,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2521,6 +2427,7 @@ "target_id": "insight-outcome-recommendation" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.insight_outcome_recommendation_job.aws_cloudwatch_event_rule.schedule", @@ -2566,6 +2473,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2595,6 +2503,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2659,6 +2568,7 @@ "target_id": "intuit" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.intuit_job.aws_cloudwatch_event_rule.schedule", @@ -2704,6 +2614,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2745,6 +2656,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2774,6 +2686,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2838,6 +2751,7 @@ "target_id": "ntg" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.ntg_job.aws_cloudwatch_event_rule.schedule", @@ -2883,6 +2797,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -2912,6 +2827,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -2976,6 +2892,7 @@ "target_id": "plaid" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.plaid_job.aws_cloudwatch_event_rule.schedule", @@ -3021,6 +2938,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3050,6 +2968,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3114,6 +3033,7 @@ "target_id": "reconcile-ledger" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.reconcile_ledger_job.aws_cloudwatch_event_rule.schedule", @@ -3159,6 +3079,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3200,6 +3121,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3229,6 +3151,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3293,6 +3216,7 @@ "target_id": "square-import-job" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.square_import_job.aws_cloudwatch_event_rule.schedule", @@ -3338,6 +3262,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3367,6 +3292,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3431,6 +3357,7 @@ "target_id": "sysco" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.sysco_job.aws_cloudwatch_event_rule.schedule", @@ -3476,6 +3403,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3505,6 +3433,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3569,6 +3498,7 @@ "target_id": "vendor-usages" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.vendor_usages_job.aws_cloudwatch_event_rule.schedule", @@ -3614,6 +3544,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3655,6 +3586,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] @@ -3684,6 +3616,7 @@ "tags_all": {} }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "bnVsbA==" } ] @@ -3748,6 +3681,7 @@ "target_id": "yodlee2" }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "dependencies": [ "module.yodlee2_job.aws_cloudwatch_event_rule.schedule", @@ -3793,6 +3727,7 @@ "volume": [] }, "sensitive_attributes": [], + "identity_schema_version": 0, "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" } ] -- 2.49.1 From 57a84dae115590e9abd04af2f153236f00469c9f Mon Sep 17 00:00:00 2001 From: Bryce Date: Sat, 15 Aug 2026 20:17:58 -0700 Subject: [PATCH 3/8] fix(sales-summaries): stop days falling out of balance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — plus a fourth that turned out to be a missing-data problem and is deliberately left visible. Measured over ninety days on a restored copy of production (210 clients, 18,900 client-days): 1,258 days out of balance and $69,560.10 becomes 279 days and $7,790.54 — of which 171 are not arithmetic faults at all, but days whose sales were never imported. 979 days repaired, none knocked out of balance, and not one already-balanced day altered — verified line by line (category, side, amount to the cent, account), not just on each day's bottom line. THE DATA FAULT Ten Square locations were configured against two client records each. Sales orders scoped their identifier by client; refunds, card payments, payouts and cash-drawer shifts used the bare Square id. Those attributes are :db.unique/identity, so both clients' imports resolved to a single entity and the last writer won — 3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts changed hands over time, across 19 client pairs of which only 10 are visible in today's configuration. Worse, one payment could belong to two orders. :sales-order/charges is :db/isComponent, so removing a voided order cascaded into payments the other client still needed. Fixes: client-scope the four key schemes; look the record up under both schemes so the change deploys before the migration finishes; and a migration that gives every order its own payment. Run over the whole database that is 19,040,785 orders walked, 9,100,314 payments re-keyed and 200,027 copied, ending with 17,047,142 payments scoped, none left to rename, none unscopable, and no payment owned by more than one order. Idempotent and resumable; about thirteen minutes. THE ARITHMETIC FAULTS - Refunded tips stayed on the books. get-tip summed tips by joining through :sales-order/charges, so a return-only order — no tender to join through — contributed nothing while its reversal sat unread on :sales-order/tip. Additive, not substitutive: where an order does have a tender the tender is the correct source. - Service charges were collected but never earned. Nothing read :sales-order/service-charge. Now credited for Square orders only, both signs, behind summary-service-charges. The flag is off by default, so deploying this changes nothing until a client is opted in. docs/2026-08-15-sales-summary-rollout-plan.md has the steps. WHAT IS DELIBERATELY NOT FIXED 156 of the 279 remaining days carry refunds on a record that recorded no sales at all that day, and 132 of those fall before that client's first ever order. The refunds are not theirs: ownership history shows a $35.35 refund dated 26 February belonging to NGDG that day and taken over by NGDU on 12 August, flipping between the two several times a day. Across nine records, 659 refunds worth $15,225.24 sit on a record dated before its own first order — unscoped keys let whichever import ran last take ownership. A rule closing those days was written and measured (156 days, $4,820.19, nothing broken) and then removed. An unbalanced day is the only visible signal that a restaurant's sales are not being imported; balancing it would remove the alarm and leave the fire. A comment and a test hold that decision in place. Step 9 of the rollout plan is the real fix, and it needs a business decision. SUPPORTING - Install schema attributes before the tuples that compose them. A tuple in schema.edn is built from an attribute in cloud-migration-schema.edn, so every test fixture died in setup — very likely why sales summaries had no tests before this. - Log each day's imbalance and its suspect lines. - Bound the dirty-summary scan to one client: 1,321 ms to 5.6 ms. - compare-sales-summaries lives in test/clj as auto-ap.tools.* — it is a verification harness, not part of the running application. Its docstring now warns that d/as-of cannot be used to compare summary amounts: :ledger-mapped/amount, ledger-side and account are :db/noHistory, so a recomputed summary reads back with its amounts absent and looks like a legitimate balanced day. 26 tests, 62 assertions. Co-Authored-By: Claude Opus 5 --- docs/2026-08-15-remove-voided-orders-risk.md | 115 +++ ...26-08-15-sales-summary-reconciliation.html | 682 ++++++++++++++++++ docs/2026-08-15-sales-summary-rollout-plan.md | 268 +++++++ src/clj/auto_ap/datomic.clj | 21 +- src/clj/auto_ap/datomic/sales_summaries.clj | 7 + .../jobs/rekey_square_external_ids.clj | 314 ++++++++ src/clj/auto_ap/jobs/sales_summaries.clj | 177 ++++- src/clj/auto_ap/square/core3.clj | 54 +- .../jobs/rekey_square_external_ids_test.clj | 149 ++++ .../clj/auto_ap/jobs/sales_summaries_test.clj | 183 +++++ test/clj/auto_ap/square/core3_test.clj | 118 +++ .../auto_ap/tools/compare_sales_summaries.clj | 153 ++++ 12 files changed, 2207 insertions(+), 34 deletions(-) create mode 100644 docs/2026-08-15-remove-voided-orders-risk.md create mode 100644 docs/2026-08-15-sales-summary-reconciliation.html create mode 100644 docs/2026-08-15-sales-summary-rollout-plan.md create mode 100644 src/clj/auto_ap/jobs/rekey_square_external_ids.clj create mode 100644 test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj create mode 100644 test/clj/auto_ap/jobs/sales_summaries_test.clj create mode 100644 test/clj/auto_ap/square/core3_test.clj create mode 100644 test/clj/auto_ap/tools/compare_sales_summaries.clj diff --git a/docs/2026-08-15-remove-voided-orders-risk.md b/docs/2026-08-15-remove-voided-orders-risk.md new file mode 100644 index 00000000..4c19716f --- /dev/null +++ b/docs/2026-08-15-remove-voided-orders-risk.md @@ -0,0 +1,115 @@ +--- +title: remove-voided-orders can delete another client's payments +type: risk +date: 2026-08-15 +status: open — decide before merging the re-key +--- + +# `remove-voided-orders` can delete another client's payments + +Measured on the restored backup, 2026-08-15. This risk is **pre-existing** — nothing in the +sales-summary work created it — but it is live right now, and the re-key work touches the same +data, so it should be understood before merging. + +## The mechanism, in four steps + +**1. Charges are component entities of an order.** + +```clojure +;; resources/schema.edn +{:db/ident :sales-order/charges + :db/valueType :db.type/ref + :db/isComponent true ;; <- this is the load-bearing bit + :db/cardinality :db.cardinality/many} +``` + +`:db/isComponent true` tells Datomic the charges *belong to* the order. It is what lets you +transact an order with its tenders nested inside, and it means the charges have no independent +existence as far as Datomic is concerned. + +**2. `retractEntity` on a component parent deletes the children too.** + +That is the documented behaviour of `:db/retractEntity`: it recursively retracts component +values. `square.core3/remove-voided-orders` ends with exactly that: + +```clojure +(s/map (fn [[o]] + [[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]])) +``` + +It asks Square for the last 10 days of orders, keeps the ones that should *not* be imported — +voided and cancelled orders — and retracts any of those we already stored. That is correct and +desirable on its own: a voided order should not sit in the books. + +**3. But one charge can be shared by two orders.** + +When two clients are configured on the same Square location, both import the same Square data. +Order keys embed the client, so each client gets its own order entity. Charge keys did **not** +embed the client, and `:charge/external-id` is `:db.unique/identity`, so both clients' orders +resolved to *the same charge entity*: + +``` +NGCD order 17592395490523 ──┐ + ├──> charge 17592490524 ← one entity, two parents +NGCC order 17592395511722 ──┘ +``` + +**4. So retracting one order deletes a charge the other order still points at.** + +Datomic sees a component and removes it. The surviving order keeps its line items — its sales — +but its tender is gone. The day then shows revenue with no payment against it, the summary goes +out of balance, and the payment is gone from the current database value. (History retains it, so +it is recoverable by someone who knows to look, but nothing in the app will show it again.) + +## How exposed are we + +Measured over the 10 contended clients across 2026-07-13 → 08-14: + +| | | +|---|---| +| Charges examined | 56,829 | +| **Referenced by more than one order** | **35,870 (63%)** | + +So this is not a theoretical corner. Roughly two thirds of the charges in that population have +two parents, and any voided order among them takes a charge down with it. + +The exposure window for *new* damage is the rolling 10 days `remove-voided-orders` searches, but +the shared charges themselves span the whole period the locations were double-configured. + +## What changes after Phase 0 and the re-key, and what doesn't + +- **Phase 0 (done on the restore)** stops new sharing: only one client per location imports now, + so no new order pairs form. +- **The re-key (done for refunds and the contended clients' charges)** makes sharing structurally + impossible going forward, because a charge key now contains the client code. +- **Neither retroactively splits the 35,870 charges that are already shared.** They still have two + parents. Until they are split, `remove-voided-orders` remains capable of deleting a payment + belonging to the other client. + +This is why plan §3.3 forbids retracting anything — including any historical cleanup of the +duplicate clients' data — until a verification query shows zero charges with more than one parent. + +## Options, roughly in order of preference + +1. **Split the shared charges, then let removal run normally.** Re-import the affected window now + that keys are client-scoped, so each client creates its own charge entity. This reuses the + import path rather than hand-constructing component entities. Verify with a query for charges + having more than one referencing order; it must reach zero. + +2. **Guard the retraction.** Before retracting an order, check whether any of its charges are + referenced by another order; detach those (retract the `:sales-order/charges` ref rather than + the charge) and retract the rest. Small, contained change, and it makes the operation safe + regardless of what shape the data is in — worth doing on its own merits even after a split. + +3. **Do nothing and accept it.** Only defensible once every location has a single client *and* + the historical shared charges are gone. Not true today. + +## What I did about it during the validation run + +I ran the import on the restore with `remove-voided-orders` **skipped**, and ran the other steps +(`upsert-locations`, `upsert`, `upsert-payouts`, `upsert-refunds`) normally. That kept the +validation faithful to how the import behaves without risking silent payment loss in the data +the measurements were about to be taken from. + +**Nothing in the production system has been changed.** This note is about a risk that already +exists there. diff --git a/docs/2026-08-15-sales-summary-reconciliation.html b/docs/2026-08-15-sales-summary-reconciliation.html new file mode 100644 index 00000000..2399a1de --- /dev/null +++ b/docs/2026-08-15-sales-summary-reconciliation.html @@ -0,0 +1,682 @@ +Ninety-Day Reconciliation + + +
+ +
+
Sales summaries · measured on a restored production backup
+

Ninety-Day Reconciliation

+

Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — and a fourth, found late, that is a missing-data problem wearing a balancing problem's clothes. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.

+
+ Window 2026-05-10 → 2026-08-07 + Client-days 18,900 + Clients 210 + Nothing in production was changed +
+
+ +
+
+
+ Today's calculation, ninety days re-run + $69,560.10 + 1,258 days out of balance · 93.34% clean +
+ +
+ The same ninety days, fixes on + $7,790.54 + 279 days out of balance · 98.52% clean +
+
+ +
+
979client-days brought into balance
+
0days knocked out of balance
+
88.8%of the variance removed
+
0payments shared between two clients
+
+ +
+

In one sentence: a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.

+

How the two figures above were produced. Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth 979 days and $61,769.56, not the larger number a stale baseline would have shown.

+

Most of what is left is not a balancing fault at all, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.

+
+
+ +
+

The four problems

+ +
+

1. Two client records sharing one Square location

+
+

For the business: ten restaurant locations were set up twice in the system, as two separate clients. Both were importing from Square. Because the two records competed for the same payments and refunds, a refund would belong to one client for twenty minutes, then the other — so a day's books could gain or lose a refund depending on nothing but timing. On 2026-07-23 one client's summary was missing a $71.94 refund entirely, and was out of balance by exactly that amount.

+

technical Sales orders scoped their identifier by client (square/order/<code>-<loc>-<id>), but refunds, card charges, payouts and cash-drawer shifts did not — they used the bare Square id. Those attributes are :db.unique/identity, so both clients' imports resolved to a single entity and the last writer won.

+

Reading ownership out of the database's own history, this had actually happened to 3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts. And it has involved 19 client pairs, of which only 10 are visible in today's configuration — nine more contended in the past and the configuration has since changed, so no point-in-time check would find them.

+
+
+ +
+

2. One payment record owned by two orders

+
+

For the business: the same collision meant a single card payment could be attached to both clients' copies of an order. That is worse than untidy. The nightly import removes orders Square reports as voided, and removing an order also removes its payments — so cancelling one client's order could silently delete the other client's payment, leaving a day showing sales with no money against them.

+

technical :sales-order/charges is declared :db/isComponent true, so [:db/retractEntity <order>] cascades into the charges. In a 20,000-order sample of the affected clients, 11,469 charges had two parent orders. This is why remove-voided-orders was left switched off during testing.

+
+
+ +
+

3. Tips refunded, and service charges credited nowhere

+
+

For the business: two arithmetic faults, both of which overstated or understated a day.

+
    +
  • Refunded tips stayed on the books. When a guest was refunded, the tip came back too — but the summary still counted the original tip as income. On one NGLK day the books credited $482.94 of tips beside a $60.00 refund of that very tip.
    • +
  • Service charges were collected but never earned. A catering or auto-gratuity charge is inside the card payment the customer makes, so it arrived as money taken — but no line recorded it as money earned. One NTPT order carried $427.10 that was credited to nothing at all; the largest single instance was $1,344.86 in one day.
    • +
+

technical get-tip summed tips by joining through :sales-order/charges, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on :sales-order/tip. Nothing at all read :sales-order/service-charge.

+
+
+ +
+

4. Refunds on records whose sales were never imported not fixed — deliberately

+
+

This is why the duplicated restaurants looked so much worse than everyone else. Of the days still failing after the first three fixes, 155 of the 158 on duplicate records had no sales orders at all — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.

+

The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded any order shows 132 of 156 fall before that client had a single order in the system — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.

+

Where the refunds came from. Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to NGDG that same day, and was taken over by NGDU on 12 August. Others flip between the two records several times a day across 12–15 August. NGDU's first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.

+

Across the nine records, 659 refunds worth $15,225.24 sit on a record dated before that record's first order. Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.

+

Why it is deliberately left out of balance. The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and it closes 156 of the 279 remaining days and $4,820.19. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.

+

technical get-returns sums :sales-order/returns over orders scanned for the date. With no orders the sum is nil and no Returns line is written, while get-refund-items still credits Card Refunds from the sales-refund records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.

+
+
+
+ +
+

What the fixes actually are

+
+

Five changes. The first three stop two clients from sharing a record; the last two record + money that was being collected but not booked. Each is small — the difficulty was knowing + which line to change, not writing it. A sixth was written and then removed; it is described + at the end because the reasoning matters more than the code did.

+
+ +

1 · Put the client in the record's name

+
+

Every imported record has an identifier the importer uses to decide "have I seen this + before?". Sales orders already included the client; refunds, card payments, payouts and + cash-drawer shifts did not, which is precisely why two clients could land on one record.

+
+
+ 
;; before — the bare Square id, identical for both clients
+(str "square/refund/" (:id r))          ;; square/refund/NOkQOTIiJULWN6…
+
+;; after
+(scoped-key "square/refund/" client location (:id r))
+;; square/refund/NGCD-CD-NOkQOTIiJULWN6…
+
+(defn scoped-key [prefix client location id]
+  (str prefix (:client/code client) "-"
+       (:square-location/client-location location) "-" id))
+
+
+

Applied at five places in the Square importer: order payments, refunds, payouts (twice — + the record itself and the lookup that finds it) and cash-drawer shifts. ezCater orders + already did this and needed no change.

+
+ +

2 · Find the existing record before writing, under either name

+
+

This is the one that makes the change safe to deploy. The identifiers are unique keys, so + the importer relies on "same id, same record". Rename them and the next import matches + nothing — and would quietly create a second copy of every refund and payment in the + system, leaving the originals orphaned. So the importer looks up the record explicitly, + new name first, old name second, and writes to whichever it finds.

+
+
+ 
(defn existing-id [db attr prefix client location id]
+  (when id
+    (or (dc/entid db [attr (scoped-key prefix client location id)])   ;; new scheme
+        (dc/entid db [attr (str prefix id)]))))                       ;; legacy scheme
+
+
+

The result is pinned as the record's id on the way in, so the write lands on the existing + row regardless of which name it currently carries. The proof this worked is a count that did + not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was + re-named, and afterwards there were still exactly 144,688 payouts and 69,291 cash-drawer + shifts — the same figures as at the restore point. Refunds went from 50,986 to 51,986, + and all 1,000 of those came from the live Square import run afterwards, not from the renaming. + Had the fallback lookup been missing, each of these would have doubled instead.

+
+ +

3 · Give every order its own payment record

+
+

Renaming stops new collisions but does not undo old ones: a payment already shared + by two orders is still one row with two owners. The migration walks each order's payments and, + where another order has already claimed one, makes that order its own copy with the same + amounts and points the order at the copy.

+
+
+ 
;; for each order, for each of its payments:
+:keep   → first order to claim it; rename in place
+:clone  → copy type, total, tip, tax, date, processor, note, receipt link
+           set the copy's client and location to this order's
+           retract this order's link to the shared payment
+           link it to the copy instead
+
+
+

Run over the whole database that was 9,100,314 renamed and 200,027 copied, + and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across + 400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it + reported making, which is the check that it created what it meant to and nothing else.

+

One subtlety worth recording, because it bit us: the Square id has to be recovered from the + record's current owner rather than by trimming a fixed prefix. Client codes contain dashes — + N-30003 — so a pattern cannot tell where the client name ends and the + Square id begins. Getting this wrong scoped some records twice and doubled their tender.

+
+ +

4 · Count tips that were handed back

+
+

Tips were summed by walking from the order to its payments. A refund-only order has no + payment attached, so its negative tip was invisible. The fix adds those tips rather than + replacing the calculation.

+
+
+ 
;; before
+:ledger-mapped/amount (tendered-tip c date)
+
+;; after
+:ledger-mapped/amount (+ (tendered-tip c date)
+                         (untendered-tip c date))
+
+;; untendered-tip — tips on orders with no payment attached
+[?e :sales-order/tip ?tip]
+(not [?e :sales-order/charges])
+
+
+

Adding rather than replacing is deliberate. Where an order does have a + payment, the payment is the correct source: real orders exist whose payment carries a tip the + order does not — an auto-gratuity recorded as a service charge, or a wallet tip missing from + the order totals. Reading the order instead would have dropped those. Three tests hold this + in place: the refund case must change, and the tendered and ordinary cases must not.

+
+ +

5 · Credit Square service charges, both signs

+
+

Nothing read the service-charge field at all. A new line credits it, for Square orders only + and for negative amounts as well as positive.

+
+
+ 
[?e :sales-order/service-charge ?service-charge]
+(or-join [?e]
+         [?e :sales-order/vendor :vendor/ccp-square]
+         (and (not [?e :sales-order/vendor])
+              [?e :sales-order/external-id ?external-id]
+              [(clojure.string/starts-with? ?external-id "square/order/")]))
+
+
+

Why the vendor test has two branches. ezCater service charges are commission + the platform deducts from the restaurant, not money the diner hands over, so crediting them + would make a day worse rather than better — hence the Square-only condition. But whole eras of + Square orders carry no vendor field at all, and a test on vendor alone would silently credit + nothing. The second branch falls back to the order's own identifier.

+

Why negatives matter. A returned catering fee arrives as a negative service + charge and is already deducted from the day's returns; dropping negatives would lose the + reversal. The line sits behind a per-client switch, off by default, so it can be turned on a + few restaurants at a time.

+
+ +

6 · The change that was written, measured, and then taken out

+
+

Worth recording, because the arithmetic case for it is good and someone will propose it + again. Where a day has refunds and no sales orders whatsoever, book a Returns + debit equal to that day's refunds:

+
+
+ 
(defn- refund-only-returns [c date]
+  (when-not (traded? c date)
+    (let [amount (refunded-total c date)]
+      (when-not (zero? amount) amount))))
+
+
+

It works. Measured over the same ninety days it closed 156 days and $4,820.19, + knocked nothing out of balance, and altered no already-balanced day — the guard makes it + incapable of touching a day that traded.

+

It was removed anyway. Those days are not quiet days; they are days whose sales were never + imported, and closing them removes the only visible sign of that. What is left in the code is + a comment saying so and a test asserting the day stays out of balance, so the next + person to notice the arithmetic finds the reasoning before they find the fix.

+
+ +

Supporting changes

+
+ + + + + + + + +
ChangeWhy
Log each day's imbalance and its suspect linesan out-of-balance day was only visible by opening the screen; now it can be queried
Stop the dirty-summary scan at the client boundaryit read every later client's summaries too — 1,321 ms to 5.6 ms per client
Split the recompute driver into a per-client functionlets a backfill spread clients across threads instead of grinding one at a time
Install schema attributes before the tuples that compose themthe test suite could not build an empty database at all, so no test could run
+
+
+

That last one is worth a sentence for engineers: transact-schema installed + schema.edn then cloud-migration-schema.edn, but a composite tuple in the first file is built + from an attribute in the second. Datomic will not create a tuple before its members exist, so + every test fixture died in setup. It is very likely why sales summaries had no tests before + this work.

+
+
+ +
+

What each fix is worth

+
+

The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates. All 18,900 client-day summaries in the window are included, whether or not the restaurant traded that day.

+
+
+ + + + + + + +
StageDays out of balanceCleanTotal variance
Today's calculation, ninety days re-run1,25893.34%$69,560.10
+ refunded tips97194.86%$66,414.39
+ service charges27998.52%$7,790.54
+
+
+

Deduplication is not a row in this table, and that is deliberate. Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that $66,589.75 is what the three arithmetic fixes are worth on their own, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.

+
+ +

Day-by-day effect of each change

+
+ + + + + + + +
ChangeUnchangedInto balanceOut of balanceBalanced days alteredMoney moved
Refunded tips18,59028700$3,712.67
Service charges18,20869200$58,349.85
Both, end to end17,91697900$61,769.56
+
+
+

Neither fix touched a day that was already correct. Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered — 17,916 summaries came out byte-identical, and every one of the 984 that moved was already wrong. Service charges are by far the larger of the two, moving $58,349.85 against the tip fix's $3,712.67.

+

That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.

+

The two fixes account for the repair exactly. Adding up the untendered-tip and service-charge amounts for the 979 repaired days leaves a residue of 0.0000000013 against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a third effect.

+
+
+ +
+

What it looks like on the page

+
+

Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.

+
+ +

A refunded tip — NGLK, 2026-08-04

+
+ + + + + + + + + +
LineBeforeAfter
Tip482.94422.94
Card Refunds60.0060.00
Total money taken10,094.8110,094.81
Total money earned10,154.8110,094.81
Out of balance by−60.000.00
+
+
+

The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY: a tip of −60.00 on an order with no payment attached at all.

+
+ +

A service charge — NTPT, 2026-08-06

+
+ + + + + + + + + +
LineBeforeAfter
Service Chargesnot shown427.10
Card Payments4,975.894,975.89
Total money taken7,777.207,777.20
Total money earned7,350.107,777.20
Out of balance by+427.100.00
+
+ +

The largest repairs of each kind

+
+ + + + + + + + + + +
ClientDateLineBeforeAfterDay closed
NGPA2026-06-04Service Chargesnot shown1,344.86+1,344.86 → 0
NTPT2026-08-06Service Chargesnot shown427.10+427.10 → 0
N-300032026-05-27Service Chargesnot shown405.83+405.83 → 0
NGFL2026-05-19Tip238.4670.42−168.04 → 0
NGMI2026-07-09Tip230.0180.01−150.00 → 0
NGVA2026-07-03Tip152.6640.12−112.54 → 0
+
+
+

In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.

+
+
+ +
+

What was done to the data, and how it was checked

+
+

Every step below was performed against a restored copy of the production database. Production itself was never touched.

+
+
+ + + + + + + + + + + +
StepResult
Deactivate the duplicate client at each shared location10 locations · shared locations remaining: 0
Walk every order in the database19,040,785 orders
Give every order its own payment record9,100,314 re-keyed · 200,027 copied
Payments owned by two orders0 across 400,000 orders checked
Client-scope refunds, payouts and cash-drawer shiftscounts unchanged · 0 collisions
Live Square import afterwards0 orders with duplicated payment · 0 shared payments
Ownership changes after the change0 refunds · 0 payouts · 0 shifts
+
+
+

The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.

+
+ +
+ The whole analysis was run again from nothing, and landed in the same place +

Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. At the point the two runs are comparable — before the refund-only fix below, which the earlier run did not have — the end state matched to the cent: 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.

+
+ +
+ A bug in this work, found by measuring rather than reading +

The first attempt at copying shared payments derived each payment's Square identifier by stripping a fixed prefix. That is right the first time a payment is seen, but once it has been re-keyed to one client, a second order meeting it later read the already-scoped key as the identifier and scoped it twice — NGCD-CD-NGCC-CC-<id>. The importer then created a fresh payment, doubling the tender on five clients by $3,000–$7,000 each. It was caught because the totals were absurd, not because the code looked wrong. The fix recovers the scope from the record itself; client codes contain dashes, so it cannot be done by pattern. A test now runs the step one order at a time, which is the arrangement that exposes it.

+
+
+ +
+

What is still out of balance

+
+

279 client-days out of 18,900, totalling $7,790.54. Most of it is not a balancing fault, and the split is the point:

+
+
+ + + + + + + +
What the remainder actually isDaysVariance
Sales never imported — refunds on a record with no orders at all that day156$4,988.53
Processing fee on a day with no trading — the same shape, from the payout side15$974.99
Real trading days with genuine discrepancies108$1,995.36
+
+
+

Two thirds of what is left is a data-completeness problem, not an arithmetic one, and it is being left visible on purpose. Of those 156 days, 132 fall before their client had recorded a single order — they are periods with no sales in the system at all. The nine records involved hold 659 refunds worth $15,225.24 dated before their own first order.

+

The 15 fee-only days are the same story from the payout side: a processing fee landing on a day with no trading. Closing them needs the payout modelled, not another rule in the summary.

+
+
+ + + + + + + + + + +
Of the 108 real trading daysDaysVarianceWhat it is
NGBR1$299.42Square recorded $6,358.99 of tender against $6,059.57 of order totals on 2026-08-06 — the gap itself, not a summary fault
NGBK1$299.42
NGDL1$810.24an order on one record whose payment is on the twin — the mirror image of the refund case
NGMV5$259.38late May, undiagnosed
NGEB4$199.09ezCater fee treatment — an open question
everyone else96$127.81mostly till rounding — pennies a day
+
+
+

The clusters on NGMV and NGEB are unexplained and worth a look. They are visible only because the window is ninety days; a thirty-day view does not reach them.

+
+
+ +
+

How complete is this

+
+

The importer understands both the old and new record names, so the change can be deployed + before the renaming finishes. That tolerance is a bridge, not a destination — while any + record still carries an unscoped name, two clients can land on it and the guarantee rests on + a convention rather than on the data. So the renaming was run to completion and measured.

+
+
+ + + + + + + + +
Record typeTotalClient-scopedStill to renameCannot be scoped
Card payments17,047,14217,047,14200
Refunds51,98651,98600
Payouts144,688144,652036
Cash-drawer shifts69,29169,29100
+
+
+

Every record in the database now carries its owner's name, and the migration proposes no + further changes: asked what is left to do, it answers zero on all four record types. The 36 + payouts are ones with no client or location recorded anywhere, on the record itself or on + anything referring to it, so there is nothing to name them after.

+

Renaming had to be driven from orders, because a payment's rightful owner is whichever + order refers to it — so completing it meant walking all 19,040,785 orders, not + just the clients that look shared today. Nine client pairs contended in the past without + sharing a location now, and a migration scoped to the current configuration would have missed + every one of them.

+

A caution about how completeness is counted. 282,649 card payments carry no + client attribute of their own — they are stubs the payout path creates, never referenced by an + order. A gate that checks the attribute reports these as "no owner" and looks like a gap. They + are not: their names are scoped, recovered from the deposit that holds them. The figures above + are counted the harder way, by asking the migration what it would still change, which resolves + each record's owner through whatever refers to it. Reading the attribute alone would have + understated completeness by a quarter of a million records — and an early draft of this report + did exactly that.

+
+ +
+ + + + + + +
Shared payments after the migrationCountMeaning
Owned by more than one order0whether the orders belong to different clients or the same one
checked across400,000 ordersspread through the whole database
+
+
+

The problem this work exists to solve is gone: no payment answers to two orders, so the + component relationship means what it says and deleting an order can no longer take another + order's money with it.

+

One behaviour changed when the pass was run over everything, and it is worth + recording. Where Square splits one tender across two of a single client's own orders, + the earlier design left the payment shared on purpose — both orders compute the same name, so + there is no second name for a copy to take. That rule only holds for two orders processed in + the same batch. Run across nineteen million orders in batches of two thousand, such pairs + almost always fall in different batches and the second order now takes a copy. Inside the + ninety-day window this changed nothing measurable: the recompute after the database-wide pass + matched the one before it to the cent. Outside the window it has not been measured, and it + should be before this runs against production.

+

The guard on remove-voided-orders is still worth having regardless. It is + cheap, and it makes the safety a property of the deletion rather than of the migration having + been run first.

+
+ +
+ Re-running is safe, and that was proved at full scale +

After the complete pass, asking the migration what it would change next returns + nothing — 17,047,142 payments examined, none to rename, none unscopable. A + record that already carries the right name is left untouched, so the migration can be stopped, + resumed, or repeated without consequence.

+

Its speed is worth a note for whoever schedules it: the whole nineteen million orders were + walked in about thirteen minutes. An earlier attempt appeared to be + transactor-bound and was projected at two days, which is why a previous run narrowed it to the + analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process + driving the migration — freeing held memory took an unrelated recompute from 17 client-days a + minute to 4,515. Nothing about the database or the transactor needed to change.

+
+ +
+

What follows from the gate reading zero. unscoped-report counts + these figures on demand. Now that unscoped is zero across the board, the importer's + understanding of the old name form can be removed — at which point two clients sharing a + location becomes structurally incapable of producing a shared record, rather than prevented by + a convention that a future import could quietly break. That removal is the one remaining step + of this piece of work.

+
+
+ +
+

Decisions and risks still open

+
+ + + + + + + + + +
ItemWho decidesWhy it matters
Which client record survives at each shared locationthe businessthe newer record generally has no history before the split, so keeping it loses years of the location's books
Which revenue account service charges post toaccountingcurrently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances
659 refunds on records that have no sales for themthe business, then engineeringthe top open item. $15,225.24 dated before the holding record's own first order. Either the missing sales get imported, or the refunds move to the record that has them — but the books cannot close until one of the two happens
Whether to correct records the wrong client already ownsthe businessthe fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared
remove-voided-ordersengineeringsafe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes
+
+ +
+ Two operational findings, unrelated to the summaries +

The production backup had not written a restore point since 2025-03-10 — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.

+

The database server was sized for a toy dataset: a 2 GB cache against 27 GB of data. Worth checking what production is set to.

+

Slowness here was misdiagnosed twice, in the same direction. Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in thirteen minutes. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.

+
+
+ +
+

How to check any of this technical

+
+ 
;; the restored database, untouched production as of 2026-08-14 22:52
+(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore"))
+
+;; the two orders behind the worked examples
+(d/pull (d/db conn) '[*] [:sales-order/external-id
+                          "square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"])
+(d/pull (d/db conn) '[*] [:sales-order/external-id
+                          "square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"])
+
+;; the gate: no payment may have two parent orders
+(rk/charges-with-multiple-parents (d/db conn) orders)   ;; => 0
+
+;; ownership history — which records ever changed client
+(->> (d/datoms (d/history (d/db conn)) :aevt :sales-refund/client)
+     (filter :added)
+     (reduce (fn [m d] (update m (:e d) (fnil conj #{}) (:v d))) {})
+     (filter (fn [[_ owners]] (> (count owners) 1)))
+     count)
+
+
+

The comparison tool is committed as auto-ap.jobs.compare-sales-summaries. Unit tests: lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test.

+
+ +
+ Do not compare summaries with as-of — a correction to how this was measured +

The obvious way to audit a recompute is to read the database at a point before it and diff: + Datomic keeps every past value, so no snapshot is needed. That is what + compare-sales-summaries was built to do, and for summary amounts it does not work. + :ledger-mapped/amount, :ledger-mapped/ledger-side and + :ledger-mapped/account are all declared :db/noHistory true, so + superseded values are discarded rather than retained. A historical read of a summary that has + since been recomputed can return its lines with the categories intact and the amounts simply + absent — which reads as a legitimate all-zero summary, not as an error.

+

Every figure in this report is therefore taken from a live read of the database immediately + after each pass, captured and stored outside it, and the before/after comparison is done + between those two captures. No historical read is involved anywhere in the numbers above. The + tool remains useful for categories and for which days changed; its docstring overstates what it + can recover, and that is worth correcting before someone relies on it for amounts.

+
+
+ +
+ Measured 2026-08-15 against integreat-prod-restore, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch worktree-sales-summary-balance. + A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 279 remaining days, 189 are material — but 171 of those are days with no sales imported, not arithmetic faults. + Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on as-of above. +
+ +
diff --git a/docs/2026-08-15-sales-summary-rollout-plan.md b/docs/2026-08-15-sales-summary-rollout-plan.md new file mode 100644 index 00000000..9a720aea --- /dev/null +++ b/docs/2026-08-15-sales-summary-rollout-plan.md @@ -0,0 +1,268 @@ +# Sales-summary balancing — rollout plan + +Steps to execute, in order. Every step is either reversible or verifiable before the next one +begins. The one behaviour change that alters a client's books is behind a per-client feature flag +that is **off by default**, so merging and deploying this branch changes nothing on its own. + +Measured on a restored copy of production (backup point `209608347`), 210 clients over +2026-05-10 → 2026-08-07: **1,258 client-days out of balance / $69,560.10 → 279 days / $7,790.54**, +with zero days knocked out of balance and zero already-balanced days altered. + +Of the 279 days left, **171 are not balancing faults** — they are days where a client's sales were +never imported while its refunds were. Step 9 is about those, and it is the most important thing in +this document. + +--- + +## Before you start + +| | | +|---|---| +| Flag introduced | `summary-service-charges` — off by default | +| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` | +| Expected migration runtime | ~13 minutes for 19M orders on a warm cache | +| Nothing here touches | invoices, payments, the ledger, or any client without the flag set | + +**One prerequisite that is not code.** Ten Square locations are configured against two client records +each. Someone in the business has to decide which record survives at each. The newer record usually +has no history from before the split, so keeping it loses years of that location's books. Do this +before step 3. + +--- + +## Step 1 — Deploy the code + +Deploy the branch as normal. The flag is absent from every client, so: + +- tips are calculated exactly as they are today, +- no `Service Charges` line is written. + +The only changes that take effect immediately are the safe ones: imbalance logging, the +dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix, +and the importer's new client-scoped keys. + +**The importer starts writing client-scoped keys straight away, and reads both schemes.** That is +deliberate and is what makes the deploy independent of the migration. Do not remove the legacy +lookup in `square.core3/existing-id` yet — see step 8. + +**Verify before moving on.** After one nightly import cycle: + +```clojure +;; refunds, payouts and shifts must not have doubled +(count (d/datoms (d/db conn) :aevt :sales-refund/external-id)) +(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id)) +(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id)) +``` + +Compare against the same counts taken immediately before deploy. Growth should be ordinary daily +volume. A near-doubling means the legacy fallback is not working — **stop and roll back the deploy**. + +--- + +## Step 2 — Guard `remove-voided-orders` + +Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so +retracting an order cascades into its payments. Until step 3 finishes there are still payments with +two parent orders, and deleting one client's voided order can take the other client's payment with +it. + +Either leave `remove-voided-orders` switched off until step 3 completes, or change it to detach a +payment that has more than one parent rather than delete it. Detaching is worth doing regardless — +it makes the safety a property of the deletion rather than of the migration having been run first. + +See `docs/2026-08-15-remove-voided-orders-risk.md`. + +--- + +## Step 3 — Retire the duplicate client records + +Business decision from the top of this document. Deactivate the losing record's Square location so +the importer stops fetching for it. The record itself stays; its history is untouched. + +**Verify:** no Square location is configured against two active client records. + +--- + +## Step 4 — Run the migration + +```clojure +(require '[auto-ap.jobs.rekey-square-external-ids :as rk]) + +;; read-only first — check :collisions is empty for every attribute +(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys) + +;; then the whole thing +(rk/migrate-all! 2000) +``` + +Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that +already carries the right name is skipped, so it can be stopped and re-run without consequence. + +If it appears to crawl, the cause is almost certainly garbage collection in the process driving it, +not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free +retained memory in the REPL and re-measure before changing anything about the database. + +**Verify — all four must read zero to migrate and zero unscopable:** + +```clojure +(rk/unscoped-report (d/db conn)) +(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys) +;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0} + +;; and the gate that this work exists for +(rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn)))) +;; => 0 +``` + +Note `unscoped-report`'s `:no-owner` column is not a gap: ~283k payout-stub payments carry no +`:charge/client` attribute of their own, so it cannot verify them by attribute. `plan` resolves +ownership through whatever refers to them and is the figure to trust. + +--- + +## Step 5 — Recompute summaries, flags still off + +```clojure +(require '[auto-ap.jobs.sales-summaries :as ss]) +(ss/refresh-sales-summaries 90) +``` + +This is the pass that banks the deduplication. **Capture the result before going further** — you +will need it as the baseline for step 6, and it cannot be reconstructed afterwards: + +```clojure +(require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath +(def before (cmp/summaries-in (d/db conn) start end)) +(spit "before.edn" (pr-str before)) +``` + +> **Do not use `d/as-of` to compare summary amounts.** `:ledger-mapped/amount`, `ledger-side` and +> `account` are `:db/noHistory`, so past values are discarded. A summary that has since been +> recomputed reads back through `as-of` with its amounts *absent*, which looks like a legitimate +> balanced day. Capture live, before and after, and diff the captures. + +--- + +## Step 6 — Turn the flag on, a few restaurants at a time + +Needs accounting sign-off first: `summary-service-charges` posts to **49000 Service Income**, chosen +so the work could be measured. It affects reporting, never whether a day balances. + +```clojure +@(d/transact conn [{:db/id [:client/code "NGxx"] + :client/feature-flags ["summary-service-charges"]}]) +(ss/refresh-sales-summaries 90) +``` + +Start with two or three restaurants, confirm, then widen. + +**Verify** against the capture from step 5: + +```clojure +(def after (cmp/summaries-in (d/db conn) start end)) +(cmp/compare-window ...) ; both arguments live database values, never as-of +``` + +The two numbers that matter — both were zero across all 18,900 client-days in testing: + +- `:balanced->unbalanced` must be **0** +- previously-balanced days whose lines changed must be **0** + +If either is non-zero, retract the flag for the affected clients and re-run step 5. The flag is the +rollback: removing it restores today's behaviour exactly. + +--- + +## Step 7 — Re-enable `remove-voided-orders` + +Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2. + +--- + +## Step 8 — Remove the legacy key lookup + +Only once `plan` reports `:to-migrate 0` and has stayed there through several import cycles. Drop +the second branch of `square.core3/existing-id`. At that point two clients sharing a location +becomes structurally incapable of producing a shared record, rather than prevented by a convention a +future import could break. + +This is the last step and there is no hurry. + +--- + +## Step 9 — Deal with the refunds that have no sales behind them + +**The most important item in this document, and the only one that is not just execution.** + +156 of the 279 remaining days are a record carrying refunds on a day it recorded no sales at all. +132 of those fall *before that client's first ever order*. Nine records are affected, holding +**659 refunds worth $15,225.24 dated before their own first order**: + +| Client | First order | Refunds before it | Value | +|---|---|---:|---:| +| NGDL | 2026-07-29 | 96 | $2,751.50 | +| NGWN | 2026-07-30 | 95 | $2,218.61 | +| NGDU | 2026-08-02 | 94 | $2,188.44 | +| NG4S | 2026-05-29 | 78 | $2,168.30 | +| NGPS | 2026-05-26 | 81 | $2,167.60 | +| NGCT | 2026-08-01 | 98 | $1,789.49 | +| NGVC | 2026-08-02 | 95 | $1,594.17 | +| NGDS | 2026-01-14 | 22 | $347.13 | + +They did not make those refunds. The database's own ownership history shows a $35.35 refund dated +26 February belonging to `NGDG` that day and taken over by `NGDU` on 12 August, with others flipping +between the two several times a day across 12–15 August. Unscoped refund keys meant whichever import +ran last took ownership — fault 1, seen from the other end. Steps 1 and 4 stop it recurring; they do +not undo it. + +Note `NG4S` and `NGPS` are **not** among the ten shared locations, so this is wider than the +duplicate records. + +Two ways to close it, and the business has to pick: + +1. **Import the missing sales.** Correct if these records are meant to have their own books. Their + refunds are real and the matching orders exist in Square. +2. **Move the refunds to the record that has the sales.** Correct if the record is a duplicate that + should never have had books of its own. + +Before either, check whether the surviving record already holds the same Square refund — within the +ninety-day window all 192 orphans did, but **outside it payouts and shifts have no counterpart at +all**, so a blanket delete would destroy the only copy of some records. + +```clojure +;; per client: refunds dated before that client's own first order +(let [first-order (->> (d/q '[:find [?d ...] :in $ ?c + :where [?o :sales-order/client ?c] [?o :sales-order/date ?d]] + (d/db conn) [:client/code "NGDU"]) + (reduce (fn [a b] (if (.before a b) a b))))] + (->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c + :where [?r :sales-refund/client ?c]] + (d/db conn) [:client/code "NGDU"]) + (filter #(.before (:sales-refund/date %) first-order)) + count)) +``` + +**Until this is resolved those days stay out of balance, on purpose.** A summary change to close +them was written and measured — it works, closes 156 days and $4,820.19, and breaks nothing — and it +was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not +being imported. A test asserts the day stays unbalanced so nobody closes it without reading this. + +--- + +## What this will not fix + +The 108 real trading days still out of balance, $1,995.36 — the NGBR/NGBK tender gap ($6,358.99 of +tender against $6,059.57 of order totals on one day), the ezCater fee question, and unexplained +clusters on NGMV and NGEB. + +Plus 15 days / $974.99 where a processing fee lands on a day with no trading — the same shape as +step 9 but from the payout side, so it needs the payout modelled rather than a rule in the summary. + +--- + +## Two operational findings, unrelated to the summaries + +- **The production backup had not written a restore point since 2025-03-10** — about seventeen + months — although data files were still uploading daily. Worth an alert on restore-point age. +- **The database server is sized for a much smaller dataset**: a 2 GB cache against 27 GB of data. + Worth checking what production is set to. diff --git a/src/clj/auto_ap/datomic.clj b/src/clj/auto_ap/datomic.clj index 80b9271e..af6a9ad2 100644 --- a/src/clj/auto_ap/datomic.clj +++ b/src/clj/auto_ap/datomic.clj @@ -875,13 +875,22 @@ (defn all-schema [] (edn/read-string (slurp (io/resource "schema.edn")))) -(defn transact-schema [conn] - @(dc/transact conn - (edn/read-string (slurp (io/resource "schema.edn")))) +(defn transact-schema + "Installs the schema in two passes: every plain attribute first, then every composite tuple. - ;; this is temporary for any new stuff that needs to be asserted for cloud migration. - @(dc/transact conn - (edn/read-string (slurp (io/resource "cloud-migration-schema.edn"))))) + A tuple can only be created once the attributes it composes already exist, and the pieces are + spread across both files — `:journal-entry-line/running-balance-tuple` lives in schema.edn + while one of its members, `:journal-entry-line/running-balance`, lives in + cloud-migration-schema.edn. Transacting the files in order therefore cannot install that tuple + against an empty database. Long-lived databases never hit it because those attributes went in + years apart." + [conn] + (let [schema (concat (edn/read-string (slurp (io/resource "schema.edn"))) + ;; this is temporary for any new stuff that needs to be asserted for cloud migration. + (edn/read-string (slurp (io/resource "cloud-migration-schema.edn")))) + {tuples true plain false} (group-by #(contains? % :db/tupleAttrs) schema)] + (when (seq plain) @(dc/transact conn plain)) + (when (seq tuples) @(dc/transact conn tuples)))) (defn backoff [n] (let [base-timeout 500 diff --git a/src/clj/auto_ap/datomic/sales_summaries.clj b/src/clj/auto_ap/datomic/sales_summaries.clj index c8d2dd69..8624889b 100644 --- a/src/clj/auto_ap/datomic/sales_summaries.clj +++ b/src/clj/auto_ap/datomic/sales_summaries.clj @@ -36,6 +36,13 @@ (defn balanced? [items] (dollars= (total-debits items) (total-credits items))) +(defn imbalance + "Signed debits minus credits. `balanced?` answers yes or no; this says by how much and in + which direction, so a day that does not balance can be logged and queried rather than only + rendered red. Positive means the tender side exceeds what revenue accounts for." + [items] + (- (total-debits items) (total-credits items))) + (defn accepted? "True once a summary is finished: every line is mapped to an account and debits equal credits. This is the same condition the sales summaries grid renders as \"Balanced\", and diff --git a/src/clj/auto_ap/jobs/rekey_square_external_ids.clj b/src/clj/auto_ap/jobs/rekey_square_external_ids.clj new file mode 100644 index 00000000..693fc55e --- /dev/null +++ b/src/clj/auto_ap/jobs/rekey_square_external_ids.clj @@ -0,0 +1,314 @@ +(ns auto-ap.jobs.rekey-square-external-ids + "One-shot migration re-keying Square entities to client-scoped external ids. + + Refunds, charges, Square payouts (expected deposits) and cash drawer shifts all carry keys with + no client scoping, so two clients configured on the same Square location share a single entity: + its owner flips every time either client imports. Sales orders and ezCater orders already scope + their keys by client and location; this brings the rest in line. + + Measured on a restored production backup, ownership had actually changed on 3,387 refunds, + 4,069 expected deposits and 2,628 cash drawer shifts, across 19 distinct client pairs — nine of + which no longer share a location in the current configuration and so are invisible to any + point-in-time check. + + Run AFTER the importer knows how to resolve both key schemes (`square.core3/existing-id`). + Running it first would be harmless, but the importer would then re-create legacy-keyed + entities on its next pass. + + The migration is idempotent: an entity already carrying its scoped key is skipped, so it can + be re-run over a partially migrated database." + (:require + [auto-ap.datomic :refer [conn]] + [auto-ap.logging :as alog] + [datomic.api :as dc])) + +(def refund-prefix "square/refund/") +(def charge-prefix "square/charge/") +(def deposit-prefix "square/payout/") +(def shift-prefix "square/cash-drawer-shift/") + +(defn- scope-of + "`[client-code location]` for an entity, or nil when it cannot be determined. + + Charges are the awkward case: about an eighth of them carry neither `:charge/client` nor + `:charge/location`. Those are stubs minted by the payout path, which asserts an external id + alone and lets unique-identity upsert bring a bare entity into being, plus older tender + records that predate the client attribute. None are orphaned, so the scope is recovered from + whatever references them — the sales order first, then the expected deposit." + [db attr e] + (let [ent (dc/entity db e) + pair (fn [code loc] (when (and code loc) [code loc]))] + (or (case attr + :sales-refund/external-id (pair (:client/code (:sales-refund/client ent)) + (:sales-refund/location ent)) + :charge/external-id (pair (:client/code (:charge/client ent)) + (:charge/location ent)) + :expected-deposit/external-id (pair (:client/code (:expected-deposit/client ent)) + (:expected-deposit/location ent)) + :cash-drawer-shift/external-id (pair (:client/code (:cash-drawer-shift/client ent)) + (:cash-drawer-shift/location ent))) + (when-let [o (:e (first (dc/datoms db :vaet e :sales-order/charges)))] + (let [oe (dc/entity db o)] + (pair (:client/code (:sales-order/client oe)) (:sales-order/location oe)))) + (when-let [d (:e (first (dc/datoms db :vaet e :expected-deposit/charges)))] + (let [de (dc/entity db d)] + (pair (:client/code (:expected-deposit/client de)) (:expected-deposit/location de))))))) + +(defn planned-key + "`[eid new-key]` for an entity that still needs re-keying, or nil when it is already scoped or + cannot be scoped at all. + + Detection compares against the key this entity *should* have rather than pattern-matching the + id, because Square ids may themselves contain dashes and no pattern separates the two schemes + reliably. That also makes the migration idempotent." + [db attr prefix datom] + (let [old (:v datom)] + (when-let [[code loc] (scope-of db attr (:e datom))] + (let [scoped-prefix (str prefix code "-" loc "-")] + (when-not (.startsWith ^String old scoped-prefix) + [(:e datom) (str scoped-prefix (subs old (count prefix)))]))))) + +(defn plan + "Everything the migration would change, plus what it cannot touch. Read-only — run this and + check `:collisions` is empty before transacting anything." + [db attr prefix] + (let [acc (reduce (fn [acc d] + (let [acc (update acc :total inc)] + (if-let [[e new-key] (planned-key db attr prefix d)] + (-> acc + (update :to-migrate inc) + (update :new-keys conj! [e new-key])) + (if (scope-of db attr (:e d)) + (update acc :already-scoped inc) + (update acc :unscopable inc))))) + {:total 0 :to-migrate 0 :already-scoped 0 :unscopable 0 :new-keys (transient [])} + (dc/datoms db :aevt attr))] + (update acc :new-keys persistent!))) + +(defn collisions + "Any two entities that would land on the same new key. Must be empty: a collision would merge + two entities into one and lose whichever lost." + [new-keys] + (->> new-keys + (group-by second) + (keep (fn [[k es]] (when (> (count es) 1) [k (mapv first es)]))) + vec)) + +(defn migrate! + "Asserts the new external id on each planned entity. The attribute is cardinality one, so the + legacy value is retracted by the same assertion and the entity keeps its identity — nothing is + created and nothing is deleted. + + Returns the number of entities re-keyed." + [attr new-keys batch-size] + (let [total (count new-keys)] + (alog/info ::migrating :attr attr :count total) + (doseq [[i batch] (map-indexed vector (partition-all batch-size new-keys))] + @(dc/transact conn (for [[e new-key] batch] {:db/id e attr new-key})) + (when (zero? (mod i 20)) + (alog/info ::migrated :attr attr :done (* i batch-size) :of total))) + total)) + +(def charge-copy-attrs + "Everything a charge carries in its own right. `:charge/client+date` is a tuple Datomic + maintains, and `:charge/external-id` is set separately, so neither is copied." + [:charge/type-name :charge/total :charge/tip :charge/tax :charge/date + :charge/processor :charge/note :charge/reference-link]) + +(defn- raw-square-id + "The Square id inside a charge's external id, with any client scoping removed. + + Stripping only the `square/charge/` prefix is not enough. Once a charge has been scoped to one + client, a second order processing the same charge would read `NGCC-CC-` as the id and scope + it again, producing `square/charge/NGCD-CD-NGCC-CC-`. The importer then computes the + correct single-scoped key, fails to find it, and creates a second charge — silently doubling + the tender. + + Client codes may themselves contain dashes, so the scope cannot be recognised by pattern. It is + recovered from the entity instead: whoever the charge currently belongs to is exactly whose + scope its key carries." + [db charge old] + (let [ent (dc/entity db charge) + code (:client/code (:charge/client ent)) + loc (:charge/location ent) + owner-prefix (when (and code loc) (str charge-prefix code "-" loc "-"))] + (cond + (and owner-prefix (.startsWith ^String old ^String owner-prefix)) (subs old (count owner-prefix)) + (.startsWith ^String old charge-prefix) (subs old (count charge-prefix)) + :else old))) + +(defn- charge-plan-for-order + "What one order needs doing to its charges. + + `:keep` — no other order has claimed this charge yet, so this order takes it and it is renamed + in place. `:clone` — another order already owns it, so this order needs its own copy. + + Splitting is per client, not per order. Where two orders of the SAME client and location refer + to one payment — Square splitting a tender across orders, or an amendment — they are left + sharing it deliberately. Both would compute the same name, so there is no second name to give + a copy, and more importantly a copy would double that client's takings for the day. The + component cascade still applies to those, which is why the retraction guard on + `remove-voided-orders` is needed regardless of this migration. + + Whether a charge is claimed is read from the charge itself: an unclaimed one still carries the + bare `square/charge/` form. Deciding it that way rather than by remembering + every charge seen so far is what lets this run across all 16 million of them — the alternative + needs a map of the entire table in memory. `batch-seen` covers only the orders inside one + transaction, where the database snapshot cannot yet show a claim made moments earlier." + [db order-eid batch-seen] + (let [oe (dc/entity db order-eid) + code (:client/code (:sales-order/client oe)) + loc (:sales-order/location oe)] + (when (and code loc) + (for [d (dc/datoms db :eavt order-eid :sales-order/charges) + :let [charge (:v d) + old (:v (first (dc/datoms db :eavt charge :charge/external-id)))] + :when old + :let [raw (raw-square-id db charge old) + new-key (str charge-prefix code "-" loc "-" raw) + claimed (get @batch-seen charge) + unclaimed? (and (= old (str charge-prefix raw)) (nil? claimed))] + ;; nothing to do when the charge already answers to this order's name, or when + ;; another order in this same batch has just claimed it under that very name — + ;; that is the same-client case, which stays shared + :when (and (not= old new-key) (not= claimed new-key))] + {:order order-eid :charge charge :new-key new-key :raw raw + :client (:db/id (:sales-order/client oe)) :location loc + :action (if unclaimed? :keep :clone)})))) + +(defn split-and-rekey-charges! + "Gives every order its own charge entity, keyed by that order's client and location. + + Where two clients were configured on one Square location, both clients' orders resolved to a + single charge, because charge keys carried no client. Re-keying alone does not undo that — it + hands the one entity to whichever client is looked at first and leaves the other order pointing + at a charge it does not own. Since `:sales-order/charges` is a component attribute, that is not + merely untidy: retracting either order would delete a charge the other one still needs. + + So a shared charge is cloned. The first order to claim it keeps it, re-keyed to that order's + scope; every other order gets a copy carrying the same amounts, scoped to its own client, and + has its reference repointed. Afterwards no charge has more than one parent order and the + component relationship means what it says. + + `orders` is the collection of order entity ids to process — typically every order belonging to + the clients that share, or have ever shared, a Square location." + [orders batch-size] + (let [cloned (atom 0) + rekeyed (atom 0)] + (doseq [batch (partition-all batch-size orders)] + (let [db (dc/db conn) + batch-seen (atom {}) + tx (doall + (for [o batch + plan (charge-plan-for-order db o batch-seen) + :let [{:keys [charge new-key action client location]} plan] + tx-item (if (= :keep action) + (do (swap! batch-seen assoc charge new-key) + (swap! rekeyed inc) + ;; record who claimed it: the owner is how a later order + ;; recovers the Square id from an already-scoped key + [{:db/id charge + :charge/external-id new-key + :charge/client client + :charge/location location}]) + (let [ent (dc/entity db charge) + copy (reduce (fn [m a] (if-some [v (get ent a)] + (assoc m a (if (map? v) (:db/id v) v)) + m)) + {} charge-copy-attrs)] + (swap! cloned inc) + [(assoc copy + :db/id new-key + :charge/external-id new-key + :charge/client client + :charge/location location) + [:db/retract o :sales-order/charges charge] + {:db/id o :sales-order/charges new-key}]))] + tx-item))] + (when (seq tx) @(dc/transact conn tx)))) + (alog/info ::split-charges :rekeyed @rekeyed :cloned @cloned) + {:rekeyed @rekeyed :cloned @cloned})) + +(defn charges-with-multiple-parents + "The §3.3 gate. Must read zero once the split has run: while any charge has two parent orders, + retracting either order deletes the other one's payment." + [db orders] + (->> orders + (mapcat (fn [o] (map :v (dc/datoms db :eavt o :sales-order/charges)))) + distinct + (filter (fn [c] (> (reduce (fn [n _] (inc n)) 0 (dc/datoms db :vaet c :sales-order/charges)) 1))) + count)) + +(def scoped-attrs + "Every Square-imported entity whose key must carry its client, with the prefix and where to + read the owner from." + [{:attr :sales-refund/external-id :prefix refund-prefix + :client :sales-refund/client :location :sales-refund/location} + {:attr :charge/external-id :prefix charge-prefix + :client :charge/client :location :charge/location} + {:attr :expected-deposit/external-id :prefix deposit-prefix + :client :expected-deposit/client :location :expected-deposit/location} + {:attr :cash-drawer-shift/external-id :prefix shift-prefix + :client :cash-drawer-shift/client :location :cash-drawer-shift/location}]) + +(defn unscoped-report + "Counts, per entity type, how many keys are already client-scoped, how many still carry the + legacy unscoped form, and how many have no owner to scope by. + + This is the completeness gate. The importer tolerates both key schemes on purpose, so that the + change can be deployed before the migration finishes — but that tolerance is a transition, not + a resting place. While `:legacy` is above zero the database is in a mixed state and a stray + unscoped record can still be adopted by whichever client imports it first. Once every count + reads zero the fallback lookup in `square.core3/existing-id` can be removed and the guarantee + becomes structural rather than conventional." + [db] + (into {} + (for [{:keys [attr prefix client location]} scoped-attrs] + [attr (reduce (fn [acc d] + (let [e (dc/entity db (:e d)) + code (:client/code (client e)) + loc (location e) + scoped (when (and code loc) (str prefix code "-" loc "-"))] + (cond + (and scoped (.startsWith ^String (:v d) ^String scoped)) (update acc :scoped inc) + (nil? scoped) (update acc :no-owner inc) + :else (update acc :legacy inc)))) + {:scoped 0 :legacy 0 :no-owner 0} + (dc/datoms db :aevt attr))]))) + +(defn all-order-ids + "Every sales order in the database, streamed." + [db] + (map :e (dc/datoms db :aevt :sales-order/external-id))) + +(defn migrate-all! + "The complete migration, over the whole database rather than a chosen subset. + + Splitting is driven from orders, because a payment's rightful owner is whichever order refers + to it — so every order has to be walked, not merely the clients that share a location today. + Nine client pairs contended in the past and no longer share one; their records are still mixed, + and a migration scoped to the current configuration would miss every one of them. + + Returns the split counts and the completeness report, which should read zero legacy across the + board when this finishes." + [batch-size] + (let [split (split-and-rekey-charges! (all-order-ids (dc/db conn)) batch-size)] + (doseq [{:keys [attr prefix]} scoped-attrs + :when (not= attr :charge/external-id)] + (let [p (plan (dc/db conn) attr prefix)] + (when-let [c (seq (collisions (:new-keys p)))] + (throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)}))) + (migrate! attr (:new-keys p) batch-size))) + ;; charges no order refers to — payout stubs — are scoped from the deposit that holds them + (let [p (plan (dc/db conn) :charge/external-id charge-prefix)] + (when (seq (:new-keys p)) (migrate! :charge/external-id (:new-keys p) batch-size))) + {:split split :completeness (unscoped-report (dc/db conn))})) + +(defn counts + "Entity totals, for the before/after assertion that is this migration's real safety net: if + either number moves, the re-key created duplicates instead of updating in place." + [db] + {:refunds (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :sales-refund/external-id)) + :charges (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :charge/external-id)) + :deposits (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :expected-deposit/external-id)) + :shifts (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :cash-drawer-shift/external-id))}) diff --git a/src/clj/auto_ap/jobs/sales_summaries.clj b/src/clj/auto_ap/jobs/sales_summaries.clj index 52391a1b..980df62b 100644 --- a/src/clj/auto_ap/jobs/sales_summaries.clj +++ b/src/clj/auto_ap/jobs/sales_summaries.clj @@ -62,15 +62,24 @@ {:ledger-mapped/ledger-side [:db/ident]} {:ledger-mapped/account [:db/id]}]) -(defn dirty-sales-summaries [c] +(defn dirty-sales-summaries + "The client's dirty summaries, with enough of each item to evaluate and re-transact it. + + `index-pull` returns a lazy seq running from `:start` to the END of the index, so this must + stop at the client boundary rather than filter: `:sales-summary/client+dirty` sorts by client + first, so every later client's summaries sit beyond this client's and filtering would walk all + of them — for every client — pulling their items on the way. That is quadratic in the number of + summaries, and it showed up as a full refresh degrading from ~180 client-days a minute to ~3 as + the summary count grew." + [c] (let [client-id (dc/entid (dc/db conn) c)] (->> (dc/index-pull (dc/db conn) {:index :avet :selector (conj '[:sales-summary/date :sales-summary/client :db/id] {:sales-summary/items item-read}) :start [:sales-summary/client+dirty [client-id true]]}) - (filter (fn [sales-summary] - (= client-id (:db/id (:sales-summary/client sales-summary)))))))) + (take-while (fn [sales-summary] + (= client-id (:db/id (:sales-summary/client sales-summary)))))))) (def default-days "How far back the scheduled refresh looks for summaries that still need recomputing." @@ -147,8 +156,18 @@ date)) 0.0))) +(def service-charges-account + "Where a credited Square service charge lands. 49000 is the existing \"Service Income\" + revenue account, which is the closest fit for auto-gratuity and catering fees. + + NEEDS ACCOUNTING SIGN-OFF before `service-charges-flag` is enabled for any client: the wrong + account misstates revenue, and a category with no account at all keeps a day from ever + reaching accepted, since `accepted?` requires every line to be mapped." + 49000) + (def name->number {"gyros and pitas" 40111 + "service charges" service-charges-account "returns" 41300 "card payments" 75460 "cash payments" 75452 @@ -292,21 +311,48 @@ [[c] date date])) 0.0)}) -(defn- get-tip [c date] +(defn- tendered-tip + "Tips read off the tenders, which is where a tip actually settles." + [c date] + (or (ffirst (dc/q '[:find (sum ?tip) + :with ?c + :in $ [?clients ?start-date ?end-date] + :where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] + [?e :sales-order/charges ?c] + [?c :charge/tip ?tip]] + (dc/db conn) + [[c] date date])) + 0.0)) + +(defn- untendered-tip + "Tips on orders that carry no tender at all. A return-only order reverses its tip on + `:sales-order/tip` but has no charge to join through, so the reversal is invisible to + `tendered-tip` and the day ends up crediting a tip that was handed back." + [c date] + (or (ffirst (dc/q '[:find (sum ?tip) + :with ?e + :in $ [?clients ?start-date ?end-date] + :where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] + [?e :sales-order/tip ?tip] + (not [?e :sales-order/charges])] + (dc/db conn) + [[c] date date])) + 0.0)) + +(defn- get-tip + "Tendered tips plus the tips on untendered orders. Additive rather than substitutive on + purpose: where an order does have a tender, the tender is the correct source, and real + orders exist whose tender carries a tip their `:sales-order/tip` does not — auto-gratuity + booked as a service charge, and wallet tips absent from the net amounts. Reading the order + instead of the tender would drop those." + [c date] {:ledger-mapped/ledger-side :ledger-side/credit :sales-summary-item/sort-order 2 :db/id (str (java.util.UUID/randomUUID)) :sales-summary-item/category "Tip" - :ledger-mapped/amount (or (ffirst (dc/q '[:find (sum ?tip) - :with ?c - :in $ [?clients ?start-date ?end-date] - :where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] - [?e :sales-order/charges ?c] - [?c :charge/tip ?tip]] - (dc/db conn) - [[c] date date])) - 0.0)}) + :ledger-mapped/amount (+ (tendered-tip c date) + (untendered-tip c date))}) (defn- get-sales [c date] (let [sales (->> (dc/q '[:find ?category (sum ?total) (sum ?tax) (sum ?discount) @@ -332,6 +378,14 @@ :ledger-mapped/amount (- (+ total discount) tax) #_#_:ledger-mapped/account nil}))) +;; A day carrying refunds and no sales at all is left out of balance on purpose. It is tempting +;; to close it by booking a return against the day's refunds — the arithmetic works, and no +;; trading day could be affected. Do not. Those days are overwhelmingly not "a refund settled +;; while the restaurant was shut": they are days whose *orders were never imported*, on client +;; records that took ownership of another record's refunds through the unscoped keys this branch +;; fixes. Balancing them would convert the only signal that a client's sales are missing into +;; silence. See `docs/2026-08-15-sales-summary-rollout-plan.md`. + (defn- get-returns [c date] (when-let [amount (ffirst (dc/q '[:find (sum ?r) :with ?e @@ -349,12 +403,82 @@ :ledger-mapped/amount amount :ledger-mapped/ledger-side :ledger-side/debit})) -(defn sales-summaries-v2 [] - (doseq [[c client-code] (dc/q '[:find ?c ?client-code - :in $ - :where [?c :client/code ?client-code]] - (dc/db conn)) - {:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)] +(def service-charges-flag + "Per-client rollout lever for crediting Square service charges, in the same style as + `new-square` and `import-custom-amount`. Absent, the summary behaves exactly as it does + today." + "summary-service-charges") + +(defn- service-charges-enabled? [c] + (contains? (set (:client/feature-flags (dc/pull (dc/db conn) '[:client/feature-flags] c))) + service-charges-flag)) + +(defn service-charge-total + "Square service charges for the day, both signs. + + A service charge is collected inside the card tender but nothing credits it, so every order + carrying one leaves the day short by exactly that amount. Both signs matter: a returned + catering fee arrives as a negative service charge and is subtracted back out of + `:sales-order/returns`, so dropping negatives would lose the reversal. + + The vendor gate is load-bearing — ezCater service charges are commission deducted from the + restaurant rather than collected from the diner, and crediting those would make things worse. + It matches on `:sales-order/vendor` where that is set and falls back to the external id + prefix where it is not, because whole eras of Square orders carry no vendor attribute at all + and a gate on vendor alone silently credits nothing. + + Kept separate from the rollout flag so the arithmetic can be measured on its own." + [c date] + (ffirst (dc/q '[:find (sum ?service-charge) + :with ?e + :in $ [?clients ?start-date ?end-date] + :where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] + [?e :sales-order/service-charge ?service-charge] + (or-join [?e] + [?e :sales-order/vendor :vendor/ccp-square] + (and (not [?e :sales-order/vendor]) + [?e :sales-order/external-id ?external-id] + [(clojure.string/starts-with? ?external-id "square/order/")]))] + (dc/db conn) + [[c] date date]))) + +(defn- get-service-charges + "The day's service charges as a summary item, for clients opted in to the rollout." + [c date] + (when (service-charges-enabled? c) + (when-let [amount (service-charge-total c date)] + (when-not (zero? amount) + {:db/id (str (java.util.UUID/randomUUID)) + :sales-summary-item/category "Service Charges" + :sales-summary-item/sort-order 2 + :ledger-mapped/amount amount + :ledger-mapped/ledger-side :ledger-side/credit})))) + +(def ^:private suspect-categories + "The terms a balancing investigation keeps returning to. Logged beside the imbalance so a + day's shape can be read out of the logs without re-running the job." + ["Tip" "Service Charges" "Returns" "Card Refunds" "Cash Refunds" "Food App Refunds"]) + +(defn- suspect-totals + "Amounts for `suspect-categories` present on this day, omitting the ones that are zero." + [items] + (into {} + (for [category suspect-categories + :let [amount (->> items + (filter #(= category (:sales-summary-item/category %))) + (map #(:ledger-mapped/amount % 0.0)) + (reduce + 0.0))] + :when (not (zero? amount))] + [category amount]))) + +(defn refresh-client! + "Recomputes every dirty summary for one client. + + Split out of the driver loop so a client's work stands on its own: it can be run for a single + client, and a backfill over the whole history can spread clients across threads instead of + grinding through the largest ones one day at a time." + [c client-code] + (doseq [{:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)] (mu/with-context {:client-code client-code :date date} (alog/info ::updating) @@ -370,6 +494,7 @@ (cons (get-fees c date)) (cons (get-tax c date)) (cons (get-tip c date)) + (cons (get-service-charges c date)) (cons (get-returns c date)) (filter identity) (map (fn [z] @@ -385,10 +510,22 @@ (if (seq (:sales-summary/items result)) (do (alog/info ::upserting-summaries - :category-count (count (:sales-summary/items result))) + :category-count (count (:sales-summary/items result)) + :imbalance (d-ss/imbalance all-items) + :balanced? (d-ss/balanced? all-items) + :suspect-totals (suspect-totals all-items)) @(dc/transact conn [[:upsert-sales-summary result]])) @(dc/transact conn [{:db/id id :sales-summary/dirty false}])))))) +(defn sales-summaries-v2 + "Recomputes every dirty summary, client by client." + [] + (doseq [[c client-code] (dc/q '[:find ?c ?client-code + :in $ + :where [?c :client/code ?client-code]] + (dc/db conn))] + (refresh-client! c client-code))) + (defn reset-summaries [] @(dc/transact conn (->> (dc/q '[:find ?sos :in $ diff --git a/src/clj/auto_ap/square/core3.clj b/src/clj/auto_ap/square/core3.clj index 77ab94e1..cfa7a7c0 100644 --- a/src/clj/auto_ap/square/core3.clj +++ b/src/clj/auto_ap/square/core3.clj @@ -269,6 +269,29 @@ 0.0 [])) +(defn scoped-key + "Client-scoped external id, in the shape sales order keys already use. + + Without the client and location in the key, two clients configured on the same Square location + collide on a single entity: a refund changes owner every time either client imports, and one + charge ends up shared between both clients' orders." + [prefix client location id] + (str prefix (:client/code client) "-" (:square-location/client-location location) "-" id)) + +(defn existing-id + "Entity id of the refund or charge this id already refers to, trying the client-scoped key + first and the legacy unscoped key second. + + This is what makes re-keying safe. These external ids are `:db.unique/identity`, so the import + relies on upsert-by-identity; changing the key format on its own would match nothing and + Datomic would create a SECOND entity for every refund and charge, orphaning the original under + its legacy key. Pinning the result as `:db/id` makes the write land on the existing entity + whichever scheme it currently carries." + [db attr prefix client location id] + (when id + (or (dc/entid db [attr (scoped-key prefix client location id)]) + (dc/entid db [attr (str prefix id)])))) + (defn tender->charge [order client location t] (remove-nils #:charge @@ -278,8 +301,9 @@ :note (:note t) :location (:square-location/client-location location) :reference-link (str (url/url "https://squareup.com/receipt/preview" (:id t))) + :db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location (:id t)) :external-id (when (:id t) - (str "square/charge/" (:id t))) + (scoped-key "square/charge/" client location (:id t))) :processor (cond (#{"OTHER" "THIRD_PARTY_CARD"} (:type t)) (condp = (some-> (:note t) str/lower-case) @@ -509,7 +533,7 @@ (try (->> (for [payout payouts :let [best-sales-date (some->> (dc/q '[:find ?s4 (count ?s) - :in $ ?payout-id + :in $ [?payout-id ...] :where [?payout :expected-deposit/external-id ?payout-id] [?payout :expected-deposit/charges ?c] @@ -519,7 +543,8 @@ [(auto-ap.time/localize ?s2) ?s3] [(clj-time.coerce/to-local-date ?s3) ?s4]] (dc/db conn) - (str "square/payout/" (:id payout))) + [(scoped-key "square/payout/" client location (:id payout)) + (str "square/payout/" (:id payout))]) (sort-by last) last first @@ -543,7 +568,10 @@ (:db/id client) (amount->money (:amount_money payout))))] :when (not equivalent-already-exists?)] - #:expected-deposit {:external-id (str "square/payout/" (:id payout)) + #:expected-deposit {:db/id (or (existing-id (dc/db conn) :expected-deposit/external-id + "square/payout/" client location (:id payout)) + (str "square/payout/" (:id payout))) + :external-id (scoped-key "square/payout/" client location (:id payout)) :vendor :vendor/ccp-square :status :expected-deposit-status/pending :total (amount->money (:amount_money payout)) @@ -561,7 +589,11 @@ (coerce/to-date))) :charges (reverse (->> (:payout_entries payout) (filter (comp :payment_id :type_charge_details)) - (map (fn [p] {:charge/external-id (str "square/charge/" (:payment_id (:type_charge_details p)))}))))}) + (map (fn [p] + (let [payment-id (:payment_id (:type_charge_details p))] + (remove-nils + {:charge/external-id (scoped-key "square/charge/" client location payment-id) + :db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location payment-id)}))))))}) (filter :expected-deposit/date) (into [])) (catch Throwable e @@ -585,7 +617,8 @@ (de/chain (get-payment client (:payment_id r)) (fn [payment] - #:sales-refund {:external-id (str "square/refund/" (:id r)) + #:sales-refund {:db/id (existing-id (dc/db conn) :sales-refund/external-id "square/refund/" client l (:id r)) + :external-id (scoped-key "square/refund/" client l (:id r)) :vendor :vendor/ccp-square :total (amount->money (:amount_money r)) :fee (transduce @@ -703,7 +736,10 @@ (de/chain (get-cash-shift client (:id s)) (fn [cash-drawer-shift] - #:cash-drawer-shift {:external-id (str "square/cash-drawer-shift/" (:id cash-drawer-shift)) + #:cash-drawer-shift {:db/id (or (existing-id (dc/db conn) :cash-drawer-shift/external-id + "square/cash-drawer-shift/" client l (:id cash-drawer-shift)) + (str "square/cash-drawer-shift/" (:id cash-drawer-shift))) + :external-id (scoped-key "square/cash-drawer-shift/" client l (:id cash-drawer-shift)) :vendor :vendor/ccp-square :paid-in (amount->money (:cash_paid_in_money cash-drawer-shift)) :paid-out (amount->money (:cash_paid_out_money cash-drawer-shift)) @@ -723,10 +759,12 @@ :when (:square-location/client-location square-location)] (upsert-cash-shifts client square-location)))) ([client location] + (upsert-cash-shifts client location (time/plus (time/now) (time/days -75)) (time/now))) + ([client location start end] (with-context-as {:source "Square cash shift loading" :client (:client/code client)} lc - (de/chain (cash-drawer-shifts client location) + (de/chain (cash-drawer-shifts client location start end) (fn [cash-shifts] (mu/with-context lc (try diff --git a/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj b/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj new file mode 100644 index 00000000..a5825462 --- /dev/null +++ b/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj @@ -0,0 +1,149 @@ +(ns auto-ap.jobs.rekey-square-external-ids-test + (:require + [auto-ap.datomic :refer [conn]] + [auto-ap.integration.util :refer [setup-test-data wrap-setup]] + [auto-ap.jobs.rekey-square-external-ids :as sut] + [clojure.string] + [clojure.test :refer [deftest is testing use-fixtures]] + [datomic.api :as dc])) + +(use-fixtures :each wrap-setup) + +(def sales-date #inst "2026-08-01T07:00:00.000-00:00") + +(defn- charge-count [] + (count (dc/q '[:find ?e :where [?e :charge/external-id]] (dc/db conn)))) + +(defn- charges-of [order] + (->> (dc/datoms (dc/db conn) :eavt order :sales-order/charges) + (map :v) + (map (fn [c] {:eid c + :key (:v (first (dc/datoms (dc/db conn) :eavt c :charge/external-id))) + :total (:v (first (dc/datoms (dc/db conn) :eavt c :charge/total))) + :tip (:v (first (dc/datoms (dc/db conn) :eavt c :charge/tip)))})) + vec)) + +(defn- parents-of [charge] + (reduce (fn [n _] (inc n)) 0 (dc/datoms (dc/db conn) :vaet charge :sales-order/charges))) + +(defn- two-orders-sharing-one-charge [] + (let [{:strs [test-client-id]} (setup-test-data []) + other (get-in @(dc/transact conn [{:db/id "other" :client/code "NGCC"}]) [:tempids "other"]) + tx @(dc/transact conn [{:db/id "charge" + :charge/external-id "square/charge/shared1" + :charge/type-name "CARD" + :charge/total 120.0 + :charge/tip 20.0} + {:db/id "order-a" + :sales-order/external-id "square/order/NGCD-CD-o1" + :sales-order/client test-client-id + :sales-order/location "CD" + :sales-order/date sales-date + :sales-order/charges ["charge"]} + {:db/id "order-b" + :sales-order/external-id "square/order/NGCC-CC-o1" + :sales-order/client other + :sales-order/location "CC" + :sales-order/date sales-date + :sales-order/charges ["charge"]}])] + {:order-a (get-in tx [:tempids "order-a"]) + :order-b (get-in tx [:tempids "order-b"]) + :charge (get-in tx [:tempids "charge"]) + :code-a (:client/code (dc/entity (dc/db conn) test-client-id)) + :code-b "NGCC"})) + +(deftest a-shared-charge-starts-with-two-parents + (testing "the condition under test really exists before the split runs" + (let [{:keys [charge]} (two-orders-sharing-one-charge)] + (is (= 1 (charge-count))) + (is (= 2 (parents-of charge)) + "one charge entity, referenced by both clients' orders")))) + +(deftest split-gives-each-order-its-own-charge + (testing "each order ends up with its own charge, scoped to its own client, carrying the same + amounts — so the component relationship means what it says and retracting one order + cannot delete the other's payment" + (let [{:keys [order-a order-b code-a code-b]} (two-orders-sharing-one-charge) + result (sut/split-and-rekey-charges! [order-a order-b] 100)] + (is (= {:rekeyed 1 :cloned 1} result) "first order keeps it, second gets a copy") + (is (= 2 (charge-count)) "exactly one new entity was created") + + (let [a (charges-of order-a) + b (charges-of order-b)] + (is (= 1 (count a))) + (is (= 1 (count b))) + (is (= (str "square/charge/" code-a "-CD-shared1") (:key (first a)))) + (is (= (str "square/charge/" code-b "-CC-shared1") (:key (first b)))) + (is (not= (:eid (first a)) (:eid (first b))) "two distinct entities") + (is (= 120.0 (:total (first a)) (:total (first b))) "amounts copied") + (is (= 20.0 (:tip (first a)) (:tip (first b))) "tips copied") + (is (= 1 (parents-of (:eid (first a))))) + (is (= 1 (parents-of (:eid (first b)))) + "no charge has more than one parent order any more"))))) + +(deftest split-leaves-an-unshared-charge-alone + (testing "an order that already owns its charge outright is only re-keyed, never cloned" + (let [{:strs [test-client-id]} (setup-test-data []) + tx @(dc/transact conn [{:db/id "charge" + :charge/external-id "square/charge/solo1" + :charge/total 50.0} + {:db/id "order" + :sales-order/external-id "square/order/NGCD-CD-o2" + :sales-order/client test-client-id + :sales-order/location "CD" + :sales-order/date sales-date + :sales-order/charges ["charge"]}]) + order (get-in tx [:tempids "order"]) + code (:client/code (dc/entity (dc/db conn) test-client-id))] + (is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [order] 100))) + (is (= 1 (charge-count)) "nothing was created") + (is (= (str "square/charge/" code "-CD-solo1") (:key (first (charges-of order)))))))) + +(deftest split-is-idempotent + (testing "re-running over an already-split database changes nothing further" + (let [{:keys [order-a order-b]} (two-orders-sharing-one-charge)] + (sut/split-and-rekey-charges! [order-a order-b] 100) + (let [after-first (charge-count)] + (is (= {:rekeyed 0 :cloned 0} (sut/split-and-rekey-charges! [order-a order-b] 100)) + "every charge already carries the key its order expects, so there is nothing to do") + (is (= after-first (charge-count)) "and no further entities appear"))))) + +(deftest clone-is-not-double-scoped-across-batches + (testing "with a batch size of one, the second order sees a charge already carrying the first + client's scope. It must clone using the underlying Square id, not re-scope the scoped + key — otherwise the entity ends up keyed NGCD-CD-NGCC-CC-, the importer computes + the correct key, misses, and creates a second charge that doubles the tender." + (let [{:keys [order-a order-b code-a code-b]} (two-orders-sharing-one-charge)] + (sut/split-and-rekey-charges! [order-a order-b] 1) + (let [ka (:key (first (charges-of order-a))) + kb (:key (first (charges-of order-b)))] + (is (= (str "square/charge/" code-a "-CD-shared1") ka)) + (is (= (str "square/charge/" code-b "-CC-shared1") kb)) + (is (not (clojure.string/includes? kb (str code-a "-CD"))) + "the clone carries one scope, not two") + (is (= 2 (charge-count))))))) + +(deftest two-orders-of-the-same-client-keep-sharing + (testing "one payment covering two of the SAME client's orders is left shared, on purpose. + + There is no second name to give a copy — both orders compute the same one — and a copy + would double that client's takings for the day. The component cascade still reaches + these, which is why remove-voided-orders needs its own guard." + (let [{:strs [test-client-id]} (setup-test-data []) + tx @(dc/transact conn [{:db/id "charge" + :charge/external-id "square/charge/same1" + :charge/total 75.0} + {:db/id "o1" :sales-order/external-id "square/order/x-1" + :sales-order/client test-client-id :sales-order/location "CD" + :sales-order/date sales-date :sales-order/charges ["charge"]} + {:db/id "o2" :sales-order/external-id "square/order/x-2" + :sales-order/client test-client-id :sales-order/location "CD" + :sales-order/date sales-date :sales-order/charges ["charge"]}]) + o1 (get-in tx [:tempids "o1"]) o2 (get-in tx [:tempids "o2"]) + code (:client/code (dc/entity (dc/db conn) test-client-id))] + (is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [o1 o2] 100)) + "renamed once, not copied") + (is (= 1 (charge-count)) "no copy was made, so the takings are not doubled") + (is (= (str "square/charge/" code "-CD-same1") (:key (first (charges-of o1))))) + (is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2)))) + "both orders still point at the one payment")))) diff --git a/test/clj/auto_ap/jobs/sales_summaries_test.clj b/test/clj/auto_ap/jobs/sales_summaries_test.clj new file mode 100644 index 00000000..a0bfec6c --- /dev/null +++ b/test/clj/auto_ap/jobs/sales_summaries_test.clj @@ -0,0 +1,183 @@ +(ns auto-ap.jobs.sales-summaries-test + (:require + [auto-ap.datomic :refer [conn]] + [auto-ap.datomic.sales-summaries :as d-ss] + [auto-ap.integration.util :refer [setup-test-data wrap-setup]] + [auto-ap.jobs.sales-summaries :as sut] + [clojure.test :refer [deftest is testing use-fixtures]] + [datomic.api :as dc])) + +(use-fixtures :each wrap-setup) + +(def sales-date #inst "2026-08-01T07:00:00.000-00:00") + +(defn- order + "A sales order on `sales-date`, carrying whatever the case under test needs. The external id + is Square-shaped by default because `get-service-charges` falls back to it when an order has + no `:sales-order/vendor`." + [client id attrs] + (merge {:db/id (str "order-" id) + :sales-order/external-id (str "square/order/TEST-" id) + :sales-order/client client + :sales-order/date sales-date + :sales-order/total 100.0} + attrs)) + +(defn- charge [id attrs] + (merge {:db/id (str "charge-" id) + :charge/external-id (str "square/charge/" id) + :charge/type-name "CARD" + :charge/total 100.0} + attrs)) + +(defn- tip-for [client] + (:ledger-mapped/amount (#'sut/get-tip client sales-date))) + +(defn- service-charges-for [client] + (#'sut/get-service-charges client sales-date)) + +(defn- enable-service-charges! [client] + @(dc/transact conn [{:db/id client + :client/feature-flags [sut/service-charges-flag]}])) + +(deftest tip-counts-a-reversal-on-an-untendered-order + (testing "a return-only order has no tender to join through, so its negative tip must come + from the order or the day credits a tip that was handed back" + (let [{:strs [test-client-id]} (setup-test-data [])] + @(dc/transact conn [(order test-client-id "return-only" {:sales-order/tip -12.0})]) + (is (= -12.0 (tip-for test-client-id)))))) + +(deftest tip-on-a-tendered-order-still-comes-from-the-tender + (testing "the tender carries a tip the order does not — auto-gratuity booked as a service + charge. Reading the order instead of the tender would drop it." + (let [{:strs [test-client-id]} (setup-test-data [])] + @(dc/transact conn [(order test-client-id "tendered" + {:sales-order/tip 0.0 + :sales-order/charges [(charge "tendered" {:charge/tip 50.0})]})]) + (is (= 50.0 (tip-for test-client-id)))))) + +(deftest tip-on-an-ordinary-order-is-counted-once + (testing "an order that agrees with its tender is not double counted by the additive form" + (let [{:strs [test-client-id]} (setup-test-data [])] + @(dc/transact conn [(order test-client-id "ordinary" + {:sales-order/tip 5.0 + :sales-order/charges [(charge "ordinary" {:charge/tip 5.0})]})]) + (is (= 5.0 (tip-for test-client-id)))))) + +(deftest service-charges-need-the-feature-flag + (testing "without the flag the summary behaves exactly as it does today" + (let [{:strs [test-client-id]} (setup-test-data [])] + @(dc/transact conn [(order test-client-id "square-sc" + {:sales-order/vendor :vendor/ccp-square + :sales-order/service-charge 50.0})]) + (is (nil? (service-charges-for test-client-id)))))) + +(deftest service-charges-credit-square-orders + (testing "a service charge rides along in the tender, so it needs a credit to match" + (let [{:strs [test-client-id]} (setup-test-data [])] + (enable-service-charges! test-client-id) + @(dc/transact conn [(order test-client-id "square-sc" + {:sales-order/vendor :vendor/ccp-square + :sales-order/service-charge 50.0})]) + (let [item (service-charges-for test-client-id)] + (is (= 50.0 (:ledger-mapped/amount item))) + (is (= :ledger-side/credit (:ledger-mapped/ledger-side item))) + (is (= "Service Charges" (:sales-summary-item/category item))))))) + +(deftest service-charges-count-both-signs + (testing "a returned catering fee arrives as a negative service charge and is subtracted back + out of returns, so dropping negatives loses the reversal" + (let [{:strs [test-client-id]} (setup-test-data [])] + (enable-service-charges! test-client-id) + @(dc/transact conn [(order test-client-id "refunded-fee" + {:sales-order/vendor :vendor/ccp-square + :sales-order/service-charge -140.0})]) + (is (= -140.0 (:ledger-mapped/amount (service-charges-for test-client-id))))))) + +(deftest service-charges-exclude-non-square-vendors + (testing "ezCater service charges are commission deducted from the restaurant rather than + collected from the diner, so crediting them would make the day worse" + (let [{:strs [test-client-id]} (setup-test-data [])] + (enable-service-charges! test-client-id) + @(dc/transact conn [(order test-client-id "ezcater-sc" + {:sales-order/external-id "ezcater/order/TEST-ezcater-sc" + :sales-order/vendor :vendor/ccp-ezcater + :sales-order/service-charge -75.0})]) + (is (nil? (service-charges-for test-client-id)))))) + +(deftest service-charges-recognise-square-orders-that-carry-no-vendor + (testing "whole eras of Square orders have no :sales-order/vendor at all; a gate on vendor + alone would silently credit nothing" + (let [{:strs [test-client-id]} (setup-test-data [])] + (enable-service-charges! test-client-id) + @(dc/transact conn [(order test-client-id "vendorless" {:sales-order/service-charge 12.5})]) + (is (= 12.5 (:ledger-mapped/amount (service-charges-for test-client-id))))))) + +(deftest service-charges-ignore-vendorless-orders-from-other-sources + (testing "the external id fallback is Square-specific, not a catch-all for missing vendors" + (let [{:strs [test-client-id]} (setup-test-data [])] + (enable-service-charges! test-client-id) + @(dc/transact conn [(order test-client-id "ezcater-vendorless" + {:sales-order/external-id "ezcater/order/TEST-ezcater-vendorless" + :sales-order/service-charge -75.0})]) + (is (nil? (service-charges-for test-client-id)))))) + +(defn- refund + "A card refund on `sales-date`. The client+date tuple is set explicitly because + `scan-sales-refunds` walks that index rather than the plain attributes." + [client id total] + {:db/id (str "refund-" id) + :sales-refund/external-id (str "square/refund/TEST-" id) + :sales-refund/client client + :sales-refund/date sales-date + :sales-refund/client+date [client sales-date] + :sales-refund/type "CARD" + :sales-refund/total total}) + +(defn- returns-for [client] + (#'sut/get-returns client sales-date)) + +(deftest a-refund-with-no-sales-leaves-the-day-out-of-balance + (testing "deliberate, and load-bearing. Booking a return against the day's refunds would close + it and is tempting for that reason. But a day with refunds and no sales at all is + overwhelmingly a day whose ORDERS WERE NEVER IMPORTED — on a restored copy of + production, 132 of 156 such days fell before their client's first ever synced order. + Balancing them would turn the only signal that a client's sales are missing into + silence. If this test starts failing, read the rollout plan before changing it." + (let [{:strs [test-client-id]} (setup-test-data [])] + @(dc/transact conn [(refund test-client-id "no-sales" 40.0)]) + (is (nil? (returns-for test-client-id)) + "no return is invented for a day that recorded no sales") + (is (= -40.0 (d-ss/imbalance (sut/get-refund-items test-client-id sales-date))) + "so the day stays out of balance by the refunded amount, visibly")))) + +(deftest a-day-that-traded-books-its-own-return + (testing "the ordinary case: the return comes from the day's orders, never from its refunds" + (let [{:strs [test-client-id]} (setup-test-data [])] + @(dc/transact conn [(order test-client-id "traded" {:sales-order/returns 7.0}) + (refund test-client-id "same-day" 40.0)]) + (is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id))))))) + +(deftest dirty-summaries-stop-at-the-client-boundary + (testing "every dirty day for the client is returned, and none belonging to another client. + + :sales-summary/client+dirty sorts by client, so an unbounded index scan would walk + every later client's summaries too — correct, but quadratic in the summary count." + (let [{:strs [test-client-id]} (setup-test-data []) + other (get-in @(dc/transact conn [{:db/id "other" :client/code (str "OTHER" (rand-int 100000))}]) + [:tempids "other"]) + day (fn [client d dirty?] + {:sales-summary/client client + :sales-summary/date d + :sales-summary/dirty dirty?})] + @(dc/transact conn [(day test-client-id #inst "2026-08-01T07:00:00.000-00:00" true) + (day test-client-id #inst "2026-08-02T07:00:00.000-00:00" true) + (day test-client-id #inst "2026-08-03T07:00:00.000-00:00" false) + (day other #inst "2026-08-01T07:00:00.000-00:00" true) + (day other #inst "2026-08-02T07:00:00.000-00:00" true)]) + (let [mine (sut/dirty-sales-summaries test-client-id)] + (is (= 2 (count mine)) "both dirty days, and not the clean one") + (is (every? #(= test-client-id (:db/id (:sales-summary/client %))) mine) + "and nothing belonging to the other client")) + (is (= 2 (count (sut/dirty-sales-summaries other))) + "the other client's own dirty days are still found")))) diff --git a/test/clj/auto_ap/square/core3_test.clj b/test/clj/auto_ap/square/core3_test.clj new file mode 100644 index 00000000..b99eeeef --- /dev/null +++ b/test/clj/auto_ap/square/core3_test.clj @@ -0,0 +1,118 @@ +(ns auto-ap.square.core3-test + (:require + [auto-ap.datomic :refer [conn]] + [auto-ap.integration.util :refer [setup-test-data wrap-setup]] + [auto-ap.square.core3 :as sut] + [clojure.test :refer [deftest is testing use-fixtures]] + [datomic.api :as dc])) + +(use-fixtures :each wrap-setup) + +(def client {:client/code "NGCD"}) +(def location {:square-location/client-location "CD"}) + +(defn- refund-count [] + (count (dc/q '[:find ?e :where [?e :sales-refund/external-id]] (dc/db conn)))) + +(defn- resolve-refund [id] + (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" client location id)) + +(deftest scoped-key-carries-client-and-location + (testing "the same shape sales order keys already use, so a shared location cannot contend" + (is (= "square/refund/NGCD-CD-abc" (sut/scoped-key "square/refund/" client location "abc"))) + (is (= "square/charge/NGCD-CD-xyz" (sut/scoped-key "square/charge/" client location "xyz"))))) + +(deftest legacy-keyed-entity-is-updated-not-duplicated + (testing "an entity still carrying its unscoped key is found and re-keyed in place. + + This is the sharpest hazard in the migration: these external ids are + :db.unique/identity, so writing the new key without resolving the old one first + matches nothing and creates a second entity, orphaning the original." + (setup-test-data []) + @(dc/transact conn [{:db/id "r" + :sales-refund/external-id "square/refund/abc" + :sales-refund/total 10.0}]) + (is (= 1 (refund-count))) + (let [eid (resolve-refund "abc")] + (is (some? eid) "resolves an entity carrying the legacy key") + @(dc/transact conn [{:db/id eid + :sales-refund/external-id (sut/scoped-key "square/refund/" client location "abc") + :sales-refund/total 10.0}]) + (is (= 1 (refund-count)) "no second entity was created") + (is (= eid (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/NGCD-CD-abc"])) + "the same entity now answers to the scoped key") + (is (nil? (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/abc"])) + "and no longer to the legacy one")))) + +(deftest already-scoped-entity-resolves-by-its-new-key + (testing "re-running the importer after migration finds the entity by the scoped key, so the + migration is not undone and nothing is duplicated" + (setup-test-data []) + @(dc/transact conn [{:db/id "r" + :sales-refund/external-id "square/refund/NGCD-CD-abc" + :sales-refund/total 10.0}]) + (is (= (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/NGCD-CD-abc"]) + (resolve-refund "abc"))) + (is (= 1 (refund-count))))) + +(deftest unknown-id-resolves-to-nothing + (testing "a refund never seen before has no id to pin, so the importer creates it fresh" + (setup-test-data []) + (is (nil? (resolve-refund "never-seen"))))) + +(deftest two-clients-on-one-location-get-their-own-entities + (testing "the point of the re-key: with the client in the key, a second client importing the + same Square refund creates its own entity instead of taking ownership of the first" + (setup-test-data []) + (let [other {:client/code "NGCC"} + other-loc {:square-location/client-location "CC"}] + @(dc/transact conn [{:db/id "r" + :sales-refund/external-id (sut/scoped-key "square/refund/" client location "shared") + :sales-refund/total 10.0}]) + (is (nil? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" other other-loc "shared")) + "the second client does not resolve onto the first client's entity") + @(dc/transact conn [{:db/id "r2" + :sales-refund/external-id (sut/scoped-key "square/refund/" other other-loc "shared") + :sales-refund/total 10.0}]) + (is (= 2 (refund-count)) "two stable entities, one per client, rather than one that flips")))) + +(deftest payouts-and-shifts-are-client-scoped-too + (testing "expected deposits and cash drawer shifts are fetched per location, so two clients on + one location collide on them exactly as refunds and charges did" + (is (= "square/payout/NGCD-CD-po1" + (sut/scoped-key "square/payout/" client location "po1"))) + (is (= "square/cash-drawer-shift/NGCD-CD-sh1" + (sut/scoped-key "square/cash-drawer-shift/" client location "sh1"))))) + +(deftest legacy-keyed-deposit-is-updated-not-duplicated + (testing "a payout still carrying its unscoped key is found and re-keyed in place" + (setup-test-data []) + @(dc/transact conn [{:db/id "d" + :expected-deposit/external-id "square/payout/po1" + :expected-deposit/total 100.0}]) + (let [eid (sut/existing-id (dc/db conn) :expected-deposit/external-id "square/payout/" client location "po1")] + (is (some? eid) "resolves the entity carrying the legacy key") + @(dc/transact conn [{:db/id eid + :expected-deposit/external-id (sut/scoped-key "square/payout/" client location "po1") + :expected-deposit/total 100.0}]) + (is (= 1 (count (dc/q '[:find ?e :where [?e :expected-deposit/external-id]] (dc/db conn)))) + "no second deposit was created") + (is (nil? (dc/entid (dc/db conn) [:expected-deposit/external-id "square/payout/po1"])) + "the legacy key is gone")))) + +(deftest two-clients-get-their-own-deposit + (testing "with the client in the key, a second client importing the same Square payout creates + its own entity instead of taking ownership of the first" + (setup-test-data []) + (let [other {:client/code "NGCC"} + other-loc {:square-location/client-location "CC"}] + @(dc/transact conn [{:db/id "d" + :expected-deposit/external-id (sut/scoped-key "square/payout/" client location "shared") + :expected-deposit/total 100.0}]) + (is (nil? (sut/existing-id (dc/db conn) :expected-deposit/external-id "square/payout/" other other-loc "shared")) + "the second client does not resolve onto the first client's deposit") + @(dc/transact conn [{:db/id "d2" + :expected-deposit/external-id (sut/scoped-key "square/payout/" other other-loc "shared") + :expected-deposit/total 100.0}]) + (is (= 2 (count (dc/q '[:find ?e :where [?e :expected-deposit/external-id]] (dc/db conn)))) + "two stable entities, one per client, rather than one that flips")))) diff --git a/test/clj/auto_ap/tools/compare_sales_summaries.clj b/test/clj/auto_ap/tools/compare_sales_summaries.clj new file mode 100644 index 00000000..5b4fdc5e --- /dev/null +++ b/test/clj/auto_ap/tools/compare_sales_summaries.clj @@ -0,0 +1,153 @@ +(ns auto-ap.tools.compare-sales-summaries + "Compares sales summaries between two points in the same database. + + A verification tool, not part of the running application: it lives on the test/dev classpath so + nothing in production can depend on it. Load it from a REPL when auditing a recompute. + + The question this exists to answer is narrower than \"did the totals improve\": it is *which + days changed, and were any of them already balanced*. A day that was balanced before and still + balances after can still have had its line amounts move, and that is a real change to the + books even though no red turns green. Counting only balanced/unbalanced transitions would hide + it entirely. + + DO NOT USE `compare-against` — OR ANY `d/as-of` DATABASE — TO COMPARE AMOUNTS. The obvious + reading is that Datomic keeps every past value, so a recompute can be audited against what was + there before with no snapshot. That does not hold here: `:ledger-mapped/amount`, + `:ledger-mapped/ledger-side` and `:ledger-mapped/account` are all `:db/noHistory true`, so + superseded values are discarded rather than retained. A summary that has since been recomputed + reads back through `as-of` with its categories intact and its amounts *absent* — which is + indistinguishable from a legitimate all-zero day, and quietly turns every rewritten summary + into a false \"was balanced, still balances\". + + To compare amounts, capture `summaries-in` from a live `(d/db conn)` immediately after each + run, keep the two captures outside the database, and diff those. `compare-window` is safe when + both arguments are live database values; only the historical read is unsound. Categories and + which-days-changed do survive `as-of`, since `:sales-summary-item/category` retains history." + (:require + [auto-ap.datomic :refer [conn]] + [auto-ap.datomic.sales-summaries :as d-ss] + [clj-time.coerce :as c] + [datomic.api :as dc])) + +(def item-read + [:sales-summary-item/category + :sales-summary-item/manual? + :ledger-mapped/amount + {:ledger-mapped/ledger-side [:db/ident]} + {:ledger-mapped/account [:account/numeric-code]}]) + +(defn- cents + "Amounts are doubles carrying float noise, so compare them at the cent — the unit the books are + actually kept in. Without this, 182.87000000000003 and 182.87 read as a change." + [x] + (Math/round (* 100.0 (double (or x 0.0))))) + +(defn- line + "One item reduced to what a reader would call \"the number\": category, side, amount, account." + [item] + {:category (:sales-summary-item/category item) + :side (get-in item [:ledger-mapped/ledger-side :db/ident]) + :cents (cents (:ledger-mapped/amount item)) + :account (get-in item [:ledger-mapped/account :account/numeric-code])}) + +(defn summaries-in + "`{[client-code date] {:lines … :imbalance … :balanced?}}` for every summary in `[start end)`. + + Keyed by client code and date rather than entity id so the two sides line up even if an entity + were recreated between the points being compared." + [db start end] + (->> (dc/q {:find [(list 'pull '?s [:sales-summary/date + {:sales-summary/client [:client/code]} + {:sales-summary/items item-read}])] + :in '[$ ?start ?end] + :where '[[?s :sales-summary/date ?d] + [(>= ?d ?start)] + [(< ?d ?end)]]} + db (c/to-date start) (c/to-date end)) + (map first) + (reduce (fn [acc s] + (let [items (map d-ss/<-pulled-item (:sales-summary/items s))] + (assoc acc + [(get-in s [:sales-summary/client :client/code]) (:sales-summary/date s)] + {:lines (frequencies (map line (:sales-summary/items s))) + :imbalance (d-ss/imbalance items) + :balanced? (d-ss/balanced? items)}))) + {}))) + +(defn- classify + "How one client-day differs. `:numbers-changed` is the interesting one — the lines themselves + moved, whether or not the day's balance status did." + [before after] + (cond + (nil? before) :added + (nil? after) :removed + (= (:lines before) (:lines after)) :identical + :else :numbers-changed)) + +(defn compare-window + "Compares every summary in `[start end)` between two database values. + + Returns per-day rows plus the tallies worth reporting, including the one that is easy to miss: + days that were **already balanced** and whose numbers moved anyway." + [before-db after-db start end] + (let [before (summaries-in before-db start end) + after (summaries-in after-db start end) + rows (for [k (distinct (concat (keys before) (keys after))) + :let [b (get before k) a (get after k)]] + {:client (first k) + :date (second k) + :change (classify b a) + :was-balanced? (:balanced? b) + :now-balanced? (:balanced? a) + :before-imbalance (:imbalance b) + :after-imbalance (:imbalance a) + :lines-before (:lines b) + :lines-after (:lines a)}) + rows (vec rows) + changed (filter #(= :numbers-changed (:change %)) rows)] + {:rows rows + :tally {:compared (count rows) + :identical (count (filter #(= :identical (:change %)) rows)) + :numbers-changed (count changed) + :added (count (filter #(= :added (:change %)) rows)) + :removed (count (filter #(= :removed (:change %)) rows))} + :balance-transitions + {:unbalanced->balanced (count (filter #(and (false? (:was-balanced? %)) (true? (:now-balanced? %))) rows)) + :balanced->unbalanced (count (filter #(and (true? (:was-balanced? %)) (false? (:now-balanced? %))) rows)) + :stayed-balanced (count (filter #(and (true? (:was-balanced? %)) (true? (:now-balanced? %))) rows)) + :stayed-unbalanced (count (filter #(and (false? (:was-balanced? %)) (false? (:now-balanced? %))) rows))} + :previously-balanced-and-changed + (->> changed (filter :was-balanced?) vec)})) + +(defn line-diff + "Which categories actually moved on one row, as `{category [before-cents after-cents]}`. For + reading a handful of rows by hand once the tallies point at them." + [row] + (let [by-cat (fn [lines] (reduce (fn [m [l n]] (assoc m (:category l) (* n (:cents l)))) {} lines)) + b (by-cat (:lines-before row)) + a (by-cat (:lines-after row))] + (->> (distinct (concat (keys b) (keys a))) + (keep (fn [cat] + (let [x (get b cat 0) y (get a cat 0)] + (when (not= x y) [cat [(/ x 100.0) (/ y 100.0)]])))) + (into {})))) + +(defn compare-against + "Compare the current database against its own past value at basis `t`. + + UNSOUND FOR AMOUNTS — see the namespace docstring. The amount, side and account attributes are + `:db/noHistory`, so any summary rewritten since `t` reads back with no amounts and appears + balanced. Kept only for comparing categories and identifying which days changed." + [t start end] + (let [db (dc/db conn)] + (compare-window (dc/as-of db t) db start end))) + +(comment + ;; the restore point, i.e. production's own summaries before any of this work + (def result (compare-against 209608347 + (clj-time.core/date-time 2026 7 15) + (clj-time.core/date-time 2026 8 14))) + (:tally result) + (:balance-transitions result) + (count (:previously-balanced-and-changed result)) + (map line-diff (take 3 (:previously-balanced-and-changed result)))) -- 2.49.1 From 10d0d01b8212d547cf44359b5d14b7c35df447ff Mon Sep 17 00:00:00 2001 From: Bryce Date: Sun, 16 Aug 2026 08:09:53 -0700 Subject: [PATCH 4/8] fix(square): do not resolve a record that belongs to another client MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two clients on one Square location double a day's tender in the window between deploying and finishing the migration. Reproduced end to end: 1. charge X carries the legacy key square/charge/P and belongs to client B's order 2. client A's payout import resolves X through existing-id's legacy fallback and renames it into A's scope 3. client B's next order import matches neither scheme, so it mints a second charge 4. :sales-order/charges is cardinality-many and orders transact as plain maps, so nothing retracts the first B's order ends up holding two charges for one payment — $200 of tender for a $100 payment — and running the migration on that state produces square/charge/BBB-LB-AAA-LA-P, the same double-scoped shape that already doubled tender on five clients once during this work. existing-id's legacy branch now declines any record already owned by a different client, reading the owner attribute and, for charges that predate :charge/client, the client of the referencing order. Declining is also correct on its merits: the write then lands on this client's own copy, which is what the scoped keys exist to create. The payout path also writes :charge/client/:charge/location alongside the key, so a charge's scope and its owner can no longer disagree. The guard is transitional and gets deleted with the legacy branch it protects, at rollout step 9. Rollout resequenced for the decision to leave duplicate client records active: no deactivation, no "which record survives" call, and the risk window closed by pausing the importer across deploy + migrate rather than by removing one of the two writers. Both records converge to independent stable histories once every key carries its owner. Also from review: - migrate-all! now collision-checks the charge pass like the other three attributes instead of discovering a clash mid-run over 17M rows - split-and-rekey-charges! logs progress every 200 batches; an interrupted 19M-order run left no trail - unscoped-report's docstring no longer promises a zero its :no-owner column cannot reach; plan is named as the authoritative signal - the rollout's pre-flight asked for a :collisions key plan never returns, so it silently passed on every database - the multi-parent gate sampled (take 400000 (all-order-ids db)), which streams :aevt — ascending entity id — and so read the OLDEST 2% of orders: 2019-12-31 to 2021-06-03, before any of the contention it looks for. Now every order of the last year via the client+date index, 5,159,787 on the restored copy, reading 0 - the report claimed same-client pairs get copied once batches split them. They do not, at any batch size; verified at batch-size 1 and now pinned by a test 30 tests, 72 assertions. Co-Authored-By: Claude Opus 5 --- ...26-08-15-sales-summary-reconciliation.html | 42 ++-- docs/2026-08-15-sales-summary-rollout-plan.md | 202 ++++++++++++------ .../jobs/rekey_square_external_ids.clj | 32 ++- src/clj/auto_ap/square/core3.clj | 169 +++++++-------- .../jobs/rekey_square_external_ids_test.clj | 27 +++ test/clj/auto_ap/square/core3_test.clj | 83 +++++++ 6 files changed, 375 insertions(+), 180 deletions(-) diff --git a/docs/2026-08-15-sales-summary-reconciliation.html b/docs/2026-08-15-sales-summary-reconciliation.html index 2399a1de..d2f27b2c 100644 --- a/docs/2026-08-15-sales-summary-reconciliation.html +++ b/docs/2026-08-15-sales-summary-reconciliation.html @@ -230,6 +230,17 @@ shifts — the same figures as at the restore point. Refunds went from 50,986 to 51,986, and all 1,000 of those came from the live Square import run afterwards, not from the renaming. Had the fallback lookup been missing, each of these would have doubled instead.

+

The fallback also has to refuse. Reading the old name is what stops + duplicates; reading anyone's old name is what creates them. Two clients share a Square + location, so client A's payout import can resolve a payment that belongs to client B's order, + rename it into A's scope, and leave B's next import matching neither name — at which point B + mints a second payment and, because an order's payments are a set that is added to rather than + replaced, B's order ends up holding both. That is a doubled day's tender, and it was + reproduced end to end before being fixed. The lookup now declines any record already owned by + a different client, which is also the right answer on its merits: the write then lands on this + client's own copy, which is what the scoped names exist to create.

+

This is transitional. Once no legacy names remain, the fallback and the refusal are deleted + together and the guarantee stops depending on either.

3 · Give every order its own payment record

@@ -250,7 +261,7 @@

Run over the whole database that was 9,100,314 renamed and 200,027 copied, and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across - 400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it + every order of the last year. The record count rose by about 200,027 — the number of copies it reported making, which is the check that it created what it meant to and nothing else.

One subtlety worth recording, because it bit us: the Square id has to be recovered from the record's current owner rather than by trimming a fixed prefix. Client codes contain dashes — @@ -454,10 +465,9 @@ - - + @@ -465,7 +475,8 @@
StepResult
Deactivate the duplicate client at each shared location10 locations · shared locations remaining: 0
Walk every order in the database19,040,785 orders
Give every order its own payment record9,100,314 re-keyed · 200,027 copied
Payments owned by two orders0 across 400,000 orders checked
Payments owned by two orders0 across every order of the last year — 5,159,787
Client-scope refunds, payouts and cash-drawer shiftscounts unchanged · 0 collisions
Live Square import afterwards0 orders with duplicated payment · 0 shared payments
Ownership changes after the change0 refunds · 0 payouts · 0 shifts

-

The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.

+

The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, close to the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)

+

The measurement above was taken with the duplicate client records deactivated, and that is not how this will be deployed. Leaving both records live is the intended configuration — the re-key is what separates them — but it means the deactivation that made this measurement clean will not be there. The gap that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why existing-id now refuses to resolve a record belonging to another client.

@@ -568,15 +579,20 @@

The problem this work exists to solve is gone: no payment answers to two orders, so the component relationship means what it says and deleting an order can no longer take another order's money with it.

-

One behaviour changed when the pass was run over everything, and it is worth - recording. Where Square splits one tender across two of a single client's own orders, - the earlier design left the payment shared on purpose — both orders compute the same name, so - there is no second name for a copy to take. That rule only holds for two orders processed in - the same batch. Run across nineteen million orders in batches of two thousand, such pairs - almost always fall in different batches and the second order now takes a copy. Inside the - ninety-day window this changed nothing measurable: the recompute after the database-wide pass - matched the one before it to the cent. Outside the window it has not been measured, and it - should be before this runs against production.

+

Where Square splits one tender across two of a single client's own orders, the + payment stays shared — at any batch size. Both orders compute the same name, so there + is no second name for a copy to take, and a copy would double that client's takings for the + day. The mechanism is worth stating precisely, because it is not obvious from reading: once + the first order re-keys the payment it also writes the owner attributes in the same + transaction, so a later order recovers the bare Square id from those, computes the name the + payment already carries, and the guard (not= old new-key) drops the + row before any copy decision is reached. Verified by running the migration at a batch size of + one, which forces the two orders into separate batches: no copy is made.

+

An earlier draft of this report claimed the opposite — that such pairs would be + copied once the batches split them — and flagged it as unmeasured risk to check before + production. That was wrong, and it is recorded here rather than quietly deleted because it did + real damage: an independent reviewer cited this paragraph as evidence and raised a defect that + does not exist. A test now pins the behaviour at batch size one.

The guard on remove-voided-orders is still worth having regardless. It is cheap, and it makes the safety a property of the deletion rather than of the migration having been run first.

diff --git a/docs/2026-08-15-sales-summary-rollout-plan.md b/docs/2026-08-15-sales-summary-rollout-plan.md index 9a720aea..a7a3dd71 100644 --- a/docs/2026-08-15-sales-summary-rollout-plan.md +++ b/docs/2026-08-15-sales-summary-rollout-plan.md @@ -9,8 +9,8 @@ Measured on a restored copy of production (backup point `209608347`), 210 client with zero days knocked out of balance and zero already-balanced days altered. Of the 279 days left, **171 are not balancing faults** — they are days where a client's sales were -never imported while its refunds were. Step 9 is about those, and it is the most important thing in -this document. +never imported while its refunds were. Step 10 is about those, and it is the most important +thing in this document. --- @@ -23,16 +23,54 @@ this document. | Expected migration runtime | ~13 minutes for 19M orders on a warm cache | | Nothing here touches | invoices, payments, the ledger, or any client without the flag set | -**One prerequisite that is not code.** Ten Square locations are configured against two client records -each. Someone in the business has to decide which record survives at each. The newer record usually -has no history from before the split, so keeping it loses years of that location's books. Do this -before step 3. +**Client configuration is left exactly as it is.** Ten Square locations are configured against two +client records each, and both stay active. The re-key is what resolves them: once every record +carries its owner in its key, each client's import resolves only its own records and the two +records keep independent, stable histories. No "which record survives" decision is needed, and +nothing is deactivated. + +The consequence to be aware of: each Square payment, refund, payout and shift at a shared location +becomes **two entities, one per client record** — by design. That is the stable end state, not a +duplicate to clean up. If any report or export aggregates across client records, one restaurant's +takings would be counted twice at that layer. Nothing in this work changes that either way. + +**The only window of risk is between deploying and finishing the migration**, while legacy keys +still exist for a client to resolve. Steps 2–5 exist to make that window effectively zero. --- -## Step 1 — Deploy the code +## Step 1 — Guard `remove-voided-orders` -Deploy the branch as normal. The flag is absent from every client, so: +Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so +retracting an order cascades into its payments. Until step 4 finishes there are still payments with +two parent orders, and deleting one client's voided order can take the other client's payment with +it. + +Either leave `remove-voided-orders` switched off until step 4 verifies clean, or change it to detach +a payment that has more than one parent rather than delete it. Detaching is worth doing regardless — +it makes the safety a property of the deletion rather than of the migration having been run first. + +See `docs/2026-08-15-remove-voided-orders-risk.md`. + +--- + +## Step 2 — Pause the Square importer + +**This is what makes the deploy safe, and it is easy to skip.** Steps 2 through 5 should be one +maintenance action, not separate days' work. + +While legacy keys exist, `square.core3/existing-id` falls back to them — and at a shared location +that is the one code path that can reach across client records. Running the migration with imports +paused means no client is resolving keys while the keys are being rewritten, so the window closes +entirely rather than merely narrowing. + +The migration itself takes about **13 minutes** for all 19M orders, so the pause is short. + +--- + +## Step 3 — Deploy the code + +Deploy the branch. The flag is absent from every client, so: - tips are calculated exactly as they are today, - no `Service Charges` line is written. @@ -41,60 +79,33 @@ The only changes that take effect immediately are the safe ones: imbalance loggi dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix, and the importer's new client-scoped keys. -**The importer starts writing client-scoped keys straight away, and reads both schemes.** That is -deliberate and is what makes the deploy independent of the migration. Do not remove the legacy -lookup in `square.core3/existing-id` yet — see step 8. - -**Verify before moving on.** After one nightly import cycle: - -```clojure -;; refunds, payouts and shifts must not have doubled -(count (d/datoms (d/db conn) :aevt :sales-refund/external-id)) -(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id)) -(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id)) -``` - -Compare against the same counts taken immediately before deploy. Growth should be ordinary daily -volume. A near-doubling means the legacy fallback is not working — **stop and roll back the deploy**. - ---- - -## Step 2 — Guard `remove-voided-orders` - -Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so -retracting an order cascades into its payments. Until step 3 finishes there are still payments with -two parent orders, and deleting one client's voided order can take the other client's payment with -it. - -Either leave `remove-voided-orders` switched off until step 3 completes, or change it to detach a -payment that has more than one parent rather than delete it. Detaching is worth doing regardless — -it makes the safety a property of the deletion rather than of the migration having been run first. - -See `docs/2026-08-15-remove-voided-orders-risk.md`. - ---- - -## Step 3 — Retire the duplicate client records - -Business decision from the top of this document. Deactivate the losing record's Square location so -the importer stops fetching for it. The record itself stays; its history is untouched. - -**Verify:** no Square location is configured against two active client records. +**The importer reads both key schemes**, so the deploy does not depend on the migration having +finished. Two protections cover the interval before it does: imports are paused (step 2), and +`existing-id` refuses to resolve a record that already belongs to a different client. Do not remove +the legacy lookup yet — see step 9. --- ## Step 4 — Run the migration +Run it immediately after the deploy, while imports are still paused. + ```clojure (require '[auto-ap.jobs.rekey-square-external-ids :as rk]) -;; read-only first — check :collisions is empty for every attribute -(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys) +;; read-only first — no two entities may want the same key. `plan` does NOT return a +;; :collisions key; you have to hand its :new-keys to `collisions` yourself. +(rk/collisions (:new-keys (rk/plan (d/db conn) :charge/external-id rk/charge-prefix))) +;; => [] (anything else: stop, do not migrate) ;; then the whole thing (rk/migrate-all! 2000) ``` +`migrate-all!` runs this same check itself, on every attribute including charges, and throws +rather than transacting if it finds one. Running it by hand first just means finding out before +the 13-minute walk rather than partway through it. + Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that already carries the right name is skipped, so it can be stopped and re-run without consequence. @@ -102,25 +113,69 @@ If it appears to crawl, the cause is almost certainly garbage collection in the not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free retained memory in the REPL and re-measure before changing anything about the database. -**Verify — all four must read zero to migrate and zero unscopable:** +**Verify.** Two checks, doing two different jobs — run both. + +**(a) Completeness, across everything.** `plan` must report nothing left to do, for all four +attributes: ```clojure -(rk/unscoped-report (d/db conn)) (dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys) ;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0} +``` -;; and the gate that this work exists for -(rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn)))) +Read `:to-migrate 0` **and** `:unscopable 0`. This is the authoritative signal, and it covers all +17M charges. + +`unscoped-report` is useful colour but is not the gate: its `:no-owner` column never reaches zero +for charges, because ~283k payout stubs carry no `:charge/client` of their own and it classifies +by attribute rather than by resolving ownership. Judge completeness by `plan`. + +**(b) The safety gate for the cascade** — no payment may answer to two orders, or re-enabling +`remove-voided-orders` in step 8 can delete a payment another order still needs. Check **every** +order in the last year, with no sampling: + +```clojure +(let [db (d/db conn) + cs (map first (d/q '[:find ?c :where [?c :client/code _]] db)) + year (java.util.Date. (- (.getTime (java.util.Date.)) (long (* 365 86400000))))] + (rk/charges-with-multiple-parents + db (map first (iol-ion.query/scan-sales-orders db cs year nil)))) ;; => 0 ``` -Note `unscoped-report`'s `:no-owner` column is not a gap: ~283k payout-stub payments carry no -`:charge/client` attribute of their own, so it cannot verify them by attribute. `plan` resolves -ownership through whatever refers to them and is the figure to trust. +On the restored copy that is 5,159,787 orders — 27% of the table — via the +`:sales-order/client+date` index. A year is chosen deliberately: `remove-voided-orders` only ever +deletes orders Square reports as voided, which are recent, so that is where the destructive risk +lives. Completeness across all of history is check (a)'s job, not this one. + +> Do **not** sample this with `(take n (rk/all-order-ids db))`. `all-order-ids` streams `:aevt`, +> which is ascending entity id, so a `take` returns the *oldest* orders — on the restored copy the +> first 400,000 are all from 2019–2021, before any of the contention this gate looks for. It would +> report a confident zero having inspected none of the relevant data. --- -## Step 5 — Recompute summaries, flags still off +## Step 5 — Resume the Square importer + +Only once step 4's two checks read clean. The maintenance window ends here. + +The first cycle after resuming is the one to watch. Compare these against the same counts taken +immediately before the deploy — growth should be ordinary daily volume: + +```clojure +(count (d/datoms (d/db conn) :aevt :sales-refund/external-id)) +(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id)) +(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id)) +(count (d/datoms (d/db conn) :aevt :charge/external-id)) +``` + +A near-doubling of any of them means records are being created rather than matched — **stop and +roll back the deploy.** Charges are included deliberately: they are the one that doubles a client's +takings rather than merely duplicating a row. + +--- + +## Step 6 — Recompute summaries, flags still off ```clojure (require '[auto-ap.jobs.sales-summaries :as ss]) @@ -128,7 +183,7 @@ ownership through whatever refers to them and is the figure to trust. ``` This is the pass that banks the deduplication. **Capture the result before going further** — you -will need it as the baseline for step 6, and it cannot be reconstructed afterwards: +will need it as the baseline for step 7, and it cannot be reconstructed afterwards: ```clojure (require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath @@ -143,7 +198,7 @@ will need it as the baseline for step 6, and it cannot be reconstructed afterwar --- -## Step 6 — Turn the flag on, a few restaurants at a time +## Step 7 — Turn the flag on, a few restaurants at a time Needs accounting sign-off first: `summary-service-charges` posts to **49000 Service Income**, chosen so the work could be measured. It affects reporting, never whether a day balances. @@ -156,7 +211,7 @@ so the work could be measured. It affects reporting, never whether a day balance Start with two or three restaurants, confirm, then widen. -**Verify** against the capture from step 5: +**Verify** against the capture from step 6: ```clojure (def after (cmp/summaries-in (d/db conn) start end)) @@ -168,29 +223,34 @@ The two numbers that matter — both were zero across all 18,900 client-days in - `:balanced->unbalanced` must be **0** - previously-balanced days whose lines changed must be **0** -If either is non-zero, retract the flag for the affected clients and re-run step 5. The flag is the +If either is non-zero, retract the flag for the affected clients and re-run step 6. The flag is the rollback: removing it restores today's behaviour exactly. --- -## Step 7 — Re-enable `remove-voided-orders` +## Step 8 — Re-enable `remove-voided-orders` -Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2. +Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 1. --- -## Step 8 — Remove the legacy key lookup +## Step 9 — Remove the legacy key lookup -Only once `plan` reports `:to-migrate 0` and has stayed there through several import cycles. Drop -the second branch of `square.core3/existing-id`. At that point two clients sharing a location -becomes structurally incapable of producing a shared record, rather than prevented by a convention a -future import could break. +**Schedule this; do not leave it open-ended.** Both client records at a shared location stay active +permanently, so the legacy fallback in `square.core3/existing-id` is the one code path that can ever +reach across them. Deleting it is what turns the guarantee from conventional into structural. -This is the last step and there is no hurry. +Once `plan` reports `:to-migrate 0` and has stayed there through several import cycles, drop the +legacy branch of `existing-id` — and with it `owned-by-other-client?`, which exists only to make +that branch safe while it lives. After this, two clients on one location are structurally incapable +of resolving onto each other's records, and no ordering discipline is required to keep it that way. + +Until it is done, the protection is the guard plus the maintenance window, both of which depend on +people doing the right thing. That is the reason not to let this drift. --- -## Step 9 — Deal with the refunds that have no sales behind them +## Step 10 — Deal with the refunds that have no sales behind them **The most important item in this document, and the only one that is not just execution.** @@ -256,7 +316,7 @@ tender against $6,059.57 of order totals on one day), the ezCater fee question, clusters on NGMV and NGEB. Plus 15 days / $974.99 where a processing fee lands on a day with no trading — the same shape as -step 9 but from the payout side, so it needs the payout modelled rather than a rule in the summary. +step 10 but from the payout side, so it needs the payout modelled rather than a rule in the summary. --- diff --git a/src/clj/auto_ap/jobs/rekey_square_external_ids.clj b/src/clj/auto_ap/jobs/rekey_square_external_ids.clj index 693fc55e..43aba8a0 100644 --- a/src/clj/auto_ap/jobs/rekey_square_external_ids.clj +++ b/src/clj/auto_ap/jobs/rekey_square_external_ids.clj @@ -195,7 +195,11 @@ [orders batch-size] (let [cloned (atom 0) rekeyed (atom 0)] - (doseq [batch (partition-all batch-size orders)] + (doseq [[i batch] (map-indexed vector (partition-all batch-size orders))] + (when (zero? (mod i 200)) + ;; the whole-database run walks 19M orders; without a trail an interrupted run leaves + ;; no way to tell how far it got short of querying the data by hand + (alog/info ::splitting :orders-done (* i batch-size) :rekeyed @rekeyed :cloned @cloned)) (let [db (dc/db conn) batch-seen (atom {}) tx (doall @@ -255,12 +259,19 @@ "Counts, per entity type, how many keys are already client-scoped, how many still carry the legacy unscoped form, and how many have no owner to scope by. - This is the completeness gate. The importer tolerates both key schemes on purpose, so that the - change can be deployed before the migration finishes — but that tolerance is a transition, not - a resting place. While `:legacy` is above zero the database is in a mixed state and a stray - unscoped record can still be adopted by whichever client imports it first. Once every count - reads zero the fallback lookup in `square.core3/existing-id` can be removed and the guarantee - becomes structural rather than conventional." + The importer tolerates both key schemes on purpose, so that the change can be deployed before + the migration finishes — but that tolerance is a transition, not a resting place. While + `:legacy` is above zero the database is in a mixed state and a stray unscoped record can still + be adopted by whichever client imports it first. **`:legacy` reaching zero on every attribute is + the done-signal**, and it is what licenses removing the fallback lookup in + `square.core3/existing-id`. + + `:no-owner` is NOT part of that signal and never reaches zero for charges. It counts entities + whose own `:charge/client`/`:charge/location` are absent — around an eighth of charges, the + payout stubs `migrate!` only ever gives an external id — so this report structurally cannot + verify them even when their keys are perfectly scoped. It classifies by attribute; `plan` + resolves ownership through whatever references the entity. Ask `plan` for the authoritative + answer: `:to-migrate 0` with `:unscopable 0` means there is nothing left to do." [db] (into {} (for [{:keys [attr prefix client location]} scoped-attrs] @@ -299,8 +310,13 @@ (when-let [c (seq (collisions (:new-keys p)))] (throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)}))) (migrate! attr (:new-keys p) batch-size))) - ;; charges no order refers to — payout stubs — are scoped from the deposit that holds them + ;; charges no order refers to — payout stubs — are scoped from the deposit that holds them. + ;; Collision-checked like the others: this is the largest attribute in the database, so it is + ;; the last one that should discover a clash as a mid-run exception. (let [p (plan (dc/db conn) :charge/external-id charge-prefix)] + (when-let [c (seq (collisions (:new-keys p)))] + (throw (ex-info "two entities would take the same key" + {:attr :charge/external-id :collisions (count c)}))) (when (seq (:new-keys p)) (migrate! :charge/external-id (:new-keys p) batch-size))) {:split split :completeness (unscoped-report (dc/db conn))})) diff --git a/src/clj/auto_ap/square/core3.clj b/src/clj/auto_ap/square/core3.clj index cfa7a7c0..97a2a329 100644 --- a/src/clj/auto_ap/square/core3.clj +++ b/src/clj/auto_ap/square/core3.clj @@ -27,11 +27,9 @@ "Authorization" (str "Bearer " (:client/square-auth-token client)) "Content-Type" "application/json"})) - (defn ->square-date [d] (f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d)) - (def manifold-api-stream (let [stream (s/stream 100)] (->> stream @@ -42,10 +40,10 @@ (de/loop [attempt 0] (-> (de/chain (de/future-with (ex/execute-pool) #_(log/info ::request-started - :url (:url request) - :attempt attempt - :source "Square 3" - :background-job "Square 3") + :url (:url request) + :attempt attempt + :source "Square 3" + :background-job "Square 3") (try (client/request (assoc request :socket-timeout 10000 @@ -104,7 +102,6 @@ :exception error)) [])))) - (def item-cache (atom {})) (defn fetch-catalog [client i v] @@ -124,13 +121,11 @@ #(do (swap! item-cache assoc i %) %)))) - (defn fetch-catalog-cache [client i version] (if (get @item-cache i) (de/success-deferred (get @item-cache i)) (fetch-catalog client i version))) - (defn item->category-name-impl [client item version] (capture-context->lc (cond (:item_id (:item_variation_data item)) @@ -161,7 +156,6 @@ :item item) "Uncategorized")))) - (defn item-id->category-name [client i version] (capture-context->lc (-> [client i] @@ -226,7 +220,6 @@ (concat (:orders result) continued-results)))) (:orders result))))))) - (defn search ([client location start end] (capture-context->lc @@ -250,11 +243,9 @@ (concat (:orders result) continued-results)))) (:orders result)))))))) - (defn amount->money [amt] (* 0.01 (or (:amount amt) 0.0))) - ;; to get totals: (comment (reduce @@ -278,6 +269,30 @@ [prefix client location id] (str prefix (:client/code client) "-" (:square-location/client-location location) "-" id)) +(def ^:private owner-attr + "Where each Square-imported entity records the client it belongs to." + {:charge/external-id :charge/client + :sales-refund/external-id :sales-refund/client + :expected-deposit/external-id :expected-deposit/client + :cash-drawer-shift/external-id :cash-drawer-shift/client}) + +(defn- owned-by-other-client? + "Whether `e` already belongs to a client other than `client-eid`. + + Reads the entity's own owner attribute, and for a charge falls back to the client of whichever + sales order refers to it — charges predating `:charge/client` still have orders, and those are + exactly the ones that can be taken by the wrong client." + [db attr e client-eid] + (let [ent (dc/entity db e) + owner (or (:db/id ((owner-attr attr) ent)) + (when (= attr :charge/external-id) + (some->> (first (dc/datoms db :vaet e :sales-order/charges)) + :e + (dc/entity db) + :sales-order/client + :db/id)))] + (and owner (not= owner client-eid)))) + (defn existing-id "Entity id of the refund or charge this id already refers to, trying the client-scoped key first and the legacy unscoped key second. @@ -286,11 +301,24 @@ relies on upsert-by-identity; changing the key format on its own would match nothing and Datomic would create a SECOND entity for every refund and charge, orphaning the original under its legacy key. Pinning the result as `:db/id` makes the write land on the existing entity - whichever scheme it currently carries." + whichever scheme it currently carries. + + The legacy branch will not take a record that already belongs to a different client. Without + that check, two clients on one Square location double money during the window between deploying + and finishing the migration: client A's payout import resolves B's charge by its bare key and + renames it into A's scope, B's next order import then matches neither scheme and mints a second + charge, and because `:sales-order/charges` is cardinality-many nothing retracts the first — so + B's order carries two charges for one payment. Declining is also the right answer on its merits: + the write then lands on this client's own copy, which is what the scoped keys exist to create. + + Once the migration has run there are no legacy keys left for this branch to find, and both it + and the guard can be deleted together." [db attr prefix client location id] (when id (or (dc/entid db [attr (scoped-key prefix client location id)]) - (dc/entid db [attr (str prefix id)])))) + (when-let [legacy (dc/entid db [attr (str prefix id)])] + (when-not (owned-by-other-client? db attr legacy (:db/id client)) + legacy))))) (defn tender->charge [order client location t] (remove-nils @@ -304,7 +332,7 @@ :db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location (:id t)) :external-id (when (:id t) (scoped-key "square/charge/" client location (:id t))) - :processor (cond + :processor (cond (#{"OTHER" "THIRD_PARTY_CARD"} (:type t)) (condp = (some-> (:note t) str/lower-case) "doordash" :ccp-processor/doordash @@ -377,7 +405,7 @@ #:sales-order {:date (if (= "Invoices" (:name (:source order))) (when (:closed_at order) - (coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles")))) + (coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles")))) (coerce/to-date (time/to-time-zone (coerce/to-date-time (:created_at order)) (time/time-zone-for-id "America/Los_Angeles")))) :client (:db/id client) :location (:square-location/client-location location) @@ -439,7 +467,6 @@ :client client :location location))))))) - (defn get-payment [client p] (de/chain (manifold-api-call {:url (str "https://connect.squareup.com/v2/payments/" p) @@ -448,7 +475,6 @@ :body :payment)) - (defn continue-payout-entry-list [c l poi cursor] (capture-context->lc lc (de/chain @@ -593,6 +619,12 @@ (let [payment-id (:payment_id (:type_charge_details p))] (remove-nils {:charge/external-id (scoped-key "square/charge/" client location payment-id) + ;; the owner attributes must travel with the key: `raw-square-id` and + ;; `scope-of` both recover a charge's scope from them, and a key + ;; scoped to one client while the owner says another is what makes + ;; the migration write square/charge/B-LB-A-LA- + :charge/client (:db/id client) + :charge/location (:square-location/client-location location) :db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location payment-id)}))))))}) (filter :expected-deposit/date) (into [])) @@ -651,7 +683,6 @@ :count (count x)) @(dc/transact-async conn x)))))))) - (defn upsert-payouts ([client] (apply de/zip @@ -700,7 +731,6 @@ (log/info ::done-loading-refunds))))))) - (defn get-cash-shift [client id] (de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id)) :method :get @@ -864,8 +894,6 @@ d1 d2)) - - (defn remove-voided-orders ([client] (apply de/zip @@ -892,7 +920,7 @@ (:sales-order/external-id o)))))) (s/map (fn [[o]] [[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]])) - + (s/reduce into []))) (fn [results] @@ -901,31 +929,26 @@ (log/info ::removing-orders :count (count x)) @(dc/transact-async conn x))))) - (de/catch (fn [e] - (log/warn ::couldnt-remove :error e) - nil) )))))) + (de/catch (fn [e] + (log/warn ::couldnt-remove :error e) + nil))))))) -#_(comment - (require 'auto-ap.time-reader) +#_(comment + (require 'auto-ap.time-reader) - @(let [[c [l]] (get-square-client-and-location "DBFS") ] - (log/peek :x [ c l]) - (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29") + @(let [[c [l]] (get-square-client-and-location "DBFS")] + (log/peek :x [c l]) + (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29")) - ) + @(let [[c [l]] (get-square-client-and-location "NGAK")] + (log/peek :x [c l]) - @(let [[c [l]] (get-square-client-and-location "NGAK") ] - (log/peek :x [ c l]) - - (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15")) - (doseq [c (get-square-clients)] - (try - @(remove-voided-orders c) - (catch Exception e - nil))) - - - ) + (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15")) + (doseq [c (get-square-clients)] + (try + @(remove-voided-orders c) + (catch Exception e + nil)))) (defn upsert-all [& clients] (capture-context->lc @@ -994,8 +1017,6 @@ [:clients clients] @(apply upsert-all clients))) - - (comment (defn refunds-raw-cont ([client l cursor so-far] @@ -1025,9 +1046,8 @@ (->> @(let [[c [l]] (get-square-client-and-location "NGGG")] - (search c l (time/now) (time/plus (time/now) (time/days -1)))) - + (filter (fn [r] (str/starts-with? (:created_at r) "2024-03-14")))) @@ -1035,7 +1055,6 @@ (->> @(let [[c [l]] (get-square-client-and-location "NGGG")] - (refunds-raw-cont c l nil [])) (filter (fn [r] (str/starts-with? (:created_at r) "2024-03-14"))))) @@ -1069,13 +1088,8 @@ []))] [(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row) (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)]) :separator \tab) - - - - - ;; => - +;; => (require 'auto-ap.time-reader) @@ -1084,27 +1098,16 @@ (clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")] l - (def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00" #clj-time/date-time "2025-02-28T00:00:00-08:00")) - (take 10 (map #(first (deref (order->sales-order c l %))) z))) + (take 10 (map #(first (deref (order->sales-order c l %))) z)))) - - ) - - - - - (->> z + (->> z (filter (fn [o] (seq (filter (comp #{"OTHER"} :type) (:tenders o))))) (filter #(not (:name (:source %)))) - (count) - - ) - - - + (count)) + (doseq [[code] (seq (dc/q '[:find ?code :in $ :where [?o :sales-order/date ?d] @@ -1113,32 +1116,22 @@ [?o :sales-order/client ?c] [?c :client/code ?code]] (dc/db conn))) - :let [[c [l]] (get-square-client-and-location code) - ] + :let [[c [l]] (get-square-client-and-location code)] order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now)) - :when (= "Invoices" (:name (:source order) )) + :when (= "Invoices" (:name (:source order))) :let [[sales-order] @(order->sales-order c l order)]] - + (when (should-import-order? order) (println "DATE IS" (:sales-order/date sales-order)) (when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after? #clj-time/date-time "2026-2-16T00:00:00-08:00")) (println "WOULD UPDATE" sales-order) - @(dc/transact auto-ap.datomic/conn [sales-order]) - ) - #_@(dc/transact ) - (println "DONE")) - - - ) + @(dc/transact auto-ap.datomic/conn [sales-order])) + #_@(dc/transact) + (println "DONE"))) #_(filter (comp #{"OTHER"} :type) (mapcat :tenders z)) - @(let [[c [l]] (get-square-client-and-location "NGRY")] #_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01")) - (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY" ))) - - ) - - ) + (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY"))))) diff --git a/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj b/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj index a5825462..cc0f646e 100644 --- a/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj +++ b/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj @@ -147,3 +147,30 @@ (is (= (str "square/charge/" code "-CD-same1") (:key (first (charges-of o1))))) (is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2)))) "both orders still point at the one payment")))) + +(deftest two-orders-of-the-same-client-keep-sharing-across-batches + (testing "batch size does not change the same-client rule, which the sibling test cannot show + because both its orders land in one batch. + + Once the first order re-keys the charge it also writes :charge/client/:charge/location, + so the second order's raw-square-id takes its owner branch, new-key reconstructs the key + the charge already has, and the (not= old new-key) guard drops the row before :action is + read. A clone here would double that client's takings for the day." + (let [{:strs [test-client-id]} (setup-test-data []) + tx @(dc/transact conn [{:db/id "charge" + :charge/external-id "square/charge/same2" + :charge/total 75.0} + {:db/id "o1" :sales-order/external-id "square/order/y-1" + :sales-order/client test-client-id :sales-order/location "CD" + :sales-order/date sales-date :sales-order/charges ["charge"]} + {:db/id "o2" :sales-order/external-id "square/order/y-2" + :sales-order/client test-client-id :sales-order/location "CD" + :sales-order/date sales-date :sales-order/charges ["charge"]}]) + o1 (get-in tx [:tempids "o1"]) o2 (get-in tx [:tempids "o2"]) + code (:client/code (dc/entity (dc/db conn) test-client-id))] + (is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [o1 o2] 1)) + "batch size 1 puts the two orders in separate batches, and still no copy is made") + (is (= 1 (charge-count)) "one payment, not two") + (is (= (str "square/charge/" code "-CD-same2") (:key (first (charges-of o1))))) + (is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2)))) + "both orders still point at the one payment")))) diff --git a/test/clj/auto_ap/square/core3_test.clj b/test/clj/auto_ap/square/core3_test.clj index b99eeeef..6d772fd7 100644 --- a/test/clj/auto_ap/square/core3_test.clj +++ b/test/clj/auto_ap/square/core3_test.clj @@ -76,6 +76,89 @@ :sales-refund/total 10.0}]) (is (= 2 (refund-count)) "two stable entities, one per client, rather than one that flips")))) +(deftest legacy-key-of-another-client-is-not-claimed + (testing "the legacy fallback must not hand one client a record that already belongs to another. + + Without this, two clients on one Square location double money in the window between + deploying and finishing the migration." + (setup-test-data []) + (let [tx @(dc/transact conn [{:db/id "mine" :client/code (str "MINE" (rand-int 100000))} + {:db/id "theirs" :client/code (str "THEIRS" (rand-int 100000))}]) + mine {:db/id (get-in tx [:tempids "mine"]) :client/code "MINE"} + theirs-id (get-in tx [:tempids "theirs"])] + @(dc/transact conn [{:db/id "r" + :sales-refund/external-id "square/refund/abc" + :sales-refund/client theirs-id + :sales-refund/total 10.0}]) + (is (nil? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" + mine location "abc")) + "a legacy-keyed refund owned by another client is left alone") + (is (some? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" + {:db/id theirs-id :client/code "THEIRS"} location "abc")) + "its own client still resolves it, so re-keying in place still works")))) + +(deftest a-charge-is-owned-by-the-client-of-the-order-that-refers-to-it + (testing "charges predating :charge/client still have orders, and those are exactly the ones + that could be taken by the wrong client" + (let [{:strs [test-client-id]} (setup-test-data []) + other (get-in @(dc/transact conn [{:db/id "o" :client/code (str "OTHER" (rand-int 100000))}]) + [:tempids "o"])] + @(dc/transact conn [{:db/id "c" :charge/external-id "square/charge/p1" :charge/total 50.0} + {:db/id "ord" :sales-order/external-id "square/order/x-1" + :sales-order/client test-client-id :sales-order/location "CD" + :sales-order/date #inst "2026-06-03T07:00:00.000-00:00" + :sales-order/charges ["c"]}]) + (is (nil? (sut/existing-id (dc/db conn) :charge/external-id "square/charge/" + {:db/id other :client/code "OTHER"} location "p1")) + "ownership is read from the referencing order when :charge/client is absent")))) + +(deftest deploy-window-does-not-double-a-second-clients-tender + (testing "the P0 this guard exists for, end to end. + + Client A's payout import reaches for a payment whose charge belongs to client B's + order. If A were allowed to re-key it, B's next order import would match neither + scheme, mint a second charge, and — since :sales-order/charges is cardinality-many — + leave B's order holding two charges for one payment." + (let [{:strs [test-client-id]} (setup-test-data []) + b-code (:client/code (dc/entity (dc/db conn) test-client-id)) + b {:db/id test-client-id :client/code b-code} + b-loc {:square-location/client-location "LB"} + a-id (get-in @(dc/transact conn [{:db/id "a" :client/code (str "AAA" (rand-int 100000))}]) + [:tempids "a"]) + a {:db/id a-id :client/code (:client/code (dc/entity (dc/db conn) a-id))} + a-loc {:square-location/client-location "LA"} + tx @(dc/transact conn [{:db/id "x" :charge/external-id "square/charge/P" + :charge/total 100.0 :charge/type-name "CARD" + :charge/client test-client-id :charge/location "LB"} + {:db/id "ob" :sales-order/external-id "square/order/b-1" + :sales-order/client test-client-id :sales-order/location "LB" + :sales-order/date #inst "2026-06-03T07:00:00.000-00:00" + :sales-order/charges ["x"]}]) + order-b (get-in tx [:tempids "ob"]) + charges-of (fn [o] (map :v (dc/datoms (dc/db conn) :eavt o :sales-order/charges)))] + ;; client A's payout import touches the same Square payment + @(dc/transact conn [(into {} (remove (comp nil? val)) + {:charge/external-id (sut/scoped-key "square/charge/" a a-loc "P") + :charge/client a-id + :charge/location "LA" + :db/id (sut/existing-id (dc/db conn) :charge/external-id + "square/charge/" a a-loc "P")})]) + ;; client B's order re-imports + @(dc/transact conn [{:db/id order-b + :sales-order/charges + [(sut/tender->charge {:id "b-1" :created_at "2026-06-03T12:00:00Z"} + b b-loc {:id "P" :type "CARD" + :amount_money {:amount 10000 + :currency "USD"}})]}]) + (is (= 1 (count (charges-of order-b))) + "B's order still holds exactly one charge for the one payment") + (is (= 100.0 (reduce + 0.0 (map #(:charge/total (dc/entity (dc/db conn) %)) + (charges-of order-b)))) + "so the day's tender is not doubled") + (is (= (str "square/charge/" b-code "-LB-P") + (:charge/external-id (dc/entity (dc/db conn) (first (charges-of order-b))))) + "and B's own charge was re-keyed in place rather than abandoned")))) + (deftest payouts-and-shifts-are-client-scoped-too (testing "expected deposits and cash drawer shifts are fetched per location, so two clients on one location collide on them exactly as refunds and charges did" -- 2.49.1 From 967e77e4437eb7c565b2ca6e69f068f505d7cd36 Mon Sep 17 00:00:00 2001 From: Bryce Date: Sun, 16 Aug 2026 08:35:02 -0700 Subject: [PATCH 5/8] feat(square): walk the migration newest month first, so stopping early is safe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit migrate-all! drove the charge split from all-order-ids, which streams :aevt — ascending entity id, so oldest first. On the production copy that means the first several hours are spent on 2019 and 2020 data no import will ever read, leaving the recent end (the part the importer actually touches) for last. Interrupt it there and the data is unmigrated exactly where it matters. Now: - refunds, payouts and cash-drawer shifts run first. Together they are ~266k records and take seconds, so an interruption cannot leave them half done. - the long order walk then runs a month at a time from the current month backwards, logging ::month-complete with per-month counts. Stop it after any month and everything from that month forward is fully scoped, so imports can resume against a partially migrated database and the older tail can be finished later — the re-run skips what is done. While a tail remains unmigrated, existing-id's ownership guard is what keeps it safe. order-months-newest-first tiles [start end] windows with no gaps (each month ends the day before the next begins) and is bounded below by a constant comfortably older than the oldest order. Windows are walked via the :sales-order/client+date index. Verified against the restored copy: 141 windows from 2026-08 back, August returning 241,126 orders and July 476,235, each dated inside its window. all-order-ids keeps its old behaviour but its docstring now warns that a prefix of it is the oldest orders, not a sample — the trap that made an earlier verification gate read 2019-2021 data. 31 tests, 76 assertions. Co-Authored-By: Claude Opus 5 --- docs/2026-08-15-sales-summary-rollout-plan.md | 28 ++++++- .../jobs/rekey_square_external_ids.clj | 77 ++++++++++++++++--- .../jobs/rekey_square_external_ids_test.clj | 17 ++++ 3 files changed, 112 insertions(+), 10 deletions(-) diff --git a/docs/2026-08-15-sales-summary-rollout-plan.md b/docs/2026-08-15-sales-summary-rollout-plan.md index a7a3dd71..62421cc2 100644 --- a/docs/2026-08-15-sales-summary-rollout-plan.md +++ b/docs/2026-08-15-sales-summary-rollout-plan.md @@ -109,6 +109,26 @@ the 13-minute walk rather than partway through it. Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that already carries the right name is skipped, so it can be stopped and re-run without consequence. +**It is also ordered so that stopping early is survivable.** Refunds, payouts and cash-drawer +shifts go first — a quarter of a million records, seconds of work — so an interruption cannot catch +them half done. The long part then walks orders **a month at a time, from the current month +backwards**, logging `::month-complete` as each finishes: + +``` +::month-complete :month "2026-08" :rekeyed 118203 :cloned 2244 +::month-complete :month "2026-07" :rekeyed 241887 :cloned 4611 +... +``` + +That ordering is the recovery plan. If it dies, everything from the last logged month forward is +fully scoped — and that recent window is what the importer actually reads — so **you can resume +imports against a partially migrated database** and finish the older tail later. Walking oldest +first would have spent the first several hours on 2019 data no import will touch, leaving exactly +the wrong end done. + +If you do resume imports mid-migration, the ownership guard in `existing-id` is what keeps the +unmigrated tail safe: a client cannot resolve onto another client's legacy-keyed record. + If it appears to crawl, the cause is almost certainly garbage collection in the process driving it, not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free retained memory in the REPL and re-measure before changing anything about the database. @@ -157,7 +177,13 @@ lives. Completeness across all of history is check (a)'s job, not this one. ## Step 5 — Resume the Square importer -Only once step 4's two checks read clean. The maintenance window ends here. +Normally: once step 4's two checks read clean. The maintenance window ends here. + +**If the migration did not finish**, you do not have to wait for it. Resume imports once the +`::month-complete` log covers the window your importer reads — the last 75 days for payouts and +cash-drawer shifts, and whatever range the order import is configured for. Then re-run +`migrate-all!` afterwards to walk the remaining older months; it will skip everything already done. +Run the step 4 checks again once it does finish. The first cycle after resuming is the one to watch. Compare these against the same counts taken immediately before the deploy — growth should be ordinary daily volume: diff --git a/src/clj/auto_ap/jobs/rekey_square_external_ids.clj b/src/clj/auto_ap/jobs/rekey_square_external_ids.clj index 43aba8a0..0c324e60 100644 --- a/src/clj/auto_ap/jobs/rekey_square_external_ids.clj +++ b/src/clj/auto_ap/jobs/rekey_square_external_ids.clj @@ -20,7 +20,8 @@ (:require [auto-ap.datomic :refer [conn]] [auto-ap.logging :as alog] - [datomic.api :as dc])) + [datomic.api :as dc] + [iol-ion.query])) (def refund-prefix "square/refund/") (def charge-prefix "square/charge/") @@ -288,10 +289,51 @@ (dc/datoms db :aevt attr))]))) (defn all-order-ids - "Every sales order in the database, streamed." + "Every sales order in the database, streamed in `:aevt` order — which is ascending entity id, + so OLDEST first. Fine for counting; wrong for anything that takes a prefix. `(take n ...)` of + this returns the oldest n orders, not a sample: on the production copy the first 400,000 are + all from 2019 to 2021. Use `order-months-newest-first` to walk the data in migration order." [db] (map :e (dc/datoms db :aevt :sales-order/external-id))) +(def ^:private earliest-orders + "How far back the month walk goes. Comfortably before the oldest order in the database + (2019-12-31 on the production copy); months with no orders cost one index seek per client." + #inst "2015-01-01T00:00:00.000-00:00") + +(defn- ->date [^java.time.LocalDate d] + (java.util.Date/from (.toInstant (.atStartOfDay d (java.time.ZoneId/systemDefault))))) + +(defn order-months-newest-first + "`[start end]` month windows from now back to `earliest`, newest month first. + + The migration walks months in this order on purpose. It is the difference between an + interrupted run leaving the data safe to import against and leaving it dangerous: the importer + works on recent data, so having the newest months fully scoped is what lets imports resume + while the older tail is still unmigrated. Walking oldest-first would spend hours on 2019 before + touching anything this month's import will read. + + Windows tile without gaps — each month's end is the day before the next month's start — and + because the migration is idempotent an order landing in two windows is a no-op the second + time, so boundary precision is not safety-critical." + ([] (order-months-newest-first earliest-orders)) + ([^java.util.Date earliest] + (let [zone (java.time.ZoneId/systemDefault) + floor (java.time.YearMonth/from (.toLocalDate (.atZone (.toInstant earliest) zone)))] + (->> (iterate (fn [^java.time.YearMonth m] (.minusMonths m 1)) (java.time.YearMonth/now zone)) + (take-while (fn [^java.time.YearMonth m] (not (.isBefore m floor)))) + (map (fn [^java.time.YearMonth m] + [(->date (.atDay m 1)) (->date (.atEndOfMonth m))])))))) + +(defn orders-in-window + "Sales order ids for every client between `start` and `end` inclusive, via the + `:sales-order/client+date` index." + [db clients start end] + (map first (iol-ion.query/scan-sales-orders db clients start end))) + +(defn- all-client-ids [db] + (map first (dc/q '[:find ?c :where [?c :client/code _]] db))) + (defn migrate-all! "The complete migration, over the whole database rather than a chosen subset. @@ -300,16 +342,33 @@ Nine client pairs contended in the past and no longer share one; their records are still mixed, and a migration scoped to the current configuration would miss every one of them. + **Ordered so that an interrupted run is recoverable.** Refunds, payouts and cash-drawer shifts + go first: together they are a quarter of a million records and take seconds, so finishing them + up front means an interruption cannot leave them half done. The long part — walking every order + to split shared charges — then runs a month at a time from the current month backwards, logging + each month as it completes. Stop it after any month and the data from that month forward is + fully scoped, which is the part the importer reads, so imports can resume against it while the + older tail waits. Re-running picks up where it left off because each month's work is idempotent. + Returns the split counts and the completeness report, which should read zero legacy across the board when this finishes." [batch-size] - (let [split (split-and-rekey-charges! (all-order-ids (dc/db conn)) batch-size)] - (doseq [{:keys [attr prefix]} scoped-attrs - :when (not= attr :charge/external-id)] - (let [p (plan (dc/db conn) attr prefix)] - (when-let [c (seq (collisions (:new-keys p)))] - (throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)}))) - (migrate! attr (:new-keys p) batch-size))) + (doseq [{:keys [attr prefix]} scoped-attrs + :when (not= attr :charge/external-id)] + (let [p (plan (dc/db conn) attr prefix)] + (when-let [c (seq (collisions (:new-keys p)))] + (throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)}))) + (migrate! attr (:new-keys p) batch-size))) + (let [clients (all-client-ids (dc/db conn)) + split (reduce (fn [acc [start end]] + (let [ids (orders-in-window (dc/db conn) clients start end) + r (split-and-rekey-charges! ids batch-size)] + (alog/info ::month-complete + :month (subs (str (.toInstant ^java.util.Date start)) 0 7) + :rekeyed (:rekeyed r) :cloned (:cloned r)) + (merge-with + acc r))) + {:rekeyed 0 :cloned 0} + (order-months-newest-first))] ;; charges no order refers to — payout stubs — are scoped from the deposit that holds them. ;; Collision-checked like the others: this is the largest attribute in the database, so it is ;; the last one that should discover a clash as a mid-run exception. diff --git a/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj b/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj index cc0f646e..b2395c7f 100644 --- a/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj +++ b/test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj @@ -148,6 +148,23 @@ (is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2)))) "both orders still point at the one payment")))) +(deftest the-month-walk-runs-newest-first-and-leaves-no-gaps + (testing "order matters operationally, not just cosmetically: the importer reads recent data, so + an interrupted migration is only safe to resume imports against if the newest months + are the ones already done. Walking :aevt instead would start in 2019." + (let [windows (sut/order-months-newest-first #inst "2026-01-01T12:00:00.000-00:00") + starts (map first windows)] + (is (seq windows)) + (is (apply > (map #(.getTime ^java.util.Date %) starts)) + "strictly descending — newest month first") + (is (every? (fn [[[next-start _] [_ prev-end]]] + (= (.getTime ^java.util.Date next-start) + (+ (.getTime ^java.util.Date prev-end) (* 24 60 60 1000)))) + (partition 2 1 windows)) + "each window ends the day before the next one starts, so no order falls between them") + (is (every? (fn [[s e]] (.before ^java.util.Date s ^java.util.Date e)) windows) + "and every window is non-empty")))) + (deftest two-orders-of-the-same-client-keep-sharing-across-batches (testing "batch size does not change the same-client rule, which the sibling test cannot show because both its orders land in one batch. -- 2.49.1 From 4a1817711d45c7ce265b75269b84530f12e0b3a7 Mon Sep 17 00:00:00 2001 From: Bryce Date: Sun, 16 Aug 2026 09:59:32 -0700 Subject: [PATCH 6/8] docs: re-measure from a fresh restore with the duplicates left active MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The report's figures were taken with one client record of each shared pair deactivated and a live Square import run afterwards. That is no longer how this deploys — both records stay live and the re-key is what separates them — so the numbers described a configuration that will not exist. Re-run from scratch: fresh restore of backup point 209608347 (verified back to 16,545,495 charges and legacy keys before starting), the new month-wise migration, then two full ninety-day recomputes. before 1,451 days out of balance / $81,023.96 after 542 days out of balance / $20,239.00 915 into balance, 6 out, 17,916 summaries untouched Two corrections to claims that no longer hold: - "zero days knocked out of balance" is now six. All six are the same shape — the Tip line falls by a round amount and the day breaks by exactly that — and all six are on shared-location records. Each is a tip reversed on one record whose refund went to its twin: the fix makes a hidden mis-attribution visible rather than causing one. - the migration takes ~38 minutes, not ~13. The month-wise walk adds per-month index overhead, and the earlier figure predated it. What did NOT move is the part that should not. For the 190 clients that do not share a Square location the residue is 119 days and $1,730.61 in both runs, with the same five restaurants accounting for it (NG4S, NGMV, NGEB, NGPS, N-30012). The arithmetic fixes behave identically whatever is done to the duplicates, which is a stronger check than either run alone. $18,508.39 of the $20,239.00 — 91% — sits on the twenty shared-location records. The report now states plainly what retiring the duplicates would be worth (~$12,000 of variance across ~260 client-days per ninety days), while noting the comparison is not perfectly isolated because the earlier run also included a live import. Migration re-measured: 16,236,839 re-keyed, 500,438 cloned, plan reports {:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0}, and the multi-parent gate reads 0 across all 5,158,470 orders of the last year. Step 10's refunds re-measured too: 660 worth $15,237.02 dated before their own record's first order, 140 of the 171 no-sales days falling before it. 31 tests, 76 assertions. Co-Authored-By: Claude Opus 5 --- ...26-08-15-sales-summary-reconciliation.html | 131 ++++++++++-------- docs/2026-08-15-sales-summary-rollout-plan.md | 44 +++--- 2 files changed, 98 insertions(+), 77 deletions(-) diff --git a/docs/2026-08-15-sales-summary-reconciliation.html b/docs/2026-08-15-sales-summary-reconciliation.html index d2f27b2c..e2224ae5 100644 --- a/docs/2026-08-15-sales-summary-reconciliation.html +++ b/docs/2026-08-15-sales-summary-reconciliation.html @@ -105,27 +105,27 @@
Today's calculation, ninety days re-run - $69,560.10 - 1,258 days out of balance · 93.34% clean + $81,023.96 + 1,451 days out of balance · 92.32% clean
The same ninety days, fixes on - $7,790.54 - 279 days out of balance · 98.52% clean + $20,239.00 + 542 days out of balance · 97.13% clean
-
979client-days brought into balance
-
0days knocked out of balance
-
88.8%of the variance removed
+
915client-days brought into balance
+
6days knocked out of balance
+
75.0%of the variance removed
0payments shared between two clients

In one sentence: a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.

-

How the two figures above were produced. Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth 979 days and $61,769.56, not the larger number a stale baseline would have shown.

+

How the two figures above were produced. Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth a net 909 days and $60,784.96, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.

Most of what is left is not a balancing fault at all, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.

@@ -165,11 +165,11 @@

4. Refunds on records whose sales were never imported not fixed — deliberately

-

This is why the duplicated restaurants looked so much worse than everyone else. Of the days still failing after the first three fixes, 155 of the 158 on duplicate records had no sales orders at all — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.

-

The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded any order shows 132 of 156 fall before that client had a single order in the system — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.

+

This is why the duplicated restaurants looked so much worse than everyone else. Of the days still failing after the first three fixes, 155 of the 423 on shared-location records had no sales orders at all — the summary consisted of nothing but refunds and their fees, with no sales for them to reduce.

+

The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded any order shows 140 of 171 fall before that client had a single order in the system — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.

Where the refunds came from. Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to NGDG that same day, and was taken over by NGDU on 12 August. Others flip between the two records several times a day across 12–15 August. NGDU's first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.

-

Across the nine records, 659 refunds worth $15,225.24 sit on a record dated before that record's first order. Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.

-

Why it is deliberately left out of balance. The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and it closes 156 of the 279 remaining days and $4,820.19. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.

+

Across the nine records, 660 refunds worth $15,237.02 sit on a record dated before that record's first order. Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.

+

Why it is deliberately left out of balance. The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and on this data it closes 171 of the 542 remaining days and $5,795.18. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.

technical get-returns sums :sales-order/returns over orders scanned for the date. With no orders the sum is nil and no Returns line is written, while get-refund-items still credits Card Refunds from the sales-refund records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.

@@ -226,9 +226,8 @@

The result is pinned as the record's id on the way in, so the write lands on the existing row regardless of which name it currently carries. The proof this worked is a count that did not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was - re-named, and afterwards there were still exactly 144,688 payouts and 69,291 cash-drawer - shifts — the same figures as at the restore point. Refunds went from 50,986 to 51,986, - and all 1,000 of those came from the live Square import run afterwards, not from the renaming. + re-named, and afterwards there were still exactly 50,986 refunds, 144,688 payouts and + 69,291 cash-drawer shifts — the same three figures as at the restore point. Had the fallback lookup been missing, each of these would have doubled instead.

The fallback also has to refuse. Reading the old name is what stops duplicates; reading anyone's old name is what creates them. Two clients share a Square @@ -259,9 +258,9 @@ link it to the copy instead

-

Run over the whole database that was 9,100,314 renamed and 200,027 copied, +

Run over the whole database that was 16,236,839 renamed and 500,438 copied, and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across - every order of the last year. The record count rose by about 200,027 — the number of copies it + every order of the last year. The record count rose by about 500,438 — the number of copies it reported making, which is the check that it created what it meant to and nothing else.

One subtlety worth recording, because it bit us: the Square id has to be recovered from the record's current owner rather than by trimming a fixed prefix. Client codes contain dashes — @@ -333,7 +332,7 @@ (when-not (zero? amount) amount))))

-

It works. Measured over the same ninety days it closed 156 days and $4,820.19, +

It works. Measured over the same ninety days it closed 171 days and $5,795.18, knocked nothing out of balance, and altered no already-balanced day — the guard makes it incapable of touching a day that traded.

It was removed anyway. Those days are not quiet days; they are days whose sales were never @@ -372,9 +371,9 @@ - - - + + +
StageDays out of balanceCleanTotal variance
Today's calculation, ninety days re-run1,25893.34%$69,560.10
+ refunded tips97194.86%$66,414.39
+ service charges27998.52%$7,790.54
Today's calculation, ninety days re-run1,45192.32%$81,023.96
+ refunded tips1,19993.66%$78,522.85
+ service charges54297.13%$20,239.00

@@ -387,16 +386,18 @@ - - - + + +
ChangeUnchangedInto balanceOut of balanceBalanced days alteredMoney moved
Refunded tips18,59028700$3,712.67
Service charges18,20869200$58,349.85
Both, end to end17,91697900$61,769.56
Refunded tips18,58825266$3,777.67
Service charges18,20466300$58,923.85
Both, end to end17,91691566$60,784.96
-

Neither fix touched a day that was already correct. Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered — 17,916 summaries came out byte-identical, and every one of the 984 that moved was already wrong. Service charges are by far the larger of the two, moving $58,349.85 against the tip fix's $3,712.67.

+

Six days were knocked out of balance, and they are worth understanding rather than hiding. Across all 18,900 client-days, 17,916 summaries came out byte-identical and 978 of the 984 that moved were already wrong. The six exceptions all have one shape: the Tip line drops by a round amount — $10, $15, $20, $30, $30, $50 — and the day breaks by exactly that. They are all on shared-location records (NGDG, NGEZ, NGDA, NGBK).

+

That is the tip fix working, not failing. Each is a tip that was handed back: the reversal sits on this record's order, but the refund that should offset it went to the record's twin. Before the fix the day balanced by accident, because the reversal was ignored. After it, the day correctly shows that half the transaction is filed elsewhere. The honest description is that the fix converts a hidden mis-attribution into a visible one — which is the same trade the fourth problem below makes deliberately.

+

Service charges are by far the larger of the two fixes, moving $58,923.85 against the tip fix's $3,777.67.

That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.

-

The two fixes account for the repair exactly. Adding up the untendered-tip and service-charge amounts for the 979 repaired days leaves a residue of 0.0000000013 against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a third effect.

+

The two fixes account for every day they moved, exactly. Adding up the untendered-tip and service-charge amounts across all 984 changed days leaves a residue of 0.0000000002. Nothing else moved those days; there is no unexplained remainder hiding a third effect, and the six that broke are accounted for by the same arithmetic as the 915 that healed.

@@ -465,9 +466,9 @@ - - - + + + @@ -475,13 +476,13 @@
StepResult
Walk every order in the database19,040,785 orders
Give every order its own payment record9,100,314 re-keyed · 200,027 copied
Payments owned by two orders0 across every order of the last year — 5,159,787
Walk every order in the database, newest month first19,040,785 orders · 38 minutes
Give every order its own payment record16,236,839 re-keyed · 500,438 copied
Payments owned by two orders0 across every order of the last year — 5,158,470
Client-scope refunds, payouts and cash-drawer shiftscounts unchanged · 0 collisions
Live Square import afterwards0 orders with duplicated payment · 0 shared payments
Ownership changes after the change0 refunds · 0 payouts · 0 shifts
-

The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, close to the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)

+

The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added 500,438, matching the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)

The measurement above was taken with the duplicate client records deactivated, and that is not how this will be deployed. Leaving both records live is the intended configuration — the re-key is what separates them — but it means the deactivation that made this measurement clean will not be there. The gap that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why existing-id now refuses to resolve a record belonging to another client.

The whole analysis was run again from nothing, and landed in the same place -

Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. At the point the two runs are comparable — before the refund-only fix below, which the earlier run did not have — the end state matched to the cent: 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.

+

Everything above was rebuilt from a fresh restore of the production backup, three times over, each time from the backup point itself rather than from a database an earlier run had touched: restore, re-key and split across all nineteen million orders, then two full ninety-day recomputes. The runs used different preparation — one deactivated the duplicate records and ran a live Square import, this one leaves the configuration exactly as production has it — so their headline figures differ, and that difference is itself the most useful measurement in this report. What did not move is the part that should not: for the 190 clients that do not share a Square location, the residue is 119 days and $1,730.61 in both runs, with the same five restaurants accounting for it. The arithmetic fixes behave identically no matter what is done to the duplicates.

@@ -493,37 +494,48 @@

What is still out of balance

-

279 client-days out of 18,900, totalling $7,790.54. Most of it is not a balancing fault, and the split is the point:

+

542 client-days out of 18,900, totalling $20,239.00. Almost all of it sits + on one group of restaurants, and that is the finding:

- + - - - + +
What the remainder actually isDaysVariance
Where the remainder sitsDaysVarianceShare
Sales never imported — refunds on a record with no orders at all that day156$4,988.53
Processing fee on a day with no trading — the same shape, from the payout side15$974.99
Real trading days with genuine discrepancies108$1,995.36
The twenty records that share a Square location423$18,508.3991%
Every other client — 190 of the 210119$1,730.619%
-

Two thirds of what is left is a data-completeness problem, not an arithmetic one, and it is being left visible on purpose. Of those 156 days, 132 fall before their client had recorded a single order — they are periods with no sales in the system at all. The nine records involved hold 659 refunds worth $15,225.24 dated before their own first order.

-

The 15 fee-only days are the same story from the payout side: a processing fee landing on a day with no trading. Closing them needs the payout modelled, not another rule in the summary.

+

For every restaurant that is not one of the ten duplicated pairs, this reproduces + to the penny. 119 days and $1,730.61, with the same five clients accounting for + almost all of it — NG4S at $1,066.61, NGMV at $259.38, NGEB at $199.09, NGPS at $172.82, + N-30012 at $30.31, and 91 further days totalling $2.40 of till rounding. Those are the + numbers an earlier run produced on a differently prepared database, which is a stronger + check on the two arithmetic fixes than any single measurement: they behave identically + whatever is done to the duplicates.

+

The $18,508.39 on the twenty shared-location records is the cost of leaving both + records live. Each restaurant now keeps two sets of books, and the history behind + them was never split: refunds claimed by whichever record imported them first, orders that + went to the other, tips reversed on one side and refunded on the other. Re-keying makes that + attribution stable — it stops moving — but it does not make it right.

+

Of that remainder, 155 days and $4,567.53 are days those records had no sales imported at + all, which is the fourth problem above and is deliberately left visible. The other 268 days + are real trading days on which the two records disagree about who owns what.

-
- - - - - - - - - - -
Of the 108 real trading daysDaysVarianceWhat it is
NGBR1$299.42Square recorded $6,358.99 of tender against $6,059.57 of order totals on 2026-08-06 — the gap itself, not a summary fault
NGBK1$299.42
NGDL1$810.24an order on one record whose payment is on the twin — the mirror image of the refund case
NGMV5$259.38late May, undiagnosed
NGEB4$199.09ezCater fee treatment — an open question
everyone else96$127.81mostly till rounding — pennies a day
-
-
-

The clusters on NGMV and NGEB are unexplained and worth a look. They are visible only because the window is ninety days; a thirty-day view does not reach them.

+
+ What retiring the duplicate records would be worth +

An earlier measurement of the same window, taken with one record of each pair deactivated + and a live Square import run afterwards, left 279 days and $7,790.54 instead + of 542 and $20,239.00. Most of that difference is the twenty shared records: with the + duplicates retired their share fell from $18,508.39 to $6,059.93.

+

That is not an argument that the configuration is wrong — two live records is a deliberate + choice, and the re-key is what makes it safe. It is a number to weigh: leaving both active + costs roughly $12,000 of unexplained variance across 260 client-days per + ninety days, carried on ten restaurants, until the historical attribution behind them is + redistributed. The two figures are not perfectly isolated — that earlier run also included a + live import, which backfilled data this one does not have — so treat it as the right order of + magnitude rather than an exact price.

@@ -539,8 +551,8 @@ - - + + @@ -601,11 +613,12 @@
Re-running is safe, and that was proved at full scale

After the complete pass, asking the migration what it would change next returns - nothing — 17,047,142 payments examined, none to rename, none unscopable. A + nothing — 17,045,933 payments examined, none to rename, none unscopable. A record that already carries the right name is left untouched, so the migration can be stopped, resumed, or repeated without consequence.

Its speed is worth a note for whoever schedules it: the whole nineteen million orders were - walked in about thirteen minutes. An earlier attempt appeared to be + walked in about thirty-eight minutes, month by month from the current month + backwards so that stopping early leaves the recent end done. An earlier attempt appeared to be transactor-bound and was projected at two days, which is why a previous run narrowed it to the analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process driving the migration — freeing held memory took an unrelated recompute from 17 client-days a @@ -641,7 +654,7 @@ Two operational findings, unrelated to the summaries

The production backup had not written a restore point since 2025-03-10 — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.

The database server was sized for a toy dataset: a 2 GB cache against 27 GB of data. Worth checking what production is set to.

-

Slowness here was misdiagnosed twice, in the same direction. Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in thirteen minutes. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.

+

Slowness here was misdiagnosed twice, in the same direction. Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in well under an hour. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.

@@ -691,7 +704,7 @@
Measured 2026-08-15 against integreat-prod-restore, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch worktree-sales-summary-balance. - A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 279 remaining days, 189 are material — but 171 of those are days with no sales imported, not arithmetic faults. + A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 542 remaining days, 452 are material — but 171 are days with no sales imported rather than arithmetic faults, and 423 of the 542 sit on the twenty records that share a Square location. Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on as-of above.
diff --git a/docs/2026-08-15-sales-summary-rollout-plan.md b/docs/2026-08-15-sales-summary-rollout-plan.md index 62421cc2..0fd020f7 100644 --- a/docs/2026-08-15-sales-summary-rollout-plan.md +++ b/docs/2026-08-15-sales-summary-rollout-plan.md @@ -5,12 +5,17 @@ begins. The one behaviour change that alters a client's books is behind a per-cl that is **off by default**, so merging and deploying this branch changes nothing on its own. Measured on a restored copy of production (backup point `209608347`), 210 clients over -2026-05-10 → 2026-08-07: **1,258 client-days out of balance / $69,560.10 → 279 days / $7,790.54**, -with zero days knocked out of balance and zero already-balanced days altered. +2026-05-10 → 2026-08-07, with the duplicate client records left active exactly as they will be in +production: **1,451 client-days out of balance / $81,023.96 → 542 days / $20,239.00**. 915 days +came into balance and 6 broke — all six being tips reversed on one record whose refund sits on its +twin, which the fix makes visible rather than causes. -Of the 279 days left, **171 are not balancing faults** — they are days where a client's sales were -never imported while its refunds were. Step 10 is about those, and it is the most important -thing in this document. +**Where the remaining $20,239.00 sits is the thing to read.** $18,508.39 of it — 91% — is on the +twenty records that share a Square location. For the other 190 clients the residue is 119 days and +$1,730.61, which reproduces an earlier run on a differently prepared database to the penny. + +Of the total, 171 days are not balancing faults at all — they are days where a client's sales were +never imported while its refunds were. Step 10 is about those. --- @@ -20,7 +25,7 @@ thing in this document. |---|---| | Flag introduced | `summary-service-charges` — off by default | | Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` | -| Expected migration runtime | ~13 minutes for 19M orders on a warm cache | +| Expected migration runtime | ~38 minutes for 19M orders on a warm cache | | Nothing here touches | invoices, payments, the ledger, or any client without the flag set | **Client configuration is left exactly as it is.** Ten Square locations are configured against two @@ -64,7 +69,8 @@ that is the one code path that can reach across client records. Running the migr paused means no client is resolving keys while the keys are being rewritten, so the window closes entirely rather than merely narrowing. -The migration itself takes about **13 minutes** for all 19M orders, so the pause is short. +The migration itself takes about **38 minutes** for all 19M orders, so the pause is short — and +if you need it shorter, see step 5: you can resume imports before it finishes. --- @@ -104,9 +110,9 @@ Run it immediately after the deploy, while imports are still paused. `migrate-all!` runs this same check itself, on every attribute including charges, and throws rather than transacting if it finds one. Running it by hand first just means finding out before -the 13-minute walk rather than partway through it. +the 38-minute walk rather than partway through it. -Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that +Runs in about thirty-eight minutes over 19M orders. It is **idempotent and resumable** — a record that already carries the right name is skipped, so it can be stopped and re-run without consequence. **It is also ordered so that stopping early is survivable.** Refunds, payouts and cash-drawer @@ -140,7 +146,7 @@ attributes: ```clojure (dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys) -;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0} +;; => {:total 17045933 :to-migrate 0 :already-scoped 17045933 :unscopable 0} ``` Read `:to-migrate 0` **and** `:unscopable 0`. This is the authoritative signal, and it covers all @@ -163,7 +169,7 @@ order in the last year, with no sampling: ;; => 0 ``` -On the restored copy that is 5,159,787 orders — 27% of the table — via the +On the restored copy that is 5,158,470 orders — 27% of the table — via the `:sales-order/client+date` index. A year is chosen deliberately: `remove-voided-orders` only ever deletes orders Square reports as voided, which are recent, so that is where the destructive risk lives. Completeness across all of history is check (a)'s job, not this one. @@ -280,16 +286,16 @@ people doing the right thing. That is the reason not to let this drift. **The most important item in this document, and the only one that is not just execution.** -156 of the 279 remaining days are a record carrying refunds on a day it recorded no sales at all. -132 of those fall *before that client's first ever order*. Nine records are affected, holding -**659 refunds worth $15,225.24 dated before their own first order**: +171 of the 542 remaining days are a record carrying refunds on a day it recorded no sales at all. +140 of those fall *before that client's first ever order*. Nine records are affected, holding +**660 refunds worth $15,237.02 dated before their own first order**: | Client | First order | Refunds before it | Value | |---|---|---:|---:| | NGDL | 2026-07-29 | 96 | $2,751.50 | | NGWN | 2026-07-30 | 95 | $2,218.61 | | NGDU | 2026-08-02 | 94 | $2,188.44 | -| NG4S | 2026-05-29 | 78 | $2,168.30 | +| NG4S | 2026-05-29 | 79 | $2,180.08 | | NGPS | 2026-05-26 | 81 | $2,167.60 | | NGCT | 2026-08-01 | 98 | $1,789.49 | | NGVC | 2026-08-02 | 95 | $1,594.17 | @@ -329,7 +335,7 @@ all**, so a blanket delete would destroy the only copy of some records. ``` **Until this is resolved those days stay out of balance, on purpose.** A summary change to close -them was written and measured — it works, closes 156 days and $4,820.19, and breaks nothing — and it +them was written and measured — it works, closes 171 days and $5,795.18, and breaks nothing — and it was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not being imported. A test asserts the day stays unbalanced so nobody closes it without reading this. @@ -337,11 +343,13 @@ being imported. A test asserts the day stays unbalanced so nobody closes it with ## What this will not fix -The 108 real trading days still out of balance, $1,995.36 — the NGBR/NGBK tender gap ($6,358.99 of +The 371 real trading days still out of balance, $14,443.82 — of which $13,940.86 is on the twenty +shared-location records, where the two books disagree about who owns what. For every other client +it is 103 days and $502.96. Named causes: the NGBR/NGBK tender gap ($6,358.99 of tender against $6,059.57 of order totals on one day), the ezCater fee question, and unexplained clusters on NGMV and NGEB. -Plus 15 days / $974.99 where a processing fee lands on a day with no trading — the same shape as +Plus 16 days / $1,227.65 where a processing fee lands on a day with no trading — the same shape as step 10 but from the payout side, so it needs the payout modelled rather than a rule in the summary. --- -- 2.49.1 From 1558851c1829fbd6efddf9139d71ee1ed8fd82dd Mon Sep 17 00:00:00 2001 From: Bryce Date: Sun, 16 Aug 2026 17:58:59 -0700 Subject: [PATCH 7/8] feat(square): backfill history so shared-location records converge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-keying stops two client records on one Square location fighting over a record, but it does not make their books equal. Sales orders have always been keyed by client, so each record built its own order history from the start. Refunds, payouts and cash-drawer shifts were not, so only ONE record holds each of them — whichever imported it last. The migration freezes that ownership rather than evening it out. The record left without them shows returns from its own orders and no refunds against them. NGBK held 158,535 orders and five refunds. On 2026-05-11 both NGBK and NGBR held the same 221 orders; NGBK had no refunds, NGBR had two worth $2,232.29, and NGBK was out by $2,232.29 to the cent. Rather than manufacture copies, ask Square again. Client-scoped keys mean each record now creates its own copy of whatever it reads, so replaying a window converges the two histories with no code inventing a duplicate. `backfill-history` does that for a date range across orders, payouts, refunds and shifts. After it, all ten pairs held matching order and refund counts. Fixes a capped read found by doing this: the refunds import asked Square for a location's refunds and read only the first page — no cursor, no date range. Square pages at a hundred, so a location with more refunds silently returned a hundred and the response looked complete. That is why each twin held almost exactly 100 refunds and why an earlier import added exactly 1,000 across ten locations. `refunds`/`upsert-refunds` now follow the cursor and accept a window. Re-measured from the same fresh restore, duplicates left active: before 1,191 days out of balance / $70,276.50 after 122 days out of balance / $2,379.45 (32 material) 1,069 into balance, 0 out, 0 already-balanced days altered The shared records went from 423 days / $18,508.39 to 3 days / $648.84 — NGBK and NGBR at $299.42 each (the known tender-versus-order gap) and NGDA at $50.00 (auto-gratuity as a service charge). The zero-regression guarantee is restored too: the six days that broke without the backfill were tips reversed on one record whose refund sat on the twin, and all six closed once both sides had their own copy. This beats retiring the duplicate records, which left 279 days and $7,790.54, and it needs no decision about whose history to abandon. Unchanged across every run: for the 190 clients that do not share a location, 119 days and $1,730.61, same five restaurants. Cost: 5.9 hours for ninety days across twenty records. Every Square call shares one 25 req/s throttle, refunds and shifts cost one API call each, and backfill-history imports three clients at a time. Reads are not the limit — existing-id measures 32 microseconds. 31 tests, 76 assertions. Co-Authored-By: Claude Opus 5 --- ...26-08-15-sales-summary-reconciliation.html | 153 ++++++++++++------ docs/2026-08-15-sales-summary-rollout-plan.md | 94 ++++++++--- src/clj/auto_ap/square/core3.clj | 89 ++++++++-- 3 files changed, 251 insertions(+), 85 deletions(-) diff --git a/docs/2026-08-15-sales-summary-reconciliation.html b/docs/2026-08-15-sales-summary-reconciliation.html index e2224ae5..130be1e1 100644 --- a/docs/2026-08-15-sales-summary-reconciliation.html +++ b/docs/2026-08-15-sales-summary-reconciliation.html @@ -105,27 +105,27 @@
Today's calculation, ninety days re-run - $81,023.96 - 1,451 days out of balance · 92.32% clean + $70,276.50 + 1,191 days out of balance · 93.70% clean
The same ninety days, fixes on - $20,239.00 - 542 days out of balance · 97.13% clean + $2,379.45 + 122 days out of balance · 99.35% clean
-
915client-days brought into balance
-
6days knocked out of balance
-
75.0%of the variance removed
+
1,069client-days brought into balance
+
0days knocked out of balance
+
96.6%of the variance removed
0payments shared between two clients

In one sentence: a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, and service charges customers paid were credited to nothing.

-

How the two figures above were produced. Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth a net 909 days and $60,784.96, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.

+

How the two figures above were produced. Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth 1,069 days and $67,897.05, not the larger number a stale baseline would have shown. Both passes ran with the duplicate client records left active, which is how this will actually be deployed.

Most of what is left is not a balancing fault at all, and the section on the fourth problem explains why deliberately leaving it unbalanced is the right call.

@@ -169,7 +169,7 @@

The obvious reading is that the refund simply settled on a closed day. It is the wrong one. Checking each of those days against the date its client first recorded any order shows 140 of 171 fall before that client had a single order in the system — for seven of the nine records affected, every single one does. These are not quiet days. They are periods where the sales were never imported at all.

Where the refunds came from. Reading the database's own ownership history settles it. A $35.35 refund dated 26 February belonged to NGDG that same day, and was taken over by NGDU on 12 August. Others flip between the two records several times a day across 12–15 August. NGDU's first order is 2 August; it holds 94 refunds dated before it existed as a trading record. It never made them — it inherited seven months of the other record's refunds, because the refund key carried no client and whichever import ran last took ownership. That is fault 1, seen from the other end.

Across the nine records, 660 refunds worth $15,237.02 sit on a record dated before that record's first order. Nothing is lost and nothing is double-counted — the money is real and the surviving record has its own copy — but it is filed against a set of books that has no sales to put it against.

-

Why it is deliberately left out of balance. The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and on this data it closes 171 of the 542 remaining days and $5,795.18. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.

+

Why it is deliberately left out of balance. The day can be closed in one line: book a return equal to the day's refunds whenever the client recorded no sales. It is safe by construction — no trading day could be touched — and on this data it closes 16 of the 122 remaining days and $1,227.65. It was built, measured, and then removed, because it is the wrong thing to do. An unbalanced day is the only visible signal that a restaurant's sales are not being imported. Making the arithmetic agree would remove the alarm and leave the fire.

technical get-returns sums :sales-order/returns over orders scanned for the date. With no orders the sum is nil and no Returns line is written, while get-refund-items still credits Card Refunds from the sales-refund records. The imbalance is the correct output for the input; the input is what is wrong. A test now pins this behaviour in place so it is not "fixed" by someone reading only the arithmetic.

@@ -371,9 +371,9 @@
Record typeTotalClient-scopedStill to renameCannot be scoped
Card payments17,047,14217,047,14200
Refunds51,98651,98600
Card payments17,045,93317,045,93300
Refunds50,98650,98600
Payouts144,688144,652036
Cash-drawer shifts69,29169,29100
- - - + + +
StageDays out of balanceCleanTotal variance
Today's calculation, ninety days re-run1,45192.32%$81,023.96
+ refunded tips1,19993.66%$78,522.85
+ service charges54297.13%$20,239.00
Today's calculation, ninety days re-run1,19193.70%$70,276.50
+ refunded tips89095.29%$67,032.09
+ service charges12299.35%$2,379.45
@@ -386,15 +386,23 @@ - - - + + +
ChangeUnchangedInto balanceOut of balanceBalanced days alteredMoney moved
Refunded tips18,58825266$3,777.67
Service charges18,20466300$58,923.85
Both, end to end17,91691566$60,784.96
Refunded tips18,57130100$4,027.21
Service charges18,12976800$64,752.64
Both, end to end17,8271,06900$67,897.05
-

Six days were knocked out of balance, and they are worth understanding rather than hiding. Across all 18,900 client-days, 17,916 summaries came out byte-identical and 978 of the 984 that moved were already wrong. The six exceptions all have one shape: the Tip line drops by a round amount — $10, $15, $20, $30, $30, $50 — and the day breaks by exactly that. They are all on shared-location records (NGDG, NGEZ, NGDA, NGBK).

-

That is the tip fix working, not failing. Each is a tip that was handed back: the reversal sits on this record's order, but the refund that should offset it went to the record's twin. Before the fix the day balanced by accident, because the reversal was ignored. After it, the day correctly shows that half the transaction is filed elsewhere. The honest description is that the fix converts a hidden mis-attribution into a visible one — which is the same trade the fourth problem below makes deliberately.

+

Neither fix touched a day that was already correct. Across all 18,900 + client-days no balanced day was knocked out of balance, and no balanced day had a single + figure altered — 17,827 summaries came out byte-identical, and every one of the 1,073 that + moved was already wrong.

+

That claim did not hold on the first attempt, and how it was recovered is the useful part. + Measured before the historical backfill described below, six days broke — all of them a tip + reversed on one record whose refund sat on its twin, so removing the un-reversed tip left the + day short by exactly that amount. Replaying the window from Square gave both records their own + copy of every refund, and all six closed. The fix was never wrong; it was reading half a + transaction.

Service charges are by far the larger of the two fixes, moving $58,923.85 against the tip fix's $3,777.67.

That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.

The two fixes account for every day they moved, exactly. Adding up the untendered-tip and service-charge amounts across all 984 changed days leaves a residue of 0.0000000002. Nothing else moved those days; there is no unexplained remainder hiding a third effect, and the six that broke are accounted for by the same arithmetic as the 915 that healed.

@@ -477,12 +485,12 @@

The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added 500,438, matching the number of copies it reported making. (Close, not exact: the counter increments while the transaction is being assembled, so two copies that resolve onto one entity are counted twice. It is a good check, not a proof.)

-

The measurement above was taken with the duplicate client records deactivated, and that is not how this will be deployed. Leaving both records live is the intended configuration — the re-key is what separates them — but it means the deactivation that made this measurement clean will not be there. The gap that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why existing-id now refuses to resolve a record belonging to another client.

+

The measurement above was taken with both client records of each pair left live, which is how this deploys. Nothing is deactivated and no business decision about which restaurant's history survives is needed. The risk that opens is narrow and specific: while any record still carries a legacy key, a second client can resolve onto it. That is why the deployment runs the migration with imports paused, and why existing-id now refuses to resolve a record belonging to another client.

The whole analysis was run again from nothing, and landed in the same place -

Everything above was rebuilt from a fresh restore of the production backup, three times over, each time from the backup point itself rather than from a database an earlier run had touched: restore, re-key and split across all nineteen million orders, then two full ninety-day recomputes. The runs used different preparation — one deactivated the duplicate records and ran a live Square import, this one leaves the configuration exactly as production has it — so their headline figures differ, and that difference is itself the most useful measurement in this report. What did not move is the part that should not: for the 190 clients that do not share a Square location, the residue is 119 days and $1,730.61 in both runs, with the same five restaurants accounting for it. The arithmetic fixes behave identically no matter what is done to the duplicates.

+

Everything above was rebuilt from a fresh restore of the production backup, several times over, each time from the backup point itself rather than from a database an earlier run had touched: restore, re-key and split across all nineteen million orders, then two full ninety-day recomputes. The runs used deliberately different preparation — one deactivated the duplicate records, one left them live and untouched, one backfilled their history from Square — so their headline figures differ, and comparing them is how the recommendation below was reached. What did not move is the part that should not: for the 190 clients that do not share a Square location the residue is 119 days and $1,730.61 in every run, with the same five restaurants accounting for it. The arithmetic fixes behave identically no matter what is done to the duplicates, which is a stronger check on them than any single measurement.

@@ -494,48 +502,89 @@

What is still out of balance

-

542 client-days out of 18,900, totalling $20,239.00. Almost all of it sits - on one group of restaurants, and that is the finding:

+

122 client-days out of 18,900, totalling $2,379.45 — and only 32 of + those are above ten cents.

- + - - + +
Where the remainder sitsDaysVarianceShare
Where the remainder sitsDaysVariance
The twenty records that share a Square location423$18,508.3991%
Every other client — 190 of the 210119$1,730.619%
The twenty records that share a Square location3$648.84
Every other client — 190 of the 210119$1,730.61
-

For every restaurant that is not one of the ten duplicated pairs, this reproduces - to the penny. 119 days and $1,730.61, with the same five clients accounting for - almost all of it — NG4S at $1,066.61, NGMV at $259.38, NGEB at $199.09, NGPS at $172.82, - N-30012 at $30.31, and 91 further days totalling $2.40 of till rounding. Those are the - numbers an earlier run produced on a differently prepared database, which is a stronger - check on the two arithmetic fixes than any single measurement: they behave identically - whatever is done to the duplicates.

-

The $18,508.39 on the twenty shared-location records is the cost of leaving both - records live. Each restaurant now keeps two sets of books, and the history behind - them was never split: refunds claimed by whichever record imported them first, orders that - went to the other, tips reversed on one side and refunded on the other. Re-keying makes that - attribution stable — it stops moving — but it does not make it right.

-

Of that remainder, 155 days and $4,567.53 are days those records had no sales imported at - all, which is the fourth problem above and is deliberately left visible. The other 268 days - are real trading days on which the two records disagree about who owns what.

+

The shared-location records are now the clean part of the book. Three days + between all twenty of them: NGBK and NGBR at $299.42 each on 2026-08-06, which is the Square + tender-versus-order-total gap described below and not an attribution fault, and NGDA at $50.00, + an auto-gratuity booked as a service charge. Before the backfill those same records carried + 423 days and $18,508.39.

+

The other 119 days have not moved across any run of this analysis. Four + separate rebuilds — different databases, different preparation, one with the duplicates + deactivated and one without — all land on 119 days and $1,730.61, with the same five + restaurants accounting for almost all of it:

+
+
+ + + + + + + + + + +
ClientDaysVarianceWhat it is
NG4S10$1,066.61refunds arriving for a record with no sales imported — the fourth problem
NGMV5$259.38late May, undiagnosed
NGEB4$199.09ezCater fee treatment — an open question
NGPS7$172.82undiagnosed
N-300122$30.31late May, undiagnosed
everyone else91$2.40till rounding — pennies a day
+
+
+

Sixteen of the 122 are days a record had no sales imported at all, worth $1,227.65 — the + fourth problem, still deliberately visible. The clusters on NGMV, NGPS and NGEB are + unexplained and worth a look, though at under $650 across sixteen days they are no longer + urgent.

+
+
+ +
+

Making the duplicated restaurants match

+
+

Re-keying stops the two records fighting, but on its own it does not make them equal, and + the difference is worth stating plainly because it decides whether the books close.

+

Orders were always duplicated; refunds never were. A sales order's + identifier has always carried its client, so each of the two records built its own order + history from the start. Refunds, payouts and cash-drawer shifts did not, so only ONE record + holds each of them — whichever imported it last. The migration freezes that ownership rather + than evening it out. The record left without them shows returns from its own orders and no + refunds to set against them, and is out of balance by exactly the amount its twin is holding.

+

On 2026-05-11 both NGBK and NGBR held the same 221 orders. NGBK had no refunds; NGBR had + two, worth $2,232.29; and NGBK's books were out by $2,232.29 to the cent. Across the whole + database NGBK held 158,535 orders and five refunds.

+
+
+ The fix is to ask Square again, not to manufacture copies +

With client-scoped keys in place, every record now creates its own copy of whatever it + reads. So replaying the window from Square is all that is needed: each record imports the same + refunds independently and the two histories converge, without any code inventing a duplicate + and having to be trusted about it. backfill-history does exactly that for a date + range, and after it every one of the ten pairs held matching order and refund counts.

+

It closed 420 of the 423 days the shared records were carrying, and + $17,859.55 of the $18,508.39. It is also what recovered the zero-regression guarantee above.

- What retiring the duplicate records would be worth -

An earlier measurement of the same window, taken with one record of each pair deactivated - and a live Square import run afterwards, left 279 days and $7,790.54 instead - of 542 and $20,239.00. Most of that difference is the twenty shared records: with the - duplicates retired their share fell from $18,508.39 to $6,059.93.

-

That is not an argument that the configuration is wrong — two live records is a deliberate - choice, and the re-key is what makes it safe. It is a number to weigh: leaving both active - costs roughly $12,000 of unexplained variance across 260 client-days per - ninety days, carried on ten restaurants, until the historical attribution behind them is - redistributed. The two figures are not perfectly isolated — that earlier run also included a - live import, which backfilled data this one does not have — so treat it as the right order of - magnitude rather than an exact price.

+ One capped read, found by doing this +

The refunds import asked Square for a location's refunds and read the first page of the + answer — no cursor, no date range. Square pages at a hundred, so a location with more than a + hundred refunds silently returned a hundred, and the response looked complete. That is why the + twins each held almost exactly 100 refunds, and why an earlier import added exactly 1,000 + across ten locations. Following the cursor is a few lines; the reason it went unnoticed for so + long is that a capped list is indistinguishable from a short one.

+
+
+

This turned out to be a better answer than retiring the duplicate records. + An earlier measurement that deactivated one record of each pair left 279 days and $7,790.54. + Backfilling instead, with both records live, leaves 122 days and $2,379.45 — + and it needs no business decision about which restaurant's history to abandon.

@@ -704,7 +753,7 @@
Measured 2026-08-15 against integreat-prod-restore, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch worktree-sales-summary-balance. - A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 542 remaining days, 452 are material — but 171 are days with no sales imported rather than arithmetic faults, and 423 of the 542 sit on the twenty records that share a Square location. + A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 122 remaining days only 32 are material, and just 3 of them sit on the twenty records that share a Square location. Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on as-of above.
diff --git a/docs/2026-08-15-sales-summary-rollout-plan.md b/docs/2026-08-15-sales-summary-rollout-plan.md index 0fd020f7..69c8a845 100644 --- a/docs/2026-08-15-sales-summary-rollout-plan.md +++ b/docs/2026-08-15-sales-summary-rollout-plan.md @@ -6,16 +6,17 @@ that is **off by default**, so merging and deploying this branch changes nothing Measured on a restored copy of production (backup point `209608347`), 210 clients over 2026-05-10 → 2026-08-07, with the duplicate client records left active exactly as they will be in -production: **1,451 client-days out of balance / $81,023.96 → 542 days / $20,239.00**. 915 days -came into balance and 6 broke — all six being tips reversed on one record whose refund sits on its -twin, which the fix makes visible rather than causes. +production: **1,191 client-days out of balance / $70,276.50 → 122 days / $2,379.45**, of which only +32 are above ten cents. 1,069 days came into balance, none broke, and no already-balanced day had a +figure altered. -**Where the remaining $20,239.00 sits is the thing to read.** $18,508.39 of it — 91% — is on the -twenty records that share a Square location. For the other 190 clients the residue is 119 days and -$1,730.61, which reproduces an earlier run on a differently prepared database to the penny. +Of the $2,379.45 left, just **$648.84 across 3 days** is on the twenty shared-location records. The +other 119 days and $1,730.61 belong to ordinary clients and have not moved across any run of this +analysis. -Of the total, 171 days are not balancing faults at all — they are days where a client's sales were -never imported while its refunds were. Step 10 is about those. +Getting the shared records there needs step 5 — a historical backfill from Square. Without it they +carry 423 days and $18,508.39, because re-keying stops the two records fighting but does not give +each its own copy of the refunds. --- @@ -40,7 +41,7 @@ duplicate to clean up. If any report or export aggregates across client records, takings would be counted twice at that layer. Nothing in this work changes that either way. **The only window of risk is between deploying and finishing the migration**, while legacy keys -still exist for a client to resolve. Steps 2–5 exist to make that window effectively zero. +still exist for a client to resolve. Steps 2–6 exist to make that window effectively zero. --- @@ -61,7 +62,7 @@ See `docs/2026-08-15-remove-voided-orders-risk.md`. ## Step 2 — Pause the Square importer -**This is what makes the deploy safe, and it is easy to skip.** Steps 2 through 5 should be one +**This is what makes the deploy safe, and it is easy to skip.** Steps 2 through 6 should be one maintenance action, not separate days' work. While legacy keys exist, `square.core3/existing-id` falls back to them — and at a shared location @@ -70,7 +71,7 @@ paused means no client is resolving keys while the keys are being rewritten, so entirely rather than merely narrowing. The migration itself takes about **38 minutes** for all 19M orders, so the pause is short — and -if you need it shorter, see step 5: you can resume imports before it finishes. +if you need it shorter, see step 6: you can resume imports before it finishes. --- @@ -88,7 +89,7 @@ and the importer's new client-scoped keys. **The importer reads both key schemes**, so the deploy does not depend on the migration having finished. Two protections cover the interval before it does: imports are paused (step 2), and `existing-id` refuses to resolve a record that already belongs to a different client. Do not remove -the legacy lookup yet — see step 9. +the legacy lookup yet — see step 10. --- @@ -157,7 +158,7 @@ for charges, because ~283k payout stubs carry no `:charge/client` of their own a by attribute rather than by resolving ownership. Judge completeness by `plan`. **(b) The safety gate for the cascade** — no payment may answer to two orders, or re-enabling -`remove-voided-orders` in step 8 can delete a payment another order still needs. Check **every** +`remove-voided-orders` in step 9 can delete a payment another order still needs. Check **every** order in the last year, with no sampling: ```clojure @@ -181,9 +182,54 @@ lives. Completeness across all of history is check (a)'s job, not this one. --- -## Step 5 — Resume the Square importer +## Step 5 — Backfill the shared-location clients from Square -Normally: once step 4's two checks read clean. The maintenance window ends here. +**Skip this and the ten duplicated restaurants stay badly out of balance.** It is the difference +between 122 client-days out of balance and 542. + +Sales orders have always been keyed by client, so both records of a pair built their own order +history. Refunds, payouts and cash-drawer shifts were not, so only ONE record holds each of them. +Re-keying freezes that ownership; it does not even it out. The record left without them shows +returns from its own orders and no refunds against them — NGBK held 158,535 orders and five +refunds — and is out of balance by exactly what its twin is holding. + +Rather than manufacture copies, ask Square again. Client-scoped keys mean each record now creates +its own copy of whatever it reads, so replaying the window makes the two histories converge: + +```clojure +(require '[auto-ap.square.core3 :as sq]) +(require '[clj-time.core :as t]) + +@(apply sq/backfill-history + (t/date-time 2026 5 10) (t/date-time 2026 8 9) + ["NGBK" "NGBR" "NGCD" "NGCC" "NGVG" "NGVC" "NGEZ" "NGJS" "NGDG" "NGDU" + "NGDV" "NGDS" "NGWC" "NGWN" "NGHY" "NGHA" "NGDA" "NGDL" "NGCL" "NGCT"]) +``` + +**Verify** — every pair should hold matching order and refund counts in the window: + +```clojure +;; per pair, per side: window orders and window refunds. The two sides should agree. +``` + +Measured on the restored copy: all ten pairs matched afterwards, and the shared records went from +423 days and $18,508.39 out of balance to 3 days and $648.84. + +**Budget an overnight run.** This took **5.9 hours** for ninety days across the twenty records. +Every Square call in the process shares one 25-requests-per-second throttle, refunds and shifts cost +one API call per record, and `backfill-history` imports three clients at a time — raise its +`s/buffer` if you need it faster. Neither the database nor the transactor is the limit; reads +measured at 32 µs. + +It must run **after** the migration. Run before, and it imports against legacy keys and leaves more +to migrate. + +--- + +## Step 6 — Resume the Square importer + +Normally: once step 4's two checks read clean and step 5's backfill has finished. The maintenance +window ends here. **If the migration did not finish**, you do not have to wait for it. Resume imports once the `::month-complete` log covers the window your importer reads — the last 75 days for payouts and @@ -207,7 +253,7 @@ takings rather than merely duplicating a row. --- -## Step 6 — Recompute summaries, flags still off +## Step 7 — Recompute summaries, flags still off ```clojure (require '[auto-ap.jobs.sales-summaries :as ss]) @@ -215,7 +261,7 @@ takings rather than merely duplicating a row. ``` This is the pass that banks the deduplication. **Capture the result before going further** — you -will need it as the baseline for step 7, and it cannot be reconstructed afterwards: +will need it as the baseline for step 8, and it cannot be reconstructed afterwards: ```clojure (require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath @@ -230,7 +276,7 @@ will need it as the baseline for step 7, and it cannot be reconstructed afterwar --- -## Step 7 — Turn the flag on, a few restaurants at a time +## Step 8 — Turn the flag on, a few restaurants at a time Needs accounting sign-off first: `summary-service-charges` posts to **49000 Service Income**, chosen so the work could be measured. It affects reporting, never whether a day balances. @@ -243,7 +289,7 @@ so the work could be measured. It affects reporting, never whether a day balance Start with two or three restaurants, confirm, then widen. -**Verify** against the capture from step 6: +**Verify** against the capture from step 7: ```clojure (def after (cmp/summaries-in (d/db conn) start end)) @@ -255,18 +301,18 @@ The two numbers that matter — both were zero across all 18,900 client-days in - `:balanced->unbalanced` must be **0** - previously-balanced days whose lines changed must be **0** -If either is non-zero, retract the flag for the affected clients and re-run step 6. The flag is the +If either is non-zero, retract the flag for the affected clients and re-run step 7. The flag is the rollback: removing it restores today's behaviour exactly. --- -## Step 8 — Re-enable `remove-voided-orders` +## Step 9 — Re-enable `remove-voided-orders` Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 1. --- -## Step 9 — Remove the legacy key lookup +## Step 10 — Remove the legacy key lookup **Schedule this; do not leave it open-ended.** Both client records at a shared location stay active permanently, so the legacy fallback in `square.core3/existing-id` is the one code path that can ever @@ -282,7 +328,7 @@ people doing the right thing. That is the reason not to let this drift. --- -## Step 10 — Deal with the refunds that have no sales behind them +## Step 11 — Deal with the refunds that have no sales behind them **The most important item in this document, and the only one that is not just execution.** @@ -350,7 +396,7 @@ tender against $6,059.57 of order totals on one day), the ezCater fee question, clusters on NGMV and NGEB. Plus 16 days / $1,227.65 where a processing fee lands on a day with no trading — the same shape as -step 10 but from the payout side, so it needs the payout modelled rather than a rule in the summary. +step 11 but from the payout side, so it needs the payout modelled rather than a rule in the summary. --- diff --git a/src/clj/auto_ap/square/core3.clj b/src/clj/auto_ap/square/core3.clj index 97a2a329..86f4e78b 100644 --- a/src/clj/auto_ap/square/core3.clj +++ b/src/clj/auto_ap/square/core3.clj @@ -632,15 +632,44 @@ (log/error ::transform-payout-failed :exception e))))))))) -(defn refunds - ([client l] - (de/chain (manifold-api-call {:url (str "https://connect.squareup.com/v2/refunds?location_id=" (:square-location/square-id l)) - :method :get +(defn- refund-list + "Every refund Square has for this location in `[start end]`, following the cursor to the end. - :headers (client-base-headers client) - :as :json}) + The list endpoint returns one page at a time. Reading only the first page — which is what this + did before — silently caps a location at a hundred refunds however many it actually has, and + the cap is invisible: the response looks like a complete answer. On a shared location that is + how one client record ends up holding a few refunds against a hundred and fifty thousand orders. + + `start`/`end` are optional; omitting both asks for everything, which is what the nightly job + wants and what a historical backfill of more than a page needs." + ([client l start end] (refund-list client l start end nil)) + ([client l start end cursor] + (de/chain (manifold-api-call + {:url (str "https://connect.squareup.com/v2/refunds" + "?" + (url/map->query + (cond-> {:location_id (:square-location/square-id l) + :limit 100} + start (assoc :begin_time (->square-date start)) + end (assoc :end_time (->square-date end)) + cursor (assoc :cursor cursor)))) + :method :get + :headers (client-base-headers client) + :as :json}) :body - :refunds + (fn [result] + (log/info ::refunds-page + :count (count (:refunds result)) + :more? (boolean (not-empty (:cursor result)))) + (if (not-empty (:cursor result)) + (de/chain (refund-list client l start end (:cursor result)) + (fn [more] (concat (:refunds result) more))) + (:refunds result)))))) + +(defn refunds + ([client l] (refunds client l nil nil)) + ([client l start end] + (de/chain (refund-list client l start end) (fn [refunds] (->> refunds (filter (fn [r] (= "COMPLETED" (:status r)))) @@ -711,11 +740,12 @@ (for [square-location (:client/square-locations client) :when (:square-location/client-location square-location)] (upsert-refunds client square-location)))) - ([client location] + ([client location] (upsert-refunds client location nil nil)) + ([client location start end] (with-context-as {:source "Square refunds loading" :client (:client/code client)} lc - (de/chain (refunds client location) + (de/chain (refunds client location start end) (fn [refunds] (mu/with-context lc (try @@ -1011,6 +1041,47 @@ (s/realize-each) (s/reduce conj [])))) +(defn backfill-history + "Re-imports orders, payouts, refunds and cash-drawer shifts for `[start end]`, one client at a + time, for every square location the client has. + + This exists for the shared-location case. Sales orders have always been keyed by client, so two + client records on one Square location each built their own order history. Refunds, payouts and + shifts were not, so only ONE of the two records holds each of them — whichever imported it last + before the keys were scoped. Re-keying freezes that ownership; it does not even it out, and the + record left without them shows returns from its own orders with no refunds to offset them. + + Rather than manufacture copies, this asks Square again. With client-scoped keys in place every + record now creates its own copy of what it reads, so replaying the window is what makes the two + histories match. Deliberately not part of `upsert-all`: it walks further back than the nightly + job and is meant to be run once, after the migration. + + Run it AFTER `rekey-square-external-ids/migrate-all!`. Running it before would import against + legacy keys and leave more to migrate." + [start end & client-codes] + (with-context-as {:source "Square historical backfill"} lc + (->> (apply get-square-clients client-codes) + (s/->source) + (s/map (fn [client] + (with-context-as (merge lc {:client (:client/code client)}) lc + (-> + (apply de/zip + (for [l (:client/square-locations client) + :when (:square-location/client-location l)] + (de/chain + (upsert client l start end) + (fn [_] (upsert-payouts client l start end)) + (fn [_] (upsert-refunds client l start end)) + (fn [_] (upsert-cash-shifts client l start end)) + (fn [_] (log/info ::backfilled + :location (:square-location/client-location l)))))) + (de/catch (fn [e] + (mu/with-context lc + (log/info ::backfill-failed :severity :error :exception e)))))))) + (s/buffer 3) + (s/realize-each) + (s/reduce conj [])))) + (defn do-upsert-all [& clients] (mu/trace ::upsert-all -- 2.49.1 From 79a4b457b0e5502181bae7190855966eb0a4fcc9 Mon Sep 17 00:00:00 2001 From: Bryce Date: Sun, 16 Aug 2026 19:04:50 -0700 Subject: [PATCH 8/8] docs: bring the rollout plan's later steps up to date with the backfill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The headline and the new backfill step were current, but step 11 and "what this will not fix" still carried pre-backfill figures. The backfill did not just change the totals — it changed step 11's population. Before it, nine records held refunds dated before their own first order: 660 refunds worth $15,237.02. Seven of those were shared-location twins whose refunds only looked orphaned because their orders had never been imported; replaying the window gave them their orders and the refunds stopped predating them. Two are left, and they are a different case — neither shares a Square location, so there is no twin holding the other half: NG4S first order 2026-05-29 79 refunds $2,180.08 10 days NGPS first order 2026-05-26 81 refunds $2,167.60 7 days Step 11 now recommends trying backfill-history on them first, with a window reaching back before their first order, since that is exactly what resolved the other seven. "What this will not fix" re-measured: 122 days / $2,379.45, of which 106 are real trading days ($1,151.80) and 16 are refunds with no sales imported ($1,227.65). Only 3 of the trading days are on shared-location records, all already diagnosed. The other 103 days and $502.96 have been identical in every run of this analysis — deactivated, live, and backfilled — and are the floor this work reaches. Also adds the backfill's ~5.9 hour runtime to the up-front table, and notes that "two entities per Square object" holds automatically for new imports but needs step 5 for existing history. Co-Authored-By: Claude Opus 5 --- docs/2026-08-15-sales-summary-rollout-plan.md | 85 +++++++++++-------- 1 file changed, 48 insertions(+), 37 deletions(-) diff --git a/docs/2026-08-15-sales-summary-rollout-plan.md b/docs/2026-08-15-sales-summary-rollout-plan.md index 69c8a845..8cb551d4 100644 --- a/docs/2026-08-15-sales-summary-rollout-plan.md +++ b/docs/2026-08-15-sales-summary-rollout-plan.md @@ -27,6 +27,7 @@ each its own copy of the refunds. | Flag introduced | `summary-service-charges` — off by default | | Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` | | Expected migration runtime | ~38 minutes for 19M orders on a warm cache | +| Backfill runtime (step 5) | ~5.9 hours for 90 days across the 20 shared-location records — an overnight job | | Nothing here touches | invoices, payments, the ledger, or any client without the flag set | **Client configuration is left exactly as it is.** Ten Square locations are configured against two @@ -40,6 +41,11 @@ becomes **two entities, one per client record** — by design. That is the stabl duplicate to clean up. If any report or export aggregates across client records, one restaurant's takings would be counted twice at that layer. Nothing in this work changes that either way. +That holds automatically for everything imported *from now on*, because the keys carry the client. +It does **not** hold for history: refunds, payouts and shifts already in the database exist only +once, on whichever record imported them last, and re-keying freezes that rather than evening it out. +Step 5 is what brings the existing history into the same shape. + **The only window of risk is between deploying and finishing the migration**, while legacy keys still exist for a client to resolve. Steps 2–6 exist to make that window effectively zero. @@ -332,56 +338,51 @@ people doing the right thing. That is the reason not to let this drift. **The most important item in this document, and the only one that is not just execution.** -171 of the 542 remaining days are a record carrying refunds on a day it recorded no sales at all. -140 of those fall *before that client's first ever order*. Nine records are affected, holding -**660 refunds worth $15,237.02 dated before their own first order**: +16 of the 122 remaining days are a record carrying refunds on a day it recorded no sales at all, +and all 16 fall before that client's first ever order. Two clients are affected, holding **160 +refunds worth $4,347.68 dated before their own first order**: -| Client | First order | Refunds before it | Value | -|---|---|---:|---:| -| NGDL | 2026-07-29 | 96 | $2,751.50 | -| NGWN | 2026-07-30 | 95 | $2,218.61 | -| NGDU | 2026-08-02 | 94 | $2,188.44 | -| NG4S | 2026-05-29 | 79 | $2,180.08 | -| NGPS | 2026-05-26 | 81 | $2,167.60 | -| NGCT | 2026-08-01 | 98 | $1,789.49 | -| NGVC | 2026-08-02 | 95 | $1,594.17 | -| NGDS | 2026-01-14 | 22 | $347.13 | +| Client | First order | Refunds before it | Value | Days out of balance | +|---|---|---:|---:|---:| +| NG4S | 2026-05-29 | 79 | $2,180.08 | 10 | +| NGPS | 2026-05-26 | 81 | $2,167.60 | 7 | -They did not make those refunds. The database's own ownership history shows a $35.35 refund dated -26 February belonging to `NGDG` that day and taken over by `NGDU` on 12 August, with others flipping -between the two several times a day across 12–15 August. Unscoped refund keys meant whichever import -ran last took ownership — fault 1, seen from the other end. Steps 1 and 4 stop it recurring; they do -not undo it. +**Step 5's backfill already resolved the other seven.** Before it, nine records were in this state +holding 660 refunds worth $15,237.02 — but seven of them were shared-location twins whose refunds +only looked orphaned because their orders had never been imported. Replaying the window gave them +their orders, and the refunds stopped predating them. -Note `NG4S` and `NGPS` are **not** among the ten shared locations, so this is wider than the -duplicate records. +NG4S and NGPS are different: neither shares a Square location, so there is no twin holding the other +half. Their sales genuinely are not in the system for the period their refunds cover. The database's +own ownership history is the evidence to check — for the twins it showed refunds changing hands +between the two records; for these two there is no second record to have taken them from. Two ways to close it, and the business has to pick: -1. **Import the missing sales.** Correct if these records are meant to have their own books. Their - refunds are real and the matching orders exist in Square. -2. **Move the refunds to the record that has the sales.** Correct if the record is a duplicate that - should never have had books of its own. +1. **Import the missing sales.** Correct if these records are meant to have their own books. Try + `backfill-history` for them first, with a window reaching back before their first order — that is + exactly what fixed the seven, and it is one command. +2. **Move the refunds to the record that has the sales.** Correct only if the refunds were misfiled + onto a record that should not have books of its own. -Before either, check whether the surviving record already holds the same Square refund — within the -ninety-day window all 192 orphans did, but **outside it payouts and shifts have no counterpart at -all**, so a blanket delete would destroy the only copy of some records. +Start with (1): it is cheap, reversible in the sense that it only adds what Square reports, and it +is already proven to work on this exact symptom. ```clojure ;; per client: refunds dated before that client's own first order (let [first-order (->> (d/q '[:find [?d ...] :in $ ?c :where [?o :sales-order/client ?c] [?o :sales-order/date ?d]] - (d/db conn) [:client/code "NGDU"]) + (d/db conn) [:client/code "NG4S"]) (reduce (fn [a b] (if (.before a b) a b))))] (->> (d/q '[:find [(pull ?r [:sales-refund/date :sales-refund/total]) ...] :in $ ?c :where [?r :sales-refund/client ?c]] - (d/db conn) [:client/code "NGDU"]) + (d/db conn) [:client/code "NG4S"]) (filter #(.before (:sales-refund/date %) first-order)) count)) ``` **Until this is resolved those days stay out of balance, on purpose.** A summary change to close -them was written and measured — it works, closes 171 days and $5,795.18, and breaks nothing — and it +them was written and measured — it works, closes 16 days and $1,227.65, and breaks nothing — and it was removed, because an unbalanced day is the only visible signal that a restaurant's sales are not being imported. A test asserts the day stays unbalanced so nobody closes it without reading this. @@ -389,14 +390,24 @@ being imported. A test asserts the day stays unbalanced so nobody closes it with ## What this will not fix -The 371 real trading days still out of balance, $14,443.82 — of which $13,940.86 is on the twenty -shared-location records, where the two books disagree about who owns what. For every other client -it is 103 days and $502.96. Named causes: the NGBR/NGBK tender gap ($6,358.99 of -tender against $6,059.57 of order totals on one day), the ezCater fee question, and unexplained -clusters on NGMV and NGEB. +122 client-days over ninety days, $2,379.45, of which only 32 are above ten cents. -Plus 16 days / $1,227.65 where a processing fee lands on a day with no trading — the same shape as -step 11 but from the payout side, so it needs the payout modelled rather than a rule in the summary. +| | Days | Variance | | +|---|---:|---:|---| +| Real trading days with genuine discrepancies | 106 | $1,151.80 | see below | +| Refunds on a record with no sales imported | 16 | $1,227.65 | step 11 — deliberately visible | + +Of the 106 trading days, only **3 are on shared-location records** — $648.84 in total, and all three +are already diagnosed: NGBK and NGBR at $299.42 each on 2026-08-06, where Square recorded $6,358.99 +of tender against $6,059.57 of order totals (the gap itself, not a summary fault), and NGDA at +$50.00, an auto-gratuity booked as a service charge. + +The other 103 days come to **$502.96 across 190 clients** — a few dollars here and there, mostly +till rounding, plus small undiagnosed clusters on NGMV ($259.38 over 5 days) and NGEB ($199.09 over +4 days, an ezCater fee-treatment question). Those two are worth a look but are not urgent. + +That 103-day, $502.96 figure has been identical in every run of this analysis — with the duplicates +deactivated, with them live, and with them backfilled. It is the floor this work reaches. --- -- 2.49.1