notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

fix(square): do not resolve a record that belongs to another client

Browse Source 
Two clients on one Square location double a day's tender in the window
between deploying and finishing the migration. Reproduced end to end:

  1. charge X carries the legacy key square/charge/P and belongs to
     client B's order
  2. client A's payout import resolves X through existing-id's legacy
     fallback and renames it into A's scope
  3. client B's next order import matches neither scheme, so it mints a
     second charge
  4. :sales-order/charges is cardinality-many and orders transact as
     plain maps, so nothing retracts the first

B's order ends up holding two charges for one payment — $200 of tender
for a $100 payment — and running the migration on that state produces
square/charge/BBB-LB-AAA-LA-P, the same double-scoped shape that already
doubled tender on five clients once during this work.

existing-id's legacy branch now declines any record already owned by a
different client, reading the owner attribute and, for charges that
predate :charge/client, the client of the referencing order. Declining is
also correct on its merits: the write then lands on this client's own
copy, which is what the scoped keys exist to create. The payout path also
writes :charge/client/:charge/location alongside the key, so a charge's
scope and its owner can no longer disagree.

The guard is transitional and gets deleted with the legacy branch it
protects, at rollout step 9.

Rollout resequenced for the decision to leave duplicate client records
active: no deactivation, no "which record survives" call, and the risk
window closed by pausing the importer across deploy + migrate rather than
by removing one of the two writers. Both records converge to independent
stable histories once every key carries its owner.

Also from review:

- migrate-all! now collision-checks the charge pass like the other three
  attributes instead of discovering a clash mid-run over 17M rows
- split-and-rekey-charges! logs progress every 200 batches; an
  interrupted 19M-order run left no trail
- unscoped-report's docstring no longer promises a zero its :no-owner
  column cannot reach; plan is named as the authoritative signal
- the rollout's pre-flight asked for a :collisions key plan never
  returns, so it silently passed on every database
- the multi-parent gate sampled (take 400000 (all-order-ids db)), which
  streams :aevt — ascending entity id — and so read the OLDEST 2% of
  orders: 2019-12-31 to 2021-06-03, before any of the contention it
  looks for. Now every order of the last year via the client+date index,
  5,159,787 on the restored copy, reading 0
- the report claimed same-client pairs get copied once batches split
  them. They do not, at any batch size; verified at batch-size 1 and now
  pinned by a test

30 tests, 72 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Bryce
2026-08-16 08:09:53 -07:00
parent 57a84dae11
commit 10d0d01b82
6 changed files with 375 additions and 180 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
src/clj/auto_ap/jobs/rekey_square_external_ids.clj
View File

@@ -195,7 +195,11 @@
[orders batch-size]
(let [cloned (atom 0)
rekeyed (atom 0)]
(doseq [batch (partition-all batch-size orders)]
(doseq [[i batch] (map-indexed vector (partition-all batch-size orders))]
(when (zero? (mod i 200))
;; the whole-database run walks 19M orders; without a trail an interrupted run leaves
;; no way to tell how far it got short of querying the data by hand
(alog/info ::splitting :orders-done (* i batch-size) :rekeyed @rekeyed :cloned @cloned))
(let [db (dc/db conn)
batch-seen (atom {})
tx (doall
@@ -255,12 +259,19 @@
"Counts, per entity type, how many keys are already client-scoped, how many still carry the
legacy unscoped form, and how many have no owner to scope by.
This is the completeness gate. The importer tolerates both key schemes on purpose, so that the
change can be deployed before the migration finishes — but that tolerance is a transition, not
a resting place. While `:legacy` is above zero the database is in a mixed state and a stray
unscoped record can still be adopted by whichever client imports it first. Once every count
reads zero the fallback lookup in `square.core3/existing-id` can be removed and the guarantee
becomes structural rather than conventional."
The importer tolerates both key schemes on purpose, so that the change can be deployed before
the migration finishes — but that tolerance is a transition, not a resting place. While
`:legacy` is above zero the database is in a mixed state and a stray unscoped record can still
be adopted by whichever client imports it first. **`:legacy` reaching zero on every attribute is
the done-signal**, and it is what licenses removing the fallback lookup in
`square.core3/existing-id`.
`:no-owner` is NOT part of that signal and never reaches zero for charges. It counts entities
whose own `:charge/client`/`:charge/location` are absent — around an eighth of charges, the
payout stubs `migrate!` only ever gives an external id — so this report structurally cannot
verify them even when their keys are perfectly scoped. It classifies by attribute; `plan`
resolves ownership through whatever references the entity. Ask `plan` for the authoritative
answer: `:to-migrate 0` with `:unscopable 0` means there is nothing left to do."
[db]
(into {}
(for [{:keys [attr prefix client location]} scoped-attrs]
@@ -299,8 +310,13 @@
(when-let [c (seq (collisions (:new-keys p)))]
(throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
(migrate! attr (:new-keys p) batch-size)))
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them.
;; Collision-checked like the others: this is the largest attribute in the database, so it is
;; the last one that should discover a clash as a mid-run exception.
(let [p (plan (dc/db conn) :charge/external-id charge-prefix)]
(when-let [c (seq (collisions (:new-keys p)))]
(throw (ex-info "two entities would take the same key"
{:attr :charge/external-id :collisions (count c)})))
(when (seq (:new-keys p)) (migrate! :charge/external-id (:new-keys p) batch-size)))
{:split split :completeness (unscoped-report (dc/db conn))}))

169
src/clj/auto_ap/square/core3.clj
View File

@@ -27,11 +27,9 @@
"Authorization" (str "Bearer " (:client/square-auth-token client))
"Content-Type" "application/json"}))
(defn ->square-date [d]
(f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d))
(def manifold-api-stream
(let [stream (s/stream 100)]
(->> stream
@@ -42,10 +40,10 @@
(de/loop [attempt 0]
(-> (de/chain (de/future-with (ex/execute-pool)
#_(log/info ::request-started
:url (:url request)
:attempt attempt
:source "Square 3"
:background-job "Square 3")
:url (:url request)
:attempt attempt
:source "Square 3"
:background-job "Square 3")
(try
(client/request (assoc request
:socket-timeout 10000
@@ -104,7 +102,6 @@
:exception error))
[]))))
(def item-cache (atom {}))
(defn fetch-catalog [client i v]
@@ -124,13 +121,11 @@
#(do (swap! item-cache assoc i %)
%))))
(defn fetch-catalog-cache [client i version]
(if (get @item-cache i)
(de/success-deferred (get @item-cache i))
(fetch-catalog client i version)))
(defn item->category-name-impl [client item version]
(capture-context->lc
(cond (:item_id (:item_variation_data item))
@@ -161,7 +156,6 @@
:item item)
"Uncategorized"))))
(defn item-id->category-name [client i version]
(capture-context->lc
(-> [client i]
@@ -226,7 +220,6 @@
(concat (:orders result) continued-results))))
(:orders result)))))))
(defn search
([client location start end]
(capture-context->lc
@@ -250,11 +243,9 @@
(concat (:orders result) continued-results))))
(:orders result))))))))
(defn amount->money [amt]
(* 0.01 (or (:amount amt) 0.0)))
;; to get totals:
(comment
(reduce
@@ -278,6 +269,30 @@
[prefix client location id]
(str prefix (:client/code client) "-" (:square-location/client-location location) "-" id))
(def ^:private owner-attr
"Where each Square-imported entity records the client it belongs to."
{:charge/external-id :charge/client
:sales-refund/external-id :sales-refund/client
:expected-deposit/external-id :expected-deposit/client
:cash-drawer-shift/external-id :cash-drawer-shift/client})
(defn- owned-by-other-client?
"Whether `e` already belongs to a client other than `client-eid`.
Reads the entity's own owner attribute, and for a charge falls back to the client of whichever
sales order refers to it — charges predating `:charge/client` still have orders, and those are
exactly the ones that can be taken by the wrong client."
[db attr e client-eid]
(let [ent (dc/entity db e)
owner (or (:db/id ((owner-attr attr) ent))
(when (= attr :charge/external-id)
(some->> (first (dc/datoms db :vaet e :sales-order/charges))
:e
(dc/entity db)
:sales-order/client
:db/id)))]
(and owner (not= owner client-eid))))
(defn existing-id
"Entity id of the refund or charge this id already refers to, trying the client-scoped key
first and the legacy unscoped key second.
@@ -286,11 +301,24 @@
relies on upsert-by-identity; changing the key format on its own would match nothing and
Datomic would create a SECOND entity for every refund and charge, orphaning the original under
its legacy key. Pinning the result as `:db/id` makes the write land on the existing entity
whichever scheme it currently carries."
whichever scheme it currently carries.
The legacy branch will not take a record that already belongs to a different client. Without
that check, two clients on one Square location double money during the window between deploying
and finishing the migration: client A's payout import resolves B's charge by its bare key and
renames it into A's scope, B's next order import then matches neither scheme and mints a second
charge, and because `:sales-order/charges` is cardinality-many nothing retracts the first — so
B's order carries two charges for one payment. Declining is also the right answer on its merits:
the write then lands on this client's own copy, which is what the scoped keys exist to create.
Once the migration has run there are no legacy keys left for this branch to find, and both it
and the guard can be deleted together."
[db attr prefix client location id]
(when id
(or (dc/entid db [attr (scoped-key prefix client location id)])
(dc/entid db [attr (str prefix id)]))))
(when-let [legacy (dc/entid db [attr (str prefix id)])]
(when-not (owned-by-other-client? db attr legacy (:db/id client))
legacy)))))
(defn tender->charge [order client location t]
(remove-nils
@@ -304,7 +332,7 @@
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location (:id t))
:external-id (when (:id t)
(scoped-key "square/charge/" client location (:id t)))
:processor (cond
:processor (cond
(#{"OTHER" "THIRD_PARTY_CARD"} (:type t))
(condp = (some-> (:note t) str/lower-case)
"doordash" :ccp-processor/doordash
@@ -377,7 +405,7 @@
#:sales-order
{:date (if (= "Invoices" (:name (:source order)))
(when (:closed_at order)
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
(coerce/to-date (time/to-time-zone (coerce/to-date-time (:created_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
:client (:db/id client)
:location (:square-location/client-location location)
@@ -439,7 +467,6 @@
:client client
:location location)))))))
(defn get-payment [client p]
(de/chain (manifold-api-call
{:url (str "https://connect.squareup.com/v2/payments/" p)
@@ -448,7 +475,6 @@
:body
:payment))
(defn continue-payout-entry-list [c l poi cursor]
(capture-context->lc lc
(de/chain
@@ -593,6 +619,12 @@
(let [payment-id (:payment_id (:type_charge_details p))]
(remove-nils
{:charge/external-id (scoped-key "square/charge/" client location payment-id)
;; the owner attributes must travel with the key: `raw-square-id` and
;; `scope-of` both recover a charge's scope from them, and a key
;; scoped to one client while the owner says another is what makes
;; the migration write square/charge/B-LB-A-LA-<id>
:charge/client (:db/id client)
:charge/location (:square-location/client-location location)
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location payment-id)}))))))})
(filter :expected-deposit/date)
(into []))
@@ -651,7 +683,6 @@
:count (count x))
@(dc/transact-async conn x))))))))
(defn upsert-payouts
([client]
(apply de/zip
@@ -700,7 +731,6 @@
(log/info ::done-loading-refunds)))))))
(defn get-cash-shift [client id]
(de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id))
:method :get
@@ -864,8 +894,6 @@
d1
d2))
(defn remove-voided-orders
([client]
(apply de/zip
@@ -892,7 +920,7 @@
(:sales-order/external-id o))))))
(s/map (fn [[o]]
[[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]]))
(s/reduce into [])))
(fn [results]
@@ -901,31 +929,26 @@
(log/info ::removing-orders
:count (count x))
@(dc/transact-async conn x)))))
(de/catch (fn [e]
(log/warn ::couldnt-remove :error e)
nil) ))))))
(de/catch (fn [e]
(log/warn ::couldnt-remove :error e)
nil)))))))
#_(comment
(require 'auto-ap.time-reader)
#_(comment
(require 'auto-ap.time-reader)
@(let [[c [l]] (get-square-client-and-location "DBFS") ]
(log/peek :x [ c l])
(search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29")
@(let [[c [l]] (get-square-client-and-location "DBFS")]
(log/peek :x [c l])
(search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29"))
)
@(let [[c [l]] (get-square-client-and-location "NGAK")]
(log/peek :x [c l])
@(let [[c [l]] (get-square-client-and-location "NGAK") ]
(log/peek :x [ c l])
(remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
(doseq [c (get-square-clients)]
(try
@(remove-voided-orders c)
(catch Exception e
nil)))
)
(remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
(doseq [c (get-square-clients)]
(try
@(remove-voided-orders c)
(catch Exception e
nil))))
(defn upsert-all [& clients]
(capture-context->lc
@@ -994,8 +1017,6 @@
[:clients clients]
@(apply upsert-all clients)))
(comment
(defn refunds-raw-cont
([client l cursor so-far]
@@ -1025,9 +1046,8 @@
(->>
@(let [[c [l]] (get-square-client-and-location "NGGG")]
(search c l (time/now) (time/plus (time/now) (time/days -1))))
(filter (fn [r]
(str/starts-with? (:created_at r) "2024-03-14"))))
@@ -1035,7 +1055,6 @@
(->>
@(let [[c [l]] (get-square-client-and-location "NGGG")]
(refunds-raw-cont c l nil []))
(filter (fn [r]
(str/starts-with? (:created_at r) "2024-03-14")))))
@@ -1069,13 +1088,8 @@
[]))]
[(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row) (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
:separator \tab)
;; =>
;; =>
(require 'auto-ap.time-reader)
@@ -1084,27 +1098,16 @@
(clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")]
l
(def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00"
#clj-time/date-time "2025-02-28T00:00:00-08:00"))
(take 10 (map #(first (deref (order->sales-order c l %))) z)))
(take 10 (map #(first (deref (order->sales-order c l %))) z))))
)
(->> z
(->> z
(filter (fn [o]
(seq (filter (comp #{"OTHER"} :type) (:tenders o)))))
(filter #(not (:name (:source %))))
(count)
)
(count))
(doseq [[code] (seq (dc/q '[:find ?code
:in $
:where [?o :sales-order/date ?d]
@@ -1113,32 +1116,22 @@
[?o :sales-order/client ?c]
[?c :client/code ?code]]
(dc/db conn)))
:let [[c [l]] (get-square-client-and-location code)
]
:let [[c [l]] (get-square-client-and-location code)]
order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now))
:when (= "Invoices" (:name (:source order) ))
:when (= "Invoices" (:name (:source order)))
:let [[sales-order] @(order->sales-order c l order)]]
(when (should-import-order? order)
(println "DATE IS" (:sales-order/date sales-order))
(when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after? #clj-time/date-time "2026-2-16T00:00:00-08:00"))
(println "WOULD UPDATE" sales-order)
@(dc/transact auto-ap.datomic/conn [sales-order])
)
#_@(dc/transact )
(println "DONE"))
)
@(dc/transact auto-ap.datomic/conn [sales-order]))
#_@(dc/transact)
(println "DONE")))
#_(filter (comp #{"OTHER"} :type) (mapcat :tenders z))
@(let [[c [l]] (get-square-client-and-location "NGRY")]
#_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
(order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY" )))
)
)
(order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY")))))