notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(square): backfill history so shared-location records converge

Browse Source 
Re-keying stops two client records on one Square location fighting over a
record, but it does not make their books equal. Sales orders have always
been keyed by client, so each record built its own order history from the
start. Refunds, payouts and cash-drawer shifts were not, so only ONE
record holds each of them — whichever imported it last. The migration
freezes that ownership rather than evening it out.

The record left without them shows returns from its own orders and no
refunds against them. NGBK held 158,535 orders and five refunds. On
2026-05-11 both NGBK and NGBR held the same 221 orders; NGBK had no
refunds, NGBR had two worth $2,232.29, and NGBK was out by $2,232.29 to
the cent.

Rather than manufacture copies, ask Square again. Client-scoped keys mean
each record now creates its own copy of whatever it reads, so replaying a
window converges the two histories with no code inventing a duplicate.
`backfill-history` does that for a date range across orders, payouts,
refunds and shifts. After it, all ten pairs held matching order and
refund counts.

Fixes a capped read found by doing this: the refunds import asked Square
for a location's refunds and read only the first page — no cursor, no
date range. Square pages at a hundred, so a location with more refunds
silently returned a hundred and the response looked complete. That is why
each twin held almost exactly 100 refunds and why an earlier import added
exactly 1,000 across ten locations. `refunds`/`upsert-refunds` now follow
the cursor and accept a window.

Re-measured from the same fresh restore, duplicates left active:

  before  1,191 days out of balance / $70,276.50
  after     122 days out of balance / $2,379.45   (32 material)
            1,069 into balance, 0 out, 0 already-balanced days altered

The shared records went from 423 days / $18,508.39 to 3 days / $648.84 —
NGBK and NGBR at $299.42 each (the known tender-versus-order gap) and
NGDA at $50.00 (auto-gratuity as a service charge). The zero-regression
guarantee is restored too: the six days that broke without the backfill
were tips reversed on one record whose refund sat on the twin, and all
six closed once both sides had their own copy.

This beats retiring the duplicate records, which left 279 days and
$7,790.54, and it needs no decision about whose history to abandon.

Unchanged across every run: for the 190 clients that do not share a
location, 119 days and $1,730.61, same five restaurants.

Cost: 5.9 hours for ninety days across twenty records. Every Square call
shares one 25 req/s throttle, refunds and shifts cost one API call each,
and backfill-history imports three clients at a time. Reads are not the
limit — existing-id measures 32 microseconds.

31 tests, 76 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Bryce
2026-08-16 17:58:59 -07:00
parent 4a1817711d
commit 1558851c18
3 changed files with 251 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
src/clj/auto_ap/square/core3.clj
View File

@@ -632,15 +632,44 @@
(log/error ::transform-payout-failed
:exception e)))))))))
(defn refunds
([client l]
(de/chain (manifold-api-call {:url (str "https://connect.squareup.com/v2/refunds?location_id=" (:square-location/square-id l))
:method :get
(defn- refund-list
"Every refund Square has for this location in `[start end]`, following the cursor to the end.
:headers (client-base-headers client)
:as :json})
The list endpoint returns one page at a time. Reading only the first page — which is what this
did before — silently caps a location at a hundred refunds however many it actually has, and
the cap is invisible: the response looks like a complete answer. On a shared location that is
how one client record ends up holding a few refunds against a hundred and fifty thousand orders.
`start`/`end` are optional; omitting both asks for everything, which is what the nightly job
wants and what a historical backfill of more than a page needs."
([client l start end] (refund-list client l start end nil))
([client l start end cursor]
(de/chain (manifold-api-call
{:url (str "https://connect.squareup.com/v2/refunds"
"?"
(url/map->query
(cond-> {:location_id (:square-location/square-id l)
:limit 100}
start (assoc :begin_time (->square-date start))
end (assoc :end_time (->square-date end))
cursor (assoc :cursor cursor))))
:method :get
:headers (client-base-headers client)
:as :json})
:body
:refunds
(fn [result]
(log/info ::refunds-page
:count (count (:refunds result))
:more? (boolean (not-empty (:cursor result))))
(if (not-empty (:cursor result))
(de/chain (refund-list client l start end (:cursor result))
(fn [more] (concat (:refunds result) more)))
(:refunds result))))))
(defn refunds
([client l] (refunds client l nil nil))
([client l start end]
(de/chain (refund-list client l start end)
(fn [refunds]
(->> refunds
(filter (fn [r] (= "COMPLETED" (:status r))))
@@ -711,11 +740,12 @@
(for [square-location (:client/square-locations client)
:when (:square-location/client-location square-location)]
(upsert-refunds client square-location))))
([client location]
([client location] (upsert-refunds client location nil nil))
([client location start end]
(with-context-as {:source "Square refunds loading"
:client (:client/code client)} lc
(de/chain (refunds client location)
(de/chain (refunds client location start end)
(fn [refunds]
(mu/with-context lc
(try
@@ -1011,6 +1041,47 @@
(s/realize-each)
(s/reduce conj []))))
(defn backfill-history
"Re-imports orders, payouts, refunds and cash-drawer shifts for `[start end]`, one client at a
time, for every square location the client has.
This exists for the shared-location case. Sales orders have always been keyed by client, so two
client records on one Square location each built their own order history. Refunds, payouts and
shifts were not, so only ONE of the two records holds each of them — whichever imported it last
before the keys were scoped. Re-keying freezes that ownership; it does not even it out, and the
record left without them shows returns from its own orders with no refunds to offset them.
Rather than manufacture copies, this asks Square again. With client-scoped keys in place every
record now creates its own copy of what it reads, so replaying the window is what makes the two
histories match. Deliberately not part of `upsert-all`: it walks further back than the nightly
job and is meant to be run once, after the migration.
Run it AFTER `rekey-square-external-ids/migrate-all!`. Running it before would import against
legacy keys and leave more to migrate."
[start end & client-codes]
(with-context-as {:source "Square historical backfill"} lc
(->> (apply get-square-clients client-codes)
(s/->source)
(s/map (fn [client]
(with-context-as (merge lc {:client (:client/code client)}) lc
(->
(apply de/zip
(for [l (:client/square-locations client)
:when (:square-location/client-location l)]
(de/chain
(upsert client l start end)
(fn [_] (upsert-payouts client l start end))
(fn [_] (upsert-refunds client l start end))
(fn [_] (upsert-cash-shifts client l start end))
(fn [_] (log/info ::backfilled
:location (:square-location/client-location l))))))
(de/catch (fn [e]
(mu/with-context lc
(log/info ::backfill-failed :severity :error :exception e))))))))
(s/buffer 3)
(s/realize-each)
(s/reduce conj []))))
(defn do-upsert-all [& clients]
(mu/trace
::upsert-all