notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

perf(sales-summaries): stop the dirty-summary scan at the client boundary

Browse Source 
dirty-sales-summaries index-pulled from [client-id true] and then filtered by
client. index-pull returns a lazy seq running to the END of the index, and a
lazy filter does not stop it, so for every client the job walked every summary
belonging to every client sorting after it — pulling their items along the way.
Quadratic in the number of summaries.

take-while stops at the client boundary instead, which is safe because
:sales-summary/client+dirty sorts by client first, so a client's dirty
summaries are contiguous from that start point.

Measured on a 14,458-summary database: 1,321ms -> 5.6ms per client, a 237x
improvement, with identical results. A full refresh had been degrading from
~180 client-days a minute to ~3 as summaries accumulated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Bryce
2026-08-15 08:44:52 -07:00
parent c5a6e8ab87
commit 4882c1c9cf
2 changed files with 36 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
src/clj/auto_ap/jobs/sales_summaries.clj
View File

@@ -62,15 +62,24 @@
{:ledger-mapped/ledger-side [:db/ident]}
{:ledger-mapped/account [:db/id]}])
(defn dirty-sales-summaries [c]
(defn dirty-sales-summaries
"The client's dirty summaries, with enough of each item to evaluate and re-transact it.
`index-pull` returns a lazy seq running from `:start` to the END of the index, so this must
stop at the client boundary rather than filter: `:sales-summary/client+dirty` sorts by client
first, so every later client's summaries sit beyond this client's and filtering would walk all
of them — for every client — pulling their items on the way. That is quadratic in the number of
summaries, and it showed up as a full refresh degrading from ~180 client-days a minute to ~3 as
the summary count grew."
[c]
(let [client-id (dc/entid (dc/db conn) c)]
(->> (dc/index-pull (dc/db conn)
{:index :avet
:selector (conj '[:sales-summary/date :sales-summary/client :db/id]
{:sales-summary/items item-read})
:start [:sales-summary/client+dirty [client-id true]]})
(filter (fn [sales-summary]
(= client-id (:db/id (:sales-summary/client sales-summary))))))))
(take-while (fn [sales-summary]
(= client-id (:db/id (:sales-summary/client sales-summary))))))))
(def default-days
"How far back the scheduled refresh looks for summaries that still need recomputing."

24
test/clj/auto_ap/jobs/sales_summaries_test.clj
View File

@@ -120,3 +120,27 @@
{:sales-order/external-id "ezcater/order/TEST-ezcater-vendorless"
:sales-order/service-charge -75.0})])
(is (nil? (service-charges-for test-client-id))))))
(deftest dirty-summaries-stop-at-the-client-boundary
(testing "every dirty day for the client is returned, and none belonging to another client.
:sales-summary/client+dirty sorts by client, so an unbounded index scan would walk
every later client's summaries too — correct, but quadratic in the summary count."
(let [{:strs [test-client-id]} (setup-test-data [])
other (get-in @(dc/transact conn [{:db/id "other" :client/code (str "OTHER" (rand-int 100000))}])
[:tempids "other"])
day (fn [client d dirty?]
{:sales-summary/client client
:sales-summary/date d
:sales-summary/dirty dirty?})]
@(dc/transact conn [(day test-client-id #inst "2026-08-01T07:00:00.000-00:00" true)
(day test-client-id #inst "2026-08-02T07:00:00.000-00:00" true)
(day test-client-id #inst "2026-08-03T07:00:00.000-00:00" false)
(day other #inst "2026-08-01T07:00:00.000-00:00" true)
(day other #inst "2026-08-02T07:00:00.000-00:00" true)])
(let [mine (sut/dirty-sales-summaries test-client-id)]
(is (= 2 (count mine)) "both dirty days, and not the clean one")
(is (every? #(= test-client-id (:db/id (:sales-summary/client %))) mine)
"and nothing belonging to the other client"))
(is (= 2 (count (sut/dirty-sales-summaries other)))
"the other client's own dirty days are still found"))))