notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

perf(sales-summaries): stop the dirty-summary scan at the client boundary

Browse Source 
dirty-sales-summaries index-pulled from [client-id true] and then filtered by
client. index-pull returns a lazy seq running to the END of the index, and a
lazy filter does not stop it, so for every client the job walked every summary
belonging to every client sorting after it — pulling their items along the way.
Quadratic in the number of summaries.

take-while stops at the client boundary instead, which is safe because
:sales-summary/client+dirty sorts by client first, so a client's dirty
summaries are contiguous from that start point.

Measured on a 14,458-summary database: 1,321ms -> 5.6ms per client, a 237x
improvement, with identical results. A full refresh had been degrading from
~180 client-days a minute to ~3 as summaries accumulated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Bryce
2026-08-15 08:44:52 -07:00
parent c5a6e8ab87
commit 4882c1c9cf
2 changed files with 36 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
src/clj/auto_ap/jobs/sales_summaries.clj
View File

@@ -62,15 +62,24 @@
{:ledger-mapped/ledger-side [:db/ident]}
{:ledger-mapped/account [:db/id]}])
(defn dirty-sales-summaries [c]
(defn dirty-sales-summaries
"The client's dirty summaries, with enough of each item to evaluate and re-transact it.
`index-pull` returns a lazy seq running from `:start` to the END of the index, so this must
stop at the client boundary rather than filter: `:sales-summary/client+dirty` sorts by client
first, so every later client's summaries sit beyond this client's and filtering would walk all
of them — for every client — pulling their items on the way. That is quadratic in the number of
summaries, and it showed up as a full refresh degrading from ~180 client-days a minute to ~3 as
the summary count grew."
[c]
(let [client-id (dc/entid (dc/db conn) c)]
(->> (dc/index-pull (dc/db conn)
{:index :avet
:selector (conj '[:sales-summary/date :sales-summary/client :db/id]
{:sales-summary/items item-read})
:start [:sales-summary/client+dirty [client-id true]]})
(filter (fn [sales-summary]
(= client-id (:db/id (:sales-summary/client sales-summary))))))))
(take-while (fn [sales-summary]
(= client-id (:db/id (:sales-summary/client sales-summary))))))))
(def default-days
"How far back the scheduled refresh looks for summaries that still need recomputing."