fix(sales-summaries): stop days falling out of balance
Three faults were leaving restaurant days out of balance — one in the data, two in the arithmetic — plus a fourth that turned out to be a missing-data problem and is deliberately left visible. Measured over ninety days on a restored copy of production (210 clients, 18,900 client-days): 1,258 days out of balance and $69,560.10 becomes 279 days and $7,790.54 — of which 171 are not arithmetic faults at all, but days whose sales were never imported. 979 days repaired, none knocked out of balance, and not one already-balanced day altered — verified line by line (category, side, amount to the cent, account), not just on each day's bottom line. THE DATA FAULT Ten Square locations were configured against two client records each. Sales orders scoped their identifier by client; refunds, card payments, payouts and cash-drawer shifts used the bare Square id. Those attributes are :db.unique/identity, so both clients' imports resolved to a single entity and the last writer won — 3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts changed hands over time, across 19 client pairs of which only 10 are visible in today's configuration. Worse, one payment could belong to two orders. :sales-order/charges is :db/isComponent, so removing a voided order cascaded into payments the other client still needed. Fixes: client-scope the four key schemes; look the record up under both schemes so the change deploys before the migration finishes; and a migration that gives every order its own payment. Run over the whole database that is 19,040,785 orders walked, 9,100,314 payments re-keyed and 200,027 copied, ending with 17,047,142 payments scoped, none left to rename, none unscopable, and no payment owned by more than one order. Idempotent and resumable; about thirteen minutes. THE ARITHMETIC FAULTS - Refunded tips stayed on the books. get-tip summed tips by joining through :sales-order/charges, so a return-only order — no tender to join through — contributed nothing while its reversal sat unread on :sales-order/tip. Additive, not substitutive: where an order does have a tender the tender is the correct source. - Service charges were collected but never earned. Nothing read :sales-order/service-charge. Now credited for Square orders only, both signs, behind summary-service-charges. The flag is off by default, so deploying this changes nothing until a client is opted in. docs/2026-08-15-sales-summary-rollout-plan.md has the steps. WHAT IS DELIBERATELY NOT FIXED 156 of the 279 remaining days carry refunds on a record that recorded no sales at all that day, and 132 of those fall before that client's first ever order. The refunds are not theirs: ownership history shows a $35.35 refund dated 26 February belonging to NGDG that day and taken over by NGDU on 12 August, flipping between the two several times a day. Across nine records, 659 refunds worth $15,225.24 sit on a record dated before its own first order — unscoped keys let whichever import ran last take ownership. A rule closing those days was written and measured (156 days, $4,820.19, nothing broken) and then removed. An unbalanced day is the only visible signal that a restaurant's sales are not being imported; balancing it would remove the alarm and leave the fire. A comment and a test hold that decision in place. Step 9 of the rollout plan is the real fix, and it needs a business decision. SUPPORTING - Install schema attributes before the tuples that compose them. A tuple in schema.edn is built from an attribute in cloud-migration-schema.edn, so every test fixture died in setup — very likely why sales summaries had no tests before this. - Log each day's imbalance and its suspect lines. - Bound the dirty-summary scan to one client: 1,321 ms to 5.6 ms. - compare-sales-summaries lives in test/clj as auto-ap.tools.* — it is a verification harness, not part of the running application. Its docstring now warns that d/as-of cannot be used to compare summary amounts: :ledger-mapped/amount, ledger-side and account are :db/noHistory, so a recomputed summary reads back with its amounts absent and looks like a legitimate balanced day. 26 tests, 62 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
183
test/clj/auto_ap/jobs/sales_summaries_test.clj
Normal file
183
test/clj/auto_ap/jobs/sales_summaries_test.clj
Normal file
@@ -0,0 +1,183 @@
|
||||
(ns auto-ap.jobs.sales-summaries-test
|
||||
(:require
|
||||
[auto-ap.datomic :refer [conn]]
|
||||
[auto-ap.datomic.sales-summaries :as d-ss]
|
||||
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
|
||||
[auto-ap.jobs.sales-summaries :as sut]
|
||||
[clojure.test :refer [deftest is testing use-fixtures]]
|
||||
[datomic.api :as dc]))
|
||||
|
||||
(use-fixtures :each wrap-setup)
|
||||
|
||||
(def sales-date #inst "2026-08-01T07:00:00.000-00:00")
|
||||
|
||||
(defn- order
|
||||
"A sales order on `sales-date`, carrying whatever the case under test needs. The external id
|
||||
is Square-shaped by default because `get-service-charges` falls back to it when an order has
|
||||
no `:sales-order/vendor`."
|
||||
[client id attrs]
|
||||
(merge {:db/id (str "order-" id)
|
||||
:sales-order/external-id (str "square/order/TEST-" id)
|
||||
:sales-order/client client
|
||||
:sales-order/date sales-date
|
||||
:sales-order/total 100.0}
|
||||
attrs))
|
||||
|
||||
(defn- charge [id attrs]
|
||||
(merge {:db/id (str "charge-" id)
|
||||
:charge/external-id (str "square/charge/" id)
|
||||
:charge/type-name "CARD"
|
||||
:charge/total 100.0}
|
||||
attrs))
|
||||
|
||||
(defn- tip-for [client]
|
||||
(:ledger-mapped/amount (#'sut/get-tip client sales-date)))
|
||||
|
||||
(defn- service-charges-for [client]
|
||||
(#'sut/get-service-charges client sales-date))
|
||||
|
||||
(defn- enable-service-charges! [client]
|
||||
@(dc/transact conn [{:db/id client
|
||||
:client/feature-flags [sut/service-charges-flag]}]))
|
||||
|
||||
(deftest tip-counts-a-reversal-on-an-untendered-order
|
||||
(testing "a return-only order has no tender to join through, so its negative tip must come
|
||||
from the order or the day credits a tip that was handed back"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(order test-client-id "return-only" {:sales-order/tip -12.0})])
|
||||
(is (= -12.0 (tip-for test-client-id))))))
|
||||
|
||||
(deftest tip-on-a-tendered-order-still-comes-from-the-tender
|
||||
(testing "the tender carries a tip the order does not — auto-gratuity booked as a service
|
||||
charge. Reading the order instead of the tender would drop it."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(order test-client-id "tendered"
|
||||
{:sales-order/tip 0.0
|
||||
:sales-order/charges [(charge "tendered" {:charge/tip 50.0})]})])
|
||||
(is (= 50.0 (tip-for test-client-id))))))
|
||||
|
||||
(deftest tip-on-an-ordinary-order-is-counted-once
|
||||
(testing "an order that agrees with its tender is not double counted by the additive form"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(order test-client-id "ordinary"
|
||||
{:sales-order/tip 5.0
|
||||
:sales-order/charges [(charge "ordinary" {:charge/tip 5.0})]})])
|
||||
(is (= 5.0 (tip-for test-client-id))))))
|
||||
|
||||
(deftest service-charges-need-the-feature-flag
|
||||
(testing "without the flag the summary behaves exactly as it does today"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(order test-client-id "square-sc"
|
||||
{:sales-order/vendor :vendor/ccp-square
|
||||
:sales-order/service-charge 50.0})])
|
||||
(is (nil? (service-charges-for test-client-id))))))
|
||||
|
||||
(deftest service-charges-credit-square-orders
|
||||
(testing "a service charge rides along in the tender, so it needs a credit to match"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-service-charges! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "square-sc"
|
||||
{:sales-order/vendor :vendor/ccp-square
|
||||
:sales-order/service-charge 50.0})])
|
||||
(let [item (service-charges-for test-client-id)]
|
||||
(is (= 50.0 (:ledger-mapped/amount item)))
|
||||
(is (= :ledger-side/credit (:ledger-mapped/ledger-side item)))
|
||||
(is (= "Service Charges" (:sales-summary-item/category item)))))))
|
||||
|
||||
(deftest service-charges-count-both-signs
|
||||
(testing "a returned catering fee arrives as a negative service charge and is subtracted back
|
||||
out of returns, so dropping negatives loses the reversal"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-service-charges! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "refunded-fee"
|
||||
{:sales-order/vendor :vendor/ccp-square
|
||||
:sales-order/service-charge -140.0})])
|
||||
(is (= -140.0 (:ledger-mapped/amount (service-charges-for test-client-id)))))))
|
||||
|
||||
(deftest service-charges-exclude-non-square-vendors
|
||||
(testing "ezCater service charges are commission deducted from the restaurant rather than
|
||||
collected from the diner, so crediting them would make the day worse"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-service-charges! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "ezcater-sc"
|
||||
{:sales-order/external-id "ezcater/order/TEST-ezcater-sc"
|
||||
:sales-order/vendor :vendor/ccp-ezcater
|
||||
:sales-order/service-charge -75.0})])
|
||||
(is (nil? (service-charges-for test-client-id))))))
|
||||
|
||||
(deftest service-charges-recognise-square-orders-that-carry-no-vendor
|
||||
(testing "whole eras of Square orders have no :sales-order/vendor at all; a gate on vendor
|
||||
alone would silently credit nothing"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-service-charges! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "vendorless" {:sales-order/service-charge 12.5})])
|
||||
(is (= 12.5 (:ledger-mapped/amount (service-charges-for test-client-id)))))))
|
||||
|
||||
(deftest service-charges-ignore-vendorless-orders-from-other-sources
|
||||
(testing "the external id fallback is Square-specific, not a catch-all for missing vendors"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
(enable-service-charges! test-client-id)
|
||||
@(dc/transact conn [(order test-client-id "ezcater-vendorless"
|
||||
{:sales-order/external-id "ezcater/order/TEST-ezcater-vendorless"
|
||||
:sales-order/service-charge -75.0})])
|
||||
(is (nil? (service-charges-for test-client-id))))))
|
||||
|
||||
(defn- refund
|
||||
"A card refund on `sales-date`. The client+date tuple is set explicitly because
|
||||
`scan-sales-refunds` walks that index rather than the plain attributes."
|
||||
[client id total]
|
||||
{:db/id (str "refund-" id)
|
||||
:sales-refund/external-id (str "square/refund/TEST-" id)
|
||||
:sales-refund/client client
|
||||
:sales-refund/date sales-date
|
||||
:sales-refund/client+date [client sales-date]
|
||||
:sales-refund/type "CARD"
|
||||
:sales-refund/total total})
|
||||
|
||||
(defn- returns-for [client]
|
||||
(#'sut/get-returns client sales-date))
|
||||
|
||||
(deftest a-refund-with-no-sales-leaves-the-day-out-of-balance
|
||||
(testing "deliberate, and load-bearing. Booking a return against the day's refunds would close
|
||||
it and is tempting for that reason. But a day with refunds and no sales at all is
|
||||
overwhelmingly a day whose ORDERS WERE NEVER IMPORTED — on a restored copy of
|
||||
production, 132 of 156 such days fell before their client's first ever synced order.
|
||||
Balancing them would turn the only signal that a client's sales are missing into
|
||||
silence. If this test starts failing, read the rollout plan before changing it."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(refund test-client-id "no-sales" 40.0)])
|
||||
(is (nil? (returns-for test-client-id))
|
||||
"no return is invented for a day that recorded no sales")
|
||||
(is (= -40.0 (d-ss/imbalance (sut/get-refund-items test-client-id sales-date)))
|
||||
"so the day stays out of balance by the refunded amount, visibly"))))
|
||||
|
||||
(deftest a-day-that-traded-books-its-own-return
|
||||
(testing "the ordinary case: the return comes from the day's orders, never from its refunds"
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])]
|
||||
@(dc/transact conn [(order test-client-id "traded" {:sales-order/returns 7.0})
|
||||
(refund test-client-id "same-day" 40.0)])
|
||||
(is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id)))))))
|
||||
|
||||
(deftest dirty-summaries-stop-at-the-client-boundary
|
||||
(testing "every dirty day for the client is returned, and none belonging to another client.
|
||||
|
||||
:sales-summary/client+dirty sorts by client, so an unbounded index scan would walk
|
||||
every later client's summaries too — correct, but quadratic in the summary count."
|
||||
(let [{:strs [test-client-id]} (setup-test-data [])
|
||||
other (get-in @(dc/transact conn [{:db/id "other" :client/code (str "OTHER" (rand-int 100000))}])
|
||||
[:tempids "other"])
|
||||
day (fn [client d dirty?]
|
||||
{:sales-summary/client client
|
||||
:sales-summary/date d
|
||||
:sales-summary/dirty dirty?})]
|
||||
@(dc/transact conn [(day test-client-id #inst "2026-08-01T07:00:00.000-00:00" true)
|
||||
(day test-client-id #inst "2026-08-02T07:00:00.000-00:00" true)
|
||||
(day test-client-id #inst "2026-08-03T07:00:00.000-00:00" false)
|
||||
(day other #inst "2026-08-01T07:00:00.000-00:00" true)
|
||||
(day other #inst "2026-08-02T07:00:00.000-00:00" true)])
|
||||
(let [mine (sut/dirty-sales-summaries test-client-id)]
|
||||
(is (= 2 (count mine)) "both dirty days, and not the clean one")
|
||||
(is (every? #(= test-client-id (:db/id (:sales-summary/client %))) mine)
|
||||
"and nothing belonging to the other client"))
|
||||
(is (= 2 (count (sut/dirty-sales-summaries other)))
|
||||
"the other client's own dirty days are still found"))))
|
||||
Reference in New Issue
Block a user