notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity
Files
f8ef7918ef223bb31351b6870675194bed737205
integreat/src/clj/auto_ap/datomic.clj
Bryce f8ef7918ef fix(sales-summaries): stop days falling out of balance 
Four faults were leaving restaurant days out of balance — one in the
data, three in the arithmetic. Measured over ninety days on a restored
copy of production (210 clients, 18,900 client-days): 1,258 days out of
balance and $69,560.10 becomes 123 days and $2,970.35, of which only 33
are above ten cents.

1,135 days repaired, none knocked out of balance, and not one
already-balanced day altered — verified line by line (category, side,
amount to the cent, account), not just on each day's bottom line.

THE DATA FAULT

Ten Square locations were configured against two client records each.
Sales orders scoped their identifier by client; refunds, card payments,
payouts and cash-drawer shifts used the bare Square id. Those attributes
are :db.unique/identity, so both clients' imports resolved to a single
entity and the last writer won — 3,387 refunds, 4,069 payouts and 2,628
cash-drawer shifts changed hands over time, across 19 client pairs of
which only 10 are visible in today's configuration.

Worse, one payment could belong to two orders. :sales-order/charges is
:db/isComponent, so removing a voided order cascaded into payments the
other client still needed.

Fixes: client-scope the four key schemes; look the record up under both
schemes so the change deploys before the migration finishes; and a
migration that gives every order its own payment. Run over the whole
database that is 19,040,785 orders walked, 9,100,314 payments re-keyed
and 200,027 copied, ending with 17,047,142 payments scoped, none left to
rename, none unscopable, and no payment owned by more than one order.
Idempotent and resumable; about thirteen minutes.

THE ARITHMETIC FAULTS

- Refunded tips stayed on the books. get-tip summed tips by joining
  through :sales-order/charges, so a return-only order — no tender to
  join through — contributed nothing while its reversal sat unread on
  :sales-order/tip. Additive, not substitutive: where an order does have
  a tender the tender is the correct source.

- Service charges were collected but never earned. Nothing read
  :sales-order/service-charge. Now credited for Square orders only, both
  signs, behind summary-service-charges.

- A refund on a day with no sales had nothing to offset it. Refunds are
  credited on the day the money goes back; the return that offsets them
  is read from that day's orders. get-returns now falls back to the day's
  refunded total, but only where the client recorded no sales orders at
  all — with no orders there is no order-derived return to double-count
  and no trading day can be moved. Behind summary-refund-only-returns.

Both flags are off by default, so deploying this changes nothing until a
client is opted in. docs/2026-08-15-sales-summary-rollout-plan.md has the
steps.

SUPPORTING

- Install schema attributes before the tuples that compose them. A tuple
  in schema.edn is built from an attribute in cloud-migration-schema.edn,
  so every test fixture died in setup — very likely why sales summaries
  had no tests before this.
- Log each day's imbalance and its suspect lines.
- Bound the dirty-summary scan to one client: 1,321 ms to 5.6 ms.
- compare-sales-summaries lives in test/clj as auto-ap.tools.* — it is a
  verification harness, not part of the running application. Its
  docstring now warns that d/as-of cannot be used to compare summary
  amounts: :ledger-mapped/amount, ledger-side and account are
  :db/noHistory, so a recomputed summary reads back with its amounts
  absent and looks like a legitimate balanced day.

28 tests, 65 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 19:19:04 -07:00

956 lines
36 KiB
Clojure
Raw Blame History

(ns auto-ap.datomic
(:require
[iol-ion.tx.pay]
[iol-ion.tx.plus]
[iol-ion.tx.propose-invoice]
[iol-ion.tx.reset-rels]
[iol-ion.tx.reset-scalars]
[iol-ion.tx.upsert-entity]
[iol-ion.tx.upsert-invoice]
[iol-ion.tx.upsert-ledger]
[iol-ion.tx.upsert-transaction]
[iol-ion.tx.upsert-sales-summary-ledger]
[com.github.ivarref.gen-fn :refer [gen-fn! datomic-fn]]
[auto-ap.utils :refer [default-pagination-size by]]
[clojure.edn :as edn]
[config.core :refer [env]]
[datomic.api :as dc]
[com.brunobonacci.mulog :as mu]
[mount.core :as mount]
[clojure.java.io :as io]
[datomic.db :refer [id-literal]]
[datomic.function :refer [construct]]
[auto-ap.logging :as alog])
(:import
(java.util UUID)))
(def uri (:datomic-url env))
#_(mount/defstate client
:start (dc/client (:client-config env))
:stop nil)
(mount/defstate conn
:start (dc/connect uri)
:stop nil)
#_(def uri "datomic:mem://datomic-transactor:4334/invoice")
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
#_(defn create-database []
(d/create-database uri))
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
#_(defn drop-database []
(d/delete-database uri))
(defn remove-nils [m]
(let [result (reduce-kv
(fn [m k v]
(if (not (nil? v))
(assoc m k v)
m
))
{}
m)]
(if (seq result)
result
nil)))
(def vendor-schema
[{:db/ident :vendor/original-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "Original id from the old system"}
{:db/ident :vendor/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A vendor's human-friendly name"}
{:db/ident :vendor/code
:db/valueType :db.type/string
:db/unique :db.unique/identity
:db/cardinality :db.cardinality/one
:db/doc "A vendor's computer-friendly name"}
{:db/ident :vendor/print-as
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Set if you want to override how this vendor's name is printed on checks"}
{:db/ident :vendor/email
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A vendor's email address"}
{:db/ident :vendor/phone
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A vendor's phone number"}
{:db/ident :vendor/invoice-reminder-schedule
:db/valueType :db.type/keyword
:db/cardinality :db.cardinality/one
:db/doc "How often to email this vendor about invoices"}
{:db/ident :vendor/default-expense-account
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "The vendor's default expense account"}
{:db/ident :vendor/primary-contact
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/isComponent true
:db/doc "The vendor's primary contact"}
{:db/ident :vendor/secondary-contact
:db/valueType :db.type/ref
:db/isComponent true
:db/cardinality :db.cardinality/one
:db/doc "The vendor's secondary contact"}
{:db/ident :vendor/address
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/isComponent true
:db.install/_attribute :db.part/db
:db/doc "The vendor's address"}
])
(def client-schema
[{:db/ident :client/original-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "Original id from the old system"}
{:db/ident :client/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A client's human-friendly name"}
{:db/ident :client/email
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "hello@example.com"}
{:db/ident :client/code
:db/valueType :db.type/string
:db/unique :db.unique/identity
:db/cardinality :db.cardinality/one
:db/doc "A client's computer-friendly name"}
{:db/ident :client/signature-file
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A check signature image"}
{:db/ident :client/locations
:db/valueType :db.type/string
:db/cardinality :db.cardinality/many
:db/doc "A client's locations"}
{:db/ident :client/address
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/isComponent true
:db/doc "Address of the client"}
{:db/ident :client/bank-accounts
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db/isComponent true
:db/doc "Bank accounts for the client"}])
(def address-schema
[
{:db/ident :address/street1
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "123 main st"}
{:db/ident :address/street2
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Apt A"}
{:db/ident :address/city
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Campbell"}
{:db/ident :address/state
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "CA"}
{:db/ident :address/zip
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "95014"}])
(def contact-schema
[
{:db/ident :contact/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "John Smith"}
{:db/ident :contact/phone
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "4255773578"}
{:db/ident :contact/email
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "hello@example.com"}])
(def bank-account-schema
[{:db/ident :bank-account/external-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "Identifier for bank account"}
{:db/ident :bank-account/original-id
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "matching the orignal tuple of [company,bank]"}
{:db/ident :bank-account/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A1127 Chase Card"}
{:db/ident :bank-account/bank-name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "e.g. Bank of America"}
{:db/ident :bank-account/bank-code
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The code to list under the bank's name"}
{:db/ident :bank-account/routing
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The bank's routing number"}
{:db/ident :bank-account/number
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The account number"}
{:db/ident :bank-account/type
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/isComponent true
:db/doc "The type of account number, either :bank-account-type/check or :bank-account-type/cash"}
{:db/ident :bank-account/yodlee-account-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "yodlee's account identifier"}
{:db/ident :bank-account/check-number
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "Current check number"}
{:db/ident :bank-account-type/check}
{:db/ident :bank-account-type/cash}])
(def invoice-schema
[{:db/ident :invoice/original-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "Original id in old system"}
{:db/ident :invoice/invoice-number
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A vendor-specified number for the invoice"}
{:db/ident :invoice/customer-identifier
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "An identifier found to suggest the customer"}
{:db/ident :invoice/status
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "Status of payment/import of the invoice [:paid, :unpaid]"}
{:db/ident :invoice/client
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "Which client this invoice is for"}
{:db/ident :invoice/vendor
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "Which vendor this invoice is for"}
{:db/ident :invoice/date
:db/valueType :db.type/instant
:db/cardinality :db.cardinality/one
:db/doc "Date for this invoice"}
{:db/ident :invoice/total
:db/valueType :db.type/double
:db/cardinality :db.cardinality/one
:db/doc "Total $ for this invoice"}
{:db/ident :invoice/outstanding-balance
:db/valueType :db.type/double
:db/cardinality :db.cardinality/one
:db/doc "The unpaid balance of this invoice"}
{:db/ident :invoice/default-location
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The default location that expense-accounts will be created with for this invoice"}
{:db/ident :invoice/default-expense-account
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "The default expense account for this invoice"}
{:db/ident :invoice/expense-accounts
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db/isComponent true
:db/doc "The expense account categories for this invoice"}
{:db/ident :invoice-status/paid}
{:db/ident :invoice-status/unpaid}
{:db/ident :invoice-status/voided}])
(def invoice-expense-account-schema
[{:db/ident :invoice-expense-account/original-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "Original id in old system"}
{:db/ident :invoice-expense-account/expense-account-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "The code for the expense account"}
{:db/ident :invoice-expense-account/location
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Location for this expense account"}
{:db/ident :invoice-expense-account/amount
:db/valueType :db.type/double
:db/cardinality :db.cardinality/one
:db/doc "The amount that this contributes to"}])
(def payment-schema
[{:db/ident :payment/original-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "The id in the old system"}
{:db/ident :payment/s3-uuid
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The uuid that matches the key for this check"}
{:db/ident :payment/s3-key
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The s3 key with pdf of this check"}
{:db/ident :payment/s3-url
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The s3 url with pdf of this check"}
{:db/ident :payment/check-number
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "The check number"}
{:db/ident :payment/memo
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The check's memo line"}
{:db/ident :payment/date
:db/valueType :db.type/instant
:db/cardinality :db.cardinality/one
:db/doc "The date the payment was made"}
{:db/ident :payment/amount
:db/valueType :db.type/double
:db/cardinality :db.cardinality/one
:db/doc "The amount that was paid to the vendor"}
{:db/ident :payment/paid-to
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Who the paid was made out to"}
{:db/ident :payment/status
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The status of the payment [:pending :cleared :voided]"}
{:db/ident :payment/type
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The type of the payment [:cash :check :debit]"}
{:db/ident :payment/pdf-data
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "raw data used to generate check pdf"}
;; relations
{:db/ident :payment/vendor
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The vendor for which this payment was for"}
{:db/ident :payment/client
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The client for which this payment"}
{:db/ident :payment/bank-account
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The bank account that was used to pay"}
{:db/ident :payment/invoices
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db/doc "Any invoices this payment is related to"}
;; enums
{:db/ident :payment-status/pending}
{:db/ident :payment-status/voided}
{:db/ident :payment-status/cleared}
{:db/ident :payment-type/cash}
{:db/ident :payment-type/check}
{:db/ident :payment-type/debit}
])
(def invoice-payment-schema
[{:db/ident :invoice-payment/original-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "The id in the old system"}
{:db/ident :invoice-payment/amount
:db/valueType :db.type/double
:db/cardinality :db.cardinality/one
:db/doc "The amount that was paid to this invoice"}
;; relations
{:db/ident :invoice-payment/invoice
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The invoice for this payment"}
{:db/ident :invoice-payment/payment
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The total payment for this payment"}])
(def transaction-schema
[{:db/ident :transaction/original-id
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "The id in the old system"}
{:db/ident :transaction/amount
:db/valueType :db.type/double
:db/cardinality :db.cardinality/one
:db/doc "The amount of the transaction"}
{:db/ident :transaction/description-original
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "full description of the transaction"}
{:db/ident :transaction/description-simple
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "short description of the transaction"}
{:db/ident :transaction/merchant-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "a yodlee id for the merchant"}
{:db/ident :transaction/merchant-name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "a name for the merchant"}
{:db/ident :transaction/id
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "A key to match against"}
{:db/ident :transaction/date
:db/valueType :db.type/instant
:db/cardinality :db.cardinality/one
:db/doc "Date that the transaction showed up"}
{:db/ident :transaction/post-date
:db/valueType :db.type/instant
:db/cardinality :db.cardinality/one
:db/doc "Date that the transaction posted"}
{:db/ident :transaction/type
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Yodlee description of the transaction"}
{:db/ident :transaction/status
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Yodlee status of the transaction"}
{:db/ident :transaction/account-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "Yodlee account id"}
{:db/ident :transaction/check-number
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "The check number that was parsed from the description"}
;; relations
{:db/ident :transaction/vendor
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "Vendor for who we think this transaction is from"}
{:db/ident :transaction/client
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "Client for who we think this transaction is for"}
{:db/ident :transaction/bank-account
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The bank accout used for this transaction"}
{:db/ident :transaction/payment
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The payment that this transaction matched to"}
])
(def user-schema
[{:db/ident :user/original-id
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "The id in the old system"}
{:db/ident :user/provider
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Provider for oauth for the user"}
{:db/ident :user/provider-id
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "the id from the provider"}
{:db/ident :user/role
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/one
:db/doc "The role [:user :admin :none]"}
{:db/ident :user/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "name of the user"}
{:db/ident :user/clients
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db/doc "The clients this user can view"}
;;enums
{:db/ident :user-role/admin}
{:db/ident :user-role/user}
{:db/ident :user-role/none}
])
(def base-schema
[ address-schema contact-schema vendor-schema client-schema bank-account-schema invoice-schema invoice-expense-account-schema payment-schema invoice-payment-schema transaction-schema user-schema])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-vendors [_]
[[]])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-clients [_]
[[]])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-invoices [_]
[[]])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-payments [_]
[[]])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-invoices-payments [_]
[[]])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-invoices-expense-accounts [_]
[[]])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-transactions [_]
[[]])
#_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
(defn migrate-users [_]
[[]])
(defn merge-query [query-part-1 query-part-2]
(-> query-part-1
(update-in [:query :find] into (get-in query-part-2 [:query :find]))
(update-in [:query :in] into (get-in query-part-2 [:query :in]))
(update-in [:query :where] into (get-in query-part-2 [:query :where]))
(update-in [:args] into (get-in query-part-2 [:args]))))
(defn add-sorter-fields [q sort-map args]
(reduce
(fn [q {:keys [sort-key] :as z}]
(merge-query q
{:query {:find [(symbol (str "?sort-" sort-key))]
:where (sort-map
sort-key
(println "Warning, trying to sort by unsupported field" sort-key, "sort map" (pr-str sort-map)))}}))
q
(:sort args)))
(defn add-sorter-fields-2 [q sort-map args]
(reduce
(fn [q {:keys [sort-key]}]
(merge-query q
{:query {:find [(last (last (sort-map
sort-key
(println "Warning, trying to sort by unsupported field" sort-key))))]
:where (sort-map
sort-key
(println "Warning, trying to sort by unsupported field" sort-key))}}))
q
(:sort args)))
(defn apply-sort-3 [args results]
(let [sort-bys (conj (into [] (:sort args))
{:sort-key "default" :asc (if (contains? args :default-asc?)
(:default-asc? args)
true)})
length (count sort-bys)
comparator (fn [xs ys]
(reduce
(fn [_ i]
(let [comparison (if (:asc (nth sort-bys i))
(compare (nth xs i) (nth ys i))
(compare (nth ys i) (nth xs i)))]
(if (not= 0 comparison)
(reduced comparison)
0)))
0
(range length)))]
(sort comparator results)))
;; TODO replace COULD JUST BE SORT-3
(defn apply-sort-4 [args results]
(let [sort-bys (-> []
(into (:sort args))
(conj {:sort-key "default" :asc (if (contains? args :default-asc?)
(:default-asc? args)
true)})
(conj {:sort-key "e" :asc true}))
length (count sort-bys)
comparator (fn [xs ys]
(reduce
(fn [_ i]
(let [comparison (if (:asc (nth sort-bys i))
(compare (nth xs i) (nth ys i))
(compare (nth ys i) (nth xs i)))]
(if (not= 0 comparison)
(reduced comparison)
0)))
0
(range length)))]
(sort comparator results)))
(defn apply-pagination-raw [args results]
{:entries (->> results
(drop (or (:start args) 0))
(take (or (:count args) (:per-page args) default-pagination-size)))
:count (count results)})
(defn apply-pagination [args results]
{:ids (->> results
(drop (or (:start args) 0))
(take (or (:count args )
(:per-page args)
default-pagination-size))
(map last))
:count (count results)
:all-ids (map last results)})
(defn audit-transact-batch [txes id]
(let [batch-id (.toString (java.util.UUID/randomUUID))]
(reduce
(fn [full-tx batch]
(let [batch (conj (vec batch) {:db/id "datomic.tx"
:audit/user (str (:user/role id) "-" (:user/name id))
:audit/batch batch-id})
_ (mu/log ::transacting-batch
:batch batch-id
:count (count batch))
tx-result @(dc/transact-async conn batch)]
(cond-> full-tx
(:tx-data full-tx) (update :tx-data #(into % (:tx-data tx-result)))
(not (:tx-data full-tx)) (assoc :tx-data (vec (:tx-data tx-result)))
(not (:db-before full-tx)) (assoc :db-before (:db-before tx-result))
true (assoc :db-after (:db-after tx-result))
true (update :tempids merge (:tempids tx-result)))))
{}
(partition-all 200 txes))))
(defn audit-transact [txes id]
(try
@(dc/transact-async conn (conj txes {:db/id "datomic.tx"
:audit/user (str (:user/role id) "-" (:user/name id))}))
(catch Exception e
(mu/log ::transaction-error
:exception e
:level :error
:tx txes)
(throw e)
)))
(defn pull-many [db read ids ]
(->> (dc/q '[:find (pull ?e r)
:in $ [?e ...] r]
db
ids
read)
(map first)))
(defn pull-many-by-id [db read ids ]
(into {}
(map (fn [[e]]
[(:db/id e) e]))
(dc/q '[:find (pull ?e r)
:in $ [?e ...] r]
db
ids
read)))
(defn random-tempid []
(str (UUID/randomUUID)))
(defn pull-id [db id]
(if (sequential? id)
(ffirst (dc/q '[:find ?i
:in $ [?a ?v]
:where [?i ?a ?v]]
db
id))
id))
(defn pull-attr [db k id]
(get (dc/pull db [k] id) k))
(defn pull-ref [db k id]
(:db/id (pull-attr db k id)))
#_(comment
(dc/pull (dc/db conn) '[*] 175921860633685)
(upsert-entity (dc/db conn) {:db/id 175921860633685 :invoice/invoice-number nil :invoice/date #inst "2021-01-01" :invoice/expense-accounts [:reset-rels [{:db/id "new" :invoice-expense-account/amount 1}]]})
(upsert-entity (dc/db conn) {:invoice/client #:db{:id 79164837221949},
:invoice/status #:db{:id 101155069755470, :ident :invoice-status/paid},
:invoice/due #inst "2020-12-23T08:00:00.000-00:00",
:invoice/invoice-number "12648",
:invoice/import-status
:import-status/imported,
:invoice/vendor nil,
:invoice/date #inst "2020-12-16T08:00:00.000-00:00",
:entity/migration-key 17592234924273,
:db/id 175921860633685,
:invoice/outstanding-balance 0.0,
:invoice/expense-accounts
[{:entity/migration-key 17592234924274,
:invoice-expense-account/location nil
:invoice-expense-account/amount 360.0,
:invoice-expense-account/account #:db{:id 92358976759248}}],})
#_(dc/pull (dc/db conn) auto-ap.datomic.clients 79164837221904)
(upsert-entity (dc/db conn) {:client/name "20Twenty - WG Development LLC",
:client/square-locations
[{:db/id 83562883711605,
:entity/migration-key 17592258901782,
:square-location/square-id "L2579ATQ0X1ET",
:square-location/name "20Twenty",
:square-location/client-location "WG"}],
:client/square-auth-token
"EAAAEEr749Ea6AdPTdngsmUPwIM3ETbPwcx3QQl_NS0KWuIL-JNzAg4f3W9DGQhb",
:client/bank-accounts
[{:bank-account/sort-order 2,
:bank-account/include-in-reports true,
:bank-account/number "3467",
:bank-account/code "20TY-WFCC3467",
:bank-account/locations ["WG"],
:entity/migration-key 17592245102834,
:bank-account/current-balance 11160.289999999979,
:bank-account/name "Wells Fargo CC - 3467",
:db/id 83562883732805,
:bank-account/start-date #inst "2021-12-01T08:00:00.000-00:00",
:bank-account/visible true,
:bank-account/type
#:db{:id 101155069755504, :ident :bank-account-type/credit},
:bank-account/intuit-bank-account #:db{:id 105553116286744},
:bank-account/integration-status
{:db/id 74766790691480,
:entity/migration-key 17592267080690,
:integration-status/last-updated #inst "2022-08-23T03:47:44.892-00:00",
:integration-status/last-attempt #inst "2022-08-23T03:47:44.892-00:00",
:integration-status/state
#:db{:id 101155069755529, :ident :integration-state/success}},
:bank-account/bank-name "Wells Fargo"}
{:bank-account/sort-order 0,
:bank-account/include-in-reports true,
:bank-account/numeric-code 11301,
:bank-account/check-number 301,
:bank-account/number "1734742859",
:bank-account/code "20TY-WF2882",
:bank-account/locations ["WG"],
:bank-account/bank-code "11-4288/1210 4285",
:entity/migration-key 17592241193004,
:bank-account/current-balance -47342.54000000085,
:bank-account/name "Wells Fargo Main - 2859",
:db/id 83562883732846,
:bank-account/start-date #inst "2021-12-01T08:00:00.000-00:00",
:bank-account/visible true,
:bank-account/type
#:db{:id 101155069755468, :ident :bank-account-type/check},
:bank-account/intuit-bank-account #:db{:id 105553116286745},
:bank-account/routing "121042882",
:bank-account/integration-status
{:db/id 74766790691458,
:entity/migration-key 17592267080255,
:integration-status/last-updated #inst "2022-08-23T03:46:45.879-00:00",
:integration-status/last-attempt #inst "2022-08-23T03:46:45.879-00:00",
:integration-status/state
#:db{:id 101155069755529, :ident :integration-state/success}},
:bank-account/bank-name "Wells Fargo"}
{:bank-account/sort-order 1,
:bank-account/include-in-reports true,
:bank-account/numeric-code 20101,
:bank-account/yodlee-account-id 27345526,
:bank-account/number "41006",
:bank-account/code "20TY-Amex41006",
:bank-account/locations ["WG"],
:entity/migration-key 17592241193006,
:bank-account/current-balance 9674.069999999963,
:bank-account/name "Amex - 41006",
:db/id 83562883732847,
:bank-account/visible true,
:bank-account/type
#:db{:id 101155069755504, :ident :bank-account-type/credit},
:bank-account/bank-name "American Express"}
{:bank-account/sort-order 3,
:bank-account/include-in-reports true,
:bank-account/numeric-code 11101,
:bank-account/code "20TY-0",
:bank-account/locations ["WG"],
:entity/migration-key 17592241193005,
:bank-account/current-balance 0.0,
:bank-account/name "CASH",
:db/id 83562883732848,
:bank-account/visible true,
:bank-account/type
#:db{:id 101155069755469, :ident :bank-account-type/cash}}],
:entity/migration-key 17592241193003,
:db/id 79164837221904,
:client/address
{:db/id 105553116285906,
:entity/migration-key 17592250661126,
:address/street1 "1389 Lincoln Ave",
:address/city "San Jose",
:address/state "CA",
:address/zip "95125"},
:client/code "NY",
:client/locations ["WE" "NG"],
:client/square-integration-status
{:db/id 74766790691447,
:entity/migration-key 17592267072653,
:integration-status/last-updated #inst "2022-08-23T13:09:16.082-00:00",
:integration-status/last-attempt #inst "2022-08-23T13:08:47.018-00:00",
:integration-status/state
#:db{:id 101155069755529, :ident :integration-state/success}}})
)
(defn install-functions []
@(dc/transact conn
(edn/read-string {:readers {'db/id id-literal
'db/fn construct}} (slurp (io/resource "functions.edn")))))
(defn all-schema []
(edn/read-string (slurp (io/resource "schema.edn"))))
(defn transact-schema
"Installs the schema in two passes: every plain attribute first, then every composite tuple.
A tuple can only be created once the attributes it composes already exist, and the pieces are
spread across both files — `:journal-entry-line/running-balance-tuple` lives in schema.edn
while one of its members, `:journal-entry-line/running-balance`, lives in
cloud-migration-schema.edn. Transacting the files in order therefore cannot install that tuple
against an empty database. Long-lived databases never hit it because those attributes went in
years apart."
[conn]
(let [schema (concat (edn/read-string (slurp (io/resource "schema.edn")))
;; this is temporary for any new stuff that needs to be asserted for cloud migration.
(edn/read-string (slurp (io/resource "cloud-migration-schema.edn"))))
{tuples true plain false} (group-by #(contains? % :db/tupleAttrs) schema)]
(when (seq plain) @(dc/transact conn plain))
(when (seq tuples) @(dc/transact conn tuples))))
(defn backoff [n]
(let [base-timeout 500
max-timeout 300000 ; 5 minutes
max-retries 10
backoff-time (* base-timeout (Math/pow 2 (min n max-retries)))]
(min (+ backoff-time (rand-int base-timeout)) max-timeout)))
(defn transact-with-backoff
([tx ] (transact-with-backoff tx 0))
([tx attempt]
(try
@(dc/transact conn tx)
(catch Exception e
(if (< attempt 10)
(do
(Thread/sleep (backoff attempt))
(mu/log ::transact-failed
:exception e
:status "WARN")
(transact-with-backoff tx (inc attempt)))
(throw e))))))
(defn visible-clients [id]
(cond
(= (:user/role id) "none")
#{}
(= (:user/role id) "admin")
(->> (dc/q '[:find (pull ?e [:db/id :client/code])
:in $
:where [?e :client/code]]
(dc/db conn))
(map first)
(map :db/id)
(into #{}))
(#{"manager" "user" "power-user"} (:user/role id))
(into #{}
(map :db/id (:user/clients id [])))))
(defn query2 [query]
(apply dc/q (:query query) (:args query)))
(defn observable-q [query]
nil)
(defn observable-query [query]
(mu/with-context {:query (pr-str (:query query))
:args (pr-str (:args query))}
(mu/trace ::query
[]
(let [query-results (dc/query {:query (:query query)
:args (:args query)
:query-stats true
:io-context ::hello})]
(alog/info ::query-stats
:io-stats (pr-str (:io-stats query-results))
:query-stats (pr-str (:query-stats query-results)))
(:ret query-results)))))
Reference in New Issue View Git Blame Copy Permalink