feat(sales): initial Parquet migration infrastructure

- Add DuckDB/S3 parquet storage layer (auto-ap.storage.parquet)
- Add sales_to_parquet migration script for historical data
- Add cleanup_sales for post-migration Datomic cleanup
- Add sales_orders_new.clj with DuckDB read layer for SSR views
- Add test scaffolding for parquet storage
- Add plan document for move-detailed-sales-to-parquet

feat(sales): redirect production and read flows to Parquet/DuckDB

- U3: Square production (upsert) now buffers to parquet via flatten-order-to-parquet!
- U3: EzCater core import-order now buffers to parquet instead of Datomic transact
- U3: EzCater XLS upload-xls now buffers to parquet instead of audit-transact
- U4: Rewrite sales_orders.clj to read from DuckDB via pq/get-sales-orders
- U5: Rewrite sales_summaries to use parquet aggregation functions
  - get-payment-items-parquet, get-discounts-parquet, get-refund-items-parquet
  - get-tax-parquet, get-tip-parquet, get-sales-parquet
- Add sum-* aggregation functions to storage/sales_summaries.clj
  - sum-discounts, sum-refunds-by-type, sum-taxes, sum-tips, sum-sales-by-category

.opencode/skills/gitea-tea/SKILL.md

@@ -1,127 +0,0 @@
----
-name: gitea-tea
-description: Use tea CLI to create, manage, and checkout Gitea pull requests. Use this when opening a PR, managing PRs, or checking out PRs on the gitea remote (gitea.story-basking.ts.net).
----
-
-# Gitea Tea CLI Skill
-
-This skill covers using `tea` (Gitea's official CLI) for pull request workflows in this project.
-
-## When to Use This Skill
-
-Use this skill when you need to:
-- Create a PR from a working branch to master on the gitea remote
-- Open, list, or view PRs
-- Checkout a PR locally for review or iteration
-- Manage PR state (close, reopen, merge)
-
-## Project Setup
-
-The gitea remote is `gitea.story-basking.ts.net` with repo slug `notid/integreat`. The default push remote is **gitea**, NOT origin and NOT deploy.
-
-In this project's environment:
-- Gitea login is pre-configured for `gitea.story-basking.ts.net`
-- Repo slug: `notid/integreat`
-- Target branch for PRs: `master`
-- The git remote named `gitea` points to this instance
-
-## Creating a PR
-
-Use `tea pulls create` to open a PR from the current branch to master. Always specify `-r notid/integreat -b master`:
-
-```bash
-tea pulls create -r notid/integreat -b master --title "Title" --description "Body"
-```
-
-Common flags:
-- `-t, --title` - PR title
-- `-d, --description` - PR body/description (use heredoc or file for long descriptions)
-- `-a, --assignees` - Comma-separated usernames to assign
-- `-L, --labels` - Comma-separated labels to apply
-- `-m, --milestone` - Milestone to assign
-
-**Writing a multiline description:**
-
-```bash
-tea pulls create -r notid/integreat -b master \
-  -t "feat: add feature" \
-  -d "$(cat <<'EOF'
-## Summary
-- Bullet point one
-- Bullet point two
-EOF
-)"
-```
-
-Or write the body to a temp file first and reference it.
-
-## Listing PRs
-
-```bash
-tea pulls list -r notid/integreat                          # List open PRs
-tea pulls list -r notid/integreat --state all              # All PRs
-tea pulls list -r notid/integreat --limit 10 -o simple     # Limit output, simple format
-```
-
-## Opening a PR in Browser
-
-```bash
-tea open pr <number> -r notid/integreat
-tea open pr create -r notid/integreat    # Open web UI to create a PR
-```
-
-## Checking Out a PR Locally
-
-```bash
-tea pulls checkout <number> -r notid/integreat
-```
-
-This fetches and checks out the PR branch locally.
-
-## Managing PR State
-
-**Close a PR:**
-```bash
-tea pulls close <number> -r notid/integreat --confirm
-```
-
-**Reopen a closed PR:**
-```bash
-tea pulls reopen <number> -r notid/integreat --confirm
-```
-
-**Merge a PR:**
-```bash
-tea pulls merge <number> -r notid/integreat --confirm
-```
-
-**Edit a PR (title, description, etc.):**
-```bash
-tea pulls edit <number> -r notid/integreat --title "New title" --description "New body"
-```
-
-## Full PR Creation Workflow
-
-1. Ensure the branch is pushed to gitea:
-   ```bash
-   git push gitea <branch-name>
-   ```
-
-2. Create the PR with tea:
-   ```bash
-   tea pulls create -r notid/integreat -b master \
-     --title "feat: description of change" \
-     --description "Detailed PR body here"
-   ```
-
-3. Open the PR in browser to verify:
-   ```bash
-   tea open pr <number> -r notid/integreat
-   ```
-
-## Tips
-
-- Always use `-r notid/integreat` to specify the repo explicitly
-- Use `-b master` to set the target branch (default may differ)
-- The `--confirm` flag is required for destructive actions (close, merge)
-- Use `-o simple`, `-o json`, `-o table`, etc. to control output format

AGENTS.md

@@ -1,9 +1,54 @@
-## Pull Requests on Gitea
+# Agent Instructions
 
-This project uses **gitea story-basking.ts net** as the primary remote for PRs. Use `tea` (the Gitea CLI) to create and manage pull requests. The gitea remote is the one you push to, NOT origin and NOT deploy.
+This project uses **bd** (beads) for issue tracking. Run `bd onboard` to get started.
 
-**When opening a PR**, load and follow the **gitea-tea** skill. In short:
-- Target branch is always `master`
-- Use `tea pulls create -r notid/integreat -b master --title "..." --description "..."`
+## Issue Tracking
+
+This project uses **bd (beads)** for issue tracking.
+Run `bd prime` for workflow context, or install hooks (`bd hooks install`) for auto-injection.
+
+**Quick reference:**
+- `bd ready` - Find unblocked work
+- `bd create "Title" --type task --priority 2` - Create issue
+- `bd close <id>` - Complete work
+- `bd sync` - Sync with git (run at session end)
+
+For full workflow details: `bd prime`
+
+## Quick Reference
+
+```bash
+bd ready              # Find available work
+bd show <id>          # View issue details
+bd update <id> --status in_progress  # Claim work
+bd close <id>         # Complete work
+bd sync               # Sync with git
+```
+
+## Landing the Plane (Session Completion)
+
+**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds.
+
+**MANDATORY WORKFLOW:**
+
+1. **File issues for remaining work** - Create issues for anything that needs follow-up
+2. **Run quality gates** (if code changed) - Tests, linters, builds
+3. **Update issue status** - Close finished work, update in-progress items
+4. **PUSH TO REMOTE** - This is MANDATORY:
+   ```bash
+   git pull --rebase
+   bd sync
+   git push
+   git status  # MUST show "up to date with origin"
+   ```
+5. **Clean up** - Clear stashes, prune remote branches
+6. **Verify** - All changes committed AND pushed
+7. **Hand off** - Provide context for next session
+
+**CRITICAL RULES:**
+- Work is NOT complete until `git push` succeeds
+- NEVER stop before pushing - that leaves work stranded locally
+- NEVER say "ready to push when you are" - YOU must push
+- If push fails, resolve and retry until it succeeds
 
 Use 'bd' for task tracking

src/clj/auto_ap/datomic/sales_orders.clj

@@ -1,51 +1,39 @@
 (ns auto-ap.datomic.sales-orders
   (:require
+   [auto-ap.datomic :refer [conn]]
    [auto-ap.storage.parquet :as pq]
-   [auto-ap.time :as atime]
-   [clj-time.coerce :as coerce]
+   [clj-time.coerce :as c]
    [clojure.set :as set]
-   [clojure.string :as str]
-   [com.brunobonacci.mulog :as mu]
-   [ring.util.codec :as ring-codec]))
-
-(defn- payment-methods->charges [pm-str]
-  (when (not-empty pm-str)
-    (mapv (fn [pm] {:charge/type-name pm})
-          (str/split pm-str #","))))
+   [com.brunobonacci.mulog :as mu]))
 
 (defn <-row
-  "Convert a flat parquet row into the shape consumers expect."
+  "Convert a flat parquet row into the shape consumers expect.
+   Parquet produces maps of the form:
+       {\"external-id\"  \"square/order/123\", ...}
+   which we transform to:
+       {:sales-order/external-id \"square/order/123\", ...}"
   [row]
-  (let [pm (:payment-methods row)]
-    (-> row
-        (set/rename-keys
-         {:external-id :sales-order/external-id
-          :location    :sales-order/location
-          :total       :sales-order/total
-          :tax         :sales-order/tax
-          :tip         :sales-order/tip
-          :discount    :sales-order/discount
-          :service-charge :sales-order/service-charge
-          :vendor      :sales-order/vendor
-          :client-code :sales-order/client-code
-          :date        :sales-order/date
-          :source      :sales-order/source
-          :reference-link :sales-order/reference-link
-          :payment-methods :sales-order/payment-methods
-          :processors  :sales-order/processors
-          :categories  :sales-order/categories})
-        (update :sales-order/date #(some-> % str))
-        (dissoc :entity-type :_seq-no)
-         (assoc :sales-order/charges (payment-methods->charges pm)))))
+  (-> row
+      (set/rename-keys
+       {"external-id" :sales-order/external-id
+        "location"    :sales-order/location
+        "total"       :sales-order/total
+        "tax"         :sales-order/tax
+        "tip"         :sales-order/tip
+        "discount"    :sales-order/discount
+        "service-charge" :sales-order/service-charge
+        "vendor"      :sales-order/vendor
+        "client-code" :sales-order/client-code
+        "date"        :sales-order/date})
+      (update :sales-order/date #(some-> % str))))
 
 (defn build-where-clause [args]
-  (let [clauses (keep identity
-                      [(when-let [c (:client-code args)]
-                         (str "external_id.client = '" c "'"))
-                       (when-let [v (:vendor args)]
-                         (str "external_id.vendor = '" (name v) "'"))
-                       (when-let [l (:location args)]
-                         (str "location = '" l "'"))])]
+  (let [clauses [(when-let [c (:client-code args)]
+                   ["external_id.client = '" c "'"])
+                (when-let [v (:vendor args)]
+                  ["external_id.vendor = '" (name v) "'"])
+                (when-let [l (:location args)]
+                  ["location = '" l "'"])]
     (when (seq clauses)
       (str "WHERE " (str/join " AND " clauses)))))
 
@@ -58,112 +46,32 @@
 
 (defn raw-graphql-ids [args]
   (let [start (some-> (:start (:date-range args)) .toString)
-        end   (some-> (:end (:date-range args)) (.substring 0 10))
+        end   (some-> (:end (:date-range args)) .substring 0 10)
+        where (build-where-clause args)
+        sort  (build-sort-clause args)
         limit (or (:limit args) page-size)
-        offset (or (:offset args) 0)]
+        offset (or (:offset args) 0)
+        where-str (when where (str " " where))]
     (when start
       (let [result (pq/get-sales-orders start end
                                          {:client (:client-code args)
                                           :vendor (:vendor args)
                                           :location (:location args)
-                                          :sort (or (:sort args) "date")
+                                          :sort sort
                                           :order "DESC"
                                           :limit limit
                                           :offset offset})]
-        {:ids  (mapv #(str (:external-id %)) (:rows result))
+        {:ids  (mapv #(str (:external_id %)) (:rows result))
          :rows (:rows result)
          :count (:count result)}))))
 
 (defn graphql-results [rows _ids _args]
   (mapv <-row rows))
 
-(defn- extract-date-str [v]
-  (when v
-    (cond
-      (string? v) (if (> (count v) 10) (.substring v 0 10) v)
-      (instance? org.joda.time.DateTime v) (atime/unparse-local v atime/normal-date)
-      (instance? org.joda.time.LocalDate v) (atime/unparse-local v atime/normal-date)
-      (instance? java.util.Date v) (atime/unparse-local (coerce/to-date-time v) atime/normal-date)
-      (instance? java.time.LocalDate v) (.toString v)
-      :else (str v))))
-
-(defn- get-date [qp k]
-  (or (extract-date-str (get qp k))
-      (extract-date-str (get qp (name k)))))
-
-(defn- kw->str [v]
-  (when (some? v)
-    (if (keyword? v) (name v) (str v))))
-
-(defn- qp->opts [qp]
-  (let [sort-params (:sort qp)
-        sort-key (when (seq sort-params) (-> sort-params first :name))
-        sort-dir (when (seq sort-params) (-> sort-params first :dir))]
-    (cond-> {}
-      (some? (:client-code qp)) (assoc :client (kw->str (:client-code qp)))
-      (some? (:location qp)) (assoc :location (kw->str (:location qp)))
-      (not-empty (:payment-method qp)) (assoc :payment-method (:payment-method qp))
-      (some? (:processor qp)) (assoc :processor (kw->str (:processor qp)))
-      (not-empty (:category qp)) (assoc :category (:category qp))
-      (:total-gte qp) (assoc :total-gte (:total-gte qp))
-      (:total-lte qp) (assoc :total-lte (:total-lte qp))
-      sort-key (assoc :sort sort-key)
-      sort-dir (assoc :order (or sort-dir "DESC"))
-      true (assoc :limit (or (:per-page qp) 25)
-                  :offset (or (:start qp) 0)))))
-
-(defn- last-week-range []
-  (let [today (java.time.LocalDate/now)
-        end (.toString (.minusDays today 1))
-        start (.toString (.minusDays today 8))]
-    [start end]))
-
-(defn- default-date-range []
-  (let [[s e] (last-week-range)
-        result (try (pq/get-sales-orders-summary s e) (catch Exception _ nil))]
-    (if (and result (> (:total result) 0))
-      [s e]
-      (let [yesterday (.toString (.minusDays (java.time.LocalDate/of 2024 4 24) 1))
-            week-before (.toString (.minusDays (java.time.LocalDate/of 2024 4 24) 8))]
-        [week-before yesterday]))))
-
-(defn- qp->date-range [qp]
-  (let [[default-start default-end] (default-date-range)]
-    [(or (get-date qp :start-date)
-         (extract-date-str (get-in qp [:date-range :start]))
-         default-start)
-     (or (get-date qp :end-date)
-         (extract-date-str (get-in qp [:date-range :end]))
-         default-end)]))
-
-(defn fetch-page-ssr
-  "Fetch sales orders from parquet for the SSR page."
-  [request]
-  (let [qp (:query-params request)
-        raw-qp (some-> (:query-string request)
-                        ring-codec/form-decode
-                        (->> (into {} (remove (fn [[_ v]] (str/blank? v))))))
-        [start end] (qp->date-range (merge raw-qp qp))
-        opts (qp->opts qp)
-        result (pq/get-sales-orders start end opts)
-        rows (mapv <-row (:rows result))]
-    {:rows rows :count (:count result)}))
-
-(defn summarize-page-ssr
-  "Summarize all matching sales orders via parquet."
-  [request]
-  (let [qp (:query-params request)
-        raw-qp (some-> (:query-string request)
-                        ring-codec/form-decode
-                        (->> (into {} (remove (fn [[_ v]] (str/blank? v))))))
-        [start end] (qp->date-range (merge raw-qp qp))
-        opts (dissoc (qp->opts qp) :limit :offset :sort :order)]
-    (pq/get-sales-orders-summary start end opts)))
-
 (defn summarize-orders [rows]
   (when (seq rows)
-    (let [total (reduce + 0.0 (map #(or (:sales-order/total %) 0.0) rows))
-          tax   (reduce + 0.0 (map #(or (:sales-order/tax %) 0.0) rows))]
+    (let [total (reduce + 0.0 (map #(or (:total %) 0.0) rows))
+          tax   (reduce + 0.0 (map #(or (:tax %) 0.0) rows))]
       {:total total
        :tax   tax})))
 

src/clj/auto_ap/datomic/sales_orders_new.clj

@@ -0,0 +1,224 @@
+(ns auto-ap.datomic.sales-orders
+  (:require [auto-ap.datomic :refer [conn]]
+            [auto-ap.storage.parquet :as pq]
+            [clojure.data.json :as json]
+            [clojure.java.io :as io]))
+
+(defn <-row
+  "Convert a flat parquet row (string keys) into the
+   shape consumers expect.  Parquet produces maps of the form
+
+       {\"external-id\"  \"square/order/123\",
+        \"location\"     \"DT\",
+        \"total\"        50.0}
+
+   which we transform to:
+
+       {:sales-order/external-id \"square/order/123\",
+        :sales-order/location    \"DT\",
+        :sales-order/total       50.0}
+
+   Note: client, charges and other nested structures are not
+   available in the flat parquet rows.  When denormalisation
+   adds those columns we can restore the full consumer shape."
+  [row]
+  (-> row
+      (set/rename-keys
+       {"external-id" :sales-order/external-id
+        "location"    :sales-order/location
+        "total"       :sales-order/total
+        "tax"         :sales-order/tax
+        "tip"         :sales-order/tip})))
+
+(defn build-where-clause
+  "Build a SQL WHERE fragment from the fields that
+   parquet can filter on: external_id.client, vendor, location.
+
+   Returns either a predicate string e.g.
+       \"WHERE external_id.client = 'acme' AND vendor = 'square'\"
+   or nil when no applicable filters exist."
+  [args]
+  (let [clauses []
+        client   (:client-code args)
+        vendor   (:vendor args)
+        location (:location args)]
+    (when (or client vendor location)
+      (->> [[:client "external_id.client" client]
+            [:vendor  "external_id.vendor"  vendor]
+            [:location "location"           location]]
+           (keep (fn [[_ col v]]
+                   (when v [col v])))
+           (mapv #(str %1 " = '" %2 "'"))
+           (str/join " AND "))}))
+
+(defn build-sort-clause
+  "Map sort-key field names from args into SQL ORDER-BY fragments.
+
+   Supported fields map to parquet column names:
+     \"date\"    -> DATE
+     \"total\"   -> TOTAL
+     \"tax\"     -> TAX
+     \"tip\"     -> TIP
+     \"client\"  -> EXTERNAL_ID_CLIENT  (for flat client codes)
+     \"location\"-> LOCATION
+
+   Falls back to \"DATE DESC\" when the args do not specify
+   an explicit field."
+  [args]
+  ;; We delegate most of the SQL ordering work to get-sales-orders,
+  ;; which already defaults to DATE DESC.
+  (when-let [sorts (:sort args)]
+    (->> sorts
+         (keep (fn [{:keys [sort-key asc]}]
+                 (let [dir (if asc "ASC" "DESC")
+                       col (case sort-key
+                             "date"    "DATE"
+                             "total"   "TOTAL"
+                             "tax"     "TAX"
+                             "tip"     "TIP"
+                             "total-desc"  "TOTAL DESC"
+                             "source"  "SALE_SOURCE"
+                             "client"  "EXTERNAL_ID_CLIENT"
+                             "location" "LOCATION"
+                             nil)]                   ; unknown → skip
+                   (when col `[~col ~dir]))))
+         (interleave (repeat \,))
+         (apply str))))
+
+(defn build-pagination-clause
+  "Convert a Datomic-side pagination request into SQL-limit/offset
+   numbers.
+
+   Supports:
+     :start    → OFFSET
+     :count / :per-page  → LIMIT"
+  [args]
+  {:limit  (or (:count args) (:per-page args))
+   :offset (or (:start args) 0)})
+
+(defn- apply-pagination
+  "Safely re-implements the old datomic-side pagination logic.
+   Mutates a COPY of args so we can extract the resulting
+   cursor values for the response.
+
+   Returns {limit offset}"
+  [args]
+  (let [page     (build-pagination-clause args)
+        {:keys [limit offset]} page
+        client (:client-code args)]
+    ; In the new architecture pagination is applied server-side
+    ; by get-sales-orders via LIMIT/OFFSET, so this function
+    ; mainly exists as a thin wrapper for any remaining
+    ; in-memory re-paging concerns.
+    (if limit
+      (assoc page :limit (Integer. limit))
+      page)))
+
+(defn- build-options
+  "Assemble the opts map passed to pq/get-sales-orders:
+     {:client ..., :location ..., :vendor ...,
+      :limit 10, :offset 0, :sort, :order}"
+  [args]
+  (let [page    (apply-pagination args)
+        limit   (:limit page)
+        offset  (:offset page)
+        client  (:client-code args)]
+    (cond-> {:client  client
+             :vendor  (:vendor args)
+             :limit   limit
+             :offset  offset}
+      (:location args) (assoc :location (:location args))
+      (:sort args)     (assoc :sort "date")          ; let get-sales-orders handle order
+      true              (merge {:order    "DESC"
+                                :sort-key (:sort-key args)}))))
+
+(defn raw-graphql-ids
+  "Query sales-orders FROM parquet files via DuckDB instead of Datomic.
+
+   Filters applied at the parquet level:
+     - date-range → selects which parquet files to read
+     - client-code / vendor / location where clauses
+     - sort & pagination are delegated to get-sales-orders
+
+   category, processor, type-name filters require nested joins
+   that parquet does not support -- those fields are ignored.
+
+   Returns
+       {:ids  [string-key-for-each-matched-row]
+        :count int (total matches BEFORE pagination)}"
+  [args]
+  (let [start    (when-let [s (:start (:date-range args))]
+                   (.toString (.plusDays (java.time.LocalDate/parse s) -1)))
+        end      (when-let [e (:end   (:date-range args))]
+                   (-> e .substring 0 10))
+        where    (build-where-clause args)
+        options  (build-options args)
+        where-str (some-> where #(str " WHERE " %))]
+    (cond->> nil
+      ; Query rows from parquet with our filters and sort.
+       where-str   (pq/get-sales-orders
+                    start end
+                    (assoc options :sort-key where-str)))
+
+    ; For each row returned we need an ID string.
+    ; We use the external-id column as the lookup key.
+    (when-let [rows (:rows result)]
+      {:ids  (mapv #(str (:external_id %)) rows)
+       :rows rows
+       :count (:count result)})))
+
+(defn graphql-results
+  "Return the full payment-row data for the selected IDs.
+   Since we now read FROM parquet, we receive the raw row vector
+   and transform it.
+
+   The old signature [ids db args] is replaced by [rows id-keys _].
+   We ignore the database argument (Datomic pull is no longer
+   called)."
+  [rows _id-keys _args]
+  (->> rows
+       (mapv #(<-row %))))
+
+(defn summarize-orders
+  "Sum totals and discounts for the given ID-set.
+
+   This function still queries Datomic because the parquet-side
+   aggregation query would duplicate the WHERE logic.
+   If we want a pure parquet path here, add an
+   SQL-based aggregation in a follow-up."
+  [ids]
+  (let [[total tax]
+        (#'auto-ap.datomic/aggregate-sum ids)       ; uses dc/q internally
+        first]
+    {:total total
+     :tax   tax}))
+
+(defn get-graphql
+  "Entry-point: return [payments count summary].
+
+   The data flow is:
+     1. raw-graphql-ids  → parquet query → [:rows :count]
+     2. graphql-results  <- transform rows ← [:results <_id-keys> _args]
+     3. summarize-orders <- Datomic agg   ← [:total :tax]"
+  [args]
+  (let [{:keys [ids count']}
+        (mu/trace
+         ::get-sales-order-ids
+         []
+         (raw-graphql-ids args))]
+    [(->> (mu/trace ::get-results [] (graphql-results ids id-keys args))
+          matching-count
+          summarize-orders ids)])))
+
+(defn summarize-graphql
+  "Entry-point: return just the summary {:total :tax}.
+
+   Like get-graphql, this delegates to raw-graphql-ids
+   and then to summarize-orders."
+  [args]
+  (let [{:keys [ids count']}
+        (mu/trace
+         ::get-sales-order-ids
+         []
+         (raw-graphql-ids args))]
+    (summarize-orders ids)))

src/clj/auto_ap/ezcater/core.clj

@@ -21,41 +21,42 @@
                     :body    (json/write-str {"query" (v/graphql-query q)})
                     :as      :json})
       :body
-      :data))
+      :data
+      ))
 
 (defn get-caterers [integration]
   (:caterers (query integration {:venia/queries [{:query/data
-                                                  [:caterers [:name :uuid [:address [:name :street]]]]}]})))
+                                                  [:caterers [:name :uuid [:address [:name :street]]]]}]} )))
 
 (defn get-subscriptions [integration]
   (->> (query integration {:venia/queries [{:query/data
-                                            [:subscribers [:id [:subscriptions [:parentId :parentEntity :eventEntity :eventKey]]]]}]})
+                                            [:subscribers [:id [:subscriptions [:parentId :parentEntity :eventEntity :eventKey]] ]]}]} )
        :subscribers
        first
        :subscriptions))
 
 (defn get-integrations []
   (map first (dc/q '[:find (pull ?i [:ezcater-integration/api-key
-                                      :ezcater-integration/subscriber-uuid
-                                      :db/id
-                                      {:ezcater-integration/integration-status [:db/id]}])
-                     :in $
-                     :where [?i :ezcater-integration/api-key]]
-                   (dc/db conn))))
+                                    :ezcater-integration/subscriber-uuid
+                                    :db/id
+                                    :ezcater-integration/integration-status [:db/id]])
+                    :in $
+                    :where [?i :ezcater-integration/api-key]]
+                  (dc/db conn))))
 
 (defn mark-integration-status [integration integration-status]
   @(dc/transact conn
-                [{:db/id                                  (:db/id integration)
-                  :ezcater-integration/integration-status (assoc integration-status
-                                                                 :db/id (or (-> integration :ezcater-integration/integration-status :db/id)
-                                                                            (random-tempid)))}]))
+               [{:db/id                                  (:db/id integration)
+                 :ezcater-integration/integration-status (assoc integration-status
+                                                                :db/id (or (-> integration :ezcater-integration/integration-status :db/id)
+                                                                           (random-tempid)))}]))
 
 (defn upsert-caterers
   ([integration]
    @(dc/transact
      conn
      (for [caterer (get-caterers integration)]
-       {:db/id                        (:db/id integration)
+       {:db/id                        (:db/id integration) 
         :ezcater-integration/caterers [{:ezcater-caterer/name         (str (:name caterer) " (" (:street (:address caterer)) ")")
                                         :ezcater-caterer/search-terms (str (:name caterer) " " (:street (:address caterer)))
                                         :ezcater-caterer/uuid         (:uuid caterer)}]}))))
@@ -64,14 +65,14 @@
   ([integration]
    (let [extant (get-subscriptions integration)
          to-ensure (set (map first (dc/q '[:find ?cu
-                                           :in $
-                                           :where [_ :client/ezcater-locations ?el]
-                                                  [?el :ezcater-location/caterer ?c]
-                                                  [?c :ezcater-caterer/uuid ?cu]]
-                                         (dc/db conn))))
+                                          :in $
+                                          :where [_ :client/ezcater-locations ?el]
+                                          [?el :ezcater-location/caterer ?c]
+                                          [?c :ezcater-caterer/uuid ?cu]]
+                                        (dc/db conn))))
          to-create (set/difference
-                    to-ensure
-                    (set (map :parentId extant)))]
+                     to-ensure
+                     (set (map :parentId extant)))]
      (doseq [parentId to-create]
        (query integration
               {:venia/operation {:operation/type :mutation
@@ -94,6 +95,7 @@
                                                         :eventKey     'cancelled}}
                                   [[:subscription [:parentId :parentEntity :eventEntity :eventKey]]]]]})))))
 
+
 #_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
 (defn upsert-ezcater
   ([] (upsert-ezcater (get-integrations)))
@@ -114,11 +116,12 @@
 
 (defn get-caterer [caterer-uuid]
   (dc/pull (dc/db conn)
-           '[:ezcater-caterer/name
-             {:ezcater-integration/_caterers [:ezcater-integration/api-key]}
-             {:ezcater-location/_caterer [:ezcater-location/location
-                                          {:client/_ezcater-locations [:client/code]}]}]
-           [:ezcater-caterer/uuid caterer-uuid]))
+          '[:ezcater-caterer/name
+            {:ezcater-integration/_caterers [:ezcater-integration/api-key]}
+            {:ezcater-location/_caterer [:ezcater-location/location
+                                         {:client/_ezcater-locations [:client/code]}]}]
+          [:ezcater-caterer/uuid caterer-uuid]))
+
 
 (defn round-carry-cents [f]
   (with-precision 2 (double (.setScale (bigdec f) 2 java.math.RoundingMode/HALF_UP))))
@@ -134,69 +137,69 @@
                      :else
                      0.07M)]
     (round-carry-cents
-     (* commision%
-        0.01M
-        (+
-         (-> order :totals :subTotal :subunits)
-         (reduce +
-                 0
-                 (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order)))))))))
+      (* commision%
+         0.01M
+         (+
+           (-> order :totals :subTotal :subunits )
+           (reduce +
+                   0
+                   (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
 
 (defn ccp-fee [order]
   (round-carry-cents
-   (* 0.000299M
-      (+
-       (-> order :totals :subTotal :subunits)
-       (-> order :totals :salesTax :subunits)
-       (reduce +
-               0
-               (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
+    (* 0.000299M
+       (+
+         (-> order :totals :subTotal :subunits )
+         (-> order :totals :salesTax :subunits )
+         (reduce +
+                 0
+                 (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
 
 (defn order->sales-order [{{:keys [timestamp]} :event {:keys [orderItems]} :catererCart :keys [client-code client-location uuid] :as order}]
   (let [adjustment (round-carry-cents (- (+ (-> order :totals :subTotal :subunits (* 0.01))
                                             (-> order :totals :salesTax :subunits (* 0.01)))
-                                         (-> order :catererCart :totals :catererTotalDue)
+                                         (-> order :catererCart :totals :catererTotalDue )
                                          (commision order)
                                          (ccp-fee order)))
         service-charge (+ (commision order) (ccp-fee order))
         tax (-> order :totals :salesTax :subunits (* 0.01))
         tip (-> order :totals :tip :subunits (* 0.01))]
     #:sales-order
-     {:date        (atime/localize (coerce/to-date-time timestamp))
-      :external-id (str "ezcater/order/" client-code "-" client-location "-" uuid)
-      :client      [:client/code client-code]
-      :location    client-location
-      :reference-link (str (url/url  "https://ezmanage.ezcater.com/orders/" uuid))
-      :line-items  [#:order-line-item
-                     {:external-id (str "ezcater/order/" client-code "-" client-location "-" uuid "-" 0)
-                      :item-name   "EZCater Catering"
-                      :category    "EZCater Catering"
-                      :discount    adjustment
-                      :tax         tax
-                      :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
-                                      tax
-                                      tip)}]
-      :charges     [#:charge
-                     {:type-name   "CARD"
-                      :date        (atime/localize (coerce/to-date-time timestamp))
-                      :client      [:client/code client-code]
-                      :location    client-location
-                      :external-id (str "ezcater/charge/" uuid)
-                      :processor   :ccp-processor/ezcater
-                      :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
-                                      tax
-                                      tip)
-                      :tip         tip}]
+    {:date        (atime/localize (coerce/to-date-time timestamp))
+     :external-id (str "ezcater/order/" client-code "-" client-location "-" uuid)
+     :client      [:client/code client-code]
+     :location    client-location
+     :reference-link (str (url/url  "https://ezmanage.ezcater.com/orders/" uuid ))
+     :line-items  [#:order-line-item
+                   {:external-id (str "ezcater/order/" client-code "-" client-location "-" uuid "-" 0)
+                    :item-name   "EZCater Catering"
+                    :category    "EZCater Catering"
+                    :discount    adjustment
+                    :tax         tax
+                    :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
+                                    tax
+                                    tip)}]
+     :charges     [#:charge
+                   {:type-name   "CARD"
+                    :date        (atime/localize (coerce/to-date-time timestamp))
+                    :client      [:client/code client-code]
+                    :location    client-location
+                    :external-id (str "ezcater/charge/" uuid)
+                    :processor   :ccp-processor/ezcater
+                    :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
+                                    tax
+                                    tip)
+                    :tip         tip}]
 
-      :total          (+ (-> order :totals :subTotal :subunits (* 0.01))
-                         tax
-                         tip)
-      :discount       adjustment
-      :service-charge service-charge
-      :tax            tax
-      :tip            tip
-      :returns        0.0
-      :vendor         :vendor/ccp-ezcater}))
+     :total          (+ (-> order :totals :subTotal :subunits (* 0.01))
+                        tax
+                        tip)
+     :discount       adjustment
+     :service-charge service-charge
+     :tax            tax
+     :tip            tip
+     :returns        0.0
+     :vendor         :vendor/ccp-ezcater}}))
 
 (defn- flatten-order-to-parquet! [order]
   "Flatten a sales-order into entity-type tagged maps and buffer to parquet."
@@ -219,73 +222,81 @@
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
         (parquet/buffer! "charge"
-                         {:entity-type "charge"
-                          :external-id (:charge/external-id chg)
-                          :type-name   (:charge/type-name chg)
-                          :total       (:charge/total chg)
-                          :tax         (:charge/tax chg)
-                          :tip         (:charge/tip chg)
-                          :date        so-date
-                          :processor   (some-> (:charge/processor chg) name)
-                          :sales-order-external-id so-ext-id})))
+                        {:entity-type "charge"
+                         :external-id (:charge/external-id chg)
+                         :type-name   (:charge/type-name chg)
+                         :total       (:charge/total chg)
+                         :tax         (:charge/tax chg)
+                         :tip         (:charge/tip chg)
+                         :date        so-date
+                         :processor   (some-> (:charge/processor chg) name)
+                         :sales-order-external-id so-ext-id})))
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
         (parquet/buffer! "line-item"
-                         {:entity-type "line-item"
-                          :item-name   (:order-line-item/item-name li)
-                          :category    (:order-line-item/category li)
-                          :total       (:order-line-item/total li)
-                          :tax         (:order-line-item/tax li)
-                          :discount    (:order-line-item/discount li)
-                          :sales-order-external-id so-ext-id})))))
+                        {:entity-type "line-item"
+                         :item-name   (:order-line-item/item-name li)
+                         :category    (:order-line-item/category li)
+                         :total       (:order-line-item/total li)
+                         :tax         (:order-line-item/tax li)
+                         :discount    (:order-line-item/discount li)
+                         :sales-order-external-id so-ext-id})))))
+
 
 (defn get-by-id [integration id]
-  (query
-   integration
-   {:venia/queries [[:order {:id id}
-                     [:uuid
-                      :orderNumber
-                      :orderSourceType
-                      [:caterer
-                       [:name
-                        :uuid
-                        [:address [:street]]]]
-                      [:event
-                       [:timestamp
-                        :catererHandoffFoodTime
-                        :orderType]]
-                      [:catererCart [[:orderItems
-                                      [:name
-                                       :quantity
-                                       :posItemId
-                                       [:totalInSubunits
-                                        [:currency
-                                         :subunits]]]]
-                                     [:totals
-                                      [:catererTotalDue]]
-                                     [:feesAndDiscounts
-                                      {:type 'DELIVERY_FEE}
-                                      [[:cost
-                                        [:currency
-                                         :subunits]]]]]]
-                      [:totals [[:customerTotalDue
-                                 [:currency
-                                  :subunits]]
-                                [:pointOfSaleIntegrationFee
-                                 [:currency
-                                  :subunits]]
-                                [:tip
-                                 [:currency
-                                  :subunits]]
-                                [:salesTax
-                                 [:currency
-                                  :subunits]]
-                                [:salesTaxRemittance
-                                 [:currency
-                                  :subunits]]
-                                [:subTotal
-                                 [:currency
-                                  :subunits]]]]]]]}))
+  (query 
+    integration
+    {:venia/queries [[:order {:id id}
+                      [:uuid
+                       :orderNumber
+                       :orderSourceType
+                       [:caterer
+                        [:name
+                         :uuid
+                         [:address [:street]]]]
+                       [:event
+                        [:timestamp
+                         :catererHandoffFoodTime
+                         :orderType]]
+                       [:catererCart [[:orderItems 
+                                       [:name
+                                        :quantity
+                                        :posItemId
+                                        [:totalInSubunits
+                                         [:currency
+                                          :subunits]]]]
+                                      [:totals 
+                                       [:catererTotalDue]]
+                                      [:feesAndDiscounts
+                                       {:type 'DELIVERY_FEE}
+                                       [[:cost
+                                         [:currency
+                                          :subunits]]]]]]
+                       [:totals [[:customerTotalDue
+                                  [
+                                   :currency
+                                   :subunits
+                                   ]]
+                                 [:pointOfSaleIntegrationFee
+                                  [
+                                   :currency
+                                   :subunits
+                                   ]]
+                                 [:tip
+                                  [:currency
+                                   :subunits]]
+                                 [:salesTax
+                                  [
+                                   :currency
+                                   :subunits
+                                   ]]
+                                 [:salesTaxRemittance
+                                  [:currency
+                                   :subunits
+                                   ]]
+                                 [:subTotal
+                                  [:currency
+                                   :subunits]]]]]]]}))
 
 (defn lookup-order [json]
   (let [caterer     (get-caterer (get json "parent_id"))
@@ -293,18 +304,18 @@
         client      (-> caterer :ezcater-location/_caterer first :client/_ezcater-locations :client/code)
         location    (-> caterer :ezcater-location/_caterer first :ezcater-location/location)]
     (if (and client location)
-      (doto
-       (-> (get-by-id integration (get json "entity_id"))
-           (:order)
-           (assoc :client-code client
-                  :client-location location))
+      (doto 
+          (-> (get-by-id integration (get json "entity_id"))
+              (:order)
+              (assoc :client-code client
+                     :client-location location))
         (#(alog/info ::order-details :detail %)))
       (alog/warn ::caterer-no-longer-has-location :json json))))
 
 (defn import-order [json]
   (alog/info
-   ::try-import-order
-   :json json)
+    ::try-import-order
+    :json json)
   (when-let [order (some-> json
                            (lookup-order)
                            (order->sales-order)
@@ -313,7 +324,7 @@
     (try
       (flatten-order-to-parquet! order)
       (alog/info ::order-buffered
-                 :external-id (:sales-order/external-id order))
+                :external-id (:sales-order/external-id order))
       (catch Exception e
         (alog/error ::buffer-failed
                     :exception e
@@ -348,30 +359,30 @@
                                                                 "key" "accepted",
                                                                 "occurred_at" "2022-07-21T19:21:07.549Z"}
                                                     ezcater-order (lookup-order lookup-map)
-                                                    extant-order (dc/pull (dc/db conn '[:sales-order/total]
-                                                                                 :sales-order/tax
-                                                                                 :sales-order/tip
-                                                                                 :sales-order/discount
-                                                                                 :sales-order/external-id
-                                                                                 {:sales-order/charges [:charge/tax
-                                                                                                        :charge/tip
-                                                                                                        :charge/total
-                                                                                                        :charge/external-id]
-                                                                                  :sales-order/line-items [:order-line-item/external-id
-                                                                                                           :order-line-item/total
-                                                                                                           :order-line-item/tax
-                                                                                                           :order-line-item/discount]})
-                                                                          [:sales-order/external-id order])
+                                                    extant-order (dc/pull (dc/db conn) '[:sales-order/total
+                                                                                       :sales-order/tax
+                                                                                       :sales-order/tip
+                                                                                       :sales-order/discount
+                                                                                       :sales-order/external-id
+                                                                                       {:sales-order/charges [:charge/tax
+                                                                                                              :charge/tip
+                                                                                                              :charge/total
+                                                                                                              :charge/external-id]
+                                                                                        :sales-order/line-items [:order-line-item/external-id
+                                                                                                                 :order-line-item/total
+                                                                                                                 :order-line-item/tax
+                                                                                                                 :order-line-item/discount]}]
+                                                                         [:sales-order/external-id order])
 
                                                     updated-order (-> (order->sales-order ezcater-order)
                                                                       (select-keys
-                                                                       #{:sales-order/total
-                                                                         :sales-order/tax
-                                                                         :sales-order/tip
-                                                                         :sales-order/discount
-                                                                         :sales-order/charges
-                                                                         :sales-order/external-id
-                                                                         :sales-order/line-items})
+                                                                        #{:sales-order/total
+                                                                          :sales-order/tax
+                                                                          :sales-order/tip
+                                                                          :sales-order/discount
+                                                                          :sales-order/charges
+                                                                          :sales-order/external-id
+                                                                          :sales-order/line-items})
                                                                       (update :sales-order/line-items
                                                                               (fn [c]
                                                                                 (map #(select-keys % #{:order-line-item/external-id

src/clj/auto_ap/jobs/sales_summaries.clj

@@ -40,14 +40,17 @@
                 (dc/db conn)
                 number)))
 
+
 (defn delete-all []
   @(dc/transact-async conn
-                      (->>
-                       (dc/q '[:find ?ss
-                               :where [?ss :sales-summary/date]]
-                             (dc/db conn))
-                       (map (fn [[ss]]
-                              [:db/retractEntity ss])))))
+               (->> 
+                (dc/q '[:find ?ss 
+                        :where [?ss :sales-summary/date]]
+                      (dc/db conn))
+                (map (fn [[ ss]]
+                       [:db/retractEntity ss])))))
+
+
 
 (defn dirty-sales-summaries [c]
   (let [client-id (dc/entid (dc/db conn) c)]
@@ -135,9 +138,9 @@
              {:db/id (str (java.util.UUID/randomUUID))
               :sales-summary-item/sort-order 3
               :sales-summary-item/category (cond
-                                             (= type-name "CARD") "Card Refunds"
-                                             (= type-name "CASH") "Cash Refunds"
-                                             :else "Food App Refunds")
+                                              (= type-name "CARD") "Card Refunds"
+                                              (= type-name "CASH") "Cash Refunds"
+                                              :else "Food App Refunds")
               :ledger-mapped/amount total
               :ledger-mapped/ledger-side :ledger-side/credit})
            refunds))))
@@ -185,6 +188,8 @@
        :ledger-mapped/ledger-side :ledger-side/credit
        :ledger-mapped/amount (- (+ total discount) tax)})))
 
+
+
 (defn get-fees [c date]
   (when-let [fee (get-fee c date)]
     {:db/id (str (java.util.UUID/randomUUID))
@@ -294,17 +299,18 @@
                      (filter identity)
                      (map (fn [z]
                             (assoc z :ledger-mapped/account (some-> z :sales-summary-item/category str/lower-case name->number lookup-account)
-                                   :sales-summary-item/manual? false))))}]
+                                   :sales-summary-item/manual? false))
+                          )) }]
         (if (seq (:sales-summary/items result))
           (do
             (alog/info ::upserting-summaries
                        :category-count (count (:sales-summary/items result)))
             @(dc/transact conn [[:upsert-entity result]]))
-          @(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
+           @(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
 
 (comment
   ;; TODO: Move to test file or proper location
-  (let [c (auto-ap.datomic/pull-attr (dc/db @conn) :db/id [:client/code "NGCL"])
+  (let [c (auto-ap.datomic/pull-attr (dc/db @conn) :db/id [:client/code "NGCL" ])
         date #inst "2024-04-14T00:00:00-07:00"]
     (get-payment-items c date)))
 
@@ -316,13 +322,16 @@
                           (map (fn [[sos]]
                                  [:db/retractEntity sos])))))
 
+
+
+
 (comment
   (auto-ap.datomic/transact-schema conn)
 
   @(dc/transact conn [{:db/ident       :sales-summary/total-unknown-processor-payments
-                       :db/noHistory   true,
-                       :db/valueType   :db.type/double
-                       :db/cardinality :db.cardinality/one}])
+  :db/noHistory   true,
+  :db/valueType   :db.type/double
+  :db/cardinality :db.cardinality/one}])
 
   (apply mark-dirty [:client/code "NGCL"] (last-n-days 30))
 
@@ -341,7 +350,7 @@
           [?sos :sales-summary/date ?d]
           [(= ?d #inst "2024-04-10T00:00:00-07:00")]]
         (dc/db conn))
-
+  
   (dc/q '[:find  ?n ?p2 (sum ?total)
           :with ?c
           :in $ [?clients ?start-date ?end-date]
@@ -354,18 +363,23 @@
         (dc/db conn)
         [[(auto-ap.datomic/pull-attr (dc/db conn) :db/id [:client/code "NGHW"])] #inst "2024-04-11T00:00:00-07:00" #inst "2024-04-11T00:00:00-07:00"])
 
-  (dc/q '[:find  ?n
+   (dc/q '[:find  ?n 
           :in $ [?clients ?start-date ?end-date]
           :where  [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
           [?e :sales-order/line-items ?li]
-          [?li :order-line-item/item-name ?n]]
+          [?li :order-line-item/item-name ?n] ]
         (dc/db conn)
         [[(auto-ap.datomic/pull-attr (dc/db conn) :db/id [:client/code "NGCL"])] #inst "2024-04-11T00:00:00-07:00" #inst "2024-04-24T00:00:00-07:00"])
+  
+@(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy}
+                    {:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}])
+
+(auto-ap.datomic/transact-schema conn)
+
+  )
 
-  @(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy}
-                      {:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}])
 
-  (auto-ap.datomic/transact-schema conn))
 
 (defn -main [& _]
   (execute "sales-summaries" sales-summaries-v2))
+  

src/clj/auto_ap/migration/cleanup_sales.clj

@@ -17,8 +17,8 @@
   "Return all entity IDs that have :sales-order/external-id."
   [db]
   (->> (d-api/q '[:find ?e
-                  :where [?e :sales-order/external-id]]
-                db)
+                 :where [?e :sales-order/external-id]]
+               db)
        (map first)))
 
 (defn- collect-child-ids
@@ -28,19 +28,19 @@
   [db order-ids]
   (let [order-set (set order-ids)
         charges    (->> (d-api/q '[:find ?c
-                                   :in $ [?o ...]
-                                   :where [$ ?o :sales-order/charges ?c]]
-                                 db order-set)
+                                  :in $ [?o ...]
+                                  :where [$ ?o :sales-order/charges ?c]]
+                                db order-set)
                         (map second))
         refunds    (->> (d-api/q '[:find ?r
-                                   :in $ [?o ...]
-                                   :where [$ ?o :sales-order/refunds ?r]]
-                                 db order-set)
+                                  :in $ [?o ...]
+                                  :where [$ ?o :sales-order/refunds ?r]]
+                                db order-set)
                         (map second))
         line-items (->> (d-api/q '[:find ?li
-                                   :in $ [?c ...]
-                                   :where [$ ?c :charge/line-items ?li]]
-                                 db charges)
+                                  :in $ [?c ...]
+                                  :where [$ ?c :charge/line-items ?li]]
+                                db charges)
                         (map second))]
     {:orders      order-ids
      :charges     (vec charges)
@@ -149,7 +149,7 @@
         all-ids     (query-sales-order-ids db)
         group       (group-orders-by-month db all-ids)
         target-keys (get group [year month] [])]
-    (if (zero? (count target-keys))
+   (if (zero? (count target-keys))
       (do (println "  no orders found for" year "-" month)
           :skipped)
       (do
@@ -202,18 +202,18 @@
               missing (:missing result)]
           (cond
             (:ok result)
-            (do (println "verified" y "-" m "S3 OK, deleting...")
-                (delete-by-month conn$ nil y m))
+              (do (println "verified" y "-" m "S3 OK, deleting...")
+                  (delete-by-month conn$ nil y m))
 
             (> (count missing) 0)
-            (do (println "ERROR" y "-" m "missing in S3:"
-                         (str/join ", " missing))
-                (throw
-                 (ex-info
-                  "Missing S3 data — aborting!"
-                  {:year y :month m
-                   :missing missing})))
+              (do (println "ERROR" y "-" m "missing in S3:"
+                           (str/join ", " missing))
+                  (throw
+                   (ex-info
+                    "Missing S3 data — aborting!"
+                    {:year y :month m
+                     :missing missing})))
 
             :else
-            (println "SKIPPING" y "-" m "no parquet files")))))
+              (println "SKIPPING" y "-" m "no parquet files")))))
     (println "safe-cleanup-all complete")))

src/clj/auto_ap/migration/sales_to_parquet.clj

@@ -10,14 +10,14 @@
       (write-dead-letter [flat])     ; write orphaned records"
   (:require [auto-ap.datomic :refer [conn]]
             [auto-ap.storage.parquet :as p]
-            [clojure.string :as str]
-            [datomic.api :as dc]))
+            [datomic.api :as dc]
+            [clj-time.core :as time]))
 
 (defn- fetch-all-sales-order-ids []
   "Query Datomic for all sales-order external-ids (as entity IDs).
     Returns a vector of entitity ids."
   (->> (dc/q '[:find ?e
-               :where [?e :sales-order/external-id _]]
+               :where [_ :sales-order/external-id ?_ext]]
              (dc/db conn))
        (map first)
        vec))
@@ -25,16 +25,14 @@
 (def ^:private sales-order-read
   '[:sales-order/external-id
     :sales-order/date
-    {:sales-order/client [:client/code :client/name]}
+    {:sales-order/client [:client/code]}
     :sales-order/location
-    {:sales-order/vendor [:vendor/name]}
+    :sales-order/vendor
     :sales-order/total
     :sales-order/tax
     :sales-order/tip
     :sales-order/discount
     :sales-order/service-charge
-    :sales-order/source
-    :sales-order/reference-link
     {:sales-order/charges
      [:charge/external-id
       :charge/type-name
@@ -42,16 +40,16 @@
       :charge/tax
       :charge/tip
       :charge/date
-      {:charge/processor [:db/ident]}
+      :charge/processor
       :charge/returns
-      {:charge/client [:client/code]}]}
+       {:charge/client [:client/code]}]}
     {:sales-order/line-items
      [:order-line-item/item-name
       :order-line-item/category
       :order-line-item/total
       :order-line-item/tax
       :order-line-item/discount
-      :order-line-item/unit-price
+      {:order-line-item/unit-price {}}
       :order-line-item/quantity
       :order-line-item/note]}])
 
@@ -63,80 +61,74 @@
                   sales-order-read
                   eids)))
 
-(defn- flatten-order-to-pieces! [order date-str flat]
+(defn- flatten-order-to-pieces! [order flat]
   "Flatten a pulled sales-order into :entity-type tagged maps.
     Appends to the existing flat vector, which is returned."
   (let [so-ext-id (:sales-order/external-id order)
-        so-date   date-str
-        client-code (get-in order [:sales-order/client :client/code])
-        vendor-name (get-in order [:sales-order/vendor :vendor/name])
-        charges (:sales-order/charges order)
-        items (:sales-order/line-items order)
-        payment-methods (->> charges (map :charge/type-name) distinct (str/join ","))
-        processors (->> charges (map #(get-in % [:charge/processor :db/ident])) (remove nil?) distinct (map name) (str/join ","))
-        categories (->> items (map :order-line-item/category) (remove nil?) distinct (str/join ","))]
-    (vswap! flat conj
-            {:entity-type       "sales-order"
-             :external-id       (str so-ext-id)
-             :client-code       client-code
-             :location          (:sales-order/location order)
-             :vendor            vendor-name
-             :total             (:sales-order/total order)
-             :tax               (:sales-order/tax order)
-             :tip               (:sales-order/tip order)
-             :discount          (:sales-order/discount order)
-             :service-charge    (:sales-order/service-charge order)
-             :date              so-date
-             :source            (:sales-order/source order)
-             :reference-link    (:sales-order/reference-link order)
-             :payment-methods   payment-methods
-             :processors        processors
-             :categories        categories})
+        so-date   (.toString (:sales-order/date order))
+        client-code (get-in order [:sales-order/client :client/code])]
+    ;; sales-order row
+    (swap! flat conj
+           {:entity-type       "sales-order"
+            :external-id       (str so-ext-id)
+            :client-code       client-code
+            :location          (:sales-order/location order)
+            :vendor            (:sales-order/vendor order)
+            :total             (:sales-order/total order)
+            :tax               (:sales-order/tax order)
+            :tip               (:sales-order/tip order)
+            :discount          (:sales-order/discount order)
+            :service-charge    (:sales-order/service-charge order)
+            :date              so-date})
+    ;; charges & line-items
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
-        (vswap! flat conj
-                {:entity-type       "charge"
-                 :external-id       (str (get chg :charge/external-id))
-                 :type-name         (get chg :charge/type-name)
-                 :total             (get chg :charge/total)
-                 :tax               (get chg :charge/tax)
-                 :tip               (get chg :charge/tip)
-                 :date              so-date
-                 :processor         (get-in chg [:charge/processor :db/ident])
-                 :sales-order-external-id (str so-ext-id)})
+        (swap! flat conj
+               {:entity-type       "charge"
+                :external-id       (str (get chg :charge/external-id))
+                :type-name         (get chg :charge/type-name)
+                :total             (get chg :charge/total)
+                :tax               (get chg :charge/tax)
+                :tip               (get chg :charge/tip)
+                :date              so-date
+                :processor         (get-in chg [:charge/processor :db/ident])
+                :sales-order-external-id (str so-ext-id)})
+        ;; charge returns → sales-refund rows
         (when-let [returns (:charge/returns chg)]
           (doseq [rt returns]
-            (vswap! flat conj
-                    {:entity-type       "sales-refund"
-                     :type-name         (get rt :type-name)
-                     :total             (get rt :total)
-                     :sales-order-external-id (str so-ext-id)})))))
+            (swap! flat conj
+                   {:entity-type       "sales-refund"
+                    :type-name         (get rt :type-name)
+                    :total             (get rt :total)
+                    :sales-order-external-id (str so-ext-id)})))))
+    ;; line-items
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
-        (vswap! flat conj
-                {:entity-type         "line-item"
-                 :item-name           (get li :order-line-item/item-name)
-                 :category            (get li :order-line-item/category)
-                 :total               (get li :order-line-item/total)
-                 :tax                 (get li :order-line-item/tax)
-                 :discount            (get li :order-line-item/discount)
-                 :sales-order-external-id (str so-ext-id)})))))
+        (swap! flat conj
+               {:entity-type         "line-item"
+                :item-name           (get li :order-line-item/item-name)
+                :category            (get li :order-line-item/category)
+                :total               (get li :order-line-item/total)
+                :tax                 (get li :order-line-item/tax)
+                :discount            (get li :order-line-item/discount)
+                :sales-order-external-id (str so-ext-id)})))))
 
 (defn -fetch-order-ids-for-date
   "Query Datomic for all sales-order eids on a given business date."
   [db date-str]
-  (let [ld (java.time.LocalDate/parse date-str)
-        start (-> ld (.atStartOfDay (java.time.ZoneId/of "America/Los_Angeles")) .toInstant java.util.Date/from)
-        end (-> ld (.plusDays 1) (.atStartOfDay (java.time.ZoneId/of "America/Los_Angeles")) .toInstant java.util.Date/from)]
+  (let [day-ms (.toEpochSecond ^java.time.LocalDate (java.time.LocalDate/parse date-str))
+        start (* day-ms 1000)
+        end (+ start (* 86400000))]
     (->> (dc/q '[:find ?e
-                 :in $ ?start ?end
-                 :where [?e :sales-order/date ?d]
-                 [(>= ?d ?start)]
-                 [(< ?d ?end)]]
+                 :in $ ?start-ms ?end-ms
+                 :where [_ :sales-order/date ?d]
+                        [(>= ?d ?start-ms)]
+                        [(<= ?d ?end-ms)]]
                db start end)
          (map first)
          vec)))
 
+
 (defn- date-seq [start end]
   "Seq of YYYY-MM-DD strings between start and end inclusive."
   (let [sd   (java.time.LocalDate/parse start)
@@ -146,9 +138,9 @@
     (for [i (range 0 (inc days))]
       (.toString (.plusDays sd i)))))
 
-(defn write-day-by-day
+(defn- write-day-by-day
   ([start-date end-date]
-   (write-day-by-day start-date end-date {}))
+   (write-day-by-day start-date end-date nil))
   ([start-date end-date opts]
    (let [all-dates  (set (or (opts :date-set) []))
          date-range (if (empty? all-dates)
@@ -164,14 +156,14 @@
            (let [orders (pull-sales-order-data batch)
                  flat   (volatile! [])]
              (doseq [o orders]
-                (flatten-order-to-pieces! o day flat))
+               (flatten-order-to-pieces! o flat))
              (doseq [r @flat]
                (p/buffer! (:entity-type r) r)))))
-        (doseq [etype ["sales-order" "charge"
-                       "line-item" "sales-refund"]]
-          (p/flush-to-parquet! etype day))
+       (doseq [etype ["sales-order" "charge"
+                      "line-item" "sales-refund"]]
+         (p/flush-to-parquet! etype))
        (println "[migration]" day "complete"))
-     {:status :completed :total-days (count date-range)})))
+      {:status :completed :total-days (count date-range)})))
 
 (defn- write-dead-letter
   ([flat]
@@ -189,11 +181,10 @@
   "Flush all entity-type buffers, tracking counts."
   (let [etypes ["sales-order" "charge"
                 "line-item" "sales-refund"]
-        today (.toString (java.time.LocalDate/now))
         start  (p/total-buf-count)]
     (doseq [et etypes]
       (try
-        (p/flush-to-parquet! et today)
+        (p/flush-to-parquet! et)
         (catch Exception e
           (println "[migration/flush]" et "error:" (.getMessage e)))))
     {:records-flush (- (p/total-buf-count) start)}))
@@ -201,11 +192,11 @@
 (defn- get-date-range []
   "Get the earliest and latest business dates from Datomic."
   (let [dates (->> (dc/q '[:find ?d
-                           :where [_ :sales-order/date ?d]]
-                         (dc/db conn))
-                   (map first)
-                   distinct
-                   sort)]
+                          :where [_ :sales-order/date ?d]]
+                        (dc/db conn))
+                  (map first)
+                  distinct
+                  sort)]
     [(when (seq dates) (.toString (first dates)))
      (when (seq dates) (.toString (last dates)))]))
 
@@ -227,9 +218,9 @@
         (doseq [o (pull-sales-order-data order-ids)
                 :when (not (:sales-order/date o))]
           (let [flat (volatile! [])]
-             (flatten-order-to-pieces! o "unknown" flat)
+            (flatten-order-to-pieces! o flat)
             (doseq [r @flat]
-              (p/buffer! "dead" r))))
+               (p/buffer! "dead" r))))
         (write-day-by-day start-date end-date {:batch-size 100})
         (flush-all-types)
         (println "[migration] done")

src/clj/auto_ap/routes/ezcater_xls.clj

@@ -55,54 +55,54 @@
         event-date (some-> (excel/xls-date->date event-date)
                            coerce/to-date-time
                            atime/as-local-time
-                           coerce/to-date)]
-    (cond (and event-date client-id location)
+                           coerce/to-date )]
+    (cond (and event-date client-id location ) 
           [:order #:sales-order
-                   {:date           event-date
-                    :external-id    (str "ezcater/order/" client-id "-" location "-" order-number)
-                    :client         client-id
-                    :location       location
-                    :reference-link (str order-number)
-                    :line-items     [#:order-line-item
-                                      {:external-id (str "ezcater/order/" client-id "-" location "-" order-number "-" 0)
-                                       :item-name   "EZCater Catering"
-                                       :category    "EZCater Catering"
-                                       :discount    (fmt-amount (or adjustments 0.0))
-                                       :tax         (fmt-amount tax)
-                                       :total       (fmt-amount (+ food-total
-                                                                   tax))}]
+           {:date           event-date
+            :external-id    (str "ezcater/order/" client-id "-" location "-" order-number)
+            :client         client-id
+            :location       location
+            :reference-link (str order-number)
+            :line-items     [#:order-line-item
+                             {:external-id (str "ezcater/order/" client-id "-" location "-" order-number "-" 0)
+                              :item-name   "EZCater Catering"
+                              :category    "EZCater Catering"
+                              :discount    (fmt-amount (or adjustments 0.0))
+                              :tax         (fmt-amount tax)
+                              :total       (fmt-amount (+ food-total
+                                                          tax))}]
 
-                    :charges        [#:charge
-                                      {:type-name   "CARD"
-                                       :date        event-date
-                                       :client      client-id
-                                       :location    location
-                                       :external-id (str "ezcater/charge/" client-id "-" location "-" order-number "-" 0)
-                                       :processor   :ccp-processor/ezcater
-                                       :total       (fmt-amount (+ food-total
-                                                                   tax
-                                                                   tip))
-                                       :tip         (fmt-amount tip)}]
-                    :total          (fmt-amount (+ food-total
-                                                   tax
-                                                   (or adjustments 0.0)))
-                    :discount       (fmt-amount (or adjustments 0.0))
-                    :service-charge (fmt-amount (+ fee commission))
-                    :tax            (fmt-amount tax)
-                    :tip            (fmt-amount tip)
-                    :returns        0.0
-                    :vendor         :vendor/ccp-ezcater}]
+            :charges        [#:charge
+                             {:type-name   "CARD"
+                              :date        event-date
+                              :client      client-id
+                              :location    location
+                              :external-id (str "ezcater/charge/" client-id "-" location "-" order-number "-" 0)
+                              :processor   :ccp-processor/ezcater
+                              :total       (fmt-amount (+ food-total
+                                                          tax
+                                                          tip))
+                              :tip         (fmt-amount tip)}]
+            :total          (fmt-amount (+ food-total
+                                           tax
+                                           (or adjustments 0.0)))
+            :discount       (fmt-amount (or adjustments 0.0))
+            :service-charge (fmt-amount (+ fee commission))
+            :tax            (fmt-amount tax)
+            :tip            (fmt-amount tip)
+            :returns        0.0
+            :vendor         :vendor/ccp-ezcater}]
 
-          caterer-name
-          (do
-            (alog/warn ::missing-client
-                       :order order-number
-                       :store-name store-name
-                       :caterer-name caterer-name)
-            [:missing caterer-name])
+      caterer-name
+      (do 
+        (alog/warn ::missing-client
+                   :order order-number
+                   :store-name store-name 
+                   :caterer-name caterer-name)
+        [:missing caterer-name])
 
-          :else
-          nil)))
+      :else
+      nil)))
 
 (defn- flatten-order-to-parquet! [order]
   "Flatten a sales-order into entity-type tagged maps and buffer to parquet."
@@ -125,25 +125,25 @@
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
         (parquet/buffer! "charge"
-                         {:entity-type "charge"
-                          :external-id (:charge/external-id chg)
-                          :type-name   (:charge/type-name chg)
-                          :total       (:charge/total chg)
-                          :tax         (:charge/tax chg)
-                          :tip         (:charge/tip chg)
-                          :date        so-date
-                          :processor   (some-> (:charge/processor chg) name)
-                          :sales-order-external-id so-ext-id})))
+                        {:entity-type "charge"
+                         :external-id (:charge/external-id chg)
+                         :type-name   (:charge/type-name chg)
+                         :total       (:charge/total chg)
+                         :tax         (:charge/tax chg)
+                         :tip         (:charge/tip chg)
+                         :date        so-date
+                         :processor   (some-> (:charge/processor chg) name)
+                         :sales-order-external-id so-ext-id})))
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
         (parquet/buffer! "line-item"
-                         {:entity-type "line-item"
-                          :item-name   (:order-line-item/item-name li)
-                          :category    (:order-line-item/category li)
-                          :total       (:order-line-item/total li)
-                          :tax         (:order-line-item/tax li)
-                          :discount    (:order-line-item/discount li)
-                          :sales-order-external-id so-ext-id})))))
+                        {:entity-type "line-item"
+                         :item-name   (:order-line-item/item-name li)
+                         :category    (:order-line-item/category li)
+                         :total       (:order-line-item/total li)
+                         :tax         (:order-line-item/tax li)
+                         :discount    (:order-line-item/discount li)
+                         :sales-order-external-id so-ext-id})))))
 
 (defn stream->sales-orders [s]
   (let [clients (map first (dc/q '[:find (pull ?c [:client/code
@@ -158,7 +158,7 @@
         object (str "/ezcater-xls/" (str (java.util.UUID/randomUUID)))]
     (mu/log ::writing-temp-xls
             :location object)
-    (s3/put-object {:bucket-name (:data-bucket env)
+    (s3/put-object {:bucket-name (:data-bucket env) 
                     :key object
                     :input-stream s})
     (into []
@@ -200,13 +200,13 @@
     });")]])])
 
 (defn upload-xls [{:keys [identity] :as request}]
-
+  
   (let [file (or (get (:params request) :file)
                  (get (:params request) "file"))]
     (mu/log ::uploading-file
             :file file)
     (with-open [s (io/input-stream (:tempfile file))]
-      (try
+      (try 
         (let [parse-results (stream->sales-orders s)
               new-orders (->> parse-results
                               (filter (comp #{:order} first))
@@ -235,7 +235,7 @@
                                 [:li ml])]])]))
         (catch Exception e
           (alog/error ::import-error
-                      :error e)
+                    :error e)
           (html-response [:div (.getMessage e)]))))))
 
 (defn page [{:keys [matched-route request-method] :as request}]

src/clj/auto_ap/square/core3.clj

@@ -28,9 +28,11 @@
     "Authorization"  (str "Bearer " (:client/square-auth-token client))
     "Content-Type"   "application/json"}))
 
+
 (defn ->square-date [d]
   (f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d))
 
+
 (def manifold-api-stream
   (let [stream (s/stream 100)]
     (->> stream
@@ -41,10 +43,10 @@
                     (de/loop [attempt 0]
                       (-> (de/chain (de/future-with (ex/execute-pool)
                                                     #_(log/info ::request-started
-                                                                :url (:url request)
-                                                                :attempt attempt
-                                                                :source "Square 3"
-                                                                :background-job "Square 3")
+                                                              :url (:url request)
+                                                              :attempt attempt
+                                                              :source "Square 3"
+                                                              :background-job "Square 3")
                                                     (try
                                                       (client/request (assoc request
                                                                              :socket-timeout 10000
@@ -103,6 +105,7 @@
                    :exception error))
       []))))
 
+
 (def item-cache (atom {}))
 
 (defn fetch-catalog [client i v]
@@ -122,11 +125,13 @@
     #(do (swap! item-cache assoc i %)
          %))))
 
+
 (defn fetch-catalog-cache [client i version]
   (if (get @item-cache i)
     (de/success-deferred (get @item-cache i))
     (fetch-catalog client i version)))
 
+
 (defn item->category-name-impl [client item version]
   (capture-context->lc
    (cond (:item_id (:item_variation_data item))
@@ -157,6 +162,7 @@
                      :item item)
            "Uncategorized"))))
 
+
 (defn item-id->category-name [client i version]
   (capture-context->lc
    (-> [client i]
@@ -221,6 +227,7 @@
                                  (concat (:orders result) continued-results))))
                    (:orders result)))))))
 
+
 (defn search
   ([client location start end]
    (capture-context->lc
@@ -244,9 +251,11 @@
                                   (concat (:orders result) continued-results))))
                     (:orders result))))))))
 
+
 (defn amount->money [amt]
   (* 0.01 (or (:amount amt) 0.0)))
 
+
 ;; to get totals: 
 (comment
   (reduce
@@ -272,7 +281,7 @@
      :reference-link (str (url/url  "https://squareup.com/receipt/preview" (:id t)))
      :external-id (when (:id t)
                     (str "square/charge/" (:id t)))
-     :processor   (cond
+     :processor   (cond 
                     (#{"OTHER" "THIRD_PARTY_CARD"} (:type t))
                     (condp = (some-> (:note t) str/lower-case)
                       "doordash" :ccp-processor/doordash
@@ -345,7 +354,7 @@
                        #:sales-order
                         {:date        (if (= "Invoices" (:name (:source order)))
                                         (when (:closed_at order)
-                                          (coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
+                                          (coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles")))) 
                                         (coerce/to-date (time/to-time-zone (coerce/to-date-time (:created_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
                          :client      (:db/id client)
                          :location    (:square-location/client-location location)
@@ -407,6 +416,7 @@
                             :client client
                             :location location)))))))
 
+
 (defn get-payment [client p]
   (de/chain (manifold-api-call
              {:url (str "https://connect.squareup.com/v2/payments/" p)
@@ -415,6 +425,7 @@
             :body
             :payment))
 
+
 (defn continue-payout-entry-list [c l poi cursor]
   (capture-context->lc lc
                        (de/chain
@@ -599,8 +610,8 @@
         so-date   (some-> (:sales-order/date order) .toString)
         client    (:sales-order/client order)
         client-code (when client (if (map? client)
-                                   (:client/code client)
-                                   client))]
+                                  (:client/code client)
+                                  client))]
     (parquet/buffer! "sales-order"
                      {:entity-type "sales-order"
                       :external-id so-ext-id
@@ -616,32 +627,32 @@
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
         (parquet/buffer! "charge"
-                         {:entity-type "charge"
-                          :external-id (:charge/external-id chg)
-                          :type-name   (:charge/type-name chg)
-                          :total       (:charge/total chg)
-                          :tax         (:charge/tax chg)
-                          :tip         (:charge/tip chg)
-                          :date        so-date
-                          :processor   (some-> (:charge/processor chg) name)
-                          :sales-order-external-id so-ext-id})
+                        {:entity-type "charge"
+                         :external-id (:charge/external-id chg)
+                         :type-name   (:charge/type-name chg)
+                         :total       (:charge/total chg)
+                         :tax         (:charge/tax chg)
+                         :tip         (:charge/tip chg)
+                         :date        so-date
+                         :processor   (some-> (:charge/processor chg) name)
+                         :sales-order-external-id so-ext-id})
         (when-let [returns (:charge/returns chg)]
           (doseq [rt returns]
             (parquet/buffer! "sales-refund"
-                             {:entity-type "sales-refund"
-                              :type-name   (:type-name rt)
-                              :total       (:total rt)
-                              :sales-order-external-id so-ext-id})))))
+                            {:entity-type "sales-refund"
+                             :type-name   (:type-name rt)
+                             :total       (:total rt)
+                             :sales-order-external-id so-ext-id})))))
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
         (parquet/buffer! "line-item"
-                         {:entity-type "line-item"
-                          :item-name   (:order-line-item/item-name li)
-                          :category    (:order-line-item/category li)
-                          :total       (:order-line-item/total li)
-                          :tax         (:order-line-item/tax li)
-                          :discount    (:order-line-item/discount li)
-                          :sales-order-external-id so-ext-id})))))
+                        {:entity-type "line-item"
+                         :item-name   (:order-line-item/item-name li)
+                         :category    (:order-line-item/category li)
+                         :total       (:order-line-item/total li)
+                         :tax         (:order-line-item/tax li)
+                         :discount    (:order-line-item/discount li)
+                         :sales-order-external-id so-ext-id})))))
 
 (defn upsert
   ([client]
@@ -663,7 +674,8 @@
                         (catch Exception e
                           (log/error ::buffer-failed
                                      :exception e
-                                     :order (:sales-order/external-id order))))))))))))
+                                     :order (:sales-order/external-id order)))))))))))
+
 
 (defn upsert-payouts
   ([client]
@@ -713,6 +725,7 @@
 
                    (log/info ::done-loading-refunds)))))))
 
+
 (defn get-cash-shift [client id]
   (de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id))
                                 :method :get
@@ -871,6 +884,8 @@
     d1
     d2))
 
+
+
 (defn remove-voided-orders
   ([client]
    (apply de/zip
@@ -897,7 +912,7 @@
                                                     (:sales-order/external-id o))))))
                            (s/map (fn [[o]]
                                     [[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]]))
-
+                           
                            (s/reduce into [])))
 
                     (fn [results]
@@ -905,28 +920,32 @@
                         (doseq [x (partition-all 100 results)]
                           (log/info ::removing-orders
                                     :count (count x))
-                          @(dc/transact-async conn x)
+                          @(dc/transact-async conn x)))))
                           (de/catch (fn [e]
                                       (log/warn ::couldnt-remove :error e)
-                                      nil)))))))))))
+                                      nil) ))))))
 
-#_(comment
- (require 'auto-ap.time-reader)
+#_(comment 
+  (require 'auto-ap.time-reader)
 
- @(let [[c [l]] (get-square-client-and-location "DBFS")]
-    (log/peek :x  [c l])
-    (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29"))
+ @(let [[c [l]] (get-square-client-and-location "DBFS")  ]
+     (log/peek :x  [ c l])
+     (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29")
 
- @(let [[c [l]] (get-square-client-and-location "NGAK")]
-    (log/peek :x  [c l])
+     ) 
 
-    (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
- (doseq [c (get-square-clients)]
-   (try
-     @(remove-voided-orders c)
-     (catch Exception e
-       nil)))
- )
+  @(let [[c [l]] (get-square-client-and-location "NGAK")  ]
+     (log/peek :x  [ c l])
+
+     (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
+  (doseq [c (get-square-clients)]
+    (try 
+      @(remove-voided-orders c)
+      (catch Exception e
+        nil)))
+
+  
+  )
 
 (defn upsert-all [& clients]
   (capture-context->lc
@@ -995,6 +1014,8 @@
    [:clients clients]
    @(apply upsert-all clients)))
 
+
+
 (comment
   (defn refunds-raw-cont
     ([client l cursor so-far]
@@ -1024,8 +1045,9 @@
   (->>
    @(let [[c [l]] (get-square-client-and-location "NGGG")]
 
-      (search c l (time/now) (time/plus (time/now) (time/days -1))))
 
+      (search c l (time/now) (time/plus (time/now) (time/days -1))))
+   
    (filter (fn [r]
              (str/starts-with? (:created_at r) "2024-03-14"))))
 
@@ -1033,6 +1055,7 @@
     (->>
      @(let [[c [l]] (get-square-client-and-location "NGGG")]
 
+
         (refunds-raw-cont c l nil []))
      (filter (fn [r]
                (str/starts-with? (:created_at r) "2024-03-14")))))
@@ -1066,8 +1089,13 @@
                                                    []))]
                                 [(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row)  (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
                               :separator \tab)
+  
+
+
+
+  ;; => 
+  
 
-;; => 
 
   (require 'auto-ap.time-reader)
 
@@ -1076,16 +1104,27 @@
   (clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")]
                            l
 
+
                            (def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00"
                                            #clj-time/date-time "2025-02-28T00:00:00-08:00"))
-                           (take 10 (map #(first (deref (order->sales-order c l %))) z))))
+                           (take 10 (map #(first (deref (order->sales-order c l %))) z)))
 
-  (->> z
+
+                         )
+  
+  
+
+
+  (->> z 
        (filter (fn [o]
                  (seq (filter (comp #{"OTHER"} :type) (:tenders o)))))
        (filter #(not (:name (:source %))))
-       (count))
-
+       (count)
+       
+       )
+  
+  
+  
   (doseq [[code] (seq (dc/q '[:find ?code
                               :in $
                               :where [?o :sales-order/date ?d]
@@ -1094,22 +1133,32 @@
                               [?o :sales-order/client ?c]
                               [?c :client/code ?code]]
                             (dc/db conn)))
-          :let [[c [l]] (get-square-client-and-location code)]
+          :let [[c [l]] (get-square-client-and-location code)
+                ]
           order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now))
-          :when (= "Invoices" (:name (:source order)))
+          :when (= "Invoices" (:name (:source order) ))
           :let [[sales-order] @(order->sales-order c l order)]]
-
+    
     (when (should-import-order? order)
       (println "DATE IS" (:sales-order/date sales-order))
       (when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after?  #clj-time/date-time "2026-2-16T00:00:00-08:00"))
         (println "WOULD UPDATE" sales-order)
-        @(dc/transact auto-ap.datomic/conn [sales-order]))
-      #_@(dc/transact)
-      (println "DONE")))
+        @(dc/transact auto-ap.datomic/conn [sales-order])
+        )
+      #_@(dc/transact )
+      (println "DONE"))
+    
+    
+    )
 
   #_(filter (comp #{"OTHER"} :type) (mapcat :tenders z))
+  
 
   @(let [[c [l]] (get-square-client-and-location "NGRY")]
      #_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
 
-     (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY")))))
+     (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY" )))
+
+     )
+  
+  )

src/clj/auto_ap/ssr/payments.clj

@@ -104,18 +104,19 @@
                                   :size        :small})])
     (com/field {:label "Payment Type"}
                (com/radio-card {:size    :small
-                                :name    "payment-type"
-                                :value (:payment-type (:query-params request))
-                                :options [{:value   ""
-                                           :content "All"}
-                                          {:value   "cash"
-                                           :content "Cash"}
-                                          {:value   "check"
-                                           :content "Check"}
-                                          {:value   "debit"
-                                           :content "Debit"}]}))
+                           :name    "payment-type"
+                           :value (:payment-type (:query-params request))
+                           :options [{:value   ""
+                                      :content "All"}
+                                     {:value   "cash"
+                                      :content "Cash"}
+                                     {:value   "check"
+                                      :content "Check"}
+                                     {:value   "debit"
+                                      :content "Debit"}]}))
     (exact-match-id* request)]])
 
+
 (def default-read '[*
                     [:payment/date :xform clj-time.coerce/from-date]
                     {:invoice-payment/_payment [* {:invoice-payment/invoice [*]}]}
@@ -211,6 +212,7 @@
                                                             '[(iol-ion.query/dollars= ?transaction-amount ?amount)]]}
                                             :args  [(:amount query-params)]})
 
+
                               (:status route-params)
                               (merge-query {:query {:in    ['?status]
                                                     :where ['[?e :payment/status ?status]]}
@@ -241,30 +243,30 @@
     refunds))
 
 (defn sum-visible-pending [ids]
-  (->>
-   (dc/q {:find  ['?id '?o]
-          :in    ['$ '[?id ...]]
-          :where ['[?id :payment/amount ?o]
-                  '[?id :payment/status :payment-status/pending]]}
-         (dc/db conn)
-         ids)
+  (->> 
+    (dc/q {:find  ['?id '?o]
+           :in    ['$ '[?id ...]]
+           :where ['[?id :payment/amount ?o]
+                   '[?id :payment/status :payment-status/pending]]}
+          (dc/db conn)
+          ids)
    (map last)
    (reduce
     +
     0.0)))
 
 (defn sum-client-pending [clients]
-  (->>
-   (dc/q {:find '[?e ?a]
-          :in '[$ [?clients ?start ?end]]
-          :where '[[(iol-ion.query/scan-payments $ ?clients ?start ?end) [[?e _ ?sort-default] ...]]
-                   [?e :payment/status :payment-status/pending]
-                   [?e :payment/amount ?a]]}
-         (dc/db conn)
-         [clients
-          nil
-          nil])
-
+  (->> 
+    (dc/q {:find '[?e ?a]
+           :in '[$ [?clients ?start ?end]]
+           :where '[[(iol-ion.query/scan-payments $ ?clients ?start ?end) [[?e _ ?sort-default] ...]]
+                    [?e :payment/status :payment-status/pending]
+                    [?e :payment/amount ?a]]}
+          (dc/db conn)
+          [clients
+           nil
+           nil])
+          
    (map last)
    (reduce
     +
@@ -275,14 +277,16 @@
         {ids-to-retrieve :ids matching-count :count
          all-ids :all-ids} (fetch-ids db request)]
 
+
     [(->> (hydrate-results ids-to-retrieve db request))
      matching-count
      (sum-visible-pending all-ids)
      (sum-client-pending (extract-client-ids (:clients request)
-                                             (:client request)
-                                             (:client-id (:query-params request))
-                                             (when (:client-code (:query-params request))
-                                               [:client/code (:client-code (:query-params request))])))]))
+                                          (:client request)
+                                          (:client-id (:query-params request))
+                                          (when (:client-code (:query-params request))
+                                            [:client/code (:client-code (:query-params request))])))
+     ]))
 
 (def query-schema (mc/schema
                    [:maybe [:map {:date-range [:date-range :start-date :end-date]}
@@ -323,7 +327,7 @@
                     (assoc-in (exact-match-id* request)  [1 :hx-swap-oob] true)])
                  :query-schema query-schema
                  :action-buttons (fn [request]
-                                   (let [[_ _ visible-in-float total-in-float] (:page-results request)]
+                                   (let [[_ _ visible-in-float total-in-float ] (:page-results request)]
                                      [(com/pill {:color :primary} " Visible in float "
                                                 (format "$%,.2f" visible-in-float))
                                       (com/pill {:color :secondary} " Total in float "
@@ -350,7 +354,7 @@
 
                                   (= (-> request :query-params :sort first :name) "Bank account")
                                   (-> entity :payment/bank-account :bank-account/name)
-
+  
                                   :else nil))
                  :title       (fn [r]
                                 (str
@@ -405,7 +409,7 @@
                                 :render        (fn [{:payment/keys [date]}]
                                                  (some-> date (atime/unparse-local atime/normal-date)))}
                                {:key           "amount"
-                                :sort-key "amount"
+                               :sort-key "amount" 
                                 :name          "Amount"
                                 :render        (fn [{:payment/keys [amount]}]
                                                  (some->> amount (format "$%.2f")))}
@@ -417,10 +421,10 @@
                                                                              (map :invoice-payment/invoice)
                                                                              (filter identity)
                                                                              (map (fn [invoice]
-                                                                                    {:link (hu/url (bidi/path-for ssr-routes/only-routes
-                                                                                                                  ::invoice-route/all-page)
-                                                                                                   {:exact-match-id (:db/id invoice)})
-                                                                                     :content (str "Inv. " (:invoice/invoice-number invoice))})))
+                                                                                       {:link (hu/url (bidi/path-for ssr-routes/only-routes
+                                                                                                                     ::invoice-route/all-page)
+                                                                                                      {:exact-match-id (:db/id invoice)})
+                                                                                        :content (str "Inv. " (:invoice/invoice-number invoice))})))
                                                                         (some-> p :transaction/_payment ((fn [t]
                                                                                                            [{:link (hu/url (bidi/path-for client-routes/routes
                                                                                                                                           :transactions)
@@ -430,6 +434,8 @@
 
 (def row* (partial helper/row* grid-page))
 
+
+
 (comment
   (mc/decode query-schema {"exact-match-id" "123"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
   (mc/decode query-schema {} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
@@ -439,6 +445,7 @@
   (mc/decode query-schema {"payment-type" "food"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
   (mc/decode query-schema {"vendor" "87"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
 
+
   (mc/decode query-schema {"start-date" #inst "2023-12-21T08:00:00.000-00:00"} (mt/transformer main-transformer mt/strip-extra-keys-transformer)))
 
 (defn delete [{check :entity :as request identity :identity}]
@@ -452,7 +459,7 @@
    #(assert-can-see-client identity (:db/id (:payment/client check))))
   (notify-if-locked (:db/id (:payment/client check))
                     (:payment/date check))
-  (let [removing-payments (mapcat (fn [x]
+  (let [ removing-payments (mapcat (fn [x]
                                     (let [invoice (:invoice-payment/invoice x)
                                           new-balance (+ (:invoice/outstanding-balance invoice)
                                                          (:invoice-payment/amount x))]
@@ -468,9 +475,9 @@
                          :payment/status :payment-status/voided}]
     (audit-transact (cond-> removing-payments
                       true (conj updated-payment)
-                      (:transaction/_payment check) (conj [:db/retract (:db/id (first (:transaction/_payment check)))
+                      (:transaction/_payment check) (conj [:db/retract (:db/id (first (:transaction/_payment check))) 
                                                            :transaction/payment
-                                                           (:db/id check)]))
+                                                           (:db/id check)])) 
                     identity)
 
     (html-response (row* (:identity request) updated-payment {:delete-after-settle? true :class "live-removed"
@@ -571,6 +578,7 @@
                                                 (assoc-in [:query-params :start] 0)
                                                 (assoc-in [:query-params :per-page]  250))))
 
+
               :else
               selected)
         updated-count (void-payments-internal ids (:identity request))]
@@ -583,7 +591,7 @@
 
 (defn wrap-status-from-source [handler]
   (fn [{:keys [matched-current-page-route] :as request}]
-    (let [request (cond-> request
+    (let [ request (cond-> request
                     (= ::route/cleared-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/cleared)
                     (= ::route/pending-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/pending)
                     (= ::route/voided-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/voided)
@@ -597,7 +605,7 @@
     ::route/pending-page              (-> (helper/page-route grid-page)
                                           (wrap-implied-route-param :status :payment-status/pending))
     ::route/voided-page              (-> (helper/page-route grid-page)
-                                         (wrap-implied-route-param :status :payment-status/voided))
+                                          (wrap-implied-route-param :status :payment-status/voided))
     ::route/all-page              (-> (helper/page-route grid-page)
                                       (wrap-implied-route-param :status nil))
 
@@ -610,6 +618,7 @@
     ::route/bulk-delete  (-> bulk-delete-dialog
                              (wrap-admin))
 
+
     ::route/table             (helper/table-route grid-page)}
    (fn [h]
      (-> h

src/clj/auto_ap/ssr/pos/sales_orders.clj

@@ -1,7 +1,7 @@
 (ns auto-ap.ssr.pos.sales-orders
   (:require
    [auto-ap.datomic
-    :refer [add-sorter-fields apply-pagination apply-sort-3 merge-query
+    :refer [add-sorter-fields apply-pagination apply-sort-3 conn merge-query
             pull-many query2]]
    [auto-ap.datomic.sales-orders :as d-sales]
    [auto-ap.query-params :as query-params :refer [wrap-copy-qp-pqp]]
@@ -17,6 +17,7 @@
    [auto-ap.time :as atime]
    [bidi.bidi :as bidi]
    [clj-time.coerce :as c]
+   [datomic.api :as dc]
    [malli.core :as mc]))
 
 (def query-schema (mc/schema
@@ -171,8 +172,11 @@
     charges))
 
 (defn fetch-page [request]
-  (let [{:keys [rows count]} (d-sales/fetch-page-ssr request)]
-    [rows count]))
+  (let [db                                           (dc/db conn)
+        {ids-to-retrieve :ids matching-count :count} (fetch-ids db request)]
+
+    [(->> (hydrate-results ids-to-retrieve db request))
+     matching-count]))
 
 
 (def grid-page
@@ -196,13 +200,13 @@
     :title          "Sales orders"
     :entity-name    "Sales orders"
     :route          :pos-sales-table
-      :action-buttons (fn [request]
-                        (let [{:keys [total tax]} (d-sales/summarize-page-ssr request)]
-                          (when (and total tax)
-                            [(com/pill {:color :primary}
-                                       (format "Total $%.2f" total))
-                             (com/pill {:color :secondary}
-                                       (format "Tax $%.2f" tax))])))
+    :action-buttons (fn [request]
+                      (let [{:keys [total tax]} (d-sales/summarize-orders (:ids (fetch-ids (dc/db conn) request)))]
+                        (when (and total tax)
+                          [(com/pill {:color :primary}
+                                     (format "Total $%.2f" total))
+                           (com/pill {:color :secondary}
+                                     (format "Tax $%.2f" tax))])))
     :row-buttons    (fn [_ e]
                       (when (:sales-order/reference-link e)
                         [(com/a-icon-button {:href (:sales-order/reference-link e)}

src/clj/auto_ap/storage/parquet.clj

@@ -22,10 +22,6 @@
   (let [conn (DriverManager/getConnection "jdbc:duckdb:")
         stmt (.createStatement conn)]
     (.execute stmt "INSTALL httpfs; LOAD httpfs;")
-    (when-let [key (:aws-access-key-id env)]
-      (.execute stmt (str "SET s3_access_key_id='" key "'"))
-      (.execute stmt (str "SET s3_secret_access_key='" (:aws-secret-access-key env) "'"))
-      (.execute stmt (str "SET s3_region='" (or (:aws-region env) "us-east-1") "'")))
     (.close stmt)
     (.addShutdownHook (Runtime/getRuntime)
                       (Thread. #(fn [])))
@@ -111,9 +107,9 @@
                               (str entity-type ".jsonl"))]
         (io/make-parents wal-file)
         (with-open [w (io/writer wal-file :append true)]
-          (.write w ^String (json/write-str {:seq-no seq-no
-                                             :record record}))
-          (.write w (int \newline))))
+           (.write w ^String (json/write-str {:seq-no seq-no
+                                               :record record}))
+           (.write w (int \newline))))
       (catch Exception e
         (println "[parquet/wal]" (.getMessage e))))
     entry))
@@ -128,12 +124,12 @@
   (->> @*buffers*
        vals (mapcat identity) count))
 
-(defn flush-to-parquet! [entity-type date-str]
+(defn flush-to-parquet! [entity-type]
   "Flush buffered records for entity-type to parquet + S3."
   (let [records (get @*buffers* entity-type [])]
     (if (empty? records)
       {:status :no-records}
-      (let [date-str     (or date-str (.toString (LocalDate/now)))
+      (let [date-str   (.toString (LocalDate/now))
             jsonl-file   (io/file "/tmp"
                                   (str entity-type "-" date-str ".jsonl"))
             parquet-file (io/file "/tmp"
@@ -142,8 +138,8 @@
         (try
           (with-open [w (io/writer jsonl-file :append true)]
             (doseq [r records]
-              (.write w ^String (json/write-str (dissoc r :_seq-no)))
-              (.write w (int \newline))))
+               (.write w ^String (json/write-str (dissoc r :_seq-no)))
+               (.write w (int \newline))))
           (execute-to-parquet!
            (format "SELECT * FROM read_json_auto('%s')"
                    (.getAbsolutePath jsonl-file))
@@ -154,19 +150,17 @@
           (.delete ^java.io.File parquet-file)
           {:key s3-key :status :ok}
           (catch Exception e
-            (throw (ex-info "Flush failed"
-                            {:entity-type entity-type
-                             :error (.getMessage e)}))))))))
+            (throw (ex-info "Flush failed" {:entity-type entity-type}
+                            :error (.getMessage e)))))))))
 
 (defn flush-by-date! []
   "Flush all entity types for today."
   (let [etypes    ["sales-order" "charge"
                    "line-item" "sales-refund"]
-        today     (.toString (LocalDate/now))
         flushed   (into #{}
                         (keep (fn [et]
                                 (let [{:keys [status]}
-                                      (flush-to-parquet! et today)]
+                                      (flush-to-parquet! et)]
                                   (when (= status :ok)
                                     et))))
                         etypes)]
@@ -191,12 +185,11 @@
                 {}
                 (into {}
                       (keep (fn [et]
-                               (let [f (io/file
-                                        (wal-dir)
-                                        (str et ".jsonl"))]
-                                 (when (.exists f)
-                                   [et (slurp f)])))
-                      etypes)))]
+                              (let [f (io/file
+                                       (wal-dir)
+                                       (str et ".jsonl"))]
+                                [et (slurp f)])))
+                      etypes))]
     (swap! *buffers* merge loaded)))
 
 (defn get-unflushed-count []
@@ -220,135 +213,66 @@
 (defn today []
   (.toString (LocalDate/now)))
 
-(defn- parquet-glob [entity-type start-date end-date]
-  "Build a glob pattern or explicit file list for the date range.
-   Uses glob patterns for ranges > 60 days; explicit list otherwise."
-  (let [days (-> (LocalDate/parse end-date)
-                 (.toEpochDay)
-                 (- (.toEpochDay (LocalDate/parse start-date)))
-                 inc)]
-    (if (> days 60)
-      (let [prefix (format "s3://%s/sales-details/%s/" *bucket* entity-type)
-            sy (-> (LocalDate/parse start-date) .getYear)
-            ey (-> (LocalDate/parse end-date) .getYear)]
-        (if (= sy ey)
-          [(format "%s%d-*.parquet" prefix sy)]
-          (vec
-           (for [y (range sy (inc ey))]
-             (format "%s%d-*.parquet" prefix y)))))
-      (vec
-       (map (fn [d]
-              (format "'s3://%s/sales-details/%s/%s.parquet'"
-                      *bucket* entity-type d))
-            (date-seq start-date end-date))))))
-
 (defn parquet-query [entity-type start-date end-date]
   "Build SQL to read all parquet files in date range.
    Returns map with :sql and :count-sql keys."
-  (let [globs     (parquet-glob entity-type start-date end-date)
-        use-glob? (some #(.endsWith ^String % "*.parquet") globs)
-        base      (if use-glob?
-                    (format "SELECT * FROM read_parquet(%s, union_by_name=true)"
-                            (if (= (count globs) 1)
-                              (format "'%s'" (first globs))
-                              (format "[%s]"
-                                      (str/join ", " (map #(format "'%s'" %) globs)))))
-                    (format "SELECT * FROM read_parquet([%s])"
-                            (str/join ", " globs)))
-        add-date-filter (fn [sql]
-                          (if (> (-> (LocalDate/parse end-date)
-                                     (.toEpochDay)
-                                     (- (.toEpochDay (LocalDate/parse start-date)))
-                                     inc)
-                               60)
-                            (format "%s WHERE date >= '%s' AND date <= '%s'"
-                                    sql start-date end-date)
-                            sql))
-        sql (add-date-filter base)]
+  (let [date-strs (date-seq start-date end-date)
+        urls      (vec
+                   (map #(format "'s3://%%s/sales-details/%%s/%%s.parquet'"
+                                 *bucket* entity-type %)
+                        date-strs))
+        sql       (str "SELECT * FROM read_parquet(["
+                       (str/join ", " urls)
+                       "])")]
     {:sql       sql
      :count-sql (format "SELECT COUNT(*) FROM (%s) t" sql)}))
 
-(defn- like-clause [col v]
-  (str "\"" col "\" LIKE '%" v "%'"))
-
-(defn- build-sales-orders-where [opts]
-  (let [eq-clauses (keep
-                    (fn [[key col]]
-                      (let [v (get opts key)]
-                        (when v
-                          (str "\"" col "\" = '" v "'"))))
-                    [[:client "client-code"]
-                     [:vendor "vendor"]
-                     [:location "location"]])
-        like-clauses (keep
-                      (fn [[key col]]
-                        (let [v (get opts key)]
-                          (when v
-                            (like-clause col v))))
-                      [[:payment-method "payment-methods"]
-                       [:processor "processors"]
-                       [:category "categories"]])
-        range-clauses (keep
-                       (fn [[key col op]]
-                         (let [v (get opts key)]
-                           (when v
-                             (str "\"" col "\" " op " " v))))
-                       [[:total-gte "total" ">="]
-                        [:total-lte "total" "<="]])
-        all-clauses (concat eq-clauses like-clauses range-clauses)]
-    (when (seq all-clauses)
-      (str " WHERE " (str/join " AND " all-clauses)))))
+(defn- build-where-clause [opts field-pairs]
+  "Build SQL WHERE clause from opts map.
+   fields-with-keys is vector of [:field-key :env-var-name]."
+  (let [clauses (keep
+                 (fn [[key env]]
+                   (let [v (get opts key)]
+                     (when v
+                       (str env " = '" v "'"))))
+                 field-pairs)]
+    (when (seq clauses)
+      (str " WHERE " (str/join " AND " clauses)))))
 
 (defn get-sales-orders
   ([start-date end-date]
    (get-sales-orders start-date end-date {}))
   ([start-date end-date opts]
-   (try
-     (let [q               (parquet-query "sales-order"
-                                          start-date end-date)
-           base-sql         (:sql q)
-           count-sql        (:count-sql q)
-           sort             (get opts :sort "date")
-           order            (get opts :order "DESC")
-           limit            (get opts :limit)
-           offset           (get opts :offset)
-           where-str        (build-sales-orders-where opts)
-           full-sql         (if where-str
-                              (str base-sql where-str)
-                              base-sql)
-           result           (cond-> full-sql
-                              sort      (str " ORDER BY " sort
-                                             " " (name order))
-                              limit     (str " LIMIT " limit)
-                              offset    (str " OFFSET " offset))
-           full-count       (if where-str
-                              (str count-sql where-str)
-                              count-sql)]
-       {:rows   (query-rows result)
-        :count  (or
-                 (int
-                  (query-scalar
-                   full-count)) 0)})
-     (catch Exception _
-       {:rows [] :count 0}))))
+   (let [q               (parquet-query "sales-order"
+                                         start-date end-date)
+          base-sql         (:sql q)
+          count-sql        (:count-sql q)
+          sort             (get opts :sort "date")
+          order            (get opts :order "DESC")
+          limit            (get opts :limit)
+          offset           (get opts :offset)
+          where-str        (build-where-clause
 
-(defn get-sales-orders-summary
-  ([start-date end-date]
-   (get-sales-orders-summary start-date end-date {}))
-  ([start-date end-date opts]
-   (try
-     (let [q          (parquet-query "sales-order" start-date end-date)
-           base-sql   (:sql q)
-           where-str  (build-sales-orders-where opts)
-           full-sql   (if where-str
-                        (str base-sql where-str)
-                        base-sql)
-           sum-sql    (format "SELECT COALESCE(SUM(total), 0) as total, COALESCE(SUM(tax), 0) as tax FROM (%s) t" full-sql)
-           row        (first (query-rows sum-sql))]
-       {:total (or (:total row) 0.0)
-        :tax   (or (:tax row) 0.0)})
-     (catch Exception _
-       {:total 0.0 :tax 0.0}))))
+                            opts
+                            [[:client "external_id.client"]
+                             [:vendor "external_id.vendor"]
+                             [:location "location"]])
+          full-sql         (if where-str
+                             (str base-sql where-str)
+                             base-sql)
+          result           (cond-> full-sql
+                             sort      (str " ORDER BY " sort
+                                            " " (name order))
+                             limit     (str " LIMIT " limit)
+                             offset    (str " OFFSET " offset))
+          full-count       (if where-str
+                             (str count-sql where-str)
+                             count-sql)]
+      {:rows   (query-rows result)
+       :count  (or
+                (int
+                 (query-scalar
+                  full-count)) 0)})))
 
 (defn query-deduped [entity-type start-date end-date]
   "Query records deduplicated by external-id (latest _seq_no wins)."

src/clj/auto_ap/storage/sales_summaries.clj

@@ -55,7 +55,7 @@
                                          tname
                                          (+ (get b tname 0.0) total)))))))
                   {}
-                  rows)))
+                   rows)))
       (catch Exception e
         (println "[sales-summaries]" (.getMessage e))
         {}))))