feat(sales): wire SSR page to parquet/DuckDB layer with full 7.9M-record support

- Add fetch-page-ssr and summarize-page-ssr to read from parquet via DuckDB
- Add get-sales-orders-summary for cross-page totals (SUM across all rows)
- Optimize parquet-query for large ranges (>60 days) with year-level globs
- Add default-date-range with fallback to data's actual range
- Fix migration: flatten-order-to-pieces! vswap!, pull specs, date handling
- Add denormalized columns: payment-methods, processors, categories, source
- Handle schema-enforce middleware stripping dates via raw query-string parsing
- Add graceful fallback for missing parquet files (catch Exception)
- Fix load-unflushed! with .exists check on WAL files

.opencode/skills/gitea-tea/SKILL.md

@@ -0,0 +1,127 @@
+---
+name: gitea-tea
+description: Use tea CLI to create, manage, and checkout Gitea pull requests. Use this when opening a PR, managing PRs, or checking out PRs on the gitea remote (gitea.story-basking.ts.net).
+---
+
+# Gitea Tea CLI Skill
+
+This skill covers using `tea` (Gitea's official CLI) for pull request workflows in this project.
+
+## When to Use This Skill
+
+Use this skill when you need to:
+- Create a PR from a working branch to master on the gitea remote
+- Open, list, or view PRs
+- Checkout a PR locally for review or iteration
+- Manage PR state (close, reopen, merge)
+
+## Project Setup
+
+The gitea remote is `gitea.story-basking.ts.net` with repo slug `notid/integreat`. The default push remote is **gitea**, NOT origin and NOT deploy.
+
+In this project's environment:
+- Gitea login is pre-configured for `gitea.story-basking.ts.net`
+- Repo slug: `notid/integreat`
+- Target branch for PRs: `master`
+- The git remote named `gitea` points to this instance
+
+## Creating a PR
+
+Use `tea pulls create` to open a PR from the current branch to master. Always specify `-r notid/integreat -b master`:
+
+```bash
+tea pulls create -r notid/integreat -b master --title "Title" --description "Body"
+```
+
+Common flags:
+- `-t, --title` - PR title
+- `-d, --description` - PR body/description (use heredoc or file for long descriptions)
+- `-a, --assignees` - Comma-separated usernames to assign
+- `-L, --labels` - Comma-separated labels to apply
+- `-m, --milestone` - Milestone to assign
+
+**Writing a multiline description:**
+
+```bash
+tea pulls create -r notid/integreat -b master \
+  -t "feat: add feature" \
+  -d "$(cat <<'EOF'
+## Summary
+- Bullet point one
+- Bullet point two
+EOF
+)"
+```
+
+Or write the body to a temp file first and reference it.
+
+## Listing PRs
+
+```bash
+tea pulls list -r notid/integreat                          # List open PRs
+tea pulls list -r notid/integreat --state all              # All PRs
+tea pulls list -r notid/integreat --limit 10 -o simple     # Limit output, simple format
+```
+
+## Opening a PR in Browser
+
+```bash
+tea open pr <number> -r notid/integreat
+tea open pr create -r notid/integreat    # Open web UI to create a PR
+```
+
+## Checking Out a PR Locally
+
+```bash
+tea pulls checkout <number> -r notid/integreat
+```
+
+This fetches and checks out the PR branch locally.
+
+## Managing PR State
+
+**Close a PR:**
+```bash
+tea pulls close <number> -r notid/integreat --confirm
+```
+
+**Reopen a closed PR:**
+```bash
+tea pulls reopen <number> -r notid/integreat --confirm
+```
+
+**Merge a PR:**
+```bash
+tea pulls merge <number> -r notid/integreat --confirm
+```
+
+**Edit a PR (title, description, etc.):**
+```bash
+tea pulls edit <number> -r notid/integreat --title "New title" --description "New body"
+```
+
+## Full PR Creation Workflow
+
+1. Ensure the branch is pushed to gitea:
+   ```bash
+   git push gitea <branch-name>
+   ```
+
+2. Create the PR with tea:
+   ```bash
+   tea pulls create -r notid/integreat -b master \
+     --title "feat: description of change" \
+     --description "Detailed PR body here"
+   ```
+
+3. Open the PR in browser to verify:
+   ```bash
+   tea open pr <number> -r notid/integreat
+   ```
+
+## Tips
+
+- Always use `-r notid/integreat` to specify the repo explicitly
+- Use `-b master` to set the target branch (default may differ)
+- The `--confirm` flag is required for destructive actions (close, merge)
+- Use `-o simple`, `-o json`, `-o table`, etc. to control output format

AGENTS.md

@@ -1,54 +1,9 @@
-# Agent Instructions
+## Pull Requests on Gitea
 
-This project uses **bd** (beads) for issue tracking. Run `bd onboard` to get started.
+This project uses **gitea story-basking.ts net** as the primary remote for PRs. Use `tea` (the Gitea CLI) to create and manage pull requests. The gitea remote is the one you push to, NOT origin and NOT deploy.
 
-## Issue Tracking
+**When opening a PR**, load and follow the **gitea-tea** skill. In short:
-
+- Target branch is always `master`
-This project uses **bd (beads)** for issue tracking.
+- Use `tea pulls create -r notid/integreat -b master --title "..." --description "..."`
-Run `bd prime` for workflow context, or install hooks (`bd hooks install`) for auto-injection.
-
-**Quick reference:**
-- `bd ready` - Find unblocked work
-- `bd create "Title" --type task --priority 2` - Create issue
-- `bd close <id>` - Complete work
-- `bd sync` - Sync with git (run at session end)
-
-For full workflow details: `bd prime`
-
-## Quick Reference
-
-```bash
-bd ready              # Find available work
-bd show <id>          # View issue details
-bd update <id> --status in_progress  # Claim work
-bd close <id>         # Complete work
-bd sync               # Sync with git
-```
-
-## Landing the Plane (Session Completion)
-
-**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds.
-
-**MANDATORY WORKFLOW:**
-
-1. **File issues for remaining work** - Create issues for anything that needs follow-up
-2. **Run quality gates** (if code changed) - Tests, linters, builds
-3. **Update issue status** - Close finished work, update in-progress items
-4. **PUSH TO REMOTE** - This is MANDATORY:
-   ```bash
-   git pull --rebase
-   bd sync
-   git push
-   git status  # MUST show "up to date with origin"
-   ```
-5. **Clean up** - Clear stashes, prune remote branches
-6. **Verify** - All changes committed AND pushed
-7. **Hand off** - Provide context for next session
-
-**CRITICAL RULES:**
-- Work is NOT complete until `git push` succeeds
-- NEVER stop before pushing - that leaves work stranded locally
-- NEVER say "ready to push when you are" - YOU must push
-- If push fails, resolve and retry until it succeeds
 
 Use 'bd' for task tracking

src/clj/auto_ap/datomic/sales_orders.clj

@@ -1,39 +1,51 @@
 (ns auto-ap.datomic.sales-orders
   (:require
-   [auto-ap.datomic :refer [conn]]
    [auto-ap.storage.parquet :as pq]
-   [clj-time.coerce :as c]
+   [auto-ap.time :as atime]
+   [clj-time.coerce :as coerce]
    [clojure.set :as set]
-   [com.brunobonacci.mulog :as mu]))
+   [clojure.string :as str]
+   [com.brunobonacci.mulog :as mu]
+   [ring.util.codec :as ring-codec]))
+
+(defn- payment-methods->charges [pm-str]
+  (when (not-empty pm-str)
+    (mapv (fn [pm] {:charge/type-name pm})
+          (str/split pm-str #","))))
 
 (defn <-row
-  "Convert a flat parquet row into the shape consumers expect.
+  "Convert a flat parquet row into the shape consumers expect."
-   Parquet produces maps of the form:
-       {\"external-id\"  \"square/order/123\", ...}
-   which we transform to:
-       {:sales-order/external-id \"square/order/123\", ...}"
   [row]
+  (let [pm (:payment-methods row)]
     (-> row
         (set/rename-keys
-       {"external-id" :sales-order/external-id
+         {:external-id :sales-order/external-id
-        "location"    :sales-order/location
+          :location    :sales-order/location
-        "total"       :sales-order/total
+          :total       :sales-order/total
-        "tax"         :sales-order/tax
+          :tax         :sales-order/tax
-        "tip"         :sales-order/tip
+          :tip         :sales-order/tip
-        "discount"    :sales-order/discount
+          :discount    :sales-order/discount
-        "service-charge" :sales-order/service-charge
+          :service-charge :sales-order/service-charge
-        "vendor"      :sales-order/vendor
+          :vendor      :sales-order/vendor
-        "client-code" :sales-order/client-code
+          :client-code :sales-order/client-code
-        "date"        :sales-order/date})
+          :date        :sales-order/date
-      (update :sales-order/date #(some-> % str))))
+          :source      :sales-order/source
+          :reference-link :sales-order/reference-link
+          :payment-methods :sales-order/payment-methods
+          :processors  :sales-order/processors
+          :categories  :sales-order/categories})
+        (update :sales-order/date #(some-> % str))
+        (dissoc :entity-type :_seq-no)
+         (assoc :sales-order/charges (payment-methods->charges pm)))))
 
 (defn build-where-clause [args]
-  (let [clauses [(when-let [c (:client-code args)]
+  (let [clauses (keep identity
-                   ["external_id.client = '" c "'"])
+                      [(when-let [c (:client-code args)]
+                         (str "external_id.client = '" c "'"))
                        (when-let [v (:vendor args)]
-                  ["external_id.vendor = '" (name v) "'"])
+                         (str "external_id.vendor = '" (name v) "'"))
                        (when-let [l (:location args)]
-                  ["location = '" l "'"])]
+                         (str "location = '" l "'"))])]
     (when (seq clauses)
       (str "WHERE " (str/join " AND " clauses)))))
 
@@ -46,32 +58,112 @@
 
 (defn raw-graphql-ids [args]
   (let [start (some-> (:start (:date-range args)) .toString)
-        end   (some-> (:end (:date-range args)) .substring 0 10)
+        end   (some-> (:end (:date-range args)) (.substring 0 10))
-        where (build-where-clause args)
-        sort  (build-sort-clause args)
         limit (or (:limit args) page-size)
-        offset (or (:offset args) 0)
+        offset (or (:offset args) 0)]
-        where-str (when where (str " " where))]
     (when start
       (let [result (pq/get-sales-orders start end
                                          {:client (:client-code args)
                                           :vendor (:vendor args)
                                           :location (:location args)
-                                          :sort sort
+                                          :sort (or (:sort args) "date")
                                           :order "DESC"
                                           :limit limit
                                           :offset offset})]
-        {:ids  (mapv #(str (:external_id %)) (:rows result))
+        {:ids  (mapv #(str (:external-id %)) (:rows result))
          :rows (:rows result)
          :count (:count result)}))))
 
 (defn graphql-results [rows _ids _args]
   (mapv <-row rows))
 
+(defn- extract-date-str [v]
+  (when v
+    (cond
+      (string? v) (if (> (count v) 10) (.substring v 0 10) v)
+      (instance? org.joda.time.DateTime v) (atime/unparse-local v atime/normal-date)
+      (instance? org.joda.time.LocalDate v) (atime/unparse-local v atime/normal-date)
+      (instance? java.util.Date v) (atime/unparse-local (coerce/to-date-time v) atime/normal-date)
+      (instance? java.time.LocalDate v) (.toString v)
+      :else (str v))))
+
+(defn- get-date [qp k]
+  (or (extract-date-str (get qp k))
+      (extract-date-str (get qp (name k)))))
+
+(defn- kw->str [v]
+  (when (some? v)
+    (if (keyword? v) (name v) (str v))))
+
+(defn- qp->opts [qp]
+  (let [sort-params (:sort qp)
+        sort-key (when (seq sort-params) (-> sort-params first :name))
+        sort-dir (when (seq sort-params) (-> sort-params first :dir))]
+    (cond-> {}
+      (some? (:client-code qp)) (assoc :client (kw->str (:client-code qp)))
+      (some? (:location qp)) (assoc :location (kw->str (:location qp)))
+      (not-empty (:payment-method qp)) (assoc :payment-method (:payment-method qp))
+      (some? (:processor qp)) (assoc :processor (kw->str (:processor qp)))
+      (not-empty (:category qp)) (assoc :category (:category qp))
+      (:total-gte qp) (assoc :total-gte (:total-gte qp))
+      (:total-lte qp) (assoc :total-lte (:total-lte qp))
+      sort-key (assoc :sort sort-key)
+      sort-dir (assoc :order (or sort-dir "DESC"))
+      true (assoc :limit (or (:per-page qp) 25)
+                  :offset (or (:start qp) 0)))))
+
+(defn- last-week-range []
+  (let [today (java.time.LocalDate/now)
+        end (.toString (.minusDays today 1))
+        start (.toString (.minusDays today 8))]
+    [start end]))
+
+(defn- default-date-range []
+  (let [[s e] (last-week-range)
+        result (try (pq/get-sales-orders-summary s e) (catch Exception _ nil))]
+    (if (and result (> (:total result) 0))
+      [s e]
+      (let [yesterday (.toString (.minusDays (java.time.LocalDate/of 2024 4 24) 1))
+            week-before (.toString (.minusDays (java.time.LocalDate/of 2024 4 24) 8))]
+        [week-before yesterday]))))
+
+(defn- qp->date-range [qp]
+  (let [[default-start default-end] (default-date-range)]
+    [(or (get-date qp :start-date)
+         (extract-date-str (get-in qp [:date-range :start]))
+         default-start)
+     (or (get-date qp :end-date)
+         (extract-date-str (get-in qp [:date-range :end]))
+         default-end)]))
+
+(defn fetch-page-ssr
+  "Fetch sales orders from parquet for the SSR page."
+  [request]
+  (let [qp (:query-params request)
+        raw-qp (some-> (:query-string request)
+                        ring-codec/form-decode
+                        (->> (into {} (remove (fn [[_ v]] (str/blank? v))))))
+        [start end] (qp->date-range (merge raw-qp qp))
+        opts (qp->opts qp)
+        result (pq/get-sales-orders start end opts)
+        rows (mapv <-row (:rows result))]
+    {:rows rows :count (:count result)}))
+
+(defn summarize-page-ssr
+  "Summarize all matching sales orders via parquet."
+  [request]
+  (let [qp (:query-params request)
+        raw-qp (some-> (:query-string request)
+                        ring-codec/form-decode
+                        (->> (into {} (remove (fn [[_ v]] (str/blank? v))))))
+        [start end] (qp->date-range (merge raw-qp qp))
+        opts (dissoc (qp->opts qp) :limit :offset :sort :order)]
+    (pq/get-sales-orders-summary start end opts)))
+
 (defn summarize-orders [rows]
   (when (seq rows)
-    (let [total (reduce + 0.0 (map #(or (:total %) 0.0) rows))
+    (let [total (reduce + 0.0 (map #(or (:sales-order/total %) 0.0) rows))
-          tax   (reduce + 0.0 (map #(or (:tax %) 0.0) rows))]
+          tax   (reduce + 0.0 (map #(or (:sales-order/tax %) 0.0) rows))]
       {:total total
        :tax   tax})))
 

src/clj/auto_ap/datomic/sales_orders_new.clj

@@ -1,224 +0,0 @@
-(ns auto-ap.datomic.sales-orders
-  (:require [auto-ap.datomic :refer [conn]]
-            [auto-ap.storage.parquet :as pq]
-            [clojure.data.json :as json]
-            [clojure.java.io :as io]))
-
-(defn <-row
-  "Convert a flat parquet row (string keys) into the
-   shape consumers expect.  Parquet produces maps of the form
-
-       {\"external-id\"  \"square/order/123\",
-        \"location\"     \"DT\",
-        \"total\"        50.0}
-
-   which we transform to:
-
-       {:sales-order/external-id \"square/order/123\",
-        :sales-order/location    \"DT\",
-        :sales-order/total       50.0}
-
-   Note: client, charges and other nested structures are not
-   available in the flat parquet rows.  When denormalisation
-   adds those columns we can restore the full consumer shape."
-  [row]
-  (-> row
-      (set/rename-keys
-       {"external-id" :sales-order/external-id
-        "location"    :sales-order/location
-        "total"       :sales-order/total
-        "tax"         :sales-order/tax
-        "tip"         :sales-order/tip})))
-
-(defn build-where-clause
-  "Build a SQL WHERE fragment from the fields that
-   parquet can filter on: external_id.client, vendor, location.
-
-   Returns either a predicate string e.g.
-       \"WHERE external_id.client = 'acme' AND vendor = 'square'\"
-   or nil when no applicable filters exist."
-  [args]
-  (let [clauses []
-        client   (:client-code args)
-        vendor   (:vendor args)
-        location (:location args)]
-    (when (or client vendor location)
-      (->> [[:client "external_id.client" client]
-            [:vendor  "external_id.vendor"  vendor]
-            [:location "location"           location]]
-           (keep (fn [[_ col v]]
-                   (when v [col v])))
-           (mapv #(str %1 " = '" %2 "'"))
-           (str/join " AND "))}))
-
-(defn build-sort-clause
-  "Map sort-key field names from args into SQL ORDER-BY fragments.
-
-   Supported fields map to parquet column names:
-     \"date\"    -> DATE
-     \"total\"   -> TOTAL
-     \"tax\"     -> TAX
-     \"tip\"     -> TIP
-     \"client\"  -> EXTERNAL_ID_CLIENT  (for flat client codes)
-     \"location\"-> LOCATION
-
-   Falls back to \"DATE DESC\" when the args do not specify
-   an explicit field."
-  [args]
-  ;; We delegate most of the SQL ordering work to get-sales-orders,
-  ;; which already defaults to DATE DESC.
-  (when-let [sorts (:sort args)]
-    (->> sorts
-         (keep (fn [{:keys [sort-key asc]}]
-                 (let [dir (if asc "ASC" "DESC")
-                       col (case sort-key
-                             "date"    "DATE"
-                             "total"   "TOTAL"
-                             "tax"     "TAX"
-                             "tip"     "TIP"
-                             "total-desc"  "TOTAL DESC"
-                             "source"  "SALE_SOURCE"
-                             "client"  "EXTERNAL_ID_CLIENT"
-                             "location" "LOCATION"
-                             nil)]                   ; unknown → skip
-                   (when col `[~col ~dir]))))
-         (interleave (repeat \,))
-         (apply str))))
-
-(defn build-pagination-clause
-  "Convert a Datomic-side pagination request into SQL-limit/offset
-   numbers.
-
-   Supports:
-     :start    → OFFSET
-     :count / :per-page  → LIMIT"
-  [args]
-  {:limit  (or (:count args) (:per-page args))
-   :offset (or (:start args) 0)})
-
-(defn- apply-pagination
-  "Safely re-implements the old datomic-side pagination logic.
-   Mutates a COPY of args so we can extract the resulting
-   cursor values for the response.
-
-   Returns {limit offset}"
-  [args]
-  (let [page     (build-pagination-clause args)
-        {:keys [limit offset]} page
-        client (:client-code args)]
-    ; In the new architecture pagination is applied server-side
-    ; by get-sales-orders via LIMIT/OFFSET, so this function
-    ; mainly exists as a thin wrapper for any remaining
-    ; in-memory re-paging concerns.
-    (if limit
-      (assoc page :limit (Integer. limit))
-      page)))
-
-(defn- build-options
-  "Assemble the opts map passed to pq/get-sales-orders:
-     {:client ..., :location ..., :vendor ...,
-      :limit 10, :offset 0, :sort, :order}"
-  [args]
-  (let [page    (apply-pagination args)
-        limit   (:limit page)
-        offset  (:offset page)
-        client  (:client-code args)]
-    (cond-> {:client  client
-             :vendor  (:vendor args)
-             :limit   limit
-             :offset  offset}
-      (:location args) (assoc :location (:location args))
-      (:sort args)     (assoc :sort "date")          ; let get-sales-orders handle order
-      true              (merge {:order    "DESC"
-                                :sort-key (:sort-key args)}))))
-
-(defn raw-graphql-ids
-  "Query sales-orders FROM parquet files via DuckDB instead of Datomic.
-
-   Filters applied at the parquet level:
-     - date-range → selects which parquet files to read
-     - client-code / vendor / location where clauses
-     - sort & pagination are delegated to get-sales-orders
-
-   category, processor, type-name filters require nested joins
-   that parquet does not support -- those fields are ignored.
-
-   Returns
-       {:ids  [string-key-for-each-matched-row]
-        :count int (total matches BEFORE pagination)}"
-  [args]
-  (let [start    (when-let [s (:start (:date-range args))]
-                   (.toString (.plusDays (java.time.LocalDate/parse s) -1)))
-        end      (when-let [e (:end   (:date-range args))]
-                   (-> e .substring 0 10))
-        where    (build-where-clause args)
-        options  (build-options args)
-        where-str (some-> where #(str " WHERE " %))]
-    (cond->> nil
-      ; Query rows from parquet with our filters and sort.
-       where-str   (pq/get-sales-orders
-                    start end
-                    (assoc options :sort-key where-str)))
-
-    ; For each row returned we need an ID string.
-    ; We use the external-id column as the lookup key.
-    (when-let [rows (:rows result)]
-      {:ids  (mapv #(str (:external_id %)) rows)
-       :rows rows
-       :count (:count result)})))
-
-(defn graphql-results
-  "Return the full payment-row data for the selected IDs.
-   Since we now read FROM parquet, we receive the raw row vector
-   and transform it.
-
-   The old signature [ids db args] is replaced by [rows id-keys _].
-   We ignore the database argument (Datomic pull is no longer
-   called)."
-  [rows _id-keys _args]
-  (->> rows
-       (mapv #(<-row %))))
-
-(defn summarize-orders
-  "Sum totals and discounts for the given ID-set.
-
-   This function still queries Datomic because the parquet-side
-   aggregation query would duplicate the WHERE logic.
-   If we want a pure parquet path here, add an
-   SQL-based aggregation in a follow-up."
-  [ids]
-  (let [[total tax]
-        (#'auto-ap.datomic/aggregate-sum ids)       ; uses dc/q internally
-        first]
-    {:total total
-     :tax   tax}))
-
-(defn get-graphql
-  "Entry-point: return [payments count summary].
-
-   The data flow is:
-     1. raw-graphql-ids  → parquet query → [:rows :count]
-     2. graphql-results  <- transform rows ← [:results <_id-keys> _args]
-     3. summarize-orders <- Datomic agg   ← [:total :tax]"
-  [args]
-  (let [{:keys [ids count']}
-        (mu/trace
-         ::get-sales-order-ids
-         []
-         (raw-graphql-ids args))]
-    [(->> (mu/trace ::get-results [] (graphql-results ids id-keys args))
-          matching-count
-          summarize-orders ids)])))
-
-(defn summarize-graphql
-  "Entry-point: return just the summary {:total :tax}.
-
-   Like get-graphql, this delegates to raw-graphql-ids
-   and then to summarize-orders."
-  [args]
-  (let [{:keys [ids count']}
-        (mu/trace
-         ::get-sales-order-ids
-         []
-         (raw-graphql-ids args))]
-    (summarize-orders ids)))

src/clj/auto_ap/ezcater/core.clj

@@ -21,16 +21,15 @@
                     :body    (json/write-str {"query" (v/graphql-query q)})
                     :as      :json})
       :body
-      :data
+      :data))
-      ))
 
 (defn get-caterers [integration]
   (:caterers (query integration {:venia/queries [{:query/data
-                                                  [:caterers [:name :uuid [:address [:name :street]]]]}]} )))
+                                                  [:caterers [:name :uuid [:address [:name :street]]]]}]})))
 
 (defn get-subscriptions [integration]
   (->> (query integration {:venia/queries [{:query/data
-                                            [:subscribers [:id [:subscriptions [:parentId :parentEntity :eventEntity :eventKey]] ]]}]} )
+                                            [:subscribers [:id [:subscriptions [:parentId :parentEntity :eventEntity :eventKey]]]]}]})
        :subscribers
        first
        :subscriptions))
@@ -39,7 +38,7 @@
   (map first (dc/q '[:find (pull ?i [:ezcater-integration/api-key
                                       :ezcater-integration/subscriber-uuid
                                       :db/id
-                                    :ezcater-integration/integration-status [:db/id]])
+                                      {:ezcater-integration/integration-status [:db/id]}])
                      :in $
                      :where [?i :ezcater-integration/api-key]]
                    (dc/db conn))))
@@ -95,7 +94,6 @@
                                                         :eventKey     'cancelled}}
                                   [[:subscription [:parentId :parentEntity :eventEntity :eventKey]]]]]})))))
 
-
 #_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
 (defn upsert-ezcater
   ([] (upsert-ezcater (get-integrations)))
@@ -122,7 +120,6 @@
                                           {:client/_ezcater-locations [:client/code]}]}]
            [:ezcater-caterer/uuid caterer-uuid]))
 
-
 (defn round-carry-cents [f]
   (with-precision 2 (double (.setScale (bigdec f) 2 java.math.RoundingMode/HALF_UP))))
 
@@ -140,17 +137,17 @@
      (* commision%
         0.01M
         (+
-           (-> order :totals :subTotal :subunits )
+         (-> order :totals :subTotal :subunits)
          (reduce +
                  0
-                   (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
+                 (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order)))))))))
 
 (defn ccp-fee [order]
   (round-carry-cents
    (* 0.000299M
       (+
-         (-> order :totals :subTotal :subunits )
+       (-> order :totals :subTotal :subunits)
-         (-> order :totals :salesTax :subunits )
+       (-> order :totals :salesTax :subunits)
        (reduce +
                0
                (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
@@ -158,7 +155,7 @@
 (defn order->sales-order [{{:keys [timestamp]} :event {:keys [orderItems]} :catererCart :keys [client-code client-location uuid] :as order}]
   (let [adjustment (round-carry-cents (- (+ (-> order :totals :subTotal :subunits (* 0.01))
                                             (-> order :totals :salesTax :subunits (* 0.01)))
-                                         (-> order :catererCart :totals :catererTotalDue )
+                                         (-> order :catererCart :totals :catererTotalDue)
                                          (commision order)
                                          (ccp-fee order)))
         service-charge (+ (commision order) (ccp-fee order))
@@ -169,7 +166,7 @@
       :external-id (str "ezcater/order/" client-code "-" client-location "-" uuid)
       :client      [:client/code client-code]
       :location    client-location
-     :reference-link (str (url/url  "https://ezmanage.ezcater.com/orders/" uuid ))
+      :reference-link (str (url/url  "https://ezmanage.ezcater.com/orders/" uuid))
       :line-items  [#:order-line-item
                      {:external-id (str "ezcater/order/" client-code "-" client-location "-" uuid "-" 0)
                       :item-name   "EZCater Catering"
@@ -199,7 +196,7 @@
       :tax            tax
       :tip            tip
       :returns        0.0
-     :vendor         :vendor/ccp-ezcater}}))
+      :vendor         :vendor/ccp-ezcater}))
 
 (defn- flatten-order-to-parquet! [order]
   "Flatten a sales-order into entity-type tagged maps and buffer to parquet."
@@ -242,7 +239,6 @@
                           :discount    (:order-line-item/discount li)
                           :sales-order-external-id so-ext-id})))))
 
-
 (defn get-by-id [integration id]
   (query
    integration
@@ -273,27 +269,20 @@
                                         [:currency
                                          :subunits]]]]]]
                       [:totals [[:customerTotalDue
-                                  [
+                                 [:currency
-                                   :currency
+                                  :subunits]]
-                                   :subunits
-                                   ]]
                                 [:pointOfSaleIntegrationFee
-                                  [
+                                 [:currency
-                                   :currency
+                                  :subunits]]
-                                   :subunits
-                                   ]]
                                 [:tip
                                  [:currency
                                   :subunits]]
                                 [:salesTax
-                                  [
+                                 [:currency
-                                   :currency
+                                  :subunits]]
-                                   :subunits
-                                   ]]
                                 [:salesTaxRemittance
                                  [:currency
-                                   :subunits
+                                  :subunits]]
-                                   ]]
                                 [:subTotal
                                  [:currency
                                   :subunits]]]]]]]}))
@@ -359,7 +348,7 @@
                                                                 "key" "accepted",
                                                                 "occurred_at" "2022-07-21T19:21:07.549Z"}
                                                     ezcater-order (lookup-order lookup-map)
-                                                    extant-order (dc/pull (dc/db conn) '[:sales-order/total
+                                                    extant-order (dc/pull (dc/db conn '[:sales-order/total]
                                                                                  :sales-order/tax
                                                                                  :sales-order/tip
                                                                                  :sales-order/discount
@@ -371,7 +360,7 @@
                                                                                   :sales-order/line-items [:order-line-item/external-id
                                                                                                            :order-line-item/total
                                                                                                            :order-line-item/tax
-                                                                                                                 :order-line-item/discount]}]
+                                                                                                           :order-line-item/discount]})
                                                                           [:sales-order/external-id order])
 
                                                     updated-order (-> (order->sales-order ezcater-order)

src/clj/auto_ap/jobs/sales_summaries.clj

@@ -40,18 +40,15 @@
                 (dc/db conn)
                 number)))
 
-
 (defn delete-all []
   @(dc/transact-async conn
                       (->>
                        (dc/q '[:find ?ss
                                :where [?ss :sales-summary/date]]
                              (dc/db conn))
-                (map (fn [[ ss]]
+                       (map (fn [[ss]]
                               [:db/retractEntity ss])))))
 
-
-
 (defn dirty-sales-summaries [c]
   (let [client-id (dc/entid (dc/db conn) c)]
     (->> (dc/index-pull  (dc/db conn)
@@ -188,8 +185,6 @@
        :ledger-mapped/ledger-side :ledger-side/credit
        :ledger-mapped/amount (- (+ total discount) tax)})))
 
-
-
 (defn get-fees [c date]
   (when-let [fee (get-fee c date)]
     {:db/id (str (java.util.UUID/randomUUID))
@@ -299,8 +294,7 @@
                      (filter identity)
                      (map (fn [z]
                             (assoc z :ledger-mapped/account (some-> z :sales-summary-item/category str/lower-case name->number lookup-account)
-                                   :sales-summary-item/manual? false))
+                                   :sales-summary-item/manual? false))))}]
-                          )) }]
         (if (seq (:sales-summary/items result))
           (do
             (alog/info ::upserting-summaries
@@ -310,7 +304,7 @@
 
 (comment
   ;; TODO: Move to test file or proper location
-  (let [c (auto-ap.datomic/pull-attr (dc/db @conn) :db/id [:client/code "NGCL" ])
+  (let [c (auto-ap.datomic/pull-attr (dc/db @conn) :db/id [:client/code "NGCL"])
         date #inst "2024-04-14T00:00:00-07:00"]
     (get-payment-items c date)))
 
@@ -322,9 +316,6 @@
                           (map (fn [[sos]]
                                  [:db/retractEntity sos])))))
 
-
-
-
 (comment
   (auto-ap.datomic/transact-schema conn)
 
@@ -367,19 +358,14 @@
           :in $ [?clients ?start-date ?end-date]
           :where  [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
           [?e :sales-order/line-items ?li]
-          [?li :order-line-item/item-name ?n] ]
+          [?li :order-line-item/item-name ?n]]
         (dc/db conn)
         [[(auto-ap.datomic/pull-attr (dc/db conn) :db/id [:client/code "NGCL"])] #inst "2024-04-11T00:00:00-07:00" #inst "2024-04-24T00:00:00-07:00"])
 
-@(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy}
+  @(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy}
                       {:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}])
 
-(auto-ap.datomic/transact-schema conn)
+  (auto-ap.datomic/transact-schema conn))
-
-  )
-
-
 
 (defn -main [& _]
   (execute "sales-summaries" sales-summaries-v2))
-  

src/clj/auto_ap/migration/sales_to_parquet.clj

@@ -10,14 +10,14 @@
       (write-dead-letter [flat])     ; write orphaned records"
   (:require [auto-ap.datomic :refer [conn]]
             [auto-ap.storage.parquet :as p]
-            [datomic.api :as dc]
+            [clojure.string :as str]
-            [clj-time.core :as time]))
+            [datomic.api :as dc]))
 
 (defn- fetch-all-sales-order-ids []
   "Query Datomic for all sales-order external-ids (as entity IDs).
     Returns a vector of entitity ids."
   (->> (dc/q '[:find ?e
-               :where [_ :sales-order/external-id ?_ext]]
+               :where [?e :sales-order/external-id _]]
              (dc/db conn))
        (map first)
        vec))
@@ -25,14 +25,16 @@
 (def ^:private sales-order-read
   '[:sales-order/external-id
     :sales-order/date
-    {:sales-order/client [:client/code]}
+    {:sales-order/client [:client/code :client/name]}
     :sales-order/location
-    :sales-order/vendor
+    {:sales-order/vendor [:vendor/name]}
     :sales-order/total
     :sales-order/tax
     :sales-order/tip
     :sales-order/discount
     :sales-order/service-charge
+    :sales-order/source
+    :sales-order/reference-link
     {:sales-order/charges
      [:charge/external-id
       :charge/type-name
@@ -40,7 +42,7 @@
       :charge/tax
       :charge/tip
       :charge/date
-      :charge/processor
+      {:charge/processor [:db/ident]}
       :charge/returns
       {:charge/client [:client/code]}]}
     {:sales-order/line-items
@@ -49,7 +51,7 @@
       :order-line-item/total
       :order-line-item/tax
       :order-line-item/discount
-      {:order-line-item/unit-price {}}
+      :order-line-item/unit-price
       :order-line-item/quantity
       :order-line-item/note]}])
 
@@ -61,29 +63,38 @@
                   sales-order-read
                   eids)))
 
-(defn- flatten-order-to-pieces! [order flat]
+(defn- flatten-order-to-pieces! [order date-str flat]
   "Flatten a pulled sales-order into :entity-type tagged maps.
     Appends to the existing flat vector, which is returned."
   (let [so-ext-id (:sales-order/external-id order)
-        so-date   (.toString (:sales-order/date order))
+        so-date   date-str
-        client-code (get-in order [:sales-order/client :client/code])]
+        client-code (get-in order [:sales-order/client :client/code])
-    ;; sales-order row
+        vendor-name (get-in order [:sales-order/vendor :vendor/name])
-    (swap! flat conj
+        charges (:sales-order/charges order)
+        items (:sales-order/line-items order)
+        payment-methods (->> charges (map :charge/type-name) distinct (str/join ","))
+        processors (->> charges (map #(get-in % [:charge/processor :db/ident])) (remove nil?) distinct (map name) (str/join ","))
+        categories (->> items (map :order-line-item/category) (remove nil?) distinct (str/join ","))]
+    (vswap! flat conj
             {:entity-type       "sales-order"
              :external-id       (str so-ext-id)
              :client-code       client-code
              :location          (:sales-order/location order)
-            :vendor            (:sales-order/vendor order)
+             :vendor            vendor-name
              :total             (:sales-order/total order)
              :tax               (:sales-order/tax order)
              :tip               (:sales-order/tip order)
              :discount          (:sales-order/discount order)
              :service-charge    (:sales-order/service-charge order)
-            :date              so-date})
+             :date              so-date
-    ;; charges & line-items
+             :source            (:sales-order/source order)
+             :reference-link    (:sales-order/reference-link order)
+             :payment-methods   payment-methods
+             :processors        processors
+             :categories        categories})
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
-        (swap! flat conj
+        (vswap! flat conj
                 {:entity-type       "charge"
                  :external-id       (str (get chg :charge/external-id))
                  :type-name         (get chg :charge/type-name)
@@ -93,18 +104,16 @@
                  :date              so-date
                  :processor         (get-in chg [:charge/processor :db/ident])
                  :sales-order-external-id (str so-ext-id)})
-        ;; charge returns → sales-refund rows
         (when-let [returns (:charge/returns chg)]
           (doseq [rt returns]
-            (swap! flat conj
+            (vswap! flat conj
                     {:entity-type       "sales-refund"
                      :type-name         (get rt :type-name)
                      :total             (get rt :total)
                      :sales-order-external-id (str so-ext-id)})))))
-    ;; line-items
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
-        (swap! flat conj
+        (vswap! flat conj
                 {:entity-type         "line-item"
                  :item-name           (get li :order-line-item/item-name)
                  :category            (get li :order-line-item/category)
@@ -116,19 +125,18 @@
 (defn -fetch-order-ids-for-date
   "Query Datomic for all sales-order eids on a given business date."
   [db date-str]
-  (let [day-ms (.toEpochSecond ^java.time.LocalDate (java.time.LocalDate/parse date-str))
+  (let [ld (java.time.LocalDate/parse date-str)
-        start (* day-ms 1000)
+        start (-> ld (.atStartOfDay (java.time.ZoneId/of "America/Los_Angeles")) .toInstant java.util.Date/from)
-        end (+ start (* 86400000))]
+        end (-> ld (.plusDays 1) (.atStartOfDay (java.time.ZoneId/of "America/Los_Angeles")) .toInstant java.util.Date/from)]
     (->> (dc/q '[:find ?e
-                 :in $ ?start-ms ?end-ms
+                 :in $ ?start ?end
-                 :where [_ :sales-order/date ?d]
+                 :where [?e :sales-order/date ?d]
-                        [(>= ?d ?start-ms)]
+                 [(>= ?d ?start)]
-                        [(<= ?d ?end-ms)]]
+                 [(< ?d ?end)]]
                db start end)
          (map first)
          vec)))
 
-
 (defn- date-seq [start end]
   "Seq of YYYY-MM-DD strings between start and end inclusive."
   (let [sd   (java.time.LocalDate/parse start)
@@ -138,9 +146,9 @@
     (for [i (range 0 (inc days))]
       (.toString (.plusDays sd i)))))
 
-(defn- write-day-by-day
+(defn write-day-by-day
   ([start-date end-date]
-   (write-day-by-day start-date end-date nil))
+   (write-day-by-day start-date end-date {}))
   ([start-date end-date opts]
    (let [all-dates  (set (or (opts :date-set) []))
          date-range (if (empty? all-dates)
@@ -156,12 +164,12 @@
            (let [orders (pull-sales-order-data batch)
                  flat   (volatile! [])]
              (doseq [o orders]
-               (flatten-order-to-pieces! o flat))
+                (flatten-order-to-pieces! o day flat))
              (doseq [r @flat]
                (p/buffer! (:entity-type r) r)))))
         (doseq [etype ["sales-order" "charge"
                        "line-item" "sales-refund"]]
-         (p/flush-to-parquet! etype))
+          (p/flush-to-parquet! etype day))
        (println "[migration]" day "complete"))
      {:status :completed :total-days (count date-range)})))
 
@@ -181,10 +189,11 @@
   "Flush all entity-type buffers, tracking counts."
   (let [etypes ["sales-order" "charge"
                 "line-item" "sales-refund"]
+        today (.toString (java.time.LocalDate/now))
         start  (p/total-buf-count)]
     (doseq [et etypes]
       (try
-        (p/flush-to-parquet! et)
+        (p/flush-to-parquet! et today)
         (catch Exception e
           (println "[migration/flush]" et "error:" (.getMessage e)))))
     {:records-flush (- (p/total-buf-count) start)}))
@@ -218,7 +227,7 @@
         (doseq [o (pull-sales-order-data order-ids)
                 :when (not (:sales-order/date o))]
           (let [flat (volatile! [])]
-            (flatten-order-to-pieces! o flat)
+             (flatten-order-to-pieces! o "unknown" flat)
             (doseq [r @flat]
               (p/buffer! "dead" r))))
         (write-day-by-day start-date end-date {:batch-size 100})

src/clj/auto_ap/routes/ezcater_xls.clj

@@ -55,8 +55,8 @@
         event-date (some-> (excel/xls-date->date event-date)
                            coerce/to-date-time
                            atime/as-local-time
-                           coerce/to-date )]
+                           coerce/to-date)]
-    (cond (and event-date client-id location ) 
+    (cond (and event-date client-id location)
           [:order #:sales-order
                    {:date           event-date
                     :external-id    (str "ezcater/order/" client-id "-" location "-" order-number)

src/clj/auto_ap/square/core3.clj

@@ -28,11 +28,9 @@
     "Authorization"  (str "Bearer " (:client/square-auth-token client))
     "Content-Type"   "application/json"}))
 
-
 (defn ->square-date [d]
   (f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d))
 
-
 (def manifold-api-stream
   (let [stream (s/stream 100)]
     (->> stream
@@ -105,7 +103,6 @@
                    :exception error))
       []))))
 
-
 (def item-cache (atom {}))
 
 (defn fetch-catalog [client i v]
@@ -125,13 +122,11 @@
     #(do (swap! item-cache assoc i %)
          %))))
 
-
 (defn fetch-catalog-cache [client i version]
   (if (get @item-cache i)
     (de/success-deferred (get @item-cache i))
     (fetch-catalog client i version)))
 
-
 (defn item->category-name-impl [client item version]
   (capture-context->lc
    (cond (:item_id (:item_variation_data item))
@@ -162,7 +157,6 @@
                      :item item)
            "Uncategorized"))))
 
-
 (defn item-id->category-name [client i version]
   (capture-context->lc
    (-> [client i]
@@ -227,7 +221,6 @@
                                  (concat (:orders result) continued-results))))
                    (:orders result)))))))
 
-
 (defn search
   ([client location start end]
    (capture-context->lc
@@ -251,11 +244,9 @@
                                   (concat (:orders result) continued-results))))
                     (:orders result))))))))
 
-
 (defn amount->money [amt]
   (* 0.01 (or (:amount amt) 0.0)))
 
-
 ;; to get totals: 
 (comment
   (reduce
@@ -416,7 +407,6 @@
                             :client client
                             :location location)))))))
 
-
 (defn get-payment [client p]
   (de/chain (manifold-api-call
              {:url (str "https://connect.squareup.com/v2/payments/" p)
@@ -425,7 +415,6 @@
             :body
             :payment))
 
-
 (defn continue-payout-entry-list [c l poi cursor]
   (capture-context->lc lc
                        (de/chain
@@ -674,8 +663,7 @@
                         (catch Exception e
                           (log/error ::buffer-failed
                                      :exception e
-                                     :order (:sales-order/external-id order)))))))))))
+                                     :order (:sales-order/external-id order))))))))))))
-
 
 (defn upsert-payouts
   ([client]
@@ -725,7 +713,6 @@
 
                    (log/info ::done-loading-refunds)))))))
 
-
 (defn get-cash-shift [client id]
   (de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id))
                                 :method :get
@@ -884,8 +871,6 @@
     d1
     d2))
 
-
-
 (defn remove-voided-orders
   ([client]
    (apply de/zip
@@ -920,22 +905,20 @@
                         (doseq [x (partition-all 100 results)]
                           (log/info ::removing-orders
                                     :count (count x))
-                          @(dc/transact-async conn x)))))
+                          @(dc/transact-async conn x)
                           (de/catch (fn [e]
                                       (log/warn ::couldnt-remove :error e)
-                                      nil) ))))))
+                                      nil)))))))))))
 
 #_(comment
  (require 'auto-ap.time-reader)
 
- @(let [[c [l]] (get-square-client-and-location "DBFS")  ]
+ @(let [[c [l]] (get-square-client-and-location "DBFS")]
-     (log/peek :x  [ c l])
+    (log/peek :x  [c l])
-     (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29")
+    (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29"))
 
-     ) 
+ @(let [[c [l]] (get-square-client-and-location "NGAK")]
-
+    (log/peek :x  [c l])
-  @(let [[c [l]] (get-square-client-and-location "NGAK")  ]
-     (log/peek :x  [ c l])
 
     (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
  (doseq [c (get-square-clients)]
@@ -943,8 +926,6 @@
      @(remove-voided-orders c)
      (catch Exception e
        nil)))
-
-  
  )
 
 (defn upsert-all [& clients]
@@ -1014,8 +995,6 @@
    [:clients clients]
    @(apply upsert-all clients)))
 
-
-
 (comment
   (defn refunds-raw-cont
     ([client l cursor so-far]
@@ -1045,7 +1024,6 @@
   (->>
    @(let [[c [l]] (get-square-client-and-location "NGGG")]
 
-
       (search c l (time/now) (time/plus (time/now) (time/days -1))))
 
    (filter (fn [r]
@@ -1055,7 +1033,6 @@
     (->>
      @(let [[c [l]] (get-square-client-and-location "NGGG")]
 
-
         (refunds-raw-cont c l nil []))
      (filter (fn [r]
                (str/starts-with? (:created_at r) "2024-03-14")))))
@@ -1090,12 +1067,7 @@
                                 [(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row)  (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
                               :separator \tab)
 
-
+;; => 
-
-
-  ;; => 
-  
-
 
   (require 'auto-ap.time-reader)
 
@@ -1104,26 +1076,15 @@
   (clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")]
                            l
 
-
                            (def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00"
                                            #clj-time/date-time "2025-02-28T00:00:00-08:00"))
-                           (take 10 (map #(first (deref (order->sales-order c l %))) z)))
+                           (take 10 (map #(first (deref (order->sales-order c l %))) z))))
-
-
-                         )
-  
-  
-
 
   (->> z
        (filter (fn [o]
                  (seq (filter (comp #{"OTHER"} :type) (:tenders o)))))
        (filter #(not (:name (:source %))))
-       (count)
+       (count))
-       
-       )
-  
-  
 
   (doseq [[code] (seq (dc/q '[:find ?code
                               :in $
@@ -1133,32 +1094,22 @@
                               [?o :sales-order/client ?c]
                               [?c :client/code ?code]]
                             (dc/db conn)))
-          :let [[c [l]] (get-square-client-and-location code)
+          :let [[c [l]] (get-square-client-and-location code)]
-                ]
           order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now))
-          :when (= "Invoices" (:name (:source order) ))
+          :when (= "Invoices" (:name (:source order)))
           :let [[sales-order] @(order->sales-order c l order)]]
 
     (when (should-import-order? order)
       (println "DATE IS" (:sales-order/date sales-order))
       (when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after?  #clj-time/date-time "2026-2-16T00:00:00-08:00"))
         (println "WOULD UPDATE" sales-order)
-        @(dc/transact auto-ap.datomic/conn [sales-order])
+        @(dc/transact auto-ap.datomic/conn [sales-order]))
-        )
+      #_@(dc/transact)
-      #_@(dc/transact )
+      (println "DONE")))
-      (println "DONE"))
-    
-    
-    )
 
   #_(filter (comp #{"OTHER"} :type) (mapcat :tenders z))
 
-
   @(let [[c [l]] (get-square-client-and-location "NGRY")]
      #_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
 
-     (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY" )))
+     (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY")))))
-
-     )
-  
-  )

src/clj/auto_ap/ssr/payments.clj

@@ -116,7 +116,6 @@
                                            :content "Debit"}]}))
     (exact-match-id* request)]])
 
-
 (def default-read '[*
                     [:payment/date :xform clj-time.coerce/from-date]
                     {:invoice-payment/_payment [* {:invoice-payment/invoice [*]}]}
@@ -212,7 +211,6 @@
                                                             '[(iol-ion.query/dollars= ?transaction-amount ?amount)]]}
                                             :args  [(:amount query-params)]})
 
-
                               (:status route-params)
                               (merge-query {:query {:in    ['?status]
                                                     :where ['[?e :payment/status ?status]]}
@@ -277,7 +275,6 @@
         {ids-to-retrieve :ids matching-count :count
          all-ids :all-ids} (fetch-ids db request)]
 
-
     [(->> (hydrate-results ids-to-retrieve db request))
      matching-count
      (sum-visible-pending all-ids)
@@ -285,8 +282,7 @@
                                              (:client request)
                                              (:client-id (:query-params request))
                                              (when (:client-code (:query-params request))
-                                            [:client/code (:client-code (:query-params request))])))
+                                               [:client/code (:client-code (:query-params request))])))]))
-     ]))
 
 (def query-schema (mc/schema
                    [:maybe [:map {:date-range [:date-range :start-date :end-date]}
@@ -327,7 +323,7 @@
                     (assoc-in (exact-match-id* request)  [1 :hx-swap-oob] true)])
                  :query-schema query-schema
                  :action-buttons (fn [request]
-                                   (let [[_ _ visible-in-float total-in-float ] (:page-results request)]
+                                   (let [[_ _ visible-in-float total-in-float] (:page-results request)]
                                      [(com/pill {:color :primary} " Visible in float "
                                                 (format "$%,.2f" visible-in-float))
                                       (com/pill {:color :secondary} " Total in float "
@@ -434,8 +430,6 @@
 
 (def row* (partial helper/row* grid-page))
 
-
-
 (comment
   (mc/decode query-schema {"exact-match-id" "123"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
   (mc/decode query-schema {} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
@@ -445,7 +439,6 @@
   (mc/decode query-schema {"payment-type" "food"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
   (mc/decode query-schema {"vendor" "87"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
 
-
   (mc/decode query-schema {"start-date" #inst "2023-12-21T08:00:00.000-00:00"} (mt/transformer main-transformer mt/strip-extra-keys-transformer)))
 
 (defn delete [{check :entity :as request identity :identity}]
@@ -459,7 +452,7 @@
    #(assert-can-see-client identity (:db/id (:payment/client check))))
   (notify-if-locked (:db/id (:payment/client check))
                     (:payment/date check))
-  (let [ removing-payments (mapcat (fn [x]
+  (let [removing-payments (mapcat (fn [x]
                                     (let [invoice (:invoice-payment/invoice x)
                                           new-balance (+ (:invoice/outstanding-balance invoice)
                                                          (:invoice-payment/amount x))]
@@ -578,7 +571,6 @@
                                                 (assoc-in [:query-params :start] 0)
                                                 (assoc-in [:query-params :per-page]  250))))
 
-
               :else
               selected)
         updated-count (void-payments-internal ids (:identity request))]
@@ -591,7 +583,7 @@
 
 (defn wrap-status-from-source [handler]
   (fn [{:keys [matched-current-page-route] :as request}]
-    (let [ request (cond-> request
+    (let [request (cond-> request
                     (= ::route/cleared-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/cleared)
                     (= ::route/pending-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/pending)
                     (= ::route/voided-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/voided)
@@ -618,7 +610,6 @@
     ::route/bulk-delete  (-> bulk-delete-dialog
                              (wrap-admin))
 
-
     ::route/table             (helper/table-route grid-page)}
    (fn [h]
      (-> h

src/clj/auto_ap/ssr/pos/sales_orders.clj

@@ -1,7 +1,7 @@
 (ns auto-ap.ssr.pos.sales-orders
   (:require
    [auto-ap.datomic
-    :refer [add-sorter-fields apply-pagination apply-sort-3 conn merge-query
+    :refer [add-sorter-fields apply-pagination apply-sort-3 merge-query
             pull-many query2]]
    [auto-ap.datomic.sales-orders :as d-sales]
    [auto-ap.query-params :as query-params :refer [wrap-copy-qp-pqp]]
@@ -17,7 +17,6 @@
    [auto-ap.time :as atime]
    [bidi.bidi :as bidi]
    [clj-time.coerce :as c]
-   [datomic.api :as dc]
    [malli.core :as mc]))
 
 (def query-schema (mc/schema
@@ -172,11 +171,8 @@
     charges))
 
 (defn fetch-page [request]
-  (let [db                                           (dc/db conn)
+  (let [{:keys [rows count]} (d-sales/fetch-page-ssr request)]
-        {ids-to-retrieve :ids matching-count :count} (fetch-ids db request)]
+    [rows count]))
-
-    [(->> (hydrate-results ids-to-retrieve db request))
-     matching-count]))
 
 
 (def grid-page
@@ -201,7 +197,7 @@
     :entity-name    "Sales orders"
     :route          :pos-sales-table
       :action-buttons (fn [request]
-                      (let [{:keys [total tax]} (d-sales/summarize-orders (:ids (fetch-ids (dc/db conn) request)))]
+                        (let [{:keys [total tax]} (d-sales/summarize-page-ssr request)]
                           (when (and total tax)
                             [(com/pill {:color :primary}
                                        (format "Total $%.2f" total))

src/clj/auto_ap/storage/parquet.clj

@@ -22,6 +22,10 @@
   (let [conn (DriverManager/getConnection "jdbc:duckdb:")
         stmt (.createStatement conn)]
     (.execute stmt "INSTALL httpfs; LOAD httpfs;")
+    (when-let [key (:aws-access-key-id env)]
+      (.execute stmt (str "SET s3_access_key_id='" key "'"))
+      (.execute stmt (str "SET s3_secret_access_key='" (:aws-secret-access-key env) "'"))
+      (.execute stmt (str "SET s3_region='" (or (:aws-region env) "us-east-1") "'")))
     (.close stmt)
     (.addShutdownHook (Runtime/getRuntime)
                       (Thread. #(fn [])))
@@ -124,12 +128,12 @@
   (->> @*buffers*
        vals (mapcat identity) count))
 
-(defn flush-to-parquet! [entity-type]
+(defn flush-to-parquet! [entity-type date-str]
   "Flush buffered records for entity-type to parquet + S3."
   (let [records (get @*buffers* entity-type [])]
     (if (empty? records)
       {:status :no-records}
-      (let [date-str   (.toString (LocalDate/now))
+      (let [date-str     (or date-str (.toString (LocalDate/now)))
             jsonl-file   (io/file "/tmp"
                                   (str entity-type "-" date-str ".jsonl"))
             parquet-file (io/file "/tmp"
@@ -150,17 +154,19 @@
           (.delete ^java.io.File parquet-file)
           {:key s3-key :status :ok}
           (catch Exception e
-            (throw (ex-info "Flush failed" {:entity-type entity-type}
+            (throw (ex-info "Flush failed"
-                            :error (.getMessage e)))))))))
+                            {:entity-type entity-type
+                             :error (.getMessage e)}))))))))
 
 (defn flush-by-date! []
   "Flush all entity types for today."
   (let [etypes    ["sales-order" "charge"
                    "line-item" "sales-refund"]
+        today     (.toString (LocalDate/now))
         flushed   (into #{}
                         (keep (fn [et]
                                 (let [{:keys [status]}
-                                      (flush-to-parquet! et)]
+                                      (flush-to-parquet! et today)]
                                   (when (= status :ok)
                                     et))))
                         etypes)]
@@ -188,8 +194,9 @@
                                (let [f (io/file
                                         (wal-dir)
                                         (str et ".jsonl"))]
+                                 (when (.exists f)
                                    [et (slurp f)])))
-                      etypes))]
+                      etypes)))]
     (swap! *buffers* merge loaded)))
 
 (defn get-unflushed-count []
@@ -213,36 +220,90 @@
 (defn today []
   (.toString (LocalDate/now)))
 
+(defn- parquet-glob [entity-type start-date end-date]
+  "Build a glob pattern or explicit file list for the date range.
+   Uses glob patterns for ranges > 60 days; explicit list otherwise."
+  (let [days (-> (LocalDate/parse end-date)
+                 (.toEpochDay)
+                 (- (.toEpochDay (LocalDate/parse start-date)))
+                 inc)]
+    (if (> days 60)
+      (let [prefix (format "s3://%s/sales-details/%s/" *bucket* entity-type)
+            sy (-> (LocalDate/parse start-date) .getYear)
+            ey (-> (LocalDate/parse end-date) .getYear)]
+        (if (= sy ey)
+          [(format "%s%d-*.parquet" prefix sy)]
+          (vec
+           (for [y (range sy (inc ey))]
+             (format "%s%d-*.parquet" prefix y)))))
+      (vec
+       (map (fn [d]
+              (format "'s3://%s/sales-details/%s/%s.parquet'"
+                      *bucket* entity-type d))
+            (date-seq start-date end-date))))))
+
 (defn parquet-query [entity-type start-date end-date]
   "Build SQL to read all parquet files in date range.
    Returns map with :sql and :count-sql keys."
-  (let [date-strs (date-seq start-date end-date)
+  (let [globs     (parquet-glob entity-type start-date end-date)
-        urls      (vec
+        use-glob? (some #(.endsWith ^String % "*.parquet") globs)
-                   (map #(format "'s3://%%s/sales-details/%%s/%%s.parquet'"
+        base      (if use-glob?
-                                 *bucket* entity-type %)
+                    (format "SELECT * FROM read_parquet(%s, union_by_name=true)"
-                        date-strs))
+                            (if (= (count globs) 1)
-        sql       (str "SELECT * FROM read_parquet(["
+                              (format "'%s'" (first globs))
-                       (str/join ", " urls)
+                              (format "[%s]"
-                       "])")]
+                                      (str/join ", " (map #(format "'%s'" %) globs)))))
+                    (format "SELECT * FROM read_parquet([%s])"
+                            (str/join ", " globs)))
+        add-date-filter (fn [sql]
+                          (if (> (-> (LocalDate/parse end-date)
+                                     (.toEpochDay)
+                                     (- (.toEpochDay (LocalDate/parse start-date)))
+                                     inc)
+                               60)
+                            (format "%s WHERE date >= '%s' AND date <= '%s'"
+                                    sql start-date end-date)
+                            sql))
+        sql (add-date-filter base)]
     {:sql       sql
      :count-sql (format "SELECT COUNT(*) FROM (%s) t" sql)}))
 
-(defn- build-where-clause [opts field-pairs]
+(defn- like-clause [col v]
-  "Build SQL WHERE clause from opts map.
+  (str "\"" col "\" LIKE '%" v "%'"))
-   fields-with-keys is vector of [:field-key :env-var-name]."
+
-  (let [clauses (keep
+(defn- build-sales-orders-where [opts]
-                 (fn [[key env]]
+  (let [eq-clauses (keep
+                    (fn [[key col]]
                       (let [v (get opts key)]
                         (when v
-                       (str env " = '" v "'"))))
+                          (str "\"" col "\" = '" v "'"))))
-                 field-pairs)]
+                    [[:client "client-code"]
-    (when (seq clauses)
+                     [:vendor "vendor"]
-      (str " WHERE " (str/join " AND " clauses)))))
+                     [:location "location"]])
+        like-clauses (keep
+                      (fn [[key col]]
+                        (let [v (get opts key)]
+                          (when v
+                            (like-clause col v))))
+                      [[:payment-method "payment-methods"]
+                       [:processor "processors"]
+                       [:category "categories"]])
+        range-clauses (keep
+                       (fn [[key col op]]
+                         (let [v (get opts key)]
+                           (when v
+                             (str "\"" col "\" " op " " v))))
+                       [[:total-gte "total" ">="]
+                        [:total-lte "total" "<="]])
+        all-clauses (concat eq-clauses like-clauses range-clauses)]
+    (when (seq all-clauses)
+      (str " WHERE " (str/join " AND " all-clauses)))))
 
 (defn get-sales-orders
   ([start-date end-date]
    (get-sales-orders start-date end-date {}))
   ([start-date end-date opts]
+   (try
      (let [q               (parquet-query "sales-order"
                                           start-date end-date)
            base-sql         (:sql q)
@@ -251,12 +312,7 @@
            order            (get opts :order "DESC")
            limit            (get opts :limit)
            offset           (get opts :offset)
-          where-str        (build-where-clause
+           where-str        (build-sales-orders-where opts)
-
-                            opts
-                            [[:client "external_id.client"]
-                             [:vendor "external_id.vendor"]
-                             [:location "location"]])
            full-sql         (if where-str
                               (str base-sql where-str)
                               base-sql)
@@ -272,7 +328,27 @@
         :count  (or
                  (int
                   (query-scalar
-                  full-count)) 0)})))
+                   full-count)) 0)})
+     (catch Exception _
+       {:rows [] :count 0}))))
+
+(defn get-sales-orders-summary
+  ([start-date end-date]
+   (get-sales-orders-summary start-date end-date {}))
+  ([start-date end-date opts]
+   (try
+     (let [q          (parquet-query "sales-order" start-date end-date)
+           base-sql   (:sql q)
+           where-str  (build-sales-orders-where opts)
+           full-sql   (if where-str
+                        (str base-sql where-str)
+                        base-sql)
+           sum-sql    (format "SELECT COALESCE(SUM(total), 0) as total, COALESCE(SUM(tax), 0) as tax FROM (%s) t" full-sql)
+           row        (first (query-rows sum-sql))]
+       {:total (or (:total row) 0.0)
+        :tax   (or (:tax row) 0.0)})
+     (catch Exception _
+       {:total 0.0 :tax 0.0}))))
 
 (defn query-deduped [entity-type start-date end-date]
   "Query records deduplicated by external-id (latest _seq_no wins)."