fix(sales): address review findings — 10 safe_auto fixes

- Fix pq-files using date-seq (daily) vs actual monthly parquet partitions
- Fix safe-cleanup-all destructure [[_ y m]] -> [[y m]] against [year month]
- Fix shutdown hook no-op: Thread wrapping #(fn []) now actually closes conn
- Fix query-deduped: PARTITION BY "external-id" not sales_order.external_id
- Fix :client_code -> :client-code key mismatch in get-payment-items-parquet
- Fix object-exists? downloading full S3 objects; use head-object instead
- Fix date-seq silently producing wrong range when start > end; now throws
- Remove duplicate private get-fees that shadowed public version
- Deduplicate date-seq: remove from sales_to_parquet, use p/date-seq
- Wrap run-perf-tests in (comment ...) to prevent execution on lein test
- Make month-seq public so sales_summaries.clj can use it

.claude/skills/clojure-eval/SKILL.md

@@ -0,0 +1,174 @@
+---
+name: clojure-eval
+description: Evaluate Clojure code via nREPL using clj-nrepl-eval. Use this when you need to test code, check if edited files compile, verify function behavior, or interact with a running REPL session.
+---
+
+# Clojure REPL Evaluation
+
+## When to Use This Skill
+
+Use this skill when you need to:
+- **Verify that edited Clojure files compile and load correctly**
+- Test function behavior interactively
+- Check the current state of the REPL
+- Debug code by evaluating expressions
+- Require or load namespaces for testing
+- Validate that code changes work before committing
+
+## How It Works
+
+The `clj-nrepl-eval` command evaluates Clojure code against an nREPL server. **Session state persists between evaluations**, so you can require a namespace in one evaluation and use it in subsequent calls. Each host:port combination maintains its own session file.
+
+## Instructions
+
+### 0. Discover and select nREPL server
+
+First, discover what nREPL servers are running in the current directory:
+
+```bash
+clj-nrepl-eval --discover-ports
+```
+
+This will show all nREPL servers (Clojure, Babashka, shadow-cljs, etc.) running in the current project directory.
+
+**Then use the AskUserQuestion tool:**
+
+- **If ports are discovered:** Prompt user to select which nREPL port to use:
+  - **question:** "Which nREPL port would you like to use?"
+  - **header:** "nREPL Port"
+  - **options:** Present each discovered port as an option with:
+    - **label:** The port number 
+    - **description:** The server type and status (e.g., "Clojure nREPL server in current directory")
+  - Include up to 4 discovered ports as options
+  - The user can select "Other" to enter a custom port number
+
+- **If no ports are discovered:** Prompt user how to start an nREPL server:
+  - **question:** "No nREPL servers found. How would you like to start one?"
+  - **header:** "Start nREPL"
+  - **options:**
+    - **label:** "deps.edn alias", **description:** "Find and use an nREPL alias in deps.edn"
+    - **label:** "Leiningen", **description:** "Start nREPL using 'lein repl'"
+  - The user can select "Other" for alternative methods or if they already have a server running on a specific port
+
+IMPORTANT: IF you start a REPL do not supply a port let the nREPL start and return the port that it was started on.
+
+### 1. Evaluate Clojure Code
+
+> Evaluation automatically connects to the given port
+
+Use the `-p` flag to specify the port and pass your Clojure code.
+
+**Recommended: Pass code as a command-line argument:**
+```bash
+clj-nrepl-eval -p <PORT> "(+ 1 2 3)"
+```
+
+**For multiple expressions (single line):**
+```bash
+clj-nrepl-eval -p <PORT> "(def x 10) (+ x 20)"
+```
+
+**Alternative: Using heredoc (may require permission approval for multiline commands):**
+```bash
+clj-nrepl-eval -p <PORT> <<'EOF'
+(def x 10)
+(+ x 20)
+EOF
+```
+
+**Alternative: Via stdin pipe:**
+```bash
+echo "(+ 1 2 3)" | clj-nrepl-eval -p <PORT>
+```
+
+### 2. Display nREPL Sessions
+
+**Discover all nREPL servers in current directory:**
+```bash
+clj-nrepl-eval --discover-ports
+```
+Shows all running nREPL servers in the current project directory, including their type (clj/bb/basilisp) and whether they match the current working directory.
+
+**Check previously connected sessions:**
+```bash
+clj-nrepl-eval --connected-ports
+```
+Shows only connections you have made before (appears after first evaluation on a port).
+
+### 3. Common Patterns
+
+**Require a namespace (always use :reload to pick up changes):**
+```bash
+clj-nrepl-eval -p <PORT> "(require '[my.namespace :as ns] :reload)"
+```
+
+**Test a function after requiring:**
+```bash
+clj-nrepl-eval -p <PORT> "(ns/my-function arg1 arg2)"
+```
+
+**Check if a file compiles:**
+```bash
+clj-nrepl-eval -p <PORT> "(require 'my.namespace :reload)"
+```
+
+**Multiple expressions:**
+```bash
+clj-nrepl-eval -p <PORT> "(def x 10) (* x 2) (+ x 5)"
+```
+
+**Complex multiline code (using heredoc):**
+```bash
+clj-nrepl-eval -p <PORT> <<'EOF'
+(def x 10)
+(* x 2)
+(+ x 5)
+EOF
+```
+*Note: Heredoc syntax may require permission approval.*
+
+**With custom timeout (in milliseconds):**
+```bash
+clj-nrepl-eval -p <PORT> --timeout 5000 "(long-running-fn)"
+```
+
+**Reset the session (clears all state):**
+```bash
+clj-nrepl-eval -p <PORT> --reset-session
+clj-nrepl-eval -p <PORT> --reset-session "(def x 1)"
+```
+
+## Available Options
+
+- `-p, --port PORT` - nREPL port (required)
+- `-H, --host HOST` - nREPL host (default: 127.0.0.1)
+- `-t, --timeout MILLISECONDS` - Timeout (default: 120000 = 2 minutes)
+- `-r, --reset-session` - Reset the persistent nREPL session
+- `-c, --connected-ports` - List previously connected nREPL sessions
+- `-d, --discover-ports` - Discover nREPL servers in current directory
+- `-h, --help` - Show help message
+
+## Important Notes
+
+- **Prefer command-line arguments:** Pass code as quoted strings: `clj-nrepl-eval -p <PORT> "(+ 1 2 3)"` - works with existing permissions
+- **Heredoc for complex code:** Use heredoc (`<<'EOF' ... EOF`) for truly multiline code, but note it may require permission approval
+- **Sessions persist:** State (vars, namespaces, loaded libraries) persists across invocations until the nREPL server restarts or `--reset-session` is used
+- **Automatic delimiter repair:** The tool automatically repairs missing or mismatched parentheses
+- **Always use :reload:** When requiring namespaces, use `:reload` to pick up recent changes
+- **Default timeout:** 2 minutes (120000ms) - increase for long-running operations
+- **Input precedence:** Command-line arguments take precedence over stdin
+
+## Typical Workflow
+
+1. Discover nREPL servers: `clj-nrepl-eval --discover-ports`
+2. Use **AskUserQuestion** tool to prompt user to select a port
+3. Require namespace:
+   ```bash
+   clj-nrepl-eval -p <PORT> "(require '[my.ns :as ns] :reload)"
+   ```
+4. Test function:
+   ```bash
+   clj-nrepl-eval -p <PORT> "(ns/my-fn ...)"
+   ```
+5. Iterate: Make changes, re-require with `:reload`, test again
+

.claude/skills/clojure-eval/examples.md

@@ -0,0 +1,82 @@
+# clj-nrepl-eval Examples
+
+## Discovery
+
+```bash
+clj-nrepl-eval --connected-ports
+```
+
+## Heredoc for Multiline Code
+
+```bash
+clj-nrepl-eval -p 7888 <<'EOF'
+(defn greet [name]
+  (str "Hello, " name "!"))
+
+(greet "Claude")
+EOF
+```
+
+### Heredoc Simplifies String Escaping
+
+Heredoc avoids shell escaping issues with quotes, backslashes, and special characters:
+
+```bash
+# With heredoc - no escaping needed
+clj-nrepl-eval -p 7888 <<'EOF'
+(def regex #"\\d{3}-\\d{4}")
+(def message "She said \"Hello!\" and waved")
+(def path "C:\\Users\\name\\file.txt")
+(println message)
+EOF
+
+# Without heredoc - requires complex escaping
+clj-nrepl-eval -p 7888 "(def message \"She said \\\"Hello!\\\" and waved\")"
+```
+
+## Working with Project Namespaces
+
+```bash
+# Test a function after requiring
+clj-nrepl-eval -p 7888 <<'EOF'
+(require '[clojure-mcp-light.delimiter-repair :as dr] :reload)
+(dr/delimiter-error? "(defn foo [x]")
+EOF
+```
+
+## Verify Compilation After Edit
+
+```bash
+# If this returns nil, the file compiled successfully
+clj-nrepl-eval -p 7888 "(require 'clojure-mcp-light.hook :reload)"
+```
+
+## Session Management
+
+```bash
+# Reset session if state becomes corrupted
+clj-nrepl-eval -p 7888 --reset-session
+```
+
+## Common Workflow Patterns
+
+### Load, Test, Iterate
+
+```bash
+# After editing a file, reload and test in one command
+clj-nrepl-eval -p 7888 <<'EOF'
+(require '[my.namespace :as ns] :reload)
+(ns/my-function test-data)
+EOF
+```
+
+### Run Tests After Changes
+
+```bash
+clj-nrepl-eval -p 7888 <<'EOF'
+(require '[my.project.core :as core] :reload)
+(require '[my.project.core-test :as test] :reload)
+(clojure.test/run-tests 'my.project.core-test)
+EOF
+```
+

.gitattributes

@@ -0,0 +1,3 @@
+
+# Use bd merge for beads JSONL files
+.beads/issues.jsonl merge=beads

.opencode/skills/gitea-tea/SKILL.md

@@ -0,0 +1,127 @@
+---
+name: gitea-tea
+description: Use tea CLI to create, manage, and checkout Gitea pull requests. Use this when opening a PR, managing PRs, or checking out PRs on the gitea remote (gitea.story-basking.ts.net).
+---
+
+# Gitea Tea CLI Skill
+
+This skill covers using `tea` (Gitea's official CLI) for pull request workflows in this project.
+
+## When to Use This Skill
+
+Use this skill when you need to:
+- Create a PR from a working branch to master on the gitea remote
+- Open, list, or view PRs
+- Checkout a PR locally for review or iteration
+- Manage PR state (close, reopen, merge)
+
+## Project Setup
+
+The gitea remote is `gitea.story-basking.ts.net` with repo slug `notid/integreat`. The default push remote is **gitea**, NOT origin and NOT deploy.
+
+In this project's environment:
+- Gitea login is pre-configured for `gitea.story-basking.ts.net`
+- Repo slug: `notid/integreat`
+- Target branch for PRs: `master`
+- The git remote named `gitea` points to this instance
+
+## Creating a PR
+
+Use `tea pulls create` to open a PR from the current branch to master. Always specify `-r notid/integreat -b master`:
+
+```bash
+tea pulls create -r notid/integreat -b master --title "Title" --description "Body"
+```
+
+Common flags:
+- `-t, --title` - PR title
+- `-d, --description` - PR body/description (use heredoc or file for long descriptions)
+- `-a, --assignees` - Comma-separated usernames to assign
+- `-L, --labels` - Comma-separated labels to apply
+- `-m, --milestone` - Milestone to assign
+
+**Writing a multiline description:**
+
+```bash
+tea pulls create -r notid/integreat -b master \
+  -t "feat: add feature" \
+  -d "$(cat <<'EOF'
+## Summary
+- Bullet point one
+- Bullet point two
+EOF
+)"
+```
+
+Or write the body to a temp file first and reference it.
+
+## Listing PRs
+
+```bash
+tea pulls list -r notid/integreat                          # List open PRs
+tea pulls list -r notid/integreat --state all              # All PRs
+tea pulls list -r notid/integreat --limit 10 -o simple     # Limit output, simple format
+```
+
+## Opening a PR in Browser
+
+```bash
+tea open pr <number> -r notid/integreat
+tea open pr create -r notid/integreat    # Open web UI to create a PR
+```
+
+## Checking Out a PR Locally
+
+```bash
+tea pulls checkout <number> -r notid/integreat
+```
+
+This fetches and checks out the PR branch locally.
+
+## Managing PR State
+
+**Close a PR:**
+```bash
+tea pulls close <number> -r notid/integreat --confirm
+```
+
+**Reopen a closed PR:**
+```bash
+tea pulls reopen <number> -r notid/integreat --confirm
+```
+
+**Merge a PR:**
+```bash
+tea pulls merge <number> -r notid/integreat --confirm
+```
+
+**Edit a PR (title, description, etc.):**
+```bash
+tea pulls edit <number> -r notid/integreat --title "New title" --description "New body"
+```
+
+## Full PR Creation Workflow
+
+1. Ensure the branch is pushed to gitea:
+   ```bash
+   git push gitea <branch-name>
+   ```
+
+2. Create the PR with tea:
+   ```bash
+   tea pulls create -r notid/integreat -b master \
+     --title "feat: description of change" \
+     --description "Detailed PR body here"
+   ```
+
+3. Open the PR in browser to verify:
+   ```bash
+   tea open pr <number> -r notid/integreat
+   ```
+
+## Tips
+
+- Always use `-r notid/integreat` to specify the repo explicitly
+- Use `-b master` to set the target branch (default may differ)
+- The `--confirm` flag is required for destructive actions (close, merge)
+- Use `-o simple`, `-o json`, `-o table`, etc. to control output format

AGENTS.md

@@ -0,0 +1,9 @@
+## Pull Requests on Gitea
+
+This project uses **gitea story-basking.ts net** as the primary remote for PRs. Use `tea` (the Gitea CLI) to create and manage pull requests. The gitea remote is the one you push to, NOT origin and NOT deploy.
+
+**When opening a PR**, load and follow the **gitea-tea** skill. In short:
+- Target branch is always `master`
+- Use `tea pulls create -r notid/integreat -b master --title "..." --description "..."`
+
+Use 'bd' for task tracking

docs/plans/2026-02-08-refactor-rebase-invoice-templates-integrate-branches-plan.md

@@ -0,0 +1,376 @@
+---
+title: Rebase Invoice Templates, Merge to Master, and Integrate Branches
+type: refactor
+date: 2026-02-08
+---
+
+# Rebase Invoice Templates, Merge to Master, and Integrate Branches
+
+## Overview
+
+This plan outlines a series of git operations to reorganize the branch structure by:
+1. Creating a rebase commit with all invoice template changes
+2. Applying those changes onto `master`
+3. Removing them from the current `clauding` branch
+4. Merging `master` back into `clauding`
+5. Finally merging `clauding` into `get-transactions2-page-working`
+
+## Current State
+
+### Branch Structure (as of Feb 8, 2026)
+
+```
+master (dc021b8c)
+  ├─ deploy/master (dc021b8c)
+  └─ (other branches)
+       └─ clauding (0155d91e) - HEAD
+            ├─ 16 commits ahead of master
+            └─ Contains invoice template work for Bonanza Produce
+                 ├─ db1cb194 Add Bonanza Produce invoice template
+                 ├─ ec754233 Improve Bonanza Produce customer identifier extraction
+                 ├─ af7bc324 Add location extraction for Bonanza Produce invoices
+                 ├─ 62107c99 Extract customer name and address for Bonanza Produce
+                 ├─ 7ecd569e Add invoice-template-creator skill for automated template generation
+                 └─ 0155d91e Add Bonanza Produce multi-invoice statement template
+```
+
+### Merge Base
+
+- **Merge base between `clauding` and `master`**: `dc021b8c`
+- **Commits on `clauding` since merge base**: 16 commits
+- **Invoice template commits**: 6 commits (db1cb194 through 0155d91e)
+
+## Problem Statement
+
+The current branch structure has:
+1. Invoice template work mixed with other feature development in `clauding`
+2. No clear separation between invoice template changes and transaction page work
+3. A desire to get invoice template changes merged to `master` independently
+4. A need to reorganize branches to prepare for merging `get-transactions2-page-working`
+
+## Proposed Solution
+
+Use git rebase and merge operations to create a cleaner branch hierarchy:
+
+1. **Create a new branch** (`invoice-templates-rebased`) with only invoice template commits
+2. **Rebase those commits** onto current `master`
+3. **Merge** this clean branch to `master`
+4. **Remove invoice template commits** from `clauding` branch
+5. **Merge `master` into `clauding`** to sync
+6. **Merge `clauding` into `get-transactions2-page-working`**
+
+## Implementation Steps
+
+### Phase 1: Extract and Rebase Invoice Templates
+
+#### Step 1.1: Identify Invoice Template Commits
+
+```bash
+# From clauding branch, find the range of invoice template commits
+git log --oneline --reverse dc021b8c..clauding
+```
+
+**Invoice template commits to extract** (6 commits in order):
+1. `db1cb194` - Add Bonanza Produce invoice template
+2. `ec754233` - Improve Bonanza Produce customer identifier extraction
+3. `af7bc324` - Add location extraction for Bonanza Produce invoices
+4. `62107c99` - Extract customer name and address for Bonanza Produce
+5. `7ecd569e` - Add invoice-template-creator skill for automated template generation
+6. `0155d91e` - Add Bonanza Produce multi-invoice statement template
+
+#### Step 1.2: Create Rebased Branch
+
+```bash
+# Create a new branch from master with only invoice template commits
+git checkout master
+git pull origin master  # Ensure master is up to date
+git checkout -b invoice-templates-rebased
+
+# Cherry-pick the invoice template commits in order
+git cherry-pick db1cb194
+git cherry-pick ec754233
+git cherry-pick af7bc324
+git cherry-pick 62107c99
+git cherry-pick 7ecd569e
+git cherry-pick 0155d91e
+
+# Resolve any conflicts that arise during cherry-pick
+# Run tests after each cherry-pick if conflicts occur
+```
+
+#### Step 1.3: Verify Rebased Branch
+
+```bash
+# Verify the commits are correctly applied
+git log --oneline master..invoice-templates-rebased
+
+# Run tests to ensure invoice templates still work
+lein test auto-ap.parse.templates-test
+```
+
+#### Step 1.4: Merge to Master
+
+```bash
+# Merge the clean invoice templates to master
+git checkout master
+git merge invoice-templates-rebased --no-edit
+
+# Push to remote
+git push origin master
+```
+
+### Phase 2: Clean Up Clauding Branch
+
+#### Step 2.1: Remove Invoice Template Commits from Clauding
+
+```bash
+# From clauding branch, find the commit before the first invoice template
+git log --oneline clauding | grep -B1 "db1cb194"
+
+# Suppose that's commit X, rebase clauding to remove invoice templates
+git checkout clauding
+
+# Option A: Interactive rebase (recommended for cleanup)
+git rebase -i <commit-before-invoice-templates>
+
+# In the editor, delete lines corresponding to invoice template commits:
+# db1cb194
+# ec754233
+# af7bc324
+# 62107c99
+# 7ecd569e
+# 0155d91e
+
+# Save and exit to rebase
+
+# Resolve any conflicts that occur
+# Run tests after rebase
+```
+
+**OR**
+
+```bash
+# Option B: Hard reset to commit before invoice templates
+# Identify the commit hash before db1cb194 (let's call it COMMIT_X)
+git reset --hard COMMIT_X
+
+# Then add back any non-invoice template commits from clauding
+# (commits after the invoice templates that should remain)
+git cherry-pick <non-invoice-commits-if-any>
+```
+
+#### Step 2.2: Verify Clauding Branch Cleanup
+
+```bash
+# Verify invoice template commits are removed
+git log --oneline | grep -i "bonanza"  # Should be empty
+
+# Verify other commits remain
+git log --oneline -20
+
+# Run tests to ensure nothing broke
+lein test
+```
+
+#### Step 2.3: Force Push Updated Clauding
+
+```bash
+# Force push the cleaned branch (use --force-with-lease for safety)
+git push --force-with-lease origin clauding
+```
+
+### Phase 3: Sync Clauding with Master
+
+#### Step 3.1: Merge Master into Clauding
+
+```bash
+git checkout clauding
+git merge master --no-edit
+
+# Resolve any conflicts
+# Run tests
+```
+
+#### Step 3.2: Push Synced Clauding
+
+```bash
+git push origin clauding
+```
+
+### Phase 4: Final Merge to get-transactions2-page-working
+
+#### Step 4.1: Merge Clauding to get-transactions2-page-working
+
+```bash
+git checkout get-transactions2-page-working
+git merge clauding --no-edit
+
+# Resolve any conflicts
+# Run tests
+```
+
+#### Step 4.2: Push Final Branch
+
+```bash
+git push origin get-transactions2-page-working
+```
+
+## Acceptance Criteria
+
+### Pre-operations Validation
+- [ ] All invoice template commits identified correctly (6 commits)
+- [ ] Merge base commit (`dc021b8c`) confirmed
+- [ ] Current branch state documented
+- [ ] Team notified of branch manipulation
+
+### Post-Rebase Validation
+- [ ] `invoice-templates-rebased` branch created from `master`
+- [ ] All 6 invoice template commits applied correctly
+- [ ] All invoice template tests pass
+- [ ] No conflicts or unexpected changes during cherry-pick
+
+### Post-Master Validation
+- [ ] Invoice templates merged to `master`
+- [ ] Changes pushed to remote `master`
+- [ ] CI/CD passes on `master`
+
+### Post-Cleanup Validation
+- [ ] `clauding` branch has only non-invoice template commits
+- [ ] No Bonanza Produce commits remain in `clauding` history
+- [ ] All `clauding` tests pass
+- [ ] Force push successful
+
+### Post-Sync Validation
+- [ ] `clauding` merged with `master`
+- [ ] All conflicts resolved
+- [ ] Changes pushed to remote
+
+### Final Merge Validation
+- [ ] `get-transactions2-page-working` merged with `clauding`
+- [ ] All conflicts resolved
+- [ ] Final tests pass
+- [ ] Changes pushed to remote
+
+## Success Metrics
+
+- **Branch structure**: Invoice templates cleanly separated on `master`
+- **Commit history**: Linear, no duplicate invoice template commits
+- **Tests passing**: 100% of existing tests pass after each step
+- **No data loss**: All work preserved in appropriate branches
+- **Branch clarity**: Each branch has a clear, focused purpose
+
+## Dependencies & Risks
+
+### Dependencies
+- [ ] All current work on `clauding` should be backed up or committed
+- [ ] Team should be aware of branch manipulation to avoid force pushing
+- [ ] CI/CD should be monitored during operations
+
+### Risks
+1. **Force push risk**: Force pushing `clauding` will rewrite history
+   - **Mitigation**: Use `--force-with-lease`, notify team beforehand
+   
+2. **Conflict resolution**: Multiple merge/conflict resolution points
+   - **Mitigation**: Test after each step, resolve conflicts carefully
+   
+3. **Work loss**: Potential to lose commits if operations go wrong
+   - **Mitigation**: Create backups, verify each step before proceeding
+   
+4. **CI/CD disruption**: Force pushes may affect CI/CD pipelines
+   - **Mitigation**: Coordinate with team, avoid during active deployments
+
+### Contingency Plan
+
+If something goes wrong:
+1. **Recover `clauding` branch**: 
+   ```bash
+   git checkout clauding
+   git reset --hard origin/clauding  # Restore from remote backup
+   ```
+
+2. **Recover master**:
+   ```bash
+   git checkout master
+   git reset --hard origin/master  # Restore from deploy/master
+   ```
+
+3. **Manual cherry-pick recovery**: If rebasing failed, manually cherry-pick remaining commits
+
+## Alternative Approaches Considered
+
+### Approach 1: Squash and Merge
+**Pros**: Single clean commit, simple history
+**Cons**: Loses individual commit history and context
+
+**Rejected because**: Team uses merge commits (not squash), and individual commit history is valuable for tracking invoice template development.
+
+### Approach 2: Keep Branches Separate
+**Pros**: No branch manipulation needed
+**Cons**: Branches remain tangled, harder to track progress
+
+**Rejected because**: Goal is to cleanly separate invoice templates from transaction work.
+
+### Approach 3: Rebase Clauding Onto Master
+**Pros**: Linear history
+**Cons**: Requires force push, may lose merge context
+
+**Rejected because**: Current team workflow uses merge commits, and merging master into clauding preserves the integration point.
+
+### Approach 4: Create New Branch Instead of Cleanup
+**Pros**: Less risky, preserves full history
+**Cons**: Accumulates branches, harder to track
+
+**Rejected because**: Goal is cleanup and reorganization, not preservation.
+
+## Related Work
+
+- **Previous invoice template work**: `2026-02-07-feat-add-invoice-template-03881260-plan.md`
+- **Current branch structure**: `clauding` has hierarchical relationship with `get-transactions2-page-working`
+- **Team git workflow**: Uses merge commits (not rebasing), per repo research
+
+## References & Research
+
+### Internal References
+- **Branch management patterns**: Repo research analysis (see `task_id: ses_3c2287be8ffe9icFi5jHEspaqh`)
+- **Invoice template location**: `src/clj/auto_ap/parse/templates.clj`
+- **Current branch structure**: Git log analysis
+
+### Git Operations Documentation
+- **Cherry-pick**: `git cherry-pick <commit>`
+- **Interactive rebase**: `git rebase -i <base>`
+- **Force push with lease**: `git push --force-with-lease`
+- **Merge commits**: `git merge <branch> --no-edit`
+
+### File Locations
+- Templates: `src/clj/auto_ap/parse/templates.clj`
+- Parser logic: `src/clj/auto_ap/parse.clj`
+- Invoice PDF: `dev-resources/INVOICE - 03881260.pdf`
+
+## Testing Plan
+
+### Before Each Major Step
+```bash
+# Verify current branch state
+git branch -vv
+git log --oneline -10
+
+# Run all tests
+lein test
+
+# Run specific invoice template tests
+lein test auto-ap.parse.templates-test
+```
+
+### After Each Major Step
+- Verify commit count and order
+- Run full test suite
+- Check for unintended changes
+- Verify remote branch state matches local
+
+## Notes
+
+- **Team coordination**: Inform team before force pushing to avoid conflicts
+- **Backup strategy**: All commits are preserved in the rebase process
+- **Testing**: Verify at each step to catch issues early
+- **Safety first**: Use `--force-with-lease` instead of `--force`
+- **Documentation**: This plan serves as documentation for the operation

src/clj/auto_ap/datomic/sales_orders.clj

@@ -1,39 +1,51 @@
 (ns auto-ap.datomic.sales-orders
   (:require
-   [auto-ap.datomic :refer [conn]]
    [auto-ap.storage.parquet :as pq]
-   [clj-time.coerce :as c]
+   [auto-ap.time :as atime]
+   [clj-time.coerce :as coerce]
    [clojure.set :as set]
-   [com.brunobonacci.mulog :as mu]))
+   [clojure.string :as str]
+   [com.brunobonacci.mulog :as mu]
+   [ring.util.codec :as ring-codec]))
+
+(defn- payment-methods->charges [pm-str]
+  (when (not-empty pm-str)
+    (mapv (fn [pm] {:charge/type-name pm})
+          (str/split pm-str #","))))
 
 (defn <-row
-  "Convert a flat parquet row into the shape consumers expect.
-   Parquet produces maps of the form:
-       {\"external-id\"  \"square/order/123\", ...}
-   which we transform to:
-       {:sales-order/external-id \"square/order/123\", ...}"
+  "Convert a flat parquet row into the shape consumers expect."
   [row]
-  (-> row
-      (set/rename-keys
-       {"external-id" :sales-order/external-id
-        "location"    :sales-order/location
-        "total"       :sales-order/total
-        "tax"         :sales-order/tax
-        "tip"         :sales-order/tip
-        "discount"    :sales-order/discount
-        "service-charge" :sales-order/service-charge
-        "vendor"      :sales-order/vendor
-        "client-code" :sales-order/client-code
-        "date"        :sales-order/date})
-      (update :sales-order/date #(some-> % str))))
+  (let [pm (:payment-methods row)]
+    (-> row
+        (set/rename-keys
+         {:external-id :sales-order/external-id
+          :location    :sales-order/location
+          :total       :sales-order/total
+          :tax         :sales-order/tax
+          :tip         :sales-order/tip
+          :discount    :sales-order/discount
+          :service-charge :sales-order/service-charge
+          :vendor      :sales-order/vendor
+          :client-code :sales-order/client-code
+          :date        :sales-order/date
+          :source      :sales-order/source
+          :reference-link :sales-order/reference-link
+          :payment-methods :sales-order/payment-methods
+          :processors  :sales-order/processors
+          :categories  :sales-order/categories})
+        (update :sales-order/date #(some-> % str))
+        (dissoc :entity-type :_seq-no)
+         (assoc :sales-order/charges (payment-methods->charges pm)))))
 
 (defn build-where-clause [args]
-  (let [clauses [(when-let [c (:client-code args)]
-                   ["external_id.client = '" c "'"])
-                (when-let [v (:vendor args)]
-                  ["external_id.vendor = '" (name v) "'"])
-                (when-let [l (:location args)]
-                  ["location = '" l "'"])]
+  (let [clauses (keep identity
+                      [(when-let [c (:client-code args)]
+                         (str "external_id.client = '" c "'"))
+                       (when-let [v (:vendor args)]
+                         (str "external_id.vendor = '" (name v) "'"))
+                       (when-let [l (:location args)]
+                         (str "location = '" l "'"))])]
     (when (seq clauses)
       (str "WHERE " (str/join " AND " clauses)))))
 
@@ -46,32 +58,114 @@
 
 (defn raw-graphql-ids [args]
   (let [start (some-> (:start (:date-range args)) .toString)
-        end   (some-> (:end (:date-range args)) .substring 0 10)
-        where (build-where-clause args)
-        sort  (build-sort-clause args)
+        end   (some-> (:end (:date-range args)) (.substring 0 10))
         limit (or (:limit args) page-size)
-        offset (or (:offset args) 0)
-        where-str (when where (str " " where))]
+        offset (or (:offset args) 0)]
     (when start
       (let [result (pq/get-sales-orders start end
                                          {:client (:client-code args)
                                           :vendor (:vendor args)
                                           :location (:location args)
-                                          :sort sort
+                                          :sort (or (:sort args) "date")
                                           :order "DESC"
                                           :limit limit
                                           :offset offset})]
-        {:ids  (mapv #(str (:external_id %)) (:rows result))
+        {:ids  (mapv #(str (:external-id %)) (:rows result))
          :rows (:rows result)
          :count (:count result)}))))
 
 (defn graphql-results [rows _ids _args]
   (mapv <-row rows))
 
+(defn- extract-date-str [v]
+  (when v
+    (cond
+      (instance? org.joda.time.DateTime v) (atime/unparse-local v atime/normal-date)
+      (instance? org.joda.time.LocalDate v) (atime/unparse-local v atime/normal-date)
+      (instance? java.util.Date v) (atime/unparse-local (coerce/to-date-time v) atime/normal-date)
+      (instance? java.time.LocalDate v) (.toString v)
+      (string? v) (if (re-find #"^\d{2}/\d{2}/\d{4}" v)
+                    (-> (java.time.LocalDate/parse v (java.time.format.DateTimeFormatter/ofPattern "MM/dd/yyyy"))
+                        .toString)
+                    (if (> (count v) 10) (.substring v 0 10) v))
+      :else (str v))))
+
+(defn- get-date [qp k]
+  (or (extract-date-str (get qp k))
+      (extract-date-str (get qp (name k)))))
+
+(defn- kw->str [v]
+  (when (some? v)
+    (if (keyword? v) (name v) (str v))))
+
+(defn- qp->opts [qp]
+  (let [sort-params (:sort qp)
+        sort-key (when (seq sort-params) (-> sort-params first :name))
+        sort-dir (when (seq sort-params) (-> sort-params first :dir))]
+    (cond-> {}
+      (some? (:client-code qp)) (assoc :client (kw->str (:client-code qp)))
+      (some? (:location qp)) (assoc :location (kw->str (:location qp)))
+      (not-empty (:payment-method qp)) (assoc :payment-method (:payment-method qp))
+      (some? (:processor qp)) (assoc :processor (kw->str (:processor qp)))
+      (not-empty (:category qp)) (assoc :category (:category qp))
+      (:total-gte qp) (assoc :total-gte (:total-gte qp))
+      (:total-lte qp) (assoc :total-lte (:total-lte qp))
+      sort-key (assoc :sort sort-key)
+      sort-dir (assoc :order (or sort-dir "DESC"))
+      true (assoc :limit (or (:per-page qp) 25)
+                  :offset (or (:start qp) 0)))))
+
+(defn- default-date-range []
+  (let [today (.toString (java.time.LocalDate/now))
+        week-ago (.toString (.minusDays (java.time.LocalDate/now) 7))]
+    [week-ago today]))
+
+(defn- qp->date-range [qp]
+  (let [[default-start default-end] (default-date-range)]
+    [(or (get-date qp :start-date)
+         (extract-date-str (get-in qp [:date-range :start]))
+         default-start)
+     (or (get-date qp :end-date)
+         (extract-date-str (get-in qp [:date-range :end]))
+         default-end)]))
+
+(defn- request->client-codes [request]
+  (let [clients (:clients request)
+        codes (keep :client/code clients)]
+    (when (seq codes) codes)))
+
+(defn fetch-page-ssr
+  "Fetch sales orders from parquet for the SSR page."
+  [request]
+  (let [qp (:query-params request)
+        raw-qp (some-> (:query-string request)
+                        ring-codec/form-decode
+                        (->> (into {} (remove (fn [[_ v]] (str/blank? v))))))
+        [start end] (qp->date-range (merge raw-qp qp))
+        opts (qp->opts qp)
+        client-codes (request->client-codes request)
+        opts (if client-codes (assoc opts :client-codes client-codes) opts)
+        result (pq/get-sales-orders start end opts)
+        rows (mapv <-row (:rows result))]
+    {:rows rows :count (:count result)}))
+
+(defn summarize-page-ssr
+  "Summarize all matching sales orders via parquet."
+  [request]
+  (let [qp (:query-params request)
+        raw-qp (some-> (:query-string request)
+                        ring-codec/form-decode
+                        (->> (into {} (remove (fn [[_ v]] (str/blank? v))))))
+        [start end] (qp->date-range (merge raw-qp qp))
+        opts (dissoc (qp->opts qp) :limit :offset :sort :order)
+        client-codes (request->client-codes request)
+        opts (if client-codes (assoc opts :client-codes client-codes) opts)]
+    (pq/get-sales-orders-summary start end opts)))
+
 (defn summarize-orders [rows]
   (when (seq rows)
-    (let [total (reduce + 0.0 (map #(or (:total %) 0.0) rows))
-          tax   (reduce + 0.0 (map #(or (:tax %) 0.0) rows))]
+    (let [total (reduce + 0.0 (map #(or (:sales-order/total %) 0.0) rows))
+          tax   (reduce + 0.0 (map #(or (:sales-order/tax %) 0.0) rows))]
       {:total total
        :tax   tax})))
 

src/clj/auto_ap/datomic/sales_orders_new.clj

@@ -1,224 +0,0 @@
-(ns auto-ap.datomic.sales-orders
-  (:require [auto-ap.datomic :refer [conn]]
-            [auto-ap.storage.parquet :as pq]
-            [clojure.data.json :as json]
-            [clojure.java.io :as io]))
-
-(defn <-row
-  "Convert a flat parquet row (string keys) into the
-   shape consumers expect.  Parquet produces maps of the form
-
-       {\"external-id\"  \"square/order/123\",
-        \"location\"     \"DT\",
-        \"total\"        50.0}
-
-   which we transform to:
-
-       {:sales-order/external-id \"square/order/123\",
-        :sales-order/location    \"DT\",
-        :sales-order/total       50.0}
-
-   Note: client, charges and other nested structures are not
-   available in the flat parquet rows.  When denormalisation
-   adds those columns we can restore the full consumer shape."
-  [row]
-  (-> row
-      (set/rename-keys
-       {"external-id" :sales-order/external-id
-        "location"    :sales-order/location
-        "total"       :sales-order/total
-        "tax"         :sales-order/tax
-        "tip"         :sales-order/tip})))
-
-(defn build-where-clause
-  "Build a SQL WHERE fragment from the fields that
-   parquet can filter on: external_id.client, vendor, location.
-
-   Returns either a predicate string e.g.
-       \"WHERE external_id.client = 'acme' AND vendor = 'square'\"
-   or nil when no applicable filters exist."
-  [args]
-  (let [clauses []
-        client   (:client-code args)
-        vendor   (:vendor args)
-        location (:location args)]
-    (when (or client vendor location)
-      (->> [[:client "external_id.client" client]
-            [:vendor  "external_id.vendor"  vendor]
-            [:location "location"           location]]
-           (keep (fn [[_ col v]]
-                   (when v [col v])))
-           (mapv #(str %1 " = '" %2 "'"))
-           (str/join " AND "))}))
-
-(defn build-sort-clause
-  "Map sort-key field names from args into SQL ORDER-BY fragments.
-
-   Supported fields map to parquet column names:
-     \"date\"    -> DATE
-     \"total\"   -> TOTAL
-     \"tax\"     -> TAX
-     \"tip\"     -> TIP
-     \"client\"  -> EXTERNAL_ID_CLIENT  (for flat client codes)
-     \"location\"-> LOCATION
-
-   Falls back to \"DATE DESC\" when the args do not specify
-   an explicit field."
-  [args]
-  ;; We delegate most of the SQL ordering work to get-sales-orders,
-  ;; which already defaults to DATE DESC.
-  (when-let [sorts (:sort args)]
-    (->> sorts
-         (keep (fn [{:keys [sort-key asc]}]
-                 (let [dir (if asc "ASC" "DESC")
-                       col (case sort-key
-                             "date"    "DATE"
-                             "total"   "TOTAL"
-                             "tax"     "TAX"
-                             "tip"     "TIP"
-                             "total-desc"  "TOTAL DESC"
-                             "source"  "SALE_SOURCE"
-                             "client"  "EXTERNAL_ID_CLIENT"
-                             "location" "LOCATION"
-                             nil)]                   ; unknown → skip
-                   (when col `[~col ~dir]))))
-         (interleave (repeat \,))
-         (apply str))))
-
-(defn build-pagination-clause
-  "Convert a Datomic-side pagination request into SQL-limit/offset
-   numbers.
-
-   Supports:
-     :start    → OFFSET
-     :count / :per-page  → LIMIT"
-  [args]
-  {:limit  (or (:count args) (:per-page args))
-   :offset (or (:start args) 0)})
-
-(defn- apply-pagination
-  "Safely re-implements the old datomic-side pagination logic.
-   Mutates a COPY of args so we can extract the resulting
-   cursor values for the response.
-
-   Returns {limit offset}"
-  [args]
-  (let [page     (build-pagination-clause args)
-        {:keys [limit offset]} page
-        client (:client-code args)]
-    ; In the new architecture pagination is applied server-side
-    ; by get-sales-orders via LIMIT/OFFSET, so this function
-    ; mainly exists as a thin wrapper for any remaining
-    ; in-memory re-paging concerns.
-    (if limit
-      (assoc page :limit (Integer. limit))
-      page)))
-
-(defn- build-options
-  "Assemble the opts map passed to pq/get-sales-orders:
-     {:client ..., :location ..., :vendor ...,
-      :limit 10, :offset 0, :sort, :order}"
-  [args]
-  (let [page    (apply-pagination args)
-        limit   (:limit page)
-        offset  (:offset page)
-        client  (:client-code args)]
-    (cond-> {:client  client
-             :vendor  (:vendor args)
-             :limit   limit
-             :offset  offset}
-      (:location args) (assoc :location (:location args))
-      (:sort args)     (assoc :sort "date")          ; let get-sales-orders handle order
-      true              (merge {:order    "DESC"
-                                :sort-key (:sort-key args)}))))
-
-(defn raw-graphql-ids
-  "Query sales-orders FROM parquet files via DuckDB instead of Datomic.
-
-   Filters applied at the parquet level:
-     - date-range → selects which parquet files to read
-     - client-code / vendor / location where clauses
-     - sort & pagination are delegated to get-sales-orders
-
-   category, processor, type-name filters require nested joins
-   that parquet does not support -- those fields are ignored.
-
-   Returns
-       {:ids  [string-key-for-each-matched-row]
-        :count int (total matches BEFORE pagination)}"
-  [args]
-  (let [start    (when-let [s (:start (:date-range args))]
-                   (.toString (.plusDays (java.time.LocalDate/parse s) -1)))
-        end      (when-let [e (:end   (:date-range args))]
-                   (-> e .substring 0 10))
-        where    (build-where-clause args)
-        options  (build-options args)
-        where-str (some-> where #(str " WHERE " %))]
-    (cond->> nil
-      ; Query rows from parquet with our filters and sort.
-       where-str   (pq/get-sales-orders
-                    start end
-                    (assoc options :sort-key where-str)))
-
-    ; For each row returned we need an ID string.
-    ; We use the external-id column as the lookup key.
-    (when-let [rows (:rows result)]
-      {:ids  (mapv #(str (:external_id %)) rows)
-       :rows rows
-       :count (:count result)})))
-
-(defn graphql-results
-  "Return the full payment-row data for the selected IDs.
-   Since we now read FROM parquet, we receive the raw row vector
-   and transform it.
-
-   The old signature [ids db args] is replaced by [rows id-keys _].
-   We ignore the database argument (Datomic pull is no longer
-   called)."
-  [rows _id-keys _args]
-  (->> rows
-       (mapv #(<-row %))))
-
-(defn summarize-orders
-  "Sum totals and discounts for the given ID-set.
-
-   This function still queries Datomic because the parquet-side
-   aggregation query would duplicate the WHERE logic.
-   If we want a pure parquet path here, add an
-   SQL-based aggregation in a follow-up."
-  [ids]
-  (let [[total tax]
-        (#'auto-ap.datomic/aggregate-sum ids)       ; uses dc/q internally
-        first]
-    {:total total
-     :tax   tax}))
-
-(defn get-graphql
-  "Entry-point: return [payments count summary].
-
-   The data flow is:
-     1. raw-graphql-ids  → parquet query → [:rows :count]
-     2. graphql-results  <- transform rows ← [:results <_id-keys> _args]
-     3. summarize-orders <- Datomic agg   ← [:total :tax]"
-  [args]
-  (let [{:keys [ids count']}
-        (mu/trace
-         ::get-sales-order-ids
-         []
-         (raw-graphql-ids args))]
-    [(->> (mu/trace ::get-results [] (graphql-results ids id-keys args))
-          matching-count
-          summarize-orders ids)])))
-
-(defn summarize-graphql
-  "Entry-point: return just the summary {:total :tax}.
-
-   Like get-graphql, this delegates to raw-graphql-ids
-   and then to summarize-orders."
-  [args]
-  (let [{:keys [ids count']}
-        (mu/trace
-         ::get-sales-order-ids
-         []
-         (raw-graphql-ids args))]
-    (summarize-orders ids)))

src/clj/auto_ap/ezcater/core.clj

@@ -21,42 +21,41 @@
                     :body    (json/write-str {"query" (v/graphql-query q)})
                     :as      :json})
       :body
-      :data
-      ))
+      :data))
 
 (defn get-caterers [integration]
   (:caterers (query integration {:venia/queries [{:query/data
-                                                  [:caterers [:name :uuid [:address [:name :street]]]]}]} )))
+                                                  [:caterers [:name :uuid [:address [:name :street]]]]}]})))
 
 (defn get-subscriptions [integration]
   (->> (query integration {:venia/queries [{:query/data
-                                            [:subscribers [:id [:subscriptions [:parentId :parentEntity :eventEntity :eventKey]] ]]}]} )
+                                            [:subscribers [:id [:subscriptions [:parentId :parentEntity :eventEntity :eventKey]]]]}]})
        :subscribers
        first
        :subscriptions))
 
 (defn get-integrations []
   (map first (dc/q '[:find (pull ?i [:ezcater-integration/api-key
-                                    :ezcater-integration/subscriber-uuid
-                                    :db/id
-                                    :ezcater-integration/integration-status [:db/id]])
-                    :in $
-                    :where [?i :ezcater-integration/api-key]]
-                  (dc/db conn))))
+                                      :ezcater-integration/subscriber-uuid
+                                      :db/id
+                                      {:ezcater-integration/integration-status [:db/id]}])
+                     :in $
+                     :where [?i :ezcater-integration/api-key]]
+                   (dc/db conn))))
 
 (defn mark-integration-status [integration integration-status]
   @(dc/transact conn
-               [{:db/id                                  (:db/id integration)
-                 :ezcater-integration/integration-status (assoc integration-status
-                                                                :db/id (or (-> integration :ezcater-integration/integration-status :db/id)
-                                                                           (random-tempid)))}]))
+                [{:db/id                                  (:db/id integration)
+                  :ezcater-integration/integration-status (assoc integration-status
+                                                                 :db/id (or (-> integration :ezcater-integration/integration-status :db/id)
+                                                                            (random-tempid)))}]))
 
 (defn upsert-caterers
   ([integration]
    @(dc/transact
      conn
      (for [caterer (get-caterers integration)]
-       {:db/id                        (:db/id integration) 
+       {:db/id                        (:db/id integration)
         :ezcater-integration/caterers [{:ezcater-caterer/name         (str (:name caterer) " (" (:street (:address caterer)) ")")
                                         :ezcater-caterer/search-terms (str (:name caterer) " " (:street (:address caterer)))
                                         :ezcater-caterer/uuid         (:uuid caterer)}]}))))
@@ -65,14 +64,14 @@
   ([integration]
    (let [extant (get-subscriptions integration)
          to-ensure (set (map first (dc/q '[:find ?cu
-                                          :in $
-                                          :where [_ :client/ezcater-locations ?el]
-                                          [?el :ezcater-location/caterer ?c]
-                                          [?c :ezcater-caterer/uuid ?cu]]
-                                        (dc/db conn))))
+                                           :in $
+                                           :where [_ :client/ezcater-locations ?el]
+                                                  [?el :ezcater-location/caterer ?c]
+                                                  [?c :ezcater-caterer/uuid ?cu]]
+                                         (dc/db conn))))
          to-create (set/difference
-                     to-ensure
-                     (set (map :parentId extant)))]
+                    to-ensure
+                    (set (map :parentId extant)))]
      (doseq [parentId to-create]
        (query integration
               {:venia/operation {:operation/type :mutation
@@ -95,7 +94,6 @@
                                                         :eventKey     'cancelled}}
                                   [[:subscription [:parentId :parentEntity :eventEntity :eventKey]]]]]})))))
 
-
 #_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]}
 (defn upsert-ezcater
   ([] (upsert-ezcater (get-integrations)))
@@ -116,12 +114,11 @@
 
 (defn get-caterer [caterer-uuid]
   (dc/pull (dc/db conn)
-          '[:ezcater-caterer/name
-            {:ezcater-integration/_caterers [:ezcater-integration/api-key]}
-            {:ezcater-location/_caterer [:ezcater-location/location
-                                         {:client/_ezcater-locations [:client/code]}]}]
-          [:ezcater-caterer/uuid caterer-uuid]))
-
+           '[:ezcater-caterer/name
+             {:ezcater-integration/_caterers [:ezcater-integration/api-key]}
+             {:ezcater-location/_caterer [:ezcater-location/location
+                                          {:client/_ezcater-locations [:client/code]}]}]
+           [:ezcater-caterer/uuid caterer-uuid]))
 
 (defn round-carry-cents [f]
   (with-precision 2 (double (.setScale (bigdec f) 2 java.math.RoundingMode/HALF_UP))))
@@ -137,69 +134,69 @@
                      :else
                      0.07M)]
     (round-carry-cents
-      (* commision%
-         0.01M
-         (+
-           (-> order :totals :subTotal :subunits )
-           (reduce +
-                   0
-                   (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
+     (* commision%
+        0.01M
+        (+
+         (-> order :totals :subTotal :subunits)
+         (reduce +
+                 0
+                 (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order)))))))))
 
 (defn ccp-fee [order]
   (round-carry-cents
-    (* 0.000299M
-       (+
-         (-> order :totals :subTotal :subunits )
-         (-> order :totals :salesTax :subunits )
-         (reduce +
-                 0
-                 (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
+   (* 0.000299M
+      (+
+       (-> order :totals :subTotal :subunits)
+       (-> order :totals :salesTax :subunits)
+       (reduce +
+               0
+               (map (comp :subunits :cost) (:feesAndDiscounts (:catererCart order))))))))
 
 (defn order->sales-order [{{:keys [timestamp]} :event {:keys [orderItems]} :catererCart :keys [client-code client-location uuid] :as order}]
   (let [adjustment (round-carry-cents (- (+ (-> order :totals :subTotal :subunits (* 0.01))
                                             (-> order :totals :salesTax :subunits (* 0.01)))
-                                         (-> order :catererCart :totals :catererTotalDue )
+                                         (-> order :catererCart :totals :catererTotalDue)
                                          (commision order)
                                          (ccp-fee order)))
         service-charge (+ (commision order) (ccp-fee order))
         tax (-> order :totals :salesTax :subunits (* 0.01))
         tip (-> order :totals :tip :subunits (* 0.01))]
     #:sales-order
-    {:date        (atime/localize (coerce/to-date-time timestamp))
-     :external-id (str "ezcater/order/" client-code "-" client-location "-" uuid)
-     :client      [:client/code client-code]
-     :location    client-location
-     :reference-link (str (url/url  "https://ezmanage.ezcater.com/orders/" uuid ))
-     :line-items  [#:order-line-item
-                   {:external-id (str "ezcater/order/" client-code "-" client-location "-" uuid "-" 0)
-                    :item-name   "EZCater Catering"
-                    :category    "EZCater Catering"
-                    :discount    adjustment
-                    :tax         tax
-                    :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
-                                    tax
-                                    tip)}]
-     :charges     [#:charge
-                   {:type-name   "CARD"
-                    :date        (atime/localize (coerce/to-date-time timestamp))
-                    :client      [:client/code client-code]
-                    :location    client-location
-                    :external-id (str "ezcater/charge/" uuid)
-                    :processor   :ccp-processor/ezcater
-                    :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
-                                    tax
-                                    tip)
-                    :tip         tip}]
+     {:date        (atime/localize (coerce/to-date-time timestamp))
+      :external-id (str "ezcater/order/" client-code "-" client-location "-" uuid)
+      :client      [:client/code client-code]
+      :location    client-location
+      :reference-link (str (url/url  "https://ezmanage.ezcater.com/orders/" uuid))
+      :line-items  [#:order-line-item
+                     {:external-id (str "ezcater/order/" client-code "-" client-location "-" uuid "-" 0)
+                      :item-name   "EZCater Catering"
+                      :category    "EZCater Catering"
+                      :discount    adjustment
+                      :tax         tax
+                      :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
+                                      tax
+                                      tip)}]
+      :charges     [#:charge
+                     {:type-name   "CARD"
+                      :date        (atime/localize (coerce/to-date-time timestamp))
+                      :client      [:client/code client-code]
+                      :location    client-location
+                      :external-id (str "ezcater/charge/" uuid)
+                      :processor   :ccp-processor/ezcater
+                      :total       (+ (-> order :totals :subTotal :subunits (* 0.01))
+                                      tax
+                                      tip)
+                      :tip         tip}]
 
-     :total          (+ (-> order :totals :subTotal :subunits (* 0.01))
-                        tax
-                        tip)
-     :discount       adjustment
-     :service-charge service-charge
-     :tax            tax
-     :tip            tip
-     :returns        0.0
-     :vendor         :vendor/ccp-ezcater}}))
+      :total          (+ (-> order :totals :subTotal :subunits (* 0.01))
+                         tax
+                         tip)
+      :discount       adjustment
+      :service-charge service-charge
+      :tax            tax
+      :tip            tip
+      :returns        0.0
+      :vendor         :vendor/ccp-ezcater}))
 
 (defn- flatten-order-to-parquet! [order]
   "Flatten a sales-order into entity-type tagged maps and buffer to parquet."
@@ -222,81 +219,73 @@
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
         (parquet/buffer! "charge"
-                        {:entity-type "charge"
-                         :external-id (:charge/external-id chg)
-                         :type-name   (:charge/type-name chg)
-                         :total       (:charge/total chg)
-                         :tax         (:charge/tax chg)
-                         :tip         (:charge/tip chg)
-                         :date        so-date
-                         :processor   (some-> (:charge/processor chg) name)
-                         :sales-order-external-id so-ext-id})))
+                         {:entity-type "charge"
+                          :external-id (:charge/external-id chg)
+                          :type-name   (:charge/type-name chg)
+                          :total       (:charge/total chg)
+                          :tax         (:charge/tax chg)
+                          :tip         (:charge/tip chg)
+                          :date        so-date
+                          :processor   (some-> (:charge/processor chg) name)
+                          :sales-order-external-id so-ext-id})))
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
         (parquet/buffer! "line-item"
-                        {:entity-type "line-item"
-                         :item-name   (:order-line-item/item-name li)
-                         :category    (:order-line-item/category li)
-                         :total       (:order-line-item/total li)
-                         :tax         (:order-line-item/tax li)
-                         :discount    (:order-line-item/discount li)
-                         :sales-order-external-id so-ext-id})))))
-
+                         {:entity-type "line-item"
+                          :item-name   (:order-line-item/item-name li)
+                          :category    (:order-line-item/category li)
+                          :total       (:order-line-item/total li)
+                          :tax         (:order-line-item/tax li)
+                          :discount    (:order-line-item/discount li)
+                          :sales-order-external-id so-ext-id})))))
 
 (defn get-by-id [integration id]
-  (query 
-    integration
-    {:venia/queries [[:order {:id id}
-                      [:uuid
-                       :orderNumber
-                       :orderSourceType
-                       [:caterer
-                        [:name
-                         :uuid
-                         [:address [:street]]]]
-                       [:event
-                        [:timestamp
-                         :catererHandoffFoodTime
-                         :orderType]]
-                       [:catererCart [[:orderItems 
-                                       [:name
-                                        :quantity
-                                        :posItemId
-                                        [:totalInSubunits
-                                         [:currency
-                                          :subunits]]]]
-                                      [:totals 
-                                       [:catererTotalDue]]
-                                      [:feesAndDiscounts
-                                       {:type 'DELIVERY_FEE}
-                                       [[:cost
-                                         [:currency
-                                          :subunits]]]]]]
-                       [:totals [[:customerTotalDue
-                                  [
-                                   :currency
-                                   :subunits
-                                   ]]
-                                 [:pointOfSaleIntegrationFee
-                                  [
-                                   :currency
-                                   :subunits
-                                   ]]
-                                 [:tip
-                                  [:currency
-                                   :subunits]]
-                                 [:salesTax
-                                  [
-                                   :currency
-                                   :subunits
-                                   ]]
-                                 [:salesTaxRemittance
-                                  [:currency
-                                   :subunits
-                                   ]]
-                                 [:subTotal
-                                  [:currency
-                                   :subunits]]]]]]]}))
+  (query
+   integration
+   {:venia/queries [[:order {:id id}
+                     [:uuid
+                      :orderNumber
+                      :orderSourceType
+                      [:caterer
+                       [:name
+                        :uuid
+                        [:address [:street]]]]
+                      [:event
+                       [:timestamp
+                        :catererHandoffFoodTime
+                        :orderType]]
+                      [:catererCart [[:orderItems
+                                      [:name
+                                       :quantity
+                                       :posItemId
+                                       [:totalInSubunits
+                                        [:currency
+                                         :subunits]]]]
+                                     [:totals
+                                      [:catererTotalDue]]
+                                     [:feesAndDiscounts
+                                      {:type 'DELIVERY_FEE}
+                                      [[:cost
+                                        [:currency
+                                         :subunits]]]]]]
+                      [:totals [[:customerTotalDue
+                                 [:currency
+                                  :subunits]]
+                                [:pointOfSaleIntegrationFee
+                                 [:currency
+                                  :subunits]]
+                                [:tip
+                                 [:currency
+                                  :subunits]]
+                                [:salesTax
+                                 [:currency
+                                  :subunits]]
+                                [:salesTaxRemittance
+                                 [:currency
+                                  :subunits]]
+                                [:subTotal
+                                 [:currency
+                                  :subunits]]]]]]]}))
 
 (defn lookup-order [json]
   (let [caterer     (get-caterer (get json "parent_id"))
@@ -304,18 +293,18 @@
         client      (-> caterer :ezcater-location/_caterer first :client/_ezcater-locations :client/code)
         location    (-> caterer :ezcater-location/_caterer first :ezcater-location/location)]
     (if (and client location)
-      (doto 
-          (-> (get-by-id integration (get json "entity_id"))
-              (:order)
-              (assoc :client-code client
-                     :client-location location))
+      (doto
+       (-> (get-by-id integration (get json "entity_id"))
+           (:order)
+           (assoc :client-code client
+                  :client-location location))
         (#(alog/info ::order-details :detail %)))
       (alog/warn ::caterer-no-longer-has-location :json json))))
 
 (defn import-order [json]
   (alog/info
-    ::try-import-order
-    :json json)
+   ::try-import-order
+   :json json)
   (when-let [order (some-> json
                            (lookup-order)
                            (order->sales-order)
@@ -324,7 +313,7 @@
     (try
       (flatten-order-to-parquet! order)
       (alog/info ::order-buffered
-                :external-id (:sales-order/external-id order))
+                 :external-id (:sales-order/external-id order))
       (catch Exception e
         (alog/error ::buffer-failed
                     :exception e
@@ -359,30 +348,30 @@
                                                                 "key" "accepted",
                                                                 "occurred_at" "2022-07-21T19:21:07.549Z"}
                                                     ezcater-order (lookup-order lookup-map)
-                                                    extant-order (dc/pull (dc/db conn) '[:sales-order/total
-                                                                                       :sales-order/tax
-                                                                                       :sales-order/tip
-                                                                                       :sales-order/discount
-                                                                                       :sales-order/external-id
-                                                                                       {:sales-order/charges [:charge/tax
-                                                                                                              :charge/tip
-                                                                                                              :charge/total
-                                                                                                              :charge/external-id]
-                                                                                        :sales-order/line-items [:order-line-item/external-id
-                                                                                                                 :order-line-item/total
-                                                                                                                 :order-line-item/tax
-                                                                                                                 :order-line-item/discount]}]
-                                                                         [:sales-order/external-id order])
+                                                    extant-order (dc/pull (dc/db conn '[:sales-order/total]
+                                                                                 :sales-order/tax
+                                                                                 :sales-order/tip
+                                                                                 :sales-order/discount
+                                                                                 :sales-order/external-id
+                                                                                 {:sales-order/charges [:charge/tax
+                                                                                                        :charge/tip
+                                                                                                        :charge/total
+                                                                                                        :charge/external-id]
+                                                                                  :sales-order/line-items [:order-line-item/external-id
+                                                                                                           :order-line-item/total
+                                                                                                           :order-line-item/tax
+                                                                                                           :order-line-item/discount]})
+                                                                          [:sales-order/external-id order])
 
                                                     updated-order (-> (order->sales-order ezcater-order)
                                                                       (select-keys
-                                                                        #{:sales-order/total
-                                                                          :sales-order/tax
-                                                                          :sales-order/tip
-                                                                          :sales-order/discount
-                                                                          :sales-order/charges
-                                                                          :sales-order/external-id
-                                                                          :sales-order/line-items})
+                                                                       #{:sales-order/total
+                                                                         :sales-order/tax
+                                                                         :sales-order/tip
+                                                                         :sales-order/discount
+                                                                         :sales-order/charges
+                                                                         :sales-order/external-id
+                                                                         :sales-order/line-items})
                                                                       (update :sales-order/line-items
                                                                               (fn [c]
                                                                                 (map #(select-keys % #{:order-line-item/external-id

src/clj/auto_ap/jobs/sales_summaries.clj

@@ -40,17 +40,14 @@
                 (dc/db conn)
                 number)))
 
-
 (defn delete-all []
   @(dc/transact-async conn
-               (->> 
-                (dc/q '[:find ?ss 
-                        :where [?ss :sales-summary/date]]
-                      (dc/db conn))
-                (map (fn [[ ss]]
-                       [:db/retractEntity ss])))))
-
-
+                      (->>
+                       (dc/q '[:find ?ss
+                               :where [?ss :sales-summary/date]]
+                             (dc/db conn))
+                       (map (fn [[ss]]
+                              [:db/retractEntity ss])))))
 
 (defn dirty-sales-summaries [c]
   (let [client-id (dc/entid (dc/db conn) c)]
@@ -103,7 +100,7 @@
   (let [date-str (.toString date)]
     (when-let [rows (seq (pq/query-deduped "charge" date-str date-str))]
       (let [client-code (if (map? c) (:client/code c) c)
-            filtered (filter #(= client-code (:client_code %)) rows)]
+             filtered (filter #(= client-code (:client-code %)) rows)]
         (reduce
          (fn [acc {:keys [processor type-name total]}]
            (update acc
@@ -138,21 +135,13 @@
              {:db/id (str (java.util.UUID/randomUUID))
               :sales-summary-item/sort-order 3
               :sales-summary-item/category (cond
-                                              (= type-name "CARD") "Card Refunds"
-                                              (= type-name "CASH") "Cash Refunds"
-                                              :else "Food App Refunds")
+                                             (= type-name "CARD") "Card Refunds"
+                                             (= type-name "CASH") "Cash Refunds"
+                                             :else "Food App Refunds")
               :ledger-mapped/amount total
               :ledger-mapped/ledger-side :ledger-side/credit})
            refunds))))
 
-(defn- get-fees [c date]
-  (when-let [fee (get-fee c date)]
-    {:db/id (str (java.util.UUID/randomUUID))
-     :sales-summary-item/sort-order 2
-     :sales-summary-item/category "Fees"
-     :ledger-mapped/amount fee
-     :ledger-mapped/ledger-side :ledger-side/debit}))
-
 (defn- get-tax-parquet [c date]
   (let [client-code (if (map? c) (:client/code c) c)
         date-str (.toString date)
@@ -188,8 +177,6 @@
        :ledger-mapped/ledger-side :ledger-side/credit
        :ledger-mapped/amount (- (+ total discount) tax)})))
 
-
-
 (defn get-fees [c date]
   (when-let [fee (get-fee c date)]
     {:db/id (str (java.util.UUID/randomUUID))
@@ -299,18 +286,17 @@
                      (filter identity)
                      (map (fn [z]
                             (assoc z :ledger-mapped/account (some-> z :sales-summary-item/category str/lower-case name->number lookup-account)
-                                   :sales-summary-item/manual? false))
-                          )) }]
+                                   :sales-summary-item/manual? false))))}]
         (if (seq (:sales-summary/items result))
           (do
             (alog/info ::upserting-summaries
                        :category-count (count (:sales-summary/items result)))
             @(dc/transact conn [[:upsert-entity result]]))
-           @(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
+          @(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
 
 (comment
   ;; TODO: Move to test file or proper location
-  (let [c (auto-ap.datomic/pull-attr (dc/db @conn) :db/id [:client/code "NGCL" ])
+  (let [c (auto-ap.datomic/pull-attr (dc/db @conn) :db/id [:client/code "NGCL"])
         date #inst "2024-04-14T00:00:00-07:00"]
     (get-payment-items c date)))
 
@@ -322,16 +308,13 @@
                           (map (fn [[sos]]
                                  [:db/retractEntity sos])))))
 
-
-
-
 (comment
   (auto-ap.datomic/transact-schema conn)
 
   @(dc/transact conn [{:db/ident       :sales-summary/total-unknown-processor-payments
-  :db/noHistory   true,
-  :db/valueType   :db.type/double
-  :db/cardinality :db.cardinality/one}])
+                       :db/noHistory   true,
+                       :db/valueType   :db.type/double
+                       :db/cardinality :db.cardinality/one}])
 
   (apply mark-dirty [:client/code "NGCL"] (last-n-days 30))
 
@@ -350,7 +333,7 @@
           [?sos :sales-summary/date ?d]
           [(= ?d #inst "2024-04-10T00:00:00-07:00")]]
         (dc/db conn))
-  
+
   (dc/q '[:find  ?n ?p2 (sum ?total)
           :with ?c
           :in $ [?clients ?start-date ?end-date]
@@ -363,23 +346,18 @@
         (dc/db conn)
         [[(auto-ap.datomic/pull-attr (dc/db conn) :db/id [:client/code "NGHW"])] #inst "2024-04-11T00:00:00-07:00" #inst "2024-04-11T00:00:00-07:00"])
 
-   (dc/q '[:find  ?n 
+  (dc/q '[:find  ?n
           :in $ [?clients ?start-date ?end-date]
           :where  [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
           [?e :sales-order/line-items ?li]
-          [?li :order-line-item/item-name ?n] ]
+          [?li :order-line-item/item-name ?n]]
         (dc/db conn)
         [[(auto-ap.datomic/pull-attr (dc/db conn) :db/id [:client/code "NGCL"])] #inst "2024-04-11T00:00:00-07:00" #inst "2024-04-24T00:00:00-07:00"])
-  
-@(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy}
-                    {:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}])
-
-(auto-ap.datomic/transact-schema conn)
-
-  )
 
+  @(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy}
+                      {:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}])
 
+  (auto-ap.datomic/transact-schema conn))
 
 (defn -main [& _]
   (execute "sales-summaries" sales-summaries-v2))
-  

src/clj/auto_ap/migration/cleanup_sales.clj

@@ -17,8 +17,8 @@
   "Return all entity IDs that have :sales-order/external-id."
   [db]
   (->> (d-api/q '[:find ?e
-                 :where [?e :sales-order/external-id]]
-               db)
+                  :where [?e :sales-order/external-id]]
+                db)
        (map first)))
 
 (defn- collect-child-ids
@@ -28,19 +28,19 @@
   [db order-ids]
   (let [order-set (set order-ids)
         charges    (->> (d-api/q '[:find ?c
-                                  :in $ [?o ...]
-                                  :where [$ ?o :sales-order/charges ?c]]
-                                db order-set)
+                                   :in $ [?o ...]
+                                   :where [$ ?o :sales-order/charges ?c]]
+                                 db order-set)
                         (map second))
         refunds    (->> (d-api/q '[:find ?r
-                                  :in $ [?o ...]
-                                  :where [$ ?o :sales-order/refunds ?r]]
-                                db order-set)
+                                   :in $ [?o ...]
+                                   :where [$ ?o :sales-order/refunds ?r]]
+                                 db order-set)
                         (map second))
         line-items (->> (d-api/q '[:find ?li
-                                  :in $ [?c ...]
-                                  :where [$ ?c :charge/line-items ?li]]
-                                db charges)
+                                   :in $ [?c ...]
+                                   :where [$ ?c :charge/line-items ?li]]
+                                 db charges)
                         (map second))]
     {:orders      order-ids
      :charges     (vec charges)
@@ -111,11 +111,12 @@
       (.toString (java.time.LocalDate/ofEpochDay d)))))
 
 (defn- object-exists?
-  "Check if an S3 object exists by attempting get-object."
+  "Check if an S3 object exists via head-object (no download)."
   [key]
   (try
     (s3/get-object {:bucket-name pq/*bucket*
-                    :key         key})
+                    :key         key}
+                   {:request-method :head})
     true
     (catch com.amazonaws.services.s3.model.AmazonS3Exception _
       false)))
@@ -149,7 +150,7 @@
         all-ids     (query-sales-order-ids db)
         group       (group-orders-by-month db all-ids)
         target-keys (get group [year month] [])]
-   (if (zero? (count target-keys))
+    (if (zero? (count target-keys))
       (do (println "  no orders found for" year "-" month)
           :skipped)
       (do
@@ -196,24 +197,24 @@
     (println "=== safe-cleanup-all"
              "months:" (count months)
              "dry-run? =" DRY-RUN?)
-    (doseq [[_ y m] months]
+    (doseq [[y m] months]
       (when-not DRY-RUN?
         (let [result  (verify-month-in-s3? y m)
               missing (:missing result)]
           (cond
             (:ok result)
-              (do (println "verified" y "-" m "S3 OK, deleting...")
-                  (delete-by-month conn$ nil y m))
+            (do (println "verified" y "-" m "S3 OK, deleting...")
+                (delete-by-month conn$ nil y m))
 
             (> (count missing) 0)
-              (do (println "ERROR" y "-" m "missing in S3:"
-                           (str/join ", " missing))
-                  (throw
-                   (ex-info
-                    "Missing S3 data — aborting!"
-                    {:year y :month m
-                     :missing missing})))
+            (do (println "ERROR" y "-" m "missing in S3:"
+                         (str/join ", " missing))
+                (throw
+                 (ex-info
+                  "Missing S3 data — aborting!"
+                  {:year y :month m
+                   :missing missing})))
 
             :else
-              (println "SKIPPING" y "-" m "no parquet files")))))
+            (println "SKIPPING" y "-" m "no parquet files")))))
     (println "safe-cleanup-all complete")))

src/clj/auto_ap/migration/sales_to_parquet.clj

@@ -10,14 +10,14 @@
       (write-dead-letter [flat])     ; write orphaned records"
   (:require [auto-ap.datomic :refer [conn]]
             [auto-ap.storage.parquet :as p]
-            [datomic.api :as dc]
-            [clj-time.core :as time]))
+            [clojure.string :as str]
+            [datomic.api :as dc]))
 
 (defn- fetch-all-sales-order-ids []
   "Query Datomic for all sales-order external-ids (as entity IDs).
     Returns a vector of entitity ids."
   (->> (dc/q '[:find ?e
-               :where [_ :sales-order/external-id ?_ext]]
+               :where [?e :sales-order/external-id _]]
              (dc/db conn))
        (map first)
        vec))
@@ -25,14 +25,16 @@
 (def ^:private sales-order-read
   '[:sales-order/external-id
     :sales-order/date
-    {:sales-order/client [:client/code]}
+    {:sales-order/client [:client/code :client/name]}
     :sales-order/location
-    :sales-order/vendor
+    {:sales-order/vendor [:vendor/name]}
     :sales-order/total
     :sales-order/tax
     :sales-order/tip
     :sales-order/discount
     :sales-order/service-charge
+    :sales-order/source
+    :sales-order/reference-link
     {:sales-order/charges
      [:charge/external-id
       :charge/type-name
@@ -40,16 +42,16 @@
       :charge/tax
       :charge/tip
       :charge/date
-      :charge/processor
+      {:charge/processor [:db/ident]}
       :charge/returns
-       {:charge/client [:client/code]}]}
+      {:charge/client [:client/code]}]}
     {:sales-order/line-items
      [:order-line-item/item-name
       :order-line-item/category
       :order-line-item/total
       :order-line-item/tax
       :order-line-item/discount
-      {:order-line-item/unit-price {}}
+      :order-line-item/unit-price
       :order-line-item/quantity
       :order-line-item/note]}])
 
@@ -61,92 +63,89 @@
                   sales-order-read
                   eids)))
 
-(defn- flatten-order-to-pieces! [order flat]
+(defn- flatten-order-to-pieces! [order date-str flat]
   "Flatten a pulled sales-order into :entity-type tagged maps.
     Appends to the existing flat vector, which is returned."
   (let [so-ext-id (:sales-order/external-id order)
-        so-date   (.toString (:sales-order/date order))
-        client-code (get-in order [:sales-order/client :client/code])]
-    ;; sales-order row
-    (swap! flat conj
-           {:entity-type       "sales-order"
-            :external-id       (str so-ext-id)
-            :client-code       client-code
-            :location          (:sales-order/location order)
-            :vendor            (:sales-order/vendor order)
-            :total             (:sales-order/total order)
-            :tax               (:sales-order/tax order)
-            :tip               (:sales-order/tip order)
-            :discount          (:sales-order/discount order)
-            :service-charge    (:sales-order/service-charge order)
-            :date              so-date})
-    ;; charges & line-items
+        so-date   date-str
+        client-code (get-in order [:sales-order/client :client/code])
+        vendor-name (get-in order [:sales-order/vendor :vendor/name])
+        charges (:sales-order/charges order)
+        items (:sales-order/line-items order)
+        payment-methods (->> charges (map :charge/type-name) distinct (str/join ","))
+        processors (->> charges (map #(get-in % [:charge/processor :db/ident])) (remove nil?) distinct (map name) (str/join ","))
+        categories (->> items (map :order-line-item/category) (remove nil?) distinct (str/join ","))]
+    (vswap! flat conj
+            {:entity-type       "sales-order"
+             :external-id       (str so-ext-id)
+             :client-code       client-code
+             :location          (:sales-order/location order)
+             :vendor            vendor-name
+             :total             (:sales-order/total order)
+             :tax               (:sales-order/tax order)
+             :tip               (:sales-order/tip order)
+             :discount          (:sales-order/discount order)
+             :service-charge    (:sales-order/service-charge order)
+             :date              so-date
+             :source            (:sales-order/source order)
+             :reference-link    (:sales-order/reference-link order)
+             :payment-methods   payment-methods
+             :processors        processors
+             :categories        categories})
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
-        (swap! flat conj
-               {:entity-type       "charge"
-                :external-id       (str (get chg :charge/external-id))
-                :type-name         (get chg :charge/type-name)
-                :total             (get chg :charge/total)
-                :tax               (get chg :charge/tax)
-                :tip               (get chg :charge/tip)
-                :date              so-date
-                :processor         (get-in chg [:charge/processor :db/ident])
-                :sales-order-external-id (str so-ext-id)})
-        ;; charge returns → sales-refund rows
+        (vswap! flat conj
+                {:entity-type       "charge"
+                 :external-id       (str (get chg :charge/external-id))
+                 :type-name         (get chg :charge/type-name)
+                 :total             (get chg :charge/total)
+                 :tax               (get chg :charge/tax)
+                 :tip               (get chg :charge/tip)
+                 :date              so-date
+                 :processor         (get-in chg [:charge/processor :db/ident])
+                 :sales-order-external-id (str so-ext-id)})
         (when-let [returns (:charge/returns chg)]
           (doseq [rt returns]
-            (swap! flat conj
-                   {:entity-type       "sales-refund"
-                    :type-name         (get rt :type-name)
-                    :total             (get rt :total)
-                    :sales-order-external-id (str so-ext-id)})))))
-    ;; line-items
+            (vswap! flat conj
+                    {:entity-type       "sales-refund"
+                     :type-name         (get rt :type-name)
+                     :total             (get rt :total)
+                     :sales-order-external-id (str so-ext-id)})))))
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
-        (swap! flat conj
-               {:entity-type         "line-item"
-                :item-name           (get li :order-line-item/item-name)
-                :category            (get li :order-line-item/category)
-                :total               (get li :order-line-item/total)
-                :tax                 (get li :order-line-item/tax)
-                :discount            (get li :order-line-item/discount)
-                :sales-order-external-id (str so-ext-id)})))))
+        (vswap! flat conj
+                {:entity-type         "line-item"
+                 :item-name           (get li :order-line-item/item-name)
+                 :category            (get li :order-line-item/category)
+                 :total               (get li :order-line-item/total)
+                 :tax                 (get li :order-line-item/tax)
+                 :discount            (get li :order-line-item/discount)
+                 :sales-order-external-id (str so-ext-id)})))))
 
 (defn -fetch-order-ids-for-date
   "Query Datomic for all sales-order eids on a given business date."
   [db date-str]
-  (let [day-ms (.toEpochSecond ^java.time.LocalDate (java.time.LocalDate/parse date-str))
-        start (* day-ms 1000)
-        end (+ start (* 86400000))]
+  (let [ld (java.time.LocalDate/parse date-str)
+        start (-> ld (.atStartOfDay (java.time.ZoneId/of "America/Los_Angeles")) .toInstant java.util.Date/from)
+        end (-> ld (.plusDays 1) (.atStartOfDay (java.time.ZoneId/of "America/Los_Angeles")) .toInstant java.util.Date/from)]
     (->> (dc/q '[:find ?e
-                 :in $ ?start-ms ?end-ms
-                 :where [_ :sales-order/date ?d]
-                        [(>= ?d ?start-ms)]
-                        [(<= ?d ?end-ms)]]
+                 :in $ ?start ?end
+                 :where [?e :sales-order/date ?d]
+                 [(>= ?d ?start)]
+                 [(< ?d ?end)]]
                db start end)
          (map first)
          vec)))
 
-
-(defn- date-seq [start end]
-  "Seq of YYYY-MM-DD strings between start and end inclusive."
-  (let [sd   (java.time.LocalDate/parse start)
-        ed   (java.time.LocalDate/parse end)
-        days (int (Math/abs (- (.toEpochDay sd)
-                               (.toEpochDay ed))))]
-    (for [i (range 0 (inc days))]
-      (.toString (.plusDays sd i)))))
-
-(defn- write-day-by-day
+(defn write-day-by-day
   ([start-date end-date]
-   (write-day-by-day start-date end-date nil))
+   (write-day-by-day start-date end-date {}))
   ([start-date end-date opts]
    (let [all-dates  (set (or (opts :date-set) []))
-         date-range (if (empty? all-dates)
-                      (date-seq start-date end-date)
-                      (filter all-dates
-                              (date-seq start-date end-date)))
+          date-range (if (empty? all-dates)
+                       (p/date-seq start-date end-date)
+                       (filter all-dates
+                               (p/date-seq start-date end-date)))
          batch-size (or (opts :batch-size) 100)]
      (doseq [^String day date-range]
        (println "[migration] processing" day)
@@ -156,14 +155,14 @@
            (let [orders (pull-sales-order-data batch)
                  flat   (volatile! [])]
              (doseq [o orders]
-               (flatten-order-to-pieces! o flat))
+                (flatten-order-to-pieces! o day flat))
              (doseq [r @flat]
                (p/buffer! (:entity-type r) r)))))
-       (doseq [etype ["sales-order" "charge"
-                      "line-item" "sales-refund"]]
-         (p/flush-to-parquet! etype))
+        (doseq [etype ["sales-order" "charge"
+                       "line-item" "sales-refund"]]
+          (p/flush-to-parquet! etype day))
        (println "[migration]" day "complete"))
-      {:status :completed :total-days (count date-range)})))
+     {:status :completed :total-days (count date-range)})))
 
 (defn- write-dead-letter
   ([flat]
@@ -181,10 +180,11 @@
   "Flush all entity-type buffers, tracking counts."
   (let [etypes ["sales-order" "charge"
                 "line-item" "sales-refund"]
+        today (.toString (java.time.LocalDate/now))
         start  (p/total-buf-count)]
     (doseq [et etypes]
       (try
-        (p/flush-to-parquet! et)
+        (p/flush-to-parquet! et today)
         (catch Exception e
           (println "[migration/flush]" et "error:" (.getMessage e)))))
     {:records-flush (- (p/total-buf-count) start)}))
@@ -192,11 +192,11 @@
 (defn- get-date-range []
   "Get the earliest and latest business dates from Datomic."
   (let [dates (->> (dc/q '[:find ?d
-                          :where [_ :sales-order/date ?d]]
-                        (dc/db conn))
-                  (map first)
-                  distinct
-                  sort)]
+                           :where [_ :sales-order/date ?d]]
+                         (dc/db conn))
+                   (map first)
+                   distinct
+                   sort)]
     [(when (seq dates) (.toString (first dates)))
      (when (seq dates) (.toString (last dates)))]))
 
@@ -218,9 +218,9 @@
         (doseq [o (pull-sales-order-data order-ids)
                 :when (not (:sales-order/date o))]
           (let [flat (volatile! [])]
-            (flatten-order-to-pieces! o flat)
+             (flatten-order-to-pieces! o "unknown" flat)
             (doseq [r @flat]
-               (p/buffer! "dead" r))))
+              (p/buffer! "dead" r))))
         (write-day-by-day start-date end-date {:batch-size 100})
         (flush-all-types)
         (println "[migration] done")

src/clj/auto_ap/routes/ezcater_xls.clj

@@ -55,54 +55,54 @@
         event-date (some-> (excel/xls-date->date event-date)
                            coerce/to-date-time
                            atime/as-local-time
-                           coerce/to-date )]
-    (cond (and event-date client-id location ) 
+                           coerce/to-date)]
+    (cond (and event-date client-id location)
           [:order #:sales-order
-           {:date           event-date
-            :external-id    (str "ezcater/order/" client-id "-" location "-" order-number)
-            :client         client-id
-            :location       location
-            :reference-link (str order-number)
-            :line-items     [#:order-line-item
-                             {:external-id (str "ezcater/order/" client-id "-" location "-" order-number "-" 0)
-                              :item-name   "EZCater Catering"
-                              :category    "EZCater Catering"
-                              :discount    (fmt-amount (or adjustments 0.0))
-                              :tax         (fmt-amount tax)
-                              :total       (fmt-amount (+ food-total
-                                                          tax))}]
+                   {:date           event-date
+                    :external-id    (str "ezcater/order/" client-id "-" location "-" order-number)
+                    :client         client-id
+                    :location       location
+                    :reference-link (str order-number)
+                    :line-items     [#:order-line-item
+                                      {:external-id (str "ezcater/order/" client-id "-" location "-" order-number "-" 0)
+                                       :item-name   "EZCater Catering"
+                                       :category    "EZCater Catering"
+                                       :discount    (fmt-amount (or adjustments 0.0))
+                                       :tax         (fmt-amount tax)
+                                       :total       (fmt-amount (+ food-total
+                                                                   tax))}]
 
-            :charges        [#:charge
-                             {:type-name   "CARD"
-                              :date        event-date
-                              :client      client-id
-                              :location    location
-                              :external-id (str "ezcater/charge/" client-id "-" location "-" order-number "-" 0)
-                              :processor   :ccp-processor/ezcater
-                              :total       (fmt-amount (+ food-total
-                                                          tax
-                                                          tip))
-                              :tip         (fmt-amount tip)}]
-            :total          (fmt-amount (+ food-total
-                                           tax
-                                           (or adjustments 0.0)))
-            :discount       (fmt-amount (or adjustments 0.0))
-            :service-charge (fmt-amount (+ fee commission))
-            :tax            (fmt-amount tax)
-            :tip            (fmt-amount tip)
-            :returns        0.0
-            :vendor         :vendor/ccp-ezcater}]
+                    :charges        [#:charge
+                                      {:type-name   "CARD"
+                                       :date        event-date
+                                       :client      client-id
+                                       :location    location
+                                       :external-id (str "ezcater/charge/" client-id "-" location "-" order-number "-" 0)
+                                       :processor   :ccp-processor/ezcater
+                                       :total       (fmt-amount (+ food-total
+                                                                   tax
+                                                                   tip))
+                                       :tip         (fmt-amount tip)}]
+                    :total          (fmt-amount (+ food-total
+                                                   tax
+                                                   (or adjustments 0.0)))
+                    :discount       (fmt-amount (or adjustments 0.0))
+                    :service-charge (fmt-amount (+ fee commission))
+                    :tax            (fmt-amount tax)
+                    :tip            (fmt-amount tip)
+                    :returns        0.0
+                    :vendor         :vendor/ccp-ezcater}]
 
-      caterer-name
-      (do 
-        (alog/warn ::missing-client
-                   :order order-number
-                   :store-name store-name 
-                   :caterer-name caterer-name)
-        [:missing caterer-name])
+          caterer-name
+          (do
+            (alog/warn ::missing-client
+                       :order order-number
+                       :store-name store-name
+                       :caterer-name caterer-name)
+            [:missing caterer-name])
 
-      :else
-      nil)))
+          :else
+          nil)))
 
 (defn- flatten-order-to-parquet! [order]
   "Flatten a sales-order into entity-type tagged maps and buffer to parquet."
@@ -125,25 +125,25 @@
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
         (parquet/buffer! "charge"
-                        {:entity-type "charge"
-                         :external-id (:charge/external-id chg)
-                         :type-name   (:charge/type-name chg)
-                         :total       (:charge/total chg)
-                         :tax         (:charge/tax chg)
-                         :tip         (:charge/tip chg)
-                         :date        so-date
-                         :processor   (some-> (:charge/processor chg) name)
-                         :sales-order-external-id so-ext-id})))
+                         {:entity-type "charge"
+                          :external-id (:charge/external-id chg)
+                          :type-name   (:charge/type-name chg)
+                          :total       (:charge/total chg)
+                          :tax         (:charge/tax chg)
+                          :tip         (:charge/tip chg)
+                          :date        so-date
+                          :processor   (some-> (:charge/processor chg) name)
+                          :sales-order-external-id so-ext-id})))
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
         (parquet/buffer! "line-item"
-                        {:entity-type "line-item"
-                         :item-name   (:order-line-item/item-name li)
-                         :category    (:order-line-item/category li)
-                         :total       (:order-line-item/total li)
-                         :tax         (:order-line-item/tax li)
-                         :discount    (:order-line-item/discount li)
-                         :sales-order-external-id so-ext-id})))))
+                         {:entity-type "line-item"
+                          :item-name   (:order-line-item/item-name li)
+                          :category    (:order-line-item/category li)
+                          :total       (:order-line-item/total li)
+                          :tax         (:order-line-item/tax li)
+                          :discount    (:order-line-item/discount li)
+                          :sales-order-external-id so-ext-id})))))
 
 (defn stream->sales-orders [s]
   (let [clients (map first (dc/q '[:find (pull ?c [:client/code
@@ -158,7 +158,7 @@
         object (str "/ezcater-xls/" (str (java.util.UUID/randomUUID)))]
     (mu/log ::writing-temp-xls
             :location object)
-    (s3/put-object {:bucket-name (:data-bucket env) 
+    (s3/put-object {:bucket-name (:data-bucket env)
                     :key object
                     :input-stream s})
     (into []
@@ -200,13 +200,13 @@
     });")]])])
 
 (defn upload-xls [{:keys [identity] :as request}]
-  
+
   (let [file (or (get (:params request) :file)
                  (get (:params request) "file"))]
     (mu/log ::uploading-file
             :file file)
     (with-open [s (io/input-stream (:tempfile file))]
-      (try 
+      (try
         (let [parse-results (stream->sales-orders s)
               new-orders (->> parse-results
                               (filter (comp #{:order} first))
@@ -235,7 +235,7 @@
                                 [:li ml])]])]))
         (catch Exception e
           (alog/error ::import-error
-                    :error e)
+                      :error e)
           (html-response [:div (.getMessage e)]))))))
 
 (defn page [{:keys [matched-route request-method] :as request}]

src/clj/auto_ap/square/core3.clj

@@ -28,11 +28,9 @@
     "Authorization"  (str "Bearer " (:client/square-auth-token client))
     "Content-Type"   "application/json"}))
 
-
 (defn ->square-date [d]
   (f/unparse (f/formatter "YYYY-MM-dd'T'HH:mm:ssZZ") d))
 
-
 (def manifold-api-stream
   (let [stream (s/stream 100)]
     (->> stream
@@ -43,10 +41,10 @@
                     (de/loop [attempt 0]
                       (-> (de/chain (de/future-with (ex/execute-pool)
                                                     #_(log/info ::request-started
-                                                              :url (:url request)
-                                                              :attempt attempt
-                                                              :source "Square 3"
-                                                              :background-job "Square 3")
+                                                                :url (:url request)
+                                                                :attempt attempt
+                                                                :source "Square 3"
+                                                                :background-job "Square 3")
                                                     (try
                                                       (client/request (assoc request
                                                                              :socket-timeout 10000
@@ -105,7 +103,6 @@
                    :exception error))
       []))))
 
-
 (def item-cache (atom {}))
 
 (defn fetch-catalog [client i v]
@@ -125,13 +122,11 @@
     #(do (swap! item-cache assoc i %)
          %))))
 
-
 (defn fetch-catalog-cache [client i version]
   (if (get @item-cache i)
     (de/success-deferred (get @item-cache i))
     (fetch-catalog client i version)))
 
-
 (defn item->category-name-impl [client item version]
   (capture-context->lc
    (cond (:item_id (:item_variation_data item))
@@ -162,7 +157,6 @@
                      :item item)
            "Uncategorized"))))
 
-
 (defn item-id->category-name [client i version]
   (capture-context->lc
    (-> [client i]
@@ -227,7 +221,6 @@
                                  (concat (:orders result) continued-results))))
                    (:orders result)))))))
 
-
 (defn search
   ([client location start end]
    (capture-context->lc
@@ -251,11 +244,9 @@
                                   (concat (:orders result) continued-results))))
                     (:orders result))))))))
 
-
 (defn amount->money [amt]
   (* 0.01 (or (:amount amt) 0.0)))
 
-
 ;; to get totals: 
 (comment
   (reduce
@@ -281,7 +272,7 @@
      :reference-link (str (url/url  "https://squareup.com/receipt/preview" (:id t)))
      :external-id (when (:id t)
                     (str "square/charge/" (:id t)))
-     :processor   (cond 
+     :processor   (cond
                     (#{"OTHER" "THIRD_PARTY_CARD"} (:type t))
                     (condp = (some-> (:note t) str/lower-case)
                       "doordash" :ccp-processor/doordash
@@ -294,7 +285,9 @@
                       (condp = (:name (:source order))
                         "GRUBHUB"          :ccp-processor/grubhub
                         "UBEREATS"         :ccp-processor/uber-eats
+                        "Uber Eats"         :ccp-processor/uber-eats
                         "DOORDASH"         :ccp-processor/doordash
+                        "DoorDash"         :ccp-processor/doordash
                         "Koala"            :ccp-processor/koala
                         "koala-production" :ccp-processor/koala
                         :ccp-processor/na))
@@ -350,7 +343,10 @@
                        (s/reduce conj []))]
                      [(remove-nils
                        #:sales-order
-                        {:date        (coerce/to-date (time/to-time-zone (coerce/to-date-time (:created_at order)) (time/time-zone-for-id "America/Los_Angeles")))
+                        {:date        (if (= "Invoices" (:name (:source order)))
+                                        (when (:closed_at order)
+                                          (coerce/to-date (time/to-time-zone (coerce/to-date-time (:closed_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
+                                        (coerce/to-date (time/to-time-zone (coerce/to-date-time (:created_at order)) (time/time-zone-for-id "America/Los_Angeles"))))
                          :client      (:db/id client)
                          :location    (:square-location/client-location location)
                          :external-id (str "square/order/" (:client/code client) "-" (:square-location/client-location location) "-" (:id order))
@@ -380,6 +376,9 @@
                                     ;; sometimes orders stay open in square. At least one payment
                                     ;; is needed to import, in order to avoid importing orders in-progress.
   (and
+   (if (= "Invoices" (:name (:source order)))
+     (boolean (:closed_at order))
+     true)
    (or (> (count (:tenders order)) 0)
        (seq (:returns order)))
    (or (= #{} (set (map #(:status (:card_details %)) (:tenders order))))
@@ -408,7 +407,6 @@
                             :client client
                             :location location)))))))
 
-
 (defn get-payment [client p]
   (de/chain (manifold-api-call
              {:url (str "https://connect.squareup.com/v2/payments/" p)
@@ -417,7 +415,6 @@
             :body
             :payment))
 
-
 (defn continue-payout-entry-list [c l poi cursor]
   (capture-context->lc lc
                        (de/chain
@@ -602,8 +599,8 @@
         so-date   (some-> (:sales-order/date order) .toString)
         client    (:sales-order/client order)
         client-code (when client (if (map? client)
-                                  (:client/code client)
-                                  client))]
+                                   (:client/code client)
+                                   client))]
     (parquet/buffer! "sales-order"
                      {:entity-type "sales-order"
                       :external-id so-ext-id
@@ -619,32 +616,32 @@
     (when-let [charges (:sales-order/charges order)]
       (doseq [chg charges]
         (parquet/buffer! "charge"
-                        {:entity-type "charge"
-                         :external-id (:charge/external-id chg)
-                         :type-name   (:charge/type-name chg)
-                         :total       (:charge/total chg)
-                         :tax         (:charge/tax chg)
-                         :tip         (:charge/tip chg)
-                         :date        so-date
-                         :processor   (some-> (:charge/processor chg) name)
-                         :sales-order-external-id so-ext-id})
+                         {:entity-type "charge"
+                          :external-id (:charge/external-id chg)
+                          :type-name   (:charge/type-name chg)
+                          :total       (:charge/total chg)
+                          :tax         (:charge/tax chg)
+                          :tip         (:charge/tip chg)
+                          :date        so-date
+                          :processor   (some-> (:charge/processor chg) name)
+                          :sales-order-external-id so-ext-id})
         (when-let [returns (:charge/returns chg)]
           (doseq [rt returns]
             (parquet/buffer! "sales-refund"
-                            {:entity-type "sales-refund"
-                             :type-name   (:type-name rt)
-                             :total       (:total rt)
-                             :sales-order-external-id so-ext-id})))))
+                             {:entity-type "sales-refund"
+                              :type-name   (:type-name rt)
+                              :total       (:total rt)
+                              :sales-order-external-id so-ext-id})))))
     (when-let [items (:sales-order/line-items order)]
       (doseq [li items]
         (parquet/buffer! "line-item"
-                        {:entity-type "line-item"
-                         :item-name   (:order-line-item/item-name li)
-                         :category    (:order-line-item/category li)
-                         :total       (:order-line-item/total li)
-                         :tax         (:order-line-item/tax li)
-                         :discount    (:order-line-item/discount li)
-                         :sales-order-external-id so-ext-id})))))
+                         {:entity-type "line-item"
+                          :item-name   (:order-line-item/item-name li)
+                          :category    (:order-line-item/category li)
+                          :total       (:order-line-item/total li)
+                          :tax         (:order-line-item/tax li)
+                          :discount    (:order-line-item/discount li)
+                          :sales-order-external-id so-ext-id})))))
 
 (defn upsert
   ([client]
@@ -666,8 +663,7 @@
                         (catch Exception e
                           (log/error ::buffer-failed
                                      :exception e
-                                     :order (:sales-order/external-id order)))))))))))
-
+                                     :order (:sales-order/external-id order))))))))))))
 
 (defn upsert-payouts
   ([client]
@@ -717,7 +713,6 @@
 
                    (log/info ::done-loading-refunds)))))))
 
-
 (defn get-cash-shift [client id]
   (de/chain (manifold-api-call {:url (str (url/url "https://connect.squareup.com/v2/cash-drawers/shifts" id))
                                 :method :get
@@ -876,8 +871,6 @@
     d1
     d2))
 
-
-
 (defn remove-voided-orders
   ([client]
    (apply de/zip
@@ -904,7 +897,7 @@
                                                     (:sales-order/external-id o))))))
                            (s/map (fn [[o]]
                                     [[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]]))
-                           
+
                            (s/reduce into [])))
 
                     (fn [results]
@@ -912,28 +905,28 @@
                         (doseq [x (partition-all 100 results)]
                           (log/info ::removing-orders
                                     :count (count x))
-                          @(dc/transact-async conn x)))))
+                          @(dc/transact-async conn x)
                           (de/catch (fn [e]
                                       (log/warn ::couldnt-remove :error e)
-                                      nil) ))))))
+                                      nil)))))))))))
 
-#_(comment 
-  (require 'auto-ap.time-reader)
+#_(comment
+ (require 'auto-ap.time-reader)
 
-  
+ @(let [[c [l]] (get-square-client-and-location "DBFS")]
+    (log/peek :x  [c l])
+    (search c l #clj-time/date-time "2026-03-28" #clj-time/date-time "2026-03-29"))
 
-  @(let [[c [l]] (get-square-client-and-location "NGAK")  ]
-     (log/peek :x  [ c l])
+ @(let [[c [l]] (get-square-client-and-location "NGAK")]
+    (log/peek :x  [c l])
 
-     (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
-  (doseq [c (get-square-clients)]
-    (try 
-      @(remove-voided-orders c)
-      (catch Exception e
-        nil)))
-
-  
-  )
+    (remove-voided-orders c l #clj-time/date-time "2024-04-11" #clj-time/date-time "2024-04-15"))
+ (doseq [c (get-square-clients)]
+   (try
+     @(remove-voided-orders c)
+     (catch Exception e
+       nil)))
+ )
 
 (defn upsert-all [& clients]
   (capture-context->lc
@@ -1002,8 +995,6 @@
    [:clients clients]
    @(apply upsert-all clients)))
 
-
-
 (comment
   (defn refunds-raw-cont
     ([client l cursor so-far]
@@ -1030,19 +1021,18 @@
                                      :headers (client-base-headers client)
                                      :as      :json})
                  :body)))
-(->>
-     @(let [[c [l]] (get-square-client-and-location "NGGG")]
+  (->>
+   @(let [[c [l]] (get-square-client-and-location "NGGG")]
 
+      (search c l (time/now) (time/plus (time/now) (time/days -1))))
 
-        (search c l (time/plus (time/now))))
-     (filter (fn [r]
-               (str/starts-with? (:created_at r) "2024-03-14"))))
+   (filter (fn [r]
+             (str/starts-with? (:created_at r) "2024-03-14"))))
 
   (def refs
     (->>
      @(let [[c [l]] (get-square-client-and-location "NGGG")]
 
-
         (refunds-raw-cont c l nil []))
      (filter (fn [r]
                (str/starts-with? (:created_at r) "2024-03-14")))))
@@ -1053,36 +1043,31 @@
 
     (map (fn [r] @(get-payment c (:payment_id r))) refs))
 
-(get-square-client-and-location "NGGB")
+  (get-square-client-and-location "NGGB")
 
   (def my-results
     (let [[c [l]] (get-square-client-and-location "NGFA")]))
 
   (clojure.data.csv/write-csv *out*
-                            (for [c  (get-square-clients)
-                                  l (:client/square-locations c)
-                                  :when (:square-location/client-location l)
-                                  bad-row (try (->> @(search c l (coerce/to-date-time #inst "2024-04-01T00:00:00-07:00") (coerce/to-date-time #inst "2024-04-15T23:59:00-07:00"))
-                                                    (filter #(not (should-import-order? %)))
-                                                    (map #(first (deref (order->sales-order c l %))))
-                                                    (filter (fn already-exists [o]
-                                                              (when (:sales-order/external-id o)
-                                                                (seq (dc/q '[:find ?i
-                                                                             :in $ ?ei
-                                                                             :where [?i :sales-order/external-id ?ei]]
-                                                                           (dc/db conn)
-                                                                           (:sales-order/external-id o)))))))
-                                               (catch Exception e
-                                                 []))]
-                              [(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row)  (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
-                            :separator \tab)
-  
-
-
-
-  ;; => 
-
+                              (for [c  (get-square-clients)
+                                    l (:client/square-locations c)
+                                    :when (:square-location/client-location l)
+                                    bad-row (try (->> @(search c l (coerce/to-date-time #inst "2024-04-01T00:00:00-07:00") (coerce/to-date-time #inst "2024-04-15T23:59:00-07:00"))
+                                                      (filter #(not (should-import-order? %)))
+                                                      (map #(first (deref (order->sales-order c l %))))
+                                                      (filter (fn already-exists [o]
+                                                                (when (:sales-order/external-id o)
+                                                                  (seq (dc/q '[:find ?i
+                                                                               :in $ ?ei
+                                                                               :where [?i :sales-order/external-id ?ei]]
+                                                                             (dc/db conn)
+                                                                             (:sales-order/external-id o)))))))
+                                                 (catch Exception e
+                                                   []))]
+                                [(:client/code c) (atime/unparse-local (clj-time.coerce/to-date-time (:sales-order/date bad-row)) atime/normal-date) (:sales-order/total bad-row)  (:sales-order/tax bad-row) (:sales-order/tip bad-row) (:db/id bad-row)])
+                              :separator \tab)
 
+;; => 
 
   (require 'auto-ap.time-reader)
 
@@ -1091,35 +1076,40 @@
   (clojure.pprint/pprint (let [[c [l]] (get-square-client-and-location "NGVT")]
                            l
 
-
                            (def z @(search c l #clj-time/date-time "2025-02-23T00:00:00-08:00"
-                                             #clj-time/date-time "2025-02-28T00:00:00-08:00"))
-                           (take 10 (map #(first (deref (order->sales-order c l %))) z)))
+                                           #clj-time/date-time "2025-02-28T00:00:00-08:00"))
+                           (take 10 (map #(first (deref (order->sales-order c l %))) z))))
 
-
-                         )
-  
-  
-
-
-  (->> z 
+  (->> z
        (filter (fn [o]
                  (seq (filter (comp #{"OTHER"} :type) (:tenders o)))))
        (filter #(not (:name (:source %))))
-       (count)
-       
-       )
-  (doseq [c (get-square-clients)]
-    (println "Upserting" (:client/name c))
-    @(upsert c))
+       (count))
+
+  (doseq [[code] (seq (dc/q '[:find ?code
+                              :in $
+                              :where [?o :sales-order/date ?d]
+                              [(>= ?d #inst "2026-01-01")]
+                              [?o :sales-order/source "Invoices"]
+                              [?o :sales-order/client ?c]
+                              [?c :client/code ?code]]
+                            (dc/db conn)))
+          :let [[c [l]] (get-square-client-and-location code)]
+          order @(search c l #clj-time/date-time "2026-01-01T00:00:00-08:00" (time/now))
+          :when (= "Invoices" (:name (:source order)))
+          :let [[sales-order] @(order->sales-order c l order)]]
+
+    (when (should-import-order? order)
+      (println "DATE IS" (:sales-order/date sales-order))
+      (when (some-> (:sales-order/date sales-order) coerce/to-date-time (time/after?  #clj-time/date-time "2026-2-16T00:00:00-08:00"))
+        (println "WOULD UPDATE" sales-order)
+        @(dc/transact auto-ap.datomic/conn [sales-order]))
+      #_@(dc/transact)
+      (println "DONE")))
 
   #_(filter (comp #{"OTHER"} :type) (mapcat :tenders z))
-  
 
-  (let [[c [l]] (get-square-client-and-location "LFHH")]
-    (search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
+  @(let [[c [l]] (get-square-client-and-location "NGRY")]
+     #_(search c l (clj-time.coerce/from-date #inst "2025-02-28") (clj-time.coerce/from-date #inst "2025-03-01"))
 
-    (:order (get-order c l "CLjQqkzVfGa82o5hEFUrGtUGO6QZY" ))
-    )
-  
-  )
+     (order->sales-order c l (:order (get-order c l "KdvwntmfMNTKBu8NOocbxatOs18YY")))))

src/clj/auto_ap/ssr/components/date_range.clj

@@ -1,11 +1,13 @@
 (ns auto-ap.ssr.components.date-range
   (:require [auto-ap.ssr.components :as com]
+            [auto-ap.ssr.components.buttons :as but]
+            [auto-ap.ssr.svg :as svg]
             [auto-ap.time :as atime]
             [clj-time.coerce :as c]
             [clj-time.core :as t]
             [clj-time.periodic :as per]))
 
-(defn date-range-field [{:keys [value id]}]
+(defn date-range-field [{:keys [value id apply-button?]}]
   [:div {:id id}
    (com/field {:label "Date Range"}
               [:div.space-y-4
@@ -21,11 +23,17 @@
                                                       (atime/unparse-local atime/normal-date))
                                  :placeholder "Date"
                                  :size        :small
-                                 :class "shrink"})
+                                 :class "shrink date-filter-input"})
 
                 (com/date-input {:name        "end-date"
                                  :value       (some-> (:end value)
                                                       (atime/unparse-local atime/normal-date))
                                  :placeholder "Date"
                                  :size        :small
-                                 :class "shrink"})]])])
+                                 :class "shrink date-filter-input"})
+                (when apply-button?
+                  (but/button- {:color :secondary
+                                :size :small
+                                :type "button"
+                                "x-on:click" "$dispatch('datesApplied')"}
+                               "Apply"))]])])

src/clj/auto_ap/ssr/invoices.clj

@@ -37,7 +37,7 @@
    [auto-ap.ssr.invoice.common :refer [default-read]]
    [auto-ap.ssr.invoice.import :as invoice-import]
    [auto-ap.ssr.invoice.new-invoice-wizard :as new-invoice-wizard :refer [location-select*]]
-   [auto-ap.ssr.pos.common :refer [date-range-field*]]
+   [auto-ap.ssr.components.date-range :as dr]
    [auto-ap.ssr.svg :as svg]
    [auto-ap.ssr.utils
              :refer [apply-middleware-to-all-handlers assert-schema
@@ -77,7 +77,7 @@
     [:div {:id "exact-match-id-tag"}]))
 
 (defn filters [request]
-  [:form#invoice-filters {"hx-trigger"   "change delay:500ms, keyup changed from:.hot-filter delay:1000ms"
+  [:form#invoice-filters {"hx-trigger"   "datesApplied, change delay:500ms from:.filter-trigger, keyup changed from:.hot-filter delay:1000ms"
                           "hx-get"       (bidi/path-for ssr-routes/only-routes
                                                         ::route/table)
                           "hx-target"    "#entity-table"
@@ -92,7 +92,8 @@
                                :url (bidi/path-for ssr-routes/only-routes :vendor-search)
                                :value       (:vendor (:query-params request))
                                :value-fn :db/id
-                               :content-fn :vendor/name}))
+                               :content-fn :vendor/name
+                               :class       "filter-trigger"}))
     (com/field {:label "Account"}
                (com/typeahead {:name        "account"
                                :id          "account"
@@ -100,8 +101,12 @@
                                :value       (:account (:query-params request))
                                :value-fn :db/id
                                :content-fn #(:account/name (d-accounts/clientize (dc/pull (dc/db conn) d-accounts/default-read (:db/id %))
-                                                                                 (:db/id (:client request))))}))
-    (date-range-field* request)
+                                                                                 (:db/id (:client request))))
+                               :class       "filter-trigger"}))
+    (dr/date-range-field {:value {:start (:start-date (:query-params request))
+                                 :end (:end-date (:query-params request))}
+                        :id "date-range"
+                        :apply-button? true})
     (com/field {:label "Check #"}
                (com/text-input {:name        "check-number"
                                 :id          "check-number"
@@ -486,7 +491,10 @@
                  :fetch-page         fetch-page
                  :oob-render
                  (fn [request]
-                   [(assoc-in (date-range-field* request) [1 :hx-swap-oob] true)
+                   [(assoc-in (dr/date-range-field {:value {:start (:start-date (:query-params request))
+                                                           :end (:end-date (:query-params request))}
+                                                   :id "date-range"
+                                                   :apply-button? true}) [1 :hx-swap-oob] true)
                     (assoc-in (exact-match-id* request)  [1 :hx-swap-oob] true)])
                  :query-schema query-schema
                  :parse-query-params (fn [p]

src/clj/auto_ap/ssr/ledger/balance_sheet.clj

@@ -81,7 +81,7 @@
           data (into []
                      (for [client-id client-ids
                            d date
-                           [client-id account-id location debits credits balance count] (iol-ion.query/detailed-account-snapshot (dc/db conn) client-id (coerce/to-date (time/plus d (time/days 1))))
+                           [client-id account-id location debits credits balance count] (iol-ion.query/detailed-account-snapshot (dc/db conn) client-id (coerce/to-date d))
                            :let [account ((or (lookup-account client-id) {}) account-id)]]
                        {:client-id client-id
                         :account-id account-id

src/clj/auto_ap/ssr/payments.clj

@@ -104,19 +104,18 @@
                                   :size        :small})])
     (com/field {:label "Payment Type"}
                (com/radio-card {:size    :small
-                           :name    "payment-type"
-                           :value (:payment-type (:query-params request))
-                           :options [{:value   ""
-                                      :content "All"}
-                                     {:value   "cash"
-                                      :content "Cash"}
-                                     {:value   "check"
-                                      :content "Check"}
-                                     {:value   "debit"
-                                      :content "Debit"}]}))
+                                :name    "payment-type"
+                                :value (:payment-type (:query-params request))
+                                :options [{:value   ""
+                                           :content "All"}
+                                          {:value   "cash"
+                                           :content "Cash"}
+                                          {:value   "check"
+                                           :content "Check"}
+                                          {:value   "debit"
+                                           :content "Debit"}]}))
     (exact-match-id* request)]])
 
-
 (def default-read '[*
                     [:payment/date :xform clj-time.coerce/from-date]
                     {:invoice-payment/_payment [* {:invoice-payment/invoice [*]}]}
@@ -212,7 +211,6 @@
                                                             '[(iol-ion.query/dollars= ?transaction-amount ?amount)]]}
                                             :args  [(:amount query-params)]})
 
-
                               (:status route-params)
                               (merge-query {:query {:in    ['?status]
                                                     :where ['[?e :payment/status ?status]]}
@@ -243,30 +241,30 @@
     refunds))
 
 (defn sum-visible-pending [ids]
-  (->> 
-    (dc/q {:find  ['?id '?o]
-           :in    ['$ '[?id ...]]
-           :where ['[?id :payment/amount ?o]
-                   '[?id :payment/status :payment-status/pending]]}
-          (dc/db conn)
-          ids)
+  (->>
+   (dc/q {:find  ['?id '?o]
+          :in    ['$ '[?id ...]]
+          :where ['[?id :payment/amount ?o]
+                  '[?id :payment/status :payment-status/pending]]}
+         (dc/db conn)
+         ids)
    (map last)
    (reduce
     +
     0.0)))
 
 (defn sum-client-pending [clients]
-  (->> 
-    (dc/q {:find '[?e ?a]
-           :in '[$ [?clients ?start ?end]]
-           :where '[[(iol-ion.query/scan-payments $ ?clients ?start ?end) [[?e _ ?sort-default] ...]]
-                    [?e :payment/status :payment-status/pending]
-                    [?e :payment/amount ?a]]}
-          (dc/db conn)
-          [clients
-           nil
-           nil])
-          
+  (->>
+   (dc/q {:find '[?e ?a]
+          :in '[$ [?clients ?start ?end]]
+          :where '[[(iol-ion.query/scan-payments $ ?clients ?start ?end) [[?e _ ?sort-default] ...]]
+                   [?e :payment/status :payment-status/pending]
+                   [?e :payment/amount ?a]]}
+         (dc/db conn)
+         [clients
+          nil
+          nil])
+
    (map last)
    (reduce
     +
@@ -277,16 +275,14 @@
         {ids-to-retrieve :ids matching-count :count
          all-ids :all-ids} (fetch-ids db request)]
 
-
     [(->> (hydrate-results ids-to-retrieve db request))
      matching-count
      (sum-visible-pending all-ids)
      (sum-client-pending (extract-client-ids (:clients request)
-                                          (:client request)
-                                          (:client-id (:query-params request))
-                                          (when (:client-code (:query-params request))
-                                            [:client/code (:client-code (:query-params request))])))
-     ]))
+                                             (:client request)
+                                             (:client-id (:query-params request))
+                                             (when (:client-code (:query-params request))
+                                               [:client/code (:client-code (:query-params request))])))]))
 
 (def query-schema (mc/schema
                    [:maybe [:map {:date-range [:date-range :start-date :end-date]}
@@ -327,7 +323,7 @@
                     (assoc-in (exact-match-id* request)  [1 :hx-swap-oob] true)])
                  :query-schema query-schema
                  :action-buttons (fn [request]
-                                   (let [[_ _ visible-in-float total-in-float ] (:page-results request)]
+                                   (let [[_ _ visible-in-float total-in-float] (:page-results request)]
                                      [(com/pill {:color :primary} " Visible in float "
                                                 (format "$%,.2f" visible-in-float))
                                       (com/pill {:color :secondary} " Total in float "
@@ -354,7 +350,7 @@
 
                                   (= (-> request :query-params :sort first :name) "Bank account")
                                   (-> entity :payment/bank-account :bank-account/name)
-  
+
                                   :else nil))
                  :title       (fn [r]
                                 (str
@@ -409,7 +405,7 @@
                                 :render        (fn [{:payment/keys [date]}]
                                                  (some-> date (atime/unparse-local atime/normal-date)))}
                                {:key           "amount"
-                               :sort-key "amount" 
+                                :sort-key "amount"
                                 :name          "Amount"
                                 :render        (fn [{:payment/keys [amount]}]
                                                  (some->> amount (format "$%.2f")))}
@@ -421,10 +417,10 @@
                                                                              (map :invoice-payment/invoice)
                                                                              (filter identity)
                                                                              (map (fn [invoice]
-                                                                                       {:link (hu/url (bidi/path-for ssr-routes/only-routes
-                                                                                                                     ::invoice-route/all-page)
-                                                                                                      {:exact-match-id (:db/id invoice)})
-                                                                                        :content (str "Inv. " (:invoice/invoice-number invoice))})))
+                                                                                    {:link (hu/url (bidi/path-for ssr-routes/only-routes
+                                                                                                                  ::invoice-route/all-page)
+                                                                                                   {:exact-match-id (:db/id invoice)})
+                                                                                     :content (str "Inv. " (:invoice/invoice-number invoice))})))
                                                                         (some-> p :transaction/_payment ((fn [t]
                                                                                                            [{:link (hu/url (bidi/path-for client-routes/routes
                                                                                                                                           :transactions)
@@ -434,8 +430,6 @@
 
 (def row* (partial helper/row* grid-page))
 
-
-
 (comment
   (mc/decode query-schema {"exact-match-id" "123"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
   (mc/decode query-schema {} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
@@ -445,7 +439,6 @@
   (mc/decode query-schema {"payment-type" "food"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
   (mc/decode query-schema {"vendor" "87"} (mt/transformer main-transformer mt/strip-extra-keys-transformer))
 
-
   (mc/decode query-schema {"start-date" #inst "2023-12-21T08:00:00.000-00:00"} (mt/transformer main-transformer mt/strip-extra-keys-transformer)))
 
 (defn delete [{check :entity :as request identity :identity}]
@@ -459,7 +452,7 @@
    #(assert-can-see-client identity (:db/id (:payment/client check))))
   (notify-if-locked (:db/id (:payment/client check))
                     (:payment/date check))
-  (let [ removing-payments (mapcat (fn [x]
+  (let [removing-payments (mapcat (fn [x]
                                     (let [invoice (:invoice-payment/invoice x)
                                           new-balance (+ (:invoice/outstanding-balance invoice)
                                                          (:invoice-payment/amount x))]
@@ -475,9 +468,9 @@
                          :payment/status :payment-status/voided}]
     (audit-transact (cond-> removing-payments
                       true (conj updated-payment)
-                      (:transaction/_payment check) (conj [:db/retract (:db/id (first (:transaction/_payment check))) 
+                      (:transaction/_payment check) (conj [:db/retract (:db/id (first (:transaction/_payment check)))
                                                            :transaction/payment
-                                                           (:db/id check)])) 
+                                                           (:db/id check)]))
                     identity)
 
     (html-response (row* (:identity request) updated-payment {:delete-after-settle? true :class "live-removed"
@@ -578,7 +571,6 @@
                                                 (assoc-in [:query-params :start] 0)
                                                 (assoc-in [:query-params :per-page]  250))))
 
-
               :else
               selected)
         updated-count (void-payments-internal ids (:identity request))]
@@ -591,7 +583,7 @@
 
 (defn wrap-status-from-source [handler]
   (fn [{:keys [matched-current-page-route] :as request}]
-    (let [ request (cond-> request
+    (let [request (cond-> request
                     (= ::route/cleared-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/cleared)
                     (= ::route/pending-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/pending)
                     (= ::route/voided-page matched-current-page-route) (assoc-in [:route-params :status] :payment-status/voided)
@@ -605,7 +597,7 @@
     ::route/pending-page              (-> (helper/page-route grid-page)
                                           (wrap-implied-route-param :status :payment-status/pending))
     ::route/voided-page              (-> (helper/page-route grid-page)
-                                          (wrap-implied-route-param :status :payment-status/voided))
+                                         (wrap-implied-route-param :status :payment-status/voided))
     ::route/all-page              (-> (helper/page-route grid-page)
                                       (wrap-implied-route-param :status nil))
 
@@ -618,7 +610,6 @@
     ::route/bulk-delete  (-> bulk-delete-dialog
                              (wrap-admin))
 
-
     ::route/table             (helper/table-route grid-page)}
    (fn [h]
      (-> h

src/clj/auto_ap/ssr/pos/sales_orders.clj

@@ -1,7 +1,7 @@
 (ns auto-ap.ssr.pos.sales-orders
   (:require
    [auto-ap.datomic
-    :refer [add-sorter-fields apply-pagination apply-sort-3 conn merge-query
+    :refer [add-sorter-fields apply-pagination apply-sort-3 merge-query
             pull-many query2]]
    [auto-ap.datomic.sales-orders :as d-sales]
    [auto-ap.query-params :as query-params :refer [wrap-copy-qp-pqp]]
@@ -17,7 +17,6 @@
    [auto-ap.time :as atime]
    [bidi.bidi :as bidi]
    [clj-time.coerce :as c]
-   [datomic.api :as dc]
    [malli.core :as mc]))
 
 (def query-schema (mc/schema
@@ -172,11 +171,8 @@
     charges))
 
 (defn fetch-page [request]
-  (let [db                                           (dc/db conn)
-        {ids-to-retrieve :ids matching-count :count} (fetch-ids db request)]
-
-    [(->> (hydrate-results ids-to-retrieve db request))
-     matching-count]))
+  (let [{:keys [rows count]} (d-sales/fetch-page-ssr request)]
+    [rows count]))
 
 
 (def grid-page
@@ -200,13 +196,13 @@
     :title          "Sales orders"
     :entity-name    "Sales orders"
     :route          :pos-sales-table
-    :action-buttons (fn [request]
-                      (let [{:keys [total tax]} (d-sales/summarize-orders (:ids (fetch-ids (dc/db conn) request)))]
-                        (when (and total tax)
-                          [(com/pill {:color :primary}
-                                     (format "Total $%.2f" total))
-                           (com/pill {:color :secondary}
-                                     (format "Tax $%.2f" tax))])))
+      :action-buttons (fn [request]
+                        (let [{:keys [total tax]} (d-sales/summarize-page-ssr request)]
+                          (when (and total tax)
+                            [(com/pill {:color :primary}
+                                       (format "Total $%.2f" total))
+                             (com/pill {:color :secondary}
+                                       (format "Tax $%.2f" tax))])))
     :row-buttons    (fn [_ e]
                       (when (:sales-order/reference-link e)
                         [(com/a-icon-button {:href (:sales-order/reference-link e)}

src/clj/auto_ap/storage/parquet.clj

@@ -3,7 +3,9 @@
             [amazonica.aws.s3 :as s3]
             [clojure.java.io :as io]
             [clojure.string :as str]
-            [clojure.data.json :as json])
+            [clojure.data.json :as json]
+            [clojure.core.cache :as cache]
+            [com.brunobonacci.mulog :as mu])
   (:import (java.sql DriverManager)
            (java.time LocalDate)))
 
@@ -14,7 +16,10 @@
   (str "s3://" *bucket* "/" filename))
 
 (defn parquet-key [entity-type date-str]
-  (str parquet-prefix "/" entity-type "/" date-str ".parquet"))
+  (let [month-str (if (= (count date-str) 10)
+                    (subs date-str 0 7)
+                    date-str)]
+    (str parquet-prefix "/" entity-type "/" month-str ".parquet")))
 
 (def db (atom nil))
 
@@ -22,9 +27,16 @@
   (let [conn (DriverManager/getConnection "jdbc:duckdb:")
         stmt (.createStatement conn)]
     (.execute stmt "INSTALL httpfs; LOAD httpfs;")
+    (when-let [key (:aws-access-key-id env)]
+      (.execute stmt (str "SET s3_access_key_id='" key "'"))
+      (.execute stmt (str "SET s3_secret_access_key='" (:aws-secret-access-key env) "'"))
+      (.execute stmt (str "SET s3_region='" (or (:aws-region env) "us-east-1") "'")))
+    (.execute stmt "PRAGMA enable_object_cache")
+    (.execute stmt "SET temp_directory='/tmp/duckdb-temp'")
+    (.execute stmt "SET memory_limit='2GB'")
     (.close stmt)
     (.addShutdownHook (Runtime/getRuntime)
-                      (Thread. #(fn [])))
+                      (Thread. #(when-let [c @db] (.close ^java.sql.Connection c))))
     (reset! db conn)))
 
 (defn disconnect! []
@@ -49,34 +61,52 @@
       (.execute stmt sql)
       nil)))
 
+(defn- sql-snippet [sql] (subs sql 0 (min (count sql) 500)))
+
 (defn query-scalar [sql]
-  (with-duckdb
-    (let [stmt (.createStatement conn)
-          rs   (.executeQuery stmt sql)]
-      (when (.next rs)
-        (.getObject rs 1)))))
+  (mu/trace ::query-scalar
+            [:sql (sql-snippet sql)]
+            (with-duckdb
+              (let [stmt (.createStatement conn)
+                    rs   (.executeQuery stmt sql)]
+                (when (.next rs)
+                  (.getObject rs 1))))))
+
+(def ^:private count-cache
+  (atom (-> (cache/ttl-cache-factory {} :ttl 1800000)
+            (cache/lru-cache-factory :threshold 256))))
+
+(defn- cached-count [sql]
+  (if-let [v (find @count-cache sql)]
+    (do (mu/log ::count-cache :hit true :sql (sql-snippet sql)) (val v))
+    (do (mu/log ::count-cache :hit false :sql (sql-snippet sql))
+        (let [result (query-scalar sql)]
+          (swap! count-cache assoc sql result)
+          result))))
 
 (defn query-rows [sql]
-  (with-duckdb
-    (let [stmt      (.createStatement conn)
-          rs        (.executeQuery stmt sql)
-          meta      (.getMetaData rs)
-          col-count (.getColumnCount meta)
-          cols      (vec (for [i (range 1 (inc col-count))]
-                           (keyword (.getColumnLabel meta i))))]
-      (loop [rows []]
-        (if (.next rs)
-          (recur (conj rows
-                       (zipmap cols
-                               (vec (for [i (range 1 (inc col-count))]
-                                      (.getObject rs i))))))
-          rows)))))
+  (mu/trace ::query-rows
+            [:sql (sql-snippet sql)]
+            (with-duckdb
+              (let [stmt      (.createStatement conn)
+                    rs        (.executeQuery stmt sql)
+                    meta      (.getMetaData rs)
+                    col-count (.getColumnCount meta)
+                    cols      (vec (for [i (range 1 (inc col-count))]
+                                     (keyword (.getColumnLabel meta i))))]
+                (loop [rows []]
+                  (if (.next rs)
+                    (recur (conj rows
+                                 (zipmap cols
+                                         (vec (for [i (range 1 (inc col-count))]
+                                                (.getObject rs i))))))
+                    rows))))))
 
 (defn execute-to-parquet! [sql ^String parquet-path]
   (with-duckdb
     (let [stmt (.createStatement conn)]
       (.execute stmt
-                (format "COPY (%s) TO '%s' (FORMAT PARQUET, OVERWRITE_OR_IGNORE)"
+                (format "COPY (%s) TO '%s' (FORMAT PARQUET, OVERWRITE_OR_IGNORE, ROW_GROUP_SIZE 10000, COMPRESSION 'zstd')"
                         sql parquet-path))
       (io/file parquet-path))))
 
@@ -107,9 +137,9 @@
                               (str entity-type ".jsonl"))]
         (io/make-parents wal-file)
         (with-open [w (io/writer wal-file :append true)]
-           (.write w ^String (json/write-str {:seq-no seq-no
-                                               :record record}))
-           (.write w (int \newline))))
+          (.write w ^String (json/write-str {:seq-no seq-no
+                                             :record record}))
+          (.write w (int \newline))))
       (catch Exception e
         (println "[parquet/wal]" (.getMessage e))))
     entry))
@@ -124,43 +154,60 @@
   (->> @*buffers*
        vals (mapcat identity) count))
 
-(defn flush-to-parquet! [entity-type]
-  "Flush buffered records for entity-type to parquet + S3."
+(defn flush-to-parquet! [entity-type date-str]
+  "Flush buffered records for entity-type to monthly parquet + S3.
+   Reads existing monthly file (if any), merges with new records, and uploads.
+   Uses temp table to ensure ROW_GROUP_SIZE is respected (DuckDB ignores it
+   when reading directly from S3 via COPY)."
   (let [records (get @*buffers* entity-type [])]
     (if (empty? records)
       {:status :no-records}
-      (let [date-str   (.toString (LocalDate/now))
+      (let [date-str     (or date-str (.toString (LocalDate/now)))
+            s3-key       (parquet-key entity-type date-str)
+            s3-url       (s3-location s3-key)
             jsonl-file   (io/file "/tmp"
                                   (str entity-type "-" date-str ".jsonl"))
             parquet-file (io/file "/tmp"
                                   (str entity-type "-" date-str ".parquet"))
-            s3-key       (parquet-key entity-type date-str)]
+            tbl          (format "\"_flush_%s_%s\""
+                                 (clojure.string/replace entity-type "-" "_")
+                                 (subs date-str 0 7))]
         (try
           (with-open [w (io/writer jsonl-file :append true)]
             (doseq [r records]
-               (.write w ^String (json/write-str (dissoc r :_seq-no)))
-               (.write w (int \newline))))
-          (execute-to-parquet!
-           (format "SELECT * FROM read_json_auto('%s')"
-                   (.getAbsolutePath jsonl-file))
-           (.getAbsolutePath parquet-file))
-          (upload-parquet! parquet-file s3-key)
-          (clear-buffer! entity-type)
-          (.delete ^java.io.File jsonl-file)
-          (.delete ^java.io.File parquet-file)
-          {:key s3-key :status :ok}
+              (.write w ^String (json/write-str (dissoc r :_seq-no)))
+              (.write w (int \newline))))
+          (let [existing-sql (format
+                              "SELECT * FROM read_parquet('%s', union_by_name=true)"
+                              s3-url)
+                new-sql      (format
+                              "SELECT * FROM read_json_auto('%s')"
+                              (.getAbsolutePath jsonl-file))]
+            (execute! (format "CREATE OR REPLACE TABLE %s AS SELECT * FROM (%s UNION ALL %s) ORDER BY \"client-code\", date"
+                              tbl existing-sql new-sql))
+            (execute! (format "COPY (SELECT * FROM %s) TO '%s' (FORMAT PARQUET, OVERWRITE_OR_IGNORE, ROW_GROUP_SIZE 10000, COMPRESSION 'zstd')"
+                              tbl (.getAbsolutePath parquet-file)))
+            (execute! (format "DROP TABLE IF EXISTS %s" tbl))
+            (upload-parquet! parquet-file s3-key)
+            (clear-buffer! entity-type)
+            (.delete ^java.io.File jsonl-file)
+            (.delete ^java.io.File parquet-file)
+            {:key s3-key :status :ok})
           (catch Exception e
-            (throw (ex-info "Flush failed" {:entity-type entity-type}
-                            :error (.getMessage e)))))))))
+            (execute! (format "DROP TABLE IF EXISTS %s" tbl))
+            (throw (ex-info "Flush failed"
+                            {:entity-type entity-type
+                             :error (.getMessage e)}))))))))
 
 (defn flush-by-date! []
   "Flush all entity types for today."
   (let [etypes    ["sales-order" "charge"
                    "line-item" "sales-refund"]
+        today     (.toString (LocalDate/now))
         flushed   (into #{}
                         (keep (fn [et]
                                 (let [{:keys [status]}
-                                      (flush-to-parquet! et)]
+                                      (flush-to-parquet! et today)]
                                   (when (= status :ok)
                                     et))))
                         etypes)]
@@ -188,7 +235,8 @@
                               (let [f (io/file
                                        (wal-dir)
                                        (str et ".jsonl"))]
-                                [et (slurp f)])))
+                                (when (.exists f)
+                                  [et (slurp f)]))))
                       etypes))]
     (swap! *buffers* merge loaded)))
 
@@ -203,76 +251,163 @@
 (defn date-seq [start end]
   "Seq of YYYY-MM-DD strings between start and end inclusive."
   (let [sd   (LocalDate/parse start)
-        ed   (LocalDate/parse end)
-        days (int (Math/abs
-                   (- (.toEpochDay sd)
-                      (.toEpochDay ed))))]
-    (for [i (range 0 (inc days))]
-      (.toString (.plusDays sd i)))))
+        ed   (LocalDate/parse end)]
+    (when (.isAfter sd ed)
+      (throw (ex-info "date-seq: start must be <= end" {:start start :end end})))
+    (let [days (int (- (.toEpochDay ed)
+                       (.toEpochDay sd)))]
+      (for [i (range 0 (inc days))]
+        (.toString (.plusDays sd i))))))
 
 (defn today []
   (.toString (LocalDate/now)))
 
+(def ^:private mm-dd-yyyy (java.time.format.DateTimeFormatter/ofPattern "MM/dd/yyyy"))
+
+(defn- normalize-date-str [s]
+  (when s
+    (if (re-find #"^\d{2}/\d{2}/\d{4}" s)
+      (.toString (LocalDate/parse s mm-dd-yyyy))
+      (if (> (count s) 10) (subs s 0 10) s))))
+
+(defn month-seq [start-date end-date]
+  "Seq of YYYY-MM strings between start-date and end-date inclusive."
+  (let [sd (LocalDate/parse (normalize-date-str start-date))
+        ed (LocalDate/parse (normalize-date-str end-date))]
+    (loop [months [] cur sd]
+      (if (.isAfter cur ed)
+        months
+        (recur (conj months (.toString (.withDayOfMonth cur 1)))
+               (.plusMonths cur 1))))))
+
+(defn- parquet-glob [entity-type start-date end-date]
+  "Build explicit file list for the date range using monthly partitions.
+   Monthly files mean only 1-3 files for typical queries, 12 for a full year."
+  (let [prefix (format "s3://%s/sales-details/%s/" *bucket* entity-type)]
+    (vec
+     (map (fn [m]
+            (format "'%s%s.parquet'" prefix m))
+          (month-seq start-date end-date)))))
+
 (defn parquet-query [entity-type start-date end-date]
-  "Build SQL to read all parquet files in date range.
-   Returns map with :sql and :count-sql keys."
-  (let [date-strs (date-seq start-date end-date)
-        urls      (vec
-                   (map #(format "'s3://%%s/sales-details/%%s/%%s.parquet'"
-                                 *bucket* entity-type %)
-                        date-strs))
-        sql       (str "SELECT * FROM read_parquet(["
-                       (str/join ", " urls)
-                       "])")]
+  "Build SQL to read monthly parquet files in date range.
+   Uses explicit file list (monthly = few files) + WHERE date filter.
+   Normalizes date formats (handles MM/dd/yyyy from UI)."
+  (let [sd    (normalize-date-str start-date)
+        ed    (normalize-date-str end-date)
+        files (parquet-glob entity-type sd ed)
+        base  (format "SELECT * FROM read_parquet([%s], union_by_name=true)"
+                      (str/join ", " files))
+        sql   (format "%s WHERE date >= '%s' AND date <= '%s'"
+                      base sd ed)]
     {:sql       sql
      :count-sql (format "SELECT COUNT(*) FROM (%s) t" sql)}))
 
-(defn- build-where-clause [opts field-pairs]
-  "Build SQL WHERE clause from opts map.
-   fields-with-keys is vector of [:field-key :env-var-name]."
-  (let [clauses (keep
-                 (fn [[key env]]
-                   (let [v (get opts key)]
-                     (when v
-                       (str env " = '" v "'"))))
-                 field-pairs)]
-    (when (seq clauses)
-      (str " WHERE " (str/join " AND " clauses)))))
+(defn- like-clause [col v]
+  (str "\"" col "\" LIKE '%" v "%'"))
+
+(defn- build-sales-orders-where [opts]
+  (let [eq-clauses (keep
+                    (fn [[key col]]
+                      (let [v (get opts key)]
+                        (when v
+                          (str "\"" col "\" = '" v "'"))))
+                    [[:client "client-code"]
+                     [:vendor "vendor"]
+                     [:location "location"]])
+        in-clauses (keep
+                    (fn [[key col]]
+                      (let [vs (get opts key)]
+                        (when (seq vs)
+                          (str "\"" col "\" IN ("
+                               (str/join ", " (map #(str "'" % "'") vs))
+                               ")"))))
+                    [[:client-codes "client-code"]])
+        like-clauses (keep
+                      (fn [[key col]]
+                        (let [v (get opts key)]
+                          (when v
+                            (like-clause col v))))
+                      [[:payment-method "payment-methods"]
+                       [:processor "processors"]
+                       [:category "categories"]])
+        range-clauses (keep
+                       (fn [[key col op]]
+                         (let [v (get opts key)]
+                           (when v
+                             (str "\"" col "\" " op " " v))))
+                       [[:total-gte "total" ">="]
+                        [:total-lte "total" "<="]])
+        all-clauses (concat eq-clauses in-clauses like-clauses range-clauses)]
+    (when (seq all-clauses)
+      (str/join " AND " all-clauses))))
 
 (defn get-sales-orders
   ([start-date end-date]
    (get-sales-orders start-date end-date {}))
   ([start-date end-date opts]
-   (let [q               (parquet-query "sales-order"
-                                         start-date end-date)
-          base-sql         (:sql q)
-          count-sql        (:count-sql q)
-          sort             (get opts :sort "date")
-          order            (get opts :order "DESC")
-          limit            (get opts :limit)
-          offset           (get opts :offset)
-          where-str        (build-where-clause
+   (mu/trace ::get-sales-orders
+             [:start-date start-date :end-date end-date :opts opts]
+             (try
+               (let [q               (parquet-query "sales-order" start-date end-date)
+                     base-sql         (:sql q)
+                     has-where?       (str/includes? base-sql " WHERE ")
+                     sort             (get opts :sort "date")
+                     order            (get opts :order "DESC")
+                     limit            (get opts :limit)
+                     offset           (get opts :offset)
+                     extra-clauses    (build-sales-orders-where opts)
+                     full-sql         (if extra-clauses
+                                        (str base-sql (if has-where? " AND " " WHERE ") extra-clauses)
+                                        base-sql)
+                     data-sql         (cond-> full-sql
+                                        sort  (str " ORDER BY " sort " " (name order))
+                                        limit (str " LIMIT " limit)
+                                        offset (str " OFFSET " offset))
+                     count-sql        (format "SELECT COUNT(*) FROM (%s) t" full-sql)]
+                 (mu/log ::get-sales-orders :data-sql data-sql :count-sql count-sql)
+                 (let [cnt  (cached-count count-sql)
+                       rows (query-rows data-sql)]
+                   {:rows  rows
+                    :count (or (int cnt) 0)}))
+               (catch Exception e
+                 (mu/log ::get-sales-orders :error e :start-date start-date :end-date end-date :opts opts)
+                 (throw e))))))
 
-                            opts
-                            [[:client "external_id.client"]
-                             [:vendor "external_id.vendor"]
-                             [:location "location"]])
-          full-sql         (if where-str
-                             (str base-sql where-str)
-                             base-sql)
-          result           (cond-> full-sql
-                             sort      (str " ORDER BY " sort
-                                            " " (name order))
-                             limit     (str " LIMIT " limit)
-                             offset    (str " OFFSET " offset))
-          full-count       (if where-str
-                             (str count-sql where-str)
-                             count-sql)]
-      {:rows   (query-rows result)
-       :count  (or
-                (int
-                 (query-scalar
-                  full-count)) 0)})))
+(def ^:private summary-cache
+  (atom (-> (cache/ttl-cache-factory {} :ttl 1800000)
+            (cache/lru-cache-factory :threshold 256))))
+
+(defn- cached-summary [sql]
+  (if-let [v (find @summary-cache sql)]
+    (do (mu/log ::summary-cache :hit true :sql (sql-snippet sql)) v)
+    (do (mu/log ::summary-cache :hit false :sql (sql-snippet sql))
+        (let [result (let [row (first (query-rows sql))]
+                       {:total (or (:total row) 0.0)
+                        :tax   (or (:tax row) 0.0)})]
+          (swap! summary-cache assoc sql result)
+          result))))
+
+(defn get-sales-orders-summary
+  ([start-date end-date]
+   (get-sales-orders-summary start-date end-date {}))
+  ([start-date end-date opts]
+   (mu/trace ::get-sales-orders-summary
+             [:start-date start-date :end-date end-date :opts opts]
+             (try
+               (let [q              (parquet-query "sales-order" start-date end-date)
+                     base-sql       (:sql q)
+                     has-where?     (str/includes? base-sql " WHERE ")
+                     extra-clauses  (build-sales-orders-where opts)
+                     full-sql       (if extra-clauses
+                                      (str base-sql (if has-where? " AND " " WHERE ") extra-clauses)
+                                      base-sql)
+                     sum-sql        (format "SELECT COALESCE(SUM(total), 0) as total, COALESCE(SUM(tax), 0) as tax FROM (%s) t" full-sql)]
+                 (mu/log ::get-sales-orders-summary :sum-sql sum-sql)
+                 (cached-summary sum-sql))
+               (catch Exception e
+                 (mu/log ::get-sales-orders-summary :error e :start-date start-date :end-date end-date :opts opts)
+                 (throw e))))))
 
 (defn query-deduped [entity-type start-date end-date]
   "Query records deduplicated by external-id (latest _seq_no wins)."
@@ -280,7 +415,7 @@
     (query-rows
      (str (:sql q)
           " QUALIFY ROW_NUMBER() OVER"
-          "   (PARTITION BY sales_order.external_id"
+          "   (PARTITION BY \"external-id\""
           " ORDER BY _seq_no DESC) = 1"))))
 
 (defn query-by-entity-id [entity-type external-id

src/clj/auto_ap/storage/sales_summaries.clj

@@ -15,12 +15,12 @@
       0.0)))
 
 (defn- pq-files [entity-type start-date end-date]
-  "Vector of S3 parquet file paths for date range."
-  (let [dates (p/date-seq start-date end-date)]
+  "Vector of S3 parquet file paths for date range (monthly partitions)."
+  (let [months (p/month-seq start-date end-date)]
     (vec
      (map #(str "'s3://" p/*bucket*
                 "/sales-details/" entity-type "/"
-                % ".parquet") dates))))
+                % ".parquet") months))))
 
 (defn sum-payments-by-type [client-id start-date end-date]
   "Return {processor-key -> {type-name-string -> total-double}}."
@@ -55,7 +55,7 @@
                                          tname
                                          (+ (get b tname 0.0) total)))))))
                   {}
-                   rows)))
+                  rows)))
       (catch Exception e
         (println "[sales-summaries]" (.getMessage e))
         {}))))

test/clj/auto_ap/parse/templates_test.clj

@@ -51,3 +51,22 @@
       (is (= "720.33" (:total (nth results 1))))
       (is (= "853.16" (:total (nth results 2))))
       (is (= "1066.60" (:total (nth results 3)))))))
+
+(deftest parse-bonanza-produce-invoice-03882095
+  (testing "Should parse Bonanza Produce invoice 03882095 with customer identifier including address"
+    (let [pdf-file (io/file "dev-resources/INVOICE - 03882095.pdf")
+          pdf-text (:out (clojure.java.shell/sh "pdftotext" "-layout" (str pdf-file) "-"))
+          results (sut/parse pdf-text)
+          result (first results)]
+      (is (some? results) "parse should return a result")
+      (is (some? result) "Template should match and return a result")
+      (when result
+        (is (= "Bonanza Produce" (:vendor-code result)))
+        (is (= "03882095" (:invoice-number result)))
+        (let [d (:date result)]
+          (is (= 2026 (time/year d)))
+          (is (= 1 (time/month d)))
+          (is (= 23 (time/day d))))
+        (is (= "NICK THE GREEK" (:customer-identifier result)))
+        (is (= "600 VISTA WAY" (str/trim (:account-number result))))
+        (is (= "946.24" (:total result)))))))

test/clj/auto_ap/storage/perf_test.clj

@@ -108,5 +108,6 @@
     (finally
       (p/disconnect!))))
 
-(run-perf-tests)
-(println "\n=== Done ===")
+(comment
+  (run-perf-tests)
+  (println "\n=== Done ==="))