notid/integreat
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Merge pull request 'feat(transactions): port manual bank-transaction import to SSR' (#9) from integreat-add-transaction-manual into staging

Browse Source 
Reviewed-on: #9
This commit was merged in pull request #9.
This commit is contained in:
notid
2026-06-01 21:06:52 -07:00
parent b6649a3d1d 9cc3418b1b
commit 569e52d1c1
8 changed files with 1062 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
docs/ideation/2026-06-01-manual-transaction-import-ssr-ideation.md Normal file
View File

@@ -0,0 +1,176 @@
---
date: 2026-06-01
topic: manual-transaction-import-ssr
focus: Port the master-branch "manual import transactions" feature to the SSR/alpine/htmx stack, modeled on the SSR ledger import, preserving all validations, starting from a failing e2e test, with minimal core-component change.
mode: repo-grounded
---
# Ideation: Porting Manual Bank-Transaction Import to SSR
## Grounding Context (Codebase)
Three reference points were read in full:
**1. The master feature (what we must reproduce).**
- UI: `src/cljs/auto_ap/views/pages/transactions/manual.cljs` — a re-frame modal titled "Import Transactions" with a single `<textarea>` ("Yodlee manual import table"). User pastes tab-separated Yodlee data, clicks "Import". POSTs `(:data)` as EDN to `/api/transactions/batch-upload`.
- Route/handler: `src/clj/auto_ap/routes/invoices.clj:241` `batch-upload-transactions` → `assert-admin`, then `manual/import-batch (manual/tabulate-data data)`.
- Parsing: `src/clj/auto_ap/import/manual.clj` — `tabulate-data` reads CSV with `\tab` separator, drops the header row, and maps **fixed positional columns**: `[:status :raw-date :description-original :high-level-category nil nil :amount nil nil nil nil nil :bank-account-code :client-code]`.
- Per-row mapping/validation: `manual->transaction` uses `import.manual.common/assoc-or-error` to accumulate errors for: **client lookup** (by bank-account-code → client), **bank-account lookup** (by code), **date parse** (`parse-date`, requires `MM/dd/yyyy`), **amount parse** (`parse-amount`).
- Engine: `import.manual/import-batch` builds lookups, calls `t/start-import-batch :import-source/manual`, applies `t/apply-synthetic-ids` (dedupe key), imports only rows with no `:errors`, returns stats `{:import-batch/imported ... :failed-validation N :sample-error "..."}`.
- Deeper validations live in `src/clj/auto_ap/import/transactions.clj` `categorize-transaction`: status must be `"POSTED"`, transaction date must be after `bank-account/start-date` and after `client/locked-until`, duplicate detection via `:transaction/id` (extant cache), missing client/bank-account/id → `:error`, plus suppression. Synthetic-id (`apply-synthetic-ids`/`synthetic-key`) gives idempotent re-import.
**2. The SSR ledger import (the pattern to emulate).**
- `src/clj/auto_ap/ssr/ledger.clj` implements a **dedicated two-stage page**, not a modal:
- `external-import-text-form*` (~L246): alpine `x-data {clipboard}`, a hidden `text-area` bound `x-model`, an `hx-post` to `::route/external-import-parse` triggered on a `"pasted"` event; a "Load from clipboard" button reads `navigator.clipboard`.
- `external-import-parse` (~L373): re-renders `external-import-form*` with `:just-parsed? true`.
- `external-import-table-form*` (~L117): renders parsed rows into an **editable** `com/data-grid-card` — each cell is a `fc/with-field` text/money input (form-cursor round-trips values + errors); per-row error/warning badge with tooltip; "Import" `hx-post`s to `::route/external-import-import`.
- Validation: malli `parse-form-schema` (~L341) with `:decode/string tsv->import-data` + `:decode/arbitrary` (vector→map) enforces shape (min-length, `clj-date-schema`, `money`, location max 2). Business validation in `add-errors`/`table->entries` (~L380–504) accumulates `[message status]` pairs (`:error`/`:warn`) at entry and line-item level; `flatten-errors` maps them to form-cursor field paths `[[:table idx] message status]`.
- `import-ledger` (~L554): splits good/ignored(warn-only)/bad entries; `throw+` `:field-validation` with `:form-errors` if any bad; upserts hidden vendors; retracts+re-inserts good entries (idempotent via `:journal-entry/external-id`); touches solr; returns `{:successful :ignored :form-errors}`. `external-import-import` wraps it in an `html-response` with an `hx-trigger` notification.
**3. THE KEY DISCOVERY — scaffolding already exists, handlers do not.**
- `src/cljc/auto_ap/routes/transactions.cljc` **already declares** the route names, mirroring ledger exactly:
```
"/external-new" ::external-page
"/external-import-new" {"" ::external-import-page
"/parse" ::external-import-parse
"/import" ::external-import-import}
```
- But `src/clj/auto_ap/ssr/transaction.clj` `key->handler` (L101) wires **only** `page/table/csv/bank-account-filter/bulk-delete` (+ `edit/` + `bulk-code/`). **No handler exists for any `external-import-*` route.** The aside nav (`ssr/components/aside.clj`) wires the *ledger* import nav (`::ledger-routes/external-import-page`) but there is no transaction-import nav entry. So the routes are declared dead ends; the gap is handlers + UI + validation + nav + tests.
**Test conventions.** `test/clj/auto_ap/ssr/ledger_test.clj` is the template: pure-fn tests (`tsv->import-data`, `trim-header`, `line->id`, `flatten-errors`), validation tests (`add-errors` per error type), tx-building tests (`entry->tx`), and end-to-end import tests (`import-ledger` against Datomic via `wrap-setup` + `setup-test-data` + `admin-token`). `test/clj/auto_ap/test-server.clj` is a Playwright/browser e2e harness with `wrap-test-auth` and seeded `setup-test-data`. Existing engine unit tests: `test/clj/auto_ap/import/transactions_test.clj` already covers `categorize-transaction`.
## Topic Axes
- **Input/paste fidelity** — keeping the exact Yodlee positional-column paste payload (req #1)
- **UI flow & surface** — modal vs. dedicated two-stage paste→review→import page (req #2)
- **Validation architecture** — where shape vs. business rules live, and how errors surface (req #3)
- **Core/backend reuse** — how much of `import.transactions` / `import.manual` is reused vs. reimplemented (req #5)
- **Test strategy** — failing-first e2e, then unit coverage (req #4)
## The Central Design Fork
Requirements #1 ("paste the exact same content") and #2 ("follow ledger patterns") pull in slightly different directions, but resolve cleanly:
- **Input format stays identical** — the user still copies the same Yodlee table and pastes the same tab-separated positional columns. We reuse `manual/columns` + `manual/tabulate-data` for the *paste shape*; we do **not** switch to ledger's named-header columns.
- **Everything downstream follows ledger** — dedicated page, clipboard paste, editable review grid, per-row error/warning badges, re-validate loop, notification on import.
The one decision genuinely worth the user's input is **how much of the ledger UX to adopt**: the full editable review grid (idea #5, higher value, more work) vs. a lighter paste→validate→summary page closer to master (still SSR, less scope). Both are presented below as ranked survivors; this is the seed question for brainstorming.
## Ranked Ideas
### 1. Wire the pre-scaffolded routes into a dedicated two-stage SSR import page
**Description:** Implement `external-import-page` / `external-import-parse` / `external-import-import` handlers in a new `src/clj/auto_ap/ssr/transaction/import.clj`, add them to `ssr/transaction.clj` `key->handler` (with the same middleware stack: `wrap-must {:activity :view :subject :transaction}`, `wrap-client-redirect-unauthenticated`, etc.), and add a transaction "Import" nav entry in `aside.clj` parallel to the ledger one. The page structure mirrors `ledger/external-import-page`: breadcrumb → clipboard script → paste form → review table.
**Axis:** UI flow & surface
**Basis:** `direct:` `routes/transactions.cljc` already declares `::external-import-page/parse/import` but `ssr/transaction.clj:101 key->handler` wires none of them; `ssr/ledger.clj:686 key->handler` shows the exact wiring shape to copy.
**Rationale:** The routing contract already exists and is unused — the cheapest, lowest-risk foundation, and it's what makes req #2 ("like the ledger import") literally true at the routing/page level.
**Downsides:** Adds a new namespace; needs a nav entry and middleware parity to avoid auth gaps.
**Confidence:** 95%
**Complexity:** Low
**Status:** Unexplored
### 2. Preserve the exact Yodlee positional paste format (req #1)
**Description:** Reuse `import.manual/columns` + `tabulate-data` (or a malli `:decode/string` wrapper around them) for parsing the pasted TSV, instead of ledger's named-header `tsv->import-data`. Keep the same column positions (`:status :raw-date :description-original ... :amount ... :bank-account-code :client-code`) so users paste identical content.
**Axis:** Input/paste fidelity
**Basis:** `direct:` Requirement #1 ("Paste the exact same type of content as was in the master branch version") + `import/manual.clj:10` fixed `columns` vector; the master textarea label is literally "Yodlee manual import table".
**Rationale:** Users' upstream copy/paste habit (and the Yodlee export shape) is the contract that must not break; the ledger named-header approach would silently change what content is valid.
**Downsides:** Positional columns are brittle if Yodlee changes column order — but that's already the status quo, not a regression.
**Confidence:** 90%
**Complexity:** Low
**Status:** Unexplored
### 3. Reuse the `import.transactions` engine as the backend (req #5)
**Description:** The import handler maps parsed rows → `:transaction/*` maps via the existing `manual->transaction` shape, then drives `import.transactions/start-import-batch :import-source/manual` + `import-transaction!` + `finish!` + `get-stats` — exactly as `import.manual/import-batch` does today. The SSR layer is presentation + pre-validation only; the categorization, rule-matching, payment/deposit clearing, synthetic-id dedupe, and audit-transact paths are untouched.
**Axis:** Core/backend reuse
**Basis:** `direct:` Requirement #5 ("Minimally, if at all, change any core components") + `import/manual.clj:32 import-batch` already encapsulates the whole engine call; `import/transactions.clj` is covered by `transactions_test.clj`.
**Rationale:** This is the battle-tested core. Re-implementing it ledger-style would risk dropping validations (`categorize-transaction`) and duplicate a large, audited code path. Wrapping it keeps core change near zero.
**Downsides:** `import-batch` returns summary stats, not per-row form-cursor errors — so the pre-validation layer (idea #4) must surface row errors *before* the engine runs.
**Confidence:** 90%
**Complexity:** Medium
**Status:** Unexplored
### 4. Two-tier validation: malli shape-parse + a transaction `add-errors` business layer
**Description:** Mirror ledger's split. Tier 1: a malli `parse-form-schema` decodes the pasted TSV and coerces/validates *shape* (date parses as `MM/dd/yyyy`, amount parses, required fields present) — reusing `manual.common/parse-date`/`parse-amount` as `:decode`/predicate fns. Tier 2: a transaction-specific `add-errors`/`table->entries` adds *business* errors/warnings by reusing the predicates already encoded in `categorize-transaction`: client-not-found (`:error`), bank-account-not-found (`:error`), date before `bank-account/start-date` (`:warn`/`:not-ready`), client locked-until (`:warn`), status not `POSTED` (`:not-ready`), already-imported/extant (`:warn`). Errors are `[message status]` pairs surfaced via `flatten-errors` → form-cursor field paths.
**Axis:** Validation architecture
**Basis:** `direct:` Requirement #3 ("every validation maintained… but doesn't have to follow the same structure — make it like the ledger import"). Maps master validations (`manual.clj:23-30`, `transactions.clj:191-225 categorize-transaction`) onto ledger's `add-errors`/`flatten-errors` shape (`ledger.clj:380-519`).
**Rationale:** Gives the ledger-style inline error UX while guaranteeing 1:1 validation parity — each master check becomes an explicit `add-errors` clause, which is also directly unit-testable (one test per error type, like `ledger_test/add-errors-test`).
**Downsides:** Some checks (extant/duplicate, start-date, locked-until) need a DB read at validation time that master does lazily inside the engine — must decide whether to pre-check or let the engine's stats report them. Risk of double-validation drift if engine and pre-validator disagree.
**Confidence:** 80%
**Complexity:** High
**Status:** Unexplored
### 5. Editable review grid with per-row error/warning badges and a re-validate loop
**Description:** After paste+parse, render rows into a `com/data-grid-card` where each field (date, amount, description, bank-account-code, client-code) is an editable `fc/with-field` input, with `com/validated-field` error display and a per-row alert badge+tooltip — exactly like `ledger/external-import-table-form*`. The user can fix a wrong client/bank-account code or date inline and re-submit; only clean rows import, warn-only rows are skipped, error rows block. A "Show table" toggle keeps the default view compact.
**Axis:** UI flow & surface / validation surfacing
**Basis:** `direct:` Requirement #2 ("follow slightly better design patterns, like how the ledger import works"). `ledger.clj:117-244` is the editable-grid implementation to copy; master has no inline correction at all (fire-and-forget modal + summary stats).
**Rationale:** This is the concrete UX upgrade over master and the main reason to model on ledger — turning "paste, pray, read a stats blob" into "paste, see exactly which rows are wrong and why, fix them, import."
**Downsides:** Highest-effort idea; form-cursor round-tripping of an editable grid is the trickiest part of the ledger code. If scope must shrink, a read-only review table + summary (lighter survivor) is the fallback.
**Confidence:** 75%
**Complexity:** High
**Status:** Unexplored
### 6. Preserve idempotent re-import via synthetic-id duplicate detection, surfaced as "already imported"
**Description:** Keep `apply-synthetic-ids`/`synthetic-key` so re-pasting the same export is idempotent (the engine categorizes extant rows as `:extant` and skips them). Surface this in the review grid as a `:warn`-level "already imported" badge rather than silently dropping it, so the user understands why a row didn't import.
**Axis:** Validation architecture
**Basis:** `direct:` `import/transactions.clj:405-421 apply-synthetic-ids` + `categorize-transaction:192-225` extant handling. This is an existing master behavior that req #3 requires us to maintain.
**Rationale:** Duplicate-safety is an easy validation to lose in a port; making it visible (vs. master's opaque stats) is a small, high-trust UX win that costs almost nothing on top of idea #5.
**Downsides:** Requires a DB read of existing `:transaction/id`s at validation time (or reading it back from engine stats post-import).
**Confidence:** 80%
**Complexity:** Low
**Status:** Unexplored
### 7. Start with a failing Playwright e2e, then backfill ledger_test-style unit coverage (req #4)
**Description:** First commit: a failing e2e (against `test-server`) that dev-logs in, navigates dashboard → transactions → Import, pastes a known-good Yodlee TSV into the paste box, asserts parsed rows render, clicks Import, and asserts the transactions appear / a "N imported" notification fires. It fails initially (no handler/nav). Then make it pass incrementally: route+page (idea #1) → parse (#2/#4 tier 1) → review grid (#5) → import via engine (#3) → business validation (#4 tier 2). Backstop with unit tests mirroring `ledger_test.clj`: `tabulate-data`/parse, each `add-errors` validation clause, and an end-to-end `import` test against Datomic. Reuse the existing `transactions_test.clj` `categorize-transaction` coverage as the validation-parity oracle.
**Axis:** Test strategy
**Basis:** `direct:` Requirement #4 ("Write detailed acceptance criteria, and start with a failing e2e test, making it pass over time"). `test-server.clj` already provides the browser harness + test auth; `ledger_test.clj` provides the unit-test template.
**Rationale:** A red e2e pins down the acceptance contract before any handler exists and gives an unambiguous "done" signal; the unit layer locks in validation parity clause-by-clause so req #3 can't silently regress.
**Downsides:** e2e clipboard paste may need a direct `type`-into-textarea path (or a test seam) since `navigator.clipboard.read()` is awkward to drive headless — plan a paste fallback the test can use.
**Confidence:** 85%
**Complexity:** Medium
**Status:** Unexplored
## Draft Acceptance Criteria (seed for brainstorm/plan, per req #4)
**Routing & access**
- [ ] `GET /transactions/external-import-new` renders an import page (admin-gated, same middleware as other transaction routes); 401/redirect for unauthenticated.
- [ ] A "Import" nav entry appears under the transactions section, active on the import route.
**Paste & parse (req #1)**
- [ ] Pasting the exact master Yodlee TSV (same positional columns) parses into the same field set as `manual/tabulate-data`.
- [ ] Header row is dropped; blank rows ignored.
- [ ] `POST …/parse` re-renders the page with a "N rows found" banner and the review table.
**Validation parity (req #3)** — each must produce a visible, row-attributed message:
- [ ] Client not found for bank-account-code → error.
- [ ] Bank account not found by code → error.
- [ ] Date not `MM/dd/yyyy` / unparseable → error.
- [ ] Amount unparseable → error.
- [ ] Status ≠ `POSTED` → not-imported (warn/not-ready).
- [ ] Date before `bank-account/start-date` → not-imported (not-ready).
- [ ] Date on/before `client/locked-until` → not-imported (not-ready).
- [ ] Already-imported (synthetic-id extant) row → skipped, surfaced as warn.
- [ ] Missing client / bank-account / id → error.
**Import (req #5, minimal core change)**
- [ ] `POST …/import` runs the existing `import.transactions` engine via the `:import-source/manual` batch path; no change to `categorize-transaction`/`import-transaction!`/`apply-synthetic-ids`.
- [ ] Only clean rows import; warn-only rows skipped; any error blocks (or imports clean rows + reports errors — match master's "import valid, report failed-validation").
- [ ] Success notification reports counts (imported / skipped / errors), mirroring master's stats.
- [ ] Re-importing the same paste is idempotent (no duplicates).
**Tests (req #4)**
- [ ] A Playwright e2e covering paste → review → import → assert, committed red first, green at the end.
- [ ] Unit tests per validation clause + a Datomic-backed end-to-end import test, modeled on `ledger_test.clj`.
## Failing-First e2e: concrete starting point
Add `test/clj/auto_ap/ssr/transaction/import_test.clj` (unit) and a Playwright spec driven through `test-server`. The e2e is the first artifact and is expected to fail because no `external-import` handler is wired in `ssr/transaction.clj`. Make it green by walking ideas #1 → #2 → #5 → #3 → #4 in that order; the unit suite grows alongside #4.
## Rejection Summary
| # | Idea | Reason Rejected |
|---|------|-----------------|
| 1 | Switch paste format to ledger's named-header columns | Violates req #1 — users paste the exact Yodlee positional export; changing valid input is a silent regression |
| 2 | Keep it a re-frame/CLJS modal | Branch eliminated the CLJS app; contradicts the whole port. (An SSR htmx *modal* was considered but rejected vs. the dedicated page — ledger uses a page and the editable review grid needs the room) |
| 3 | Reimplement validation entirely ledger-style, ignoring `import.transactions` | Duplicates audited `categorize-transaction` logic, risks dropping validations (req #3), and churns core (violates req #5) |
| 4 | Async/streaming import for large pastes | Scope overrun — master is synchronous; YAGNI for the manual paste workflow |
| 5 | Add CSV file-upload alongside paste | Scope overrun — not part of the master manual-import feature |
| 6 | Replace `import-batch` stats with a bespoke result type | Unnecessary core change; the existing stats map already carries imported/failed/sample-error |

275
docs/plans/2026-06-01-001-feat-manual-transaction-import-ssr-plan.md Normal file
View File

@@ -0,0 +1,275 @@
---
date: 2026-06-01
type: feat
status: active
plan_id: 2026-06-01-001
title: "feat: Port manual bank-transaction import to SSR (alpine/htmx)"
depth: standard
origin: docs/ideation/2026-06-01-manual-transaction-import-ssr-ideation.md
---
# feat: Port Manual Bank-Transaction Import to SSR
## Summary
Port the master-branch "manual import transactions" feature into the SSR/alpinejs/htmx stack by implementing the `external-import` handlers that `src/cljc/auto_ap/routes/transactions.cljc` already declares but that no handler currently serves. The feature is a dedicated two-stage page — paste the same Yodlee positional-column TSV → an editable review grid with per-row error/warning badges → import — modeled directly on the SSR ledger import (`src/clj/auto_ap/ssr/ledger.clj`). Validation follows the ledger's `add-errors` shape but preserves every master validation, and the actual write reuses the existing `auto-ap.import.transactions` engine unchanged.
---
## Problem Frame
On `master`, admins import bank transactions by pasting a tab-separated Yodlee export into a re-frame modal (`src/cljs/auto_ap/views/pages/transactions/manual.cljs`) that POSTs EDN to `/api/transactions/batch-upload`. This branch removed the ClojureScript React app and re-implemented the transactions surface server-side, but the manual-import feature was never ported — so admins on this branch cannot manually import transactions at all.
The route names are already scaffolded (`::external-page`, `::external-import-page`, `::external-import-parse`, `::external-import-import` in `routes/transactions.cljc`) but `src/clj/auto_ap/ssr/transaction.clj` wires no handlers for them — they are declared dead ends. The work is to fill that gap with handlers + UI + validation + nav + tests, mirroring the already-shipped ledger import.
---
## Scope Boundaries
**In scope**
- A dedicated SSR import page at `/transaction2/external-import-new` (+ `/parse`, `/import` sub-routes), admin-gated with the same middleware posture as other transaction routes and the ledger import.
- Reuse of the exact Yodlee positional-column paste format (no named-header columns).
- An editable review grid with inline per-field editing and per-row error/warning badges (ledger-style, form-cursor driven).
- Two-tier validation preserving every master validation, with the agreed severity split.
- Import via the existing `auto-ap.import.transactions` engine, block-whole-batch on hard errors.
- A transactions-section "Import" nav entry and an import-result notification.
- A Playwright e2e (committed failing first) plus unit/integration tests modeled on `test/clj/auto_ap/ssr/ledger_test.clj`.
### Deferred to Follow-Up Work
- CSV file upload as an alternative to paste.
- Asynchronous/streaming import for very large pastes.
- Any change to `categorize-transaction` or engine internals.
**Outside this change**
- Named-header column format (rejected — would silently change valid input).
- A bespoke import-result type replacing the engine's stats map.
---
## Key Technical Decisions
1. **Full editable review grid, block-whole-batch on hard error** (from brainstorm). Any remaining hard error blocks the entire import (ledger behavior: `throw+ {:type :field-validation ...}`, re-render the grid with errors highlighted); warn-level rows skip just that row and the rest import. Rationale: with an editable grid, the user can fix fixable problems inline, so "nothing imports until clean-or-skippable" is the coherent contract.
2. **Severity split between fixable errors and inherent warnings.**
- **Hard errors (block, must fix inline):** unparseable/invalid date (must match `MM/dd/yyyy`), unparseable amount, unknown client code (no client for the bank-account-code), unknown bank-account code, missing required fields.
- **Warnings (skip that row, import the rest):** status ≠ `"POSTED"`, transaction date before `bank-account/start-date`, date on/before `client/locked-until`, already-imported (synthetic-id `extant`).
Rationale: fixable problems are correctable by editing a cell; inherent skip-conditions are facts about the data/account that editing cannot change, so they should not block the batch — this also reproduces master's "import valid, report the rest" outcome for those rows.
3. **Reuse the exact Yodlee positional paste format** (req #1). Parse with the master positional `columns` mapping (`auto-ap.import.manual/columns` + `tabulate-data` shape), not ledger's named-header `tsv->import-data`. Rationale: admins paste an unchanged Yodlee export; changing valid input is a silent regression.
4. **Reuse the `import.transactions` engine unchanged** (req #5). The import handler maps reviewed rows → `:transaction/*` maps (the `auto-ap.import.manual/manual->transaction` shape) and drives `start-import-batch :import-source/manual` → `import-transaction!` → `finish!` → `get-stats`, with `apply-synthetic-ids` for dedupe — exactly as `auto-ap.import.manual/import-batch` does today. The SSR layer is presentation + pre-validation only.
5. **Preview/engine parity via shared predicates** (the key design tension). The warn-level conditions shown in the grid before import (`not-ready` from start-date/locked-until, `extant`/already-imported, non-`POSTED`) and the engine's write-time `categorize-transaction` decisions must not drift. Decision: the pre-validation layer computes warn conditions by calling the **same** predicate functions the engine uses (`auto-ap.import.transactions/categorize-transaction` and its inputs — `get-existing` for extant, the bank-account `start-date`/`locked-until` checks), rather than re-deriving parallel logic. The grid is advisory display; the engine remains authoritative at write time, and because both read the same functions they agree. Hard-error (fixable) validations have no engine equivalent and live only in the pre-validation layer / malli schema.
6. **Testable paste path.** The ledger import populates a hidden textarea from `navigator.clipboard` via an alpine `@click`/`paste` handler, which is awkward to drive in headless Playwright. Decision: keep the "Load from clipboard" affordance, but ensure the paste textarea is fillable and that a `pasted`/`change` trigger fires the parse `hx-post`, so the e2e can set the value and dispatch the event without the clipboard API. (Implementation detail of how the trigger is wired is deferred to execution.)
---
## High-Level Technical Design
Two-stage flow mirroring `ssr/ledger.clj`, on the transactions surface:
```
GET /transaction2/external-import-new -> external-import-page (paste form + empty review area)
POST /transaction2/external-import-new/parse -> external-import-parse (decode TSV -> validate -> render editable grid)
POST /transaction2/external-import-new/import -> external-import-import (re-validate -> if any hard error: re-render grid (blocked);
else run import.transactions engine on clean rows,
skip warn rows, return notification with stats)
```
*This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
Validation is two-tier:
- **Tier 1 (shape, hard errors):** a malli parse-form-schema decodes the pasted TSV positionally (reusing the master column order) and coerces/flags shape problems — date parses as `MM/dd/yyyy`, amount parses, required fields present.
- **Tier 2 (business):** a transaction `add-errors`/`table->entries` pass attaches `[message status]` pairs (`:error` / `:warn`) per row, with the hard/warn split from Decision 2, computing warn conditions from the shared engine predicates (Decision 5). `flatten-errors` maps them onto form-cursor field paths for the editable grid.
---
## Implementation Units
Build order is **failing-e2e-first** (req #4): U1 lands the red acceptance test, then U2–U7 turn it green incrementally. Each feature-bearing unit also grows the unit-test suite in `test/clj/auto_ap/ssr/transaction/import_test.clj`.
### U1. Failing e2e acceptance test + deterministic import seed
**Goal:** Commit the end-to-end acceptance test (expected to fail) that defines "done", plus the deterministic test fixture it needs.
**Requirements:** req #4; advances acceptance criteria AC-1, AC-2, AC-9, AC-10.
**Dependencies:** none.
**Files:**
- `e2e/transaction-import.spec.ts` (new)
- `test/clj/auto_ap/test_server.clj` (modify `seed-test-data` to give the seeded bank account a **fixed** `:bank-account/code`, e.g. `"TEST-CHK"`, since `test-bank-account` otherwise assigns a random code)
**Approach:** Mirror `e2e/transaction-navigation.spec.ts` conventions (`x-clients: "mine"` header, `page.goto`, locators). The spec navigates to `/transaction2/external-import-new`, fills the paste box with a known-good Yodlee TSV whose bank-account-code/client-code match the seed (`TEST` client, `TEST-CHK` bank account), triggers parse, asserts parsed rows render in the review grid, clicks Import, and asserts a success notification with an imported count and that the imported transaction is visible on `/transaction2`. Include a second scenario pasting a row with an unknown client code and asserting a blocking error badge + that nothing imports. Drive paste by filling the textarea and dispatching the parse trigger (Decision 6), not the clipboard API.
**Patterns to follow:** `e2e/transaction-navigation.spec.ts`, `e2e/bulk-code-transactions.spec.ts`; seed shape in `test/clj/auto_ap/test_server.clj` `seed-test-data`.
**Test scenarios:**
- Covers AE/AC-1, AC-2: paste valid TSV → rows render → import → "N imported" notification → transaction appears on the list page.
- Covers AC-9: paste TSV with an unknown client code → row shows a blocking error badge, Import is blocked, no transaction created.
- Edge: empty paste → no rows / friendly empty state (assert no crash).
**Verification:** `npx playwright test e2e/transaction-import.spec.ts` runs and **fails** at this unit (no handler yet); the seed change does not break existing e2e specs (`npx playwright test` green except the new file).
**Execution note:** Start red. This is the acceptance contract; do not weaken it to pass — make U2–U7 satisfy it.
### U2. Wire routes and render the import page shell
**Goal:** Make `/transaction2/external-import-new` serve a real page with the correct admin middleware; wire `parse`/`import` routes to placeholder handlers.
**Requirements:** AC-1, AC-12 (auth); req #2.
**Dependencies:** U1.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (new — namespace for the import handlers)
- `src/clj/auto_ap/ssr/transaction.clj` (merge the new `key->handler` entries into the existing map at the `key->handler` def)
**Approach:** Create `external-import-page` returning a `base-page` + `com/page` with breadcrumb ("Transactions" → "Import"), the clipboard helper script, and a forms container (initially just the paste form placeholder). Wire `::route/external-import-page`, `::route/external-import-parse`, `::route/external-import-import` into the transaction `key->handler` with the same middleware chain ledger uses for its import routes (`wrap-schema-enforce`/`wrap-form-4xx-2`/`wrap-schema-decode`/`wrap-nested-form-params` on parse/import) under the transaction page middleware (`wrap-must {:activity :import :subject :transaction}` analogous to ledger's `:subject :ledger`, `wrap-client-redirect-unauthenticated`). Confirm the correct `:activity`/`:subject` against the permissions model.
**Patterns to follow:** `src/clj/auto_ap/ssr/ledger.clj` `external-import-page` and `key->handler` (~lines 276–318, 686–718); `src/clj/auto_ap/ssr/transaction.clj` existing `key->handler` (~line 101).
**Test scenarios:**
- Happy path: `GET` the page as admin → 200, renders the paste form container.
- Error/auth: unauthenticated request → redirect/401 per `wrap-client-redirect-unauthenticated`.
**Verification:** Page loads at the route in the running app and in `test_server`; the e2e gets past navigation (still fails later in the flow).
### U3. Paste + parse using the master positional column format
**Goal:** Parse the pasted Yodlee TSV (exact master columns) into rows and render them; wire the paste form's `pasted`-triggered `hx-post` to the parse handler.
**Requirements:** req #1, req #2; AC-1, AC-3, AC-4.
**Dependencies:** U2.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `external-import-text-form*`, `external-import-parse`, the parse malli schema, and a positional `tsv->rows` decode)
- `test/clj/auto_ap/ssr/transaction/import_test.clj` (new)
**Approach:** Reuse the master column order from `auto-ap.import.manual/columns` and `tabulate-data` (CSV read with `\tab`, drop header) to map positional columns. Define a malli `parse-form-schema` (ledger-style) whose `:table` field uses a `:decode/string` that runs the positional parse and a per-row `:decode/arbitrary` to build row maps; encode Tier-1 shape constraints (date `MM/dd/yyyy`, amount parses, required fields) reusing `auto-ap.import.manual.common/parse-date`/`parse-amount` semantics. `external-import-parse` re-renders the forms fragment with `:just-parsed? true`. Keep the paste textarea fillable and fire the parse trigger on a `pasted`/`change` event (Decision 6).
**Patterns to follow:** `ssr/ledger.clj` `external-import-text-form*`, `external-import-parse`, `tsv->import-data`, `parse-form-schema` (~lines 246–375); `import/manual.clj` `columns`/`tabulate-data`; `import/manual/common.clj` `parse-date`/`parse-amount`.
**Test scenarios:**
- Happy path: a known Yodlee TSV string decodes to the expected row count with the expected field keys/values (positional mapping correct).
- Header handling: first row dropped; blank rows ignored.
- Edge: amount with currency formatting parses; amount unparseable flagged at Tier 1.
- Edge: date not `MM/dd/yyyy` flagged at Tier 1; valid date parses.
- Covers AC-3: pasting the exact master column layout yields the same field set master's `tabulate-data` produced.
**Verification:** After paste, the parsed rows render (read-only at this unit is acceptable); parse unit tests green.
### U4. Editable review grid with per-row error/warning badges
**Goal:** Render parsed rows into an editable `data-grid` where each field is editable and per-row error/warning badges show, with a "Show table" toggle and an Import button.
**Requirements:** req #2; AC-5, AC-6.
**Dependencies:** U3.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `external-import-table-form*`, `external-import-form*`)
**Approach:** Mirror `ledger/external-import-table-form*` using form-cursor (`fc/start-form`, `fc/with-field`, `fc/cursor-map`, `fc/field-value`/`field-name`/`field-errors`) and `com/data-grid-card` / `com/validated-field` / `com/text-input` / `com/money-input`. Columns reflect the transaction row shape (date, description, amount, bank-account-code, client-code, status). Per-row badge summarizes that row's error/warn state with a tooltip listing messages (red for `:error`, yellow for `:warn`). A parsed-summary banner shows row count + error/warning pill counts. Values round-trip on re-submit so inline edits persist.
**Patterns to follow:** `ssr/ledger.clj` `external-import-table-form*` and `external-import-form*` (~lines 117–274).
**Test scenarios:**
- Test expectation: none for pure rendering structure beyond what U5 exercises — but include: rows with no errors render without a badge; rows with errors render a red badge; rows with only warnings render a yellow badge (assert via the rendered hiccup/markup in a handler-level test once U5 attaches errors).
**Verification:** Parsed grid is visibly editable; badges appear once U5 attaches errors; e2e can see rows.
### U5. Two-tier validation preserving every master validation
**Goal:** Attach hard-error and warning statuses to rows per the severity split, reusing the engine's predicates for the warn conditions so the preview matches the engine.
**Requirements:** req #3, req #5 (Decision 5); AC-7, AC-8, AC-9.
**Dependencies:** U3 (Tier 1 shape errors), U4 (badges to display them).
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `add-errors`, `table->entries`, `flatten-errors`, `entry-error-types` analogues)
- `test/clj/auto_ap/ssr/transaction/import_test.clj` (extend)
**Approach:** Build a transaction `add-errors` that, given lookups (client-by-bank-account-code, bank-account-by-code, bank-account `start-date`/`locked-until`, existing transaction ids), assigns:
- **Hard errors:** unknown client code, unknown bank-account code, missing required fields. (Tier-1 date/amount errors already present from U3.)
- **Warnings:** status ≠ `POSTED`; date before `bank-account/start-date`; date on/before `client/locked-until`; already-imported (synthetic-id present in existing ids).
Compute the warn conditions by calling the same functions the engine uses — `auto-ap.import.transactions/categorize-transaction` (and its inputs `get-existing`, the `apply-synthetic-ids` key) — rather than parallel logic (Decision 5). `flatten-errors` maps `[message status]` onto form-cursor field paths so badges render against the right rows. Map every master validation explicitly (see `import/manual.clj manual->transaction` and `import/transactions.clj categorize-transaction`).
**Patterns to follow:** `ssr/ledger.clj` `add-errors`/`table->entries`/`flatten-errors`/`entry-error-types` (~lines 380–523); `import/transactions.clj` `categorize-transaction`/`get-existing`/`apply-synthetic-ids`.
**Test scenarios (one per validation, modeled on `ledger_test/add-errors-test`):**
- Hard error: unknown client code → `:error` with a clear message.
- Hard error: unknown bank-account code → `:error`.
- Hard error: missing required field → `:error`.
- (Tier 1) invalid date / unparseable amount → `:error`.
- Warning: status ≠ `POSTED` → `:warn`, row skipped.
- Warning: date before `bank-account/start-date` → `:warn`.
- Warning: date on/before `client/locked-until` → `:warn`.
- Warning: already-imported (extant synthetic id) → `:warn`.
- Parity: a row the grid marks clean is categorized `:import` by `categorize-transaction`; a row marked warn-skip is categorized to the matching non-`:import` action (assert grid preview agrees with engine).
- Pass-through: a fully valid row has no errors/warnings.
**Verification:** Validation unit tests green; badges reflect the correct severities in the grid.
### U6. Import via the existing engine, block-on-error, with notification
**Goal:** Implement `external-import-import`: block the whole batch if any hard error remains; otherwise run the `import.transactions` engine on clean rows (skipping warn rows) and return a result notification.
**Requirements:** req #5, Decisions 1 & 4; AC-2, AC-9, AC-10, AC-11.
**Dependencies:** U5.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `rows->transactions`, `import-transactions`, `external-import-import`)
**Approach:** Re-validate submitted (possibly edited) rows via U5. If any `:error` rows remain, `throw+ {:type :field-validation :form-errors ... :form-params ...}` and re-render the grid with errors (the `wrap-form-4xx-2` middleware handles re-render) — nothing imports. Otherwise map clean rows → `:transaction/*` maps using the `auto-ap.import.manual/manual->transaction` shape, apply `apply-synthetic-ids`, then drive `start-import-batch :import-source/manual` → `import-transaction!` (per row) → `finish!` → `get-stats`. Warn-only rows are excluded from the engine input (skipped). Return `html-response` re-rendering the form with an `hx-trigger` notification reporting counts from the engine stats (imported / skipped / not-ready / extant), mirroring master's stats surface.
**Patterns to follow:** `ssr/ledger.clj` `import-ledger` + `external-import-import` (~lines 554–684); `import/manual.clj` `import-batch` (engine driving); `import/manual.clj` `manual->transaction`.
**Test scenarios (modeled on `ledger_test` `import-ledger-*` tests, against Datomic via `wrap-setup`/`setup-test-data`/`admin-token`):**
- Happy path: all-clean batch → engine imports all rows; stats report the imported count; transactions exist in the DB afterward.
- Block-on-error: a batch with one hard-error row → throws `:field-validation`; **no** transactions are created (assert DB unchanged).
- Warning skip: a batch with one warn-only row (e.g., non-`POSTED`) and clean rows → clean rows import, warn row skipped, stats reflect the skip.
- Idempotency: importing the same paste twice → second run imports 0 new (extant/synthetic-id dedupe); no duplicates.
- Integration: imported transaction carries `:import-source/manual` and is categorized/coded by the engine as it would be for any import (engine unchanged).
**Verification:** Import unit/integration tests green; the e2e's import step succeeds and the transaction appears on the list page.
### U7. Transactions "Import" nav entry + final polish
**Goal:** Add an "Import" entry to the transactions section nav (parallel to the ledger import nav) and finish the parsed-summary banner / notification copy.
**Requirements:** req #2; AC-1, AC-11.
**Dependencies:** U2 (route exists), U6 (notification exists).
**Files:**
- `src/clj/auto_ap/ssr/components/aside.clj` (add a transactions "Import" nav button + mark active on `::transaction-routes/external-import-page`)
**Approach:** Mirror the ledger import nav entry in `aside.clj` — add a sub-menu button under the transactions section linking to `::transaction-routes/external-import-page`, active-highlighted on that matched route. Confirm the banner shows row counts + error/warning pills (from U4) and the success notification copy matches the engine stats.
**Patterns to follow:** `ssr/components/aside.clj` ledger import nav (~lines 360–366) and the transactions sub-menu (~lines 285–298).
**Test scenarios:**
- Test expectation: none (navigation markup) — covered indirectly by the e2e navigating via the nav link; optionally assert the nav button renders with the correct href on the import route.
**Verification:** Full `e2e/transaction-import.spec.ts` passes; nav link is present and active on the import page.
---
## Acceptance Criteria
**Routing & access**
- **AC-1.** `GET /transaction2/external-import-new` renders the import page for an admin; an "Import" nav entry under the transactions section links to it and is active there.
- **AC-12.** Unauthenticated access redirects/401s per the standard transaction-route middleware.
**Paste & parse (req #1)**
- **AC-3.** Pasting the exact master Yodlee positional TSV parses into the same field set as `auto-ap.import.manual/tabulate-data`; the header row is dropped and blank rows ignored.
- **AC-4.** `POST .../parse` re-renders the page with a "N rows found" banner and the review grid.
**Review grid (req #2)**
- **AC-5.** Parsed rows render in an editable grid; each field is editable and inline edits persist across re-submit.
- **AC-6.** Each row shows an error badge (red) when it has a hard error, a warning badge (yellow) when it has only warnings, and no badge when clean; badges list messages on hover.
**Validation parity (req #3)** — each produces a visible, row-attributed message:
- **AC-7.** Hard errors block: client not found, bank account not found, date not `MM/dd/yyyy`, amount unparseable, missing required field.
- **AC-8.** Warnings skip just that row: status ≠ `POSTED`, date before `bank-account/start-date`, date on/before `client/locked-until`, already-imported.
- **AC-9.** With any remaining hard error, clicking Import blocks the whole batch (nothing imports) and re-renders the grid with errors highlighted.
**Import (req #5)**
- **AC-2.** `POST .../import` imports the clean rows via the existing `import.transactions` engine on the `:import-source/manual` batch path; the success notification reports counts (imported / skipped / not-ready / extant).
- **AC-10.** Re-importing the same paste is idempotent — no duplicate transactions (synthetic-id dedupe preserved).
- **AC-11.** `categorize-transaction` and the engine internals are unchanged by this work.
**Tests (req #4)**
- The Playwright e2e `e2e/transaction-import.spec.ts` exists, was committed failing first, and passes at the end.
- Unit/integration tests in `test/clj/auto_ap/ssr/transaction/import_test.clj` cover each validation clause and the end-to-end import flow against Datomic.
---
## Test Strategy
- **e2e (Playwright):** `e2e/transaction-import.spec.ts`, driven through `test/clj/auto_ap/test_server.clj` (real routes, injected test auth). Committed red in U1, green by U7.
- **Unit/integration (clojure.test):** `test/clj/auto_ap/ssr/transaction/import_test.clj`, modeled on `test/clj/auto_ap/ssr/ledger_test.clj` — pure parse/format tests, one validation test per clause, and Datomic-backed import tests via `wrap-setup` / `setup-test-data` / `admin-token`. Run with `clj-nrepl-eval -p PORT "(clojure.test/run-tests 'auto-ap.ssr.transaction.import-test)"` per AGENTS.md (preferred over `lein test`).
- **Validation-parity oracle:** existing `test/clj/auto_ap/import/transactions_test.clj` (`categorize-transaction`) backs Decision 5 — the warn-condition predicates the grid reuses are already under test.
---
## System-Wide Impact
- **Editing discipline (AGENTS.md):** all Clojure edits go through the clojure-mcp editing tools (or `@clojure-author`), not raw file edits; use `clojure-eval`/`clj-nrepl-eval` to compile-check and run tests. Run `lein cljfmt fix` before committing; `clj-paren-repair` on any file that won't compile.
- **Shared component reuse:** the feature composes existing `ssr/components` (`data-grid-card`, `validated-field`, `text-input`, `money-input`, `button`, `checkbox`, `errors`, `pill`, `form-errors`) and `ssr/form-cursor` — no core-component changes expected (req #5). If a component genuinely needs a new option, prefer an additive, backward-compatible change and flag it.
- **Test fixture change:** giving the seeded bank account a deterministic `:bank-account/code` in `test_server.clj` could affect other e2e specs that assume the random code; U1 verifies the existing suite stays green.
- **Permissions:** confirm the `wrap-must` `:activity`/`:subject` for the import routes matches the permission model (ledger uses `{:activity :import :subject :ledger}`); use the transaction equivalent.
---
## Risks & Mitigations
- **Preview/engine drift (highest risk).** Mitigated by Decision 5 — share the engine's predicates for warn conditions; the parity test in U5 asserts the grid and `categorize-transaction` agree.
- **Headless clipboard paste.** Mitigated by Decision 6 — fillable textarea + explicit parse trigger so the e2e never needs `navigator.clipboard`.
- **form-cursor round-tripping of an editable grid** is the trickiest ledger mechanic to copy; mitigate by mirroring `external-import-table-form*` closely and testing edit-persist-on-resubmit early (U4).
- **Positional column brittleness** is inherited from master (Yodlee column order); not a regression, and out of scope to fix here.
---
## Deferred Implementation Notes (execution-time unknowns)
- Exact helper/function names in the new `import.clj` namespace.
- The precise malli `:decode` wiring for positional parsing (reuse vs. thin wrapper around `tabulate-data`).
- The exact `:activity`/`:subject` keyword for the import-route `wrap-must` (verify against the permissions model).
- The seeded bank-account code value and whether any existing e2e needs adjustment after making it deterministic.
- Final notification/banner copy.

125
e2e/transaction-import.spec.ts Normal file
View File

@@ -0,0 +1,125 @@
import { test, expect } from '@playwright/test';
// The SSR manual transaction import accepts the exact Yodlee positional-column
// TSV format from the master branch. Column order (14 columns), per
// auto-ap.import.manual/columns:
// 0:status 1:raw-date 2:description-original 3:high-level-category
// 4,5:(unused) 6:amount 7..11:(unused) 12:bank-account-code 13:client-code
//
// The test server (auto-ap.test-server) seeds client "TEST" with a bank
// account whose code is the deterministic "TEST-CHK" (see seed-test-data).
const IMPORT_PATH = '/transaction2/external-import-new';
function yodleeRow(opts: {
status?: string;
date?: string;
description?: string;
category?: string;
amount?: string;
bankAccountCode?: string;
clientCode?: string;
}): string {
const cols = new Array(14).fill('');
cols[0] = opts.status ?? 'POSTED';
cols[1] = opts.date ?? '';
cols[2] = opts.description ?? '';
cols[3] = opts.category ?? '';
cols[6] = opts.amount ?? '';
cols[12] = opts.bankAccountCode ?? '';
cols[13] = opts.clientCode ?? '';
return cols.join('\t');
}
function yodleeTsv(rows: string[]): string {
// First line is a header that the importer drops.
const header = new Array(14).fill('');
header[0] = 'Status';
header[1] = 'Date';
header[2] = 'Description';
header[6] = 'Amount';
header[12] = 'Bank Account';
header[13] = 'Client';
return [header.join('\t'), ...rows].join('\n');
}
async function gotoImport(page: any) {
await page.setExtraHTTPHeaders({ 'x-clients': '"mine"' });
await page.goto(IMPORT_PATH);
}
async function pasteAndParse(page: any, tsv: string) {
const textarea = page.locator('#parse-form textarea').first();
await textarea.fill(tsv);
// A visible "Parse" button submits the paste form (htmx swaps in the grid).
await page.getByRole('button', { name: /parse/i }).click();
await page.waitForTimeout(800);
}
test.describe('Manual Transaction Import (SSR)', () => {
test('renders the import page with a paste box', async ({ page }) => {
await gotoImport(page);
await expect(page.locator('#parse-form textarea').first()).toBeVisible();
});
test('paste -> parse -> review grid -> import a valid transaction', async ({ page }) => {
await gotoImport(page);
const description = 'E2E Imported Coffee';
const tsv = yodleeTsv([
yodleeRow({
date: '01/15/2024',
description,
category: 'Food',
amount: '12.50',
bankAccountCode: 'TEST-CHK',
clientCode: 'TEST',
}),
]);
await pasteAndParse(page, tsv);
// The review grid renders the parsed row as editable inputs (the
// description lives in an input value, so assert on the input, not text).
await expect(page.locator('input[value="TEST-CHK"]').first()).toBeVisible();
await expect(page.locator(`input[value="${description}"]`).first()).toBeVisible();
// Import the clean batch.
await page.getByRole('button', { name: /^import$/i }).click();
await page.waitForTimeout(1000);
// The imported transaction shows up on the transactions list.
await page.goto('/transaction2?date-range=all');
await page.waitForSelector('table tbody tr');
await expect(page.getByText(description)).toBeVisible();
});
test('blocks the whole batch when a row has an unknown bank-account code', async ({ page }) => {
await gotoImport(page);
const description = 'E2E Blocked Row';
const tsv = yodleeTsv([
yodleeRow({
date: '01/16/2024',
description,
amount: '20.00',
bankAccountCode: 'NOPE-DOES-NOT-EXIST',
clientCode: 'TEST',
}),
]);
await pasteAndParse(page, tsv);
// The grid surfaces a blocking error for the bad row. The importer reuses
// the master-branch message wording ("Cannot find bank account by code …").
await expect(page.getByText(/cannot find bank account/i).first()).toBeVisible();
// Importing does not create the transaction (batch blocked).
await page.getByRole('button', { name: /^import$/i }).click();
await page.waitForTimeout(800);
await page.goto('/transaction2?date-range=all');
await page.waitForSelector('table tbody tr');
await expect(page.getByText(description)).toHaveCount(0);
});
});

6
src/clj/auto_ap/ssr/components/aside.clj
View File

@@ -304,6 +304,12 @@
:hx-boost "true"
:hx-include "#transaction-filters"}
"Approved")
(when (is-admin? (:identity request))
(menu-button- {:href (bidi/path-for ssr-routes/only-routes
::transaction-routes/external-import-page)
:active? (= ::transaction-routes/external-import-page (:matched-route request))
:hx-boost "true"}
"Import"))
(when (can? (:identity request)
{:subject :transaction :activity :insights})
(menu-button- {:href (bidi/path-for ssr-routes/only-routes

2
src/clj/auto_ap/ssr/transaction.clj
View File

@@ -19,6 +19,7 @@
grid-page query-schema
wrap-status-from-source]]
[auto-ap.ssr.transaction.edit :as edit :refer [transaction-account-row*]]
[auto-ap.ssr.transaction.import :as t-import]
[auto-ap.ssr.utils
:refer [apply-middleware-to-all-handlers entity-id html-response
many-entity modal-response percentage ref->enum-schema
@@ -101,6 +102,7 @@
(def key->handler
(merge edit/key->handler
bulk-code/key->handler
t-import/key->handler
(apply-middleware-to-all-handlers
{::route/page page
::route/approved-page (-> page (wrap-implied-route-param :status :transaction-approval-status/approved))

300
src/clj/auto_ap/ssr/transaction/import.clj Normal file
View File

@@ -0,0 +1,300 @@
(ns auto-ap.ssr.transaction.import
"SSR manual bank-transaction import. Mirrors the SSR ledger import
(auto-ap.ssr.ledger) but accepts the exact master-branch Yodlee
positional-column TSV and drives the existing
auto-ap.import.transactions engine (via auto-ap.import.manual/import-batch)
unchanged. Two-stage flow: paste -> editable review grid -> import."
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.graphql.utils :refer [assert-admin]]
[auto-ap.import.manual :as manual]
[auto-ap.import.transactions :as t]
[auto-ap.permissions :refer [wrap-must]]
[auto-ap.routes.transactions :as route]
[auto-ap.routes.utils :refer [wrap-client-redirect-unauthenticated]]
[auto-ap.ssr-routes :as ssr-routes]
[auto-ap.ssr.components :as com]
[auto-ap.ssr.form-cursor :as fc]
[auto-ap.ssr.hx :as hx]
[auto-ap.ssr.nested-form-params :refer [wrap-nested-form-params]]
[auto-ap.ssr.ui :refer [base-page]]
[auto-ap.ssr.utils
:refer [apply-middleware-to-all-handlers html-response
wrap-form-4xx-2 wrap-schema-decode wrap-schema-enforce]]
[bidi.bidi :as bidi]
[clojure.data.csv :as csv]
[clojure.java.io :as io]
[clojure.string :as str]
[datomic.api :as dc]
[malli.core :as mc]
[slingshot.slingshot :refer [throw+]]))
;; ---------------------------------------------------------------------------
;; Parsing (positional Yodlee columns, identical to master)
;; ---------------------------------------------------------------------------
(defn tsv->rows
"Decode a pasted tab-separated Yodlee export into a vector of raw column
vectors. Drops the header row (like auto-ap.import.manual/tabulate-data) and
skips blank lines. No-op when already decoded."
[data]
(if (string? data)
(with-open [r (io/reader (char-array data))]
(into []
(comp (drop 1)
(filter (fn [row] (some (fn [c] (seq (str/trim (or c "")))) row))))
(csv/read-csv r :separator \tab)))
data))
(defn vector->row
"Map a raw column vector onto the master positional column keys."
[t]
(if (vector? t)
(into {} (filter first (map vector manual/columns t)))
t))
(def parse-form-schema
(mc/schema
[:map
[:table {:min 1
:error/message "Paste should contain at least one row to import"
:decode/string tsv->rows}
[:vector {:coerce? true}
[:map {:decode/arbitrary vector->row}
[:status {:optional true} [:maybe :string]]
[:raw-date {:optional true} [:maybe :string]]
[:description-original {:optional true} [:maybe :string]]
[:high-level-category {:optional true} [:maybe :string]]
[:amount {:optional true} [:maybe :string]]
[:bank-account-code {:optional true} [:maybe :string]]
[:client-code {:optional true} [:maybe :string]]]]]]))
;; ---------------------------------------------------------------------------
;; Validation (two-tier, preserving every master validation)
;; ---------------------------------------------------------------------------
(defn- bank-account-code->client [db]
(into {} (dc/q '[:find ?bac ?c
:where
[?c :client/bank-accounts ?ba]
[?ba :bank-account/code ?bac]]
db)))
(defn- bank-account-code->bank-account [db]
(into {} (dc/q '[:find ?bac ?ba
:where [?ba :bank-account/code ?bac]]
db)))
(defn warn-message
"Map a non-:import engine categorization to a [message :warn] pair, or nil
when the row will import cleanly."
[action]
(case action
:extant ["Already imported — skipped" :warn]
:not-ready ["Not ready (before account start date, client locked, or not posted) — skipped" :warn]
:suppressed ["Suppressed — skipped" :warn]
nil))
(defn classify-table
"Given parsed row maps, return {:form-errors {:table {idx [[msg status]...]}}
:has-errors? bool}. Hard (fixable) errors come from
manual/manual->transaction; warnings come from the engine's own
categorize-transaction so the grid preview matches what the import will do."
[rows]
(let [db (dc/db conn)
client-lookup (bank-account-code->client db)
ba-lookup (bank-account-code->bank-account db)
indexed (map-indexed
(fn [i row]
(assoc (manual/manual->transaction row ba-lookup client-lookup)
::idx i))
rows)
with-ids (t/apply-synthetic-ids indexed)
ba-cache (atom {})
existing-cache (atom {})
entries (->> with-ids
(map (fn [txn]
(let [idx (::idx txn)
hard (mapv (fn [e] [(:info e) :error]) (:errors txn))
warn (when (and (empty? hard)
(:transaction/bank-account txn))
(let [ba-id (:transaction/bank-account txn)
ba (or (get @ba-cache ba-id)
(get (swap! ba-cache assoc ba-id
(dc/pull db t/bank-account-pull ba-id))
ba-id))
existing (or (get @existing-cache ba-id)
(get (swap! existing-cache assoc ba-id
(t/get-existing ba-id))
ba-id))]
(warn-message (t/categorize-transaction txn ba existing))))]
[idx (cond-> hard warn (conj warn))])))
(sort-by first))
form-errors {:table (into {} (filter (fn [[_ errs]] (seq errs)) entries))}
has-errors? (boolean (some (fn [[_ errs]] (some (fn [[_ s]] (= :error s)) errs)) entries))]
{:form-errors form-errors
:has-errors? has-errors?}))
;; ---------------------------------------------------------------------------
;; Views
;; ---------------------------------------------------------------------------
(defn- row-badge [errors]
(when (seq errors)
[:div.p-1.flex.flex-col.gap-1
(for [[m s] errors]
[:div.text-xs {:class (if (= :error s) "text-red-600" "text-yellow-600")} m])]))
(defn- parsed-banner [request]
(let [errs (->> (:form-errors request) :table vals (mapcat identity))
n-err (count (filter (fn [[_ s]] (= :error s)) errs))
n-warn (count (filter (fn [[_ s]] (= :warn s)) errs))
n-rows (count (:table (:form-params request)))]
[:div.bg-green-50.text-green-700.rounded.p-3.my-2
(format "%,d rows parsed. " n-rows)
(when (pos? n-err)
[:span.text-red-700.font-semibold (format "%d error(s) must be fixed. " n-err)])
(when (pos? n-warn)
[:span.text-yellow-700.font-semibold (format "%d warning row(s) will be skipped. " n-warn)])]))
(defn external-import-text-form* [request]
(fc/start-form
(or (:form-params request) {}) (:form-errors request)
[:form#parse-form {:hx-post (bidi/path-for ssr-routes/only-routes ::route/external-import-parse)
:hx-target "#forms"
:hx-swap "outerHTML"}
(fc/with-field :table
[:div.flex.flex-col.gap-2
(com/errors {:errors (when (string? (fc/field-errors)) (fc/field-errors))})
(com/text-area {:name (fc/field-name)
:rows 6
:class "w-full font-mono text-xs"
:placeholder "Paste your Yodlee transaction export (tab-separated, including the header row) here"})])
(com/button {:color :primary :type "submit"} "Parse")]))
(defn external-import-table-form* [request]
(fc/start-form
(:form-params request) (:form-errors request)
(fc/with-field :table
(when (seq (fc/field-value))
[:div.mt-4 {:x-data (hx/json {"showTable" true})}
(when (:just-parsed? request)
(parsed-banner request))
[:form {:hx-post (bidi/path-for ssr-routes/only-routes ::route/external-import-import)
:hx-target "#forms"
:hx-swap "outerHTML"
:autocomplete "off"}
[:div.flex.gap-4.items-center.my-2
(com/checkbox {"@click" "showTable=!showTable"} "Show table")
(com/button {:color :primary :type "submit"} "Import")]
[:div {:x-show "showTable"}
(com/data-grid-card
{:id "transaction-import-data"
:route nil
:title "Transactions to import"
:paginate? false
:headers [(com/data-grid-header {} "Date")
(com/data-grid-header {} "Description")
(com/data-grid-header {} "Amount")
(com/data-grid-header {} "Bank Account")
(com/data-grid-header {} "Client")
(com/data-grid-header {} "Status")
(com/data-grid-header {} "")]
:rows
(fc/cursor-map
(fn [_]
(let [row-errors (fc/field-errors)]
(com/data-grid-row
{}
(com/data-grid-cell {} (fc/with-field :raw-date
(com/text-input {:value (fc/field-value) :name (fc/field-name) :class "w-28"})))
(com/data-grid-cell {} (fc/with-field :description-original
(com/text-input {:value (fc/field-value) :name (fc/field-name)})))
(com/data-grid-cell {} (fc/with-field :amount
(com/money-input {:value (fc/field-value) :name (fc/field-name) :class "w-28"})))
(com/data-grid-cell {} (fc/with-field :bank-account-code
(com/text-input {:value (fc/field-value) :name (fc/field-name) :class "w-28"})))
(com/data-grid-cell {} (fc/with-field :client-code
(com/text-input {:value (fc/field-value) :name (fc/field-name) :class "w-24"})))
(com/data-grid-cell {} [:span.text-xs.text-gray-500 (fc/with-field :status (fc/field-value))])
(com/data-grid-cell {:class "align-top"} (row-badge row-errors))))))}
nil)]]]))))
(defn external-import-form* [request]
[:div#forms
(external-import-text-form* request)
(external-import-table-form* request)])
(defn external-import-page [request]
(base-page
request
(com/page {:nav com/main-aside-nav
:client-selection (:client-selection request)
:clients (:clients request)
:client (:client request)
:identity (:identity request)
:request request}
(com/breadcrumbs {}
[:a {:href (bidi/path-for ssr-routes/only-routes ::route/page)} "Transactions"]
[:a {:href (bidi/path-for ssr-routes/only-routes ::route/external-import-page)} "Import"])
(external-import-form* request))
"Import Transactions"))
;; ---------------------------------------------------------------------------
;; Handlers
;; ---------------------------------------------------------------------------
(defn external-import-parse [request]
(let [{:keys [form-errors]} (classify-table (:table (:form-params request)))]
(html-response
(external-import-form* (assoc request :form-errors form-errors :just-parsed? true)))))
(defn import-transactions
"Validate the (possibly edited) rows. Block the whole batch when any hard
error remains; otherwise run the existing import engine on the rows. Returns
the engine stats."
[request]
(assert-admin (:identity request))
(let [rows (:table (:form-params request))
{:keys [form-errors has-errors?]} (classify-table rows)]
(when has-errors?
(throw+ {:type :field-validation
:form-errors form-errors
:form-params (:form-params request)}))
(let [user (or (:user/name (:identity request))
(:user (:identity request))
"SSR import")]
(manual/import-batch rows user))))
(defn external-import-import [request]
(let [stats (import-transactions request)
imported (:import-batch/imported stats 0)
extant (:import-batch/extant stats 0)
not-ready (:import-batch/not-ready stats 0)
errored (+ (:import-batch/error stats 0) (:failed-validation stats 0))]
(html-response
(external-import-form* (assoc request :form-params {} :form-errors {}))
:headers {"hx-trigger"
(hx/json {"notification"
(format "%d imported, %d already imported, %d not ready, %d errored."
imported extant not-ready errored)})})))
;; ---------------------------------------------------------------------------
;; Routing
;; ---------------------------------------------------------------------------
(def key->handler
(apply-middleware-to-all-handlers
{::route/external-import-page external-import-page
::route/external-import-parse (-> external-import-parse
(wrap-schema-enforce :form-schema parse-form-schema)
(wrap-form-4xx-2 external-import-parse)
(wrap-schema-decode :form-schema parse-form-schema))
::route/external-import-import (-> external-import-import
(wrap-schema-enforce :form-schema parse-form-schema)
(wrap-form-4xx-2 external-import-parse)
(wrap-nested-form-params))}
(fn [h]
(-> h
(wrap-must {:activity :import :subject :transaction})
(wrap-client-redirect-unauthenticated)))))

164
test/clj/auto_ap/ssr/transaction/import_test.clj Normal file
View File

@@ -0,0 +1,164 @@
(ns auto-ap.ssr.transaction.import-test
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.integration.util :refer [admin-token setup-test-data test-bank-account
test-client wrap-setup]]
[auto-ap.ssr.transaction.import :as sut]
[auto-ap.ssr.utils :refer [main-transformer]]
[clojure.test :refer [deftest is testing use-fixtures]]
[datomic.api :as dc]
[malli.core :as mc]
[slingshot.slingshot :refer [try+]]))
(use-fixtures :each wrap-setup)
(defn- seed-client! []
(setup-test-data
[(test-client :db/id "import-client"
:client/code "TEST"
:client/locations ["DT"]
:client/bank-accounts [(test-bank-account :db/id "import-ba"
:bank-account/code "TEST-CHK")])]))
(defn- txn-count []
(or (dc/q '[:find (count ?e) . :where [?e :transaction/id]] (dc/db conn)) 0))
(defn- import! [rows]
(sut/import-transactions {:form-params {:table rows} :identity (admin-token)}))
;; =============================================================================
;; Pure parsing — tsv->rows, vector->row, parse-form-schema
;; =============================================================================
(deftest tsv->rows-test
(testing "Drops the header row and parses tab-separated columns"
(let [tsv "Status\tDate\tDescription\nPOSTED\t01/15/2024\tCoffee"
rows (sut/tsv->rows tsv)]
(is (= 1 (count rows)))
(is (= ["POSTED" "01/15/2024" "Coffee"] (first rows)))))
(testing "Skips blank lines"
(is (= 1 (count (sut/tsv->rows "h1\th2\nPOSTED\tx\n\t\n")))))
(testing "No-op on already-decoded data"
(is (= [{:raw-date "x"}] (sut/tsv->rows [{:raw-date "x"}])))))
(deftest vector->row-test
(testing "Maps the exact master positional columns"
(let [row (sut/vector->row
["POSTED" "01/15/2024" "Coffee" "Food" "" "" "12.50" "" "" "" "" "" "TEST-CHK" "TEST"])]
(is (= "POSTED" (:status row)))
(is (= "01/15/2024" (:raw-date row)))
(is (= "Coffee" (:description-original row)))
(is (= "12.50" (:amount row)))
(is (= "TEST-CHK" (:bank-account-code row)))
(is (= "TEST" (:client-code row))))))
(deftest parse-form-schema-test
(testing "Decodes a pasted Yodlee TSV string into row maps"
(let [tsv (str "Status\tDate\tDescription\t\t\t\tAmount\t\t\t\t\t\tBank\tClient\n"
"POSTED\t01/15/2024\tCoffee\tFood\t\t\t12.50\t\t\t\t\t\tTEST-CHK\tTEST")
decoded (mc/decode sut/parse-form-schema {:table tsv} main-transformer)]
(is (= 1 (count (:table decoded))))
(is (= "TEST-CHK" (:bank-account-code (first (:table decoded))))))))
;; =============================================================================
;; Validation — classify-table (hard errors + warnings, preserving master)
;; =============================================================================
(deftest classify-hard-errors-test
(seed-client!)
(testing "Unknown bank-account code is a hard error"
(let [{:keys [form-errors has-errors?]}
(sut/classify-table [{:raw-date "01/15/2024" :amount "1.00" :bank-account-code "NOPE"}])]
(is has-errors?)
(is (some (fn [[m _]] (re-find #"bank account" m)) (get-in form-errors [:table 0])))))
(testing "Unknown client fires independently when the bank account exists but is linked to no client"
@(dc/transact conn [{:db/id "orphan-ba"
:bank-account/code "ORPHAN-CHK"
:bank-account/type :bank-account-type/check}])
(let [{:keys [form-errors has-errors?]}
(sut/classify-table [{:raw-date "01/15/2024" :amount "1.00" :bank-account-code "ORPHAN-CHK"}])
msgs (map first (get-in form-errors [:table 0]))]
(is has-errors?)
(is (some #(re-find #"Cannot find client" %) msgs)
"client-not-found error fires")
(is (not (some #(re-find #"bank account by code" %) msgs))
"bank-account-not-found error does not fire because the bank account exists")))
(testing "Invalid date is a hard error"
(let [{:keys [form-errors has-errors?]}
(sut/classify-table [{:raw-date "not-a-date" :amount "1.00" :bank-account-code "TEST-CHK"}])]
(is has-errors?)
(is (some (fn [[m _]] (re-find #"(?i)mm/dd/yyyy|date" m)) (get-in form-errors [:table 0]))))))
(deftest classify-clean-test
(seed-client!)
(testing "A fully valid row produces no errors or warnings"
(let [{:keys [form-errors has-errors?]}
(sut/classify-table [{:raw-date "01/15/2024" :description-original "Coffee"
:amount "12.50" :bank-account-code "TEST-CHK" :client-code "TEST"}])]
(is (not has-errors?))
(is (empty? (get-in form-errors [:table 0]))))))
(deftest classify-not-ready-warning-test
(testing "A date before the bank-account start-date is a (skippable) warning, not an error"
(setup-test-data
[(test-client :db/id "import-client"
:client/code "TEST"
:client/locations ["DT"]
:client/bank-accounts [(test-bank-account :db/id "import-ba"
:bank-account/code "TEST-CHK"
:bank-account/start-date #inst "2030-01-01")])])
(let [{:keys [form-errors has-errors?]}
(sut/classify-table [{:raw-date "01/15/2024" :description-original "Early"
:amount "5.00" :bank-account-code "TEST-CHK" :client-code "TEST"}])]
(is (not has-errors?) "warnings do not block")
(is (some (fn [[_ s]] (= :warn s)) (get-in form-errors [:table 0]))))))
;; =============================================================================
;; Import flow — import-transactions (engine reuse, block, idempotency, skip)
;; =============================================================================
(deftest import-clean-test
(seed-client!)
(testing "Clean rows import via the engine and persist"
(let [stats (import! [{:raw-date "01/15/2024" :description-original "Coffee"
:amount "12.50" :bank-account-code "TEST-CHK" :client-code "TEST"}])]
(is (= 1 (:import-batch/imported stats)))
(is (= 1 (txn-count))))))
(deftest import-blocks-on-hard-error-test
(seed-client!)
(testing "Any hard error blocks the whole batch — nothing is written"
(is (= :blocked
(try+
(import! [{:raw-date "01/15/2024" :amount "1.00" :bank-account-code "NOPE"}])
:did-not-throw
(catch [:type :field-validation] _ :blocked))))
(is (= 0 (txn-count)))))
(deftest import-idempotent-test
(seed-client!)
(testing "Re-importing the same paste is idempotent (extant), no duplicates"
(let [row [{:raw-date "01/15/2024" :description-original "Coffee"
:amount "12.50" :bank-account-code "TEST-CHK" :client-code "TEST"}]]
(import! row)
(let [stats (import! row)]
(is (= 0 (:import-batch/imported stats)))
(is (= 1 (:import-batch/extant stats)))
(is (= 1 (txn-count)))))))
(deftest import-skips-warning-rows-test
(testing "Warning rows (not-ready) are skipped, not imported, without blocking"
(setup-test-data
[(test-client :db/id "import-client"
:client/code "TEST"
:client/locations ["DT"]
:client/bank-accounts [(test-bank-account :db/id "import-ba"
:bank-account/code "TEST-CHK"
:bank-account/start-date #inst "2030-01-01")])])
(let [stats (import! [{:raw-date "01/15/2024" :description-original "Early"
:amount "5.00" :bank-account-code "TEST-CHK" :client-code "TEST"}])]
(is (= 0 (:import-batch/imported stats)))
(is (= 1 (:import-batch/not-ready stats)))
(is (= 0 (txn-count))))))

28
test/clj/auto_ap/test_server.clj
View File

@@ -67,7 +67,7 @@
[(assoc (test-client :db/id "client-id"
:client/code "TEST"
:client/locations ["DT"])
:client/bank-accounts [(test-bank-account :db/id "bank-account-id")])
:client/bank-accounts [(test-bank-account :db/id "bank-account-id" :bank-account/code "TEST-CHK")])
(test-client :db/id "client-id-2"
:client/code "TEST2"
:client/locations ["NY"])
@@ -135,19 +135,19 @@
:payment/status :payment-status/pending
:payment/date #inst "2023-06-15")
;; Transaction and unpaid invoice for link testing
(test-transaction :db/id "transaction-id-unpaid"
:transaction/client "client-id"
:transaction/bank-account "bank-account-id"
:transaction/amount -150.0
:transaction/description-original "Transaction for unpaid invoice link"
:transaction/approval-status :transaction-approval-status/unapproved)
(test-transaction :db/id "transaction-id-feedback"
:transaction/client "client-id"
:transaction/bank-account "bank-account-id"
:transaction/amount 400.0
:transaction/description-original "Transaction for feedback review"
:transaction/approval-status :transaction-approval-status/requires-feedback)
(test-invoice :db/id "invoice-unpaid-id"
(test-transaction :db/id "transaction-id-unpaid"
:transaction/client "client-id"
:transaction/bank-account "bank-account-id"
:transaction/amount -150.0
:transaction/description-original "Transaction for unpaid invoice link"
:transaction/approval-status :transaction-approval-status/unapproved)
(test-transaction :db/id "transaction-id-feedback"
:transaction/client "client-id"
:transaction/bank-account "bank-account-id"
:transaction/amount 400.0
:transaction/description-original "Transaction for feedback review"
:transaction/approval-status :transaction-approval-status/requires-feedback)
(test-invoice :db/id "invoice-unpaid-id"
:invoice/client "client-id"
:invoice/vendor "vendor-id"
:invoice/total 150.0