notid/integreat
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(transactions): port manual bank-transaction import to SSR

Browse Source 
Implement the SSR/alpine/htmx manual transaction import, wiring the
already-declared but unhandled ::external-import-page/parse/import routes.
Mirrors the SSR ledger import: paste the exact master-branch Yodlee
positional-column TSV, review parsed rows in an editable grid with per-row
error/warning badges, and import. Every master validation is preserved and
the existing import.transactions engine is reused unchanged
(via import.manual/import-batch), so core components are untouched.

- New ns auto-ap.ssr.transaction.import (page, paste/parse, editable grid,
  two-tier validation, import handler) + admin-only transactions Import nav.
- Two-tier validation: fixable problems (bad date/amount, unknown client or
  bank-account code, missing fields) are hard errors that block the whole
  batch; inherent skip-conditions (non-POSTED, before start-date/locked,
  already-imported) are warnings computed from the engine's own
  categorize-transaction so the grid preview matches the import result.
- Tests: failing-first Playwright e2e (e2e/transaction-import.spec.ts) plus
  unit/integration coverage (ssr/transaction/import_test.clj, 10 tests).
- Deterministic bank-account code in the e2e seed.

Plan: docs/plans/2026-06-01-001-feat-manual-transaction-import-ssr-plan.md

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bryce
2026-06-01 11:18:28 -07:00
parent b6649a3d1d
commit a1098b28f8
8 changed files with 1056 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
docs/ideation/2026-06-01-manual-transaction-import-ssr-ideation.md Normal file
View File

@@ -0,0 +1,176 @@
---
date: 2026-06-01
topic: manual-transaction-import-ssr
focus: Port the master-branch "manual import transactions" feature to the SSR/alpine/htmx stack, modeled on the SSR ledger import, preserving all validations, starting from a failing e2e test, with minimal core-component change.
mode: repo-grounded
---
# Ideation: Porting Manual Bank-Transaction Import to SSR
## Grounding Context (Codebase)
Three reference points were read in full:
**1. The master feature (what we must reproduce).**
- UI: `src/cljs/auto_ap/views/pages/transactions/manual.cljs` — a re-frame modal titled "Import Transactions" with a single `<textarea>` ("Yodlee manual import table"). User pastes tab-separated Yodlee data, clicks "Import". POSTs `(:data)` as EDN to `/api/transactions/batch-upload`.
- Route/handler: `src/clj/auto_ap/routes/invoices.clj:241` `batch-upload-transactions` → `assert-admin`, then `manual/import-batch (manual/tabulate-data data)`.
- Parsing: `src/clj/auto_ap/import/manual.clj` — `tabulate-data` reads CSV with `\tab` separator, drops the header row, and maps **fixed positional columns**: `[:status :raw-date :description-original :high-level-category nil nil :amount nil nil nil nil nil :bank-account-code :client-code]`.
- Per-row mapping/validation: `manual->transaction` uses `import.manual.common/assoc-or-error` to accumulate errors for: **client lookup** (by bank-account-code → client), **bank-account lookup** (by code), **date parse** (`parse-date`, requires `MM/dd/yyyy`), **amount parse** (`parse-amount`).
- Engine: `import.manual/import-batch` builds lookups, calls `t/start-import-batch :import-source/manual`, applies `t/apply-synthetic-ids` (dedupe key), imports only rows with no `:errors`, returns stats `{:import-batch/imported ... :failed-validation N :sample-error "..."}`.
- Deeper validations live in `src/clj/auto_ap/import/transactions.clj` `categorize-transaction`: status must be `"POSTED"`, transaction date must be after `bank-account/start-date` and after `client/locked-until`, duplicate detection via `:transaction/id` (extant cache), missing client/bank-account/id → `:error`, plus suppression. Synthetic-id (`apply-synthetic-ids`/`synthetic-key`) gives idempotent re-import.
**2. The SSR ledger import (the pattern to emulate).**
- `src/clj/auto_ap/ssr/ledger.clj` implements a **dedicated two-stage page**, not a modal:
- `external-import-text-form*` (~L246): alpine `x-data {clipboard}`, a hidden `text-area` bound `x-model`, an `hx-post` to `::route/external-import-parse` triggered on a `"pasted"` event; a "Load from clipboard" button reads `navigator.clipboard`.
- `external-import-parse` (~L373): re-renders `external-import-form*` with `:just-parsed? true`.
- `external-import-table-form*` (~L117): renders parsed rows into an **editable** `com/data-grid-card` — each cell is a `fc/with-field` text/money input (form-cursor round-trips values + errors); per-row error/warning badge with tooltip; "Import" `hx-post`s to `::route/external-import-import`.
- Validation: malli `parse-form-schema` (~L341) with `:decode/string tsv->import-data` + `:decode/arbitrary` (vector→map) enforces shape (min-length, `clj-date-schema`, `money`, location max 2). Business validation in `add-errors`/`table->entries` (~L380–504) accumulates `[message status]` pairs (`:error`/`:warn`) at entry and line-item level; `flatten-errors` maps them to form-cursor field paths `[[:table idx] message status]`.
- `import-ledger` (~L554): splits good/ignored(warn-only)/bad entries; `throw+` `:field-validation` with `:form-errors` if any bad; upserts hidden vendors; retracts+re-inserts good entries (idempotent via `:journal-entry/external-id`); touches solr; returns `{:successful :ignored :form-errors}`. `external-import-import` wraps it in an `html-response` with an `hx-trigger` notification.
**3. THE KEY DISCOVERY — scaffolding already exists, handlers do not.**
- `src/cljc/auto_ap/routes/transactions.cljc` **already declares** the route names, mirroring ledger exactly:
```
"/external-new" ::external-page
"/external-import-new" {"" ::external-import-page
"/parse" ::external-import-parse
"/import" ::external-import-import}
```
- But `src/clj/auto_ap/ssr/transaction.clj` `key->handler` (L101) wires **only** `page/table/csv/bank-account-filter/bulk-delete` (+ `edit/` + `bulk-code/`). **No handler exists for any `external-import-*` route.** The aside nav (`ssr/components/aside.clj`) wires the *ledger* import nav (`::ledger-routes/external-import-page`) but there is no transaction-import nav entry. So the routes are declared dead ends; the gap is handlers + UI + validation + nav + tests.
**Test conventions.** `test/clj/auto_ap/ssr/ledger_test.clj` is the template: pure-fn tests (`tsv->import-data`, `trim-header`, `line->id`, `flatten-errors`), validation tests (`add-errors` per error type), tx-building tests (`entry->tx`), and end-to-end import tests (`import-ledger` against Datomic via `wrap-setup` + `setup-test-data` + `admin-token`). `test/clj/auto_ap/test-server.clj` is a Playwright/browser e2e harness with `wrap-test-auth` and seeded `setup-test-data`. Existing engine unit tests: `test/clj/auto_ap/import/transactions_test.clj` already covers `categorize-transaction`.
## Topic Axes
- **Input/paste fidelity** — keeping the exact Yodlee positional-column paste payload (req #1)
- **UI flow & surface** — modal vs. dedicated two-stage paste→review→import page (req #2)
- **Validation architecture** — where shape vs. business rules live, and how errors surface (req #3)
- **Core/backend reuse** — how much of `import.transactions` / `import.manual` is reused vs. reimplemented (req #5)
- **Test strategy** — failing-first e2e, then unit coverage (req #4)
## The Central Design Fork
Requirements #1 ("paste the exact same content") and #2 ("follow ledger patterns") pull in slightly different directions, but resolve cleanly:
- **Input format stays identical** — the user still copies the same Yodlee table and pastes the same tab-separated positional columns. We reuse `manual/columns` + `manual/tabulate-data` for the *paste shape*; we do **not** switch to ledger's named-header columns.
- **Everything downstream follows ledger** — dedicated page, clipboard paste, editable review grid, per-row error/warning badges, re-validate loop, notification on import.
The one decision genuinely worth the user's input is **how much of the ledger UX to adopt**: the full editable review grid (idea #5, higher value, more work) vs. a lighter paste→validate→summary page closer to master (still SSR, less scope). Both are presented below as ranked survivors; this is the seed question for brainstorming.
## Ranked Ideas
### 1. Wire the pre-scaffolded routes into a dedicated two-stage SSR import page
**Description:** Implement `external-import-page` / `external-import-parse` / `external-import-import` handlers in a new `src/clj/auto_ap/ssr/transaction/import.clj`, add them to `ssr/transaction.clj` `key->handler` (with the same middleware stack: `wrap-must {:activity :view :subject :transaction}`, `wrap-client-redirect-unauthenticated`, etc.), and add a transaction "Import" nav entry in `aside.clj` parallel to the ledger one. The page structure mirrors `ledger/external-import-page`: breadcrumb → clipboard script → paste form → review table.
**Axis:** UI flow & surface
**Basis:** `direct:` `routes/transactions.cljc` already declares `::external-import-page/parse/import` but `ssr/transaction.clj:101 key->handler` wires none of them; `ssr/ledger.clj:686 key->handler` shows the exact wiring shape to copy.
**Rationale:** The routing contract already exists and is unused — the cheapest, lowest-risk foundation, and it's what makes req #2 ("like the ledger import") literally true at the routing/page level.
**Downsides:** Adds a new namespace; needs a nav entry and middleware parity to avoid auth gaps.
**Confidence:** 95%
**Complexity:** Low
**Status:** Unexplored
### 2. Preserve the exact Yodlee positional paste format (req #1)
**Description:** Reuse `import.manual/columns` + `tabulate-data` (or a malli `:decode/string` wrapper around them) for parsing the pasted TSV, instead of ledger's named-header `tsv->import-data`. Keep the same column positions (`:status :raw-date :description-original ... :amount ... :bank-account-code :client-code`) so users paste identical content.
**Axis:** Input/paste fidelity
**Basis:** `direct:` Requirement #1 ("Paste the exact same type of content as was in the master branch version") + `import/manual.clj:10` fixed `columns` vector; the master textarea label is literally "Yodlee manual import table".
**Rationale:** Users' upstream copy/paste habit (and the Yodlee export shape) is the contract that must not break; the ledger named-header approach would silently change what content is valid.
**Downsides:** Positional columns are brittle if Yodlee changes column order — but that's already the status quo, not a regression.
**Confidence:** 90%
**Complexity:** Low
**Status:** Unexplored
### 3. Reuse the `import.transactions` engine as the backend (req #5)
**Description:** The import handler maps parsed rows → `:transaction/*` maps via the existing `manual->transaction` shape, then drives `import.transactions/start-import-batch :import-source/manual` + `import-transaction!` + `finish!` + `get-stats` — exactly as `import.manual/import-batch` does today. The SSR layer is presentation + pre-validation only; the categorization, rule-matching, payment/deposit clearing, synthetic-id dedupe, and audit-transact paths are untouched.
**Axis:** Core/backend reuse
**Basis:** `direct:` Requirement #5 ("Minimally, if at all, change any core components") + `import/manual.clj:32 import-batch` already encapsulates the whole engine call; `import/transactions.clj` is covered by `transactions_test.clj`.
**Rationale:** This is the battle-tested core. Re-implementing it ledger-style would risk dropping validations (`categorize-transaction`) and duplicate a large, audited code path. Wrapping it keeps core change near zero.
**Downsides:** `import-batch` returns summary stats, not per-row form-cursor errors — so the pre-validation layer (idea #4) must surface row errors *before* the engine runs.
**Confidence:** 90%
**Complexity:** Medium
**Status:** Unexplored
### 4. Two-tier validation: malli shape-parse + a transaction `add-errors` business layer
**Description:** Mirror ledger's split. Tier 1: a malli `parse-form-schema` decodes the pasted TSV and coerces/validates *shape* (date parses as `MM/dd/yyyy`, amount parses, required fields present) — reusing `manual.common/parse-date`/`parse-amount` as `:decode`/predicate fns. Tier 2: a transaction-specific `add-errors`/`table->entries` adds *business* errors/warnings by reusing the predicates already encoded in `categorize-transaction`: client-not-found (`:error`), bank-account-not-found (`:error`), date before `bank-account/start-date` (`:warn`/`:not-ready`), client locked-until (`:warn`), status not `POSTED` (`:not-ready`), already-imported/extant (`:warn`). Errors are `[message status]` pairs surfaced via `flatten-errors` → form-cursor field paths.
**Axis:** Validation architecture
**Basis:** `direct:` Requirement #3 ("every validation maintained… but doesn't have to follow the same structure — make it like the ledger import"). Maps master validations (`manual.clj:23-30`, `transactions.clj:191-225 categorize-transaction`) onto ledger's `add-errors`/`flatten-errors` shape (`ledger.clj:380-519`).
**Rationale:** Gives the ledger-style inline error UX while guaranteeing 1:1 validation parity — each master check becomes an explicit `add-errors` clause, which is also directly unit-testable (one test per error type, like `ledger_test/add-errors-test`).
**Downsides:** Some checks (extant/duplicate, start-date, locked-until) need a DB read at validation time that master does lazily inside the engine — must decide whether to pre-check or let the engine's stats report them. Risk of double-validation drift if engine and pre-validator disagree.
**Confidence:** 80%
**Complexity:** High
**Status:** Unexplored
### 5. Editable review grid with per-row error/warning badges and a re-validate loop
**Description:** After paste+parse, render rows into a `com/data-grid-card` where each field (date, amount, description, bank-account-code, client-code) is an editable `fc/with-field` input, with `com/validated-field` error display and a per-row alert badge+tooltip — exactly like `ledger/external-import-table-form*`. The user can fix a wrong client/bank-account code or date inline and re-submit; only clean rows import, warn-only rows are skipped, error rows block. A "Show table" toggle keeps the default view compact.
**Axis:** UI flow & surface / validation surfacing
**Basis:** `direct:` Requirement #2 ("follow slightly better design patterns, like how the ledger import works"). `ledger.clj:117-244` is the editable-grid implementation to copy; master has no inline correction at all (fire-and-forget modal + summary stats).
**Rationale:** This is the concrete UX upgrade over master and the main reason to model on ledger — turning "paste, pray, read a stats blob" into "paste, see exactly which rows are wrong and why, fix them, import."
**Downsides:** Highest-effort idea; form-cursor round-tripping of an editable grid is the trickiest part of the ledger code. If scope must shrink, a read-only review table + summary (lighter survivor) is the fallback.
**Confidence:** 75%
**Complexity:** High
**Status:** Unexplored
### 6. Preserve idempotent re-import via synthetic-id duplicate detection, surfaced as "already imported"
**Description:** Keep `apply-synthetic-ids`/`synthetic-key` so re-pasting the same export is idempotent (the engine categorizes extant rows as `:extant` and skips them). Surface this in the review grid as a `:warn`-level "already imported" badge rather than silently dropping it, so the user understands why a row didn't import.
**Axis:** Validation architecture
**Basis:** `direct:` `import/transactions.clj:405-421 apply-synthetic-ids` + `categorize-transaction:192-225` extant handling. This is an existing master behavior that req #3 requires us to maintain.
**Rationale:** Duplicate-safety is an easy validation to lose in a port; making it visible (vs. master's opaque stats) is a small, high-trust UX win that costs almost nothing on top of idea #5.
**Downsides:** Requires a DB read of existing `:transaction/id`s at validation time (or reading it back from engine stats post-import).
**Confidence:** 80%
**Complexity:** Low
**Status:** Unexplored
### 7. Start with a failing Playwright e2e, then backfill ledger_test-style unit coverage (req #4)
**Description:** First commit: a failing e2e (against `test-server`) that dev-logs in, navigates dashboard → transactions → Import, pastes a known-good Yodlee TSV into the paste box, asserts parsed rows render, clicks Import, and asserts the transactions appear / a "N imported" notification fires. It fails initially (no handler/nav). Then make it pass incrementally: route+page (idea #1) → parse (#2/#4 tier 1) → review grid (#5) → import via engine (#3) → business validation (#4 tier 2). Backstop with unit tests mirroring `ledger_test.clj`: `tabulate-data`/parse, each `add-errors` validation clause, and an end-to-end `import` test against Datomic. Reuse the existing `transactions_test.clj` `categorize-transaction` coverage as the validation-parity oracle.
**Axis:** Test strategy
**Basis:** `direct:` Requirement #4 ("Write detailed acceptance criteria, and start with a failing e2e test, making it pass over time"). `test-server.clj` already provides the browser harness + test auth; `ledger_test.clj` provides the unit-test template.
**Rationale:** A red e2e pins down the acceptance contract before any handler exists and gives an unambiguous "done" signal; the unit layer locks in validation parity clause-by-clause so req #3 can't silently regress.
**Downsides:** e2e clipboard paste may need a direct `type`-into-textarea path (or a test seam) since `navigator.clipboard.read()` is awkward to drive headless — plan a paste fallback the test can use.
**Confidence:** 85%
**Complexity:** Medium
**Status:** Unexplored
## Draft Acceptance Criteria (seed for brainstorm/plan, per req #4)
**Routing & access**
- [ ] `GET /transactions/external-import-new` renders an import page (admin-gated, same middleware as other transaction routes); 401/redirect for unauthenticated.
- [ ] A "Import" nav entry appears under the transactions section, active on the import route.
**Paste & parse (req #1)**
- [ ] Pasting the exact master Yodlee TSV (same positional columns) parses into the same field set as `manual/tabulate-data`.
- [ ] Header row is dropped; blank rows ignored.
- [ ] `POST …/parse` re-renders the page with a "N rows found" banner and the review table.
**Validation parity (req #3)** — each must produce a visible, row-attributed message:
- [ ] Client not found for bank-account-code → error.
- [ ] Bank account not found by code → error.
- [ ] Date not `MM/dd/yyyy` / unparseable → error.
- [ ] Amount unparseable → error.
- [ ] Status ≠ `POSTED` → not-imported (warn/not-ready).
- [ ] Date before `bank-account/start-date` → not-imported (not-ready).
- [ ] Date on/before `client/locked-until` → not-imported (not-ready).
- [ ] Already-imported (synthetic-id extant) row → skipped, surfaced as warn.
- [ ] Missing client / bank-account / id → error.
**Import (req #5, minimal core change)**
- [ ] `POST …/import` runs the existing `import.transactions` engine via the `:import-source/manual` batch path; no change to `categorize-transaction`/`import-transaction!`/`apply-synthetic-ids`.
- [ ] Only clean rows import; warn-only rows skipped; any error blocks (or imports clean rows + reports errors — match master's "import valid, report failed-validation").
- [ ] Success notification reports counts (imported / skipped / errors), mirroring master's stats.
- [ ] Re-importing the same paste is idempotent (no duplicates).
**Tests (req #4)**
- [ ] A Playwright e2e covering paste → review → import → assert, committed red first, green at the end.
- [ ] Unit tests per validation clause + a Datomic-backed end-to-end import test, modeled on `ledger_test.clj`.
## Failing-First e2e: concrete starting point
Add `test/clj/auto_ap/ssr/transaction/import_test.clj` (unit) and a Playwright spec driven through `test-server`. The e2e is the first artifact and is expected to fail because no `external-import` handler is wired in `ssr/transaction.clj`. Make it green by walking ideas #1 → #2 → #5 → #3 → #4 in that order; the unit suite grows alongside #4.
## Rejection Summary
| # | Idea | Reason Rejected |
|---|------|-----------------|
| 1 | Switch paste format to ledger's named-header columns | Violates req #1 — users paste the exact Yodlee positional export; changing valid input is a silent regression |
| 2 | Keep it a re-frame/CLJS modal | Branch eliminated the CLJS app; contradicts the whole port. (An SSR htmx *modal* was considered but rejected vs. the dedicated page — ledger uses a page and the editable review grid needs the room) |
| 3 | Reimplement validation entirely ledger-style, ignoring `import.transactions` | Duplicates audited `categorize-transaction` logic, risks dropping validations (req #3), and churns core (violates req #5) |
| 4 | Async/streaming import for large pastes | Scope overrun — master is synchronous; YAGNI for the manual paste workflow |
| 5 | Add CSV file-upload alongside paste | Scope overrun — not part of the master manual-import feature |
| 6 | Replace `import-batch` stats with a bespoke result type | Unnecessary core change; the existing stats map already carries imported/failed/sample-error |

275
docs/plans/2026-06-01-001-feat-manual-transaction-import-ssr-plan.md Normal file
View File

@@ -0,0 +1,275 @@
---
date: 2026-06-01
type: feat
status: active
plan_id: 2026-06-01-001
title: "feat: Port manual bank-transaction import to SSR (alpine/htmx)"
depth: standard
origin: docs/ideation/2026-06-01-manual-transaction-import-ssr-ideation.md
---
# feat: Port Manual Bank-Transaction Import to SSR
## Summary
Port the master-branch "manual import transactions" feature into the SSR/alpinejs/htmx stack by implementing the `external-import` handlers that `src/cljc/auto_ap/routes/transactions.cljc` already declares but that no handler currently serves. The feature is a dedicated two-stage page — paste the same Yodlee positional-column TSV → an editable review grid with per-row error/warning badges → import — modeled directly on the SSR ledger import (`src/clj/auto_ap/ssr/ledger.clj`). Validation follows the ledger's `add-errors` shape but preserves every master validation, and the actual write reuses the existing `auto-ap.import.transactions` engine unchanged.
---
## Problem Frame
On `master`, admins import bank transactions by pasting a tab-separated Yodlee export into a re-frame modal (`src/cljs/auto_ap/views/pages/transactions/manual.cljs`) that POSTs EDN to `/api/transactions/batch-upload`. This branch removed the ClojureScript React app and re-implemented the transactions surface server-side, but the manual-import feature was never ported — so admins on this branch cannot manually import transactions at all.
The route names are already scaffolded (`::external-page`, `::external-import-page`, `::external-import-parse`, `::external-import-import` in `routes/transactions.cljc`) but `src/clj/auto_ap/ssr/transaction.clj` wires no handlers for them — they are declared dead ends. The work is to fill that gap with handlers + UI + validation + nav + tests, mirroring the already-shipped ledger import.
---
## Scope Boundaries
**In scope**
- A dedicated SSR import page at `/transaction2/external-import-new` (+ `/parse`, `/import` sub-routes), admin-gated with the same middleware posture as other transaction routes and the ledger import.
- Reuse of the exact Yodlee positional-column paste format (no named-header columns).
- An editable review grid with inline per-field editing and per-row error/warning badges (ledger-style, form-cursor driven).
- Two-tier validation preserving every master validation, with the agreed severity split.
- Import via the existing `auto-ap.import.transactions` engine, block-whole-batch on hard errors.
- A transactions-section "Import" nav entry and an import-result notification.
- A Playwright e2e (committed failing first) plus unit/integration tests modeled on `test/clj/auto_ap/ssr/ledger_test.clj`.
### Deferred to Follow-Up Work
- CSV file upload as an alternative to paste.
- Asynchronous/streaming import for very large pastes.
- Any change to `categorize-transaction` or engine internals.
**Outside this change**
- Named-header column format (rejected — would silently change valid input).
- A bespoke import-result type replacing the engine's stats map.
---
## Key Technical Decisions
1. **Full editable review grid, block-whole-batch on hard error** (from brainstorm). Any remaining hard error blocks the entire import (ledger behavior: `throw+ {:type :field-validation ...}`, re-render the grid with errors highlighted); warn-level rows skip just that row and the rest import. Rationale: with an editable grid, the user can fix fixable problems inline, so "nothing imports until clean-or-skippable" is the coherent contract.
2. **Severity split between fixable errors and inherent warnings.**
- **Hard errors (block, must fix inline):** unparseable/invalid date (must match `MM/dd/yyyy`), unparseable amount, unknown client code (no client for the bank-account-code), unknown bank-account code, missing required fields.
- **Warnings (skip that row, import the rest):** status ≠ `"POSTED"`, transaction date before `bank-account/start-date`, date on/before `client/locked-until`, already-imported (synthetic-id `extant`).
Rationale: fixable problems are correctable by editing a cell; inherent skip-conditions are facts about the data/account that editing cannot change, so they should not block the batch — this also reproduces master's "import valid, report the rest" outcome for those rows.
3. **Reuse the exact Yodlee positional paste format** (req #1). Parse with the master positional `columns` mapping (`auto-ap.import.manual/columns` + `tabulate-data` shape), not ledger's named-header `tsv->import-data`. Rationale: admins paste an unchanged Yodlee export; changing valid input is a silent regression.
4. **Reuse the `import.transactions` engine unchanged** (req #5). The import handler maps reviewed rows → `:transaction/*` maps (the `auto-ap.import.manual/manual->transaction` shape) and drives `start-import-batch :import-source/manual` → `import-transaction!` → `finish!` → `get-stats`, with `apply-synthetic-ids` for dedupe — exactly as `auto-ap.import.manual/import-batch` does today. The SSR layer is presentation + pre-validation only.
5. **Preview/engine parity via shared predicates** (the key design tension). The warn-level conditions shown in the grid before import (`not-ready` from start-date/locked-until, `extant`/already-imported, non-`POSTED`) and the engine's write-time `categorize-transaction` decisions must not drift. Decision: the pre-validation layer computes warn conditions by calling the **same** predicate functions the engine uses (`auto-ap.import.transactions/categorize-transaction` and its inputs — `get-existing` for extant, the bank-account `start-date`/`locked-until` checks), rather than re-deriving parallel logic. The grid is advisory display; the engine remains authoritative at write time, and because both read the same functions they agree. Hard-error (fixable) validations have no engine equivalent and live only in the pre-validation layer / malli schema.
6. **Testable paste path.** The ledger import populates a hidden textarea from `navigator.clipboard` via an alpine `@click`/`paste` handler, which is awkward to drive in headless Playwright. Decision: keep the "Load from clipboard" affordance, but ensure the paste textarea is fillable and that a `pasted`/`change` trigger fires the parse `hx-post`, so the e2e can set the value and dispatch the event without the clipboard API. (Implementation detail of how the trigger is wired is deferred to execution.)
---
## High-Level Technical Design
Two-stage flow mirroring `ssr/ledger.clj`, on the transactions surface:
```
GET /transaction2/external-import-new -> external-import-page (paste form + empty review area)
POST /transaction2/external-import-new/parse -> external-import-parse (decode TSV -> validate -> render editable grid)
POST /transaction2/external-import-new/import -> external-import-import (re-validate -> if any hard error: re-render grid (blocked);
else run import.transactions engine on clean rows,
skip warn rows, return notification with stats)
```
*This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
Validation is two-tier:
- **Tier 1 (shape, hard errors):** a malli parse-form-schema decodes the pasted TSV positionally (reusing the master column order) and coerces/flags shape problems — date parses as `MM/dd/yyyy`, amount parses, required fields present.
- **Tier 2 (business):** a transaction `add-errors`/`table->entries` pass attaches `[message status]` pairs (`:error` / `:warn`) per row, with the hard/warn split from Decision 2, computing warn conditions from the shared engine predicates (Decision 5). `flatten-errors` maps them onto form-cursor field paths for the editable grid.
---
## Implementation Units
Build order is **failing-e2e-first** (req #4): U1 lands the red acceptance test, then U2–U7 turn it green incrementally. Each feature-bearing unit also grows the unit-test suite in `test/clj/auto_ap/ssr/transaction/import_test.clj`.
### U1. Failing e2e acceptance test + deterministic import seed
**Goal:** Commit the end-to-end acceptance test (expected to fail) that defines "done", plus the deterministic test fixture it needs.
**Requirements:** req #4; advances acceptance criteria AC-1, AC-2, AC-9, AC-10.
**Dependencies:** none.
**Files:**
- `e2e/transaction-import.spec.ts` (new)
- `test/clj/auto_ap/test_server.clj` (modify `seed-test-data` to give the seeded bank account a **fixed** `:bank-account/code`, e.g. `"TEST-CHK"`, since `test-bank-account` otherwise assigns a random code)
**Approach:** Mirror `e2e/transaction-navigation.spec.ts` conventions (`x-clients: "mine"` header, `page.goto`, locators). The spec navigates to `/transaction2/external-import-new`, fills the paste box with a known-good Yodlee TSV whose bank-account-code/client-code match the seed (`TEST` client, `TEST-CHK` bank account), triggers parse, asserts parsed rows render in the review grid, clicks Import, and asserts a success notification with an imported count and that the imported transaction is visible on `/transaction2`. Include a second scenario pasting a row with an unknown client code and asserting a blocking error badge + that nothing imports. Drive paste by filling the textarea and dispatching the parse trigger (Decision 6), not the clipboard API.
**Patterns to follow:** `e2e/transaction-navigation.spec.ts`, `e2e/bulk-code-transactions.spec.ts`; seed shape in `test/clj/auto_ap/test_server.clj` `seed-test-data`.
**Test scenarios:**
- Covers AE/AC-1, AC-2: paste valid TSV → rows render → import → "N imported" notification → transaction appears on the list page.
- Covers AC-9: paste TSV with an unknown client code → row shows a blocking error badge, Import is blocked, no transaction created.
- Edge: empty paste → no rows / friendly empty state (assert no crash).
**Verification:** `npx playwright test e2e/transaction-import.spec.ts` runs and **fails** at this unit (no handler yet); the seed change does not break existing e2e specs (`npx playwright test` green except the new file).
**Execution note:** Start red. This is the acceptance contract; do not weaken it to pass — make U2–U7 satisfy it.
### U2. Wire routes and render the import page shell
**Goal:** Make `/transaction2/external-import-new` serve a real page with the correct admin middleware; wire `parse`/`import` routes to placeholder handlers.
**Requirements:** AC-1, AC-12 (auth); req #2.
**Dependencies:** U1.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (new — namespace for the import handlers)
- `src/clj/auto_ap/ssr/transaction.clj` (merge the new `key->handler` entries into the existing map at the `key->handler` def)
**Approach:** Create `external-import-page` returning a `base-page` + `com/page` with breadcrumb ("Transactions" → "Import"), the clipboard helper script, and a forms container (initially just the paste form placeholder). Wire `::route/external-import-page`, `::route/external-import-parse`, `::route/external-import-import` into the transaction `key->handler` with the same middleware chain ledger uses for its import routes (`wrap-schema-enforce`/`wrap-form-4xx-2`/`wrap-schema-decode`/`wrap-nested-form-params` on parse/import) under the transaction page middleware (`wrap-must {:activity :import :subject :transaction}` analogous to ledger's `:subject :ledger`, `wrap-client-redirect-unauthenticated`). Confirm the correct `:activity`/`:subject` against the permissions model.
**Patterns to follow:** `src/clj/auto_ap/ssr/ledger.clj` `external-import-page` and `key->handler` (~lines 276–318, 686–718); `src/clj/auto_ap/ssr/transaction.clj` existing `key->handler` (~line 101).
**Test scenarios:**
- Happy path: `GET` the page as admin → 200, renders the paste form container.
- Error/auth: unauthenticated request → redirect/401 per `wrap-client-redirect-unauthenticated`.
**Verification:** Page loads at the route in the running app and in `test_server`; the e2e gets past navigation (still fails later in the flow).
### U3. Paste + parse using the master positional column format
**Goal:** Parse the pasted Yodlee TSV (exact master columns) into rows and render them; wire the paste form's `pasted`-triggered `hx-post` to the parse handler.
**Requirements:** req #1, req #2; AC-1, AC-3, AC-4.
**Dependencies:** U2.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `external-import-text-form*`, `external-import-parse`, the parse malli schema, and a positional `tsv->rows` decode)
- `test/clj/auto_ap/ssr/transaction/import_test.clj` (new)
**Approach:** Reuse the master column order from `auto-ap.import.manual/columns` and `tabulate-data` (CSV read with `\tab`, drop header) to map positional columns. Define a malli `parse-form-schema` (ledger-style) whose `:table` field uses a `:decode/string` that runs the positional parse and a per-row `:decode/arbitrary` to build row maps; encode Tier-1 shape constraints (date `MM/dd/yyyy`, amount parses, required fields) reusing `auto-ap.import.manual.common/parse-date`/`parse-amount` semantics. `external-import-parse` re-renders the forms fragment with `:just-parsed? true`. Keep the paste textarea fillable and fire the parse trigger on a `pasted`/`change` event (Decision 6).
**Patterns to follow:** `ssr/ledger.clj` `external-import-text-form*`, `external-import-parse`, `tsv->import-data`, `parse-form-schema` (~lines 246–375); `import/manual.clj` `columns`/`tabulate-data`; `import/manual/common.clj` `parse-date`/`parse-amount`.
**Test scenarios:**
- Happy path: a known Yodlee TSV string decodes to the expected row count with the expected field keys/values (positional mapping correct).
- Header handling: first row dropped; blank rows ignored.
- Edge: amount with currency formatting parses; amount unparseable flagged at Tier 1.
- Edge: date not `MM/dd/yyyy` flagged at Tier 1; valid date parses.
- Covers AC-3: pasting the exact master column layout yields the same field set master's `tabulate-data` produced.
**Verification:** After paste, the parsed rows render (read-only at this unit is acceptable); parse unit tests green.
### U4. Editable review grid with per-row error/warning badges
**Goal:** Render parsed rows into an editable `data-grid` where each field is editable and per-row error/warning badges show, with a "Show table" toggle and an Import button.
**Requirements:** req #2; AC-5, AC-6.
**Dependencies:** U3.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `external-import-table-form*`, `external-import-form*`)
**Approach:** Mirror `ledger/external-import-table-form*` using form-cursor (`fc/start-form`, `fc/with-field`, `fc/cursor-map`, `fc/field-value`/`field-name`/`field-errors`) and `com/data-grid-card` / `com/validated-field` / `com/text-input` / `com/money-input`. Columns reflect the transaction row shape (date, description, amount, bank-account-code, client-code, status). Per-row badge summarizes that row's error/warn state with a tooltip listing messages (red for `:error`, yellow for `:warn`). A parsed-summary banner shows row count + error/warning pill counts. Values round-trip on re-submit so inline edits persist.
**Patterns to follow:** `ssr/ledger.clj` `external-import-table-form*` and `external-import-form*` (~lines 117–274).
**Test scenarios:**
- Test expectation: none for pure rendering structure beyond what U5 exercises — but include: rows with no errors render without a badge; rows with errors render a red badge; rows with only warnings render a yellow badge (assert via the rendered hiccup/markup in a handler-level test once U5 attaches errors).
**Verification:** Parsed grid is visibly editable; badges appear once U5 attaches errors; e2e can see rows.
### U5. Two-tier validation preserving every master validation
**Goal:** Attach hard-error and warning statuses to rows per the severity split, reusing the engine's predicates for the warn conditions so the preview matches the engine.
**Requirements:** req #3, req #5 (Decision 5); AC-7, AC-8, AC-9.
**Dependencies:** U3 (Tier 1 shape errors), U4 (badges to display them).
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `add-errors`, `table->entries`, `flatten-errors`, `entry-error-types` analogues)
- `test/clj/auto_ap/ssr/transaction/import_test.clj` (extend)
**Approach:** Build a transaction `add-errors` that, given lookups (client-by-bank-account-code, bank-account-by-code, bank-account `start-date`/`locked-until`, existing transaction ids), assigns:
- **Hard errors:** unknown client code, unknown bank-account code, missing required fields. (Tier-1 date/amount errors already present from U3.)
- **Warnings:** status ≠ `POSTED`; date before `bank-account/start-date`; date on/before `client/locked-until`; already-imported (synthetic-id present in existing ids).
Compute the warn conditions by calling the same functions the engine uses — `auto-ap.import.transactions/categorize-transaction` (and its inputs `get-existing`, the `apply-synthetic-ids` key) — rather than parallel logic (Decision 5). `flatten-errors` maps `[message status]` onto form-cursor field paths so badges render against the right rows. Map every master validation explicitly (see `import/manual.clj manual->transaction` and `import/transactions.clj categorize-transaction`).
**Patterns to follow:** `ssr/ledger.clj` `add-errors`/`table->entries`/`flatten-errors`/`entry-error-types` (~lines 380–523); `import/transactions.clj` `categorize-transaction`/`get-existing`/`apply-synthetic-ids`.
**Test scenarios (one per validation, modeled on `ledger_test/add-errors-test`):**
- Hard error: unknown client code → `:error` with a clear message.
- Hard error: unknown bank-account code → `:error`.
- Hard error: missing required field → `:error`.
- (Tier 1) invalid date / unparseable amount → `:error`.
- Warning: status ≠ `POSTED` → `:warn`, row skipped.
- Warning: date before `bank-account/start-date` → `:warn`.
- Warning: date on/before `client/locked-until` → `:warn`.
- Warning: already-imported (extant synthetic id) → `:warn`.
- Parity: a row the grid marks clean is categorized `:import` by `categorize-transaction`; a row marked warn-skip is categorized to the matching non-`:import` action (assert grid preview agrees with engine).
- Pass-through: a fully valid row has no errors/warnings.
**Verification:** Validation unit tests green; badges reflect the correct severities in the grid.
### U6. Import via the existing engine, block-on-error, with notification
**Goal:** Implement `external-import-import`: block the whole batch if any hard error remains; otherwise run the `import.transactions` engine on clean rows (skipping warn rows) and return a result notification.
**Requirements:** req #5, Decisions 1 & 4; AC-2, AC-9, AC-10, AC-11.
**Dependencies:** U5.
**Files:**
- `src/clj/auto_ap/ssr/transaction/import.clj` (add `rows->transactions`, `import-transactions`, `external-import-import`)
**Approach:** Re-validate submitted (possibly edited) rows via U5. If any `:error` rows remain, `throw+ {:type :field-validation :form-errors ... :form-params ...}` and re-render the grid with errors (the `wrap-form-4xx-2` middleware handles re-render) — nothing imports. Otherwise map clean rows → `:transaction/*` maps using the `auto-ap.import.manual/manual->transaction` shape, apply `apply-synthetic-ids`, then drive `start-import-batch :import-source/manual` → `import-transaction!` (per row) → `finish!` → `get-stats`. Warn-only rows are excluded from the engine input (skipped). Return `html-response` re-rendering the form with an `hx-trigger` notification reporting counts from the engine stats (imported / skipped / not-ready / extant), mirroring master's stats surface.
**Patterns to follow:** `ssr/ledger.clj` `import-ledger` + `external-import-import` (~lines 554–684); `import/manual.clj` `import-batch` (engine driving); `import/manual.clj` `manual->transaction`.
**Test scenarios (modeled on `ledger_test` `import-ledger-*` tests, against Datomic via `wrap-setup`/`setup-test-data`/`admin-token`):**
- Happy path: all-clean batch → engine imports all rows; stats report the imported count; transactions exist in the DB afterward.
- Block-on-error: a batch with one hard-error row → throws `:field-validation`; **no** transactions are created (assert DB unchanged).
- Warning skip: a batch with one warn-only row (e.g., non-`POSTED`) and clean rows → clean rows import, warn row skipped, stats reflect the skip.
- Idempotency: importing the same paste twice → second run imports 0 new (extant/synthetic-id dedupe); no duplicates.
- Integration: imported transaction carries `:import-source/manual` and is categorized/coded by the engine as it would be for any import (engine unchanged).
**Verification:** Import unit/integration tests green; the e2e's import step succeeds and the transaction appears on the list page.
### U7. Transactions "Import" nav entry + final polish
**Goal:** Add an "Import" entry to the transactions section nav (parallel to the ledger import nav) and finish the parsed-summary banner / notification copy.
**Requirements:** req #2; AC-1, AC-11.
**Dependencies:** U2 (route exists), U6 (notification exists).
**Files:**
- `src/clj/auto_ap/ssr/components/aside.clj` (add a transactions "Import" nav button + mark active on `::transaction-routes/external-import-page`)
**Approach:** Mirror the ledger import nav entry in `aside.clj` — add a sub-menu button under the transactions section linking to `::transaction-routes/external-import-page`, active-highlighted on that matched route. Confirm the banner shows row counts + error/warning pills (from U4) and the success notification copy matches the engine stats.
**Patterns to follow:** `ssr/components/aside.clj` ledger import nav (~lines 360–366) and the transactions sub-menu (~lines 285–298).
**Test scenarios:**
- Test expectation: none (navigation markup) — covered indirectly by the e2e navigating via the nav link; optionally assert the nav button renders with the correct href on the import route.
**Verification:** Full `e2e/transaction-import.spec.ts` passes; nav link is present and active on the import page.
---
## Acceptance Criteria
**Routing & access**
- **AC-1.** `GET /transaction2/external-import-new` renders the import page for an admin; an "Import" nav entry under the transactions section links to it and is active there.
- **AC-12.** Unauthenticated access redirects/401s per the standard transaction-route middleware.
**Paste & parse (req #1)**
- **AC-3.** Pasting the exact master Yodlee positional TSV parses into the same field set as `auto-ap.import.manual/tabulate-data`; the header row is dropped and blank rows ignored.
- **AC-4.** `POST .../parse` re-renders the page with a "N rows found" banner and the review grid.
**Review grid (req #2)**
- **AC-5.** Parsed rows render in an editable grid; each field is editable and inline edits persist across re-submit.
- **AC-6.** Each row shows an error badge (red) when it has a hard error, a warning badge (yellow) when it has only warnings, and no badge when clean; badges list messages on hover.
**Validation parity (req #3)** — each produces a visible, row-attributed message:
- **AC-7.** Hard errors block: client not found, bank account not found, date not `MM/dd/yyyy`, amount unparseable, missing required field.
- **AC-8.** Warnings skip just that row: status ≠ `POSTED`, date before `bank-account/start-date`, date on/before `client/locked-until`, already-imported.
- **AC-9.** With any remaining hard error, clicking Import blocks the whole batch (nothing imports) and re-renders the grid with errors highlighted.
**Import (req #5)**
- **AC-2.** `POST .../import` imports the clean rows via the existing `import.transactions` engine on the `:import-source/manual` batch path; the success notification reports counts (imported / skipped / not-ready / extant).
- **AC-10.** Re-importing the same paste is idempotent — no duplicate transactions (synthetic-id dedupe preserved).
- **AC-11.** `categorize-transaction` and the engine internals are unchanged by this work.
**Tests (req #4)**
- The Playwright e2e `e2e/transaction-import.spec.ts` exists, was committed failing first, and passes at the end.
- Unit/integration tests in `test/clj/auto_ap/ssr/transaction/import_test.clj` cover each validation clause and the end-to-end import flow against Datomic.
---
## Test Strategy
- **e2e (Playwright):** `e2e/transaction-import.spec.ts`, driven through `test/clj/auto_ap/test_server.clj` (real routes, injected test auth). Committed red in U1, green by U7.
- **Unit/integration (clojure.test):** `test/clj/auto_ap/ssr/transaction/import_test.clj`, modeled on `test/clj/auto_ap/ssr/ledger_test.clj` — pure parse/format tests, one validation test per clause, and Datomic-backed import tests via `wrap-setup` / `setup-test-data` / `admin-token`. Run with `clj-nrepl-eval -p PORT "(clojure.test/run-tests 'auto-ap.ssr.transaction.import-test)"` per AGENTS.md (preferred over `lein test`).
- **Validation-parity oracle:** existing `test/clj/auto_ap/import/transactions_test.clj` (`categorize-transaction`) backs Decision 5 — the warn-condition predicates the grid reuses are already under test.
---
## System-Wide Impact
- **Editing discipline (AGENTS.md):** all Clojure edits go through the clojure-mcp editing tools (or `@clojure-author`), not raw file edits; use `clojure-eval`/`clj-nrepl-eval` to compile-check and run tests. Run `lein cljfmt fix` before committing; `clj-paren-repair` on any file that won't compile.
- **Shared component reuse:** the feature composes existing `ssr/components` (`data-grid-card`, `validated-field`, `text-input`, `money-input`, `button`, `checkbox`, `errors`, `pill`, `form-errors`) and `ssr/form-cursor` — no core-component changes expected (req #5). If a component genuinely needs a new option, prefer an additive, backward-compatible change and flag it.
- **Test fixture change:** giving the seeded bank account a deterministic `:bank-account/code` in `test_server.clj` could affect other e2e specs that assume the random code; U1 verifies the existing suite stays green.
- **Permissions:** confirm the `wrap-must` `:activity`/`:subject` for the import routes matches the permission model (ledger uses `{:activity :import :subject :ledger}`); use the transaction equivalent.
---
## Risks & Mitigations
- **Preview/engine drift (highest risk).** Mitigated by Decision 5 — share the engine's predicates for warn conditions; the parity test in U5 asserts the grid and `categorize-transaction` agree.
- **Headless clipboard paste.** Mitigated by Decision 6 — fillable textarea + explicit parse trigger so the e2e never needs `navigator.clipboard`.
- **form-cursor round-tripping of an editable grid** is the trickiest ledger mechanic to copy; mitigate by mirroring `external-import-table-form*` closely and testing edit-persist-on-resubmit early (U4).
- **Positional column brittleness** is inherited from master (Yodlee column order); not a regression, and out of scope to fix here.
---
## Deferred Implementation Notes (execution-time unknowns)
- Exact helper/function names in the new `import.clj` namespace.
- The precise malli `:decode` wiring for positional parsing (reuse vs. thin wrapper around `tabulate-data`).
- The exact `:activity`/`:subject` keyword for the import-route `wrap-must` (verify against the permissions model).
- The seeded bank-account code value and whether any existing e2e needs adjustment after making it deterministic.
- Final notification/banner copy.