integreat/docs/plans/2026-06-01-001-feat-manual-transaction-import-ssr-plan.md

date, type, status, plan_id, title, depth, origin

date	type	status	plan_id	title	depth	origin
2026-06-01	feat	active	2026-06-01-001	feat: Port manual bank-transaction import to SSR (alpine/htmx)	standard	docs/ideation/2026-06-01-manual-transaction-import-ssr-ideation.md

feat: Port Manual Bank-Transaction Import to SSR

Summary

Port the master-branch "manual import transactions" feature into the SSR/alpinejs/htmx stack by implementing the external-import handlers that src/cljc/auto_ap/routes/transactions.cljc already declares but that no handler currently serves. The feature is a dedicated two-stage page — paste the same Yodlee positional-column TSV → an editable review grid with per-row error/warning badges → import — modeled directly on the SSR ledger import (src/clj/auto_ap/ssr/ledger.clj). Validation follows the ledger's add-errors shape but preserves every master validation, and the actual write reuses the existing auto-ap.import.transactions engine unchanged.

---

Problem Frame

On master, admins import bank transactions by pasting a tab-separated Yodlee export into a re-frame modal (src/cljs/auto_ap/views/pages/transactions/manual.cljs) that POSTs EDN to /api/transactions/batch-upload. This branch removed the ClojureScript React app and re-implemented the transactions surface server-side, but the manual-import feature was never ported — so admins on this branch cannot manually import transactions at all.

The route names are already scaffolded (::external-page, ::external-import-page, ::external-import-parse, ::external-import-import in routes/transactions.cljc) but src/clj/auto_ap/ssr/transaction.clj wires no handlers for them — they are declared dead ends. The work is to fill that gap with handlers + UI + validation + nav + tests, mirroring the already-shipped ledger import.

---

Scope Boundaries

In scope

• A dedicated SSR import page at /transaction2/external-import-new (+ /parse, /import sub-routes), admin-gated with the same middleware posture as other transaction routes and the ledger import.
• Reuse of the exact Yodlee positional-column paste format (no named-header columns).
• An editable review grid with inline per-field editing and per-row error/warning badges (ledger-style, form-cursor driven).
• Two-tier validation preserving every master validation, with the agreed severity split.
• Import via the existing auto-ap.import.transactions engine, block-whole-batch on hard errors.
• A transactions-section "Import" nav entry and an import-result notification.
• A Playwright e2e (committed failing first) plus unit/integration tests modeled on test/clj/auto_ap/ssr/ledger_test.clj.

Deferred to Follow-Up Work

• CSV file upload as an alternative to paste.
• Asynchronous/streaming import for very large pastes.
• Any change to categorize-transaction or engine internals.

Outside this change

• Named-header column format (rejected — would silently change valid input).
• A bespoke import-result type replacing the engine's stats map.

---

Key Technical Decisions

1. Full editable review grid, block-whole-batch on hard error (from brainstorm). Any remaining hard error blocks the entire import (ledger behavior: throw+ {:type :field-validation ...}, re-render the grid with errors highlighted); warn-level rows skip just that row and the rest import. Rationale: with an editable grid, the user can fix fixable problems inline, so "nothing imports until clean-or-skippable" is the coherent contract.

2. Severity split between fixable errors and inherent warnings.

   • Hard errors (block, must fix inline): unparseable/invalid date (must match MM/dd/yyyy), unparseable amount, unknown client code (no client for the bank-account-code), unknown bank-account code, missing required fields.
   • Warnings (skip that row, import the rest): status ≠ "POSTED", transaction date before bank-account/start-date, date on/before client/locked-until, already-imported (synthetic-id extant). Rationale: fixable problems are correctable by editing a cell; inherent skip-conditions are facts about the data/account that editing cannot change, so they should not block the batch — this also reproduces master's "import valid, report the rest" outcome for those rows.

3. Reuse the exact Yodlee positional paste format (req #1). Parse with the master positional columns mapping (auto-ap.import.manual/columns + tabulate-data shape), not ledger's named-header tsv->import-data. Rationale: admins paste an unchanged Yodlee export; changing valid input is a silent regression.

4. Reuse the import.transactions engine unchanged (req #5). The import handler maps reviewed rows → :transaction/* maps (the auto-ap.import.manual/manual->transaction shape) and drives start-import-batch :import-source/manual → import-transaction! → finish! → get-stats, with apply-synthetic-ids for dedupe — exactly as auto-ap.import.manual/import-batch does today. The SSR layer is presentation + pre-validation only.

5. Preview/engine parity via shared predicates (the key design tension). The warn-level conditions shown in the grid before import (not-ready from start-date/locked-until, extant/already-imported, non-POSTED) and the engine's write-time categorize-transaction decisions must not drift. Decision: the pre-validation layer computes warn conditions by calling the same predicate functions the engine uses (auto-ap.import.transactions/categorize-transaction and its inputs — get-existing for extant, the bank-account start-date/locked-until checks), rather than re-deriving parallel logic. The grid is advisory display; the engine remains authoritative at write time, and because both read the same functions they agree. Hard-error (fixable) validations have no engine equivalent and live only in the pre-validation layer / malli schema.

6. Testable paste path. The ledger import populates a hidden textarea from navigator.clipboard via an alpine @click/paste handler, which is awkward to drive in headless Playwright. Decision: keep the "Load from clipboard" affordance, but ensure the paste textarea is fillable and that a pasted/change trigger fires the parse hx-post, so the e2e can set the value and dispatch the event without the clipboard API. (Implementation detail of how the trigger is wired is deferred to execution.)

---

High-Level Technical Design

Two-stage flow mirroring ssr/ledger.clj, on the transactions surface:

GET  /transaction2/external-import-new          -> external-import-page      (paste form + empty review area)
POST /transaction2/external-import-new/parse     -> external-import-parse      (decode TSV -> validate -> render editable grid)
POST /transaction2/external-import-new/import    -> external-import-import     (re-validate -> if any hard error: re-render grid (blocked);
                                                                                else run import.transactions engine on clean rows,
                                                                                skip warn rows, return notification with stats)

This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.

Validation is two-tier:

• Tier 1 (shape, hard errors): a malli parse-form-schema decodes the pasted TSV positionally (reusing the master column order) and coerces/flags shape problems — date parses as MM/dd/yyyy, amount parses, required fields present.
• Tier 2 (business): a transaction add-errors/table->entries pass attaches [message status] pairs (:error / :warn) per row, with the hard/warn split from Decision 2, computing warn conditions from the shared engine predicates (Decision 5). flatten-errors maps them onto form-cursor field paths for the editable grid.

---

Implementation Units

Build order is failing-e2e-first (req #4): U1 lands the red acceptance test, then U2–U7 turn it green incrementally. Each feature-bearing unit also grows the unit-test suite in test/clj/auto_ap/ssr/transaction/import_test.clj.

U1. Failing e2e acceptance test + deterministic import seed

Goal: Commit the end-to-end acceptance test (expected to fail) that defines "done", plus the deterministic test fixture it needs. Requirements: req #4; advances acceptance criteria AC-1, AC-2, AC-9, AC-10. Dependencies: none. Files:

• e2e/transaction-import.spec.ts (new)
• test/clj/auto_ap/test_server.clj (modify seed-test-data to give the seeded bank account a fixed :bank-account/code, e.g. "TEST-CHK", since test-bank-account otherwise assigns a random code) Approach: Mirror e2e/transaction-navigation.spec.ts conventions (x-clients: "mine" header, page.goto, locators). The spec navigates to /transaction2/external-import-new, fills the paste box with a known-good Yodlee TSV whose bank-account-code/client-code match the seed (TEST client, TEST-CHK bank account), triggers parse, asserts parsed rows render in the review grid, clicks Import, and asserts a success notification with an imported count and that the imported transaction is visible on /transaction2. Include a second scenario pasting a row with an unknown client code and asserting a blocking error badge + that nothing imports. Drive paste by filling the textarea and dispatching the parse trigger (Decision 6), not the clipboard API. Patterns to follow: e2e/transaction-navigation.spec.ts, e2e/bulk-code-transactions.spec.ts; seed shape in test/clj/auto_ap/test_server.clj seed-test-data. Test scenarios:
• Covers AE/AC-1, AC-2: paste valid TSV → rows render → import → "N imported" notification → transaction appears on the list page.
• Covers AC-9: paste TSV with an unknown client code → row shows a blocking error badge, Import is blocked, no transaction created.
• Edge: empty paste → no rows / friendly empty state (assert no crash). Verification: npx playwright test e2e/transaction-import.spec.ts runs and fails at this unit (no handler yet); the seed change does not break existing e2e specs (npx playwright test green except the new file). Execution note: Start red. This is the acceptance contract; do not weaken it to pass — make U2–U7 satisfy it.

U2. Wire routes and render the import page shell

Goal: Make /transaction2/external-import-new serve a real page with the correct admin middleware; wire parse/import routes to placeholder handlers. Requirements: AC-1, AC-12 (auth); req #2. Dependencies: U1. Files:

• src/clj/auto_ap/ssr/transaction/import.clj (new — namespace for the import handlers)
• src/clj/auto_ap/ssr/transaction.clj (merge the new key->handler entries into the existing map at the key->handler def) Approach: Create external-import-page returning a base-page + com/page with breadcrumb ("Transactions" → "Import"), the clipboard helper script, and a forms container (initially just the paste form placeholder). Wire ::route/external-import-page, ::route/external-import-parse, ::route/external-import-import into the transaction key->handler with the same middleware chain ledger uses for its import routes (wrap-schema-enforce/wrap-form-4xx-2/wrap-schema-decode/wrap-nested-form-params on parse/import) under the transaction page middleware (wrap-must {:activity :import :subject :transaction} analogous to ledger's :subject :ledger, wrap-client-redirect-unauthenticated). Confirm the correct :activity/:subject against the permissions model. Patterns to follow: src/clj/auto_ap/ssr/ledger.clj external-import-page and key->handler (~lines 276–318, 686–718); src/clj/auto_ap/ssr/transaction.clj existing key->handler (~line 101). Test scenarios:
• Happy path: GET the page as admin → 200, renders the paste form container.
• Error/auth: unauthenticated request → redirect/401 per wrap-client-redirect-unauthenticated. Verification: Page loads at the route in the running app and in test_server; the e2e gets past navigation (still fails later in the flow).

U3. Paste + parse using the master positional column format

Goal: Parse the pasted Yodlee TSV (exact master columns) into rows and render them; wire the paste form's pasted-triggered hx-post to the parse handler. Requirements: req #1, req #2; AC-1, AC-3, AC-4. Dependencies: U2. Files:

• src/clj/auto_ap/ssr/transaction/import.clj (add external-import-text-form*, external-import-parse, the parse malli schema, and a positional tsv->rows decode)
• test/clj/auto_ap/ssr/transaction/import_test.clj (new) Approach: Reuse the master column order from auto-ap.import.manual/columns and tabulate-data (CSV read with \tab, drop header) to map positional columns. Define a malli parse-form-schema (ledger-style) whose :table field uses a :decode/string that runs the positional parse and a per-row :decode/arbitrary to build row maps; encode Tier-1 shape constraints (date MM/dd/yyyy, amount parses, required fields) reusing auto-ap.import.manual.common/parse-date/parse-amount semantics. external-import-parse re-renders the forms fragment with :just-parsed? true. Keep the paste textarea fillable and fire the parse trigger on a pasted/change event (Decision 6). Patterns to follow: ssr/ledger.clj external-import-text-form*, external-import-parse, tsv->import-data, parse-form-schema (~lines 246–375); import/manual.clj columns/tabulate-data; import/manual/common.clj parse-date/parse-amount. Test scenarios:
• Happy path: a known Yodlee TSV string decodes to the expected row count with the expected field keys/values (positional mapping correct).
• Header handling: first row dropped; blank rows ignored.
• Edge: amount with currency formatting parses; amount unparseable flagged at Tier 1.
• Edge: date not MM/dd/yyyy flagged at Tier 1; valid date parses.
• Covers AC-3: pasting the exact master column layout yields the same field set master's tabulate-data produced. Verification: After paste, the parsed rows render (read-only at this unit is acceptable); parse unit tests green.

U4. Editable review grid with per-row error/warning badges

Goal: Render parsed rows into an editable data-grid where each field is editable and per-row error/warning badges show, with a "Show table" toggle and an Import button. Requirements: req #2; AC-5, AC-6. Dependencies: U3. Files:

• src/clj/auto_ap/ssr/transaction/import.clj (add external-import-table-form*, external-import-form*) Approach: Mirror ledger/external-import-table-form* using form-cursor (fc/start-form, fc/with-field, fc/cursor-map, fc/field-value/field-name/field-errors) and com/data-grid-card / com/validated-field / com/text-input / com/money-input. Columns reflect the transaction row shape (date, description, amount, bank-account-code, client-code, status). Per-row badge summarizes that row's error/warn state with a tooltip listing messages (red for :error, yellow for :warn). A parsed-summary banner shows row count + error/warning pill counts. Values round-trip on re-submit so inline edits persist. Patterns to follow: ssr/ledger.clj external-import-table-form* and external-import-form* (~lines 117–274). Test scenarios:
• Test expectation: none for pure rendering structure beyond what U5 exercises — but include: rows with no errors render without a badge; rows with errors render a red badge; rows with only warnings render a yellow badge (assert via the rendered hiccup/markup in a handler-level test once U5 attaches errors). Verification: Parsed grid is visibly editable; badges appear once U5 attaches errors; e2e can see rows.

U5. Two-tier validation preserving every master validation

Goal: Attach hard-error and warning statuses to rows per the severity split, reusing the engine's predicates for the warn conditions so the preview matches the engine. Requirements: req #3, req #5 (Decision 5); AC-7, AC-8, AC-9. Dependencies: U3 (Tier 1 shape errors), U4 (badges to display them). Files:

• src/clj/auto_ap/ssr/transaction/import.clj (add add-errors, table->entries, flatten-errors, entry-error-types analogues)
• test/clj/auto_ap/ssr/transaction/import_test.clj (extend) Approach: Build a transaction add-errors that, given lookups (client-by-bank-account-code, bank-account-by-code, bank-account start-date/locked-until, existing transaction ids), assigns:
• Hard errors: unknown client code, unknown bank-account code, missing required fields. (Tier-1 date/amount errors already present from U3.)
• Warnings: status ≠ POSTED; date before bank-account/start-date; date on/before client/locked-until; already-imported (synthetic-id present in existing ids). Compute the warn conditions by calling the same functions the engine uses — auto-ap.import.transactions/categorize-transaction (and its inputs get-existing, the apply-synthetic-ids key) — rather than parallel logic (Decision 5). flatten-errors maps [message status] onto form-cursor field paths so badges render against the right rows. Map every master validation explicitly (see import/manual.clj manual->transaction and import/transactions.clj categorize-transaction). Patterns to follow: ssr/ledger.clj add-errors/table->entries/flatten-errors/entry-error-types (~lines 380–523); import/transactions.clj categorize-transaction/get-existing/apply-synthetic-ids. Test scenarios (one per validation, modeled on ledger_test/add-errors-test):
• Hard error: unknown client code → :error with a clear message.
• Hard error: unknown bank-account code → :error.
• Hard error: missing required field → :error.
• (Tier 1) invalid date / unparseable amount → :error.
• Warning: status ≠ POSTED → :warn, row skipped.
• Warning: date before bank-account/start-date → :warn.
• Warning: date on/before client/locked-until → :warn.
• Warning: already-imported (extant synthetic id) → :warn.
• Parity: a row the grid marks clean is categorized :import by categorize-transaction; a row marked warn-skip is categorized to the matching non-:import action (assert grid preview agrees with engine).
• Pass-through: a fully valid row has no errors/warnings. Verification: Validation unit tests green; badges reflect the correct severities in the grid.

U6. Import via the existing engine, block-on-error, with notification

Goal: Implement external-import-import: block the whole batch if any hard error remains; otherwise run the import.transactions engine on clean rows (skipping warn rows) and return a result notification. Requirements: req #5, Decisions 1 & 4; AC-2, AC-9, AC-10, AC-11. Dependencies: U5. Files:

• src/clj/auto_ap/ssr/transaction/import.clj (add rows->transactions, import-transactions, external-import-import) Approach: Re-validate submitted (possibly edited) rows via U5. If any :error rows remain, throw+ {:type :field-validation :form-errors ... :form-params ...} and re-render the grid with errors (the wrap-form-4xx-2 middleware handles re-render) — nothing imports. Otherwise map clean rows → :transaction/* maps using the auto-ap.import.manual/manual->transaction shape, apply apply-synthetic-ids, then drive start-import-batch :import-source/manual → import-transaction! (per row) → finish! → get-stats. Warn-only rows are excluded from the engine input (skipped). Return html-response re-rendering the form with an hx-trigger notification reporting counts from the engine stats (imported / skipped / not-ready / extant), mirroring master's stats surface. Patterns to follow: ssr/ledger.clj import-ledger + external-import-import (~lines 554–684); import/manual.clj import-batch (engine driving); import/manual.clj manual->transaction. Test scenarios (modeled on ledger_test import-ledger-* tests, against Datomic via wrap-setup/setup-test-data/admin-token):
• Happy path: all-clean batch → engine imports all rows; stats report the imported count; transactions exist in the DB afterward.
• Block-on-error: a batch with one hard-error row → throws :field-validation; no transactions are created (assert DB unchanged).
• Warning skip: a batch with one warn-only row (e.g., non-POSTED) and clean rows → clean rows import, warn row skipped, stats reflect the skip.
• Idempotency: importing the same paste twice → second run imports 0 new (extant/synthetic-id dedupe); no duplicates.
• Integration: imported transaction carries :import-source/manual and is categorized/coded by the engine as it would be for any import (engine unchanged). Verification: Import unit/integration tests green; the e2e's import step succeeds and the transaction appears on the list page.

U7. Transactions "Import" nav entry + final polish

Goal: Add an "Import" entry to the transactions section nav (parallel to the ledger import nav) and finish the parsed-summary banner / notification copy. Requirements: req #2; AC-1, AC-11. Dependencies: U2 (route exists), U6 (notification exists). Files:

• src/clj/auto_ap/ssr/components/aside.clj (add a transactions "Import" nav button + mark active on ::transaction-routes/external-import-page) Approach: Mirror the ledger import nav entry in aside.clj — add a sub-menu button under the transactions section linking to ::transaction-routes/external-import-page, active-highlighted on that matched route. Confirm the banner shows row counts + error/warning pills (from U4) and the success notification copy matches the engine stats. Patterns to follow: ssr/components/aside.clj ledger import nav (~lines 360–366) and the transactions sub-menu (~lines 285–298). Test scenarios:
• Test expectation: none (navigation markup) — covered indirectly by the e2e navigating via the nav link; optionally assert the nav button renders with the correct href on the import route. Verification: Full e2e/transaction-import.spec.ts passes; nav link is present and active on the import page.

---

Acceptance Criteria

Routing & access

• AC-1. GET /transaction2/external-import-new renders the import page for an admin; an "Import" nav entry under the transactions section links to it and is active there.
• AC-12. Unauthenticated access redirects/401s per the standard transaction-route middleware.

Paste & parse (req #1)

• AC-3. Pasting the exact master Yodlee positional TSV parses into the same field set as auto-ap.import.manual/tabulate-data; the header row is dropped and blank rows ignored.
• AC-4. POST .../parse re-renders the page with a "N rows found" banner and the review grid.

Review grid (req #2)

• AC-5. Parsed rows render in an editable grid; each field is editable and inline edits persist across re-submit.
• AC-6. Each row shows an error badge (red) when it has a hard error, a warning badge (yellow) when it has only warnings, and no badge when clean; badges list messages on hover.

Validation parity (req #3) — each produces a visible, row-attributed message:

• AC-7. Hard errors block: client not found, bank account not found, date not MM/dd/yyyy, amount unparseable, missing required field.
• AC-8. Warnings skip just that row: status ≠ POSTED, date before bank-account/start-date, date on/before client/locked-until, already-imported.
• AC-9. With any remaining hard error, clicking Import blocks the whole batch (nothing imports) and re-renders the grid with errors highlighted.

Import (req #5)

• AC-2. POST .../import imports the clean rows via the existing import.transactions engine on the :import-source/manual batch path; the success notification reports counts (imported / skipped / not-ready / extant).
• AC-10. Re-importing the same paste is idempotent — no duplicate transactions (synthetic-id dedupe preserved).
• AC-11. categorize-transaction and the engine internals are unchanged by this work.

Tests (req #4)

• The Playwright e2e e2e/transaction-import.spec.ts exists, was committed failing first, and passes at the end.
• Unit/integration tests in test/clj/auto_ap/ssr/transaction/import_test.clj cover each validation clause and the end-to-end import flow against Datomic.

---

Test Strategy

• e2e (Playwright): e2e/transaction-import.spec.ts, driven through test/clj/auto_ap/test_server.clj (real routes, injected test auth). Committed red in U1, green by U7.
• Unit/integration (clojure.test): test/clj/auto_ap/ssr/transaction/import_test.clj, modeled on test/clj/auto_ap/ssr/ledger_test.clj — pure parse/format tests, one validation test per clause, and Datomic-backed import tests via wrap-setup / setup-test-data / admin-token. Run with clj-nrepl-eval -p PORT "(clojure.test/run-tests 'auto-ap.ssr.transaction.import-test)" per AGENTS.md (preferred over lein test).
• Validation-parity oracle: existing test/clj/auto_ap/import/transactions_test.clj (categorize-transaction) backs Decision 5 — the warn-condition predicates the grid reuses are already under test.

---

System-Wide Impact

• Editing discipline (AGENTS.md): all Clojure edits go through the clojure-mcp editing tools (or @clojure-author), not raw file edits; use clojure-eval/clj-nrepl-eval to compile-check and run tests. Run lein cljfmt fix before committing; clj-paren-repair on any file that won't compile.
• Shared component reuse: the feature composes existing ssr/components (data-grid-card, validated-field, text-input, money-input, button, checkbox, errors, pill, form-errors) and ssr/form-cursor — no core-component changes expected (req #5). If a component genuinely needs a new option, prefer an additive, backward-compatible change and flag it.
• Test fixture change: giving the seeded bank account a deterministic :bank-account/code in test_server.clj could affect other e2e specs that assume the random code; U1 verifies the existing suite stays green.
• Permissions: confirm the wrap-must :activity/:subject for the import routes matches the permission model (ledger uses {:activity :import :subject :ledger}); use the transaction equivalent.

---

Risks & Mitigations

• Preview/engine drift (highest risk). Mitigated by Decision 5 — share the engine's predicates for warn conditions; the parity test in U5 asserts the grid and categorize-transaction agree.
• Headless clipboard paste. Mitigated by Decision 6 — fillable textarea + explicit parse trigger so the e2e never needs navigator.clipboard.
• form-cursor round-tripping of an editable grid is the trickiest ledger mechanic to copy; mitigate by mirroring external-import-table-form* closely and testing edit-persist-on-resubmit early (U4).
• Positional column brittleness is inherited from master (Yodlee column order); not a regression, and out of scope to fix here.

---

Deferred Implementation Notes (execution-time unknowns)

• Exact helper/function names in the new import.clj namespace.
• The precise malli :decode wiring for positional parsing (reuse vs. thin wrapper around tabulate-data).
• The exact :activity/:subject keyword for the import-route wrap-must (verify against the permissions model).
• The seeded bank-account code value and whether any existing e2e needs adjustment after making it deterministic.
• Final notification/banner copy.