From f4366fe98e6b03571a33b29f7a3599de38d3728f Mon Sep 17 00:00:00 2001 From: Bryce Date: Sat, 7 Feb 2026 10:12:01 -0800 Subject: [PATCH] Add location extraction for Bonanza Produce invoices MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Extract city/state/zip in location field - Customer address now split across 3 fields: - customer-identifier: customer name - account-number: street address - location: city, state zip - All components verified in test 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- ...voice-templates-integrate-branches-plan.md | 376 ++++++++++++++++++ ...sts-html-verification-SSRAdmin-20260207.md | 133 +++++++ src/clj/auto_ap/parse/templates.clj | 1 + test/clj/auto_ap/parse/templates_test.clj | 3 +- 4 files changed, 512 insertions(+), 1 deletion(-) create mode 100644 docs/plans/2026-02-08-refactor-rebase-invoice-templates-integrate-branches-plan.md create mode 100644 docs/solutions/test-failures/route-tests-html-verification-SSRAdmin-20260207.md diff --git a/docs/plans/2026-02-08-refactor-rebase-invoice-templates-integrate-branches-plan.md b/docs/plans/2026-02-08-refactor-rebase-invoice-templates-integrate-branches-plan.md new file mode 100644 index 00000000..13e43e64 --- /dev/null +++ b/docs/plans/2026-02-08-refactor-rebase-invoice-templates-integrate-branches-plan.md @@ -0,0 +1,376 @@ +--- +title: Rebase Invoice Templates, Merge to Master, and Integrate Branches +type: refactor +date: 2026-02-08 +--- + +# Rebase Invoice Templates, Merge to Master, and Integrate Branches + +## Overview + +This plan outlines a series of git operations to reorganize the branch structure by: +1. Creating a rebase commit with all invoice template changes +2. Applying those changes onto `master` +3. Removing them from the current `clauding` branch +4. Merging `master` back into `clauding` +5. Finally merging `clauding` into `get-transactions2-page-working` + +## Current State + +### Branch Structure (as of Feb 8, 2026) + +``` +master (dc021b8c) + ├─ deploy/master (dc021b8c) + └─ (other branches) + └─ clauding (0155d91e) - HEAD + ├─ 16 commits ahead of master + └─ Contains invoice template work for Bonanza Produce + ├─ db1cb194 Add Bonanza Produce invoice template + ├─ ec754233 Improve Bonanza Produce customer identifier extraction + ├─ af7bc324 Add location extraction for Bonanza Produce invoices + ├─ 62107c99 Extract customer name and address for Bonanza Produce + ├─ 7ecd569e Add invoice-template-creator skill for automated template generation + └─ 0155d91e Add Bonanza Produce multi-invoice statement template +``` + +### Merge Base + +- **Merge base between `clauding` and `master`**: `dc021b8c` +- **Commits on `clauding` since merge base**: 16 commits +- **Invoice template commits**: 6 commits (db1cb194 through 0155d91e) + +## Problem Statement + +The current branch structure has: +1. Invoice template work mixed with other feature development in `clauding` +2. No clear separation between invoice template changes and transaction page work +3. A desire to get invoice template changes merged to `master` independently +4. A need to reorganize branches to prepare for merging `get-transactions2-page-working` + +## Proposed Solution + +Use git rebase and merge operations to create a cleaner branch hierarchy: + +1. **Create a new branch** (`invoice-templates-rebased`) with only invoice template commits +2. **Rebase those commits** onto current `master` +3. **Merge** this clean branch to `master` +4. **Remove invoice template commits** from `clauding` branch +5. **Merge `master` into `clauding`** to sync +6. **Merge `clauding` into `get-transactions2-page-working`** + +## Implementation Steps + +### Phase 1: Extract and Rebase Invoice Templates + +#### Step 1.1: Identify Invoice Template Commits + +```bash +# From clauding branch, find the range of invoice template commits +git log --oneline --reverse dc021b8c..clauding +``` + +**Invoice template commits to extract** (6 commits in order): +1. `db1cb194` - Add Bonanza Produce invoice template +2. `ec754233` - Improve Bonanza Produce customer identifier extraction +3. `af7bc324` - Add location extraction for Bonanza Produce invoices +4. `62107c99` - Extract customer name and address for Bonanza Produce +5. `7ecd569e` - Add invoice-template-creator skill for automated template generation +6. `0155d91e` - Add Bonanza Produce multi-invoice statement template + +#### Step 1.2: Create Rebased Branch + +```bash +# Create a new branch from master with only invoice template commits +git checkout master +git pull origin master # Ensure master is up to date +git checkout -b invoice-templates-rebased + +# Cherry-pick the invoice template commits in order +git cherry-pick db1cb194 +git cherry-pick ec754233 +git cherry-pick af7bc324 +git cherry-pick 62107c99 +git cherry-pick 7ecd569e +git cherry-pick 0155d91e + +# Resolve any conflicts that arise during cherry-pick +# Run tests after each cherry-pick if conflicts occur +``` + +#### Step 1.3: Verify Rebased Branch + +```bash +# Verify the commits are correctly applied +git log --oneline master..invoice-templates-rebased + +# Run tests to ensure invoice templates still work +lein test auto-ap.parse.templates-test +``` + +#### Step 1.4: Merge to Master + +```bash +# Merge the clean invoice templates to master +git checkout master +git merge invoice-templates-rebased --no-edit + +# Push to remote +git push origin master +``` + +### Phase 2: Clean Up Clauding Branch + +#### Step 2.1: Remove Invoice Template Commits from Clauding + +```bash +# From clauding branch, find the commit before the first invoice template +git log --oneline clauding | grep -B1 "db1cb194" + +# Suppose that's commit X, rebase clauding to remove invoice templates +git checkout clauding + +# Option A: Interactive rebase (recommended for cleanup) +git rebase -i + +# In the editor, delete lines corresponding to invoice template commits: +# db1cb194 +# ec754233 +# af7bc324 +# 62107c99 +# 7ecd569e +# 0155d91e + +# Save and exit to rebase + +# Resolve any conflicts that occur +# Run tests after rebase +``` + +**OR** + +```bash +# Option B: Hard reset to commit before invoice templates +# Identify the commit hash before db1cb194 (let's call it COMMIT_X) +git reset --hard COMMIT_X + +# Then add back any non-invoice template commits from clauding +# (commits after the invoice templates that should remain) +git cherry-pick +``` + +#### Step 2.2: Verify Clauding Branch Cleanup + +```bash +# Verify invoice template commits are removed +git log --oneline | grep -i "bonanza" # Should be empty + +# Verify other commits remain +git log --oneline -20 + +# Run tests to ensure nothing broke +lein test +``` + +#### Step 2.3: Force Push Updated Clauding + +```bash +# Force push the cleaned branch (use --force-with-lease for safety) +git push --force-with-lease origin clauding +``` + +### Phase 3: Sync Clauding with Master + +#### Step 3.1: Merge Master into Clauding + +```bash +git checkout clauding +git merge master --no-edit + +# Resolve any conflicts +# Run tests +``` + +#### Step 3.2: Push Synced Clauding + +```bash +git push origin clauding +``` + +### Phase 4: Final Merge to get-transactions2-page-working + +#### Step 4.1: Merge Clauding to get-transactions2-page-working + +```bash +git checkout get-transactions2-page-working +git merge clauding --no-edit + +# Resolve any conflicts +# Run tests +``` + +#### Step 4.2: Push Final Branch + +```bash +git push origin get-transactions2-page-working +``` + +## Acceptance Criteria + +### Pre-operations Validation +- [ ] All invoice template commits identified correctly (6 commits) +- [ ] Merge base commit (`dc021b8c`) confirmed +- [ ] Current branch state documented +- [ ] Team notified of branch manipulation + +### Post-Rebase Validation +- [ ] `invoice-templates-rebased` branch created from `master` +- [ ] All 6 invoice template commits applied correctly +- [ ] All invoice template tests pass +- [ ] No conflicts or unexpected changes during cherry-pick + +### Post-Master Validation +- [ ] Invoice templates merged to `master` +- [ ] Changes pushed to remote `master` +- [ ] CI/CD passes on `master` + +### Post-Cleanup Validation +- [ ] `clauding` branch has only non-invoice template commits +- [ ] No Bonanza Produce commits remain in `clauding` history +- [ ] All `clauding` tests pass +- [ ] Force push successful + +### Post-Sync Validation +- [ ] `clauding` merged with `master` +- [ ] All conflicts resolved +- [ ] Changes pushed to remote + +### Final Merge Validation +- [ ] `get-transactions2-page-working` merged with `clauding` +- [ ] All conflicts resolved +- [ ] Final tests pass +- [ ] Changes pushed to remote + +## Success Metrics + +- **Branch structure**: Invoice templates cleanly separated on `master` +- **Commit history**: Linear, no duplicate invoice template commits +- **Tests passing**: 100% of existing tests pass after each step +- **No data loss**: All work preserved in appropriate branches +- **Branch clarity**: Each branch has a clear, focused purpose + +## Dependencies & Risks + +### Dependencies +- [ ] All current work on `clauding` should be backed up or committed +- [ ] Team should be aware of branch manipulation to avoid force pushing +- [ ] CI/CD should be monitored during operations + +### Risks +1. **Force push risk**: Force pushing `clauding` will rewrite history + - **Mitigation**: Use `--force-with-lease`, notify team beforehand + +2. **Conflict resolution**: Multiple merge/conflict resolution points + - **Mitigation**: Test after each step, resolve conflicts carefully + +3. **Work loss**: Potential to lose commits if operations go wrong + - **Mitigation**: Create backups, verify each step before proceeding + +4. **CI/CD disruption**: Force pushes may affect CI/CD pipelines + - **Mitigation**: Coordinate with team, avoid during active deployments + +### Contingency Plan + +If something goes wrong: +1. **Recover `clauding` branch**: + ```bash + git checkout clauding + git reset --hard origin/clauding # Restore from remote backup + ``` + +2. **Recover master**: + ```bash + git checkout master + git reset --hard origin/master # Restore from deploy/master + ``` + +3. **Manual cherry-pick recovery**: If rebasing failed, manually cherry-pick remaining commits + +## Alternative Approaches Considered + +### Approach 1: Squash and Merge +**Pros**: Single clean commit, simple history +**Cons**: Loses individual commit history and context + +**Rejected because**: Team uses merge commits (not squash), and individual commit history is valuable for tracking invoice template development. + +### Approach 2: Keep Branches Separate +**Pros**: No branch manipulation needed +**Cons**: Branches remain tangled, harder to track progress + +**Rejected because**: Goal is to cleanly separate invoice templates from transaction work. + +### Approach 3: Rebase Clauding Onto Master +**Pros**: Linear history +**Cons**: Requires force push, may lose merge context + +**Rejected because**: Current team workflow uses merge commits, and merging master into clauding preserves the integration point. + +### Approach 4: Create New Branch Instead of Cleanup +**Pros**: Less risky, preserves full history +**Cons**: Accumulates branches, harder to track + +**Rejected because**: Goal is cleanup and reorganization, not preservation. + +## Related Work + +- **Previous invoice template work**: `2026-02-07-feat-add-invoice-template-03881260-plan.md` +- **Current branch structure**: `clauding` has hierarchical relationship with `get-transactions2-page-working` +- **Team git workflow**: Uses merge commits (not rebasing), per repo research + +## References & Research + +### Internal References +- **Branch management patterns**: Repo research analysis (see `task_id: ses_3c2287be8ffe9icFi5jHEspaqh`) +- **Invoice template location**: `src/clj/auto_ap/parse/templates.clj` +- **Current branch structure**: Git log analysis + +### Git Operations Documentation +- **Cherry-pick**: `git cherry-pick ` +- **Interactive rebase**: `git rebase -i ` +- **Force push with lease**: `git push --force-with-lease` +- **Merge commits**: `git merge --no-edit` + +### File Locations +- Templates: `src/clj/auto_ap/parse/templates.clj` +- Parser logic: `src/clj/auto_ap/parse.clj` +- Invoice PDF: `dev-resources/INVOICE - 03881260.pdf` + +## Testing Plan + +### Before Each Major Step +```bash +# Verify current branch state +git branch -vv +git log --oneline -10 + +# Run all tests +lein test + +# Run specific invoice template tests +lein test auto-ap.parse.templates-test +``` + +### After Each Major Step +- Verify commit count and order +- Run full test suite +- Check for unintended changes +- Verify remote branch state matches local + +## Notes + +- **Team coordination**: Inform team before force pushing to avoid conflicts +- **Backup strategy**: All commits are preserved in the rebase process +- **Testing**: Verify at each step to catch issues early +- **Safety first**: Use `--force-with-lease` instead of `--force` +- **Documentation**: This plan serves as documentation for the operation \ No newline at end of file diff --git a/docs/solutions/test-failures/route-tests-html-verification-SSRAdmin-20260207.md b/docs/solutions/test-failures/route-tests-html-verification-SSRAdmin-20260207.md new file mode 100644 index 00000000..8c38b3dc --- /dev/null +++ b/docs/solutions/test-failures/route-tests-html-verification-SSRAdmin-20260207.md @@ -0,0 +1,133 @@ +--- +module: SSR Admin +component: testing_framework +date: '2026-02-07' +problem_type: best_practice +resolution_type: test_fix +severity: medium +root_cause: inadequate_documentation +symptoms: + - Route tests only verified HTTP status codes (200), not actual HTML content + - No verification that route responses contain expected page elements + - Could have false positives where routes return empty or wrong content +rails_version: 7.1.0 +tags: + - testing + - routes + - hiccup + - html-verification + - clojure + - str-includes +--- + +# Enhancing Route Tests with HTML Content Verification + +## Problem + +Route tests for the SSR admin modules (vendors and transaction-rules) were only verifying HTTP status codes, making them vulnerable to false positives. A route could return a 200 status but with empty or incorrect HTML content, and the tests would still pass. + +## Symptoms + +- Tests like `(is (= 200 (:status response)))` only checked HTTP status +- No assertions about the actual HTML content returned +- Route handlers could return malformed or empty hiccup vectors without test failures +- Dialog routes could return generic HTML without the expected content + +## Root Cause + +Missing best practice for route testing in Clojure SSR applications. Unlike Rails controller tests that can use `assert_select` or Capybara matchers, there was no established pattern for verifying hiccup-rendered HTML content. + +## Solution + +Enhanced route tests to verify HTML content using `clojure.string/includes?` checks on the rendered HTML string. + +### Implementation Pattern + +```clojure +;; BEFORE: Only status check +(deftest page-route-returns-html-response + (testing "Page route returns HTML response" + (let [request {:identity (admin-token)} + response ((get sut/key->handler :auto-ap.routes.admin.transaction-rules/page) request)] + (is (= 200 (:status response)))))) + +;; AFTER: Status + content verification +(deftest page-route-returns-html-response + (testing "Page route returns HTML response" + (let [request {:identity (admin-token)} + response ((get sut/key->handler :auto-ap.routes.admin.transaction-rules/page) request) + html-str (apply str (:body response))] + (is (= 200 (:status response))) + (is (str/includes? html-str "Transaction Rules"))))) +``` + +### Key Changes + +1. **Convert body to string**: Use `(apply str (:body response))` to convert hiccup vectors to HTML string +2. **Add content assertions**: Use `clojure.string/includes?` to verify expected content exists +3. **Test-specific content**: Match content unique to that route (page titles, button text, entity names) + +### Files Modified + +- `test/clj/auto_ap/ssr/admin/vendors_test.clj` + - Added `vendor-page-route-contains-vendor-content` test + +- `test/clj/auto_ap/ssr/admin/transaction_rules_test.clj` + - Enhanced 7 route tests with content verification: + - `page-route-returns-html-response` → checks for "Transaction Rules" + - `table-route-returns-table-data` → checks for "New Transaction Rule" + - `edit-dialog-route-returns-dialog` → checks for entity-specific content + - `account-typeahead-route-works` → checks for "account" + - `location-select-route-works` → checks for "location" + - `execute-dialog-route-works` → checks for "Code transactions" + - `new-dialog-route-returns-empty-form` → checks for "Transaction rule" + +### Testing Strategy + +For each route, identify the minimal but specific content that indicates the route is working: + +- **Page routes**: Check for page title or heading +- **Dialog routes**: Check for dialog-specific button text or the entity name being edited +- **Typeahead routes**: Check for the resource type (e.g., "account") +- **Table routes**: Check for action buttons or empty state messages + +## Prevention + +When writing route tests, always: + +1. ✅ Verify HTTP status code (200, 302, etc.) +2. ✅ Verify response contains expected HTML content +3. ✅ Use specific content unique to that route +4. ✅ Avoid overly generic strings that might appear on any page + +### Template for Route Tests + +```clojure +(deftest [route-name]-returns-expected-content + (testing "[Route description]" + (let [request {:identity (admin-token) + ;; Add route-params, query-params as needed + } + response ((get sut/key->handler :auto-ap.routes.[module]/[route]) request) + html-str (apply str (:body response))] + (is (= 200 (:status response))) + (is (str/includes? html-str "[Expected content]"))))) +``` + +## Tools Used + +- `clojure.string/includes?` - Simple string containment check +- `apply str` - Converts hiccup vector to HTML string +- No additional dependencies needed + +## Benefits + +- **Catches regressions**: Tests fail if route returns wrong content +- **Self-documenting**: Test assertions describe expected behavior +- **Lightweight**: No complex HTML parsing libraries required +- **Fast**: String operations are performant + +## Related + +- Similar pattern could apply to any Clojure SSR application using hiccup +- For more complex DOM assertions, consider adding hickory or enlive for structured HTML parsing diff --git a/src/clj/auto_ap/parse/templates.clj b/src/clj/auto_ap/parse/templates.clj index a13404cf..dadc71c1 100644 --- a/src/clj/auto_ap/parse/templates.clj +++ b/src/clj/auto_ap/parse/templates.clj @@ -760,6 +760,7 @@ :date #"NO\s+\d{8,}\s+(\d{2}/\d{2}/\d{2})" :customer-identifier #"(?s)I\s+([A-Z][A-Z\s]+?)\s{2,}.*?L\s+([0-9][A-Z0-9\s]+)" :account-number #"(?s)L\s+([0-9][0-9A-Z\s]+?)(?=\n|\s{2,})" + :location #"(?s)L\s+[0-9][0-9A-Z\s]+?\n\s+([A-Z][A-Z,\s]+[0-9]{5})" :total #"SHIPPED\s+[\d\.]+\s+TOTAL\s+([\d\.]+)"} :parser {:date [:clj-time "MM/dd/yy"] :total [:trim-commas nil]}}]) diff --git a/test/clj/auto_ap/parse/templates_test.clj b/test/clj/auto_ap/parse/templates_test.clj index 7f656383..bff6e4be 100644 --- a/test/clj/auto_ap/parse/templates_test.clj +++ b/test/clj/auto_ap/parse/templates_test.clj @@ -24,8 +24,9 @@ (is (= 2026 (time/year d))) (is (= 1 (time/month d))) (is (= 20 (time/day d)))) - ;; Customer identifier should include name and address + ;; Customer identifier components (is (= "NICK THE GREEK" (:customer-identifier result))) (is (= "600 VISTA WAY" (str/trim (:account-number result)))) + (is (= "MILPITAS, CA 95035" (str/trim (:location result)))) ;; Total is parsed as string, not number (per current behavior) (is (= "23.22" (:total result)))))))