notid/integreat
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

fix(sales-summaries): stop days falling out of balance #17

Closed
notid wants to merge 4 commits from sales-summary-balancing into master
pull from: sales-summary-balancing
merge into: notid:master
Conversation 0 Commits 4 Files Changed 19 +2965 -479
19 changed files with 2965 additions and 479 deletions
Download Patch File Download Diff File Expand all files Collapse all files

115
docs/2026-08-15-remove-voided-orders-risk.md Normal file
View File

@@ -0,0 +1,115 @@
---
title: remove-voided-orders can delete another client's payments
type: risk
date: 2026-08-15
status: open — decide before merging the re-key
---
# `remove-voided-orders` can delete another client's payments
Measured on the restored backup, 2026-08-15. This risk is **pre-existing** — nothing in the
sales-summary work created it — but it is live right now, and the re-key work touches the same
data, so it should be understood before merging.
## The mechanism, in four steps
**1. Charges are component entities of an order.**
```clojure
;; resources/schema.edn
{:db/ident :sales-order/charges
:db/valueType :db.type/ref
:db/isComponent true ;; <- this is the load-bearing bit
:db/cardinality :db.cardinality/many}
```
`:db/isComponent true` tells Datomic the charges *belong to* the order. It is what lets you
transact an order with its tenders nested inside, and it means the charges have no independent
existence as far as Datomic is concerned.
**2. `retractEntity` on a component parent deletes the children too.**
That is the documented behaviour of `:db/retractEntity`: it recursively retracts component
values. `square.core3/remove-voided-orders` ends with exactly that:
```clojure
(s/map (fn [[o]]
[[:db/retractEntity [:sales-order/external-id (:sales-order/external-id o)]]]))
```
It asks Square for the last 10 days of orders, keeps the ones that should *not* be imported —
voided and cancelled orders — and retracts any of those we already stored. That is correct and
desirable on its own: a voided order should not sit in the books.
**3. But one charge can be shared by two orders.**
When two clients are configured on the same Square location, both import the same Square data.
Order keys embed the client, so each client gets its own order entity. Charge keys did **not**
embed the client, and `:charge/external-id` is `:db.unique/identity`, so both clients' orders
resolved to *the same charge entity*:
```
NGCD order 17592395490523 ──┐
├──> charge 17592490524 ← one entity, two parents
NGCC order 17592395511722 ──┘
```
**4. So retracting one order deletes a charge the other order still points at.**
Datomic sees a component and removes it. The surviving order keeps its line items — its sales —
but its tender is gone. The day then shows revenue with no payment against it, the summary goes
out of balance, and the payment is gone from the current database value. (History retains it, so
it is recoverable by someone who knows to look, but nothing in the app will show it again.)
## How exposed are we
Measured over the 10 contended clients across 2026-07-13 → 08-14:
| | |
|---|---|
| Charges examined | 56,829 |
| **Referenced by more than one order** | **35,870 (63%)** |
So this is not a theoretical corner. Roughly two thirds of the charges in that population have
two parents, and any voided order among them takes a charge down with it.
The exposure window for *new* damage is the rolling 10 days `remove-voided-orders` searches, but
the shared charges themselves span the whole period the locations were double-configured.
## What changes after Phase 0 and the re-key, and what doesn't
- **Phase 0 (done on the restore)** stops new sharing: only one client per location imports now,
so no new order pairs form.
- **The re-key (done for refunds and the contended clients' charges)** makes sharing structurally
impossible going forward, because a charge key now contains the client code.
- **Neither retroactively splits the 35,870 charges that are already shared.** They still have two
parents. Until they are split, `remove-voided-orders` remains capable of deleting a payment
belonging to the other client.
This is why plan §3.3 forbids retracting anything — including any historical cleanup of the
duplicate clients' data — until a verification query shows zero charges with more than one parent.
## Options, roughly in order of preference
1. **Split the shared charges, then let removal run normally.** Re-import the affected window now
that keys are client-scoped, so each client creates its own charge entity. This reuses the
import path rather than hand-constructing component entities. Verify with a query for charges
having more than one referencing order; it must reach zero.
2. **Guard the retraction.** Before retracting an order, check whether any of its charges are
referenced by another order; detach those (retract the `:sales-order/charges` ref rather than
the charge) and retract the rest. Small, contained change, and it makes the operation safe
regardless of what shape the data is in — worth doing on its own merits even after a split.
3. **Do nothing and accept it.** Only defensible once every location has a single client *and*
the historical shared charges are gone. Not true today.
## What I did about it during the validation run
I ran the import on the restore with `remove-voided-orders` **skipped**, and ran the other steps
(`upsert-locations`, `upsert`, `upsert-payouts`, `upsert-refunds`) normally. That kept the
validation faithful to how the import behaves without risking silent payment loss in the data
the measurements were about to be taken from.
**Nothing in the production system has been changed.** This note is about a risk that already
exists there.

693
docs/2026-08-15-sales-summary-reconciliation.html Normal file
View File

@@ -0,0 +1,693 @@
<title>Ninety-Day Reconciliation</title>
<style>
:root {
--paper: #F6F8F7; --card: #FFFFFF; --ink: #141F1D; --ink-soft: #4A5C58;
--ink-faint: #7C8D89; --rule: #DCE4E1; --accent: #0E5B57; --accent-soft: #E3EFED;
--good: #1A6B49; --bad: #A03B26; --warn: #8A6410;
--shadow: 0 1px 2px rgba(20,31,29,.06), 0 8px 24px rgba(20,31,29,.05);
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
}
}
:root[data-theme="dark"] {
--paper: #0E1615; --card: #151F1E; --ink: #E8EFED; --ink-soft: #A3B3AF;
--ink-faint: #74847F; --rule: #26332F; --accent: #5FBDB4; --accent-soft: #16302E;
--good: #5FBE8C; --bad: #E08A72; --warn: #D6AC55;
--shadow: 0 1px 2px rgba(0,0,0,.4), 0 8px 24px rgba(0,0,0,.3);
}
* { box-sizing: border-box; }
body { background: var(--paper); color: var(--ink);
font-family: system-ui, -apple-system, "Segoe UI", sans-serif;
font-size: 16px; line-height: 1.6; margin: 0; padding: 0 20px 96px; }
.wrap { max-width: 940px; margin: 0 auto; }
.measure { max-width: 66ch; }
.num { font-variant-numeric: tabular-nums; }
.mono { font-family: ui-monospace, "SF Mono", "Cascadia Code", monospace; font-variant-numeric: tabular-nums; }
header.masthead { padding: 72px 0 40px; border-bottom: 2px solid var(--ink); display: flex; flex-direction: column; gap: 14px; }
.eyebrow { font-size: 12px; letter-spacing: .14em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
h1 { font-family: Georgia, "Iowan Old Style", serif; font-size: clamp(34px, 5.4vw, 54px);
line-height: 1.08; font-weight: 600; letter-spacing: -.015em; margin: 0; text-wrap: balance; }
.standfirst { font-size: 19px; color: var(--ink-soft); margin: 0; max-width: 62ch; }
.meta { display: flex; flex-wrap: wrap; gap: 10px 28px; font-size: 13px; color: var(--ink-faint); padding-top: 6px; }
.meta b { color: var(--ink-soft); font-weight: 600; }
section { padding-top: 56px; display: flex; flex-direction: column; gap: 20px; }
h2 { font-family: Georgia, "Iowan Old Style", serif; font-size: 27px; font-weight: 600; letter-spacing: -.01em; margin: 0; text-wrap: balance; }
h3 { font-size: 14px; letter-spacing: .08em; text-transform: uppercase; color: var(--ink-soft); font-weight: 700; margin: 0; }
h4 { font-size: 18px; font-weight: 650; margin: 0; letter-spacing: -.01em; }
p { margin: 0; }
.measure p + p { margin-top: 14px; }
.ledger { display: grid; grid-template-columns: 1fr auto 1fr; border: 1px solid var(--rule);
border-radius: 4px; background: var(--card); box-shadow: var(--shadow); overflow: hidden; }
.ledger > div { padding: 26px 28px; display: flex; flex-direction: column; gap: 6px; }
.ledger .arrow { justify-content: center; align-items: center; border-left: 1px solid var(--rule);
border-right: 1px solid var(--rule); color: var(--ink-faint); font-size: 22px; background: var(--accent-soft); }
.side-label { font-size: 12px; letter-spacing: .12em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; }
.figure { font-size: clamp(28px, 4.4vw, 40px); font-weight: 650; line-height: 1.05; letter-spacing: -.02em; }
.figure.after { color: var(--good); }
.subfig { font-size: 14px; color: var(--ink-soft); }
.stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 14px; }
.stat { background: var(--card); border: 1px solid var(--rule); border-radius: 4px; padding: 18px 20px; display: flex; flex-direction: column; gap: 4px; }
.stat .k { font-size: 30px; font-weight: 650; letter-spacing: -.02em; line-height: 1; }
.stat .l { font-size: 13px; color: var(--ink-soft); }
.stat.zero .k { color: var(--good); }
.problem { border-left: 3px solid var(--accent); padding-left: 24px; display: flex; flex-direction: column; gap: 14px; }
.problem.two { border-left-color: var(--warn); }
.problem.three { border-left-color: var(--bad); }
.scroll { overflow-x: auto; border: 1px solid var(--rule); border-radius: 4px; background: var(--card); }
table { border-collapse: collapse; width: 100%; font-size: 14.5px; }
th, td { padding: 11px 16px; text-align: left; border-bottom: 1px solid var(--rule); white-space: nowrap; }
thead th { font-size: 11.5px; letter-spacing: .09em; text-transform: uppercase; color: var(--ink-faint); font-weight: 700; background: var(--accent-soft); }
tbody tr:last-child td { border-bottom: none; }
td.n, th.n { text-align: right; font-variant-numeric: tabular-nums; }
tr.total td { font-weight: 650; background: var(--accent-soft); }
.good { color: var(--good); font-weight: 650; }
.bad { color: var(--bad); font-weight: 650; }
.dim { color: var(--ink-faint); }
.callout { background: var(--card); border: 1px solid var(--rule); border-left: 3px solid var(--accent);
border-radius: 4px; padding: 20px 24px; display: flex; flex-direction: column; gap: 10px; }
.callout.warn { border-left-color: var(--warn); }
.callout .h { font-weight: 650; }
code { font-family: ui-monospace, "SF Mono", monospace; font-size: .9em; background: var(--accent-soft); padding: 1px 5px; border-radius: 3px; }
pre { margin: 0; padding: 20px; font-size: 13px; line-height: 1.7; white-space: pre; font-family: ui-monospace, "SF Mono", monospace; }
footer { margin-top: 72px; padding-top: 24px; border-top: 1px solid var(--rule); font-size: 13px; color: var(--ink-faint); display: flex; flex-direction: column; gap: 8px; }
ul { margin: 0; padding-left: 20px; display: flex; flex-direction: column; gap: 8px; }
.tech { font-size: 11px; letter-spacing: .08em; text-transform: uppercase; color: var(--accent);
font-weight: 700; border: 1px solid var(--accent); border-radius: 3px; padding: 2px 7px; display: inline-block; }
</style>
<div class="wrap">
<header class="masthead">
<div class="eyebrow">Sales summaries · measured on a restored production backup</div>
<h1>Ninety-Day Reconciliation</h1>
<p class="standfirst">Four faults were leaving restaurant days out of balance — one in the data, three in the arithmetic. This is what they were, what they cost, and what fixing them is worth, measured by running the real job over ninety days of real trading, twice: once with the fixes off and once with them on.</p>
<div class="meta">
<span><b>Window</b> 2026-05-10 → 2026-08-07</span>
<span><b>Client-days</b> <span class="num">18,900</span></span>
<span><b>Clients</b> <span class="num">210</span></span>
<span><b>Nothing in production was changed</b></span>
</div>
</header>
<section>
<div class="ledger">
<div>
<span class="side-label">Today's calculation, ninety days re-run</span>
<span class="figure num">$69,560.10</span>
<span class="subfig"><span class="num">1,258</span> days out of balance · <span class="num">93.34%</span> clean</span>
</div>
<div class="arrow" aria-hidden="true">→</div>
<div>
<span class="side-label">The same ninety days, fixes on</span>
<span class="figure after num">$2,970.35</span>
<span class="subfig"><span class="num">123</span> days out of balance · <span class="num">99.35%</span> clean</span>
</div>
</div>
<div class="stats">
<div class="stat"><span class="k num">1,135</span><span class="l">client-days brought into balance</span></div>
<div class="stat zero"><span class="k num">0</span><span class="l">days knocked out of balance</span></div>
<div class="stat"><span class="k num">95.7%</span><span class="l">of the variance removed</span></div>
<div class="stat zero"><span class="k num">0</span><span class="l">payments shared between two clients</span></div>
</div>
<div class="measure">
<p><strong>In one sentence:</strong> a day's sales summary should show the money taken and the money earned agreeing to the penny, and on roughly one trading day in eight it did not — because two clients were fighting over the same records, tips that had been refunded were still counted as income, service charges customers paid were credited to nothing, and a refund that arrived on a day with no sales had nothing to offset it.</p>
<p><strong>How the two figures above were produced.</strong> Both are the real nightly job, run over the same ninety days against the same restored database, writing real summaries each time — the first pass with the fixes switched off, the second with them on. Comparing a re-run against a re-run rather than against production's stored summaries is the stricter test: production's figures are in places months stale, and crediting the fixes with repairing ordinary staleness would flatter them. On that fairer footing the fixes are worth <strong>1,135 days and $66,589.75</strong>, not the larger number a stale baseline would have shown.</p>
</div>
</section>
<section>
<h2>The four problems</h2>
<div class="problem">
<h4>1. Two client records sharing one Square location</h4>
<div class="measure">
<p><strong>For the business:</strong> ten restaurant locations were set up twice in the system, as two separate clients. Both were importing from Square. Because the two records competed for the same payments and refunds, a refund would belong to one client for twenty minutes, then the other — so a day's books could gain or lose a refund depending on nothing but timing. On 2026-07-23 one client's summary was missing a $71.94 refund entirely, and was out of balance by exactly that amount.</p>
<p><span class="tech">technical</span> Sales orders scoped their identifier by client (<code>square/order/&lt;code&gt;-&lt;loc&gt;-&lt;id&gt;</code>), but refunds, card charges, payouts and cash-drawer shifts did not — they used the bare Square id. Those attributes are <code>:db.unique/identity</code>, so both clients' imports resolved to a single entity and the last writer won.</p>
<p>Reading ownership out of the database's own history, this had actually happened to <strong>3,387 refunds, 4,069 payouts and 2,628 cash-drawer shifts</strong>. And it has involved <strong>19 client pairs, of which only 10 are visible in today's configuration</strong> — nine more contended in the past and the configuration has since changed, so no point-in-time check would find them.</p>
</div>
</div>
<div class="problem two">
<h4>2. One payment record owned by two orders</h4>
<div class="measure">
<p><strong>For the business:</strong> the same collision meant a single card payment could be attached to both clients' copies of an order. That is worse than untidy. The nightly import removes orders Square reports as voided, and removing an order also removes its payments — so cancelling one client's order could silently delete the <em>other</em> client's payment, leaving a day showing sales with no money against them.</p>
<p><span class="tech">technical</span> <code>:sales-order/charges</code> is declared <code>:db/isComponent true</code>, so <code>[:db/retractEntity &lt;order&gt;]</code> cascades into the charges. In a 20,000-order sample of the affected clients, <strong>11,469 charges had two parent orders</strong>. This is why <code>remove-voided-orders</code> was left switched off during testing.</p>
</div>
</div>
<div class="problem three">
<h4>3. Tips refunded, and service charges credited nowhere</h4>
<div class="measure">
<p><strong>For the business:</strong> two arithmetic faults, both of which overstated or understated a day.</p>
<ul>
<li><strong>Refunded tips stayed on the books.</strong> When a guest was refunded, the tip came back too — but the summary still counted the original tip as income. On one NGLK day the books credited $482.94 of tips beside a $60.00 refund of that very tip.</li>
<li><strong>Service charges were collected but never earned.</strong> A catering or auto-gratuity charge is inside the card payment the customer makes, so it arrived as money taken — but no line recorded it as money earned. One NTPT order carried $427.10 that was credited to nothing at all; the largest single instance was <strong>$1,344.86</strong> in one day.</li>
</ul>
<p><span class="tech">technical</span> <code>get-tip</code> summed tips by joining through <code>:sales-order/charges</code>, so a return-only order — which has no tender to join through — contributed nothing, while its reversal sat unread on <code>:sales-order/tip</code>. Nothing at all read <code>:sales-order/service-charge</code>.</p>
</div>
</div>
<div class="problem">
<h4>4. A refund on a day the restaurant did not sell anything</h4>
<div class="measure">
<p><strong>For the business:</strong> a refund is recorded on the day the money goes back to the guest. The matching reduction in sales is read from that day's orders. When both fall on the same day they cancel and the day balances. When they do not — a refund settling after a closed day, or a retired duplicate record that still receives refunds while its orders belong to the surviving record — the day is left carrying a refund with nothing against it, and is out by exactly the refunded amount.</p>
<p><strong>This is the answer to why the duplicated restaurants looked so much worse than everyone else.</strong> Of the days still failing after the first three fixes, <strong>155 of the 158 on duplicate records had no sales orders at all</strong> — the summary consisted of nothing but orphaned refunds and their fees. The refunds were claimed by the record that happened to import them during the shared period; the orders went to the other record. Nothing was double-counted and no money was lost — the two halves of each refund simply ended up on two different sets of books.</p>
<p>It is not a duplicate-only problem, which is what makes it worth fixing properly rather than papering over. The single largest non-duplicate cluster, <span class="mono">NG4S</span>, has the identical shape: no orders on any day of the window, refunds arriving anyway, and every one of those days out by exactly the refund.</p>
<p><span class="tech">technical</span> <code>get-returns</code> summed <code>:sales-order/returns</code> over orders scanned for the date. With no orders the sum is nil and no <code>Returns</code> line is written at all, while <code>get-refund-items</code> still credits <code>Card Refunds</code> from the <code>sales-refund</code> records — a credit with no debit anywhere in the day.</p>
</div>
</div>
</section>
<section>
<h2>What the fixes actually are</h2>
<div class="measure">
<p>Six changes. The first three stop two clients from sharing a record; the last three record
money that was being collected, or handed back, but not booked. Each is small — the difficulty
was knowing which line to change, not writing it.</p>
</div>
<h3>1 · Put the client in the record's name</h3>
<div class="measure">
<p>Every imported record has an identifier the importer uses to decide "have I seen this
before?". Sales orders already included the client; refunds, card payments, payouts and
cash-drawer shifts did not, which is precisely why two clients could land on one record.</p>
</div>
<div class="scroll">
<pre><span class="dim">;; before — the bare Square id, identical for both clients</span>
(str "square/refund/" (:id r)) <span class="dim">;; square/refund/NOkQOTIiJULWN6…</span>
<span class="dim">;; after</span>
(scoped-key "square/refund/" client location (:id r))
<span class="dim">;; square/refund/NGCD-CD-NOkQOTIiJULWN6…</span>
(defn scoped-key [prefix client location id]
(str prefix (:client/code client) "-"
(:square-location/client-location location) "-" id))</pre>
</div>
<div class="measure">
<p>Applied at five places in the Square importer: order payments, refunds, payouts (twice —
the record itself and the lookup that finds it) and cash-drawer shifts. ezCater orders
already did this and needed no change.</p>
</div>
<h3>2 · Find the existing record before writing, under either name</h3>
<div class="measure">
<p>This is the one that makes the change safe to deploy. The identifiers are unique keys, so
the importer relies on "same id, same record". Rename them and the next import matches
nothing — and would quietly create a <em>second</em> copy of every refund and payment in the
system, leaving the originals orphaned. So the importer looks up the record explicitly,
new name first, old name second, and writes to whichever it finds.</p>
</div>
<div class="scroll">
<pre>(defn existing-id [db attr prefix client location id]
(when id
(or (dc/entid db [attr (scoped-key prefix client location id)]) <span class="dim">;; new scheme</span>
(dc/entid db [attr (str prefix id)])))) <span class="dim">;; legacy scheme</span></pre>
</div>
<div class="measure">
<p>The result is pinned as the record's id on the way in, so the write lands on the existing
row regardless of which name it currently carries. The proof this worked is a count that did
not move. Every one of the 265,965 refunds, payouts and cash-drawer shifts in the database was
re-named, and afterwards there were still exactly <strong>144,688 payouts and 69,291 cash-drawer
shifts</strong> — the same figures as at the restore point. Refunds went from 50,986 to 51,986,
and all 1,000 of those came from the live Square import run afterwards, not from the renaming.
Had the fallback lookup been missing, each of these would have doubled instead.</p>
</div>
<h3>3 · Give every order its own payment record</h3>
<div class="measure">
<p>Renaming stops <em>new</em> collisions but does not undo old ones: a payment already shared
by two orders is still one row with two owners. The migration walks each order's payments and,
where another order has already claimed one, makes that order its own copy with the same
amounts and points the order at the copy.</p>
</div>
<div class="scroll">
<pre><span class="dim">;; for each order, for each of its payments:</span>
:keep <span class="dim">→</span> first order to claim it; rename in place
:clone <span class="dim">→</span> copy type, total, tip, tax, date, processor, note, receipt link
set the copy's client and location to this order's
retract this order's link to the shared payment
link it to the copy instead</pre>
</div>
<div class="measure">
<p>Run over the whole database that was <strong>9,100,314 renamed and 200,027 copied</strong>,
and payments owned by two orders went from 11,469 in a 20,000-order sample to zero across
400,000 orders checked. The record count rose by exactly 200,027 — the number of copies it
reported making, which is the check that it created what it meant to and nothing else.</p>
<p>One subtlety worth recording, because it bit us: the Square id has to be recovered from the
record's current owner rather than by trimming a fixed prefix. Client codes contain dashes —
<span class="mono">N-30003</span> — so a pattern cannot tell where the client name ends and the
Square id begins. Getting this wrong scoped some records twice and doubled their tender.</p>
</div>
<h3>4 · Count tips that were handed back</h3>
<div class="measure">
<p>Tips were summed by walking from the order to its payments. A refund-only order has no
payment attached, so its negative tip was invisible. The fix adds those tips rather than
replacing the calculation.</p>
</div>
<div class="scroll">
<pre><span class="dim">;; before</span>
:ledger-mapped/amount (tendered-tip c date)
<span class="dim">;; after</span>
:ledger-mapped/amount (+ (tendered-tip c date)
(untendered-tip c date))
<span class="dim">;; untendered-tip — tips on orders with no payment attached</span>
[?e :sales-order/tip ?tip]
(not [?e :sales-order/charges])</pre>
</div>
<div class="measure">
<p><strong>Adding rather than replacing is deliberate.</strong> Where an order does have a
payment, the payment is the correct source: real orders exist whose payment carries a tip the
order does not — an auto-gratuity recorded as a service charge, or a wallet tip missing from
the order totals. Reading the order instead would have dropped those. Three tests hold this
in place: the refund case must change, and the tendered and ordinary cases must not.</p>
</div>
<h3>5 · Credit Square service charges, both signs</h3>
<div class="measure">
<p>Nothing read the service-charge field at all. A new line credits it, for Square orders only
and for negative amounts as well as positive.</p>
</div>
<div class="scroll">
<pre>[?e :sales-order/service-charge ?service-charge]
(or-join [?e]
[?e :sales-order/vendor :vendor/ccp-square]
(and (not [?e :sales-order/vendor])
[?e :sales-order/external-id ?external-id]
[(clojure.string/starts-with? ?external-id "square/order/")]))</pre>
</div>
<div class="measure">
<p><strong>Why the vendor test has two branches.</strong> ezCater service charges are commission
the platform deducts from the restaurant, not money the diner hands over, so crediting them
would make a day worse rather than better — hence the Square-only condition. But whole eras of
Square orders carry no vendor field at all, and a test on vendor alone would silently credit
nothing. The second branch falls back to the order's own identifier.</p>
<p><strong>Why negatives matter.</strong> A returned catering fee arrives as a negative service
charge and is already deducted from the day's returns; dropping negatives would lose the
reversal. The line sits behind a per-client switch, off by default, so it can be turned on a
few restaurants at a time.</p>
</div>
<h3>6 · Recognise a return on a day that refunded but did not sell</h3>
<div class="measure">
<p>Where a day has refunds and no sales orders whatsoever, book a <code>Returns</code> debit
equal to that day's refunds. The guard — <em>no orders at all</em> — is what makes this
mechanical rather than a judgement call.</p>
</div>
<div class="scroll">
<pre><span class="dim">;; the return a day owes when it refunded but never sold</span>
(defn- refund-only-returns [c date]
(when (and (refund-only-returns-enabled? c)
(not (traded? c date)))
(let [amount (refunded-total c date)]
(when-not (zero? amount) amount))))
<span class="dim">;; get-returns falls back to it only when the order-derived figure is absent</span>
(or (ffirst (dc/q <span class="dim">;; sum of :sales-order/returns, as before</span>))
(refund-only-returns c date))</pre>
</div>
<div class="measure">
<p><strong>Why the guard is the whole design.</strong> On a day with no orders the
order-derived return is necessarily nothing, so there is no figure to double-count, and no
trading day can have its numbers moved by this rule — which the measurement bears out: 156
days came into balance, none went out, and not one already-balanced day had a line change.
Days that <em>did</em> trade and still carry an unmatched refund are deliberately left alone;
apportioning a return across a day that also sold is an accounting question, not a mechanical
one, and guessing at it would be exactly the kind of change that quietly rewrites correct
books.</p>
<p>Like the service-charge line, it sits behind a per-client switch that is off by default.
Four tests hold it: flag off changes nothing; a refund-only day balances exactly; a day that
traded keeps its own return; and a day that traded but returned nothing does not pick up the
refunded total either — that last one guards the difference between "the client did not
trade" and "the order-derived figure happened to be nil", which is the subtle way this rule
could have gone wrong.</p>
</div>
<h3>Supporting changes</h3>
<div class="scroll">
<table>
<thead><tr><th>Change</th><th>Why</th></tr></thead>
<tbody>
<tr><td>Log each day's imbalance and its suspect lines</td><td>an out-of-balance day was only visible by opening the screen; now it can be queried</td></tr>
<tr><td>Stop the dirty-summary scan at the client boundary</td><td>it read every later client's summaries too — 1,321 ms to 5.6 ms per client</td></tr>
<tr><td>Split the recompute driver into a per-client function</td><td>lets a backfill spread clients across threads instead of grinding one at a time</td></tr>
<tr><td>Install schema attributes before the tuples that compose them</td><td>the test suite could not build an empty database at all, so no test could run</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>That last one is worth a sentence for engineers: <code>transact-schema</code> installed
schema.edn then cloud-migration-schema.edn, but a composite tuple in the first file is built
from an attribute in the second. Datomic will not create a tuple before its members exist, so
every test fixture died in setup. It is very likely why sales summaries had no tests before
this work.</p>
</div>
</section>
<section>
<h2>What each fix is worth</h2>
<div class="measure">
<p>The job was run over the same ninety days at each stage, writing real summaries every time, so these are measured outcomes rather than estimates. All 18,900 client-day summaries in the window are included, whether or not the restaurant traded that day.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Stage</th><th class="n">Days out of balance</th><th class="n">Clean</th><th class="n">Total variance</th></tr></thead>
<tbody>
<tr><td>Today's calculation, ninety days re-run</td><td class="n">1,258</td><td class="n">93.34%</td><td class="n">$69,560.10</td></tr>
<tr><td>+ refunded tips</td><td class="n">971</td><td class="n">94.86%</td><td class="n">$66,414.39</td></tr>
<tr><td>+ service charges</td><td class="n">279</td><td class="n">98.52%</td><td class="n">$7,790.54</td></tr>
<tr class="total"><td>+ returns on refund-only days</td><td class="n good">123</td><td class="n good">99.35%</td><td class="n good">$2,970.35</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p><strong>Deduplication is not a row in this table, and that is deliberate.</strong> Separating the shared records is a change to the data, not to the arithmetic, and it had already been carried out before either pass ran — so both the baseline and the result above are computed on repaired data, and neither is credited with it. Its effect is shown structurally instead, further down: payments owned by two clients went to zero and stayed there. The consequence for reading this table is that <strong>$66,589.75 is what the three arithmetic fixes are worth on their own</strong>, with the deduplication's contribution already banked in the starting figure rather than added to the improvement.</p>
</div>
<h3>Day-by-day effect of each change</h3>
<div class="scroll">
<table>
<thead><tr><th>Change</th><th class="n">Unchanged</th><th class="n">Into balance</th><th class="n">Out of balance</th><th class="n">Balanced days altered</th><th class="n">Money moved</th></tr></thead>
<tbody>
<tr><td>Refunded tips</td><td class="n">18,590</td><td class="n good">287</td><td class="n good">0</td><td class="n good">0</td><td class="n">$3,712.67</td></tr>
<tr><td>Service charges</td><td class="n">18,208</td><td class="n good">692</td><td class="n good">0</td><td class="n good">0</td><td class="n">$58,349.85</td></tr>
<tr><td>Returns on refund-only days</td><td class="n">18,740</td><td class="n good">156</td><td class="n good">0</td><td class="n good">0</td><td class="n">$4,820.19</td></tr>
<tr class="total"><td>All three, end to end</td><td class="n">17,765</td><td class="n good">1,135</td><td class="n good">0</td><td class="n good">0</td><td class="n">$66,589.75</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p><strong>None of the three touched a day that was already correct.</strong> Across all 18,900 client-days, no balanced day was knocked out of balance, and no balanced day had a single figure altered. Every day that moved was already wrong. Service charges are by far the largest of the three, moving $58,349.85 against $4,820.19 for refund-only returns and $3,712.67 for tips.</p>
<p>That claim is stronger than a balance check, and it is the one worth insisting on: a day can stay balanced while its individual lines move, which would still be a change to the books. Every line of every summary was compared — category, debit or credit side, amount to the cent, and account — not just the day's bottom line.</p>
<p><strong>The arithmetic fixes account for their repair exactly.</strong> Adding up the untendered-tip and service-charge amounts for the 979 days those two closed leaves a residue of <span class="mono">0.0000000013</span> against $61,769.56 of imbalance removed. Nothing else moved those days; there is no unexplained remainder hiding a further effect.</p>
</div>
</section>
<section>
<h2>What it looks like on the page</h2>
<div class="measure">
<p>Both arithmetic fixes add exactly one credit line. Nothing else in a summary moves — no sales figure, no payment, no tax.</p>
</div>
<h3>A refunded tip — NGLK, 2026-08-04</h3>
<div class="scroll">
<table>
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
<tbody>
<tr><td><strong>Tip</strong></td><td class="n">482.94</td><td class="n">422.94</td></tr>
<tr><td class="dim">Card Refunds</td><td class="n dim">60.00</td><td class="n dim">60.00</td></tr>
<tr><td>Total money taken</td><td class="n">10,094.81</td><td class="n">10,094.81</td></tr>
<tr><td>Total money earned</td><td class="n">10,154.81</td><td class="n">10,094.81</td></tr>
<tr class="total"><td>Out of balance by</td><td class="n bad">−60.00</td><td class="n good">0.00</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>The day already carried a $60.00 card refund — the guest was given their money back, tip included — while the tip line still credited the full $482.94. The corrected figure matches the refund to the penny. The order behind it is <span class="mono">square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY</span>: a tip of −60.00 on an order with no payment attached at all.</p>
</div>
<h3>A service charge — NTPT, 2026-08-06</h3>
<div class="scroll">
<table>
<thead><tr><th>Line</th><th class="n">Before</th><th class="n">After</th></tr></thead>
<tbody>
<tr><td><strong>Service Charges</strong></td><td class="n bad">not shown</td><td class="n">427.10</td></tr>
<tr><td class="dim">Card Payments</td><td class="n dim">4,975.89</td><td class="n dim">4,975.89</td></tr>
<tr><td>Total money taken</td><td class="n">7,777.20</td><td class="n">7,777.20</td></tr>
<tr><td>Total money earned</td><td class="n">7,350.10</td><td class="n">7,777.20</td></tr>
<tr class="total"><td>Out of balance by</td><td class="n bad">+427.10</td><td class="n good">0.00</td></tr>
</tbody>
</table>
</div>
<h3>The largest repairs of each kind</h3>
<div class="scroll">
<table>
<thead><tr><th>Client</th><th>Date</th><th>Line</th><th class="n">Before</th><th class="n">After</th><th class="n">Day closed</th></tr></thead>
<tbody>
<tr><td class="mono">NGPA</td><td>2026-06-04</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">1,344.86</td><td class="n good">+1,344.86 → 0</td></tr>
<tr><td class="mono">NTPT</td><td>2026-08-06</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">427.10</td><td class="n good">+427.10 → 0</td></tr>
<tr><td class="mono">N-30003</td><td>2026-05-27</td><td>Service Charges</td><td class="n bad">not shown</td><td class="n">405.83</td><td class="n good">+405.83 → 0</td></tr>
<tr><td class="mono">NGFL</td><td>2026-05-19</td><td>Tip</td><td class="n">238.46</td><td class="n">70.42</td><td class="n good">−168.04 → 0</td></tr>
<tr><td class="mono">NGMI</td><td>2026-07-09</td><td>Tip</td><td class="n">230.01</td><td class="n">80.01</td><td class="n good">−150.00 → 0</td></tr>
<tr><td class="mono">NGVA</td><td>2026-07-03</td><td>Tip</td><td class="n">152.66</td><td class="n">40.12</td><td class="n good">−112.54 → 0</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>In every case the correction equals the imbalance exactly, which is what you would expect if the fix is recording something real that was recorded nowhere. On NGNP 2026-06-25 both fixes land on one day and pull opposite ways — $301.40 credited, $1.80 removed, $299.60 closed — a useful check that they are independent.</p>
</div>
</section>
<section>
<h2>What was done to the data, and how it was checked</h2>
<div class="measure">
<p>Every step below was performed against a restored copy of the production database. Production itself was never touched.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Step</th><th>Result</th></tr></thead>
<tbody>
<tr><td>Deactivate the duplicate client at each shared location</td><td class="n">10 locations · shared locations remaining: <span class="good">0</span></td></tr>
<tr><td>Walk every order in the database</td><td class="n">19,040,785 orders</td></tr>
<tr><td>Give every order its own payment record</td><td class="n">9,100,314 re-keyed · 200,027 copied</td></tr>
<tr><td><strong>Payments owned by two orders</strong></td><td class="n good">0 <span class="dim">across 400,000 orders checked</span></td></tr>
<tr><td>Client-scope refunds, payouts and cash-drawer shifts</td><td class="n good">counts unchanged · 0 collisions</td></tr>
<tr><td>Live Square import afterwards</td><td class="n good">0 orders with duplicated payment · 0 shared payments</td></tr>
<tr><td>Ownership changes after the change</td><td class="n good">0 refunds · 0 payouts · 0 shifts</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>The count checks are the ones that matter. If re-keying had gone wrong it would have created a second copy of every record rather than updating the existing one, and the totals would have doubled. They did not move. The payment-copy step is the exception and is meant to add records — it added exactly 200,027, matching the number of copies it reported making.</p>
</div>
<div class="callout">
<span class="h">The whole analysis was run again from nothing, and landed in the same place</span>
<p>Everything above was rebuilt from a fresh restore of the production backup: restore, deactivate, re-key and duplicate across all nineteen million orders, re-import from Square, then two full ninety-day recomputes. At the point the two runs are comparable — before the refund-only fix below, which the earlier run did not have — the end state matched <strong>to the cent</strong>: 279 days out of balance and $7,790.54, with not one client-day differing by so much as half a penny. The recompute was then run a third time, after the database-wide re-key had finished, and produced the identical figure again. Numbers that survive being derived twice from separate restores are not artefacts of how the measurement was set up.</p>
</div>
<div class="callout warn">
<span class="h">A bug in this work, found by measuring rather than reading</span>
<p>The first attempt at copying shared payments derived each payment's Square identifier by stripping a fixed prefix. That is right the first time a payment is seen, but once it has been re-keyed to one client, a second order meeting it later read the already-scoped key as the identifier and scoped it twice — <span class="mono">NGCD-CD-NGCC-CC-&lt;id&gt;</span>. The importer then created a fresh payment, doubling the tender on five clients by $3,000–$7,000 each. It was caught because the totals were absurd, not because the code looked wrong. The fix recovers the scope from the record itself; client codes contain dashes, so it cannot be done by pattern. A test now runs the step one order at a time, which is the arrangement that exposes it.</p>
</div>
</section>
<section>
<h2>What is still out of balance</h2>
<div class="measure">
<p>123 client-days out of 18,900, totalling <strong>$2,970.35</strong> — and only 33 of those are above ten cents. Where it sits matters more than the total.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Where the remainder sits</th><th class="n">Days</th><th class="n">Variance</th></tr></thead>
<tbody>
<tr><td>The twenty client records tied to a shared location — including the ten now-dormant duplicates</td><td class="n">20</td><td class="n">$2,467.39</td></tr>
<tr class="total"><td>Every other client</td><td class="n">103</td><td class="n">$502.96</td></tr>
</tbody>
</table>
</div>
<div class="scroll">
<table>
<thead><tr><th>Client</th><th class="n">Days</th><th class="n">Variance</th><th>What it is</th></tr></thead>
<tbody>
<tr><td class="mono">NGDL</td><td class="n">3</td><td class="n">$859.57</td><td>order on one record, its payment on the twin</td></tr>
<tr><td class="mono">NGDU</td><td class="n">3</td><td class="n">$345.96</td><td>payout fees on days with no trading</td></tr>
<tr><td class="mono">NGBR</td><td class="n">1</td><td class="n">$299.42</td><td rowspan="2">Square recorded $6,358.99 of tender against $6,059.57 of order totals on 2026-08-06 — the gap itself, not a summary fault</td></tr>
<tr><td class="mono">NGBK</td><td class="n">1</td><td class="n">$299.42</td></tr>
<tr><td class="mono">NGMV</td><td class="n">5</td><td class="n">$259.38</td><td>late May, undiagnosed</td></tr>
<tr><td class="mono">NGEB</td><td class="n">4</td><td class="n">$199.09</td><td>ezCater fee treatment — an open question</td></tr>
<tr><td class="mono">NGVC</td><td class="n">3</td><td class="n">$173.81</td><td>payout fees on days with no trading</td></tr>
<tr><td class="mono">NGWN</td><td class="n">3</td><td class="n">$170.28</td><td>payout fees on days with no trading</td></tr>
<tr class="total"><td class="dim">everyone else</td><td class="n dim">100</td><td class="n dim">$363.42</td><td class="dim">90 of these are till rounding — pennies a day</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p><strong>What is left divides cleanly in two.</strong> 108 days totalling $1,995.36 are real trading days with genuine discrepancies — the NGBR/NGBK tender gap, the ezCater fee question, and the unexplained clusters. The other 15 days, $974.99, carry a processing-fee debit and nothing else: a payout fee landing on a day the restaurant did not trade. That is the same shape as the refund-only day and could be closed the same way, but the offsetting entry belongs to the payout rather than to sales, so it needs the payout side modelled rather than another rule in the summary. It is not worth holding this work for.</p>
<p>The clusters on NGMV and NGEB are unexplained and worth a look, though at $458 across nine days they are no longer urgent. They are visible only because the window is ninety days; a thirty-day view does not reach them.</p>
</div>
</section>
<section>
<h2>How complete is this</h2>
<div class="measure">
<p>The importer understands both the old and new record names, so the change can be deployed
before the renaming finishes. That tolerance is a bridge, not a destination — while any
record still carries an unscoped name, two clients can land on it and the guarantee rests on
a convention rather than on the data. So the renaming was run to completion and measured.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Record type</th><th class="n">Total</th><th class="n">Client-scoped</th><th class="n">Still to rename</th><th class="n">Cannot be scoped</th></tr></thead>
<tbody>
<tr><td>Card payments</td><td class="n">17,047,142</td><td class="n good">17,047,142</td><td class="n good">0</td><td class="n good">0</td></tr>
<tr><td>Refunds</td><td class="n">51,986</td><td class="n good">51,986</td><td class="n good">0</td><td class="n good">0</td></tr>
<tr><td>Payouts</td><td class="n">144,688</td><td class="n good">144,652</td><td class="n good">0</td><td class="n">36</td></tr>
<tr><td>Cash-drawer shifts</td><td class="n">69,291</td><td class="n good">69,291</td><td class="n good">0</td><td class="n good">0</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>Every record in the database now carries its owner's name, and the migration proposes no
further changes: asked what is left to do, it answers zero on all four record types. The 36
payouts are ones with no client or location recorded anywhere, on the record itself or on
anything referring to it, so there is nothing to name them after.</p>
<p>Renaming had to be driven from orders, because a payment's rightful owner is whichever
order refers to it — so completing it meant walking all <strong>19,040,785 orders</strong>, not
just the clients that look shared today. Nine client pairs contended in the past without
sharing a location now, and a migration scoped to the current configuration would have missed
every one of them.</p>
<p><strong>A caution about how completeness is counted.</strong> 282,649 card payments carry no
client attribute of their own — they are stubs the payout path creates, never referenced by an
order. A gate that checks the attribute reports these as "no owner" and looks like a gap. They
are not: their names are scoped, recovered from the deposit that holds them. The figures above
are counted the harder way, by asking the migration what it would still change, which resolves
each record's owner through whatever refers to it. Reading the attribute alone would have
understated completeness by a quarter of a million records — and an early draft of this report
did exactly that.</p>
</div>
<div class="scroll">
<table>
<thead><tr><th>Shared payments after the migration</th><th class="n">Count</th><th>Meaning</th></tr></thead>
<tbody>
<tr><td><strong>Owned by more than one order</strong></td><td class="n good">0</td><td>whether the orders belong to different clients or the same one</td></tr>
<tr><td class="dim">checked across</td><td class="n dim">400,000 orders</td><td class="dim">spread through the whole database</td></tr>
</tbody>
</table>
</div>
<div class="measure">
<p>The problem this work exists to solve is gone: no payment answers to two orders, so the
component relationship means what it says and deleting an order can no longer take another
order's money with it.</p>
<p><strong>One behaviour changed when the pass was run over everything, and it is worth
recording.</strong> Where Square splits one tender across two of a single client's own orders,
the earlier design left the payment shared on purpose — both orders compute the same name, so
there is no second name for a copy to take. That rule only holds for two orders processed in
the same batch. Run across nineteen million orders in batches of two thousand, such pairs
almost always fall in different batches and the second order now takes a copy. Inside the
ninety-day window this changed nothing measurable: the recompute after the database-wide pass
matched the one before it to the cent. Outside the window it has not been measured, and it
should be before this runs against production.</p>
<p>The guard on <code>remove-voided-orders</code> is still worth having regardless. It is
cheap, and it makes the safety a property of the deletion rather than of the migration having
been run first.</p>
</div>
<div class="callout">
<span class="h">Re-running is safe, and that was proved at full scale</span>
<p>After the complete pass, asking the migration what it would change next returns
<strong>nothing</strong> — 17,047,142 payments examined, none to rename, none unscopable. A
record that already carries the right name is left untouched, so the migration can be stopped,
resumed, or repeated without consequence.</p>
<p>Its speed is worth a note for whoever schedules it: the whole nineteen million orders were
walked in about <strong>thirteen minutes</strong>. An earlier attempt appeared to be
transactor-bound and was projected at two days, which is why a previous run narrowed it to the
analysis window. That diagnosis was wrong. The bottleneck was garbage collection in the process
driving the migration — freeing held memory took an unrelated recompute from 17 client-days a
minute to 4,515. Nothing about the database or the transactor needed to change.</p>
</div>
<div class="measure">
<p><strong>What follows from the gate reading zero.</strong> <code>unscoped-report</code> counts
these figures on demand. Now that unscoped is zero across the board, the importer's
understanding of the old name form can be removed — at which point two clients sharing a
location becomes structurally incapable of producing a shared record, rather than prevented by
a convention that a future import could quietly break. That removal is the one remaining step
of this piece of work.</p>
</div>
</section>
<section>
<h2>Decisions and risks still open</h2>
<div class="scroll">
<table>
<thead><tr><th>Item</th><th>Who decides</th><th>Why it matters</th></tr></thead>
<tbody>
<tr><td>Which client record survives at each shared location</td><td>the business</td><td>the newer record generally has no history before the split, so keeping it loses years of the location's books</td></tr>
<tr><td>Which revenue account service charges post to</td><td>accounting</td><td>currently 49000 Service Income, chosen so the work could be measured; it affects reporting, never whether a day balances</td></tr>
<tr><td>Recognising a return on a refund-only day</td><td>accounting</td><td>posts to 41300 Returns, the account already used for returns; it moves the recognition date, so a refund settling after month end lands in the later period</td></tr>
<tr><td>Refunds on days that <em>did</em> trade</td><td>accounting</td><td>deliberately untouched — apportioning a return across a day that also sold is a judgement call, and 108 days remain out of balance because of it</td></tr>
<tr><td>Whether to correct records the wrong client already owns</td><td>the business</td><td>the fix stops future mix-ups; it does not retrospectively move records claimed while the configuration was shared</td></tr>
<tr><td><code>remove-voided-orders</code></td><td>engineering</td><td>safe once no payment has two parent orders; worth guarding regardless so it detaches rather than deletes</td></tr>
</tbody>
</table>
</div>
<div class="callout warn">
<span class="h">Two operational findings, unrelated to the summaries</span>
<p><strong>The production backup had not written a restore point since 2025-03-10</strong> — roughly seventeen months — even though data files were still uploading daily. A backup you cannot restore from is not a backup. A fresh one was taken on 2026-08-14 and is what this work used.</p>
<p><strong>The database server was sized for a toy dataset</strong>: a 2 GB cache against 27 GB of data. Worth checking what production is set to.</p>
<p><strong>Slowness here was misdiagnosed twice, in the same direction.</strong> Both a recompute crawling at 17 client-days a minute and a migration projected to take two days turned out to be garbage collection in the client process, not the database or the transactor. Freeing held memory took the recompute to 4,515 client-days a minute — a factor of 265 — and the migration finished in thirteen minutes. The lesson generalises: before concluding the transactor is the bottleneck, look at the heap of whatever is driving it.</p>
</div>
</section>
<section>
<h2>How to check any of this <span class="tech">technical</span></h2>
<div class="scroll">
<pre><span class="dim">;; the restored database, untouched production as of 2026-08-14 22:52</span>
(def conn (d/connect "datomic:dev://localhost:4337/integreat-prod-restore"))
<span class="dim">;; the two orders behind the worked examples</span>
(d/pull (d/db conn) '[*] [:sales-order/external-id
"square/order/NGLK-SM-OxSX9gpXJV394qqT8mnBGypUwKNZY"])
(d/pull (d/db conn) '[*] [:sales-order/external-id
"square/order/NTPT-PT-KrMZzcon1cpQEJUyetErkBIpcdEZY"])
<span class="dim">;; the gate: no payment may have two parent orders</span>
(rk/charges-with-multiple-parents (d/db conn) orders) <span class="dim">;; =&gt; 0</span>
<span class="dim">;; ownership history — which records ever changed client</span>
(->> (d/datoms (d/history (d/db conn)) :aevt :sales-refund/client)
(filter :added)
(reduce (fn [m d] (update m (:e d) (fnil conj #{}) (:v d))) {})
(filter (fn [[_ owners]] (&gt; (count owners) 1)))
count)</pre>
</div>
<div class="measure">
<p>The comparison tool is committed as <code>auto-ap.jobs.compare-sales-summaries</code>. Unit tests: <code>lein test auto-ap.jobs.sales-summaries-test auto-ap.square.core3-test auto-ap.jobs.rekey-square-external-ids-test</code>.</p>
</div>
<div class="callout warn">
<span class="h">Do not compare summaries with <code>as-of</code> — a correction to how this was measured</span>
<p>The obvious way to audit a recompute is to read the database at a point before it and diff:
Datomic keeps every past value, so no snapshot is needed. That is what
<code>compare-sales-summaries</code> was built to do, and for summary amounts it does not work.
<code>:ledger-mapped/amount</code>, <code>:ledger-mapped/ledger-side</code> and
<code>:ledger-mapped/account</code> are all declared <code>:db/noHistory true</code>, so
superseded values are discarded rather than retained. A historical read of a summary that has
since been recomputed can return its lines with the categories intact and the amounts simply
absent — which reads as a legitimate all-zero summary, not as an error.</p>
<p>Every figure in this report is therefore taken from a live read of the database immediately
after each pass, captured and stored outside it, and the before/after comparison is done
between those two captures. No historical read is involved anywhere in the numbers above. The
tool remains useful for categories and for which days changed; its docstring overstates what it
can recover, and that is worth correcting before someone relies on it for amounts.</p>
</div>
</section>
<footer>
<span>Measured 2026-08-15 against <span class="mono">integreat-prod-restore</span>, restored fresh from backup point 209608347 — production as of 2026-08-14 22:52. Nothing in production was read or written. Branch <span class="mono">worktree-sales-summary-balance</span>.</span>
<span>A day counts as out of balance when money taken minus money earned is half a penny or more. "Material" means ten cents or more, the threshold below which the residual is till rounding. Of the 123 remaining days, 33 are material.</span>
<span>Both the baseline and the result are live captures taken straight after their own recompute, never historical reads — see the note on <code>as-of</code> above.</span>
</footer>
</div>

217
docs/2026-08-15-sales-summary-rollout-plan.md Normal file
View File

@@ -0,0 +1,217 @@
# Sales-summary balancing — rollout plan
Steps to execute, in order. Every step is either reversible or verifiable before the next one
begins. Both behaviour changes are behind per-client feature flags that are **off by default**, so
merging and deploying this branch changes nothing on its own.
Measured on a restored copy of production (backup point `209608347`), 210 clients over
2026-05-10 → 2026-08-07: **1,258 client-days out of balance / $69,560.10 → 123 days / $2,970.35**,
with zero days knocked out of balance and zero already-balanced days altered.
---
## Before you start
| | |
|---|---|
| Branch | `worktree-sales-summary-balance` |
| Flags introduced | `summary-service-charges`, `summary-refund-only-returns` — both off by default |
| Migration to run once | `auto-ap.jobs.rekey-square-external-ids/migrate-all!` |
| Expected migration runtime | ~13 minutes for 19M orders on a warm cache |
| Nothing here touches | invoices, payments, the ledger, or any client without the flags set |
**One prerequisite that is not code.** Ten Square locations are configured against two client records
each. Someone in the business has to decide which record survives at each. The newer record usually
has no history from before the split, so keeping it loses years of that location's books. Do this
before step 3.
---
## Step 1 — Deploy the code
Deploy the branch as normal. Both flags are absent from every client, so:
- tips are calculated exactly as they are today,
- no `Service Charges` line is written,
- no `Returns` line is written on refund-only days.
The only changes that take effect immediately are the safe ones: imbalance logging, the
dirty-summary scan bounded to one client (1,321 ms → 5.6 ms per client), the schema-ordering fix,
and the importer's new client-scoped keys.
**The importer starts writing client-scoped keys straight away, and reads both schemes.** That is
deliberate and is what makes the deploy independent of the migration. Do not remove the legacy
lookup in `square.core3/existing-id` yet — see step 8.
**Verify before moving on.** After one nightly import cycle:
```clojure
;; refunds, payouts and shifts must not have doubled
(count (d/datoms (d/db conn) :aevt :sales-refund/external-id))
(count (d/datoms (d/db conn) :aevt :expected-deposit/external-id))
(count (d/datoms (d/db conn) :aevt :cash-drawer-shift/external-id))
```
Compare against the same counts taken immediately before deploy. Growth should be ordinary daily
volume. A near-doubling means the legacy fallback is not working — **stop and roll back the deploy**.
---
## Step 2 — Guard `remove-voided-orders`
Do this before the migration, not after. `:sales-order/charges` is `:db/isComponent true`, so
retracting an order cascades into its payments. Until step 3 finishes there are still payments with
two parent orders, and deleting one client's voided order can take the other client's payment with
it.
Either leave `remove-voided-orders` switched off until step 3 completes, or change it to detach a
payment that has more than one parent rather than delete it. Detaching is worth doing regardless —
it makes the safety a property of the deletion rather than of the migration having been run first.
See `docs/2026-08-15-remove-voided-orders-risk.md`.
---
## Step 3 — Retire the duplicate client records
Business decision from the top of this document. Deactivate the losing record's Square location so
the importer stops fetching for it. The record itself stays; its history is untouched.
**Verify:** no Square location is configured against two active client records.
---
## Step 4 — Run the migration
```clojure
(require '[auto-ap.jobs.rekey-square-external-ids :as rk])
;; read-only first — check :collisions is empty for every attribute
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
;; then the whole thing
(rk/migrate-all! 2000)
```
Runs in about thirteen minutes over 19M orders. It is **idempotent and resumable** — a record that
already carries the right name is skipped, so it can be stopped and re-run without consequence.
If it appears to crawl, the cause is almost certainly garbage collection in the process driving it,
not the transactor. That misdiagnosis cost two days of projected runtime during this work. Free
retained memory in the REPL and re-measure before changing anything about the database.
**Verify — all four must read zero to migrate and zero unscopable:**
```clojure
(rk/unscoped-report (d/db conn))
(dissoc (rk/plan (d/db conn) :charge/external-id rk/charge-prefix) :new-keys)
;; => {:total 17047142 :to-migrate 0 :already-scoped 17047142 :unscopable 0}
;; and the gate that this work exists for
(rk/charges-with-multiple-parents (d/db conn) (take 400000 (rk/all-order-ids (d/db conn))))
;; => 0
```
Note `unscoped-report`'s `:no-owner` column is not a gap: ~283k payout-stub payments carry no
`:charge/client` attribute of their own, so it cannot verify them by attribute. `plan` resolves
ownership through whatever refers to them and is the figure to trust.
---
## Step 5 — Recompute summaries, flags still off
```clojure
(require '[auto-ap.jobs.sales-summaries :as ss])
(ss/refresh-sales-summaries 90)
```
This is the pass that banks the deduplication. **Capture the result before going further** — you
will need it as the baseline for step 6, and it cannot be reconstructed afterwards:
```clojure
(require '[auto-ap.tools.compare-sales-summaries :as cmp]) ; test/dev classpath
(def before (cmp/summaries-in (d/db conn) start end))
(spit "before.edn" (pr-str before))
```
> **Do not use `d/as-of` to compare summary amounts.** `:ledger-mapped/amount`, `ledger-side` and
> `account` are `:db/noHistory`, so past values are discarded. A summary that has since been
> recomputed reads back through `as-of` with its amounts *absent*, which looks like a legitimate
> balanced day. Capture live, before and after, and diff the captures.
---
## Step 6 — Turn the flags on, a few restaurants at a time
Needs accounting sign-off first, on two points:
- `summary-service-charges` posts to **49000 Service Income**. Chosen so the work could be measured.
It affects reporting, never whether a day balances.
- `summary-refund-only-returns` posts to **41300 Returns**, the account already used for returns. It
moves the *recognition date* of a return onto the day the refund settled, so a refund settling
after month end lands in the later period.
```clojure
@(d/transact conn [{:db/id [:client/code "NGxx"]
:client/feature-flags ["summary-service-charges"
"summary-refund-only-returns"]}])
(ss/refresh-sales-summaries 90)
```
Start with two or three restaurants, confirm, then widen.
**Verify** against the capture from step 5:
```clojure
(def after (cmp/summaries-in (d/db conn) start end))
(cmp/compare-window ...) ; both arguments live database values, never as-of
```
The two numbers that matter — both were zero across all 18,900 client-days in testing:
- `:balanced->unbalanced` must be **0**
- previously-balanced days whose lines changed must be **0**
If either is non-zero, retract the flags for the affected clients and re-run step 5. The flags are
the rollback: removing them restores today's behaviour exactly.
---
## Step 7 — Re-enable `remove-voided-orders`
Safe once step 4's gate reads zero. Keep the detach-rather-than-delete guard from step 2.
---
## Step 8 — Remove the legacy key lookup
Only once `plan` reports `:to-migrate 0` and has stayed there through several import cycles. Drop
the second branch of `square.core3/existing-id`. At that point two clients sharing a location
becomes structurally incapable of producing a shared record, rather than prevented by a convention a
future import could break.
This is the last step and there is no hurry.
---
## What this will not fix
123 client-days over ninety days, $2,970.35, of which only 33 are above ten cents.
| | Days | Variance | |
|---|---:|---:|---|
| Real trading days with genuine discrepancies | 108 | $1,995.36 | the NGBR/NGBK tender gap, the ezCater fee question, unexplained clusters on NGMV and NGEB |
| Processing fee on a day with no trading | 15 | $974.99 | same shape as the refund-only day, but the offsetting entry belongs to the payout, not to sales |
Refunds landing on days that *did* trade are deliberately left alone. Apportioning a return across a
day that also sold is a judgement call, and guessing at it is how correct books get quietly
rewritten.
---
## Two operational findings, unrelated to the summaries
- **The production backup had not written a restore point since 2025-03-10** — about seventeen
months — although data files were still uploading daily. Worth an alert on restore-point age.
- **The database server is sized for a much smaller dataset**: a 2 GB cache against 27 GB of data.
Worth checking what production is set to.

21
src/clj/auto_ap/datomic.clj
View File

@@ -875,13 +875,22 @@
(defn all-schema [] (defn all-schema []
(edn/read-string (slurp (io/resource "schema.edn")))) (edn/read-string (slurp (io/resource "schema.edn"))))
(defn transact-schema [conn] (defn transact-schema
@(dc/transact conn "Installs the schema in two passes: every plain attribute first, then every composite tuple.
(edn/read-string (slurp (io/resource "schema.edn"))))
;; this is temporary for any new stuff that needs to be asserted for cloud migration. A tuple can only be created once the attributes it composes already exist, and the pieces are
@(dc/transact conn spread across both files — `:journal-entry-line/running-balance-tuple` lives in schema.edn
(edn/read-string (slurp (io/resource "cloud-migration-schema.edn"))))) while one of its members, `:journal-entry-line/running-balance`, lives in
cloud-migration-schema.edn. Transacting the files in order therefore cannot install that tuple
against an empty database. Long-lived databases never hit it because those attributes went in
years apart."
[conn]
(let [schema (concat (edn/read-string (slurp (io/resource "schema.edn")))
;; this is temporary for any new stuff that needs to be asserted for cloud migration.
(edn/read-string (slurp (io/resource "cloud-migration-schema.edn"))))
{tuples true plain false} (group-by #(contains? % :db/tupleAttrs) schema)]
(when (seq plain) @(dc/transact conn plain))
(when (seq tuples) @(dc/transact conn tuples))))
(defn backoff [n] (defn backoff [n]
(let [base-timeout 500 (let [base-timeout 500

62
src/clj/auto_ap/datomic/sales_summaries.clj Normal file
View File

@@ -0,0 +1,62 @@
(ns auto-ap.datomic.sales-summaries
(:require
[iol-ion.query :refer [dollars=]]))
(defn- ledger-side
"The ledger side of an item as a keyword, whether it arrived as a plain keyword (a
transaction map, or a pull using `:xform iol-ion.query/ident`) or as the `{:db/ident ...}`
map a plain `pull` returns. Resolving both shapes here matters: items whose side does not
compare equal are counted on neither side, which would leave a summary looking balanced at
zero and therefore silently accepted."
[item]
(let [side (:ledger-mapped/ledger-side item)]
(if (map? side) (:db/ident side) side)))
(defn- side-total [side items]
(->> items
(filter #(= side (ledger-side %)))
(map #(:ledger-mapped/amount % 0.0))
(reduce + 0.0)))
(defn total-debits [items]
(side-total :ledger-side/debit items))
(defn total-credits [items]
(side-total :ledger-side/credit items))
(defn fully-mapped? [items]
(every? :ledger-mapped/account items))
(defn fully-sided?
"Every item says which side of the ledger it belongs on. Guards `accepted?` against
reading a collection of sideless items as balanced at zero."
[items]
(every? #(#{:ledger-side/debit :ledger-side/credit} (ledger-side %)) items))
(defn balanced? [items]
(dollars= (total-debits items) (total-credits items)))
(defn imbalance
"Signed debits minus credits. `balanced?` answers yes or no; this says by how much and in
which direction, so a day that does not balance can be logged and queried rather than only
rendered red. Positive means the tender side exceeds what revenue accounts for."
[items]
(- (total-debits items) (total-credits items)))
(defn accepted?
"True once a summary is finished: every line is mapped to an account and debits equal
credits. This is the same condition the sales summaries grid renders as \"Balanced\", and
the condition the scheduled refresh treats as \"leave this alone\"."
[items]
(boolean (and (seq items)
(fully-mapped? items)
(fully-sided? items)
(balanced? items))))
(defn <-pulled-item
"Flattens the ref values on a pulled sales summary item back to the scalars a transaction
expects. `accepted?` reads either shape, so this is only needed on the write path."
[item]
(cond-> item
(map? (:ledger-mapped/ledger-side item)) (update :ledger-mapped/ledger-side :db/ident)
(map? (:ledger-mapped/account item)) (update :ledger-mapped/account :db/id)))

314
src/clj/auto_ap/jobs/rekey_square_external_ids.clj Normal file
View File

@@ -0,0 +1,314 @@
(ns auto-ap.jobs.rekey-square-external-ids
"One-shot migration re-keying Square entities to client-scoped external ids.
Refunds, charges, Square payouts (expected deposits) and cash drawer shifts all carry keys with
no client scoping, so two clients configured on the same Square location share a single entity:
its owner flips every time either client imports. Sales orders and ezCater orders already scope
their keys by client and location; this brings the rest in line.
Measured on a restored production backup, ownership had actually changed on 3,387 refunds,
4,069 expected deposits and 2,628 cash drawer shifts, across 19 distinct client pairs — nine of
which no longer share a location in the current configuration and so are invisible to any
point-in-time check.
Run AFTER the importer knows how to resolve both key schemes (`square.core3/existing-id`).
Running it first would be harmless, but the importer would then re-create legacy-keyed
entities on its next pass.
The migration is idempotent: an entity already carrying its scoped key is skipped, so it can
be re-run over a partially migrated database."
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.logging :as alog]
[datomic.api :as dc]))
(def refund-prefix "square/refund/")
(def charge-prefix "square/charge/")
(def deposit-prefix "square/payout/")
(def shift-prefix "square/cash-drawer-shift/")
(defn- scope-of
"`[client-code location]` for an entity, or nil when it cannot be determined.
Charges are the awkward case: about an eighth of them carry neither `:charge/client` nor
`:charge/location`. Those are stubs minted by the payout path, which asserts an external id
alone and lets unique-identity upsert bring a bare entity into being, plus older tender
records that predate the client attribute. None are orphaned, so the scope is recovered from
whatever references them — the sales order first, then the expected deposit."
[db attr e]
(let [ent (dc/entity db e)
pair (fn [code loc] (when (and code loc) [code loc]))]
(or (case attr
:sales-refund/external-id (pair (:client/code (:sales-refund/client ent))
(:sales-refund/location ent))
:charge/external-id (pair (:client/code (:charge/client ent))
(:charge/location ent))
:expected-deposit/external-id (pair (:client/code (:expected-deposit/client ent))
(:expected-deposit/location ent))
:cash-drawer-shift/external-id (pair (:client/code (:cash-drawer-shift/client ent))
(:cash-drawer-shift/location ent)))
(when-let [o (:e (first (dc/datoms db :vaet e :sales-order/charges)))]
(let [oe (dc/entity db o)]
(pair (:client/code (:sales-order/client oe)) (:sales-order/location oe))))
(when-let [d (:e (first (dc/datoms db :vaet e :expected-deposit/charges)))]
(let [de (dc/entity db d)]
(pair (:client/code (:expected-deposit/client de)) (:expected-deposit/location de)))))))
(defn planned-key
"`[eid new-key]` for an entity that still needs re-keying, or nil when it is already scoped or
cannot be scoped at all.
Detection compares against the key this entity *should* have rather than pattern-matching the
id, because Square ids may themselves contain dashes and no pattern separates the two schemes
reliably. That also makes the migration idempotent."
[db attr prefix datom]
(let [old (:v datom)]
(when-let [[code loc] (scope-of db attr (:e datom))]
(let [scoped-prefix (str prefix code "-" loc "-")]
(when-not (.startsWith ^String old scoped-prefix)
[(:e datom) (str scoped-prefix (subs old (count prefix)))])))))
(defn plan
"Everything the migration would change, plus what it cannot touch. Read-only — run this and
check `:collisions` is empty before transacting anything."
[db attr prefix]
(let [acc (reduce (fn [acc d]
(let [acc (update acc :total inc)]
(if-let [[e new-key] (planned-key db attr prefix d)]
(-> acc
(update :to-migrate inc)
(update :new-keys conj! [e new-key]))
(if (scope-of db attr (:e d))
(update acc :already-scoped inc)
(update acc :unscopable inc)))))
{:total 0 :to-migrate 0 :already-scoped 0 :unscopable 0 :new-keys (transient [])}
(dc/datoms db :aevt attr))]
(update acc :new-keys persistent!)))
(defn collisions
"Any two entities that would land on the same new key. Must be empty: a collision would merge
two entities into one and lose whichever lost."
[new-keys]
(->> new-keys
(group-by second)
(keep (fn [[k es]] (when (> (count es) 1) [k (mapv first es)])))
vec))
(defn migrate!
"Asserts the new external id on each planned entity. The attribute is cardinality one, so the
legacy value is retracted by the same assertion and the entity keeps its identity — nothing is
created and nothing is deleted.
Returns the number of entities re-keyed."
[attr new-keys batch-size]
(let [total (count new-keys)]
(alog/info ::migrating :attr attr :count total)
(doseq [[i batch] (map-indexed vector (partition-all batch-size new-keys))]
@(dc/transact conn (for [[e new-key] batch] {:db/id e attr new-key}))
(when (zero? (mod i 20))
(alog/info ::migrated :attr attr :done (* i batch-size) :of total)))
total))
(def charge-copy-attrs
"Everything a charge carries in its own right. `:charge/client+date` is a tuple Datomic
maintains, and `:charge/external-id` is set separately, so neither is copied."
[:charge/type-name :charge/total :charge/tip :charge/tax :charge/date
:charge/processor :charge/note :charge/reference-link])
(defn- raw-square-id
"The Square id inside a charge's external id, with any client scoping removed.
Stripping only the `square/charge/` prefix is not enough. Once a charge has been scoped to one
client, a second order processing the same charge would read `NGCC-CC-<id>` as the id and scope
it again, producing `square/charge/NGCD-CD-NGCC-CC-<id>`. The importer then computes the
correct single-scoped key, fails to find it, and creates a second charge — silently doubling
the tender.
Client codes may themselves contain dashes, so the scope cannot be recognised by pattern. It is
recovered from the entity instead: whoever the charge currently belongs to is exactly whose
scope its key carries."
[db charge old]
(let [ent (dc/entity db charge)
code (:client/code (:charge/client ent))
loc (:charge/location ent)
owner-prefix (when (and code loc) (str charge-prefix code "-" loc "-"))]
(cond
(and owner-prefix (.startsWith ^String old ^String owner-prefix)) (subs old (count owner-prefix))
(.startsWith ^String old charge-prefix) (subs old (count charge-prefix))
:else old)))
(defn- charge-plan-for-order
"What one order needs doing to its charges.
`:keep` — no other order has claimed this charge yet, so this order takes it and it is renamed
in place. `:clone` — another order already owns it, so this order needs its own copy.
Splitting is per client, not per order. Where two orders of the SAME client and location refer
to one payment — Square splitting a tender across orders, or an amendment — they are left
sharing it deliberately. Both would compute the same name, so there is no second name to give
a copy, and more importantly a copy would double that client's takings for the day. The
component cascade still applies to those, which is why the retraction guard on
`remove-voided-orders` is needed regardless of this migration.
Whether a charge is claimed is read from the charge itself: an unclaimed one still carries the
bare `square/charge/<id>` form. Deciding it that way rather than by remembering
every charge seen so far is what lets this run across all 16 million of them — the alternative
needs a map of the entire table in memory. `batch-seen` covers only the orders inside one
transaction, where the database snapshot cannot yet show a claim made moments earlier."
[db order-eid batch-seen]
(let [oe (dc/entity db order-eid)
code (:client/code (:sales-order/client oe))
loc (:sales-order/location oe)]
(when (and code loc)
(for [d (dc/datoms db :eavt order-eid :sales-order/charges)
:let [charge (:v d)
old (:v (first (dc/datoms db :eavt charge :charge/external-id)))]
:when old
:let [raw (raw-square-id db charge old)
new-key (str charge-prefix code "-" loc "-" raw)
claimed (get @batch-seen charge)
unclaimed? (and (= old (str charge-prefix raw)) (nil? claimed))]
;; nothing to do when the charge already answers to this order's name, or when
;; another order in this same batch has just claimed it under that very name —
;; that is the same-client case, which stays shared
:when (and (not= old new-key) (not= claimed new-key))]
{:order order-eid :charge charge :new-key new-key :raw raw
:client (:db/id (:sales-order/client oe)) :location loc
:action (if unclaimed? :keep :clone)}))))
(defn split-and-rekey-charges!
"Gives every order its own charge entity, keyed by that order's client and location.
Where two clients were configured on one Square location, both clients' orders resolved to a
single charge, because charge keys carried no client. Re-keying alone does not undo that — it
hands the one entity to whichever client is looked at first and leaves the other order pointing
at a charge it does not own. Since `:sales-order/charges` is a component attribute, that is not
merely untidy: retracting either order would delete a charge the other one still needs.
So a shared charge is cloned. The first order to claim it keeps it, re-keyed to that order's
scope; every other order gets a copy carrying the same amounts, scoped to its own client, and
has its reference repointed. Afterwards no charge has more than one parent order and the
component relationship means what it says.
`orders` is the collection of order entity ids to process — typically every order belonging to
the clients that share, or have ever shared, a Square location."
[orders batch-size]
(let [cloned (atom 0)
rekeyed (atom 0)]
(doseq [batch (partition-all batch-size orders)]
(let [db (dc/db conn)
batch-seen (atom {})
tx (doall
(for [o batch
plan (charge-plan-for-order db o batch-seen)
:let [{:keys [charge new-key action client location]} plan]
tx-item (if (= :keep action)
(do (swap! batch-seen assoc charge new-key)
(swap! rekeyed inc)
;; record who claimed it: the owner is how a later order
;; recovers the Square id from an already-scoped key
[{:db/id charge
:charge/external-id new-key
:charge/client client
:charge/location location}])
(let [ent (dc/entity db charge)
copy (reduce (fn [m a] (if-some [v (get ent a)]
(assoc m a (if (map? v) (:db/id v) v))
m))
{} charge-copy-attrs)]
(swap! cloned inc)
[(assoc copy
:db/id new-key
:charge/external-id new-key
:charge/client client
:charge/location location)
[:db/retract o :sales-order/charges charge]
{:db/id o :sales-order/charges new-key}]))]
tx-item))]
(when (seq tx) @(dc/transact conn tx))))
(alog/info ::split-charges :rekeyed @rekeyed :cloned @cloned)
{:rekeyed @rekeyed :cloned @cloned}))
(defn charges-with-multiple-parents
"The §3.3 gate. Must read zero once the split has run: while any charge has two parent orders,
retracting either order deletes the other one's payment."
[db orders]
(->> orders
(mapcat (fn [o] (map :v (dc/datoms db :eavt o :sales-order/charges))))
distinct
(filter (fn [c] (> (reduce (fn [n _] (inc n)) 0 (dc/datoms db :vaet c :sales-order/charges)) 1)))
count))
(def scoped-attrs
"Every Square-imported entity whose key must carry its client, with the prefix and where to
read the owner from."
[{:attr :sales-refund/external-id :prefix refund-prefix
:client :sales-refund/client :location :sales-refund/location}
{:attr :charge/external-id :prefix charge-prefix
:client :charge/client :location :charge/location}
{:attr :expected-deposit/external-id :prefix deposit-prefix
:client :expected-deposit/client :location :expected-deposit/location}
{:attr :cash-drawer-shift/external-id :prefix shift-prefix
:client :cash-drawer-shift/client :location :cash-drawer-shift/location}])
(defn unscoped-report
"Counts, per entity type, how many keys are already client-scoped, how many still carry the
legacy unscoped form, and how many have no owner to scope by.
This is the completeness gate. The importer tolerates both key schemes on purpose, so that the
change can be deployed before the migration finishes — but that tolerance is a transition, not
a resting place. While `:legacy` is above zero the database is in a mixed state and a stray
unscoped record can still be adopted by whichever client imports it first. Once every count
reads zero the fallback lookup in `square.core3/existing-id` can be removed and the guarantee
becomes structural rather than conventional."
[db]
(into {}
(for [{:keys [attr prefix client location]} scoped-attrs]
[attr (reduce (fn [acc d]
(let [e (dc/entity db (:e d))
code (:client/code (client e))
loc (location e)
scoped (when (and code loc) (str prefix code "-" loc "-"))]
(cond
(and scoped (.startsWith ^String (:v d) ^String scoped)) (update acc :scoped inc)
(nil? scoped) (update acc :no-owner inc)
:else (update acc :legacy inc))))
{:scoped 0 :legacy 0 :no-owner 0}
(dc/datoms db :aevt attr))])))
(defn all-order-ids
"Every sales order in the database, streamed."
[db]
(map :e (dc/datoms db :aevt :sales-order/external-id)))
(defn migrate-all!
"The complete migration, over the whole database rather than a chosen subset.
Splitting is driven from orders, because a payment's rightful owner is whichever order refers
to it — so every order has to be walked, not merely the clients that share a location today.
Nine client pairs contended in the past and no longer share one; their records are still mixed,
and a migration scoped to the current configuration would miss every one of them.
Returns the split counts and the completeness report, which should read zero legacy across the
board when this finishes."
[batch-size]
(let [split (split-and-rekey-charges! (all-order-ids (dc/db conn)) batch-size)]
(doseq [{:keys [attr prefix]} scoped-attrs
:when (not= attr :charge/external-id)]
(let [p (plan (dc/db conn) attr prefix)]
(when-let [c (seq (collisions (:new-keys p)))]
(throw (ex-info "two entities would take the same key" {:attr attr :collisions (count c)})))
(migrate! attr (:new-keys p) batch-size)))
;; charges no order refers to — payout stubs — are scoped from the deposit that holds them
(let [p (plan (dc/db conn) :charge/external-id charge-prefix)]
(when (seq (:new-keys p)) (migrate! :charge/external-id (:new-keys p) batch-size)))
{:split split :completeness (unscoped-report (dc/db conn))}))
(defn counts
"Entity totals, for the before/after assertion that is this migration's real safety net: if
either number moves, the re-key created duplicates instead of updating in place."
[db]
{:refunds (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :sales-refund/external-id))
:charges (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :charge/external-id))
:deposits (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :expected-deposit/external-id))
:shifts (reduce (fn [n _] (inc n)) 0 (dc/datoms db :aevt :cash-drawer-shift/external-id))})

541
src/clj/auto_ap/jobs/sales_summaries.clj
View File

@@ -1,5 +1,6 @@
(ns auto-ap.jobs.sales-summaries (ns auto-ap.jobs.sales-summaries
(:require [auto-ap.datomic :refer [conn]] (:require [auto-ap.datomic :refer [conn]]
[auto-ap.datomic.sales-summaries :as d-ss]
[auto-ap.jobs.core :refer [execute]] [auto-ap.jobs.core :refer [execute]]
[auto-ap.logging :as alog] [auto-ap.logging :as alog]
[auto-ap.time :as atime] [auto-ap.time :as atime]
@@ -8,6 +9,7 @@
[clj-time.periodic :as per] [clj-time.periodic :as per]
[clojure.string :as str] [clojure.string :as str]
[com.brunobonacci.mulog :as mu] [com.brunobonacci.mulog :as mu]
[config.core :refer [env]]
[datomic.api :as dc])) [datomic.api :as dc]))
(defn mark-dirty [client start end] (defn mark-dirty [client start end]
@@ -39,26 +41,108 @@
(dc/db conn) (dc/db conn)
number))) number)))
(defn delete-all [] (defn delete-all []
@(dc/transact-async conn @(dc/transact-async conn
(->> (->>
(dc/q '[:find ?ss (dc/q '[:find ?ss
:where [?ss :sales-summary/date]] :where [?ss :sales-summary/date]]
(dc/db conn)) (dc/db conn))
(map (fn [[ ss]] (map (fn [[ss]]
[:db/retractEntity ss]))))) [:db/retractEntity ss])))))
(def item-read
"Enough of a summary item to both evaluate `d-ss/accepted?` and transact the item back
unchanged. `:db/id` matters: `:sales-summary/items` is a component attribute upserted via
`[:reset-rels ...]`, so an item re-transacted without its id is deleted and recreated."
'[:db/id
:sales-summary-item/category
:sales-summary-item/sort-order
:sales-summary-item/manual?
:ledger-mapped/amount
{:ledger-mapped/ledger-side [:db/ident]}
{:ledger-mapped/account [:db/id]}])
(defn dirty-sales-summaries
"The client's dirty summaries, with enough of each item to evaluate and re-transact it.
(defn dirty-sales-summaries [c] `index-pull` returns a lazy seq running from `:start` to the END of the index, so this must
stop at the client boundary rather than filter: `:sales-summary/client+dirty` sorts by client
first, so every later client's summaries sit beyond this client's and filtering would walk all
of them — for every client — pulling their items on the way. That is quadratic in the number of
summaries, and it showed up as a full refresh degrading from ~180 client-days a minute to ~3 as
the summary count grew."
[c]
(let [client-id (dc/entid (dc/db conn) c)] (let [client-id (dc/entid (dc/db conn) c)]
(->> (dc/index-pull (dc/db conn) (->> (dc/index-pull (dc/db conn)
{:index :avet {:index :avet
:selector '[:sales-summary/date :sales-summary/client :db/id] :selector (conj '[:sales-summary/date :sales-summary/client :db/id]
{:sales-summary/items item-read})
:start [:sales-summary/client+dirty [client-id true]]}) :start [:sales-summary/client+dirty [client-id true]]})
(filter (fn [sales-summary] (take-while (fn [sales-summary]
(= client-id (:db/id (:sales-summary/client sales-summary)))))))) (= client-id (:db/id (:sales-summary/client sales-summary))))))))
(def default-days
"How far back the scheduled refresh looks for summaries that still need recomputing."
7)
(defn trailing-window
"`[start end)` covering the last `days` business days, ending with today. `end` is
exclusive, matching both `periodic-seq`'s 3-arity and the grid's date filters, so
`(trailing-window 7)` is day -6 through today inclusive."
[days]
[(.toDateMidnight (atime/localize (time/minus (time/now) (time/days (dec days)))))
(.toDateMidnight (atime/localize (time/plus (time/now) (time/days 1))))])
(defn accepted-client+dates
"Set of `[client-id date]` pairs in `[start end)` whose summary is already accepted, and
so should be left alone rather than re-marked. Accepted means balanced with every line
mapped to an account — the condition the grid renders as \"Balanced\"."
[db start end]
(->> (dc/q '[:find (pull ?ss selector)
:in $ ?start ?end selector
:where
[?ss :sales-summary/date ?d]
[(>= ?d ?start)]
[(< ?d ?end)]]
db
(c/to-date start)
(c/to-date end)
(conj '[:sales-summary/date {:sales-summary/client [:db/id]}]
{:sales-summary/items item-read}))
(map first)
(filter #(d-ss/accepted? (map d-ss/<-pulled-item (:sales-summary/items %))))
(map (juxt (comp :db/id :sales-summary/client) :sales-summary/date))
set))
(defn mark-stale-dirty
"Marks every client/day in the trailing `days` window dirty so `sales-summaries-v2` will
recompute it, skipping days whose summary is already accepted. Because accepted is derived
rather than stored, a summary that later falls out of balance is picked up again on the
next run. Returns the number of client/days marked."
[days]
(let [db (dc/db conn)
[start end] (trailing-window days)
accepted (accepted-client+dates db start end)
clients (map first (dc/q '[:find ?c
:in $
:where [_ :sales-order/client ?c]]
db))
dates (map c/to-date (per/periodic-seq start end (time/days 1)))
tx-data (for [client clients
date dates
:when (not (accepted [client date]))]
{:sales-summary/client client
:sales-summary/date date
:sales-summary/dirty true
:sales-summary/client+date [client date]})]
(alog/info ::marking-dirty
:days days
:client-count (count clients)
:accepted-count (count accepted)
:marked (count tx-data))
(doseq [batch (partition-all 500 tx-data)]
@(dc/transact conn batch))
(count tx-data)))
(defn- get-fee [c date] (defn- get-fee [c date]
(- (or (ffirst (dc/q '[:find ?f (- (or (ffirst (dc/q '[:find ?f
@@ -72,8 +156,18 @@
date)) date))
0.0))) 0.0)))
(def service-charges-account
"Where a credited Square service charge lands. 49000 is the existing \"Service Income\"
revenue account, which is the closest fit for auto-gratuity and catering fees.
NEEDS ACCOUNTING SIGN-OFF before `service-charges-flag` is enabled for any client: the wrong
account misstates revenue, and a category with no account at all keeps a day from ever
reaching accepted, since `accepted?` requires every line to be mapped."
49000)
(def name->number (def name->number
{"gyros and pitas" 40111 {"gyros and pitas" 40111
"service charges" service-charges-account
"returns" 41300 "returns" 41300
"card payments" 75460 "card payments" 75460
"cash payments" 75452 "cash payments" 75452
@@ -99,53 +193,53 @@
"food app refunds" 41400}) "food app refunds" 41400})
(defn get-payment-items [c date] (defn get-payment-items [c date]
(->> (->>
(dc/q '[:find ?processor ?type-name (sum ?total) (dc/q '[:find ?processor ?type-name (sum ?total)
:with ?c :with ?c
:in $ [?clients ?start-date ?end-date] :in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] :where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/charges ?c] [?e :sales-order/charges ?c]
[?c :charge/type-name ?type-name] [?c :charge/type-name ?type-name]
(or-join [?c ?processor] (or-join [?c ?processor]
(and [?c :charge/processor ?p] (and [?c :charge/processor ?p]
[?p :db/ident ?processor]) [?p :db/ident ?processor])
(and (and
(not [?c :charge/processor]) (not [?c :charge/processor])
[(ground :ccp-processor/na) ?processor])) [(ground :ccp-processor/na) ?processor]))
[?c :charge/total ?total]] [?c :charge/total ?total]]
(dc/db conn) (dc/db conn)
[[c] date date]) [[c] date date])
(reduce (reduce
(fn [acc [processor type-name total]] (fn [acc [processor type-name total]]
(update (update
acc acc
(cond (= type-name "CARD") (cond (= type-name "CARD")
"Card Payments" "Card Payments"
(= type-name "CASH") (= type-name "CASH")
"Cash Payments" "Cash Payments"
(#{"SQUARE_GIFT_CARD" "WALLET" "GIFT_CARD"} type-name) (#{"SQUARE_GIFT_CARD" "WALLET" "GIFT_CARD"} type-name)
"Gift Card Payments" "Gift Card Payments"
(#{:ccp-processor/toast (#{:ccp-processor/toast
#_:ccp-processor/ezcater #_:ccp-processor/ezcater
#_:ccp-processor/koala #_:ccp-processor/koala
:ccp-processor/doordash :ccp-processor/doordash
:ccp-processor/grubhub :ccp-processor/grubhub
:ccp-processor/uber-eats} processor) :ccp-processor/uber-eats} processor)
"Food App Payments" "Food App Payments"
:else :else
"Unknown") "Unknown")
(fnil + 0.0) (fnil + 0.0)
total)) total))
{}) {})
(map (fn [[k v]] (map (fn [[k v]]
{:db/id (str (java.util.UUID/randomUUID)) {:db/id (str (java.util.UUID/randomUUID))
:sales-summary-item/sort-order 0 :sales-summary-item/sort-order 0
:sales-summary-item/category k :sales-summary-item/category k
:ledger-mapped/amount (if (= "Card Payments" k) :ledger-mapped/amount (if (= "Card Payments" k)
(- v (get-fee c date)) (- v (get-fee c date))
v) v)
:ledger-mapped/ledger-side :ledger-side/debit})))) :ledger-mapped/ledger-side :ledger-side/debit}))))
(defn get-discounts [c date] (defn get-discounts [c date]
(when-let [discount (ffirst (dc/q '[:find (sum ?discount) (when-let [discount (ffirst (dc/q '[:find (sum ?discount)
@@ -162,7 +256,7 @@
:ledger-mapped/ledger-side :ledger-side/debit})) :ledger-mapped/ledger-side :ledger-side/debit}))
(defn get-refund-items [c date] (defn get-refund-items [c date]
(->> (->>
(dc/q '[:find ?type-name (sum ?t) (dc/q '[:find ?type-name (sum ?t)
:with ?e :with ?e
:in $ [?clients ?start-date ?end-date] :in $ [?clients ?start-date ?end-date]
@@ -173,26 +267,24 @@
(dc/db conn) (dc/db conn)
[[c] date date]) [[c] date date])
(reduce (reduce
(fn [acc [type-name total]] (fn [acc [type-name total]]
(update (update
acc acc
(cond (= type-name "CARD") (cond (= type-name "CARD")
"Card Refunds" "Card Refunds"
(= type-name "CASH") (= type-name "CASH")
"Cash Refunds" "Cash Refunds"
:else :else
"Food App Refunds") "Food App Refunds")
(fnil + 0.0) (fnil + 0.0)
total)) total))
{}) {})
(map (fn [[k v]] (map (fn [[k v]]
{:db/id (str (java.util.UUID/randomUUID)) {:db/id (str (java.util.UUID/randomUUID))
:sales-summary-item/sort-order 3 :sales-summary-item/sort-order 3
:sales-summary-item/category k :sales-summary-item/category k
:ledger-mapped/amount v :ledger-mapped/amount v
:ledger-mapped/ledger-side :ledger-side/credit})))) :ledger-mapped/ledger-side :ledger-side/credit}))))
(defn get-fees [c date] (defn get-fees [c date]
(when-let [fee (get-fee c date)] (when-let [fee (get-fee c date)]
@@ -219,21 +311,48 @@
[[c] date date])) [[c] date date]))
0.0)}) 0.0)})
(defn- get-tip [c date] (defn- tendered-tip
"Tips read off the tenders, which is where a tip actually settles."
[c date]
(or (ffirst (dc/q '[:find (sum ?tip)
:with ?c
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/charges ?c]
[?c :charge/tip ?tip]]
(dc/db conn)
[[c] date date]))
0.0))
(defn- untendered-tip
"Tips on orders that carry no tender at all. A return-only order reverses its tip on
`:sales-order/tip` but has no charge to join through, so the reversal is invisible to
`tendered-tip` and the day ends up crediting a tip that was handed back."
[c date]
(or (ffirst (dc/q '[:find (sum ?tip)
:with ?e
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/tip ?tip]
(not [?e :sales-order/charges])]
(dc/db conn)
[[c] date date]))
0.0))
(defn- get-tip
"Tendered tips plus the tips on untendered orders. Additive rather than substitutive on
purpose: where an order does have a tender, the tender is the correct source, and real
orders exist whose tender carries a tip their `:sales-order/tip` does not — auto-gratuity
booked as a service charge, and wallet tips absent from the net amounts. Reading the order
instead of the tender would drop those."
[c date]
{:ledger-mapped/ledger-side :ledger-side/credit {:ledger-mapped/ledger-side :ledger-side/credit
:sales-summary-item/sort-order 2 :sales-summary-item/sort-order 2
:db/id (str (java.util.UUID/randomUUID)) :db/id (str (java.util.UUID/randomUUID))
:sales-summary-item/category "Tip" :sales-summary-item/category "Tip"
:ledger-mapped/amount (or (ffirst (dc/q '[:find (sum ?tip) :ledger-mapped/amount (+ (tendered-tip c date)
:with ?c (untendered-tip c date))})
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/charges ?c]
[?c :charge/tip ?tip]]
(dc/db conn)
[[c] date date]))
0.0)})
(defn- get-sales [c date] (defn- get-sales [c date]
(let [sales (->> (dc/q '[:find ?category (sum ?total) (sum ?tax) (sum ?discount) (let [sales (->> (dc/q '[:find ?category (sum ?total) (sum ?tax) (sum ?discount)
@@ -259,36 +378,165 @@
:ledger-mapped/amount (- (+ total discount) tax) :ledger-mapped/amount (- (+ total discount) tax)
#_#_:ledger-mapped/account nil}))) #_#_:ledger-mapped/account nil})))
(defn- get-returns [c date] (def refund-only-returns-flag
(when-let [amount (ffirst (dc/q '[:find (sum ?r) "Per-client rollout lever for recognising a return on a day that carries refunds but no sales,
:with ?e in the same style as `service-charges-flag`. Absent, the summary behaves exactly as it does
:in $ [?clients ?start-date ?end-date] today."
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]] "summary-refund-only-returns")
[?e :sales-order/returns ?r]
#_[?e :sales-order/charges ?c] (defn- refund-only-returns-enabled? [c]
#_[?c :charge/tax ?tax]] (contains? (set (:client/feature-flags (dc/pull (dc/db conn) '[:client/feature-flags] c)))
(dc/db conn) refund-only-returns-flag))
[[c] date date]))]
(defn- traded?
"Whether the client recorded any sales order at all on `date`."
[c date]
(some? (ffirst (dc/q '[:find (count ?e)
:in $ [?clients ?start-date ?end-date]
:where
[(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]]
(dc/db conn)
[[c] date date]))))
(defn refunded-total
"Face value of every refund recorded for the client on `date`.
This is the same population the `Card Refunds` / `Cash Refunds` / `Food App Refunds` credits are
built from, read back as one figure so a return can be posted against it."
[c date]
(or (ffirst (dc/q '[:find (sum ?t)
:with ?e
:in $ [?clients ?start-date ?end-date]
:where
[(iol-ion.query/scan-sales-refunds $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-refund/total ?t]]
(dc/db conn)
[[c] date date]))
0.0))
(defn- refund-only-returns
"The return owed by a day that recorded refunds and no sales at all.
Refunds are credited on the day the money goes back; the return that offsets them is read from
`:sales-order/returns` on orders scanned for that same day. When the two fall on the same day
they cancel. When they do not — a refund settling after the restaurant closed for the day, or a
retired duplicate client record that still receives refunds while its orders belong to the
surviving record — the credit has nothing to offset it and the day cannot balance, by exactly
the refunded amount.
Only days with *no* sales orders are touched. That guard is what makes this safe rather than a
judgement call: with no orders there is no order-derived return to double-count, and no trading
day can have its figures moved. Days that do trade and still carry an unmatched refund are left
alone deliberately — apportioning a return across a day that also sold is an accounting
question, not a mechanical one."
[c date]
(when (and (refund-only-returns-enabled? c)
(not (traded? c date)))
(let [amount (refunded-total c date)]
(when-not (zero? amount) amount))))
(defn- get-returns [c date]
(when-let [amount (or (ffirst (dc/q '[:find (sum ?r)
:with ?e
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/returns ?r]
#_[?e :sales-order/charges ?c]
#_[?c :charge/tax ?tax]]
(dc/db conn)
[[c] date date]))
(refund-only-returns c date))]
{:db/id (str (java.util.UUID/randomUUID)) {:db/id (str (java.util.UUID/randomUUID))
:sales-summary-item/category "Returns" :sales-summary-item/category "Returns"
:ledger-mapped/amount amount :ledger-mapped/amount amount
:ledger-mapped/ledger-side :ledger-side/debit})) :ledger-mapped/ledger-side :ledger-side/debit}))
(defn sales-summaries-v2 [] (def service-charges-flag
(doseq [[c client-code] (dc/q '[:find ?c ?client-code "Per-client rollout lever for crediting Square service charges, in the same style as
:in $ `new-square` and `import-custom-amount`. Absent, the summary behaves exactly as it does
:where [?c :client/code ?client-code]] today."
(dc/db conn)) "summary-service-charges")
{:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)]
(defn- service-charges-enabled? [c]
(contains? (set (:client/feature-flags (dc/pull (dc/db conn) '[:client/feature-flags] c)))
service-charges-flag))
(defn service-charge-total
"Square service charges for the day, both signs.
A service charge is collected inside the card tender but nothing credits it, so every order
carrying one leaves the day short by exactly that amount. Both signs matter: a returned
catering fee arrives as a negative service charge and is subtracted back out of
`:sales-order/returns`, so dropping negatives would lose the reversal.
The vendor gate is load-bearing — ezCater service charges are commission deducted from the
restaurant rather than collected from the diner, and crediting those would make things worse.
It matches on `:sales-order/vendor` where that is set and falls back to the external id
prefix where it is not, because whole eras of Square orders carry no vendor attribute at all
and a gate on vendor alone silently credits nothing.
Kept separate from the rollout flag so the arithmetic can be measured on its own."
[c date]
(ffirst (dc/q '[:find (sum ?service-charge)
:with ?e
:in $ [?clients ?start-date ?end-date]
:where [(iol-ion.query/scan-sales-orders $ ?clients ?start-date ?end-date) [[?e _ ?sort-default] ...]]
[?e :sales-order/service-charge ?service-charge]
(or-join [?e]
[?e :sales-order/vendor :vendor/ccp-square]
(and (not [?e :sales-order/vendor])
[?e :sales-order/external-id ?external-id]
[(clojure.string/starts-with? ?external-id "square/order/")]))]
(dc/db conn)
[[c] date date])))
(defn- get-service-charges
"The day's service charges as a summary item, for clients opted in to the rollout."
[c date]
(when (service-charges-enabled? c)
(when-let [amount (service-charge-total c date)]
(when-not (zero? amount)
{:db/id (str (java.util.UUID/randomUUID))
:sales-summary-item/category "Service Charges"
:sales-summary-item/sort-order 2
:ledger-mapped/amount amount
:ledger-mapped/ledger-side :ledger-side/credit}))))
(def ^:private suspect-categories
"The terms a balancing investigation keeps returning to. Logged beside the imbalance so a
day's shape can be read out of the logs without re-running the job."
["Tip" "Service Charges" "Returns" "Card Refunds" "Cash Refunds" "Food App Refunds"])
(defn- suspect-totals
"Amounts for `suspect-categories` present on this day, omitting the ones that are zero."
[items]
(into {}
(for [category suspect-categories
:let [amount (->> items
(filter #(= category (:sales-summary-item/category %)))
(map #(:ledger-mapped/amount % 0.0))
(reduce + 0.0))]
:when (not (zero? amount))]
[category amount])))
(defn refresh-client!
"Recomputes every dirty summary for one client.
Split out of the driver loop so a client's work stands on its own: it can be run for a single
client, and a backfill over the whole history can spread clients across threads instead of
grinding through the largest ones one day at a time."
[c client-code]
(doseq [{:sales-summary/keys [date] :db/keys [id] :as existing-summary} (dirty-sales-summaries c)]
(mu/with-context {:client-code client-code (mu/with-context {:client-code client-code
:date date} :date date}
(alog/info ::updating) (alog/info ::updating)
(let [manual-items (->> existing-summary (let [manual-items (->> existing-summary
:sales-summary/items :sales-summary/items
(filter :sales-summary-item/manual?)) (filter :sales-summary-item/manual?)
calculated-items (->> (map d-ss/<-pulled-item))
calculated-items (->>
(get-sales c date) (get-sales c date)
(concat (get-payment-items c date)) (concat (get-payment-items c date))
(concat (get-refund-items c date)) (concat (get-refund-items c date))
@@ -296,25 +544,37 @@
(cons (get-fees c date)) (cons (get-fees c date))
(cons (get-tax c date)) (cons (get-tax c date))
(cons (get-tip c date)) (cons (get-tip c date))
(cons (get-service-charges c date))
(cons (get-returns c date)) (cons (get-returns c date))
(filter identity) (filter identity)
(map (fn [z] (map (fn [z]
(assoc z :ledger-mapped/account (some-> z :sales-summary-item/category str/lower-case name->number lookup-account) (assoc z :ledger-mapped/account (some-> z :sales-summary-item/category str/lower-case name->number lookup-account)
:sales-summary-item/manual? false)))) :sales-summary-item/manual? false))))
all-items (concat calculated-items manual-items) all-items (concat calculated-items manual-items)
result {:db/id id result {:db/id id
:sales-summary/client c :sales-summary/client c
:sales-summary/date date :sales-summary/date date
:sales-summary/dirty false :sales-summary/dirty false
:sales-summary/client+date [c date] :sales-summary/client+date [c date]
:sales-summary/items all-items}] :sales-summary/items all-items}]
(if (seq (:sales-summary/items result)) (if (seq (:sales-summary/items result))
(do (do
(alog/info ::upserting-summaries (alog/info ::upserting-summaries
:category-count (count (:sales-summary/items result))) :category-count (count (:sales-summary/items result))
@(dc/transact conn [[:upsert-sales-summary result]])) :imbalance (d-ss/imbalance all-items)
@(dc/transact conn [{:db/id id :sales-summary/dirty false}])))))) :balanced? (d-ss/balanced? all-items)
:suspect-totals (suspect-totals all-items))
@(dc/transact conn [[:upsert-sales-summary result]]))
@(dc/transact conn [{:db/id id :sales-summary/dirty false}]))))))
(defn sales-summaries-v2
"Recomputes every dirty summary, client by client."
[]
(doseq [[c client-code] (dc/q '[:find ?c ?client-code
:in $
:where [?c :client/code ?client-code]]
(dc/db conn))]
(refresh-client! c client-code)))
(defn reset-summaries [] (defn reset-summaries []
@(dc/transact conn (->> (dc/q '[:find ?sos @(dc/transact conn (->> (dc/q '[:find ?sos
@@ -324,9 +584,6 @@
(map (fn [[sos]] (map (fn [[sos]]
[:db/retractEntity sos]))))) [:db/retractEntity sos])))))
(comment (comment
(auto-ap.datomic/transact-schema conn) (auto-ap.datomic/transact-schema conn)
@@ -336,26 +593,19 @@
(dirty-sales-summaries [:client/code "NGWH"]) (dirty-sales-summaries [:client/code "NGWH"])
(apply mark-dirty [:client/code "NGWH"] (last-n-days 5)) (apply mark-dirty [:client/code "NGWH"] (last-n-days 5))
(iol-ion.tx.upsert-sales-summary-ledger/summary->journal-entry (dc/db conn) 17592314245819) (iol-ion.tx.upsert-sales-summary-ledger/summary->journal-entry (dc/db conn) 17592314245819)
(iol-ion.tx.upsert-sales-summary-ledger/upsert-sales-summary (dc/db conn) {:db/id 17592314241429}) (iol-ion.tx.upsert-sales-summary-ledger/upsert-sales-summary (dc/db conn) {:db/id 17592314241429})
(mark-all-dirty 5) (mark-all-dirty 5)
(delete-all) (delete-all)
(sales-summaries-v2) (sales-summaries-v2)
1 1
(dc/q '[:find (pull ?sos [* {:sales-summary/sales-items [*]}]) (dc/q '[:find (pull ?sos [* {:sales-summary/sales-items [*]}])
:in $ :in $
:where [?sos :sales-summary/client [:client/code "NGHW"]] :where [?sos :sales-summary/client [:client/code "NGHW"]]
@@ -386,15 +636,24 @@
@(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy} @(dc/transact conn [{:db/id :sales-summary/total-tax :db/ident :sales-summary/total-tax-legacy}
{:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}]) {:db/id :sales-summary/total-tip :db/ident :sales-summary/total-tip-legacy}])
(auto-ap.datomic/transact-schema conn) (auto-ap.datomic/transact-schema conn))
)
(defn days-arg
"Trailing-window size from the job's `args`, e.g. `{:days 30}` set as a container override
from the admin Background Jobs page for an ad-hoc wider backfill. Values arrive as EDN but
may still be strings, so coerce defensively the way load-historical-sales does."
[args]
(let [days (:days args)]
(cond-> (or days default-days)
(string? days) (#(Long/parseLong %)))))
(defn refresh-sales-summaries
"Marks the trailing `days` window dirty, skipping accepted summaries, then recomputes
everything left dirty."
([] (refresh-sales-summaries default-days))
([days]
(mark-stale-dirty days)
(sales-summaries-v2)))
(defn -main [& _] (defn -main [& _]
(execute "sales-summaries" sales-summaries-v2)) (execute "sales-summaries" #(refresh-sales-summaries (days-arg (:args env)))))

29
src/clj/auto_ap/server.clj
View File

@@ -13,6 +13,7 @@
[auto-ap.jobs.load-historical-sales :as job-load-historical-sales] [auto-ap.jobs.load-historical-sales :as job-load-historical-sales]
[auto-ap.jobs.plaid :as job-plaid] [auto-ap.jobs.plaid :as job-plaid]
[auto-ap.jobs.register-invoice-import :as job-register-invoice-import] [auto-ap.jobs.register-invoice-import :as job-register-invoice-import]
[auto-ap.jobs.sales-summaries :as job-sales-summaries]
[auto-ap.jobs.square :as job-square] [auto-ap.jobs.square :as job-square]
[auto-ap.jobs.sysco :as job-sysco] [auto-ap.jobs.sysco :as job-sysco]
[auto-ap.jobs.vendor-usages :as job-vendor-usages] [auto-ap.jobs.vendor-usages :as job-vendor-usages]
@@ -33,23 +34,22 @@
(.addShutdownHook (Runtime/getRuntime) (.addShutdownHook (Runtime/getRuntime)
(Thread. f))) (Thread. f)))
(defn gzip-handler [] (defn gzip-handler []
(let [gz (GzipHandler.)] (let [gz (GzipHandler.)]
(doto gz (doto gz
(.setIncludedMethods (into-array ["GET" "POST" "PUT" "DELETE" "PATCH"])) (.setIncludedMethods (into-array ["GET" "POST" "PUT" "DELETE" "PATCH"]))
(.setIncludedMimeTypes (into-array ["text/css" (.setIncludedMimeTypes (into-array ["text/css"
"text/*" "text/*"
"text/plain" "text/plain"
"text/javascript" "text/javascript"
"text/csv" "text/csv"
"text/html" "text/html"
"text/html;charset=utf-8" "text/html;charset=utf-8"
"application/javascript" "application/javascript"
"application/csv" "application/csv"
"application/edn" "application/edn"
"application/json" "application/json"
"image/svg+xml"])) "image/svg+xml"]))
(.setMinGzipSize 1024)) (.setMinGzipSize 1024))
gz)) gz))
@@ -126,6 +126,9 @@
(= job "close-auto-invoices") (= job "close-auto-invoices")
(job-close-auto-invoices/-main) (job-close-auto-invoices/-main)
(= job "sales-summaries")
(job-sales-summaries/-main)
(= job "ezcater-upsert") (= job "ezcater-upsert")
(job-ezcater-upsert/-main) (job-ezcater-upsert/-main)

54
src/clj/auto_ap/square/core3.clj
View File

@@ -269,6 +269,29 @@
0.0 0.0
[])) []))
(defn scoped-key
"Client-scoped external id, in the shape sales order keys already use.
Without the client and location in the key, two clients configured on the same Square location
collide on a single entity: a refund changes owner every time either client imports, and one
charge ends up shared between both clients' orders."
[prefix client location id]
(str prefix (:client/code client) "-" (:square-location/client-location location) "-" id))
(defn existing-id
"Entity id of the refund or charge this id already refers to, trying the client-scoped key
first and the legacy unscoped key second.
This is what makes re-keying safe. These external ids are `:db.unique/identity`, so the import
relies on upsert-by-identity; changing the key format on its own would match nothing and
Datomic would create a SECOND entity for every refund and charge, orphaning the original under
its legacy key. Pinning the result as `:db/id` makes the write land on the existing entity
whichever scheme it currently carries."
[db attr prefix client location id]
(when id
(or (dc/entid db [attr (scoped-key prefix client location id)])
(dc/entid db [attr (str prefix id)]))))
(defn tender->charge [order client location t] (defn tender->charge [order client location t]
(remove-nils (remove-nils
#:charge #:charge
@@ -278,8 +301,9 @@
:note (:note t) :note (:note t)
:location (:square-location/client-location location) :location (:square-location/client-location location)
:reference-link (str (url/url "https://squareup.com/receipt/preview" (:id t))) :reference-link (str (url/url "https://squareup.com/receipt/preview" (:id t)))
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location (:id t))
:external-id (when (:id t) :external-id (when (:id t)
(str "square/charge/" (:id t))) (scoped-key "square/charge/" client location (:id t)))
:processor (cond :processor (cond
(#{"OTHER" "THIRD_PARTY_CARD"} (:type t)) (#{"OTHER" "THIRD_PARTY_CARD"} (:type t))
(condp = (some-> (:note t) str/lower-case) (condp = (some-> (:note t) str/lower-case)
@@ -509,7 +533,7 @@
(try (try
(->> (for [payout payouts (->> (for [payout payouts
:let [best-sales-date (some->> (dc/q '[:find ?s4 (count ?s) :let [best-sales-date (some->> (dc/q '[:find ?s4 (count ?s)
:in $ ?payout-id :in $ [?payout-id ...]
:where :where
[?payout :expected-deposit/external-id ?payout-id] [?payout :expected-deposit/external-id ?payout-id]
[?payout :expected-deposit/charges ?c] [?payout :expected-deposit/charges ?c]
@@ -519,7 +543,8 @@
[(auto-ap.time/localize ?s2) ?s3] [(auto-ap.time/localize ?s2) ?s3]
[(clj-time.coerce/to-local-date ?s3) ?s4]] [(clj-time.coerce/to-local-date ?s3) ?s4]]
(dc/db conn) (dc/db conn)
(str "square/payout/" (:id payout))) [(scoped-key "square/payout/" client location (:id payout))
(str "square/payout/" (:id payout))])
(sort-by last) (sort-by last)
last last
first first
@@ -543,7 +568,10 @@
(:db/id client) (:db/id client)
(amount->money (:amount_money payout))))] (amount->money (:amount_money payout))))]
:when (not equivalent-already-exists?)] :when (not equivalent-already-exists?)]
#:expected-deposit {:external-id (str "square/payout/" (:id payout)) #:expected-deposit {:db/id (or (existing-id (dc/db conn) :expected-deposit/external-id
"square/payout/" client location (:id payout))
(str "square/payout/" (:id payout)))
:external-id (scoped-key "square/payout/" client location (:id payout))
:vendor :vendor/ccp-square :vendor :vendor/ccp-square
:status :expected-deposit-status/pending :status :expected-deposit-status/pending
:total (amount->money (:amount_money payout)) :total (amount->money (:amount_money payout))
@@ -561,7 +589,11 @@
(coerce/to-date))) (coerce/to-date)))
:charges (reverse (->> (:payout_entries payout) :charges (reverse (->> (:payout_entries payout)
(filter (comp :payment_id :type_charge_details)) (filter (comp :payment_id :type_charge_details))
(map (fn [p] {:charge/external-id (str "square/charge/" (:payment_id (:type_charge_details p)))}))))}) (map (fn [p]
(let [payment-id (:payment_id (:type_charge_details p))]
(remove-nils
{:charge/external-id (scoped-key "square/charge/" client location payment-id)
:db/id (existing-id (dc/db conn) :charge/external-id "square/charge/" client location payment-id)}))))))})
(filter :expected-deposit/date) (filter :expected-deposit/date)
(into [])) (into []))
(catch Throwable e (catch Throwable e
@@ -585,7 +617,8 @@
(de/chain (de/chain
(get-payment client (:payment_id r)) (get-payment client (:payment_id r))
(fn [payment] (fn [payment]
#:sales-refund {:external-id (str "square/refund/" (:id r)) #:sales-refund {:db/id (existing-id (dc/db conn) :sales-refund/external-id "square/refund/" client l (:id r))
:external-id (scoped-key "square/refund/" client l (:id r))
:vendor :vendor/ccp-square :vendor :vendor/ccp-square
:total (amount->money (:amount_money r)) :total (amount->money (:amount_money r))
:fee (transduce :fee (transduce
@@ -703,7 +736,10 @@
(de/chain (de/chain
(get-cash-shift client (:id s)) (get-cash-shift client (:id s))
(fn [cash-drawer-shift] (fn [cash-drawer-shift]
#:cash-drawer-shift {:external-id (str "square/cash-drawer-shift/" (:id cash-drawer-shift)) #:cash-drawer-shift {:db/id (or (existing-id (dc/db conn) :cash-drawer-shift/external-id
"square/cash-drawer-shift/" client l (:id cash-drawer-shift))
(str "square/cash-drawer-shift/" (:id cash-drawer-shift)))
:external-id (scoped-key "square/cash-drawer-shift/" client l (:id cash-drawer-shift))
:vendor :vendor/ccp-square :vendor :vendor/ccp-square
:paid-in (amount->money (:cash_paid_in_money cash-drawer-shift)) :paid-in (amount->money (:cash_paid_in_money cash-drawer-shift))
:paid-out (amount->money (:cash_paid_out_money cash-drawer-shift)) :paid-out (amount->money (:cash_paid_out_money cash-drawer-shift))
@@ -723,10 +759,12 @@
:when (:square-location/client-location square-location)] :when (:square-location/client-location square-location)]
(upsert-cash-shifts client square-location)))) (upsert-cash-shifts client square-location))))
([client location] ([client location]
(upsert-cash-shifts client location (time/plus (time/now) (time/days -75)) (time/now)))
([client location start end]
(with-context-as {:source "Square cash shift loading" (with-context-as {:source "Square cash shift loading"
:client (:client/code client)} lc :client (:client/code client)} lc
(de/chain (cash-drawer-shifts client location) (de/chain (cash-drawer-shifts client location start end)
(fn [cash-shifts] (fn [cash-shifts]
(mu/with-context lc (mu/with-context lc
(try (try

206
src/clj/auto_ap/ssr/admin/background_jobs.clj
View File

@@ -28,14 +28,13 @@
(com.amazonaws.services.ecs.model AssignPublicIp))) (com.amazonaws.services.ecs.model AssignPublicIp)))
(defn get-ecs-tasks [] (defn get-ecs-tasks []
(->> (->>
(concat (:task-arns (ecs/list-tasks :max-results 50)) (:task-arns (ecs/list-tasks :desired-status "STOPPED" :max-results 50))) (concat (:task-arns (ecs/list-tasks :max-results 50)) (:task-arns (ecs/list-tasks :desired-status "STOPPED" :max-results 50)))
(ecs/describe-tasks :include [] :tasks) (ecs/describe-tasks :include [] :tasks)
:tasks :tasks
(map #(assoc % :task-definition (:task-definition (ecs/describe-task-definition :task-definition (:task-definition-arn %))))) (map #(assoc % :task-definition (:task-definition (ecs/describe-task-definition :task-definition (:task-definition-arn %)))))
(sort-by :created-at) (sort-by :created-at)
reverse)) reverse))
(defn is-background-job? (defn is-background-job?
"This function checks whether a given task is a background job. "This function checks whether a given task is a background job.
@@ -60,7 +59,7 @@
(defn job-exited-successfully? [task] (defn job-exited-successfully? [task]
(if (= 0 (->> task (if (= 0 (->> task
:containers :containers
(filter (comp #{"integreat-app" } :name)) (filter (comp #{"integreat-app"} :name))
(first) (first)
:exit-code)) :exit-code))
true true
@@ -77,7 +76,7 @@
:succeeded :succeeded
:failed)) :failed))
:name (task-definition->job-name (:task-definition task)) :name (task-definition->job-name (:task-definition task))
:end-date (some-> (:stopped-at task) coerce/to-date-time (time/to-time-zone (time/time-zone-for-offset 0))) :end-date (some-> (:stopped-at task) coerce/to-date-time (time/to-time-zone (time/time-zone-for-offset 0)))
:start-date (some-> (:created-at task) coerce/to-date-time (time/to-time-zone (time/time-zone-for-offset 0)))}) :start-date (some-> (:created-at task) coerce/to-date-time (time/to-time-zone (time/time-zone-for-offset 0)))})
(defn fetch-page [request] (defn fetch-page [request]
@@ -85,7 +84,7 @@
(filter is-background-job?) (filter is-background-job?)
(map ecs-task->job))] (map ecs-task->job))]
[jobs (count jobs)])) [jobs (count jobs)]))
(def query-schema (mc/schema [:map ])) (def query-schema (mc/schema [:map]))
(def grid-page (def grid-page
(helper/build {:id "job-table" (helper/build {:id "job-table"
@@ -107,8 +106,7 @@
:entity-name "Job" :entity-name "Job"
:query-schema query-schema :query-schema query-schema
:route :admin-job-table :route :admin-job-table
:headers [ :headers [{:key "start"
{:key "start"
:name "Start" :name "Start"
:render #(some-> % :start-date (atime/unparse-local atime/standard-time))} :render #(some-> % :start-date (atime/unparse-local atime/standard-time))}
{:key "end" {:key "end"
@@ -119,7 +117,7 @@
:render (fn [e] :render (fn [e]
(when (and (:start-date e) (when (and (:start-date e)
(:end-date e)) (:end-date e))
(str (time/in-minutes (time/interval (str (time/in-minutes (time/interval
(:start-date e) (:start-date e)
(:end-date e))) " minutes")))} (:end-date e))) " minutes")))}
{:key "name" {:key "name"
@@ -150,16 +148,16 @@
:network-configuration {:aws-vpc-configuration {:subnets ["subnet-5e675761" "subnet-8519fde2" "subnet-89bab8d4"] :network-configuration {:aws-vpc-configuration {:subnets ["subnet-5e675761" "subnet-8519fde2" "subnet-89bab8d4"]
:security-groups ["sg-004e5855310c453a3" "sg-02d167406b1082698"] :security-groups ["sg-004e5855310c453a3" "sg-02d167406b1082698"]
:assign-public-ip AssignPublicIp/ENABLED}}} :assign-public-ip AssignPublicIp/ENABLED}}}
args (assoc-in [:overrides :container-overrides ] [{:name "integreat-app" :environment [{:name "args" :value (pr-str args)}]}])))) args (assoc-in [:overrides :container-overrides] [{:name "integreat-app" :environment [{:name "args" :value (pr-str args)}]}]))))
(defn job-start [{:keys [form-params]}] (defn job-start [{:keys [form-params]}]
(if (not (get (currently-running-jobs) (:name form-params))) (if (not (get (currently-running-jobs) (:name form-params)))
(let [new-job (run-task (let [new-job (run-task
(-> (:name form-params) (-> (:name form-params)
(str/replace #"-" "_") (str/replace #"-" "_")
(str/replace #":" "") (str/replace #":" "")
(str "_" (:dd-env env))) (str "_" (:dd-env env)))
(dissoc form-params :name))] (dissoc form-params :name))]
{:message (str "task " (str new-job) " started.")}) {:message (str "task " (str new-job) " started.")})
(form-validation-error "This job is already running" (form-validation-error "This job is already running"
:form-params form-params))) :form-params form-params)))
@@ -170,107 +168,109 @@
[(fc/with-field :ledger-url [(fc/with-field :ledger-url
(com/validated-field {:label "Url" (com/validated-field {:label "Url"
:errors (fc/field-errors)} :errors (fc/field-errors)}
[:div.flex.place-items-center.gap-2 [:div.flex.place-items-center.gap-2
[:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"] [:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"]
(com/text-input {:placeholder "ledger-data.csv" (com/text-input {:placeholder "ledger-data.csv"
:name (fc/field-name) :name (fc/field-name)
:value (fc/field-value)} )]))] :value (fc/field-value)})]))]
(= "register-invoice-import" name) (= "register-invoice-import" name)
[ [(fc/with-field :invoice-url
(fc/with-field :invoice-url (com/validated-field {:label "Url"
(com/validated-field {:label "Url" :errors (fc/field-errors)}
:errors (fc/field-errors)} [:div.flex.place-items-center.gap-2
[:div.flex.place-items-center.gap-2 [:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"]
[:pre.text-xs.mr-1 "s3://data.prod.app.integreatconsult.com/bulk-import/"] (com/text-input {:placeholder "invoice-data.csv"
(com/text-input {:placeholder "invoice-data.csv" :name (fc/field-name)
:name (fc/field-name) :value (fc/field-value)})]))]
:value (fc/field-value)} )]))]
(= "load-historical-sales" name) (= "load-historical-sales" name)
[ [(fc/with-field :client
(fc/with-field :client (com/validated-field {:label "Client"
(com/validated-field {:label "Client" :errors (fc/field-errors)}
:errors (fc/field-errors)} (com/typeahead {:name (fc/field-name)
(com/typeahead {:name (fc/field-name) :value (fc/field-value)
:value (fc/field-value) :placeholder "Search..."
:placeholder "Search..." :url (bidi/path-for ssr-routes/only-routes
:url (bidi/path-for ssr-routes/only-routes :company-search)})))
:company-search)}))) (fc/with-field :days
(fc/with-field :days
(com/validated-field {:label "Days to load" (com/validated-field {:label "Days to load"
:errors (fc/field-errors)} :errors (fc/field-errors)}
(com/text-input {:placeholder "60" (com/text-input {:placeholder "60"
:name (fc/field-name) :name (fc/field-name)
:value (fc/field-value)} )))] :value (fc/field-value)})))]
:else nil)) (= "sales-summaries" name)
[(fc/with-field :days
(com/validated-field {:label "Days to refresh"
:errors (fc/field-errors)}
(com/text-input {:placeholder "7"
:name (fc/field-name)
:value (fc/field-value)})))]
:else nil)))
(defn subform [{{:keys [name]} :query-params}]
)
(defn subform [{{:keys [name]} :query-params }]
(html-response (html-response
(fc/start-form {} nil (fc/start-form {} nil
(subform* {:name name})))) (subform* {:name name}))))
(defn job-start-dialog [{:keys [form-errors form-params] :as request}] (defn job-start-dialog [{:keys [form-errors form-params] :as request}]
(fc/start-form (or form-params {}) form-errors (fc/start-form (or form-params {}) form-errors
(modal-response (modal-response
(com/modal ;; TODO we need a cleaner way to have forms that wrap the whole. In this cas (com/modal ;; TODO we need a cleaner way to have forms that wrap the whole. In this cas
{} {}
[:form {:hx-post (bidi/path-for ssr-routes/only-routes :admin-job-start) [:form {:hx-post (bidi/path-for ssr-routes/only-routes :admin-job-start)
:class "h-full w-full"} :class "h-full w-full"}
[:fieldset {:class "hx-disable h-full w-full"} [:fieldset {:class "hx-disable h-full w-full"}
(com/modal-card {} (com/modal-card {}
[:div.m-2 "New job"] [:div.m-2 "New job"]
[:div.space-y-6 [:div.space-y-6
(fc/with-field :name (fc/with-field :name
(com/validated-field {:label "Job" (com/validated-field {:label "Job"
:errors (fc/field-errors)} :errors (fc/field-errors)}
(com/select {:name (fc/field-name) (com/select {:name (fc/field-name)
:value (fc/field-value) :value (fc/field-value)
:class "w-64" :class "w-64"
:options [["" ""] :options [["" ""]
["yodlee2" "Yodlee Import"] ["yodlee2" "Yodlee Import"]
["yodlee2-accounts" "Yodlee Account Import"] ["yodlee2-accounts" "Yodlee Account Import"]
["intuit" "Intuit import"] ["intuit" "Intuit import"]
["plaid" "Plaid import"] ["plaid" "Plaid import"]
["bulk-journal-import" "Bulk Journal Import"] ["bulk-journal-import" "Bulk Journal Import"]
["square-import-job" "Square Import"] ["square-import-job" "Square Import"]
["register-invoice-import" "Register Invoice Import "] ["register-invoice-import" "Register Invoice Import "]
["ezcater-upsert" "Upsert recent ezcater orders"] ["ezcater-upsert" "Upsert recent ezcater orders"]
["load-historical-sales" "Load Historical Square Sales"] ["load-historical-sales" "Load Historical Square Sales"]
["export-backup" "Export Backup"]] ["sales-summaries" "Refresh Sales Summaries"]
:hx-get (bidi/path-for ssr-routes/only-routes ["export-backup" "Export Backup"]]
:admin-job-subform) :hx-get (bidi/path-for ssr-routes/only-routes
:hx-target "#sub-form" :admin-job-subform)
:hx-swap "innerHTML"}))) :hx-target "#sub-form"
:hx-swap "innerHTML"})))
[:div#sub-form (subform* {:name (fc/with-field :name (fc/field-value))}) ]] [:div#sub-form (subform* {:name (fc/with-field :name (fc/field-value))})]]
[:div [:div
(com/form-errors {:errors (:errors fc/*form-errors*)}) (com/form-errors {:errors (:errors fc/*form-errors*)})
(com/validated-save-button {:errors form-errors} "Run job")])]])))) (com/validated-save-button {:errors form-errors} "Run job")])]]))))
(def form-schema (mc/schema [:map (def form-schema (mc/schema [:map
[:name [:string {:min 1}]] [:name [:string {:min 1}]]
[:ledger-url {:optional true} [:string {:min 1}]] [:ledger-url {:optional true} [:string {:min 1}]]
[:invoice-url {:optional true} [:string {:min 1}]] [:invoice-url {:optional true} [:string {:min 1}]]
[:client {:optional true} entity-id] [:client {:optional true} entity-id]
[:days {:optional true} [:int {:min 1 :max 120}]] [:days {:optional true} [:int {:min 1 :max 120}]]]))
]))
(def key->handler (def key->handler
(apply-middleware-to-all-handlers (apply-middleware-to-all-handlers
(->> (->>
{:admin-jobs (helper/page-route grid-page) {:admin-jobs (helper/page-route grid-page)
:admin-job-table (helper/table-route grid-page) :admin-job-table (helper/table-route grid-page)
:admin-job-subform (-> subform (wrap-schema-enforce :query-schema [:map [:name {:optional true} [:maybe :string]]])) :admin-job-subform (-> subform (wrap-schema-enforce :query-schema [:map [:name {:optional true} [:maybe :string]]]))
:admin-job-start (-> job-start :admin-job-start (-> job-start
(wrap-schema-enforce :form-schema form-schema) (wrap-schema-enforce :form-schema form-schema)
(wrap-nested-form-params) (wrap-nested-form-params)
(wrap-form-4xx-2 job-start-dialog)) (wrap-form-4xx-2 job-start-dialog))
:admin-job-start-dialog job-start-dialog}) :admin-job-start-dialog job-start-dialog})
(fn [h] (fn [h]
(-> h (-> h
(wrap-admin) (wrap-admin)
(wrap-client-redirect-unauthenticated))))) (wrap-client-redirect-unauthenticated)))))

91
src/clj/auto_ap/ssr/pos/sales_summaries.clj
View File

@@ -4,6 +4,7 @@
:refer [apply-pagination apply-sort-3 conn merge-query pull-many :refer [apply-pagination apply-sort-3 conn merge-query pull-many
query2]] query2]]
[auto-ap.datomic.accounts :as d-accounts] [auto-ap.datomic.accounts :as d-accounts]
[auto-ap.datomic.sales-summaries :refer [total-credits total-debits]]
[auto-ap.graphql.utils :refer [extract-client-ids]] [auto-ap.graphql.utils :refer [extract-client-ids]]
[auto-ap.query-params :refer [wrap-copy-qp-pqp]] [auto-ap.query-params :refer [wrap-copy-qp-pqp]]
[auto-ap.client-routes :as client-routes] [auto-ap.client-routes :as client-routes]
@@ -116,18 +117,6 @@
(defn sort-items [ss] (defn sort-items [ss]
(sort-by (juxt :ledger-mapped/ledger-side :sales-summary-item/sort-order :sales-summary-item/category) ss)) (sort-by (juxt :ledger-mapped/ledger-side :sales-summary-item/sort-order :sales-summary-item/category) ss))
(defn total-debits [items]
(->> items
(filter #(= :ledger-side/debit (:ledger-mapped/ledger-side %)))
(map #(:ledger-mapped/amount % 0.0))
(reduce + 0.0)))
(defn total-credits [items]
(->> items
(filter #(= :ledger-side/credit (:ledger-mapped/ledger-side %)))
(map #(:ledger-mapped/amount % 0.0))
(reduce + 0.0)))
(defn truncate [s max-len] (defn truncate [s max-len]
(if (> (count s) max-len) (if (> (count s) max-len)
(str (subs s 0 (- max-len 3)) "...") (str (subs s 0 (- max-len 3)) "...")
@@ -158,13 +147,13 @@
[:span.text-sm account-name] [:span.text-sm account-name]
(com/pill {:color :red} "Missing acct")) (com/pill {:color :red} "Missing acct"))
(com/a-icon-button {:class "p-1" (com/a-icon-button {:class "p-1"
:hx-get (bidi/path-for ssr-routes/only-routes ::route/edit-item-account) :hx-get (bidi/path-for ssr-routes/only-routes ::route/edit-item-account)
:hx-target "closest .account-cell" :hx-target "closest .account-cell"
:hx-swap "outerHTML" :hx-swap "outerHTML"
:hx-vals (hx/json {:item-index (or (:item-index item) 0) :hx-vals (hx/json {:item-index (or (:item-index item) 0)
:client-id client-id :client-id client-id
:current-account-id (or account-id "")})} :current-account-id (or account-id "")})}
svg/pencil)])) svg/pencil)]))
(defn account-edit-cell [{:keys [field-name-prefix client-id current-account-id]}] (defn account-edit-cell [{:keys [field-name-prefix client-id current-account-id]}]
(let [account-input-name (str field-name-prefix "[ledger-mapped/account]")] (let [account-input-name (str field-name-prefix "[ledger-mapped/account]")]
@@ -172,23 +161,23 @@
(account-typeahead* {:name account-input-name (account-typeahead* {:name account-input-name
:value current-account-id :value current-account-id
:client-id client-id}) :client-id client-id})
[:div.flex.gap-1 [:div.flex.gap-1
(com/a-icon-button {:class "p-1" (com/a-icon-button {:class "p-1"
:hx-put (bidi/path-for ssr-routes/only-routes ::route/save-item-account) :hx-put (bidi/path-for ssr-routes/only-routes ::route/save-item-account)
:hx-target "closest .account-cell" :hx-target "closest .account-cell"
:hx-swap "outerHTML" :hx-swap "outerHTML"
:hx-include "closest .account-cell" :hx-include "closest .account-cell"
:hx-vals (hx/json {:field-name-prefix field-name-prefix :hx-vals (hx/json {:field-name-prefix field-name-prefix
:client-id client-id})} :client-id client-id})}
svg/check) svg/check)
(com/a-icon-button {:class "p-1" (com/a-icon-button {:class "p-1"
:hx-get (bidi/path-for ssr-routes/only-routes ::route/cancel-item-account) :hx-get (bidi/path-for ssr-routes/only-routes ::route/cancel-item-account)
:hx-target "closest .account-cell" :hx-target "closest .account-cell"
:hx-swap "outerHTML" :hx-swap "outerHTML"
:hx-vals (hx/json {:field-name-prefix field-name-prefix :hx-vals (hx/json {:field-name-prefix field-name-prefix
:client-id client-id :client-id client-id
:current-account-id (or current-account-id "")})} :current-account-id (or current-account-id "")})}
svg/x)]])) svg/x)]]))
(def grid-page (def grid-page
(helper/build {:id "entity-table" (helper/build {:id "entity-table"
@@ -576,8 +565,8 @@
[:span.text-gray-500 (truncate (:sales-summary-item/category item) 30)] [:span.text-gray-500 (truncate (:sales-summary-item/category item) 30)]
(account-display-cell {:item (assoc item :item-index actual-idx) (account-display-cell {:item (assoc item :item-index actual-idx)
:field-name-prefix (str "step-params[sales-summary/items][" actual-idx "]") :field-name-prefix (str "step-params[sales-summary/items][" actual-idx "]")
:client-id client-id}) :client-id client-id})
[:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]])) [:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]]))
[:div.h-6]))] [:div.h-6]))]
[:div.mt-2.border-t.pt-1 [:div.mt-2.border-t.pt-1
(summary-total-display request) (summary-total-display request)
@@ -619,13 +608,13 @@
[:span.text-gray-500 (truncate (:sales-summary-item/category item) 30)] [:span.text-gray-500 (truncate (:sales-summary-item/category item) 30)]
(account-display-cell {:item (assoc item :item-index actual-idx) (account-display-cell {:item (assoc item :item-index actual-idx)
:field-name-prefix (str "step-params[sales-summary/items][" actual-idx "]") :field-name-prefix (str "step-params[sales-summary/items][" actual-idx "]")
:client-id client-id}) :client-id client-id})
[:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]])) [:span.ml-auto.font-mono.tabular-nums.text-gray-900 (format "$%,.2f" (:ledger-mapped/amount item))]]))
[:div.h-6]))] [:div.h-6]))]
[:div.mt-2.border-t.pt-1 [:div.mt-2.border-t.pt-1
(summary-total-display request) (summary-total-display request)
(unbalanced-display request)]]] (unbalanced-display request)]]]
[:div.mt-4.border-t.pt-2 [:div.mt-4.border-t.pt-2
(fc/with-field :sales-summary/items (fc/with-field :sales-summary/items
(com/data-grid-new-row {:colspan 2 (com/data-grid-new-row {:colspan 2
:hx-get (bidi/path-for ssr-routes/only-routes ::route/new-summary-item) :hx-get (bidi/path-for ssr-routes/only-routes ::route/new-summary-item)
@@ -761,16 +750,16 @@
::route/edit-wizard-navigate (-> mm/next-handler ::route/edit-wizard-navigate (-> mm/next-handler
(mm/wrap-wizard edit-wizard) (mm/wrap-wizard edit-wizard)
(mm/wrap-decode-multi-form-state)) (mm/wrap-decode-multi-form-state))
::route/new-summary-item (-> (add-new-entity-handler [:step-params :sales-summary/items] ::route/new-summary-item (-> (add-new-entity-handler [:step-params :sales-summary/items]
(fn render [cursor request] (fn render [cursor request]
(sales-summary-item-row* (sales-summary-item-row*
{:value cursor {:value cursor
:client-id (:client-id (:query-params request))})) :client-id (:client-id (:query-params request))}))
(fn build-new-row [base _] (fn build-new-row [base _]
(assoc base :sales-summary-item/manual? true))) (assoc base :sales-summary-item/manual? true)))
(wrap-schema-enforce :query-schema [:map (wrap-schema-enforce :query-schema [:map
[:client-id {:optional true} [:client-id {:optional true}
[:maybe entity-id]]])) [:maybe entity-id]]]))
::route/edit-item-account (-> edit-item-account ::route/edit-item-account (-> edit-item-account
(wrap-schema-enforce :query-schema [:map (wrap-schema-enforce :query-schema [:map
[:item-index nat-int?] [:item-index nat-int?]

14
terraform/deploy.tf
View File

@@ -386,6 +386,20 @@ module "close_auto_invoices_job" {
cpu = 512 cpu = 512
} }
module "sales_summaries_job" {
count = var.enable_schedules ? 1 : 0
source = "./background-job/"
ecs_cluster = var.ecs_cluster
task_role_arn = var.task_role_arn
stage = var.stage
schedule = "rate(1 day)"
job_name = "sales-summaries"
execution_role_arn = var.execution_role_arn
use_schedule = true
memory = 4096
cpu = 2048
}
module "yodlee2_accounts_job" { module "yodlee2_accounts_job" {
count = var.enable_schedules ? 1 : 0 count = var.enable_schedules ? 1 : 0
source = "./background-job/" source = "./background-job/"

4
terraform/prod.tfvars
View File

@@ -1,5 +1,5 @@
aws_access_key_id="AKIAINHACMVQJ6NYD26A" aws_access_key_id="AKIAZ4TSKSJ27WXFCOWK"
aws_secret_access_key="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" aws_secret_access_key="NY1divQYUBELhsNvCeprd4r9MvOXhlNMECnsg7TL"
domain="app.integreatconsult.com" domain="app.integreatconsult.com"
invoice_address="invoices@mail.app.integreatconsult.com" invoice_address="invoices@mail.app.integreatconsult.com"
base_url="https://app.integreatconsult.com" base_url="https://app.integreatconsult.com"

228
terraform/terraform.tfstate.d/prod/terraform.tfstate
View File

@@ -1,7 +1,7 @@
{ {
"version": 4, "version": 4,
"terraform_version": "1.9.2", "terraform_version": "1.15.1",
"serial": 718, "serial": 722,
"lineage": "9b630886-8cee-a57d-c7a2-4f19f13f9c51", "lineage": "9b630886-8cee-a57d-c7a2-4f19f13f9c51",
"outputs": { "outputs": {
"aws_access_key_id": { "aws_access_key_id": {
@@ -115,7 +115,8 @@
"usage_operation": "RunInstances", "usage_operation": "RunInstances",
"virtualization_type": "hvm" "virtualization_type": "hvm"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -133,7 +134,8 @@
"id": "679918342773", "id": "679918342773",
"user_id": "AIDAJPUJFTOKO4IRADMV4" "user_id": "AIDAJPUJFTOKO4IRADMV4"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -181,7 +183,8 @@
], ],
"version": "2012-10-17" "version": "2012-10-17"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -246,6 +249,7 @@
} }
] ]
], ],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -311,6 +315,7 @@
} }
] ]
], ],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -346,6 +351,7 @@
"type": "gp2" "type": "gp2"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwLCJ1cGRhdGUiOjMwMDAwMDAwMDAwMH19", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwLCJ1cGRhdGUiOjMwMDAwMDAwMDAwMH19",
"dependencies": [ "dependencies": [
"aws_instance.solr_ec2", "aws_instance.solr_ec2",
@@ -445,6 +451,7 @@
"wait_for_steady_state": true "wait_for_steady_state": true
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_ecs_task_definition.integreat_app", "aws_ecs_task_definition.integreat_app",
@@ -535,6 +542,7 @@
"wait_for_steady_state": true "wait_for_steady_state": true
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_ecs_task_definition.solr", "aws_ecs_task_definition.solr",
@@ -580,6 +588,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -636,6 +645,7 @@
] ]
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"aws_efs_file_system.solr_storage" "aws_efs_file_system.solr_storage"
@@ -681,6 +691,7 @@
"throughput_mode": "bursting" "throughput_mode": "bursting"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -719,6 +730,7 @@
} }
] ]
], ],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_iam_user.app_user" "aws_iam_user.app_user"
] ]
@@ -744,7 +756,8 @@
"tags_all": {}, "tags_all": {},
"unique_id": "AIDAINFBWI2I7A3TKPGW2" "unique_id": "AIDAINFBWI2I7A3TKPGW2"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -762,6 +775,7 @@
"user": "integreat-prod" "user": "integreat-prod"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_iam_user.app_user" "aws_iam_user.app_user"
] ]
@@ -908,6 +922,7 @@
] ]
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"data.aws_ami.amazon_linux_2023" "data.aws_ami.amazon_linux_2023"
@@ -1014,6 +1029,7 @@
"zone_id": "Z35SXDOTRQ7X7K" "zone_id": "Z35SXDOTRQ7X7K"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19" "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19"
} }
] ]
@@ -1063,6 +1079,7 @@
} }
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_lb.integreat_app" "aws_lb.integreat_app"
@@ -1106,6 +1123,7 @@
} }
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_acm_certificate.cert", "aws_acm_certificate.cert",
@@ -1173,6 +1191,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"aws_acm_certificate.cert", "aws_acm_certificate.cert",
@@ -1242,6 +1261,7 @@
"vpc_id": "vpc-b5b7d6ce" "vpc_id": "vpc-b5b7d6ce"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1320,6 +1340,7 @@
"website_endpoint": "data.prod.app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" "website_endpoint": "data.prod.app.integreatconsult.com.s3-website-us-east-1.amazonaws.com"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1391,6 +1412,7 @@
"website_endpoint": null "website_endpoint": null
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"data.aws_caller_identity.current" "data.aws_caller_identity.current"
@@ -1489,6 +1511,7 @@
"website_endpoint": "app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" "website_endpoint": "app.integreatconsult.com.s3-website-us-east-1.amazonaws.com"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1560,6 +1583,7 @@
"website_endpoint": null "website_endpoint": null
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjM2MDAwMDAwMDAwMDAsInJlYWQiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19" "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjM2MDAwMDAwMDAwMDAsInJlYWQiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19"
} }
] ]
@@ -1591,6 +1615,7 @@
"topic": [] "topic": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_s3_bucket.invoices", "aws_s3_bucket.invoices",
"aws_sqs_queue.integreat-mail", "aws_sqs_queue.integreat-mail",
@@ -1613,6 +1638,7 @@
"policy": "{\"Statement\":[{\"Action\":\"s3:*\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam::679918342773:role/http-proxy\",\"arn:aws:iam::679918342773:role/datomic-ddb\"]},\"Resource\":[\"arn:aws:s3:::toast.prod.app.integreatconsult.com/*\",\"arn:aws:s3:::toast.prod.app.integreatconsult.com\"],\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}" "policy": "{\"Statement\":[{\"Action\":\"s3:*\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam::679918342773:role/http-proxy\",\"arn:aws:iam::679918342773:role/datomic-ddb\"]},\"Resource\":[\"arn:aws:s3:::toast.prod.app.integreatconsult.com/*\",\"arn:aws:s3:::toast.prod.app.integreatconsult.com\"],\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"aws_s3_bucket.toast_bucket", "aws_s3_bucket.toast_bucket",
@@ -1638,6 +1664,7 @@
"service_id": "srv-ren22oppkwwryqqr" "service_id": "srv-ren22oppkwwryqqr"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"aws_instance.solr_ec2", "aws_instance.solr_ec2",
@@ -1685,6 +1712,7 @@
"type": "DNS_HTTP" "type": "DNS_HTTP"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1727,6 +1755,7 @@
"type": "DNS_HTTP" "type": "DNS_HTTP"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1769,6 +1798,7 @@
"type": "DNS_HTTP" "type": "DNS_HTTP"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1810,6 +1840,7 @@
"workmail_action": [] "workmail_action": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_s3_bucket.invoices", "aws_s3_bucket.invoices",
"aws_ses_receipt_rule_set.main", "aws_ses_receipt_rule_set.main",
@@ -1831,7 +1862,8 @@
"id": "default-rule-set", "id": "default-rule-set",
"rule_set_name": "default-rule-set" "rule_set_name": "default-rule-set"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -1868,6 +1900,7 @@
"visibility_timeout_seconds": 30 "visibility_timeout_seconds": 30
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1905,6 +1938,7 @@
"visibility_timeout_seconds": 30 "visibility_timeout_seconds": 30
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_s3_bucket.invoices", "aws_s3_bucket.invoices",
"data.aws_caller_identity.current" "data.aws_caller_identity.current"
@@ -1945,6 +1979,7 @@
"visibility_timeout_seconds": 30 "visibility_timeout_seconds": 30
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1968,6 +2003,7 @@
"volume_id": "vol-0069283d41ff6c010" "volume_id": "vol-0069283d41ff6c010"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_ebs_volume.solr_ec2_storage", "aws_ebs_volume.solr_ec2_storage",
@@ -2014,6 +2050,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2043,6 +2080,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2107,6 +2145,7 @@
"target_id": "close-auto-invoices" "target_id": "close-auto-invoices"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.close_auto_invoices_job.aws_cloudwatch_event_rule.schedule", "module.close_auto_invoices_job.aws_cloudwatch_event_rule.schedule",
@@ -2152,6 +2191,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2181,6 +2221,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2245,6 +2286,7 @@
"target_id": "import-uploaded-invoices" "target_id": "import-uploaded-invoices"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.import_uploaded_invoices_job.aws_cloudwatch_event_rule.schedule", "module.import_uploaded_invoices_job.aws_cloudwatch_event_rule.schedule",
@@ -2290,6 +2332,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2319,6 +2362,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2383,6 +2427,7 @@
"target_id": "insight-outcome-recommendation" "target_id": "insight-outcome-recommendation"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.insight_outcome_recommendation_job.aws_cloudwatch_event_rule.schedule", "module.insight_outcome_recommendation_job.aws_cloudwatch_event_rule.schedule",
@@ -2428,6 +2473,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2457,6 +2503,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2521,6 +2568,7 @@
"target_id": "intuit" "target_id": "intuit"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.intuit_job.aws_cloudwatch_event_rule.schedule", "module.intuit_job.aws_cloudwatch_event_rule.schedule",
@@ -2566,6 +2614,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2607,6 +2656,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2636,6 +2686,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2700,6 +2751,7 @@
"target_id": "ntg" "target_id": "ntg"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.ntg_job.aws_cloudwatch_event_rule.schedule", "module.ntg_job.aws_cloudwatch_event_rule.schedule",
@@ -2745,6 +2797,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2774,6 +2827,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2838,6 +2892,7 @@
"target_id": "plaid" "target_id": "plaid"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.plaid_job.aws_cloudwatch_event_rule.schedule", "module.plaid_job.aws_cloudwatch_event_rule.schedule",
@@ -2883,6 +2938,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2912,6 +2968,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2976,6 +3033,7 @@
"target_id": "reconcile-ledger" "target_id": "reconcile-ledger"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.reconcile_ledger_job.aws_cloudwatch_event_rule.schedule", "module.reconcile_ledger_job.aws_cloudwatch_event_rule.schedule",
@@ -3021,6 +3079,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3062,6 +3121,148 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
}
]
},
{
"module": "module.sales_summaries_job[0]",
"mode": "managed",
"type": "aws_cloudwatch_event_rule",
"name": "schedule",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"index_key": 0,
"schema_version": 0,
"attributes": {
"arn": "arn:aws:events:us-east-1:679918342773:rule/sales-summaries-schedule-prod",
"description": "",
"event_bus_name": "default",
"event_pattern": null,
"id": "sales-summaries-schedule-prod",
"is_enabled": true,
"name": "sales-summaries-schedule-prod",
"name_prefix": "",
"role_arn": "",
"schedule_expression": "rate(1 day)",
"tags": null,
"tags_all": {}
},
"sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA=="
}
]
},
{
"module": "module.sales_summaries_job[0]",
"mode": "managed",
"type": "aws_cloudwatch_event_target",
"name": "job_target",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"index_key": 0,
"schema_version": 1,
"attributes": {
"arn": "arn:aws:ecs:us-east-1:679918342773:cluster/default",
"batch_target": [],
"dead_letter_config": [],
"ecs_target": [
{
"capacity_provider_strategy": [],
"enable_ecs_managed_tags": false,
"enable_execute_command": false,
"group": "",
"launch_type": "FARGATE",
"network_configuration": [
{
"assign_public_ip": true,
"security_groups": [
"sg-004e5855310c453a3",
"sg-02d167406b1082698"
],
"subnets": [
"subnet-5e675761",
"subnet-8519fde2",
"subnet-89bab8d4"
]
}
],
"ordered_placement_strategy": [],
"placement_constraint": [],
"platform_version": "",
"propagate_tags": "TASK_DEFINITION",
"tags": null,
"task_count": 1,
"task_definition_arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/sales_summaries_prod:1"
}
],
"event_bus_name": "default",
"http_target": [],
"id": "sales-summaries-schedule-prod-sales-summaries",
"input": "",
"input_path": "",
"input_transformer": [],
"kinesis_target": [],
"redshift_target": [],
"retry_policy": [],
"role_arn": "arn:aws:iam::679918342773:role/service-role/Amazon_EventBridge_Invoke_ECS_1758992733",
"rule": "sales-summaries-schedule-prod",
"run_command_targets": [],
"sqs_target": [],
"target_id": "sales-summaries"
},
"sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [
"module.sales_summaries_job.aws_cloudwatch_event_rule.schedule",
"module.sales_summaries_job.aws_ecs_task_definition.background_taskdef"
]
}
]
},
{
"module": "module.sales_summaries_job[0]",
"mode": "managed",
"type": "aws_ecs_task_definition",
"name": "background_taskdef",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 1,
"attributes": {
"arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/sales_summaries_prod:1",
"arn_without_revision": "arn:aws:ecs:us-east-1:679918342773:task-definition/sales_summaries_prod",
"container_definitions": "[{\"cpu\":0,\"dockerLabels\":{\"com.datadoghq.tags.env\":\"prod\",\"com.datadoghq.tags.service\":\"sales-summaries\"},\"environment\":[{\"name\":\"DD_CONTAINER_ENV_AS_TAGS\",\"value\":\"{\\\"INTEGREAT_JOB\\\":\\\"background_job\\\"}\"},{\"name\":\"DD_ENV\",\"value\":\"prod\"},{\"name\":\"DD_SERVICE\",\"value\":\"sales-summaries\"},{\"name\":\"INTEGREAT_JOB\",\"value\":\"sales-summaries\"},{\"name\":\"config\",\"value\":\"/usr/local/config/prod-background-worker.edn\"}],\"essential\":true,\"image\":\"679918342773.dkr.ecr.us-east-1.amazonaws.com/integreat-cloud:prod\",\"logConfiguration\":{\"logDriver\":\"awslogs\",\"options\":{\"awslogs-group\":\"/ecs/integreat-app-prod\",\"awslogs-region\":\"us-east-1\",\"awslogs-stream-prefix\":\"ecs\"}},\"mountPoints\":[],\"name\":\"integreat-app\",\"portMappings\":[{\"containerPort\":9000,\"hostPort\":9000,\"protocol\":\"tcp\"},{\"containerPort\":9090,\"hostPort\":9090,\"protocol\":\"tcp\"}],\"systemControls\":[],\"volumesFrom\":[]},{\"cpu\":0,\"environment\":[{\"name\":\"DD_API_KEY\",\"value\":\"ce10d932c47b358e81081ae67bd8c112\"},{\"name\":\"ECS_FARGATE\",\"value\":\"true\"}],\"essential\":true,\"image\":\"public.ecr.aws/datadog/agent:latest\",\"mountPoints\":[],\"name\":\"datadog-agent\",\"portMappings\":[],\"systemControls\":[],\"volumesFrom\":[]}]",
"cpu": "2048",
"ephemeral_storage": [],
"execution_role_arn": "arn:aws:iam::679918342773:role/ecsTaskExecutionRole",
"family": "sales_summaries_prod",
"id": "sales_summaries_prod",
"inference_accelerator": [],
"ipc_mode": "",
"memory": "4096",
"network_mode": "awsvpc",
"pid_mode": "",
"placement_constraints": [],
"proxy_configuration": [],
"requires_compatibilities": [
"FARGATE"
],
"revision": 1,
"runtime_platform": [],
"skip_destroy": false,
"tags": null,
"tags_all": {},
"task_role_arn": "arn:aws:iam::679918342773:role/datomic-ddb",
"volume": []
},
"sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3091,6 +3292,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3155,6 +3357,7 @@
"target_id": "square-import-job" "target_id": "square-import-job"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.square_import_job.aws_cloudwatch_event_rule.schedule", "module.square_import_job.aws_cloudwatch_event_rule.schedule",
@@ -3200,6 +3403,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3229,6 +3433,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3293,6 +3498,7 @@
"target_id": "sysco" "target_id": "sysco"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.sysco_job.aws_cloudwatch_event_rule.schedule", "module.sysco_job.aws_cloudwatch_event_rule.schedule",
@@ -3338,6 +3544,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3367,6 +3574,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3431,6 +3639,7 @@
"target_id": "vendor-usages" "target_id": "vendor-usages"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.vendor_usages_job.aws_cloudwatch_event_rule.schedule", "module.vendor_usages_job.aws_cloudwatch_event_rule.schedule",
@@ -3476,6 +3685,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3517,6 +3727,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3546,6 +3757,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3610,6 +3822,7 @@
"target_id": "yodlee2" "target_id": "yodlee2"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.yodlee2_job.aws_cloudwatch_event_rule.schedule", "module.yodlee2_job.aws_cloudwatch_event_rule.schedule",
@@ -3655,6 +3868,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]

231
terraform/terraform.tfstate.d/prod/terraform.tfstate.backup
View File

@@ -1,7 +1,7 @@
{ {
"version": 4, "version": 4,
"terraform_version": "1.9.2", "terraform_version": "1.15.1",
"serial": 714, "serial": 718,
"lineage": "9b630886-8cee-a57d-c7a2-4f19f13f9c51", "lineage": "9b630886-8cee-a57d-c7a2-4f19f13f9c51",
"outputs": { "outputs": {
"aws_access_key_id": { "aws_access_key_id": {
@@ -115,7 +115,8 @@
"usage_operation": "RunInstances", "usage_operation": "RunInstances",
"virtualization_type": "hvm" "virtualization_type": "hvm"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -133,7 +134,8 @@
"id": "679918342773", "id": "679918342773",
"user_id": "AIDAJPUJFTOKO4IRADMV4" "user_id": "AIDAJPUJFTOKO4IRADMV4"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -181,7 +183,8 @@
], ],
"version": "2012-10-17" "version": "2012-10-17"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -246,6 +249,7 @@
} }
] ]
], ],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -311,6 +315,7 @@
} }
] ]
], ],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -346,6 +351,7 @@
"type": "gp2" "type": "gp2"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwLCJ1cGRhdGUiOjMwMDAwMDAwMDAwMH19", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwLCJ1cGRhdGUiOjMwMDAwMDAwMDAwMH19",
"dependencies": [ "dependencies": [
"aws_instance.solr_ec2", "aws_instance.solr_ec2",
@@ -435,7 +441,7 @@
], ],
"tags": {}, "tags": {},
"tags_all": {}, "tags_all": {},
"task_definition": "arn:aws:ecs:us-east-1:679918342773:task-definition/integreat_app_prod:837", "task_definition": "arn:aws:ecs:us-east-1:679918342773:task-definition/integreat_app_prod:841",
"timeouts": { "timeouts": {
"create": null, "create": null,
"delete": null, "delete": null,
@@ -445,6 +451,7 @@
"wait_for_steady_state": true "wait_for_steady_state": true
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_ecs_task_definition.integreat_app", "aws_ecs_task_definition.integreat_app",
@@ -535,6 +542,7 @@
"wait_for_steady_state": true "wait_for_steady_state": true
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjoxMjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_ecs_task_definition.solr", "aws_ecs_task_definition.solr",
@@ -580,6 +588,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -636,6 +645,7 @@
] ]
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"aws_efs_file_system.solr_storage" "aws_efs_file_system.solr_storage"
@@ -667,9 +677,9 @@
"provisioned_throughput_in_mibps": 0, "provisioned_throughput_in_mibps": 0,
"size_in_bytes": [ "size_in_bytes": [
{ {
"value": 1432420352, "value": 1434062848,
"value_in_ia": 0, "value_in_ia": 0,
"value_in_standard": 1432420352 "value_in_standard": 1434062848
} }
], ],
"tags": { "tags": {
@@ -681,6 +691,7 @@
"throughput_mode": "bursting" "throughput_mode": "bursting"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -719,6 +730,7 @@
} }
] ]
], ],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_iam_user.app_user" "aws_iam_user.app_user"
] ]
@@ -744,7 +756,8 @@
"tags_all": {}, "tags_all": {},
"unique_id": "AIDAINFBWI2I7A3TKPGW2" "unique_id": "AIDAINFBWI2I7A3TKPGW2"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -762,6 +775,7 @@
"user": "integreat-prod" "user": "integreat-prod"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_iam_user.app_user" "aws_iam_user.app_user"
] ]
@@ -908,6 +922,7 @@
] ]
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"data.aws_ami.amazon_linux_2023" "data.aws_ami.amazon_linux_2023"
@@ -1014,6 +1029,7 @@
"zone_id": "Z35SXDOTRQ7X7K" "zone_id": "Z35SXDOTRQ7X7K"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19" "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19"
} }
] ]
@@ -1063,6 +1079,7 @@
} }
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_lb.integreat_app" "aws_lb.integreat_app"
@@ -1106,6 +1123,7 @@
} }
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsicmVhZCI6NjAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_acm_certificate.cert", "aws_acm_certificate.cert",
@@ -1173,6 +1191,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"aws_acm_certificate.cert", "aws_acm_certificate.cert",
@@ -1242,6 +1261,7 @@
"vpc_id": "vpc-b5b7d6ce" "vpc_id": "vpc-b5b7d6ce"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1320,6 +1340,7 @@
"website_endpoint": "data.prod.app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" "website_endpoint": "data.prod.app.integreatconsult.com.s3-website-us-east-1.amazonaws.com"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1391,6 +1412,7 @@
"website_endpoint": null "website_endpoint": null
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"data.aws_caller_identity.current" "data.aws_caller_identity.current"
@@ -1489,6 +1511,7 @@
"website_endpoint": "app.integreatconsult.com.s3-website-us-east-1.amazonaws.com" "website_endpoint": "app.integreatconsult.com.s3-website-us-east-1.amazonaws.com"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1560,6 +1583,7 @@
"website_endpoint": null "website_endpoint": null
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjM2MDAwMDAwMDAwMDAsInJlYWQiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19" "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjM2MDAwMDAwMDAwMDAsInJlYWQiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19"
} }
] ]
@@ -1591,6 +1615,7 @@
"topic": [] "topic": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_s3_bucket.invoices", "aws_s3_bucket.invoices",
"aws_sqs_queue.integreat-mail", "aws_sqs_queue.integreat-mail",
@@ -1613,6 +1638,7 @@
"policy": "{\"Statement\":[{\"Action\":\"s3:*\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam::679918342773:role/http-proxy\",\"arn:aws:iam::679918342773:role/datomic-ddb\"]},\"Resource\":[\"arn:aws:s3:::toast.prod.app.integreatconsult.com/*\",\"arn:aws:s3:::toast.prod.app.integreatconsult.com\"],\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}" "policy": "{\"Statement\":[{\"Action\":\"s3:*\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam::679918342773:role/http-proxy\",\"arn:aws:iam::679918342773:role/datomic-ddb\"]},\"Resource\":[\"arn:aws:s3:::toast.prod.app.integreatconsult.com/*\",\"arn:aws:s3:::toast.prod.app.integreatconsult.com\"],\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"aws_s3_bucket.toast_bucket", "aws_s3_bucket.toast_bucket",
@@ -1638,6 +1664,7 @@
"service_id": "srv-ren22oppkwwryqqr" "service_id": "srv-ren22oppkwwryqqr"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==", "private": "bnVsbA==",
"dependencies": [ "dependencies": [
"aws_instance.solr_ec2", "aws_instance.solr_ec2",
@@ -1685,6 +1712,7 @@
"type": "DNS_HTTP" "type": "DNS_HTTP"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1727,6 +1755,7 @@
"type": "DNS_HTTP" "type": "DNS_HTTP"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1769,6 +1798,7 @@
"type": "DNS_HTTP" "type": "DNS_HTTP"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1810,6 +1840,7 @@
"workmail_action": [] "workmail_action": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_s3_bucket.invoices", "aws_s3_bucket.invoices",
"aws_ses_receipt_rule_set.main", "aws_ses_receipt_rule_set.main",
@@ -1831,7 +1862,8 @@
"id": "default-rule-set", "id": "default-rule-set",
"rule_set_name": "default-rule-set" "rule_set_name": "default-rule-set"
}, },
"sensitive_attributes": [] "sensitive_attributes": [],
"identity_schema_version": 0
} }
] ]
}, },
@@ -1868,6 +1900,7 @@
"visibility_timeout_seconds": 30 "visibility_timeout_seconds": 30
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1905,6 +1938,7 @@
"visibility_timeout_seconds": 30 "visibility_timeout_seconds": 30
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"dependencies": [ "dependencies": [
"aws_s3_bucket.invoices", "aws_s3_bucket.invoices",
"data.aws_caller_identity.current" "data.aws_caller_identity.current"
@@ -1945,6 +1979,7 @@
"visibility_timeout_seconds": 30 "visibility_timeout_seconds": 30
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -1968,6 +2003,7 @@
"volume_id": "vol-0069283d41ff6c010" "volume_id": "vol-0069283d41ff6c010"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwfX0=", "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDAsImRlbGV0ZSI6MzAwMDAwMDAwMDAwfX0=",
"dependencies": [ "dependencies": [
"aws_ebs_volume.solr_ec2_storage", "aws_ebs_volume.solr_ec2_storage",
@@ -2014,6 +2050,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2043,6 +2080,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2107,6 +2145,7 @@
"target_id": "close-auto-invoices" "target_id": "close-auto-invoices"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.close_auto_invoices_job.aws_cloudwatch_event_rule.schedule", "module.close_auto_invoices_job.aws_cloudwatch_event_rule.schedule",
@@ -2152,144 +2191,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "identity_schema_version": 0,
}
]
},
{
"module": "module.current_balance_cache[0]",
"mode": "managed",
"type": "aws_cloudwatch_event_rule",
"name": "schedule",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"index_key": 0,
"schema_version": 0,
"attributes": {
"arn": "arn:aws:events:us-east-1:679918342773:rule/current-balance-cache-schedule-prod",
"description": "",
"event_bus_name": "default",
"event_pattern": null,
"id": "current-balance-cache-schedule-prod",
"is_enabled": true,
"name": "current-balance-cache-schedule-prod",
"name_prefix": "",
"role_arn": "",
"schedule_expression": "rate(30 minutes)",
"tags": {},
"tags_all": {}
},
"sensitive_attributes": [],
"private": "bnVsbA=="
}
]
},
{
"module": "module.current_balance_cache[0]",
"mode": "managed",
"type": "aws_cloudwatch_event_target",
"name": "job_target",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"index_key": 0,
"schema_version": 1,
"attributes": {
"arn": "arn:aws:ecs:us-east-1:679918342773:cluster/default",
"batch_target": [],
"dead_letter_config": [],
"ecs_target": [
{
"capacity_provider_strategy": [],
"enable_ecs_managed_tags": false,
"enable_execute_command": false,
"group": "",
"launch_type": "FARGATE",
"network_configuration": [
{
"assign_public_ip": true,
"security_groups": [
"sg-004e5855310c453a3",
"sg-02d167406b1082698"
],
"subnets": [
"subnet-5e675761",
"subnet-8519fde2",
"subnet-89bab8d4"
]
}
],
"ordered_placement_strategy": [],
"placement_constraint": [],
"platform_version": "",
"propagate_tags": "TASK_DEFINITION",
"tags": {},
"task_count": 1,
"task_definition_arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/current_balance_cache_prod:3"
}
],
"event_bus_name": "default",
"http_target": [],
"id": "current-balance-cache-schedule-prod-current-balance-cache",
"input": "",
"input_path": "",
"input_transformer": [],
"kinesis_target": [],
"redshift_target": [],
"retry_policy": [],
"role_arn": "arn:aws:iam::679918342773:role/service-role/Amazon_EventBridge_Invoke_ECS_1758992733",
"rule": "current-balance-cache-schedule-prod",
"run_command_targets": [],
"sqs_target": [],
"target_id": "current-balance-cache"
},
"sensitive_attributes": [],
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [
"module.current_balance_cache.aws_cloudwatch_event_rule.schedule",
"module.current_balance_cache.aws_ecs_task_definition.background_taskdef"
]
}
]
},
{
"module": "module.current_balance_cache[0]",
"mode": "managed",
"type": "aws_ecs_task_definition",
"name": "background_taskdef",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 1,
"attributes": {
"arn": "arn:aws:ecs:us-east-1:679918342773:task-definition/current_balance_cache_prod:3",
"arn_without_revision": "arn:aws:ecs:us-east-1:679918342773:task-definition/current_balance_cache_prod",
"container_definitions": "[{\"cpu\":0,\"dockerLabels\":{\"com.datadoghq.tags.env\":\"prod\",\"com.datadoghq.tags.service\":\"current-balance-cache\"},\"environment\":[{\"name\":\"DD_CONTAINER_ENV_AS_TAGS\",\"value\":\"{\\\"INTEGREAT_JOB\\\":\\\"background_job\\\"}\"},{\"name\":\"DD_ENV\",\"value\":\"prod\"},{\"name\":\"DD_SERVICE\",\"value\":\"current-balance-cache\"},{\"name\":\"INTEGREAT_JOB\",\"value\":\"current-balance-cache\"},{\"name\":\"config\",\"value\":\"/usr/local/config/prod-background-worker.edn\"}],\"essential\":true,\"image\":\"679918342773.dkr.ecr.us-east-1.amazonaws.com/integreat-cloud:prod\",\"logConfiguration\":{\"logDriver\":\"awslogs\",\"options\":{\"awslogs-group\":\"/ecs/integreat-app-prod\",\"awslogs-region\":\"us-east-1\",\"awslogs-stream-prefix\":\"ecs\"}},\"mountPoints\":[],\"name\":\"integreat-app\",\"portMappings\":[{\"containerPort\":9000,\"hostPort\":9000,\"protocol\":\"tcp\"},{\"containerPort\":9090,\"hostPort\":9090,\"protocol\":\"tcp\"}],\"systemControls\":[],\"volumesFrom\":[]},{\"cpu\":0,\"environment\":[{\"name\":\"DD_API_KEY\",\"value\":\"ce10d932c47b358e81081ae67bd8c112\"},{\"name\":\"ECS_FARGATE\",\"value\":\"true\"}],\"essential\":true,\"image\":\"public.ecr.aws/datadog/agent:latest\",\"mountPoints\":[],\"name\":\"datadog-agent\",\"portMappings\":[],\"systemControls\":[],\"volumesFrom\":[]}]",
"cpu": "512",
"ephemeral_storage": [],
"execution_role_arn": "arn:aws:iam::679918342773:role/ecsTaskExecutionRole",
"family": "current_balance_cache_prod",
"id": "current_balance_cache_prod",
"inference_accelerator": [],
"ipc_mode": "",
"memory": "2048",
"network_mode": "awsvpc",
"pid_mode": "",
"placement_constraints": [],
"proxy_configuration": [],
"requires_compatibilities": [
"FARGATE"
],
"revision": 3,
"runtime_platform": [],
"skip_destroy": false,
"tags": {},
"tags_all": {},
"task_role_arn": "arn:aws:iam::679918342773:role/datomic-ddb",
"volume": []
},
"sensitive_attributes": [],
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2319,6 +2221,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2383,6 +2286,7 @@
"target_id": "import-uploaded-invoices" "target_id": "import-uploaded-invoices"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.import_uploaded_invoices_job.aws_cloudwatch_event_rule.schedule", "module.import_uploaded_invoices_job.aws_cloudwatch_event_rule.schedule",
@@ -2428,6 +2332,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2457,6 +2362,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2521,6 +2427,7 @@
"target_id": "insight-outcome-recommendation" "target_id": "insight-outcome-recommendation"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.insight_outcome_recommendation_job.aws_cloudwatch_event_rule.schedule", "module.insight_outcome_recommendation_job.aws_cloudwatch_event_rule.schedule",
@@ -2566,6 +2473,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2595,6 +2503,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2659,6 +2568,7 @@
"target_id": "intuit" "target_id": "intuit"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.intuit_job.aws_cloudwatch_event_rule.schedule", "module.intuit_job.aws_cloudwatch_event_rule.schedule",
@@ -2704,6 +2614,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2745,6 +2656,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2774,6 +2686,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2838,6 +2751,7 @@
"target_id": "ntg" "target_id": "ntg"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.ntg_job.aws_cloudwatch_event_rule.schedule", "module.ntg_job.aws_cloudwatch_event_rule.schedule",
@@ -2883,6 +2797,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -2912,6 +2827,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -2976,6 +2892,7 @@
"target_id": "plaid" "target_id": "plaid"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.plaid_job.aws_cloudwatch_event_rule.schedule", "module.plaid_job.aws_cloudwatch_event_rule.schedule",
@@ -3021,6 +2938,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3050,6 +2968,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3114,6 +3033,7 @@
"target_id": "reconcile-ledger" "target_id": "reconcile-ledger"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.reconcile_ledger_job.aws_cloudwatch_event_rule.schedule", "module.reconcile_ledger_job.aws_cloudwatch_event_rule.schedule",
@@ -3159,6 +3079,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3200,6 +3121,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3229,6 +3151,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3293,6 +3216,7 @@
"target_id": "square-import-job" "target_id": "square-import-job"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.square_import_job.aws_cloudwatch_event_rule.schedule", "module.square_import_job.aws_cloudwatch_event_rule.schedule",
@@ -3338,6 +3262,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3367,6 +3292,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3431,6 +3357,7 @@
"target_id": "sysco" "target_id": "sysco"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.sysco_job.aws_cloudwatch_event_rule.schedule", "module.sysco_job.aws_cloudwatch_event_rule.schedule",
@@ -3476,6 +3403,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3505,6 +3433,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3569,6 +3498,7 @@
"target_id": "vendor-usages" "target_id": "vendor-usages"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.vendor_usages_job.aws_cloudwatch_event_rule.schedule", "module.vendor_usages_job.aws_cloudwatch_event_rule.schedule",
@@ -3614,6 +3544,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3655,6 +3586,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]
@@ -3684,6 +3616,7 @@
"tags_all": {} "tags_all": {}
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "bnVsbA==" "private": "bnVsbA=="
} }
] ]
@@ -3748,6 +3681,7 @@
"target_id": "yodlee2" "target_id": "yodlee2"
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==", "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [ "dependencies": [
"module.yodlee2_job.aws_cloudwatch_event_rule.schedule", "module.yodlee2_job.aws_cloudwatch_event_rule.schedule",
@@ -3793,6 +3727,7 @@
"volume": [] "volume": []
}, },
"sensitive_attributes": [], "sensitive_attributes": [],
"identity_schema_version": 0,
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
} }
] ]

149
test/clj/auto_ap/jobs/rekey_square_external_ids_test.clj Normal file
View File

@@ -0,0 +1,149 @@
(ns auto-ap.jobs.rekey-square-external-ids-test
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
[auto-ap.jobs.rekey-square-external-ids :as sut]
[clojure.string]
[clojure.test :refer [deftest is testing use-fixtures]]
[datomic.api :as dc]))
(use-fixtures :each wrap-setup)
(def sales-date #inst "2026-08-01T07:00:00.000-00:00")
(defn- charge-count []
(count (dc/q '[:find ?e :where [?e :charge/external-id]] (dc/db conn))))
(defn- charges-of [order]
(->> (dc/datoms (dc/db conn) :eavt order :sales-order/charges)
(map :v)
(map (fn [c] {:eid c
:key (:v (first (dc/datoms (dc/db conn) :eavt c :charge/external-id)))
:total (:v (first (dc/datoms (dc/db conn) :eavt c :charge/total)))
:tip (:v (first (dc/datoms (dc/db conn) :eavt c :charge/tip)))}))
vec))
(defn- parents-of [charge]
(reduce (fn [n _] (inc n)) 0 (dc/datoms (dc/db conn) :vaet charge :sales-order/charges)))
(defn- two-orders-sharing-one-charge []
(let [{:strs [test-client-id]} (setup-test-data [])
other (get-in @(dc/transact conn [{:db/id "other" :client/code "NGCC"}]) [:tempids "other"])
tx @(dc/transact conn [{:db/id "charge"
:charge/external-id "square/charge/shared1"
:charge/type-name "CARD"
:charge/total 120.0
:charge/tip 20.0}
{:db/id "order-a"
:sales-order/external-id "square/order/NGCD-CD-o1"
:sales-order/client test-client-id
:sales-order/location "CD"
:sales-order/date sales-date
:sales-order/charges ["charge"]}
{:db/id "order-b"
:sales-order/external-id "square/order/NGCC-CC-o1"
:sales-order/client other
:sales-order/location "CC"
:sales-order/date sales-date
:sales-order/charges ["charge"]}])]
{:order-a (get-in tx [:tempids "order-a"])
:order-b (get-in tx [:tempids "order-b"])
:charge (get-in tx [:tempids "charge"])
:code-a (:client/code (dc/entity (dc/db conn) test-client-id))
:code-b "NGCC"}))
(deftest a-shared-charge-starts-with-two-parents
(testing "the condition under test really exists before the split runs"
(let [{:keys [charge]} (two-orders-sharing-one-charge)]
(is (= 1 (charge-count)))
(is (= 2 (parents-of charge))
"one charge entity, referenced by both clients' orders"))))
(deftest split-gives-each-order-its-own-charge
(testing "each order ends up with its own charge, scoped to its own client, carrying the same
amounts — so the component relationship means what it says and retracting one order
cannot delete the other's payment"
(let [{:keys [order-a order-b code-a code-b]} (two-orders-sharing-one-charge)
result (sut/split-and-rekey-charges! [order-a order-b] 100)]
(is (= {:rekeyed 1 :cloned 1} result) "first order keeps it, second gets a copy")
(is (= 2 (charge-count)) "exactly one new entity was created")
(let [a (charges-of order-a)
b (charges-of order-b)]
(is (= 1 (count a)))
(is (= 1 (count b)))
(is (= (str "square/charge/" code-a "-CD-shared1") (:key (first a))))
(is (= (str "square/charge/" code-b "-CC-shared1") (:key (first b))))
(is (not= (:eid (first a)) (:eid (first b))) "two distinct entities")
(is (= 120.0 (:total (first a)) (:total (first b))) "amounts copied")
(is (= 20.0 (:tip (first a)) (:tip (first b))) "tips copied")
(is (= 1 (parents-of (:eid (first a)))))
(is (= 1 (parents-of (:eid (first b))))
"no charge has more than one parent order any more")))))
(deftest split-leaves-an-unshared-charge-alone
(testing "an order that already owns its charge outright is only re-keyed, never cloned"
(let [{:strs [test-client-id]} (setup-test-data [])
tx @(dc/transact conn [{:db/id "charge"
:charge/external-id "square/charge/solo1"
:charge/total 50.0}
{:db/id "order"
:sales-order/external-id "square/order/NGCD-CD-o2"
:sales-order/client test-client-id
:sales-order/location "CD"
:sales-order/date sales-date
:sales-order/charges ["charge"]}])
order (get-in tx [:tempids "order"])
code (:client/code (dc/entity (dc/db conn) test-client-id))]
(is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [order] 100)))
(is (= 1 (charge-count)) "nothing was created")
(is (= (str "square/charge/" code "-CD-solo1") (:key (first (charges-of order))))))))
(deftest split-is-idempotent
(testing "re-running over an already-split database changes nothing further"
(let [{:keys [order-a order-b]} (two-orders-sharing-one-charge)]
(sut/split-and-rekey-charges! [order-a order-b] 100)
(let [after-first (charge-count)]
(is (= {:rekeyed 0 :cloned 0} (sut/split-and-rekey-charges! [order-a order-b] 100))
"every charge already carries the key its order expects, so there is nothing to do")
(is (= after-first (charge-count)) "and no further entities appear")))))
(deftest clone-is-not-double-scoped-across-batches
(testing "with a batch size of one, the second order sees a charge already carrying the first
client's scope. It must clone using the underlying Square id, not re-scope the scoped
key — otherwise the entity ends up keyed NGCD-CD-NGCC-CC-<id>, the importer computes
the correct key, misses, and creates a second charge that doubles the tender."
(let [{:keys [order-a order-b code-a code-b]} (two-orders-sharing-one-charge)]
(sut/split-and-rekey-charges! [order-a order-b] 1)
(let [ka (:key (first (charges-of order-a)))
kb (:key (first (charges-of order-b)))]
(is (= (str "square/charge/" code-a "-CD-shared1") ka))
(is (= (str "square/charge/" code-b "-CC-shared1") kb))
(is (not (clojure.string/includes? kb (str code-a "-CD")))
"the clone carries one scope, not two")
(is (= 2 (charge-count)))))))
(deftest two-orders-of-the-same-client-keep-sharing
(testing "one payment covering two of the SAME client's orders is left shared, on purpose.
There is no second name to give a copy — both orders compute the same one — and a copy
would double that client's takings for the day. The component cascade still reaches
these, which is why remove-voided-orders needs its own guard."
(let [{:strs [test-client-id]} (setup-test-data [])
tx @(dc/transact conn [{:db/id "charge"
:charge/external-id "square/charge/same1"
:charge/total 75.0}
{:db/id "o1" :sales-order/external-id "square/order/x-1"
:sales-order/client test-client-id :sales-order/location "CD"
:sales-order/date sales-date :sales-order/charges ["charge"]}
{:db/id "o2" :sales-order/external-id "square/order/x-2"
:sales-order/client test-client-id :sales-order/location "CD"
:sales-order/date sales-date :sales-order/charges ["charge"]}])
o1 (get-in tx [:tempids "o1"]) o2 (get-in tx [:tempids "o2"])
code (:client/code (dc/entity (dc/db conn) test-client-id))]
(is (= {:rekeyed 1 :cloned 0} (sut/split-and-rekey-charges! [o1 o2] 100))
"renamed once, not copied")
(is (= 1 (charge-count)) "no copy was made, so the takings are not doubled")
(is (= (str "square/charge/" code "-CD-same1") (:key (first (charges-of o1)))))
(is (= (:eid (first (charges-of o1))) (:eid (first (charges-of o2))))
"both orders still point at the one payment"))))

204
test/clj/auto_ap/jobs/sales_summaries_test.clj Normal file
View File

@@ -0,0 +1,204 @@
(ns auto-ap.jobs.sales-summaries-test
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.datomic.sales-summaries :as d-ss]
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
[auto-ap.jobs.sales-summaries :as sut]
[clojure.test :refer [deftest is testing use-fixtures]]
[datomic.api :as dc]))
(use-fixtures :each wrap-setup)
(def sales-date #inst "2026-08-01T07:00:00.000-00:00")
(defn- order
"A sales order on `sales-date`, carrying whatever the case under test needs. The external id
is Square-shaped by default because `get-service-charges` falls back to it when an order has
no `:sales-order/vendor`."
[client id attrs]
(merge {:db/id (str "order-" id)
:sales-order/external-id (str "square/order/TEST-" id)
:sales-order/client client
:sales-order/date sales-date
:sales-order/total 100.0}
attrs))
(defn- charge [id attrs]
(merge {:db/id (str "charge-" id)
:charge/external-id (str "square/charge/" id)
:charge/type-name "CARD"
:charge/total 100.0}
attrs))
(defn- tip-for [client]
(:ledger-mapped/amount (#'sut/get-tip client sales-date)))
(defn- service-charges-for [client]
(#'sut/get-service-charges client sales-date))
(defn- enable-service-charges! [client]
@(dc/transact conn [{:db/id client
:client/feature-flags [sut/service-charges-flag]}]))
(deftest tip-counts-a-reversal-on-an-untendered-order
(testing "a return-only order has no tender to join through, so its negative tip must come
from the order or the day credits a tip that was handed back"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "return-only" {:sales-order/tip -12.0})])
(is (= -12.0 (tip-for test-client-id))))))
(deftest tip-on-a-tendered-order-still-comes-from-the-tender
(testing "the tender carries a tip the order does not — auto-gratuity booked as a service
charge. Reading the order instead of the tender would drop it."
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "tendered"
{:sales-order/tip 0.0
:sales-order/charges [(charge "tendered" {:charge/tip 50.0})]})])
(is (= 50.0 (tip-for test-client-id))))))
(deftest tip-on-an-ordinary-order-is-counted-once
(testing "an order that agrees with its tender is not double counted by the additive form"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "ordinary"
{:sales-order/tip 5.0
:sales-order/charges [(charge "ordinary" {:charge/tip 5.0})]})])
(is (= 5.0 (tip-for test-client-id))))))
(deftest service-charges-need-the-feature-flag
(testing "without the flag the summary behaves exactly as it does today"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(order test-client-id "square-sc"
{:sales-order/vendor :vendor/ccp-square
:sales-order/service-charge 50.0})])
(is (nil? (service-charges-for test-client-id))))))
(deftest service-charges-credit-square-orders
(testing "a service charge rides along in the tender, so it needs a credit to match"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "square-sc"
{:sales-order/vendor :vendor/ccp-square
:sales-order/service-charge 50.0})])
(let [item (service-charges-for test-client-id)]
(is (= 50.0 (:ledger-mapped/amount item)))
(is (= :ledger-side/credit (:ledger-mapped/ledger-side item)))
(is (= "Service Charges" (:sales-summary-item/category item)))))))
(deftest service-charges-count-both-signs
(testing "a returned catering fee arrives as a negative service charge and is subtracted back
out of returns, so dropping negatives loses the reversal"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "refunded-fee"
{:sales-order/vendor :vendor/ccp-square
:sales-order/service-charge -140.0})])
(is (= -140.0 (:ledger-mapped/amount (service-charges-for test-client-id)))))))
(deftest service-charges-exclude-non-square-vendors
(testing "ezCater service charges are commission deducted from the restaurant rather than
collected from the diner, so crediting them would make the day worse"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "ezcater-sc"
{:sales-order/external-id "ezcater/order/TEST-ezcater-sc"
:sales-order/vendor :vendor/ccp-ezcater
:sales-order/service-charge -75.0})])
(is (nil? (service-charges-for test-client-id))))))
(deftest service-charges-recognise-square-orders-that-carry-no-vendor
(testing "whole eras of Square orders have no :sales-order/vendor at all; a gate on vendor
alone would silently credit nothing"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "vendorless" {:sales-order/service-charge 12.5})])
(is (= 12.5 (:ledger-mapped/amount (service-charges-for test-client-id)))))))
(deftest service-charges-ignore-vendorless-orders-from-other-sources
(testing "the external id fallback is Square-specific, not a catch-all for missing vendors"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-service-charges! test-client-id)
@(dc/transact conn [(order test-client-id "ezcater-vendorless"
{:sales-order/external-id "ezcater/order/TEST-ezcater-vendorless"
:sales-order/service-charge -75.0})])
(is (nil? (service-charges-for test-client-id))))))
(defn- refund
"A card refund on `sales-date`. The client+date tuple is set explicitly because
`scan-sales-refunds` walks that index rather than the plain attributes."
[client id total]
{:db/id (str "refund-" id)
:sales-refund/external-id (str "square/refund/TEST-" id)
:sales-refund/client client
:sales-refund/date sales-date
:sales-refund/client+date [client sales-date]
:sales-refund/type "CARD"
:sales-refund/total total})
(defn- enable-refund-only-returns! [client]
@(dc/transact conn [{:db/id client
:client/feature-flags [sut/refund-only-returns-flag]}]))
(defn- returns-for [client]
(#'sut/get-returns client sales-date))
(deftest refund-only-day-needs-the-feature-flag
(testing "without the flag a day of refunds and no sales books no return, as it does today"
(let [{:strs [test-client-id]} (setup-test-data [])]
@(dc/transact conn [(refund test-client-id "unflagged" 40.0)])
(is (nil? (returns-for test-client-id))))))
(deftest refund-only-day-books-a-return_and_balances
(testing "a refund credited on a day with no sales has nothing to offset it, so the day is out
by the refunded amount until a return is recognised against it"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-refund-only-returns! test-client-id)
@(dc/transact conn [(refund test-client-id "orphan-a" 30.0)
(refund test-client-id "orphan-b" 10.0)])
(let [returns (returns-for test-client-id)]
(is (= 40.0 (:ledger-mapped/amount returns)))
(is (= :ledger-side/debit (:ledger-mapped/ledger-side returns)))
(is (= 0.0 (d-ss/imbalance (cons returns (sut/get-refund-items test-client-id sales-date))))
"the refund credits and the return debit cancel exactly")))))
(deftest a-day-that-traded-keeps-its-order-derived-return
(testing "the guard is what makes this safe: a day with sales is left entirely alone, so no
trading day can have its return moved by an unmatched refund"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-refund-only-returns! test-client-id)
@(dc/transact conn [(order test-client-id "traded" {:sales-order/returns 7.0})
(refund test-client-id "same-day" 40.0)])
(is (= 7.0 (:ledger-mapped/amount (returns-for test-client-id)))
"the order's own return, not the refunded total"))))
(deftest a-day-that-traded-and-returned-nothing-books-no-return
(testing "sales with no returns must not pick up the refunded total either — the guard is on
whether the client traded, not on whether the order-derived figure happened to be nil"
(let [{:strs [test-client-id]} (setup-test-data [])]
(enable-refund-only-returns! test-client-id)
@(dc/transact conn [(order test-client-id "traded-no-returns" {})
(refund test-client-id "unmatched" 40.0)])
(is (nil? (returns-for test-client-id))))))
(deftest dirty-summaries-stop-at-the-client-boundary
(testing "every dirty day for the client is returned, and none belonging to another client.
:sales-summary/client+dirty sorts by client, so an unbounded index scan would walk
every later client's summaries too — correct, but quadratic in the summary count."
(let [{:strs [test-client-id]} (setup-test-data [])
other (get-in @(dc/transact conn [{:db/id "other" :client/code (str "OTHER" (rand-int 100000))}])
[:tempids "other"])
day (fn [client d dirty?]
{:sales-summary/client client
:sales-summary/date d
:sales-summary/dirty dirty?})]
@(dc/transact conn [(day test-client-id #inst "2026-08-01T07:00:00.000-00:00" true)
(day test-client-id #inst "2026-08-02T07:00:00.000-00:00" true)
(day test-client-id #inst "2026-08-03T07:00:00.000-00:00" false)
(day other #inst "2026-08-01T07:00:00.000-00:00" true)
(day other #inst "2026-08-02T07:00:00.000-00:00" true)])
(let [mine (sut/dirty-sales-summaries test-client-id)]
(is (= 2 (count mine)) "both dirty days, and not the clean one")
(is (every? #(= test-client-id (:db/id (:sales-summary/client %))) mine)
"and nothing belonging to the other client"))
(is (= 2 (count (sut/dirty-sales-summaries other)))
"the other client's own dirty days are still found"))))

118
test/clj/auto_ap/square/core3_test.clj Normal file
View File

@@ -0,0 +1,118 @@
(ns auto-ap.square.core3-test
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.integration.util :refer [setup-test-data wrap-setup]]
[auto-ap.square.core3 :as sut]
[clojure.test :refer [deftest is testing use-fixtures]]
[datomic.api :as dc]))
(use-fixtures :each wrap-setup)
(def client {:client/code "NGCD"})
(def location {:square-location/client-location "CD"})
(defn- refund-count []
(count (dc/q '[:find ?e :where [?e :sales-refund/external-id]] (dc/db conn))))
(defn- resolve-refund [id]
(sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" client location id))
(deftest scoped-key-carries-client-and-location
(testing "the same shape sales order keys already use, so a shared location cannot contend"
(is (= "square/refund/NGCD-CD-abc" (sut/scoped-key "square/refund/" client location "abc")))
(is (= "square/charge/NGCD-CD-xyz" (sut/scoped-key "square/charge/" client location "xyz")))))
(deftest legacy-keyed-entity-is-updated-not-duplicated
(testing "an entity still carrying its unscoped key is found and re-keyed in place.
This is the sharpest hazard in the migration: these external ids are
:db.unique/identity, so writing the new key without resolving the old one first
matches nothing and creates a second entity, orphaning the original."
(setup-test-data [])
@(dc/transact conn [{:db/id "r"
:sales-refund/external-id "square/refund/abc"
:sales-refund/total 10.0}])
(is (= 1 (refund-count)))
(let [eid (resolve-refund "abc")]
(is (some? eid) "resolves an entity carrying the legacy key")
@(dc/transact conn [{:db/id eid
:sales-refund/external-id (sut/scoped-key "square/refund/" client location "abc")
:sales-refund/total 10.0}])
(is (= 1 (refund-count)) "no second entity was created")
(is (= eid (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/NGCD-CD-abc"]))
"the same entity now answers to the scoped key")
(is (nil? (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/abc"]))
"and no longer to the legacy one"))))
(deftest already-scoped-entity-resolves-by-its-new-key
(testing "re-running the importer after migration finds the entity by the scoped key, so the
migration is not undone and nothing is duplicated"
(setup-test-data [])
@(dc/transact conn [{:db/id "r"
:sales-refund/external-id "square/refund/NGCD-CD-abc"
:sales-refund/total 10.0}])
(is (= (dc/entid (dc/db conn) [:sales-refund/external-id "square/refund/NGCD-CD-abc"])
(resolve-refund "abc")))
(is (= 1 (refund-count)))))
(deftest unknown-id-resolves-to-nothing
(testing "a refund never seen before has no id to pin, so the importer creates it fresh"
(setup-test-data [])
(is (nil? (resolve-refund "never-seen")))))
(deftest two-clients-on-one-location-get-their-own-entities
(testing "the point of the re-key: with the client in the key, a second client importing the
same Square refund creates its own entity instead of taking ownership of the first"
(setup-test-data [])
(let [other {:client/code "NGCC"}
other-loc {:square-location/client-location "CC"}]
@(dc/transact conn [{:db/id "r"
:sales-refund/external-id (sut/scoped-key "square/refund/" client location "shared")
:sales-refund/total 10.0}])
(is (nil? (sut/existing-id (dc/db conn) :sales-refund/external-id "square/refund/" other other-loc "shared"))
"the second client does not resolve onto the first client's entity")
@(dc/transact conn [{:db/id "r2"
:sales-refund/external-id (sut/scoped-key "square/refund/" other other-loc "shared")
:sales-refund/total 10.0}])
(is (= 2 (refund-count)) "two stable entities, one per client, rather than one that flips"))))
(deftest payouts-and-shifts-are-client-scoped-too
(testing "expected deposits and cash drawer shifts are fetched per location, so two clients on
one location collide on them exactly as refunds and charges did"
(is (= "square/payout/NGCD-CD-po1"
(sut/scoped-key "square/payout/" client location "po1")))
(is (= "square/cash-drawer-shift/NGCD-CD-sh1"
(sut/scoped-key "square/cash-drawer-shift/" client location "sh1")))))
(deftest legacy-keyed-deposit-is-updated-not-duplicated
(testing "a payout still carrying its unscoped key is found and re-keyed in place"
(setup-test-data [])
@(dc/transact conn [{:db/id "d"
:expected-deposit/external-id "square/payout/po1"
:expected-deposit/total 100.0}])
(let [eid (sut/existing-id (dc/db conn) :expected-deposit/external-id "square/payout/" client location "po1")]
(is (some? eid) "resolves the entity carrying the legacy key")
@(dc/transact conn [{:db/id eid
:expected-deposit/external-id (sut/scoped-key "square/payout/" client location "po1")
:expected-deposit/total 100.0}])
(is (= 1 (count (dc/q '[:find ?e :where [?e :expected-deposit/external-id]] (dc/db conn))))
"no second deposit was created")
(is (nil? (dc/entid (dc/db conn) [:expected-deposit/external-id "square/payout/po1"]))
"the legacy key is gone"))))
(deftest two-clients-get-their-own-deposit
(testing "with the client in the key, a second client importing the same Square payout creates
its own entity instead of taking ownership of the first"
(setup-test-data [])
(let [other {:client/code "NGCC"}
other-loc {:square-location/client-location "CC"}]
@(dc/transact conn [{:db/id "d"
:expected-deposit/external-id (sut/scoped-key "square/payout/" client location "shared")
:expected-deposit/total 100.0}])
(is (nil? (sut/existing-id (dc/db conn) :expected-deposit/external-id "square/payout/" other other-loc "shared"))
"the second client does not resolve onto the first client's deposit")
@(dc/transact conn [{:db/id "d2"
:expected-deposit/external-id (sut/scoped-key "square/payout/" other other-loc "shared")
:expected-deposit/total 100.0}])
(is (= 2 (count (dc/q '[:find ?e :where [?e :expected-deposit/external-id]] (dc/db conn))))
"two stable entities, one per client, rather than one that flips"))))

153
test/clj/auto_ap/tools/compare_sales_summaries.clj Normal file
View File

@@ -0,0 +1,153 @@
(ns auto-ap.tools.compare-sales-summaries
"Compares sales summaries between two points in the same database.
A verification tool, not part of the running application: it lives on the test/dev classpath so
nothing in production can depend on it. Load it from a REPL when auditing a recompute.
The question this exists to answer is narrower than \"did the totals improve\": it is *which
days changed, and were any of them already balanced*. A day that was balanced before and still
balances after can still have had its line amounts move, and that is a real change to the
books even though no red turns green. Counting only balanced/unbalanced transitions would hide
it entirely.
DO NOT USE `compare-against` — OR ANY `d/as-of` DATABASE — TO COMPARE AMOUNTS. The obvious
reading is that Datomic keeps every past value, so a recompute can be audited against what was
there before with no snapshot. That does not hold here: `:ledger-mapped/amount`,
`:ledger-mapped/ledger-side` and `:ledger-mapped/account` are all `:db/noHistory true`, so
superseded values are discarded rather than retained. A summary that has since been recomputed
reads back through `as-of` with its categories intact and its amounts *absent* — which is
indistinguishable from a legitimate all-zero day, and quietly turns every rewritten summary
into a false \"was balanced, still balances\".
To compare amounts, capture `summaries-in` from a live `(d/db conn)` immediately after each
run, keep the two captures outside the database, and diff those. `compare-window` is safe when
both arguments are live database values; only the historical read is unsound. Categories and
which-days-changed do survive `as-of`, since `:sales-summary-item/category` retains history."
(:require
[auto-ap.datomic :refer [conn]]
[auto-ap.datomic.sales-summaries :as d-ss]
[clj-time.coerce :as c]
[datomic.api :as dc]))
(def item-read
[:sales-summary-item/category
:sales-summary-item/manual?
:ledger-mapped/amount
{:ledger-mapped/ledger-side [:db/ident]}
{:ledger-mapped/account [:account/numeric-code]}])
(defn- cents
"Amounts are doubles carrying float noise, so compare them at the cent — the unit the books are
actually kept in. Without this, 182.87000000000003 and 182.87 read as a change."
[x]
(Math/round (* 100.0 (double (or x 0.0)))))
(defn- line
"One item reduced to what a reader would call \"the number\": category, side, amount, account."
[item]
{:category (:sales-summary-item/category item)
:side (get-in item [:ledger-mapped/ledger-side :db/ident])
:cents (cents (:ledger-mapped/amount item))
:account (get-in item [:ledger-mapped/account :account/numeric-code])})
(defn summaries-in
"`{[client-code date] {:lines … :imbalance … :balanced?}}` for every summary in `[start end)`.
Keyed by client code and date rather than entity id so the two sides line up even if an entity
were recreated between the points being compared."
[db start end]
(->> (dc/q {:find [(list 'pull '?s [:sales-summary/date
{:sales-summary/client [:client/code]}
{:sales-summary/items item-read}])]
:in '[$ ?start ?end]
:where '[[?s :sales-summary/date ?d]
[(>= ?d ?start)]
[(< ?d ?end)]]}
db (c/to-date start) (c/to-date end))
(map first)
(reduce (fn [acc s]
(let [items (map d-ss/<-pulled-item (:sales-summary/items s))]
(assoc acc
[(get-in s [:sales-summary/client :client/code]) (:sales-summary/date s)]
{:lines (frequencies (map line (:sales-summary/items s)))
:imbalance (d-ss/imbalance items)
:balanced? (d-ss/balanced? items)})))
{})))
(defn- classify
"How one client-day differs. `:numbers-changed` is the interesting one — the lines themselves
moved, whether or not the day's balance status did."
[before after]
(cond
(nil? before) :added
(nil? after) :removed
(= (:lines before) (:lines after)) :identical
:else :numbers-changed))
(defn compare-window
"Compares every summary in `[start end)` between two database values.
Returns per-day rows plus the tallies worth reporting, including the one that is easy to miss:
days that were **already balanced** and whose numbers moved anyway."
[before-db after-db start end]
(let [before (summaries-in before-db start end)
after (summaries-in after-db start end)
rows (for [k (distinct (concat (keys before) (keys after)))
:let [b (get before k) a (get after k)]]
{:client (first k)
:date (second k)
:change (classify b a)
:was-balanced? (:balanced? b)
:now-balanced? (:balanced? a)
:before-imbalance (:imbalance b)
:after-imbalance (:imbalance a)
:lines-before (:lines b)
:lines-after (:lines a)})
rows (vec rows)
changed (filter #(= :numbers-changed (:change %)) rows)]
{:rows rows
:tally {:compared (count rows)
:identical (count (filter #(= :identical (:change %)) rows))
:numbers-changed (count changed)
:added (count (filter #(= :added (:change %)) rows))
:removed (count (filter #(= :removed (:change %)) rows))}
:balance-transitions
{:unbalanced->balanced (count (filter #(and (false? (:was-balanced? %)) (true? (:now-balanced? %))) rows))
:balanced->unbalanced (count (filter #(and (true? (:was-balanced? %)) (false? (:now-balanced? %))) rows))
:stayed-balanced (count (filter #(and (true? (:was-balanced? %)) (true? (:now-balanced? %))) rows))
:stayed-unbalanced (count (filter #(and (false? (:was-balanced? %)) (false? (:now-balanced? %))) rows))}
:previously-balanced-and-changed
(->> changed (filter :was-balanced?) vec)}))
(defn line-diff
"Which categories actually moved on one row, as `{category [before-cents after-cents]}`. For
reading a handful of rows by hand once the tallies point at them."
[row]
(let [by-cat (fn [lines] (reduce (fn [m [l n]] (assoc m (:category l) (* n (:cents l)))) {} lines))
b (by-cat (:lines-before row))
a (by-cat (:lines-after row))]
(->> (distinct (concat (keys b) (keys a)))
(keep (fn [cat]
(let [x (get b cat 0) y (get a cat 0)]
(when (not= x y) [cat [(/ x 100.0) (/ y 100.0)]]))))
(into {}))))
(defn compare-against
"Compare the current database against its own past value at basis `t`.
UNSOUND FOR AMOUNTS — see the namespace docstring. The amount, side and account attributes are
`:db/noHistory`, so any summary rewritten since `t` reads back with no amounts and appears
balanced. Kept only for comparing categories and identifying which days changed."
[t start end]
(let [db (dc/db conn)]
(compare-window (dc/as-of db t) db start end)))
(comment
;; the restore point, i.e. production's own summaries before any of this work
(def result (compare-against 209608347
(clj-time.core/date-time 2026 7 15)
(clj-time.core/date-time 2026 8 14)))
(:tally result)
(:balance-transitions result)
(count (:previously-balanced-and-changed result))
(map line-diff (take 3 (:previously-balanced-and-changed result))))