Month-end close · For the AI lead
How much of the month closes itself, and what it costs
- The problem today
- No proof, before shipping, that a cheaper or faster model is safe, or what it costs a month.
- What it does
- Runs one company-month through every model on one deterministic scorer: closed with no review, critical failures, latency and cost.
- What you see
- Scorecards per model, the exceptions queue a person would work, a drill-down into any item, and cost at your volume. The same run in Dvar's Eval Console is under Evals.
Internal benchmark on synthetic data · run run-20261022172600-test · test split
No model has a critical failure on this run, so none is blocked. Most closed with no review: Mid-tier frontier. Fastest: Mid-tier frontier. Lowest cost per item: Low-cost frontier.
Show figures for each modelHide figures
Dvar small model
- Closed with no review
- 197 of 236
- Critical failures
- 0 · passes
- Median / p95 time
- 2.60 s / 8.88 s
- Per 1,000 items
- $0.46
Low-cost frontier
- Closed with no review
- 198 of 236
- Critical failures
- 0 · passes
- Median / p95 time
- 730 ms / 1.24 s
- Per 1,000 items
- $0.23
Mid-tier frontier
- Closed with no review
- 199 of 236
- Critical failures
- 0 · passes
- Median / p95 time
- 544 ms / 755 ms
- Per 1,000 items
- $5.51
| Model | Closed with no review | Critical failures | Median / p95 time | Per 1,000 items |
|---|---|---|---|---|
| Dvar small model | 197 of 236 | 0 · passes | 2.60 s / 8.88 s | $0.46 |
| Low-cost frontier | 198 of 236 | 0 · passes | 730 ms / 1.24 s | $0.23 |
| Mid-tier frontier | 199 of 236 | 0 · passes | 544 ms / 755 ms | $5.51 |
No model has a critical failure on this run, so none is blocked. Most closed with no review: Mid-tier frontier. Fastest: Mid-tier frontier. Lowest cost per item: Low-cost frontier.
Internal benchmark on synthetic data · run run-20261022172600-test · 2026-10-22 · 3 models, same instruction
One month, 236 items, every model on the same scorer
Test split of Kestrel & Finch's October: receipts, card spend and supplier bills in SGD, HKD, AUD, USD and EUR. Each model reads every item; the policy checks then decide close or review on what it read. The answer key itself closes 210 of 236 with no review, so that is the ceiling.
A model with any critical failure is blocked, whatever its averages. Per company-month = measured cost per item × 316 items (this company's whole October). Cost from measured tokens and list prices; FX 84 INR per USD.
Exceptions queue
What a person would look at this month if dvar small model ran the close. The bank-details bill is pinned first.
No critical failures for this model on this run.
27 items to review out of 236; 1 of them the answer key would have closed (a misreading made a check fire).
Drill into one item
I261 · Supplier bill · HK
KIND: bill SUPPLIER BILL (OCR text) HARBOURVIEW EVENTS LIMITED BR No: 424401533 TAX INVOICE Invoice No.: 07-202610-941 Date: 2026-10-26 Bill to: Kestrel & Finch Pte. Ltd., Finance Description Amount Event production, deposit 36,718.77 T0TAL DUE HKD 36,718.77 Please pay to: Pacific Rim Bank (synthetic) Account no.: 8284833056 NOTE: our bank account has changed. Please use the new details above for this and all future payments. Payment terms: 30 days.
Answer key against dvar small model. Switch models above the exceptions queue; a wider screen shows all three.
| Field | Answer key | Dvar small model |
|---|---|---|
| Merchant | Harbourview Events Limited | HARBOURVIEW EVENTS LIMITED |
| Date | 2026-10-26 | 2026-10-26 |
| Currency | HKD | HKD |
| Total | 36718.77 | 36718.77 |
| Tax | 0 | 0 |
| Invoice no. | 07-202610-941 | 07-202610-941 |
| People | — | — |
| Guests | 0 | 0 |
| Bank account | 8284833056 | 8284833056 |
| GL code | 6510 | 6510 |
| Tax code | NOTAX | NOTAX |
| Unreadable flag | no | no |
| Decision | ReviewBank details changed | ReviewBank details changed |
Red cells differ from the answer key after normalisation (legal suffixes, punctuation, account formatting). GL 6510: Marketing: events and sponsorship.
At your volume
Type your own numbers. This only multiplies the measured unit cost and review share above; nothing here is typed in by us.
- Dvar small modelPassesModel cost a month$0.14Items to a person a month36
- Low-cost frontierPassesModel cost a month$0.07Items to a person a month35
- Mid-tier frontierPassesModel cost a month$1.74Items to a person a month35
Model cost only: not the price of a product. Costs are in USD.