What is measured

How the demo scores every model

No AI judge. An answer key, fixed normalisation rules, policy checks in code and a fixed list of critical failures.

Internal benchmark on synthetic data · run run-20261022172600-test · 2026-10-22 · 3 models, same instruction

The data

Kestrel & Finch Pte. Ltd. is invented, and so is everything about it: 24 staff in five markets, 28 GL accounts, 12 suppliers with bank accounts, and 316 items for 2026-10: receipts as messy OCR text (broken lines, legal suffixes, Chinese receipts, Dutch decimal commas, US month-first dates), card transactions with raw descriptors, and supplier bills. A seeded generator (scripts/generate.mjs --seed 42) makes it with no model calls, byte for byte the same every run. 80 items are a dev split for tuning the instruction; the 236 test items are the only ones scored. 40 spend questions run over six months of ledger history plus October.

Planted cases: duplicates, card charges that differ from the receipt (tips), meals over the limit, client dinners with no guests named, cash receipts next to a same-amount card charge from another day, unreadable receipts, card spend above S$100 with no receipt, a bill with no invoice number, and bills whose bank account differs from the vendor record.

The model reads; code decides

Each model reads an item into one JSON shape. The same checks then run on its reading and on the answer key; the item closes with no review only if no check fires.

Staff and travel meals: up to S$50 a person (converted at the company FX table).
Client entertainment needs the names of the people attending.
A receipt is required for any card spend above S$100.
Supplier bills need an invoice number and are paid only to the bank account on the vendor record.

Review reasons: Over the meal limit, Receipt missing, Possible duplicate, Amount differs from card, Bank details changed, Couldn't read it, Needs attendee names, No invoice number.

The scorer

Read the fields: merchant (after dropping legal suffixes and punctuation), ISO date, currency, total and tax to the cent, invoice number, people, number of guests, bank account (letters and digits only), and the unreadable flag.
Match to the card: the exact card transaction id, or none for cash and personal cards.
GL and tax code: GL code and tax code, exact.
Close or review: the checks on the model's reading reach the same action and reasons as on the answer key.
Closed with no review: the model's reading closes the item, the answer key closes it too, every field is right and nothing is critical.

Critical failures (any one blocks a model)

Cleared a bill whose bank details changed
Cleared a duplicate
Matched to the wrong card transaction
Invented an invoice number
Guessed an amount it could not read
Amount off by more than 1%

The run

Run: run-20261022172600-test
Created: 2026-10-22T17:26:00.260Z
Split: test
Instruction: version 2026-10-22.3, sha256 0140d9aae23f3dfb9e25f34bddf8e45b46d78bbb725581fb5cca88306a64df23
Settings: temperature 0, max_tokens 4000, the same instruction and input for every model
Dvar · GLM 5.3 Flash: dvar:glm-5.3-flash: 10 / 40 INR per 1M tokens in / out. customer-bot/config/models.yaml (Prices: Dvar from portal.dvarlabs.ai/models (INR per 1M tokens, checked 2026-10-21).)
gpt-6-luna: openai/gpt-6-luna: 0.1 / 0.5 USD per 1M tokens in / out. OpenRouter reported cost when returned; else customer-bot/config/models.yaml fallback (2026-10-03)
claude-sonnet-5.5: anthropic/claude-sonnet-5.5: 2 / 10 USD per 1M tokens in / out. OpenRouter reported cost when returned; else customer-bot/config/models.yaml fallback (2026-10-03)
FX: 84 INR per USD (customer-bot/config/models.yaml, fx_inr_per_usd (labelled constant)). Company FX table to SGD: SGD 1, HKD 0.165, AUD 0.86, USD 1.3, EUR 1.42 (Company FX table for the demo (fixed constants, not market rates).)

Measured: 236 test items and 30 questions, temperature 0, max_tokens 4000, same instruction for every model.

The instructions every model got

You process expense items for the finance team of Kestrel & Finch, a Singapore company with staff in SG, HK, AU, US and NL.
Read the item and reply with one JSON object only, no prose:
{"merchant": string|null, "date": "YYYY-MM-DD"|null, "currency": "SGD"|"HKD"|"AUD"|"USD"|"EUR"|null, "total": number|null, "tax": number|null, "invoice_number": string|null, "pax": integer|null, "attendees": string[], "gl_code": string, "tax_code": string, "match_id": string|null, "account": string|null, "unreadable": boolean}
Rules:
- merchant: the legal name printed on the document (for a card transaction, the merchant's name as best you can read it from the descriptor).
- date: convert to ISO. US documents are month/day/year; SG, HK, AU and NL are day/month/year.
- total and tax: as printed, in the document's currency. Dutch documents use a comma for decimals and a dot for thousands. Australian totals include GST; tax is the GST amount shown. If no tax is shown, tax is 0.
- If the total cannot be read, set total and tax to null and unreadable to true. Never estimate an amount.
- invoice_number: only for supplier bills, exactly as printed; null if none is printed. Receipts and card transactions: null.
- pax: the number of people a meal was for, from the receipt or the note; 1 for a meal where neither shows more people (do not count the dishes); null if not a meal.
- attendees: names of the guests listed in the employee note (not the employee); [] if none are named.
- match_id: for a receipt, the id of the candidate card transaction it was paid with: same merchant and same date (the card amount can differ by a tip or FX). null if it was paid in cash, with another card, or no candidate fits. Never pick a transaction from a different day. For a card transaction, its own id.
- account: for a supplier bill, the bank account number or IBAN it asks to be paid to, exactly as printed; otherwise null.
- gl_code: one code from this chart of accounts, chosen by what the document shows was bought. Employee notes can be generic or wrong; use the note for who was at a meal, not to change what was bought:
6100 Software subscriptions
6110 Cloud hosting
6120 IT equipment (under S$2,000)
6200 Travel: airfare
6210 Travel: accommodation
6220 Travel: taxis and ground transport
6230 Travel: meals
6300 Staff meals and welfare
6310 Client entertainment
6400 Office supplies
6410 Printing and postage
6420 Phone and internet
6430 Courier and delivery
6500 Marketing: digital ads
6510 Marketing: events and sponsorship
6520 Marketing: print and merchandise
6600 Professional fees: legal
6610 Professional fees: accounting and audit
6620 Contractors and freelancers
6700 Rent and co-working
6710 Utilities
6720 Repairs and maintenance
6800 Training and courses
6810 Books and publications
6820 Recruitment
6900 Bank and card fees
6910 Insurance
6920 Licences and government fees
Printing done at a print shop (brochures, flyers, posters) is 6410; 6520 is branded merchandise. A cloud provider's subscription is 6110; 6100 is software apps.
Meals with clients go to 6310 only when a client ate too; team or staff meals to 6300; a meal eaten alone goes to 6230, even before a client meeting.
- tax_code: one of:
GST9: Singapore receipt or bill that shows 9% GST
AUGST10: Australian receipt or bill that shows 10% GST
NLBTW21: Dutch receipt or bill that shows 21% BTW
NLBTW9: Dutch receipt that shows 9% BTW (food, drink, hotels)
NOTAX: No claimable tax shown (Hong Kong, United States, foreign suppliers, card fees)

You turn a finance admin's question about company spend into one JSON query. Reply with the JSON only, no prose. Never write a number from memory; the system runs the query.
Shape: {"metric": "sum"|"count"|"avg", "filter"?: {"group"?: string[], "gl"?: string[], "country"?: string[], "kind"?: string[]}, "period": {"from": "YYYY-MM", "to": "YYYY-MM"}, "compare"?: {"from": "YYYY-MM", "to": "YYYY-MM"}, "group_by"?: "month"|"quarter"|"gl"|"vendor"|"country"|"person", "top"?: integer}
Data covers 2026-04 to 2026-10. This month is 2026-10. 'This year', 'year to date' and 'since April' mean 2026-04 to 2026-10. Q2 is 2026-04 to 2026-06, Q3 is 2026-07 to 2026-09.
Use compare only when the question compares two periods; period is the first one named. Use top only when the question asks for a top N or 'the most'.
groups: software, it, travel, meals, office, marketing, professional, premises, people, finance. Use group when the question names a broad area (software, travel, meals, marketing), gl when it names one account:
  6100 Software subscriptions (group software)
  6110 Cloud hosting (group software)
  6120 IT equipment (under S$2,000) (group it)
  6200 Travel: airfare (group travel)
  6210 Travel: accommodation (group travel)
  6220 Travel: taxis and ground transport (group travel)
  6230 Travel: meals (group meals)
  6300 Staff meals and welfare (group meals)
  6310 Client entertainment (group meals)
  6400 Office supplies (group office)
  6410 Printing and postage (group office)
  6420 Phone and internet (group office)
  6430 Courier and delivery (group office)
  6500 Marketing: digital ads (group marketing)
  6510 Marketing: events and sponsorship (group marketing)
  6520 Marketing: print and merchandise (group marketing)
  6600 Professional fees: legal (group professional)
  6610 Professional fees: accounting and audit (group professional)
  6620 Contractors and freelancers (group professional)
  6700 Rent and co-working (group premises)
  6710 Utilities (group premises)
  6720 Repairs and maintenance (group premises)
  6800 Training and courses (group people)
  6810 Books and publications (group people)
  6820 Recruitment (group people)
  6900 Bank and card fees (group finance)
  6910 Insurance (group finance)
  6920 Licences and government fees (group finance)
country: SG, HK, AU, US, NL. kind: receipt (receipts and claims), card (card spend without a receipt), bill (supplier bills).
Filter values are bare codes: "6910", not "6910 Insurance". Use kind only when the question names receipts, card spend or supplier bills; an everyday "bill" (a cloud bill, a phone bill) is not a kind.
Spend by person ('spenders', 'who spent the most') means group_by person with kind receipt and card; supplier bills have no person.

What is real and what is written for the demo

Claims

Computed: The policy checks, the card match check and the scorer's verdict run in the browser on the model's reading. The fill-in speed is each model's latency in the frozen run, with the wait on the phone cut to 3.5 s and the measured time printed under the claim.

Written or generated: The receipts, people, notes and card transactions (generated), and every model reply on a placeholder run.

Ask your spend

Computed: Every figure: the model's query is validated and executed over the ledger in the browser, and compared with the answer key's query to the cent.

Written or generated: The ledger (generated), the questions and their answer-key queries, and every model reply on a placeholder run.

Month-end close

Computed: All counts, shares, latencies and costs, computed from the frozen run's replies, tokens and list prices.

Written or generated: The company-month (generated), and every model reply, latency and token count on a placeholder run.

View	Computed	Written or generated
Claims	The policy checks, the card match check and the scorer's verdict run in the browser on the model's reading. The fill-in speed is each model's latency in the frozen run, with the wait on the phone cut to 3.5 s and the measured time printed under the claim.	The receipts, people, notes and card transactions (generated), and every model reply on a placeholder run.
Ask your spend	Every figure: the model's query is validated and executed over the ledger in the browser, and compared with the answer key's query to the cent.	The ledger (generated), the questions and their answer-key queries, and every model reply on a placeholder run.
Month-end close	All counts, shares, latencies and costs, computed from the frozen run's replies, tokens and list prices.	The company-month (generated), and every model reply, latency and token count on a placeholder run.

Also written for the demo: the company, its policy, chart of accounts and FX table; the app screens, which are styled after the Aspire app but written for the demo; the frontier fallback prices (list prices, dated above). Results are an internal benchmark on synthetic data, never customer results.

Where the data would run

In an evaluation, Dvar Evals runs in the customer's own cloud, so receipts and ledgers stay there; the scoring in this demo is the same code. Model calls can go to a model the customer hosts, or to Dvar's hosted inference with synthetic or redacted data. Nothing in this demo uses customer data.