Workflows / aspire-claims-receipt-reading · Calibration · gold v1 × rubric v1

Calibrate the judge

Locked
159 of 33 labelledevery topic coveredJudge: anthropic/claude-opus-5.5 Change

159 rows in gold v1 · 159 ready · 0 not ready

Keyboard shortcutsY yes · N no · 1 today · 2 alternative · E own · F flag · Enter save · J/K next or previous row · U undo

Only a member labels. You can read the rows and the result.

Row 1 of 140 of 159 answered

United States receipts · 14 labelled, 3 needed · picked automatically, the topic furthest behind · en

Your earlier answers aren't shown (someone labelled this row after you). Answer again and pick the reference to update them.

Same input for both

System prompt · 1 turn · no tools · max 4000 tokens · temperature 0

Open conversation

System You process expense items for the finance team of Kestrel & Finch, a Singapore company with staff in SG, HK, AU, US and NL. Read the item and reply with one JSON object only, no prose: {"merchant": string|null, "date": "YYYY-MM-DD"|null, "currency": "SGD"|"HKD"|"AUD"|"USD"|"EUR"|null, "total": number|null, "tax": number|null, "invoice_number": string|null, "pax": integer|null, "attendees": string[], "gl_code": string, "tax_code": string, "match_id": string|null, "account": string|null, "unreadable": boolean} Rules: - merchant: the legal name printed on the document (for a card transaction, the merchant's name as best you can read it from the descriptor). - date: convert to ISO. US documents are month/day/year; SG, HK, AU and NL are day/month/year. - total and tax: as printed, in the document's currency. Dutch documents use a comma for decimals and a dot for thousands. Australian totals include GST; tax is the GST amount shown. If no tax is shown, tax is 0. - If the total cannot be read, set total and tax to null and unreadable to true. Never estimate an amount. - invoice_number: only for supplier bills, exactly as printed; null if none is printed. Receipts and card transactions: null. - pax: the number of people a meal was for, from the receipt or the note; 1 for a meal where neither shows more people (do not count the dishes); null if not a meal. - attendees: names of the guests listed in the employee note (not the employee); [] if none are named. - match_id: for a receipt, the id of the candidate card transaction it was paid with: same merchant and same date (the card amount can differ by a tip or FX). null if it was paid in cash, with another card, or no candidate fits. Never pick a transaction from a different day. For a card transaction, its own id. - account: for a supplier bill, the bank account number or IBAN it asks to be paid to, exactly as printed; otherwise null. - gl_code: one code from this chart of accounts, chosen by what the document shows was bought. Employee notes can be generic or wrong; use the note for who was at a meal, not to change what was bought: 6100 Software subscriptions 6110 Cloud hosting 6120 IT equipment (under S$2,000) 6200 Travel: airfare 6210 Travel: accommodation 6220 Travel: taxis and ground transport 6230 Travel: meals 6300 Staff meals and welfare 6310 Client entertainment 6400 Office supplies 6410 Printing and postage 6420 Phone and internet 6430 Courier and delivery 6500 Marketing: digital ads 6510 Marketing: events and sponsorship 6520 Marketing: print and merchandise 6600 Professional fees: legal 6610 Professional fees: accounting and audit 6620 Contractors and freelancers 6700 Rent and co-working 6710 Utilities 6720 Repairs and maintenance 6800 Training and courses 6810 Books and publications 6820 Recruitment 6900 Bank and card fees 6910 Insurance 6920 Licences and government fees Printing done at a print shop (brochures, flyers, posters) is 6410; 6520 is branded merchandise. A cloud provider's subscription is 6110; 6100 is software apps. Meals with clients go to 6310 only when a client ate too; team or staff meals to 6300; a meal eaten alone goes to 6230, even before a client meeting. - tax_code: one of: GST9: Singapore receipt or bill that shows 9% GST AUGST10: Australian receipt or bill that shows 10% GST NLBTW21: Dutch receipt or bill that shows 21% BTW NLBTW9: Dutch receipt that shows 9% BTW (food, drink, hotels) NOTAX: No claimable tax shown (Hong Kong, United States, foreign suppliers, card fees)

Customer

KIND: receipt EMPLOYEE: <PERSON_1> (US), corporate card ****6952 EMPLOYEE NOTE: Printed brochures for the event RECEIPT (OCR text) LOOMWORK SOFTWARE INC. 278 Hill St #07-12 New York Date: 10/02/2026 14:03 ------------------------------ Monthly subscription 270.17 ------------------------- TOTAL USD 270.17 Paid: CASH Thank you! CANDIDATE CARD TRANSACTIONS T0139 2026-10-06 ADREACH*ADS US USD 1427.38 T0140 2026-10-01 NIMBUSGRID CLOUD US USD 218.22 T0141 2026-10-01 ADREACH*ADS US USD 797.29

Todayanthropic/claude-sonnet-5.528 words1

{"merchant": "LOOMWORK SOFTWARE INC.", "date": "2026-10-02", "currency": "USD", "total": 270.17, "tax": 0, "invoice_number": null, "pax": null, "attendees": [], "gl_code": "6100", "tax_code": "NOTAX", "match_id": null, "account": null, "unreadable": false}

Alternative—0 words2

None (empty or expired)

Check 1 of 7

fields_read_correctly counts ×2

Merchant, date, currency, total, tax, people and guests are read as printed.

Reference

Judge's verdict

Answer every check first. The judge's verdicts show after, so they can't bias you.

Judge check · gold v1 with rubric v1

Judge: anthropic/claude-opus-5.5 · your key

Judge check. One row per check, with how far the judge agrees with you.
CheckAgreement (κ)How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted.LabelledTrust
fields_read_correctly—159 · 159 Y, 0 NNeeds 5 more labels
gl_and_tax_code_correct0.00159 · 150 Y, 9 NNot trusted
card_match_correct—159 · 159 Y, 0 NNeeds 5 more labels
decision_correct—159 · 159 Y, 0 NNeeds 5 more labels
no_wrong_card_match—159 · 159 Y, 0 NNeeds 5 more labels
no_wrong_amount—159 · 159 Y, 0 NNeeds 5 more labels
no_duplicate_cleared—159 · 159 Y, 0 NNeeds 5 more labels

A check is trusted at κ 0.8How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted. or more on 30 or more labelled rows, with at least 5 Yes and 5 No. A fifth of the rows is held out of every rubric fix; κ more than 0.1 lower there marks the check possibly overfit.

Locked: gold v1 with rubric v1 by ops.member · 20 Oct · 159 of 159 rows verified

Instructions for Dvar (optional)

None saved.

Rubric v17 checks by ops.member · 20 Oct
  • fields_read_correctly

    Merchant, date, currency, total, tax, people and guests are read as printed.

    ×2
  • gl_and_tax_code_correct

    The GL code and tax code match the answer key.

  • card_match_correct

    The claim is matched to the right card line, or to none.

  • decision_correct

    The policy checks on this reading reach the same close-or-review decision as on the answer key.

  • no_wrong_card_match

    Never matches a receipt to a card line it was not paid with.

    Required
  • no_wrong_amount

    Never guesses an unreadable amount, and never misreads the total by more than 1%.

    Required
  • no_duplicate_cleared

    Never lets a receipt that was already claimed close with no review.

    Required
Version historyv1 approved by risk.member · 20 Oct
Rubric v1by ops.member · approved by risk.member · 20 Oct
  • fields_read_correctly

    Merchant, date, currency, total, tax, people and guests are read as printed.

    ×2
  • gl_and_tax_code_correct

    The GL code and tax code match the answer key.

  • card_match_correct

    The claim is matched to the right card line, or to none.

  • decision_correct

    The policy checks on this reading reach the same close-or-review decision as on the answer key.

  • no_wrong_card_match

    Never matches a receipt to a card line it was not paid with.

    Required
  • no_wrong_amount

    Never guesses an unreadable amount, and never misreads the total by more than 1%.

    Required
  • no_duplicate_cleared

    Never lets a receipt that was already claimed close with no review.

    Required
——