Agent Autopsy

01

The failure.

When will my refund arrive? The agent searches the knowledge base and the top hit is the refund article: same id, same title, the 2025 text that says 3 working days. The current article says 5. Nothing on either document says which is in force, so the model does what retrieval told it to and the customer is told the wrong date with full confidence. This is Bench Starter's own recorded miss on T-006, taken apart.

what to watch for The tool result at step 3 says 3 working days and carries no date. The reply at step 4 repeats it.

02

What it cost.

The two runs side by side. Same ticket, same tools, same prices; one safeguard apart.

cost, failure

$0.0038

cost, fix

$0.0039

tokens in

2,910 → 2,980

model calls

3 → 3

The failure cost about the same as the fix. Price was not the signal here; the reply was.

03

The fix: freshness metadata.

Every document in the index carries effective_from and, when replaced, superseded_by. Retrieval filters superseded documents out before ranking, returns the effective date with each hit, and the system prompt asks the model to cite it. The eval catches the symptom either way: the response case expects the phrase the current policy uses.

def search_kb(args):
    hits = rank(query, [a for a in articles if a.get("superseded_by") is None])
    return [{**a, "effective_from": a["effective_from"]} for a in hits[:2]]

Before: search_kb over an index that still holds the 2025 refund article (3 working days) under the same id and title as the current one (5 working days); nothing on the document says which is in force. After: every article carries effective_from, superseded versions are dropped from the index, and the reply cites the date it relied on.

04

The same ticket, with the fix in.

The identical scripted model, the identical tools; only the safeguard differs.

05

How the eval would have caught this.

The response case for T-006 expects "5 working days" and fails on the stale reply, which is exactly the miss Bench Starter keeps on purpose in its committed run. The hallucination case passes both times: the stale number was in a tool result. The trajectory is the same before and after.

Gate: demo-bench-starter ci.yml, job 'api quality and evals': uv run python -m evals.run (exits 1 under any threshold)

• evalset.json: T-006-response
• results/latest.json: the kept miss