Agent Autopsy

01

The failure.

A seat-count dispute. The agent looks up the account, the invoice, and then searches the knowledge base six times with slightly different queries. search_kb has no top-k, so each search returns every article that matches at all, full text, and the loop appends all of it to the context. By the eleventh model call the request is 12,800 tokens; the ticket costs 22 times the median. No single step looks wrong, and the step limit never fires, because there are only twelve of them.

what to watch for The tokens-in column doubles, then doubles again. Watch the cost of each individual model call, not just the total.

02

What it cost.

The two runs side by side. Same ticket, same tools, same prices; one safeguard apart.

cost, failure

$0.0834

cost, fix

$0.0093

tokens in

80,857 → 7,782

model calls

12 → 6

The failure cost 8.9 times the fixed run for the same ticket.

03

The fix: idempotency key, and a tool that returns what was asked.

search_kb goes back to its two best articles. Every tool call is keyed on the tool name and its normalised arguments; a repeat is answered from the earlier result as a pointer, so the model can still see it, the context does not grow by it, and a flaky tool is never retried with a side effect. A per-task cost ceiling belongs behind this as the backstop: this run finished at $0.009, and a ceiling of $0.02 would have stopped the original at step 8.

def cached(run):
    def call(args):
        key = canonical(name, args)
        if key in cache:
            return {"cached": True, "same_as_call": cache[key]}
        cache[key] = call_number
        return run(args)
    return call

Before: search_kb with no top-k: every article that matches at all comes back with its full body, and the loop appends all of it to the context on every step; MAX_MODEL_CALLS at 20, so the step limit never fires. After: search_kb back to its two best articles, plus an idempotency key on every tool call: a repeat of the same call returns a pointer to the earlier result instead of the result again, so the context stops growing by it.

04

The same ticket, with the fix in.

The identical scripted model, the identical tools; only the safeguard differs.

05

How the eval would have caught this.

The response and hallucination cases pass before and after: the reply was right both times, which is the point of this autopsy. The trajectory case fails both times because the model asked for search_kb more than once; after the fix that repeat cost nothing. The eval that catches a cost blow-up is the cost line in results/latest.json, compared against the per-task budget in the contract.

Gate: demo-bench-starter ci.yml, job 'api quality and evals': uv run python -m evals.run (exits 1 under any threshold)

• evals/run.py: cost.per_task_usd
• results/latest.json: cost