Agent Autopsy

01

The failure.

A customer asks when a refund will arrive. The model calls search_kb("refund timing"), gets the refund policy, and calls it again. And again. Each call re-sends the whole transcript, so the context grows by about 250 tokens a step and the price of each call climbs with it. Nothing is wrong with the tool; the model has decided the answer is one more search away.

what to watch for Forty identical tool calls. The tokens-in column rises on every model step while the output never changes.

02

What it cost.

The two runs side by side. Same ticket, same tools, same prices; one safeguard apart.

cost, failure

$0.2360

cost, fix

$0.0053

tokens in

230,430 → 4,365

model calls

40 → 4

The failure never produced a reply: stopped at the model-call limit.

03

The fix: max-steps hook and an idempotency key.

Two rules, both deterministic. MAX_MODEL_CALLS goes back to 8: enough for the longest legitimate task in the eval set, low enough to fire in testing. In front of every tool call, a repeated_call rule keys the call on the tool name and its normalised arguments; the second call with the same key is refused with a pointer to the step that already holds the result. The model is told, not silently short-circuited, so the trace shows the decision.

MAX_MODEL_CALLS = 8

def check_tool_call(ticket_id, step, name, args):
    key = canonical(name, args)          # tool name + sorted, normalised args
    if key in seen:
        return blocked("repeated_call",
            f"{name} was already called with these arguments at step {seen[key]}")
    seen[key] = step
    return real_check_tool_call(ticket_id, step, name, args)

Before: guardrails.MAX_MODEL_CALLS raised from 8 to 40, the kind of limit that is set high enough to never fire in testing; the model repeats search_kb('refund timing') until it does. After: MAX_MODEL_CALLS back at 8, plus a repeated_call rule in front of check_tool_call: the same tool with the same arguments (idempotency key) is refused the second time and the model is pointed at the step that already has the result.

04

The same ticket, with the fix in.

The identical scripted model, the identical tools; only the safeguard differs.

05

How the eval would have caught this.

Two of the three Bench Starter cases for T-006 fail on the loop: there is no final answer, and the agent did not finish. After the fix both pass. The trajectory case still fails, and should: the agent asked for search_kb twice, and the eval scores the decision, not the damage. The guard contained it; the eval reports it.

Gate: demo-bench-starter ci.yml, job 'api quality and evals': uv run python -m evals.run (exits 1 under any threshold)

• evals/run.py: score_safety
• guardrails.py: MAX_MODEL_CALLS