The problem
Agents rarely fail because the model says something wrong. They fail in the system around it. A refund call times out after the money already moved, the agent retries, and the customer gets refunded twice. A tool errors and the agent says "Done!" anyway. A run crashes halfway and redoes steps on restart.
Most agent demos only test the happy path, with tools that always respond correctly. Few test what happens when a tool fails, which is when an agent costs real money.
What I built
A runner that drops an agent into a fake world (payments, email, helpdesk, inventory and so on) and breaks it on cue: timeouts, lost replies, rate limits, stale data, a crash right after step three.
Every real effect lands in a ledger. Plain functions read the ledger and the agent's final answer and decide pass or fail, with the evidence attached. You get a JSON file, an HTML report and a CI exit code.
System design
Break the world. Judge the side effects
- ScenarioYAML · task, faults, checks
- RunAgent vs simulated services
- LedgerEvery effect, applied or deduped
- VerdictsOracles + validity → report
Explore the full architecture
25 scenarios across six failure categories. Reference agents, framework adapters, or any agent over MCP or HTTP.
Design notes
An open benchmark and fault-injection runner for AI agents. A YAML scenario seeds deterministic simulated services, the runner fires faults at chosen calls, and every real effect lands in a ledger that oracles turn into verdicts.
Judge effects, not transcripts
Verdicts come from the side-effect ledger and the agent’s final claim, so a retry that charges twice fails even when the transcript reads fine.
Deterministic by design
Simulated services honour real semantics such as idempotency keys, and a scripted model replaces the LLM in framework mode, so every run reproduces exactly with no API key.
A fault that never fired is not a test
A run only counts if the faults and crash it declares actually happened. Otherwise it is reported invalid, never as a pass.
Calls I'd make again
- No model grading another model. Verdicts come from the ledger, so they are reproducible.
- Deterministic by default. A scripted model and a seeded clock mean anyone can rerun a result for free and get the same bytes.
- A run only counts if its fault actually fired. Some agents quit before the fault, and I didn't want that scored as a pass.
- The fake services follow real rules like idempotency keys, so doing the right thing actually gets rewarded.
The hard parts
- Faking "it worked, but the reply got lost", then proving the duplicate from the ledger rather than from logs.
- Killing an agent after an exact step and checking it resumes without redoing finished work.
- Keeping framework adapters honest. They add no retries or checks of their own, so results show each framework's real defaults.
What's still rough
Deterministic mode tests the harness, not the model's judgement, and the report says so. Agents connected over MCP can't be crashed and restarted yet. Next up is that restart protocol and a comparison with durable runtimes like Temporal.
