Like the work? Buy me a coffee
Selected projects

Case study · Open source

AgentPostmortem

An open benchmark that tests how AI agents handle failing tools.

An open benchmark that breaks an AI agent's tools on purpose, safely, and checks what the agent actually did, not what it said it did.

Role
Solo, design to release
Stack
Python, Pydantic, YAML, pytest
Works with
LangGraph, OpenAI Agents SDK, Claude Agent SDK, anything over MCP or HTTP
Ships as
agentpostmortem-bench on PyPI, Apache-2.0

The problem

Agents rarely fail because the model says something wrong. They fail in the system around it. A refund call times out after the money already moved, the agent retries, and the customer gets refunded twice. A tool errors and the agent says "Done!" anyway. A run crashes halfway and redoes steps on restart.

Most agent demos only test the happy path, with tools that always respond correctly. Few test what happens when a tool fails, which is when an agent costs real money.

What I built

A runner that drops an agent into a fake world (payments, email, helpdesk, inventory and so on) and breaks it on cue: timeouts, lost replies, rate limits, stale data, a crash right after step three.

Every real effect lands in a ledger. Plain functions read the ledger and the agent's final answer and decide pass or fail, with the evidence attached. You get a JSON file, an HTML report and a CI exit code.

System design

Break the world. Judge the side effects

Read how it works
Walk through the system
AgentPostmortem architectureA YAML scenario declares the task, the faults and the checks. The runner seeds deterministic simulated services and fires each fault at its chosen call while the agent under test works through its tools. Every real effect is written to a side-effect ledger. Oracles judge the ledger and the agent's final claim, a validity check confirms the declared faults actually fired, and the results land in results.json and report.html, which can gate CI.
  1. ScenarioYAML · task, faults, checks
  2. RunAgent vs simulated services
  3. LedgerEvery effect, applied or deduped
  4. VerdictsOracles + validity → report
Explore the full architecture
AgentPostmortem architectureA YAML scenario declares the task, the faults and the checks. The runner seeds deterministic simulated services and fires each fault at its chosen call while the agent under test works through its tools. Every real effect is written to a side-effect ledger. Oracles judge the ledger and the agent's final claim, a validity check confirms the declared faults actually fired, and the results land in results.json and report.html, which can gate CI.

25 scenarios across six failure categories. Reference agents, framework adapters, or any agent over MCP or HTTP.

Design notes

An open benchmark and fault-injection runner for AI agents. A YAML scenario seeds deterministic simulated services, the runner fires faults at chosen calls, and every real effect lands in a ledger that oracles turn into verdicts.

Judge effects, not transcripts

Verdicts come from the side-effect ledger and the agent’s final claim, so a retry that charges twice fails even when the transcript reads fine.

Deterministic by design

Simulated services honour real semantics such as idempotency keys, and a scripted model replaces the LLM in framework mode, so every run reproduces exactly with no API key.

A fault that never fired is not a test

A run only counts if the faults and crash it declares actually happened. Otherwise it is reported invalid, never as a pass.

Calls I'd make again

  • No model grading another model. Verdicts come from the ledger, so they are reproducible.
  • Deterministic by default. A scripted model and a seeded clock mean anyone can rerun a result for free and get the same bytes.
  • A run only counts if its fault actually fired. Some agents quit before the fault, and I didn't want that scored as a pass.
  • The fake services follow real rules like idempotency keys, so doing the right thing actually gets rewarded.

The hard parts

  • Faking "it worked, but the reply got lost", then proving the duplicate from the ledger rather than from logs.
  • Killing an agent after an exact step and checking it resumes without redoing finished work.
  • Keeping framework adapters honest. They add no retries or checks of their own, so results show each framework's real defaults.

What's still rough

Deterministic mode tests the harness, not the model's judgement, and the report says so. Agents connected over MCP can't be crashed and restarted yet. Next up is that restart protocol and a comparison with durable runtimes like Temporal.

Next case studyBoardEject