Derek Ashmore

The AI Agent Testing Pyramid: A Hands-On Workshop

AI agents are entering production at an unprecedented scale without the automated testing practices that continuous delivery demands. The core obstacle: agents are non-deterministic, which breaks the expected output == actual output assumption that traditional test suites rely on. This workshop puts the AI Agent Testing Pyramid into practice. Working with a pre-built Research Summarizer Agent, attendees will use Claude Code to write tests at the first two levels of the pyramid: deterministic unit tests with mocked LLMs and constrained model tests at temperature=0 . I'll also provide a live demonstration of LLM-as-judge evaluation at Level 3. We'll cover the mechanics of mocking orchestration logic, designing assertions that tolerate model variance, and structuring a test suite that runs in CI without burning your API budget. By the end of the session, you'll have written working tests for a real AI agent and leave with a repeatable pattern you can apply to your own systems immediately. Prerequisites: - Claude Code and Python installed and functional, and workshop repo cloned Key Takeaways: - Hands-on experience writing deterministic unit tests and constrained model tests for a real agent using Claude Code - A mental model for mocking LLM calls that separates orchestration logic from model behavior - Practical cost and speed guidance so your test suite remains sustainable as your agent evolves - Critical pitfalls that break continuous delivery for AI systems, and how to avoid them