Jay Gordon

AI coding agents are becoming users of your APIs, documentation, tools, and services, and they don’t behave like human developers. Experiences that seem intuitive to people can send an agent down the wrong path, but traditional developer-experience testing won’t reveal why.

Unlike human usability studies, agent behavior can be reproduced at scale: you can run the same developer scenario in a sandbox and inspect every action, tool call, error, and outcome.

In this talk, we’ll introduce Project Scope, an open-source system for evaluating real developer scenarios across combinations of models, coding-agent harnesses, and extensions such as skills, MCP servers, and instruction files. Drawing on twelve months of experiments, we’ll share what we learned about designing services that agents can use successfully.

You’ll leave with a practical approach for measuring agent experience, diagnosing failures, and improving your APIs, documentation, and agent integrations.