Joe Tyler

A new coding model tops SWE-Bench on Monday. Should you roll it out to your team on Tuesday? A new tool or MCP server is released. Should you use it straight away? Public benchmarks can’t answer these questions, because none of them were built for your code.

Without knowing how the models were trained, it’s difficult to know whether labs have “benchmaxxed” on popular open benchmarks like SWE-Bench or Terminal-bench. Even more recent benchmarks like DeepSWE which is written by hand to avoid contamination, or FrontierCode which includes a measure of production code quality, are not built for the repositories or workflows you’ll be using AI coding tools for in your day-to-day.

In short: these benchmarks are all relevant, but none of them have been tailored for your use cases, and therefore none can be trusted to give a realistic prediction of how a coding model or agent will perform in your domain.

At Sonar, as the leader in independent verification of AI-generated and developer-written code, our research team has invested significant time and resources into understanding the qualities and defects of code generated by AI coding tools and models. This is done to answer questions the community keeps asking: “Which coding model writes code that is easiest to maintain?” or “Where should I focus my code review when using this model?”. We use these tools ourselves, so we built our own custom benchmark, which allows us to independently verify the performance of each new model, agent, tool or configuration in workflows that reflect our developers’ AI coding tool usage. We do this to compare these options and recommend tooling, but also as an evaluation tool for our custom models to hill climb on through post-training.

In this talk, Joe Tyler, Lead AI Researcher at Sonar, will look at the important aspects when designing evaluations to be indicative of production quality code, before walking through how we built a custom benchmark, and what we learnt when comparing leading open and closed source coding agents.