What Developers Are Really Saying About AI Agent Evals and Harness Engineering

Something shifted at AI Native DevCon London 2026. The conversations on the expo floor weren't about whether AI would change software development. They were about how to make it work reliably at scale, and the gap between teams seeing real returns and those still searching for them. Questions about AI agent evals, harness engineering, and actual ROI came up, unprompted, in almost every exchange.

Simon Maple spent the two-day conference at The Brewery in central London talking with attendees, speakers, and exhibitors, capturing a cross-section of where practitioners genuinely are. What emerged is a picture of an industry moving from excitement to execution, with a few persistent blockers standing between intention and impact.

From Vanity Metrics to Meaningful AI Agent Evaluation

The sharpest recurring theme was measurement. Chris Baty, an engineer of 20 years who spoke at the conference about lessons learned building AI-first product teams inside a consultancy, argued that vanity metrics are the most dangerous habit teams can fall into. "Avoid vanity metrics at all costs, like code, commits or tokens," he observed. "Think about the actual outcomes you want your team and find the friction."

His point cuts to a real tension in how AI adoption gets reported internally. Token usage is easy to measure. Whether software is reaching real users, and whether those users are finding value, is harder. The framing, outcomes over output, reflects what a growing number of engineering leaders are arriving at: the bottleneck isn't code generation speed, it's the whole delivery pipeline.

This connects directly to the growing interest in AI agent evaluation, the practice of systematically testing whether an agent is producing correct, reliable, and useful outputs. Ryan, a tech lead attending the conference, cited evals as one of his biggest takeaways. "We've got skills spread all over the place and kind of want to consolidate them," he noted. "And look into the evals."

The two problems appear to be related. Without evaluation frameworks, it's difficult to know whether a skill is working correctly. Without consolidation, there's no clear place to run those evaluations from. Harness engineering, building the scaffolding around an agent including hooks, verifiers, and context controls, was something multiple attendees mentioned as a concept they were encountering for the first time. It wasn't in the CFP when the conference opened months ago; by the time the doors closed, it had become one of the defining topics of the event.

Real ROI Numbers and the MCP Servers Effect

Alongside the measurement conversation, the conference produced some concrete ROI figures worth examining. Cian O Maidin, president and founder of NearForm, shared two case studies from active client work.

On an AML project with a financial institution, the team used AI agents to process compliance cases that would previously have required 3,000 to 4,000 hours of human time. "I think we cleared like a total of AML cases in 20 minutes," he explained. On a legacy modernisation project, a codebase that would have cost over 10 million euros pre-AI is coming in at under 2 million, in roughly 35% of the time.

Those numbers suggest that when ROI appears, it tends to be dramatic rather than incremental. But they also highlight a pattern: the biggest gains tend to appear in specific, well-scoped tasks rather than as a general productivity lift across a whole team.

Martin, a developer at Train Guard attending with an AI automation focus, offered a useful framing for why that might be. "The force multiplier has been the tooling around the models," he noted. "Skills, MCP servers, they've really sort of delivered recency of information to the models that really make them much more capable." Model capability alone doesn't explain the gains. How context is delivered, how skills are structured, and how results are verified seems to be a significant variable, one that the MCP ecosystem is beginning to address in a systematic way.

Change Management and the Question of Who Ships

Two other themes from the floor are worth considering together. Manny Saka, a software engineer turned AI and ML practitioner with 25 years of experience, returned to the planning question that tends to get lost in discussions about agent speed. He drew on Einstein's observation about spending 55 minutes understanding a problem before spending five minutes solving it, and applied it directly to agentic workflows. The risk, he argued, isn't that developers are using AI too much, but that they're using it before they've properly understood what they're building.

At the other end of the spectrum, AutonomyAI CEO Adir Ben Yehuda made the case that the people doing the building might not always be developers at all. Their product, Fei Studio, aims to enable product managers and designers to raise pull requests directly on production code, by replacing CLI interfaces with a web interface and giving non-technical contributors guardrails rather than constraints.

When asked what one thing organisations aren't spending enough time on, Adir gave a one-word answer: "Change management." The tools exist to move faster. Whether organisations are restructuring their workflows, their teams, and their expectations to take advantage seems to be the harder, slower work.

That observation found an echo in a conversation with a representative from Snyk, who described the attendee population in terms of a spectrum: from teams that are cautious and curious, testing carefully, to those that are "YOLO-ing it." The interesting thing, they suggested, is that both positions contain genuine wisdom. The cautious teams are drawing on hard-won knowledge about what breaks at scale. The aggressive teams are discovering things that theory alone couldn't predict.

The conversations at DevCon London 2026 suggest an industry that has moved past the introductory phase. The questions now are operational: how to evaluate agents, how to structure tooling, how to make the case for genuine ROI, and how to bring an organisation along for the change. Worth a listen for any team in the middle of those questions.