AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
86%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Joe Tyler · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=aJGdUhUmerg
Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.
For topic navigation, see outline.md.
Cue 1 — 00:00:00,000 --> 00:00:02,721
Yeah. So thank you everyone for joining.
Cue 2 — 00:00:02,761 --> 00:00:07,921
So as Simon said, I'm Joe Tyler and I'm a researcher at Sonar.
Cue 3 — 00:00:07,961 --> 00:00:11,001
And I'm going to be talking about some of the work we've been doing in our research
Cue 4 — 00:00:11,041 --> 00:00:15,101
team to evaluate coding agents in realistic settings.
Cue 5 — 00:00:15,841 --> 00:00:20,421
And one of the applications of this within our team is to post train coding agents
Cue 6 — 00:00:20,841 --> 00:00:22,781
to be to be better for our team.
Cue 7 — 00:00:23,401 --> 00:00:27,761
Okay. So in this talk, I'm going to cover why we benchmark why an agent is more than
Cue 8 — 00:00:27,801 --> 00:00:32,441
just a model. What we what we learn from public benchmarks, but then also where
Cue 9 — 00:00:32,481 --> 00:00:36,721
public benchmarks fail using some popular benchmarks that you might have heard of as
Cue 10 — 00:00:36,761 --> 00:00:42,401
examples. And then using those failures, we will see some design requirements on
Cue 11 — 00:00:42,441 --> 00:00:44,901
what will make a better benchmark for you.
Cue 12 — 00:00:45,241 --> 00:00:50,421
I'll run through how we did that for ourselves and then go through some findings
Cue 13 — 00:00:50,721 --> 00:00:56,441
that we that we saw when we compared a lot of open and closed source models and
Cue 14 — 00:00:56,481 --> 00:01:00,941
harnesses on these benchmarks, and we analyzed the trajectories.
Cue 15 — 00:01:02,081 --> 00:01:05,681
Okay, I'll get started. So section one, what do benchmarks tell us.
Cue 16 — 00:01:05,721 --> 00:01:11,261
Why do we benchmark and how do scores compare with what users experience?
Cue 17 — 00:01:12,841 --> 00:01:16,361
So benchmarks have long been a part of model releases.
Cue 18 — 00:01:16,401 --> 00:01:19,261
They give empirical evidence of.
Cue 19 — 00:01:21,001 --> 00:01:25,821
A models ability on any task that requires reasoning or understanding.
Cue 20 — 00:01:26,961 --> 00:01:30,341
We know that we can't we can't just read capabilities of models off their spec
Cue 21 — 00:01:30,381 --> 00:01:35,001
sheet. So models, even if we know the exact architecture, the exact number of
Cue 22 — 00:01:35,041 --> 00:01:37,801
parameters of a model and even what those parameters actually are,
Cue 23 — 00:01:37,841 --> 00:01:44,401
there's no way that we can actually understand what what the model's abilities will
Cue 24 — 00:01:44,441 --> 00:01:46,901
be. Sorry.
Cue 25 — 00:01:48,281 --> 00:01:52,521
No worries. And yeah, and as I said, agents are more than just its models.
Cue 26 — 00:01:52,561 --> 00:01:57,841
So even if you see benchmark scores in a model release, how you use that,
Cue 27 — 00:01:57,881 --> 00:02:01,081
that model in terms of which harness you use, what configuration you use,
Cue 28 — 00:02:01,121 --> 00:02:03,021
that's going to change your experience a lot.
Cue 29 — 00:02:03,081 --> 00:02:07,281
And finally, scores aren't ever going to match your true experience of using those
Cue 30 — 00:02:07,321 --> 00:02:11,761
agents. So I'm sure like everyone has has felt that when they've seen a new model
Cue 31 — 00:02:11,801 --> 00:02:15,001
come out that that beats the frontier.
Cue 32 — 00:02:15,041 --> 00:02:19,041
And when you actually use that model, you don't really feel any tangible difference.
Cue 33 — 00:02:19,081 --> 00:02:24,001
Whereas sometimes, like I can think of Opus 4.5 back in November,
Cue 34 — 00:02:24,041 --> 00:02:27,801
December last year, that model didn't jump on the benchmarks that much,
Cue 35 — 00:02:27,841 --> 00:02:31,601
but it really did feel like it was a lot more capable than than models that had come
Cue 36 — 00:02:31,641 --> 00:02:32,541
before it.
Cue 37 — 00:02:33,801 --> 00:02:38,601
And this, this, this difference between how they feel and what they score in the
Cue 38 — 00:02:38,641 --> 00:02:41,201
benchmark isn't, isn't a contradiction.
Cue 39 — 00:02:41,241 --> 00:02:45,121
So each benchmark is measuring specific things, and those just might not be the
Cue 40 — 00:02:45,161 --> 00:02:49,361
things that you need to measure to understand how it's going to feel for you when
Cue 41 — 00:02:49,401 --> 00:02:54,581
you actually use the model. And this leads to the questions that we get asked a lot.
Cue 42 — 00:02:54,921 --> 00:02:58,481
So they want to know. Teams want to know should we use a new model when it comes
Cue 43 — 00:02:58,521 --> 00:03:01,461
out. Should we try out a new harness.
Cue 44 — 00:03:02,401 --> 00:03:06,161
And also which of our workflows in our software factory actually need a frontier
Cue 45 — 00:03:06,201 --> 00:03:10,481
model? Or could a smaller model with a different harness do just as good a job or
Cue 46 — 00:03:10,521 --> 00:03:11,541
even better?
Cue 47 — 00:03:12,921 --> 00:03:15,601
And we want to ask this because I come from Sonar.
Cue 48 — 00:03:15,641 --> 00:03:19,501
So we're a global leader in AI code verification and governance.
Cue 49 — 00:03:19,601 --> 00:03:24,921
We're trusted by 75% of the Fortune 100 companies to bring confidence to take
Cue 50 — 00:03:24,961 --> 00:03:28,521
software factories. And personally in our in our.
Cue 51 — 00:03:28,561 --> 00:03:33,241
I work in our research team, and we are very interested in the qualities and defects
Cue 52 — 00:03:33,281 --> 00:03:38,481
of AI coding tools, including models, agents, harnesses.
Cue 53 — 00:03:38,521 --> 00:03:43,141
And we want to know how they how they perform in real developer contexts.
Cue 54 — 00:03:43,441 --> 00:03:47,181
And then we're also interested in how we can how we can improve them.
Cue 55 — 00:03:48,121 --> 00:03:48,921
So,
Cue 56 — 00:03:50,321 --> 00:03:54,601
I've maybe hinted at a few failings of benchmarks so far, but now I'm going to dive
Cue 57 — 00:03:54,641 --> 00:03:59,181
more into why they fail, even from a research perspective.
Cue 58 — 00:03:59,761 --> 00:04:03,101
So the first failure that I'm going to discuss is data leakage.
Cue 59 — 00:04:03,201 --> 00:04:07,781
So benchmarks tend to be built from open source repositories.
Cue 60 — 00:04:07,921 --> 00:04:11,741
And they're a great resource to find realistic engineering.
Cue 61 — 00:04:12,761 --> 00:04:14,941
Examples of realistic engineering.
Cue 62 — 00:04:15,281 --> 00:04:20,921
But the problem is that these same open source repositories probably appear in the
Cue 63 — 00:04:20,961 --> 00:04:24,581
data sets that have been used to train the models in the first place.
Cue 64 — 00:04:24,641 --> 00:04:30,401
This means that models have an unfair advantage, and you you can't tell if a model
Cue 65 — 00:04:30,441 --> 00:04:34,441
is actually reasoning for a problem, or it's just remembering something that's seen
Cue 66 — 00:04:34,481 --> 00:04:40,601
in its pre-training data. One example of this, notably a famous example,
Cue 67 — 00:04:40,641 --> 00:04:45,181
is SWE-bench, which is which was a very popular benchmark introduced by OpenAI.
Cue 68 — 00:04:46,081 --> 00:04:52,721
However, when researchers I can't remember from which
Cue 69 — 00:04:52,761 --> 00:04:58,281
institution they're from, but when researchers looked at providing just issue
Cue 70 — 00:04:58,321 --> 00:05:05,121
messages to the models, they found the best model, when
Cue 71 — 00:05:05,161 --> 00:05:10,201
given an issue from SWE-bench could name the buggy file 76% of the time,
Cue 72 — 00:05:10,241 --> 00:05:13,841
but when it was a series of other repositories, still public repositories.
Cue 73 — 00:05:13,881 --> 00:05:20,761
But, when it weren't within SWE-bench, there was only only 53% of the
Cue 74 — 00:05:20,801 --> 00:05:23,221
time. Could they name the buggy file?
Cue 75 — 00:05:23,761 --> 00:05:29,661
So one fix for this came with a benchmark from Datacurve called Deep SWE,
Cue 76 — 00:05:30,041 --> 00:05:35,401
where they took open source projects and they wrote a real feature,
Cue 77 — 00:05:35,441 --> 00:05:38,001
real new features, but they never merge them.
Cue 78 — 00:05:38,041 --> 00:05:39,621
So the
Cue 79 — 00:05:41,161 --> 00:05:46,841
the changes never got put into the history, which gave a really realistic assessment
Cue 80 — 00:05:46,881 --> 00:05:48,481
of the models at the time it came out.
Cue 81 — 00:05:48,521 --> 00:05:50,801
However, now these these tests are now public.
Cue 82 — 00:05:50,841 --> 00:05:54,461
So again, there's sort of a shelf life to these public benchmarks.
Cue 83 — 00:05:55,241 --> 00:06:00,681
And this shows the importance to evaluate models and agents on code that they that
Cue 84 — 00:06:00,721 --> 00:06:05,261
they haven't seen by using private repositories for example.
Cue 85 — 00:06:06,241 --> 00:06:11,641
So, failure two: passing the tests is not the same as being ready to merge.
Cue 86 — 00:06:11,681 --> 00:06:18,361
So pretty much every software engineering benchmark until recently was just built on
Cue 87 — 00:06:18,401 --> 00:06:23,881
unit tests and can check certain things about the code, but they might not check
Cue 88 — 00:06:23,921 --> 00:06:27,821
that the the change fully matches the intent of the prompt.
Cue 89 — 00:06:28,281 --> 00:06:33,881
They might not check that the code is passing all the quality requirements that you
Cue 90 — 00:06:33,921 --> 00:06:37,761
need in order to merge it. So there was METR.
Cue 91 — 00:06:37,801 --> 00:06:44,661
They did a study where they asked a lot of open source maintainers to look at,
Cue 92 — 00:06:46,161 --> 00:06:50,681
pull requests that had been created for SWE-bench on a load of the leading models at
Cue 93 — 00:06:50,721 --> 00:06:53,601
the time. And those were the red red dots there.
Cue 94 — 00:06:53,641 --> 00:07:00,521
And those were the merge rates from the, from the models when scored on SWE-bench.
Cue 95 — 00:07:00,561 --> 00:07:07,001
But then actually the maintainers would only merge them about 24% less often on on
Cue 96 — 00:07:07,041 --> 00:07:11,241
average. There's a gap here between just going off the unit tests and actually a
Cue 97 — 00:07:11,281 --> 00:07:13,941
more holistic assessment of the code.
Cue 98 — 00:07:14,081 --> 00:07:18,601
And there's a couple of, more recent benchmarks that tried to target this.
Cue 99 — 00:07:18,641 --> 00:07:25,601
So frontier code use a load of different rubrics for code quality and and intent
Cue 100 — 00:07:25,641 --> 00:07:31,441
of the code. And they even though you know the best one of the best models here is
Cue 101 — 00:07:31,481 --> 00:07:35,041
over 60% on SWE-bench. But when you put them through frontier code,
Cue 102 — 00:07:35,081 --> 00:07:38,301
even the best model at the time, which I think was Claude Opus five,
Cue 103 — 00:07:38,401 --> 00:07:44,881
only scored 13.4%. We also have a benchmark publicly that
Cue 104 — 00:07:44,921 --> 00:07:51,721
addresses this issue. So unit tests alone don't tell you whether the code is ready
Cue 105 — 00:07:51,761 --> 00:07:57,041
to merge. So our benchmark also looks for Java and Python problems.
Cue 106 — 00:07:57,081 --> 00:08:00,921
We look at unit tests, but we also use SonarQube to analyze the code,
Cue 107 — 00:08:00,961 --> 00:08:06,681
see how many bugs vulnerabilities, bad maintainability practices spill into the
Cue 108 — 00:08:06,721 --> 00:08:09,481
code. We also look at how complex the code is.
Cue 109 — 00:08:09,521 --> 00:08:14,181
And all of this is to answer how long is your code going to spend?
Cue 110 — 00:08:15,081 --> 00:08:18,441
In the review process. So that's that's become the bottleneck,
Cue 111 — 00:08:18,481 --> 00:08:20,521
I think, with agentic coding these days.
Cue 112 — 00:08:20,561 --> 00:08:25,541
And this, this benchmark should give you a more realistic assessment of how,
Cue 113 — 00:08:25,601 --> 00:08:27,641
how complex that code is going to be reviewed.
Cue 114 — 00:08:27,681 --> 00:08:31,901
So this is available online. So leaderboard.
Cue 115 — 00:08:32,401 --> 00:08:33,801
And you can check check this out.
Cue 116 — 00:08:33,841 --> 00:08:36,481
But this is still this is still a public benchmark.
Cue 117 — 00:08:36,521 --> 00:08:39,701
So it's still not it's still not what we're looking for.
Cue 118 — 00:08:40,801 --> 00:08:41,841
And this leads up to this point.
Cue 119 — 00:08:41,881 --> 00:08:46,061
So no public benchmark was built from your code or your workflows.
Cue 120 — 00:08:46,281 --> 00:08:49,021
So this failure is a lack of context.
Cue 121 — 00:08:50,041 --> 00:08:56,561
Teams will give agents their code bases, their conventions and their pull
Cue 122 — 00:08:56,601 --> 00:09:02,941
requests, but the public benchmarks will never, will never measure these things.
Cue 123 — 00:09:03,081 --> 00:09:06,814
Benchmarks all designed to answer very specific questions like SWE-bench with the
Cue 124 — 00:09:06,854 --> 00:09:09,461
unit tests. Terminal-Bench, which is for.
Cue 125 — 00:09:11,281 --> 00:09:12,961
Agents working in the command line.
Cue 126 — 00:09:13,001 --> 00:09:16,921
Again, that's all based on unit tests, frontier code and our leaderboard address
Cue 127 — 00:09:16,961 --> 00:09:18,921
more factors about merging the code.
Cue 128 — 00:09:18,961 --> 00:09:25,401
But still they don't have context of really how your how your developers are working
Cue 129 — 00:09:25,441 --> 00:09:27,821
or how your software factory is working.
Cue 130 — 00:09:27,921 --> 00:09:31,861
And therefore you need context of of your own workflows to measure.
Cue 131 — 00:09:34,001 --> 00:09:38,801
This is and the study I'm showing here, this is from Scale AI with their benchmark
Cue 132 — 00:09:38,841 --> 00:09:43,041
called SWE-Bench Pro. So they built their own version of SWE-bench.
Cue 133 — 00:09:43,081 --> 00:09:45,261
And they had a load of the
Cue 134 — 00:09:46,601 --> 00:09:50,321
examples which were on public code bases and a load that were on private.
Cue 135 — 00:09:50,361 --> 00:09:54,801
And you can see that either because the private benchmarks haven't appeared in the
Cue 136 — 00:09:54,841 --> 00:09:59,601
training sets, or the fact that the private benchmarks might be more niche,
Cue 137 — 00:09:59,641 --> 00:10:02,201
more specialized than what you see in public datasets.
Cue 138 — 00:10:02,241 --> 00:10:05,641
We see that there's a there's a gap in the performance of the leading models at the
Cue 139 — 00:10:05,681 --> 00:10:08,981
time. So all of these,
Cue 140 — 00:10:10,801 --> 00:10:13,801
bring us to a set of design requirements for what?
Cue 141 — 00:10:13,841 --> 00:10:16,561
How we wanted to build our benchmark for ourselves.
Cue 142 — 00:10:16,601 --> 00:10:20,441
So we wanted no data leakage. We wanted to use private repositories to make sure
Cue 143 — 00:10:20,481 --> 00:10:22,901
nothing had been seen in the pre-training data.
Cue 144 — 00:10:23,041 --> 00:10:25,681
We didn't want to just use unit tests.
Cue 145 — 00:10:25,721 --> 00:10:28,441
So we have the we want to score.
Cue 146 — 00:10:28,481 --> 00:10:32,481
how mergeable it is by using static analysis, a load of quality rubrics,
Cue 147 — 00:10:32,521 --> 00:10:37,301
intent rubrics using calibrated LLM judges.
Cue 148 — 00:10:37,841 --> 00:10:43,501
We also, as I said, we use we use SonarQube to analyze for security issues and bugs.
Cue 149 — 00:10:45,361 --> 00:10:48,681
We make sure that we're using real workflows that our team is,
Cue 150 — 00:10:48,721 --> 00:10:52,861
that our team is building, and that we run in our software factories.
Cue 151 — 00:10:53,481 --> 00:10:57,121
And we evaluate the whole stack starts from the first section.
Cue 152 — 00:10:57,161 --> 00:11:00,361
But yeah, it's important not just look at the model in itself,
Cue 153 — 00:11:00,401 --> 00:11:05,701
but also the harness and any configuration you might want to change.
Cue 154 — 00:11:06,121 --> 00:11:07,781
So how do we build this.
Cue 155 — 00:11:09,881 --> 00:11:12,841
So this is roughly the five stage pipeline that we adopted.
Cue 156 — 00:11:12,881 --> 00:11:20,061
We looked through our own our own history for different commits
Cue 157 — 00:11:20,161 --> 00:11:27,461
new features. And we dropped any like trivial commits such as merges reverts.
Cue 158 — 00:11:28,041 --> 00:11:33,361
We then use an LLM to just go through and label the type of work that's being worked
Cue 159 — 00:11:33,401 --> 00:11:38,121
on. We also use some heuristics to classify how complex the change is,
Cue 160 — 00:11:38,161 --> 00:11:41,641
which will look at how many lines of code have been changed,
Cue 161 — 00:11:41,681 --> 00:11:45,761
how many files were touched, maybe how many commits were were part of the pull
Cue 162 — 00:11:45,801 --> 00:11:50,521
request. And then using this, using those features, we balance the data set.
Cue 163 — 00:11:50,561 --> 00:11:54,801
So we don't want any any workflow to be represented too much.
Cue 164 — 00:11:54,841 --> 00:11:59,321
We want a nice spread. We want a spread in in difficulties of the task that we're
Cue 165 — 00:11:59,361 --> 00:12:05,721
asking. And then we use an LLM and then review by hand the the briefs to make sure
Cue 166 — 00:12:05,761 --> 00:12:12,861
that we, are describing a sensible task to, to ask the agent.
Cue 167 — 00:12:13,961 --> 00:12:18,961
And of course, this is just so this give us about 200 tasks based on work that's
Cue 168 — 00:12:19,001 --> 00:12:22,201
done in our repositories. Of course if you did this, it would,
Cue 169 — 00:12:22,241 --> 00:12:26,541
it would, it would differ. But like this is exactly the point.
Cue 170 — 00:12:27,401 --> 00:12:31,601
And yeah, for each task we have a sandboxed, sandboxed environment.
Cue 171 — 00:12:31,641 --> 00:12:34,541
We use all the scoring techniques that I've mentioned.
Cue 172 — 00:12:34,801 --> 00:12:37,841
We make sure that all of the judges that we're using are calibrated.
Cue 173 — 00:12:37,881 --> 00:12:44,161
So LLM judges are a really quick way to assess assess results in a
Cue 174 — 00:12:44,201 --> 00:12:48,241
benchmark. But you have to make sure that the judge isn't biased,
Cue 175 — 00:12:48,281 --> 00:12:54,921
like judges can be biased by. There's been a few studies, including on bias as being
Cue 176 — 00:12:54,961 --> 00:12:58,781
judges being biased towards longer outputs.
Cue 177 — 00:12:58,881 --> 00:13:04,481
Also, judges tend to be biased towards code or output that's been generated by their
Cue 178 — 00:13:04,521 --> 00:13:09,461
own family. So Claude biases towards Claude, GPT towards GPT.
Cue 179 — 00:13:09,961 --> 00:13:14,401
So it's good. You need to take a small sample of your data, read it for each
Cue 180 — 00:13:14,441 --> 00:13:17,261
workflow, and make sure the judge is calibrated.
Cue 181 — 00:13:19,161 --> 00:13:25,921
And yeah, as I and because everything is random with we repeat the
Cue 182 — 00:13:25,961 --> 00:13:29,521
run several times. And for this for what I'm about to show you,
Cue 183 — 00:13:29,561 --> 00:13:34,361
we use t-distribution confidence intervals because that's quite a good way to
Cue 184 — 00:13:34,401 --> 00:13:37,501
measure confidence when you have a small a small sample size.
Cue 196 — 00:14:19,358 --> 00:14:23,965
So what do we find. So I'm going to show you a high level.
Cue 197 — 00:14:25,305 --> 00:14:30,745
This is I chose five models to show you a mixture of open source closed source
Cue 198 — 00:14:30,785 --> 00:14:33,205
models from from recent times.
Cue 199 — 00:14:33,945 --> 00:14:41,005
And we defined a success metric which was like a complete success for the task.
Cue 200 — 00:14:41,105 --> 00:14:43,625
So the model has to resolve the unit tests.
Cue 201 — 00:14:43,665 --> 00:14:47,805
All the judges have to score it top marks across quality.
Cue 202 — 00:14:48,665 --> 00:14:51,765
Whether the change matches the intent of the prompt.
Cue 203 — 00:14:52,065 --> 00:14:57,545
And we also use the analyzer to make sure we don't have any serious security or
Cue 204 — 00:14:57,585 --> 00:14:59,645
reliability issues in the code.
Cue 205 — 00:15:00,465 --> 00:15:04,525
And you can see the results sort of speak for themselves.
Cue 206 — 00:15:04,585 --> 00:15:09,505
You can see for the success rate, the confidence intervals intervals are quite wide,
Cue 207 — 00:15:09,545 --> 00:15:14,465
but generally the large open models do very well.
Cue 208 — 00:15:14,505 --> 00:15:16,725
Around 73% score.
Cue 209 — 00:15:18,225 --> 00:15:23,145
Whereas the models from OpenAI rank slightly further down.
Cue 210 — 00:15:23,185 --> 00:15:25,865
But there's not too much difference there.
Cue 211 — 00:15:25,905 --> 00:15:28,025
When we look at the cost, though, there's a there's a big gap.
Cue 212 — 00:15:28,065 --> 00:15:34,445
So GLM 5.3 Flash, which is a smaller version of GLM 5.3, does
Cue 213 — 00:15:35,825 --> 00:15:40,165
does a lot better in terms of cost compared to its larger version.
Cue 214 — 00:15:41,305 --> 00:15:47,985
And we see the that GLM 5.3 Flash, we believe on average gives the
Cue 215 — 00:15:48,025 --> 00:15:54,865
best optimal or not optimum on our tasks between success rate and cost per
Cue 216 — 00:15:54,905 --> 00:16:00,345
success. And so high level we wanted to look like why is why is this happening?
Cue 217 — 00:16:00,385 --> 00:16:04,005
It seems like there's a big gap to these larger open source models.
Cue 218 — 00:16:04,905 --> 00:16:09,345
We saw that they they have a lot more steps in their trajectories.
Cue 219 — 00:16:09,385 --> 00:16:12,365
And this grows the context each time.
Cue 220 — 00:16:12,945 --> 00:16:16,945
I mean, that's just a high level, but I can I want to dive into more more of what
Cue 221 — 00:16:16,985 --> 00:16:22,065
we're seeing here. So I'm just this isn't this isn't our framework,
Cue 222 — 00:16:22,105 --> 00:16:28,325
but I really like it. So this is from a paper called What Resolve Hides.
Cue 223 — 00:16:28,945 --> 00:16:30,865
And it's a framework called Trace Probe.
Cue 224 — 00:16:30,905 --> 00:16:37,085
So rather than you can see here they they categorize everything into
Cue 225 — 00:16:37,185 --> 00:16:43,745
searches reads, edits, tool calls reversion these like typical steps that an agent
Cue 226 — 00:16:43,785 --> 00:16:47,865
might take in its trajectory. And they can use these to then map between different
Cue 227 — 00:16:47,905 --> 00:16:50,925
agent trajectories. So it works a lot.
Cue 228 — 00:16:51,145 --> 00:16:52,125
It works,
Cue 229 — 00:16:54,065 --> 00:16:56,305
a lot nicer than than having to.
Cue 230 — 00:16:56,345 --> 00:16:59,505
I'm sure some of you have like, gone in and tried to figure out what your agents
Cue 231 — 00:16:59,545 --> 00:17:01,345
doing and have to look step by step.
Cue 232 — 00:17:01,385 --> 00:17:04,605
I think this gives a really nice, framework.
Cue 233 — 00:17:04,665 --> 00:17:06,345
And then it looks at patterns.
Cue 234 — 00:17:06,385 --> 00:17:10,545
So it will look for like bad behaviors, whether an agent gets stuck searching the
Cue 235 — 00:17:10,585 --> 00:17:15,105
same files again and again, or of an agent goes to edit a file and then a few turns
Cue 236 — 00:17:15,145 --> 00:17:16,785
later goes to reedit that same file.
Cue 237 — 00:17:16,825 --> 00:17:18,845
That's a behavior that we don't want either.
Cue 238 — 00:17:19,425 --> 00:17:23,785
And then there's some more complicated stuff you can do comparing comparing
Cue 239 — 00:17:23,825 --> 00:17:29,005
different agents. So what do we see only when we apply this to our own benchmark.
Cue 240 — 00:17:29,865 --> 00:17:35,605
So yeah this is this is interesting when we when we look at the the tasks that the
Cue 241 — 00:17:35,745 --> 00:17:40,465
agents got wrong on our benchmarks, we see what happens with the failures.
Cue 242 — 00:17:40,505 --> 00:17:46,105
There are certain models. So GPT-6 when it starts to get a benchmark wrong,
Cue 243 — 00:17:46,145 --> 00:17:50,945
it decides to stop more quickly than these other open source models we're seeing.
Cue 244 — 00:17:50,985 --> 00:17:57,905
So I mean, Kimi K3 has averaged like 105, 105 steps, which is like four times
Cue 245 — 00:17:57,945 --> 00:18:00,325
as much as GPT-6.
Cue 246 — 00:18:02,545 --> 00:18:06,025
And when we when we actually look into like why this happens,
Cue 247 — 00:18:06,065 --> 00:18:09,805
we looked at Flash because that's the model we were interested in earlier.
Cue 248 — 00:18:10,185 --> 00:18:17,185
And Flash only fails a very small amount of the, easy tasks and medium ones.
Cue 249 — 00:18:17,225 --> 00:18:20,625
This is using a slightly different success metric to to earlier.
Cue 250 — 00:18:20,665 --> 00:18:27,545
This is about unit tests, but the on the hard tasks it fails a
Cue 251 — 00:18:27,585 --> 00:18:32,305
lot more of them. And on those hard tasks that's quite costly because it it uses
Cue 252 — 00:18:32,345 --> 00:18:35,145
like three times as many steps as the other models.
Cue 253 — 00:18:35,185 --> 00:18:40,585
So this tells us that when like GLM 5.3 Flash works very well.
Cue 254 — 00:18:40,625 --> 00:18:46,865
It's very cost effective on the on the easy and medium tasks as we as we classified
Cue 255 — 00:18:46,905 --> 00:18:53,625
them. Another lens to look at, maybe beyond just the GLM 5.3 Flash as,
Cue 256 — 00:18:53,665 --> 00:18:55,965
the best model would be security.
Cue 257 — 00:18:56,585 --> 00:19:00,765
So this is using the SonarQube security analysis.
Cue 258 — 00:19:01,505 --> 00:19:03,825
And we saw across all the models.
Cue 259 — 00:19:03,865 --> 00:19:08,525
This is a number of Sonar issues that got introduced across our benchmarks.
Cue 260 — 00:19:08,585 --> 00:19:13,765
Then security which is a fairly rare, a rare type of type of issue.
Cue 261 — 00:19:14,385 --> 00:19:20,265
We saw that GPT-6. And even though they produce more issues overall,
Cue 262 — 00:19:20,305 --> 00:19:23,865
which will be like maintainability or reliability issues, we see very strong
Cue 263 — 00:19:23,905 --> 00:19:26,105
performance in security. And it drops.
Cue 264 — 00:19:26,145 --> 00:19:28,965
They both drop down below the open source models.
Cue 265 — 00:19:30,625 --> 00:19:34,065
And then I thought, okay, let's not just look at all security issues.
Cue 266 — 00:19:34,105 --> 00:19:37,445
I want to look at like the real blocker and high security issues.
Cue 267 — 00:19:37,485 --> 00:19:41,805
So this is stuff that if it went into production, it's likely to
Cue 268 — 00:19:43,385 --> 00:19:48,085
yeah, it's very easy to target and it will have a big impact if it is targeted.
Cue 269 — 00:19:48,585 --> 00:19:53,245
And we found even then the the number of
Cue 270 — 00:19:54,665 --> 00:19:58,485
issues with GPT-6 is a lot lower than the others.
Cue 271 — 00:19:58,705 --> 00:20:02,385
And I then I wanted to look at what type of what type of workflow this happens most
Cue 272 — 00:20:02,425 --> 00:20:08,065
for. And you can see there's this big difference between GPT-6 and the open source
Cue 273 — 00:20:08,105 --> 00:20:10,545
models stays for new features.
Cue 274 — 00:20:10,585 --> 00:20:16,569
So this would tell us that for security sensitive tasks you should use GPT-6 Sol or
Cue 275 — 00:20:16,609 --> 00:20:22,665
GPT-6 Terra. And especially if you're doing new feature work.
Cue 276 — 00:20:22,705 --> 00:20:25,385
So this could be something. This might give you an insight that in your software
Cue 277 — 00:20:25,425 --> 00:20:30,145
factory, when you're implementing new features, you want to use a security sensitive
Cue 278 — 00:20:30,185 --> 00:20:31,725
model like one of these.
Cue 279 — 00:20:33,025 --> 00:20:37,845
And then finally, I'll briefly touch on, harnesses.
Cue 280 — 00:20:38,345 --> 00:20:44,525
So this is the same GPT-6 Terra we saw just then was very good with security.
Cue 281 — 00:20:44,785 --> 00:20:48,065
And now we look at swapping out the harness.
Cue 282 — 00:20:48,105 --> 00:20:53,705
So these are three popular harnesses, OpenCode, which is very popular I use it I use
Cue 283 — 00:20:53,745 --> 00:21:00,505
it quite a lot, Pi and Codex. And you can see the model achieves
Cue 284 — 00:21:00,545 --> 00:21:05,345
roughly the same success rate across the three harnesses, a little bit less on on
Cue 285 — 00:21:05,385 --> 00:21:11,905
Codex. And you can see there's actually a reasonably significant difference in cost,
Cue 286 — 00:21:11,945 --> 00:21:14,865
especially between OpenCode, Pi and Codex.
Cue 287 — 00:21:14,905 --> 00:21:21,745
So which I found quite interesting because, I mean, GPT-6 would have been trained
Cue 288 — 00:21:21,785 --> 00:21:24,605
like within within Codex I'd imagine.
Cue 289 — 00:21:27,105 --> 00:21:28,625
So yeah, this actually reminds me of that.
Cue 290 — 00:21:28,665 --> 00:21:30,685
There was a talk by Anthropic yesterday over.
Cue 291 — 00:21:30,725 --> 00:21:32,985
They're talking about like de-harnessing your model.
Cue 292 — 00:21:33,025 --> 00:21:35,525
And I think this shows that you should always
Cue 293 — 00:21:36,785 --> 00:21:40,065
you should always look at swapping out your harness, because it might be that it's
Cue 294 — 00:21:40,105 --> 00:21:44,665
wasting tokens and not actually giving you a better success rate when you when you
Cue 295 — 00:21:44,705 --> 00:21:45,965
reevaluate it.
Cue 296 — 00:21:47,705 --> 00:21:52,165
Okay. So to summarize so what what we learned from this.
Cue 297 — 00:21:52,305 --> 00:21:57,205
So for easy tasks you can you can use GLM 5.3 Flash.
Cue 298 — 00:21:58,225 --> 00:22:04,845
It had one of the best success rates and it was very cheap per person task.
Cue 299 — 00:22:05,265 --> 00:22:11,345
But then we saw that for harder tasks GLM 5.3 Flash would actually tend to get stuck
Cue 300 — 00:22:11,385 --> 00:22:17,485
in loops. It would tend to reedit files and this causes it to to fail.
Cue 301 — 00:22:18,105 --> 00:22:23,025
Whereas GPT-6 Terra or GPT-6 Sol, they will stop when they can,
Cue 302 — 00:22:23,065 --> 00:22:25,665
when they've figured out that they're they're not going to be able to solve the
Cue 303 — 00:22:25,705 --> 00:22:32,545
task. And then for new features, I honed in on that workflow because it's
Cue 304 — 00:22:32,585 --> 00:22:37,625
very security sensitive. We found that, again, the OpenAI models are the best to go
Cue 305 — 00:22:37,665 --> 00:22:42,685
for. And then finally, what we learned from looking at the harness was Terra
Cue 306 — 00:22:42,725 --> 00:22:48,325
actually costs more when you run it in the Codex harness than it did when we ran it
Cue 307 — 00:22:48,465 --> 00:22:50,005
in OpenCode or Pi.
Cue 308 — 00:22:51,305 --> 00:22:53,625
But yeah, this is this is like very niche.
Cue 309 — 00:22:53,665 --> 00:22:56,785
This is just based on the benchmarks that we ran on, like a handful of other
Cue 310 — 00:22:56,825 --> 00:22:59,425
repositories. But I do find it quite interesting.
Cue 311 — 00:22:59,465 --> 00:23:05,225
And, I mean, the takeaway is that this is like, this is not you're not going to find
Cue 312 — 00:23:05,265 --> 00:23:07,065
this by looking at a public leaderboard.
Cue 313 — 00:23:07,105 --> 00:23:10,485
And you could if you do this for yourselves, you'll be able to find out,
Cue 314 — 00:23:10,865 --> 00:23:13,625
different workflows within your software factories or different agents.
Cue 315 — 00:23:13,665 --> 00:23:16,405
You'll be able to find these sort of insights out.
Cue 316 — 00:23:18,705 --> 00:23:20,605
And yeah, just some quick takeaways.
Cue 317 — 00:23:21,065 --> 00:23:23,945
So public benchmarks, I still I still think they're useful.
Cue 318 — 00:23:23,985 --> 00:23:26,765
I still think people should report them when they release models.
Cue 319 — 00:23:27,025 --> 00:23:31,785
They just give a general sense of model performance, but they should just give you a
Cue 320 — 00:23:31,825 --> 00:23:36,225
short list of candidates. They can't actually give you a decision of which model you
Cue 321 — 00:23:36,265 --> 00:23:39,765
should use. And your agent, and especially in your software factory,
Cue 322 — 00:23:40,665 --> 00:23:43,065
you should always build tasks from your context.
Cue 323 — 00:23:43,105 --> 00:23:47,705
Like I can't tell you exactly what the best things are to measure within your code
Cue 324 — 00:23:47,745 --> 00:23:50,525
base. Everyone's going to be very different.
Cue 325 — 00:23:50,785 --> 00:23:56,405
I'd always recommend using deterministic interpretable verifiers when you can,
Cue 326 — 00:23:57,225 --> 00:24:00,665
but if you can't use an LLM judge, it's very quick.
Cue 327 — 00:24:00,705 --> 00:24:02,525
But make sure you calibrate it.
Cue 328 — 00:24:02,945 --> 00:24:05,185
And finally, always be curious.
Cue 329 — 00:24:05,225 --> 00:24:10,385
Like look beyond just the high level success rate and how expensive stuff is per run
Cue 330 — 00:24:10,425 --> 00:24:15,825
on average. And use tools like trace probe, or looking at the trajectories directly
Cue 331 — 00:24:15,865 --> 00:24:20,085
and try and figure out the real trends for different tasks.
Cue 332 — 00:24:20,905 --> 00:24:26,405
So yeah, I've got some references there and thank you for listening.
Cue 333 — 00:24:26,625 --> 00:24:29,085
That's my LinkedIn and happy to speak.
Cue 334 — 00:24:34,445 --> 00:24:35,965
Awesome. Thank you. Awesome. Thank you very much.
Cue 335 — 00:24:36,005 --> 00:24:37,725
Do we have some time questions.
Cue 336 — 00:24:37,765 --> 00:24:40,585
Any one with questions I want to share.
Cue 337 — 00:24:47,285 --> 00:24:50,085
Appreciate it. I just want to ask like more open ended.
Cue 338 — 00:24:50,125 --> 00:24:54,045
Like have you looked at like testing like single agent versus like agent swarms and
Cue 339 — 00:24:54,085 --> 00:24:57,205
like performance of like a single agent trying to complete a hard task versus like
Cue 340 — 00:24:57,245 --> 00:25:00,085
swarm of like smaller agents completing a task and just what your experience is
Cue 341 — 00:25:00,125 --> 00:25:04,325
without. Yeah. I mean on this benchmark we haven't we haven't done that yet.
Cue 342 — 00:25:04,365 --> 00:25:10,485
I think that's that's another great application of a benchmark like this to do,
Cue 343 — 00:25:10,525 --> 00:25:14,725
like a swarm or if you have some, if you have like some AI tool internally that you
Cue 344 — 00:25:14,765 --> 00:25:18,405
want to that you want to test because you have this sandbox environment,
Cue 345 — 00:25:18,445 --> 00:25:21,565
you can set it up in any way you want and, and measure that,
Cue 346 — 00:25:21,605 --> 00:25:23,645
measure the costs. So yeah, absolutely.
Cue 347 — 00:25:23,685 --> 00:25:24,685
I think you can extend it to that.
Cue 348 — 00:25:24,725 --> 00:25:27,185
We didn't do that. But that would be a really interesting.
Cue 349 — 00:25:28,485 --> 00:25:32,325
Can you share more on your calibration process for an LLM judge.
Cue 350 — 00:25:32,365 --> 00:25:36,685
And you previously mentioned manual like some manual steps involved.
Cue 351 — 00:25:36,725 --> 00:25:37,525
Yeah.
Cue 352 — 00:25:38,725 --> 00:25:45,525
Sure. So we have a across like let me take a like the
Cue 353 — 00:25:45,565 --> 00:25:47,705
intent rubric. So that would be
Cue 354 — 00:25:49,445 --> 00:25:54,965
you'd get your we use an LLM to create the prompt and then we'd have the reference
Cue 355 — 00:25:55,005 --> 00:26:01,945
change. And then we would give our LLM judge all of the,
Cue 356 — 00:26:02,925 --> 00:26:06,985
all of the combinations, and you'd get sort of ten, ten predictions on scores.
Cue 357 — 00:26:07,125 --> 00:26:11,725
And then we would ask like get real developers to look at the,
Cue 358 — 00:26:11,765 --> 00:26:13,545
the changes against,
Cue 359 — 00:26:16,445 --> 00:26:21,265
yeah. Look at the changes against the rubric that was defined and make sure that you
Cue 360 — 00:26:21,805 --> 00:26:24,165
like and you can create like with changes from different models,
Cue 361 — 00:26:24,205 --> 00:26:27,285
different combinations, and you'll get lots of different quality of answers.
Cue 362 — 00:26:27,325 --> 00:26:30,065
And you go through and you make sure that the,
Cue 363 — 00:26:31,645 --> 00:26:38,205
the rankings of the like the empirical rankings of what the LLM judge decided,
Cue 364 — 00:26:38,245 --> 00:26:40,605
matches with what your labeler would say.
Cue 365 — 00:26:40,645 --> 00:26:42,125
And if not, then you need to go.
Cue 366 — 00:26:42,165 --> 00:26:45,725
You need to look back at the prompt of your judge and try and change it,
Cue 367 — 00:26:45,765 --> 00:26:50,285
because you might have overly specified something, or you might have missed out some
Cue 368 — 00:26:50,325 --> 00:26:53,065
detail that let it, let it, let it hack the reward.
Cue 369 — 00:26:53,597 --> 00:26:55,637
Please give it up for Joe. All right.
Cue 370 — 00:26:55,677 --> 00:26:56,537
Thank you.
.tessl-plugin
skills
talk-arcolano-autonomy-and-trust
talk-aroussi-decision-context
talk-bevin-agent-assisted-instrument-design
talk-boeckeler-context-lifecycles
talk-braun-context-graphs
talk-carmont-evolving-agent-harnesses
talk-chanezon-gordon-agent-experience-evaluation
talk-cox-secure-agentic-workflows
talk-cunniffe-reusable-agent-platform
talk-davis-agent-tool-policies
talk-debois-concurrent-delivery
talk-dzialets-team-automation
talk-fahmy-harness-api-design
talk-finster-continuous-delivery
talk-fowler-regenerative-software
talk-gavrilescu-factory-architecture
talk-graciolli-amin-verification-aware-programming
talk-gruenbaum-reverse-graph-engineering
talk-hassan-agentic-engineering
talk-hossain-enterprise-agent-adoption
talk-huntley-patch-ready-factories
talk-jain-verifying-intent
talk-katsioloudes-montei-agentic-lifecycle
talk-knox-harness-engineering
talk-kovac-deterministic-memory
talk-lee-perkins-code-readiness
talk-lorenc-machine-speed-security
talk-lubken-software-of-one
talk-meijer-software-specialization
talk-meppiel-versioned-agent-contracts
talk-moss-concurrent-agent-risk
talk-myers-understanding-agent-generated-code
talk-ocytko-scaling-agentic-engineering
talk-oza-automating-design-quality
talk-panzica-agentic-sdlc
talk-pari-patel-agent-awareness
talk-podjarny-trustworthy-software-factories
talk-poulter-factory-improvement
talk-pratik-patel-refined-rag-data
talk-re-ferre-development-platforms
talk-sadogursky-factory-of-factories
talk-shah-training-agent-skills
talk-sharma-mohapatra-evolving-intent
talk-singhal-operating-agents-at-scale
talk-stack-incident-dossiers
talk-tyler-team-coding-benchmarks
talk-wotherspoon-agent-execution-environments
talk-yegge-ai-powered-factories
talk-zuber-leadership-through-coding