CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-carmont-evolving-agent-harnesses/references/

Corrected transcript: Building a harness that keeps up with the latest models

Nick Carmont · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=IiAvs9hiQZ0

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:01,054 --> 00:00:04,174

Hi everyone, and thanks so much for having me.

Cue 2 — 00:00:04,214 --> 00:00:09,034

And huge thank you to AI DevCon and Tessl for putting this all together.

Cue 3 — 00:00:09,784 --> 00:00:13,234

I am one of those people that gets very excited about model releases,

Cue 4 — 00:00:13,494 --> 00:00:16,594

so I think I am within one of within that bucket.

Cue 5 — 00:00:17,174 --> 00:00:22,906

And the topic for today is building a harness that keeps up with those latest models

Cue 6 — 00:00:23,094 --> 00:00:27,894

specifically around experiences in the un-harnessing SDLC loop.

Cue 7 — 00:00:27,934 --> 00:00:32,734

In short, it's not just about adding stuff to your harness and keep on adding

Cue 8 — 00:00:32,774 --> 00:00:37,774

instructions, but it's also about how do you ensure that you keep the instructions

Cue 9 — 00:00:37,814 --> 00:00:41,974

and the harness around the model quite lean and quite high signal.

Cue 10 — 00:00:42,014 --> 00:00:43,814

And how do you do that continuously?

Cue 11 — 00:00:43,854 --> 00:00:45,934

So I know this is the talk before lunch.

Cue 12 — 00:00:45,974 --> 00:00:47,814

Hopefully you're not too hungry.

Cue 13 — 00:00:47,854 --> 00:00:50,054

And if you are hungry, you're hungry.

Cue 14 — 00:00:50,094 --> 00:00:54,014

To learn more about how to optimize your harness with with new models.

Cue 15 — 00:00:54,054 --> 00:00:57,094

So I'm Nick, member of technical staff on our Applied AI team.

Cue 16 — 00:00:57,134 --> 00:01:00,734

I get the privilege of sitting between our research team, our model team,

Cue 17 — 00:01:00,774 --> 00:01:04,894

and some of our most strategic customers, particularly startups,

Cue 18 — 00:01:04,934 --> 00:01:09,814

and being on a hands on technical level, helping them with every model release.

Cue 19 — 00:01:09,854 --> 00:01:15,614

How do we make sure that this model and your harness and evals best benefit from

Cue 20 — 00:01:15,654 --> 00:01:20,134

that intelligence, and how, when that is the case, can we help that for your use

Cue 21 — 00:01:20,174 --> 00:01:23,654

case? And when the model has some feedback that's needed, how can I channel that.

Cue 22 — 00:01:23,694 --> 00:01:27,794

Best to to the model research team in order to to optimize that loop.

Cue 23 — 00:01:29,334 --> 00:01:33,354

So this is the kind of loop that I go through.

Cue 24 — 00:01:33,494 --> 00:01:37,374

I help AI native startups optimize their harness their evals with each model

Cue 25 — 00:01:37,414 --> 00:01:40,774

release, and any time that the model requires feedback.

Cue 26 — 00:01:40,814 --> 00:01:44,094

I then work with our research team in order to kind of channel that back.

Cue 27 — 00:01:44,134 --> 00:01:48,694

And recently, for me personally, it's felt like almost like a 100 meter Olympic

Cue 28 — 00:01:48,734 --> 00:01:52,814

sprints. We had a Opus 5.5 release not too long ago.

Cue 29 — 00:01:52,854 --> 00:01:55,894

Then we had Sonnet 5.5 and Haiku 5.5.

Cue 30 — 00:01:55,934 --> 00:02:00,094

So it's been a busy period. But this is typically how it works,

Cue 31 — 00:02:00,134 --> 00:02:04,934

where a new model ships. I then work with customers to rerun their evals,

Cue 32 — 00:02:04,974 --> 00:02:09,574

see how that model performs on on their use case, and then it's a matter of seeing

Cue 33 — 00:02:09,614 --> 00:02:12,294

okay. Should we adjust the model?

Cue 34 — 00:02:12,334 --> 00:02:14,614

Should I kind of bring that feedback back?

Cue 35 — 00:02:14,654 --> 00:02:18,494

Is there anything in the harness that requires optimization and is there anything in

Cue 36 — 00:02:18,534 --> 00:02:22,934

the eval? Let's say that it's starting to saturate that needs adjustment.

Cue 37 — 00:02:22,974 --> 00:02:28,094

And then helping those customers ship that new model, potentially towards millions

Cue 38 — 00:02:28,134 --> 00:02:31,774

of customers to be able to, as soon as possible, benefit from those gains.

Cue 39 — 00:02:31,814 --> 00:02:34,594

So with Opus 5.5, that was quite nice.

Cue 40 — 00:02:34,694 --> 00:02:39,534

Customers were able to receive an increase in intelligence at a lower cost because

Cue 41 — 00:02:39,574 --> 00:02:44,694

of its token efficiencies. But this is a little bit of of the loop that I typically

Cue 42 — 00:02:44,734 --> 00:02:47,554

go through and help some of our, our customers on.

Cue 43 — 00:02:49,774 --> 00:02:51,974

This sometimes feels like Narnia.

Cue 44 — 00:02:52,014 --> 00:02:55,839

And the reason is, I don't know how many have watched Narnia:

Cue 45 — 00:02:55,879 --> 00:02:58,034

The Lion, the Witch and the Wardrobe.

Cue 46 — 00:02:58,294 --> 00:03:01,574

My life revolves around, well, sorry.

Cue 47 — 00:03:01,614 --> 00:03:03,934

The three topics today I'll talk about is the model, the harness,

Cue 48 — 00:03:03,974 --> 00:03:08,934

and the eval. Why? Claude Code cut its system prompt over 80%.

Cue 49 — 00:03:08,974 --> 00:03:14,494

And a bit about the loop. It's not just about adding instructions and skills,

Cue 50 — 00:03:14,534 --> 00:03:18,374

but if you're working on a software factory on loops, a lot of the fundamentals end

Cue 51 — 00:03:18,414 --> 00:03:23,334

up compounding. And so having the fundamentals right and lean becomes ever more

Cue 52 — 00:03:23,374 --> 00:03:25,514

important as you go scaling things.

Cue 53 — 00:03:25,854 --> 00:03:29,774

So I said this tends to be a bit like Narnia.

Cue 54 — 00:03:29,814 --> 00:03:33,014

So instead of The Lion, the Witch and the Wardrobe, I work with the model,

Cue 55 — 00:03:33,054 --> 00:03:37,854

the harness or the eval. So every model that drops is understanding okay,

Cue 56 — 00:03:37,894 --> 00:03:42,494

is it the model that isn't performing correctly on this use case or requires

Cue 57 — 00:03:42,534 --> 00:03:47,654

adjustment? Is it the harness which is restricting the model from being able to do

Cue 58 — 00:03:47,694 --> 00:03:49,534

something in the specific use case?

Cue 59 — 00:03:49,574 --> 00:03:55,254

Or is there things within the eval that are, for example, saturated or there might

Cue 60 — 00:03:55,294 --> 00:03:57,534

be even in this actually happens quite often.

Cue 61 — 00:03:57,574 --> 00:04:01,414

There are specific use cases in which it's actually an impossible task.

Cue 62 — 00:04:01,454 --> 00:04:05,094

So I've had some startup customers be like, oh, even the frontier models are like

Cue 63 — 00:04:05,134 --> 00:04:12,054

only reaching 50%. And then the the maximum anyone can achieve on that task is 50%,

Cue 64 — 00:04:12,094 --> 00:04:15,234

because the rest of the 50% are are impossible tasks.

Cue 65 — 00:04:15,454 --> 00:04:20,494

And so it's a it's a bit of this loop of optimization when you get a new model in

Cue 66 — 00:04:20,534 --> 00:04:25,054

order to how to best optimize it so that the intelligence flows towards the end

Cue 67 — 00:04:25,094 --> 00:04:29,354

customer, and they get the best experience and the highest value from this.

Cue 68 — 00:04:29,814 --> 00:04:34,274

And there tends to be some telltale signs as to which is the problem.

Cue 69 — 00:04:34,414 --> 00:04:37,454

One way to think about it is let's say this is a driving exam,

Cue 70 — 00:04:37,494 --> 00:04:41,174

and I unfortunately had a lot of problems with driving exams and probably not the

Cue 71 — 00:04:41,214 --> 00:04:44,774

best driver. But let's treat the model as like the engine.

Cue 72 — 00:04:44,814 --> 00:04:49,774

The harness is like a car. The model in the harness like a driver you have like the

Cue 73 — 00:04:49,814 --> 00:04:53,654

instructions, you have the engine, you have like the intelligence and the eval is

Cue 74 — 00:04:53,694 --> 00:04:58,894

like the driving exam. And so what happens often is let's say that you switch in a

Cue 75 — 00:04:58,934 --> 00:05:02,194

new model and you can see that it performs better or worse.

Cue 76 — 00:05:02,294 --> 00:05:05,734

Some signs are okay on higher efforts or new models.

Cue 77 — 00:05:05,774 --> 00:05:09,694

We start to see potentially the model increasing in performance or reducing in

Cue 78 — 00:05:09,734 --> 00:05:13,214

performance. And so we can start to see, okay, as we increase the effort,

Cue 79 — 00:05:13,254 --> 00:05:20,174

let's say how much acceleration you put on the engine, is the car going faster,

Cue 80 — 00:05:20,214 --> 00:05:22,394

is it improving in performance etc..

Cue 81 — 00:05:22,854 --> 00:05:28,916

The harness around the car, one of the telltale signs of the harness being what's

Cue 82 — 00:05:28,956 --> 00:05:35,454

kind of holding back potentially this use case is, if you compare a more

Cue 83 — 00:05:35,494 --> 00:05:39,654

stripped-back version of the harness versus your current harness and you see

Cue 84 — 00:05:39,694 --> 00:05:43,854

increases in performance, it can start to be a sign that the harness is actually

Cue 85 — 00:05:43,894 --> 00:05:48,014

what's holding things back. And so I'll go over some examples later that I've seen.

Cue 86 — 00:05:48,054 --> 00:05:53,734

But often some teams have built this really elaborate harness with tons of

Cue 87 — 00:05:53,774 --> 00:05:58,774

instructions, tons of steps, and a lot of work has been done in increasing and

Cue 88 — 00:05:58,814 --> 00:06:04,594

making this like bigger in terms of instructions, but not much has has been done.

Cue 89 — 00:06:05,094 --> 00:06:08,774

And how do we reduce this? How do we make this as high signal as possible and flow

Cue 90 — 00:06:08,814 --> 00:06:14,654

quite well. And so one sign starts to be, okay, let's actually run the model with a

Cue 91 — 00:06:14,694 --> 00:06:18,034

leaner harness or no harness versus the current one.

Cue 92 — 00:06:18,214 --> 00:06:21,654

And it starts to be a little bit more like treating this like a science experiment

Cue 93 — 00:06:21,694 --> 00:06:23,214

rather than an engineering one.

Cue 94 — 00:06:23,254 --> 00:06:27,414

How do we make sure that the harness that's selected is the most optimal for the use

Cue 95 — 00:06:27,454 --> 00:06:30,634

case? It's not just one that we decided to kind of put forward.

Cue 96 — 00:06:31,294 --> 00:06:35,574

And then the eval, you should never trust the eval kind of blindly.

Cue 97 — 00:06:35,614 --> 00:06:38,814

It's always about. So let's say this is a driving test.

Cue 98 — 00:06:38,854 --> 00:06:45,054

And part of the exam marks is that you have to go into a road which is closed

Cue 99 — 00:06:45,094 --> 00:06:50,494

temporarily, and so you can run your eval, and then no one's scoring very high on

Cue 100 — 00:06:50,534 --> 00:06:55,554

this eval. And the telltale sign is like, don't just look at the results.

Cue 101 — 00:06:55,654 --> 00:06:58,674

Actually look at what the harness and the model are telling you.

Cue 102 — 00:06:58,814 --> 00:07:03,294

Are they saying, hey, this is an impossible task because I can't go left because the

Cue 103 — 00:07:03,334 --> 00:07:08,034

road is closed, for example. And this happens a lot more than you'd expect.

Cue 104 — 00:07:08,454 --> 00:07:12,634

A lot of and as they get harder and harder and start to saturate,

Cue 105 — 00:07:12,974 --> 00:07:16,854

it actually becomes harder to really look at like what good looks like and the

Cue 106 — 00:07:16,894 --> 00:07:19,274

things that only frontier models can achieve.

Cue 107 — 00:07:20,374 --> 00:07:24,494

You have to be very specific and concrete and really close to the problem to really

Cue 108 — 00:07:24,534 --> 00:07:29,374

define a good eval. So those are the three things that I tend to help customers with

Cue 109 — 00:07:29,414 --> 00:07:32,754

and optimize in terms of model, harness and eval.

Cue 110 — 00:07:35,294 --> 00:07:39,814

And in terms of the model itself, it is true that every model can have a little bit

Cue 111 — 00:07:39,854 --> 00:07:42,154

of its own traits and personality.

Cue 112 — 00:07:42,294 --> 00:07:49,074

So one of the observations was with the switch from older models to Claude 5 models.

Cue 113 — 00:07:50,374 --> 00:07:55,874

A lot of the things that so if you worked with Sonnet 3.5 or Opus 4.5,

Cue 114 — 00:07:56,134 --> 00:07:59,054

you often had to tell it a lot how to do things.

Cue 115 — 00:07:59,094 --> 00:08:01,434

And apart from like what to do.

Cue 116 — 00:08:02,174 --> 00:08:07,174

And one of the good things of some of the latest models is a lot of the common

Cue 117 — 00:08:07,214 --> 00:08:11,694

things that you would tend to prompt that are kind of universal start to be baked

Cue 118 — 00:08:11,734 --> 00:08:17,414

into the model. And so I've spent maybe two months with customers really focused on

Cue 119 — 00:08:17,454 --> 00:08:20,194

system prompts and like over verification.

Cue 120 — 00:08:20,454 --> 00:08:25,364

You can't believe the amount of times in skills system prompts in MD.

Cue 121 — 00:08:25,454 --> 00:08:29,894

I had this all over my computer, which is like really verify absolutely everything

Cue 122 — 00:08:29,934 --> 00:08:32,474

you do, create tests and run tests.

Cue 123 — 00:08:32,654 --> 00:08:36,494

But the problem was that these same instructions were replicated across so many

Cue 124 — 00:08:36,534 --> 00:08:39,934

different steps, in so many different parts of like the instructions and system

Cue 125 — 00:08:39,974 --> 00:08:46,094

prompt. And the thing is, starting with Opus 5 and the rest of the family,

Cue 126 — 00:08:46,134 --> 00:08:50,054

verification is something that's very heavily built into the models instructions

Cue 127 — 00:08:50,094 --> 00:08:55,054

anyways, even if you don't tell it to verify, it will go about doing that.

Cue 128 — 00:08:55,094 --> 00:09:00,254

And so one of the consequences starts being that, if you just drop a model into an

Cue 129 — 00:09:00,294 --> 00:09:04,114

existing harness, there starts to be duplicated efforts.

Cue 130 — 00:09:04,454 --> 00:09:10,094

So I saw with some customers costs go up like 30% and then go down to like 0 or 5

Cue 131 — 00:09:10,134 --> 00:09:14,300

plus percent. Once that you really lean in and you have de-duplicated the

Cue 132 — 00:09:14,340 --> 00:09:16,114

instructions of like verification.

Cue 133 — 00:09:16,214 --> 00:09:21,274

All this to say, the models, as they grow more intelligent are not just,

Cue 134 — 00:09:21,974 --> 00:09:26,254

having differences in patterns of behaviors, but they're also tend to bake in more

Cue 135 — 00:09:26,294 --> 00:09:29,674

instructions that were otherwise in the system prompt.

Cue 136 — 00:09:30,094 --> 00:09:33,854

And this actually makes it cheaper because you're now not paying for certain tokens.

Cue 137 — 00:09:33,894 --> 00:09:36,954

It's already kind of built in to part of the behavior.

Cue 138 — 00:09:39,214 --> 00:09:42,334

So this is just a the same piece here.

Cue 139 — 00:09:42,374 --> 00:09:45,394

Here's an example of instructions to verify every file.

Cue 140 — 00:09:45,534 --> 00:09:48,434

And here just saying the exact instructions.

Cue 141 — 00:09:48,774 --> 00:09:53,434

The main point here being make sure to verify your verifications.

Cue 142 — 00:09:53,774 --> 00:09:59,634

And there are certain things that actually benefit from a lean model as the as the

Cue 143 — 00:09:59,934 --> 00:10:05,094

models get more intelligent. So that was a bit about the model,

Cue 144 — 00:10:05,134 --> 00:10:07,114

the harness and the eval.

Cue 145 — 00:10:09,094 --> 00:10:13,934

One quick one here is I used to be a startup CTO, and so I used to build websites

Cue 146 — 00:10:13,974 --> 00:10:17,594

from scratch and SVG animations, a bunch of embarrassing stuff now.

Cue 147 — 00:10:18,014 --> 00:10:21,334

And it's crazy to think to have coded from from scratch a bunch.

Cue 148 — 00:10:21,374 --> 00:10:25,134

But one of the things in software engineering is the concept of bit rot.

Cue 149 — 00:10:25,174 --> 00:10:29,574

That is, your code base gets bigger and older.

Cue 150 — 00:10:29,614 --> 00:10:32,834

There are certain parts that start to, for example, decay.

Cue 151 — 00:10:33,094 --> 00:10:38,394

And now that even in the previous talk, the cost of software is so cheap,

Cue 152 — 00:10:38,454 --> 00:10:42,694

the rate of bit decay happens a lot faster because there's a lot more code being

Cue 153 — 00:10:42,734 --> 00:10:46,894

built at once. And one thing to think about for about your harness,

Cue 154 — 00:10:46,934 --> 00:10:51,494

which is, yeah, everything around the AI model to power your use case is that it

Cue 155 — 00:10:51,534 --> 00:10:55,334

also has a shelf life, and you should see it as something dynamic and constantly

Cue 156 — 00:10:55,374 --> 00:10:59,234

improving, because the objective isn't to build a big harness,

Cue 157 — 00:10:59,294 --> 00:11:03,854

it's to have the model plus harness power, the best use case,

Cue 158 — 00:11:03,894 --> 00:11:05,814

and the highest value for the end user.

Cue 159 — 00:11:05,854 --> 00:11:09,634

And so you should look at this as a as a constant evolution in cycle.

Cue 160 — 00:11:11,734 --> 00:11:16,294

To give an example, this summer, the Claude Code team cut the system prompt by over

Cue 161 — 00:11:16,334 --> 00:11:20,614

80%. And that's quite a hard core thing to do because you're like,

Cue 162 — 00:11:20,654 --> 00:11:23,854

oh, how would that affect customers usage, etc..

Cue 163 — 00:11:23,894 --> 00:11:28,454

And they saw that this is one of the biggest benefits because as mentioned just now,

Cue 164 — 00:11:28,494 --> 00:11:31,814

the model is starting to bake in a lot of the best practices.

Cue 165 — 00:11:31,854 --> 00:11:33,394

And by

Cue 166 — 00:11:34,614 --> 00:11:40,514

and just keeping the highest signal 20% of instructions and tooling for the model,

Cue 167 — 00:11:40,774 --> 00:11:44,294

it's able to find the best solution itself for most of these problems.

Cue 168 — 00:11:44,334 --> 00:11:48,374

And all this to say, with a lot of startup customers, there is a find that just the

Cue 169 — 00:11:48,414 --> 00:11:52,694

leaner the harness, the more new models and new intelligence can actually benefit

Cue 170 — 00:11:52,734 --> 00:11:57,814

the use case. And it's kind of the equivalent of having being a manager and having

Cue 171 — 00:11:57,854 --> 00:12:03,194

an employee. And instead of telling it, micromanaging and saying how to do things

Cue 172 — 00:12:03,534 --> 00:12:07,814

super strictly, you allow this in very intelligent employee to really express and

Cue 173 — 00:12:07,854 --> 00:12:09,174

find the best way to do things.

Cue 174 — 00:12:09,214 --> 00:12:13,554

And really defining with high signal the what you want to achieve.

Cue 175 — 00:12:15,774 --> 00:12:22,254

And there's kind of two main theories as to why works un-harnessing is just kind of

Cue 176 — 00:12:22,294 --> 00:12:25,054

keeping your harness lean and deleting parts of it.

Cue 177 — 00:12:25,094 --> 00:12:29,294

The first one is mentioned is because the models start to bake this in a lot.

Cue 178 — 00:12:29,334 --> 00:12:32,934

So the verification part, the running tests, the creating tests,

Cue 179 — 00:12:32,974 --> 00:12:38,474

etc. the models actually do a lot that you don't realize they do automatically.

Cue 180 — 00:12:38,974 --> 00:12:43,214

And so with some customers, we've deleted a bunch of prompts that we thought were

Cue 181 — 00:12:43,254 --> 00:12:47,534

important, and then the model actually did better without those prompts,

Cue 182 — 00:12:47,574 --> 00:12:50,494

because the models now bake in a bunch of those instructions.

Cue 183 — 00:12:50,534 --> 00:12:54,734

So it's something to consider is when you're doing un-harnessing the models will

Cue 184 — 00:12:54,774 --> 00:12:58,654

make quite a bit in. And then a bunch of my time goes into this,

Cue 185 — 00:12:58,694 --> 00:13:04,334

which is quite confusing, but it's fewer contradictions when you have like a harness

Cue 186 — 00:13:04,374 --> 00:13:10,374

or a prompt that's quite lean, it's easier to manage, but it's unlikely as well that

Cue 187 — 00:13:10,414 --> 00:13:15,294

there's contradictions. One likely example that I find, and it's really painful for

Cue 188 — 00:13:15,334 --> 00:13:22,134

both me for for technical teams, etc., is that one example was in a file.

Cue 189 — 00:13:22,174 --> 00:13:27,854

It said run the test pipeline after every commit, and this test pipeline took like

Cue 190 — 00:13:27,894 --> 00:13:31,854

ten minutes to run, which is a completely other problem, but it took quite a while

Cue 191 — 00:13:31,894 --> 00:13:37,534

to run. Then there was a system prompt that said to only do it in certain scenarios,

Cue 192 — 00:13:37,574 --> 00:13:40,694

and then in certain skills for specific actions.

Cue 193 — 00:13:40,734 --> 00:13:47,294

It's said to not run tests. And so then you look at the transcripts of the model and

Cue 194 — 00:13:47,334 --> 00:13:50,414

it's super confused because it's like, which one should I do?

Cue 195 — 00:13:50,454 --> 00:13:54,974

And there's so much tokens that then go wasted as to like trying to understand what

Cue 196 — 00:13:55,014 --> 00:13:58,114

this ambiguous problem in this and big problem, what it should do.

Cue 197 — 00:13:58,454 --> 00:14:03,494

All this to say that part of the fix for this is just having a leaner harness in the

Cue 198 — 00:14:03,534 --> 00:14:08,294

first place. Like having just the concrete like high signal instructions that you

Cue 199 — 00:14:08,334 --> 00:14:12,138

need within the skills within system prompts within your CLAUDE.md.

Cue 200 — 00:14:12,178 --> 00:14:19,064

And I think people are often surprised as to how lean CLAUDE.md files are

Cue 201 — 00:14:19,104 --> 00:14:23,754

within Anthropic, really defining just what needs to be there.

Cue 202 — 00:14:26,174 --> 00:14:29,214

And here are a couple of examples of how you can go about doing that.

Cue 203 — 00:14:29,254 --> 00:14:35,134

There's a couple of blog posts, but part of it was defining what it used to be that

Cue 204 — 00:14:35,174 --> 00:14:39,374

you have to define how to do things, and it's a bit of a movement towards how what

Cue 205 — 00:14:39,414 --> 00:14:46,374

to do. There used to be a lot of emphasis on examples and defining exactly what the

Cue 206 — 00:14:46,414 --> 00:14:51,294

model should return. One thing that's interesting is is a very big pattern.

Cue 207 — 00:14:51,334 --> 00:14:55,774

To put an example in your system prompt, like the model gets something and this is

Cue 208 — 00:14:55,814 --> 00:15:00,254

what you should return. Examples can be a bit tricky because they are very useful in

Cue 209 — 00:15:00,294 --> 00:15:03,614

kind of telling the model how to do things, but by giving an example,

Cue 210 — 00:15:03,654 --> 00:15:08,654

sometimes you're biasing the model to only be replying in a specific way or do

Cue 211 — 00:15:08,694 --> 00:15:12,054

something as a result. And so all this to say is, yeah, again,

Cue 212 — 00:15:12,094 --> 00:15:15,094

the leanness of the harness and really keeping what what is needed.

Cue 213 — 00:15:15,134 --> 00:15:19,754

There was a push before as well of loading a bunch of context up front.

Cue 214 — 00:15:19,974 --> 00:15:26,154

So it used to be that like when you use a coding agent or coding CLI,

Cue 215 — 00:15:26,254 --> 00:15:29,494

it would inject, I don't know, files or content up front.

Cue 216 — 00:15:29,534 --> 00:15:32,014

There's a movement towards progressive disclosure.

Cue 217 — 00:15:32,054 --> 00:15:36,534

So the models are very smart. And so allowing them to kind of explore the file base

Cue 218 — 00:15:36,574 --> 00:15:39,014

or file system and find it for themselves.

Cue 219 — 00:15:39,054 --> 00:15:40,234

So there's a couple.

Cue 220 — 00:15:44,634 --> 00:15:50,154

There we go. There's a couple of these patterns that I vouch for for you to look at.

Cue 221 — 00:15:50,194 --> 00:15:52,814

But the common theme is simplification.

Cue 222 — 00:15:52,874 --> 00:15:57,014

Even the models, when you ask it to do something will tend to add.

Cue 223 — 00:15:57,074 --> 00:15:59,954

And part of our roles is also to simplify and keep things high.

Cue 224 — 00:15:59,994 --> 00:16:04,074

Signal have a complete command over what the right instructions are,

Cue 225 — 00:16:04,114 --> 00:16:08,834

and it's the best way for us to really understand and and promote the best

Cue 226 — 00:16:08,874 --> 00:16:09,814

practices.

Cue 238 — 00:16:52,081 --> 00:16:57,581

I've been talking about the ways that, like going towards a new model can negatively

Cue 239 — 00:16:57,721 --> 00:17:01,601

affect things, and how to optimize evals and harness all this to say,

Cue 240 — 00:17:01,641 --> 00:17:05,321

like typically it does mean more intelligence, kind of better value in better use

Cue 241 — 00:17:05,361 --> 00:17:09,161

case. I mean, this is on a compaction eval as you go progressing across the

Cue 242 — 00:17:09,201 --> 00:17:12,641

different models, it's the performance increased.

Cue 243 — 00:17:12,681 --> 00:17:17,841

Same with like BrowseComp. But if you are in the cases in which you see,

Cue 244 — 00:17:17,881 --> 00:17:24,381

hey, new model has released and my use case hasn't seen an increase in performance,

Cue 245 — 00:17:24,601 --> 00:17:31,181

it is something to question. And as creating code becomes really cheap,

Cue 246 — 00:17:31,641 --> 00:17:35,961

having that leanness within what you built and the instructions still becomes ever

Cue 247 — 00:17:36,001 --> 00:17:37,141

more important.

Cue 248 — 00:17:39,761 --> 00:17:40,561

So.

Cue 249 — 00:17:42,401 --> 00:17:47,881

Before kind of coming a little bit more to a close, this is a bit of the loop that I

Cue 250 — 00:17:47,921 --> 00:17:51,061

start to really promote with a bunch of this customers.

Cue 251 — 00:17:52,001 --> 00:17:58,161

I'm calling it the SDLC, but it's it's about how do you just not just add towards

Cue 252 — 00:17:58,201 --> 00:18:02,081

your harness your instructions, your skills, but what is a way that you can actually

Cue 253 — 00:18:02,121 --> 00:18:07,721

like very quickly have a bit of a loop where you can start to really keep what's

Cue 254 — 00:18:07,761 --> 00:18:12,161

high signal. And just like the Claude Code team in cutting out 80% of the system

Cue 255 — 00:18:12,201 --> 00:18:16,721

prompt and improving performance, you really iterate as a scientist to keep what

Cue 256 — 00:18:16,761 --> 00:18:19,101

really matters, and you keep experimenting.

Cue 257 — 00:18:19,201 --> 00:18:21,881

So this can look as, first of all, un-harnessing.

Cue 258 — 00:18:21,921 --> 00:18:23,841

I gave the example of like verification.

Cue 259 — 00:18:23,881 --> 00:18:27,361

Let's say that you you can do this yourself or give it to Claude or your agent,

Cue 260 — 00:18:27,401 --> 00:18:32,781

etc. you say remove all of the verification steps within my system prompts,

Cue 261 — 00:18:33,361 --> 00:18:36,601

then you're baseline. You can run your evals, try on different efforts,

Cue 262 — 00:18:36,641 --> 00:18:42,041

etc. you then see okay it actually increased performance, reduce costs,

Cue 263 — 00:18:42,081 --> 00:18:46,721

reduce latency. But there's one edge case for a couple edge cases that are missing.

Cue 264 — 00:18:46,761 --> 00:18:51,321

So then you can harness and then you can ship that and continue that within the loop

Cue 265 — 00:18:51,361 --> 00:18:57,041

using your evals to reassess. And what's interesting about this loop is you can

Cue 266 — 00:18:57,081 --> 00:19:02,361

actually just get Claude to really do this on a kind of experimental basis with some

Cue 267 — 00:19:02,401 --> 00:19:04,961

customers we've run. I'm not sure.

Cue 268 — 00:19:05,001 --> 00:19:09,561

I'm guessing folks have run kind of /goal or these kind of like self-improvement

Cue 269 — 00:19:09,601 --> 00:19:14,761

loops. It's basically you give Claude a command, an outcome and then it iterates and

Cue 270 — 00:19:14,801 --> 00:19:17,321

creates experiments until it optimizes on that.

Cue 271 — 00:19:17,361 --> 00:19:18,601

This is a really good example.

Cue 272 — 00:19:18,641 --> 00:19:23,121

So with a customer we said what is the most lean and high impact version of your

Cue 273 — 00:19:23,161 --> 00:19:28,621

harness look like. And it went through the steps of removing pieces re-baselining,

Cue 274 — 00:19:28,661 --> 00:19:31,901

re-harnessing shipping and then continuing within this loop.

Cue 275 — 00:19:32,441 --> 00:19:39,121

And with that, I've seen in the order of like a 40% reduction in cost for some use

Cue 276 — 00:19:39,161 --> 00:19:42,241

cases where there was too much verification, where there was bloat in certain

Cue 277 — 00:19:42,281 --> 00:19:46,081

places, etc.. And all this to say, it's not just about adding,

Cue 278 — 00:19:46,121 --> 00:19:51,401

it's also about just how do you have a process within your team that you ensure you

Cue 279 — 00:19:51,441 --> 00:19:55,081

have the most high signal version of the instructions you're trying to give,

Cue 280 — 00:19:55,121 --> 00:19:59,661

where code is just a way to to be able to to address those instructions.

Cue 281 — 00:20:02,581 --> 00:20:06,761

The other piece that we've talked quite a bit about, models and harnesses,

Cue 282 — 00:20:07,141 --> 00:20:12,421

I work a lot with the folks that run the evals within each AI native startup,

Cue 283 — 00:20:12,461 --> 00:20:14,841

and some of the fastest growing and leading ones.

Cue 284 — 00:20:16,221 --> 00:20:21,861

I think it's really underappreciated how important the person running evals is

Cue 285 — 00:20:21,901 --> 00:20:25,161

within a technical team that's running an AI use case.

Cue 286 — 00:20:25,421 --> 00:20:29,461

It's not just someone who's running the equivalent of like a integration test or a

Cue 287 — 00:20:29,501 --> 00:20:35,021

unit test or an end to end test, this person is defining what good looks like in an

Cue 288 — 00:20:35,061 --> 00:20:39,621

era where intelligence scales and constantly almost like as a product manager,

Cue 289 — 00:20:39,661 --> 00:20:45,001

defining, okay, where are the model capabilities now for our use case?

Cue 290 — 00:20:45,141 --> 00:20:48,501

Where should it go next? What is it struggling with?

Cue 291 — 00:20:48,541 --> 00:20:53,541

What can be improved and having that overlap in terms of capabilities with what is

Cue 292 — 00:20:53,581 --> 00:20:57,121

the highest ROI and value pieces for our end customers?

Cue 293 — 00:20:57,301 --> 00:20:58,741

And so looking at your person.

Cue 294 — 00:20:58,781 --> 00:21:03,021

Running evals is not just someone who's like managing test coverage,

Cue 295 — 00:21:03,061 --> 00:21:08,301

but is almost like an AI product manager managing what good looks like and ensuring

Cue 296 — 00:21:08,341 --> 00:21:11,261

the user value and capability scale.

Cue 297 — 00:21:11,301 --> 00:21:16,581

With each model release becomes huge, and some of the main recommendations are with

Cue 298 — 00:21:16,621 --> 00:21:21,341

evals starting with a small amount of like really high grade,

Cue 299 — 00:21:21,381 --> 00:21:23,201

almost like manual,

Cue 300 — 00:21:24,421 --> 00:21:27,141

cases within your eval that you really understand.

Cue 301 — 00:21:27,181 --> 00:21:31,421

And then using that to really scale it, you want to grade the outcome and not just

Cue 302 — 00:21:31,461 --> 00:21:35,561

the steps in between, obviously track the cost and the latency.

Cue 303 — 00:21:35,741 --> 00:21:40,661

But the reason this eval person is also very important is because it's not just

Cue 304 — 00:21:40,701 --> 00:21:44,081

about looking at the number, it's also reading the transcripts.

Cue 305 — 00:21:44,461 --> 00:21:49,361

And you often see that some of the really hard use cases are impossible.

Cue 306 — 00:21:49,461 --> 00:21:54,321

And you only see that by seeing the model venting within its transcripts and logs.

Cue 307 — 00:21:54,781 --> 00:21:59,741

And this person becomes so vital in being able to really iterate the perfect version

Cue 308 — 00:21:59,781 --> 00:22:03,561

of what an evaluation system for your AI use case looks like.

Cue 309 — 00:22:06,961 --> 00:22:09,861

So we've touched a lot about these changes.

Cue 310 — 00:22:10,241 --> 00:22:13,921

There's models that have a bigger change in ones that have a smaller one.

Cue 311 — 00:22:13,961 --> 00:22:19,481

That change from an Opus 5 to 5.5 hopefully was quite pleasant to folks,

Cue 312 — 00:22:19,521 --> 00:22:20,921

even internally. Before release.

Cue 313 — 00:22:20,961 --> 00:22:23,961

It was one of my favorite models in terms of its token efficiency,

Cue 314 — 00:22:24,001 --> 00:22:29,121

etc. we see this as a big win, but this is a model that, with your existing Opus 5

Cue 315 — 00:22:29,161 --> 00:22:32,001

prompts, should perform quite well without the changes.

Cue 316 — 00:22:32,041 --> 00:22:35,241

There's models that will have a big difference, other ones that will have a smaller

Cue 317 — 00:22:35,281 --> 00:22:38,761

difference within patterns. It's obviously part of my job personally,

Cue 318 — 00:22:38,801 --> 00:22:43,261

but also community generally to make that change as as easy as possible.

Cue 319 — 00:22:44,601 --> 00:22:49,521

But that's also a piece it's you shouldn't expect the same exact changes per per

Cue 320 — 00:22:49,561 --> 00:22:50,661

model release.

Cue 321 — 00:22:52,841 --> 00:22:56,541

And I wanted to leave it off with three questions.

Cue 322 — 00:22:56,921 --> 00:23:00,841

Hopefully that un-harnessing SDLC loop is an interesting one,

Cue 323 — 00:23:00,881 --> 00:23:04,961

but there's typically such a focus on adding the.

Cue 324 — 00:23:05,001 --> 00:23:09,343

The three questions that I'd like to ask technical teams is how efficiently can you

Cue 325 — 00:23:09,383 --> 00:23:12,541

change a harness? Can you test a harness change?

Cue 326 — 00:23:12,921 --> 00:23:18,821

If you were to like remove 80% of your system prompt like the Claude Code team did,

Cue 327 — 00:23:19,201 --> 00:23:21,401

how easy would you be able to test that?

Cue 328 — 00:23:21,441 --> 00:23:24,121

Because there's a lot of gains and flexibility that then gives you,

Cue 329 — 00:23:24,161 --> 00:23:27,581

if you start off with a leaner code base within your harness.

Cue 330 — 00:23:28,601 --> 00:23:31,641

And this often means also making your harness in a more modular way.

Cue 331 — 00:23:31,681 --> 00:23:33,661

So it is easier to do those changes.

Cue 332 — 00:23:34,841 --> 00:23:39,241

Which parts of the harness do you see as temporary as intelligence scales?

Cue 333 — 00:23:39,281 --> 00:23:43,121

Are there parts that you think are kind of crutches just because the model

Cue 334 — 00:23:43,161 --> 00:23:49,881

intelligence isn't there? Or as this use case scales, it can start to take on higher

Cue 335 — 00:23:49,921 --> 00:23:55,161

capabilities. And then last but not least, like not just looking at the numbers,

Cue 336 — 00:23:55,201 --> 00:23:59,521

but also just like really truly understanding on that eval situation.

Cue 337 — 00:23:59,561 --> 00:24:03,501

What are the model transcripts saying about your eval and your harness?

Cue 338 — 00:24:04,481 --> 00:24:08,001

There's a couple customer that I saw that did something quite interesting,

Cue 339 — 00:24:08,041 --> 00:24:12,701

which was they introduced an ability for the models to vent.

Cue 340 — 00:24:13,601 --> 00:24:17,001

So as the model goes into acting on use cases in production,

Cue 341 — 00:24:17,041 --> 00:24:20,081

it starts being like, hey, this was annoying, etc.

Cue 342 — 00:24:20,121 --> 00:24:25,241

and it's actually really interesting to be able to have that type of feedback,

Cue 343 — 00:24:25,281 --> 00:24:28,681

because these models are very intelligent and probably have good ideas that then

Cue 344 — 00:24:28,721 --> 00:24:31,501

contribute towards your system, your harness, etc..

Cue 345 — 00:24:32,041 --> 00:24:35,841

So yeah, really understanding what are the transcripts saying and not just what the

Cue 346 — 00:24:35,881 --> 00:24:36,941

numbers are saying.

Cue 347 — 00:24:38,321 --> 00:24:40,681

I just want to say that that's the end of my slides.

Cue 348 — 00:24:40,721 --> 00:24:42,841

Thank you so much for listening to me before lunch.

Cue 349 — 00:24:42,881 --> 00:24:45,001

Hopefully this was this was interesting.

Cue 350 — 00:24:45,041 --> 00:24:47,501

And yeah. Thank you.

Cue 351 — 00:24:53,081 --> 00:24:55,321

Would love to take any questions.

Cue 352 — 00:24:55,361 --> 00:25:00,761

And hopefully I think part of the paradigm shift is looking at smaller harnesses and

Cue 353 — 00:25:00,801 --> 00:25:04,161

leaner ones, not just bigger ones, and then looking at yourself if you're working on

Cue 354 — 00:25:04,201 --> 00:25:09,041

harnesses and evals as scientists and not just engineers, but I'll take any

Cue 355 — 00:25:09,081 --> 00:25:11,281

questions. We have time for a couple of questions.

Cue 356 — 00:25:11,321 --> 00:25:14,521

Please raise your hand so I can bring you a mic, I promise.

Cue 357 — 00:25:14,561 --> 00:25:16,181

You're not as loud as you think.

Cue 358 — 00:25:17,521 --> 00:25:22,881

Hey, how do you avoid eval hacking in in a in a setup like this,

Cue 359 — 00:25:22,921 --> 00:25:26,301

when you remove 80% of your of your prompts?

Cue 360 — 00:25:28,281 --> 00:25:34,401

I mean, even when you don't replace 80% of the prompt, eval hacking becomes one of

Cue 361 — 00:25:34,441 --> 00:25:38,321

the most important things. I think it's also the importance of the person.

Cue 362 — 00:25:38,361 --> 00:25:45,161

Managing evals has increased exponentially over time, because as the models become

Cue 363 — 00:25:45,201 --> 00:25:49,481

more and more intelligent, it becomes harder to stop things like eval hacking.

Cue 364 — 00:25:49,521 --> 00:25:55,701

And so, yeah, just to say, even without that, that becomes a very important part.

Cue 365 — 00:25:55,801 --> 00:25:59,641

I think the only real way that we've seen is obviously there's a bunch of sandboxing

Cue 366 — 00:25:59,681 --> 00:26:02,761

things that you can start to look at, like you used to just run the evals on your

Cue 367 — 00:26:02,801 --> 00:26:07,281

computer and then looking at like a secure sandbox that you can actually run them

Cue 368 — 00:26:07,321 --> 00:26:11,561

in. That's kind of airtight, that you start to look at the transcripts and seeing,

Cue 369 — 00:26:11,601 --> 00:26:13,621

okay, is it cheating in certain ways?

Cue 370 — 00:26:14,201 --> 00:26:18,361

All this to say is having a higher degree of skepticism when you run evals is to the

Cue 371 — 00:26:18,401 --> 00:26:23,201

numbers and looking a little bit more about the how and the transcripts instead to

Cue 372 — 00:26:23,241 --> 00:26:25,141

try and see what's what's valid or not.

Cue 373 — 00:26:25,281 --> 00:26:27,021

Basically, it's a continuous effort.

Cue 374 — 00:26:28,121 --> 00:26:32,241

So if you just hit the goal, like it would end up with something that was like over

Cue 375 — 00:26:32,281 --> 00:26:34,501

optimized for whatever evals you had.

Cue 376 — 00:26:35,161 --> 00:26:38,601

I think also the folks. So it used to be sometimes like there wasn't someone

Cue 377 — 00:26:38,641 --> 00:26:42,561

dedicated on evals and technical teams, then someone became dedicated,

Cue 378 — 00:26:42,601 --> 00:26:47,321

then some people started become dedicated towards just the sandboxing of the eval,

Cue 379 — 00:26:47,361 --> 00:26:49,781

and then someone running the use case in the eval.

Cue 380 — 00:26:50,001 --> 00:26:53,401

And so I think, yeah, just as this is scaled, there starts to be more

Cue 381 — 00:26:53,441 --> 00:26:57,501

responsibilities to making sure that you're not reward hacking,

Cue 382 — 00:26:57,681 --> 00:27:00,321

which tends to sometimes be the easiest thing for the models,

Cue 383 — 00:27:00,361 --> 00:27:04,321

hopefully with more aligned models, that also starts to be solved.

Cue 384 — 00:27:04,361 --> 00:27:09,321

But but there is a lot of efforts going into just sandboxing the evals and making

Cue 385 — 00:27:09,361 --> 00:27:13,981

sure that, that there's not another path into than to complete it.

Cue 386 — 00:27:14,321 --> 00:27:15,181

Thanks.

Cue 387 — 00:27:17,041 --> 00:27:19,681

You kind of answered a little bit, but I was wondering.

Cue 388 — 00:27:19,721 --> 00:27:23,481

Sorry. I'm sort of here. If if you could give some insight in what you're doing with

Cue 389 — 00:27:23,521 --> 00:27:28,161

customers to help them avoid overfitting, whether and how they know when is a good

Cue 390 — 00:27:28,201 --> 00:27:30,061

time to throw away bad evals.

Cue 391 — 00:27:33,286 --> 00:27:37,606

I think it it it usually depends on the exact evals.

Cue 392 — 00:27:37,646 --> 00:27:42,046

So like typically it would be a matter of like looking into the transcripts

Cue 393 — 00:27:42,086 --> 00:27:48,046

specifically like in terms of like a thumb in the like a rule of thumb generally,

Cue 394 — 00:27:48,086 --> 00:27:52,206

it would be that we try to kind of have evals, like the sweet spot is that some of

Cue 395 — 00:27:52,246 --> 00:27:56,886

the best models are achieving like 60 to 80% performance, but I wouldn't over index

Cue 396 — 00:27:56,926 --> 00:27:59,206

on that. But that's typically one.

Cue 397 — 00:27:59,246 --> 00:28:04,326

And then ones where we see that consistently with newer models and higher efforts,

Cue 398 — 00:28:04,366 --> 00:28:10,246

especially if there isn't kind of a nice at least somewhat slope in terms of like

Cue 399 — 00:28:10,286 --> 00:28:15,006

performance. With more intelligence, you start to have to ask real reasons as to why

Cue 400 — 00:28:15,046 --> 00:28:18,346

that isn't scaling. So with some teams, I saw, for example,

Cue 401 — 00:28:19,806 --> 00:28:24,966

new Frontier models were released and the effort curve was like completely flat.

Cue 402 — 00:28:25,006 --> 00:28:30,886

But all of the frontier models started to being 1% away from each other at the 50%

Cue 403 — 00:28:30,926 --> 00:28:38,006

mark. And so you have to start asking like, why are they not able to do the top 50%?

Cue 404 — 00:28:38,046 --> 00:28:42,126

And it turns out that a couple of those are like either impossible problems or the

Cue 405 — 00:28:42,166 --> 00:28:46,686

models intelligence actually were able to see that something that was previously

Cue 406 — 00:28:46,726 --> 00:28:49,706

considered is like a right answer is not actually a right answer.

Cue 407 — 00:28:50,046 --> 00:28:52,806

And so, so, yeah, there's a couple of things.

Cue 408 — 00:28:52,846 --> 00:28:57,926

I think there's no way to go around it apart from like exploring into the details,

Cue 409 — 00:28:57,966 --> 00:29:02,046

but typically typically having frontier models in the 60 to 80% mark.

Cue 410 — 00:29:02,086 --> 00:29:06,306

And then some type of like nice Pareto is like effort and model scale.

Cue 411 — 00:29:09,244 --> 00:29:15,224

Thank you. You mentioned a few times like this a scientist and an engineer.

Cue 412 — 00:29:15,964 --> 00:29:19,004

Building evals. I'm just like, kind of curious not to get too philosophical.

Cue 413 — 00:29:19,044 --> 00:29:22,704

Like what? What are the traits that actually, are important here?

Cue 414 — 00:29:24,684 --> 00:29:29,444

It's a it's a good question. I think it's for me, it's a somewhat of an analogy.

Cue 415 — 00:29:29,484 --> 00:29:32,564

I'd say I'd sit more on the engineering side than the science side,

Cue 416 — 00:29:32,604 --> 00:29:36,324

but I think, so traditionally been a software engineer and CTO,

Cue 417 — 00:29:36,364 --> 00:29:42,084

etc.. I think this kind of work has really pushed me towards being more on the

Cue 418 — 00:29:42,124 --> 00:29:43,664

science side in the sense of,

Cue 419 — 00:29:44,964 --> 00:29:48,044

I guess on the engineering side, there's always been scientific pieces where maybe

Cue 420 — 00:29:48,084 --> 00:29:51,004

you're optimizing for latency, for performance, etc.

Cue 421 — 00:29:51,044 --> 00:29:55,024

but I think in terms of when you're removing 80% of your system prompt,

Cue 422 — 00:29:55,084 --> 00:30:00,484

there has to be like quite a decoupling from what you have now and a real focus on

Cue 423 — 00:30:00,524 --> 00:30:03,944

like the outcome and the exploration and the learnings towards getting there.

Cue 424 — 00:30:04,404 --> 00:30:06,844

I think engineering has always had like parts of a scientist,

Cue 425 — 00:30:06,884 --> 00:30:09,144

and so maybe it's a little bit more of a spectrum.

Cue 426 — 00:30:09,204 --> 00:30:15,964

But as an engineer, I think what's been cool is I used to have to develop

Cue 427 — 00:30:16,004 --> 00:30:19,204

everything by hand or websites, etc.

Cue 428 — 00:30:19,244 --> 00:30:22,804

now I treat everything I build as an experiment towards like some type of bigger

Cue 429 — 00:30:22,844 --> 00:30:25,244

outcome instead of the main and the only thing.

Cue 430 — 00:30:25,284 --> 00:30:29,204

And so seeing everything is type, some type of iteration of constant improvement I

Cue 431 — 00:30:29,244 --> 00:30:31,804

think maybe has been the mindset mindset shift.

Cue 432 — 00:30:31,844 --> 00:30:34,524

I think it's a spectrum. So there are different degrees.

Cue 433 — 00:30:34,564 --> 00:30:36,784

But that's been my experience so far.

skills

talk-carmont-evolving-agent-harnesses

tile.json