AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
86%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Nick Carmont · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=IiAvs9hiQZ0
Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.
For topic navigation, see outline.md.
Cue 1 — 00:00:01,054 --> 00:00:04,174
Hi everyone, and thanks so much for having me.
Cue 2 — 00:00:04,214 --> 00:00:09,034
And huge thank you to AI DevCon and Tessl for putting this all together.
Cue 3 — 00:00:09,784 --> 00:00:13,234
I am one of those people that gets very excited about model releases,
Cue 4 — 00:00:13,494 --> 00:00:16,594
so I think I am within one of within that bucket.
Cue 5 — 00:00:17,174 --> 00:00:22,906
And the topic for today is building a harness that keeps up with those latest models
Cue 6 — 00:00:23,094 --> 00:00:27,894
specifically around experiences in the un-harnessing SDLC loop.
Cue 7 — 00:00:27,934 --> 00:00:32,734
In short, it's not just about adding stuff to your harness and keep on adding
Cue 8 — 00:00:32,774 --> 00:00:37,774
instructions, but it's also about how do you ensure that you keep the instructions
Cue 9 — 00:00:37,814 --> 00:00:41,974
and the harness around the model quite lean and quite high signal.
Cue 10 — 00:00:42,014 --> 00:00:43,814
And how do you do that continuously?
Cue 11 — 00:00:43,854 --> 00:00:45,934
So I know this is the talk before lunch.
Cue 12 — 00:00:45,974 --> 00:00:47,814
Hopefully you're not too hungry.
Cue 13 — 00:00:47,854 --> 00:00:50,054
And if you are hungry, you're hungry.
Cue 14 — 00:00:50,094 --> 00:00:54,014
To learn more about how to optimize your harness with with new models.
Cue 15 — 00:00:54,054 --> 00:00:57,094
So I'm Nick, member of technical staff on our Applied AI team.
Cue 16 — 00:00:57,134 --> 00:01:00,734
I get the privilege of sitting between our research team, our model team,
Cue 17 — 00:01:00,774 --> 00:01:04,894
and some of our most strategic customers, particularly startups,
Cue 18 — 00:01:04,934 --> 00:01:09,814
and being on a hands on technical level, helping them with every model release.
Cue 19 — 00:01:09,854 --> 00:01:15,614
How do we make sure that this model and your harness and evals best benefit from
Cue 20 — 00:01:15,654 --> 00:01:20,134
that intelligence, and how, when that is the case, can we help that for your use
Cue 21 — 00:01:20,174 --> 00:01:23,654
case? And when the model has some feedback that's needed, how can I channel that.
Cue 22 — 00:01:23,694 --> 00:01:27,794
Best to to the model research team in order to to optimize that loop.
Cue 23 — 00:01:29,334 --> 00:01:33,354
So this is the kind of loop that I go through.
Cue 24 — 00:01:33,494 --> 00:01:37,374
I help AI native startups optimize their harness their evals with each model
Cue 25 — 00:01:37,414 --> 00:01:40,774
release, and any time that the model requires feedback.
Cue 26 — 00:01:40,814 --> 00:01:44,094
I then work with our research team in order to kind of channel that back.
Cue 27 — 00:01:44,134 --> 00:01:48,694
And recently, for me personally, it's felt like almost like a 100 meter Olympic
Cue 28 — 00:01:48,734 --> 00:01:52,814
sprints. We had a Opus 5.5 release not too long ago.
Cue 29 — 00:01:52,854 --> 00:01:55,894
Then we had Sonnet 5.5 and Haiku 5.5.
Cue 30 — 00:01:55,934 --> 00:02:00,094
So it's been a busy period. But this is typically how it works,
Cue 31 — 00:02:00,134 --> 00:02:04,934
where a new model ships. I then work with customers to rerun their evals,
Cue 32 — 00:02:04,974 --> 00:02:09,574
see how that model performs on on their use case, and then it's a matter of seeing
Cue 33 — 00:02:09,614 --> 00:02:12,294
okay. Should we adjust the model?
Cue 34 — 00:02:12,334 --> 00:02:14,614
Should I kind of bring that feedback back?
Cue 35 — 00:02:14,654 --> 00:02:18,494
Is there anything in the harness that requires optimization and is there anything in
Cue 36 — 00:02:18,534 --> 00:02:22,934
the eval? Let's say that it's starting to saturate that needs adjustment.
Cue 37 — 00:02:22,974 --> 00:02:28,094
And then helping those customers ship that new model, potentially towards millions
Cue 38 — 00:02:28,134 --> 00:02:31,774
of customers to be able to, as soon as possible, benefit from those gains.
Cue 39 — 00:02:31,814 --> 00:02:34,594
So with Opus 5.5, that was quite nice.
Cue 40 — 00:02:34,694 --> 00:02:39,534
Customers were able to receive an increase in intelligence at a lower cost because
Cue 41 — 00:02:39,574 --> 00:02:44,694
of its token efficiencies. But this is a little bit of of the loop that I typically
Cue 42 — 00:02:44,734 --> 00:02:47,554
go through and help some of our, our customers on.
Cue 43 — 00:02:49,774 --> 00:02:51,974
This sometimes feels like Narnia.
Cue 44 — 00:02:52,014 --> 00:02:55,839
And the reason is, I don't know how many have watched Narnia:
Cue 45 — 00:02:55,879 --> 00:02:58,034
The Lion, the Witch and the Wardrobe.
Cue 46 — 00:02:58,294 --> 00:03:01,574
My life revolves around, well, sorry.
Cue 47 — 00:03:01,614 --> 00:03:03,934
The three topics today I'll talk about is the model, the harness,
Cue 48 — 00:03:03,974 --> 00:03:08,934
and the eval. Why? Claude Code cut its system prompt over 80%.
Cue 49 — 00:03:08,974 --> 00:03:14,494
And a bit about the loop. It's not just about adding instructions and skills,
Cue 50 — 00:03:14,534 --> 00:03:18,374
but if you're working on a software factory on loops, a lot of the fundamentals end
Cue 51 — 00:03:18,414 --> 00:03:23,334
up compounding. And so having the fundamentals right and lean becomes ever more
Cue 52 — 00:03:23,374 --> 00:03:25,514
important as you go scaling things.
Cue 53 — 00:03:25,854 --> 00:03:29,774
So I said this tends to be a bit like Narnia.
Cue 54 — 00:03:29,814 --> 00:03:33,014
So instead of The Lion, the Witch and the Wardrobe, I work with the model,
Cue 55 — 00:03:33,054 --> 00:03:37,854
the harness or the eval. So every model that drops is understanding okay,
Cue 56 — 00:03:37,894 --> 00:03:42,494
is it the model that isn't performing correctly on this use case or requires
Cue 57 — 00:03:42,534 --> 00:03:47,654
adjustment? Is it the harness which is restricting the model from being able to do
Cue 58 — 00:03:47,694 --> 00:03:49,534
something in the specific use case?
Cue 59 — 00:03:49,574 --> 00:03:55,254
Or is there things within the eval that are, for example, saturated or there might
Cue 60 — 00:03:55,294 --> 00:03:57,534
be even in this actually happens quite often.
Cue 61 — 00:03:57,574 --> 00:04:01,414
There are specific use cases in which it's actually an impossible task.
Cue 62 — 00:04:01,454 --> 00:04:05,094
So I've had some startup customers be like, oh, even the frontier models are like
Cue 63 — 00:04:05,134 --> 00:04:12,054
only reaching 50%. And then the the maximum anyone can achieve on that task is 50%,
Cue 64 — 00:04:12,094 --> 00:04:15,234
because the rest of the 50% are are impossible tasks.
Cue 65 — 00:04:15,454 --> 00:04:20,494
And so it's a it's a bit of this loop of optimization when you get a new model in
Cue 66 — 00:04:20,534 --> 00:04:25,054
order to how to best optimize it so that the intelligence flows towards the end
Cue 67 — 00:04:25,094 --> 00:04:29,354
customer, and they get the best experience and the highest value from this.
Cue 68 — 00:04:29,814 --> 00:04:34,274
And there tends to be some telltale signs as to which is the problem.
Cue 69 — 00:04:34,414 --> 00:04:37,454
One way to think about it is let's say this is a driving exam,
Cue 70 — 00:04:37,494 --> 00:04:41,174
and I unfortunately had a lot of problems with driving exams and probably not the
Cue 71 — 00:04:41,214 --> 00:04:44,774
best driver. But let's treat the model as like the engine.
Cue 72 — 00:04:44,814 --> 00:04:49,774
The harness is like a car. The model in the harness like a driver you have like the
Cue 73 — 00:04:49,814 --> 00:04:53,654
instructions, you have the engine, you have like the intelligence and the eval is
Cue 74 — 00:04:53,694 --> 00:04:58,894
like the driving exam. And so what happens often is let's say that you switch in a
Cue 75 — 00:04:58,934 --> 00:05:02,194
new model and you can see that it performs better or worse.
Cue 76 — 00:05:02,294 --> 00:05:05,734
Some signs are okay on higher efforts or new models.
Cue 77 — 00:05:05,774 --> 00:05:09,694
We start to see potentially the model increasing in performance or reducing in
Cue 78 — 00:05:09,734 --> 00:05:13,214
performance. And so we can start to see, okay, as we increase the effort,
Cue 79 — 00:05:13,254 --> 00:05:20,174
let's say how much acceleration you put on the engine, is the car going faster,
Cue 80 — 00:05:20,214 --> 00:05:22,394
is it improving in performance etc..
Cue 81 — 00:05:22,854 --> 00:05:28,916
The harness around the car, one of the telltale signs of the harness being what's
Cue 82 — 00:05:28,956 --> 00:05:35,454
kind of holding back potentially this use case is, if you compare a more
Cue 83 — 00:05:35,494 --> 00:05:39,654
stripped-back version of the harness versus your current harness and you see
Cue 84 — 00:05:39,694 --> 00:05:43,854
increases in performance, it can start to be a sign that the harness is actually
Cue 85 — 00:05:43,894 --> 00:05:48,014
what's holding things back. And so I'll go over some examples later that I've seen.
Cue 86 — 00:05:48,054 --> 00:05:53,734
But often some teams have built this really elaborate harness with tons of
Cue 87 — 00:05:53,774 --> 00:05:58,774
instructions, tons of steps, and a lot of work has been done in increasing and
Cue 88 — 00:05:58,814 --> 00:06:04,594
making this like bigger in terms of instructions, but not much has has been done.
Cue 89 — 00:06:05,094 --> 00:06:08,774
And how do we reduce this? How do we make this as high signal as possible and flow
Cue 90 — 00:06:08,814 --> 00:06:14,654
quite well. And so one sign starts to be, okay, let's actually run the model with a
Cue 91 — 00:06:14,694 --> 00:06:18,034
leaner harness or no harness versus the current one.
Cue 92 — 00:06:18,214 --> 00:06:21,654
And it starts to be a little bit more like treating this like a science experiment
Cue 93 — 00:06:21,694 --> 00:06:23,214
rather than an engineering one.
Cue 94 — 00:06:23,254 --> 00:06:27,414
How do we make sure that the harness that's selected is the most optimal for the use
Cue 95 — 00:06:27,454 --> 00:06:30,634
case? It's not just one that we decided to kind of put forward.
Cue 96 — 00:06:31,294 --> 00:06:35,574
And then the eval, you should never trust the eval kind of blindly.
Cue 97 — 00:06:35,614 --> 00:06:38,814
It's always about. So let's say this is a driving test.
Cue 98 — 00:06:38,854 --> 00:06:45,054
And part of the exam marks is that you have to go into a road which is closed
Cue 99 — 00:06:45,094 --> 00:06:50,494
temporarily, and so you can run your eval, and then no one's scoring very high on
Cue 100 — 00:06:50,534 --> 00:06:55,554
this eval. And the telltale sign is like, don't just look at the results.
Cue 101 — 00:06:55,654 --> 00:06:58,674
Actually look at what the harness and the model are telling you.
Cue 102 — 00:06:58,814 --> 00:07:03,294
Are they saying, hey, this is an impossible task because I can't go left because the
Cue 103 — 00:07:03,334 --> 00:07:08,034
road is closed, for example. And this happens a lot more than you'd expect.
Cue 104 — 00:07:08,454 --> 00:07:12,634
A lot of and as they get harder and harder and start to saturate,
Cue 105 — 00:07:12,974 --> 00:07:16,854
it actually becomes harder to really look at like what good looks like and the
Cue 106 — 00:07:16,894 --> 00:07:19,274
things that only frontier models can achieve.
Cue 107 — 00:07:20,374 --> 00:07:24,494
You have to be very specific and concrete and really close to the problem to really
Cue 108 — 00:07:24,534 --> 00:07:29,374
define a good eval. So those are the three things that I tend to help customers with
Cue 109 — 00:07:29,414 --> 00:07:32,754
and optimize in terms of model, harness and eval.
Cue 110 — 00:07:35,294 --> 00:07:39,814
And in terms of the model itself, it is true that every model can have a little bit
Cue 111 — 00:07:39,854 --> 00:07:42,154
of its own traits and personality.
Cue 112 — 00:07:42,294 --> 00:07:49,074
So one of the observations was with the switch from older models to Claude 5 models.
Cue 113 — 00:07:50,374 --> 00:07:55,874
A lot of the things that so if you worked with Sonnet 3.5 or Opus 4.5,
Cue 114 — 00:07:56,134 --> 00:07:59,054
you often had to tell it a lot how to do things.
Cue 115 — 00:07:59,094 --> 00:08:01,434
And apart from like what to do.
Cue 116 — 00:08:02,174 --> 00:08:07,174
And one of the good things of some of the latest models is a lot of the common
Cue 117 — 00:08:07,214 --> 00:08:11,694
things that you would tend to prompt that are kind of universal start to be baked
Cue 118 — 00:08:11,734 --> 00:08:17,414
into the model. And so I've spent maybe two months with customers really focused on
Cue 119 — 00:08:17,454 --> 00:08:20,194
system prompts and like over verification.
Cue 120 — 00:08:20,454 --> 00:08:25,364
You can't believe the amount of times in skills system prompts in MD.
Cue 121 — 00:08:25,454 --> 00:08:29,894
I had this all over my computer, which is like really verify absolutely everything
Cue 122 — 00:08:29,934 --> 00:08:32,474
you do, create tests and run tests.
Cue 123 — 00:08:32,654 --> 00:08:36,494
But the problem was that these same instructions were replicated across so many
Cue 124 — 00:08:36,534 --> 00:08:39,934
different steps, in so many different parts of like the instructions and system
Cue 125 — 00:08:39,974 --> 00:08:46,094
prompt. And the thing is, starting with Opus 5 and the rest of the family,
Cue 126 — 00:08:46,134 --> 00:08:50,054
verification is something that's very heavily built into the models instructions
Cue 127 — 00:08:50,094 --> 00:08:55,054
anyways, even if you don't tell it to verify, it will go about doing that.
Cue 128 — 00:08:55,094 --> 00:09:00,254
And so one of the consequences starts being that, if you just drop a model into an
Cue 129 — 00:09:00,294 --> 00:09:04,114
existing harness, there starts to be duplicated efforts.
Cue 130 — 00:09:04,454 --> 00:09:10,094
So I saw with some customers costs go up like 30% and then go down to like 0 or 5
Cue 131 — 00:09:10,134 --> 00:09:14,300
plus percent. Once that you really lean in and you have de-duplicated the
Cue 132 — 00:09:14,340 --> 00:09:16,114
instructions of like verification.
Cue 133 — 00:09:16,214 --> 00:09:21,274
All this to say, the models, as they grow more intelligent are not just,
Cue 134 — 00:09:21,974 --> 00:09:26,254
having differences in patterns of behaviors, but they're also tend to bake in more
Cue 135 — 00:09:26,294 --> 00:09:29,674
instructions that were otherwise in the system prompt.
Cue 136 — 00:09:30,094 --> 00:09:33,854
And this actually makes it cheaper because you're now not paying for certain tokens.
Cue 137 — 00:09:33,894 --> 00:09:36,954
It's already kind of built in to part of the behavior.
Cue 138 — 00:09:39,214 --> 00:09:42,334
So this is just a the same piece here.
Cue 139 — 00:09:42,374 --> 00:09:45,394
Here's an example of instructions to verify every file.
Cue 140 — 00:09:45,534 --> 00:09:48,434
And here just saying the exact instructions.
Cue 141 — 00:09:48,774 --> 00:09:53,434
The main point here being make sure to verify your verifications.
Cue 142 — 00:09:53,774 --> 00:09:59,634
And there are certain things that actually benefit from a lean model as the as the
Cue 143 — 00:09:59,934 --> 00:10:05,094
models get more intelligent. So that was a bit about the model,
Cue 144 — 00:10:05,134 --> 00:10:07,114
the harness and the eval.
Cue 145 — 00:10:09,094 --> 00:10:13,934
One quick one here is I used to be a startup CTO, and so I used to build websites
Cue 146 — 00:10:13,974 --> 00:10:17,594
from scratch and SVG animations, a bunch of embarrassing stuff now.
Cue 147 — 00:10:18,014 --> 00:10:21,334
And it's crazy to think to have coded from from scratch a bunch.
Cue 148 — 00:10:21,374 --> 00:10:25,134
But one of the things in software engineering is the concept of bit rot.
Cue 149 — 00:10:25,174 --> 00:10:29,574
That is, your code base gets bigger and older.
Cue 150 — 00:10:29,614 --> 00:10:32,834
There are certain parts that start to, for example, decay.
Cue 151 — 00:10:33,094 --> 00:10:38,394
And now that even in the previous talk, the cost of software is so cheap,
Cue 152 — 00:10:38,454 --> 00:10:42,694
the rate of bit decay happens a lot faster because there's a lot more code being
Cue 153 — 00:10:42,734 --> 00:10:46,894
built at once. And one thing to think about for about your harness,
Cue 154 — 00:10:46,934 --> 00:10:51,494
which is, yeah, everything around the AI model to power your use case is that it
Cue 155 — 00:10:51,534 --> 00:10:55,334
also has a shelf life, and you should see it as something dynamic and constantly
Cue 156 — 00:10:55,374 --> 00:10:59,234
improving, because the objective isn't to build a big harness,
Cue 157 — 00:10:59,294 --> 00:11:03,854
it's to have the model plus harness power, the best use case,
Cue 158 — 00:11:03,894 --> 00:11:05,814
and the highest value for the end user.
Cue 159 — 00:11:05,854 --> 00:11:09,634
And so you should look at this as a as a constant evolution in cycle.
Cue 160 — 00:11:11,734 --> 00:11:16,294
To give an example, this summer, the Claude Code team cut the system prompt by over
Cue 161 — 00:11:16,334 --> 00:11:20,614
80%. And that's quite a hard core thing to do because you're like,
Cue 162 — 00:11:20,654 --> 00:11:23,854
oh, how would that affect customers usage, etc..
Cue 163 — 00:11:23,894 --> 00:11:28,454
And they saw that this is one of the biggest benefits because as mentioned just now,
Cue 164 — 00:11:28,494 --> 00:11:31,814
the model is starting to bake in a lot of the best practices.
Cue 165 — 00:11:31,854 --> 00:11:33,394
And by
Cue 166 — 00:11:34,614 --> 00:11:40,514
and just keeping the highest signal 20% of instructions and tooling for the model,
Cue 167 — 00:11:40,774 --> 00:11:44,294
it's able to find the best solution itself for most of these problems.
Cue 168 — 00:11:44,334 --> 00:11:48,374
And all this to say, with a lot of startup customers, there is a find that just the
Cue 169 — 00:11:48,414 --> 00:11:52,694
leaner the harness, the more new models and new intelligence can actually benefit
Cue 170 — 00:11:52,734 --> 00:11:57,814
the use case. And it's kind of the equivalent of having being a manager and having
Cue 171 — 00:11:57,854 --> 00:12:03,194
an employee. And instead of telling it, micromanaging and saying how to do things
Cue 172 — 00:12:03,534 --> 00:12:07,814
super strictly, you allow this in very intelligent employee to really express and
Cue 173 — 00:12:07,854 --> 00:12:09,174
find the best way to do things.
Cue 174 — 00:12:09,214 --> 00:12:13,554
And really defining with high signal the what you want to achieve.
Cue 175 — 00:12:15,774 --> 00:12:22,254
And there's kind of two main theories as to why works un-harnessing is just kind of
Cue 176 — 00:12:22,294 --> 00:12:25,054
keeping your harness lean and deleting parts of it.
Cue 177 — 00:12:25,094 --> 00:12:29,294
The first one is mentioned is because the models start to bake this in a lot.
Cue 178 — 00:12:29,334 --> 00:12:32,934
So the verification part, the running tests, the creating tests,
Cue 179 — 00:12:32,974 --> 00:12:38,474
etc. the models actually do a lot that you don't realize they do automatically.
Cue 180 — 00:12:38,974 --> 00:12:43,214
And so with some customers, we've deleted a bunch of prompts that we thought were
Cue 181 — 00:12:43,254 --> 00:12:47,534
important, and then the model actually did better without those prompts,
Cue 182 — 00:12:47,574 --> 00:12:50,494
because the models now bake in a bunch of those instructions.
Cue 183 — 00:12:50,534 --> 00:12:54,734
So it's something to consider is when you're doing un-harnessing the models will
Cue 184 — 00:12:54,774 --> 00:12:58,654
make quite a bit in. And then a bunch of my time goes into this,
Cue 185 — 00:12:58,694 --> 00:13:04,334
which is quite confusing, but it's fewer contradictions when you have like a harness
Cue 186 — 00:13:04,374 --> 00:13:10,374
or a prompt that's quite lean, it's easier to manage, but it's unlikely as well that
Cue 187 — 00:13:10,414 --> 00:13:15,294
there's contradictions. One likely example that I find, and it's really painful for
Cue 188 — 00:13:15,334 --> 00:13:22,134
both me for for technical teams, etc., is that one example was in a file.
Cue 189 — 00:13:22,174 --> 00:13:27,854
It said run the test pipeline after every commit, and this test pipeline took like
Cue 190 — 00:13:27,894 --> 00:13:31,854
ten minutes to run, which is a completely other problem, but it took quite a while
Cue 191 — 00:13:31,894 --> 00:13:37,534
to run. Then there was a system prompt that said to only do it in certain scenarios,
Cue 192 — 00:13:37,574 --> 00:13:40,694
and then in certain skills for specific actions.
Cue 193 — 00:13:40,734 --> 00:13:47,294
It's said to not run tests. And so then you look at the transcripts of the model and
Cue 194 — 00:13:47,334 --> 00:13:50,414
it's super confused because it's like, which one should I do?
Cue 195 — 00:13:50,454 --> 00:13:54,974
And there's so much tokens that then go wasted as to like trying to understand what
Cue 196 — 00:13:55,014 --> 00:13:58,114
this ambiguous problem in this and big problem, what it should do.
Cue 197 — 00:13:58,454 --> 00:14:03,494
All this to say that part of the fix for this is just having a leaner harness in the
Cue 198 — 00:14:03,534 --> 00:14:08,294
first place. Like having just the concrete like high signal instructions that you
Cue 199 — 00:14:08,334 --> 00:14:12,138
need within the skills within system prompts within your CLAUDE.md.
Cue 200 — 00:14:12,178 --> 00:14:19,064
And I think people are often surprised as to how lean CLAUDE.md files are
Cue 201 — 00:14:19,104 --> 00:14:23,754
within Anthropic, really defining just what needs to be there.
Cue 202 — 00:14:26,174 --> 00:14:29,214
And here are a couple of examples of how you can go about doing that.
Cue 203 — 00:14:29,254 --> 00:14:35,134
There's a couple of blog posts, but part of it was defining what it used to be that
Cue 204 — 00:14:35,174 --> 00:14:39,374
you have to define how to do things, and it's a bit of a movement towards how what
Cue 205 — 00:14:39,414 --> 00:14:46,374
to do. There used to be a lot of emphasis on examples and defining exactly what the
Cue 206 — 00:14:46,414 --> 00:14:51,294
model should return. One thing that's interesting is is a very big pattern.
Cue 207 — 00:14:51,334 --> 00:14:55,774
To put an example in your system prompt, like the model gets something and this is
Cue 208 — 00:14:55,814 --> 00:15:00,254
what you should return. Examples can be a bit tricky because they are very useful in
Cue 209 — 00:15:00,294 --> 00:15:03,614
kind of telling the model how to do things, but by giving an example,
Cue 210 — 00:15:03,654 --> 00:15:08,654
sometimes you're biasing the model to only be replying in a specific way or do
Cue 211 — 00:15:08,694 --> 00:15:12,054
something as a result. And so all this to say is, yeah, again,
Cue 212 — 00:15:12,094 --> 00:15:15,094
the leanness of the harness and really keeping what what is needed.
Cue 213 — 00:15:15,134 --> 00:15:19,754
There was a push before as well of loading a bunch of context up front.
Cue 214 — 00:15:19,974 --> 00:15:26,154
So it used to be that like when you use a coding agent or coding CLI,
Cue 215 — 00:15:26,254 --> 00:15:29,494
it would inject, I don't know, files or content up front.
Cue 216 — 00:15:29,534 --> 00:15:32,014
There's a movement towards progressive disclosure.
Cue 217 — 00:15:32,054 --> 00:15:36,534
So the models are very smart. And so allowing them to kind of explore the file base
Cue 218 — 00:15:36,574 --> 00:15:39,014
or file system and find it for themselves.
Cue 219 — 00:15:39,054 --> 00:15:40,234
So there's a couple.
Cue 220 — 00:15:44,634 --> 00:15:50,154
There we go. There's a couple of these patterns that I vouch for for you to look at.
Cue 221 — 00:15:50,194 --> 00:15:52,814
But the common theme is simplification.
Cue 222 — 00:15:52,874 --> 00:15:57,014
Even the models, when you ask it to do something will tend to add.
Cue 223 — 00:15:57,074 --> 00:15:59,954
And part of our roles is also to simplify and keep things high.
Cue 224 — 00:15:59,994 --> 00:16:04,074
Signal have a complete command over what the right instructions are,
Cue 225 — 00:16:04,114 --> 00:16:08,834
and it's the best way for us to really understand and and promote the best
Cue 226 — 00:16:08,874 --> 00:16:09,814
practices.
Cue 238 — 00:16:52,081 --> 00:16:57,581
I've been talking about the ways that, like going towards a new model can negatively
Cue 239 — 00:16:57,721 --> 00:17:01,601
affect things, and how to optimize evals and harness all this to say,
Cue 240 — 00:17:01,641 --> 00:17:05,321
like typically it does mean more intelligence, kind of better value in better use
Cue 241 — 00:17:05,361 --> 00:17:09,161
case. I mean, this is on a compaction eval as you go progressing across the
Cue 242 — 00:17:09,201 --> 00:17:12,641
different models, it's the performance increased.
Cue 243 — 00:17:12,681 --> 00:17:17,841
Same with like BrowseComp. But if you are in the cases in which you see,
Cue 244 — 00:17:17,881 --> 00:17:24,381
hey, new model has released and my use case hasn't seen an increase in performance,
Cue 245 — 00:17:24,601 --> 00:17:31,181
it is something to question. And as creating code becomes really cheap,
Cue 246 — 00:17:31,641 --> 00:17:35,961
having that leanness within what you built and the instructions still becomes ever
Cue 247 — 00:17:36,001 --> 00:17:37,141
more important.
Cue 248 — 00:17:39,761 --> 00:17:40,561
So.
Cue 249 — 00:17:42,401 --> 00:17:47,881
Before kind of coming a little bit more to a close, this is a bit of the loop that I
Cue 250 — 00:17:47,921 --> 00:17:51,061
start to really promote with a bunch of this customers.
Cue 251 — 00:17:52,001 --> 00:17:58,161
I'm calling it the SDLC, but it's it's about how do you just not just add towards
Cue 252 — 00:17:58,201 --> 00:18:02,081
your harness your instructions, your skills, but what is a way that you can actually
Cue 253 — 00:18:02,121 --> 00:18:07,721
like very quickly have a bit of a loop where you can start to really keep what's
Cue 254 — 00:18:07,761 --> 00:18:12,161
high signal. And just like the Claude Code team in cutting out 80% of the system
Cue 255 — 00:18:12,201 --> 00:18:16,721
prompt and improving performance, you really iterate as a scientist to keep what
Cue 256 — 00:18:16,761 --> 00:18:19,101
really matters, and you keep experimenting.
Cue 257 — 00:18:19,201 --> 00:18:21,881
So this can look as, first of all, un-harnessing.
Cue 258 — 00:18:21,921 --> 00:18:23,841
I gave the example of like verification.
Cue 259 — 00:18:23,881 --> 00:18:27,361
Let's say that you you can do this yourself or give it to Claude or your agent,
Cue 260 — 00:18:27,401 --> 00:18:32,781
etc. you say remove all of the verification steps within my system prompts,
Cue 261 — 00:18:33,361 --> 00:18:36,601
then you're baseline. You can run your evals, try on different efforts,
Cue 262 — 00:18:36,641 --> 00:18:42,041
etc. you then see okay it actually increased performance, reduce costs,
Cue 263 — 00:18:42,081 --> 00:18:46,721
reduce latency. But there's one edge case for a couple edge cases that are missing.
Cue 264 — 00:18:46,761 --> 00:18:51,321
So then you can harness and then you can ship that and continue that within the loop
Cue 265 — 00:18:51,361 --> 00:18:57,041
using your evals to reassess. And what's interesting about this loop is you can
Cue 266 — 00:18:57,081 --> 00:19:02,361
actually just get Claude to really do this on a kind of experimental basis with some
Cue 267 — 00:19:02,401 --> 00:19:04,961
customers we've run. I'm not sure.
Cue 268 — 00:19:05,001 --> 00:19:09,561
I'm guessing folks have run kind of /goal or these kind of like self-improvement
Cue 269 — 00:19:09,601 --> 00:19:14,761
loops. It's basically you give Claude a command, an outcome and then it iterates and
Cue 270 — 00:19:14,801 --> 00:19:17,321
creates experiments until it optimizes on that.
Cue 271 — 00:19:17,361 --> 00:19:18,601
This is a really good example.
Cue 272 — 00:19:18,641 --> 00:19:23,121
So with a customer we said what is the most lean and high impact version of your
Cue 273 — 00:19:23,161 --> 00:19:28,621
harness look like. And it went through the steps of removing pieces re-baselining,
Cue 274 — 00:19:28,661 --> 00:19:31,901
re-harnessing shipping and then continuing within this loop.
Cue 275 — 00:19:32,441 --> 00:19:39,121
And with that, I've seen in the order of like a 40% reduction in cost for some use
Cue 276 — 00:19:39,161 --> 00:19:42,241
cases where there was too much verification, where there was bloat in certain
Cue 277 — 00:19:42,281 --> 00:19:46,081
places, etc.. And all this to say, it's not just about adding,
Cue 278 — 00:19:46,121 --> 00:19:51,401
it's also about just how do you have a process within your team that you ensure you
Cue 279 — 00:19:51,441 --> 00:19:55,081
have the most high signal version of the instructions you're trying to give,
Cue 280 — 00:19:55,121 --> 00:19:59,661
where code is just a way to to be able to to address those instructions.
Cue 281 — 00:20:02,581 --> 00:20:06,761
The other piece that we've talked quite a bit about, models and harnesses,
Cue 282 — 00:20:07,141 --> 00:20:12,421
I work a lot with the folks that run the evals within each AI native startup,
Cue 283 — 00:20:12,461 --> 00:20:14,841
and some of the fastest growing and leading ones.
Cue 284 — 00:20:16,221 --> 00:20:21,861
I think it's really underappreciated how important the person running evals is
Cue 285 — 00:20:21,901 --> 00:20:25,161
within a technical team that's running an AI use case.
Cue 286 — 00:20:25,421 --> 00:20:29,461
It's not just someone who's running the equivalent of like a integration test or a
Cue 287 — 00:20:29,501 --> 00:20:35,021
unit test or an end to end test, this person is defining what good looks like in an
Cue 288 — 00:20:35,061 --> 00:20:39,621
era where intelligence scales and constantly almost like as a product manager,
Cue 289 — 00:20:39,661 --> 00:20:45,001
defining, okay, where are the model capabilities now for our use case?
Cue 290 — 00:20:45,141 --> 00:20:48,501
Where should it go next? What is it struggling with?
Cue 291 — 00:20:48,541 --> 00:20:53,541
What can be improved and having that overlap in terms of capabilities with what is
Cue 292 — 00:20:53,581 --> 00:20:57,121
the highest ROI and value pieces for our end customers?
Cue 293 — 00:20:57,301 --> 00:20:58,741
And so looking at your person.
Cue 294 — 00:20:58,781 --> 00:21:03,021
Running evals is not just someone who's like managing test coverage,
Cue 295 — 00:21:03,061 --> 00:21:08,301
but is almost like an AI product manager managing what good looks like and ensuring
Cue 296 — 00:21:08,341 --> 00:21:11,261
the user value and capability scale.
Cue 297 — 00:21:11,301 --> 00:21:16,581
With each model release becomes huge, and some of the main recommendations are with
Cue 298 — 00:21:16,621 --> 00:21:21,341
evals starting with a small amount of like really high grade,
Cue 299 — 00:21:21,381 --> 00:21:23,201
almost like manual,
Cue 300 — 00:21:24,421 --> 00:21:27,141
cases within your eval that you really understand.
Cue 301 — 00:21:27,181 --> 00:21:31,421
And then using that to really scale it, you want to grade the outcome and not just
Cue 302 — 00:21:31,461 --> 00:21:35,561
the steps in between, obviously track the cost and the latency.
Cue 303 — 00:21:35,741 --> 00:21:40,661
But the reason this eval person is also very important is because it's not just
Cue 304 — 00:21:40,701 --> 00:21:44,081
about looking at the number, it's also reading the transcripts.
Cue 305 — 00:21:44,461 --> 00:21:49,361
And you often see that some of the really hard use cases are impossible.
Cue 306 — 00:21:49,461 --> 00:21:54,321
And you only see that by seeing the model venting within its transcripts and logs.
Cue 307 — 00:21:54,781 --> 00:21:59,741
And this person becomes so vital in being able to really iterate the perfect version
Cue 308 — 00:21:59,781 --> 00:22:03,561
of what an evaluation system for your AI use case looks like.
Cue 309 — 00:22:06,961 --> 00:22:09,861
So we've touched a lot about these changes.
Cue 310 — 00:22:10,241 --> 00:22:13,921
There's models that have a bigger change in ones that have a smaller one.
Cue 311 — 00:22:13,961 --> 00:22:19,481
That change from an Opus 5 to 5.5 hopefully was quite pleasant to folks,
Cue 312 — 00:22:19,521 --> 00:22:20,921
even internally. Before release.
Cue 313 — 00:22:20,961 --> 00:22:23,961
It was one of my favorite models in terms of its token efficiency,
Cue 314 — 00:22:24,001 --> 00:22:29,121
etc. we see this as a big win, but this is a model that, with your existing Opus 5
Cue 315 — 00:22:29,161 --> 00:22:32,001
prompts, should perform quite well without the changes.
Cue 316 — 00:22:32,041 --> 00:22:35,241
There's models that will have a big difference, other ones that will have a smaller
Cue 317 — 00:22:35,281 --> 00:22:38,761
difference within patterns. It's obviously part of my job personally,
Cue 318 — 00:22:38,801 --> 00:22:43,261
but also community generally to make that change as as easy as possible.
Cue 319 — 00:22:44,601 --> 00:22:49,521
But that's also a piece it's you shouldn't expect the same exact changes per per
Cue 320 — 00:22:49,561 --> 00:22:50,661
model release.
Cue 321 — 00:22:52,841 --> 00:22:56,541
And I wanted to leave it off with three questions.
Cue 322 — 00:22:56,921 --> 00:23:00,841
Hopefully that un-harnessing SDLC loop is an interesting one,
Cue 323 — 00:23:00,881 --> 00:23:04,961
but there's typically such a focus on adding the.
Cue 324 — 00:23:05,001 --> 00:23:09,343
The three questions that I'd like to ask technical teams is how efficiently can you
Cue 325 — 00:23:09,383 --> 00:23:12,541
change a harness? Can you test a harness change?
Cue 326 — 00:23:12,921 --> 00:23:18,821
If you were to like remove 80% of your system prompt like the Claude Code team did,
Cue 327 — 00:23:19,201 --> 00:23:21,401
how easy would you be able to test that?
Cue 328 — 00:23:21,441 --> 00:23:24,121
Because there's a lot of gains and flexibility that then gives you,
Cue 329 — 00:23:24,161 --> 00:23:27,581
if you start off with a leaner code base within your harness.
Cue 330 — 00:23:28,601 --> 00:23:31,641
And this often means also making your harness in a more modular way.
Cue 331 — 00:23:31,681 --> 00:23:33,661
So it is easier to do those changes.
Cue 332 — 00:23:34,841 --> 00:23:39,241
Which parts of the harness do you see as temporary as intelligence scales?
Cue 333 — 00:23:39,281 --> 00:23:43,121
Are there parts that you think are kind of crutches just because the model
Cue 334 — 00:23:43,161 --> 00:23:49,881
intelligence isn't there? Or as this use case scales, it can start to take on higher
Cue 335 — 00:23:49,921 --> 00:23:55,161
capabilities. And then last but not least, like not just looking at the numbers,
Cue 336 — 00:23:55,201 --> 00:23:59,521
but also just like really truly understanding on that eval situation.
Cue 337 — 00:23:59,561 --> 00:24:03,501
What are the model transcripts saying about your eval and your harness?
Cue 338 — 00:24:04,481 --> 00:24:08,001
There's a couple customer that I saw that did something quite interesting,
Cue 339 — 00:24:08,041 --> 00:24:12,701
which was they introduced an ability for the models to vent.
Cue 340 — 00:24:13,601 --> 00:24:17,001
So as the model goes into acting on use cases in production,
Cue 341 — 00:24:17,041 --> 00:24:20,081
it starts being like, hey, this was annoying, etc.
Cue 342 — 00:24:20,121 --> 00:24:25,241
and it's actually really interesting to be able to have that type of feedback,
Cue 343 — 00:24:25,281 --> 00:24:28,681
because these models are very intelligent and probably have good ideas that then
Cue 344 — 00:24:28,721 --> 00:24:31,501
contribute towards your system, your harness, etc..
Cue 345 — 00:24:32,041 --> 00:24:35,841
So yeah, really understanding what are the transcripts saying and not just what the
Cue 346 — 00:24:35,881 --> 00:24:36,941
numbers are saying.
Cue 347 — 00:24:38,321 --> 00:24:40,681
I just want to say that that's the end of my slides.
Cue 348 — 00:24:40,721 --> 00:24:42,841
Thank you so much for listening to me before lunch.
Cue 349 — 00:24:42,881 --> 00:24:45,001
Hopefully this was this was interesting.
Cue 350 — 00:24:45,041 --> 00:24:47,501
And yeah. Thank you.
Cue 351 — 00:24:53,081 --> 00:24:55,321
Would love to take any questions.
Cue 352 — 00:24:55,361 --> 00:25:00,761
And hopefully I think part of the paradigm shift is looking at smaller harnesses and
Cue 353 — 00:25:00,801 --> 00:25:04,161
leaner ones, not just bigger ones, and then looking at yourself if you're working on
Cue 354 — 00:25:04,201 --> 00:25:09,041
harnesses and evals as scientists and not just engineers, but I'll take any
Cue 355 — 00:25:09,081 --> 00:25:11,281
questions. We have time for a couple of questions.
Cue 356 — 00:25:11,321 --> 00:25:14,521
Please raise your hand so I can bring you a mic, I promise.
Cue 357 — 00:25:14,561 --> 00:25:16,181
You're not as loud as you think.
Cue 358 — 00:25:17,521 --> 00:25:22,881
Hey, how do you avoid eval hacking in in a in a setup like this,
Cue 359 — 00:25:22,921 --> 00:25:26,301
when you remove 80% of your of your prompts?
Cue 360 — 00:25:28,281 --> 00:25:34,401
I mean, even when you don't replace 80% of the prompt, eval hacking becomes one of
Cue 361 — 00:25:34,441 --> 00:25:38,321
the most important things. I think it's also the importance of the person.
Cue 362 — 00:25:38,361 --> 00:25:45,161
Managing evals has increased exponentially over time, because as the models become
Cue 363 — 00:25:45,201 --> 00:25:49,481
more and more intelligent, it becomes harder to stop things like eval hacking.
Cue 364 — 00:25:49,521 --> 00:25:55,701
And so, yeah, just to say, even without that, that becomes a very important part.
Cue 365 — 00:25:55,801 --> 00:25:59,641
I think the only real way that we've seen is obviously there's a bunch of sandboxing
Cue 366 — 00:25:59,681 --> 00:26:02,761
things that you can start to look at, like you used to just run the evals on your
Cue 367 — 00:26:02,801 --> 00:26:07,281
computer and then looking at like a secure sandbox that you can actually run them
Cue 368 — 00:26:07,321 --> 00:26:11,561
in. That's kind of airtight, that you start to look at the transcripts and seeing,
Cue 369 — 00:26:11,601 --> 00:26:13,621
okay, is it cheating in certain ways?
Cue 370 — 00:26:14,201 --> 00:26:18,361
All this to say is having a higher degree of skepticism when you run evals is to the
Cue 371 — 00:26:18,401 --> 00:26:23,201
numbers and looking a little bit more about the how and the transcripts instead to
Cue 372 — 00:26:23,241 --> 00:26:25,141
try and see what's what's valid or not.
Cue 373 — 00:26:25,281 --> 00:26:27,021
Basically, it's a continuous effort.
Cue 374 — 00:26:28,121 --> 00:26:32,241
So if you just hit the goal, like it would end up with something that was like over
Cue 375 — 00:26:32,281 --> 00:26:34,501
optimized for whatever evals you had.
Cue 376 — 00:26:35,161 --> 00:26:38,601
I think also the folks. So it used to be sometimes like there wasn't someone
Cue 377 — 00:26:38,641 --> 00:26:42,561
dedicated on evals and technical teams, then someone became dedicated,
Cue 378 — 00:26:42,601 --> 00:26:47,321
then some people started become dedicated towards just the sandboxing of the eval,
Cue 379 — 00:26:47,361 --> 00:26:49,781
and then someone running the use case in the eval.
Cue 380 — 00:26:50,001 --> 00:26:53,401
And so I think, yeah, just as this is scaled, there starts to be more
Cue 381 — 00:26:53,441 --> 00:26:57,501
responsibilities to making sure that you're not reward hacking,
Cue 382 — 00:26:57,681 --> 00:27:00,321
which tends to sometimes be the easiest thing for the models,
Cue 383 — 00:27:00,361 --> 00:27:04,321
hopefully with more aligned models, that also starts to be solved.
Cue 384 — 00:27:04,361 --> 00:27:09,321
But but there is a lot of efforts going into just sandboxing the evals and making
Cue 385 — 00:27:09,361 --> 00:27:13,981
sure that, that there's not another path into than to complete it.
Cue 386 — 00:27:14,321 --> 00:27:15,181
Thanks.
Cue 387 — 00:27:17,041 --> 00:27:19,681
You kind of answered a little bit, but I was wondering.
Cue 388 — 00:27:19,721 --> 00:27:23,481
Sorry. I'm sort of here. If if you could give some insight in what you're doing with
Cue 389 — 00:27:23,521 --> 00:27:28,161
customers to help them avoid overfitting, whether and how they know when is a good
Cue 390 — 00:27:28,201 --> 00:27:30,061
time to throw away bad evals.
Cue 391 — 00:27:33,286 --> 00:27:37,606
I think it it it usually depends on the exact evals.
Cue 392 — 00:27:37,646 --> 00:27:42,046
So like typically it would be a matter of like looking into the transcripts
Cue 393 — 00:27:42,086 --> 00:27:48,046
specifically like in terms of like a thumb in the like a rule of thumb generally,
Cue 394 — 00:27:48,086 --> 00:27:52,206
it would be that we try to kind of have evals, like the sweet spot is that some of
Cue 395 — 00:27:52,246 --> 00:27:56,886
the best models are achieving like 60 to 80% performance, but I wouldn't over index
Cue 396 — 00:27:56,926 --> 00:27:59,206
on that. But that's typically one.
Cue 397 — 00:27:59,246 --> 00:28:04,326
And then ones where we see that consistently with newer models and higher efforts,
Cue 398 — 00:28:04,366 --> 00:28:10,246
especially if there isn't kind of a nice at least somewhat slope in terms of like
Cue 399 — 00:28:10,286 --> 00:28:15,006
performance. With more intelligence, you start to have to ask real reasons as to why
Cue 400 — 00:28:15,046 --> 00:28:18,346
that isn't scaling. So with some teams, I saw, for example,
Cue 401 — 00:28:19,806 --> 00:28:24,966
new Frontier models were released and the effort curve was like completely flat.
Cue 402 — 00:28:25,006 --> 00:28:30,886
But all of the frontier models started to being 1% away from each other at the 50%
Cue 403 — 00:28:30,926 --> 00:28:38,006
mark. And so you have to start asking like, why are they not able to do the top 50%?
Cue 404 — 00:28:38,046 --> 00:28:42,126
And it turns out that a couple of those are like either impossible problems or the
Cue 405 — 00:28:42,166 --> 00:28:46,686
models intelligence actually were able to see that something that was previously
Cue 406 — 00:28:46,726 --> 00:28:49,706
considered is like a right answer is not actually a right answer.
Cue 407 — 00:28:50,046 --> 00:28:52,806
And so, so, yeah, there's a couple of things.
Cue 408 — 00:28:52,846 --> 00:28:57,926
I think there's no way to go around it apart from like exploring into the details,
Cue 409 — 00:28:57,966 --> 00:29:02,046
but typically typically having frontier models in the 60 to 80% mark.
Cue 410 — 00:29:02,086 --> 00:29:06,306
And then some type of like nice Pareto is like effort and model scale.
Cue 411 — 00:29:09,244 --> 00:29:15,224
Thank you. You mentioned a few times like this a scientist and an engineer.
Cue 412 — 00:29:15,964 --> 00:29:19,004
Building evals. I'm just like, kind of curious not to get too philosophical.
Cue 413 — 00:29:19,044 --> 00:29:22,704
Like what? What are the traits that actually, are important here?
Cue 414 — 00:29:24,684 --> 00:29:29,444
It's a it's a good question. I think it's for me, it's a somewhat of an analogy.
Cue 415 — 00:29:29,484 --> 00:29:32,564
I'd say I'd sit more on the engineering side than the science side,
Cue 416 — 00:29:32,604 --> 00:29:36,324
but I think, so traditionally been a software engineer and CTO,
Cue 417 — 00:29:36,364 --> 00:29:42,084
etc.. I think this kind of work has really pushed me towards being more on the
Cue 418 — 00:29:42,124 --> 00:29:43,664
science side in the sense of,
Cue 419 — 00:29:44,964 --> 00:29:48,044
I guess on the engineering side, there's always been scientific pieces where maybe
Cue 420 — 00:29:48,084 --> 00:29:51,004
you're optimizing for latency, for performance, etc.
Cue 421 — 00:29:51,044 --> 00:29:55,024
but I think in terms of when you're removing 80% of your system prompt,
Cue 422 — 00:29:55,084 --> 00:30:00,484
there has to be like quite a decoupling from what you have now and a real focus on
Cue 423 — 00:30:00,524 --> 00:30:03,944
like the outcome and the exploration and the learnings towards getting there.
Cue 424 — 00:30:04,404 --> 00:30:06,844
I think engineering has always had like parts of a scientist,
Cue 425 — 00:30:06,884 --> 00:30:09,144
and so maybe it's a little bit more of a spectrum.
Cue 426 — 00:30:09,204 --> 00:30:15,964
But as an engineer, I think what's been cool is I used to have to develop
Cue 427 — 00:30:16,004 --> 00:30:19,204
everything by hand or websites, etc.
Cue 428 — 00:30:19,244 --> 00:30:22,804
now I treat everything I build as an experiment towards like some type of bigger
Cue 429 — 00:30:22,844 --> 00:30:25,244
outcome instead of the main and the only thing.
Cue 430 — 00:30:25,284 --> 00:30:29,204
And so seeing everything is type, some type of iteration of constant improvement I
Cue 431 — 00:30:29,244 --> 00:30:31,804
think maybe has been the mindset mindset shift.
Cue 432 — 00:30:31,844 --> 00:30:34,524
I think it's a spectrum. So there are different degrees.
Cue 433 — 00:30:34,564 --> 00:30:36,784
But that's been my experience so far.
.tessl-plugin
skills
talk-arcolano-autonomy-and-trust
talk-aroussi-decision-context
talk-bevin-agent-assisted-instrument-design
talk-boeckeler-context-lifecycles
talk-braun-context-graphs
talk-carmont-evolving-agent-harnesses
talk-chanezon-gordon-agent-experience-evaluation
talk-cox-secure-agentic-workflows
talk-cunniffe-reusable-agent-platform
talk-davis-agent-tool-policies
talk-debois-concurrent-delivery
talk-dzialets-team-automation
talk-fahmy-harness-api-design
talk-finster-continuous-delivery
talk-fowler-regenerative-software
talk-gavrilescu-factory-architecture
talk-graciolli-amin-verification-aware-programming
talk-gruenbaum-reverse-graph-engineering
talk-hassan-agentic-engineering
talk-hossain-enterprise-agent-adoption
talk-huntley-patch-ready-factories
talk-jain-verifying-intent
talk-katsioloudes-montei-agentic-lifecycle
talk-knox-harness-engineering
talk-kovac-deterministic-memory
talk-lee-perkins-code-readiness
talk-lorenc-machine-speed-security
talk-lubken-software-of-one
talk-meijer-software-specialization
talk-meppiel-versioned-agent-contracts
talk-moss-concurrent-agent-risk
talk-myers-understanding-agent-generated-code
talk-ocytko-scaling-agentic-engineering
talk-oza-automating-design-quality
talk-panzica-agentic-sdlc
talk-pari-patel-agent-awareness
talk-podjarny-trustworthy-software-factories
talk-poulter-factory-improvement
talk-pratik-patel-refined-rag-data
talk-re-ferre-development-platforms
talk-sadogursky-factory-of-factories
talk-shah-training-agent-skills
talk-sharma-mohapatra-evolving-intent
talk-singhal-operating-agents-at-scale
talk-stack-incident-dossiers
talk-tyler-team-coding-benchmarks
talk-wotherspoon-agent-execution-environments
talk-yegge-ai-powered-factories
talk-zuber-leadership-through-coding