AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
86%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Nandini Singhal · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=BtGqqd7rbAI
Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.
For topic navigation, see outline.md.
Cue 1 — 00:00:00,220 --> 00:00:02,080
Hi everyone. Thank you so much.
Cue 2 — 00:00:02,540 --> 00:00:07,180
My name is Nandini and I am a software engineer at Datadog and I work on the
Cue 3 — 00:00:07,220 --> 00:00:10,940
streaming platform team. So my day to day job is working on Kafka.
Cue 4 — 00:00:10,980 --> 00:00:12,780
But that's not what this talk is about.
Cue 5 — 00:00:12,820 --> 00:00:16,220
So this talk is about operating coding agents at scale.
Cue 6 — 00:00:16,260 --> 00:00:21,340
And like a lot of companies, we've given coding agents to pretty much all of our
Cue 7 — 00:00:21,380 --> 00:00:26,300
engineers. And it turns out enabling the coding agent is sort of an easy job.
Cue 8 — 00:00:26,340 --> 00:00:29,600
But what I want to talk about is what comes after that.
Cue 9 — 00:00:30,220 --> 00:00:32,900
So we keep changing the set up every week.
Cue 10 — 00:00:32,940 --> 00:00:37,580
And the hard part is actually knowing whether each of those changes helped.
Cue 11 — 00:00:37,620 --> 00:00:40,520
And I'm going to be covering what changes I'm talking about.
Cue 12 — 00:00:41,060 --> 00:00:45,500
What does a good change look like and what do we do about that?
Cue 13 — 00:00:45,540 --> 00:00:50,820
So let me get started. So at Datadog we have thousands of engineers which are using
Cue 14 — 00:00:50,860 --> 00:00:54,060
coding agents. And it's not a single setup for every engineer.
Cue 15 — 00:00:54,100 --> 00:00:58,100
Right. Because every engineer has their own setup of different agents.
Cue 16 — 00:00:58,140 --> 00:01:00,780
Different terminals could be different configurations.
Cue 17 — 00:01:00,820 --> 00:01:06,020
Different models could be some people using open weight models and some have each
Cue 18 — 00:01:06,060 --> 00:01:08,220
different settings, different parameters.
Cue 19 — 00:01:08,260 --> 00:01:14,180
And the thing that makes this interesting is on the right when the platform changes
Cue 20 — 00:01:14,220 --> 00:01:18,860
centrally, like at an organization level, say let's say the default model is
Cue 21 — 00:01:18,900 --> 00:01:23,060
changed, or let's say the reasoning effort, or let's say the company has decided to
Cue 22 — 00:01:23,100 --> 00:01:27,260
put a cap on the context, or let's say we decided to try out a new plugin,
Cue 23 — 00:01:27,300 --> 00:01:33,660
or we decided to try out a new MCP tool that change lands on every single engineers
Cue 24 — 00:01:33,700 --> 00:01:38,980
machine. And so we are constantly making those changes every week as new models are
Cue 25 — 00:01:39,020 --> 00:01:42,020
evolving, new things and technologies are coming up.
Cue 26 — 00:01:42,060 --> 00:01:47,020
And so this talk is really about how we figured out what each of those changes did.
Cue 27 — 00:01:47,060 --> 00:01:49,980
And so people talk a lot about the effectiveness of AI.
Cue 28 — 00:01:50,020 --> 00:01:52,860
But what is it exactly mean and how do we measure that.
Cue 29 — 00:01:52,900 --> 00:01:58,020
And so before going into that, it helps to see where exactly agents are being used,
Cue 30 — 00:01:58,060 --> 00:02:02,100
because it's not just really the engineers and their laptops that that are using
Cue 31 — 00:02:02,140 --> 00:02:07,140
these AI agents. Right. So where exactly agents are being used today.
Cue 32 — 00:02:07,180 --> 00:02:11,460
So they show up in different stack, different places in the stack.
Cue 33 — 00:02:11,500 --> 00:02:16,300
And I've ordered them by how much control we have at the top of the pipeline.
Cue 34 — 00:02:16,340 --> 00:02:22,940
We have an every pull request for code review and the risk scoring and let's say the
Cue 35 — 00:02:22,980 --> 00:02:24,820
auto approval for lower risk changes.
Cue 36 — 00:02:24,860 --> 00:02:26,340
So that's the top of the stack.
Cue 37 — 00:02:26,380 --> 00:02:30,260
And then you have the operational layer when something breaks investigating
Cue 38 — 00:02:30,300 --> 00:02:34,920
incidents, triaging alerts and checking where let's say a deployment went bad.
Cue 39 — 00:02:35,260 --> 00:02:39,220
And then you have like managed agents which are basically running on a schedule.
Cue 40 — 00:02:39,260 --> 00:02:44,860
And so they do things like vulnerability fixes, adding, let's say missing test cases
Cue 41 — 00:02:44,900 --> 00:02:48,860
or cleaning a bad code, which is maybe dead or unused.
Cue 42 — 00:02:48,900 --> 00:02:52,620
And so at the bottom layer is basically where the engineers lie,
Cue 43 — 00:02:52,660 --> 00:02:57,540
which is basically the interactive agents, like the ID or the CLI that the engineers
Cue 44 — 00:02:57,580 --> 00:03:02,100
use on a day to day basis. And so the arrow is really important because it shows how
Cue 45 — 00:03:02,140 --> 00:03:07,200
much control the we have over the model and the configuration.
Cue 46 — 00:03:07,340 --> 00:03:09,260
And so at the top we pick the model.
Cue 47 — 00:03:09,300 --> 00:03:13,740
And so we control the cost and the quality across the organization and the bottom.
Cue 48 — 00:03:13,780 --> 00:03:15,380
It's basically the engineers choice.
Cue 49 — 00:03:15,420 --> 00:03:21,660
Right. So it's it's that's where most of the usage is and also where it's very hard
Cue 50 — 00:03:21,700 --> 00:03:26,720
to measure. And so most of what I talk about today is that bottom layer.
Cue 51 — 00:03:26,980 --> 00:03:30,260
And whenever we compare notes and data with other companies,
Cue 52 — 00:03:30,300 --> 00:03:32,700
the same four questions keep popping up.
Cue 53 — 00:03:32,740 --> 00:03:37,020
And what are those questions? And I hear everybody here talk about these four
Cue 54 — 00:03:37,060 --> 00:03:40,700
questions, which is like who who is actually using this?
Cue 55 — 00:03:40,740 --> 00:03:45,180
And like what's the adoption rate of a particular model or let's say configuration.
Cue 56 — 00:03:45,220 --> 00:03:49,780
And does it really stick? And the second question is like the productivity aspect,
Cue 57 — 00:03:49,820 --> 00:03:54,380
is it really helping or is it just producing more code for people to review?
Cue 58 — 00:03:54,420 --> 00:03:57,220
And the third question is basically about the cost.
Cue 59 — 00:03:57,260 --> 00:03:59,540
And the cost question is super important as well.
Cue 60 — 00:03:59,580 --> 00:04:04,340
Like to understand why is the bill of AI growing and what you can actually do about
Cue 61 — 00:04:04,380 --> 00:04:08,380
it. And the fourth question is also super important, which is about security.
Cue 62 — 00:04:08,420 --> 00:04:10,900
And the security question is about is it safe?
Cue 63 — 00:04:10,940 --> 00:04:16,340
Who can call which model? Where does our code go, who can merge the code on its own
Cue 64 — 00:04:16,380 --> 00:04:19,660
and things of that sort. And so I'm going to use these four questions as the
Cue 65 — 00:04:19,700 --> 00:04:24,620
backbone of the talk. And for each one I'll show you why it's really difficult and
Cue 66 — 00:04:24,660 --> 00:04:28,980
then how we address it. And then right at the end I'll come back and summarize what
Cue 67 — 00:04:29,020 --> 00:04:33,100
we learned. And so we've just done the four questions.
Cue 68 — 00:04:33,140 --> 00:04:37,020
And so I'll go through why each one is hard.
Cue 69 — 00:04:37,060 --> 00:04:40,840
And then I'll show you our approach and how it evolved over time.
Cue 70 — 00:04:41,500 --> 00:04:45,540
And so the main part of the talk, which is basically the four layers which I'm going
Cue 71 — 00:04:45,580 --> 00:04:50,640
to be talking about. So let's start with the first question which is adoption.
Cue 72 — 00:04:50,740 --> 00:04:52,980
And so adoptions really simple right.
Cue 73 — 00:04:53,020 --> 00:04:56,380
Like adoption sounds like it should be one simple number like how many people use
Cue 74 — 00:04:56,420 --> 00:05:00,780
the model. And it's really actually not because it's a funnel,
Cue 75 — 00:05:00,820 --> 00:05:07,420
because it each step decides like, does someone even have access to the model?
Cue 76 — 00:05:07,460 --> 00:05:11,980
First of all, and at an organization level, that's basically like device management.
Cue 77 — 00:05:12,020 --> 00:05:16,860
And so let's say the company decides to install this manage setting to every laptop.
Cue 78 — 00:05:16,900 --> 00:05:21,120
And then they did they actually like set it up.
Cue 79 — 00:05:21,500 --> 00:05:23,780
And we how can we identify that.
Cue 80 — 00:05:23,820 --> 00:05:28,220
So we identify that because the gateway sees their first call that they have signed
Cue 81 — 00:05:28,260 --> 00:05:30,020
up with their own personal token.
Cue 82 — 00:05:30,060 --> 00:05:34,460
And then we have to also measure their usage that they use it actually this week.
Cue 83 — 00:05:34,500 --> 00:05:36,640
And that's basically sessions.
Cue 84 — 00:05:37,020 --> 00:05:40,540
And so you have to categorize the sessions by let's say the team by what kind of
Cue 85 — 00:05:40,580 --> 00:05:43,260
work they're doing, which comes from, again, the gateway.
Cue 86 — 00:05:43,300 --> 00:05:48,300
And there is a tool called Trajectory, which is basically the gateway for Datadog
Cue 87 — 00:05:48,340 --> 00:05:53,400
and which is our like the session capture tool and more on that will come later.
Cue 88 — 00:05:53,740 --> 00:05:58,660
And the bottom layer is basically really using it a month later.
Cue 89 — 00:05:58,700 --> 00:06:03,220
And that's basically retention because people can start like getting hyped up about
Cue 90 — 00:06:03,260 --> 00:06:06,020
a model or a tool, and then they're not even using it.
Cue 91 — 00:06:06,060 --> 00:06:10,500
So it's important to understand the adoption rate.
Cue 92 — 00:06:10,540 --> 00:06:15,280
And the question two is basically productivity, which is where it gets hard.
Cue 93 — 00:06:16,060 --> 00:06:20,720
And I want to start productivity with an outside view because it sets expectations.
Cue 94 — 00:06:20,900 --> 00:06:24,660
There is a study from DORA, which is basically the Google Cloud's research group
Cue 95 — 00:06:24,700 --> 00:06:31,100
behind the State of DevOps report, and they put out a report in April on the ROI of
Cue 96 — 00:06:31,140 --> 00:06:33,380
AI assisted software development.
Cue 97 — 00:06:33,420 --> 00:06:39,220
And one of their findings is that most organizations get a bit worse before they get
Cue 98 — 00:06:39,260 --> 00:06:41,580
better. And so there's always a dip.
Cue 99 — 00:06:41,620 --> 00:06:44,820
And teams are like learning new workflows and there's more code.
Cue 100 — 00:06:44,860 --> 00:06:49,460
So there's more to review and testing and approvals become more bottleneck.
Cue 101 — 00:06:49,500 --> 00:06:53,740
And so the point is that the gains come from the engineering system,
Cue 102 — 00:06:53,780 --> 00:06:57,300
which is built around the tool and not from like getting a better tool.
Cue 103 — 00:06:57,340 --> 00:07:02,580
And I find that really reassuring because when someone asks why things have not
Cue 104 — 00:07:02,620 --> 00:07:09,100
magically improved, like from past two months ago, it means that the dip is a real
Cue 105 — 00:07:09,140 --> 00:07:12,180
problem. And so that's what really the the talk is about.
Cue 106 — 00:07:12,220 --> 00:07:18,540
And we will see that the review bottleneck is real in the next number that we are
Cue 107 — 00:07:18,580 --> 00:07:24,380
going to see. And so the bottleneck is this is a study again from Faros,
Cue 108 — 00:07:24,420 --> 00:07:29,700
which they published a study about the AI productivity paradox based on the
Cue 109 — 00:07:29,740 --> 00:07:32,780
telemetry from over 10,000 individuals.
Cue 110 — 00:07:32,820 --> 00:07:39,060
And it looks great. It shows that 21% tasks were more were completed almost daily.
Cue 111 — 00:07:39,100 --> 00:07:43,580
2x the number of pull requests are getting merged, and PRs that are about two and a
Cue 112 — 00:07:43,620 --> 00:07:49,300
half times bigger. But but review time is also increased significantly.
Cue 113 — 00:07:49,340 --> 00:07:52,540
Almost 90% of the team's time has now increased in reviews.
Cue 114 — 00:07:52,580 --> 00:07:56,460
And the bugs which are generated per developer have gone up as well.
Cue 115 — 00:07:56,500 --> 00:08:01,020
And so the company level delivery is not significantly improved.
Cue 116 — 00:08:01,060 --> 00:08:05,940
And so more output from the agent doesn't automatically translate into more shipped
Cue 117 — 00:08:05,980 --> 00:08:10,020
or production code. And a lot of it just becomes more code for somebody else to
Cue 118 — 00:08:10,060 --> 00:08:17,020
review. And so the numbers most teams look at first can tell those two situations
Cue 119 — 00:08:17,060 --> 00:08:19,920
apart. So how do you measure productivity.
Cue 120 — 00:08:19,980 --> 00:08:24,140
And so these are the three numbers everybody builds a dashboard for.
Cue 121 — 00:08:24,180 --> 00:08:27,340
Basically they build a dashboard which is like having the number of tokens,
Cue 122 — 00:08:27,380 --> 00:08:29,500
the number of sessions and tool calls.
Cue 123 — 00:08:29,540 --> 00:08:33,780
And they all go up and to the right, which feels great.
Cue 124 — 00:08:33,820 --> 00:08:37,740
And the catch is that each one can go up for a good reason or a bad reason.
Cue 125 — 00:08:37,780 --> 00:08:40,460
And the chart looks exactly the same either way.
Cue 126 — 00:08:40,500 --> 00:08:45,820
So tokens go up when real work is happening, but also when the context is bloated.
Cue 127 — 00:08:45,860 --> 00:08:50,700
And so sessions go up when adoption grows, but also when people keep retrying and
Cue 128 — 00:08:50,740 --> 00:08:55,820
tool calls go up when the agent is making progress, but also when it's lost.
Cue 129 — 00:08:55,860 --> 00:09:02,400
And so a single agent stuck on a flaky test pushes all of these metrics.
Cue 130 — 00:09:02,780 --> 00:09:08,580
And so none of these metrics really tell you anything about how much of the code
Cue 131 — 00:09:08,620 --> 00:09:12,020
actually got shipped. And so even when you pick better metrics,
Cue 132 — 00:09:12,060 --> 00:09:16,214
there is a second problem, which is tying a result back to the change that caused
Cue 133 — 00:09:16,254 --> 00:09:17,054
it.
Cue 134 — 00:09:18,220 --> 00:09:21,380
And so why is it so hard to tell what a change did.
Cue 135 — 00:09:21,420 --> 00:09:25,220
So there are three reasons. It's hard to say what any single change did.
Cue 136 — 00:09:25,260 --> 00:09:29,340
First, the model and the harness often shipped together.
Cue 137 — 00:09:29,380 --> 00:09:34,340
So when a new Claude Code, let's say, release comes out, it often comes up with a
Cue 138 — 00:09:34,380 --> 00:09:37,500
new model. And so you can't always change like one thing at a time.
Cue 139 — 00:09:37,540 --> 00:09:40,940
What happens is agents are not deterministic.
Cue 140 — 00:09:40,980 --> 00:09:44,820
So you if you run the same task twice and you get different results.
Cue 141 — 00:09:44,860 --> 00:09:48,540
So one good run of a task really doesn't prove much.
Cue 142 — 00:09:48,580 --> 00:09:53,340
And so the main important thing is that the outcomes that we care about,
Cue 143 — 00:09:53,380 --> 00:09:56,820
they come very late. And which is basically did the PR get reviewed.
Cue 144 — 00:09:56,860 --> 00:10:00,040
Did the PR get merged and did the PR get reverted?
Cue 145 — 00:10:00,820 --> 00:10:04,340
And that reward could happen, let's say days or weeks after the merge.
Cue 146 — 00:10:04,380 --> 00:10:07,080
And so who's tracking all of that really?
Cue 147 — 00:10:07,620 --> 00:10:13,840
And it's not just us. METR is struggling with this too, which is basically a
Cue 148 — 00:10:13,980 --> 00:10:19,300
research organization who can, like, run control studies on the AI and developer
Cue 149 — 00:10:19,340 --> 00:10:25,100
productivity. They published a study and basically that they're redesigning their
Cue 150 — 00:10:25,140 --> 00:10:31,220
experiment because so many now developers won't work without AI that the comparison
Cue 151 — 00:10:31,260 --> 00:10:33,280
study that they have is broken.
Cue 152 — 00:10:33,460 --> 00:10:36,820
And so people are running several agents at once.
Cue 153 — 00:10:36,860 --> 00:10:42,780
And so we at Datadog have stopped waiting for a perfect answer.
Cue 154 — 00:10:42,820 --> 00:10:46,820
And so we measure continuously inside our own system instead.
Cue 155 — 00:10:46,860 --> 00:10:52,680
And so the first step is being clear about what each kind of metric is actually for.
Cue 156 — 00:10:53,780 --> 00:10:56,860
So we gave our metrics two separate jobs.
Cue 157 — 00:10:56,900 --> 00:11:00,840
So outcomes are basically what we decide.
Cue 158 — 00:11:01,260 --> 00:11:05,200
So what is the outcome. Did the change get merged and did it stay merged.
Cue 159 — 00:11:05,420 --> 00:11:07,580
Did it need a follow up or did it get reverted.
Cue 160 — 00:11:07,620 --> 00:11:10,900
And what did it cost per merge change.
Cue 161 — 00:11:10,940 --> 00:11:14,540
So those are the important numbers that we actually make the calls on.
Cue 162 — 00:11:14,580 --> 00:11:18,940
So activity is what explains when the outcome is delivered.
Cue 163 — 00:11:18,980 --> 00:11:22,240
And tokens basically tell us where the money is spent.
Cue 164 — 00:11:22,860 --> 00:11:28,220
Sessions basically tell us how people are working and the tool that the tool agent
Cue 165 — 00:11:28,260 --> 00:11:32,860
calls. They tell us what the agent was actually doing because let's be real,
Cue 166 — 00:11:32,900 --> 00:11:37,620
the agent could be just, let's say, spinning up or making a lot of redundant calls.
Cue 167 — 00:11:37,660 --> 00:11:42,580
So I look at activity dashboards all the time, and they're really useful for
Cue 168 — 00:11:42,620 --> 00:11:48,400
debugging. And we just don't use them to declare that something worked.
Cue 169 — 00:11:48,860 --> 00:11:54,660
So that's productivity. Which basically brings us to question number three,
Cue 170 — 00:11:54,700 --> 00:12:01,500
which is basically cost. And the reason cost is confusing is that the bill of the
Cue 171 — 00:12:01,540 --> 00:12:04,100
AI agent is basically a multiplication.
Cue 172 — 00:12:04,140 --> 00:12:07,100
And this is a graph which I really like.
Cue 173 — 00:12:07,140 --> 00:12:11,980
And this breakdown is basically adapted from a post that Uber Engineering published.
Cue 174 — 00:12:12,020 --> 00:12:17,780
And if you look at this graph, it's very interesting because the spend is actually
Cue 175 — 00:12:17,820 --> 00:12:25,020
the users times sessions per user times tokens per request,
Cue 176 — 00:12:25,060 --> 00:12:30,860
times price per token, and each term multiplies the next.
Cue 177 — 00:12:30,900 --> 00:12:34,980
And so the first two in green are the adoption.
Cue 178 — 00:12:35,020 --> 00:12:37,860
And you want those to keep growing in an organization.
Cue 179 — 00:12:37,900 --> 00:12:42,540
And the middle three in purple are basically how much work the agent does per
Cue 180 — 00:12:42,580 --> 00:12:48,140
session. And those you actually want to shrink because you want fewer retry loops.
Cue 181 — 00:12:48,180 --> 00:12:50,020
You want the right docs in context.
Cue 182 — 00:12:50,060 --> 00:12:52,300
So it needs the fewer requests.
Cue 183 — 00:12:52,340 --> 00:12:54,900
And you want caching and you want compression.
Cue 184 — 00:12:54,940 --> 00:12:57,340
So each request is basically small.
Cue 185 — 00:12:57,380 --> 00:13:01,300
And the last one is basically something that you don't have control over because the
Cue 186 — 00:13:01,340 --> 00:13:06,820
last one is price per token, which you mostly choose via the default model that you
Cue 187 — 00:13:06,860 --> 00:13:10,220
are setting as a benchmark for your organization.
Cue 188 — 00:13:10,260 --> 00:13:16,460
So when the total goes up, that alone doesn't tell you if it's good news or bad
Cue 189 — 00:13:16,500 --> 00:13:21,260
news, right? And most of these terms aren't really controlled by individual
Cue 190 — 00:13:21,300 --> 00:13:25,020
engineers. They controlled by our default settings.
Cue 191 — 00:13:25,060 --> 00:13:27,940
And so that's where defaults are super important.
Cue 192 — 00:13:27,980 --> 00:13:30,240
Which brings me to my next slide.
Cue 193 — 00:13:31,660 --> 00:13:34,400
The default is a product decision.
Cue 194 — 00:13:34,580 --> 00:13:38,740
And so every row here in this table if you look at it is a setting that we change
Cue 195 — 00:13:38,780 --> 00:13:40,840
centrally in the organization.
Cue 196 — 00:13:40,900 --> 00:13:46,580
And every one of them saves something which is costing something else.
Cue 197 — 00:13:46,620 --> 00:13:51,780
So it's like a trade off. And so if you choose to decide to go with a cheaper
Cue 198 — 00:13:51,820 --> 00:13:57,980
default model, it lowers the price per token, but it can hurt the quality on hard
Cue 199 — 00:13:58,020 --> 00:14:03,460
tasks. If you go with lower reasoning, it cuts the output tokens,
Cue 200 — 00:14:03,500 --> 00:14:10,220
but it can cost you correctness, a tighter context cap, or aggressive compaction.
Cue 201 — 00:14:10,260 --> 00:14:13,980
It can make your request smaller, but it can break prompt caching,
Cue 202 — 00:14:14,020 --> 00:14:17,020
and maybe it can make the agent lose track.
Cue 203 — 00:14:17,060 --> 00:14:22,540
So routing requests to different models or using cheaper models for sub agents.
Cue 204 — 00:14:22,580 --> 00:14:26,200
It saves on price but hurts cache reuse.
Cue 205 — 00:14:26,420 --> 00:14:29,140
And so I'll show you a real example of that in a few minutes.
Cue 206 — 00:14:29,180 --> 00:14:36,020
But even adding helpful plugins, skills, or MCP tools has a cost because their
Cue 207 — 00:14:36,060 --> 00:14:39,160
definitions are sent along with every request.
Cue 208 — 00:14:39,300 --> 00:14:45,500
And so a default setting reaches every engineer in the organization at once.
Cue 209 — 00:14:45,540 --> 00:14:50,540
And each of this, in my opinion, is a product decision, not just a configuration
Cue 210 — 00:14:50,580 --> 00:14:54,080
tweak that you can say, okay, you can roll out an organization because,
Cue 211 — 00:14:55,060 --> 00:14:59,620
that defines the cost. Which brings us to the question number four,
Cue 212 — 00:14:59,660 --> 00:15:03,820
which is security. And I'll come back to that near the end, because our answer there
Cue 213 — 00:15:03,860 --> 00:15:06,040
is again mostly about defaults.
Cue 214 — 00:15:06,140 --> 00:15:08,860
So that's why these questions are hard.
Cue 215 — 00:15:08,900 --> 00:15:11,500
Now how do we actually measure all of these.
Cue 216 — 00:15:11,540 --> 00:15:14,820
And our setup has actually built over time.
Cue 217 — 00:15:14,860 --> 00:15:19,700
It's like one piece at a time that has taken iterations and evolved.
Cue 218 — 00:15:19,740 --> 00:15:22,640
And so how did our approach evolve?
Cue 230 — 00:16:05,140 --> 00:16:12,020
So here's how it evolved. We started by routing agent traffic through a single AI
Cue 231 — 00:16:12,060 --> 00:16:15,780
gateway, so every model call went through one place.
Cue 232 — 00:16:15,820 --> 00:16:17,740
So we tried a couple of cost ideas.
Cue 233 — 00:16:17,780 --> 00:16:22,720
We tried a model router. We tried a token reduction tool called RTK.
Cue 234 — 00:16:22,940 --> 00:16:28,260
We scaled up our internal coding evaluations so that we could see where the agents
Cue 235 — 00:16:28,300 --> 00:16:31,940
struggle. We tested a compression proxy.
Cue 236 — 00:16:31,980 --> 00:16:35,960
In those evals. We tune the default model.
Cue 237 — 00:16:36,180 --> 00:16:41,300
We rolled out compression, moved to a newer model, which is the default,
Cue 238 — 00:16:41,340 --> 00:16:45,820
and then we added like spend guardrails that every company is trying to enforce.
Cue 239 — 00:16:45,860 --> 00:16:50,860
And the pattern was that each of these steps raised a question the signal wasn't
Cue 240 — 00:16:50,900 --> 00:16:52,900
able to answer in the previous step.
Cue 241 — 00:16:52,940 --> 00:16:57,920
And that's how we ended up with four layers of visibility or observability.
Cue 242 — 00:16:57,980 --> 00:17:01,820
And the stories are what I will in the pink ones, if you notice in this,
Cue 243 — 00:17:01,860 --> 00:17:04,800
are basically the stories that I'm going to walk you through.
Cue 244 — 00:17:04,980 --> 00:17:07,760
And so these are the four layers.
Cue 245 — 00:17:08,420 --> 00:17:13,360
And so I've ordered them from fastest to slowest feedback.
Cue 246 — 00:17:13,940 --> 00:17:16,460
So if you look at the top it's the gateway.
Cue 247 — 00:17:16,500 --> 00:17:21,500
As I mentioned you the gateway sees every model call and you hear back in like
Cue 248 — 00:17:21,540 --> 00:17:25,100
immediate like seconds because you would know exactly in the organization which
Cue 249 — 00:17:25,140 --> 00:17:28,720
engineer is making which tool called because it goes via the gateway.
Cue 250 — 00:17:28,860 --> 00:17:32,360
The trace is they see each agent session.
Cue 251 — 00:17:32,820 --> 00:17:36,740
So maybe it could take some minutes because now you are capturing traces for every
Cue 252 — 00:17:36,780 --> 00:17:39,420
single tool call or agent call that's happening.
Cue 253 — 00:17:39,460 --> 00:17:43,140
The evals, they test the change before anyone uses it.
Cue 254 — 00:17:43,180 --> 00:17:47,620
So basically the evals are basically like a testing harness which runs in the like a
Cue 255 — 00:17:47,660 --> 00:17:53,260
periodic cron job or nightly. And those are important to identify how the model
Cue 256 — 00:17:53,300 --> 00:17:56,120
works. I'm going to cover those in a bit.
Cue 257 — 00:17:56,500 --> 00:18:01,260
And so the rollout is basically what is going to roll out the change in front of the
Cue 258 — 00:18:01,300 --> 00:18:06,100
engineers. And then we are going to measure how the actual productivity is.
Cue 259 — 00:18:06,140 --> 00:18:10,140
And at the bottom is basically the outcome, which basically is the work that got
Cue 260 — 00:18:10,180 --> 00:18:13,040
shipped that and did it stay shipped.
Cue 261 — 00:18:13,460 --> 00:18:18,180
And so everything lands in Datadog, which is the LLM Observability,
Cue 262 — 00:18:18,220 --> 00:18:19,860
the dashboards and the monitors.
Cue 263 — 00:18:19,900 --> 00:18:23,840
And so we'll go through them in order, starting with the gateway.
Cue 264 — 00:18:25,460 --> 00:18:30,740
And so the idea is simple, that gateway is basically one way in for every model
Cue 265 — 00:18:30,780 --> 00:18:36,620
call. And so the agent on each laptop has basically the managed settings that point
Cue 266 — 00:18:36,660 --> 00:18:38,980
our traffic to the AI gateway.
Cue 267 — 00:18:39,020 --> 00:18:44,380
And there's a short lived token which ties the call to a specific person to identify
Cue 268 — 00:18:44,420 --> 00:18:49,020
the identity. And behind that, it's the same models and providers as before.
Cue 269 — 00:18:49,060 --> 00:18:51,420
There's just a single front door.
Cue 270 — 00:18:51,460 --> 00:18:57,180
And so what happens is this AI gateway gives us the usage, the errors,
Cue 271 — 00:18:57,220 --> 00:19:00,140
the latency and cost. In Datadog.
Cue 272 — 00:19:00,180 --> 00:19:02,700
It breaks it down by team and model.
Cue 273 — 00:19:02,740 --> 00:19:07,820
And so it basically just adds a few milliseconds of overhead per call.
Cue 274 — 00:19:07,860 --> 00:19:11,180
But now you're able to get an entire visibility across the stack.
Cue 275 — 00:19:11,220 --> 00:19:17,260
And it does come up with some upkeep because you have to now like maintain this AI
Cue 276 — 00:19:17,300 --> 00:19:20,820
gateway. And it's a shared dependency.
Cue 277 — 00:19:20,860 --> 00:19:24,780
So if it's slow, everyone will obviously feel the burden of it.
Cue 278 — 00:19:24,820 --> 00:19:26,000
And so
Cue 279 — 00:19:27,260 --> 00:19:29,900
it has one blind spot if you can spot it.
Cue 280 — 00:19:29,940 --> 00:19:34,940
So what does AI it will not give you is basically it knows the amount of tokens you
Cue 281 — 00:19:34,980 --> 00:19:38,860
are spending, but you it doesn't know what the agent did with that.
Cue 282 — 00:19:38,900 --> 00:19:43,380
And so if you're just FYI, if you're building your AI gateway yourself,
Cue 283 — 00:19:43,420 --> 00:19:46,580
you can use of off the shelf AI gateway.
Cue 284 — 00:19:46,620 --> 00:19:53,460
You can use like LiteLLM proxy, or you can also use like Datadog AI Gateway or Envoy
Cue 285 — 00:19:53,500 --> 00:19:56,200
AI gateway. And there are multiple off the shelf options.
Cue 286 — 00:19:56,620 --> 00:19:57,600
And so
Cue 287 — 00:19:58,840 --> 00:20:05,300
before we get to that blind spot, here's a story about where the gateway was exactly
Cue 288 — 00:20:05,340 --> 00:20:11,100
what got the problem. And so this is the model router that we piloted.
Cue 289 — 00:20:11,140 --> 00:20:14,580
And so the model router basically sits in front of the models,
Cue 290 — 00:20:14,620 --> 00:20:19,540
and for every request, every turn of a session, It guesses how hard that request is
Cue 291 — 00:20:19,580 --> 00:20:25,580
and picks a model. So it basically routes the the traffic to different models based
Cue 292 — 00:20:25,620 --> 00:20:28,780
on the hardness or the problem itself.
Cue 293 — 00:20:28,820 --> 00:20:33,620
And so the easy request would go to a small model that's cheap and fast,
Cue 294 — 00:20:33,660 --> 00:20:35,620
and the hard ones go to the big model.
Cue 295 — 00:20:35,660 --> 00:20:39,260
And which is more expensive on paper it makes sense right.
Cue 296 — 00:20:39,300 --> 00:20:43,740
Like that. That would work. So you would expect like the same quality lower costs.
Cue 297 — 00:20:43,780 --> 00:20:47,260
Everybody should be happy. And we piloted this.
Cue 298 — 00:20:47,300 --> 00:20:52,160
And the cost per user actually went up about 16%.
Cue 299 — 00:20:52,580 --> 00:20:56,700
That's that's not that's not the savings we expected.
Cue 300 — 00:20:56,740 --> 00:21:03,160
So we dug into why. So there were two reasons and both came out of the data itself.
Cue 301 — 00:21:03,380 --> 00:21:06,220
The first is that it overrode people's choices.
Cue 302 — 00:21:06,260 --> 00:21:10,620
And people like the engineers who had deliberately picked the small model,
Cue 303 — 00:21:10,660 --> 00:21:13,780
suddenly had about half of their traffic going to the big one.
Cue 304 — 00:21:13,820 --> 00:21:18,540
And the second is caching, and a prompt cache belongs to one model.
Cue 305 — 00:21:18,580 --> 00:21:23,660
So every time the router switched models and in the middle of the session,
Cue 306 — 00:21:23,700 --> 00:21:27,380
all of that context basically had to be sent to another model.
Cue 307 — 00:21:27,420 --> 00:21:31,180
And so the people on the big model, the cache hit rate dropped,
Cue 308 — 00:21:31,220 --> 00:21:33,740
and the cache misses are obviously super expensive.
Cue 309 — 00:21:33,780 --> 00:21:39,700
And it basically told us about the cost and the told us about the quality of this.
Cue 310 — 00:21:39,740 --> 00:21:44,500
And then what we took away from this is that the defaults and the caching,
Cue 311 — 00:21:44,540 --> 00:21:50,360
they move the bill, but they don't really like apply for a clever routing.
Cue 312 — 00:21:50,660 --> 00:21:55,300
And so remember what was the blind spot with the the AI gateway?
Cue 313 — 00:21:55,340 --> 00:21:59,620
It basically tells you what you spent, but it doesn't tell you what the agent did.
Cue 314 — 00:21:59,660 --> 00:22:02,880
So that's what the second layer is for, which is traces.
Cue 315 — 00:22:12,020 --> 00:22:18,180
And so talking about traces, every agent that you use in the organization and it's
Cue 316 — 00:22:18,220 --> 00:22:22,198
different events. Like if you use Claude Code, Codex, somebody could be using Pi,
Cue 317 — 00:22:22,238 --> 00:22:23,940
somebody could be using Cursor.
Cue 318 — 00:22:23,980 --> 00:22:26,360
They all log things differently.
Cue 319 — 00:22:26,460 --> 00:22:32,460
And so we built Trajectory which is basically a capture layer that turns each agents
Cue 320 — 00:22:32,500 --> 00:22:34,900
events into a common session format.
Cue 321 — 00:22:34,940 --> 00:22:37,620
So it's like a schema for all of these models.
Cue 322 — 00:22:37,660 --> 00:22:41,420
It's open source. It's on GitHub under Datadog Trajectory.
Cue 323 — 00:22:41,460 --> 00:22:42,780
And so it's linked at the end.
Cue 324 — 00:22:42,820 --> 00:22:47,240
So I'll show you that. And so once everything is in that same format,
Cue 325 — 00:22:47,340 --> 00:22:50,380
a session is basically like a distributed trace.
Cue 326 — 00:22:50,420 --> 00:22:56,340
And so in this example the agent loads the context, reads the file edits like let's
Cue 327 — 00:22:56,380 --> 00:22:57,980
say three of them runs the test.
Cue 328 — 00:22:58,020 --> 00:23:03,500
The test fails, it retries, and the retry is where the extra time and tokens went
Cue 329 — 00:23:03,540 --> 00:23:07,860
right. So the gateway would have only seen that the extra tokens were spent.
Cue 330 — 00:23:07,900 --> 00:23:12,340
And by default we only like capture the structure and timing of a session.
Cue 331 — 00:23:12,380 --> 00:23:18,020
So not the prompts or the code, but traces have an advantage that basically we would
Cue 332 — 00:23:18,060 --> 00:23:21,020
be able to see the three trials that are happening across the prompt and things of
Cue 333 — 00:23:21,060 --> 00:23:24,560
that sort. But what is the blind spot for traces?
Cue 334 — 00:23:25,220 --> 00:23:31,660
So any guesses? And so the blind spot for traces is that they will tell you what
Cue 335 — 00:23:31,700 --> 00:23:34,640
actually happened in the prompt, but not exactly.
Cue 336 — 00:23:34,700 --> 00:23:39,000
Is this prompt going to work better with another model or harness.
Cue 337 — 00:23:39,100 --> 00:23:43,200
And so answering that question is basically the job of evals.
Cue 338 — 00:23:44,820 --> 00:23:50,020
And so this is talking about evals I'm sure many of you know here about evals.
Cue 339 — 00:23:50,060 --> 00:23:53,740
And so we I do we run our own evals.
Cue 340 — 00:23:53,780 --> 00:23:57,460
And so the first thing people would ask us is like, why not just look at some
Cue 341 — 00:23:57,500 --> 00:24:00,180
existing public evals and run those?
Cue 342 — 00:24:00,220 --> 00:24:05,540
And the problem is that if you look at the leaderboard like it answers a different
Cue 343 — 00:24:05,580 --> 00:24:09,900
question. So it uses public tasks on public repositories.
Cue 344 — 00:24:09,940 --> 00:24:12,180
So it tests a model basically on its own.
Cue 345 — 00:24:12,220 --> 00:24:15,400
And it mostly just scores pass or fail.
Cue 346 — 00:24:15,660 --> 00:24:18,740
So it tells you which model is better in general.
Cue 347 — 00:24:18,780 --> 00:24:20,780
So if you ask somebody oh, which model do you use?
Cue 348 — 00:24:20,820 --> 00:24:25,642
People like in the previous talk they were talking about are people use Claude Code
Cue 349 — 00:24:25,682 --> 00:24:30,260
or Codex or specific IDEs about which task performs under which model.
Cue 350 — 00:24:30,300 --> 00:24:37,400
They don't have an opinion. And so our engineers, they don't use a model like
Cue 351 — 00:24:37,580 --> 00:24:43,180
on its own. So they use a basically a model inside a harness with our config and our
Cue 352 — 00:24:43,220 --> 00:24:49,060
code. So the evals are basically real, like the company Datadog work in our
Cue 353 — 00:24:49,100 --> 00:24:52,300
repositories and tests. So they have the entire setup together.
Cue 354 — 00:24:52,340 --> 00:24:55,920
And so the question the answer is much like narrow.
Cue 355 — 00:24:56,380 --> 00:25:00,540
Should this be, let's say our default model or right now what's the config.
Cue 356 — 00:25:00,580 --> 00:25:06,020
So the eval is basically going to evaluate the model on the specific task that you
Cue 357 — 00:25:06,060 --> 00:25:09,000
are doing day to day, and check the performance of it.
Cue 358 — 00:25:10,020 --> 00:25:15,320
So what exactly is an eval. Every eval has four parts.
Cue 359 — 00:25:15,700 --> 00:25:20,520
The task is basically a prompt taken from let's say actual work that you're doing
Cue 360 — 00:25:20,740 --> 00:25:25,580
using the framework conventions that you're actually using in a day to day job.
Cue 361 — 00:25:25,620 --> 00:25:30,860
The repository is pinned to a single commit, and so every run starts from exactly
Cue 362 — 00:25:30,900 --> 00:25:36,380
the same code build tools. And the setup under the test is the model plus the
Cue 363 — 00:25:36,420 --> 00:25:42,620
harness plus the config, which is basically how you test, from engineers
Cue 364 — 00:25:42,660 --> 00:25:45,660
perspective. And because like I said, they shipped together.
Cue 365 — 00:25:45,700 --> 00:25:50,380
So the grader, which is the evaluator, basically combines all of the automated
Cue 366 — 00:25:50,420 --> 00:25:56,120
checks with an LLM judge and it basically runs every night.
Cue 367 — 00:25:56,340 --> 00:26:01,780
We pick the tests, we set up the repositories, run the setup multiple times because
Cue 368 — 00:26:01,820 --> 00:26:04,500
of the non determinism and create the run.
Cue 369 — 00:26:04,540 --> 00:26:10,100
And then we have an LLM read the whole run.
Cue 370 — 00:26:10,140 --> 00:26:16,020
And so we explain why did the score result in this kind of a score.
Cue 371 — 00:26:16,060 --> 00:26:20,340
And so there's like a nightly dashboard which we have which compare the setup
Cue 372 — 00:26:20,380 --> 00:26:23,780
against the current defaults. And then check the performance of the model on these
Cue 373 — 00:26:23,820 --> 00:26:28,240
tests. And this includes like the benchmarking completely.
Cue 374 — 00:26:28,300 --> 00:26:32,740
And let's, let's look at it from like a concrete example.
Cue 375 — 00:26:32,780 --> 00:26:38,660
So how does it get scored. So let's say the task is to read a secret using our
Cue 376 — 00:26:38,700 --> 00:26:44,700
current vault pattern. And the known end state is the pattern our docs describe,
Cue 377 — 00:26:44,740 --> 00:26:46,660
which is basically test passing.
Cue 378 — 00:26:46,700 --> 00:26:49,180
So we run the same setup five times.
Cue 379 — 00:26:49,220 --> 00:26:51,380
Each run is scored on four things.
Cue 380 — 00:26:51,420 --> 00:26:56,760
Do the test pass, do the checks on the diff, find the right pattern?
Cue 381 — 00:26:56,900 --> 00:27:02,500
Did the agent take the right actions and what does the judge say?
Cue 382 — 00:27:02,540 --> 00:27:07,700
So the score is not just like a 0 or 1, like a binary like pass or fail.
Cue 383 — 00:27:07,740 --> 00:27:13,060
And look at the run. Two it's interesting, the test passed, but it only got a score
Cue 384 — 00:27:13,100 --> 00:27:19,540
of 0.6. And the step five is basically the explanation because it copied an older
Cue 385 — 00:27:19,580 --> 00:27:24,180
vault pattern from a neighboring service instead of the documented one.
Cue 386 — 00:27:24,220 --> 00:27:30,820
And so this explanation is what lets us group failures across thousands of runs and
Cue 387 — 00:27:30,860 --> 00:27:36,340
see patterns and running evals like this every night is what changed our default
Cue 388 — 00:27:36,380 --> 00:27:38,160
model at Datadog twice.
Cue 389 — 00:27:40,100 --> 00:27:45,940
And so, like I said, actually changed the default model at Datadog twice because we
Cue 390 — 00:27:45,980 --> 00:27:49,820
started on a larger model, and the eval showed that a smaller model got the same
Cue 391 — 00:27:49,860 --> 00:27:53,620
results on most tasks while being faster and cheaper.
Cue 392 — 00:27:53,660 --> 00:27:58,060
So we move the default to a smaller model, and few months later,
Cue 393 — 00:27:58,100 --> 00:28:03,420
a newer, larger model came out, and in our evals it was about half the cost of the
Cue 394 — 00:28:03,460 --> 00:28:06,060
smaller one. And so we switched again.
Cue 395 — 00:28:06,100 --> 00:28:12,100
So it's the same harness and the same tasks and a different answer basically that's
Cue 396 — 00:28:12,140 --> 00:28:17,700
evolving. So the evolve the eval result has a shelf life if it makes sense because
Cue 397 — 00:28:17,740 --> 00:28:22,980
the models improve and they need less steering harnesses make tool calls more
Cue 398 — 00:28:23,020 --> 00:28:27,700
reliable. And so it's worth rerunning everything when the model changes.
Cue 399 — 00:28:27,740 --> 00:28:32,420
And this is basically another example.
Cue 400 — 00:28:32,460 --> 00:28:35,340
So this is the compression proxy from the timeline.
Cue 401 — 00:28:35,380 --> 00:28:39,200
And the reason to compress. Why would you want to compress is because
Cue 402 — 00:28:40,660 --> 00:28:46,340
if you have like less amount of data per request, it would basically reduce the cost
Cue 403 — 00:28:46,380 --> 00:28:48,460
of your request significantly.
Cue 404 — 00:28:48,500 --> 00:28:51,980
And so if you look at this diagram, each dot is one eval result.
Cue 405 — 00:28:52,020 --> 00:28:57,460
And the cost went down by more than half and the average score remained about the
Cue 406 — 00:28:57,500 --> 00:29:01,700
same. So if it only looked at the average we would have shipped it.
Cue 407 — 00:29:01,740 --> 00:29:04,740
But two tasks actually drop down to zero.
Cue 408 — 00:29:04,780 --> 00:29:09,620
If you look at the left and the compression actually removes something those tasks
Cue 409 — 00:29:09,660 --> 00:29:13,780
dependent on so evals actually let us catch that before, you know,
Cue 410 — 00:29:13,820 --> 00:29:16,020
rolling it out across the organization.
Cue 411 — 00:29:16,060 --> 00:29:18,980
So it was actually run on a fixed set of tasks.
Cue 412 — 00:29:19,020 --> 00:29:21,700
It's not what engineers do day to day.
Cue 413 — 00:29:21,740 --> 00:29:26,960
And so the last layer is basically rollouts, which is where we test the real work.
Cue 414 — 00:29:27,900 --> 00:29:31,860
And so we treat the config change just like a code deployment.
Cue 415 — 00:29:31,900 --> 00:29:35,260
Basically every variant it gets a version tag.
Cue 416 — 00:29:35,300 --> 00:29:38,460
And that tag rides along with every model call.
Cue 417 — 00:29:38,500 --> 00:29:42,160
Small group gets it first. It's a very controlled rollout.
Cue 418 — 00:29:42,300 --> 00:29:44,780
Use device management group for that.
Cue 419 — 00:29:44,820 --> 00:29:47,700
And then we compare the test group against the base group.
Cue 420 — 00:29:47,740 --> 00:29:53,260
And so only after that, like all of this process, it becomes a default model for
Cue 421 — 00:29:53,300 --> 00:29:56,980
everyone. And because every call carries the config version,
Cue 422 — 00:29:57,020 --> 00:30:02,020
every dashboard already has the split without building anything from new.
Cue 423 — 00:30:02,060 --> 00:30:06,160
And so here's a rollout that gave us an answer quickly.
Cue 424 — 00:30:06,700 --> 00:30:10,720
This is the RTK test that I was talking about from the timeline.
Cue 425 — 00:30:11,300 --> 00:30:16,620
And we compared them across a group of days, and the test group actually made more
Cue 426 — 00:30:16,660 --> 00:30:20,140
requests, hit more errors and ended up costing more.
Cue 427 — 00:30:20,180 --> 00:30:23,820
So shrinking one of the bill made the others grow.
Cue 428 — 00:30:23,860 --> 00:30:27,020
And it's the same multiplication that I talked about from the cost slide.
Cue 429 — 00:30:27,060 --> 00:30:31,820
And so because both groups ran side by side, we had a clear answer in a week.
Cue 430 — 00:30:31,860 --> 00:30:36,500
And so that leaves the slowest signal of all, which is the outcome.
Cue 431 — 00:30:36,540 --> 00:30:38,540
And the outcomes could take weeks.
Cue 432 — 00:30:38,580 --> 00:30:41,900
They're also basically the easiest way to fool yourself.
Cue 433 — 00:30:41,940 --> 00:30:46,780
And so we have to be very careful here because we tested a bunch of candidate like
Cue 434 — 00:30:46,820 --> 00:30:51,660
code quality signals. And so the question that we want to answer here is does this
Cue 435 — 00:30:51,700 --> 00:30:58,020
PR need a follow up PR. And so most of the PR did not hold up because they needed
Cue 436 — 00:30:58,060 --> 00:31:03,900
some of the other bug fixing. And the numbers say that only one out of ten actually
Cue 437 — 00:31:03,940 --> 00:31:06,400
held up. So that's pretty shocking.
Cue 438 — 00:31:06,780 --> 00:31:12,620
And so let me put the four layers together, which is basically on the left.
Cue 439 — 00:31:12,660 --> 00:31:16,420
Think of every change coming in and going through the four checkpoints,
Cue 440 — 00:31:16,460 --> 00:31:20,100
the gateway it catches, the cost and failure spikes.
Cue 441 — 00:31:20,140 --> 00:31:24,900
That was the routing. And so the trace is they catch the loops and the stalls like
Cue 442 — 00:31:24,940 --> 00:31:31,300
retry loops. The evals basically catch the quality regressions and rollouts which
Cue 443 — 00:31:31,340 --> 00:31:33,280
catch the real world breakage.
Cue 444 — 00:31:33,580 --> 00:31:37,380
And so only anything that passes all four of them becomes the default.
Cue 445 — 00:31:37,420 --> 00:31:41,340
And that covers basically the adoption, the productivity and the cost.
Cue 446 — 00:31:41,380 --> 00:31:44,500
And so I said I'd come back to the fourth question, security.
Cue 447 — 00:31:44,540 --> 00:31:50,820
So here it is. Our answer to security is the same idea as everything else.
Cue 448 — 00:31:50,860 --> 00:31:52,800
And so the question,
Cue 449 — 00:31:54,140 --> 00:31:58,480
every guardrail is basically a central default, so nobody has to remember it.
Cue 450 — 00:31:58,700 --> 00:32:05,300
Who's calling it? Every call basically carries a short lived token tied to one
Cue 451 — 00:32:05,340 --> 00:32:10,660
person. And which model? Like so agents are configured for approved.
Cue 452 — 00:32:10,700 --> 00:32:16,100
Why does only which tools MCP servers get vetted before they touch anything.
Cue 453 — 00:32:16,140 --> 00:32:23,060
And yeah, so basically what merges is like auto approval only happens when every
Cue 454 — 00:32:23,100 --> 00:32:26,180
security, reliability and compliance check has passed.
Cue 455 — 00:32:26,220 --> 00:32:30,580
So with all the four covered, let me go back to the questions from the start and
Cue 456 — 00:32:30,620 --> 00:32:37,440
give you the answers. So adoption my answer is make it the default and then
Cue 457 — 00:32:37,860 --> 00:32:42,700
see how it pans. Basically who who stays using them, the configuration and who is
Cue 458 — 00:32:42,740 --> 00:32:46,480
not using it. And that's basically controlled by device management,
Cue 459 — 00:32:46,700 --> 00:32:49,780
the gateway and the traces. They will show you the weekly usage.
Cue 460 — 00:32:49,820 --> 00:32:54,860
And if people are dropping out, that's basically your warning productivity.
Cue 461 — 00:32:54,900 --> 00:32:58,720
You should just change by outcomes and not activity.
Cue 462 — 00:32:59,100 --> 00:33:02,960
So merged and kept PRs, not reverted PRs.
Cue 463 — 00:33:03,860 --> 00:33:10,660
And so use evals before releasing and small rollout before everyone cost manage
Cue 464 — 00:33:10,700 --> 00:33:13,040
the unit costs, not the total.
Cue 465 — 00:33:13,260 --> 00:33:19,780
For us, defaults and caching moves the bill more than routing and security.
Cue 466 — 00:33:19,820 --> 00:33:22,660
Make every guardrail a central default.
Cue 467 — 00:33:22,700 --> 00:33:29,100
And so here's what I would suggest if the takeaways from this talk overall.
Cue 468 — 00:33:29,140 --> 00:33:34,340
So one is measure the outcome not activity because tokens explain where the money
Cue 469 — 00:33:34,380 --> 00:33:36,520
went but they don't prove really value.
Cue 470 — 00:33:36,620 --> 00:33:42,500
And two set up a gateway and a trace format because you don't want to be changing
Cue 471 — 00:33:42,540 --> 00:33:45,180
between models. And then trying to work out the schema.
Cue 472 — 00:33:45,220 --> 00:33:48,800
Being able to compare across agents really matters a lot.
Cue 473 — 00:33:48,900 --> 00:33:53,320
Three benchmark on your own work, which is basically the real tasks.
Cue 474 — 00:33:53,740 --> 00:33:59,900
And look at the failures, not just the averages Four, fix your defaults and caching
Cue 475 — 00:33:59,940 --> 00:34:01,860
before you try anything clever.
Cue 476 — 00:34:01,900 --> 00:34:07,780
And five ship configuration like code, which means like version it,
Cue 477 — 00:34:07,820 --> 00:34:12,220
roll it out to a small group first, watch it, make it easy to roll back,
Cue 478 — 00:34:12,260 --> 00:34:17,900
and everything I mentioned in the resources today is covered in this slide if you're
Cue 479 — 00:34:17,940 --> 00:34:24,580
interested. So this has links to some of those important blogs and do try
Cue 480 — 00:34:24,620 --> 00:34:28,620
Trajectory. It's open source and that's it from me.
Cue 481 — 00:34:28,660 --> 00:34:30,120
Thank you all for listening.
.tessl-plugin
skills
talk-arcolano-autonomy-and-trust
talk-aroussi-decision-context
talk-bevin-agent-assisted-instrument-design
talk-boeckeler-context-lifecycles
talk-braun-context-graphs
talk-carmont-evolving-agent-harnesses
talk-chanezon-gordon-agent-experience-evaluation
talk-cox-secure-agentic-workflows
talk-cunniffe-reusable-agent-platform
talk-davis-agent-tool-policies
talk-debois-concurrent-delivery
talk-dzialets-team-automation
talk-fahmy-harness-api-design
talk-finster-continuous-delivery
talk-fowler-regenerative-software
talk-gavrilescu-factory-architecture
talk-graciolli-amin-verification-aware-programming
talk-gruenbaum-reverse-graph-engineering
talk-hassan-agentic-engineering
talk-hossain-enterprise-agent-adoption
talk-huntley-patch-ready-factories
talk-jain-verifying-intent
talk-katsioloudes-montei-agentic-lifecycle
talk-knox-harness-engineering
talk-kovac-deterministic-memory
talk-lee-perkins-code-readiness
talk-lorenc-machine-speed-security
talk-lubken-software-of-one
talk-meijer-software-specialization
talk-meppiel-versioned-agent-contracts
talk-moss-concurrent-agent-risk
talk-myers-understanding-agent-generated-code
talk-ocytko-scaling-agentic-engineering
talk-oza-automating-design-quality
talk-panzica-agentic-sdlc
talk-pari-patel-agent-awareness
talk-podjarny-trustworthy-software-factories
talk-poulter-factory-improvement
talk-pratik-patel-refined-rag-data
talk-re-ferre-development-platforms
talk-sadogursky-factory-of-factories
talk-shah-training-agent-skills
talk-sharma-mohapatra-evolving-intent
talk-singhal-operating-agents-at-scale
talk-stack-incident-dossiers
talk-tyler-team-coding-benchmarks
talk-wotherspoon-agent-execution-environments
talk-yegge-ai-powered-factories
talk-zuber-leadership-through-coding