CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-singhal-operating-agents-at-scale/references/

Corrected transcript: Operating Coding Agents at Scale

Nandini Singhal · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=BtGqqd7rbAI

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:00,220 --> 00:00:02,080

Hi everyone. Thank you so much.

Cue 2 — 00:00:02,540 --> 00:00:07,180

My name is Nandini and I am a software engineer at Datadog and I work on the

Cue 3 — 00:00:07,220 --> 00:00:10,940

streaming platform team. So my day to day job is working on Kafka.

Cue 4 — 00:00:10,980 --> 00:00:12,780

But that's not what this talk is about.

Cue 5 — 00:00:12,820 --> 00:00:16,220

So this talk is about operating coding agents at scale.

Cue 6 — 00:00:16,260 --> 00:00:21,340

And like a lot of companies, we've given coding agents to pretty much all of our

Cue 7 — 00:00:21,380 --> 00:00:26,300

engineers. And it turns out enabling the coding agent is sort of an easy job.

Cue 8 — 00:00:26,340 --> 00:00:29,600

But what I want to talk about is what comes after that.

Cue 9 — 00:00:30,220 --> 00:00:32,900

So we keep changing the set up every week.

Cue 10 — 00:00:32,940 --> 00:00:37,580

And the hard part is actually knowing whether each of those changes helped.

Cue 11 — 00:00:37,620 --> 00:00:40,520

And I'm going to be covering what changes I'm talking about.

Cue 12 — 00:00:41,060 --> 00:00:45,500

What does a good change look like and what do we do about that?

Cue 13 — 00:00:45,540 --> 00:00:50,820

So let me get started. So at Datadog we have thousands of engineers which are using

Cue 14 — 00:00:50,860 --> 00:00:54,060

coding agents. And it's not a single setup for every engineer.

Cue 15 — 00:00:54,100 --> 00:00:58,100

Right. Because every engineer has their own setup of different agents.

Cue 16 — 00:00:58,140 --> 00:01:00,780

Different terminals could be different configurations.

Cue 17 — 00:01:00,820 --> 00:01:06,020

Different models could be some people using open weight models and some have each

Cue 18 — 00:01:06,060 --> 00:01:08,220

different settings, different parameters.

Cue 19 — 00:01:08,260 --> 00:01:14,180

And the thing that makes this interesting is on the right when the platform changes

Cue 20 — 00:01:14,220 --> 00:01:18,860

centrally, like at an organization level, say let's say the default model is

Cue 21 — 00:01:18,900 --> 00:01:23,060

changed, or let's say the reasoning effort, or let's say the company has decided to

Cue 22 — 00:01:23,100 --> 00:01:27,260

put a cap on the context, or let's say we decided to try out a new plugin,

Cue 23 — 00:01:27,300 --> 00:01:33,660

or we decided to try out a new MCP tool that change lands on every single engineers

Cue 24 — 00:01:33,700 --> 00:01:38,980

machine. And so we are constantly making those changes every week as new models are

Cue 25 — 00:01:39,020 --> 00:01:42,020

evolving, new things and technologies are coming up.

Cue 26 — 00:01:42,060 --> 00:01:47,020

And so this talk is really about how we figured out what each of those changes did.

Cue 27 — 00:01:47,060 --> 00:01:49,980

And so people talk a lot about the effectiveness of AI.

Cue 28 — 00:01:50,020 --> 00:01:52,860

But what is it exactly mean and how do we measure that.

Cue 29 — 00:01:52,900 --> 00:01:58,020

And so before going into that, it helps to see where exactly agents are being used,

Cue 30 — 00:01:58,060 --> 00:02:02,100

because it's not just really the engineers and their laptops that that are using

Cue 31 — 00:02:02,140 --> 00:02:07,140

these AI agents. Right. So where exactly agents are being used today.

Cue 32 — 00:02:07,180 --> 00:02:11,460

So they show up in different stack, different places in the stack.

Cue 33 — 00:02:11,500 --> 00:02:16,300

And I've ordered them by how much control we have at the top of the pipeline.

Cue 34 — 00:02:16,340 --> 00:02:22,940

We have an every pull request for code review and the risk scoring and let's say the

Cue 35 — 00:02:22,980 --> 00:02:24,820

auto approval for lower risk changes.

Cue 36 — 00:02:24,860 --> 00:02:26,340

So that's the top of the stack.

Cue 37 — 00:02:26,380 --> 00:02:30,260

And then you have the operational layer when something breaks investigating

Cue 38 — 00:02:30,300 --> 00:02:34,920

incidents, triaging alerts and checking where let's say a deployment went bad.

Cue 39 — 00:02:35,260 --> 00:02:39,220

And then you have like managed agents which are basically running on a schedule.

Cue 40 — 00:02:39,260 --> 00:02:44,860

And so they do things like vulnerability fixes, adding, let's say missing test cases

Cue 41 — 00:02:44,900 --> 00:02:48,860

or cleaning a bad code, which is maybe dead or unused.

Cue 42 — 00:02:48,900 --> 00:02:52,620

And so at the bottom layer is basically where the engineers lie,

Cue 43 — 00:02:52,660 --> 00:02:57,540

which is basically the interactive agents, like the ID or the CLI that the engineers

Cue 44 — 00:02:57,580 --> 00:03:02,100

use on a day to day basis. And so the arrow is really important because it shows how

Cue 45 — 00:03:02,140 --> 00:03:07,200

much control the we have over the model and the configuration.

Cue 46 — 00:03:07,340 --> 00:03:09,260

And so at the top we pick the model.

Cue 47 — 00:03:09,300 --> 00:03:13,740

And so we control the cost and the quality across the organization and the bottom.

Cue 48 — 00:03:13,780 --> 00:03:15,380

It's basically the engineers choice.

Cue 49 — 00:03:15,420 --> 00:03:21,660

Right. So it's it's that's where most of the usage is and also where it's very hard

Cue 50 — 00:03:21,700 --> 00:03:26,720

to measure. And so most of what I talk about today is that bottom layer.

Cue 51 — 00:03:26,980 --> 00:03:30,260

And whenever we compare notes and data with other companies,

Cue 52 — 00:03:30,300 --> 00:03:32,700

the same four questions keep popping up.

Cue 53 — 00:03:32,740 --> 00:03:37,020

And what are those questions? And I hear everybody here talk about these four

Cue 54 — 00:03:37,060 --> 00:03:40,700

questions, which is like who who is actually using this?

Cue 55 — 00:03:40,740 --> 00:03:45,180

And like what's the adoption rate of a particular model or let's say configuration.

Cue 56 — 00:03:45,220 --> 00:03:49,780

And does it really stick? And the second question is like the productivity aspect,

Cue 57 — 00:03:49,820 --> 00:03:54,380

is it really helping or is it just producing more code for people to review?

Cue 58 — 00:03:54,420 --> 00:03:57,220

And the third question is basically about the cost.

Cue 59 — 00:03:57,260 --> 00:03:59,540

And the cost question is super important as well.

Cue 60 — 00:03:59,580 --> 00:04:04,340

Like to understand why is the bill of AI growing and what you can actually do about

Cue 61 — 00:04:04,380 --> 00:04:08,380

it. And the fourth question is also super important, which is about security.

Cue 62 — 00:04:08,420 --> 00:04:10,900

And the security question is about is it safe?

Cue 63 — 00:04:10,940 --> 00:04:16,340

Who can call which model? Where does our code go, who can merge the code on its own

Cue 64 — 00:04:16,380 --> 00:04:19,660

and things of that sort. And so I'm going to use these four questions as the

Cue 65 — 00:04:19,700 --> 00:04:24,620

backbone of the talk. And for each one I'll show you why it's really difficult and

Cue 66 — 00:04:24,660 --> 00:04:28,980

then how we address it. And then right at the end I'll come back and summarize what

Cue 67 — 00:04:29,020 --> 00:04:33,100

we learned. And so we've just done the four questions.

Cue 68 — 00:04:33,140 --> 00:04:37,020

And so I'll go through why each one is hard.

Cue 69 — 00:04:37,060 --> 00:04:40,840

And then I'll show you our approach and how it evolved over time.

Cue 70 — 00:04:41,500 --> 00:04:45,540

And so the main part of the talk, which is basically the four layers which I'm going

Cue 71 — 00:04:45,580 --> 00:04:50,640

to be talking about. So let's start with the first question which is adoption.

Cue 72 — 00:04:50,740 --> 00:04:52,980

And so adoptions really simple right.

Cue 73 — 00:04:53,020 --> 00:04:56,380

Like adoption sounds like it should be one simple number like how many people use

Cue 74 — 00:04:56,420 --> 00:05:00,780

the model. And it's really actually not because it's a funnel,

Cue 75 — 00:05:00,820 --> 00:05:07,420

because it each step decides like, does someone even have access to the model?

Cue 76 — 00:05:07,460 --> 00:05:11,980

First of all, and at an organization level, that's basically like device management.

Cue 77 — 00:05:12,020 --> 00:05:16,860

And so let's say the company decides to install this manage setting to every laptop.

Cue 78 — 00:05:16,900 --> 00:05:21,120

And then they did they actually like set it up.

Cue 79 — 00:05:21,500 --> 00:05:23,780

And we how can we identify that.

Cue 80 — 00:05:23,820 --> 00:05:28,220

So we identify that because the gateway sees their first call that they have signed

Cue 81 — 00:05:28,260 --> 00:05:30,020

up with their own personal token.

Cue 82 — 00:05:30,060 --> 00:05:34,460

And then we have to also measure their usage that they use it actually this week.

Cue 83 — 00:05:34,500 --> 00:05:36,640

And that's basically sessions.

Cue 84 — 00:05:37,020 --> 00:05:40,540

And so you have to categorize the sessions by let's say the team by what kind of

Cue 85 — 00:05:40,580 --> 00:05:43,260

work they're doing, which comes from, again, the gateway.

Cue 86 — 00:05:43,300 --> 00:05:48,300

And there is a tool called Trajectory, which is basically the gateway for Datadog

Cue 87 — 00:05:48,340 --> 00:05:53,400

and which is our like the session capture tool and more on that will come later.

Cue 88 — 00:05:53,740 --> 00:05:58,660

And the bottom layer is basically really using it a month later.

Cue 89 — 00:05:58,700 --> 00:06:03,220

And that's basically retention because people can start like getting hyped up about

Cue 90 — 00:06:03,260 --> 00:06:06,020

a model or a tool, and then they're not even using it.

Cue 91 — 00:06:06,060 --> 00:06:10,500

So it's important to understand the adoption rate.

Cue 92 — 00:06:10,540 --> 00:06:15,280

And the question two is basically productivity, which is where it gets hard.

Cue 93 — 00:06:16,060 --> 00:06:20,720

And I want to start productivity with an outside view because it sets expectations.

Cue 94 — 00:06:20,900 --> 00:06:24,660

There is a study from DORA, which is basically the Google Cloud's research group

Cue 95 — 00:06:24,700 --> 00:06:31,100

behind the State of DevOps report, and they put out a report in April on the ROI of

Cue 96 — 00:06:31,140 --> 00:06:33,380

AI assisted software development.

Cue 97 — 00:06:33,420 --> 00:06:39,220

And one of their findings is that most organizations get a bit worse before they get

Cue 98 — 00:06:39,260 --> 00:06:41,580

better. And so there's always a dip.

Cue 99 — 00:06:41,620 --> 00:06:44,820

And teams are like learning new workflows and there's more code.

Cue 100 — 00:06:44,860 --> 00:06:49,460

So there's more to review and testing and approvals become more bottleneck.

Cue 101 — 00:06:49,500 --> 00:06:53,740

And so the point is that the gains come from the engineering system,

Cue 102 — 00:06:53,780 --> 00:06:57,300

which is built around the tool and not from like getting a better tool.

Cue 103 — 00:06:57,340 --> 00:07:02,580

And I find that really reassuring because when someone asks why things have not

Cue 104 — 00:07:02,620 --> 00:07:09,100

magically improved, like from past two months ago, it means that the dip is a real

Cue 105 — 00:07:09,140 --> 00:07:12,180

problem. And so that's what really the the talk is about.

Cue 106 — 00:07:12,220 --> 00:07:18,540

And we will see that the review bottleneck is real in the next number that we are

Cue 107 — 00:07:18,580 --> 00:07:24,380

going to see. And so the bottleneck is this is a study again from Faros,

Cue 108 — 00:07:24,420 --> 00:07:29,700

which they published a study about the AI productivity paradox based on the

Cue 109 — 00:07:29,740 --> 00:07:32,780

telemetry from over 10,000 individuals.

Cue 110 — 00:07:32,820 --> 00:07:39,060

And it looks great. It shows that 21% tasks were more were completed almost daily.

Cue 111 — 00:07:39,100 --> 00:07:43,580

2x the number of pull requests are getting merged, and PRs that are about two and a

Cue 112 — 00:07:43,620 --> 00:07:49,300

half times bigger. But but review time is also increased significantly.

Cue 113 — 00:07:49,340 --> 00:07:52,540

Almost 90% of the team's time has now increased in reviews.

Cue 114 — 00:07:52,580 --> 00:07:56,460

And the bugs which are generated per developer have gone up as well.

Cue 115 — 00:07:56,500 --> 00:08:01,020

And so the company level delivery is not significantly improved.

Cue 116 — 00:08:01,060 --> 00:08:05,940

And so more output from the agent doesn't automatically translate into more shipped

Cue 117 — 00:08:05,980 --> 00:08:10,020

or production code. And a lot of it just becomes more code for somebody else to

Cue 118 — 00:08:10,060 --> 00:08:17,020

review. And so the numbers most teams look at first can tell those two situations

Cue 119 — 00:08:17,060 --> 00:08:19,920

apart. So how do you measure productivity.

Cue 120 — 00:08:19,980 --> 00:08:24,140

And so these are the three numbers everybody builds a dashboard for.

Cue 121 — 00:08:24,180 --> 00:08:27,340

Basically they build a dashboard which is like having the number of tokens,

Cue 122 — 00:08:27,380 --> 00:08:29,500

the number of sessions and tool calls.

Cue 123 — 00:08:29,540 --> 00:08:33,780

And they all go up and to the right, which feels great.

Cue 124 — 00:08:33,820 --> 00:08:37,740

And the catch is that each one can go up for a good reason or a bad reason.

Cue 125 — 00:08:37,780 --> 00:08:40,460

And the chart looks exactly the same either way.

Cue 126 — 00:08:40,500 --> 00:08:45,820

So tokens go up when real work is happening, but also when the context is bloated.

Cue 127 — 00:08:45,860 --> 00:08:50,700

And so sessions go up when adoption grows, but also when people keep retrying and

Cue 128 — 00:08:50,740 --> 00:08:55,820

tool calls go up when the agent is making progress, but also when it's lost.

Cue 129 — 00:08:55,860 --> 00:09:02,400

And so a single agent stuck on a flaky test pushes all of these metrics.

Cue 130 — 00:09:02,780 --> 00:09:08,580

And so none of these metrics really tell you anything about how much of the code

Cue 131 — 00:09:08,620 --> 00:09:12,020

actually got shipped. And so even when you pick better metrics,

Cue 132 — 00:09:12,060 --> 00:09:16,214

there is a second problem, which is tying a result back to the change that caused

Cue 133 — 00:09:16,254 --> 00:09:17,054

it.

Cue 134 — 00:09:18,220 --> 00:09:21,380

And so why is it so hard to tell what a change did.

Cue 135 — 00:09:21,420 --> 00:09:25,220

So there are three reasons. It's hard to say what any single change did.

Cue 136 — 00:09:25,260 --> 00:09:29,340

First, the model and the harness often shipped together.

Cue 137 — 00:09:29,380 --> 00:09:34,340

So when a new Claude Code, let's say, release comes out, it often comes up with a

Cue 138 — 00:09:34,380 --> 00:09:37,500

new model. And so you can't always change like one thing at a time.

Cue 139 — 00:09:37,540 --> 00:09:40,940

What happens is agents are not deterministic.

Cue 140 — 00:09:40,980 --> 00:09:44,820

So you if you run the same task twice and you get different results.

Cue 141 — 00:09:44,860 --> 00:09:48,540

So one good run of a task really doesn't prove much.

Cue 142 — 00:09:48,580 --> 00:09:53,340

And so the main important thing is that the outcomes that we care about,

Cue 143 — 00:09:53,380 --> 00:09:56,820

they come very late. And which is basically did the PR get reviewed.

Cue 144 — 00:09:56,860 --> 00:10:00,040

Did the PR get merged and did the PR get reverted?

Cue 145 — 00:10:00,820 --> 00:10:04,340

And that reward could happen, let's say days or weeks after the merge.

Cue 146 — 00:10:04,380 --> 00:10:07,080

And so who's tracking all of that really?

Cue 147 — 00:10:07,620 --> 00:10:13,840

And it's not just us. METR is struggling with this too, which is basically a

Cue 148 — 00:10:13,980 --> 00:10:19,300

research organization who can, like, run control studies on the AI and developer

Cue 149 — 00:10:19,340 --> 00:10:25,100

productivity. They published a study and basically that they're redesigning their

Cue 150 — 00:10:25,140 --> 00:10:31,220

experiment because so many now developers won't work without AI that the comparison

Cue 151 — 00:10:31,260 --> 00:10:33,280

study that they have is broken.

Cue 152 — 00:10:33,460 --> 00:10:36,820

And so people are running several agents at once.

Cue 153 — 00:10:36,860 --> 00:10:42,780

And so we at Datadog have stopped waiting for a perfect answer.

Cue 154 — 00:10:42,820 --> 00:10:46,820

And so we measure continuously inside our own system instead.

Cue 155 — 00:10:46,860 --> 00:10:52,680

And so the first step is being clear about what each kind of metric is actually for.

Cue 156 — 00:10:53,780 --> 00:10:56,860

So we gave our metrics two separate jobs.

Cue 157 — 00:10:56,900 --> 00:11:00,840

So outcomes are basically what we decide.

Cue 158 — 00:11:01,260 --> 00:11:05,200

So what is the outcome. Did the change get merged and did it stay merged.

Cue 159 — 00:11:05,420 --> 00:11:07,580

Did it need a follow up or did it get reverted.

Cue 160 — 00:11:07,620 --> 00:11:10,900

And what did it cost per merge change.

Cue 161 — 00:11:10,940 --> 00:11:14,540

So those are the important numbers that we actually make the calls on.

Cue 162 — 00:11:14,580 --> 00:11:18,940

So activity is what explains when the outcome is delivered.

Cue 163 — 00:11:18,980 --> 00:11:22,240

And tokens basically tell us where the money is spent.

Cue 164 — 00:11:22,860 --> 00:11:28,220

Sessions basically tell us how people are working and the tool that the tool agent

Cue 165 — 00:11:28,260 --> 00:11:32,860

calls. They tell us what the agent was actually doing because let's be real,

Cue 166 — 00:11:32,900 --> 00:11:37,620

the agent could be just, let's say, spinning up or making a lot of redundant calls.

Cue 167 — 00:11:37,660 --> 00:11:42,580

So I look at activity dashboards all the time, and they're really useful for

Cue 168 — 00:11:42,620 --> 00:11:48,400

debugging. And we just don't use them to declare that something worked.

Cue 169 — 00:11:48,860 --> 00:11:54,660

So that's productivity. Which basically brings us to question number three,

Cue 170 — 00:11:54,700 --> 00:12:01,500

which is basically cost. And the reason cost is confusing is that the bill of the

Cue 171 — 00:12:01,540 --> 00:12:04,100

AI agent is basically a multiplication.

Cue 172 — 00:12:04,140 --> 00:12:07,100

And this is a graph which I really like.

Cue 173 — 00:12:07,140 --> 00:12:11,980

And this breakdown is basically adapted from a post that Uber Engineering published.

Cue 174 — 00:12:12,020 --> 00:12:17,780

And if you look at this graph, it's very interesting because the spend is actually

Cue 175 — 00:12:17,820 --> 00:12:25,020

the users times sessions per user times tokens per request,

Cue 176 — 00:12:25,060 --> 00:12:30,860

times price per token, and each term multiplies the next.

Cue 177 — 00:12:30,900 --> 00:12:34,980

And so the first two in green are the adoption.

Cue 178 — 00:12:35,020 --> 00:12:37,860

And you want those to keep growing in an organization.

Cue 179 — 00:12:37,900 --> 00:12:42,540

And the middle three in purple are basically how much work the agent does per

Cue 180 — 00:12:42,580 --> 00:12:48,140

session. And those you actually want to shrink because you want fewer retry loops.

Cue 181 — 00:12:48,180 --> 00:12:50,020

You want the right docs in context.

Cue 182 — 00:12:50,060 --> 00:12:52,300

So it needs the fewer requests.

Cue 183 — 00:12:52,340 --> 00:12:54,900

And you want caching and you want compression.

Cue 184 — 00:12:54,940 --> 00:12:57,340

So each request is basically small.

Cue 185 — 00:12:57,380 --> 00:13:01,300

And the last one is basically something that you don't have control over because the

Cue 186 — 00:13:01,340 --> 00:13:06,820

last one is price per token, which you mostly choose via the default model that you

Cue 187 — 00:13:06,860 --> 00:13:10,220

are setting as a benchmark for your organization.

Cue 188 — 00:13:10,260 --> 00:13:16,460

So when the total goes up, that alone doesn't tell you if it's good news or bad

Cue 189 — 00:13:16,500 --> 00:13:21,260

news, right? And most of these terms aren't really controlled by individual

Cue 190 — 00:13:21,300 --> 00:13:25,020

engineers. They controlled by our default settings.

Cue 191 — 00:13:25,060 --> 00:13:27,940

And so that's where defaults are super important.

Cue 192 — 00:13:27,980 --> 00:13:30,240

Which brings me to my next slide.

Cue 193 — 00:13:31,660 --> 00:13:34,400

The default is a product decision.

Cue 194 — 00:13:34,580 --> 00:13:38,740

And so every row here in this table if you look at it is a setting that we change

Cue 195 — 00:13:38,780 --> 00:13:40,840

centrally in the organization.

Cue 196 — 00:13:40,900 --> 00:13:46,580

And every one of them saves something which is costing something else.

Cue 197 — 00:13:46,620 --> 00:13:51,780

So it's like a trade off. And so if you choose to decide to go with a cheaper

Cue 198 — 00:13:51,820 --> 00:13:57,980

default model, it lowers the price per token, but it can hurt the quality on hard

Cue 199 — 00:13:58,020 --> 00:14:03,460

tasks. If you go with lower reasoning, it cuts the output tokens,

Cue 200 — 00:14:03,500 --> 00:14:10,220

but it can cost you correctness, a tighter context cap, or aggressive compaction.

Cue 201 — 00:14:10,260 --> 00:14:13,980

It can make your request smaller, but it can break prompt caching,

Cue 202 — 00:14:14,020 --> 00:14:17,020

and maybe it can make the agent lose track.

Cue 203 — 00:14:17,060 --> 00:14:22,540

So routing requests to different models or using cheaper models for sub agents.

Cue 204 — 00:14:22,580 --> 00:14:26,200

It saves on price but hurts cache reuse.

Cue 205 — 00:14:26,420 --> 00:14:29,140

And so I'll show you a real example of that in a few minutes.

Cue 206 — 00:14:29,180 --> 00:14:36,020

But even adding helpful plugins, skills, or MCP tools has a cost because their

Cue 207 — 00:14:36,060 --> 00:14:39,160

definitions are sent along with every request.

Cue 208 — 00:14:39,300 --> 00:14:45,500

And so a default setting reaches every engineer in the organization at once.

Cue 209 — 00:14:45,540 --> 00:14:50,540

And each of this, in my opinion, is a product decision, not just a configuration

Cue 210 — 00:14:50,580 --> 00:14:54,080

tweak that you can say, okay, you can roll out an organization because,

Cue 211 — 00:14:55,060 --> 00:14:59,620

that defines the cost. Which brings us to the question number four,

Cue 212 — 00:14:59,660 --> 00:15:03,820

which is security. And I'll come back to that near the end, because our answer there

Cue 213 — 00:15:03,860 --> 00:15:06,040

is again mostly about defaults.

Cue 214 — 00:15:06,140 --> 00:15:08,860

So that's why these questions are hard.

Cue 215 — 00:15:08,900 --> 00:15:11,500

Now how do we actually measure all of these.

Cue 216 — 00:15:11,540 --> 00:15:14,820

And our setup has actually built over time.

Cue 217 — 00:15:14,860 --> 00:15:19,700

It's like one piece at a time that has taken iterations and evolved.

Cue 218 — 00:15:19,740 --> 00:15:22,640

And so how did our approach evolve?

Cue 230 — 00:16:05,140 --> 00:16:12,020

So here's how it evolved. We started by routing agent traffic through a single AI

Cue 231 — 00:16:12,060 --> 00:16:15,780

gateway, so every model call went through one place.

Cue 232 — 00:16:15,820 --> 00:16:17,740

So we tried a couple of cost ideas.

Cue 233 — 00:16:17,780 --> 00:16:22,720

We tried a model router. We tried a token reduction tool called RTK.

Cue 234 — 00:16:22,940 --> 00:16:28,260

We scaled up our internal coding evaluations so that we could see where the agents

Cue 235 — 00:16:28,300 --> 00:16:31,940

struggle. We tested a compression proxy.

Cue 236 — 00:16:31,980 --> 00:16:35,960

In those evals. We tune the default model.

Cue 237 — 00:16:36,180 --> 00:16:41,300

We rolled out compression, moved to a newer model, which is the default,

Cue 238 — 00:16:41,340 --> 00:16:45,820

and then we added like spend guardrails that every company is trying to enforce.

Cue 239 — 00:16:45,860 --> 00:16:50,860

And the pattern was that each of these steps raised a question the signal wasn't

Cue 240 — 00:16:50,900 --> 00:16:52,900

able to answer in the previous step.

Cue 241 — 00:16:52,940 --> 00:16:57,920

And that's how we ended up with four layers of visibility or observability.

Cue 242 — 00:16:57,980 --> 00:17:01,820

And the stories are what I will in the pink ones, if you notice in this,

Cue 243 — 00:17:01,860 --> 00:17:04,800

are basically the stories that I'm going to walk you through.

Cue 244 — 00:17:04,980 --> 00:17:07,760

And so these are the four layers.

Cue 245 — 00:17:08,420 --> 00:17:13,360

And so I've ordered them from fastest to slowest feedback.

Cue 246 — 00:17:13,940 --> 00:17:16,460

So if you look at the top it's the gateway.

Cue 247 — 00:17:16,500 --> 00:17:21,500

As I mentioned you the gateway sees every model call and you hear back in like

Cue 248 — 00:17:21,540 --> 00:17:25,100

immediate like seconds because you would know exactly in the organization which

Cue 249 — 00:17:25,140 --> 00:17:28,720

engineer is making which tool called because it goes via the gateway.

Cue 250 — 00:17:28,860 --> 00:17:32,360

The trace is they see each agent session.

Cue 251 — 00:17:32,820 --> 00:17:36,740

So maybe it could take some minutes because now you are capturing traces for every

Cue 252 — 00:17:36,780 --> 00:17:39,420

single tool call or agent call that's happening.

Cue 253 — 00:17:39,460 --> 00:17:43,140

The evals, they test the change before anyone uses it.

Cue 254 — 00:17:43,180 --> 00:17:47,620

So basically the evals are basically like a testing harness which runs in the like a

Cue 255 — 00:17:47,660 --> 00:17:53,260

periodic cron job or nightly. And those are important to identify how the model

Cue 256 — 00:17:53,300 --> 00:17:56,120

works. I'm going to cover those in a bit.

Cue 257 — 00:17:56,500 --> 00:18:01,260

And so the rollout is basically what is going to roll out the change in front of the

Cue 258 — 00:18:01,300 --> 00:18:06,100

engineers. And then we are going to measure how the actual productivity is.

Cue 259 — 00:18:06,140 --> 00:18:10,140

And at the bottom is basically the outcome, which basically is the work that got

Cue 260 — 00:18:10,180 --> 00:18:13,040

shipped that and did it stay shipped.

Cue 261 — 00:18:13,460 --> 00:18:18,180

And so everything lands in Datadog, which is the LLM Observability,

Cue 262 — 00:18:18,220 --> 00:18:19,860

the dashboards and the monitors.

Cue 263 — 00:18:19,900 --> 00:18:23,840

And so we'll go through them in order, starting with the gateway.

Cue 264 — 00:18:25,460 --> 00:18:30,740

And so the idea is simple, that gateway is basically one way in for every model

Cue 265 — 00:18:30,780 --> 00:18:36,620

call. And so the agent on each laptop has basically the managed settings that point

Cue 266 — 00:18:36,660 --> 00:18:38,980

our traffic to the AI gateway.

Cue 267 — 00:18:39,020 --> 00:18:44,380

And there's a short lived token which ties the call to a specific person to identify

Cue 268 — 00:18:44,420 --> 00:18:49,020

the identity. And behind that, it's the same models and providers as before.

Cue 269 — 00:18:49,060 --> 00:18:51,420

There's just a single front door.

Cue 270 — 00:18:51,460 --> 00:18:57,180

And so what happens is this AI gateway gives us the usage, the errors,

Cue 271 — 00:18:57,220 --> 00:19:00,140

the latency and cost. In Datadog.

Cue 272 — 00:19:00,180 --> 00:19:02,700

It breaks it down by team and model.

Cue 273 — 00:19:02,740 --> 00:19:07,820

And so it basically just adds a few milliseconds of overhead per call.

Cue 274 — 00:19:07,860 --> 00:19:11,180

But now you're able to get an entire visibility across the stack.

Cue 275 — 00:19:11,220 --> 00:19:17,260

And it does come up with some upkeep because you have to now like maintain this AI

Cue 276 — 00:19:17,300 --> 00:19:20,820

gateway. And it's a shared dependency.

Cue 277 — 00:19:20,860 --> 00:19:24,780

So if it's slow, everyone will obviously feel the burden of it.

Cue 278 — 00:19:24,820 --> 00:19:26,000

And so

Cue 279 — 00:19:27,260 --> 00:19:29,900

it has one blind spot if you can spot it.

Cue 280 — 00:19:29,940 --> 00:19:34,940

So what does AI it will not give you is basically it knows the amount of tokens you

Cue 281 — 00:19:34,980 --> 00:19:38,860

are spending, but you it doesn't know what the agent did with that.

Cue 282 — 00:19:38,900 --> 00:19:43,380

And so if you're just FYI, if you're building your AI gateway yourself,

Cue 283 — 00:19:43,420 --> 00:19:46,580

you can use of off the shelf AI gateway.

Cue 284 — 00:19:46,620 --> 00:19:53,460

You can use like LiteLLM proxy, or you can also use like Datadog AI Gateway or Envoy

Cue 285 — 00:19:53,500 --> 00:19:56,200

AI gateway. And there are multiple off the shelf options.

Cue 286 — 00:19:56,620 --> 00:19:57,600

And so

Cue 287 — 00:19:58,840 --> 00:20:05,300

before we get to that blind spot, here's a story about where the gateway was exactly

Cue 288 — 00:20:05,340 --> 00:20:11,100

what got the problem. And so this is the model router that we piloted.

Cue 289 — 00:20:11,140 --> 00:20:14,580

And so the model router basically sits in front of the models,

Cue 290 — 00:20:14,620 --> 00:20:19,540

and for every request, every turn of a session, It guesses how hard that request is

Cue 291 — 00:20:19,580 --> 00:20:25,580

and picks a model. So it basically routes the the traffic to different models based

Cue 292 — 00:20:25,620 --> 00:20:28,780

on the hardness or the problem itself.

Cue 293 — 00:20:28,820 --> 00:20:33,620

And so the easy request would go to a small model that's cheap and fast,

Cue 294 — 00:20:33,660 --> 00:20:35,620

and the hard ones go to the big model.

Cue 295 — 00:20:35,660 --> 00:20:39,260

And which is more expensive on paper it makes sense right.

Cue 296 — 00:20:39,300 --> 00:20:43,740

Like that. That would work. So you would expect like the same quality lower costs.

Cue 297 — 00:20:43,780 --> 00:20:47,260

Everybody should be happy. And we piloted this.

Cue 298 — 00:20:47,300 --> 00:20:52,160

And the cost per user actually went up about 16%.

Cue 299 — 00:20:52,580 --> 00:20:56,700

That's that's not that's not the savings we expected.

Cue 300 — 00:20:56,740 --> 00:21:03,160

So we dug into why. So there were two reasons and both came out of the data itself.

Cue 301 — 00:21:03,380 --> 00:21:06,220

The first is that it overrode people's choices.

Cue 302 — 00:21:06,260 --> 00:21:10,620

And people like the engineers who had deliberately picked the small model,

Cue 303 — 00:21:10,660 --> 00:21:13,780

suddenly had about half of their traffic going to the big one.

Cue 304 — 00:21:13,820 --> 00:21:18,540

And the second is caching, and a prompt cache belongs to one model.

Cue 305 — 00:21:18,580 --> 00:21:23,660

So every time the router switched models and in the middle of the session,

Cue 306 — 00:21:23,700 --> 00:21:27,380

all of that context basically had to be sent to another model.

Cue 307 — 00:21:27,420 --> 00:21:31,180

And so the people on the big model, the cache hit rate dropped,

Cue 308 — 00:21:31,220 --> 00:21:33,740

and the cache misses are obviously super expensive.

Cue 309 — 00:21:33,780 --> 00:21:39,700

And it basically told us about the cost and the told us about the quality of this.

Cue 310 — 00:21:39,740 --> 00:21:44,500

And then what we took away from this is that the defaults and the caching,

Cue 311 — 00:21:44,540 --> 00:21:50,360

they move the bill, but they don't really like apply for a clever routing.

Cue 312 — 00:21:50,660 --> 00:21:55,300

And so remember what was the blind spot with the the AI gateway?

Cue 313 — 00:21:55,340 --> 00:21:59,620

It basically tells you what you spent, but it doesn't tell you what the agent did.

Cue 314 — 00:21:59,660 --> 00:22:02,880

So that's what the second layer is for, which is traces.

Cue 315 — 00:22:12,020 --> 00:22:18,180

And so talking about traces, every agent that you use in the organization and it's

Cue 316 — 00:22:18,220 --> 00:22:22,198

different events. Like if you use Claude Code, Codex, somebody could be using Pi,

Cue 317 — 00:22:22,238 --> 00:22:23,940

somebody could be using Cursor.

Cue 318 — 00:22:23,980 --> 00:22:26,360

They all log things differently.

Cue 319 — 00:22:26,460 --> 00:22:32,460

And so we built Trajectory which is basically a capture layer that turns each agents

Cue 320 — 00:22:32,500 --> 00:22:34,900

events into a common session format.

Cue 321 — 00:22:34,940 --> 00:22:37,620

So it's like a schema for all of these models.

Cue 322 — 00:22:37,660 --> 00:22:41,420

It's open source. It's on GitHub under Datadog Trajectory.

Cue 323 — 00:22:41,460 --> 00:22:42,780

And so it's linked at the end.

Cue 324 — 00:22:42,820 --> 00:22:47,240

So I'll show you that. And so once everything is in that same format,

Cue 325 — 00:22:47,340 --> 00:22:50,380

a session is basically like a distributed trace.

Cue 326 — 00:22:50,420 --> 00:22:56,340

And so in this example the agent loads the context, reads the file edits like let's

Cue 327 — 00:22:56,380 --> 00:22:57,980

say three of them runs the test.

Cue 328 — 00:22:58,020 --> 00:23:03,500

The test fails, it retries, and the retry is where the extra time and tokens went

Cue 329 — 00:23:03,540 --> 00:23:07,860

right. So the gateway would have only seen that the extra tokens were spent.

Cue 330 — 00:23:07,900 --> 00:23:12,340

And by default we only like capture the structure and timing of a session.

Cue 331 — 00:23:12,380 --> 00:23:18,020

So not the prompts or the code, but traces have an advantage that basically we would

Cue 332 — 00:23:18,060 --> 00:23:21,020

be able to see the three trials that are happening across the prompt and things of

Cue 333 — 00:23:21,060 --> 00:23:24,560

that sort. But what is the blind spot for traces?

Cue 334 — 00:23:25,220 --> 00:23:31,660

So any guesses? And so the blind spot for traces is that they will tell you what

Cue 335 — 00:23:31,700 --> 00:23:34,640

actually happened in the prompt, but not exactly.

Cue 336 — 00:23:34,700 --> 00:23:39,000

Is this prompt going to work better with another model or harness.

Cue 337 — 00:23:39,100 --> 00:23:43,200

And so answering that question is basically the job of evals.

Cue 338 — 00:23:44,820 --> 00:23:50,020

And so this is talking about evals I'm sure many of you know here about evals.

Cue 339 — 00:23:50,060 --> 00:23:53,740

And so we I do we run our own evals.

Cue 340 — 00:23:53,780 --> 00:23:57,460

And so the first thing people would ask us is like, why not just look at some

Cue 341 — 00:23:57,500 --> 00:24:00,180

existing public evals and run those?

Cue 342 — 00:24:00,220 --> 00:24:05,540

And the problem is that if you look at the leaderboard like it answers a different

Cue 343 — 00:24:05,580 --> 00:24:09,900

question. So it uses public tasks on public repositories.

Cue 344 — 00:24:09,940 --> 00:24:12,180

So it tests a model basically on its own.

Cue 345 — 00:24:12,220 --> 00:24:15,400

And it mostly just scores pass or fail.

Cue 346 — 00:24:15,660 --> 00:24:18,740

So it tells you which model is better in general.

Cue 347 — 00:24:18,780 --> 00:24:20,780

So if you ask somebody oh, which model do you use?

Cue 348 — 00:24:20,820 --> 00:24:25,642

People like in the previous talk they were talking about are people use Claude Code

Cue 349 — 00:24:25,682 --> 00:24:30,260

or Codex or specific IDEs about which task performs under which model.

Cue 350 — 00:24:30,300 --> 00:24:37,400

They don't have an opinion. And so our engineers, they don't use a model like

Cue 351 — 00:24:37,580 --> 00:24:43,180

on its own. So they use a basically a model inside a harness with our config and our

Cue 352 — 00:24:43,220 --> 00:24:49,060

code. So the evals are basically real, like the company Datadog work in our

Cue 353 — 00:24:49,100 --> 00:24:52,300

repositories and tests. So they have the entire setup together.

Cue 354 — 00:24:52,340 --> 00:24:55,920

And so the question the answer is much like narrow.

Cue 355 — 00:24:56,380 --> 00:25:00,540

Should this be, let's say our default model or right now what's the config.

Cue 356 — 00:25:00,580 --> 00:25:06,020

So the eval is basically going to evaluate the model on the specific task that you

Cue 357 — 00:25:06,060 --> 00:25:09,000

are doing day to day, and check the performance of it.

Cue 358 — 00:25:10,020 --> 00:25:15,320

So what exactly is an eval. Every eval has four parts.

Cue 359 — 00:25:15,700 --> 00:25:20,520

The task is basically a prompt taken from let's say actual work that you're doing

Cue 360 — 00:25:20,740 --> 00:25:25,580

using the framework conventions that you're actually using in a day to day job.

Cue 361 — 00:25:25,620 --> 00:25:30,860

The repository is pinned to a single commit, and so every run starts from exactly

Cue 362 — 00:25:30,900 --> 00:25:36,380

the same code build tools. And the setup under the test is the model plus the

Cue 363 — 00:25:36,420 --> 00:25:42,620

harness plus the config, which is basically how you test, from engineers

Cue 364 — 00:25:42,660 --> 00:25:45,660

perspective. And because like I said, they shipped together.

Cue 365 — 00:25:45,700 --> 00:25:50,380

So the grader, which is the evaluator, basically combines all of the automated

Cue 366 — 00:25:50,420 --> 00:25:56,120

checks with an LLM judge and it basically runs every night.

Cue 367 — 00:25:56,340 --> 00:26:01,780

We pick the tests, we set up the repositories, run the setup multiple times because

Cue 368 — 00:26:01,820 --> 00:26:04,500

of the non determinism and create the run.

Cue 369 — 00:26:04,540 --> 00:26:10,100

And then we have an LLM read the whole run.

Cue 370 — 00:26:10,140 --> 00:26:16,020

And so we explain why did the score result in this kind of a score.

Cue 371 — 00:26:16,060 --> 00:26:20,340

And so there's like a nightly dashboard which we have which compare the setup

Cue 372 — 00:26:20,380 --> 00:26:23,780

against the current defaults. And then check the performance of the model on these

Cue 373 — 00:26:23,820 --> 00:26:28,240

tests. And this includes like the benchmarking completely.

Cue 374 — 00:26:28,300 --> 00:26:32,740

And let's, let's look at it from like a concrete example.

Cue 375 — 00:26:32,780 --> 00:26:38,660

So how does it get scored. So let's say the task is to read a secret using our

Cue 376 — 00:26:38,700 --> 00:26:44,700

current vault pattern. And the known end state is the pattern our docs describe,

Cue 377 — 00:26:44,740 --> 00:26:46,660

which is basically test passing.

Cue 378 — 00:26:46,700 --> 00:26:49,180

So we run the same setup five times.

Cue 379 — 00:26:49,220 --> 00:26:51,380

Each run is scored on four things.

Cue 380 — 00:26:51,420 --> 00:26:56,760

Do the test pass, do the checks on the diff, find the right pattern?

Cue 381 — 00:26:56,900 --> 00:27:02,500

Did the agent take the right actions and what does the judge say?

Cue 382 — 00:27:02,540 --> 00:27:07,700

So the score is not just like a 0 or 1, like a binary like pass or fail.

Cue 383 — 00:27:07,740 --> 00:27:13,060

And look at the run. Two it's interesting, the test passed, but it only got a score

Cue 384 — 00:27:13,100 --> 00:27:19,540

of 0.6. And the step five is basically the explanation because it copied an older

Cue 385 — 00:27:19,580 --> 00:27:24,180

vault pattern from a neighboring service instead of the documented one.

Cue 386 — 00:27:24,220 --> 00:27:30,820

And so this explanation is what lets us group failures across thousands of runs and

Cue 387 — 00:27:30,860 --> 00:27:36,340

see patterns and running evals like this every night is what changed our default

Cue 388 — 00:27:36,380 --> 00:27:38,160

model at Datadog twice.

Cue 389 — 00:27:40,100 --> 00:27:45,940

And so, like I said, actually changed the default model at Datadog twice because we

Cue 390 — 00:27:45,980 --> 00:27:49,820

started on a larger model, and the eval showed that a smaller model got the same

Cue 391 — 00:27:49,860 --> 00:27:53,620

results on most tasks while being faster and cheaper.

Cue 392 — 00:27:53,660 --> 00:27:58,060

So we move the default to a smaller model, and few months later,

Cue 393 — 00:27:58,100 --> 00:28:03,420

a newer, larger model came out, and in our evals it was about half the cost of the

Cue 394 — 00:28:03,460 --> 00:28:06,060

smaller one. And so we switched again.

Cue 395 — 00:28:06,100 --> 00:28:12,100

So it's the same harness and the same tasks and a different answer basically that's

Cue 396 — 00:28:12,140 --> 00:28:17,700

evolving. So the evolve the eval result has a shelf life if it makes sense because

Cue 397 — 00:28:17,740 --> 00:28:22,980

the models improve and they need less steering harnesses make tool calls more

Cue 398 — 00:28:23,020 --> 00:28:27,700

reliable. And so it's worth rerunning everything when the model changes.

Cue 399 — 00:28:27,740 --> 00:28:32,420

And this is basically another example.

Cue 400 — 00:28:32,460 --> 00:28:35,340

So this is the compression proxy from the timeline.

Cue 401 — 00:28:35,380 --> 00:28:39,200

And the reason to compress. Why would you want to compress is because

Cue 402 — 00:28:40,660 --> 00:28:46,340

if you have like less amount of data per request, it would basically reduce the cost

Cue 403 — 00:28:46,380 --> 00:28:48,460

of your request significantly.

Cue 404 — 00:28:48,500 --> 00:28:51,980

And so if you look at this diagram, each dot is one eval result.

Cue 405 — 00:28:52,020 --> 00:28:57,460

And the cost went down by more than half and the average score remained about the

Cue 406 — 00:28:57,500 --> 00:29:01,700

same. So if it only looked at the average we would have shipped it.

Cue 407 — 00:29:01,740 --> 00:29:04,740

But two tasks actually drop down to zero.

Cue 408 — 00:29:04,780 --> 00:29:09,620

If you look at the left and the compression actually removes something those tasks

Cue 409 — 00:29:09,660 --> 00:29:13,780

dependent on so evals actually let us catch that before, you know,

Cue 410 — 00:29:13,820 --> 00:29:16,020

rolling it out across the organization.

Cue 411 — 00:29:16,060 --> 00:29:18,980

So it was actually run on a fixed set of tasks.

Cue 412 — 00:29:19,020 --> 00:29:21,700

It's not what engineers do day to day.

Cue 413 — 00:29:21,740 --> 00:29:26,960

And so the last layer is basically rollouts, which is where we test the real work.

Cue 414 — 00:29:27,900 --> 00:29:31,860

And so we treat the config change just like a code deployment.

Cue 415 — 00:29:31,900 --> 00:29:35,260

Basically every variant it gets a version tag.

Cue 416 — 00:29:35,300 --> 00:29:38,460

And that tag rides along with every model call.

Cue 417 — 00:29:38,500 --> 00:29:42,160

Small group gets it first. It's a very controlled rollout.

Cue 418 — 00:29:42,300 --> 00:29:44,780

Use device management group for that.

Cue 419 — 00:29:44,820 --> 00:29:47,700

And then we compare the test group against the base group.

Cue 420 — 00:29:47,740 --> 00:29:53,260

And so only after that, like all of this process, it becomes a default model for

Cue 421 — 00:29:53,300 --> 00:29:56,980

everyone. And because every call carries the config version,

Cue 422 — 00:29:57,020 --> 00:30:02,020

every dashboard already has the split without building anything from new.

Cue 423 — 00:30:02,060 --> 00:30:06,160

And so here's a rollout that gave us an answer quickly.

Cue 424 — 00:30:06,700 --> 00:30:10,720

This is the RTK test that I was talking about from the timeline.

Cue 425 — 00:30:11,300 --> 00:30:16,620

And we compared them across a group of days, and the test group actually made more

Cue 426 — 00:30:16,660 --> 00:30:20,140

requests, hit more errors and ended up costing more.

Cue 427 — 00:30:20,180 --> 00:30:23,820

So shrinking one of the bill made the others grow.

Cue 428 — 00:30:23,860 --> 00:30:27,020

And it's the same multiplication that I talked about from the cost slide.

Cue 429 — 00:30:27,060 --> 00:30:31,820

And so because both groups ran side by side, we had a clear answer in a week.

Cue 430 — 00:30:31,860 --> 00:30:36,500

And so that leaves the slowest signal of all, which is the outcome.

Cue 431 — 00:30:36,540 --> 00:30:38,540

And the outcomes could take weeks.

Cue 432 — 00:30:38,580 --> 00:30:41,900

They're also basically the easiest way to fool yourself.

Cue 433 — 00:30:41,940 --> 00:30:46,780

And so we have to be very careful here because we tested a bunch of candidate like

Cue 434 — 00:30:46,820 --> 00:30:51,660

code quality signals. And so the question that we want to answer here is does this

Cue 435 — 00:30:51,700 --> 00:30:58,020

PR need a follow up PR. And so most of the PR did not hold up because they needed

Cue 436 — 00:30:58,060 --> 00:31:03,900

some of the other bug fixing. And the numbers say that only one out of ten actually

Cue 437 — 00:31:03,940 --> 00:31:06,400

held up. So that's pretty shocking.

Cue 438 — 00:31:06,780 --> 00:31:12,620

And so let me put the four layers together, which is basically on the left.

Cue 439 — 00:31:12,660 --> 00:31:16,420

Think of every change coming in and going through the four checkpoints,

Cue 440 — 00:31:16,460 --> 00:31:20,100

the gateway it catches, the cost and failure spikes.

Cue 441 — 00:31:20,140 --> 00:31:24,900

That was the routing. And so the trace is they catch the loops and the stalls like

Cue 442 — 00:31:24,940 --> 00:31:31,300

retry loops. The evals basically catch the quality regressions and rollouts which

Cue 443 — 00:31:31,340 --> 00:31:33,280

catch the real world breakage.

Cue 444 — 00:31:33,580 --> 00:31:37,380

And so only anything that passes all four of them becomes the default.

Cue 445 — 00:31:37,420 --> 00:31:41,340

And that covers basically the adoption, the productivity and the cost.

Cue 446 — 00:31:41,380 --> 00:31:44,500

And so I said I'd come back to the fourth question, security.

Cue 447 — 00:31:44,540 --> 00:31:50,820

So here it is. Our answer to security is the same idea as everything else.

Cue 448 — 00:31:50,860 --> 00:31:52,800

And so the question,

Cue 449 — 00:31:54,140 --> 00:31:58,480

every guardrail is basically a central default, so nobody has to remember it.

Cue 450 — 00:31:58,700 --> 00:32:05,300

Who's calling it? Every call basically carries a short lived token tied to one

Cue 451 — 00:32:05,340 --> 00:32:10,660

person. And which model? Like so agents are configured for approved.

Cue 452 — 00:32:10,700 --> 00:32:16,100

Why does only which tools MCP servers get vetted before they touch anything.

Cue 453 — 00:32:16,140 --> 00:32:23,060

And yeah, so basically what merges is like auto approval only happens when every

Cue 454 — 00:32:23,100 --> 00:32:26,180

security, reliability and compliance check has passed.

Cue 455 — 00:32:26,220 --> 00:32:30,580

So with all the four covered, let me go back to the questions from the start and

Cue 456 — 00:32:30,620 --> 00:32:37,440

give you the answers. So adoption my answer is make it the default and then

Cue 457 — 00:32:37,860 --> 00:32:42,700

see how it pans. Basically who who stays using them, the configuration and who is

Cue 458 — 00:32:42,740 --> 00:32:46,480

not using it. And that's basically controlled by device management,

Cue 459 — 00:32:46,700 --> 00:32:49,780

the gateway and the traces. They will show you the weekly usage.

Cue 460 — 00:32:49,820 --> 00:32:54,860

And if people are dropping out, that's basically your warning productivity.

Cue 461 — 00:32:54,900 --> 00:32:58,720

You should just change by outcomes and not activity.

Cue 462 — 00:32:59,100 --> 00:33:02,960

So merged and kept PRs, not reverted PRs.

Cue 463 — 00:33:03,860 --> 00:33:10,660

And so use evals before releasing and small rollout before everyone cost manage

Cue 464 — 00:33:10,700 --> 00:33:13,040

the unit costs, not the total.

Cue 465 — 00:33:13,260 --> 00:33:19,780

For us, defaults and caching moves the bill more than routing and security.

Cue 466 — 00:33:19,820 --> 00:33:22,660

Make every guardrail a central default.

Cue 467 — 00:33:22,700 --> 00:33:29,100

And so here's what I would suggest if the takeaways from this talk overall.

Cue 468 — 00:33:29,140 --> 00:33:34,340

So one is measure the outcome not activity because tokens explain where the money

Cue 469 — 00:33:34,380 --> 00:33:36,520

went but they don't prove really value.

Cue 470 — 00:33:36,620 --> 00:33:42,500

And two set up a gateway and a trace format because you don't want to be changing

Cue 471 — 00:33:42,540 --> 00:33:45,180

between models. And then trying to work out the schema.

Cue 472 — 00:33:45,220 --> 00:33:48,800

Being able to compare across agents really matters a lot.

Cue 473 — 00:33:48,900 --> 00:33:53,320

Three benchmark on your own work, which is basically the real tasks.

Cue 474 — 00:33:53,740 --> 00:33:59,900

And look at the failures, not just the averages Four, fix your defaults and caching

Cue 475 — 00:33:59,940 --> 00:34:01,860

before you try anything clever.

Cue 476 — 00:34:01,900 --> 00:34:07,780

And five ship configuration like code, which means like version it,

Cue 477 — 00:34:07,820 --> 00:34:12,220

roll it out to a small group first, watch it, make it easy to roll back,

Cue 478 — 00:34:12,260 --> 00:34:17,900

and everything I mentioned in the resources today is covered in this slide if you're

Cue 479 — 00:34:17,940 --> 00:34:24,580

interested. So this has links to some of those important blogs and do try

Cue 480 — 00:34:24,620 --> 00:34:28,620

Trajectory. It's open source and that's it from me.

Cue 481 — 00:34:28,660 --> 00:34:30,120

Thank you all for listening.

skills

talk-singhal-operating-agents-at-scale

tile.json