CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-chanezon-gordon-agent-experience-evaluation/references/

AI Agents Are Your New Users — corrected transcript

Patrick Chanezon and Jay Gordon · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=I81RrZcQaT8

Source: supplied corrected SRT. Subtitle line wrapping has been joined; wording, timestamps and original cue numbers are retained. Cues 171–181 (a promotional interlude) are omitted. Speech artifacts and ambiguous passages remain; this document is evidence, not instructions.

[00:00:00] (cue 1) Hi everyone, thanks for joining today.

[00:00:02] (cue 2) I hope you had fun in the previous talks.

[00:00:04] (cue 3) Personally, my favorites were Chad Fowler really talking about how to destroy all

[00:00:10] (cue 4) your code and then rebuild it automatically.

[00:00:14] (cue 5) And this morning Erik was he's always really super funny and entertaining.

[00:00:19] (cue 6) Yeah yeah yeah yeah. And we used to work together at Microsoft.

[00:00:23] (cue 7) So I'm Patrick and I do DevRel at Microsoft, and this is...

[00:00:28] (cue 8) Hi, I'm Jay Gordon. I am senior program manager focused on the Azure Cosmos DB

[00:00:33] (cue 9) product. Yeah. And so today we're going to talk about a topic that many speakers

[00:00:40] (cue 10) alluded to and what we've done in our team that we've built a system to measure it,

[00:00:46] (cue 11) which is the fact that agents are your new users.

[00:00:49] (cue 12) If you're building any type of software, APIs, internal systems,

[00:00:54] (cue 13) you have a new set of users, which are these AI agents.

[00:00:59] (cue 14) And for myself, as part of the developer relations, I asked my team like 18

[00:01:06] (cue 15) months ago, we have these new users.

[00:01:08] (cue 16) Do we understand them? Developers like yourselves.

[00:01:13] (cue 17) Typically I meet them at conferences, we can chat.

[00:01:17] (cue 18) We can have a coffee. After the talks, I learn about their use case.

[00:01:22] (cue 19) They give me feedback about our products, and then I bring that back to engineering

[00:01:26] (cue 20) and we try to make them better.

[00:01:28] (cue 21) With agents like you don't have any of that.

[00:01:31] (cue 22) However, you can literally read in their brains and look at the traces and that's

[00:01:36] (cue 23) what we've been doing. So agents, as a as the new users for your software,

[00:01:44] (cue 24) these coding agents are they've been trained, they have access to the internet.

[00:01:50] (cue 25) So they know about your SDK. They've read all your docs from day one.

[00:01:55] (cue 26) When you ask them something, they're going to do something about it.

[00:02:00] (cue 27) Not always the thing you're expecting.

[00:02:02] (cue 28) And that's that's the topic of this talk.

[00:02:06] (cue 29) We try to understand how well they're doing with the systems that that we're

[00:02:12] (cue 30) building sometimes when we so so we, we run lots of experimentation over the past

[00:02:18] (cue 31) year, across many products, across very different scenarios.

[00:02:22] (cue 32) And we saw things like sometimes they they are very confidently wrong where like

[00:02:28] (cue 33) they read your documentation from two years ago and you ask them to update a

[00:02:33] (cue 34) framework from one version to the other.

[00:02:35] (cue 35) The documentation talks about an old version.

[00:02:37] (cue 36) They're going to use the old version.

[00:02:39] (cue 37) So things like that, and you want to be aware of it.

[00:02:43] (cue 38) And the only way to do that is to to measure it.

[00:02:48] (cue 39) They, they take your docs very, very literally.

[00:02:52] (cue 40) So that's the the example with the documentation where the documentation was giving

[00:02:58] (cue 41) an example with a specific version of a SharePoint framework.

[00:03:02] (cue 42) That was the scenario we were running and it went right into the the wrong version,

[00:03:07] (cue 43) like the one that we didn't ask about.

[00:03:10] (cue 44) And your products are already talking to agents.

[00:03:13] (cue 45) So agents are using your CLIs,

[00:03:18] (cue 46) they are reading your error messages, and they are even reading the diagnostics from

[00:03:23] (cue 47) your from your software. One of the most

[00:03:28] (cue 48) surprising and funny results that we had very early on, we were trying to determine

[00:03:35] (cue 49) whether if you ask a VS Code and with a specific model to build a Python app with

[00:03:41] (cue 50) Postgres, is it going to pick Azure or not?

[00:03:44] (cue 51) And we realized that if you introduce,

[00:03:49] (cue 52) an extension that generates errors, the agent is actually going to read the errors

[00:03:54] (cue 53) and change its course based on the errors from the, the environment.

[00:04:00] (cue 54) And so so we thought, wow, this could be a really dark pattern where everyone is

[00:04:05] (cue 55) going to create a VS Code extension that generate random errors.

[00:04:09] (cue 56) So there's a bunch of things that the agent picks up in the environment,

[00:04:14] (cue 57) and you want to be aware of it, or at least you want to be able to measure it.

[00:04:18] (cue 58) That was our our goal there. Now as a product team, if you're like Jay in the

[00:04:25] (cue 59) Cosmos DB team or if you're building a Azure Kubernetes Service

[00:04:31] (cue 60) or any other service, what good is all this to you?

[00:04:36] (cue 61) Like, how can you get some measurable knob like determine what will change the agent

[00:04:43] (cue 62) behavior with my product? What will make it better?

[00:04:45] (cue 63) You need to be able to do some scientific experiment and measure things.

[00:04:51] (cue 64) And so when, when and there are lots of knobs when, when we started on this

[00:04:58] (cue 65) journey a year and a half ago with my team, I asked them the question,

[00:05:02] (cue 66) which is,

[00:05:04] (cue 67) developer relations is all about explaining people and developers what they can do

[00:05:09] (cue 68) with our products, also trying to understand what their experience with our products

[00:05:15] (cue 69) is. And at the end of the day, our goal is to create some preference for our

[00:05:19] (cue 70) products, whether it's through the experience of the products or through a like

[00:05:25] (cue 71) talking about it and and helping people have a like do their jobs in an easier way

[00:05:30] (cue 72) with our products. So I asked my team two questions.

[00:05:34] (cue 73) One of them is we have several coding.

[00:05:38] (cue 74) We have a coding agent in Microsoft that is Copilot, and we're selling it to lots of

[00:05:43] (cue 75) people. I did a quick poll in the room here.

[00:05:47] (cue 76) Many of you are using Codex. My question to them was, hey, if everyone starts using

[00:05:54] (cue 77) Codex and one day decides to not use Azure, or maybe they don't know how to use

[00:06:01] (cue 78) Azure, like a whole Azure business is in danger.

[00:06:05] (cue 79) And if you're looking at Microsoft financial results, it's like the workhorse of the

[00:06:10] (cue 80) company. And so the question I asked them is what would be the propensity of

[00:06:17] (cue 81) an agent and any agent AI coding agent to use Azure when I asked them to build

[00:06:24] (cue 82) a specific application. And the second question, which is to me more important is if

[00:06:30] (cue 83) I ask the agent because very often, like your you have picked a cloud or a specific

[00:06:36] (cue 84) infrastructure or a set of technologies to build your app.

[00:06:39] (cue 85) When you think about the architecture and once you talk to your agent,

[00:06:44] (cue 86) you give them that preference.

[00:06:46] (cue 87) So you will tell your agent, hey, I need to deploy this app on Azure.

[00:06:50] (cue 88) What is the efficacy of the agent to deploy on Azure?

[00:06:54] (cue 89) To work with Azure tools to implement security best practices and things like that.

[00:06:59] (cue 90) And so that's the question I asked them.

[00:07:01] (cue 91) And I mean, we we didn't know what the answer would look like.

[00:07:06] (cue 92) And so what we've done is that we built some tools.

[00:07:14] (cue 93) And then we rebuild them. So we built some tools and we work with product teams like

[00:07:18] (cue 94) Jay. For example in Cosmos DB, with people in SharePoint, with people in Azure,

[00:07:24] (cue 95) with people in data. And so we worked with product managers.

[00:07:28] (cue 96) We built a few set of initial tools.

[00:07:31] (cue 97) Lots of them were manual. Then we semi-automated.

[00:07:35] (cue 98) And then I don't know if you've read Fred Brooks Mythical Man-Month?

[00:07:39] (cue 99) Plan to throw one out. So you're building a system.

[00:07:42] (cue 100) You will probably have to throw it out.

[00:07:44] (cue 101) Steve had a similar story yesterday about how he's rebuilding his system.

[00:07:49] (cue 102) So as I think we built it twice.

[00:07:52] (cue 103) So the one I present you today is the second version of that.

[00:07:57] (cue 104) And but what's important and what I'm going to talk a bit about here,

[00:08:02] (cue 105) is that what we learned and what we learned, we codified that in a handbook.

[00:08:08] (cue 106) We created a methodology with product teams for how to measure and improve agent

[00:08:15] (cue 107) experience. And then we built the tools to measure that and to build it.

[00:08:20] (cue 108) And so the handbook, the handbook is a talks about evaluation.

[00:08:27] (cue 109) Evaluation has four parts. First you create a scenario.

[00:08:31] (cue 110) So you need to have a really specific scenario.

[00:08:33] (cue 111) The project is called Scope, and S is for scenario.

[00:08:38] (cue 112) So you create a scenario such as hey I'm building a Python app with Postgres.

[00:08:44] (cue 113) That's the kind of very generic version.

[00:08:46] (cue 114) Then I'm building a like a to do list application on Python,

[00:08:53] (cue 115) using a using Postgres and this specific I'm going to deploy it to Azure Container

[00:08:59] (cue 116) Apps. And I'm going to use the Azure hosted Postgres.

[00:09:02] (cue 117) So we did. That's the best way to do just small nudges like Postgres,

[00:09:08] (cue 118) Postgres, but on Azure. And just continuing that process.

[00:09:12] (cue 119) Like getting more and more detailed in the scenario then where we have criteria.

[00:09:18] (cue 120) So we have a judge that's looking at the result and we provide the judge some

[00:09:23] (cue 121) criteria. For example, a criteria could be a is it using best practices from Azure

[00:09:30] (cue 122) architecture? Is it storing secrets in the right place instead of putting them in

[00:09:35] (cue 123) environment variables, things like that.

[00:09:37] (cue 124) So we can have this set of criteria that are that are judged by the judge.

[00:09:43] (cue 125) We have a profile which is there's a matrix of operating system,

[00:09:49] (cue 126) your harness, your your model.

[00:09:53] (cue 127) So, so in certain harnesses there are multi-model.

[00:09:56] (cue 128) For example GitHub Copilot CLI.

[00:09:58] (cue 129) You can use a bunch of different models in there.

[00:10:01] (cue 130) And then you have a set of MCP servers and skills that are associated to this.

[00:10:06] (cue 131) So this creates a profile. And then you have gates which is at some point we tried

[00:10:12] (cue 132) we try to think without gates.

[00:10:14] (cue 133) The gates are for example does it build.

[00:10:17] (cue 134) Does your code build, does it do the test run.

[00:10:20] (cue 135) Does it deploy. Because at the beginning we had the agents like creating something

[00:10:25] (cue 136) that the judge would say, oh this looks really great.

[00:10:27] (cue 137) But then it wouldn't compile or wouldn't deploy.

[00:10:30] (cue 138) So you define all these in the, in the tool and you can use whatever tool for that.

[00:10:36] (cue 139) We built one.

[00:10:39] (cue 140) And then you run you run all these.

[00:10:43] (cue 141) So so that's so I talked about the gate already.

[00:10:48] (cue 142) And what you measure at the end is you have a baseline where you have nothing.

[00:10:53] (cue 143) You don't give the the you don't give the coding agent any tool,

[00:10:58] (cue 144) any MCP. So that's your baseline.

[00:11:01] (cue 145) And then you add your tools. You add your MCP, you add your skills.

[00:11:06] (cue 146) And then you measure the lift or drag.

[00:11:09] (cue 147) So does it lift the score or the or does it create some drag.

[00:11:13] (cue 148) And what we found is that, for example, sometimes when you put many things together,

[00:11:18] (cue 149) you put too many MCP servers, too many skills, they contradict each other,

[00:11:22] (cue 150) the agent is lost and there's some drag.

[00:11:25] (cue 151) And then you can look at the cost as well.

[00:11:27] (cue 152) And this year, if you're looking at the coding agent or the model prices,

[00:11:32] (cue 153) usually the x axis is in log scale, which means you pay 10x for the new models.

[00:11:39] (cue 154) And when that happens, people start looking really at cost.

[00:11:42] (cue 155) And here we can measure cost as well and take that into into account in the

[00:11:47] (cue 156) equation. So the steps for measuring coding agent experience with your with your API

[00:11:54] (cue 157) is you design a scenario. You write an evaluation criteria for the for the judge

[00:12:00] (cue 158) explaining what good looks like.

[00:12:02] (cue 159) Then you choose a profile which is harness, model extensions and the OS.

[00:12:09] (cue 160) We that sometimes when you have a very generic scenario, if you ask a coding agent

[00:12:15] (cue 161) on Windows versus Linux, on Linux it will do that in Python by default,

[00:12:22] (cue 162) and on Windows in .NET. So even that has an influence.

[00:12:27] (cue 163) The path of your directory has an influence.

[00:12:30] (cue 164) So we discovered a lot of really weird things like that.

[00:12:34] (cue 165) Then you need to make many runs.

[00:12:36] (cue 166) So a lot of evaluation tools are running on your laptop.

[00:12:40] (cue 167) We designed a system that runs on Kubernetes.

[00:12:42] (cue 168) It also scales with KEDA and and so you can have many runs and you want to run many

[00:12:48] (cue 169) runs to have statistically valid results.

[00:12:52] (cue 170) And then you generate a readout and then you can investigate the trajectories.

[00:13:40] (cue 182) So we are. So we built that playbook.

[00:13:44] (cue 183) You can you can find,

[00:13:48] (cue 184) you can find a skill. We have a skill that you can add in your agent so that your

[00:13:53] (cue 185) agent knows about the playbook and can design the experiments for you.

[00:13:56] (cue 186) And then to put the playbook in action.

[00:13:58] (cue 187) We built a system called Scope.

[00:14:01] (cue 188) So that's the implementation of the playbook.

[00:14:03] (cue 189) It's a pretty simple system where it's running on Kubernetes.

[00:14:09] (cue 190) You can define all these things, the task criteria etc.

[00:14:13] (cue 191) define some containers for your different agents.

[00:14:16] (cue 192) Give them the tokens. Then you have a portal, a CLI and a REST API.

[00:14:23] (cue 193) So your agent has a choice for how it can interact with Scope.

[00:14:26] (cue 194) And then Scope is running on top of AKS.

[00:14:29] (cue 195) You can run it locally on your laptop with Docker Compose, but it's designed to run

[00:14:33] (cue 196) on Kubernetes with an auto scale, and then it's running all the scenarios,

[00:14:38] (cue 197) run the gates, and then gives you a readout that you can then analyze,

[00:14:44] (cue 198) and you can go into the ATIF trajectories of each of the agents that are problematic

[00:14:50] (cue 199) to try to understand what to do better.

[00:14:52] (cue 200) And so our goal with this longer term is to have like agents read all these and then

[00:14:58] (cue 201) go do the fixes automatically in the loop.

[00:15:00] (cue 202) We're not there yet. So there's a practitioner skill that you can use for your for

[00:15:07] (cue 203) your readouts. And let me show you a quick demo.

[00:15:11] (cue 204) So the demo in this demo the scenario I'm going to use is a,

[00:15:16] (cue 205) there's a specific library that Pamela Fox, one of our colleagues wrote called

[00:15:23] (cue 206) Jason. There's a bug in there.

[00:15:25] (cue 207) I'm just going to point to a GitHub

[00:15:28] (cue 208) to the URL of the issue for that library to fix.

[00:15:33] (cue 209) And I'm going to tell my my prompt will be, just go fix this bug in that in that

[00:15:38] (cue 210) library. And what I'm trying to measure here is the quality of the GitHub of

[00:15:45] (cue 211) GitHub as a tool used by agents.

[00:15:49] (cue 212) And so I'm going to do three things.

[00:15:51] (cue 213) I'm going to have a baseline where I don't have any tool in there.

[00:15:55] (cue 214) I define nothing, just curl, so it can curl to get access to GitHub with my token.

[00:16:03] (cue 215) Else I have the GitHub MCP so that the GitHub team built or the GitHub CLI.

[00:16:09] (cue 216) So I'm going to do a poll first.

[00:16:11] (cue 217) Which one do you think is the best agent experience between these three curl,

[00:16:16] (cue 218) GitHub, GitHub CLI or of the MCP server?

[00:16:24] (cue 219) Yeah, yeah, okay, MCP, okay different opinions.

[00:16:30] (cue 220) So we we run that test. I'm just going to show you the what it looks like.

[00:16:35] (cue 221) So that's Scope. That's the Scope UI.

[00:16:38] (cue 222) So in there I can create what is called a resource.

[00:16:41] (cue 223) So here the resource is something that your tool has access to.

[00:16:45] (cue 224) And here it's like a fake GitHub implementation that the system can interact with so

[00:16:52] (cue 225) that it's always in the same state for every runs.

[00:16:55] (cue 226) So it's basically a container emulating GitHub.

[00:16:59] (cue 227) Then I'm going to create different profiles.

[00:17:03] (cue 228) So here it's the GitHub simulator.

[00:17:04] (cue 229) I'm going to create a few profiles.

[00:17:06] (cue 230) One for the GitHub CLI, one for just the baseline and one for the MCP server.

[00:17:13] (cue 231) So I can define MCP servers and skills and add them in there.

[00:17:17] (cue 232) And then I'm going to launch it with a specific runs.

[00:17:21] (cue 233) And so I'm I'm asking it to do several runs for each of these for the same scenario.

[00:17:27] (cue 234) And at the end I'm going to look at what the results are, what the results look

[00:17:33] (cue 235) like. So that's how I submit a run in the system.

[00:17:36] (cue 236) So that's when you're using the UI for the portal.

[00:17:40] (cue 237) And what I'm going to do is I'm going to get out of the.

[00:17:45] (cue 238) So that's what a run result looks like.

[00:17:48] (cue 239) And I'm just going to go in the in the UI in there.

[00:17:53] (cue 240) This is the the app itself. And so this is the the previous thing.

[00:17:58] (cue 241) So here you can see the task I have the prompt text the different features.

[00:18:04] (cue 242) So as I have you can model things with features.

[00:18:08] (cue 243) And then you have your results with with the runs around it.

[00:18:15] (cue 244) So that's that's kind of what the,

[00:18:20] (cue 245) what Scope does for you.

[00:18:24] (cue 246) Let me just get out of this. And for the previous run.

[00:18:29] (cue 247) So we run 15 baseline runs, 15 for MCP and 15 for the GitHub CLI.

[00:18:35] (cue 248) The results were something like that, where it run like

[00:18:42] (cue 249) every, every experiment that we did, they fixed the bug and they it ran.

[00:18:48] (cue 250) It worked well. So you could say if you're doing an evaluation where is this.

[00:18:53] (cue 251) Like what is it useful to. What is useful is that even for something that works

[00:18:59] (cue 252) really well, a GitHub that is very well known by agents, there are some differences.

[00:19:03] (cue 253) So here when you start analyzing the data, you can do these violin diagrams here.

[00:19:09] (cue 254) And what you can see is that the GitHub CLI is both much faster for the agent to

[00:19:15] (cue 255) use, but also consumes much less token.

[00:19:18] (cue 256) And here for many people, the experience with with agents, they're thinking

[00:19:25] (cue 257) I'm going to use a specific like coding harness with a specific model.

[00:19:31] (cue 258) Should I use the large model for everything or can I go to a smaller model?

[00:19:36] (cue 259) And here we can measure exactly like with the tools that we are using in a specific

[00:19:41] (cue 260) company. And the model that we're using, like how much token are consumed and how

[00:19:47] (cue 261) fast it is. And then I can do some quality versus quantity and time

[00:19:54] (cue 262) compromise. So that's that's what Scope is bringing to the table.

[00:19:59] (cue 263) Now I wanted to talk about like the GitHub experience I showed you is a toy example

[00:20:05] (cue 264) Jay's been working with a team for months improving the experience on Cosmos DB.

[00:20:12] (cue 265) Yeah. Hey my name is Jay Gordon.

[00:20:14] (cue 266) I work on the Azure Cosmos DB team.

[00:20:16] (cue 267) I'm really looking at growth and being able to get the product into more people's

[00:20:21] (cue 268) hands. And one of those things is agent-led growth.

[00:20:23] (cue 269) That's a big part of what people are doing.

[00:20:25] (cue 270) But what is Cosmos DB? I have to get this part out.

[00:20:29] (cue 271) It is a fully managed NoSQL database service on Azure.

[00:20:33] (cue 272) It supports some of the biggest corporations in the world.

[00:20:38] (cue 273) And where's the rest of my slide?

[00:20:39] (cue 274) There we go. No managed and has support for vector database services.

[00:20:44] (cue 275) You can use it for full text search, hybrid search, all the cool things.

[00:20:49] (cue 276) It's used by these great companies.

[00:20:51] (cue 277) We are the backing database for every single message that is stored in ChatGPT.

[00:20:58] (cue 278) Anyone use that thing a little bit?

[00:21:00] (cue 279) Right? Cool. We got through that part.

[00:21:03] (cue 280) So this is something that I believe I've been using this term a lot.

[00:21:09] (cue 281) And the reason why is that most coding agents put out some really,

[00:21:15] (cue 282) really sloppy stuff. Let's let's face it, we all use the term AI slop.

[00:21:19] (cue 283) It is is what it is. So what we did was we used Scope to help us

[00:21:25] (cue 284) determine what was the difference between what bare models did and what having

[00:21:30] (cue 285) context in. So what we wanted to find out is does it build?

[00:21:34] (cue 286) Does it run, and does Cosmos DB actually work well?

[00:21:38] (cue 287) Is it implemented well? And so what we did was we asked two real questions

[00:21:45] (cue 288) choosing Cosmos DB. Well how did it do propensity Patrick talked about propensity

[00:21:51] (cue 289) for a second. So if I say I want you to build this application,

[00:21:55] (cue 290) whatever it is a CRUD application, and I want you to store the data in a NoSQL

[00:21:59] (cue 291) database, I want to know what every coding agent is telling me to do,

[00:22:04] (cue 292) because propensity is important.

[00:22:07] (cue 293) If I just said a generic prompt, as in NoSQL and I didn't actually mention a cloud,

[00:22:12] (cue 294) I didn't mention cloud at all.

[00:22:14] (cue 295) It won't build it for a cloud, it won't really infer that.

[00:22:17] (cue 296) So you're add a little nudge, you add a little bit more context.

[00:22:21] (cue 297) And so that's when you start. And I am so sorry.

[00:22:26] (cue 298) That's when you start looking at efficacy.

[00:22:28] (cue 299) And what is efficacy. It's to see if the code actually works if it's right.

[00:22:33] (cue 300) Because propensity just means that eventually you can figure out ways to have small

[00:22:37] (cue 301) amounts of inference and to be able to add, refine out exactly what was chosen,

[00:22:43] (cue 302) what was not chosen by the agent, and then efficacy.

[00:22:46] (cue 303) It was chosen by the agent, but it actually used it correctly.

[00:22:49] (cue 304) That's the thing we need to find out.

[00:22:51] (cue 305) So we built an agent kit and we wanted to basically put a Cosmos DB in your pocket.

[00:22:58] (cue 306) If you keep a person in your pocket, that's your decision.

[00:23:00] (cue 307) But at least we give you someone to help you along with building against what it is.

[00:23:06] (cue 308) It's a source. It's an open source collection of agent skills that gives AI coding

[00:23:11] (cue 309) assistants guidance on data modeling, partition keys, queries,

[00:23:15] (cue 310) and SDK usage. Because one of the biggest problems that people sometimes when they

[00:23:19] (cue 311) start working with something like a database is with an agent,

[00:23:24] (cue 312) it'll just put out whatever it knows, but it doesn't necessarily know the right

[00:23:28] (cue 313) things could train on old data.

[00:23:30] (cue 314) It could simply have access to old documentation.

[00:23:34] (cue 315) And so what we did was we did exactly what Patrick said.

[00:23:39] (cue 316) We built scenarios, we told it exactly what we wanted to build.

[00:23:43] (cue 317) And in this case, we weren't looking for propensity as much,

[00:23:47] (cue 318) but we were looking for efficacy.

[00:23:48] (cue 319) So we gave it just some real basic criteria that we needed to meet.

[00:23:53] (cue 320) And this is the expected scale.

[00:23:56] (cue 321) And so, you know, notice what had asked for.

[00:23:59] (cue 322) It asked for real scale requirements because that can help make a determination on

[00:24:04] (cue 323) how actually Cosmos DB was implemented.

[00:24:07] (cue 324) So we had the same prompt and five kinds of profiles.

[00:24:11] (cue 325) Those are the setups that Patrick was talking about.

[00:24:14] (cue 326) And so we kind of went through this process.

[00:24:17] (cue 327) We created a control version, no context, bare models.

[00:24:21] (cue 328) And we wanted to go across the entire spectrum if we could.

[00:24:24] (cue 329) Claude, Codex, Copilot on Windows, Copilot CLI, or I should say Copilot on VS

[00:24:31] (cue 330) Code on Windows. So this team at Scope built me all these options and I needed to

[00:24:37] (cue 331) take it and then go ahead and I can add a little more.

[00:24:40] (cue 332) So I wanted to test how it just responded with having the GitHub Copilot for Azure

[00:24:45] (cue 333) MCP server, the same thing with documentation, the Microsoft Learn,

[00:24:49] (cue 334) and then the agent kit. The thing that we wanted to improve that these simply didn't

[00:24:55] (cue 335) have enough context for. And then we also wanted to just try agent kits with MCP

[00:25:00] (cue 336) servers. What was the combo like?

[00:25:02] (cue 337) And you know, we got three kinds of individuals.

[00:25:05] (cue 338) And then we ran to compare. So.

[00:25:12] (cue 339) We basically created these rule rubrics.

[00:25:15] (cue 340) We used an LLM judge. It's a little bit older version.

[00:25:18] (cue 341) As we were iterating you could see GPT-4.1.

[00:25:21] (cue 342) We've since updated that to a more modern version.

[00:25:23] (cue 343) Runtime checks. So we selected apps to run against the emulator.

[00:25:28] (cue 344) We also did this against real versions eventually with the code that was produced,

[00:25:33] (cue 345) but we used the emulator because then we can go ahead and actually make sure the

[00:25:36] (cue 346) test work. And then we did an A versus B comparison to determine what did the

[00:25:42] (cue 347) baseline say. And then B when we added context what came from there.

[00:25:46] (cue 348) So Scope, like I said, compared with the bare models or the agent kit application

[00:25:51] (cue 349) scale, scale, expose those patterns that just a single demo.

[00:25:57] (cue 350) So you can go into your agent and you can just say build me a CRUD application.

[00:26:02] (cue 351) But if you do it once, you're only get one version and it might not be the right

[00:26:05] (cue 352) version, it might be just something that it scraped off the internet and said,

[00:26:09] (cue 353) all right, well, here you go. Here's your CRUD app.

[00:26:12] (cue 354) It may not be what you were actually looking to do.

[00:26:14] (cue 355) So what we built were eight different eight M prompts.

[00:26:18] (cue 356) We tried five different languages: .NET, C#, Java, JavaScript,

[00:26:23] (cue 357) Node.js, Python, TypeScript. And we also started on Rust.

[00:26:26] (cue 358) Rust is still a little shaky, but you know, we actually are going to have that open

[00:26:32] (cue 359) source Rust driver for a Cosmos DB.

[00:26:35] (cue 360) Very, very I think it's GA in like a week or two.

[00:26:38] (cue 361) So for you Rust people give us a shot.

[00:26:41] (cue 362) And then we compared two setups.

[00:26:44] (cue 363) We used bare models and regular models with agent kit, and we all just ran

[00:26:51] (cue 364) it over and over and over again.

[00:26:52] (cue 365) We did hundreds and hundreds. And so each snapshot of code, every single one of

[00:26:58] (cue 366) those iterations that were done, produced a certain segment of code that were a

[00:27:02] (cue 367) snapshot of code that we eventually wanted to learn.

[00:27:05] (cue 368) We wanted to prove whatever the behavior is.

[00:27:07] (cue 369) So the first step we would review, we would just map the generated app dependencies,

[00:27:13] (cue 370) deployment assets, things like that.

[00:27:15] (cue 371) We would scan the snapshot against the agent kit, so we would grab all of the

[00:27:20] (cue 372) snapshots that were generated, all that code, and then we would go ahead and ask the

[00:27:24] (cue 373) agent kit, did it do these things right?

[00:27:26] (cue 374) Did it model data correctly? Did it pick the right partition key.

[00:27:30] (cue 375) Then we would do some cross-checking.

[00:27:32] (cue 376) We would look against the documentation and say, I literally would pull the entire

[00:27:37] (cue 377) documentation repo for anything Microsoft had out there about Cosmos DB,

[00:27:42] (cue 378) and then had it do comparative analysis against what was an agent kit and what code

[00:27:47] (cue 379) was produced. And then it was able to kind of find ways to make sure that the code

[00:27:52] (cue 380) built. Like Patrick said, we want our code to build and then eventually we want it

[00:27:57] (cue 381) to run. So I went through this whole process across, just like I said,

[00:28:01] (cue 382) countless amounts of iterations.

[00:28:04] (cue 383) And, you know, one of those iterations was a Python leaderboard.

[00:28:09] (cue 384) We basically just had one prompt and we gave it.

[00:28:13] (cue 385) We got better guidance as we tested.

[00:28:16] (cue 386) So we provided more context and then eventually we got better outputs.

[00:28:20] (cue 387) So we did 45 baseline 25 reruns.

[00:28:24] (cue 388) And then we actually made some changes to our agent kit.

[00:28:27] (cue 389) You can see the PR number right there.

[00:28:29] (cue 390) And so we found that one thing that we found was that there was a rule that gave the

[00:28:36] (cue 391) right approach, but it only gave it in C#.

[00:28:39] (cue 392) So I ran the Python evaluation and I didn't get the correct information.

[00:28:43] (cue 393) Matter of fact it put C# in.

[00:28:47] (cue 394) Cool. So one of the things that we were able to do is we took the usage of

[00:28:52] (cue 395) hierarchical partition keys, and it was a part of the application requirements,

[00:28:58] (cue 396) and 47% were in the baseline with the agent kit, based on us doing iterative

[00:29:05] (cue 397) tests with Scope, we were able to up that to 100% offset.

[00:29:11] (cue 398) It was really, really causing some real bad,

[00:29:17] (cue 399) ability to have quick queries.

[00:29:19] (cue 400) It would slow down. It would use way too much, resources would cost more.

[00:29:23] (cue 401) So we figured out a way to get that down from 40%, 47% to 0% auto scale.

[00:29:30] (cue 402) We were able to get all of these to actually choose our auto scale

[00:29:36] (cue 403) model instead of one of our standard provisions.

[00:29:39] (cue 404) RUs, our request units, are units of scale that equate to how much money people

[00:29:45] (cue 405) spend on Cosmos DB. That's a big part of this.

[00:29:48] (cue 406) We want to make sure that we help them save money through context.

[00:29:52] (cue 407) So we also kept finding the same problems.

[00:29:55] (cue 408) Documentation that needed to be updated.

[00:29:58] (cue 409) 45 agent kit issues and 81 documentation issues were handled through our testing SDK

[00:30:05] (cue 410) correctness. We made sure that the wrong packages dependency lifecycle things.

[00:30:09] (cue 411) We found all that and then we found efficient data access.

[00:30:12] (cue 412) So cross partition scans which would definitely slow down how your data is accessed

[00:30:16] (cue 413) during queries, things like that.

[00:30:18] (cue 414) Point reads. And then we wanted to make sure that safe data design,

[00:30:22] (cue 415) mutable partitions, things like that was built in.

[00:30:26] (cue 416) We wanted a reliable source guidance, so we didn't want it to use broken samples.

[00:30:30] (cue 417) Something from some random GitHub repo that doesn't have the right information.

[00:30:34] (cue 418) We want to avoid that. So the findings shipped is reusable improvements.

[00:30:41] (cue 419) What did we do. We we basically try to have every one of these iterations and turn

[00:30:47] (cue 420) into a step that changes. So what do we do.

[00:30:49] (cue 421) We created 20 different agent kit issues.

[00:30:52] (cue 422) We fixed 27 documentation issues.

[00:30:56] (cue 423) We picked 47 merged back resolutions.

[00:30:59] (cue 424) So those were the total amount of PRs that got merged in and put into production for

[00:31:04] (cue 425) our agent kit. And we saw our agent kit outcomes improve.

[00:31:09] (cue 426) We saw our documentation improved based on all this testing,

[00:31:13] (cue 427) and we went ahead and we were able to improve our products.

[00:31:16] (cue 428) So by actually being a customer of an internal product, we were actually able to

[00:31:21] (cue 429) help you improve it too. So I gave feedback every single time.

[00:31:26] (cue 430) And his team took every bit of feedback to heart and built out a product that not

[00:31:32] (cue 431) just suited for me, but for everyone at Microsoft and now in the open source world

[00:31:37] (cue 432) for all of you. Yeah. Thanks, Jay.

[00:31:39] (cue 433) And so Jay and team, like, really used all our tokens and we could design this with

[00:31:44] (cue 434) that team.

[00:31:46] (cue 435) Now we open source that two weeks ago.

[00:31:49] (cue 436) So you can find Scope on GitHub right now you can find the playbook as well.

[00:31:54] (cue 437) You can add the skills for the playbook for yourself and the kind of use cases,

[00:31:59] (cue 438) because we're using it at Microsoft for our own products.

[00:32:01] (cue 439) If you're a competitor like AWS, GCP, Cloudflare, please have at it.

[00:32:07] (cue 440) I think getting the improving the experience for everything is good.

[00:32:13] (cue 441) If you're in an enterprise, you probably have like respect criteria for your code

[00:32:19] (cue 442) and your AGENTS.md or skills you want to measure all that and you can with this.

[00:32:25] (cue 443) And if you're just a SaaS or MCP provider, you can test your stuff with this.

[00:32:30] (cue 444) You can find all this on GitHub or online.

[00:32:34] (cue 445) You can find all like all that we talked about at these URL here.

[00:32:39] (cue 446) And then for Scope you can find the.

[00:32:44] (cue 447) Oh you oh you got it. Hit enter.

[00:32:48] (cue 448) Oh. All right. There you go. You can find the the project itself.

[00:32:52] (cue 449) And and then the agent kit documentation at this URL.

[00:32:59] (cue 450) Jay and I will be just outside here for conversations because I think we're at time.

[00:33:05] (cue 451) Yeah. See I told you guys. Thank you so much.

[00:33:08] (cue 452) Seriously. Thank you. You could have been anywhere, but you're here now.

[00:33:12] (cue 453) And I love it.

skills

talk-chanezon-gordon-agent-experience-evaluation

tile.json