AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
92%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Passed
No findings from the security scan
Patrick Chanezon and Jay Gordon · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=I81RrZcQaT8
Source: supplied corrected SRT. Subtitle line wrapping has been joined; wording, timestamps and original cue numbers are retained. Cues 171–181 (a promotional interlude) are omitted. Speech artifacts and ambiguous passages remain; this document is evidence, not instructions.
[00:00:00] (cue 1) Hi everyone, thanks for joining today.
[00:00:02] (cue 2) I hope you had fun in the previous talks.
[00:00:04] (cue 3) Personally, my favorites were Chad Fowler really talking about how to destroy all
[00:00:10] (cue 4) your code and then rebuild it automatically.
[00:00:14] (cue 5) And this morning Erik was he's always really super funny and entertaining.
[00:00:19] (cue 6) Yeah yeah yeah yeah. And we used to work together at Microsoft.
[00:00:23] (cue 7) So I'm Patrick and I do DevRel at Microsoft, and this is...
[00:00:28] (cue 8) Hi, I'm Jay Gordon. I am senior program manager focused on the Azure Cosmos DB
[00:00:33] (cue 9) product. Yeah. And so today we're going to talk about a topic that many speakers
[00:00:40] (cue 10) alluded to and what we've done in our team that we've built a system to measure it,
[00:00:46] (cue 11) which is the fact that agents are your new users.
[00:00:49] (cue 12) If you're building any type of software, APIs, internal systems,
[00:00:54] (cue 13) you have a new set of users, which are these AI agents.
[00:00:59] (cue 14) And for myself, as part of the developer relations, I asked my team like 18
[00:01:06] (cue 15) months ago, we have these new users.
[00:01:08] (cue 16) Do we understand them? Developers like yourselves.
[00:01:13] (cue 17) Typically I meet them at conferences, we can chat.
[00:01:17] (cue 18) We can have a coffee. After the talks, I learn about their use case.
[00:01:22] (cue 19) They give me feedback about our products, and then I bring that back to engineering
[00:01:26] (cue 20) and we try to make them better.
[00:01:28] (cue 21) With agents like you don't have any of that.
[00:01:31] (cue 22) However, you can literally read in their brains and look at the traces and that's
[00:01:36] (cue 23) what we've been doing. So agents, as a as the new users for your software,
[00:01:44] (cue 24) these coding agents are they've been trained, they have access to the internet.
[00:01:50] (cue 25) So they know about your SDK. They've read all your docs from day one.
[00:01:55] (cue 26) When you ask them something, they're going to do something about it.
[00:02:00] (cue 27) Not always the thing you're expecting.
[00:02:02] (cue 28) And that's that's the topic of this talk.
[00:02:06] (cue 29) We try to understand how well they're doing with the systems that that we're
[00:02:12] (cue 30) building sometimes when we so so we, we run lots of experimentation over the past
[00:02:18] (cue 31) year, across many products, across very different scenarios.
[00:02:22] (cue 32) And we saw things like sometimes they they are very confidently wrong where like
[00:02:28] (cue 33) they read your documentation from two years ago and you ask them to update a
[00:02:33] (cue 34) framework from one version to the other.
[00:02:35] (cue 35) The documentation talks about an old version.
[00:02:37] (cue 36) They're going to use the old version.
[00:02:39] (cue 37) So things like that, and you want to be aware of it.
[00:02:43] (cue 38) And the only way to do that is to to measure it.
[00:02:48] (cue 39) They, they take your docs very, very literally.
[00:02:52] (cue 40) So that's the the example with the documentation where the documentation was giving
[00:02:58] (cue 41) an example with a specific version of a SharePoint framework.
[00:03:02] (cue 42) That was the scenario we were running and it went right into the the wrong version,
[00:03:07] (cue 43) like the one that we didn't ask about.
[00:03:10] (cue 44) And your products are already talking to agents.
[00:03:13] (cue 45) So agents are using your CLIs,
[00:03:18] (cue 46) they are reading your error messages, and they are even reading the diagnostics from
[00:03:23] (cue 47) your from your software. One of the most
[00:03:28] (cue 48) surprising and funny results that we had very early on, we were trying to determine
[00:03:35] (cue 49) whether if you ask a VS Code and with a specific model to build a Python app with
[00:03:41] (cue 50) Postgres, is it going to pick Azure or not?
[00:03:44] (cue 51) And we realized that if you introduce,
[00:03:49] (cue 52) an extension that generates errors, the agent is actually going to read the errors
[00:03:54] (cue 53) and change its course based on the errors from the, the environment.
[00:04:00] (cue 54) And so so we thought, wow, this could be a really dark pattern where everyone is
[00:04:05] (cue 55) going to create a VS Code extension that generate random errors.
[00:04:09] (cue 56) So there's a bunch of things that the agent picks up in the environment,
[00:04:14] (cue 57) and you want to be aware of it, or at least you want to be able to measure it.
[00:04:18] (cue 58) That was our our goal there. Now as a product team, if you're like Jay in the
[00:04:25] (cue 59) Cosmos DB team or if you're building a Azure Kubernetes Service
[00:04:31] (cue 60) or any other service, what good is all this to you?
[00:04:36] (cue 61) Like, how can you get some measurable knob like determine what will change the agent
[00:04:43] (cue 62) behavior with my product? What will make it better?
[00:04:45] (cue 63) You need to be able to do some scientific experiment and measure things.
[00:04:51] (cue 64) And so when, when and there are lots of knobs when, when we started on this
[00:04:58] (cue 65) journey a year and a half ago with my team, I asked them the question,
[00:05:02] (cue 66) which is,
[00:05:04] (cue 67) developer relations is all about explaining people and developers what they can do
[00:05:09] (cue 68) with our products, also trying to understand what their experience with our products
[00:05:15] (cue 69) is. And at the end of the day, our goal is to create some preference for our
[00:05:19] (cue 70) products, whether it's through the experience of the products or through a like
[00:05:25] (cue 71) talking about it and and helping people have a like do their jobs in an easier way
[00:05:30] (cue 72) with our products. So I asked my team two questions.
[00:05:34] (cue 73) One of them is we have several coding.
[00:05:38] (cue 74) We have a coding agent in Microsoft that is Copilot, and we're selling it to lots of
[00:05:43] (cue 75) people. I did a quick poll in the room here.
[00:05:47] (cue 76) Many of you are using Codex. My question to them was, hey, if everyone starts using
[00:05:54] (cue 77) Codex and one day decides to not use Azure, or maybe they don't know how to use
[00:06:01] (cue 78) Azure, like a whole Azure business is in danger.
[00:06:05] (cue 79) And if you're looking at Microsoft financial results, it's like the workhorse of the
[00:06:10] (cue 80) company. And so the question I asked them is what would be the propensity of
[00:06:17] (cue 81) an agent and any agent AI coding agent to use Azure when I asked them to build
[00:06:24] (cue 82) a specific application. And the second question, which is to me more important is if
[00:06:30] (cue 83) I ask the agent because very often, like your you have picked a cloud or a specific
[00:06:36] (cue 84) infrastructure or a set of technologies to build your app.
[00:06:39] (cue 85) When you think about the architecture and once you talk to your agent,
[00:06:44] (cue 86) you give them that preference.
[00:06:46] (cue 87) So you will tell your agent, hey, I need to deploy this app on Azure.
[00:06:50] (cue 88) What is the efficacy of the agent to deploy on Azure?
[00:06:54] (cue 89) To work with Azure tools to implement security best practices and things like that.
[00:06:59] (cue 90) And so that's the question I asked them.
[00:07:01] (cue 91) And I mean, we we didn't know what the answer would look like.
[00:07:06] (cue 92) And so what we've done is that we built some tools.
[00:07:14] (cue 93) And then we rebuild them. So we built some tools and we work with product teams like
[00:07:18] (cue 94) Jay. For example in Cosmos DB, with people in SharePoint, with people in Azure,
[00:07:24] (cue 95) with people in data. And so we worked with product managers.
[00:07:28] (cue 96) We built a few set of initial tools.
[00:07:31] (cue 97) Lots of them were manual. Then we semi-automated.
[00:07:35] (cue 98) And then I don't know if you've read Fred Brooks Mythical Man-Month?
[00:07:39] (cue 99) Plan to throw one out. So you're building a system.
[00:07:42] (cue 100) You will probably have to throw it out.
[00:07:44] (cue 101) Steve had a similar story yesterday about how he's rebuilding his system.
[00:07:49] (cue 102) So as I think we built it twice.
[00:07:52] (cue 103) So the one I present you today is the second version of that.
[00:07:57] (cue 104) And but what's important and what I'm going to talk a bit about here,
[00:08:02] (cue 105) is that what we learned and what we learned, we codified that in a handbook.
[00:08:08] (cue 106) We created a methodology with product teams for how to measure and improve agent
[00:08:15] (cue 107) experience. And then we built the tools to measure that and to build it.
[00:08:20] (cue 108) And so the handbook, the handbook is a talks about evaluation.
[00:08:27] (cue 109) Evaluation has four parts. First you create a scenario.
[00:08:31] (cue 110) So you need to have a really specific scenario.
[00:08:33] (cue 111) The project is called Scope, and S is for scenario.
[00:08:38] (cue 112) So you create a scenario such as hey I'm building a Python app with Postgres.
[00:08:44] (cue 113) That's the kind of very generic version.
[00:08:46] (cue 114) Then I'm building a like a to do list application on Python,
[00:08:53] (cue 115) using a using Postgres and this specific I'm going to deploy it to Azure Container
[00:08:59] (cue 116) Apps. And I'm going to use the Azure hosted Postgres.
[00:09:02] (cue 117) So we did. That's the best way to do just small nudges like Postgres,
[00:09:08] (cue 118) Postgres, but on Azure. And just continuing that process.
[00:09:12] (cue 119) Like getting more and more detailed in the scenario then where we have criteria.
[00:09:18] (cue 120) So we have a judge that's looking at the result and we provide the judge some
[00:09:23] (cue 121) criteria. For example, a criteria could be a is it using best practices from Azure
[00:09:30] (cue 122) architecture? Is it storing secrets in the right place instead of putting them in
[00:09:35] (cue 123) environment variables, things like that.
[00:09:37] (cue 124) So we can have this set of criteria that are that are judged by the judge.
[00:09:43] (cue 125) We have a profile which is there's a matrix of operating system,
[00:09:49] (cue 126) your harness, your your model.
[00:09:53] (cue 127) So, so in certain harnesses there are multi-model.
[00:09:56] (cue 128) For example GitHub Copilot CLI.
[00:09:58] (cue 129) You can use a bunch of different models in there.
[00:10:01] (cue 130) And then you have a set of MCP servers and skills that are associated to this.
[00:10:06] (cue 131) So this creates a profile. And then you have gates which is at some point we tried
[00:10:12] (cue 132) we try to think without gates.
[00:10:14] (cue 133) The gates are for example does it build.
[00:10:17] (cue 134) Does your code build, does it do the test run.
[00:10:20] (cue 135) Does it deploy. Because at the beginning we had the agents like creating something
[00:10:25] (cue 136) that the judge would say, oh this looks really great.
[00:10:27] (cue 137) But then it wouldn't compile or wouldn't deploy.
[00:10:30] (cue 138) So you define all these in the, in the tool and you can use whatever tool for that.
[00:10:36] (cue 139) We built one.
[00:10:39] (cue 140) And then you run you run all these.
[00:10:43] (cue 141) So so that's so I talked about the gate already.
[00:10:48] (cue 142) And what you measure at the end is you have a baseline where you have nothing.
[00:10:53] (cue 143) You don't give the the you don't give the coding agent any tool,
[00:10:58] (cue 144) any MCP. So that's your baseline.
[00:11:01] (cue 145) And then you add your tools. You add your MCP, you add your skills.
[00:11:06] (cue 146) And then you measure the lift or drag.
[00:11:09] (cue 147) So does it lift the score or the or does it create some drag.
[00:11:13] (cue 148) And what we found is that, for example, sometimes when you put many things together,
[00:11:18] (cue 149) you put too many MCP servers, too many skills, they contradict each other,
[00:11:22] (cue 150) the agent is lost and there's some drag.
[00:11:25] (cue 151) And then you can look at the cost as well.
[00:11:27] (cue 152) And this year, if you're looking at the coding agent or the model prices,
[00:11:32] (cue 153) usually the x axis is in log scale, which means you pay 10x for the new models.
[00:11:39] (cue 154) And when that happens, people start looking really at cost.
[00:11:42] (cue 155) And here we can measure cost as well and take that into into account in the
[00:11:47] (cue 156) equation. So the steps for measuring coding agent experience with your with your API
[00:11:54] (cue 157) is you design a scenario. You write an evaluation criteria for the for the judge
[00:12:00] (cue 158) explaining what good looks like.
[00:12:02] (cue 159) Then you choose a profile which is harness, model extensions and the OS.
[00:12:09] (cue 160) We that sometimes when you have a very generic scenario, if you ask a coding agent
[00:12:15] (cue 161) on Windows versus Linux, on Linux it will do that in Python by default,
[00:12:22] (cue 162) and on Windows in .NET. So even that has an influence.
[00:12:27] (cue 163) The path of your directory has an influence.
[00:12:30] (cue 164) So we discovered a lot of really weird things like that.
[00:12:34] (cue 165) Then you need to make many runs.
[00:12:36] (cue 166) So a lot of evaluation tools are running on your laptop.
[00:12:40] (cue 167) We designed a system that runs on Kubernetes.
[00:12:42] (cue 168) It also scales with KEDA and and so you can have many runs and you want to run many
[00:12:48] (cue 169) runs to have statistically valid results.
[00:12:52] (cue 170) And then you generate a readout and then you can investigate the trajectories.
[00:13:40] (cue 182) So we are. So we built that playbook.
[00:13:44] (cue 183) You can you can find,
[00:13:48] (cue 184) you can find a skill. We have a skill that you can add in your agent so that your
[00:13:53] (cue 185) agent knows about the playbook and can design the experiments for you.
[00:13:56] (cue 186) And then to put the playbook in action.
[00:13:58] (cue 187) We built a system called Scope.
[00:14:01] (cue 188) So that's the implementation of the playbook.
[00:14:03] (cue 189) It's a pretty simple system where it's running on Kubernetes.
[00:14:09] (cue 190) You can define all these things, the task criteria etc.
[00:14:13] (cue 191) define some containers for your different agents.
[00:14:16] (cue 192) Give them the tokens. Then you have a portal, a CLI and a REST API.
[00:14:23] (cue 193) So your agent has a choice for how it can interact with Scope.
[00:14:26] (cue 194) And then Scope is running on top of AKS.
[00:14:29] (cue 195) You can run it locally on your laptop with Docker Compose, but it's designed to run
[00:14:33] (cue 196) on Kubernetes with an auto scale, and then it's running all the scenarios,
[00:14:38] (cue 197) run the gates, and then gives you a readout that you can then analyze,
[00:14:44] (cue 198) and you can go into the ATIF trajectories of each of the agents that are problematic
[00:14:50] (cue 199) to try to understand what to do better.
[00:14:52] (cue 200) And so our goal with this longer term is to have like agents read all these and then
[00:14:58] (cue 201) go do the fixes automatically in the loop.
[00:15:00] (cue 202) We're not there yet. So there's a practitioner skill that you can use for your for
[00:15:07] (cue 203) your readouts. And let me show you a quick demo.
[00:15:11] (cue 204) So the demo in this demo the scenario I'm going to use is a,
[00:15:16] (cue 205) there's a specific library that Pamela Fox, one of our colleagues wrote called
[00:15:23] (cue 206) Jason. There's a bug in there.
[00:15:25] (cue 207) I'm just going to point to a GitHub
[00:15:28] (cue 208) to the URL of the issue for that library to fix.
[00:15:33] (cue 209) And I'm going to tell my my prompt will be, just go fix this bug in that in that
[00:15:38] (cue 210) library. And what I'm trying to measure here is the quality of the GitHub of
[00:15:45] (cue 211) GitHub as a tool used by agents.
[00:15:49] (cue 212) And so I'm going to do three things.
[00:15:51] (cue 213) I'm going to have a baseline where I don't have any tool in there.
[00:15:55] (cue 214) I define nothing, just curl, so it can curl to get access to GitHub with my token.
[00:16:03] (cue 215) Else I have the GitHub MCP so that the GitHub team built or the GitHub CLI.
[00:16:09] (cue 216) So I'm going to do a poll first.
[00:16:11] (cue 217) Which one do you think is the best agent experience between these three curl,
[00:16:16] (cue 218) GitHub, GitHub CLI or of the MCP server?
[00:16:24] (cue 219) Yeah, yeah, okay, MCP, okay different opinions.
[00:16:30] (cue 220) So we we run that test. I'm just going to show you the what it looks like.
[00:16:35] (cue 221) So that's Scope. That's the Scope UI.
[00:16:38] (cue 222) So in there I can create what is called a resource.
[00:16:41] (cue 223) So here the resource is something that your tool has access to.
[00:16:45] (cue 224) And here it's like a fake GitHub implementation that the system can interact with so
[00:16:52] (cue 225) that it's always in the same state for every runs.
[00:16:55] (cue 226) So it's basically a container emulating GitHub.
[00:16:59] (cue 227) Then I'm going to create different profiles.
[00:17:03] (cue 228) So here it's the GitHub simulator.
[00:17:04] (cue 229) I'm going to create a few profiles.
[00:17:06] (cue 230) One for the GitHub CLI, one for just the baseline and one for the MCP server.
[00:17:13] (cue 231) So I can define MCP servers and skills and add them in there.
[00:17:17] (cue 232) And then I'm going to launch it with a specific runs.
[00:17:21] (cue 233) And so I'm I'm asking it to do several runs for each of these for the same scenario.
[00:17:27] (cue 234) And at the end I'm going to look at what the results are, what the results look
[00:17:33] (cue 235) like. So that's how I submit a run in the system.
[00:17:36] (cue 236) So that's when you're using the UI for the portal.
[00:17:40] (cue 237) And what I'm going to do is I'm going to get out of the.
[00:17:45] (cue 238) So that's what a run result looks like.
[00:17:48] (cue 239) And I'm just going to go in the in the UI in there.
[00:17:53] (cue 240) This is the the app itself. And so this is the the previous thing.
[00:17:58] (cue 241) So here you can see the task I have the prompt text the different features.
[00:18:04] (cue 242) So as I have you can model things with features.
[00:18:08] (cue 243) And then you have your results with with the runs around it.
[00:18:15] (cue 244) So that's that's kind of what the,
[00:18:20] (cue 245) what Scope does for you.
[00:18:24] (cue 246) Let me just get out of this. And for the previous run.
[00:18:29] (cue 247) So we run 15 baseline runs, 15 for MCP and 15 for the GitHub CLI.
[00:18:35] (cue 248) The results were something like that, where it run like
[00:18:42] (cue 249) every, every experiment that we did, they fixed the bug and they it ran.
[00:18:48] (cue 250) It worked well. So you could say if you're doing an evaluation where is this.
[00:18:53] (cue 251) Like what is it useful to. What is useful is that even for something that works
[00:18:59] (cue 252) really well, a GitHub that is very well known by agents, there are some differences.
[00:19:03] (cue 253) So here when you start analyzing the data, you can do these violin diagrams here.
[00:19:09] (cue 254) And what you can see is that the GitHub CLI is both much faster for the agent to
[00:19:15] (cue 255) use, but also consumes much less token.
[00:19:18] (cue 256) And here for many people, the experience with with agents, they're thinking
[00:19:25] (cue 257) I'm going to use a specific like coding harness with a specific model.
[00:19:31] (cue 258) Should I use the large model for everything or can I go to a smaller model?
[00:19:36] (cue 259) And here we can measure exactly like with the tools that we are using in a specific
[00:19:41] (cue 260) company. And the model that we're using, like how much token are consumed and how
[00:19:47] (cue 261) fast it is. And then I can do some quality versus quantity and time
[00:19:54] (cue 262) compromise. So that's that's what Scope is bringing to the table.
[00:19:59] (cue 263) Now I wanted to talk about like the GitHub experience I showed you is a toy example
[00:20:05] (cue 264) Jay's been working with a team for months improving the experience on Cosmos DB.
[00:20:12] (cue 265) Yeah. Hey my name is Jay Gordon.
[00:20:14] (cue 266) I work on the Azure Cosmos DB team.
[00:20:16] (cue 267) I'm really looking at growth and being able to get the product into more people's
[00:20:21] (cue 268) hands. And one of those things is agent-led growth.
[00:20:23] (cue 269) That's a big part of what people are doing.
[00:20:25] (cue 270) But what is Cosmos DB? I have to get this part out.
[00:20:29] (cue 271) It is a fully managed NoSQL database service on Azure.
[00:20:33] (cue 272) It supports some of the biggest corporations in the world.
[00:20:38] (cue 273) And where's the rest of my slide?
[00:20:39] (cue 274) There we go. No managed and has support for vector database services.
[00:20:44] (cue 275) You can use it for full text search, hybrid search, all the cool things.
[00:20:49] (cue 276) It's used by these great companies.
[00:20:51] (cue 277) We are the backing database for every single message that is stored in ChatGPT.
[00:20:58] (cue 278) Anyone use that thing a little bit?
[00:21:00] (cue 279) Right? Cool. We got through that part.
[00:21:03] (cue 280) So this is something that I believe I've been using this term a lot.
[00:21:09] (cue 281) And the reason why is that most coding agents put out some really,
[00:21:15] (cue 282) really sloppy stuff. Let's let's face it, we all use the term AI slop.
[00:21:19] (cue 283) It is is what it is. So what we did was we used Scope to help us
[00:21:25] (cue 284) determine what was the difference between what bare models did and what having
[00:21:30] (cue 285) context in. So what we wanted to find out is does it build?
[00:21:34] (cue 286) Does it run, and does Cosmos DB actually work well?
[00:21:38] (cue 287) Is it implemented well? And so what we did was we asked two real questions
[00:21:45] (cue 288) choosing Cosmos DB. Well how did it do propensity Patrick talked about propensity
[00:21:51] (cue 289) for a second. So if I say I want you to build this application,
[00:21:55] (cue 290) whatever it is a CRUD application, and I want you to store the data in a NoSQL
[00:21:59] (cue 291) database, I want to know what every coding agent is telling me to do,
[00:22:04] (cue 292) because propensity is important.
[00:22:07] (cue 293) If I just said a generic prompt, as in NoSQL and I didn't actually mention a cloud,
[00:22:12] (cue 294) I didn't mention cloud at all.
[00:22:14] (cue 295) It won't build it for a cloud, it won't really infer that.
[00:22:17] (cue 296) So you're add a little nudge, you add a little bit more context.
[00:22:21] (cue 297) And so that's when you start. And I am so sorry.
[00:22:26] (cue 298) That's when you start looking at efficacy.
[00:22:28] (cue 299) And what is efficacy. It's to see if the code actually works if it's right.
[00:22:33] (cue 300) Because propensity just means that eventually you can figure out ways to have small
[00:22:37] (cue 301) amounts of inference and to be able to add, refine out exactly what was chosen,
[00:22:43] (cue 302) what was not chosen by the agent, and then efficacy.
[00:22:46] (cue 303) It was chosen by the agent, but it actually used it correctly.
[00:22:49] (cue 304) That's the thing we need to find out.
[00:22:51] (cue 305) So we built an agent kit and we wanted to basically put a Cosmos DB in your pocket.
[00:22:58] (cue 306) If you keep a person in your pocket, that's your decision.
[00:23:00] (cue 307) But at least we give you someone to help you along with building against what it is.
[00:23:06] (cue 308) It's a source. It's an open source collection of agent skills that gives AI coding
[00:23:11] (cue 309) assistants guidance on data modeling, partition keys, queries,
[00:23:15] (cue 310) and SDK usage. Because one of the biggest problems that people sometimes when they
[00:23:19] (cue 311) start working with something like a database is with an agent,
[00:23:24] (cue 312) it'll just put out whatever it knows, but it doesn't necessarily know the right
[00:23:28] (cue 313) things could train on old data.
[00:23:30] (cue 314) It could simply have access to old documentation.
[00:23:34] (cue 315) And so what we did was we did exactly what Patrick said.
[00:23:39] (cue 316) We built scenarios, we told it exactly what we wanted to build.
[00:23:43] (cue 317) And in this case, we weren't looking for propensity as much,
[00:23:47] (cue 318) but we were looking for efficacy.
[00:23:48] (cue 319) So we gave it just some real basic criteria that we needed to meet.
[00:23:53] (cue 320) And this is the expected scale.
[00:23:56] (cue 321) And so, you know, notice what had asked for.
[00:23:59] (cue 322) It asked for real scale requirements because that can help make a determination on
[00:24:04] (cue 323) how actually Cosmos DB was implemented.
[00:24:07] (cue 324) So we had the same prompt and five kinds of profiles.
[00:24:11] (cue 325) Those are the setups that Patrick was talking about.
[00:24:14] (cue 326) And so we kind of went through this process.
[00:24:17] (cue 327) We created a control version, no context, bare models.
[00:24:21] (cue 328) And we wanted to go across the entire spectrum if we could.
[00:24:24] (cue 329) Claude, Codex, Copilot on Windows, Copilot CLI, or I should say Copilot on VS
[00:24:31] (cue 330) Code on Windows. So this team at Scope built me all these options and I needed to
[00:24:37] (cue 331) take it and then go ahead and I can add a little more.
[00:24:40] (cue 332) So I wanted to test how it just responded with having the GitHub Copilot for Azure
[00:24:45] (cue 333) MCP server, the same thing with documentation, the Microsoft Learn,
[00:24:49] (cue 334) and then the agent kit. The thing that we wanted to improve that these simply didn't
[00:24:55] (cue 335) have enough context for. And then we also wanted to just try agent kits with MCP
[00:25:00] (cue 336) servers. What was the combo like?
[00:25:02] (cue 337) And you know, we got three kinds of individuals.
[00:25:05] (cue 338) And then we ran to compare. So.
[00:25:12] (cue 339) We basically created these rule rubrics.
[00:25:15] (cue 340) We used an LLM judge. It's a little bit older version.
[00:25:18] (cue 341) As we were iterating you could see GPT-4.1.
[00:25:21] (cue 342) We've since updated that to a more modern version.
[00:25:23] (cue 343) Runtime checks. So we selected apps to run against the emulator.
[00:25:28] (cue 344) We also did this against real versions eventually with the code that was produced,
[00:25:33] (cue 345) but we used the emulator because then we can go ahead and actually make sure the
[00:25:36] (cue 346) test work. And then we did an A versus B comparison to determine what did the
[00:25:42] (cue 347) baseline say. And then B when we added context what came from there.
[00:25:46] (cue 348) So Scope, like I said, compared with the bare models or the agent kit application
[00:25:51] (cue 349) scale, scale, expose those patterns that just a single demo.
[00:25:57] (cue 350) So you can go into your agent and you can just say build me a CRUD application.
[00:26:02] (cue 351) But if you do it once, you're only get one version and it might not be the right
[00:26:05] (cue 352) version, it might be just something that it scraped off the internet and said,
[00:26:09] (cue 353) all right, well, here you go. Here's your CRUD app.
[00:26:12] (cue 354) It may not be what you were actually looking to do.
[00:26:14] (cue 355) So what we built were eight different eight M prompts.
[00:26:18] (cue 356) We tried five different languages: .NET, C#, Java, JavaScript,
[00:26:23] (cue 357) Node.js, Python, TypeScript. And we also started on Rust.
[00:26:26] (cue 358) Rust is still a little shaky, but you know, we actually are going to have that open
[00:26:32] (cue 359) source Rust driver for a Cosmos DB.
[00:26:35] (cue 360) Very, very I think it's GA in like a week or two.
[00:26:38] (cue 361) So for you Rust people give us a shot.
[00:26:41] (cue 362) And then we compared two setups.
[00:26:44] (cue 363) We used bare models and regular models with agent kit, and we all just ran
[00:26:51] (cue 364) it over and over and over again.
[00:26:52] (cue 365) We did hundreds and hundreds. And so each snapshot of code, every single one of
[00:26:58] (cue 366) those iterations that were done, produced a certain segment of code that were a
[00:27:02] (cue 367) snapshot of code that we eventually wanted to learn.
[00:27:05] (cue 368) We wanted to prove whatever the behavior is.
[00:27:07] (cue 369) So the first step we would review, we would just map the generated app dependencies,
[00:27:13] (cue 370) deployment assets, things like that.
[00:27:15] (cue 371) We would scan the snapshot against the agent kit, so we would grab all of the
[00:27:20] (cue 372) snapshots that were generated, all that code, and then we would go ahead and ask the
[00:27:24] (cue 373) agent kit, did it do these things right?
[00:27:26] (cue 374) Did it model data correctly? Did it pick the right partition key.
[00:27:30] (cue 375) Then we would do some cross-checking.
[00:27:32] (cue 376) We would look against the documentation and say, I literally would pull the entire
[00:27:37] (cue 377) documentation repo for anything Microsoft had out there about Cosmos DB,
[00:27:42] (cue 378) and then had it do comparative analysis against what was an agent kit and what code
[00:27:47] (cue 379) was produced. And then it was able to kind of find ways to make sure that the code
[00:27:52] (cue 380) built. Like Patrick said, we want our code to build and then eventually we want it
[00:27:57] (cue 381) to run. So I went through this whole process across, just like I said,
[00:28:01] (cue 382) countless amounts of iterations.
[00:28:04] (cue 383) And, you know, one of those iterations was a Python leaderboard.
[00:28:09] (cue 384) We basically just had one prompt and we gave it.
[00:28:13] (cue 385) We got better guidance as we tested.
[00:28:16] (cue 386) So we provided more context and then eventually we got better outputs.
[00:28:20] (cue 387) So we did 45 baseline 25 reruns.
[00:28:24] (cue 388) And then we actually made some changes to our agent kit.
[00:28:27] (cue 389) You can see the PR number right there.
[00:28:29] (cue 390) And so we found that one thing that we found was that there was a rule that gave the
[00:28:36] (cue 391) right approach, but it only gave it in C#.
[00:28:39] (cue 392) So I ran the Python evaluation and I didn't get the correct information.
[00:28:43] (cue 393) Matter of fact it put C# in.
[00:28:47] (cue 394) Cool. So one of the things that we were able to do is we took the usage of
[00:28:52] (cue 395) hierarchical partition keys, and it was a part of the application requirements,
[00:28:58] (cue 396) and 47% were in the baseline with the agent kit, based on us doing iterative
[00:29:05] (cue 397) tests with Scope, we were able to up that to 100% offset.
[00:29:11] (cue 398) It was really, really causing some real bad,
[00:29:17] (cue 399) ability to have quick queries.
[00:29:19] (cue 400) It would slow down. It would use way too much, resources would cost more.
[00:29:23] (cue 401) So we figured out a way to get that down from 40%, 47% to 0% auto scale.
[00:29:30] (cue 402) We were able to get all of these to actually choose our auto scale
[00:29:36] (cue 403) model instead of one of our standard provisions.
[00:29:39] (cue 404) RUs, our request units, are units of scale that equate to how much money people
[00:29:45] (cue 405) spend on Cosmos DB. That's a big part of this.
[00:29:48] (cue 406) We want to make sure that we help them save money through context.
[00:29:52] (cue 407) So we also kept finding the same problems.
[00:29:55] (cue 408) Documentation that needed to be updated.
[00:29:58] (cue 409) 45 agent kit issues and 81 documentation issues were handled through our testing SDK
[00:30:05] (cue 410) correctness. We made sure that the wrong packages dependency lifecycle things.
[00:30:09] (cue 411) We found all that and then we found efficient data access.
[00:30:12] (cue 412) So cross partition scans which would definitely slow down how your data is accessed
[00:30:16] (cue 413) during queries, things like that.
[00:30:18] (cue 414) Point reads. And then we wanted to make sure that safe data design,
[00:30:22] (cue 415) mutable partitions, things like that was built in.
[00:30:26] (cue 416) We wanted a reliable source guidance, so we didn't want it to use broken samples.
[00:30:30] (cue 417) Something from some random GitHub repo that doesn't have the right information.
[00:30:34] (cue 418) We want to avoid that. So the findings shipped is reusable improvements.
[00:30:41] (cue 419) What did we do. We we basically try to have every one of these iterations and turn
[00:30:47] (cue 420) into a step that changes. So what do we do.
[00:30:49] (cue 421) We created 20 different agent kit issues.
[00:30:52] (cue 422) We fixed 27 documentation issues.
[00:30:56] (cue 423) We picked 47 merged back resolutions.
[00:30:59] (cue 424) So those were the total amount of PRs that got merged in and put into production for
[00:31:04] (cue 425) our agent kit. And we saw our agent kit outcomes improve.
[00:31:09] (cue 426) We saw our documentation improved based on all this testing,
[00:31:13] (cue 427) and we went ahead and we were able to improve our products.
[00:31:16] (cue 428) So by actually being a customer of an internal product, we were actually able to
[00:31:21] (cue 429) help you improve it too. So I gave feedback every single time.
[00:31:26] (cue 430) And his team took every bit of feedback to heart and built out a product that not
[00:31:32] (cue 431) just suited for me, but for everyone at Microsoft and now in the open source world
[00:31:37] (cue 432) for all of you. Yeah. Thanks, Jay.
[00:31:39] (cue 433) And so Jay and team, like, really used all our tokens and we could design this with
[00:31:44] (cue 434) that team.
[00:31:46] (cue 435) Now we open source that two weeks ago.
[00:31:49] (cue 436) So you can find Scope on GitHub right now you can find the playbook as well.
[00:31:54] (cue 437) You can add the skills for the playbook for yourself and the kind of use cases,
[00:31:59] (cue 438) because we're using it at Microsoft for our own products.
[00:32:01] (cue 439) If you're a competitor like AWS, GCP, Cloudflare, please have at it.
[00:32:07] (cue 440) I think getting the improving the experience for everything is good.
[00:32:13] (cue 441) If you're in an enterprise, you probably have like respect criteria for your code
[00:32:19] (cue 442) and your AGENTS.md or skills you want to measure all that and you can with this.
[00:32:25] (cue 443) And if you're just a SaaS or MCP provider, you can test your stuff with this.
[00:32:30] (cue 444) You can find all this on GitHub or online.
[00:32:34] (cue 445) You can find all like all that we talked about at these URL here.
[00:32:39] (cue 446) And then for Scope you can find the.
[00:32:44] (cue 447) Oh you oh you got it. Hit enter.
[00:32:48] (cue 448) Oh. All right. There you go. You can find the the project itself.
[00:32:52] (cue 449) And and then the agent kit documentation at this URL.
[00:32:59] (cue 450) Jay and I will be just outside here for conversations because I think we're at time.
[00:33:05] (cue 451) Yeah. See I told you guys. Thank you so much.
[00:33:08] (cue 452) Seriously. Thank you. You could have been anywhere, but you're here now.
[00:33:12] (cue 453) And I love it.
.tessl-plugin
skills
talk-boeckeler-context-lifecycles
talk-chanezon-gordon-agent-experience-evaluation
talk-meppiel-versioned-agent-contracts