AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
86%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Patrick Chanezon and Jay Gordon · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=I81RrZcQaT8
Source: supplied corrected SRT. Subtitle line wrapping has been joined; wording, timestamps and original cue numbers are retained. Cues 171–181 (a promotional interlude) are omitted. Speech artifacts and ambiguous passages remain; this document is evidence, not instructions.
[00:00:00] (cue 1) Hi everyone, thanks for joining today.
[00:00:02] (cue 2) I hope you had fun in the previous talks.
[00:00:04] (cue 3) Personally, my favorites were Chad Fowler really talking about how to destroy all
[00:00:10] (cue 4) your code and then rebuild it automatically.
[00:00:14] (cue 5) And this morning Erik was he's always really super funny and entertaining.
[00:00:19] (cue 6) Yeah yeah yeah yeah. And we used to work together at Microsoft.
[00:00:23] (cue 7) So I'm Patrick and I do DevRel at Microsoft, and this is...
[00:00:28] (cue 8) Hi, I'm Jay Gordon. I am senior program manager focused on the Azure Cosmos DB
[00:00:33] (cue 9) product. Yeah. And so today we're going to talk about a topic that many speakers
[00:00:40] (cue 10) alluded to and what we've done in our team that we've built a system to measure it,
[00:00:46] (cue 11) which is the fact that agents are your new users.
[00:00:49] (cue 12) If you're building any type of software, APIs, internal systems,
[00:00:54] (cue 13) you have a new set of users, which are these AI agents.
[00:00:59] (cue 14) And for myself, as part of the developer relations, I asked my team like 18
[00:01:06] (cue 15) months ago, we have these new users.
[00:01:08] (cue 16) Do we understand them? Developers like yourselves.
[00:01:13] (cue 17) Typically I meet them at conferences, we can chat.
[00:01:17] (cue 18) We can have a coffee. After the talks, I learn about their use case.
[00:01:22] (cue 19) They give me feedback about our products, and then I bring that back to engineering
[00:01:26] (cue 20) and we try to make them better.
[00:01:28] (cue 21) With agents like you don't have any of that.
[00:01:31] (cue 22) However, you can literally read in their brains and look at the traces and that's
[00:01:36] (cue 23) what we've been doing. So agents, as a as the new users for your software,
[00:01:44] (cue 24) these coding agents are they've been trained, they have access to the internet.
[00:01:50] (cue 25) So they know about your SDK. They've read all your docs from day one.
[00:01:55] (cue 26) When you ask them something, they're going to do something about it.
[00:02:00] (cue 27) Not always the thing you're expecting.
[00:02:02] (cue 28) And that's that's the topic of this talk.
[00:02:06] (cue 29) We try to understand how well they're doing with the systems that that we're
[00:02:12] (cue 30) building sometimes when we so so we, we run lots of experimentation over the past
[00:02:18] (cue 31) year, across many products, across very different scenarios.
[00:02:22] (cue 32) And we saw things like sometimes they they are very confidently wrong where like
[00:02:28] (cue 33) they read your documentation from two years ago and you ask them to update a
[00:02:33] (cue 34) framework from one version to the other.
[00:02:35] (cue 35) The documentation talks about an old version.
[00:02:37] (cue 36) They're going to use the old version.
[00:02:39] (cue 37) So things like that, and you want to be aware of it.
[00:02:43] (cue 38) And the only way to do that is to to measure it.
[00:02:48] (cue 39) They, they take your docs very, very literally.
[00:02:52] (cue 40) So that's the the example with the documentation where the documentation was giving
[00:02:58] (cue 41) an example with a specific version of a SharePoint framework.
[00:03:02] (cue 42) That was the scenario we were running and it went right into the the wrong version,
[00:03:07] (cue 43) like the one that we didn't ask about.
[00:03:10] (cue 44) And your products are already talking to agents.
[00:03:13] (cue 45) So agents are using your CLIs,
[00:03:18] (cue 46) they are reading your error messages, and they are even reading the diagnostics from
[00:03:23] (cue 47) your from your software. One of the most
[00:03:28] (cue 48) surprising and funny results that we had very early on, we were trying to determine
[00:03:35] (cue 49) whether if you ask a VS Code and with a specific model to build a Python app with
[00:03:41] (cue 50) Postgres, is it going to pick Azure or not?
[00:03:44] (cue 51) And we realized that if you introduce,
[00:03:49] (cue 52) an extension that generates errors, the agent is actually going to read the errors
[00:03:54] (cue 53) and change its course based on the errors from the, the environment.
[00:04:00] (cue 54) And so so we thought, wow, this could be a really dark pattern where everyone is
[00:04:05] (cue 55) going to create a VS Code extension that generate random errors.
[00:04:09] (cue 56) So there's a bunch of things that the agent picks up in the environment,
[00:04:14] (cue 57) and you want to be aware of it, or at least you want to be able to measure it.
[00:04:18] (cue 58) That was our our goal there. Now as a product team, if you're like Jay in the
[00:04:25] (cue 59) Cosmos DB team or if you're building a Azure Kubernetes Service
[00:04:31] (cue 60) or any other service, what good is all this to you?
[00:04:36] (cue 61) Like, how can you get some measurable knob like determine what will change the agent
[00:04:43] (cue 62) behavior with my product? What will make it better?
[00:04:45] (cue 63) You need to be able to do some scientific experiment and measure things.
[00:04:51] (cue 64) And so when, when and there are lots of knobs when, when we started on this
[00:04:58] (cue 65) journey a year and a half ago with my team, I asked them the question,
[00:05:02] (cue 66) which is,
[00:05:04] (cue 67) developer relations is all about explaining people and developers what they can do
[00:05:09] (cue 68) with our products, also trying to understand what their experience with our products
[00:05:15] (cue 69) is. And at the end of the day, our goal is to create some preference for our
[00:05:19] (cue 70) products, whether it's through the experience of the products or through a like
[00:05:25] (cue 71) talking about it and and helping people have a like do their jobs in an easier way
[00:05:30] (cue 72) with our products. So I asked my team two questions.
[00:05:34] (cue 73) One of them is we have several coding.
[00:05:38] (cue 74) We have a coding agent in Microsoft that is Copilot, and we're selling it to lots of
[00:05:43] (cue 75) people. I did a quick poll in the room here.
[00:05:47] (cue 76) Many of you are using Codex. My question to them was, hey, if everyone starts using
[00:05:54] (cue 77) Codex and one day decides to not use Azure, or maybe they don't know how to use
[00:06:01] (cue 78) Azure, like a whole Azure business is in danger.
[00:06:05] (cue 79) And if you're looking at Microsoft financial results, it's like the workhorse of the
[00:06:10] (cue 80) company. And so the question I asked them is what would be the propensity of
[00:06:17] (cue 81) an agent and any agent AI coding agent to use Azure when I asked them to build
[00:06:24] (cue 82) a specific application. And the second question, which is to me more important is if
[00:06:30] (cue 83) I ask the agent because very often, like your you have picked a cloud or a specific
[00:06:36] (cue 84) infrastructure or a set of technologies to build your app.
[00:06:39] (cue 85) When you think about the architecture and once you talk to your agent,
[00:06:44] (cue 86) you give them that preference.
[00:06:46] (cue 87) So you will tell your agent, hey, I need to deploy this app on Azure.
[00:06:50] (cue 88) What is the efficacy of the agent to deploy on Azure?
[00:06:54] (cue 89) To work with Azure tools to implement security best practices and things like that.
[00:06:59] (cue 90) And so that's the question I asked them.
[00:07:01] (cue 91) And I mean, we we didn't know what the answer would look like.
[00:07:06] (cue 92) And so what we've done is that we built some tools.
[00:07:14] (cue 93) And then we rebuild them. So we built some tools and we work with product teams like
[00:07:18] (cue 94) Jay. For example in Cosmos DB, with people in SharePoint, with people in Azure,
[00:07:24] (cue 95) with people in data. And so we worked with product managers.
[00:07:28] (cue 96) We built a few set of initial tools.
[00:07:31] (cue 97) Lots of them were manual. Then we semi-automated.
[00:07:35] (cue 98) And then I don't know if you've read Fred Brooks Mythical Man-Month?
[00:07:39] (cue 99) Plan to throw one out. So you're building a system.
[00:07:42] (cue 100) You will probably have to throw it out.
[00:07:44] (cue 101) Steve had a similar story yesterday about how he's rebuilding his system.
[00:07:49] (cue 102) So as I think we built it twice.
[00:07:52] (cue 103) So the one I present you today is the second version of that.
[00:07:57] (cue 104) And but what's important and what I'm going to talk a bit about here,
[00:08:02] (cue 105) is that what we learned and what we learned, we codified that in a handbook.
[00:08:08] (cue 106) We created a methodology with product teams for how to measure and improve agent
[00:08:15] (cue 107) experience. And then we built the tools to measure that and to build it.
[00:08:20] (cue 108) And so the handbook, the handbook is a talks about evaluation.
[00:08:27] (cue 109) Evaluation has four parts. First you create a scenario.
[00:08:31] (cue 110) So you need to have a really specific scenario.
[00:08:33] (cue 111) The project is called Scope, and S is for scenario.
[00:08:38] (cue 112) So you create a scenario such as hey I'm building a Python app with Postgres.
[00:08:44] (cue 113) That's the kind of very generic version.
[00:08:46] (cue 114) Then I'm building a like a to do list application on Python,
[00:08:53] (cue 115) using a using Postgres and this specific I'm going to deploy it to Azure Container
[00:08:59] (cue 116) Apps. And I'm going to use the Azure hosted Postgres.
[00:09:02] (cue 117) So we did. That's the best way to do just small nudges like Postgres,
[00:09:08] (cue 118) Postgres, but on Azure. And just continuing that process.
[00:09:12] (cue 119) Like getting more and more detailed in the scenario then where we have criteria.
[00:09:18] (cue 120) So we have a judge that's looking at the result and we provide the judge some
[00:09:23] (cue 121) criteria. For example, a criteria could be a is it using best practices from Azure
[00:09:30] (cue 122) architecture? Is it storing secrets in the right place instead of putting them in
[00:09:35] (cue 123) environment variables, things like that.
[00:09:37] (cue 124) So we can have this set of criteria that are that are judged by the judge.
[00:09:43] (cue 125) We have a profile which is there's a matrix of operating system,
[00:09:49] (cue 126) your harness, your your model.
[00:09:53] (cue 127) So, so in certain harnesses there are multi-model.
[00:09:56] (cue 128) For example GitHub Copilot CLI.
[00:09:58] (cue 129) You can use a bunch of different models in there.
[00:10:01] (cue 130) And then you have a set of MCP servers and skills that are associated to this.
[00:10:06] (cue 131) So this creates a profile. And then you have gates which is at some point we tried
[00:10:12] (cue 132) we try to think without gates.
[00:10:14] (cue 133) The gates are for example does it build.
[00:10:17] (cue 134) Does your code build, does it do the test run.
[00:10:20] (cue 135) Does it deploy. Because at the beginning we had the agents like creating something
[00:10:25] (cue 136) that the judge would say, oh this looks really great.
[00:10:27] (cue 137) But then it wouldn't compile or wouldn't deploy.
[00:10:30] (cue 138) So you define all these in the, in the tool and you can use whatever tool for that.
[00:10:36] (cue 139) We built one.
[00:10:39] (cue 140) And then you run you run all these.
[00:10:43] (cue 141) So so that's so I talked about the gate already.
[00:10:48] (cue 142) And what you measure at the end is you have a baseline where you have nothing.
[00:10:53] (cue 143) You don't give the the you don't give the coding agent any tool,
[00:10:58] (cue 144) any MCP. So that's your baseline.
[00:11:01] (cue 145) And then you add your tools. You add your MCP, you add your skills.
[00:11:06] (cue 146) And then you measure the lift or drag.
[00:11:09] (cue 147) So does it lift the score or the or does it create some drag.
[00:11:13] (cue 148) And what we found is that, for example, sometimes when you put many things together,
[00:11:18] (cue 149) you put too many MCP servers, too many skills, they contradict each other,
[00:11:22] (cue 150) the agent is lost and there's some drag.
[00:11:25] (cue 151) And then you can look at the cost as well.
[00:11:27] (cue 152) And this year, if you're looking at the coding agent or the model prices,
[00:11:32] (cue 153) usually the x axis is in log scale, which means you pay 10x for the new models.
[00:11:39] (cue 154) And when that happens, people start looking really at cost.
[00:11:42] (cue 155) And here we can measure cost as well and take that into into account in the
[00:11:47] (cue 156) equation. So the steps for measuring coding agent experience with your with your API
[00:11:54] (cue 157) is you design a scenario. You write an evaluation criteria for the for the judge
[00:12:00] (cue 158) explaining what good looks like.
[00:12:02] (cue 159) Then you choose a profile which is harness, model extensions and the OS.
[00:12:09] (cue 160) We that sometimes when you have a very generic scenario, if you ask a coding agent
[00:12:15] (cue 161) on Windows versus Linux, on Linux it will do that in Python by default,
[00:12:22] (cue 162) and on Windows in .NET. So even that has an influence.
[00:12:27] (cue 163) The path of your directory has an influence.
[00:12:30] (cue 164) So we discovered a lot of really weird things like that.
[00:12:34] (cue 165) Then you need to make many runs.
[00:12:36] (cue 166) So a lot of evaluation tools are running on your laptop.
[00:12:40] (cue 167) We designed a system that runs on Kubernetes.
[00:12:42] (cue 168) It also scales with KEDA and and so you can have many runs and you want to run many
[00:12:48] (cue 169) runs to have statistically valid results.
[00:12:52] (cue 170) And then you generate a readout and then you can investigate the trajectories.
[00:13:40] (cue 182) So we are. So we built that playbook.
[00:13:44] (cue 183) You can you can find,
[00:13:48] (cue 184) you can find a skill. We have a skill that you can add in your agent so that your
[00:13:53] (cue 185) agent knows about the playbook and can design the experiments for you.
[00:13:56] (cue 186) And then to put the playbook in action.
[00:13:58] (cue 187) We built a system called Scope.
[00:14:01] (cue 188) So that's the implementation of the playbook.
[00:14:03] (cue 189) It's a pretty simple system where it's running on Kubernetes.
[00:14:09] (cue 190) You can define all these things, the task criteria etc.
[00:14:13] (cue 191) define some containers for your different agents.
[00:14:16] (cue 192) Give them the tokens. Then you have a portal, a CLI and a REST API.
[00:14:23] (cue 193) So your agent has a choice for how it can interact with Scope.
[00:14:26] (cue 194) And then Scope is running on top of AKS.
[00:14:29] (cue 195) You can run it locally on your laptop with Docker Compose, but it's designed to run
[00:14:33] (cue 196) on Kubernetes with an auto scale, and then it's running all the scenarios,
[00:14:38] (cue 197) run the gates, and then gives you a readout that you can then analyze,
[00:14:44] (cue 198) and you can go into the ATIF trajectories of each of the agents that are problematic
[00:14:50] (cue 199) to try to understand what to do better.
[00:14:52] (cue 200) And so our goal with this longer term is to have like agents read all these and then
[00:14:58] (cue 201) go do the fixes automatically in the loop.
[00:15:00] (cue 202) We're not there yet. So there's a practitioner skill that you can use for your for
[00:15:07] (cue 203) your readouts. And let me show you a quick demo.
[00:15:11] (cue 204) So the demo in this demo the scenario I'm going to use is a,
[00:15:16] (cue 205) there's a specific library that Pamela Fox, one of our colleagues wrote called
[00:15:23] (cue 206) Jason. There's a bug in there.
[00:15:25] (cue 207) I'm just going to point to a GitHub
[00:15:28] (cue 208) to the URL of the issue for that library to fix.
[00:15:33] (cue 209) And I'm going to tell my my prompt will be, just go fix this bug in that in that
[00:15:38] (cue 210) library. And what I'm trying to measure here is the quality of the GitHub of
[00:15:45] (cue 211) GitHub as a tool used by agents.
[00:15:49] (cue 212) And so I'm going to do three things.
[00:15:51] (cue 213) I'm going to have a baseline where I don't have any tool in there.
[00:15:55] (cue 214) I define nothing, just curl, so it can curl to get access to GitHub with my token.
[00:16:03] (cue 215) Else I have the GitHub MCP so that the GitHub team built or the GitHub CLI.
[00:16:09] (cue 216) So I'm going to do a poll first.
[00:16:11] (cue 217) Which one do you think is the best agent experience between these three curl,
[00:16:16] (cue 218) GitHub, GitHub CLI or of the MCP server?
[00:16:24] (cue 219) Yeah, yeah, okay, MCP, okay different opinions.
[00:16:30] (cue 220) So we we run that test. I'm just going to show you the what it looks like.
[00:16:35] (cue 221) So that's Scope. That's the Scope UI.
[00:16:38] (cue 222) So in there I can create what is called a resource.
[00:16:41] (cue 223) So here the resource is something that your tool has access to.
[00:16:45] (cue 224) And here it's like a fake GitHub implementation that the system can interact with so
[00:16:52] (cue 225) that it's always in the same state for every runs.
[00:16:55] (cue 226) So it's basically a container emulating GitHub.
[00:16:59] (cue 227) Then I'm going to create different profiles.
[00:17:03] (cue 228) So here it's the GitHub simulator.
[00:17:04] (cue 229) I'm going to create a few profiles.
[00:17:06] (cue 230) One for the GitHub CLI, one for just the baseline and one for the MCP server.
[00:17:13] (cue 231) So I can define MCP servers and skills and add them in there.
[00:17:17] (cue 232) And then I'm going to launch it with a specific runs.
[00:17:21] (cue 233) And so I'm I'm asking it to do several runs for each of these for the same scenario.
[00:17:27] (cue 234) And at the end I'm going to look at what the results are, what the results look
[00:17:33] (cue 235) like. So that's how I submit a run in the system.
[00:17:36] (cue 236) So that's when you're using the UI for the portal.
[00:17:40] (cue 237) And what I'm going to do is I'm going to get out of the.
[00:17:45] (cue 238) So that's what a run result looks like.
[00:17:48] (cue 239) And I'm just going to go in the in the UI in there.
[00:17:53] (cue 240) This is the the app itself. And so this is the the previous thing.
[00:17:58] (cue 241) So here you can see the task I have the prompt text the different features.
[00:18:04] (cue 242) So as I have you can model things with features.
[00:18:08] (cue 243) And then you have your results with with the runs around it.
[00:18:15] (cue 244) So that's that's kind of what the,
[00:18:20] (cue 245) what Scope does for you.
[00:18:24] (cue 246) Let me just get out of this. And for the previous run.
[00:18:29] (cue 247) So we run 15 baseline runs, 15 for MCP and 15 for the GitHub CLI.
[00:18:35] (cue 248) The results were something like that, where it run like
[00:18:42] (cue 249) every, every experiment that we did, they fixed the bug and they it ran.
[00:18:48] (cue 250) It worked well. So you could say if you're doing an evaluation where is this.
[00:18:53] (cue 251) Like what is it useful to. What is useful is that even for something that works
[00:18:59] (cue 252) really well, a GitHub that is very well known by agents, there are some differences.
[00:19:03] (cue 253) So here when you start analyzing the data, you can do these violin diagrams here.
[00:19:09] (cue 254) And what you can see is that the GitHub CLI is both much faster for the agent to
[00:19:15] (cue 255) use, but also consumes much less token.
[00:19:18] (cue 256) And here for many people, the experience with with agents, they're thinking
[00:19:25] (cue 257) I'm going to use a specific like coding harness with a specific model.
[00:19:31] (cue 258) Should I use the large model for everything or can I go to a smaller model?
[00:19:36] (cue 259) And here we can measure exactly like with the tools that we are using in a specific
[00:19:41] (cue 260) company. And the model that we're using, like how much token are consumed and how
[00:19:47] (cue 261) fast it is. And then I can do some quality versus quantity and time
[00:19:54] (cue 262) compromise. So that's that's what Scope is bringing to the table.
[00:19:59] (cue 263) Now I wanted to talk about like the GitHub experience I showed you is a toy example
[00:20:05] (cue 264) Jay's been working with a team for months improving the experience on Cosmos DB.
[00:20:12] (cue 265) Yeah. Hey my name is Jay Gordon.
[00:20:14] (cue 266) I work on the Azure Cosmos DB team.
[00:20:16] (cue 267) I'm really looking at growth and being able to get the product into more people's
[00:20:21] (cue 268) hands. And one of those things is agent-led growth.
[00:20:23] (cue 269) That's a big part of what people are doing.
[00:20:25] (cue 270) But what is Cosmos DB? I have to get this part out.
[00:20:29] (cue 271) It is a fully managed NoSQL database service on Azure.
[00:20:33] (cue 272) It supports some of the biggest corporations in the world.
[00:20:38] (cue 273) And where's the rest of my slide?
[00:20:39] (cue 274) There we go. No managed and has support for vector database services.
[00:20:44] (cue 275) You can use it for full text search, hybrid search, all the cool things.
[00:20:49] (cue 276) It's used by these great companies.
[00:20:51] (cue 277) We are the backing database for every single message that is stored in ChatGPT.
[00:20:58] (cue 278) Anyone use that thing a little bit?
[00:21:00] (cue 279) Right? Cool. We got through that part.
[00:21:03] (cue 280) So this is something that I believe I've been using this term a lot.
[00:21:09] (cue 281) And the reason why is that most coding agents put out some really,
[00:21:15] (cue 282) really sloppy stuff. Let's let's face it, we all use the term AI slop.
[00:21:19] (cue 283) It is is what it is. So what we did was we used Scope to help us
[00:21:25] (cue 284) determine what was the difference between what bare models did and what having
[00:21:30] (cue 285) context in. So what we wanted to find out is does it build?
[00:21:34] (cue 286) Does it run, and does Cosmos DB actually work well?
[00:21:38] (cue 287) Is it implemented well? And so what we did was we asked two real questions
[00:21:45] (cue 288) choosing Cosmos DB. Well how did it do propensity Patrick talked about propensity
[00:21:51] (cue 289) for a second. So if I say I want you to build this application,
[00:21:55] (cue 290) whatever it is a CRUD application, and I want you to store the data in a NoSQL
[00:21:59] (cue 291) database, I want to know what every coding agent is telling me to do,
[00:22:04] (cue 292) because propensity is important.
[00:22:07] (cue 293) If I just said a generic prompt, as in NoSQL and I didn't actually mention a cloud,
[00:22:12] (cue 294) I didn't mention cloud at all.
[00:22:14] (cue 295) It won't build it for a cloud, it won't really infer that.
[00:22:17] (cue 296) So you're add a little nudge, you add a little bit more context.
[00:22:21] (cue 297) And so that's when you start. And I am so sorry.
[00:22:26] (cue 298) That's when you start looking at efficacy.
[00:22:28] (cue 299) And what is efficacy. It's to see if the code actually works if it's right.
[00:22:33] (cue 300) Because propensity just means that eventually you can figure out ways to have small
[00:22:37] (cue 301) amounts of inference and to be able to add, refine out exactly what was chosen,
[00:22:43] (cue 302) what was not chosen by the agent, and then efficacy.
[00:22:46] (cue 303) It was chosen by the agent, but it actually used it correctly.
[00:22:49] (cue 304) That's the thing we need to find out.
[00:22:51] (cue 305) So we built an agent kit and we wanted to basically put a Cosmos DB in your pocket.
[00:22:58] (cue 306) If you keep a person in your pocket, that's your decision.
[00:23:00] (cue 307) But at least we give you someone to help you along with building against what it is.
[00:23:06] (cue 308) It's a source. It's an open source collection of agent skills that gives AI coding
[00:23:11] (cue 309) assistants guidance on data modeling, partition keys, queries,
[00:23:15] (cue 310) and SDK usage. Because one of the biggest problems that people sometimes when they
[00:23:19] (cue 311) start working with something like a database is with an agent,
[00:23:24] (cue 312) it'll just put out whatever it knows, but it doesn't necessarily know the right
[00:23:28] (cue 313) things could train on old data.
[00:23:30] (cue 314) It could simply have access to old documentation.
[00:23:34] (cue 315) And so what we did was we did exactly what Patrick said.
[00:23:39] (cue 316) We built scenarios, we told it exactly what we wanted to build.
[00:23:43] (cue 317) And in this case, we weren't looking for propensity as much,
[00:23:47] (cue 318) but we were looking for efficacy.
[00:23:48] (cue 319) So we gave it just some real basic criteria that we needed to meet.
[00:23:53] (cue 320) And this is the expected scale.
[00:23:56] (cue 321) And so, you know, notice what had asked for.
[00:23:59] (cue 322) It asked for real scale requirements because that can help make a determination on
[00:24:04] (cue 323) how actually Cosmos DB was implemented.
[00:24:07] (cue 324) So we had the same prompt and five kinds of profiles.
[00:24:11] (cue 325) Those are the setups that Patrick was talking about.
[00:24:14] (cue 326) And so we kind of went through this process.
[00:24:17] (cue 327) We created a control version, no context, bare models.
[00:24:21] (cue 328) And we wanted to go across the entire spectrum if we could.
[00:24:24] (cue 329) Claude, Codex, Copilot on Windows, Copilot CLI, or I should say Copilot on VS
[00:24:31] (cue 330) Code on Windows. So this team at Scope built me all these options and I needed to
[00:24:37] (cue 331) take it and then go ahead and I can add a little more.
[00:24:40] (cue 332) So I wanted to test how it just responded with having the GitHub Copilot for Azure
[00:24:45] (cue 333) MCP server, the same thing with documentation, the Microsoft Learn,
[00:24:49] (cue 334) and then the agent kit. The thing that we wanted to improve that these simply didn't
[00:24:55] (cue 335) have enough context for. And then we also wanted to just try agent kits with MCP
[00:25:00] (cue 336) servers. What was the combo like?
[00:25:02] (cue 337) And you know, we got three kinds of individuals.
[00:25:05] (cue 338) And then we ran to compare. So.
[00:25:12] (cue 339) We basically created these rule rubrics.
[00:25:15] (cue 340) We used an LLM judge. It's a little bit older version.
[00:25:18] (cue 341) As we were iterating you could see GPT-4.1.
[00:25:21] (cue 342) We've since updated that to a more modern version.
[00:25:23] (cue 343) Runtime checks. So we selected apps to run against the emulator.
[00:25:28] (cue 344) We also did this against real versions eventually with the code that was produced,
[00:25:33] (cue 345) but we used the emulator because then we can go ahead and actually make sure the
[00:25:36] (cue 346) test work. And then we did an A versus B comparison to determine what did the
[00:25:42] (cue 347) baseline say. And then B when we added context what came from there.
[00:25:46] (cue 348) So Scope, like I said, compared with the bare models or the agent kit application
[00:25:51] (cue 349) scale, scale, expose those patterns that just a single demo.
[00:25:57] (cue 350) So you can go into your agent and you can just say build me a CRUD application.
[00:26:02] (cue 351) But if you do it once, you're only get one version and it might not be the right
[00:26:05] (cue 352) version, it might be just something that it scraped off the internet and said,
[00:26:09] (cue 353) all right, well, here you go. Here's your CRUD app.
[00:26:12] (cue 354) It may not be what you were actually looking to do.
[00:26:14] (cue 355) So what we built were eight different eight M prompts.
[00:26:18] (cue 356) We tried five different languages: .NET, C#, Java, JavaScript,
[00:26:23] (cue 357) Node.js, Python, TypeScript. And we also started on Rust.
[00:26:26] (cue 358) Rust is still a little shaky, but you know, we actually are going to have that open
[00:26:32] (cue 359) source Rust driver for a Cosmos DB.
[00:26:35] (cue 360) Very, very I think it's GA in like a week or two.
[00:26:38] (cue 361) So for you Rust people give us a shot.
[00:26:41] (cue 362) And then we compared two setups.
[00:26:44] (cue 363) We used bare models and regular models with agent kit, and we all just ran
[00:26:51] (cue 364) it over and over and over again.
[00:26:52] (cue 365) We did hundreds and hundreds. And so each snapshot of code, every single one of
[00:26:58] (cue 366) those iterations that were done, produced a certain segment of code that were a
[00:27:02] (cue 367) snapshot of code that we eventually wanted to learn.
[00:27:05] (cue 368) We wanted to prove whatever the behavior is.
[00:27:07] (cue 369) So the first step we would review, we would just map the generated app dependencies,
[00:27:13] (cue 370) deployment assets, things like that.
[00:27:15] (cue 371) We would scan the snapshot against the agent kit, so we would grab all of the
[00:27:20] (cue 372) snapshots that were generated, all that code, and then we would go ahead and ask the
[00:27:24] (cue 373) agent kit, did it do these things right?
[00:27:26] (cue 374) Did it model data correctly? Did it pick the right partition key.
[00:27:30] (cue 375) Then we would do some cross-checking.
[00:27:32] (cue 376) We would look against the documentation and say, I literally would pull the entire
[00:27:37] (cue 377) documentation repo for anything Microsoft had out there about Cosmos DB,
[00:27:42] (cue 378) and then had it do comparative analysis against what was an agent kit and what code
[00:27:47] (cue 379) was produced. And then it was able to kind of find ways to make sure that the code
[00:27:52] (cue 380) built. Like Patrick said, we want our code to build and then eventually we want it
[00:27:57] (cue 381) to run. So I went through this whole process across, just like I said,
[00:28:01] (cue 382) countless amounts of iterations.
[00:28:04] (cue 383) And, you know, one of those iterations was a Python leaderboard.
[00:28:09] (cue 384) We basically just had one prompt and we gave it.
[00:28:13] (cue 385) We got better guidance as we tested.
[00:28:16] (cue 386) So we provided more context and then eventually we got better outputs.
[00:28:20] (cue 387) So we did 45 baseline 25 reruns.
[00:28:24] (cue 388) And then we actually made some changes to our agent kit.
[00:28:27] (cue 389) You can see the PR number right there.
[00:28:29] (cue 390) And so we found that one thing that we found was that there was a rule that gave the
[00:28:36] (cue 391) right approach, but it only gave it in C#.
[00:28:39] (cue 392) So I ran the Python evaluation and I didn't get the correct information.
[00:28:43] (cue 393) Matter of fact it put C# in.
[00:28:47] (cue 394) Cool. So one of the things that we were able to do is we took the usage of
[00:28:52] (cue 395) hierarchical partition keys, and it was a part of the application requirements,
[00:28:58] (cue 396) and 47% were in the baseline with the agent kit, based on us doing iterative
[00:29:05] (cue 397) tests with Scope, we were able to up that to 100% offset.
[00:29:11] (cue 398) It was really, really causing some real bad,
[00:29:17] (cue 399) ability to have quick queries.
[00:29:19] (cue 400) It would slow down. It would use way too much, resources would cost more.
[00:29:23] (cue 401) So we figured out a way to get that down from 40%, 47% to 0% auto scale.
[00:29:30] (cue 402) We were able to get all of these to actually choose our auto scale
[00:29:36] (cue 403) model instead of one of our standard provisions.
[00:29:39] (cue 404) RUs, our request units, are units of scale that equate to how much money people
[00:29:45] (cue 405) spend on Cosmos DB. That's a big part of this.
[00:29:48] (cue 406) We want to make sure that we help them save money through context.
[00:29:52] (cue 407) So we also kept finding the same problems.
[00:29:55] (cue 408) Documentation that needed to be updated.
[00:29:58] (cue 409) 45 agent kit issues and 81 documentation issues were handled through our testing SDK
[00:30:05] (cue 410) correctness. We made sure that the wrong packages dependency lifecycle things.
[00:30:09] (cue 411) We found all that and then we found efficient data access.
[00:30:12] (cue 412) So cross partition scans which would definitely slow down how your data is accessed
[00:30:16] (cue 413) during queries, things like that.
[00:30:18] (cue 414) Point reads. And then we wanted to make sure that safe data design,
[00:30:22] (cue 415) mutable partitions, things like that was built in.
[00:30:26] (cue 416) We wanted a reliable source guidance, so we didn't want it to use broken samples.
[00:30:30] (cue 417) Something from some random GitHub repo that doesn't have the right information.
[00:30:34] (cue 418) We want to avoid that. So the findings shipped is reusable improvements.
[00:30:41] (cue 419) What did we do. We we basically try to have every one of these iterations and turn
[00:30:47] (cue 420) into a step that changes. So what do we do.
[00:30:49] (cue 421) We created 20 different agent kit issues.
[00:30:52] (cue 422) We fixed 27 documentation issues.
[00:30:56] (cue 423) We picked 47 merged back resolutions.
[00:30:59] (cue 424) So those were the total amount of PRs that got merged in and put into production for
[00:31:04] (cue 425) our agent kit. And we saw our agent kit outcomes improve.
[00:31:09] (cue 426) We saw our documentation improved based on all this testing,
[00:31:13] (cue 427) and we went ahead and we were able to improve our products.
[00:31:16] (cue 428) So by actually being a customer of an internal product, we were actually able to
[00:31:21] (cue 429) help you improve it too. So I gave feedback every single time.
[00:31:26] (cue 430) And his team took every bit of feedback to heart and built out a product that not
[00:31:32] (cue 431) just suited for me, but for everyone at Microsoft and now in the open source world
[00:31:37] (cue 432) for all of you. Yeah. Thanks, Jay.
[00:31:39] (cue 433) And so Jay and team, like, really used all our tokens and we could design this with
[00:31:44] (cue 434) that team.
[00:31:46] (cue 435) Now we open source that two weeks ago.
[00:31:49] (cue 436) So you can find Scope on GitHub right now you can find the playbook as well.
[00:31:54] (cue 437) You can add the skills for the playbook for yourself and the kind of use cases,
[00:31:59] (cue 438) because we're using it at Microsoft for our own products.
[00:32:01] (cue 439) If you're a competitor like AWS, GCP, Cloudflare, please have at it.
[00:32:07] (cue 440) I think getting the improving the experience for everything is good.
[00:32:13] (cue 441) If you're in an enterprise, you probably have like respect criteria for your code
[00:32:19] (cue 442) and your AGENTS.md or skills you want to measure all that and you can with this.
[00:32:25] (cue 443) And if you're just a SaaS or MCP provider, you can test your stuff with this.
[00:32:30] (cue 444) You can find all this on GitHub or online.
[00:32:34] (cue 445) You can find all like all that we talked about at these URL here.
[00:32:39] (cue 446) And then for Scope you can find the.
[00:32:44] (cue 447) Oh you oh you got it. Hit enter.
[00:32:48] (cue 448) Oh. All right. There you go. You can find the the project itself.
[00:32:52] (cue 449) And and then the agent kit documentation at this URL.
[00:32:59] (cue 450) Jay and I will be just outside here for conversations because I think we're at time.
[00:33:05] (cue 451) Yeah. See I told you guys. Thank you so much.
[00:33:08] (cue 452) Seriously. Thank you. You could have been anywhere, but you're here now.
[00:33:12] (cue 453) And I love it.
.tessl-plugin
skills
talk-arcolano-autonomy-and-trust
talk-aroussi-decision-context
talk-bevin-agent-assisted-instrument-design
talk-boeckeler-context-lifecycles
talk-braun-context-graphs
talk-carmont-evolving-agent-harnesses
talk-chanezon-gordon-agent-experience-evaluation
talk-cox-secure-agentic-workflows
talk-cunniffe-reusable-agent-platform
talk-davis-agent-tool-policies
talk-debois-concurrent-delivery
talk-dzialets-team-automation
talk-fahmy-harness-api-design
talk-finster-continuous-delivery
talk-fowler-regenerative-software
talk-gavrilescu-factory-architecture
talk-graciolli-amin-verification-aware-programming
talk-gruenbaum-reverse-graph-engineering
talk-hassan-agentic-engineering
talk-hossain-enterprise-agent-adoption
talk-huntley-patch-ready-factories
talk-jain-verifying-intent
talk-katsioloudes-montei-agentic-lifecycle
talk-knox-harness-engineering
talk-kovac-deterministic-memory
talk-lee-perkins-code-readiness
talk-lorenc-machine-speed-security
talk-lubken-software-of-one
talk-meijer-software-specialization
talk-meppiel-versioned-agent-contracts
talk-moss-concurrent-agent-risk
talk-myers-understanding-agent-generated-code
talk-ocytko-scaling-agentic-engineering
talk-oza-automating-design-quality
talk-panzica-agentic-sdlc
talk-pari-patel-agent-awareness
talk-podjarny-trustworthy-software-factories
talk-poulter-factory-improvement
talk-pratik-patel-refined-rag-data
talk-re-ferre-development-platforms
talk-sadogursky-factory-of-factories
talk-shah-training-agent-skills
talk-sharma-mohapatra-evolving-intent
talk-singhal-operating-agents-at-scale
talk-stack-incident-dossiers
talk-tyler-team-coding-benchmarks
talk-wotherspoon-agent-execution-environments
talk-yegge-ai-powered-factories
talk-zuber-leadership-through-coding