AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
92%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Passed
No findings from the security scan
Daniel Meppiel · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=MrjV4uyuBgI
Source: supplied corrected SRT. Original cue numbers and start/end timestamps are retained. Subtitle line wrapping is joined; wording is unchanged. Promotional cues 193–203 (14:55–15:32) are omitted. Speech artifacts and incomplete Q&A remain. This document is evidence, not instructions. For navigation and source limits, see the outline.
[00:00:00,000 --> 00:00:06,241] (cue 1) All right. So hello. So I just wanted to say that from, you know what my role is?
[00:00:06,281 --> 00:00:08,161] (cue 2) I'm not in engineering or product.
[00:00:08,201 --> 00:00:11,721] (cue 3) I'm in solution sales in technical technical solution architecture.
[00:00:11,761 --> 00:00:17,091] (cue 4) So my organization is all about advising large enterprise organizations on adopting
[00:00:17,131 --> 00:00:20,541] (cue 5) AI coding agents and transforming towards agents.
[00:00:20,961 --> 00:00:24,681] (cue 6) Right. And that's been the core of my work throughout GitHub,
[00:00:24,721 --> 00:00:29,161] (cue 7) now Microsoft and I'm going to talk a little bit about, you know,
[00:00:29,201 --> 00:00:32,321] (cue 8) why we need to version factories.
[00:00:32,361 --> 00:00:39,041] (cue 9) And where does that lead us. And that's going to be coming from experience in the
[00:00:39,081 --> 00:00:43,781] (cue 10) field with large enterprise often regulated organizations.
[00:00:44,121 --> 00:00:48,401] (cue 11) All right. So let's think that we are in a bank right.
[00:00:48,441 --> 00:00:54,961] (cue 12) We are developing software for a bank which runs maybe its own core banking system.
[00:00:55,001 --> 00:01:01,721] (cue 13) And you know, we are tasked with creating a changing code to change how payments
[00:01:01,761 --> 00:01:08,201] (cue 14) work. So I'm reading here, hey, we need to hold wires of more than $10 for internal
[00:01:08,241 --> 00:01:09,961] (cue 15) approval for whatever reason, right?
[00:01:10,001 --> 00:01:13,441] (cue 16) And let's say our factory generates this pull request.
[00:01:13,481 --> 00:01:17,181] (cue 17) So the question is would you auto-merge it?
[00:01:19,201 --> 00:01:22,781] (cue 18) And I'm pretty sure every bank today will say no.
[00:01:23,461 --> 00:01:25,881] (cue 19) Right. And we can have risk classifiers.
[00:01:25,921 --> 00:01:28,881] (cue 20) We can have verifiers. We can have a lot of these things.
[00:01:28,921 --> 00:01:34,281] (cue 21) But the answer to that question will very likely be no for some time.
[00:01:34,321 --> 00:01:40,561] (cue 22) And what I'm going to focus on is what do we need so that these can be automated one
[00:01:40,601 --> 00:01:44,721] (cue 23) way, one day confidently. All right.
[00:01:44,761 --> 00:01:50,001] (cue 24) So recapping, I think by now we all know what an agent software factory ideal or
[00:01:50,041 --> 00:01:55,641] (cue 25) concept is. We get the intent in, our agentic software factory runs an automated
[00:01:55,681 --> 00:02:00,741] (cue 26) agent process to basically transform this into code.
[00:02:00,881 --> 00:02:06,561] (cue 27) Code changes. Now the problem with that is that we create new bottlenecks.
[00:02:06,601 --> 00:02:09,781] (cue 28) And this has been repeated over and over and over yesterday.
[00:02:09,921 --> 00:02:11,961] (cue 29) One of those bottlenecks is CI.
[00:02:12,001 --> 00:02:16,721] (cue 30) We get so many requests, CI runs everything again and this becomes a bottleneck.
[00:02:16,761 --> 00:02:19,041] (cue 31) And the other one is us humans.
[00:02:19,081 --> 00:02:22,361] (cue 32) And we just heard in the keynote that we need to remove all humans.
[00:02:22,401 --> 00:02:28,121] (cue 33) Right. And if we don't do this agentic software factories cannot scale in large
[00:02:28,161 --> 00:02:33,121] (cue 34) organizations either. Right. So that's another piece of the problem that we need to
[00:02:33,161 --> 00:02:38,721] (cue 35) solve today. We seem to be converging on the idea of a risk classifier.
[00:02:38,761 --> 00:02:45,241] (cue 36) Right. Which is basically hey, we identify what the code change is changing and then
[00:02:45,281 --> 00:02:51,001] (cue 37) route it to a human only when the risk is, you know, justifying that.
[00:02:51,041 --> 00:02:57,321] (cue 38) But this is, as I said before, this is not enough for large regulated enterprise
[00:02:57,361 --> 00:03:02,661] (cue 39) organizations, right? Because even when we auto-merge in this scenario,
[00:03:02,761 --> 00:03:08,281] (cue 40) then there are of questions about which agentic software factory created that
[00:03:08,321 --> 00:03:14,581] (cue 41) change. And do we trust it? There is no trust yet, just based on a risk classifier.
[00:03:15,721 --> 00:03:19,921] (cue 42) Every single time I talk with large enterprises going through this journey,
[00:03:19,961 --> 00:03:24,881] (cue 43) there are four elements that come up in conversations all the time about what's
[00:03:24,921 --> 00:03:28,001] (cue 44) needed. The first is verification.
[00:03:28,041 --> 00:03:34,321] (cue 45) How can we be sure that what the software factory produced as code meets the intent
[00:03:34,361 --> 00:03:40,921] (cue 46) we have? And that's it? Well, the second one is about, hey, did that code
[00:03:40,961 --> 00:03:46,081] (cue 47) change? Was that code change authored following our standards,
[00:03:46,121 --> 00:03:48,161] (cue 48) our platform engineering standards?
[00:03:48,201 --> 00:03:54,921] (cue 49) Our compliance frameworks? Did it did it follow our coding styles and so on?
[00:03:54,961 --> 00:03:58,261] (cue 50) Right. Which are often already in place.
[00:03:58,801 --> 00:04:00,961] (cue 51) The third one is traceability.
[00:04:01,001 --> 00:04:06,161] (cue 52) Like, you know, if I have a code change that I have merged that came from a software
[00:04:06,201 --> 00:04:10,681] (cue 53) factory that used, I don't know how many skills, how can I trace that back if
[00:04:10,721 --> 00:04:14,601] (cue 54) something goes wrong in production, how can I trace that back to the skill that I
[00:04:14,641 --> 00:04:20,841] (cue 55) need to improve? Or the factory that was compromised and and the last one is
[00:04:20,881 --> 00:04:25,641] (cue 56) security, and it's a little bit related to the, to the rest,
[00:04:25,681 --> 00:04:30,861] (cue 57) which is, you know, agentic software factories become a new surface of attack,
[00:04:30,961 --> 00:04:34,841] (cue 58) and it becomes a very lucrative and a very interesting surface of attack,
[00:04:34,881 --> 00:04:38,881] (cue 59) because now these software factories are going to generate all the code,
[00:04:38,921 --> 00:04:41,981] (cue 60) right? They will remove all humans from coding.
[00:04:42,561 --> 00:04:45,741] (cue 61) So the security aspect is super important.
[00:04:46,441 --> 00:04:52,321] (cue 62) Now I want us to think about how we build trust for classic software before the AI
[00:04:52,361 --> 00:04:58,051] (cue 63) era. And how did we build that, technically speaking, right.
[00:04:58,481 --> 00:05:02,161] (cue 64) To me, software runs on a fabric of trust.
[00:05:02,201 --> 00:05:04,561] (cue 65) Kind of, if I can use that metaphor.
[00:05:04,601 --> 00:05:09,081] (cue 66) There are several elements that brought us to where we are today,
[00:05:09,121 --> 00:05:13,961] (cue 67) where software runs every a lot of mission critical applications.
[00:05:14,001 --> 00:05:16,121] (cue 68) It all starts with versioning.
[00:05:16,161 --> 00:05:20,881] (cue 69) It starts with versioning because versioning is about giving an identity to software
[00:05:20,921 --> 00:05:23,801] (cue 70) artifacts. The fingerprint. Sure.
[00:05:23,841 --> 00:05:26,961] (cue 71) We see we see version 1.0, 2.0.
[00:05:27,001 --> 00:05:31,541] (cue 72) But behind that we have a digest and we have a fingerprint.
[00:05:31,881 --> 00:05:38,641] (cue 73) Once we have that, we can start creating the rest of the elements that compose this
[00:05:38,681 --> 00:05:43,641] (cue 74) fabric of trust like inventories, provenance, the evidence like.
[00:05:43,681 --> 00:05:49,881] (cue 75) And we have created a lot of standards so far to standardize this and speak the same
[00:05:49,921 --> 00:05:54,721] (cue 76) language. When we check, what should we run and what do we trust?
[00:05:54,761 --> 00:05:57,801] (cue 77) What software do we trust to be run in our environments?
[00:05:57,841 --> 00:06:02,621] (cue 78) Right? Like SLSA, CycloneDX, in-toto, and so on.
[00:06:03,801 --> 00:06:08,661] (cue 79) We invented signatures. And finally, ultimately with all of the above,
[00:06:08,761 --> 00:06:15,441] (cue 80) we could create admission. Admission checks, basically a policy that's unique to
[00:06:15,481 --> 00:06:19,801] (cue 81) each enterprise, which says, all right, I'm going to check the inventory of the bill
[00:06:19,841 --> 00:06:21,481] (cue 82) of materials of this software.
[00:06:21,521 --> 00:06:25,641] (cue 83) I'm going to check which organization created it and signed it.
[00:06:25,681 --> 00:06:30,801] (cue 84) Right. And based on all those things, I'm going to validate that this software can
[00:06:30,841 --> 00:06:37,168] (cue 85) run in my bank. All right. And so the question is when an agentic software factory
[00:06:37,208 --> 00:06:40,481] (cue 86) has generated software, where do we leave all of this.
[00:06:40,521 --> 00:06:44,041] (cue 87) What is the fabric of trust. Right.
[00:06:44,081 --> 00:06:47,821] (cue 88) I don't think we have a lot about that today yet.
[00:06:49,281 --> 00:06:53,761] (cue 89) So we need to build that and to build that.
[00:06:53,801 --> 00:06:59,761] (cue 90) I like to think about a factory as an agent program as software.
[00:06:59,801 --> 00:07:06,921] (cue 91) Right. Because we saw yesterday, for example, how Tessl, with the loops concept.
[00:07:06,961 --> 00:07:11,641] (cue 92) And some other vendors like ourselves with GitHub automations,
[00:07:11,681 --> 00:07:16,241] (cue 93) we call them automations. At the end of the day, it's a bunch of skills,
[00:07:16,281 --> 00:07:21,401] (cue 94) a bunch of prompts running a process that we have written in natural language.
[00:07:21,441 --> 00:07:26,781] (cue 95) Right? And a factory is a collection of those that are chained together.
[00:07:27,041 --> 00:07:31,261] (cue 96) That looks to me like a program, like a software program, chaining functions.
[00:07:31,801 --> 00:07:36,761] (cue 97) And so if it's a program, we should version it because it's the first thread of the
[00:07:36,801 --> 00:07:37,981] (cue 98) trust fabric.
[00:07:39,961 --> 00:07:45,441] (cue 99) The other problem we have to solve today is that in the proto factories we're
[00:07:45,481 --> 00:07:49,801] (cue 100) starting to create, we are gluing together a lot of loose parts.
[00:07:49,841 --> 00:07:56,401] (cue 101) Right. We saw skills, prompts, scripts, maybe some proxies and elements on the
[00:07:56,441 --> 00:08:01,081] (cue 102) networking layer, on the sandbox layer, environment layer, infrastructure layer.
[00:08:01,121 --> 00:08:03,421] (cue 103) It's a lot of loose parts right.
[00:08:04,041 --> 00:08:08,721] (cue 104) So we have all those loose parts with which we can compose a factory and build that
[00:08:08,761 --> 00:08:14,241] (cue 105) ourselves. And the other alternative is just go with a vendor.
[00:08:14,281 --> 00:08:18,841] (cue 106) Right. Which are starting now to propose a way to create factories.
[00:08:18,881 --> 00:08:21,144] (cue 107) Right. In the case of GitHub it's automations.
[00:08:21,184 --> 00:08:23,464] (cue 108) In the case of Tessl, it's loops.
[00:08:23,504 --> 00:08:27,481] (cue 109) In the case of Codex, no, of Cursor, it's also automations, right.
[00:08:27,521 --> 00:08:30,161] (cue 110) But that's getting us locked in right.
[00:08:30,201 --> 00:08:34,481] (cue 111) It's a nice UI, UX and so on. But once we build our factory there,
[00:08:34,521 --> 00:08:38,251] (cue 112) it's very hard to take it somewhere else and run it in another harness one day.
[00:08:38,291 --> 00:08:45,201] (cue 113) Right. So what's missing in between, in my opinion, is a single unique
[00:08:45,241 --> 00:08:52,081] (cue 114) way. Standard way to define the smallest element that defines a unit of work,
[00:08:52,121 --> 00:08:53,741] (cue 115) of agent work.
[00:08:55,161 --> 00:08:56,781] (cue 116) What do I mean with this?
[00:08:58,561 --> 00:09:03,281] (cue 117) This metaphor, I think, is going to make us converge.
[00:09:03,321 --> 00:09:06,561] (cue 118) So I call it agent contracts, right?
[00:09:06,601 --> 00:09:08,641] (cue 119) I don't call it automations. I don't call it loops.
[00:09:08,681 --> 00:09:14,561] (cue 120) I call it agent contracts in the same way that when we sign up for a job with a
[00:09:14,601 --> 00:09:21,521] (cue 121) company, we sign a contract. I do think that what can help us standardize and make
[00:09:21,561 --> 00:09:26,341] (cue 122) software factories portable are agent contracts.
[00:09:26,401 --> 00:09:33,121] (cue 123) As a simple markdown file where we define the assignment or we admit we basically
[00:09:33,161 --> 00:09:36,801] (cue 124) has an input external input the assignment.
[00:09:36,841 --> 00:09:43,541] (cue 125) The assignment can change when they when they the agent can be asked to perform one
[00:09:43,641 --> 00:09:47,161] (cue 126) one software change in one piece of software the next day another one.
[00:09:47,201 --> 00:09:53,281] (cue 127) But what doesn't change is the role, the role in which the agent signs itself for in
[00:09:53,321 --> 00:09:58,361] (cue 128) this contract, the dependencies, what's made available for the agent to perform this
[00:09:58,401 --> 00:10:04,161] (cue 129) job, and the budget. Like how much are we happy to pay for right?
[00:10:04,201 --> 00:10:09,881] (cue 130) For this job to get done? It resembles a lot the kind of work contracts that we sign
[00:10:09,921 --> 00:10:15,161] (cue 131) in real life, right? And what we get back are the results of the checks,
[00:10:15,201 --> 00:10:21,897] (cue 132) the verification and the deliverables, like software changes or change diffs or
[00:10:21,937 --> 00:10:28,881] (cue 133) pull requests. Right. So once we are able to define
[00:10:28,921 --> 00:10:33,881] (cue 134) agent contracts in a simple markdown file with front matter YAML or,
[00:10:33,921 --> 00:10:38,301] (cue 135) you know, that's a possible format for it, then we can version it.
[00:10:39,321 --> 00:10:44,121] (cue 136) This contract can link the skills and everything that's required for the agent to do
[00:10:44,161 --> 00:10:49,701] (cue 137) the job, and it becomes a program an agentic program.
[00:10:50,201 --> 00:10:54,281] (cue 138) It becomes a program that is versioned, that allows composition.
[00:10:54,321 --> 00:10:57,601] (cue 139) We can start composing contracts between themselves.
[00:10:57,641 --> 00:11:03,041] (cue 140) It allows portability if more harnesses and vendors accept it,
[00:11:03,081 --> 00:11:08,501] (cue 141) and it's identifiable, and it allows us to start building the fabric of trust.
[00:11:08,681 --> 00:11:13,921] (cue 142) All right. So a factory can be thought of as a graph of contracts.
[00:11:13,961 --> 00:11:18,881] (cue 143) Like if we have an agent contract, that's for planning another one for a spec,
[00:11:18,921 --> 00:11:22,281] (cue 144) then another one for building and implementing the actual code,
[00:11:22,321 --> 00:11:24,581] (cue 145) for documentation, for reviewing.
[00:11:24,801 --> 00:11:30,061] (cue 146) For a lot of different tasks, we chain them together and we have our factory.
[00:11:30,441 --> 00:11:34,361] (cue 147) We give it a version, we get the fingerprint and we have the identity of the
[00:11:34,401 --> 00:11:36,341] (cue 148) factory. Right.
[00:11:37,641 --> 00:11:42,741] (cue 149) And so if factories are programs, then once we have that definition,
[00:11:42,801 --> 00:11:49,601] (cue 150) what runs it. Right. And so I want to get my I want to get it to
[00:11:49,641 --> 00:11:56,701] (cue 151) run right now. So let's let's imagine we have basically the
[00:11:57,041 --> 00:11:59,921] (cue 152) the pull request example that I showed at the beginning.
[00:11:59,961 --> 00:12:05,881] (cue 153) The intent here defined in markdown hold large wires to new right.
[00:12:05,921 --> 00:12:11,761] (cue 154) This is intent. It can come in the form of a of an issue of a markdown file,
[00:12:11,801 --> 00:12:14,761] (cue 155) or a message from a colleague on Slack or whatever it is.
[00:12:14,801 --> 00:12:18,721] (cue 156) Right. I have defined here, I have defined it here in a markdown file.
[00:12:18,761 --> 00:12:25,561] (cue 157) So here we have that right. And this is our core banking software
[00:12:25,601 --> 00:12:29,281] (cue 158) Contoso Payments whatever. Okay great.
[00:12:29,321 --> 00:12:33,641] (cue 159) And then we have our factory. This is our these are all public repositories.
[00:12:33,681 --> 00:12:35,761] (cue 160) You will get access to that later.
[00:12:35,801 --> 00:12:40,381] (cue 161) And so in our factory we have a contract folder.
[00:12:40,641 --> 00:12:45,361] (cue 162) And this contract folders we have a bunch of contracts like from the graph that
[00:12:45,401 --> 00:12:52,201] (cue 163) you've seen before. And for example the build contract which specifies hey as input
[00:12:52,241 --> 00:12:57,561] (cue 164) I need the intent, the request, then the specification from the previous contract I
[00:12:57,601 --> 00:13:03,001] (cue 165) will produce a diff and I will import this specific skill.
[00:13:03,041 --> 00:13:05,361] (cue 166) Right. And I have this budget and so on.
[00:13:05,401 --> 00:13:10,561] (cue 167) And this is the the role the role of the agent contract.
[00:13:10,601 --> 00:13:15,241] (cue 168) All right. So those are those are our program files.
[00:13:15,281 --> 00:13:21,761] (cue 169) And then we have this APM YAML file which can be thought of as a package.json file
[00:13:21,801 --> 00:13:24,881] (cue 170) in the npm world. This is from agent package manager.
[00:13:24,921 --> 00:13:26,681] (cue 171) And it's defining our dependencies.
[00:13:26,721 --> 00:13:31,181] (cue 172) Like what are the skills that are required for me to run this factory?
[00:13:31,241 --> 00:13:36,721] (cue 173) And once we have this manifest here defining the dependencies and they get
[00:13:36,761 --> 00:13:43,481] (cue 174) installed, then our contracts can link them, import them as modules as we do in a in
[00:13:43,521 --> 00:13:46,241] (cue 175) a usual programming languages.
[00:13:46,281 --> 00:13:52,841] (cue 176) Right. And so with that I'll use apmx.
[00:13:52,881 --> 00:13:56,961] (cue 177) And you can see the analogy here to the npm world or UV or whatever.
[00:13:57,001 --> 00:14:02,881] (cue 178) Right. We have npm, npx, we have uv, uvx, you can see the analogies and I will use
[00:14:02,921 --> 00:14:06,961] (cue 179) that to run the factory and I will run it.
[00:14:07,001 --> 00:14:09,601] (cue 180) And I hope you can see that in two terminals.
[00:14:09,641 --> 00:14:14,241] (cue 181) One is going to basically run the factory on copilot as a harness.
[00:14:14,281 --> 00:14:16,221] (cue 182) I'm going to launch that right now.
[00:14:16,481 --> 00:14:19,861] (cue 183) And on the right hand side it's going to run it on OpenCode.
[00:14:20,361 --> 00:14:24,901] (cue 184) All right. And you can see that the first thing that's happening is that agent
[00:14:24,941 --> 00:14:29,241] (cue 185) package manager, which is akin to npm, is reading the manifest,
[00:14:29,281 --> 00:14:31,841] (cue 186) installing the dependencies, locking them in a lock file.
[00:14:31,881 --> 00:14:34,081] (cue 187) And then the factory can run as a program.
[00:14:34,121 --> 00:14:37,321] (cue 188) And we can see here the chaining of the contracts.
[00:14:37,361 --> 00:14:42,121] (cue 189) The first one is called planning being run on the harness of choice.
[00:14:42,161 --> 00:14:48,801] (cue 190) So these are the of how we can have a portable open format that can eventually run
[00:14:48,841 --> 00:14:52,641] (cue 191) on any harness. Right. I will leave that running.
[00:14:52,681 --> 00:14:53,481] (cue 192) Okay.
[00:15:36,360 --> 00:15:39,240] (cue 204) And let's come back to, to the presentation.
[00:15:39,280 --> 00:15:43,920] (cue 205) Okay. So APM I have introduced that briefly.
[00:15:43,960 --> 00:15:45,840] (cue 206) It's a package manager for agents.
[00:15:45,880 --> 00:15:50,000] (cue 207) It's an open source project. I started that independently.
[00:15:50,040 --> 00:15:55,320] (cue 208) And then as I am employed by Microsoft, I transfer it to the Microsoft Open Source
[00:15:55,360 --> 00:16:00,720] (cue 209) organization. All right. The intent here is to move it to a neutral foundation
[00:16:00,760 --> 00:16:03,520] (cue 210) working on that. And it is pretty simple.
[00:16:03,560 --> 00:16:06,040] (cue 211) We have a manifest with the dependencies.
[00:16:06,080 --> 00:16:09,280] (cue 212) We have a lock file. That's when we install.
[00:16:09,320 --> 00:16:14,240] (cue 213) And then we can create a policy in the enterprise which says what are the
[00:16:14,280 --> 00:16:18,240] (cue 214) dependencies we want to allow and what are the dependencies that we want to deny.
[00:16:18,280 --> 00:16:20,580] (cue 215) And many other cool things. All right.
[00:16:20,840 --> 00:16:26,760] (cue 216) And so APM is about running these modules as programs.
[00:16:26,800 --> 00:16:28,360] (cue 217) They contracts as programs. Right.
[00:16:28,400 --> 00:16:32,800] (cue 218) Which depend on skills, MCPs, hooks and whatever it is.
[00:16:32,840 --> 00:16:36,120] (cue 219) Right. So we can run it on any coding agent.
[00:16:36,160 --> 00:16:39,760] (cue 220) And as a result we are going to get our pull request, our diff.
[00:16:39,800 --> 00:16:41,876] (cue 221) But we are also going to get a receipt.
[00:16:41,916 --> 00:16:44,200] (cue 222) We are going to get a receipt.
[00:16:44,240 --> 00:16:47,280] (cue 223) And that's the second important element here.
[00:16:47,320 --> 00:16:51,700] (cue 224) One is the contracts and the other is the receipt of the run.
[00:16:52,920 --> 00:16:54,680] (cue 225) And so why is the receipt important.
[00:16:54,720 --> 00:16:57,760] (cue 226) All right let's go back to to how this works.
[00:16:57,800 --> 00:17:02,800] (cue 227) We get intent in. We have the factory as a as a chain of contracts or loops or
[00:17:02,840 --> 00:17:04,520] (cue 228) automations or whatever it is.
[00:17:04,560 --> 00:17:08,748] (cue 229) Right. It generates the diff, of course, but also the receipt.
[00:17:08,788 --> 00:17:15,560] (cue 230) And the receipt is signed. The receipt is signed by whichever machine which run our
[00:17:15,600 --> 00:17:20,880] (cue 231) factory, which we want, we choose to trust as the runner of our factory.
[00:17:20,920 --> 00:17:24,920] (cue 232) All right then we have our rules, our policies which define,
[00:17:24,960 --> 00:17:26,840] (cue 233) hey, which factories do we trust?
[00:17:26,880 --> 00:17:30,040] (cue 234) Which skills do we allow? In which ones do we deny?
[00:17:30,080 --> 00:17:32,280] (cue 235) Oh, we heard about that skill that was compromised.
[00:17:32,320 --> 00:17:33,920] (cue 236) We're going to deny it there, right?
[00:17:33,960 --> 00:17:37,480] (cue 237) It's our set of rules. And then we have a gate.
[00:17:37,520 --> 00:17:42,000] (cue 238) And that's typically implemented in CI/CD in the pull request targeting main.
[00:17:42,040 --> 00:17:45,640] (cue 239) Right. Those are the checks that can run in CI/CD.
[00:17:45,680 --> 00:17:51,032] (cue 240) And I've chosen to implement that as a demonstrator with a command,
[00:17:51,072 --> 00:17:57,920] (cue 241) apmx audit. It's just going to read the receipt read the policy check whether there
[00:17:57,960 --> 00:18:03,600] (cue 242) is you know, there is alignment between those and block the the pull request or auto
[00:18:03,640 --> 00:18:04,460] (cue 243) merge it.
[00:18:06,000 --> 00:18:08,140] (cue 244) All right. So let's look at the receipt.
[00:18:08,280 --> 00:18:14,600] (cue 245) The receipt is based on open standards that we all know about in the software supply
[00:18:14,640 --> 00:18:20,480] (cue 246) chain world. So if we look at the first block that's based on a provenance and it
[00:18:20,520 --> 00:18:25,840] (cue 247) gives us traceability, it allows us to understand that the output,
[00:18:25,880 --> 00:18:32,611] (cue 248) the diff that was generated comes from this factory because they are chained
[00:18:32,651 --> 00:18:36,120] (cue 249) through their fingerprints, through the digests.
[00:18:36,160 --> 00:18:41,640] (cue 250) Right. The versions. Right. Then we have the results of the checks,
[00:18:41,680 --> 00:18:45,880] (cue 251) the checks that have been defined in each contract and are immutable by the agents.
[00:18:45,920 --> 00:18:47,960] (cue 252) The agents cannot change those checks.
[00:18:48,000 --> 00:18:51,360] (cue 253) Remember that agents cannot be trusted, right?
[00:18:51,400 --> 00:18:53,520] (cue 254) They can change the rules of the game.
[00:18:53,560 --> 00:18:59,320] (cue 255) So we need to also cryptographically chain the results of the checks and the checks
[00:18:59,360 --> 00:19:01,320] (cue 256) themselves in the respect and the evidence.
[00:19:01,360 --> 00:19:03,500] (cue 257) All right. And that's based on in-toto.
[00:19:04,000 --> 00:19:06,640] (cue 258) And then we have our agent bill of materials.
[00:19:06,680 --> 00:19:08,680] (cue 259) It's akin to the software bill of materials.
[00:19:08,720 --> 00:19:11,520] (cue 260) It's about what's the composition of the factory.
[00:19:11,560 --> 00:19:15,320] (cue 261) Right. And finally we sign it.
[00:19:15,360 --> 00:19:20,640] (cue 262) We sign it with a trusted signer that cannot be the same machine that run the
[00:19:20,680 --> 00:19:23,000] (cue 263) factory because we cannot trust the agents.
[00:19:23,040 --> 00:19:25,200] (cue 264) Right. So it has to be something else.
[00:19:25,240 --> 00:19:28,060] (cue 265) And that gives us security, right?
[00:19:28,760 --> 00:19:34,040] (cue 266) And ultimately, it's this receipt which allows us to build that fabric of trust.
[00:19:34,080 --> 00:19:38,280] (cue 267) Right. So we get the identity as the factory version of fingerprint.
[00:19:38,320 --> 00:19:43,160] (cue 268) We get the inventory as the agent bill of materials out of the APM lock file.
[00:19:43,200 --> 00:19:49,089] (cue 269) Right. We get the provenance with a we get the evidence with in-toto,
[00:19:49,129 --> 00:19:53,150] (cue 270) thanks to the check results and the checks themselves chained to the receipt,
[00:19:53,190 --> 00:19:59,320] (cue 271) chained to the output. And we get, you know, we get attested provenance with the
[00:19:59,360 --> 00:20:06,160] (cue 272) signature. And ultimately we can create a gate to automate based
[00:20:06,200 --> 00:20:10,220] (cue 273) on actual evidence, not just risk classification.
[00:20:10,480 --> 00:20:15,240] (cue 274) Right. With a simple command, for example APM audit.
[00:20:15,280 --> 00:20:19,040] (cue 275) My goal here is not to position this tool or that tool, but what's needed,
[00:20:19,080 --> 00:20:22,380] (cue 276) what we need as an industry to create trust.
[00:20:22,440 --> 00:20:23,240] (cue 277) So
[00:20:24,560 --> 00:20:27,880] (cue 278) what happens then? All right. So the model is pretty simple.
[00:20:27,920 --> 00:20:29,120] (cue 279) We have the receipt. We have the gate.
[00:20:29,160 --> 00:20:35,540] (cue 280) We have the rules. We have that command APM audit that we can run
[00:20:36,240 --> 00:20:40,320] (cue 281) on pull requests as a check which is going to look at the evidence,
[00:20:40,360 --> 00:20:42,320] (cue 282) look at the diff as the output.
[00:20:42,360 --> 00:20:45,760] (cue 283) Make sure that there is the integrity, the provenance, the checks passed,
[00:20:45,800 --> 00:20:50,400] (cue 284) that there was no tampering, that we trust the signer, that the policy that we have
[00:20:50,440 --> 00:20:55,600] (cue 285) defined is respected. And then we're going to have a result either valid or invalid.
[00:20:55,640 --> 00:21:01,360] (cue 286) Right. And that's the verification of, you know, that fabric of trust that we have
[00:21:01,400 --> 00:21:04,560] (cue 287) chosen to to basically accept.
[00:21:04,600 --> 00:21:08,660] (cue 288) Right. All right. So I'm going to move faster here.
[00:21:09,800 --> 00:21:10,660] (cue 289) Okay.
[00:21:12,160 --> 00:21:18,480] (cue 290) What this allows us to do is to basically forgo human review,
[00:21:18,520 --> 00:21:24,280] (cue 291) forgo CI processes that are creating bottlenecks and auto merge,
[00:21:24,320 --> 00:21:28,880] (cue 292) but not just based on risk classification, on our personal confidence or very smart
[00:21:28,920 --> 00:21:34,560] (cue 293) engineers we have hired who say, yes, I trust my factory, but actual evidence that
[00:21:34,600 --> 00:21:40,320] (cue 294) we can trace back. And then when an auditor comes in, because we are regulated
[00:21:40,360 --> 00:21:44,840] (cue 295) organization like a bank or a pharma company, if something goes wrong,
[00:21:44,880 --> 00:21:51,640] (cue 296) we can point exactly at our controls that follow a certain regulatory framework.
[00:21:51,680 --> 00:21:56,400] (cue 297) We can point back to the skill or the factory that we decided to trust.
[00:21:56,440 --> 00:21:59,880] (cue 298) We can point back to what went wrong, and we can take action,
[00:21:59,920 --> 00:22:02,860] (cue 299) and we can place the responsibility where it has to be.
[00:22:04,440 --> 00:22:09,760] (cue 300) All right. So there are several questions that this approach help us answer.
[00:22:09,800 --> 00:22:12,780] (cue 301) Right. So for example production broke.
[00:22:13,240 --> 00:22:17,960] (cue 302) And so if production broke and everything in production was built by a factory.
[00:22:18,000 --> 00:22:22,800] (cue 303) Then the question is which factory is responsible and which step?
[00:22:22,840 --> 00:22:29,600] (cue 304) Precisely because we have this receipt that chains the outputs to the evidence.
[00:22:29,640 --> 00:22:35,120] (cue 305) Then we can trace back from the source code back to the receipt and back to the
[00:22:35,160 --> 00:22:39,560] (cue 306) factory, thanks to the versioning, which starts everything because it assigns an
[00:22:39,600 --> 00:22:40,660] (cue 307) identity.
[00:22:41,920 --> 00:22:46,640] (cue 308) Another question we can answer is hey, a skill in the open source world that we are
[00:22:46,680 --> 00:22:50,000] (cue 309) using was compromised. So who's exposed?
[00:22:50,040 --> 00:22:54,880] (cue 310) Just look at the agent bill of materials or every single receipt that you stored.
[00:22:54,920 --> 00:23:00,240] (cue 311) Right. And then find which factories are using those and patch it,
[00:23:00,280 --> 00:23:05,120] (cue 312) revoke it in the rules. The gate will start forbidding any output from those
[00:23:05,160 --> 00:23:08,620] (cue 313) factories from going to production until you fix that.
[00:23:08,920 --> 00:23:14,680] (cue 314) All right. Okay. So then next question the checks pass the checks that we have
[00:23:14,720 --> 00:23:18,560] (cue 315) defined in the contracts, which are exactly our our quality bar.
[00:23:18,600 --> 00:23:22,400] (cue 316) Now they passed but did the agent change them right.
[00:23:22,440 --> 00:23:26,560] (cue 317) We have seen agents gaming the checks consistently.
[00:23:26,600 --> 00:23:29,160] (cue 318) So hey checks lives in the contracts.
[00:23:29,200 --> 00:23:35,040] (cue 319) They don't. They are not expected to be created by the same agent who work in the
[00:23:35,080 --> 00:23:38,040] (cue 320) factory. So they cannot be changed.
[00:23:38,080 --> 00:23:41,360] (cue 321) They are cryptographically tied into the receipt.
[00:23:41,400 --> 00:23:47,640] (cue 322) So if somebody tampers those, you are going to find out and they are enforced in the
[00:23:47,680 --> 00:23:50,120] (cue 323) gate with the audit command. Right?
[00:23:50,160 --> 00:23:53,460] (cue 324) So it's not possible to tamper those.
[00:23:54,000 --> 00:23:58,480] (cue 325) All right. And so let's imagine hey we want to add new checks because of course the
[00:23:58,520 --> 00:24:03,660] (cue 326) checks at the end become here our bar for automate for trust right.
[00:24:04,040 --> 00:24:07,020] (cue 327) So how can we ensure that they become mandatory.
[00:24:07,480 --> 00:24:11,120] (cue 328) Well if we create new checks these changes the factory.
[00:24:11,160 --> 00:24:14,940] (cue 329) And so we can basically release a new factory version.
[00:24:15,240 --> 00:24:18,400] (cue 330) Right. If we change the checks which are tied in the contracts,
[00:24:18,440 --> 00:24:20,680] (cue 331) then we have a new version of our factory.
[00:24:20,720 --> 00:24:25,320] (cue 332) If we create a new version of our factory, then we can go to our policy file and
[00:24:25,360 --> 00:24:32,120] (cue 333) say, now we are only going to accept diffs coming from factories
[00:24:32,160 --> 00:24:34,480] (cue 334) above version 2.1 for example.
[00:24:34,520 --> 00:24:41,340] (cue 335) Right. And so our apmx audit command, the gate will forbid any output coming from
[00:24:41,760 --> 00:24:45,620] (cue 336) older factory versions. All right.
[00:24:46,440 --> 00:24:49,920] (cue 337) And so well there is another question that we can answer.
[00:24:49,960 --> 00:24:52,880] (cue 338) You know, in the policy we can also define, hey, what are the skills that the
[00:24:52,920 --> 00:24:55,080] (cue 339) factories can use are allowed to use.
[00:24:55,120 --> 00:24:58,260] (cue 340) Again, we have that policy in the rules.
[00:24:59,300 --> 00:25:02,600] (cue 341) The dependencies are tied in the manifest in the lock file.
[00:25:02,640 --> 00:25:07,740] (cue 342) We can check with the audit command and we can get based on that.
[00:25:08,080 --> 00:25:13,340] (cue 343) Those are just some examples of what this approach can help us with.
[00:25:13,800 --> 00:25:20,240] (cue 344) So now if I go back to the factory runs, I can see on the copilot run
[00:25:20,280 --> 00:25:22,920] (cue 345) that the different contracts were run.
[00:25:22,960 --> 00:25:26,620] (cue 346) Planning a specification, build a review.
[00:25:27,080 --> 00:25:33,260] (cue 347) What allows the state to flow through the contracts is basically files,
[00:25:33,440 --> 00:25:34,920] (cue 348) right? So it's markdown files.
[00:25:34,960 --> 00:25:37,520] (cue 349) It's diffs. It's whatever files that go into disk.
[00:25:37,560 --> 00:25:41,280] (cue 350) So a contract generates files and under contract will consume those files.
[00:25:41,320 --> 00:25:48,000] (cue 351) Right. We see it completed and we can see the outputs it generated the receipt that
[00:25:48,040 --> 00:25:51,880] (cue 352) was generated in total in an in-toto file.
[00:25:51,920 --> 00:25:58,500] (cue 353) And I can run the audit command quickly to show hey
[00:25:58,600 --> 00:26:02,780] (cue 354) now eventually if we run this audit command which should run in CI/CD,
[00:26:04,400 --> 00:26:09,680] (cue 355) can we check that the integrity is there, that the provenance is the one that we
[00:26:09,720 --> 00:26:13,120] (cue 356) that we decided to trust that the checks ran, that they were valid,
[00:26:13,160 --> 00:26:16,540] (cue 357) etc.. So we can see that happening there locally.
[00:26:16,640 --> 00:26:19,920] (cue 358) Of course here we didn't sign the receipt.
[00:26:19,960 --> 00:26:26,440] (cue 359) So the check doesn't pass. Eventually what we need to do is to add
[00:26:26,480 --> 00:26:31,300] (cue 360) that add that command as a check to a pull request.
[00:26:31,360 --> 00:26:35,700] (cue 361) So I have here a pull request generated by the factory.
[00:26:36,880 --> 00:26:41,420] (cue 362) And as a check I have a single one, the audit command.
[00:26:41,640 --> 00:26:46,736] (cue 363) We should have also code scanning and stuff like that which runs the apmx audit
[00:26:46,776 --> 00:26:52,960] (cue 364) command. And the basically all the checks pass the receipt is signed.
[00:26:53,000 --> 00:26:56,920] (cue 365) It was signed by the environment which run the factory.
[00:26:56,960 --> 00:26:59,800] (cue 366) You know it can be CI/CD, it can be our Kubernetes cluster.
[00:26:59,840 --> 00:27:02,520] (cue 367) It can be wherever you decide to run your factories.
[00:27:02,560 --> 00:27:07,720] (cue 368) And that becomes a valid statement that allows us to say, okay,
[00:27:07,760 --> 00:27:09,800] (cue 369) can we auto-merge this now? Right.
[00:27:09,840 --> 00:27:13,580] (cue 370) And we have traceability and all those properties I spoke about.
[00:27:14,920 --> 00:27:21,460] (cue 371) All right. So I'm going to close by saying we we seem to be all converging
[00:27:21,520 --> 00:27:28,280] (cue 372) into something like some sort of definition of the smallest unit of what a factory
[00:27:28,320 --> 00:27:31,600] (cue 373) is. You know, Tessl was speaking yesterday about loops.
[00:27:31,640 --> 00:27:34,880] (cue 374) We have a lot of vendors speaking about automations.
[00:27:34,920 --> 00:27:36,780] (cue 375) I mean, speaking about contracts.
[00:27:37,880 --> 00:27:40,880] (cue 376) What I do think is that we need a shared format.
[00:27:40,920 --> 00:27:46,600] (cue 377) If we do want portability of of software factories, we need a shared format,
[00:27:46,640 --> 00:27:50,960] (cue 378) a shared standard, and we need not to reinvent the wheel.
[00:27:51,000 --> 00:27:55,660] (cue 379) We created a lot of this fabric of trust in the past for classic software.
[00:27:55,720 --> 00:28:00,280] (cue 380) Let's not reinvent the wheel and use the lessons learned to build the same fabric of
[00:28:00,320 --> 00:28:02,960] (cue 381) trust for agentic software factories.
[00:28:03,000 --> 00:28:05,740] (cue 382) And let's do that in the open, right?
[00:28:05,960 --> 00:28:10,160] (cue 383) So we need to define factories as code in an open format.
[00:28:10,200 --> 00:28:15,180] (cue 384) We need to version them because it all starts with versioning which is identity.
[00:28:15,240 --> 00:28:19,600] (cue 385) And we need then we can then get on evidence on actual evidence.
[00:28:19,640 --> 00:28:23,140] (cue 386) And and we get the traceability, the provenance, the portability.
[00:28:23,360 --> 00:28:26,020] (cue 387) We skip CI we skip peer reviews.
[00:28:26,360 --> 00:28:30,760] (cue 388) And I think this is how agentic software factories can scale to large regulated
[00:28:30,800 --> 00:28:36,360] (cue 389) enterprises. All right. So this is a proposition for an open format that I have
[00:28:36,400 --> 00:28:42,880] (cue 390) published. It's a draft I think this is the perfect conference to invite you to
[00:28:42,920 --> 00:28:44,940] (cue 391) participate in the conversation.
[00:28:45,040 --> 00:28:50,320] (cue 392) I hope that from there, we'll be able one day to say, hey, we're going to change the
[00:28:50,360 --> 00:28:55,260] (cue 393) payment systems in the bank and we can even auto merge.
[00:28:56,000 --> 00:28:57,480] (cue 394) All right. Thank you very much.
[00:28:57,520 --> 00:29:01,840] (cue 395) With that, I leave you with the QR code for all the content.
[00:29:01,880 --> 00:29:05,660] (cue 396) Thank you. And any questions. I think we have one minute.
[00:29:06,120 --> 00:29:08,140] (cue 397) All right. Thanks.
[00:29:10,290 --> 00:29:13,600] (cue 398) Thank you. Thank you. Talk to you about the contents of it.
[00:29:13,640 --> 00:29:15,680] (cue 399) Thank you. Daniel. That was an amazing session.
[00:29:15,720 --> 00:29:18,960] (cue 400) Thanks. You're going to be doing questions now or do you want to stage?
[00:29:19,000 --> 00:29:22,000] (cue 401) Are you have time? Then I can pick a couple.
[00:29:22,040 --> 00:29:24,120] (cue 402) Of quick questions you want to ask you now.
[00:29:24,160 --> 00:29:27,180] (cue 403) Oh, there we go. I'll come down.
[00:29:29,320 --> 00:29:31,660] (cue 404) Let me go down. I can do the exercise.
[00:29:33,240 --> 00:29:37,280] (cue 405) So as part of the checks that you're defining is part of these kind of these agents.
[00:29:37,320 --> 00:29:39,840] (cue 406) First off, I love the kind of like, OPA design of all of this.
[00:29:39,880 --> 00:29:42,100] (cue 407) I think that makes a ton of sense.
[00:29:42,160 --> 00:29:48,280] (cue 408) But in regards to the checks, are you kind of alluding to kind of them like almost
[00:29:48,320 --> 00:29:51,340] (cue 409) being a replacement more for like unit and integration tests?
[00:29:51,760 --> 00:29:57,360] (cue 410) In that or is it more a specific kind of like a volatile of the specific code that
[00:29:57,400 --> 00:30:01,050] (cue 411) you're producing as part of that individual agent execution?
[00:30:01,090 --> 00:30:06,080] (cue 412) Yeah, yeah. So the checks have to be deterministic.
[00:30:06,120 --> 00:30:10,720] (cue 413) In my view. The checks that you define in a contract ought to be deterministic
[00:30:10,760 --> 00:30:15,880] (cue 414) because that's that's basically the harness around the model that you can basically
[00:30:15,920 --> 00:30:22,880] (cue 415) trust that elevates trust. So a check could be hey what is if we have a contract for
[00:30:22,920 --> 00:30:27,360] (cue 416) writing specifications, then maybe there is a specific format in what you want to
[00:30:27,400 --> 00:30:29,960] (cue 417) make sure that the specifications are written right.
[00:30:30,000 --> 00:30:35,040] (cue 418) So that could be one check if you have a contract for performing pull review pull
[00:30:35,080 --> 00:30:37,640] (cue 419) request reviews, then maybe it's about the format.
[00:30:37,680 --> 00:30:44,400] (cue 420) Maybe if you have one for building diffs actual code, then maybe it's about
[00:30:44,440 --> 00:30:49,840] (cue 421) hey, you can attach this piece of code or you have to follow this style.
[00:30:49,880 --> 00:30:52,540] (cue 422) So linting and stuff like that, right?
[00:30:52,630 --> 00:30:53,430] (cue 423) Yeah.
[00:31:03,880 --> 00:31:06,390] (cue 424) Staging and so on. And so
[00:31:09,300 --> 00:31:12,240] (cue 425) to load on CI. Exactly. How that
[00:31:24,480 --> 00:31:29,000] (cue 426) emerged. The idea is that you skip this running the checks again in CI because you
[00:31:29,040 --> 00:31:31,040] (cue 427) have the evidence that they have run.
[00:31:31,080 --> 00:31:34,520] (cue 428) You have trusted evidence sign that they have run and they have passed.
[00:31:34,560 --> 00:31:39,520] (cue 429) So the idea is that you can you you don't need to run them again on CI centrally.
[00:31:39,560 --> 00:31:44,220] (cue 430) So shifting left testing as much as possible in a in a trusted way.
[00:31:44,970 --> 00:31:49,140] (cue 431) Yeah. You go one more time for sure.
[00:31:50,480 --> 00:31:54,440] (cue 432) Hello. Waterfall was bad idea for humans.
[00:31:54,480 --> 00:31:57,660] (cue 433) Why would waterfall be a good idea for a AI?
[00:31:58,160 --> 00:32:00,720] (cue 434) Sorry. Can can you? Because they held of echo.
[00:32:00,760 --> 00:32:06,000] (cue 435) I didn't hear you well. So waterfall was bad idea for humans?
[00:32:06,040 --> 00:32:10,340] (cue 436) Yeah. Why would waterfall be a good idea for AI?
[00:32:11,800 --> 00:32:17,120] (cue 437) So, you know, I think waterfall did very well in some situations.
[00:32:17,160 --> 00:32:18,680] (cue 438) It still does well in some situations.
[00:32:18,720 --> 00:32:21,160] (cue 439) And I agree with you that it's not a solution for everything.
[00:32:21,200 --> 00:32:24,580] (cue 440) I think your question comes from the graph that you've seen,
[00:32:24,680 --> 00:32:27,560] (cue 441) which seems to be a waterfall approach.
[00:32:27,600 --> 00:32:33,000] (cue 442) I do think we are going to see forms in which there's nothing to do with waterfall.
[00:32:33,040 --> 00:32:39,680] (cue 443) Even when you do a scrum and all this, you follow a certain approach to
[00:32:39,720 --> 00:32:44,720] (cue 444) something. We are speaking about automation here, so I agree with you that waterfall
[00:32:44,760 --> 00:32:45,720] (cue 445) is not the answer for everything.
[00:32:45,760 --> 00:32:47,620] (cue 446) And this was just one example.
[00:32:47,760 --> 00:32:52,800] (cue 447) I think we can very likely be able to model another sort of process in the form of
[00:32:52,840 --> 00:32:54,180] (cue 448) contracts as well.
.tessl-plugin
skills
talk-boeckeler-context-lifecycles
talk-chanezon-gordon-agent-experience-evaluation
talk-meppiel-versioned-agent-contracts