AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
86%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Landon Cox · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=hM-xZk3Wv0I
Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.
For topic navigation, see outline.md.
Cue 1 — 00:00:00,129 --> 00:00:01,989
How's everybody doing? Doing okay.
Cue 2 — 00:00:02,249 --> 00:00:05,689
Awesome. All right, so my name is Landon.
Cue 3 — 00:00:05,729 --> 00:00:08,296
I am a researcher at GitHub Next.
Cue 4 — 00:00:08,336 --> 00:00:12,149
So GitHub Next is GitHub's research arm.
Cue 5 — 00:00:12,409 --> 00:00:16,909
I'm also at Microsoft Research, which you might guess is Microsoft's research arm.
Cue 6 — 00:00:17,409 --> 00:00:21,969
So I lead security for GitHub Agentic Workflows.
Cue 7 — 00:00:22,009 --> 00:00:25,349
And so that's why, this talk is on security.
Cue 8 — 00:00:25,926 --> 00:00:30,820
Okay. So to start out, I'm just going to kind of explain what GitHub Agentic
Cue 9 — 00:00:30,860 --> 00:00:34,946
Workflows are and why you might be interested in using them.
Cue 10 — 00:00:35,686 --> 00:00:41,866
So GitHub Agentic Workflows started as a project by Peli and Don,
Cue 11 — 00:00:42,326 --> 00:00:48,306
a little over a year ago. I joined in December of 2025 to work on security.
Cue 12 — 00:00:48,926 --> 00:00:53,086
And this project has really taken off like nothing I've ever worked on.
Cue 13 — 00:00:53,126 --> 00:00:58,286
So you have a researcher, and my typical span for getting something into people's
Cue 14 — 00:00:58,326 --> 00:01:02,366
hands is, you know, you think of a problem, you work on a paper for a couple of
Cue 15 — 00:01:02,406 --> 00:01:05,966
years, you get it published, you hope a product group takes notice,
Cue 16 — 00:01:06,006 --> 00:01:07,986
and then they actually put it into practice.
Cue 17 — 00:01:08,526 --> 00:01:13,206
And the rate at which we've been able to get stuff done in this project because of
Cue 18 — 00:01:13,246 --> 00:01:16,846
coding agents is completely different from anything else that I've ever worked on.
Cue 19 — 00:01:16,886 --> 00:01:22,006
So like I said, I joined in December of 2025 roughly.
Cue 20 — 00:01:22,046 --> 00:01:26,646
We hit technical preview in February, we then hit public preview in June.
Cue 21 — 00:01:26,686 --> 00:01:32,406
And then just last week at Universe, we announced that we were GA general
Cue 22 — 00:01:32,446 --> 00:01:38,006
availability. So, GitHub is now standing behind this project and will be.
Cue 23 — 00:01:38,046 --> 00:01:40,666
So we'll be doing so for the foreseeable future.
Cue 24 — 00:01:41,486 --> 00:01:47,686
So one of the things that has allowed us to get so much done in such a small period
Cue 25 — 00:01:47,726 --> 00:01:52,974
of time with such, frankly, a small team, is that we've been dogfooding agentic
Cue 26 — 00:01:53,014 --> 00:01:57,326
workflows the entire time. And so that's part of what I'm going to talk about in a
Cue 27 — 00:01:57,366 --> 00:02:01,006
few slides is just like, what is the experience of using agentic workflows?
Cue 28 — 00:02:01,046 --> 00:02:05,986
And what what does your repo look like if you build with agentic workflows?
Cue 29 — 00:02:07,246 --> 00:02:10,406
Okay. So what are they in the first place?
Cue 30 — 00:02:10,446 --> 00:02:12,506
So this is an automation platform.
Cue 31 — 00:02:12,806 --> 00:02:16,886
It's an automation platform that runs on top of GitHub Actions if you're familiar
Cue 32 — 00:02:16,926 --> 00:02:20,446
with GitHub Actions. This is GitHub's CI/CD platform.
Cue 33 — 00:02:20,486 --> 00:02:24,386
We basically build on all of the primitives of GitHub Actions.
Cue 34 — 00:02:24,606 --> 00:02:30,286
And the way that you create an agent workflow is that you tell an agent what it is
Cue 35 — 00:02:30,326 --> 00:02:36,046
that you want to automate, and then the agent spits out a markdown file that's
Cue 36 — 00:02:36,086 --> 00:02:40,646
composed of two pieces. The first part is the prompt that will end up being
Cue 37 — 00:02:40,686 --> 00:02:45,366
delivered to an agent. And then the second part is a bunch of front matter with tons
Cue 38 — 00:02:45,406 --> 00:02:51,326
of configuration. Now you as a user of agent workflows, you can look at this if you
Cue 39 — 00:02:51,366 --> 00:02:54,286
want, but there's no need for you to.
Cue 40 — 00:02:54,326 --> 00:03:01,246
But that front matter is then deterministically compiled into an actions YAML.
Cue 41 — 00:03:01,286 --> 00:03:06,426
And then you just run that YAML like you would anything else on Actions.
Cue 42 — 00:03:06,606 --> 00:03:12,446
Now part of what the deterministic compiler is doing is it is instantiating a
Cue 43 — 00:03:12,486 --> 00:03:18,806
security architecture in the YAML to make sure that your code and your secrets are
Cue 44 — 00:03:18,846 --> 00:03:21,726
safe. And that's kind of like where I come into play.
Cue 45 — 00:03:21,766 --> 00:03:26,386
So that architecture is more or less what I'm responsible for.
Cue 46 — 00:03:27,606 --> 00:03:31,086
Now, underneath all that, we have an extensible reasoning layer.
Cue 47 — 00:03:31,126 --> 00:03:35,486
So we support almost any coding agent you might want to use.
Cue 48 — 00:03:35,526 --> 00:03:39,892
We support Copilot. Of course we support Claude.
Cue 49 — 00:03:39,932 --> 00:03:43,946
We support Codex, Pi, Gemini et cetera, etc..
Cue 50 — 00:03:44,166 --> 00:03:48,906
Okay. So that's kind of an overview of what agentic workflows are.
Cue 51 — 00:03:49,166 --> 00:03:50,886
Why would you want to use it. Okay.
Cue 52 — 00:03:50,926 --> 00:03:55,646
So very quick. Let's say you're a repo maintainer and you start a project and
Cue 53 — 00:03:55,686 --> 00:03:57,346
everything is going whoops.
Cue 54 — 00:04:06,486 --> 00:04:08,446
Okay. I'll just speed through this.
Cue 55 — 00:04:08,486 --> 00:04:12,006
Everything is going great. You're getting lots of stars, although the animation is
Cue 56 — 00:04:12,046 --> 00:04:18,446
not showing. And then after a while, the technical debt starts to pile up.
Cue 57 — 00:04:18,486 --> 00:04:22,486
Okay, so the technical debt, this is all of the each of the stuff,
Cue 58 — 00:04:22,526 --> 00:04:26,386
you know, you're supposed to be doing, but you don't really want to do.
Cue 59 — 00:04:26,526 --> 00:04:28,806
So that's saying, like, you know, your docs are out of sync.
Cue 60 — 00:04:28,846 --> 00:04:33,726
The issues with bug reports and feature requests, they start to pile up.
Cue 61 — 00:04:33,766 --> 00:04:37,046
Your tests are out of date. You've got copypasta everywhere.
Cue 62 — 00:04:37,086 --> 00:04:43,286
Your logging is insufficient. All this stuff that you know is really not fun to do,
Cue 63 — 00:04:43,326 --> 00:04:44,866
but, you know, you should be doing.
Cue 64 — 00:04:45,686 --> 00:04:51,126
And so agentic workflows gives you a way to apply continuous automation.
Cue 65 — 00:04:51,166 --> 00:04:54,446
Again, I'm missing my animations to all of that.
Cue 66 — 00:04:54,486 --> 00:04:56,426
So let me see if it works here.
Cue 67 — 00:04:57,606 --> 00:05:00,486
There we go. Okay. So I'm just going to click there.
Cue 68 — 00:05:00,526 --> 00:05:06,626
So GitHub Agentic Workflows is a way to just go in and clean up all of your messes.
Cue 69 — 00:05:07,766 --> 00:05:13,426
So we designed GitHub Agentic Workflows with three design principles in mind.
Cue 70 — 00:05:13,606 --> 00:05:18,286
The first one is that you want to assign the dirty work to your agentic workflows
Cue 71 — 00:05:18,326 --> 00:05:23,786
okay. Things like testing, logging docs, refactoring, dependency work.
Cue 72 — 00:05:24,326 --> 00:05:28,806
This is all the stuff that you can automate very easily and that lets your devs
Cue 73 — 00:05:28,846 --> 00:05:32,746
focus on features and the stuff they actually want to be doing anyway.
Cue 74 — 00:05:33,246 --> 00:05:37,466
And you want this stuff to run continuously and automatically.
Cue 75 — 00:05:37,886 --> 00:05:41,846
So from personal experience, I can just say it's amazing to just kind of wake up
Cue 76 — 00:05:41,886 --> 00:05:48,246
every morning and have 3 or 4 hours with all of this toil and drudgery already done
Cue 77 — 00:05:48,286 --> 00:05:52,886
and packaged up into a bunch of bite sized pieces that are really easy for me to
Cue 78 — 00:05:52,926 --> 00:05:57,006
inspect and merge. And that kind of leads into the third design principle,
Cue 79 — 00:05:57,046 --> 00:06:03,326
which is workflows work best when they're dealing with bite sized chunks of work.
Cue 80 — 00:06:03,366 --> 00:06:07,486
Okay. So every PR should be tens of lines of code that are easy.
Cue 81 — 00:06:07,526 --> 00:06:11,026
That's really easy for you to inspect and merge.
Cue 82 — 00:06:11,246 --> 00:06:14,526
Maybe hundreds of lines, although that's pretty rare.
Cue 83 — 00:06:14,566 --> 00:06:16,866
And thousands of lines. Absolutely not.
Cue 84 — 00:06:16,926 --> 00:06:22,526
So breaking things down into these bite sized chunks of work is good,
Cue 85 — 00:06:22,566 --> 00:06:25,406
both for the humans and for the agents.
Cue 86 — 00:06:25,446 --> 00:06:28,986
So it's good for the humans because it lets you get through the quickly.
Cue 87 — 00:06:29,286 --> 00:06:34,086
But it's good for the agents because it lets them apply all of their context to the
Cue 88 — 00:06:34,126 --> 00:06:38,406
problem at hand. And so they do a very good job with the small task.
Cue 89 — 00:06:38,446 --> 00:06:40,186
Okay, now
Cue 90 — 00:06:41,566 --> 00:06:46,046
what happens? This is okay. So this is a graph of the two repos,
Cue 91 — 00:06:46,086 --> 00:06:49,746
the security repos that, I'm in charge of.
Cue 92 — 00:06:49,886 --> 00:06:53,626
So we have an MCP gateway, and then we have an agent firewall.
Cue 93 — 00:06:53,686 --> 00:06:55,406
You can just think of them as blue and green.
Cue 94 — 00:06:55,446 --> 00:06:57,026
It's not really that important.
Cue 95 — 00:06:57,166 --> 00:07:01,606
And this is a breakdown of all of the PRs that were categorized.
Cue 96 — 00:07:01,646 --> 00:07:07,246
I think it's the last 2000 PRs, that came out of an agent workflow.
Cue 97 — 00:07:07,286 --> 00:07:09,706
Sort of what what are they? What are they related to?
Cue 98 — 00:07:09,846 --> 00:07:15,046
I think it's kind of interesting that most of the PRs that come out of workflows are
Cue 99 — 00:07:15,086 --> 00:07:16,986
refactoring and deduplication.
Cue 100 — 00:07:17,206 --> 00:07:21,426
So this was kind of a philosophical shift for us.
Cue 101 — 00:07:22,126 --> 00:07:26,326
I don't know we have with you guys, but when we talk to people within our big
Cue 102 — 00:07:26,366 --> 00:07:32,766
companies, both Microsoft and GitHub, there's a lot of, desire to just keep slop out
Cue 103 — 00:07:32,806 --> 00:07:35,066
of the repo and that slows people down.
Cue 104 — 00:07:35,646 --> 00:07:40,686
For us, we just kind of accept that some of the slops going to get in and we work
Cue 105 — 00:07:40,726 --> 00:07:42,926
really hard to clean it up. Right.
Cue 106 — 00:07:42,966 --> 00:07:46,526
And so we're not going to stress out over a little bit of slop getting into the
Cue 107 — 00:07:46,566 --> 00:07:51,486
repo. Instead we're going to automate and just run things all the time to clean that
Cue 108 — 00:07:51,526 --> 00:07:53,526
up. Help us clean it up. Right.
Cue 109 — 00:07:53,566 --> 00:07:55,166
And that's part of your technical debt.
Cue 110 — 00:07:55,206 --> 00:08:00,246
You can think of the slop that creeps into your code base as really no different
Cue 111 — 00:08:00,286 --> 00:08:01,846
than your docs getting out of sync.
Cue 112 — 00:08:01,886 --> 00:08:06,606
So, you know, GitHub Agentic Workflows are great for just cleaning up all all that
Cue 113 — 00:08:06,646 --> 00:08:07,446
mess.
Cue 114 — 00:08:08,806 --> 00:08:12,086
Okay. And then you can kind of see the other categories that cover like test
Cue 115 — 00:08:12,126 --> 00:08:18,326
coverage, library dependencies, syncing with external APIs, logging and
Cue 116 — 00:08:18,366 --> 00:08:19,766
observability. I'll return to it.
Cue 117 — 00:08:19,806 --> 00:08:22,826
That's actually a really important part of our system too.
Cue 118 — 00:08:23,606 --> 00:08:27,266
Okay. And then you can look at, well, what does the code base actually look like.
Cue 119 — 00:08:27,406 --> 00:08:32,006
I think somebody earlier was talking about, you know, sloppy unit tests that are
Cue 120 — 00:08:32,046 --> 00:08:34,366
very little value. I'm sure we have tons of those.
Cue 121 — 00:08:34,406 --> 00:08:37,546
You can see here that half of our code base is tests.
Cue 122 — 00:08:37,806 --> 00:08:41,416
I will definitely not vouch for the quality of all of these tests.
Cue 123 — 00:08:41,456 --> 00:08:45,006
The only thing I can say is that we're constantly cleaning them up.
Cue 124 — 00:08:45,046 --> 00:08:49,946
So there's probably some. Again, we don't mind a little slop getting in.
Cue 125 — 00:08:50,846 --> 00:08:53,006
But I think the ratios here are kind of interesting.
Cue 126 — 00:08:53,046 --> 00:08:55,946
So, you know, half our code base is tests.
Cue 127 — 00:08:56,726 --> 00:08:59,386
About 25% is the implementation.
Cue 128 — 00:08:59,566 --> 00:09:03,646
And I think the next category is also really important to point out,
Cue 129 — 00:09:03,686 --> 00:09:07,526
which is 10% of our code base is documentation and specification.
Cue 130 — 00:09:07,566 --> 00:09:11,806
So this is both to help us coordinate across repos.
Cue 131 — 00:09:11,846 --> 00:09:18,366
We use lots of and schemas. But then it also allows us to share
Cue 132 — 00:09:18,406 --> 00:09:24,126
like architectural designs with our agents as they're working on the repo.
Cue 133 — 00:09:24,166 --> 00:09:27,506
So we think all of this stuff kind of keeps everything in line.
Cue 134 — 00:09:29,126 --> 00:09:35,046
So I think the best articulation of what it's like to use agentic workflows is this
Cue 135 — 00:09:35,086 --> 00:09:41,586
blog post from Don Syme. So Don actually spoke at DevCon in London over the summer.
Cue 136 — 00:09:41,806 --> 00:09:47,486
It was a great talk. And part of what he talked about is the experience of coming
Cue 137 — 00:09:47,526 --> 00:09:51,266
back to some open source repos that he had let rot.
Cue 138 — 00:09:51,326 --> 00:09:56,686
So Don had maintained, repos for the programming language F#.
Cue 139 — 00:09:56,726 --> 00:10:02,006
And then he went off and he did other things, and he had hundreds of open issues
Cue 140 — 00:10:02,046 --> 00:10:04,526
that are just accumulated over the course of years.
Cue 141 — 00:10:04,566 --> 00:10:09,846
And so over the course of the weekend, he went in with workflows and started to chip
Cue 142 — 00:10:09,886 --> 00:10:11,566
away at all that technical debt.
Cue 143 — 00:10:11,606 --> 00:10:16,126
And he writes about what a joy it was to actually go through this process,
Cue 144 — 00:10:16,166 --> 00:10:20,906
because he was kind of reviving something that he cared very deeply about.
Cue 145 — 00:10:21,046 --> 00:10:24,546
And now it was finally back in a state that he could be proud of.
Cue 146 — 00:10:24,646 --> 00:10:26,286
So I encourage you to read that blog post.
Cue 147 — 00:10:26,326 --> 00:10:32,706
To me, it's like the best, argument for why you would adopt agentic workflows.
Cue 148 — 00:10:34,766 --> 00:10:37,966
Okay. So I mentioned that I lead security.
Cue 149 — 00:10:38,006 --> 00:10:41,986
The reason I'm here on stage is because I'm here to talk about security.
Cue 150 — 00:10:42,366 --> 00:10:48,326
So what about security? So we have kind of three design principles.
Cue 151 — 00:10:48,366 --> 00:10:51,246
You'll see. This was common with Guy's keynote too.
Cue 152 — 00:10:51,286 --> 00:10:53,446
There's always three. Three is a nice number.
Cue 153 — 00:10:53,486 --> 00:10:57,226
So there's three design principles that we applied to our security model.
Cue 154 — 00:10:57,686 --> 00:11:03,646
The first one is by far the most important, which is we never allow agents to access
Cue 155 — 00:11:03,686 --> 00:11:06,186
auth tokens or other secrets directly.
Cue 156 — 00:11:06,406 --> 00:11:11,026
And everything kind of is built on that foundational promise to our users.
Cue 157 — 00:11:11,526 --> 00:11:16,766
So if your agent is prompt injected or goes off the rails, we want to be able to
Cue 158 — 00:11:16,806 --> 00:11:22,926
tell people and people, be confident that that agent is not going to suddenly share
Cue 159 — 00:11:22,966 --> 00:11:24,986
the auth tokens somewhere else.
Cue 160 — 00:11:25,086 --> 00:11:29,686
Okay. Now, what that means is it informs how we architect everything.
Cue 161 — 00:11:29,726 --> 00:11:33,106
I'll show that in a, in a in the next slide.
Cue 162 — 00:11:34,446 --> 00:11:39,566
The second two or the other two design principles are about how we control reads and
Cue 163 — 00:11:39,606 --> 00:11:45,286
how we control writes. So the middle one is how we prevent our workflows from being
Cue 164 — 00:11:45,326 --> 00:11:47,966
exposed to untrustworthy content.
Cue 165 — 00:11:48,006 --> 00:11:53,046
So, you know, we're GitHub. There are comments from randos all over the place.
Cue 166 — 00:11:53,086 --> 00:11:55,306
You don't necessarily want your workflow.
Cue 167 — 00:11:56,286 --> 00:11:58,406
Reading that content, it could be dangerous.
Cue 168 — 00:11:58,446 --> 00:12:03,006
So we have a mechanism called integrity filtering, which lets users control the
Cue 169 — 00:12:03,046 --> 00:12:05,626
level of trust they're willing to expose their agent to.
Cue 170 — 00:12:05,686 --> 00:12:07,586
I'm not going to talk about that one so much.
Cue 171 — 00:12:07,806 --> 00:12:11,886
The third one, how we deal with writes, I think, is actually very important for the
Cue 172 — 00:12:11,926 --> 00:12:17,366
mental model of how workflows works, which is something called safe outputs.
Cue 173 — 00:12:17,406 --> 00:12:20,466
So our agents run in a read only mode.
Cue 174 — 00:12:20,766 --> 00:12:26,206
We don't allow them to make any modifications at the time that they're running.
Cue 175 — 00:12:26,246 --> 00:12:31,606
Instead, what we do is we collect all of the writes that they want to perform.
Cue 176 — 00:12:31,646 --> 00:12:37,878
And then after the agent has exited, we basically go over all of those writes
Cue 177 — 00:12:37,918 --> 00:12:41,846
deterministically and apply policy to remove things that aren't allowed.
Cue 178 — 00:12:41,886 --> 00:12:43,326
So I can give you a concrete example.
Cue 179 — 00:12:43,366 --> 00:12:47,666
Let's say that your agent goes off the rails and creates a thousand issues.
Cue 180 — 00:12:48,086 --> 00:12:52,966
Your workflow can have a policy that says the agent is only allowed to create one
Cue 181 — 00:12:53,006 --> 00:12:56,406
issue, and so we'll check the other 999.
Cue 182 — 00:12:56,446 --> 00:13:02,686
But you can apply all sorts of deterministic, checks on outputs using this process.
Cue 183 — 00:13:02,726 --> 00:13:07,606
And I think one of the things that is really helpful about the way that this is a
Cue 184 — 00:13:07,646 --> 00:13:12,046
very opinionated security model, by the way, it may not be perfect for all kinds of
Cue 185 — 00:13:12,086 --> 00:13:18,646
work. But our experience has been that because you are, you feel safe using
Cue 186 — 00:13:18,686 --> 00:13:21,926
agentic workflows, you can work really fast.
Cue 187 — 00:13:21,966 --> 00:13:26,446
Right. So good. Good guardrails help people move fast.
Cue 188 — 00:13:26,486 --> 00:13:28,586
And that's been one of our mantras.
Cue 189 — 00:13:29,166 --> 00:13:36,226
Okay. So let me talk about how we architect things to keep secrets away from agents.
Cue 190 — 00:13:36,366 --> 00:13:40,206
Basically, we run our agent in a micro VM.
Cue 191 — 00:13:40,246 --> 00:13:43,786
We support Cloud Hypervisor. We support NV.
Cue 192 — 00:13:45,006 --> 00:13:48,886
That's where it's supported. We also have a bunch of customers on self-hosted
Cue 193 — 00:13:48,926 --> 00:13:51,806
runners and other kinds of infra that don't support micro VMs.
Cue 194 — 00:13:51,846 --> 00:13:55,986
In that case, the agent runs in a container which is less than ideal.
Cue 195 — 00:13:56,606 --> 00:14:01,746
But regardless of whether it's in a micro VM or in a container,
Cue 196 — 00:14:02,006 --> 00:14:06,646
we have a series of proxies and gateways that hold all of the auth and keep them
Cue 197 — 00:14:06,686 --> 00:14:11,886
away from the agent. Okay, so in this case, I've highlighted the API proxy which
Cue 198 — 00:14:11,926 --> 00:14:17,766
proxies all of the calls. And then the GitHub MCP server which holds the credentials
Cue 199 — 00:14:17,806 --> 00:14:24,386
for accessing GitHub. So even if the agent becomes compromised,
Cue 200 — 00:14:25,166 --> 00:14:29,186
and as long as it stays in its sandbox, it's not going to jump out.
Cue 201 — 00:14:31,326 --> 00:14:37,406
Because we've separated everything into these different gateways and proxies.
Cue 202 — 00:14:37,446 --> 00:14:40,926
It gives us a tremendous amount of insight into what's going on.
Cue 203 — 00:14:40,966 --> 00:14:45,626
And I mentioned that we spent a lot of time improving our observability and logging.
Cue 204 — 00:14:46,166 --> 00:14:48,326
And so we log everything. Right.
Cue 205 — 00:14:48,366 --> 00:14:52,886
So from the agent side, we log the agent conversation.
Cue 206 — 00:14:52,926 --> 00:14:55,606
We log every single MCP tool call.
Cue 207 — 00:14:55,646 --> 00:15:02,406
We log all of the API calls through the API calls, including the token burn on each
Cue 208 — 00:15:02,446 --> 00:15:07,766
of those calls. We log all of the outputs, and then we log what's going on with our
Cue 209 — 00:15:07,806 --> 00:15:12,686
network firewall. And so this is really important for security.
Cue 210 — 00:15:12,726 --> 00:15:15,206
It's also a hugely important for debugging.
Cue 211 — 00:15:15,246 --> 00:15:19,686
And then it turns out and I also give a talk on token efficiency.
Cue 212 — 00:15:19,726 --> 00:15:24,606
This is hugely important for closing the token efficiency loop because you can take
Cue 213 — 00:15:24,646 --> 00:15:27,326
all of this data and we store it in GitHub Actions.
Cue 214 — 00:15:27,366 --> 00:15:30,546
Or you can plug in your OTel endpoint if you want.
Cue 215 — 00:15:30,766 --> 00:15:35,506
And then you can just sort of like look at what happened and optimize your workflow.
Cue 216 — 00:15:35,846 --> 00:15:39,146
And I think somebody said this earlier I can't remember which of the speakers.
Cue 217 — 00:15:40,006 --> 00:15:43,886
But the truth is that all of the inputs to your agent workflow,
Cue 218 — 00:15:43,926 --> 00:15:49,966
the prompts and the skills and all that stuff, that is also technical debt.
Cue 219 — 00:15:50,006 --> 00:15:54,726
And so part of the technical debt that we are trying to remove here is we can look
Cue 220 — 00:15:54,766 --> 00:15:58,846
at these logs. We can see whether or not everything that's part of the workflow is
Cue 221 — 00:15:58,886 --> 00:16:02,046
strictly required. And then we can modify workflows.
Cue 222 — 00:16:02,086 --> 00:16:07,826
So we have a great sort of closing the loop self optimization path here.
Cue 223 — 00:16:09,806 --> 00:16:13,051
Okay. I mentioned how we handle writes.
Cue 224 — 00:16:13,091 --> 00:16:17,546
Basically all of the writes are buffered through the safe outputs.
Cue 225 — 00:16:17,646 --> 00:16:20,746
The agent basically runs in a read only mode.
Cue 226 — 00:16:21,246 --> 00:16:27,995
And then we inspect what happens after the agent has after the agent has exited.
Cue 238 — 00:17:09,791 --> 00:17:13,911
Okay. So what this looks at looks like is that after the agent job exits,
Cue 239 — 00:17:13,951 --> 00:17:19,491
we have exit exits. We have this file full of things that the agent wanted to do,
Cue 240 — 00:17:19,871 --> 00:17:22,491
and we haven't applied it to GitHub at this point.
Cue 241 — 00:17:22,671 --> 00:17:26,371
And then we go through a policy that's part of that front matter I mentioned.
Cue 242 — 00:17:26,871 --> 00:17:30,271
And we just toss everything that is not compliant with the policy.
Cue 243 — 00:17:30,311 --> 00:17:34,431
So if your agent tries to, I don't know, write a comment and you don't allow it,
Cue 244 — 00:17:34,471 --> 00:17:40,351
then we get rid of that. If your agent tries to insert a URL into the PR that you
Cue 245 — 00:17:40,391 --> 00:17:41,871
don't recognize, what you check that too.
Cue 246 — 00:17:41,911 --> 00:17:47,111
So you can very easily, deterministically get rid of stuff you don't want to happen
Cue 247 — 00:17:47,151 --> 00:17:51,351
here. So again, it's meant to make you feel very, very safe,
Cue 248 — 00:17:51,391 --> 00:17:56,611
right? Anything that survives that policy check we then apply.
Cue 249 — 00:17:57,191 --> 00:18:01,811
So if you wanted to write just a single issue, it will generate a single issue.
Cue 250 — 00:18:05,131 --> 00:18:11,851
Okay. So that's kind of where agentic workflows is and kind of
Cue 251 — 00:18:11,891 --> 00:18:15,311
has been for the last couple of months.
Cue 252 — 00:18:15,851 --> 00:18:18,451
But we're also kind of excited about where things are going.
Cue 253 — 00:18:18,491 --> 00:18:21,911
You know, we're researchers, so we're always looking towards the future.
Cue 254 — 00:18:22,291 --> 00:18:26,891
And for us, the thing that is coming, maybe it's already there.
Cue 255 — 00:18:26,931 --> 00:18:31,831
You're already there. Are these large scale cross repo workflows.
Cue 256 — 00:18:32,011 --> 00:18:35,731
Okay. So, you know, we had our single developer that had to worry about technical
Cue 257 — 00:18:35,771 --> 00:18:40,151
debt. Now imagine you're an admin or you're responsible for,
Cue 258 — 00:18:40,971 --> 00:18:45,451
tens of thousands. You have some customers with hundreds of thousands of repos,
Cue 259 — 00:18:45,491 --> 00:18:50,131
and you actually have to figure out how to get rid of all of the technical debt in
Cue 260 — 00:18:50,171 --> 00:18:51,191
all of those.
Cue 261 — 00:18:52,891 --> 00:18:58,351
So that's why we are building something called Central Agent Ops.
Cue 262 — 00:18:59,011 --> 00:19:04,671
And this is kind of like a centralized way to manage view,
Cue 263 — 00:19:06,011 --> 00:19:11,071
distribute agent workflows across an entire organization.
Cue 264 — 00:19:12,491 --> 00:19:14,911
So Peli is working very hard on this.
Cue 265 — 00:19:15,571 --> 00:19:21,831
Here is the URL. If you want to look at you can you can take a look yourself.
Cue 266 — 00:19:22,331 --> 00:19:25,131
I should mention everything we do is open source.
Cue 267 — 00:19:25,171 --> 00:19:29,611
So, you know, you can take a look at our repos, take what you want,
Cue 268 — 00:19:29,651 --> 00:19:32,691
leave behind what you don't want, point your agent at it and say,
Cue 269 — 00:19:32,731 --> 00:19:37,071
hey, go. Go take what is what looks good here.
Cue 270 — 00:19:37,811 --> 00:19:39,551
Yeah. We're an open source platform.
Cue 271 — 00:19:40,491 --> 00:19:41,751
Okay, so that's cool.
Cue 272 — 00:19:43,091 --> 00:19:48,411
So as the security guy, I look at this cross repo, large scale stuff and I have to
Cue 273 — 00:19:48,451 --> 00:19:52,411
think about, okay, where are the security problems in this?
Cue 274 — 00:19:52,451 --> 00:19:57,131
In this world? And so that's what I'm going to talk about next.
Cue 275 — 00:19:57,171 --> 00:19:59,811
I will warn you, this is kind of the first time I've talked about this stuff.
Cue 276 — 00:19:59,851 --> 00:20:03,711
So if the presentation is a little clunky, just blame it on,
Cue 277 — 00:20:04,011 --> 00:20:08,911
the newness of it all. Okay, so here's a scenario I worry about.
Cue 278 — 00:20:09,051 --> 00:20:12,271
This is one that has come up a couple of times.
Cue 279 — 00:20:12,411 --> 00:20:15,351
And we have customers with exactly this scenario, by the way,
Cue 280 — 00:20:15,531 --> 00:20:20,411
which is let's say that you have a public repo and a private repo,
Cue 281 — 00:20:20,451 --> 00:20:25,611
and they're paired. So the public repo is the place where your customers submit
Cue 282 — 00:20:25,651 --> 00:20:29,191
feature request. They submit bug reports.
Cue 283 — 00:20:29,291 --> 00:20:32,891
They sort of ask you what the what's going on with your project.
Cue 284 — 00:20:32,931 --> 00:20:35,891
And then the implementation is, is private.
Cue 285 — 00:20:35,931 --> 00:20:40,631
Okay. So there's a lot of companies who have things set up on GitHub like this.
Cue 286 — 00:20:40,931 --> 00:20:47,631
And like any public facing repo, these public repos are now inundated with
Cue 287 — 00:20:47,771 --> 00:20:49,971
coding agent and other sorts of slop.
Cue 288 — 00:20:50,011 --> 00:20:51,691
Like you can't handle all the load.
Cue 289 — 00:20:51,731 --> 00:20:55,411
So you might think, well, what I really want to do is I want to have an agent that
Cue 290 — 00:20:55,451 --> 00:21:00,531
can handle these issues for me, instead of forcing my developers to handle them.
Cue 291 — 00:21:00,571 --> 00:21:04,631
Okay. And you'll you'll see what the problem is here.
Cue 292 — 00:21:04,731 --> 00:21:07,171
All right. So the agent picks up the issue.
Cue 293 — 00:21:07,211 --> 00:21:11,411
You'll notice the issue is asking about some library dependency in the
Cue 294 — 00:21:11,451 --> 00:21:16,931
implementation. The agent then, because it has read access to the private repo goes
Cue 295 — 00:21:16,971 --> 00:21:19,251
off and reads the private repo.
Cue 296 — 00:21:19,291 --> 00:21:24,251
The private repo does have the answer to the question, but it's got a ton of other
Cue 297 — 00:21:24,291 --> 00:21:30,471
stuff, right? So I don't know if somebody wrote the launch data Project Orchid.
Cue 298 — 00:21:30,651 --> 00:21:32,511
I don't know what Project Orchid is.
Cue 299 — 00:21:33,691 --> 00:21:37,531
Wrote that into the repo, and now the coding agent has internalized that in its
Cue 300 — 00:21:37,571 --> 00:21:39,971
context. It's just trying to be super helpful.
Cue 301 — 00:21:40,011 --> 00:21:43,411
So it closes out the issue and writes a comment and says, yes,
Cue 302 — 00:21:43,451 --> 00:21:46,691
version two is compatible with your integration, by the way.
Cue 303 — 00:21:46,731 --> 00:21:49,911
It will support Project Orchid as of June 12th.
Cue 304 — 00:21:50,131 --> 00:21:53,011
And I think the interesting thing here, this is this is an example,
Cue 305 — 00:21:53,051 --> 00:21:54,551
right. It's not it's not real.
Cue 306 — 00:21:55,131 --> 00:21:57,771
But the the agent wasn't necessarily attacked.
Cue 307 — 00:21:57,811 --> 00:22:01,031
It's just really chatty and trying to be helpful.
Cue 308 — 00:22:01,371 --> 00:22:07,571
Right. So it can be worse if you are if the agent is being prompt injected.
Cue 309 — 00:22:07,611 --> 00:22:12,371
Right. But even just through being overly chatty over sharing,
Cue 310 — 00:22:12,411 --> 00:22:18,291
your coding agent that has access to a private repo and is now writing to a public
Cue 311 — 00:22:18,331 --> 00:22:23,631
repo can very easily inadvertently expose stuff you don't want it to expose.
Cue 312 — 00:22:24,691 --> 00:22:27,791
Okay, so what are we doing about this?
Cue 313 — 00:22:29,571 --> 00:22:31,531
We have three design principles again.
Cue 314 — 00:22:31,571 --> 00:22:37,291
The first is that we have what we call the primary agent that only deals with public
Cue 315 — 00:22:37,331 --> 00:22:41,911
data. Okay, so we sort of isolate this public agent.
Cue 316 — 00:22:42,171 --> 00:22:47,811
It only sees public data. The primary never touches private data.
Cue 317 — 00:22:47,851 --> 00:22:54,571
Okay. Instead what it can do is it can launch what we call an enclave agent that
Cue 318 — 00:22:54,611 --> 00:22:57,691
can handle interacting with the private repo.
Cue 319 — 00:22:57,731 --> 00:23:02,171
And the way that it does this, it is specifies the private repo that the agent
Cue 320 — 00:23:02,211 --> 00:23:07,811
should interact with. It gives that enclave agent a prompt, and then it also
Cue 321 — 00:23:07,851 --> 00:23:11,531
specifies a schema. That schema for the response is going to be really,
Cue 322 — 00:23:11,571 --> 00:23:16,211
really important. Okay. So the important thing here is that the primary agent never
Cue 323 — 00:23:16,251 --> 00:23:19,891
sees the private repo. Instead it launches an isolated enclave agent.
Cue 324 — 00:23:19,931 --> 00:23:24,311
The enclave agent can see the private repo and nothing else.
Cue 325 — 00:23:25,051 --> 00:23:29,051
It receives the prompt, it acts on the prompt, and when it's done with its work,
Cue 326 — 00:23:29,091 --> 00:23:35,491
it has to return. It's answer that in a way that's compliant with the schema that
Cue 327 — 00:23:35,531 --> 00:23:41,331
the primary agent created. Now you might say, okay, that's fine.
Cue 328 — 00:23:41,371 --> 00:23:46,851
It's schema compliant. That doesn't necessarily prevent it from saying something it
Cue 329 — 00:23:46,891 --> 00:23:51,691
shouldn't. And so there is a third thing here, which is we impose an information
Cue 330 — 00:23:51,731 --> 00:23:55,131
budget on that schema for the responses.
Cue 331 — 00:23:55,171 --> 00:23:58,291
And that budget is set by the user.
Cue 332 — 00:23:58,331 --> 00:24:01,651
So the user defines a budget. That could be eight bits.
Cue 333 — 00:24:01,691 --> 00:24:05,471
It could be 64 bits. It can be whatever you want it to be.
Cue 334 — 00:24:06,251 --> 00:24:10,871
And then we have a trusted broker that makes sure that the responses coming back
Cue 335 — 00:24:10,971 --> 00:24:16,131
from the enclave to the primary stay within that information budget.
Cue 336 — 00:24:16,171 --> 00:24:22,611
So, for example, if you want your enclave to only answer some number of yes or no
Cue 337 — 00:24:22,651 --> 00:24:28,811
questions and not leak the date of when a feature ships, then you would restrict
Cue 338 — 00:24:28,851 --> 00:24:30,851
that budget and then we would enforce it.
Cue 339 — 00:24:30,891 --> 00:24:33,771
The actual answers stay within it.
Cue 340 — 00:24:33,811 --> 00:24:37,851
And I think the important thing here, well, let me go through an example to show you
Cue 341 — 00:24:37,891 --> 00:24:40,311
what, what this looks like in practice.
Cue 342 — 00:24:40,811 --> 00:24:45,811
Okay. So now in our workflow we have two agents.
Cue 343 — 00:24:45,851 --> 00:24:49,091
We've got the primary agent which was the one that was running before.
Cue 344 — 00:24:49,131 --> 00:24:52,011
And we've got this new isolated enclave agent.
Cue 345 — 00:24:52,051 --> 00:24:56,911
We've got the same to repos the public one and the private one.
Cue 346 — 00:24:57,371 --> 00:25:03,051
Okay. So we get our our public issue that is asking about compatibility.
Cue 347 — 00:25:03,091 --> 00:25:08,651
The primary agent starts to work on that, realizes that in order to answer it,
Cue 348 — 00:25:08,691 --> 00:25:11,771
it needs access to the private repo.
Cue 349 — 00:25:11,811 --> 00:25:14,531
It also knows it can't access the private repo.
Cue 350 — 00:25:14,571 --> 00:25:19,691
So it instantiates an enclave agent that will then handle all of the interactions
Cue 351 — 00:25:19,731 --> 00:25:21,151
with the private repo.
Cue 352 — 00:25:23,051 --> 00:25:26,871
The enclave agent then interacts with the private repo.
Cue 353 — 00:25:26,931 --> 00:25:33,531
It checks for compatibility, and then it creates a response back to the primary
Cue 354 — 00:25:33,571 --> 00:25:39,531
agent. And the response, as I mentioned, has to be compliant with the schema and the
Cue 355 — 00:25:39,571 --> 00:25:40,811
channel width you could think of.
Cue 356 — 00:25:40,851 --> 00:25:47,091
The information width of that schema has to be compliant with the user's budget
Cue 357 — 00:25:47,131 --> 00:25:49,291
information budget that it assigns.
Cue 358 — 00:25:49,331 --> 00:25:54,451
Okay. So it's nice is that the primary agent now gets the answer to its simple
Cue 359 — 00:25:54,491 --> 00:26:01,331
question and nothing more. And then it composes a reply publicly and say,
Cue 360 — 00:26:01,371 --> 00:26:02,911
hey, everything is fine.
Cue 361 — 00:26:04,691 --> 00:26:10,331
Compatibility, version two is compatible and it doesn't say anything about anything
Cue 362 — 00:26:10,371 --> 00:26:14,191
else happening in the in the private repo.
Cue 363 — 00:26:14,491 --> 00:26:21,091
Okay. So one of the key things here is, you know, what is this information budget
Cue 364 — 00:26:21,131 --> 00:26:28,031
stuff. So we have an enclave information budget hierarchy on the extreme is
Cue 365 — 00:26:28,931 --> 00:26:33,931
no limits. In other words, you can provide as much information as you want.
Cue 366 — 00:26:33,971 --> 00:26:39,011
We're just going to require that you put it inside of a schema on the extreme
Cue 367 — 00:26:39,051 --> 00:26:44,171
trusted part. You can use open ended strings in the public part.
Cue 368 — 00:26:44,211 --> 00:26:46,251
You're not allowed to use open ended strings.
Cue 369 — 00:26:46,291 --> 00:26:51,171
You have to restrict the schema to enums and bools and small numbers,
Cue 370 — 00:26:51,211 --> 00:26:52,931
basically. But there's no budget.
Cue 371 — 00:26:52,971 --> 00:26:58,631
You can pack and encode as much information in bools and enums as you want.
Cue 372 — 00:26:58,731 --> 00:27:05,491
Okay, now kind of like in the middle are these, these other
Cue 373 — 00:27:05,531 --> 00:27:08,171
two levels, which we call internal and confidential.
Cue 374 — 00:27:08,211 --> 00:27:11,951
Here again, we're imposing limits on what the schema can be.
Cue 375 — 00:27:12,091 --> 00:27:13,971
There's no open ended strings.
Cue 376 — 00:27:14,011 --> 00:27:16,611
It has to be bools and enums and stuff like that.
Cue 377 — 00:27:16,651 --> 00:27:22,451
But we have a stricter bit width on that, a budget on that, on the communication.
Cue 378 — 00:27:22,491 --> 00:27:25,371
So internal is 64 bits confidential is eight bits.
Cue 379 — 00:27:25,411 --> 00:27:28,451
And then the other extreme is just completely sealed.
Cue 380 — 00:27:28,491 --> 00:27:29,491
You can't really say anything.
Cue 381 — 00:27:29,531 --> 00:27:35,171
So this is like fire and forget enclave go off and do something and then I won't
Cue 382 — 00:27:35,211 --> 00:27:37,131
learn anything after you're done.
Cue 383 — 00:27:37,171 --> 00:27:42,251
Okay. So the important thing here is that you as the workflow author,
Cue 384 — 00:27:42,291 --> 00:27:46,171
you set the information budget somewhere in this hierarchy.
Cue 385 — 00:27:46,211 --> 00:27:51,611
You set the information budget, and then the primary agent with that budget gets to
Cue 386 — 00:27:51,651 --> 00:27:56,451
choose the semantics of the schema so it can say what each of those enums within the
Cue 387 — 00:27:56,491 --> 00:28:00,531
budget mean, because it knows best it's trying to perform some work.
Cue 388 — 00:28:00,571 --> 00:28:02,551
We're not going to decide that in advance.
Cue 389 — 00:28:02,891 --> 00:28:08,311
But we are going to enforce an information budget on on that schema.
Cue 390 — 00:28:09,211 --> 00:28:13,171
Okay. All right. So all of this is is pretty new.
Cue 391 — 00:28:13,211 --> 00:28:17,231
It went out. It went out with GA last week.
Cue 392 — 00:28:17,971 --> 00:28:20,811
I put up a blog post last night talking about all this stuff.
Cue 393 — 00:28:20,851 --> 00:28:23,251
If you want to dive in. Feel free to reach out.
Cue 394 — 00:28:23,291 --> 00:28:28,051
You know, if you have questions or feedback on on this stuff I think are a really
Cue 395 — 00:28:28,091 --> 00:28:33,851
exciting security primitive, especially as you move up to cross repo large scale
Cue 396 — 00:28:33,891 --> 00:28:34,991
workflows.
.tessl-plugin
skills
talk-arcolano-autonomy-and-trust
talk-aroussi-decision-context
talk-bevin-agent-assisted-instrument-design
talk-boeckeler-context-lifecycles
talk-braun-context-graphs
talk-carmont-evolving-agent-harnesses
talk-chanezon-gordon-agent-experience-evaluation
talk-cox-secure-agentic-workflows
talk-cunniffe-reusable-agent-platform
talk-davis-agent-tool-policies
talk-debois-concurrent-delivery
talk-dzialets-team-automation
talk-fahmy-harness-api-design
talk-finster-continuous-delivery
talk-fowler-regenerative-software
talk-gavrilescu-factory-architecture
talk-graciolli-amin-verification-aware-programming
talk-gruenbaum-reverse-graph-engineering
talk-hassan-agentic-engineering
talk-hossain-enterprise-agent-adoption
talk-huntley-patch-ready-factories
talk-jain-verifying-intent
talk-katsioloudes-montei-agentic-lifecycle
talk-knox-harness-engineering
talk-kovac-deterministic-memory
talk-lee-perkins-code-readiness
talk-lorenc-machine-speed-security
talk-lubken-software-of-one
talk-meijer-software-specialization
talk-meppiel-versioned-agent-contracts
talk-moss-concurrent-agent-risk
talk-myers-understanding-agent-generated-code
talk-ocytko-scaling-agentic-engineering
talk-oza-automating-design-quality
talk-panzica-agentic-sdlc
talk-pari-patel-agent-awareness
talk-podjarny-trustworthy-software-factories
talk-poulter-factory-improvement
talk-pratik-patel-refined-rag-data
talk-re-ferre-development-platforms
talk-sadogursky-factory-of-factories
talk-shah-training-agent-skills
talk-sharma-mohapatra-evolving-intent
talk-singhal-operating-agents-at-scale
talk-stack-incident-dossiers
talk-tyler-team-coding-benchmarks
talk-wotherspoon-agent-execution-environments
talk-yegge-ai-powered-factories
talk-zuber-leadership-through-coding