AI Native DevCon 2026 New York — practical agent skills grounded in conference talks
86%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Geoffrey Huntley · AI Native DevCon New York 2026
Recording: https://www.youtube.com/watch?v=2aV2XYo8GVE
Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.
For topic navigation, see outline.md.
Cue 1 — 00:00:00,460 --> 00:00:02,800
Hey, everyone. Thanks for joining me here today.
Cue 2 — 00:00:03,340 --> 00:00:06,900
Rather than talking about AI and all the rest, I'm just going to talk about some
Cue 3 — 00:00:06,940 --> 00:00:12,020
engineering because deeply, I believe the way to solve some of the problems we have
Cue 4 — 00:00:12,060 --> 00:00:13,760
is actually through engineering.
Cue 5 — 00:00:13,940 --> 00:00:17,600
And I'm going to put forward to you some ideas.
Cue 6 — 00:00:18,700 --> 00:00:22,180
Today and tomorrow you're going to be seeing a whole bunch of talks about how do you
Cue 7 — 00:00:22,220 --> 00:00:24,900
build software faster with factories and all the rest.
Cue 8 — 00:00:24,940 --> 00:00:25,740
Cool.
Cue 9 — 00:00:27,020 --> 00:00:32,200
Maybe if that's the wrong thing, how do you do security incident response faster,
Cue 10 — 00:00:33,340 --> 00:00:37,820
right. And it turns out if you optimize for your security incident response times,
Cue 11 — 00:00:37,860 --> 00:00:40,160
it also accelerates your build times.
Cue 12 — 00:00:40,580 --> 00:00:43,520
So I'm going to show you how I do some things.
Cue 13 — 00:00:44,180 --> 00:00:47,320
So two weeks ago it was pretty wild right?
Cue 14 — 00:00:48,660 --> 00:00:55,360
It was pretty cool. Folks like wow, you mean open code got breached.
Cue 15 — 00:00:55,980 --> 00:00:59,220
Then the Prime Minister of Japan was just going, hey everyone,
Cue 16 — 00:00:59,260 --> 00:01:00,600
you should update PHP.
Cue 17 — 00:01:03,140 --> 00:01:08,340
Next thing you know, people were like, okay, there's a vulnerability in HTTP stack
Cue 18 — 00:01:08,380 --> 00:01:12,960
that allows you to turn off a computer that's running.
Cue 19 — 00:01:14,140 --> 00:01:20,940
And then we got into a 12 year old, a 12 year old security exploit in
Cue 20 — 00:01:20,980 --> 00:01:26,100
GitLab that led to a compromise to and remote code execution.
Cue 21 — 00:01:26,140 --> 00:01:33,400
Ouch. It turns out the the answer to life in the universe is not 42.
Cue 22 — 00:01:34,060 --> 00:01:39,844
It is four. It is four. The magic number that unlocks all the encryption on the
Cue 23 — 00:01:39,884 --> 00:01:46,240
internet is four. If you don't know what happened, a a certificate authority,
Cue 24 — 00:01:47,380 --> 00:01:54,340
was incompetent and they put their master password, the
Cue 25 — 00:01:54,380 --> 00:01:59,800
master key, as four. And you could sign any other SSL with that if you knew the
Cue 26 — 00:01:59,840 --> 00:02:02,480
number four. Four opens all doors on the internet.
Cue 27 — 00:02:03,100 --> 00:02:06,720
So we started wondering, gosh, is it time to turn off the internet yet?
Cue 28 — 00:02:08,580 --> 00:02:12,980
And if you're kind of wondering, like, this is what we're looking at,
Cue 29 — 00:02:13,020 --> 00:02:18,900
folks, this is what we're looking at, you might have seen the GitHub quadratic graph
Cue 30 — 00:02:18,940 --> 00:02:23,420
of like everyone raising pull requests and GitHub having some availability issues
Cue 31 — 00:02:23,460 --> 00:02:25,760
because of it. Because the agent take off.
Cue 32 — 00:02:27,020 --> 00:02:29,280
This is security take off folks.
Cue 33 — 00:02:30,260 --> 00:02:35,720
And I'm putting forward is this is a start of the new normal.
Cue 34 — 00:02:36,820 --> 00:02:39,560
This is the start of the new normal that.
Cue 35 — 00:02:41,860 --> 00:02:45,580
Is normal now. Now it could be it could be debated.
Cue 36 — 00:02:45,620 --> 00:02:50,420
Maybe this is just a small bump in the line of the road until,
Cue 37 — 00:02:50,460 --> 00:02:55,000
like, all the software has been patched, maybe they will go down.
Cue 38 — 00:02:55,140 --> 00:02:58,520
Maybe they'll go down. But agents are writing software now.
Cue 39 — 00:02:59,060 --> 00:03:01,940
And agents have been trained on existing code, and existing code will have these
Cue 40 — 00:03:01,980 --> 00:03:06,960
security vulnerabilities. So argue that this is the start of the new normal.
Cue 41 — 00:03:08,820 --> 00:03:13,520
And unfortunately cyber is only going to accelerate from this point forward out.
Cue 42 — 00:03:13,900 --> 00:03:19,340
Like it's going to be really weird, folks, because like, I suspect you're going to
Cue 43 — 00:03:19,380 --> 00:03:25,080
need more inferencing capacity than your attackers to remain secure.
Cue 44 — 00:03:26,220 --> 00:03:30,140
Figure about that. Think about the companies right now and they're thinking,
Cue 45 — 00:03:30,180 --> 00:03:35,520
oh no, we can only do like $100 a month token budget per employee.
Cue 46 — 00:03:37,340 --> 00:03:41,180
Yeah. No. Yeah. Not so. This is one of the things I think it's going to be really,
Cue 47 — 00:03:41,220 --> 00:03:46,000
really different is as soon as you publish your any code on GitHub,
Cue 48 — 00:03:46,140 --> 00:03:51,820
the end result. Net result is some person with racks of inferencing hardware is
Cue 49 — 00:03:51,860 --> 00:03:56,420
going to be running autonomously scanning natural response and then autonomously
Cue 50 — 00:03:56,460 --> 00:03:59,080
tool calling to actually exploit that exploit.
Cue 51 — 00:04:00,060 --> 00:04:01,460
This is the new state of the normal.
Cue 52 — 00:04:01,500 --> 00:04:08,260
It's getting weird. And I guess I wrote this almost a year ago now it's a
Cue 53 — 00:04:08,300 --> 00:04:12,660
blog post called libraries. And according to the question, why do we even have open
Cue 54 — 00:04:12,700 --> 00:04:17,500
source libraries? And I got grilled to hell on Hacker News when I published this.
Cue 55 — 00:04:17,540 --> 00:04:20,360
But every day that goes by, it's become even more true.
Cue 56 — 00:04:21,060 --> 00:04:27,080
I'll explain. The reason we had open source libraries was because code was precious.
Cue 57 — 00:04:27,180 --> 00:04:33,780
It was hard to write, so we decided to concentrate into libraries to make reuse
Cue 58 — 00:04:33,820 --> 00:04:38,500
easy. And we also did this to make hiring easier, right?
Cue 59 — 00:04:38,540 --> 00:04:44,100
React famously by Facebook was kind of like of a play to standardize,
Cue 60 — 00:04:44,140 --> 00:04:47,080
making it easy for them to hire frontend developers.
Cue 61 — 00:04:47,340 --> 00:04:52,280
So we've done all these things in our practices to optimize for humans,
Cue 62 — 00:04:52,900 --> 00:04:57,880
and I don't think that we need libraries anymore.
Cue 63 — 00:04:58,380 --> 00:05:03,820
Now, I'm not saying that we should go out there and completely like vibe code up
Cue 64 — 00:05:03,860 --> 00:05:08,140
some cryptography. If you're a cryptographer, go do it, please.
Cue 65 — 00:05:08,180 --> 00:05:12,760
But I'm not. So there is going to be like a trusted base and a trusted core.
Cue 66 — 00:05:13,300 --> 00:05:18,180
But you think about think about your supply chain.
Cue 67 — 00:05:18,220 --> 00:05:21,960
How much time do you spend on Dependabot updates?
Cue 68 — 00:05:23,900 --> 00:05:27,460
Cool. How much time do you spend doing security incident response?
Cue 69 — 00:05:27,500 --> 00:05:31,500
Because the Dependabot update has been compromised because the upstream author is
Cue 70 — 00:05:31,540 --> 00:05:37,220
being compromised. Right. So all the all already we're in these negative time
Cue 71 — 00:05:37,260 --> 00:05:43,100
savings just from the from the what we do because we consume these libraries because
Cue 72 — 00:05:43,140 --> 00:05:47,440
the prior was the prior was writing software was hard.
Cue 73 — 00:05:47,780 --> 00:05:53,980
So it's been a year now. I don't use much open source anymore.
Cue 74 — 00:05:54,020 --> 00:05:59,220
Sure, I'll use the Linux kernel and like DNS servers and BIND and all the lower
Cue 75 — 00:05:59,260 --> 00:06:05,140
level stuff. But as I start getting into high level stuff, I just prompt the agent
Cue 76 — 00:06:05,180 --> 00:06:06,400
for what I want.
Cue 77 — 00:06:07,620 --> 00:06:10,260
You have to find what makes you comfortable.
Cue 78 — 00:06:10,300 --> 00:06:14,140
It will feel very unnatural at first when you first start doing it.
Cue 79 — 00:06:14,180 --> 00:06:15,720
But here's the thing.
Cue 80 — 00:06:17,140 --> 00:06:22,500
If it's like well and like a small and well specified, maybe like a,
Cue 81 — 00:06:22,540 --> 00:06:26,580
like a, an argument parser on a command line application, sure,
Cue 82 — 00:06:26,620 --> 00:06:31,340
you could take that dependency, but you could just like ask Claude and run in a loop
Cue 83 — 00:06:31,380 --> 00:06:33,340
to build that command line parser.
Cue 84 — 00:06:33,380 --> 00:06:36,000
And something magical happens when you do that.
Cue 85 — 00:06:36,260 --> 00:06:40,500
It's built to your use case and your needs, and you don't need to interface with
Cue 86 — 00:06:40,540 --> 00:06:45,260
someone on GitHub asking them to take your pull request a merge of what else have
Cue 87 — 00:06:45,300 --> 00:06:49,960
you. It turns out if you have to tool call a human, that's not AGI.
Cue 88 — 00:06:51,700 --> 00:06:58,700
So the best thing you can really do is either vendor that open source software and
Cue 89 — 00:06:58,740 --> 00:07:00,600
on your file system check it in.
Cue 90 — 00:07:00,740 --> 00:07:04,960
And that way the agent can modify it and extend it to its needs.
Cue 91 — 00:07:05,180 --> 00:07:08,800
That's one approach. I'm going to show you a different approach today.
Cue 92 — 00:07:10,220 --> 00:07:13,240
And that's something that I'm calling like Patch the Planet.
Cue 93 — 00:07:13,860 --> 00:07:20,320
You see, patching is perceived to be hard by people, but it doesn't have to be.
Cue 94 — 00:07:20,380 --> 00:07:23,500
So one of the things that makes patching software hard is like,
Cue 95 — 00:07:23,540 --> 00:07:26,540
let's say you wanted to patch something on GitHub, you know,
Cue 96 — 00:07:26,580 --> 00:07:31,940
and you got to figure out how that piece of software actually builds and that
Cue 97 — 00:07:31,980 --> 00:07:36,100
toolchain and oh my God, I'm going to have a new CI job just for that one thing.
Cue 98 — 00:07:36,140 --> 00:07:37,940
And you do it on a per language basis.
Cue 99 — 00:07:37,980 --> 00:07:40,940
And each open source library has a different build toolchain.
Cue 100 — 00:07:40,980 --> 00:07:43,900
So this just gets really, really, really hard.
Cue 101 — 00:07:43,940 --> 00:07:47,280
It doesn't have to be folks. Does not have to be.
Cue 102 — 00:07:48,620 --> 00:07:52,040
And then let's say you've actually patched it, you're like, God.
Cue 103 — 00:07:53,980 --> 00:07:56,680
Oh, I guess I'm going to host that in an artifact repo now.
Cue 104 — 00:07:57,100 --> 00:08:00,380
And now going to think about artifact tokens and all this complexity.
Cue 105 — 00:08:00,420 --> 00:08:02,340
We're making things needlessly complex.
Cue 106 — 00:08:02,380 --> 00:08:07,180
There's been solutions. Today I'm going to introduce you to something called Nix.
Cue 107 — 00:08:07,220 --> 00:08:09,260
Nix has been around since 2003.
Cue 108 — 00:08:09,300 --> 00:08:13,620
I've been using it for years. You might hear that Nix sucks.
Cue 109 — 00:08:13,660 --> 00:08:14,720
It does.
Cue 110 — 00:08:16,770 --> 00:08:18,480
Like there is no silver bullet.
Cue 111 — 00:08:18,740 --> 00:08:23,840
But the thing that you most commonly hear about Nix is it's hard to use.
Cue 112 — 00:08:24,350 --> 00:08:26,920
It is where it was,
Cue 113 — 00:08:28,420 --> 00:08:35,200
right? You see, the agents know how to drive a drive, an outcome.
Cue 114 — 00:08:35,620 --> 00:08:38,060
If you want to use Nix, you just need to go to a.
Cue 115 — 00:08:38,100 --> 00:08:41,860
You just need to know that Nix exists and you can actually prompt an agent.
Cue 116 — 00:08:41,900 --> 00:08:44,760
Hey, I want Nix here and you get Nix.
Cue 117 — 00:08:45,180 --> 00:08:47,020
You can do that with Bazel and Buck2.
Cue 118 — 00:08:47,060 --> 00:08:50,460
Now those those two technologies also suck, you know.
Cue 119 — 00:08:50,500 --> 00:08:52,620
And like you speak with any eng leader.
Cue 120 — 00:08:52,660 --> 00:08:57,500
They got a story where they almost got fired because they staffed a team of five to
Cue 121 — 00:08:57,540 --> 00:08:58,980
introduce some of these technologies.
Cue 122 — 00:08:59,020 --> 00:09:01,640
And it took a long time to upskill all the developers.
Cue 123 — 00:09:01,940 --> 00:09:08,740
This stuff is no longer true. You can now use a highly complex tools simply
Cue 124 — 00:09:08,780 --> 00:09:11,260
that are high powered. It has got to know about it, and you've got to do it,
Cue 125 — 00:09:11,300 --> 00:09:14,240
and you've got to falsify your priors.
Cue 126 — 00:09:14,580 --> 00:09:19,120
So the language sucks, I don't care, I just want the outcome.
Cue 127 — 00:09:19,220 --> 00:09:23,380
So let's get into the outcome up here on my GitHub.
Cue 128 — 00:09:23,420 --> 00:09:29,820
It's a thing called nix demo. After this, clone it down and have a look.
Cue 129 — 00:09:29,860 --> 00:09:34,720
Have a look at it because it's got some pretty interesting things in there.
Cue 130 — 00:09:36,220 --> 00:09:38,092
I use something called devenv.
Cue 131 — 00:09:38,132 --> 00:09:44,160
Devenv is just like kind of like a nice introduction to Nix.
Cue 132 — 00:09:44,620 --> 00:09:49,132
And what you do is you define your packages, your services that you did and what
Cue 133 — 00:09:49,172 --> 00:09:55,640
have you in the Nix. And that one file can generate out your developer laptops,
Cue 134 — 00:09:56,460 --> 00:09:57,720
your CI/CD,
Cue 135 — 00:09:59,220 --> 00:10:03,880
your agent sandboxes, and your production Docker images.
Cue 136 — 00:10:04,980 --> 00:10:07,060
You see what I'm going here? We've got a single source of truth,
Cue 137 — 00:10:07,100 --> 00:10:10,860
folks. How many people right now are spinning up teams to figure out what they're
Cue 138 — 00:10:10,900 --> 00:10:12,920
going to do about agent sandboxes?
Cue 139 — 00:10:13,420 --> 00:10:16,980
And are they are they just duplicating environments?
Cue 140 — 00:10:17,020 --> 00:10:20,980
Because I bet your bottom dollar right now in some of your companies,
Cue 141 — 00:10:21,020 --> 00:10:25,460
your laptop, you know, and it's not reproducible like you're onboard a new employee
Cue 142 — 00:10:25,500 --> 00:10:29,740
or you lose a laptop that's like 48 hours to set up your developer environment
Cue 143 — 00:10:29,780 --> 00:10:33,700
again. CI/CD, there's drift. So it works on your computer.
Cue 144 — 00:10:33,740 --> 00:10:37,160
But like CI/CD says it doesn't.
Cue 145 — 00:10:37,260 --> 00:10:40,060
Next, you know, you got another source of drift the agent sandboxes.
Cue 146 — 00:10:40,100 --> 00:10:45,300
And then, of course, you say it's all working and done, but the doesn't work on the
Cue 147 — 00:10:45,340 --> 00:10:49,680
production Docker image I'm getting at is there's a solution here.
Cue 148 — 00:10:52,220 --> 00:10:56,540
And interesting enough Nix actually can produce would it produces these Docker
Cue 149 — 00:10:56,580 --> 00:11:01,740
images. They're incredibly efficient because the way how Docker works is it
Cue 150 — 00:11:01,780 --> 00:11:05,480
basically stacks file system layers on top of each other and squashes them.
Cue 151 — 00:11:05,740 --> 00:11:10,600
Nix is a graph, and it's able to use that graph to build a very optimized image.
Cue 152 — 00:11:10,940 --> 00:11:11,740
Cool.
Cue 153 — 00:11:13,060 --> 00:11:17,520
I'm going to quickly come across here and show you some cool things.
Cue 154 — 00:11:18,580 --> 00:11:22,600
So I want to show you what version of git here.
Cue 155 — 00:11:23,300 --> 00:11:27,100
So I'm running 254. I'm going to use some magic.
Cue 156 — 00:11:27,140 --> 00:11:28,540
Imagine that I'm a software engineer.
Cue 157 — 00:11:28,580 --> 00:11:30,320
I just got a brand new computer
Cue 158 — 00:11:31,980 --> 00:11:35,480
and I want to onboard into the company.
Cue 159 — 00:11:39,300 --> 00:11:45,626
Onboarded. If you think about the things like, like NVM or like
Cue 160 — 00:11:45,666 --> 00:11:51,140
Python version manager, etc., it's this for all your operating system dependencies,
Cue 161 — 00:11:51,180 --> 00:11:55,440
including the FFI boundaries. It goes past the programming language.
Cue 162 — 00:11:55,940 --> 00:11:58,800
And let's have a look at the version of git in here
Cue 163 — 00:12:00,340 --> 00:12:05,140
Cue 164 — 00:12:05,180 --> 00:12:11,080
It's not just a higher version of git on GitHub in the GitHub repo has the answer.
Cue 165 — 00:12:25,500 --> 00:12:30,000
This version of git does not support force push.
Cue 166 — 00:12:32,140 --> 00:12:35,820
You see, what I've done is I've monkey patched for my agent,
Cue 167 — 00:12:35,860 --> 00:12:41,140
sandbox, my local engineering laptops, and wherever I use this git and I've got a
Cue 168 — 00:12:41,180 --> 00:12:46,260
single source of git and anything else that needs git as a as a transitory
Cue 169 — 00:12:46,300 --> 00:12:49,840
dependency also gets my monkey patch version of git.
Cue 170 — 00:12:50,020 --> 00:12:52,720
So how do you stop an agent from doing force push?
Cue 171 — 00:12:52,780 --> 00:12:55,900
Because you know it hill climbs and it does murderous things.
Cue 172 — 00:12:55,940 --> 00:13:00,140
When it hill climbs and causes incidents, you move that capability from the
Cue 173 — 00:13:00,180 --> 00:13:03,980
underlying thing. So I do these type of tricks all over the place.
Cue 174 — 00:13:04,020 --> 00:13:07,500
The best way to stop a model from being able to do something is remove the
Cue 175 — 00:13:07,540 --> 00:13:11,500
capability for it to be able to do it, and sometimes the best way is just to monkey
Cue 176 — 00:13:11,540 --> 00:13:15,480
patch the underlying underlying system to be able to support these behaviors.
Cue 177 — 00:13:15,620 --> 00:13:19,240
So yeah, let's have a look how that works.
Cue 178 — 00:13:25,940 --> 00:13:26,740
Cool.
Cue 179 — 00:13:28,700 --> 00:13:31,280
So it's up here in GitHub, nix-demo.
Cue 180 — 00:13:31,340 --> 00:13:32,720
And it's really simple.
Cue 181 — 00:13:34,300 --> 00:13:38,900
For the older older people in the room of my generation, you probably remember patch
Cue 182 — 00:13:38,940 --> 00:13:44,540
files. Open source used to be distributed as a collection of patch files.
Cue 183 — 00:13:44,580 --> 00:13:50,460
Used to be a contributor. All this, like everything, must be merged upstream on
Cue 184 — 00:13:50,500 --> 00:13:54,680
GitHub. That is, that was never a specified in open source.
Cue 185 — 00:13:55,140 --> 00:13:58,640
And like we used this is how we used to do software development.
Cue 186 — 00:13:58,900 --> 00:14:04,600
So what happens here is I have a closure that defines git.
Cue 187 — 00:14:04,660 --> 00:14:06,840
And then I just had my custom patches.
Cue 188 — 00:14:07,260 --> 00:14:08,820
How did I get the custom patches?
Cue 189 — 00:14:08,860 --> 00:14:13,780
I prompted Claude, can you please use Nix and create me a custom overlay that
Cue 190 — 00:14:13,820 --> 00:14:20,620
removes force? Push. Done. Next thing you know, I now have a whole
Cue 191 — 00:14:20,660 --> 00:14:25,980
bunch of Unix patches that automatically applied to the git when I'm in that
Cue 192 — 00:14:26,020 --> 00:14:29,900
operating environment. But it's not just that environment.
Cue 193 — 00:14:29,940 --> 00:14:32,220
It could be a Docker, it could be production host.
Cue 194 — 00:14:32,260 --> 00:14:37,980
What does have you? This one little thing, this one little pattern turns everything
Cue 195 — 00:14:38,020 --> 00:14:40,080
in the entire world patchable.
Cue 196 — 00:14:40,700 --> 00:14:47,120
No longer are you waiting for someone upstream to merge a bug fix.
Cue 197 — 00:14:47,290 --> 00:14:50,000
You just route around it, just route around it.
Cue 198 — 00:14:50,100 --> 00:14:55,380
What happens is all software becomes infinitely patched, and that is the behavior
Cue 199 — 00:14:55,420 --> 00:14:58,820
that you need to for security incident response.
Cue 200 — 00:14:58,860 --> 00:15:01,420
It's also a behavior. You also need your engineering.
Cue 201 — 00:15:01,460 --> 00:15:03,820
How many times you try to figure out what you're going to do,
Cue 202 — 00:15:03,860 --> 00:15:07,380
you take a fork, something's wrong or you want to add this capability.
Cue 203 — 00:15:07,420 --> 00:15:12,300
So there's two paths. You either yourself, you generate a first hand and minimize
Cue 204 — 00:15:12,340 --> 00:15:16,520
the dependency. But if you can't minimize the dependency you can use Nix to monkey
Cue 205 — 00:15:16,560 --> 00:15:19,280
patch the world. Really cool.
Cue 206 — 00:15:20,620 --> 00:15:23,980
And you're probably wondering, there's all sorts of little magic somewhere.
Cue 207 — 00:15:24,020 --> 00:15:27,500
I'm hiding some skeletons in the closet in this regard.
Cue 208 — 00:15:27,540 --> 00:15:31,780
No, it's quite literally in here.
Cue 209 — 00:15:31,820 --> 00:15:35,720
I import the overlay and Nix handles it all.
Cue 210 — 00:15:37,300 --> 00:15:41,988
Like mise. You might think mise is good software.
Cue 211 — 00:15:42,028 --> 00:15:47,020
It's not. Try this because mise This is just a more like a package manager like
Cue 212 — 00:15:47,060 --> 00:15:50,620
homebrew or etc. etc.. Does it have this level functionality?
Cue 213 — 00:15:50,660 --> 00:15:54,560
It allows you to patch anything, including the kernel of the operating system,
Cue 214 — 00:15:55,180 --> 00:15:58,680
and then deploy that custom version across your fleet,
Cue 215 — 00:15:59,940 --> 00:16:02,240
because Nix does and Nix can.
Cue 216 — 00:16:05,900 --> 00:16:06,880
Because.
Cue 217 — 00:16:11,300 --> 00:16:14,540
When I wrote this blog post, I guess this is like a month ago,
Cue 218 — 00:16:14,580 --> 00:16:18,900
I talked about how OpenSSL, if there's a critical fault in this,
Cue 219 — 00:16:18,940 --> 00:16:21,420
is before what happened two weeks ago.
Cue 220 — 00:16:21,460 --> 00:16:26,180
And this is the key. You think about something like OpenSSL.
Cue 221 — 00:16:26,220 --> 00:16:29,340
You could update OpenSSL. That's cool.
Cue 222 — 00:16:29,380 --> 00:16:32,920
But what about when it's a transitive dependency.
Cue 223 — 00:16:33,940 --> 00:16:37,900
So OpenSSL is used in Nginx and Postgres.
Cue 224 — 00:16:37,940 --> 00:16:42,480
So it's not just a matter of like patching the dependency of like OpenSSL.
Cue 225 — 00:16:42,960 --> 00:16:44,460
If you wanted to manually patch.
Cue 226 — 00:16:44,500 --> 00:16:48,807
This is why we never did it. But it turns out if you have a single source of truth
Cue 227 — 00:16:48,847 --> 00:16:53,180
for OpenSSL, and you update that monkey patch similar to I did with git,
Cue 228 — 00:16:53,220 --> 00:16:58,820
the end result it will recompile the world and any transitory dependencies will get
Cue 229 — 00:16:58,860 --> 00:17:02,040
that custom version. The world is infinitely.
Cue 230 — 00:17:02,940 --> 00:17:08,340
Now Nix has some pretty cool caching, and it has remote builders similar to Bazel
Cue 231 — 00:17:08,380 --> 00:17:13,660
and and so it's not a matter of like, oh, it's going to compile.
Cue 232 — 00:17:13,700 --> 00:17:17,740
It's going to take forever. No, your CI when it validates.
Cue 233 — 00:17:17,780 --> 00:17:20,100
It's just going to compile the software and it publishes the cache.
Cue 234 — 00:17:20,140 --> 00:17:24,320
And your developer environment is sandboxes consume in that case simple.
Cue 235 — 00:17:24,380 --> 00:17:28,260
So I guess the really the question I want to put to you is how much productivity did
Cue 236 — 00:17:28,300 --> 00:17:29,840
you lose two weeks ago
Cue 237 — 00:17:31,180 --> 00:17:35,360
during the security incidents that happened because.
Cue 238 — 00:17:37,540 --> 00:17:39,920
I didn't lose any sleep, folks,
Cue 239 — 00:17:41,260 --> 00:17:44,920
day to day, I'm a field CTO at a company called Antithesis.
Cue 240 — 00:17:45,820 --> 00:17:50,300
We do a lot in testing. Come pick my brain at lunchtime and I can show you some
Cue 241 — 00:17:50,340 --> 00:17:53,740
really cool stuff. I literally just went into Claude and said,
Cue 242 — 00:17:53,780 --> 00:17:54,580
hey,
Cue 243 — 00:17:55,940 --> 00:18:00,120
because I got the single source of truth is my fleet affected?
Cue 244 — 00:18:01,060 --> 00:18:07,820
And it goes. Yeah. And from that point forward, it started monkey patching the what
Cue 245 — 00:18:07,860 --> 00:18:10,900
it needed. It updated the refs and the dependencies and like what,
Cue 246 — 00:18:10,940 --> 00:18:13,900
all the pins and it did the monkey patches as I need be.
Cue 247 — 00:18:13,940 --> 00:18:17,660
And because that source of truth or the transitory dependencies automatically
Cue 248 — 00:18:17,700 --> 00:18:20,680
recompiled done. I guess
Cue 249 — 00:18:22,060 --> 00:18:25,020
I'm enjoying the Nirvana life over there.
Cue 250 — 00:18:25,060 --> 00:18:27,740
You could be too. You just got to know that existed.
Cue 251 — 00:18:27,780 --> 00:18:30,540
So I hope that you take this opportunity to play with this idea,
Cue 252 — 00:18:30,580 --> 00:18:34,920
because you're you're probably over there in a root of pain.
Cue 253 — 00:18:35,220 --> 00:18:38,780
And it turns out if you optimize to get to this type of thing,
Cue 254 — 00:18:38,820 --> 00:18:41,300
your security incident response is faster.
Cue 255 — 00:18:41,340 --> 00:18:46,080
Your productivity for like factories and everything else also gets faster.
Cue 256 — 00:18:46,420 --> 00:18:49,120
It turns out the answer for AI is engineering.
Cue 257 — 00:18:49,940 --> 00:18:52,440
And there was one cute trick I did.
Cue 258 — 00:18:52,820 --> 00:18:56,700
It's a really one trick I did back over here.
Cue 259 — 00:18:56,740 --> 00:19:03,180
I said, hey, am I affected? And I had to do some temporary monkey patches to like
Cue 260 — 00:19:03,220 --> 00:19:08,440
similar to the git overlay. Well, I issued another prompt and that prompt was,
Cue 261 — 00:19:08,820 --> 00:19:14,320
hey, can you update the AGENTS.md and then like in seven days time,
Cue 262 — 00:19:14,500 --> 00:19:17,840
can you, can you like get rid of my patches?
Cue 263 — 00:19:18,660 --> 00:19:24,308
So instead of creating a Jira ticket, I just added like a reminder my AGENTS.md and
Cue 264 — 00:19:24,348 --> 00:19:28,460
the AGENTS.md when like the next developer was using or agent that picked off off
Cue 265 — 00:19:28,500 --> 00:19:34,880
that machine just automatically just did the house cleaning task kind of cool.
Cue 266 — 00:19:35,020 --> 00:19:40,100
So it automatically backed out that monkey patch role for my pins because the
Cue 267 — 00:19:40,140 --> 00:19:44,700
upstream had been fixed. And, you know, this is like the security maintenance ever
Cue 268 — 00:19:44,740 --> 00:19:48,140
done. And every time I have a security is like security type thing,
Cue 269 — 00:19:48,180 --> 00:19:52,240
I just do these two tricks. It's pretty nice.
Cue 270 — 00:19:53,260 --> 00:19:57,960
Again the repo is g, so github.com g.
Cue 271 — 00:19:58,020 --> 00:19:58,920
nix-demo.
Cue 283 — 00:20:41,620 --> 00:20:45,540
And I have ten minutes, so with do some bonus.
Cue 284 — 00:20:45,580 --> 00:20:48,160
Ponderous. I finished this security thing.
Cue 285 — 00:20:50,180 --> 00:20:51,640
Let's talk about loops.
Cue 286 — 00:20:52,900 --> 00:20:55,040
Let's talk about loops so
Cue 287 — 00:20:56,660 --> 00:21:02,860
everyone has their different ways on how to do loop engineering and and all these
Cue 288 — 00:21:02,900 --> 00:21:07,876
different things. There is no best practices in my way is very different to Steve's
Cue 289 — 00:21:07,916 --> 00:21:14,140
way, the the way my way it kind of differs is surprise.
Cue 290 — 00:21:14,180 --> 00:21:17,760
Nix has a very powerful testing primitive.
Cue 291 — 00:21:17,900 --> 00:21:23,000
Nix is the biggest GitHub open source community on GitHub by far.
Cue 292 — 00:21:23,860 --> 00:21:27,580
If you're here GitHub having issues, it's normally nix packages causing the
Cue 293 — 00:21:27,620 --> 00:21:28,520
problems.
Cue 294 — 00:21:29,940 --> 00:21:33,560
And why this is interesting and why this comes to loop engineering.
Cue 295 — 00:21:45,220 --> 00:21:49,400
It's because when I do my loops, I give my agents access to sudo.
Cue 296 — 00:21:50,340 --> 00:21:54,720
They run as root. Would you do that on an Ubuntu machine?
Cue 297 — 00:21:55,180 --> 00:21:59,180
Hell no. I won't have time. Find me later.
Cue 298 — 00:21:59,220 --> 00:22:01,100
I'll go into it because I just don't want to.
Cue 299 — 00:22:01,140 --> 00:22:05,700
Just talking my book. But there are some unique properties on a developer throwaway
Cue 300 — 00:22:05,740 --> 00:22:10,420
machine where I can use nix with sudo, and I can run these loops and it's,
Cue 301 — 00:22:10,460 --> 00:22:14,420
it allows it to introspect every element from a database to journals and what have
Cue 302 — 00:22:14,460 --> 00:22:18,820
you got permissions. And it's safe because I've designed my operating environment to
Cue 303 — 00:22:18,860 --> 00:22:20,600
be able to do these types of things.
Cue 304 — 00:22:20,820 --> 00:22:27,360
Now. Why it matters. Why it matters Nix has a testing framework
Cue 305 — 00:22:27,400 --> 00:22:31,080
called NixOS test. I want you to imagine
Cue 306 — 00:22:32,540 --> 00:22:39,340
that you have a web server, maybe two web servers and a
Cue 307 — 00:22:39,380 --> 00:22:40,880
load balancer in front,
Cue 308 — 00:22:42,300 --> 00:22:46,320
and they're going to be on different network VLANs and all these types of things.
Cue 309 — 00:22:47,180 --> 00:22:51,360
How would you test that normally be really hard.
Cue 310 — 00:22:51,620 --> 00:22:54,600
I'm more speaking to the DevOps type people here.
Cue 311 — 00:22:54,900 --> 00:22:58,220
You normally really hard. It turns out here with Nix you spin up,
Cue 312 — 00:22:58,260 --> 00:23:02,020
you define what you need. It creates all the machines and you can assert these
Cue 313 — 00:23:02,060 --> 00:23:05,760
network rules are true about how the network works.
Cue 314 — 00:23:06,740 --> 00:23:09,440
I guess you're kind of wondering, why am I talking about this?
Cue 315 — 00:23:09,740 --> 00:23:14,180
It's because when I do my loops, I put the entire operating system and the entire
Cue 316 — 00:23:14,220 --> 00:23:17,600
fleet and the higher tier stack under test.
Cue 317 — 00:23:18,100 --> 00:23:23,020
It's integrated from day one. So I'm using this same primitive,
Cue 318 — 00:23:23,060 --> 00:23:27,100
not just a monkey patch of the world, but when I have my unit of work that I've
Cue 319 — 00:23:27,140 --> 00:23:31,460
done, the agents being done, I'm actually testing it in on the production,
Cue 320 — 00:23:31,820 --> 00:23:34,900
the exact what would be production infrastructure, because I'd define my
Cue 321 — 00:23:34,940 --> 00:23:38,700
infrastructure here as code, and it's integrated everywhere for.
Cue 322 — 00:23:38,740 --> 00:23:42,460
And that means I can deploy that increment somewhere as a preview environment.
Cue 323 — 00:23:42,500 --> 00:23:43,600
Really simple.
Cue 324 — 00:23:45,900 --> 00:23:52,060
So I'm hearing you correctly because you're running production off of Nix flakes.
Cue 325 — 00:23:52,100 --> 00:23:55,580
Yeah. You can just spin up an environment look just like production somewhere.
Cue 326 — 00:23:55,620 --> 00:24:00,155
Yes, the hell out of it and assume that if that is green, when you apply that Nix
Cue 327 — 00:24:00,195 --> 00:24:03,740
flake to production. Will be correct for like database etc..
Cue 328 — 00:24:03,780 --> 00:24:08,900
Now having said do not run nix do not run agents with nix as root on production.
Cue 329 — 00:24:08,940 --> 00:24:11,980
Hosts do not connect agents to production databases.
Cue 330 — 00:24:12,020 --> 00:24:16,260
Ever. The best thing to do if you want to do some sort of reproduction is somehow
Cue 331 — 00:24:16,300 --> 00:24:21,740
have somehow workflow that duplicates the database and restrict the heck out of the
Cue 332 — 00:24:21,780 --> 00:24:27,420
outbound network. So Deb, parity at this point is Dev Perry.
Cue 333 — 00:24:27,460 --> 00:24:32,340
Yeah, equals equals. But but it's it's time for structure.
Cue 334 — 00:24:32,380 --> 00:24:33,860
One source of truth, one way up.
Cue 335 — 00:24:33,900 --> 00:24:35,600
It's very powerful folks.
Cue 336 — 00:24:36,980 --> 00:24:41,040
In the couple of minutes that's left, I'll run you through some general theory.
Cue 337 — 00:24:43,860 --> 00:24:46,220
It was not just about loops with Ralph.
Cue 338 — 00:24:46,260 --> 00:24:50,000
It was. It was meant to be about stopping the loop from closing.
Cue 339 — 00:24:50,380 --> 00:24:56,240
So what is the loop? The loop is it generates code, and then I guess it commits code
Cue 340 — 00:24:57,300 --> 00:25:01,340
now and generates code, compiles software, etc..
Cue 341 — 00:25:01,380 --> 00:25:07,120
So the best thing to do to stop the loop from closing is to do deterministic checks.
Cue 342 — 00:25:07,180 --> 00:25:09,740
These aren't skills. This is engineering again.
Cue 343 — 00:25:09,780 --> 00:25:16,340
So the idea is you can stop git from being pushed if the software doesn't
Cue 344 — 00:25:16,380 --> 00:25:20,120
compile. Well this is called a pre commit hook or PreK.
Cue 345 — 00:25:20,540 --> 00:25:24,660
And it turns out you can engineer quite a lot of back pressure using these hooks.
Cue 346 — 00:25:24,700 --> 00:25:29,220
And then you can do all sorts of things like static analysis that this one domain
Cue 347 — 00:25:29,260 --> 00:25:30,980
cannot be coupled to another domain.
Cue 348 — 00:25:31,020 --> 00:25:33,940
Let's say you've got an API domain and you've got a SQL server,
Cue 349 — 00:25:33,980 --> 00:25:37,860
a SQL domain. Well, you don't want the SQL referencing the API.
Cue 350 — 00:25:37,900 --> 00:25:42,740
That would be normally an architecture, but the agents are pretty,
Cue 351 — 00:25:42,780 --> 00:25:46,340
pretty dumb. They'll do it. So you're probably thinking like,
Cue 352 — 00:25:46,380 --> 00:25:48,540
how do I solve my code review problems?
Cue 353 — 00:25:48,580 --> 00:25:53,860
Simple. You engineer away the need to do code review through engineering these types
Cue 354 — 00:25:53,900 --> 00:26:00,020
of checks. So what you do is this static, static and semantic analysis using the
Cue 355 — 00:26:00,060 --> 00:26:02,380
language service of your particular programming language.
Cue 356 — 00:26:02,420 --> 00:26:04,560
There's so much you can do. You just got to ask.
Cue 357 — 00:26:04,820 --> 00:26:10,180
The next thing is unit test. The problem with unit test is they'll only test for the
Cue 358 — 00:26:10,220 --> 00:26:14,020
things that you specify. Edge cases slip through.
Cue 359 — 00:26:14,060 --> 00:26:16,020
So there's a thing called property based testing.
Cue 360 — 00:26:16,060 --> 00:26:16,860
What
Cue 361 — 00:26:18,100 --> 00:26:21,140
ever since late 2013 you should use it.
Cue 362 — 00:26:21,180 --> 00:26:25,600
It's really good. Let's say you got a function to reverse a string.
Cue 363 — 00:26:26,300 --> 00:26:32,520
And the implementation here is faulty because it does not support Unicode or emojis.
Cue 364 — 00:26:33,140 --> 00:26:35,220
Because I never wrote the test, this would slip through.
Cue 365 — 00:26:35,260 --> 00:26:37,340
This is a class A category problem.
Cue 366 — 00:26:37,380 --> 00:26:43,100
What's different with a property based testing is you have a test generator that
Cue 367 — 00:26:43,140 --> 00:26:48,100
would punch in random strings, emojis, big strings, long strings,
Cue 368 — 00:26:48,140 --> 00:26:52,980
what else have you? And it will just keep on doing this and exploring the state
Cue 369 — 00:26:53,020 --> 00:26:56,000
space to ensure that your property holds true.
Cue 370 — 00:26:56,700 --> 00:27:00,420
And the property is. A string is reversible and the string does reverse.
Cue 371 — 00:27:00,460 --> 00:27:02,260
So a different way of testing.
Cue 372 — 00:27:02,300 --> 00:27:06,060
But I'm not just talking about string reversing here.
Cue 373 — 00:27:06,100 --> 00:27:12,700
I'm talking about like if a customer does a refund, they only get refunded
Cue 374 — 00:27:12,740 --> 00:27:16,960
once across the life cycle, across all your infrastructure of microservices.
Cue 375 — 00:27:17,260 --> 00:27:21,280
The property is a little bit weird to think about, but it's really cool.
Cue 376 — 00:27:22,580 --> 00:27:27,860
Day to day field CTO at a company called Antithesis, validation and verification of
Cue 377 — 00:27:27,900 --> 00:27:32,280
software is what we do. We have a library here called Bombadil.
Cue 378 — 00:27:32,580 --> 00:27:35,940
You should ditch Playwright folks, and I'm not just speaking my book.
Cue 379 — 00:27:35,980 --> 00:27:37,960
You should literally ditch it.
Cue 380 — 00:27:38,650 --> 00:27:40,540
Bombadil allows you to write a property.
Cue 381 — 00:27:40,580 --> 00:27:46,580
A property is the user always has the ability to log out like this.
Cue 382 — 00:27:46,620 --> 00:27:50,980
Right. And so what Bombadil will do is it will put your website a web the actual
Cue 383 — 00:27:51,020 --> 00:27:56,660
like just your plain like JavaScript TypeScript application in the web browser.
Cue 384 — 00:27:56,700 --> 00:28:00,400
And it will put your application through through hell.
Cue 385 — 00:28:00,780 --> 00:28:04,740
It'll rotate the screen, it will change the screen dimensions,
Cue 386 — 00:28:04,780 --> 00:28:07,100
it will go big screens, full screen, etc.
Cue 387 — 00:28:07,140 --> 00:28:13,040
and it will try doing all these random acts of violence to your web application
Cue 388 — 00:28:13,340 --> 00:28:17,020
until that property holds true or false.
Cue 389 — 00:28:17,060 --> 00:28:20,080
A counter example of where it failed.
Cue 390 — 00:28:20,380 --> 00:28:24,660
And when you have that counterexample, how it failed, you get a reproduction of how
Cue 391 — 00:28:24,700 --> 00:28:29,180
it has failed. So you know the exact test case that cool.
Cue 392 — 00:28:29,220 --> 00:28:30,560
So that's front end.
Cue 393 — 00:28:33,100 --> 00:28:39,620
I still believe something that is still to this day about February,
Cue 394 — 00:28:39,660 --> 00:28:44,220
we woke up and we saw PewDiePie was using this and we still don't even know how,
Cue 395 — 00:28:44,260 --> 00:28:46,340
because this is not popular knowledge.
Cue 396 — 00:28:46,380 --> 00:28:50,520
This is academic knowledge. We have no idea how PewDiePie is using Bombadil,
Cue 397 — 00:28:50,580 --> 00:28:53,320
but he is. Maybe it's in the model weights.
Cue 398 — 00:28:53,580 --> 00:28:59,920
Okay, we also have Hegel. Hegel is a small core library,
Cue 399 — 00:29:01,180 --> 00:29:04,580
written in Rust. And then we put sugar on top for each one of the different
Cue 400 — 00:29:04,620 --> 00:29:10,000
languages out there. You might also know Hegel from its successor.
Cue 401 — 00:29:10,860 --> 00:29:14,560
It's from its predecessor. And that's Hypothesis.
Cue 402 — 00:29:15,140 --> 00:29:19,180
Same team, same tech. But we're bringing this all to all programming languages,
Cue 403 — 00:29:19,220 --> 00:29:23,800
doing these two things in your code base, apart from solving your patching woes,
Cue 404 — 00:29:23,900 --> 00:29:27,060
is a massive uplift within your organization.
Cue 405 — 00:29:27,100 --> 00:29:29,940
Massive uplift, and they're just free there.
Cue 406 — 00:29:29,980 --> 00:29:30,830
You just got to know about it.
Cue 407 — 00:29:30,870 --> 00:29:34,540
You got to use it. Finally, if you like these types of talks,
Cue 408 — 00:29:34,580 --> 00:29:38,880
which just cold hard engineering, we have a YouTube channel,
Cue 409 — 00:29:39,140 --> 00:29:43,940
Bug Bash. You'll find talks on there from like even Don Syme at GitHub who's
Cue 410 — 00:29:43,980 --> 00:29:45,420
actually building the software factories.
Cue 411 — 00:29:45,460 --> 00:29:50,240
We had them. We had them over in Copenhagen a couple of months ago.
Cue 412 — 00:29:50,420 --> 00:29:53,260
There's Jane Street on there talking about how they're doing software factories.
Cue 413 — 00:29:53,300 --> 00:29:57,440
If you like nerd nerd stuff, that's real, go check out our YouTube channel.
Cue 414 — 00:29:58,460 --> 00:29:59,360
Thank you.
.tessl-plugin
skills
talk-arcolano-autonomy-and-trust
talk-aroussi-decision-context
talk-bevin-agent-assisted-instrument-design
talk-boeckeler-context-lifecycles
talk-braun-context-graphs
talk-carmont-evolving-agent-harnesses
talk-chanezon-gordon-agent-experience-evaluation
talk-cox-secure-agentic-workflows
talk-cunniffe-reusable-agent-platform
talk-davis-agent-tool-policies
talk-debois-concurrent-delivery
talk-dzialets-team-automation
talk-fahmy-harness-api-design
talk-finster-continuous-delivery
talk-fowler-regenerative-software
talk-gavrilescu-factory-architecture
talk-graciolli-amin-verification-aware-programming
talk-gruenbaum-reverse-graph-engineering
talk-hassan-agentic-engineering
talk-hossain-enterprise-agent-adoption
talk-huntley-patch-ready-factories
talk-jain-verifying-intent
talk-katsioloudes-montei-agentic-lifecycle
talk-knox-harness-engineering
talk-kovac-deterministic-memory
talk-lee-perkins-code-readiness
talk-lorenc-machine-speed-security
talk-lubken-software-of-one
talk-meijer-software-specialization
talk-meppiel-versioned-agent-contracts
talk-moss-concurrent-agent-risk
talk-myers-understanding-agent-generated-code
talk-ocytko-scaling-agentic-engineering
talk-oza-automating-design-quality
talk-panzica-agentic-sdlc
talk-pari-patel-agent-awareness
talk-podjarny-trustworthy-software-factories
talk-poulter-factory-improvement
talk-pratik-patel-refined-rag-data
talk-re-ferre-development-platforms
talk-sadogursky-factory-of-factories
talk-shah-training-agent-skills
talk-sharma-mohapatra-evolving-intent
talk-singhal-operating-agents-at-scale
talk-stack-incident-dossiers
talk-tyler-team-coding-benchmarks
talk-wotherspoon-agent-execution-environments
talk-yegge-ai-powered-factories
talk-zuber-leadership-through-coding