CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-huntley-patch-ready-factories/references/

Corrected transcript: Hack the planet? Patch the planet.

Geoffrey Huntley · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=2aV2XYo8GVE

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:00,460 --> 00:00:02,800

Hey, everyone. Thanks for joining me here today.

Cue 2 — 00:00:03,340 --> 00:00:06,900

Rather than talking about AI and all the rest, I'm just going to talk about some

Cue 3 — 00:00:06,940 --> 00:00:12,020

engineering because deeply, I believe the way to solve some of the problems we have

Cue 4 — 00:00:12,060 --> 00:00:13,760

is actually through engineering.

Cue 5 — 00:00:13,940 --> 00:00:17,600

And I'm going to put forward to you some ideas.

Cue 6 — 00:00:18,700 --> 00:00:22,180

Today and tomorrow you're going to be seeing a whole bunch of talks about how do you

Cue 7 — 00:00:22,220 --> 00:00:24,900

build software faster with factories and all the rest.

Cue 8 — 00:00:24,940 --> 00:00:25,740

Cool.

Cue 9 — 00:00:27,020 --> 00:00:32,200

Maybe if that's the wrong thing, how do you do security incident response faster,

Cue 10 — 00:00:33,340 --> 00:00:37,820

right. And it turns out if you optimize for your security incident response times,

Cue 11 — 00:00:37,860 --> 00:00:40,160

it also accelerates your build times.

Cue 12 — 00:00:40,580 --> 00:00:43,520

So I'm going to show you how I do some things.

Cue 13 — 00:00:44,180 --> 00:00:47,320

So two weeks ago it was pretty wild right?

Cue 14 — 00:00:48,660 --> 00:00:55,360

It was pretty cool. Folks like wow, you mean open code got breached.

Cue 15 — 00:00:55,980 --> 00:00:59,220

Then the Prime Minister of Japan was just going, hey everyone,

Cue 16 — 00:00:59,260 --> 00:01:00,600

you should update PHP.

Cue 17 — 00:01:03,140 --> 00:01:08,340

Next thing you know, people were like, okay, there's a vulnerability in HTTP stack

Cue 18 — 00:01:08,380 --> 00:01:12,960

that allows you to turn off a computer that's running.

Cue 19 — 00:01:14,140 --> 00:01:20,940

And then we got into a 12 year old, a 12 year old security exploit in

Cue 20 — 00:01:20,980 --> 00:01:26,100

GitLab that led to a compromise to and remote code execution.

Cue 21 — 00:01:26,140 --> 00:01:33,400

Ouch. It turns out the the answer to life in the universe is not 42.

Cue 22 — 00:01:34,060 --> 00:01:39,844

It is four. It is four. The magic number that unlocks all the encryption on the

Cue 23 — 00:01:39,884 --> 00:01:46,240

internet is four. If you don't know what happened, a a certificate authority,

Cue 24 — 00:01:47,380 --> 00:01:54,340

was incompetent and they put their master password, the

Cue 25 — 00:01:54,380 --> 00:01:59,800

master key, as four. And you could sign any other SSL with that if you knew the

Cue 26 — 00:01:59,840 --> 00:02:02,480

number four. Four opens all doors on the internet.

Cue 27 — 00:02:03,100 --> 00:02:06,720

So we started wondering, gosh, is it time to turn off the internet yet?

Cue 28 — 00:02:08,580 --> 00:02:12,980

And if you're kind of wondering, like, this is what we're looking at,

Cue 29 — 00:02:13,020 --> 00:02:18,900

folks, this is what we're looking at, you might have seen the GitHub quadratic graph

Cue 30 — 00:02:18,940 --> 00:02:23,420

of like everyone raising pull requests and GitHub having some availability issues

Cue 31 — 00:02:23,460 --> 00:02:25,760

because of it. Because the agent take off.

Cue 32 — 00:02:27,020 --> 00:02:29,280

This is security take off folks.

Cue 33 — 00:02:30,260 --> 00:02:35,720

And I'm putting forward is this is a start of the new normal.

Cue 34 — 00:02:36,820 --> 00:02:39,560

This is the start of the new normal that.

Cue 35 — 00:02:41,860 --> 00:02:45,580

Is normal now. Now it could be it could be debated.

Cue 36 — 00:02:45,620 --> 00:02:50,420

Maybe this is just a small bump in the line of the road until,

Cue 37 — 00:02:50,460 --> 00:02:55,000

like, all the software has been patched, maybe they will go down.

Cue 38 — 00:02:55,140 --> 00:02:58,520

Maybe they'll go down. But agents are writing software now.

Cue 39 — 00:02:59,060 --> 00:03:01,940

And agents have been trained on existing code, and existing code will have these

Cue 40 — 00:03:01,980 --> 00:03:06,960

security vulnerabilities. So argue that this is the start of the new normal.

Cue 41 — 00:03:08,820 --> 00:03:13,520

And unfortunately cyber is only going to accelerate from this point forward out.

Cue 42 — 00:03:13,900 --> 00:03:19,340

Like it's going to be really weird, folks, because like, I suspect you're going to

Cue 43 — 00:03:19,380 --> 00:03:25,080

need more inferencing capacity than your attackers to remain secure.

Cue 44 — 00:03:26,220 --> 00:03:30,140

Figure about that. Think about the companies right now and they're thinking,

Cue 45 — 00:03:30,180 --> 00:03:35,520

oh no, we can only do like $100 a month token budget per employee.

Cue 46 — 00:03:37,340 --> 00:03:41,180

Yeah. No. Yeah. Not so. This is one of the things I think it's going to be really,

Cue 47 — 00:03:41,220 --> 00:03:46,000

really different is as soon as you publish your any code on GitHub,

Cue 48 — 00:03:46,140 --> 00:03:51,820

the end result. Net result is some person with racks of inferencing hardware is

Cue 49 — 00:03:51,860 --> 00:03:56,420

going to be running autonomously scanning natural response and then autonomously

Cue 50 — 00:03:56,460 --> 00:03:59,080

tool calling to actually exploit that exploit.

Cue 51 — 00:04:00,060 --> 00:04:01,460

This is the new state of the normal.

Cue 52 — 00:04:01,500 --> 00:04:08,260

It's getting weird. And I guess I wrote this almost a year ago now it's a

Cue 53 — 00:04:08,300 --> 00:04:12,660

blog post called libraries. And according to the question, why do we even have open

Cue 54 — 00:04:12,700 --> 00:04:17,500

source libraries? And I got grilled to hell on Hacker News when I published this.

Cue 55 — 00:04:17,540 --> 00:04:20,360

But every day that goes by, it's become even more true.

Cue 56 — 00:04:21,060 --> 00:04:27,080

I'll explain. The reason we had open source libraries was because code was precious.

Cue 57 — 00:04:27,180 --> 00:04:33,780

It was hard to write, so we decided to concentrate into libraries to make reuse

Cue 58 — 00:04:33,820 --> 00:04:38,500

easy. And we also did this to make hiring easier, right?

Cue 59 — 00:04:38,540 --> 00:04:44,100

React famously by Facebook was kind of like of a play to standardize,

Cue 60 — 00:04:44,140 --> 00:04:47,080

making it easy for them to hire frontend developers.

Cue 61 — 00:04:47,340 --> 00:04:52,280

So we've done all these things in our practices to optimize for humans,

Cue 62 — 00:04:52,900 --> 00:04:57,880

and I don't think that we need libraries anymore.

Cue 63 — 00:04:58,380 --> 00:05:03,820

Now, I'm not saying that we should go out there and completely like vibe code up

Cue 64 — 00:05:03,860 --> 00:05:08,140

some cryptography. If you're a cryptographer, go do it, please.

Cue 65 — 00:05:08,180 --> 00:05:12,760

But I'm not. So there is going to be like a trusted base and a trusted core.

Cue 66 — 00:05:13,300 --> 00:05:18,180

But you think about think about your supply chain.

Cue 67 — 00:05:18,220 --> 00:05:21,960

How much time do you spend on Dependabot updates?

Cue 68 — 00:05:23,900 --> 00:05:27,460

Cool. How much time do you spend doing security incident response?

Cue 69 — 00:05:27,500 --> 00:05:31,500

Because the Dependabot update has been compromised because the upstream author is

Cue 70 — 00:05:31,540 --> 00:05:37,220

being compromised. Right. So all the all already we're in these negative time

Cue 71 — 00:05:37,260 --> 00:05:43,100

savings just from the from the what we do because we consume these libraries because

Cue 72 — 00:05:43,140 --> 00:05:47,440

the prior was the prior was writing software was hard.

Cue 73 — 00:05:47,780 --> 00:05:53,980

So it's been a year now. I don't use much open source anymore.

Cue 74 — 00:05:54,020 --> 00:05:59,220

Sure, I'll use the Linux kernel and like DNS servers and BIND and all the lower

Cue 75 — 00:05:59,260 --> 00:06:05,140

level stuff. But as I start getting into high level stuff, I just prompt the agent

Cue 76 — 00:06:05,180 --> 00:06:06,400

for what I want.

Cue 77 — 00:06:07,620 --> 00:06:10,260

You have to find what makes you comfortable.

Cue 78 — 00:06:10,300 --> 00:06:14,140

It will feel very unnatural at first when you first start doing it.

Cue 79 — 00:06:14,180 --> 00:06:15,720

But here's the thing.

Cue 80 — 00:06:17,140 --> 00:06:22,500

If it's like well and like a small and well specified, maybe like a,

Cue 81 — 00:06:22,540 --> 00:06:26,580

like a, an argument parser on a command line application, sure,

Cue 82 — 00:06:26,620 --> 00:06:31,340

you could take that dependency, but you could just like ask Claude and run in a loop

Cue 83 — 00:06:31,380 --> 00:06:33,340

to build that command line parser.

Cue 84 — 00:06:33,380 --> 00:06:36,000

And something magical happens when you do that.

Cue 85 — 00:06:36,260 --> 00:06:40,500

It's built to your use case and your needs, and you don't need to interface with

Cue 86 — 00:06:40,540 --> 00:06:45,260

someone on GitHub asking them to take your pull request a merge of what else have

Cue 87 — 00:06:45,300 --> 00:06:49,960

you. It turns out if you have to tool call a human, that's not AGI.

Cue 88 — 00:06:51,700 --> 00:06:58,700

So the best thing you can really do is either vendor that open source software and

Cue 89 — 00:06:58,740 --> 00:07:00,600

on your file system check it in.

Cue 90 — 00:07:00,740 --> 00:07:04,960

And that way the agent can modify it and extend it to its needs.

Cue 91 — 00:07:05,180 --> 00:07:08,800

That's one approach. I'm going to show you a different approach today.

Cue 92 — 00:07:10,220 --> 00:07:13,240

And that's something that I'm calling like Patch the Planet.

Cue 93 — 00:07:13,860 --> 00:07:20,320

You see, patching is perceived to be hard by people, but it doesn't have to be.

Cue 94 — 00:07:20,380 --> 00:07:23,500

So one of the things that makes patching software hard is like,

Cue 95 — 00:07:23,540 --> 00:07:26,540

let's say you wanted to patch something on GitHub, you know,

Cue 96 — 00:07:26,580 --> 00:07:31,940

and you got to figure out how that piece of software actually builds and that

Cue 97 — 00:07:31,980 --> 00:07:36,100

toolchain and oh my God, I'm going to have a new CI job just for that one thing.

Cue 98 — 00:07:36,140 --> 00:07:37,940

And you do it on a per language basis.

Cue 99 — 00:07:37,980 --> 00:07:40,940

And each open source library has a different build toolchain.

Cue 100 — 00:07:40,980 --> 00:07:43,900

So this just gets really, really, really hard.

Cue 101 — 00:07:43,940 --> 00:07:47,280

It doesn't have to be folks. Does not have to be.

Cue 102 — 00:07:48,620 --> 00:07:52,040

And then let's say you've actually patched it, you're like, God.

Cue 103 — 00:07:53,980 --> 00:07:56,680

Oh, I guess I'm going to host that in an artifact repo now.

Cue 104 — 00:07:57,100 --> 00:08:00,380

And now going to think about artifact tokens and all this complexity.

Cue 105 — 00:08:00,420 --> 00:08:02,340

We're making things needlessly complex.

Cue 106 — 00:08:02,380 --> 00:08:07,180

There's been solutions. Today I'm going to introduce you to something called Nix.

Cue 107 — 00:08:07,220 --> 00:08:09,260

Nix has been around since 2003.

Cue 108 — 00:08:09,300 --> 00:08:13,620

I've been using it for years. You might hear that Nix sucks.

Cue 109 — 00:08:13,660 --> 00:08:14,720

It does.

Cue 110 — 00:08:16,770 --> 00:08:18,480

Like there is no silver bullet.

Cue 111 — 00:08:18,740 --> 00:08:23,840

But the thing that you most commonly hear about Nix is it's hard to use.

Cue 112 — 00:08:24,350 --> 00:08:26,920

It is where it was,

Cue 113 — 00:08:28,420 --> 00:08:35,200

right? You see, the agents know how to drive a drive, an outcome.

Cue 114 — 00:08:35,620 --> 00:08:38,060

If you want to use Nix, you just need to go to a.

Cue 115 — 00:08:38,100 --> 00:08:41,860

You just need to know that Nix exists and you can actually prompt an agent.

Cue 116 — 00:08:41,900 --> 00:08:44,760

Hey, I want Nix here and you get Nix.

Cue 117 — 00:08:45,180 --> 00:08:47,020

You can do that with Bazel and Buck2.

Cue 118 — 00:08:47,060 --> 00:08:50,460

Now those those two technologies also suck, you know.

Cue 119 — 00:08:50,500 --> 00:08:52,620

And like you speak with any eng leader.

Cue 120 — 00:08:52,660 --> 00:08:57,500

They got a story where they almost got fired because they staffed a team of five to

Cue 121 — 00:08:57,540 --> 00:08:58,980

introduce some of these technologies.

Cue 122 — 00:08:59,020 --> 00:09:01,640

And it took a long time to upskill all the developers.

Cue 123 — 00:09:01,940 --> 00:09:08,740

This stuff is no longer true. You can now use a highly complex tools simply

Cue 124 — 00:09:08,780 --> 00:09:11,260

that are high powered. It has got to know about it, and you've got to do it,

Cue 125 — 00:09:11,300 --> 00:09:14,240

and you've got to falsify your priors.

Cue 126 — 00:09:14,580 --> 00:09:19,120

So the language sucks, I don't care, I just want the outcome.

Cue 127 — 00:09:19,220 --> 00:09:23,380

So let's get into the outcome up here on my GitHub.

Cue 128 — 00:09:23,420 --> 00:09:29,820

It's a thing called nix demo. After this, clone it down and have a look.

Cue 129 — 00:09:29,860 --> 00:09:34,720

Have a look at it because it's got some pretty interesting things in there.

Cue 130 — 00:09:36,220 --> 00:09:38,092

I use something called devenv.

Cue 131 — 00:09:38,132 --> 00:09:44,160

Devenv is just like kind of like a nice introduction to Nix.

Cue 132 — 00:09:44,620 --> 00:09:49,132

And what you do is you define your packages, your services that you did and what

Cue 133 — 00:09:49,172 --> 00:09:55,640

have you in the Nix. And that one file can generate out your developer laptops,

Cue 134 — 00:09:56,460 --> 00:09:57,720

your CI/CD,

Cue 135 — 00:09:59,220 --> 00:10:03,880

your agent sandboxes, and your production Docker images.

Cue 136 — 00:10:04,980 --> 00:10:07,060

You see what I'm going here? We've got a single source of truth,

Cue 137 — 00:10:07,100 --> 00:10:10,860

folks. How many people right now are spinning up teams to figure out what they're

Cue 138 — 00:10:10,900 --> 00:10:12,920

going to do about agent sandboxes?

Cue 139 — 00:10:13,420 --> 00:10:16,980

And are they are they just duplicating environments?

Cue 140 — 00:10:17,020 --> 00:10:20,980

Because I bet your bottom dollar right now in some of your companies,

Cue 141 — 00:10:21,020 --> 00:10:25,460

your laptop, you know, and it's not reproducible like you're onboard a new employee

Cue 142 — 00:10:25,500 --> 00:10:29,740

or you lose a laptop that's like 48 hours to set up your developer environment

Cue 143 — 00:10:29,780 --> 00:10:33,700

again. CI/CD, there's drift. So it works on your computer.

Cue 144 — 00:10:33,740 --> 00:10:37,160

But like CI/CD says it doesn't.

Cue 145 — 00:10:37,260 --> 00:10:40,060

Next, you know, you got another source of drift the agent sandboxes.

Cue 146 — 00:10:40,100 --> 00:10:45,300

And then, of course, you say it's all working and done, but the doesn't work on the

Cue 147 — 00:10:45,340 --> 00:10:49,680

production Docker image I'm getting at is there's a solution here.

Cue 148 — 00:10:52,220 --> 00:10:56,540

And interesting enough Nix actually can produce would it produces these Docker

Cue 149 — 00:10:56,580 --> 00:11:01,740

images. They're incredibly efficient because the way how Docker works is it

Cue 150 — 00:11:01,780 --> 00:11:05,480

basically stacks file system layers on top of each other and squashes them.

Cue 151 — 00:11:05,740 --> 00:11:10,600

Nix is a graph, and it's able to use that graph to build a very optimized image.

Cue 152 — 00:11:10,940 --> 00:11:11,740

Cool.

Cue 153 — 00:11:13,060 --> 00:11:17,520

I'm going to quickly come across here and show you some cool things.

Cue 154 — 00:11:18,580 --> 00:11:22,600

So I want to show you what version of git here.

Cue 155 — 00:11:23,300 --> 00:11:27,100

So I'm running 254. I'm going to use some magic.

Cue 156 — 00:11:27,140 --> 00:11:28,540

Imagine that I'm a software engineer.

Cue 157 — 00:11:28,580 --> 00:11:30,320

I just got a brand new computer

Cue 158 — 00:11:31,980 --> 00:11:35,480

and I want to onboard into the company.

Cue 159 — 00:11:39,300 --> 00:11:45,626

Onboarded. If you think about the things like, like NVM or like

Cue 160 — 00:11:45,666 --> 00:11:51,140

Python version manager, etc., it's this for all your operating system dependencies,

Cue 161 — 00:11:51,180 --> 00:11:55,440

including the FFI boundaries. It goes past the programming language.

Cue 162 — 00:11:55,940 --> 00:11:58,800

And let's have a look at the version of git in here

Cue 163 — 00:12:00,340 --> 00:12:05,140

  1. Well that's interesting. What's what's so special about this version of git?

Cue 164 — 00:12:05,180 --> 00:12:11,080

It's not just a higher version of git on GitHub in the GitHub repo has the answer.

Cue 165 — 00:12:25,500 --> 00:12:30,000

This version of git does not support force push.

Cue 166 — 00:12:32,140 --> 00:12:35,820

You see, what I've done is I've monkey patched for my agent,

Cue 167 — 00:12:35,860 --> 00:12:41,140

sandbox, my local engineering laptops, and wherever I use this git and I've got a

Cue 168 — 00:12:41,180 --> 00:12:46,260

single source of git and anything else that needs git as a as a transitory

Cue 169 — 00:12:46,300 --> 00:12:49,840

dependency also gets my monkey patch version of git.

Cue 170 — 00:12:50,020 --> 00:12:52,720

So how do you stop an agent from doing force push?

Cue 171 — 00:12:52,780 --> 00:12:55,900

Because you know it hill climbs and it does murderous things.

Cue 172 — 00:12:55,940 --> 00:13:00,140

When it hill climbs and causes incidents, you move that capability from the

Cue 173 — 00:13:00,180 --> 00:13:03,980

underlying thing. So I do these type of tricks all over the place.

Cue 174 — 00:13:04,020 --> 00:13:07,500

The best way to stop a model from being able to do something is remove the

Cue 175 — 00:13:07,540 --> 00:13:11,500

capability for it to be able to do it, and sometimes the best way is just to monkey

Cue 176 — 00:13:11,540 --> 00:13:15,480

patch the underlying underlying system to be able to support these behaviors.

Cue 177 — 00:13:15,620 --> 00:13:19,240

So yeah, let's have a look how that works.

Cue 178 — 00:13:25,940 --> 00:13:26,740

Cool.

Cue 179 — 00:13:28,700 --> 00:13:31,280

So it's up here in GitHub, nix-demo.

Cue 180 — 00:13:31,340 --> 00:13:32,720

And it's really simple.

Cue 181 — 00:13:34,300 --> 00:13:38,900

For the older older people in the room of my generation, you probably remember patch

Cue 182 — 00:13:38,940 --> 00:13:44,540

files. Open source used to be distributed as a collection of patch files.

Cue 183 — 00:13:44,580 --> 00:13:50,460

Used to be a contributor. All this, like everything, must be merged upstream on

Cue 184 — 00:13:50,500 --> 00:13:54,680

GitHub. That is, that was never a specified in open source.

Cue 185 — 00:13:55,140 --> 00:13:58,640

And like we used this is how we used to do software development.

Cue 186 — 00:13:58,900 --> 00:14:04,600

So what happens here is I have a closure that defines git.

Cue 187 — 00:14:04,660 --> 00:14:06,840

And then I just had my custom patches.

Cue 188 — 00:14:07,260 --> 00:14:08,820

How did I get the custom patches?

Cue 189 — 00:14:08,860 --> 00:14:13,780

I prompted Claude, can you please use Nix and create me a custom overlay that

Cue 190 — 00:14:13,820 --> 00:14:20,620

removes force? Push. Done. Next thing you know, I now have a whole

Cue 191 — 00:14:20,660 --> 00:14:25,980

bunch of Unix patches that automatically applied to the git when I'm in that

Cue 192 — 00:14:26,020 --> 00:14:29,900

operating environment. But it's not just that environment.

Cue 193 — 00:14:29,940 --> 00:14:32,220

It could be a Docker, it could be production host.

Cue 194 — 00:14:32,260 --> 00:14:37,980

What does have you? This one little thing, this one little pattern turns everything

Cue 195 — 00:14:38,020 --> 00:14:40,080

in the entire world patchable.

Cue 196 — 00:14:40,700 --> 00:14:47,120

No longer are you waiting for someone upstream to merge a bug fix.

Cue 197 — 00:14:47,290 --> 00:14:50,000

You just route around it, just route around it.

Cue 198 — 00:14:50,100 --> 00:14:55,380

What happens is all software becomes infinitely patched, and that is the behavior

Cue 199 — 00:14:55,420 --> 00:14:58,820

that you need to for security incident response.

Cue 200 — 00:14:58,860 --> 00:15:01,420

It's also a behavior. You also need your engineering.

Cue 201 — 00:15:01,460 --> 00:15:03,820

How many times you try to figure out what you're going to do,

Cue 202 — 00:15:03,860 --> 00:15:07,380

you take a fork, something's wrong or you want to add this capability.

Cue 203 — 00:15:07,420 --> 00:15:12,300

So there's two paths. You either yourself, you generate a first hand and minimize

Cue 204 — 00:15:12,340 --> 00:15:16,520

the dependency. But if you can't minimize the dependency you can use Nix to monkey

Cue 205 — 00:15:16,560 --> 00:15:19,280

patch the world. Really cool.

Cue 206 — 00:15:20,620 --> 00:15:23,980

And you're probably wondering, there's all sorts of little magic somewhere.

Cue 207 — 00:15:24,020 --> 00:15:27,500

I'm hiding some skeletons in the closet in this regard.

Cue 208 — 00:15:27,540 --> 00:15:31,780

No, it's quite literally in here.

Cue 209 — 00:15:31,820 --> 00:15:35,720

I import the overlay and Nix handles it all.

Cue 210 — 00:15:37,300 --> 00:15:41,988

Like mise. You might think mise is good software.

Cue 211 — 00:15:42,028 --> 00:15:47,020

It's not. Try this because mise This is just a more like a package manager like

Cue 212 — 00:15:47,060 --> 00:15:50,620

homebrew or etc. etc.. Does it have this level functionality?

Cue 213 — 00:15:50,660 --> 00:15:54,560

It allows you to patch anything, including the kernel of the operating system,

Cue 214 — 00:15:55,180 --> 00:15:58,680

and then deploy that custom version across your fleet,

Cue 215 — 00:15:59,940 --> 00:16:02,240

because Nix does and Nix can.

Cue 216 — 00:16:05,900 --> 00:16:06,880

Because.

Cue 217 — 00:16:11,300 --> 00:16:14,540

When I wrote this blog post, I guess this is like a month ago,

Cue 218 — 00:16:14,580 --> 00:16:18,900

I talked about how OpenSSL, if there's a critical fault in this,

Cue 219 — 00:16:18,940 --> 00:16:21,420

is before what happened two weeks ago.

Cue 220 — 00:16:21,460 --> 00:16:26,180

And this is the key. You think about something like OpenSSL.

Cue 221 — 00:16:26,220 --> 00:16:29,340

You could update OpenSSL. That's cool.

Cue 222 — 00:16:29,380 --> 00:16:32,920

But what about when it's a transitive dependency.

Cue 223 — 00:16:33,940 --> 00:16:37,900

So OpenSSL is used in Nginx and Postgres.

Cue 224 — 00:16:37,940 --> 00:16:42,480

So it's not just a matter of like patching the dependency of like OpenSSL.

Cue 225 — 00:16:42,960 --> 00:16:44,460

If you wanted to manually patch.

Cue 226 — 00:16:44,500 --> 00:16:48,807

This is why we never did it. But it turns out if you have a single source of truth

Cue 227 — 00:16:48,847 --> 00:16:53,180

for OpenSSL, and you update that monkey patch similar to I did with git,

Cue 228 — 00:16:53,220 --> 00:16:58,820

the end result it will recompile the world and any transitory dependencies will get

Cue 229 — 00:16:58,860 --> 00:17:02,040

that custom version. The world is infinitely.

Cue 230 — 00:17:02,940 --> 00:17:08,340

Now Nix has some pretty cool caching, and it has remote builders similar to Bazel

Cue 231 — 00:17:08,380 --> 00:17:13,660

and and so it's not a matter of like, oh, it's going to compile.

Cue 232 — 00:17:13,700 --> 00:17:17,740

It's going to take forever. No, your CI when it validates.

Cue 233 — 00:17:17,780 --> 00:17:20,100

It's just going to compile the software and it publishes the cache.

Cue 234 — 00:17:20,140 --> 00:17:24,320

And your developer environment is sandboxes consume in that case simple.

Cue 235 — 00:17:24,380 --> 00:17:28,260

So I guess the really the question I want to put to you is how much productivity did

Cue 236 — 00:17:28,300 --> 00:17:29,840

you lose two weeks ago

Cue 237 — 00:17:31,180 --> 00:17:35,360

during the security incidents that happened because.

Cue 238 — 00:17:37,540 --> 00:17:39,920

I didn't lose any sleep, folks,

Cue 239 — 00:17:41,260 --> 00:17:44,920

day to day, I'm a field CTO at a company called Antithesis.

Cue 240 — 00:17:45,820 --> 00:17:50,300

We do a lot in testing. Come pick my brain at lunchtime and I can show you some

Cue 241 — 00:17:50,340 --> 00:17:53,740

really cool stuff. I literally just went into Claude and said,

Cue 242 — 00:17:53,780 --> 00:17:54,580

hey,

Cue 243 — 00:17:55,940 --> 00:18:00,120

because I got the single source of truth is my fleet affected?

Cue 244 — 00:18:01,060 --> 00:18:07,820

And it goes. Yeah. And from that point forward, it started monkey patching the what

Cue 245 — 00:18:07,860 --> 00:18:10,900

it needed. It updated the refs and the dependencies and like what,

Cue 246 — 00:18:10,940 --> 00:18:13,900

all the pins and it did the monkey patches as I need be.

Cue 247 — 00:18:13,940 --> 00:18:17,660

And because that source of truth or the transitory dependencies automatically

Cue 248 — 00:18:17,700 --> 00:18:20,680

recompiled done. I guess

Cue 249 — 00:18:22,060 --> 00:18:25,020

I'm enjoying the Nirvana life over there.

Cue 250 — 00:18:25,060 --> 00:18:27,740

You could be too. You just got to know that existed.

Cue 251 — 00:18:27,780 --> 00:18:30,540

So I hope that you take this opportunity to play with this idea,

Cue 252 — 00:18:30,580 --> 00:18:34,920

because you're you're probably over there in a root of pain.

Cue 253 — 00:18:35,220 --> 00:18:38,780

And it turns out if you optimize to get to this type of thing,

Cue 254 — 00:18:38,820 --> 00:18:41,300

your security incident response is faster.

Cue 255 — 00:18:41,340 --> 00:18:46,080

Your productivity for like factories and everything else also gets faster.

Cue 256 — 00:18:46,420 --> 00:18:49,120

It turns out the answer for AI is engineering.

Cue 257 — 00:18:49,940 --> 00:18:52,440

And there was one cute trick I did.

Cue 258 — 00:18:52,820 --> 00:18:56,700

It's a really one trick I did back over here.

Cue 259 — 00:18:56,740 --> 00:19:03,180

I said, hey, am I affected? And I had to do some temporary monkey patches to like

Cue 260 — 00:19:03,220 --> 00:19:08,440

similar to the git overlay. Well, I issued another prompt and that prompt was,

Cue 261 — 00:19:08,820 --> 00:19:14,320

hey, can you update the AGENTS.md and then like in seven days time,

Cue 262 — 00:19:14,500 --> 00:19:17,840

can you, can you like get rid of my patches?

Cue 263 — 00:19:18,660 --> 00:19:24,308

So instead of creating a Jira ticket, I just added like a reminder my AGENTS.md and

Cue 264 — 00:19:24,348 --> 00:19:28,460

the AGENTS.md when like the next developer was using or agent that picked off off

Cue 265 — 00:19:28,500 --> 00:19:34,880

that machine just automatically just did the house cleaning task kind of cool.

Cue 266 — 00:19:35,020 --> 00:19:40,100

So it automatically backed out that monkey patch role for my pins because the

Cue 267 — 00:19:40,140 --> 00:19:44,700

upstream had been fixed. And, you know, this is like the security maintenance ever

Cue 268 — 00:19:44,740 --> 00:19:48,140

done. And every time I have a security is like security type thing,

Cue 269 — 00:19:48,180 --> 00:19:52,240

I just do these two tricks. It's pretty nice.

Cue 270 — 00:19:53,260 --> 00:19:57,960

Again the repo is g, so github.com g.

Cue 271 — 00:19:58,020 --> 00:19:58,920

nix-demo.

Cue 283 — 00:20:41,620 --> 00:20:45,540

And I have ten minutes, so with do some bonus.

Cue 284 — 00:20:45,580 --> 00:20:48,160

Ponderous. I finished this security thing.

Cue 285 — 00:20:50,180 --> 00:20:51,640

Let's talk about loops.

Cue 286 — 00:20:52,900 --> 00:20:55,040

Let's talk about loops so

Cue 287 — 00:20:56,660 --> 00:21:02,860

everyone has their different ways on how to do loop engineering and and all these

Cue 288 — 00:21:02,900 --> 00:21:07,876

different things. There is no best practices in my way is very different to Steve's

Cue 289 — 00:21:07,916 --> 00:21:14,140

way, the the way my way it kind of differs is surprise.

Cue 290 — 00:21:14,180 --> 00:21:17,760

Nix has a very powerful testing primitive.

Cue 291 — 00:21:17,900 --> 00:21:23,000

Nix is the biggest GitHub open source community on GitHub by far.

Cue 292 — 00:21:23,860 --> 00:21:27,580

If you're here GitHub having issues, it's normally nix packages causing the

Cue 293 — 00:21:27,620 --> 00:21:28,520

problems.

Cue 294 — 00:21:29,940 --> 00:21:33,560

And why this is interesting and why this comes to loop engineering.

Cue 295 — 00:21:45,220 --> 00:21:49,400

It's because when I do my loops, I give my agents access to sudo.

Cue 296 — 00:21:50,340 --> 00:21:54,720

They run as root. Would you do that on an Ubuntu machine?

Cue 297 — 00:21:55,180 --> 00:21:59,180

Hell no. I won't have time. Find me later.

Cue 298 — 00:21:59,220 --> 00:22:01,100

I'll go into it because I just don't want to.

Cue 299 — 00:22:01,140 --> 00:22:05,700

Just talking my book. But there are some unique properties on a developer throwaway

Cue 300 — 00:22:05,740 --> 00:22:10,420

machine where I can use nix with sudo, and I can run these loops and it's,

Cue 301 — 00:22:10,460 --> 00:22:14,420

it allows it to introspect every element from a database to journals and what have

Cue 302 — 00:22:14,460 --> 00:22:18,820

you got permissions. And it's safe because I've designed my operating environment to

Cue 303 — 00:22:18,860 --> 00:22:20,600

be able to do these types of things.

Cue 304 — 00:22:20,820 --> 00:22:27,360

Now. Why it matters. Why it matters Nix has a testing framework

Cue 305 — 00:22:27,400 --> 00:22:31,080

called NixOS test. I want you to imagine

Cue 306 — 00:22:32,540 --> 00:22:39,340

that you have a web server, maybe two web servers and a

Cue 307 — 00:22:39,380 --> 00:22:40,880

load balancer in front,

Cue 308 — 00:22:42,300 --> 00:22:46,320

and they're going to be on different network VLANs and all these types of things.

Cue 309 — 00:22:47,180 --> 00:22:51,360

How would you test that normally be really hard.

Cue 310 — 00:22:51,620 --> 00:22:54,600

I'm more speaking to the DevOps type people here.

Cue 311 — 00:22:54,900 --> 00:22:58,220

You normally really hard. It turns out here with Nix you spin up,

Cue 312 — 00:22:58,260 --> 00:23:02,020

you define what you need. It creates all the machines and you can assert these

Cue 313 — 00:23:02,060 --> 00:23:05,760

network rules are true about how the network works.

Cue 314 — 00:23:06,740 --> 00:23:09,440

I guess you're kind of wondering, why am I talking about this?

Cue 315 — 00:23:09,740 --> 00:23:14,180

It's because when I do my loops, I put the entire operating system and the entire

Cue 316 — 00:23:14,220 --> 00:23:17,600

fleet and the higher tier stack under test.

Cue 317 — 00:23:18,100 --> 00:23:23,020

It's integrated from day one. So I'm using this same primitive,

Cue 318 — 00:23:23,060 --> 00:23:27,100

not just a monkey patch of the world, but when I have my unit of work that I've

Cue 319 — 00:23:27,140 --> 00:23:31,460

done, the agents being done, I'm actually testing it in on the production,

Cue 320 — 00:23:31,820 --> 00:23:34,900

the exact what would be production infrastructure, because I'd define my

Cue 321 — 00:23:34,940 --> 00:23:38,700

infrastructure here as code, and it's integrated everywhere for.

Cue 322 — 00:23:38,740 --> 00:23:42,460

And that means I can deploy that increment somewhere as a preview environment.

Cue 323 — 00:23:42,500 --> 00:23:43,600

Really simple.

Cue 324 — 00:23:45,900 --> 00:23:52,060

So I'm hearing you correctly because you're running production off of Nix flakes.

Cue 325 — 00:23:52,100 --> 00:23:55,580

Yeah. You can just spin up an environment look just like production somewhere.

Cue 326 — 00:23:55,620 --> 00:24:00,155

Yes, the hell out of it and assume that if that is green, when you apply that Nix

Cue 327 — 00:24:00,195 --> 00:24:03,740

flake to production. Will be correct for like database etc..

Cue 328 — 00:24:03,780 --> 00:24:08,900

Now having said do not run nix do not run agents with nix as root on production.

Cue 329 — 00:24:08,940 --> 00:24:11,980

Hosts do not connect agents to production databases.

Cue 330 — 00:24:12,020 --> 00:24:16,260

Ever. The best thing to do if you want to do some sort of reproduction is somehow

Cue 331 — 00:24:16,300 --> 00:24:21,740

have somehow workflow that duplicates the database and restrict the heck out of the

Cue 332 — 00:24:21,780 --> 00:24:27,420

outbound network. So Deb, parity at this point is Dev Perry.

Cue 333 — 00:24:27,460 --> 00:24:32,340

Yeah, equals equals. But but it's it's time for structure.

Cue 334 — 00:24:32,380 --> 00:24:33,860

One source of truth, one way up.

Cue 335 — 00:24:33,900 --> 00:24:35,600

It's very powerful folks.

Cue 336 — 00:24:36,980 --> 00:24:41,040

In the couple of minutes that's left, I'll run you through some general theory.

Cue 337 — 00:24:43,860 --> 00:24:46,220

It was not just about loops with Ralph.

Cue 338 — 00:24:46,260 --> 00:24:50,000

It was. It was meant to be about stopping the loop from closing.

Cue 339 — 00:24:50,380 --> 00:24:56,240

So what is the loop? The loop is it generates code, and then I guess it commits code

Cue 340 — 00:24:57,300 --> 00:25:01,340

now and generates code, compiles software, etc..

Cue 341 — 00:25:01,380 --> 00:25:07,120

So the best thing to do to stop the loop from closing is to do deterministic checks.

Cue 342 — 00:25:07,180 --> 00:25:09,740

These aren't skills. This is engineering again.

Cue 343 — 00:25:09,780 --> 00:25:16,340

So the idea is you can stop git from being pushed if the software doesn't

Cue 344 — 00:25:16,380 --> 00:25:20,120

compile. Well this is called a pre commit hook or PreK.

Cue 345 — 00:25:20,540 --> 00:25:24,660

And it turns out you can engineer quite a lot of back pressure using these hooks.

Cue 346 — 00:25:24,700 --> 00:25:29,220

And then you can do all sorts of things like static analysis that this one domain

Cue 347 — 00:25:29,260 --> 00:25:30,980

cannot be coupled to another domain.

Cue 348 — 00:25:31,020 --> 00:25:33,940

Let's say you've got an API domain and you've got a SQL server,

Cue 349 — 00:25:33,980 --> 00:25:37,860

a SQL domain. Well, you don't want the SQL referencing the API.

Cue 350 — 00:25:37,900 --> 00:25:42,740

That would be normally an architecture, but the agents are pretty,

Cue 351 — 00:25:42,780 --> 00:25:46,340

pretty dumb. They'll do it. So you're probably thinking like,

Cue 352 — 00:25:46,380 --> 00:25:48,540

how do I solve my code review problems?

Cue 353 — 00:25:48,580 --> 00:25:53,860

Simple. You engineer away the need to do code review through engineering these types

Cue 354 — 00:25:53,900 --> 00:26:00,020

of checks. So what you do is this static, static and semantic analysis using the

Cue 355 — 00:26:00,060 --> 00:26:02,380

language service of your particular programming language.

Cue 356 — 00:26:02,420 --> 00:26:04,560

There's so much you can do. You just got to ask.

Cue 357 — 00:26:04,820 --> 00:26:10,180

The next thing is unit test. The problem with unit test is they'll only test for the

Cue 358 — 00:26:10,220 --> 00:26:14,020

things that you specify. Edge cases slip through.

Cue 359 — 00:26:14,060 --> 00:26:16,020

So there's a thing called property based testing.

Cue 360 — 00:26:16,060 --> 00:26:16,860

What

Cue 361 — 00:26:18,100 --> 00:26:21,140

ever since late 2013 you should use it.

Cue 362 — 00:26:21,180 --> 00:26:25,600

It's really good. Let's say you got a function to reverse a string.

Cue 363 — 00:26:26,300 --> 00:26:32,520

And the implementation here is faulty because it does not support Unicode or emojis.

Cue 364 — 00:26:33,140 --> 00:26:35,220

Because I never wrote the test, this would slip through.

Cue 365 — 00:26:35,260 --> 00:26:37,340

This is a class A category problem.

Cue 366 — 00:26:37,380 --> 00:26:43,100

What's different with a property based testing is you have a test generator that

Cue 367 — 00:26:43,140 --> 00:26:48,100

would punch in random strings, emojis, big strings, long strings,

Cue 368 — 00:26:48,140 --> 00:26:52,980

what else have you? And it will just keep on doing this and exploring the state

Cue 369 — 00:26:53,020 --> 00:26:56,000

space to ensure that your property holds true.

Cue 370 — 00:26:56,700 --> 00:27:00,420

And the property is. A string is reversible and the string does reverse.

Cue 371 — 00:27:00,460 --> 00:27:02,260

So a different way of testing.

Cue 372 — 00:27:02,300 --> 00:27:06,060

But I'm not just talking about string reversing here.

Cue 373 — 00:27:06,100 --> 00:27:12,700

I'm talking about like if a customer does a refund, they only get refunded

Cue 374 — 00:27:12,740 --> 00:27:16,960

once across the life cycle, across all your infrastructure of microservices.

Cue 375 — 00:27:17,260 --> 00:27:21,280

The property is a little bit weird to think about, but it's really cool.

Cue 376 — 00:27:22,580 --> 00:27:27,860

Day to day field CTO at a company called Antithesis, validation and verification of

Cue 377 — 00:27:27,900 --> 00:27:32,280

software is what we do. We have a library here called Bombadil.

Cue 378 — 00:27:32,580 --> 00:27:35,940

You should ditch Playwright folks, and I'm not just speaking my book.

Cue 379 — 00:27:35,980 --> 00:27:37,960

You should literally ditch it.

Cue 380 — 00:27:38,650 --> 00:27:40,540

Bombadil allows you to write a property.

Cue 381 — 00:27:40,580 --> 00:27:46,580

A property is the user always has the ability to log out like this.

Cue 382 — 00:27:46,620 --> 00:27:50,980

Right. And so what Bombadil will do is it will put your website a web the actual

Cue 383 — 00:27:51,020 --> 00:27:56,660

like just your plain like JavaScript TypeScript application in the web browser.

Cue 384 — 00:27:56,700 --> 00:28:00,400

And it will put your application through through hell.

Cue 385 — 00:28:00,780 --> 00:28:04,740

It'll rotate the screen, it will change the screen dimensions,

Cue 386 — 00:28:04,780 --> 00:28:07,100

it will go big screens, full screen, etc.

Cue 387 — 00:28:07,140 --> 00:28:13,040

and it will try doing all these random acts of violence to your web application

Cue 388 — 00:28:13,340 --> 00:28:17,020

until that property holds true or false.

Cue 389 — 00:28:17,060 --> 00:28:20,080

A counter example of where it failed.

Cue 390 — 00:28:20,380 --> 00:28:24,660

And when you have that counterexample, how it failed, you get a reproduction of how

Cue 391 — 00:28:24,700 --> 00:28:29,180

it has failed. So you know the exact test case that cool.

Cue 392 — 00:28:29,220 --> 00:28:30,560

So that's front end.

Cue 393 — 00:28:33,100 --> 00:28:39,620

I still believe something that is still to this day about February,

Cue 394 — 00:28:39,660 --> 00:28:44,220

we woke up and we saw PewDiePie was using this and we still don't even know how,

Cue 395 — 00:28:44,260 --> 00:28:46,340

because this is not popular knowledge.

Cue 396 — 00:28:46,380 --> 00:28:50,520

This is academic knowledge. We have no idea how PewDiePie is using Bombadil,

Cue 397 — 00:28:50,580 --> 00:28:53,320

but he is. Maybe it's in the model weights.

Cue 398 — 00:28:53,580 --> 00:28:59,920

Okay, we also have Hegel. Hegel is a small core library,

Cue 399 — 00:29:01,180 --> 00:29:04,580

written in Rust. And then we put sugar on top for each one of the different

Cue 400 — 00:29:04,620 --> 00:29:10,000

languages out there. You might also know Hegel from its successor.

Cue 401 — 00:29:10,860 --> 00:29:14,560

It's from its predecessor. And that's Hypothesis.

Cue 402 — 00:29:15,140 --> 00:29:19,180

Same team, same tech. But we're bringing this all to all programming languages,

Cue 403 — 00:29:19,220 --> 00:29:23,800

doing these two things in your code base, apart from solving your patching woes,

Cue 404 — 00:29:23,900 --> 00:29:27,060

is a massive uplift within your organization.

Cue 405 — 00:29:27,100 --> 00:29:29,940

Massive uplift, and they're just free there.

Cue 406 — 00:29:29,980 --> 00:29:30,830

You just got to know about it.

Cue 407 — 00:29:30,870 --> 00:29:34,540

You got to use it. Finally, if you like these types of talks,

Cue 408 — 00:29:34,580 --> 00:29:38,880

which just cold hard engineering, we have a YouTube channel,

Cue 409 — 00:29:39,140 --> 00:29:43,940

Bug Bash. You'll find talks on there from like even Don Syme at GitHub who's

Cue 410 — 00:29:43,980 --> 00:29:45,420

actually building the software factories.

Cue 411 — 00:29:45,460 --> 00:29:50,240

We had them. We had them over in Copenhagen a couple of months ago.

Cue 412 — 00:29:50,420 --> 00:29:53,260

There's Jane Street on there talking about how they're doing software factories.

Cue 413 — 00:29:53,300 --> 00:29:57,440

If you like nerd nerd stuff, that's real, go check out our YouTube channel.

Cue 414 — 00:29:58,460 --> 00:29:59,360

Thank you.

skills

talk-huntley-patch-ready-factories

tile.json