CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-poulter-factory-improvement/references/

Corrected transcript: Did It Actually Improve the Software Factory?

Chris Poulter · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=b7LVk105nsA

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:00,380 --> 00:00:02,900

Hello. So my name is Chris Poulter.

Cue 2 — 00:00:02,940 --> 00:00:06,020

I'm based out of London. And I'm a founder.

Cue 3 — 00:00:06,060 --> 00:00:09,460

I've been working on a software factory for a couple of years now.

Cue 4 — 00:00:09,500 --> 00:00:11,780

From before it was called a software factory.

Cue 5 — 00:00:11,820 --> 00:00:16,220

So what I'm going to do today is share some of my learnings along the way in terms

Cue 6 — 00:00:16,260 --> 00:00:20,280

of improving the factory. So what we're going to cover today.

Cue 7 — 00:00:20,380 --> 00:00:24,740

First of all, is why improvements are rarely linear and they're almost always a

Cue 8 — 00:00:24,780 --> 00:00:26,860

trade off. We're going to see loads of examples of this.

Cue 9 — 00:00:26,900 --> 00:00:28,760

Quite embarrassing ones I'm sure.

Cue 10 — 00:00:29,340 --> 00:00:33,620

Why? Atomic decomposition breaking things down into the smallest possible chunks

Cue 11 — 00:00:33,660 --> 00:00:36,200

really enables compound improvements.

Cue 12 — 00:00:36,420 --> 00:00:40,260

We're also going to have a look at why improvements need a separate lifecycle to

Cue 13 — 00:00:40,300 --> 00:00:43,300

your code, whether it lives with your code or not, it's different matter.

Cue 14 — 00:00:43,340 --> 00:00:47,820

You definitely need a different life cycle and probably the hottest take of all.

Cue 15 — 00:00:47,860 --> 00:00:51,600

Why? Sandboxes are the wrong primitive for a software factory.

Cue 16 — 00:00:52,380 --> 00:00:58,100

So let's go back to what probably for me with the birth of real agentic coding,

Cue 17 — 00:00:58,140 --> 00:01:03,380

which was June 2024, right? Which is when Sonnet 3.5 came out.

Cue 18 — 00:01:03,420 --> 00:01:08,220

Now, the reason this was kind of a big deal is because it was the first model which

Cue 19 — 00:01:08,260 --> 00:01:13,900

could really do tool calling quite well, and that enabled us to do longer running

Cue 20 — 00:01:13,940 --> 00:01:18,700

agentic tasks. And, and that kind of changed things from before,

Cue 21 — 00:01:18,740 --> 00:01:22,580

where we're mostly kind of asking questions or doing autocomplete,

Cue 22 — 00:01:22,620 --> 00:01:26,300

that kind of thing. Now, the double whammy on this was it was only about a month

Cue 23 — 00:01:26,340 --> 00:01:29,193

later. that Cursor had brought out Composer.

Cue 24 — 00:01:29,233 --> 00:01:33,660

And Composer was the first real kind of tool that you had, which would allow you to

Cue 25 — 00:01:33,700 --> 00:01:38,420

do multi file editing. Yeah. And again, this was a big at the time because it's like

Cue 26 — 00:01:38,460 --> 00:01:40,700

oh now we could do useful things with an agent.

Cue 27 — 00:01:40,740 --> 00:01:44,980

Now we can make real changes to our code base right.

Cue 28 — 00:01:45,020 --> 00:01:49,540

But as anyone who worked with those tools at the time probably remembers.

Cue 29 — 00:01:49,580 --> 00:01:53,620

Sure, it might might give you some working code, but it would probably make some

Cue 30 — 00:01:53,660 --> 00:01:56,030

really weird choices along the way.

Cue 31 — 00:01:56,070 --> 00:02:00,500

Right. So there's so many cases of, oh, well, I didn't bother with this shared

Cue 32 — 00:02:00,540 --> 00:02:01,980

function. I just created a new one.

Cue 33 — 00:02:02,020 --> 00:02:05,460

Isn't it great? Yeah. Or I'm using this library now.

Cue 34 — 00:02:05,500 --> 00:02:07,340

You don't worry about that one over there.

Cue 35 — 00:02:07,380 --> 00:02:11,260

So very immediately clear. Just because it can change the code doesn't mean it's any

Cue 36 — 00:02:11,300 --> 00:02:17,420

good at it. So Cursor had introduced rules, which just it was the earliest version

Cue 37 — 00:02:17,460 --> 00:02:21,340

of skills, probably where we could just pass a prompt into kind of every session as

Cue 38 — 00:02:21,380 --> 00:02:27,140

it started. So the first iteration of this really for me was think with me before

Cue 39 — 00:02:27,180 --> 00:02:31,020

you build, because we knew it wasn't going to go out and look at the code base,

Cue 40 — 00:02:31,060 --> 00:02:33,420

it wasn't going to think through the choices it made.

Cue 41 — 00:02:33,460 --> 00:02:36,540

I had some really simple instructions in there telling it, look,

Cue 42 — 00:02:36,580 --> 00:02:41,940

when when a user asks you to do something, first of all, we search the code base,

Cue 43 — 00:02:41,980 --> 00:02:45,660

see what's there already, then think about three different ways which you could

Cue 44 — 00:02:45,700 --> 00:02:47,100

actually approach this problem.

Cue 45 — 00:02:47,140 --> 00:02:51,660

Come back to me with a recommendation, but then wait for my go ahead before you

Cue 46 — 00:02:51,700 --> 00:02:56,300

actually build it. And that was quite effective for individual changes.

Cue 47 — 00:02:56,340 --> 00:03:01,480

But what happens when you want to build a whole feature or even an entire system?

Cue 48 — 00:03:02,500 --> 00:03:06,260

So around this time I was working on Augmented OS.

Cue 49 — 00:03:06,300 --> 00:03:10,660

I think I was raising my pre-seed round at the time, and that was a workflow engine

Cue 50 — 00:03:10,700 --> 00:03:14,940

for AI native businesses. So what it really did was connect to all the existing

Cue 51 — 00:03:14,980 --> 00:03:20,060

platforms, built a data model specific to that business, and that allowed them to

Cue 52 — 00:03:20,100 --> 00:03:25,100

build workflows on top of all that, so that you could assign some tasks to agents,

Cue 53 — 00:03:25,140 --> 00:03:29,820

you could have some sign some tasks to deterministic code and some tasks to humans

Cue 54 — 00:03:29,860 --> 00:03:32,440

to oversee and approve and that kind of thing.

Cue 55 — 00:03:33,180 --> 00:03:38,880

And what I did was a massive exercise in spec-driven development of the entire

Cue 56 — 00:03:38,920 --> 00:03:42,660

system from scratch. Right. This is a durable workflow engine.

Cue 57 — 00:03:42,700 --> 00:03:47,860

They're very nitty gritty, quite hard to work on, and ended up over the space of a

Cue 58 — 00:03:47,900 --> 00:03:52,340

month while supposedly a holiday in Portugal, kind of coming up with this monster

Cue 59 — 00:03:52,380 --> 00:03:58,140

spec of 251 files, about 75,000 lines of specs.

Cue 60 — 00:03:58,180 --> 00:04:01,340

Right? And while that's great while you're building it, what happens when you need

Cue 61 — 00:04:01,380 --> 00:04:05,820

to iterate or suddenly you've added another module and that impacts other things?

Cue 62 — 00:04:05,860 --> 00:04:10,300

Your important decisions are now getting buried in documents where you can't see

Cue 63 — 00:04:10,340 --> 00:04:15,860

them. And there was kind of a moment at one point where I realized I'm building this

Cue 64 — 00:04:15,900 --> 00:04:20,740

workflow engine for businesses, but actually I need it for coding because it was the

Cue 65 — 00:04:20,780 --> 00:04:25,960

same problem. How can you divide work between agents and humans in a governed way?

Cue 66 — 00:04:26,420 --> 00:04:29,930

So what I did is I started to adapt the system that I was building for agentic

Cue 67 — 00:04:29,970 --> 00:04:33,220

coding. So if we think about the kind of integrations we need in software

Cue 68 — 00:04:33,260 --> 00:04:37,620

development, well, issue trackers, source control, development environments,

Cue 69 — 00:04:37,660 --> 00:04:40,900

those are all critical when we're building code, right?

Cue 70 — 00:04:40,940 --> 00:04:44,340

The data model was very interesting because what we'd realized if we take the

Cue 71 — 00:04:44,380 --> 00:04:49,060

example of decisions. These are decisions which are just getting buried in in a chat

Cue 72 — 00:04:49,100 --> 00:04:51,340

in the first place, or in specs later on.

Cue 73 — 00:04:51,380 --> 00:04:56,740

But actually, if we could turn these into typed artifacts where we're recording in a

Cue 74 — 00:04:56,780 --> 00:05:00,580

consistent way, exactly what it is and how it relates to other things,

Cue 75 — 00:05:00,620 --> 00:05:03,800

then actually we can use this to drive the workflow engine.

Cue 76 — 00:05:04,580 --> 00:05:10,060

And what I was finding was that as you scale up agentic coding and you have lots of

Cue 77 — 00:05:10,100 --> 00:05:12,220

different things on the go, I'm sure we've all been there.

Cue 78 — 00:05:12,260 --> 00:05:13,380

It's hard to find what you want.

Cue 79 — 00:05:13,420 --> 00:05:16,400

And so an inbox for that is incredibly valuable.

Cue 80 — 00:05:17,220 --> 00:05:20,700

So just a quick aside on how Workflow Engine would work when you're working with

Cue 81 — 00:05:20,740 --> 00:05:25,740

agentic coding. Well, typically a workflow node, you'd have inputs and outputs,

Cue 82 — 00:05:25,780 --> 00:05:29,060

right. So in this case you'd want the context.

Cue 83 — 00:05:29,100 --> 00:05:33,640

So you could say we have to have this artifact in order to run this workflow.

Cue 84 — 00:05:33,700 --> 00:05:39,420

Yeah. Instructions would generally be versioned skills because again,

Cue 85 — 00:05:39,460 --> 00:05:43,060

as a durable workflow engine, what we want to make sure is if we run this again

Cue 86 — 00:05:43,100 --> 00:05:46,780

later, that it's going to have exactly the same inputs, because that's where the

Cue 87 — 00:05:46,820 --> 00:05:48,860

magic comes from as we're about to see.

Cue 88 — 00:05:48,900 --> 00:05:52,500

Now, obviously, you want to be able to configure all the normal stuff like harness,

Cue 89 — 00:05:52,540 --> 00:05:57,420

model effort and everything like that, but also tools, because if your agent is

Cue 90 — 00:05:57,460 --> 00:06:00,500

doing some research, it doesn't need to be able to edit code,

Cue 91 — 00:06:00,540 --> 00:06:03,500

it doesn't need to connect to external environments.

Cue 92 — 00:06:03,540 --> 00:06:07,300

So by breaking it down into individual tasks, you can do a lot with the security

Cue 93 — 00:06:07,340 --> 00:06:11,980

side of stuff as well. But where the benefit really comes in is that now you've got

Cue 94 — 00:06:12,020 --> 00:06:16,100

individual tasks as part of workflows, which you're reusing for almost every piece

Cue 95 — 00:06:16,140 --> 00:06:20,660

of work that you do. And what that means is that you very quickly build up runtime

Cue 96 — 00:06:20,700 --> 00:06:23,740

evaluations, which tell you what's working and what isn't.

Cue 97 — 00:06:23,780 --> 00:06:28,620

Now, I'm sure the obvious outcome from that is that, hey, what about model choice?

Cue 98 — 00:06:28,660 --> 00:06:31,660

Yeah, a lot of the time. Don't know about you lot, but you know,

Cue 99 — 00:06:31,700 --> 00:06:36,340

for me, I choose the most expensive model because I know it's most likely to work.

Cue 100 — 00:06:36,380 --> 00:06:40,540

But I also know there's a lot of things that I do which could probably work on the

Cue 101 — 00:06:40,580 --> 00:06:46,420

cheaper models. So by using runtime evaluations from this kind of workflow engine,

Cue 102 — 00:06:46,460 --> 00:06:49,780

then you can actually have certainty over a large number of runs,

Cue 103 — 00:06:49,820 --> 00:06:54,060

what works and what trade offs you want to make between, in this case,

Cue 104 — 00:06:54,100 --> 00:06:59,940

quality and cost. So really this was the start of the software factory as I saw it,

Cue 105 — 00:06:59,980 --> 00:07:04,780

because the critical thing I think with software factories is it you're separating

Cue 106 — 00:07:04,820 --> 00:07:08,420

how you do the work from the actual work that you're doing.

Cue 107 — 00:07:08,460 --> 00:07:12,580

Yeah. And so now we could start to think about how do we improve the factory,

Cue 108 — 00:07:12,620 --> 00:07:14,920

or how do we improve the way that we do work.

Cue 109 — 00:07:15,620 --> 00:07:21,500

Now, one early example, which really pointed to me as to why we need to separate out

Cue 110 — 00:07:21,540 --> 00:07:26,540

the life cycle of these improvements from the code, as we discussed earlier,

Cue 111 — 00:07:26,580 --> 00:07:29,260

was when an issue went wrong. Right.

Cue 112 — 00:07:29,300 --> 00:07:32,900

So I think it kind of ended up costing lots and lots of money,

Cue 113 — 00:07:32,940 --> 00:07:36,380

a bit like the one we saw earlier and then ended up getting dumped anyway.

Cue 114 — 00:07:36,420 --> 00:07:41,220

So one of the first and most useful workflows I ever built in the engine was the

Cue 115 — 00:07:41,260 --> 00:07:43,100

retrospective workflow, right?

Cue 116 — 00:07:43,140 --> 00:07:48,180

Because that allows you to take in all of all of the run information,

Cue 117 — 00:07:48,220 --> 00:07:52,580

all of the artifacts from the run, all of the traces and telemetry and say,

Cue 118 — 00:07:52,620 --> 00:07:54,680

well, something went wrong. Why?

Cue 119 — 00:07:54,740 --> 00:07:58,820

Most importantly, what would need to improve to make sure that this didn't happen

Cue 120 — 00:07:58,860 --> 00:08:02,980

next time? So I ran a retrospective and it said, well, look,

Cue 121 — 00:08:03,020 --> 00:08:06,380

we're not checking for this really important thing in this skill.

Cue 122 — 00:08:06,420 --> 00:08:09,740

So let's add a few lines, make sure that it has a look at it in future.

Cue 123 — 00:08:09,780 --> 00:08:15,780

Great problem sorted. So what happens deploy that a few weeks later.

Cue 124 — 00:08:15,820 --> 00:08:20,180

Kind of realize like okay it was overfitting right?

Cue 125 — 00:08:20,220 --> 00:08:22,500

We solved the problem with that one particular issue.

Cue 126 — 00:08:22,540 --> 00:08:24,300

It's not helping anything else.

Cue 127 — 00:08:24,340 --> 00:08:28,180

So what's happened now is taking longer because the model is doing more work.

Cue 128 — 00:08:28,220 --> 00:08:32,660

It's costing more money and we're not really seeing any improvement from it.

Cue 129 — 00:08:32,700 --> 00:08:37,020

Yeah. So again, this was probably the point where I realized that all improvements

Cue 130 — 00:08:37,060 --> 00:08:41,700

are trade offs and is very hard for model to understand trade offs and systems.

Cue 131 — 00:08:41,740 --> 00:08:45,860

Right. And so really it should be you deciding what trade offs you're happy with and

Cue 132 — 00:08:45,900 --> 00:08:52,660

what you're not. So where we came to from here is, is a much better life cycle

Cue 133 — 00:08:52,700 --> 00:08:56,460

for improvements. And if we take this particular example of maybe changing the way a

Cue 134 — 00:08:56,500 --> 00:08:59,660

skill works and where we want the agent to put their attention.

Cue 135 — 00:08:59,700 --> 00:09:02,860

So now instead of just saying, yeah, that looks good.

Cue 136 — 00:09:02,900 --> 00:09:07,460

Let's merge it. Move on. Instead, what we can do is in the beauty of a workflow

Cue 137 — 00:09:07,500 --> 00:09:10,020

engine is you can rerun anything at any time, right?

Cue 138 — 00:09:10,060 --> 00:09:14,260

So you can say, okay, let's rerun this workflow on this issue ten times with the

Cue 139 — 00:09:14,300 --> 00:09:17,420

change and without and make sure that it helps locally.

Cue 140 — 00:09:17,460 --> 00:09:18,940

Right. Do we get better results.

Cue 141 — 00:09:18,980 --> 00:09:21,620

Is it cheaper. Whatever. The improvement we want to see is.

Cue 142 — 00:09:21,660 --> 00:09:24,740

But then critically after that we can say okay.

Cue 143 — 00:09:24,780 --> 00:09:28,140

But we want to make sure it helps everything and we're not regressing overall.

Cue 144 — 00:09:28,180 --> 00:09:32,740

So you might have a regression set of 50 to 100 different things,

Cue 145 — 00:09:32,780 --> 00:09:34,780

which are typical day to day work.

Cue 146 — 00:09:34,820 --> 00:09:39,240

And now we can rerun your improvement changes through the entire regression set.

Cue 147 — 00:09:39,340 --> 00:09:43,780

Yeah. Now that gives you the confidence to say before it's deployed that yes,

Cue 148 — 00:09:43,820 --> 00:09:47,940

this is actually going to improve the software factory rather than it's going to

Cue 149 — 00:09:47,980 --> 00:09:52,520

overfit for an issue and cost me more money like the previous improvement did.

Cue 150 — 00:09:53,700 --> 00:09:57,460

So really what you end up overall is it's kind of another frontier,

Cue 151 — 00:09:57,500 --> 00:09:59,260

right? Where where we're looking at.

Cue 152 — 00:09:59,300 --> 00:10:01,700

Should we have another evaluator here?

Cue 153 — 00:10:01,740 --> 00:10:03,820

Should we do additional tests on this?

Cue 154 — 00:10:03,860 --> 00:10:08,100

And we're always trying to evaluate different parameters against each other to think

Cue 155 — 00:10:08,140 --> 00:10:11,240

about what trade off we're most happy with.

Cue 156 — 00:10:12,660 --> 00:10:17,900

Now, I spent a long time building the workflow engine.

Cue 157 — 00:10:17,940 --> 00:10:22,500

Like I said, they are very complex, nitty gritty, and a lot of things can go wrong.

Cue 158 — 00:10:22,540 --> 00:10:27,540

So you can kind of imagine the elation when I finally got to run the workflow in

Cue 159 — 00:10:27,580 --> 00:10:34,720

prod for the first time, and I sat there waiting and waiting and waiting

Cue 160 — 00:10:34,780 --> 00:10:40,500

and 90 seconds. After I'd hit start on the workflow, an agent suddenly started doing

Cue 161 — 00:10:40,540 --> 00:10:45,580

work. Yeah. Now, if I take this back a step, I think we've all been told that you

Cue 162 — 00:10:45,620 --> 00:10:50,346

need isolation for agents. And so there was an assumption on my part that sandboxes

Cue 163 — 00:10:50,386 --> 00:10:55,880

are the way to go. Right? You put an agent in a sandbox and it's contained.

Cue 164 — 00:10:55,940 --> 00:10:58,380

It can't do too much damage. It has what it needs.

Cue 165 — 00:10:58,420 --> 00:11:05,300

Everything's great. Right? And really, a lot of that 90 seconds probably

Cue 166 — 00:11:05,340 --> 00:11:07,180

came from bad engineering choices from me.

Cue 167 — 00:11:07,220 --> 00:11:10,820

I'm not going to lie. Right. But there's certain things that you just can't get away

Cue 168 — 00:11:10,860 --> 00:11:15,340

from, right? If you're working with an agent in a sandbox, you have to have a copy

Cue 169 — 00:11:15,380 --> 00:11:18,300

of the source which which you're working on.

Cue 170 — 00:11:18,340 --> 00:11:24,860

Right? The right checkpoint. Now, most engineering now is working at a pace and it's

Cue 171 — 00:11:24,900 --> 00:11:28,420

constantly changing. And most of your issues are probably based on different

Cue 172 — 00:11:28,460 --> 00:11:31,340

checkpoints. So it's not like you can just say, hey, it's fine.

Cue 173 — 00:11:31,380 --> 00:11:32,820

We've got a snapshot over here.

Cue 174 — 00:11:32,860 --> 00:11:36,460

Let's fire it up. You're always going to have to do something to get the source in

Cue 175 — 00:11:36,500 --> 00:11:40,580

place on that. And on top of that, because you've got an agent working in a sandbox,

Cue 176 — 00:11:40,620 --> 00:11:43,660

you're having to do a lot of work for security, right?

Cue 177 — 00:11:43,700 --> 00:11:45,020

They can change anything in there.

Cue 178 — 00:11:45,060 --> 00:11:47,580

So you need to have the confidence that what they've done is right.

Cue 179 — 00:11:47,620 --> 00:11:49,780

You need to be triggering it in the first place.

Cue 180 — 00:11:49,820 --> 00:11:53,740

Yeah. And you need to be checking the artifacts that they generate and pushing them

Cue 181 — 00:11:53,780 --> 00:11:57,780

back into your system. So all of these things were starting to push towards the fact

Cue 182 — 00:11:57,820 --> 00:12:00,800

that, like, maybe sandboxes weren't the right primitive here.

Cue 183 — 00:12:01,700 --> 00:12:05,380

Finally, there was there was kind of a barrier which you just can't really get

Cue 184 — 00:12:05,420 --> 00:12:10,500

around with sandboxes. Generally when you create them, they have the set resources,

Cue 185 — 00:12:10,540 --> 00:12:15,100

which you start with. Yeah. So if you consider a session maybe where you're planning

Cue 186 — 00:12:15,140 --> 00:12:20,273

on doing some research. So really just reviewing the code, understanding what's

Cue 187 — 00:12:20,313 --> 00:12:23,020

there already so that you've got the groundwork set for something that you're

Cue 188 — 00:12:23,060 --> 00:12:27,060

building. You might start that with a one gig sandbox in terms of memory,

Cue 189 — 00:12:27,100 --> 00:12:29,620

because you don't need to do a lot with it, right?

Cue 190 — 00:12:29,660 --> 00:12:33,140

But what happens then if there's some uncertainty and actually you need to run some

Cue 191 — 00:12:33,180 --> 00:12:36,860

tests to make sure there isn't a bug in that code before you go and produce a

Cue 192 — 00:12:36,900 --> 00:12:40,600

report? Well, you might not have enough resources to do that.

Cue 193 — 00:12:41,140 --> 00:12:45,180

So maybe you run with bigger sandboxes and it costs you more money,

Cue 194 — 00:12:45,220 --> 00:12:47,620

and then maybe you need to do something that you still can't.

Cue 195 — 00:12:47,660 --> 00:12:52,340

So sandbox resource sizing can be very difficult when you're working at scale,

Cue 196 — 00:12:52,380 --> 00:12:55,320

and you need to be able to do different things very quickly.

Cue 197 — 00:12:55,660 --> 00:13:00,460

So having had this success in terms of decomposing the workflows into the smallest

Cue 198 — 00:13:00,500 --> 00:13:06,460

possible components, I wondered, well, what happens if you decompose a harness into

Cue 199 — 00:13:06,500 --> 00:13:09,920

different components so that you can optimize each of these differently?

Cue 211 — 00:13:52,260 --> 00:13:57,420

And that's a really, really interesting concept, because what we start to see is

Cue 212 — 00:13:57,460 --> 00:14:02,140

that the agent loop itself, as we said earlier, it's very simple.

Cue 213 — 00:14:02,180 --> 00:14:06,180

It can be as little as 300 lines of code, but it's relatively trusted code.

Cue 214 — 00:14:06,220 --> 00:14:08,780

All you're doing is you're looping your model, right.

Cue 215 — 00:14:08,820 --> 00:14:11,860

You're making sure the request keep going to the model and that's it,

Cue 216 — 00:14:11,900 --> 00:14:15,060

right? It doesn't inherently have access to the file system.

Cue 217 — 00:14:15,100 --> 00:14:17,100

It can't inherently execute code.

Cue 218 — 00:14:17,140 --> 00:14:23,140

It can't do anything right. So actually you can have multiple agent loops or

Cue 219 — 00:14:23,180 --> 00:14:27,620

multiple harnesses running in one environment with no isolation or security

Cue 220 — 00:14:27,660 --> 00:14:31,460

concerns. Yeah. So that was the kind of first point where it's like,

Cue 221 — 00:14:31,500 --> 00:14:36,500

okay, so if we can now do this, then what about those tools which it definitely

Cue 222 — 00:14:36,540 --> 00:14:41,060

needs access to. And again, that's where it gets really, really interesting because

Cue 223 — 00:14:41,100 --> 00:14:45,620

running a factory with a control plane, you kind of already have a source of truth

Cue 224 — 00:14:45,660 --> 00:14:49,100

for those things. So your platform tools already exist.

Cue 225 — 00:14:49,140 --> 00:14:52,760

Yeah. And all you need to do now is connect via MCP.

Cue 226 — 00:14:53,140 --> 00:14:55,740

Great. It's got access to the tools that you need.

Cue 227 — 00:14:55,780 --> 00:15:00,140

And as long as you sort out your security properly, then it only has access to the

Cue 228 — 00:15:00,180 --> 00:15:02,920

tools that it needs to do that specific job.

Cue 229 — 00:15:03,500 --> 00:15:10,060

Now also MCP brought out skills over MCP, which is another really good innovation

Cue 230 — 00:15:10,100 --> 00:15:14,340

because what this now means is that we don't need to worry about what version of

Cue 231 — 00:15:14,380 --> 00:15:19,060

skills are in the repo. We don't need to worry about whether it's the latest or when

Cue 232 — 00:15:19,100 --> 00:15:23,340

things change. How do we get that across all the different work which is going on?

Cue 233 — 00:15:23,380 --> 00:15:26,220

Instead, it comes from the control plane.

Cue 234 — 00:15:26,260 --> 00:15:29,900

Skills are versioned, and we know that anytime something is executing,

Cue 235 — 00:15:29,940 --> 00:15:32,480

it's working on the latest version of the skill.

Cue 236 — 00:15:32,540 --> 00:15:36,300

It's not what the talks about today, but skills have a separate lifecycle in the

Cue 237 — 00:15:36,340 --> 00:15:40,400

same way that code does. And yeah, improvements in general.

Cue 238 — 00:15:41,020 --> 00:15:43,020

So now getting to the toffee bits right.

Cue 239 — 00:15:43,060 --> 00:15:45,740

Our agent actually needs to do some stuff with our code.

Cue 240 — 00:15:45,780 --> 00:15:48,520

So it needs to read and it needs to write.

Cue 241 — 00:15:49,140 --> 00:15:53,220

But here's the thing. Like having to have a copy of the source on every single

Cue 242 — 00:15:53,260 --> 00:15:56,740

sandbox is very inefficient in the first place, and it's generally tied to a

Cue 243 — 00:15:56,780 --> 00:16:02,020

specific version. So what happens if instead you have one tool for all your agents

Cue 244 — 00:16:02,060 --> 00:16:06,180

which, like git, can instantly access any checkpoint, right?

Cue 245 — 00:16:06,220 --> 00:16:10,340

What if you can read any checkpoint, write any checkpoint at any period in time?

Cue 246 — 00:16:10,380 --> 00:16:13,060

Right. Suddenly that's your problem solved.

Cue 247 — 00:16:13,100 --> 00:16:17,580

Yeah. And actually, that's quite doable with modern file systems.

Cue 248 — 00:16:17,620 --> 00:16:21,940

I mean, I think it takes less than 70 milliseconds or something like that to spin up

Cue 249 — 00:16:21,980 --> 00:16:26,180

a new writable volume, which is based on a current checkpoint.

Cue 250 — 00:16:26,220 --> 00:16:30,340

And, you know, like I say, complete isolation, no security issues.

Cue 251 — 00:16:30,380 --> 00:16:33,880

So that part seemed like it was solvable.

Cue 252 — 00:16:33,940 --> 00:16:37,300

And then you come back to, to the real bugbear, which with our execution

Cue 253 — 00:16:37,340 --> 00:16:43,100

environments. Right. And what we really need here is, is something dynamic,

Cue 254 — 00:16:43,140 --> 00:16:47,560

right. Actually 95% of the time.

Cue 255 — 00:16:47,740 --> 00:16:52,380

Plus I think on latest traces, agents aren't executing anything.

Cue 256 — 00:16:52,420 --> 00:16:54,420

Right. So it's not really a problem.

Cue 257 — 00:16:54,460 --> 00:16:57,460

Yeah. Now the few times they are when they've done their work,

Cue 258 — 00:16:57,500 --> 00:16:59,220

they've built something they want to test.

Cue 259 — 00:16:59,260 --> 00:17:02,540

They want to check. Now we need an execution environment.

Cue 260 — 00:17:02,580 --> 00:17:07,180

But the beauty of separating it is that now you can have a look at it and go,

Cue 261 — 00:17:07,220 --> 00:17:08,900

what are they trying to run? Right.

Cue 262 — 00:17:08,940 --> 00:17:10,620

We've got set type checks. Okay.

Cue 263 — 00:17:10,660 --> 00:17:15,460

It's running this one. How much was do we need to actually execute this now?

Cue 264 — 00:17:15,500 --> 00:17:19,580

And what we can start doing is dynamically provisioning the environments they

Cue 265 — 00:17:19,620 --> 00:17:24,160

actually need to execute whatever it is that they need to do.

Cue 266 — 00:17:24,500 --> 00:17:28,540

So this becomes something else that your factory can optimize.

Cue 267 — 00:17:28,580 --> 00:17:32,220

Is that how do we optimize the environments for execution?

Cue 268 — 00:17:32,260 --> 00:17:36,700

Yeah. And actually it doesn't take a lot of data to get very good at that.

Cue 269 — 00:17:36,740 --> 00:17:43,540

Yeah. So what like I say, what we end up with is components which you can run in

Cue 270 — 00:17:43,580 --> 00:17:49,780

different places. Yeah. So the agentic runtime now I could run my agent loop on

Cue 271 — 00:17:49,820 --> 00:17:54,880

Daytona or Cloudflare is quite popular because it doesn't require a lot of RAM.

Cue 272 — 00:17:54,980 --> 00:17:57,300

I could run share VMs on another provider.

Cue 273 — 00:17:57,340 --> 00:17:59,420

I could do this here. I could do that there.

Cue 274 — 00:17:59,460 --> 00:18:05,100

Right. But actually, what I found is that when you're only using the resources which

Cue 275 — 00:18:05,140 --> 00:18:09,540

you actually need, it's not taking anywhere near as much as if you had individual

Cue 276 — 00:18:09,580 --> 00:18:16,380

sandboxes. And actually, in our case, I mean, I tend to average about ten agents at

Cue 277 — 00:18:16,420 --> 00:18:21,700

a given time. Running are probably a lot more during the day and a lot less at

Cue 278 — 00:18:21,740 --> 00:18:23,580

night. But that's kind of the average.

Cue 279 — 00:18:23,620 --> 00:18:28,980

And what's really interesting is that actually one VM can handle almost all of this

Cue 280 — 00:18:29,020 --> 00:18:32,460

except for the control plane, which you always want to be kind of cloud and separate

Cue 281 — 00:18:32,500 --> 00:18:37,740

anyway. So if we come back to that original problem of like,

Cue 282 — 00:18:37,780 --> 00:18:39,540

yay, I've finished the workflow engine.

Cue 283 — 00:18:39,580 --> 00:18:41,880

Oh god, it took 90 seconds to load.

Cue 284 — 00:18:42,260 --> 00:18:46,620

So now the load time went from 90 seconds to two seconds.

Cue 285 — 00:18:46,660 --> 00:18:49,300

And there's probably more we could shave off that if we wanted to.

Cue 286 — 00:18:49,340 --> 00:18:52,740

But like, now we're in the ballpark of where we actually want to be.

Cue 287 — 00:18:52,780 --> 00:18:55,600

And we've got a whole load of advantages along the way.

Cue 288 — 00:18:56,100 --> 00:19:00,740

Yeah. Now the secondary, which I wasn't really thinking about when I started working

Cue 289 — 00:19:00,780 --> 00:19:03,700

on this new runtime, was cost.

Cue 290 — 00:19:03,740 --> 00:19:09,580

Because if you're running ten sandboxes at any given moment with an average of four

Cue 291 — 00:19:09,620 --> 00:19:14,020

gig RAM so your agents can actually edit and execute, then that's probably costing

Cue 292 — 00:19:14,060 --> 00:19:17,500

you a decent amount of money. Post dollars a month.

Cue 293 — 00:19:17,540 --> 00:19:21,380

If you're running less RAM, sure it's going to cost you money.

Cue 294 — 00:19:21,420 --> 00:19:27,840

But the thing is, if you can handle it all on one VM, it's costing $53 a month now,

Cue 295 — 00:19:28,100 --> 00:19:35,020

and the capacity is probably there to run about 50% more without increasing the cost

Cue 296 — 00:19:35,060 --> 00:19:39,260

at all. And again, it all comes just because we've decomposed and said,

Cue 297 — 00:19:39,300 --> 00:19:42,900

what are the different elements that we need and how do we optimize that?

Cue 298 — 00:19:42,940 --> 00:19:46,300

And again, that would be my thesis that the factory should be in charge of those

Cue 299 — 00:19:46,340 --> 00:19:47,960

optimizations as well.

Cue 300 — 00:19:49,220 --> 00:19:53,140

So another embarrassing story after my wonderful 90-second startup.

Cue 301 — 00:19:53,180 --> 00:19:56,940

So, CI. I'm sure we all love CI.

Cue 302 — 00:19:56,980 --> 00:20:02,000

And we've seen it kind of change in front of our eyes with agentic coding.

Cue 303 — 00:20:02,580 --> 00:20:06,380

But at one point, I think it was just before I'd finished a workflow engine.

Cue 304 — 00:20:06,420 --> 00:20:10,380

I was starting to get quite a few bugs creep into production.

Cue 305 — 00:20:10,420 --> 00:20:15,560

Right. And the problem when you're dogfooding your own project and you have bugs,

Cue 306 — 00:20:15,660 --> 00:20:17,540

everything comes to a standstill, right?

Cue 307 — 00:20:17,580 --> 00:20:19,580

And you have to drop whatever you're doing and get it sorted,

Cue 308 — 00:20:19,620 --> 00:20:21,340

because otherwise you can't keep developing.

Cue 309 — 00:20:21,380 --> 00:20:23,380

So it's a really painful position.

Cue 310 — 00:20:23,420 --> 00:20:25,620

So it's like, no, it's fine. I know what to do.

Cue 311 — 00:20:25,660 --> 00:20:27,700

I'm going to set aside a couple of days.

Cue 312 — 00:20:27,740 --> 00:20:32,460

I'm going to do loads of optimization, put together great everything that I need.

Cue 313 — 00:20:32,500 --> 00:20:35,820

I'm going to shard it nicely. I'm going to use Blacksmith because I've heard that

Cue 314 — 00:20:35,860 --> 00:20:38,140

that's much cheaper and faster than GitHub Actions.

Cue 315 — 00:20:38,180 --> 00:20:40,940

I'll get everything going and I'll forget about it.

Cue 316 — 00:20:40,980 --> 00:20:43,120

And it did. It would work. It was great.

Cue 317 — 00:20:43,300 --> 00:20:48,180

Except a free trial. Two weeks later, I get an email saying,

Cue 318 — 00:20:48,220 --> 00:20:53,060

here's your invoice for $250. So always read the small print.

Cue 319 — 00:20:53,100 --> 00:20:55,960

They only cover about $10 worth on the free trial.

Cue 320 — 00:20:56,700 --> 00:21:00,700

So clearly again, we're talking about improvements and trade offs.

Cue 321 — 00:21:00,740 --> 00:21:04,790

This didn't feel scalable to me, particularly when you now take the knowledge of how

Cue 322 — 00:21:04,830 --> 00:21:07,360

how much it costs to run the agent runtime.

Cue 323 — 00:21:07,620 --> 00:21:12,460

So the next one, like I said, this was just before we kind of got the workflow

Cue 324 — 00:21:12,500 --> 00:21:16,900

engine live on sandboxes. The next one was, okay, let's run them locally.

Cue 325 — 00:21:16,940 --> 00:21:20,740

Instead, let's cut down the number of tests to the core and what we really need,

Cue 326 — 00:21:20,780 --> 00:21:23,180

and let's just run that when we're about to merge.

Cue 327 — 00:21:23,220 --> 00:21:29,020

But then what happens is, okay, it's running quicker now and we got less problems,

Cue 328 — 00:21:29,060 --> 00:21:34,520

except if 2 or 3 agents wanted to do this at the same time, the laptop would crash,

Cue 329 — 00:21:34,740 --> 00:21:36,740

right? We'd run into issues there.

Cue 330 — 00:21:36,780 --> 00:21:41,480

So another one where it's like, okay, the trade offs here aren't really worth doing.

Cue 331 — 00:21:42,060 --> 00:21:42,860

Now,

Cue 332 — 00:21:44,060 --> 00:21:50,020

having had the individual components coming together now, a workflow engine and the

Cue 333 — 00:21:50,060 --> 00:21:53,900

agent runtime, it was becoming more and more clear that there's actually another

Cue 334 — 00:21:53,940 --> 00:21:59,060

solution here, because when you think about talking about our dynamic execution

Cue 335 — 00:21:59,100 --> 00:22:04,340

environments and what the agents need to run, you can start to see a smaller and

Cue 336 — 00:22:04,380 --> 00:22:09,020

smaller gap between what happens in the agent runtime and what would traditionally

Cue 337 — 00:22:09,060 --> 00:22:14,420

happen during CI. Yeah. So the question in my mind was, how do we bring these things

Cue 338 — 00:22:14,460 --> 00:22:19,020

together? Yeah. Now, from an actual execution point of view,

Cue 339 — 00:22:19,060 --> 00:22:22,540

that's dead easy. You just run your CI in the same environments that we've been

Cue 340 — 00:22:22,580 --> 00:22:27,620

talking about, right? But then the problem becomes, how do you do that in an

Cue 341 — 00:22:27,660 --> 00:22:32,220

efficient way? Because actually most of the tests that we run in CI,

Cue 342 — 00:22:32,260 --> 00:22:36,060

we don't really need at all. But it's so much effort to work out what we do we don't

Cue 343 — 00:22:36,100 --> 00:22:38,740

need. We end up running them anyway, right?

Cue 344 — 00:22:38,780 --> 00:22:44,460

So the question in my mind was, how do we get to a point where we can say with

Cue 345 — 00:22:44,500 --> 00:22:49,300

certainty for one edit exactly what tests, checks, linting, standards,

Cue 346 — 00:22:49,340 --> 00:22:52,560

anything else that you want to run for that specific edit?

Cue 347 — 00:22:53,260 --> 00:22:59,020

And the answer came in the form of code graph or symbol graph if you like.

Cue 348 — 00:22:59,060 --> 00:23:05,260

Yeah. Because what did is essentially starting to move away from the file and folder

Cue 349 — 00:23:05,300 --> 00:23:11,220

system of git and more towards a graph database where you know and you understand

Cue 350 — 00:23:11,260 --> 00:23:14,180

the relationships. And if you look at a symbol for a function,

Cue 351 — 00:23:14,220 --> 00:23:17,700

you can see the relationship to the input and output contracts.

Cue 352 — 00:23:17,740 --> 00:23:19,700

You can see upstream and downstream.

Cue 353 — 00:23:19,740 --> 00:23:22,320

You can see what tests are running there and why.

Cue 354 — 00:23:23,140 --> 00:23:26,860

And interestingly enough, you can link back to those decisions that we talked about

Cue 355 — 00:23:26,900 --> 00:23:28,900

right at the beginning of the presentation.

Cue 356 — 00:23:28,940 --> 00:23:34,180

So suddenly you can start to build a very, very rich graph of your code,

Cue 357 — 00:23:34,220 --> 00:23:36,460

which the agents can use for efficiency.

Cue 358 — 00:23:36,500 --> 00:23:39,960

But also we can start to think about CI.

Cue 359 — 00:23:40,580 --> 00:23:47,500

Yeah. So spoiler it worked. So like most things when you're improving

Cue 360 — 00:23:47,540 --> 00:23:50,940

the factory, you probably need to do quite a lot of work on your code base as well.

Cue 361 — 00:23:50,980 --> 00:23:54,260

So there are a lot of patterns which we were using to call functions,

Cue 362 — 00:23:54,300 --> 00:23:57,980

which weren't great because it was quite hard to follow cheaply.

Cue 363 — 00:23:58,020 --> 00:24:04,420

So it became a part in part kind of refactoring and part improving detection.

Cue 364 — 00:24:04,460 --> 00:24:09,300

Right. But once these exercises are finished, we're getting down to about 15 seconds

Cue 365 — 00:24:09,340 --> 00:24:12,620

for an edit. Now, I know that sounds a lot because we're thinking,

Cue 366 — 00:24:12,660 --> 00:24:16,640

well, I wouldn't want to sit there working on something and sitting around for 15

Cue 367 — 00:24:16,680 --> 00:24:18,820

seconds before I got the go ahead, right.

Cue 368 — 00:24:18,860 --> 00:24:23,540

But these are agents, right? And what we've really done is we've replaced the entire

Cue 369 — 00:24:23,580 --> 00:24:30,580

huge loop of of integrating to the latest version, opening the PR,

Cue 370 — 00:24:30,620 --> 00:24:35,900

letting the CI run, then getting the results back and then having to go and fix it

Cue 371 — 00:24:35,940 --> 00:24:38,080

and go through the whole loop again.

Cue 372 — 00:24:38,260 --> 00:24:41,660

But what we're really doing now is we're pulling all of that forward,

Cue 373 — 00:24:41,700 --> 00:24:46,740

you know, that kind of shift left which really embodies factories from the DevOps.

Cue 374 — 00:24:46,780 --> 00:24:47,940

And we're pulling that forward.

Cue 375 — 00:24:47,980 --> 00:24:52,260

So now when there's an issue a bit like I say test type, check,

Cue 376 — 00:24:52,300 --> 00:24:56,002

whatever it might be, the agent sees it straight away while they're still in

Cue 377 — 00:24:56,042 --> 00:24:59,620

session, they've still got the context, they've still got the tools that they need.

Cue 378 — 00:24:59,660 --> 00:25:04,820

Yeah. And they're able to actually do it much, much quicker than if they were

Cue 379 — 00:25:04,860 --> 00:25:06,560

waiting for the CI to finish.

Cue 380 — 00:25:07,980 --> 00:25:11,860

So we've talked about a range of things over improvements.

Cue 381 — 00:25:11,900 --> 00:25:14,220

So let's let's let's just kind of recap quickly right.

Cue 382 — 00:25:14,260 --> 00:25:18,580

So first of all we started by saying improvements usually a trade off right.

Cue 383 — 00:25:18,620 --> 00:25:21,820

And we've seen how there's lots of examples here where you've got kind of better

Cue 384 — 00:25:21,860 --> 00:25:26,200

quality. But actually it ended up costing more money or taking more time.

Cue 385 — 00:25:26,260 --> 00:25:29,540

Yeah. Second of all decomposition.

Cue 386 — 00:25:29,580 --> 00:25:34,300

Right. It's more obvious to think about atomic decomposition when it comes to tasks

Cue 387 — 00:25:34,340 --> 00:25:38,260

and workflows, but we can see how effective that can be with your agent runtime as

Cue 388 — 00:25:38,300 --> 00:25:41,660

well. And actually decomposing the harness into an agent runtime,

Cue 389 — 00:25:41,700 --> 00:25:46,680

which replaces your sandboxes and your CI, is quite feasible at the moment.

Cue 390 — 00:25:48,140 --> 00:25:52,060

So we also established that we need a separate life cycle to code.

Cue 391 — 00:25:52,100 --> 00:25:55,460

Right. Because your improvements shouldn't be done by agents.

Cue 392 — 00:25:55,500 --> 00:25:59,700

They should be proved through evidence that they actually improve your factory

Cue 393 — 00:25:59,740 --> 00:26:01,900

without causing regressions along the way.

Cue 394 — 00:26:01,940 --> 00:26:06,460

Yeah. And finally, like I said, the hottest take of all sandbox is the wrong runtime

Cue 395 — 00:26:06,500 --> 00:26:09,380

for factory. Yeah, happy to have that discussion.

Cue 396 — 00:26:09,420 --> 00:26:13,780

But for me, it seems fairly clear that we can do a lot more when we break down those

Cue 397 — 00:26:13,820 --> 00:26:17,280

components and say, what do we need and how do we optimize them?

Cue 398 — 00:26:17,660 --> 00:26:22,100

So to come back to the question that this talk is about, did it actually improve the

Cue 399 — 00:26:22,140 --> 00:26:26,860

software factory? I think the simplest way to think about it is don't ask what it

Cue 400 — 00:26:26,900 --> 00:26:29,280

made better ask what it made worse.

Cue 401 — 00:26:29,380 --> 00:26:30,180

Thank you.

skills

talk-poulter-factory-improvement

tile.json