CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-moss-concurrent-agent-risk/references/

Corrected transcript: Dealing with Agentic Coding Risk: What Breaks Going From 1 Agent to 10

James Moss · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=B4sTAoiyR_k

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:00,460 --> 00:00:03,180

Morning, everyone. Nice to meet you all.

Cue 2 — 00:00:03,220 --> 00:00:07,940

Welcome to the lower level. We're kind of in a weird little hidden away little spot

Cue 3 — 00:00:07,980 --> 00:00:09,820

down here, but, I think it's quite nice.

Cue 4 — 00:00:09,860 --> 00:00:11,100

I quite like it. Sort of small.

Cue 5 — 00:00:11,140 --> 00:00:13,980

Just a few of us. It's the best room.

Cue 6 — 00:00:14,020 --> 00:00:14,920

Thank you.

Cue 7 — 00:00:16,740 --> 00:00:18,380

Okay. Before we get into things about me.

Cue 8 — 00:00:18,420 --> 00:00:22,360

My name is James. I'm a member of the technical staff at Tessl.

Cue 9 — 00:00:22,740 --> 00:00:28,175

I've got around 25 years of software engineering, so doing this quite a long time at

Cue 10 — 00:00:28,215 --> 00:00:30,520

Tessl. I work on a bunch of different stuff.

Cue 11 — 00:00:30,580 --> 00:00:35,780

This year, I've been leading software engineering on our context inventory product

Cue 12 — 00:00:35,820 --> 00:00:38,140

and also working on the factory as well.

Cue 13 — 00:00:38,180 --> 00:00:41,860

So some of the stuff that Guy just talked about, you've seen in the keynote we're

Cue 14 — 00:00:41,900 --> 00:00:43,200

working on as well.

Cue 15 — 00:00:44,420 --> 00:00:48,700

And I love software factories, which is what I'm going to be talking a bit about

Cue 16 — 00:00:48,740 --> 00:00:54,300

that today. I think what I really love is actually the kind of productivity gains

Cue 17 — 00:00:54,340 --> 00:00:58,900

that they enable. I've been doing startups now for about 15 years,

Cue 18 — 00:00:58,940 --> 00:01:01,460

and more than ever, I can kind of build something end to end,

Cue 19 — 00:01:01,500 --> 00:01:04,500

just on my own and ship it to a customer and get it live very,

Cue 20 — 00:01:04,540 --> 00:01:07,680

very quickly. So yeah, big fan of that.

Cue 21 — 00:01:08,220 --> 00:01:10,540

So what are we actually talking about today?

Cue 22 — 00:01:10,580 --> 00:01:13,860

I want to give you a bit of a run through of some of the issues that we've run into

Cue 23 — 00:01:13,900 --> 00:01:17,740

while we've been building our, our kind of factory product and the learnings that

Cue 24 — 00:01:17,780 --> 00:01:21,960

came out of those issues. Kind of over the last six months.

Cue 25 — 00:01:23,100 --> 00:01:27,840

I've categorized these into around five topics.

Cue 26 — 00:01:29,300 --> 00:01:30,800

I just lost the mouse.

Cue 27 — 00:01:32,180 --> 00:01:37,220

Ultimately, you have multiple coding agents sharing one delivery system.

Cue 28 — 00:01:37,260 --> 00:01:40,860

So using things like GitHub or Linear, we've got these tools that have been built

Cue 29 — 00:01:40,900 --> 00:01:44,460

for humans and not for agent first workflows.

Cue 30 — 00:01:44,500 --> 00:01:46,700

And that ends up causing problems.

Cue 31 — 00:01:46,740 --> 00:01:50,820

So there's the five topics we're going to jump into today are shared state.

Cue 32 — 00:01:50,860 --> 00:01:55,220

So where you've got agents both using the same thing or working on the same kind of

Cue 33 — 00:01:55,260 --> 00:02:00,420

area of the code, stale assumptions where the agent is doing some work and the world

Cue 34 — 00:02:00,460 --> 00:02:04,480

kind of changes it around it, and it doesn't update its understanding of the world

Cue 35 — 00:02:05,300 --> 00:02:09,940

convergence where the agent is doing work, it's spinning its wheels,

Cue 36 — 00:02:09,980 --> 00:02:12,700

it's using tokens, but you're not actually kind of going anywhere.

Cue 37 — 00:02:12,740 --> 00:02:18,640

You're not sort of getting close to some kind of target and duplicate work,

Cue 38 — 00:02:19,100 --> 00:02:22,760

and kind of error states and recovery and how you kind of recover from those.

Cue 39 — 00:02:23,100 --> 00:02:27,500

And then finally looking at kind of control and budgets when things do spiral out of

Cue 40 — 00:02:27,540 --> 00:02:32,420

control, what can you do? And kind of stop conditions that you need to have in place

Cue 41 — 00:02:32,460 --> 00:02:33,280

for that.

Cue 42 — 00:02:34,900 --> 00:02:39,720

So a bit of a brief introduction to software factories at Tessl.

Cue 43 — 00:02:39,980 --> 00:02:43,160

And you notice that factories there has an s.

Cue 44 — 00:02:43,620 --> 00:02:47,800

So we have not just built one software factory at Tessl with many.

Cue 45 — 00:02:48,180 --> 00:02:51,820

I think we're probably on our fourth iteration at the moment.

Cue 46 — 00:02:51,860 --> 00:02:56,620

I would say, some of them were experiments, some of them were just toys,

Cue 47 — 00:02:56,660 --> 00:03:00,320

things we throw away. But we learn a bunch along the way.

Cue 48 — 00:03:00,700 --> 00:03:04,960

And yeah, hopefully you guys are going to learn from some mistakes that we've made.

Cue 49 — 00:03:05,180 --> 00:03:10,260

So a bit of history, we kind of first kicked off our initials work and investigation

Cue 50 — 00:03:10,300 --> 00:03:14,740

on this back in April and playing around with some sort of ticket to power

Cue 51 — 00:03:14,780 --> 00:03:18,660

experiments. So looking at Linear and able to delegate to agents and then Linear,

Cue 52 — 00:03:18,700 --> 00:03:22,240

but very much not kind of a fully agent.

Cue 53 — 00:03:22,940 --> 00:03:28,100

We we gave a small team some time and some space to work on this.

Cue 54 — 00:03:28,140 --> 00:03:34,300

And after around a couple of weeks were they came up with something that we called

Cue 55 — 00:03:34,340 --> 00:03:39,160

Kikimora. So this was our first kind of internal version of a factory.

Cue 56 — 00:03:39,460 --> 00:03:44,760

For those that don't know, Kikimora is a Slavic mythological creature.

Cue 57 — 00:03:44,860 --> 00:03:47,980

She comes and visits you in the middle of the night in your house.

Cue 58 — 00:03:48,020 --> 00:03:52,740

If you if your house is clean and tidy, she'll actually help make your house even

Cue 59 — 00:03:52,780 --> 00:03:58,020

tidier and cleaner. But if your house is dirty and messy, then she's going to mess

Cue 60 — 00:03:58,060 --> 00:04:00,800

it up even more, right? She's going to break your pots and pans.

Cue 61 — 00:04:01,220 --> 00:04:02,740

Kind of a weird name for a factory, right?

Cue 62 — 00:04:02,780 --> 00:04:04,940

Like, why would you want a factory that comes and breaks loads of stuff,

Cue 63 — 00:04:04,980 --> 00:04:10,420

but we kind of saw it as if we have our context and everything else in a good state.

Cue 64 — 00:04:10,460 --> 00:04:13,100

Then we know that the factory will make that better.

Cue 65 — 00:04:13,140 --> 00:04:15,780

And likewise, conversely, if things are not in a good state,

Cue 66 — 00:04:15,820 --> 00:04:17,920

then we're going to end up in a real mess.

Cue 67 — 00:04:20,260 --> 00:04:24,840

So that initial version was just built on one of our developers laptops.

Cue 68 — 00:04:25,140 --> 00:04:27,840

She was running it from her laptop.

Cue 69 — 00:04:27,900 --> 00:04:31,980

So we would often get messages like this if you can't see it at the back,

Cue 70 — 00:04:32,020 --> 00:04:35,040

says Kikimora is going offline while I commute to DevCon.

Cue 71 — 00:04:36,020 --> 00:04:39,560

So every time she'd close the lid on her laptop, that factory would pause.

Cue 72 — 00:04:40,300 --> 00:04:42,220

But it was great. It was really compelling.

Cue 73 — 00:04:42,260 --> 00:04:43,980

People were excited about using it.

Cue 74 — 00:04:44,020 --> 00:04:48,300

By early June, we'd put that on to an EC2 instance, and we'd started rolling that

Cue 75 — 00:04:48,340 --> 00:04:51,820

out to the entire company, and we saw pretty fast adoption, which I'll talk about in

Cue 76 — 00:04:51,860 --> 00:04:52,660

a minute.

Cue 77 — 00:04:54,500 --> 00:05:01,220

By the end of sorry, by mid-June, we had around 31% of all of our PRs going through

Cue 78 — 00:05:01,260 --> 00:05:04,000

Kikimora through that initial version of the software factory.

Cue 79 — 00:05:04,420 --> 00:05:09,960

And by the end of July, we'd got up to 85% of all the PRs within our,

Cue 80 — 00:05:10,000 --> 00:05:12,360

our company going through there.

Cue 81 — 00:05:13,220 --> 00:05:17,640

And we learned a bunch of stuff kind of going through this process.

Cue 82 — 00:05:18,500 --> 00:05:24,600

And then lastly, in early October, we swapped over from Kikimora to our Tessl

Cue 83 — 00:05:24,640 --> 00:05:28,080

factory product, which is what Guy was talking about on stage earlier,

Cue 84 — 00:05:28,260 --> 00:05:30,140

that kind of distilled all the learnings.

Cue 85 — 00:05:30,180 --> 00:05:34,960

A lot of the stuff that we talk about today into kind of best practices.

Cue 86 — 00:05:35,220 --> 00:05:37,860

And actually it was surprisingly easy to switch over as well,

Cue 87 — 00:05:37,900 --> 00:05:39,180

which is which is pretty awesome.

Cue 88 — 00:05:39,220 --> 00:05:42,620

I think we had one morning where someone said, hey, we're switching over.

Cue 89 — 00:05:42,660 --> 00:05:45,000

All of it started working. It's kind of amazing.

Cue 90 — 00:05:45,140 --> 00:05:47,220

And that's because we're not really working in the factory.

Cue 91 — 00:05:47,260 --> 00:05:50,880

We're working around the factory, which again, I'll talk about in a minute.

Cue 92 — 00:05:52,420 --> 00:05:55,080

So I'm looking at some of the numbers,

Cue 93 — 00:05:56,580 --> 00:05:58,880

and talking about kind of the rollout.

Cue 94 — 00:06:00,020 --> 00:06:05,260

This is PRs merged per week in a monorepo, which is basically all the work that's

Cue 95 — 00:06:05,300 --> 00:06:11,780

done across 2026. So you can see there pre Kikimora, we were we were

Cue 96 — 00:06:11,820 --> 00:06:18,120

merging around I think it was 100, 150 PRs a week.

Cue 97 — 00:06:19,020 --> 00:06:23,420

Obviously PRs are not like, you know, a great definitive metric for measuring like

Cue 98 — 00:06:23,460 --> 00:06:25,020

productivity within the company.

Cue 99 — 00:06:25,060 --> 00:06:28,220

As much as CTOs would kind of love that to be the case.

Cue 100 — 00:06:28,260 --> 00:06:31,300

But there are kind of good like directional indicator in terms of like,

Cue 101 — 00:06:31,340 --> 00:06:32,620

what are we shipping? What's going on.

Cue 102 — 00:06:32,660 --> 00:06:34,840

Right. And so,

Cue 103 — 00:06:36,100 --> 00:06:39,180

you can see here, as soon as we rolled that out, we started seeing a huge uptick,

Cue 104 — 00:06:39,220 --> 00:06:43,020

right. It was actually three and a half times the number of PRs that we were able to

Cue 105 — 00:06:43,060 --> 00:06:48,420

start shipping per week just by kind of using this factory model today,

Cue 106 — 00:06:48,460 --> 00:06:52,100

I'm sure we got up to 85%. Today it's jumped down a little bit.

Cue 107 — 00:06:52,140 --> 00:06:55,740

We're actually still shipping more PRs, but as a percentage of the work,

Cue 108 — 00:06:55,780 --> 00:06:58,840

we've sort of dropped back a bit in terms of what goes through the factory.

Cue 109 — 00:06:58,980 --> 00:07:01,940

Some of the stuff around things like front end and some of the design pieces,

Cue 110 — 00:07:01,980 --> 00:07:05,520

we're doing a lot more design through the factory, sorry, not through the factory.

Cue 111 — 00:07:06,820 --> 00:07:10,580

And so folks are keen to maybe run that on, on their local machine rather than put

Cue 112 — 00:07:10,620 --> 00:07:12,060

it through something a bit more automated.

Cue 113 — 00:07:12,100 --> 00:07:14,640

So still seeing lots of usage.

Cue 114 — 00:07:15,580 --> 00:07:19,260

And something important to note during the time is the team size hasn't changed,

Cue 115 — 00:07:19,300 --> 00:07:23,780

right? So this is still the same same team size, but we've got this huge kind of

Cue 116 — 00:07:23,820 --> 00:07:24,620

uptick.

Cue 117 — 00:07:27,060 --> 00:07:29,860

I guess some quick level setting before we continue.

Cue 118 — 00:07:29,900 --> 00:07:31,540

So what do we mean by a factory?

Cue 119 — 00:07:31,580 --> 00:07:37,940

So in our world, we basically have this kind of tickets to PR to merge flow.

Cue 120 — 00:07:37,980 --> 00:07:41,540

So you start off with a ticket within something like we use Linear,

Cue 121 — 00:07:41,580 --> 00:07:43,700

but obviously something like GitHub issues or Jira.

Cue 122 — 00:07:43,740 --> 00:07:47,660

A human and agent have worked together to write that and refine that ticket into

Cue 123 — 00:07:47,700 --> 00:07:49,860

something meaningful that you can deliver.

Cue 124 — 00:07:49,900 --> 00:07:53,380

Our tickets often include a lot of the technical implementation as well.

Cue 125 — 00:07:53,420 --> 00:07:56,060

Not too much. We don't want to steer the implementation agent too much,

Cue 126 — 00:07:56,100 --> 00:08:02,420

but enough that we kind of track that in there, and we can sort of track the things

Cue 127 — 00:08:02,460 --> 00:08:05,420

that we think are important within that ticket that we want the agent to hit.

Cue 128 — 00:08:05,460 --> 00:08:09,580

We then have implementation agent that runs in the cloud, completely separate

Cue 129 — 00:08:09,620 --> 00:08:15,780

context window goes and builds that and passes it over to review agent within a PR

Cue 130 — 00:08:15,820 --> 00:08:21,220

within GitHub. Again the implementation and the review agent are separate entities.

Cue 131 — 00:08:21,260 --> 00:08:23,800

There's no kind of shared context window there.

Cue 132 — 00:08:24,500 --> 00:08:26,680

We also have a process called shepherding.

Cue 133 — 00:08:27,020 --> 00:08:30,700

Some people call it babysitting, but I think we've all experienced a world where you

Cue 134 — 00:08:30,740 --> 00:08:34,820

have to look after a PR and, you know, look at the comments that are coming up on

Cue 135 — 00:08:34,860 --> 00:08:38,420

that PR and chase people up and, you know, check for merge conflicts and do all

Cue 136 — 00:08:38,460 --> 00:08:40,860

those kinds of, like, boring, tedious things that no one likes doing.

Cue 137 — 00:08:40,900 --> 00:08:45,000

So we built an agent called Shepherd, which does all that stuff for you.

Cue 138 — 00:08:45,580 --> 00:08:48,280

And basically you can just leave it running.

Cue 139 — 00:08:48,540 --> 00:08:52,060

Finally, a human will, then, when they're happy with the state,

Cue 140 — 00:08:52,100 --> 00:08:55,720

will approve that and merge that, and that gets deployed to prod.

Cue 141 — 00:08:55,780 --> 00:08:58,820

Some of our PRs are also auto-merging as well.

Cue 142 — 00:08:58,860 --> 00:09:00,980

So we actually haven't got on the diagram.

Cue 143 — 00:09:01,020 --> 00:09:07,080

But we do a kind of sizing estimate based on the complexity of the ticket.

Cue 144 — 00:09:07,900 --> 00:09:12,860

And if it's small size or low risk, then we'll just allow that to be kind of merge

Cue 145 — 00:09:12,900 --> 00:09:16,040

straight into main and go out to production.

Cue 146 — 00:09:17,220 --> 00:09:21,620

The important thing here is that all these things here, this entire life cycle,

Cue 147 — 00:09:21,660 --> 00:09:25,700

is not triggered by a prompts, but actually triggered by events,

Cue 148 — 00:09:25,740 --> 00:09:27,940

things that are happening in the in the stock.

Cue 149 — 00:09:27,980 --> 00:09:32,940

Right. So no one is manually pressing a button to turn the ticket into to pass the

Cue 150 — 00:09:32,980 --> 00:09:34,840

ticket to the implementation agent.

Cue 151 — 00:09:35,060 --> 00:09:38,500

They just create a ticket, they put a label on it, or they delegate it to Tessl

Cue 152 — 00:09:38,540 --> 00:09:42,240

within Linear. And this whole thing kicks off right?

Cue 153 — 00:09:42,860 --> 00:09:46,040

The agents themselves own the individual steps.

Cue 154 — 00:09:46,140 --> 00:09:49,660

So as I mentioned, there's this kind of isolation between each between each of these

Cue 155 — 00:09:49,700 --> 00:09:53,660

different steps. The human is responsible for the entire outcome.

Cue 156 — 00:09:53,700 --> 00:09:56,380

Right. We're not quite at the level of just saying YOLO.

Cue 157 — 00:09:56,420 --> 00:10:00,000

You guys, you guys do it or just chill and do nothing.

Cue 158 — 00:10:01,940 --> 00:10:05,860

And then the important thing here is that the agents are working in the same repos

Cue 159 — 00:10:05,900 --> 00:10:08,100

as the humans, right? So it's not in a separate place.

Cue 160 — 00:10:08,140 --> 00:10:11,580

They're using the same skills the human would be if they were working with an agent

Cue 161 — 00:10:11,620 --> 00:10:13,120

on their local machine.

Cue 162 — 00:10:14,580 --> 00:10:18,660

And they're using the same GitHub services, Linear services,

Cue 163 — 00:10:18,700 --> 00:10:25,660

those kinds of things. So let's jump into a bit more about these five topics that

Cue 164 — 00:10:25,700 --> 00:10:27,920

I mentioned. So first one shared state.

Cue 165 — 00:10:29,020 --> 00:10:33,560

So this is where the agents are colliding over the same same things.

Cue 166 — 00:10:34,140 --> 00:10:40,540

So we we basically have kind of a simplified illustration of the overlap here.

Cue 167 — 00:10:40,580 --> 00:10:46,300

So we started seeing a bunch of painful merge conflicts where agents were sort of

Cue 168 — 00:10:46,340 --> 00:10:49,780

working on the same task, updating the same file, but not really kind of getting

Cue 169 — 00:10:49,820 --> 00:10:54,580

anywhere. And so what we realized is that independent tickets do not kind of

Cue 170 — 00:10:54,620 --> 00:10:56,380

guarantee independent changes.

Cue 171 — 00:10:56,420 --> 00:11:00,500

So with one one agent, these edits might have happened sequentially.

Cue 172 — 00:11:00,540 --> 00:11:04,480

Right. That one agent can work out what order it needs to apply changes to a file.

Cue 173 — 00:11:04,660 --> 00:11:09,100

When you are scaling that many, many agents you know, they're not aware of,

Cue 174 — 00:11:09,140 --> 00:11:12,640

of what the other agents are doing and how how they're going to,

Cue 175 — 00:11:14,340 --> 00:11:16,240

conflict with each other, basically.

Cue 176 — 00:11:17,500 --> 00:11:22,460

So, yeah, like I said, some of these some of these, edits are sorry,

Cue 177 — 00:11:22,500 --> 00:11:24,060

some of these conflicts are pretty obvious, right?

Cue 178 — 00:11:24,100 --> 00:11:27,820

You know, here, for example, two agents are working in the same file.

Cue 179 — 00:11:27,860 --> 00:11:32,640

They're working on some billing file, both making changes at the same time.

Cue 180 — 00:11:32,780 --> 00:11:35,100

So this is quite simple. Just two, two agents.

Cue 181 — 00:11:35,140 --> 00:11:36,820

But imagine scaling it to ten, right.

Cue 182 — 00:11:36,860 --> 00:11:39,680

The the problem is kind of exponential.

Cue 183 — 00:11:40,260 --> 00:11:45,100

There's also a kind of class of problem here where, agents might be working on

Cue 184 — 00:11:45,140 --> 00:11:50,740

apparently separate things, but there are conflicting changes that they're maybe not

Cue 185 — 00:11:50,780 --> 00:11:52,780

aware about that you would understand as a human right.

Cue 186 — 00:11:52,820 --> 00:11:57,687

So we use Drizzle as our ORM within the world of Node.

Cue 187 — 00:11:57,727 --> 00:12:01,740

Don't get me started on Node ORMs because there's lots and lots of terrible ones,

Cue 188 — 00:12:01,780 --> 00:12:04,360

but Drizzle is the one we're using for whatever reason.

Cue 189 — 00:12:05,100 --> 00:12:07,940

And it has a terrible migration system.

Cue 190 — 00:12:07,980 --> 00:12:11,240

It's shocking about it doesn't even have down migrations.

Cue 191 — 00:12:12,140 --> 00:12:16,420

But what happens here is you edit schema file and two separate edits to two

Cue 192 — 00:12:16,460 --> 00:12:22,020

different schemas that are unrelated end up causing contention on this journal JSON

Cue 193 — 00:12:22,060 --> 00:12:26,940

file. So you one agent might make a change to the user schema,

Cue 194 — 00:12:26,980 --> 00:12:31,980

maybe add a new field birth whatever, and the other agent is editing something

Cue 195 — 00:12:32,020 --> 00:12:33,740

completely different related to payments.

Cue 196 — 00:12:33,780 --> 00:12:37,540

But because both of those changes get recorded in the journal JSON,

Cue 197 — 00:12:37,580 --> 00:12:38,700

you end up getting conflicts there.

Cue 198 — 00:12:38,740 --> 00:12:42,200

And again, multiply this by many, many agents.

Cue 199 — 00:12:42,660 --> 00:12:47,900

And you really start to see churn and lots and lots of merge conflicts and things

Cue 200 — 00:12:47,940 --> 00:12:51,460

just not moving through their factory because they're just constantly conflicting

Cue 201 — 00:12:51,500 --> 00:12:52,440

with each other.

Cue 202 — 00:12:56,380 --> 00:13:01,980

So as I mentioned, obviously this kind of scales up exponentially as you are adding

Cue 203 — 00:13:02,020 --> 00:13:04,240

more and more agents working in parallel.

Cue 204 — 00:13:04,660 --> 00:13:07,924

Right now, we've kind of solved this by being a bit more clever about how we

Cue 205 — 00:13:07,964 --> 00:13:12,360

structure the work. So thinking about that sort of ticket process.

Cue 206 — 00:13:13,740 --> 00:13:17,820

At that point we think about staging, and we work with an agent to work out what the

Cue 207 — 00:13:17,860 --> 00:13:20,580

changes we need to make. We'll try and make a lot of these sort of schema changes

Cue 208 — 00:13:20,620 --> 00:13:24,320

earlier on, rather than like later in the set of work.

Cue 209 — 00:13:25,060 --> 00:13:29,140

We also have a bunch of skills to help agents fix these kinds of conflicts by

Cue 210 — 00:13:29,180 --> 00:13:33,300

default. Not many, not many models seem to know about Drizzle and how it's kind of

Cue 211 — 00:13:33,340 --> 00:13:35,660

migration. Journal file works.

Cue 212 — 00:13:35,700 --> 00:13:38,340

They just want to update and edit that journal file directly,

Cue 213 — 00:13:38,380 --> 00:13:40,960

which is not good. You're in a bad place if you do that.

Cue 214 — 00:13:41,100 --> 00:13:44,060

And so we've given them a bunch of skills on how to resolve those things as well,

Cue 215 — 00:13:44,100 --> 00:13:48,120

which kind of increased, success rates.

Cue 216 — 00:13:48,540 --> 00:13:52,020

The other thing is thinking about the kind of technology choices that we pick,

Cue 217 — 00:13:52,060 --> 00:13:55,220

right? Like, we picked Drizzle before we were in this world,

Cue 218 — 00:13:55,260 --> 00:13:58,540

and we're kind of bound to it because it's a key part of our architecture.

Cue 219 — 00:13:58,580 --> 00:14:03,100

But really, we should try and look at something different, something that doesn't

Cue 220 — 00:14:03,140 --> 00:14:07,700

cause this contention, like the technology choices that you make can affect how

Cue 221 — 00:14:07,740 --> 00:14:11,420

successful multiple agents can be working within your repository.

Cue 222 — 00:14:11,460 --> 00:14:13,660

I think Drizzle is about to change the migration system.

Cue 223 — 00:14:13,700 --> 00:14:16,780

I think in the 1.0 release, which we're looking forward to, which is going to remove

Cue 224 — 00:14:16,820 --> 00:14:19,640

that that, journal JSON file.

Cue 225 — 00:14:23,300 --> 00:14:25,580

So next up is stale assumptions.

Cue 226 — 00:14:25,620 --> 00:14:29,360

So another agent is changing the world where yours is working.

Cue 227 — 00:14:29,780 --> 00:14:34,460

So you give an agent a ticket and it kicks off starting some work.

Cue 228 — 00:14:34,500 --> 00:14:37,140

And that point of starting essentially has a snapshot of the world.

Cue 229 — 00:14:37,180 --> 00:14:42,110

Right. So here, for example, we're looking at version 12 of the code version three

Cue 230 — 00:14:42,150 --> 00:14:45,580

of the tickets. And maybe there's been a review at some point on that PR.

Cue 231 — 00:14:46,060 --> 00:14:47,880

And the agent is doing some work.

Cue 232 — 00:14:48,820 --> 00:14:53,480

But what happens in the meantime is other agents are doing work in other branches on

Cue 233 — 00:14:53,780 --> 00:14:57,700

other tickets, and they might be merging that work before your agent gets to it.

Cue 234 — 00:14:57,740 --> 00:15:04,320

Right. And so what you end up in a kind of situation is where you have is that

Cue 235 — 00:15:04,660 --> 00:15:10,980

the agent basically is making changes in a world where its reality has changed.

Cue 236 — 00:15:11,020 --> 00:15:17,020

Essentially, something we kept seeing is that we had an issue with merge conflicts.

Cue 237 — 00:15:17,060 --> 00:15:22,780

Actually, we didn't notice it first, but to the agent to get its work merged into

Cue 238 — 00:15:22,820 --> 00:15:27,700

main rebase main, and it would just silently throw away the other work then been

Cue 239 — 00:15:27,740 --> 00:15:30,160

done. And we didn't realize at first.

Cue 240 — 00:15:30,700 --> 00:15:34,140

Sometimes obviously tests start failing, sometimes the agent would edit the test as

Cue 241 — 00:15:34,180 --> 00:15:37,060

well to stop them failing. And we didn't know this was happening.

Cue 242 — 00:15:37,100 --> 00:15:38,820

Right. All this stuff is happening out of band.

Cue 243 — 00:15:38,860 --> 00:15:41,680

We're not sitting there watching over the agent and what it's doing.

Cue 244 — 00:15:42,740 --> 00:15:43,940

But someone started catching it.

Cue 245 — 00:15:43,980 --> 00:15:45,460

We started having people being like, where's that?

Cue 246 — 00:15:45,500 --> 00:15:47,460

Where's that button gone? Like that was there last week.

Cue 247 — 00:15:47,500 --> 00:15:51,260

The buttons disappeared. So some of this can be done with better testing and kind of

Cue 248 — 00:15:51,300 --> 00:15:52,640

better isolation.

Cue 249 — 00:15:55,500 --> 00:15:56,720

Sorry. Wants to say,

Cue 250 — 00:15:57,980 --> 00:15:58,800

but,

Cue 251 — 00:16:00,380 --> 00:16:06,740

yeah, we we solved it by having another kind of trigger that basically ran as part

Cue 252 — 00:16:06,780 --> 00:16:12,260

of the PR, which is able to notice these kinds of edits being made and stop agent

Cue 253 — 00:16:12,300 --> 00:16:16,220

from doing it so the agent would still try and do these kinds of merge conflict

Cue 254 — 00:16:16,260 --> 00:16:18,580

resolutions. Where would you delete the person's code?

Cue 255 — 00:16:18,620 --> 00:16:21,500

But we had another agent that was watching that and was able to say,

Cue 256 — 00:16:21,540 --> 00:16:22,700

hey, you've done something wrong there.

Cue 257 — 00:16:22,740 --> 00:16:24,840

Revert that. Here's what you should be doing.

Cue 258 — 00:16:26,740 --> 00:16:30,580

So that's stale assumptions. Convergence is an interesting one.

Cue 259 — 00:16:30,620 --> 00:16:36,060

So here this is where is each iteration actually getting us closer to an acceptable

Cue 260 — 00:16:36,100 --> 00:16:39,400

result. From 1 to 100. And don't stop.

Cue 261 — 00:16:40,020 --> 00:16:41,680

Got it. Yeah. All right. Okay.

Cue 262 — 00:16:42,740 --> 00:16:47,220

One. And then the next day we can say two, and we'll just go until we get 100 ready.

Cue 263 — 00:16:47,260 --> 00:16:50,860

I'll start one. Yeah. I'm ready, so I'll go after you.

Cue 264 — 00:16:50,900 --> 00:16:53,140

Let's keep it smooth and we'll just count straight through.

Cue 265 — 00:16:53,180 --> 00:16:56,460

Go for it. Awesome. So I'll say one.

Cue 266 — 00:16:56,500 --> 00:16:58,900

You say two and we'll just keep that rhythm going.

Cue 267 — 00:16:58,940 --> 00:17:01,620

Ready? Here we go. One. Exactly.

Cue 268 — 00:17:01,660 --> 00:17:04,060

I'm ready. I'll jump in with two.

Cue 269 — 00:17:04,100 --> 00:17:07,100

Right after you. Perfect. Excellent.

Cue 270 — 00:17:07,140 --> 00:17:08,820

So I said one knows your turn.

Cue 271 — 00:17:08,860 --> 00:17:12,120

Go ahead. And two. And we'll take the pace on the way up 200.

Cue 272 — 00:17:12,460 --> 00:17:15,420

All right. Perfect. So I'll say two, and then you'll say three.

Cue 273 — 00:17:15,460 --> 00:17:17,160

And we'll keep going one after the other.

Cue 274 — 00:17:17,200 --> 00:17:20,000

Just don't have to explain to each other how to count.

Cue 275 — 00:17:23,300 --> 00:17:26,260

So basically, that kind of thing was happening within our factory,

Cue 276 — 00:17:26,300 --> 00:17:29,620

right? Like we were sitting there just like, what is going on with this crazy

Cue 277 — 00:17:29,660 --> 00:17:32,780

situation? So here's here's a real example of that kind of in practice.

Cue 278 — 00:17:32,820 --> 00:17:36,520

So this is a very simple PR raised by a colleague, Dougal.

Cue 279 — 00:17:37,020 --> 00:17:39,600

Just adding one new API field to,

Cue 280 — 00:17:40,860 --> 00:17:43,960

to a response in API writing. Pretty simple.

Cue 281 — 00:17:44,380 --> 00:17:48,980

There's 1700 lines changed on this, but most of that is kind of auto generated API

Cue 282 — 00:17:49,020 --> 00:17:52,000

contracts, OpenAPI JSON file is that kind of thing.

Cue 283 — 00:17:52,820 --> 00:17:55,840

But a lot of bad things started happening.

Cue 284 — 00:17:55,940 --> 00:17:58,480

Can anyone see the issue here?

Cue 285 — 00:18:01,700 --> 00:18:03,460

Yeah, actually, it's quite a small screen.

Cue 286 — 00:18:03,500 --> 00:18:09,420

So, if I was zoomed in here, basically the conversation are in very,

Cue 287 — 00:18:09,460 --> 00:18:11,860

very short order. Got to 106 comments.

Cue 288 — 00:18:11,900 --> 00:18:14,140

Right. Like that wasn't a human doing that.

Cue 289 — 00:18:14,180 --> 00:18:20,080

There was a bunch of agents. So we ended up having 11 review rounds over 23 hours.

Cue 290 — 00:18:20,860 --> 00:18:22,660

The agents were kind of finding things.

Cue 291 — 00:18:22,700 --> 00:18:28,220

It was quite nitty comments. But, it basically meant that we kind of in this,

Cue 292 — 00:18:28,260 --> 00:18:30,840

like, death spiral of agents talking to each other.

Cue 293 — 00:18:31,740 --> 00:18:36,620

I think we, for some reason, we were testing a bunch of different code review tools

Cue 294 — 00:18:36,660 --> 00:18:38,800

at the same time. We don't know why we did that.

Cue 295 — 00:18:39,060 --> 00:18:41,300

But here you can see we had some stuff going through GitHub Actions.

Cue 296 — 00:18:41,340 --> 00:18:43,620

We had agent tests, which is one for a while.

Cue 297 — 00:18:43,660 --> 00:18:46,700

We had our Tessl code review. We also had CodeRabbit as well,

Cue 298 — 00:18:46,740 --> 00:18:50,600

which isn't on this list, but that was definitely in the mix as well.

Cue 299 — 00:18:51,220 --> 00:18:54,980

And so what was happening, we basically worked was being done,

Cue 300 — 00:18:55,020 --> 00:18:56,220

but we weren't really getting anywhere.

Cue 301 — 00:18:56,260 --> 00:18:59,460

So the initial work had been done immediately to kind of a good standard,

Cue 302 — 00:18:59,500 --> 00:19:03,820

but then we had multiple agents sort of chiming in, and they were all responsible

Cue 303 — 00:19:03,860 --> 00:19:08,140

for the exact same task, but we didn't have any kind of alignment on what our

Cue 304 — 00:19:08,180 --> 00:19:11,720

definition of good looked like for each of these code review agents.

Cue 305 — 00:19:12,940 --> 00:19:14,080

You can see that.

Cue 317 — 00:19:57,140 --> 00:20:01,100

So that's kind of like an extreme example, I guess.

Cue 318 — 00:20:01,140 --> 00:20:05,140

Like most people aren't really running four different code reviewers on every PR,

Cue 319 — 00:20:05,180 --> 00:20:08,900

but even with one review bot, you can kind of see the same sort of issues being

Cue 320 — 00:20:08,940 --> 00:20:12,580

raised on each round. So it's worded slightly differently and each time they're

Cue 321 — 00:20:12,620 --> 00:20:17,660

worded differently, the agent goes off and decides to try and implement whatever

Cue 322 — 00:20:17,700 --> 00:20:20,940

change that was asked for. So how do we fix this?

Cue 323 — 00:20:20,980 --> 00:20:24,300

First of all, recognizing that repetition.

Cue 324 — 00:20:24,340 --> 00:20:27,940

And then each finding that comes out of that repetition doesn't need to be a

Cue 325 — 00:20:27,980 --> 00:20:29,500

distinct action that needs to be taken.

Cue 326 — 00:20:29,540 --> 00:20:34,620

So these kind of findings that happen in this review cycle need to be kind of

Cue 327 — 00:20:34,660 --> 00:20:39,300

aggregated against the existing findings that that bot or other bots have found

Cue 328 — 00:20:39,340 --> 00:20:44,360

within the PR. We should also look at making sure we've got consistent standards

Cue 329 — 00:20:44,740 --> 00:20:48,900

between all these different agents on these different pieces within our factory

Cue 330 — 00:20:48,940 --> 00:20:55,804

pipeline. We do this via skills and things like obviously your AGENTS.md or your

Cue 331 — 00:20:55,844 --> 00:21:01,300

CLAUDE.md. And making sure that what you what you think is good is clearly defined.

Cue 332 — 00:21:01,340 --> 00:21:05,280

And every agent that is working in that repo understands that as well.

Cue 333 — 00:21:05,700 --> 00:21:09,380

And then finally allowing a human to jump in as well.

Cue 334 — 00:21:09,420 --> 00:21:14,100

So having some kind of condition, we'll talk about stop conditions shortly,

Cue 335 — 00:21:14,140 --> 00:21:17,500

but having some kind of condition that will detect this and then say,

Cue 336 — 00:21:17,540 --> 00:21:24,540

hey, this is not good, and pull in a human to ask for change or changing the scope,

Cue 337 — 00:21:24,580 --> 00:21:28,440

whatever makes sense there. So that's convergence.

Cue 338 — 00:21:29,460 --> 00:21:35,880

Next up is duplicate work and kind of how we recover from from duplicate work.

Cue 339 — 00:21:35,980 --> 00:21:41,680

So if we think about our factory it's effectively distributed state machine.

Cue 340 — 00:21:42,060 --> 00:21:46,330

But it's also running across multiple third party systems that,

Cue 341 — 00:21:46,370 --> 00:21:51,300

that you don't own. Right. So if you think about it, we have tickets in Linear.

Cue 342 — 00:21:51,340 --> 00:21:55,220

We have our kind of cloud execution environments, those sandboxes.

Cue 343 — 00:21:55,260 --> 00:21:59,320

We have our PRs on GitHub. We have CI runs.

Cue 344 — 00:21:59,420 --> 00:22:02,740

We have review comments. We have things being merged.

Cue 345 — 00:22:02,780 --> 00:22:08,540

We have agent logs. All of these different elements are all distributed for most

Cue 346 — 00:22:08,580 --> 00:22:13,740

people anyway. And to kind of calculate our view of the world,

Cue 347 — 00:22:13,780 --> 00:22:16,800

we need to look at all of these things to understand what is happening.

Cue 348 — 00:22:18,220 --> 00:22:21,420

Luckily, none of this stuff is like new in terms of system design,

Cue 349 — 00:22:21,460 --> 00:22:24,660

right? Like, we've been building fault tolerant, distributed systems for a while

Cue 350 — 00:22:24,700 --> 00:22:28,980

now, and so we can kind of go back to first principles and ensure things like

Cue 351 — 00:22:29,020 --> 00:22:33,020

atomicity and idempotency are kind of part of our system design and the way we think

Cue 352 — 00:22:33,060 --> 00:22:37,560

about our factory. And they are effectively fault tolerant.

Cue 353 — 00:22:37,940 --> 00:22:39,840

Things are going to break. Yeah.

Cue 354 — 00:22:40,300 --> 00:22:43,940

This stuff is so, so hard to test, like, without using production.

Cue 355 — 00:22:43,980 --> 00:22:47,340

Like, you know, how how do you mock GitHub, for example, or like,

Cue 356 — 00:22:47,380 --> 00:22:49,600

or Linear? Like you just can't do these things.

Cue 357 — 00:22:51,260 --> 00:22:52,460

And so things will definitely break.

Cue 358 — 00:22:52,500 --> 00:22:54,560

So how do you how do you recover from those.

Cue 359 — 00:22:55,380 --> 00:22:59,460

So I want to tell you a quick story about an issue we had that caused us a bunch of

Cue 360 — 00:22:59,500 --> 00:23:06,020

problems where a comment on a PR got counted twice and two different

Cue 361 — 00:23:06,060 --> 00:23:10,940

workers responded. Which we didn't initially notice.

Cue 362 — 00:23:10,980 --> 00:23:13,640

So sometimes the later worker.

Cue 363 — 00:23:14,540 --> 00:23:18,260

You could work out that the work was already being done and did nothing.

Cue 364 — 00:23:18,300 --> 00:23:20,600

There was like a no op, nothing happened there.

Cue 365 — 00:23:20,700 --> 00:23:26,740

But sometimes, depending on timing, the second agent could sorry,

Cue 366 — 00:23:26,780 --> 00:23:30,300

the second worker could then also do some work and also react to those changes.

Cue 367 — 00:23:30,340 --> 00:23:34,300

And obviously the kind of output that comes out of that is going to be different.

Cue 368 — 00:23:34,340 --> 00:23:36,220

Right. You give two agents the same task.

Cue 369 — 00:23:36,260 --> 00:23:38,440

They're never going to agree and do the same things.

Cue 370 — 00:23:39,460 --> 00:23:42,760

It's effectively non determinism and it's super hard to debug.

Cue 371 — 00:23:42,860 --> 00:23:45,800

We were testing with only a single worker.

Cue 372 — 00:23:46,380 --> 00:23:49,840

And obviously when there's a single worker you don't get into this state.

Cue 373 — 00:23:50,180 --> 00:23:52,720

And in prod, it was a very different story.

Cue 374 — 00:23:53,180 --> 00:23:59,580

In the end, we rewrote the queue entirely to basically handle handle this.

Cue 375 — 00:23:59,620 --> 00:24:05,700

So, we defined what was kind of a logical piece of work and which worker currently

Cue 376 — 00:24:05,740 --> 00:24:10,660

owned that. I think the team that did this was so, did formal verification as well,

Cue 377 — 00:24:10,700 --> 00:24:13,000

which is definitely something I'm not an expert in.

Cue 378 — 00:24:13,100 --> 00:24:16,060

But they also formally verified the queue that we rebuilt as well,

Cue 379 — 00:24:16,100 --> 00:24:19,160

that it did what we needed it to do.

Cue 380 — 00:24:21,220 --> 00:24:26,540

So fifth piece here is around control and budgets.

Cue 381 — 00:24:26,580 --> 00:24:30,380

So, actually stopping when things are going, going crazy.

Cue 382 — 00:24:30,420 --> 00:24:32,920

So we've got another PR again.

Cue 383 — 00:24:33,980 --> 00:24:38,540

The work done here, unlike the last PR that we looked at, is much more complex.

Cue 384 — 00:24:38,580 --> 00:24:42,640

It's the last one was just adding a field to an API result.

Cue 385 — 00:24:42,780 --> 00:24:47,340

This is a bunch of new functionality being added to our verifies feature.

Cue 386 — 00:24:47,380 --> 00:24:49,460

So it ends up running over six days.

Cue 387 — 00:24:49,500 --> 00:24:55,280

No one really was like keeping an eye on this and racked up $800 spend in tokens.

Cue 388 — 00:24:55,460 --> 00:24:57,500

And then the worst thing is it didn't even get merged, right?

Cue 389 — 00:24:57,540 --> 00:25:01,520

We just closed it because. Because it was just a bunch of slop.

Cue 390 — 00:25:04,300 --> 00:25:09,020

And if you thought 105 conversations were bad, this one has 533.

Cue 391 — 00:25:09,060 --> 00:25:15,380

So it was going crazy. So all those comments, all that conversation was basically a

Cue 392 — 00:25:15,420 --> 00:25:20,540

bunch of bots, including an agent that was running on a developers machine that was

Cue 393 — 00:25:20,580 --> 00:25:24,120

watching the PR as well. So what happened here?

Cue 394 — 00:25:24,900 --> 00:25:29,780

What was the mistake we made? Well, we were kind of like, not entirely oblivious to

Cue 395 — 00:25:29,820 --> 00:25:31,060

the fact that this might happen, right?

Cue 396 — 00:25:31,100 --> 00:25:33,420

We weren't just going to give a bunch of bots access to GitHub and say,

Cue 397 — 00:25:33,460 --> 00:25:38,820

go crazy, right? So we built in this limit of ten kind of bot interactions in a row,

Cue 398 — 00:25:38,860 --> 00:25:40,180

and we thought ten was a nice number.

Cue 399 — 00:25:40,220 --> 00:25:42,040

We thought, sure, that would make sense.

Cue 400 — 00:25:43,020 --> 00:25:44,180

And this was actually in the code.

Cue 401 — 00:25:44,220 --> 00:25:47,940

This wasn't just context. The agent can work its way around that.

Cue 402 — 00:25:47,980 --> 00:25:51,720

This is like a physical restriction that was that was hard coded.

Cue 403 — 00:25:52,260 --> 00:25:57,100

And so we we were detecting in those in the threads within GitHub who's a bot who's

Cue 404 — 00:25:57,140 --> 00:26:02,960

a human and kind of making sure that everything was being adhered to.

Cue 405 — 00:26:03,340 --> 00:26:09,940

But we had two issues here. One was that logic was flawed, whether the human whether

Cue 406 — 00:26:09,980 --> 00:26:14,060

the user was a bot or not. Nothing particularly amazing here.

Cue 407 — 00:26:14,100 --> 00:26:16,200

It was just a simple logic error.

Cue 408 — 00:26:16,740 --> 00:26:22,780

And so whenever the bot posted a comment, it was able to reset that counter back to

Cue 409 — 00:26:22,820 --> 00:26:28,200

zero. And so that ended up reading obviously as human intent.

Cue 410 — 00:26:28,500 --> 00:26:31,720

And so we basically had no guardrails at that point.

Cue 411 — 00:26:32,460 --> 00:26:37,260

On top of this, we also had the developer who is kind of owning this was running an

Cue 412 — 00:26:37,300 --> 00:26:41,180

agent locally on the machine, as I said, but it was acting on their behalf as a

Cue 413 — 00:26:41,220 --> 00:26:44,920

human. So it was posting into the, the PR,

Cue 414 — 00:26:46,860 --> 00:26:50,260

and it looked like a human to our systems, but obviously we know it wasn't a human.

Cue 415 — 00:26:50,300 --> 00:26:54,940

It was Claude watching the PR and and chiming in and like, you know,

Cue 416 — 00:26:54,980 --> 00:26:57,000

pouring gasoline on the fire. Really.

Cue 417 — 00:26:57,940 --> 00:27:01,680

So yeah, this was this continued for six days, kind of unnoticed.

Cue 418 — 00:27:03,460 --> 00:27:07,560

And and obviously we fixed we fix the issue with the bot detection,

Cue 419 — 00:27:08,340 --> 00:27:10,180

which was one way of fixing it.

Cue 420 — 00:27:10,220 --> 00:27:13,500

What we actually want to do, and we haven't done this yet, is actually start

Cue 421 — 00:27:13,540 --> 00:27:16,620

thinking about a budget for a work item.

Cue 422 — 00:27:16,660 --> 00:27:20,800

So rather than thinking about those individual number of turns,

Cue 423 — 00:27:20,940 --> 00:27:23,620

actually thinking about what is this actually costing us?

Cue 424 — 00:27:23,660 --> 00:27:27,520

So looking at the cost of the tokens, looking at the cost of things like CI runs.

Cue 425 — 00:27:28,100 --> 00:27:32,700

And budgeting against that. I think personally, if you've got a couple of bots or

Cue 426 — 00:27:32,740 --> 00:27:35,700

agents talking to each other and they have 40 turns of talking to each other,

Cue 427 — 00:27:35,740 --> 00:27:39,090

but the work that's coming out is good and it's under budget and it's only costing

Cue 428 — 00:27:39,130 --> 00:27:41,460

you $5, then that's fine, right?

Cue 429 — 00:27:41,500 --> 00:27:44,880

Like, why try and limit that sort of like level of interaction?

Cue 430 — 00:27:45,300 --> 00:27:48,860

But yeah, this is our kind of plan that we want to move to, but we're not quite

Cue 431 — 00:27:48,900 --> 00:27:49,960

there yet. So.

Cue 432 — 00:27:52,900 --> 00:27:54,800

So some key takeaways.

Cue 433 — 00:27:56,460 --> 00:28:01,660

Most of everything that the factory is driving at the moment is designed for a human

Cue 434 — 00:28:01,700 --> 00:28:04,980

at the wheel. Right. So if you take a look at things like GitHub,

Cue 435 — 00:28:05,020 --> 00:28:11,444

for example, the way the GitHub apps API is designed, they can't approve their own

Cue 436 — 00:28:11,484 --> 00:28:15,540

PRs. So we have a factory app, which is the thing that will create PRs and put

Cue 437 — 00:28:15,580 --> 00:28:19,420

labels on things and do all that kind of stuff, but it can't approve.

Cue 438 — 00:28:19,460 --> 00:28:23,820

It can't auto approve its own PRs because of a limitation of the GitHub API.

Cue 439 — 00:28:23,860 --> 00:28:28,060

They never designed it to think oh, 5 or 6 agents will be running as part of this

Cue 440 — 00:28:28,100 --> 00:28:31,000

app installation. And so we should enable that.

Cue 441 — 00:28:32,060 --> 00:28:33,400

Things like CI,

Cue 442 — 00:28:35,740 --> 00:28:39,620

every push gets a full run. I'm sure most people just have CI set up this way.

Cue 443 — 00:28:39,660 --> 00:28:43,040

Just because it's the way we've always been done it, we've always done it right.

Cue 444 — 00:28:43,500 --> 00:28:46,860

We just, you know, if someone does a push or a commit, we we just trigger a bunch

Cue 445 — 00:28:46,900 --> 00:28:48,420

of, like, automated processes.

Cue 446 — 00:28:48,460 --> 00:28:52,100

Why not hand that some of that stuff off to an agent and let it decide when it wants

Cue 447 — 00:28:52,140 --> 00:28:54,620

to run CI. I think these days the models are good enough that,

Cue 448 — 00:28:54,660 --> 00:28:58,740

you know, you could trust them to make a good judgment on that and maybe put in

Cue 449 — 00:28:58,780 --> 00:29:00,660

other protections around things like running.

Cue 450 — 00:29:00,700 --> 00:29:03,340

CI on main, for example. Anthropic have done a bunch of this recently.

Cue 451 — 00:29:03,380 --> 00:29:05,180

They posted a blog post about this a couple of weeks ago.

Cue 452 — 00:29:05,220 --> 00:29:07,360

So definitely worth looking at.

Cue 453 — 00:29:07,780 --> 00:29:09,700

And then things like issue trackers, right?

Cue 454 — 00:29:09,740 --> 00:29:12,760

Traditionally issue trackers just have like one assignee.

Cue 455 — 00:29:12,800 --> 00:29:16,860

Linear added the ability to delegate to an agent like last year maybe,

Cue 456 — 00:29:16,900 --> 00:29:20,080

which is super cool, but why not delegate to like three agents or,

Cue 457 — 00:29:20,820 --> 00:29:22,660

a mixture of those different kind of settings?

Cue 458 — 00:29:22,700 --> 00:29:26,980

Right. Like all these, all these restrictions are because these tools and most of

Cue 459 — 00:29:27,020 --> 00:29:30,420

the tools we're using have been built on a world that was not designed for agents.

Cue 460 — 00:29:30,460 --> 00:29:35,020

Right. And especially not many coding agents working in parallel at speed all the

Cue 461 — 00:29:35,060 --> 00:29:35,860

time.

Cue 462 — 00:29:39,020 --> 00:29:42,440

The next piece I want to talk about briefly is just around autonomy.

Cue 463 — 00:29:42,940 --> 00:29:47,580

So everything we talked about today allowed us to earn a level of trust in the,

Cue 464 — 00:29:47,620 --> 00:29:52,500

in the factory. We didn't just turn it all on and one day or day one and let it just

Cue 465 — 00:29:52,540 --> 00:29:53,780

do everything that I talked about.

Cue 466 — 00:29:53,820 --> 00:29:55,060

We kind of started small, right?

Cue 467 — 00:29:55,100 --> 00:29:57,680

We started with some small elements.

Cue 468 — 00:29:57,740 --> 00:30:00,740

And as we kind of gained confidence that the factory was competent,

Cue 469 — 00:30:00,780 --> 00:30:04,860

are doing those things, we gave it more and more autonomy and we were able to kind

Cue 470 — 00:30:04,900 --> 00:30:06,040

of step back more.

Cue 471 — 00:30:07,340 --> 00:30:10,820

This means that we can start kind of shifting left and shifting right.

Cue 472 — 00:30:10,860 --> 00:30:15,620

So in terms of shifting left, we are starting to do be even kind of less involved in

Cue 473 — 00:30:15,660 --> 00:30:21,260

the code. We are doing more and more work and that kind of upfront ticket planning

Cue 474 — 00:30:21,300 --> 00:30:23,060

section, but we could shift even further.

Cue 475 — 00:30:23,100 --> 00:30:27,180

Left, right. Like as me, as an engineer, I can start taking on more of the kind of

Cue 476 — 00:30:27,220 --> 00:30:30,460

product manager roles. I could start getting my agent to connect to things like

Cue 477 — 00:30:30,500 --> 00:30:33,740

PostHog, which we use for product analytics and making decisions about,

Cue 478 — 00:30:33,780 --> 00:30:34,900

should we even build this feature?

Cue 479 — 00:30:34,940 --> 00:30:37,160

Does it make sense? Will anyone use it?

Cue 480 — 00:30:37,900 --> 00:30:41,220

And then on the right hand side, thinking about stuff after it gets deployed,

Cue 481 — 00:30:41,260 --> 00:30:45,900

like doing testing in QA, in prod things, or maybe updating docs,

Cue 482 — 00:30:45,940 --> 00:30:49,480

things that, that I don't want to do anymore.

Cue 483 — 00:30:51,540 --> 00:30:56,220

And then lastly, what you do by instinct, always record it in context and make sure

Cue 484 — 00:30:56,260 --> 00:30:58,740

that agents have access to that context.

Cue 485 — 00:30:58,780 --> 00:31:01,740

Right. So things that you just do instinctively as a human.

Cue 486 — 00:31:01,780 --> 00:31:05,520

Make sure it's written down. Make sure it's in a in a tool or a skill.

Cue 487 — 00:31:05,580 --> 00:31:09,660

And the agents can kind of access that and use that as they need to to solve the

Cue 488 — 00:31:09,700 --> 00:31:12,240

problem. So having to repeat yourself all the time,

Cue 489 — 00:31:14,660 --> 00:31:16,080

and that's it. Thank you.

Cue 490 — 00:31:22,420 --> 00:31:27,020

I'm not sure how we'd have time just a little over, but we have a 30 minute coffee

Cue 491 — 00:31:27,060 --> 00:31:30,340

break right after this. So, guys, feel free to come up and ask some questions.

Cue 492 — 00:31:30,380 --> 00:31:32,280

Another round of applause. Please forgive me.

Cue 493 — 00:31:38,100 --> 00:31:40,280

One more thing. I've got to do a bit of a pitch.

Cue 494 — 00:31:40,340 --> 00:31:43,520

So we've obviously just launched our brand new software factory.

Cue 495 — 00:31:44,420 --> 00:31:48,670

Scan the QR code. If you go and sign up and use the email that you've registered

Cue 496 — 00:31:48,710 --> 00:31:52,700

with at DevCon, you will get $500 of Tessl credits.

Cue 497 — 00:31:52,740 --> 00:31:56,900

So basically free agent tokens and you can go and start just playing around with

Cue 498 — 00:31:56,940 --> 00:32:00,280

this. So yeah. Yeah. Please, please have a go.

Cue 499 — 00:32:00,540 --> 00:32:01,600

Thanks, everyone.

skills

talk-moss-concurrent-agent-risk

tile.json