CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-lee-perkins-code-readiness/references/

Corrected transcript: Code Readiness for Agentic Development

Tracy Lee & Coston Perkins · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=0DQwjoPqolw

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:00,000 --> 00:00:04,847

thanks all for being here. So for the past 30 days.

Cue 2 — 00:00:05,147 --> 00:00:09,227

What if I told you that one of my engineers had 99?

Cue 3 — 00:00:09,267 --> 00:00:14,567

It's actually 99.6% of their code fully written by AI.

Cue 4 — 00:00:15,107 --> 00:00:17,307

That's just merged into production.

Cue 5 — 00:00:17,347 --> 00:00:19,727

It's not written or edited by a developer.

Cue 6 — 00:00:20,627 --> 00:00:24,587

Now, if you think about your last pull request, how many times did you all have to

Cue 7 — 00:00:24,627 --> 00:00:28,267

step in before it was merged? Like whether that was correcting a prompt,

Cue 8 — 00:00:28,307 --> 00:00:33,867

a rerun, a manual setup, anybody getting 99 to 100% of their code fully written by

Cue 9 — 00:00:33,907 --> 00:00:36,027

AI. Oh, I see some hands. One.

Cue 10 — 00:00:36,067 --> 00:00:39,487

Two. Three. Three. Okay. Four.

Cue 11 — 00:00:39,627 --> 00:00:43,007

Okay. Five. Yes, I love it. That's awesome.

Cue 12 — 00:00:43,147 --> 00:00:48,287

How about, like 50%? Anybody have 50% their code a few more.

Cue 13 — 00:00:48,387 --> 00:00:52,427

Okay, good good good. 10%. All hands.

Cue 14 — 00:00:52,467 --> 00:00:56,567

No, that was no hand. Okay, we got one hand over here.

Cue 15 — 00:00:57,547 --> 00:01:01,267

So I think another thing to think about when we say 99%, because,

Cue 16 — 00:01:01,307 --> 00:01:05,047

you know, you might be like, well, what's 99%, right?

Cue 17 — 00:01:05,267 --> 00:01:11,947

But this is actually 231 pull request that this team member was actually able to

Cue 18 — 00:01:11,987 --> 00:01:15,387

get merged again with like no developer code.

Cue 19 — 00:01:15,427 --> 00:01:17,427

And that was over a 30 day period.

Cue 20 — 00:01:17,467 --> 00:01:22,747

So what's unique about this is that 85% of all those pull requests were merged on

Cue 21 — 00:01:22,787 --> 00:01:27,487

the first try, meaning like nothing was changed after that pull request was opened

Cue 22 — 00:01:27,587 --> 00:01:32,307

and then most of the other 34, the agent basically fixed its own work.

Cue 23 — 00:01:32,347 --> 00:01:36,307

So whether that was like a failed check or something from a review comment or

Cue 24 — 00:01:36,347 --> 00:01:39,187

anything like that. How many of you all are experiencing that?

Cue 25 — 00:01:39,227 --> 00:01:41,447

Am I going to have like the same show of hands?

Cue 26 — 00:01:41,547 --> 00:01:45,847

4 or 5? I'll just pretend that I see like 4 or 5 hands up there.

Cue 27 — 00:01:47,107 --> 00:01:51,267

So, you know, I think another thing to understand as we all go through this AI

Cue 28 — 00:01:51,307 --> 00:01:58,107

journey is that no, no code written by developers still leaves plenty of work

Cue 29 — 00:01:58,147 --> 00:02:00,987

for us to do, right? Like, we still have to set the requirements,

Cue 30 — 00:02:01,027 --> 00:02:06,387

review pull requests, make the calls, and having something merge into production is

Cue 31 — 00:02:06,427 --> 00:02:08,547

a weak evidence of actual quality, right?

Cue 32 — 00:02:08,587 --> 00:02:13,627

So that's also our jobs. And then also watching what breaks after merge.

Cue 33 — 00:02:13,667 --> 00:02:18,427

Thankfully, of the 230 that were merged, nothing has been reverted.

Cue 34 — 00:02:18,467 --> 00:02:23,047

So crossing fingers for the next 30 days for that to be a reality.

Cue 35 — 00:02:23,907 --> 00:02:28,807

So if you're not experiencing that 99% of your code being written by AI,

Cue 36 — 00:02:29,147 --> 00:02:33,167

I would actually venture to say that the problem is not you.

Cue 37 — 00:02:33,507 --> 00:02:36,787

The problem is typically code base readiness.

Cue 38 — 00:02:36,827 --> 00:02:40,627

And when we go into organizations and talk to different teams,

Cue 39 — 00:02:40,667 --> 00:02:45,507

I feel like most of what we see again is that development teams are actually not

Cue 40 — 00:02:45,547 --> 00:02:51,287

able to fully utilize the power of AI because their code base is just aren't ready.

Cue 41 — 00:02:51,667 --> 00:02:54,907

My name is Tracy. I'm the CEO of a company called This Dot Labs.

Cue 42 — 00:02:54,947 --> 00:03:00,805

We're a software consultancy. I you can find me on Twitter on X at ladyleet or on

Cue 43 — 00:03:00,845 --> 00:03:04,743

LinkedIn as well. I'm a Google Developer Expert and Microsoft MVP.

Cue 44 — 00:03:04,783 --> 00:03:09,907

I'm a former GitHub Star and RxJS core team alum, and then I do a lot of stuff on

Cue 45 — 00:03:09,947 --> 00:03:15,787

the the social medias, YouTube, so the Modern Web podcast, Context Window,

Cue 46 — 00:03:15,827 --> 00:03:19,927

etc. so you can find a bunch of our content on YouTube at This Dot Media.

Cue 47 — 00:03:21,027 --> 00:03:26,127

I'm Coston Perkins. I lead tiger teams and consulting services and do a lot of

Cue 48 — 00:03:26,227 --> 00:03:28,687

development engineering.

Cue 49 — 00:03:30,667 --> 00:03:32,667

We get to work with some pretty cool orgs.

Cue 50 — 00:03:32,707 --> 00:03:38,047

So I'm originally from the Bay area, so Stripe, DocuSign, Roblox,

Cue 51 — 00:03:38,087 --> 00:03:40,887

Twilio, Cloudflare, and Accenture.

Cue 52 — 00:03:40,987 --> 00:03:44,627

A lot of these companies, we basically go in and help from an innovation

Cue 53 — 00:03:44,667 --> 00:03:48,207

perspective. It's been really fun working with OpenAI as well on,

Cue 54 — 00:03:48,707 --> 00:03:50,927

the stuff that they're doing with Codex.

Cue 55 — 00:03:51,187 --> 00:03:55,027

But if you need help or anything like that, it's, you know, we're always happy to

Cue 56 — 00:03:55,067 --> 00:03:59,147

chat, but it's just been really fun to just watch the evolution of,

Cue 57 — 00:03:59,187 --> 00:04:04,787

like, how all these teams are utilizing AI and kind of learning.

Cue 58 — 00:04:04,827 --> 00:04:05,987

Everyone's learning from each other.

Cue 59 — 00:04:06,027 --> 00:04:09,567

So it's a it's a fun adventure that we're in.

Cue 60 — 00:04:10,347 --> 00:04:13,667

So we're going to be talking about agentic development, obviously.

Cue 61 — 00:04:13,707 --> 00:04:19,387

And you know, this right here is sort of the we're going to when we talk in this

Cue 62 — 00:04:19,427 --> 00:04:24,227

talk about legacy development processes, it's crazy that we're calling this legacy.

Cue 63 — 00:04:24,267 --> 00:04:26,147

But I want you to think about this.

Cue 64 — 00:04:26,187 --> 00:04:30,627

So this is what we consider like developer-in-the-loop agentic development.

Cue 65 — 00:04:30,667 --> 00:04:33,787

And that's the thing that's becoming a legacy development process for us.

Cue 66 — 00:04:33,827 --> 00:04:39,667

Right. So in this process, if you think about it, you go to step one,

Cue 67 — 00:04:39,707 --> 00:04:42,167

run the app, you know, you're doing it with AI.

Cue 68 — 00:04:42,587 --> 00:04:44,107

You're building to the team standards.

Cue 69 — 00:04:44,147 --> 00:04:46,907

With AI, you're checking acceptance criteria.

Cue 70 — 00:04:46,947 --> 00:04:49,867

With AI, you're fixing failing checks with AI.

Cue 71 — 00:04:49,907 --> 00:04:53,827

Right? It's becoming really repetitive because you're doing every single step as a

Cue 72 — 00:04:53,867 --> 00:04:54,987

developer. But you're doing it.

Cue 73 — 00:04:55,027 --> 00:05:01,987

Yes, with AI. But that process, again, is becoming the new manual way

Cue 74 — 00:05:02,027 --> 00:05:05,707

of doing things. Every time you pick up a ticket, you have to do the same thing over

Cue 75 — 00:05:05,747 --> 00:05:11,267

and over and over again, where a new reality could actually be creating a system

Cue 76 — 00:05:11,307 --> 00:05:15,427

around this process and not having to be a developer in the loop.

Cue 77 — 00:05:15,467 --> 00:05:18,867

And I assume the five people that raise their hands, that 100% of their code is

Cue 78 — 00:05:18,907 --> 00:05:21,787

generated by AI are probably experiencing this.

Cue 79 — 00:05:21,827 --> 00:05:25,407

So I'd love to hear from you guys after to see what you all are doing as well.

Cue 80 — 00:05:26,307 --> 00:05:29,587

That's right. We want developers out of the loop.

Cue 81 — 00:05:29,627 --> 00:05:33,507

And to do this we're we're using a pattern engineers have trusted for a very long

Cue 82 — 00:05:33,547 --> 00:05:37,147

time for migrating off of legacy systems.

Cue 83 — 00:05:37,187 --> 00:05:41,447

And that's the strangler pattern, the strangler pattern roots,

Cue 84 — 00:05:42,067 --> 00:05:44,867

you know, one piece at a time, from the old system to the new.

Cue 85 — 00:05:44,907 --> 00:05:48,007

And that's how we think about building agentic workflows.

Cue 86 — 00:05:48,227 --> 00:05:52,947

We take people out of that loop, and we're actively mapping parts of the engineering

Cue 87 — 00:05:52,987 --> 00:05:59,527

stack and, and handing them to scripts that run pre-configured steps.

Cue 88 — 00:05:59,907 --> 00:06:03,667

Planning this way gives everyone one honest measure of progress,

Cue 89 — 00:06:03,707 --> 00:06:08,107

and that's, you know, how are we doing migrating from this?

Cue 90 — 00:06:08,147 --> 00:06:14,947

You know, a specific list of steps in our workflow to this new workflow.

Cue 91 — 00:06:14,987 --> 00:06:20,707

Playing this, our clients are using agentic workflows that now turn requirements

Cue 92 — 00:06:20,747 --> 00:06:25,047

into pull requests. And that's kind of what we're going to be talking about today,

Cue 93 — 00:06:25,427 --> 00:06:27,127

building those workflows.

Cue 94 — 00:06:31,307 --> 00:06:35,307

So if you're sort of working in the legacy developer in the loop process,

Cue 95 — 00:06:35,347 --> 00:06:38,667

right, then we are saying that like a person is always still involved,

Cue 96 — 00:06:38,707 --> 00:06:40,707

they're still determining what's next.

Cue 97 — 00:06:40,747 --> 00:06:44,927

What's done, what to paste in, what should we move into a workflow.

Cue 98 — 00:06:46,907 --> 00:06:53,227

And like, I think the general idea is that everything you're doing that seems

Cue 99 — 00:06:53,267 --> 00:06:56,867

manual, that seems repetitive, should actually be moved into a workflow.

Cue 100 — 00:06:56,907 --> 00:07:01,007

So hopefully most of you are familiar with Claude workflows as a good example,

Cue 101 — 00:07:01,187 --> 00:07:05,387

but a workflow is basically just a script that runs agents and scripts,

Cue 102 — 00:07:05,427 --> 00:07:10,327

right? So you can use it to break any task you're working on into small pieces,

Cue 103 — 00:07:10,627 --> 00:07:14,247

giving them to the agent one at a time, recording what's done.

Cue 104 — 00:07:14,387 --> 00:07:18,707

One recommendation I recently gave a talk at S, and it was more about the mentality

Cue 105 — 00:07:18,747 --> 00:07:22,867

of how you should start rethinking about, you know, how you're working with AI as a

Cue 106 — 00:07:22,907 --> 00:07:25,867

developer. And I think the biggest takeaway was like, okay, well,

Cue 107 — 00:07:25,907 --> 00:07:29,307

Tracy, what's next? And I think one recommendation is just, hey,

Cue 108 — 00:07:29,347 --> 00:07:33,067

just start with one workflow. What is the task that you repeat the most that you

Cue 109 — 00:07:33,107 --> 00:07:36,947

keep having to correct? And if you find yourself repeating prompts or something like

Cue 110 — 00:07:36,987 --> 00:07:40,787

that, that is basically a really good indicator that that should just be moved into

Cue 111 — 00:07:40,827 --> 00:07:41,807

a workflow.

Cue 112 — 00:07:44,267 --> 00:07:49,267

So this is an example of a real migration workflow helping a client convert a code

Cue 113 — 00:07:49,307 --> 00:07:51,507

base from CoffeeScript to TypeScript.

Cue 114 — 00:07:51,547 --> 00:07:58,427

I think this is a common type of the modernization is a common use case for AI

Cue 115 — 00:07:58,467 --> 00:08:02,827

these days, and at first everything of this migration was handled in a chat.

Cue 116 — 00:08:02,867 --> 00:08:07,827

Then the workflow was composed within like a slash command, just basically one big

Cue 117 — 00:08:07,867 --> 00:08:13,187

agent prompt. After that, we pulled everything out into a script and we could just

Cue 118 — 00:08:13,227 --> 00:08:18,287

pass, you know, a task into this workflow and the workflow would handle it.

Cue 119 — 00:08:19,267 --> 00:08:25,187

This shows a person leaving the orchestration role, and, it doesn't really show if

Cue 120 — 00:08:25,227 --> 00:08:30,347

the code is right, but we're going to continue iterating on this pattern in this

Cue 121 — 00:08:30,387 --> 00:08:31,187

talk.

Cue 122 — 00:08:33,787 --> 00:08:35,587

So step one is just running the app, right?

Cue 123 — 00:08:35,627 --> 00:08:37,107

I think this is the biggest thing.

Cue 124 — 00:08:37,147 --> 00:08:42,207

And it's funny that just running the app matters so much when it comes to

Cue 125 — 00:08:42,467 --> 00:08:45,787

development. But that's the first problem to solve.

Cue 126 — 00:08:45,827 --> 00:08:48,307

Like can your agent actually run your app without you?

Cue 127 — 00:08:48,347 --> 00:08:53,027

One thing we see a lot in engineering orgs is just like the setup to run an app is

Cue 128 — 00:08:53,067 --> 00:08:55,027

really dependent on developers, right?

Cue 129 — 00:08:55,067 --> 00:08:58,827

It's really normal. And I'm sure all of you experience this to see engineers being

Cue 130 — 00:08:58,867 --> 00:09:02,547

dependent on access to external services, databases, auth.

Cue 131 — 00:09:02,587 --> 00:09:06,867

Right. There's there's so much there that just can't an agent just can't go in and

Cue 132 — 00:09:06,907 --> 00:09:07,707

run.

Cue 133 — 00:09:09,147 --> 00:09:12,247

No, you're still there. Oh, sorry.

Cue 134 — 00:09:13,147 --> 00:09:17,747

Yeah. So, like, we believe that one command should generally be all that it's

Cue 135 — 00:09:17,787 --> 00:09:22,667

required to start an app. And that should also start your databases,

Cue 136 — 00:09:22,707 --> 00:09:29,494

your services, anything. Your app depends on seeded users for like key

Cue 137 — 00:09:29,534 --> 00:09:34,747

roles in your database. And all of this should be able to run in a VM or sandbox.

Cue 138 — 00:09:34,787 --> 00:09:41,287

That's the target. And if your agent can't do this, like if the agent can't start

Cue 139 — 00:09:41,327 --> 00:09:43,947

the app, it can't run the test, it can't click through features,

Cue 140 — 00:09:43,987 --> 00:09:48,207

it can't take screenshots or reviewer needs, can't record videos.

Cue 141 — 00:09:48,987 --> 00:09:52,887

And you know, we want your agent to be doing all of that.

Cue 142 — 00:09:53,107 --> 00:09:57,387

And, you know, you compare this to like getting a new hire set up in your code base.

Cue 143 — 00:09:57,427 --> 00:10:02,787

Like, you know, you pay that tax once learning all the ins and outs and API keys and

Cue 144 — 00:10:02,827 --> 00:10:06,727

everything. The agent's going to pay it every single time it runs.

Cue 145 — 00:10:06,907 --> 00:10:10,547

So you want to start thinking about simplifying your process.

Cue 146 — 00:10:10,587 --> 00:10:15,507

And if the app depends on outside services, you want to stand up local versions of

Cue 147 — 00:10:15,547 --> 00:10:21,567

those outside services like storage, ML, containerized, whatever you got to do to,

Cue 148 — 00:10:22,067 --> 00:10:26,567

you know, reduce the friction for the agent to be able to work in a sandbox.

Cue 149 — 00:10:26,987 --> 00:10:30,447

It's great if you if your agent can just, you know, work offline.

Cue 150 — 00:10:30,507 --> 00:10:34,287

That's that's a big plus. Besides the agent being able to connect.

Cue 151 — 00:10:34,707 --> 00:10:39,087

And if it takes more than one command on a clean machine, like you should fix that.

Cue 152 — 00:10:39,227 --> 00:10:42,427

I think the great part about this is it's also so useful for humans.

Cue 153 — 00:10:42,467 --> 00:10:45,747

Yeah. Yeah. It makes onboarding a lot easier.

Cue 154 — 00:10:45,787 --> 00:10:47,527

And we've seen that with our clients.

Cue 155 — 00:10:48,267 --> 00:10:54,067

So here's what that looks like at a at a big energy software company that we work

Cue 156 — 00:10:54,107 --> 00:10:57,507

with. We did this on more than five legacy apps.

Cue 157 — 00:10:57,547 --> 00:11:03,247

And like they were all very difficult tons of services, a lot of dependencies.

Cue 158 — 00:11:03,947 --> 00:11:08,747

And we witnessed the developers setting these things up by hand before anything

Cue 159 — 00:11:08,787 --> 00:11:14,227

would run on every task. Now each one of them, you know, each one of these

Cue 160 — 00:11:14,267 --> 00:11:16,567

applications come with a single command.

Cue 161 — 00:11:16,977 --> 00:11:20,307

They can run in a sandbox and the agent runs.

Cue 162 — 00:11:20,347 --> 00:11:22,947

You know, these applications end in on their own.

Cue 163 — 00:11:22,987 --> 00:11:28,387

And this is critical, critical for AI to work at this modern pace where we're

Cue 164 — 00:11:28,427 --> 00:11:29,567

pushing toward.

Cue 165 — 00:11:31,427 --> 00:11:37,557

So the next thing is, you know, you've been in the loop to build to the team

Cue 166 — 00:11:37,597 --> 00:11:41,087

standards. We want to fix that.

Cue 167 — 00:11:41,867 --> 00:11:45,387

You want to give every agent the team's engineering standards.

Cue 168 — 00:11:45,427 --> 00:11:50,027

Your agent needs good context and good enforcement of those standards.

Cue 169 — 00:11:50,067 --> 00:11:55,667

Context is just like the the cultural things, and maybe documents about like

Cue 170 — 00:11:55,707 --> 00:11:58,887

architecture, standards, quality bar.

Cue 171 — 00:11:59,427 --> 00:12:04,587

You can build a references to things like your design tokens,

Cue 172 — 00:12:04,627 --> 00:12:10,227

spec components to reuse. And a lot of this is in your AGENTS.md,

Cue 173 — 00:12:10,267 --> 00:12:16,067

but you can also input these things in and directly into your prompts in your

Cue 174 — 00:12:16,107 --> 00:12:19,727

workflow. And then enforcement is just guardrails.

Cue 175 — 00:12:19,947 --> 00:12:23,327

They should be deterministic and fail on drift like your lint rules.

Cue 176 — 00:12:25,387 --> 00:12:29,347

Tests anything that your agent can read and fix, and you want it to be able to read

Cue 177 — 00:12:29,387 --> 00:12:32,087

those errors and react to them.

Cue 178 — 00:12:34,427 --> 00:12:38,207

And I can't stress enough, make your culture deterministic.

Cue 179 — 00:12:39,467 --> 00:12:44,927

You know, it's so easy to build lint rules now, like your agent can just write them.

Cue 180 — 00:12:44,987 --> 00:12:50,447

So put everything into linting, you know, make sure you have formatter type checks,

Cue 181 — 00:12:51,827 --> 00:12:56,607

you know, abuse git hooks just to defend the code base from the agent.

Cue 182 — 00:12:56,827 --> 00:13:00,527

All of these things you should be having in your code base now.

Cue 183 — 00:13:02,187 --> 00:13:06,707

And we're getting closer to my favorite example of where our workflow can save the

Cue 184 — 00:13:06,747 --> 00:13:11,927

day. And you want to put deterministic work outside of the model.

Cue 185 — 00:13:12,867 --> 00:13:16,807

You know, like if, if it has one right answer, you want to script it.

Cue 186 — 00:13:17,027 --> 00:13:19,887

So that can be like converting something, formatting something,

Cue 187 — 00:13:20,027 --> 00:13:25,027

generating something. Lint test build and your workflow can run them.

Cue 188 — 00:13:25,067 --> 00:13:27,667

And then you know anything that needs judgment.

Cue 189 — 00:13:27,707 --> 00:13:32,147

Have your model work on it. It can fix something that a converter couldn't do.

Cue 190 — 00:13:32,187 --> 00:13:36,487

Choose type some names, fit the change to nearby code.

Cue 191 — 00:13:37,787 --> 00:13:42,807

And now we're at the moment where I talk about the example that I like.

Cue 192 — 00:13:43,427 --> 00:13:47,427

And that's the CoffeeScript to TypeScript migration, because it was a big learning

Cue 193 — 00:13:47,467 --> 00:13:52,107

opportunity for us in early 2025 about workflows.

Cue 194 — 00:13:52,147 --> 00:13:55,557

When we started, we were doing the migrations with slash commands and,

Cue 195 — 00:13:55,597 --> 00:13:59,107

you know, just like steps in a big prompt and maybe referencing other slash

Cue 196 — 00:13:59,147 --> 00:14:05,827

commands. And one one of those steps ran a CLI tool

Cue 197 — 00:14:05,867 --> 00:14:10,707

and that did a deterministic 1 to 1 conversion of CoffeeScript to TypeScript.

Cue 198 — 00:14:10,747 --> 00:14:14,107

Then an agent would follow up and handle types, cleanup and polish.

Cue 199 — 00:14:14,147 --> 00:14:15,507

And we got a really nice conversion.

Cue 200 — 00:14:15,547 --> 00:14:21,707

It worked really well one day on a developers machine, the CLI wasn't available,

Cue 201 — 00:14:21,747 --> 00:14:27,667

and instead of throwing an error and failing right there, the agent converted the

Cue 202 — 00:14:27,707 --> 00:14:33,827

CoffeeScript to TypeScript itself, and it did a pretty good job converting the code.

Cue 203 — 00:14:33,867 --> 00:14:37,747

I mean, like, you wouldn't notice it, just like reviewing the PR,

Cue 204 — 00:14:37,787 --> 00:14:41,587

but it introduced behavioral change in the application, which was a big failure from

Cue 205 — 00:14:41,627 --> 00:14:47,307

our point of view. And what we learned was that the CLI had to come out of the

Cue 206 — 00:14:47,347 --> 00:14:53,147

agent's hands. And we can assure the conversion runs by composing a script that runs

Cue 207 — 00:14:53,187 --> 00:15:00,027

the conversion and then and then follow it up with agent steps and

Cue 208 — 00:15:00,067 --> 00:15:03,487

back and forth between scripts and agents is what we now call workflow.

Cue 209 — 00:15:03,707 --> 00:15:07,467

And we really like workflows. I think that's like such an important thing,

Cue 210 — 00:15:07,507 --> 00:15:11,867

right? This whole idea that, like, you really have to take anything that's

Cue 211 — 00:15:11,907 --> 00:15:17,467

deterministic out of whatever you're doing with agents and have it sort of somewhere

Cue 212 — 00:15:17,507 --> 00:15:20,267

else. Because, you know, the biggest thing we're trying to deal with,

Cue 213 — 00:15:20,307 --> 00:15:24,227

right, with agents is like, how do you deal with all the non-deterministic stuff and

Cue 214 — 00:15:24,267 --> 00:15:28,927

how do you get the same result every single time on the things that actually matter?

Cue 226 — 00:16:11,927 --> 00:16:15,667

Okay, so now we have two steps handed over.

Cue 227 — 00:16:16,327 --> 00:16:19,947

Now. Step three check acceptance criteria.

Cue 228 — 00:16:20,447 --> 00:16:23,507

We we never want to let a worker be its own judge.

Cue 229 — 00:16:23,607 --> 00:16:27,947

We want quality gates that you know every step of the workflow.

Cue 230 — 00:16:29,647 --> 00:16:34,847

So, you know, the first type of quality check is just deterministic checks,

Cue 231 — 00:16:34,887 --> 00:16:36,267

lint tests, build.

Cue 232 — 00:16:38,007 --> 00:16:42,967

You, you know, you want your you want to prove what, what you can with deterministic

Cue 233 — 00:16:43,007 --> 00:16:43,807

tests.

Cue 234 — 00:16:45,167 --> 00:16:51,327

The next one is, is a judge. This is just a second agent following up a step,

Cue 235 — 00:16:51,367 --> 00:16:56,487

you know, an agent step. And just like evaluating, did it do what it was supposed to

Cue 236 — 00:16:56,527 --> 00:17:02,307

and then, you know, like pass or fail true or false.

Cue 237 — 00:17:02,487 --> 00:17:05,127

And then we can talk about what we do after that.

Cue 238 — 00:17:05,167 --> 00:17:08,687

And then the last is just human like, is it the right change?

Cue 239 — 00:17:08,727 --> 00:17:14,487

Is the risk okay. Like, there's still, you know, that developer taking a look at the

Cue 240 — 00:17:14,527 --> 00:17:20,107

code. So I want to I want to dive deeper into this.

Cue 241 — 00:17:21,167 --> 00:17:24,667

You want to give every step of explicit definition of done.

Cue 242 — 00:17:24,767 --> 00:17:27,107

So now we're starting to look at a workflow.

Cue 243 — 00:17:28,207 --> 00:17:31,127

We have three, three like 3 to 7.

Cue 244 — 00:17:31,167 --> 00:17:36,047

Are the steps prompt. Let's keep the plan, test my test right,

Cue 245 — 00:17:36,087 --> 00:17:42,899

this file, etc.. Line two is just a flag on this workflow: LLM as a

Cue 246 — 00:17:42,939 --> 00:17:46,447

judge. And why is this just a flag?

Cue 247 — 00:17:46,487 --> 00:17:51,327

Because all it's doing is reusing the same prompt that we use for LLM as a judge.

Cue 248 — 00:17:51,367 --> 00:17:56,407

Like did the step do what it was asked if the file is missing or a task was skipped?

Cue 249 — 00:17:56,447 --> 00:18:01,147

It fails and a step reruns with the with the judges feedback.

Cue 250 — 00:18:01,607 --> 00:18:04,827

And then the last is a script step.

Cue 251 — 00:18:05,847 --> 00:18:11,307

So this next step needs that file that was created in by the agent.

Cue 252 — 00:18:11,727 --> 00:18:16,367

And what we can do is just have a deterministic check, like does the file exist?

Cue 253 — 00:18:16,407 --> 00:18:21,127

And if it, if it doesn't exist, like, you know, fail the workflow,

Cue 254 — 00:18:21,167 --> 00:18:25,967

stop there or maybe auto fix, you know, you can always go back to the previous step

Cue 255 — 00:18:26,007 --> 00:18:28,867

and and with some feedback.

Cue 256 — 00:18:30,897 --> 00:18:34,447

I'll give you a great example of why having just like that separate LLM as a judge

Cue 257 — 00:18:34,487 --> 00:18:39,207

is so important. And I've definitely seen people over engineer this where they're

Cue 258 — 00:18:39,247 --> 00:18:43,607

like, well, I'm having one model review and then another model review,

Cue 259 — 00:18:43,647 --> 00:18:46,927

and I'm having them argue with each other and, you know, people who are really,

Cue 260 — 00:18:46,967 --> 00:18:50,967

really deep into this just really, really have at it, if you will.

Cue 261 — 00:18:51,007 --> 00:18:51,967

You know, we're all engineers.

Cue 262 — 00:18:52,007 --> 00:18:56,007

So we like to sort of, you know, as much as possible, but, you know,

Cue 263 — 00:18:56,047 --> 00:19:00,167

just simply having a separate LLM as a judge is just so important.

Cue 264 — 00:19:00,207 --> 00:19:03,847

Right. Like we were recently completing this AngularJS migration,

Cue 265 — 00:19:03,887 --> 00:19:09,447

and agents kept reporting a step as done when you know the task,

Cue 266 — 00:19:09,487 --> 00:19:11,807

some tasks from this plan had actually been skipped.

Cue 267 — 00:19:11,847 --> 00:19:16,967

So the judge would basically work, read the work against the plan.

Cue 268 — 00:19:17,007 --> 00:19:20,567

It found the missing test and it sent the step back.

Cue 269 — 00:19:20,607 --> 00:19:22,607

And this is all before humans touching it, right?

Cue 270 — 00:19:22,647 --> 00:19:25,707

This is an agent working with an agent being its judge.

Cue 271 — 00:19:25,887 --> 00:19:31,607

And then the agent was able to go correct what the judge found and finish it before

Cue 272 — 00:19:31,647 --> 00:19:36,207

any person looked at it. And, you know, again, we're really trying to again decrease

Cue 273 — 00:19:36,247 --> 00:19:41,047

the amount of time the developer has to be in the loop.

Cue 274 — 00:19:41,087 --> 00:19:44,567

And again, this is just such a great example of why the worker should really just

Cue 275 — 00:19:44,607 --> 00:19:46,827

never be the judge itself.

Cue 276 — 00:19:48,047 --> 00:19:55,007

And so step four is fixed. Failing checks, you know, in the loop without you

Cue 277 — 00:19:55,047 --> 00:19:58,667

there, you want to run CI on your workflow.

Cue 278 — 00:19:59,087 --> 00:20:00,587

You want agents to

Cue 279 — 00:20:01,967 --> 00:20:08,767

to, you know, basically catch any CI issues before it gets pushed

Cue 280 — 00:20:08,807 --> 00:20:13,507

and have to actually reach maybe your out of the loop CI.

Cue 281 — 00:20:14,567 --> 00:20:16,367

Your workflow should run the same lint test.

Cue 282 — 00:20:16,407 --> 00:20:21,047

Build commands that CI runs before anything is pushed.

Cue 283 — 00:20:21,087 --> 00:20:27,007

And you know, a failure is going to get caught by by an agent right away in cases

Cue 284 — 00:20:27,047 --> 00:20:32,887

like this. And, you know, outside of the loop, you can have your regular real CI.

Cue 285 — 00:20:32,927 --> 00:20:37,287

But I've been seeing a lot of folks do really clever things to prevent like

Cue 286 — 00:20:37,327 --> 00:20:42,767

duplicated work, you know, especially with custom harnesses and factories and

Cue 287 — 00:20:42,807 --> 00:20:48,727

everything. It's there's a lot of innovation to, to save time and be able to push

Cue 288 — 00:20:48,767 --> 00:20:50,027

faster here.

Cue 289 — 00:20:52,007 --> 00:20:55,767

So the next step is writing the pull request, which I think a lot of you are

Cue 290 — 00:20:55,807 --> 00:20:59,987

familiar. How many of you all are using AI to write your pull requests?

Cue 291 — 00:21:00,207 --> 00:21:02,427

Raise of hands? I see a lot more.

Cue 292 — 00:21:02,647 --> 00:21:06,447

Okay, cool. Yeah, this is something that we should all be familiar with since it's

Cue 293 — 00:21:06,487 --> 00:21:08,787

been done for a while now, right?

Cue 294 — 00:21:08,887 --> 00:21:13,767

But the difference here is writing a request for a team member who has never seen

Cue 295 — 00:21:13,807 --> 00:21:16,647

the implementation is a different story.

Cue 296 — 00:21:16,687 --> 00:21:20,887

So one thing we're trying to do is speed up the understanding of the work that was

Cue 297 — 00:21:20,927 --> 00:21:26,407

implemented. So nowadays and this is great with AI because AI can basically do all

Cue 298 — 00:21:26,447 --> 00:21:30,847

the work for you. Every pull request needs to carry its own evidence,

Cue 299 — 00:21:30,887 --> 00:21:35,407

right? A developer is just again out of the loop in the implementation.

Cue 300 — 00:21:35,447 --> 00:21:37,887

So we also need to be more careful at the same time.

Cue 301 — 00:21:37,927 --> 00:21:40,147

Speed it up, but also be more careful.

Cue 302 — 00:21:40,247 --> 00:21:44,967

On most teams. The longest wait in delivery is review, right?

Cue 303 — 00:21:45,007 --> 00:21:48,587

We're working on this one client right now, and there are some PRs that have been

Cue 304 — 00:21:48,627 --> 00:21:53,827

open for two years. So that's how long this migration has been going on.

Cue 305 — 00:21:54,847 --> 00:21:58,127

But you know, without evidence, if you don't have evidence in each one of those pull

Cue 306 — 00:21:58,167 --> 00:22:01,567

requests, then a review has to pull down the branch, run it,

Cue 307 — 00:22:01,607 --> 00:22:03,207

and work out what the change was meant to be.

Cue 308 — 00:22:03,247 --> 00:22:04,767

I'm sure all of you have done this before.

Cue 309 — 00:22:04,807 --> 00:22:07,047

Had to deal with this before. You can actually even judge anything.

Cue 310 — 00:22:07,087 --> 00:22:11,707

And it's exhausting and annoying, especially with the influx of pull requests.

Cue 311 — 00:22:11,887 --> 00:22:13,807

So you know what should really happen.

Cue 312 — 00:22:13,847 --> 00:22:18,247

And again, it's so easy to set up with AI these days is every pull request needs to

Cue 313 — 00:22:18,287 --> 00:22:24,007

like list each requirement mark if the requirements are met or not with screenshots

Cue 314 — 00:22:24,047 --> 00:22:29,267

with a short video of the change and the result of every single check.

Cue 315 — 00:22:29,447 --> 00:22:32,407

So if you're not sending that up now, it's definitely a best practice that you

Cue 316 — 00:22:32,447 --> 00:22:35,507

should be instilling within your your teams.

Cue 317 — 00:22:35,567 --> 00:22:39,687

Yeah. And you know your agents seeing all all of this to the screenshots,

Cue 318 — 00:22:39,727 --> 00:22:41,727

composing a video for you whatever.

Cue 319 — 00:22:41,767 --> 00:22:47,067

Like your agents being able to review all of that and self-correct.

Cue 320 — 00:22:47,647 --> 00:22:49,527

And we you know, we've done this.

Cue 321 — 00:22:49,567 --> 00:22:55,987

And here's what happened. Reviews got faster without good evidence of pull requests.

Cue 322 — 00:22:57,247 --> 00:23:01,287

You know, can take take a while because a developer has to pull down the code,

Cue 323 — 00:23:01,327 --> 00:23:06,087

you know, read all the code, look at the app that while it's running or whatever.

Cue 324 — 00:23:06,127 --> 00:23:12,647

But once you have a workflow, you know, automatically present the

Cue 325 — 00:23:12,687 --> 00:23:14,587

evidence in the pull requests.

Cue 326 — 00:23:15,647 --> 00:23:21,127

People reviewing your code can have a lot more confidence, understand it faster,

Cue 327 — 00:23:21,167 --> 00:23:26,987

provide feedback faster, and get through that review cycle much faster.

Cue 328 — 00:23:28,327 --> 00:23:31,627

So here's one more look at how we're designing workflows.

Cue 329 — 00:23:32,087 --> 00:23:37,567

Task comes in and the the workflow plans it into pieces.

Cue 330 — 00:23:37,607 --> 00:23:42,967

So each piece runs an agent step and a judge checks that work right away like after

Cue 331 — 00:23:43,007 --> 00:23:48,927

every step. Then scripts run, lint tests build, when a judge or a script fails,

Cue 332 — 00:23:48,967 --> 00:23:53,827

a step retries. And, you know, we typically set this for like five times,

Cue 333 — 00:23:54,527 --> 00:23:59,287

before a person sees it. Every step runs in a working environment with the team

Cue 334 — 00:23:59,327 --> 00:24:02,247

standards and in the relevant prompts.

Cue 335 — 00:24:02,287 --> 00:24:05,247

And what comes out is a pull requests carrying its own evidence.

Cue 336 — 00:24:05,287 --> 00:24:12,267

And CI checks it again. And, you know, most of the time, like I think every time,

Cue 337 — 00:24:13,447 --> 00:24:17,707

you know, everything's good at that point because CI is running in the workflows.

Cue 338 — 00:24:18,487 --> 00:24:21,907

So no single part does much alone.

Cue 339 — 00:24:22,687 --> 00:24:27,887

And, you know, a judge really has nothing to look at without a running app and an

Cue 340 — 00:24:27,927 --> 00:24:31,447

agent without scripts. I mean, they're just going to improvise.

Cue 341 — 00:24:31,487 --> 00:24:38,127

So together they compound. And that's how one team member got to 230 pull requests

Cue 342 — 00:24:38,167 --> 00:24:44,937

out of 231 with like, exceptional quality and like, not much churn

Cue 343 — 00:24:44,977 --> 00:24:48,707

on, merge conflicts or review feedback.

Cue 344 — 00:24:51,127 --> 00:24:58,007

So we are we've had six steps handed over and this is an

Cue 345 — 00:24:58,047 --> 00:25:00,847

interesting one. Respond to production signals.

Cue 346 — 00:25:00,887 --> 00:25:02,987

I wonder how many people are doing this.

Cue 347 — 00:25:03,727 --> 00:25:09,007

Our numbers, you know, we're stop at the pull request, but the work keeps going.

Cue 348 — 00:25:09,047 --> 00:25:13,767

After merge, production signals are the next step to strangle like an error or

Cue 349 — 00:25:13,807 --> 00:25:19,007

performance issue. Showing up in the logs should kick off an agent using one of

Cue 350 — 00:25:19,047 --> 00:25:21,487

those workflows. A Claude workflow.

Cue 351 — 00:25:21,527 --> 00:25:28,587

Custom workflow that triages it and finds the cause, opens pull requests,

Cue 352 — 00:25:29,167 --> 00:25:31,047

and includes all of the proof.

Cue 353 — 00:25:31,087 --> 00:25:33,047

Like this is how we know it's fixed.

Cue 354 — 00:25:33,087 --> 00:25:38,047

What is like. I'm seeing graphs in my pull requests now of agents proving that

Cue 355 — 00:25:38,087 --> 00:25:40,907

things are, you know, fixed now.

Cue 356 — 00:25:41,367 --> 00:25:43,287

And it follows the same workflow process.

Cue 357 — 00:25:43,327 --> 00:25:48,727

There's a lot to automate, and we'd love to talk to you about it into all this.

Cue 358 — 00:25:48,767 --> 00:25:53,127

So, yeah. It's a fun adventure.

Cue 359 — 00:25:53,167 --> 00:25:56,287

And this is like, definitely like a new reality that we're moving into,

Cue 360 — 00:25:56,327 --> 00:26:00,967

right? This like, you know, again, we talk about this like legacy developer in the

Cue 361 — 00:26:01,007 --> 00:26:05,327

loop AI development process. And the world is just moving so fast.

Cue 362 — 00:26:05,367 --> 00:26:07,647

Right. But we're just able to automate so much.

Cue 363 — 00:26:07,687 --> 00:26:13,887

Right. But you know, the there's more to just getting from like a ticket or a task

Cue 364 — 00:26:13,927 --> 00:26:18,327

to a PR a the most important thing is us as developers of course,

Cue 365 — 00:26:18,367 --> 00:26:22,827

need to make sure that we are checking to see if it's good.

Cue 366 — 00:26:22,927 --> 00:26:26,587

So people are still needed because you know what's actually worth building?

Cue 367 — 00:26:27,287 --> 00:26:29,867

You know who's writing the requirements?

Cue 368 — 00:26:30,727 --> 00:26:37,007

I talk a lot about this. As you know, the engineers that are the most impactful and

Cue 369 — 00:26:37,047 --> 00:26:39,727

the ones that are going to keep their jobs these days are the ones who are more

Cue 370 — 00:26:39,767 --> 00:26:42,907

product oriented, right? The ones who are closer to the business.

Cue 371 — 00:26:43,487 --> 00:26:46,167

But really, you know, as engineers were designing the system,

Cue 372 — 00:26:46,207 --> 00:26:47,647

we're designing the workflows.

Cue 373 — 00:26:47,687 --> 00:26:51,607

It's much to me it's more exciting because, like, you basically get to be an

Cue 374 — 00:26:51,647 --> 00:26:56,027

architect, even as a junior developer, you know, you get to do more fun stuff,

Cue 375 — 00:26:56,327 --> 00:27:00,207

checking, you know, you're designing the checks, you're designing the escalation

Cue 376 — 00:27:00,247 --> 00:27:02,807

rules, you're deciding what's what.

Cue 377 — 00:27:02,847 --> 00:27:06,727

Risk is acceptable, right? Like what should the agent actually be working on?

Cue 378 — 00:27:06,767 --> 00:27:09,587

What part of the code base, what goes to production?

Cue 379 — 00:27:10,287 --> 00:27:13,807

And so agents are definitely taking over the execution part of things.

Cue 380 — 00:27:13,847 --> 00:27:16,767

But the decisions, right. The important decisions are really staying with us.

Cue 381 — 00:27:16,807 --> 00:27:22,247

And for an organization, I feel like that's really the shift engineer spending less

Cue 382 — 00:27:22,287 --> 00:27:26,487

time sort of doing the steps and more designing the system.

Cue 383 — 00:27:26,527 --> 00:27:32,687

And it's it's been a really fun shift as well as an agency because now you're having

Cue 384 — 00:27:32,727 --> 00:27:33,547

like AI,

Cue 385 — 00:27:34,847 --> 00:27:40,367

you're having AI request proposals and AI is requesting, like these crazy design

Cue 386 — 00:27:40,407 --> 00:27:42,807

documents before they even talk to you.

Cue 387 — 00:27:42,847 --> 00:27:46,727

Right. Like this company that's using AI to try to get bids.

Cue 388 — 00:27:46,767 --> 00:27:49,967

And it's like, well, you know, all of these design docs can just get fed into an

Cue 389 — 00:27:50,007 --> 00:27:54,367

agent. So as a consultancy, it's fun to just sort of figure that that little dance

Cue 390 — 00:27:54,407 --> 00:27:59,087

out. But I would say, like as you're going back to your job and,

Cue 391 — 00:27:59,127 --> 00:28:02,407

and thinking about how to do this, really thinking about, hey,

Cue 392 — 00:28:02,447 --> 00:28:07,527

what's like every time that somebody steps in on an agent's work,

Cue 393 — 00:28:07,567 --> 00:28:12,327

right. Is it a prompt that you keep saying or rewriting?

Cue 394 — 00:28:12,367 --> 00:28:14,567

Is it a rerun? Is it a manual setup?

Cue 395 — 00:28:14,607 --> 00:28:18,487

Is it a review comment that you keep leaving because you have to in order to make it

Cue 396 — 00:28:18,527 --> 00:28:23,407

work? Choose one thing that's repetitive and safe to remove and then just build the

Cue 397 — 00:28:23,447 --> 00:28:27,087

workflow. I was at an IBM developer conference.

Cue 398 — 00:28:27,127 --> 00:28:31,407

The other was the last week, and somebody on the IBM Bob team,

Cue 399 — 00:28:31,447 --> 00:28:35,067

which is their coding agent, was like, just pick one workflow.

Cue 400 — 00:28:35,327 --> 00:28:39,647

IBM Bob is very workflow focused, but, you know, just build that workflow,

Cue 401 — 00:28:39,687 --> 00:28:44,607

build that script every single day, you know, whatever check that replaces the thing

Cue 402 — 00:28:44,647 --> 00:28:48,167

that you're doing manually. And then make sure that you're actually measuring

Cue 403 — 00:28:48,207 --> 00:28:52,327

autonomy and quality together so that you see, like both what merges without help

Cue 404 — 00:28:52,367 --> 00:28:55,447

and then also what breaks after it does.

Cue 405 — 00:28:55,487 --> 00:28:56,647

And then just pick the next one.

Cue 406 — 00:28:56,687 --> 00:29:00,567

Right. And so Coston mentioned the strangler pattern earlier on in the talk.

Cue 407 — 00:29:00,607 --> 00:29:02,167

And that's kind of what it is, right?

Cue 408 — 00:29:02,207 --> 00:29:07,287

Do that. Run it on your own process and then prove that it works and repeat.

Cue 409 — 00:29:07,327 --> 00:29:11,187

So before we wrap up there are a few ways to keep this conversation going.

Cue 410 — 00:29:11,487 --> 00:29:14,807

If you don't know this, like we're out on the internet a lot and we do a lot of

Cue 411 — 00:29:14,847 --> 00:29:17,367

really fun things for executives.

Cue 412 — 00:29:17,407 --> 00:29:21,007

I spend a lot of time in the executive networks, and so there's an AI executive

Cue 413 — 00:29:21,047 --> 00:29:26,167

workshop that I run. If you are a VP or up, you know, you should come to it and to

Cue 414 — 00:29:26,207 --> 00:29:31,627

see how I use AI, not in the development world.

Cue 415 — 00:29:32,607 --> 00:29:35,967

My engineering manager told me I use 3 billion tokens in the past 30 days,

Cue 416 — 00:29:36,007 --> 00:29:38,607

so I don't know if I should be proud or what, but, you know,

Cue 417 — 00:29:38,647 --> 00:29:42,207

that's what she told me. And so anyways, the executive workshops.

Cue 418 — 00:29:42,247 --> 00:29:45,647

Great. If you're seeing your leader, sort of like, you know,

Cue 419 — 00:29:45,687 --> 00:29:49,407

pointing down at you and trying to one of the reasons why I'm hosting this is

Cue 420 — 00:29:49,447 --> 00:29:53,527

because, it's really important for executives to, like, get on board and use AI,

Cue 421 — 00:29:53,567 --> 00:29:55,767

right? So they better understand what developers are doing.

Cue 422 — 00:29:55,807 --> 00:30:00,407

We also host an AI workshop. So if you're interested in sort of getting into the

Cue 423 — 00:30:00,447 --> 00:30:02,367

more advanced topics, you should definitely do that.

Cue 424 — 00:30:02,407 --> 00:30:04,687

It's taught by or CTO and then see.

Cue 425 — 00:30:04,727 --> 00:30:08,887

So connect is a great way. Again, if you're like VP level up to just come and talk

Cue 426 — 00:30:08,927 --> 00:30:12,507

to other executives about what they're doing.

Cue 427 — 00:30:12,807 --> 00:30:16,047

It's really fun to just sort of watch the trends every few months and how those

Cue 428 — 00:30:16,087 --> 00:30:22,237

changes. So you can check out thisdot.co for all of that, or you can find myself in

Cue 429 — 00:30:22,407 --> 00:30:28,087

on X on LinkedIn. You know, we're happy to come do like a lunch and learn or

Cue 430 — 00:30:28,127 --> 00:30:31,167

anything like that and come talk to you all about all the things.

Cue 431 — 00:30:31,207 --> 00:30:33,067

So thanks all for having us.

skills

talk-lee-perkins-code-readiness

tile.json