CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-jain-verifying-intent/references/

Corrected transcript: We Stopped Reading Diffs and Started Verifying Intent

Ankit Jain · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=yKoP9E1NL-w

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:00,096 --> 00:00:01,636

Thank you everyone for joining.

Cue 2 — 00:00:01,976 --> 00:00:04,176

Really excited to have all of you here.

Cue 3 — 00:00:04,216 --> 00:00:06,696

And we're going to be talking about code reviews.

Cue 4 — 00:00:06,736 --> 00:00:08,796

So who doesn't love code reviews?

Cue 5 — 00:00:09,456 --> 00:00:13,759

Before we get started, as Macey said, one of the things I would like to plug is our

Cue 6 — 00:00:13,799 --> 00:00:19,016

Hangar community. It's a DX community, which is where we bring in amazing speakers

Cue 7 — 00:00:19,056 --> 00:00:23,556

to talk about different challenges around developer experience as long as developers

Cue 8 — 00:00:23,816 --> 00:00:28,896

exist. We would always want to build better tools for them, so this is a great place

Cue 9 — 00:00:28,936 --> 00:00:30,616

to come join. It's a free community.

Cue 10 — 00:00:30,656 --> 00:00:33,176

You can all sign up. It's community.

Cue 11 — 00:00:33,216 --> 00:00:39,696

So let's get rolling. Slide. Okay.

Cue 12 — 00:00:39,736 --> 00:00:43,956

So I will share our battles with code reviews in the last one year.

Cue 13 — 00:00:44,376 --> 00:00:50,616

But we also sell tools. We have built these tools along the ride as we evolve our

Cue 14 — 00:00:50,656 --> 00:00:54,236

code reviews. So please be suspicious of me.

Cue 15 — 00:00:55,816 --> 00:01:00,056

So I'll start with. This is an article I wrote now almost a year ago,

Cue 16 — 00:01:00,096 --> 00:01:02,556

which is like a decade in AI times.

Cue 17 — 00:01:03,296 --> 00:01:04,816

I talked about code reviews. Dead.

Cue 18 — 00:01:04,856 --> 00:01:07,496

This was on Latent Space. You can find this on QR code.

Cue 19 — 00:01:07,536 --> 00:01:09,016

I'm not going to go too much into details.

Cue 20 — 00:01:09,056 --> 00:01:13,976

I feel like this is already a little bit dated, but a lot of my learnings from

Cue 21 — 00:01:14,016 --> 00:01:15,796

there. This is written in March.

Cue 22 — 00:01:16,416 --> 00:01:19,996

A lot of my learnings from there will be something we will discuss here today.

Cue 23 — 00:01:21,496 --> 00:01:28,336

And yeah, essentially it talks about creating frameworks and layers to replace

Cue 24 — 00:01:28,376 --> 00:01:32,576

code reviews. So that's kind of like some of the baseline of some of the things we

Cue 25 — 00:01:32,616 --> 00:01:36,476

will talk about. And I will talk about things where I was wrong.

Cue 26 — 00:01:37,736 --> 00:01:40,756

So a little bit of like overview a year.

Cue 27 — 00:01:41,536 --> 00:01:45,816

We went 4x. So everyone's code volume is increasing.

Cue 28 — 00:01:45,856 --> 00:01:49,256

You know, like writing code is almost zero cost today.

Cue 29 — 00:01:49,296 --> 00:01:51,636

You can create PR, you can throw them away.

Cue 30 — 00:01:52,416 --> 00:01:55,976

We are about the same team we were a year ago.

Cue 31 — 00:01:56,016 --> 00:02:01,456

And we are producing 4x code. One of the things which has shifted and I'm guessing

Cue 32 — 00:02:01,496 --> 00:02:06,336

everyone already has seen that experienced last November, actually almost a year

Cue 33 — 00:02:06,376 --> 00:02:13,136

ago, where we moved from human assistant humans writing code and AI assisted to now

Cue 34 — 00:02:13,176 --> 00:02:16,436

predominantly AI writing code with human assistants.

Cue 35 — 00:02:16,656 --> 00:02:20,996

And that is kind of like part of the reason why the volumes have increased.

Cue 36 — 00:02:22,936 --> 00:02:25,236

So our battles that code reviews.

Cue 37 — 00:02:25,856 --> 00:02:31,896

So last December we were this is the first time we properly introduced an AI

Cue 38 — 00:02:31,936 --> 00:02:34,096

reviewer. This was our first attempt.

Cue 39 — 00:02:34,136 --> 00:02:38,216

And it's an example of one of the PR that I reviewed.

Cue 40 — 00:02:38,256 --> 00:02:43,496

So what was happening was we had AI review the PR and humans also after that doing

Cue 41 — 00:02:43,536 --> 00:02:49,976

the review. An interesting pattern we noticed is most of the comments from AI were

Cue 42 — 00:02:50,016 --> 00:02:54,436

mostly small nits. We didn't even care about it as much, but it did find one bug,

Cue 43 — 00:02:54,496 --> 00:02:57,856

which was great. You know, even finding one bug in your code is important.

Cue 44 — 00:02:57,896 --> 00:02:59,876

That's part of the process as well.

Cue 45 — 00:02:59,936 --> 00:03:06,016

And one interesting thing we did was humans review were predominantly decisions and

Cue 46 — 00:03:06,056 --> 00:03:09,356

opinions, which I find was actually very helpful.

Cue 47 — 00:03:09,816 --> 00:03:13,356

But again, this wasn't really like very effective.

Cue 48 — 00:03:13,816 --> 00:03:16,856

So the second attempt and this is probably our worst attempt,

Cue 49 — 00:03:16,896 --> 00:03:21,736

honestly we did was we told everyone, okay, you have four business hours.

Cue 50 — 00:03:21,776 --> 00:03:24,056

Every review has to be completed in four business hours.

Cue 51 — 00:03:24,096 --> 00:03:26,796

We wanted people to kind of like be faster at reviews.

Cue 52 — 00:03:27,496 --> 00:03:31,336

And the goal was like, first review, we don't expect you to approve PR in first four

Cue 53 — 00:03:31,376 --> 00:03:36,076

business hours. So that kind of like, you know, our velocity can increase.

Cue 54 — 00:03:36,416 --> 00:03:40,756

But obviously what we noticed is although the first reviews got faster,

Cue 55 — 00:03:41,536 --> 00:03:43,436

we started seeing a lot of rubber-stamping.

Cue 56 — 00:03:43,536 --> 00:03:47,416

Nobody's going to keep up on the reviews and specially for senior engineers.

Cue 57 — 00:03:47,456 --> 00:03:51,376

Now we are seeing more and more that they're like struggling to keep up on their own

Cue 58 — 00:03:51,416 --> 00:03:54,136

work at some point, like our one of our senior engineers was like,

Cue 59 — 00:03:54,176 --> 00:03:56,856

no, I'm not going to do reviews anymore, right?

Cue 60 — 00:03:56,896 --> 00:04:02,236

So we moved to faster reviews, but the quality of reviews itself went down.

Cue 61 — 00:04:03,856 --> 00:04:07,256

This was attempt three. This was back in March.

Cue 62 — 00:04:07,296 --> 00:04:12,496

So eight months ago. And this is where kind of like we started seeing signs of

Cue 63 — 00:04:12,536 --> 00:04:16,876

better quality. What we did was we created a self review skill.

Cue 64 — 00:04:17,256 --> 00:04:20,416

What that means is essentially it's not like off the shelf review.

Cue 65 — 00:04:20,456 --> 00:04:22,136

We essentially wrote our own skill.

Cue 66 — 00:04:22,176 --> 00:04:27,136

Every developer who would create a PR now has to do this self review before they

Cue 67 — 00:04:27,176 --> 00:04:33,256

submit it. And a lot of bugs. Effectively what ended up happening is we started

Cue 68 — 00:04:33,296 --> 00:04:34,676

catching them earlier.

Cue 69 — 00:04:37,496 --> 00:04:42,696

The challenge though was we are still spending that time reviewing the diffs after

Cue 70 — 00:04:42,736 --> 00:04:48,296

the fact. And the other thing is, as the skill found some issues,

Cue 71 — 00:04:48,336 --> 00:04:51,056

they were resolved, but they never got surfaced.

Cue 72 — 00:04:51,096 --> 00:04:54,376

Which means it's like any kind of findings we had got lost in,

Cue 73 — 00:04:54,416 --> 00:04:56,096

sort of like the self review skill as well.

Cue 74 — 00:04:56,136 --> 00:04:59,936

But this was still by far our best attempt from there.

Cue 75 — 00:04:59,976 --> 00:05:03,156

And this is kind of like also around the time when I wrote that article.

Cue 76 — 00:05:06,016 --> 00:05:10,216

So before we kind of like, you know, talk about framework, let's talk about like why

Cue 77 — 00:05:10,256 --> 00:05:14,376

do we even do reviews? This was a research done in 2013.

Cue 78 — 00:05:14,416 --> 00:05:21,296

It's been a while ago, but 44% of people who were asked said they'd do it to find

Cue 79 — 00:05:21,336 --> 00:05:25,456

defects. However, when we reviewed the comments, analyze the comments.

Cue 80 — 00:05:25,496 --> 00:05:27,676

Like this. Is the research done by Microsoft?

Cue 81 — 00:05:27,776 --> 00:05:34,216

Only 14% actually were defect related and 30% is this gap.

Cue 82 — 00:05:34,256 --> 00:05:36,336

The whole talk is about like this.

Cue 83 — 00:05:36,376 --> 00:05:39,586

This is sort of like the mindset where we believe that majority of the reviews,

Cue 84 — 00:05:39,626 --> 00:05:41,776

we're doing it actually for defect finding.

Cue 85 — 00:05:41,816 --> 00:05:43,876

But there is a different value.

Cue 86 — 00:05:44,296 --> 00:05:46,756

Code review is not about the code.

Cue 87 — 00:05:47,496 --> 00:05:50,076

Code review is not about finding bugs.

Cue 88 — 00:05:50,616 --> 00:05:53,036

Code review is to build your mental models.

Cue 89 — 00:05:53,216 --> 00:05:58,656

Like we work in teams, we are not solo developers who are building projects and kind

Cue 90 — 00:05:58,696 --> 00:06:00,456

of like doing everything by themselves.

Cue 91 — 00:06:00,496 --> 00:06:04,416

You need to educate people, you need to collaborate and you need to build mental

Cue 92 — 00:06:04,456 --> 00:06:08,216

models. Code review is about knowledge sharing.

Cue 93 — 00:06:08,256 --> 00:06:12,816

If we today stop doing the reviews, we would not know what our teammates are doing.

Cue 94 — 00:06:12,856 --> 00:06:14,776

We would not understand how the systems work.

Cue 95 — 00:06:14,816 --> 00:06:18,956

And that is the main value that we have for code reviews today.

Cue 96 — 00:06:21,496 --> 00:06:26,656

So here's the thing. Even today, human code reviews do find bugs,

Cue 97 — 00:06:26,696 --> 00:06:32,276

so they are valuable. However, we are not very good at it.

Cue 98 — 00:06:33,016 --> 00:06:40,096

Like as humans, this was also a research done back in 2006 where once your code

Cue 99 — 00:06:40,136 --> 00:06:45,136

lines go beyond 400 and you're spending over an hour doing a review,

Cue 100 — 00:06:45,176 --> 00:06:50,216

the quality of finding defects drops significantly and your AI changes are already

Cue 101 — 00:06:50,256 --> 00:06:53,176

kind of like, you know, your median is already beyond that.

Cue 102 — 00:06:53,216 --> 00:06:57,576

So that means the way we are doing today is not sustainable.

Cue 103 — 00:06:57,616 --> 00:07:01,276

It's not something which is actually even providing high quality.

Cue 104 — 00:07:02,936 --> 00:07:06,556

And the reason is diff was always a bad place to review.

Cue 105 — 00:07:06,736 --> 00:07:12,216

The diff was always a place to do review because you are one reviewer,

Cue 106 — 00:07:12,256 --> 00:07:14,196

you're just reading from memory

Cue 107 — 00:07:15,456 --> 00:07:18,496

and the diff is showing you what change.

Cue 108 — 00:07:18,536 --> 00:07:23,496

It doesn't tell you why. So think about it like we talk about AI being not

Cue 109 — 00:07:23,536 --> 00:07:25,616

deterministic. Humans are two.

Cue 110 — 00:07:25,656 --> 00:07:30,096

If you actually do a review in two different times, very likely you will find two

Cue 111 — 00:07:30,136 --> 00:07:34,056

different things. And that happens because we build these systems which are

Cue 112 — 00:07:34,096 --> 00:07:38,316

completely non-deterministic. And as humans we are also a reflection of that too.

Cue 113 — 00:07:39,656 --> 00:07:42,216

So here's sort of like a review, which I did.

Cue 114 — 00:07:42,256 --> 00:07:44,856

And you can see kind of like I left a bunch of comments.

Cue 115 — 00:07:44,896 --> 00:07:46,916

There's a bunch of comments from bot as well.

Cue 116 — 00:07:48,736 --> 00:07:51,216

One thing I would like you to see is sort of like, you know,

Cue 117 — 00:07:51,256 --> 00:07:52,936

a lot of bot comments are still noise.

Cue 118 — 00:07:52,976 --> 00:07:55,936

And I do agree that AI is still finds bugs.

Cue 119 — 00:07:55,976 --> 00:08:02,256

So I'm not asking you to take away that, but a lot of reviews that we do can be

Cue 120 — 00:08:02,296 --> 00:08:07,856

separated into buckets. So as you can see, some of the review feedback we are giving

Cue 121 — 00:08:07,896 --> 00:08:12,336

is about code quality, and the others we are doing are sort of like opinions and

Cue 122 — 00:08:12,376 --> 00:08:16,756

decisions. And once you start understanding that difference,

Cue 123 — 00:08:16,936 --> 00:08:20,196

it makes you reimagine what reviews could be.

Cue 124 — 00:08:20,256 --> 00:08:24,496

So in that particular example we had eight threads from bot.

Cue 125 — 00:08:24,536 --> 00:08:26,636

Only one of them was a brutal bug.

Cue 126 — 00:08:26,816 --> 00:08:31,596

Three were rules, which was about code quality and two were actual decisions.

Cue 127 — 00:08:32,376 --> 00:08:36,456

So when we look at reviews, it's important you actually see the difference between

Cue 128 — 00:08:36,496 --> 00:08:38,996

what is a rule and what's an opinion.

Cue 129 — 00:08:39,176 --> 00:08:44,256

And we need both of them. But the mechanism to detect them, the mechanism to review

Cue 130 — 00:08:44,296 --> 00:08:49,156

them can be different. Which brings us to our framework.

Cue 131 — 00:08:50,776 --> 00:08:55,296

So this is the new shape of review that I propose.

Cue 132 — 00:08:55,336 --> 00:08:59,816

And yes, we had one step of review and I'm suggesting you six.

Cue 133 — 00:08:59,856 --> 00:09:03,456

But hear me out and we will kind of like go into more details here.

Cue 134 — 00:09:03,496 --> 00:09:05,216

But let me kind of like just give you an overview.

Cue 135 — 00:09:05,256 --> 00:09:10,956

We talk about argue, which is about actually having agents argue with each other,

Cue 136 — 00:09:11,216 --> 00:09:16,456

capture where we capture some of the decisions and some of the opinions of the

Cue 137 — 00:09:16,496 --> 00:09:22,596

agents and humans codify where we try to codify all of our review,

Cue 138 — 00:09:22,816 --> 00:09:26,796

structural review, the semantic review capabilities.

Cue 139 — 00:09:27,096 --> 00:09:32,016

We talk about proof, which is like, how do we actually verify and prove that these

Cue 140 — 00:09:32,056 --> 00:09:38,736

rules are addressed? Debate. And this is one of the most important things.

Cue 141 — 00:09:38,776 --> 00:09:42,976

This was actually something I missed entirely in my article that I wrote in March,

Cue 142 — 00:09:43,016 --> 00:09:44,816

which is about the role of humans.

Cue 143 — 00:09:44,856 --> 00:09:47,856

And the review is to debate, to express opinions.

Cue 144 — 00:09:47,896 --> 00:09:50,616

It's not about identifying bugs and issues.

Cue 145 — 00:09:50,656 --> 00:09:55,416

It's about aligning on the mental models, making sure we agree on the system

Cue 146 — 00:09:55,456 --> 00:10:00,456

architectures, making sure we actually are able to collaborate with others and own

Cue 147 — 00:10:00,496 --> 00:10:03,776

is the ownership. Because, you know, I cannot own things.

Cue 148 — 00:10:03,816 --> 00:10:06,116

They're not responsible effectively.

Cue 149 — 00:10:06,496 --> 00:10:08,076

So let's just start with argue.

Cue 150 — 00:10:08,576 --> 00:10:13,216

So we talk about argue as let the agents argue before the PR is created.

Cue 151 — 00:10:13,256 --> 00:10:18,296

Now I argue that as AI reviews, even though AI reviews are important,

Cue 152 — 00:10:18,336 --> 00:10:21,316

you don't necessarily need to do that on a PR.

Cue 153 — 00:10:21,416 --> 00:10:26,136

You can actually identify and let let let run a local agent,

Cue 154 — 00:10:26,176 --> 00:10:29,556

which actually does the review, because all you're trying to do is identify

Cue 155 — 00:10:29,616 --> 00:10:31,636

inconsistencies in the system.

Cue 156 — 00:10:32,496 --> 00:10:34,856

So this is an example of what we did.

Cue 157 — 00:10:34,896 --> 00:10:37,896

Attempt three, as I explained, which was a review skill.

Cue 158 — 00:10:37,936 --> 00:10:39,136

You can write your own review skill.

Cue 159 — 00:10:39,176 --> 00:10:43,536

I'm not going to share sort of like our own versions because everyone has their own

Cue 160 — 00:10:43,576 --> 00:10:47,456

sort of different systems. They have like own ways of defining what the review looks

Cue 161 — 00:10:47,496 --> 00:10:51,096

like, but you make it argue with the coding agent.

Cue 162 — 00:10:51,136 --> 00:10:56,976

So what you can do here is having two different goals for two different agents and

Cue 163 — 00:10:57,016 --> 00:11:01,256

let them debate. What ends up happening is effectively you're trying to come up with

Cue 164 — 00:11:01,296 --> 00:11:06,336

consensus and anything which remains open or the decisions that humans should make.

Cue 165 — 00:11:06,376 --> 00:11:11,296

So you're now reducing the amount of toil humans have to do to only look at the

Cue 166 — 00:11:11,336 --> 00:11:13,236

inconsistencies with the agents

Cue 167 — 00:11:14,576 --> 00:11:18,316

every day. So every disagreement, then we write it down.

Cue 168 — 00:11:20,856 --> 00:11:25,216

So as humans, as I said, now you, instead of reviewing the diff,

Cue 169 — 00:11:25,256 --> 00:11:27,316

you read what they disagree about.

Cue 170 — 00:11:27,696 --> 00:11:31,576

So in this particular example we have a few tags as you can see.

Cue 171 — 00:11:31,616 --> 00:11:35,496

So agents sort of like did agree and disagree on certain things.

Cue 172 — 00:11:35,536 --> 00:11:40,056

Some things were decided. Some things agents said like okay this is an acceptable

Cue 173 — 00:11:40,096 --> 00:11:45,656

risk. And then certain things are follow ups which now become easier for you to

Cue 174 — 00:11:45,696 --> 00:11:50,576

track. And certain things are human calls which now are surfaced for humans to make

Cue 175 — 00:11:50,616 --> 00:11:55,216

final decisions on. So all you're doing is essentially creating a better framework

Cue 176 — 00:11:55,256 --> 00:12:00,176

for agents and humans to work together, where both of them are using their strengths

Cue 177 — 00:12:00,216 --> 00:12:05,196

versus using kind of like creating more toil for humans.

Cue 178 — 00:12:06,416 --> 00:12:13,096

Capture. So capture. We talk about what do we capture from a coding

Cue 179 — 00:12:13,136 --> 00:12:14,836

session or like from a.

Cue 180 — 00:12:16,776 --> 00:12:22,796

From a code change. So we talk about capturing the why and the how.

Cue 181 — 00:12:23,176 --> 00:12:27,536

So what does that mean? A typical answer today is spec, and I know some of my

Cue 182 — 00:12:27,576 --> 00:12:29,656

friends here are also going to talk about specifications.

Cue 183 — 00:12:29,696 --> 00:12:33,416

So I'm not going to, you know, make too much fun of specs.

Cue 184 — 00:12:33,456 --> 00:12:36,116

But there is an inherent challenge with specs.

Cue 185 — 00:12:36,576 --> 00:12:41,476

Specs can work, but writing good specs require time.

Cue 186 — 00:12:42,696 --> 00:12:47,296

And the biggest challenge for specs for me is that there is no feedback loop.

Cue 187 — 00:12:47,336 --> 00:12:51,536

Once you write spec and you start implementing, you might find some things which you

Cue 188 — 00:12:51,576 --> 00:12:53,376

completely forgot about in the spec.

Cue 189 — 00:12:53,416 --> 00:12:56,416

Do you actually go up and update your specs back?

Cue 190 — 00:12:56,456 --> 00:12:58,936

If you do, that's great. You can continue doing it.

Cue 191 — 00:12:58,976 --> 00:13:02,456

But in reality, most people who write specs, essentially your specs and your

Cue 192 — 00:13:02,496 --> 00:13:05,516

implementation starts differing the moment you create a change.

Cue 193 — 00:13:06,656 --> 00:13:10,356

So these drifts make it really hard for you to manage specs.

Cue 194 — 00:13:10,576 --> 00:13:14,636

In fact, this is like the waterfall model that we moved away from 20 years ago.

Cue 206 — 00:13:57,259 --> 00:13:59,839

So what can really replace specs.

Cue 207 — 00:13:59,939 --> 00:14:05,059

We talk about intent. And this is sort of like what one of our biggest learning was,

Cue 208 — 00:14:05,099 --> 00:14:07,959

is when we think about why a change is happening.

Cue 209 — 00:14:08,019 --> 00:14:10,479

Intent is a great way to think about it.

Cue 210 — 00:14:11,659 --> 00:14:14,459

We also now talk about the acceptance criteria.

Cue 211 — 00:14:14,499 --> 00:14:19,539

Many of you may have already seen this in the past, where Gherkin was a very popular

Cue 212 — 00:14:19,579 --> 00:14:24,539

framework to define the acceptance criteria using BDD behavioral driven development.

Cue 213 — 00:14:24,579 --> 00:14:30,699

And because of AI, this framework now becomes a lot easier for you to adopt because

Cue 214 — 00:14:30,739 --> 00:14:34,819

acceptance criteria can be defined in English, but at the same time it can then be

Cue 215 — 00:14:34,859 --> 00:14:36,359

validated with agents.

Cue 216 — 00:14:37,659 --> 00:14:42,379

And the beauty of this is this works with one line bugfix as well as a thousand line

Cue 217 — 00:14:42,419 --> 00:14:47,579

feature. So a specification may feel overwhelming when you're making a two line

Cue 218 — 00:14:47,619 --> 00:14:53,359

bugfix, but with this framework, you can actually do that even for a one line fix.

Cue 219 — 00:14:54,339 --> 00:14:57,979

And where do they live? They actually already live in your sessions.

Cue 220 — 00:14:58,019 --> 00:15:02,539

So if you're using any of your coding sessions, intent is what you start your

Cue 221 — 00:15:02,579 --> 00:15:07,099

session with. An acceptance criteria is every time an agent stops to ask your

Cue 222 — 00:15:07,139 --> 00:15:11,919

questions, every time there are opinions and decisions made on the behavior,

Cue 223 — 00:15:12,099 --> 00:15:14,279

those are your acceptance criteria.

Cue 224 — 00:15:14,539 --> 00:15:18,659

But today we throw them away as soon as the session ends.

Cue 225 — 00:15:18,699 --> 00:15:20,459

That information always disappears.

Cue 226 — 00:15:20,499 --> 00:15:23,179

You capture some of those things in a pull request description,

Cue 227 — 00:15:23,219 --> 00:15:27,139

but they're not structured. They might have misinformation and there is no way to

Cue 228 — 00:15:27,179 --> 00:15:28,599

actually validate it.

Cue 229 — 00:15:30,859 --> 00:15:37,739

So codify. This is where your structures, your code patterns become very

Cue 230 — 00:15:37,779 --> 00:15:44,239

important. And we talk about how we can turn your repeated comments into rules.

Cue 231 — 00:15:44,859 --> 00:15:50,099

So this is what we did. We mined one year of our comments which we are already

Cue 232 — 00:15:50,139 --> 00:15:51,999

living in our GitHub pull requests.

Cue 233 — 00:15:52,979 --> 00:15:57,699

First we did a reduce, which is essentially we took those comments and removed some

Cue 234 — 00:15:57,739 --> 00:16:03,699

of the noise coming from bots we extracted, which is essentially we classified every

Cue 235 — 00:16:03,739 --> 00:16:05,499

comment. As I said, some of them are decisions.

Cue 236 — 00:16:05,539 --> 00:16:10,159

Some of them can be rules, some of them are just nits we don't care about.

Cue 237 — 00:16:10,459 --> 00:16:15,059

And finally, we distilled where we are identifying common patterns of comments

Cue 238 — 00:16:15,099 --> 00:16:17,919

you're making and converting them to rules.

Cue 239 — 00:16:18,139 --> 00:16:21,439

This essentially creates what we call invariants.

Cue 240 — 00:16:22,179 --> 00:16:26,139

Out of 11,000 comments that we reviewed.

Cue 241 — 00:16:26,179 --> 00:16:32,039

We got 212 candidates and effectively 64 rules out of them.

Cue 242 — 00:16:32,539 --> 00:16:35,499

And these are examples of some of our invariants.

Cue 243 — 00:16:35,539 --> 00:16:38,699

As you can see, some of them are pretty structured, which is like GraphQL schema

Cue 244 — 00:16:38,739 --> 00:16:43,339

patterns, but some of them are also sort of like okay, back end code reuse as an

Cue 245 — 00:16:43,379 --> 00:16:45,659

existing implementation, you don't have front end code.

Cue 246 — 00:16:45,699 --> 00:16:48,379

Be intelligent and repeating the patterns.

Cue 247 — 00:16:48,419 --> 00:16:53,699

Now if you see some of them can actually be your linters, which already many of you

Cue 248 — 00:16:53,739 --> 00:16:55,779

are probably building many of these linters.

Cue 249 — 00:16:55,819 --> 00:17:00,179

But with agents, the beauty of this is you can actually build rules which may not

Cue 250 — 00:17:00,219 --> 00:17:02,299

necessarily be covered with linters.

Cue 251 — 00:17:02,339 --> 00:17:07,779

And this is an invariant, as an example, how what we actually store in the system

Cue 252 — 00:17:07,819 --> 00:17:11,299

first is a rule itself, which is every config change writes.

Cue 253 — 00:17:11,339 --> 00:17:16,899

So kind of like explaining what the rule is scope which is which particular changes

Cue 254 — 00:17:16,939 --> 00:17:20,059

applies. This rule you don't want to validate every rule in every change.

Cue 255 — 00:17:20,099 --> 00:17:23,959

If you're making a front end change, it doesn't need to be part of the back end.

Cue 256 — 00:17:25,339 --> 00:17:30,079

We talk about verified by, which is a process on how this can be verified.

Cue 257 — 00:17:30,139 --> 00:17:34,459

Many times you can actually spin up a service and have agents test it out.

Cue 258 — 00:17:34,499 --> 00:17:38,359

That could be verification. Sometimes it could just be reading the code itself.

Cue 259 — 00:17:40,059 --> 00:17:43,219

Provenance is a trail of how this was generated.

Cue 260 — 00:17:43,259 --> 00:17:47,599

This becomes important from audit purposes, like why we even created this rule.

Cue 261 — 00:17:47,739 --> 00:17:51,399

And finally, and one of the most important one is ownership.

Cue 262 — 00:17:51,459 --> 00:17:55,099

If you are building these rules, this is not something you build once and forget

Cue 263 — 00:17:55,139 --> 00:17:58,259

about it. Your code evolves. So do your rules.

Cue 264 — 00:17:58,299 --> 00:18:02,839

So it's important that these rules are owned by the right people and the teams.

Cue 265 — 00:18:03,979 --> 00:18:08,199

And the best thing of these rules is they get cheaper over time.

Cue 266 — 00:18:09,259 --> 00:18:13,619

So we start with this invariant, which is essentially a rule created by a model by

Cue 267 — 00:18:13,659 --> 00:18:15,439

analyzing your comments.

Cue 268 — 00:18:17,739 --> 00:18:23,330

What we do with that is essentially have your agents essentially convert that into

Cue 269 — 00:18:23,370 --> 00:18:26,979

ast-grep patterns. If you're not familiar with ast-grep, think of this as sort of

Cue 270 — 00:18:27,019 --> 00:18:30,259

like smart linters. They are completely deterministic.

Cue 271 — 00:18:30,299 --> 00:18:34,459

But essentially what we are doing is using LLMs to create more deterministic

Cue 272 — 00:18:34,499 --> 00:18:40,219

patterns. The advantage of this is now the error rates are like the variation rates

Cue 273 — 00:18:40,259 --> 00:18:42,279

become zero.

Cue 274 — 00:18:43,619 --> 00:18:47,309

We run this in shadow mode, which essentially means is we do a validation once with

Cue 275 — 00:18:47,349 --> 00:18:50,399

the model and once with the rule, and see how the results are.

Cue 276 — 00:18:50,539 --> 00:18:52,859

And effectively after that we promote it.

Cue 277 — 00:18:52,899 --> 00:18:55,699

Where it means is we are only validating it with the models,

Cue 278 — 00:18:55,739 --> 00:18:57,979

sorry, only validating it with the rules.

Cue 279 — 00:18:58,019 --> 00:19:01,859

Deterministic rules. So here's an example of one of the rules we had,

Cue 280 — 00:19:01,899 --> 00:19:04,379

which is every request needs a timeout.

Cue 281 — 00:19:04,419 --> 00:19:07,299

And this is entirely generated by an LLM.

Cue 282 — 00:19:07,339 --> 00:19:09,219

We did not write it ourselves.

Cue 283 — 00:19:09,259 --> 00:19:14,279

And the beauty of this is now this can be validated completely deterministically.

Cue 284 — 00:19:15,259 --> 00:19:20,659

So out of the 64 rules that we created, 19 of them are now deterministic.

Cue 285 — 00:19:20,699 --> 00:19:24,699

They run in under 0.3 seconds.

Cue 286 — 00:19:24,739 --> 00:19:29,019

And we get exactly the same rule of verdict every time.

Cue 287 — 00:19:29,059 --> 00:19:32,119

Which means now we build deterministic rules using AI.

Cue 288 — 00:19:33,939 --> 00:19:39,019

So remember every comment your team repeats is a guardrail that you haven't written

Cue 289 — 00:19:39,059 --> 00:19:44,239

yet. Every comment your team repeats is a guardrail you haven't written yet.

Cue 290 — 00:19:45,339 --> 00:19:49,939

Prove. How do we actually validate everything that we have written so far?

Cue 291 — 00:19:49,979 --> 00:19:51,799

All these different mechanics.

Cue 292 — 00:19:53,859 --> 00:19:56,359

So how a criteria gets proven.

Cue 293 — 00:19:57,299 --> 00:19:59,779

The first thing we do is create a plan.

Cue 294 — 00:19:59,819 --> 00:20:02,159

Each criteria becomes a scenario.

Cue 295 — 00:20:02,819 --> 00:20:08,139

We spin up a preview environment, which is essentially similar to how you're running

Cue 296 — 00:20:08,179 --> 00:20:11,019

doing local testing. In dev environments, you spin up a server,

Cue 297 — 00:20:11,059 --> 00:20:15,059

and using that plan essentially let the agents drive it.

Cue 298 — 00:20:15,099 --> 00:20:18,499

So in case of a front end, you can actually have your agents capture the

Cue 299 — 00:20:18,539 --> 00:20:25,219

screenshots, capture all of this evidence and as as like it clicks through the UI.

Cue 300 — 00:20:25,259 --> 00:20:28,219

So you might already familiar if you've already used Playwright-type systems,

Cue 301 — 00:20:28,259 --> 00:20:30,339

it's doing the same thing, but it's doing with the agents.

Cue 302 — 00:20:30,379 --> 00:20:35,099

The beauty of that is now you can have these tests be generated in real time versus

Cue 303 — 00:20:35,139 --> 00:20:40,059

having something pre-planned. And all of this now becomes the evidence.

Cue 304 — 00:20:40,099 --> 00:20:43,179

So the in case of front end, you have screenshots.

Cue 305 — 00:20:43,219 --> 00:20:46,879

In case of backends you may have like API responses, you may have,

Cue 306 — 00:20:47,619 --> 00:20:52,019

you know, database snapshots, which is essentially creating all of these rules to or

Cue 307 — 00:20:52,059 --> 00:20:55,619

like evidence, which becomes your proof that the code works.

Cue 308 — 00:20:55,659 --> 00:21:00,379

So this is not just about a green check from an AI review.

Cue 309 — 00:21:00,419 --> 00:21:04,559

It's about actually how do you validate the change really works.

Cue 310 — 00:21:06,139 --> 00:21:09,379

So here's another example. We have a rule for config change.

Cue 311 — 00:21:09,419 --> 00:21:13,939

What we are doing is taking a screenshot before making a config change in our code

Cue 312 — 00:21:13,979 --> 00:21:16,859

base, and seeing whether an audit trail gets created.

Cue 313 — 00:21:16,899 --> 00:21:23,339

So essentially the entire test is now run by AI, and all we do as humans is look at

Cue 314 — 00:21:23,379 --> 00:21:26,299

the evidence, whether this is actually getting satisfied.

Cue 315 — 00:21:26,339 --> 00:21:29,619

And this is a rule. Now we apply for every config change we make,

Cue 316 — 00:21:29,659 --> 00:21:32,219

which means this. Now we don't have to ever worry about it.

Cue 317 — 00:21:32,259 --> 00:21:36,119

As your code evolves, these rules continue capturing more space.

Cue 318 — 00:21:37,979 --> 00:21:41,379

And different criteria requires different type of proof.

Cue 319 — 00:21:41,419 --> 00:21:44,339

So as I mentioned, kind of like things that users see.

Cue 320 — 00:21:44,379 --> 00:21:46,139

You make them a screenshot. Done.

Cue 321 — 00:21:46,179 --> 00:21:48,259

DOM snapshots all of those things.

Cue 322 — 00:21:48,299 --> 00:21:51,239

If it's an API contract, you look at the responses.

Cue 323 — 00:21:51,499 --> 00:21:55,219

If there are side effects, then you kind of look at the network request.

Cue 324 — 00:21:55,259 --> 00:22:00,619

A code becomes an ast-grep as I mentioned, and behavior over time can be sort of

Cue 325 — 00:22:00,659 --> 00:22:01,839

like a test run.

Cue 326 — 00:22:05,419 --> 00:22:09,499

And debate. This is the part that I missed entirely in the essay,

Cue 327 — 00:22:09,539 --> 00:22:14,499

and this is the part which I find is still the most important when we think about

Cue 328 — 00:22:14,539 --> 00:22:18,419

reviews, when we think about the role of humans in the software development

Cue 329 — 00:22:18,459 --> 00:22:19,319

lifecycle.

Cue 330 — 00:22:20,699 --> 00:22:25,419

So we talk about debating decisions, but where do decisions even come from?

Cue 331 — 00:22:25,459 --> 00:22:28,039

I've already kind of like given some of those examples.

Cue 332 — 00:22:28,739 --> 00:22:32,399

A decision comes from while you build.

Cue 333 — 00:22:32,539 --> 00:22:37,739

So when I say you build, yes, code is written by agents, but you as human is

Cue 334 — 00:22:37,779 --> 00:22:42,219

involved in participating in that conversation when you're doing it.

Cue 335 — 00:22:42,259 --> 00:22:44,119

Interactive sessions with the agent.

Cue 336 — 00:22:44,179 --> 00:22:48,779

So whenever agent make choices with you, it's making those decisions.

Cue 337 — 00:22:48,819 --> 00:22:53,339

So you can write a very small skill which actually tells your agent to export all of

Cue 338 — 00:22:53,379 --> 00:22:57,759

those decisions as you write, as you kind of like, publish a change.

Cue 339 — 00:22:59,659 --> 00:23:03,179

Similarly, as your coding agent argues with a review agent now,

Cue 340 — 00:23:03,219 --> 00:23:07,299

today you leave some comments in GitHub, but you can also capture those things.

Cue 341 — 00:23:07,339 --> 00:23:10,219

And any time an agent is making a change based on a comment,

Cue 342 — 00:23:10,259 --> 00:23:13,939

there are likely some decisions being made and it's important that you actually

Cue 343 — 00:23:13,979 --> 00:23:18,059

capture those decisions, because this is something that would require and make it

Cue 344 — 00:23:18,099 --> 00:23:22,599

easy for humans to understand how your code is changing.

Cue 345 — 00:23:23,819 --> 00:23:29,459

And similarly, once your verification fails, agent is again making certain changes.

Cue 346 — 00:23:29,499 --> 00:23:31,699

All of those things also qualifies decisions.

Cue 347 — 00:23:31,739 --> 00:23:36,019

Agent may also change its acceptance criteria, which is also important to you for

Cue 348 — 00:23:36,059 --> 00:23:37,799

you to capture as decisions.

Cue 349 — 00:23:40,179 --> 00:23:45,739

And every decision on the PR. So this is an example of some of the decisions and

Cue 350 — 00:23:45,779 --> 00:23:49,699

also outcomes out of it. As you can see, some of those things is decided.

Cue 351 — 00:23:49,739 --> 00:23:53,459

Some of them are fixed, which is like there was a bug or maybe a test failure which

Cue 352 — 00:23:53,499 --> 00:23:58,619

got fixed. And some of them requires humans to participate as well,

Cue 353 — 00:23:58,659 --> 00:24:01,659

as there are certain things which an agent will say, okay, let's create a separate

Cue 354 — 00:24:01,699 --> 00:24:05,979

PR for this. What you're doing here is creating not just an audit trail,

Cue 355 — 00:24:06,019 --> 00:24:10,039

but also a breadcrumbs of how your things are evolving.

Cue 356 — 00:24:10,539 --> 00:24:16,299

And the beauty of all of this is now, what do you do with them?

Cue 357 — 00:24:16,339 --> 00:24:20,719

Like, you can actually skim what was settled.

Cue 358 — 00:24:21,179 --> 00:24:26,819

You can settle when an agent cannot, which means like it requires human decision

Cue 359 — 00:24:26,859 --> 00:24:31,959

making. You can reopen any decision that you disagree with.

Cue 360 — 00:24:32,259 --> 00:24:36,299

And I think the most important at this point is this is your review surface.

Cue 361 — 00:24:36,339 --> 00:24:38,259

You're not reviewing the code.

Cue 362 — 00:24:38,299 --> 00:24:42,499

You're actually reviewing these decisions and you're having discussions with the

Cue 363 — 00:24:42,539 --> 00:24:45,479

author. This is how you're building your mental models.

Cue 364 — 00:24:46,939 --> 00:24:50,659

And the beauty of all of this is decisions outlive a pull request.

Cue 365 — 00:24:50,699 --> 00:24:53,979

A pull request is more rigid. Nobody goes and looks back to the comments.

Cue 366 — 00:24:54,019 --> 00:24:57,299

But these decisions are something which are now accessible.

Cue 367 — 00:24:57,339 --> 00:25:02,299

You can ask your agents to go read your decisions from last one week and identify

Cue 368 — 00:25:02,339 --> 00:25:08,059

inconsistencies. You can analyze your weekly results, your decisions.

Cue 369 — 00:25:08,099 --> 00:25:09,539

You can build your agent memory.

Cue 370 — 00:25:09,579 --> 00:25:13,539

Your agents now have access to all of these decisions which have been made across

Cue 371 — 00:25:13,579 --> 00:25:17,159

your entire codebase, not just the ones that your agent made.

Cue 372 — 00:25:17,939 --> 00:25:21,099

You can track follow ups. So any decisions which said like, okay,

Cue 373 — 00:25:21,139 --> 00:25:22,579

we'll follow up in another PR.

Cue 374 — 00:25:22,619 --> 00:25:24,219

Now you have a good traceability.

Cue 375 — 00:25:24,259 --> 00:25:27,139

You have easy ways to track all of those follow ups.

Cue 376 — 00:25:27,179 --> 00:25:28,729

You can connect it with your Jira.

Cue 377 — 00:25:28,769 --> 00:25:31,919

Now your follow-ups and Jira tickets are also traceable.

Cue 378 — 00:25:32,899 --> 00:25:37,459

And some of these decisions, repeated decisions now become your rules.

Cue 379 — 00:25:37,499 --> 00:25:40,319

So your rules library also keeps on evolving.

Cue 380 — 00:25:42,299 --> 00:25:45,559

Which comes to the last step which is ownership.

Cue 381 — 00:25:46,019 --> 00:25:50,299

You know models are great at doing things, but they are terrible when it comes to

Cue 382 — 00:25:50,339 --> 00:25:53,799

accountability. So somebody needs to own them.

Cue 383 — 00:25:54,139 --> 00:25:59,219

Responsibility has moved. We used to say the author's own the responsibility of code

Cue 384 — 00:25:59,259 --> 00:26:04,239

change. Sometimes we say that reviewers also own partial responsibility,

Cue 385 — 00:26:04,619 --> 00:26:10,819

but the reality today is, as teams, we should be owning the rules that govern the

Cue 386 — 00:26:10,859 --> 00:26:16,059

change. If we own these rules that get verified, we build stricter controls.

Cue 387 — 00:26:16,099 --> 00:26:21,739

We build stricter guardrails, which provides us at higher confidence on how to

Cue 388 — 00:26:21,779 --> 00:26:23,119

verify the changes.

Cue 389 — 00:26:24,539 --> 00:26:30,099

So responsibility today is not line by line review, but owning the rules that check

Cue 390 — 00:26:30,139 --> 00:26:34,719

the code and maintaining the understanding of the systems.

Cue 391 — 00:26:36,179 --> 00:26:38,539

So a year later. Quick numbers.

Cue 392 — 00:26:38,579 --> 00:26:40,979

Today we are merging four times the changes.

Cue 393 — 00:26:41,019 --> 00:26:44,979

And the median change takes about 3.5 hours for us.

Cue 394 — 00:26:45,019 --> 00:26:47,979

And the amount of human reviews are actually very low.

Cue 395 — 00:26:48,019 --> 00:26:52,139

Most of the reviews come from our verification systems, the number of reverts and

Cue 396 — 00:26:52,179 --> 00:26:54,799

the stability. And this is the most important things.

Cue 397 — 00:26:54,899 --> 00:26:59,039

Our stability has went up. Our number of reverts have gone down by 40%.

Cue 398 — 00:27:00,899 --> 00:27:04,919

I'll skip over this. But yeah, same size a year apart.

Cue 399 — 00:27:05,179 --> 00:27:06,619

I will share most of those things.

Cue 400 — 00:27:06,659 --> 00:27:12,099

But the most important thing is you are reading the intent, the decisions and the

Cue 401 — 00:27:12,139 --> 00:27:15,959

criteria, the rules. You're not reading the diffs.

Cue 402 — 00:27:17,779 --> 00:27:23,959

So where our time goes today is refining the rules, debugging decisions,

Cue 403 — 00:27:24,179 --> 00:27:29,339

debating decisions, sorry, and improving the verification system so we get stronger

Cue 404 — 00:27:29,379 --> 00:27:33,179

and stronger proof. And this is how we build better systems for the future.

Cue 405 — 00:27:33,219 --> 00:27:35,719

This is how we scale our AI systems.

Cue 406 — 00:27:37,219 --> 00:27:39,939

First 90 days is hard. You know, I'm not going to lie.

Cue 407 — 00:27:39,979 --> 00:27:42,919

First month we were doing double the work.

Cue 408 — 00:27:43,579 --> 00:27:46,427

Later on, we kind of like started seeing value.

Cue 409 — 00:27:46,467 --> 00:27:48,419

Invariants actually started catching things.

Cue 410 — 00:27:48,459 --> 00:27:53,799

And by end of month three we started completely stopped reading the diffs.

Cue 411 — 00:27:55,059 --> 00:27:58,639

So homework for everyone start tomorrow.

Cue 412 — 00:27:58,739 --> 00:28:01,219

Mine your last 1000 review comments.

Cue 413 — 00:28:01,259 --> 00:28:05,259

Take that top 20 that repeat and write them as checks.

Cue 414 — 00:28:05,299 --> 00:28:06,979

You can also put that in your CI.

Cue 415 — 00:28:07,019 --> 00:28:11,579

You don't have to use any of our products, but start creating these rules today.

Cue 416 — 00:28:11,619 --> 00:28:16,339

Stop reading the diffs. Read the intent, read the acceptance criteria,

Cue 417 — 00:28:16,379 --> 00:28:20,279

read the decisions and will for tomorrow.

Cue 418 — 00:28:21,219 --> 00:28:25,139

Thank you everyone and check out Aviator Verify.

Cue 419 — 00:28:25,179 --> 00:28:29,459

This is what we are building. I don't know if we have time for question,

Cue 420 — 00:28:29,499 --> 00:28:32,339

otherwise. Time for one question.

Cue 421 — 00:28:32,379 --> 00:28:36,659

One question. No. Better be good and I'll be here if you want to ask more questions.

Cue 422 — 00:28:36,699 --> 00:28:37,679

But yes.

Cue 423 — 00:28:40,579 --> 00:28:46,179

So there is this tribal knowledge that everybody knows that it's never written in

Cue 424 — 00:28:46,219 --> 00:28:49,099

review, but everybody knows. Have you thought of how to pull that out?

Cue 425 — 00:28:49,139 --> 00:28:50,339

So I love everything you're doing.

Cue 426 — 00:28:50,379 --> 00:28:52,699

But there's another part of this with yes.

Cue 427 — 00:28:52,739 --> 00:28:56,459

So that's a great question. This is why we mine these comments as well.

Cue 428 — 00:28:56,499 --> 00:29:01,359

Because a lot of tribal knowledge today gets shared during code review comments.

Cue 429 — 00:29:01,819 --> 00:29:06,339

Right. So our goal is kind of like using those code review comments as an initial

Cue 430 — 00:29:06,379 --> 00:29:09,899

corpus to build some of those rules which actually has these knowledge.

Cue 431 — 00:29:09,939 --> 00:29:13,499

But these more are the corner ones, the ones that are like, everybody knows we don't

Cue 432 — 00:29:13,539 --> 00:29:16,859

do this around here. And the problem is the agent is not everybody,

Cue 433 — 00:29:16,899 --> 00:29:22,099

right. So that corner case is where it just you'll be the dumbest person the planet

Cue 434 — 00:29:22,139 --> 00:29:24,669

to do this. And everybody, everybody comments on it.

Cue 435 — 00:29:24,709 --> 00:29:27,619

And that's part of especially on big projects like ten years,

Cue 436 — 00:29:27,659 --> 00:29:31,659

10 million lines of code. We I find this as agents are really suffering on that.

Cue 437 — 00:29:31,699 --> 00:29:35,619

No. 100%. And I think this is why we also believe that you have to have humans in

Cue 438 — 00:29:35,659 --> 00:29:40,379

the loop. Right. So big. Our one of our biggest perspective here is like,

Cue 439 — 00:29:40,419 --> 00:29:43,499

how do we actually create the right responsibility for the humans?

Cue 440 — 00:29:43,539 --> 00:29:47,939

And being able to, for instance, review those decisions also helps you with that

Cue 441 — 00:29:47,979 --> 00:29:50,819

because now you're not like spending time reading every diff,

Cue 442 — 00:29:50,859 --> 00:29:53,899

because a lot of code can be just boilerplate code.

Cue 443 — 00:29:53,939 --> 00:29:57,459

What you're reading is actually some of those code decisions and also helps you kind

Cue 444 — 00:29:57,499 --> 00:30:02,499

of like have those opinions. So yes, I mean, in some ways these are frameworks and

Cue 445 — 00:30:02,539 --> 00:30:07,779

capabilities we are building to, you know, capture some of those tribal knowledge.

Cue 446 — 00:30:07,819 --> 00:30:09,959

This is not something which happens overnight.

Cue 447 — 00:30:11,139 --> 00:30:12,979

Cool. All right. Thank you everyone.

Cue 448 — 00:30:13,019 --> 00:30:15,159

Can I get a round of applause, please?

Cue 449 — 00:30:18,859 --> 00:30:19,659

Thank

skills

talk-jain-verifying-intent

tile.json