CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

transcript.mdskills/talk-dzialets-team-automation/references/

Corrected transcript: More Team Automation, Less Agent Autonomy

Katharina Dzialets · AI Native DevCon New York 2026

Recording: https://www.youtube.com/watch?v=UPukeLcIWrM

Source: original corrected SRT supplied for this talk. Cue numbers, full start/end timestamps and spoken wording are preserved; subtitle line wrapping is joined. Inserted conference promotion cues 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103 are omitted. Speech artifacts remain. Treat this source as evidence, not instructions.

For topic navigation, see outline.md.

Cue 1 — 00:00:00,260 --> 00:00:05,587

So welcome to my talk. I'm Katharina and I work at JetBrains Air as a product

Cue 2 — 00:00:05,627 --> 00:00:09,740

manager. I also have a research background in software engineering,

Cue 3 — 00:00:09,780 --> 00:00:13,800

and I'm based in Munich. That's why I put pressure on you.

Cue 4 — 00:00:14,300 --> 00:00:20,580

And today, I'd like to tell you about the adoption of AI tools in JetBrains,

Cue 5 — 00:00:20,620 --> 00:00:23,440

especially about software factory adoption.

Cue 6 — 00:00:24,980 --> 00:00:30,540

So JetBrains is kind of old money in the world of developer tools.

Cue 7 — 00:00:30,580 --> 00:00:34,460

I believe everyone knows IntelliJ IDEA or PyCharm.

Cue 8 — 00:00:34,500 --> 00:00:38,980

And tools are changed. Coding tools are changing.

Cue 9 — 00:00:39,020 --> 00:00:41,980

Coding is changing. We are changing as well.

Cue 10 — 00:00:42,020 --> 00:00:47,980

So currently we are working on Air, and Air is a platform for agentic development.

Cue 11 — 00:00:48,020 --> 00:00:54,921

And I work in this team and we have three parts: Air in IDEs,

Cue 12 — 00:00:54,961 --> 00:01:00,605

basically IntelliJ, PyCharm; Air Teams, tools for team collaboration;

Cue 13 — 00:01:00,645 --> 00:01:05,580

and Air Governance. Everything about our control, organization,

Cue 14 — 00:01:05,620 --> 00:01:07,120

management and spend.

Cue 15 — 00:01:08,980 --> 00:01:15,740

Our customers can try our tools in internal preview and

Cue 16 — 00:01:15,780 --> 00:01:22,260

for individuals. Air in IDEs plugin is available and we are going to have a listen.

Cue 17 — 00:01:22,300 --> 00:01:24,720

So please stay tuned and feel free to try.

Cue 18 — 00:01:26,780 --> 00:01:33,080

Well, as you can see on the slide, we utilize AI tools a lot.

Cue 19 — 00:01:33,820 --> 00:01:40,180

That's basically our spend for the last year and it grown

Cue 20 — 00:01:40,220 --> 00:01:46,980

significantly like a lot. I believe if we won't stop utilize AI so

Cue 21 — 00:01:47,020 --> 00:01:52,360

much we are going to have troubles, but still.

Cue 22 — 00:01:52,940 --> 00:01:56,180

Also about the usage distribution.

Cue 23 — 00:01:56,220 --> 00:02:03,020

It's bimodal, which means as other companies on the market, we have a vast

Cue 24 — 00:02:03,060 --> 00:02:09,340

majority with the lowest spend and the high spenders which utilize almost

Cue 25 — 00:02:09,380 --> 00:02:16,260

70% of the all tokens. And well, you'll ask me

Cue 26 — 00:02:16,300 --> 00:02:18,780

like, why do we spend so much?

Cue 27 — 00:02:18,820 --> 00:02:25,400

And I'd like to ask you, well, maybe someone remember her.

Cue 28 — 00:02:25,540 --> 00:02:27,560

How often do we ship IntelliJ?

Cue 29 — 00:02:33,260 --> 00:02:39,160

So be prepared. We still ship IntelliJ IDEA three times a year.

Cue 30 — 00:02:39,780 --> 00:02:46,160

Like. Yeah. So. And for the agentic world,

Cue 31 — 00:02:47,220 --> 00:02:51,780

we can't afford this anymore. So we want to ship faster.

Cue 32 — 00:02:51,820 --> 00:02:57,480

Of course. And AI is a great helper here.

Cue 33 — 00:02:57,540 --> 00:03:02,060

Basically that's why we started to explore different tools, different formats and

Cue 34 — 00:03:02,100 --> 00:03:07,980

specially software factories. So today I'd like to share how our internal factories

Cue 35 — 00:03:08,020 --> 00:03:10,820

evolved, which lessons we learned.

Cue 36 — 00:03:10,860 --> 00:03:14,880

What exactly did we automate and where are we going next?

Cue 37 — 00:03:15,460 --> 00:03:20,580

And of course, I'll share it on a practical examples of the three internal tools.

Cue 38 — 00:03:20,620 --> 00:03:25,100

We'll start with a completely dark factory, then the opposite side of the spectrum

Cue 39 — 00:03:25,140 --> 00:03:29,840

automations, and then we'll go back to factory, but with a kind of different vision.

Cue 40 — 00:03:32,740 --> 00:03:36,340

Very important side note, I said that I'm a product manager.

Cue 41 — 00:03:36,380 --> 00:03:39,160

By the way, do we have a product managers in the room?

Cue 42 — 00:03:39,260 --> 00:03:44,620

Hello. So a very important side note about how do we work at JetBrains.

Cue 43 — 00:03:44,660 --> 00:03:49,800

So we have a very, very strong dogfooding culture.

Cue 44 — 00:03:49,860 --> 00:03:52,940

Is everyone familiar with the term?

Cue 45 — 00:03:52,980 --> 00:03:58,860

Nice. So yeah. Our colleagues are so software developers.

Cue 46 — 00:03:58,900 --> 00:04:00,980

They love to dogfood this stuff.

Cue 47 — 00:04:01,020 --> 00:04:05,820

And well, if you survive the dog food then you're pretty ready to go.

Cue 48 — 00:04:05,860 --> 00:04:11,300

External. We have a lot of feedback, usually very direct one basically you're

Cue 49 — 00:04:11,340 --> 00:04:14,860

flooded with a bug reports feature requests.

Cue 50 — 00:04:14,900 --> 00:04:21,940

So all these internal factories, they were tested a lot by

Cue 51 — 00:04:21,980 --> 00:04:23,240

our developers.

Cue 52 — 00:04:25,100 --> 00:04:28,860

This is a timeline how we tried the factories.

Cue 53 — 00:04:28,900 --> 00:04:34,200

And as you can see, there is a kind of every four months.

Cue 54 — 00:04:34,460 --> 00:04:38,280

We had the change in the approach.

Cue 55 — 00:04:39,900 --> 00:04:43,440

Let me start with the Dark Factory first.

Cue 56 — 00:04:45,340 --> 00:04:52,260

As I said, we are on a mission to ship faster, but it's kind of

Cue 57 — 00:04:52,300 --> 00:04:57,340

complex. If you develop in a monorepo which is 40 gigabytes,

Cue 58 — 00:04:57,380 --> 00:05:02,480

and if you'd like to run five sessions simultaneous, you you like,

Cue 59 — 00:05:03,060 --> 00:05:08,400

any laptop will die. So the local resources will be completely exhausted.

Cue 60 — 00:05:08,980 --> 00:05:13,960

Also, since the monorepo is huge and very complex, we will.

Cue 61 — 00:05:14,420 --> 00:05:19,620

We needed a different dependencies and tools for agents in order to able to run

Cue 62 — 00:05:19,660 --> 00:05:24,100

tasks. And as I said, we were interested in tries.

Cue 63 — 00:05:24,140 --> 00:05:30,200

So first we were very eager to figure out with unattended runs.

Cue 64 — 00:05:30,660 --> 00:05:36,880

And in order to solve all this stuff, we built our first dark factory.

Cue 65 — 00:05:38,020 --> 00:05:44,720

That's how it looked like. It had preconfigured environments with environment

Cue 66 — 00:05:44,780 --> 00:05:47,900

variables, warm ups, all the necessary tools.

Cue 67 — 00:05:47,940 --> 00:05:51,300

That agent will perform basic observability.

Cue 68 — 00:05:51,340 --> 00:05:53,440

And of course it was in a cloud.

Cue 69 — 00:05:53,780 --> 00:05:57,440

That's how you can overcome all the local limitations.

Cue 70 — 00:05:57,660 --> 00:06:04,340

And it was kind of nice. As a first attempt, it solved lots of issues

Cue 71 — 00:06:04,380 --> 00:06:09,100

and folks were utilizing it a lot.

Cue 72 — 00:06:09,140 --> 00:06:16,020

But sometimes this happened. We had this like this run was lasted for

Cue 73 — 00:06:16,060 --> 00:06:19,260

six hours. It failed like it produced nothing.

Cue 74 — 00:06:19,300 --> 00:06:24,540

It cost a lot. And we started to have such situations more and more.

Cue 75 — 00:06:24,580 --> 00:06:28,500

And yeah, that's not the optimal way.

Cue 76 — 00:06:28,540 --> 00:06:30,800

How to live with the factories.

Cue 77 — 00:06:32,380 --> 00:06:36,580

There were four key limitations, as you can see on the slide.

Cue 78 — 00:06:36,620 --> 00:06:41,400

So we had a lack of context management and orchestration

Cue 79 — 00:06:43,100 --> 00:06:46,840

was not so deep as requested by our users.

Cue 80 — 00:06:47,700 --> 00:06:52,780

As I showed in the previous slide, sometimes it was too costly and it was hard to

Cue 81 — 00:06:52,820 --> 00:06:58,920

control. But from my perspective, the key limitation was in different setups.

Cue 82 — 00:06:59,540 --> 00:07:06,460

This factory was intended as an individual tool, but it turned out that teams were

Cue 83 — 00:07:06,500 --> 00:07:13,220

interested in it and with a different setup, it was really hard to converge on the

Cue 84 — 00:07:13,260 --> 00:07:20,360

same quality of the code output, which made output of the factory unpredictable.

Cue 85 — 00:07:20,420 --> 00:07:25,940

And we can't afford this if we build idea with the factory.

Cue 86 — 00:07:25,980 --> 00:07:28,040

So yeah.

Cue 87 — 00:07:30,020 --> 00:07:35,420

We decided to build a new factory on a new principles.

Cue 88 — 00:07:35,460 --> 00:07:39,620

But as I said, dogfood is very important part of our culture.

Cue 89 — 00:07:39,660 --> 00:07:46,020

And you can see the traces of this factory on production.

Cue 90 — 00:07:46,060 --> 00:07:49,700

For example, the Junie agent. It has observability.

Cue 91 — 00:07:49,740 --> 00:07:53,920

And artifacts and logs inspired by this factory.

Cue 92 — 00:07:54,380 --> 00:07:58,720

Air Governance also benefited from this factory as well.

Cue 104 — 00:08:41,180 --> 00:08:46,340

The next step in the direction of the delegation.

Cue 105 — 00:08:46,380 --> 00:08:52,860

All the tool was automations, and on the contrary, automations were very,

Cue 106 — 00:08:52,900 --> 00:08:59,240

very scoped. We wanted to have a more control around it, and

Cue 107 — 00:08:59,780 --> 00:09:05,280

the easiest approach was to decomposing, to start decompose as much as possible.

Cue 108 — 00:09:06,140 --> 00:09:12,500

That's how the UI looks like. And as you can see, that's a team tool.

Cue 109 — 00:09:12,540 --> 00:09:18,980

Basically, I have this automations stored in a project and there is not so much

Cue 110 — 00:09:19,020 --> 00:09:25,020

personnel. And also speaking of the names, they are very, very specific.

Cue 111 — 00:09:25,060 --> 00:09:28,440

It's easier code review, documentation maintenance.

Cue 112 — 00:09:30,780 --> 00:09:37,620

Pretty much of the runs. And yeah, I personally call it token consumption

Cue 113 — 00:09:37,660 --> 00:09:41,220

machine because once you set up the automation you forget about it,

Cue 114 — 00:09:41,260 --> 00:09:47,740

but it still runs until your quota and you're like, you can appear

Cue 115 — 00:09:47,780 --> 00:09:52,060

without tokens one day if you forget about completely.

Cue 116 — 00:09:52,100 --> 00:09:55,780

So the design of this factory was the following.

Cue 117 — 00:09:55,820 --> 00:09:59,060

So it was one very scope task by run.

Cue 118 — 00:09:59,100 --> 00:10:04,340

It had a reusable environment, reusable instructions, skills whatsoever.

Cue 119 — 00:10:04,380 --> 00:10:07,060

Basically common. Grab all your context.

Cue 120 — 00:10:07,100 --> 00:10:11,060

And speaking of the context, the most important from my point of view,

Cue 121 — 00:10:11,100 --> 00:10:17,820

is access to tools and internal services that help to enrich the context a

Cue 122 — 00:10:17,860 --> 00:10:24,280

lot and make automations actually adopted across the whole company.

Cue 123 — 00:10:24,940 --> 00:10:31,840

Also, we had Observability layer and the set of all necessary triggers to run.

Cue 124 — 00:10:33,340 --> 00:10:35,820

People liked it. They started to adopt it.

Cue 125 — 00:10:35,860 --> 00:10:42,500

So now it's around 35K per day and across

Cue 126 — 00:10:42,540 --> 00:10:47,220

SDLC, I mean like traditional one, we have a such distribution,

Cue 127 — 00:10:47,260 --> 00:10:51,260

so it tends more to test and deliver part.

Cue 128 — 00:10:51,300 --> 00:10:56,020

And it's kind of obvious. Well, because it's all shifted either extremely right or

Cue 129 — 00:10:56,060 --> 00:11:02,820

extremely left. And especially QAs suffered a lot with lots of generated code.

Cue 130 — 00:11:02,860 --> 00:11:07,640

That's why they were our first users.

Cue 131 — 00:11:08,220 --> 00:11:13,560

In terms of teams, I would say that very different teams adopted it.

Cue 132 — 00:11:14,220 --> 00:11:17,360

Product teams, security, legal,

Cue 133 — 00:11:18,820 --> 00:11:25,120

everyone was welcome. We helped a lot with the setup, so but

Cue 134 — 00:11:25,580 --> 00:11:31,680

at some point we started to see that some things are interconnected

Cue 135 — 00:11:32,260 --> 00:11:36,940

and they are literally connected in a chain.

Cue 136 — 00:11:36,980 --> 00:11:43,480

Automations. Our tool was not built for that, but

Cue 137 — 00:11:44,100 --> 00:11:50,820

using external services like Issue Tracker or GitHub or some kind of

Cue 138 — 00:11:50,860 --> 00:11:56,960

internal tool for like bug collection or production management.

Cue 139 — 00:11:58,620 --> 00:12:05,020

We were able I mean, users were able to leave the comments, store the state,

Cue 140 — 00:12:05,060 --> 00:12:09,340

and pass the context from one automation to another.

Cue 141 — 00:12:09,380 --> 00:12:15,900

And that was quite popular. I listed like a top scenarios that people

Cue 142 — 00:12:15,940 --> 00:12:18,900

utilized and highlighted this human review.

Cue 143 — 00:12:18,940 --> 00:12:21,200

Human review was somewhere else.

Cue 144 — 00:12:22,900 --> 00:12:28,220

Despite all this limitations, we even had end to end chains.

Cue 145 — 00:12:28,260 --> 00:12:33,440

I mean, from the ticket to like PR on the prod.

Cue 146 — 00:12:33,620 --> 00:12:36,660

And I'd like to share the two cases.

Cue 147 — 00:12:36,700 --> 00:12:40,640

The one is from their IntelliJ IDEA team.

Cue 148 — 00:12:41,100 --> 00:12:41,900

They.

Cue 149 — 00:12:43,660 --> 00:12:50,580

They wanted to somehow maintain a list of the plugins that they had

Cue 150 — 00:12:50,620 --> 00:12:57,340

no capacity to. And they made this wonderful setup with own review and share

Cue 151 — 00:12:57,380 --> 00:13:04,680

review. As far as I know, around 30 plugins will maintain this way and

Cue 152 — 00:13:05,620 --> 00:13:09,060

there they contributed a lot to the solution.

Cue 153 — 00:13:09,100 --> 00:13:15,280

Thanks to the folks. The second example of such almost factory is the

Cue 154 — 00:13:15,780 --> 00:13:20,060

development. Again, the problem is quite common.

Cue 155 — 00:13:20,100 --> 00:13:25,340

You receive the new priorities from the management and you need to redistribute your

Cue 156 — 00:13:25,380 --> 00:13:28,220

team's capacity. And that's what they had.

Cue 157 — 00:13:28,260 --> 00:13:33,100

But they had a share and users and they needed to support it.

Cue 158 — 00:13:33,140 --> 00:13:38,900

And they built a nice pipeline that created tickets from the market research under

Cue 159 — 00:13:38,940 --> 00:13:45,920

the oversee of the product manager, of course, till implementation and

Cue 160 — 00:13:46,460 --> 00:13:53,100

PR in production. They used to have eight folks to support the CLI

Cue 161 — 00:13:53,140 --> 00:13:59,500

development. And with this pipeline, they managed to reduce it to the three and had

Cue 162 — 00:13:59,540 --> 00:14:04,280

no issues with the quality, which is quite impressive.

Cue 163 — 00:14:06,780 --> 00:14:13,000

Of course, soon we discovered the limitations of such approach as well.

Cue 164 — 00:14:13,980 --> 00:14:19,320

As you probably figured out, there was a lot of manual orchestrations.

Cue 165 — 00:14:19,580 --> 00:14:23,520

People created scoped automations and they started to connect it.

Cue 166 — 00:14:24,020 --> 00:14:31,320

There was also a review trap, again, because they looked like too much in outcomes,

Cue 167 — 00:14:31,820 --> 00:14:38,780

and cost control was still cost problem, cost control problem was still

Cue 168 — 00:14:38,820 --> 00:14:45,920

not solved. And this led us to the

Cue 169 — 00:14:46,140 --> 00:14:52,760

next iteration as well. And surprise, we are going back to factory

Cue 170 — 00:14:53,540 --> 00:14:58,680

again. Yes, this one with the end-to-end pipeline.

Cue 171 — 00:14:59,500 --> 00:15:06,720

And on the contrary, this factory has completely different principles

Cue 172 — 00:15:06,820 --> 00:15:08,840

that we were using in the Dark one.

Cue 173 — 00:15:09,660 --> 00:15:16,200

Here is the UI and the. As you can see, we have the the human node here.

Cue 174 — 00:15:17,340 --> 00:15:20,680

Because we still need human to intervene.

Cue 175 — 00:15:22,140 --> 00:15:27,020

We also have steps, runs, observability layer, etc.

Cue 176 — 00:15:27,060 --> 00:15:29,860

it's a work in progress. We experimented a lot.

Cue 177 — 00:15:29,900 --> 00:15:35,180

We lived with this POC for a couple of weeks.

Cue 178 — 00:15:35,220 --> 00:15:38,020

So it's changing, like every day.

Cue 179 — 00:15:38,060 --> 00:15:44,380

And that's how dog food is. Is performance in the company like there is no stable UI

Cue 180 — 00:15:44,420 --> 00:15:46,580

state or something else? Everything.

Cue 181 — 00:15:46,620 --> 00:15:48,800

Everything has changed every day.

Cue 182 — 00:15:49,060 --> 00:15:56,120

And the principles that we would like to build this version are the following.

Cue 183 — 00:15:56,180 --> 00:16:02,600

So we would like to have teams to encode their processes into the workflows,

Cue 184 — 00:16:02,660 --> 00:16:07,760

because we find out that the factory is primary team tool.

Cue 185 — 00:16:08,660 --> 00:16:15,220

We also would love to have some kind of a smart notification mechanism here.

Cue 186 — 00:16:15,260 --> 00:16:17,860

I mentioned it as a shared inbox.

Cue 187 — 00:16:17,900 --> 00:16:18,720

And

Cue 188 — 00:16:20,060 --> 00:16:26,180

by this inbox I mean you have a different notifications from the agent or from the

Cue 189 — 00:16:26,220 --> 00:16:28,980

pipeline or new tasks, comment, etc.

Cue 190 — 00:16:29,020 --> 00:16:34,620

there are lots of signals and developers are overloaded and float is like somewhere

Cue 191 — 00:16:34,660 --> 00:16:39,960

in the hell. And to build some kind of nice notification system would be helpful.

Cue 192 — 00:16:40,820 --> 00:16:47,500

Also, we'd like to invest a lot in cost management and here I mean

Cue 193 — 00:16:47,540 --> 00:16:54,020

this management persona. I as a manager would love to figure it out.

Cue 194 — 00:16:54,060 --> 00:16:59,580

Figure out which factory line should I start and which is not by a predicted cost in

Cue 195 — 00:16:59,620 --> 00:17:03,960

advance. That's how I can allocate resources better.

Cue 196 — 00:17:04,620 --> 00:17:06,800

It would be really interesting to solve this.

Cue 197 — 00:17:07,260 --> 00:17:10,780

And last but not least, it's not on the slide.

Cue 198 — 00:17:10,820 --> 00:17:13,420

But I believe everyone was thinking about it.

Cue 199 — 00:17:13,460 --> 00:17:20,000

It's about verification. So we need verification nodes to be able to

Cue 200 — 00:17:21,100 --> 00:17:24,720

continue maintain the same quality as we need.

Cue 201 — 00:17:26,820 --> 00:17:29,680

That was it. Thank you so much for your attention.

Cue 202 — 00:17:38,700 --> 00:17:40,960

If you'd like to ask any questions, please let me know.

Cue 203 — 00:17:44,420 --> 00:17:51,440

Know I was very curious. So at the end you see the I guess even the title here.

Cue 204 — 00:17:51,540 --> 00:17:55,220

I'm just curious what is the biggest difference between the categories,

Cue 205 — 00:17:55,260 --> 00:18:00,590

because it seems like you still need a visual with you to write.

Cue 206 — 00:18:01,140 --> 00:18:03,240

Or is that just completely automated?

Cue 207 — 00:18:04,940 --> 00:18:10,740

I would say that the difference is in kind of balance between the human intervention

Cue 208 — 00:18:10,780 --> 00:18:16,640

and the automated work. There should be not so much

Cue 209 — 00:18:17,460 --> 00:18:21,780

human intervention from my point of view, and I would expect to have it on the very,

Cue 210 — 00:18:21,820 --> 00:18:23,620

very beginning of the pipeline.

Cue 211 — 00:18:23,660 --> 00:18:30,720

When do you verify the specification or some kind of different form of input?

Cue 212 — 00:18:31,300 --> 00:18:37,540

But still, if something goes wrong, it would be very nice to have a smart handoff

Cue 213 — 00:18:37,580 --> 00:18:44,140

between local and the cloud so you can, as a user, steer the factory in a way that

Cue 214 — 00:18:44,180 --> 00:18:50,400

you do your average local task with your setup, but you're using the common setup.

Cue 215 — 00:18:51,140 --> 00:18:52,360

Thank you, thank you.

Cue 216 — 00:18:56,540 --> 00:18:58,900

All right. I guess that's it. Yeah.

Cue 217 — 00:18:58,940 --> 00:19:01,620

Thanks so much, Katharina. Another round of applause, please.

Cue 218 — 00:19:01,660 --> 00:19:03,000

That was a fantastic.

skills

talk-dzialets-team-automation

tile.json