Transcript — The Humans Architect the System, the AI Writes the Code

Speaker-label warning. The source transcript has no per-speaker labels. The talk itself is single-speaker (Paul Stack), so attribution to Paul is reliable for the main body. The Q&A section contains audience questions whose askers are not identified — attribute them only as "an audience member" / "a questioner". The transcript also contains speech-to-text artifacts that should be preserved verbatim when quoting: e.g. "marriage" almost certainly = "merge", "scale"/"scales" in some places = "skill"/"skills", "claw code" = "Claude Code", "polarizer" = "pull request", "via coda" = "vibe-coding", "QR code" likely = "Codex" (or another coding agent), "Eldest One Club" may be a mis-transcription of the new company name, "marriage gains" = "merge gates", "marriage code" = "merged code", "feel the right" = "fix the [code] right", "yoga search" = "you go search" or similar, "polarizer is open by my agent" = "PR was opened by my agent". Do not silently correct these in quotes; you may add [likely: merge] in square brackets if it aids the user.

Participants metadata: the user supplied no participants list. The questioners in Q&A are anonymous.

Section 1 — Cold open & framing

Pause it here just for now. Line of code am I written since? I don't know, maybe end of end of January. This is how we work. This is how I work. I'm not here telling you this is how you should work. Trust me, it's very liberating when you don't need to care about the code anymore. But I promise you it's like a very interesting direction or a very interesting way of doing. I'm going to shoot [show?] in our process in detail. And you can take what's useful. Everything I'm going to show you is open source. You can play around with it. You can have a look at it. You can see what craziness in there. But by the end of it, I want you to think through, like, the types of things you can do.

Okay. The distinction here is between write the code. And building the machine that writes the code. Okay. And the AI era, that's the very interesting thing. We hear it called software factories. Dark software factories, all these different things. It all boils down to a very similar thing of gray cinnamon [great common?] process.

So what we're going to cover here today is why we stop writing code. Okay. What changed when we did, how we designed guidelines? What is our architecture first pattern meet home? Probably a little bit later, but that created me. And how AI gives you back the work that we actually got into the industry to go for.

Section 2 — What changed at System Initiative

So. The bottleneck is no longer faking [taking?] in software. Okay. It used to be that we had to think about, you know, how we took an idea, how do we push that idea through to users when we tested that hypothesis and the time scale to do that was a huge amount of investment. And we had to sink a lot of up front thought and process into it.

Okay. At the company I work at. Previously, it was called System Initiative just last week we spun up a brand new company based on the lack [back?] of this, and it's called Eldest One Club. At System Initiative, we went, we spent six years building a profit [product]. Okay. We're trying to revolutionize infrastructure as code and the ways through infrastructure is code. It was the most beautifully, and I really need [mean?] it, the most beautifully architected rust that you've ever seen. Okay. It was an incredibly well designed distributed system that run many microservices that talk to each other through event buses and did a lot of stuff. End of January, we threw every single line of the. Grid [gridded? — likely "we threw every single line of that out"]. Every single day [line].

And it's not because the code is bad, it's because the product needed to rethink for the age of the year [age of AI/the agent]. The product wasn't going to resonate as we went into AI and pushed forward. We let, unfortunately, we let the team go at that time, and five of us stayed behind, five of the whole company staying behind. And on January the 25th, we started with a mirror [Miro] board. With not a single line of code written in the entire system. We were like, right, let's redesign something from scratch and think through what we actually did.

This slide I originally first gave this talk two months ago, and I said two months. Absolutely thrilled by what we have. Four months and below the way that we'll be film. Ing [feeling]. Okay. And I genuinely need not call [not quite — likely "genuinely mean it"] because I've built it, but because I've actually built the product that I've always wanted to work with as an operations person. Yes, I am an ops person, a developer. I've been writing code for a long period of time. And so I have a very. Big, big interest in what we're actually trying to build here. I worked at HashiCorp build in Terraform. I worked at Pulumi, building Pulumi. I've worked in the developer tool space for a while. Someone [I'm someone] incredibly opinionated, incredibly opinionated. So for me, it's, like, very important to build something that I really care about.

Section 3 — Why Paul fell out of love with code

So the question is. If I'm not writing the code. What am I doing instead? Actually, it's even better. Okay. I fell out of loaf [love] of code many two years ago. And the reason I fell out of love with code is because I was on a zoom call with five other people, and they were talking about the naming conventions of nat [NATS] cues [queues]. I just, I wanted to, like, lose my mind. Okay? Least interest in playing the world [thing in the world] for me personally, the people I was on the call with loved it. They're so intrigued by it. I really wanted to be perfect [it perfect]. I didn't care.

Okay. I just want to point out something. I'm from northern ireland. If I swear, it just happens. I don't even realize it just slips out, so I apologize in advance.

Section 4 — What he does now — architecture, not syntax

Okay, so for me, not what I actually do is even though I own the product, because I'm, like, actually, like, the person responsible for it, but I actually only architecture that goes with the product. I care about the designs, the constraints, the invariants, the things that sort of make this system coherent for a user across time. The engineers are getting consistently good output. When they actually express what they want architecture.

And we're gonna [wrote a] blog post about this. Maybe three weeks ago called the vibes don't scale. Okay, it doesn't. Okay. TLDR in the blog post, they don't scale. Okay. If you're via coda [vibe-coding] and there's no idea of what you're trying to do, can you give it a single line prompt? You're going to get somewhere. Is it going to be the most secure, the most useful tool over time? Probably not. You're going to have to invest some serious time. And you're probably spending a lot of tokens in order to do.

So for us, we try to make them specific, unambiguous constraints. Which means we violent [violently] handwriting code. Absolutely valid [forbid?]. We cannot, none of us as Engineers or people that work in the company, we can't send a pull request. When it's written by us. It'll be. It'll be deleted, removed. It won't even be closed. It'll be literally just deleted as a pull request. It's not a guideline. It's not a preference. It is literally how we do things. Agents write every single line of code. Okay, you can go to our rep [repo]. Even the agents write the code. I promise you, it's not awful. I would like to say it's great. It's not. I mean, there's going to be things that you'll look at and go, I don't like that. I don't like how that's led [laid out]. That's taste. Okay? That's, you know, that's the least important part of it. Naming the vengeance [conventions]. How many times have we argued on pull requests? As software engineers over naming conventions on how many lines you'd like a class have before it's refactored? I once had a ridiculous, absolutely ridiculous conversation where somebody says, if a class is over a hundred lines, it should be micro seconds [microservices]. I'm like, come on. This is stupid. This is absolute crazy.

Section 5 — The no-PR policy

Okay, so for us, it's much more about the fact that we give the lms [LLMs], our agents very strict operational and design guideline. S. Okay, that's what we work on on the day-to-day basis. We work in the system design. We work on the constraints that the agent has to work within. We work with all of the pieces that the agent has to form really strict owner of the Bible. These are the patterns, the boundaries, the architectural constraints, you know, the invariants that we must hold, things that have failed in the past where you need to feed that data back into the agent doesn't make that same mistake again. Because you know what agents are bond [bound to do], they will make mistakes. Actually. They'll tell you they haven't made a mistake, but they absolutely have made these things. So this is not for us documentation. This is actually executable constraints. And this is what's really, really important.

Okay. Also, because we don't write code. We don't accept anybody else's code. We are an open source tool. We'll talk about that a little later. I believe truly that the norms of open source are now changing in an AI world. So many projects are being overwhelmed. By AI terrain [-generated?] tools, by AI driven code that is going into the system and they're just not willing to maintain it. So many people, the tools are running this in order to stop it from happening. Mitchell Hashimoto created a tool called fudge [Fudge / Nudge?]. Okay. Where you actually have to budge [vouch?] and say that you're not an agent and that you actually, you know, you're the right person.

It's like center of code. We don't accept any pull request in any way, shape or form. And it's not because I don't trust your code. I absolutely do not. But what I'm trying to do is I am trying to maintain the integrity of our supply chain. I have end users of a binary. The last thing I want to do is to try and open an attack surface where somebody who is inevitably smarter than me. Will be able to inject code into the system. And you see so many of these problems happening all over the internet. Today. Okay? Trivia [Trivy?]. Absolutely, like destroyed a couple of months ago. Okay. We saw many shy [shai-hulud?] to attack last week where 364 npm packages are actually at risk in the system.

So, you know, we're very much a case of AI will generate really plausible wealth [well-]format test and code at scale. And if we have to author and look at that code, we're never going to be able to distinguish what is a supply chain attack. Versus what is actually a change in the code itself. So we just found it [shut it down]. I'm not sure we got a lot of pushback from people. They're like, oh, you know, that, that doesn't really seem great. I absolutely will accept you, like driving our product forward. Give me issues. Give me feature requests. Give me design specifications that you think are more interesting. Give the ideas where the code is not great and the system will actually help drive it forward.

Section 6 — End-to-end flow: triage → plan ↔ adversarial → human arbiter → UAT → release

So for us. We have a full flow from idea to marriage [merge] code. And our flow looks kind of like this. Okay, so we have a start and you triage an issue. Okay. We actually wrote, funnily enough, with an AI world, we run a skill. Okay, everyone is writing skills. Our skill is backed by CLI. Okay. Our skills, like our skill is, is just yard rails [guardrails] running, like, the correct CLI fountain [function] to do it.

But effectively what's happening here is we triage an issue or a feature request. It then gets classified. It then goes into a loop of plant [plan] generated review, plant generated review of our generator review. And that's an adversarial review. Okay, so it's great a plan [it'll create a plan]. You know, an agent will say, this is the best one from what you do. Then I have written, I've written a very grumby [grumpy] adversarial review that says you don't trust any code in the world. You have to prove that it's secure. You have to prove that it has no injection attacks. You have to prove that it's architecturally complete as to the guidelines. And these two agents basically go into this loop to figure out where it gets in the right thing.

We get to five. Okay. We'll only ever allow five loops. Before it steps [stops]. And it says, hey, we can't agree in this. You have to be the arbiter of what's going on. At that point. I have to step. Human is in the look [loop] mode at this point in time. Planners generated plan a secure base [secured by] and adversarial. Human has to say, this is the right feature. This is the right bug fix. Here is the reasons why.

We then started punitive [executive — likely "we then start to execute"]. After implementation, we recreate it. If it's above [a bug], we try and verify it. If it's a feature, then it goes into pull request open. Pull request open goes through another set of reviews. If it's failed and it's back in the so on and so forth, then we go in to notify or, excuse me, release it, and then we notify the users who request it and we are done. Okay.

This isn't a whole pile of shell scripts because that would be horrible to maintain. This is actually basically a state machine inside a skeleton [skill]. Incredibly simple. But actually incredibly powerful because that can extend the state any. More [anymore].

Section 7 — CLAUDE.md as executable contract

For us, the entire entry point of the system is our claw.id [CLAUDE.md]. yes, we have QR code [Codex?] users. Okay. Agents. You can swap in and you can do the same thing. For us. It's welcome [Claude]. Okay. It is an executable contract. It is literally the center of gravity in our entire system. And therefore every claw code [Claude Code] or sea ice [CI? — or codex] session will raid [read] that claw code by default.

Now our claw code is slightly different. We don't like embed old [a lot of?] like words. We're not good with words or software engineers. We have things like this. Okay. TypeScript. It must be strict. No pennies [anys] because any's aren't the devil. They must be named exports. No defaults. It has the agplyright [AGPL copyright] on the top of every file. No fire forget promises. Don't lose people's memories. Right. That architecture. Everything has to have long adjacent [JSON?] endpoints. Everything has to import from moz [mod]. You should never leak into the past [public?] because that's an implementation detail. These are the types of constraints that we care about. Okay. These are the types of things that are very, very important.

And then there's always a line at the bottom of our claw code that basically says, hey, if you hit a non-obvious problem during a session that's going to trip up future users, future cloud sessions, future agent sessions, record it and propose an update. So you pull it down before we continue.

Section 8 — Test strategy & merge gates

So when we, when then we went to a complete test strategy, if [it's] unit test integration test contract test, property test, architectural tests, and that's the end of the binary. Once the binary is built, we then take that build binary and ship it across to a different repo. We take it completely out of the context of everything else. And we run it as a user. We run effectively all uat test. Run this script. Here's the intent of what it's going to do. Run this command. Here is what it's going to produce. Never like hard coded for words [forwards]. That would be crazy.

And then we have another set of tests that sit on top adversarial test. Okay. Those tests are. Try and inject it. Try and kill the process in the middle. Try and kill the data. Try and remove it. Try and do everything that a user will ultimately do. No bunny state. They just do it. And that's just what happens. The user is your best user. Your best QA of your entire system. We try and treat those bugs that come in when we feed them back into the adversaries.

The nrci [our CI] takes over. CI for us actually has five marriage gains [merge gates]. Okay, we have our code review because we need a code review. That's just the way it works. But then we have an adversarial review. As soon as everything's broken, try and prove that it's broken. Then we have a user experience review. Check the CLI. Make sure that it doesn't regress the output. Make sure that the commands are the correct structure. We try and go non player [noun verb?]. So we try and keep that sort of, you know, thing create, say, get, etc. Etc. So we try and ensure that consistency and we have to encode that as a review step. Then, of course, we need a CI security review. Because all sorts of cheap [shady?] people, they're going to be trying to eject [inject] stuff into my CI pipeline. So I want to catch it before it even starts. And then the last one, which is amazing consider where we are in everything guy [going] said, it's for us skill checks are incredibly important. Okay. Not just the actual skill, but the content, the format, the layout, the triggers, that is a hard gate for us. Our system is incredibly set up to be driven via an agent. So therefore the agent experience must be really good. So we literally will block any regressions in content or experience as we go through.

When everything passes, it merges itself.

Section 9 — Self-merging PR demo & numbers

I'm not kidding. Okay. The reason I'm not kidding is. If I show you. This pull request merged just before I got on stage. And you can see the polarizer is open [the PR was opened] by my agent. It had first review. Then this one passed. Then it goes a big walk [bigwig?] interview [review]. I can't do that. Agent then figured out that it had the pushing changes. Right here. Keeps going because it's all context. It's incredibly important. There's a new change. And the agent was actually able to go and reproduce itself, figure out that everything was fine. These suggestions are all okay and then it automatically sounds and. Kicked it [auto-merges and ships it]. Okay, we're not doing anything like out of the ordinary here. We're just trusting the process that we continually hold in that process.

So for us, the gate is our UAT. Just because someone is marriage [merged] doesn't mean it automatically gets to an end user. We have this whole level of test that, that goes through and we catch regressions. Every single day before it goes to end users. Absolutely every single day. It blocks our pipeline and nobody else can, like, publish a release out there until that's fixed.

Now, the most important thing is here. Is in our UAT repo. We have a line that says that tests are the source of Truth. Don't change the test. Always feel the right [fix the code] if it's a regression bind. Er [finder].

Okay? And what we've ended up with in the last 30 days is 295 issues have been opened, and that's issues, not just bugs that feed [feature] your request, Etc. I ship [shipped] 217 of them. I closed 81 as either duplicates are just not things that we're going to do. The median time to triage. That's, this is not work hours. This is like elapsed hours in a day is 4.6 hours. And from triage the whole way through the ship, including all of those gates is 1.6 hours.

Okay, there's five people the company I work at. Each of us has a claw code. Max Pro, which is 200. And then we spend about 1500 to two thousand dollars per month. On the whole CI review process. We're able to ship software. Many times faster now for three thousand dollars per month than we were ever able to do before. It's really not that.

Section 10 — Self-debugging agent / removing bus factor

But what we've actually managed to do is we taught the system to debug itself. Okay, so swamp actually knows when it encounters an error. It will check the source control of the code out at the version. That it has the error mat [at]. It will try and reproduce the error and then it will open an issue in their behalf. It will even check if the issue has been fixed at a later version before it happens. So we're trying to include this guardrail in and what we're doing.

And we end up with this, like, like really tight look [loop] with bugs. Which means the bugs that we get from the agent are incredibly well formed and they have a lot of great context that goes with it. So for us, we're not really just automating code. We're actually building the whole system that keeps the system running and keeps it going.

And what we care about here is we don't think about spider [bus factor?]. Anymore. We all work in teams where one person is like the overarching architecture person who you know you need to go to ask about the part of the system that hasn't been changed every six months. Or every year. Okay. We do an infrastructure as code. We do it in a platform. We do an actual software all the time. Okay. We're trying to remove that bus Factor because we're trying to make the agent aware of the system. An agent is in a fresh content every single time.

In the past. I don't even know how long. Maybe two months. I have not had a rum slash player [/clear] in my context. Moment. Because every record [every run / every Claude] is a brand new cloth work tree [Claude worktree] which keeps it very focused, very specific, and it means that it actually cares just about. The context and not. Everyone.

Section 11 — How to get started small

So when can you get started? Okay, since the whole point of why we're in a conference like this. Okay? The smallest possible way to do this, I talked to so many people about this. Okay, turn your conventions into constraints. Sit down and think about the constraint that right now, if you put an agent on a piece of software in your source control, then it doesn't know it will be able to get from the source control itself. That's the first thing that you can absolutely do. And those constraints are really important.

Encode that one thing. That only one person in the organization can software at IT organization knows there is definitely one in every org. 100. Okay, run that loop once. End to end. Figure out where it breaks. And that's the next constraint that you have to go. You always have to start really small and just find you. You just can't push an agent at your entire silos [Silicon?] code repository. Just imagine that things are going to be great. It will not be. It absolutely will not be. But if you do this right and in this small, controlled way, what ends up happening.

Section 12 — Closing: "intent is the new architecture"

Is that AI is going to give you back the work that you got into the industry to do. So the user problems, generating systems for end users, have an architecture that system design.

I blog [blogged] last week saying the words intent is the new architecture. Both previous sessions in this room, guys and then Dana's, who's a great reflection of exactly where people are going in that respect. And I love seeing both of them. Because context is the new code. And then I started talking about the fact that intent is how you drive the system move forward. You can see that a lot of us are starting to come to the same agreement, the same consensus about what is actually happening.

Section 13 — Live triage demo of issue #518

So as I told you, I kicked off a swamp image [issue?] triage. Before I started this. Talk. Okay. I started with the command at the top that says triage issue number 518. It loaded a scale [skill]. It ran a couple of commands. These are actual swap [swamp] commands. These are rcli [our CLI] commands which set up the guardrails for the system.

Then what it does. Is it fetches the details of the issue, starts to understand the code base. It started to read the operations files. It started to read all of the context that it actually means [needs]. It then brings me back a list of summary of findings. Here's what I found. Here's what's happening. And then because of that, it can classify it and it can basically say, hey, this is a bug. I have everything I need to write the plan. Then it builds a very structured plant [plan]. And because it's a very structured ground [plan?], it's actually able to give me a really detailed output. Of what the plan is plus any of the adversary warnings that are happening.

Okay. This is not, honestly, when I say this is not. Like super smart. This is just building guidelines and guardrails. This is actually what I'm talking about here.

Where is my screen? Yeah. There we go. So. It's really important here that what we're in a situation to do. Is create in this setup, this structure, this different piece that you need in order to build this factory, this, this system that allows your system to be able to drive forward.

Okay. And then, like, for us, we built it in swamp club. So small dashclub.com [swamp-club.com / swampclub], you can go and have a look at how we built it, because, as I said, every single thing here is open source. All of the extra triage, all of these pieces, all of these factories are all open source as well. So for us, we love being able to show this off. This is like our pride and joy at this moment in time because we're having such a great time building the software that we are.

You can find me on the internet. At stack 72. I've written countless blogs over the last four weeks about how we built this, and it's a stack 72.dev. And I believe I have some time for. Questions.

Section 14 — Q&A: bottleneck

Right there.

[Audience member]: Hi, that's a great talk. Thanks very much. Is your, what's your bottleneck now? Like, because you're running this on floor [Claude?] max or you weren't running on your laptops, is that the bottleneck that you wish you could put this on the cloud or, like, wonderful questions?

What's our like? Bottleneck is deciding what to build. Okay. I don't run autonomous agents that sit in triage every issue coming in. That would be math [mad]. The reason I'm going to be mad at this is that somebody is going to reject something. So there's always human looks at it. Is this virtual [a real thing]? Is this something we care about? That's our the thinking. It's actually, you know, deciding what to build. Because when I can build so fast, I can spend 10 agents at the same time. But if it's the wrong features for the users, I'm just going to create this monolithic system that doesn't really make any sense and has no sort of drive and little consistency going forward. That's our bottom line. Our bottleneck is my thinking. Thinking through what is important for our users.

Section 15 — Q&A: how humans arbitrate

So let's put it up.

[Audience member]: You said with the, your system, the life cycle review, you know that it's, it's automated until sometimes something gets flagged where the agents can [can't?] create [agree] and therefore that's going to come to the loop. What when the human comes in that point, how do they get there in a context? To be able to decide, you know, which way to go? Just like, you know, the agents have been churning so much knowledge that when you, I mean, probably there are different ways, depending on the context, but you have penny start [a starting] dates for pirate headwind [private head-wind?].

Wonderful question. Okay, so we, the question is, how do we as humans arbitrate when the agents are actually not able to agree? So for us, it's like every time we run that command, that, that plan, that adversary, if you [view], that's capture[d], that's structured output within our own system, hence one [hands-on?]. So we can actually do smoke [some?] the query and say, hey, give me the four things that can't agree on. Tell me where it can agree, because all it is is just data, right? It's literally just data where we can actually ask questions and we can, like, because it's, and it's so easy to understand the difference, and then we can make the decision.

And if at the end of that week, that makes sense of it, we go, right, yoga search [you go search?] and start a new one up because we've only lost, like, 10 minutes and there's no issue. We're not like, we're not spending thousands of dollars. I mean, there's literally a very short period of time that we've been able to do it. So we get [can] spin up as many as we actually need to.

Section 16 — Q&A: open-source / commercial model

[Audience member]: Could you expand a little bit on your open source model, especially the commercial model? I'm a big fan. I completely agree that we are Reinventing how open source works, and I like to approach it. But what's your promotional [commercial?] motivation?

Okay, so we are in the business of solid software licenses. Okay. That is literally that. It's like a red hot [Red Hat] model. Okay. And the reason that it's that way is because people not care more about supply chain and they care more about the maintainability and the drive-through that. So if somebody wants to take our software, Fork it, build it, do whatever they want. Go for your life. It's hplv3 [AGPLv3], right? It's the way it works. You can't build a competing company because that's not a [in the] terms of services. But at the same time, like, we are, we are actively working with a lot of people who want that license. We want that, like, that care, that maintainability of the software, they don't want to have to maintain this themselves. That's our license. Plate.

Section 17 — Q&A: does this scale to large orgs?

[Audience member]: Thank you so much for this talk. I think the question that I have. I'm sure T5 [there's 5?] people in the team right now. Right? Is the approach applicable on large scale build where you have, like, 100 different services and charity [shared?] scale, the main expertise.

What a wonderful question. Is this approach scalable to other people? Dana said it really well in the last. Talk. When you have a guardrails in place, when you have this system in place, everyone within the org is actually empowered to do the right thing. We don't believe that we need to scale the company out to hundreds of engineers to be able to do this. Right. What we need is we actually care about people who have architectural vision on product usability and care to drive the system forward. Once they are there, once they know what we're trying to build, we do it. Then multiple agents can happen and say, I have five aims [agents] running right now this second of my laptop. You know what I mean? And that's the way it's going to be.

I truly believe that in the future of software in this way, a junior has become really important. Really important. Okay? Because right now, when junior's commitment engineer works, they do low level stops [stuff]. Like, open small pieces of color [code] because you can't give them the system. You can juniors really can learn architecture constraints of the system. They don't need to care about the syntax of the code anymore. Sure, they can read it. Sure, it's really important for them to do it, but they actually land [learn?] in a faster way about the constraints and systems thinking that they've ever done before.

So for us, it's about making sure that we keep that core group of people who are actually entrenched and making sure that they use [our] feedback is really good.

Section 18 — Q&A: is this the honeymoon period?

I created a, I hear these questions all the time. I've recreated a bunch of issues. These are all questions I've had before where the, like, things like we judge [chucked?] six years of work, you throw it away, and you're happy with four months. This is the honeymoon period. Absolutely. 100. Am I going to be like this in four months time? I probably have redesigned the system four times in the limit [the interim]. It's just the way this is just the way the world works. Maybe I'll even do a bong [bun?] style and rewrite it in rust. I don't do that. I'm like, you know, we're learning a bunch. Anybody who's actually come find me on the hangaran [hangar / hangout?] for the rest of the night. Thank you so much. Enjoy the rest of your.

Talk starts in eight minutes. Here and in the other places. So you've got enough time to get some water. Coke [Folk?] me with problems with the app. You got to find another room. Back very soon. Thank you. That was great. I look forward to playing.