Transcript — Vibe Coding: Is this really the best we can do?

Attribution warning. This transcript has no per-speaker labels. There are two voices: (1) an unnamed MC / host who introduces Farley at the start and handles conference housekeeping (rail strikes, party, ratings) at the end, and (2) Dave Farley, who delivers the main talk. When quoting, attribute housekeeping content to "the MC" (not Farley) and main-talk content to Farley. The transcript also contains speech-to-text artefacts (e.g. "copper" appears to be a mistranscription of "cop-out"; "tools" appears as "tours" in one place; "stuck control" likely means "stock control") — preserved verbatim below.

---

Section 1 — MC introduction

Okay. Hello, everyone. How's everyone feeling? All the benefits to seeing you. I'm totally not with you. I'm. Tired. I'm tired. My brain's tired. But we have one session left, and it's actually in a really good session. I'm not gonna say we're just. I'm not gonna say we saved the best till last because that's probably not doing justice for our other speakers as well. But I'm really interested in this session in day five. It's a great. A great speaker. So vibe coding. Is this the best we can do? I think this is going to be an interesting look at the processes and the practices we do today with live coding, what needs to go forward into our new practices. This is the last session of the day before the party. So afterwards, I'll let you know a little bit more about what the party's doing. But in the meantime, please make sure you're still doing your ratings and things like that on the app. But without further Ado, please give it up to Dave Farley.

Section 2 — Opening: what's durable

Thank you. I'm the last speaker between you and beer. So I'll try not to go on too long. I'm interested in the things that seem durable. And that may seem odd when we're talking about such a disruptive change as we are talking about when we're talking about. The adoption of AI for programming. But I think there's quite a lot of things that are durable. There's a lot of behavior. S that have stood the test of time and seem to continue to be important. And I'm not the only person that's been saying that today. If you've been listening carefully, there's been people saying very similar things as we go through. I'd like to look at maybe some of the reasons why this matters. And question some of the things that we may otherwise think about the new world of agentic programming. I think one of the things that we can certainly say, though, is that whatever does come next, it's going to be different. This is a huge change. This is a sea change in the way that our industry. And probably our industry is just a forum and how our world is going to work. So what does this really mean? What does this mean in terms of the impact on people like us who are involved in the production of software at the moment? And I suppose to some degree the Pathfinders for everybody else in the world as they start to adapt to some of these things.

Section 3 — Provocations

I think a starting point, if you'll forgive me for being slightly provocative, some of the differences that we talk about are kind of just done. So I would argue that vibe coding programming with natural languages. While having a place. Are also kind of bad ideas. And I want to talk a little bit about what I mean by that, and we'll get into it. I think sucking all junior developers, AI will write all the code. We don't need no programmers. Some of these are kind of right. Some of them are properly true, and some of these are a terrible idea. I think AI generated tests. Also a terrible idea. So I'd like to explore all of these things in a bit more detail. And there's quite a lot of nuance in this because there are aspects of these where they're all true. But let's talk about it.

Section 4 — What are tests for?

So first of all, I'd like to, I'm an old school card carrying test driven development developer. So let's start with tests, because that's where you're supposed to start. What's the test for? Is it to prove success? Not really. Is it to find our mistakes? Well, kind of, but not really. Is it to challenge my genius as a programmer? Certainly not. It's not. Those are not really what tests are for. They're not what make tests useful. I would argue that what tests are for us, they're our form of measurement. They are equivalent of a carpenter having a tape measure in his pocket that he can measure things with. We use tests to figure out that we're doing the right things. We use tests to figure out that when we've done them, they continue to be the right things. They're measurements to see if we've achieved our goals. But for that, we've got to define what our goals are. And we can't infer the goals from the solution because the solution can always be wrong.

So anybody that's selling you. Free AI testing that will come in and look at your existing system and test it, that might have some value, but it's not the same value as proper testing that we need as part of the development process. It's a different thing. And the reason for that is fairly obviously. If I, if I write a function, something like this, calculate tax. And I'm going to return 50 times the amount that I put in. That's a very bad place to live if that's what your tax looks like. But if that's my starting point, the only thing that an agentic AI or any other kind of AI can do is look at that and say, okay. Here's my test. It's just going to reinforce the wrongness that I wrote into that function. All that a test can do at that point. Because the only input it's got is the test is the, is the working code. It's verify that the working code does what the working code does. That has its uses. That's very good if you want to refactor the code, do behavior preserving change and not break anything. But it's rubbish if you want to develop a new system and verify that this new system continues to do or does the right things, because now it's proving that it does the wrong thing. Because that's where we started from. So that's not giving us our ability, an ability to. Define what we really wanted.

AI generated tests. If the code is the only input, we can only verify that the code remains the same. We can't infer the goals from the solution because the solution could always be wrong. So they're mostly a dumb idea. They have a, they have a place, but mostly a dumb idea. They tend to be a copper for people who don't, can't be bothered to state their goals. And I think that's a problem because specifying the goals is kind of what our job is. It's kind of what we're here to do as software developers.

Section 5 — What is a program for?

My next question, what's a program for? To Define a sequence of instructions? Well, not really. That's not the goal. It might be what it is. But to encode algorithms, that's not the goal again of, of a program. That's not what we do it for. That's not the value that it brings. To implement our brilliant design again, that's some of the self-fulfillment that I, as a program that might get from it, but that's not the goal. That's, that's not the goal. That's not the reason that people pay me to do it. So the commercial pressure is not, not of those things. That's not really what it's for as much as we might like to think of it in that way as programmers.

Section 6 — Three goals of programming languages

Programming language, I would argue, have three goals as tools. So first, they are tools that have been designed to help us to organize our thinking about a problem. They allow us to explore the surface area of a problem and kind of understand it in more detail, in more depth, and they're designed to work that way. Programming languages aren't difficult because we want it to be obscure and obtruse. We wanted tours [tools] that would help us to explore the problems in, in these kind of way. They're designed to help us to do that. They're also a means of communicating our understanding with other programmers, other humans. So they are communication tool between us so we can work as part of a team and understand what we are in some technical detail, exploring the depth and the breadth and the complexity of a, of the problem that we're trying to solve and the solution that we've come up with. Encoded as a program, in a programming language. And ultimately, they're there to tell a computer what to do. But that's kind of the last part because assembly language instructions inform computer what to do perfectly well. And we, not many of us spend their time writing programs in assembly language.

Section 7 — Three techniques embodied in programming languages

There are also three techniques that are embodied in programming languages that are, I think, are important for us to think about and to remember when we're talking about programming as a discipline. The first is we have a simple, consistent grammar. It allows us to express our ideas in, in a concrete, specific, precise way. Precisely enough to be executable by a computer. They are unambiguous expression of our intent in that respect and that this, this matters. And they're repeatable and deterministic in terms of execution. And this matters a lot, too. If we write something in the programming language of our choice and run it twice, given the same inputs, we're going to get the same result. Every time. That's important. It means that we can reasonably better. It means that we can test it. It means that we can understand what it means and we design systems that are more or less repeatable in that sense, but the tools themselves are repeatable.

Section 8 — How natural language fails on each

And it is natural language measure up to that because natural language, somebody said today, I think they quoted Andrej Karpathy saying that the programming language of the future is English. I don't like that idea very much for the reasons that I'm talking about. It's not precise enough. It's too vague. Natural language helps us. How does that measure or helps us through organize our thinking about a problem? Well, it's too vague, really. Natural language is open to misinterpretation. If anybody's been around, I can't really see you very well. But if you've got a gray beard like mine, you know, you've been around, you know. I bet that you've been in a situation where somebody's coming down from on high and given a very vague instruction to a bunch of programmers gone away and they've been really disappointed in the results because they didn't tell us in enough detail. That's really common. It's really easy to misinterpret what, what the goal really is. Natural language is always open to interpretation. Time flies like an arrow. These are some time flies, liking arrows.

Vibe coding alone is simply not good enough if we're just chatting with the computer to express our needs. That's not enough. We need tools that allow us to be more precise than that, to be more specific than that, to be better prompts at what it is that we want to achieve. That doesn't mean that we need to look at all the code and are only allowed to do this in a programming language. I mean, I'm in the category of it's been a long time now since I've written any code by hand because my AI agent writes all the code. But I've been more precise than just natural language in some respects in specifying what I want from it. I'm able to get what I want from it by using a precise prescriptive version of natural language in order to be able to prompt it. And that's what I'm really talking about here.

So does it communicate our understanding with other humans? It's a bit too ambiguous to do that very well because we, as we've, as we all know. Does it tell a computer what we want him to do? It needs to be precise for that. So it's very easy, unless what we're telling it is something very obvious, like asking our AI assistant to implement fizz buzz for us. It might do that in a repeated way, but otherwise it's probably not. It's going to be a little bit different and give us something a little bit different every time. Certainly in the earlier versions of AI coding assistance, we've all seen that. Where we've asked the system to build something, you ask it the same thing again, you get something very different. That certainly happened to me a few times.

So we've got these, what about these three techniques that we have? Simple, consistent formal grammar. Does natural language give us that? No, it's complex and inconsistent. Unambiguous expression of intent. No, it's ambiguous and vague. Repeatable deterministic execution. Not repeat or not deterministic. Not version controllable as a result. We can, if we version control the language that we prompted our AI with, that's not enough to get back to the same result again. On its own.

Section 9 — Three problems AI programming presents

So programming with a [AI] presents three important problems for us to replace traditional programming. So how do we specify what we want with precision? That's problem one. Problem two is how do we confirm that we got what we wanted? This is the verification problem. And problem three is that we don't work in the same way as the machines. They don't work in the same way as us, by the way, which you look at that the way that we do things is that we tend to deal with problems with, with the limits of our own context window. So what will fit inside our head at a time? We might have a picture of what it is that we want to build. But then small change by small change, the way that we deal with it is that we do it incrementally. We do a small piece of work. We evaluate that piece of work and then we, then we move on to the next small piece of work.

I would argue that high quality system development. In software is always an incremental process of learning and discovery. We are always exploring the problems base and trying to understand it over time, bit by bit by bit. I go further than that. I would say that's all engineering, all engineering is a, is an. Act of. Incrementally improving on what we know about the problem. And dealing with the, with the new things that we learn as we do that over time. And that's a hard part of software development, hard part of engineering in general. And then we've got to verify that we got what we wanted and it does all of the things that we need in a repeatable, reliable way. And that's a hard part of software development as well. And what have we done? We sped up the coding bit. That was the easy part of software development, which is good, it's great. It does accelerate us. But it also moves the bottleneck. If you've ever read The Goal, the theory of constraints, that's what we've done. We've just moved the bottom in [bottleneck] to somewhere else. And actually, we probably haven't moved the bottom in because the bottoming was nearly always about verification and release rather than figuring out what do we really want in rather than the encoding of it if we were any good. As delivery agents.

So an AI works differently. And AI works, it kind of builds this context, this picture. And then it goes bang and it makes a huge change. Probably until recently you kind of rewrite the whole system every time. And there's nothing really to stop it while it's building your stuck control [stock control] system to write space invaders. Instead, if we haven't told it not to, if we haven't defined what our goals are and are in, in a position where we're able to evaluate it, that part of the problem here. Is that a lot of people that talk about using AI, AIs and AI agents to help with coding. Are often talking from a perspective of building small simple systems. I don't know about all of you, but those aren't the sorts of systems that I generally build professionally. That's certainly how I've played with the eyes [AIs]. I've written space invaders with an AI agent, and I've done simple things like that. All those nice, straightforward things. But my professional building of software has been for bigger, more complicated systems than that. And that's not good enough. Just, just being able to check them manually to see if they're okay. He's [It's] not even close to being good enough to verify that we get what we wanted.

So. If the AI is making huge steps and changing things in one big, big, big change at a time, and we're not able to verify that we got what we wanted effectively, how do we learn? How do we find our understanding? How do we reliably update our solution to improve it? And how do we regenerate if it's regenerating it from scratch every time? He [It] can't. It's not really helping us to learn and adapt and figure out where we are and figure out what we need to do next. It's, it's, it's a problem. The last two problems, problems two and three mean that we lose the, our ability to work incrementally. And that is fundamental to building complex systems of any kind, but certainly complex software systems. It's a, this process of learning and discovery. So how do we keep our ability to learn, discover and incrementally add to our systems as we go? We have to have, we have to have a means that can keep up with the rate at which we can produce software.

Section 10 — The 12,000 lines/day anecdote

I spoke to somebody not very long ago who, who's engaged wholeheartedly in the adoption of agentic programming using multiple agents to build systems. He's writing a game. He reckons he may, he writes 12,000 lines of code per day. No human being can review 12,000 lines of code per day. No human being can test, manually test the output of 12,000 lines a day behaviorally to figure out whether it's doing the right things. We've got to find ways of speeding up the rate of verification and assurance to tell whether we got what we wanted to keep up with that speed of generation of code. It's the only thing that makes sense, it seems to me, if we're not to suffer from the theory of constraint problem.

How do we confirm then that we always get what we want? So first we've got to specify what we want. Clearly reproduce in a reproducible manner. And then we've got to be able to verify repeatably that we still have it after every change. If you're familiar with my work with Continuous Delivery and so on, that might sound familiar. It's because it is, it's the same that we've been doing when we're talking about practicing continuous delivery. It's table state [table stakes]. We have to have this vast cycle of verifying that we go what we want.

Section 11 — Past vs future: specification

So how should programming change to keep up with all of these? So problem one, specifying what we want with precision. In the past, the way that we did this was that a program was a precise solution encoded as algorithms. Weirdly, we kind of lost what the problem was. It's kind of implicit in our solution, but we keep that in our heads and we build a solution and that would. Be a definition that we'd work to. But the solution itself was implicit. So in this, in my example, this is a routing algorithm. So I'd like to find my way home is the outcome that I'm trying to achieve here.

In the future. A program will be a precise description of what it is that we want, I think, and coded as specifications translated into execute or [executable] instructions by the AI that will verify that we got what we wanted. And what we want now is explicitly part of, if you like, a program. So the program moves away from being. Solution focus, to being a more accurate description of the problem that we're trying to solve. I'm describing a form of spectrum [spec-um? specification] development, but the specification takes the form of executable specifications, tests, if you like, that both specify what we want and also verify that we got it.

What it takes to achieve that program, I've just, I'm just repeating what I said, designing a domain specific language, if you like, for specifying what it is that we want from our software with a deal of precision. And then specifying what we will aim to achieve in detail all of the time. And those are, those are all of the behaviors of the system. Not just the conventional user behaviors that we might think about. We're not talking about sketchy testing of the system, but all of the behaviors of the system. So how fast would we learn the results? But if performance matters, how secure do we want it to be if we want security system to be secure? Because all of those are contextual. You can't have the AI to infer that to make it up because it depends. The real, you know, how security you want to your system today, you'd be stupid to apply the level of security to a single user game than you would if you were building a system for a bank. Because you'd be over engineering the game and probably under engineering the bank. So you need to be specific about what it is that you're trying to achieve. In all of the dimensions of architecture and design. So some of those are behaviors of the system that we need to specify too.

Section 12 — Past vs future: verification

The second problem, how does it, how about that? So how do we confirm that we got what we wanted? In the past, we hope that our solution solved the problem. Language verified some aspects of this in terms of the syntax that we use, particularly if we use types, languages and stuff like that. But they didn't, it didn't work to verify functional correctness. If you were a well behaved test driven development developer, then testing would fill that gap and would verify that our code did what we wanted it to do from that in that sense. But otherwise maybe not.

In the future executable specifications will work as the verification of our system as well as the specification of what it is that we aim to achieve. I think. This makes the AI, we can do this in human language. It doesn't have to be difficult or hard. The way that I do it is that I teach my AI how to do the kind of BDD style tests that I like to use. And I specify what I want. And my AI fills in the gaps from a relatively simple English descriptions of what it is, what the specification that I, that I, of what I want. AI generates solutions that maps the specifications. And now we can run those tests against those solutions. We can verify that the AI is doing the right thing by giving test test values that it hasn't seen before, so it can't cheat the tests. And get feedback that we got, we actually did get what we wanted.

What it takes to achieve that. The programmer. Invents problem specific DSL. Defines all of the desired behavior the system has prompts to the AI in the form of BDD style specifications and the AI. Takes the specifications, builds test infrastructure to make them executable. Generates code to fulfill all of the executable specifications. And we've got what we wanted. We've got the precise definition of the behaviors that we want and a verification that we got those behaviors.

Section 13 — Past vs future: regaining incrementalism

Regaining, the last of the three regaining incrementalism. So in the past, we would work iteratively in small steps adding new tests and code. We would gather feedback validated by our deployment pipelines. Build systems incrementally treat each change as an experiment and value empirical learning.

In the future. We go to work iteratively in small steps adding new tests and code. The AI will generate the code. Gather feedback to validate every change. Build systems incrementally, many small changes treating changes and experiments and value empirical learnings. It's exactly the same. So the stuff, if you've been practicing continuous delivery. As two of the speakers that I heard speaking today. Said. At least implicitly. Then things become easier. Because that's what we're doing the same things. We're evaluating stuff as we go, working in small steps and we're understanding where we are at old [all] times and so are our AI agents.

What does this take to regain the incrementalism in this sense? Well, we've version controlled the specifications because those are now in effect program. We run the tests ourselves in a deployment pipeline. Otherwise the AI is going to cheat. And we're going to stop the AI gaming the test by using the sorts of techniques that I mentioned about maybe keeping some tests back and not showing the AI for what's going to. Cost. So it can't cheat. Them.

Section 14 — Worked example: flight planning

Most of this that I've described is how acceptance testing and behavior driven development already works. So this is a teddy thing [a tidy thing] necessarily new if you've been working this way before. This is a really simple step. Here's an example that I've used. I've done this now several times, many times for building real systems. This is not a real system. This is an example from a training course that I did an exercise with. So here are some specimens kinds of specifications that I'm talking about. If you think about what I'm trying to describe, we start off with the vanquish [vision] of what we want. We get to a slightly more formal version of describing that, which is a user story. And then we come up with some examples that would demonstrate that the feature of the story describes exists. Those are our examples, those are our executable specifications. That's what these things are on the screen. These are the ones that prompted building a system that actually did this. It actually did flight planning in this way.

Section 15 — Conclusions

My conclusions are that natural language alone is not good enough. BDD style domain specific languages are the best alternative for prompting effectively and giving us the combination of precise specification of what we want. And built in verification that we got it. Automating testing from a solution is a bit of a joke. It's a niche. It has some utility, but it's a corner case. It doesn't solve the real problem. We should treat skills to analyze and decompose problems as the real core of our discipline. That's the real discipline that we as engineers, technologists bring. It's that way of reasoning about a problem of decomposing it and thinking about the corner cases. Thinking about kind of the architectural level.

One of the ways of thinking about all of this is that effectively what we're doing is we're involved. In invoking a fifth generation programming language, a fifth generation programming system. And I think that's where we're at. So we need a different model for the role of AI. AI assistance is rather like the compiler. We're not going to care for very much longer at all if we do now. About the code that it generates because we'll verify that we've got the results. And as long as we get the results, who cares about the implementation? It's rather like the move from assembly language programming to high level language programming. I don't know if anybody in room is old enough to remember that I don't quite, but I was an assembly language programmer for a while. And for a while, I used to look at what the assembly language was that was generated by my compilers, but I haven't done that for years. And most people have never done that. So why should we care about what the code that was generating was if we can verify that the behaviors that we wanted are the ones that we got.

If you'd like to see a demonstration of this in use, there's an open source project called N-able that's worth a look. That applies this thinking and promotes this way of development. So. That. I'm going to skip over here because I've got. The end of my talk. Thank you very much.

Section 16 — MC outro (NOT Farley)

Big thank you today. Thank you. We don't actually have time for questions, unfortunately, but please come up and grab Dave afterwards or of course the party is the great time for good questions. I've got a few important announcements, so please, please remain with me for a few minutes. But first of all, I just want to say thank you to the live stream. We had over two and a half thousand views across the live stream on this stage. And every other track is recorded by the way, both today and tomorrow. So while this is a big thank you to those in the live stream, I just want to say that all sessions will be made available from both today on all tracks as well as the tracks tomorrow as well. And we'll make them available to you very soon. So for now, thanks very much and we'll see you on the next hey native [AI native], DevCon virtual session. Thank you. I think we're offline now. So now we can talk honestly. Cool.

So. Actually first really, really important thing that I just want to make everyone fully aware of is unfortunately tomorrow we do have rail strikes. But I want to let you know about what is working and what is not working. So for tomorrow's tube state tube strikes, the circle line, the piccadilly lines will both be fully shut. The parts of the metropolitan line and central line will be suspended. But other lines are expected to run just as a reduced service as they're expecting half drivers to be on strike. What we'll be working is buses, london over ground, the elizabeth line will be fully working. The DLR will be working and tram services as well, although they will be expected to be busier just because of the others being down. The closest is probably Elizabeth flying straight to founder [Farringdon]. That should be fully working. As an incentive, don't forget we have over 3,000 pounds worth to give away at the closing. So there's a little incentive for you for the session.

Okay, a little more about tomorrow. The workshops are actually fully booked. But if you haven't booked on it and you really, really want to attend one, I don't expect all 60 folks are going to be there on time. And as soon as we kick off. If they're not there. Those spots become available. So we're first in, first out in the physical queue at that door. So I don't know how many spots will be available, but feel free to queue up and we'll try and get you in. If there is high demand, so please do on the app go ahead and say, you know, join the waitlist. If there is high demand and you are local, we will try and rerun those sessions in person in the Tessl office at King's Cross. And we will send an invite to folks who are on the waitlist and also generally if folks want to come and join that as well. A couple of other things. Yep, please do go ahead onto the app and rate every session that you attended today. That really helps us and it really helps other speakers as well.

And finally, let's talk about the party. So party goes on from six, which is 10 minutes ago till nine. It's outside the front door, so you have to walk past it to leave. So there's no escape, you have no reason not to come. There'll be wine, there'll be beer, there'll be cocktails, and there'll be soff [soft] drinks. It's at 10 o'clock, 10 in the morning start tomorrow. So we've got you back a little bit there. 10 o'clock in the morning start at 9am breakfast here as well. There is a ton of food stalls that will be serving throughout the entire night. So if there is a big queue at a food stall, don't feel like it's all going to go. You can just, you know, you can just join a little bit later when the queues are shorter and you will definitely get food. And yeah, that's it. I think everyone's hopefully taking a huge amount of learnings throughout the day. So now is our time to kind of like kick back a little bit, have a drink, have some soft drinks, eat some food and just chat with each other. So thank you very, very much for attending the whole day. We'll hopefully look forward to seeing you tomorrow. And let's continue all right.