From building CoPilot as GitHub CTO to building a code foundation model at Poolside: Jason Warner explores the AI Code Generation journey and where it leads.

Episode Description

In this episode of AI Native Dev, Guy Podjarny sits down with Jason Warner, a veteran in software development and AI. Jason's impressive career includes roles such as CTO at GitHub, where he contributed to the creation of GitHub Actions and GitHub Copilot, GM at Heroku, and positions at Canonical and Redpoint Ventures. Now, as the founder of Poolside, Jason focuses on developing state-of-the-art language models for software development. This episode covers the evolution of code generation, the impact of AI on software development, the challenges and limitations of current models, and speculations on the future role of developers in an AI-native world.

Resources

• Follow Poolside on Twitter: @poolsideai
• Visit Poolside's website: poolside.ai

Chapters

[00:00:16] Introduction - Guy Podjarny introduces Jason Warner and outlines his impressive career.

[00:01:39] Jason Warner's Journey to Poolside - Jason's career from GitHub to founding Poolside.

[00:02:27] Foundations of Code Generation with LLMs - How Large Language Models (LLMs) are revolutionizing code generation.

[00:04:15] Code-Specific Models vs. General Purpose Models - The differences and benefits of domain-specific models.

[00:09:04] The Evolution of Code Generation Models - Examples and the current state of code generation.

[00:15:49] Challenges and Limitations of Current LLMs - Identifying the boundaries of current models in code generation.

[00:32:47] Training on Customer Code and Context - The importance of customer-specific data in model training.

[00:44:29] The Future of AI in Software Development - Speculations on the future of AI and autonomous developers.

[00:54:16] The Role of Developers in an AI-Native World - How the role of developers is likely to evolve.

Full Script

[00:00:16] Guy Podjarny: Hello, everyone. Welcome back to the AI Native Dev. today I am going to have a really fun conversation with Jason Warner. Jason and I go a fair bit back, Jason and I, it's eight years. It's always,

[00:00:29] Jason Warner: I look at eight

[00:00:30] Guy Podjarny: all these numbers get a little bit big. Probably biggest relevant claim to fame over here is he was GitHub's CTO, at the time that, built, that he built up Copilot and Actions and a bunch of other, fun things over there.

[00:00:42] Guy Podjarny: Before that, you were the GM at Heroku, right? Building up a lot of end to end developer workflows, Canonical, before that, and then now you are after a stint on the dark side, at Redpoint Ventures, you are running Poolside. You founded and running Poolside, which is a code focused foundation model.

[00:01:02] Guy Podjarny: Does that sound right? Did I miss anything, anything key here?

[00:01:05] Jason Warner: No, you got it. if you tore through my background, which was, yeah, investor for a couple of years, but, GitHub, Heroku, and Canonical basically means that I'm pretty useless if it's got a motor, but useful if it's got a keyboard, that's an easy way to understand what I do.And now we're building Poolside, which is, it's a frontier AI company, so think OpenAI, Anthropic, etc.

[00:01:25] Jason Warner: focused on software as a, as the domain and the discipline, and so focused for, as developers. We're building the, state of the art language models for software development, specifically. And there's a host of things that means, but that's what we do.

[00:01:39] Guy Podjarny: Yeah. And we'll dig into a bunch of those. So thanks for coming onto the show, Jason. And, so Jason, maybe start by, helping us level set a bit on the foundations of this thing. We're going to talk a lot about code generation and doing so with LLMs, or these foundation models. can I ask you to just ground us a little bit about like, why does this thing even work?

[00:01:59] Guy Podjarny: Like what are the sort of the fundamentals of code generations with LLM? And why is it different now?

[00:02:05] Jason Warner: I think, the best way to describe this, how I just try to describe this to folks who are not in the field, which is just that these, let me take a step back and say something that maybe we don't typically do, which is we owe OpenAI a debt of gratitude. From a commercialization, from a corporation perspective, they went and they did a lot of hard work for a bunch of years, and when it wasn't really in vogue to do it, and they proved out a bunch of things.

[00:02:27] Jason Warner: And one of the things that they really proved out is that, all the scaling laws are going to hold. All of the various things that we postulated about as an industry or in academics might be true, they said were true. And they just had the guts to go do it and throw a whole bunch of money at it.

[00:02:43] Jason Warner: And effectively what we've seen is that as the, as these models get larger, we throw more data at it and more compute at it, what they're able to do is effectively solve more complex problems. And solving is a air quote thing that we're going to have to get into in a bit, just think of these things as,

[00:02:59] Jason Warner: understanding enough, from their internal systems to be able to answer questions. And that's the way I look at this. It's like, Hey, you asked a question. It has got a whole bunch of representative state inside of its, inside of its network. And it knows how to navigate that. And you'll hear a lot of us in the industry,

[00:03:16] Jason Warner: And you've seen this yourself say, we don't necessarily know how these things work. No, we know how they work. It's all math, what we can't say is exactly what the answer is going to be every time. That is a thing that we can't do because what they're effectively doing is they're saying based upon the probabilistic,

[00:03:31] Jason Warner: XYZ, I'm going to answer it this way this time.

[00:03:34] Guy Podjarny: Yeah. More, more than we understand how do we educate them, but we don't necessarily know precisely

[00:03:41] Jason Warner: we nudge

[00:03:42] Jason Warner: would learn and what is it that they would distill from Yeah. How do we nudge these things into a way that is acceptable to us as humans to engage with them?

[00:03:48] Guy Podjarny: Yeah. And another sort of basic question is, code generation, in the early conversations, there was a lot of conversation about how do the LLMs even find out about code, And the premise was that the initial learning was on comments and text that revolved around that code, but that subsequently the state of the art has improved and some sort of dedicated code specific content was created.

[00:04:15] Guy Podjarny: Is that correct? Can you tell us maybe a couple of words about that evolution?

[00:04:18] Jason Warner: There's some history here, which is you end up in the early days, you ended up with code specific models. And actually there's a whole host of people who are still building code specific models. Models who are trained exclusively on source code itself. And then you have the OpenAI's and Anthropics who are general purpose, who have code in their training data.

[00:04:35] Jason Warner: And then you have us who are domain specific or focusing on software. So we have slightly different orientations, but what we saw early, what OpenAI say we as an industry as the global, we as an but OpenAI typically saw, and sometimes the academics saw, was if you take a general purpose model, which is all NLP data and you say, Hey, we're going to just take all the NL data, but all of a sudden you start adding code to it, it gets smarter.

[00:04:59] Jason Warner: And by smarter, it means it's reasoning and it's logic and all that sort of stuff starts to improve. Then there was a whole bunch of research on to why that is. And then a whole bunch of us who are more pragmatic just said, that's effing cool. We don't actually care about the why in that specific case, but holy crap, adding code to this

[00:05:12] Guy Podjarny: it can do some amazing things.

[00:05:14] Jason Warner: You can do some amazing things and that's effectively where OpenAI, Anthropic, and a bunch of general purpose models are is like, okay, we're going to add code to this because it's going to make the general reasoning capabilities of the models better. And then, but a side effect of that is that it's going to be able to answer code questions as well.

[00:05:30] Jason Warner: Now it's going to be limited in understanding of software, but it's going to be very good at code in the moment. And then the code specific ones like the, the small language models, code only models are really good at things like ghost writing or code completion because what they're effectively doing is this is my wording here and this is where Copilot started as well is fancy code completion is like looking at the next, the line that you're trying to write and it's going to complete the last little bit of it.

[00:05:52] Jason Warner: What are the next tokens in that? It's just next token prediction again, but at a different level, a different type of thing. And then the function level, etc. And then, what is happening and, Poolside is representative of this and there's others doing this as well is that you're flipping it around and say, okay, so why don't we focus on software as the domain?

[00:06:09] Jason Warner: And then what can we do by to advance the state of the art in that way, similar to what's going to happen or is happening right now in robotics, or it's going to happen in drug discovery. And you'll have domain specific ones who folks who are saying, I'm going to go solve this problem specifically.

[00:06:23] Guy Podjarny: So it,applies the same techniques and maybe the same learnings that we had around scaling and maybe even a little bit of the combination, of the text around the code or going from code to software that maybe the original, the original, AI models on it oftentimes are quite dismissive of anything that was, that was outside the, the code itself.

[00:06:46] Guy Podjarny: So even taking advantage of our human understanding as we would as developers, to better understand the code itself, but if before it was like a lot of text, a lot of text, add some codes to that text, understand everything together, and then maybe start removing the text, focus more and more on the code, but with maybe some of the practices, I guess, would you say we anchor today more in, a focused, generic model that we have refined and then from here now we're augmenting or does it more resemble, the AI models of the past augmented with, LLM.

[00:07:19] Guy Podjarny: I think if you're asking about the general purpose models that exist today, like the Claudes and the GPT 4 and things of that nature, what I, what they do, in my opinion, is They're really good at language, and they're applying similar techniques to the code side of the fence. And the result of that is that they understand what, how to answer questions relative to code.

[00:07:42] Jason Warner: Now on the inner workings of, I'm not as familiar with the inner workings of GPT 4s or the Claudes and exactly how they're applying those techniques. But I know from us, what we do is we actually have, if you think about our neural network that we're building on Poolside, we actually have slightly different techniques.

[00:07:55] Jason Warner: You, you treat a code data set and, and all data sets slightly different and the interleaved portions of those involve understanding what is which, like a comment is treated in one way, but it's got to have context. It's relative to the context of the code and the code is relative to the context of the comment, so it has to be able to understand both, but you're going to apply slightly different,learning techniques to, to those because they, they require slightly different learning techniques.

[00:08:20] Guy Podjarny: Yeah. Yeah. And that makes sense. There's a whole world of knowledge that is not anchored in language, but rather anchored in code. So let's, we'll go.

[00:08:28] Jason Warner: Code is super fascinating because a comma. A misplaced comma in code is incredibly bad. It's like changes the work, not work, functionality, but a misplaced comma in language can be bad, but for the most part, humans who read these things can overlook that, but the compiler can't overlook that.

[00:08:46] Guy Podjarny: It's a very different notion. Instrumentation, the telemetry, the understanding of whether something was or was not correct in the context of code, is there's a lot more opportunity there versus maybe in language in which oftentimes implications and whether you got it right or not are a lot harder to deduce.

[00:09:03] Guy Podjarny: So

[00:09:04] Jason Warner: we can get into this one later in the

[00:09:06] Guy Podjarny: yeah, absolutely. So before we dig, we're going to unravel some of these different types and strands on it. So there's still a little bit on the level set. Let's talk a little bit about what is the current state of the art. So I'm curious, you dig into it, you have your own, one of the sort of the top models in the code side.

[00:09:22] Guy Podjarny: How, what is the most impressed that you've been from a co generation? If you think about examples you've recently seen, or maybe, edge cases in, in how you challenge your own models and models around.

[00:09:34] Jason Warner: I think there's a couple of, that always impress me, whether it be, what we're building or what somebody else has built as well, but these two that kind of always stand out is somebody who doesn't know anything about software, but they want to solve a problem. And for, and they've, they're able to go from a nothing to a something.

[00:09:53] Jason Warner: That is just a monumentally impressive

[00:09:55] Jason Warner: thing,and whenever you could think about with experience, how long that might have taken somebody typically, and the reduction of cycles, and time, saving have in there, that's just a really impressive thing. And you can see this all the time on Twitter these days.

[00:10:10] Jason Warner: People who say, I could I use X to go build this thing in some amount of hours? And they couldn't fathom themselves doing this before. That's just impressive. Now the other one, and my joke here is anytime I ask any model about ffmpeg and it actually gives me a usable, real answer that works at the command line, I'm always impressed because ffmpeg is an effing nightmare.

[00:10:30] Jason Warner: Yeah. So many flags,It's so many flags. And that's, it's just one of those ridiculous things. It's not, nothing's turing complete until it can answer ffmpeg questions. But the one that really impresses me is when, and if you think about how I'm answering this question, I'm talking about novice developer, I'm talking about experienced developer, so in an experienced developer, all of a sudden you get some sort of weird error message popped up from a thing that blew up and then you just cut and paste that thing into it, or you ask it to say, what is this?

[00:10:58] Jason Warner: And it actually knows. It can give you the solution to that problem. It can point you to the thing. And I remember like my own experience for the first time I saw this was with GPT 3 when it was doing a certain thing where we were building Copilot and we were navigating this and I piped in,some sort of log that we had and I was like, it showed me not in the code, but it showed me like where to think about the problem happening.

[00:11:18] Jason Warner: And it wasn't super obvious. I didn't spend a lot of time with it, the error message, but I was like, Oh wow, that was actually correct. And then recently when I was doing with what we're building, something blew up, and I piped in the, error message and it literally pointed at the piece of code. It said, this is where this is broken, this whole thing.

[00:11:37] Jason Warner: And it was a long, long, like very kind of complex, hard to parse error message. And I was like, Oh, I wish I was not 45 years old. I wish I was 20 again or 15

[00:11:49] Guy Podjarny: much time this would have saved you.

[00:11:51] Jason Warner: Because this is a completely different next 25 years of my career.

[00:11:56] Guy Podjarny: Yeah. and I think,we didn't really properly introduce these terms, but how much do you think this is retrieval versus reasoning? So how muchof these types of examples are really around sparing someone the needs to do something that would have been repetitive versus some sort of deep understanding that has been, embedded somewhere between, within those neural networks,

[00:12:21] Guy Podjarny: to help some reach conclusions.

[00:12:22] Jason Warner: Hard to answer, right? This is one of those you're asking me almost a gut feel

[00:12:26] Guy Podjarny: That's true. Yeah.

[00:12:27] Jason Warner: And so how I'm answering this is going to be maybe dissatisfying to folks, but I would say a lot of this is obviously going to be retrieval. This is going to be an understands, it's seen, it's whatever, but there's obviously cases where it could not have possibly have seen enough data on that thing to be able to answer it.

[00:12:44] Jason Warner: And, my, my answer to this is I always ask questions, in niche programming languages to Poolside and I ask about Crystal, I ask about Pony, I ask about these weird languages that a lot of people don't know about. But, you can see it having kind of intelligence around problems. And this is where it gets interesting.

[00:13:03] Jason Warner: It's like the problem that you're trying to solve is X. So the specific implementation in this, syntax, the language would be Y. That first part is actually going to be what we might classically call intelligence. It understands the problem and understands how to solve that problem, and then it's applying it to this.

[00:13:20] Jason Warner: That's what we as humans would do. And so that is super impressive. That said, we are, we have, the ramp that we have is ridiculous. Like where people think we are in the moment is, wherever you think of it is we're at the lower end of that curve. What we're going to see in the next five to 10 years is going to be ridiculous.

[00:13:37] Guy Podjarny: Yeah. It's going to be dramatic. Yeah. but I love that example because indeed, when you're, when you evolve in your career and your skills as a software developer, you become much more naturally polyglot. Like you understand the concepts of it at the beginning. You're a lot more, I use, this type of polyfill or whatever in JavaScript, right?

[00:13:54] Guy Podjarny: I use these classes. I use this in the SDK. you are indeed more retrieval. You're just repeating a pattern you might've learned, or you sit down and you think deeply. Over time you build intuition and that intuition is oftentimes transcends language. It's. it's about concepts of understanding how the systems work.

[00:14:07] Guy Podjarny: And so I love the esoteric, programming language, example, cause it almost proves out the polyglot nature, of the model.

[00:14:15] Jason Warner: And you can ask, it's always interesting to me because I think you can ask a question, you can do two things. Like whenever you're evaluating these things, the more experienced people versus less experienced and like naive approach to evaluating these things, like people might grab a random LLM off the streets and say, I'm going to go against human eval, I'm going to go against all these other things.

[00:14:33] Jason Warner: And a lot of that stuff is just, one, it's BS because people train on the data now and that's just not a thing that you

[00:14:38] Guy Podjarny: Yeah, of course. Once you set a benchmark and it's public, that's the downside of it.

[00:14:42] Jason Warner: but the other is ask it to do things that would be to apply concepts, and this is again like in, if I were to do X and Y, how would I do it? And again, it has to understand X and it has to understand what it would mean to do it in Y, and those two things are the interesting bits.

[00:14:57] Jason Warner: So if you were to say, I wanted to go do X and Rex. Or I want to go to X in Cobol or Fortran, and then I want to go to X in Java or Perl or Python. So completely different in Haskell or whatever it is that you have to understand that it's applying two different things here. It's not just saying, this is how you do X in Rust.

[00:15:17] Jason Warner: It has to understand the both sides of that fence to be able

[00:15:19] Guy Podjarny: Yeah. Yeah. Super interesting. So indeed, maybe like before we dig into indeed the, all sorts of the forks in the road, of this, of this domain. what would you say is currently, the boundaries of,of the LLMs today? And, and maybe which ones are a little bit surprising. I think it's easy enough to just imagine something that is massively complex, but what would you say are the things That today's state of the art code gen LLMs, yourselves included, can't yet get over.

[00:15:49] Jason Warner: So I think that let's just baseline where I think we are, which is all of these things are great code assistants. So in my words, what I say is we're in a dev led, AI assisted world. The developer is still front and center, hands on keyboard, asking questions, or getting some feedback, whether it be code assistance in terms of like ghost writing and code completion or asking in a side by side chat kind of world, but it's assisting the developer to a degree.

[00:16:13] Jason Warner: And yes, it might be writing lines of code and they have to accept it or whatever it be, but it's still assisting the developers.and I think it does well, like the top tier thing, I think all of these things do well is they do more, they do better explaining. So if you think about a 10/10, 9/10, 8/10 it's in the upper tier explaining.

[00:16:30] Jason Warner: So if you have a file or a repo or whatever, what does this do? It's really good at that. Highlighting code, explain this to me, that sort of thing. Or just asking generic questions like in Java, if I were wanted to do this, how would I do it? Explain how I would do that. It's very good at that.

[00:16:49] Jason Warner: Tier down from that is actually the root generation of the code. I think it's not as good at generating code that none of them are. even though they're, if you look at them objectively, that's still a strong function, but if you have to tier them, that's that.Now, where I think all of them, still have a lot of ways to go is jumping from AI assistant to the developer to an autonomous junior developer.

[00:17:11] Jason Warner: And this is like this holy grail kind of category where we keep on saying, these things are junior developers at this point, and that's not true. None of these things are junior developers at this point. You still need a developer to hands on keyboard. And I think what we've been trying to do is work around the model limitations by putting agents around them and all that sort of stuff.

[00:17:27] Jason Warner: But we've done this multiple times with GPT 4. And eventually the models themselves become so much more capable that the agents themselves will fall away as well, because the capabilities will be inherent in the models. So where I think that we do not have these strong capabilities is on the, quote unquote model agency side of the fence, which is why we're building a lot of those tools.

[00:17:48] Guy Podjarny: So they're not going to work autonomously.You could tool them to work semi autonomously if you want, but there's limitations in that. That a, is that primarily a reliability problem? Is it, because if they're a useful assistant, if they're an assistant that keeps getting it wrong, they're also useless, but if they get it right often enough, I think the delta between autonomy and an assistant, at least one dimension of it is the notion of reliability.

[00:18:13] Guy Podjarny: How likely is it to get it right?

[00:18:15] Jason Warner: But this is like in like gut here at the moment, which is that, let's just call it like a,degrading graph at some point, if you start to talk to GPT 4, any generic model about software as an example, and you ask it to continue to get more and more specific as it goes, the answers actually will decrease over time because there's get the broader and broader context, there's too much stuff being pulled in, which is why so many people go after context.

[00:18:40] Jason Warner: So imagine you want to go after a very difficult problem and you need to keep going back and forth and keep putting more and more information into this. Your answers are going to get less precise and it feels like it's going to actually degrade in time. Now, again, I'm saying feel here, because again, this is, I'm using these things

[00:18:56] Guy Podjarny: It's very hard to measure these things as well, as we pointed out, the benchmarks are a way to do it, but they are, they're useful in some fashion, it's still a measuring stick, but they're clearly only one lens

[00:19:07] Jason Warner: Yeah, I think there's also ways around even what I'm talking about here, because you can start saying to, to, you're going to broaden the context when this, you're going to stuff everything that you wanted from the previous into this and see what it can do. And if the model has been trained to understand that you can, it could get some degree of confidence out of that.

[00:19:21] Jason Warner: Or you can say, Hey, we're going to start doing embeddings and we're started doing all this stuff around the model itself. More how I think about this is, people always want that ask the question, will software developers exist? Because they view a world where somebody is going to ask a singular question, create me a Twitter clone.

[00:19:37] Jason Warner: And then out pops a Twitter clone. and that is a Holy grail type of scenario that I do think exists in the future, but we're not there today because what happens in between there, here and there is so much plumbing of that system. That's going to be required to go do that. So it's not just the AI system anymore.

[00:19:53] Guy Podjarny: It's like everything around it. Now I think that, interestingly, I think the AI will actually get to that point. that's what I, all of us who are building models, I think, believe. And but we're just not there at all. let's, let's maybe start bifurcating some of these different types. First let's dig a little bit into this notion of a generic versus a code specific model. So we touched a bit on that already, in saying, the types of sources, but still, I don't know if you can, how would you draw the line between a generic OpenAI, GPT type model versus a code focused, model like Poolside?

[00:20:28] Guy Podjarny: So I think,the way to answer this question, the way I've tried to answer this question is what I believe happens in the world. And so let me take a core belief I think all of us have. So let's just take. Anthropic, OpenAI, Poolside, because the two leading general purpose models, the first domain specific for software, as an example.

[00:20:49] Jason Warner: So why didn't we go build a general purpose model and then start trying to say Hey, we're going to market it towards the software side of the fence? We have a belief and our belief is that in an infinite world, so infinite compute, infinite energy, infinite data, infinite, all of us at the same time.

[00:21:06] Jason Warner: The general purpose model is the way to go toward to get to AGI, because like we all are doing the exact same thing. We're trying to use as much compute, throw as much data at this thing, had it learn as fast as possible, etc, grow as large as possible, smart as possible, as dense as possible.

[00:21:18] Guy Podjarny: We've already learned, some of these learnings into ChatGPT was that at least to a substantial degree, these scaling laws have worked, give it more data, give it more compute. It produces better results.

[00:21:29] Jason Warner: Yes. So we, I think we would all say that we believe the same thing. So in that, in the infinite or unbounded world, we'd all be doing The same thing, but we don't live in an unbounded world. We have a lot of constraints. So we have constraints on energy, we have constraints on how many, chips you can network at the moment, even in the future, how much data we have in the world, and then how we're gonna spend our time to focus these things.

[00:21:52] Jason Warner: And now, so my, belief in pool size beliefs and others who are going to build like robotics specific ones, or,drug discovery specific ones would be that the focus yields higher results, better results. And so the bifurcation here is that, the, we're going to focus our compute efforts, we're going to focus our data efforts, we're going to focus our fine tuning efforts to be around the domain.

[00:22:13] Jason Warner: And then we're going to say, a good example here is that Poolside is actually a very good general purpose model. You ask it butterfly questions or to write a song or a poem, it does it. But, it's not our focus, we don't care, we're not going to market towards it, and we're not going to really fine tune it that way.

[00:22:30] Jason Warner: We're going to spend all of our energy and efforts this way. So the bifurcation is really going to be about focus. How we think about what we're going to care about, how we're going to care about our data sets, what we're going to craft the human responses for, how we're going to think about the users interacting with the system.

[00:22:44] Jason Warner: And that's just not a first decision. And it is, it should very much be a first decision, but it's also a pragmatic knock on decision for even, we don't care that it regresses on butterfly anatomy when we start to add a ton of synthetic, data to this, we just don't. it's gotta be good enough, it doesn't have to be excellent at that.

[00:23:02] Guy Podjarny: Yeah, because at some point, more data can translate into more distraction, above and beyond, there's the fundamental, just like sheer dollars, computes, like how much do you funnel into it, but there's also the element of, of, of focus, if you give it more information, it has to factor in more things.

[00:23:20] Guy Podjarny: It's harder.

[00:23:21] Jason Warner: And so a good example here is our distribution. We're 50 50, so we're 50 NL data and 50 percent code data. So if you just think about it from the data distribution perspective, what it is, and if you think about all LLMs, it's just distribution over their data. So the more that they know about non software, the more that they know about that, but the more they know about software, the more they know about software, the more associations they can make and think about our fine tuning data sets as an example.

[00:23:45] Jason Warner: And obviously we've become known for the, what we call reinforcement learning via code execution feedback, which is effectively building a massive simulator environment to generate a ton of synthetic data that augments, our models. And this allows us to get out of the conundrum of not having enough data in the world.

[00:23:59] Jason Warner: We're actually generating our way out of not having enough data, but that's a bunch of our secret sauce. And, but it's also the focus on the domain matters here because that's such a large effort that, I would say that nearly half of our engineering, half of our entire team is focused on that one problem.

[00:24:12] Jason Warner: And it's not, it's a problem that you would only really tackle if you cared deeply about software as your prime use case.

[00:24:18] Guy Podjarny: Yeah. And this is, it is very interesting and it's, I think the, maybe like a, maybe a perspective here is that code, if you want to understand it, is not just the words around it. It's not just the static analysis of it. It's also observing that code in production. So you're learning, your ability to understand,

[00:24:39] Guy Podjarny: correctness from incorrectness, track the journey. I remember you and I have had conversations back when you were at GitHub, about GitHub futures and where you would it go? And even back then, and of course your Heroku routes and all that, you always were passionate about closing the loop, between the code and the, and the execution of it.

[00:24:58] Guy Podjarny: But it really builds on that perception that, there's unique insight or like very powerful insight that you can achieve from observing code.

[00:25:08] Jason Warner: There's a host of things. So there's this one magical workflow and I made the joke earlier that I'm useless around engines, and that sort of thing. But there's one workflow that I always think about and have been thinking about for the past 15 years. And it's essentially every time a developer touches a key on their keyboard,

[00:25:24] Jason Warner: to what happens in productions. That whole magic kinetic chain in between touching the single key all the way to something happening in production. And if you think about what the most perfect system for a developer would be was if you even touch a single key production changes and updates and all of the different safeguards and all the things that were, we should be in place would be in place.

[00:25:46] Jason Warner: Now that's not a thing that you, we can do. You can't theoretically collapse the entire SDLC down to that single magical workflow, but you can think about it that way. And they all matter. There's a whole bunch of things that matter. The keyboard down here matters. What's happening in production matters.

[00:26:01] Jason Warner: Observing what's happening with the software matters. Seeing what, how the software changes is an example, so even if we're talking about AI systems, not just looking at the code, but seeing how code has changed, even historically as you're training these things, or in the future, understanding what is happening here so you can actually answer certain questions.

[00:26:18] Jason Warner: All of that stuff matters. And so again, this is why we're focused on the domain. So when, and I get very obsessed over two things here. So one is the developer experience, not just the answer, being intelligent, answering the questions, but also if we have that kinetic chain, what can we do for developers in the future?

[00:26:33] Jason Warner: How can we make things better, faster, simpler, smooth all of the rough edges around the industry.

[00:26:37] Guy Podjarny: And I think all of this unique data is, sounds like it can be very very powerful

[00:26:45] Guy Podjarny: and you're seeing it and literally betting the company on it and you know I can absolutely see how that gives you insights that really makes you better, but I think your case is two fold, one is that there is unique data that I think that if you were to bother and invest the time and ingenuity and all of that in, in achieving and producing this data oftentimes, then, you can actually make your models a lot better at getting a better developer experience, but also at just producing better results, producing better software on it.

[00:27:14] Guy Podjarny: The second is that even if a generic model was to bother, which is a big if, because they have a lot of other fish to fry, but even if they were to bother and invest in building all that time, the fact that would represent a smaller percentage of their total data of their total,neural network and things that the model has to factor in would lead them to produce inferior results.

[00:27:37] Jason Warner: The first one, 100 percent absolutely, and it's something we've proven out, it's the basis for what we've done our most recent fundraiser on, we've got all the data on that, yes. The second one requires a little bit of context. The answer is yes. Given the choice between two roughly equivalent, let's call them like, Decently close to each other types of things.

[00:28:00] Jason Warner: Somebody who's focused on a set of things, and then over here on the software side of the fence, always going to produce better results. That said, if we're talking about epoch differences, if we're talking about the difference between a GPT 2 and a GPT 5, it's very difficult to make up a gap between a GPT 2 with specific orientation to a GPT 5 general purpose.

[00:28:19] Jason Warner: So it's, this is opening eyes entire thesis, which is we don't care about anything else except for scale. That is the only thing that we care about. So if I were to say, answer that question by saying, the only thing that matters is the fact that I'm focused, I would be dead wrong because I too have to keep up by increasing my core models, reasoning capabilities by increasing its neural network size and all of the different things that other people have gone through.

[00:28:44] Jason Warner: And it's the focus that gives me the also the other advantages. If, again, let's just take out to the future. If, I am roughly, or any model is roughly GPT 2 sized. Very unlikely that no matter how much time and energy and effort they focus, they'd be anywhere even close to what GPT 5 will be

[00:29:05] Guy Podjarny: Yep.

[00:29:05] Jason Warner: it does require people, even

[00:29:08] Guy Podjarny: be in the ballpark. You still

[00:29:09] Jason Warner: have to be in the ballpark.

[00:29:10] Guy Podjarny: sort of caliber to be able to produce those. And I guess, how does that, play out in models, for instance, like Llama? and we've seen a variety of others recently, especially in the open source side, although you see this in the hosted one as well, which is, you have Llama and you have Code Llama.

[00:29:25] Guy Podjarny: And I guess my, external perspective is probably a decent chunk of that is it's, take the code related subset of data and just run the training separately. So in their case, maybe they didn't give the level of investment and focus on the custom and unique data that you have, but they did avoid distractions by training a model separately, maybe on code.

[00:29:49] Guy Podjarny: Is that, am I understanding it correctly?

[00:29:51] Jason Warner: It's a little bit more nuanced than that. Just think of it as Llama and Code Llama are like similar families of things. This is more of an orientation. So the size won't be there. I would not expect that anybody at that Meta, whether the rumors that they're going to release a four and a 5 billion parameter Llama3 here soon, something like that, general purpose Llama3.

[00:30:08] Jason Warner: They'll have a Code Llama variant of that, and it'll just be some sort of, it'll be a smaller one so that people can use it in various different ways. It's probably. if I were to think about internal to what they've done, it's probably some either distillation of that to be smaller, more focused, or some earlier checkpoint or something like that.

[00:30:27] Jason Warner: And this then said, okay, from this checkpoint, what we're going to do is we're just going to apply more, focused, code data of some sort. But I honestly don't fully understand exactly what they're doing internally to this. But what I can say is use an analogy here, I think is useful, which people, cause I think people get very confused about, code specific, software specific, general purpose. And I just like to say the following typically, which is, let's say you're an athlete and you're early in your career, you're young or whatever, there's always the focus saying, don't specify your athletic career too early. Just be a general purpose athlete.

[00:31:03] Jason Warner: Just be very good. Just be very good at what you do, but at some point, if you want to be the best in the world, you're going to have to say basketball's my sport or golf is my sport because they're very different.

[00:31:14] Guy Podjarny: And they require a base level of athleticism to a degree in each one, but they're different and you can focus on them, given your own skillset.

[00:31:21] Jason Warner: I think what, the entire industry needs to wrap its head around is that applies here too, that you, because we don't live in an infinite, unbounded world, we do have to say, Hey, this is the thing that we're naturally going to be excellent at or best at. And then these other things, we might still be good.

[00:31:36] Jason Warner: You put LeBron James. Who's, the best in the world at, basketball player, on a pickleball court as an example,he's, his base level of athleticism is gonna beat a lot of people in the world. But there's gonna be pickleball specific people who are gonna beat LeBron James, who they would never have the same level of athleticism to him to a degree.

[00:31:54] Jason Warner: So it's just a, it's a weird thing for us to wrap our head around because we just assume that one system should solve all these things. It's never gonna be the case.

[00:32:01] Guy Podjarny: Yeah, no, understood. And it sounds like there was a decent amount of overlap to your GPT 2 versus GPT 5 analogy. There is significant state of the art that you still need to keep up with. You still need to do a great job at, but then the subsequent focus helps you, both in terms of the creativity of the time investment of where

[00:32:22] Guy Podjarny: the novel stuff happens, but also just the fundamental, how do you focus the system? Because you only care about this one thing, which makes a lot of sense. So maybe let's talk about another version of it, which is training on customer code, and, we talked a little bit, talked about the difference between,maybe tuning on a customer code context, maybe on the other side, actual straight out, some training runs that run on it.

[00:32:47] Guy Podjarny: Maybe for starters, tell us a little bit, what does it mean to, to tune or train on customer code, just to make sure we refer to the same thing.

[00:32:57] Jason Warner: Yeah, the way that I refer to this is basically that you're going to want to add the customer's confusing term here, but I always say customer's context to the system because you want the customer's, information to be available to the models, whether it be baked into the model or around the model.

[00:33:17] Jason Warner: And the reason why I think that's important, I find that vital is because as a programmer, you can think about this as one way. I have a tool and then I have a whole bunch of documentation around it. So if the tool is abstract from all the documentation or all the other systems, like there's a bridge that needs to be built between those, that's all effectively you're about doing is Hey, I'm gonna build that bridge.

[00:33:38] Jason Warner: And whether or not you bake it into the model or you bake it around the model or however you already think about it, but it's important that you not ignore that. Because if you ignore it, it's, an abstract tool. it's just a tool that you can apply and then you have to apply to your context.

[00:33:50] Guy Podjarny: Yep. Yep. So you want to sort of build that in. And I think that makes a lot of sense to me, even just the notion of knowing which APIs exist and what are some of the best practices.

[00:34:02] Jason Warner: I think it's more important in software by the way, than it is in a lot of other domains where I think it's also important, but it, it's more important to know what they do as an example, Hey, at X corporation, we use these libraries, or this is our standard, or this is our entire approach to SDLC or our entire approach to deployments, or here's how, configuration files look or feel, or, all that sort of stuff.

[00:34:25] Jason Warner: And we all know how it is. You walk into a corporation day one, you're like. How do we do things here?

[00:34:30] Guy Podjarny: Yeah, I think that's probably the best analogy is if you come along as a, even a very capable developer, you come in day one, you can't just write some code, you want to look at some code around you to understand

[00:34:41] Jason Warner: And project to project is different because legacy projects or even modern projects have different standards and stuff like this and it becomes like a nightmare.

[00:34:49] Guy Podjarny: And if you just go off like a bull in a china shop and do things your way, that would be frowned upon and it wouldn't be very collaborative. You wouldn't be able, wouldn't be able to work with others terribly well. But I wonder how much is that knowledge significant for the actual model operation?

[00:35:08] Guy Podjarny: So how much is it, Hey, look at my code or look at my projects code and, or my organization's code, whatever it is, and summarize out of it, the, whatever my norms, some distilled two, ten, a hundred page summary of the, this is how we write code, in this organization, and then almost introduce that into the context versus how much is it an actual,

[00:35:35] Guy Podjarny: deep understanding of the systems themselves that is embedded in the code, right? When you look at the code, you see everybody avoiding a certain service, right? You're seeing, everybody,applying a certain pattern to, I don't know, scaling, right? Or how do you, uh, shed loads, so I guess how much is it generic distilling, maybe even human interpretable and reviewable

[00:35:57] Guy Podjarny: information about this as our best practices versus needing to be actually attached to the sometimes substantial amounts of code that exist in the enterprise.

[00:36:07] Jason Warner: So let's take a second to, to bifurcate two things. So one is in, as the world sees what Poolside is building, and this will make sense too, but I always quantify things into two different modes and it's easiest to visualize it this way is, are you asking a webpage a question, or are you asking a, are you interacting with these AI systems in your IDE?

[00:36:28] Jason Warner: And there's two different kind of modes that you would expect reasonably to have inside

[00:36:32] Jason Warner: now we at Poolside have embedded this in all the different contexts you can do this. But imagine you're on the webpage, you're just saying hey, how would I do this? Or I'm trying to do this, what would happen here?

[00:36:41] Jason Warner: And in that case, what's useful is the explaining. Or the giving some sort of an answer and going back and forth and all that. So imagine that you're working inside an enterprise, and you say, like, how do we do deployments here? Or how do we do AWS deployments or GCP deployments? Explain that to me. And it's a really good back and forth.

[00:36:57] Jason Warner: It's an internally documentation system or, okay, so on the bedrock project that we were working on together or the bedrock project. How do we do this? And to compare that to the substrate project,that sort of thing, that is incredibly useful in a way that becomes specific to the context of the customer, but it has nothing to do with the code per se, it has to do with the standards and what they've done and all that, but the context matters.

[00:37:21] Jason Warner: So then the code

[00:37:22] Guy Podjarny: But naturally it would need to be able to look at things broadly. It would need to see all of Project A and all of Project B to be able to contrast that above and

[00:37:32] Jason Warner: and how do you make those associations? How does it understand how to do that? that's why it's different than just fine tuning and rag and all that sort of stuff, but baking into the model. It matters there because of the associations matter. The context from what it knows generically to the specific context matter because it's making those associations and its ability to discern and understand and reason about it.

[00:37:51] Jason Warner: Now on the code side of the fence, it's, I think it's even more obvious, which is as an example, like this is how we guard against overflows here, that sort of thing, and you can see it as it sees more of a customer's code specific data in that instance here. It's going to say, Oh,

[00:38:06] Jason Warner: this is, I've never seen this style before. I have seen this style before, but I've seen it all the time now. And so that's what they do. That's what this customer does in this orientation. And now that is before we even get into feedback loops. So a good example here might be like, let's say that you're working at a video game studio and they do the,the buffer overflow thing,

[00:38:28] Guy Podjarny: they solve the problem Some security protection. Yeah.

[00:38:30] Jason Warner: Whatever. And some model. As it's writing code for an interaction with a developer, it suggests the wrong one. Something more generic, something that was more seen in the real world. Sorry, the real world, but like the generic

[00:38:43] Guy Podjarny: In the generic world versus the choices made in this organization.

[00:38:47] Jason Warner: So what you might do then is interact with the model, say, nope, that's wrong. That's not what we do. We do it this way. Can you change it to do it this way? Then that feedback loop is incredibly important because that piece of data right there becomes an incredible, and it's not something you've trained on.

[00:39:01] Jason Warner: It's not in the corpus. per se, but it's the tuning data there that matters. And that actually, human interaction with the system matters because that becomes a very high value piece of data to update and say, okay, you know what, I can't show them this way, do it this way, because this, they've already tuned me and said that I need to go this way.

[00:39:19] Guy Podjarny: Yeah. Yeah. That makes a lot of sense. And I guess from a quantity perspective as you've seen these monstrous sort of enterprise codebases. Does it come down to a hundred thousand small decisions like this versus distilling a hundred big rules for new codes that is being generated?

[00:39:40] Jason Warner: So no corporation in the world, like we all talk about data here in the AI side of the fence and the funny, dirty little secret of data in the AI world is we all have the exact same access to data. We have the web data, the code data. We're all doing the same thing. And no matter what corporation you are in the world, or a corporation has, they can never match at all the size of that data.

[00:40:02] Jason Warner: So what you have to do is you have to understand how you're going to favor client data or customer data inside of that. You can never rely on quantity of data for that. So what would you, what somebody would need to do is say, okay, so what we're going to do is we're going to, the model looks this way, and then we're going to favor some of the interactions this way, some of the later layers this way.

[00:40:22] Jason Warner: and then what you then do is you have to further favor person, I would say preference data. So if you were to interact with a system and say,no, we, this is how we want to go do that, you have to favor, continue to favor that preference

[00:40:35] Guy Podjarny: Yeah. And then you guide those.

[00:40:36] Jason Warner: Because you're never going to have the quantity,never going to get to the quantity of data in that scenario.

[00:40:40] Guy Podjarny: Yeah. Makes sense. And I guess maybe one last question on that topic. When I had Des on the show from Intercom, he commented on how oftentimes at Intercom, with their sort of autonomous AI support agent, Fin, one of the common experiences for new customers is that they turn it on and they realize that Fin mimics the the knowledge base that they have very well, but the knowledge base is bad.

[00:41:08] Guy Podjarny: And, they have this knowledge base and there was this,informal knowledge about what is good and what is bad and what you should and you shouldn't use within the knowledge base amidst the support organization and just never really manifested in it and it made me wonder whether a code has the same thing just within the organization.

[00:41:25] Guy Podjarny: Some things that you have in your code you want to repeat, and some things that you have in your code you don't want to repeat, how does that, how do you deal with that challenge?

[00:41:35] Jason Warner: I think it's, as prevalent in code, if not more, and in fact, when we're interacting with customers, we're talking about what we do and how we do it. The most common retort from them is, what if I don't want my data to be favored in that sort of way, particularly with legacy data as an example and it's important that legacy data be in

[00:41:55] Guy Podjarny: we make mistakes and sometimes, you don't want those

[00:41:57] Jason Warner: this has been

[00:41:59] Guy Podjarny: because you had the wrong people in the organization or they're just mistakes over the time. Things have not matured well, not aged well.

[00:42:05] Jason Warner: It's always good to be able to say, well give us representative good and bad examples like show us so that we understand how we can tune that system for you, it's just like with anything in life. You can't just go in and just YOLO something like you know my least favorite valuations of these systems is somebody to go in and just like point in time just asking a single question get a single response and say this is good or bad? You just can't.

[00:42:29] Jason Warner: You have to understand each of these systems is different. And when you're interacting with a customer, it's important that you say, Okay, give us some of the positive context that you want and some of the negative. And negative is a harsh term, but what I mean is Hey, don't do it this way, or

[00:42:42] Guy Podjarny: Things you

[00:42:42] Jason Warner: you without.

[00:42:43] Guy Podjarny: you do not wanna repeat.

[00:42:44] Jason Warner: Exactly. And we all understand, and the code side of the fence, I think they have a lot of that, actually. They have more representative examples of these things.but if they don't have the negative side, because a lot of people don't keep negative examples around, the positive is also good.

[00:42:56] Jason Warner: Like here's 10 modern projects that we really want. this is the style of thing that we want to go do. Here's all of the data, but this is the style type of deal.And you can do that. This is all about customer engagement. These are not like one click setting systems. You have to think of these things as like these very large kind of complex systems.

[00:43:12] Guy Podjarny: Yeah. But what do you mean, Jason? We want magic. We want you to come along and just next, next. And it needs to work and understand all of my system. Isn't that the, the promise of AI? Sounds there's still a little bit of, expressing your preferences, involved in that. Maybe let's shift gears a little bit and talk about the evolution of, of these code models.

[00:43:30] Guy Podjarny: The general perception amidst many people that I speak to, is that, of all the domains in which LLMs can improve or AI foundation models, can improve, code is, is high up there because of the ability to validate whether you got something correct. When you compose a poem, you don't necessarily know if that poem is correct, even when you compose an image,

[00:43:54] Guy Podjarny: it's hard. Code offers you an opportunity, in concept at least, to, to, for instance, you can see if it compiled, right? You can see, but maybe you can get a lot more detailed. How do you see this? First of all, is it correct? Do you expect it the ability to generate codes to evolve faster than maybe other domains.

[00:44:14] Guy Podjarny: And, and second is how much faster is it? Is it dramatically faster or is it just, Hey, this will be like 10 percent ahead of this other domain. But all in all, it's really all about the quality of the data and, like other domains.

[00:44:29] Jason Warner: One of the core beliefs that we have at Poolside, and again, I'm going to corporate speak a little bit, I'm trying not to do that. That's more of a. I don't want to talk generic here, but one of the reasons why we did the way we've done it at Poolside is we actually believe true human level reasoning and planning will emerge in the domain of software first, because of the, some of the luxuries that we have in software.

[00:44:50] Jason Warner: And primarily that software is very close, if not as close as you can possibly get to a deterministic system with a massive corpus of seed data. So again, going back to our simulator system, our reinforcement learning via code execution back system, that allows us a set of luxuries.

[00:45:05] Jason Warner: And one of the luxuries is generate, a ton of synthetic data that we can, will be unique to us and the models will emerge and evolve better in that way. But software, inherently, is more bounded, like it's testable, it's executable, it's, it is verifiable. There's a whole bunch of things that we've not as an industry explored yet, including Poolside, we have not explored this.

[00:45:24] Jason Warner: We're a domain specific company and it's just, we're not there yet. We will get there eventually, but could you imagine a world and, marry all of these things together marry my entire history of Canonical, Heroku, GitHub, and at Poolside. But could you imagine when you're interacting with a system, you're asking it to do something, and it could pre verify that the generation or the code that you've done, one, adheres to all your STLC, but also two, is, runnable, executable, and is running for you on a system.

[00:45:48] Jason Warner: These are the promise and magic that you can have here, but it's much harder. This is a harder problem to go solve because we're talking about the entirety of software. Again, not just writing a poem or an email, which the human brain is going to interpret. We're talking about these systems.

[00:46:01] Jason Warner: So it's a harder technical problem, but it's also an easier verifiable system so that the magic you can build around this is going to be pretty, pretty incredible.

[00:46:10] Guy Podjarny: Yeah, as compared to art. I think the other aspect of it is that there might be less societal objections maybe because I think when we talk about a lot of definitely the arts. I don't know about legal or drug discovery. I guess it has its own It's just harder to find out whether you got it, but everybody's in favor of us getting it right

[00:46:31] Jason Warner: So I think there's two things here that I've personally experienced. So one is developers have preference. So you code review all of the things, right? People are going to say, I don't like this style or I don't like whatever, but you can't argue with running. You can't like, there's always an old saying in software, which is, source code wins.

[00:46:45] Jason Warner: So if you get into a debate at a theoretical level, typically the person who writes, the code is going to dominate and dictate the rest of the conversation because everyone's reacting to the code itself. So code wins at the end of the day and that you're going to keep pushing that in this case here, you know people can argue all the time over strategy differences at a legal level like this isn't even how we want to go approach this problem so I think there's some luxuries here.

[00:47:12] Jason Warner: I also think that people are more open to no matter what we think in the moment on Twitter about, developers being lost to AI systems, people are more open to advancing systems here because they're used to everything changing every five or ten years. No one writes configuration files the way they used to 10 years ago.

[00:47:30] Jason Warner: No one does X the way they used to five years ago. So we're used to this. So we're open to this kind of change.

[00:47:37] Guy Podjarny: As opposed to legal or education or these domains that have actually been quite stuck for quite a while

[00:47:45] Jason Warner: And the old trope about developer, the lazy developer tends to win too, because they're going to automate most of the systems and they're going to do this. Like this is the ultimate tool for the lazy developer in a way. And we just remember that we like to reward people who think creatively that way and really use the tools as maximum their ability instead of just having to,be a human in the loop 100 percent of the time.

[00:48:05] Guy Podjarny: Right. So this is the perfect segue indeed into the maybe futures section of the conversation, not that we haven't spoken a ton about futures over here, but maybe let's talk a little bit about the person, about the developer, and maybe about this sort of AI Native development reality, this is the future reality in which you are developing software,

[00:48:24] Guy Podjarny: in, in a way that assumes AI and powerful AI is at your fingertips. So I guess the first question I wanted to ask you is, you referred a lot to the notion of understanding the code, and it's my perception that when,any model analyzes code, it needs to decipher out of it all sorts of levels of understanding about the code. It needs to understand how it operates. Okay, how does it move bits around and what operations does it do? It needs to understand what it does, especially if it wants to interact with the prompts, like whenever adds this ad within the, checkout process, it needs to know what the checkout process is and maybe even a little bit of the why, if it wanted to,to interact and explain things in a good way.

[00:49:10] Guy Podjarny: So I guess what I was curious is when I look at all the workflows that exist today about AI assisted development, it feels like you come in with some language that is sometimes at either of these three levels, and you make an ask, and you expect the, model to understand the code, understand all of these different facets related to the request right now, do all of its voodoo and come back with a kind of proposals of here are the changes I'm going to make,

[00:49:39] Guy Podjarny: sometimes we see these sort of plans in between and then somewhere in there, sometimes you get it right, sometimes you get it wrong. you, as in the model, sometimes they get it right. But all of the conclusions are only on the action. There's never really an opportunity and we think a lot about this sort of notion of spec centric development, or it doesn't feel like in that process, there's ever an opportunity to critique these layers or to have some, I think of prompts as like interaction models.

[00:50:08] Guy Podjarny: It's my UX. It's I clicked the button. I give the prompt versus something that is more long lived. So I'm sharing more, but this pet peeve, this is am I thinking about this correctly? Does the model indeed try to decipher these different levels? Does it get represented? Should it get represented in the future in a better way?

[00:50:24] Jason Warner: I think this is part of the evolution that we're going to go through. So the way that I've always thought about these things, which is, and again, like the interactions that we see today and where I think people's heads are going to go in the future, and we see elements of this, but for the most part, you ask a model, a question, whatever it be, and then you get an output.

[00:50:42] Jason Warner: Then the output is code, or the output is an answer, right? But it could be code itself. But you don't actually know what steps the model went through. Now you can ask the model to explain itself, but it's always a post hoc, rational, you actually want to know the thought process. And again, Holy Grail, we talk about chain of thought, we talk about all these different types of things, but,

[00:50:59] Jason Warner: this is what we actually all want to see these things do in the future, because we want to see that part. That part's the interesting part, and if you think about this as how you took from the input to the output with the middle part. This is the show your work part in school, this is where all the points were.

[00:51:16] Jason Warner: You might've gotten a 10 point question. You might've gotten two for the correct answer, but it was eight for showing your work

[00:51:22] Guy Podjarny: the way. Yeah.

[00:51:23] Jason Warner: And that's what we need to get to. We need to get to the show your work part. Now, what will happen is by the time, once we get to the show your work part and everyone builds enough trust over the show your work part, we will then start to hide the show your work part and just trust the output, but right now we're actually starting to ask people to trust the output without showing your work and it's a natural thing to talk about.

[00:51:44] Jason Warner: How do I, can I trust this? Now, there, there's what you said too is, there's a lot of folks who really want to go from like PRD. hey, I want you to take a PRD. I want to explain what I want. I want it in detail. And then I want to see this thing pop out the other side of this fence. We're not there.

[00:51:59] Jason Warner: there's a long way to go from a PRD down to here. But there's intermediate steps. That we are going to need to go build now to get to this point. And that's where we are, is again, we can build these things in place now. But, we gotta make sure we under, we as an industry, I'm using the global we here, don't oversell people.

[00:52:16] Jason Warner: And unfortunately, that will happen. There will be a lot of snake oil in this. There will be a lot of snake oil.

[00:52:22] Guy Podjarny: And inconsistent one as well. Like things that sometimes they work, sometimes they're magical and sometimes

[00:52:28] Jason Warner: That's the, that's probably the more frustrating part for folks like you and I on this, we're building companies around this, which is yeah, they can work portion of the time with a lot of glue and bailing wire, but it's not going to work mostly because of these reasons,

[00:52:38] Guy Podjarny: Yeah. Which diminishes the value of getting it right and

[00:52:41] Jason Warner: Right? and the way I've always, I tried to describe this to folks is, don't look at a demo on Twitter. Don't look at a video out there on Twitter and then say, Okay, I'm gonna take my entire company, board that spaceship, and go explore space in that. Because you're just doomed for failure, that's not going to be a thing.

[00:52:58] Jason Warner: The life support systems and all the complex systems that are in place there. You have to make sure you understand the inner workings.

[00:53:03] Guy Podjarny: So I think though back to that PRD, it's not doable today, but when you think about the developer of tomorrow, is that what they're writing? Do they need to understand code? Once our understanding of code and sort of the LLMs got better at knowing it, does a developer still interact with code? If not, do they interact with that PRD?

[00:53:23] Jason Warner: Yeah, I've always said there's two types of developers in the world. And obviously that's untrue because it's always more than just two, but it's easy to quantify.

[00:53:30] Guy Podjarny: Simplistically.

[00:53:31] Jason Warner: So there's two types of developers. There's the developer that wants to build an application and everything else doesn't matter. So I'm going to solve a problem in building application.

[00:53:38] Jason Warner: I need everything else to get out of the way. Source code, the tools, all that sort of stuff. That's a thing and a means to an end. So they're not fiddling with their Neovims, they're not fiddling. Like they probably got a, very basic setup, they just really care about solving the problem, and, going off and making money and blah, blah.

[00:53:52] Jason Warner: Then there's people who love to chase the stack. They're, they love that sort of thing. Now, obviously opposite ends of the spectrum, and I'm being very generic and,distilling this down. So I think in the future, you're going to see a reinforcement of both sides of this, so the people who chase a stack trace are going to be giving massive amounts of super time because they can further chase stack traces and things that they couldn't before, or do things deeper and better and faster.

[00:54:16] Jason Warner: They'll be given a brand new set of things, but those will be, they'll, will classically interact with systems in some way,with these systems, but the people who just want to go build a thing, they may never, see source code in the future. They may get a mid journey style output, and very likely to be honest with you, a mid journey style output of running software.

[00:54:35] Jason Warner: And when something breaks, they ask the system, what broke, how do you fix it? And a couple of, fixed mechanisms are tried until one works and then they go. And a lot of software does not, like a lot of particularly low complexity software does not require the type of attention that some folks might give it over here, and we just need to be honest about that.

[00:54:55] Jason Warner: This is not low code, no code, but it's an idea behind some of these things. I don't think people are necessarily going to always have to interact with source code in the future.

[00:55:01] Guy Podjarny: Yeah. And I think they need to have an understanding of the system. I guess I think about this a fair bit, probably such a lot, although you're building the fundamental engine for this as well, which is there's the notion of how do you produce the software? So I wrote a PRD and then there's everything else around software,

[00:55:22] Guy Podjarny: that is, how do you version it? How do you package it? How do you? How do you distribute it? How do you roll it out gradually, which I guess gets into operations and things like that. And I guess what we imagine here at Tessl is something thatthat entire development process centers around some form of a spec or other kind of guidelines, maybe tests, maybe verifications, versus revolving around the code,

[00:55:51] Jason Warner: I think previously, let's break the world up into a couple of different moments in time, as an example. June of 21, when we released GitHub, Copilot beta, essentially, or I forget what we actually deemed it, it was beta or alpha, I can't remember. But every line of code before that was human endeavored, and the most important artifact was the code itself, because that was the discrete unit of interaction with these systems.

[00:56:13] Jason Warner: And since we've entered this dev led, AI assisted world where Pandora's out of the, box type of Pandora's box is open. this thing is here to stay. At some point in the future, the source code again becomes another artifact, but it's not only the most important artifact because it's the intentions.

[00:56:30] Jason Warner: So just like all the software has always been, what was this code supposed to do? And so the code itself, people have always said writing the code is interesting, but reading the code, explaining the code, understanding the code, is actually where a lot of developers spend their time interacting with the system.

[00:56:46] Jason Warner: So I'm a believer in the fact that source code is still the most important thing. In the future, are we going to be able to get away with it? No, I think source code is always going to be around, but developers might not interact with the source code the same way that, I myself don't have to interact with,

[00:56:58] Jason Warner: YAML, the way I had to just even five years ago, there's systems that allow me to navigate that in ways I didn't have to before, or I couldn't before, sorry.

[00:57:06] Guy Podjarny: Yeah. Interact with it at a, at a higher level of abstraction. Yeah. And either way, it will be a very interesting journey. That's for sure.

[00:57:12] Jason Warner: Yeah, and the way I've always said this, people ask me about Heroku actually quite a bit, because we had a grand plan at Heroku back, because back before the acquisition, post the acquisition, all that. And Heroku is job not

[00:57:23] Guy Podjarny: acquisition by a Salesforce,

[00:57:24] Jason Warner: at Salesforce,

[00:57:25] Guy Podjarny: it, a good, but 15 years ago, give or take.

[00:57:29] Jason Warner: At some point, yeah, wow.

[00:57:30] Guy Podjarny: Yeah. Been a while. We're old, Jason.

[00:57:33] Jason Warner: But Heroku is effectively also job not done. It's not finished yet, because Heroku set at the highest level. It was like, hey, we're going to make software development deployment super easy. Git push Heroku master. It's going to be effing magic. You're going to love this thing. But Heroku, particularly as post acquisition, we'd always wanted to start escaping, hatching out.

[00:57:52] Jason Warner: Because this is what happens, 80 percent of the time git push Heroku master is going to be great and you never have to think about it. But some portion of the time you got to escape hatch out and do something a little bit more complex or bespoke or whatever. And then some portion of that time, still de minimis more, you're going to have to escape hatch out.

[00:58:07] Jason Warner: And the point is, these systems need to look like this. All of these, even AI systems, we can talk about base case and then edge case. And base case will be like, most of them are just going to work at some point. Like the specs are going to work. You're going to find a certain X, like a spec, and you're going to be able to deploy the code from the AI systems.

[00:58:23] Jason Warner: And you're not going to have to think about the code, but at some point you're going to start having to escape, hatch out, and do more esoteric, more complex, more bespoke things. That they need to support that. If they don't support that, then we're in trouble, because then you're bound into a box that you don't understand.

[00:58:36] Guy Podjarny: Yep. Yep. And then I guess we don't even know what to ask for at the moment. because we're all

[00:58:40] Jason Warner: And this is what I think developers are going to be really doing in the future, is understanding how to ask the questions, again, I don't think a developer of the future is ever going to be able to get away from the fundamentals of software development, because they have to be able to understand the systems here.

[00:58:53] Jason Warner: It's the same way as you say, a hardware engineer always has to understand how to build these systems, even if they're not focused on that aspect of the system. They have to understand the basics of the systems just because.

[00:59:02] Jason Warner: Yep. Yep. Makes sense. We think a lot about a Developer either going down the route of the Product Manager, as we discussed before, or going down the route of the Architect. of having more, more understanding of how the moving parts interact, even if they have less of a need to, and maybe actually to an extent, they are not too far apart from correlating to your two,No, I think that's actually, I've always joked with people because I've, I've said to myself, I'm, I was a very average developer, the way my brain worked, it was the lines of code were difficult for me to see, but I could see the packages and I could see the systems. So as I extracted myself out, I got much better at a technical level and as a CTO, it was much easier for me to be a very good CTO versus a very good developer, as weird as that sounds.

[00:59:46] Jason Warner: But it's just the way it worked. But I think that the systems allow. That sort of thing, so you can see yourself going PM or going architect and understanding that now there's going to be people who still miss the lines of code and they're going to want to write more of that themselves, but that's like somebody who wants to hold the brush and be the artist versus somebody who's going to use mid journey 90 percent of the time.

[01:00:04] Guy Podjarny: Yep. And there might be some room for them and they might just not be the center. Jason, and this has been fascinating. I guess if people want to track Poolside and, and know what's coming from it, cause I know you don't, you're not yet just available for them to go off and try it out.

[01:00:18] Guy Podjarny: What should they do?

[01:00:19] Jason Warner: You can follow us on Twitter @poolsideai,and then go to website poolside.ai and sign up and we'll send out updates.

[01:00:25] Guy Podjarny: Perfect. I'm looking forward to seeing the wonders that come out. Jason, thanks a lot for joining us today and sharing all the, the great wisdom, super, super interesting nuggets.

[01:00:34] Jason Warner: Thanks, guys. Thanks for having me. It's been fun, man.

[01:00:37] Guy Podjarny: And thanks everybody for tuning in. And I hope you join us for the next one.

Podcast theme music by Transistor.fm. Learn how to start a podcast here.