Live Monthly: News on DeepSeek, Stargate, StackBlitz (Bolt.new) funding, prompting, and more

Episode Description

Join hosts Simon Maple and Guy Podjarny in this insightful episode as they discuss the latest trends and innovations in AI technology. Featuring the impressive advancements by DeepSeek, this episode highlights how these new developments are affecting major players like Nvidia and altering the AI landscape. The hosts also explore the significant investments in AI infrastructure and discuss the shifting role of developers in an AI-driven world. With expert insights from industry leaders, this episode is a must-listen for anyone interested in the future of AI.

Chapters

1. [00:00:00] - Introduction to the AI Native Dev Podcast
2. [00:01:00] - DeepSeek's Game-Changing Model
3. [00:02:00] - Exploring DeepSeek's Innovations
4. [00:05:00] - Implications for the AI Industry
5. [00:08:00] - Market Reactions and Competition
6. [00:10:00] - AI Infrastructure Investments
7. [00:16:00] - The Role of Developers
8. [00:20:00] - Fine-Tuning vs. RAG Debate
9. [00:27:00] - The Future of Prompt Engineering
10. [00:36:00] - Traditional Programming Education in an AI World

The Innovations of DeepSeek

The podcast begins with a discussion on DeepSeek's new open-source model, which Simon Maple describes as "mightily exciting and even some might say game-changing." The hosts delve into the technical innovations that allow DeepSeek to train models at a significantly lower cost than traditional methods, referencing Simon's comment on the model's training cost being "sub 6 million."

Key Innovations

1. Mixture of Experts (MOE): DeepSeek employs the MOE approach, using only relevant experts for specific tasks, which Guy Podjarny explains as "dramatically reducing" the resources needed.
2. Low-Level Optimization: The company has optimized below CUDA, working directly with GPUs, a necessity due to export restrictions. Guy highlights this as a prime example of "necessity being the mother of all invention."
3. Reinforcement Learning: Instead of relying heavily on human feedback, DeepSeek uses models to evaluate and improve reasoning, which Guy notes as reducing the "amount of human feedback used."

Implications for the AI Industry

The advancements by DeepSeek have sent "shockwaves" through the market, affecting major players like Nvidia, as Simon notes with the significant drop in their market value. The discussion highlights how these developments challenge the traditional barriers and open the field to more competition.

Key Implications

1. Cost Reduction: With models being cheaper to train, the barrier to entry for new competitors is lowered, leading to increased competition.
2. Business Models: The commoditization of AI models could change the business dynamics, as Guy Podjarny points out, questioning if foundational models can remain a viable business.
3. Market Impact: The episode notes the significant market reactions, with Simon quoting a "700 billion reduction in market cap" for Nvidia.

AI Infrastructure Investment Trends

The podcast also covers the massive $500 billion investment in AI infrastructure by companies like OpenAI, Oracle, and SoftBank. Simon and Guy discuss whether this investment is justified in light of recent advancements that reduce the cost of model training.

Discussion Highlights

- Geopolitical Dynamics: Guy Podjarny mentions the interesting geopolitical implications, with announcements made from the White House and the involvement of various international players.
- Continued Relevance: Despite cost reductions, the need for significant infrastructure investment remains, as computing power is still a bottleneck.

The Role of Developers in the Age of AI

A significant portion of the episode is dedicated to discussing the evolving role of developers with the rise of AI. The conversation touches on whether traditional programming education is still relevant and how developers can adapt to these changes.

Full Script
‍

Simon Maple: [00:00:00\] You're listening to the AI Native Dev brought to you by Tessl.

That oh so familiar music once again, Guypo. It's now in my dreams. It's Yeah, first thing you think about when you wake up, first thing you hear. Guy, it's that time again. It's that monthly. And this is our first time of the month. You're saying it's the time of the month. It's the first month.

Explains why you're so antsy, like a little bit. That's every week and that's most days. I didn't want to go there. So yeah, we've had a lot of news. I'm going to jump straight into the news and then cover some of the some of the hot topics that were on the podcast this month.

But where do we start guy? What's the hottest news right now that if you scroll through Twitter? Or LinkedIn. You don't have to go far before

Guy Podjarny: seeing somewhere a near China. Maybe India. I don't think I've ever spoken like a true European [00:01:00\] on the about the same yeah, clearly DeepSeek, that happened a little bit of news mightily exciting and even some might say game changing new model, new open source model from from DeepSeek, from that startup.

Simon Maple: And there's a few things that are very interesting. The shift speed at which you can pull responses from DeepSeek. How competitive it is, the quality of it in comparison to the OpenAI models. And I would say that how cheap it is to actually train when you think about the, the pre training that is required.

Some of the numbers 5\. 6 million to train. There was a with an asterix. There was a tweet I saw with Sam Altman in a car, in some fast car. And it's this car cost more than the DeepSeek model. Yeah, exactly.

Guy Podjarny: So there's a bunch of a whole bunch. Maybe we unravel a little bit of sort of the innovations and the and then we go a little bit to the implications of it, but just quickly enumerate.

And this is easily, one or multiple episodes, on it. And maybe we do a. a deep dive on it actually.

Simon Maple: I think, yeah, we're looking at thinking about a deep dive or an AMA or something like that. So let us know. Because there's a lot on it.

Guy Podjarny: And [00:02:00\] if you have questions, we're going to cover some of this, but if you're listening and you have questions or disagree with some of the things that we say do share that in the chat.

So a few things that happened, one is indeed, maybe I'll start from the cost. And they have been able to train this model at a cost of sub 6 million. A little bit of an asterisk where after the lab was prepared, but still has been quite vetted. I've spoken to quite a few people about this, and it feels quite logical.

The way they've done it is a sequence of innovations, really. One is that they, and a bunch of them have actually published before, but the penny didn't drop until this one one is this notion of a mixture of experts and MOE. And so what they do is there's a bunch of experts within the model. And every time that they train or iterate on some piece, they don't use all the parameters.

They only use the slice that belongs to that expert or set of experts, or maybe a quarter of them. It's saying when I'm coding, I don't need my biology experts to be there with me, right? It's an element of what is needed, what is important to be able to do that. So that dramatically reduces that they've done some very technical super valuable innovations around [00:03:00\] the low level optimization to their sort of key value cache and how do they deal with memory that eventually deals with larger context windows in a much cheaper fashion.

It gets mighty technical and to achieve that, they've actually gone below CUDA. CUDA is the development environment when you build on top of NVIDIA and it's a very dominant environment. And they've gone lower level, at that then than that and really worked on the GPUs because they had to work with the age 800 and not the age 100\.

That that Nvidia produces because of export restrictions from the US . So interesting. Like the in invention is the yeah. Or necessity is the mother of all invention.

Yeah.

And so they figured out a way to do that. I think those are the the primary ones with those innovations.

They've been able to train that. And then this the last bit that I would mention is that they've done a lot less. This is more for the reasoning parts. All of those are like the V2, V3 innovations they've done. And then the last bit is that they've actually reduced substantially the amount of human feedback that they've used.

And instead of iterating on human feedback, they've used models, the [00:04:00\] evaluation for models with reinforcement learning, to be able to train the reasoning model of it. And they've done some HF at the end. They released something called R10, which is the more raw form of it. And then R1, the reasoning model that just came out, that kind of caused all this commotion does use a little bit more tuning by human feedback.

To be able to produce results that are more otherwise it mixed languages. They did all sorts of funky things for their statements. And so a whole bunch of those, I probably didn't even number all of them that came down into this sort of new model that is an open weights model that has been published.

So a lot of like impressive stuff. And just to say, there's been a bunch of doubters. No, they didn't really cost that much. They didn't really have those inventions from all I could tell. A lot of this is like truly innovative and fine maybe they don't precisely disclose the number.

Maybe that 5\. 6 million is a is a little bit after some preparation. But these are still orders of magnitudes of improvements that append the resources required and the approaches taken to build modern models and how reproducible is this?

Simon Maple: Obviously there's a ton of [00:05:00\] things that the DeepSeek folks have been very innovative about.

Like you say, sometimes due to necessity in terms of what they're restricted to. Are these the kind of things that OpenAI and other models can actually say, Oh, yeah, that's great. What if we did that? Or is there like IP restrictions? Or can they actually start?

Guy Podjarny: So it seems like a lot of it is very reproducible.

A lot of it is core concepts. It changes a little bit of what it is that you might have invested in. So probably the CUDA versus lower level is a good example of that. And I don't know how NVIDIA would respond. Would they actually enhance CUDA to allow it to do some of these lower level things? I don't have enough sort of depth in the precise difference to say, but at the moment they have to code it differently.

But I think, generally, a lot of these things are are absolutely reproducible. People have already been saying so or doing so. I think Hugging Face said something like there's already 500 further refinements building on top of it, fine tuning it, or doing post training on it to train it further.

And yeah, I think everybody will be embracing a bunch of these, and I think what's important to say here, [00:06:00\] though, that it appends the primary drivers of what makes someone a leader in the space, right?

The interesting thing is just what are the drivers for training a good model? Suddenly if you can train a model for 6 million, suddenly you don't need the ridiculous kind of cash kind of coffers that these big, the Anthropic, the OpenAI, that the cloud giants have to be able to train a model.

And so that's a very big deal. It implies more competition would kick in. And would be able to build models or tune models and it takes away a lot of the moat, maybe that some of the giants have, two is the inference actually running it is faster and cheaper by substantial amounts on. As a result of that, you can use the models in many more places. And so you can introduce features, AI features that might be like super cheap, like the raise, the commoditization of how much value, how much can you charge really for an AI call will drop as well. And so between the fact they're gonna be more competitors and the business model of an AI model on its [00:07:00\] own will diminish substantially.

You're left with a very different business model of who is going to win. And then maybe, I don't know how much lesser or not this is, but the reliance on reinforcement learning from human feedback then that, that reliance, if you've built, if you're amazing at doing stuff with human feedback, reinforcement learning, then you, that might not come into play.

Like you probably still need some of that to, have the language model answer in one language, to a human. Or have Claude's personality. But I think it just casts into doubt all sorts of existing forces in the industry, such as how much compute would NVIDIA sell, such as, could the giant models retain their leadership?

And in general, could building a foundational model be a viable business? Yeah. Because if you can't really charge for it. So I think it might be that the answer to all of those is back to where it was before. But at the moment, it really puts all of those into question, and I think it's quite likely that it would disrupt this industry dramatically.

Simon Maple: Absolutely. And you can see that from the shock waves that [00:08:00\] sent through the market. Just some numbers here S\&P 500 fell by 1\. 4%. The NASDAQ dropped by 2\. 3%. Nvidia dropped by 16 percent and I saw some stats actually from the RedMonk folks on a blog that they created, I think it was Steve O'Grady, Steven O'Grady.

He said in video was worth around 3\. 6 trillion and that's right, 3\. 6 trillion. Is that right? That sounds right. Which went down to 2\. 9 trillion. So 700 billion reduction in market cap,

Guy Podjarny: yeah. Ridiculous. And you can claim the 3\. 6 was somewhat ridiculous.

Simon Maple: Yeah, that's true. That's fair.

Guy Podjarny: I think it's the assumption like all of these are multiples based on the durability of growth and revenues.

Simon Maple: And as soon as the doubt that's it starts dropping now. We'll talk about this deeper. I think we need to have a session dedicated.

Guy Podjarny: We should have a deeper one and I think there's a lot to do it.

What I would leave people with is it's not just another model that came out and it might really shake up who are the movers and shakers, who [00:09:00\] are the key players in the space and where is it that you can use AI and all sorts of use cases for AI for LLMs in which you want to run 1, 000 different. alternatives, for a thing you're trying to do, maybe that becomes viable.

So I think it'll be very interesting. And that's without even opening the sort of China versus U. S. kind of ego hit or the chip restrictions. Such a massive topic. Super interesting. Took up my weekend, though.

Simon Maple: Before we move on, Guy, let me throw you back. I don't know if you can remember this, but almost around 20 days ago we had a 2025 predictions and one of your predictions was there won't be a single model that runs away with it.

There'll be multi models in 2025\. I would still agree with that. Actually.

Guy Podjarny: Yeah. It's still true. It's just they changed now who that model is.

Simon Maple: More dramatic fashion.

Guy Podjarny: Yeah. I would say it's even more true now. Excellent. Excellent.

Simon Maple: And it's good to see a model coming out of nowhere.

Actually really shaking. This isn't Claude with a very innovative next step. This is a new model coming out of nowhere. That's really shaking up.

Guy Podjarny: And we're still [00:10:00\] running our evaluation, so we don't have the numbers. But generally the consensus is that R1 doesn't surpass o1. It roughly matches it.

But it roughly matches it with a lot less resources, and an open one, and runs faster, and so with a bunch of advantages. And we'll see about O3.

Simon Maple: Cool. The next piece, which is talking about ridiculous amounts of money, $500 billion announced as investment through from a number of different groups. OpenAI, Oracle, SoftBank from Japan MGX tech investment arm of the UAE government for this project, Stargate.

So we've got Stargate, we've got DeepSeek. These all starting to sound like hollywood style movie right?

Guy Podjarny: I wouldn't put it past them to create a holiday movie on these things on it. I didn't really can Stargate. I didn't mean, Stargate's a great thing. It is. So star listeners like,

Simon Maple: Oh yeah. So stargate 500 billion. And this is for AI infrastructure, which when we actually see now how cheaply models can be like a bill and things like that, is that actually a [00:11:00\] realistic amount? What's your feeling on this?

Guy Podjarny: Yeah. First just outline what happened. So open AI combined with Oracle and SoftBank announced this sort of big infrastructure.

I

Simon Maple: think it's OpenAI and SoftBank are leading it, I think.

Guy Podjarny: Correct. Yeah. And you've got or maybe Oracle. Anyways, there are the players in there and then, Microsoft is involved a little bit, there's a bunch of other players.

Simon Maple: Interesting no, no Tesla, Elon Musk, no Elon Musk. Yeah, I think that's surprising.

I

Guy Podjarny: gotta say, I don't wanna get political here, but like probably the most interesting thing is like the dynamic between Trump. They've announced this from the White House, although it doesn't appear like Trump had really much to do it with it. But they gave him a lot of credits. Yeah. Removing AI regulations maybe to do this.

So it, interesting, on it and the relationship with Musk. Let's not get political, but that was an interesting aspect of it. Otherwise I think, big investment in infrastructure and that it's interesting for OpenAI to be relevant here. I don't think this is moots, like I think the perception right now, the understanding in the world of AI is that compute is still a barrier.

And so even if you improve it by a dramatic amount by order of magnitude [00:12:00\] it's still probably true that power that that, chip creation and volume that sort of big data centers are still a kinda bottleneck. . And then, maybe I'm not entirely sure what they capture into AI infrastructure But at the very least it will be power and GPU creation, right? . And factories. And so I don't think it's a waste on it. I think mostly, again, a bunch of a battle of the giants and a little bit interesting geopolitically to see who the player is.

Simon Maple: Yeah. Yeah.

Interesting piece of news and I think, yeah top, yeah, indeed.

Guy Podjarny: Yeah. And probably more interesting than open AI's other announcement of the operator of the operator, AI agent.

Simon Maple: Yeah. This is an interesting one. It's still research preview, but it reminds me a little bit of the, was it Cursor, that did the ability to pretty much just run through your PC and look at your screen and start learning things from your screen?

And I think it's interacting. I think it's Anthropic that did it. It's interesting. Cursor has that, yeah. A number of times I keep getting Claude and Cursor, all these C

Guy Podjarny: A I entities. And so yeah, Anthropic has that I think Gemini has a version of it as well. Yeah, I found that to be a little [00:13:00\] bit of a me too element.

They get better results. I frankly don't have a lot of benefits there. I think what's really interesting is this probably. Really it's bad news, all of these investments for the UI paths of the world and the ones that are just trying to do this robotic process automation because the ability to interact with the screen as a human would is very valuable and it is a scary agent capability. A lot of people are boasting it, I'm still a security person in heart. I feel sure, just unleash this probabilistic AI agent on your computer and just allow them to click whatever button that I don't think I'm going to be doing that.

You'd run to get a tea, if you needed a cup of tea, you'd run and sprint back, wouldn't you, just in case. Post post doughnuts people's lacks this a habit we have when someone forgets the to lock their screen write doughnuts on the screen can be a hidden agents that whenever they leave their screen unlocked, they type doughnut into their slack.

Simon Maple: At least you only have to buy doughnuts in that case. Who knows what would happen? Indeed. Yeah.

Guy Podjarny: Who knows? The the other commitments you would have made. Yeah, exactly.

Simon Maple: So let's wrap up with some great news from [00:14:00\] Slack from Stack Blitz in fact. And then we'll cover a question from Charchris.

So our friends at StackBlitz raised 105 million in a Series B. Eric Simmons provided some stats based on that. Eric is the CEO. He actually gave a a session at AI Native Development Conference. So if you're interested to learn more about Bolt, we had a session on that.

So feel free to, to check out that that session on the AI native. They've seen zero to 20 million in ARR in just two months. That's amazing, isn't it? That would normally take a company and if they did that in a year, that would be amazing.

Guy Podjarny: But I'll add a bit of an asterisk, which is as far as I understand.

They call it ARR, but they're slightly abusing the term there. People have been coming in and buying in a given month, a whole bunch of. credits and those credits amounts to if you multiply them by 12, 20 million a year, which is still a very impressive that whatever it is, a two and a half, so three million, I don't do that too.

It's shy of 2 million. My math here is failing me. 1\. 67 million a dollar in a given month. So that's still a lot to spend and especially going from zero. Yeah. I [00:15:00\] would not quite equate it to a durable, renewable subscription revenue that ARR implies, but still really great.

And clearly also driven drove the investment which is well earned. Bolt is really fun and yeah happy for the team there.

Simon Maple: And I have a 2 million registered users, which is very impressive. That is very impressive. Yeah. I'm going to quickly jump back before we jump into some of the topics that we discussed on the podcast.

I just want to quickly cover a question from Charchris, which I know you love talking about motes in projects and companies. And Charchris comment is. Where will the moat exist for these giants if entrants are undercutting them so much? It's still crazy to me how much better anecdotally Claude is than other models are for coding and for other tasks. Is the moat going to be around the quality of human feedback in RLHF?

It will be interesting to see how the buyer picks their foundational model in future if they pick Yeah, a single one specific.

Guy Podjarny: Yeah, I think it's it's really interesting. And I think maybe the question right now, especially with the sort of massive dollars being spent into these AI [00:16:00\] foundational companies.

I think we should separate, especially with Claude, the underlying models capabilities from the app player. What I would say is the underlying model capabilities. I do think the DeepSeek has definitely shaken that up and and up until if you had asked this a week ago, probably the answer would be very much on scale and compute and access to those and it felt like new entrants cannot come in.

Now it's much more cast into doubt. And so I think it's very hard to think about the foundational model sort of mode. And especially when you think of how much it can feed a company, right? Can they do it? And so I think it becomes even more commoditized or it becomes commoditized even faster.

I think it's different to think about the app player. When you think about Claude, I believe, and this is an opinion that Claude's. advantage at the moment. There's the strength of the Anthropic model, which, for us, for instance, we seen the stats is like just edging 4o, in some temporary stats we have.

And definitely is a compelling model. But then I think the success [00:17:00\] of Claude oftentimes comes to has to do with with their system prompt with their choice of tone of voice, with how it works that way, which is more of an app preference. And I think more impacted by human feedback and probably product sensibilities.

So it's hard for me to believe that there isn't some amount of data sort of cycle when you see what people did and didn't like, which is your own unique data that OpenAI, Anthropic and those that are in power right now are able to harvest. What we don't know is what percentage advantage would that give, right?

If you've got some sort of math innovation, does that outdo all of that extra data if you have some novel paths. So I guess what I would say is really a good question. My guess infrastructure like foundation models are infrastructure and they will generally get commoditized and eventually go the route of being a part of an infrastructure stack and they will have innovations that will be exceptions that will maybe be specialized model, but a lot of them will end up that path and I think a lot of the value capture will be in the app layer.

That's [00:18:00\] definitely our bet at Tessl, in which we're not looking to build a model. We're looking to build a development platform that assumes generation is available and is strong. And I think you don't pick a model. Like I think you want to, you can lean into a model when it delivers the results as you would to any sort of partner and vendor, but I think committing your business to thinking that one model will outdo to the others.

As I said, in the predictions, I think will be a mistake.

Simon Maple: Yeah. It must be high stress when you're working for a model creator that's constantly fighting to either. If they are ahead, they're waiting for someone else to, jump over them. And it's just, it's constant.

Guy Podjarny: They capture a huge value, but they're also losing huge money in the process.

Maybe the flip side of that is they're now able to continue, at least for a while, capture that value and maybe dramatically reduce their costs. So they might actually become profitable much faster, but that's, it's questionable whether that is durable profitability.

Simon Maple: Yeah. Amazing. Thank you very much Charchris.

And Charchris is from old Jersey. I like that. I like that phrasing old Jersey rather than New Jersey. Okay, let's jump into some of the sessions that we [00:19:00\] have, we had some great people on the podcast this month we had we started off with Macey Baker, Macey Bakes then we moved on to, I think it was a Dion Dion session next with Quinn Slack CEO of Then we had we had Hannah Foxwell, who I chatted with about a great DevOps and now AI leader as well.

I'm talking about the change in the development organization that and how we can learn from the DevOps culture change that we of course went through. And then today we have Guy Gur-Ari who was talking about a number of things, but included who is the He's the co founder of Augment,

Guy Podjarny: I'm not actually sure what his title is, I think he's heading up research.

Okay, there we go. Yeah, do you know, as soon as I started that, I'm like, oh crap, what is he? He just went with co founder. Yeah, absolutely. Co founder and very important person at Augment. If we say He leads the research arm, title aside, he we say

Simon Maple: co founder confidently, it'll just turn into that.

Guy Podjarny: You just shouldn't have opened that. It's co founder of Augment, just let it flow with it.

Simon Maple: So Let's move on. So one of the things that I'll call him [00:20:00\] Guy, Guy G, one of the, one of the things that Guy G talked about was fine tuning versus RAG. And it's actually interesting because Simon Last had a lot to say on fine tuning.

As well, one of the quotes from guy was the more you think about it, if we saw large quality games from fine tuning, I'd say this, the story would be different because we didn't see those. We felt RAG is just a much better fit from what we were trying to do. More and more people seem to be moving, not moving away necessarily, but investing in other spaces rather than fine tuning.

Guy Podjarny: Yeah, and I think first of all, if you haven't caught the episode, you absolutely should. It's a brilliant episode with Guy. On it. A fun anecdote, if you don't listen to it, is Guy, and I actually were in the same group ages ago too many years to name, with the gray on it with there were four guys named Guy, and it's part of the reason I go by Guypo and it was GuyGur but he managed to shake that off.

And it's just Guy . And I stayed with Guypo as the yeah.

Simon Maple: There were five GuyGur's though, in that group,

Guy Podjarny: no, there are four Guys, four

Simon Maple: or five GuyGurs cause a Geiger counter told me.

There we go.

Guy Podjarny: So we're going to move on. We're going to move on. It's it's this brilliant time [00:21:00\] moment.

So anyways, you should very much check that out. Yeah, brilliant insights. And in general Guy had a lot to say for context a little bit Augment is a coding assistant or in that family.

And it specializes or sort of its primary claim to fame is in its ability to understand existing code bases and learn them. And so we had a conversation about how do you learn a code base effectively? And if you've been following the the podcast, you might've listened to Jason Warner a bunch of months ago talking about how they train on a customer's code base.

And that gives them a bunch of advantages to his description about but understanding the code base and putting it inside, which You know, sounds reasonable. Guy challenged that, what kind of when he did their analysis they felt that fine tuning, they've tried to fine tune on it and they felt like it gives them incremental improvement over the original kind of model or the untrained model in terms of adhering to the code base that practices, right? Or mimicking what is happening in the code base. I thought that was really interesting. I think it's interesting in [00:22:00\] two ways. And sorry, I should point out that he said they're seeing a lot less impact from fine tuning as compared to RAG in which they index and chunk the the code base and they pull in, simplifying here the right bit of, relevant code from your system to, to the context at that sort of point in time, and so they can complete the code correctly. And it's really interesting. It's interesting to contrast these two approaches. I think one is, two brilliant people saying opposite things is always interesting. Yeah. I do want to point out to that poolside, again this is, as I understand it, they're training a model from scratch. So they're training the model from scratch and then they're adding the fine tuning. While Augment takes an existing open weights model. And they are doing post training on it. They're giving it reinforcement learning and fine tuning and making it a better coding model.

And then they're putting RAG on top of that. And it's a slightly different path. I don't know I, I found it interesting. Logically, it's easy to think, Okay, if you know a lot a million things. And I came along and I told you a hundred more things and they just get blended [00:23:00\] into the what, and you're probably going to act on the million things more than a hundred and you bring that in.

So if you use that equation, it makes sense that you would see RAG more relevant because in RAG I actually am giving you an exercise and I'm giving you the specific relevant context, from the textbook right now.

Simon Maple: Do you see that having a long lifetime in AI or do you think at some point, models will be able to accept much larger pieces of context and recognize the importance and the value for which piece of context is important without necessarily needing to say I can tell from my RAG that, these are the important pieces.

I'll add them into my context.

Guy Podjarny: I don't know. I feel like my other analogy, when you think about the post, training. I don't know. Okay. I find I'm reluctant to say fine tuning. Yeah. I think training on a code base can be done with fine tuning, more correct questions and answers and can be done with reinforcement learning in which the model generate something and is basically being told.

Is that correct? Is that not correct? Maybe [00:24:00\] you wouldn't without more information. And if we think about all of that is post training, this is what can you do in post training? I feel like that's more Hey, someone came on to the company and they went through an onboarding process, and I think we know onboarding processes are valuable, right?

They were trained at that point, as a human would on what is the code base, what's important here, what's not important, and they got that information ahead of time. And then they approach some piece of code and they apply that learning. I think that's an effective approach.

The RAG is almost like giving you that sort of coding exercise and then someone senior coming along at the time. So it's here's a coding exercise. Here's some relevant pieces from the code base that you might want to follow on. And so I can also see how that might be sometimes better. So I think both are effective approaches.

It's hard to anticipate what is different. What I found a bit more tactically harder to argue with is that training is a complicated and slow process. And even with DeppSeek and all of that stuff and RAG is much more dynamic. And so when [00:25:00\] you're making code changes, all the time and your team is making code changes and all the time that context of what is happening in the code base changes quite dynamically and you probably can't retrain the model on a very frequent basis.

You definitely can't do that live to the code changes you're making right now. And so you have to do some amount of RAG anyway for those runtime changes. And so if I had to bet, I would say RAG feels better. RAG has all sorts of other security advantages. You can control what data is and is not allowed to be accessed.

And so cool. I think, and it's maybe like a little bit more debuggable because fine tuning is you tell the brain more. It's like you had someone in the class and that they don't know it. It's like, why don't they know the right answer? I have to go through the whole class again, see, cause I don't know precisely why they didn't catch this point while RAG is more debuggable.

So I know. And I think that's what Simon Last was feels more practical. No,

Simon Maple: I don't think the level of investment that went into the model training. that's, and the debugging to try and work out, okay, this is a black box. [00:26:00\] Something's gone wrong. How do I debug this? And the level of debugging that was required for such small ease, it was too much.

And that's

Guy Podjarny: Although like post training is getting pretty fancy and pretty good. And like a reinforcement learning is a big deal. I don't know, can I, from a math perspective, can you take the existing models that have been trained on your code and then just run the sort of the RL piece for the latest code changes on a nightly basis.

Maybe that's feasible.

Yeah.

I don't really know. So I feel like both are legitimate tools in the hands of people. And choice is good. And I don't know, like I spoke to as we're hiring, I spoke to this really smart guy building a model and he talked about how the prestige in the world of AI research has gone from pre training to post training, right?

It feels like there's still a lot of science to be unlocked and a lot of improvement there. So I think, the game is happening right now. The game is afoot in the post training world. And so it's possible that this is not a won or lost game. We'll move on to prompt [00:27:00\] engineering.

Simon Maple: Can you phrase that again? Yeah prompt engineering, I sound like Yoda if I just do it the other way around. So prompt engineering, now prompt engineering was something that was mentioned in actually in every episode that we did this year. Guy G mentioned how important prompt engineering is today.

But actually how he expects this to reduce over time, because he feels the models are going to get better naturally as a result. I guess more able to do things with less. Now Macey actually a few weeks before had a very similar thing. And I asked Macey the question, it's, we can invest in it now and we get great gains.

What's that going to be like in a year's time or two years or three years? Very similar viewpoint. She still thinks it's going to be very important in a year. This isn't going away short term, but over time. This will reduce again as models get better. And what do you think, like how do you feel about this?

I feel like certainly at Tessl, chatting with people like Macey and folks like that the amount of benefit you can get from a well written prompt, in fact, even going back many episodes where you hear the folks from Tabnine, for example, where they almost have an [00:28:00\] interpreted prompt almost.

The user writes a prompt , if they put that prompt directly to the LLM, it wouldn't provide them with a good response, but they know if this is written in a subtly different way, it would get a far better prompt on the Sourcegraph episode as well.

They talk about how they almost reuse prompt and we'll get into that in a second. But yeah, I think. The way you ask for something the level of depth you go into the context you try and provide all of that gives such a massive impact way beyond I would say anything that you know, even training can get to Training is the training is great for the detail if the prompt is already, right?

But the massive win is clearly in the prompt space today. Yeah, now the Sourcegraph one was very interesting because one of the big Sourcegraph customers is Stripe and they very publicly blogged about how Stripe use or rather share prompts, what a good prompt looks like and how it's great rather than one person who writes this specific prompt to generate tests or to write good docs for them, they want to share [00:29:00\] that and they say, here's a prompt, anyone can use it now if you want to write tests on this, just change this bit of code, just change this bit of text as to what you want the prompts to be on and the test to be on and it will prompt and everything looks good. The Sourcegraph loved that. And so what they did is they built this prompt library into their platform. And one of the lovely things that Quinn mentioned was one of the customers, 80 percent of the prompts that were generated were from this prompt library.

And that's wonderful to cause it's so reusable and the value they must get from that, not from a speed point of view of someone prompting, but actually the value they get the best responses from the LLM. And so they rely upon those prompt libraries essentially. And I love that because you could get your AI engineers, you could get your senior developers to write the best prompts for the core golden path that you want people to use and then if yeah, if you want to do other more complex things or very specific things, sure, write that by yourself, but keep our prompt library in mind and maybe alter existing prompts or whatever. But I absolutely love that.

Guy Podjarny: Yeah. [00:30:00\] I find it really interesting.

I feel like within prompt engineering there, there's an aspect of prompt engineering, which is offer the model a hundred dollars if they get it right, or tell them your life depends on it or things like that, which, I think these need to go away.

Like they're probably just, quirks in the behavior right now. And I, yeah. I'd like to believe and I do believe, that they would that they would go away. There's another aspect of prompt engineering, which is not the model can't read your mind. And a place in which you have a picture in your mind and you say Hey, Simon, write a blog post.

There are like many interpretations of a blog post and I might have in my mind a two page blog post that is aimed at, whatever, a developer audience that doesn't have deep AI expertise on it. And maybe you have a totally different picture in your head.

And so the likelihood of you just knowing what I meant is is low. So if you just basically want the mainstream, right? If you want the thing that people generally think about, right? If I say whatever, make a cup of tea, putting milk aside. There's a whole conversation around the ratio [00:31:00\] the relationship between the milk placing and the hot water placing and tea.

So maybe a bad example. Maybe not everybody wants a cup of tea doing it. But let's assume you're a normal person. And and I say, I want a cup of tea that there's only you'd assume a mug, you'd assume a certain quantity of water. So if that's indeed what you have in mind.

But that's fine. So I think that element of learning to be specific . I think stays here. And and I do love, I think Macey kept comparing, interacting with the LLM, with interacting with humans.

Yeah.

And I, I love that. I think I, I relate to it. I feel some of it is clarifying what it is that you mean , by what you say.

So that piece I think is here to stay. And then I think the third bit I found interesting is just the longevity piece of it of details like, Macey has a great trick, but it's not a trick. It's like a practice, that she's she's also applied here. We've been applying here at Tessl around this task framing.

I think a different way to think about it is that the ordering matters, like the earlier you are in the prompt, the more weights. I think you say the earlier [00:32:00\] the prompt you say it, the more impactful it would be. And I don't know that you can see models improving that you can see that maybe looking a little bit more blocks.

I think actually Sourcegraph swaps things out as well, right? I think in the previous episode we had with someone from Sourcegraph, they talked about how they swap out the prompt. And if someone to indeed something like the prompt libraries, so they might, it might be a user improvement, right?

And swap it out to to something smarter. And so I think those are examples of things that I think the system can get better. So I guess long story short is I think prompt engineering probably is a skill that we should all get better at. But I think less the sort of stand on one leg and wink just so you get the you get the result.

Yep. What I find interesting is the is the question about the different models. And if I wanted to convince you to do something, I might need to use different approaches, like cup of tea than I would if I needed to, I don't know convince Sam to do someone right now.

Community manager, a great community manager. And if if it's two individuals, you might need to make different emphasis. You might need to fill in different gaps. [00:33:00\] So it's interesting whether the models and even personas within them end up having a little bit of those.

Simon Maple: So Macey did kinda like mention that there are very, there are specifics in terms of how you prompt different models and she's actually doing a lot.

Guy Podjarny: Claude will ask you, how do you feel about that? Yeah, exactly. Yeah. Yeah.

Simon Maple: Cool. There's a quote here. I'm not sure if it's actually relevant to this section or not.

The Dion's quote. Yeah, that's true.

Guy Podjarny: So I love look, I should point out, we had so the episode with Quinn Slack was great. It's confusing a little bit. Quinn Slack is his last name. The co founder CEO of of Sourcegraph. And him and Dion were it was a bit of a love fest of they're both, Dion comes from Augment.

They both love coding systems. And so maybe I have two things to say about it. One substance and one less. I'll start with the less. I loved Dion's, at some point Dion said what if a developer was able to sit on the shoulders of the entire dev team, but not in a creepy way? I found it just like, why did he have to add?

What is the creepy way that he had in mind around sitting on the developer team's app? So we're going to have to get to the bottom of it. I've not yet had the conversation with Dion about this.

Simon Maple: I think [00:34:00\] if someone were to sit on, literally sit on the shoulders of an entire development team I'm more surprised you don't find that creepy.

So hang on, if you, let me put you in the situation, right? We have a dev team out here in I do that every day. No, come on Guy, you've made your bed, now let's lie in it. You've got an entire development team out there. You're telling me that if you sat on their shoulders, literally, physically sat on their shoulders.

Neither you or they would find it creepy. So in the podcast Yes or no, that's a yes, no, that's a yes, no answer, Guy.

Guy Podjarny: In the podcast prompt engineering that you might do when you think about what is redundant to say, right? It wasn't like without pouring water on them or, without kicking them in the face.

It was a not in a creepy way. I'm finding this conversation creepy now. It's a very it's a very interesting choice of words. Okay. That aside I hope you enjoyed that sort of tangent. I find it highly entertaining. I rewinded and listened to that sentence again. And the I think the other piece to just point out about that episode, which I thought was great and very much kind of worth listening to, is that there's constant allusions to the fact that developers like to write code and that this is very useful to how [00:35:00\] people write code today.

And I think I would just add to this sort of caveat, which is, Okay, today, but what about tomorrow? Is it about the more and more code can be generated by the LLMs and you would prompt easier, more and more easily with libraries and others, and they would produce vast amounts of code, and then the developer would basically need to review that code and sift through it and find the cases in which they didn't get it right. Thats what we are leaning on for validation..

I don't think anybody thinks that is the correct destination. And so just what I lacked a little bit with the coding assistance, just add that critique. But I think of mightily like that episode and those conversations were one of the most sort of practical, how do you look at coding assistance in the enterprise and how do you get real value today from this coding assistance with some good tips about about Sourgegraph itself and its capabilities.

Simon Maple: Absolutely. Onto a third, maybe our last section here about does traditional programming education hold up in the age of AI? And I love this. I love this question. So I always love, I [00:36:00\] really enjoyed my journey of education from university all the way through up to all my learnings at the various companies that I've been at and in the industry.

But, there are kids, not kids, sorry, I'm showing my age now. There are people, students at university and colleges asking what they need to learn. And we had some really interesting discussions. Hannah, as a great example, actually doesn't think there's actually too much different to learn things to learn.

There's more of a push on to the how to build the app. What are the most important parts of the app? She actually mentioned a really interesting thing, which talks about if you remove, what's moving, what's changing is the bottleneck. And the bottleneck is changing from development being the long term thing that you're actually having to do in a workflow or in a cycle to actually ship a feature.

And I actually. If that's fast and easy and very repeatable, then actually the biggest pain point, the biggest bottleneck will actually be on the product side. And in two ways, really, first of all not just shipping everything because then you'll just get massive application, [00:37:00\] the challenge then becomes prioritizing.

And actually the cycle of build, measure, learn. So you have to build something nice, thin part of a feature. You measure it. Is this right? Are people using it the right way? You learn from it and then you continue to build. So it's almost like that iteration is going to be the new norm rather than having to think that much further ahead.

And so it's we're almost going to be almost A, B testing a little bit in prod or something like that, do this resonate with you?

Guy Podjarny: Yeah. I I think you started the question thinking about sort of education, right? And how does it work? And degrees today, when you look at degrees today, they probably include more pieces around design and business case and user needs than degrees maybe, 15 years ago, right?

Or a decade ago, even I think the appreciation that, one is that software creation is getting easier. It has been for a long time. And two is that the winning software is no longer the threshold of who can create this application. It's the threshold of who creates the correct application and the scalable system.

And so I do think that those needs will grow. So it does [00:38:00\] resonate to me that probably more emphasis will go there. And I think we've spoken before about this sort of two, two paths, right? One aspect of it is this notion of understand the user needs, have the taste, the preferences, the, So the design, the zeitgeist right of what would work on.

That's very much towards that sort of product manager and and business need. And I think Quinn was referring to that as well and talking about how the new developer would be a much more reliant on. The importance of understanding business needs. So I think that's good. I do think there's another path which is more architectural which is more about understanding how the systems work and what are the trade offs you're making there.

And it comes back to like things that you have multiple correct answers to, but they're correct on different paths. So I do think that both of those are there I asked a Guy Gur-Ari a slightly different question. Then I asked him, would you like, maybe trying to make it tougher, say, Hey, would you recommend to your kid right now?

Would you want your kid right now to get into a computer science degree? Yeah.

[00:39:00\] Yeah.

And yeah, his answer was that his answer changes on a monthly basis. What would your answer be?

I think I'm on a no side right now. I feel like it depends on what it is that you're seeking. So if the purpose is a career in software development, I think my answer would be no.

I feel I have for two reasons. One is I indeed, I think what is being taught today in computer science is just not the right skills. Like some elements of it are valuable, but a lot of them will actually become anti patterns. And so there's if I could pick some quarter of the computer science degree, I think it would still be valuable.

And two is that I don't think computer science degrees conceptually cannot become more relevant, but I just have low confidence that the academic institutions will adapt fast enough. And so it almost feels like indeed a business degree or some form of what my guess is that a new form of degree will get created, which will be called something that is not computer science.

It will [00:40:00\] be, and I guess it's debatable whether that's university material with the whole research arm behind it, or is it just a practical type of certification that really aims at software development with the competence of AI and in this world. And my guess is that, like this is a vocational path and what you want is something that's a lot more academy style, bootcamp style type path. And so I think who would we hire for instance, in that sort of a decade from now, I think we would want to hire someone who maybe has done something in a university as in they've proven they can withstand that complexity. They have the ability to sit down and get through something hard.

They have some foundations in, I don't know, math or in business or in something like that. But then subsequently that they have the ability to create those. I think that would probably be a more desirable profile to hire for than someone who can code very well and has the deep understanding of operating system.

Simon Maple: So to summarize that to summarize your response to that question then, it's [00:41:00\] presumably you don't mind if your kids go into what a future development role looks like, which could be the more on the architect or more on the product side, but you don't feel like the education path today is actually reflecting of those future jobs.

Correct. So the skills are irrelevant in five, 10 years or whatever.

Guy Podjarny: Yeah. Or or a big, good portion of them both either be irrelevant and then it would miss a whole pile of others, for the time that they built them. I think software development would be very needed.

Yeah. Yeah. A decade in two decades in three decades.

But I think it would look very different. And so what are the core competencies and who's the best institution? What's the best path to doing it? I think probably there is no correct path right now. I think the sort of the, this was a bit of a trick question. Cause if I ask that about law today, Yeah. I think all of those would have a little bit of this questionable answer.

Yeah.

I can just say concretely is my kids are both not into programming. Yeah. I'm couldn't really gather. They're both like curious about it, but [00:42:00\] not don't feel passionate.

They're 13 and 15\. So like it's been a while, I think it's set and I think a couple years ago I was a lot more bothered by it than I am today.

Simon Maple: Actually, it's weird. My son's just picking his options at school and one of the things that he did mention was I want to do computer science And I said, Okay, why do you want to do that?

And I think a lot of that is because of what me and my wife both do is in computing. And I don't know how I felt about that. To be honest, I think I'm not sure if it was just because I know they largely teach python, but not just the language. But it's also perhaps because of exactly like you say he's years away from his first job.

How much of that is actually going to be written by the time he even goes into it.

Guy Podjarny: It will probably still be more practical than studying Latin. I agree. Or even to an extent aspects of his chemistry. I thought you were going to joke and say Java for a second.

Simon Maple: But Latin, I'll take Latin.

Guy Podjarny: Okay.

Java will become easier with AI. That's a prediction. Java will become easier to write with AI because you can actually write the substance and then have all of that [00:43:00\] 90 percent overhead be written for you with AI.

Simon Maple: Excellent. So one other thing which actually going back to the Quinn and Dion topic was one of the things that they I said, and you mentioned it was a bit of an echo chamber that wasn't it?

It was a bit of a fanboy Kool Aid drinking contest, but would we really need to code? Is an interesting question and the topic was brought up at devs like to code. Do devs like to code or do devs like to create?

Guy Podjarny: Ooh, that's a good question. It's a question for the audience, right?

Oh, there we go. I think I think devs like to code. And I think they like to create. But a coding has this video game aspect to it, right? It's like a challenge that you get and you can always level up, right? You can take on a challenge, you can take a task or something at a certain level of complexity, you can solve it and then you can do something harder and harder.

You can complete the same thing. And at some point that level gets boring. So you get bored of doing things at that level of challenge and you go higher and higher. And I think as you go higher the coding itself becomes less of the thing that is that is appealing and the problem solving becomes more [00:44:00\] so at first you, you have the tool, how to whatever wire electrical circuits is the same as like eventually you get to, to impact to logic, to higher resolution problems.

And so I think it's almost like a seniority type element. I don't know, like I, I am a mightily rusty developer and I still like to code because there's a, there's like a playing with Lego type element to it. Yeah. You don't mean you're a rust developer.

I'm not a rust developer, although with ChatGPT or whatever, it's a LLMs I can probably get by.

Yeah.

Guy Podjarny: But I think I'm just, today. I wouldn't trust myself to someone has told me my production code license has expired , like I I should not put anything in production, but coding is still fun. I wrote something, four slash with my son four or five months ago over the summer.

And, that was like a lot of fun. Yeah. But I think, carpentry is fun as well. And we still have factories, it's not the way that we do. And so I think craft almost like the hobby element of it is fun. But I think the profession will become software creation and not coding.

Yeah, we'll see. I don't know. Like a lot of people will be pissed by this. No. Yeah, probably. Yeah, I know a lot [00:45:00\] of people.

Simon Maple: But the question is, are they hanging on to the right thing? How pissed will they be when they realize Oh, actually, and they actually start finding a love for creation without the coding, are they going to actually think actually a lot of the, I get a lot of the joy from this.

And actually I can get to that joy quicker almost. What is the joy? And I think yeah, there's probably a lot of people who are maybe hung up on the wrong things. Maybe some people will absolutely love the coding. But yeah we'll see.

Guy Podjarny: We have a question though,

Simon Maple: when, yeah, I was just gonna add that in.

So yeah, more of a comment. It's more of a comment, but it's interesting 'cause it actually, I did another podcast episode just earlier today in fact. And Farhath the guy who I was chatting with talked about the fact that he can, what his path to development was, he starts in Claude and he does some prototyping in Claude.

He says he tries that. Then he throws it away. Pretty much. He understands them, the data model he wants to create, and then he goes into Cursor and starts building with understand the data model.

He built the back end and the front end and so forth. And I think this is really interesting with the POCs and the prototypes to test theories. And this is amazing. And Andy [00:46:00\] says he's not a he's not a developer, right?

Guy Podjarny: So that's I think for the podcast listeners might want to call this out to say that. What Andy says that on the topic of product versus dev bottleneck as a product manager. I'm finding it super useful to use Claude etc to generate POC and prototypes to test theories and build POCs to validate with the business. Yeah, and I think that's super relevant. And his point is, I agree as a methodology, that's really good.

But then the other aspect of it as we talk about whether developers like to code is who's a developer. And I think today coding is a bottleneck. There's a skill, a competency that you have to build towards it. To me, I find the sort of the, that basically LLMs can take away, right? They can initially, I think, probably the most immediate beneficiaries are those people that we think of as like developer adjacent product managers, security people, design people who already think about software, already think about creation and they can get with this.

And I find the best analogy is image generation as someone who can't draw to save his life. I find image generation amazing. Like I can [00:47:00\] suddenly create visuals that I could never do before. And I find that, very I'm happy about it. And I think people who are really great at drawing are much more ambivalent about it because they feel like they have this sort of unique skill that or not unique, but rare rare enough that is being commoditized, right?

That is being reduced. So I think there are many developers, but there are many more non developers and and many of those will be able to create software. And so I think that also kicks in

Simon Maple: And I think one of the other things about people ,when we had mobile apps to make so easy to make mobile apps, you had a lot of developers that weren't familiar with maybe some of the typical flows of how to create an application, how to use frameworks and things like that, and they didn't have consideration for certain things.

Whatever that is, whether it's reliability or high availability. And Andy actually asks this question, which we're gonna answer in two minutes, Guy, because that's when we, that's when we close. I'm good at short answers. Yeah, I know. I knew that, right? So Andy also says, Hey, Simon Guy, I'd love to hear your thoughts on what are new or emerging security threats that you're considering or worried about AI native applications.[00:48:00\]

And this is one problem, right? We try and build it. We can build quick. We can also build poorly. quickly as well. Yeah, there are a ton of these. What are your? Yeah, there are many concerns.

Guy Podjarny: First of all, I would say that we had a great episode with Caleb Sima earlier on the podcast, and I think we covered a bunch of those, and I highly recommend it.

He's brilliant. And we had a good chat. I probably actually was more engaged there with opinions than even in most episodes.

Simon Maple: And Liran Tal as well. There was another session with Laurent L. So very practical. So we have both of those answers.

Guy Podjarny: I would say they are real. Fundamentally, when you're building applications with AI, the primary concerns are the same as when you're building software.

So you have to think about security testing for vulnerabilities. You have to think about the libraries that you're using. You have to think about threat modeling and what are you bringing in? And the fact that you've used the coding systems or the other is the secondary piece to it.

If you're talking about LLM powered application. So it's an application that has an LLM within that and it behaves and it access it. It does open a [00:49:00\] new kind of worms that is that is a bit hard to wrangle right now. And it's especially problematic if the LLM, one pulls in data that is not supposed to be accessible to everybody.

And for instance, if it can access an HR system that your employees can then ask questions about that data then if the model knows it, generally we're prompt injection today. You can get the data out. So you have to segregate those out. And two related is the question of what tools and actions can the model create?

Because again, If the model is able to call those tools it's quite likely that it can be tricked into calling them in a nefarious fashion. And so I think there, I think the craft is still being built. There's more information in those in those podcast episodes. And I wouldn't hold back building right now. But I would say probably wise to not start from your most security sensitive applications. Try to focus or at least not the functionality that is security sensitive. Yes.

Simon Maple: Awesome. And that wasn't the alarm for the end of this [00:50:00\] podcast. But weirdly, it was very well time as well. Time.

Yeah. So thank you all for asking questions and joining us . And Guy pleasure is always super fun.

Guy Podjarny: Don't forget to check out the Guy \-Gur-Ari episode and a lot of great other episodes we have coming for you in the coming months.

Simon Maple: Absolutely. Going forward, we've got a ton more content all being edited and ready for you.

So make sure you subscribe on Apple podcast or Spotify or wherever you normally consume your podcast episodes. Or YouTube. Or YouTube, of course. Follow us on YouTube. And you'll catch all the next great content coming next month. Thanks all for tuning in and we'll see you again next time. Thank you.

‍

‍

Podcast theme music by [Transistor.fm](https://transistor.fm/?via=music). Learn how to start a [podcast](https://transistor.fm/how-to-start-a-podcast/?via=music) here.

‍