Unlocking the Full Potential of AI Assistants for Developers with Peter Guagenti from Tabnine

In this episode, Simon Maple sits down with Peter Guagenti, President and CMO of Tabnine, to discuss the transformative power of AI code assistants and how they are revolutionizing the software development lifecycle. Developers will gain valuable insights into optimizing these tools for productivity and future trends in AI-driven development.

Episode Description

In this podcast episode, Simon Maple and Peter Guagenti explore the evolution, current capabilities, and future potential of AI code assistants like Tabnine. Peter shares his extensive experience and insights into how these tools have transitioned from code completion aids to comprehensive assistants throughout the Software Development Life Cycle (SDLC). The discussion also addresses common concerns among developers, such as job security and accuracy of AI-generated code. Listeners will learn practical tips for leveraging AI tools effectively and hear predictions about the future of AI in software development. Join us as we delve into this fascinating topic with an industry expert.

Resources

• Tabnine
• ChatGPT

Chapters

[00:00:15] Introduction [00:00:52] Background on Tabnine [00:03:24] Evolution of AI Code Assistants [00:06:24] Current Capabilities [00:10:32] Addressing Skepticism [00:15:34] Practical Tips for Developers [00:19:48] The Future of AI in Software Development [00:30:42] Insights on Prompt Engineering [00:36:13] Balancing Context and Simplicity [00:41:06] Matching Hype with Reality

Full Script

[00:00:15] Simon Maple: On today's episode, we're hot on the heels of last week's Categorization of AI Tools session to talk about one of the hottest topics that developers are using with open arms, or sometimes just looking at using, or maybe showing great skepticism around. We're going to talk to you about how you can level up your usage of code completion tools, and how you can make sure that the tools you're using are really set up correctly, and everything, is available to them to provide you with the best suggestions possible. And joining me on this podcast is Peter Guagenti, president and CMO at Tabnine. Welcome Peter. Thank you, Simon. Tell us a little bit about yourself.

[00:00:52] Simon Maple: Tell us a little bit about what you do, what president and CMO means at Tabnine.

[00:00:58] Peter Guagenti: I'll, maybe I'll frame it with Tabnine first. So Tabnine is the originator of the AI code assistant category. And so Tabnine as a business actually started back in 2018. When LLMs first emerged and it was abundantly clear that these language models, if the language was highly structured and repeatable and well documented, that it was very good at them.

[00:01:18] Peter Guagenti: And what languages are more well documented and repeatable than programming languages. So a company, released the very first, both built-in LLM and an AI code assistant for Java and, operated as an independent business for many years, selling directly to developers. this was like the hidden secret.

[00:01:37] Peter Guagenti: If you were a high performing developer, this is the kind of stuff you had running in your IDE to make you a little bit faster and to eliminate some of the mundane and the day to day for you. And as the LLMs emerged and kept getting stronger and stronger, all, of a sudden, call it two and a half years ago when ChatGPT came out, AI obviously has been around for, most of us have been working on it for seven, eight years.

[00:01:57] Peter Guagenti: and then what happened was all of a sudden with ChatGPT, everybody understood what was possible, and then all of a sudden, then every CEO was asking every CIO, what are you doing? What are you doing with AI? We have to be on top of this, and so fear of missing out, they start pushing down, and, what was pretty consistently used were AI code assistants.

[00:02:14] Peter Guagenti: So I met the founders of Tabnine. I'm a long time, enterprise software guy. I used to build apps over the first 15 years of my career. And then the last 15 had been in companies like Nginx, built and scaled that business. Super proud of my work in open source. Most recently was at Cockroach Labs, behind CockroachDB, built that business up.

[00:02:33] Peter Guagenti: And as the buyer for Tabnine shifted from being the individual developer, swiping a credit card, to all of a sudden, CIOs and heads of engineering looking at this. I was able to come on, join the founders and I help with go to market strategy. I help them think through product strategy, and really just how the space is going to evolve and how we as a business adapt to support our customers.

[00:02:54] Simon Maple: Amazing. And it must've been such a rollercoaster ride the last few years, last, really probably four years or so, but over the last six years at Tabnine, you were there before it was cool, before people were really talking about it and it must've been nice to effectively have an already running established business, such that when the hype came in and everyone started wanting to jump on this bandwagon, you're already there, business as usual.

[00:03:18] Simon Maple: And it was just a case of, can we execute to this greater level of,of demand, with people trying to do that.

[00:03:24] Peter Guagenti: It's very true. And what's really remarkable is,there's like a, there's a pretty dramatic evolution of the species happening right now and it's happening in sort of three ways.

[00:03:32] Peter Guagenti: Like the models themselves are getting stronger and stronger. We're all learning how to use them better. And what you realize is the models themselves are not the answer, right? It's all of the work we do around it, the app, the UI, all the prompt engineering, then all the context we feed it and how we actually interact with the model and the learning that we all are going through,

[00:03:53] Peter Guagenti: it's an incredible clip. Like we are just, we're discovering how to get more and more out of these LLMs and more and more deliver more and more value to the user. Then, obviously as a business, the customers pulling us in, and that doesn't happen very often. I've built some great companies over the years.

[00:04:09] Peter Guagenti: I've built some great products over the years and many times you're focused on a pain and you're just really trying to get that product out there. Gartner estimated that as of the end of last year, only about 10 percent of enterprise software developers were using these tools.

[00:04:24] Peter Guagenti: But they estimate by the end of 2027, that number will be almost 80%, right? I've never seen software adopt, get adopted that fast. Never in my whole career.

[00:04:33] Simon Maple: And weirdly you think, and weirdly you think in 2027, three years away. You'd think, why are a hundred percent of people not going to be using this?

[00:04:41] Simon Maple: With the way the LLMs are going to be growing and expanding, it's who is this one in five, developers that are not using something? It's because you think, even if you're not using it today, which, yeah, there are reasons why people might want to not want to use it today, surely in three years, it's going to be so much more improved as well, because we've seen the level of growth and the level of,how the LLMs have got better and more finely tuned.

[00:05:02] Simon Maple: Yeah, it's a no brainer, right? People are going to be out of business if they're not going to be doing this.

[00:05:06] Peter Guagenti: I agree. I think I would argue if you're not using them today, you have to ask yourself why, right? I mean, there are, there's enough variety and diversity in the tool chain itself where,

[00:05:17] Peter Guagenti: you can have incredible tooling that fits your corporate standards, right? Because the only real question about two thirds of CIOs are concerned about data privacy with Gen AI. So they're very concerned about shipping their most valuable data and insight off to a third party and having it processed offsite.

[00:05:34] Peter Guagenti: And about a third are concerned about copyright and, license infringement, which also, you know, pretty fair concern given all the stuff that's unsettled copyright law with AI right now. However. with this explosion of investment we've all made in the last five years, you've got private models, you've got privately deployed versions of AI code assistants like, like Tabnine, you've got, you've got a diversity of relationships you can have around the models and what content it's trained on.

[00:05:59] Peter Guagenti: You can even train these code assistants now on your own code instead of having to worry about third parties. So I feel like there's enough diversity and these tools are incredibly powerful. Incredibly powerful. And, if you think about state of the art 18 months ago, or maybe two years ago, was simple code completion as you typed, right?

[00:06:15] Peter Guagenti: That was state of the art. And it was pretty remarkable, right? It's predicting the function you're writing. It's auto filling classes and, it's making you just work so much faster.

[00:06:24] Peter Guagenti: Yeah. But then, state of the art than even six months later, 12 months later was an assistant that you're chatting with.

[00:06:31] Peter Guagenti: Right now, the state of the art is these code assistants that are, that touch every single part of the SDLC and they know your code base and they know your standards and they're actually even interacting with you in a way that behaves more like an on boarding engineer on your team. So where are we going to be in three years?

[00:06:47] Peter Guagenti: it's, it's so hard to predict, isn't it? Yeah. Yeah. I think the, my belief on this, I've lived through a lot of these transitions, right? And I think the change for software development here is as radical as the change we went through, starting with no version control, doing everything local, and then doing everything waterfall and doing everything on a local machine compared to the way we do it today, where we've got continuous integration, continuous delivery, we haven't done an agile, we're doing everything in cloud infrastructure, doing composable infrastructure.

[00:07:16] Peter Guagenti: If you took a software engineer. In 1998 writing web applications and then you drop them in 2018 and you show them how they did it. they wouldn't even understand what was happening.

[00:07:26] Peter Guagenti: It would be so confusing. I think we're about to go through the same thing with AI enabled software development.

[00:07:31] Simon Maple: Yeah. and we mentioned if there's 80 percent of people, then. That's going to be on it in, in, in 2027, still 20 percent of people, not, there are a number of reasons, right? I guess some, if you just look today, there are going to be some people who's maybe their companies, are saying, look, we're going to put a blanket ban on, on AI tools for whatever, corporate reasons.

[00:07:50] Simon Maple: Some devs, are going to love it. Some devs aren't going to love it. I guess some devs really, those who have really adopted it and embraced it, really levels up their productivity and they see it as an assistant. Other devs, they think it's going to take their jobs. How do you reply, how do you respond to people who are like really concerned that, yes, it's an assistant today, but where's it going to be tomorrow?

[00:08:12] Peter Guagenti: Yeah, look, I think the capability of these things are getting stronger and stronger and there, there will be an evolution that occurs. We started with AI code completion, we're now at the tail end of the era of what I call AI assistance. And this is true, not just for software development, but other categories as well, but we're at the pointy end of the spear as software developers living through these, transformations.

[00:08:34] Peter Guagenti: Really, right now, we're in what I call the AI agent era, where, we've crafted the capabilities of these AI agents to accomplish very specific tasks, but with full automation. You think about, documentation, you think about testing, we have AI agents that will, you generate a test, run the test, return the results and help you understand what code needs to change.

[00:08:55] Peter Guagenti: That's a fully contained agent and we will get to a place of where eventually we have AI engineers, right? Where we just feed it requirements and then it comes back with a function or an application or componentry. And we'll do super rapid iteration on those with that. And so I think what I'd say for those who are concerned, is yeah, the reality is your job is going to change.

[00:09:19] Peter Guagenti: I think though, the concern that AI is going to take your job is it's fairly overblown, right? I think it is fairly overblown. What we actually see in the people who use AI assistance the best is it levels them up. Instead of having to be in the weeds, blocking and tackling, executing specific things, they get to think more like architects, they get to think more like designers, right?

[00:09:42] Peter Guagenti: Where they're really looking and saying, Oh, what's the construction of this application? How do I want it to interact?

[00:09:46] Peter Guagenti: and I think that happens pretty regularly with automation and almost every job function. In software development, you think about like for those of us who started our careers in the nineties, people used to write memory management in their application, right?

[00:09:59] Peter Guagenti: Does anyone listening to this podcast even understand how to do memory management? Probably not, right? we think about, when we first started building web applications, we had to do a ton of work around networking. We don't think about the network anymore, right? All of that stuff is totally resolved for us.

[00:10:13] Peter Guagenti: And I think the, this is going to be a pair of probably an order of magnitude of a bigger change than those examples, but I think they're going to be very similar, right?

[00:10:20] Peter Guagenti: Where, if you're somebody who is a systemic thinker and you're thinking about the components of an application, you think about, you have dozens or hundreds of microservices and each of those microservices need to interact with each other, you're still going to do that.

[00:10:32] Peter Guagenti: You're still going to think that way. It's just the last course that's going to be done for you by the machine.

[00:10:37] Simon Maple: Yeah, you're still an app creator, you're still creating a solution, but you're not necessarily doing that, the writing of the code, which arguably is as my co-host on this podcast, when we chatted last week, coding is still a small amount of the job, you're not as much as we love to think a developer is going to sit in their IDE for five, six hours a day, that's not always the case.

[00:10:59] Simon Maple: And there's lots of other things that need to be done around that. And a number of developers that are using code completion tools who are more skeptical, of the code completion tools, some people try it and they say, I think this is more, maybe the more senior devs who know what to expect or know what they want, in terms of implementing, something that they're trying to achieve.

[00:11:20] Simon Maple: They say okay, this, the styling's wrong, or I would do it this way. This is a more performant way. And actually they know very specifically what they want from the implementation. And if they don't get that, maybe first, second, third go, then they think this isn't a, an efficiency play for me at all.

[00:11:34] Simon Maple: It's actually more frustrating for me. Is this something you hear from,developers? Is this something? Yeah. That happens a lot,

[00:11:40] Peter Guagenti: Yeah, and I think there's a bit of truth to it, right? At the end of the day You know, if you're, if you've got very firm and specific expectations of the structure of the code and the specific uses that, reuses from existing code and things like that, then yeah, I think these tools can feel a little frustrating at times.

[00:11:58] Peter Guagenti: And I think there is a, the thing for your listeners to wrap their head around, though, is it really does come down to the tool itself then at that point, because I think a lot of people's experience with these tools were not actually using the tools, right? They were not using a code assistant. Instead, they were just going and putting in coding questions into a chatbot, right?

[00:12:17] Peter Guagenti: And, what you end up with is an ignorant genius when you do that, right? These things, it's like when you start talking to one of these chatbots about software development, it's like somebody read a book on software development, right? And so they're just gonna, they're gonna parrot back what they understand from what they read.

[00:12:31] Peter Guagenti: But that's not in the right context, right? even the most rudimentary code assistants, they're actually taking what you're asking and they're doing prompt engineering and, changes around that to make sure that what you're actually getting is useful in your use case.

[00:12:45] Simon Maple: And

[00:12:45] Peter Guagenti: then those of us who are really advanced like Tabnine and some others, we also then use a great deal of context. And that's a pretty dramatic change. If you think about, I'm a car guy, so forgive the analogy if it doesn't resonate, but if you and I, if you and I met on the street and didn't know each other and you said, Hey, I heard you're a car guy.

[00:13:01] Peter Guagenti: What car should I buy? You gotta give me a hundred answers, right? And they're all academically correct, right? But I don't know you. I don't know what your use case is. So I'm just going to give you a bunch of options. This is what consumer report says. And this is what the car magazines like. This is what Top Gear likes.

[00:13:13] Peter Guagenti: but if you're my best friend, and I know you just had your third child and you're really concerned about safety and you really care about,concern about climate change. So maybe I'm recommending a Volvo electric SUV because I know you, right? And the code assistants are no different.

[00:13:28] Peter Guagenti: If you ask how to write a function, in a language and it doesn't know anything about you, it's going to give you something that came out of an O'Reilly book, right? It's going to be something that's solid, right? But it may not be relevant. And that's where I think senior devs get frustrated.

[00:13:40] Peter Guagenti: However, change it around. now give that same code assistant full awareness of your code base. Like we, we look at 18 different things just within the IDE alone, even before we hook in through Git to your global code base. Now you ask that question and you're getting something that's very specific to you, right?

[00:13:55] Peter Guagenti: So the way we write this function is this way, right? We call this API. And that's a lot of the evolution that, that each of us are doing right now, because what we want is a hundred percent acceptance rate, right? We want you to interact with the, with the code assistant, and have it come back with something you're like, that's exactly what I meant, right?

[00:14:13] Peter Guagenti: Yeah. And we have to shape the tools to be able to do that. And I think we're, I think we're pretty dramatic. just to give you a data point on this Simon, we added local code base awareness, early this year, right? And we just what we could access within the IDE. And in doing so, our acceptance rate for unaltered code that came back went up 40%.

[00:14:33] Peter Guagenti: Just by reading what was open in the IDE. and then we added in global code based awareness. Now we're adding in non code sources of information. Then we're adding standards and practices that are documented in the code. how specific will we get? We want that a hundred percent acceptance rate.

[00:14:46] Simon Maple: And your acceptance for unaltered code there. That means that the developer is accepting the Tabnine suggestion without making any changes themselves.

[00:14:55] Peter Guagenti: Without making any changes. Exactly.

[00:14:56] Simon Maple: Wow. And that's 40%. And so I guess 40 percent that's amazing. And I guess there's two, like when we think about styling,

[00:15:02] Simon Maple: I guess there's two, almost aspects to it. There's one is the very, developers, very specific style as in,I prefer to use X versus Y and it's it's just the way that develop, we go back years with whole tabs and spaces, If we can't even agree on that, there's big style questions.

[00:15:18] Simon Maple: Then there's the question of, I want you, to use essentially the, like you say, the SDKs, the APIs that we are using for the rest of our projects. And there is a way that we write this code internally. And I think that second one is super, super important. I think the first one is actually, if it gets done this way, rather than that way, is it cosmetic?

[00:15:41] Simon Maple: versus actually something more architectural or structural. And I think that the latter piece, which is using these other pieces of our project, don't try and reinvent everything in different ways. let's architect this project, this application,this, this larger environment, and let's make sure we make good use of each of those components and pieces.

[00:15:59] Simon Maple: And so I think that's really interesting, because for me, the cosmetic piece is take it or leave it. If it gets done one way, that's fine. If it gets done another, that's equally fine. And I wonder when we think about that kind of style, is that even going to be important in years to come? Because the more we dive into the AI space, who's actually reading the code?

[00:16:19] Simon Maple: who's doing it? Is it going to be other AI agents and bots that are actually going to be doing that reading that generating the tests, the code reviews and so forth? And if that's the case, what, what value does individual style actually have? if we are, like you say, thinking more about it from that, feature by feature level, versus the individual lines of code, any thoughts on that or breaking those?

[00:16:41] Peter Guagenti: Yeah, no, I think you're,you hit a few things in there that I think are really quite interesting and worth unpacking, right? The first and foremost, I think most of the earliest generation of generative AI tools have been focused on giving, giving more and more generation, right?

[00:16:54] Peter Guagenti: More and more knowledge, able to do more and more things, but we've discovered is now we're at a point. If we really want to add value, we actually have to make them do more specific things, like we actually have to start constraining, right? If anything, we were, we've gotten into the point of where it's we could do tons and tons of things.

[00:17:09] Peter Guagenti: It kept getting smarter and more, more code could generate more things could be automatically generated. But then now we're looking at it and saying, is it the right quality? Does it meet our standards, right? is do we apply our corporate policies against that? And we think that that's the next evolution of the species here in this right is constraints, right? We call it coaching or expert guidance. we have actually a private preview right now with our enterprise customers. We have the ability for them to give us very specific standards in plain English, right? It doesn't, we can read something that was a document.

[00:17:41] Peter Guagenti: Somebody wrote and it's looking at Confluent somewhere. And then use that to actually constrain what's done and constrain it both in the IDE. So when things are formatting is a good example, we follow the Google Java coding standards. Okay, then that's how the code is structured. Yeah, that's what it looks like.

[00:17:56] Peter Guagenti: So everyone can read it, right? Or even more importantly, things like we never, we never actually talk, to this API in this way because it's a security vulnerability. So we just don't do that, right? And we can also capture that code review. So we can see it when code is being pushed out and in, in reviews, you're in the pull request and says, this violates the standards.

[00:18:16] Peter Guagenti: This violates the standards. This is what a change looks like in order to adopt it. That to me, is going to be really transformative, right? Because, you think about the process here, you want to have all these standards. It could be security, could be performance, could be readability, could be all those things.

[00:18:32] Peter Guagenti: If we're not actually evaluating the code, validating it before it goes out, then we're in trouble, right? And the best organizations do that manually today, AI can do this incredibly quickly. So that's a big one. And I think to your point around,how much is machine, is readability of this really going to matter?

[00:18:49] Peter Guagenti: I would actually argue we're going to end up in a place of where AI is actually generating a lot of this code and generating actually whole functions or whole applications. And if you really let them run and you have it optimized to meet your standards, it may not be human readable, right? And you think about what it takes to actually make code run really fast, right?

[00:19:08] Peter Guagenti: You may make some decisions in how the code is structured in order to optimize for performance. And then a human's going to struggle to look at it. So does that matter? If you have an AI generator, AI code generation happening, then you have AI code validation, which is separate, different prompts, different ways of looking at it.

[00:19:25] Peter Guagenti: Then, the AI will be able to read it, right? It'll be able to understand what's going on, so you don't have to understand it, right? What you do have to understand, and this is, I think, really critical, if we're going to get to that point where it becomes less human readable, then you need to have absolute trust that what the agents are doing follows your standards and probably is held to a higher standard than the humans are on your team today.

[00:19:48] Simon Maple: Yeah, very interesting. And I think it's actually, if you draw a parallel, um, with, with how,software development has evolved over time. Anyway, people going from an assembly language to a C compiler, maybe even to Java and actually, okay, now we're compiling this to object code, etc, etc.

We abstract ourselves away from the instruction language, and the byte code, etc. And when we think about it that way, when we think, okay, there's, this is effectively an interpretation, right? And if I'm writing some Java source code and that, that, compiles it into object code.

[00:20:19] Simon Maple: Is anyone actually looking at the byte code, looking at the load and stores, etc? And when that goes down to an assembly language, is anyone actually thinking about that? And it's it's almost like a mental, muscle to almost be comfortable with abstracting ourselves away from that and actually putting the trust in.

[00:20:34] Simon Maple: And perhaps that's one of the things like, it's that level of trust that we generally need to get better at with AI so long as we have the right guardrails and things like that going forward. One other, one other point I'd love to quickly make actually is, is one thing which has always bugged me as a developer, which is when you look at a project, as that project grows, people use different packages and different, they pull in different dependencies and things when they want to do the same task.

[00:21:04] Simon Maple: And it's we've all been there where we've seen a project with, five different JSON parsers and all this thing because people don't understand or people don't realize that other people have had to do the same thing and have already potentially pulled something in or they do realize someone's potentially pulled something in but they don't know where to look they don't know they can't find it and I think when we think about a tool like Tabnine for example that does build that context around an entire project and does use, that's going to be far more effective at recognizing when I need to perform a certain task if that task has been performed in a similar way in my project and use that same approach versus trying to effectively recreate the wheel every single time I want to do some basic core, functionality.

[00:21:51] Simon Maple: And I think that is, is a real value, which is a pain point because it grows our supply chain. it messes up our,the code in, in, in terms of, doing things in different ways, potentially having different formats and so forth, so I find that a really big value to, to clean the code up a little bit as well as talking about how readable it is.

[00:22:12] Peter Guagenti: Yeah, no, I think that's, I think that's right. And I think this is the value of what we call global code base awareness, right? So if the AI code assistant can see your entire repo or can see everything that you're doing, you can optimize for things like reuse, right? You can optimize things like,keeping your volume of code down to a minimum.

[00:22:34] Peter Guagenti: Right by calling these functions repeatedly. Yeah. Or I think even when we think about refactoring that a human pouring over an entire code base and trying to identify where you have those issues, where you've rewritten things that's a hugely time consuming task. And nobody does it, right?

[00:22:49] Peter Guagenti: Nobody does it because we're, we've got too much on our plates to build the next application or to fix something that, that has failed on us. And so this is exactly the kind of thing we want to,we call it an off, we call it an offshoring to silicone. Like we want to actually push all of those things out to the AI agents to do for us.

[00:23:09] Peter Guagenti: And refactoring and restructuring code is the kind of thing that it's very good at today. And it's only going to get stronger.

[00:23:15] Simon Maple: Offshore, offshoring to silicon is my new favorite phrase. I'm gonna, I'm gonna make sure I use that every day now.

[00:23:20] Peter Guagenti: I can't, I can't take credit for that.

[00:23:23] Peter Guagenti: That's our, CTO, Eran Yahab. He, he's been in deep learning and automation for developer tools for two decades, right? Yeah. He, he has this thing that's effectively, he calls it the fundamental theorem of, of automation, which is the amount of effort required to get a system to interact the way you want it to has to be lower than, the amount of effort would have taken just to do it yourself.

[00:23:48] Peter Guagenti: So the offering, it was always that, I could move things over to another team. I could outsource this to another place and, you know, AI. Up until just a few years ago wasn't to the point of where you could just do that and say just delegate and now you can and it's getting better and better at that degree of delegation.

[00:24:05] Simon Maple: Yeah.One thing I want to move us onto a little bit is more kind of things that we can have the listeners take away in terms of what can, how can they improve, both the likelihood they're going to be successful with code completion and also how they can change the way they interact with these tools.

[00:24:22] Simon Maple: And I want to start with, I guess when we think of code completion tools, there are many code completion tools out there, obviously, obviously you work for Tabnine, but I'd love to hear maybe from a, maybe from the Tabnine experience, actually, there are going to be lots of tools that do things differently, and there will be some better at certain things, some better at others.

[00:24:38] Simon Maple: from the Tabnine point of view, what are the core things which you've found your users have really found value in your platform, in terms of, actually making them more successful. A great example there was that 40 percent increase from the context. would you say there are certain things that, that Tabnine have learned from this?

[00:24:57] Peter Guagenti: Yeah, well, we're, I mean, I think about what these tools are capable of today and what their strengths are. They're, it's way more than code completion now, right? It's already, truly an assistant throughout the SDLC, right? We see them incredibly successful, in the obvious things of generating functions, right?

[00:25:16] Peter Guagenti: And then generating componentry as you're writing an application, but they're also incredibly good at, refactoring code, identifying errors and defects, right? And you think about where we spend most of our time. We don't spend most of our time writing new code, right? We spend most of our time actually in maintenance, right?

[00:25:31] Peter Guagenti: And so they're incredibly good at that. You think about refactoring or identifying bugs or, the sort of mundane tasks we have to go through when we're doing updates and upgrades and we have to make changes, like they're really good at that. And that's something to lean into.

[00:25:45] Peter Guagenti: And then. they're incredibly good at these sort of discrete AI agent tasks. So like Tabnine has an onboarding agent. So when you first open a new project, or let's say you're somebody who maybe worked on that project quite some time ago, and you forgot how it's structured. It will walk you through the project.

[00:26:02] Peter Guagenti: This is what it does. this is how it's structured. This is what it interacts with. These are the other systems that it talks to. These are the dependencies, all the stuff that you would have a senior engineer who works on the project every day do with you. So it's really good at that.

[00:26:13] Peter Guagenti: It's really good at things like automated test generation, automated documentation. so the, there, the guidance I'd give to your listeners is, look, these tools actually touch every part of what you do. And, like instead of going to search and go and try to find examples of how to structure a certain function or how somebody else solved a problem, you can actually just ask tab nine in the IDEIt will come back with options, and it'll actually come back with incremental prompts, like what else you should be asking. So you can do discovery. And I'd encourage to just try it, right? There's, it's a new pattern, it's a new way of working. You will discover very quickly where it is great. You'll also discover very quickly where it's maybe not as great.

[00:26:53] Peter Guagenti: You're also going to discover very quickly how you have to interact with it, right? I think as vendors, we're very focused on, I want to understand the intent of the developer, and then I want to structure the tool to respond so they don't have to think about it. They should just be able to work the way they want to work.

[00:27:08] Peter Guagenti: But there is a little bit of a difference still with these things around how you have to interact with it versus maybe how you would put in a Google search or how you would ask a peer, right? They're a little bit different behavior today. And so start small. Start incremental, try each of these individual tasks and then start pushing the boundaries and start figuring out, where they really perform for you.

[00:27:28] Peter Guagenti: And it's like having a, it's truly like having a pair of programmer today. It's like having somebody sitting next to you. You have to build that relationship, right? You have to know what it's going to be good at for you and what, how it fits the way you want to work. And I think that's a key takeaway.

[00:27:39] Peter Guagenti: If you think you can just get in and give it a, a requirements document, it's going to spit out perfection and you probably have inflated expectations.

[00:27:46] Simon Maple: Yeah. Yeah. And you mentioned the kind of like different agents and, I've seen so many good use of agents in, in, in the industry.

[00:27:53] Simon Maple: And I think they're, people are really leaning into the agents as a good way of, as a good way of performing very specific, different tasks. Um,in terms of the agents, do you use different models for the agents for different tasks, or is it, Is it, just the kind of like different backgrounds and different goals and tasks that the agents are trying to specifically do, or is it a mixture of the two?

[00:28:14] Peter Guagenti: So today we're not changing models underneath each of the agents. we, Tabnine is one of the, just a couple of who actually have switchable models in general. So we have our own models that we developed, one that is fully private and another one that is fully private and also is only trained on permissively licensed code.

[00:28:29] Peter Guagenti: So there's no security risk of, copyright infringement risk.

[00:28:32] Peter Guagenti: and then we have all the major vendors, we work with Cohere and we work with Anthropic and OpenAI so you can try it all. And what we've discovered is for the most part. The LLMs are reaching, if not commodity, at least utility, right? they're all converging towards the same alert, same level of performance, where the agents come in.

[00:28:49] Peter Guagenti: Basically the agent is trained on a discrete task. We've done all of the prompt engineering and all of the work in order to work through that task. And then we use specific context and the specific context really matters, right? Becauseit's not enough for the LLM to respond. You actually have to give it appropriate framing for your problem, right?

[00:29:07] Peter Guagenti: And that appropriate framing usually is something in how you work, something in the code base itself, and then it also needs to know what to look for, like that onboarding agent is actually doing, you, you click a button that says onboard me to this project, but it's actually doing a bunch of back and forth with the LLM and it's actually giving it specific context from within the project and it's, it's got a vector database behind it that actually has some of these key characteristics.

[00:29:30] Peter Guagenti: So there's a. There's a certain amount of sort of app construction, maybe, is how I think about it. just like you build an app today, you have to build all the logic and you have to build the data that it works with and how it responds. That's basically how these agents are constructed.

[00:29:42] Simon Maple: Yeah.

[00:29:42] Peter Guagenti: And it's really powerful, right? And I do think, though, at one point, either the models will end up being, being so similar that we won't worry about different agency, different models, or We might actually make the models a little bit smaller and more specialized for performance, right?

[00:30:00] Peter Guagenti: And in that case, then yeah, maybe we use a slightly different model for each of these things. And there's a debate around some of that as well, like what is the right way forward? And this is why we have switchable and all these other things, because we want to experiment with all of it and see what gives us the best performance.

[00:30:12] Simon Maple: Yeah, it's interesting you talk about the model size and things like that, because it's probably not going to be long before we see models on small devices, very limited,execution pair and things like that. Of course, Tabnine, actually, they offer a, you offer a, an offline mode as well, right?

[00:30:25] Simon Maple: We effectively run all that locally, as well as being able to is that something that developers use a lot or edge case?

[00:30:33] Peter Guagenti: We have a number of customers who are air gapped, right? So we tend to work with, large enterprises who really make privacy and security paramount in what they do.

[00:30:42] Peter Guagenti: And that's, banks, it's pharmaceutical companies, it's manufacturers,it's anybody who's really looking and saying,I want to control where the data goes.

[00:30:49] Peter Guagenti: And even we've gotten more into government and defense, we work with, for example, one of the, one of the, departments of energy, for engineering part of the grid and things like that, that you cannot ever make available on the open internet,

[00:31:02] Peter Guagenti: And those users, they like that, right?

[00:31:04] Peter Guagenti: And so for us, we have self contained instances of Tabnine running in their private infrastructure with its own models, and it's fully air gapped. and on top of that, we actually had created our original code completion models were on device, so we actually still have an on device model.

[00:31:20] Peter Guagenti: So if you need to actually not even have the laptop that you're working on connected to something else, we can do that, and, we work with Intel and AMD and some of these others and some of their advancements, and I think it's going to be sooner rather than later that you're actually able to run pretty sizable models locally.

[00:31:38] Peter Guagenti: And we saw this in other places, right? We saw this in, in, in other areas of compute where we thought everything would have to be on some giant machine, and you think about the, the compute power on my phone now is, greater than the supercomputers we were running in the nineties.

[00:31:52] Peter Guagenti: And so I think. this idea of being able to actually have, functionality that's happening, on device versus in your private environment versus requires cloud computing power and all of those things. It's great that we can actually tap into all of that because the customer at the end of the day needs to decide,

[00:32:09] Peter Guagenti: what is most appropriate for their use case, and as vendors, we need that degree of flexibility to support them.

[00:32:14] Simon Maple: Amazing. Amazing. It's amazing how fast it's progressing like to think, of the possibility of having this on small devices so quickly is mind blowing. So, let's talk about prompt engineering a little bit as well.

[00:32:27] Simon Maple: In terms of how users, interact with, with the model or with the tool. You're going to get, I bet, so much variation in terms of people putting too much, not enough, ambiguity in the requests. How, what advice would you give to developers who are trying to, trying to give good, or try to, be more effective in the responses that the tool is going to provide them with short, succinct messages?

[00:32:50] Peter Guagenti: Yeah, I have, I'm of mixed feelings on this because I do feel as product developers on this, we need to actually meet the user where they are.

[00:32:57] Peter Guagenti: And so like for when you talk about prompt engineering, for the most part, what we try to do is in the UX actually understand and interpret intention.

[00:33:06] Peter Guagenti: And then that's what we leverage to go and interact with both the model and with our other agents, and, the goal there is for them to not have to think about it and say, I asked that question badly. Let me ask it a different way. If you're a regular user of ChatGPT, for example,

[00:33:21] Peter Guagenti: you may actually restructure your query a bunch of different ways because it's not doing much in the way of interpretation of context. There's some context if you leave things open and you can feed it some data, but it's not like these other generative AI tools that are built for purpose, like code assistance.

[00:33:36] Peter Guagenti: And those, we really are trying to understand, if you're asking a specific type of question, you've got a specific application open, specific files open, I see errors. that have been generated in the IDE, I should be able to understand all that and say, okay, what's your, why you're asking that question?

[00:33:50] Peter Guagenti: Do you really want this?

[00:33:51] Peter Guagenti: And so that's the kind of. That's the kind of engineering that's actually happening in these generative AI tools, right? And I think there's a sort of misunderstanding that, if I just, to make a generative AI application, I just need an LLM and call the API with a couple of questions.

[00:34:03] Peter Guagenti: All of a sudden I have an application. You have a pretty mediocre application if you do that, right? And maybe an unsuccessful one. The real engineering comes around, what we call it, what we think of as three legged stool. Yes, you have the model, right? But then you have to have really good prompt and structure and good user experience to understand and intention and to shape that intention then into how you're interacting with the model.

[00:34:24] Peter Guagenti: And then the third leg of the stool is context, right? Okay. What else am I feeding into the LLM alongside those prompts to make sure that I'm getting the correct answer back? And by correct, not is it academically correct, but it's what you meant. Okay. It's what your intention was. And I think we're all working through that.

[00:34:42] Peter Guagenti: I think what I would encourage your listeners to do is if you're not getting what you want in these interactions, give it more context. use a tool like Tabnine, open the files that are relevant to that, so it can see them and it understands what's going on. Or, if you're trying to do this through a third party tool, it doesn't allow you to do that.

[00:35:00] Peter Guagenti: Copy and paste. Additional context in right? Give it some more detail and then try expanding on your query, right? Try expanding on what you're asking for, give it more detail, right? It's just like a human right. If you are, if you and I are collaborating on a project and you're looking for help, the more context you give me, the more successful I will be assisting you.

[00:35:21] Peter Guagenti: And we see the same exact thing with generative ai.

[00:35:24] Simon Maple: Yeah. and it's interesting actually, when we talk about giving it more. Like when I go to a conference and I present, for example, I know if I don't have one, if I have more than one point per slide, let's say I have three takeaway points from a slide, that'd be a really important one, a less important one, and something that's absolute bullshit, that's just a story that I made up.

[00:35:42] Simon Maple: They'll take away that story and that would be the thing that they remember from that presentation or that slide. Is there, if you provide one thing.the audience or the LLM will focus on that thing. Is there a balance between not enough too much?

[00:35:55] Peter Guagenti: Yeah, that's a great call out.

[00:35:57] Peter Guagenti: It's there, there is such thing as too much context, right? Cause then it's, then it just doesn't understand what's going on or it's just overwhelmed and it's trying to understand, okay, how do we weigh all of these things? I think. Larger and larger context windows and more and more things that are fed in what we've not seen that actually correlates with better outcomes.

[00:36:13] Peter Guagenti: What instead we've seen is the right context results in better outcomes. And, I think that's something, it's hard for someone to do individually, right? It's hard for somebody to, to apply that themselves. But I think this is where we earn our money, right? Is actually understanding that.

[00:36:32] Peter Guagenti: So like I said, in the IDE,we touched something like 18 different sources. We don't read all of it for every prompt and every agent, right? We're using what's appropriate in the agent. you're going and asking for a fix of broken code, of course, we're going to read errors, right?

[00:36:47] Peter Guagenti: At some point we'll hook into observability tools, we can see actually what's happening right in the application of production, right? I'm not going to go read documentation about something and feed that when I'm trying to fix code, right? I have to keep it to a discrete and finite thing, and it's just the way, it's just the way humans work.

[00:37:02] Peter Guagenti: Think about when you're starting a task, what are the things you want to have available to you when you're starting that task? What's appropriate for what you're trying to do? That's a lot of the engineering we're doing now is figuring out what are the most appropriate things to leverage as context.

[00:37:14] Simon Maple: Yeah, very interesting. And I think when we think about,when we look at code completion or AI, or just code generation in general, we can almost drop into this fantasy land a little bit where we say, look, I'm gonna, I'm gonna write this one liner or a short prompt, and I'm gonna have this whole application generated for me.

[00:37:33] Simon Maple: What, do you have advice in terms of, the scale at which we should be asking for things to be generated. Should we be looking big, middle, small?

[00:37:42] Peter Guagenti: it's really funny is I,we get feedback from our customers all the time, right? we have, we're relentless about this.

[00:37:47] Peter Guagenti: We have a free product. So we have over a million monthly developers on our free product. Wow. These are like heavy users. This is not, I think my. total IDE downloads or something like 70 million or something like that. But these are people who use it all the time, right? And they're constantly in the pool.

[00:38:01] Peter Guagenti: And so they're very vocal about what works and what doesn't. And we had a hiccup with somebody who was trying to do something that was basically the super structured ask. It was basically trying to output a Python app, like that, that, that processed some data and did some interpretation, and when I saw it come through, I was like, is that even possible?

[00:38:20] Peter Guagenti: Like, they're basically asking for something that is a structured set of requirements that is outputting a basically a complete functioning sort of small application. It's like a microservice and we'd had some hiccups, which is why we had heard about it, but it was actually a connection issue on their side that it wasn't working once we fixed the connection issue on their side,

[00:38:39] Peter Guagenti: then it worked and I was shocked, like I was actually shocked. And I probably shouldn't be at this point because I feel like the tools we are getting, making them smarter and smarter every day, and I think I am equally amazed, and not when I see that happen, I do think though, you don't want to start there, right?

[00:38:58] Peter Guagenti: You don't want to take a JIRA ticket, copy the whole thing and drop it in and hope that it outputs the entire functioning service. I think there are some of these discrete things that are absolutely working today. We actually have a prototype of what we call JIRA to code, where it's literally just go in and it takes a ticket and then outputs,in this case, uh,JS app, which is fully functioning with UX and everything, but those are, I think, still further apart.

[00:39:22] Peter Guagenti: So today, start with discrete tasks, start with things that, like I need this function, I need this specific. component that I'm trying to do, but then push the edges and see what it will do. And I think you might, I have been surprised. You might be surprised as well.

[00:39:36] Simon Maple: Yeah. And I think we, we talk a lot about, training, fine training and fine tuning LLMs and things like that, but a lot of it is really the other way around as well. We need to be better at using these tools to get the most out of these tools.

[00:39:50] Peter Guagenti: It's going to be, it's going to be something of a meeting in the middle where we really land, right?

[00:39:56] Peter Guagenti: Because the, like I said, I feel that we have a responsibility as product designers and developers to make sure that it does just work the way you want. This is the whole point. I started my career and I remember all the debates were, do you even need GUI, right? Like, Why aren't we just talking to the computer in command line the way the computer wants to be spoken to, right?

[00:40:14] Simon Maple: Yeah.

[00:40:15] Peter Guagenti: And I always thought that was ridiculous, these are our machines, right? We want them to behave the way we want them to behave, and I think that is on us, but I think there's going to be some changed behavior here, right? There's going to be new ways of working and just be open to that, right?

[00:40:27] Peter Guagenti: As a developer, I feel like if you've been a developer for any period of time, 10, 15, 20 years, you've already seen so much radical transformation in your career in the last 20 years. Like I've been building web apps since 94 and it feels like it's a different lifetime. And I think, yeah, AI is going to be some radical change, but we're good at that.

[00:40:47] Peter Guagenti: Like we, as a community are really good at that. We're always constantly evolving anyway, just be open to the change and open to adapting it and you will both get stronger and you will also understand what requirements we need to be, we need to be responding to as product developers as well.

[00:41:06] Simon Maple: And finally,let's take a look back at where we are today in terms of matching the hype with the actual,products and capabilities and technologies.

[00:41:16] Simon Maple: Are we at the, are we at the right level in terms of what we're saying AI is today or how far are we trailing behind?

[00:41:26] Peter Guagenti: It's a tough question, because I feel like it depends on who you talk to. I think the cheerleaders are doing what they always do. And they're probably way more hyped up than what these things are actually capable of today.

[00:41:40] Peter Guagenti: But it's going to get there quickly, can you really call them wrong? At the end of the day, I'm always. I'm effectively a sales and marketing guy. That's really what I am. I have,I've worked as an engineer. I've done enough of that, but I've always been a, I've always been a sales market guy, right?

[00:41:55] Peter Guagenti: And my peers frustrate me when I see them promising things or overblowing what the thing is actually capable of, cause I think it diminishes your credibility, right? Your job and in technology. And when you make products like these, it's just to expose the inherent truths of the product, if you do that, then people will use it.

[00:42:12] Peter Guagenti: People buy it, so I do think the hype train is a little much some days with Gen AI. But then again, we also keep matching the expectation with reality and we're very sure how long that's going to last, but it definitely is happening, and then I see the other side too. I see a lot of skeptics who are,

[00:42:30] Peter Guagenti: who are like, no, these things are garbage, they don't really work. And I think that's also not correct either, right? I think there's capability in these tools. One, one of the things that, that I was just reading is, LLMs are now effectively at the intellect of a grade schooler.

[00:42:47] Peter Guagenti: That's pretty powerful, right? That's pretty great, but it's also the intellect of a grade schooler. That's, that should frame your expectations. So there's some remarkable capability, which you should lean into, but there's still a lot of room to develop and grow, which you should acknowledge.

[00:43:00] Simon Maple: Yeah, absolutely. It's the question of would you let a grade schooler commit code into your code base? Other than that, yeah. obviously there's a difference between the smartness of of the pattern matching, the correlation, but yeah there's certainly something there that we can learn.

[00:43:14] Peter Guagenti: So you know what's really remarkable about that make that comment is I think StackOverflow a few years ago they do their normal developer studying and I remember reading you know the number of people who, they always ask when did you write your first code? When did that come and actually The highest performing developers started in grade school.

[00:43:31] Peter Guagenti: So here we are, so that grade school or that we've got as a code assistant today is going to be a hell of an engineer 20 years.

[00:43:38] Simon Maple: Looking forward to that. Amazing. Peter, it's been absolutely great fun and,very good to, to hear,your and Tabnine's opinions on code completion.

[00:43:46] Simon Maple: Thank you so much.

[00:43:47] Peter Guagenti: Thank you, Simon. Appreciate the time.

Podcast theme music by Transistor.fm. Learn how to start a podcast here.