Intercom Co-Founder on AI Autonomy, Adapting Dev to LLMs and what AI Native means: Insights from Des Traynor
In this episode, Guy Podjarny chats with Des Traynor, co-founder of Intercom, about the transformative journey of integrating AI into their customer support platform. Discover the strategic decisions, challenges, and future outlook of AI in customer service.
Episode Description
In this enlightening episode of the AI Native Dev podcast, Des shares invaluable insights into the strategic decisions and challenges faced along the way of Intercom's AI journey, highlighting the pivotal moments that shaped Intercom's AI-first approach. The discussion delves into the evolution from AI-assisted tools to autonomous AI agents, organizational changes, and the future of AI-powered customer support. Des also emphasizes the importance of up-to-date knowledge bases and disciplined adoption of AI tools. This episode is a must-listen for anyone looking to understand the impact of AI on customer service and the broader implications for development practices.
Key Resources Mentioned
- Intercom's AI-native features: Fin and Fin Copilot
Chapters
[00:00:00] Introduction and Guest Introduction
[00:03:12] Overview of Intercom and AI Integration
[00:05:47] The Impact of GPT-4 on Intercom's Strategy
[00:08:18] Challenges of Adapting Developers to AI Workflows
[00:13:40] Transitioning from Traditional to AI-First Product Development
[00:19:46] Hiring and Skills for an AI-First Engineering Team
[00:24:48] The Role of Chat-Based Interfaces in AI-Driven Products
[00:31:13] Evaluating the Effectiveness of AI Solutions
[00:42:32] Future of AI in Customer Support and Product Development
[00:55:01] Closing Thoughts and Future Directions
Full Script
[00:00:15] Guy Podjarny: Hello, everyone. Welcome back to the AI Native Dev. on the show today, we have Des Traynor, who is the co founder of Intercom, who are one of the earliest companies to really properly embrace AI, so we'll hear a lot about those types of learnings, and Des is also an active angel investor,
[00:00:30] Guy Podjarny: so we'll have interesting perspectives about AI for, for things beyond the intercom sphere. So thanks for coming onto the show, Tess.
[00:00:37] Des Traynor: Thanks for having me, Guy. Excited.
[00:00:40] Guy Podjarny: We're going to talk a lot about, autonomy of AI and testing and learnings, but I think people need a bit more context if they're not familiar with, with intercom specifically with Fin, with your AI agent.
[00:00:51] Des Traynor: Sure. I mean, Intercom, we started life as a sort of general purpose customer communication tool. So like a way that internet businesses could talk to their customers and vice versa.I'll skip the 10 years in between, but yeah, so let's just say two years ago we went all in on customer support and, two, I guess maybe four weeks later, ChatGPT launched and then AI first customer support became the most obvious sort of future of what it meant to build that customer service.
[00:01:17] Des Traynor: So we jumped on that basically, I think ChatGPT dropped November 30th. By December 1st, we were working on what would be like the sort of future of our platform, and the huge components of that was,AI native features, and then later on, Fin, and then more recently Fin Copilot. But we're basically taking an AI first approach to, to building out a support platform,
[00:01:37] Des Traynor: cause we think that's obviously where the future of basically every industry, but certainly customer service.
[00:01:42] Guy Podjarny: Yeah, no, super interesting, and I think, I guess I think of intercom oftentimes as the one that pioneered or at least popularized the, the sort of chat based support, that is in line, typically bottom right corner.
[00:01:53] Guy Podjarny: The little speak bubble, speech bubble, on websites. So I guess you were very, it felt relatively natural as a chat product to, to embrace it, like ChatGPT felt, reminiscent or felt, familiar?
[00:02:04] Des Traynor: Yeah, exactly right. Like I think chatbots are what happened when two mega trends, one was like the rise of messaging and the other one was like the rise of AI combined.
[00:02:14] Des Traynor: And then chatbots basically, had multiple false starts, like you probably remember the if this then back bots. Yeah, and then they they slightly matured and matured again, but realistically generative AI is the thing that made them like not shit, which is where the industry needed to get to.
[00:02:29] Des Traynor: But I think, yeah, like we, we were like certainly, foundational and pioneering in moving support to being an always on thing, so you could literally talk to the people behind a Tessl or a Snyk or whatever, just by engaging in the bottom right hand corner and asking them questions, get a quick answer. You know, and like that was, for the longest time, that was our intercoms, hundreds of millions of revenue came from, and then obviously, how does the world change with those answers can be quite damn good and generated in zero seconds by an autonomous agent that can Find out answers and see what's been said before.
[00:03:03] Des Traynor: And disambiguate and qualify and follow up and all sorts of stuff like that.The nature of it all changes and that's what we've been like wrestling with since basically every day since.
[00:03:12] Guy Podjarny: Yeah. Yeah. Super interesting. So we'll dig into those, but I think maybe to give us a bit of a chronological order, maybe,you, you had, you spoke in various podcasts on this sort of code red moment of hey, we, we see this, you had, if I remember correctly, Fergal, your head of AI, saw ChatGPT, got you all looking into it and you all, agreed that, you shouldn't increment, hey, let's put three people on it, but rather route a bunch of people into this and really bet on this approach.
[00:03:40] Guy Podjarny: I think what I found interesting is, that implies in that moment, which is clearly insightful in hindsight, and, you really embrace it, hopefully, reaping some benefits, of that today. it does imply that you routed a good number of people, probably, that were not previously familiar with working with LLMs or these sort of statistical systems, and specifically a good number of devs to work on it.
[00:04:01] Guy Podjarny: Is that correct? And I guess, how did that, how easy or hard was it to get a bunch of people acclimatized to this sort of new way of working?
[00:04:11] Des Traynor: So the timeline goes something like, ChatGPT launches on a Thursday, it's Thursday evening in Dublin, Ireland, where we're playing with it. I'm slacking with our, VP Fergal of he's our VP of AI.
[00:04:23] Des Traynor: We really, he's very convinced and he convinces me with examples that like, this is a thing, this is a big thing. I speak with Owen on Friday and spoke with him through the weekend and Owen's our CEO in Intercom and ultimately this is a time in a company where you need an extremely decisive and fast moving CEO who's not in any way, say, paralyzed by risk or fear.
[00:04:42] Des Traynor: Almost just let's go. I think we should really go for it. So by Monday, we got to work, prioritizing the first wave of the features, which actually wasn't Fin, it was stuff we built inside the inbox to augment the support workflow. So things like automatic summarization, or things like reframing, rewriting, all the sort of stuff that a support rep does all the time.
[00:05:01] Des Traynor: We were just trying to like, basically augment their workflow by removing all the undifferentiated heavy lifting. And we released those features in about seven weeks or so, and we probably put outside of the AI group who obviously would always be working in this area. We probably pushed maybe like 20 more people at that time. Most of our inbox teams and some of the surrounding teams got to work on it.
[00:05:23] Des Traynor: By January, we started to get word that GPT 4 was coming, and that was the big unlock, and that was where we were like, Oh, GPT 4 had, we had the ability to prompt that in such a way that it wouldn't hallucinate at the same rate that 3.5 was. And that's when we realized, if this thing, if we can control the hallucinogenic properties of this, and dial them down, we can really build the end to end answer to damn question bot.
[00:05:47] Des Traynor: So we launched that in March.
[00:05:48] Guy Podjarny: And before, so before you go into that,so before ChatGPT 4, or sorry, before GPT 4. the tools that were not internal, they were a customer facing tools, but they were more, I guess, kind of fair to call them AI assisted. They're sort of attended. Hey, I'm going to help you do a bunch of that groundwork, but it's to your customers, just not to confuse.
[00:06:05] Guy Podjarny: It's not to your support people. It's to the support people that are,
[00:06:08] Des Traynor: Yeah. The terms get trickier. Obviously we have a support team. That's not who I'm talking about, but our customers are people who support their customers. So when we say, like we say, agent facing is the tools we built, like as in customer support agents,
[00:06:22] Des Traynor: got access to these tools, so if they're dealing with a ticket or a conversation, they could click a button that would generate a summary of it. Their customers, our customers didn't see it. That was what Fin unlocked. Fin was like the bot that could actually talk to our customers on our customers behalf.
[00:06:36] Guy Podjarny: Yeah, understood. Cool. Okay, so you've got it, so you jump on the bandwagon, you don't go right away to the biggest audience that would see this, you would go to the, you went to the medium audience that would see this, which is your, but it's the people paying you. So, I mean, if you care to kind of make them happy, and to like you, and you give them attended a helpful.
[00:06:56] Des Traynor: It's also for what it's worth, like this, there's probably something generalized about this, but it was like the lower risk team to build as well, because if it generates a bad summarization, that's not a fatal outcome, however, giving an awful answer to somebody, automatically, and then nobody knowing it happened is actually a far bit more dangerous outcome.
[00:07:12] Des Traynor: So that's why we figured it was an easier dip your toe type feature.
[00:07:15] Guy Podjarny: Yeah, no, perfect. That sounds super wise. And we'll dig a bit more into like attended and autonomous and all that a bit later. Okay, cool. But so you have GPT 4. Then you see that Fin, you see the opportunity of Fin.
[00:07:26] Guy Podjarny: And then you do allocate more people to come work?
[00:07:29] Des Traynor: Yeah, I think at that point it grew to maybe a hundred people or so, across cause like teams from our messenger team, various different backend teams, Fin works off, like articles. So our knowledge based team who owns our, like your docs product, they got involved as well.
[00:07:43] Des Traynor: So it became like a company wide initiative in loads of ways. And then all of our marketing shift and focus on this specific product. So very quickly it became a huge chunk, if not the majority of the R&D or was working on either direct this product directly or the ancillary products that feeds this product.
[00:08:01] Guy Podjarny: We have the pleasure, dubious pleasure of sort of building LLM based products at Tessl, I've had some of that at Snyk. It's different. They're annoying. They are hard to evaluate whether you're doing your job correctly or not, sometimes costs are different.
[00:08:18] Guy Podjarny: CICD is different. Some of the tools are quite immature. and we're talking, probably a year before that experience that I'm referring to here. How hard was it for a developer who previously didn't work in that surrounding to adapt it? How did you help the dev team, not just, hey, I reprioritize this feature or am I overblowing the change?
[00:08:40] Des Traynor: I don't know. The change is big. It's actually been a subject of a lot of internal debate. Um,I'll give the umbrella answer and then we can get to specifics in a little bit, but like generally speaking, everything about how we build software products changes when you bring in generative AI,probability based systems or whatever you want to call it.
[00:09:00] Des Traynor: There's classic product development is some sort of double diamond. It's like research, make decisions, build like ship and then see if it works, whatever. You have like extra diamonds all over the place when it's like, AI, because it's, you don't start with here's the user problem, let's solve it.
[00:09:14] Des Traynor: You, you tend to start with what's technically possible to begin with. And then you say, hey, does this new technical capability, let's say, hey, we can read PDFs in a second. Okay. does that map onto any problems that we have to solve for our users? And then you Find one of those. And then, so then you've worked out, this thing can be built and it seems like it solves this problem.
[00:09:34] Des Traynor: Even then, everyone's DNA and software is, so build it and ship it and move on to the next feature. What's the big deal? And, I think the problem is once these things go live, you still don't know if it works or not. so you have to do so much, you know, whereas in typical boring bread and butter B2B SaaS, someone's oh, I wonder, can we merge expenses?
[00:09:52] Des Traynor: Okay. So you get two receipts, you have to combine them. Oh, okay. If you ship that feature and people use it, you're like, great. You're not sitting here going, ooh, I wonder if it works. You're assuming that, that they're like the mathematics worked or whatever.
[00:10:03] Des Traynor: Whereas, in all these systems, like once it goes live, it goes through a whole other, like almost like product evolution, which is it actually working the way we thought it did?
[00:10:13] Des Traynor: And obviously we do our best to not just test these systems on what we would call happy paths. A happy path is like, how do I reset my password? An unhappy path is, hey, my account is stored in my husband's name and it's my credit card and I don't even need to move it to his credit card, but also we need to change address at the same time, right?
[00:10:31] Des Traynor: That's where like these systems struggle massively. They're all brilliant at like, how do I reset my password? In the same way that if me and Einstein sit down and take a maths test and it's just basic arithmetic. I look as if I'm as smart as Einstein, so you have to have this higher pass filter to get through, but nothing like reality bats last in AI, it's the last person to step up.
[00:10:52] Des Traynor: And, and reality tells you like, hey, Fin works great in all these circumstances, but here's a pocket that it doesn't work. And here's an area that I can't reach or whatever. So I think, the first thing I'd say to the question of enabling devs, empowering devs to work in this LLM first world is like getting out of your head, the idea that things just simply do or don't work, cause it's all a spectrum now, getting out of your head, the idea that your core job is to just,clarify or remove ambiguity. Ambiguity is all over the place.
[00:11:19] Des Traynor: It's and your job is it's such a, there's like an art and science to like threading all the needles to Finding the useful solutions. And that's before we've actually sat down in front of a text editor and started banging out code, that's as in working out what problem we could solve, why we think it's like most of your prototypes in this era are like written by just prompting an LLM and seeing what it does with certain prompts and seeing what it does in response to certain asks, and basically, if you can get to a consistent, reliable set of behaviours,
[00:11:50] Des Traynor: then you can start to think about how would we productize this? What are the API calls? And then what's the UI look like? And then how would we describe it? All that. We have so much like pre work to do. All that said, there is obviously around any given like product, if we just, I'll take Fin because it's one of our best.
[00:12:07] Des Traynor: There is still a lot of like bread and butter B2B SaaS shit you have to build. someone has to import docs and somebody has to have permissions and it has to be charged for it, so there's a pricing feature. Like it's not all, it's not like you're an LLM engineer or a bust. But it's just, it's the secondary features that are kind of like fall into the classic like, our reporting engine still draws graphs.
[00:12:26] Des Traynor: That's not new.
[00:12:27] Guy Podjarny: yeah. Like any, like engine,if you built any sort of proper product at the end of the day, the engine is 10, 20 percent of the surface of the product.
[00:12:35] Des Traynor: Exactly that. And and I think, the, that there's a whole thin wrapper, thick wrapper debate that happens a lot in the industry.
[00:12:40] Des Traynor: And I think like Fin is very much like a thick wrapper. There's a lot around Fin that needs to be there to make it work really well, such that the actual, the pieces are like, the little like the resolution engine that sits inside of it, that is a part of the puzzle. It's a really important high impact part, but it alone isn't sufficient,
[00:12:59] Des Traynor: it still needs the 16 other things, the vector search, the the content import and all that sort of stuff.
[00:13:04] Guy Podjarny: so it sounds like, it sounds like a massive change. it sounds like the, a lot of the changes in the, is in the sum, the total of the product. There is a good chunk of people that are still working on things that don't require them to change their, ways of working, but still a good portion that did,maybe in the total. So I guess, are there some key takeaways, like I'm sure you've tried things and some work you had a little bit of a headstart cause they didn't have some AI capabilities, you had.
[00:13:32] Des Traynor: We had an AI product live since I think 2017, 2018. So we definitely have prior experience here,
[00:13:39] Des Traynor: and that was one of the reasons we can move fast, cause we are able to reuse a lot of our existing features. For example, like the vector search engine, we'd already built one for a previous product, etc. So we were, that's one of the reasons why we were like out of the gates so fast. I, if I was to really say like the, the biggest lessons I've observed, like to be clear, what I'm skipping over here is I'm sure there are libraries and APIs and all sorts of stuff you need to read and get good at, etc.
[00:14:00] Des Traynor: But I think um,to actually be a performant engineer in this era requires some amount of ability to interrogate data or perform data science, to measure outcomes, to give you a very simple example, when we make a change, like we've ran probably like 50 different A/B tests on Fin's performance, and it's very tempting,
[00:14:24] Des Traynor: everyone comes in, who might join the group and say something like, oh, I have a cool feature, I could add, how about we make it so that, blah? Like, how about we make it so that it'll always ask this question at the end of a thing? And what you don't realize is like all of these things are different iterations of a resolution engine whose job is ultimately to given a question, answer it.
[00:14:44] Des Traynor: If you break that, it doesn't matter how cool your little addition was. Now the thing is that's not a binary yes no condition. It's like today's Fin's performance is about 50% of, so 50% of inbound support volume will be resolved by Fin.If somebody rolls out a cool feature that brings that 50 down to 40, that's a problem.
[00:15:02] Des Traynor: Yeah.
[00:15:02] Des Traynor: If they do that and don't even know to check or to realize or to look at its performance over a trailing 30 days, whatever, that's an even bigger problem. So there's a very much you need to work out. You have to understand, trick to working with these systems is to like, bottle up a probabilistic outcome and productize it and you have to work out who is allowed play with the probabilities of that,
[00:15:25] Des Traynor: and understand that it's not like two different worlds. It's very easy for somebody in our messenger team to damage the performance of Fin, even though they don't write to that code base. But if they frame it wrong, and they introduce Fin in the wrong way, it'll change what people write. And if it changes what people write, then it'll change how Fin performs, so it's really the extra skill required here is the ability to assess Does this thing do the job it's supposed to do at a really high performance level? And you have to have some way to measure that, some way to test it, have bake offs of different engines against each other, some way to have an acceptance framework for when, all right, that's been live for 30 days, no one's complained, it looks like we got a 4 percent increase in resolutions, ship it.
[00:16:07] Des Traynor: It's like that, none of those skills really existed before in, I'd say in most companies, but definitely not in Intercom.
[00:16:14] Guy Podjarny: Yeah, no, it sounds fascinating. I guess the good news is there's some analogy between what you're describing and thestate of the art in terms of continuous deployment, right?
[00:16:23] Guy Podjarny: If you're working with feature flags, if you're in Facebook and you're shipping, I forget the name of their sort of, feature system there to some gradual, number of people, then there's some similarity of it, but this is much more en masse, much more predictable. And pretty much, does it kill almost substantially the value of a sort of CICD assessment for anything that isn't the more bread and butter SaaS piece?
[00:16:45] Guy Podjarny: How much having run with this now for a bit, how trustworthy or how, revealing are the results of the synthetic tests or the CICDs type, the pre deployment tests that you have?
[00:16:59] Des Traynor: We have two tests. We have one test we perform internally, which is expensive to run, which is like our torture test.
[00:17:04] Des Traynor: So let's just say. Love the name, torture test. Yeah. . because it's, it's it's all the psychotic scenarios that Fin will get put into in the real world. But we were, we wanna make sure it doesn't like, barf. So we don't do this unless we have good reason, but let's just say like GPT 4.0 drops or something like that, 4.0. the tempting thing to do, and you'll see a lot of our competitors do this, is just holy shit, new model, plug it in. We're good. And, and I think what that says to me is it speaks to the immaturity of their rollout. like we also know the same API endpoints to ping if you want 4.0. I think it's not, we weren't sitting here going, Oh, I wonder how do you do this? It says,our approach to such a scenario is Okay, let's first of all, run this through the torture test and our torture test is there to deliberately, give us hard situations where we have a very clear outcome.
[00:17:50] Des Traynor: So for example, let's say Tessl is using Fin. Would you want Tessl's bot to recommend competitors over Tessl? Would you want it to would you want it to answer a tier three question? if it didn't know the answer, how would you want it to behave? Would you want it to make up an answer?
[00:18:05] Des Traynor: there's a long variety, like genuinely hundreds of scenarios where most people agree on what the right outcome is, but like GPT 4.0 might not agree with that. So this is where like you have to control and contain the behavior. So once it clears the torture test, and we look at its performance, then we'll move to look, let's put this live, let's see how it works in actual reality, which oftentimes is just.
[00:18:28] Guy Podjarny: In a gradual fashion, like you wouldn't turn it on for everybody, you would graduate, you would,
[00:18:32] Des Traynor: Yes, absolutely.
[00:18:34] Des Traynor: And I think there is a lot of parallels to CICD, except for the sheer, It's almost say if you're Facebook running on CICD and oh, let's move the poke button to the left, from the left to the right or something like that. Okay. That probably will change human behavior and it's probably in a hard to guess way.
[00:18:47] Des Traynor: We have that problem, and then on top of that, we have the, the fact of, we don't even know what poke does most of the time, but 90%, yeah.
[00:18:53] Guy Podjarny: Would it show up on the right or on the left on know uncertainty,
[00:18:58] Des Traynor: yeah. yeah. and everyone, I'll get this, like when you multiply, in probabilities they tend to grow.
[00:19:03] Guy Podjarny: Yeah, super interesting. So this is fascinating. So you move to, to really go all in on AI, you're highlighting that there's still a lot of like core SaaS that has to be built for the full experience on it. but you have to change your total approach to product to, in some respects, really embrace the sort of continuous deployment, type, type model, but really probably more supercharged.
[00:19:24] Guy Podjarny: And that requires developers to have a lot more ability to interrogate, data to work in that fashion, how, has that changed? Like when you look at the team, are there either, did you have to recalibrate and actually move some people out of that team or hire for some skills and maybe a bit more gently, for new people that you hire,
[00:19:43] Guy Podjarny: is there a difference in the skill set you're now searching for?
[00:19:46] Des Traynor: Yeah, we distinguish right now between engineers in our AI group and engineers in what we call a core product. Generally speaking, we look for we're like Ruby on Rails shop. We tend to hire like, engineers that are full stack or close to full stack.
[00:20:01] Des Traynor: And we still need a lot of them because they said the reporting engine for Fin has like date pickers and graphs and charts and sliders and all the shit you'd expect. And none of that is is like head scratchy type stuff. It's an average CS degree produces people who can do all that.
[00:20:15] Des Traynor: The extra skills we look for within the AI group is everything around, data science. It's everything around basically like data driven decisions, and it's got there's a hard requirement on the ability to interrogate performance. That's what we actually, need. We tend to hire it in that group.
[00:20:28] Des Traynor: So a lot of our products begin life inside the AI group until we are sure that we've got the core engine performing. Then we, when we, when it comes to productizing it and building out all of the, as you said, the sort of the SaaS standard wrapping type stuff that's necessary to utilize the engine.
[00:20:45] Des Traynor: That's when we hand it over to people, and for those folks, we're not sitting here going, oh, I wonder what the date picker is going to do today. We know how to do all that sort of stuff. So we right now, I think this will change over time, but right now we just basically say there is pockets of our sort of overall engineering work where we deal with significant uncertainty,
[00:21:02] Des Traynor: that is that needs containment. And there are people who usually write against the LLMs and they take scenarios, do testing, all that sort of stuff. And then there's the rest of the work, which is like, productizing what, what has been coming out of the AI group. So we haven't changed our kind of core deFinition of an engineer, but when we're hiring engineers to join the AI group, we have extra requirements.
[00:21:21] Des Traynor: If you're a brilliant research scientist who's wonderful at data and is really good, I'd say programming, but has never really built a web app before. Historically, we might not have had much of a job for you.
[00:21:30] Des Traynor: Now we might've been our infra team or something like that, but now we're like, it doesn't actually bother us. you don't need to be like brilliant, like f**king React or Ember or whatever to perform in the AI group, but you do need to have some skills along those lines to actually productize, which is why you'd probably join R&D if you have that skill set.
[00:21:47] Guy Podjarny: Yeah, interesting, but it also sounds like the AI engineering is not all PhDs in machine learning. it is software engineers with a data bias or somewhere between. Yeah, it's machine learning, engineers with a software bias and vice versa.
[00:22:03] Des Traynor: There is a higher percentage of machine, of PhDs in the AI group for sure.
[00:22:07] Des Traynor: I think a lot of that comes from, Fergal's desire to have people who are very capable of running their own, data analysis on their own work. and typically that those skills tend to get most perhaps battle hardened in research labs. So I'm guessing, I'm confident there's more PhDs or MSCs or whatever in that group.
[00:22:23] Des Traynor: I'd say the reason is what I said, which is just the ability to perform your own analysis at a pretty deep level, without needing much oversight, which you might not get in a regular computer science degree. But, but we've had like multiple successful engineers leave regular stuff and go and join the AI group and perform really well.
[00:22:40] Guy Podjarny: Yep. So it's different skills. I'm curious a little bit, whether this would lead into kind of research entities or will the industry of developing on top of these AI go maybe the way of design and others in which you'll find yourself wanting to have, trials, or maybe now it's quads of teams or domains have associated AI accompaniment.
[00:23:01] Des Traynor: It's a great question. so right now,it's, It's an opinionated thing we're doing to say the AI group exists here and people who build products that wrap the AI group stuff to actually commercialize it exists elsewhere. There are benefits, costs and benefits in a sense, right? You can guess all of the challenges of having people like hinge on, like right now in a classic Intercom engineering team, there's like an engineering manager, a designer, a product manager, and maybe five engineers.
[00:23:25] Des Traynor: What they don't yet specifically have on every team is like the AI person,
[00:23:29] Des Traynor: because the actual products like aren't built like, in a distributed way. Products are built in a centralized way in the AI group, and as I said, that engine then gets taken out and the engine is what gets, the product built around it.
[00:23:42] Des Traynor: And then that's what we launch.
[00:23:44] Guy Podjarny: Yeah. Yeah. Interesting. We'll see how that evolves and how that counters the other best practice that evolves around, less of full stack teams, right?
[00:23:51] Des Traynor: Yeah. Like it's very clear the trade offs we'd make in both directions to be fair.
[00:23:55] Guy Podjarny: Yeah. So maybe let's shift gears a little bit and talk about the learnings on the product side.
[00:24:00] Guy Podjarny: So you, as we mentioned, you had a chat based product. You also had the AI,maybe, headstarts, a little bit in the organization. But still, you had a chat based product, so the interface, or at least, it's easy for me from the outside, it's easy to understand how ChatGPT will interact with Fin or will, you know,
[00:24:18] Guy Podjarny: augment or even create Fin doing it. How significant do you think it is to have already an existing chat interface or language interface? Do you think the same urgency of the sort of code red that you pulled is appropriate in other domains? Or is it really like an order of magnitude more urgent for people that are already talking,but are already chat based.
[00:24:44] Des Traynor: Do you mean like specifically in customer service or do you mean broadly in the industry?
[00:24:48] Guy Podjarny: I think in the industry, in industries as a whole, if you just want to talk about products or how significant was it that your product was already chat based?
[00:24:55] Des Traynor: I, yeah, it's a good question. I haven't really thought about this too much.
[00:24:58] Des Traynor: I'll tell you, it's, it made it very damn obvious what we have to do next.because, obviously overnight people got used to just typing, into ChatGPT. and ChatGPT seems so smart that like anywhere else you type that's not ChatGPT, seems dumb, right? or Claude or any of those. so I think, like for a lot of our competitors who had say, maybe who worked in customer support, but had a ticket first mindset or an email support first mindset, I think it probably was very alarming to them because they were like, oh, shit.
[00:25:28] Des Traynor: Intercom had this right all along in that if chat was going to happen and AI, if AI was going to happen then chat was going to be the obvious, conduit. so you saw, we saw a lot of people scrambling to copy our messenger as best as they could, and then also go and try and copy our Fin as best as they could too. So I think within support, it was obvious, but if I just take a slightly broader thing,I think chat UI is a fascinating area.
[00:25:49] Des Traynor: And I think the next year we'll see. And voice is part of this as well. But let me just say a few different things. One is like what chat driven UI offers us is a way to use products where you know, the outcome you want, and you know how to explain what it is that you want, but up until now, the UI has been the barrier that you didn't know how to use.
[00:26:15] Des Traynor: So to give you a simple example, maybe you do or don't know how to use Photoshop, right? Maybe you know a little bit, a few layers and a couple of effects, but you don't really get it. So if I gave you a photo and say, can you remove the foreground from the background? or can you make Des's t shirt green instead of white or whatever?
[00:26:31] Des Traynor: You know it as you want. You probably have an idea of your end outcome, but you probably just don't know how to do it. And that's an interesting barrier that like UI design and the whole usability era of 2005, 6, 7, 8, 9 was about this. It was about like, how can we make all of these arcane technologies easier to use?
[00:26:49] Des Traynor: It turns out the easiest UI to use will often be one where you just need to explain what it is you want to do. So you probably have somewhere in the depths of Tessl an Excel spreadsheet that tracks your revenue growth, or will, you will at some point. And you might want to be like, how can I forecast this out for three years and assume the growth degrades and churn up ticks and retention and then or,
[00:27:10] Des Traynor: and the answer is, lots of formulae that you don't know off by heart, and yet you're in a world where you actually do know the thing you want to do. So what chat and then what soon will voice will be just, I think when Apple launches Apple intelligence for real, and assuming it doesn't suck, what I think you'll see is, everyone just gets used to saying things, almost speaking to the computer, more like it's an actual server to actually understand some and the sort of net effect is people will,
[00:27:37] Des Traynor: assume that they can use software because they just need to say what it is that they want. So I think products that haven't heretofore adopted a chat driven UI might want to think about it because it might just, or like a voice driven UI, because whilst there are still some obvious power tools where like pointing and clicking and dragging, I'm thinking about an example, let's say you're editing a movie and you want to overlap time zone, sorry, timelines and all sorts of, there are clear power tools for that.
[00:28:04] Des Traynor: But those power tools are going to be much more like the back of the toolbox and not like the hammer or screwdriver. I think you, you might end up thinking about like voice UI and chat UI as the hammer and screwdriver as in everyone has one. And then if you need to, you've got the big dirty f**king toolbox that you keep in the garage for occasions when you need it.
[00:28:25] Des Traynor: And I think you'll be surprised how many things you can get done just by saying the thing you want to do. I Even these are like those Facebook, Ray Bans, the Metaglasses, sorry. Even that is like quite impressive for what it can get you done. And that's just meta, meta is like whatever it's called,
[00:28:41] Des Traynor: Llama.I think everyone's interaction mode with software is going to change and they're just going to get used to, not learning how to use Microsoft Word or Figma or whatever. And in a load of cases, unless it's a specialist tool, just saying the thing you want done and getting it done. And that, that just changes the nature of software and it massively democratizes it like.
[00:29:03] Des Traynor: In two years time, everyone might be brilliant at Excel, that might just be the case because, anyway, so that, that's what think like some people started like we, we were a chat UI all along for loads of different reasons. And so it was like, say, Slack.
[00:29:16] Guy Podjarny: Supercharges, like it just, it makes a product ten times better, but it doesn't require a change in how users interact with it.
[00:29:25] Guy Podjarny: It just succeeds that much more often.
[00:29:27] Des Traynor: I think that's totally correct. And even if I hear our listeners, have driven a modern, fancy car where you can press a button and say, please turn on the air conditioning and defrost the windscreen or something like that,that's actually when it works.
[00:29:40] Des Traynor: And again, that's a massive asterisk for a lot of these things, but when it works, it's way better than these weird buttons on your dashboard. You're kind of stop and hoping, but if you can just say the thing you want and get it, that is, that's remarkable.
[00:29:52] Guy Podjarny: Yeah. That's a very, that's a very useful,
[00:29:55] Guy Podjarny: and, and I agree with, like good analogy of areas in which a chat based interface will help. I think what's interesting, and so maybe connecting it a little bit to, to code. I guess two questions. First just on interface and in AI dev tools, two of the most prominent, use cases that are already getting adoption are code completion in the IDE Copilots of the world, and, next to them, pull request related items, a review, a change on it. and I guess when I think about those naively, I feel like the completion piece isn't chat based. By default, it's completion. It is text based, a different one, while the pull request review is chat based.
[00:30:36] Guy Podjarny: It's asynchronous, so it's interesting. You compared it to a ticket based approach. Maybe it's, it'll be good to maybe hear that distinction. I guess I, I wonder, based on your experience, generally, like what you've seen in the year or the experience with Fin, do you feel the LLMs are more aligned
[00:30:52] Guy Podjarny: to one of those, approaches or to something else, likewould the advantage of the chat based sort of approach, that exists in pull requests, give it an edge, of not of adoption, at least, forget about capability even, but just like adoption because people are already used to interacting with this through chats.
[00:31:13] Des Traynor: Yeah, there's a bit there. I think that when I think about what completion actually is, which is let's just say you, like you, you stub out a method and you put in a kind of pseudo code comment saying, here's what's going to happen, and then you just hit the magic go button and the thing you want it to do.
[00:31:29] Des Traynor: It is, it's another variation of the describing how I want something, describing the outcome I want in a way and getting it. and saving me like the pain of the interface in between, which sometimes is point and click, and sometimes it's just typing out a for loop or whatever, right? Yeah.
[00:31:47] Des Traynor: Yeah. Um,I think they, there is like a similarity, I put it this way. I don't know if that binary exists between let's just say the, completion and say like a state based thing, like review this pull request or whatever, they're not miles apart from each other. I think chat UI basically offers people a way to say, write the thing you're trying to do and hit return and it'll get done.
[00:32:06] Des Traynor: And I think completion is like a different instantiation of the same principle, which is like describe roughly loosely what it is you're trying to get done, hit the magic button and see if it gets done. If you don't like it, provide more context, and then, like the risk with say going, like in the case of most of the completion stuff I've seen, that's just GitHub Copilot is what I'm most familiar with, but I've looked at Cursor and a few of the other ones.
[00:32:30] Des Traynor: What I find, like it's quite a risky move, because if it doesn't work, it's shit. And we've dealt with this quite a bit and we can talk about that when we talk about our own assistive product, Copilot, where to some degree. The measure of how confident you are has to drive your UI decision,
[00:32:44] Des Traynor: because if you're like, so maybe just to get into it, to exemplify that point. If you're sure that nine, nine and a half times out of 10, you're going to be right. Then you can design it such that it naturally assumes it's correct and embeds it into that way, so it just goes, so rather than popping you a dialogue saying, I think you're trying to write a loop, does this look like, and you have to say yes and all that, you just inject it directly because you're, you have the confidence.
[00:33:07] Des Traynor: That makes sense. To give you an example of a change we made, we were building a effectively completion for customer support teams. So a new conversation comes into the inbox and you read the conversation and you start typing. And what we were hoping we could do would be like tab, auto complete and hit send.
[00:33:25] Des Traynor: And it would not be a wonderful experience. And in practice, it's not like when you test this on the the happy path, it's very easy to solve. When you test this on the torture test or anything like real world usage, it doesn't work at the same rate at all. It blends in shit like PII, it confuses scenarios, it confuses types of users, it says too much, it says too little.
[00:33:50] Des Traynor: because of all that, The right place, the right UI design to wrap around this was not injected into the text area so that somebody can press return. And instead, what we actually built was a right hand sidebar called Copilot, where you can diagnose the issue, select the pieces, search knowledge bases, look for related conversations, where you basically have a chat on the right hand side.
[00:34:11] Des Traynor: We're like the Copilot who's effectively an expert trained on every conversation that's ever happened in your business.
[00:34:15] Guy Podjarny: Right.
[00:34:16] Des Traynor: And, and like the reason we built all that other software was because we couldn't do the thing we wanted. If the thing we wanted was not going to be feasible.
[00:34:23] Guy Podjarny: Which would be inline, the, and so I love that you're saying two things. One is completion and, Whatever, a chat based interactions are not that distinct in both cases you're describing. I love the comment example because comment is very much a chat. you're talking to your future self, in a second here, I'm going to write this thing or in my sort of future person who comes in to, to read this code.
[00:34:43] Guy Podjarny: This is what the next bits can. So that, that aligns with your previous example of, why do I need to know which icon to press on the car dashboard? Can you just tell me to turn on the air conditioning? so I like that, but then to an extent, even as you type it in, all of that can be seen, which I guess it is as a chat, you're talking to it, whether or not you're doing it in line or not, whether it's chats offline is really, it's just a matter of confidence.
[00:35:07] Guy Podjarny: If you have, doing it, I do think like I, I relate to that. I feel like there's a little bit of pushback in that, the chat thing is, is, is often wrong. And I think the inline option gives you, every time I press a key, you have an opportunity to give me something different, right?
[00:35:24] Guy Podjarny: If the chat on the side told me something and then if I didn't press the copy button out of that five seconds later it would give me something else and five seconds later it would I would turn off that pane pretty quickly. but in the,in the code that it runs.
[00:35:38] Des Traynor: Yeah, that's there's an interesting point there, which is like, how much context are you willing to feed before, the more context you feed, the more cost you pay to use this feature, but also the greater the chance of certainty.
[00:35:48] Des Traynor: A classic intercom situation would be like, some people write into support and their first question is, hi are you there, right? Copilot is not going to do anything useful for you with that, right? The answer is yes, I am really here, my name is Really Des, I am here to support you. And they say, I have a problem.
[00:36:02] Des Traynor: We're still getting nowhere, right? And I say, what's the damn problem? And they say, the screen doesn't work. And I'm like, you have to really press it and all that. So to some degree you have to do, sometimes you have to do a lot of work to feed it enough states such that it'll guess correctly what it should do next.
[00:36:15] Des Traynor: And a version of that might be like,if you're playing with, say,Claude and it's like its ability to, say, generate complete apps or whatever, you can say build me a game, or you can say, build me a game called Pac Man, or you can say, build me a game called Pac Man, where there's a yellow character in a maze with ghosts, and he runs around and eats pills, and the more state you give it, the more cost you're paying, but also,the more, precision you get in output.
[00:36:35] Des Traynor: it's almost I always used to say to people,going back to, the, the jobs to be done, school of thought, that we used to talk about a lot back in the day in intercom, I would often say,if someone says, I am hungry, you don't know what to get. If someone says, I'm running to an airport, starving, and, I need something to eat that's going to fill me up while I'm on the go, and I only have one hand free, then you're like, here's a slice of pizza, it's like the problem describes the solution, the more detail on the problem, the more detail on the solution, and I think there's a similarity there for prompt as well.
[00:37:01] Des Traynor: And what you're describing is if you're a pretty good coder, which I'm guessing you are, and you're like knocking out like a pretty descriptive, code. And you're like halfway through a method that you've probably thrown a few comments into along the way. Yeah, it's gonna be able to tab auto complete the rest of the entire file for you.
[00:37:16] Des Traynor: if you start off with please build me Pac Man or whatever, it might be like, did you want a 3D engine? Do you want 2D? it's gonna misfire a lot more. So I just think prompting or like giving context to the engine is a kind of an ROI based thing. And the benefit of these like really specific LLMs is that they make a few assumptions on your behalf, which reduces the investment and increases return.
[00:37:38] Guy Podjarny: Yeah, no, I think that all makes sense. I think the, so the other lens of this equation, to, to your analogy on the UI that I really liked is your ability to specify it. So you give example of, I want to turn that shirt red. So the instruction is simple. Your desire is simple, but the application of it is complicated.
[00:37:58] Guy Podjarny: Sometimes the instruction is not that trivial, right? Like when you think,we talk about AI native software development. We talk about it being spec centric versus code centric. And one of the challenges we're playing with and learning is how do you specify, how do you know to say that Pac Man is 2D versus 3D?
[00:38:14] Guy Podjarny: How do you know which questions to ask, and,and sometimes proactively, which information to volunteer. And I think like maybe importantly, how do you draw the line, because if you don't say something, then the LLM will provide, will make assumptions, right? Assuming you haven't blocked it from doing that.
[00:38:33] Guy Podjarny: And you can give it context about yourself, about coding in the world, about the application. How do you control? the degree of freedom that you, that you want to give it.
[00:38:43] Des Traynor: Let me like,if you can cast your mind back to when you were learning how to code and at some point somebody introduced you to the concept of pseudo code.
[00:38:51] Des Traynor: And specifically, maybe it wasn't you, maybe you were that damn good, but somebody in your class was probably much better at pseudo code than they were at code. Which always led to the discussion of, why can't you just compile pseudo code? Because that would be a lot damn easier, right? And the answer, as we all learn over time, because a lot of us, I certainly did, I'm sure you did as well, we tried to write a pseudo code compiler.
[00:39:12] Des Traynor: And the answer always, This comes out to follow, which is if you want specificity in output, you need specificity in input. And the modern day version of this is, why does all the AI writing sound the same? The answer is because you, the brilliance of these things is that you give it such a small prompt and it can be so evocative and prescriptive.
[00:39:31] Des Traynor: How? How does it know whether to use the word brilliant or gorgeous? How does it know? How does it know any of these things? Because it's making assumptions and it's costing you, like it's robbing you of that ability for, sorry, your desire to short prompt is robbing you of the ability to add in detailed description, which means you're not going to control the output either.
[00:39:50] Des Traynor: So that's ultimately the paradox we're playing with here is we want to have that somehow the smallest possible set of inputs, but we also want extreme diversity in the set of outputs and the reality is much like the reason you can't we can't actually compile pseudo code or like you can, but you, it's not the same strength as an actual full programming language, is because the desire for detail is at odds with the desire for minimal prompting.
[00:40:14] Des Traynor: So to your question, it's it's the amount of description you're willing to provide a scenario will reward you on the far side with the detail and, the LLMs can say let's always, it's let's say we build an LLM for a keyboard powered 2D game. Now I can probably just say build Pacman or build Arkanoid or something like that, right?
[00:40:34] Des Traynor: We've baked in so much stuff, already, or it doesn't even need to LLM level. It could be at a silent prompt behind the scenes level or something like that. But, but in doing so, we've cost ourselves, we've shrank our world massively. So this is this is, I think going to be one of the, big tests we see as LLMs evolve over the next couple of years is do these verticalized LLMs truly outperform?
[00:40:55] Des Traynor: Because in some sense, their power comes from their specificity, but that also means that they've had to delete whole chunks of the world to achieve that. You know what I mean?
[00:41:04] Guy Podjarny: Yeah, I do. And I think it's super interesting. I think one, angle on that is that sometimes you want to be creative, but oftentimes you want to work within an ecosystem or within a context.
[00:41:12] Guy Podjarny: So like within an organization, you don't want to every time explain it. If you are a builder of 3d games for the, whatever, a Nintendo switch, and that's what we are, you're doing. You don't need to say that every time to the LLM to talk about, an application.
[00:41:26] Des Traynor: In your world, I often wonder like an LLM trained only on the entire WordPress ecosystem.
[00:41:33] Des Traynor: Would it be better or worse done in LLM training than all code ever when it comes to writing WordPress plugins? And it's hard, I don't know the answer to that question. I want to say the whole world will be better, but actually from a raw productivity, get it right every time, blah, blah, blah point of view, like when you shrink the world, maybe, and like our version of this, like we look at building, say LLMs, training them on a, because we're support conversations, blah, blah, blah, all that sort of stuff.
[00:41:55] Des Traynor: But it's not obvious to me, do we want a support rep of the future to have only ever heard support conversations, or should they have read Shakespeare? it's not obvious. I don't think we actually know the answer, in, in society yet. How much generalizable knowledge is useful in verticalized instances, basically.
[00:42:13] Guy Podjarny: Yeah. And I think I actually really want to drill into that sort of future AI native support. I'm going to keep that to the end, but maybe one tactical question on is like, how significant is it in terms of success today when you give Fin the existing knowledge base? I'm sure you have some customers that have populated Fin with a lot of information versus some that populated.
[00:42:32] Guy Podjarny: So I guess like how significant is it for it to get the right answer? And are you seeing any already some gaps of, hey, look, it's it's becoming robotic. It's only if I gave it so much information and it really stays within the lines and that's nice, but it means, it, it can deal actually worse than others with, with problems that are just on the edge.
[00:42:51] Des Traynor: Yeah. So the right content is is a huge part of the, what makes Fin work. having access to up to date for honestly, the most common experience an unhappy Fin customer has is they give us their knowledge base and Fin starts reading their knowledge base and replying as if the knowledge base was true.
[00:43:09] Des Traynor: And that triggers a realization of our knowledge base is years out of date. Our support team knew this, but they never had time to update it. And, so like that is a thing. Like the right information, like Fin deifies the content that you give it over its general intellect, if
[00:43:22] Guy Podjarny: so it will prioritize that. And then like the interesting thing that we often observe is, we do a lot of work to control, hallucinations and we do that because the nature of Fin is, if you're like have a hundred, hundred thousand conversations a month in the future with Tessl, and let's say a Fin has like a 0.2% hallucination rate, which it doesn't, but if it did, it means whatever, 200 people are getting the wrong answer, and you don't even know, and they might even follow the instructions in the answer, and they don't even know either, so all of this can go wrong. However, the work we do to control hallucinations can often suppress our resolution engine, if Yeah.
[00:43:56] Des Traynor: But because, basically, if Fin, let's say Fin is really good for up to 50%, if you want, if you're willing to hallucinate, you can get a lot higher than that. but you're just stepping into the unknown. So your actual resolutions will go, but so will your bad answers. Or, so there's a tension there of like, how, how much risk do you want to bake in if you like?
[00:44:16] Guy Podjarny: Yeah. And do you, have you considered engaging the customer with that? have you considered. saying the customer, Hey, I don't know that answer. I can try and get more creative, but then I might get it wrong. Do you want that?
[00:44:27] Des Traynor: Fin currently would do , it'll give you a sense of its own confidence.
[00:44:31] Des Traynor: So if you say, it's a store open on Monday at 10 AM, it will say, yes. if you say it's a store open on Christmas day at 3 PM, it will say, I don't know, here's the information I've read, which suggests that the answer might be blah, but you should verify yourself. And I think that's in line with how we would want a human to behave, which is a lot of what we're trying to do is like mimic a kind of best in class support rep,
[00:44:54] Des Traynor: and I, cause like in the future, what we want Fin to do is if it doesn't know, Fin should jump into Slack and post and say, Hey guys, question for you that I don't have the answer to. And then the people who reply to Fin in Slack, Fin will take that, say it back to the user and then possibly go and update the knowledge base so that it has the information for the future.
[00:45:09] Des Traynor: that's the full end to end workflow you'd like of a support app, like proper autonomy.
[00:45:13] Guy Podjarny: Yeah, no, I think that's very valuable. I'm curious. so thinking about that context, it's interesting to apply it to the world of code generation. There's a, there's, a set of really formidable products and companies right now looking to improve code completion, AI code generation as a whole,
[00:45:31] Guy Podjarny: by indexing an enterprise's existing code base.I don't know. I'm kind of asking, for a prediction. Do you think, I can see how they would realize most of their code base is crap and they would do it's almost like the knowledge base being out of date. It's no. I don't want to replicate this sort of debt.
[00:45:47] Guy Podjarny: Yeah. code based on it don't replicate these bad practices of the past. I can also see constraint, but also at the same time, it's pretty easy to understand how, Hey, you could learn organizational behaviors and it would seem a bit more like a developer in this company, I don't know.
[00:46:01] Guy Podjarny: if you were giving advice to, to anybody building that type of, of, code completion that is infused with existing organizational code base information. What would you say?
[00:46:12] Des Traynor: Yeah, there's a slight difference. I'll answer directly in a sec. But there's a slight difference.
[00:46:17] Des Traynor: the knowledge based thing is real. And it's definitely, if you give, I'm sure, crappy code will get you more crappy code. And in our world, the wrong docs will get you the wrong answers. The,there are some differences in that. Like I think, people more naturally, how would you say, maintain their code base perhaps than they do,their knowledge base, which can actually sit on a shelf somewhere.
[00:46:38] Des Traynor: it's also the case that, that, I like most companies. Like they'll give a good talk about style guide and tone of voice. But actually, if you go and read 10 different docs by 10 different support people, they're written in 10 different styles and it's totally fine. Whereas I think that companies try to, through linting and other stuff, try to enforce a kind of standard, a bit more of a standard.
[00:46:56] Des Traynor: when I think, what would it look like, to, let's just say, train up or take as input a company's entire code base, like I don't know, but what I'd say is generally speaking, you get more of what you give. So the challenges you'd have is, let's say your early code was shit and over time you got really good.
[00:47:13] Des Traynor: If it's a 80 20 shit code versus good, you're going to, advance or propagate that is one worry I'd have. Secondly, I worry how do you perform like cultural transformations? cause if the, if you're saying, Hey guys, we need to really like care about performance from this point onwards.
[00:47:28] Des Traynor: Like you could possibly try and prompt an LLM to be like, let's care about that. But in general, it's, if it's learned entirely off what it's seen so far, it's just going to keep doing what it's seen so far. And then there are like, I guess there's other stuff where going back to our earlier point, if you're asking it to do something that it hasn't seen at all before, that has no predicate in the existing codebase.
[00:47:48] Des Traynor: It'll probably end up like riffing or, doing its best, but they'll probably be like some, it'll probably take you a new direction so you might not have gone, which is sometimes good and sometimes very bad. Yeah. Yeah. It's a challenge. Like in general.
[00:48:01] Guy Podjarny: Interesting. Yeah. to think about how would, how to address that in product, over time to, allow the edges of creativity to shine at the same time, maintain predictability, which I guess is also true in humans, right?
[00:48:13] Des Traynor: Yeah, it's so much of what we see. you look at say Figma AI, right? Like the ability to draw interfaces. There's two different ways people respond to that. Some people are like, Oh my God, I can't believe you're going to take away all my like pointy clicky pixel playing. And other people are like, hallelujah.
[00:48:29] Des Traynor: You mean I don't have to drag boxes around for like 25 minutes just to get to the point where I designed something new in the world? That's often like what, the argument we'll make we will remove all the undifferentiated heavy lifting. if you might remember, DHH's Rails demo, look at what I'm not doing.
[00:48:45] Des Traynor: The reason people were excited by that, I think it was like a blog or something he stubbed out in 10 seconds. The reason people were excited because all the undifferentiated, boring shit in programming was taken away or made very, again, with bits of Ruby or whatever, so that you could actually do something new in the world.
[00:48:59] Des Traynor: And I think you could argue that,the proper adoption of, AI native code generation would mean that like people only write semantically deep, really meaningful, lines of code that really matter to the true nature of the product that they're building, which paradoxically means that code completion will become less useful, right?
[00:49:19] Des Traynor: The reason for that is because,if AI is generating like 90 percent of the undifferentiated shit. The piece it can't generate has no precedent. So that's, and I would say the same thing. Like an interesting thing that happens when people adopt Fin is the average handling time of their, and the number of tickets dealt with per day of their support team goes up, not down.
[00:49:40] Des Traynor: Why is that? Cause all the easy shit's gone.and what that means is, so what your humans are left with is the stuff that really requires human intellect. And I think you'll see this in code generation as well. We'll generate all the easy stuff, and then what's left It's the stuff that's not easy to generate.
[00:49:54] Des Traynor: There you go. You better start writing it. Yeah.
[00:49:56] Guy Podjarny: And that story kind of repeats in AI a fair bit, which is sometimes people need the simple tasks because they want that as a break. Yeah. So we talked. a lot about some product learnings and about building a product that's AI powered. I actually talked about adapting the dev practices as well.
[00:50:12] Guy Podjarny: Do you feel the same urgency around becoming AI powered? Like, how do you think about your own dev team and your team beyond dev as well, using AI powered tools? Do you feel the same urgency about it? Have you done anything? What's your view there?
[00:50:29] Des Traynor: I think, all of our engineers or a vast majority of them, have and use things like Copilot, so I, I believe and I'm sure like, one challenge for us is just that we're a little bit at scale. Like we're like a thousand people, we have substantial infrastructure, etc.
[00:50:45] Des Traynor: So like a lot of the cool advancements, whether it's like AI powered, let's just say end to end testing, like an AI version of Selenium and all that sort of stuff. I think all of those are coming out. These days, but they might just need to mature a little bit before we can actually properly adopt them.
[00:51:00] Des Traynor: But I think there's definitely, I don't know if it's urgency so much as well as desire to like to get on board with being an AI, an AI first, let's just say R&D or product org in general. We also use there's a lot of other areas like AI summaries of meetings, AI, AI, analysis, like one of the things we look at a lot is we got like thousands of feature requests.
[00:51:20] Des Traynor: We use AI to roll up all that. So we use it in all the obvious ways, or at least what I hope are obvious to most people on our marketing side, I would like us to use more of it, and certainly on sales. I think there's probably opportunities as well, as it relates to like either, qualification or prospecting or any of that.
[00:51:37] Des Traynor: There are great tools out there. I've invested in a chunk of them. things like 11x for like basically an SD org in a box type products. There are, there will be like end to end marketing tools, which will like start campaigns, turn on AdWords, send prospects, emails, all that sort of stuff.
[00:51:53] Des Traynor: I don't think we're adopting anything yet, amazing. There, there's other stuff that we're looking at, like things like Synthesia for like synthetic video, avatars, all that sort of shit. I think that, that will also be like a part of the future. What do I feel urgency there? I think I'll, I probably want to make sure that we're moving as fast as is, setting out.
[00:52:10] Des Traynor: I should be clear, I don't control these areas. there are a lot of people make these decisions, but I hope we're moving as fast as is sensible. I also know that a lot of these tools, a lot of the AI tools, if we're being really honest, they don't have in my mind enough discipline regarding, testing and making sure that they
[00:52:28] Des Traynor: both deliver the outcomes and don't deliver negative outcomes. Like quite often, I'm sure all of your listeners have had this experience. They sign up for some tool that talks a big game in the marketing website. They download it. They try to do the thing, whether it's generate the report or write the email or whatever, and it barfs.
[00:52:46] Des Traynor: And I think you know,you have to be cautious in your adoption of AI. Especially if you're not scale business, because the last thing you want is it is like a shitty email going to like
[00:52:58] Guy Podjarny: it's interesting. Cause it's the same, it's the same comment on, you don't know whether the product works or not that you talked about when you're building product.
[00:53:07] Guy Podjarny: The same as a consumer, right? You're consuming a product. You don't know if it works. The fact that it worked in the demo, the fact that it worked five times doesn't mean it would work a sixth. And so you have to have
[00:53:18] Des Traynor: It's 100% that Guy, but it's also yeah. Having seen a lot of these companies up close and personal, I don't,I don't have conviction that a lot of them have enough, discipline or hygiene around what they will roll out, and I mean that genuinely, as in, it's very tempting to be like, hey, Claude, Claude's new Frontier model just dropped, stick it in and pick go and see what happens, right?
[00:53:40] Des Traynor: If there's real customers on the far side of that, people are going to get affected, in in not so great ways, and that's no fault of Anthropic, it's just, I don't, I'm not sure that all these like white hot new super scaling AI startups are applying enough discipline to the products that they're rolling out,
[00:53:57] Des Traynor: and because of that, I think it's probably riskier in some areas. We put a lot of work into making sure people can evaluate Fin before they go, we really want people to feel comfortable about what's going on in front of their customers, but a lot of these products don't give you that opportunity.
[00:54:10] Des Traynor: They just say, import your contacts and we'll do the rest. And you're like, I don't need that in my life.
[00:54:17] Guy Podjarny: Yeah. Hope for the best is the idea. Yeah, exactly. That's before you even opened up authorization or, any of those types of, concerns.I want to talk a little bit about attended versus autonomous, and then maybe Finish off with some AI native chat.
[00:54:31] Guy Podjarny: You mentioned Fin a bunch of times as an autonomous agent, and I think of it as that as well. How do you think about autonomy, in, in the world of AI, even for you, for Fin, what's, what does it mean that it's autonomous? And I guess maybe to play devil's advocate, like for, if a customer is there, a customer is customer, right?
[00:54:51] Guy Podjarny: Your customer's customer is there and says, yes, you've addressed my problem or not. does that qualify as autonomous? How do you delineate those?
[00:55:01] Des Traynor: Yeah. So, I will get to the answer, but I'll just give a bit of structure here. If you think about any given product, it's like a set of workflows.
[00:55:08] Des Traynor: So like customer support, it has the triage workflow. It has the handle a customer query workflow. It has the updated engineer workflow. And you log in and your dashboard is usually just a nav bar that takes you to various different workflows. So the first wave of AI products, and indeed the first stuff we shipped, which was like, hey, automatically summarize.
[00:55:26] Des Traynor: That just takes a single step out of an existing workflow and does it automatically. And sometimes you can take two or three steps out of workflow and do those automatically, but it's still an attended workflow. So as in somebody is still, going and copying the summary and pasting it and going over here and clicking that, and they might get a suggestion to do this and they'll accept the suggestion and they'll go,
[00:55:45] Des Traynor: but that's just augmenting human capacity. And, and a lot of the like first round of AI, CodeAI, NotionAI or whatever, they were just basically saying, here's a few cool tools for your team who already use this product. And that's what they were all just charging,and it's another $9 on the seat price,
[00:55:59] Des Traynor: cause it's we're basically augmenting the human capacity a little bit by taking away some of it. And I think that's that was the gen one of AI products that we saw happen over the first half of 2023.
[00:56:09] Guy Podjarny: Yeah. Still most of them today, probably.
[00:56:10] Des Traynor: Yeah. Yeah. A lot of them are still just that.
[00:56:12] Des Traynor: The next wave is let's take a whole workflow end to end. And what you'll see there is something like, Hey, one of our most common workflows is like optimize all of our ads or review all of our campaigns and cancel all the most, cancel all the least performant ones and double the budget on the expensive ones or the performant ones.
[00:56:29] Des Traynor: And if you can actually do all of those steps in a row, like the AI is handing over to itself, and it's like basically every single step is automated such that you basically just tell the users, don't worry, this product is always optimizing your odds. There is no UI for that. you just trust us.
[00:56:45] Des Traynor: We'll give you a report at the end of the month, right? That's like the Gen 2 products, which I think,you can think of as hey, a whole chunk of the job is now gone. That's great. We saved, we've saved massive amounts of human time here. It's at least equivalent perhaps to salaries or whatever.
[00:56:59] Des Traynor: And then the Gen3, which is where I think we get to like deep, proper, full, broad scale autonomy, is when you're replacing orgs. You're not replacing humans, but you're replacing orgs. So if you take 11x as one example, 11x is like an SDR team in a box. And I really do mean team. Like here's a funny example that I think this will distinguish the three worlds for you.
[00:57:20] Des Traynor: In one week, in mid 2023, I got pitched to three different startups, and this all in one week. Startup one on a Tuesday was, we'll be, we'll help your sales team write emails. We'll do the prospecting, we'll do the research, find the details, find a relevant intro point, and we'll write a great email for you.
[00:57:37] Des Traynor: The second one was like, you import your CSV, we'll research on all of them, and we'll send the emails too. Oh wow, if the second one exists, the first one doesn't, right? because it's, clearly that's The third one I got was, you just describe the customers you want, like VPs of engineering, and you can go, and we're going to go find them, find their email addresses, write the emails, send them out, set up the calendar of invites, and the rest of yours will just get f**king demos to be scheduled or whatever, right?
[00:58:06] Des Traynor: And you see,that's,the way this world is going, right? And I think, like the, you have autonomy of the human and you have autonomy of the org, and before that, the safe, the dip your toe type features is, let's just make it easy to do common repetitive tasks that humans have to do.
[00:58:20] Des Traynor: That's roughly how I model out the space.
[00:58:23] Guy Podjarny: Yeah, no, I think I understand and you're describing it once again, like with the completion and the context as a continuum and you say the reason to magic attended versus autonomous type lens fully, it's more about the size of the task.
[00:58:37] Guy Podjarny: You could say that you summarize emails autonomously, but it's not like a sufficiently.
[00:58:43] Des Traynor: Yeah, it's not what I call like an end to end workflow.
[00:58:46] Guy Podjarny: For the outcome, if the outcome is sufficiently, is the actual kind of goal of a workflow, then you can say that it establishes or completes this workflow autonomously.
[00:58:54] Des Traynor: Yeah, the way I think about it more is whatever business need is presented, does it get resolved without human intervention? And if the human intervention is just to click next and okay and next, that's still human scaled in some sense. You still need to have humans in the mix. So that's I haven't thought that out fully, but that's roughly what's in my mind is basically, when you properly automate the flow, no one's thinking about it or talking about it in the business.
[00:59:17] Des Traynor: It's like, why is the electricity working above my head? I don't know. I don't care. versus,something that's slightly more complicated might be like, Hey, every day I'm involved in checking my calendar and making sure about, it's got to do with the kind of thing happened without a human input.
[00:59:30] Guy Podjarny: Yeah. Yeah. I think that makes sense. So I guess maybe this leads to a good kind of close off of AI native. So if you were to decouple from today, and you were to imagine five plus years from now, 10 years from now, what is an AI native support process? Like how do we even consume, fulfill, support, in that world.
[00:59:57] Des Traynor: I honestly, I think the nature of service changes. I think you'll see, every product, every website, every service, every app will have, a concept of how you get help. And it'll probably possibly, it could be audio, it could be video, it could be video avatar, it could be human, it could be any of those things.
[01:00:14] Des Traynor: I think it'll be both proactive and reactive. it'll, if I log in and my credit card's about to expire, it'll be like, yo, your credit card's about to expire. Click here to fix it or whatever. it won't wait for shit to go wrong. It'll make sure that things go right.I think we'll, humanity will have gotten used to and expect sub second replies to things.
[01:00:32] Des Traynor: So the idea of thank you for your ticket. We'll get back to you in 24 to 72 hours will be like batshit. It will be simply like people expect to type problems and get answers. I think the, the specificity and detail of the answers will dramatically improve even from where it is today. So you'll get an answer that is better than a human would give you, can go into further detail.
[01:00:52] Des Traynor: and I think that the norm will therefore be like, Most things go right, so there's just a quality of using software will dramatically improve because I think you'll have like autonomous agents proactively improving the support experience and proactively improving the customer experience in general.
[01:01:09] Des Traynor: And, and I think when something goes wrong, It will, it'll have two meaningful outcomes. One is it'll get put right immediately. The bot will like, escalate, raise an issue, issue a refund, resend, or teach. Everything that's the needs. The bot will also ship you a CSAT at the end and be like, hey guy, how did I do?
[01:01:26] Des Traynor: Ran me five out of five or whatever. The bot will probably also do like an outcome report on the back end and be like, Guy's t shirt didn't arrive. Here's all the chain of events that happened. We think it's because we used this provider in the Middle East. We think the data didn't ship on time because they blah, it'll like do a, what's the word?
[01:01:40] Des Traynor: Like a post mortem almost in every incident. So I just think like software and systems will get dramatically better. Now, The question is, what's the rest of the world look like around all that? Cause you know,
[01:01:51] Guy Podjarny: And notably, I think, I think there's a very compelling kind of reality and I like it.
[01:01:55] Guy Podjarny: There's interesting questions on it that we'll have to unravel in another one, but I guess the one question just to ask on it is what's the role of a, as a support person in that world? Like you previously talked about how they elevate into the sort of the bigger problems. Does that continue?
[01:02:10] Des Traynor: I think they control the intelligence and they control the policies. So I think, okay, businesses will still have to have opinions about what type of customer service they want to deliver, what expectations they want to set, what brand they want to project. But in 10 years time, if you miss your Ryanair flight, I don't think AI is the reason why Ryanair is going to be like, no worries, we'll just put you on the next one for free.
[01:02:31] Des Traynor: Like I think that's an actual business policy of Ryanair. so like somebody needs to tell the bot that that's how the refund policy works. So I think it there'll be a master of policy, a master of intelligence, of tone of voice, of control, perhaps safety harnesses too, like there probably will still be a like escape switch for hello, Revolut, I have been hacked and all my money is draining from my bank.
[01:02:51] Des Traynor: You probably don't want like a chirpy bot replying to that one. so I think there'll be controlling the system and controlling the intelligence. but I'm also like, I don't, it's quite possible that like that the future support team is like 10 percent the size of the current ones we see today.
[01:03:04] Des Traynor: And that's just part of the evolution of technology.
[01:03:08] Guy Podjarny: Yeah. Yeah. I think it's very interesting to think about the roles that, that form and the power of them. And to an extent, the more power you have, it could be that there's, 10 times as many services on it. So the same number of people in the world and just fewer of them in every organization.
[01:03:22] Guy Podjarny: Absolutely. Yeah. Des, thanks for coming on and sharing kind of these perspectives on, on, AI, tools and,how to, They change development and support. And, yeah, I can't wait to see how Fin continues to evolve and solve all of our problems in all of the vendors that we use. Cool.
[01:03:38] Des Traynor: Same with Tessl. thank you very much for having me. It's been a fun chat.
[01:03:42] Guy Podjarny: Cool. And thanks everybody for tuning in and I hope you join us for the next one.
Podcast theme music by Transistor.fm. Learn how to start a podcast here.