Beyond Coding Assistants: Cursor as an API, Coding with Gestures and more with Patrick Debois

Episode Description

In this captivating episode of the AI Native Dev podcast, Simon Maple welcomes back Patrick Debois, a leading figure in the tech industry famously known as the "Father of DevOps." Patrick shares his transformative journey of integrating AI into his coding practices, discussing the evolution and impact of AI tools like GitHub Copilot and ChatGPT. With his profound experience in software delivery and automation, Patrick provides valuable insights into the role of AI in coding, covering topics from AI code generation to the significance of user experience and security in AI tools. Tune in to learn from Patrick's experiences and explore the potential of AI to revolutionize your software development processes.

Chapters

1. [00:00:00] Introduction to AI in Coding with Patrick Debois
2. [00:01:00] Early Experiences with AI Tools: ChatGPT & Copilot
3. [00:04:00] The UX Challenge in AI Tools
4. [00:07:00] Context and Customization in AI Code Generation
5. [00:12:00] Enhancing AI Tools: Patrick's Customizations
6. [00:22:00] Voice Interaction and Accessibility in Coding
7. [00:27:00] Security Concerns with AI Tools
8. [00:31:00] AI Tools for Frontend and DevOps
9. [00:34:00] Patrick's Recommendations for AI Tools
10. [00:35:00] Conclusion and Future of AI in Development

The Evolution of AI in Coding

Throughout the podcast, Patrick Debois highlights the transformative role of AI in coding, emphasizing its ability to automate code generation and standardize workflows. "There's definitely a link between DevOps doing a lot of automation but particularly excited about how we can now automate more of our code generation," he mentions. This connection between DevOps and AI is pivotal, as both focus on enhancing efficiency and collaboration within software teams. The integration of AI in coding not only streamlines processes but also enhances the consistency and security of software development. AI agents act as tireless assistants to developers, boosting productivity by handling repetitive tasks and allowing developers to focus on more complex challenges.

Patrick illustrates that AI is not just a tool for automation but a means to elevate the quality and reliability of the code, creating a more efficient development environment. By utilizing AI, developers can ensure that their code adheres to best practices, reducing the likelihood of errors and improving overall software quality. This is particularly important in today's fast-paced development landscape, where time-to-market is crucial, and maintaining high-quality standards is essential for success.

The evolution of AI in coding has been marked by significant advancements in technology and its application in software development workflows. As AI tools continue to improve, they offer new possibilities for automating tasks and optimizing processes, ultimately enhancing the productivity and creativity of developers. Patrick's insights underscore the importance of embracing AI as a valuable resource in the development process, encouraging developers to explore its potential and leverage its capabilities to achieve their goals.

First Encounters with AI Coding Tools

Reflecting on his initial experiences, Patrick recounts his journey with AI tools like ChatGPT and GitHub Copilot. He describes the transition from basic code completion to more advanced chat-based interactions, noting significant improvements in AI's ability to generate executable code. "Initially it was like, okay, here's a block of code and it rarely ran. Now it's a lot more refined on immediately getting the code into your editor," Patrick explains. This evolution highlights the rapid advancements in AI technology and its growing effectiveness in assisting developers.

The early iterations, while not perfect, laid the groundwork for the sophisticated tools we see today, enabling developers to interact more naturally with AI and leverage its capabilities to enhance their coding practices. As AI tools have evolved, they have become more intuitive and user-friendly, allowing developers to integrate them seamlessly into their workflows. This has led to increased adoption and reliance on AI tools, as developers recognize their potential to improve efficiency and productivity.

Patrick's experiences with these early tools provide valuable insights into the challenges and opportunities presented by AI in coding. By understanding the evolution of AI tools, developers can better appreciate their capabilities and limitations, enabling them to make informed decisions about how to integrate AI into their workflows. This understanding is crucial for maximizing the benefits of AI tools and ensuring that they are used effectively to enhance the software development process.

The Impact of User Experience (UX) in AI Tools

User experience plays a crucial role in the effectiveness of AI coding tools. Patrick emphasizes the importance of maintaining developer flow and improving interaction with AI tools. He highlights Cursor's approach, which includes features like multi-file changes and conversational aspects that enhance coding efficiency. "If you're coding and you're somehow in the flow, right? You want it to be snappy, right? So you want it to be, hey, I need information when I want it," Patrick explains. The design of AI tools should prioritize seamless integration into the developer's workflow, reducing friction and enhancing the overall coding experience.

A well-designed UX allows developers to focus on their tasks without unnecessary interruptions, making AI tools an invaluable asset in the development process. By providing intuitive interfaces and streamlined interactions, AI tools can enhance the developer's ability to work efficiently and effectively. This is particularly important in complex coding environments, where maintaining focus and momentum is essential for productivity and success.

Patrick's insights into the importance of UX in AI tools underscore the need for continuous improvement and innovation in this area. By prioritizing UX, developers and tool creators can ensure that AI tools are not only functional but also enjoyable to use, encouraging widespread adoption and integration into development workflows. This focus on UX is crucial for maximizing the potential of AI tools and ensuring that they deliver the desired benefits to developers and the broader software development community.

The Role of Context in AI Code Generation

Providing context and rules is essential for improving AI-generated code. Patrick discusses the balance between detailed prompts and context to achieve desired results. "I have a base set of requirements I put in my project that I reuse across projects," he explains. By supplying AI tools with the necessary context, developers can refine prompts for better AI output, making coding more intuitive and reliable. The ability to specify guidelines and rules ensures that AI-generated code aligns with the developer's standards and expectations.

This customization capability is crucial for integrating AI tools into diverse projects and maintaining code quality and consistency. By tailoring AI tools to the specific needs and requirements of each project, developers can ensure that the generated code meets their expectations and adheres to best practices. This is particularly important in complex projects, where maintaining consistency and quality is essential for success.

Patrick's experiences highlight the importance of providing context and guidance to AI tools to achieve optimal results. By understanding the role of context in AI code generation, developers can leverage AI tools more effectively, ensuring that they deliver the desired outcomes and enhance the software development process. This understanding is crucial for maximizing the benefits of AI tools and ensuring that they are used effectively to support the developer's goals and objectives.

Enhancements and Customizations in AI Tools

Patrick shares his customizations to Cursor, integrating it with his workflow by creating an API to compare different AI tools. This innovation allows him to use multiple AI tools to achieve optimal coding results. "The first one is, so having looked a lot at the pipelines and, did my fair share of trying to make it work," Patrick shares. By leveraging an API, he can seamlessly switch between tools like Copilot and Cursor, harnessing the strengths of each to enhance his coding practices.

This approach demonstrates the flexibility and potential of AI in software development, encouraging developers to explore and innovate with the tools at their disposal. By customizing and enhancing AI tools, developers can tailor them to their specific needs and requirements, ensuring that they deliver the desired benefits and outcomes. This flexibility is crucial for maximizing the potential of AI tools and ensuring that they support the developer's goals and objectives.

Patrick's experiences with enhancing and customizing AI tools underscore the importance of innovation and experimentation in the software development process. By exploring new possibilities and pushing the boundaries of what's possible, developers can unlock the full potential of AI tools and drive progress and innovation in the software development industry.

Voice Interaction and Accessibility in Coding

Exploring voice-controlled coding, Patrick discusses his innovations using gesture recognition for enhanced productivity. By integrating OpenAI's real-time interaction, he improves the coding experience, making it more accessible and efficient for developers. "I made a little thing that watches me on the camera. And when I did this, instead of typing, it was listening to me in voice mode," Patrick explains.

This approach allows developers to interact with their code through voice commands, reducing the need for traditional input methods and enabling a more natural and intuitive coding experience. Voice interaction not only enhances productivity but also opens up new possibilities for accessibility in the development process. By providing alternative input methods, developers can ensure that their workflows are inclusive and accessible to a wider range of users.

Patrick's innovations in voice interaction and accessibility highlight the importance of exploring new possibilities and pushing the boundaries of what's possible in software development. By embracing new technologies and approaches, developers can enhance the coding experience and ensure that it is accessible and inclusive for all users. This focus on accessibility and innovation is crucial for maximizing the potential of AI tools and ensuring that they deliver the desired benefits and outcomes.

Security and Trust in AI Tools

Security is a key concern when leveraging AI tools with screen access. Patrick discusses the security implications of Anthropic's screen scraping technology, emphasizing the balance between functionality and security. "I think it does in the way that I would run this on a virtual desktop and be sure what I give access to it," Patrick notes. Ensuring the privacy and security of sensitive information is paramount when using AI tools, and developers must take precautions to protect their data.

By implementing robust authorization measures and maintaining control over what AI tools can access, developers can leverage AI effectively while safeguarding their information. This is particularly important in today's digital landscape, where data breaches and security threats are prevalent. By prioritizing security and trust, developers can ensure that their AI tools are used safely and effectively, minimizing the risk of unauthorized access and data breaches.

Patrick's insights into the importance of security and trust in AI tools underscore the need for continuous improvement and innovation in this area. By prioritizing security and trust, developers and tool creators can ensure that AI tools are used safely and effectively, delivering the desired benefits and outcomes while minimizing the risk of security threats and data breaches.

AI Tools for Frontend and DevOps

In addition to his explorations with AI in coding, Patrick highlights additional tools like V0 for frontend development and runme.dev for DevOps commands. These tools integrate seamlessly with existing development practices, showcasing the versatility and adaptability of AI in enhancing software workflows. By incorporating AI tools into various aspects of the development process, developers can streamline tasks and improve overall efficiency.

Patrick's experiences demonstrate the potential of AI to transform not only coding but also other critical areas of software development, empowering developers to achieve more with less effort. By leveraging AI tools, developers can enhance their workflows and ensure that they are working efficiently and effectively. This is particularly important in today's fast-paced development landscape, where time-to-market is crucial, and maintaining high-quality standards is essential for success.

By exploring and embracing AI tools for frontend and DevOps, developers can unlock new possibilities and drive progress and innovation in the software development industry. Patrick's insights underscore the importance of exploring new possibilities and pushing the boundaries of what's possible, encouraging developers to embrace AI and its potential to enhance their workflows and achieve their goals.

Conclusion

Patrick's experiences with AI-assisted coding tools demonstrate their potential to transform software development workflows. By exploring and adapting these tools, developers can enhance productivity and innovation in their projects. Patrick's journey serves as an inspiration for developers to embrace AI and its possibilities in coding. As AI continues to evolve and improve, it presents exciting opportunities for developers to push the boundaries of what's possible in software development, paving the way for a future where AI and human creativity work hand in hand to drive progress and innovation.

In conclusion, Patrick's insights and experiences highlight the transformative potential of AI in coding and software development. By leveraging AI tools, developers can enhance their workflows and achieve their goals, driving progress and innovation in the software development industry. As AI continues to evolve and improve, it presents exciting opportunities for developers to explore new possibilities and push the boundaries of what's possible in software development. By embracing AI and its potential, developers can unlock new possibilities and drive progress and innovation in their projects, ensuring that they are working efficiently and effectively to achieve their goals.

Full Script

Patrick Debois: [00:00:00] I think an LLM combined with tools and combined with agents, they give you massively better results. So I made a little thing that watches me on the camera. And when I did this, instead of typing, it was listening to me in voice mode. And so now I can switch between two modes and I'm just sitting on the sofa.

Simon Maple: You're listening to the AI Native Dev brought to you by Tessl.

On today's episode, joining us once again, is Mr. Patrick Debois, the father of DevOps, but this time around, we're not going to be talking about DevOps, Patrick, it's going to be a nice, refreshing change for you, right? We're going to be talking about the journey that you had using coding assistants, learning to develop [00:01:00] using AI as part of your workflow.

I've been watching from afar, sometimes quite close, seeing how you've been playing around with various, these editors tooling systems. It's been really fun to watch the passion that you have in, AI in general, but also how it can actually help you build software. So let's jump straight in.

Patrick, father of DevOps and many other accolades. But today coder Patrick. Tell us a little bit about your background, first of all, briefly in the, why don't we do the coding space? Cause I think everyone knows you from the DevOps space. Coding, Patrick, what does that look like?

Patrick Debois: There's definitely a link between DevOps doing a lot of automation but particularly excited about how we can now automate more of our code generation.

And I see it also a way to standardize the way we work, make it more consistent. make it more secure. There's a loads of opportunities that machines are tireless helping us to do this. Where in the past, part of my job was VP engineering, scaling this out to all the engineers, the best practices.

Now I have [00:02:00] AI that can help developers to do this all day and kind of get better for the good cause. So yeah. Yeah.

Simon Maple: Awesome. So let's go back to the early stages of the first time trying out AI as part of your coding style what did you use? Was it copilot?

Was it earlier than copilot?

Patrick Debois: I think at that time, ChatGPT came out. The first plugins were actually more like ChatGPT pilot so there were like VS codes. I know there was GitHub Copilot, but at the time I was maybe not doing too much of the coding, but there was a difference between doing it in line in your editor by commenting and making sure it was generating or auto completion thing.

And the next thing that came about a little bit with ChatGPT was the fact that we could ask it questions and to do things. So that kind of chat pane on the side helped to do things. We've come a long way. Initially it was like, okay, here's a block of code and it rarely ran. Now it's a lot more refined on immediately getting the code into your [00:03:00] editor and doing there.

But that kind of was the switch, the inline editing was not always working too well at that time. But I'm talking like, early, a couple of years ago.

Simon Maple: Inline editing being changing existing code or?

Patrick Debois: No, more like you write a comment and then it auto completes to the comments. And then the auto completion was a little bit better, but we called a completion before.

So that was helpful, but not good enough at that time. Now it's tremendously better, but that was for me. It's nice, but it sometimes gets in my way. And then the chat came. And that was nice, but it also felt like, why am I copy and pasting this all the time? You felt like going out to stack overflow, coming back, it was not exactly what you needed, but it was a new way, a second way that we were Interacting with LLMs or AI models to generate code that we would do and generate in our code basis as well.

Simon Maple: It strikes me here and thinking back to a blog post, actually that Guypo wrote, which we [00:04:00] mentioned on the monthly a few weeks back. He wrote about this matrix of change of trust. And it's really interesting to see here. And we see this with a number of tools whereby various tools, let's take ChatGPT.

In fact, let's take something like Claude. With artifacts is a great example you can use it as an interface to ask questions to write to make changes to code and that's fine so long as you don't want to change code and the change piece there. The change in workflow is massive. Because we're no longer an IDE.

We're not writing code. So the way in which we're interacting is a very big change. Then there are other tools like Cursor then allow for that conversational aspect was it, what did I call it in Cursor? It's compose composer. That's the composer.

And it allows you to say I want to change this, or I want that, multi file change, whatever it is. But it pushes it straight back into the IDE. From the change perspective while it's considered an advancement because it reduces the copy pasting that we need to do.

It [00:05:00] pushes us back into more the existing workflow rather than potentially what could be in the later down the road. Do you think we see adoption for various tools, maybe like more like the Cursors and things like that, because they're just far closer to where we are comfortable working and is that going to almost like trump various other tools like a, like whether it's from Anthropic or ChatGPT and things like that.

Patrick Debois: Interestingly enough, the chat interface, when the ChatGPT plugins came, they already had things like, oh, can you refactor this code? Can you explain me this code? All those kinds of functionality came extra, but they fell on the side. And as you say, now we can do this like with our code base and it's writing these things in the code base.

I think, whether you using Codium or Cody or like a lot of people have looked at it like this way of doing this together, I think what Cursor cracked was UX problem of us expressing what we wanted, the intent we [00:06:00] wanted, changing that code. That's something, we could have done with copy and paste, but they streamline the process, they optimize the process.

And then the next part they did is when we're reviewing all the edited code, a brilliant thing they did is they're not just showing us the diff. They're not just showing us the inline thing. They're showing us a condensed version of what is changing. So they're helping us to validate the code that has been changing.

And I think that's the biggest nice aspects that A it brought us into the flow Hey, we're writing and we're expressing, is it good? And then we were reviewing. So all these kinds of things got polished in the workflow.

Simon Maple: And I saw that and I think it was a LinkedIn tweet that, sorry, A LinkedIn tweet.

Yeah. LinkedIn post. A LinkedIn post. A LinkedIn face, Insta Post that you did? Yes. Excellent. And . Yeah. Where it was basically three things. You had the discussion, the diff so literally like a Git diff right. What was removed? What was added, and then a shorthand. These are the things that it's describing.

It's done so effectively, almost like a code review on what it's done. In [00:07:00] case there are big changes. So it's the UX that kind of cracked that. And was that one of the big problems when you were looking at whether it's copilot or something in the early days? What was the killer feature?

I guess you were missing in those early days to move to the next step. And what was that next step?

Patrick Debois: If you're coding and you're somehow in the flow, right? You want it to be snappy, right? So you want it to be, hey, I need information when I want it. I want to review and the first time that cycle with Cursor felt like, oh, this is, It keeps me into the flow.

The other things were like, okay, now I go somewhere on the side and in the past, it might've been a stack overflow. Then it was a chat, but now everything came into this is the problem I focus on. This is the next thing I need to think about. Give me the result. Oh, this is the next thing you need to think about.

And kind of that amazing flow started to happen. It doesn't mean that the answers were perfect yet. But then the brilliant thing was if you're doing code completion you get one completion and you're like, ah, it's not what I wanted. And you're like, okay escape, because it's not what you want.

Cursor [00:08:00] brought multiple line completions and multi completions as well. So they brought that into steroids. But then when you use compose, you just say I want a file, All that does it like this and, print out that and it gives you an answer. And when you're not happy with the answer, you just say change this.

Yeah. And then you see, so you're having that conversational aspect, not on the chat, but immediately with your code and with the whole code base. So not just the one file, it understands everything of your code base. So it was aligning multiple things that were there before. In a kind of streamline the process that me as a developer, I found myself more and more just not typing the code, but just nudging the LLM or the tool to say this is what I want.

No, a little bit more like that. Oh, that's good. That's not good. So that reviewing process got like insanely different.

Simon Maple: So let's talk about that in terms of an accuracy point of view.

It's funny when you talk, a lot of people who are still a little bit [00:09:00] skeptical about AI coding or just not what I'm trying to push it off a little bit too much.

Very often you hear different levels of acceptance in terms of the, how accurate the suggestions are, whether it's what they want, whether it's even the style of implementation that they want, maybe they want to do it in a different way. Is it something that you care about enough in terms of the way?

It's something's been implemented. Do you care about it? Like I wanted to use this type of set or collection or whatever versus this one. Do you care about that anymore or is it more the functionality that you want it to achieve that you're actually mostly discussing with the chat?

Patrick Debois: It's a dilemma. Do we still care about the code, how it looks like? Or do we just care?

Simon Maple: Is it your code anymore or is it your description of what the code should do?

Patrick Debois: I think my argument would be is that I still care in a way that when it fails as a human, I still want to understand how it works and why it works.

And navigating my code base could not just be a bunch of files naming a. py [00:10:00] and b. py. So there is a cognitive piece that I want to still understand what it is. What I see myself more and more do is I have a specific problem that I want to solve. I start off almost with a blank kind of project. And I iterate on that using Cursor and compose.

So it's like a very well defined thing. And then I bring this into my bigger project. And then in that bigger project, I have more rules specified on how I want things. So one of the cool things of course is, besides taking the whole code base, having the compose was that you describe a little bit of a rules that the generation should adhere to.

Like I'm coding in react and my front end. I always want to have an API test written in open API spec. I want all those things. So more and more, I could write my set of rules, like they call it Cursor rules and Cursor while generating takes that into as not just the [00:11:00] context of your code base, but it also takes that like your guidelines that it wants to take to generate the code that kind of sometimes gets in the way for a smaller project.

Yeah. But for a bigger project. That's how you make it consistent across the whole code base.

Simon Maple: Yeah. And I presume, even with the smaller projects, there will be basic things that you will want to inherit from general development. Correct. And so we'll start with that.

So in terms of how we provide this type of context, there'll probably be some various system prompts that we'll always want to use. Are you adding these, are you creating these as like files, like supporting files that you include and reference in the composer? Or is there another way you include them?

Cause you obviously you don't want to provide too much context or overwhelm it with huge number of instructions. So you want to be almost a little bit surgical and saying, I'm working on the backend here. I want you to use these backend requirements essentially, when you're creating these suggestions

Patrick Debois: So I have a base set of requirements I put in my project that I reuse across projects.

And then while doing [00:12:00] specific stuff, I add specific files. Let's say I'm doing front end stuff. Yeah. Then I'm specifically saying at front end specs, do this. So that kind of is a balance between. What I'm trying to achieve in the code and it's the same thing with adding documentation, depending on what I want to bring in is something that I decide on the moment.

Simon Maple: Yeah.

Patrick Debois: So if you're specifying too much, it sometimes gets upset or it has conflicts and the code's not always that good. Yeah. So I, I do tweak this in small bits when I'm asking it to generate the code while I'm doing this.

Simon Maple: And if you didn't add that kind of like context or like reference these various files, presumably you would still get to the same end point, but you would then have to iterate on its response more and more into the prompt.

Yeah. So you have more of a discussion going forward. So actually it's not quite how I would do that. I want to do it like this. It's interesting. There was a good quote from Des Traynor actually. He said something along the lines of the, our [00:13:00] willingness or our ability to want to write short prompts is robbing us from the need to be able to provide it with the detail it needs in order to give me the right answer. And it feels like this is effectively doing both. It provides us with the longer prompt the information it needs to say, this is what I want you. This is how I want you to provide me the good answers, but we're still being able to short prompt it.

We're still being able to say, look, I want this, but we don't have to say exactly the detail of how we want. So you're almost like adding that little bit of background knowledge into the responses.

Patrick Debois: But it's like humans. When I say do a certain thing. And we have a lot of history and a lot of context are already, with two words. And so if I bring in a lot of context, in my editor and I provided with a set of rules, it can infer from the things that I want more information. A lot of the tools have like prompt compilers, they convert your simple prompt to a more complex prompt to get better results.

So this is definitely one of the ways they're getting, but the only way they can do this, if [00:14:00] you give them more context or specificity.

Simon Maple: Getting that detail out of your mind, actually out of your mind. Yeah. Yeah. Interesting. Another tool that you used was Aider, right? You use that before Cursor or, yeah, that was before Cursor, preCursor.

Patrick Debois: So Aider is one of those open source tools to generate code. It's a little bit different from other tools in that it is more CLI based. So their approach was, you're coding, but at the same time you're running it like a demon in your CLI and you give it what you want. Similar to, a chat and it changed the code.

I think the quality was really good at what it was generating. But there were a few quirks that worked against me. One is it was auto committing codes that he thought was good. Which kind of ah of course you could disable it, but it was like, one of those things do we really want this?

They had some cool things. Like I can talk with voice to it. And so I can have that iteration look more on a verbal thing, which is also something I tried with Cursor to make it like talk and have that conversation because I can just sit on the bench and talk [00:15:00] to it. Why are we still typing?

And it's almost like that peer programming thing as well, right?

Simon Maple: We actually, when, a lot of the time when you just sat there at a keyboard, sometimes, you skip through various checks in your mind. Whereas actually, when you're trying to say something. You're thinking it through more and so as a result, you yeah, I guess you, you like fine tune exactly what you're asking for, which is interesting.

Patrick Debois: But Aider has, what's nice is they blog a lot about how they do things. So they talk about how they do their internal diffing and sending small chunks to the LLM. So it's not overloaded with the whole code base. So they kind of work from there. This is also something Cursor does is they almost prioritize which part of the context are useful to solve your part of your problems.

So which is like optimizing for speed and context, because if you give them too much it's just a problem. Yeah. Something that I recently added is when the code is being generated, they have the notion of an architect reviewing your code. It's just the LLM taking another identity, but it critiques your code that's being generated.

So you [00:16:00] don't have to do it yourself, but obviously latency plays into a big thing there because you want your code quick and you don't want to wait like for a minute. So there's a lot of caching optimization of the tools happening, but they're very open about it. And it's pretty cool. You can learn a lot about the source.

Simon Maple: Yeah. How good would you say those types of tools are truly understanding code? And do you feel, I see a lot of tools either mixing AI, so LLM versus a symbolic AI or something like that, or using static analysis of codes to better understand and model the architecture or the AST of your code, what's your experience been in those various approaches in truly providing good feedback versus just a very general feedback which can be hit and miss.

Patrick Debois: It's one of those conceptions that people, they'll look at ChatGPT, they ask it to generate something and they don't like what they see. And they dismiss the whole option. I think an LLM combined with tools and combined with [00:17:00] agents, they give you massively better results, but what could go into some of the tools you mentioned that when it's cutting up your code, you don't want it to be cut in the middle of a sentence or a line of your code.

It needs to understand your code. That's why people are using AST. So the syntax tree, abstract syntax tree to parse things and chunk it up to the right pieces. So that's one layer of kind of improving.

Simon Maple: Which I was guessing when you were like talking about when the tools use various parts of the project or the code as context, because it knows those are relevant pieces.

I was wondering if it was using something like an AST to work that out.

Patrick Debois: Most of them are doing this to correctly. Divide the pieces in the right pieces. They need to look up or they need to change. So that's one thing then imagine the codes being generated, what you want in like a code generation pipeline, you want it to somehow specify and generate the tests that [00:18:00] it's doing. One of the tests could be like, I take the prompt and I convert it somehow into a functional test. Some of, I forgot the name, but some of the code editors actually are doing this first, the right to test first. And they help you write the tests and then they have the LLM write the code.

Because then you have your test harness and they have a better chance of success because the whole thing about TDD was actually refining what you wanted and how you wanted to see it. And then they give better hints to the, so that's another thing. There's loads of other tools. You can lint things when it's generated.

Is it valid syntax? That's an easy one. So you can feed that back. Hey, there was an error or that you can run the code if it's a piece, but then you need sandboxing, there's all kinds of security issues. Can I run this? But in a pipeline, some of those parts you could do and you can see whether it executes and does not have like issues or it actually runs the tests as also an execute and give that back.

There's [00:19:00] other things people have been putting in there, there's so many tools that, we use in our head, but these LLMs could use. Another one is code smells.

Simon Maple: Yeah.

Patrick Debois: So if I'm editing the code and my code smells increase, I do not want to have that code. So they feed this back again, people from code scene, they do a lot of stuff kind of detecting because a lot of the refactoring is you do not want your code smells to go up.

And so feeding this back into the loop Hey, it's not okay. So I'll generate another. And that's under the hood. It's not like the end user see it, but obviously the longer the pipeline gets longer the latency, but if you look at coding agents that just run and run, so not specifically in your code editor, they could do this all day, right?

So they can just run and see if it's optimized and go from there. There's other pieces of optimization that you have. Maybe you fine tune your model based on the code, based on your guidelines, deliberately inject errors [00:20:00] and see if they're caught and they're not good. So there's like a whole thing that people are doing this.

And the other thing that people have learned is when they do changes is that do smaller changes and not like the big chunk with all the context, but it's almost like the humans. They need to understand almost like the, your code editor view plus and minus one. And that's the context they give it.

And then they stuff a lot of context around it, but people are coming back from stuffing all the code into the LLM and see what happens.

Simon Maple: Too much context, basically overloading it with context. Let's talk about what you did with Cursor. I think that was super interesting beyond going beyond Cursor.

You made a couple of updates yourself built upon a Cursor. Tell us a little bit about some of the adjustments you made. First of all, what was the problem that caused you to do this? And and then how did you do that?

Patrick Debois: The first one is, so having looked a lot at the pipelines and, did my fair share of trying to make it work.

Sometimes you just want to use it. All right. And as Cursor was one of the better ones and like help that really give good [00:21:00] results. I thought it will be really nice to use that kind of whole logic inside of my own project. But Cursor is an IDE. It's not supposed to do any editing and so on. So what I did is I wrote a combination of a VS code plugin that allows you to type and an open CVE project that looks at your screen and see what our Cursor is open and then grabs the text and so on.

So eventually it was able to wrap the IDE to an API. So I could just say run this, give me the change that you generated and go from there. And my whole idea was that I can now compare Cursor with Copilot, with all the stuff, and I just use the best tool, but most of the tools were closed in a way that they're in the IDE and you can't do anything about it.

So that was my, I wanted to do this and then I could drive this code generation and add this as a tool to my agent.

Simon Maple: Yeah.

Patrick Debois: So I got the best of all breeds. So

Simon Maple: I guess there's a couple of next steps on that. The first one is you could effectively automate it to say, [00:22:00] look, this is what I want.

Use five tools go and do it in the way you want to do it. Provide me with the results. I can then choose which result I want to use. And so you're effectively running a team of developers that are playing off each other. And you can say, actually, this is the one I want, and these are the reasons why I don't want the others.

That's, one way. A second way could be to automate that and then say, is there a way I could actually take a tool that actually reviews that code and identifies the best of the options and suggests the best option to me and to why this was the best option around other codes. So there's like levels of you effectively being almost like a team lead of these AI developer agents.

Yep. And you're managing that team effectively.

Patrick Debois: Yeah, indeed. And then you have the reviewing agents, the code writing agent and Yeah. So that was my motivation to play and make that into an API. But

Simon Maple: How successful was that in terms of then being able to get results back that you would be more likely to accept.

Patrick Debois: It wasn't about being more likely to accept, but it was more like I can use it for my own tools.

Simon Maple: Right?

Patrick Debois: Yes. So that [00:23:00] was the usefulness. No, I didn't have to write the whole pipeline. Somebody wrote it for me and I can just use it.

Simon Maple: Consume that data however you want in your next step.

Patrick Debois: And the second tool was about, we mentioned this already before, I find myself reviewing code more and then commenting on the code.

And I first tried, I think, a voice recording assistive tool, because then I wouldn't have to type anymore, right? So I was like, okay. But I found that even VSCode has a voice assistant. It is only typing into the code editor. It was not typing into the Cursor editor or kind of all the fields.

Then I went to an accessibility tool with the voice. But then I figured I was doing this and my wife came in and started talking to me. I got like things on my screen. I do this? And sometimes I still needed that keyboard for the shortcut for activation. And I didn't want to do this.

I actually came across an auto tool called Cursorless. There's no kind of connection between both, but they [00:24:00] had a fascinating view about using sounds like to activate certain things. And then they would say little rats putter, which means click on the next word with a red p in the editor.

So they annotate the whole

Simon Maple: A little bit like what three words Yeah, so it's very complex.

Patrick Debois: So I figured like, how can I do this? And what I ultimately ended up with was I thought when you want to have somebody listen to you, what do you do? You raise your finger, right? Okay.

So I made a little thing that watches me on the camera. And when I did this, instead of typing, it was listening to me in voice mode. But then there's a difference between like, when I say do this command is it now typing or do I want it to execute?

Simon Maple: Yeah.

Patrick Debois: So I basically switch between this is typing and this is executing a command.

Simon Maple: Yeah.

Patrick Debois: And so now I can switch between two modes and I'm just sitting on the sofa saying, okay, this is the two fingers, compose, activate.

Simon Maple: Yeah.

Patrick Debois: It [00:25:00] brings up composer.

Simon Maple: Yeah.

Patrick Debois: Then I say, I'm typing what I want. And then I say, okay, so I'm just switching with my fingers.

Simon Maple: And if you turn the fingers round, you get very angry at the idea.

If you choose to, yeah.

Patrick Debois: Anyway, I'm a big fan of the OpenAI real time interaction. A lot of people think of it as clunky, but it is able to talk conversationally back to you in a very good way. And the other thing they solved was I can interrupt the speech. So it's not okay. Damn. No, it's reading off my whole diff.

Yeah. Yeah. That's not what I want. Yeah. So I had some shortcuts by saying open compose and it was listening for the words open compose and kind of activating it on that. But then I went a step further and I thought but the LLM can just look at my texts and I hooked it up with function calling.

And so if it thinks it should trigger the function, open compose, however, I say it or [00:26:00] whatever it determines that it's going to bring up open composer. So now I don't have to specifically say any more okay, this function is with that trigger word. If you don't know the trigger word, it's just listening on the conversation.

And I just can keep adding more tools. Oh, no, I'm going into my browser. And when I go into my browser, you probably want to take a screenshot and you want to upload it into my Cursor. Because then you can look and generate me the front end code that I wanted. So this is just a conversational way that I brought up.

Simon Maple: That's brilliant. I love it. And I would love to almost see, if there was a whiteboard behind us, for example, and we're drawing out some architecture and it can start recognizing the architecture and almost using that as a format that we can describe and get code written in certain places, there's a ton of things where it can take more than just the text input and actually take visual cues, like you just mentioned, as well.

Patrick Debois: Like you would ask what's on my screen?

Yeah. And it would tell you what's on your screen. Yeah. How do I improve it? Yeah. Here's two suggestions.

Simon Maple: Now [00:27:00] recently, certainly at the time of recording this, but maybe a few weeks ago when for people who are listening. There was some recent additions to Athropic's Claude whereby there is a beta that has been released or beta for our US friends, they effectively have some screen scraping technology that can identify what's in your screen.

And even interact. So you're effectively having a coder who can write into your IDE, whatever it is, even if there's no support for AI and things like that in that IDE, because Anthropic is now looking at your screen. How do you feel about that kind of thing?

Patrick Debois: First of all, I think it makes sense.

When I would use ChatGPT and I'm in a store, what do I do? I take a picture and say, okay, what do I need to know? And we've been in the IDE using this on the OS. It's very simple, similar to what I said. What's on my screen? How can you improve this? So now it's the next step. What they did is it's allowed to execute something locally.

Simon Maple: Yeah.

Patrick Debois: But it is just another tool. But [00:28:00] now they run this locally. I think this was an evolution. Also, Microsoft has a way that they record everything that ever went on your screen, and you're able to recall, remember that spreadsheet, Simon, that we worked on last Friday or something. Boom.

Simon Maple: Oh my God. Oh my God.

You didn't, you didn't see that spreadsheet. Yeah. Yeah.

Patrick Debois: No. So that kind of is in eminence in a way that the tools are just getting more on the local IDE and one of the things like with my tooling, I had to determine it's what's on my screen as well, because is it now Cursor or was it my browser?

So the machines were looking at the context. I was able to do this by looking at the process list and which was, when it was on my screen, but you could definitely look at other ways. The other thing I had to do was I had to look where a certain dial was and you could train a model, say, look at the screen, where's the certain dial, but then tomorrow there's something else.

And then if you would have an if statement list of saying if a dial than that, it will not be very [00:29:00] efficient. But if I would train a model that just understands all the software tooling we're using today and recognizes what kind of screen, what are the actions you're supposed to be taking when you have this in front of your screen.

Simon Maple: Yeah.

Patrick Debois: I know people are trying to make this even more self learning is they look at. Again, it's hooking it up with a tool. They look at it from the accessibility inspector, like they'll look at the buttons on the screen and they'll look like, are they clickable? Can we do something with that? Then they look at the hints of the screen, what they can do.

So this has been a little bit going on in a research project called OS world where they're already doing this in a virtual machine and you can spin this up and it gives you a desktop. And then they try all out all these scenarios, open my screen, open my browser, do this. And it's very similar to what I've described, like hooking up with the function calling.

Once you know what's there, then you can take the next step on kind of the those things, would I trust it?

Simon Maple: Security was the next kind of thing. Cause it, once it basically [00:30:00] has access to whatever's on your screen, you don't know what's going to pop up in your screen next. And how do you control what it should and shouldn't have access to?

And I think there's a wider problem here ,in and around authorization and things like that. And we've talked with people, Tamar Yehoshua, who, gave some really interesting thoughts about how Glean actually have their authorization as a step before the LLM working out what its users actually do have access to and then providing the LLM with what the user should access versus have the LLM try and work that out and understand that because you have to assume the LLM is like a user in that respect.

You don't have that control necessarily when you allow someone to completely look at your screen. So does that worry you at all from the security point of view, personal data, anything like that?

Patrick Debois: I think it does in the way that I would run this on a virtual desktop and be sure what I give access to it.

Obviously not everybody, is supposed to be tech savvy. At a certain point you [00:31:00] understand what is allowed and what isn't allowed. I think there is an inherent problem with the LLM in that there is hallucination. Yes, we're trying to reduce it, but we are never sure what it's going to do.

So we might restrict what it's able to do. That's our best thing. But then obviously if you want it to change document files. Where do you stop and then you get a tedious thing. Like when it's going to ask you every time, can I change this doc? And you're just going to say, okay, whatever. So that compliance is worrisome in a way that we know the technology inherently is not always repeatable, that it's always giving the same result.

Simon Maple: But it's incredible to be able to share context across applications. That's going to be so powerful, I think. And the intent as well to be able to say, Oh, I'm in this app. This is the level that the user requires this detail on, but now I'm in an IDE. I need to create actual implementation and go a much, much lower level to provide value.

Amazing. Before we close, I'd love to talk a little bit of more breadth in terms of when you are coding, what [00:32:00] other assistant tools have you played with that has really added value, most value to your coding experience, thinking here, whether it's test support, whether it's documentation support, whether it's anything else like that.

Patrick Debois: The one that I quite often use is V0, which is for a front end generation. Yeah,

Simon Maple: yeah.

Patrick Debois: So that's helps you. And I think it's nice that you specify the website that you want. They generate a code and you can iterate on that.

And now you can sync this back into your repo as well. So it's an additional tool. The Cursors and the like are not really focused on that, but it helps you with the front end kind of stuff. I've been exploring a tool called runme.dev, which is more of a tool on writing notebooks for DevOps commands.

Another way of not writing all the scripts, but keeping mixture of documentation in there. The documentation I usually do in line, like one of my specification is like whenever you write something, add documentation. So there's a lot more tooling around that. And the [00:33:00] other tool is now I have to think it was e2b if I'm not mistaking for sandboxing, I always mess the letters up.

So that was three, but for a sandbox execution of functions. So I get like a fast execution solution.

Simon Maple: All a little bit similar to a kind of like a replit style, would you say, or slightly different?

Patrick Debois: The replit, it is similar, but this one is very focused on sandboxing and just gotcha.

So it's easy to integrate in your code if you want something to happen. Yeah. And the other tool, gosh, I use all the tools that don't remember the name, but I think the other one was AI inspects and it allows you to do much like evaluations. But in your VS code, it's like a, somewhere a government project.

But what's really nice. And most of the evaluations, they take a long time. They're run somewhere else, like as Langsmith and so on. But this plugin allows me to keep it into the VS code and have the runs also across multiple [00:34:00] LLMs and see which one was the best for this kind of aspect. Nice. So those are a few add ons to the tools.

Simon Maple: Yeah. Amazing. And for people who are just thinking about starting off and wanting to try something what kind of, of all the tools that you've mentioned, is Cursor the one that you would start off with or something else?

Patrick Debois: I guess if you're very much into front end, you go to the V0.

Yeah. If you're a little bit more into the generate whole applications, there's another one called Bolt. new. And then Cursor, if you're day to day code writing back end front end. That's the one I would do. And then there's the obvious ones like Anthropic artifacts that help you as well for generating some of this stuff.

So those are a typical, tool base. But I think what's happening is that all the tools. They're learning from each other and hopefully, you have everything somewhere close into one tool which is interesting. For example, one of the next GitHub Copilots they will look at your test coverage and then as you just tests [00:35:00] also something you can use.

Everybody's exploring and maybe that was another reason why to build an API around it, because then I can use whatever tool for the right job.

Simon Maple: Amazing, Patrick, it's been a blast as always to have a chat and thank you for so much like very practical, hands on tips and tooling suggestions for people.

So No, no excuses now go out and have a play and see what you think. Let us know what you're using, what you're enjoying. Thanks again, Patrick, for joining us twice.

The first person twice on the, on the AI Native Dev.

Patrick Debois: My pleasure. In a couple of months, we'll automate it ourselves. Absolutely. During the podcast, like notebook LLM.

Simon Maple: Yeah, there we go. There we go. And we can just relax on a beach somewhere, watching our own jobs being done better than we could. Amazing. Thanks very much and join see you on the next episode.

Thanks for tuning in. Join us next time on the AI Native Dev brought [00:36:00] to you by Tessl .

Podcast theme music by Transistor.fm. Learn how to start a podcast here.

‍