Transcript — Welcome to AI Native DevCon (Tldraw AI Canvas Experiments)

⚠️ Speaker-label warning. This transcript has no per-speaker labels. The talk metadata names Simon Maple as speaker, but the spoken content reads as a Tldraw team member demoing the Tldraw SDK ("we're on Tldraw right now", "if you go to tldraw.dev", "our customers"). It may be that Simon Maple introduced/hosted but a Tldraw presenter delivered the content. When quoting, prefer "the presenter said..." rather than naming a specific person, unless the user has already established attribution. The transcript also contains substantial speech-to-text artifacts (garbled words, fragments, missing punctuation) — preserve them verbatim rather than correcting.

Participants: Not listed.

Section 1 — Intro & framing

Having a startup that makes an SDK. The problem is that you can build cool stuff with this SDK is because of the SDK, what it can do. Over the last couple of years, a lot of those experiments have involved AI. And, well, let me show you. Okay, so we're on Tldraw right now. Actually. We're not on tilter.com. So overall all those pitches, but it really is. I mean, just to show you, there really is a normal canvas on the, on the cans of this is, you know, a YouTube video that I can still play with, even though it's, it's rich in better content. It's the same way that notion kind of like, it's like a document editor, but it's also just normal itching those days out and stuff like that. Same thing for Tldraw.

Section 2 — November 2023: GPT-4 vision launches & "make real"

In 2023. Which just could be framing for the rest of the talk. It's all going to be demos from here on. These, these AI products that we've experimented with. November 2023, GPT-4 with vision just launches. This is the first time that a really good vision model is available via like a developer API. That API is in preview. You get like 30 requests a day or something like that. It's ridiculously expensive, but it's the first time that people can start playing around with products for this. So I don't know if you were on Twitter in November 2023. You probably saw this tweet about make real. And say that because, like, it was the first kind of, like. Using AI to produce code. Kind of thing to really. Break containment, like kind of escape from normal. Like, kind of Niche. And in the community type of tweets and go really broad. The, the closest thing to this that proceeded in was a carcass. You know, menu, like visualizer type of app. This first kind of coding stuff. And, and, yeah, the basic idea is that you have a canvas where you can make stuff, right? You draw diagrams and things like that. And you have an AI that can see, let's see if we can get the AI to make the stuff that we're drawing. Right.

Section 3 — Demo: text-prompted timer; drawing-prompted timer; annotation as input

So, you know, Well, the mountains have gotten a lot better since 2023, but the, the application still works. I'll show you. So normally you're prompting, and maybe you are just using text, then basically it's, it's this, right? I want, like, an orange device. Shows a timer 00 button start stop at time. And models are especially recent. It's much better than it was in 2023. Like, totally capable of, of working with that and adding to it and so on. So wonderful. I have my little tire, and I use nothing but a text to, to get. There. But. With make real, you can kind of get similar results. Just with the, the drawing. Right? Like, if I took this and just copied it as a screenshot and, like, just pasted the screenshot into a person or something, said, hey, can you make this for me? And, then, yeah, we would be able to do that as well.

One of the cool things we found about this. And again, you have to remember 2023, right? There was no bytecode wasn't working yet. That came later. There was no lovable. This whole idea of AI coding. Well, just hadn't arrived in the same way as, as it did in 2024. For a lot of people, this was like the first time you've ever. Created something technical at all. Because it kind of code and they could use their existing skills, which was like John rectangles. Something fun. We found about or something fun that we found out about this kind of way of using mechanics, though, is that, you know, if I wanted to change something about just a text to, to represent, I could have probably said it, like, no, no, make it blue. You know, or, you know, asking these kind of complex positional things. Like, he'd switch the position of these buttons and so on. But what is, what is cool about this is that even though this is a kind of a working website that has been created from. Created from, from an image, right, it's sitting on the canvas, it can kind of do these. Stuff like this. I can just kind of draw on top of it. And I could say, like. Really. And. Maybe make the. I don't know, do something like this. And then in the same way that I selected all this and put the button to make it turn into a prototype. I can select all this and put it in and it will send that new screenshot, which includes the website itself. To the model saying, hey, can you go? All right. Switching buttons didn't work. But the grid that you can, you can use this to kind of iterate on the, the website using annotation. This was such a cool, cool thing to discover that, that annotation could be an input to these language models.

Section 4 — Annotation-as-input generalized; customer examples

And in fact, if I close the tabs, but if you go to some of the customers that have used the SDK now, like, like Google and mixed more stitch or prep lead and Luma, whatever Runway, this idea of using annotations as an input part of the prompt is something that has gone, people have taken really, really far.

Section 5 — Demo: dot-matrix camera filter from a sketch

So let me see how far I can push this with, I don't know, good model of using Gemini 3 flash preview, and then we'll go a little bit harder. I mean, we switched to model. By the way, this is all bring your own token. Right. We haven't monetized this at all. So you have to. You have to trust me not to steal your API token. I promise. Let's say we have something like this. Right? This is much more complex for me to, to prompt this with, with language would be quite complex. Right. I want to use these camera feed. I want to have switches for dot matrix or colors. I want to control for how big the, the dot size is. I want to have a little panel at the bottom with, like, thumbnails of the previous things. And it's like, you can do it. You could do it with, you know, supplementing both language and maybe fitment design or something else. But children and Tldraw, the canvas is a really, really cool. Place to make that decision to have the kind of iteration loop and to have that back and forth. And like, like any good cooking show, I do have emergency, you know, finished products over here. So we'll see if it actually uses my camera here. Otherwise we'll need to, we'll need to change the product. It's kind of a little bit like it. It's interesting. It kind of used to say design that we just. Before. You can imagine it creates the markup and then it creates the. That's not bad. Right? I mean, all right. Assume it actually works, though. Coloring. Color works. The dot matrix that works. That size. You know. And can I capture. Can. Pretty. Cool, right? Now, this didn't have any, well, I guess we've had a couple words in it, but, you know, this is just an image as input. We got a thing that's output and an interior that could continue to. Mark it up and see if I can create later. Iterations and stuff. Like that.

Section 6 — Tldraw computer: branching & multi-step prompts

So maybe it was like one of the first. AI projects in the, we ever played with. Right. And this was like, oh, wow vision models. That's, that's what it's at. Right? And we did do, do other stuff. Like we created this whole thing called Tldraw computer. The channel isn't the right user interface One of the things that you can do outside of one b chaplain is is branching. Branching conversations, as well as, like, multistep prompts. Way that it's, like, repeatable It's very hard to do in chat. Trivial to do using kind of code or some workflow editors and not exist But but hard to be creative with, I would say. So the campus is a great place to do that because you can drink graphs. You can create I'm gonna change this to actually use use my own drive. We need a Real tennis? Is that like real ultimate Foot's to the last restroom. I So each one of these things is a little prompt. It has a kind of some some structure to it, but it's it's repeatable. It kinda generates its own kinda script. And then executes a script. Based on what its inputs are. In this case, the inputs are whatever is pointing to the So write a newspaper article summarizing the championship on these rules. Based on the rules for the game, text. We also have little text and this should be wrong that are drew. Which is leading us to to design a three d model of the logo. And we're still generating the newspaper article which will be used less time. Lefford towards the victory and inaugural Okay? Cool. Actually, this is much more complex and interesting than I thought. Do you guys have tennis? With different rules? That looks fun. If you wanted to to remix this all you have to do is kind of change the input. Right? And then run run Want? From? From this

Section 7 — Why canvas: collaboration as the killer feature

What do we really want from a canvas? This this going? We we Done an agreel We've done a computer. It's kinda, like, associated things You know? And part of that was just thinking about, like, how do we use the killer feature of Canvas is collaboration. Right? I wanna be able to to work with other people and something very complex. I wanna be able to just kinda have a 10, 15 people in the same document and for it to sort of have the affordances to to come in for all that. So we started on this really long journey of of getting an AI to see the canvas and to understand what's on the canvas and also be able to act. The canvas. So same way that I can create shapes and and manipulate the canvas contents I wanted the AI to be cat.

Section 8 — Agent harness demo (2025)

This is as far as I know The agent finish So this is 2025. So we tried to rip off cursors of our it was in 2025 exactly. And really is a full, like, kinda agent of the It's gonna look. It's gonna and there you go. I can even reject or accept this That's wonderful. Perfect. Right? I wanna I wanna let's make a little table here. Of course, I'm not gonna kinda tell This is a table because it can On So the stuff that it made, this is not an image. This is a Actually, these steps Just reviewing this work. Hopefully, it doesn't say Perfect. Yeah. Great. That's all we won. Right? But can't be pretty good. And some more screens. My work. Work rooms. It's a much more complex And talk about, like, kinda where we wanted to go next. This was cool. We had this harness Not terribly good at moving cats around, but it it is pretty good at for example, a tree. But it was tracking the sidebar. This agent. Tracking the sidebar.

Section 9 — Fairies: agents as canvas citizens

So from there, we we wanted to bring it onto the canvas the same way that you know, other users on the Canvas So that led us to to to fairies to to these guys. So each one of these is instance of that that harness that that lives of on the canvas. And while it started with just being a way of visualizing the the agents. Where are they? Which ones exist? Which one is which? Right? You can kind of configure them change their hat. Or because we needed two things, like, can change the length the legs. Hold them. They don't like that. Yeah. But the third, you can talk to them as well. Make them move as x. You can you can visualize the state of the the agent as well. So whether it's thinking, or reviewing or or working cool. Make a move. And o and you can ask them to to do things. And, again, it's pretty much the same part as as we looked at. But they're they can exist simultaneously. They even know about where each other This is worse than life coding, by the way. Let me let me let me try this. Maybe you can get them to work. As a team. So if I grab all three and I say, play a full game. Right? Leader. And will Plan. The the race will change colors to They're on the scene. Same team. By the way, this is also clarity. You could have multiple people kind of sharing the same experience. Coordinator comes up with a to do list and then delegates those tasks. To different affairs, different agents. So first move you can kinda read your other c check. Well. There. Wonderful. Second one Oh, this is gonna be bad. I'm just getting on on bailing in this. Alright. Taking an entire line of peer eating, turning into cleaning up your stuff, actually is really cool. Of my favorite parts about this, as I said, this is a labyrinth so you can kinda share a link. At ariesneutildraw.com. It's free You can spend money. Blah. Reviewer, I believe, is also looking to see whether someone's won or not the But this is a narrative. If I had 10 other people, working on this document with me, in this experience with me. They would all have three fairies each. And my fairies would choose to see their fairies and their mine. And while they the orchestration interest within the single user's Having 10 people on the same document, we done this. Having a collaborative experience on the same document, together with 30 various 30 agents And and for that just to work, for the UI and the the application they have afforded this is for not only human AI collaboration, The hemming I I really encourage you to try because it'll be kind of fun. Like all of these demos, should kinda point towards, like, a possibility that maybe makes more sense than more narrow version of this experience that is gonna be tied to this. To become a user or the product.

Section 10 — Desktop app, local MCP, and "code mode"

With newest things that we're we're doing with the same sort of I suppose, patient harness. And that is the desktop app. So we wanna have like, an MCP, like, experience where you're able to to use and any agent to to kinda work with the campus. Okay? Like, I wanna be able to say, hey. Here's my code. Can you just whip up a quick diagram of, like, how this thing is working? And then I wanna be able to, like, change the diagram say, okay. I've changed the way that this works in the diagram. Can you, like, bring that back to code? Update the code to match, you know, the That would the kinda trapped inside of an active application. So one of my favorite that communicate with the application using like, a local MCP server. Maybe paper When you think about And we couldn't recreate that same sort of experience then. You know, like from the Fairies app or from the agent kind of as a sidebar starter to which, by the way, if you wanna build with that, you can have all the code to do But then we we kinda realized that were looking at code mode. I don't know if you kind of tracked this, the idea that well, actually, agents are just kinda better rather than making these calls if you just give them code and let the next it and then kinda generate the tool calls from there. But Tldraw has, like, a runtime API. You can say, like So since it was local, we're like, let's just give the AI direct access to to the editor and let's see what it does with it. And well, the the answer is does. Incredibly crazy metaprogramming and top of of, like, So we started asking questions about can you make this interactive? Can we make this do stuff? Like, make the hype of this little circle affect the height of guy. Right? Can you can you make this? Make these buttons actually work? Kinda kinda back to make real, but I'll operate it with within the the canvas itself. Now we don't have primitives for you know, when this is split to do this, when the Like, you can you can you have enough of the, like, low level find a way to program And that's exactly what what Claude has done for his It's essentially like script injecting the application. Application in order to make these things work. But we've we've given it a doorway to to do that. Okay? Silly example. But like totally unexpected. For for us. You know, we we had a the first time we we saw this, we were like, hey. Grab all the photos on our on my desktop. Put them into your draw. Give me a little of Desktop, search bar that uses the native embeddings on Filter the list. And it just figured out a way to do it. If you I don't think you election will work. But the the code that it uses to do it is some of the most insane using Python to template JavaScript inside of a mash call. Like, the the craziest, weirdest code that I've ever written in my life. But it totally works. And and as a a way of doing, like, an know like, shoes. You know, I I could do this and then and say, The only thing is that it's it's it's slow. So not gonna I'm not gonna wait. But that's all kind of pointless. But we've seen customers, different teams, kinda use

Section 11 — Close / CTAs

If you do want to, you can go to tldraw.dev. To use the SDK. Learn more about that. Or if you just need it for you, free whiteboard? That is on a And that is is my show. I have time for some questions. So thank you.