Transcript -- Agents on the canvas with tldraw

Speaker-label warning. This transcript has no per-speaker labels. The public schedule names Steve Ruiz as the speaker, and the spoken content is a tldraw product demo. The transcript also contains substantial speech-to-text artifacts (garbled words, fragments, missing punctuation); preserve them verbatim in direct quotations.

Participants: Not listed.

Section 1 — Intro & framing

Having a startup that makes an SDK. The problem is that you can build cool stuff with this SDK is because of the SDK, what it can do. Over the last couple of years, a lot of those experiments have involved AI. And, well, let me show you. Okay, so we're on Tldraw right now. Actually. We're not on tilter.com. So overall all those pitches, but it really is. I mean, just to show you, there really is a normal canvas on the, on the cans of this is, you know, a YouTube video that I can still play with, even though it's, it's rich in better content. It's the same way that notion kind of like, it's like a document editor, but it's also just normal itching those days out and stuff like that. Same thing for Tldraw.

Section 2 — November 2023: GPT-4 vision launches & "make real"

In 2023. Which just could be framing for the rest of the talk. It's all going to be demos from here on. These, these AI products that we've experimented with. November 2023, GPT-4 with vision just launches. This is the first time that a really good vision model is available via like a developer API. That API is in preview. You get like 30 requests a day or something like that. It's ridiculously expensive, but it's the first time that people can start playing around with products for this. So I don't know if you were on Twitter in November 2023. You probably saw this tweet about make real. And say that because, like, it was the first kind of, like. Using AI to produce code. Kind of thing to really. Break containment, like kind of escape from normal. Like, kind of Niche. And in the community type of tweets and go really broad. The, the closest thing to this that proceeded in was a carcass. You know, menu, like visualizer type of app. This first kind of coding stuff. And, and, yeah, the basic idea is that you have a canvas where you can make stuff, right? You draw diagrams and things like that. And you have an AI that can see, let's see if we can get the AI to make the stuff that we're drawing. Right.

Section 3 — Demo: text-prompted timer; drawing-prompted timer; annotation as input

So, you know, Well, the mountains have gotten a lot better since 2023, but the, the application still works. I'll show you. So normally you're prompting, and maybe you are just using text, then basically it's, it's this, right? I want, like, an orange device. Shows a timer 00 button start stop at time. And models are especially recent. It's much better than it was in 2023. Like, totally capable of, of working with that and adding to it and so on. So wonderful. I have my little tire, and I use nothing but a text to, to get. There. But. With make real, you can kind of get similar results. Just with the, the drawing. Right? Like, if I took this and just copied it as a screenshot and, like, just pasted the screenshot into a person or something, said, hey, can you make this for me? And, then, yeah, we would be able to do that as well.

One of the cool things we found about this. And again, you have to remember 2023, right? There was no bytecode wasn't working yet. That came later. There was no lovable. This whole idea of AI coding. Well, just hadn't arrived in the same way as, as it did in 2024. For a lot of people, this was like the first time you've ever. Created something technical at all. Because it kind of code and they could use their existing skills, which was like John rectangles. Something fun. We found about or something fun that we found out about this kind of way of using mechanics, though, is that, you know, if I wanted to change something about just a text to, to represent, I could have probably said it, like, no, no, make it blue. You know, or, you know, asking these kind of complex positional things. Like, he'd switch the position of these buttons and so on. But what is, what is cool about this is that even though this is a kind of a working website that has been created from. Created from, from an image, right, it's sitting on the canvas, it can kind of do these. Stuff like this. I can just kind of draw on top of it. And I could say, like. Really. And. Maybe make the. I don't know, do something like this. And then in the same way that I selected all this and put the button to make it turn into a prototype. I can select all this and put it in and it will send that new screenshot, which includes the website itself. To the model saying, hey, can you go? All right. Switching buttons didn't work. But the grid that you can, you can use this to kind of iterate on the, the website using annotation. This was such a cool, cool thing to discover that, that annotation could be an input to these language models.

Section 4 — Annotation-as-input generalized; customer examples

And in fact, if I close the tabs, but if you go to some of the customers that have used the SDK now, like, like Google and mixed more stitch or prep lead and Luma, whatever Runway, this idea of using annotations as an input part of the prompt is something that has gone, people have taken really, really far.

Section 5 — Demo: dot-matrix camera filter from a sketch

Operational credential and local-device details are redacted. Safe summary: the presenter demonstrates that sketches and annotations can express UI intent that would be hard to specify in text alone, making the canvas a strong iteration surface for AI-generated prototypes.

Section 6 — Tldraw computer: branching & multi-step prompts

So maybe it was like one of the first AI projects we ever played with. The presenter then moves to Tldraw computer: a canvas-based way to represent branching conversations and repeatable multi-step prompt workflows. Operational details are omitted. Safe summary: the canvas makes it easier to see, branch, remix, and rerun structured creative workflows than a linear chat interface.

Section 7 — Why canvas: collaboration as the killer feature

What do we really want from a canvas? This this going? We we Done an agreel We've done a computer. It's kinda, like, associated things You know? And part of that was just thinking about, like, how do we use the killer feature of Canvas is collaboration. Right? I wanna be able to to work with other people and something very complex. I wanna be able to just kinda have a 10, 15 people in the same document and for it to sort of have the affordances to to come in for all that. So we started on this really long journey of of getting an AI to see the canvas and to understand what's on the canvas and also be able to act. The canvas. So same way that I can create shapes and and manipulate the canvas contents I wanted the AI to be cat.

Section 8 — Agent harness demo (2025)

This is as far as I know The agent finish So this is 2025. So we tried to rip off cursors of our it was in 2025 exactly. And really is a full, like, kinda agent of the It's gonna look. It's gonna and there you go. I can even reject or accept this That's wonderful. Perfect. Right? I wanna I wanna let's make a little table here. Of course, I'm not gonna kinda tell This is a table because it can On So the stuff that it made, this is not an image. This is a Actually, these steps Just reviewing this work. Hopefully, it doesn't say Perfect. Yeah. Great. That's all we won. Right? But can't be pretty good. And some more screens. My work. Work rooms. It's a much more complex And talk about, like, kinda where we wanted to go next. This was cool. We had this harness Not terribly good at moving cats around, but it it is pretty good at for example, a tree. But it was tracking the sidebar. This agent. Tracking the sidebar.

Section 9 — Fairies: agents as canvas citizens

So from there, we we wanted to bring it onto the canvas the same way that you know, other users on the Canvas So that led us to to to fairies to to these guys. So each one of these is instance of that that harness that that lives of on the canvas. And while it started with just being a way of visualizing the the agents. Where are they? Which ones exist? Which one is which? Right? You can kind of configure them change their hat. Or because we needed two things, like, can change the length the legs. Hold them. They don't like that. Yeah. But the third, you can talk to them as well. Make them move as x. You can you can visualize the state of the the agent as well. So whether it's thinking, or reviewing or or working cool. Make a move. And o and you can ask them to to do things. And, again, it's pretty much the same part as as we looked at. But they're they can exist simultaneously. They even know about where each other This is worse than life coding, by the way. Let me let me let me try this. Maybe you can get them to work. As a team. So if I grab all three and I say, play a full game. Right? Leader. And will Plan. The the race will change colors to They're on the scene. Same team. By the way, this is also clarity. You could have multiple people kind of sharing the same experience. Coordinator comes up with a to do list and then delegates those tasks. To different affairs, different agents. So first move you can kinda read your other c check. Well. There. Wonderful. Second one Oh, this is gonna be bad. I'm just getting on on bailing in this. Alright. Taking an entire line of peer eating, turning into cleaning up your stuff, actually is really cool. Of my favorite parts about this, as I said, this is a labyrinth so you can kinda share a link. At ariesneutildraw.com. It's free You can spend money. Blah. Reviewer, I believe, is also looking to see whether someone's won or not the But this is a narrative. If I had 10 other people, working on this document with me, in this experience with me. They would all have three fairies each. And my fairies would choose to see their fairies and their mine. And while they the orchestration interest within the single user's Having 10 people on the same document, we done this. Having a collaborative experience on the same document, together with 30 various 30 agents And and for that just to work, for the UI and the the application they have afforded this is for not only human AI collaboration, The hemming I I really encourage you to try because it'll be kind of fun. Like all of these demos, should kinda point towards, like, a possibility that maybe makes more sense than more narrow version of this experience that is gonna be tied to this. To become a user or the product.

Section 10 — Desktop/canvas agent integration

Operational details from the desktop integration demo are redacted. Safe summary: the presenter explores how agents might collaborate with a canvas application, but the defensible design lesson is to expose explicit authorized APIs, keep local-resource access narrow and visible, and make generated actions reviewable and reversible.

Section 11 — Close / CTAs

If you do want to, you can go to tldraw.dev. To use the SDK. Learn more about that. Or if you just need it for you, free whiteboard? That is on a And that is is my show. I have time for some questions. So thank you.