Outline — Welcome to AI Native DevCon (Tldraw AI Canvas Experiments)

Speaker

Listed: Simon Maple — Head of Developer Relations at Tessl, AI Native Dev co-host. Previously Field CTO and VP Developer Relations at Snyk, ZeroTurnaround, and IBM. Java Champion (2014), JavaOne Rockstar (2014, 2017), Duke's Choice award winner, Virtual JUG founder, London Java Community co-leader.

Caveat [inferred]: The transcript content reads as a Tldraw team member demoing the Tldraw SDK ("we're on Tldraw right now", "if you go to tldraw.dev"). The talk may be hosted/introduced by Simon Maple at AI Native DevCon but delivered by a Tldraw presenter. Treat speaker attribution with care.

Abstract [inferred]

A demo-driven tour of Tldraw's AI experiments since November 2023: starting with "make real" (drawing → working website via GPT-4 vision), exploring annotation as prompt input, branching prompt workflows on Tldraw computer, multi-agent "fairies" collaborating on a shared canvas, and a desktop app where agents use "code mode" against Tldraw's runtime API.

Thesis

The canvas is an under-appreciated surface for AI interaction: it lets users supply drawings and annotations as input alongside text, supports branching/repeatable workflows that are awkward in chat, naturally accommodates multi-agent + multi-human collaboration, and — when paired with a runtime API ("code mode") — lets agents do metaprogramming that would be impossible through narrowly-scoped tool calls.

Section TOC

1. Intro & framing (lines ~1–10) — "Having a startup that makes an SDK"; Tldraw is "a normal canvas" that supports rich content like a YouTube video; framing for a demo-heavy talk.
2. November 2023: GPT-4 vision launches & "make real" (lines ~10–25) — First good vision model via developer API; "make real" tweet "really broke containment"; canvas + AI to render drawings as working artifacts.
3. Demo: text-prompted timer; drawing-prompted timer; annotation as input (lines ~25–45) — Building a timer via text, then via screenshot of a drawing, then iterating by drawing annotations on top of the rendered website.
4. Annotation-as-input generalized; customer examples (lines ~45–55) — Google, Stitch, Replit, Luma, Runway have "taken really, really far" the idea of annotation as part of the prompt.
5. Demo: dot-matrix camera filter app from a sketch (lines ~55–75) — Pushing complexity: camera feed, dot-matrix/colour switches, dot-size control, thumbnail panel — generated from one sketch via Gemini 3 Flash preview.
6. Tldraw computer: branching & multi-step prompts on canvas (lines ~75–100) — Chat is bad for branching and repeatable multi-step prompts; canvas is great for it. Demo: a "real ultimate tennis" workflow generating rules, a newspaper article, and a 3D logo.
7. Why canvas: collaboration as the killer feature (lines ~100–110) — 10–15 people in the same document; getting AI to see and act on the canvas.
8. Agent harness demo (2025) (lines ~110–125) — "Ripped off cursor"; full agent that can be accepted/rejected; can create tables, screens, track work in a sidebar.
9. Fairies: agents as canvas citizens (lines ~125–150) — Each fairy is an instance of the harness living on the canvas; configurable (hats, leg length); states (thinking/reviewing/working) are visualized; multi-fairy coordination (Coordinator delegates to-do items); compatible with multi-user collaboration (10 people × 3 fairies = 30 agents on one doc). Try at aries.tldraw.com.
10. Desktop app + local MCP + "code mode" (lines ~150–180) — Goal: any agent works with the canvas via local MCP; "code mode" idea — give agents direct access to Tldraw's runtime API rather than narrow tool calls. Results: "incredibly crazy metaprogramming", interactive scripts, a photo-search demo using native embeddings, "the craziest, weirdest code that I've ever written in my life. But it totally works." Caveat: it's slow.
11. Close / CTAs (lines ~180–end) — tldraw.dev for the SDK; free whiteboard available; Q&A.

Terminology glossary (speaker's own framing)

• Make real — The November 2023 demo/app where drawings on the Tldraw canvas are sent as a screenshot to a vision model (GPT-4 with vision) which produces a working website. Speaker: "the basic idea is that you have a canvas where you can make stuff, right? You draw diagrams and things like that. And you have an AI that can see, let's see if we can get the AI to make the stuff that we're drawing."
• Annotation as input — Drawing on top of an already-rendered artifact on the canvas and re-sending it (image + drawings) to the model as the next prompt iteration. Speaker: "annotation could be an input to these language models... this idea of using annotations as an input part of the prompt is something that has gone, people have taken really, really far."
• Tldraw computer — A canvas-based environment for branching conversations and repeatable multi-step prompts. Speaker: "branching conversations, as well as, like, multistep prompts. Way that it's, like, repeatable. It's very hard to do in chat."
• Agent harness — Tldraw's agent that can see and act on the canvas (create shapes, manipulate contents, tracked in a sidebar). Initially modeled on Cursor's UX ("we tried to rip off cursor").
• Fairies — Visualized agents living on the canvas as configurable characters; each is an instance of the agent harness. State (thinking/reviewing/working) is visible; they can be talked to, moved, and coordinated. Available at aries.tldraw.com.
• Code mode — Giving the agent direct access to the editor's runtime API rather than predefined tool calls. Speaker: "agents are just kinda better rather than making these calls if you just give them code and let the [agent write] it and then kinda generate the tool calls from there."

Named frameworks / concepts introduced

• Drawing → working artifact via vision model (make real, 2023)
• Annotation-as-prompt-input iteration loop
• Branching + repeatable multi-step prompt workflows on a canvas (Tldraw computer)
• Multi-agent collaboration on a shared canvas, integrated with multi-user collaboration
• Code mode — runtime-API access > narrow tool calls for agents

Open questions / not covered

• Detailed pricing, business model, or monetization of the experimental demos ("we haven't monetized this at all"; "bring your own token").
• Specific model performance benchmarks or evaluation methodology.
• How "code mode" handles security/sandboxing concerns when agents have direct runtime API access.
• How multi-agent orchestration is implemented under the hood (only the "Coordinator delegates to fairies" UX is described).
• Any comparison with bolt.new, Lovable, Cursor beyond brief mentions.
• Roadmap dates or release timing for the desktop app and MCP integration.
• Q&A content — the talk ends with "I have time for some questions" but no Q&A is in the transcript.