Brian Douglas

The beginners guide to training AI on your own code

Every AI agent call generates training data. Most teams throw it away. Tapes is an open source telemetry proxy that intercepts LLM API calls and builds a content-addressable Merkle DAG of every conversation turn, with zero instrumentation required. The agent points its API base URL at the proxy and keeps running. Every session is cryptographically verified and forkable.

We publish these events to Kafka. Flink SQL runs continuous anomaly detection against the stream: stuck loop detection in tumbling windows, token spike detection against rolling averages, and behavioral drift monitoring. When Flink catches something, it publishes an alert to a separate topic. The next agent run reads those signals before it starts. That's the self-healing loop.

But the more interesting thing happens after. The Merkle DAG structure means every conversation turn is content-addressable and filterable. Successful sessions become supervised fine-tuning data. Pair them against failures and you get preference data for DPO. We built this pipeline by running parallel agents speedrunning Gameboy games. Hundreds of sessions, labeled by outcome, streamed through Kafka, filtered by Flink, and used to train QLoRA adapters on Qwen 7B running on a single consumer GPU.

The result: a closed loop where your agents generate the data that trains the next version of the model they run on. No external training data. No third-party model outputs. Your code, your agent traces, your model. Essentially embed the skills right into the model.

This session covers the full pipeline end to end: tapes proxy to Kafka, Flink SQL anomaly queries, the alerts feedback loop, and extracting clean training datasets from raw telemetry to fine-tune local models. Live demo included. All code is open source.