Context engineering AI agents has quietly become one of the harder problems in software teams' AI adoption, and a recent conversation on The AI Native Dev makes the case with an unusually sharp example. Simon Boudrias, who leads the Language Foundations group at Datadog, told host Guy Podjarny about the moment his team deleted an entire folder of AI context files that had been maintained for over a year, expecting performance to drop. Instead, their evals improved. It's a small story with a large implication: the steering documents teams write for their coding agents age faster than most people assume, and letting go of them can matter more than adding new ones.

Boudrias runs the part of Datadog's engineering organization that serves around 4,000 developers, and the AI dev group he built a year ago sits inside that mandate. The conversation traces the whole arc of how that group came to exist, starting with a scrappy internal pilot of Cursor at the beginning of 2025. The team aimed for maybe 100 to 200 developers to try it. Within a month, more than a thousand were using it daily. "This is pure pull," Podjarny observed, and Boudrias agreed: nobody had to sell the tool, they just had to open the door.

Why Datadog Built Evals Before It Built Rules

For informational purposes: an AI agent eval, in Boudrias's telling, is a repeatable test that replays a real scenario against an agent and scores whether it produced the correct outcome, most usefully anchored to ground truth a team already has, like past incidents.

Cursor's adoption curve outran Datadog's internal processes fast enough that, for a stretch, a product manager was manually adding developers' emails to an access list because the intended provisioning system hadn't caught up. That kind of organic, slightly chaotic pull is a pattern Podjarny has seen across the industry: teams that try to gate AI tool adoption tend to fight it, while teams that get out of the way tend to get flooded.

What's less common is what Datadog did next. Rather than layering on more rules, the team invested early in AI agent evaluation. Boudrias described the first concrete application as code review: instead of building a generically "smart" reviewer, his team replayed pull requests that Datadog knew had caused real production incidents, then measured whether an AI reviewer would have caught the problem before it shipped. That single design choice sidesteps one of the hardest problems in AI agent evals, which is defining what "correct" even means. When the ground truth is "this caused an outage," the eval writes itself.

This is a pattern worth naming as a mental model: rather than trying to specify correctness in the abstract, anchor the eval in something that already happened. It's the same instinct, later in the conversation, that shapes how Datadog wants to write future evals at all: extracting real usage patterns from agent trajectory data rather than asking developers to author test cases by hand, since, as Boudrias put it, most developers don't want to write evals any more than they want to write documentation or tests.

The Context Deletion Experiment

The episode's most quotable moment centers on context engineering directly. Datadog's frontend team noticed that a steering document, written roughly a year earlier when Sonnet 3.5 was the frontier model, hadn't been revisited since. The hunch was simple: is this file still earning its keep in a world of Opus 4.8 and GPT-5.5? So the team tried something blunt. They deleted it entirely and reran their evals.

Performance went up. Not stayed flat, improved. Boudrias's explanation is worth sitting with: a lot of what gets written into a steering document early on is basic scaffolding a weaker model genuinely needed, like a paragraph explaining how to use Yarn. A frontier model already has that in its training data. The document isn't just neutral clutter at that point; it appears to actively compete for the model's attention with newer, more relevant context, a dynamic Podjarny described as context rot, the same decay familiar from software that ages from useful to useless to actively harmful.

What made the decision defensible, rather than reckless, was that Datadog had eval data to back it. Boudrias was candid that removing something and not replacing it with anything is a hard call to make with confidence, especially at a scale where a mistake touches thousands of engineers. He traced the reluctance to a kind of institutional loss aversion: once someone writes something into an AGENTS.md file, proposing to cut it invites the question of whether it's secretly useful to somebody, somewhere. Data is what breaks the stalemate. Without it, as Boudrias put it, deleting anything "would be pretty controversial."

Where Open-Weight Models and Hiring Fit In

The conversation also touches on where open-weight models like GLM 5.2 sit relative to frontier options for tasks like code review: based on Datadog's own evals, Boudrias suggested an open-weight model would need to perform roughly 50% better than today's baseline to be a fully viable substitute, with the caveat that creative, less-defined tasks are harder to benchmark than narrow, repeatable ones like formatting fixes.

Perhaps the most immediately actionable segment of the episode is Boudrias's account of rebuilding technical interviews. He was candid that he'd always found LeetCode-style interviews low signal, and that AI adoption gave his team a legitimate reason to redesign them. The new format hands a candidate a large, unfamiliar real codebase and asks them to reason through an actual engineering problem with an agent, testing judgment and tradeoff-making rather than whether someone can produce working code in 40 minutes.

Taken together, the episode makes a case that's easy to miss amid the noise around agent frameworks: the teams getting real value from AI coding tools aren't necessarily the ones with the most sophisticated harness. They're the ones willing to measure what's actually working, and brave enough to delete what isn't.