Agent Benchmarks Need To Measure The Whole Workflow

At AI Native DevCon London, I talked about benchmarking in the agent era because the workloads we care about have changed faster than the benchmarks we use to measure them.

At NVIDIA, we spend a lot of time thinking about inference performance. That means performance in the real world, not only performance on clean single-turn prompts. For a while, many benchmark conversations assumed a chatbot shape: one user input, one model response, a small context window, relatively short outputs, and no tools.

That is not what agentic coding workloads look like anymore.

In "Benchmarking the Agent Era: Measuring Performance Beyond the LLM," I wanted to separate model performance from system performance. The model still matters, but once agents start calling tools, reading files, building context, and looping through many turns, the benchmark has to measure the whole trajectory. You can also watch the full recording.

Chatbot Benchmarks Do Not Match Agent Workloads

The old shape was simple. A user asked a question. The model returned an answer. Context length might be one to four thousand tokens. Output length was usually small. There were no tool calls.

In agentic workloads, the shape is different. A single user request can lead to dozens of turns. The model may be called again and again. Sequence lengths can grow from tens of thousands to hundreds of thousands of tokens. Output length can increase. Tools become part of the normal path.

That difference matters because the benchmark that scored well yesterday may not tell you much about the workload you are actually running today.

I used a coding-agent example in the talk: a user asks an agent to fix a flaky test in a codebase. The first model call does not solve the whole task. It decides which file to inspect. A tool reads the file. The result goes back to the model. The model decides where to look next. Another tool runs. That loop can continue for twenty, thirty, or forty turns depending on the complexity of the task.

That is not one inference call. It is a trajectory.

Agents Mix GPU Work And CPU Work

One detail that gets missed in simple benchmarks is where the work actually runs.

In the agentic trajectory, the model calls run on GPUs. The tool calls often run on CPUs. If a tool reads files, searches a repository, runs shell commands, or gathers information from another system, the GPU may be idle while the CPU-side work happens.

That changes capacity planning. If you assume every user keeps the GPU busy all the time, you may underestimate how many concurrent users the system can support. If tool calls create large CPU-side delays, you may have gaps where GPUs are available for other work.

Ignoring tools therefore gives you the wrong concurrency picture. The benchmark may say your hardware can support one number of users, while the real agentic workload could support more or less depending on the shape of the tool calls.

The point is not that tools are free. The point is that they change the system. A real benchmark has to include them.

Prefix Caching Becomes Bread And Butter

The first optimization I discussed was prefix caching.

In an agentic workflow, each new turn builds on the previous history. By the time the agent is deep into the task, much of the context has already been processed before. If the system can cache that work, it does not need to recompute the whole prefix every time.

That is critical for time to first token. If caching is working, the model only has to process the new part of the context. If caching is not working, each turn can become much more expensive because the system keeps reprocessing information it has already seen.

This is why agent benchmarks need to test caching behavior directly. A benchmark with single-turn fixed input and output shapes does not expose whether the serving system can handle the repeated, growing context that agents create.

For agentic coding workloads, prefix caching is not a small optimization. It is foundational.

Routing Has To Find The Cache

Caching is not enough if the next turn lands on the wrong replica.

In production, the same model may be served across multiple replicas. If turn one lands on replica A, that replica has the relevant cache. If turn two is routed round-robin to replica B, the cache may not be there. The system loses the benefit.

That is why KV cache-aware routing matters. The serving layer needs to send later turns to the place where the useful cache exists, or use another strategy that preserves the cache benefit.

There are different ways to do this. A system can use session IDs to keep related requests together. It can also use more explicitly cache-aware routing. The important thing is that the benchmark should reflect this issue. A benchmark that turns this off or ignores multi-turn trajectories misses one of the major performance properties of agentic workloads.

Speculative Decoding Changes Output Speed

The third optimization I discussed was speculative decoding.

The high-level idea is that a smaller draft model proposes several tokens ahead. The larger target model then verifies those tokens. If the draft model is sufficiently aligned with what the target model would have produced, the system can generate multiple tokens in one pass instead of producing them one by one.

This can create a meaningful speedup on the decode side, where the model is producing output. Newer models may include multi-token prediction components that make this easier to turn on.

But there is a measurement trap. Speculative decoding changes the distribution of token speeds. If you look at the wrong percentile, you may accidentally penalize a useful optimization. In the talk, I showed how one percentile view could make the optimized run look worse even though the broader distribution was better.

The lesson is that agent benchmarking has to look at distributions, not only averages.

Means And Medians Are Not Enough

In agentic systems, user experience often breaks in the tails.

If most requests are fast but a meaningful tail is slow, users will still feel the system as unreliable. For time to first token, left is good and right is bad. For token speed, the interpretation changes because slow generation is the problem. Either way, the mean can hide what users actually experience.

That is why I focused on percentile-based thinking. P95, P25, and similar distribution metrics can be more useful than mean or median, but only if you choose the percentile that matches the user experience and the optimization you are evaluating.

The benchmark should help you answer questions such as: how many users can this hardware sustain while meeting a time-to-first-token SLO? How fast are tokens generated for the important part of the distribution? What happens when caching is warm? What happens before it is warm?

Those questions are closer to production reality than "how fast is one model response?"

Measure Steady State, Not Only Startup

Agentic workloads also take time to stabilize.

At the beginning of a run, caches may be cold. Routing may not yet have enough useful history. The system may still be in a transient state. If you measure performance too early, you might conclude that a deployment fails its SLO even though it would pass once the workload stabilizes.

This is especially important for caching-heavy agent workloads. The performance curve can improve as the system warms up. So the measurement needs to show whether the metrics have flattened out over time.

If the benchmark only captures a transient window, it can lead teams to overprovision hardware or reject a deployment that would have worked.

A Better Benchmark Starts With Real Trajectories

I pointed to the Artificial Analysis agent benchmark as an example of the field moving in the right direction.

The important shift is that it tries to include the properties I discussed: multi-turn trajectories, cache behavior, speculative decoding, SLO-based measurement, and real-world workload shape. The question becomes: for a given hardware setup and a given SLO, how many users can the system sustain?

That is a much more useful question for production planning than a single model score.

It also opens the door to per-watt and per-dollar views, which matter when teams are making infrastructure decisions. Performance is not only speed. It is speed under cost, power, concurrency, and quality constraints.

The Missing Work Is Still Large

This field is still early.

In the talk, I focused mainly on single-agent coding trajectories. That is already more realistic than chatbot benchmarks, but it is not the end state. Multi-agent workflows create additional complexity. Long-running sessions can continue for days, with caches and memory growing over time. Heterogeneous workloads may behave very differently from coding agents. CPU coupling needs deeper treatment. Quality under load has to be measured alongside speed.

The final metric may not be time to first token or token speed at all. For many teams, task completion time will matter more. For others, the right question will be how quickly the agent completes the task while still producing a correct result.

That is why I think benchmarking the agent era means measuring beyond the LLM. The model call is only one part of the system. The real workload includes context growth, tool calls, caching, routing, decode behavior, CPU and GPU interaction, SLOs, tail latency, and quality.

Benchmarks for the chatbot era have caught up to chatbot workloads. Benchmarks for the agent era are still being built. The next step is to make them look like the systems we actually run.

The full version of this argument was presented at AI Native DevCon London. To go deeper, watch the full recording.