CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flame-graph-analyzer

Reads CPU flame-graph output from py-spy (Python), async-profiler (JVM), Go pprof, or Node.js `perf_hooks` / clinic.js: identifies the hot path (top sample-time frames), classifies the bottleneck (CPU-bound vs lock contention vs allocator pressure), and proposes the next investigation step. Use when a perf regression is bisected to a commit but the hot path inside it is unclear; for tail-latency percentiles use latency-percentile-analyzer, for GC pauses specifically use jvm-gc-tuning, and for a slow SQL hot path use db-query-plan-analyzer.

80

Quality

100%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

SKILL.md

name:
flame-graph-analyzer
description:
Reads CPU flame-graph output from py-spy (Python), async-profiler (JVM), Go pprof, or Node.js `perf_hooks` / clinic.js: identifies the hot path (top sample-time frames), classifies the bottleneck (CPU-bound vs lock contention vs allocator pressure), and proposes the next investigation step. Use when a perf regression is bisected to a commit but the hot path inside it is unclear; for tail-latency percentiles use latency-percentile-analyzer, for GC pauses specifically use jvm-gc-tuning, and for a slow SQL hot path use db-query-plan-analyzer.

flame-graph-analyzer

Overview

Canonical flame-graph reference: brendan-gregg-flame. The widest leaf (bottom of the stack) is the hot path.

Each runtime produces flame graphs from its own profiler - py-spy (Python), async-profiler (JVM), Go's built-in pprof, clinic.js flame (Node.js), and perf / eBPF (native C/C++/Rust). This skill is language-agnostic: it consumes the profiler output (SVG, JSON, or folded-stacks .txt) and surfaces a hypothesis the engineer can act on. Per-runtime capture commands live in references/capturing-and-interpreting-flame-graphs.md.

When to use

  • A perf regression has been bisected but the introducing commit touches multiple functions; the team needs to know which function is the actual hot path.
  • A load test under k6-load-testing or a sibling shows latency growth, but the API code hasn't visibly changed - the flame graph reveals the runtime cause.
  • A production incident showed CPU saturation; the team has captured a profile and needs to triage before reducing fleet size.
  • An EXPLAIN ANALYZE trace suggests CPU is the bottleneck rather than I/O - the flame graph confirms.

How to use

  1. Capture a profile under realistic load with the runtime's profiler and export folded stacks - per-runtime capture commands in references/capturing-and-interpreting-flame-graphs.md.
  2. Sort the folded stacks by sample count and read the top 5 leaves (the actual working code, not framework wrappers).
  3. Classify each hot leaf's sample-time signature (CPU-bound vs allocator pressure vs lock contention vs I/O-misclassified) using the table below.
  4. Propose the next step for the dominant leaf; the category-to-fix remediation catalog is in references/capturing-and-interpreting-flame-graphs.md.
  5. Re-profile after the change to confirm the delta, then hand off to perf-budget-gate to confirm the regression delta closes.

Reading the hot path

Read the folded stacks (or extract them from the SVG / JSON), sort by sample count, and identify the top leaves - the bottom-of-stack frames that are the actual working code, not framework wrappers:

# top 5 leaves by sample count
sort -k2 -n -r folded.txt | head -5

Each hot leaf's sample-time signature points to one category:

CategorySignature in the flame graph
CPU-bound (hot algo)A wide leaf in user code (a regex, a JSON serializer, a hash function).
Allocator pressureWide GC frames (gc::scavenge, Java GC, gc.collect).
Lock contentionWide synchronization frames (pthread_mutex_lock, Object.wait, parking).
I/O wait misclassifiedOn-CPU profilers don't show I/O blocks; switch to wall-clock profiling.
Reflection / dynamic dispatchWide reflection.invoke, method_missing, getattr chains.
Logging overheadWide log.format, Logger.debug, serialization for log lines.

Worked example

A Node.js endpoint regressed after a bisected commit. Capture under load, then read the hot path end to end.

Capture (Node.js clinic.js flame; other runtimes in the reference):

npx clinic flame -- node app.js
# drive load against the process, then stop it; clinic writes flame.html + folded stacks

Sort the folded stacks and read the top leaves:

sort -k2 -n -r folded.txt | head -5
main;handleRequest;serializeJson;JSON.stringify   4521
main;handleRequest;dbQuery;Array.from              2103
main;handleRequest;dbQuery;parseRows               1832
main;authCheck;jwt.verify;crypto.createHash        904
main;handleRequest;serializeJson;Buffer.from       312

Emit the analysis:

## Flame graph analysis - `flame.html`

**Runtime:** node
**Profile duration:** 30s under load

| Rank | Sample share | Stack (leaf)                          | Category |
|-----:|-------------:|---------------------------------------|----------|
| 1    | 38%          | `JSON.stringify` (in `serializeJson`) | CPU-bound hot algo |
| 2    | 17%          | `Array.from` (in `dbQuery`)           | Allocator pressure |
| 3    | 15%          | `parseRows` (in `dbQuery`)            | CPU-bound hot algo |
| 4    | 8%           | `jwt.verify` (in `authCheck`)         | CPU-bound hot algo (crypto) |
| 5    | 3%           | `Buffer.from` (in `serializeJson`)    | Allocator pressure |

### Hypothesis
The serialization path is load-bearing: rank 1 (`JSON.stringify`, 38%) plus
rank 5 (`Buffer.from`, 3%) account for ~41% of sampled time combined.

### Recommended next step
1. Switch to a streaming JSON serializer (`fast-json-stringify` in Node,
   `orjson` in Python, Jackson's `JsonGenerator` in the JVM) to eliminate the
   intermediate string allocation.
2. Re-profile; expect rank 1 to drop below 10%.
3. Hand off to `perf-budget-gate` to confirm the regression delta closes.

More interpreted cases (GC pressure, lock contention, reflection overhead) and the category-to-fix remediation catalog are in references/capturing-and-interpreting-flame-graphs.md.

Anti-patterns

Anti-patternWhy it failsFix
Reading the SVG visually only, no quantitative dataEasy to mis-judge widths; biases toward dramatic-looking deep stacks.Always work from folded stacks; sort by sample count.
Profiling under load that is too lowOne request per second can't expose contention or allocator pressure.Profile under realistic load - pair with k6-load-testing.
Optimizing rank 5 first because rank 1 looks "structural"Premature optimization; misses the dominant cost.Always start with rank 1; only descend if rank 1 is genuinely framework-bound (e.g. event_loop).
On-CPU profiler for an I/O-bound workloadI/O wait doesn't appear; the flame graph shows what's running, not what's waiting.Use wall-clock / off-CPU profiling for I/O-bound workloads.
Single 30-second capture under highly variable loadThe sample is unrepresentative.Capture multiple samples across the load-test duration; merge.

Limitations

  • Symbolication. Without debug symbols, the flame graph shows hex addresses. Profile with debug info enabled in CI builds intended for analysis.
  • Inlining. Aggressive inlining (especially in the JVM hot-path optimizer) can hide functions; the flame graph shows the post-inlining shape, which may not match the source.
  • Sampling vs. tracing. Sample-based flame graphs give relative weights; for absolute timings of specific operations, use a tracer instead.

References

SKILL.md

tile.json