load-testing-overview

Load testing drives concurrent, sustained traffic at a deployed system and measures what happens to response time, throughput, and errors while that traffic runs. It assumes correctness and asks whether the system stays fast and correct under load, and where it stops being either. It needs a deployed environment, realistic data volumes, and a real network path; run one against a laptop dev server and you measured the laptop.

Pick a tool from what is already in the repo

Five tools cover almost every case. Read the table top to bottom and stop at the first row that is true of your project. The rows are ordered so that hard constraints (protocol, who authors the tests) beat preferences (language).

Two follow-on constraints that change the answer:

• Tests must be reviewable in a pull request. k6, Gatling, Locust, and Artillery all store tests as ordinary source files. JMeter plans are .jmx files produced by the GUI and passed to the runner with jmeter -n -t my_test.jmx -l log.jtl (JMeter get started). Teams commonly find those generated plan files hard to diff in review; that is a practitioner observation, not a documented limitation. If code review of the test itself matters to you, that alone rules JMeter out.
• CI must fail the build with no extra plumbing. k6 and Artillery both exit non-zero on a failed budget out of the box (see the threshold section below). With JMeter you get a .jtl results file and have to assert on it yourself.

If you truly have no constraint, choose k6. It has the shortest path from zero to a failing build, which is the only path that matters at the start.

When the table lands on Gatling or Locust, the full authoring deep dive lives in this skill: references/gatling.md (Simulation class DSL, injectOpen vs injectClosed, setUp().assertions() as the CI gate, Maven/Gradle/sbt runs) and references/locust.md (HttpUser + @task locustfile structure, headless and distributed runs, CSV-based CI gating). For k6 and JMeter, use the dedicated k6-load-testing and jmeter-load-testing skills.

The six load profiles

Newcomers say "load test" for all six of these and then argue past each other. Each name answers a different question, and the profiles are defined by k6 as follows (k6 test types):

Two of these are routinely confused. Stress holds an above-average load steady and watches how the system copes; breakpoint keeps increasing load until the system fails, so its output is a number (the ceiling), not a verdict. Soak uses ordinary load and long duration on purpose: it exists to catch memory leaks, connection-pool exhaustion, and log-disk growth, which a 10-minute run cannot see.

Run smoke first, always. A broken script under 300 VUs produces a very convincing graph of nothing.

Open vs closed workload models

This is the single most misunderstood idea in the field, and it decides whether your numbers mean anything.

• Closed model: you hold the number of concurrent users constant. New work starts only when old work finishes. In k6's words, "In the closed model, VU iterations start only when the last iteration finishes," so "the target system's response time can influence the throughput of the test" (k6 open vs closed).
• Open model: you hold the arrival rate constant. "In the open model, on the other hand, VUs arrive independently of iteration completion," so "the response times of the target system no longer influence the load on the target system" (k6 open vs closed).

Why it matters: under a closed model, when the system slows down, your test automatically applies less load, which hides the problem exactly when it starts. k6 names this: "In some testing literature, this problem is known as coordinated omission" (k6 open vs closed).

Real user traffic on a public web service is open (people keep arriving whether or not you are coping); Gatling frames it the same way, noting open systems "have no control over the number of concurrent users" while closed systems cap that number (Gatling workload models). Closed is the right model for a fixed-size worker pool, a call-centre queue, or a system behind a hard concurrency limit.

For how each tool names its open and closed executors, and the Gatling "users means open, not closed" trap spelled out, see references/tool-executors.md.

Run your first test with k6

Install (k6 install):

brew install k6                     # macOS
choco install k6                    # Windows, Chocolatey
winget install k6 --source winget   # Windows, winget
docker pull grafana/k6              # any platform

Write script.js:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 10,
  duration: '30s',
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95th percentile under 500ms
    http_req_failed:   ['rate<0.01'],  // under 1% failed requests
  },
};

export default function () {
  const res = http.get('https://your-service.example.com/health');
  check(res, { 'status 200': (r) => r.status === 200 });
  sleep(1);
}

Run it (k6 running):

k6 run script.js
k6 run --vus 10 --duration 30s script.js   # same shape, set from the CLI

What success looks like: a live progress table during the run, then an end-of-run summary in which every threshold line is marked with a green check. http_req_duration reports avg, min, med, max, p(90), and p(95) (k6 metrics). If any threshold fails, "the little green checkmark next to the threshold name would be a red cross and k6 would exit with a non-zero exit code" (k6 thresholds). That non-zero exit is your entire CI gate: no plugin, no parser, no dashboard.

That first run is a smoke test. Only after it is green should you raise VUs or switch to an arrival-rate executor.

Percentiles, not averages

Report and gate on percentiles. An average response time is a single number produced by summing everything and dividing, so a handful of 8-second requests disappear into thousands of 40ms ones. The p95 and p99 are the response times that 5% and 1% of requests exceeded, which is the experience of your slowest users and, on a page that makes 20 backend calls, the experience of most sessions.

This is why every tool in the list exposes percentiles as first-class threshold targets: k6 with http_req_duration: ['p(95)<200'] (k6 thresholds) and Artillery with p95: 200 in its ensure block (Artillery ensure).

Practical rules, which are practitioner convention rather than a documented standard:

• Quote p50, p95, and p99 together. p50 alone flatters you; p99 alone is noisy at low request counts, and a p99 over 200 requests is two data points.
• Never average percentiles across separate runs or separate load generators. The result is not a percentile of anything.
• Compare like with like: same profile, same duration, same environment. A p95 from a 30-second smoke run is not comparable to one from a two-hour soak.

A test without a threshold is just a graph

If a run cannot fail, nobody will ever act on it. Define the pass/fail budget before the run, and derive the numbers from your service level objectives, not from whatever the first run happened to produce.

k6 states this directly: "Thresholds are the pass/fail criteria that you define for your test metrics. If the performance of the system under test (SUT) does not meet the conditions of your threshold, the test finishes with a failed status" (k6 thresholds). Set abortOnFail on a threshold to stop the run the moment the condition goes false instead of burning the full duration (k6 thresholds).

Artillery's ensure extension is the same idea in YAML: "Artillery can validate if a metric's value meets a predefined threshold. If it doesn't, it will exit with a non-zero exit code" (Artillery ensure).

The k6 guidance on where to run these is worth taking seriously: "As a general rule on pre-release environments, we should run our larger tests with quality gates, Pass/Fail criteria that validate SLOs or reliability goals," but "Unless your verification process is mature, do not rely entirely on Pass/Fail results to guarantee the reliability of releases" (k6 automated performance testing). Gate the short smoke and average-load runs in CI; run stress, spike, soak, and breakpoint on a schedule against a dedicated environment.

Performance incident workflow

When an alert, APM spike, or customer report signals a live performance incident, the goal is to localize the dominant cause under time pressure. Required inputs: the affected endpoint (or service name) plus the observed symptom (p95 latency, error rate, CPU saturation, or DB load). Do not proceed without an endpoint.

Step 1 - Confirm and reproduce

Run a smoke k6 script against the affected endpoint to confirm the symptom is reproducible and measure its current magnitude. Per k6 running docs, a minimal confirmation run with a thresholds block:

export const options = {
  stages: [{ duration: '60s', target: 20 }],
  thresholds: {
    http_req_duration: ['p(95)<500'],
    http_req_failed:   ['rate<0.01'],
  },
};

Run it with --summary-export=summary.json --quiet and parse the result:

jq -r '.metrics | to_entries[] | select(.value.thresholds) | .key + ": " + (.value.thresholds | to_entries | map("\(.key) -> \(if .value.ok then "PASS" else "FAIL" end)") | join(", "))' summary.json

Per k6 thresholds docs, a non-zero exit and "ok": false on a threshold confirms the regression is deterministic before investing in deeper diagnosis. If the run passes all thresholds, the incident may be intermittent or already resolved - state that explicitly and stop.

Step 2 - Flame-graph the hot path

With the service running under the k6 load from Step 1, capture a CPU profile using flame-graph-analyzer: run the runtime-appropriate profiler (py-spy / async-profiler / Go pprof / clinic.js flame) for 30 seconds under live load, sort folded stacks by sample count, surface the top 5 leaf frames, and classify each (CPU-bound hot algo, allocator pressure, lock contention, reflection overhead). The widest leaf in the flame graph is the hot path per Brendan Gregg's canonical flame-graph reference. If the flame graph shows DB-bound frames (e.g. pg_send_query_blocking, mysql_send_query) as the dominant cost, the bottleneck is database-side - proceed directly to Step 3 and skip app-side remediation. If a flame graph cannot be captured (no profiler available, no access to the process), state the blocker; do not guess the hot path from code review alone.

Step 3 - Detect slow queries

If Step 2 points to DB-bound cost (or is inconclusive), use db-query-plan-analyzer: capture EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON, SETTINGS) for the suspect query (PostgreSQL) or EXPLAIN ANALYZE (MySQL 8.0+) per pg-explain docs, identify the dominant plan node (Seq Scan, Sort spill, Nested Loop with high inner-side row count), and emit the candidate CREATE INDEX or query rewrite. Find the hottest node with jq:

jq '[.. | objects | select(.["Node Type"] and .["Actual Total Time"]) | {node: .["Node Type"], time: .["Actual Total Time"]}] | sort_by(-.time) | .[0]' plan.json

Step 4 - Localize and recommend

Combine the k6 confirmation delta, the flame-graph top frame, and the slow-query plan node into a single cause statement, one of:

• App-side CPU: dominant hot path is in user code (e.g. JSON.stringify, a hash function, a regex) - recommend algorithm or serialization change.
• App-side allocator: GC frames dominate - recommend object pooling or streaming serialization.
• DB-side: Seq Scan or sort spill dominates - emit the CREATE INDEX candidate.
• Mixed: both app-side and DB-side cost are significant - order recommendations by sample share descending.

Emit a triage report: symptom confirmed (observed vs budget), flame-graph findings table (rank, sample share, leaf stack, category), slow-query findings if DB-bound, the localized cause paragraph, and recommended actions ordered by impact - ending with a re-run of the k6 confirmation test after the fix. If the cause is still inconclusive after Steps 2 and 3, the introducing commit is unknown: hand off to the regression-bisector agent (qa-flake-triage) in its perf-measurement mode to bisect for it.

Traps that catch newcomers first

The seven mistakes that most often invalidate a first effort - load testing the generator itself, running JMeter's GUI as the generator, one URL / one account, presenting a closed-model VU count as capacity, skipping warm-up, extrapolating from a scaled-down environment, and recording latency without error rate - are detailed with citations in references/traps.md.

Related skills

Optional deeper dives, if you have them installed: k6-load-testing (including latency-percentile interpretation in its references), jmeter-load-testing, flame-graph-analyzer, db-query-plan-analyzer. Gatling and Locust deep dives live in this skill's references/gatling.md and references/locust.md.