jmeter-load-testing

Overview

Apache JMeter is the long-running JVM-native load-testing tool. Tests are XML .jmx files authored in JMeter's GUI; CI runs them headless via the jmeter CLI (jmeter-getstarted).

The official guidance is explicit:

"GUI mode should only be used for creating the test script, CLI mode (NON GUI) must be used for load testing." (jmeter-getstarted)

This skill covers the CLI / CI side. Authoring is GUI-driven and out of scope here - see the JMeter user manual for the Thread Group / HTTP Sampler / Assertion authoring flow.

When to use

• The project already has .jmx test plans.
• The team is on the JVM and prefers JMeter's mature ecosystem (BlazeMeter, Taurus, plugin manager, distributed mode).
• A specific protocol is needed that JMeter has first-class support for: JDBC, JMS, MQTT, FTP, LDAP, SOAP - k6's plugin landscape is weaker for niche protocols.

If the team is starting fresh, evaluate k6-load-testing (developer-friendly JS), gatling-load-testing (JVM DSL), or locust-load-testing (Python) before adopting JMeter - XML authoring has a steep learning curve.

Install

JMeter requires Java 8 or higher with JAVA_HOME set. Download the latest release from jmeter.apache.org and extract - there is no installer (jmeter-getstarted).

For Docker-based CI, official images are at apache/jmeter. Pin to a specific tag rather than latest.

Running

Canonical CLI invocation

Per jmeter-getstarted:

jmeter -n -t test.jmx -l results.jtl

With HTML dashboard

Per jmeter-getstarted:

jmeter -n -t test.jmx -l results.jtl -e -o report_folder

The dashboard shows percentiles, throughput, error rates, response- time graphs, and per-sampler breakdowns - the canonical JMeter output for human review.

Other useful flags

For multi-environment runs, parameterize the test plan with ${__P(api.base.url, default)} and override at the CLI:

jmeter -n -t orders.jmx -l results.jtl \
  -Japi.base.url=https://staging.example.com \
  -Japi.token=$API_TOKEN

Parsing results

The JTL file is the structured artifact for CI gating. Default JTL columns (CSV): timeStamp, elapsed, label, responseCode, responseMessage, threadName, dataType, success, failureMessage, bytes, sentBytes, grpThreads, allThreads, URL, Latency, IdleTime, Connect.

Quick-and-dirty pass/fail with awk:

# Count errors
awk -F',' 'NR>1 && $8=="false"' results.jtl | wc -l

# Compute mean response time
awk -F',' 'NR>1 { sum+=$2; n++ } END { print sum/n }' results.jtl

For richer parsing, use the JMeter HTML report's statistics.json under the report folder - it contains the percentiles per sampler in JSON form.

CI integration

# .github/workflows/jmeter.yml
name: load-test

on:
  pull_request:
    paths: ['tests/load/**.jmx']
  schedule:
    - cron: '0 4 * * *'

jobs:
  jmeter:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - name: Run JMeter via Docker
        env:
          API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
        run: |
          docker run --rm \
            -v "$PWD:/work" \
            -w /work \
            apache/jmeter \
              -n -t tests/load/orders.jmx \
              -l results.jtl \
              -e -o report \
              -Japi.token=$API_TOKEN \
              -Japi.base.url=https://staging.example.com

      - name: Pass/fail gate
        run: |
          ERRORS=$(awk -F',' 'NR>1 && $8=="false"' results.jtl | wc -l)
          if [ "$ERRORS" -gt 10 ]; then
            echo "::error::Got $ERRORS errors (>10 threshold)"
            exit 1
          fi

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: jmeter-report
          path: |
            report/
            results.jtl
          retention-days: 14

The Docker invocation pattern keeps the CI runner clean - no Java / JMeter install on the runner.

Anti-patterns

Limitations

• XML authoring. .jmx files are not human-friendly; reviews of test-plan changes are awkward.
• Memory hungry. JMeter's per-sample memory overhead is high vs. k6 / Gatling.
• Single-machine VU limits. ~1000 threads per machine before JVM contention; for higher loads, use distributed mode (master + remote slaves) or move to a tool with a smaller per-VU footprint.
• Plugin ecosystem fragmentation. Many JMeter plugins are abandoned; vet plugin maintenance status before depending on one.

References

• jmeter-getstarted - canonical CLI flags, GUI-vs-CLI guidance, dashboard generation.
• Apache JMeter user manual - https://jmeter.apache.org/usermanual/
• k6-load-testing, gatling-load-testing, locust-load-testing - alternatives by stack.
• perf-budget-gate - downstream gate aggregating multiple runner verdicts.