steady-state-hypothesis-validator

Overview

A chaos experiment's steady-state hypothesis is the contract that determines whether an experiment is scientifically useful or a no-op. Per principlesofchaos.org Principle 1:

"Focus on the measurable output of a system, rather than internal attributes of the system. Measurements of that output over a short period of time constitute a proxy for the system's steady state."

A hypothesis that cannot be measured, has no baseline, or would not move under the injected fault produces a verdict that means nothing. This skill runs five pre-flight checks against the hypothesis block before any tooling executes, catching bad hypotheses while the cost of fixing them is low.

The Chaos Toolkit steady-state-hypothesis block

The steady-state-hypothesis object requires a title and one or more probes, each with a provider and a tolerance gate; if a probe's return value fails its tolerance before the method, the experiment bails before running. The full field list, the eight supported tolerance forms (scalar, boolean, string, range, membership, regex, JSONPath, range object), and the pre-/post-method evaluation flow are in references/chaostoolkit-tolerance.md, per chaostoolkit.org/reference/api/experiment/ and chaostoolkit.org/reference/concepts/.

The five pre-flight checks

Check 1 - Metric is measurable and observable

The probe must query a real data source the team can access right now: a Prometheus query endpoint, a Datadog API, an HTTP health endpoint, a process exit code. The metric must already be instrumented.

Fail signals:

• The probe provider points to a dashboard URL rather than an API endpoint, or the only way to evaluate the metric is to read it manually.
• The metric name is a made-up label not yet emitted by any service.
• The probe requires credentials or tooling not available in the environment where the experiment will run.

Pass signal: The team can run the probe in isolation right now and get a numeric or boolean return value.

Check 2 - A recent baseline exists

Per principlesofchaos.org: "Measurements of that output over a short period of time constitute a proxy for the system's steady state." The tolerance must be anchored to observed behavior, not a guess.

Fail signals:

• The threshold is a round number with no supporting measurement (e.g., >= 99% when the service has never been measured).
• The most recent baseline measurement is older than 30 days, or predates the last deployment.
• The baseline was taken during a known incident or load spike.

Pass signal: The team can cite a dashboard, runbook, or monitoring record showing the metric's typical value over the past 7-30 days in normal production or staging traffic.

Check 3 - Tolerance is numerically meaningful and SLI-backed

The tolerance bounds must reflect a real service-level indicator (SLI), not an arbitrary threshold that would never be breached even during a real incident.

Fail signals:

• Tolerance is so wide it accepts total degradation, i.e. the value is "system is completely down" rather than "noticeably degraded" (e.g., >= 0%).
• Tolerance is tighter than the metric's normal noise band, so it fails spuriously in baseline conditions.
• No SLO or SLI document backs the threshold choice.

Pass signal: The threshold maps to a published SLO, an error budget line, or a documented user-impact threshold (e.g., checkout completion >= 95% because below that the on-call alert fires).

Diagnostic questions:

• What SLO or alert threshold does this tolerance align with?
• At what value would a real on-call alert fire?
• Has this metric ever breached this threshold under normal operations?

Check 4 - Measurement window is defined

A probe without a defined measurement window can return a point-in-time value that is unrepresentative of system behavior. The measured_over or equivalent window annotation in the experiment YAML must be present.

Fail signals:

• The probe queries a single-sample endpoint with no aggregation window.
• The experiment YAML specifies measured_over: 0 or omits it entirely.
• A single HTTP status check is used as a proxy for sustained service health.

Pass signal: The probe measures an aggregated value over a window of at least 1 minute (longer for low-traffic services). For Prometheus: a rate() or avg_over_time() expression with an explicit range vector. For Datadog: a rollup with a defined time window.

# Acceptable: aggregated over a window
probes:
  - name: checkout-completion-rate
    type: probe
    provider:
      type: http
      url: "https://metrics.internal/query?expr=avg_over_time(checkout_success_rate[5m])"
    tolerance:
      type: range
      range: [95.0, 100.0]

# Risky: single-sample point-in-time check
probes:
  - name: homepage-status
    type: probe
    provider:
      type: http
      url: "https://app.example.com/"
    tolerance: 200

Check 5 - The metric moves under the target failure mode

The most important check: would the injected fault actually cause this metric to change? A probe that is decoupled from the fault being injected produces a vacuous result.

Fail signals:

• The fault is a database connection failure, but the probe measures frontend CPU usage - the probe is on a data path the fault never touches.
• The fault affects one region, but the probe aggregates globally across all regions, so the failure is averaged away.
• The service has circuit-breaker or cached fallbacks that prevent the fault from ever reaching the probe's data path.

Pass signal: The team can trace the fault's propagation path from injection point to the metric's data source and confirm at least one step in that path directly affects the metric.

Diagnostic questions:

• Draw the call graph from injection point to the probe's data source. Is there a direct path?
• Does the service have a fallback that would mask the fault from this metric entirely?
• If this experiment "held" (metric stayed in tolerance), would that mean the system is resilient, or just that the metric is unrelated?

Worked example

Experiment: inject 500ms network latency on the payment-service pod; hypothesis is that checkout completion rate stays >= 95%.

steady-state-hypothesis:
  title: "Checkout completion rate stays above 95% under payment-service latency"
  probes:
    - name: checkout-completion-rate
      type: probe
      provider:
        type: http
        url: "https://metrics.internal/query?expr=avg_over_time(checkout_success_rate[5m])"
        timeout: 10
      tolerance:
        type: range
        range: [95.0, 100.0]

Pre-flight verdict against each check:

Verdict: hypothesis is sound. Proceed to experiment execution.

Hard-reject conditions

These patterns block execution outright - a Hard-reject triggered: yes in the output below. Each maps to the check that catches it, and each subsumes the soft anti-patterns that share its cause; do not proceed until resolved.

Output format

Emit one row per probe in the hypothesis block, then a summary verdict:

Steady-State Hypothesis Pre-Flight Report
==========================================
Experiment: <title>
Fault: <fault description>

Probe: <probe name>
  Check 1 (measurable):  PASS / FAIL - <reason>
  Check 2 (baseline):    PASS / FAIL - <reason>
  Check 3 (SLI-backed):  PASS / FAIL - <reason>
  Check 4 (window):      PASS / FAIL - <reason>
  Check 5 (moves):       PASS / FAIL - <reason>

Verdict: SOUND / UNSOUND
  Blocking issues: <list or "none">
  Hard-reject triggered: yes / no
  Recommended action: <proceed | revise probe | replace metric | add baseline>

Limitations

• This skill validates the hypothesis specification, not the live system. Instrumentation gaps (Check 1) or stale baselines (Check 2) can only be confirmed by running the probe manually before authoring the experiment.
• SLI-backing (Check 3) requires access to the team's SLO documents or alert configuration. If neither exists, the team should define a threshold and document the rationale in the experiment YAML before running.
• Check 5 (metric moves) is a reasoning exercise, not an automated trace. For complex microservice graphs, draw the dependency diagram manually.
• The Chaos Toolkit tolerance schema does not enforce window-based aggregation; a point-in-time HTTP probe is syntactically valid even if it is a poor steady-state indicator. This skill flags it as a warning, not a hard error (unless the probe returns a constant).

References

• principlesofchaos.org - Principle 1: "Build a Hypothesis around Steady State Behavior." Defines measurable output, throughput, error rates, latency percentiles as valid steady-state metrics.
• chaostoolkit.org/reference/api/experiment/ - steady-state-hypothesis block specification: required fields (title, probes), tolerance types (scalar, range, regex, jsonpath, probe), and evaluation semantics.
• chaostoolkit.org/reference/concepts/ - Experiment lifecycle: pre-method vs. post-method hypothesis check, bail-out behavior when pre-check fails, deviation detection after method.
• chaos-experiment-author - upstream skill that authors the experiment (Step 1 defines the hypothesis this skill validates).
• prod-canary-validator (in the qa-shift-right plugin) - provides the production steady-state metrics that can anchor hypothesis baselines (Check 2).