PromQL Query Generator

When to Use This Skill

Use this skill when you need to:

• Write a PromQL query for error rates, latency percentiles, or availability.
• Author recording rules or SLO burn-rate alerting rules.
• Aggregate metrics across labels using RED or USE monitoring patterns.
• Build queries for Prometheus counters, gauges, histograms, or summaries.
• Draft Kubernetes or availability queries from the patterns in assets/.

When Not to Use

• Validating or optimising an existing query: use promql-validator instead.
• Non-Prometheus query languages (LogQL, MetricsQL dialect specifics, SQL): use the matching tool.
• Configuring Prometheus scrape targets or storage: that is server config, not query authoring.
• Diagnosing why a metric is missing from Prometheus: check up{job="..."} and the scrape config instead.

Philosophy

• Match the function to the metric type. rate() is for counters, histogram_quantile() for histograms; applying rate() to a gauge is meaningless.
• Error rate is a ratio. Express it as rated errors over rated totals across the same window, not as raw counts.
• Control cardinality on purpose. Choose by()/without() labels deliberately so a query cannot explode a dashboard.
• Alert on sustained burn. SLO alerts use multi-window burn rates and a for clause, not a single instantaneous threshold.
• Plan before you write. Confirm goal, metric type and parameters with the user before generating any query.

Interactive Query Planning Workflow

CRITICAL: Always engage the user in collaborative planning before generating any query. Never skip the planning phase.

Workflow (7 stages)

1. Understand the goal: Ask what the user wants to monitor (request rate, error rate, latency, resource usage, availability, SLO tracking) and the use case (dashboard, alert, recording rule, ad-hoc).
2. Identify metrics: Confirm metric names, types (counter/gauge/histogram/summary), and relevant labels. Suggest common naming patterns if uncertain.
3. Determine parameters: Confirm time range, label filters, aggregation, and thresholds. If the user already specified values (e.g., "5-minute window", "> 5% error rate"), acknowledge them as pre-filled defaults and allow quick confirmation rather than re-asking.
4. Present the query plan: Before writing any code, present a plain-English plan (goal, query structure, expected output, example interpretation) and ask for confirmation and offer the options: "Yes, generate this query" / "Modify [aspect]" / "Show alternatives".
5. Generate the query: Once confirmed, read the relevant reference file(s) before writing code, cite the applicable pattern, and apply the best practices below.
6. Validate: Automatically invoke promql-validator. Display structured results (syntax, best practices, explanation). Fix any issues and re-validate until all checks pass.
7. Deliver: Provide the final query, plain-English explanation, usage instructions (dashboard / alert / recording rule), customization notes, and related query suggestions.

Ask vs. Infer: If the user's request already clearly specifies goal, use case, and context, acknowledge those details instead of re-asking. Only ask for missing or ambiguous information.

Best Practices for Query Generation

Always consult the relevant reference file before writing code.

Key Rules

1. Always add label filters: reduces cardinality and improves performance.
2. Match functions to metric types: rate()/increase() on counters; *_over_time() or direct use for gauges; histogram_quantile() for histograms.
3. Prefer by()/without() on all aggregations.
4. Prefer exact label matches over regex when the value is known.
5. Use recording rules for queries that are expensive or reused frequently (naming: level:metric:operations).
6. Format multi-line for complex queries.

Core Patterns

# Request rate (counter)
sum(rate(http_requests_total{job="api-server"}[5m])) by (endpoint)

# Error rate ratio
sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="api-server"}[5m]))

# P95 latency (classic histogram)
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api-server"}[5m]))
)

# P95 latency (native histogram, Prometheus 3.x+)
histogram_quantile(0.95,
  sum by (job) (rate(http_request_duration_seconds[5m]))
)

# Availability
(count(up{job="api-server"} == 1) / count(up{job="api-server"})) * 100

# Burn rate (99.9% SLO, 1h window)
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="api"}[1h]))
) / 0.001

# Multi-window burn-rate alert (page: 2% budget in 1h, burn rate 14.4)
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="api"}[1h]))
) > 14.4 * 0.001
and
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m]))
) > 14.4 * 0.001

For complete SLO patterns, Native Histogram functions (histogram_count, histogram_sum, histogram_fraction), subqueries, offset/@ modifiers, vector matching, and Kubernetes patterns: see the assets/ files.

Validation Checklist

After generating, invoke promql-validator and display results in this format:

## PromQL Validation Results

### Syntax Check
- Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR
- Issues: [list any syntax errors]

### Best Practices Check
- Status: ✅ OPTIMIZED / ⚠️ CAN BE IMPROVED / ❌ HAS ISSUES
- Issues: [list problems found]
- Suggestions: [list optimizations]

### Query Explanation
- What it measures: [plain English]
- Output labels: [label list or "None (scalar)"]
- Expected result structure: [instant vector / scalar / etc.]

Fix all issues and re-validate until clean.

Verifying Generated Rules

Check the shipped rule files, or your own, with promtool before loading them into Prometheus:

promtool check rules assets/alerting_rules.yaml
promtool check rules assets/recording_rules.yaml

Run a candidate query against a live server to confirm it returns series:

curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=sum by (job) (rate(http_requests_total[5m]))'

Query a recorded series instead of repeating its expression:

job:http_requests:rate5m{job="api-server"}

Use *_over_time() for a gauge rather than rate():

avg_over_time(node_memory_MemAvailable_bytes[5m])

Browse the reusable query templates with a plain listing:

grep -n "^# " assets/common_queries.promql

Alerting and Recording Rule Snippets

# Alerting rule with for clause
alert: HighErrorRate
expr: |
  (
    sum(rate(http_requests_total{status_code=~"5.."}[5m]))
    / sum(rate(http_requests_total[5m]))
  ) > 0.05
for: 10m

# Recording rule (naming: level:metric:operations)
- record: job:http_requests:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))

Anti-Patterns

Each entry below pairs a reason with a BAD and a GOOD query. See references/best_practices.md for further performance and cardinality guidance.

NEVER use rate() on a gauge metric

WHY: rate() computes per-second rate of increase and assumes monotonically increasing counters. Applied to a gauge, it produces nonsensical results because gauges can decrease.

BAD: rate(node_memory_MemFree_bytes[5m]): memory is a gauge

GOOD: node_memory_MemFree_bytes (direct use) or delta(node_memory_MemFree_bytes[5m]) for change over time

NEVER query a histogram with avg() across quantile labels

WHY: Summary metrics expose pre-aggregated {quantile="0.95"} labels that cannot be re-aggregated across instances. Using avg() on them produces statistically meaningless results.

BAD: avg(http_request_duration_seconds{quantile="0.95"})

GOOD: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))): use histogram type with histogram_quantile()

NEVER use a high-cardinality label in by() without filtering first

WHY: Aggregating by user_id, request_id, or pod without label filters produces thousands of series, overwhelming dashboards and recording rules.

BAD: sum by (user_id) (rate(http_requests_total[5m]))

GOOD: sum by (job, status_code) (rate(http_requests_total{job="api"}[5m])): filter and aggregate on stable, low-cardinality labels

NEVER use increase() for alerting thresholds

WHY: increase() is extrapolated and can return non-integer values on sparse counters. For alerting, rate() produces stable per-second thresholds that are scrape-interval independent.

BAD: increase(http_requests_total{status=~"5.."}[5m]) > 10

GOOD: rate(http_requests_total{status=~"5.."}[5m]) > 0.033 (2 errors/minute = 0.033/second)

NEVER omit the for clause on alert rules

WHY: Alerts without for fire immediately on a single evaluation, causing false positives from transient spikes. The for clause requires the condition to be true for a sustained period.

BAD: alert: HighErrorRate with expr: error_rate > 0.05 and no for

GOOD: Add for: 5m to require the condition to hold for 5 minutes before firing

NEVER express an error rate as a raw count

WHY: Raw error counts rise with traffic, so a busy service looks worse than a failing quiet one. A ratio of rated errors over rated totals is comparable across load.

BAD: sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m])) used as the alert expression.

GOOD: sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api-server"}[5m]))

NEVER write a query before the user has confirmed the plan

WHY: Guessing the metric name, type or window produces a query that parses but answers the wrong question. The planning workflow exists to catch this before code is written.

BAD: Generate rate(http_requests_total[5m]) straight from "show me errors".

GOOD: Present the plain-English plan at stage 4 and wait for confirmation, then generate.

NEVER deliver a query without validating it

WHY: Stage 6 of the workflow catches syntax errors and best-practice issues before the user pastes the query into a dashboard or rule file.

BAD: Hand over the first draft and stop.

GOOD: Run the validator, show the Validation Checklist output, fix issues and re-validate until clean.

NEVER use a regex matcher when the label value is known

WHY: Equality matchers are cheaper and unambiguous, while a loose regex such as =~".*" can match far more series than intended. Key Rule 4 prefers exact matches.

BAD: http_requests_total{job=~"api.*"} when the job is api-server.

GOOD: http_requests_total{job="api-server"}

NEVER repeat an expensive expression across dashboards and alerts

WHY: Each copy is evaluated separately and the copies drift apart over time. A recording rule computes it once and gives every consumer the same series.

BAD: Pasting sum by (job) (rate(http_requests_total[5m])) into several panels and alert rules.

GOOD: Define - record: job:http_requests:rate5m once and query job:http_requests:rate5m everywhere.

NEVER use a bare irate() window for alert thresholds

WHY: irate() only looks at the last two samples, so a single noisy scrape can trip or clear an alert. rate() over a window smooths this out.

BAD: irate(http_requests_total{job="api-server", status_code=~"5.."}[1m]) > 0.05

GOOD: rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]) > 0.05

NEVER mix metrics from different windows in one ratio

WHY: An error rate that divides a 5m rate by a 1h rate compares unlike quantities and gives a misleading percentage.

BAD: sum(rate(errors_total[5m])) / sum(rate(requests_total[1h]))

GOOD: sum(rate(errors_total[5m])) / sum(rate(requests_total[5m]))

Documentation Lookup

1. Documentation MCP server (preferred): resolve prometheus, then fetch docs with the relevant topic.
2. Fallback WebSearch: "Prometheus PromQL [function/operator] documentation [version] examples"

Error Handling Quick Reference

References

Internal:

• PromQL Functions: all PromQL functions with examples; read for specific function syntax questions
• PromQL Patterns: RED/USE method patterns, alerting rules, recording rules; read for standard monitoring patterns
• Best Practices: anti-patterns, performance optimization, cardinality management; read when optimizing queries
• Metric Types: counter/gauge/histogram/summary guide; read to confirm correct function choice
• Common Queries: reusable request rate, error rate, latency, and availability query templates
• RED Method: complete RED method (Rate, Errors, Duration) implementation
• USE Method: complete USE method (Utilization, Saturation, Errors) implementation
• SLO Patterns: SLO, error budget, burn rate, and multi-window alerting patterns
• Alerting Rules: example alerting rules with thresholds and for clauses
• Recording Rules: example recording rules with level:metric:operations naming
• Kubernetes Patterns: kube-state-metrics, cAdvisor, and vector matching examples