LogQL Query Generator

When to Use This Skill

• Creating LogQL queries for log analysis, dashboards, or alerting
• Converting log analysis requirements into LogQL expressions
• Troubleshooting applications through log analysis
• Working with structured logs (JSON, logfmt)

Interactive Query Planning Workflow

CRITICAL: Always engage the user in collaborative planning before generating queries.

Stage 1: Understand the Goal

Gather requirements using AskUserQuestion:

1. Primary Goal: Error analysis, performance tracking, security monitoring, debugging, pattern detection?
2. Use Case: Dashboard, alerting rule, ad-hoc troubleshooting, metrics generation?
3. Context: Application/service, environment, time range, log format?

Stage 2: Identify Log Sources

1. Labels: What labels identify your logs? (job, namespace, app, level, service_name)
2. Log Format: JSON, logfmt, plain text, or custom?
3. Strategy: Use labels for stream selection (indexed), line filters for content (not indexed)

Stage 3: Determine Query Parameters

1. Query Type: Log query (return lines) or metric query (calculate values)?
2. Filtering: Stream selector {job="app"}, line filters |= "error", label filters | status >= 500
3. Parsing: | json, | logfmt, | pattern "<ip> - <user>", | regexp "(?P<field>...)"
4. Aggregation: count_over_time(), rate(), sum by (label), quantile_over_time()
5. Time Range: [5m], [1h], [24h]

Stage 4: Plan, Validate & Consult References

Before generating code, present a plain-English plan and confirm with the user via AskUserQuestion:

## LogQL Query Plan

**Goal**: [Description]
**Query Structure**:
1. Select streams: `{label="value"}`
2. Filter lines: [operations]
3. Parse logs: [parser]
4. Aggregate: [function]

**Does this match your intentions?**

Once confirmed, MANDATORY: use the Read tool to consult the appropriate reference files before generating. Do NOT rely on prior knowledge or cached information.

Paths to use with the Read tool:

Read(".claude/skills/logql-generator/assets/common_queries.logql")   # Query patterns
Read(".claude/skills/logql-generator/references/best_practices.md")  # Optimization and anti-patterns

Why this matters: Reference files contain battle-tested patterns and edge cases not covered in the skill overview. Explicit consultation during each skill execution ensures you use the latest patterns and prevents syntax errors.

Stage 5: Generate Query

Best Practices

1. Specific Stream Selectors: {namespace="prod", app="api", level="error"} not just {namespace="prod"}
2. Filter Order: Line filter → parse → label filter (fastest to slowest)
3. Parser Performance: pattern > logfmt > json > regexp

Core Query Patterns

Log Filtering:

{job="app"} |= "error" |= "timeout"        # Contains both
{job="app"} |~ "error|fatal|critical"       # Regex match
{job="app"} != "debug"                      # Exclude

JSON/logfmt Parsing:

{app="api"} | json | level="error" | status_code >= 500
{app="app"} | logfmt | caller="database.go"

Pattern Extraction:

{job="nginx"} | pattern "<ip> - - [<_>] \"<method> <path>\" <status> <size>"

Metrics:

# Rate
rate({job="app"} | json | level="error" [5m])

# Count by label
sum by (app) (count_over_time({namespace="prod"} | json [5m]))

# Error percentage
sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m])) * 100

# Latency percentiles
quantile_over_time(0.95, {app="api"} | json | unwrap duration [5m])

# Top N
topk(10, sum by (error_type) (count_over_time({job="app"} | json | level="error" [1h])))

Formatting:

{job="app"} | json | line_format "{{.level}}: {{.message}}"
{job="app"} | json | label_format env="{{.environment}}"

IP Filtering (prefer label filter after parsing for precision):

{job="nginx"} | logfmt | remote_addr = ip("192.168.4.0/24")

Stage 5a: Incremental Query Building (Educational/Debugging)

When to use this stage:

• User is learning LogQL
• Complex multi-stage queries
• Debugging query issues
• User explicitly requests step-by-step explanation

Present the query construction incrementally:

## Building Your Query Step-by-Step

### Step 1: Stream Selector (verify logs exist)

```logql
{app="api"}

Test this first to confirm logs are flowing

Step 2: Add Line Filter (fast pre-filtering)

{app="api"} |= "error"

Reduces data before parsing

Step 3: Add Parser (extract fields)

{app="api"} |= "error" | json

Now you can filter on extracted labels

Step 4: Add Label Filter (precise filtering)

{app="api"} |= "error" | json | level="error"

Final filter on parsed data

Step 5: Add Aggregation (if metric query)

sum(count_over_time({app="api"} |= "error" | json | level="error" [5m]))

Complete metric query

**Benefits of incremental building:**
1. Identify which step breaks (no results, parse errors)
2. Understand performance impact of each operation
3. Debug unexpected results by testing each stage
4. Learn LogQL query structure naturally

**Use AskUserQuestion** to offer incremental mode:
- Option: "Show step-by-step construction" vs "Show final query only"

### Stage 6: Provide Usage

1. **Final Query** with explanation
2. **How to Use**: Grafana panel, Loki alerting rules, `logcli query`, HTTP API
3. **Customization**: Labels to modify, thresholds to tune

## Advanced Techniques

### Multiple Parsers

```logql
{app="api"} | json | regexp "user_(?P<user_id>\\d+)"

Unwrap for Numeric Metrics

sum(sum_over_time({app="api"} | json | unwrap duration [5m]))

Pattern Match Operators (Loki 3.0+, 10x faster than regex)

{service_name=`app`} |> "<_> level=debug <_>"

Logical Operators

{app="api"} | json | (status_code >= 400 and status_code < 500) or level="error"

Offset Modifier

sum(rate({app="api"} | json | level="error" [5m])) - sum(rate({app="api"} | json | level="error" [5m] offset 1d))

Label Operations

{app="api"} | json | keep namespace, pod, level
{app="api"} | json | drop pod, instance

Note: LogQL has no dedup or distinct operators. Use metric aggregations like sum by (field) for programmatic deduplication.

Loki 3.x Key Features

Structured Metadata

High-cardinality data without indexing (trace_id, user_id, request_id):

# Filter AFTER stream selector, NOT in it
{app="api"} | trace_id="abc123" | json | level="error"

Query Acceleration (Bloom Filters)

Place structured metadata filters BEFORE parsers:

# ACCELERATED
{cluster="prod"} | detected_level="error" | logfmt | json
# NOT ACCELERATED
{cluster="prod"} | logfmt | json | detected_level="error"

approx_topk (Probabilistic)

approx_topk(10, sum by (endpoint) (rate({app="api"}[5m])))

vector() for Alerting

sum(count_over_time({app="api"} | json | level="error" [5m])) or vector(0)

Automatic Labels

• service_name: Auto-populated from container name
• detected_level: Auto-detected when discover_log_levels: true (stored as structured metadata)

Quick Function Reference

Log Range Aggregations (most common)

Unwrapped Range Aggregations (most common)

Aggregation Operators

sum, avg, min, max, count, stddev, topk, bottomk, approx_topk, sort, sort_desc

With grouping: sum by (label1, label2) or sum without (label1)

Conversion Functions

label_replace()

label_replace(rate({job="api"} |= "err" [1m]), "foo", "$1", "service", "(.*):.*")

Parser Reference

logfmt

| logfmt [--strict] [--keep-empty]

• --strict: Error on malformed entries
• --keep-empty: Keep standalone keys

JSON

| json                                           # All fields
| json method="request.method", status="response.status"  # Specific fields
| json servers[0], headers="request.headers[\"User-Agent\"]"  # Nested/array

Template Functions

Common functions for line_format and label_format:

String: trim, upper, lower, replace, trunc, substr, printf, contains, hasPrefix Math: add, sub, mul, div, addf, subf, floor, ceil, round Date: date, now, unixEpoch, toDate, duration_seconds Regex: regexReplaceAll, count Other: fromJson, default, int, float64, __line__, __timestamp__

See assets/common_queries.logql for detailed usage.

Alerting Rules

# Alert when error rate exceeds 5%
(sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m]))) > 0.05

# With vector() to avoid "no data"
sum(rate({app="api"} | json | level="error" [5m])) or vector(0) > 10

Error Handling

Documentation Lookup

When to Fetch External Documentation (MANDATORY)

Trigger context7 MCP or WebSearch when the query involves ANY of these:

How to Use

context7 MCP (preferred - authoritative docs):

1. mcp__context7__resolve-library-id with libraryName="grafana loki"
2. mcp__context7__get-library-docs with context7CompatibleLibraryID and topic="[specific topic]"

WebSearch (fallback for latest features):

WebSearch query: "Grafana Loki LogQL [topic] documentation [year]"

Example Workflow

When user asks for "error tracking with trace correlation in Loki 3.x":

1. Recognize trigger: "Loki 3.x" + "trace" → structured metadata
2. Fetch docs: mcp__context7__get-library-docs with topic="structured metadata trace_id"
3. Apply patterns from docs to generate accurate query

Resources

• assets/common_queries.logql: Comprehensive query examples
• references/best_practices.md: 39+ LogQL best practices, performance optimization, anti-patterns

Guidelines

1. Always plan interactively - Present plain-English plan before generating
2. Use AskUserQuestion - Gather requirements and confirm plans
3. MUST use Read tool for complex queries - See Stage 4 for the authoritative table of when and which files to read; do NOT skip this step or rely on prior knowledge
4. Fetch docs for advanced features - Use context7 MCP when Loki 3.x features, approx_topk, or unclear syntax is involved (see Documentation Lookup triggers)
5. Offer incremental building - For learning or debugging, present step-by-step query construction (see Stage 5a)
6. Explain queries - What it does, how to interpret results
7. Prioritize performance - Specific selectors, filter early, simpler parsers

Version Notes

• Loki 3.0+: Bloom filters, structured metadata, pattern match operators (|>, !>)
• Loki 3.3+: approx_topk function
• Loki 3.5+: Promtail deprecated (use Grafana Alloy)
• Loki 3.6+: Horizontally scalable compactor, Loki UI as Grafana plugin

Deprecations: Promtail (use Alloy), BoltDB store (use TSDB with v13 schema)