OpenTelemetry Skill

Core Principles

Use these defaults:

1. Stability over Features: Check otelcol-contrib stability (Alpha/Beta/Stable); warn before production use of non-stable components.

2. Convention over Configuration: Prefer OpenTelemetry Semantic Conventions over custom attribute names.

3. Protocol Unification: Default to OTLP gRPC (port 4317); use OTLP HTTP (port 4318) when gRPC is unavailable due to agent, proxy, browser, or backend constraints.

4. Deterministic Routing Keys: For load-balancing exporters, use stable, deterministic routing keys — traceID for tail-sampling stickiness, tenant_id or cluster for tenant/shard routing. Normalize non-string attributes before routing.

5. Safety First: Prioritize collector stability (memory limiters, persistent queues, backpressure) over data completeness. Dropping data is preferable to crashing the collector.

6. Cardinality Awareness: High-cardinality attributes (>100 unique values) must NOT be metric dimensions — use traces or logs instead.

7. Security by Default: Redact PII, enable TLS for cross-network communication, and authenticate all collector endpoints.

8. Cross-Field Consistency: Treat collector reviews as systems reviews, not YAML linting. Compare processor order, memory limits, replica strategy, routing, metric temporality state, queue storage medium, PDB/HPA settings, and OTTL attribute types together before calling a config "safe".

Pre-Flight Checklist

Before generating config/code, confirm these. If unknown, ask first:

1. Signal volume — High traffic (>10k RPS) or low volume? Drives sampling/scaling. → sampling.md, collector.md
2. Cardinality risk — Any unbounded metric attributes (user/request/session IDs)? Move to traces/logs. → instrumentation.md
3. Resiliency — Is restart/outage data loss acceptable? If no, use file_storage + persistent queues. → collector.md
4. Trust boundaries — Any public-network hops? Require TLS + mTLS. → security.md
5. Deployment target — Kubernetes, EC2, Lambda, or containers? → architecture.md

Eval-Critical Response Minimums

When user requests match these patterns, include these points explicitly:

• Collector setup: include memory_limiter; keep it first in each pipeline processors list; explain OOM-prevention rationale.
• Metric dimension request for user_id: refuse; explain time-series explosion risk; suggest traces and bounded metric dimensions.
• Kubernetes tail sampling: Gateway (Deployment) tier, loadbalancing with routing_key: traceID, Headless Service (clusterIP: None), error+10% policies, Beta stability caution.
• Claude Code telemetry: include CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative, ~/.claude/settings.json persistence; OTEL_LOG_USER_PROMPTS/OTEL_LOG_TOOL_DETAILS default to false — warn against enabling in shared/production environments without PII controls; avoid session.id as a metric dimension.

Existing Configuration Review Mode

When the user provides an existing collector config, Helm values file, or Kubernetes manifest, audit it for internal contradictions before proposing edits.

Always compare:

1. Memory limiter vs pod limit — limit_mib must stay below the container memory limit with headroom for the Go runtime and internal buffers.
2. Stateful processing vs scaling — tail_sampling, spanmetrics, and servicegraph require sticky routing when replicas can exceed 1.
3. Exporter durability vs outage tolerance — disabled retries and no persistent queue imply data loss during backend failures.
4. Network exposure vs deployment mode — hostPort is usually a DaemonSet/node-local choice, not a horizontally scaled gateway Deployment default.
5. Rollout settings together — review replicaCount, HPA minReplicas, PodDisruptionBudget, and rolling update settings as one unit.
6. OTTL/filter correctness — keep attribute types consistent (for example bool vs string) and prefer current semantic convention keys such as http.response.status_code.
7. Dead config — flag processors/exporters/extensions that are declared but never referenced by any pipeline.
8. Metric temporality and state — deltatocumulative / cumulativetodelta are stateful conversions; verify source temporality, backend expectations, restart tolerance, and any replica/routing assumptions before enabling them.
9. Queue storage backend — file_storage needs local locking-safe storage; ReadWriteMany / EFS / NFS-style volumes are not safe defaults.

Progressive Disclosure: Context Triggers

Load detailed reference documentation only when the user's request matches a trigger. This keeps context lean.

Production Baseline Configuration

Use these defaults unless the user specifies otherwise. This is a copy-paste-ready starting point:

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"
  file_storage/queue:
    directory: /var/lib/otelcol/queue
    timeout: 10s
    compaction:
      on_start: true
      on_rebound: false

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlp:
    endpoint: "your-backend:4317"
    sending_queue:
      enabled: true
      storage: file_storage/queue
      num_consumers: 4
      queue_size: 1024
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 300s
  # otlphttp:                        # HTTP exporter — use when backend requires HTTP
  #   endpoint: "https://your-backend:4318"
  #   sending_queue: { enabled: true, storage: file_storage/queue }

service:
  extensions: [health_check, file_storage/queue]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Key defaults:

• memory_limiter must be first in every processor chain.
• batch reduces exporter network calls.
• file_storage preserves queues across restarts only when the collector returns to the same host/volume. In Kubernetes, back /var/lib/otelcol/queue with a ReadWriteOnce block-backed PVC rather than RWX/network storage.
• health_check binds to localhost (not 0.0.0.0) in shared networks.
• Prefer OTLP gRPC (port 4317) for receivers and exporters. Fall back to OTLP HTTP (port 4318) when gRPC is unavailable.

Validation & Error Recovery

Always include validation checkpoints when delivering configurations.

Validate before deploying

# Syntax and structural validation (local binary)
otelcol validate --config config.yaml

# Container-based dry-run (no outbound traffic)
docker run --rm -v $(pwd)/config.yaml:/etc/otelcol/config.yaml \
  otel/opentelemetry-collector-contrib:latest \
  validate --config /etc/otelcol/config.yaml

Verify a live pipeline

# Health endpoint — returns 200 when collector is ready
curl -sf http://localhost:13133/ && echo "healthy"

# Tail logs for pipeline errors and dropped data
kubectl logs -l app=otelcol -f | grep -E "error|dropped|refused|timeout"

# Collector self-metrics (Prometheus scrape)
curl -s http://localhost:8888/metrics | grep -E "otelcol_processor_dropped|otelcol_exporter_send_failed"

Error recovery guidance

Anti-Patterns to Avoid

The most critical patterns are listed here. See anti-patterns.md for the full annotated catalog.

❌ Placing memory_limiter anywhere except first in the processor chain ❌ Using high-cardinality attributes (user_id, trace_id) as metric dimensions ❌ Exposing pprof (1777), zpages (55679) on 0.0.0.0 in production ❌ Using tail_sampling without sticky session load balancing (loadbalancing exporter) ❌ Omitting batch processor (causes excessive network calls) ❌ Calling a config "fine" because it parses, without checking memory limits, sticky routing, exporter durability, and rollout settings together

Version and Compatibility

• Use compatibility.md for fast-moving version floors and AI agent support details.
• Keep SKILL.md focused on routing logic, guardrails, and production defaults rather than inline release tracking.