Investigate and root-cause production issues from live OpenTelemetry telemetry (traces, logs, metrics) through the Kopai connector or the Kopai SDK in TypeScript code mode. Use this skill to debug errors and error-rate spikes, investigate latency/slowness and timeouts, trace requests across services, find failing or cascading services, and correlate logs to a trace — including vague symptom reports like 'why is my API slow', 'getting 500 errors', 'service is down', 'requests are timing out', or 'something is failing in prod', even when the user never says 'traces' or 'observability'. This analyzes existing telemetry to find a cause. Do NOT use it to add instrumentation (use otel-instrumentation), to build dashboards or visualizations (use create-dashboard), or to fix non-telemetry problems like failing CI, lint errors, unit tests, or SQL query tuning. For on-call alerts where ONCALL.md exists, use kopai-triage if it is installed (the Kopai plugin's on-call skills).
91%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Debug production issues from telemetry (traces, logs, metrics). Kopai's analytical
power — error rate, throughput, latency, group-by, time-series bucketing — lives in
the query API, which the Kopai connector's query tool and the SDK both expose
and the CLI does not. So lead every investigation with queries, through the connector
or a short TypeScript script; the CLI stays as a quick-lookup fallback.
otel-instrumentation skill).query tool, or @kopai/sdk
(npm i @kopai/sdk, which ships the query API and the .kopairc reader).| You have | Use |
|---|---|
A Kopai query tool (Claude Tag channel, claude.ai connector, or Claude Code with the Kopai MCP server) | Connector mode below, for most investigations |
| A sandbox or project where you can run Node, and a question the connector answers awkwardly (many queries in a loop, joining results, arithmetic across rows) | Code mode |
| Neither | The CLI fallback at the end |
Both modes run the same queries: the workflow, patterns and gotchas below apply to each.
The Kopai connector exposes two read-only tools: query, which takes one
KopaiQuery object under query, and metrics_discover. Their names carry
a prefix that depends on how Kopai is connected (mcp__kopai__query,
mcp__claude_ai_Kopai__query, …); use whichever is present. The tool's
own schema is authoritative for field names, row caps and the aggregate
cap; on invalid_input, repair the query from the returned issues.
Every SDK builder call in this skill has a JSON equivalent. Measure ops
are upper-case (errorRate → ERROR_RATE, avg → AVG, p95 → P95),
.summary()/.timeSeries("5m") become output, and .timeRelative("1h")
becomes timeDimension. Step 1 of the workflow, ranking services by
error rate:
{
"signal": "traces",
"mode": "aggregate",
"measures": [
{ "op": "ERROR_RATE", "as": "error_rate" },
{ "op": "COUNT", "as": "spans" }
],
"dimensions": ["service.name"],
"timeDimension": { "type": "relative", "lookback": "1h" },
"output": { "type": "summary" },
"orderBy": [
{ "type": "measure", "alias": "error_rate", "direction": "desc" }
],
"limit": 20
}And step 3, the logs of one trace:
{
"signal": "logs",
"mode": "raw",
"filters": [{ "column": "TraceId", "op": "eq", "value": "<traceId>" }],
"timeDimension": { "type": "relative", "lookback": "1h" },
"limit": 100
}For an incident, prefer an absolute window (startTime/endTime, ISO
datetimes) over relative, so the evidence you cite doesn't slide.
Differences from the SDK in connector mode:
Duration comparisons (gt, lte, …) take a number of nanoseconds,
not a string like "1s": { "column": "Duration", "op": "gt", "value": 1000000000 }.raw mode and 500 in aggregate mode (buckets ×
groups for a time series). An over-cap result is refused, not
truncated: narrow the window, coarsen the granularity, or filter groups."ERROR" or "SERVER"
silently match nothing. Use the exact values ("Error", "Server",
"Client"), and treat an unexpected zero as a wrong value first.Credentials. In connector mode Claude never handles a Kopai token. In a Claude Tag channel, Kopai access is a credential the admin attached to the channel, applied outside the sandbox. Never ask anyone to paste a token into a thread.
Code mode in a Claude Tag sandbox. The sandbox has no .kopairc and
needs none: requests to api.kopai.app get the channel's Kopai credential
attached on the way out. Connect with
clientFromConfig({ url: "https://api.kopai.app/v2" }) and no token. If
requests come back unauthorized, the channel's Kopai connection doesn't
cover that address; use connector mode and tell the user which host was
refused.
Connect — clientFromConfig() reads .kopairc exactly like the CLI:
import { clientFromConfig } from "@kopai/sdk/node";
// Reads ./.kopairc then ~/.kopairc; defaults to http://localhost:8000.
// Override any field: clientFromConfig({ url, token, configPath, timeout }).
const client = clientFromConfig();Run — npx tsx rca.mts. Use the .mts extension: it is always an ES module, so
top-level await works without depending on the host project's package.json "type".
(If tsx is missing: node --experimental-strip-types rca.mts.)
Build a query with kq, run it with client.query(q). client.query(q) is the
single clean way to run any built query and is fully typed — rows autocomplete and
typos are compile errors. Aggregate queries return { data }; raw queries return
{ data, nextCursor }. Result rows are fully typed — iterate data directly and never
cast to any/any[]. Aggregate measure values are number (so row.error_rate * 100
just works — no String(...)/cast), grouped dimensions are present (row["service.name"]),
and .timeSeries() rows add bucket_start:
const { data } = await client.query(q); // typed rows — do NOT write `data as any[]`
for (const row of data) {
console.log(
row["service.name"],
`${(row.error_rate * 100).toFixed(1)}%`,
row.rps
);
}Output — stdout is your result. End with console.log(JSON.stringify(data, null, 2)),
and wrap .build() + the query call in try/catch:
import { kq, KopaiQueryBuildError } from "@kopai/sdk";
try {
const q = kq.traces
.aggregate() /* … */
.build();
const { data } = await client.query(q);
console.log(JSON.stringify(data, null, 2));
} catch (e) {
if (e instanceof KopaiQueryBuildError)
console.error(e.issues); // {path,message}[]
else throw e;
}.timeRelative("1h") or .timeAbsolute(startISO, endISO). Lookback/granularity match ^[1-9]\d*[smhdw]$ ("30s", "15m", "2h", "7d").StatusCode is exactly "Unset" | "Ok" | "Error" (title case). Successful spans are usually "Unset", not "Ok". This enum is type-checked: a wrong value like "ERROR" is a compile error, so the casing trap is caught for you. Prefer the errorRate measure (counts errors server-side) over filtering the raw value anyway.service.name (dotted), never ServiceName. ServiceName is not a queryable column — filtering on it silently matches nothing. (In result rows the value comes back keyed as you grouped it, e.g. row["service.name"].)p50–p999) are ClickHouse-only. On SQLite they fail at query time (KopaiError: Percentile measures … not yet supported on the sqlite backend). Lead latency with avg/max of Duration; treat percentiles as a ClickHouse upgrade in try/catch.rate/increase on kq.metrics("Sum") (needs .where(f => f.eq("IsMonotonic", true))), histogramQuantile/histogramAvg/rate("Count", …) on kq.metrics("Histogram"). Never avg/p95 of a counter's Value or a histogram's Sum/Count. A histogram percentile is an estimate inside one bucket, so write it as "p95 ≈ 430 ms" ("p95 > 430 ms" when _high is null, as the value is then only a lower bound). _low/_high show how precise that estimate is: use them to judge whether a change is real, and tell the user in plain words when a change is within that precision. See references/metric-filters.md.Duration filters accept duration strings with units s/m/h/d/w — f.gt("Duration", "1s"), f.lte("Duration", "2h") (also gte/lt). No sub-second units; for sub-second thresholds pass a nanosecond number. (Result rows still report Duration in nanoseconds — avg/max of Duration come back in ns: 1ms = 1e6, 1s = 1e9.)kq.metrics("Gauge")…, kq.metrics("Sum")… (type ∈ "Gauge" | "Sum" | "Histogram" | "ExponentialHistogram" | "Summary"). The builder arg auto-pins the MetricType, so no manual .where(f => f.eq("MetricType", …)). Value columns are typed per type: Gauge/Sum → "Value"; Histogram/ExponentialHistogram → "Count" | "Sum" | "Min" | "Max"; Summary → "Count" | "Sum".searchTraces/searchLogs/searchMetrics are async iterables (auto-paginate) — consume with for await (const row of client.searchLogs({ … })) { … }, do not await them as an array. For a single page use searchTracesPage/searchLogsPage/searchMetricsPage → { data, nextCursor }. Limits: kq .limit() caps at 10000; search* filters cap at 1000.Find the failing work — rank services by error rate and throughput. errorRate
handles StatusCode server-side, so you never guess the value:
const q = kq.traces
.aggregate()
.measure((m) => m.errorRate("error_rate"))
.measure((m) => m.throughput("rps"))
.measure((m) => m.count("spans"))
.dimension("service.name")
.timeRelative("1h")
.summary()
.orderByMeasure("error_rate", "desc")
.build();
const { data } = await client.query(q);For log-first triage, pull error-level logs by SeverityNumber >= 17 (catches
ERROR/FATAL regardless of text casing):
client.query(kq.logs.raw().where(f => f.gte("SeverityNumber", 17)).timeRelative("1h").limit(50).build()).
See workflow-find-errors.
Get full trace context — const spans = await client.getTrace(traceId). Inspect
Duration (bottlenecks), ParentSpanId (call chain), StatusMessage, SpanKind.
See workflow-get-context.
Correlate logs to the trace —
client.query(kq.logs.raw().where(f => f.eq("TraceId", traceId)).timeRelative("1h").build())
(or iterate client.searchLogs({ traceId })). Look for the earliest error and its
stack trace. See workflow-correlate-logs.
Quantify impact / pinpoint onset — the upgrade the CLI can't do. Bucket the metric over time to see when it started and how wide the blast radius is. Watch the retention floor: if the earliest bucket is already elevated, the true onset is at or before it — say so rather than calling the floor a "step change":
const q = kq.traces
.aggregate()
.measure((m) => m.errorRate("error_rate"))
.measure((m) => m.avg("Duration", "avg_ns"))
.measure((m) => m.max("Duration", "max_ns"))
.dimension("service.name")
.timeRelative("3h")
.timeSeries("5m")
.orderByMeasure("error_rate", "desc")
.build();
const { data } = await client.query(q);When the signal is a metric rather than a trace, choose the type up front and let it
auto-pin MetricType — e.g. bucket a Gauge over time:
const q = kq
.metrics("Gauge")
.aggregate()
.measure((m) => m.avg("Value", "avg_value"))
.where((f) => f.eq("MetricName", "process.cpu.utilization"))
.timeRelative("3h")
.timeSeries("5m")
.build();
const { data } = await client.query(q);See workflow-check-metrics.
Present findings — root cause with evidence (specific TraceIds, log entries,
metric deltas), blast radius, and a suggested fix. Offer to build an incident
dashboard (see the create-dashboard skill). See workflow-identify-cause.
// rca-triage.mts — run: npx tsx rca-triage.mts
import { kq, KopaiQueryBuildError } from "@kopai/sdk";
import { clientFromConfig } from "@kopai/sdk/node";
const client = clientFromConfig();
try {
const q = kq.traces
.aggregate()
.measure((m) => m.errorRate("error_rate"))
.measure((m) => m.throughput("rps"))
.measure((m) => m.avg("Duration", "avg_ns"))
.measure((m) => m.max("Duration", "max_ns"))
.measure((m) => m.count("spans"))
.dimension("service.name")
.timeRelative("1h")
.summary()
.orderByMeasure("error_rate", "desc")
.build();
const { data } = await client.query(q); // typed rows
console.log(JSON.stringify(data, null, 2));
} catch (e) {
if (e instanceof KopaiQueryBuildError) console.error(e.issues);
else throw e;
}workflow-find-errors - Find error traces and error-level logsworkflow-get-context - Get full trace contextworkflow-correlate-logs - Correlate logs with a traceworkflow-check-metrics - Quantify impact with aggregate/time-series queriesworkflow-identify-cause - Identify root cause & present findingspattern-http-errors - HTTP error debuggingpattern-slow-requests - Slow request analysispattern-distributed - Distributed failure tracingpattern-log-driven - Log-driven investigationRead rules/<rule-name>.md for details.
errorRate measure over filtering StatusCode (counted server-side).client.query(q) — it is fully typed (rows autocomplete, typos compile-error).Duration with strings (f.gt("Duration", "1s")); result rows still report Duration in nanoseconds. The earliest error in a trace chain is usually closest to root cause..timeSeries(granularity) to find when a regression started — and don't mistake the retention floor for the onset.kq.metrics("Gauge")… (auto-pins MetricType, types the value columns).avg/max of Duration; switch to p95/p99 on ClickHouse.The CLI is fine for a single lookup, but has no query/aggregation command. Note the
StatusCode value is the literal "Error".
npx @kopai/cli traces search --status-code Error --limit 20 --json
npx @kopai/cli traces get <traceId> --json
npx @kopai/cli logs search --trace-id <traceId> --severity-min 17 --json
npx @kopai/cli metrics discover --json