test-observability

Integrate Playwright tests with OpenTelemetry, Grafana, Prometheus, Loki, and Tempo. Use when debugging test failures across distributed systems, measuring test performance, creating test dashboards, or correlating tests with backend traces.

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Test Observability

Name: test-observability
Rating: 67.2 (1 reviews)
Author: fernandezbaptiste

Overview

Connect your Playwright tests to your observability stack (Grafana, Prometheus, Loki, Tempo) for:

Trace Correlation: See full trace from browser click → backend → database
Test Metrics: Dashboard with pass rates, durations, flakiness
Log Aggregation: Test logs alongside application logs
Failure Analysis: Quickly identify root cause across distributed systems

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Playwright     │────▶│  OTEL Collector  │────▶│  Tempo (Traces) │
│  Tests          │     │                  │────▶│  Loki (Logs)    │
│                 │     │                  │────▶│  Prometheus     │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                          │
                                                          ▼
                                                  ┌─────────────────┐
                                                  │    Grafana      │
                                                  │  Test Dashboard │
                                                  └─────────────────┘

Quick Start: OTEL Reporter

1. Install Dependencies

npm install playwright-opentelemetry-reporter @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

2. Configure Reporter

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-opentelemetry-reporter', {
      serviceName: 'playwright-tests',
      endpoint: process.env.OTEL_ENDPOINT || 'http://localhost:4318/v1/traces',
    }],
  ],
});

3. Run Tests

# With local OTEL collector
OTEL_ENDPOINT=http://localhost:4318/v1/traces npx playwright test

# With Railway OTEL collector
OTEL_ENDPOINT=http://otel-collector.railway.internal:4318/v1/traces npx playwright test

4. View in Grafana

Open Grafana → Explore → Tempo
Search for service.name = "playwright-tests"
See test spans with duration, status, steps

Tracetest Integration (Advanced)

Tracetest enables trace-based testing - assert on any span in the distributed trace.

Installation

npm install @tracetest/playwright

Configuration

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html'],
    ['@tracetest/playwright', {
      serverUrl: process.env.TRACETEST_URL || 'http://localhost:11633',
      apiKey: process.env.TRACETEST_API_KEY,
    }],
  ],
});

Trace-Based Assertions

// tests/checkout.spec.ts
import { test, expect } from '@playwright/test';
import { Tracetest } from '@tracetest/playwright';

test('checkout creates order', async ({ page }) => {
  const tracetest = new Tracetest();

  // Start trace capture
  await tracetest.capture();

  // Perform user action
  await page.goto('/checkout');
  await page.click('#place-order');
  await expect(page.locator('.order-confirmation')).toBeVisible();

  // Assert on backend trace!
  await tracetest.assertOnTrace({
    assertions: [
      // API response time
      {
        selector: 'span[name="POST /api/orders"]',
        assertion: 'attr:http.status_code = 201',
      },
      // Database query time
      {
        selector: 'span[name="INSERT orders"]',
        assertion: 'attr:db.duration < 100ms',
      },
      // Payment service
      {
        selector: 'span[name="payment.process"]',
        assertion: 'attr:payment.status = "success"',
      },
    ],
  });
});

Benefits

80% faster debugging: See exactly where failures occur
Full visibility: Browser → API → Database → Services
Backend assertions: Test database queries, service calls
Performance validation: Assert on span durations

Grafana Dashboard

Import Dashboard

Open Grafana → Dashboards → Import
Upload dashboards/test-results-dashboard.json
Select Prometheus and Tempo data sources

Key Panels

Panel	Query	Purpose
Pass Rate	`sum(playwright_test_passed) / sum(playwright_test_total)`	Overall health
Test Duration (p95)	`histogram_quantile(0.95, playwright_test_duration_bucket)`	Performance
Failed Tests	`playwright_test_failed{status="failed"}`	Quick triage
Flaky Tests	`playwright_test_retries > 1`	Identify flakiness
Slowest Tests	`topk(10, playwright_test_duration)`	Optimization targets

Custom Metrics

// Export custom metrics from tests
import { test } from '@playwright/test';
import { metrics } from '@opentelemetry/api';

const testMeter = metrics.getMeter('playwright-tests');
const testDuration = testMeter.createHistogram('test.duration');
const testCounter = testMeter.createCounter('test.count');

test('custom metrics', async ({ page }) => {
  const start = Date.now();

  await page.goto('/');
  await page.click('.action');

  // Record custom metric
  testDuration.record(Date.now() - start, {
    test: 'homepage-action',
    browser: 'chromium',
  });

  testCounter.add(1, { status: 'passed' });
});

Railway Integration

Connect to your existing Railway observability stack.

Environment Variables

# Set in Railway service or .env
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.railway.internal:4318
OTEL_SERVICE_NAME=playwright-tests

Playwright Config

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['playwright-opentelemetry-reporter', {
      serviceName: process.env.OTEL_SERVICE_NAME || 'playwright-tests',
      endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
      headers: {
        // Add auth if needed
        'Authorization': `Bearer ${process.env.OTEL_AUTH_TOKEN}`,
      },
    }],
  ],
});

Verify Connection

# Test OTEL collector is receiving data
curl -X POST http://otel-collector.railway.internal:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'
# Should return 200 OK

Log Correlation

Send test logs to Loki alongside traces.

Winston + OTEL

// tests/fixtures/logger.ts
import winston from 'winston';
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';

export const logger = winston.createLogger({
  transports: [
    new winston.transports.Console(),
    // Send to OTEL collector → Loki
    new OTLPLogTransport({
      endpoint: process.env.OTEL_ENDPOINT,
    }),
  ],
});

// In tests
import { logger } from './fixtures/logger';

test('with logging', async ({ page }) => {
  logger.info('Starting test', { test: 'checkout' });

  await page.goto('/checkout');
  logger.debug('Page loaded', { url: page.url() });

  await page.click('#submit');
  logger.info('Order submitted');
});

View Logs in Grafana

1. Grafana → Explore → Loki
2. Query: {service="playwright-tests"}
3. Filter: |= "error" or |json | level="error"

Alerting

Set up alerts for test failures.

Prometheus Alert Rules

# alerts/test-alerts.yml
groups:
  - name: playwright-tests
    rules:
      - alert: TestFailureRate
        expr: |
          (sum(rate(playwright_test_failed[5m])) / sum(rate(playwright_test_total[5m]))) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High test failure rate (>10%)"

      - alert: TestDurationSpike
        expr: |
          histogram_quantile(0.95, playwright_test_duration_bucket) > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Test duration p95 > 30 seconds"

Grafana Alert

Edit dashboard panel
Alert tab → Create alert
Condition: avg() of A is above 0.1
Notifications: Slack, email, PagerDuty

Workflow: Debugging with Traces

1. Test Fails in CI

❌ tests/checkout.spec.ts:15 - Order creation failed
   Expected: Order confirmation visible
   Actual: Error page shown

2. Find Trace in Grafana

Grafana → Explore → Tempo
Query: service.name="playwright-tests" AND name="checkout.spec.ts"

3. Analyze Trace

Browser click "Place Order" (50ms)
  └─ POST /api/orders (200ms)
       └─ Validate cart (10ms)
       └─ Process payment ❌ (timeout after 30s)
           └─ Payment gateway unreachable
       └─ Create order (skipped)

4. Root Cause

Found: Payment gateway timeout caused order failure.

Without traces: Would have debugged browser-side for hours.

Best Practices

1. Add Trace Context to Tests

import { context, trace } from '@opentelemetry/api';

test('with trace context', async ({ page }) => {
  const tracer = trace.getTracer('playwright');

  await tracer.startActiveSpan('checkout-flow', async (span) => {
    span.setAttribute('test.name', 'checkout');
    span.setAttribute('test.browser', 'chromium');

    await page.goto('/checkout');
    await page.click('#submit');

    span.setStatus({ code: 1 }); // OK
    span.end();
  });
});

2. Propagate Context to Backend

// Ensure trace ID passes from browser to backend
test('trace propagation', async ({ page }) => {
  // Get current trace ID
  const traceId = trace.getActiveSpan()?.spanContext().traceId;

  // Add to request headers
  await page.route('**/*', route => {
    route.continue({
      headers: {
        ...route.request().headers(),
        'traceparent': `00-${traceId}-${spanId}-01`,
      },
    });
  });
});

3. Tag Tests for Filtering

test('with tags', async ({ page }) => {
  const span = trace.getActiveSpan();
  span?.setAttribute('test.suite', 'e2e');
  span?.setAttribute('test.priority', 'critical');
  span?.setAttribute('test.feature', 'checkout');

  // Now filter in Grafana: test.feature="checkout"
});

References

references/tracetest-integration.md - Full Tracetest setup
references/otel-reporter-setup.md - OTEL reporter configuration
references/grafana-dashboards.md - Dashboard creation guide
dashboards/test-results-dashboard.json - Ready-to-import dashboard

Test observability transforms debugging from guesswork to precision - see the full picture from click to database.

Repository: fernandezbaptiste/Skrillz
Path: skills/test-observability/SKILL.md
Commit: 93ed392

Last updated: 1 day ago
First committed: 8 months ago

Also appears in

fernandezbaptiste/Skrillz

In sync

since Sep 12, 2026

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.