opentelemetry

OpenTelemetry with Grafana stack. Covers OTel SDK instrumentation for Go/Java/Python/Node.js/.NET, OTLP protocol and endpoint configuration, sending telemetry to Grafana Cloud via OTLP endpoint, Grafana Alloy as OTel collector, sampling strategies, Kubernetes OTel Operator, and migration from other observability tools. Use when instrumenting apps with OTel, configuring OTLP endpoints, setting up collectors, or migrating to OpenTelemetry.

Quality

82%

Does it follow best practices?

Impact

—

No eval scenarios have been run

Securityby

Advisory

Suggest reviewing before use

OpenTelemetry with Grafana

Name: opentelemetry
Rating: 68.8 (1 reviews)
Author: grafana

Overview

OpenTelemetry (OTel) is a vendor-neutral framework for collecting observability data (metrics, logs, traces, profiles). Grafana Labs integrates it as a core strategy, offering a full stack to collect, ingest, store, analyze, and visualize telemetry data.

Four-Step Implementation Model

Instrument - Add telemetry using Grafana SDKs, Beyla (eBPF), or upstream OTel SDKs
Pipeline - Build processing infrastructure with Grafana Alloy or OTel Collector
Ingest - Route data to Grafana Cloud OTLP endpoint or self-managed backends
Analyze - Dashboards, alerts, Application Observability, Drilldown apps

Grafana Backends

Signal	Backend
Metrics	Grafana Mimir
Logs	Grafana Loki
Traces	Grafana Tempo
Profiles	Grafana Pyroscope

OTLP Endpoint and Authentication

Grafana Cloud OTLP Endpoint

Grafana Cloud exposes a managed OTLP gateway endpoint:

https://otlp-gateway-<region>.grafana.net/otlp

Example regions: prod-us-east-0, prod-eu-west-0, prod-ap-southeast-0

Full example:

https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Authentication - Basic Auth

Grafana Cloud OTLP uses HTTP Basic Auth:

Username: Grafana Cloud Instance ID (numeric, e.g. 123456)
Password: Grafana Cloud API token (with MetricsPublisher, LogsPublisher, TracesPublisher roles)

Via environment variable (recommended)

# Base64-encode "instanceID:apiToken"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n '123456:glc_eyJ...' | base64)"

Via Alloy environment variables

export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Direct Send (no collector) - Environment Variables

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64(instanceID:apiToken)>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=production"

Instrumentation by Language

Go

Requirements: Go 1.22+

Install packages:

go get "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" \
  "go.opentelemetry.io/contrib/instrumentation/runtime" \
  "go.opentelemetry.io/otel" \
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \
  "go.opentelemetry.io/otel/sdk" \
  "go.opentelemetry.io/otel/sdk/metric"

Run with environment variables:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
go run .

See references/instrumentation.md for full Go code example.

Java (Grafana Distribution - JVM Agent)

Requirements: JDK 8+

Download: grafana-opentelemetry-java.jar from https://github.com/grafana/grafana-opentelemetry-java/releases

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=shoppingcart,service.namespace=ecommerce,deployment.environment=production" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
java -javaagent:/path/to/grafana-opentelemetry-java.jar -jar myapp.jar

Optional: Data saver mode (reduces metric cardinality):

export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Debug:

export OTEL_JAVAAGENT_DEBUG=true
# Enable console output alongside OTLP
export OTEL_TRACES_EXPORTER=otlp,console
export OTEL_METRICS_EXPORTER=otlp,console
export OTEL_LOGS_EXPORTER=otlp,console

Node.js

Install:

npm install --save @opentelemetry/api
npm install --save @opentelemetry/auto-instrumentations-node

Run:

OTEL_TRACES_EXPORTER="otlp" \
OTEL_METRICS_EXPORTER="otlp" \
OTEL_LOGS_EXPORTER="otlp" \
OTEL_NODE_RESOURCE_DETECTORS="env,host,os" \
OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
node app.js

Warning: Bundlers like @vercel/ncc can break auto-instrumentation hooks.

See references/instrumentation.md for manual SDK setup example.

Python

Install:

pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
opentelemetry-instrument python app.py

Multi-process servers (Gunicorn, uWSGI): implement post-fork hooks to reinitialize OTel providers per worker.

.NET (Grafana Distribution)

Install NuGet:

dotnet add package Grafana.OpenTelemetry

ASP.NET Core setup:

using Grafana.OpenTelemetry;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry()
    .WithTracing(configure => configure.UseGrafana())
    .WithMetrics(configure => configure.UseGrafana());
builder.Logging.AddOpenTelemetry(options => options.UseGrafana());

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
dotnet run

Requirements: .NET 6+ or .NET Framework 4.6.2+

See references/instrumentation.md for full .NET examples.

Beyla (eBPF - Language Agnostic)

Grafana Beyla instruments at the network layer - no code changes required, works with any language.

# Docker
docker run --rm -it \
  --privileged \
  -e BEYLA_SERVICE_NAME=myapp \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
  -v /sys/kernel/security:/sys/kernel/security \
  grafana/beyla

Verify with: curl http://localhost:9090/metrics

Full docs: https://grafana.com/docs/beyla/

Grafana Alloy Collector

Grafana Alloy is the recommended OTel Collector distribution. It combines upstream OTel Collector components with Prometheus exporters for infrastructure + application observability correlation.

Why Use a Collector?

Cost control: Aggregate, sample, and drop data before sending
Reliability: Buffer and retry on connection failures
Enrichment: Add resource attributes, transform, redact, and route data

Alloy Ports

Port	Protocol	Purpose
4317	gRPC	OTLP gRPC receiver
4318	HTTP	OTLP HTTP/protobuf receiver

Application -> Alloy -> Grafana Cloud

Application env vars (point to local Alloy):

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Alloy config env vars (Alloy -> Grafana Cloud):

export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...

See references/collector-config.md for full Alloy configuration.

Kubernetes Setup

Option 1: Grafana Kubernetes Monitoring Helm Chart (recommended)

The Grafana Kubernetes Monitoring Helm chart deploys Alloy with OTLP receivers pre-configured.

Enable "OTLP Receivers" in the Cluster Configuration tab
Get gRPC/HTTP endpoints from "Configure Application Instrumentation" section
Point apps to the in-cluster Alloy endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT=<GRPC_ENDPOINT_FROM_HELM>
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Option 2: OpenTelemetry Operator

Install via official docs, then use Instrumentation CR for auto-injection:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otelcol:4317
  propagators:
    - tracecontext
    - baggage
  java:
    # Use Grafana distribution image
    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1
  nodejs: {}
  python: {}

Inject into pods with annotation:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    # or: inject-nodejs, inject-python, inject-dotnet

See references/collector-config.md for Kubernetes Alloy Helm values and OTel Collector YAML.

Sampling Strategies

Head-Based Sampling

Decision made at trace start - low overhead, may miss rare errors.

Environment variable (probability sampler):

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # 10% of traces

Alloy head sampling config:

otelcol.processor.probabilistic_sampler "default" {
  sampling_percentage = 10
  output {
    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

Tail-Based Sampling

Decision made after all spans collected - can sample based on outcome (e.g. keep all errors).

Alloy tail sampling config:

otelcol.processor.tail_sampling "default" {
  decision_wait            = "10s"
  num_traces               = 100000
  expected_new_traces_per_sec = 10

  policy {
    name = "keep-errors"
    type = "status_code"
    status_code {
      status_codes = ["ERROR"]
    }
  }

  policy {
    name = "probabilistic-sample"
    type = "probabilistic"
    probabilistic {
      sampling_percentage = 10
    }
  }

  output {
    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

Key Environment Variables Reference

Variable	Description	Example
`OTEL_EXPORTER_OTLP_ENDPOINT`	OTLP receiver URL	`https://otlp-gateway-prod-us-east-0.grafana.net/otlp`
`OTEL_EXPORTER_OTLP_PROTOCOL`	Transport protocol	`grpc` or `http/protobuf`
`OTEL_EXPORTER_OTLP_HEADERS`	Auth headers	`Authorization=Basic <base64>`
`OTEL_RESOURCE_ATTRIBUTES`	Service metadata	`service.name=myapp,service.namespace=team,deployment.environment=prod`
`OTEL_TRACES_EXPORTER`	Trace exporter type	`otlp`
`OTEL_METRICS_EXPORTER`	Metrics exporter type	`otlp`
`OTEL_LOGS_EXPORTER`	Logs exporter type	`otlp`
`OTEL_SERVICE_NAME`	Service name (shorthand)	`myapp`
`OTEL_TRACES_SAMPLER`	Sampler type	`parentbased_traceidratio`
`OTEL_TRACES_SAMPLER_ARG`	Sampler argument	`0.1` (10%)

Key Resource Attributes

Attribute	Purpose	Example
`service.name`	Service identifier	`shoppingcart`
`service.namespace`	Groups related services	`ecommerce`
`deployment.environment`	Environment tier	`production`, `staging`
`service.version`	App version	`1.2.3`

Useful Links

Grafana OTel docs: https://grafana.com/docs/opentelemetry/
Grafana Cloud OTLP: https://grafana.com/docs/grafana-cloud/send-data/otlp/
Grafana Java Agent: https://github.com/grafana/grafana-opentelemetry-java
Grafana .NET SDK: https://github.com/grafana/grafana-opentelemetry-dotnet
Grafana Alloy: https://grafana.com/docs/alloy/
Grafana Beyla: https://grafana.com/docs/beyla/
OTel Collector: https://opentelemetry.io/docs/collector/
OTel Operator: https://opentelemetry.io/docs/kubernetes/operator/

Repository: grafana/skills
Commit: 53bb349

Last updated: 1 day ago
Created: 1 day ago

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.