langfuse-tracing

Overview

Langfuse is a production LLM observability platform (per lf-gh).

Step 1 - Install

Per lf-gh:

pip install langfuse

For TypeScript:

npm install @langfuse/tracing

Set up project credentials per Langfuse self-hosted or cloud project (LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY + LANGFUSE_HOST).

Step 2 - Instrument with @observe

Per langfuse.com/docs/sdk/python/decorators:

Python:

from langfuse import observe

@observe(name="llm-call", as_type="generation")  # auto-captures inputs, outputs, timings, errors
async def my_async_llm_call(prompt_text):
    return "LLM response"

TypeScript - create an observation, do work inside it, then end it:

import { startActiveObservation } from "@langfuse/tracing";

const { observation, end } = startActiveObservation({
  name: "llm-call",
  type: "generation",
  input: { prompt: promptText },
});

try {
  const result = await callMyLLM(promptText);
  observation.update({ output: result });
  return result;
} finally {
  end();
}

Validation: After your first instrumented call, open the Langfuse UI → Traces. You should see a new trace with nested observations, inputs, outputs, and timings. If no trace appears within ~30 s:

• Confirm LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST are set in the process environment.
• Call await langfuse.flushAsync() (TS) / langfuse.flush() (Python) before process exit to force-drain the queue.
• Check langfuse.debug() output for connection errors.

Step 3 - Update current observation with metadata

Per lf-py-deco:

from langfuse import get_client

langfuse = get_client()

with langfuse.start_as_current_observation(as_type="span", name="data-processing"):
    langfuse.update_current_span(metadata={"step1_complete": True})

Common metadata fields used in production:

• model - the model name (e.g., claude-haiku-4-5)
• model_parameters - temperature / top_p / max_tokens
• usage - input/output tokens, cost
• tags - environment (prod / staging), feature flag, customer ID
• level - DEBUG / DEFAULT / WARNING / ERROR

Step 4 - Score traces

Per langfuse.com/docs/scores:

langfuse.score(
    trace_id="...",
    name="answer_relevance",
    value=0.87,           # numeric (0-1); also supports categorical (string) and boolean
    comment="Judged by GPT-4 rubric"
)

Scores can come from:

• Manual review (human raters via the Langfuse UI)
• Automated eval (run a metric in batch, score by trace_id)
• User feedback (thumbs-up / thumbs-down from the app)

Validation: After scoring, open the Langfuse UI → Traces → select the trace. The score should appear in the Scores panel. If missing, confirm the trace_id matches an existing trace and that the score call did not raise an exception.

Step 5 - Datasets for offline eval

Datasets are collections of (input, expected_output) items built from production traces, CSV / JSONL imports, or the UI. Run one against your app to diff vs a baseline:

items = langfuse.get_dataset_items(dataset_id="...")
for item in items:
    actual = my_llm_app(item.input)
    item.run(actual)  # links the run back to the dataset for diff vs baseline

See references/datasets.md for building datasets from production traces, the lf-ds API signature, and the run-validation checkpoint.

Step 6 - Prompt management

Fetch the current production prompt at runtime and pin its version so prompt drift is traceable:

from langfuse import get_client

langfuse = get_client()

# Fetch the production-labelled prompt; falls back to cache if offline
prompt = langfuse.get_prompt("my-prompt-name", label="production")
compiled = prompt.compile(user_input=user_query)

# Pass compiled text to your LLM call
response = my_llm_call(compiled)

Iterate prompt text in the Langfuse UI and roll out new versions per environment (production / staging labels) without code deploys. Version history and A/B comparison are available in the UI.

Validation: After an instrumented call using a managed prompt, open the Langfuse UI → Traces → select the trace. The prompt name and version should appear in the trace metadata, confirming prompt version is pinned and attributable.

Step 7 - CI integration

Langfuse is observability-side, not pre-deploy CI-side. Post-deploy CI patterns query the Langfuse API for recent traces and assert on aggregate metrics (a score-query gate, eval-on-trace, cost-regression, score-based alerting). Runnable gate script and the full pattern list: references/ci-integration.md.

Anti-patterns

Version notes

Per lf-gh, the SDK was rewritten in v4 and released in March 2026; this skill targets the v4 API. v3 patterns are no longer supported - pin the SDK version in requirements, and for v3 codebases see the upstream migration guide.

Limitations

• Langfuse cloud is hosted; for data-residency-strict teams, self-host (well-supported but operational overhead).
• SDK and score-API signatures may drift between versions; always consult langfuse.com/docs and lf-scores when authoring wiring against any rapidly-evolving endpoint.
• Integration with eval frameworks evolves rapidly; consult langfuse.com/docs for current Promptfoo / DeepEval / Ragas integration patterns.

References

• lf-gh - Python SDK repo, install, version note
• lf-py-deco - @observe decorator + observation patterns
• lf-scores - score API
• lf-ds - datasets
• lf-docs - full documentation
• promptfoo-evaluation, deepeval-evaluation, ragas-evaluation, giskard-llm - pre-deploy eval sister tools