Advanced Topics

Advanced LangSmith features including OpenTelemetry integration, data anonymization, and annotation queues for human-in-the-loop workflows.

Overview

LangSmith provides three advanced features for specialized use cases:

• OpenTelemetry Integration - Standards-based instrumentation and distributed tracing
• Data Anonymization - Remove sensitive information from traces
• Annotation Queues - Human-in-the-loop feedback workflows

OpenTelemetry Integration

What It Does

Enables standards-based distributed tracing using OpenTelemetry instrumentation libraries. Export traces to LangSmith while maintaining compatibility with the OpenTelemetry ecosystem.

When to Use

• You already use OpenTelemetry in your stack
• Need standards-compliant distributed tracing across services
• Want to instrument both LLM and non-LLM operations
• Require integration with existing observability tools

Key Capabilities

• initializeOTEL(config) - One-line setup with automatic instrumentation
• LangSmithOTLPTraceExporter - OTLP exporter configured for LangSmith
• Semantic convention constants (GEN_AI_*) - Standard attribute names for LLM operations
• Support for multiple instrumentation libraries (HTTP, Express, MongoDB, Redis, etc.)
• W3C trace context propagation for distributed systems

→ Read OpenTelemetry Documentation

Data Anonymization

What It Does

Removes sensitive information from traces before sending to LangSmith. Supports pattern-based rules, custom functions, and advanced processors for comprehensive PII protection.

When to Use

• Working with user data containing PII
• Tracing operations that include credentials or API keys
• Compliance with privacy regulations (GDPR, CCPA, HIPAA)
• Protecting sensitive business information

Key Capabilities

• createAnonymizer(replacer, options) - Create anonymizers from rules, functions, or processors
• Three approaches: Rule-based (regex patterns), Function-based (custom logic), Processor-based (structured)
• Comprehensive pattern library (emails, SSNs, phone numbers, credit cards, API keys, etc.)
• Path-based selective anonymization with exclusions
• Integration with Client, traceable, RunTree, and LangChain

→ Read Anonymization Documentation

Annotation Queues

What It Does

Provides structured queues for human review of LLM outputs. Enables quality assurance workflows, training data collection, and comparative model evaluation with human feedback.

When to Use

• Human-in-the-loop evaluation workflows
• Quality assurance for production outputs
• Building human-annotated training datasets
• Comparative model evaluation by human reviewers
• Edge case analysis and categorization

Key Capabilities

• createAnnotationQueue(options) - Create queues with rubric instructions
• addRunsToAnnotationQueue(params) - Add runs for review
• getRunFromAnnotationQueue(queueId, index) - Retrieve runs for annotation
• getSizeFromAnnotationQueue(queueId) - Monitor queue size
• Integration with feedback system for storing annotations
• Multiple sampling strategies (random, stratified, uncertainty-based, active learning)

→ Read Annotation Queues Documentation

Choosing the Right Feature

Use OpenTelemetry if you need:

• ✓ Standards-based observability
• ✓ Distributed tracing across multiple services
• ✓ Integration with existing OTEL tooling
• ✓ Both LLM and non-LLM instrumentation

Use Anonymization if you need:

• ✓ PII protection in traces
• ✓ Compliance with privacy regulations
• ✓ Credential/secret redaction
• ✓ Selective data hiding based on paths

Use Annotation Queues if you need:

• ✓ Human review of model outputs
• ✓ Quality assurance workflows
• ✓ Human-annotated training data
• ✓ Comparative human evaluation

Complete Integration Example

Combining all three advanced features in a production application:

import { initializeOTEL } from "langsmith/experimental/otel/setup";
import { createAnonymizer } from "langsmith/anonymizer";
import { Client } from "langsmith";
import { traceable } from "langsmith/traceable";

// 1. Initialize OpenTelemetry for distributed tracing
initializeOTEL({
  projectName: "production-app",
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

// 2. Create anonymizer to protect sensitive data
const anonymizer = createAnonymizer([
  { pattern: /\b[\w\.-]+@[\w\.-]+\.\w+\b/g, replace: "[EMAIL]" },
  { pattern: /\bsk-[a-zA-Z0-9]+\b/g, replace: "[API_KEY]" },
  { pattern: /\b\d{3}-\d{2}-\d{4}\b/g, replace: "[SSN]" },
]);

// 3. Create client with anonymization enabled
const client = new Client({
  apiKey: process.env.LANGSMITH_API_KEY,
  anonymizer,
});

// 4. Create annotation queue for human quality review
const qaQueue = await client.createAnnotationQueue({
  name: "Production QA",
  description: "Human review of low-confidence outputs",
  rubricInstructions: "Rate accuracy, helpfulness, and safety on 1-5 scale",
});

// 5. Create traced function with anonymization
const processUserQuery = traceable(
  async (input: { email: string; query: string }) => {
    // Process query - traced by both OTEL and LangSmith
    const response = await handleQuery(input.query);

    return {
      response,
      userEmail: input.email,  // Will be anonymized in traces
      confidence: response.confidence,
    };
  },
  {
    name: "process_user_query",
    run_type: "chain",
    processInputs: anonymizer,   // Anonymize inputs
    processOutputs: anonymizer,  // Anonymize outputs
  }
);

// 6. Production workflow with all features
async function productionWorkflow(email: string, query: string) {
  // Process query (OpenTelemetry + LangSmith tracing, anonymized)
  const result = await processUserQuery({ email, query });

  // If confidence is low, add to human review queue
  if (result.confidence < 0.7) {
    await client.addRunsToAnnotationQueue({
      queueId: qaQueue.id,
      runIds: [result.runId],  // From trace context
    });

    console.log(`Low confidence (${result.confidence}) - added to human review queue`);
  }

  return result;
}

// Example usage
await productionWorkflow(
  "user@example.com",
  "Show me data for SSN 123-45-6789"
);
// Traces will show: "[EMAIL]" and "SSN [SSN]" (anonymized)
// Low-confidence outputs automatically queued for human review
// All traces captured by OpenTelemetry + LangSmith

Related Documentation

• OpenTelemetry Integration - Complete OTEL documentation
• Data Anonymization - Complete anonymization guide
• Annotation Queues - Complete queue workflows
• Getting Started - Installation and first steps
• Tracing - Core tracing functionality
• Client API - Complete API reference