0
# Tracing and Observability
1

2
MLflow's tracing capabilities provide comprehensive observability for LLM applications, GenAI workflows, and distributed ML systems. The system captures execution traces, spans, and performance metrics with automatic instrumentation and custom span creation for debugging and monitoring complex AI applications.
3

4
## Capabilities
5

6
### Tracing Configuration and Setup
7

8
Functions for configuring tracing behavior, setting up providers, and managing tracing destinations.
9

10
```python { .api }
11
def configure(disable_propagation=None, sampling_rate=None):
12
    """
13
    Configure global tracing settings.
14
    
15
    Parameters:
16
    - disable_propagation: bool, optional - Disable trace context propagation
17
    - sampling_rate: float, optional - Trace sampling rate (0.0 to 1.0)
18
    """
19

20
def enable(experiment_id=None):
21
    """
22
    Enable MLflow tracing for current session.
23
    
24
    Parameters:
25
    - experiment_id: str, optional - Experiment ID for traces
26
    """
27

28
def disable():
29
    """
30
    Disable MLflow tracing for current session.
31
    """
32

33
def reset():
34
    """
35
    Reset tracing configuration to defaults.
36
    """
37

38
def set_destination(destination):
39
    """
40
    Set tracing destination for spans and traces.
41
    
42
    Parameters:
43
    - destination: str or TraceDestination - Destination for trace data
44
    """
45
```
46

47
### Trace and Span Management
48

49
Core functions for creating, managing, and ending traces and spans with hierarchical relationships.
50

51
```python { .api }
52
def start_span(name, span_type=None, inputs=None, attributes=None):
53
    """
54
    Start a new span in current trace context.
55
    
56
    Parameters:
57
    - name: str - Span name for identification
58
    - span_type: str, optional - Type of span (LLM, CHAIN, AGENT, etc.)
59
    - inputs: Any, optional - Input data for the span
60
    - attributes: dict, optional - Additional span attributes
61
    
62
    Returns:
63
    Span object representing the active span
64
    """
65

66
@contextmanager
67
def trace(name, span_type=None, inputs=None, attributes=None):
68
    """
69
    Context manager for automatic span lifecycle management.
70
    
71
    Parameters:
72
    - name: str - Span name for identification
73
    - span_type: str, optional - Type of span
74
    - inputs: Any, optional - Input data for the span
75
    - attributes: dict, optional - Additional span attributes
76
    
77
    Yields:
78
    Span object for the duration of the context
79
    """
80

81
def get_current_active_span():
82
    """
83
    Get currently active span in execution context.
84
    
85
    Returns:
86
    Span object or None if no active span
87
    """
88

89
def end_span():
90
    """
91
    End the currently active span.
92
    """
93

94
def set_span_attribute(key, value):
95
    """
96
    Set attribute on current active span.
97
    
98
    Parameters:
99
    - key: str - Attribute key
100
    - value: Any - Attribute value (will be serialized)
101
    """
102

103
def set_span_attributes(attributes):
104
    """
105
    Set multiple attributes on current active span.
106
    
107
    Parameters:
108
    - attributes: dict - Dictionary of attributes to set
109
    """
110

111
def set_span_chat_tools(tools):
112
    """
113
    Set chat tools metadata for current span.
114
    
115
    Parameters:
116
    - tools: list - List of tool specifications for chat models
117
    """
118
```
119

120
### Display and Visualization
121

122
Functions for controlling trace display in notebook environments and visualization settings.
123

124
```python { .api }
125
def enable_notebook_display():
126
    """
127
    Enable automatic trace display in Jupyter notebooks.
128
    """
129

130
def disable_notebook_display():
131
    """
132
    Disable automatic trace display in Jupyter notebooks.
133
    """
134
```
135

136
### Span Types and Status
137

138
Enumeration classes for span types and status codes used in tracing.
139

140
```python { .api }
141
class SpanType:
142
    """Enumeration of span types for categorizing operations."""
143
    UNKNOWN = "UNKNOWN"
144
    LLM = "LLM"
145
    CHAT_MODEL = "CHAT_MODEL"
146
    CHAIN = "CHAIN"
147
    AGENT = "AGENT"
148
    TOOL = "TOOL"
149
    RETRIEVER = "RETRIEVER"
150
    EMBEDDING = "EMBEDDING"
151
    RERANKER = "RERANKER"
152
    PARSER = "PARSER"
153
    
154
class SpanStatus:
155
    """Enumeration of span status codes."""
156
    OK = "OK"
157
    ERROR = "ERROR"
158
    CANCELLED = "CANCELLED"
159
    TIMEOUT = "TIMEOUT"
160

161
class SpanStatusCode:
162
    """Standard status codes for spans."""
163
    UNSET = "UNSET"
164
    OK = "OK"  
165
    ERROR = "ERROR"
166
```
167

168
### Automatic Instrumentation
169

170
Decorators and utilities for automatic instrumentation of functions and methods.
171

172
```python { .api }
173
@mlflow_trace
174
def instrumented_function(param1, param2):
175
    """
176
    Decorator for automatic function instrumentation.
177
    
178
    The decorator automatically creates spans for function execution,
179
    capturing inputs, outputs, and execution metadata.
180
    """
181

182
def instrument_langchain():
183
    """
184
    Enable automatic instrumentation for LangChain components.
185
    """
186

187
def instrument_openai():
188
    """
189
    Enable automatic instrumentation for OpenAI API calls.
190
    """
191

192
def instrument_anthropic():
193
    """
194
    Enable automatic instrumentation for Anthropic API calls.
195
    """
196

197
def instrument_transformers():
198
    """
199
    Enable automatic instrumentation for Transformers library.
200
    """
201
```
202

203
### Trace Retrieval and Analysis
204

205
Functions for searching, retrieving, and analyzing traces after execution.
206

207
```python { .api }
208
def get_trace(trace_id):
209
    """
210
    Retrieve trace by ID.
211
    
212
    Parameters:
213
    - trace_id: str - Unique trace identifier
214
    
215
    Returns:
216
    Trace object with all spans and metadata
217
    """
218

219
def search_traces(experiment_ids=None, filter_string=None, max_results=None):
220
    """
221
    Search traces matching criteria.
222
    
223
    Parameters:
224
    - experiment_ids: list, optional - List of experiment IDs to search
225
    - filter_string: str, optional - Filter expression for traces
226
    - max_results: int, optional - Maximum number of traces to return
227
    
228
    Returns:
229
    List of Trace objects matching criteria
230
    """
231

232
def get_last_active_trace():
233
    """
234
    Get the most recently active trace.
235
    
236
    Returns:
237
    Trace object or None if no recent traces
238
    """
239
```
240

241
### Span Events and Logging
242

243
Functions for adding events, logs, and structured data to spans during execution.
244

245
```python { .api }
246
def log_span_event(name, attributes=None, timestamp=None):
247
    """
248
    Log event to current active span.
249
    
250
    Parameters:
251
    - name: str - Event name
252
    - attributes: dict, optional - Event attributes
253
    - timestamp: int, optional - Event timestamp (defaults to current time)
254
    """
255

256
def set_span_output(output):
257
    """
258
    Set output data for current active span.
259
    
260
    Parameters:
261
    - output: Any - Output data (will be serialized)
262
    """
263

264
def set_span_inputs(inputs):
265
    """
266
    Set input data for current active span.
267
    
268
    Parameters:
269
    - inputs: Any - Input data (will be serialized)
270
    """
271

272
def add_span_exception(exception):
273
    """
274
    Record exception in current active span.
275
    
276
    Parameters:
277
    - exception: Exception - Exception object to record
278
    """
279
```
280

281
## Usage Examples
282

283
### Basic Tracing Setup
284

285
```python
286
import mlflow
287
import mlflow.tracing
288

289
# Configure tracing
290
mlflow.tracing.configure(
291
    disable_propagation=False,
292
    sampling_rate=1.0  # Trace all requests
293
)
294

295
# Enable tracing with specific experiment
296
mlflow.tracing.enable(experiment_id="123")
297

298
# Enable notebook display for interactive development
299
mlflow.tracing.enable_notebook_display()
300

301
# Simple function tracing
302
@mlflow.tracing.trace
303
def process_text(text):
304
    # Simulate text processing
305
    processed = text.upper().strip()
306
    return processed
307

308
# Use traced function
309
result = process_text("hello world")
310
print(f"Processed: {result}")
311

312
# Disable tracing when done
313
mlflow.tracing.disable()
314
```
315

316
### Manual Span Creation
317

318
```python
319
import mlflow.tracing
320
from mlflow.tracing import SpanType, SpanStatus
321

322
# Start experiment and enable tracing
323
mlflow.set_experiment("llm-application")
324
mlflow.tracing.enable()
325

326
# Create manual spans with hierarchy
327
with mlflow.tracing.trace("user_query_processing", SpanType.CHAIN) as parent_span:
328
    # Set input data
329
    mlflow.tracing.set_span_inputs({"query": "What is machine learning?"})
330
    
331
    # Child span for retrieval
332
    with mlflow.tracing.trace("document_retrieval", SpanType.RETRIEVER) as retrieval_span:
333
        # Simulate document retrieval
334
        documents = ["Doc1", "Doc2", "Doc3"]
335
        mlflow.tracing.set_span_output({"retrieved_docs": documents})
336
        mlflow.tracing.set_span_attribute("num_documents", len(documents))
337
    
338
    # Child span for LLM processing
339
    with mlflow.tracing.trace("llm_generation", SpanType.LLM) as llm_span:
340
        mlflow.tracing.set_span_inputs({
341
            "prompt": "Answer based on context",
342
            "context": documents
343
        })
344
        
345
        # Simulate LLM call
346
        response = "Machine learning is a subset of AI..."
347
        mlflow.tracing.set_span_output({"response": response})
348
        mlflow.tracing.set_span_attribute("model_name", "gpt-4")
349
        mlflow.tracing.set_span_attribute("tokens_used", 150)
350
    
351
    # Set final output
352
    mlflow.tracing.set_span_output({"final_answer": response})
353
```
354

355
### Automatic Instrumentation
356

357
```python
358
import mlflow.tracing
359
from mlflow.tracing import mlflow_trace
360

361
# Decorate functions for automatic tracing
362
@mlflow_trace(span_type=SpanType.EMBEDDING)
363
def get_embeddings(texts):
364
    """Get embeddings for list of texts."""
365
    # Simulate embedding generation
366
    embeddings = [[0.1, 0.2, 0.3] for _ in texts]
367
    return embeddings
368

369
@mlflow_trace(span_type=SpanType.RETRIEVER)
370
def retrieve_documents(query_embedding, top_k=5):
371
    """Retrieve relevant documents."""
372
    # Simulate document retrieval
373
    documents = [f"Document {i}" for i in range(top_k)]
374
    return documents
375

376
@mlflow_trace(span_type=SpanType.LLM)
377
def generate_response(query, context_docs):
378
    """Generate response using LLM."""
379
    # Simulate LLM generation
380
    response = f"Based on {len(context_docs)} documents: Answer here"
381
    return response
382

383
# Enable tracing
384
mlflow.tracing.enable()
385

386
# Use instrumented functions - traces created automatically
387
query = "What is MLflow?"
388
query_embedding = get_embeddings([query])
389
relevant_docs = retrieve_documents(query_embedding[0])
390
response = generate_response(query, relevant_docs)
391

392
print(f"Response: {response}")
393
```
394

395
### Error Handling and Status
396

397
```python
398
import mlflow.tracing
399
from mlflow.tracing import SpanStatus
400

401
mlflow.tracing.enable()
402

403
def risky_operation(data):
404
    """Function that might fail."""
405
    if len(data) == 0:
406
        raise ValueError("Empty data provided")
407
    return sum(data)
408

409
# Handle errors in traces
410
with mlflow.tracing.trace("data_processing") as span:
411
    try:
412
        mlflow.tracing.set_span_inputs({"data": [1, 2, 3]})
413
        result = risky_operation([1, 2, 3])
414
        mlflow.tracing.set_span_output({"result": result})
415
        # Span automatically gets OK status
416
        
417
    except Exception as e:
418
        # Record exception in span
419
        mlflow.tracing.add_span_exception(e)
420
        # Span automatically gets ERROR status
421
        raise
422

423
# Example with empty data (will fail)
424
try:
425
    with mlflow.tracing.trace("failing_operation") as span:
426
        mlflow.tracing.set_span_inputs({"data": []})
427
        result = risky_operation([])
428
except ValueError as e:
429
    print(f"Operation failed: {e}")
430
```
431

432
### Chat Model Tracing
433

434
```python
435
import mlflow.tracing
436
from mlflow.tracing import SpanType
437

438
mlflow.tracing.enable()
439

440
# Simulate chat model with tools
441
def chat_with_tools(messages, tools=None):
442
    """Simulate chat model with tool calling capability."""
443
    
444
    with mlflow.tracing.trace("chat_completion", SpanType.CHAT_MODEL) as span:
445
        # Set chat tools if provided
446
        if tools:
447
            mlflow.tracing.set_span_chat_tools(tools)
448
        
449
        # Set input messages
450
        mlflow.tracing.set_span_inputs({
451
            "messages": messages,
452
            "tools": tools or []
453
        })
454
        
455
        # Set model attributes
456
        mlflow.tracing.set_span_attributes({
457
            "model_name": "gpt-4-turbo",
458
            "temperature": 0.7,
459
            "max_tokens": 1000
460
        })
461
        
462
        # Simulate tool calling
463
        if tools and "calculator" in [t["name"] for t in tools]:
464
            with mlflow.tracing.trace("tool_call", SpanType.TOOL) as tool_span:
465
                mlflow.tracing.set_span_inputs({
466
                    "tool_name": "calculator",
467
                    "arguments": {"expression": "2 + 2"}
468
                })
469
                tool_result = 4
470
                mlflow.tracing.set_span_output({"result": tool_result})
471
        
472
        # Generate final response
473
        response = {
474
            "role": "assistant",
475
            "content": "The calculation result is 4."
476
        }
477
        
478
        mlflow.tracing.set_span_output({"response": response})
479
        return response
480

481
# Define available tools
482
tools = [
483
    {
484
        "name": "calculator",
485
        "description": "Perform mathematical calculations",
486
        "parameters": {
487
            "type": "object",
488
            "properties": {
489
                "expression": {"type": "string"}
490
            }
491
        }
492
    }
493
]
494

495
# Use chat model with tools
496
messages = [{"role": "user", "content": "What is 2 + 2?"}]
497
response = chat_with_tools(messages, tools=tools)
498
```
499

500
### Trace Analysis and Retrieval
501

502
```python
503
import mlflow
504
import mlflow.tracing
505

506
# Set up experiment
507
mlflow.set_experiment("trace_analysis")
508
mlflow.tracing.enable()
509

510
# Generate some traces
511
for i in range(5):
512
    with mlflow.tracing.trace(f"operation_{i}") as span:
513
        mlflow.tracing.set_span_inputs({"iteration": i})
514
        # Simulate varying execution times
515
        import time
516
        time.sleep(0.1 * i)
517
        mlflow.tracing.set_span_output({"processed": True})
518

519
# Retrieve and analyze traces
520
experiment = mlflow.get_experiment_by_name("trace_analysis")
521
traces = mlflow.tracing.search_traces(
522
    experiment_ids=[experiment.experiment_id],
523
    max_results=10
524
)
525

526
print(f"Found {len(traces)} traces")
527

528
for trace in traces:
529
    print(f"Trace ID: {trace.request_id}")
530
    print(f"  Start time: {trace.timestamp_ms}")
531
    print(f"  Duration: {trace.execution_time_ms}ms")
532
    print(f"  Status: {trace.status}")
533
    
534
    # Analyze spans
535
    for span in trace.spans:
536
        print(f"    Span: {span.name} ({span.span_type})")
537
        print(f"      Duration: {span.end_time_ns - span.start_time_ns}ns")
538
        if span.inputs:
539
            print(f"      Inputs: {span.inputs}")
540
        if span.outputs:
541
            print(f"      Outputs: {span.outputs}")
542

543
# Get specific trace
544
if traces:
545
    last_trace = mlflow.tracing.get_trace(traces[0].request_id)
546
    print(f"\nDetailed trace: {last_trace.request_id}")
547
    print(f"Execution time: {last_trace.execution_time_ms}ms")
548
```
549

550
### Integration with LangChain
551

552
```python
553
import mlflow.tracing
554

555
# Enable automatic LangChain instrumentation
556
mlflow.tracing.instrument_langchain()
557
mlflow.tracing.enable()
558

559
# LangChain components will be automatically traced
560
from langchain.llms import OpenAI
561
from langchain.chains import LLMChain
562
from langchain.prompts import PromptTemplate
563

564
# Create LangChain components (automatically instrumented)
565
prompt_template = PromptTemplate(
566
    input_variables=["topic"],
567
    template="Write a short summary about {topic}"
568
)
569

570
llm = OpenAI(temperature=0.7)
571
chain = LLMChain(llm=llm, prompt=prompt_template)
572

573
# Run chain - automatically traced
574
with mlflow.tracing.trace("langchain_summary_generation"):
575
    result = chain.run(topic="machine learning")
576
    print(f"Summary: {result}")
577

578
# Traces will show LangChain component hierarchy
579
# with automatic span creation for each component
580
```
581

582
## Types
583

584
```python { .api }
585
from mlflow.entities import Trace, Span, SpanEvent
586
from mlflow.tracing.provider import SpanType, SpanStatus, SpanStatusCode
587

588
class Trace:
589
    request_id: str
590
    experiment_id: str
591
    timestamp_ms: int
592
    execution_time_ms: int
593
    status: str
594
    request_metadata: Dict[str, Any]
595
    tags: Dict[str, str]
596
    spans: List[Span]
597

598
class Span:
599
    name: str
600
    context: SpanContext
601
    parent_id: Optional[str]
602
    start_time_ns: int
603
    end_time_ns: int
604
    status_code: str
605
    status_message: str
606
    attributes: Dict[str, Any]
607
    events: List[SpanEvent]
608
    span_type: str
609
    inputs: Optional[Any]
610
    outputs: Optional[Any]
611

612
class SpanContext:
613
    span_id: str
614
    trace_id: str
615

616
class SpanEvent:
617
    name: str
618
    timestamp_ns: int
619
    attributes: Dict[str, Any]
620

621
class SpanType:
622
    UNKNOWN: str
623
    LLM: str
624
    CHAT_MODEL: str
625
    CHAIN: str
626
    AGENT: str
627
    TOOL: str
628
    RETRIEVER: str
629
    EMBEDDING: str
630
    RERANKER: str
631
    PARSER: str
632

633
class SpanStatus:
634
    OK: str
635
    ERROR: str
636
    CANCELLED: str
637
    TIMEOUT: str
638

639
class SpanStatusCode:
640
    UNSET: str
641
    OK: str
642
    ERROR: str
643

644
# Context manager and decorator types
645
from typing import ContextManager, Callable, Any
646

647
def trace(
648
    name: str,
649
    span_type: Optional[str] = None,
650
    inputs: Optional[Any] = None,
651
    attributes: Optional[Dict[str, Any]] = None
652
) -> ContextManager[Span]: ...
653

654
def mlflow_trace(
655
    span_type: Optional[str] = None,
656
    name: Optional[str] = None
657
) -> Callable[[Callable], Callable]: ...
658
```