0
# Distributed Tracing and Diagnostics
1

2
Azure Core provides comprehensive distributed tracing capabilities with OpenTelemetry integration, automatic HTTP instrumentation, and flexible span management. The tracing system enables end-to-end observability across Azure SDK operations with W3C Trace Context compliance for interoperability.
3

4
## Core Components
5

6
### AbstractSpan
7

8
Protocol interface for distributed tracing span implementations with context management and attribute support.
9

10
```python { .api }
11
from azure.core.tracing import AbstractSpan, SpanKind
12
from typing import Optional, Dict, Union, Any, Type, ContextManager, Callable
13

14
class AbstractSpan(Protocol[SpanType]):
15
    def __init__(self, span: Optional[SpanType] = None, name: Optional[str] = None, **kwargs: Any) -> None: ...
16
    
17
    # Span lifecycle
18
    def span(self, name: str = "child_span", **kwargs: Any) -> AbstractSpan[SpanType]: ...
19
    def start(self) -> None: ...
20
    def finish(self) -> None: ...
21
    
22
    # Context manager support
23
    def __enter__(self) -> AbstractSpan[SpanType]: ...
24
    def __exit__(self, exception_type: Optional[Type[BaseException]], 
25
                exception_value: Optional[BaseException], 
26
                traceback: Any) -> None: ...
27
    
28
    # Span properties
29
    @property
30
    def kind(self) -> Optional[SpanKind]: ...
31
    @kind.setter
32
    def kind(self, value: SpanKind) -> None: ...
33
    
34
    @property
35
    def span_instance(self) -> SpanType: ...
36
    
37
    # Attribute management
38
    def add_attribute(self, key: str, value: Union[str, int]) -> None: ...
39
    def set_http_attributes(self, request: Any, response: Optional[Any] = None) -> None: ...
40
    
41
    # Trace context operations
42
    def to_header(self) -> Dict[str, str]: ...
43
    def get_trace_parent(self) -> str: ...
44
    
45
    # Class methods for context management
46
    @classmethod
47
    def link(cls, traceparent: str, attributes: Optional[Dict[str, Any]] = None) -> None: ...
48
    @classmethod
49
    def link_from_headers(cls, headers: Dict[str, str], attributes: Optional[Dict[str, Any]] = None) -> None: ...
50
    @classmethod
51
    def get_current_span(cls) -> SpanType: ...
52
    @classmethod
53
    def get_current_tracer(cls) -> Any: ...
54
    @classmethod
55
    def set_current_span(cls, span: SpanType) -> None: ...
56
    @classmethod
57
    def set_current_tracer(cls, tracer: Any) -> None: ...
58
    @classmethod
59
    def change_context(cls, span: SpanType) -> ContextManager[SpanType]: ...
60
    @classmethod
61
    def with_current_context(cls, func: Callable) -> Callable: ...
62
```
63

64
### SpanKind
65

66
Enumeration for categorizing spans based on their role in distributed traces.
67

68
```python { .api }
69
from azure.core.tracing import SpanKind
70
from enum import Enum
71

72
class SpanKind(Enum):
73
    UNSPECIFIED = 1  # Default span kind
74
    SERVER = 2       # Handles incoming requests
75
    CLIENT = 3       # Makes outgoing requests
76
    PRODUCER = 4     # Initiates operations (messaging)
77
    CONSUMER = 5     # Processes operations from producers
78
    INTERNAL = 6     # Internal application operations
79
```
80

81
### TracingOptions
82

83
Configuration options for controlling tracing behavior.
84

85
```python { .api }
86
from azure.core.tracing import TracingOptions
87
from typing import TypedDict, Mapping, Union, Sequence
88

89
AttributeValue = Union[str, bool, int, float, Sequence[str], Sequence[bool], Sequence[int], Sequence[float]]
90
Attributes = Mapping[str, AttributeValue]
91

92
class TracingOptions(TypedDict, total=False):
93
    enabled: bool           # Override global tracing setting
94
    attributes: Attributes  # Additional span attributes
95
```
96

97
### Link
98

99
Represents references between spans across service boundaries.
100

101
```python { .api }
102
from azure.core.tracing import Link
103
from typing import Dict, Optional
104

105
class Link:
106
    def __init__(self, headers: Dict[str, str], attributes: Optional[Attributes] = None) -> None: ...
107
    
108
    headers: Dict[str, str]
109
    attributes: Optional[Attributes]
110
```
111

112
## Basic Usage
113

114
### Manual Span Management
115

116
```python
117
from azure.core.tracing import AbstractSpan, SpanKind
118

119
def manual_tracing_example():
120
    # Create span with context manager
121
    with AbstractSpan(name="operation", kind=SpanKind.CLIENT) as span:
122
        span.add_attribute("user.id", "12345")
123
        span.add_attribute("operation.type", "data_fetch")
124
        
125
        try:
126
            # Your operation code here
127
            result = perform_operation()
128
            span.add_attribute("operation.success", True)
129
            span.add_attribute("result.count", len(result))
130
            return result
131
        except Exception as e:
132
            span.add_attribute("error.type", type(e).__name__)
133
            span.add_attribute("error.message", str(e))
134
            raise
135

136
def nested_spans_example():
137
    with AbstractSpan(name="parent_operation", kind=SpanKind.INTERNAL) as parent:
138
        parent.add_attribute("operation.phase", "initialization")
139
        
140
        # Create child span
141
        with parent.span(name="child_operation", kind=SpanKind.CLIENT) as child:
142
            child.add_attribute("target.service", "external-api")
143
            perform_external_call()
144
        
145
        parent.add_attribute("operation.phase", "completion")
146
```
147

148
### Decorator-Based Tracing
149

150
```python
151
from azure.core.tracing.decorator import distributed_trace
152
from azure.core.tracing import SpanKind
153

154
@distributed_trace
155
def simple_traced_function():
156
    """Automatically traced with function name as span name"""
157
    return process_data()
158

159
@distributed_trace(
160
    name_of_span="custom_operation_name",
161
    kind=SpanKind.CLIENT,
162
    tracing_attributes={"service.name": "my-service", "operation.type": "fetch"}
163
)
164
def custom_traced_function(user_id: str):
165
    """Traced with custom span name, kind, and attributes"""
166
    return fetch_user_data(user_id)
167

168
# Async version
169
from azure.core.tracing.decorator_async import distributed_trace_async
170

171
@distributed_trace_async(name_of_span="async_operation", kind=SpanKind.CLIENT)
172
async def async_traced_function():
173
    """Async function with automatic tracing"""
174
    return await async_operation()
175
```
176

177
### HTTP Request Tracing
178

179
```python
180
from azure.core.tracing import AbstractSpan, SpanKind
181
from azure.core.rest import HttpRequest
182

183
def http_request_tracing():
184
    request = HttpRequest("GET", "https://api.example.com/data")
185
    
186
    with AbstractSpan(name="http_call", kind=SpanKind.CLIENT) as span:
187
        # Automatically sets HTTP attributes
188
        span.set_http_attributes(request)
189
        
190
        # Make request
191
        response = client.send_request(request)
192
        
193
        # Update span with response information
194
        span.set_http_attributes(request, response)
195
        
196
        return response
197
```
198

199
## OpenTelemetry Integration
200

201
### OpenTelemetrySpan
202

203
Complete AbstractSpan implementation using OpenTelemetry spans.
204

205
```python
206
from azure.core.tracing.ext.opentelemetry_span import OpenTelemetrySpan
207
from azure.core.tracing import SpanKind
208

209
# Basic OpenTelemetry span usage
210
with OpenTelemetrySpan(name="azure_operation", kind=SpanKind.CLIENT) as span:
211
    span.add_attribute("azure.namespace", "Microsoft.Storage")
212
    span.add_attribute("azure.service", "BlobService")
213
    
214
    # Your Azure SDK operation
215
    result = perform_azure_operation()
216

217
# With links to parent traces
218
parent_headers = {"traceparent": "00-trace-id-span-id-01"}
219
links = [Link(parent_headers, {"link.type": "follows_from"})]
220

221
with OpenTelemetrySpan(name="linked_operation", links=links) as span:
222
    # Operation that follows from another trace
223
    process_linked_operation()
224
```
225

226
### Context Propagation
227

228
```python
229
from azure.core.tracing import AbstractSpan
230

231
def context_propagation_example():
232
    with AbstractSpan(name="parent") as parent_span:
233
        # Get trace context headers for propagation
234
        trace_headers = parent_span.to_header()
235
        
236
        # Headers contain W3C Trace Context
237
        # traceparent: 00-trace-id-span-id-flags
238
        # tracestate: (optional state information)
239
        
240
        # Pass headers to downstream service
241
        request = HttpRequest("POST", "/downstream", headers=trace_headers)
242
        response = client.send_request(request)
243
        
244
        return response
245

246
def incoming_trace_context():
247
    # Receive headers from upstream service
248
    incoming_headers = {
249
        "traceparent": "00-12345678901234567890123456789012-1234567890123456-01"
250
    }
251
    
252
    # Create links to connect traces
253
    AbstractSpan.link_from_headers(incoming_headers, {"link.type": "follows_from"})
254
```
255

256
## Pipeline Integration
257

258
### DistributedTracingPolicy
259

260
Automatic HTTP request tracing through the pipeline system.
261

262
```python
263
from azure.core.pipeline.policies import DistributedTracingPolicy
264
from azure.core import PipelineClient
265

266
# Pipeline client with automatic tracing
267
client = PipelineClient(
268
    base_url="https://api.service.com",
269
    policies=[
270
        DistributedTracingPolicy(),
271
        # ... other policies
272
    ]
273
)
274

275
# All HTTP requests through this client are automatically traced
276
request = HttpRequest("GET", "/api/data")
277
response = client.send_request(request)  # Automatically creates span
278
```
279

280
### Custom Instrumentation Configuration
281

282
```python
283
from azure.core.pipeline.policies import DistributedTracingPolicy
284

285
# Custom instrumentation configuration
286
instrumentation_config = {
287
    "span_name": "custom_http_call",
288
    "attributes": {
289
        "service.name": "my-azure-client",
290
        "service.version": "1.0.0"
291
    }
292
}
293

294
policy = DistributedTracingPolicy(instrumentation_config=instrumentation_config)
295
```
296

297
## Advanced Features
298

299
### Custom Tracer Integration
300

301
```python
302
from azure.core.tracing import AbstractSpan
303

304
class CustomTracer:
305
    def create_span(self, name: str, **kwargs):
306
        # Your custom span implementation
307
        return CustomSpan(name, **kwargs)
308

309
# Set custom tracer globally
310
AbstractSpan.set_current_tracer(CustomTracer())
311

312
# All spans will now use your custom implementation
313
with AbstractSpan(name="operation") as span:
314
    # Uses CustomTracer.create_span
315
    pass
316
```
317

318
### Span Suppression
319

320
```python
321
from azure.core.settings import settings
322

323
# Disable tracing globally
324
settings.tracing_enabled = False
325

326
# Or per operation using TracingOptions
327
@distributed_trace(tracing_attributes={"enabled": False})
328
def untraced_operation():
329
    # This operation won't be traced
330
    pass
331
```
332

333
### Error Handling and Status
334

335
```python
336
from azure.core.tracing import AbstractSpan, SpanKind
337

338
def error_handling_example():
339
    with AbstractSpan(name="operation_with_errors", kind=SpanKind.INTERNAL) as span:
340
        try:
341
            risky_operation()
342
            span.add_attribute("operation.status", "success")
343
        except ValueError as e:
344
            # Automatic error attributes
345
            span.add_attribute("error.type", "ValueError")
346
            span.add_attribute("error.message", str(e))
347
            span.add_attribute("operation.status", "error")
348
            raise
349
        except Exception as e:
350
            span.add_attribute("error.type", type(e).__name__)
351
            span.add_attribute("error.message", str(e))
352
            span.add_attribute("operation.status", "error")
353
            raise
354
```
355

356
### Async Context Preservation
357

358
```python
359
import asyncio
360
from azure.core.tracing import AbstractSpan
361
from azure.core.tracing.decorator_async import distributed_trace_async
362

363
@distributed_trace_async(name_of_span="async_coordinator")
364
async def async_coordinator():
365
    # Context is automatically preserved across await boundaries
366
    tasks = [
367
        async_task_1(),
368
        async_task_2(),
369
        async_task_3()
370
    ]
371
    
372
    # All tasks will be child spans of the coordinator span
373
    results = await asyncio.gather(*tasks)
374
    return results
375

376
@distributed_trace_async(name_of_span="async_task", kind=SpanKind.INTERNAL)
377
async def async_task_1():
378
    await asyncio.sleep(1)  # Simulated async work
379
    return "task1_result"
380
```
381

382
### Batch Operations
383

384
```python
385
from azure.core.tracing import AbstractSpan, SpanKind
386

387
def batch_operation_tracing():
388
    with AbstractSpan(name="batch_processor", kind=SpanKind.INTERNAL) as batch_span:
389
        batch_span.add_attribute("batch.size", 100)
390
        batch_span.add_attribute("batch.type", "data_processing")
391
        
392
        for i, item in enumerate(batch_items):
393
            # Create span for each item or group of items
394
            with batch_span.span(name=f"process_item_{i}", kind=SpanKind.INTERNAL) as item_span:
395
                item_span.add_attribute("item.id", item.id)
396
                item_span.add_attribute("item.type", item.type)
397
                
398
                try:
399
                    result = process_item(item)
400
                    item_span.add_attribute("processing.success", True)
401
                except Exception as e:
402
                    item_span.add_attribute("processing.success", False)
403
                    item_span.add_attribute("error.type", type(e).__name__)
404
                    # Continue processing other items
405
                    continue
406
```
407

408
## Configuration and Settings
409

410
### Global Settings
411

412
```python
413
from azure.core.settings import settings
414

415
# Enable/disable tracing globally
416
settings.tracing_enabled = True
417

418
# Configure tracing options
419
settings.tracing_implementation = "OpenTelemetry"  # or custom implementation
420
```
421

422
### Per-Operation Configuration
423

424
```python
425
# Using TracingOptions
426
tracing_options = {
427
    "enabled": True,
428
    "attributes": {
429
        "user.id": "12345",
430
        "operation.priority": "high"
431
    }
432
}
433

434
@distributed_trace(tracing_attributes=tracing_options)
435
def configured_operation():
436
    pass
437
```
438

439
## Key Features
440

441
**OpenTelemetry Integration**: Full compatibility with OpenTelemetry standards and tooling.
442

443
**W3C Trace Context**: Automatic propagation of trace context using W3C standards for interoperability.
444

445
**Automatic HTTP Instrumentation**: Pipeline policy automatically traces HTTP requests with semantic conventions.
446

447
**Flexible Span Management**: Support for manual span creation, decorators, and context managers.
448

449
**Cross-Service Linking**: Link spans across service boundaries for complete trace visibility.
450

451
**Error Handling**: Automatic error capture with structured error attributes.
452

453
**Async Support**: Full async/await support with proper context preservation.
454

455
**Custom Tracer Support**: Plugin architecture for custom tracing implementations.
456

457
**Performance Optimized**: Minimal overhead when tracing is disabled, efficient span creation.
458

459
The distributed tracing system provides comprehensive observability for Azure SDK operations while maintaining flexibility for custom implementations and seamless integration with existing observability infrastructure.