0
# Media Management
1

2
The MediaManager provides functionality to resolve media references in Langfuse trace data. Media content (images, audio, video, PDFs, etc.) is stored separately and referenced using special string identifiers. The MediaManager resolves these references back to usable content like base64 data URIs.
3

4
## Capabilities
5

6
### Resolve Media References
7

8
Replace media reference strings in an object with base64 data URIs by fetching the actual media content from Langfuse.
9

10
```typescript { .api }
11
/**
12
 * Resolves media reference strings in an object to base64 data URIs
13
 *
14
 * Recursively traverses the input object looking for media reference strings
15
 * in the format "@@@langfuseMedia:...@@@". When found, fetches the actual
16
 * media content and replaces the reference with a base64 data URI.
17
 *
18
 * If fetching fails for a reference, a warning is logged and the reference
19
 * string is left unchanged in the output.
20
 *
21
 * @template T - The type of the object being processed
22
 * @param params - Configuration object for resolving references
23
 * @returns A deep copy of the input with media references resolved
24
 */
25
async resolveReferences<T>(
26
  params: LangfuseMediaResolveMediaReferencesParams<T>
27
): Promise<T>;
28

29
type LangfuseMediaResolveMediaReferencesParams<T> = {
30
  /** The object to process for media references */
31
  obj: T;
32
  /** The format to resolve media references to (currently only "base64DataUri" is supported) */
33
  resolveWith: "base64DataUri";
34
  /** Maximum depth to traverse when processing nested objects (default: 10) */
35
  maxDepth?: number;
36
};
37
```
38

39
**Usage Examples:**
40

41
```typescript
42
import { LangfuseClient } from '@langfuse/client';
43

44
const langfuse = new LangfuseClient();
45

46
// Simple object with media reference
47
const obj = {
48
  image: "@@@langfuseMedia:type=image/jpeg|id=abc123|source=bytes@@@",
49
  text: "This is a regular string"
50
};
51

52
const resolved = await langfuse.media.resolveReferences({
53
  obj,
54
  resolveWith: "base64DataUri"
55
});
56

57
// Result:
58
// {
59
//   image: "data:image/jpeg;base64,/9j/4AAQSkZJRg...",
60
//   text: "This is a regular string"
61
// }
62
```
63

64
**Nested Objects and Arrays:**
65

66
```typescript
67
// Complex nested structure
68
const traceData = {
69
  observations: [
70
    {
71
      input: {
72
        message: "Process this image",
73
        image: "@@@langfuseMedia:type=image/png|id=img001|source=bytes@@@"
74
      },
75
      output: {
76
        result: "Processed",
77
        thumbnail: "@@@langfuseMedia:type=image/jpeg|id=img002|source=bytes@@@"
78
      },
79
      metadata: {
80
        attachments: [
81
          "@@@langfuseMedia:type=application/pdf|id=doc001|source=bytes@@@",
82
          "@@@langfuseMedia:type=audio/wav|id=aud001|source=bytes@@@"
83
        ]
84
      }
85
    }
86
  ]
87
};
88

89
// Resolves all media references throughout the nested structure
90
const resolved = await langfuse.media.resolveReferences({
91
  obj: traceData,
92
  resolveWith: "base64DataUri"
93
});
94

95
// All media references are now base64 data URIs
96
console.log(resolved.observations[0].input.image);
97
// "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
98

99
console.log(resolved.observations[0].metadata.attachments[0]);
100
// "data:application/pdf;base64,JVBERi0xLjcKJeLjz9MKMSAw..."
101
```
102

103
**Controlling Traversal Depth:**
104

105
```typescript
106
// Deeply nested structure
107
const deepObject = {
108
  level1: {
109
    level2: {
110
      level3: {
111
        level4: {
112
          level5: {
113
            media: "@@@langfuseMedia:type=image/jpeg|id=deep001|source=bytes@@@"
114
          }
115
        }
116
      }
117
    }
118
  }
119
};
120

121
// Limit traversal depth
122
const resolved = await langfuse.media.resolveReferences({
123
  obj: deepObject,
124
  resolveWith: "base64DataUri",
125
  maxDepth: 3  // Will not process beyond level 3
126
});
127

128
// Media at level 5 will not be resolved due to maxDepth limit
129
```
130

131
**Resolving Trace Data:**
132

133
```typescript
134
// Fetch trace from API (contains media references)
135
const trace = await langfuse.api.trace.get('trace-id-123');
136

137
// Resolve all media references in the trace
138
const resolvedTrace = await langfuse.media.resolveReferences({
139
  obj: trace,
140
  resolveWith: "base64DataUri"
141
});
142

143
// Now you can use the resolved media content
144
if (resolvedTrace.observations) {
145
  for (const obs of resolvedTrace.observations) {
146
    if (obs.input?.image) {
147
      // image is now a base64 data URI, can be displayed in browser
148
      console.log('Image data URI:', obs.input.image);
149
    }
150
  }
151
}
152
```
153

154
**Error Handling:**
155

156
```typescript
157
// If media fetch fails, the reference string remains unchanged
158
const obj = {
159
  validMedia: "@@@langfuseMedia:type=image/jpeg|id=valid123|source=bytes@@@",
160
  invalidMedia: "@@@langfuseMedia:type=image/jpeg|id=invalid999|source=bytes@@@"
161
};
162

163
const resolved = await langfuse.media.resolveReferences({
164
  obj,
165
  resolveWith: "base64DataUri"
166
});
167

168
// Valid media is resolved
169
console.log(resolved.validMedia);
170
// "data:image/jpeg;base64,/9j/4AAQSkZJRg..."
171

172
// Invalid media remains as reference (warning logged)
173
console.log(resolved.invalidMedia);
174
// "@@@langfuseMedia:type=image/jpeg|id=invalid999|source=bytes@@@"
175
```
176

177
### Parse Media Reference String
178

179
Parse a media reference string into its component parts (static method).
180

181
```typescript { .api }
182
/**
183
 * Parses a media reference string into a ParsedMediaReference object
184
 *
185
 * Media reference string format:
186
 * "@@@langfuseMedia:type=<mime>|id=<id>|source=<source>@@@"
187
 *
188
 * @param referenceString - The reference string to parse
189
 * @returns Parsed media reference with mediaId, source, and contentType
190
 * @throws Error if the reference string is invalid or missing required fields
191
 */
192
static parseReferenceString(
193
  referenceString: string
194
): ParsedMediaReference;
195

196
type ParsedMediaReference = {
197
  /** The unique identifier for the media content */
198
  mediaId: string;
199
  /** The source type of the media (e.g., "bytes", "base64_data_uri") */
200
  source: string;
201
  /** The MIME type of the media content */
202
  contentType: MediaContentType;
203
};
204
```
205

206
**Usage Examples:**
207

208
```typescript
209
import { MediaManager } from '@langfuse/client';
210

211
// Parse a valid reference string
212
const referenceString = "@@@langfuseMedia:type=image/jpeg|id=abc-123|source=bytes@@@";
213

214
const parsed = MediaManager.parseReferenceString(referenceString);
215

216
console.log(parsed);
217
// {
218
//   mediaId: "abc-123",
219
//   source: "bytes",
220
//   contentType: "image/jpeg"
221
// }
222
```
223

224
**Extracting Information:**
225

226
```typescript
227
// Use parsed information to understand media type
228
const ref = "@@@langfuseMedia:type=application/pdf|id=doc001|source=base64_data_uri@@@";
229
const parsed = MediaManager.parseReferenceString(ref);
230

231
console.log(`Media ID: ${parsed.mediaId}`);          // "doc001"
232
console.log(`Content Type: ${parsed.contentType}`);  // "application/pdf"
233
console.log(`Source: ${parsed.source}`);              // "base64_data_uri"
234

235
// Determine media category
236
if (parsed.contentType.startsWith('image/')) {
237
  console.log('This is an image');
238
} else if (parsed.contentType.startsWith('audio/')) {
239
  console.log('This is audio');
240
} else if (parsed.contentType === 'application/pdf') {
241
  console.log('This is a PDF document');
242
}
243
```
244

245
**Error Handling:**
246

247
```typescript
248
// Invalid format - missing prefix
249
try {
250
  MediaManager.parseReferenceString("type=image/jpeg|id=123|source=bytes@@@");
251
} catch (error) {
252
  console.error(error.message);
253
  // "Reference string does not start with '@@@langfuseMedia:type='"
254
}
255

256
// Invalid format - missing suffix
257
try {
258
  MediaManager.parseReferenceString("@@@langfuseMedia:type=image/jpeg|id=123|source=bytes");
259
} catch (error) {
260
  console.error(error.message);
261
  // "Reference string does not end with '@@@'"
262
}
263

264
// Invalid format - missing required fields
265
try {
266
  MediaManager.parseReferenceString("@@@langfuseMedia:type=image/jpeg|id=123@@@");
267
} catch (error) {
268
  console.error(error.message);
269
  // "Missing required fields in reference string"
270
}
271
```
272

273
**Finding References in Strings:**
274

275
```typescript
276
// Extract and parse all media references from a string
277
const text = `
278
  Image 1: @@@langfuseMedia:type=image/png|id=img1|source=bytes@@@
279
  Image 2: @@@langfuseMedia:type=image/jpeg|id=img2|source=bytes@@@
280
  PDF: @@@langfuseMedia:type=application/pdf|id=doc1|source=bytes@@@
281
`;
282

283
const regex = /@@@langfuseMedia:.+?@@@/g;
284
const matches = text.match(regex);
285

286
if (matches) {
287
  const parsed = matches.map(ref => MediaManager.parseReferenceString(ref));
288

289
  console.log('Found media references:');
290
  parsed.forEach(p => {
291
    console.log(`- ${p.contentType}: ${p.mediaId}`);
292
  });
293
  // - image/png: img1
294
  // - image/jpeg: img2
295
  // - application/pdf: doc1
296
}
297
```
298

299
## Reference String Format
300

301
Media reference strings follow a specific format that encodes media metadata:
302

303
```
304
@@@langfuseMedia:type=<mime-type>|id=<media-id>|source=<source-type>@@@
305
```
306

307
**Components:**
308

309
- **Prefix:** `@@@langfuseMedia:`
310
- **type:** MIME type of the media (e.g., `image/jpeg`, `application/pdf`)
311
- **id:** Unique identifier for the media content
312
- **source:** Source type (e.g., `bytes`, `base64_data_uri`)
313
- **Suffix:** `@@@`
314

315
**Examples:**
316

317
```typescript
318
// Image reference
319
"@@@langfuseMedia:type=image/jpeg|id=abc-123-def|source=bytes@@@"
320

321
// PDF reference
322
"@@@langfuseMedia:type=application/pdf|id=doc-456|source=base64_data_uri@@@"
323

324
// Audio reference
325
"@@@langfuseMedia:type=audio/wav|id=sound-789|source=bytes@@@"
326

327
// Video reference
328
"@@@langfuseMedia:type=video/mp4|id=clip-012|source=bytes@@@"
329
```
330

331
## Types
332

333
### MediaContentType
334

335
Union type representing all supported MIME types for media content.
336

337
```typescript { .api }
338
type MediaContentType =
339
  // Images
340
  | "image/png"
341
  | "image/jpeg"
342
  | "image/jpg"
343
  | "image/webp"
344
  | "image/gif"
345
  | "image/svg+xml"
346
  | "image/tiff"
347
  | "image/bmp"
348
  // Audio
349
  | "audio/mpeg"
350
  | "audio/mp3"
351
  | "audio/wav"
352
  | "audio/ogg"
353
  | "audio/oga"
354
  | "audio/aac"
355
  | "audio/mp4"
356
  | "audio/flac"
357
  // Video
358
  | "video/mp4"
359
  | "video/webm"
360
  // Text
361
  | "text/plain"
362
  | "text/html"
363
  | "text/css"
364
  | "text/csv"
365
  // Documents
366
  | "application/pdf"
367
  | "application/msword"
368
  | "application/vnd.ms-excel"
369
  // Other
370
  | "application/zip"
371
  | "application/json"
372
  | "application/xml"
373
  | "application/octet-stream";
374

375
// Constants for type-safe access
376
const MediaContentType = {
377
  // Images
378
  ImagePng: "image/png",
379
  ImageJpeg: "image/jpeg",
380
  ImageJpg: "image/jpg",
381
  ImageWebp: "image/webp",
382
  ImageGif: "image/gif",
383
  ImageSvgXml: "image/svg+xml",
384
  ImageTiff: "image/tiff",
385
  ImageBmp: "image/bmp",
386
  // Audio
387
  AudioMpeg: "audio/mpeg",
388
  AudioMp3: "audio/mp3",
389
  AudioWav: "audio/wav",
390
  AudioOgg: "audio/ogg",
391
  AudioOga: "audio/oga",
392
  AudioAac: "audio/aac",
393
  AudioMp4: "audio/mp4",
394
  AudioFlac: "audio/flac",
395
  // Video
396
  VideoMp4: "video/mp4",
397
  VideoWebm: "video/webm",
398
  // Text
399
  TextPlain: "text/plain",
400
  TextHtml: "text/html",
401
  TextCss: "text/css",
402
  TextCsv: "text/csv",
403
  // Documents
404
  ApplicationPdf: "application/pdf",
405
  ApplicationMsword: "application/msword",
406
  ApplicationMsExcel: "application/vnd.ms-excel",
407
  // Other
408
  ApplicationZip: "application/zip",
409
  ApplicationJson: "application/json",
410
  ApplicationXml: "application/xml",
411
  ApplicationOctetStream: "application/octet-stream",
412
} as const;
413
```
414

415
### ParsedMediaReference
416

417
Result type from parsing a media reference string.
418

419
```typescript { .api }
420
type ParsedMediaReference = {
421
  /** The unique identifier for the media content */
422
  mediaId: string;
423
  /** The source type of the media */
424
  source: string;
425
  /** The MIME type of the media content */
426
  contentType: MediaContentType;
427
};
428
```
429

430
### LangfuseMediaResolveMediaReferencesParams
431

432
Parameters for the `resolveReferences` method.
433

434
```typescript { .api }
435
type LangfuseMediaResolveMediaReferencesParams<T> = {
436
  /** The object to process for media references */
437
  obj: T;
438
  /** The format to resolve media references to */
439
  resolveWith: "base64DataUri";
440
  /** Maximum depth to traverse when processing nested objects (default: 10) */
441
  maxDepth?: number;
442
};
443
```
444

445
## Common Use Cases
446

447
### Working with Images
448

449
```typescript
450
import { LangfuseClient } from '@langfuse/client';
451

452
const langfuse = new LangfuseClient();
453

454
// Fetch trace with image references
455
const trace = await langfuse.api.trace.get('trace-123');
456

457
// Resolve image references
458
const resolved = await langfuse.media.resolveReferences({
459
  obj: trace,
460
  resolveWith: "base64DataUri"
461
});
462

463
// Use resolved images in HTML
464
if (resolved.observations) {
465
  for (const obs of resolved.observations) {
466
    if (obs.input?.screenshot) {
467
      // Create an img element with the data URI
468
      const img = document.createElement('img');
469
      img.src = obs.input.screenshot; // data:image/png;base64,...
470
      document.body.appendChild(img);
471
    }
472
  }
473
}
474
```
475

476
### Processing PDF Documents
477

478
```typescript
479
// Object with PDF reference
480
const data = {
481
  document: "@@@langfuseMedia:type=application/pdf|id=doc123|source=bytes@@@"
482
};
483

484
// Resolve to base64 data URI
485
const resolved = await langfuse.media.resolveReferences({
486
  obj: data,
487
  resolveWith: "base64DataUri"
488
});
489

490
// Use in iframe or download
491
console.log(resolved.document);
492
// "data:application/pdf;base64,JVBERi0xLjcK..."
493

494
// Display in iframe
495
const iframe = document.createElement('iframe');
496
iframe.src = resolved.document;
497
document.body.appendChild(iframe);
498
```
499

500
### Handling Audio Files
501

502
```typescript
503
// Trace with audio recording reference
504
const trace = {
505
  input: {
506
    recording: "@@@langfuseMedia:type=audio/wav|id=rec001|source=bytes@@@"
507
  }
508
};
509

510
// Resolve audio reference
511
const resolved = await langfuse.media.resolveReferences({
512
  obj: trace,
513
  resolveWith: "base64DataUri"
514
});
515

516
// Play audio in browser
517
const audio = new Audio(resolved.input.recording);
518
audio.play();
519

520
// Or use in HTML audio element
521
const audioElement = document.createElement('audio');
522
audioElement.src = resolved.input.recording;
523
audioElement.controls = true;
524
document.body.appendChild(audioElement);
525
```
526

527
### Processing Video Content
528

529
```typescript
530
// Video reference in observation
531
const observation = {
532
  output: {
533
    generatedVideo: "@@@langfuseMedia:type=video/mp4|id=vid001|source=bytes@@@"
534
  }
535
};
536

537
// Resolve video
538
const resolved = await langfuse.media.resolveReferences({
539
  obj: observation,
540
  resolveWith: "base64DataUri"
541
});
542

543
// Display video
544
const video = document.createElement('video');
545
video.src = resolved.output.generatedVideo;
546
video.controls = true;
547
video.width = 640;
548
document.body.appendChild(video);
549
```
550

551
### Batch Processing Multiple Traces
552

553
```typescript
554
// Fetch multiple traces
555
const traces = await langfuse.api.trace.list({ limit: 50 });
556

557
// Resolve all media references across all traces
558
const resolvedTraces = await langfuse.media.resolveReferences({
559
  obj: traces,
560
  resolveWith: "base64DataUri"
561
});
562

563
// Process each trace with resolved media
564
for (const trace of resolvedTraces.data) {
565
  console.log(`Processing trace: ${trace.id}`);
566

567
  // Access resolved media content
568
  if (trace.observations) {
569
    for (const obs of trace.observations) {
570
      if (obs.input?.image?.startsWith('data:image/')) {
571
        console.log('Found resolved image');
572
      }
573
    }
574
  }
575
}
576
```
577

578
### Selective Media Resolution
579

580
```typescript
581
// Parse references first to decide what to resolve
582
const obj = {
583
  largeVideo: "@@@langfuseMedia:type=video/mp4|id=large001|source=bytes@@@",
584
  smallImage: "@@@langfuseMedia:type=image/jpeg|id=small001|source=bytes@@@"
585
};
586

587
// Extract reference strings
588
const videoRef = obj.largeVideo;
589
const imageRef = obj.smallImage;
590

591
// Parse to check content type
592
const videoParsed = MediaManager.parseReferenceString(videoRef);
593
const imageParsed = MediaManager.parseReferenceString(imageRef);
594

595
// Only resolve images (skip large videos)
596
const filtered = {
597
  smallImage: obj.smallImage,
598
  // Exclude largeVideo
599
};
600

601
const resolved = await langfuse.media.resolveReferences({
602
  obj: filtered,
603
  resolveWith: "base64DataUri"
604
});
605

606
console.log('Resolved only small image, skipped large video');
607
```
608

609
## Best Practices
610

611
### Memory Management
612

613
Be mindful of memory usage when resolving large media files or processing many objects:
614

615
```typescript
616
// Good: Process in batches
617
const traces = await langfuse.api.trace.list({ limit: 10 });
618
const resolved = await langfuse.media.resolveReferences({
619
  obj: traces,
620
  resolveWith: "base64DataUri"
621
});
622

623
// Less ideal: Processing too many large files at once
624
// const manyTraces = await langfuse.api.trace.list({ limit: 1000 });
625
// May cause memory issues if traces contain large media
626
```
627

628
### Error Resilience
629

630
The `resolveReferences` method handles individual fetch failures gracefully:
631

632
```typescript
633
// Some references may fail to fetch, but the method continues
634
const obj = {
635
  validImage: "@@@langfuseMedia:type=image/jpeg|id=valid|source=bytes@@@",
636
  missingImage: "@@@langfuseMedia:type=image/jpeg|id=deleted|source=bytes@@@",
637
  anotherValidImage: "@@@langfuseMedia:type=image/png|id=valid2|source=bytes@@@"
638
};
639

640
const resolved = await langfuse.media.resolveReferences({
641
  obj,
642
  resolveWith: "base64DataUri"
643
});
644

645
// Valid images are resolved, invalid ones remain as references
646
// Warnings are logged for failed fetches
647
```
648

649
### Caching Results
650

651
Consider caching resolved media to avoid repeated fetches:
652

653
```typescript
654
const cache = new Map<string, string>();
655

656
async function getResolvedMedia(referenceString: string): Promise<string> {
657
  if (cache.has(referenceString)) {
658
    return cache.get(referenceString)!;
659
  }
660

661
  const resolved = await langfuse.media.resolveReferences({
662
    obj: { media: referenceString },
663
    resolveWith: "base64DataUri"
664
  });
665

666
  cache.set(referenceString, resolved.media);
667
  return resolved.media;
668
}
669

670
// Use cached resolver
671
const image1 = await getResolvedMedia(
672
  "@@@langfuseMedia:type=image/jpeg|id=abc|source=bytes@@@"
673
);
674
const image2 = await getResolvedMedia(
675
  "@@@langfuseMedia:type=image/jpeg|id=abc|source=bytes@@@" // Cache hit
676
);
677
```
678

679
### Type Safety
680

681
Use TypeScript generics to maintain type information:
682

683
```typescript
684
interface TraceWithMedia {
685
  id: string;
686
  observations: Array<{
687
    input?: {
688
      image?: string;
689
    };
690
  }>;
691
}
692

693
// Type is preserved through resolution
694
const trace: TraceWithMedia = await langfuse.api.trace.get('trace-123');
695
const resolved: TraceWithMedia = await langfuse.media.resolveReferences({
696
  obj: trace,
697
  resolveWith: "base64DataUri"
698
});
699

700
// TypeScript knows the structure
701
console.log(resolved.observations[0].input?.image);
702
```
703

704
### Controlling Recursion
705

706
Use `maxDepth` to prevent excessive recursion on deeply nested structures:
707

708
```typescript
709
// Limit traversal for performance
710
const resolved = await langfuse.media.resolveReferences({
711
  obj: veryNestedObject,
712
  resolveWith: "base64DataUri",
713
  maxDepth: 5  // Don't traverse deeper than 5 levels
714
});
715
```
716

717
## Integration Patterns
718

719
### With Web Applications
720

721
```typescript
722
// React example
723
import { LangfuseClient } from '@langfuse/client';
724
import { useState, useEffect } from 'react';
725

726
function TraceViewer({ traceId }: { traceId: string }) {
727
  const [resolvedTrace, setResolvedTrace] = useState(null);
728
  const langfuse = new LangfuseClient();
729

730
  useEffect(() => {
731
    async function loadTrace() {
732
      const trace = await langfuse.api.trace.get(traceId);
733
      const resolved = await langfuse.media.resolveReferences({
734
        obj: trace,
735
        resolveWith: "base64DataUri"
736
      });
737
      setResolvedTrace(resolved);
738
    }
739
    loadTrace();
740
  }, [traceId]);
741

742
  if (!resolvedTrace) return <div>Loading...</div>;
743

744
  return (
745
    <div>
746
      {resolvedTrace.observations?.map((obs, idx) => (
747
        <div key={idx}>
748
          {obs.input?.image && (
749
            <img src={obs.input.image} alt="Trace input" />
750
          )}
751
        </div>
752
      ))}
753
    </div>
754
  );
755
}
756
```
757

758
### With Server-Side Processing
759

760
```typescript
761
// Node.js server example
762
import { LangfuseClient } from '@langfuse/client';
763
import express from 'express';
764

765
const app = express();
766
const langfuse = new LangfuseClient();
767

768
app.get('/api/trace/:traceId', async (req, res) => {
769
  try {
770
    const trace = await langfuse.api.trace.get(req.params.traceId);
771

772
    // Resolve media references before sending to client
773
    const resolved = await langfuse.media.resolveReferences({
774
      obj: trace,
775
      resolveWith: "base64DataUri"
776
    });
777

778
    res.json(resolved);
779
  } catch (error) {
780
    res.status(500).json({ error: error.message });
781
  }
782
});
783

784
app.listen(3000);
785
```
786

787
### With Data Processing Pipelines
788

789
```typescript
790
// ETL pipeline example
791
import { LangfuseClient } from '@langfuse/client';
792

793
const langfuse = new LangfuseClient();
794

795
async function exportTracesWithMedia() {
796
  const traces = await langfuse.api.trace.list({ limit: 100 });
797

798
  // Resolve all media in batch
799
  const resolved = await langfuse.media.resolveReferences({
800
    obj: traces,
801
    resolveWith: "base64DataUri"
802
  });
803

804
  // Export to file or database with resolved media
805
  for (const trace of resolved.data) {
806
    await saveToDatabase(trace); // Media is now embedded as base64
807
  }
808
}
809
```
810