0
# Media Handling
1

2
Rich media support with content-addressable storage, SHA-256 hashing, and media reference tags for embedding images, audio, video, and documents in Langfuse traces.
3

4
## Capabilities
5

6
### LangfuseMedia Class
7

8
A class for wrapping media objects for upload to Langfuse with automatic content hashing and reference tag generation.
9

10
```typescript { .api }
11
class LangfuseMedia {
12
  constructor(params: LangfuseMediaParams);
13

14
  async getId(): Promise<string | null>;
15
  async getSha256Hash(): Promise<string | undefined>;
16
  async getTag(): Promise<string | null>;
17

18
  get contentLength(): number | undefined;
19
  get base64DataUri(): string | null;
20

21
  toJSON(): string | null;
22
}
23

24
type LangfuseMediaParams =
25
  | {
26
      source: "base64_data_uri";
27
      base64DataUri: string;
28
    }
29
  | {
30
      source: "bytes";
31
      contentBytes: Uint8Array;
32
      contentType: MediaContentType;
33
    };
34

35
type MediaContentType =
36
  // Images
37
  | "image/png" | "image/jpeg" | "image/jpg" | "image/webp"
38
  | "image/gif" | "image/svg+xml" | "image/tiff" | "image/bmp"
39
  // Audio
40
  | "audio/mpeg" | "audio/mp3" | "audio/wav" | "audio/ogg"
41
  | "audio/oga" | "audio/aac" | "audio/mp4" | "audio/flac"
42
  // Video
43
  | "video/mp4" | "video/webm"
44
  // Text
45
  | "text/plain" | "text/html" | "text/css" | "text/csv"
46
  // Documents
47
  | "application/pdf" | "application/msword" | "application/vnd.ms-excel"
48
  // Archives
49
  | "application/zip"
50
  // Data
51
  | "application/json" | "application/xml" | "application/octet-stream";
52
```
53

54
**Import:**
55

56
```typescript
57
import { LangfuseMedia, type LangfuseMediaParams, type MediaContentType } from '@langfuse/core';
58
```
59

60
#### Constructor
61

62
Creates a new LangfuseMedia instance from either a base64 data URI or raw bytes.
63

64
```typescript { .api }
65
constructor(params: LangfuseMediaParams)
66
```
67

68
**Parameters:**
69

70
**Option 1: Base64 Data URI**
71
- `params.source: "base64_data_uri"` - Indicates media is from a data URI
72
- `params.base64DataUri: string` - Complete data URI (e.g., "data:image/png;base64,...")
73

74
**Option 2: Raw Bytes**
75
- `params.source: "bytes"` - Indicates media is from raw bytes
76
- `params.contentBytes: Uint8Array` - The raw content bytes
77
- `params.contentType: MediaContentType` - MIME type of the content
78

79
**Usage Examples:**
80

81
```typescript
82
import { LangfuseMedia } from '@langfuse/core';
83

84
// From base64 data URI (common with web APIs)
85
const mediaFromUri = new LangfuseMedia({
86
  source: "base64_data_uri",
87
  base64DataUri: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
88
});
89

90
// From raw bytes (common with file system operations)
91
const imageBytes = new Uint8Array([137, 80, 78, 71, 13, 10, 26, 10, ...]);
92
const mediaFromBytes = new LangfuseMedia({
93
  source: "bytes",
94
  contentBytes: imageBytes,
95
  contentType: "image/png"
96
});
97

98
// From file in Node.js
99
import { readFile } from 'fs/promises';
100

101
const fileBuffer = await readFile('screenshot.png');
102
const mediaFromFile = new LangfuseMedia({
103
  source: "bytes",
104
  contentBytes: new Uint8Array(fileBuffer),
105
  contentType: "image/png"
106
});
107

108
// From canvas in browser
109
const canvas = document.getElementById('myCanvas') as HTMLCanvasElement;
110
const dataUri = canvas.toDataURL('image/png');
111
const mediaFromCanvas = new LangfuseMedia({
112
  source: "base64_data_uri",
113
  base64DataUri: dataUri
114
});
115
```
116

117
#### getId()
118

119
Gets a unique content-based identifier for the media.
120

121
```typescript { .api }
122
/**
123
 * Gets a unique identifier derived from the SHA-256 hash of the content
124
 * @returns The media ID (22 characters) or null if hash generation failed
125
 */
126
async getId(): Promise<string | null>;
127
```
128

129
**Returns**: A 22-character URL-safe base64-encoded string derived from the SHA-256 hash, or null if unavailable.
130

131
**Description**: The ID is deterministic - identical content always produces the same ID, enabling content-addressable storage and deduplication.
132

133
**Usage Example:**
134

135
```typescript
136
const media = new LangfuseMedia({
137
  source: "base64_data_uri",
138
  base64DataUri: "data:image/png;base64,iVBORw0..."
139
});
140

141
const mediaId = await media.getId();
142
console.log(mediaId);  // "A1B2C3D4E5F6G7H8I9J0K1"
143

144
// Same content = same ID (deduplication)
145
const media2 = new LangfuseMedia({
146
  source: "base64_data_uri",
147
  base64DataUri: "data:image/png;base64,iVBORw0..."
148
});
149
const mediaId2 = await media2.getId();
150
console.log(mediaId === mediaId2);  // true
151
```
152

153
#### getSha256Hash()
154

155
Gets the SHA-256 hash of the media content.
156

157
```typescript { .api }
158
/**
159
 * Gets the SHA-256 hash of the content as a base64-encoded string
160
 * @returns The hash or undefined if unavailable
161
 */
162
async getSha256Hash(): Promise<string | undefined>;
163
```
164

165
**Returns**: Base64-encoded SHA-256 hash or undefined if content is unavailable.
166

167
**Usage Example:**
168

169
```typescript
170
const media = new LangfuseMedia({
171
  source: "bytes",
172
  contentBytes: new Uint8Array([72, 101, 108, 108, 111]),
173
  contentType: "text/plain"
174
});
175

176
const hash = await media.getSha256Hash();
177
console.log(hash);  // Base64-encoded SHA-256 hash
178

179
// Use for integrity verification
180
const verifyHash = await media.getSha256Hash();
181
console.log(hash === verifyHash);  // true
182
```
183

184
#### getTag()
185

186
Gets the media reference tag for embedding in trace data.
187

188
```typescript { .api }
189
/**
190
 * Gets the media reference tag for embedding in trace attributes
191
 * @returns The media tag or null if required data is missing
192
 */
193
async getTag(): Promise<string | null>;
194
```
195

196
**Returns**: A special tag string in the format `@@@langfuseMedia:type=<contentType>|id=<mediaId>|source=<source>@@@`, or null if required data is missing.
197

198
**Description**: This tag can be embedded in trace input/output or observation data. When viewed in the Langfuse UI, the tag is replaced with the actual media content (image viewer, video player, etc.).
199

200
**Usage Example:**
201

202
```typescript
203
const media = new LangfuseMedia({
204
  source: "base64_data_uri",
205
  base64DataUri: "data:image/png;base64,iVBORw0KGgoAAAA..."
206
});
207

208
const tag = await media.getTag();
209
console.log(tag);
210
// "@@@langfuseMedia:type=image/png|id=A1B2C3D4E5F6G7H8I9J0K1|source=base64_data_uri@@@"
211

212
// Embed in trace data
213
import { LangfuseAPIClient } from '@langfuse/core';
214

215
const client = new LangfuseAPIClient({ /* ... */ });
216

217
await client.ingestion.batch({
218
  batch: [{
219
    type: 'trace-create',
220
    id: 'event-1',
221
    timestamp: new Date().toISOString(),
222
    body: {
223
      id: 'trace-1',
224
      name: 'Image Processing',
225
      input: {
226
        description: 'Processing this image:',
227
        image: tag  // Embedded media tag
228
      },
229
      output: {
230
        result: 'Image processed successfully'
231
      }
232
    }
233
  }]
234
});
235
```
236

237
#### contentLength (getter)
238

239
Gets the length of the media content in bytes.
240

241
```typescript { .api }
242
/**
243
 * Gets the content length in bytes
244
 */
245
get contentLength(): number | undefined;
246
```
247

248
**Returns**: The content size in bytes, or undefined if no content is available.
249

250
**Usage Example:**
251

252
```typescript
253
const media = new LangfuseMedia({
254
  source: "bytes",
255
  contentBytes: new Uint8Array(1024 * 100),  // 100 KB
256
  contentType: "image/jpeg"
257
});
258

259
console.log(`Size: ${media.contentLength} bytes`);  // "Size: 102400 bytes"
260

261
// Check size before upload
262
if (media.contentLength && media.contentLength > 10 * 1024 * 1024) {
263
  console.warn('Media exceeds 10MB, consider compression');
264
}
265
```
266

267
#### base64DataUri (getter)
268

269
Gets the media content as a base64 data URI.
270

271
```typescript { .api }
272
/**
273
 * Gets the complete base64 data URI
274
 */
275
get base64DataUri(): string | null;
276
```
277

278
**Returns**: The complete data URI string (e.g., "data:image/png;base64,..."), or null if no content.
279

280
**Usage Example:**
281

282
```typescript
283
const media = new LangfuseMedia({
284
  source: "bytes",
285
  contentBytes: new Uint8Array([72, 101, 108, 108, 111]),
286
  contentType: "text/plain"
287
});
288

289
const dataUri = media.base64DataUri;
290
console.log(dataUri);  // "data:text/plain;base64,SGVsbG8="
291

292
// Use in HTML img tag
293
const img = document.createElement('img');
294
img.src = dataUri;
295
document.body.appendChild(img);
296
```
297

298
#### toJSON()
299

300
Serializes the media to JSON (returns the base64 data URI).
301

302
```typescript { .api }
303
/**
304
 * Serializes the media to JSON
305
 * @returns The base64 data URI or null
306
 */
307
toJSON(): string | null;
308
```
309

310
**Returns**: The base64 data URI, or null if no content is available.
311

312
**Usage Example:**
313

314
```typescript
315
const media = new LangfuseMedia({
316
  source: "base64_data_uri",
317
  base64DataUri: "data:image/png;base64,iVBORw0..."
318
});
319

320
const json = JSON.stringify({ screenshot: media });
321
// Automatically calls toJSON()
322
console.log(json);
323
// '{"screenshot":"data:image/png;base64,iVBORw0..."}'
324

325
// Parse back
326
const parsed = JSON.parse(json);
327
const restoredMedia = new LangfuseMedia({
328
  source: "base64_data_uri",
329
  base64DataUri: parsed.screenshot
330
});
331
```
332

333
### ParsedMediaReference Type
334

335
Represents a parsed media reference from trace data.
336

337
```typescript { .api }
338
interface ParsedMediaReference {
339
  mediaId: string;
340
  source: string;
341
  contentType: MediaContentType;
342
}
343
```
344

345
**Import:**
346

347
```typescript
348
import { type ParsedMediaReference } from '@langfuse/core';
349
```
350

351
**Properties:**
352
- `mediaId: string` - The unique media identifier (22 characters)
353
- `source: string` - The source type ("base64_data_uri" or "bytes")
354
- `contentType: MediaContentType` - The MIME type of the media
355

356
## Usage Patterns
357

358
### Basic Image Upload
359

360
```typescript
361
import { LangfuseMedia, LangfuseAPIClient } from '@langfuse/core';
362

363
async function uploadScreenshot(imageDataUri: string) {
364
  // Create media object
365
  const media = new LangfuseMedia({
366
    source: "base64_data_uri",
367
    base64DataUri: imageDataUri
368
  });
369

370
  // Get media tag for embedding
371
  const mediaTag = await media.getTag();
372

373
  // Create trace with embedded media
374
  const client = new LangfuseAPIClient({ /* ... */ });
375

376
  await client.ingestion.batch({
377
    batch: [{
378
      type: 'trace-create',
379
      id: 'event-1',
380
      timestamp: new Date().toISOString(),
381
      body: {
382
        id: 'trace-1',
383
        name: 'UI Screenshot',
384
        input: {
385
          action: 'screenshot_captured',
386
          image: mediaTag
387
        }
388
      }
389
    }]
390
  });
391
}
392
```
393

394
### Multiple Media Types
395

396
```typescript
397
import { LangfuseMedia } from '@langfuse/core';
398

399
async function processMultimodalInput(
400
  image: Uint8Array,
401
  audio: Uint8Array,
402
  text: string
403
) {
404
  // Create media objects
405
  const imageMedia = new LangfuseMedia({
406
    source: "bytes",
407
    contentBytes: image,
408
    contentType: "image/jpeg"
409
  });
410

411
  const audioMedia = new LangfuseMedia({
412
    source: "bytes",
413
    contentBytes: audio,
414
    contentType: "audio/mp3"
415
  });
416

417
  // Get tags
418
  const imageTag = await imageMedia.getTag();
419
  const audioTag = await audioMedia.getTag();
420

421
  // Create trace with multiple media
422
  await client.ingestion.batch({
423
    batch: [{
424
      type: 'trace-create',
425
      id: 'event-1',
426
      timestamp: new Date().toISOString(),
427
      body: {
428
        id: 'trace-1',
429
        name: 'Multimodal Processing',
430
        input: {
431
          text: text,
432
          image: imageTag,
433
          audio: audioTag
434
        }
435
      }
436
    }]
437
  });
438
}
439
```
440

441
### File System Integration (Node.js)
442

443
```typescript
444
import { LangfuseMedia } from '@langfuse/core';
445
import { readFile } from 'fs/promises';
446
import { extname } from 'path';
447

448
const mimeTypes: Record<string, MediaContentType> = {
449
  '.png': 'image/png',
450
  '.jpg': 'image/jpeg',
451
  '.jpeg': 'image/jpeg',
452
  '.gif': 'image/gif',
453
  '.pdf': 'application/pdf',
454
  '.mp3': 'audio/mp3',
455
  '.mp4': 'video/mp4'
456
};
457

458
async function createMediaFromFile(filePath: string) {
459
  // Read file
460
  const buffer = await readFile(filePath);
461
  const ext = extname(filePath).toLowerCase();
462
  const contentType = mimeTypes[ext];
463

464
  if (!contentType) {
465
    throw new Error(`Unsupported file type: ${ext}`);
466
  }
467

468
  // Create media
469
  return new LangfuseMedia({
470
    source: "bytes",
471
    contentBytes: new Uint8Array(buffer),
472
    contentType: contentType
473
  });
474
}
475

476
// Usage
477
const media = await createMediaFromFile('./screenshot.png');
478
const tag = await media.getTag();
479
```
480

481
### Canvas Capture (Browser)
482

483
```typescript
484
import { LangfuseMedia } from '@langfuse/core';
485

486
function captureCanvas(canvas: HTMLCanvasElement) {
487
  // Convert canvas to data URI
488
  const dataUri = canvas.toDataURL('image/png');
489

490
  // Create media
491
  return new LangfuseMedia({
492
    source: "base64_data_uri",
493
    base64DataUri: dataUri
494
  });
495
}
496

497
// Usage
498
const canvas = document.getElementById('myCanvas') as HTMLCanvasElement;
499
const media = captureCanvas(canvas);
500
const tag = await media.getTag();
501
```
502

503
### Media Deduplication
504

505
```typescript
506
import { LangfuseMedia } from '@langfuse/core';
507

508
async function trackMediaUsage(dataUri: string) {
509
  const media = new LangfuseMedia({
510
    source: "base64_data_uri",
511
    base64DataUri: dataUri
512
  });
513

514
  const mediaId = await media.getId();
515

516
  // Check if already uploaded
517
  const mediaCache = new Map<string, boolean>();
518

519
  if (mediaCache.has(mediaId!)) {
520
    console.log('Media already uploaded, reusing reference');
521
  } else {
522
    console.log('New media, uploading');
523
    mediaCache.set(mediaId!, true);
524
  }
525

526
  return media.getTag();
527
}
528
```
529

530
### Size Validation
531

532
```typescript
533
import { LangfuseMedia } from '@langfuse/core';
534

535
async function createMediaWithValidation(
536
  contentBytes: Uint8Array,
537
  contentType: MediaContentType,
538
  maxSize: number = 10 * 1024 * 1024  // 10 MB
539
) {
540
  if (contentBytes.length > maxSize) {
541
    throw new Error(
542
      `Media size ${contentBytes.length} exceeds maximum ${maxSize} bytes`
543
    );
544
  }
545

546
  const media = new LangfuseMedia({
547
    source: "bytes",
548
    contentBytes,
549
    contentType
550
  });
551

552
  console.log(`Created media: ${media.contentLength} bytes, ${contentType}`);
553
  return media;
554
}
555
```
556

557
### Compression Before Upload
558

559
```typescript
560
import { LangfuseMedia } from '@langfuse/core';
561

562
async function compressAndCreateMedia(
563
  imageDataUri: string,
564
  maxSizeKb: number = 500
565
) {
566
  // Load image
567
  const img = new Image();
568
  img.src = imageDataUri;
569
  await new Promise(resolve => img.onload = resolve);
570

571
  // Create canvas for compression
572
  const canvas = document.createElement('canvas');
573
  const ctx = canvas.getContext('2d')!;
574

575
  // Calculate new dimensions
576
  let width = img.width;
577
  let height = img.height;
578
  const maxDimension = 1920;
579

580
  if (width > maxDimension || height > maxDimension) {
581
    if (width > height) {
582
      height = Math.round((height * maxDimension) / width);
583
      width = maxDimension;
584
    } else {
585
      width = Math.round((width * maxDimension) / height);
586
      height = maxDimension;
587
    }
588
  }
589

590
  canvas.width = width;
591
  canvas.height = height;
592
  ctx.drawImage(img, 0, 0, width, height);
593

594
  // Try different quality levels
595
  let quality = 0.9;
596
  let compressedDataUri: string;
597

598
  do {
599
    compressedDataUri = canvas.toDataURL('image/jpeg', quality);
600
    quality -= 0.1;
601
  } while (
602
    compressedDataUri.length > maxSizeKb * 1024 * 1.37 &&
603
    quality > 0.1
604
  );
605

606
  // Create media from compressed image
607
  return new LangfuseMedia({
608
    source: "base64_data_uri",
609
    base64DataUri: compressedDataUri
610
  });
611
}
612
```
613

614
## Type Definitions
615

616
```typescript { .api }
617
class LangfuseMedia {
618
  constructor(params: LangfuseMediaParams);
619
  async getId(): Promise<string | null>;
620
  async getSha256Hash(): Promise<string | undefined>;
621
  async getTag(): Promise<string | null>;
622
  get contentLength(): number | undefined;
623
  get base64DataUri(): string | null;
624
  toJSON(): string | null;
625
}
626

627
type LangfuseMediaParams =
628
  | {
629
      source: "base64_data_uri";
630
      base64DataUri: string;
631
    }
632
  | {
633
      source: "bytes";
634
      contentBytes: Uint8Array;
635
      contentType: MediaContentType;
636
    };
637

638
type MediaContentType =
639
  | "image/png" | "image/jpeg" | "image/jpg" | "image/webp"
640
  | "image/gif" | "image/svg+xml" | "image/tiff" | "image/bmp"
641
  | "audio/mpeg" | "audio/mp3" | "audio/wav" | "audio/ogg"
642
  | "audio/oga" | "audio/aac" | "audio/mp4" | "audio/flac"
643
  | "video/mp4" | "video/webm"
644
  | "text/plain" | "text/html" | "text/css" | "text/csv"
645
  | "application/pdf" | "application/msword" | "application/vnd.ms-excel"
646
  | "application/zip"
647
  | "application/json" | "application/xml" | "application/octet-stream";
648

649
interface ParsedMediaReference {
650
  mediaId: string;
651
  source: string;
652
  contentType: MediaContentType;
653
}
654
```
655

656
## Best Practices
657

658
1. **Content Type Accuracy**: Always specify the correct MIME type for reliable media handling
659
2. **Size Limits**: Consider compressing large media before upload to reduce storage costs
660
3. **Deduplication**: Leverage content-based IDs to avoid uploading duplicate media
661
4. **Error Handling**: Always check for null returns from async methods (getId, getTag)
662
5. **Browser Compatibility**: Use appropriate polyfills for crypto APIs in older browsers
663
6. **Memory Management**: Process large files in streams rather than loading entirely in memory
664
7. **Format Selection**: Prefer web-optimized formats (JPEG for photos, PNG for graphics, WebP when supported)
665
8. **Metadata**: Include descriptive context around media in trace input/output, not just the tag
666
9. **Security**: Never include sensitive information in media that might be stored long-term
667
10. **Testing**: Test media handling with various file sizes and types during development
668