0
# Rendering and Effects
1

2
Visual rendering utilities for creating masks, overlays, and special effects from segmentation results. Provides functions for background blur, body part blur, mask generation, and custom visualization effects.
3

4
## Capabilities
5

6
### Mask Generation
7

8
Create binary and colored masks from segmentation results.
9

10
```typescript { .api }
11
/**
12
 * Creates a binary mask image from segmentation results
13
 * @param segmentation - Segmentation result (person or part segmentation)
14
 * @param foreground - Foreground color (default: transparent)
15
 * @param background - Background color (default: opaque black)
16
 * @param drawContour - Draw contour lines around segments
17
 * @param foregroundIds - Which part IDs are considered foreground
18
 * @returns ImageData containing the mask
19
 */
20
function toMask(
21
  segmentation: SemanticPersonSegmentation | SemanticPartSegmentation | PersonSegmentation[] | PartSegmentation[],
22
  foreground?: Color,
23
  background?: Color,
24
  drawContour?: boolean,
25
  foregroundIds?: number[]
26
): ImageData;
27

28
/**
29
 * Creates colored visualization of body part segmentation
30
 * @param partSegmentation - Part segmentation result
31
 * @param partColors - RGB colors for each part (default: rainbow colors)
32
 * @returns ImageData with colored parts
33
 */
34
function toColoredPartMask(
35
  partSegmentation: SemanticPartSegmentation | PartSegmentation[],
36
  partColors?: Array<[number, number, number]>
37
): ImageData;
38

39
interface Color {
40
  r: number;  // Red (0-255)
41
  g: number;  // Green (0-255)
42
  b: number;  // Blue (0-255)
43
  a: number;  // Alpha (0-255)
44
}
45
```
46

47
### Background Effects
48

49
Create background blur and bokeh effects for virtual backgrounds.
50

51
```typescript { .api }
52
/**
53
 * Creates bokeh/background blur effect while keeping people in focus
54
 * @param canvas - Target canvas for rendering
55
 * @param image - Source image/video element
56
 * @param personSegmentation - Person segmentation result
57
 * @param backgroundBlurAmount - Background blur intensity (1-20)
58
 * @param edgeBlurAmount - Edge smoothing blur (0-20)
59
 * @param flipHorizontal - Flip result horizontally
60
 */
61
function drawBokehEffect(
62
  canvas: HTMLCanvasElement | OffscreenCanvas,
63
  image: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
64
  personSegmentation: SemanticPersonSegmentation | PersonSegmentation[],
65
  backgroundBlurAmount?: number,
66
  edgeBlurAmount?: number,
67
  flipHorizontal?: boolean
68
): void;
69

70
/**
71
 * Draws image with mask overlay on canvas
72
 * @param canvas - Target canvas for rendering
73
 * @param image - Source image/video element
74
 * @param maskImage - Mask to overlay (from toMask function)
75
 * @param maskOpacity - Mask opacity (0-1)
76
 * @param maskBlurAmount - Blur radius for mask edges (0-20)
77
 * @param flipHorizontal - Flip result horizontally
78
 */
79
function drawMask(
80
  canvas: HTMLCanvasElement | OffscreenCanvas,
81
  image: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
82
  maskImage: ImageData | null,
83
  maskOpacity?: number,
84
  maskBlurAmount?: number,
85
  flipHorizontal?: boolean
86
): void;
87

88
/**
89
 * Draws image with pixelated mask effect
90
 * @param canvas - Target canvas for rendering
91
 * @param image - Source image/video element
92
 * @param maskImage - Mask to overlay
93
 * @param maskOpacity - Mask opacity (0-1)
94
 * @param maskBlurAmount - Blur radius for mask edges (0-20)
95
 * @param flipHorizontal - Flip result horizontally
96
 * @param pixelCellWidth - Pixel cell size for pixelation effect
97
 */
98
function drawPixelatedMask(
99
  canvas: HTMLCanvasElement | OffscreenCanvas,
100
  image: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
101
  maskImage: ImageData,
102
  maskOpacity?: number,
103
  maskBlurAmount?: number,
104
  flipHorizontal?: boolean,
105
  pixelCellWidth?: number
106
): void;
107
```
108

109
### Body Part Effects
110

111
Target specific body parts for privacy or creative effects.
112

113
```typescript { .api }
114
/**
115
 * Blurs specific body parts (e.g., faces for privacy)
116
 * @param canvas - Target canvas for rendering  
117
 * @param image - Source image/video element
118
 * @param partSegmentation - Part segmentation result
119
 * @param bodyPartIdsToBlur - Array of part IDs to blur (default: [0,1] for faces)
120
 * @param backgroundBlurAmount - Blur intensity (1-20)
121
 * @param edgeBlurAmount - Edge smoothing blur (0-20)
122
 * @param flipHorizontal - Flip result horizontally
123
 */
124
function blurBodyPart(
125
  canvas: HTMLCanvasElement | OffscreenCanvas,
126
  image: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
127
  partSegmentation: SemanticPartSegmentation | PartSegmentation[],
128
  bodyPartIdsToBlur?: number[],
129
  backgroundBlurAmount?: number,
130
  edgeBlurAmount?: number,
131
  flipHorizontal?: boolean
132
): void;
133
```
134

135
## Usage Examples
136

137
### Basic Background Blur
138

139
```typescript
140
import * as bodyPix from '@tensorflow-models/body-pix';
141

142
const net = await bodyPix.load();
143
const video = document.getElementById('webcam');
144
const canvas = document.getElementById('output');
145

146
// Create background blur effect
147
const segmentation = await net.segmentPerson(video);
148
bodyPix.drawBokehEffect(canvas, video, segmentation, 7, 3, true);
149
```
150

151
### Custom Mask Creation
152

153
```typescript
154
// Create custom colored mask
155
const segmentation = await net.segmentPerson(imageElement);
156
const greenScreenMask = bodyPix.toMask(segmentation,
157
  { r: 0, g: 255, b: 0, a: 255 },     // Green foreground
158
  { r: 255, g: 0, b: 255, a: 255 }    // Magenta background
159
);
160

161
// Apply mask with custom styling
162
bodyPix.drawMask(canvas, imageElement, greenScreenMask, 0.8, 2);
163
```
164

165
### Face Privacy Protection
166

167
```typescript
168
// Blur faces for privacy
169
const partSegmentation = await net.segmentPersonParts(imageElement);
170
const FACE_PARTS = [0, 1]; // left_face, right_face
171

172
bodyPix.blurBodyPart(canvas, imageElement, partSegmentation, FACE_PARTS, 15, 3);
173
```
174

175
### Part-Specific Visualization
176

177
```typescript
178
// Create rainbow-colored body part visualization
179
const partSegmentation = await net.segmentPersonParts(imageElement);
180
const coloredParts = bodyPix.toColoredPartMask(partSegmentation);
181

182
// Display on canvas
183
const ctx = canvas.getContext('2d');
184
ctx.putImageData(coloredParts, 0, 0);
185

186
// Create hand-only highlight
187
const HAND_PARTS = [10, 11];
188
const handMask = bodyPix.toMask(partSegmentation,
189
  { r: 255, g: 255, b: 0, a: 200 },   // Semi-transparent yellow
190
  { r: 0, g: 0, b: 0, a: 0 },         // Transparent background
191
  false,
192
  HAND_PARTS
193
);
194

195
bodyPix.drawMask(canvas, imageElement, handMask, 0.7);
196
```
197

198
### Real-time Video Effects
199

200
```typescript
201
async function applyVideoEffect() {
202
  const video = document.getElementById('webcam');
203
  const canvas = document.getElementById('output');
204
  
205
  // Resize canvas to match video
206
  canvas.width = video.videoWidth;
207
  canvas.height = video.videoHeight;
208
  
209
  const segmentation = await net.segmentPerson(video, {
210
    internalResolution: 'medium',
211
    segmentationThreshold: 0.7
212
  });
213
  
214
  // Apply different effects based on user selection
215
  const effectType = document.getElementById('effect-select').value;
216
  
217
  switch (effectType) {
218
    case 'blur':
219
      bodyPix.drawBokehEffect(canvas, video, segmentation, 10, 5, true);
220
      break;
221
      
222
    case 'mask':
223
      const mask = bodyPix.toMask(segmentation);
224
      bodyPix.drawMask(canvas, video, mask, 0.8);
225
      break;
226
      
227
    case 'pixelate':
228
      const pixelMask = bodyPix.toMask(segmentation);
229
      bodyPix.drawPixelatedMask(canvas, video, pixelMask, 1.0, 0, true, 8);
230
      break;
231
  }
232
  
233
  // Continue for next frame
234
  requestAnimationFrame(applyVideoEffect);
235
}
236
```
237

238
### Advanced Custom Effects
239

240
```typescript
241
// Create gradient mask effect
242
function createGradientMask(segmentation: SemanticPersonSegmentation): ImageData {
243
  const { data, width, height } = segmentation;
244
  const maskData = new Uint8ClampedArray(width * height * 4);
245
  
246
  for (let i = 0; i < data.length; i++) {
247
    const isPerson = data[i] === 1;
248
    const x = i % width;
249
    const y = Math.floor(i / width);
250
    
251
    if (isPerson) {
252
      // Create gradient based on position
253
      const gradientIntensity = x / width;
254
      maskData[i * 4] = Math.floor(255 * gradientIntensity);     // R
255
      maskData[i * 4 + 1] = Math.floor(255 * (1 - gradientIntensity)); // G
256
      maskData[i * 4 + 2] = 128;  // B
257
      maskData[i * 4 + 3] = 255;  // A
258
    } else {
259
      // Transparent background
260
      maskData[i * 4 + 3] = 0;
261
    }
262
  }
263
  
264
  return new ImageData(maskData, width, height);
265
}
266

267
const segmentation = await net.segmentPerson(imageElement);
268
const gradientMask = createGradientMask(segmentation);
269
bodyPix.drawMask(canvas, imageElement, gradientMask, 0.8);
270
```
271

272
## Performance Tips
273

274
- **Canvas Reuse**: Reuse the same canvas element for better performance
275
- **Effect Caching**: Cache mask ImageData when possible for repeated effects
276
- **Resolution Matching**: Match canvas size to video/image dimensions
277
- **Blur Optimization**: Lower blur amounts improve performance significantly
278
- **Frame Skipping**: Skip frames during video processing for better real-time performance
279

280
## Common Use Cases
281

282
- **Video conferencing** background replacement and blur
283
- **Social media filters** and creative effects  
284
- **Privacy protection** with selective blurring
285
- **Green screen effects** and chroma key replacement
286
- **Medical imaging** visualization and analysis
287
- **Security applications** with person highlighting
288
- **Educational tools** for anatomy and body part identification
289
- **Gaming and AR** applications with real-time effects