tessl/npm-tensorflow-models--body-pix

Real-time person and body part segmentation using TensorFlow.js for web browsers with machine learning models.

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Securityby

Pending

The risk profile of this skill

Overview

Eval results

Files

BodyPix

Name: tessl/npm-tensorflow-models--body-pix
Author: tessl

BodyPix is a TensorFlow.js library that provides real-time person and body part segmentation in web browsers using machine learning models. It enables pixel-level classification of human figures and their 24 distinct body parts within images or video streams, supporting both single and multi-person scenarios with configurable neural network architectures.

Package Information

Package Name: @tensorflow-models/body-pix
Package Type: npm
Language: TypeScript
Installation: npm install @tensorflow-models/body-pix
Peer Dependencies: @tensorflow/tfjs-core, @tensorflow/tfjs-converter, @tensorflow/tfjs-backend-webgl

Core Imports

import * as bodyPix from '@tensorflow-models/body-pix';

Named imports:

import { load, BodyPix, toMask, drawBokehEffect } from '@tensorflow-models/body-pix';

For CommonJS:

const bodyPix = require('@tensorflow-models/body-pix');

Basic Usage

import * as bodyPix from '@tensorflow-models/body-pix';

// Load the model
const net = await bodyPix.load();

// Get image/video element
const imageElement = document.getElementById('person');

// Segment person
const segmentation = await net.segmentPerson(imageElement);

// Create and apply background blur effect
const canvas = document.getElementById('canvas');
bodyPix.drawBokehEffect(canvas, imageElement, segmentation);

Architecture

BodyPix is built around several key components:

Model Loading: Configurable loading of MobileNet or ResNet architectures with different accuracy/performance trade-offs
Segmentation Engine: Core neural network inference providing person and body part segmentation
Multi-Person Support: Instance-level segmentation supporting multiple people in a single image
Rendering Pipeline: Utilities for mask generation, visualization, and special effects like background blur
Pose Integration: Each segmentation result includes associated pose keypoints for enhanced functionality

Capabilities

Model Loading and Configuration

Load and configure BodyPix models with different architectures and performance characteristics.

function load(config?: ModelConfig): Promise<BodyPix>;

interface ModelConfig {
  architecture: 'MobileNetV1' | 'ResNet50';
  outputStride: 8 | 16 | 32;
  multiplier?: 0.50 | 0.75 | 1.0;
  modelUrl?: string;
  quantBytes?: 1 | 2 | 4;
}

Model Loading

Person Segmentation

Segment people in images and videos with semantic and instance-level segmentation options.

class BodyPix {
  segmentPerson(
    input: BodyPixInput,
    config?: PersonInferenceConfig
  ): Promise<SemanticPersonSegmentation>;
  
  segmentMultiPerson(
    input: BodyPixInput,
    config?: MultiPersonInstanceInferenceConfig
  ): Promise<PersonSegmentation[]>;
}

type BodyPixInput = ImageData | HTMLImageElement | HTMLCanvasElement | HTMLVideoElement | OffscreenCanvas | tf.Tensor3D;

Person Segmentation

Body Part Segmentation

Segment 24 distinct body parts with detailed classification for advanced applications.

class BodyPix {
  segmentPersonParts(
    input: BodyPixInput,
    config?: PersonInferenceConfig
  ): Promise<SemanticPartSegmentation>;
  
  segmentMultiPersonParts(
    input: BodyPixInput,
    config?: MultiPersonInstanceInferenceConfig
  ): Promise<PartSegmentation[]>;
}

Body Part Segmentation

Rendering and Effects

Create visual effects, masks, and overlays from segmentation results.

function toMask(
  segmentation: SemanticPersonSegmentation | PersonSegmentation[],
  foreground?: Color,
  background?: Color
): ImageData;

function drawBokehEffect(
  canvas: HTMLCanvasElement | OffscreenCanvas,
  image: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
  personSegmentation: SemanticPersonSegmentation | PersonSegmentation[],
  backgroundBlurAmount?: number,
  edgeBlurAmount?: number,
  flipHorizontal?: boolean
): void;

function blurBodyPart(
  canvas: HTMLCanvasElement | OffscreenCanvas,
  image: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
  partSegmentation: SemanticPartSegmentation | PartSegmentation[],
  bodyPartIdsToBlur?: number[],
  backgroundBlurAmount?: number,
  edgeBlurAmount?: number
): void;

Rendering and Effects

Utility Functions

Helper functions for pose manipulation, tensor operations, and coordinate transformations.

function flipPoseHorizontal(pose: Pose, imageWidth: number): Pose;

function resizeAndPadTo(
  imageTensor: tf.Tensor3D,
  [targetH, targetW]: [number, number],
  flipHorizontal?: boolean
): {
  resizedAndPadded: tf.Tensor3D,
  paddedBy: [[number, number], [number, number]]
};

Utility Functions

Core Types

interface PersonSegmentation {
  data: Uint8Array;    // Binary mask (0=background, 1=person)
  width: number;       // Mask width
  height: number;      // Mask height
  pose: Pose;         // Associated pose keypoints
}

interface SemanticPersonSegmentation {
  data: Uint8Array;    // Combined mask for all people
  width: number;       // Mask width
  height: number;      // Mask height
  allPoses: Pose[];   // All detected poses
}

interface PartSegmentation {
  data: Int32Array;    // Part IDs (0-23) or -1 for background
  width: number;       // Mask width
  height: number;      // Mask height
  pose: Pose;         // Associated pose keypoints
}

interface SemanticPartSegmentation {
  data: Int32Array;    // Combined part IDs for all people
  width: number;       // Mask width
  height: number;      // Mask height
  allPoses: Pose[];   // All detected poses
}

interface Pose {
  keypoints: Keypoint[];  // 17 body keypoints
  score: number;         // Overall pose confidence (0-1)
}

interface Keypoint {
  score: number;         // Keypoint confidence (0-1)
  position: Vector2D;    // Pixel coordinates
  part: string;         // Body part name
}

interface Vector2D {
  x: number;            // X coordinate
  y: number;            // Y coordinate
}

interface Color {
  r: number;            // Red (0-255)
  g: number;            // Green (0-255)
  b: number;            // Blue (0-255)
  a: number;            // Alpha (0-255)
}

Constants

const PART_CHANNELS: string[] = [
  'left_face',           // 0
  'right_face',          // 1
  'left_upper_arm_front', // 2
  'left_upper_arm_back',  // 3
  'right_upper_arm_front', // 4
  'right_upper_arm_back',  // 5
  'left_lower_arm_front',  // 6
  'left_lower_arm_back',   // 7
  'right_lower_arm_front', // 8
  'right_lower_arm_back',  // 9
  'left_hand',            // 10
  'right_hand',           // 11
  'torso_front',          // 12
  'torso_back',           // 13
  'left_upper_leg_front', // 14
  'left_upper_leg_back',  // 15
  'right_upper_leg_front', // 16
  'right_upper_leg_back',  // 17
  'left_lower_leg_front', // 18
  'left_lower_leg_back',  // 19
  'right_lower_leg_front', // 20
  'right_lower_leg_back',  // 21
  'left_feet',            // 22
  'right_feet'            // 23
];

const version: string;
// Library version

Workspace: tessl
Visibility: Public
Created: 7 months ago
Last updated: 3 months ago
Describes: pkg:npm/@tensorflow-models/body-pix@2.2.x
Publish Source: CLI
Badge