CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/maven-org-bytedeco--libfreenect2

JavaCPP Presets for libfreenect2 providing Java bindings for Kinect for Windows v2 device drivers

Pending
Overview
Eval results
Files

pipeline-configuration.mddocs/

Pipeline Configuration

Multiple processing pipeline implementations optimized for different hardware configurations and performance requirements. Pipelines handle the decoding of raw sensor data into usable RGB, depth, and infrared frames.

Capabilities

PacketPipeline Base Class

Abstract base class for all pipeline implementations providing common interface for RGB and depth packet processing.

/**
 * Base class for other pipeline classes.
 * Methods in this class are reserved for internal use.
 */
abstract class PacketPipeline {
    /** Default constructor */
    PacketPipeline();
    
    /** Get RGB packet parser for color data processing */
    DataCallback getRgbPacketParser();
    
    /** Get IR packet parser for depth/IR data processing */
    DataCallback getIrPacketParser();
    
    /** Get RGB packet processor instance */
    RgbPacketProcessor getRgbPacketProcessor();
    
    /** Get depth packet processor instance */
    DepthPacketProcessor getDepthPacketProcessor();
}

CPU Pipeline

CPU-based processing pipeline suitable for systems without GPU acceleration or for maximum compatibility.

/**
 * Pipeline with CPU depth processing.
 * Most compatible option but may be slower than GPU-accelerated pipelines.
 */
class CpuPacketPipeline extends PacketPipeline {
    /** Default constructor for CPU-only processing */
    CpuPacketPipeline();
}

Usage Examples:

// Create CPU pipeline for maximum compatibility
PacketPipeline pipeline = new CpuPacketPipeline();

// Open device with CPU pipeline
Freenect2Device device = freenect2.openDevice(serial, pipeline);

// CPU pipeline is automatically managed - no additional configuration needed
if (device != null) {
    System.out.println("Device opened with CPU pipeline");
    device.start();
}

OpenGL Pipeline

OpenGL-accelerated processing pipeline for improved performance on systems with OpenGL support.

/**
 * Pipeline with OpenGL depth processing.
 * Requires OpenGL support and may provide better performance than CPU pipeline.
 */
class OpenGLPacketPipeline extends PacketPipeline {
    /** Default constructor for OpenGL processing */
    OpenGLPacketPipeline();
    
    /** 
     * Constructor with OpenGL context and debug options
     * @param parent_opengl_context Existing OpenGL context to share (or null)
     * @param debug Enable debug output for troubleshooting
     */
    OpenGLPacketPipeline(Pointer parent_opengl_context, @Cast("bool") boolean debug);
}

Usage Examples:

// Create OpenGL pipeline with default settings
PacketPipeline pipeline = new OpenGLPacketPipeline();

// Create OpenGL pipeline with debug enabled
PacketPipeline debugPipeline = new OpenGLPacketPipeline(null, true);

// Open device with OpenGL pipeline
try {
    Freenect2Device device = freenect2.openDevice(serial, pipeline);
    if (device != null) {
        System.out.println("Device opened with OpenGL pipeline");
        device.start();
    }
} catch (Exception e) {
    System.err.println("OpenGL pipeline failed, falling back to CPU");
    PacketPipeline cpuPipeline = new CpuPacketPipeline();
    Freenect2Device device = freenect2.openDevice(serial, cpuPipeline);
}

Dump Pipeline

Debug pipeline that dumps raw packet data for analysis and troubleshooting.

/**
 * Pipeline that dumps packets for debugging purposes.
 * Useful for analyzing raw sensor data and troubleshooting issues.
 * Provides access to internal depth processing tables.
 */
class DumpPacketPipeline extends PacketPipeline {
    /** Constructor for dump pipeline */
    DumpPacketPipeline();
    
    /** Get depth P0 tables required for depth decoding */
    @Cast("const unsigned char*") BytePointer getDepthP0Tables(@Cast("size_t*") SizeTPointer length);
    
    /** Get depth X lookup table */
    @Const FloatPointer getDepthXTable(@Cast("size_t*") SizeTPointer length);
    
    /** Get depth Z lookup table */
    @Const FloatPointer getDepthZTable(@Cast("size_t*") SizeTPointer length);
    
    /** Get depth value lookup table */
    @Const ShortPointer getDepthLookupTable(@Cast("size_t*") SizeTPointer length);
}

Packet Processing Components

Individual components for processing different types of sensor data.

/**
 * Callback interface for packet data processing
 */
abstract class DataCallback {
    // Implementation varies by packet type
}

/**
 * RGB packet processor for color frame data
 */
abstract class RgbPacketProcessor {
    // Processes color sensor packets into RGB frames
}

/**
 * Depth packet processor for depth/IR frame data  
 */
abstract class DepthPacketProcessor {
    // Processes ToF sensor packets into depth and IR frames
}

/**
 * Container for packet pipeline components
 */
class PacketPipelineComponents {
    // Aggregates processing components
}

Pipeline Selection Guidelines

Choosing the Right Pipeline

CpuPacketPipeline:

  • Best compatibility across all systems
  • No GPU dependencies
  • Slower processing but stable
  • Recommended for: servers, headless systems, compatibility testing

OpenGLPacketPipeline:

  • Hardware-accelerated processing
  • Better performance on systems with OpenGL support
  • May have driver dependencies
  • Recommended for: desktop applications, real-time processing

DumpPacketPipeline:

  • For debugging and analysis only
  • Not suitable for production use
  • Helpful for understanding raw sensor data

Performance Considerations

// Pipeline performance comparison example
long startTime, endTime;

// Test CPU pipeline
PacketPipeline cpuPipeline = new CpuPacketPipeline();
Freenect2Device device = freenect2.openDevice(serial, cpuPipeline);
device.start();

startTime = System.currentTimeMillis();
// Capture 10 frames for timing
for (int i = 0; i < 10; i++) {
    if (listener.waitForNewFrame(frames, 1000)) {
        listener.release(frames);
    }
}
endTime = System.currentTimeMillis();
long cpuTime = endTime - startTime;

device.stop();
device._close();

// Test OpenGL pipeline
PacketPipeline glPipeline = new OpenGLPacketPipeline();
device = freenect2.openDevice(serial, glPipeline);
device.start();

startTime = System.currentTimeMillis();
// Capture same 10 frames
for (int i = 0; i < 10; i++) {
    if (listener.waitForNewFrame(frames, 1000)) {
        listener.release(frames);
    }
}
endTime = System.currentTimeMillis();
long glTime = endTime - startTime;

System.out.println("CPU Pipeline: " + cpuTime + "ms");
System.out.println("OpenGL Pipeline: " + glTime + "ms");
System.out.println("Performance gain: " + (cpuTime / (float)glTime) + "x");

Error Handling and Fallbacks

/**
 * Robust pipeline selection with fallback support
 */
public static Freenect2Device openDeviceWithBestPipeline(
        Freenect2 freenect2, String serial) {
    
    // Try pipelines in order of preference
    PacketPipeline[] pipelines = {
        new OpenGLPacketPipeline(),  // Best performance
        new CpuPacketPipeline()      // Fallback compatibility
    };
    
    String[] pipelineNames = {"OpenGL", "CPU"};
    
    for (int i = 0; i < pipelines.length; i++) {
        try {
            Freenect2Device device = freenect2.openDevice(serial, pipelines[i]);
            if (device != null) {
                System.out.println("Successfully opened device with " + 
                                 pipelineNames[i] + " pipeline");
                return device;
            }
        } catch (Exception e) {
            System.err.println(pipelineNames[i] + " pipeline failed: " + 
                             e.getMessage());
            if (i == pipelines.length - 1) {
                System.err.println("All pipelines failed!");
                return null;
            }
        }
    }
    
    return null;
}

Pipeline Lifecycle Management

public class PipelineManager {
    private Freenect2 freenect2;
    private Freenect2Device device;
    private PacketPipeline currentPipeline;
    
    public boolean initializeWithPipeline(String pipelineType) {
        try {
            // Create pipeline based on type
            switch (pipelineType.toLowerCase()) {
                case "cpu":
                    currentPipeline = new CpuPacketPipeline();
                    break;
                case "opengl":
                    currentPipeline = new OpenGLPacketPipeline();
                    break;
                case "debug":
                    currentPipeline = new DumpPacketPipeline();
                    break;
                default:
                    throw new IllegalArgumentException("Unknown pipeline: " + pipelineType);
            }
            
            // Open device with selected pipeline
            String serial = freenect2.getDefaultDeviceSerialNumber().getString();
            device = freenect2.openDevice(serial, currentPipeline);
            
            if (device == null) {
                System.err.println("Failed to open device with " + pipelineType + " pipeline");
                return false;
            }
            
            System.out.println("Initialized with " + pipelineType + " pipeline");
            return true;
            
        } catch (Exception e) {
            System.err.println("Pipeline initialization failed: " + e.getMessage());
            return false;
        }
    }
    
    public void cleanup() {
        if (device != null) {
            device.stop();
            device._close();
            device = null;
        }
        currentPipeline = null;
    }
}

Install with Tessl CLI

npx tessl i tessl/maven-org-bytedeco--libfreenect2

docs

device-management.md

frame-processing.md

index.md

logging.md

pipeline-configuration.md

registration-geometry.md

tile.json