0
# Hardware Acceleration
1

2
GPU and specialized hardware acceleration support for high-performance video encoding, decoding, and processing operations with CUDA, VAAPI, DXVA2, VideoToolbox, and other acceleration APIs.
3

4
## Capabilities
5

6
### Hardware Device Management
7

8
#### Device Type Discovery
9

10
```java { .api }
11
/**
12
 * Find hardware device type by name
13
 * @param name Device type name (e.g., "cuda", "vaapi", "dxva2")
14
 * @return Hardware device type constant or AV_HWDEVICE_TYPE_NONE if not found
15
 */
16
int av_hwdevice_find_type_by_name(String name);
17

18
/**
19
 * Get hardware device type name
20
 * @param type Hardware device type constant
21
 * @return Device type name string
22
 */
23
BytePointer av_hwdevice_get_type_name(int type);
24

25
/**
26
 * Iterate through available hardware device types
27
 * @param prev Previous device type (use AV_HWDEVICE_TYPE_NONE to start)
28
 * @return Next available device type or AV_HWDEVICE_TYPE_NONE when done
29
 */
30
int av_hwdevice_iterate_types(int prev);
31
```
32

33
**Usage Example:**
34

35
```java
36
import org.bytedeco.ffmpeg.avutil.*;
37
import static org.bytedeco.ffmpeg.global.avutil.*;
38

39
// Find CUDA device type
40
int cudaType = av_hwdevice_find_type_by_name("cuda");
41
if (cudaType != AV_HWDEVICE_TYPE_NONE) {
42
    System.out.println("CUDA acceleration available");
43
}
44

45
// List all available hardware acceleration types
46
int deviceType = av_hwdevice_iterate_types(AV_HWDEVICE_TYPE_NONE);
47
while (deviceType != AV_HWDEVICE_TYPE_NONE) {
48
    BytePointer typeName = av_hwdevice_get_type_name(deviceType);
49
    System.out.println("Available: " + typeName.getString());
50
    deviceType = av_hwdevice_iterate_types(deviceType);
51
}
52
```
53

54
### Hardware Device Context Management
55

56
#### Creating Device Contexts
57

58
```java { .api }
59
/**
60
 * Allocate hardware device context for specified type
61
 * @param type Hardware device type
62
 * @return Reference to device context buffer or null on failure
63
 */
64
AVBufferRef av_hwdevice_ctx_alloc(int type);
65

66
/**
67
 * Initialize hardware device context
68
 * @param ref Reference to allocated device context
69
 * @return 0 on success, negative AVERROR on failure
70
 */
71
int av_hwdevice_ctx_init(AVBufferRef ref);
72

73
/**
74
 * Create hardware device context from device reference
75
 * @param device_ctx Pointer to store created context reference
76
 * @param type Hardware device type
77
 * @param device Device identifier (can be null for default)
78
 * @param opts Options dictionary for device creation
79
 * @param flags Creation flags (reserved, pass 0)
80
 * @return 0 on success, negative AVERROR on failure
81
 */
82
int av_hwdevice_ctx_create(@ByPtrPtr AVBufferRef device_ctx, int type,
83
                          String device, AVDictionary opts, int flags);
84
```
85

86
**Usage Example:**
87

88
```java
89
import org.bytedeco.ffmpeg.avutil.*;
90
import static org.bytedeco.ffmpeg.global.avutil.*;
91

92
// Create CUDA device context
93
AVBufferRef cudaDeviceRef = new AVBufferRef(null);
94
int result = av_hwdevice_ctx_create(cudaDeviceRef, AV_HWDEVICE_TYPE_CUDA, null, null, 0);
95

96
if (result >= 0) {
97
    System.out.println("CUDA device context created successfully");
98
    
99
    // Use device context for hardware acceleration
100
    try {
101
        // Hardware operations here
102
    } finally {
103
        av_buffer_unref(cudaDeviceRef); // Clean up when done
104
    }
105
} else {
106
    System.err.println("Failed to create CUDA context: " + result);
107
}
108
```
109

110
### Hardware Frames Management
111

112
#### Frame Context Operations
113

114
```java { .api }
115
/**
116
 * Allocate hardware frames context
117
 * @param device_ref Reference to hardware device context
118
 * @return Reference to frames context buffer or null on failure
119
 */
120
AVBufferRef av_hwframe_ctx_alloc(AVBufferRef device_ref);
121

122
/**
123
 * Initialize hardware frames context
124
 * @param ref Reference to allocated frames context
125
 * @return 0 on success, negative AVERROR on failure
126
 */
127
int av_hwframe_ctx_init(AVBufferRef ref);
128

129
/**
130
 * Get hardware frames buffer
131
 * @param frame AVFrame to receive hardware buffer
132
 * @param hwframe_ctx Hardware frames context reference
133
 * @param flags Allocation flags (reserved, pass 0)
134
 * @return 0 on success, negative AVERROR on failure
135
 */
136
int av_hwframe_get_buffer(AVFrame frame, AVBufferRef hwframe_ctx, int flags);
137
```
138

139
### Hardware Frame Transfer Operations
140

141
#### Data Transfer Methods
142

143
```java { .api }
144
/**
145
 * Transfer data from hardware frame to software frame
146
 * @param dst Destination software frame
147
 * @param src Source hardware frame
148
 * @param flags Transfer flags (reserved, pass 0)
149
 * @return 0 on success, negative AVERROR on failure
150
 */
151
int av_hwframe_transfer_data(AVFrame dst, AVFrame src, int flags);
152

153
/**
154
 * Transfer data from software frame to hardware frame
155
 * @param dst Destination hardware frame
156
 * @param src Source software frame
157
 * @param flags Transfer flags (reserved, pass 0)
158
 * @return 0 on success, negative AVERROR on failure
159
 */
160
int av_hwframe_transfer_data(AVFrame dst, AVFrame src, int flags);
161

162
/**
163
 * Map hardware frame to accessible memory
164
 * @param dst Destination frame for mapped data
165
 * @param src Source hardware frame
166
 * @param flags Mapping flags (AV_HWFRAME_MAP_*)
167
 * @return 0 on success, negative AVERROR on failure
168
 */
169
int av_hwframe_map(AVFrame dst, AVFrame src, int flags);
170
```
171

172
**Usage Example:**
173

174
```java
175
// Transfer hardware frame to software for CPU processing
176
AVFrame hwFrame = /* hardware decoded frame */;
177
AVFrame swFrame = av_frame_alloc();
178

179
// Configure software frame
180
swFrame.format(AV_PIX_FMT_YUV420P);
181
swFrame.width(hwFrame.width());
182
swFrame.height(hwFrame.height());
183
av_frame_get_buffer(swFrame, 32);
184

185
// Transfer from hardware to software
186
int result = av_hwframe_transfer_data(swFrame, hwFrame, 0);
187
if (result >= 0) {
188
    // Process software frame on CPU
189
    System.out.println("Frame transferred to CPU memory");
190
} else {
191
    System.err.println("Transfer failed: " + result);
192
}
193

194
av_frame_free(swFrame);
195
```
196

197
## Hardware Device Types
198

199
### Supported Acceleration APIs
200

201
```java { .api }
202
// Hardware device type constants
203
int AV_HWDEVICE_TYPE_NONE = 0;        // No hardware acceleration
204
int AV_HWDEVICE_TYPE_VDPAU = 1;       // NVIDIA VDPAU (Linux)
205
int AV_HWDEVICE_TYPE_CUDA = 2;        // NVIDIA CUDA
206
int AV_HWDEVICE_TYPE_VAAPI = 3;       // Intel/AMD VA-API (Linux)
207
int AV_HWDEVICE_TYPE_DXVA2 = 4;       // Microsoft DirectX Video Acceleration 2
208
int AV_HWDEVICE_TYPE_QSV = 5;         // Intel Quick Sync Video
209
int AV_HWDEVICE_TYPE_VIDEOTOOLBOX = 6; // Apple VideoToolbox (macOS/iOS)
210
int AV_HWDEVICE_TYPE_D3D11VA = 7;     // Direct3D 11 Video Acceleration
211
int AV_HWDEVICE_TYPE_DRM = 8;         // Direct Rendering Manager (Linux)
212
int AV_HWDEVICE_TYPE_OPENCL = 9;      // OpenCL acceleration
213
int AV_HWDEVICE_TYPE_MEDIACODEC = 10; // Android MediaCodec
214
int AV_HWDEVICE_TYPE_VULKAN = 11;     // Vulkan API
215
```
216

217
### Platform-Specific Acceleration
218

219
| Platform | Recommended Types | Description |
220
|----------|------------------|-------------|
221
| **Windows** | `DXVA2`, `D3D11VA`, `CUDA` | DirectX-based or NVIDIA GPU |
222
| **Linux** | `VAAPI`, `VDPAU`, `CUDA` | Intel/AMD integrated or NVIDIA |
223
| **macOS** | `VIDEOTOOLBOX` | Apple hardware acceleration |
224
| **Android** | `MEDIACODEC` | Android hardware codecs |
225
226
## Codec Hardware Acceleration Setup
227

228
### Hardware Decoder Configuration
229

230
```java { .api }
231
/**
232
 * Get codec hardware configuration at specified index
233
 * @param codec Target codec
234
 * @param index Configuration index (0-based)
235
 * @return Hardware configuration or null if not available
236
 */
237
AVCodecHWConfig avcodec_get_hw_config(AVCodec codec, int index);
238
```
239

240
**Usage Example:**
241

242
```java
243
import org.bytedeco.ffmpeg.avcodec.*;
244
import static org.bytedeco.ffmpeg.global.avcodec.*;
245

246
// Find H.264 decoder with hardware support
247
AVCodec h264Decoder = avcodec_find_decoder(AV_CODEC_ID_H264);
248

249
// Check hardware configurations
250
AVCodecHWConfig hwConfig;
251
int configIndex = 0;
252
while ((hwConfig = avcodec_get_hw_config(h264Decoder, configIndex)) != null) {
253
    int deviceType = hwConfig.device_type();
254
    int pixelFormat = hwConfig.pix_fmt();
255
    
256
    System.out.println("Hardware config " + configIndex + ":");
257
    System.out.println("  Device type: " + av_hwdevice_get_type_name(deviceType).getString());
258
    System.out.println("  Pixel format: " + pixelFormat);
259
    
260
    configIndex++;
261
}
262
```
263

264
### Setting Up Hardware Decoding
265

266
```java
267
import org.bytedeco.ffmpeg.avcodec.*;
268
import org.bytedeco.ffmpeg.avutil.*;
269
import static org.bytedeco.ffmpeg.global.avcodec.*;
270
import static org.bytedeco.ffmpeg.global.avutil.*;
271

272
// 1. Create hardware device context
273
AVBufferRef hwDeviceCtx = new AVBufferRef(null);
274
int result = av_hwdevice_ctx_create(hwDeviceCtx, AV_HWDEVICE_TYPE_CUDA, null, null, 0);
275
if (result < 0) {
276
    throw new RuntimeException("Failed to create hardware device context");
277
}
278

279
// 2. Set up codec context
280
AVCodec decoder = avcodec_find_decoder(AV_CODEC_ID_H264);
281
AVCodecContext codecCtx = avcodec_alloc_context3(decoder);
282

283
// 3. Configure hardware acceleration
284
codecCtx.hw_device_ctx(av_buffer_ref(hwDeviceCtx));
285

286
// 4. Open decoder with hardware acceleration
287
result = avcodec_open2(codecCtx, decoder, (PointerPointer)null);
288
if (result < 0) {
289
    throw new RuntimeException("Failed to open hardware decoder");
290
}
291

292
try {
293
    // Decode frames - output will be in GPU memory
294
    AVFrame hwFrame = av_frame_alloc();
295
    
296
    // Send packet for hardware decoding
297
    avcodec_send_packet(codecCtx, packet);
298
    
299
    // Receive hardware-decoded frame
300
    while (avcodec_receive_frame(codecCtx, hwFrame) >= 0) {
301
        // hwFrame contains GPU-accelerated decoded data
302
        // Transfer to CPU if needed for processing
303
        AVFrame swFrame = av_frame_alloc();
304
        av_hwframe_transfer_data(swFrame, hwFrame, 0);
305
        
306
        // Process swFrame on CPU or keep hwFrame on GPU
307
        
308
        av_frame_free(swFrame);
309
    }
310
    
311
    av_frame_free(hwFrame);
312
} finally {
313
    avcodec_free_context(codecCtx);
314
    av_buffer_unref(hwDeviceCtx);
315
}
316
```
317

318
## Frame Mapping Operations
319

320
### Memory Access Flags
321

322
```java { .api }
323
// Frame mapping flags for av_hwframe_map()
324
int AV_HWFRAME_MAP_READ = 1 << 0;      // Map for reading
325
int AV_HWFRAME_MAP_WRITE = 1 << 1;     // Map for writing
326
int AV_HWFRAME_MAP_OVERWRITE = 1 << 2; // Overwrite existing data
327
int AV_HWFRAME_MAP_DIRECT = 1 << 3;    // Direct mapping (avoid copies)
328
```
329

330
**Usage Example:**
331

332
```java
333
// Map hardware frame for read access
334
AVFrame mappedFrame = av_frame_alloc();
335
int result = av_hwframe_map(mappedFrame, hwFrame, AV_HWFRAME_MAP_READ);
336

337
if (result >= 0) {
338
    // mappedFrame provides CPU-accessible pointers to GPU data
339
    // Process mapped data without full transfer
340
    
341
    av_frame_unref(mappedFrame); // Unmap when done
342
}
343
av_frame_free(mappedFrame);
344
```
345

346
## Performance Considerations
347

348
### Best Practices
349

350
1. **Keep Data on GPU**: Minimize transfers between GPU and CPU memory
351
2. **Batch Operations**: Process multiple frames before transferring
352
3. **Use Appropriate Formats**: Choose GPU-native pixel formats when possible
353
4. **Pool Frames**: Reuse hardware frame buffers to avoid allocation overhead
354

355
### Common Pixel Formats for Hardware
356

357
```java
358
// Hardware-optimized pixel formats
359
AV_PIX_FMT_CUDA     // NVIDIA CUDA format
360
AV_PIX_FMT_VAAPI    // Intel VA-API format  
361
AV_PIX_FMT_DXVA2_VLD // DirectX Video Acceleration
362
AV_PIX_FMT_VIDEOTOOLBOX // Apple VideoToolbox format
363
```
364

365
## Error Handling
366

367
Hardware acceleration can fail due to driver issues, insufficient GPU memory, or unsupported configurations:
368

369
```java
370
// Always check for hardware acceleration support
371
int deviceType = av_hwdevice_find_type_by_name("cuda");
372
if (deviceType == AV_HWDEVICE_TYPE_NONE) {
373
    System.err.println("CUDA not available, falling back to software decoding");
374
    // Use software decoder as fallback
375
}
376

377
// Handle device creation failures
378
AVBufferRef deviceCtx = new AVBufferRef(null);
379
int result = av_hwdevice_ctx_create(deviceCtx, deviceType, null, null, 0);
380
if (result < 0) {
381
    System.err.println("Hardware device creation failed: " + result);
382
    // Implement software fallback
383
}
384
```
385

386
## Integration with Encoding/Decoding Pipeline
387

388
Hardware acceleration integrates seamlessly with the standard FFmpeg encode/decode workflow, providing significant performance improvements for supported codecs and hardware configurations.