0
# VR and Input Handling
1

2
Virtual reality support, keyboard and mouse input handling for interactive simulations, teleoperation, and user interfaces. Essential for immersive simulation experiences and human-in-the-loop applications.
3

4
## Capabilities
5

6
### VR Support
7

8
```python { .api }
9
def getVREvents(deviceTypeFilter=None, physicsClientId=0):
10
    """
11
    Get VR controller events and state.
12
    
13
    Args:
14
        deviceTypeFilter (int, optional): Filter by device type
15
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
16
    
17
    Returns:
18
        list: VR events with controller positions, orientations, and button states
19
    """
20

21
def setVRCameraState(cameraDistance, yaw, pitch, cameraTargetPosition, physicsClientId=0):
22
    """
23
    Set VR camera properties.
24
    
25
    Args:
26
        cameraDistance (float): Distance from target
27
        yaw (float): Yaw angle in degrees
28
        pitch (float): Pitch angle in degrees
29
        cameraTargetPosition (list): Target position [x, y, z]
30
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
31
    """
32
```
33

34
### Input Events
35

36
```python { .api }
37
def getKeyboardEvents(physicsClientId=0):
38
    """
39
    Get keyboard input events.
40
    
41
    Args:
42
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
43
    
44
    Returns:
45
        dict: Dictionary of keyboard events with key codes and states
46
              Format: {keycode: keystate} where keystate includes:
47
              - wasTriggered: whether key was pressed this frame
48
              - isPressed: whether key is currently held down
49
    """
50

51
def getMouseEvents(physicsClientId=0):
52
    """
53
    Get mouse input events.
54
    
55
    Args:
56
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
57
    
58
    Returns:
59
        list: List of mouse events containing:
60
              - eventType: type of mouse event (MOUSE_MOVE, MOUSE_BUTTON, etc.)
61
              - mousePosX, mousePosY: mouse position
62
              - buttonIndex: mouse button index
63
              - buttonState: button state (pressed/released)
64
    """
65
```
66

67
### Utility Functions
68

69
```python { .api }
70
def getAPIVersion():
71
    """
72
    Get PyBullet API version.
73
    
74
    Returns:
75
        int: API version number
76
    """
77

78
def isNumpyEnabled():
79
    """
80
    Check if NumPy support is enabled.
81
    
82
    Returns:
83
        bool: True if NumPy is available for array operations
84
    """
85

86
def setAdditionalSearchPath(path):
87
    """
88
    Add additional search path for loading URDF and other data files.
89
    
90
    Args:
91
        path (str): Directory path to add to search paths
92
    """
93

94
def setTimeOut(timeOutInSeconds, physicsClientId=0):
95
    """
96
    Set timeout for physics simulation operations.
97
    
98
    Args:
99
        timeOutInSeconds (float): Timeout duration in seconds
100
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
101
    """
102
```
103

104
### Plugin System
105

106
```python { .api }
107
def loadPlugin(pluginPath, postFix="", physicsClientId=0):
108
    """
109
    Load physics engine plugin.
110
    
111
    Args:
112
        pluginPath (str): Path to plugin library
113
        postFix (str, optional): Plugin name postfix. Defaults to "".
114
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
115
    
116
    Returns:
117
        int: Plugin unique ID
118
    """
119

120
def unloadPlugin(pluginUniqueId, physicsClientId=0):
121
    """
122
    Unload physics engine plugin.
123
    
124
    Args:
125
        pluginUniqueId (int): Plugin unique ID from loadPlugin
126
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
127
    """
128

129
def executePluginCommand(pluginUniqueId, textArgument="", intArgs=None, floatArgs=None, physicsClientId=0):
130
    """
131
    Execute command in loaded plugin.
132
    
133
    Args:
134
        pluginUniqueId (int): Plugin unique ID
135
        textArgument (str, optional): Text argument for plugin. Defaults to "".
136
        intArgs (list, optional): Integer arguments for plugin
137
        floatArgs (list, optional): Float arguments for plugin
138
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
139
    
140
    Returns:
141
        int: Plugin execution result
142
    """
143
```
144

145
### Advanced Functions
146

147
```python { .api }
148
def vhacd(fileName, fileNameOut, alpha=0.04, resolution=100000, depth=20, concavity=0.025, planeDownsampling=4, convexhullDownsampling=4, pca=0, mode=0, maxNumVerticesPerCH=64, minVolumePerCH=0.0001):
149
    """
150
    Perform volumetric hierarchical approximate convex decomposition (V-HACD).
151
    
152
    Args:
153
        fileName (str): Input mesh file path
154
        fileNameOut (str): Output file path for decomposed mesh
155
        alpha (float, optional): Controls surface vs volume sampling. Defaults to 0.04.
156
        resolution (int, optional): Voxel resolution. Defaults to 100000.
157
        depth (int, optional): Maximum number of clipping planes. Defaults to 20.
158
        concavity (float, optional): Maximum allowed concavity. Defaults to 0.025.
159
        planeDownsampling (int, optional): Plane downsampling rate. Defaults to 4.
160
        convexhullDownsampling (int, optional): Convex hull downsampling. Defaults to 4.
161
        pca (int, optional): Enable/disable PCA. Defaults to 0.
162
        mode (int, optional): Decomposition mode. Defaults to 0.
163
        maxNumVerticesPerCH (int, optional): Max vertices per convex hull. Defaults to 64.
164
        minVolumePerCH (float, optional): Min volume per convex hull. Defaults to 0.0001.
165
    
166
    Returns:
167
        list: List of decomposed convex hulls
168
    """
169

170
def submitProfileTiming(name, physicsClientId=0):
171
    """
172
    Submit profile timing information.
173
    
174
    Args:
175
        name (str): Profile timing name/label
176
        physicsClientId (int, optional): Physics client ID. Defaults to 0.
177
    """
178
```
179

180
## Usage Examples
181

182
### Keyboard and Mouse Input
183

184
```python
185
import pybullet as p
186
import time
187

188
p.connect(p.GUI)
189
p.loadURDF("plane.urdf")
190
cube_id = p.loadURDF("cube_small.urdf", [0, 0, 1])
191

192
print("Use WASD keys to move the cube, mouse to look around")
193
print("Press ESC to exit")
194

195
cube_pos = [0, 0, 1]
196

197
while True:
198
    # Get keyboard events
199
    keys = p.getKeyboardEvents()
200
    
201
    # Handle key presses
202
    if ord('w') in keys and keys[ord('w')] & p.KEY_IS_DOWN:
203
        cube_pos[1] += 0.01
204
    if ord('s') in keys and keys[ord('s')] & p.KEY_IS_DOWN:
205
        cube_pos[1] -= 0.01
206
    if ord('a') in keys and keys[ord('a')] & p.KEY_IS_DOWN:
207
        cube_pos[0] -= 0.01
208
    if ord('d') in keys and keys[ord('d')] & p.KEY_IS_DOWN:
209
        cube_pos[0] += 0.01
210
    
211
    # Check for ESC key to exit
212
    if p.B3G_ESCAPE in keys and keys[p.B3G_ESCAPE] & p.KEY_WAS_TRIGGERED:
213
        break
214
    
215
    # Update cube position
216
    p.resetBasePositionAndOrientation(cube_id, cube_pos, [0, 0, 0, 1])
217
    
218
    # Get mouse events
219
    mouse_events = p.getMouseEvents()
220
    for event in mouse_events:
221
        if event[0] == p.MOUSE_MOVE_EVENT:
222
            print(f"Mouse moved to: {event[1]}, {event[2]}")
223
    
224
    p.stepSimulation()
225
    time.sleep(1./60.)
226

227
p.disconnect()
228
```
229

230
### VR Support
231

232
```python
233
import pybullet as p
234
import time
235

236
# Connect with VR support (requires VR headset)
237
try:
238
    p.connect(p.SHARED_MEMORY)  # or p.GUI with VR enabled
239
    
240
    # Load scene
241
    p.loadURDF("plane.urdf")
242
    robot_id = p.loadURDF("r2d2.urdf", [0, 0, 1])
243
    
244
    # Set up VR camera
245
    p.setVRCameraState(
246
        cameraDistance=2.0,
247
        yaw=0,
248
        pitch=-30,
249
        cameraTargetPosition=[0, 0, 1]
250
    )
251
    
252
    print("VR mode active - use VR controllers to interact")
253
    
254
    for i in range(1000):
255
        # Get VR controller events
256
        vr_events = p.getVREvents()
257
        
258
        for event in vr_events:
259
            # Process VR controller input
260
            # event contains controller position, orientation, button states
261
            controller_pos = event[1]  # Controller position
262
            controller_orn = event[2]  # Controller orientation
263
            buttons = event[6]         # Button states
264
            
265
            # Example: Use trigger to apply force
266
            if buttons[33] & p.VR_BUTTON_IS_DOWN:  # Trigger pressed
267
                p.applyExternalForce(
268
                    robot_id, -1, [0, 0, 5], controller_pos, p.WORLD_FRAME
269
                )
270
        
271
        p.stepSimulation()
272
        time.sleep(1./90.)  # VR typically runs at 90Hz
273
        
274
except Exception as e:
275
    print(f"VR not available: {e}")
276
    # Fall back to regular GUI mode
277
    p.connect(p.GUI)
278
```
279

280
### Plugin System Usage
281

282
```python
283
import pybullet as p
284

285
p.connect(p.GUI)
286

287
# Load a custom physics plugin
288
try:
289
    plugin_id = p.loadPlugin("custom_physics_plugin.so")
290
    print(f"Loaded plugin with ID: {plugin_id}")
291
    
292
    # Execute plugin command
293
    result = p.executePluginCommand(
294
        plugin_id,
295
        textArgument="initialize",
296
        intArgs=[1, 2, 3],
297
        floatArgs=[0.1, 0.2, 0.3]
298
    )
299
    print(f"Plugin command result: {result}")
300
    
301
    # Use plugin during simulation...
302
    
303
    # Unload plugin when done
304
    p.unloadPlugin(plugin_id)
305
    print("Plugin unloaded")
306
    
307
except Exception as e:
308
    print(f"Plugin loading failed: {e}")
309
```
310

311
### Utility Functions
312

313
```python
314
import pybullet as p
315
import pybullet_data
316

317
# Check API version and NumPy support
318
print(f"PyBullet API version: {p.getAPIVersion()}")
319
print(f"NumPy enabled: {p.isNumpyEnabled()}")
320

321
# Set up data search paths
322
p.setAdditionalSearchPath(pybullet_data.getDataPath())
323
p.setAdditionalSearchPath("./custom_models/")
324

325
# Connect and configure timeout
326
p.connect(p.GUI)
327
p.setTimeOut(10.0)  # 10 second timeout for operations
328

329
# Load objects using search paths
330
plane_id = p.loadURDF("plane.urdf")  # Found via search path
331
robot_id = p.loadURDF("r2d2.urdf")   # Found via search path
332

333
print("Simulation ready with configured search paths and timeout")
334
```
335

336
## Input Constants
337

338
```python
339
# Keyboard key constants
340
p.KEY_IS_DOWN        # Key is currently pressed
341
p.KEY_WAS_TRIGGERED  # Key was just pressed this frame
342
p.KEY_WAS_RELEASED   # Key was just released this frame
343

344
# Special keys
345
p.B3G_ESCAPE        # Escape key
346
p.B3G_RETURN        # Enter key
347
p.B3G_SPACE         # Space bar
348
p.B3G_LEFT_ARROW    # Left arrow key
349
p.B3G_RIGHT_ARROW   # Right arrow key
350
p.B3G_UP_ARROW      # Up arrow key
351
p.B3G_DOWN_ARROW    # Down arrow key
352

353
# Mouse events
354
p.MOUSE_MOVE_EVENT   # Mouse movement
355
p.MOUSE_BUTTON_EVENT # Mouse button press/release
356

357
# VR button constants  
358
p.VR_BUTTON_IS_DOWN     # VR button pressed
359
p.VR_BUTTON_WAS_TRIGGERED # VR button just pressed
360
p.VR_BUTTON_WAS_RELEASED  # VR button just released
361
```
362

363
## Best Practices
364

365
1. **Event Processing** - Check input events every frame to avoid missing user interactions
366
2. **Key State Checking** - Use appropriate key state flags (IS_DOWN vs WAS_TRIGGERED) for different behaviors
367
3. **VR Frame Rate** - Maintain 90Hz for VR applications to prevent motion sickness
368
4. **Plugin Safety** - Always unload plugins properly to prevent memory leaks
369
5. **Search Path Management** - Set search paths early in your application initialization
370
6. **Timeout Configuration** - Set reasonable timeouts for network-based physics clients
371
7. **Input Validation** - Validate plugin commands and file paths before execution