XR Interaction Test

Test 12 suites covering XR interaction behaviors: entity discovery, ECS registration, ray interaction, poke/touch, dual-mode, cross-entity isolation, input mode switching, rapid poke cycles, audio, UI panel, and stability.

All tool calls go through npx iwsdk from the example workspace. The helper below keeps the existing MCP-style tool names, but it resolves them through iwsdk mcp inspect and then executes the matching CLI command directly.

Configuration:

• EXAMPLE_DIR: /Users/felixz/Projects/immersive-web-sdk/examples/poke
• ROOT: /Users/felixz/Projects/immersive-web-sdk

SHORTHAND: Throughout this document, MCPCALL means this shell function:

MCPCALL() {
  local tool=""
  local args=""
  local timeout=""
  while [ "$#" -gt 0 ]; do
    case "$1" in
      --tool) tool="$2"; shift 2 ;;
      --args) args="$2"; shift 2 ;;
      --timeout) timeout="$2"; shift 2 ;;
      *) echo "Unknown argument: $1" >&2; return 1 ;;
    esac
  done

  node --input-type=module - "$tool" "${args:-}" "${timeout:-}" <<'EOF'
import { spawnSync } from 'node:child_process';

const [toolName, rawArgs, timeout] = process.argv.slice(2);
const inspect = spawnSync('npx', ['iwsdk', 'mcp', 'inspect'], {
  cwd: process.cwd(),
  encoding: 'utf8',
});
if (inspect.status !== 0) {
  if (inspect.stderr) process.stderr.write(inspect.stderr);
  process.exit(inspect.status ?? 1);
}

const parsed = JSON.parse(inspect.stdout);
const tool = parsed.data.tools.find((entry) => entry.mcpName === toolName);
if (!tool) {
  console.error(`Unknown tool: ${toolName}`);
  process.exit(1);
}

const cliArgs = ['iwsdk', ...tool.cliPath.split(' ')];
if (rawArgs) cliArgs.push('--input-json', rawArgs);
if (timeout) cliArgs.push('--timeout', timeout);

const result = spawnSync('npx', cliArgs, {
  cwd: process.cwd(),
  encoding: 'utf8',
});
if (result.stdout) process.stdout.write(result.stdout);
if (result.stderr) process.stderr.write(result.stderr);
process.exit(result.status ?? 1);
EOF
}

Tool calling pattern: Every tool call is a Bash command using the MCPCALL shorthand:

MCPCALL --tool <TOOL_NAME> --args '<JSON_ARGS>' 2>/dev/null

• <TOOL_NAME> uses MCP-style names (e.g. browser_reload_page, xr_accept_session, xr_look_at). The shell helper resolves them to direct CLI commands.
• <JSON_ARGS> is a JSON object string. Omit --args if no arguments needed.
• Output is JSON on stdout. Parse it to check assertions.
• Use --timeout 20000 for operations that may take longer (reload, accept_session, animate_to, screenshot).
• Running from the example workspace (or a child directory within it) is required so npx iwsdk can resolve the nearest IWSDK app root.

IMPORTANT: Run each Bash command one at a time. Parse the JSON output and verify assertions before moving to the next command. Do NOT chain multiple MCPCALL commands together.

IMPORTANT: When the instructions say "wait N seconds", use sleep N as a separate Bash command.

Step 1: Install Dependencies

cd /Users/felixz/Projects/immersive-web-sdk/examples/poke && npm run fresh:install

Wait for this to complete before proceeding.

Step 2: Start Dev Server

Start the dev server as a background task using the Bash tool's run_in_background: true parameter:

cd /Users/felixz/Projects/immersive-web-sdk/examples/poke && npm run dev

IMPORTANT: This command MUST be run with run_in_background: true on the Bash tool — do NOT append & to the command itself.

Once the background task is launched, poll the output for Vite's ready message (up to 60s). You can also run npx iwsdk dev status from the example directory until state.running becomes true. You do not need to extract or manage the port yourself; all subsequent MCPCALL commands resolve the active runtime through the CLI.

If the server fails to start within 60 seconds, report FAIL for all suites and skip to Step 5.

Step 3: Verify Connectivity

MCPCALL --tool ecs_list_systems 2>/dev/null

This must return JSON with a list of systems. If it fails:

1. Check /tmp/iwsdk-dev-interactions.log for errors
2. Try killing and restarting the server (Step 2)
3. If it still fails, report FAIL for all suites and skip to Step 5

Step 4: Run Test Suites

Pre-test Setup

Run these commands in order:

1. MCPCALL --tool browser_reload_page --timeout 20000 2>/dev/null Then: sleep 3

2. MCPCALL --tool xr_accept_session --timeout 20000 2>/dev/null Then: sleep 2

3. MCPCALL --tool browser_get_console_logs --args '{"count":20,"level":["error"]}' 2>/dev/null Assert: No error-level logs. Warnings about audio autoplay are acceptable.

Suite 1: Entity Discovery

Discover all testable entities dynamically. These entity indices are used by all subsequent suites.

Test 1.1: Find Robot Entity

MCPCALL --tool ecs_find_entities --args '{"withComponents":["Robot"]}' 2>/dev/null

Assert: Exactly 1 entity. Save its entityIndex as <robot>.

Test 1.2: Find Panel Entity

MCPCALL --tool ecs_find_entities --args '{"withComponents":["PanelUI"]}' 2>/dev/null

Assert: Exactly 1 entity. Save its entityIndex as <panel>.

Test 1.3: Get Robot World Position

MCPCALL --tool scene_get_hierarchy --args '{"maxDepth":3}' 2>/dev/null

Find the robot's Object3D UUID (match entityIndex = <robot>). Then:

MCPCALL --tool scene_get_object_transform --args '{"uuid":"<robot-uuid>"}' 2>/dev/null

Save positionRelativeToXROrigin as <robot-pos>. Expected near (0, 0.95, -1.5).

Test 1.4: Get Panel World Position Same approach — find panel's UUID from hierarchy, query transform.

MCPCALL --tool scene_get_object_transform --args '{"uuid":"<panel-uuid>"}' 2>/dev/null

Save positionRelativeToXROrigin as <panel-pos>. Expected near (0, 1.5, -1.4).

Suite 2: ECS Registration

Test 2.1: List Systems

MCPCALL --tool ecs_list_systems 2>/dev/null

Assert these systems are present: RobotSystem, PanelSystem, InputSystem, AudioSystem, PanelUISystem.

Test 2.2: List Components

MCPCALL --tool ecs_list_components 2>/dev/null

Assert these components are registered:

• Robot
• PanelUI (with fields: config, maxWidth, maxHeight)
• AudioSource (with fields: src, loop, _loaded, _isPlaying, _playRequested)
• RayInteractable
• PokeInteractable
• ScreenSpace

Suite 3: Ray Interaction on Robot

Test 3.1: Ray Hover

MCPCALL --tool xr_look_at --args '{"device":"controller-right","target":{"x":<robot-pos.x>,"y":<robot-pos.y>,"z":<robot-pos.z>},"moveToDistance":1.0}' 2>/dev/null

Then: sleep 1

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: Hovered present, Pressed absent.

Test 3.2: Ray Select

MCPCALL --tool xr_set_select_value --args '{"device":"controller-right","value":1}' 2>/dev/null

Then: sleep 0.5

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: Both Hovered and Pressed present.

Test 3.3: Ray Release

MCPCALL --tool xr_set_select_value --args '{"device":"controller-right","value":0}' 2>/dev/null

Then: sleep 0.5

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: Hovered present, Pressed absent.

Test 3.4: Ray Unhover

MCPCALL --tool xr_look_at --args '{"device":"controller-right","target":{"x":5,"y":1.5,"z":0}}' 2>/dev/null

Then: sleep 1

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered absent.

Suite 4: Poke Interaction on Robot

The touch pointer uses a SphereIntersector with two thresholds:

• hoverRadius: 0.2m (20cm) — triggers hover
• downRadius: 0.02m (2cm) — triggers auto-select (pointerdown)

Test 4.1: Position Near Robot

MCPCALL --tool xr_set_transform --args '{"device":"controller-right","position":{"x":<robot-pos.x>,"y":<robot-pos.y>,"z":<z+0.3>},"orientation":{"pitch":0,"yaw":180,"roll":0}}' 2>/dev/null

(where <z+0.3> = <robot-pos.z> + 0.3)

Test 4.2: Slow Animate Through Robot

MCPCALL --tool xr_animate_to --args '{"device":"controller-right","position":{"x":<robot-pos.x>,"y":<robot-pos.y>,"z":<z-0.3>},"duration":2.5}' --timeout 20000 2>/dev/null

(where <z-0.3> = <robot-pos.z> - 0.3) Then: sleep 1.5

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: At least Hovered present. Pressed may also be present.

Test 4.3: Pull Back

MCPCALL --tool xr_animate_to --args '{"device":"controller-right","position":{"x":0.3,"y":1.5,"z":-0.3},"duration":0.3}' --timeout 20000 2>/dev/null

Then: sleep 0.5

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: Neither Hovered nor Pressed present.

Suite 5: Ray Interaction on Panel

Test 5.1: Ray Hover

MCPCALL --tool xr_look_at --args '{"device":"controller-right","target":{"x":<panel-pos.x>,"y":<panel-pos.y>,"z":<panel-pos.z>},"moveToDistance":0.8}' 2>/dev/null

Then: sleep 1

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered present.

Test 5.2: Click

MCPCALL --tool xr_select --args '{"device":"controller-right","duration":0.2}' 2>/dev/null

Then: sleep 0.5

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered still present.

Test 5.3: Unhover

MCPCALL --tool xr_look_at --args '{"device":"controller-right","target":{"x":5,"y":1.5,"z":0}}' 2>/dev/null

Then: sleep 1

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered absent.

Suite 6: Dual-Mode Interaction (Panel — Ray + Poke)

Test 6.1: Ray Hover from Distance

MCPCALL --tool xr_look_at --args '{"device":"controller-right","target":{"x":<panel-pos.x>,"y":<panel-pos.y>,"z":<panel-pos.z>},"moveToDistance":0.8}' 2>/dev/null

Then: sleep 1

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered present.

Test 6.2: Poke on Panel

MCPCALL --tool xr_set_transform --args '{"device":"controller-right","position":{"x":<panel-pos.x>,"y":<panel-pos.y>,"z":<pz+0.2>},"orientation":{"pitch":0,"roll":0,"yaw":0}}' 2>/dev/null

(where <pz+0.2> = <panel-pos.z> + 0.2)

MCPCALL --tool xr_animate_to --args '{"device":"controller-right","position":{"x":<panel-pos.x>,"y":<panel-pos.y>,"z":<pz-0.01>},"duration":3}' --timeout 20000 2>/dev/null

(where <pz-0.01> = <panel-pos.z> - 0.01 — stop just past the panel surface, NOT far behind it)

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: Both Hovered and Pressed present.

Test 6.3: Poke Release

MCPCALL --tool xr_animate_to --args '{"device":"controller-right","position":{"x":0.3,"y":1.5,"z":-0.3},"duration":0.3}' --timeout 20000 2>/dev/null

Then: sleep 0.5

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: Neither present.

Suite 7: Cross-Entity Isolation

Test 7.1: Only Target Entity Gets Hovered

MCPCALL --tool xr_set_transform --args '{"device":"controller-right","position":{"x":<rx+0.1>,"y":<robot-pos.y>,"z":<rz+0.3>},"orientation":{"pitch":0,"roll":0,"yaw":180}}' 2>/dev/null

(where <rx+0.1> = <robot-pos.x> + 0.1, <rz+0.3> = <robot-pos.z> + 0.3) Then: sleep 1

Check robot:

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered present on robot.

Check panel:

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["Hovered","Pressed"]}' 2>/dev/null

Assert: No interaction components on panel.

Suite 8: Input Mode Switching

Test 8.1: Hand Hover

MCPCALL --tool xr_set_input_mode --args '{"mode":"hand"}' 2>/dev/null

MCPCALL --tool xr_set_transform --args '{"device":"hand-right","position":{"x":<rx+0.1>,"y":<robot-pos.y>,"z":<rz+0.3>}}' 2>/dev/null

Then: sleep 1

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered present.

Test 8.2: Switch Back to Controllers

MCPCALL --tool xr_set_input_mode --args '{"mode":"controller"}' 2>/dev/null

MCPCALL --tool xr_set_transform --args '{"device":"controller-right","position":{"x":0.3,"y":1.5,"z":-0.3},"orientation":{"pitch":0,"roll":0,"yaw":0}}' 2>/dev/null

Then: sleep 1

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<robot>,"components":["Hovered"]}' 2>/dev/null

Assert: Hovered absent (clean transition).

Suite 9: Rapid Poke Cycles (Regression)

Test that multiple poke-release cycles all clean up properly (no stuck Pressed).

For each of 3 cycles:

1. Position at {x: <robot-pos.x>, y: <robot-pos.y>, z: <robot-pos.z> + 0.4} with yaw 180:

   MCPCALL --tool xr_set_transform --args '{"device":"controller-right","position":{"x":<robot-pos.x>,"y":<robot-pos.y>,"z":<rz+0.4>},"orientation":{"pitch":0,"yaw":180,"roll":0}}' 2>/dev/null

2. Animate through:

   MCPCALL --tool xr_animate_to --args '{"device":"controller-right","position":{"x":<robot-pos.x>,"y":<robot-pos.y>,"z":<rz-0.3>},"duration":1.5}' --timeout 20000 2>/dev/null

3. sleep 1.5, then query <robot> for ["Hovered","Pressed"]. Assert: at least Hovered or Pressed present.
4. Animate back:

   MCPCALL --tool xr_animate_to --args '{"device":"controller-right","position":{"x":<robot-pos.x>,"y":<robot-pos.y>,"z":<rz+0.5>},"duration":0.3}' --timeout 20000 2>/dev/null

5. sleep 0.5, then query <robot> for ["Hovered","Pressed"]. Assert: neither present.

All 3 cycles must pass.

Suite 10: Audio

Test 10.1: Find Audio Entities

MCPCALL --tool ecs_find_entities --args '{"withComponents":["AudioSource"]}' 2>/dev/null

Assert: At least 1 entity found. Use the first as <audio>.

Test 10.2: Verify Audio Loaded

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<audio>,"components":["AudioSource"]}' 2>/dev/null

Assert: _loaded = true, src contains chime.mp3.

Test 10.3: Trigger Playback

MCPCALL --tool ecs_set_component --args '{"entityIndex":<audio>,"componentId":"AudioSource","field":"loop","value":true}' 2>/dev/null

MCPCALL --tool ecs_set_component --args '{"entityIndex":<audio>,"componentId":"AudioSource","field":"_playRequested","value":true}' 2>/dev/null

Note: _playRequested is consumed within one frame.

Test 10.4: Verify Playback State

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<audio>,"components":["AudioSource"]}' 2>/dev/null

Assert: _isPlaying = true (loop is on).

Test 10.5: Stop Playback

MCPCALL --tool ecs_set_component --args '{"entityIndex":<audio>,"componentId":"AudioSource","field":"_stopRequested","value":true}' 2>/dev/null

Suite 11: UI Panel Verification

Test 11.1: Panel Loading

MCPCALL --tool ecs_query_entity --args '{"entityIndex":<panel>,"components":["PanelUI","PanelDocument","ScreenSpace"]}' 2>/dev/null

Assert:

• PanelUI.config contains welcome.json
• PanelUI.maxWidth approximately 0.5, PanelUI.maxHeight approximately 0.4
• PanelDocument component IS present (proves async panel loading succeeded)
• ScreenSpace component IS present

Test 11.2: Visual Confirmation

MCPCALL --tool browser_screenshot --timeout 20000 2>/dev/null

Assert: returns a screenshotPath (PNG file saved to /tmp).

Suite 12: Stability Check

MCPCALL --tool browser_get_console_logs --args '{"count":50,"level":["error","warn"]}' 2>/dev/null

Assert: No error-level logs. Warnings about AudioContext autoplay policy are acceptable. Pre-existing 404 resource errors from page load are acceptable.

Step 5: Cleanup & Results

Kill the dev server:

cd /Users/felixz/Projects/immersive-web-sdk/examples/poke && npx iwsdk dev down

Output a summary table:

| Suite                         | Result    |
|-------------------------------|-----------|
| 1. Entity Discovery           | PASS/FAIL |
| 2. ECS Registration           | PASS/FAIL |
| 3. Ray Interaction (Robot)    | PASS/FAIL |
| 4. Poke Interaction (Robot)   | PASS/FAIL |
| 5. Ray Interaction (Panel)    | PASS/FAIL |
| 6. Dual-Mode (Panel)          | PASS/FAIL |
| 7. Cross-Entity Isolation     | PASS/FAIL |
| 8. Input Mode Switching       | PASS/FAIL |
| 9. Rapid Poke Cycles          | PASS/FAIL |
| 10. Audio                     | PASS/FAIL |
| 11. UI Panel                  | PASS/FAIL |
| 12. Stability                 | PASS/FAIL |

If any suite fails, include which assertion failed and actual vs expected values.

Recovery

If at any point a transient error occurs (server crash, WebSocket timeout, connection refused, etc.) that is NOT caused by a source code bug:

1. Stop the dev server: cd /Users/felixz/Projects/immersive-web-sdk/examples/poke && npx iwsdk dev down
2. Restart: re-run Step 2 to start a fresh dev server
3. Re-run the Pre-test Setup (reload, accept session)
4. Retry the failed suite

Only give up after one retry attempt per suite. If the same suite fails twice, mark it FAIL and continue to the next suite.

Known Issues & Workarounds

Poke timing sensitivity

The slow animation in poke suites (2-2.5 seconds) is critical. The poke system uses a 2cm downRadius threshold — if the controller moves too fast, it can skip past the threshold between frames.

Audio autoplay

Browsers block audio autoplay until user gesture. The _playRequested flag may silently fail. If _isPlaying is false, this is a browser policy issue, not a bug.

One-shot flags consumed immediately

_playRequested and _stopRequested are processed and reset to false within a single frame.

Entity indices change on reload

Never cache entity indices across page reloads. Always re-discover via ecs_find_entities.

Touch pointer not enabled

toggleSubPointer('touch', true) must be called when entities are created, not just at init time.

Pressed stuck after poke pull-back

Fixed: processTouchLifecycle now dispatches pointer.up() when intersection is lost in SELECT state.