SmartHome Video Anomaly Benchmark

Benchmark suite evaluating VLM models on video anomaly detection across 7 smart home categories — based on the SmartHome-Bench dataset (1,203 YouTube smart home camera clips).

Key differentiator from HomeSec-Bench: This benchmark requires video understanding (multi-frame sampling), not single-frame analysis.

Setup

Requires npm install + system dependencies. The deployment agent should:

1. Check for yt-dlp and ffmpeg in PATH
2. Run npm install in the skill directory

Entry script: scripts/run-benchmark.cjs

Verification

node scripts/run-benchmark.cjs --help

Quick Start

As an Aegis Skill (automatic)

When spawned by Aegis, configuration is injected via environment variables. The benchmark downloads video clips, samples frames, evaluates with VLM, and generates an HTML report.

Standalone

# Run with local VLM (subset mode, 50 videos)
node scripts/run-benchmark.cjs --vlm http://localhost:5405

# Quick test with 10 videos
node scripts/run-benchmark.cjs --vlm http://localhost:5405 --max-videos 10

# Full benchmark (all curated clips)
node scripts/run-benchmark.cjs --vlm http://localhost:5405 --mode full

# Filter by category
node scripts/run-benchmark.cjs --vlm http://localhost:5405 --categories "Wildlife,Security"

# Skip download (re-evaluate cached videos)
node scripts/run-benchmark.cjs --vlm http://localhost:5405 --skip-download

# Skip report auto-open
node scripts/run-benchmark.cjs --vlm http://localhost:5405 --no-open

Configuration

Environment Variables (set by Aegis)

Note: This is a VLM-only benchmark. An LLM gateway is not required.

User Configuration (config.yaml)

This skill includes a config.yaml that defines user-configurable parameters. Aegis parses this at install time and renders a config panel in the UI. Values are delivered via AEGIS_SKILL_PARAMS.

CLI Arguments (standalone fallback)

Protocol

Aegis → Skill (env vars)

AEGIS_VLM_URL=http://localhost:5405
AEGIS_SKILL_ID=smarthome-bench
AEGIS_SKILL_PARAMS={}

Skill → Aegis (stdout, JSON lines)

{"event": "ready", "model": "SmolVLM2-2.2B", "system": "Apple M3"}
{"event": "suite_start", "suite": "Wildlife"}
{"event": "test_result", "suite": "Wildlife", "test": "smartbench_0003", "status": "pass", "timeMs": 4500}
{"event": "suite_end", "suite": "Wildlife", "passed": 12, "failed": 3}
{"event": "complete", "passed": 78, "total": 105, "timeMs": 480000, "reportPath": "/path/to/report.html"}

Human-readable output goes to stderr (visible in Aegis console tab).

Test Suites (7 Categories)

Each clip is evaluated for binary anomaly detection: the VLM predicts normal (0) or abnormal (1), compared against expert annotations.

Metrics

Per-category and overall:

• Accuracy — correct predictions / total
• Precision — true positives / predicted positives
• Recall — true positives / actual positives
• F1-Score — harmonic mean of precision & recall
• Confusion Matrix — TP, FP, TN, FN breakdown

Results

Results are saved to ~/.aegis-ai/smarthome-bench/ as JSON. An HTML report with per-category breakdown, confusion matrix, and model comparison is auto-generated.

Requirements

• Node.js ≥ 18
• npm install (for openai SDK dependency)
• yt-dlp (video download from YouTube)
• ffmpeg (frame extraction from video clips)
• Running VLM server (must support multi-image input)

Citation

Based on SmartHome-Bench: A Comprehensive Benchmark for Video Anomaly Detection in Smart Homes Using Multi-Modal Foundation Models.