Video Agent - Short-form, Explainer & Demo Router

Scheduling

Goal

Generate finished .mp4 videos through a key-optional, 3-tier (CLI-first / MCP / guided) provider router while preserving deterministic asset buses (script -> timing -> render-spec), reproducible manifests, cost controls, and capture-path safety.

Intent signature

• User asks for a short-form video, shorts/reels clip, TikTok/YouTube Short, explainer, demo, walkthrough, or screencast.
• User wants a topic, README, code, or data turned into a narrated, captioned video.
• Another skill needs shared video-generation infrastructure (script -> assets -> render).

When to use

• Generating short-form video (shorts / reels) from a topic or brief (--mode shorts, 9:16)
• Generating an explainer from a README, code, or data set (--mode explainer, 16:9 / 9:16)
• Producing a demo / walkthrough from a screen capture file (--mode demo --source file, 16:9)
• Supervised headed web-app capture of any URL (--mode demo --source web --url <url>) — a human drives the on-screen flow; the tool only opens a headed browser and records. Example categories are equal and illustrative only: demo, walkthrough, onboarding clip, bug repro, app-review screencast.
• Re-rendering an existing run deterministically from render-spec.json
• Other skills needing video-generation infrastructure (shared invocation via --format json)

When NOT to use

• Generating a single still image -> use oma-image
• Generating a slide deck / presentation -> use oma-slide (this skill calls it internally for explainer frames)
• Generating speech audio only (no video) -> use oma-voice
• Non-linear video editing of an existing finished mp4 -> out of scope (OpenCut-MCP deferred)
• Supervised headed web capture is in-scope (--source web); live streaming is out of scope
• Interactive HTML explainer document (not a video) -> use oma-explainer

Expected inputs

• A brief (topic / README path / data) plus optional mode, aspect, locale, captions, visual, voice, music, duration, compositor, capture path, seed
• For demo --source file: a screen-capture file path (--capture) or Cap availability
• For demo --source web: a target --url (any URL — local/staging/prod), optional --device/--ready-selector/--show-cursor/--polish/--capture-timeout; capture size is derived from --aspect/--device (no hardcoded size); a resolvable Playwright + an interactive TTY (else the run falls back to the guided protocol)
• Authentication/environment state for oma-voice (Voicebox MCP), oma-image vendors, and optional Pexels / Pixelle keys

Expected outputs

• A run directory under .agents/results/videos/<timestamp>-<shortid>-<mode>/
• Deterministic asset bus: script.json, timing.json, render-spec.json
• audio/, visuals/, captions.srt / captions.vtt, the rendered <mode>-<slug>.mp4
• manifest.json with providers, asset hashes, cost breakdown, and exit code

outputs:
  - name: video
    description: rendered <mode>-<slug>.mp4 in the run directory
    artifact: ".agents/results/videos/*/*.mp4"
    required: true
  - name: manifest
    description: reproducibility record (providers, asset sha256 hashes, cost breakdown, exit code)
    artifact: ".agents/results/videos/*/manifest.json"
    required: true
  - name: render-spec
    description: deterministic compute boundary consumed by `oma video render <runDir>`
    artifact: ".agents/results/videos/*/render-spec.json"
    required: true
  - name: captions
    description: key-free caption tracks aligned to timing.json
    artifact: ".agents/results/videos/*/captions.srt"
    required: false

Dependencies

• oma video generate CLI + central error module (exit codes aligned with oma search fetch)
• oma-voice (Voicebox MCP), oma-image, oma-slide as key-free fallback providers
• Vendored Remotion project at resources/remotion/ (compositor)
• resources/vendor-matrix.md, resources/execution-protocol.md, resources/prompt-tips.md, config/video-config.yaml

Control-flow features

• Branches by mode (shorts / explainer / demo), aspect, visual strategy, provider availability, cost threshold, capture requirement, and path safety
• Runs a per-capability fallback chain (real key/resource -> key-free fallback) per backend rule 11
• Reads briefs/captures and writes assets, render-spec, and manifests
• Calls external resources: Voicebox MCP, oma-image vendors, Remotion toolchain, optional Pexels / Pixelle

Structural Flow

Entry

1. Validate that the brief carries enough mode/topic signal (or infer the mode from keywords).
2. For demo, confirm a capture path exists (or Cap is available); otherwise enter the guided protocol.
3. Resolve defaults from config/video-config.yaml -> env vars -> CLI flags; check output path safety and limits.

Scenes

1. PREPARE: Resolve mode/aspect/locale, clarify or amplify the brief, choose the visual + compositor strategy.
2. ACQUIRE: Probe provider availability (voice / visual / caption / compositor), validate capture path, check cost.
3. ACT: Run the mode pipeline — script -> (voice ∥ visuals ∥ captions) -> render-spec -> compositor render.
4. VERIFY: Validate every asset-bus schema, manifest hashes, exit code, and the output mp4.
5. FINALIZE: Return the run-dir path, the mp4 path, and any provider/coverage warnings.

Transitions

• If the brief lacks a clear mode, infer from keywords (shorts/reels -> shorts; README/code -> explainer; capture -> demo) and show the user the inferred plan before generating.
• If the selected visual provider key is absent (Pexels / Pixelle), fall through the chain to the key-free oma-image stills + Ken Burns fallback and annotate coverage.
• If demo --source web has a --url, dispatch the headed web-capture path (human-driven flow, ENTER to stop); if Playwright is unresolvable OR there is no interactive TTY, fall back to the guided protocol (no hang).
• If demo --source file has no capture and Cap is unavailable, emit the guided capture protocol and stop (exit code maps to capture-required).
• If estimated cost (Pixelle / RunningHub credits) exceeds the guardrail, require confirmation unless bypassed.

Failure and recovery

• If a provider is unavailable, try the next provider in the capability's order; only chain exhaustion is a stage failure.
• If the Remotion toolchain is not bootstrapped, point the user to oma video doctor --install (one-time: deps + headless shell + Pretendard font); fall back to the MPT compositor where applicable (MPT itself needs a one-time oma video doctor --install-mpt).
• If Voicebox MCP is down, fall back to estimated timing (still produces captions). A whisper.cpp hop between the two is reserved but not yet wired (TODO(oma-deferred): whisper-cpp).
• If the brief locale is non-source, translate via oma-translator (key-free); if absent, warn and keep source text.

Exit

• Success: <mode>-<slug>.mp4 and manifest.json exist in the run directory; all schemas validate.
• Partial success: video renders with a key-free fallback in place of a paid provider; coverage is annotated in warnings.
• Failure: no video is produced and the route/cost/capture/auth/safety blocker is explicit in the exit code + manifest.

Logical Operations

Actions

Tools and instruments

• oma video generate, oma video doctor, oma video list-providers, oma video render
• Provider adapters: AgentScript, oma-voice, oma-image, oma-slide, Pexels, Pixelle, oma-captions, Cap, Remotion, MPT
• Vendored Remotion project (resources/remotion/), prompt tips, vendor matrix, video config

Canonical command path

oma video doctor
oma video generate "<brief>" --mode shorts --aspect auto --captions tiktok --format json

Explainer from a README, with a deterministic seed:

oma video generate "explain this project" --mode explainer --aspect 16:9 --seed 42 --out ./out

Demo from a screen capture:

oma video generate "feature walkthrough" --mode demo --capture <absolute-path>.mp4

Deterministic re-render from an existing run:

oma video render .agents/results/videos/20260603-143052-ab12cd-shorts

Resource scope

Preconditions

• Brief carries enough signal for the mode, or the user approves the inferred/amplified plan.
• Output path is inside $PWD (or --allow-external-out is set).
• For demo --source file: the capture path exists, is absolute/$PWD-guarded, and is a valid format.
• For demo --source web: a --url is supplied (else SchemaValidationError); a resolvable Playwright + an interactive TTY exist (else the run falls back to the guided protocol).
• Required provider availability holds for the chosen (non-fallback) path, or the fallback is acceptable.

Effects and side effects

• Creates a run directory with assets, render-spec, captions, mp4, and manifest.
• oma-voice plays narration on the speakers as a side effect of synthesis.
• May call paid or rate-limited providers (Pexels / Pixelle / RunningHub) only when keys are present.

Guardrails

1. Clarify or infer before invoking: if the mode/topic is ambiguous, infer the mode from keywords and show the user the plan, or ask. Do NOT silently render from a vague brief. See Clarification Protocol below.
2. Key-optional dispatch (backend rule 11): every external capability has a real (key/resource) path AND a key-free fallback. Paid providers (Pexels, Pixelle) auto-enable only when their env key is present; otherwise the chain falls through to oma-image stills + Ken Burns. Default providers (oma-voice local, oma-image, oma-slide, Remotion) are key-free, so auto-triggering on keywords is safe.
3. Cost guardrail: confirm before runs whose estimated cost is >= cost.guardrail_usd ($0.20, configurable) or --max-usd. --yes / OMA_VIDEO_YES=1 bypass. Local/free paths carry zero cost.
4. Path safety: output paths outside $PWD require --allow-external-out. --capture is absolutized, $PWD-guarded, and format-validated; external assets are copied into the run dir and hashed (no URL refs).
5. Cancellable: SIGINT/SIGTERM aborts in-flight provider calls, the render, and the orchestrator.
6. Deterministic outputs: render-spec.json + asset files (+ seed + embedded Pretendard font) are the determinism boundary. oma video doctor --install fetches the Pretendard woff2 once; if offline, the render gracefully falls back to system fonts, and byte-identical output across machines is only guaranteed once the font is present. Re-rendering the same render-spec is byte-stable; OMA_VIDEO_MOCK=1 replays golden fixtures.
7. Limits: limits.max_duration_sec = 180, limits.max_scenes = 40 (wall-time + memory bound).
8. Community-MCP consent: Pixelle-MCP is off by default and requires one-time explicit consent + source review before connecting; RunningHub credits gate on --max-usd.
9. Demo is human-in-the-loop: capture is performed by a human. For --source file the skill guides but does not screen-record autonomously; for --source web the tool only opens a headed browser and records while the human drives the entire on-screen flow (interactive ENTER to stop). The mechanism prescribes nothing about what the flow is or what the recording is for.
10. Web-capture security: NO credential automation of any kind — if a flow needs a login, a human performs it. The driver runs as a subprocess under the resolved Playwright (never imported into the CLI). The --url and any query tokens are masked in logs and in manifest.json; credentials are never stored or printed. Recording and all outputs are confined to the run dir. On-screen sensitive input is captured as-is — the user controls the flow. Multi-page navigation (popup / new tab / redirect) is recorded generically, with no assumption about the flow's shape.
11. Web capture is key-optional + non-blocking: web capture is the real branch; the guided protocol is the fallback when Playwright is unresolvable OR there is no interactive TTY (CI / -y / no stdin) — the run falls back to guided and never hangs. Live capture is outside the determinism boundary, so the manifest records nondeterministic: true.
12. Headed capture needs a display: web capture launches a headed Chromium so the human can drive the flow. On display-less hosts (CI / Linux without X), pass --capture-stop duration:<sec>|selector:<css> — the driver then runs headless (resources/playwright/record.mjs --headless 1) — or expect a capture error / guided fallback.
13. Run-dir retention: .agents/results/videos/<run>/ accumulates one directory per run (assets + mp4 + manifest) and is never auto-pruned; the user deletes old run directories manually.
14. Exit codes align with oma search fetch (0 ok, 1 generic, 2 safety, 3 not-found, 4 invalid-input, 5 auth-required, 6 timeout).

Clarification Protocol

Before invoking oma video generate, the calling agent runs this checklist. If any answer is "no / unknown", clarify or infer-and-confirm with the user first.

Required signal (must be present or inferable):

• Mode: shorts / explainer / demo? Infer from keywords (shorts/reels/쇼츠/릴스 -> shorts; README/code/data/explain/설명 -> explainer; demo/walkthrough/capture/데모 -> demo).
• Topic / source: what is the video about? (a topic, a README/code path, a capture file, or — for demo --source web — a --url)
• For demo: --source file (a --capture path) or --source web (a --url)? For --source web, state up front that a human drives the on-screen flow and that the tool never automates any login.

Strongly recommended (ask if absent AND not inferable):

• Aspect: 9:16 (shorts/reels), 16:9 (explainer/demo), 1:1, or auto (snaps to the mode default).
• Locale: narration + caption language (default from config; translated via oma-translator when non-source).
• Captions: tiktok (centered, static windowed cues), lower-third, or none.
• Duration: target seconds (<= 180) or auto (derived from the script).
• Voice / music: voice profile or none; music upbeat / calm / none. The default voice is none → a silent video with estimated caption timing. Pass --voice <profile> (a Voicebox profile) whenever narration is expected. Music is rendered offline by Strudel and mixed at −18 dB; it needs a one-time oma video doctor --install-strudel and degrades to no music without it.

Amplification shortcut. For a one-line brief (e.g. "shorts about Jeju coffee"), do not pop a questionnaire if the request is genuinely simple. Instead amplify inline and show the user the inferred plan before invoking:

User: "make a short about Jeju coffee" Agent: "I'll generate this as: mode shorts, 9:16, ~30s, oma-image stills with Ken Burns, TikTok captions, locale en, calm music. Proceed, or adjust mode/aspect/voice?"

Skip clarification when the user authored a full brief (mode + topic + aspect + captions). Respect their flags verbatim.

Output language. Narration and on-screen text are authored in the requested locale. Image-generation prompts passed to oma-image are sent in English (image models are trained predominantly on English captions); translate the user's request and show the translated version during amplification.

Modes

3-Tier Integration

Invocation

Standalone

/oma-video make a 30s short about Jeju coffee
/oma-video --mode explainer --aspect 16:9 explain this project from the README
/oma-video --mode demo --source file --capture ~/recordings/walkthrough.mp4 feature demo
/oma-video --mode demo --source web --url http://localhost:3000 record my app flow
/oma-video --mode demo --source web --url <url> --ready-selector "#app" --polish onboarding clip

Shell CLI

oma video generate "<brief>" [--mode shorts|explainer|demo] \
                             [--aspect 9:16|16:9|1:1|auto] [--locale <lang>] \
                             [--captions tiktok|lower-third|none] \
                             [--visual auto|generate|stock|aigc|slide] \
                             [--voice <profile>|none] [--music upbeat|calm|cinematic|lofi|piano|none] \
                             [--duration <sec>|auto] [--compositor remotion|mpt] \
                             [--capture <path>] \
                             [--source file|web] [--url <url>] [--device <name>] \
                             [--ready-selector <css>] [--show-cursor] [--polish] \
                             [--capture-timeout <sec>] [--capture-stop duration:<sec>|selector:<css>] \
                             [--out <dir>] [--allow-external-out] \
                             [--max-usd <n>] [--seed <n>] [--timeout 600] [-y] \
                             [--dry-run] [--script <path>] \
                             [--format text|json] [--no-brief-in-manifest]
# --script: inject the agent-authored script.json (agent-as-key). Without it the
#   CLI builds its own skeleton script from the brief — always pass the script
#   the agent wrote so narration/on-screen text/visual prompts are honored.
# --source web: headed browser at --url; capture size derived from --aspect/--device (no hardcoded size).
# A human drives the on-screen flow; press ENTER to stop. NO credential automation. --url/tokens masked.
# Non-interactive (CI / -y / no TTY) or unresolvable Playwright -> falls back to the guided protocol (no hang).
# --capture-stop gives CI a non-interactive stop (duration / selector) in place of the ENTER prompt.
oma video doctor          # readiness report only (no install): Node/Chromium/FFmpeg · Remotion project · Pretendard font · MPT · Playwright · Voicebox MCP · oma-image vendors · Pixelle-MCP · Cap
oma video doctor --install             # one-time: vendored Remotion deps + Chrome Headless Shell + Pretendard font fetch (offline -> warn, system-font fallback)
oma video doctor --install-mpt         # one-time: MoneyPrinterTurbo checkout (clone + venv + deps) for --compositor mpt
oma video doctor --install-playwright  # one-time: npm i playwright + chromium (web capture)
oma video list-providers  # availability + key/fallback status
oma video render <runDir>  # re-render from render-spec.json (deterministic)

Shared Infrastructure (from other skills)

Other skills call oma video generate --format json and parse the JSON envelope ({exitCode, runDir, manifestPath, scriptPath, renderSpecPath, warnings, error}) from stdout. There is no outputs key — read output/asset paths from the manifest at manifestPath. The deterministic boundary is render-spec.json + assets, so a downstream consumer can re-render via oma video render <runDir> without re-running script/voice/visual generation.

Output Layout

.agents/results/videos/
└── 20260603-143052-ab12cd-shorts/        # {timestamp}-{shortid}-{mode}
    ├── script.json                        # determinism boundary start
    ├── timing.json
    ├── render-spec.json                   # deterministic compute boundary
    ├── audio/
    │   └── narration-01.wav               # single narration track (ALL lines joined — not per-scene)
    ├── visuals/
    │   └── scene-01.jpg …
    ├── captions.srt
    ├── captions.vtt
    ├── shorts-<slug>.mp4
    └── manifest.json                      # reproducibility record

Audio Sync & Captions Format

• Narration is one wav: oma-voice joins every scene line into a single audio/narration-01.wav, referenced by render-spec.audio.narration. There are no per-scene narration-NN.wav files.
• Timing: per-line offsets live in timing.json (voicebox-stt -> estimated; the tts-native and whisper-cpp source values are reserved but not yet wired — TODO(oma-deferred): whisper-cpp); scene boundaries and caption cues are derived from it.
• Captions: key-free .srt (+ .vtt) built from timing.json; render-spec.captions.file points at the .srt. The compositor renders static windowed cues — the cue active at the current frame, CSS-wrapped (no per-word animation).
• Music: --music <preset> renders a BGM bed with Strudel and mixes it under narration at render-spec.audio.musicGainDb (default −18 dB). The bed is generated offline (headless Chrome + OfflineAudioContext), so it needs no key, no network, and no audio device — a 30s bed renders in well under a second.
  • Presets: calm (sustained pad + arpeggio), upbeat (bright plucks), cinematic (drone build to a lead), lofi (warm chords, swung ticks), piano (neoclassical arpeggio). Each preset picks its key and mode from the run seed, so the same preset sounds different run to run without a second pattern.
  • Artifacts in the run dir: music/bgm.wav (mixed by the compositor), music/bgm.mp3 (preview), music/bgm-raw.wav (pre-master), and music/pattern.strudel — the source that produced them, editable and re-renderable by hand.
  • Level: every bed is normalised to −14 LUFS with a static gain before a peak limiter, so musicGainDb means the same thing for every preset. Normalisation is deliberately not loudnorm's one-pass mode, which flattens the arrangement arc.
  • Opt-in install: @strudel/* is AGPL-3.0-or-later while the oma CLI is MIT, so the deps are never bundled and never installed implicitly. Run oma video doctor --install-strudel once. The CLI never imports Strudel — it spawns resources/strudel/render.mjs as a subprocess, the same boundary the Remotion / Playwright projects use.
  • Fallback: a missing install, a missing Chrome, or a failed render degrades to no music with a warning. The run still succeeds and audio.music stays unset (never a dangling staticFile() ref).
  • Determinism: the built-in beds are oscillator-only (sine / triangle / square / sawtooth), which render byte-identically on replay. Noise sounds (white / pink / brown) draw from Math.random() and would break that, so the templates avoid them.

References

Follow resources/execution-protocol.md step by step. See resources/vendor-matrix.md for provider precheck + fallback-chain rules. Author --script files against resources/script-schema.md (full field reference + example; schemaVersion: "1.0" is required). Use resources/prompt-tips.md for writing effective briefs per mode. Before submitting, run resources/checklist.md. The vendored Remotion compositor lives at resources/remotion/ (see its README.md). The web-capture driver lives at resources/playwright/record.mjs (runs as a subprocess under the resolved Playwright; never imported into the CLI). The MPT fallback compositor driver lives at resources/mpt/driver.py (consumed by the CLI's mpt-project internals).

Configuration

Project-specific settings: config/video-config.yaml. Env vars: OMA_VIDEO_DEFAULT_MODE, OMA_VIDEO_DEFAULT_OUT, OMA_VIDEO_YES, PEXELS_API_KEY, RUNNINGHUB_API_KEY (+ POLLINATIONS_API_KEY via oma-image), OMA_VIDEO_MOCK, OMA_VIDEO_PLAYWRIGHT_DIR (web-capture Playwright override), OMA_VIDEO_PWTEST (opt-in web-capture e2e).

• Execution steps: resources/execution-protocol.md
• Vendor matrix: resources/vendor-matrix.md
• Prompt tips: resources/prompt-tips.md
• Checklist: resources/checklist.md
• Remotion compositor: resources/remotion/README.md
• Context loading: ../_shared/core/context-loading.md