Talking-Head Reels (VO spine + b-roll)

Read this when the project mode is talking-head. The montage pipeline cuts picture to a beat grid; here the spoken take is the spine and picture is cut to fit it. Everything else (probe, contact sheets, normalize, grade, export) is shared with music-video mode.

No script yet? The script comes first and it is its own job:

Skill(skill: "yap-writer")

Length and budget

These run 60–90s, not the 30s a montage targets — a spoken point needs roughly 150–200 words to land. Set --target-duration accordingly at init; the 30s default will truncate the script.

Keep b-roll under ~40% of the runtime. This format is opinion, and the face carries it. Screen recordings and product/logo cards are illustration, not the reel; a stretch of them longer than a few seconds reads as a slideshow with narration. Cap any single app-screen beat around 3s.

Scripting to a teleprompter is normal here, which makes the take clean but front-loads dead air at line breaks — see Tightening a take below.

The shape

VO take  ├───────── continuous speech, never re-timed ─────────┤
picture  │ face │ b-roll │ b-roll │ face │ b-roll │  face+CTA  │

• Face segments keep their true source in-points. A face segment placed at a different in-point than its audio came from breaks lip-sync instantly. Do not let a generator pick these.
• B-roll covers the middle, where the talk is explanatory and the face adds nothing. Opening and closing on the face is what makes it read as a person talking to you rather than a slideshow.
• B-roll here is often not footage. Logo cards, headline cards, and app screen recordings are the usual material (see Capturing an app window below). Stills go in raw/ like any clip and normalize into Ken Burns motion (kb field) — a static card held for three seconds reads as dead air, a drifting one does not. Screen recordings are ordinary video, but probe flags them as duration outliers; extract the few seconds you want into work/extracts/ first.
• Third-party logos and press imagery carry usage terms. Keep a short note of what came from where alongside the project; nominative/editorial use of a company's marks in a commentary reel is not a licence to restyle them.
• The plan has no music key unless you want a quiet bed. render_reel.py treats a music-less plan as VO-spine and uses the clips' own audio at 0dB; add --natural-audio -6 to duck it under a bed.

Cut plan shape

Segments are ordinary cut_plan.json segments — the difference is how you pick in values, not the format.

{
  "segments": [
    {"clip": "work/mezz/take_01.mp4", "in": 0.0,  "duration": 4.2,  "note": "face — hook line"},
    {"clip": "work/mezz/take_01.mp4", "in": 4.2,  "duration": 6.0,  "note": "face — setup"},
    {"clip": "work/mezz/broll_02.mp4","in": 12.5, "duration": 3.4,  "note": "b-roll over VO 10.2-13.6"},
    {"clip": "work/mezz/take_01.mp4", "in": 13.6, "duration": 5.1,  "note": "face — payoff + CTA"}
  ]
}

The b-roll problem: the renderer concatenates segments, so a b-roll segment replaces its slice of VO audio instead of overlaying it. Two ways out:

1. Cut b-roll from the take's own timeline — keep the VO contiguous by letting b-roll segments carry no audio, then lay the full VO take over the finished picture as a separate pass (see Overlay pass below). Correct, one extra step.
2. Accept the swap on short b-roll — under ~1s the audio break is inaudible under a bed. Fine for accent cuts, wrong for anything longer.

B-roll that must not interrupt the VO

Segments are concatenated, so a b-roll segment replaces its slice of voiceover. Under about a second that is inaudible; longer and it punches a hole in the narration. --vo-overlay lays the continuous take over the cut picture:

python3 scripts/render_reel.py plan/cut_plan.json --grade clean \
    --vo-overlay raw/take_01.mp4 --vo-start 0 --out work/master.mp4

Picture cuts freely; the voice runs unbroken. --vo-start is where in the VO file the reel's t=0 sits. With this flag the segments' own audio is ignored entirely, so face in-points still have to line up with the VO timeline — --vo-overlay fixes the holes, not a drifting edit.

Removing a word or a phrase

tighten_vo.py removes silence; trim_words.py removes speech — a filler word, a fluffed clause, a line that did not survive:

python3 scripts/trim_words.py words.json --video raw/take.mp4 --list
python3 scripts/trim_words.py words.json --video raw/take.mp4 \
    --out work/extracts/take_trimmed.mp4 --remove "to a plan" --breath 0.12

Phrases are matched case- and punctuation-insensitively against the transcript. A phrase appearing more than once is refused, not guessed — disambiguate with --occurrence N, or use --remove-range I:J with indices from --list. --breath leaves held silence at each join so the cut does not sound clipped; zero sounds clipped.

It writes the same keep-range cut list as tighten_vo.py, so --emit-cuts here replays at 4K through tighten_vo.py --apply-cuts.

Re-transcribe after trimming — every word timestamp past the first cut has moved, so captions built on the old transcript will drift.

Dropping abandoned takes

When a speaker fluffs a line they usually back up and start the sentence again with the same opening. --drop-restarts finds those and removes the EARLIER attempt, keeping the completed one:

python3 scripts/trim_words.py words.json --video raw/take.mp4 --drop-restarts --dry-run

Always --dry-run first. It prints what it would cut; a false positive removes a real sentence. --drop-fillers separately removes standalone "um"/"uh".

Groups are split on punctuation or a pause, because an abandoned line is often left unpunctuated by the transcriber — the pause after it is the reliable signal.

Proving the cut still says what you meant

Every audio-driven edit here can clip a word at a join: tighten_vo.py trims to a margin, trim_words.py cuts on word boundaries, and a few frames either way takes a consonant with them. The result plays fine and is subtly wrong, and check_cuts.py cannot see it — it looks at pictures.

python3 scripts/verify_cut.py work/master.mp4 \
    --words work/take.words.json --cuts work/cuts.json

It re-transcribes the finished file and diffs it against the words the plan expected to survive. Below 95% similarity it fails and names the missing words; the fix is a larger --margin or --breath, then re-cut.

With no transcription backend installed it reports SKIPPED, not PASS — an unverifiable check must never read as a passing one.

Cleaning the audio

Room tone — HVAC, traffic, laptop fans — is the usual failing of a take recorded outside a booth. Clean it before tightening, because tighten_vo.py finds pauses by audio level and a noise floor makes real silence look loud:

python3 scripts/clean_audio.py raw/take_01.mp4 --out work/extracts/take_01_clean.mp4

Defaults to ffmpeg's afftdn — no install, no model file. --engine arnndn uses RNNoise if you supply a .rnnn model; --engine deepfilternet is the best quality and falls back to afftdn when deepFilter is not on PATH. --nr tunes the reduction in dB; past about 20 speech starts sounding underwater.

Do not pass --loudnorm for takes staying in this pipeline. export_variants.py already normalizes every platform master to -14 LUFS, and normalizing twice compounds pumping. The flag exists for audio leaving the pipeline, and runs a genuine two-pass (measure, then correct).

Tightening a take (pause removal)

Spoken takes carry dead air, and teleprompter reads carry more of it. Cut it:

python3 scripts/tighten_vo.py raw/take_01.mp4 --out work/extracts/take_01_tight.mp4

Prefers auto-editor (pip install auto-editor) and falls back to an ffmpeg silencedetect pass when it is absent — the two agree to within about a tenth of a second, so the fallback is not a downgrade. Sections are cut, never sped up, so pitch is untouched. Tuning lives in the script's --threshold and --margin help; raise the threshold if breaths survive, lower it if words get clipped, and never set margin to zero — it sounds clipped.

Editing at 1080 and reproducing at 4K

auto-editor is not resolution-independent — the same audio in a 1080 and a 4K copy of one take can lose different amounts, which shifts every face in-point and forces a re-plan. Capture the decisions once, replay them:

python3 scripts/tighten_vo.py take_1080.mp4 --out t.mp4  --emit-cuts work/cuts.json
python3 scripts/tighten_vo.py take_4k.mp4  --out t4k.mp4 --apply-cuts work/cuts.json

--emit-cuts forces the ffmpeg analyser, which reads audio only and so gives the same ranges at any resolution; --apply-cuts does no detection at all and replays the recorded ranges exactly. Verified identical to the millisecond across 1080 and 4K. The list is plain JSON — reviewable, and editable by hand if a range needs nudging.

It prints a JSON summary with how much came out. Re-probe the tightened file before planning against it — its timeline no longer matches the original take, so every face in-point you had is now wrong.

If you are reading silences by hand instead, note that silencedetect and volumedetect log at info level: -v error hides their output entirely. Use -hide_banner and read stderr.

Making the cards

Logo, headline, and stat cards render as real motion clips at mezzanine spec:

python3 scripts/make_cards.py cards.json --outdir work/cards

Each entry needs a title; kicker, subtitle, duration, bg, accent, and logo are optional. Output is ordinary video from that point on — put it in raw/ and probe it like any clip. Requires Node 18+, and the first run installs Remotion into <outdir>/_project, which takes several minutes and a few hundred MB. Without Node, build the card as a still image instead and let normalize_clips.py apply Ken Burns motion — the kb field exists for this.

Motion is deliberately restrained: a card is illustration under a voice, not a title sequence. Keep any single card near 3s and the whole card budget under the ~40% ceiling above.

Capturing an app window

A full-display grab records whatever the compositor last painted, and drags in the menu bar, toolbar, and cursor. A browser tab driven over the CDP/debug bridge is worse: macOS and Chrome throttle its repaints, so the capture holds stale frames and the motion all arrives in a burst at the end — measured at 8 frozen seconds out of 13.

capture_window.py targets one window by its window id, so only that window's content lands in the frame:

python3 scripts/capture_window.py list --filter chrome
python3 scripts/capture_window.py shot --window "Iron Trainer" --out work/broll/f001.png

--window takes a window id or a title/owner substring. A substring resolves to the LARGEST match, which is the content window rather than a tooltip or panel.

For motion, work stop-motion: drive the UI one step, capture one frame, repeat. Each frame is pulled on demand, so it can never be stale relative to the step that produced it. burst does the unattended version on a timer:

python3 scripts/capture_window.py burst --window 12345 --out work/broll/seq \
    --frames 90 --interval 0.1
python3 scripts/capture_window.py assemble --frames work/broll/seq \
    --out work/broll/b_app.mp4 --fps 30 --dedupe --crop 2992:1660:0:214

screencapture manages roughly 3-5 fps, so over-sample a stepped animation and let --dedupe keep one frame per rendered state; it also trims the idle head and tail spent waiting for the UI to move. --crop cuts browser chrome out of the frame.

Output is ordinary video — put it in raw/ and probe it like any clip. Needs pyobjc-framework-Quartz and Screen Recording permission for the terminal app running it; without either, check_deps.py names it and the rest of the pipeline is unaffected.

Keep it inside the ~40% b-roll ceiling above — cap any single app-screen beat around 3s.

A shot nobody filmed

plan_broll.py turns the take's transcript into timed slots where b-roll fits — budget, hook and CTA guards, beat caps — and leaves the prompt blank on each. Reading what a slot says and deciding whether it earns a picture is the part the script deliberately does not do.

When a beat needs a frame the footage does not cover, gen_images.py renders stills that stay coherent across cuts — see references/generated-stills.md. The rule that matters: additions and modifications REGENERATE, removals EDIT.

Which capture tool

They split on who drives the UI, not on quality:

screencapture manages 3-5 fps, so capture_window.py reads as a slideshow on live motion. The reverse trap is worse: OpenScreen CANNOT fix CDP-throttled repaints, since no capture API sees frames the compositor never painted. Point it at a debugged browser tab and the frozen-frame problem comes straight back.

python3 scripts/capture_screencast.py sources --filter chrome
python3 scripts/capture_screencast.py record --window "Iron Trainer" \
    --duration 20 --into raw/ --name app_demo.mp4

The capture is symlinked into raw/ and is an ordinary clip from there — probe, normalize, plan, grade, export as usual. ScreenCaptureKit writes variable frame rate (measured 55 fps on a window capture), so probe before planning against it; normalize_clips.py puts it on the mezzanine CFR grid.

Needs OpenScreen (MIT, github.com/getopenscreen/openscreen) plus Screen Recording permission. Verify what you installed — an unsigned look-alike distributes under this name from a different publisher. The script refuses to run anything not signed by TeamIdentifier=M4LK7C6S84.

Polished app demos

record gives raw motion. demo runs OpenScreen's full path — record, edit the project, render — to produce ONE finished b-roll clip with cursor-driven zoom and on-screen notes:

python3 scripts/capture_screencast.py demo --window "Iron Trainer" \
    --duration 20 --spec plan/demo_spec.json --auto-zoom \
    --into raw/ --name app_demo.mp4

The spec talks in seconds, like a cut plan:

{
  "zooms": [{"at": 2.0, "until": 6.0, "depth": 3, "cx": 0.5, "cy": 0.4}],
  "notes": [{"at": 0.5, "until": 4.0, "text": "One-click setup"}],
  "auto_zoom": true
}

--auto-zoom is the reason this path exists: cursor dwell detection needs the .cursor.json telemetry, and ffmpeg has no equivalent. Everything else the pipeline does better.

This renders a CLIP, not a reel. It lands in raw/ and the pipeline still owns the cut, the grade and both gates — one editor per timeline. Keep it inside the ~40% b-roll ceiling, roughly 3s per app beat, so a 20s take is material to cut from, not the beat itself.

Skip --audio for talking-head work: the VO comes from the pipeline, and --audio-mode mix ducks the recording under it while replace drops it entirely. App clips usually want neither.

Keeping the head in frame

A centre crop decapitates anyone framed high in the shot, which is most talking-head takes — people leave headroom, so the face sits above centre.

python3 scripts/reframe_offset.py work/master.mp4 --height 1350

It samples faces with Apple Vision and places them near the upper third. It takes a low quantile rather than the median on purpose: the error is asymmetric, since guessing low crops a head off and guessing high only wastes headroom, and with two people a median lands between them and matches neither. Without pyobjc Vision it says so instead of silently centre-cropping.

Transitions on source changes

render_reel.py concatenates with -c copy — hard cuts. To put a hit on each change of picture:

python3 scripts/transitions.py work/master.mp4 plan/cut_plan.json --style flash

Run it on the graded master, BEFORE captions are burned. Only cuts where the source clip changes get one: a word trim inside a single take is meant to be invisible, and a flash there announces the edit.

Every style is duration-preserving, keyed to a time window rather than consuming frames. That matters because a talking-head reel lays a continuous VO over the whole timeline with --vo-overlay, and a crossfade would shorten the picture and desync everything after it. Verified: a 28.733s master comes back 28.733s. If you want real dissolves they belong in the cut plan, with segment durations padded to pay for them.

Captions are mandatory here

These reels are watched muted. Two options, and they are not equivalent.

Plain — render_reel.py --srt captions.srt burns libass text. Fast, no extra dependency, but libass BorderStyle=3 draws a hard rectangle: no rounded corners, no per-word emphasis. Fine for a rough cut.

Styled — gen_captions.py produces the rounded-card look modern IG/TikTok captions use. Needs ImageMagick and word-level timestamps.

mlx_whisper take.wav --word-timestamps True --output-format json
python3 scripts/gen_captions.py review words.json --out work/captions/phrases.json
# read and fix phrases.json, then:
python3 scripts/gen_captions.py burn work/captions/phrases.json \
    --video work/master.mp4 --out work/master_captioned.mp4

The review stage is not optional. Whisper mishears proper nouns and numbers — "70.3" becomes "17.3", "borrowed" becomes "Boren". Read the phrase list and fix it before burning; re-rendering after the fact wastes a full pass.

Emphasis is explicit, never guessed: set "emphasis": [2] on a phrase to italicise its third word in the accent colour.

Position defaults to 0.72 of frame height — lower third, deliberately off the face. Raise it toward 1.0 to sit lower.

Card geometry is authored at 1080 wide and scaled to the real frame, so a 4K master gets proportionally sized captions rather than half-size ones.

Match --crf to the master you are burning onto. The burn re-encodes, so leaving it at the default while the master was rendered at a lower CRF throws that quality away — measured at 80 Mbps down to 44 on one archival master.

Caption against the final cut, not the raw take — timings from the raw take will not match a tightened or b-rolled timeline.

What still gets checked

check_cuts.py works unchanged, but read its FROZEN_CUT flags loosely on face segments: a locked-off talking head genuinely has near-identical frames across a cut. Black frames and duration drift are still real defects.