Reel Builder

Process steps in order. Do not skip ahead.

A batch, human-in-the-loop pipeline turning raw clips into platform-ready vertical reels, in either of two formats. ffmpeg-based, with optional Whisper (captions) and Gemini (hook copy, thumbnails).

Pipeline Overview

┌ music-video ─▶ 5.BEATS ─┐
raw ─▶ 1.FORMAT ─▶ 2.PROBE ─▶ 3.INSPECT ─▶ 4.NORMALIZE┤                         ├▶ 6.CUT PLAN + DRAFT ─▶ ⏸ GATE 1
       (ask!)      (metadata) (look at it)            └ talking-head ──────────┘                            │
  final exports ◀── 8.EXPORT ◀── ⏸ GATE 2 ◀── 7.FINAL RENDER + QC ◀── (approve / edit plan) ◀───────────────┘

The format is the first question, not an assumption. Steps 2–4 and 7–8 are identical for both; only beat detection and how the cut plan is built differ.

Two mandatory human review gates: cut plan approval (Gate 1 — always present the plan together with a cheap 540p preview) and final approval (Gate 2 — graded render before export). Never skip either.

Never plan blind. View every contact sheet (Step 3) before proposing a cut plan — a clip named "nyc iceberg.mp4" can be a 44-minute conference talk, and fitness clips can carry burned-in captions that disqualify them.

Requirements

Required: ffmpeg + ffprobe (brew install ffmpeg); for music-video, a .venv with librosa + soundfile (macOS refuses system pip, PEP 668). Homebrew ffmpeg has no drawtext — text goes through subtitles/libass.

Optional: auto-editor, deepfilternet, Node 18+, moviepy, yt-dlp, mlx-whisper, a Gemini API key — each named with its cost in references/dependencies.md.

Run python3 scripts/check_deps.py first, every session — it exits non-zero on a missing required tool, and pipeline.py status blocks on the same check. Never silently substitute a lesser tool for a missing one: each fallback loses something specific, so name the loss and let the user choose. Scripts record what ran (beats.json → source; tighten_vo.py / clean_audio.py → engine_requested, substituted) — report a substitution, never pass its output off as the preferred tool's. What each missing tool costs: references/dependencies.md.

Working Directory Layout

One project folder per reel:

mkdir my-reel && cd my-reel
python3 scripts/pipeline.py init --target-duration 30 --mode music-video --resolution 1080

Scaffolds raw/ music/ work/ plan/ preview/ exports/ plus project.md (why — settled decisions) and state.json (where — stage state, gates, and the four Step 1 answers, which route every later status / next). Layout, flags, gate bookkeeping: references/project-setup.md.

Footage that already exists anywhere — external drive, sibling folder — is symlinked into raw/, never copied. Replace the empty raw/ that init created:

rmdir raw && ln -s "$(cd /path/to/footage && pwd)" raw

Resolve the target to an absolute path, as above — the project folder usually sits below the footage, so a bare ln -s footage raw dangles and probe_clips.py reports zero clips.

Verify before probing: ls -l raw shows the link and ls raw/ | head lists real files.

raw/ is permanent structure, symlink included — never delete it. Session cleanup clears intermediates under work/ only; removing the link orphans clips.json and every mezzanine path through it. Every script follows symlinks. A removable source must stay mounted to re-probe or re-normalize; work/ mezzanines keep renders working if it isn't.

pipeline.py is the state machine. Stage completion is read from disk; staleness chains through mtimes, so editing the cut plan flips preview/master/exports to stale.

Session-start ritual — MANDATORY, in this order, before touching any media:

1. python3 scripts/pipeline.py status — where the project stands; run next whenever unsure.
2. Read project.md — the human's settled decisions; never re-ask them.
3. Create the venv if it is missing — do this before any check, and never pip3 install into system Python (macOS refuses it, PEP 668):

   python3 -m venv .venv && .venv/bin/pip install librosa soundfile

   Run detect_beats.py with .venv/bin/python3 from then on.
4. Then gate on the preflight — it exits non-zero when a required tool is missing:

   python3 scripts/check_deps.py --mode music-video

   If it reports anything missing, install it and re-run until it passes; reporting the gap is not completing this step. Do not touch media until it exits clean. Then tell the user which optional tools are absent and what each costs.

During the session, pipeline.py gate 1|2 presented|approved|nle-wait|reopen records every human decision the moment it happens.

Step-by-Step Workflow

Step 1 — Ask what they're making

Ask all four before touching any media, and ask explicitly — never infer them from the footage. A folder of action clips is just as plausibly b-roll for a talking-head reel, and every one of these choices is expensive to change after normalize.

1. Format. Music video — beat-synced montage; needs music, no speech. Or talking head ("yapping", piece-to-camera) — a spoken take carries it with b-roll over the middle; needs a VO take, music optional. Talking-head runs 60–90s, so raise --target-duration well above the 30s montage default.
2. Aspect. 9:16 reel/short (default). If it isn't going out as a reel or short, ask whether they want 16:9 — a landscape YouTube cut is a different deliverable and exports one youtube.mp4 instead of the three vertical targets.
3. Resolution. 1080 (default) or 4k — 4K for an archival or YouTube-primary master; 4K normalize/render is markedly slower.
4. Captions. Yes or no. Talking-head reels are watched muted, so recommend yes there; transcription must happen before Gate 2, so the answer changes the plan.

Trade-offs behind each: references/project-setup.md.

If the user already settled something ("cut this to the beat", "make it landscape for YouTube"), don't re-ask — state what you're proceeding with. If they're unsure on format, ask what the reel is for: a mood piece is music-video, an explanation or opinion is talking-head.

Record all four before probing:

python3 scripts/pipeline.py init --target-duration 75 --mode talking-head \
  --aspect 9:16 --resolution 4k --captions yes

status then shows them ([talking-head · 9:16 · 4k · captions]) and adds --aspect/--resolution to the commands it emits, so the choices can't drift apart. Record them in project.md too. Proceed to Step 2 — the next three steps are the same for every combination.

Step 2 — Probe & inventory

python3 scripts/probe_clips.py raw/ --target-duration 30 --out work/clips.json

Produces work/clips.json with duration, resolution, fps, codec, rotation, and color transfer (detecting iPhone HDR/HLG and DJI D-Log) per clip. Still photos (jpg/png/webp/heic) are inventoried too and become Ken Burns clips at normalize time (--still-duration 8; for HEIC ffmpeg can't read, sips -s format jpeg first). Clips longer than 10× --target-duration are flagged as duration outliers. Don't normalize one whole — extract the usable segment (ffmpeg -ss START -to END -i clip -c copy work/extracts/name.mp4, bounds from its Step 3 sheet) and re-probe.

Summarize to the user: total clips, runtime, outliers, problem files, and which look log/HDR. Ask what story/order they have in mind if filenames don't say — filenames are hints; Step 3 verifies them.

Step 3 — Look at the footage (contact sheets)

python3 scripts/contact_sheet.py work/clips.json --outdir work/sheets

One dense frame grid per clip across its full duration (--interval 1.0 for denser). Clips run concurrently; the sampler picks decode or keyframe-seek by measured cost. View every sheet (Read the PNGs) before planning. Look for:

• Disqualifiers: burned-in captions/lyrics, screen recordings, watermarks, long static stretches. Intermittent text is the classic trap — it appears mid-clip only and sparse sampling misses it; re-run a borderline clip at --interval 0.5.
• The hook and the closer: the most kinetic, instantly readable moment opens the reel; the payoff/smile/CTA moment closes it. Note which clips (and roughly where in them) these live — feed them to build_cut_plan.py --hook/--close.
• Where the action is in each clip, so in-points and pan values (Step 4) aren't guesses.

Report rejects and picks with reasons; record them in project.md and remove rejected clips from clips.json.

Step 4 — Normalize to mezzanine

Mixed sources can't be concatenated directly. Normalize to a uniform intermediate:

python3 scripts/normalize_clips.py work/clips.json --outdir work/mezz --profile auto

Default mezzanine: 1080x1920 (9:16), 30fps, H.264 high-bitrate, BT.709 SDR, stereo 48kHz. The script auto-crops landscape footage to center-cut vertical; pass --pan left|center|right per clip in clips.json if the subject is off-center (the contact sheets from Step 3 show you where the subject is). Camera-specific color handling (HLG→SDR tonemap, D-Log→Rec709) is applied automatically based on probe data.

--aspect and --resolution set the frame (see Step 1); pass the same values at init, here, and at render. Cropping across aspect throws away one axis — a landscape source cut to 9:16 yields only ~1215px of width and upscales to fill — so the script names each affected clip on stderr.

Still photos become Ken Burns clips here, at mezzanine spec, so later steps treat them as ordinary footage. Direction via each image's kb field (auto/in/out/left/right); prefer in for a centred subject, since a pan travels the still's full width.

D-Log clips convert via a bundled LUT, applied only to dlog-profile clips, never to SDR footage — a conversion LUT on Rec.709 material crushes blacks and pushes colour out of gamut. --dlog-lut official.cube uses DJI's exact per-model LUT. Details: references/ffmpeg-recipes.md.

Outputs are atomic (.tmp.mp4, renamed on completion), but still gate on the process exiting, not on files appearing. Slow on 4K sources — if backgrounded, wait for exit before rendering.

Step 5 — Beat detection (music-video only)

Talking-head reels skip this step entirely — cuts land on speech, not on a grid, and pipeline.py drops the stage from status in that mode. If the user wants a quiet music bed under the VO, drop the file in music/ and go straight to Step 6; no beat analysis is needed for a bed. Proceed to Step 6.

python3 scripts/detect_beats.py music/track.mp3 --out work/beats.json

Outputs BPM, beat timestamps, and detected downbeats (stronger cut candidates). No librosa? Use --bpm 120 (or the track's known BPM) for a fixed grid.

No music in music/ yet? Stop and ask the user — beat-synced cuts are the core value, so don't silently proceed without a track. Offer three options:

1. They drop a file into music/ (mp3/m4a/wav).
2. They paste a YouTube link — python3 scripts/fetch_music.py "<url>" --outdir music/ extracts a beat-ready m4a plus an .srt of lyrics when the video has them (useful for picking music_start at a lyric). --section 1:10-1:50 grabs just part of a long track. Requires yt-dlp. Always remind them about music rights: platform in-app music libraries don't cover audio baked into an uploaded render.
3. Proceed with a fixed rhythm (--bpm) and re-sync when they pick a track — last resort.

Step 6 — Cut plan + draft preview (⏸ HUMAN GATE 1)

Both formats produce the same artifact — plan/cut_plan.json — but reach it differently. Take the branch matching the mode from Step 1, then rejoin below.

Music video: build_cut_plan.py generates it against the beat grid, using a phase-based pacing structure with a 2-beat minimum cut. --score-in-points keeps only in-points that contain a person (Apple Vision, macOS) — worth it on footage with a subject, pointless on POV. Read references/montage-cuts.md for the command, the phase table, and the two known weaknesses worth checking by hand.

Talking-head: no generator — author the plan yourself and read references/talking-head.md first. What decides whether it works: face segments keep their true source in-points (anything else breaks lip-sync), b-roll covers the explanatory middle while the face opens and closes, and the plan carries no music key unless the user wants a bed. Audio prep is clean_audio.py then tighten_vo.py; b-roll over ~1s needs --vo-overlay. After any audio-driven edit run verify_cut.py — it re-transcribes the cut and fails if a join ate a word, which check_cuts.py cannot see. That reference also covers restart removal, word trims, 4K cut replay, and cards.

Audit the plan's content first — python3 scripts/audit_plan.py plan/cut_plan.json --out work/plan_audit.png renders one labeled thumbnail per segment. An in-point landing on empty track or sky is dead space; check_cuts.py samples across cuts for defects, this samples within segments for content.

Then render a cheap draft and QC it before the human sees anything:

python3 scripts/render_reel.py plan/cut_plan.json --preview --out preview/draft_v1.mp4
python3 scripts/check_cuts.py plan/cut_plan.json preview/draft_v1.mp4 --out work/qc_v1

--preview renders fast at low res. check_cuts.py writes a frame-pair sheet across every cut plus a report JSON, flagging black frames, frozen cuts, and duration drift. View it and fix what it finds first.

Present Gate 1 as plan + preview together — the readable timeline and the draft video. Record it (pipeline.py gate 1 presented), then ask which way they want to go:

1. Approve — pipeline.py gate 1 approved (it refuses if the preview went stale), proceed to Step 7.
2. Describe changes — reorder, swap clips, adjust hook or pacing. Edit cut_plan.json, re-render the draft (bump versions), return to Gate 1.
3. Fine-tune in their NLE (Final Cut / Resolve) — offer this whenever feedback turns frame-level.

Hook advice: the first 1.5–2s decides scroll-past.

Option 3 — NLE round trip. Export FCPXML, pause while they edit, re-import the cuts. Procedure and the mandatory gate 1 nle-wait rule: references/nle-round-trip.md.

Step 7 — Final render + QC (⏸ HUMAN GATE 2)

python3 scripts/render_reel.py plan/cut_plan.json --grade gritty --out work/master.mp4
python3 scripts/check_cuts.py plan/cut_plan.json work/master.mp4 --out work/qc_master

Grades: gritty (teal/orange, vignetted), rapha (muted editorial, no vignette — softer on faces), cine (contrast + grain, colour-faithful), punch (hardest), clean, none, or --lut file.cube. Undecided? python3 scripts/compare_grades.py plan/cut_plan.json --out work/grades.png tiles the same frames under every grade in a couple of seconds — pick from the sheet instead of rendering the reel repeatedly. Bundled LUTs in assets/luts/looks/ are Rec.709-safe; assets/luts/conversion/ is D-Log only. Filter math in references/ffmpeg-recipes.md.

--natural-audio -18 keeps ambient clip sound under the music; default is music-only. Captions: --srt burns plain libass text; scripts/gen_captions.py gives the rounded-card look with per-word emphasis, and its review stage surfaces the transcript for correction first — Whisper mishears names and numbers. Match its --crf to the master. See references/talking-head.md.

Talking-head: captions are mandatory — these reels are watched muted. Time them against the final cut. Prefer --grade clean; the gritty look's cool shadows and vignette flatter action footage and do faces no favors. A music-less plan needs no --natural-audio (the VO is already the spine at 0dB); with a bed, duck the voice deliberately, e.g. --natural-audio -6.

render_moviepy.py is an alternative engine accepting the same plan, for A/B comparison — slower, slightly different audio mix; see references/render-engines.md. Ask before switching; don't swap engines mid-project.

--crf 12 --audio-bitrate 320k renders an archival master. Raising resolution rarely helps: a 9:16 crop from landscape 4K is only ~1215px wide natively, so 1080 is already effectively native and 2160 upscales.

QC the master, show it (pipeline.py gate 2 presented), and export only when they say ship it (gate 2 approved).

Step 8 — Export platform variants

python3 scripts/export_variants.py work/master.mp4 --outdir exports/ --thumb-at 1.5 \
    --thumb-text "HOOK LINE"   # optional: magazine-cover text on styled thumbnails

Produces:

• exports/tiktok.mp4, exports/ig_reels.mp4, exports/yt_shorts.mp4 — all 1080x1920 H.264, platform-tuned bitrate/loudness (specs in references/ffmpeg-recipes.md)
• exports/thumbnail.jpg — frame grab at --thumb-at seconds (pick a strong hook frame)
• exports/thumbnail-modern.jpg + exports/thumbnail-80s.jpg — sport-magazine-styled versions of the same frame, generated when a Gemini API key is set; --no-stylize skips. See references/captions-and-gemini.md.

Present all thumbnails and let the user pick the cover — that's a brand call, not yours. Eyeball the styled ones first (Read the JPEGs) for face fidelity and text spelling, and mention once per project that AI-stylized imagery may warrant platform disclosure.

Then offer caption + hashtags per platform (voice in references/brand-style.md, else ask). Deliver exports/.

Batch Mode (multiple reels)

For a week's batch: one project folder per reel, run Steps 1–6 for all reels first, present every plan + preview in one review sitting, then render and export the approved ones together.

Reference Files

• references/dependencies.md — required vs optional tools, what each missing one costs, and the two fakes to refuse. Read at preflight or when a tool is missing.
• references/montage-cuts.md — music-video cut plans: build_cut_plan.py, phase pacing table, generator weaknesses. Read when mode is music-video.
• references/talking-head.md — VO-spine format: plan shape, fixed face in-points, b-roll overlay pass, audio cleanup, tightening, cards. Read when mode is talking-head.
• references/ffmpeg-recipes.md — camera profiles, grade filter chains, export specs, troubleshooting (A/V desync, rotation, VFR). Read when a script default fails or manual control is wanted.
• references/project-setup.md — layout, init flags, state vs project.md, gate bookkeeping. Read when scaffolding or when state looks wrong.
• references/nle-round-trip.md — FCPXML export/import and the hands-off rule. Read when Gate 1 feedback turns frame-level.
• references/render-engines.md — ffmpeg vs MoviePy, and how to A/B them. Read when asked to compare engines.
• references/captions-and-gemini.md — Whisper captions, burn-in styling, Gemini hook copy and thumbnails. Read for captions or AI copy/imagery.
• references/brand-style.md — brand palette, caption style, voice. Prefer the user's own brand doc if they have one.

Principles

• Trust pipeline.py status over memory — a stale preview/master is a fact, not an opinion. Record gate decisions the moment they happen.
• Never modify or delete raw/ — including the symlink itself, which survives cleanup. Extracted segments go in work/extracts/; disposables live only under work/.
• Never plan from metadata alone — view the contact sheets first.
• Every creative decision surfaces to the human before an expensive render — always as something watchable, not just a text plan.
• Gate on process exit, never on output files appearing — "almost done" background jobs produce moov-atom errors downstream.
• Prefer script defaults; drop to manual ffmpeg (recipes reference) only when defaults fail.
• Report progress on long jobs and validate outputs (ffprobe) before declaring success.