Reel Builder

A batch, human-in-the-loop pipeline that turns a folder of raw clips + a music track into platform-ready vertical reels. Built around ffmpeg with optional Whisper (captions) and Gemini API (hook copy, captions/hashtags, thumbnails).

Pipeline Overview

raw clips ─▶ 1.PROBE ──▶ 2.INSPECT ──▶ 3.NORMALIZE ──▶ 4.BEATS ──▶ 5.CUT PLAN + DRAFT PREVIEW ─▶ ⏸ GATE 1
             (metadata)  (look at the footage)                                                       │
final exports ◀── 7.EXPORT ◀── ⏸ GATE 2 ◀── 6.FINAL RENDER + QC ◀── (approve / edit plan) ◀─────────┘

Two mandatory human review gates: cut plan approval (Gate 1 — always present the plan together with a cheap 540p preview; humans can't judge a cut list, they can judge 15 seconds of video) and final approval (Gate 2 — graded render before export). Never skip these — the whole point of the batch flow is that the human signs off on creative decisions, not renders.

Never plan blind. Probe metadata tells you nothing about what's in the footage. Step 2 (contact sheets) exists because filenames lie: a clip called "nyc iceberg.mp4" can be a 44-minute conference talk, and fitness clips can carry burned-in captions or talking heads that disqualify them. Look at every clip's sheet before proposing a cut plan.

Requirements

• ffmpeg + ffprobe (required): brew install ffmpeg
  • Note: Homebrew's ffmpeg is built without drawtext — no recipe in this skill relies on it. Text overlays go through the subtitles filter (libass, included) or are added in post by the platform apps.
• Python 3.10+ with librosa + soundfile for beat detection. Modern macOS Python is externally managed (PEP 668), so pip3 install librosa fails system-wide — create a venv:

  python3 -m venv .venv && source .venv/bin/activate
  pip install librosa soundfile

  Use .venv/bin/python3 for detect_beats.py afterwards. If librosa is unavailable, the pipeline falls back to fixed-interval cuts (--bpm flag).
• Optional — music from YouTube: yt-dlp (brew install yt-dlp) for fetch_music.py when the user has no local music file.
• Optional — captions: mlx-whisper (best on Apple Silicon: pip install mlx-whisper, same venv) or whisper.cpp. See references/captions-and-gemini.md.
• Optional — AI copy + stylized thumbnails: a Gemini API key in GEMINI_API_KEY (or GOOGLE_AI_API_KEY / GOOGLE_API_KEY) enables hook-copy drafts and sport-magazine thumbnail styling (Nano Banana). See references/captions-and-gemini.md.

Check these at the start of a session (ffmpeg -version, python3 -c "import librosa") and tell the user what's missing rather than failing mid-pipeline.

Working Directory Layout

Create a project folder per reel and scaffold it:

mkdir my-reel && cd my-reel
python3 scripts/pipeline.py init --target-duration 30

my-reel/
├── project.md     # session memory — read first, update after every gate
├── state.json     # pipeline state (gates, config) — managed by pipeline.py
├── raw/           # user's original clips (never modified) — may be a symlink
├── music/         # one audio track (mp3/m4a/wav)
├── work/          # mezzanine files, sheets/, extracts/, beats.json, clips.json
├── plan/          # cut_plan.json (the human-editable artifact), reel.fcpxml (NLE round trip)
├── preview/       # low-res draft renders
└── exports/       # final platform deliverables

pipeline.py is the pipeline's state machine — trust it over memory. Stage completion is detected from artifacts on disk, staleness chains through file mtimes (edit the cut plan → preview/master/exports flip to stale automatically), and gate decisions are recorded in state.json. The ritual:

• python3 scripts/pipeline.py status — run at the start of every session (with project.md) and whenever unsure where things stand
• python3 scripts/pipeline.py next — the single next action, with the exact command
• python3 scripts/pipeline.py gate 1|2 presented|approved|nle-wait|reopen — record every human decision the moment it happens; it refuses approvals when the preview/master is stale, and enforces the NLE pause

The step-by-step below is the why and the creative guidance; pipeline.py tracks the where are we.

Footage on an external drive? Don't copy gigabytes — symlink it: ln -s /Volumes/Drive/shoot-folder raw. Works with every script. Caveat to tell the user: the drive must stay mounted for any re-normalization or re-probe later in the project; the mezzanine files in work/ keep renders working if it isn't.

pipeline.py init copies assets/project_template.md to project.md. Read it at the start of every session and update it after each review gate — record the human's decisions (approved hook, trims, grade tweaks) so later sessions and batch reels never re-ask settled questions; state.json tracks where the pipeline is, project.md records why. Copy the "carry-forward learnings" section into each new project.

Step-by-Step Workflow

Step 1 — Probe & inventory

python3 scripts/probe_clips.py raw/ --target-duration 30 --out work/clips.json

Produces work/clips.json with duration, resolution, fps, codec, rotation, color transfer (detects iPhone HDR/HLG and DJI D-Log flat profiles) per clip. Pass the reel's planned length as --target-duration: clips longer than 10× it are flagged as duration outliers (screen recordings, full events, talks). Don't normalize an outlier whole — extract the usable segment first (ffmpeg -ss START -to END -i clip -c copy work/extracts/name.mp4, pick START/END from its contact sheet in Step 2) and re-probe.

Summarize this to the user: total clips, total runtime, outliers, any problem files (corrupt, audio-only, unsupported), and which clips look log/HDR (they'll need the corresponding normalization profile). Ask the user roughly what story/order they have in mind if it isn't obvious from filenames — but treat filenames as hints only; Step 2 verifies them.

Step 2 — Look at the footage (contact sheets)

python3 scripts/contact_sheet.py work/clips.json --outdir work/sheets

One dense frame grid per clip, sampled across the clip's full duration (default 1 frame per ~2s; --interval 1.0 for denser). View every sheet (Read the PNGs) before planning. You are looking for:

• Disqualifiers: burned-in captions/lyrics, talking heads, screen recordings, watermarks, long static/unusable stretches. Intermittent text is the classic trap — it appears mid-clip only, which sparse sampling misses; if a clip is borderline, re-run its sheet with --interval 0.5.
• The hook and the closer: the most kinetic, instantly readable moment opens the reel; the payoff/smile/CTA moment closes it. Note which clips (and roughly where in them) these live — feed them to build_cut_plan.py --hook/--close.
• Where the action is in each clip, so in-points and pan values (Step 3) aren't guesses.

Report rejects and picks to the user (with reasons) and record them in project.md. Clips the user rejects: remove them from clips.json (or delete and re-probe) so no later step resurrects them.

Step 3 — Normalize to mezzanine

Mixed sources (iPhone 4K/60 HDR, DJI 2.7K, drone 30fps…) can't be concatenated directly. Normalize everything to a uniform intermediate:

python3 scripts/normalize_clips.py work/clips.json --outdir work/mezz --profile auto

Default mezzanine: 1080x1920 (9:16), 30fps, H.264 high-bitrate, BT.709 SDR, stereo 48kHz. The script auto-crops landscape footage to center-cut vertical; pass --pan left|center|right per clip in clips.json if the subject is off-center (the contact sheets from Step 2 show you where the subject is). Camera-specific color handling (HLG→SDR tonemap, D-Log→Rec709) is applied automatically based on probe data.

D-Log/D-Log M clips convert via the bundled assets/luts/conversion/dji-dlogm-to-rec709.cube (generated from DJI's whitepaper math — a good approximation), applied only to dlog-profile clips, never to SDR footage — a camera-conversion LUT on already-Rec.709 material crushes blacks and blows colors out of gamut. For exact conversion, have the user download the official "D-Log M to Rec.709" LUT for their drone model from dji.com's Download Center and pass it: --dlog-lut official.cube. Details, LUT generation, and manual recipes in references/ffmpeg-recipes.md.

Outputs are atomic (written as .tmp.mp4, renamed when complete), so a partial file can never be consumed downstream — but still gate on the process exiting successfully, not on files appearing. This step is slow on 4K sources; if run in the background, wait for exit before rendering.

Step 4 — Beat detection

python3 scripts/detect_beats.py music/track.mp3 --out work/beats.json

Outputs BPM, beat timestamps, and detected downbeats (stronger cut candidates). No librosa? Use --bpm 120 (or the track's known BPM) for a fixed grid.

No music in music/ yet? Stop and ask the user — beat-synced cuts are the core value, so don't silently proceed without a track. Offer three options:

1. They drop a file into music/ (mp3/m4a/wav).
2. They paste a YouTube link — fetch the audio (and lyrics, when available):

   python3 scripts/fetch_music.py "https://youtube.com/watch?v=..." --outdir music/

   Extracts a beat-detection-ready m4a plus an .srt of subtitles/lyrics if the video has them (useful for picking music_start at a specific lyric, or --srt burn-in later). For long tracks grab just the section they want: --section 1:10-1:50 (cut is frame-exact). Requires yt-dlp. Always remind them about music rights: platform in-app music libraries don't cover audio baked into an uploaded render — a muted or taken-down reel wastes the pipeline.
3. Proceed with a fixed rhythm (--bpm) and re-sync when they pick a track — last resort.

Step 5 — Cut plan + draft preview (⏸ HUMAN GATE 1)

python3 scripts/build_cut_plan.py work/clips.json work/beats.json \
  --target-duration 30 --hook run_01 --close finish_04 --out plan/cut_plan.json

The generator uses a phase-based pacing structure, every duration snapped to real beat timestamps:

Built-in guarantees: 2-beat minimum cut (1-beat flash cuts are opt-in and punch-phase-only), in-points spread across each clip's full duration (not clustered in its first seconds), and no two adjacent segments from the same clip (that's a jump-cut generator). plan/cut_plan.json is the single human-editable source of truth:

{
  "music": "music/track.mp3",
  "music_start": 0.0,
  "segments": [
    {"clip": "work/mezz/run_01.mp4", "in": 2.4, "duration": 1.85, "note": "HOOK — most kinetic shot here"},
    {"clip": "work/mezz/gym_03.mp4", "in": 7.4, "duration": 0.94, "note": "groove · 2-beat"}
  ]
}

Then render a cheap draft and QC it before the human sees anything:

python3 scripts/render_reel.py plan/cut_plan.json --preview --out preview/draft_v1.mp4
python3 scripts/check_cuts.py plan/cut_plan.json preview/draft_v1.mp4 --out work/qc_v1

--preview renders fast at 540x960. check_cuts.py writes a contact sheet of frame pairs across every cut (work/qc_v1.png) plus a report JSON — it flags black frames, frozen cuts, and duration drift. View the sheet yourself; fix flagged problems by editing the plan and re-rendering. Don't burn a human review cycle on defects a script can catch.

Present Gate 1 as plan + preview together: the readable timeline (segment order, source clip, duration, phase) and the draft video. Record it: pipeline.py gate 1 presented. Then ask the user explicitly which way they want to go — don't assume they'll direct edits through you:

1. Approve — pipeline.py gate 1 approved (it refuses if the preview went stale), proceed to Step 6.
2. Describe changes — reorder, swap clips, adjust hook, trim points, faster/slower pacing. Apply edits directly to cut_plan.json, re-render the draft (bump versions: draft_v2.mp4), return to Gate 1.
3. Fine-tune in their NLE (Final Cut / Resolve) — the hands-on option below. Offer it whenever feedback turns granular ("nudge that cut a few frames") — frame-level trimming is faster in an NLE than through conversation.

Hook advice: the first 1.5–2s decides scroll-past.

Option 3 — NLE round trip. Export the plan as an FCPXML timeline and hand it over:

python3 scripts/export_timeline.py plan/cut_plan.json --out plan/reel.fcpxml

Opens in Final Cut Pro (File > Import > XML) and DaVinci Resolve (File > Import Timeline) with every clip pre-placed at the right source timecode, phase notes as clip names, and the music on its own audio lane. Defaults to fcpxml v1.10 — Resolve rejects newer versions (--fcpxml-version 1.9 for Resolve 17).

Then STOP — the pipeline is paused while the human edits. Run pipeline.py gate 1 nle-wait (it snapshots the plan's mtime and status will alarm if anything touches cut_plan.json during the wait), note it in project.md, and tell them how to hand the result back (export fcpxml from the NLE — from Resolve choose v1.9/1.10; from FCP either .fcpxml or a .fcpxmld bundle works). Do not render, re-plan, or touch cut_plan.json in the meantime — their edit would be overwritten. When the edited file comes back:

python3 scripts/import_timeline.py plan/reel_edited.fcpxml \
    --out plan/cut_plan.json --base-plan plan/cut_plan.json

Only cuts survive the round trip — clip order, in-points, durations, music offset. NLE-side effects, transitions, titles, and retimes are ignored (the importer warns — surface those warnings to the user so they know what didn't make it); grading stays in the render step. Timeline gaps are dropped with a warning since the renderer butts segments together. After import: pipeline.py gate 1 reopen, re-render the preview, QC, and return to Gate 1 for a final confirm.

Step 6 — Final render + QC (⏸ HUMAN GATE 2)

python3 scripts/render_reel.py plan/cut_plan.json --grade gritty --out work/master.mp4
python3 scripts/check_cuts.py plan/cut_plan.json work/master.mp4 --out work/qc_master

--grade gritty applies the teal/orange "Gritty Authentic Energy" look (lifted contrast, warm mids, cool shadows, slight saturation). Alternatives: --grade clean (contrast + wb only), --grade none, or --lut path/to/file.cube. Bundled look LUTs in assets/luts/looks/ (reel-gritty / reel-punchy / reel-faded) are Rec.709 in/out — safe on any normalized footage, usable here via --lut or handed to the user for their NLE. (Don't confuse them with assets/luts/conversion/ — that one is for D-Log footage only.) Regenerate or restyle with scripts/make_luts.py — see references/ffmpeg-recipes.md. To keep ambient clip audio (gym sounds, footsteps, breathing — adds authenticity) mixed under the music, add --natural-audio -18 (dB gain for the natural track; 30ms fades at every cut prevent audio pops). Default is music-only. Grade math lives in references/ffmpeg-recipes.md. Optional caption burn-in: --srt captions.srt (generate via Whisper first — see references/captions-and-gemini.md).

QC the master, then show it to the human (pipeline.py gate 2 presented). Only move to export when they say ship it: pipeline.py gate 2 approved.

Step 7 — Export platform variants

python3 scripts/export_variants.py work/master.mp4 --outdir exports/ --thumb-at 1.5 \
    --thumb-text "HOOK LINE"   # optional: magazine-cover text on styled thumbnails

Produces:

• exports/tiktok.mp4, exports/ig_reels.mp4, exports/yt_shorts.mp4 — all 1080x1920 H.264, platform-tuned bitrate/loudness (specs in references/ffmpeg-recipes.md)
• exports/thumbnail.jpg — frame grab at --thumb-at seconds (pick a strong hook frame)
• exports/thumbnail-modern.jpg + exports/thumbnail-80s.jpg — sport-magazine-styled versions of the same real frame (Nano Banana via scripts/stylize_thumbnail.py), generated automatically when a Gemini API key is set (GEMINI_API_KEY / GOOGLE_AI_API_KEY / GOOGLE_API_KEY); --no-stylize skips. With --thumb-text they become full covers (masthead + headline typography).

Present all thumbnails to the user and let them pick the cover — plain grab vs modern vs 80s is a brand call, not yours. Before presenting, eyeball the styled ones (Read the JPEGs): check face fidelity and text spelling. Mention once per project that AI-stylized imagery may warrant platform disclosure. More styles / re-runs: references/captions-and-gemini.md.

Then offer to draft the caption + hashtags per platform (Gemini-assisted or written directly — brand voice rules in references/brand-style.md if present, otherwise ask about tone). Deliver everything in exports/ to the user.

Batch Mode (multiple reels)

For a week's batch: create one project folder per reel, run Steps 1–5 for all reels first, present all cut plans + draft previews in one review session, then render/export approved ones together. This concentrates the human's attention into one review sitting instead of interrupting them per reel.

Reference Files

• references/ffmpeg-recipes.md — camera normalization profiles (iPhone HLG, DJI D-Log, drone), grade filter chains, platform export specs, troubleshooting (A/V desync, rotation metadata, VFR footage, segment extraction from long clips). Read when a script's default handling fails or the user asks for manual control.
• references/captions-and-gemini.md — Whisper caption workflow (mlx-whisper on Apple Silicon), styling burned-in captions, Gemini API for hook copy / captions / hashtags / thumbnail generation. Read when the user wants captions or AI-generated copy/imagery.
• references/brand-style.md — the user's brand look-and-feel defaults (palette, caption style, voice). If the user has their own brand memory document, prefer that.

Principles

• Trust pipeline.py status over memory — stage state is read from disk, and a stale preview/master is a fact, not an opinion. Record gate decisions the moment they happen.
• Never modify files in raw/ — originals are sacred. (Extracted segments go in work/extracts/.)
• Never plan from metadata alone — view the contact sheets first; the footage is the truth, filenames are rumors.
• Every creative decision (clip order, hook, trims, grade choice) surfaces to the human before an expensive render — and always as something watchable, not just a text plan.
• Gate on process exit, never on output files appearing — background jobs that are "almost done" produce moov-atom errors downstream. (Scripts write atomically as a second line of defense.)
• Prefer script defaults; drop to manual ffmpeg (recipes reference) only when defaults fail.
• Report render progress on long jobs and validate outputs (ffprobe duration/streams) before declaring success.