Captions (Whisper) & AI Assist (Gemini)

Captions with Whisper

Recommended: mlx-whisper (Apple Silicon)

Fastest local option on M-series Macs (uses the Neural Engine/GPU via MLX):

pip install mlx-whisper
mlx_whisper work/master.mp4 --model mlx-community/whisper-large-v3-turbo \
  --output-format srt --output-dir work/

Model guidance: whisper-large-v3-turbo is the sweet spot (near-large accuracy, ~6x faster). For quick drafts: whisper-base.en. Talking-head reels with music underneath transcribe fine; extract voice-only audio first if music bleeds into the transcript: ffmpeg -i master.mp4 -af "highpass=f=120,lowpass=f=8000" -vn voice.wav

Alternative: whisper.cpp (brew install whisper-cpp) — similar quality, CLI: whisper-cli -m ggml-large-v3-turbo.bin -f voice.wav --output-srt.

Short-form caption style

Raw Whisper SRT lines are too long for reels. Reformat before burn-in: 1–4 words per caption event, punchy, synced to speech rhythm. Edit the SRT (each event = one line of 1–4 words) or ask Whisper for word-level timestamps (--word-timestamps True) and regroup.

Burn in via render_reel.py --srt captions.srt. Default style is white bold with black outline, centered, safe-zone bottom margin (MarginV=60 keeps clear of platform UI). Adjust in the script's force_style string. Keep captions inside the central ~80% horizontally — platform UI overlays eat the edges.

Gemini API Assist

Requires an API key in GEMINI_API_KEY (or GOOGLE_AI_API_KEY / GOOGLE_API_KEY — scripts check all three). Use the REST API directly (no SDK needed):

curl -s "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent" \
  -H "x-goog-api-key: $GEMINI_API_KEY" -H 'Content-Type: application/json' \
  -d '{"contents":[{"parts":[{"text":"PROMPT"}]}]}'

Model names change; if a 404 comes back, list models: GET /v1beta/models.

Use case 1 — Hook lines & on-screen text

Feed it the cut plan notes + what the reel shows; ask for 5 hook-text options (<6 words each, no clickbait). Present options to the user. To burn the winner in, do NOT use drawtext — Homebrew ffmpeg ships without it. Use the subtitles filter with a one-event SRT instead (same mechanism as captions, styled via force_style):

printf '1\n00:00:00,000 --> 00:00:02,500\nHOOK TEXT\n' > work/hook.srt
# then: render_reel.py --srt work/hook.srt   (adjust force_style in the script for size/position)

The winning hook line also feeds --thumb-text for the magazine-cover thumbnails (Use case 3).

Use case 2 — Captions & hashtags per platform

Prompt with: reel content summary, target audience, platform. Ask for platform-specific length (TikTok short + 3-5 tags; IG longer storytelling + tags; YT Shorts title + description). Write results to exports/captions.md. If the user has a brand voice document, include it in the prompt.

Use case 3 — Stylized magazine-cover thumbnails (stylize_thumbnail.py)

Authenticity rule: the cover is always a real frame grab, restyled — Nano Banana edits the frame (identity, pose, and composition preserved by prompt), it doesn't invent a new image. export_variants.py runs this automatically when an API key is set; run it directly for control:

python3 scripts/stylize_thumbnail.py                            # both styles, photo treatment
python3 scripts/stylize_thumbnail.py --style 80s \
    --text "NO EXCUSES" --masthead "HYROX PREP"                 # full cover with typography
python3 scripts/stylize_thumbnail.py --video work/master.mp4 --at 3.2 --style modern

Styles: modern (premium editorial — rim light, cinematic contrast) and 80s (film grain, halftone print texture, faded warmth, registration offset). Prompts are editable constants in the script — add styles there.

Landscape input (16:9/4:3 drone shots, stills) is handled: the script detects it and switches to a reframe prompt — the model recomposes to 9:16 around the subject (verified on real DJI footage; markedly better than a dumb center-crop). Two caveats to check in the output: recomposition synthesizes some background (extended deck/sky), and small-scale details like tiny jersey lettering can get approximated when the subject is far away — prefer frames where the subject is reasonably large. --text switches to full-cover mode with masthead + headline rendered by the model; keep headlines short and check spelling in the output (it's instructed to render verbatim, but verify).

Models (--model): gemini-3-pro-image (Nano Banana Pro, default — best typography and instruction following), gemini-3.1-flash-image (Nano Banana 2 — faster/cheaper, weaker text; fine for photo-only), legacy gemini-2.5-flash-image. Model names drift — on a 404, list current ones: GET /v1beta/models. Same generateContent endpoint as text calls; the image comes back as base64 inlineData.

Always show every variant (plain + styled) to the user and let them pick; check faces before publishing, and note that AI-stylized imagery may warrant platform disclosure. Use generated-from-scratch imagery only for concept covers (e.g., race-day announcements) — never as a stand-in for real training footage.

What NOT to automate

Anything claiming personal experience must come from the user, not the model — the model drafts, the human owns the story. Always show AI-drafted copy for approval before it goes into exports/.