Podcast → Transcript → Obsidian

Purpose

Manual-trigger pipeline that detects new podcast episodes via Spotify MCP, downloads audio via RSS enclosures, transcribes locally using faster-whisper, and generates structured Obsidian notes with summaries, key ideas, quotes, and backlinks.

When to Use

Prerequisites

• Spotify MCP configured in VS Code (see Setup section below) — only for podcast detection mode
• Obsidian running with CLI enabled (Settings → General → CLI)
• faster-whisper installed (pip install faster-whisper)
• feedparser installed (pip install feedparser)
• yt-dlp + ffmpeg for --url mode (pip install yt-dlp, plus ffmpeg on PATH)
• Python 3.10+

Quick Start

# Full pipeline — detect, download, transcribe, write to vault
python .github/skills/podcast-to-obsidian/scripts/pipeline.py

# Check for new episodes only (no download/transcribe)
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --check-only

# Process a specific show
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --show "Show Name"

# Dry run — downloads & transcribes but SKIPS vault write only
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --dry-run

# Process a single episode by title substring
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --episode "Nasdaq"

# Add a new show manually (without Spotify)
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --add-show --name "My Show" --rss "https://example.com/feed.xml"

# Keep audio files after run (default: purge after success)
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --keep-audio

# List all tracked shows
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --list-shows

# Transcription model selection (default: large-v3)
# Use --model base only when speed matters more than accuracy; it garbles
# proper nouns badly and those errors propagate into the generated note.
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --model base

# Retry failed episodes
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --retry-failed

# URL mode — one-off clip from X / YouTube / Vimeo / etc. (uses yt-dlp, bypasses manifest)
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --url "https://x.com/handle/status/123/video/1"

# URL mode + custom folder + title override
python .github/skills/podcast-to-obsidian/scripts/pipeline.py --url "https://youtu.be/abc" --clips-folder "Clips" --title "My Clean Title"

Pipeline Flow

1. DETECT    → Spotify MCP or fetch_webpage scrapes episode metadata
2. DIFF      → Compare episode IDs against manifest (skip processed)
3. CONFIRM   → Show user what's new, ask which to process
4. DOWNLOAD  → Fetch audio via RSS <enclosure> URL → .work/audio/
5. TRANSCRIBE → faster-whisper (local GPU/CPU) → .work/transcripts/
6. GENERATE  → AI summary (Claude CLI / OpenAI) + structured note (automatic)
7. WRITE     → Obsidian skill pipes note to vault
8. MANIFEST  → Update manifest only after successful write
9. CLEANUP   → Purge .mp3 audio files + intermediate build artifacts
10. SYNC     → Leave Obsidian running so Sync can push the new notes

Step 10 — SYNC: hand the notes to Obsidian Sync

Notes are written to the vault filesystem, and Obsidian Sync only pushes while the desktop app runs. A 02:30 scheduled run finds the app closed, so without this step the notes stay on the machine that wrote them — and a phone opening a deep link to one gets "file not found", correctly, because the file is not there. That is precisely what happened on 2026-09-19.

hand_off_to_sync() runs after the writes and prints one [sync] line. It is skipped on --dry-run and when nothing was written, never raises, and gives Sync a window rather than a guarantee — do not report a push as confirmed on the strength of it. Set OBSIDIAN_AUTOLAUNCH=0 to keep the app closed and accept the lag.

Step 1 — DETECT: Fallback Detection

Spotify MCP (SpotifyGetInfo) does not support episode URIs — it returns "Unknown qtype episode". When given a Spotify episode URL:

1. Try Spotify MCP first (may work for show-level queries)
2. If it fails, use fetch_webpage on the Spotify episode URL to scrape:
   • Episode title, show name, publish date, duration, description
3. Match the episode to a tracked show in config.json via show name
4. Use the show's RSS feed to find the audio enclosure URL

Step 6 — GENERATE: Structured Note from Transcript

Long transcripts are chunked, never truncated. Each backend used to slice the transcript to its first 12,000 words, so a 2h45m episode (~29k words) was summarized from its first 41% and the back half silently vanished from the note. Now note_generator splits anything over ~9k words into overlapping segments, extracts per-segment notes (the map phase), then synthesizes them into the final summary (the reduce phase). Requested item counts also scale with runtime — a three-hour panel show asks for ~22-32 key ideas and 6-7 deep dives, where a 20-minute interview asks for far fewer.

If a segment fails, the whole backend is abandoned rather than producing a summary with a hole in it. The parsed result is validated for required keys before use, so a response cut off by an output-token limit can no longer be salvaged into a quietly incomplete note.

This step is automatic. step_generate_notes() resolves a summary in this order and writes the note itself:

1. A pre-generated summary at .work/summaries/<transcript-stem>.json, if the orchestrator wrote one
2. Otherwise generate_ai_summary() — Claude CLI first, then the OpenAI API (OPENAI_API_KEY)
3. If neither is available the episode is failed, not skeleton-written — pass --no-ai explicitly if you actually want a template-only note

The pipeline then writes the note to the vault and updates the manifest. A normal run needs no agent intervention.

When an agent IS in the loop

An orchestrating agent adds value by auditing the generated note against the transcript, not by regenerating it. Do not write a second note — the manifest is already marked completed and the vault file already exists. Instead:

1. Read the generated note and the full transcript
2. Spot-check the back third. Chunked summarization means the model now sees the whole episode, so wholesale gaps should be gone — but verify, since this is where failures historically showed up
3. Verify speaker attribution on quotes (small models produce bare or wrong first names, and panel shows referred to only by first name — "Alex", "Dave" — need resolving to full names for the [[People/...]] links)
4. Verify [[People/...]] links resolve to real people, not homophones
5. Patch gaps in place with targeted edits

Alternatively, pre-generate the summary yourself into .work/summaries/<transcript-stem>.json before running the pipeline, and it will be used verbatim.

Writing a note by hand (fallback only)

1. Read the full transcript from .work/transcripts/<YYYY-MM-DD> - <Title>.txt
2. Identify speakers, key themes, and structure
3. Write the note following the Obsidian Note Structure template above:

• Frontmatter with tags, show, episode, dates, source: podcast-to-obsidian, and spotify_url when available
• Keep the episode title unsanitized inside the note; sanitize only the filename/path
• TL;DR as an Obsidian abstract callout
• Key Ideas as a numbered list with bolded labels and explanations
• Deep Dives (3-5 mini-essays on the most important/surprising concepts — analysis, implications, connections, what wasn't said. NOT a summary rehash.)
• Actionable Takeaways (checkbox items)
• Memorable Quotes as quote callouts with speaker attribution
• People & Topics (wiki-links: [[People/Name]], [[Topics/Topic]], [[Companies/Org]])
• Use the same section separators and layout as the generated final notes in .work/notes/

4. Pipe the note to the obsidian skill:

   $noteContent = @'
   <generated note content>
   '@ | python .github/skills/obsidian/scripts/obsidian.py create --path "Podcasts/<Show>/<YYYY-MM-DD> - <Title>.md"

5. Verify the write through the Obsidian wrapper instead of writing directly to the vault filesystem:

python .github/skills/obsidian/scripts/obsidian.py read --path "Podcasts/<Show>/<YYYY-MM-DD> - <Title>.md"

URL Mode (one-off clips)

For single web videos that aren't tracked podcast episodes (X tweets, YouTube videos, Vimeo clips, etc.), use --url to bypass RSS detection and the manifest:

python .github/skills/podcast-to-obsidian/scripts/pipeline.py --url "<web video URL>"

How it differs from podcast mode:

Title derivation: yt-dlp doesn't expose a clean title for X tweets — it uses the tweet description. The pipeline auto-derives a note title by stripping the <uploader> - prefix, dropping trailing t.co/... links, and keeping the first sentence (~100 chars). Use --title to override.

Source label / folder: auto-derived from extractor + uploader, e.g. X — @servasyy_ai, YouTube — @somechannel. Override with --show-name.

Playlists (X tweets with multiple videos): the pipeline always picks the first video. There's no equivalent of /video/N selection — re-run with a more specific URL if needed.

URL-mode flags:

New Releases Only

The pipeline never walks backwards into a show's archive. Backfill is a deliberate manual act, never something a scheduled run starts doing on its own.

Two gates apply together during detection:

The watermark advances only on status: completed — a failed episode must not raise the bar, or its retry would be filtered out as historical next run.

Episodes with no publish date in the feed are skipped while gating is active: recency can't be established, so they can't be confirmed as new releases. (Undated entries previously bypassed every gate silently.)

Drawing a line in the sand

# Everything currently in every feed becomes historical.
# Only genuinely future releases are auto-fetched from here on.
python .../pipeline.py --set-watermark today

# Or per show, or to a specific date
python .../pipeline.py --set-watermark 2026-07-18 --show "Bankless"

--set-watermark never lowers an existing watermark unless --force is given, so it can't silently re-open a backlog you already closed.

Manual backfill

# Consider episodes published on or after a date, ignoring the watermark
python .../pipeline.py --backfill-since 2026-06-01 --show "Bankless"

# Target one specific episode
python .../pipeline.py --backfill-since 2026-06-01 --episode "Nasdaq"

# Everything in the detection window, no date gate at all
python .../pipeline.py --ignore-watermark --show "Naval" --max-episodes 1

Backfill still respects max_episodes, so a large archive drains in controlled batches rather than all at once.

Adopting this on an existing manifest

# Set each show's watermark from its newest COMPLETED episode
python .../pipeline.py --seed-watermarks

Use this when a manifest predates watermarks. It reprocesses nothing — it just records where each show had already got to. Shows with no completed episodes fall back to max_age_days on their first run.

Manifest

Persistent JSON file tracking all processed episodes.

Location: config/podcast-manifest.json

Rules:

• Spotify episode ID is the primary key (RSS-only entries are keyed by RSS GUID)
• Dedup matches by key, rss_guid/spotify_id field, or normalized title + published date — the same episode may be keyed by Spotify ID (Spotify detection) or RSS GUID (RSS detection), and a match on ANY of these means already processed. When diffing via Spotify MCP, compare title + published date against manifest entries, not just IDs.
• If a match exists → skip
• If not → process and append
• Manifest updated ONLY after successful Obsidian write
• Supports manual RSS-only entries (no Spotify ID required)
• The pipeline git-commits the manifest itself when a run changes it. Without this, every successful run permanently dirties the working tree and its diff gets swept into whatever unrelated commit lands next. The commit is deliberately narrow: it uses a pathspec so a staged index and other working-tree edits are left alone, it never pushes, and a git failure is logged rather than failing a run whose real work already succeeded. Opt out with --no-commit-manifest
• Each show carries latest_published, the release watermark (see New Releases Only). It advances only on completed, never on failed, and never moves backwards

Obsidian Note Structure

Each episode produces a note at: Podcasts/<Show Name>/<YYYY-MM-DD> - <Episode Title>.md

Filename rule:

• Use the episode's published date plus title
• Strip invalid filename characters < > : " / \ | ? * AND Obsidian wikilink-breaking characters # ^ [ ] (a # in the filename breaks the [[path|label]] links in the show index)
• Trim trailing . and space after stripping
• Truncate the title portion to about 120 chars if needed
• Do not slugify the filename

The transcript is also written to the vault at Podcasts/<Show Name>/transcripts/<same filename>.md (the Obsidian CLI only creates .md notes) and that path is recorded in the manifest. If the vault copy fails, the manifest records the real .work path instead — never a path that does not exist.

---
tags: [podcast, <show-slug>, <topic-tag-1>, <topic-tag-2>]
type: podcast-note
show: "<Show Name>"
episode: "<Episode Title>"
published: YYYY-MM-DD
duration: "HH:MM:SS"
spotify_url: "<url>"
source: podcast-to-obsidian
created: YYYY-MM-DDTHH:MM:SSZ
---

# <Episode Title>

**Show:** [[<parent folder>/<Show Name>]] · 📅 YYYY-MM-DD · ⏱ HH:MM:SS
<!-- parent folder is `podcasts_folder` in podcast mode and `clips_folder`
     in URL mode — it must match where the note is written or the link
     dead-ends. Passed to generate_note() as `parent_folder`. -->

---

> [!abstract]+ TL;DR
> <2-3 sentence summary>

---

## 💡 Key Ideas

1. **Idea 1** — explanation
2. **Idea 2** — explanation
3. ...

---

## 🧠 Deep Dives

### Concept Title

2-4 paragraph mini-essay analyzing this concept in depth —
implications, connections between ideas, what wasn't said,
why it matters beyond the podcast. Pick 3-5 of the most
important/surprising concepts. Do NOT repeat Key Ideas;
add new depth and perspective.

---

## ✅ Actionable Takeaways

- [ ] Action item 1
- [ ] Action item 2

---

## 💬 Key Quotes

> [!quote] "Quote text"
> — **Speaker Name**

---

## 🔗 People & Topics

**People:** [[People/<Name>]] · [[People/<Name 2>]]

**Topics:** [[Topics/<Topic>]] · [[Topics/<Topic 2>]]

**Companies:** [[Companies/<Org>]] · [[Companies/<Org 2>]]

---

## Transcript

<collapsible full transcript>

Working Directory

work_dir resolves against SKILL_DIR, not the current working directory or the repo root. With the default .work, everything lives under:

.github/skills/podcast-to-obsidian/.work/
├── audio/         # .mp3 (+ .part during download), purged after success
├── transcripts/   # raw .txt from whisper + .meta.json completion sidecars
├── summaries/     # optional pre-generated summary JSON (see Step 6)
├── notes/         # intermediate .final.md build artifacts
├── logs/          # timestamped run logs, newest 30 retained
└── pipeline.lock  # PID lock held for the duration of a full run

Running python .../pipeline.py from the repo root does not create <repo>/.work — look under the skill directory. This has cost debugging time more than once.

Configuration

Edit scripts/config.json:

Transcription vocabulary

config/vocabulary.json holds domain terms and is applied two ways:

• initial_prompt primes the Whisper decoder toward correct spellings (prevention). Add recurring hosts, guests, companies, and jargon here.
• corrections are word-boundary, case-insensitive replacements applied to the finished transcript (cure). Keep entries phrase-scoped — a bare single word like opioid or England must never be replaced, because both have legitimate uses in the same episodes.

Terms too ambiguous to auto-correct are parked under _risky_not_applied as documentation rather than silently guessed at.

Run Logs

Every invocation tees stdout and stderr to .work/logs/<YYYYMMDD-HHMMSS>.log, including tracebacks, and prints the path on exit. This is independent of shell redirection, so scheduled and detached runs always leave a debuggable artifact. The newest 30 logs are kept.

Download progress renders as a live bar only when stdout is a TTY; redirected output gets one line per 10% instead, which keeps logs small and readable.

Spotify MCP Setup

Step 1 — Create Spotify Developer App

1. Go to https://developer.spotify.com/dashboard
2. Create a new app
3. Note your Client ID and Client Secret
4. Set Redirect URI to http://localhost:8888/callback

Step 2 — Install Spotify MCP Server

# Clone and install
git clone <spotify-mcp-repo>
cd spotify-mcp
npm install

Step 3 — Environment Variables

Add to your .env or system environment:

SPOTIFY_CLIENT_ID=<your-client-id>
SPOTIFY_CLIENT_SECRET=<your-client-secret>
SPOTIFY_REDIRECT_URI=http://localhost:8888/callback

Step 4 — Register in VS Code

Add to .vscode/settings.json or user settings:

{
  "mcp.servers": {
    "spotify": {
      "command": "node",
      "args": ["path/to/spotify-mcp/index.js"]
    }
  }
}

Restart VS Code after configuration.

Known Issues

CLI Flag Reference

Troubleshooting

Dependencies

• Composes with obsidian skill for all vault I/O
• faster-whisper for local transcription (GPU recommended)
• feedparser for RSS parsing
• Python 3.10+ (stdlib + 2 pip packages)
• Obsidian must be running with CLI enabled
• Spotify MCP server (optional — can use RSS-only mode)

Related Skills

• obsidian — Vault operations (composed — required)
• obsidian-vault-digest — Query vault for prior podcast knowledge
• obsidian-vault-linker — Link podcast notes to related content

Step N: Reflection (composable)

Invoke the skill-reflection skill with the following context:

• Calling skill: podcast-to-obsidian
• SKILL.md path: .github/skills/podcast-to-obsidian/SKILL.md
• Steps completed: list each step with pass/fail/skipped
• Friction notes: any workarounds, retries, unexpected errors, or manual interventions