HTML to Markdown Conversion

Convert HTML files to clean Markdown using pandoc.

Quick Start

Single File

pandoc -f html -t markdown input.html -o output.md

With Options

# Wrap lines at 80 characters
pandoc -f html -t markdown --wrap=auto --columns=80 input.html -o output.md

# No line wrapping (preserves original flow)
pandoc -f html -t markdown --wrap=none input.html -o output.md

# GitHub-flavored markdown
pandoc -f html -t gfm input.html -o output.md

Batch Processing

Use scripts/html2markdown.py for batch conversion:

python scripts/html2markdown.py /path/to/root/directory

Options:

• --output-dir: Specify output directory (default: in-place with .md extension)
• --flatten: Flatten all output to single directory
• --wrap: Line wrapping mode (none, auto, preserve)

Output Formats

Note: Standard markdown (-t markdown) may include pandoc-specific syntax like ::: {role="main"} div markers. Use -t gfm for cleaner, more portable output.

Common Issues

Malformed HTML

Pandoc handles most malformed HTML gracefully. For severely broken HTML, preprocess with:

tidy -q -ashtml input.html | pandoc -f html -t markdown -o output.md

Character Encoding

Force UTF-8 encoding:

pandoc -f html -t markdown --metadata encoding=utf-8 input.html -o output.md

Stripping Unwanted Elements

Pandoc converts everything. To strip scripts/styles before conversion:

from bs4 import BeautifulSoup

with open('input.html', 'r') as f:
    soup = BeautifulSoup(f.read(), 'html.parser')

for tag in soup(['script', 'style', 'nav', 'footer', 'header']):
    tag.decompose()

clean_html = str(soup)
# Then pass clean_html to pandoc via stdin

Integration with Preprocessing Pipeline

For course preprocessing workflows, the batch script outputs files suitable for concatenation into a single *_output.txt file with custom separators.