Handle websites requiring JavaScript by using curl with browser headers and validating file types.
71%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Fix and improve this skill with Tessl
tessl review fix ./benchmarks/gdpval/skills/http-response-handling/SKILL.mdUse this technique when you need to fetch content from websites that:
Use curl with a realistic User-Agent header to mimic a real browser:
curl -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -o output.html "https://example.com"Key flags:
-L — Follow redirects-A — Set User-Agent header to mimic a real browser-o — Save output to file for inspectionAfter fetching, check if you received a placeholder response instead of actual content:
# Check file size (placeholder responses are often very small)
wc -c output.html
# Check for common placeholder indicators
grep -i "javascript" output.html | head -5
grep -i "loading" output.html | head -5
grep -i "noscript" output.html | head -5Signs of a placeholder response:
Before attempting format-specific parsing, validate the file type:
# Check the file type
file output.html
# Check the actual content type (if you have the headers)
curl -I -A "Mozilla/5.0 ..." "https://example.com" | grep -i content-type
# Inspect first few lines
head -50 output.htmlCommon checks:
<!DOCTYPE or <html{ or [%PDFIf you got valid HTML content:
# Proceed with HTML parsing or extraction
grep -oP '(?<=<title>).*?(?=</title>)' output.htmlIf you got a placeholder/JS-dependent response:
If you got an unexpected file type:
# Check what was actually returned
file output.html
# Adjust your approach based on actual content
case $(file -b --mime-type output.html) in
"text/html")
# Parse as HTML
;;
"application/json")
# Parse as JSON
;;
"application/pdf")
# Handle as PDF
;;
*)
echo "Unexpected file type: $(file -b --mime-type output.html)"
;;
esac#!/bin/bash
# fetch-with-validation.sh
URL="$1"
OUTPUT="${2:-output.html}"
# Fetch with browser headers
curl -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -o "$OUTPUT" "$URL"
# Get file info
SIZE=$(wc -c < "$OUTPUT")
TYPE=$(file -b --mime-type "$OUTPUT")
echo "Downloaded: $OUTPUT"
echo "Size: $SIZE bytes"
echo "Type: $TYPE"
# Warn about potential issues
if [ "$SIZE" -lt 1000 ]; then
echo "WARNING: File is very small - may be a placeholder response"
fi
if [ "$TYPE" = "text/html" ]; then
if grep -qi "javascript\|loading\|spinner" "$OUTPUT"; then
echo "WARNING: Content may be JavaScript-dependent"
fi
fiHKUDS/OpenSpace
3827781
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.