Firecrawl Production Checklist

Overview

Pre-deployment validation checklist for applications using Firecrawl's scrape, crawl, map, and extract APIs. Covers credential management, crawl safety limits, error handling, monitoring, and rollback.

Prerequisites

Staging environment tested and passing
Production API key from firecrawl.dev/app
Monitoring infrastructure ready

Pre-Deployment Checklist

Credentials & Security

Production FIRECRAWL_API_KEY in secure vault (not in code or .env)
Key starts with fc- and is scoped to production
Different API keys for dev/staging/production
.env files in .gitignore
Webhook secrets stored securely
Git history scanned for leaked keys

Crawl Safety

All crawlUrl calls have limit parameter set
maxDepth configured to prevent unbounded crawling
includePaths / excludePaths filters applied where appropriate
Credit budget tracking implemented (daily limit alerts)
No hardcoded URLs in production code

Error Handling

429 rate limit handling with exponential backoff
402 credit exhaustion handled gracefully (no crash)
401 auth failure logged and alerted
Async crawl jobs have timeout with deadline
Fallback from crawl to individual scrape on failure
Empty markdown detection (JS rendering issues)

Monitoring & Alerting

Scrape success/failure rate tracked
Credit consumption monitored
Crawl job completion rate tracked
Alert on credit balance below threshold
Alert on error rate > 5%
Webhook delivery failures logged

Instructions

Step 1: Verify API Connectivity

set -euo pipefail
# Test production key
curl -s https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}' | jq '.success'

# Check credit balance
curl -s https://api.firecrawl.dev/v1/team/credits \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" | jq .

Step 2: Health Check Endpoint

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

export async function healthCheck() {
  const start = Date.now();
  try {
    const result = await firecrawl.scrapeUrl("https://example.com", {
      formats: ["markdown"],
    });
    return {
      status: result.success ? "healthy" : "degraded",
      latencyMs: Date.now() - start,
      hasContent: (result.markdown?.length || 0) > 0,
    };
  } catch (error: any) {
    return {
      status: "unhealthy",
      latencyMs: Date.now() - start,
      error: error.statusCode || error.message,
    };
  }
}

Step 3: Production-Safe Crawl Wrapper

export async function productionCrawl(url: string, opts: {
  maxPages: number;
  paths?: string[];
  timeout?: number;
}) {
  // Hard credit safety — never exceed configured limit
  const limit = Math.min(opts.maxPages, 500);

  const job = await firecrawl.asyncCrawlUrl(url, {
    limit,
    maxDepth: 3,
    includePaths: opts.paths,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });

  // Poll with timeout
  const deadline = Date.now() + (opts.timeout || 600000);
  let pollInterval = 2000;
  let status = await firecrawl.checkCrawlStatus(job.id);

  while (status.status === "scraping" && Date.now() < deadline) {
    await new Promise(r => setTimeout(r, pollInterval));
    pollInterval = Math.min(pollInterval * 1.5, 30000);
    status = await firecrawl.checkCrawlStatus(job.id);
  }

  if (status.status !== "completed") {
    throw new Error(`Crawl ${job.id} did not complete: ${status.status}`);
  }
  return status;
}

Step 4: Rollback Procedure

set -euo pipefail
# Immediate rollback — disable Firecrawl integration
kubectl set env deployment/app FIRECRAWL_ENABLED=false
kubectl rollout restart deployment/app

# Verify rollback
curl -s https://app.example.com/health | jq '.services.firecrawl'

Alerting Rules

Alert	Condition	Severity
API unreachable	Health check fails 3x	P1
Credits < 1000	Balance check	P2
Error rate > 5%	429/5xx rate	P2
Crawl timeout	Job stuck > 10min	P3
Auth failure	Any 401 response	P1

Resources

Next Steps

For version upgrades, see firecrawl-upgrade-migration.

firecrawl-prod-checklist