Multi-threaded Web Scraper with Cache

Build a multi-threaded web scraper that safely caches HTTP responses to avoid redundant requests when multiple threads access the same URLs.

Requirements

Your solution must:

1. Use a cached HTTP session to store responses
2. Fetch multiple URLs concurrently using Python threads
3. Ensure thread-safe cache access when multiple threads read/write simultaneously
4. Return all fetched content in a dictionary mapping URLs to response text

Implementation

@generates

Create a Python module that exports a scrape_urls function.

API

def scrape_urls(urls: list[str], num_threads: int = 5) -> dict[str, str]:
    """
    Scrape multiple URLs concurrently with safe caching.

    Args:
        urls: List of URLs to scrape
        num_threads: Number of concurrent threads to use

    Returns:
        Dictionary mapping URLs to their response content
    """
    pass

Test Cases

• When multiple threads request the same URL, only one actual HTTP request is made @test
• When 5 threads access different URLs simultaneously, all responses are cached correctly @test
• When the same URL is in the list multiple times, the cached response is reused @test

Dependencies { .dependencies }

requests-cache { .dependency }

Provides HTTP caching with thread-safe operations.

threading { .dependency }

Provides thread support for concurrent execution.