File Deduplication Tool

Build a command-line tool that scans a directory for duplicate files using fast hash-based comparison.

Requirements

The tool should:

1. Accept a directory path as a command-line argument
2. Recursively scan all files in the directory
3. Compute a hash fingerprint for each file's contents
4. Group files by their hash values to identify duplicates
5. Output a report showing which files are duplicates of each other

Output Format

For each group of duplicate files (files with identical content), print:

Duplicates found (hash: <hash_value>):
  - <file_path_1>
  - <file_path_2>
  - <file_path_3>

If no duplicates are found, print:

No duplicate files found.

Implementation Requirements

• Use the 64-bit hash variant for better collision resistance
• Process files efficiently by reading them completely before hashing
• Skip empty files or handle them appropriately
• Use hexadecimal output format for hash values in the report

Test Cases

• Given a directory with two files containing identical text "Hello World", the tool identifies them as duplicates @test
• Given a directory where all files have unique content, the tool reports no duplicates @test
• Given a directory with three files where two are identical and one is different, the tool identifies only the two identical files as duplicates @test

@generates

API

def find_duplicates(directory_path: str) -> dict[str, list[str]]:
    """
    Scan a directory for duplicate files.

    Args:
        directory_path: Path to the directory to scan

    Returns:
        Dictionary mapping hash values to lists of file paths with that hash
    """
    pass

def main():
    """Command-line entry point."""
    pass

if __name__ == "__main__":
    main()

Dependencies { .dependencies }

xxhash { .dependency }

Provides fast non-cryptographic hashing support.