tessl/pypi-pip-upgrader

An interactive pip requirements upgrader that also updates the version in your requirements.txt file

—

Pending

Overview

Eval results

Files

Package Parsing

Name: tessl/pypi-pip-upgrader
Author: tessl

Extraction and parsing of package specifications from requirements files, handling various formats including extras, comments, and pip options.

Capabilities

Package Detection from Requirements Files

Parses packages from requirements files, extracting clean package specifications while filtering out comments, pip options, and other non-package lines.

class PackagesDetector:
    def __init__(self, requirements_files):
        """
        Initialize packages detector with requirements files.
        
        Args:
            requirements_files (list): List of requirements file paths to parse
        """
    
    def get_packages(self):
        """
        Return the list of detected packages from all requirements files.
        
        Returns:
            list: List of package specification strings (e.g., "django==3.2.0")
        """
    
    def detect_packages(self, requirements_files):
        """
        Parse packages from all provided requirements files.
        
        Args:
            requirements_files (list): List of requirements file paths to process
            
        Side effects:
            Populates self.packages with parsed package specifications
        """

Requirements Line Processing

Processes individual lines from requirements files, handling various formats and filtering out non-package content.

def _process_req_line(self, line):
    """
    Process a single line from a requirements file.
    
    Args:
        line (str): Single line from requirements file
        
    Processing rules:
    - Skip empty lines and whitespace-only lines
    - Skip comment lines (starting with #)
    - Skip pip options (-f, --find-links, -i, --index-url, etc.)
    - Handle inline comments by extracting package part
    - Add valid package specifications to self.packages
    """

Supported Package Formats

Standard Package Specifications

django==3.2.0           # Exact version pin
requests>=2.25.1        # Minimum version (note: only == is processed for upgrades)
flask~=2.0.0           # Compatible version

Package Extras

django[rest]==3.2.0     # Package with extras
requests[security,socks]==2.25.1  # Multiple extras

Complex Specifications

git+https://github.com/user/repo.git#egg=package  # VCS packages (filtered out)
-e .                    # Editable installs (filtered out)
./local/package         # Local packages (filtered out)

Filtered Content

The parser filters out various pip options and non-package lines:

Pip Options (Filtered Out)

-f http://example.com/packages/     # --find-links
--find-links http://example.com/    
-i http://pypi.example.com/         # --index-url
--index-url http://pypi.example.com/
--extra-index-url http://extra.com/
--no-index
-r other-requirements.txt           # -r inclusions (handled by RequirementsDetector)
-Z                                  # --always-unzip
--always-unzip

Comments and Inline Comments

# This is a comment line              # Filtered out
django==3.2.0  # This is inline      # Extracts: django==3.2.0

Empty Lines

# Filtered out

django==3.2.0                       # Processed
                                     # Filtered out

Usage Examples

Basic Package Detection

from pip_upgrader.packages_detector import PackagesDetector

# Parse packages from requirements files
filenames = ['requirements.txt', 'requirements/dev.txt']
detector = PackagesDetector(filenames)
packages = detector.get_packages()

print(packages)
# Output: ['django==3.2.0', 'requests==2.25.1', 'pytest==6.2.4']

Example Requirements File Processing

Given a requirements file:

# requirements.txt
django==3.2.0
requests>=2.25.1  # HTTP library
flask~=2.0.0

# Development dependencies
-r requirements/base.txt
pytest==6.2.4
black==21.5.4  # Code formatter

# Index configuration  
-i https://pypi.org/simple/
--extra-index-url https://test.pypi.org/simple/

# Empty line above and below

celery[redis]==5.1.0

The PackagesDetector would extract:

[
    'django==3.2.0',
    'requests>=2.25.1',  
    'flask~=2.0.0',
    'pytest==6.2.4',
    'black==21.5.4',
    'celery[redis]==5.1.0'
]

Integration with Requirements Detection

from pip_upgrader.requirements_detector import RequirementsDetector
from pip_upgrader.packages_detector import PackagesDetector

# First detect requirements files
req_detector = RequirementsDetector(None)  # Auto-detect
filenames = req_detector.get_filenames()

# Then parse packages from those files
pkg_detector = PackagesDetector(filenames)
packages = pkg_detector.get_packages()

print(f"Found {len(packages)} packages in {len(filenames)} files")

Line Processing Details

Comment Handling

The parser handles various comment scenarios:

# Full line comment - skipped
django==3.2.0          # Package with inline comment - extracts package
    # Indented comment - skipped  
django==3.2.0 # comment with multiple # symbols - extracts package

Whitespace Handling

"django==3.2.0"        # Standard format
"  django==3.2.0  "    # Leading/trailing whitespace stripped
"\tdjango==3.2.0\n"    # Tabs and newlines handled

Pip Options Detection

The parser identifies and filters pip options using prefix matching:

# These lines are filtered out:
"-f http://example.com"
"--find-links http://example.com"  
"-i http://pypi.example.com"
"--index-url http://pypi.example.com"
"--extra-index-url http://extra.com"
"--no-index"
"-r base.txt"
"-Z"
"--always-unzip"

Error Handling

File read errors: Handled by the calling code (RequirementsDetector validates files first)
Invalid package specifications: Invalid lines are processed as-is and may cause issues in later pipeline stages
Encoding issues: Files are read with default encoding; encoding errors may cause parsing failures
Empty files: Gracefully handled, results in empty packages list

Output Format

The detector returns a list of string package specifications exactly as they appear in the requirements files (minus comments and whitespace). The format is compatible with pip and follows the requirements file specification:

package==1.0.0 (exact version)
package>=1.0.0 (minimum version)
package~=1.0.0 (compatible version)
package[extra1,extra2]==1.0.0 (with extras)

Note: For upgrade detection, the system primarily works with == specifications, as these provide the current pinned version needed for comparison with available updates.

Install with Tessl CLI