or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

browser.mdforms.mdindex.mdnavigation.mdutilities.md
tile.json

tessl/pypi-mechanicalsoup

A Python library for automating interaction with websites, providing web scraping and form submission capabilities

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/mechanicalsoup@1.4.x

To install, run

npx @tessl/cli install tessl/pypi-mechanicalsoup@1.4.0

index.mddocs/

MechanicalSoup

A Python library for automating interaction with websites. MechanicalSoup provides a simple API for web scraping and form submission, built on top of the popular Requests library for HTTP sessions and BeautifulSoup for HTML parsing. It automatically handles cookies, redirects, and can follow links and submit forms without JavaScript execution.

Package Information

  • Package Name: mechanicalsoup
  • Language: Python
  • Installation: pip install MechanicalSoup

Core Imports

import mechanicalsoup

Common usage patterns:

from mechanicalsoup import StatefulBrowser, Browser, Form
from mechanicalsoup import LinkNotFoundError, InvalidFormMethod

Basic Usage

import mechanicalsoup

# Create a browser instance
browser = mechanicalsoup.StatefulBrowser()

# Open a webpage
browser.open("https://httpbin.org/forms/post")

# Select and fill a form
browser.select_form('form[action="/post"]')
browser["custname"] = "John Doe"
browser["custtel"] = "555-1234"

# Submit the form
response = browser.submit_selected()
print(response.text)

# Navigate using links
browser.open("https://httpbin.org/")
links = browser.links()
if links:
    browser.follow_link(links[0])
    print(f"Now at: {browser.url}")

Architecture

MechanicalSoup provides a layered architecture for web automation:

  • Browser: Low-level HTTP browser with BeautifulSoup integration for basic request/response handling
  • StatefulBrowser: High-level browser that maintains navigation state, handles forms, and provides convenient web interaction methods
  • Form: HTML form manipulation class for filling fields and preparing submissions
  • Utilities: Exception classes and helper functions for error handling and form analysis

This design enables both simple scripting for basic web scraping and sophisticated automation workflows for complex multi-step interactions.

Capabilities

Core Browser Operations

Low-level HTTP browser functionality providing direct request/response handling with automatic BeautifulSoup parsing. Handles sessions, cookies, and basic web interactions.

class Browser:
    def __init__(self, session=None, soup_config=None, requests_adapters=None, 
                 raise_on_404=False, user_agent=None): ...
    def get(self, *args, **kwargs): ...
    def post(self, *args, **kwargs): ...
    def submit(self, form, url=None, **kwargs): ...

Core Browser Operations

Stateful Web Navigation

High-level browser that maintains page state and provides convenient methods for navigation, link following, and multi-step web interactions. Recommended for most applications.

class StatefulBrowser(Browser):
    @property
    def page(self): ...  # Current page BeautifulSoup object
    @property 
    def url(self): ...   # Current page URL
    @property
    def form(self): ...  # Currently selected form
    
    def open(self, url, *args, **kwargs): ...
    def select_form(self, selector="form", nr=0): ...
    def follow_link(self, link=None, **kwargs): ...

Stateful Web Navigation

Form Handling

HTML form manipulation and field setting capabilities. Supports all standard form elements including inputs, checkboxes, radio buttons, selects, and textareas.

class Form:
    def __init__(self, form): ...  # form is bs4.element.Tag
    def set(self, name, value, force=False): ...
    def __setitem__(self, name, value): ...
    def set_checkbox(self, data, uncheck_other_boxes=True): ...
    def set_select(self, data): ...

Form Handling

Utilities and Error Handling

Exception classes and utility functions for error handling and form analysis.

class LinkNotFoundError(Exception): ...
class InvalidFormMethod(LinkNotFoundError): ...

def is_multipart_file_upload(form, tag): ...

Utilities and Error Handling

Types

# Session and configuration types
from typing import Optional, Dict, Any, List, Union
from requests import Session
from bs4 import BeautifulSoup, Tag

# Common parameter types
SoupConfig = Dict[str, Any]
RequestsAdapters = Dict[str, Any]