tessl/pypi-sphinx-codeautolink
Automatic links from code examples to reference documentation.
code-processing.mddocs/
Code Block Processing
Code block analysis and transformation system that parses Python code, identifies name usage patterns, and prepares them for link generation. This system handles various Python code formats including standard code blocks, IPython console sessions, and doctests.
Capabilities
Core Parsing Functions
Main functions for extracting name usage from Python source code using AST analysis.
def parse_names(source: str, doctree_node) -> list[Name]:
"""
Parse names from Python source code.
Analyzes the abstract syntax tree of the provided source code to identify
all name accesses, import statements, and their relationships. Tracks
variable assignments and scope to build accurate name chains.
Parameters:
- source: Python source code string to analyze
- doctree_node: Docutils node for error reporting context
Returns:
- list[Name]: List of Name objects representing all identified name accesses
"""
def linenos(node: ast.AST) -> tuple[int, int]:
"""
Return lineno and end_lineno safely from AST nodes.
Provides safe access to line number information from AST nodes,
handling cases where end_lineno might not be available.
Parameters:
- node: AST node to extract line numbers from
Returns:
- tuple[int, int]: Start line number and end line number
"""Code Block Transformers
Functions that clean and normalize different Python code formats for analysis.
def clean_pycon(source: str) -> tuple[str, str]:
"""
Clean up Python console syntax to pure Python.
Converts Python console session format (with >>> and ... prompts)
to clean Python code suitable for AST analysis.
Parameters:
- source: Raw console session text with prompts
Returns:
- tuple[str, str]: (original_source, cleaned_source)
Example:
Input: ">>> import os\\n>>> os.path.join('a', 'b')"
Output: (">>> import os...", "import os\\nos.path.join('a', 'b')")
"""
def clean_ipython(source: str) -> tuple[str, str]:
"""
Clean up IPython magics and console syntax to pure Python.
Processes IPython console session format, removing magic commands
and console prompts while preserving executable Python code.
Requires IPython to be installed.
Parameters:
- source: Raw IPython session text
Returns:
- tuple[str, str]: (original_source, cleaned_source)
Example:
Input: "In [1]: %matplotlib inline\\nIn [2]: import numpy"
Output: ("In [1]: %matplotlib...", "import numpy")
"""Block Analysis System
Core classes for analyzing code blocks and managing transformations.
class SourceTransform:
"""
Represents transforms on source code.
Contains the analysis results for a single code block, including
the source code, extracted names, and metadata for link generation.
"""
source: str # Original source code
names: list[Name] # Parsed name accesses
example: CodeExample # Metadata about the example
doc_lineno: int # Line number in source document
class CodeBlockAnalyser(nodes.SparseNodeVisitor):
"""
Analyzes and transforms Python code blocks with reference links.
Visits code block nodes in the document tree, extracts Python code,
applies transformers, parses names, and prepares for link injection.
"""
def __init__(self, doctree, source_dir: str, global_preface: list[str],
custom_blocks: dict, concat_default: bool,
default_highlight_lang: str, warn_default_parse_fail: bool): ...HTML Link Application
Functions for injecting links into generated HTML output.
def link_html(document: str, out_dir: str, transforms: list[SourceTransform],
inventory: dict, custom_blocks: dict, search_css_classes: list,
builder_name: str = "html") -> None:
"""
Inject links into HTML code blocks on disk.
Post-processes generated HTML files to add clickable links to code
elements based on resolved name mappings. Modifies HTML files in place.
Parameters:
- document: Document name/path
- out_dir: Output directory containing HTML files
- transforms: List of source transformations with resolved names
- inventory: Mapping from names to documentation URLs
- custom_blocks: Custom block type processors
- search_css_classes: Additional CSS classes to search for code blocks
- builder_name: Sphinx builder name (default: "html")
"""Built-in Block Types
Constants defining supported code block language identifiers and their transformers.
BUILTIN_BLOCKS: dict = {
# Standard Python blocks (no transformation needed)
"default": None,
"python": None,
"Python": None,
"python3": None,
"py": None,
"py3": None,
"pyi": None,
"sage": None,
"bazel": None,
"starlark": None,
# Special blocks with transformers
"pycon": clean_pycon, # Python console sessions
"ipython": clean_ipython, # IPython sessions (if available)
"ipython3": clean_ipython
}AST Visitor Implementation
Core visitor class that performs detailed AST analysis to track imports and name usage.
class ImportTrackerVisitor(ast.NodeVisitor):
"""
Track imports and their use through source code.
Visits AST nodes to build a complete picture of how names are
imported, assigned, and accessed throughout the code. Handles
complex scoping rules including nested functions, classes, and
comprehensions.
"""
def __init__(self, doctree_node): ...
# Key visitor methods for different AST node types
def visit_Import(self, node: ast.Import): ...
def visit_ImportFrom(self, node: ast.ImportFrom): ...
def visit_Name(self, node: ast.Name) -> PendingAccess: ...
def visit_Attribute(self, node: ast.Attribute) -> PendingAccess: ...
def visit_Call(self, node: ast.Call) -> PendingAccess: ...
def visit_Assign(self, node: ast.Assign) -> Assignment: ...
def visit_FunctionDef(self, node: ast.FunctionDef): ...
def visit_ClassDef(self, node: ast.ClassDef): ...
# ... and many more AST node handlersUsage Examples
Custom Block Transformer
Define a custom transformer for a specialized code format:
# conf.py
def process_sage_code(source: str) -> tuple[str, str]:
"""Transform SageMath code for analysis."""
# Remove sage-specific syntax
cleaned = source.replace('sage:', 'python:')
# Apply other transformations
cleaned = cleaned.replace('%time', '# %time')
return source, cleaned
codeautolink_custom_blocks = {
'sage': process_sage_code,
'custom-python': lambda s: (s, s.replace('>>>', ''))
}Global Preface Configuration
Set up common imports for all code blocks:
# conf.py
codeautolink_global_preface = """
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
# Common setup
plt.style.use('seaborn')
np.random.seed(42)
"""Advanced HTML Processing
The extension searches for code blocks in HTML using these patterns:
# Default CSS classes searched
search_patterns = [
'highlight-python', # Standard Sphinx
'highlight-py', # Short form
'highlight-default', # Default language
'code-block', # Generic blocks
]
# Additional classes via configuration
codeautolink_search_css_classes = [
'custom-code',
'my-python-blocks'
]Error Handling in Processing
The system provides detailed error reporting for parsing failures:
# Enable detailed warnings in conf.py
codeautolink_warn_on_default_parse_fail = True
# Example error output:
# WARNING: Could not parse code block on line 45 in document.rst:
# SyntaxError: invalid syntax (<string>, line 3)
# Source: 'def invalid_syntax('Supported Code Formats
The extension recognizes and processes these code block formats:
.. code-block:: python
import os # Standard Python
.. code-block:: pycon
>>> import sys # Python console
>>> sys.version
.. code-block:: ipython
In [1]: %matplotlib inline # IPython session
In [2]: import numpy as np
.. code-block:: py3
async def main(): # Python 3 specific
await some_function()Install with Tessl CLI
npx tessl i tessl/pypi-sphinx-codeautolink