Analyze Python code for correctness using symbolic execution and SMT solving to automatically find counterexamples for functions with type annotations and contracts.
86
{
"context": "This evaluation assesses the engineer's ability to use CrossHair's symbolic execution capabilities to verify generic type constraints, particularly focusing on TypeVar binding, bounded TypeVars, and contract verification using PEP316-style contracts with preconditions and postconditions.",
"type": "weighted_checklist",
"checklist": [
{
"name": "TypeVar declaration",
"description": "Correctly declares TypeVar 'T' and bounded TypeVar 'NumericT' with appropriate constraints (int, float) at module level",
"max_score": 10
},
{
"name": "Generic tuple typing",
"description": "Uses Tuple[T, T] type annotation in swap_pair to enforce same-type constraint for both tuple elements",
"max_score": 15
},
{
"name": "PEP316 preconditions",
"description": "Implements preconditions using 'pre:' in docstrings for all four functions (swap_pair, process_numeric, get_first, transform_list) to validate inputs",
"max_score": 15
},
{
"name": "PEP316 postconditions",
"description": "Implements postconditions using 'post:' in docstrings with __return__ references to verify output correctness and type preservation",
"max_score": 20
},
{
"name": "Bounded TypeVar usage",
"description": "Uses NumericT bounded TypeVar correctly in process_numeric function signature for both parameters and return type to restrict to numeric types",
"max_score": 15
},
{
"name": "Generic List typing",
"description": "Uses List[T] type annotations in get_first and transform_list to maintain generic type relationships between input lists and return values",
"max_score": 15
},
{
"name": "Contract verification",
"description": "Contracts are verifiable by CrossHair's symbolic execution (can be checked with 'crosshair check' command without errors in contract syntax)",
"max_score": 10
}
]
}Install with Tessl CLI
npx tessl i tessl/pypi-crosshair-toolevals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10