Analyze Python code for correctness using symbolic execution and SMT solving to automatically find counterexamples for functions with type annotations and contracts.
86
{
"context": "This evaluation assesses how well the engineer uses crosshair-tool's plugin system to register custom types and contracts. The focus is on using register_type() for symbolic value creation and register_contract() for programmatic contract registration.",
"type": "weighted_checklist",
"checklist": [
{
"name": "register_type usage",
"description": "Uses crosshair's register_type() function to register a symbolic value creator for the Coordinate type",
"max_score": 25
},
{
"name": "Symbolic value creator",
"description": "Implements a creator function that returns symbolic Coordinate instances using proxy_for_type() or similar crosshair APIs for creating symbolic floats",
"max_score": 20
},
{
"name": "Constraint application",
"description": "Applies proper constraints to symbolic latitude ([-90, 90]) and longitude ([-180, 180]) values using Z3 or crosshair's constraint mechanisms",
"max_score": 15
},
{
"name": "register_contract usage",
"description": "Uses crosshair's register_contract() function to programmatically register contracts for the calculate_distance function",
"max_score": 25
},
{
"name": "Precondition specification",
"description": "Specifies a precondition function that validates both arguments are Coordinate instances (checking type or attributes)",
"max_score": 8
},
{
"name": "Postcondition specification",
"description": "Specifies a postcondition function that validates the return value is non-negative (>= 0)",
"max_score": 7
}
]
}Install with Tessl CLI
npx tessl i tessl/pypi-crosshair-toolevals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10