Analyze Python code for correctness using symbolic execution and SMT solving to automatically find counterexamples for functions with type annotations and contracts.
86
Pending
Does it follow best practices?
Impact
86%
1.24xAverage score across 10 eval scenarios
Pending
The risk profile of this skill
{
"context": "This evaluation assesses how well the engineer uses crosshair-tool's plugin system to register custom types and contracts. The focus is on using register_type() for symbolic value creation and register_contract() for programmatic contract registration.",
"type": "weighted_checklist",
"checklist": [
{
"name": "register_type usage",
"description": "Uses crosshair's register_type() function to register a symbolic value creator for the Coordinate type",
"max_score": 25
},
{
"name": "Symbolic value creator",
"description": "Implements a creator function that returns symbolic Coordinate instances using proxy_for_type() or similar crosshair APIs for creating symbolic floats",
"max_score": 20
},
{
"name": "Constraint application",
"description": "Applies proper constraints to symbolic latitude ([-90, 90]) and longitude ([-180, 180]) values using Z3 or crosshair's constraint mechanisms",
"max_score": 15
},
{
"name": "register_contract usage",
"description": "Uses crosshair's register_contract() function to programmatically register contracts for the calculate_distance function",
"max_score": 25
},
{
"name": "Precondition specification",
"description": "Specifies a precondition function that validates both arguments are Coordinate instances (checking type or attributes)",
"max_score": 8
},
{
"name": "Postcondition specification",
"description": "Specifies a postcondition function that validates the return value is non-negative (>= 0)",
"max_score": 7
}
]
}evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10