Scan dependencies for known vulnerabilities and licenses.
Overall
score
61%
{
"context": "This criteria evaluates how well the engineer uses the Safety CLI package to implement system-wide Python environment vulnerability scanning. The focus is on proper usage of Safety's scanning commands and output handling capabilities.",
"type": "weighted_checklist",
"checklist": [
{
"name": "System-wide scanning",
"description": "Uses 'safety system-scan' or 'safety scan' command to perform system-wide vulnerability scanning when no target paths are provided",
"max_score": 30
},
{
"name": "Targeted directory scanning",
"description": "Uses 'safety scan --target' or 'safety system-scan --target' with specified directory paths to scan specific locations",
"max_score": 25
},
{
"name": "Output format handling",
"description": "Uses '--output json' flag or equivalent to get structured output from Safety CLI that can be parsed programmatically",
"max_score": 20
},
{
"name": "Subprocess execution",
"description": "Uses subprocess module (subprocess.run, subprocess.check_output, or similar) to execute Safety CLI commands and capture output",
"max_score": 15
},
{
"name": "Report data extraction",
"description": "Parses Safety's JSON output to extract vulnerability information including package counts and vulnerability details",
"max_score": 10
}
]
}Install with Tessl CLI
npx tessl i tessl/pypi-safetydocs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10