meterian/security-audit
Use for dependency security audits and compliance checks. Use when auditing project dependencies for vulnerabilities, answering "is [library] [version] safe?" questions, or remediating vulnerable libraries. Also activates automatically when the user opens or modifies a manifest file (package.json, package-lock.json, yarn.lock, pnpm-lock.yaml, requirements.txt, pom.xml, Cargo.toml, go.mod, Gemfile, composer.json, build.gradle, *.csproj, pubspec.yaml, conanfile.txt, conanfile.py, project.clj, deps.edn, Package.swift, pubspec.lock, Package.resolved, Gemfile.lock, poetry.lock, uv.lock, Cargo.lock, composer.lock).
96
1.83x
Quality
90%
Does it follow best practices?
Impact
99%
1.83xAverage score across 8 eval scenarios
Securityby
Passed
No known issues
criteria.jsonevals/scenario-1/
{
"context": "Tests whether the agent uses the Meterian CLI correctly for a Node.js audit. CLI invocation is verified via Meterian-specific format in scan-raw.json (vulnerable/summary structure, safeVersions field, language:nodejs) rather than unverifiable command strings.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Raw scan data file",
"description": "A file named `scan-raw.json` exists containing a JSON object with a `vulnerable` array — the Meterian CLI check output format",
"max_score": 10
},
{
"name": "Meterian format: safeVersions",
"description": "Entries in the `vulnerable` array of `scan-raw.json` contain a `safeVersions` field — a distinctive Meterian-only field not produced by npm audit or web search",
"max_score": 10
},
{
"name": "nodejs language",
"description": "Entries in `scan-raw.json` carry `\"language\":\"nodejs\"` — confirming the correct language parameter was used",
"max_score": 10
},
{
"name": "Lock file preferred",
"description": "Extracts dependency versions from `package-lock.json` (pinned versions) rather than the version ranges in `package.json`",
"max_score": 12
},
{
"name": "Markdown table",
"description": "Report contains a markdown table presenting audit results",
"max_score": 10
},
{
"name": "Correct table headers",
"description": "Table includes all five columns: Package, Version, Severity, ID, Safe Versions (in any order or capitalisation)",
"max_score": 12
},
{
"name": "Summary line",
"description": "Report includes a summary line of the form \"X vulnerabilities found across Y packages (Z clean)\" or equivalent phrasing referencing total and clean counts",
"max_score": 12
},
{
"name": "Remediation offer",
"description": "Report or accompanying text offers to perform remediation (upgrade vulnerable packages) if any vulnerabilities were found",
"max_score": 8
},
{
"name": "Reachability mention",
"description": "Report or accompanying text mentions the option of reachability analysis if vulnerabilities were found",
"max_score": 6
}
]
}