CtrlK
BlogDocsLog inGet started
Tessl Logo

pantheon-ai/azure-pipelines-toolkit

Complete azure-pipelines toolkit with generation and validation capabilities

97

Quality

97%

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Advisory

Suggest reviewing before use

Overview
Quality
Evals
Security
Files

criteria.jsonvalidator/evals/scenario-1/

{
  "context": "Tests that the agent detects task format violations (Npm without version), an empty dependsOn value for the CD stage, and produces an accurate syntax report with a corrected pipeline file.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Unversioned task format detected",
      "description": "syntax-report.md (or equivalent) flags the `task: Npm` entry (missing @version suffix) as a task format violation",
      "max_score": 20
    },
    {
      "name": "Empty dependsOn detected",
      "description": "syntax-report.md flags the `dependsOn:` key with no value in the CD stage as a syntax error or schema violation",
      "max_score": 20
    },
    {
      "name": "Task format corrected in output",
      "description": "In the corrected azure-pipelines.yml, the unversioned `task: Npm` entry is replaced with a versioned form (e.g., Npm@1)",
      "max_score": 20
    },
    {
      "name": "dependsOn corrected in output",
      "description": "In the corrected azure-pipelines.yml, the CD stage dependsOn is either removed, set to CI, or given a valid value — no bare `dependsOn:` without a value",
      "max_score": 15
    },
    {
      "name": "Location/context provided for each error",
      "description": "syntax-report.md identifies where each error occurs (e.g., stage name, job name, task name, or approximate line)",
      "max_score": 15
    },
    {
      "name": "Correction explained",
      "description": "syntax-report.md explains the correction applied for each error, not just that an error exists",
      "max_score": 10
    }
  ]
}

tile.json