CtrlK
BlogDocsLog inGet started
Tessl Logo

pantheon-ai/azure-pipelines-toolkit

Complete azure-pipelines toolkit with generation and validation capabilities

97

Quality

97%

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Advisory

Suggest reviewing before use

Overview
Quality
Evals
Security
Files

criteria.jsonvalidator/evals/scenario-4/

{
  "context": "Tests that the agent validates both the entry-point and the template file, detects the unversioned task (PublishBuildArtifacts missing @N), the conditional insertion indentation error in the template, and explains why entry-point-only validation would miss these issues.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Unversioned PublishBuildArtifacts task detected",
      "description": "template-validation-report.md (or equivalent) flags `task: PublishBuildArtifacts` in the template as lacking a version suffix (should be PublishBuildArtifacts@2 or similar)",
      "max_score": 20
    },
    {
      "name": "Conditional insertion indentation error detected",
      "description": "template-validation-report.md identifies the `${{ if ... }}:` block — the list items under it must be indented one level deeper than the `- ${{ if ... }}:` line; a misalignment here causes the conditional step to be skipped or the YAML to parse incorrectly",
      "max_score": 20
    },
    {
      "name": "Error file attribution correct",
      "description": "template-validation-report.md correctly attributes each error to templates/build-stage.yml (not to azure-pipelines.yml)",
      "max_score": 15
    },
    {
      "name": "Entry-point-only limitation explained",
      "description": "template-validation-report.md includes an explanation of why validating only azure-pipelines.yml would not surface these template errors",
      "max_score": 15
    },
    {
      "name": "Task version corrected in output template",
      "description": "In the corrected templates/build-stage.yml, PublishBuildArtifacts is replaced with a versioned form (e.g., PublishBuildArtifacts@2)",
      "max_score": 15
    },
    {
      "name": "Corrected template file produced",
      "description": "A corrected version of templates/build-stage.yml is produced (not just a report), with the identified issues fixed",
      "max_score": 15
    }
  ]
}

tile.json