CtrlK
BlogDocsLog inGet started
Tessl Logo

pantheon-ai/gitlab-ci-toolkit

Complete GitLab CI/CD toolkit with generation and validation capabilities for pipelines and configurations

97

Quality

97%

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Advisory

Suggest reviewing before use

Overview
Quality
Evals
Security
Files

criteria.jsonvalidator/evals/scenario-3/

{
  "context": "Tests that the agent validates the included template and detects the undeclared stage 'testing' in the run_tests job, explains why entry-point-only validation misses it, and produces corrected versions of both files.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Undeclared stage in template detected",
      "description": "include-validation-report.md (or equivalent) flags that run_tests in the template references stage `testing`, which is not declared in the main file's `stages:` list",
      "max_score": 25
    },
    {
      "name": "Error correctly attributed to template file",
      "description": "include-validation-report.md identifies the error as being in `templates/jobs.gitlab-ci.yml`, not in the main `.gitlab-ci.yml`",
      "max_score": 15
    },
    {
      "name": "Entry-point limitation explained",
      "description": "include-validation-report.md includes an explanation of why validating only the main file does not surface errors in included template files",
      "max_score": 15
    },
    {
      "name": "Stage corrected in output",
      "description": "In the corrected templates/jobs.gitlab-ci.yml, the `run_tests` job references a declared stage (e.g., `test`) OR the main .gitlab-ci.yml adds `testing` to the stages list",
      "max_score": 20
    },
    {
      "name": "Corrected template file produced",
      "description": "A corrected version of `templates/jobs.gitlab-ci.yml` is produced as a file (not just described in the report)",
      "max_score": 15
    },
    {
      "name": "No new errors introduced in output",
      "description": "The corrected files do not introduce new syntax or schema violations (e.g., the needs: references in deploy_app remain valid after stage corrections)",
      "max_score": 10
    }
  ]
}

tile.json