Azure Pipelines Validator

Validates, lints, and security-scans Azure DevOps Pipeline configurations (azure-pipelines.yml, azure-pipelines.yaml). Runs four validation layers via a single orchestrator script.

Philosophy

• Static analysis is cheap and runs before the pipeline does, so catch problems at review time, not at queue time.
• Fix findings in layer order (lint, syntax, best practices, security): a syntax error can hide later findings.
• Treat security findings as merge blockers and best-practice findings as guidance that hardens over time.
• Re-run only the layer you just fixed, then run the full validation once before merging.

When to Use

• Reviewing azure-pipelines.yml or azure-pipelines.yaml before merging a change.
• Auditing a pipeline for hardcoded secrets, :latest container tags or curl | bash patterns.
• Checking task version pinning, displayName, caching and timeouts on an existing pipeline.
• Adding a validation gate to an Azure Pipelines CI job.

When Not to Use

• Executing or testing a pipeline run: this skill does static analysis only.
• Looking up current task inputs or versions: use the live documentation sources listed below.
• Validating GitLab CI or GitHub Actions files: use the matching validator skill instead.
• Generating a new pipeline from scratch: use the Azure Pipelines generator skill instead.

Basic Usage

# Full validation (all layers)
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml

Layers executed in order: 0. YAML lint (yamllint): formatting, indentation, trailing spaces

1. Syntax validation: schema, required fields, stages/jobs/steps hierarchy, task format, dependencies
2. Best practices: display names, task version pinning, pool image specificity, caching, timeouts
3. Security scan: hardcoded secrets/API keys/AWS/Azure credentials, dangerous script patterns, SSL bypasses, container :latest tags

Common Options

# Targeted runs
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --syntax-only
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only

# Skip layers
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --skip-yaml-lint
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-best-practices
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-security

# Strict mode (fail on warnings)
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict

Individual Scripts

python3 scripts/validate_syntax.py azure-pipelines.yml
python3 scripts/check_best_practices.py azure-pipelines.yml
python3 scripts/check_security.py azure-pipelines.yml

Output and Error Recovery

════════════════════════════════════════════════════════════════════════════════
  Azure Pipelines Validator
════════════════════════════════════════════════════════════════════════════════

[1/3] Running syntax validation...
✓ Syntax validation passed

[2/3] Running best practices check...
SUGGESTIONS (2):
  INFO: Line 15: Job 'BuildJob' should have displayName [missing-displayname]
  💡 Add 'displayName: "Your Job Description"' to job 'BuildJob'

  WARNING: Line 25: Task 'Npm@1' could benefit from caching [missing-cache]
  💡 Add Cache@2 task to cache dependencies and speed up builds

[3/3] Running security scan...
MEDIUM SEVERITY (1):
  MEDIUM: Line 8: Container 'linux' uses ':latest' tag [container-latest-tag]
  🔒 Pin container images to specific versions or SHA digests

When validation fails:

1. Note the rule code in brackets (e.g., [missing-displayname]): see references/ for rule details.
2. Fix the flagged line and re-run the same layer (--syntax-only, --security-only, etc.) to iterate quickly.
3. Run full validation once all targeted fixes are applied to confirm no regressions.
4. For MEDIUM/HIGH security findings, do not merge until resolved; INFO findings are advisory.

Common Scenarios

New pipeline validation

bash scripts/validate_azure_pipelines.sh new-pipeline.yml

Security audit before merge

bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only --strict

Pipeline optimisation

bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices

CI/CD self-validation in Azure Pipelines

steps:
- script: |
    bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict
  displayName: 'Validate Pipeline Configuration'

Auto-detection

Run without arguments to auto-detect azure-pipelines*.yml files in the current directory (up to 3 levels deep).

Fetching Live Documentation

The validator performs static analysis only. For dynamic lookups (task versions, input parameters, feature docs), use:

# Context7 MCP
mcp__context7__resolve-library-id("azure-pipelines")
mcp__context7__get-library-docs(context7CompatibleLibraryID, topic="deployment")

# Or WebSearch / WebFetch
WebSearch("Azure Pipelines Docker@2 task documentation 2025")
WebFetch("https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/docker-v2")

Requirements

• Python 3.10+, Bash
• PyYAML and yamllint: auto-installed in a persistent .venv if not available system-wide: no manual setup required.

# Optional manual install
pip3 install PyYAML yamllint

Troubleshooting

Anti-Patterns

NEVER skip validation on template files

WHY: Templates called from the main pipeline can contain schema violations, missing parameters, or invalid task references that only surface at runtime. Validating only the entry-point file leaves template errors undetected until the pipeline actually runs. BAD: Validate only azure-pipelines.yml and skip all files under templates/*.yml. GOOD: Run the validator on every .yml file in the pipeline directory, including all templates.

NEVER treat YAML lint warnings as informational

WHY: Azure Pipelines YAML is whitespace-significant. Indentation errors and trailing spaces can silently restructure the pipeline: turning a step into a job property, or merging two separate blocks: without any obvious parse error. BAD: Ignore yamllint warnings such as "trailing spaces" or "wrong indentation" because the pipeline appears to run. GOOD: Fix all YAML formatting issues before submitting; a clean yamllint pass is a prerequisite for a trustworthy pipeline.

NEVER use the strict mode flag on an unreviewed pipeline

WHY: --strict fails on all warnings, which is the correct setting for a CI gate. Applied to a brand-new pipeline with dozens of warnings, it produces so much noise that engineers discard the output entirely and disable validation rather than fix the root causes. BAD: Run --strict on a new pipeline, see 30 warnings, and remove validation from the workflow because "it's too noisy." GOOD: Run without --strict first, fix critical errors, then warnings, then graduate to strict mode as a CI gate.

NEVER skip security scanning before merging pipeline changes

WHY: A passing syntax validation does not imply a secure pipeline. Security scanning catches hardcoded credentials, injection vectors, and insecure patterns that syntax checks are not designed to detect. BAD: Merge a pipeline change after syntax validation passes with no security check. GOOD: Always run --security as part of the validation workflow, treating MEDIUM and HIGH findings as merge blockers.

NEVER merge a pipeline with hardcoded secrets in YAML

WHY: The security layer flags hardcoded passwords, API keys, tokens, AWS and Azure credentials (for example [hardcoded-password], [hardcoded-api-key]). Anything committed to the repository stays in history even after the line is removed. BAD: password: 'P@ssw0rd' written directly in a variables: block. GOOD: Reference a secret variable or variable group, for example $(DB_PASSWORD), and mark it secret in Azure DevOps.

NEVER leave task versions unpinned or at version zero

WHY: The best-practices layer reports [task-no-version], [task-version-zero] and [task-old-version]. An unpinned or outdated task can change behaviour between runs or run deprecated code. BAD: - task: Npm with no @ version. GOOD: - task: Npm@1 with the major version pinned.

NEVER use :latest container images or ubuntu-latest blindly

WHY: The scanner reports [container-latest-tag] and [pool-latest-image]. A floating tag means the same commit can build differently on different days. BAD: container: ubuntu:latest. GOOD: Pin the image to a specific version or a SHA digest.

NEVER pipe remote downloads straight into a shell

WHY: Patterns such as curl ... | bash and wget ... | sh are flagged by the security scan ([curl-pipe-shell], [wget-pipe-shell]). The pipeline executes whatever the remote host serves at that moment. BAD: - script: curl -sSL https://example.com/install.sh | bash. GOOD: Download the file, verify it, then run it, or use a pinned pipeline task.

NEVER disable SSL verification in pipeline scripts

WHY: The security scan flags SSL bypasses such as [insecure-ssl] and [git-disable-ssl]. Disabling verification exposes the build to man-in-the-middle tampering. BAD: - script: git config --global http.sslVerify false. GOOD: Fix the certificate chain or install the corporate CA on the agent.

NEVER ignore a missing timeout or cache finding on long jobs

WHY: The best-practices layer reports [missing-timeout] and [missing-cache]. A hung job without a timeout holds an agent until the platform default expires, and an uncached dependency install slows every run. BAD: A build job with no timeoutInMinutes that reinstalls npm packages on every run. GOOD: Set timeoutInMinutes on the job and add a Cache@2 step for dependencies.

References

• references/azure-pipelines-reference.md: full YAML syntax reference and rule definitions
• assets/examples/basic-pipeline.yml: simple CI pipeline
• assets/examples/docker-build.yml: Docker build and push
• assets/examples/deployment-pipeline.yml: multi-environment deployment with approval gates
• assets/examples/multi-platform.yml: multi-platform build matrix
• assets/examples/template-example.yml: reusable templates

# Test with a bundled example
bash scripts/validate_azure_pipelines.sh assets/examples/basic-pipeline.yml

Extending the Skill

Add custom rules to the appropriate script:

• Syntax rules → scripts/validate_syntax.py
• Best practice rules → scripts/check_best_practices.py
• Security rules → scripts/check_security.py

# Example custom best-practice rule in check_best_practices.py
def _check_custom_rule(self):
    for job in self._get_all_jobs():
        job_name = job.get('job') or job.get('deployment')
        if 'tags' not in pool:
            self.issues.append(BestPracticeIssue(
                'warning',
                self._get_line(job_name),
                f"Job '{job_name}' should specify agent tags",
                'custom-missing-tags',
                "Add 'tags' to pool to select appropriate agents"
            ))

Note: This skill validates pipeline configurations but does not execute pipelines. Use Azure DevOps Pipeline validation or Azure CLI to test actual pipeline execution.