wcag-compliance-reporter

Overview

WCAG 2.2 has 77 success criteria (SC) split across three conformance levels: 25 at A, 24 at AA, 28 at AAA (wcag-spec) - for a total of 49 SC at the AA level (which is the typical legal / contractual target in the US, EU, UK).

Per wcag-conformance:

"Level AA: 'satisfies all the Level A and Level AA success criteria'"

"All success criteria at a claimed level must be satisfied with no exceptions."

That binary verdict ("conforms or doesn't") makes per-tool output ("axe found 17 violations in 5 pages") hard to act on - the team needs to know which SC at which level on which page is failing the conformance claim.

This skill builds that report.

When to use

• A multi-page scan has produced output from one or more of: axe-core (via @axe-core/cli or axe-puppeteer), pa11y (via pa11y-ci), Lighthouse (via lighthouse-batch), WAVE, IBM Equal Access.
• A stakeholder asks "are we WCAG 2.2 AA conformant?" and wants a document, not a CLI dump.
• The team is preparing an accessibility audit / VPAT / ACR.
• A PR introduces UI changes and the team needs a "before vs after" on conformance.

If only one tool is in use, the tool's native HTML report may suffice; this skill is for aggregation across multiple tools or multiple pages.

How to use

1. Gather scan output from one or more scanners (axe-core, pa11y, Lighthouse, WAVE, IBM Equal Access) across every page in scope.
2. Normalize per tool into one SC-keyed Violation shape, mapping each tool's rule ID to the WCAG Success Criterion it covers (Step 1).
3. Aggregate the normalized violations by SC and by page (Step 2).
4. Compute the conformance verdict per level (A / AA / AAA): conformant, non-conformant, or unknown - where unknown means the scan never covered the SCs a claim at that level needs (Step 3).
5. Check page coverage against the pages-to-scan.yaml process spec and flag any page that wasn't scanned (Step 4).
6. Render the report - the executive verdict plus per-SC and per-page markdown (Step 5), then emit the machine-readable compliance.json and the CI job that produces it (Machine output and CI, below).

Step 1 - Author normalizers per upstream tool

Each scanner has its own JSON shape. The first step is normalizing to a single intermediate format keyed by SC:

interface Violation {
  page: string;                       // URL or path
  successCriterion: string;           // e.g. "1.4.3"
  level: 'A' | 'AA' | 'AAA';
  ruleId: string;                     // tool-specific rule (e.g. axe "color-contrast")
  selector: string;                   // CSS selector of failing element
  message: string;
  helpUrl?: string;                   // tool's docs
  scanner: 'axe' | 'pa11y' | 'lighthouse' | 'wave' | 'equal-access';
}

Normalizer per tool maps tool-specific rule IDs to the SC they cover, and a central sc-mapping.json holds rule-to-SC for every tool the report consumes. The per-tool tag conventions (axe tags, pa11y codes, Lighthouse's internal map), a worked normalize_axe example, and the sc-mapping.json shape live in references/tool-normalizers.md.

Step 2 - Aggregate

def aggregate(violations):
    by_sc = defaultdict(lambda: defaultdict(list))   # sc -> level -> [violation]
    by_page = defaultdict(list)                       # page -> [violation]
    pages = set()
    for v in violations:
        by_sc[v['successCriterion']][v['level']].append(v)
        by_page[v['page']].append(v)
        pages.add(v['page'])
    return {
        'by_sc': dict(by_sc),
        'by_page': dict(by_page),
        'pages': sorted(pages),
    }

Step 3 - Conformance verdict per level

Per wcag-conformance, conformance is binary per level ("All success criteria at a claimed level must be satisfied with no exceptions"). The verdict logic:

def conformance(agg, level):
    """Returns 'conformant' | 'non-conformant' | 'unknown'.
    Unknown if the scan didn't cover the SCs needed to claim conformance."""
    failing_scs = [sc for sc, by_level in agg['by_sc'].items()
                       if any(v['level'] in level_set(level) for v in
                              [vv for lvl, vs in by_level.items() for vv in vs])]
    if failing_scs:
        return 'non-conformant', failing_scs
    # Did the scan actually cover all SCs needed for this level?
    expected_scs = SC_BY_LEVEL[level]   # ground-truth list per WCAG 2.2
    covered_scs  = set(agg['by_sc'].keys())  # only SCs at least one tool checks
    missing = expected_scs - covered_scs
    if missing:
        return 'unknown', sorted(missing)
    return 'conformant', []

level_set('AA') returns {'A', 'AA'} because AA "satisfies all the Level A and Level AA success criteria" (wcag-conformance).

The "unknown" verdict is critical - automated tools cover roughly 30% of WCAG SCs. A green automated scan ≠ AA conformance. The report must surface what wasn't scanned.

Step 4 - Per-page coverage gaps

Equally important: did every relevant page get scanned at all? A checkout flow with a missing scan on the payment page can't claim conformance per wcag-conformance:

"Complete Processes: When a page is part of a multi-step process, 'all web pages in the process conform at the specified level or better.'"

Author maintains a pages-to-scan.yaml:

processes:
  checkout:
    pages:
      - /
      - /products
      - /cart
      - /checkout
      - /checkout/payment
      - /checkout/confirm
  account:
    pages:
      - /account/login
      - /account/profile
      - /account/orders

Compare scanned pages against the spec; flag missing ones in the report.

Step 5 - Render the report

Render the executive verdict first: a three-row table mapping each conformance level to its verdict and the reason.

Below the verdict, the full report adds process-coverage, failures-by-SC, per-page-detail, and coverage-gap sections - the coverage-gap table is where the "automated tools cover ~30% of SCs, the rest is manual review" reality from Step 3 lands. The complete multi-table sample report, every section with example rows, lives in references/sample-report-format.md.

Machine output and CI

Once the markdown report renders, two operational blocks finish the pipeline: a machine-readable compliance.json for downstream dashboards / ACR gates, and the CI job that runs the scanners and aggregator on every push. Both templates live in references/machine-output-and-ci.md.

Anti-patterns

Limitations

• Automated scans don't replace manual review. Roughly 30% of WCAG SCs are tool-detectable; the rest require human assessment (cognitive load, alternative text quality, color-only meaning, etc.).
• No screen-reader reality check. Tools find the absence of ARIA; they don't verify that the screen-reader narration is helpful. Pair with screen-reader-test-author from qa-accessibility.
• Per-tool drift. Rule catalogs evolve; new tool versions may add or remove SC coverage. Pin tool versions in CI; bump the SC mapping in lock-step.
• VPAT / ACR is a different document. This report informs the VPAT but isn't the VPAT itself - those documents have specific formats (Section 508, EN 301 549) that go beyond WCAG.

References

• wcag-tr - WCAG 2.2 specification: 77 SC across A / AA / AAA, four POUR principles, 13 guidelines.
• wcag-conformance - WCAG 2.2 conformance requirements: binary per-level verdict, complete-processes rule, no-exceptions rule.
• junit-xml-analysis (in the qa-test-reporting plugin) - reporter for test execution (different domain, same PR-time reporting shape).
• The qa-accessibility plugin's per-tool wrappers (axe-a11y, pa11y-a11y, lighthouse-a11y, wave-a11y, ibm-equal-access-a11y) - produce the upstream input this skill consumes.
• Per-tool normalizers + the sc-mapping.json shape (Step 1 deep dive): references/tool-normalizers.md.
• The full multi-table sample report format (Step 5 deep dive): references/sample-report-format.md.
• Machine-readable compliance.json output + the CI aggregation job: references/machine-output-and-ci.md.