bug-report-from-failure

Overview

A test failure produces structured data (XML, JSON, HTML) with everything a triager needs - assertion, stack, test name, environment. This workflow ingests that record and emits a ready-to-file bug spec.

It composes:

• bug-lifecycle-reference for initial state (New).
• severity-vs-priority-reference for severity proposal (heuristic; reviewer confirms).
• defect-taxonomy-istqb for type / root-cause hypothesis fields.
• One of jira-bug-workflow-runner, linear-bug-workflow-runner, github-issues-bug-workflow as the filing backend.

Distinct from screen-recording-driven bug reporting: this one starts from a CI failure record.

When to use

• CI test failure auto-files a triaged bug.
• Manual investigation produces structured failure data - script this rather than write the report by hand.
• Bulk-filing bugs from a regression-suite run after a change flagged many failures.

Step 1 - Ingest the failure record

The skill accepts these inputs (auto-detected by extension):

Parser bodies and per-format field shapes (JUnit, Allure, pytest, Playwright, TestNG) live in references/parsers.md. parse_junit(path) returns one dict per failing testcase; Allure's first-class severity / feature / suite labels are harvested when present.

Step 2 - Extract classification fields

For each failure, propose values for the bug report:

Severity inference rules (heuristic, reviewer confirms):

SEVERITY_FROM_ERROR = {
    "AssertionError":   "medium",
    "TimeoutError":     "high",
    "ConnectionError":  "high",
    "OutOfMemoryError": "critical",
    "SecurityException": "critical",
}

def infer_severity(failure_type, message):
    if failure_type in SEVERITY_FROM_ERROR:
        return SEVERITY_FROM_ERROR[failure_type]
    if "production" in message.lower() or "p0" in message.lower():
        return "high"
    return "medium"

Step 3 - Render the Markdown body

render_body(failure, env) emits a standard Markdown block consumed verbatim by every platform runner - test name, assertion, stack, environment, artefact links, a proposed-classification table (flagged "triager to confirm"), reproduction steps, and dupe history. Full template: references/spec-template.md.

Step 4 - Search for duplicates

Before filing, search the platform tracker for open bugs with matching title / test name. Use the per-platform skill:

# Pseudo
def find_dupes(platform, test_name):
    if platform == "jira":
        return jira_bug_workflow_runner.search_jql(
            f'project = ENG AND text ~ "{test_name}" AND issuetype = Bug'
        )
    if platform == "linear":
        return linear_bug_workflow_runner.find_dupes(TEAM_ID, test_name)
    if platform == "github":
        return github_issues_bug_workflow.search_issues(
            f'is:open label:bug "{test_name}" in:title,body'
        )

If duplicates exist, the workflow attaches a comment instead of creating a new bug.

Step 5 - File the bug

Emit a tracker-agnostic spec:

bug_spec:
  title: "Test failure: checkout fails with promo X"
  body: |
    ## Test failure
    ...
  severity: high
  priority: p2
  labels: [bug, type:regression, component:checkout]
  defect_type: Code
  component: checkout
  reproduction:
    commit: "abc123"
    command: "pytest tests/checkout/test_promo.py::test_stacked"
    environment:
      branch: main
      ci_run: "https://github.com/.../runs/123"

Then pass to the relevant platform-runner. Sample dispatcher:

def file_bug(spec, platform):
    if platform == "jira":
        return jira_bug_workflow_runner.create_bug(
            project_key="ENG",
            summary=spec["title"],
            description_text=spec["body"],
            severity=spec["severity"].capitalize(),
            priority=spec["priority"].upper(),
            labels=spec["labels"],
        )
    if platform == "linear":
        return linear_bug_workflow_runner.create_bug(
            team_id=os.environ["LINEAR_TEAM_ID"],
            title=spec["title"],
            description_md=spec["body"],
            priority=PRIORITY_MAP[spec["priority"]],
            state_id=BACKLOG_STATE_ID,
            label_ids=resolve_label_ids(spec["labels"]),
        )
    if platform == "github":
        return github_issues_bug_workflow.create_bug(
            title=spec["title"],
            body=spec["body"],
            severity=spec["severity"],
            priority=spec["priority"],
            labels=spec["labels"],
        )

Step 6 - Confirm and audit

After filing:

1. Capture the new bug's URL / identifier.
2. Append a comment to the CI run linking the bug.
3. Update a per-test "known failure" register so subsequent runs can correlate.

Worked example - pytest + JUnit → GitHub Issues

from pathlib import Path

failures = parse_junit(Path("results.xml"))
for f in failures:
    spec = {
        "title": f"Test failure: {f['test'].split('::')[-1]}",
        "body": render_body(f, env=collect_env_vars()),
        "severity": infer_severity(f["type"], f["message"]),
        "priority": "p3",  # default; reviewer adjusts
        "labels": ["bug", "auto-filed", "ci-failure"],
        "defect_type": "Code",
        "component": guess_component(f["test"]),
    }
    if find_dupes("github", f["test"]):
        # Comment on the existing issue
        continue
    issue = file_bug(spec, "github")
    print(f"Filed #{issue['number']}: {issue['html_url']}")

Anti-patterns

Limitations

• Inference is heuristic. Severity / type inference from exception class is approximate; a reviewer must confirm.
• Schemas drift. JUnit XML is informal (Ant convention); some test runners emit non-standard variants (Surefire's missing attributes, Playwright's extended fields). The parser must handle gracefully.
• Coverage is per-test-failure. Tests that crash before pytest catches them (segfault) produce no JUnit output - pair with CI step-failure detection.
• No root-cause inference. Root cause (CTAL-TA) requires human investigation; the skill leaves the field blank for triager.
• Platform-specific dedup correctness. Each tracker has its own search semantics; false negatives possible.

References

• JUnit XML de-facto schema - llg.cubic.org/docs/junit/ (community-curated canonical reference).
• Allure framework results format - docs.qameta.io/allure-report.
• TestNG XML - testng.org.
• Parser bodies and render template: references/parsers.md, references/spec-template.md.
• Composed / backend skills are listed in the Overview.