CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/flake-dashboard-author

Use for any question or task about tracking flaky-test rates or test reliability over time, including a weekly or monthly test-health report for management: a flake-rate metric from CI history (JUnit XML or JSON), a Grafana panel or Datadog CI Visibility view, the quarantine-candidate query, trend alerts and the report for Slack or a wiki.

Quality

72%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

50%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The skill body is well-sequenced and rich in concrete, copyable artifacts for building a flake-rate dashboard, but it is undermined by references to script and dashboard files that are missing from the bundle, absent validation checkpoints on the batch ingestion path, and padding such as the terminology note and fully rendered sample report. These issues consistently place it at the midpoint across all four content dimensions.

Suggestions

Ship the referenced bundle files (scripts/parse_junit.py, scripts/ingest_playwright_json.js, scripts/quarantine_candidates.sql, dashboards/flakiness-overview.json) or remove/replace the commands that invoke them so the ingestion steps and worked example are executable as written.

Add validation checkpoints to the ingestion workflow — e.g. verify expected row counts in test_runs after each load and re-validate on mismatch — so the batch/database pipeline includes a fix-and-retry loop rather than ending at 'populate and query'.

Trim the opening terminology blockquote (the 'flaky test' definition and ISTQB Glossary v4.7.1 citation) and the Grafana field-by-field explanation bullets, and move the full report template and panel JSON into a references/ file summarized by a short inline example.

DimensionReasoningScore

Conciseness

The body is dense with genuinely useful artifacts (SQL DDL, panel JSON, queries), but includes unnecessary explanation: the opening terminology blockquote defining "flaky test" and citing ISTQB Glossary v4.7.1 (a concept Claude already knows, plus a time-sensitive version number), bullet-by-bullet restatement of Grafana field behavior ("set to `\"percent\"` so Grafana formats values as `2.4%`"), and a fully rendered ~30-line sample report. This exceeds the 4 anchor's 'minor instances of over-explanation' but is far from the padding of the 1–2 anchors.

3 / 5

Actionability

Much of the guidance is concrete and executable (test_runs DDL, xmllint extraction, complete Grafana panel JSON, Datadog queries and monitor thresholds), but core commands invoke files absent from the bundle: `scripts/parse_junit.py` (Step 2's specified JUnit ingestion path), `scripts/ingest_playwright_json.js`, `scripts/quarantine_candidates.sql`, and `dashboards/flakiness-overview.json` in the worked example. The gaps are not minor — the prescribed ingestion route and the entire worked example fail as written — so this sits at the 3 anchor ('missing key details') rather than the 4.

3 / 5

Workflow Clarity

Steps 1–6 are clearly sequenced (metric → data model → panel → Datadog alternative → quarantine query → alerting) with one embedded guard ("HAVING COUNT(*) >= 10 guard prevents a test with 1 run and 1 failure from appearing as 100% flaky") and one verification note ("Verify metric existence in Metrics Explorer before building the Timeboard widget"). However the main workflow is batch database ingestion with no validation checkpoints — no row-count or ingest verification, no fix-and-retry loop — which caps it at 3 per the rubric's batch/database feedback-loop rule. It is above the 2 anchor because the sequence itself is well defined with some implicit checks.

3 / 5

Progressive Disclosure

Section structure is good (numbered steps, worked example, limitations, a clear References list of external docs), but the ~390-line body inlines content that belongs in separate files (the 40-line Grafana panel JSON, the full report template) and references `scripts/` and `dashboards/` paths that do not exist in the bundle, so the internal references mislead rather than navigate. This matches the 3 anchor (some structure, content that should be separate is inline) rather than the 2 anchor's minimal structure.

3 / 5

Total

12

/

20

Passed

Description

82%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description with explicit trigger guidance, broad natural-term coverage including tool synonyms, and a concrete deliverable enumeration. Its main weaknesses are noun-listed rather than action-led capabilities and a niche that partially overlaps the sibling quarantine skill.

DimensionReasoningScore

Specificity

Lists several specific deliverables — "a flake-rate metric from CI history (JUnit XML or JSON)", "a Grafana panel or Datadog CI Visibility view", "the quarantine-candidate query", "trend alerts and the report for Slack or a wiki" — with only minor coverage gaps. It falls short of the 5 anchor because capabilities are listed as noun deliverables rather than the concrete verb-led actions ("Extract…, fill…, merge…") of the top anchor.

4 / 5

Completeness

Both parts are present: an explicit trigger clause ("Use for any question or task about tracking flaky-test rates or test reliability over time") and a concrete enumeration of what is produced (metric, dashboard, query, alerts, report). It is not a 5 because the "what" is delivered only as a list of noun deliverables — no verb-first statement of what the skill does (e.g. "Builds…", "Computes…") — making it less explicit than the 5 anchor's action-plus-trigger pairing.

4 / 5

Trigger Term Quality

Comprehensive natural-term coverage with synonyms and format alternatives users would actually say: "flaky-test rates", "test reliability over time", "test-health report", "JUnit XML or JSON", "Grafana… or Datadog CI Visibility", "Slack or a wiki". Each deliverable is paired with an alternative phrasing, matching the 5 anchor's requirement for synonyms and file formats rather than the 4 anchor's 'a few natural terms missing'.

5 / 5

Distinctiveness Conflict Risk

The flaky-test observability niche is clear and mostly distinct, but "the quarantine-candidate query" as a listed trigger overlaps with the sibling `flaky-test-quarantine` skill referenced in the body, and "test-health report for management" could collide with generic test-reporting skills. This matches the 4 anchor (minor overlap risk with closely related skills) rather than the 5 anchor's minimal conflict risk.

4 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 3 missing

Warning

Total

15

/

16

Passed

Reviewed

On this page