CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flake-dashboard-author

Builds a persistent flakiness infrastructure dashboard from JUnit XML or JSON CI run history: defines the flake-rate metric (failures per test over a configurable window), authors the data model, generates a Grafana time-series panel JSON or configures a Datadog CI Visibility view, derives the quarantine-candidate query, and wires trend alerts. Also generates the periodic (weekly / monthly) test-suite trend report - total runs, suite duration, flakiness rate, top failing tests, time-to-green per PR, week-over-week deltas - as a markdown summary for a team Slack channel or wiki page. Use when a team needs a long-lived observability surface for test reliability, or a scheduled comparable health report on top of it.

59

Quality

74%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

53%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is artifact-dense and largely executable, with a coherent six-step build sequence and honest limitations. Its main weaknesses are structural: referenced script/dashboard files are absent from the bundle (making the worked example non-runnable), no ingestion-validation feedback loop despite database/batch operations, and a monolithic 400+ line file that inlines the entire periodic-report capability instead of splitting it into a reference file.

Suggestions

Ship the referenced files — scripts/parse_junit.py, scripts/ingest_playwright_json.js, scripts/quarantine_candidates.sql, and dashboards/flakiness-overview.json — or remove the invocations from the worked example, since the commands currently fail as bundled.

Add an explicit validation checkpoint after ingestion (e.g., 'Verify row count matches the number of <testcase> elements before importing the dashboard; if zero rows, check the run_id join') to create a validate→fix→retry loop for the database/batch operations.

Move the periodic trend report — the metric table, report template, and cadence caveats — into a references/trend-report.md file linked one level deep from a short section, cutting ~90 lines from SKILL.md.

DimensionReasoningScore

Conciseness

Mostly efficient — the bulk is executable artifacts (SQL DDL, panel JSON, queries, commands) — but there is unnecessary material: the terminology note explaining the well-known term 'flaky test' via ISTQB/Google-blog citations, a 'Key fields per the Grafana docs' bullet list that re-explains fields already visible in the JSON above it, and a ~90-line inlined report template. Not 4 because these trimmable sections are more than 'minor instances'; not 2 because the core steps are lean and artifact-driven rather than padded prose.

3 / 5

Actionability

Largely copy-paste ready: complete Grafana panel JSON, executable quarantine SQL with a NULLIF sample-count guard, xmllint extraction command, concrete Datadog monitor threshold steps, and a full worked-example pipeline. Not 5 because the worked example invokes 'scripts/ingest_playwright_json.js', 'scripts/quarantine_candidates.sql', and 'dashboards/flakiness-overview.json' which are not present in the bundle (no scripts/ or dashboards/ directories exist), so those commands are not executable as shipped; not 3 because the majority of guidance runs as-is.

4 / 5

Workflow Clarity

Steps 1–6 are clearly sequenced (metric → data model → panel → Datadog alternative → quarantine query → alerting) with cross-references between thresholds, but validation checkpoints are implicit or absent for the database/batch operations: there is no verify-after-ingest step (e.g., row-count or panel-population check) and no error-recovery loop, and the rubric's scoring note caps workflow clarity at 3 for database/batch operations missing feedback loops. The one explicit verification ('Verify metric existence in Metrics Explorer') sits in Limitations rather than in the workflow. Not 4 specifically because the DB/batch feedback-loop cap applies; not 2 because the sequence itself is coherent and well-defined.

3 / 5

Progressive Disclosure

Scored against the actual bundle (SKILL.md only — no scripts/, references/, or assets/ exist): the body references 'scripts/parse_junit.py', 'scripts/ingest_playwright_json.js', 'scripts/quarantine_candidates.sql', and 'dashboards/flakiness-overview.json' — paths that do not exist, breaking the one-level-deep reference structure. Additionally, ~90 lines of periodic-report template and metric definitions are inlined in a 400+ line single file where a separate reference file is warranted. Not 3 because the missing referenced files are a structural failure beyond 'could be better organized'; not 1 because internal section structure and navigation within the file are clear.

2 / 5

Total

12

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete, multi-part capability list, explicit 'Use when' guidance, and a well-delineated niche with tool-specific triggers. The main improvement opportunity is making the trigger clause more concrete and user-natural ('when the team wants a dashboard for flaky tests or a weekly test-health report') instead of the abstract 'long-lived observability surface' phrasing.

Suggestions

Rewrite the 'Use when' clause with concrete user-side triggers, e.g. 'Use when a team asks for a flaky-test dashboard, test-health report, or CI reliability metrics', replacing the abstract 'long-lived observability surface' phrasing.

Add the natural synonyms practitioners say — 'flaky tests', 'test stability', 'test health' — to the trigger wording so searches match user phrasing.

DimensionReasoningScore

Specificity

The description lists multiple specific concrete actions — 'defines the flake-rate metric (failures per test over a configurable window)', 'authors the data model', 'generates a Grafana time-series panel JSON or configures a Datadog CI Visibility view', 'derives the quarantine-candidate query, and wires trend alerts', plus the periodic trend report with named metrics — giving comprehensive coverage of the skill's capabilities. Not 4 because coverage is comprehensive rather than having minor gaps; every capability in the body is represented.

5 / 5

Completeness

Both parts are explicitly present: the 'what' is detailed and multi-part, and an explicit 'Use when a team needs a long-lived observability surface for test reliability, or a scheduled comparable health report' clause answers 'when'. Not 5 because the 'when' phrasing is abstract ('long-lived observability surface') rather than concrete trigger phrases a user would naturally utter; not 3 because the 'when' clause is explicit, not merely implied.

4 / 5

Trigger Term Quality

Good natural-keyword coverage: 'flakiness', 'dashboard', 'Grafana', 'Datadog', 'CI', 'JUnit XML', 'trend report', 'Slack', 'wiki', 'test reliability' — terms a practitioner would actually say. Not 5 because a few common phrasings are missing (e.g., 'flaky tests' as a noun phrase, 'test health/stability') and nothing like a file-extension cue; not 3 because coverage clearly exceeds 'some relevant keywords'.

4 / 5

Distinctiveness Conflict Risk

Clear niche — building a persistent flakiness dashboard from JUnit XML/JSON CI run history with Grafana/Datadog — with distinct trigger terms ('flake-rate', 'quarantine-candidate', 'Datadog CI Visibility') that are unlikely to fire for unrelated skills. Not 4 because the tooling-specific vocabulary gives minimal realistic overlap risk.

5 / 5

Total

18

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 3 missing

Warning

Total

15

/

16

Passed

Reviewed

Table of Contents