CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/mobile-device-matrix-toolkit

Dispatches mobile UI test runs across a 3-tier device matrix (smoke per-PR, regression per-merge, full farm at release) to control CI cost: generates per-target Appium capability configs from a central YAML, parallelises via GitHub Actions matrix strategy, and aggregates JUnit XML into a cross-device pass/fail table. Use when deciding which iOS / Android devices and OS versions to run tests on and at which stage (smoke / regression / full farm), not how to configure a specific test framework (for that, use xcuitest-suite, espresso-suite, etc.).

68

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-executed dispatcher skill: executable scripts, explicit tier-cost rationale, an anti-patterns table with concrete fixes, and a rare explicit validation warning against false-positive aggregation. It falls short of top marks on minor gaps — an undefined run-tests.sh hook, inline time-sensitive device data, and everything packed into one file rather than split into references.

Suggestions

Define or stub ./scripts/run-tests.sh (even a two-line signature showing how target name and capabilities JSON are consumed) — it is the hook every CI shard invokes but is never shown.

Trim the full 14-device release-tier YAML to representative entries plus a note, and flag device/OS versions as example data that the owning team refreshes — the inlined version numbers (iPhone 15, os 17.4, api 34) will age quickly.

Move the per-farm wiring (Step 5's Firebase/BrowserStack snippets) into a references/ file (e.g. references/farm-wiring.md) linked from the main body, keeping SKILL.md as the decision-level overview.

DimensionReasoningScore

Conciseness

Mostly lean: code blocks carry the weight, the anti-patterns table is dense, and the farm-vs-local decision table is compact and useful. Not 5 because of minor trimmable content — the full 14-device release-tier YAML listing, the test-pyramid citation ('mobile UI tests are the most expensive layer') explaining a concept Claude already knows, and time-sensitive version/device data (iPhone 15, os 17.4, api 34) inlined rather than isolated as aging example data.

4 / 5

Actionability

gen-matrix.py and aggregate-matrix.py are complete, executable scripts; the GitHub Actions matrix/dispatch YAML and the gcloud/BrowserStack commands are concrete. Not 5 because of minor gaps: ./scripts/run-tests.sh (the hook the whole CI matrix calls) is never defined, and the reusable run-matrix.yml is only shown as a jobs fragment; not 3 because the guidance is genuinely executable, not pseudocode.

4 / 5

Workflow Clarity

Clear 5-step sequence (define matrix → generate capabilities → tier dispatch → aggregate → farm-vs-local decision), and the batch aggregation risk gets an explicit checkpoint ('a shard that fails to upload its JUnit XML... reads as 0 tests, not failure — detect shards with a missing or empty report, re-run only those shards, then re-aggregate'), which satisfies the batch-operation feedback-loop requirement. Not 5 because minor validation gaps remain — no check that the generated matrix output is well-formed or that a tier's device list is non-empty before dispatch.

4 / 5

Progressive Disclosure

No bundle files exist; the single file is well-organized with clear section headers, a When-to-use section, per-step structure, anti-patterns, and a References section naming sibling skills. Not 5 because all content lives inline in one ~280-line file — the full tier YAML listing and per-farm wiring are natural candidates for separate reference files; not 3 because structure and organization are genuinely good, not merely 'some structure'.

4 / 5

Total

16

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete multi-action capability statement, explicit 'Use when' trigger guidance, and an explicit disambiguation boundary against sibling test-framework skills. The only weakness is a few missing natural trigger variations (emulator, real devices, named farms).

DimensionReasoningScore

Specificity

Lists multiple concrete, comprehensive actions covering the full pipeline: 'generates per-target Appium capability configs from a central YAML, parallelises via GitHub Actions matrix strategy, and aggregates JUnit XML into a cross-device pass/fail table', each naming the concrete mechanism and output. Not 4 because coverage is comprehensive rather than having minor gaps — generation, parallelisation, and aggregation span the whole workflow.

5 / 5

Completeness

Explicitly answers both: what ('dispatches mobile UI test runs across a 3-tier device matrix... to control CI cost: generates... parallelises... aggregates') and when ('Use when deciding which iOS / Android devices and OS versions to run tests on and at which stage'). The 'Use when' clause includes concrete trigger phrases, matching the top anchor exactly; third-person voice is used throughout.

5 / 5

Trigger Term Quality

Good natural coverage: 'which iOS / Android devices and OS versions to run tests on', 'smoke / regression / full farm', 'device matrix', 'CI cost' are phrases a user would naturally say. Not 5 because common variations like 'emulator', 'real devices', or specific farm names (Firebase Test Lab, BrowserStack) that users often use are absent; not 3 because the natural keyword set goes well beyond 'some relevant keywords'.

4 / 5

Distinctiveness Conflict Risk

Clear niche (device-matrix dispatch for CI cost control) with an explicit negative boundary: 'not how to configure a specific test framework (for that, use xcuitest-suite, espresso-suite, etc.)'. This actively routes away from sibling skills, minimising conflict risk.

5 / 5

Total

19

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 3 missing

Warning

Total

15

/

16

Passed

Reviewed

Table of Contents