testland/flaky-test-quarantine
Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.
88
84%
Does it follow best practices?
Impact
89%
1.45xAverage score across 10 eval scenarios
Low
Low-risk findings worth noting
Low
Low-risk findings.
1 low severity finding. Worth noting, but not necessarily harmful.
W011: Third-party content exposure detected (indirect prompt injection risk).
What this means
The skill exposes the agent to untrusted, user-generated content from public third-party sources, creating a risk of indirect prompt injection. This includes browsing arbitrary URLs, reading social media posts or forum comments, and analyzing content from unknown websites.
Why it was flagged
In the required runtime workflow (Step 4), the GitHub Actions job checks out the repository and greps/reads quarantine annotation source text from `tests/` (e.g., `test.fixme(...)` lines) to build `expired.txt`, which is then included in `gh issue create --body "$line"`.
Where we found it
tests/ directory source files (test.fixme annotations)
content-type · 3 sites
The CI workflow greps test source files from the tests/ directory, extracts annotation text, and passes it unsanitized into gh issue create --body, creating a path for content from test files to flow into GitHub issues.
references/ci-quarantine-report.md
25
grep -rn -B1 -A5 "test\.fixme(" tests/ \
references/ci-quarantine-report.md
56
run: bash scripts/list-expired-quarantines.sh > expired.txt
references/ci-quarantine-report.md
62
gh issue create --title "Expired quarantine: ${line%%:*}" --body "$line"
- Audited
- Security analysis