Evaluate PR Tests

Evaluates the quality, coverage, and appropriateness of tests added in a PR. Produces a structured report with actionable findings.

When to Use

• ✅ PR has tests and you want to evaluate their quality
• ⚠️ PR has no test files -- output a ❌ Fix Coverage verdict noting no tests were added; skip remaining criteria
• ✅ Reviewing whether tests adequately cover the fix
• ✅ Checking if a lighter test type could be used instead
• ✅ Before merging a PR, as part of review

Quick Start

# Auto-detect PR and base branch
pwsh .github/skills/evaluate-pr-tests/scripts/Gather-TestContext.ps1

# With explicit base branch
pwsh .github/skills/evaluate-pr-tests/scripts/Gather-TestContext.ps1 -BaseBranch "origin/main"

Workflow

Step 1: Gather Automated Context

Run the script to get file categorization, convention checks, and anti-pattern detection:

pwsh .github/skills/evaluate-pr-tests/scripts/Gather-TestContext.ps1

This produces a report at CustomAgentLogsTmp/TestEvaluation/context.md with:

• File categorization (fix files vs test files by type)
• Convention compliance checks (naming, attributes, anti-patterns)
• AutomationId consistency (HostApp ↔ test)
• Existing similar tests
• Platform scope analysis

Step 2: Understand the Fix

Read the fix files to understand:

• What changed — which code paths were modified
• Why it changed — the bug being fixed (from PR description or linked issue)
• Edge cases — what boundary conditions exist in the changed code

Step 3: Evaluate the Tests

Read each test file and evaluate against all criteria below. For each criterion, provide a verdict (✅ Pass, ⚠️ Concern, ❌ Fail) with explanation.

Step 4: Produce the Report

Output a structured evaluation report (see Output Format below).

Evaluation Criteria

1. Fix Coverage

Question: Does the test exercise the actual code paths changed by the fix?

How to check:

• Trace the test's actions through the code to the fix location
• Would the test fail if the fix were reverted?
• Does the test assert on the specific behavior that was broken?

Red flags:

• Test only checks that a page loads (doesn't exercise the fix)
• Test asserts on a different property/behavior than what was fixed
• Test interacts with the control but doesn't trigger the buggy code path

Example — Good:

// Fix: CollectionView.SelectedItem setter now clears selection when set to null
// Test: Sets SelectedItem to null and verifies selection is cleared
App.Tap("SelectItem");
App.Tap("ClearSelection");  // Sets SelectedItem = null
var text = App.FindElement("SelectionStatus").GetText();
Assert.That(text, Is.EqualTo("None"));  // Directly tests the fix

Example — Bad:

// Fix: CollectionView.SelectedItem setter
// Test: Just checks CollectionView renders (doesn't test selection clearing)
App.WaitForElement("MyCollectionView");
Assert.That(true);  // Proves nothing about the fix

2. Edge Cases & Gaps

Question: Does the test cover boundary conditions, or only the happy path?

Check for these common gaps:

How to suggest missing edge cases:

• Read the fix code and identify every conditional branch
• For each branch, check if the test covers it
• Look for if (x == null), if (x <= 0), try/catch blocks
• Consider: "What inputs would make this fix NOT work?"

3. Test Type Appropriateness

Question: Is this the lightest test type that can verify the fix?

Preference order (lightest → heaviest):

Decision tree:

Does the test need to interact with visual UI elements?
  YES → Is it checking visual layout/appearance?
    YES → UI test (VerifyScreenshot) ✅
    NO  → Could the interaction be tested via handler/control API?
      YES → Device test ⭐⭐
      NO  → UI test ✅
  NO  → Does it need a platform/native context?
    YES → Device test ⭐⭐
    NO  → Does it test XAML parsing/compilation?
      YES → XAML test ⭐
      NO  → Unit test ⭐

Common "could be lighter" patterns:

4. Convention Compliance

Automated by the script. Review the script output for:

UI Tests:

• File naming: IssueXXXXX.cs
• [Issue()] attribute on HostApp page
• [Category()] attribute — exactly ONE per test class (on the class or method, not both)
• _IssuesUITest base class
• WaitForElement before interactions
• No Task.Delay/Thread.Sleep
• No inline #if ANDROID/#if IOS
• No obsolete APIs (Application.MainPage, Frame, Device.BeginInvokeOnMainThread)
• UITestEntry/UITestEditor for screenshot tests

Unit Tests:

• [Fact] or [Theory] attributes (xUnit)

XAML Tests:

• [Test] with [Values] XamlInflator parameter
• Issue naming: MauiXXXXX

5. Flakiness Risk

Question: Is this test likely to be flaky in CI?

6. Duplicate Coverage

Question: Does a similar test already exist?

Check the "Existing Similar Tests" section of the script output. If similar tests exist:

• Is the new test covering a different scenario? → OK
• Is the new test redundant? → Flag as concern
• Could the new test be merged with an existing one? → Suggest consolidation

7. Platform Scope

Question: Does the test run on all platforms affected by the fix?

Check the "Platform Scope Analysis" from the script:

• Cross-platform fix → tests should run on all platforms
• Platform-specific fix → test on that platform is sufficient
• Fix affects iOS + MacCatalyst → both should be tested (.ios.cs compiles for both)

8. Assertion Quality

Question: Are the assertions specific enough to catch regressions?

9. Fix-Test Alignment

Question: Do the files changed by the fix align with what the test exercises?

• Map the fix files to the controls/features they affect
• Map the test to the controls/features it exercises
• Flag if test exercises a different control than the fix changes
• Flag if test only covers one platform when fix touches multiple

Red flags:

• Test class is named Issue12345 for a fix in CollectionView but only exercises Label rendering
• Fix changes Shell.cs but test only navigates a ContentPage

Output Format

Produce the evaluation report in this format:

## PR Test Evaluation Report

**PR:** #XXXXX — [Title]
**Test files evaluated:** [count]
**Fix files:** [count]

---

### Overall Verdict

[One of: ✅ Tests are adequate | ⚠️ Tests need improvement | ❌ Tests are insufficient]

[1-2 sentence summary of the most important finding]

---

### 1. Fix Coverage — [✅/⚠️/❌]

[Does the test exercise the code paths changed by the fix?]

### 2. Edge Cases & Gaps — [✅/⚠️/❌]

**Covered:**
- [edge case 1]
- [edge case 2]

**Missing:**
- [gap 1 — describe what should be tested and why]
- [gap 2]

### 3. Test Type Appropriateness — [✅/⚠️/❌]

**Current:** [UI Test / Device Test / Unit Test / XAML Test]
**Recommendation:** [Same / Could be lighter — explain why]

### 4. Convention Compliance — [✅/⚠️/❌]

[Summary from automated checks — list only issues found]

### 5. Flakiness Risk — [✅ Low / ⚠️ Medium / ❌ High]

[Specific risk factors identified]

### 6. Duplicate Coverage — [✅ No duplicates / ⚠️ Potential overlap]

[Similar existing tests found, if any]

### 7. Platform Scope — [✅/⚠️/❌]

[Does test coverage match the platforms affected by the fix?]

### 8. Assertion Quality — [✅/⚠️/❌]

[Are assertions specific enough to catch the actual bug?]

### 9. Fix-Test Alignment — [✅/⚠️/❌]

[Do the test and fix target the same code paths?]

---

### Recommendations

1. [Most important actionable recommendation]
2. [Second recommendation]
3. [...]

Output Files

Troubleshooting