desktop-test-strategy-reference

Overview

Desktop GUI testing is fragmented along two axes: operating system (Windows / macOS / Linux) and UI toolkit (native, Electron, Qt, GTK, Cocoa, WPF, WinUI, Win32). Unlike the web - where a single DOM-driving tool (Playwright / Selenium) covers every browser - each desktop OS exposes its own accessibility tree, and that tree is the substrate every reliable desktop test driver uses.

Per Microsoft Learn - UI Automation Win32 overview:

"Microsoft UI Automation is an accessibility framework that enables Windows applications to provide and consume programmatic information about user interfaces (UIs). … UI Automation also allows automated test scripts to interact with the UI."

Per Apple - Testing with Xcode, UI Testing chapter:

"UI Testing rests upon two core technologies: the XCTest framework and Accessibility."

Per the at-spi2-core project:

"Base DBus XML interfaces for accessibility, the accessibility registry daemon, and atspi library."

This skill is a pure reference consumed by the per-tool skills below and by anyone choosing a desktop test stack.

How to use this reference

1. Pick OS + toolkit from the per-OS / per-toolkit decision matrix.
2. Choose stable locators for that backend from the locator-strategy table.
3. Apply the matching async-wait primitive for the OS (XCTest waiters / FlaUI Retry / AT-SPI polling) - see references/async-waits-and-concurrency.md.
4. Pre-handle the OS elevation hazard (UAC / TCC / AT-SPI) before it blocks the run - see references/platform-hazards-and-dpi.md.

When to use

• Starting a new desktop test program - which OS + toolkit needs which driver?
• Auditing an existing desktop test suite - is the right backend in use, or is the suite working around a wrong-tool fit?
• Cross-platform desktop app (Electron, Qt, .NET MAUI) - which test tools cover which platforms?
• Onboarding a tester from web E2E - what's the mental-model difference between DOM-driving and accessibility-tree-driving?

The three OS backends

Every reliable desktop driver routes through one of three OS-native accessibility trees. Direct pixel / coordinate scripting (SikuliX, AutoIt screen scraping) is in the anti-pattern column below.

Windows - Microsoft UI Automation (UIA)

UI Automation (UIA) is Microsoft's accessibility framework for Windows desktop applications, replacing the older Microsoft Active Accessibility (MSAA) interface (msuia). UIA exposes an AutomationElement tree with control patterns (InvokePattern, ValuePattern, SelectionPattern, etc.) that describe what each element does, not just how it looks.

Per msuia:

"UI Automation is designed for experienced C/C++ developers. In general, developers need a moderate level of understanding about Component Object Model (COM) objects and interfaces, Unicode, and Windows API programming."

Higher-level language bindings (C#, PowerShell, Python via pywinauto) sit on top. For test-automation purposes, the WinAppDriver and Appium-Windows projects expose UIA as a W3C WebDriver endpoint - which is what most QA toolchains actually drive.

macOS - Apple Accessibility + XCTest

macOS GUI testing uses the same XCTest framework that ships with Xcode for unit and integration tests. Per appleuit:

"UI testing rests upon two core technologies: the XCTest framework and Accessibility."

The three foundational classes (appleuit) are:

The general UI-test pattern per appleuit:

"Use an XCUIElementQuery to find an XCUIElement. Synthesize an event and send it to the XCUIElement. Use an assertion to compare the state of the XCUIElement against an expected reference state."

Xcode's Accessibility Inspector (bundled with Xcode → Xcode → Open Developer Tool → Accessibility Inspector) is the read-side companion: it walks the same tree the tests see, lets the engineer verify accessibility identifiers exist before writing the query, and flags WCAG-style accessibility gaps in the same pass.

Linux - AT-SPI

The Linux desktop accessibility stack is AT-SPI (Assistive Technology Service Provider Interface). Per atspi2core, the canonical implementation at-spi2-core provides "Base DBus XML interfaces for accessibility, the accessibility registry daemon, and atspi library." The registry daemon (at-spi2-registryd) exposes a system-wide D-Bus service that assistive tools - and test clients - connect to.

GTK applications expose AT-SPI automatically via the atk bridge (at-spi2-atk); Qt applications expose AT-SPI via the QAccessible infrastructure (see qt-test-framework SKILL for the Qt-specific path); Electron and Chromium apps on Linux expose their own AT-SPI surface.

Python clients (dogtail, pyatspi) and the GNOME inspector tool Accerciser walk the registry to drive tests.

The toolkit overlays

Two cross-toolkit families need explicit treatment because they ride on top of (or around) the OS backends:

Electron

Electron apps are Chromium + Node.js wrapped in a native window. They expose two parallel surfaces:

1. Renderer process - a normal Chromium DOM. Page-level Playwright / Selenium / Puppeteer drive this just like a browser tab.
2. Main process - the Node.js process that owns native windows, menus, IPC, file dialogs, lifecycle.

A packaged Electron app cannot be driven by browser-only Playwright - the entry point is the packaged binary, not a URL. The Playwright _electron API (electron-playwright SKILL in this plugin) launches the packaged binary, exposes the main process as a typed ElectronApplication handle, and returns Chromium-window pages for the renderer. Legacy alternative (deprecated 2022): Spectron - migration path in electron-playwright's references/spectron-migration.md.

Qt

Qt has its own first-party test framework - QtTest - that lives in the application's process and emits events directly into the QObject event queue. It does not go through the OS accessibility tree by default. For UI tests of Qt apps that need to be driven from outside the process, the OS-native drivers (WinAppDriver on Windows, XCTest on macOS, AT-SPI on Linux via Qt's QAccessible) are the path. See qt-test-framework SKILL.

Per-OS / per-toolkit decision matrix

Cross-references to per-tool SKILLs:

• winappdriver - the Windows UIA WebDriver service, covering both the direct Selenium-client path and the Appium 2.x invocation path
• xcuitest-suite (qa-mobile plugin) - XCTest UI; its references/macos.md covers the macOS desktop delta
• AT-SPI on Linux - driven via dogtail / pyatspi clients (no dedicated skill; the backend and bootstrap are covered in this reference)
• qt-test-framework - QtTest in-process
• electron-playwright - Playwright _electron API; its references/spectron-migration.md covers legacy Spectron migrations

Choosing a driver

To pick one driver for a concrete app, first infer the app type from the project file - never from a bare directory name or README:

Then apply the decision table - one primary driver per app; a secondary fallback only where two are co-equal defensible (UWP, Win32, Qt):

Two constraints that flip or qualify a recommendation:

• Elevation (Windows). If the SUT requires admin privileges, the driver session itself must run elevated - UAC's secure desktop is outside the accessibility tree, so a non-elevated WinAppDriver / FlaUI session sees the entire elevated UI as empty (WinAppDriver #306, #2033). Signals: <requestedExecutionLevel level="requireAdministrator" /> in app.manifest; README "Run as administrator."
• Cross-OS Electron. electron-playwright is the same driver across Windows / macOS / Linux (_electron.launch() + electronApp.evaluate() per the Playwright ElectronApplication API); only the CI bootstrap differs per OS (see references/platform-hazards-and-dpi.md).

Never mix UIA2 and UIA3 in the same FlaUI test process (unsupported per the FlaUI README), and never emit one recommendation for "cross-platform desktop" without a per-OS breakout - each OS has a different accessibility backend.

Locator strategy across backends

The portable lesson from web E2E (per playwright-testing in the qa-web-e2e plugin) - accessibility-first locators - carries directly over: every desktop backend resolves elements through the same accessibility tree assistive technology uses.

The single biggest portability win across desktop platforms is: every app under test sets a stable accessibility identifier on every interactive widget. Without it, the tree's only resolution key is the human-visible label, which collapses under localisation.

Localisation rule of thumb: if the test passes on a US-English build but the same app ships in 20 locales, every Name / label-based locator is a latent failure. AccessibilityId / AutomationId / object name are locale-independent by design - make the developer set them at the source.

Coverage scope per layer

Cross-references for the upstream + downstream slots:

• Unit testing - see the per-language qa-unit-tests-* plugins.
• Visual regression - see qa-visual-regression for desktop screenshot comparison patterns.
• CI integration - desktop runners cost more than web runners; see winappdriver and qa-mobile's xcuitest-suite (references/macos.md) for GitHub-hosted vs self-hosted considerations.

Operating the tests - deep references

Once the stack is chosen, the operational depth lives in two companion references:

• Asynchronous waits + concurrency - per-OS retry primitives (XCTest waiters, FlaUI Retry, AT-SPI polling) and the per-OS parallel-test policy: references/async-waits-and-concurrency.md.
• Platform hazards + high-DPI - foreground-lock, UAC secure desktop, macOS TCC prompts, AT-SPI session enablement, and the per-monitor DPI test matrix: references/platform-hazards-and-dpi.md.

Review hazards - desktop test-code checklist

When reviewing an existing desktop UI test suite or a PR that touches desktop test files, walk these four axes. For generic, framework-agnostic test-file conventions (AAA, naming, magic numbers, assertion quality), use the test-code-critic agent in qa-test-review - this checklist covers only the desktop-specific hazards.

Hard-reject signal: a new test file where every element lookup is by Name or index (zero AutomationId / accessibilityIdentifier usage) - locale-dependent and structurally fragile; assign stable identifiers before authoring tests.

Anti-patterns

Limitations

• Apple developer documentation is behind a Cloudflare-style challenge on most "https://developer.apple.com/documentation/…" pages - automated fetches return the SPA shell only. This reference cites Apple's stable-archive testing_with_xcode chapter (appleuit) for prose; per-API surface (XCUIApplication, XCUIElement) is treated as a stable identifier in qa-mobile's xcuitest-suite SKILL.
• No single cross-platform driver. Tools that claim to be one (Appium across Windows + macOS + Linux desktop) are thin facades over the per-OS backends - coverage gaps follow the underlying backend's gaps.
• GPU-accelerated content (canvas, WebGL, custom-painted Qt widgets, Win32 DirectComposition surfaces) does not expose itself to the accessibility tree by default; any driver routed through the tree sees them as opaque rectangles.
• Cross-process focus + window-Z order is OS-policy-dependent (Windows foreground-lock, macOS Stage Manager); tests that assume a window is foreground often need explicit app.activate() calls.
• Localisation collapses label-based locators. The per-tool SKILLs all push toward AutomationId / accessibilityIdentifier / object name for this reason.

References

• Microsoft UI Automation overview - msuia.
• Apple Testing with Xcode - UI Testing chapter (appleuit).
• at-spi2-core - atspi2core.
• dogtail (Python AT-SPI client) - dogtail.
• Operational depth (with their own citations): async waits + concurrency in references/async-waits-and-concurrency.md; foreground / elevation hazards + high-DPI matrix in references/platform-hazards-and-dpi.md.
• Per-tool implementation SKILLs: winappdriver (direct + Appium invocation), qt-test-framework, electron-playwright (incl. Spectron migration), and qa-mobile's xcuitest-suite (references/macos.md for the macOS desktop delta).
• Web-side neighbour: playwright-testing - DOM-driving for browser apps (the contrast surface for desktop drivers).
• Architecture-pattern sibling: object-model-patterns (in the qa-test-review plugin) - Pattern 7 (Screen Object) is the desktop analog of POM and cites this skill for the backend substrate.