UIAutomation

Python UIAutomation for Windows provides comprehensive automation capabilities for Windows applications using Microsoft's UIAutomation framework. It enables Python developers to interact with and control Windows applications built with various UI frameworks including MFC, Windows Forms, WPF, Modern UI, Qt (partial support), Firefox, and Chrome.

Package Information

• Package Name: uiautomation
• Language: Python
• Installation: pip install uiautomation
• Supported Platforms: Windows XP SP3+, Windows Vista SP2+, Windows 7+, Windows 8/8.1/10
• Python Versions: Python 2.7, Python 3.4+
• License: Apache 2.0

Core Imports

import uiautomation

All public API is available through the main module import:

# Control classes are available directly
window = uiautomation.WindowControl()
button = uiautomation.ButtonControl()

# Global functions are available
root = uiautomation.GetRootControl()
focused = uiautomation.GetFocusedControl()

Basic Usage

import uiautomation

# Find and interact with a window
calculator = uiautomation.WindowControl(searchDepth=1, Name='Calculator')

# Find a button within the window and click it
button7 = calculator.ButtonControl(Name='Seven')
button7.Click()

# Type into an edit control
edit = calculator.EditControl(ClassName='Edit')
edit.SendKeys('123')

# Get control properties
print(f"Button name: {button7.Name}")
print(f"Button bounds: {button7.BoundingRectangle}")
print(f"Is enabled: {button7.IsEnabled}")

# Wait for controls to appear or disappear
uiautomation.WaitForExist(button7, timeout=5)

Architecture

UIAutomation is built around the Windows UI Automation framework and provides a Python interface with several key components:

• Control Hierarchy: Tree-based structure representing UI elements from desktop root to individual controls
• Control Types: 40+ specialized control classes for different UI element types (Button, Edit, List, etc.)
• Patterns: UI Automation patterns for specialized operations (Invoke, Value, Selection, etc.)
• Search System: Flexible control location using property matching with depth control for performance
• Input Simulation: Mouse and keyboard input simulation through Windows APIs
• Screen Capture: Bitmap operations for visual verification and image-based automation

The library follows a consistent API pattern where controls are located using named parameters matching their Windows properties, then manipulated through methods or patterns.

Capabilities

Control Location and Management

Core functionality for finding, accessing, and managing UI controls within the Windows control tree. Provides the foundation for all automation operations.

def GetRootControl() -> Control: ...
def GetFocusedControl() -> Control: ...
def GetForegroundControl() -> Control: ...
def ControlFromPoint(x: int, y: int) -> Control: ...
def ControlFromHandle(handle: int) -> Control: ...
def FindControl(root: Control, **kwargs) -> Control: ...

Control Management

Control Classes and Types

Comprehensive set of 40+ control classes representing all Windows UI element types, from basic buttons and text fields to complex data grids and tree views.

class Control:
    def Click(self) -> None: ...
    def SendKeys(self, keys: str) -> None: ...
    def SetFocus(self) -> None: ...
    
class WindowControl(Control): ...
class ButtonControl(Control): ...
class EditControl(Control): ...
class ListControl(Control): ...

Control Types

Input Simulation

Mouse and keyboard input simulation for automating user interactions with applications. Supports clicking, typing, dragging, and special key combinations.

def Click(x: int, y: int) -> None: ...
def SendKey(key: int, modifiers: int = 0) -> None: ...
def SendKeys(keys: str) -> None: ...
def DragDrop(x1: int, y1: int, x2: int, y2: int) -> None: ...

Input Simulation

UI Automation Patterns

Implementation of Windows UI Automation patterns for specialized control operations like invoking buttons, setting values, handling selections, and managing scrollable content.

class InvokePattern:
    def Invoke(self) -> None: ...

class ValuePattern:
    def SetValue(self, value: str) -> None: ...
    def GetValue(self) -> str: ...

class SelectionPattern:
    def GetSelection(self) -> list: ...

Automation Patterns

Screen Capture and Imaging

Bitmap operations for capturing screenshots of controls or screen areas, pixel color analysis, and image-based verification in automation scripts.

class Bitmap:
    @staticmethod
    def FromControl(control: Control) -> Bitmap: ...
    def ToFile(self, filename: str) -> None: ...
    def GetPixelColor(self, x: int, y: int) -> int: ...

Screen Capture

Windows API Integration

Comprehensive Windows API wrapper providing low-level system operations for window management, process control, and system utilities.

class Win32API:
    @staticmethod
    def SetForegroundWindow(handle: int) -> bool: ...
    @staticmethod
    def ShowWindow(handle: int, state: int) -> bool: ...
    @staticmethod
    def GetWindowText(handle: int) -> str: ...

Windows API

Logging and Debugging

Comprehensive logging system with color support and control tree enumeration for debugging automation scripts and understanding application structure.

class Logger:
    def WriteLine(self, text: str) -> None: ...
    def ColorfulWriteLine(self, text: str, color: int) -> None: ...

def LogControl(control: Control) -> None: ...
def EnumAndLogControl(control: Control, depth: int) -> None: ...

Logging and Debugging

Error Handling

Common exceptions that may be raised:

• RuntimeError: "Can not get an instance of IUIAutomation" - requires Windows update KB971513 for Windows XP
• Control not found errors when search criteria don't match existing controls
• Access denied errors when running without administrator privileges
• Timeout errors when controls don't appear within specified time limits

Requirements and Setup

• Administrator privileges recommended for complete access to all applications
• Windows update KB971513 required for Windows XP users
• Chrome automation requires --force-renderer-accessibility command line flag
• Firefox automation supports versions ≤56 or ≥60
• Metro app automation on Windows 8/8.1 requires apps to be in foreground

Types

# Version Information
VERSION: str  # Library version string (e.g., "1.1.15")

# Constants
OPERATION_WAIT_TIME: float  # Default wait time after operations (0.5 seconds)

class Point:
    def __init__(self, x: int, y: int): ...
    x: int
    y: int

class Rect:
    def __init__(self, left: int, top: int, right: int, bottom: int): ...
    left: int
    top: int
    right: int
    bottom: int

class Control:
    Name: str
    ControlType: int
    ClassName: str
    AutomationId: str
    ProcessId: int
    IsEnabled: bool
    HasKeyboardFocus: bool
    IsKeyboardFocusable: bool
    IsOffScreen: bool
    BoundingRectangle: Rect
    Handle: int