0
# UI Element Selection and Interaction
1

2
Comprehensive UI element selection using UiAutomator selectors, coordinate-based targeting, and gesture operations. Supports clicking, text input, gestures, and element property inspection.
3

4
## Capabilities
5

6
### Element Selection
7

8
Select UI elements using UiAutomator selector attributes.
9

10
```python { .api }
11
class Device:
12
    def __call__(self, **kwargs) -> UiObject:
13
        """
14
        Create UI element selector.
15

16
        Parameters:
17
        - text: Exact text match
18
        - textContains: Partial text match
19
        - textMatches: Regular expression text match
20
        - textStartsWith: Text prefix match
21
        - className: UI element class name
22
        - classNameMatches: Regular expression class name match
23
        - description: Content description match
24
        - descriptionContains: Partial description match
25
        - descriptionMatches: Regular expression description match
26
        - descriptionStartsWith: Description prefix match
27
        - resourceId: Resource ID match  
28
        - resourceIdMatches: Regular expression resource ID match
29
        - packageName: Package name match
30
        - packageNameMatches: Regular expression package name match
31
        - checkable: Boolean checkable state
32
        - checked: Boolean checked state
33
        - clickable: Boolean clickable state
34
        - longClickable: Boolean long clickable state
35
        - scrollable: Boolean scrollable state
36
        - enabled: Boolean enabled state
37
        - focusable: Boolean focusable state
38
        - focused: Boolean focused state
39
        - selected: Boolean selected state
40
        - index: Element index within parent
41
        - instance: Element instance number
42

43
        Returns:
44
        UiObject for interaction
45
        """
46

47
    def exists(self, **kwargs) -> bool:
48
        """Check if UI element exists using selector attributes"""
49
```
50

51
Usage examples:
52

53
```python
54
d = u2.connect()
55

56
# Select by text
57
button = d(text="OK")
58
login_btn = d(textContains="Login")
59
submit_btn = d(textMatches=r"Submit|Send")
60

61
# Select by resource ID
62
username_field = d(resourceId="com.example:id/username")
63
password_field = d(resourceId="com.example:id/password")
64

65
# Select by class and attributes
66
text_view = d(className="android.widget.TextView", clickable=True)
67
edit_text = d(className="android.widget.EditText", enabled=True)
68

69
# Select by index
70
first_item = d(className="android.widget.LinearLayout", index=0)
71

72
# Check existence
73
if d(text="Settings").exists:
74
    print("Settings button found")
75
```
76

77
### Element Properties and Information
78

79
Access UI element properties and attributes.
80

81
```python { .api }
82
class UiObject:
83
    @property
84
    def exists(self) -> bool:
85
        """Check if element exists in current UI"""
86

87
    @property
88
    def info(self) -> Dict[str, Any]:
89
        """Get element attributes including bounds, text, className, etc."""
90

91
    def screenshot(self, display_id: Optional[int] = None) -> Image.Image:
92
        """Take screenshot of element bounds"""
93
```
94

95
Usage examples:
96

97
```python
98
d = u2.connect()
99

100
# Get element info
101
element = d(resourceId="com.example:id/title")
102
if element.exists:
103
    info = element.info
104
    print(f"Text: {info['text']}")
105
    print(f"Bounds: {info['bounds']}")
106
    print(f"Class: {info['className']}")
107
    print(f"Clickable: {info['clickable']}")
108

109
# Screenshot element
110
element_image = element.screenshot()
111
element_image.save("element.png")
112
```
113

114
### Coordinate-based Interaction
115

116
Direct interaction using screen coordinates.
117

118
```python { .api }
119
class Device:
120
    def click(self, x: Union[float, int], y: Union[float, int]):
121
        """
122
        Click at coordinates.
123

124
        Parameters:
125
        - x: X coordinate (absolute pixels or relative 0-1)
126
        - y: Y coordinate (absolute pixels or relative 0-1)
127
        """
128

129
    def double_click(self, x, y, duration=0.1):
130
        """
131
        Double click at coordinates.
132

133
        Parameters:
134
        - x, y: Coordinates
135
        - duration: Delay between clicks in seconds
136
        """
137

138
    def long_click(self, x, y, duration: float = 0.5):
139
        """
140
        Long click at coordinates.
141

142
        Parameters:
143
        - x, y: Coordinates  
144
        - duration: Press duration in seconds
145
        """
146

147
    @property
148
    def pos_rel2abs(self):
149
        """Function to convert relative (0-1) to absolute pixel coordinates"""
150
```
151

152
Usage examples:
153

154
```python
155
d = u2.connect()
156

157
# Absolute coordinates
158
d.click(100, 200)
159
d.double_click(300, 400)
160
d.long_click(500, 600, duration=2.0)
161

162
# Relative coordinates (0-1 range)
163
d.click(0.5, 0.3)  # Center horizontally, 30% from top
164
d.click(0.1, 0.9)  # Near bottom-left corner
165

166
# Convert relative to absolute
167
convert = d.pos_rel2abs
168
abs_x, abs_y = convert(0.5, 0.5)  # Get center coordinates
169
```
170

171
### Gesture Operations
172

173
Swipe, drag, and multi-touch gestures.
174

175
```python { .api }
176
class Device:
177
    def swipe(self, fx, fy, tx, ty, duration: Optional[float] = None, steps: Optional[int] = None):
178
        """
179
        Swipe from one point to another.
180

181
        Parameters:
182
        - fx, fy: From coordinates
183
        - tx, ty: To coordinates
184
        - duration: Swipe duration in seconds
185
        - steps: Number of interpolation steps (overrides duration)
186
        """
187

188
    def swipe_points(self, points: List[Tuple[int, int]], duration: float = 0.5):
189
        """
190
        Multi-point swipe gesture.
191

192
        Parameters:
193
        - points: List of (x, y) coordinate tuples
194
        - duration: Total gesture duration
195
        """
196

197
    def drag(self, sx, sy, ex, ey, duration=0.5):
198
        """
199
        Drag from start to end coordinates.
200

201
        Parameters:
202
        - sx, sy: Start coordinates
203
        - ex, ey: End coordinates  
204
        - duration: Drag duration in seconds
205
        """
206

207
    @property
208
    def touch(self) -> TouchActions:
209
        """Touch gesture builder for complex multi-touch operations"""
210
```
211

212
Usage examples:
213

214
```python
215
d = u2.connect()
216

217
# Basic swipe gestures
218
d.swipe(100, 500, 100, 100)  # Swipe up
219
d.swipe(500, 300, 100, 300)  # Swipe left
220
d.swipe(100, 100, 500, 500, duration=2.0)  # Slow diagonal swipe
221

222
# Multi-point swipe
223
points = [(100, 100), (200, 150), (300, 100), (400, 200)]
224
d.swipe_points(points, duration=1.5)
225

226
# Drag operation
227
d.drag(100, 100, 500, 500, duration=1.0)
228

229
# Complex touch gestures
230
d.touch.down(100, 100).move(200, 200).sleep(0.5).up(200, 200)
231
```
232

233
### Advanced Gesture Support
234

235
Extended swipe functionality for common UI patterns.
236

237
```python { .api }
238
class Device:
239
    @cached_property
240
    def swipe_ext(self) -> SwipeExt:
241
        """Extended swipe gestures for common UI patterns"""
242

243
class SwipeExt:
244
    def up(self, scale: float = 0.8):
245
        """Swipe up gesture"""
246

247
    def down(self, scale: float = 0.8):
248
        """Swipe down gesture"""
249

250
    def left(self, scale: float = 0.8):
251
        """Swipe left gesture"""
252

253
    def right(self, scale: float = 0.8):
254
        """Swipe right gesture"""
255
```
256

257
Usage examples:
258

259
```python
260
d = u2.connect()
261

262
# Extended swipe gestures
263
d.swipe_ext.up(scale=0.9)    # Swipe up 90% of screen
264
d.swipe_ext.down()           # Swipe down default 80%
265
d.swipe_ext.left(scale=0.7)  # Swipe left 70% of screen
266
d.swipe_ext.right()          # Swipe right default 80%
267
```
268

269
### Element Interaction
270

271
Direct interaction with selected UI elements.
272

273
```python { .api }
274
class UiObject:
275
    def click(self, timeout: Optional[float] = None):
276
        """Click the UI element"""
277

278
    def long_click(self, duration: float = 0.5):
279
        """Long click the UI element"""
280

281
    def set_text(self, text: str):
282
        """Set text content of element"""
283

284
    def clear_text(self):
285
        """Clear text content of element"""
286

287
    def wait(self, timeout: Optional[float] = None) -> bool:
288
        """Wait for element to appear"""
289

290
    def wait_gone(self, timeout: Optional[float] = None) -> bool:
291
        """Wait for element to disappear"""
292

293
    def click_exists(self, timeout=0) -> bool:
294
        """Click element if it exists within timeout"""
295

296
    def click_gone(self, maxretry=10, interval=1.0):
297
        """Click element repeatedly until it disappears"""
298

299
    def bounds(self) -> Tuple[int, int, int, int]:
300
        """Get element bounds (left, top, right, bottom)"""
301

302
    def center(self, offset=(0.5, 0.5)) -> Tuple[int, int]:
303
        """Get element center coordinates with optional offset"""
304

305
    def drag_to(self, **kwargs):
306
        """Drag element to another element or coordinates"""
307
```
308

309
Usage examples:
310

311
```python
312
d = u2.connect()
313

314
# Element interaction
315
button = d(text="Submit")
316
button.click()
317

318
# Text input
319
username = d(resourceId="com.example:id/username")
320
username.clear_text()
321
username.set_text("john_doe")
322

323
# Wait for elements
324
loading = d(text="Loading...")
325
loading.wait(timeout=10)  # Wait up to 10 seconds
326
loading.wait_gone(timeout=30)  # Wait for loading to disappear
327

328
# Advanced element interaction
329
button = d(text="Optional Button")
330
if button.click_exists(timeout=5):  # Click if exists within 5 seconds
331
    print("Button clicked")
332

333
# Remove popup by clicking repeatedly
334
popup_close = d(resourceId="close_popup")
335
popup_close.click_gone(maxretry=5, interval=0.5)  # Click until gone
336

337
# Get element position and size
338
element = d(resourceId="target_element")
339
bounds = element.bounds()  # (left, top, right, bottom) 
340
print(f"Element bounds: {bounds}")
341

342
center_x, center_y = element.center()  # Get center point
343
print(f"Element center: ({center_x}, {center_y})")
344

345
# Get center with offset (0.8, 0.2 means 80% right, 20% down within element)
346
offset_point = element.center(offset=(0.8, 0.2))
347

348
# Drag element to another location
349
source = d(text="Drag Me")
350
target = d(text="Drop Here") 
351
source.drag_to(target)  # Drag to another element
352
source.drag_to(x=500, y=300)  # Drag to coordinates
353
```
354

355
### Element Navigation and Hierarchy
356

357
Navigate through UI hierarchy using parent-child relationships.
358

359
```python { .api }
360
class UiObject:
361
    def child(self, **kwargs) -> UiObject:
362
        """Find child element using selector attributes"""
363
    
364
    def sibling(self, **kwargs) -> UiObject:
365
        """Find sibling element using selector attributes"""
366
    
367
    def child_by_text(self, txt: str, **kwargs) -> UiObject:
368
        """Find child element by text match"""
369
    
370
    def child_by_description(self, txt: str, **kwargs) -> UiObject:
371
        """Find child element by description match"""
372
    
373
    def child_by_instance(self, inst: int, **kwargs) -> UiObject:
374
        """Find child element by instance number"""
375
```
376

377
Usage examples:
378

379
```python
380
d = u2.connect()
381

382
# Navigate through hierarchy
383
container = d(resourceId="container")
384
submit_button = container.child(text="Submit")
385
submit_button.click()
386

387
# Find specific child by text
388
menu = d(className="android.widget.LinearLayout")
389
settings_item = menu.child_by_text("Settings")
390
settings_item.click()
391

392
# Find sibling elements
393
current_item = d(text="Current Item")
394
next_item = current_item.sibling(text="Next Item")
395
next_item.click()
396
```
397

398
### Relative Positioning
399

400
Find elements relative to other elements on screen.
401

402
```python { .api }
403
class UiObject:
404
    def right(self, **kwargs) -> UiObject:
405
        """Find element to the right of current element"""
406
    
407
    def left(self, **kwargs) -> UiObject:
408
        """Find element to the left of current element"""
409
    
410
    def up(self, **kwargs) -> UiObject:
411
        """Find element above current element"""
412
    
413
    def down(self, **kwargs) -> UiObject:
414
        """Find element below current element"""
415
```
416

417
Usage examples:
418

419
```python
420
d = u2.connect()
421

422
# Relative positioning
423
username_field = d(resourceId="username")
424
password_field = username_field.down(resourceId="password")
425
password_field.set_text("secret123")
426

427
# Find button to the right of a label
428
price_label = d(text="Price:")
429
edit_button = price_label.right(text="Edit")
430
edit_button.click()
431
```
432

433
### Scrolling and Flinging
434

435
Scroll elements within containers with precise control.
436

437
```python { .api }
438
class UiObject:
439
    @property
440
    def scroll(self):
441
        """Scroll gesture builder for element"""
442
    
443
    @property  
444
    def fling(self):
445
        """Fling gesture builder for element"""
446

447
class ScrollBuilder:
448
    def forward(self, steps: int = 10): """Scroll forward (up/left)"""
449
    def backward(self, steps: int = 10): """Scroll backward (down/right)"""
450
    def toBeginning(self, steps: int = 10): """Scroll to beginning"""
451
    def toEnd(self, steps: int = 10): """Scroll to end"""
452
    def to(self, **kwargs): """Scroll until element is visible"""
453

454
class FlingBuilder:
455
    def forward(self): """Fling forward quickly"""
456
    def backward(self): """Fling backward quickly"""
457
    def toBeginning(self): """Fling to beginning"""
458
    def toEnd(self): """Fling to end"""
459
```
460

461
Usage examples:
462

463
```python
464
d = u2.connect()
465

466
# Scroll within a list or container
467
scrollable_list = d(scrollable=True)
468
scrollable_list.scroll.forward(steps=5)  # Scroll up 5 steps
469
scrollable_list.scroll.backward(steps=10)  # Scroll down 10 steps
470
scrollable_list.scroll.toBeginning()  # Scroll to top
471
scrollable_list.scroll.toEnd()  # Scroll to bottom
472

473
# Scroll until specific element is visible
474
scrollable_list.scroll.to(text="Target Item")
475

476
# Fast fling gestures
477
scrollable_list.fling.forward()  # Quick fling up
478
scrollable_list.fling.toEnd()  # Quick fling to bottom
479
```
480

481
### Advanced Gestures
482

483
Multi-finger gestures for complex interactions.
484

485
```python { .api }
486
class UiObject:
487
    def gesture(self, start1: Tuple[float, float], start2: Tuple[float, float], 
488
                end1: Tuple[float, float], end2: Tuple[float, float], steps: int = 100):
489
        """Two-finger gesture between specified points"""
490
    
491
    def pinch_in(self, percent: int = 100, steps: int = 50):
492
        """Pinch in gesture (zoom out)"""
493
    
494
    def pinch_out(self, percent: int = 100, steps: int = 50):
495
        """Pinch out gesture (zoom in)"""
496
```
497

498
Usage examples:
499

500
```python
501
d = u2.connect()
502

503
# Pinch gestures on image or map
504
image_view = d(className="ImageView")
505
image_view.pinch_out(percent=200, steps=30)  # Zoom in 200%
506
image_view.pinch_in(percent=50, steps=30)  # Zoom out 50%
507

508
# Custom two-finger gesture
509
map_view = d(resourceId="map_view")
510
# Two-finger drag from corners to center
511
map_view.gesture(start1=(0.1, 0.1), start2=(0.9, 0.9), 
512
                end1=(0.4, 0.4), end2=(0.6, 0.6), steps=50)
513
```
514

515
### Element Indexing and Iteration
516

517
Work with multiple matching elements using indexing and iteration.
518

519
```python { .api }
520
class UiObject:
521
    def __getitem__(self, index: int) -> UiObject:
522
        """Get element by index when multiple matches exist"""
523
    
524
    def __len__(self) -> int:
525
        """Get count of matching elements"""
526
    
527
    def __iter__(self):
528
        """Iterate over all matching elements"""
529
    
530
    @property
531
    def count(self) -> int:
532
        """Get count of matching elements"""
533
```
534

535
Usage examples:
536

537
```python
538
d = u2.connect()
539

540
# Work with multiple matching elements
541
buttons = d(className="android.widget.Button")
542
print(f"Found {len(buttons)} buttons")  # Get count
543
print(f"Found {buttons.count} buttons")  # Alternative count
544

545
# Access specific element by index
546
first_button = buttons[0]  # First button
547
last_button = buttons[-1]  # Last button
548
first_button.click()
549

550
# Iterate over all matching elements
551
for i, button in enumerate(buttons):
552
    print(f"Button {i}: {button.get_text()}")
553
    if button.get_text() == "Submit":
554
        button.click()
555
        break
556
```