macOS Desktop Control Guide

Core mindset: understand first, then act

Do not see an app and immediately click. When a human opens an unfamiliar app they look around first — they notice which region is business UI, where the close button is, what entry points exist. You have to do the same.

Three-phase framework:

1. Survey    → MacosSurvey(app)  — get a mental model of the app
2. Plan      → pick a path from the "Recommended path order" the survey gives you
3. Act+Verify → MacosAct executes; MacosObserve confirms the result

Skipping Survey and going straight to Act is the single most common failure mode. WeChat / QQ / Feishu / DingTalk / Teams are self-drawn — their AX tree exposes only 3 window chrome buttons (close / min / zoom). A model that guesses path=["0","0"] lands on the red close button and kills the window. Survey will tell you in advance when you're in that trap.

Tool inventory

All tools are desktop-only. They aren't registered off-macOS; don't try to call them.

Execution priority (Survey tells you this)

MacosSurvey's response contains a Recommended path order section, highest confidence first:

1. Known intents       MacosAct type="intent"         100% confidence
2. Menu navigation     MacosAct type="menu_click"     works on self-drawn apps too
3. Keyboard shortcut   MacosAct type="key"            read the shortcut from the menu tree
4. AX path click       MacosAct type="click" ref=...  only when Survey verdict is ax-rich
5. Coord click         MacosAct type="click" point=... last resort, read pixels off a screenshot

Go down the list in order. Never skip ahead. On apps classified self-drawn, step 4 is off-limits — AX path clicks on window chrome buttons are refused (you'll get WINDOW_CHROME_BLOCKED).

What a Survey response looks like

App: 微信 (WeChat.app) (com.tencent.xinWeChat, pid=86795)

AX profile: self-drawn — AXWindow children are only chrome buttons
  richness: 0.995 (203 nodes)
  windowChildRoles: [AXCloseButton, AXFullScreenButton, AXMinimizeButton]

Known intents (from registry):
  - id: open_moments    — Open Moments feed
  - id: open_chats      — Switch to chats tab
  - id: open_discover   — Open Discover tab
  ...

Menu bar (34 entries with shortcuts, sample):
  - File > Lock WeChat  (⌘L)
  - View > Chats  (⌘1)
  - View > Moments  (⌘⇧4)
  ...

Windows (2):
  - windowID=227474 visible 1060×859 "Weixin"  (main)
  - windowID=260481 hidden 710×831

Recommended path order:
  1. Known intents (highest confidence): MacosAct type="intent" — 5 registered
  2. Menu navigation: MacosAct type="menu_click" — 34 reachable entries
  3. Keyboard shortcut: MacosAct type="key"
  4. AX path click: **DO NOT USE** on this app.
  5. Coordinate click: last resort.

"Known intents" are paths that have been vetted for you. Use them first.

Typical flows

A. Open WeChat Moments

1. MacosAct { type: "launch_app", app: "WeChat" }
2. MacosSurvey { app: "WeChat" }
   → sees Known intents contains open_moments
3. MacosAct { type: "intent", app: "WeChat", intent: "open_moments" }
4. MacosObserve { appFilter: "WeChat" } — verify Moments is showing

Do NOT observe + click AX tree right after launch_app. That's the old path, and it lands on the close button.

B. Open github.com in Safari

1. MacosSurvey { app: "Safari" }
   → Known intents has open_url (requires url arg)
2. MacosAct { type: "intent", app: "Safari", intent: "open_url", args: { url: "https://github.com" } }
3. MacosObserve { appFilter: "Safari" } — verify the page loads

C. Read the screen

User asks "what's on my screen / what's in this app": one MacosObserve call — use the screenshot + AX tree to answer. No Survey, no Act needed.

D. Unregistered intent on an ax-rich app

Survey returns ax-rich but no matching intent. Prefer menu_click or key, then AX path click:

1. MacosSurvey { app: "Finder" } → ax-rich; recommends menu_click / AX path click
2. Want a new Finder window → menu bar shows "File > New Finder Window (⌘N)"
3. MacosAct { type: "key", keys: ["cmd", "n"] } — shortcut beats click
4. MacosObserve to verify

E. Multi-window (WeChat main + image preview)

1. MacosSurvey { app: "WeChat" } → Windows: 2 entries, windowIDs 227474/260481
2. MacosCaptureWindow { windowID: 227474 } — capture the main window directly (even if preview covers it)
3. Subsequent MacosAct click coords are now based on that window's image

F. Native form (AppKit app, ax-rich)

1. MacosObserve { appFilter: "..." } to get the text field ref
2. MacosAct { type: "click", ref: { app, path: [...] } } to focus
3. MacosAct { type: "type", text: "..." }
4. MacosAct { type: "ax_action", ref: { app, path: [submit button] }, action: "AXPress" }
5. MacosObserve to verify

When Survey doesn't have the intent you want

1. Look at Survey's Menu bar section — almost every app's menu bar is menu_click-able
2. If a menu item has (⌘⇧X), MacosAct type="key" is more reliable than click
3. Last resort: observe the screenshot, then coord click
4. If none of these work, stop and tell the user "I don't have a safe path for this". Don't guess.

Rules

1. First contact with an app: Survey first. Skipping Survey is the #1 failure, especially on self-drawn UIs.
2. Prefer Survey's first-row recommendation (Known intents). Going off-script usually crashes.
3. Observe after every MacosAct. UI has state; you can't predict the outcome. Exception: after a successful intent / menu_click, if no further action is needed you can skip the verify.
4. AX path click only on ax-rich apps. mixed = cautious; self-drawn = forbidden — chrome buttons are hard-blocked (WINDOW_CHROME_BLOCKED).
5. Coord click is last resort. Must follow a recent observe. After two coord clicks with no visible change, stop and say you failed to locate.
6. Sensitive apps in foreground (password manager / bank / Keychain): stop immediately. Let the user drive.
7. When permissions are missing: the settings pane was auto-opened. Tell the user (in prose) to enable OpenLoaf there, wait for confirmation. Do not retry.
8. Non-macOS desktop: return the desktop-only error, switch to BrowserAct or hand back.

Permissions

First call prompts for missing permissions; the server auto-opens the matching System Settings pane. Required:

• screen — Screen Recording (Survey / Observe / CaptureWindow all need it)
• accessibility — Accessibility (AX tree + synthetic input)

Some system-level apps need an OpenLoaf Desktop restart after granting.

Coordinate space (when falling back to point)

point.{x,y} refers to pixels on the most recent MacosObserve or MacosCaptureWindow screenshot — read them straight off the image. The tool converts to screen coords automatically; you don't handle retina scale, window offset, or multi-display yourself.

• The Screenshot: window 2120×1718 px line in observe/capture_window output is the coordinate range
• You must call observe / capture_window at least once before coord actions; otherwise the tool refuses

Safety & privacy

• Do not Survey / Observe / Act on password / bank / Keychain apps in the foreground. Hand it back to the user.
• Screenshots live in the session asset dir and are cleaned with the session. Not uploaded, not trained on.
• When uncertain about current screen state, observe first. Don't guess.

Error diagnosis

• permissionsMissing → settings pane already opened, prompt user, wait
• desktop-only → not running under OpenLoaf Desktop; switch platforms
• WINDOW_CHROME_BLOCKED → you clicked a chrome button. Pick one of the 4 suggested fallbacks (menu_click / key / coord / URL). Do NOT add confirm_window_chrome:true unless you really do mean to close the window.
• "intent not found" → call MacosSurvey to list available intents; otherwise degrade to menu_click.
• Click has no effect → re-observe; check whether the target is really actionable or was covered.
• Foreground switched mid-flow → observe's app no longer matches act's; key cmd+tab back first.