Computer Use (macOS and Windows)

GUI control goes through the script bundled with this skill. The computer-use extension is off by default, so if you are reading this the user has switched it on.

Platform support

On Windows the script only works from a WSL shell on the same machine: it reaches the Windows desktop through WSL interop, which inherits the logged-on user's session. Run from a service, an SSH session or anything on session 0 and doctor will say so — screenshots come back black and input goes nowhere. This does not work on plain Linux without Windows, and there is no macOS-side fallback for it.

Everything below is the same on both platforms. Key combos keep the macOS spelling (cmd, shift), so cmd means the Windows key on Windows.

When to use it

• Native apps, dialogs, menu bars, installers, file pickers, System Settings.
• Anything on screen that terminal cannot reach and browser cannot automate.
• For web automation prefer the browser tool while it works. If it fails because remote debugging is unavailable, fall back here immediately — do not open chrome://inspect or run the browser-setup skill unless the user asks for it.

Workflow

1. Bring the target app forward if another window may cover it:

   ruby <skill_dir>/bin/computer.rb activate "WorkBuddy"

   It needs no extra permission on either platform. If the name does not match (localized app names differ), it prints what is actually frontmost — take a screenshot to see the real window order instead of guessing.

2. Screenshot — with a labelled grid when you will need to click precise spots:

   ruby <skill_dir>/bin/computer.rb screenshot --grid

   It prints image WxH scale S origin (x,y) and writes a sidecar .json next to the PNG. The grid image is the one to read: its magenta lines and pixel labels give exact coordinates, so prefer it over writing your own crop or PIL overlay scripts. --grid 100 sets a denser step; the default is 200.

3. Read that PNG with the read tool — it is attached to the conversation as an image. Need to read small text, or click inside a region? Crop and upscale it first, with --grid:

   ruby <skill_dir>/bin/computer.rb zoom 400,300,900,600 --grid --out /tmp/clacky-shot-2.png

   Always pass --grid: its labels are in the zoomed image's own pixel space, ready to feed straight into click, so you never convert coordinates yourself. Hand-converting a downscaled image is the usual source of misclicks — do not do it. zoom also pads the region it captures, so the origin (x,y) it prints is not the region you asked for; the grid labels are what to trust, not the origin.

   Note: the read tool downscales every image to 800px wide by default. That is fine for --grid captures; pass image_max_width: 0 only when reading a screenshot --original capture, since re-downscaling would defeat it.

4. Act, always passing the image you are looking at:

   ruby <skill_dir>/bin/computer.rb click 812 430 --from /tmp/clacky-shot-1.png
   ruby <skill_dir>/bin/computer.rb type "hello world"
   ruby <skill_dir>/bin/computer.rb key "cmd+s"

   --from accepts the grid copy too (shot-1-grid.png); it resolves back to the capture's sidecar automatically.

   type and key go to whatever holds the focus, and the script cannot tell where that is — a key aimed at a field that never got focus lands on another panel and fails silently. Screenshot to confirm the focus is where you mean it, and click the field once before typing or sending. If a field ignores type altogether (common with IME-backed native inputs), paste instead — pbcopy on macOS, clip on Windows — then key "cmd+v".

5. Screenshot again to verify the result.

Locating elements

Follow this order and do not invent your own image-processing pipeline (no PIL scripts, no pixel counting):

1. Plain screenshot for overall layout.
2. --grid when you need to click something and coordinates are not obvious.
3. zoom on a region to read small text, zoom --grid when it must be clicked.
4. screenshot --original (read with image_max_width: 0) when small detail is needed across many regions at once and zooming one by one costs more rounds — full resolution costs many more tokens, so prefer zoom first.
5. Only then click, using the coordinates read off the image you last looked at.

Coordinates

• X Y are pixels of the image you last looked at, origin at its top-left corner.
• Always pass --from <that png>. Without it the script falls back to the most recent screenshot if it is under 120 seconds old; otherwise it fails and you must screenshot again.
• Never carry coordinates over from an older screenshot — the window may have moved.

Commands

Safety

• Ask the user before changing system, browser or application settings, and before anything destructive (deleting files, sending messages, installing software).
• Exit codes: 0 ok · 2 permission missing or backend unusable · 3 screenshot state missing or stale · 4 disabled by ~/.clacky/computer.yml.
• On a permission error, relay the System Settings → Privacy & Security grant it asks for and wait for the user. Do not retry in a loop.