v1.20Grok Build CLI support →

Computer

Let an agent click, type, and take screenshots inside any Mac app — the same way assistive technology does, without needing the app to have a web interface.

When to use it

agents computer drives macOS applications through a small helper app called Computer Helper. Use it when you want to automate a desktop app that has no web interface — a native finance tool, a system preferences pane, or anything a browser cannot reach.

This is macOS-only.

How the allow-list works

By default the helper is allowed to drive zero apps. You name each one you want to automate. The agent cannot click inside Mail unless you explicitly say Mail is allowed — nothing runs silently in the background.


+---------------------------+     +---------------------------+
| 1. Agent asks:            |     | 2. Request goes to        |
|    "click Send in Mail"   | --> |    Computer Helper        |
+---------------------------+     +---------------------------+
                                            |
                                            v
+---------------------------+     +---------------------------+
| 4. Helper clicks Send,    |     | 3. Helper checks          |
|    returns confirmation   | <-- |    allow-list: Mail OK    |
+---------------------------+     +---------------------------+

One-time setup

Four steps, done once per machine:

Step 1 — Install the helper app:

agents computer install-helper

Step 2 — Grant permissions in System Settings (manual):

System Settings > Privacy & Security > Accessibility     — add Computer Helper.app
System Settings > Privacy & Security > Screen Recording  — add Computer Helper.app

Step 3 — Name the apps you want to drive. Create a file at ~/.agents/permissions/groups/computer.yaml:

name: computer
allow:
  - "Computer(com.apple.mail)"
  - "Computer(com.apple.finder)"
  - "Computer(com.apple.notes)"

To find an app's identifier: defaults read /Applications/SomeApp.app/Contents/Info CFBundleIdentifier

Step 4 — Start the helper and confirm it works:

agents computer start
agents computer status            # look for trust: granted
agents computer screenshot --out /tmp/snapshot.jpg

If trust: denied appears, the System Settings step did not complete. Add the app there, then run agents computer start again.

Add an app without restarting

# Append to the permissions file, then reload
echo '  - "Computer(com.apple.calendar)"' >> ~/.agents/permissions/groups/computer.yaml
agents computer reload
# Output: policy: 4 apps allowed (com.apple.mail, ...)

Stop when you are done

Accessibility and Screen Recording are sensitive permissions. Stop the helper when you are not using it.

agents computer stop

Command reference

agents computer install-helper    # copy helper to /Applications/, write config
agents computer start             # start the helper
agents computer stop              # stop the helper
agents computer reload            # reload the allow list without restarting
agents computer status            # install state, running state, permission trust
agents computer screenshot        # capture a JPEG of the frontmost allowed app

Related