A deep dive into agent-desktop
By Flavio
Updated
Tagged: AI
agent-desktop is Vercel Labs' CLI for computer use on macOS, Windows, and Linux. Agents drive desktop apps through the accessibility tree and stable refs.
agent-desktop is a command line tool from Vercel Labs. A coding agent uses it to see a desktop app and operate it.
It reads the operating system’s accessibility tree, the same structure a screen reader uses. Each control comes back as JSON, with a ref the agent can click, type into, or read. The loop stays in your agent, because agent-desktop is the tool and not the model.
Chris Tate introduced it on 31 October 2026 and called it “agent-browser, for your whole computer.” His post, which includes a demo video, describes computer use for any coding agent on macOS, Windows, and Linux. It runs in the background while you keep working, and several agents can use the computer in parallel, each with its own cursor.
This guide covers agent-desktop 1.2.0. The changelog entry is dated 5 November 2026, and the GitHub release came out the same day. Commands, flags, and error codes come from that release, docs/json-output.md, the MCP guide, and the docs at agent-browser.dev/desktop. The Linux transcripts are from a run of 1.0.2 on Ubuntu on 1 November 2026, so their envelopes say "2.5".
The package is Apache-2.0. The npm page is agent-desktop. It needs Node 18 or newer and ships native binaries for macOS, Windows, and Linux, on x64 and ARM64.
What changed in 1.2.0?
1.2.0 adds agent-desktop mcp. That command runs a Model Context Protocol server over stdio. Each leaf CLI command becomes a tool, and a call returns that command’s JSON envelope. The client config is in how you connect it. The source is the MCP guide.
type, set-value, and clipboard-set accept --stdin. They read UTF-8 text from a pipe or a redirected file, instead of a command-line argument.
On macOS, 1.2.0 can expose and select a list row whose selection can only be changed through its container.
What changed in 1.1.0?
1.1.0 is a breaking release, with a version 2.6 migration for the JSON envelope.
Refs got stricter on Windows and Linux. A ref written before 1.1.0 now returns AMBIGUOUS_TARGET instead of acting, and so does a ref to one of several identical controls. On 1.0.2 some of those still acted, so take a fresh snapshot after you upgrade. On macOS, a ref from an older snapshot still resolves while the element its token names is alive. The rules are in how refs work below.
The macOS command host is gone. It used to stay up and exit after 15 idle minutes. Now every CLI call runs in its own process on macOS, Windows, and Linux, and the npm install, like every CLI call, stops a host an older version left running.
A surface kind the OS does not have returns INVALID_ARGS and names the kinds to use. An adapter with no snapshot surfaces at all still returns PLATFORM_NOT_SUPPORTED.
On macOS you can read the Dock and the menu bar status items:
agent-desktop snapshot --surface dock
agent-desktop snapshot --surface menu-bar-extras
open-system-surface returns either of those surfaces at once, headless, and its data has no window. With --app, menu-bar-extras reads that app’s own status item. A background app with no window, such as Control Center, Notification Center, or a menu bar app, is observed through its process. --app matches the display name, the executable name, or the bundle id, so --app ControlCenter works. Windows opens its shell surfaces and needs --headed, and Linux is limited to GNOME shell surfaces.
The release also adds type --interval-ms, a cursor overlay that shows the text caret, and C ABI major version 7. Each one is covered in its section below.
What is agent-desktop?
An agent usually has a shell and a browser. A lot of software still lives in a window with no API, like a settings pane or a dialog in a desktop app.
agent-desktop is the CLI for that window. You install one binary and the agent runs commands. Each command prints one JSON object, which the agent reads before it decides on the next command.
agent-browser is the older sibling, and it stays inside a browser. Its npm latest on 1 November 2026 was 0.41.1, described as a browser automation CLI for AI agents. You install it, then run agent-browser install to fetch Chrome for Testing. Its snapshot is the accessibility tree of a page, and its quick start clicks a bare ref such as @e2.
agent-desktop points the same idea at the whole desktop, and its refs look different on purpose. A desktop ref includes the snapshot id, like @s8f3k2p9:e8, and a bare @e8 is rejected. I mention agent-browser briefly in which browser to give your AI agents on macOS, which is about browsers rather than native windows.
| agent-desktop | agent-browser | |
|---|---|---|
| What it drives | Desktop apps on macOS, Windows, and Linux | A browser, Chrome for Testing by default |
| How it sees the UI | OS accessibility tree | Page accessibility tree |
| Ref shape | @<snapshot_id>:eN | Bare @eN in the quick start |
| Install | npm install -g agent-desktop | npm install -g agent-browser, then agent-browser install |
| Homebrew | Not in the default catalog yet | brew install agent-browser is documented |
| License | Apache-2.0 | Apache-2.0, also Vercel Labs |
How does agent-desktop work?
The agent observes, acts, and observes again. The README’s macOS example is Finder:
agent-desktop snapshot --app Finder -i
agent-desktop click @s8f3k2p9:e8
agent-desktop snapshot --app Finder -i
-i keeps interactive elements. The first snapshot assigns refs, the click uses a ref from it, and the second snapshot is how the agent checks the result. Nothing in that loop matches pixels, though there’s a separate screenshot command when you want an image.
Refs belong to one snapshot
The snapshot id is part of the ref, so @sofkkd3q8w04y:e269 names both the element and the snapshot that produced it.
On macOS a ref follows its element. A removed row returns STALE_REF, and identical controls can each be used by ref. An element with no address gets identity_not_unique.
Windows and Linux are stricter. A ref acts only when one complete read proved it was unique, and exactly one live element still matches its role and stable fields. Otherwise the command returns AMBIGUOUS_TARGET with details.kind set to identity_not_unique. That happens for a ref to one of several identical controls, a ref from a window the snapshot could not read whole, and a ref saved before 1.1.0. Identity is the role plus the stable fields, so an identical control that replaced the original is acted on.
When a snapshot cannot prove its refs, data.identity_unproven is true. Raise --timeout-ms and snapshot again.
Windows, dense apps, and headless input
If snapshot cannot tell which window you mean, it also returns AMBIGUOUS_TARGET, and lists candidate window ids in error.details. Pass one with --window-id. This case is about the window, while identity_not_unique is about the element.
For a dense app, start with a skeleton. The README uses Slack: snapshot --skeleton --app Slack -i --compact is the shallow map, then snapshot --root @s8f3k2p9:e3 -i --compact drills into one region.
Ref actions are headless by default. They go through the accessibility API, and they don’t move the pointer, take focus, synthesize keys, or use the clipboard. The global --headed flag allows physical input, and the docs say not to flip it just to recover from a headless failure. A raw --xy coordinate never implies focus.
In PowerShell @ is the splat operator, so quote the ref: '@s8f3k2p9:e8'. cmd.exe and bash don’t need the quotes.
Every command prints one envelope. Here’s a success, from the JSON contract:
{
"version": "2.6",
"ok": true,
"command": "click",
"data": {
"action": "click",
"disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }
}
}
version here is the envelope schema. The CLI release shows up inside the data of the version command, as "1.2.0" on this release.
How do you install agent-desktop?
The documented install is global npm, on macOS, Windows, or Linux:
npm install -g agent-desktop
agent-desktop version
Bun, Yarn, and pnpm can install the same package. A non-global install lands in ./node_modules/.bin, and you run it with npx agent-desktop.
The postinstall script replaces a Node launcher with a link to the native binary, which the install doc puts at about 21 ms per call. If the package manager blocks install scripts, as npm 12 does by default, the Node launcher stays and each call costs about 146 ms. Since 1.0.2 a one-time notice on stderr tells you. Approve the script with npm install-scripts approve agent-desktop, or install with --allow-scripts. After an install that skips npm scripts, or a manual install, run agent-desktop version once so it stops any command host an older version left running.
You can also download the native archive from GitHub Releases. Keep the helper beside the executable, and upgrade by replacing both from the same release. Checksums catch corruption, but to check the archive came from the project the docs give this command:
gh attestation verify <downloaded-archive> --repo vercel-labs/agent-desktop
Official Homebrew and Scoop packages are being submitted and aren’t in the default catalogs yet. The repo no longer maintains a custom tap or bucket, and an older custom install can shadow the new binary. Check with command -v agent-desktop on macOS and Linux, or Get-Command agent-desktop on Windows.
Building from source needs Rust 1.89 or later, plus Xcode command line tools on macOS 13 or newer, MSVC on Windows, or a C linker on Linux:
git clone https://github.com/vercel-labs/agent-desktop && cd agent-desktop
cargo build --release -p agent-desktop
On Linux, also build agent-desktop-linux and keep agent-desktop-linux-clipboard-owner beside the CLI. npm and the release archives already include that helper.
The Linux run on this page used a user prefix, so it did not need root:
npm install -g agent-desktop --prefix "$HOME/.local"
On that 1.0.2 install, agent-desktop version printed:
{
"version": "2.5",
"ok": true,
"command": "version",
"data": {
"os": "linux",
"target": "x86_64",
"version": "1.0.2"
}
}
The machine was Ubuntu 24.04.5 LTS, x86_64, glibc 2.39, with XFCE on X11. The docs measure on Ubuntu 24.04 or newer with GNOME 46, and set a glibc 2.35 floor. at-spi2-core was already installed. The shell was not part of the graphical session, so the run copied DISPLAY and DBUS_SESSION_BUS_ADDRESS from that session before any command could see apps. As the install doc says, a TTY or SSH shell without the session bus, the display, and the runtime directory can’t see them.
Which permissions does each OS need?
Run agent-desktop permissions before you automate anything. It reports state and does not prompt.
macOS
You need macOS 13 or newer, Intel or Apple Silicon. Grant Accessibility to the app that launches agent-desktop, whether that’s Terminal, iTerm2, Warp, VS Code, Codex, or something else. macOS 27 and later call that list Device Control and Data Access. If System Settings also lists the agent-desktop binary, grant that too.
permissions --request can prompt. permissions --guide opens System Settings at the missing permission and returns immediately, with guide set to started, already_running, or not_needed. The guide covers Accessibility and Screen Recording only, and doesn’t prompt. Since 1.0.2 it offers the app that launched the CLI, even a terminal Launch Services does not know.
Screenshots, and session start --screenshots, need Screen Recording. Opening Notification Center needs Automation permission for System Events, and Automation reports unknown when macOS would have to prompt. After you grant access, run permissions again. The launching app does not need a restart.
Windows
You need Windows 10 1809 or newer, or Server 2019 or newer, x64 or ARM64. Apps at your terminal’s integrity level need no extra grant, and permissions reports automation as not_required. An agent can read an admin app, but only an elevated agent can send it input. Windows enforces this with UIPI, and elevated input from a medium-integrity caller maps to PERM_DENIED.
The binary is unsigned. SmartScreen may warn on a browser download, antivirus may quarantine it, and Smart App Control on a clean Windows 11 install can block it. The npm install path does not trigger SmartScreen. --guide returns PLATFORM_NOT_SUPPORTED on anything but macOS.
Session 0, Server Core, the secure desktop (UAC, sign-in), a locked desktop, and another user’s session are not supported. launch takes an absolute path, or a bare name under System32 or the Windows directory, with or without .exe. notepad works, a display name such as “Google Chrome” does not.
type and right-click need --headed, and headless text goes through set-value. Minimized windows report offscreen and ref actions fail, so restore and snapshot again. These shortcuts refuse without --force: alt+f4, win+l, win+d, alt+tab, win+r, win+x, win+e, ctrl+shift+esc, and modifier supersets. In combos, cmd and meta are the Windows key.
Linux
The primary measured target is Ubuntu 24.04 or newer with GNOME 46. Ubuntu 22.04’s ATK bridge aborts on Action.GetActions, and other distributions need the at-spi2-core bridge fix the Linux skill names as dc0dc33. Run inside the signed-in user’s graphical session.
agent-desktop status activates and checks the accessibility bus. An empty supported_surfaces list means the bus did not answer, not that the desktop is empty.
X11 and Wayland both work, with differences. X11 keys need XTEST, Wayland keys need GNOME Mutter RemoteDesktop, and the Wayland clipboard needs wl-clipboard. On Wayland, app pointer input refuses and native window control is unsupported. focus-window requires X11.
On the XFCE X11 run of 1.0.2, status returned granted accessibility, automation not_required, screen recording not_required, and surfaces window, sheet, alert, and menu. permissions --guide exited 1 with the documented non-macOS result:
{
"version": "2.5",
"ok": false,
"command": "permissions",
"error": {
"code": "PLATFORM_NOT_SUPPORTED",
"message": "start_permission_guide is not supported on this platform",
"suggestion": "Use a platform/build that advertises this capability or choose a supported command",
"disposition": { "delivery": "not_delivered", "retry": "safe" }
}
}
The state root is ~/.agent-desktop. It must be private, mode 700, or snapshot fails with INTERNAL. AGENT_DESKTOP_HOME relocates it to an absolute path, used as is with nothing appended.
--app takes the name list-apps prints, or a desktop-entry name or id, but not a window title. Chromium and Qt apps need a launch flag or variable before they show up on the bus, and the Linux skill documents those.
How do you connect it to Claude Code, Codex, or Cursor?
1.2.0 ships an MCP server. agent-desktop mcp speaks Model Context Protocol over stdio, and each leaf CLI command is a tool. Add this to the client’s MCP settings:
{
"mcpServers": {
"agent-desktop": { "command": "agent-desktop", "args": ["mcp"] }
}
}
The MCP guide is the reference for that server.
A tool name is desktop_ plus the command, with hyphens turned into underscores. snapshot is desktop_snapshot. list-windows is desktop_list_windows. session start is desktop_session_start. skills get is desktop_skills_get.
Start with the skills. Call desktop_skills_get with { "name": "desktop" }, then with { "name": "platform" }.
Global flags go on mcp itself. This applies --session and --headed to every call:
agent-desktop mcp --session s1 --headed
A tool cannot pass a global flag.
The server has the same reach as the CLI. Treat it like a shell for the signed-in user.
The skill and the AGENTS.md paragraph are still the path for an agent that runs a shell. MCP is a second path, for a client that speaks MCP.
The binary carries docs that match its version:
agent-desktop skills get desktop
agent-desktop skills get platform
agent-desktop snapshot --help
skills get desktop returns markdown in the JSON envelope, under data.content. References load the same way: agent-desktop skills get desktop commands-observation, and the same for commands-interaction, commands-system, and workflows.
Install the skill from skills.sh:
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop
Or add the README’s paragraph to AGENTS.md:
Use `agent-desktop` to see and operate desktop apps. Before the first use, run
`agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.
After that, any agent that can run a shell can call the binary. The agent’s own permission modes still apply, separately from the OS accessibility grant.
Some products ship their own computer use, and those are different tools. Claude Code has a computer-use server on macOS. Codex has a Computer Use plugin in the desktop app, which I wrote up in how to use the Codex desktop app. Cursor has Cursor Computer Use on Cloud Agents and self-hosted machines.
Use the product’s own feature when you are already inside that product and the task fits it. Use agent-desktop when you want one CLI, the accessibility tree, and the same commands from a shell or from an MCP client.
Which commands does it have?
The README counts 60 command names, 56 of them operational. key-down, key-up, mouse-down, and mouse-up refuse to run until a daemon can release a held key or button.
| Group | Commands |
|---|---|
| Observe | snapshot, find, get, is, screenshot, list-surfaces |
| Act on a ref | click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to |
| Keyboard and mouse | press, hover, drag, mouse-move, mouse-click, mouse-wheel, plus the four held-input commands that refuse for now |
| Apps and windows | launch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface |
| Notifications | list-notifications, dismiss-notification, dismiss-all-notifications, notification-action |
| Clipboard | clipboard-get, clipboard-set, clipboard-clear |
| Wait | wait |
| Sessions | session, trace, cursor-overlay |
| System | status, permissions, version, batch, skills |
batch runs entries in order in one process, without making them a transaction. It rejects unknown fields and nested batch entries, and --stop-on-error stops at the first failure.
Auto-wait is on by default. A ref action polls until the target is actionable, then fails with TIMEOUT. CONCEPTS.md sets the bound at 5000 ms, and type adds 30 ms per character. --timeout-ms 0 is a single shot. launch --timeout defaults to 30000 ms and set-value --timeout-ms to 5000.
type can pace itself. type --interval-ms 60 sends one keystroke every 60 ms, and the value can be from 1 to 2000. With that flag, --timeout-ms applies to each keystroke, and the command reads the field after the last one.
Since 1.2.0, type, set-value, and clipboard-set also accept --stdin. The text is UTF-8 from a pipe or a redirected file.
Platform support, from the README:
| macOS | Windows | Linux | |
|---|---|---|---|
| Accessibility tree | Yes | Yes | Yes, AT-SPI |
| Click, type, keyboard | Yes | Yes | AT-SPI, plus X11 or GNOME keys |
| Mouse input | Yes | Yes | X11 with --headed |
| Screenshot | Yes | Yes | X11 windows, Wayland portal |
| Clipboard | Yes | Yes | Yes, Wayland needs wl-clipboard |
| App and window management | Yes | Yes | Launch and list, X11 window control |
| Notifications | Yes | Yes | GNOME Shell |
| Shell surfaces | Dock and menu bar extras | Yes | GNOME tray and Action Center |
agent-desktop skills get platform prints the table for the OS you are on. Shell surfaces go through open-system-surface and snapshot --surface, as shown in the 1.1.0 section.
Each release also ships libagent_desktop_ffi, a C library so a program can call the tool in process. agent-desktop skills get ffi covers entry points, memory, and threads, and you build it with cargo build --profile release-ffi -p agent-desktop-ffi. 1.1.0 is C ABI major 7. In it, ad_resolve_element_exact and ad_execute_ref_action_exact_with_policy always refuse with ACTION_NOT_SUPPORTED (details.kind is exact_entry_unsupported), because a C caller cannot supply the capture proof a ref now needs. Act through a snapshot ref instead.
What a Linux run of 1.0.2 returned
These commands ran on agent-desktop 1.0.2 on 1 November 2026, so the envelopes say "2.5".
The app was Mousepad, the XFCE text editor. A file /tmp/ad-test/note.txt already contained hello from agent-desktop and a newline. The launch asked Mousepad to open that file and wait for a window:
agent-desktop launch mousepad --activate --arg /tmp/ad-test/note.txt
The envelope said ok, pid 3317, window title /tmp/ad-test/note.txt - Mousepad, bounds 642 by 509, accessible, visible, not minimized. list-apps --app mousepad then showed it in the foreground.
snapshot --app mousepad -i --compact returned snapshot id sofkkd3q8w04y, ref_count 271, complete true. Most of those 271 refs were menu items, including a long filetype menu. The text field was one node:
{
"role": "textfield",
"name": null,
"ref_id": "@sofkkd3q8w04y:e269",
"available_actions": ["SetValue", "TypeText", "SetFocus"],
"states": ["focused"]
}
Reading it:
agent-desktop get @sofkkd3q8w04y:e269 --property value
data.value was "hello from agent-desktop\n".
A bare ref failed with exit code 1:
{
"version": "2.5",
"ok": false,
"command": "get",
"error": {
"code": "INVALID_ARGS",
"message": "Invalid ref_id '@e269': expected @<snapshot_id>:e<N>",
"disposition": { "delivery": "not_delivered", "retry": "safe" }
}
}
The envelope’s suggestion pointed at an example like @s8f3k2p9:e1.
Setting the same words back, without the trailing newline in the argument:
agent-desktop set-value @sofkkd3q8w04y:e269 'hello from agent-desktop'
The result was ok, with disposition.delivery at delivered_verified, retry at unsafe, and verification_scope at element_value. The step name was EditableText.SetTextContents, mechanism semantic_api. A following get showed the accessibility value without the newline.
The file was not saved, though. set-value changed the unsaved buffer, and the file on disk kept the original line, newline included, through the force-close later on.
find --app mousepad --role button --name New --exact found one button, ref @srhu9kybmxlyy:e1, state offscreen, on a new snapshot id. A skeleton snapshot of the same app returned 79 refs and complete: true.
screenshot --app mousepad /tmp/ad-test/mousepad.png wrote a PNG, 642 by 509, scale factor 1. It worked on this X11 session with screen_recording at not_required, as the Linux skill describes for X11 windows.
Closing without --force failed, because the dirty buffer had a Save Changes dialog open:
{
"ok": false,
"command": "close-app",
"error": {
"code": "ACTION_FAILED",
"message": "Process 3317 has 1 open dialog(s), so no close request was sent to them",
"disposition": { "delivery": "delivered_unverified", "retry": "unsafe" },
"details": { "cause": "dialog_pending", "dialog": "Save Changes" }
}
}
The message says no close request was sent, while the disposition says delivered_unverified. The suggestion mentioned a save prompt, or --force, which discards unsaved work.
close-app mousepad --force then succeeded, method force, delivered_verified, evidence process_instance_exited. list-apps --app mousepad returned an empty app list, and the file on disk still held the original line.
Before Mousepad, snapshot --app thunar failed with WINDOW_NOT_FOUND and the message Application 'thunar' is running but has no matching window. Thunar was in list-apps as a background app. list-windows with no app, and list-windows --app thunar, both returned "data": []. Launching with --activate is what produced a window the snapshot could see.
Can several agents use the desktop at once?
A session is a folder under ~/.agent-desktop/sessions/<id>/. session start prints a session_id, but later processes don’t attach to it by themselves. Pass --session or set AGENT_DESKTOP_SESSION, and the flag wins. Tracing is on unless you pass --no-trace. --screenshots stores unredacted PNGs, so keep those private.
The cursor overlay is presentation only. It does not move the OS pointer or change which actions are allowed. session start --cursor or cursor-overlay enable turns it on, and data.rendered: false means the setting was saved but nothing was drawn.
Since 1.1.0 the overlay shows the insertion caret after type and set-value, on macOS and Windows. The label is a black pill sized to its text, and the cursor body uses the accent colour unless you pass --fill to cursor-overlay enable. On Windows, the same command can draw --image and --pointer-image cursors.
Parallel cursors are session start --cursor --multi-agent. The harness assigns a stable --agent-id, or AGENT_DESKTOP_AGENT_ID. An id is 1 to 64 letters, digits, -, or _. Desktop UI actions then require an agent id, while observations, clipboard commands, and session admin don’t.
On the Linux run:
agent-desktop session start --name mousepad-test --cursor --multi-agent --no-trace
The envelope returned session id run-1793563261320-3497-0, trace off, cursor_overlay.enabled true, multi_agent true, and rendered false. The Linux skill says start returns rendered: false until an agent acts. On X11 each agent id then gets its own cursor, and on Wayland none is drawn. session end on that id returned the same session id and an ended_at timestamp.
Headless actions move their overlay cursor before they take the interaction lease, so several cursors travel at the same time while the actions themselves take turns. The session’s snapshots are shared. cursor-overlay disable and session end stop every cursor.
On macOS the overlay is click-through, and Reduce Motion skips travel and dwell. --travel-ms MIN,MAX needs both numbers at least 30, with MIN at or below MAX. Travel max plus --dwell-ms must be at most 700 ms, or the command returns INVALID_ARGS. On Windows, overlay travel is for headless semantic actions, and --headed moves the real cursor.
If you are splitting work across helpers, the subagents page is about context windows inside Claude Code, Codex, and Cursor. agent-desktop’s agent id is narrower, because it only names a cursor and a lease inside one desktop session.
Can you keep working while it runs?
The background behavior Tate described is the headless default. Semantic ref actions don’t steal focus or move the OS pointer, so you can keep using the machine while they run.
With --headed, physical input takes the pointer and the interaction lease serializes it. On Windows, headed type uses SendInput into the foreground queue. There’s no separate “background mode” command beyond this policy.
Don’t mix this up with Codex Computer Use on Windows, which uses foreground input of its own, as the Codex guide explains.
Two GTK clicks on one button less than about 250 ms apart act once on Linux, and both can still answer ok with delivered_unverified, so wait for the effect between clicks. A terminal text node does not accept headless type (POLICY_DENIED). A GTK4 secret field can show text, and only a password role is withheld.
How I would use agent-desktop
I build Mac apps, and I let agents test them in a VM so the clicks stay off the machine I am typing on. That setup is how to let coding agents test Mac apps in a VM. The local builds are listed in App Drawer, a Dock panel for the Mac apps in your project folders.
I would install agent-desktop inside that VM, grant Accessibility to the terminal that launches it, and put the skill paragraph in the project’s AGENTS.md. The task would be an app that has no CLI for the flow I care about: open a window, snapshot, act on a ref, snapshot again, and stop. I would rather have the agent read a button’s name from the tree than guess a coordinate from a screenshot.
I would keep it off my everyday desktop. The CLI can click, type, read window text, and close apps, and a VM with test data is the place for that. I would not point it at a logged-in mail client or a password manager on the machine I use all day.
These are the jobs I would give it:
- Launch an app, wait until it has a window, snapshot, and read a field, which is the shape of the Mousepad run above.
- Set a text field with
set-valuewhen I don’t want keystrokes or focus, then check the file separately, since the control and the file on disk can differ. - Find a control by role and name when the full tree is hundreds of refs.
- Give two subagents two agent ids and two overlay cursors, expecting their physical input to take turns.
- Screenshot one window when I want an image for myself. The agent doesn’t need it to click.
A browser flow still belongs in a browser tool, and Playwright is the one I would reach for in a test suite. agent-desktop covers the native apps next to it.
What will it refuse to do?
Protected processes cannot be closed. The match is exact, not a substring, so a lookalike name stays closable. The code is INVALID_ARGS with not_delivered, not PERM_DENIED. CONCEPTS.md defines a protected process as a session-critical OS process, one whose exit would take down the desktop session.
On macOS, quit shortcuts return POLICY_DENIED unless you pass --force: cmd+q, cmd+shift+q, cmd+alt+esc, ctrl+cmd+q, and cmd+shift+delete. Use close-app to quit.
Retry only when error.disposition.retry is safe. The contract says only not_delivered is safe to retry. delivered_unverified means something may have landed, and sending it again can double the action. recovery is present only when retry is safe.
How do you read an error?
The JSON doc defines the exit codes. 0 is success, 1 is a structured error on stdout, and 2 is a parse error that still prints an envelope with INVALID_ARGS.
error.code, error.message, and error.disposition are always there. suggestion, recovery, platform_detail, and details are omitted when empty.
| Code | Meaning |
|---|---|
PERM_DENIED | Accessibility permission not granted |
ELEMENT_NOT_FOUND | No element matched the ref or query |
APP_NOT_FOUND | App not running, or no windows. Nothing was sent |
STALE_REF | The element is gone. On macOS, its recorded address no longer matches |
AMBIGUOUS_TARGET | More than one window, or a ref that was not proven unique (identity_not_unique) |
SNAPSHOT_NOT_FOUND | Snapshot id missing or expired |
POLICY_DENIED | Physical or headed path blocked by policy |
ACTION_FAILED | Rejected, or the observed result contradicted the request |
ACTION_NOT_SUPPORTED | The target does not expose that action |
APP_UNRESPONSIVE | The matching app stopped responding |
PLATFORM_NOT_SUPPORTED | This OS does not implement the method |
TIMEOUT | The wait expired |
INVALID_ARGS | Bad arguments, a bare ref, or a surface kind this OS does not have |
WINDOW_NOT_FOUND | No such window |
NOTIFICATION_NOT_FOUND | Notification id not found |
INTERNAL | Unexpected error, including a state root that is not private |
delivery is one of unknown, not_delivered, delivery_uncertain, delivered_unverified, or delivered_verified. Treat unknown as unsafe to retry automatically.
When a snapshot runs out of budget, envelope 2.2 returns ok: true with complete: false and the tree it did observe, instead of a TIMEOUT and no tree. Read data.complete. A --root drill-down is different, and an incomplete observation there still returns TIMEOUT.
With identity_not_unique, details.retryable is false. Nothing was sent, so disposition.retry is safe, but repeating the same ref gets the same refusal. Take a new snapshot. An identical twin that shows up after capture uses details.kind saved_ref_ambiguous instead.
Is this safe to point at your real desktop?
The native CLI does not call a remote model, but what it reads can still be private. Snapshots, clipboard reads, screenshots, command output, debug HTML, and session traces can contain whatever is on the screen. The FAQ says to keep those out of public logs. Redaction in trace lines does not make a screenshot safe to publish, and --screenshots stores unredacted PNGs.
The optional JEV helper, in skills/jev-desktop/SKILL.md, is separate from the native CLI and sends goals and screen descriptions to a configured provider. Its value suppression does not remove private text from labels, titles, ancestor paths, or the goal.
agent-desktop’s own docs do not mention prompt injection from on-screen content, yet a window title or a button label is text your agent will read. Prompt injection is the general problem: untrusted text that tries to steer the agent. Anthropic’s computer use security notes say web pages and images can carry that text. They recommend a dedicated VM, minimal privileges, no sensitive logins, a domain allowlist, and a person confirming actions that have real consequences. The advice is written for their tool, and it fits any agent that can click.
agent-desktop does not document a sandbox of its own. A coding agent’s sandbox limits files and hosts for shell commands, but a desktop click can sit outside that boundary, because it talks to other apps through the accessibility API. If the task can spend money, send a message, or delete data, run it where the blast radius is a VM and a test account.
SECURITY.md is the vulnerability report path (GitHub advisories) for the CLI, the npm package, the FFI library, releases, Actions, and adapters. It is not a threat model of desktop automation.
How does it compare with other computer-use tools?
These are different products, and the last column is where they differ most.
| Tool | What it drives | How it points | Who performs the action |
|---|---|---|---|
| agent-desktop 1.2.0 | Native apps | Accessibility refs | The CLI, or its MCP server |
| agent-browser 0.41.1 | A browser | Page refs | The CLI |
| Anthropic computer use | A screen you attach | Screenshot, then coordinates. Toolset computer_toolset_20260801 | Your application runs every call |
| OpenAI computer use | A browser or desktop interface | Screenshots and mouse or keyboard via ComputerToolset | Your app supplies the driver |
| Playwright | A browser | Selectors in a test | Your test code |
| Gemini Computer Use | Browser, Android, or a desktop environment | Screenshot in, click or type coordinates out | Your code. The guide on this site uses Playwright for the browser case |
| Cursor Computer Use | Desktop apps and the iOS Simulator, inside Cursor | Cursor’s own flow | Cursor’s agent |
| Codex Computer Use | macOS and Windows apps, in the Codex app | The Computer Use plugin | Codex |
| terminal-mac | A macOS guest in the terminal | Screenshot, click, type, or a key | The agent, via MCP or the CLI |
Anthropic’s current toolset has member tools such as screenshot, left_click, type, and zoom. The docs say it is not available in Claude Managed Agents, and they point webpage-only tasks at a browser tool instead. Their reference environment includes Xvfb. Use their page for the sample code.
OpenAI’s guide says computer use lets a model operate a browser or desktop interface. Page text and controls go through BrowserToolset. A browser hosted by OpenAI is covered in a separate Agents API guide, which lists hosted partners including Browserbase, Browser Use, Daytona, and E2B.
Playwright remains the tool for a repeatable browser test. Gemini’s computer-use model is a coordinate loop on top of a screen. agent-desktop skips coordinates when the OS can name the control.
What do you do when a command fails?
Start with agent-desktop status and agent-desktop permissions. On Linux, empty surfaces usually mean the bus, not an empty desktop, so put the CLI in the graphical session’s environment.
WINDOW_NOT_FOUND while list-apps shows the process means the app has no window yet. launch --activate waits for one, which is what fixed the Thunar miss in the Linux run.
INVALID_ARGS on a ref usually means the ref is bare, or it came from a different snapshot or session. Use the qualified ref from the latest snapshot in the same session. On --surface, it means this OS does not have that kind, and the error names the ones it has.
STALE_REF means the element is gone, which on macOS includes a row whose address is dead. Snapshot again rather than reusing the old id.
AMBIGUOUS_TARGET on a window means pass --window-id. With identity_not_unique, snapshot again, because --window-id won’t help there.
PLATFORM_NOT_SUPPORTED on permissions --guide is expected on Windows and Linux. Grant permissions in the OS settings instead.
ACTION_FAILED with a dialog pending means something is asking to save. Decide whether to save, or pass --force and accept the discarded buffer.
POLICY_DENIED on type often means the action wants --headed, or the control has no headless text path. set-value is the headless write on Windows, while a Linux terminal text node refuses headless type.
PERM_DENIED on macOS is the Accessibility grant, for the app that launched the CLI. On Windows it can be an integrity mismatch: you can read an elevated app, and you cannot send it input unless the agent is elevated too.
INTERNAL right after install, on Linux, can be a state root that other users can write. Fix it with chmod 700 ~/.agent-desktop.
If npm feels slow, check whether install scripts were blocked. The Node launcher is the 146 ms path and the native link is the 21 ms one. command -v agent-desktop tells you which binary you are actually running.
FAQ
Does agent-desktop need screenshots?
No, the core loop is the accessibility tree and refs. The FAQ says agents can click, type, inspect menus, and move through windows without screenshots or pixel matching.
Is the cursor overlay the real mouse?
No, it’s a drawing on top of the session. The OS pointer stays put during headless actions, and only headed actions move it.
Will set-value save my document?
It writes the control. In the Mousepad run it updated the accessibility value and left the file on disk unchanged. Saving is a separate app command, often a button or a shortcut, and some shortcuts are blocked unless you pass --force.
Where are the docs?
The repo README, docs/install.md, docs/json-output.md, docs/faq.md, CONCEPTS.md, and the skills printed by the binary. The npm package is agent-desktop. agent-browser is vercel-labs/agent-browser.
Want me to talk about your product? You can sponsor this site.
Related posts about ai: