A Pointman extension: agent-desktop on each machine, so agents can read its apps through the OS accessibility tree and act on them: click, type, set values, press keys, scroll, launch apps. An agent anywhere sends a command to any machine with create_job(kind="desktop.run", machine=…).

It pairs with the screen extension. When a person wants to watch, the agent opens screen's live pane on their device first. On macOS, every agent then gets its own cursor, labelled with its handle and what it's doing, where it acts.

One Rust binary, bin/desktop, built for each platform with the pointman-extension crate.

The extension doesn't use npm or Homebrew. It installs one pinned agent-desktop release into its state folder (agent-desktop/<version>/): the executable with its platform helper beside it, from agent-desktop-v<version>-<target>.tar.gz on agent-desktop's GitHub releases. Before unpacking, it checks the archive against a sha256 that src/install.rs carries.

• Where the hashes come from: the release's own checksums.txt. Each archive's build provenance was checked first with gh attestation verify <archive> --repo vercel-labs/agent-desktop.
• A new version is one change to src/install.rs: VERSION and all four hashes.
• On startup: the extension installs it in the background, so the first call doesn't wait.

On Linux, the node may run as a systemd user unit. A call there gets the session's variables it lacks from /run/user/<uid>: the session bus (agent-desktop finds the AT-SPI bus through it), the Wayland socket, Hyprland's instance and X11's display. The call also gets NixOS's system and per-user bin folders on its PATH (for wl-copy), and QT_ACCESSIBILITY=1. NO_AT_BRIDGE and GTK_A11Y are removed, so apps it launches join the bus. Under Hyprland, these work: listing apps and windows, snapshots, click, type, set-value, and the clipboard. press, focus-window and pointer input need GNOME's Mutter on Wayland. Screenshots ask for consent on screen, so screen.still is the way there.

• Errors are answers. agent-desktop's error (STALE_REF, APP_NOT_FOUND…) comes back as the result, for the agent to branch on. The job only fails when agent-desktop can't run: not installed, crashed, timed out, or physical input asked for where it's off.
• Screenshots are posted, not answered. A screenshot without a path comes back as a PNG output, and the answer names it instead of carrying base64. A file it writes where it ran (screenshot … out.png) is posted too.
• A huge answer goes back as a file. Over 512 KB, the full JSON is posted as a document.
• Deadlines: each call has 60 s unless timeout says otherwise (at most 600). It can be cancelled, which kills the CLI. Output still arriving 0.5 s after the CLI exits is dropped: a process it left behind holding the pipe (wl-copy, owning a clipboard it set) isn't waited for.

Every call shares one agent-desktop session on the machine, started with --cursor --multi-agent --no-trace. Each call runs with --agent-id <agent>, so each agent gets a cursor of its own, labelled @<agent>: <doing>. The session lasts a day, then the next call starts a new one and clears the old one's files. If the session ends under a call (session_ended), the extension starts a new one and runs the call once more. A call that names its own --session or --agent-id is left as it is. With the cursor setting off, calls run without a session.

agent-desktop keeps its state (snapshots' refs, sessions) in the extension's state folder (home/, mode 0700, as it requires). The node's own AGENT_DESKTOP_* variables never reach it.

cargo test. The tests run the extension under the SDK's stand-in node with a fake agent-desktop, a shell script (tests/common) that logs each call and answers as the real one does. No test reaches GitHub or a real desktop, except cargo test -- --ignored, which downloads the pinned release and runs its version. CI runs cargo fmt, clippy and the tests on macOS and Ubuntu.

A release is a tag that's extension.toml's version: .github/workflows/release.yml builds the four platforms and attaches them.