Computer use · Chanto docs

The Computer tool lets the model work in any desktop app, not just web pages: it takes screenshots, clicks, drags, scrolls, types, presses keys and opens or focuses apps. It's off by default. Turn it on for one session with chanto --computer, or always with:

toml

[computer]
enabled = true
# background = false   # also offer it to background runs (off: runs never get it)
# max_width = 1280     # screenshot size the model sees (aspect kept)
# max_height = 800

Actions. screenshot; left_click, right_click, middle_click, double_click, triple_click and mouse_move at coordinate: [x, y]; left_click_drag from start_coordinate to coordinate; scroll (scroll_direction, scroll_amount wheel clicks, optional coordinate); type (text); key (text: Return, Escape, cmd+shift+t, ctrl+a, F5, …); open_app and focus_app (app); list_windows; cursor_position; wait. Coordinates are pixels in the latest screenshot. Screenshots of the main display are scaled to fit 1280x800, and Chanto maps them to screen points. On a Retina display the raw capture has twice as many pixels as points, and the mapping accounts for that. Screenshots need a vision model.

Platforms. On macOS it uses only built-in tools: screencapture and sips for screenshots, and osascript running JavaScript for Automation for input. Its ObjC bridge posts CoreGraphics mouse and scroll events, and System Events types text and presses keys. That means no compiler (a Swift helper would need the Xcode tools), no download (cliclick) and no new dependencies. Each action is one short-lived process. Linux works on X11 with xdotool and ImageMagick's import installed (open_app goes through gtk-launch).

On Wayland, which keeps apps from reading the screen or faking input, it uses the tools each desktop allows:

Screenshots: grim on wlroots compositors (Sway, Hyprland, river), gnome-screenshot on GNOME, spectacle on KDE; the last two need ImageMagick to resize.
Mouse and keyboard: ydotool, which works on any compositor through /dev/uinput; its daemon ydotoold must be running. ydotool moves the pointer with relative motion, so turn off pointer acceleration for clicks to land where intended. On scaled (HiDPI) displays, clicks can be off by the scale factor.
Windows: list_windows and focus_app work on Sway and Hyprland. Other compositors don't let apps list or focus windows, and the cursor position can't be read on any; the tool says so.

Windows isn't supported yet.

macOS permissions. The app Chanto runs in (Terminal, iTerm, your editor) needs Screen & System Audio Recording to take screenshots and Accessibility to click and type. Typing and keys also need Automation permission to control System Events, which macOS asks for the first time. When a permission is missing, the tool returns an error that names the setting and the app. You turn it on in System Settings → Privacy & Security, then reopen the app.

Safety.

Every action asks for approval in the default and acceptEdits modes, screenshots included, since they send your screen to the model provider. "Yes" applies to that action only, and allow rules don't skip the prompt.
In auto mode, screenshots and other look-only actions go ahead, and everything else goes to the auto-mode classifier. Plan mode only allows looking, and bypass mode allows everything.
Background runs never get the tool unless [computer] background = true. Then each action becomes a needs-input prompt in acceptEdits.
While computer use is on, the TUI status line shows ◉ computer use on (active right after an action), and the session starts with a warning.
[computer] is ignored in projects you haven't trusted.

Risks. Computer use acts with your login: the model can click, type and see anything on your screen, in any app and any signed-in account. Anything on screen can contain text written to hijack the model: a web page, an email, a document or a notification. This is prompt injection, and it can make the model try things you didn't ask for, like sending messages, changing settings or copying data somewhere. The approval prompt is your defense, so read each action before you allow it. Close windows with sensitive content first, and don't use auto or bypass mode with computer use unless the machine is a disposable VM. The tool tells the model to treat screen content as untrusted, but that doesn't make injection impossible.