@wangyong1972/dsh-computer-use-macos

English | 中文

A DeepSeek Harness (DSH) plugin that gives models a native macOS implementation of Anthropic's "Computer Use" capability: one computer tool that can screenshot the real display and drive the real mouse/keyboard, using only binaries that ship with macOS (screencapture, sips, osascript) — zero installed dependencies, zero network calls, zero compiled helper binaries.

What it does

The plugin registers a single tool, computer, that lets the model:

• take a screenshot of the real screen and see it inline in the conversation;
• move the mouse, left-/right-/double-click at a pixel coordinate;
• type text and send key combos (Return, cmd+c, ctrl+shift+t, ...);
• scroll up/down/left/right;
• read the current cursor position (best-effort);
• wait for a short, bounded duration.

Every OS interaction goes through execFile('screencapture', [...]), execFile('sips', [...]), or execFile('osascript', ['-l', 'JavaScript', ...]) with argv arrays — never a shell string built from model input. See "Security posture" below.

Screenshots are automatically downscaled before being sent to the model: any capture whose longest edge exceeds 1568px is resized in place (via the macOS-native sips -Z, preserving aspect ratio) so it stays comfortably under common vision-model per-image pixel limits — this is what makes screenshot work at all on a Retina or 6K display, whose raw capture would otherwise be rejected outright by many hosts with "Image exceeds the configured per-side pixel limit." No setting changes this target size. If on-screen text or controls become too small to read reliably in the downscaled image, increase the size of that text/UI in the app being controlled (zoom in, increase font size, etc.) rather than trying to change the display's resolution or this plugin's downscale target — matching Claude Code's own documented behavior for its native macOS computer-use tool. Every coordinate the model reads or supplies is always in this downscaled screenshot's own pixel space, never the display's raw native pixel space — the tool converts internally.

Claude-Code-parity UX features (all individually toggleable, see the config table below): a machine-wide lock so a second DSH process on the same Mac can't drive the mouse/keyboard concurrently; optionally hiding other visible apps while acting; optionally excluding the host/terminal from the screenshot it takes of itself; macOS notifications when the tool starts and stops acting; and extra warning text in the approval prompt when the frontmost app is a terminal/IDE, Finder, or a system-settings app. A global Esc-to-abort hotkey was investigated and found infeasible under this plugin's architecture — see DESIGN.md §10.5 for the full analysis.

Requirements

• macOS only ("os": ["darwin"] — installation fails loudly on any other platform).
• A DSH host with the tools and attachments services mounted (both are standard in a DSH web/desktop profile).
• A vision-capable model route for the screenshot action specifically (a text-only route gets a clear error instead of a wasted screenshot).

One-time macOS permission setup

macOS requires two manual, one-time permission grants before this plugin can do anything. This plugin cannot grant these to itself — macOS deliberately makes TCC (privacy) grants un-scriptable from the requesting process. This is the same kind of one-time dance you already went through for apps needing Full Disk Access or Screen Recording.

1. Accessibility — required for every mouse/keyboard action (mouse_move, left_click, right_click, double_click, type, key, scroll, and best-effort for cursor_position). Open System Settings → Privacy & Security → Accessibility and enable the relevant app.
2. Screen Recording — required for screenshot to return real pixels instead of a blank image (recent macOS versions silently return a blank capture, with no error, when this permission is missing). Open System Settings → Privacy & Security → Screen Recording and enable the relevant app.

Which app shows up in those lists depends on how DSH was launched. Every mouse/keyboard/screenshot action in this plugin runs through the osascript binary (or screencapture for screenshots), so TCC attributes the permission to whichever process actually executed it — in practice, this is either osascript itself, or your terminal/host app in the responsible- process chain, depending on macOS version and how DSH was started. Trigger one computer action first (it will fail with a clear message naming the missing permission), then check System Settings — the correct entry only appears in that list after the first attempt. Look for osascript, Terminal, or your terminal app.

Once both permissions are granted, retry the failed action.

Installing into a DSH profile

npm (recommended):

dsh plugin --profile <name> add @wangyong1972/dsh-computer-use-macos

Published package: @wangyong1972/dsh-computer-use-macos

Local checkout (development):

dsh plugin --profile <name> add /path/to/dsh-computer-use-macos

This plugin ships its own cordis.patch.yml (referenced from package.json's dsh.bundle.patch field), so adding it registers the computer-use-macos plugin with sensible defaults automatically.

Config reference

field	default	description
enabled	true	Master switch. When false, the computer tool is not registered at all — zero prompt-token cost.
requireConfirmation	true	Route every mutating action (click/move/type/key/scroll) through the approval seam before executing. screenshot/cursor_position/wait are never gated. Fails closed (denies) when no approval service is mounted.
allowedDisplayIndex	0	Backward-compatible default 0-based display index used whenever an action omits display. Per-action selection can override it without changing config.
screenshotFormat	'png'	Reserved for future formats; only png is supported in v1.
actionTimeoutMs	10000	Hard timeout for every screencapture/sips/osascript/ps subprocess call.
maxTypeTextLength	4096	Reject action=type calls whose text exceeds this many UTF-16 code units (rejected, never truncated).
enableMachineLock	true	Hold a machine-wide lockfile while a mutating action runs, so a second DSH process on this Mac can't drive the real mouse/keyboard concurrently. A stale lock (dead holder, or held too long) is reclaimed automatically.
hideOtherAppsWhileActing	false	Hide every other visible app while a mutating action runs, keeping only this plugin's own host process visible, then restore exactly what was hidden. Off by default — visibly intrusive, opt in deliberately.
excludeHostFromScreenshots	false	Momentarily hide the host process for just the screencapture call so screenshot never shows the terminal instead of the real target app. Off by default for the same reason as above.
enableSessionNotifications	true	Post a macOS notification when the tool starts/stops acting. Best-effort; never fails the underlying action.
enableAppRiskWarnings	true	Add extra warning text to the approval prompt when the frontmost app is a terminal/IDE, Finder, or a system-settings app. Purely additive — never changes what gets approved, no effect when requireConfirmation is false.

Click diagnostics and logging

Every mouse_move/left_click/right_click/double_click call emits one structured, privacy-safe diagnostic record via DSH's own ctx.logger() facade (never a bespoke log file, never console.*) — named computer-use-macos, so a host's own log exporter can filter on it. Example line (real format, live-verified against a real @deepseek-ai/cordis Context):

computer.left_click display=0 pixel=[500,400] point=(500,400) pid=69234 exit=0 elapsed=112ms outcome=ok cursorVerified=true

Fields: the action, allowedDisplayIndex, the requested pixel coordinate, the resolved Quartz point, the osascript subprocess's pid/exit code, elapsed time, a coarse outcome classification (ok / accessibility-denied / timeout / aborted / other-error), and whether a post-action cursor-position self-check confirmed the cursor actually arrived at the requested point. Never logged: raw stderr/stdout text, typed text, or key-combo content — see DESIGN.md §11.2 for the full privacy contract. warn-level for anything other than a clean, verified outcome; info-level otherwise.

What this can and cannot prove: the cursor self-check confirms a click's underlying CGEventPost call reached the OS and the cursor warped to the right screen point — it cannot confirm the target application actually received or reacted to the click as a click (that would need per-app accessibility-tree introspection, out of scope; see DESIGN.md §11.3). Take a follow-up screenshot to confirm a click's actual on-screen effect.

Supported actions (v1)

list_displays, screenshot, left_click, right_click, double_click, mouse_move, type, key, scroll, wait, cursor_position (best-effort).

Selecting a display

Call list_displays to obtain each active display's 0-based index, numeric CGDirectDisplayID (id), main-display flag, Quartz origin/size, backing scale, and explicit 1-based screenshotOrdinal. screenshot, cursor_position, mouse move/click actions, and scroll accept an optional display field:

• positional strings: main, leftmost, rightmost, topmost, bottommost;
• a JSON number for a 0-based active-display index;
• a decimal string for a CGDirectDisplayID, optionally prefixed with id:.

If display is omitted, the existing allowedDisplayIndex behavior is preserved. The inventory is refreshed for every call, invalid/out-of-range selectors fail before any OS input event, and coordinates remain local to the selected display's screenshot space. main must resolve to exactly one display; a positional extreme shared by two displays is ambiguous and fails closed. For scroll without coordinate, the display selector is accepted but does not move the pointer; scrolling occurs at the current cursor position as before.

Screenshot identity limitation: macOS screencapture -D accepts only a 1-based ordinal, not a CGDirectDisplayID. screenshotOrdinal is the best-known mapping derived from the same NSScreen inventory and is passed explicitly; it is not an identity-safe guarantee that Apple will keep NSScreen and screencapture ordering identical. Mouse/cursor geometry is ID-backed through CGDisplayBounds, but callers must not treat CG-ID-selected screenshots as absolutely identity-safe until the backend is replaced with direct CoreGraphics capture.

Not yet supported (deferred to v2): left_click_drag, middle_click, triple_click, hold_key, zoom, compensating for the macOS "natural scrolling" preference, and per-window/per-app (accessibility-tree) targeting. Please don't file bugs for these — they're known, intentional v1 scope cuts, not oversights.

Security posture

• No shell-string execution anywhere. Every OS command is invoked via Node's execFile with an argv array, and every mouse/keyboard action's untrusted input (text, coordinate) crosses the JXA subprocess boundary as one JSON string in argv, never as interpolated AppleScript or shell source text.
• No network calls of any kind, anywhere in this plugin.
• All numeric input is validated and clamped against the real, freshly-queried screen bounds before any mouse/keyboard event is synthesized — never trusted from the model, even if the model is only echoing a coordinate it was previously shown.
• No arbitrary file path acceptance. The only filesystem paths this plugin ever touches are its own randomly-named temp screenshot file (created, optionally downscaled in place via sips, and deleted within the same tool call) and a single fixed-path machine-wide lockfile (§10.1).
• The machine-wide lock fails closed on contention — a second process holding it is a hard denial, never a silent proceed; stale-lock reclaim is bounded by both PID-liveness and an absolute age ceiling.
• App-hiding always restores exactly what it itself hid, never a blanket "show everything," so an app the user had already hidden stays hidden.
• Per-app risk warnings are purely additive to the existing approval prompt — they can never turn an approval "ask" into an automatic "allow," and the risk table is fixed, not model- or config-editable.
• Every mutating action is gate-able via requireConfirmation, which fails closed (denies) when no approval service is mounted — never fails open.
• enabled: false fully disables the plugin: no tool registration, zero token cost.
• No postinstall/preinstall scripts, ever.
• peerDependencies, not dependencies, for every @deepseek-ai/* package this plugin composes against.

Development

pnpm install
pnpm run build      # tsc -> lib/, then copy the JXA helper verbatim
pnpm run typecheck   # tsc --noEmit
pnpm run verify      # static checks, no macOS side effects, safe for CI
pnpm test            # unit tests over lib/ (key-spec, validate, permission-errors, screenshot-scale, lock, app-tiers, notify)

Everything requiring real Accessibility/Screen Recording permissions is covered by the manual checklist in tests/e2e.manual.md.

License

MIT