DSH Computer Use

Computer Use for DeepSeek Harness — let the Harness drive real macOS apps.

Background AX actions (no cursor theft), a Claude-style floating status capsule with a Stop button, a Codex-style inline window preview inside the conversation, and a split-screen session layout that puts the target app on the left and DSH on the right.

📺 Demo — zcode-install-github-plugin.mp4 (8.4 s, 1088×998, silent) Recorded end-to-end by this plugin itself: it opens ZCode's plugin marketplace, searches for the GitHub plugin, clicks Install, and the plugin lands on disk. Every frame is a real capture — the 4 step callouts were added in post, the UI itself is untouched.

---

Why another desktop agent

Most "computer use" loops feel slow because every single keystroke costs a model round trip. This plugin is built around three ideas:

1. Local batch execution. cu_batch runs a whole click → type → key → wait sequence locally in ~300 ms instead of one model turn per action.
2. Observe by delta, not by screenshot. Every mutating call returns a compact +added / -removed / ~changed AX tree diff, so the model reads a few lines instead of a megabyte of PNG.
3. Never steal the pointer. Clicks, typing and scrolling go through AXPress or PID-targeted events. Your mouse stays where you left it and the target app does not need to be frontmost.

Features

Background-first	Click / type / key / scroll without moving the user's cursor or stealing focus
Floating capsule + halo	Non-activating top-right status pill (cu capsule) and target-window border (cu halo) with an immediate Stop button (~/.cu/pause + plugin latch)
Inline preview	Target-window screenshots are attached inline in the conversation via DSH attachments.saveImage
Split-screen session	cu_session(start) docks DSH to the right, lays the target window out on the left, and restores both on stop — the Claude prepareDisplay equivalent, and it stops Stage Manager thumbnail churn
AX delta observation	Mutating tools return the tree diff in the same call
Hybrid targeting	AX semantic search first, Apple Vision OCR fallback for canvas-heavy apps
Unicode safe	Chinese / Japanese text entry verified with read-back

Tools

Tool	Description
cu_status	Service health, Accessibility & Screen Recording grants, pause latch, active session
cu_session	start / preview / stop / resume the visual session (capsule, halo, side-docking, split screen, inline preview)
cu_windows	List on-screen windows and running apps
cu_launch	Launch by bundle id or .app path; bring an off-stage window onto the current stage
cu_snapshot	AX tree with stable e1, e2, … refs and delta diffing
cu_screenshot	Window or screen capture, attached inline
cu_click	Background click by ref or x, y, with post-click AX delta
cu_type	Background text entry (type, setvalue, action) with read-back verification
cu_key	Background key or shortcut (return, cmd+shift+g, …)
cu_scroll	Background vertical / horizontal scroll
cu_find	Hybrid search: AX first, Apple Vision OCR fallback
cu_batch	Run click / type / setvalue / key / scroll / wait / action steps locally in one call

Requirements

• macOS 14+
• DeepSeek Harness with the plugin loader (@deepseek-ai/dsh-tools, @deepseek-ai/schemastery)
• The cu native engine — this plugin is the client; the engine holds the TCC grants and posts the events. Build and install it separately (see Engine)
• Accessibility and Screen Recording granted to the engine app

Check grants at any time:

curl -s http://127.0.0.1:47777/v1/status | jq '{accessibility, screenRecording, paused, windows}'

Install

1. Build and install the engine, then confirm the service is up on http://127.0.0.1:47777.
2. Install the plugin into a DSH profile:

cd ~/.dsh/profiles/<your-profile>
pnpm add link:/path/to/dsh-computer-use

1. Add it to the profile bundle list (package.json → dsh.profile.bundles), or let the bundled cordis.patch.yml register it.
2. Restart DSH. The 12 cu_* tools appear, and the skill shows up in the / menu as /computer-use, /dsh-computer-use and /cu.

Engine

cu is a separate MIT-licensed native engine (Swift; public Apple frameworks only — Accessibility, CoreGraphics events, ScreenCaptureKit) shipping a cu CLI plus a launchd service. The plugin resolves the binary in this order:

1. CU_BIN environment variable
2. ~/Library/Application Support/cu/bin/cu
3. /Applications/Computer Use.app/Contents/MacOS/Computer Use
4. ../../../bin/cu relative to the plugin

If none is present the plugin still loads — every tool returns a structured service_unreachable error with next steps instead of crashing.

Configuration

Option	Default	Description
serviceUrl	http://127.0.0.1:47777	Local engine URL
autoSession	true	Show the floating capsule automatically when acting on a window
defaultPreviewMaxDim	960	Max pixel dimension for inline previews

Safety

• Stop is always one click away. Pressing Stop touches ~/.cu/pause; the plugin latches on it and refuses further mutating calls until cu_session(resume).
• Actions are latched, not retried. ok: true is not proof the UI changed — read the returned delta.
• No credentials, no telemetry. The plugin only talks to 127.0.0.1.

Development

node --test test/core.test.mjs

The suite covers ref caching and deltas, argv construction, the pause latch, media sniffing, one live round trip against an isolated cu demo window (background click + delta + preview), and a cu_batch smoke test.

License

MIT — see LICENSE.