DScomputer-control

Windows desktop automation for DeepSeek Harness — the agent gets eyes and hands on the local GUI.

14 tools · pure JS, zero dependencies · brings real keyboard/mouse input, occlusion-proof window screenshots and the accessibility tree into DSH.

What it does

• Enumerate apps and windows (cua_list_apps, cua_list_windows) → {app, id, title}.
• Look — cua_window_state returns a window screenshot captured with Windows.Graphics.Capture (works even when the window is covered by others) plus the accessibility (UIA) tree, which carries element_index values for precise targeting.
• Act — real input injection, not UI-Automation value writes: cua_click (single/double, by element or pixel), cua_type_text (CJK included), cua_press_key (Control_L+a, Return, …), cua_scroll, cua_drag (text selection, sliders, files), cua_set_value, cua_secondary_action.
• Launch / focus — cua_activate_window, cua_launch_app.
• Clean exit — cua_release shuts the engine down so its on-screen cursor overlay is torn down and the real pointer returns; an idle timer (releaseAfterIdleMs, default 60 s) does the same automatically if a session is left open.

Requirements

OS	Windows 10 / 11 x64
Host	DeepSeek Harness Desktop 0.1.x
Runtime	The official computer-use runtime (@oai/sky) already installed on the machine

This package does not ship the engine binary — the runtime comes from the vendor application that provides it. The plugin finds it by enumerating the vendor directories under %LOCALAPPDATA%\OpenAI\ (highest @oai/sky version, largest CLI executable), so vendor upgrades that change hash directories keep working without touching the plugin.

No admin rights required.

Install

From the plugin market, or directly:

dsh plugin add https://github.com/MuzeWinter/dscomputer-control

Then fully quit and restart DeepSeek Harness — bundle modules are cached per process, so a restart is what loads the plugin. Changes to the config block below are hot-reloaded in a few seconds and do not need a restart.

Manual installation: copy lib/ and cordis.patch.yml into <profile>/_plugins/dscomputer-control, expose it in <profile>/node_modules/, add "dscomputer-control" to dsh.profile.bundles in <profile>/package.json, and append:

- id: dscomputer-control
  name: "dscomputer-control"
  config:
    approval: auto
    releaseAfterAction: false
    releaseAfterIdleMs: 60000
    imageMode: auto
    maxTextChars: 20000
    timeoutMs: 45000

The bundled skill (skill/SKILL.md) can be copied to %USERPROFILE%\.dsh\skills\dscomputer-control\SKILL.md; skills are picked up without a restart and make the model reach for this toolset on GUI tasks.

Configuration

key	default	meaning
approval	auto	auto runs unattended; allowlist allows only approvedApps matches; deny rejects every action deterministically
approvedApps	[]	substrings matched against app title / process name (only with approval: allowlist)
imageMode	auto	auto attaches the screenshot when the active model accepts images; text returns only the tree
releaseAfterAction	false	true closes the engine after every action (safest for the pointer, but the overlay only flashes)
releaseAfterIdleMs	60000	auto-release after this much idle time; 0 disables it
maxTextChars	20000	truncation length for the accessibility tree
timeoutMs	45000	per-call timeout (registered with DSH as timeoutMs + 15000)
scrollStep	3	step size for direction-based scrolling

Tools (14)

cua_status · cua_list_apps · cua_list_windows · cua_window_state · cua_click · cua_type_text · cua_press_key · cua_scroll · cua_drag · cua_set_value · cua_secondary_action · cua_activate_window · cua_launch_app · cua_release

Usage discipline that saves turns: enumerate first ({app, id} are both required), activate before typing, prefer element_index over pixels, verify after every action, and release when done.

Behaviour and caveats

• Real input injection steals foreground focus. It activates the target window; it is not suited to silent background work. For value-setting without focus changes, the separate dsh-click plugin is a better fit — the two are complementary and can be installed side by side.
• WinUI / AUMID windows (Windows 11 Notepad, Windows Terminal) and elevated windows are not supported by the engine (foreground window did not report a process id, or UIPI blocks the injection). Classic Win32 targets work.
• Coordinates originate at the client area's top-left, excluding the title bar.
• cua_launch_app takes an app id from cua_list_apps (or a launchable executable), not a display name.
• The engine's cursor overlay is drawn while controlling; cua_release / idle auto-release removes it.

中文要点

• 给 DeepSeek Harness 加"看屏幕 + 动手"能力：14 个工具，纯 JS、零依赖。
• 枚举应用/窗口 → 窗口截图（被遮挡也能截）+ 无障碍树 → 真实输入（点击、输入中文、组合键、滚动、拖拽）。
• 需要机器上已装官方电脑控制运行时（@oai/sky），本包不含引擎二进制；插件按厂商目录枚举发现运行时，引擎升级换目录不影响。
• 安装后必须完全重启 DSH；改 config 会热重载、不用重启。
• 已知边界：WinUI/AUMID 窗口与提权窗口不支持；真实输入会抢前台焦点；坐标以客户区左上角为原点。
• 收尾用 cua_release（或等 releaseAfterIdleMs 自动释放），否则引擎的光标覆盖层会一直盖着鼠标。

Attribution

The computer-use engine and the @oai/sky runtime belong to their vendor and are not part of this package. This plugin only locates and loads them in-process on behalf of DSH; it neither ships, copies nor modifies their files.

License

MIT — covers the plugin code in this repository only.