dsh-plugin-computer-user

Virtual-desktop computer use for DeepSeek Harness. The agent sees your screen in the right sidebar and its mouse acts on that snapshot — not on your real cursor. It plays its own game; you keep your computer.

A fork of jing-hy/computer-user that adds the half it was missing: a visualization.

---

The problem

Upstream computer-user gives a text-only model genuine control of a Windows machine. It works, but you cannot see any of it:

• computer_screenshot writes a PNG to a temp directory for the model's eyes only. You never see what it is looking at.
• computer_click moves your actual cursor and clicks. While the agent works, the machine is not yours.
• You find out what happened afterwards, from the conversation.

For a long autonomous task that is a bad deal: you cannot tell whether it is about to click the wrong thing, and you cannot use your own computer meanwhile.

What this fork changes

A live snapshot in the right sidebar. Every capture the agent makes is published to a dockable panel: the screen, the agent's virtual pointer moving across it, click ripples, the text it types, the keys it presses. You watch the work in progress.

Actions land on the snapshot by default. link_mode: virtual means the agent still screenshots, still "clicks", still "types" — but that pointer lives on the panel. Nothing reaches Win32 SendInput, so your cursor, your focus, and your window layout are untouched. The agent plays in the picture; your desktop is yours.

Flip to real when you mean it. One toolbar button (or the settings card) switches link_mode to real, and now the Win32 path runs — with the panel still mirroring what happens, so real mode is visible rather than a hijack.

You can point at things. Click anywhere on the snapshot, optionally with a note, and that becomes a queued instruction with an exact screen coordinate. The agent reads them through computer_get_vision, so "click this" stops being a description problem.

The agent gets numbers, not just a picture. computer_get_vision reports the captured size, the virtual-screen origin, the capture scale, the current pointer, the frame's age, and your pending hints. Combined with the panel, the look-act-verify loop closes on both sides.

How the two halves fit together

	upstream	this fork
capture	PNG to disk for the model	same PNG, published to the sidebar
pointer	real cursor via SendInput	virtual pointer on the snapshot (default)
typing / keys	real keystrokes	drawn on the snapshot (default)
verification	screenshot again, read it	screenshot again, watch it
user input	none	click the snapshot to point, with a note
real control	always	opt-in, one switch

Everything upstream did is still here: all nine computer_* tools, the four run modes, /computer approval, the settings namespace, the output guard that bounces tool calls written as prose, and the bundled PowerShell that needs no native module.

Install

pnpm install
pnpm run build          # → lib/index.js (host) + lib/client.js (browser)

Then add the package to your DSH profile as a plugin dependency. The bundle patch in cordis.patch.yml inserts the plugin row and sets the defaults:

- insert:
    - id: dsh-computer-user
      name: '@asun-labs/dsh-plugin-computer-user'
      config:
        mode: manual
        link_mode: virtual

Settings

Settings → 电脑操作. The two that matter sit at the top:

动作落点 (link_mode)

• virtual (default) — mouse and keyboard only affect the sidebar snapshot. Your desktop and cursor are untouched.
• real — Win32 SendInput drives this machine. The agent takes the real cursor.

运行模式 (mode)

• disabled — refuse every tool call.
• readonly — screenshot, cursor, vision reads, wait.
• manual (default) — acting tools need /computer approval in the conversation.
• auto — the model may call everything.

Advanced: capture interval (how often the panel refreshes), screenshot directory, typing interval, scroll units, whether the AI may change its own modes, the output guard, and debug logging.

The panel

control	what it does
虚拟快照 / 实机接管 badge	which mode is live, at a glance
→ real / → virtual	switch link_mode in one click
− / ＋ / %	zoom; 100% is real 1:1, one panel pixel per snapshot pixel
适应 (Fit)	whole snapshot, centred, aspect ratio preserved
1:1	original pixels, scrolled if it does not fit
跟随 (Follow)	pan automatically to keep the virtual pointer visible
⟳	capture now; otherwise the panel refreshes on the configured interval

Gestures: left click = point for the agent (with the note field, if you typed one), Shift+drag = pan when zoomed in, Ctrl/⌘+wheel = zoom.

The snapshot is never stretched. It is scaled by whichever axis fits and centred on the other, so the shapes you see are the shapes on screen — which is what makes clicking it meaningful.

Architecture

src/
  index.ts         plugin body — tools, routes, capture loop, settings, /computer
  tools.ts         the 12 computer_* tools, the mode gate
  session.ts       VirtualSession — the board the panel renders (frames, overlays, hints)
  ports.ts         DesktopPort: real (SendInput) vs virtual (record only)
  geometry.ts      screen ↔ frame ↔ panel transforms (pure, shared by both halves)
  client.tsx       the sidebar panel + the settings card
  ps.ts            bundled PowerShell runner (base64 JSON payload)
  capture.ps1      DPI-aware multi-monitor screenshot  (upstream, unmodified)
  input.ps1        Win32 SendInput mouse + keyboard    (upstream, unmodified)
tests/             geometry, session, tools, gate, output guard

The split that makes the fork work is ports.ts: a tool asks a DesktopPort to click, and whether that means "really click" or "draw a click on the panel" is one swap. link_mode changes which port is returned; nothing else in the tool layer knows the difference. It also means the whole tool set is testable without touching a real desktop — the tests do exactly that. The Host keeps only the latest 24 generated panel previews and removes its generated screenshots on an orderly shutdown. A screenshot path supplied explicitly by a tool caller is not managed by this cleanup.

geometry.ts is shared verbatim by both halves, which is why a click you make on the panel and a click the agent makes through computer_click resolve to the same screen pixel instead of merely close to it.

Requirements

Windows. Tested with DSH 0.2.1-alpha.2; the package also declares the earlier 0.1.5 and 0.1.7 lines. Capture uses System.Windows.Forms/System.Drawing and input uses Win32 SendInput, both through the bundled PowerShell — no native module, no build step, no external API. On other platforms the panel loads and reports that capture and injection are unavailable.

Attribution

Forked from computer-user by jing-hy, MIT licensed. The computer_* tool design, the run-mode gate, the /computer approval flow, the settings namespace, the output guard, and both PowerShell scripts come from that project; see NOTICE. The visualization layer, the virtual/real port split, computer_get_vision, computer_set_link_mode, and the sidebar panel are this fork's additions.

License

MIT — see LICENSE.