dsh-computer-use-win

Windows computer-use bundle for DeepSeek Harness (dsh): screenshots, window and UI Automation element discovery, mouse/keyboard control, and direct Windows Settings launch.

The macOS-first sibling is Anionex/dsh-computer-use; this plugin fills the Windows gap.

What It Adds

Tool	Purpose
computer_screenshot	Capture the full screen or a specific window. Vision-capable routes receive an image; text-only routes receive a saved PNG path.
computer_list_windows	List visible top-level windows with process, pid, title, screen rectangle, and active state.
computer_list_elements	Search the Windows UI Automation tree and return control names, types, physical-pixel rectangles, and click=(x,y) centers.
computer_open_settings	Open Windows Settings through the documented ms-settings: URI.
computer_click	Move the mouse and click at absolute screen coordinates.
computer_type_text	Type literal text through clipboard paste first, with SendInput fallback. This keeps CJK text reliable under VDI/RDP filtering.
computer_press_key	Press a key or key combination. SendKeys is primary; SendInput is used for combinations SendKeys cannot spell.
computer_scroll	Scroll the mouse wheel, optionally at given coordinates.
computer_drag	Left-button drag between two screen points.
computer_move_mouse	Move the cursor without clicking.

Input-injecting tools are approval-gated by default. With dsh's Full access preset (approval=never plus sandbox=danger-full-access), the plugin treats the action as already authorized and does not call the approval prompt that would otherwise deterministically reject. Confined modes still fail closed when the approval policy is never.

How It Works

The plugin has no native binary dependency. Every capability runs through a bounded PowerShell child process:

• Capture uses System.Drawing.CopyFromScreen.
• Window discovery uses Win32 EnumWindows, GetWindowTextW, GetWindowRect, and foreground-window APIs.
• Element discovery uses Windows UI Automation, which lets text-only models locate controls without seeing screenshots.
• Pointer input uses SetCursorPos and mouse_event.
• Text input uses clipboard plus System.Windows.Forms.SendKeys as the reliable path under Sangfor/VDI filtering, with SendInput fallback.
• All PowerShell processes opt into DPI awareness and UTF-8 output so UIA rectangles, screenshots, and click coordinates use one physical-pixel coordinate space.

Install

Requires the dsh CLI on Windows.

From a checkout:

git clone https://github.com/you/dsh-computer-use-win.git
dsh plugin --profile web add ./dsh-computer-use-win

Or install a packed tarball:

pnpm install
pnpm pack
dsh plugin --profile web add ./dsh-computer-use-win-0.3.3.tgz

Verify the layer, then boot:

dsh --profile web --dump-config | grep computer-use
dsh --profile web

Configuration

Patch the computer-use-win row in your profile's cordis.patch.yml:

- id: computer-use-win
  config:
    requireApproval: true
    maxScreenshotBytes: 8388608

Set requireApproval: false only for unattended tests or disposable environments.

Known Limitations

• Windows input injection is global: clicks go to whatever is under the coordinates.
• Elevated administrator windows may ignore input from a non-elevated plugin process.
• Win-key combinations fall back to SendInput, which some VDI products filter. Prefer semantic tools such as computer_open_settings where possible.
• UI Automation coverage depends on the target application exposing useful accessibility metadata.

Discovery

DeepSeek Harness currently documents out-of-tree plugin installation through profile dependencies and recommends adding the dsh-plugin GitHub topic for discoverability. There is no separate official plugin upload endpoint in the public dsh docs.

License

MIT