VHCU

Vision · Hearing · Computer Use for DeepSeek Harness
Local-first screen understanding, system-audio transcription, and safe Windows input automation — exposed as 28 tools and a sidebar panel.

Repository: github.com/qiaozy-16/dsh-vhcu

---

[!IMPORTANT] Before installing, read this.

• Required companion: this plugin's UI is designed to work with dsh-better-sidebar. Install and enable it in the same DSH profile first. VHCU will not render a usable panel without a sidebar host.
• Windows only — DSH Desktop App only: screen capture uses Windows GDI/BitBlt, input uses SendInput, and Hear uses WASAPI loopback. macOS and Linux are not supported. 仅支持 Windows 系统下的 DSH 桌面版 (Desktop App)，不支持 Web 网页端；浏览器沙盒无法启动本地 Python 或注入键鼠。
• DSH version: 0.2.1-rc.3 is recommended (tested profile). The plugin declares compatibility for 0.2.0-rc.2, 0.2.1-rc.2, and 0.2.1-rc.3.
• This plugin can control your real mouse and keyboard. Read the Safety section before using the privacy-shield override or automatic approval of sensitive operations.

---

What is VHCU?

VHCU is a DeepSeek Harness plugin for local multimodal computer use:

• captures the primary screen and turns visual differences into structured events;
• transcribes system playback audio (not the microphone) with a local faster-whisper model;
• exposes 28 model-callable tools for observation, window capture, UIA element search, clicking, typing, hotkeys, dragging, shape drawing, and macro replay;
• fuses vision and hearing into a single 1.5-second multimodal timeline;
• keeps screenshots, audio, coordinates, and macros on the local machine.

It is intentionally not a cloud automation service. The only expected network access is the optional one-time faster-whisper model download.

---

Feature Highlights

Multimodal perception

Capability	Detail
Screen capture	mss / Windows BitBlt, primary monitor, DPI-aware
Visual diff	Lightweight NumPy diff, change-only ROI events, 1 fps idle loop
Full frame	Signed local PNG URL with image-to-physical scale_factor
Active window	Foreground title/exe for context-aware macros
Hear	WASAPI loopback → WebRTC VAD → faster-whisper base (CPU int8)
Fusion	1.5 s vision + audio windows, modalities: both / vision / hear
Untrusted data	Screen/audio text wrapped in <untrusted_screen_data> before reaching the model

Computer use

Group	Tools
Vision	vhcu_start_vision, vhcu_stop_vision, vhcu_get_vision_events, vhcu_get_full_frame, vhcu_capture_target
UIA	vhcu_find_elements、vhcu_click_element（uiautomation>=2.0.29，Windows）
Context	vhcu_get_active_window, vhcu_get_recommended_macros
Input	vhcu_mouse_click, vhcu_type_text, vhcu_key_press, vhcu_mouse_scroll, vhcu_mouse_drag, vhcu_mouse_middle_drag
Shape drawing	vhcu_draw_shape / input.draw_shape generates 16–512 high-density parabola/sine/cosine points; virtual path keeps MK_LBUTTON on every move; physical path uses 1 ms micro-delays
Humanized input	optional humanize_profile (off / precise / balanced); balanced clamp(80+0.15d,100,350)ms cubic Bezier + jitter, precise 30-80ms, virtual 80-120ms, Turbo auto-downgrades to precise; UIA InvokePattern does not move the mouse
Virtual cursor	cursorInjectionMode (physical / virtual / hybrid); Win32 PostMessage client-coordinate injection with DPI mapping; Python layered click-through overlay; hybrid only falls back before the first queued message
Persistent settings	settings.json in %LOCALAPPDATA%\dsh-vhcu (override VHCU_SETTINGS_PATH); atomic temp+rename save, defaults merge on load, visible persistence errors
Macros	vhcu_save_macro, vhcu_run_macro, vhcu_macro_open_notepad_and_type, vhcu_macro_save_file
Recording	vhcu_start_recording, vhcu_stop_recording
Clipboard	vhcu_set_memory_clipboard, vhcu_get_memory_clipboard
Hear	vhcu_start_hear, vhcu_stop_hear, vhcu_get_hear_events
Fusion	vhcu_get_multimodal_events

Macro engine

• JSON macro steps: click, double_click, right_click, type, key, scroll, drag, middle_drag, wait;
• variables such as {{x}} / {{text}}; optional image-to-physical coordinate scaling;
• 100 steps per macro, 20 macros per queue;
• built-in Blender/CAD-friendly middle-button drag (MIDDLEDOWN → 16-step interpolation → MIDDLEUP);
• sensitive-operation macros (is_destructive: true) are denied by default until the user approves them in the panel;
• optional managed mode (自动批准敏感宏（托管模式）) can skip approval, with a permanent red warning banner;
• Turbo mode allows skip_readback: true on click/hotkey/macro for fast consecutive actions, at the cost of visual verification.

UI

• Single sidebar panel with Vision / Action / Hear / Fusion tabs;
• collapsible 常规设置, 高级调试, 高级设置 sections;
• soft-pause banner: 👀 您正在接管键鼠，AI 已暂停 — real user input pauses AI injection and resumes 1.5 s after the user stops (Turbo mode: 50 ms + automatic frame refresh);
• sensitive-operation approval modal, no window.confirm;
• recording strip with 🔴 开始录制宏 / ⏹ 停止并保存.

V1.0.1 platform boundary

Area	What changed
Platform drivers	DriverFactory selects InputBackend, CaptureBackend and AudioBackend; Windows adapters wrap the existing SendInput/mss/SoundCard code.
Responsive interrupt	Token-protected loopback POST /interrupt; soft pause and emergency shutdown can interrupt a running macro or Hear step; new E_INTERRUPTED error.
Absolute path sandbox	Node injects absolute VHCU_MODEL_PATH, VHCU_MACROS_PATH and VHCU_LOG_DIR; Python rejects relative values instead of resolving against the process CWD.
Release packaging	PyInstaller onedir bundle with full Hear runtime closure; no onefile extraction into %TEMP%.

This release does not add a macOS/Linux implementation. Windows behavior and the existing safety model remain the reference implementation.

---

Architecture

flowchart LR
    U[User] --> D[DeepSeek Harness Desktop]
    D --> S[dsh-better-sidebar]
    S --> C[VHCU UI Client<br/>lib/client.js]
    D --> N[VHCU Node Host<br/>lib/index.js]
    N -->|NDJSON over stdio| P[Python Daemon<br/>vision/vhcu_daemon]
    N -->|POST /interrupt| P
    N -.->|absolute model/macro/log paths| P
    P --> SC[mss / BitBlt capture]
    P --> IN[SendInput keyboard / mouse]
    P --> HR[Hear pipeline]
    HR --> VAD[WebRTC VAD]
    HR --> ASR[faster-whisper base<br/>CPU int8]
    N -->|SSE /vhcu/events| C
    N -->|HTTP /vhcu/frame| C
    N --> MF[(macros.json)]
    P --> MD[(models/)]

Process boundaries

• Node host (lib/index.js) owns tool registration, HTTP routes, SSE, safety state, macros, and the supervisor.
• Python daemon (vision/vhcu_daemon) owns screen capture, image diffing, input injection, and Hear.
• Browser client (lib/client.js) is loaded by dsh-better-sidebar; it never touches the desktop directly.
• The supervisor validates every frame, enforces heartbeat recovery, and can restart the daemon without restarting DSH.

---

Requirements

Component	Requirement
OS	Windows 10/11 x64
DSH	0.2.1-rc.3 recommended
Sidebar host	dsh-better-sidebar (required)
Node.js	>= 20
Python	Source builds only; the full Release bundle ships the prebuilt engine
Screen	Primary monitor; DPI-aware
Audio	Any WASAPI playback device (Hear only)

The normal-user full bundle does not require a separate Python installation. Node.js is still used by the one-time node scripts/install.mjs registration command and by the DSH host.

---

Installation

There are two paths. Regular users should install the full release package. Source code building is for developers who need to modify or rebuild VHCU.

For a list of changes in V1.0.1, please refer to RELEASE_NOTES.md.

For Users: Zero-Dependency Installation

Normal usage requires no source code modification, no Python environment, no package manager, and no build steps.

1. Download the Complete Package

Get it from Releases. Download dsh-vhcu.zip.

[!IMPORTANT] Please use this complete package. It includes all necessary dependencies and the pre-built engine. Unpack it and follow the steps below.

The complete package includes:

• Plugin source code: lib/, host/, contract/, vision/
• Frontend dependencies: node_modules/
• Compiled standalone engine: dist/vhcu-daemon/
• Offline Hear model: models/
• One-time registration script: scripts/install.mjs

Extract the ZIP to a permanent directory, for example:

%LOCALAPPDATA%\dsh-vhcu\

Do not run the plugin directly from a temporary directory. Moving or cleaning up the download directory will cause DSH to lose track of the plugin.

2. Run the Registration Script

First, ensure dsh-better-sidebar is installed and enabled in the same DSH profile. VHCU's panel relies on this sidebar host.

Open PowerShell in the extracted directory and run once:

node scripts/install.mjs

If your DSH profile is not the default, use the --profile parameter:

node scripts/install.mjs --profile <your_profile_name>

Optional:

node scripts/install.mjs --dry-run    # View plan only, do not write
node scripts/install.mjs --uninstall  # Uninstall

This script only does three things: creates a directory junction, adds dsh-vhcu to dsh.profile.bundles, and adds dependencies. It will never run npm install or pip install.

3. Restart DSH

Close and restart the DSH desktop app (not just refresh the page). You should see this in the startup log:

dsh-vhcu: mounted (routes /vhcu/state|events|control|frame|settings|macros)

After restarting, open VHCU in the sidebar.

4. Hear Model (Optional)

The complete package already includes the models/ directory, so no extra download is normally needed. If the model is missing or automatic download is slow, you can manually download models.zip from Releases and extract it to the root directory.

Build from Source (Developers)

Use this path when you want to modify VHCU or rebuild the Python engine. The release ZIP is still the recommended installation for normal users.

Prerequisites

• Windows 10/11 x64
• Node.js 20 or newer (Node.js 24 is recommended)
• Python 3.10 or newer (64-bit; py -3 must resolve to it)
• Git for Windows
• PowerShell 7 (or Windows PowerShell 5.1)

1. Clone the Repository

git clone https://github.com/qiaozy-16/dsh-vhcu.git
cd dsh-vhcu

2. Install Node Dependencies

npm install

uiohook-napi is optional. If its native ABI does not match your Electron/Node runtime, the core vision and input tools still work; only global recording and user-takeover detection require it.

3. Create the Python Virtual Environment

py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

The daemon needs mss and numpy; Hear additionally needs soundcard, webrtcvad-wheels, and faster-whisper. UIA element search needs uiautomation.

4. Build the Standalone Python Engine

.\.venv\Scripts\python.exe -m pip install pyinstaller
.\.venv\Scripts\pyinstaller.exe --clean vision/vhcu_daemon.spec

The onedir bundle is written to:

dist\vhcu-daemon\vhcu-daemon.exe

Verify the build:

Get-ChildItem .\dist\vhcu-daemon

The spec deliberately does not bundle the Hear model. Keep the offline model pack in models\; the host passes its absolute path to the daemon at runtime. The dist\vhcu-daemon\ onedir folder is the release artifact; development runs use the interpreter configured through VHCU_PYTHON.

5. Register the Plugin with DSH

From the repository root, run:

node scripts/install.mjs

If your DSH profile is not the default:

node scripts/install.mjs --profile <your_profile_name>

For source-only development without building the PyInstaller bundle, point the plugin at the virtual environment instead:

$env:VHCU_PYTHON = (Resolve-Path .\.venv\Scripts\python.exe).Path

The registration script itself never runs npm install, pip install, or the build; it only links the plugin and updates the DSH profile.

6. Restart DSH

Close and restart the DSH desktop app. Confirm the startup log contains:

dsh-vhcu: mounted (routes /vhcu/state|events|control|frame|settings|macros)

For a quick developer check, place the .venv Python on VHCU_PYTHON, start DSH, and run:

npm test
npm run test:py

---

Quick Start

1. Restart DSH after installing the plugin.
2. Open the sidebar and select VHCU.
3. Click 开启视觉 (or ask the model to call vhcu_start_vision).
4. Ask for a screenshot or a click; the model should call vhcu_get_full_frame before coordinate-based actions.
5. Optional: select the Hear tab and click 开启听觉.
6. Optional: open ⚙ Settings to adjust input method, Hear sensitivity, Turbo mode, or privacy policy.

The model-facing loop is intentionally explicit:

observe (full frame) → act (click/type/key) → wait (~300–500 ms) → read back (events)

E_STATE means an action was attempted before the previous action was visually confirmed. In normal mode, call vhcu_get_vision_events or vhcu_get_full_frame first.

---

Safety

VHCU is a real computer-use tool, not a sandbox. Treat every screen and every action as consequential.

Privacy hard block

• Foreground-window titles are checked against a configurable blacklist (default includes password/login/banking terms).
• On a blocked window, VHCU returns E_PRIVACY_BLOCKED, stores no frame, and invalidates any previous frame URL.
• No vp_* or external screenshot tool should be used as a fallback during a privacy block. Stop and wait for the user.
• 关闭隐私屏蔽（开发者模式） disables this protection and is gated by an explicit confirmation; leave it off unless you fully understand the risk.

Input guards

• Four screen-corner failsafe: if the real cursor is parked in a corner, input is refused.
• Coordinate clamping to the primary monitor.
• frame_timestamp freshness check (default 1 s) for click coordinates.
• Foreground-focus guard: E_FOCUS_LOST if the target window changed.
• Rate limiting in both Node and Python.
• Real user mouse/keyboard activity triggers a soft pause (paused_by_user); AI injection is filtered via isAiInjecting.

Sensitive operations

• Macros can be marked is_destructive: true (built-in macro_close_active_window is).
• Default behavior: the panel asks for approval; deny returns E_DENIED.
• Managed mode skips the prompt only after an explicit confirmation and shows a persistent red banner.
• alt+f4, ctrl+s, and delete hotkeys require confirm_destructive: true.

Prompt injection

Visual and audio text is returned inside:

<untrusted_screen_data>...</untrusted_screen_data>

Tool descriptions explicitly tell the model that content inside is data, never instructions. VHCU does not execute commands found on screen.

Data location

• macros.json is local private state and is git-ignored.
• models/ is local and git-ignored.
• Screenshots are held in memory / short-lived local frame URLs; no cloud upload.
• No microphone capture.

---

Macro JSON

{
  "name": "blender_orbit",
  "category": "3d",
  "target_app": "blender",
  "is_destructive": false,
  "steps": [
    { "op": "middle_drag", "x1": 640, "y1": 360, "x2": 820, "y2": 420 },
    { "op": "key", "keys": ["shift", "a"] },
    { "op": "wait", "ms": 250 },
    { "op": "type", "text": "Cube" }
  ]
}

Run a single macro:

{ "name": "blender_orbit" }

Run a queue (up to 20):

{
  "macros_queue": [
    { "name": "macro_launch_app", "variables": { "app_name": "blender" } },
    { "name": "blender_orbit" }
  ]
}

Turbo mode (after enabling it in Settings):

{ "name": "blender_orbit", "skip_readback": true }

---

Configuration

Variable	Purpose
VHCU_PYTHON	Optional absolute path to a custom Python interpreter
VHCU_MODEL_PATH	Absolute model root; defaults to <plugin>/models
VHCU_MACROS_PATH	Absolute macros.json path injected by the host
VHCU_LOG_DIR	Absolute log directory contract injected by the host
VHCU_MACRO_ACK_TIMEOUT_MS	Macro queue acknowledgement deadline; default 45000
VHCU_INPUT_DRY_RUN=1	Validate input calls without injecting
VHCU_DISABLE_USER_TAKEOVER=1	Disable global user-takeover watcher (tests/automation)
VHCU_KILL_ALL_PYTHON=0	Do not kill orphan python.exe on Hear-only start
VHCU_IDLE_TIMEOUT_MS	Override idle-watchdog timeout for demos/tests
HF_ENDPOINT	Overridden to https://hf-mirror.com when downloading Hear models

Runtime settings (input method, Hear sensitivity, privacy blacklist, Turbo, managed approval) are in the ⚙ Settings panel and live in the plugin process; restarting DSH restores safe defaults for the sensitive switches.

---

Project Structure

dsh-vhcu/
├─ lib/
│  ├─ index.js              # Node host: tools, routes, SSE, supervisor, safety
│  └─ client.js             # dsh-better-sidebar browser client
├─ host/
│  ├─ lib/                  # macros, recorder, guards, fusion, supervisor
│  └─ test/                 # Node test suite
├─ contract/
│  ├─ lib/contract.js       # command whitelist, error codes
│  ├─ schema/v1/            # JSON schemas
│  └─ test/                 # contract tests
├─ vision/
│  ├─ vhcu_daemon/          # Python daemon: capture, input, Hear, protocol
│  │  ├─ drivers/           # InputBackend / CaptureBackend / AudioBackend
│  │  ├─ paths.py           # absolute resource path contract
│  │  └─ interrupts.py      # cooperative interrupt channel
│  ├─ tests/                # Python test suite
│  └─ vhcu_daemon.spec      # PyInstaller onedir build spec
├─ dist/vhcu-daemon/        # release build (gitignored)
├─ assets/icon.png          # DSH plugin-list icon (V1.0.2)
├─ scripts/install.mjs      # DSH profile installer
├─ models/                  # local model files (ignored)
├─ macros.json              # private macros (ignored)
├─ requirements.txt
├─ RELEASE_NOTES.md
├─ .gitignore
└─ README.md

---

Development

npm test          # Node: 116 tests
npm run test:py   # Python: 216 tests

Useful commands:

npm run dev                 # local browser harness
npm run demo                # host CLI demo
npm run install:plugin:dry  # inspect profile changes

Test suites use VHCU_INPUT_DRY_RUN=1 and mock Python/audio inputs; they do not move the real cursor or capture a real screen.

---

Troubleshooting

Symptom	Fix
Sidebar shows no VHCU panel	Confirm dsh-better-sidebar is installed/enabled in the same profile; restart DSH.
E_UNAVAILABLE on Hear start	Install requirements; confirm soundcard, webrtcvad-wheels, faster-whisper, numpy are in the interpreter selected by VHCU_PYTHON.
Hear downloads forever	Use the Releases offline model pack; check models/faster-whisper-base/model.bin.
E_PRIVACY_BLOCKED	The foreground window matched the privacy blacklist. Switch away or adjust the blacklist; do not use external screenshot fallback.
E_UIA_UNAVAILABLE on vhcu_find_elements	Install uiautomation>=2.0.29 (and its comtypes dependency) in the interpreter selected by VHCU_PYTHON; restart the daemon.
E_STATE after an action	Read back with vhcu_get_vision_events / vhcu_get_full_frame, or enable Turbo and pass skip_readback: true deliberately.
uiohook-napi install fails	Core tools still work; user-takeover and recording are unavailable. Use a supported Node/Electron ABI or rebuild the native module.
Blender ignores vhcu_type_text	Pass use_clipboard_fallback: true; VHCU temporarily uses the clipboard and restores it.
Shift+A not received	Pass delay_ms: 35 (try 60–100 for slow/remote apps); VHCU sends scancodes.
Middle-drag does not orbit	Confirm Blender's orbit binding is middle mouse; focus the viewport first.

---

Limitations

• Windows 10/11 only; primary monitor only.
• DPI scaling is handled, but exotic multi-monitor layouts are not a primary target.
• Hear uses the CPU base model; not a cloud/GPU transcription service.
• The side panel needs dsh-better-sidebar.
• VHCU is not a security sandbox; it runs with the current user's desktop permissions.
• Web version is unsupported: the browser sandbox cannot spawn local Python or inject keyboard/mouse input; use the DSH Desktop App.
• Real Blender/remote-desktop input behavior is environment-dependent and should be verified manually.

---

License

MIT. See LICENSE.

Third-party components and their licenses are listed in THIRD_PARTY_LICENSES.md.

VHCU is an independent community plugin and is not an official DeepSeek Harness component.