dsh-computer-browser

Desktop + browser automation for DeepSeek Harness — one tool surface that drives Windows applications and your real, signed-in Chrome.

Status: both channels live (background CDP + foreground real keyboard/mouse). Measurements and decision rules are below.

How one search task flows

Action	Which channel	Why (measured)
Send a query (address bar / URL)	background chrome_page_goto	1.5 s to the results page on a login-free site
Read results / paginate	background chrome_page_read	0.2–0.3 s per page with connection reuse; cold start eliminated
Click a public result link	background chrome_page_click (mode=auto)	navigation within 1.2 s = success; auto-escalates to foreground when it doesn't navigate
Operate something that must be visible	foreground chrome_page_click (mode=foreground)	brings the window forward and uses the real mouse, bypassing CDP clicks
Type into an in-site search box	foreground chrome_window_focus → keyboard	sites detect JS-assigned values; real keystrokes are steadier
Login/subscription sites	foreground (signed-in Chrome + KeePassXC Auto-Type)	the dedicated profile has no login state; forcing it only hits the login wall
Captcha / QR / risk control appears	foreground, or hand back to the user	human interaction is the only way
Write / submit / pay	foreground + explicit confirmation	irreversible actions
Two background clicks with no effect	auto-escalate to foreground	built into chrome_page_click as a fallback

Rule precedence: interception > login > action type > site allow-list.

Tool surface

Always present (2): computer_do (enter/leave the desktop surface) · screen_observe

Once expanded, these are added:

• Observation: screen_zoom
• Keyboard/mouse: computer_click / double_click / right_click / type / key / scroll / drag / wait
• Applications: app_list / app_launch
• Verification: computer_verify / computer_wait_for
• Other: computer_clipboard / menu / hover / stop / resume / computer_task
• Browser (background): chrome_page_read / chrome_page_click / chrome_page_goto
• Browser (foreground prep): chrome_window_focus
• Login detection: chrome_login_state (reads page facts — login URL, password field, "sign in" text, user menu, cookie count — never a site list)

Key measurements (2026-11-09)

• Background page read: 0.2–0.3 s (31 ms → 3 ms with connection reuse)
• Background navigation: fixed 1.5 s wait
• Foreground keyboard/mouse: 1–2 s per step, 4–7 s for a full chain
• Windows foreground lock: a bare SetForegroundWindow is refused; you need ShowWindow → ALT → BringWindowToTop → SetForegroundWindow, otherwise keystrokes land in DSH itself
• Real mouse clicks: SetCursorPos + mouse_event, using the screenX/screenY from locate(); wait at least 2.5–3 s before reading the URL, or you misread the result as failure
• The tool surface folds itself: two non-desktop tool calls collapse it, so a "bring forward + keys/mouse" sequence must finish inside one expanded window
• The dedicated Chrome dies with dsh (it is in the child process tree), so after a restart bring it back with --user-data-dir=<profile> --remote-debugging-port=9222

Red lines

• Passwords: only via KeePassXC Auto-Type into the foreground page — never read as text, never logged, never written to a file
• Login-required sites: background tools refuse and suggest the foreground path
• Dialogs: unrelated ones are dismissed automatically; login/QR/captcha, payment/outbound submission, and "the dialog is the task" are the three exceptions that must be offered as choices

Layout

lib/desk/register.js       tool registration (includes the two-channel chrome_page_*)
lib/desk/chrome-cdp.js     CDP session reuse / dialog handling / locate
lib/desk/policy-rules.js   site routing / confirmations / password-manager plan / channel fallback
lib/desk/win-focus.js      bring-to-front / real mouse click / window enumeration
lib/desk/surface.js        expand/collapse surface

Development

node --check lib/desk/register.js
node --check lib/desk/win-focus.js
node --check lib/desk/policy-rules.js

0.2.0 — three fixes from real user reports (2026-11-15)

Source: three field reports under D:\工作区\工作\产出\工具心得\ (57 calls / Navicat / database sessions).

1. The tool surface is flat by default. It no longer folds. In the folded state the action tools were not registered at all, so the host answered unknown tool and the caller never learned to run computer_do enter first (9 failures across three rounds). enter/exit remain as compatibility entry points.
2. Post-action verification. computer_click / double-click / right-click now take an AX fingerprint of the target window before and after (role+name sequence, element count, title; 2.5 s timeout), and the receipt says either "window content changed" or "no visible change — re-check recommended". When no fingerprint is available it still reports unverifiable rather than inventing a conclusion.
3. Three-state receipts: applied / no visible change / delivered only.

Next version (todo): unify the coordinate space with scale, degraded markers and click refusal, automatic pid for screen_zoom, atomic select_option / wait_for, a single-call desktop_state.

0.3.0 — glue between the desktop and browser tool groups (2026-11-15)

The field reports said the two groups did not hand off smoothly. This release only fixes the loose ends; cross-surface numbering stays for stage two.

1. Uniform browser receipts: chrome_page_read / click / goto / login_state now return surface: 'browser', and navigation-capable ones add urlBefore / urlAfter / changed, matching the desktop group's three-state receipt.
2. Lightweight task state: lib/desk/task-state.js remembers the current surface (desk/web), the window or tab, and the last snapshot; computer_do action="status" appends one line, and cross-surface switches are announced in the receipt. It is a status indicator, not task resumption.
3. Site policy in code: when chrome_page_read / click hits a site classified as foreground (CNKI, campus systems, trading sites), the receipt appends a hint to use real foreground input instead of relying on operator memory.

Tests: 12/12 (four new task-state unit tests).

Version mapping

npm	Git tag	What it contains
0.1.0	(initial)	Scaffold only: lib/policy.js, lib/surface.js — no desktop engine, no CDP bridge
0.2.x	v0.2.3	Desktop engine fixes from three field reports
0.3.0	v0.3.0	Current release: everything under lib/desk/, lib/web/, client/, tests/

The repository tracks the npm package from 0.3.0 onward; master and the published tarball describe the same code.