dsh-browser-view · Live Browser View for DeepSeek Harness (DSH)

English | 中文

 

A browser the agent drives, watched from a tab beside the chat. The host half owns a dedicated Chromium over the devtools protocol; the browser half contributes one right-sidebar tab type, so the page stays in view while the agent verifies it.

This package is an external plugin for DeepSeek Harness (DSH), structured as a cordis bundle: src/index.ts is the host half (Chrome process, CDP sessions, tab registry, WebSocket fan-out, agent tools) and src/client/ is the browser half (the sidebar tab body and its chip).

What it does

• A live tab, not a screenshot poll: the panel draws the page three ways and you pick: pixel frames (Page.startScreencast, the default, and the fallback for the other two), a live DOM mirror (rrweb's recorder in the page, rrweb's replayer in the panel, crisp text and event-sized traffic) and a tab capture (the tab's own surface recorded to VP8 and played with MSE, which is the lane with a real picture for a cross-origin <video>). The tab shows the active page, its URL and title, and what the agent last did.
• Eleven agent tools: browser_navigate, browser_click, browser_type, browser_press, browser_scroll, browser_snapshot, browser_screenshot, browser_console, browser_network, browser_tabs, browser_close (closes only the tabs this session opened).
• Evidence in the conversation: every browser_screenshot result carries the image, so the transcript keeps the picture the agent acted on instead of only a sentence about it.
• One dedicated browser: the plugin launches its own Chromium with a throwaway profile. It never attaches to a browser you are signed into unless you opt in with DSH_BROWSER_VIEW_ATTACH_PORT (see Security model).
• Signs itself in to this host's own GUI: a fresh profile has no session cookie, so the viewer would only ever see the 401 challenge. When a navigation lands on that challenge over loopback, the plugin asks the host's connection service for the same ticketed URL dsh web printed, and retries once. The cookie that mints lives in the throwaway profile and dies with it.

How it works

agent tool call ──► host half ──► CDP ──► dedicated Chromium
                        │
                        └─ screencast frames ──► WebSocket ──► sidebar tab body

The host half starts Chrome lazily on the first tool call or the first tab mount, pins a viewport, and keeps a ring of console entries and network requests per tab so a verification can be read back without re-running the page. A top-level document replacement clears both rings; a same-document navigation (hash, query, SPA pushState) keeps them.

Frame transport and bandwidth

Frames default to the WebSocket (JSON + base64). After mounting, the panel additionally performs a WebRTC handshake with the host: the panel creates an unreliable, unordered DataChannel (drops beat stalls, exactly what a frame stream wants) and the host answers with werift, a pure-TypeScript WebRTC implementation with zero native dependencies. Once the channel opens, binary frames travel there; the control plane (state, actions, commands, input) stays on the WebSocket. Every failure - including a single frame the channel cannot send - falls back to the WebSocket, and costs that frame only: the panel is never dropped from the broadcast.

One frame is split into chunks of at most 60 KiB, each prefixed with [u32 sequence][u8 index][u8 count], and the panel rebuilds it before decoding. This is not optional: a DataChannel message is capped at 64 KiB, while real pages routinely exceed it (measured at 1280x800 and quality 70: 111 KB for a Wikipedia article, 250 KB for a heavy DOM page), so an unchunked frame throws every single time.

Bandwidth is designed for "look when you need to, step in when you must", not for watching video:

Measure	Effect
Frame encoding	WebP at quality 50 (was JPEG at 70), visibly the same at a clearly smaller size
Capture box	Captured at 1024x640 (the browser viewport stays 1280x800); pointer coordinates are scaled back up, so clicking stays exact
Frame budget	At most about 15 fps while nobody interacts, opened to about 30 fps for three seconds after any input
Invisible means stopped	A collapsed sidebar, another chip in front, or a hidden browser tab tells the host; the last watching panel leaving stops the screencast outright - nothing is captured, encoded or sent - and returning restores it with one fresh frame

Four lanes: auto, pixels, the DOM mirror and the tab capture

The panel can render the page four ways. auto is the default, and you can pick any of the others:

Lane	How it works	What it is for
Auto (default)	The host probes the watched tab with one cheap Runtime.evaluate - does the document carry media the mirror cannot draw? - and resolves media to the capture and everything else to the mirror. It re-resolves on every navigation of the watched tab and never polls	Letting the page decide: crisp text on quiet pages, a real picture on media pages, no opinion required
Pixel stream	CDP Page.startScreencast frames, exactly as before	Anything at all - <video>, canvas, native UI, DRM - and the automatic fallback of the other lanes
DOM mirror	The host injects rrweb's recorder into the tab (Page.addScriptToEvaluateOnNewDocument for future documents, Runtime.evaluate for the loaded one), records each mutation and streams it to the panel, which replays it with rrweb's own replayer in live mode	Text and UI pages: crisp at any panel width, and traffic proportional to what changed rather than to how often the page repaints
Tab capture	Chromium is launched with --auto-select-tab-capture-source-by-title=<sentinel>, --use-fake-ui-for-media-stream and --enable-usermedia-screen-capturing; right before capturing, the host renames the watched tab to the sentinel (the flag matches the title at capture time) and asks the page for getDisplayMedia({ video: { displaySurface: 'browser' }, audio: false, preferCurrentTab: true }); the page forwards MediaRecorder chunks (video/webm;codecs=vp8, ~1.5 Mbps, 200 ms slices) over a Runtime.addBinding binding and the panel plays them with MSE	A real picture of cross-origin media. A DOM mirror has an element where a <video> is and no pixels in it, and element-level captureStream() throws SecurityError for a cross-origin CDN - which is where a page's media almost always lives. Tab capture goes through the browser's own surface, so the origin of the media never enters into it

The switch sits in the toolbar and names the lane currently on screen - for auto, the lane the page resolved to (for example "Auto (mirror)"), so a viewer always knows whether they are watching a recording; pressing it cycles auto → mirror → capture → pixels. The mirror and the capture are both read-only: a click on a replayed node or on the captured video is not mapped back to a CDP action, because neither is the page's own element tree. Input stays on the pixel lane.

Two things are deliberate about the two non-pixel lanes' cost:

• They only run while somebody watches. The recorder is armed when a visible panel asks for that lane and stopped when the last one stops asking (or hides), and it follows the watched tab, so a background tab never records. A capture that stops also puts the page's original title back and releases the display-capture tracks.
• The pixel lane stays warm until the new lane has something to draw. Frames keep arriving until the first full snapshot (mirror) or the first chunk (capture) reaches a panel, so switching lanes never trades a live picture for an empty stage. If either cannot start - the page refuses the injected script, the recorder throws, the bundled artifact is missing, the display-capture request is denied, the recorder dies mid-stream - the panel falls back to pixels automatically and shows the reason instead of a blank viewer.

Measured on this repository's own fixtures (5 second windows, one lane at a time, headless Chromium at 1280x800): on grid.html, a static page, both the pixels and the mirror are silent after the first paint; on heavy.html, 4000 absolutely-positioned nodes with 400 inline-style mutations every 50 ms, pixels cost 1169 KiB/s and the mirror 1093 KiB/s. That fixture is the mirror's worst case - it rewrites most of the document's inline styles twenty times a second - and it is worth knowing that "cheaper" is a property of the page, not of the lane: a page that mutates a little is nearly free to mirror, one that mutates everything constantly is not. The capture lane is measured against the pixel lane on the same page: a 800x500 animated av.html records at roughly 320-500 kbps of real VP8, against the pixel stream's ~1472 kbps of stills, and it is the tab's compositor output (30 fps) rather than ~10 fps of still frames.

Packaging: rrweb is a runtime dependency, and the host injects the UMD build this package ships - lib/rrweb.umd.cjs, copied from rrweb's dist by pnpm build. The host never resolves a node_modules path at runtime, because the installed layout is not this repository's. The panel's replayer rides the normal client bundle (lazily imported, and the bundle is pinned to a single file so the module loader can serve it). The capture lane needs no artifact at all: its two scripts are built in src/core/capture.ts, and the panel's player is a MediaSource it constructs itself.

Interface seats

Seat	Kind	What lands there
sidebar.right.pane.tab	keyed, session	the tab body
sidebar.right.pane.tab.title	keyed, session	the chip text and glyph

The conversation header carries no entry for this tab: the column's own tab strip and the guide capsule already open it, and a third place to click was only crowding that row.

The tab type is registered with ctx.sidebarRightTabs.register and opened with ctx.sidebarRight.openTabIn. Its kind is browser-view, deliberately not browser: the shipped @deepseek-ai/dsh-client-ui-sidebar-browser owns that kind, and a second registration in the extension band takes a kind over, which would shadow the shipped tab out of existence.

The single-occupant rightbar / rightbar.session seats are still not touched: they belong to the column itself, not to the tabs inside it.

Security model

• Loopback only: the panel socket is fenced to loopback requests; a non-local client is rejected before the upgrade. Whoever can establish the socket can also send input to the watched page (clicks, keys, wheel), so the fence and CIDR lists should stay limited to the ranges that genuinely serve the GUI.
• Opt-in CIDR widening: setting DSH_BROWSER_VIEW_ALLOWED_CIDRS to a comma-separated list of IPv4 CIDR ranges (for example 192.168.1.0/24) admits panel socket requests whose socket address falls inside a listed range. The widening keeps the other request-level gates: a request carrying forwarding headers (a proxy or tunnel in front of the host) is still refused, as is a cross-site request or one whose Origin does not match the Host. Unset or empty means loopback-only, unchanged. A listed range can drive the browser tools from that network, so list the narrowest range that actually serves the GUI; ::1 stays always allowed and the list never admits pure IPv6 addresses.
• Trusted proxy is a separate, narrower opt-in: with a reverse proxy in front of the host, the forwarded-header refusal disables the CIDR widening above (the proxy appends forwarding headers and rewrites Host). Three variables must all be set to open this path: DSH_BROWSER_VIEW_TRUSTED_PROXY_CIDRS lists the socket ranges the proxy sits on (for example 127.0.0.1/32), DSH_BROWSER_VIEW_ALLOWED_CIDRS constrains the client address taken from the last X-Forwarded-For entry, and DSH_BROWSER_VIEW_PUBLIC_ORIGIN declares the public origin the browser actually uses (comma-separated for several, for example https://100.64.0.10:30082,https://10.76.144.1:30082), which the request's Origin must equal one of. Missing any one keeps the path closed; cross-site requests stay refused. The trust boundary is real: a host inside the trusted range can forge X-Forwarded-For to impersonate a listed client, so trust only the narrowest range around the proxy itself and keep the proxy-to-host path on the machine.
• Throwaway profile: Chrome runs on a fresh --user-data-dir under the system temp directory with a random debugging port, so no cookie, session or extension of the user's real browser is reachable.
• Attach mode is opt-in and hands-off: setting DSH_BROWSER_VIEW_ATTACH_PORT to a DevTools debugging port makes the plugin drive that already-running browser instead of launching one. The plugin never enables remote debugging itself; the operator starts the browser with --remote-debugging-port=<port> on a loopback address. In attach mode the plugin does not kill the browser and does not touch any profile — dispose only closes the tabs the plugin itself created and its own protocol session. The trade-off is real: the agent is now driving a browser that holds your sessions, so in attach mode every agent tool may only operate tabs the calling session created via the tools; a foreign target (your own page) is refused with the list of allowed tabs, and browser_close closes only session-created tabs and never falls through to anything else. Only turn attach mode on for a browser profile you accept the agent working inside.
• Headless by default: nothing pops up on the desktop; the tab is the only view.
• Process lifetime: in spawn mode the Chromium child is killed when the plugin disposes. Nothing is left running behind a closed session. In attach mode the browser is yours and outlives the plugin by design.
• Token handling: the launch token is minted per navigation, held in memory, and never written to disk; the retry only happens on a loopback target whose document is DSH's own challenge, so no other local service is ever handed the token.

Known limits

• The tab is an interactive view: clicks, drags, wheel and keyboard travel from the panel into the watched page (the host-side Chromium); your input never passes through an agent session. The address bar stays read-only, and navigation remains the agent's tools' or the tab picker's job.
• The tab lives inside the right sidebar, so it needs that column; a layout that hides the column hides the tab.
• Frames are produced when the page changes, so a static page holds the last frame instead of streaming duplicates.
• Switching chips keeps the body mounted (keepMounted), so the socket survives; the frame still waits for the next change.
• The token top-up only covers the site root, because DSH answers the challenge with a 303 to ./ for GET / alone.
• One browser process per host; tabs are isolated CDP targets, not separate profiles.
• Chrome must be installed at a standard location (or named by CHROME_PATH).
• Audio is a non-goal on every lane: the mirror records DOM only, and the tab capture asks for video only (audio: false, no audio codec in the container), so <video> sound is never captured - the browser also runs with --mute-audio.

Install

This repository is a standalone bundle: install it straight from GitHub.

dsh plugin --profile web add git+https://github.com/jimorsm/dsh-browser-view

lib/ is committed, so the install runs no build script and needs no allowBuilds opt-in; pin a commit or tag with #<ref> if you want a frozen copy. To build locally instead, clone and link the checkout:

dsh plugin --profile web add link:<this checkout>

The plugin row is read at startup, so a DSH restart is required after installing or changing it.

Not published to npm. The upstream project publishes the same plugin as @linxin666/dsh-client-ui-browser-view; do not install both - they claim the same plugin row id (ui-browser-view) and the same right-sidebar tab kind (browser-view), so the later one in the loader roster wins silently.

Development

pnpm install
pnpm typecheck
pnpm test
pnpm build

Provenance

Forked from the dsh-web plugin packages/dsh-browser-view (upstream package @linxin666/dsh-client-ui-browser-view, version 0.4.10), plus this fork's own feature work: panel input relay, the opt-in trusted-proxy socket fence, and WebRTC DataChannel frame transport. See NOTICE for attribution and licensing; the upstream package name and npm scope belong to the upstream project and are not used here.