dsh-browser-panel

English | 简体中文

A browser that lives inside the DSH host: every DSH session gets its own tab in it, the AI drives that tab with browser_panel_* tools, and you watch — or take over — the very same tab from a panel in the session page of the DSH Web UI.

It exists for one stubborn problem: agents run on machines with no display, but the sites they need are behind a login. A headless browser can be scripted; it cannot scan a QR code, type an SMS code, or solve a CAPTCHA. With this plugin the human does that part inside the DSH Web UI, on the same tab the AI is already working on, and the result stays in a persistent profile — so a login performed once works in every session, now and after a restart.

The problem

• An agent host (container, VM, CI box) has no GUI, so it cannot show you a login page.
• Fully headless automation breaks at the first step that needs a person: SSO redirects, one-time passcodes, QR-code logins, CAPTCHAs, hardware keys.
• Copying cookies or storageState into the container is fragile: device-bound credentials, fingerprints, and WebAuthn simply do not transfer.

What it does

One browser, one tab per session, two control planes. The AI speaks CDP on its own tab; the human speaks the panel on that same tab. The login the human performs is exactly the session the AI keeps using — and because all sessions share one profile, every other session is logged in too.

Capability	Tool	Notes
Report state	browser_panel_status	Running? Which URL/title has this session's tab? Is a human action pending?
Open a URL	browser_panel_navigate	Starts the browser on first use; newTab replaces this session's page with a fresh tab
Read the page	browser_panel_snapshot	Title, URL, numbered inventory of clickable/typable elements, visible text
Click	browser_panel_click	By element number or by visible text; real input events
Fill a field	browser_panel_type	React/Vue-friendly insertion; submit presses Enter
Press a key	browser_panel_press	Enter, Tab, Escape, arrows, PageUp/Down, …
Scroll	browser_panel_scroll	down/up/left/right/top/bottom
Look at the page	browser_panel_screenshot	PNG returned to the model as an image attachment
Ask the human	browser_panel_ask_human	Opens this session's panel with your instruction and waits until they press 我已完成
Close the tab	browser_panel_close	Closes this session's tab; the profile (and every login) stays

Panel side (DSH Web UI):

• a 浏览器 / Browser tab in the session's view strip — the live view of that session's tab; every session shows its own;
• the sidebar entry (browser glyph) is the host-wide overview: one row per session tab — title, URL, last used, a 待接管 / needs you badge while that session waits for a person, and 结束并清理 / Close tab on each row;
• the canvas accepts your mouse, wheel, keyboard and IME input — it is not a screenshot viewer, the events are replayed into the very CDP session the AI uses;
• the hand-over banner appears in the panel of the session that asked, with a 我已完成 / Done button that resumes the waiting tool call (and an optional note back to the AI);
• a toolbar with back / forward / reload / repaint, Tab / ⇧Tab / Enter to walk a form without aiming the mouse, and 结束并清理 / Close tab.

How it works

flowchart LR
  AI["AI turn<br/>(browser_panel_* tools)"] -->|"CDP · this session's tab"| B["Chrome<br/>(persistent profile)"]
  H["You, in the DSH Web UI"] -->|"mouse · keyboard · IME"| P["Session browser panel<br/>(client half)"]
  P -->|"WebSocket /api/dsh-browser-panel/stream?session=…"| S["Host half<br/>(screencast + input replay)"]
  S -->|"Page.startScreencast (watched tab)"| B
  S -->|"Page.captureScreenshot (~7 fps, the rest)"| B
  S -->|"Input.dispatch* (that panel's tab)"| B
  B -->|"JPEG frames"| S --> P
  B -.->|"one profile: cookies, localStorage"| D[("profile dir<br/>$DSH_HOME/browser-panel/profile")]

• The host half launches one Chrome per DSH host process. With mode: auto it runs headed on a private Xvfb when Xvfb is available (better fingerprint than headless) and falls back to --headless=new otherwise.
• A session's tab is created lazily — on that session's first browser_panel_* call, or when the human opens the browser from that session's panel — and closed when the session is disposed, when the AI calls browser_panel_close, or when the human presses 结束并清理 / Close tab. Sessions are isolated in page state, not in identity: they share the profile, so a login performed once is available everywhere.
• Every tab is activated once when it is created. A Chrome target that was never activated silently discards every injected input event for the rest of its life, and one activation repairs it permanently. Afterwards a session's tab accepts input even while it is in the background — so the AI never moves your foreground.
• Chrome only emits screencast frames for the active tab. The panel you are actually looking at sends focus and owns the real Page.startScreencast; every other attached session is served by polled Page.captureScreenshot frames (~7 fps target; its first frame can be cold — seconds — while another tab owns the screencast). A freshly opened panel always gets a seeded frame, because a static page emits none on its own.
• Tools carry no session parameter: every handler reads the session id from its own execution context (exec.agent.id), so tool names and parameters stay small and a session can never steer another session's tab. A call with no owning session is an error.
• The panel is served by the host webserver, on the same origin and behind the same session authentication as the Web UI itself (connection.requestRejection). The routes that belong to a session carry its id — GET /api/dsh-browser-panel/state?session=…, POST /open|/close (session in the body), and the /stream?session=… WebSocket upgrade — while GET /sessions and GET /health describe the whole host, and POST /human-done settles a request by id. No extra port, no extra token, no tunnel.
• Human take-over is per session: two sessions can wait on a human at the same time, and a request is delivered only to the panel of its own session.
• Frames travel as JPEG over one WebSocket; input travels back as small JSON messages. Frames are dropped rather than queued when your connection falls behind.
• Nothing is written into DSH core: the plugin is one composition row.
• The measured constraints behind these choices are written down in references/PITFALLS.md (#11–#13).

Requirements

• DSH >= 0.1.5-rc.2, Node >= 22.19.
• A Chromium-family browser. Auto-detected in this order: browserPath config → google-chrome-stable / google-chrome / chromium / chromium-browser / chrome on PATH → Playwright cache (~/.cache/ms-playwright/chromium-*/…) → Puppeteer cache.
• Optional: Xvfb on PATH for headed mode. Without it the plugin runs headless automatically.

Install

Version 0.2.0. This release is deliberately breaking: the plugin used to own one shared page for the whole host, and there is no shared-browser mode any more — 0.2.0 gives every session its own tab. 0.1.x is the previous, single-page release.

# npm
dsh plugin --profile web add dsh-browser-panel

# straight from GitHub
dsh plugin --profile web add github:imroc/dsh-browser-panel

If your registry still resolves an 0.1.x build, ask for the version explicitly (dsh-browser-panel@0.2.0) or install straight from GitHub.

The package ships its built JavaScript, so nothing is compiled on install.

Restart the Web UI afterwards (adding a plugin row is a boot-time composition change):

systemctl --user restart dsh-web      # or however you run `dsh web`

Verify

1. Open the DSH Web UI. A 浏览器 / Browser tab appears in the session's view strip — click it. The sidebar also gains a browser entry, which lists the tabs of every session.

2. The tab reports that this session has no browser open yet, with a button to open one; the browser starts on demand (any browser_panel_* call opens it too), and the first frame is this session's own tab in the host Chrome.

3. Type a URL in the AI conversation and ask it to open it; the page appears in that same tab:

   Use browser_panel_navigate to open https://example.com, then browser_panel_snapshot.
   

4. Ask the AI to hand over, then complete the login yourself in the panel:

   Call browser_panel_ask_human with the instruction "请在面板里完成登录，然后点我已完成".
   

5. Log in, press 我已完成 — the tool call returns, and the AI continues with an authenticated session.

6. In a second session, open its own browser tab: it is a different tab, but it is already logged in, because the profile is shared. Close one session's tab (browser_panel_close, or 结束并清理 / Close tab) and reopen it: still logged in.

Configuration

Override any field from your own patch layer (~/.dsh/cordis.patch.yml, or the profile's cordis.patch.yml). The whole config key is replaced, so restate what you need:

- id: browser-panel
  config:
    mode: headless            # auto | headed | headless
    viewport: 1280x800
    idleShutdownMinutes: 30

Key	Default	Meaning
browserPath	''	Explicit Chrome/Chromium path; empty = auto-detect.
profileDir	''	Persistent profile; empty = $DSH_HOME/browser-panel/profile. Shared by every session.
mode	auto	auto = headed on a private Xvfb when available, else headless.
screen	1440x900x24	Geometry of the private Xvfb.
windowSize	1440x900	Chrome window size in headed mode.
viewport	1440x900	Emulated page viewport — the same for every session tab, and the panel canvas coordinate space.
xvfbDisplay	:99	Preferred X display; the next free one is used if taken.
port	0	Fixed DevTools port; 0 picks a free one.
startUrl	about:blank	First URL of each session's fresh tab.
extraArgs	[]	Extra Chrome switches.
snapshotMaxChars	4000	Page text budget per snapshot.
maxElements	80	Interactive elements listed per snapshot.
screencastQuality	60	JPEG quality of the stream (and of the polled stills).
screencastMaxWidth	1440	Maximum streamed frame width.
pollFrameMs	140	Poll interval for panels that cannot own the foreground screencast (~7 fps). Values below 60 are clamped.
askHumanTimeoutSeconds	600	Default budget of browser_panel_ask_human.
idleShutdownMinutes	0	Stop the browser — and with it every session's tab — after this much idle time; 0 never stops it.
autoStart	false	Start the browser with the host instead of on first use.
startTimeoutMs	20000	How long to wait for the DevTools endpoint after launch.

Security notes

• The panel and its WebSocket live behind the same authentication as the DSH Web UI. Anyone who can see the panel can already drive the host browser — treat Web UI access accordingly.
• Sessions are isolated in page state, not in identity: every session uses the same browser profile, and therefore the same cookies and logins. That is the point of the plugin, but it also means one session's AI can reach any site the profile is logged into.
• The browser profile holds real sessions. It stays on the host, under the DSH home, is never uploaded, and is not part of the Git repository.
• Browser traffic goes out from the container. A data-centre IP is a weaker signal than your laptop's; for sites that are aggressive about automation, mode: headed (the default whenever Xvfb is available) is the better half of the trade.
• Prefer running Chrome as a non-root user with the sandbox on; the plugin only adds --no-sandbox automatically when the host process runs as root.

Rollback

dsh plugin --profile web remove dsh-browser-panel   # or delete the row from cordis.patch.yml

Then restart the Web UI. The profile directory is left in place, so reinstalling keeps your logins. Delete $DSH_HOME/browser-panel/profile to forget them.

License

MIT