dsh-web-fetch-playwright

中文 · npm · GitHub

A DeepSeek Harness (DSH) plugin that gives the built-in web_fetch tool a Playwright/CDP backend: pages are rendered in a real browser, denoised with Readability + DOMPurify + Turndown + GFM, and returned as Markdown.

Features

• Real browser rendering — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
• Denoise pipeline — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped tool-web renderer. Inline data: images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as ![alt](data:image/png;base64,...8.9KB) so they cannot flood the body. On list-shaped pages (portals, catalogs) where the content is the links, Readability's link-density scoring picks a noise fragment — a coverage gate therefore distrusts an extraction under 250 characters or covering under 10% of the page's visible text (measured on the 163.com homepage: 96 characters of footer licensing won while 320 headlines scored near zero), and the pipeline falls back to the sanitized whole document, which on such pages keeps effectively all list content. An accepted extraction is then pruned of trailing link-dense sections — the post-article recommendation rails and share bars that ride inside the winning wrapper and survive every vocabulary-based rule (163's post_recommend cards measure 0.42–0.48 per-card link density, just under Readability's own 0.5 removal bar, and arrive as bare divs with no class to match); a trailing run of such blocks is removed only when it is a minority of its container and clearly denser than the content before it, so link-list extractions (feed rows, portal headlines) are untouched.
• PDF links — a URL that serves a PDF (inline application/pdf, a Content-Disposition: attachment download, or an application/octet-stream whose bytes start with %PDF-) is read as bytes and converted to Markdown instead of failing: pdf.js (via unpdf) supplies positioned text and real font names, and layout heuristics rebuild headings (outline/bookmarks, section numbering such as 3.1 / 第二章 / 一、, type size, standalone bold lines), paragraphs (gaps, first-line indents, cross-column/cross-page continuation, dictionary-checked de-hyphenation, CJK-aware joining), lists, monospace code blocks, column-aligned GFM tables, inline bold, and <!-- page N --> markers, while dropping running headers/footers and page numbers. See PDF links.
• Two backends — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
• Browser resolution — a configured path, a playwright CLI on $PATH, or the bundled playwright-core; CDP needs no local browser at all.
• Shared profile or reusable isolation (CDP) — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (connectOverCDP → browser.contexts()[0], like CLI CDP attach with isolation disabled, not playwright-cli open). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
• Live configuration — the Plugins page (侧边栏 → 插件 → dsh-web-fetch-playwright, dsh ≥ 0.1.7) shows the full form right on the plugin's package page (and on the row's own page via its configure control); it edits the backend, context mode, denoise toggle, and concurrency, and a committed change reaches the next fetch without a restart.
• Budget-aware — per-fetch waiting budget (25s) — when it runs out (a slow load, an endless scroll feed) the fetch stops waiting and extracts what the page already rendered instead of failing (truncated when the document never finished loading; WEB_FETCH_TIMEOUT only when nothing arrived at all or extraction itself hangs past a 10s grace); concurrency is backend-priced (maxConcurrency, default 4 local browsers / 50 CDP tabs; queued fetches fail fast with a retry hint after 10s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
• Bounded Cloudflare-challenge wait — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented cf-mitigated: challenge response header plus structural page markers), the fetch keeps the same tab and context and waits for the browser's own verification to clear it — tracking the last main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded by a built-in 15s wait, with a bounded same-tab retry (challengeRetries, default 1). When the window runs out, a bounded plain-HTTP refetch of the same URL runs through the page's own request context (challengeHttpFallback, default on) — the same cookie jar and the browser's UA, but none of Chromium's render decisions — because many edges serve the real document to such a request while withholding the interactive challenge from the browser session; the answer must itself be a non-challenge document or the fetch still fails. Failing that, the fetch fails with the distinct WEB_FETCH_CHALLENGE error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.
• Deep capture for viewport-virtualized feeds and app-shell documents — forum topics, comment threads, and timelines increasingly render only the items near the scroll position (Discourse topic streams keep every other post as an empty height-only placeholder that carries the post's id but none of its content), and app-shell document editors (Feishu docs and their kin) go one step further: the body never scrolls at all, blocks are keyed by data-block-id with no id, and off-window blocks are removed outright behind anonymous height spacers. A resting snapshot returns the skeleton or just the first screen while the bodies silently vanish. The fetch therefore runs one bounded in-page scroll walk before serialization: it walks the window or, when the document does not scroll, the page's dominant inner scroll container; it stashes what each rendered window shows (richest snapshot per id/data-block-id), steps to the bottom — the only place "the page has stopped growing" means anything — then rehydrates every degraded keyed element from its stashed snapshot, splices the removed ones back beside their surviving siblings, and restores rows that carry no DOM identity at all (react-window lists, X-style timelines) from the moment the engine removed them, in a structural clone of the page that it serializes itself (the live page is never rewritten). Pages that never react to scrolling — most of the web — are detected and skip the slow walk. Bounded by a built-in 20s budget; any failure silently degrades to the legacy capture.

How it works

Half	Location	Responsibility
Host (server)	src/	Registers the fetch provider (id playwright) into ctx.web; cordis.patch.yml pins the web seam's fetchProvider to it and enables the web_fetch tool with a 60s budget.
Browser (client)	src/client/	Registers the Playwright web fetch configuration form on the Plugins page's plugins.bundle.config / plugins.row.config seats, which stages edits and writes them through the shared settings form service.

web_fetch (tool-web)
   └─ ctx.web.fetchProvider = playwright
        ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
        ├─ cdp:   connectOverCDP(endpoint)
        ├─ PDF (inline / attachment / sniffed octet-stream) → bytes (navigation body, else request-context refetch) → pdf.js → layout heuristics → Markdown
        ├─ page.goto → settle (networkidle, best-effort) → deep-capture scroll walk (virtualized feeds; returns its merged, stripped capture when it restored anything) → otherwise strip scripting subtrees in the live DOM → page.content()
        ├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → trailing link-dense prune → DOMPurify → Turndown(GFM)
        └─ Markdown (or raw HTML when denoise is off)

Non-2xx statuses are results, not errors — the web seam's contract (@deepseek-ai/dsh-web): a fetched 404/418/500 comes back with its status code, exactly what dsh-web-fetch-http does. Chromium complicates this for status-only responses — an error status with an empty body (httpbin's /status/404) fails the whole navigation with net::ERR_HTTP_RESPONSE_CODE_FAILURE, and a 204/205 is aborted (net::ERR_ABORTED) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so web_fetch reports Fetched … (HTTP 404) instead of a WEB_PROVIDER_ERROR throw. The same recovery covers download responses — a body served with Content-Disposition: attachment (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: goto rejects with Download is starting and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; a PDF download is extracted to Markdown (see PDF links), and other undecodable downloads (a ZIP) still fail with WEB_UNSUPPORTED_CONTENT_TYPE. A recovered response that is itself a Cloudflare challenge edge still fails with WEB_FETCH_CHALLENGE; transport errors and cancellations stay errors.

Non-content scripting subtrees are removed in the live DOM before the document is read back — output-neutral by construction. The denoise pipeline (and the tool layer's own renderer on the raw-HTML path) re-parses page.content() with jsdom, and a script's text is the one thing that round-trip cannot trust: a page's runtime JS may rewrite a data script's text with a literal </script> embedded — measured on techcommunity.microsoft.com (Khoros/Aurora), whose client writes its whole Apollo cache back into script#__NEXT_DATA__ via JSON.stringify, un-escaping the wire's 712 \u003c sequences into raw < — and browsers serialize script text verbatim, so the flattened HTML string ends that script at the first embedded end tag and parses the remaining ~270KB of JSON as markup. The phantom elements that survive (unknown tags like <id>, anchors full of CSS and i18n strings) are exactly what Readability's scorer reads as article mass (measured: 126 points vs the real body's 45), so the badcase fetch of the RDP 10 AVC/H.264 article returned CSS rules and CachedAsset strings instead of the post. No string-level patch can disambiguate that serialization after the fact — only the browser's own parse holds the document's true structure — so ordinary script and template subtrees are stripped inside the live page (page.evaluate, best-effort) before page.content(). This changes nothing the pipeline ever returned: Readability deletes script/noscript nodes before extraction and scoring anyway, DOMPurify's allowlist drops script, and the tool layer's renderer removes SCRIPT/TEMPLATE wholesale — differential A/B over a real-page corpus (Wikipedia, MDN, Hacker News, the badcase page's wire HTML, plus noscript/JSON-LD/template edge cases) came back byte-identical on every page. Two deliberate carve-outs keep even the indirect consumers intact: script[type="application/ld+json"] is defused rather than deleted (</ rewritten to <\/ — still valid JSON, so Readability's metadata pass reads the same object and keeps the site-suffix-free article title, e.g. "Remote Desktop Protocol" instead of "… - Wikipedia"), and noscript is kept entirely because Readability's _unwrapNoscriptImages salvages the real image out of lazy-load <noscript><img …> wrappers. The strip also applies with denoise off, keeps the raw HTML serialization-safe for the tool layer's renderer, and shrinks what the pipeline parses (~70% on the badcase page).

Requirements

• DSH web profile (dsh web), Node.js ≥ 20.
• For the local backend: a Playwright installation with Chromium, a Chromium-family browser binary, or playwright-core with a browser in the default cache.
• For the CDP backend: any browser already running with --remote-debugging-port (e.g. chromium --headless --remote-debugging-port=9222).

Installation

From the npm registry (prebuilt — no build permission needed):

dsh plugin --profile web add dsh-web-fetch-playwright

From a GitHub repository (source — pnpm runs the prepare build; allowlist the package in profiles/web/pnpm-workspace.yaml if pnpm blocks the build script):

dsh plugin --profile web add github:chendefine/dsh-web-fetch-playwright

Or through the DSH plugin marketplace (设置 → DSH插件市场) — the repo carries the dsh-plugin topic and is indexed automatically.

After a bundle plugin is added to the profile layer stack, restart dsh web for it to load; uninstall with dsh plugin --profile web remove dsh-web-fetch-playwright and restart again.

Configuration

The Plugins page (侧边栏 → 插件) opens the form one click from the list: the dsh-web-fetch-playwright card lands on the package page with the full form between its description and its rows, and the row's own configure control opens the same form under the plugin's title. It edits the web-fetch-playwright settings section — derived from the plugin entry's Config schema — live:

Field	Default	Description
backend	local	Radio: Local Playwright or Remote CDP endpoint, each with its own nested input.
playwrightPath	(blank)	Local backend: path to a playwright executable or a Chromium-family browser binary. Blank = discover on $PATH, then fall back to the bundled playwright-core.
cdpEndpoint	127.0.0.1:9222	Remote backend: host:port, http(s)://… or ws(s)://….
shareBrowserContext	true	CDP only: Share / Isolate radio. true uses the remote default context and logins; false uses a separate context with the lifecycle determined by the scope below. Ignored locally.
contextIsolationScope	global	CDP isolation only: session creates a context on demand per DSH session and closes it when no active/pending fetches or open tabs remain (next fetch starts fresh); global reuses one fixed isolated context across all DSH sessions within this plugin instance.
denoise	true	Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert.
maxConcurrency	(auto)	How many fetches may render at once (1–200). Blank = backend default: 4 for local (each slot launches a browser) / 50 for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 10s they fail with WEB_FETCH_TIMEOUT and a hint to retry or raise this setting, rather than hanging until the tool budget aborts.
challengeRetries	1	Same-tab re-navigation attempts after a wait window runs out (0–3); any clearance cookies the browser earned stay in the context for the retry. Everything stays inside the 25s per-fetch waiting budget.
challengeHttpFallback	true	When the browser cannot clear a challenge inside its window, refetch the same URL once through the page's plain-HTTP request context (same cookies, the browser's UA, no render decision) and return that document when it is not itself a challenge. false keeps the browser-only posture.

Local backend resolution order:

1. The configured path (auto-detected as Playwright CLI or browser binary).
2. A playwright executable on $PATH (its package knows that installation's browser registry).
3. The bundled playwright-core — requires PLAYWRIGHT_BROWSERS_PATH or browsers in the default cache; otherwise the error suggests playwright install chromium.

Windows note — $PATH is scanned with the platform delimiter (;), but npm/pnpm global installs expose playwright as .cmd/.ps1 shims whose location does not walk up to the package root, so discovery may still land on step 3 (the bundled core). To drive a specific installation's browser registry, point playwrightPath at the playwright package directory or a browser executable.

CDP mode needs no local browser: the provider holds one shared connection for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a tab in the remote browser that closes when the fetch completes. Concurrency therefore counts tabs, which is why the CDP default is high (50). Unloading the plugin drops the shared connection (the remote browser itself is never closed).

The CDP context modes (share the browser's profile or not)

Choose Share the browser context (default) to open tabs in the remote browser's existing default context. Cookies and localStorage come from and are written back to that profile; the plugin never closes it. Resource filters and popup guards apply only to the fetch-owned tab.

Choose Isolate from the browser profile, then select:

• Per DSH session: create a dedicated CDP connection, isolated context and dedicated browser window on demand per DSH session ID. This also separates gateway implementations that map each connection's first context to a distinct Chromium process; different sessions never become nested contexts on one gateway connection. Concurrent fetches in the same session share that context/window in separate tabs; each fetch closes only its own tab. Context creation, tab creation, and tab-close/empty-check/context-close are serialized by a per-session FIFO mutex; fetching content stays concurrent and different sessions do not block each other. Once no fetch-owned or other open tabs remain, the context closes. A queued acquisition waits for that close before opening a fresh context. Cookies/localStorage reset after idle cleanup. The plugin verifies real CDP context and window IDs, matches newly created targets by ID, and fails explicitly if the endpoint cannot provide the isolation guarantees or the call lacks a session ID; it never falls back to the default profile.
• Global reuse across DSH sessions (default): all DSH sessions within this plugin instance reuse one fixed isolated context.

The plugin creates its own isolated contexts; it does not discover contexts created by other tools. Global isolation remains retained between fetches: completion closes only the fetch-owned tab, so cookies/localStorage survive subsequent fetches. Session isolation instead releases its context once no active/pending fetches or open tabs remain, as described above. This is in-memory reuse, not persistent storage across browser/plugin restarts; changing the context configuration retires old contexts, and teardown or connection loss/replacement discards them. Global isolation separates you from the browser profile, not from other DSH sessions within the same plugin instance.

If another CDP client deletes a plugin-owned isolated context, Chromium may not emit a Playwright context close event. On the explicit Target.createTarget “Failed to find browser context with id …” error, the plugin invalidates only that cached context and retries opening a tab once in a fresh context. Concurrent requests share the replacement; the deleted context's cookies/localStorage are lost. Other contexts and the default profile are untouched. Generic page errors are not retried by this recovery, and it never replays navigation or falls back to the default profile.

Profile-mode risk notes — it upgrades web_fetch from "anonymous read" to "acts as the browser's logged-in user":

• A fetched page that talks the agent into a GET-style state-changing URL (logout, settings change, API call) will send it with your session cookies.
• Concurrent fetches to the same site share one cookie jar; one fetch's logout or Set-Cookie affects the others.
• Output starts depending on the browser's history (A/B buckets, language preferences). The remote profile also keeps accumulating site data; the plugin never cleans it.

Persistent logins require a persistent user-data-dir. Headful (recommended — log in by hand once):

google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome-dsh-profile"

Headless server (pre-seed logins in a headful environment first): chromium --headless=new --remote-debugging-port=9222 --user-data-dir=/data/chrome-dsh-profile. Do not add --incognito or a throwaway user-data-dir — either defeats profile mode. Design notes and verified playwright-core facts live in docs/context-mode-profile.md.

Cloudflare challenge handling (bounded natural wait)

Some strict sites serve a Cloudflare interstitial before the real page. A real browser often passes the check on its own within a few seconds — but a fetch that only looks at the first response hands you the interstitial as if it were the page (the pre-0.2.5 behavior; from a repo checkout, after pnpm build — run node scripts/challenge-demo.mjs for a local simulated before/after, and node scripts/challenge-online.mjs <url> against a real site).

With the wait on (default):

1. Detection — a response carrying cf-mitigated: challenge (the documented signal for every challenge page type), or a 403/503 HTML document from a server: cloudflare edge, or the localized interstitial itself (title family like "Just a moment…", 请稍候…, "Минутку…", plus structural markers: the challenge-platform runtime's <script src>, #challenge-* elements, cf-chl-widget- frames, window._cf_chl_opt). The content-level markers are the fallback tier and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (server: cloudflare / cf-ray) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. Two refinements keep that promise for real pages: the challenge-platform script check reads real <script src> attributes only (prose, and escaped code samples that quote a path, are ignored), and it matches the runtime mount /cdn-cgi/challenge-platform/h/[b|g]/orchestrate/ positively. Everything else Cloudflare injects under that prefix — Bot Management's JavaScript Detections (scripts/jsd/, h/[b|g]/scripts/jsd/<hash>/, h/[b|g]/jsd/r/), and Precursor (scripts/precursor/) — is passive instrumentation present on every normal page of a protected zone (e.g. openrouter.ai, nodeseek.com), so it is never read as a challenge. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
2. Bounded wait, same tab and context — the fetch polls the live DOM (500ms interval) for the challenge to disappear while the browser runs its own verification; the last main-frame navigation response is tracked so the reloaded real document's status and headers are the ones reported. SPA-style clears (content swapped without any navigation) are caught by the same DOM probe.
3. Bounded retry — when a window runs out, the same tab re-navigates once (default; challengeRetries) with whatever clearance cookies the context already holds.
4. Dead-end refetch — before failing, one plain-HTTP request for the same URL through the page's request context (challengeHttpFallback, default on): the same cookie jar and the browser's User-Agent, none of Chromium's render decisions. Many Cloudflare edges serve the real document to such a request while withholding the interactive challenge from the browser session itself (measured live: v2ex.com from a datacenter IP — the browser drew an empty widget frame for 15s×2, the plain request drew the full article). The answer must itself be a non-challenge HTML document — checked at response level AND content level by the same classifiers that gate the browser path — or the fetch keeps the browser-only verdict.
5. Clear failure — WEB_FETCH_CHALLENGE (a provider-specific code the web seam's open-string code allows) naming the site, the budget spent, and the last challenge status.

Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in CDP isolated mode clearance remains within its session/global context until that context is discarded (local contexts still end with each fetch); in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by the built-in 15s wait and the 25s per-fetch waiting budget; nothing blocks forever.

Deep capture (viewport-virtualized feeds and app-shell documents)

Feed-shaped pages — Discourse topic streams (linux.do, meta.discourse.org), comment threads, timelines — render only the items near the current scroll position. At rest the live DOM holds a window's worth of real content while every item outside it is a height-only placeholder that keeps the item's identity but none of it:

<div class="post-stream--cloaked" data-post-number="1" id="post_1" style="height:297.9px !important;…">
  <h2 class="sr-only">由 Diver-QC 于 2 小时前 发布</h2>
</div>

A fetch that renders, never scrolls, and serializes one instant of that DOM hands back the skeleton — the screen-reader-only "posted by" headings survive the pipeline as if they were content, the bodies vanish, and even the denoise pipeline's coverage gate cannot notice (the page's own visible text is the same windowed fragment as the extraction). Scrolling to the bottom once does not help either: the render window moves — measured on a 40-reply linux.do topic, the resting DOM held only the first ~6 replies, a bottom jump left only the last 10, and replies past the first chunk were not in the DOM at all until a load-more sentinel near the bottom fetched them.

App-shell document editors have the same pathology one level deeper — the measured badcase is a Feishu wiki page (bocha-ai.feishu.cn/wiki/…). There the body never scrolls at all: html/body are locked at exactly one viewport and the document scrolls inside div.bear-web-x-container (measured: documentElement.scrollHeight 720 == innerHeight, window.scrollTo(0, 800) left scrollY at 0, while the container held 4534px against a 656px client). Its blocks are virtualized with two twists over Discourse: they are keyed by data-block-id and carry no id, and off-window blocks are removed outright — the whole un-rendered tail is one anonymous .bear-virtual-renderUnit-placeholder spacer with an explicit height — so there is no window scrollbar to walk, no id to stash by, and no degraded-in-place row to swap back. A fetch just returns the statically rendered first screen (measured: 7 blocks, 892 chars; the full document is ~190 blocks).

The fix is a bounded in-page scroll walk (src/autoscroll.ts), run after the settle and before the serialization strip for every HTML document:

1. Resolve the walk target: the window when the document itself scrolls (the Discourse shape); otherwise the page's dominant inner scroll container (the app-shell shape) — the element with the largest scrollable distance among computed overflow-y: auto|scroll elements, gated at half a viewport so dropdown-sized scrollers never trigger a walk. Neither scrollable means no walk.
2. Stash every keyed element — keyed by id, else data-block-id (namespaced so the two can never collide) — as a detached clone (framework-generated ids are not keys: Ember's ember123 and React useId values such as :r1:, «r1» and _r_1_ change on every render, so such elements take the unkeyed path below), keeping the richest sighting per key (a row first seen as its placeholder is upgraded when its rendered form appears), together with its nearest keyed siblings: where it sits in its own list. Entries are size-capped (~64KB each): a virtualized row is a few KB, while an element tens of KB big is a container whose serialization embeds the whole stream (measured on linux.do: six nested wrappers plus data-preloaded and svg-sprites ballooned one pass to 1.9MB of the 2MB budget); such keys are remembered and never serialized again.
3. Walk down from the top — 70% of the target's viewport per step, each a smooth 300ms-animated descent (time-interpolated ticks; scrollTop drives an inner container exactly the way window.scrollTo drives the document) plus a 300ms quiet settle for the feed engine's debounced uncloak — until the walk can hold at the bottom through a 2.5s patience window with no height growth (real feed engines fetch their next chunk with network latency; growth restarts the clock), or the step/budget caps close it. A MutationObserver watches for the walk's whole reason to exist — the page adding or removing elements in reaction to the scroll. After two careful steps with no reaction the page is treated as scroll-inert and the rest is probed with one jump to the bottom; any reaction after that jump (an uncloak, a re-cloak, a load-more append) rewinds the walk to where it jumped from and it finishes carefully. Correctness never rests on the warm-up guess, only the cost of an inert page does. The same observer harvests unkeyed rows: when the engine removes an element that has no key, the record still hands over the detached element whole, plus the sibling nodes it sat between, and that is where the merge puts it back. No layout is read, so absolute positioning and transform do not matter. Each row identity is stashed once per list (an engine that re-renders every frame removes the same row thousands of times; later sightings only anchor), from its own character budget so it can never crowd out the keyed stash. Transient UI is never harvested (role=status|alert|dialog|tooltip|progressbar|menu|…, aria-hidden, aria-busy, elements without text). A page that never reacted and issued no fetch/XHR holds only 900ms at the bottom instead of the full 2.5s patience. Measured: a ~30-viewport static article costs ~2.4s instead of exhausting the 20s budget, and a page that does not scroll pays nothing.
4. Merge into a structural clone (document.cloneNode(true)): no serialize/re-parse round trip, so a runtime-rewritten data script with a literal </script> in its text cannot corrupt the merge, and — having no browsing context — the clone runs no scripts, upgrades no custom elements, and loads no frames. Splice (container walks): stashed keys from inside the scroll container that the clone no longer holds were removed by the engine (the Feishu shape, whose placeholders are anonymous) and go back next to their recorded sibling, resolved iteratively so a removed run chains off whichever end survived; a key that rode in inside its spliced parent is skipped. Placement is structural, so it holds whatever order the walk saw rows in — including an anchored open that starts mid-document. Swap: every keyed element emptier than its snapshot (fewer child elements, or meaningfully less text — the Discourse placeholder) is replaced by it, except containers (ancestors of another element being swapped), which are assembled from those descendants instead — a container's own snapshot is one window's partial view. Restore (unkeyed rows, before the swap): a list qualifies only if the engine both removed text-bearing rows from it and added elements to it during the walk. A lone toast or a load-more append does not qualify, and neither does anything inside an aria-live="assertive", role=alert|status|timer|marquee or carousel region. Each removed row the list no longer holds goes back next to the sibling it was removed beside. An anchor that was itself removed is followed to its neighbor, and a repeat sighting resolves to wherever the row's stashed copy lands. Every sighting of a row is tried, so the first window of an anchored open is placed by the later pass that saw it beside its neighbors. A list re-rendered wholesale (innerHTML) leaves no surviving anchor; its rows go before the first text-bearing row, in removal order, because a top-down walk that ends at the bottom only loses what lay above the final window. De-duplication against the rows still present uses a per-list identity attribute when the engine sets one (data-index, data-item-index, aria-rowindex, aria-posinset, data-key, data-id, …, scoped to the list so repeated values across lists cannot collide), else the row's text. A present row with the same identity attribute that is far emptier is a placeholder and is replaced. An unkeyed stub the engine put in a row's place (short text the row contains, right beside it) is absorbed. A restored list and its ancestors count as containers for the swap, so a keyed wrapper is never replaced by its one-window snapshot. The clone is then stripped of scripting elements (the same src/strip.ts routine the live read uses) and serialized; the provider reads that string instead of the live page. If a page reacted but nothing could be restored (rows without text, canvas rendering), the walk scrolls back to where the page opened and waits for it to settle before the live read. Otherwise the provider would read the walk's last (bottom) window instead of the opening screen the legacy capture saw. A scroll-inert page is left where it is, since its DOM is the same either way.

The walk runs in both denoise modes and is bounded by a built-in 20s budget and by the fetch's remaining waiting budget; a walk cut short by the budget still merges and returns what it stashed, and the content read plus decode run in the 10s extraction grace window after it. Any failure — a page without evaluate, a rejected evaluation, an engine that refuses the walk — silently degrades to the legacy capture; a walk can only add content, never fail a fetch. In CDP profile mode the walk scrolls the fetch-owned tab in the remote browser, exactly where the fetch already renders it.

Measured on the live badcases: the linux.do topic (51 replies, ~18kpx) walks fully in one 20s budget with every post rehydrated and zero cloaked placeholders left; the Feishu wiki page walks its inner container in ~11.5s of the 20s budget (15 steps, 38 removed blocks spliced back, 191 blocks in the final DOM — the resting capture had 7) and the deep tail (both JSON examples, the full passages) lands in the markdown. On the public react-window demo (/list/fixed-row-height, 1,000 unkeyed rows), the walk previously returned only its final window (rows 129–142); the unkeyed restore returns rows 1–155 in order with no duplicates, stopping where the 20s budget ends. Remaining limits of the unkeyed path: rows that hold only images or render to canvas cannot be restored (the walk rewinds instead), and engines that recycle row nodes in place (rewriting a pooled node's text rather than removing it) are not harvested. One residual limitation, documented as measured: Feishu table blocks lazy-render their cells behind a second, in-block virtualization the walk does not chase — table headers and rows are restored, very large payload cells may stay empty.

PDF links

When the fetched URL turns out to be a PDF, the provider extracts it instead of failing with WEB_UNSUPPORTED_CONTENT_TYPE:

How the PDF arrives	What the browser does	How the bytes are obtained
application/pdf (inline)	headless Chromium downloads it; a headed or CDP-attached Chrome opens its viewer	navigation body when it is the PDF, else a refetch through the page's API request context
Content-Disposition: attachment	goto rejects with Download is starting	refetch through the API request context
application/octet-stream (and kin), .pdf path or filename	download	refetch, then the %PDF- magic decides

The refetch uses the same browser context, so cookies and logins (CDP profile mode) apply to gated PDFs. The bytes go through pdf.js (unpdf's serverless build, lazily imported — plugin start never loads it) and a layout pass (src/pdf.ts) that turns positioned text into Markdown: the largest first-page type becomes the # title (else the PDF's metadata title), headings come from the outline, numbering, larger type, or standalone bold lines; running headers/footers and page numbers are removed; paragraphs are rebuilt and joined across columns and pages; bullets, numbered items, monospace blocks, and column-aligned rows become lists, fenced code, and GFM tables; each page starts with a <!-- page N --> marker.

Limits and errors: PDFs over 50MB fail with WEB_PDF_TOO_LARGE (checked against Content-Length before downloading); reading stops after 500 pages, 100k collected characters, or the fetch deadline, and the result is flagged truncated with a note naming the last page read. A scan with no text layer fails with WEB_PDF_NO_TEXT (OCR is not supported — PDFs that carry an OCR text layer extract normally), a password-protected PDF with WEB_PDF_ENCRYPTED, and bytes pdf.js cannot parse with WEB_PDF_INVALID. Known limitations: tables without consistent column gaps (and figure labels) come out as plain text, multi-column layouts rely on the PDF's content order, and bullets drawn as vector shapes (rather than glyphs) are not recognized as list markers.

Development

pnpm install
pnpm typecheck   # tsc --noEmit
pnpm test        # vitest run (browser smoke self-skips without a browser)
pnpm build       # tsc declarations + tsdown (host ESM + client module-registration bundle)

Run the real context/window isolation, cookie/localStorage separation, concurrent provider identity, create/close race, and external-context-deletion regressions (the tests launch their own browser and local server, never the configured user browser):

PLAYWRIGHT_TEST_EXECUTABLE_PATH=/path/to/chromium pnpm exec vitest run tests/cdp-recovery.browser.spec.ts

Without an override it uses Playwright's default executable and reports skipped tests when that binary is absent; an explicit invalid path or a launch failure fails the test. Set PLAYWRIGHT_TEST_HEADED=1 with a working DISPLAY to verify real headed windows as well. A source build does not reload an already running DSH server: restart/reload the plugin host before judging the new behavior.

For an explicitly authorized connection-scoped gateway, run the opt-in regression below. It creates only test-owned contexts, fulfills a .invalid probe locally, checks cookies/localStorage and idle disconnect, then cleans up its resources. Numeric window IDs are process-local and may repeat across cloned Chromium processes; they must not be compared across gateway connections.

PLAYWRIGHT_TEST_CDP_ENDPOINT=http://your-gateway:9222 pnpm exec vitest run tests/cdp-gateway.browser.spec.ts

Repository layout:

src/
├── index.ts               # host entry: registers the fetch provider (the settings section derives from the plugin's Config schema)
├── config.ts              # schemastery schema, CDP endpoint normalizer
├── provider.ts            # WebFetchProvider: navigation, deadline, semaphore, caps
├── markdown.ts            # denoise pipeline (Readability + DOMPurify + Turndown/GFM)
├── playwright-resolve.ts  # local backend discovery (path / $PATH / bundled core)
├── types.ts               # structural Playwright types (runtime module discovered dynamically)
└── client/                # browser half: settings card, form model, locales
tests/                     # unit + provider + browser integration (self-skipping)

See CONTRIBUTING.md for the development and release workflow, and SECURITY.md for the security model and reporting policy.

Security

Same stance as the built-in HTTP provider: no SSRF / private-network protection is implemented — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself — but note that in profile mode the requests (and any state changes a malicious page tricks the agent into) carry the remote browser's logged-in sessions (see the risk notes above).

License

MIT © 2026 chendefine