dsh-screen-eye

An autonomous eye for DeepSeek Harness on macOS and Windows: the agent captures the screen and receives the picture in the same tool call, so it can look at a running app, a dialog, an error or its own UI work without asking you for a screenshot.

No native build step, no bundled binary, no dependencies.

What it adds

Two model-callable tools:

Tool	What it does
screenshot	Captures the screen and returns the image itself, as an image content block the model can see.
screen_permission	On macOS: reports whether this process is currently allowed to capture, and with action: "guide" opens the exact System Settings pane and names the entry to switch on. Not registered on Windows, which has no such permission to report.

mode selects what is captured: screen (default, the main display), display (one display, by index), region (a rectangle, whose origin is the top-left of the main display, so a monitor placed to the left of or above it takes negative coordinates), displays (which captures nothing and lists the connected screens with the index display expects and, on Windows, where each one starts), or the interactive window / select, which wait for the user to click a window or drag a rectangle. Windows has no system region picker, so there select is refused with that explanation and window means the window already in front.

Set frames above 1 and the call takes that many captures interval_ms apart and returns them all, which is how something that changes over time can be seen. This is deliberately not a GIF: the harness stores images single-frame, so an animated GIF arrives as its first frame.

Both knobs are yours to set, and the useful direction is not always "finer". How fast frames can be taken depends on the area — 155ms for a whole 4K screen but 56ms for a 1200x800 region — so a short component animation is resolved by capturing the small area it happens in, not by asking for a finer interval over everything. The reply reports the spacing actually achieved, and says so when the request could not be met. docs/motion.md has the measurements and the reasoning.

interval_ms is the knob that decides both cost and resolution: it is how finely the motion is sampled, and the frames are what a burst costs. Nothing here imposes a duration — the burst is (frames - 1) x interval_ms long, and when the call waits for a change the picture decides where it ends — so the only ceiling is timeout_ms, the budget for the whole call, which defaults to five minutes and can be raised per call. The frame ceiling is the provider's own 600 images per request rather than a number this plugin picked: each frame is one image and at most 384 vision tokens (about 380 measured), which makes the count a cost decision the caller is better placed to make than the plugin is, and the tool description spells the arithmetic out for it.

Every other call returns exactly one image. That is a deliberate constraint rather than a limitation of the system: screencapture writes one file per screen, so an unqualified capture on a multi-display Mac would produce several files while this pipeline resolves and reads a single path — leaving the others behind under names nothing here chose. The default is therefore pinned to one display, and display selects another.

Why this plugin exists

Most screenshot tooling assumes the hard part is capturing pixels. On macOS the hard part is permission, and it fails in a way that looks like a bug:

screencapture: could not create image from display

macOS gates screen capture behind the Screen Recording permission, keyed to the responsible process — the application macOS holds accountable for a whole process tree.

What decides everything here is whether there is an app identity to attribute the request to, and it leads to two different paths:

• When the host is an app — the desktop client, for instance — macOS shows a system prompt once. But the prompt only adds the app to the list, and the entry it adds is switched off: captures still fail until the user turns that switch on by hand.
• When the host has no app identity — the in-app plugin market restarts the host through a detached helper, so it is reparented to launchd with no application above it — macOS cannot attribute the request and refuses silently, without ever showing the prompt.

Either way there is one thing that fixes it: the switch being on. That cannot be done programmatically — the TCC databases are SIP-protected and tccutil only resets. So this plugin does what is actually possible:

• it detects the denial by attempting a real capture and classifying the result, rather than guessing;
• it names the entry to look for — the principle, plus the common cases — because which row holds the grant depends on how DSH was started and cannot be computed from inside the process (see the table further down);
• it opens the exact settings pane on request;
• and it returns those steps as the tool result, so the agent can hand you a fix rather than a stack trace.

Windows has no such gate, and no such story: the hard part there is that the capture can succeed and still be useless — a downscaled copy of the screen if the process is DPI-unaware, or a black frame if it is not attached to the interactive desktop. Both are answered in Windows below.

Install

dsh plugin --profile web add github:davidekingsss/dsh-screen-eye
# then restart dsh

The same command installs it on macOS and on Windows; the platform layer picks the engine.

From a local checkout, instead of a published source:

dsh plugin --profile web add -w link:/path/to/dsh-screen-eye

The plugin has no build step and no dependencies of its own, so nothing compiles at install time. On macOS a small helper is compiled later, on first capture, from source that ships with the plugin — see How it works.

Grant Screen Recording (macOS, once)

Call screenshot once. If permission is missing, the result tells you exactly what to do, and screen_permission with action: "guide" walks through it: it checks first, opens the pane only when the grant really is missing, and names the entry to look for.

Three states, only one of which is "never granted"

All three were produced and observed on macOS 26.6.2, because they behave differently:

state	system prompt	in the list	captures work
never requested	yes, on the first request	after you open Settings	no
granted, then switched off	no	yes, switch off	no
removed from the list	yes, on the next request	after you open Settings	no

The entry is named after the app that started DSH

macOS names the entry after the app bundle the request is attributed to — the app that launched the harness, not the harness. Measured on one machine, three launchers gave three different names:

how DSH was started	entry in the list
the Shortcuts droplet (运行Deepseek Harness.app)	运行 Deepseek Harness
a browser	Google Chrome
a terminal	Terminal / iTerm

So there is no single correct name to print. The tool states the principle and lists the common cases; all you have to do is match it in the list.

That 运行Deepseek Harness.app is a macOS Shortcuts droplet (CFBundleName is ShortcutDroplet). The Shortcut starts the harness detached and then exits, so it never appears in the process tree — the name exists only in the attribution macOS recorded.

The real sequence: four steps, and no retrying in between

1. Watch for the system dialog — "…wants to record this computer's screen and audio." If it appears, click Open System Settings: it adds the app to the list for you. If it does not (started from a terminal, or previously denied), open System Settings → Privacy & Security → Screen & System Audio Recording yourself.
2. Find the entry for the app you started DSH with — see the table above.
3. Turn its switch on. ⚠️ An entry the system added is OFF by default, and being in the list is not being granted. This step is yours to do.
4. Then call the tool again. No DSH restart is needed — the grant applies to the next capture.

Why step 3 has to be spelled out: doing anything between step 1 and step 3 — judging, retrying, reporting a failure — is wrong. Until the switch is on, every capture fails the same way, and that is not a fault; it is a sequence that has not finished yet.

macOS may periodically ask you to re-confirm this permission. Switching the same entry back on is enough.

Windows needs none of this — see Windows.

Configuration

All keys are optional — and all of them are editable from the harness's own settings page, Settings → Screen Eye, or by hand in ~/.dsh/settings.yaml under screen-eye:. Either way an edit reaches the next call without a restart, and the plugin's mount entry stays the base that a cleared field falls back to. docs/settings.md has the three layers, the page, and the two platform constraints behind it.

Key	Default	Meaning
outputDir	<pictures>/Screen Eye	Where captured PNGs are written — the system pictures folder, in a folder of its own so fifty screenshots do not land among the user's photographs.
locale	en	Language of the onboarding text: en or zh.
timeoutMs	300000	Budget for one whole call, including any wait_for_change. The capture gets what the wait did not spend.
frames / interval_ms	1 / 200	Frames per call and the target gap between them. At most 600, which is the provider's per-request image limit rather than a policy here — past it the extra images would be taken and then replaced with a placeholder. Each frame is one image and at most 384 vision tokens (about 380 measured), so the count is a cost decision and it is yours.
duration_ms	—	State how long the motion lasts and let the tool size the burst, instead of computing frames and interval yourself.

frames, interval_ms and duration_ms describe one burst and any two of them determine the third: frames with an interval gives the span, frames with a duration gives the interval that divides it, an interval with a duration gives how many frames fit. All three at once is refused rather than resolved.

That is also the cost lever. Each frame is an image and images are what cost, so holding the window fixed and raising interval_ms trades resolution for cost — the same motion watched with fewer images — while lowering it spends more for a finer sample. A window given alone is sampled at the ordinary frame count, not at the maximum, because the maximum is the most expensive answer and was not asked for — unless the call is waiting for a change first, where the ending is the screen's to decide and the count is headroom.

wait_for_change: true also settles when the burst ends: it stops once the picture stops changing, so frames is an upper bound and the reply says which ending happened. wait_timeout_ms (default 30000) is how long it waits for the motion to start. until_still: false opts out, recording the full window instead — the right call for a span rather than an event. | maxDimension | 4096 | Largest side, in pixels, a capture may have. 4096 is where the provider's per-image side limit lands once a request carries fifteen or more images, and a burst can carry hundreds. A capture over the cap is refused with its size named rather than resized — so a display larger than this needs the field raised to be captured whole. | | keepRecent | 50 | How many of the newest captures to keep in outputDir. A capture is a few-megabyte PNG and an agent using its eyes takes many, so the directory is bounded by default. 0 keeps everything. | | requireImageCapableModel | true | Refuse a capture when the calling model declares no image input, instead of returning a picture it cannot see. | | deleteAfterCommit | false | Delete the PNG once it is committed to the attachment store. Off by default, so the returned path stays re-readable. | | announceCapability | true | Put a standing line in the system prompt saying the agent can look at the screen, and register the screen-eye skill. Off leaves the tools defined but unannounced. See Telling the model it has eyes. |

Captures are committed without the records that would disqualify them from lossless storage. On macOS that means stripping the ICC profile, EXIF and iTXt records screencapture attaches to every shot, while the file on disk keeps them; those records decide whether the attachment store keeps the bytes or re-encodes them, so removing them is what lets a capture be stored as written instead of as WebP at quality 85. On Windows there is nothing to strip: GDI+ writes none of them, so a capture already satisfies the store's condition. The store converts to sRGB either way, so nothing that survives is lost, and the file you can open keeps its colour profile.

Retention only ever removes files this plugin wrote: regular files, direct children of outputDir, whose names match the exact shape it generates (shot-<timestamp>-<suffix>.png). It is never recursive, it never touches another naming scheme, and it never removes the capture it just returned.

# cordis.patch.yml
- insert:
    - id: screen-eye
      name: dsh-screen-eye
      config:
        locale: zh
        outputDir: /Users/me/Pictures/agent-shots
        keepRecent: 200

How it works

screenshot tool ──▶ lib/capture.mjs ──▶ the platform engine ──▶ PNG
                       │                   macOS: screencapture
                       │                   Windows: PowerShell + System.Drawing
                       └──▶ attachments.saveImage() ──▶ image content block ──▶ model

Captures are not resized to fit maxDimension, and the cap is deliberately not set lower. Both choices come from the same measurement: the harness projects every image to a route-level pixel budget before the model sees it — 640,000 pixels by default, about 1066x600 for a 16:9 screen — so a capture from a 4K, 5K, 6K or 8K display arrives as the same image. Below that budget a smaller cap cannot save a token, and resizing here would insert one more scale between what the model measures in the image and the screen coordinates region expects, which is the mapping the zoom workflow depends on.

Neither engine ships a binary: each is built on the machine that runs it. On macOS the engine of record is the system's screencapture(1) — Apple-signed, already using ScreenCaptureKit internally, and requiring no build — with a resident helper compiled once from engine.swift for the speed a per-call process cannot reach (a change check costs 22.6ms through the helper against 55.8ms through the binary). Compiling it rather than shipping it is the same reasoning as Windows': a prebuilt binary would need per-architecture builds, a fixed deployment target, and an ad-hoc signature whose hash changes on every rebuild — and a changed hash silently invalidates the user's Screen Recording grant. On Windows the engine is Windows PowerShell 5.1 — present on every install — driving System.Drawing through a shim compiled in memory for the length of one call.

The macOS helper is an optimisation, never a dependency: screencapture remains the engine of record and every failure except a denial falls back to it, so a machine with no Swift toolchain captures exactly as it always did.

The image reaches the model through the same attachment path the built-in read_image tool uses, so the value is validated, downscaled and replayed exactly like any other image in the session.

Telling the model it has eyes

A tool reaches a model through two channels that do not talk to each other: its schema, sent with every request, and a standing line in the system prompt. The schema answers how do I call this — but only for a caller that has already decided to look it up. Nothing reads it while the model is still choosing what to do, so a plugin that ships only a schema is a plugin the model has to guess at.

That is not a theory about this plugin; it is what this plugin measured. Across the sixty sessions in this machine's local store, fifty-four mention screenshot exactly once, and every one of those mentions is the same sentence — the Web surface's "the browser provides no implicit DOM, route, or screenshot context". Nine sessions ever called the tool, and six of those nine were sessions spent building or testing this plugin: two in this repository, four in temporary scratch directories on the day the first commit landed. Three sessions used it for their own work. Silence in the prompt, and a tool nobody reaches for.

So the plugin registers a section, once, beside the tool it describes:

This deployment can see the screen, and seeing is a way of finding things out rather than a last resort: when the answer is on screen rather than in a file, look instead of searching. Read text out of a running app's interface — a window title, a field value, a dialog, a message — by observing that app, whose accessibility tree hands it over as text; keep the screenshot tool for visual facts with no text to read, such as layout, colour, motion, and anything you are judging by eye. A window too small, collapsed, or covered to answer from is a reason to expand it, scroll it, or move it back into view, and then look — not a reason to go looking for the same information in caches, databases, files, or APIs. The screenshot tool captures and returns the picture in that same call, so no read_image step follows, and a burst of frames records motion instead. A capture can need Screen Recording permission; screen_permission reports whether it is granted and opens the pane that grants it.

The routing sentences were not in the first version, and a second measurement is why. Asked what video was playing, a session answered correctly and expensively: sixteen shell commands into a media app's cache database and two web API calls, to recover a title printed in the window's own title bar. Nothing had said that a running interface is a place where answers live — the line said "on screen rather than in a file", and the model read on screen as in a picture. So the two channels are named now, and so is the third case: when the view itself is the obstacle, changing the view is a step toward the answer rather than a detour away from it.

Four things about the section are deliberate:

• It is registered only when the tool mounted. A deployment with no attachment store registers no capture tool, and a prompt that advertised one would send the model after a call that returns nothing.
• The consent sentence is platform-asked. Windows registers no screen_permission and has no grant to report, so that sentence is not there. The question goes to the engine registry, the same one apply() asks.
• The skill is the long form. A screen-eye skill carries the two-channel split, why a cache is the wrong route for something already displayed, the mode table, what a scaled display means for coordinates, and what to do when nothing comes back — registered at runtime, so installing the plugin is the whole installation and no skill file can drift from the code that describes it.
• announceCapability: false removes the line, and the line can be switched back on without a restart: its text is a provider evaluated at each assembly rather than a string captured at mount. The skill registration cannot be walked back once made, so that one follows the setting as it stood at mount.

Keeping the tools without the standing line is a supported deployment, not a degraded one — it is what you want if the line ever reads wrong on your setup. There is no switch that unregisters the tools, because a plugin whose entire purpose is a tool the agent calls by itself is not improved by hiding it.

Platform support

macOS and Windows, enforced in two places: the bundle patch carries disabled: !!js process.platform !== 'darwin' && process.platform !== 'win32', so on any other platform the module is never imported, and apply() re-checks so a direct mount cannot register capture tools that have no engine.

Windows

Windows needs no permission and no onboarding — any process attached to the interactive desktop may capture it — so screen_permission is not registered there at all, and screenshot is the whole tool surface. Four things are genuinely different from macOS, and each is answered rather than documented away; docs/windows.md has the measurements.

• Scale. Windows PowerShell is DPI-unaware, and a DPI-unaware process gets a downscaled copy of the desktop: on a 3840x2160 panel at 125% it reports and captures 3072x1728. The engine declares per-monitor awareness before it reads anything, and captures the true pixels.
• A blind session. A process that is not attached to the interactive window station does not fail — CopyFromScreen returns black, which a naive engine reports as a successful capture of a black screen. The engine checks the window station and session first and refuses by name, and a frame that is black everywhere is returned with a note saying what that usually means.
• Bursts. A Windows capture costs about 380ms cold, almost regardless of area, because the engine pays a PowerShell start per call (148ms of that with nothing to do at all). It is started once and kept resident instead: the C# shim is compiled once per machine, and a warm capture costs 16-22ms. Taking a six-frame burst as six cold calls would sample a 400ms animation over two and a half seconds, so Windows takes the whole burst in one engine call and the interval in the plan becomes reachable.
• Animations that run once. A component transition plays once, and a model cannot issue its call at the moment the user clicks — measured, a burst issued as a 300ms transition begins gets zero usable frames on either platform. wait_for_change: true is the answer: "I am watching this rectangle; take the frames from the moment it moves." On a known 300ms transition the frames began 103ms after it started and 4-5 of 8 landed inside it. macOS has the same option, and since 2026-09-16 the same kind of resident helper — a check costs 22.6ms there against 55.8ms through screencapture, which is the difference between recording the last third of a 300ms transition and all of it (docs/macos-findings.md). The other end is settled by the screen too: a burst that waited for a change ends when the picture settles, so frames is an upper bound rather than a guess about how long the motion takes — measured on the same transition, 7 frames covering it and stopping there, with 3 of the 10-frame cap unused. The reply says which ending happened, and until_still: false records a fixed window instead for a caller who wants a span rather than an event.
• Two screens. Displays are listed main-first with their origins, so a region on a second screen — including one to the left of the main display, where x is negative — addresses the right pixels. Verified on a 3840x2160 main screen with a 2560x1600 screen at x = -2560, by comparing captures against each other pixel for pixel.
• select. Windows ships no system region picker, so that mode is refused with the reason and region suggested; window captures the window already in front, since there is nothing to click either.

Requirements

• macOS or Windows, with the harness's Node runtime. Neither capture path needs an extra package. Nothing is compiled at install time; on macOS a small helper is compiled from source on first capture, and the plugin captures without it if no Swift toolchain is present.
• On macOS, Screen Recording permission for the process running the harness (see above). Windows needs no grant.
• A model route that declares image input. With requireImageCapableModel left at its default, a route declared text-only is refused up front with a message naming the model, instead of silently capturing something the model cannot see. That declaration lives in the harness's model metadata, not in the model: settings.yaml can override a built-in entry, and an override listing only text will refuse captures for a model that can see perfectly well. The refusal names the model and the setting, so the fix is one line — and it takes effect without a restart.

Development

npm install
node test/selftest.mjs          # everything: 160 cases, minutes
node test/fast.mjs --list       # the sections, with their case counts
node test/fast.mjs "plugin wiring" "announcing the capability"

test/selftest.mjs is the thing that ships and the thing to run before a commit. It is also minutes long, and one section is why: live capture takes a real screenshot, records a real burst and builds the Swift engine for each of its 24 cases. The other 136 cases are pure logic and stubs and finish in about six seconds, so test/fast.mjs runs just the sections you name — the suite's own cases, read at run time, with the helpers they need pulled in automatically. It refuses to pull in live capture silently, because that is the run that would look fast and not be.

That split is worth having on purpose. A killed run is not a neutral event: the suite creates a temporary directory for its captures and removes it at the end, so a run cut short at the wrong moment leaves PNGs of the user's screen behind. The full suite has been interrupted that way repeatedly while this plugin was being written. So live capture now sweeps first: it removes any dsh-screen-eye-test-* directory in the system temporary folder whose lock file names a process that is no longer running. The lock names a pid rather than merely existing, because a lock that only said "a run made this" would protect the interrupted run forever — which the first version of the sweep did, measured rather than assumed.

The suite runs without a harness: the logic modules are imported directly and the tool definitions are exercised through a stubbed context, so it works on a machine that has never seen the harness — which is also what CI does, on macOS and on Windows. Cases that capture for real run only when the machine can actually see its own screen — Screen Recording granted on macOS, a visible desktop on Windows — so the suite stays green before the grant too.

Cases about one platform's own model are asked of that model rather than of the host, so the Windows engine's script is asserted on macOS CI and the macOS permission text is asserted on Windows. That is deliberate: a seam is only worth having if something checks the other side of it.

The three @deepseek-ai/* packages the plugin imports are pinned exactly in devDependencies, and that is deliberate: they publish the current line under the next dist-tag while their latest tag still points at a much older release, so an unpinned install resolves to the old one and the import fails.

docs/verification.md records what has actually been run — the self-test on both platforms, loader acceptance in an isolated profile, and end-to-end agent turns — and what each result does and does not prove.

docs/macos-debugging.md is a self-contained runbook for verifying the macOS half on a Mac, written to be handed to an agent that has no context: eight steps in priority order, each saying what to run, what the answer should be, what a deviation would mean, and which numbers to report. tools/motion-fixture.html is the animation it uses: a 300ms transition that fires on its own schedule, so nothing has to be clicked while a burst is waiting.

docs/macos-findings.md is what that runbook produced when it was run on a Mac on 2026-09-16. It is worth reading before trusting any macOS number anywhere else: it confirms the region mapping is exact and the still check sound, and it contradicts the claim that macOS's change check leaves "four or five frames of a 300ms transition" — measured, it left one or two, because each check paid a whole screencapture process start. That measurement is why macOS now keeps a resident helper, and why the plugin is not quite as free of compiled code as it used to say it was: the helper is built from source on the machine that runs it, inherits the Screen Recording grant rather than asking for its own, and falls back to screencapture on every failure.

License

MIT