@zibokapi/dsh-codex-computer-use

English | 中文

A standalone plugin for the DeepSeek Harness: desktop computer use on macOS — app listing, key-window capture as a screenshot plus model-readable accessibility tree, and synthesized mouse and keyboard input — cloned from OpenAI's Codex Computer Use window API. Input delivery is background-first like Codex's own execution model: semantic AX actions need no focus, and raw events ride the private SkyLight path (SLEventPostToPid with the keyboard authentication envelope, field stamps, and focus-without-raise records, ported from the MIT-licensed trycua/cua project) with a public CGEvent.postToPid fallback per symbol, so the user's foreground never changes. It needs no OpenAI components.

The design is accessibility-tree-first: computer_use_get_app_state returns a numbered, tab-indented element tree; the model acts on element indexes, with window-relative coordinates as a fallback, and later captures of the same app return diffs instead of the full tree.

Everything ships in this one package: the seam, the local Swift-daemon provider, the computer_use_* tools, the approval policy, and a standalone MCP server. The complete feature delta against OpenAI's implementation lives in docs/codex-parity.md; that reference is the parity checklist.

Install

This repository is a DSH bundle: the root package.json declares dsh.bundle.patch → cordis.patch.yml, which inserts the three host rows under this package's own subpaths (@zibokapi/dsh-codex-computer-use/*). One install brings everything.

# 1. install the plugin into a profile
dsh plugin --profile <name> add @zibokapi/dsh-codex-computer-use

# 2. build the macOS daemon and grant it permissions (once per machine)
npx @zibokapi/dsh-codex-computer-use

# 3. (re)start the profile — the computer_use_* tools are live
dsh --profile <name> web

Step 2 is the only macOS-specific part. It builds the Swift helper from the native/ directory shipped inside the installed package (Xcode Command Line Tools required — xcode-select --install), bundles and signs it into ~/.dsh/computer-use/dsh-computer-daemon.app, then explicitly asks for the two permissions the daemon needs — Accessibility and Screen Recording — through the macOS system dialogs, polling until the grants land. It writes no configuration and none is needed: with helperPath unset, the engine falls back to that exact path.

The install location is stable on purpose. macOS TCC keys each grant on the helper's bundle id, code signature, and on-disk path; a path inside node_modules would change with every plugin update, so the daemon lives under the DSH home ($DSH_HOME/computer-use) and the grants survive plugin updates.

dsh re-asks for the grants whenever the plugin loads: a missing or revoked grant simply means computer use is not loaded until it is granted (the daemon opens the matching System Settings pane on a remembered denial). dsh-codex-computer-use status reports the install at any time.

The registry tarball ships lib/, and lib/ is committed to git, so every install form — npm, a git URL, a packed tarball — works without running any build script (pnpm ≥10 blocks lifecycle scripts of git-hosted dependencies by default, so the built entries ship in the repository itself).

Lazy path

Tell your dsh:

Install this plugin package: https://github.com/geohotstan/dsh-computer-use

then run steps 2–3 above (npx @zibokapi/dsh-codex-computer-use, restart the profile).

Manual (from a checkout)

git clone https://github.com/geohotstan/dsh-computer-use
cd dsh-computer-use
pnpm install
pnpm exec dsh-codex-computer-use   # build + install the daemon, ask for permissions
pnpm run build:native              # optional: a Developer ID / self-signed build, see Permissions
cd <where you run dsh>
dsh plugin --profile <name> add ../dsh-computer-use

dsh plugin add registers the repo as a bundle layer in the profile (dsh.profile.bundles). The engine resolves the daemon in order: the helperPath config → the DSH_COMPUTER_HELPER_PATH environment override → the setup CLI's install location (~/.dsh/computer-use/dsh-computer-daemon.app/Contents/MacOS/dsh-computer-daemon). Set helperPath only to point at a custom build.

Entries

One package, five subpath entries (plus the invariant companions under ./<entry>/invariant):

Entry	Role	Loader row
./computer	Service Definition — ctx.computer	registered by computer-local
./computer-local	Local provider — resident Swift daemon (AX tree, screenshots, CGEvent input)	@zibokapi/dsh-codex-computer-use/computer-local
./computer-tools	The computer_use_* tools plus the computer-use skill	@zibokapi/dsh-codex-computer-use/computer-tools
./computer-policy	Per-app approval gate + Codex-style tier guidance + computer_use_list_granted_applications	@zibokapi/dsh-codex-computer-use/computer-policy
./computer-mcp	Standalone MCP stdio server exposing the same surface for external MCP clients	not a loader row — a binary

Compose

Authoring a composition by hand uses the same subpaths:

plugins:
  computer-engine:
    plugin: '@zibokapi/dsh-codex-computer-use/computer-local'
  computer-tools:
    plugin: '@zibokapi/dsh-codex-computer-use/computer-tools'
  computer-policy:
    plugin: '@zibokapi/dsh-codex-computer-use/computer-policy'

helperPath is optional — the engine falls back to the setup CLI's install location — and the other rows are optional too (@zibokapi/dsh-codex-computer-use/computer-policy needs an approval service mounted, e.g. @deepseek-ai/dsh-user-approval). A runnable composition lives in example/cordis.yml.

MCP server

The @zibokapi/dsh-codex-computer-use/computer-mcp bin exposes the same surface — the official ten Codex Computer Use window tools plus request_access and the three event_stream_* recording tools — as a standalone MCP stdio server over the same engine, so Codex CLI, Claude Code, or any MCP client can drive the harness's computer use. The bin keeps the harness packages external like every other entry (@deepseek-ai/dsh-subprocess-local ships node-pty, a native module that cannot be bundled), so it runs wherever pnpm install has materialized the dependencies:

# after `npx @zibokapi/dsh-codex-computer-use` the daemon needs no path:
node lib/mcp.js
# an explicit path (first argument) or the environment override also work:
node lib/mcp.js /absolute/path/to/dsh-computer-daemon.app/Contents/MacOS/dsh-computer-daemon
DSH_COMPUTER_HELPER_PATH=/absolute/path/to/dsh-computer-daemon.app/Contents/MacOS/dsh-computer-daemon dsh-computer-mcp

The server speaks newline-delimited JSON-RPC 2.0 on stdin/stdout, answers initialize / ping / tools/list / tools/call, and mirrors the official response behavior: action tools answer with the post-action state (text plus a JPEG image block when one was captured).

Permissions

The daemon needs two macOS grants, checked and requested at startup: Accessibility (tree reading and input targeting) and Screen Recording (window screenshots). Both are TCC permissions; the plugin never bypasses them. The setup command (npx @zibokapi/dsh-codex-computer-use) asks for both up front, and every plugin load re-checks: it prompts for any missing grant and opens the matching System Settings pane when macOS has already remembered a denial; while a grant is missing the plugin refuses to activate, so computer use is simply not loaded.

macOS keys each grant on the helper's bundle id, code signature, and on-disk path. The setup command installs to one fixed path (~/.dsh/computer-use) so grants persist; sign with a stable identity so they also survive rebuilds. Any code-signing certificate whose designated requirement is identifier-plus-certificate works — including a self-signed one created with Keychain Access (Certificate Assistant → Create a Certificate → Code Signing). Set it once:

DSH_COMPUTER_SIGN_IDENTITY="<certificate common name>" npx @zibokapi/dsh-codex-computer-use

The default ad-hoc signature works, but its designated requirement is the binary's cdhash, so macOS forgets the grants on every rebuild.

Develop

pnpm install
pnpm test          # fake-daemon tests; nothing touches a live desktop
pnpm run typecheck
pnpm run build     # esbuild → lib/, tsc → lib/types

Verification

• CI check job (Ubuntu): a clean pnpm install from npm — the user-facing path — then typecheck and build against the published @deepseek-ai/* packages.
• CI native job (macOS): builds and unit-tests the Swift helper daemon, runs the Node test suite — the engine is macOS-only by design (it throws a platform gate on non-darwin hosts), so the tests that boot it live here and drive a fake daemon, touching no live desktop — and exercises the setup CLI end to end (build, bundle, sign, install, status).
• CI install job (Ubuntu): runs the documented chains against the real CLI — dsh plugin --profile ci add from the checkout path, a packed tarball, and a git URL — asserts the setup binary's non-macOS guidance, then composes the profile and asserts the bundle layer registered, the three rows composed, and every @zibokapi/dsh-codex-computer-use/* row resolves and loads through the installed package.

License

MIT — see LICENSE.