dsh-vision-recognizer

English | 简体中文

Keep DeepSeek as the conversation brain, attach images anyway, and switch the image-recognition provider any time from Settings → Plugins. A vision plugin for DeepSeek Harness.

It registers an adaptive provider route (default vision-recognizer, shown as DeepSeek + 智能识图 in the model picker) that wraps the configured conversation provider. The wrapper always admits image attachments, then resolves the exact selected model: models declaring native image input receive the original image blocks directly; text-only or unknown-capability models receive text transcribed by the vision model you configure. DeepSeek remains the default wrapped conversation provider.

attached image ──▶ vision-recognizer route ──▶ selected model supports image? ── yes ─▶ native image request
                                                      │
                                                      no
                                                      ▼
                                  configured vision transcription ──▶ text-only selected model

Features

• Adaptive routing: native multimodal models receive images unchanged; only text-only or unknown-capability models invoke the configured transcription fallback.
• One-click install: dsh plugin --profile web add dsh-vision-recognizer — no build scripts, no sharp approval (no native dependencies at all).
• Configure from Settings → Plugins → Vision: pick a provider, enter an API key, override model / endpoint / token cap / timeout / marker. Saved changes take effect immediately, no restart.
• 15+ providers, domestic and international: OpenAI, Anthropic Claude, Google Gemini, OpenRouter, Azure OpenAI, Ollama (local), plus Alibaba DashScope, QwenCloud (Intl), Zhipu GLM, Baidu Qianfan, iFlytek Spark, Moonshot Kimi, Tencent Hunyuan, Volcengine Doubao, SiliconFlow. Any OpenAI-compatible endpoint works via the custom provider.
• Two wire protocols: OpenAI-compatible (/chat/completions) and native Anthropic Messages — Claude works out of the box.
• No hangs: local/anonymous endpoints get a hard 20s timeout cap, HTTP 429 fails fast, failed endpoints cool down for 60s; without a key and without local Ollama it fails fast with actionable guidance.
• Fallback chain: after the primary model fails, each fallbackModels entry is tried in order (each may target a different vendor); only after all fail does the request fail, listing every attempt.
• Content-hash cache: the same image is transcribed at most once per process (in-process, capped at 200).
• Zero-config local path: autoLocalOllama (default on) probes http://localhost:11434 and prepends a running Ollama to the chain — images never leave your machine.

Quick start

dsh plugin --profile web add dsh-vision-recognizer

Slow npm registry? dsh plugin --profile web add dsh-vision-recognizer --registry=https://registry.npmmirror.com

Install from a local checkout (development):

dsh plugin --profile web add file:/path/to/dsh-vision-recognizer

Use the file: prefix (copies the package into node_modules). A bare add . or add link:… makes pnpm symlink the package, in which case the plugin's schemastery dependency resolves from the source checkout and is not found — a general pnpm symlink-install gotcha, not a bug in the plugin.

Restart dsh web, then:

1. Pick DeepSeek + 智能识图 in the model selector;
2. Open Settings → Plugins → Vision, choose the fallback vision provider, enter an API key, save;
3. Paste an image into a conversation. A native multimodal selected model receives it directly; a text-only selected model receives the [图片转译] result.

With a native multimodal selected model, no fallback key is required. With a text-only model and no key or local Ollama, the turn fails fast with guidance instead of hanging.

Scope: adaptive fallback applies while the DeepSeek + 智能识图 wrapper route is selected. Selecting another provider route calls that route directly. rc8 does not expose a public decorator hook that can add fallback behavior to every existing provider route.

Supported providers

Provider	baseURL	Default model	Key env var	Protocol
OpenAI	https://api.openai.com/v1	gpt-4o-mini	OPENAI_API_KEY	OpenAI
Anthropic Claude	https://api.anthropic.com/v1	claude-3-5-sonnet-latest	ANTHROPIC_API_KEY	Anthropic
Google Gemini	https://generativelanguage.googleapis.com/v1beta/openai	gemini-2.0-flash	GEMINI_API_KEY	OpenAI
OpenRouter	https://openrouter.ai/api/v1	qwen/qwen-2.5-vl-72b-instruct	OPENROUTER_API_KEY	OpenAI
Azure OpenAI	user-supplied (…/openai/deployments/<deployment>)	gpt-4o-mini	AZURE_OPENAI_API_KEY	OpenAI
Ollama (local)	http://localhost:11434/v1	auto-detected	none	OpenAI
Alibaba DashScope	https://dashscope.aliyuncs.com/compatible-mode/v1	qwen-vl-max	DASHSCOPE_API_KEY	OpenAI
QwenCloud (Intl)	https://dashscope-intl.aliyuncs.com/compatible-mode/v1	qwen-vl-plus	DASHSCOPE_API_KEY	OpenAI
Zhipu GLM	https://open.bigmodel.cn/api/paas/v4	glm-4v-flash	ZHIPU_API_KEY	OpenAI
Baidu Qianfan	https://qianfan.baidubce.com/v2	ernie-4.5-vl-8k	QIANFAN_API_KEY	OpenAI
iFlytek Spark	https://spark-api-open.xf-yun.com/v1	generalv3.5	SPARK_API_KEY	OpenAI
Moonshot Kimi	https://api.moonshot.cn/v1	moonshot-v1-8k-vision-preview	MOONSHOT_API_KEY	OpenAI
Tencent Hunyuan	https://api.hunyuan.cloud.tencent.com/v1	hunyuan-vision	HUNYUAN_API_KEY	OpenAI
Volcengine Doubao	https://ark.cn-beijing.volces.com/api/v3	doubao-1.5-vision-pro-32k-250115	ARK_API_KEY	OpenAI
SiliconFlow	https://api.siliconflow.cn/v1	Qwen/Qwen2.5-VL-72B-Instruct	SILICONFLOW_API_KEY	OpenAI

Model ids drift over time; the defaults are starting points — override Model in the settings UI. Key resolution order: key entered in the UI → the provider env var → $VISION_API_KEY / $DASHSCOPE_API_KEY.

Configuration storage

Config saved from the UI is written to $DSH_HOME/vision-recognizer.json and merged over the bundle defaults at startup. cordis.patch.yml only carries factory defaults; a user cordis.patch.yml override still works as the composition-time fallback.

⚠️ patch semantics: the bundle's - insert: appends this row to the entry list. Writing a second - insert: with the same id in your own cordis.patch.yml would register the adapter twice (undefined behavior). To override individual keys, write a single top-level - id: dsh-vision-recognizer entry; better yet, use the Settings UI.

Implementation notes (for plugin authors)

The adaptive wrapper uses rc8 public interfaces only:

• ctx.llm.listModels(innerProvider) and resolveModelInfo(innerProvider, model) inspect the exact target model and rebind its metadata to the wrapper route;
• ctx.llm.prepareCall(...) delegates to the configured target provider without depending on private adapter registrations;
• proxy resolveModel advertises ['text', 'image'] so the wrapper admits images, while proxy stream uses the target's original modality declaration to choose native pass-through or transcription;
• transcription clones request messages and replaces image blocks only in the delegated request; durable session history keeps the original image references;
• the settings UI rides the settings.plugins.tab slot plus custom webServer routes, persisting config to its own JSON file.

rc8 limitations

Capability lookup and prepared target dispatch are separate public operations in rc8. A target adapter replaced by HMR in that tiny interval can race the routing decision. Nested target delegation also enters the llm/stream waterfall a second time, and DSH may strip provider-private replay metadata when wrapper and target adapters differ. Ordinary text/image history is preserved; provider-specific replay signatures may lose their optimization or fidelity until DSH exposes an atomic delegation handle.

Privacy

Native multimodal routing sends image bytes to the selected conversation provider. The text-only fallback instead sends them (base64, normally HTTPS) to the vision endpoint you configure. In either mode, image data leaves your machine unless that endpoint is local. Nothing beyond the harness's own attachment storage persists an image.

License

MIT