dsh-image-vision

Seamless image understanding for DeepSeek Harness — lets pure-text main models read pasted/dropped images and Feishu/Lark images via a configurable vision model, without modifying any host code.

A pure "everything is a plugin" implementation: it wraps a documented llm service method and listens to official agent/tool waterfalls. No packages/ files are touched, so it portably runs wherever DeepSeek Harness runs.

简体中文说明见 README.zh.md

---

What it does

• Paste / drop an image in chat — even when the main model is text-only (e.g. DeepSeek V4 Flash routed through pi-ai), the image is accepted and described to the model as text.
• Feishu / Lark document images — <image token="..."/> produced by lark_read_doc and friends is downloaded through lark-cli and described; the result is injected back as tool output.
• describe_image model tool — describes a local image file path.
• Full description, not just OCR — people, scenes, tables, charts and verbatim text are all described, driven by a prompt you can override.

Pick a vision model from ModelScope free community API (e.g. Qwen/Qwen3-VL-8B-Instruct) or SiliconFlow (e.g. Qwen/Qwen3-VL-32B-Instruct). Routes are auto-discovered from your settings.

---

Why no host code changes

The web API proxy gates image upload/model-switch by calling llm.resolveModelInfo(...).inputModalities and rejecting when the model does not declare image. Since DSH is "everything is a plugin", this plugin:

1. Wraps ctx.llm.resolveModelInfo — when the target model lacks image input and a vision route is configured, the returned capabilities gain image, so the admission gate lets the image through.
2. Listens on agent/pre-step — before the real model request, each image block is transcribed by the vision model and replaced with its text, so the text-only main model only ever receives plain text.

Both steps use documented DSH extension points. No core files are changed.

---

Requirements

• DeepSeek Harness 0.1.0-rc.6 (source checkout or installed release — both are supported, see Installation).
• Node.js >= 22.19
• A configured vision model provider (see Configuration).

---

Installation

Two deployment styles are supported: a source checkout (clone the deepseek-harness repo and run pnpm dsh web) and the installed release (npm i -g @deepseek-ai/dsh). Both go through the same dsh plugin manager; @deepseek-ai/* dependencies are provided by the host and resolved from the DSH installation anchor.

From GitHub (recommended)

npx @deepseek-ai/dsh plugin --profile web add github:VeryInt/dsh-image-vision

From a local path (development / validation)

git clone https://github.com/VeryInt/dsh-image-vision.git /path/to/dsh-image-vision
npx @deepseek-ai/dsh plugin --profile web add /path/to/dsh-image-vision

Replace <profile> with whichever profile you use (web, headless, ...). If pnpm blocks a git dependency's build/prepare script, allowlist the exact key it prints under allowBuilds in <profile>/pnpm-workspace.yaml, then re-run.

Restart Harness after installing. The plugin then appears under Settings → Plugins.

---

Configuration

Image recognition needs a vision model provider. Add an OpenAI-compatible vision endpoint in the DSH Models page (or ~/.dsh/settings.yaml), and — this is the part the Models page does not show — declare input: [text, image] on the model entry.

Example settings.yaml:

llm-pi-ai:
  providers:
    modelscope:
      displayName: ModelScope
      apiKeyEnv: MODELSCOPE_API_KEY
      api: openai-completions
      baseURL: https://api-inference.modelscope.cn/v1
      models:
        - id: Qwen/Qwen3-VL-8B-Instruct
          name: Qwen3-VL-8B
          input: [ text, image ]
    siliconflow:
      displayName: 硅基流动
      apiKeyEnv: SILICONFLOW_API_KEY
      api: openai-completions
      baseURL: https://api.siliconflow.cn/v1
      models:
        - id: Qwen/Qwen3-VL-32B-Instruct
          name: Qwen3-VL-32B-Instruct
          input: [ text, image ]

Then provide the API key in ~/.credentials.yaml or as an environment variable:

# ModelScope community access: generate at https://modelscope.cn
MODELSCOPE_API_KEY=ms-xxxxxx
# SiliconFlow: generate at https://cloud.siliconflow.cn
SILICONFLOW_API_KEY=sk-xxxxxx

Use a vision model (VL suffix). Text-only models such as Qwen/Qwen3.5-27B return empty results even when declared image-capable.

Plugin vision route

The plugin's cordis.patch.yml carries two relevant fields:

- insert:
    - id: dsh-image-vision
      name: dsh-image-vision
      config:
        provider: siliconflow      # vision provider route (empty = auto-discover)
        model: Qwen/Qwen3-VL-32B-Instruct   # vision model id (empty = auto-discover)
        # prompt: ...
        # maxImagesPerMessage: 4
        # feishuImages: true
        # maxFeishuImages: 5

Rules:

• Both provider and model set: used as-is (highest priority).
• Both empty: scan every configured provider and pick the first model declaring input: [text, image] (this is the common case).
• Changing the model = update these two fields and make sure the model declares input: [text, image] in settings.yaml.

---

Verification

After a restart:

1. Paste an image in chat — the main model should describe its content (rendered as [图片内容] …).
2. Read a Feishu/Lark doc with images — tool output gains [飞书图片 <token>] … descriptions.
3. describe_image tool — ask the model to describe a local image path.

On a transient vision failure the reply still comes back with a [图片内容识别失败：…] / [飞书图片 … 识别失败：…] placeholder rather than failing the whole turn.

---

Options

Field	Default	Description
provider	empty	Vision provider route; empty = auto-discover
model	empty	Vision model id; empty = auto-discover
prompt	(full description)	Instruction sent to the vision model
maxImagesPerMessage	4	Max images bridged per message; beyond throws
feishuImages	true	Auto-describe Feishu <image token>
maxFeishuImages	5	Max Feishu images per tool result

---

How this differs from oil-oil/dsh-vision

We looked at the community plugin oil-oil/dsh-vision before writing this one. Key differences:

	oil-oil/dsh-vision	dsh-image-vision
Mechanism	Replaces the official deepseek-official adapter with a VisionBridgeAdapter; installer disables the built-in llm-deepseek	Wraps only llm.resolveModelInfo and uses the official agent/pre-step waterfall — no adapter replacement, no built-in plugin disabled
Works with	Models routed through deepseek-official	Any pure-text route (pi-ai deepseek, etc.) because it hooks the llm service, not a specific route
Vision backends	Bundles several (zenmux / Bailian / see config / local OCR)	Reuses your settings.yaml / Models vision providers (input: [text, image])
Settings UI	Ships a "Vision Recognition" card	None — config lives in the plugin cordis.patch.yml
Packaging	TypeScript + tsdown build, many peer deps	Plain JS ESM, minimal peers, no build step

In short: oil-oil replaces the DeepSeek adapter (so it only affects deepseek-official and requires disabling the built-in plugin); this plugin only wraps a llm capability lookup and uses the official pre-step — it works for any text-only main model route and makes no destructive changes, which fits "everything is a plugin" more literally.

---

License

MIT