dsh-sensevoice-npu

Local speech-to-text for DSH that runs SenseVoiceSmall on the Intel NPU through OpenVINO, plugged into the official voice-input stack.

It registers one more provider into the stock @deepseek-ai/dsh-experimental-speech-to-text service, so the voice-input button, the settings card, the preparation UI and the speech API keep working unchanged — only the recognizer behind them changes.

Why a separate plugin

The bundled provider (dsh-experimental-speech-to-text-sensevoice) is CPU-only by construction:

lib/worker.js hard-codes provider: "cpu" for the recognizer and the VAD;
the shipped sherpa-onnx-win-x64 package is a CPU-only ONNX Runtime build, so sherpa-onnx's openvino provider logs "OpenVINOExecutionProvider is not available … Fallback to cpu";
ONNX Runtime publishes no OpenVINO build for Windows;
and OpenVINO crashes the NPU compiler on this model's default graph (unbounded dynamic dimensions from the x_length attention mask).

This plugin sidesteps all four: it bakes x_length into a constant for one frame bucket (which the NPU compiler accepts), compiles that graph for the NPU, and serves it through a small Python worker.

Layout

Path	Role
`lib/index.js`	cordis plugin: registers the provider, spawns the worker, runs preparation
`lib/worker.py`	authenticated loopback HTTP worker (one WAV per request)
`lib/sensevoice_npu.py`	frontend (fbank + LFR + CMVN), Silero VAD, NPU session, CTC decode
`runtime/bootstrap.ps1`	creates the virtual environment the worker needs

The frontend mirrors sherpa-onnx exactly: kaldi_native_fbank with the same options, ApplyLfr (7/6), and (x + neg_mean) * inv_stddev with the CMVN vectors read from the ONNX metadata — so transcripts match the CPU provider.

Install

Prerequisite: model files

The worker reads three files (this plugin does not download models — it registers with an empty downloadSources):

Path relative to `dataRoot`	Size
`models/sensevoice-onnx/model.int8.onnx`	~228 MB
`models/sensevoice-onnx/tokens.txt`	~0.3 MB
`models/silero/silero_vad.onnx`	~1.7 MB

The official dsh-experimental-speech-to-text-sensevoice provider downloads them into the same dataRoot (<DSH home>/speech-to-text/sensevoice), so let that provider prepare once, or place the files at those paths yourself.

Build the Python environment (needs a 64-bit Python 3.11–3.13 and the Intel NPU driver):
```
pwsh -File runtime/bootstrap.ps1
```

Install the package into the profile that owns the voice-input bundle (the DSH desktop app uses the desktop profile):

dsh plugin --profile desktop add "file:C:/path/to/dsh-sensevoice-npu"

pnpm copies the package, so while iterating on it replace the copy with a junction to your working tree:

$d = "$env:USERPROFILE\.dsh\profiles\desktop\node_modules\dsh-sensevoice-npu"
Remove-Item -Recurse -Force $d
New-Item -ItemType Junction -Path $d -Target C:\path\to\dsh-sensevoice-npu

The bundle's own cordis.patch.yml already selects the provider and registers the plugin, and it contains no machine-specific path: pythonPath and dataRoot default to the bootstrap.ps1 venv and DSH's speech root. Append an override to ~/.dsh/profiles/desktop/cordis.patch.yml (back it up first) only when you moved either one:
```
- id: speech-to-text-npu
  name: 'dsh-sensevoice-npu'
  config:
    pythonPath: <custom venv>\Scripts\python.exe   # only with -VenvDir
    dataRoot: <custom speech root>                 # only with a non-default DSH_HOME
```
Restart DSH. The voice-input card then lists SenseVoiceSmall · Intel NPU (NPU) and uses it by default; the first preparation compiles the NPU graph once (~2.5 min), after which starts take ~0.2 s. If the card still shows the CPU provider, pick the NPU one in the voice-input settings (an explicit choice persists over the patch default).

Configuration

Key	Default	Meaning
`pythonPath`	`%LOCALAPPDATA%\dsh-sensevoice-npu\venv\Scripts\python.exe`	worker interpreter (`runtime/bootstrap.ps1` target)
`dataRoot`	`<DSH home>/speech-to-text/sensevoice`	DSH speech root; models are read from `<dataRoot>/models`
`precision`	`int8`	`int8` (`model.int8.onnx`) or `fp32` (`model.onnx`)
`device`	`NPU`	OpenVINO device for the recogniser (`NPU`, `GPU`, `CPU`)
`vadDevice`	`CPU`	Silero VAD device — tiny model, dynamic shapes, NPU rejects it
`useVad`	`true`	trim silence and split long speech into segments
`buckets`	`[64, 96, 128, 192, 256, 384, 512, 768, 1024]` (plugin config sets `[256]`)	LFR-frame buckets the engine may compile
`prewarmBuckets`	`[256]`	buckets compiled before the worker reports ready
`idleTimeoutMs`	`600000`	release the worker after idling; compiled blobs stay cached

Each bucket costs roughly 800 MB on disk (baked IR + OpenVINO NPU cache) and one ~2.5 minute compile the first time. One bucket is enough for dictation: 256 frames ≈ 15 s of audio, and the NPU handles the padding cheaply.

Measured behaviour (Core Ultra X7 358H, Arc B390, NPU driver 32.0.100.5540)

Case	Result
Warm start (compiled blob cached)	0.21 s
4.5 s Chinese clip → 256-frame bucket	`今天天气不错，我们去公园散步吧`, 0.66 s inference
5.2 s clip	`明天上午10点开会，记得带上项目进度报告`, 0.48 s
14.5 s clip	full sentence correct, 0.65 s
First compile per bucket	100–145 s, cached afterwards

CPU is used only for the frontend (fbank + VAD, a few ms) and JSON plumbing; the encoder/decoder graph runs on the NPU. Note that the NPU is not faster than sherpa-onnx on CPU for clips this short (~0.5 s vs ~0.2 s) — the win is moving the heavy work off the CPU/GPU entirely.

Development

node --test        # 6 tests, no dependencies

They cover the registration contract (provider id, device label, diagnostic file), the portable pythonPath/dataRoot defaults, and the two config guards that would otherwise fail much later: an empty bucket list and a relative path.

Troubleshooting

Confirm the profile really loaded the plugin and see the last phase/device:

Get-Content "$env:USERPROFILE\.dsh\speech-to-text\sensevoice\npu\provider-status.json"

NPU device not visible → update the Intel NPU driver (32.0.100.5540 tested).
NPU compile failed → delete <dataRoot>/npu and retry; check free disk (~1 GB per bucket).
Transcription slow on first use of a new bucket → it is compiling; later runs reuse the cache.

sakuraboy9128-cmd/dsh-sensevoice-npu