dsh-compat-probe

A DeepSeek Harness plugin with one tool. compat_probe sends five tiny fixture requests to one OpenAI-compatible server and returns a protocol card: a fixed set of booleans saying what that server actually implements, rather than what its docs claim.

The card answers the questions that break a harness at runtime. Is the server up? Does GET /models answer? Does stream: true come back as Server-Sent Events, or as one non-streaming JSON completion because the build ignored the flag? Does it accept max_tokens, max_completion_tokens, both, or neither?

It pairs with dsh-llamacpp, which does the actual serving. This package only asks questions.

What this is not

• It does not ship GGUF weights, and it does not download any.
• It does not start llama-server or any other process. Point it at a server you already run.
• It is not Ollama, and it does not speak the Ollama API.
• It is not a load tester. Five requests, once, when the model calls the tool.
• It is not a GPU benchmark. It measures nothing about throughput, latency, or hardware.
• It is not a client for hosted OpenAI or hosted Anthropic. The default target is loopback, and a non-loopback target is refused unless the deployment opts in.

Install

dsh plugin --profile web add github:jwilson411/dsh-compat-probe

The installer reads dsh.bundle.patch from package.json and appends cordis.patch.yml to the profile's ordered bundle list. The bundle inserts one row, compat-probe, pointing at loopback.

Pin the harness packages

Install @deepseek-ai/dsh-* at 0.1.1-rc.2 explicitly. The latest dist-tag on those packages is older than the release this plugin builds against, so an unpinned install resolves backwards and defineTool will not have the output schema shape used here.

npm install @deepseek-ai/dsh-tools@0.1.1-rc.2

Configuration

Set these in the profile's cordis.patch.yml, or in a --patch overlay, on the row with id: compat-probe. An id-targeted patch replaces the row's whole config, so restate every field you mean to keep.

Key	Default	Env fallback	Notes
baseURL	http://127.0.0.1:8080/v1	DSH_COMPAT_PROBE_BASE_URL	Includes the API prefix, so /v1 is never appended twice. This is where llama-server --port 8080 listens.
apiKey	unset	DSH_COMPAT_PROBE_API_KEY	Sent as Authorization: Bearer when set, and never logged. Prefer the environment variable over writing a credential into the profile. Never a tool argument.
allowRemote	false	none, on purpose	Permit a non-loopback baseURL. No env fallback: widening where the harness sends requests should take an edit to the profile, not an exported shell variable.
model	probe	DSH_COMPAT_PROBE_MODEL	The model id named on the fixture requests. A single-model server ignores it.
timeoutMs	3000	DSH_COMPAT_PROBE_TIMEOUT_MS	Per-request budget in milliseconds.

The tool itself declares no parameters. Where to probe and which credential to send are deployment decisions, so a model cannot aim the harness at a host the operator did not name, and has no argument slot to smuggle a key into.

The card

Field	Type	Meaning
reachable	boolean	Any fixture request received an HTTP response at all.
models_ok	boolean	GET {baseURL}/models answered 2xx.
stream_sse	boolean	A stream: true chat completion came back as Server-Sent Events (text/event-stream, or data: frames).
stream_ignored_json	boolean	A stream: true chat completion came back as one non-streaming JSON chat.completion, meaning the server ignored stream.
honors_max_tokens	boolean	A POST carrying max_tokens: 1 avoided a 4xx or 5xx.
honors_max_completion_tokens	boolean	A POST carrying max_completion_tokens: 1 avoided a 4xx or 5xx.
http_status	integer or null	The first non-2xx status seen in probe order, otherwise the last successful status, otherwise null when nothing connected.
error_code	string or null	Null when every fixture answered 2xx. UNREACHABLE when a request got no answer, HTTP_ERROR when one came back non-2xx.

Failures are fields, not exceptions. A refused connection, a DNS failure, a timeout, and a 500 each land in the card, so the model reads one shape whether the server is healthy or dead.

Default-deny for non-loopback targets

Only http: and https: on a loopback host (localhost, ::1, anything in 127.0.0.0/8) are probed. Anything else is refused before a socket is opened, and the refusal is thrown rather than returned so it cannot be ignored by forgetting to read a boolean. To probe a server on another host, set allowRemote: true on the plugin row.

Loopback is judged as written, not as resolved. A hostname that happens to resolve to 127.0.0.1 is still remote to this check.

Denials carry a stable reason: INVALID_URL, SCHEME_DENIED, or NOT_LOOPBACK.

Tests

npm test

The suite is offline. Every server it probes is a node:http mock bound to 127.0.0.1:0, so the kernel picks a free port and nothing outside the machine is contacted. No GGUF weights, no llama.cpp process, no GPU, no credential. Unreachability is produced by binding a port and releasing it, not by waiting on a timeout.

Topics

dsh-plugin, deepseek-harness, llama-cpp

License

MIT. See LICENSE.