description: Evaluation layer for dsh plugin authors — behavior cases over real headless traces, scripted mock-LLM runs, and comprehension review of plugin outputs
@catheadowl/dsh-eval
English | 中文
A dsh-native (dsh = DeepSeek's deepseek-harness agent CLI) agent evaluation layer for plugin authors: behavior cases run against real headless dsh traces, while review experiments test whether fresh models understand plugin outputs.
It evaluates the assembled agent harness (the graph a plugin + profile + patch + tool registry form inside a real dsh headless run), not isolated functions; verdicts come from dsh-native session-trace projections and matchers (contract assertions), not metric scores. It is not a general agent-eval platform (no dashboard / dataset hosting / metric catalog, no benchmark ranking) and not a DeepEval / OpenAI Evals replacement — those projects proved the problem space; this package picks the dsh-native vertical solution. In deployment terms: eval frameworks embed the model calls inside the harness, while task artifacts (the SWE-bench shape) ship task + fixture + verifier and let the runner supply the agent. This package composes the two — mock cases are credential-free artifacts judged by deterministic trace verifiers, real cases put the assembled harness itself (profile-supplied model included) under contract assertions, and comprehension review dispatches blank-environment reviewers — models that see only the plugin's materialized output, nothing else — as the measuring instrument at the plugin's output surface, not as the subject.
Documentation is Chinese-first; deep contracts live in docs/ (matchers / boundary contracts / review / report structure / host wiring / known issues).
Why it exists
| Layer | Question | Granularity | Verdict | Execution |
|---|---|---|---|---|
| unit / shape test | are deterministic fields and values correct | component | automatic | the plugin's own node:test |
| behavior real | does natural-language intent pick the right tool | trajectory | trace matcher | dsh + real model |
| behavior mock | is the tool pipeline and write round-trip stable | trajectory + end state | trace matcher + workspace inspect | dsh + scripted mock LLM |
| comprehension review | can a fresh model understand the output and the next step | rubric-primary | manual rubric, converged over runs | abstract review experiment + replaceable executor |
A dsh plugin is correct when the assembled graph really wires tools, steers, prompts, and gates together — plugin unit tests cover only part of that, and "is the output understandable" is not a string regression at all. This package turns both layers into replayable evidence instead of manual trial runs.
plugin-owned experiment shared framework
fixtures + prompt + rubric + observe ──► experiment/review.mjs
│ task
▼
adapters/dsh/review.mjs ──► dsh headless
behavior *.eval.mjs ───────────────────► dsh behavior runner (trace + mock)
src/experiment/is the model- and runtime-agnostic experiment layer: blind review, live observation, byte-identical evidence across reviewers. It does not import dsh.src/adapters/dsh/is the landing layer: hands the abstract task to an isolated dsh headless run.- Your
eval/keeps only domain fixtures, projections/observe, prompts, rubrics, and cases — no runner duplication.
Install
npm i -D @deepseek-ai/dsh @catheadowl/dsh-eval
Install with npm, host co-installed in the same tree — the verified path. Other package managers and store layouts (e.g. a pnpm virtual store) can leave the dsh-llm peer without a resolvable position; for pnpm-workspace projects, give eval its own npm-managed directory (own package.json) instead of installing into the workspace — topology notes in docs/host-wiring/README.md. Pin @deepseek-ai/dsh to the same version as the CLI your profiles run on — eval drives the binary from this install tree, and a driving-vs-production generation mismatch silently produces false verdicts. Run everything from the package root under test (where eval/ lives). Node ^22.19 || >=24.
Requirements (wiring details and failure self-diagnostics in docs/host-wiring/README.md):
- the dsh host as a dev dependency (
@deepseek-ai/dsh) — its dependency tree ships a same-generationdsh-llm, so the peer needs zero manual wiring; a built checkout (apps/cli/lib/bin.js, pinned via--repo) works as an alternative host source but is optional; - the plugin under test installed into a dsh profile;
- the only form still needing manual peer wiring is an eval-only install with no host in the tree (npm then auto-installs an incompatible antique
latestof@deepseek-ai/dsh-llm— see the host-wiring doc).
Quickstart
<plugin>/eval/behavior/mock/smoke.eval.mjs:
import { toolCalled, toolCallStep, textStep } from '@catheadowl/dsh-eval'
export default {
id: 'my-first-case',
mode: 'mock', // cases default to mode 'real' — mock must say so
task: 'rename guide.md to intro.md',
async prepare(workspace) { /* optional: seed fixture files */ },
script: { steps: [toolCallStep('md_rename', { oldPath: 'guide.md', newPath: 'intro.md' }), textStep('done')] },
expect: [toolCalled('md_rename')],
}
Install the plugin under test into the profile once — the dsh plugin add write face forwards to pnpm, which must be on PATH (corepack enable pnpm or npm i -g pnpm) — then run; the CLIs are local bins, invoke them via npx:
npx dsh plugin --profile headless add <plugin dir | tarball>
npx dsh-eval run --profile headless --mode mock eval/behavior/mock
npx dsh-review --dry-run eval/comprehension # model-free dry run of the review layer
# → PASS my-first-case
# 1 selected, 1 passed, 0 failed, 0 skipped
--profile headless is the zero-setup default — the shipped template auto-materializes in an isolated home on first use. Any profile you pass must run the headless app: the case task reaches it as a positional argument, so web-based profiles reject the run. A dsh-eval.config.mjs at the package root replaces the explicit flag (see "Unified config" below). A complete real-model case (mode: 'real') is in docs/intent-cases.md; real runs need a provider key by env (see docs/host-wiring/credentials.md).
Real runs use dsh-eval run --profile <p> <case path> (the CLI resolves from the install tree; pass --repo <harness checkout> to pin a built checkout instead); all flags (--mode/--keep-artifacts/--fail-on-skip/--format/--report) are documented in docs/report.md. Real cases auto-skip without credentials (dsh resolves credentials itself); mock and dry runs need no credentials. The model a real run uses comes from the chosen profile (NOT your interactive session's current pick) — provenance, how to check what a run used, and how to pin it live in docs/host-wiring/model-selection.md.
Canonical layout
<plugin>/eval/
.gitignore # .runs/ (no path prefix)
README.md
behavior/ # optional
real/*.eval.mjs
mock/*.eval.mjs
_fixtures/
comprehension/ # optional
<name>.review.mjs
fixtures.json
prompt.md
rubric.md
Unified config dsh-eval.config.mjs
Drop one at the consumer package root; both CLIs walk upward from the working directory, and flags always override config:
export default {
profile: 'headless', // dsh profile
mode: 'mock', // behavior CLI's --mode default (review has none)
failOnSkip: false, // behavior CI gate default
report: 'eval-report.json', // --report default (anchored at the config dir)
disableRows: ['gates'], // plugin rows disabled by default; case-level declarations win
// (explicit [] = all enabled, for gate-interaction cases)
}
Unknown keys fail loudly (typos never degrade silently). The disableRows semantics and the turn-close gate boundary contract are in docs/disablerows.md.
Docs
| Doc | Topic |
|---|---|
| host-wiring | install and host wiring topic index: peer wiring (the antique-peer trap applies to host-absent installs only), CLI resolution, host contract assertions, model selection, credentials, spawn requirements |
| concepts | concept mapping from mainstream agent eval (dataset / metric / LLM-as-judge / trajectory / CI) into cases / matchers / review — behavior cases are unittest semantics, judgment frozen at authoring time |
| host-machinery | relationship to the host's own in-tree test machinery (test-support group, snapshot/e2e/bench tracks) and a scenario-to-machinery dispatch table for plugin authors |
| review | comprehension review: experiment definition, blank-environment reviewers, artifacts, the six review rules, machine judge first-pass (--judge) |
| matchers | the full trace-matcher and mock-helper set (tool face / text face / model-visible face) |
| disablerows | disableRows and the turn-close gate boundary contract |
| rowconfig | the rowConfig per-row config override contract (whole-segment replacement, restate needed keys) |
| environment | experiment-environment discipline — know and control the run environment: staging coverage boundary and leak paths, confounding (false-red / false-green), row pinning by default with case-level restore |
| cross-turn | the followups cross-turn async driving contract (driver-row swap, subagent settle wait, single-cursor mock orchestration, bounded-redispatch assertions) |
| experiments | behavior experiments: arms as case-field overrides × repeats × guards × aggregation (defineBehaviorExperiment, arm deep-merge, INVALID discipline, preregistered decision rule) |
| intent-cases | real intent-case spec: when to write one, assertion face, guards, CI semantics |
| report | machine-readable report structure (--format json / --report) |
| known-issues | known issues and workarounds (e.g. REQUEST_EXTENSION in staged homes) |
| runner-api | programmatic runner API: runEvalCase options contract, EvalRunResult fields, crossing tiers for cliPath |
| experimental | experimental subpath symbol list (escape hatch, no compatibility promise) |
Runtime guarantees
The runner uses try/finally so temp directories and links are cleaned up on every path (prepare throwing, mock validation failure, spawn errors) — the real profile store is never polluted. The behavior and review CLIs share directory scanning (skipping .runs and node_modules); the behavior CLI validates case shapes and detects cross-file duplicate ids at load time, failing as early as possible.
License: MIT. The framework's own tests and release self-checks are carried by the repository CI and do not ship with the package. Release history: CHANGELOG.md.
No comments yet. Be the first to write one.