DSH HUB
HomePlugin StorePlugin PacksCommunityRankingsResourcesPublish Guide
Plugin source
Back to catalog

CatheadOwl /

CatheadOwl/dsh-eval

Topic repository only

Agent eval framework over dsh headless runs: case runner, session-trace assertions, and a scripted mock-LLM layer for plugin intent tests.

★ 0 Stars0 Forks0 IssuesN/A Community rating0 Confirmed installs
View on GitHub
READMESource: main@a52d5c02

description: Evaluation layer for dsh plugin authors — behavior cases over real headless traces, scripted mock-LLM runs, and comprehension review of plugin outputs

@catheadowl/dsh-eval

English | 中文

A dsh-native (dsh = DeepSeek's deepseek-harness agent CLI) agent evaluation layer for plugin authors: behavior cases run against real headless dsh traces, while review experiments test whether fresh models understand plugin outputs.

It evaluates the assembled agent harness (the graph a plugin + profile + patch + tool registry form inside a real dsh headless run), not isolated functions; verdicts come from dsh-native session-trace projections and matchers (contract assertions), not metric scores. It is not a general agent-eval platform (no dashboard / dataset hosting / metric catalog, no benchmark ranking) and not a DeepEval / OpenAI Evals replacement — those projects proved the problem space; this package picks the dsh-native vertical solution. In deployment terms: eval frameworks embed the model calls inside the harness, while task artifacts (the SWE-bench shape) ship task + fixture + verifier and let the runner supply the agent. This package composes the two — mock cases are credential-free artifacts judged by deterministic trace verifiers, real cases put the assembled harness itself (profile-supplied model included) under contract assertions, and comprehension review dispatches blank-environment reviewers — models that see only the plugin's materialized output, nothing else — as the measuring instrument at the plugin's output surface, not as the subject.

Documentation is Chinese-first; deep contracts live in docs/ (matchers / boundary contracts / review / report structure / host wiring / known issues).

Why it exists

Layer Question Granularity Verdict Execution
unit / shape test are deterministic fields and values correct component automatic the plugin's own node:test
behavior real does natural-language intent pick the right tool trajectory trace matcher dsh + real model
behavior mock is the tool pipeline and write round-trip stable trajectory + end state trace matcher + workspace inspect dsh + scripted mock LLM
comprehension review can a fresh model understand the output and the next step rubric-primary manual rubric, converged over runs abstract review experiment + replaceable executor

A dsh plugin is correct when the assembled graph really wires tools, steers, prompts, and gates together — plugin unit tests cover only part of that, and "is the output understandable" is not a string regression at all. This package turns both layers into replayable evidence instead of manual trial runs.

plugin-owned experiment             shared framework
fixtures + prompt + rubric + observe ──► experiment/review.mjs
                                               │ task
                                               ▼
                                        adapters/dsh/review.mjs ──► dsh headless

behavior *.eval.mjs ───────────────────► dsh behavior runner (trace + mock)
  • src/experiment/ is the model- and runtime-agnostic experiment layer: blind review, live observation, byte-identical evidence across reviewers. It does not import dsh.
  • src/adapters/dsh/ is the landing layer: hands the abstract task to an isolated dsh headless run.
  • Your eval/ keeps only domain fixtures, projections/observe, prompts, rubrics, and cases — no runner duplication.

Install

npm i -D @deepseek-ai/dsh @catheadowl/dsh-eval

Install with npm, host co-installed in the same tree — the verified path. Other package managers and store layouts (e.g. a pnpm virtual store) can leave the dsh-llm peer without a resolvable position; for pnpm-workspace projects, give eval its own npm-managed directory (own package.json) instead of installing into the workspace — topology notes in docs/host-wiring/README.md. Pin @deepseek-ai/dsh to the same version as the CLI your profiles run on — eval drives the binary from this install tree, and a driving-vs-production generation mismatch silently produces false verdicts. Run everything from the package root under test (where eval/ lives). Node ^22.19 || >=24.

Requirements (wiring details and failure self-diagnostics in docs/host-wiring/README.md):

  • the dsh host as a dev dependency (@deepseek-ai/dsh) — its dependency tree ships a same-generation dsh-llm, so the peer needs zero manual wiring; a built checkout (apps/cli/lib/bin.js, pinned via --repo) works as an alternative host source but is optional;
  • the plugin under test installed into a dsh profile;
  • the only form still needing manual peer wiring is an eval-only install with no host in the tree (npm then auto-installs an incompatible antique latest of @deepseek-ai/dsh-llm — see the host-wiring doc).

Quickstart

<plugin>/eval/behavior/mock/smoke.eval.mjs:

import { toolCalled, toolCallStep, textStep } from '@catheadowl/dsh-eval'

export default {
  id: 'my-first-case',
  mode: 'mock',                    // cases default to mode 'real' — mock must say so
  task: 'rename guide.md to intro.md',
  async prepare(workspace) { /* optional: seed fixture files */ },
  script: { steps: [toolCallStep('md_rename', { oldPath: 'guide.md', newPath: 'intro.md' }), textStep('done')] },
  expect: [toolCalled('md_rename')],
}

Install the plugin under test into the profile once — the dsh plugin add write face forwards to pnpm, which must be on PATH (corepack enable pnpm or npm i -g pnpm) — then run; the CLIs are local bins, invoke them via npx:

npx dsh plugin --profile headless add <plugin dir | tarball>
npx dsh-eval run --profile headless --mode mock eval/behavior/mock
npx dsh-review --dry-run eval/comprehension     # model-free dry run of the review layer
# → PASS my-first-case
#   1 selected, 1 passed, 0 failed, 0 skipped

--profile headless is the zero-setup default — the shipped template auto-materializes in an isolated home on first use. Any profile you pass must run the headless app: the case task reaches it as a positional argument, so web-based profiles reject the run. A dsh-eval.config.mjs at the package root replaces the explicit flag (see "Unified config" below). A complete real-model case (mode: 'real') is in docs/intent-cases.md; real runs need a provider key by env (see docs/host-wiring/credentials.md).

Real runs use dsh-eval run --profile <p> <case path> (the CLI resolves from the install tree; pass --repo <harness checkout> to pin a built checkout instead); all flags (--mode/--keep-artifacts/--fail-on-skip/--format/--report) are documented in docs/report.md. Real cases auto-skip without credentials (dsh resolves credentials itself); mock and dry runs need no credentials. The model a real run uses comes from the chosen profile (NOT your interactive session's current pick) — provenance, how to check what a run used, and how to pin it live in docs/host-wiring/model-selection.md.

Canonical layout

<plugin>/eval/
  .gitignore                 # .runs/ (no path prefix)
  README.md
  behavior/                  # optional
    real/*.eval.mjs
    mock/*.eval.mjs
    _fixtures/
  comprehension/             # optional
    <name>.review.mjs
    fixtures.json
    prompt.md
    rubric.md

Unified config dsh-eval.config.mjs

Drop one at the consumer package root; both CLIs walk upward from the working directory, and flags always override config:

export default {
  profile: 'headless',              // dsh profile
  mode: 'mock',                     // behavior CLI's --mode default (review has none)
  failOnSkip: false,                // behavior CI gate default
  report: 'eval-report.json',       // --report default (anchored at the config dir)
  disableRows: ['gates'],           // plugin rows disabled by default; case-level declarations win
                                     // (explicit [] = all enabled, for gate-interaction cases)
}

Unknown keys fail loudly (typos never degrade silently). The disableRows semantics and the turn-close gate boundary contract are in docs/disablerows.md.

Docs

Doc Topic
host-wiring install and host wiring topic index: peer wiring (the antique-peer trap applies to host-absent installs only), CLI resolution, host contract assertions, model selection, credentials, spawn requirements
concepts concept mapping from mainstream agent eval (dataset / metric / LLM-as-judge / trajectory / CI) into cases / matchers / review — behavior cases are unittest semantics, judgment frozen at authoring time
host-machinery relationship to the host's own in-tree test machinery (test-support group, snapshot/e2e/bench tracks) and a scenario-to-machinery dispatch table for plugin authors
review comprehension review: experiment definition, blank-environment reviewers, artifacts, the six review rules, machine judge first-pass (--judge)
matchers the full trace-matcher and mock-helper set (tool face / text face / model-visible face)
disablerows disableRows and the turn-close gate boundary contract
rowconfig the rowConfig per-row config override contract (whole-segment replacement, restate needed keys)
environment experiment-environment discipline — know and control the run environment: staging coverage boundary and leak paths, confounding (false-red / false-green), row pinning by default with case-level restore
cross-turn the followups cross-turn async driving contract (driver-row swap, subagent settle wait, single-cursor mock orchestration, bounded-redispatch assertions)
experiments behavior experiments: arms as case-field overrides × repeats × guards × aggregation (defineBehaviorExperiment, arm deep-merge, INVALID discipline, preregistered decision rule)
intent-cases real intent-case spec: when to write one, assertion face, guards, CI semantics
report machine-readable report structure (--format json / --report)
known-issues known issues and workarounds (e.g. REQUEST_EXTENSION in staged homes)
runner-api programmatic runner API: runEvalCase options contract, EvalRunResult fields, crossing tiers for cliPath
experimental experimental subpath symbol list (escape hatch, no compatibility promise)

Runtime guarantees

The runner uses try/finally so temp directories and links are cleaned up on every path (prepare throwing, mock validation failure, spawn errors) — the real profile store is never polluted. The behavior and review CLIs share directory scanning (skipping .runs and node_modules); the behavior CLI validates case shapes and detects cross-file duplicate ids at load time, failing as early as possible.

License: MIT. The framework's own tests and release self-checks are carried by the repository CI and do not ship with the package. Release history: CHANGELOG.md.

—/ 5

No ratings yet

Manifest verification required

Commit a52d5c025b59

Community comments

No comments yet. Be the first to write one.

DSH HUB

A community index for DSH plugins. Not an official GitHub or DeepSeek AI product.

CommunityResourcesAPIAbout