English · 简体中文

DSH Plugin A/B Test

Test a DSH plugin change on the same tasks before you ship it.

DSH Plugin A/B Test runs your current plugin (Control) and proposed change (Candidate) in isolated DSH environments, pairs their results case by case, and produces evidence you can review before release.

It helps answer three practical questions:

• Did the Candidate improve task success?
• Did that improvement come with a meaningful regression in tokens, latency, or tool errors?
• Can someone else reproduce the result from the same inputs?

Every experiment ends with one of four deterministic outcomes: PROMOTE, REVIEW, REJECT, or INCONCLUSIVE. Even PROMOTE is an offline recommendation only—this project never changes your real DSH profile or publishes a plugin for you.

See the decision first

This is a real result from the repository's offline example, with unrelated fields omitted:

{
  "outcome": "PROMOTE",
  "validPairCount": 2,
  "invalidPairCount": 0,
  "triggeredRules": ["primary.superiority"]
}

Alongside the decision, you get raw session evidence, assertion results, pair-level deltas, and reports in JSON, Markdown, and HTML. The model does not choose the outcome; deterministic rules from the experiment manifest do.

Quick start

The current MVP runs from source and requires Node.js ^22.19.0 || >=24.0.0 and pnpm 11.19.0. The starter experiment uses an offline scripted provider, so no model API key is required.

pnpm install --frozen-lockfile

node --import tsx src/cli/bin.ts init --output ./my-experiment --json
node --import tsx src/cli/bin.ts freeze --manifest ./my-experiment/experiment.yml --output ./evidence --json
node --import tsx src/cli/bin.ts run --manifest ./my-experiment/experiment.yml --output ./evidence --json
node --import tsx src/cli/bin.ts decision --manifest ./my-experiment/experiment.yml --output ./evidence --json
node --import tsx src/cli/bin.ts report --manifest ./my-experiment/experiment.yml --output ./evidence --json

Open ./evidence/<experiment-id>/report.html to view the static report. To test your own plugin, edit experiment.yml, evals/cases.yml, and the two variant configurations created by init.

Understand the outcome

Outcome	What it means	Typical next step
PROMOTE	Evidence is sufficient, quality meets the target, and guardrails pass	Continue through your human release process
REVIEW	Results improved, but cost, latency, error rate, or variance needs judgment	Review the pair-level evidence
REJECT	A hard gate failed, a critical case regressed, or the gain was too small	Fix the Candidate and rerun
INCONCLUSIVE	There were too few valid pairs, exposure was not proven, or environments were not comparable	Complete the evidence instead of treating it as a failure

Task success is the default primary metric. You can also guard token usage, P95 latency, and tool error rate. Thresholds, minimum valid pairs, repetitions, and concurrency all live in the manifest. See the manifest reference and decision rules for details.

Why the evidence is trustworthy

• Paired tasks: Control and Candidate receive the same case, workspace fixture, and model parameters.
• Balanced order: Pair order alternates to reduce fixed first-run bias.
• Isolated environments: Each arm gets its own DSH_HOME, profile, workspace, session root, and frozen plugin artifact.
• Traceable artifacts: Sources may be a local directory, tarball, exact npm version, or pinned GitHub commit; hashes are checked again before execution.
• Proven exposure: Session events, tool calls, or plugin receipts show whether the target plugin actually participated.
• Blind comparison: An optional comparator sees anonymous A/B outputs; identity is revealed only after comparison.
• Honest infrastructure failures: Provider outages, corrupted sessions, and environment mismatches become invalid evidence or INCONCLUSIVE, not fake Candidate regressions.

Evidence is stored under one experiment directory:

<output>/<experiment-id>/
├── manifest.lock.json
├── control-artifact.json
├── candidate-artifact.json
├── pairs/<case-id>-<repetition>/
│   ├── pair.json
│   ├── measurement.json
│   ├── control/
│   └── candidate/
├── comparison.json
├── decision.json
├── report.md
└── report.html

Completed pairs are reused by later run commands, and partial state is never silently overwritten. If inputs change or evidence is damaged, use a new output root.

Connect to real DSH

Non-mock providers invoke the exact pinned version @deepseek-ai/dsh@0.1.0-rc.7. Add model credentials and other environment variables by name to extensions.environment_allowlist. Fingerprints and artifact metadata store only the allowlist hash, never the original values.

Plugin and test-command stdout, stderr, and session logs are retained as raw evidence, so integrations must still avoid printing secrets.

The CLI is a trusted local automation boundary and may run commands explicitly declared in the manifest. Optional DSH/Cordis tool entry points are model-facing, so they reject experiments containing command_test rather than becoming arbitrary command-execution tools.

The complete CLI includes init, validate, freeze, run, status, compare, decision, and report. Stable exit codes and the live-model smoke procedure are documented in the verification record.

Development

pnpm identity:check
pnpm lint
pnpm typecheck
pnpm test
pnpm test:integration
pnpm test:rename
pnpm build
pnpm pack --dry-run

This project is currently an MVP and is not published as an npm package. See the architecture and upstream audit for implementation and security boundaries.

Public names, the npm package name, and the CLI name are managed from project.identity.json. A real rename test prevents stale public identity from surviving a rename; see the identity guide. Stable protocol identifiers do not change with the project brand.