agent-model-scout
English | 简体中文
/scout <task> — Probe every configured model with one cheap planning call, rank the plans by quality vs. estimated token cost, and optionally switch this agent onto the winner.
Scout is a pre-flight check, not a router daemon: it gathers cheap intel (a plan), scores it locally, and lets you or the command make the call. It answers "which model plans this task better for the money", not "which model will finish it".
Features
- Multi-model probing: Test all configured models on the same task with a single planning call
- Quality vs. cost scoring: Rank models by utility (quality − λ·normalized cost)
- Interactive picker: Choose a model or keep current one after seeing the results
- Reasoning effort selection: For models that expose reasoning levels, pick execution intensity
- Re-probe optimization: When top models are close, re-probe them and average the results
- Calibrated token estimation: Uses real billing data to predict token usage
- Outcome log & priors: Settles dispatched tasks against session logs; per-route calibration replaces guesswork
- Self-validation: Built-in tests verify stability and accuracy
- Cordis plugin: Integrates seamlessly with DeepSeek Harness
scout · 3 candidates · λ=0.50
task: make the config loader support .toml and add tests
# MODEL Q EST.TOK U TIME NOTE
1 deepseek-official/deepseek-flash 0.74 58k 0.61 3.1s 4 steps, 2 files
2 acme/big 0.90 220k 0.40 5.7s 6 steps, 5 files
3 acme/tiny 0.11 18k 0.07 1.2s 1 step, 0 files
top plan: 1) read src/config/loader.ts 2) add a toml branch inside parseConfig() …
files: src/config/loader.ts, test/loader.test.ts
▸ recommend: deepseek-official/deepseek-flash — quality 0.74, est. 58k tokens (7 turns)
▸ dry run — add --apply to switch this agent's model.
Status
v0, manual trigger only. It answers "which model plans this task better for the money", not "which model will finish it". Learned priors, price tables, auto pre-flight, and mid-run escalation are deliberately out of scope (see Roadmap).
Why probe instead of asking models to self-score
Asking a model to estimate its own token usage or grade its own answer is unreliable: tokenizers differ per provider, models cannot count tokens, and self-evaluation carries a known self-preference bias. Scout splits the job:
| Question | Who answers |
|---|---|
| What would you do? | the candidate model (a plan, as JSON) |
| How good is that plan? | a fixed local rubric |
| How many tokens will it take? | a local cost model |
The model only ever supplies a plan. Everything that becomes a number is computed here, so the scoring is inspectable, tunable, and replayable.
Install
Requires DeepSeek Harness with a profile that has at least two configured models. The examples target the desktop profile — adjust if yours differs. All commands run in PowerShell.
1. Get the code
git clone https://github.com/ankepoipoi/agent-model-scout.git
cd agent-model-scout
2. Copy the package into the profile
Close DeepSeek Harness first, then:
$dest = "$env:USERPROFILE\.dsh\profiles\desktop\node_modules\agent-model-scout"
New-Item -ItemType Directory -Force $dest | Out-Null
Copy-Item -Recurse -Force .\package.json, .\lib $dest
Or run .\install.ps1, which does this step for you (idempotent; -DryRun shows the plan first).
3. Register the plugin
Append this entry to the end of %USERPROFILE%\.dsh\profiles\desktop\cordis.patch.yml:
- insert:
- id: scout
name: "agent-model-scout"
config:
candidates: [] # explicit "provider/model" routes; empty = auto-discover
lambda: 0.5 # cost weight: 0 = quality only, 1 = cost dominates
topK: 4 # max candidates probed when discovering
maxProbeTokens: 500 # output cap for one probe
probeTimeoutMs: 60000 # wall-clock budget for one probe
reprobeGap: 0.15 # re-probe the top-2 when their gap is below this
fixedOverheadTokens: 30000 # per-turn system prompt + tool schema cost
insert:is required. A plain- id: scoutentry patches an existing entry and is silently skipped when the id is not found.
4. Restart and verify
Restart DeepSeek Harness fully (quit from the tray, not just close the window), then run a zero-cost check:
/scout any task here --dry
--dry lists the discovered candidates without probing — it spends nothing and proves the plugin is loaded. Drop --dry for the real run.
The plugin declares no Config schema: it reads a plain object from the patch entry, so it does not couple to a schemastery version.
Configuration
- id: scout
name: "agent-model-scout"
config:
candidates: [] # explicit "provider/model" routes; empty = auto-discover
lambda: 0.5 # cost weight: 0 ranks by quality only, 1 lets cost dominate
topK: 4 # max candidates probed when discovering
maxProbeTokens: 500 # output cap for one probe
probeTimeoutMs: 60000 # wall-clock budget for one probe
probeTemperature: 0 # pin sampling so two runs are comparable
reprobeGap: 0.15 # re-probe the top-2 when their gap is below this; 0 disables
fixedOverheadTokens: 30000 # per-turn system prompt + tool schema cost of the real task
priorMinDispatches: 2 # settled dispatches a route needs before its prior calibrates estimates
Candidate resolution order: --models flag → candidates config → auto-discovery (ctx.llm.listProviders() + listModels()) → the agent's current model.
Usage
Every successful probe ends in a picker card, and a picked model runs the task immediately — the loop is closed: probe → pick → execute.
/scout
/scout make the config loader support .toml and add tests
/scout refactor the session storage layer --apply
/scout ... --models deepseek-official/deepseek-flash,acme/big --top 2
| Flag | Effect |
|---|---|
| (no task) | ask for the task, probe, pick, run |
| (task) | probe it, pick, run |
--apply |
switch onto the winner and run the task without asking |
--dry |
list the candidates that would be probed, spending nothing |
--trust |
rank from outcome-log priors alone, no probes; needs every candidate to have settled dispatches |
--models a/b,c/d |
probe exactly these routes |
--top N |
cap how many discovered candidates are probed |
--lambda X |
cost weight for this run (0 = quality only, 1 = cost dominates) |
Picking the keep-current option runs the task without switching; dismissing the card runs nothing. The hand-off is agent.followup() with a frozen user message, so the task enters the session log as an ordinary turn and the recorded model selection takes effect on that turn's first step.
A chosen model that exposes reasoning efforts gets a second card: how hard it should think while executing (its own effort ids plus 默认, the model's default). The suggestion follows the plan — risky or many-stepped work earns the deepest effort, simple work the cheapest — and --apply applies the suggestion without asking. Efforts are discovered per model via ctx.llm.resolveModelInfo(); a model without an effort catalog skips the card silently, and the keep-current path leaves the effort untouched.
The picker is rendered through ctx.userQuestions.ask(), the same service behind the ask_user_question tool, so it appears as a normal question card. On a surface without a question UI (headless), the wizard refuses and the flag-driven forms still work.
--apply calls the native agent.model.select(), which is durable in the session projection and takes effect on the next step. The decision record is the command's own log entry plus the report text in command/done, so a replayed session still shows why the model changed.
Outcome log (v0.3)
Every dispatch appends one record to ~/.dsh/agent-model-scout/outcomes.jsonl, keyed by the scout- id of the injected task message. The next /scout run reconciles pending records against the session logs (same multi-frame zstd pipeline as the self-check), matching each message id to the turn that consumed it and settling it with what actually happened: the route that ran, the step count, and the turn's billed tokens from data.usage. A record settles only after something follows its turn in the same file, because the newest turn may still be streaming.
Settled records become per-route priors. Once a route has priorMinDispatches settled dispatches, its token estimate is multiplied by its own mean actual/predicted ratio — the ranking starts pricing each model by its measured behavior instead of the shared estimate. The report shows it in two places: a cal ×N.NN marker in the row's NOTE column and a ▸ priors: footer line.
Settling also classifies the first user message after the dispatched turn: a repeat of the task or an explicit complaint (重试, retry, failed, …) is a re-ask — evidence the run underdelivered — anything else is a continuation. Re-ask rate is the only quality signal that survived the self-check (the probe rubric carries none), so priors track it per route and the picker surfaces it as 历史重问 N%.
--trust consumes those priors without probing at all: when every candidate has settled dispatches, scout ranks by measured behavior (mean actual tokens, quality = 1 − re-ask rate) and spends nothing on probes. The rows are marked prior mode because their numbers are cross-task historical averages, not per-task estimates — trust mode is for when probing itself is the cost you want to avoid. If any candidate lacks priors, --trust degrades to a normal probe run and the report says why.
The log is local, append-mostly, and best-effort: a missing or corrupt file costs nothing but priors, and reconcile never throws into /scout. Test results as an outcome signal are the remaining v0.3 work.
How the score is computed
Quality rubric (0–1)
| Term | Weight | Notes |
|---|---|---|
| structure | 0.30 | step count, saturating at 6 |
| scope | 0.15 | files named, saturating at 2 |
| concreteness | 0.25 | share of steps naming a path, symbol, or code span |
| coverage | 0.20 | plan both edits and verifies (edit + test/bash/grep/read) |
| risk | 0.05 | low full, medium half, high zero |
| confidence | 0.05 | self-reported, deliberately the smallest term |
| blockers | −0.10 | −0.05 each, capped at −0.10 |
| grounding | +0.10 / −0.20 | every planned file exists on disk (+0.10); any missing (−0.20); no files listed (0) |
Grounding outweighs every shape term on purpose: a plan that names files which do not exist is guessing. The check runs fs.existsSync against the session cwd, so absolute paths and cwd-relative paths both count.
scope saturates at two files because most tasks legitimately touch one or two; an earlier version rewarded naming many files and pushed models toward inventing extra targets.
Token estimate
The loop re-sends the whole context every turn, so input cost grows quadratically with turn count:
total ≈ (fixed + task)·turns + 1200·turns(turns+1)/2 + 500·turns
turns is the plan's est_turns, floored by step count (turns >= steps): calibration on dispatched turns showed every planned step tends to cost one call (5 steps → 5 calls). The per-turn fixed overhead (30,000) is likewise calibrated: the first two dispatched turns billed 36k–47k tokens per call — the original 3,000 assumption underestimated 7–9×. test/selfcheck.mjs section 4 re-runs this calibration (actual/predicted) whenever dispatched turns exist in the logs.
Coefficients live at the top of lib/estimate.js. Text density uses a provider-neutral heuristic (CJK ≈ 0.6 tokens/char, ASCII ≈ 0.3 tokens/char) because a shared tokenizer would be wrong for every provider except one.
Ranking
utility = quality − λ · (candidate tokens / max tokens among candidates)
Known limitations
- The rubric is a proxy, not a success rate. v0 exists to test whether plans differ enough to predict execution.
- Ranking is unstable run to run. On repeated runs of the same task, the winner has changed every time: the rubric scores plan shape, and shape varies at least as much within one model as between models. A 9-run self-check (
test/selfcheck.mjs) on one task measured mean within-model Q drift 0.45 vs mean between-model spread 0.26 — at that noise level the ranking carries no signal. The same check found deepseek models pinned at temperature 0 (±0.02 across runs) while glm-5.3 drifted 0.54, so determinism differs per provider even withtemperature: 0sent. Treat/scoutas a way to compare how models reason, not as an automatic router. - Unverified rows say so. When the session cwd cannot be resolved, planned files are not checked; the NOTE cell reads
· unverifiedand grounding stays neutral rather than awarding the bonus. - Probing costs money: one call per candidate.
topK=4is roughly 4 × (input + 500 tokens). - Tokenizers differ: the estimate is a shared density heuristic, not a provider-exact count.
- No capability filter yet: a model whose context window cannot hold the session is still probed.
- Switching costs cache: crossing providers drops the prompt cache; v0 does not price that.
- Probe requests are not in the session log: only the decision is durable.
Roadmap
- v0.1 — context-window/capability filtering, price table (USD), probe cache by task fingerprint.
- v0.2 — auto pre-flight via
agent/pre-step, budget guard, fallback chain. - v0.3 — outcome log → per-model priors. Landed: dispatch records, session-log reconciliation, per-route calibration priors, re-ask classification, probe-free
--trustmode. Remaining: test-result signal. - v1 — mid-run escalation on repeated tool errors / failing tests / loops; Web UI routing card.
Development
Pure logic (estimation, plan parsing, scoring, ranking, rendering) has no DSH dependency and runs standalone:
$env:ELECTRON_RUN_AS_NODE='1'
& "D:\deepsh\DeepSeek Harness.exe" test\run.mjs # 19 assertions
& "D:\deepsh\DeepSeek Harness.exe" test\demo.mjs # renders a report from fixtures
& "D:\deepsh\DeepSeek Harness.exe" test\selfcheck.mjs # mines session logs: score stability, probe failures, dispatches
lib/index.js consumes only the host context (ctx.commands, ctx.llm) and carries no imports from @deepseek-ai/* — third-party plugins are resolved from the profile's own package scope, where the host packages are not visible. Requests are hand-built plain objects and the probe reads raw text-delta chunks instead of a host assembler.
Publishing
Run .\publish.ps1 -Owner <your-github-name> from a normal PowerShell window (not inside a sandboxed agent, which has no network egress). With gh authenticated it creates the repo and pushes; without gh it commits locally and prints the two commands to finish in the browser.
License
MIT
No comments yet. Be the first to write one.