dsh-plugin-elevenlabs-callback

A DeepSeek Harness plugin that calls you back by voice.

You give the harness a long task and walk away. When the turn finishes, fails, or stops to ask for approval, the plugin sends a link to your phone. Opening it starts a conversation with an ElevenLabs voice agent that already knows what happened. It reads you the short version, answers questions about the run, and — once you confirm the wording — sends your next instruction (or your approve/deny) straight back into the session through webhook tools served by the plugin.

Built in a weekend on top of the ElevenLabs Agents API and the dsh plugin API. dsh is in developer preview; this plugin was written and tested against @deepseek-ai/dsh 0.1.1-rc.2 and will need updating when the plugin API changes.

Prerequisites · Install · First callback · Configuration · Try it without a harness · Real phone calls · Troubleshooting

What a callback sounds like

Agent: Hi, this is your DeepSeek Harness calling about retry-refactor. It has finished. Done. I moved the retry logic into http/retry.ts, added exponential backoff with jitter, and wrote six tests for it. Want more detail, or should I tell it what to do next? You: Tell it to run the tests again and commit if they pass. Agent: So, you want me to tell it to run the tests again and commit if they pass? You: Yes. → send_instruction → the message lands in the session as a follow-up turn Agent: I've sent the instruction. It will call back when it's done.

(Transcript from pnpm live-check, text mode, against the real agent and real webhook tools.)

The same loop was then run against a real dsh session (DeepSeek-V4-Flash, a 40-second turn with 14 tool calls that fixed a failing test and committed): the plugin sent the link, the voice agent summarised the run from the session log, and the spoken instruction arrived in the session as a follow-up turn — visible in the dsh UI as Context injection · elevenlabs-callback · instruction given by voice callback. Replies came 0.9–1.9 s after each message (average 1.3 s); the webhook was answered in 2.6 ms. A later run from a real handset (“improve the function”, spoken) produced a 29-second follow-up turn that refactored the function, passed the tests and committed, after which the plugin called back again.

How it works

sequenceDiagram
    participant H as dsh session
    participant P as plugin (in the dsh process)
    participant N as your phone
    participant E as ElevenLabs Agents

    H->>P: agent/status idle (turn/end in the session log)
    P->>P: gather last message, tool calls, duration → debrief
    P->>N: link (terminal + QR, ntfy push, desktop, or any command)
    N->>P: open link, tap Answer → conversation token
    N->>E: WebRTC call with dynamic variables (summary, status, debrief id)
    E->>N: spoken summary
    N->>E: "tell it to rerun the tests and commit"
    E->>P: POST /tools/send_instruction (X-Tool-Token, debrief id filled by the platform)
    P->>H: agent.followup(message) — or steer() if it is still running
    E->>N: "Sent. It will call back when done." → end_call

Three webhook tools are wired to the agent: get_run_details (the full last message and every tool call of the turn), send_instruction, and resolve_approval. The debrief_id argument is marked as a dynamic variable in the tool schema, so the platform fills it from the conversation itself and the language model never has to repeat an identifier.

Approvals are a race: when a tool is waiting for permission, the plugin sends a link and lets the normal keyboard prompt run. Whichever answers first wins; the other side is told it was already handled.

Prerequisites

you need	why
Node.js ≥ 22 and pnpm on your PATH	dsh plugin drives pnpm to install plugins into a profile
DeepSeek Harness @deepseek-ai/dsh 0.1.1-rc.2, already working	npx @deepseek-ai/dsh web must start and run a turn on its own (model provider and API key configured) before you add this plugin — it only listens to sessions, it does not create them
An ElevenLabs account and API key	the plugin creates one agent, three webhook tools and one workspace secret in your workspace on first start. Create the key under Settings → API Keys; the free tier is enough for a few callbacks a day (about 15 agent-minutes a month)
A public HTTPS URL for the plugin's small server: either cloudflared on your PATH (no account needed; the plugin opens a quick tunnel by itself) or your own tunnel / reverse proxy set as publicUrl	ElevenLabs must be able to reach the webhook tools, and your phone must be able to load the page. Browsers only allow the microphone on HTTPS
A phone (or any device) with a modern browser and a microphone	the page runs the call over WebRTC using @elevenlabs/client
Optional: the ntfy app	to have links pushed to your phone instead of reading a QR code off the terminal
Optional: a Twilio number	for a real ringing phone instead of a link — see Real phone calls

Tested on macOS with dsh 0.1.1-rc.2 (the npm latest at the time of writing). Desktop notifications use osascript on macOS and notify-send on Linux.

Install

Not on npm yet — install from GitHub or from a local checkout.

From GitHub (pnpm fetches the sources and builds them with the package's prepare script):

dsh plugin --profile web add github:StephenEvenson/dsh-plugin-elevenlabs-callback

pnpm refuses to run build scripts of git dependencies until you allow them. If the first add prints a warning about ignored build scripts, add this to ~/.dsh/profiles/web/pnpm-workspace.yaml and run the same add again:

allowBuilds:
  dsh-plugin-elevenlabs-callback: true

Only grant this to packages you trust — it runs the package's build at install time.

From a local checkout (no build permission needed; the checkout is linked into the profile, so git pull && pnpm build updates it):

git clone https://github.com/StephenEvenson/dsh-plugin-elevenlabs-callback.git
cd dsh-plugin-elevenlabs-callback && pnpm install && pnpm build
dsh plugin --profile web add "$PWD"

Check the layer without booting:

dsh --profile web --dump-config | grep -A2 "id: elevenlabs-callback"

Give it the API key and start the profile. The plugin reads ELEVENLABS_API_KEY from the process environment (dsh's own ~/.dsh/.env is not exported to plugins), or apiKey from the profile patch — see Configuration:

export ELEVENLABS_API_KEY=sk_...
dsh web

On start-up the plugin binds 127.0.0.1:7331, opens the tunnel, then creates (or updates) the agent and its three tools on ElevenLabs. You will see:

[elevenlabs-callback] ready — agent agent_…, links will point at https://….trycloudflare.com

Every later start reuses the same agent and tools (their ids are kept in ~/.dsh/elevenlabs-callback/agent.json) and only pushes the current settings and public URL.

Update: in a linked checkout, git pull && pnpm build; for a GitHub install, re-run the add github:… command. Restart dsh web afterwards — bundle membership only changes on restart.

Uninstall:

dsh plugin --profile web remove dsh-plugin-elevenlabs-callback

then restart dsh web. The plugin's state stays in ~/.dsh/elevenlabs-callback/ and the agent stays in your ElevenLabs workspace (delete it under Agents); remove both if you want a clean slate.

First callback

1. In the dsh web UI, give the harness something that takes longer than 20 seconds (a refactor with tests, a scripted change across files) and step away.

2. When the turn ends, the terminal running dsh web prints the link with a QR code:

   [elevenlabs-callback] Harness: my-project — Finished, waiting for you
   [elevenlabs-callback] open on your phone: https://….trycloudflare.com/d/2p4yc6?k=…
   

   With notify.ntfyTopic set, the same link is pushed to the ntfy app; with notify.desktop, it also appears as a desktop notification.

3. Open the link on your phone, tap Answer, and allow the microphone. The agent speaks first: which project, what happened, and the first sentence of the harness's last message.

4. Ask for detail (“what did it change?”), or tell it what to do next. The agent reads your instruction back and only sends it after you say yes.

5. The instruction arrives in the dsh session as a follow-up turn (as a steer message if the harness is still running). When that turn ends, you get the next link.

For approvals the flow is the same, except that the agent asks the question (“It wants to run rm -rf build. Allow it?”) and calls resolve_approval; the keyboard prompt in dsh keeps working in parallel.

Configuration

Settings live in your profile's cordis.patch.yml (~/.dsh/profiles/web/cordis.patch.yml). Target the row by its id, elevenlabs-callback. A patch replaces the whole config object, so repeat every key you want to keep:

- id: elevenlabs-callback
  config:
    apiKey: sk_...                                   # or leave out and export ELEVENLABS_API_KEY
    triggers: { finished: true, error: true, approval: true, minTurnSeconds: 20 }
    notify: { terminal: true, qr: false, desktop: true, ntfyTopic: my-private-topic, ntfyServer: https://ntfy.sh, command: "" }

key	default	meaning
apiKey	""	ElevenLabs API key; empty = read ELEVENLABS_API_KEY from the environment
publicUrl	""	https URL where the plugin server is reachable; empty = start a cloudflared quick tunnel
host, port	127.0.0.1, 7331	where the plugin server listens (0 = free port)
stateDir	$DSH_HOME/elevenlabs-callback	agent/tool ids and the generated tool token
agent.llm, agent.voiceId, agent.ttsModel	gemini-2.5-flash, Charlie, eleven_flash_v2	agent settings pushed at start-up (English agents need eleven_flash_v2 or eleven_turbo_v2)
triggers.finished / .error / .approval	true	which events send a link
triggers.minTurnSeconds	20	completed turns shorter than this are ignored — you were probably watching
linkTtlMinutes	120	how long a link can start a call
notify.terminal, notify.qr	true	print the link (and a QR code) to the terminal
notify.desktop	false	macOS / Linux desktop notification
notify.ntfyTopic, notify.ntfyServer	"", https://ntfy.sh	push the link to the ntfy app
notify.command	""	any shell command; gets DSH_CALLBACK_URL, _TITLE, _BODY in its environment
phone.toNumber, phone.fromNumber	""	ring a real phone instead of only sending a link — see below

Patch edits to config hot-reload; the plugin restarts its server and re-provisions the agent with the new settings.

Try it without a harness

cp .env.example .env            # ELEVENLABS_API_KEY, TOOL_SHARED_SECRET
pnpm install
pnpm demo                       # --status finished | needs_approval | error

pnpm demo runs the same server, tunnel and provisioning with one made-up debrief and a fake harness that prints whatever the voice agent sends. It uses its own agent (… (standalone demo)) so it never rewires the plugin's. Open the printed link on your phone and talk to it; or drive it from the terminal in text mode, which spends no audio minutes:

pnpm live-check --link "https://…/d/abc123?k=…"

This opens a real conversation over WebSocket, sends a four-line script, and checks that send_instruction was called through the public URL. Observed on the run recorded above: agent replies 1.0–2.1 s after each message (average 1.4 s); the plugin answered the webhook in 0.2 ms.

Real phone calls (optional)

Links work everywhere; a ringing phone is nicer. ElevenLabs does not sell numbers itself, so you bring one from Twilio (or any SIP trunk):

1. In the Twilio console: buy a number with voice capability (a US number is instant; other countries need address/identity documents). ElevenLabs cannot import numbers from a Twilio trial account (Twilio error 20003), so the account must be upgraded to pay-as-you-go first.

2. In the ElevenLabs dashboard: Agents → Phone numbers → Import from Twilio, using the Account SID (AC…) and Auth Token from the Twilio console (not an API key).

3. In the profile patch:

   - id: elevenlabs-callback
     config:
       phone: { toNumber: "+61400000000", fromNumber: "+15550001111" }
   

From then on every callback rings your phone through POST /v1/convai/twilio/outbound-call with the same dynamic variables the web page would use — the agent speaks first, the tools work identically — and the link is still sent as a fallback. pnpm demo --call +61… (with PHONE_FROM in .env) tries it without a harness. Twilio charges per minute on top of ElevenLabs' agent minutes.

Troubleshooting

• cloudflared: command not found / “tunnel did not report a URL” — install cloudflared, or run your own tunnel and set publicUrl. The plugin refuses non-https URLs because ElevenLabs will not call them.
• No link after a turn — the turn was shorter than triggers.minTurnSeconds, was interrupted/aborted, or ended before the plugin started (only turns that end after start-up count). Check the [elevenlabs-callback] lines in the terminal; ctx.logger output is not shown by dsh web, so everything user-facing is printed with plain console.log.
• The page loads but Answer fails — the microphone needs HTTPS and a permission grant; on iOS Safari the first tap must be a user gesture. A link older than linkTtlMinutes answers 404.
• Tools answer 401 — the shared token in stateDir/tool-secret no longer matches the workspace secret on ElevenLabs. Restart the profile; the plugin re-pushes the secret value on every start.
• The agent talks over you or in the wrong voice/model — change agent.* in the patch; the plugin PATCHes the live agent on the next start. English agents must use eleven_flash_v2 or eleven_turbo_v2.
• “Unusual activity” / quota errors from ElevenLabs — the free tier shares its monthly agent minutes across every agent in the workspace; pnpm live-check uses text mode and spends none.

Development

pnpm install
pnpm test        # 40 tests: debrief model, HTTP surface, session-log parsing, config schema,
                 # and the whole loop inside a real Cordis context with a fake agent registry
pnpm typecheck   # against the published @deepseek-ai/dsh-* type definitions
pnpm build       # tsc → lib/

Every webhook call is logged with its latency (tool=send_instruction status=ok latency_ms=0.2 …) and visible at /api/tool-log. Tool endpoints answer 200 with a structured ok/found field even when something is wrong, so the voice agent can say what happened instead of stalling on an HTTP error.

To iterate without reinstalling: link the checkout as above; after pnpm build, restart dsh web.

Security notes

• The tunnel exposes only this plugin's server, never the dsh web UI.
• Each link carries its own random key; without it the page and the token endpoint answer 404. Links expire.
• ElevenLabs authenticates to the tools with a token generated on first run, stored as a workspace secret on their side and in stateDir on yours; anything else gets 401.
• The agent is private (enable_auth), capped at one concurrent call, 50 calls a day and 5 minutes per call.
• An instruction is only delivered after the agent has read it back and heard a yes; a turn accepts one instruction per callback.
• Nothing is sent anywhere but ElevenLabs (and ntfy / Twilio if you enable them); the debrief contains the harness's last message and tool-call summaries, so treat the link like the session itself.

Compatibility

dsh	status
0.1.1-rc.2 (npm latest)	tested end to end, including a real handset call
0.1.2-alpha.x (GitHub main)	not tested; the plugin API is a developer preview and is expected to break

Peer ranges in package.json are open-ended (>=0.1.1-rc.2); pnpm typecheck runs against the pinned rc.2 type definitions.

The plugin does not resume a session that has been disposed; the link still works for reading, but send_instruction will say the session is gone.

Layout

src/index.ts              the Cordis plugin: config schema, triggers, approval race, delivery
src/harness.ts            turns a dsh session log into a debrief (pure, unit-tested)
src/debrief.ts            debrief store, per-link keys, spoken opening, dynamic variables
src/server.ts             node:http server: phone page, token endpoint, three webhook tools
src/notify.ts             terminal + QR, desktop, ntfy, shell command
src/tunnel.ts             cloudflared quick tunnel
src/elevenlabs/           agent definition as code, minimal API client, idempotent provisioning
web/                      the phone page (plain HTML/JS on the @elevenlabs/client IIFE bundle)
scripts/                  demo (no harness needed), live-check (text mode, real tools), provision
cordis.patch.yml          the layer `dsh plugin add` merges into a profile

License

MIT