dsh-youtube-study-notes

A DeepSeek Harness (dsh) plugin that turns a YouTube link into study notes — and then gets your notes involved.

Give it a link. It pulls the video's own captions, samples the frames so it can read what was on screen, stops and asks what you took from it, and merges all three into one document.

The middle step is the point. Every other video-to-notes tool produces the same summary for everyone. This one writes the notes you'd have written, if you'd been taking notes.

What it does

Step	What happens
1	Captions — yt-dlp fetches the caption track YouTube already publishes. Human-written preferred, auto-generated as fallback. No ASR, no third-party transcript sites.
2	Outline — the timestamped transcript is read in full and outlined by chapter.
3	Visual pass (optional) — ffmpeg scene-change detection samples one frame per shot; a vision model reads the on-screen text. Catches slides, code, formulas and diagrams the narration never says out loud.
4	Ask — the agent stops and asks what you noticed, in your words. Type it or dictate it.
5	Merge — transcript + on-screen evidence + your observations, into one markdown file.
6	Save — notes.md, written beside the transcript so the evidence stays with it.
7	Clean up — the frame stills and any video file are removed; the transcript stays.

What it leaves on disk

The video is never kept: step 3 downloads it into a temporary directory that is deleted when the script exits (--keep-video is the only way it survives). The frame JPEGs are removed by step 7 once notes.md exists — they are derived from someone else's video and are typically tens of megabytes, against a few kilobytes of transcript.

Kept, because it is the evidence behind every spoken claim and costs almost nothing:

transcript.md  transcript.txt  meta.json  notes.md

cleanup.py refuses to act on a directory that does not carry one of those markers, so it cannot be pointed at an unrelated folder. Pass --keep-frames to review the stills yourself.

Install

dsh plugin --profile <your-profile> add dsh-youtube-study-notes

Restart the profile afterwards — bundle changes apply at boot.

Then trigger it either way:

/youtube-study-notes https://youtu.be/VIDEO_ID

or just ask in plain language ("get me notes from this video: …"). A link plus a request for notes, a summary, or study material is enough.

Desktop clients: the desktop app manages its own profile. Use its Plugins → Add plugin flow rather than the CLI, and enable the bundle when the install finishes.

Requirements

The plugin ships no runtime code — its substance is the Skill, which drives local tools. dsh does not install these for you:

Tool	Role	Install
python3	runs the bundled scripts (standard library only)	brew install python3
ffmpeg / ffprobe	audio/video probing, scene detection, frame extraction	brew install ffmpeg
yt-dlp	caption listing and download	brew install yt-dlp

The Skill detects missing tools, names them, and waits for your confirmation before installing anything.

Enabling the visual pass

Step 3 needs a model that accepts images, and many model entries do not — even ones whose name says "vision". A model is only offered images if its configuration declares input: [text, image]; entries otherwise default to ["text"], and the harness refuses the image before attaching it with:

model "<provider>/<model>" does not declare image input

The Skill handles this: it tries read_image on a single frame first. If the session model can see images, it reads the frames directly. If not, it delegates through the workflow tool's per-agent provider / model override, and asks you once which image-capable model to use.

To make a model explicitly image-capable, declare it in your profile's cordis.patch.yml (the file your profile tells you to edit — not cordis.yml):

- id: llm-pi-ai
  name: "@deepseek-ai/dsh-llm-pi-ai"
  config:
    providers:
      your-provider:
        # ...existing provider config...
        models:
          - id: "your-vision-model"
            name: "Your Vision Model"
            contextWindow: 200000
            input:
              - text
              - image

A profile with live reload picks this up without a restart.

Only declare this if the endpoint actually serves images. Over-claiming admits an image the provider then rejects mid-turn, after the message is already durable. The declaration is not verified for you.

If you skip all of this, the plugin still works — it just produces transcript-only notes, and says so rather than pretending it saw the screen.

Limitations

• Caption-less videos stop the run. If a video publishes no caption track at all (not even auto-generated), step 1 reports which video failed and stops. Nothing is invented to fill the gap. Pair this with a local speech-to-text plugin if you need those covered.
• Auto-generated captions are rough — no punctuation, misheard proper nouns. The skill always reports which kind it got.
• Vision reads can be wrong at the character level. A period can read as a hyphen, 1 as l. Fine for prose, worth double-checking in code and formulas. The skill is instructed to flag anything load-bearing that looks ambiguous.
• Frame sampling sees shots, not motion. A diagram built up over one continuous shot yields a single frame.
• Personal study use. Captions, frames and notes are derived from someone else's work. Don't republish them.

How it's built

package.json          declares dsh.bundle → cordis.patch.yml
cordis.patch.yml      inserts one @deepseek-ai/dsh-skill-filesystem row
lib/index.js          resolveSkillRoot() — resolves skills/ from the profile's own module identity
skills/youtube-study-notes/
  SKILL.md            the workflow
  scripts/            yt_transcript.py, video_frames.py, cleanup.py
  references/         notes template and merge rules

The bundle mounts its skills/ directory as an isolated provider (includeDefaultRoots: false), so installing it contributes exactly one skill and cannot shadow or reorder the ones you already have.

License

MIT — see LICENSE.