dsh-zotero-wiki

中文 | English

DeepSeekHarness（dsh）× Zotero 插件：自动同步 Zotero 文献库，用 MinerU 解析 PDF、用 DeepSeek 阅读全文，把文献编译成可检索的 Obsidian Wiki，并让 Agent 在回答时附带可追溯的文献引用。

---

项目定位（范围）

是什么：一个 Node/Cordis 风格的 dsh 插件 + 独立 CLI。它把你的 Zotero 文献库变成一座持续更新的 Obsidian 知识库，并暴露原生工具供 Agent 检索。

完整链路：

Zotero（本地库 / Web API）
  └─ sync：增量同步元数据 → .zotero-sync/sync.db（SQLite 状态库）
      └─ parse：MinerU 官方平台解析 PDF → raw/<topic>/<bundle>/ 原文 + DeepSeek 笔记（<key>-note.md 同束存放）
          └─ compile：主题综述（_index.md）、索引、交叉链接、归档标记
              └─ query：Agent 通过 query_zotero_wiki 检索 raw 笔记，回答附 [Zotero key, §章节] 引用

明确不做（保持轻量）：向量数据库、自动聚类、多用户服务端、定时后台服务、Web UI、Zotero 注释（高亮/批注）同步、引用页码提取。

---

特性

能力	说明
双源同步	本地 zotero.sqlite（临时副本只读打开，不干扰 Zotero 进程）或 Zotero Web API（version 游标增量）
交互式范围	全部 / 指定 Collection（含子级）/ 标签过滤，选择结果持久化，启动时静默复用
PDF 智能解析	MinerU 官方开放平台（上传→轮询→Zip），解析产物落为 raw/ bundle（Markdown + images/），原始 PDF 不落盘
DeepSeek 全文笔记	deepseek-chat 全文直读（100K token 上限、尾部截断），产出中文结构化笔记（摘要/方法/结论/创新点/局限/关键引用）
优雅降级	无 PDF 或解析失败 → 基于标题+摘要生成简版笔记；后续补 PDF 自动升级全文笔记
配额与并发	单轮解析上限 MAX_PARSE_PER_SYNC（默认 50），并发 3，失败记库、下轮可重跑；支持 topic/collection/keys/limit 过滤（keys > collection > topic 优先级）
解析预览	parse_zotero_wiki 无参数调用只返回候选统计与样本（pending/failed/placeholder-upgrade，MinerU 零调用），确认后再执行
Wiki 编译	主题综述（LLM 增量改写）、index/authors/years 索引（index 表格含作者/DOI/Updated 列）、See Also 交叉链接（bundle 间相对路径）、归档标记（不删笔记）、追加式 log（`## [YYYY-MM-DD]
LINT 自检	四类确定性检查（索引一致性 / 内部链接 / raw 引用 / 图片链接与孤儿图），compile 末尾自动 auto-fix，CLI 可单独运行 lint [--report-only]
查询归档	archive_zotero_query 工具把问答归档为 wiki 归档页（含参考文献表格），联动更新 index 与 log
Agent 查询	minisearch 内存全文检索 + 中文 bigram 分词，返回片段与 [key, §章节] 章节级引用；无命中明确告知，不编造

---

当前状态与路线图

里程碑	内容	状态
M1	Zotero 双源读取与增量同步	✅ 已落地并通过真实库验收
M2	MinerU 解析 + DeepSeek 笔记生成	✅ 已落地并通过真实环境验收（7 篇全文笔记 + 37 篇摘要降级，0 失败）
M3	主题归类与 Wiki 生成	✅ 已落地并通过真实环境验收（deepseek-v4-flash 综述生成 + 幂等二跑）
M4	dsh 插件集成与查询工具	✅ 已落地并通过 dsh 真机联调
M5	测试、打包、文档完善	进行中
改进方案 1	wiki 职责收窄（bundle 双文件写盘）、archive/LINT/migrate、parse 过滤与预览、首次配置向导	✅ 已落地（阶段 A–E，typecheck + 120 单测 + CLI 冒烟）

路线图中（不阻塞使用）：向量检索增强、Zotero 注释同步、引用页码提取、术语表、定时同步。问题与建议请提 Issues。

---

架构

┌────────────────────┐      ┌─────────────────────────────────────────┐
│  Zotero 桌面端      │ ───▶ │ dsh-zotero-wiki                          │
│  / Zotero Web API  │      │                                          │
└────────────────────┘      │  sync 引擎 ──▶ SQLite 状态库             │
                            │  parse 管道 ─▶ MinerU + DeepSeek        │
┌────────────────────┐      │  compile 器 ─▶ 综述/索引/链接/归档       │
│ MinerU 官方平台     │ ◀──▶ │  query 索引 ─▶ minisearch + bigram      │
│ DeepSeek API       │      │                                          │
└────────────────────┘      │  dsh 工具：query / sync / parse          │
                            └────────────────┬────────────────────────┘
                                             ▼
                            ┌─────────────────────────────────────────┐
                            │ Obsidian Vault                           │
                            │  raw/（解析源文件） wiki/（笔记与索引）   │
                            │  references/（模板） .zotero-sync/（状态）│
                            └─────────────────────────────────────────┘

生成的 Vault 结构：

<Vault>/
├── raw/<topic>[/<subtopic>]/YYYY-MM-DD-<slug>/
│   ├── <key>.md            # 原文（全文或占位，frontmatter status: pending|fulltext）
│   ├── <key>-note.md       # LLM 笔记（type: note）
│   └── images/             # MinerU 图片（如有）
├── references/             # 5 个模板（只补缺不覆盖）
├── wiki/                   # 仅 compile 与 query 产物
│   ├── index.md / authors.md / years.md / log.md
│   └── <topic>/
│       ├── _index.md       # 主题综述（compile，含文献列表 + 参考文献表格）
│       └── <slug>.md       # archive 归档页（query 按需生成，仅 type: archive）
└── .zotero-sync/sync.db    # 同步游标、条目、处理状态（不入 git）

---

安装

环境要求

• Node.js ≥ 22.19（使用内置 node:sqlite）
• Zotero 本地库或 Web API Key
• （parse 阶段）MinerU Token（mineru.net API 管理页申请）与 DeepSeek API Key

从源码安装（MVP 阶段推荐）

git clone https://github.com/Wechsels/dsh-zotero-wiki.git && cd dsh-zotero-wiki
npm install
cp .env.example .env     # 然后按需编辑，见下方配置
npm run build

注册为 dsh 插件（本地路径安装）

npx -y @deepseek-ai/dsh plugin --profile web add file:/path/to/dsh-zotero-wiki

插件加载后：dsh 启动时自动静默增量同步并重建查询索引；聊天中即可调用三个原生工具。

---

配置

所有配置在插件目录下的 .env（参见 .env.example）。首次运行任意 CLI 命令时，若缺少必要配置会自动弹出交互式配置向导（--no-wizard 可跳过，此时打印缺失清单并退出）：

键	必填	默认	说明
ZOTERO_MODE	是	local	local / web
ZOTERO_DATA_DIR	local 可选	自动探测	含 zotero.sqlite 与 storage/ 的目录
ZOTERO_API_KEY / ZOTERO_USER_ID / ZOTERO_GROUP_ID	web 必填	—	Web API 凭据（Group 优先于 User）
OBSIDIAN_VAULT_PATH	是	—	Vault 根目录
SYNC_SCOPE_JSON	否	交互选择	持久化同步范围，如 {"mode":"all","collectionKeys":[],"tags":[]}
MINERU_TOKEN	parse 必填	—	MinerU 官方平台 Token
MINERU_MODEL_VERSION	否	pipeline	可选 vlm（质量更高）
MINERU_TIMEOUT_MS	否	1200000	单文件解析超时
DEEPSEEK_API_KEY	parse/综述必填	—	DeepSeek API Key
DEEPSEEK_MODEL	否	deepseek-chat	全文笔记生成模型
REVIEW_MODEL	否	deepseek-v4-flash	主题综述生成模型（compile）
NOTE_TOKEN_LIMIT	否	100000	每篇全文截断上限（按字符保守估算）
MAX_PARSE_PER_SYNC	否	50	单轮最多上传解析篇数（保护 MinerU 免费额度）
PARSE_CONCURRENCY	否	3	解析并发上限
REVIEW_REGEN_THRESHOLD	否	5	综述累计新增 N 篇后全量重生成
REVIEW_NOTE_CHARS	否	2000	综述生成时每篇笔记送入 LLM 的字数上限

---

用法

CLI（完整能力，首次配置入口）

node dist/cli.js sync                  # 增量同步；首次弹交互选择范围并持久化
node dist/cli.js sync --interactive    # 重新选择同步范围

node dist/cli.js parse --dry-run       # 预览待解析候选（≈ 预览模式，零外部调用）
node dist/cli.js parse --topic T       # 限定主题目录（支持两级如 A/B）
node dist/cli.js parse --collection C  # 限定 Collection（名称/key，含子合集）
node dist/cli.js parse --keys K1,K2    # 限定 Zotero key 列表（最精确）
node dist/cli.js parse --limit N       # 本次解析篇数上限（与配额取小）
node dist/cli.js parse                 # MinerU 解析 + DeepSeek 笔记生成

node dist/cli.js compile               # 综述/索引/交叉链接/归档标记（末尾自动 LINT）
node dist/cli.js compile --no-review   # 跳过 LLM 综述（无需 DEEPSEEK_API_KEY）

node dist/cli.js lint                  # LINT 检查 + auto-fix
node dist/cli.js lint --report-only    # 仅报告，不修改文件

node dist/cli.js migrate               # 存量迁移 dry-run（旧 wiki 单篇页 → raw bundle 计划表）
node dist/cli.js migrate --apply       # 执行迁移（写入 bundle 笔记、删除旧页面），随后运行 compile 重建索引
# 任意命令加 --no-wizard 可跳过首次配置向导（打印缺失清单并退出）

dsh 原生工具（Agent 侧）

工具	说明
query_zotero_wiki(query, topic?, max_results?)	检索 Wiki，返回片段、元数据与 [Zotero key, §章节] 引用；无命中返回"当前 Wiki 中没有找到相关文献"
sync_zotero_wiki()	手动增量同步并重建索引
parse_zotero_wiki(topic?, collection?, keys?, limit?)	无参数时仅返回候选预览（零 MinerU 调用）请用户确认；带参数执行解析与笔记生成（受配额限制）并重建索引
archive_zotero_query(title, body, topic?, source_keys?)	把查询回答归档为 wiki 归档页（含参考文献表格），联动更新 index 与 log

典型对话："帮我总结荧光材料方向这几年的进展" → Agent 调用 query_zotero_wiki → 基于命中片段与引用编制回答。

同步行为约定

• 启动自动同步按上次持久化的范围静默执行；交互选择只在首次或手动 --interactive 时出现
• 已有 bundle 更新/扩充，新文献新增，log.md 追加；永不物理删除笔记——Zotero 中删除的条目在笔记中标记 status: archived，恢复后自动复原
• 未归入任何 Collection 的文献跳过，待用户在 Zotero 中分类后下轮处理
• 一篇文献属于多个 Collection：每个主题各生成一份自包含 bundle（raw/<topic>/…/），processed.note_paths 记录各主题笔记的 raw 相对路径

---

开发

npm run typecheck   # tsc --noEmit
npm test            # vitest（18 个测试文件 / 82 个用例）
npm run build       # vite → dist/cli.js + dist/plugin.js
npm run dev         # 监听构建

src/
├── cli.ts          # CLI（sync / parse / compile）
├── plugin.ts       # dsh 集成层（工具注册 + 启动 sync，依赖注入可测）
├── config.ts       # .env 加载与回写
├── sources/        # Zotero 双源（local sqlite / Web API）
├── sync/           # 同步引擎与 scope 过滤
├── parse/          # MinerU 客户端、Zip 解包、单篇 pipeline（bundle 写盘）
├── llm/            # DeepSeek 客户端、prompts、截断
├── compile/        # 综述、索引、See Also、归档、参考文献表格、日志
├── query/          # minisearch 索引（raw 笔记 + _index/archive）、bigram 分词、检索
├── wiki/           # topic 映射、bundle 写盘（<key>.md + <key>-note.md）、frontmatter、模板
└── state/          # SQLite 状态库
dsh/index.js        # 插件薄壳 → dist/plugin.js

---

常见问题

Q：启动时提示 "SYNC_SCOPE_JSON 未配置"？ 在终端运行一次 node dist/cli.js sync 完成交互式范围选择；dsh 运行环境无交互终端，首次配置必须走 CLI。

Q：首次运行提示缺少配置？ CLI 在交互终端会启动配置向导逐项引导（Zotero 模式、数据目录/API Key、Vault 路径、MinerU/DeepSeek Token）；非交互环境（含 dsh 工具）返回结构化缺失清单（configured: false + missing[]，含 where_to_get 指引）。

Q：本地模式提示找不到 Zotero 数据目录？ 在 .env 显式设置 ZOTERO_DATA_DIR 指向含 zotero.sqlite 的目录（如 D:\ZoteroLibrary）。

Q：parse 很慢 / 部分篇目一直是 pending？ MinerU 免费额度为每日 2000 页高优先级，超出后排队降速；调小 MAX_PARSE_PER_SYNC 分多轮执行即可。

Q：中文查询效果不理想？ 当前为 bigram 分词的关键词检索（零原生依赖）；可尝试更具体的关键词或加 topic 过滤。向量检索增强在路线图中。

Q：npx @deepseek-ai/dsh 报 'dsh' 不是内部或外部命令？ npx 在部分 Windows 环境下无法生成可执行 shim。改用本地 dsh 源码仓的 CLI 入口即可：node <dsh 仓库>/apps/cli/lib/bin.js plugin --profile web add file:/path/to/dsh-zotero-wiki。

Q：dsh 启动报 Cannot find module .../dist/plugin.js？ 本地 file: 安装时 pnpm 打包会遵守 .gitignore；确保 package.json 的 files 字段显式包含 dist（本仓库已配置）。

Q：能在 Obsidian 里手动编辑笔记吗？ 可以。只有 ## See Also 章节、索引页（index/authors/years）与归档提示块会被自动重写，其余内容不受影响。

许可证 / License

MIT

---

---

dsh-zotero-wiki

中文 | English

A DeepSeekHarness (dsh) plugin for Zotero: automatically syncs your Zotero library, parses PDFs with MinerU, reads full texts with DeepSeek, compiles everything into a searchable Obsidian wiki, and lets your agent answer questions with traceable literature citations.

---

Positioning & Scope

What it is: a Node/Cordis-style dsh plugin plus a standalone CLI that turns your Zotero library into a continuously updated Obsidian knowledge base, exposing native tools for agent retrieval.

Full pipeline:

Zotero (local DB / Web API)
  └─ sync: incremental metadata sync → .zotero-sync/sync.db (SQLite state)
      └─ parse: MinerU PDF parsing → raw/<topic>/<bundle>/ originals + DeepSeek notes (<key>-note.md co-located in the bundle)
          └─ compile: topic reviews (_index.md), indexes, cross-links, archive marks
              └─ query: agents search raw notes via query_zotero_wiki, answers cite [Zotero key, §section]

Explicitly out of scope (kept lightweight): vector databases, automatic clustering, multi-user server, scheduled background services, Web UI, Zotero annotation/highlight sync, page-number citations.

Constitution deviation: topic directories are kept to two levels (A/B; deeper nesting collapses into the second level), unlike the full-path convention in CLAUDE.md; see the raw//wiki/ layout below.

---

Features

Feature	Description
Dual-source sync	Local zotero.sqlite (read-only temp copy, never blocks Zotero) or Zotero Web API (version-cursor incremental)
Interactive scope	All / specific Collections (incl. children) / tag filter; the choice is persisted and silently reused at startup
PDF parsing	MinerU official open platform (upload → poll → zip); results stored as raw/ bundles (Markdown + images/); original PDFs are never persisted
DeepSeek notes	deepseek-chat full-text reading (100K-token cap, tail truncation) producing structured Chinese notes (abstract / method / conclusions / contributions / limitations / key quotes)
Graceful degradation	No PDF or parse failure → concise note from title + abstract; automatically upgraded to a full-text note once a PDF is added
Quota & concurrency	Per-run parse cap MAX_PARSE_PER_SYNC (default 50), concurrency 3, failures recorded and retryable next run; topic/collection/keys/limit filters (keys > collection > topic priority)
Parse preview	parse_zotero_wiki with no args returns candidate stats & samples (pending/failed/placeholder-upgrade, zero MinerU calls); execute after confirmation
Wiki compile	LLM topic reviews (incremental rewrite), index/authors/years pages (index table with author/DOI/Updated columns), See Also cross-links (bundle-relative), archive marks (no note deletion), append-only log (`## [YYYY-MM-DD]
LINT self-check	Four deterministic checks (index consistency / internal links / raw refs / image links & orphans); auto-fix at the end of compile; standalone lint [--report-only] CLI
Query archiving	archive_zotero_query tool archives an answer as a wiki archive page (with a reference table) and updates index + log
Agent query	minisearch in-memory full-text search with Chinese bigram tokenization; returns snippets with [key, §section] citations; explicitly reports "not found", never fabricates

---

Status & Roadmap

Milestone	Scope	Status
M1	Zotero dual-source reading & incremental sync	✅ landed, verified against a real library
M2	MinerU parsing + DeepSeek note generation	✅ landed, verified in a real environment (7 full-text notes + 37 abstract fallbacks, 0 failures)
M3	Topic compilation & wiki generation	✅ landed, verified in a real environment (deepseek-v4-flash reviews + idempotent re-run)
M4	dsh plugin integration & query tools	✅ landed, verified with a real dsh installation
M5	Testing, packaging, documentation	in progress
Improvement plan 1	Narrowed wiki role (bundle dual-file writes), archive/LINT/migrate, parse filters & preview, first-run config wizard	✅ landed (stages A–E; typecheck + 120 unit tests + CLI smoke)

On the roadmap (non-blocking): vector retrieval, Zotero annotation sync, page-number citations, glossary, scheduled sync. Please file issues and suggestions on GitHub Issues.

---

Installation

Requirements

• Node.js ≥ 22.19 (uses built-in node:sqlite)
• A local Zotero library or a Zotero Web API key
• (for parse) a MinerU token (from mineru.net) and a DeepSeek API key

Install from source (recommended during MVP)

git clone https://github.com/Wechsels/dsh-zotero-wiki.git && cd dsh-zotero-wiki
npm install
cp .env.example .env     # then edit; see Configuration below
npm run build

Register as a dsh plugin (local-path install)

npx -y @deepseek-ai/dsh plugin --profile web add file:/path/to/dsh-zotero-wiki

Once loaded, dsh silently runs an incremental sync at startup and rebuilds the query index; the three native tools become available in chat.

---

Configuration

All settings live in .env inside the plugin directory (see .env.example). On first run, any CLI command with missing required settings launches an interactive setup wizard (add --no-wizard to skip and print the missing list instead). Key entries:

Key	Required	Default	Purpose
ZOTERO_MODE	yes	local	local / web
ZOTERO_DATA_DIR	local, optional	auto-detect	Directory containing zotero.sqlite and storage/
ZOTERO_API_KEY / ZOTERO_USER_ID / ZOTERO_GROUP_ID	web	—	Web API credentials (Group wins over User)
OBSIDIAN_VAULT_PATH	yes	—	Vault root
SYNC_SCOPE_JSON	no	interactive	Persisted sync scope, e.g. {"mode":"all","collectionKeys":[],"tags":[]}
MINERU_TOKEN	parse	—	MinerU official platform token
MINERU_MODEL_VERSION	no	pipeline	vlm for higher quality
DEEPSEEK_API_KEY	parse/reviews	—	DeepSeek API key
DEEPSEEK_MODEL	no	deepseek-chat	Model for full-text note generation
REVIEW_MODEL	no	deepseek-v4-flash	Model for topic review generation (compile)
NOTE_TOKEN_LIMIT	no	100000	Per-paper truncation cap (conservative char-based estimate)
MAX_PARSE_PER_SYNC	no	50	Max uploads per run (protects MinerU free quota)
PARSE_CONCURRENCY	no	3	Parse concurrency
REVIEW_REGEN_THRESHOLD	no	5	Regenerate a topic review after N accumulated new notes

---

Usage

CLI (full capabilities; first-run configuration entry point)

node dist/cli.js sync                  # incremental sync; interactive scope picker on first run
node dist/cli.js sync --interactive    # re-select sync scope

node dist/cli.js parse --dry-run       # preview pending candidates (≈ preview mode, zero external calls)
node dist/cli.js parse --topic T       # restrict to a topic directory (two levels, e.g. A/B)
node dist/cli.js parse --collection C  # restrict to a collection (name/key, includes children)
node dist/cli.js parse --keys K1,K2    # restrict to the given Zotero keys (most precise)
node dist/cli.js parse --limit N       # cap items parsed this run (min with quota)
node dist/cli.js parse                 # MinerU parsing + DeepSeek note generation

node dist/cli.js compile               # reviews / indexes / cross-links / archive marks (auto-LINT at the end)
node dist/cli.js compile --no-review   # skip LLM reviews (no DEEPSEEK_API_KEY needed)

node dist/cli.js lint                  # LINT checks + auto-fix
node dist/cli.js lint --report-only    # report only, no file changes

node dist/cli.js migrate               # legacy migration dry-run (old wiki paper pages → raw bundle plan)
node dist/cli.js migrate --apply       # execute migration (write bundle notes, delete old pages), then run compile
# any command accepts --no-wizard to skip the first-run config wizard (prints the missing list and exits)

dsh native tools (agent side)

Tool	Description
query_zotero_wiki(query, topic?, max_results?)	Search the wiki; returns snippets, metadata and [Zotero key, §section] citations; replies "nothing found in the wiki" when empty
sync_zotero_wiki()	Manual incremental sync + index rebuild
parse_zotero_wiki(topic?, collection?, keys?, limit?)	With no args returns a candidate preview (zero MinerU calls) for confirmation; with args runs parse & note generation (quota-limited) + index rebuild
archive_zotero_query(title, body, topic?, source_keys?)	Archive an answer as a wiki archive page (with a reference table), updating index + log

Sync behavior contract

• Startup sync silently reuses the last persisted scope; interactive selection only happens on first run or with --interactive
• Existing bundles are updated/extended, new items create new bundles, log.md is append-only; notes are never physically deleted — items removed from Zotero get status: archived on the note and are auto-restored if they come back
• Items without any Collection are skipped until classified in Zotero
• An item in multiple Collections: one self-contained bundle per topic (raw/<topic>/…/); processed.note_paths tracks each topic note's raw-relative path

---

Development

npm run typecheck   # tsc --noEmit
npm test            # vitest (18 test files / 82 cases)
npm run build       # vite → dist/cli.js + dist/plugin.js
npm run dev         # watch build

---

FAQ

Q: Startup says "SYNC_SCOPE_JSON not configured"? Run node dist/cli.js sync once in a terminal to complete the interactive scope selection; the dsh runtime has no interactive TTY, so first-time configuration must go through the CLI.

Q: First run complains about missing settings? On an interactive terminal the CLI starts a guided wizard (Zotero mode, data dir/API key, vault path, MinerU/DeepSeek tokens). In non-interactive environments (including dsh tools) it returns a structured missing list (configured: false + missing[] with where_to_get guidance).

Q: Local mode can't find the Zotero data directory? Set ZOTERO_DATA_DIR in .env explicitly to the directory containing zotero.sqlite.

Q: parse is slow or items stay pending? MinerU's free quota is 2000 high-priority pages/day; beyond that, jobs queue at lower priority. Lower MAX_PARSE_PER_SYNC and run in multiple rounds.

Q: Chinese search quality? Tokenization is bigram-based keyword search (zero native dependencies). Try more specific keywords or a topic filter; vector retrieval is on the roadmap.

Q: npx @deepseek-ai/dsh says 'dsh' is not recognized? npx may fail to create the executable shim on some Windows setups. Use the CLI entry from a local dsh source checkout instead: node <dsh-repo>/apps/cli/lib/bin.js plugin --profile web add file:/path/to/dsh-zotero-wiki.

Q: dsh boot fails with Cannot find module .../dist/plugin.js? For local file: installs, pnpm packing honors .gitignore; make sure package.json has an explicit files field including dist (already configured in this repo).

Q: Can I hand-edit notes in Obsidian? Yes. Only the ## See Also section, the index pages (index/authors/years) and archive notice blocks are rewritten automatically; everything else is preserved.