spa-api-extract

An Agent Skill that extracts structured data from JavaScript single-page apps (SPAs) when a normal page fetch fails.

When WebFetch (or any HTML-to-markdown fetcher) times out, returns only obfuscated/bundled JS, or returns an empty client-rendered shell, this skill switches to a reliable CLI pipeline that finds and calls the JSON API the page uses internally.

When to use

• The target page renders content client-side (React/Vue/Next/Nuxt SPA).
• The real data lives behind a JSON API, not in the initial HTML.
• The URL carries an auth token query param (e.g. ?t=xxxx).
• WebFetch timed out or returned scripts instead of readable content.

The pipeline

1. Confirm the failure signal — timeout / JS-only / empty shell → it's an SPA.
2. curl-probe headers + raw HTML skeleton (-sS -m always).
3. grep the HTML/JS for fetch() / /api/ endpoints, <script src>, and BASE-path derivation.
4. Reconstruct & request the JSON API, carrying the token param (read-only GET).
5. Parse with Python — inspect structure first, then extract to a UTF-8 file (avoids Windows GBK mojibake).
6. Verify record count, field completeness, non-empty output.

See SKILL.md for the full step-by-step commands and reference.md for the grep-regex library, common SPA data patterns (REST / GraphQL / __NEXT_DATA__ / inline window.__* JSON), auth variants, and a troubleshooting table.

Install

Drop the directory into your agent's skills folder, e.g.:

~/.qwenworkcn/skills/spa-api-extract/

The skill is plain Markdown + standard curl/python and follows the open Agent Skills convention (SKILL.md + frontmatter), so it is portable across harnesses that support the format.

DeepSeek Harness (dsh) plugin

This repo is also packaged as a DeepSeek Harness (dsh) skill plugin — dsh's model is "everything is a plugin", built on the Cordis runtime.

Layout:

package.json          # dsh manifest: `dsh.bundle.patch` + `dsh-plugin` keywords
cordis.patch.yml      # Cordis patch that inserts this plugin into the runtime tree
index.js              # entry: inject = ['skills'], apply(ctx) -> registerSkills(ctx)
lib/skills.js         # registers the root SKILL.md as one ctx.skills provider
SKILL.md              # the skill bundle (at the package root)
reference.md          # deep-dive loaded on demand
.skill-metadata.yaml  # bilingual recommended queries

Install into a dsh deployment (after @deepseek-ai/dsh-base, which mounts the skill registry):

npm install @sunwindy91/dsh-spa-api-extract
# then apply the Cordis patch shipped at ./cordis.patch.yml

The plugin registers knowledge only — it exposes no host tools and performs no network I/O at load time.

Note: the dsh ecosystem commonly nests skills under skills/<name>/SKILL.md. This plugin keeps the single bundle at the package root so the same tree also loads as a native QwenWork skill without duplicating the file; lib/skills.js reads it from there. The provider/candidate/definition shapes and frontmatter parsing match the official dsh skill providers.

Safety & compliance

• Read-only by default. Only GET is used; write/mutate endpoints require explicit user authorization.
• Tokens are secrets. Captured tokens must not be echoed into final output, committed logs, or shared files.
• Respect the target. Only scrape sites/APIs you are authorized to access, and honor their Terms of Service, robots directives, and rate limits.
• No telemetry. This skill contains no listener, no data collection, and no network callback — it is pure instruction text with placeholder examples (HOST / PATH / TOKEN).

License

MIT © 2026 sunwindy91 — see LICENSE.