DSH HUB
HomePlugin StorePlugin PacksCommunityRankingsResourcesPublish Guide
Plugin source
Back to catalog

Tannnnhauser /

Tannnnhauser/pivot-web-search

Topic repository only

Resilient multi-provider web search & content extraction for Claude Code, DeepSeek Harness, and any MCP host — quota-aware failover, JS rendering, and a CLI.

★ 2 Stars0 Forks0 IssuesN/A Community rating0 Confirmed installs
View on GitHub
READMESource: main@dfb82ed3

Pivot Web Search

Resilient multi-provider web search and page extraction for MCP hosts, the command line, and host adapters.

Python 3.10+ License: Apache-2.0 Tests: 358

What Is This?

Pivot Web Search exposes one set of shared search, fetch, and configuration services through three interfaces:

  • MCP server (pivot-web-search-mcp) — for Claude Code and any MCP-aware host
  • CLI (pivot-web-search) — for people and shell scripts, no AI host required
  • JSON bridge (pivot-web-search-bridge) — a host-neutral subprocess interface for adapters

It routes each query across multiple providers, fails over automatically when results are weak or a provider is down, tracks quota and health, and extracts page content locally — with no Anthropic API dependency, so it works on Amazon Bedrock and other API providers. A Claude Code plugin and an optional DeepSeek Harness adapter provide host-specific installation; adopting Pivot never requires patching a host's source.

DuckDuckGo works with no API key; adding an API-backed provider (Tavily, Brave, Gemini) improves reliability.

Choose Your Interface

You use… Start here
Claude Code Install the plugin
Codex Install the plugin
Another MCP host (Cursor, Claude Desktop, …) Configure pivot-web-search-mcp
Terminal / scripts / CI Install and run the CLI
DeepSeek Harness Install runtime + DSH bundle

Key Features

  • One runtime, three interfaces — the same search/fetch/config services back MCP, the CLI, and the JSON bridge.
  • Multi-provider failover — quota-aware routing across providers; weak or exhausted providers are skipped, partial results kept as fallback.
  • Super mode — query all providers in parallel, dedupe by URL, rank by cross-provider agreement.
  • Local page extraction — full content via trafilatura, with Next.js/Nuxt.js SPA fallbacks and an optional JS renderer.
  • Hot-reloadable config — add/remove/reorder providers and proxies via YAML; changes apply on the next request.
  • Actionable diagnostics — when everything fails, get per-provider failure reasons and suggestions, not a generic error.

Single-provider search fails when an API is unavailable, rate-limited, blocked, or out of quota. Pivot routes each request across configured providers, keeps usable partial results, and exposes the same behavior through MCP, CLI, and adapter-friendly JSON.

Prerequisites

  • uv — manages Python and dependencies automatically (used by every interface).
  • At least one search provider (DDG needs no API key).

Recommended: configure at least one free API key — Tavily (1000 credits/month, no card) or Brave (1000 queries/month, card required). DDG is a free fallback but can be unreliable under heavy use.

Per-host setup is covered in each Quick Start below.

Quick Starts

Claude Code plugin

Step 1 — add the marketplace (one-time):

claude plugin marketplace add https://github.com/Tannnnhauser/pivot-web-search.git

Step 2 — install:

claude plugin install pivot-web-search

The plugin prompts for configuration at install time:

Setting Description
Tavily API Key Stored in system keychain. Set to enable Tavily (or inherit TAVILY_API_KEY from the shell).
Brave Search API Key Stored in system keychain. Set to enable Brave (or inherit BRAVE_API_KEY).
Gemini API Key Stored in system keychain. Set to enable Gemini (or inherit GEMINI_SEARCH_API_KEY / GOOGLE_STUDIO_API_KEY).
Proxy URLs Comma-separated proxies to try in order. direct is always appended as the final fallback — to disable that, use ~/.pivot-web-search/proxies.yaml.

Providers are enabled automatically for whichever keys you supply; DDG is always on. Routing order and timeouts come from smart defaults — the order you supply keys does not matter. Reconfigure anytime via claude plugin configure pivot-web-search.

Verify: ask Claude Code to run WebSearchConfig with action status — you should see a provider health report.

Key resolution & macOS GUI caveat

For each provider key the plugin reads (1) the standard env var (e.g. TAVILY_API_KEY) inherited from the shell — wins if set — then (2) the /plugin UI value (injected as PIVOT_USERCONFIG_TAVILY_API_KEY). To use the UI value instead, unset TAVILY_API_KEY in your shell.

macOS: when Claude Code launches from Spotlight or the Dock it does not see ~/.zshrc exports. Start it from a terminal, set keys via the /plugin UI, or add them to the env block in ~/.claude/config.json.

Codex plugin

Add the same GitHub marketplace and install the same plugin payload:

codex plugin marketplace add Tannnnhauser/pivot-web-search --ref main
codex plugin add pivot-web-search@pivot-web-search-plugin

Start a new Codex chat after installation. Refresh future releases with:

codex plugin marketplace upgrade pivot-web-search-plugin

Codex reads provider keys from the standard environment variables listed above. DDG remains available without a key; advanced provider and proxy configuration under ~/.pivot-web-search/ is shared with every host.

MCP hosts

Any MCP-aware host (Claude Code, Claude Desktop, Cursor, …) can run the server directly via uvx — no clone, no venv. Add to your .mcp.json:

{
  "mcpServers": {
    "pivot-web-search": {
      "command": "uvx",
      "args": [
        "git+https://github.com/Tannnnhauser/pivot-web-search.git#subdirectory=plugins/pivot-web-search"
      ],
      "env": {
        "TAVILY_API_KEY": "tvly-...",
        "BRAVE_API_KEY": "BSA...",
        "GEMINI_SEARCH_API_KEY": "AI..."
      }
    }
  }
}

Pin a version with @v1.1.0:

git+https://github.com/Tannnnhauser/pivot-web-search.git@v1.1.0#subdirectory=plugins/pivot-web-search

Advanced provider/proxy config lives in ~/.pivot-web-search/*.yaml (see Configuration) and applies to every launch method.

CLI

The installed package provides pivot-web-search, a human-facing command backed by the same services as the MCP tools — for use in a terminal, a script, or CI, without going through an AI host. Install it standalone with uv:

uv tool install 'git+https://github.com/Tannnnhauser/pivot-web-search.git@v1.1.0#subdirectory=plugins/pivot-web-search'
pivot-web-search search "latest Python release" --format json
pivot-web-search fetch https://example.com --format md
pivot-web-search config status

search flags:

Flag Default Description
--max-results 5 Number of results (1–10, or 1–20 with --super)
--provider auto Force a configured provider by name
--super off Query all providers in parallel
--news off Search news instead of the general web
--timelimit — d / w / m / y recency filter
--include-answer off Include an AI answer when supported
--search-depth basic basic or advanced (Tavily)
--topic general general or news (Tavily)
--days — Limit news to recent N days
--include-domains — Domain allowlist
--exclude-domains — Domain blocklist
--include-content off Return pre-extracted page content (Brave LLM Context)
--max-content-tokens 8192 Token budget for --include-content
--region wt-wt DDG region (CLI-only; no MCP equivalent)
--format md md or json

fetch (alias extract): --query (relevance hint for JS renderers), --max-chars (per-URL truncation), --format (json default, or md).

config: positional action, status (default) or reload.

DeepSeek Harness

The optional pivot-web-search-dsh Profile Bundle registers Pivot as DSH's existing web_search and web_fetch provider through DSH's published Bundle, ctx.web, and ctx.subprocess APIs. The model keeps seeing DSH's standard tools — no second set of Pivot-specific tools. Host integrations use public extension APIs; the adapter requires no DSH fork or source patch.

Install the runtime and the Bundle:

uv tool install 'git+https://github.com/Tannnnhauser/pivot-web-search.git@v1.1.0#subdirectory=plugins/pivot-web-search'
dsh plugin --profile web add pivot-web-search-dsh

Restart the profile after adding the Bundle. It selects provider pivot as searchProvider/fetchProvider and enables DSH's tool-web entry (which the shipped web profile disables). Verify with dsh --profile web --dump-config — you should see searchProvider: pivot, fetchProvider: pivot, an enabled tool-web, and pivot-web-search-provider.

Provider keys are forwarded explicitly by cordis.patch.yml (TAVILY_API_KEY, BRAVE_API_KEY, GEMINI_SEARCH_API_KEY, GOOGLE_STUDIO_API_KEY, plus PIVOT_WEB_SEARCH_PROXIES); config files under ~/.pivot-web-search/ are read directly.

Custom providers: the bridge subprocess receives only the env vars listed in cordis.patch.yml — it does not inherit the parent shell. If a custom provider's api_key_env names a key outside the list above (e.g. a self-hosted gateway token), add that variable to the Bundle's env block or it will not reach the bridge.

Development install and removal steps are documented in integrations/deepseek-harness/.

Configuration

Pivot runs on auto-detection by default — supply API keys (UI or shell env) and matching providers are enabled with smart routing defaults. No YAML required for the common case.

For advanced setups, drop YAML into ~/.pivot-web-search/:

File Purpose
providers.yaml Take over provider config: SearXNG, custom JSON APIs, LLM-search providers, explicit priorities
proxies.yaml Take over proxy config: SOCKS5, forced-proxy (no direct fallback), per-proxy priority

Precedence is all-or-nothing per file: if a file exists, auto-detection for that concern is bypassed — list every entry you want (including DDG). Templates live in examples/.

~/.pivot-web-search/providers.yaml

Providers are tried by priority (lower = first). Same-priority providers are hedged — queried concurrently with staggered starts, first quality result wins. Without an explicit priority, smart defaults apply by type.

providers:
  - name: tavily
    type: tavily
    api_key_env: TAVILY_API_KEY

  - name: brave
    type: brave
    api_key_env: BRAVE_API_KEY

  - name: gemini
    type: gemini
    api_key_env: GEMINI_SEARCH_API_KEY
    model: gemini-2.5-flash

  - name: ddg
    type: ddg

  # Self-hosted SearXNG
  # - name: searxng-local
  #   type: searxng
  #   endpoint: "http://localhost:8888/search"

  # Generic JSON API (Serper, Google CSE, etc.) — multiple instances allowed,
  # each with independent priority, quota tracking, and circuit-breaker state.
  # - name: serper
  #   type: json_api
  #   endpoint: "https://google.serper.dev/search"
  #   api_key_env: SERPER_API_KEY
  #   method: POST
  #   request_body:
  #     q: "{{query}}"
  #     num: "{{max_results}}"
  #   response_mapping:
  #     results_path: "organic"
  #     title: "title"
  #     url: "link"
  #     snippet: "snippet"

Smart default priorities (when no explicit priority):

Type Priority Timeout
llm_search 10 15s
tavily / brave / gemini 20 4s / 4s / 20s
searxng / json_api 30 6s
ddg 90 6s

Note: an llm_search provider at priority 10 runs ahead of Tavily/Brave with a 15s timeout, so every query may take 15+s. Bump its priority above 20 if latency matters more than answer quality.

LLM Search Providers (type: llm_search)

For any model with built-in web search grounding (Perplexity Sonar Pro, OpenAI with web_search, SAP AI Core, etc.). These return an AI answer plus cited URLs. Configure via ~/.pivot-web-search/providers.yaml (template at examples/providers.yaml).

chat_completions — any /chat/completions-compatible endpoint with built-in search:

  - name: sonar-pro
    type: llm_search
    api_format: chat_completions
    endpoint: "https://api.perplexity.ai/chat/completions"
    model: sonar-pro
    api_key_env: PERPLEXITY_API_KEY
    timeout: 15

Response parsing tries, in order: search_results (Perplexity/Sonar), annotations with type: url_citation (OpenAI Chat Completions), then top-level citations.

responses — OpenAI Responses API (/responses) with the web_search tool:

  - name: gpt-web-search
    type: llm_search
    api_format: responses
    endpoint: "https://api.openai.com/v1/responses"
    model: gpt-4o
    api_key_env: OPENAI_API_KEY
    timeout: 45
    search_tool: web_search
    search_context_size: medium

Common fields:

Field Required Description
api_format No chat_completions (default) or responses
endpoint Yes Full URL to the API endpoint
model Yes Model identifier
api_key_env Yes Env var holding the API key (sent as Bearer token)
max_tokens No Max response tokens (default: 500 / 4000 for responses)
timeout No Request timeout in seconds (default: 30)
system_prompt No System prompt (chat_completions only)
headers No Additional request headers
web_search_options No Extra search options (chat_completions only)

The gemini type is also LLM-search internally (Google Search grounding) but keeps type: gemini for backward compatibility and dual-key fallback.

~/.pivot-web-search/proxies.yaml

When present it takes over completely — the install-time Proxy URLs field is ignored and direct is not auto-appended (the escape hatch for forced-proxy setups).

proxies:
  - name: direct
    url: null             # null = direct connection
    enabled: true
    priority: 1

  - name: myproxy1
    url: "http://myproxy1.example:8080"
    enabled: true
    priority: 2

  # SOCKS5 (requires PySocks: uv pip install pysocks)
  # - name: ssh-tunnel
  #   url: "socks5://127.0.0.1:1080"
  #   enabled: true
  #   priority: 3

config/fetch.yaml

Controls WebFetch behavior including the JS rendering fallback:

js_renderer: none         # none (default), "playwright", or "tavily"
max_chars: 100000         # content truncation limit

Set js_renderer: playwright for JavaScript-heavy sites (requires uv sync --extra browser).

MCP Tool Reference

Via MCP, tools are prefixed: mcp__pivot-web-search__WebSearch, mcp__pivot-web-search__WebFetch, mcp__pivot-web-search__WebSearchConfig.

WebSearch

Parameter Type Default Description
query str required Search query
provider str "auto" Force provider: auto / ddg / tavily / brave / gemini / searxng, or any registered name
super_mode bool false Query all providers in parallel
max_results int 5 1–10 (1–20 in super mode)
allowed_domains list[str] null Only include results from these domains
blocked_domains list[str] null Exclude results from these domains
news bool false Search news instead of web
timelimit str null d / w / m / y
include_answer bool false AI-generated answer summary (Tavily)
include_content bool false Pre-extracted page content (Brave LLM Context)
max_content_tokens int 8192 Token budget when include_content=true (1024–32768)
search_depth str "basic" basic or advanced — advanced costs 2x credits (Tavily)
topic str "general" general or news (Tavily)
days int null Limit news to recent N days (Tavily)

WebFetch

Parameter Type Default Description
url str / list[str] required URL(s) to extract. HTTP auto-upgrades to HTTPS. Batch mode with multiple URLs.
query str null Optional relevance query for JS-fallback renderers
max_chars int null Truncate output to this many characters (default: 100,000)

Behaviors: 15-minute per-URL cache · binary-content detection & rejection · cross-host redirect safety (blocks before following) · SPA fallback (__NEXT_DATA__ / RSC payload / __NUXT_DATA__).

WebSearchConfig

Parameter Type Default Description
action str "status" status — provider health, quota, config sources; reload — hot-reload YAML

status returns provider health, per-provider quota, and config-source annotations showing where each setting comes from (env var, YAML path, or built-in default).

How Routing and Failover Works

Request
  │
  ├─ normal mode: priority-group routing
  │   ┌─ Group 1 (priority 10): LLM Search (Perplexity, OpenAI, etc.)
  │   ├─ Group 2 (priority 20): Tavily + Brave + Gemini ← hedged (200ms stagger, first quality-gate pass wins)
  │   ├─ Group 3 (priority 30): SearXNG / json_api
  │   └─ Group 4 (priority 90): DDG (free exhaustion fallback)
  │
  │   Gates: quota-exhausted → skip | circuit-open → skip | affinity mismatch → skip
  │   Quality gate (3-tier): AI answer ≥40 chars? → unique URLs ≥2? → keyword overlap?
  │   Circuit breaker: 3 consecutive failures → OPEN (60s cooldown) → HALF_OPEN → probe
  │
  └─ super mode:    Tavily ┐
                    Brave  ┤ parallel (skip exhausted) → dedup → rank by provider count
                    Gemini ┤
                    DDG    ┘

Each provider independently tries all configured proxies (direct → myproxy1 → …) with per-host caching persisted to ~/.cache/pivot-web-search/.

Quota Management

Usage is tracked across sessions in ~/.cache/pivot-web-search/quota.json:

Provider Tracking Free tier Details
DuckDuckGo Not tracked Unlimited Free, no API key
Tavily API sync 1000 credits/month Calls GET /usage at startup for real credit data
Brave Response headers Rolling 30-day window Parses X-RateLimit-Remaining / X-RateLimit-Reset
Gemini Local (daily) Varies (resets PT midnight) Set limit via PIVOT_WEB_SEARCH_GEMINI_QUOTA

Quota-aware scheduling skips providers at 100% usage; resets on calendar rollover.

Architecture

Three interfaces sit on one set of shared services:

MCP server ─┐
CLI ────────┼─→ search / fetch / config services ─→ provider registry ─→ providers
JSON bridge ┘                                                            (DDG/Tavily/Brave/Gemini/SearXNG/json_api/llm_search)
  • server.py — FastMCP adapter (the 3 MCP tools) · cli.py — CLI adapter · machine_bridge.py — host-neutral JSON adapter
  • search_service.py / fetch_service.py / config_service.py — authoritative orchestration · presentation.py — Markdown/JSON projections
  • routing.py / quality_gate.py — priority-group routing, hedging, circuit breaker · backends.py / extraction.py / http_client.py — provider I/O, extraction, proxy failover
  • providers/ — adapter base, 6 built-ins, mtime hot-reload registry · quota.py / config.py — cross-session quota, YAML hot-reload

The Claude Code plugin payload lives under plugins/pivot-web-search/; the DSH adapter under integrations/deepseek-harness/ (never patches DSH source); YAML templates under examples/.

Testing

uv sync                                # install workspace + dev deps
uv run pytest -m "not integration" -q  # 364 offline tests
uv run pytest -m integration -vv       # 7 live network/API tests
uv run pytest                          # all 371 Python tests
uv run python scripts/codex-plugin-smoke.py  # Codex install, skill discovery, cached MCP handshake
npm --prefix integrations/deepseek-harness test
(cd integrations/deepseek-harness && npm pack --dry-run)
npm --prefix integrations/deepseek-harness run test:dsh

For local Claude Code development: claude --plugin-dir /path/to/pivot-web-search/.

Troubleshooting

Enable debug logging — set PIVOT_WEB_SEARCH_DEBUG=1; timestamped logs go to ~/.cache/pivot-web-search/server.log.

No results from any provider — run WebSearchConfig action status (or pivot-web-search config status) to check provider health and active config sources. Ensure at least one provider has a valid key (or DDG is reachable).

SSL certificate errors on macOS — the plugin uses certifi; if errors persist: uv sync --upgrade-package certifi.

DuckDuckGo rate limiting (403) — the circuit breaker bypasses DDG for 60s after consecutive failures, then probes to recover. Configure an API-backed provider for reliability.

trafilatura extraction returns empty — some JS-heavy sites need a renderer. Set js_renderer: playwright in config/fetch.yaml, then uv sync --extra browser && playwright install chromium.

Alternatives Comparison

Feature Pivot Web Search Single-provider MCP tools Built-in WebSearch
Multi-provider failover 4+ providers, auto-fallback Single point of failure N/A on Bedrock
Quota management Cross-session tracking None N/A
Super mode (parallel) All providers at once Not possible N/A
Local content extraction trafilatura + SPA fallback Usually Tavily Extract Anthropic-hosted
Interfaces MCP + CLI + JSON bridge MCP only Built-in only
Works on Bedrock Yes Yes No
Self-hosted Yes Varies No

License

Apache-2.0

—/ 5

No ratings yet

Manifest verification required

Commit dfb82ed35535

Community comments

No comments yet. Be the first to write one.

DSH HUB

A community index for DSH plugins. Not an official GitHub or DeepSeek AI product.

CommunityResourcesAPIAbout