dsh-searchops

Agent-native search, log investigation and cluster operations for DeepSeek Harness.

OpenSearch-first. Provider-neutral by design.

"Why is payment-api returning HTTP 500s?"

DeepSeek Harness
       ↓
dsh-searchops
       ↓
OpenSearch
       ↓
Errors
Patterns
Trace IDs
Context
Evidence
       ↓
Agent reasoning

Project status: pre-1.0 (v0.1.0). Every tool is strictly read-only — the plugin searches, observes and investigates; it never writes, deletes, or changes cluster state. The safety envelope (credential isolation, bounded responses, timeouts, disabled redirects, untrusted-data handling) is covered by automated tests, but the plugin has not yet been certified against every OpenSearch distribution and version.

---

Why SearchOps

An AI agent pointed at a raw POST /_search endpoint drowns. A single log index can hold billions of documents; one careless query pulls back gigabytes; a wall of 900 near-identical Connection refused lines burns tokens and tells the model nothing it could not have learned from the sentence "Connection refused — 842 times, first at 10:02, last at 10:14".

dsh-searchops is not an OpenSearch API wrapper. It is an agent-native evidence-acquisition layer for search and log systems. The plugin does the deterministic, token-expensive work — bounding, grouping, fingerprinting, correlating — and hands the agent a compact, structured evidence package. The agent does what only it can do: reason about the cause.

SearchOps = deterministic evidence acquisition
LLM       = reasoning

The core competency is not "AI can query OpenSearch" — it is "AI can efficiently acquire structured evidence from huge search/log systems without drowning itself in raw data."

---

Features

• Nine read-only tools under a provider-neutral searchops_* namespace — no opensearch_* branding, and deliberately no generic searchops_http escape hatch.
• searchops_investigate — the headline capability. One bounded call runs a fixed evidence pipeline (resolve schema → find errors → group into recurring fingerprints → pick representative events → extract trace ids → correlate context → suggest next queries) and returns a structured package.
• Deterministic log fingerprinting — collapses Connection refused to 10.1.2.3:6379 and Connection refused to 10.1.2.4:6379 into one pattern with a count, without ML and without being over-aggressive (meaningful words, short numbers and HTTP status codes survive).
• Configurable field profiles with auto-detection — never hard-codes one log schema. Bind timestampField/messageField/serviceField/levelField/ traceIdField per source, or let the plugin detect them from the mapping and fall back to convention.
• Multiple named sources — prod, staging, local; each with its own URL, auth mode, credential references and field profile. Every tool takes an optional source argument.
• A real provider abstraction — the core speaks SearchSource / SearchQuery / SearchResult / SearchProvider; OpenSearch is one adapter behind it. Elasticsearch is a documented extension point, not a fake.
• A hard security envelope — credentials never enter the model context, tool results, logs or error text; redirects are disabled so an Authorization header can never be forwarded cross-origin; every response is size- and time-bounded; all returned content is treated as untrusted data.
• Bounded by construction — every list, query and investigation discloses truncation (returned vs total) instead of silently dropping rows.

---

Architecture

                 DeepSeek Harness
                         │
                    SearchOps (this plugin)
                         │
             ┌───────────┴───────────┐
             │                       │
      High-level tools          Raw query layer
             │                       │
   logs / investigate              query
   aggregate / context               │
             │                       │
             └───────────┬───────────┘
                         │
              Domain engines (bounded, provider-neutral)
                         │
                  SearchProvider API
                         │
        ┌────────────────┴────────────────┐
        │                                 │
   OpenSearch adapter              Elasticsearch
       (v0.1)                        (future)
        │
   Authenticated, bounded HTTP client
   (timeouts · no redirects · byte caps · redaction)

Layers, from the composition root inward:

Layer	Files	Responsibility
Plugin entry	index.js	Config schema, apply(), system-prompt guidance, tool registration, internals
Tools	lib/tools/*.js	Thin searchops_* tool definitions: parameters, presentation, rendering
Domain engines	lib/logs.js, lib/aggregate.js, lib/context.js, lib/investigate.js, lib/search.js, lib/patterns.js	Bounded, provider-neutral search/log/analysis/investigation logic
Runtime	lib/runtime.js	Multi-source resolution and the authenticated, bounded HTTP client
Providers	lib/providers/index.js, lib/providers/opensearch.js	The SearchProvider registry and the OpenSearch adapter
Config & fields	lib/config.js, lib/fields.js, lib/time.js, lib/dsl.js	Source/profile resolution, field auto-detection, time ranges, query DSL builders
Primitives	lib/constants.js, lib/util.js, lib/budget.js, lib/failures.js, lib/render.js	Security bounds, sanitization, truncation budget, error model, presentation

The core never contains opensearch.xxx calls; only the adapter knows the OpenSearch REST API. See docs/ARCHITECTURE.md.

---

Quick Start

Requirements

Component	Supported baseline
Node.js	20.11 or newer
DeepSeek Harness	0.1.0-rc.6 through 0.1.5 prereleases
Search backend	OpenSearch 1.x / 2.x (REST API)

There is no build step: the plugin is plain ESM JavaScript.

Install

# released tag (preferred)
dsh plugin --profile <profile> add github:guhanfei-ai/dsh-searchops#v0.1.0

# local development
npm ci
dsh plugin --profile <profile> add link:/absolute/path/to/dsh-searchops

Restart the selected DSH profile after installing.

Configure one source

In Settings → Plugins → SearchOps, add a source. Secrets are never typed here — only credential references, whose values live in the DSH credential store:

{
  "sources": [
    {
      "name": "prod",
      "provider": "opensearch",
      "url": "https://search-prod.example.com:9200",
      "auth": "basic",
      "username": "searchops-reader"
    }
  ],
  "defaultSource": "prod",
  "allowInsecureHttp": false
}

Store the password under the reference derived from the source id (SEARCHOPS_PASSWORD_prod). Then ask the agent:

Is the prod search cluster healthy?
Show me the last hour of payment-api errors.
Investigate why payment-api is returning 500s.

---

Configuration

The plugin config is validated by a schema; unknown or malformed input is rejected with a clear, secret-free message.

Field	Type	Default	Meaning
sources	array	[]	Configured search sources (see below)
profiles	array	[]	Optional field profiles for non-standard schemas
defaultSource	string	""	Source used when a tool call omits source; with exactly one source it is the default automatically
allowInsecureHttp	boolean	false	Allow plain HTTP for non-loopback hosts. Off by default

Source object

Field	Default	Meaning
id	(system)	Read-only, system-generated unique id. Credential refs derive from it
name	—	Unique handle the agent uses to select the source (any language)
provider	opensearch	Only opensearch in v0.1; elasticsearch is a roadmap item
auth	basic	none, basic, or bearer
url	—	Base URL, e.g. https://search.example.com:9200. Must not contain credentials, a query string or a fragment
username	""	Optional non-secret username for basic auth
usernameRef	(derived)	Credential ref for the basic-auth username
passwordRef	(derived)	Credential ref for the basic-auth password
tokenRef	(derived)	Credential ref for the bearer token
profile	""	Name of the field profile bound to this source; empty = auto-detect

The resolver also accepts an object-map form ({"sources": {"prod": {...}}}) for convenience; the array form above is what the Settings UI edits.

---

Multiple Sources

Configure as many sources as you like (up to 50) and target one per call:

{
  "sources": [
    { "name": "prod",    "url": "https://search-prod.example.com:9200",    "auth": "basic",  "username": "searchops-reader" },
    { "name": "staging", "url": "https://search-staging.example.com:9200", "auth": "bearer" },
    { "name": "local",   "url": "http://localhost:9200",                   "auth": "none" }
  ],
  "defaultSource": "prod"
}

Every tool accepts an optional source argument (the source name). Omit it to use the default. Call searchops_sources first to discover valid names, providers, sanitized URLs, auth modes, bound profiles and whether credentials are configured (never the values).

Each source keeps its own URL, credentials and field profile; a request for staging never touches prod's credentials. http://localhost (loopback) is allowed without allowInsecureHttp; any other plain-HTTP host is refused unless the operator explicitly opts in.

---

Field Profiles

The log world's biggest pain is that schemas are not uniform. One index uses @timestamp / message / service.name; another uses ts / msg / app. dsh-searchops never hard-codes a single schema. Resolution order per role:

1. Profile — the field named in the source's bound profile wins.
2. Auto-detection — otherwise the plugin looks at the index mapping (or a sample) and picks the first known candidate that exists.
3. Convention — otherwise the most common name is used so a query still works.

{
  "profiles": [
    {
      "name": "ecs",
      "timestampField": "@timestamp",
      "messageField": "message",
      "serviceField": "service.name",
      "levelField": "log.level",
      "traceIdField": "trace.id"
    },
    {
      "name": "custom",
      "timestampField": "ts",
      "messageField": "msg",
      "serviceField": "app",
      "levelField": "severity",
      "traceIdField": "request_id"
    }
  ]
}

Bind a profile to a source with "profile": "ecs". Leave it empty to auto-detect. Every logs/context/investigate result reports the fields it actually used and flags which roles were guessed, so the agent can tell you when detection looks wrong.

Detection candidates include, per role: timestamp (@timestamp, timestamp, time, created_at, …), message (message, msg, log, text, …), service (service.name, service, app, application, …), level (log.level, level, severity, …), trace id (trace.id, traceId, request.id, correlation.id, …).

---

Tools

All nine tools are read-only, bounded, and return untrusted data. None requires approval because none can mutate anything.

Tool	Purpose
searchops_sources	List configured sources (names, providers, sanitized URLs, auth, profile, credentials-configured, default). Never returns secret values
searchops_status	Reachability + cluster health: distribution, version, cluster name, status, node and shard counts
searchops_indices	List indices, optionally by pattern; paged, with doc counts and store sizes
searchops_mapping	Inspect an index's fields (flattened paths + types), or a bounded raw-mapping mode
searchops_query	Bounded raw OpenSearch/Elasticsearch DSL query for power users
searchops_logs	Semantic log search by time range, service, level and free text — no need to know field names
searchops_aggregate	Group/count/metric over an index, and/or a date histogram, without pulling documents
searchops_context	The lines around one event (by timestamp) or every event sharing a trace id
searchops_investigate	A fixed, bounded investigation returning a structured evidence package

Bounds at a glance

Concern	Default	Hard cap
searchops_query size	20	200 (500 absolute)
Result offset (from)	0	10 000 (deep paging rejected)
Logs lines	50	500
Aggregate buckets	20	200
Context lines per side	20	200
Context window per side	300 s	3 600 s
Time window per call	—	31 days
Response body	—	4 MiB (streamed, abandoned early)
Per-request timeout	15 s	—
Investigate scanned docs	—	500

---

Examples

Status

searchops_status { "source": "prod" }
→ source="prod" url="https://search-prod.example.com:9200" reachable=yes
  cluster="prod-cluster" distribution=opensearch version=2.11.0 node="node-1"
  health=green nodes=3 dataNodes=2 shards: active=20 activePrimary=10 unassigned=0 …

Semantic logs

searchops_logs { "index": "logs-*", "service": "payment-api", "level": "error", "from": "now-15m" }
→ logs on "logs-*" range=now-15m..now matched=137 returned=50 took=42ms
  fields: timestamp=@timestamp message=message service=service.name level=log.level trace=trace.id
  2024-01-15T10:14:02Z ERROR [payment-api] Connection refused to db-1 trace=t1 id=…
  …

Aggregate

searchops_aggregate { "index": "logs-*", "groupBy": "service", "from": "now-15m" }
→ aggregate on "logs-*" range=now-15m..now matched=947 took=18ms
  groups by "service.name":
    payment-api count=732
    checkout    count=211
    auth        count=4

Raw query (power users)

searchops_query {
  "index": "logs-*",
  "query": { "bool": { "filter": [ { "term": { "http.response.status_code": 500 } } ] } },
  "size": 20
}

---

Investigation Workflow

searchops_investigate is the recommended entry point for an incident. This is what makes SearchOps more than es_query_logs:

User:
  Investigate payment-api errors in prod during the last 15 minutes.

Agent:
  searchops_investigate { "source": "prod", "index": "logs-*",
                          "service": "payment-api", "from": "now-15m" }

SearchOps (deterministic, bounded, read-only):
  - resolves the logs-* schema (mapping → field profile)
  - filters the time window and service, error-level only
  - groups error docs into recurring fingerprints with counts + first/last seen
  - picks a representative sample per pattern
  - extracts trace/request ids
  - pulls correlated context for the top traces
  - emits deterministic next-query suggestions

  # Investigation of "logs-*" (now-15m..now) service="payment-api"
  scanned=137 doc(s); matched=137; distinct patterns=3; error-level filter=on

  ## Recurring patterns (most frequent first)
  1. [112x] Connection refused to <ip>:<port>
     first=2024-01-15T10:02:11Z last=2024-01-15T10:14:58Z services=payment-api
     sample: Connection refused to 10.1.2.3:6379 (id=…)
     traces: t1, t2, t3, t4, t5
  2. [21x] Redis timeout after …
  3. [4x] NullPointer …

  ## Correlated traces (what else happened in the same request)
  trace=t1 events=6 services=payment-api,cache,db
     …

  ## Suggested next steps (deterministic — you decide the cause)
  - searchops_logs on logs-* with query="Connection refused" to read the raw lines…
  - searchops_context on logs-* with traceId="t1" to see the full request path…
  - searchops_aggregate on logs-* groupBy="service.name" …
  - searchops_aggregate on logs-* interval=5m to see when the errors started…

  Evidence only — the plugin does not infer root cause.

Agent:
  uses the evidence package to reason about likely causes
  (e.g. the cache tier refusing connections at 10:02, spreading to payment-api).

The plugin collects and organizes evidence; it deliberately does not decide the root cause. That reasoning is the model's job.

---

Security

dsh-searchops is read-only by design and defends every request. Full details in docs/SECURITY.md.

• Read-only. No tool writes, deletes, or mutates cluster state. Destructive operations are out of scope for v0.1 and will require dsh-human-intent authorization when they arrive.
• Credential isolation. Secrets are resolved from the DSH credential store at request time and go straight into the Authorization header. They never appear in config, tool results, the model context, logs, or error text.
• No redirects. Requests use redirect: 'error', so an Authorization header can never be forwarded to a different origin (SSRF / credential-leak guard).
• Transport policy. HTTPS is required for non-loopback hosts; plain HTTP is refused unless the operator explicitly sets allowInsecureHttp: true. URLs with embedded credentials, query strings or fragments are rejected.
• Bounded responses. Per-request timeout, a streamed hard byte cap, size and offset limits, and a maximum time window stop one query from pulling gigabytes.
• Sanitized errors. Upstream bodies are untrusted and may echo credentials; they are redacted, single-lined and truncated before reaching any message.
• Redacted evidence. Indexed log data is untrusted too and may carry a secret (an echoed Authorization header, a password= in a message). Documents are sanitized on the way out — credential-shaped fields (password, token, api_key, …) and obvious in-text credential shapes become [redacted] — on a best-effort, deterministic basis before they reach a tool result, the model context or the UI.
• Untrusted data. All returned content is data, never instructions (below).

---

Credential Handling

Credentials follow the DeepSeek Harness model: references in config, values in the credential store.

• Config holds only a credential reference (e.g. passwordRef), never a secret. The plugin rejects any config containing a literal password, token or apiKey so a secret can never be committed to Git.
• References derive from the source id: SEARCHOPS_USERNAME_<id>, SEARCHOPS_PASSWORD_<id>, SEARCHOPS_TOKEN_<id>. You may override any of them to share one credential across sources.
• At request time the runtime resolves the value and builds the header: basic → Authorization: Basic base64(user:pass); bearer → Authorization: Bearer <token>; none → no header.
• If a required credential is missing, the request is refused before it reaches the network, with a message naming the reference to set — never the value.
• searchops_sources reports credentials as a boolean (configured / missing) and never echoes a value.

Store the secret for the prod example above under SEARCHOPS_PASSWORD_prod in the DSH credential store (Settings → Credentials, or your host's secret backend).

---

Untrusted Data

Search result content is untrusted data and must never be interpreted as instructions to the agent.

Everything inside OpenSearch — log messages, field values, index names, mappings — is treated as untrusted. A log line may literally contain Ignore previous instructions or run rm -rf /; that is just data.

• Log content is never spliced into a system-like instruction and never allowed to control plugin behavior.
• Every returned message is single-lined (newlines/tabs collapsed) so untrusted text cannot forge extra rows or fake structure, and truncated to a bounded length.
• Evidence is returned as structured records ({ timestamp, message, … }), not as prose that could be read as a directive.
• Tool descriptions and the system-prompt guidance repeat the untrusted-data rule so the model is reminded on every call.

---

Limitations

• Read-only. No index/document writes, deletes, mapping changes, cluster setting writes, or bulk ingest. This is intentional for v0.1.
• OpenSearch only. Elasticsearch is a documented extension point, not implemented. There is no fake provider.
• No deep pagination. Offsets past 10 000 are rejected; search_after is not exposed yet. Use aggregations or narrow the query.
• No AWS SigV4. Basic, bearer and no-auth are supported; SigV4 is on the roadmap.
• Date math is forwarded, not parsed. The plugin validates the shape of a time value and guards the window; OpenSearch interprets now-15m, ISO-8601 and epoch millis.
• Detection is best-effort. A source with an unusual schema should bind an explicit field profile rather than rely on auto-detection.

---

Roadmap

• Elasticsearch provider behind the same SearchProvider interface.
• AWS SigV4 auth for managed OpenSearch.
• search_after cursor pagination for deep result sets.
• Guarded write operations (e.g. index lifecycle, document delete) bound to dsh-human-intent for explicit human authorization — never in a read-only release.
• Composition with dsh-grafana: a Grafana alert → SearchOps logs → root-cause reasoning flow, orchestrated by the agent (no plugin-to-plugin RPC needed).

---

License

MIT © guhanfei-ai. See LICENSE.

Interacting with the OpenSearch REST API does not bundle or redistribute any OpenSearch source code.