Searqon Docs

Searqon Technical Documentation

Searqon is a self-hosted web search, scraping, and token-chunking engine compiled into a single static Go binary. It provides a standardized data extraction layer for autonomous AI agents, local RAG pipelines, and search tooling.

Subsystem Technology / Approach Description
Discovery Multi-provider aggregator SearXNG, DuckDuckGo, Wikipedia, Arxiv with reciprocal rank fusion (RRF) and deduplication.
Stealth Scraper 4-tier escalation engine Automated fallback: Fast HTTP → Spoofed Headers → Headless CDP → Residential Proxy.
Agent Protocols MCP & UTCP First-class Model Context Protocol (stdio/SSE) and Universal Tool Calling Protocol support.
Extraction & RAG Markdown parser + BM25 chunker Strips boilerplate, generates semantic token windows with heading breadcrumbs and BM25 scores.
Storage Embedded SQLite / PostgreSQL Zero-configuration SQLite WAL mode by default; optional PostgreSQL + pgvector and Redis cache.

Architecture & Ingestion Pipeline

Searqon is organized as a decoupled, multi-stage ingestion engine. Incoming queries and extraction requests flow through security guardrails, federated discovery, 4-tier anti-bot escalation, and semantic token chunking.

System Map · Archify Blueprint Open Interactive Diagram
Searqon Architecture Blueprint
High-resolution 2048×1320 architecture map. Click to launch the interactive viewer with live path tracing, dark/light themes, and pan/zoom controls.

Quick Start & Docker Deployment

Searqon ships as a single static binary in a scratch Docker image (~8.2 MB). It requires no external database or background daemons to run.

Running with Docker

Bash
docker run -d \
  --name searqon \
  -p 7493:7493 \
  -v $(pwd)/data:/app/data \
  sooriya04/searqon:latest

Health Verification

Bash
curl -s http://localhost:7493/health | jq .
Default Network Binding

Searqon binds to 0.0.0.0:7493. Change this using the SERVER_PORT environment variable or within config/settings.yml.

Compiling from Source

Requires Go 1.22+:

Bash
git clone https://github.com/Sooriya04/Searqon.git
cd Searqon/src
go build -o searqon ./...
./searqon

Storage Backends

Searqon operates in embedded mode by default, persisting metadata, FTS5 full-text indices, and cache entries to an embedded SQLite file with WAL (Write-Ahead Logging) enabled.

SQLite (Default)

Created automatically at data/searqon.db upon first start. Includes full-text search tables with deduplicated URL hashing.

PostgreSQL + pgvector

For multi-instance deployments or high-throughput vector search across cached documents:

YAML
# config/settings.yml
database:
  driver: "postgres"
  dsn: "postgres://searqon:secret@localhost:5432/searqon?sslmode=disable"

Redis Cache Layer

When configured, Redis acts as an in-memory L1 cache ahead of the relational store:

YAML
cache:
  redis_url: "redis://localhost:6379"
  ttl: 3600 # 1 hour TTL

Universal Tool Calling Protocol (UTCP)

Searqon implements the Universal Tool Calling Protocol specification. Agents can dynamically discover available search and extraction capabilities by requesting the machine-readable manifest:

HTTP
GET /utcp/manifest HTTP/1.1
Host: localhost:7493

The returned JSON schema defines tools (search, scrape, pipeline), typing constraints, parameter requirements, and return structures ready for autonomous invocation.

Model Context Protocol (MCP)

Searqon runs a native MCP server implementation supporting both stdio and SSE transports. This enables direct integration with Claude Desktop, Continue, Cursor, and any MCP client.

Configuring Claude Desktop

Add to claude_desktop_config.json:

JSON
{
  "mcpServers": {
    "searqon": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-v", "/path/to/data:/app/data",
        "sooriya04/searqon:latest",
        "--mcp-stdio"
      ]
    }
  }
}

MCP over HTTP/SSE

Connect via Server-Sent Events at:

HTTP
GET http://localhost:7493/mcp/sse

SSE Event Stream

For interactive agent workflows, GET /search/stream streams discovery, per-URL scraping, and final synthesis events in real-time over a persistent Server-Sent Events channel.

Bash
curl -N "http://localhost:7493/search/stream?query=distributed+consensus&limit=3"
Event Type Timing Payload Contents
discovery Instant (100–300ms) Raw search results (URL list, titles, snippets, source engine).
scrape Per result completion Purified Markdown body, word count, cryptographic content hash.
synthesis Pipeline complete Consolidated, deduplicated array sorted by reciprocal rank score.
done Stream close Final execution metrics (total duration, cache hits, errors).

REST Endpoint Matrix

Verb Path Description
POST /search Synchronous multi-engine search with optional instant scrape extraction.
GET /search/stream Server-Sent Events streaming search, discovery, and scrape pipeline.
POST /scrape Fetch a single URL through the 4-tier anti-bot ladder; returns Markdown.
POST /scrape/chunked Extract and split webpage into semantic token windows with BM25 scores.
POST /scrape/batch Concurrent multi-URL scraping with bounded worker pools.
POST /pipeline End-to-end Search → Fetch → Chunk → Rank orchestration.
POST /crawl Recursive domain crawler with depth constraints and link deduplication.
POST /map Fast domain sitemap generation without deep content parsing.
GET /health Service health status and active provider list.
GET /metrics Prometheus exposition endpoint for latency, requests, and tier rates.
GET /logs Structured JSON request telemetry for audit and debugging.

Search API Reference

POST /search

Dispatches queries across configured search engines and optionally purifies top hits.

JSON Request
{
  "query": "sqlite wal mode concurrency",
  "limit": 5,
  "scrape": true,
  "provider": "searxng"
}
JSON Response
{
  "success": true,
  "data": {
    "query": "sqlite wal mode concurrency",
    "total": 5,
    "provider": "searxng",
    "duration_ms": 284,
    "results": [
      {
        "title": "Write-Ahead Logging - SQLite",
        "url": "https://sqlite.org/wal.html",
        "snippet": "WAL provides more concurrency as readers do not block writers...",
        "markdown": "# Write-Ahead Logging\n\nBy default, SQLite writes directly...",
        "word_count": 1250,
        "content_hash": "a8f3b92c4d12e6",
        "scraped": true,
        "escalation_tier": "fast_http"
      }
    ]
  }
}

Scrape & Chunking Reference

POST /scrape

JSON Request
{
  "url": "https://news.ycombinator.com",
  "format": "markdown",
  "stealth_level": "auto",
  "timeout_ms": 10000
}

POST /scrape/chunked

Splits extracted content into semantic token windows ready for vector databases.

JSON Request
{
  "url": "https://arxiv.org/abs/2312.00752",
  "chunk_size": 500,
  "chunk_overlap": 50,
  "include_bm25": true
}

Crawl & Map Reference

POST /crawl

JSON Request
{
  "url": "https://docs.kernel.org",
  "max_depth": 2,
  "max_pages": 50,
  "concurrency": 4,
  "respect_robots": true
}

Telemetry & Observability

Searqon provides real-time Prometheus metrics at GET /metrics:

Text
# HELP searqon_requests_total Total requests processed
# TYPE searqon_requests_total counter
searqon_requests_total{endpoint="/search"} 1420
searqon_requests_total{endpoint="/scrape"} 3891

# HELP searqon_escalation_tier_total Scrape attempts per escalation tier
# TYPE searqon_escalation_tier_total counter
searqon_escalation_tier_total{tier="t1_fast"} 3201
searqon_escalation_tier_total{tier="t2_stealth"} 540
searqon_escalation_tier_total{tier="t3_headless"} 122
searqon_escalation_tier_total{tier="t4_proxy"} 28

Configuration (settings.yml)

Searqon loads configuration from config/settings.yml with complete environment variable override support.

YAML
server:
  port: 7493
  host: "0.0.0.0"

database:
  driver: "sqlite"            # "sqlite" | "postgres"
  path: "data/searqon.db"     # SQLite only
  # dsn: "postgres://..."    # PostgreSQL only

cache:
  ttl: 3600                   # Cache TTL in seconds
  # redis_url: "redis://localhost:6379"

search:
  default_provider: "searxng"
  searxng_url: "http://localhost:8080"
  providers:
    - searxng
    - duckduckgo
    - wikipedia
    - arxiv

scraper:
  timeout_ms: 10000
  max_concurrency: 8
  default_tier: "auto"        # "auto" | "fast" | "stealth" | "headless"

rate_limiter:
  requests_per_minute: 60
  burst: 10

Environment Variable Overrides

Environment Variable Default Value Description
SERVER_PORT 7493 TCP listening port.
DATABASE_DRIVER sqlite Relational database backend (sqlite or postgres).
DATABASE_DSN — PostgreSQL connection string.
SEARXNG_URL http://localhost:8080 Upstream SearXNG instance endpoint.
REDIS_URL — Optional Redis L1 cache endpoint.
SEARQON_ALLOW_PRIVATE false Allows scraping private/loopback IP ranges (disables SSRF guard).

Anti-Bot Escalation Tiers

Every scraping request begins at Tier 1 and transparently steps through higher tiers upon detecting HTTP 403/429 status codes, CAPTCHA markers, or JavaScript execution hurdles.

Tier Mechanism Typical Latency Trigger Condition
Tier 1 Direct HTTP (Go net/http) 50–200ms Initial request attempt.
Tier 2 Spoofed User-Agent, Accept-Headers, TLS cipher suite 200–500ms Status 403 or basic header inspection block.
Tier 3 Headless CDP browser (Lightpanda / Camoufox) 1.5–3.5s Cloudflare 5s challenge, heavy client-side JavaScript.
Tier 4 Residential proxy pool rotation 2.5–6.0s IP rate-limiting, ASN reputation block, or geo-restriction.
SSRF Protection

Searqon strictly enforces SSRF protection. Requests attempting to resolve loopback (127.0.0.1), RFC 1918 private subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), or link-local addresses (169.254.0.0/16) are rejected immediately unless SEARQON_ALLOW_PRIVATE=true is explicitly set.

Docker Compose Reference

YAML (docker-compose.yml)
version: "3.9"

services:
  searqon:
    image: sooriya04/searqon:latest
    ports:
      - "7493:7493"
    environment:
      - SEARXNG_URL=http://searxng:8080
      - DATABASE_DRIVER=sqlite
    volumes:
      - ./data:/app/data
    depends_on:
      - searxng
    restart: unless-stopped

  searxng:
    image: searxng/searxng:latest
    ports:
      - "8080:8080"
    environment:
      - SEARXNG_BASE_URL=http://localhost:8080
    volumes:
      - ./searxng:/etc/searxng:ro
    restart: unless-stopped