Searqon Technical Documentation
Searqon is a self-hosted web search, scraping, and token-chunking engine compiled into a single static Go binary. It provides a standardized data extraction layer for autonomous AI agents, local RAG pipelines, and search tooling.
| Subsystem | Technology / Approach | Description |
|---|---|---|
| Discovery | Multi-provider aggregator | SearXNG, DuckDuckGo, Wikipedia, Arxiv with reciprocal rank fusion (RRF) and deduplication. |
| Stealth Scraper | 4-tier escalation engine | Automated fallback: Fast HTTP → Spoofed Headers → Headless CDP → Residential Proxy. |
| Agent Protocols | MCP & UTCP | First-class Model Context Protocol (stdio/SSE) and Universal Tool Calling Protocol support. |
| Extraction & RAG | Markdown parser + BM25 chunker | Strips boilerplate, generates semantic token windows with heading breadcrumbs and BM25 scores. |
| Storage | Embedded SQLite / PostgreSQL | Zero-configuration SQLite WAL mode by default; optional PostgreSQL + pgvector and Redis cache. |
Architecture & Ingestion Pipeline
Searqon is organized as a decoupled, multi-stage ingestion engine. Incoming queries and extraction requests flow through security guardrails, federated discovery, 4-tier anti-bot escalation, and semantic token chunking.
Quick Start & Docker Deployment
Searqon ships as a single static binary in a scratch Docker image (~8.2 MB). It requires no external database or background daemons to run.
Running with Docker
docker run -d \
--name searqon \
-p 7493:7493 \
-v $(pwd)/data:/app/data \
sooriya04/searqon:latest
Health Verification
curl -s http://localhost:7493/health | jq .
Searqon binds to 0.0.0.0:7493. Change this using the SERVER_PORT environment variable or within config/settings.yml.
Compiling from Source
Requires Go 1.22+:
git clone https://github.com/Sooriya04/Searqon.git
cd Searqon/src
go build -o searqon ./...
./searqon
Storage Backends
Searqon operates in embedded mode by default, persisting metadata, FTS5 full-text indices, and cache entries to an embedded SQLite file with WAL (Write-Ahead Logging) enabled.
SQLite (Default)
Created automatically at data/searqon.db upon first start. Includes full-text search tables with deduplicated URL hashing.
PostgreSQL + pgvector
For multi-instance deployments or high-throughput vector search across cached documents:
# config/settings.yml
database:
driver: "postgres"
dsn: "postgres://searqon:secret@localhost:5432/searqon?sslmode=disable"
Redis Cache Layer
When configured, Redis acts as an in-memory L1 cache ahead of the relational store:
cache:
redis_url: "redis://localhost:6379"
ttl: 3600 # 1 hour TTL
Universal Tool Calling Protocol (UTCP)
Searqon implements the Universal Tool Calling Protocol specification. Agents can dynamically discover available search and extraction capabilities by requesting the machine-readable manifest:
GET /utcp/manifest HTTP/1.1
Host: localhost:7493
The returned JSON schema defines tools (search, scrape, pipeline), typing constraints, parameter requirements, and return structures ready for autonomous invocation.
Model Context Protocol (MCP)
Searqon runs a native MCP server implementation supporting both stdio and SSE transports. This enables direct integration with Claude Desktop, Continue, Cursor, and any MCP client.
Configuring Claude Desktop
Add to claude_desktop_config.json:
{
"mcpServers": {
"searqon": {
"command": "docker",
"args": [
"run", "-i", "--rm",
"-v", "/path/to/data:/app/data",
"sooriya04/searqon:latest",
"--mcp-stdio"
]
}
}
}
MCP over HTTP/SSE
Connect via Server-Sent Events at:
GET http://localhost:7493/mcp/sse
SSE Event Stream
For interactive agent workflows, GET /search/stream streams discovery, per-URL scraping, and final synthesis events in real-time over a persistent Server-Sent Events channel.
curl -N "http://localhost:7493/search/stream?query=distributed+consensus&limit=3"
| Event Type | Timing | Payload Contents |
|---|---|---|
discovery |
Instant (100–300ms) | Raw search results (URL list, titles, snippets, source engine). |
scrape |
Per result completion | Purified Markdown body, word count, cryptographic content hash. |
synthesis |
Pipeline complete | Consolidated, deduplicated array sorted by reciprocal rank score. |
done |
Stream close | Final execution metrics (total duration, cache hits, errors). |
REST Endpoint Matrix
| Verb | Path | Description |
|---|---|---|
| POST | /search |
Synchronous multi-engine search with optional instant scrape extraction. |
| GET | /search/stream |
Server-Sent Events streaming search, discovery, and scrape pipeline. |
| POST | /scrape |
Fetch a single URL through the 4-tier anti-bot ladder; returns Markdown. |
| POST | /scrape/chunked |
Extract and split webpage into semantic token windows with BM25 scores. |
| POST | /scrape/batch |
Concurrent multi-URL scraping with bounded worker pools. |
| POST | /pipeline |
End-to-end Search → Fetch → Chunk → Rank orchestration. |
| POST | /crawl |
Recursive domain crawler with depth constraints and link deduplication. |
| POST | /map |
Fast domain sitemap generation without deep content parsing. |
| GET | /health |
Service health status and active provider list. |
| GET | /metrics |
Prometheus exposition endpoint for latency, requests, and tier rates. |
| GET | /logs |
Structured JSON request telemetry for audit and debugging. |
Search API Reference
POST /search
Dispatches queries across configured search engines and optionally purifies top hits.
{
"query": "sqlite wal mode concurrency",
"limit": 5,
"scrape": true,
"provider": "searxng"
}
{
"success": true,
"data": {
"query": "sqlite wal mode concurrency",
"total": 5,
"provider": "searxng",
"duration_ms": 284,
"results": [
{
"title": "Write-Ahead Logging - SQLite",
"url": "https://sqlite.org/wal.html",
"snippet": "WAL provides more concurrency as readers do not block writers...",
"markdown": "# Write-Ahead Logging\n\nBy default, SQLite writes directly...",
"word_count": 1250,
"content_hash": "a8f3b92c4d12e6",
"scraped": true,
"escalation_tier": "fast_http"
}
]
}
}
Scrape & Chunking Reference
POST /scrape
{
"url": "https://news.ycombinator.com",
"format": "markdown",
"stealth_level": "auto",
"timeout_ms": 10000
}
POST /scrape/chunked
Splits extracted content into semantic token windows ready for vector databases.
{
"url": "https://arxiv.org/abs/2312.00752",
"chunk_size": 500,
"chunk_overlap": 50,
"include_bm25": true
}
Crawl & Map Reference
POST /crawl
{
"url": "https://docs.kernel.org",
"max_depth": 2,
"max_pages": 50,
"concurrency": 4,
"respect_robots": true
}
Telemetry & Observability
Searqon provides real-time Prometheus metrics at GET /metrics:
# HELP searqon_requests_total Total requests processed
# TYPE searqon_requests_total counter
searqon_requests_total{endpoint="/search"} 1420
searqon_requests_total{endpoint="/scrape"} 3891
# HELP searqon_escalation_tier_total Scrape attempts per escalation tier
# TYPE searqon_escalation_tier_total counter
searqon_escalation_tier_total{tier="t1_fast"} 3201
searqon_escalation_tier_total{tier="t2_stealth"} 540
searqon_escalation_tier_total{tier="t3_headless"} 122
searqon_escalation_tier_total{tier="t4_proxy"} 28
Configuration (settings.yml)
Searqon loads configuration from config/settings.yml with complete environment variable override support.
server:
port: 7493
host: "0.0.0.0"
database:
driver: "sqlite" # "sqlite" | "postgres"
path: "data/searqon.db" # SQLite only
# dsn: "postgres://..." # PostgreSQL only
cache:
ttl: 3600 # Cache TTL in seconds
# redis_url: "redis://localhost:6379"
search:
default_provider: "searxng"
searxng_url: "http://localhost:8080"
providers:
- searxng
- duckduckgo
- wikipedia
- arxiv
scraper:
timeout_ms: 10000
max_concurrency: 8
default_tier: "auto" # "auto" | "fast" | "stealth" | "headless"
rate_limiter:
requests_per_minute: 60
burst: 10
Environment Variable Overrides
| Environment Variable | Default Value | Description |
|---|---|---|
SERVER_PORT |
7493 |
TCP listening port. |
DATABASE_DRIVER |
sqlite |
Relational database backend (sqlite or postgres). |
DATABASE_DSN |
— | PostgreSQL connection string. |
SEARXNG_URL |
http://localhost:8080 |
Upstream SearXNG instance endpoint. |
REDIS_URL |
— | Optional Redis L1 cache endpoint. |
SEARQON_ALLOW_PRIVATE |
false |
Allows scraping private/loopback IP ranges (disables SSRF guard). |
Anti-Bot Escalation Tiers
Every scraping request begins at Tier 1 and transparently steps through higher tiers upon detecting HTTP 403/429 status codes, CAPTCHA markers, or JavaScript execution hurdles.
| Tier | Mechanism | Typical Latency | Trigger Condition |
|---|---|---|---|
| Tier 1 | Direct HTTP (Go net/http) |
50–200ms | Initial request attempt. |
| Tier 2 | Spoofed User-Agent, Accept-Headers, TLS cipher suite | 200–500ms | Status 403 or basic header inspection block. |
| Tier 3 | Headless CDP browser (Lightpanda / Camoufox) | 1.5–3.5s | Cloudflare 5s challenge, heavy client-side JavaScript. |
| Tier 4 | Residential proxy pool rotation | 2.5–6.0s | IP rate-limiting, ASN reputation block, or geo-restriction. |
Searqon strictly enforces SSRF protection. Requests attempting to resolve loopback (127.0.0.1), RFC 1918 private subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), or link-local addresses (169.254.0.0/16) are rejected immediately unless SEARQON_ALLOW_PRIVATE=true is explicitly set.
Docker Compose Reference
version: "3.9"
services:
searqon:
image: sooriya04/searqon:latest
ports:
- "7493:7493"
environment:
- SEARXNG_URL=http://searxng:8080
- DATABASE_DRIVER=sqlite
volumes:
- ./data:/app/data
depends_on:
- searxng
restart: unless-stopped
searxng:
image: searxng/searxng:latest
ports:
- "8080:8080"
environment:
- SEARXNG_BASE_URL=http://localhost:8080
volumes:
- ./searxng:/etc/searxng:ro
restart: unless-stopped