Open Source Search & Extraction Engine · MIT License

Fast, self-hosted web search & scraping for developer tools.

A single Go binary that provides multi-engine search aggregation, 4-tier anti-bot escalation, and clean Markdown extraction without third-party SaaS dependencies.

$ docker run -d -p 7493:7493 -v $(pwd)/data:/app/data sooriya04/searqon:latest
Runtime Size
~8.2 MB
Scratch container image
Memory Footprint
< 20 MB
Idle resident memory
Anti-Bot Escalation
4 Tiers
Automatic challenge fallback
Storage Backends
SQLite / PG
Embedded WAL or pgvector

Core Subsystems

Designed to execute local search, anti-bot resolution, and structured extraction in a predictable, high-throughput pipeline.

System Architecture · Archify Blueprint Open Interactive Diagram
Searqon Architecture Blueprint
SUBSYSTEM 01
Multi-Engine Search Aggregation
Dispatches search queries simultaneously across SearXNG, DuckDuckGo, Wikipedia, and Arxiv. Merges disparate hit lists with reciprocal rank fusion (RRF) and deduplicates pages by cryptographic SHA-256 content hashes.
SUBSYSTEM 02
4-Tier Anti-Bot Escalation Ladder
Starts requests on fast, low-overhead HTTP. If a target presents bot barriers or JavaScript challenges, Searqon automatically escalates through browser header spoofing, headless CDP (Lightpanda/Camoufox), and residential proxies.
SUBSYSTEM 03
Token-Bounded Extraction & Chunking
Strips ads, tracking scripts, and navigation boilerplate using readability algorithms. Produces clean Markdown with word counts, heading breadcrumbs, and pre-chunked sliding token windows ready for vector database ingestion.
SUBSYSTEM 04
Agent Protocols: MCP & UTCP
Direct native support for Model Context Protocol (stdio and SSE transports) and the Universal Tool Calling Protocol (UTCP). AI agents inspect the schema manifest at runtime without custom adapter glue.

Anti-Bot Escalation Ladder

Each target URL starts at Tier 1 and transparently progresses only when challenges or blocks are encountered.

Tier Resolution Mechanism Typical Latency Trigger Threshold
Tier 1 Direct HTTP client (Go standard library net/http) 50–200ms Standard initial fetch attempt.
Tier 2 Browser header spoofing, TLS fingerprint imitation 200–500ms HTTP 403 Forbidden or static header inspection.
Tier 3 Headless CDP browser (Lightpanda / Camoufox) 1.5–3.5s Cloudflare 5s challenge, heavy client-side JavaScript.
Tier 4 Residential proxy pool rotation 2.5–6.0s IP rate-limiting, ASN block, or geographic restriction.

REST API Explorer

All endpoints accept standard JSON and run on port 7493 by default. See the complete documentation for exhaustive request options.

POST http://localhost:7493/search HTTP/1.1
Request Payload

                    
JSON Response

                    

Webmaster Policy & Robots.txt

Searqon identifies itself using a standard crawler User-Agent and strictly adheres to robots.txt rules when operating in crawl mode.

Crawler Identity (User-Agent)

Sent by the crawler and scraper tiers on non-stealth requests. Webmasters can use this identifier in robots.txt rules:

SearqonBot/2.0 (+https://github.com/Sooriya04/Searqon)

Robots.txt Directive Recipes

Sample access policies for webmasters configuring site behavior: