Searqon System Architecture

High-Throughput Web Search, 4-Tier Anti-Bot Escalation & RAG Extraction Pipeline

Searqon System Architecture High-Throughput Web Search, 4-Tier Anti-Bot Escalation & RAG Extraction Pipeline AI Agents & Clients · Claude / Cursor / LangChain · Architecture component · REST / MCP / UTCP AI Agents & Clients Claude / Cursor / LangChain REST / MCP / UTCP Security Guardrails · SSRF & Token Bucket · Searqon Standalone Engine (Go Binary) · IPv4/IPv6 Bounded Security Guardrails SSRF & Token Bucket IPv4/IPv6 Bounded HTTP & Protocol Gateway · :7493 Go stdlib router · Searqon Standalone Engine (Go Binary) · SSE / stdio HTTP & Protocol Gateway :7493 Go stdlib router SSE / stdio Telemetry & Metrics · /metrics & /logs · Searqon Standalone Engine (Go Binary) · Prometheus Telemetry & Metrics /metrics & /logs Prometheus SearXNG Engine · Local instance :8080 · External Discovery Providers · Multi-Engine SearXNG Engine Local instance :8080 Multi-Engine Public APIs · DDG / Wikipedia / Arxiv · External Discovery Providers · Direct Fallback Public APIs DDG / Wikipedia / Arxiv Direct Fallback Search & Discovery Hub · Circuit breaker & fusion · Searqon Standalone Engine (Go Binary) › Core Pipeline · RRF + BM25 Search & Discovery Hub Circuit breaker & fusion RRF + BM25 Crawler & Link Mapper · Depth & link dedup · Searqon Standalone Engine (Go Binary) › Core Pipeline · BFS Crawler Crawler & Link Mapper Depth & link dedup BFS Crawler 4-Tier Anti-Bot Engine · Auto escalation ladder · Searqon Standalone Engine (Go Binary) › Core Pipeline · CDP & Residential 4-Tier Anti-Bot Engine Auto escalation ladder CDP & Residential RAG Chunker & BM25 · Sliding token window · Searqon Standalone Engine (Go Binary) › Core Pipeline · 500t Window / 50 Overlap RAG Chunker & BM25 Sliding token window 500t Window / 50 Overlap Content Extractor · Readability & Markdown · Searqon Standalone Engine (Go Binary) › Core Pipeline · SHA-256 Provenance Content Extractor Readability & Markdown SHA-256 Provenance SQLite WAL Engine · Embedded data/searqon.db · Searqon Standalone Engine (Go Binary) · Default · FTS5 SQLite WAL Engine Embedded data/searqon.db Default · FTS5 PostgreSQL + pgvector · Scalable vector store · External Storage Services · Optional DB PostgreSQL + pgvector Scalable vector store Optional DB Redis L1 Cache · In-memory cache layer · External Storage Services · Optional :6379 Redis L1 Cache In-memory cache layer Optional :6379 JSON / SSE filter request metrics query crawl federate fallback scrape hits fetch link raw HTML purify MD FTS5 index vectors cache MD Searqon Standalone Engine (Go Binary) Core Pipeline External Discovery Providers External Storage Services Legend Backend Database Cloud Security External