The Search & Web Retrieval Layer Built for AI Agents
Searqon searches, crawls, extracts, and purifies raw web pages into clean markdown. Self-hosted, privacy-first, and highly performant.
Dual-Provider Search
Queries a local SearXNG cluster as the primary engine, falling back gracefully to DuckDuckGo Lite HTML if SearXNG is rate-limited or down.
Sub-5ms Cache Check
Maintains a dual-layer PostgreSQL cache pool. Search results (24h TTL) and page scrapes (7-day TTL) load instantly for recurring requests.
Lightpanda Rendering
Spawns the Lightpanda headless browser dynamically to render client-side JavaScript (React, Next.js) for deep text extraction.
Robots.txt Obedience
Audits remote rules, respects crawling limits, respects crawl-delays, and gracefully switches User-Agents according to site policies.
SearqonBot Verification
SearqonBot/1.0 is the web crawler utilized by self-hosted Searqon search installations. It compiles public web content to provide structured text data for users and AI summarization apps.
We value webmaster policies and work hard to ensure the bot behaves politely:
Anti-Overload Safeguards
Honors crawl delays specified in robots.txt and restricts timeouts to 7-8 seconds to avoid pinning server threads.
Opt-Out Capabilities
Allows webmasters to completely block or route SearqonBot using standard instructions in their robots.txt file.
Zero Security Intrusion
We only scrape public webpages. We never scan paths for vulnerabilities or bypass security payloads.