Fast, self-hosted web search & scraping for developer tools.
A single Go binary that provides multi-engine search aggregation, 4-tier anti-bot escalation, and clean Markdown extraction without third-party SaaS dependencies.
Core Subsystems
Designed to execute local search, anti-bot resolution, and structured extraction in a predictable, high-throughput pipeline.
Anti-Bot Escalation Ladder
Each target URL starts at Tier 1 and transparently progresses only when challenges or blocks are encountered.
| Tier | Resolution Mechanism | Typical Latency | Trigger Threshold |
|---|---|---|---|
| Tier 1 | Direct HTTP client (Go standard library net/http) |
50–200ms | Standard initial fetch attempt. |
| Tier 2 | Browser header spoofing, TLS fingerprint imitation | 200–500ms | HTTP 403 Forbidden or static header inspection. |
| Tier 3 | Headless CDP browser (Lightpanda / Camoufox) | 1.5–3.5s | Cloudflare 5s challenge, heavy client-side JavaScript. |
| Tier 4 | Residential proxy pool rotation | 2.5–6.0s | IP rate-limiting, ASN block, or geographic restriction. |
REST API Explorer
All endpoints accept standard JSON and run on port 7493 by default. See the complete documentation for exhaustive request options.
Webmaster Policy & Robots.txt
Searqon identifies itself using a standard crawler User-Agent and strictly adheres to robots.txt rules when operating in crawl mode.
Crawler Identity (User-Agent)
Sent by the crawler and scraper tiers on non-stealth requests. Webmasters can use this identifier in robots.txt rules:
Robots.txt Directive Recipes
Sample access policies for webmasters configuring site behavior: