Every URL-based source (url, sitemap,
feed, auto) flows through the same multi-tier
crawler chain. Each tier tries a different way to get readable text out
of the page. The first tier whose extracted text clears the
200-character threshold wins; failures cascade to the next tier.
| # | Tier | What it does | When it wins |
|---|---|---|---|
| 1 | plain_http |
Plain Guzzle GET with a Mozilla user-agent.
Runs through UrlSafetyGuard so private IPs and
cloud metadata services are blocked. |
Server-rendered pages (Blade apps, classic blogs, marketing sites with SSR or static export). |
| 2 | cloudflare_browser_markdown |
POST /browser-rendering/markdown. Cloudflare
spins up headless Chrome, waits for JS to settle, and emits
clean markdown. We wrap it in a minimal HTML envelope so
the extractor and chunker stay unchanged. |
JS-heavy SPAs whose final DOM is text-shaped (most React / Vue / Svelte sites). |
| 3 | cloudflare_browser |
POST /browser-rendering/content. Same headless
Chrome, but returns raw rendered HTML. Our
ReadabilityExtractor takes over. |
Atypical layouts where the markdown extractor's heuristics drop content but the regex extractor catches it (tables that don't map cleanly to GFM, deep nested article structures). |
| 4 | cloudflare_vision |
Two-stage:
|