Site crawling and indexing

What it is

A crawler is the foundation of every SEO tool. Before you can audit a page, score its speed, cluster its keywords or predict its traffic, something has to fetch it, parse it and record what it found. The crawler is that something, and in this platform it is the first pillar everything else stands on.

It is an async crawler built on httpx and BeautifulSoup. That choice matters: it means the crawler has almost no dependencies, starts in milliseconds, and can be deployed on a small VPS. It will optionally use Playwright to render JavaScript when you need it, and degrade back to plain HTTP when you do not.

How it works

You point it at a domain and it does the patient work:

• Honours robots.txt, including Crawl-delay, and applies per-host rate limits so you never hammer a server.
• Discovers URLs from internal links and your sitemap, building a frontier and crawling breadth-first to a depth you set.
• Follows and records redirect chains, so a 301 that became a 302 that became a loop is visible as a chain rather than a mystery.
• Extracts on-page signals: title, meta description, headings, canonical, hreflang, links, images, alt text and structured data.
• Detects duplicates by content hashing and canonical analysis, which is how you find the same page published at four URLs.
• Captures render timing so Core Web Vitals work even without a real browser or a field-data API.

What it does for your site

A crawl answers questions that are otherwise guesswork. How many pages does your site actually have, as opposed to how many you think it has? Which pages are orphaned, reachable by no internal link at all? Which titles are duplicated across dozens of URLs, splitting their own ranking signal? Which images are missing alt text? Which redirect chains are quietly leaking link equity?

Every one of those is a real, fixable problem with a measurable cost. The crawler finds them all in one pass, and stores the result so the audit, the graph and the forecasts can reuse it rather than re-crawling.

Where the results go

The crawl output is not an end in itself. It feeds the on-page audit, which turns pages into a prioritised fix list, and the Core Web Vitals checks. It also builds the link graph that authority metrics are computed from.

Crawls run as queued jobs, so a 5,000-page crawl does not hold a browser connection open for minutes. You get a job id back immediately and poll for progress — which is why the dashboard can show live progress on a long crawl without timing out.

Configuration