robots.txt and sitemap checker

robots.txt & sitemap checker

Type your address and we fetch the two files every crawler reads before anything else — /robots.txt and your sitemap — and tell you, in plain English, whether they are helping search engines in or shutting them out. No account, no email.

What this checks

• Is there a robots.txt at all — and does it return a real page rather than a 404 or an error.
• Does it block the whole site — a Disallow: / for everyone. This is the single most common reason a site is missing from search results, and it is usually left over from a staging server.
• Does it block your CSS or JavaScript — Google renders pages to judge them, and needs those files to do it. Blocking them makes your page look broken to the renderer while looking fine to you.
• Does it name a sitemap — so crawlers find it without being told separately.
• Is the sitemap reachable and valid — a sitemap that 404s or is malformed is silently ignored, and the pages it lists are never submitted.
• Is it empty, too large, or full of the wrong domain — each of those makes a crawler distrust or reject the file.
• Does it carry lastmod dates — the one sitemap field Google documents as a signal for when to recrawl a page.

Why these two files decide so much

A search engine reads robots.txt first. If it says stay out, the crawler does — politely, quietly, and without telling you. Nothing appears in any dashboard; the pages simply never get indexed. It is a one-line mistake with the largest possible consequence, which is why it is worth checking by hand once.

The sitemap is the other half. It is how you tell a crawler what exists, especially pages it has not found by following links. A sitemap that is not served, or not valid, is worse than none: you believe the pages have been announced, and they have not.