Sitemap and robots.txt problems are some of the most damaging technical SEO issues, precisely because they’re invisible to anyone browsing the site normally. A visitor sees a working page; a search engine sees a directive telling it to ignore that same page entirely.
A blanket disallow rule left over from staging
The single most damaging mistake: a Disallow: / rule in robots.txt, left in place after a site migrated from a staging environment to production. This tells every well-behaved crawler to ignore the entire site. Check this first, on every audit, no exceptions.
Sitemap references URLs that redirect or 404
A sitemap should only ever list final, canonical, 200-status URLs. If it includes URLs that now redirect or return a 404, it wastes crawl budget and signals a lack of maintenance. WordPress sitemap plugins usually handle this automatically, but content deleted outside the plugin’s awareness (via direct database edits, for instance) can leave stale entries.
Sitemap isn’t referenced in robots.txt
Adding a Sitemap: line to robots.txt is a simple, free way to help search engines discover your sitemap immediately rather than waiting for it to be found some other way. It costs one line and there’s no reason to skip it.
Robots.txt blocks resources the page needs to render
Blocking CSS or JavaScript files in robots.txt can prevent Google from rendering the page correctly when it evaluates it, which can affect how the page is understood and ranked. This was more common advice years ago and is now actively counterproductive.
Noindex pages still appear in the sitemap
A page marked noindex shouldn’t also appear in your sitemap — the two signals contradict each other and, in Google’s own guidance, can slow down how quickly the noindex directive is respected. Keep these two systems in sync.
How to check your own site
Fetch /robots.txt and read it literally, line by line, as a crawler would. Then open your sitemap XML and spot-check ten to fifteen URLs for 200 status codes and matching canonical tags. Both checks take under five minutes and catch a disproportionate share of indexing problems.
The free audit checks your robots.txt and sitemap automatically as part of every scan, including the blanket-disallow mistake above.
Check your robots.txt and sitemap for freeThis is step three of our complete WordPress SEO audit checklist, which covers the rest of the crawlability and indexation basics.