Back to resources

Noindex and Crawl Blocking

Accidental noindex or robots blocks can remove important pages from search results.

Why this matters

  • - Noindex removes pages from the index.
  • - Robots.txt disallow prevents crawling (not a substitute for noindex).
  • - Staging leaks can tank production URLs.

How to fix it

  1. 1. Audit meta robots and X-Robots-Tag headers.
  2. 2. Remove noindex from pages you want indexed.
  3. 3. Align robots.txt with intended crawl policy.

Weak example

Production category pages ship with meta robots noindex.

Better example

Only admin and cart routes are noindexed intentionally.