Meta Robots Tag: noindex, nofollow, and When to Use Them
What the Robots Meta Tag Does
The meta robots tag tells search engine crawlers how to handle a page. The two most important directives are index/noindex (whether to include the page in search results) and follow/nofollow (whether to follow links on the page). Unlike robots.txt, which blocks crawling, noindex allows crawling but prevents indexing.
When to Use noindex
Admin pages, login pages, thank-you pages, duplicate content, thin content pages, staging sites, paginated pages beyond page 2, search results pages. Using noindex on any of these keeps your crawl budget focused on your most valuable content.
Critical Mistakes to Avoid
Blocking CSS and JS in robots.txt while using meta robots to index — Google needs to render pages to understand them. Setting noindex on your homepage accidentally. Using both robots.txt block AND noindex — Google may not even read the noindex tag if the page is blocked from crawling.
Frequently Asked Questions
Set robots tags correctly
Generate proper meta robots tags with the free Meta Tag Generator.
Open Meta Tag Generator