Robots.txt Builder
Create robots.txt files with user-agent rules, allow/disallow paths, crawl delay, and sitemap URL. Download or copy with one click.
Frequently Asked Questions
What is robots.txt?
▼
Robots.txt is a text file placed in the root of your website that tells search engine crawlers which pages they can or cannot access.
Where should I place robots.txt?
▼
Place it in the root directory of your domain, accessible at https://yourdomain.com/robots.txt.
Does robots.txt block indexing?
▼
No, robots.txt only prevents crawling, not indexing. To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header.
What is Crawl-delay?
▼
Crawl-delay tells crawlers to wait a specified number of seconds between requests. Not all search engines support it.
Is my data safe?
▼
Yes. Everything runs in your browser. No data is sent to any server.