Robots.txt Generator
Generate robots.txt files for your website with allow/disallow rules for different user-agents. Free online robots.txt generator with sitemap support.
User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
What is Robots.txt Generator?
robots.txt is a text file placed at the root of a website (e.g., example.com/robots.txt) that tells web crawlers which pages they can and cannot access. It uses the Robots Exclusion Protocol to define rules per user-agent (Googlebot, Bingbot, etc.). A properly configured robots.txt prevents crawlers from indexing private pages, reduces server load, and guides search engines to your sitemap.
How to Use
- The default rule allows all crawlers access to the site
- Add Allow or Disallow rules for specific paths (e.g., /admin/, /private/)
- Add additional user-agents to create rules specific to certain crawlers
- Enter your sitemap URL to help crawlers discover your pages
- Copy the generated content and save it as robots.txt in your website root
Examples
Basic robots.txt
Block a specific crawler
Related Tools
Meta Tag Generator
Generate HTML meta tags for your website including title, description, viewport, charset, robots, and author. Free online meta tag generator with instant preview.
Open Graph Preview
Preview how your URL appears when shared on Facebook, LinkedIn, and Twitter. Generate Open Graph and Twitter Card meta tags. Free online OG preview tool.
URL Parser
Parse any URL into protocol, hostname, port, path, query parameters, and hash. Free online URL parser with structured table output.
HTTP Status Codes Reference
Complete reference for HTTP status codes with descriptions. Search and browse all status codes from 100 to 5xx.
Frequently Asked Questions
Where should I place robots.txt?
At the root of your domain: https://example.com/robots.txt. It must be accessible at this exact URL. If you have subdomains, each subdomain needs its own robots.txt file.
Can robots.txt prevent indexing?
No. robots.txt tells crawlers not to crawl pages, but if a page is linked from elsewhere, search engines may still index it based on those links. For true no-indexing, use the noindex meta tag or return an X-Robots-Tag HTTP header.
What is Crawl-delay?
Crawl-delay tells supporting crawlers how many seconds to wait between requests. Googlebot does not support this directive. Bingbot and YandexBot do. Use it to reduce server load from aggressive crawlers.