Robots.txt Generator

What does a robots.txt file do?

A robots.txt file at the root of a domain tells crawlers which paths they may request. It controls crawling, not indexing — a blocked URL can still appear in search results if other sites link to it, so keeping a page out of the index requires a noindex meta tag instead.

Last updated 2026-07-30

Create a correctly formatted robots.txt with allow and disallow rules, crawl-delay settings and your sitemap URL — without memorising the syntax.

How to use Robots.txt Generator

  1. Choose a preset or start blank — Pick a common configuration such as "allow everything" and adjust from there.
  2. Add your rules — Set per-crawler allow and disallow paths, and point to your sitemap.
  3. Upload the file — Copy the output into a file named robots.txt at the root of your domain.

Frequently asked questions

Does robots.txt keep a page out of Google?
No — it only prevents crawling. A blocked URL can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
Where does robots.txt go?
At the root of the domain, reachable at yoursite.com/robots.txt. Crawlers do not look anywhere else.
Should I block my CSS and JavaScript?
No. Google renders pages to evaluate them. Blocking assets stops it seeing your layout properly and can hurt rankings.

Related tools