Robots.txt Generator

Build a robots.txt to control search crawlers.

User-agent

Crawling

Disallow paths (one per line)

Sitemap URL (optional)

Save the output as robots.txt in your site's root folder (e.g. example.com/robots.txt).

robots.txt

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Allow: /

Flowing through EveryFileTool…

Processing your file right in your browser

Working…

How to use the Robots.txt Generator

Simple, free and instant - no sign-up required.

Fill in your details

Enter your page info in the fields - everything is generated in your browser, nothing is uploaded.

Adjust the options

Tweak the settings to generate exactly the markup or URL you need.

Get the code instantly

Your tags, code or URL build live as you type - no waiting and no processing queue.

Copy & paste into your site

Copy the output or download the file, then drop it into your page head, server config or CMS.

Frequently asked questions

Useful information about Robots.txt Generator.

What is a robots.txt file?

robots.txt tells search-engine crawlers which parts of your site they may or may not crawl. This tool builds one with your allow/disallow rules and an optional sitemap reference.

Where do I put robots.txt?

In your site's root directory, so it's reachable at yourdomain.com/robots.txt. Download the file from this tool and upload it there.

Does blocking a page in robots.txt remove it from Google?

No — it only stops crawling, not indexing. To keep a page out of search results, use a noindex meta tag instead (and don't block it in robots.txt, or Google can't see the noindex).

Robots.txt Generator Online - EveryFileTool