Robots.txt Generator

Enter your website origin if you want to use the sitemap suggestion.

Select All crawlers, Googlebot, or Bingbot.

Add Allow and Disallow path rules, or start with a preset.

Review the path entries and correct any validation errors.

Add an optional sitemap URL. If the field is empty, check the suggested URL based on your site origin.

Set an optional Crawl-delay if needed and supported by your target crawler.

Preview the generated robots.txt content.

Copy the text or download it as robots.txt.

You can load a sample configuration to see how the directives work or clear the form to start again.

Create a robots.txt file for your website with this free online generator. Choose which crawlers the rules apply to, add Allow and Disallow paths, include a sitemap URL, and optionally set a crawl delay. Preview the generated directives, copy the text, or download it as a robots.txt file.

Whether you manage a small website or a larger project, a correctly configured robots.txt file can help communicate which parts of your site crawlers should or should not request.

A robots.txt file is a plain-text file that provides instructions to web crawlers about which URLs they are requested to access or avoid on a website.

It is part of the Robots Exclusion Protocol. Search engines and other crawlers can check this file before crawling pages on a site.

The file must be placed at the root of the relevant hostname. For example:

https://example.com/robots.txt

A file located at https://example.com/blog/robots.txt does not control crawling across the entire example.com website. Subdomains also need their own robots.txt files.

A robots.txt file uses directives to describe crawling preferences. The generator supports the following rules.

User-agent

The User-agent directive identifies the crawler to which the following rules apply.

Common examples include:

User-agent: *

User-agent: Googlebot

User-agent: Bingbot

* applies to all crawlers that follow the Robots Exclusion Protocol.

Googlebot identifies Google's main web crawler.

Bingbot identifies Microsoft's Bing crawler.

Rules for a specific crawler can differ from the rules for all crawlers. Check the generated output to confirm which group receives each rule.

Disallow

The Disallow directive tells a compliant crawler not to request URLs matching a specified path.

For example:

User-agent: *

Disallow: /private/

This asks matching crawlers not to crawl URLs under the /private/ path.

A Disallow rule does not remove a page from search results automatically. A URL may still be indexed if other pages link to it or if the search engine discovers it elsewhere.

Allow

The Allow directive identifies a path that crawlers may access, including when a broader Disallow rule also applies.

For example:

User-agent: *

Disallow: /private/

Allow: /private/public-guide/

This asks crawlers to avoid the private section while allowing the specified guide path. Support for Allow and how conflicting paths are interpreted can vary between crawlers, so check the documentation for the search engines you target.

Sitemap

The Sitemap directive provides the absolute URL of your XML sitemap.

For example:

Sitemap: https://example.com/sitemap.xml

A sitemap can help search engines discover URLs on your website. It does not override Disallow rules or guarantee that every listed URL will be crawled or indexed.

If the sitemap field is empty and you enter a valid website origin, the generator can suggest a sitemap URL using that origin. Check that the suggested path matches the actual sitemap location on your site before publishing.

Crawl-delay

The optional Crawl-delay directive requests a delay between crawler requests.

For example:

User-agent: *

Crawl-delay: 10

This asks a crawler that supports the directive to wait 10 seconds between requests.

Crawler support varies. Googlebot does not support the Crawl-delay directive in robots.txt. Do not assume that adding it will slow down every search engine crawler. Check the documentation for each crawler before relying on this setting.

The generator supports three user-agent choices:

All crawlers (*): Create rules for all crawlers that follow the protocol.

Googlebot: Create a rule group specifically for Google's main web crawler.

Bingbot: Create a rule group specifically for Bing's crawler.

The selected user-agent determines which crawler group receives the Allow and Disallow rules. If you need different rules for different crawlers, configure the relevant groups carefully and review the generated output.

Add as many path rules as your configuration requires. Each rule specifies whether a matching path should be allowed or disallowed for the selected crawler.

Examples of paths include:

/admin/

/checkout/

/search/

/assets/

Use paths relative to the website root, normally beginning with /. Review each path carefully: a broad rule can affect more URLs than intended.

The generator validates path entries and optional website and sitemap URLs. A valid URL should use http:// or https://. A sitemap directive should contain an absolute URL, not just a relative path.

Use a preset to create a starting configuration, then edit the rules to match your website.

Allow all crawlers

A simple configuration that does not disallow any paths:

User-agent: *

Disallow:

This leaves crawling unrestricted by robots.txt rules in this group. It does not guarantee that crawlers will visit or index every page.

Block all crawlers

A configuration that asks all compliant crawlers not to access any path:

User-agent: *

Disallow: /

Use this carefully. It can prevent search engines from crawling the entire site. It is not a way to make a website private or remove already indexed URLs.

Block common private paths with an exception

A preset can add Disallow rules for common restricted sections and an Allow rule for an explicitly permitted path.

For example:

User-agent: *

Disallow: /admin/

Disallow: /account/

Disallow: /checkout/

Allow: /account/help/

This is only a crawling preference. It does not prevent visitors or malicious bots from opening those URLs.

Here is a simple example that allows general crawling, blocks selected sections, and points crawlers to a sitemap:

User-agent: *

Disallow: /admin/

Disallow: /account/

Disallow: /checkout/

Allow: /account/help/

Sitemap: https://example.com/sitemap.xml

Before publishing, confirm that the paths exist, that the exception is intentional, and that important pages and resources are not blocked accidentally.

A robots.txt file is public crawl guidance, not a security mechanism.

It cannot:

Require a password or authenticate visitors.

Protect private files, directories, or sensitive information.

Stop visitors or malicious bots from requesting a URL.

Guarantee that a blocked URL will disappear from search results.

Guarantee that every crawler will follow its instructions.

Do not use robots.txt to hide confidential information. If you need a URL removed from search results, use the relevant search engine removal tools and appropriate indexing controls rather than relying on Disallow alone.