Simple. Fast. Useful.

Choose your tool

DEVELOPER TOOL

Robots.txt Rule Generator

Build one user-agent group with explicit disallow paths and optional sitemap.

â—‡ Browser only
0 lines · 0 characters
Ready when you are
Your results will appear here.
Copy or download your result below.

Your text stays on this device.

Favorites
Recently used
LESS BUSYWORK, MORE DONE

Three steps. That’s it.

A useful toolkit for text, data, images and calculations.

  1. 01

    Add your input

    Choose a tool, enter values or select a file.

  2. 02

    Make it yours

    Adjust the options and run the action.

  3. 03

    Take the result

    Copy the result or download its file.

Robots.txt Rule Maker online

Build one user-agent group with explicit disallow paths and optional sitemap. Use this when inspecting structured input or preparing a data or code artifact. Generated commands and configurations remain text until you review and use them yourself.

How to use it

Paste the multi-line input in the format shown below. Set user agent, sitemap url. Choose Run tool. Review the output, then copy or download it.

Options and example settings

OptionChoices / valueExample default
User agentEnter user agent.*
Sitemap URLEnter sitemap url.https://example.com/sitemap.xml

Worked examples

Use this when inspecting structured input or preparing a data or code artifact. Generated commands and configurations remain text until you review and use them yourself.

Reproducible worked example

Settings: User agent: *; Sitemap URL: https://example.com/sitemap.xml

Input:

/admin/
/private/

With the input and settings above, the tool produces:

Output:

User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml

The example output has 4 lines. Compare its order, separators and whitespace with the input before copying the result.

Limitations and common mistakes

  • Robots rules control cooperative crawling, not access permissions or reliable removal from search results.
  • Combined input is bounded to 150,000 characters; output text is bounded to 1.2 million characters. Some algorithms impose smaller limits, stated in their error messages.

Frequently asked questions

What format should I enter?

Paste the multi-line input in the format shown below. The worked example shows the supported structure and actual output.

How can I create one user-agent group with explicit disallow paths and optional sitemap online?

Paste the multi-line input in the format shown below. Set user agent, sitemap url. Choose Run tool. Review the output, then copy or download it.

Where is my input processed?

The bundled tool processes input in your browser. It does not upload tool input to a conversion service. Other site plugins, analytics, advertising and browser extensions may have separate behavior.

Useful next tools

Explore other tools

Open a tool below, use the search, or choose a category to browse its full collection.