Robots.txt Builder
Build a robots.txt file with allow/disallow rules, user-agent groups, and a sitemap URL — the same page works as a robot.txt generator.
- Allow / Disallow
- AI crawlers
- Sitemap + llms.txt
A robots.txt builder writes the crawl rules you place at the site root. Crawlers request /robots.txt first. This page is also a robot.txt generator for the common misspelling: one URL, not a second thin page.
Add a user-agent group, then Allow or Disallow paths. Use * for every crawler, or name GPTBot, ClaudeBot, Google-Extended, and other AI bots. Presets fill a public allow-all file, a full block, or a block-AI set. The sitemap field becomes a Sitemap line.
Optional llms.txt extras add a comment pointing at an llms.txt URL and preview a short companion file. Everything runs in your browser. Browse more SEO tools on the SEO hub, or open the UTM builder. For pay math, use the paycheck calculator hourly. For cost basis, try the stock average calculator. Need local identifiers? Use the UUID generator.
How to use the robots.txt builder
- Start from a preset, or keep the default public group.
- Set each user-agent. Add allow or disallow paths as needed.
- Optionally block GPTBot, ClaudeBot, Google-Extended, and other AI crawlers.
- Enter your sitemap URL. Turn on llms.txt extras if you want a companion comment.
- Copy or download the generated robots.txt. Place it at the site root.
Frequently asked questions
What is a robots.txt builder?
A robots.txt builder — also used as a robot.txt generator — writes the Allow, Disallow, User-agent, and Sitemap lines for a robots.txt file so you can preview, copy, or download it without editing the file by hand.
What is the difference between a robots.txt builder and a robot.txt generator?
They are the same job. Searchers often type robot.txt generator with a singular robot, but the file crawlers request is robots.txt. This page covers both names.
Which AI crawlers can I block?
Presets and chips cover GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Claude-SearchBot, Google-Extended, CCBot, Bytespider, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, and Amazonbot. You can also type any other user-agent.
Does a sitemap line belong in robots.txt?
Yes. Add one Sitemap URL per line. Crawlers that honor the directive can discover your XML sitemap from the robots.txt file at the site root.
What are the optional llms.txt extras?
llms.txt is a companion file some sites publish for AI assistants. This robots.txt builder can add a comment pointing at your llms.txt URL and preview a short llms.txt draft. It does not upload either file.
Does this robots.txt builder upload my rules?
No. User-agent groups, allow/disallow paths, sitemap URLs, and llms.txt extras are generated in your browser. Nothing is sent to a server or stored on our side.
Other tools
Finance
Stock Average Calculator
Weighted average cost, total shares, break-even price, and unrealized P/L from multiple lots.
Finance
Paycheck Calculator Hourly
Gross pay per paycheck from an hourly rate or salary — weekly, biweekly, semimonthly, or monthly.
SEO
UTM Builder
UTM generator, maker, and link builder for campaign URLs — including bulk CSV — all in the browser.
Developer
UUID Generator
Generate UUID v4 and UUID v7 (also used as an online GUID generator) with bulk copy.
Convert
HEIC to PNG Converter
Convert HEIC and HEIF photos to PNG in the browser. Batch convert, then download — files never leave the device.