robots.txt and sitemap
robots.txt and XML sitemap generator
Build robots.txt with GPTBot rules and a simple XML sitemap.
Result
robots.txt
User-agent: * Disallow: Allow: / User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: Google-Extended Allow: / User-agent: ClaudeBot Allow: / User-agent: Anthropic-AI Allow: / User-agent: PerplexityBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: Amazonbot Allow: / User-agent: meta-externalagent Allow: /
sitemap.xml
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> </urlset>
Sponsored
Hosting and WordPress
We may earn a commission if you buy through these links, at no extra cost to you. That is one way to keep the tools free.
This robots.txt generator builds a file with User-agent: *, Allow/Disallow, directives for AI crawlers (GPTBot, Google-Extended, ClaudeBot, and others), and a Sitemap: line. It runs in the browser and never crawls the site.
The same screen builds a simple XML sitemap from a list of URLs. It does not discover pages and it does not check whether those URLs return 200.
Allowing or blocking GPTBot and peers is your policy. Training, search, and user-fetch bots are different user-agents — read the label before you hit Disallow.
robots.txt is a request, not a lock. Sensitive URLs need real authentication, not only Disallow.
None of the listed URLs are visited. The generator only formats robots.txt and XML from what you paste.
Quick habits
- Point Sitemap: at the canonical XML (a file or a sitemap index).
- Do not block CSS and JS the Google crawler needs to render the page.
- Split GPTBot (training) from search/fetch user-agents if the policy differs.
- After you publish, hit https://yoursite.com/robots.txt and check Search Console.
Frequently asked questions
Short answers about robots.txt, GPTBot, the Sitemap: line, and the XML this page emits.
- What does this generator include besides User-agent: *?
- Blocks for AI crawlers — GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, ClaudeBot, Anthropic-AI, PerplexityBot, Applebot-Extended, Bytespider, CCBot, Amazonbot, and meta-externalagent — plus a Sitemap: line if you fill in the URL.
- Does the tool crawl the site to build the sitemap?
- No. You paste the URLs. The XML is a minimal urlset with loc and today’s lastmod. No crawler and no lastmod sniffed from the page.
- Does blocking GPTBot hide the content from ChatGPT?
- Not necessarily. GPTBot is usually the training bot; ChatGPT-User and OAI-SearchBot are different agents. Set each line to match the policy instead of assuming one switch.
- Does robots.txt stop Google from indexing a URL?
- It is not a guarantee. Disallow asks crawlers not to fetch. URLs can still be indexed if they are linked. Use noindex or authentication when the URL must stay out of search.
- Can I output several Sitemap: lines?
- This field emits one line. If you have a sitemap index, point at that. Multiple Sitemap: lines are also valid — edit the copied text if you need them.
- Are the paths I paste sent to a server?
- No. The text stays in the browser until you reload. Copy robots.txt and the XML to your host.