Mon–Fri: 10:00 – 18:00 
Request a call 
robots.txt Generator
robots.txt tells crawlers which URLs they may access on your site. This generator builds a properly formatted robots.txt with form inputs — pick the bot, pick Allow or Disallow, fill in the path. Modern AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are included in the dropdown.
robots.txt is a text file at /robots.txt that tells crawlers which URLs they may access. It does not prevent indexing of already-indexed URLs — only crawling. Most sites need a robots.txt to: (1) avoid wasting crawl budget on admin pages, (2) point crawlers to the sitemap, (3) control AI crawler access.
Depends on strategy. Blocking them prevents training data scraping but also blocks citation in ChatGPT/Claude answers (which can drive traffic via LLM SEO). Most B2B sites we work with allow AI crawlers because being cited in LLM answers is more valuable than the training-data concern.
Disallow in robots.txt prevents crawling but does not remove already-indexed URLs from Google. noindex (a meta tag or HTTP header on the actual page) tells Google to remove the URL from the index. Use Disallow to save crawl budget; use noindex to remove pages from search.
Only if the bad bot respects robots.txt — most do not. robots.txt is an honour system. To block actually-malicious bots, you need WAF rules, IP blocking at the server level, or services like Cloudflare bot management.
Get a free SEO audit
See exactly what is holding your B2B SaaS site back in organic search. We review technical SEO, content gaps, and competitor positioning — no strings attached.