Mon–Fri: 10:00 – 18:00 
Request a call 
Robots.txt Tester
robots.txt rules can be tricky — overlapping patterns, Allow vs Disallow precedence, user-agent inheritance, glob matching. This tool parses your robots.txt and tells you whether a specific URL is allowed or disallowed for any user-agent. Pair it with our robots.txt generator for a full robots workflow.
Google's spec uses longest-match-wins: whichever rule has the longest path prefix wins. When two rules have the same length but different types (Allow vs Disallow), Allow wins. This tool implements that spec — Bing has slightly different behaviour but the core logic is the same.
Disallow in robots.txt prevents crawling — Googlebot will not fetch the URL. noindex (a meta tag or HTTP header on the actual page) tells Google to remove an already-indexed URL from search results. Disallow alone does NOT remove pages from the index if they were already there (Google can index URLs it cannot crawl based on external links to them).
Yes — add 'User-agent: GPTBot' followed by 'Disallow: /' to your robots.txt. Same for ClaudeBot, PerplexityBot, CCBot, Google-Extended (Gemini training). Note: blocking them prevents being cited in AI answers. Most B2B sites we work with allow AI crawlers because citation in LLM answers is more valuable than the training-data concern.
Overlapping rules are normal — for example 'Disallow: /wp-admin/' plus 'Allow: /wp-admin/admin-ajax.php'. The Allow wins for /wp-admin/admin-ajax.php (longer match), the Disallow wins for everything else under /wp-admin/. This tool shows the matching rule so you can verify the right one fires.
Get a free SEO audit
See exactly what is holding your B2B SaaS site back in organic search. We review technical SEO, content gaps, and competitor positioning — no strings attached.