FREE TOOL

robots.txt Tester

Paste a robots.txt file (or fetch it from a site), list URLs, and see which crawler can access each one, and which rule decides it. Matching follows Google's documented rules. For the background, read the robots.txt guide and the crawl budget article.

Crawlers

Result

URLGooglebotGooglebot-ImageGoogle-Extended *BingbotGPTBotOAI-SearchBotClaudeBotPerplexityBot
/✓ Allowed
no rule
✓ Allowed
no rule
✓ Allowed
no rule
✓ Allowed
no rule
✗ Blocked
line 9
✓ Allowed
no rule
✓ Allowed
no rule
✓ Allowed
no rule
/hesabim/giris✓ Allowed
line 6
✓ Allowed
no rule
✓ Allowed
line 6
✓ Allowed
line 6
✗ Blocked
line 9
✓ Allowed
line 6
✓ Allowed
line 6
✓ Allowed
line 6
/hesabim/siparisler✗ Blocked
line 4
✓ Allowed
no rule
✗ Blocked
line 4
✗ Blocked
line 4
✗ Blocked
line 9
✗ Blocked
line 4
✗ Blocked
line 4
✗ Blocked
line 4
/kategori/ayakkabi?siralama=fiyat✗ Blocked
line 5
✓ Allowed
no rule
✗ Blocked
line 5
✗ Blocked
line 5
✗ Blocked
line 9
✗ Blocked
line 5
✗ Blocked
line 5
✗ Blocked
line 5
/ozel-gorseller/kampanya.jpg✓ Allowed
no rule
✗ Blocked
line 12
✓ Allowed
no rule
✓ Allowed
no rule
✗ Blocked
line 9
✓ Allowed
no rule
✓ Allowed
no rule
✓ Allowed
no rule
/robots-txt✓ Allowed
no rule
✓ Allowed
no rule
✓ Allowed
no rule
✓ Allowed
no rule
✗ Blocked
line 9
✓ Allowed
no rule
✓ Allowed
no rule
✓ Allowed
no rule
  • Google-Extended * Not a separate crawler but a control token: crawling is done by the main crawler; this rule governs use of content in AI models.

Sitemaps: https://www.ornek.com/sitemap.xml

How to use it

  1. Enter a domain and press Fetch, or paste your robots.txt content (for example a draft you have not published yet).
  2. List the URLs to test, one per line. Full URLs or paths both work.
  3. Pick the crawlers. Add any other user-agent token in the custom field.
  4. Read the table: each cell shows allowed or blocked and the line number of the deciding rule. Fix the warnings below the table before publishing.

Frequently Asked Questions

Does blocking a URL in robots.txt remove it from Google?

No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed without its content if other pages link to it. To keep a page out of the index, allow crawling and use a noindex tag; see the technical SEO guide.

Which rule wins when Allow and Disallow both match?

Google applies the most specific rule, meaning the longest matching path. If an Allow and a Disallow rule are equally long, the less restrictive one (Allow) wins. The tool shows the deciding rule for each URL.

Should I block AI crawlers like GPTBot or ClaudeBot?

It is a business decision. Blocking training crawlers keeps content out of future model training but does not necessarily remove you from AI answers that use search. The GEO implementation guide explains the trade-offs.

Is this tool identical to Google's parser?

It follows Google's documented rules (groups, the most specific user-agent, longest match, wildcards, the 500 KiB limit), but it is not Google's own code. For critical changes, also check the robots.txt report in Search Console.

Related guides

Technical SEO Guide
Technical SEO Guide
Crawling, indexing, site architecture, Core Web Vitals, JavaScript and hreflang: run a technical SEO audit step by step.
Read the guide →
Site Migration SEO Guide
Site Migration SEO Guide
A step-by-step plan for keeping organic traffic when your domain, platform or URL structure changes.
Read the guide →
GEO Implementation Guide
GEO Implementation Guide
A five-step GEO plan, from question set to measurement, for getting your brand cited in AI answers.
Read the guide →