robots.txt Tester
Paste a robots.txt file (or fetch it from a site), list URLs, and see which crawler can access each one, and which rule decides it. Matching follows Google's documented rules. For the background, read the robots.txt guide and the crawl budget article.
Result
| URL | Googlebot | Googlebot-Image | Google-Extended * | Bingbot | GPTBot | OAI-SearchBot | ClaudeBot | PerplexityBot |
|---|---|---|---|---|---|---|---|---|
| / | ✓ Allowed no rule | ✓ Allowed no rule | ✓ Allowed no rule | ✓ Allowed no rule | ✗ Blocked line 9 | ✓ Allowed no rule | ✓ Allowed no rule | ✓ Allowed no rule |
| /hesabim/giris | ✓ Allowed line 6 | ✓ Allowed no rule | ✓ Allowed line 6 | ✓ Allowed line 6 | ✗ Blocked line 9 | ✓ Allowed line 6 | ✓ Allowed line 6 | ✓ Allowed line 6 |
| /hesabim/siparisler | ✗ Blocked line 4 | ✓ Allowed no rule | ✗ Blocked line 4 | ✗ Blocked line 4 | ✗ Blocked line 9 | ✗ Blocked line 4 | ✗ Blocked line 4 | ✗ Blocked line 4 |
| /kategori/ayakkabi?siralama=fiyat | ✗ Blocked line 5 | ✓ Allowed no rule | ✗ Blocked line 5 | ✗ Blocked line 5 | ✗ Blocked line 9 | ✗ Blocked line 5 | ✗ Blocked line 5 | ✗ Blocked line 5 |
| /ozel-gorseller/kampanya.jpg | ✓ Allowed no rule | ✗ Blocked line 12 | ✓ Allowed no rule | ✓ Allowed no rule | ✗ Blocked line 9 | ✓ Allowed no rule | ✓ Allowed no rule | ✓ Allowed no rule |
| /robots-txt | ✓ Allowed no rule | ✓ Allowed no rule | ✓ Allowed no rule | ✓ Allowed no rule | ✗ Blocked line 9 | ✓ Allowed no rule | ✓ Allowed no rule | ✓ Allowed no rule |
- Google-Extended * Not a separate crawler but a control token: crawling is done by the main crawler; this rule governs use of content in AI models.
Sitemaps: https://www.ornek.com/sitemap.xml
How to use it
- Enter a domain and press Fetch, or paste your robots.txt content (for example a draft you have not published yet).
- List the URLs to test, one per line. Full URLs or paths both work.
- Pick the crawlers. Add any other user-agent token in the custom field.
- Read the table: each cell shows allowed or blocked and the line number of the deciding rule. Fix the warnings below the table before publishing.
Frequently Asked Questions
Does blocking a URL in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed without its content if other pages link to it. To keep a page out of the index, allow crawling and use a noindex tag; see the technical SEO guide.
Which rule wins when Allow and Disallow both match?
Google applies the most specific rule, meaning the longest matching path. If an Allow and a Disallow rule are equally long, the less restrictive one (Allow) wins. The tool shows the deciding rule for each URL.
Should I block AI crawlers like GPTBot or ClaudeBot?
It is a business decision. Blocking training crawlers keeps content out of future model training but does not necessarily remove you from AI answers that use search. The GEO implementation guide explains the trade-offs.
Is this tool identical to Google's parser?
It follows Google's documented rules (groups, the most specific user-agent, longest match, wildcards, the 500 KiB limit), but it is not Google's own code. For critical changes, also check the robots.txt report in Search Console.


