# robots.txt Guide: Correct Configuration and Common Mistakes

Author: Fatih Emin Çakıroğlu (https://fatihemincakiroglu.com)
Source: https://fatihemincakiroglu.com/en/blog/robots-txt
Published: 2026-10-20
Updated: 2026-10-20

robots.txt is a text file of a few lines sitting at the root of your site, yet one wrong line can stop the whole site from being crawled. In audits I still find "Disallow: /" lines carried over from a staging environment to production.

This article covers robots.txt syntax according to Google's interpretation, how it differs from noindex, the official user-agent names of AI crawlers, how it differs from llms.txt, and the most common mistakes.

> Short answer: robots.txt tells bots which parts of a site they may crawl; on its own it does not stop a page from being indexed. To keep a page out of the index you use noindex, and the page must be crawlable for that to work. AI crawlers can be managed one by one using each company's official user-agent name.

## What is robots.txt and where should it live?

According to Google's specification, the file must sit at the root of each host, protocol and port: example.com/robots.txt. Subdomains such as blog.example.com need their own file, and a robots.txt placed in a subfolder is ignored.

The file must be UTF-8 plain text, and Google only reads the first 500 KiB. Google generally caches the file for up to 24 hours, so a change may not take effect immediately.

What happens when the file cannot be fetched also matters. A 4xx response (except 429) is treated as if no robots.txt exists, so no restrictions apply. On a 5xx response Google stops crawling for the first 12 hours, then uses the last successful version for up to 30 days.

## robots.txt syntax: the core rules

Google supports four fields: user-agent, allow, disallow and sitemap. Unsupported fields such as crawl-delay are ignored by Google. Paths are case-sensitive; * matches zero or more characters and $ marks the end of the URL.

```
User-agent: *
Disallow: /cart/
Disallow: /account/
Disallow: /*?sort=
Allow: /*?sort=*&page=

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml
```

When rules conflict, the longest, most specific match wins; on a tie the less restrictive rule applies. If there is a group for a bot by name, that bot follows only that group and ignores the * group. So when you open a group for a specific bot, you need to repeat the general rules there too.

Sitemap lines must be full URLs and are not tied to any user-agent group. I cover which URLs are worth blocking for crawl efficiency, and which are not, in the [crawl budget](https://fatihemincakiroglu.com/en/blog/crawl-butcesi) article.

## What is the difference between robots.txt and noindex?

This is the most common mix-up. robots.txt controls crawling, noindex controls indexing. Google can index a URL blocked by robots.txt without seeing its content if other pages link to it.

**robots.txt Disallow**
- Blocks crawling
- The page can still appear in the index without content
- Good for reducing crawl waste
- Works site-wide, by pattern

**noindex (meta or X-Robots-Tag)**
- Blocks indexing
- The page must be crawlable
- The right tool to remove pages from the index
- Works per page or file

The classic mistake is using both: if you add noindex and also block the page in robots.txt, Googlebot never sees the noindex. Wait for the page to drop out with noindex first, then block it if needed. For the broader crawling versus indexing distinction, see the [technical SEO guide](https://fatihemincakiroglu.com/en/guides/teknik-seo).

## Managing AI crawlers with robots.txt

AI companies often use separate bots for model training, search and fetching pages on a user's request. The table below summarises the names in each company's official documentation as of October 2026; these lists change, so check the sources again before you implement.

| User-agent | Company | Purpose per official docs |
| --- | --- | --- |
| Google-Extended | Google | Training Gemini models and grounding in Gemini Apps and Vertex AI; a control token, not a separate bot |
| GPTBot | OpenAI | Training generative AI foundation models |
| OAI-SearchBot | OpenAI | Surfacing websites in ChatGPT search features |
| ChatGPT-User | OpenAI | User-initiated actions; robots.txt rules may not apply |
| ClaudeBot | Anthropic | Collecting content that could contribute to training |
| Claude-SearchBot / Claude-User | Anthropic | Search result quality / fetching pages for a user's question |
| PerplexityBot | Perplexity | Surfacing sites in Perplexity search results; not for training |

> **Google-Extended does not switch off AI Overviews** Google's documentation says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search, so Googlebot rules and controls such as nosnippet, max-snippet and noindex apply to them.

My advice is to treat training bots and search bots separately. Blocking training while allowing search bots keeps your chance of being cited by AI assistants. You will find the brand visibility side of this decision in [brand visibility in ChatGPT](https://fatihemincakiroglu.com/en/blog/chatgpt-marka-gorunurlugu) and the implementation steps in the [GEO implementation guide](https://fatihemincakiroglu.com/en/guides/geo-uygulama). If the terms are new to you, the [AI glossary](https://fatihemincakiroglu.com/en/ai-glossary) helps.

## How robots.txt differs from llms.txt

llms.txt is a proposed file format for summarising a site's key content for language models; it neither grants nor blocks access. robots.txt is a crawling protocol that bots follow.

Google states plainly that you do not need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode. I cover what llms.txt does and when it is worth trying in the [llms.txt guide](https://fatihemincakiroglu.com/en/blog/llms-txt-rehberi).

## The robots.txt mistakes I see most often

Use the list below as a checklist before a site launch or migration:

- [ ] Has the staging "Disallow: /" made it to production?
- [ ] Are CSS, JavaScript or API endpoints blocked? Rendering will be incomplete; see [JavaScript SEO](https://fatihemincakiroglu.com/en/blog/javascript-seo).
- [ ] Are pages you want removed marked with both noindex and Disallow?
- [ ] Was a bot-specific group opened without copying the general rules into it?
- [ ] Does a path fail to match because of case (/Cart/ versus /cart/)?
- [ ] Is the file relying on noindex or crawl-delay lines? Google supports neither.
- [ ] Does each subdomain have its own robots.txt?

After a change, use the robots.txt report in Search Console to see which version Google fetched and when. If you want a strategy for AI crawlers, I help through [GEO consulting](https://fatihemincakiroglu.com/en/geo-consulting), and with technical audits through [SEO consulting](https://fatihemincakiroglu.com/en/seo-consulting); you can reach me via the [contact page](https://fatihemincakiroglu.com/en/contact).

## Frequently Asked Questions

### What happens if there is no robots.txt file?

If the file is missing or returns 404, Google crawls the site without restrictions. That is usually fine for small sites, but keeping a simple file to list your sitemap and block unnecessary areas is a good habit.

### Is robots.txt enough to remove a page from Google quickly?

No. A blocked page can stay in the index. Use the Removals tool in Search Console for fast, temporary hiding, and noindex, password protection or a 404/410 response for a permanent fix.

### Will blocking AI crawlers reduce my traffic?

It depends on the bot. Blocking training bots does not directly affect search visibility; blocking search and user-request bots can reduce your chance of being cited in that assistant. Decide bot by bot.

### Do all bots obey robots.txt?

robots.txt is not a security tool but a protocol that well-behaved bots follow. Malicious crawlers can ignore it; protect sensitive content with passwords or server-side access control.

### How do I remove images from Google Images?

You can disallow the relevant image paths for the Googlebot-Image user agent. According to Google's documentation, this token also affects some Search features that show images, not only Google Images.

### How do I edit robots.txt on Shopify or WordPress?

Platforms usually generate a default file. Shopify uses the robots.txt.liquid template, and on WordPress you edit through an SEO plugin or a physical file on the server; check the platform's official docs for the current method.

---
Please attribute as: "Fatih Emin Çakıroğlu (fatihemincakiroglu.com)". Original: https://fatihemincakiroglu.com/en/blog/robots-txt