AI Crawlers, GPTBot and llms.txt: What They Mean for Your Site

The two jobs an AI crawler can be doing

The first job is training: a bot downloads lots of pages to teach a model. The second is retrieval: a bot fetches a specific page because a user asked something, or to power an AI search result. The distinction matters because many vendors split these into separate user-agents so you can opt out of one without opting out of the other - for example, letting a page appear in an AI answer while refusing to be used for model training.

Equally important is that Google search indexing and Google's AI training are different bots: Googlebot indexes for Search, while Google-Extended governs whether content feeds Gemini model training. Blocking one does not block the other, a nuance a lot of sites get wrong.

How each bot maps to a robots.txt user-agent

Most major AI crawlers advertise a named user-agent you can target in robots.txt. The common ones and what they are for:

Because each vendor exposes its own name, you can be selective: allow Perplexity-User so your pages still answer questions, while disallowing PerplexityBot if you do not want the crawl used for their index, or block every training agent while keeping search visibility. The rules are ordinary User-agent plus Disallow groups - nothing exotic.

Allowing or blocking - and what a block really means

To welcome all AI use, you simply leave everything allowed (an allow-all robots.txt, optionally with Sitemap: lines pointing at your maps from the sitemap generator). To opt out of a bot, add a group naming it and Disallow: /.

Be honest about the limits: robots.txt is an advisory signal, not an enforcement wall. Reputable AI crawlers publish a user-agent and say they respect robots.txt, so the directives work for them - but a file cannot stop a determined scraper, and some networks have been reported ignoring it. Treat robots.txt as the correct, standard way to express your preference to compliant bots, not as copy-protection.

What llms.txt actually is (and is not)

llms.txt is a proposed Markdown file at your site root that summarises what the site is and links to its most important pages - a hand-held map for a language model, in the spirit of robots.txt and sitemap.xml but written for reading rather than crawling. The appeal is that a compact, curated text file is easier for a model to ingest accurately than raw HTML.

Set expectations honestly: llms.txt is a community proposal, not an approved standard. No search engine or model vendor has committed to reading it, and it is not a ranking factor. Think of a clean, truthful llms.txt as a low-cost bet that improves how AI systems describe your site if and when they consume it - never fabricate capabilities or figures in it just to look authoritative.

The traffic trade-off to decide on deliberately

Being cited inside an AI answer can raise brand visibility, but assistants and AI overviews often summarise the point and send back few clicks. Publishers therefore weigh reach (being the source an AI quotes) against referral loss (the answer satisfies the visit without a click-through). There is no universal right answer: a reference or tool site may welcome the exposure, while a publisher living on session traffic may restrict training crawlers and still allow answer-time retrieval.

The practical workflow is: decide per vendor and per job (training versus retrieval), express it as named user-agent groups in robots.txt, keep an accurate llms.txt if you want to court citations, then verify with the link checker and log review that the rules you wrote are the rules you meant.