Should an online store block AI crawlers?

The question hides two decisions. The first is whether AI assistants may read your pages when a shopper asks them something. The second is whether AI companies may use your content to train future models. Blocking indiscriminately answers both with no, and the first no has a commercial cost: a store that cannot be read cannot be recommended.

Each major assistant now publishes distinct user agents for these purposes. OpenAI uses GPTBot for training, OAI-SearchBot to build the index behind ChatGPT search, and ChatGPT-User when a person's request needs a page fetched. Anthropic uses ClaudeBot for training, Claude-SearchBot for its search index and Claude-User for user-initiated fetches. Perplexity uses PerplexityBot for its index and Perplexity-User for live requests. Google uses Googlebot for Search, including AI Overviews and AI Mode, and the Google-Extended token only to control Gemini training and grounding.

Once the roles are separated, the decision for a store is usually easy: allow the search and user-fetch agents on product, collection and policy pages, and decide about the training agents on brand policy rather than on traffic.

AI shopping answers are built from retrieved pages and submitted feeds. If the retriever is blocked, your products can still appear through a feed you submitted, but the assistant cannot verify the page, cannot read the specification table or the return policy, and cannot fetch the page when a shopper clicks through inside the assistant's own browsing. In practice a blocked store shows up less often and with thinner information.

There is a second loss that is easy to miss. Assistants read third-party pages about your products too, and when a shopper asks for the source, the assistant tries to open your page. A fetch that returns a block page or a challenge screen ends the interaction with a competitor's page that did load.

None of this argues for allowing everything. It argues for allowing the agents that act on a shopper's behalf, then measuring: server logs will show which user agents request which product pages, and referral traffic from chatgpt.com, perplexity.ai and similar domains shows what those reads turned into.

Whether to allow GPTBot, ClaudeBot, CCBot, Applebot-Extended, Meta-ExternalAgent or Bytespider is mostly about your view on model training and your images and copy being used for it. Blocking them does not remove you from ChatGPT search, Perplexity or Google's AI features, because those run on other agents. Allowing them does not earn you placement.

There is one nuance worth knowing. Some vendors describe their training crawler as also feeding general model knowledge that shapes how an assistant talks about a category. That is diffuse and slow, and it is not how a specific product ends up in a specific answer. The retrieval agents are.

If your brand position is that content should not train models, block the training agents and say so in your terms. If you have no strong view, the lower-effort option is to allow them and revisit if the crawl load becomes a problem, which for most stores it does not, since these crawlers fetch far fewer pages than Googlebot.

Robots.txt is a convention that well-run crawlers honour and that nothing forces. The agents named above publish their user agent strings and their IP ranges, and the major vendors document that they respect robots rules. If you need a rule enforced regardless, the place to do it is at the edge: a CDN or web application firewall rule keyed on the verified user agent and IP range, not on the string alone, since strings are trivially spoofed.

Several CDNs now offer managed controls for AI crawlers, including one-click blocking and per-crawler allowances. These are convenient, and they are blunt. Check that a managed block is not also stopping the user-fetch agents you want, and check that a bot challenge page is not being served to them, which reads to the assistant as a page with no content.

Whatever you use, log the outcome. A monthly look at requests by user agent, status code and path is enough to see whether your rules are doing what you intended.

Allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User on products, collections, policies, about and size or fit guides. Decide GPTBot, ClaudeBot, CCBot and Google-Extended by brand policy, and write the rule explicitly either way so it is a decision rather than an accident.

Disallow every agent from cart, checkout, account, wishlist, internal search results, sorted and filtered collection URLs, and any endpoint that changes state. That protects both your server and your data, and it is good practice regardless of AI.

Keep the file short and test it. Google's robots testing in Search Console covers Googlebot; for the others, fetch a product page with the user agent string set and confirm you get the page rather than a block. Then revisit the policy each quarter, because the list of agents and what each one does keeps changing.