AI Search · Updated June 2026
AI crawlers are bots that fetch web content for AI systems — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), and CCBot (Common Crawl). Each respects robots.txt, so you choose per-bot whether to allow access.
Allowing AI crawlers makes your content eligible for AI answers and citations; blocking them protects content but forfeits that discovery channel. Most merchants benefit from allowing search-oriented bots at minimum.
robots.txt is a file at a site’s root that tells crawlers which areas they may access. It now also governs AI crawlers — GPTBot, ClaudeBot, PerplexityBot — making it the control point for whether your content can appear in AI answers.
llms.txt is a proposed standard file at a website’s root that gives AI systems a curated, markdown-formatted guide to the site’s most important content. Like robots.txt for permissions or sitemap.xml for URLs, llms.txt is a map written specifically for LLMs.
Generative engine optimization (GEO) is optimizing content to be retrieved, used, and cited by AI answer engines — ChatGPT, Claude, Perplexity, Google AI Overviews. Tactics include quotable self-contained answers, explicit statistics, schema markup, llms.txt, and allowing AI crawlers.