AI Search · Updated June 2026
llms.txt is a proposed standard file at a website’s root that gives AI systems a curated, markdown-formatted guide to the site’s most important content. Like robots.txt for permissions or sitemap.xml for URLs, llms.txt is a map written specifically for LLMs.
The file typically lists key pages with one-line descriptions, organized by section. Early adopters use it to steer AI assistants toward canonical resources — improving citation accuracy when ChatGPT, Claude, or Perplexity reference the site.
It is optional but cheap insurance: a well-structured llms.txt helps AI assistants find and accurately cite your key collections, policies, and guides. Generate one free with Chakril’s llms.txt generator and upload it to your store.
AI crawlers are bots that fetch web content for AI systems — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), and CCBot (Common Crawl). Each respects robots.txt, so you choose per-bot whether to allow access.
Generative engine optimization (GEO) is optimizing content to be retrieved, used, and cited by AI answer engines — ChatGPT, Claude, Perplexity, Google AI Overviews. Tactics include quotable self-contained answers, explicit statistics, schema markup, llms.txt, and allowing AI crawlers.
robots.txt is a file at a site’s root that tells crawlers which areas they may access. It now also governs AI crawlers — GPTBot, ClaudeBot, PerplexityBot — making it the control point for whether your content can appear in AI answers.