SEO · Updated June 2026
robots.txt is a file at a site’s root that tells crawlers which areas they may access. It now also governs AI crawlers — GPTBot, ClaudeBot, PerplexityBot — making it the control point for whether your content can appear in AI answers.
Blocking AI crawlers protects content from training but also removes you from AI-driven discovery. For merchants, visibility usually wins: being cited in ChatGPT or Perplexity answers is free distribution.
AI crawlers are bots that fetch web content for AI systems — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), and CCBot (Common Crawl). Each respects robots.txt, so you choose per-bot whether to allow access.
llms.txt is a proposed standard file at a website’s root that gives AI systems a curated, markdown-formatted guide to the site’s most important content. Like robots.txt for permissions or sitemap.xml for URLs, llms.txt is a map written specifically for LLMs.
An XML sitemap is a machine-readable list of a site’s URLs that helps search engines discover and prioritize crawling. Shopify auto-generates /sitemap.xml; submit it in Google Search Console to speed up indexing of new pages.