Glossary
AI Crawler
An AI crawler is a bot operated by an AI provider that fetches web pages either to train models or to answer live user queries. Each is identified by a user agent and can be allowed or blocked in robots.txt; OpenAI, for example, documents separate agents for training and for user-triggered browsing.
In plain terms
Different bots do different jobs. Blocking the wrong one can quietly remove you from live AI answers.
Why it matters
Teams often block AI bots wholesale to protect content, then wonder why they disappeared from AI answers. The training decision and the retrieval decision should be made separately and deliberately.
How to apply it
- Audit robots.txt for every AI user agent, not just Googlebot.
- Decide training access and live retrieval access as two separate policies.
- Confirm pages render server-side; a bot that cannot execute your app sees nothing.
Example
A B2B company discovers its robots.txt blocks GPTBot entirely, which prevents OpenAI from training on its content but also, it later learns, does not affect ChatGPT's separate live-browsing agent — prompting the team to set distinct rules for training crawlers versus real-time retrieval crawlers.
Sources
Related reading
Related terms
Back to the full glossary (75 terms).
Ranking is no longer enough
You need to be cited, mentioned, and recommended.
Being cited, mentioned, and recommended are three different outcomes, and most brands only ever achieve the first one. Ranking is no longer enough because AI engines answer buyers directly and name only a short list of vendors as the recommendation — everyone else is cited in passing, if at all. Get a free AI Visibility Report to see exactly where your brand appears today across ChatGPT, Google AI Overviews, Gemini, Perplexity, and Copilot, where competitors are winning the recommendation instead, and what's keeping you from moving up the shortlist.