Glossary
AI Crawler
An AI crawler is a bot operated by an AI provider that fetches web pages either to train models or to answer live user queries. Each is identified by a user agent and can be allowed or blocked in robots.txt; OpenAI, for example, documents separate agents for training and for user-triggered browsing.
In plain terms
Different bots do different jobs. Blocking the wrong one can quietly remove you from live AI answers.
Why it matters for B2B
Teams often block AI bots wholesale to protect content, then wonder why they disappeared from AI answers. The training decision and the retrieval decision should be made separately and deliberately.
How to apply it
- Audit robots.txt for every AI user agent, not just Googlebot.
- Decide training access and live retrieval access as two separate policies.
- Confirm pages render server-side; a bot that cannot execute your app sees nothing.
Sources
Related reading
Related terms
Back to the full glossary (25 terms).
Ranking is no longer enough
You need to be cited, mentioned, and recommended.
Get a free AI Visibility Report — see exactly where your brand appears across ChatGPT, Google AI, Gemini, Perplexity, and Copilot, and where competitors are winning instead.