AI Crawler Reference
Primenza tracks discovery and policy signals for major AI crawlers listed in the product catalog. This page documents which bots Primenza recognizes — not every bot that may access your site.
Tracked AI crawlers (product catalog)
The following user-agent tokens are configured in Primenza for robots policy analysis, monitoring, and AI visibility modules:
- GPTBot — OpenAI GPT
- ClaudeBot — Anthropic Claude
- Google-Extended — Google Gemini
- PerplexityBot — Perplexity
- Amazonbot — Amazon Alexa
- CCBot — Common Crawl
- Bytespider — ByteDance
- OAI-SearchBot — OpenAI Search
How Primenza uses crawler data
Primenza analyzes robots.txt directives, crawl policy recommendations, and (where enabled) AI visibility simulations relevant to these bots. Findings are OBSERVED from fetched policy files and crawls — not from proprietary crawler logs inside third-party AI companies.
robots.txt and AI crawlers
Teams can audit whether AI-specific user-agent rules exist, conflict with broader directives, or block resources needed for answer extraction. Primenza Robots Management can generate and deploy policy updates when approved.
llms.txt and discovery
llms.txt is a separate machine-readable index for AI retrieval. It does not replace robots.txt. Primenza audits, generates, and can deploy llms.txt alongside crawler policy files. See /llms-txt for workflow details.
Limitations
Crawler adoption and behavior change without notice. A bot listed in Primenza does not guarantee it crawls your site, indexes content, or cites your brand. AI visibility measurements are samples — see /methodology.
Primenza modules relevant to AI crawlers
- Robots Management — audit and deploy robots.txt
- AI Visibility — discovery and simulation signals (plan-dependent)
- llms.txt module — generate and deploy site indexes
- Monitoring — scheduled re-checks after policy changes
References
Frequently asked questions
Should I block AI crawlers?
That is a site policy decision. Primenza helps you audit current rules and measure visibility trade-offs — it does not mandate allow or block outcomes.
Does Primenza log live crawler visits?
Primenza measures policy files, crawls, and configured visibility modules. It does not provide server access logs for every third-party bot by default.