You can now block AI crawlers and scrapers like GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Bytespider (ByteDance), and others with a single toggle using the AI bot managed ruleset. Now available for free on all plans.
The ruleset is managed by Vercel and updates automatically as new crawlers appear with no additional action required. This protection operates with zero latency impact to legitimate traffic.
For more complete coverage, combine with Bot Filter to catch AI bots that attempt to spoof user agents to disguise themselves as legitimate browsers or omit proper identification headers.
AI crawlers now generate more traffic than human users on many popular sites, driving up infrastructure costs and raising copyright and data usage concerns. Many of these crawlers do not respect robots.txt or similar directives, making manual solutions unreliable.
Enable the ruleset or learn more in the documentation.
Read more
Continue reading...
The ruleset is managed by Vercel and updates automatically as new crawlers appear with no additional action required. This protection operates with zero latency impact to legitimate traffic.
For more complete coverage, combine with Bot Filter to catch AI bots that attempt to spoof user agents to disguise themselves as legitimate browsers or omit proper identification headers.
AI crawlers now generate more traffic than human users on many popular sites, driving up infrastructure costs and raising copyright and data usage concerns. Many of these crawlers do not respect robots.txt or similar directives, making manual solutions unreliable.
Enable the ruleset or learn more in the documentation.
Read more
Continue reading...