Hello Fastly Community!
We’re excited to announce the launch of Fastly AI Bot Management!
Fastly AI Bot Management gives you the power to manage and control the behavior of AI bots that crawl and scrape website content. Detect which AI bots are accessing your content, and take action to block, intercept, or allow particular AI bots based on your own unique policies and desired responses.
With AI Bot Management, you’ll be able to:
- Understand what AI bots are crawling your content
- Control what AI bots can crawl or scrape your content
- Block any bots that take things without your consent
- Stop AI bots from costing you money
AI bots fall into two main categories:
AI Crawlers
Used primarily for training AI/LLMs and building large datasets or indexes.
AI Fetchers
Used by AI/LLMs in real-time to enrich responses to specific user queries.
To identify and address the behaviors from these bots, we’ve developed four specific signals for both verified and suspected AI bots:
Verified Bot - AI Crawler
Verified Bot - AI Fetcher
Suspected Bot - AI Crawler
Suspected Bot - AI Fetcher
Customers can use these signals in their own configurations and create their own custom Site Lists by incorporating these new signals. For participants in our Beta program, these signals have been added to the beta feature System Lists under the list name AI Bot Signals.
Check out our newest blog post for more information, or contact your account team to learn more.