Building an AI Gateway on Fastly Compute

General
1

Check out this blog from @JonathanSpeek on his process for building an AI routing gateway with Mercury2 on Fastly compute (there’s even a video demo :video_camera: ).

TL;DR – Based on the nature of the end user’s query, the gateway routes to the most efficient and cost-effective model for the job. At scale, these token utilization efficiencies add up to major cost savings!

As always, we’d love to hear more about what you’re building on Fastly, especially with AI. Let us know in the comments <3

2

Are there plans to offer managed models in the future? This would enable a one-stop service with Fastly. I would only need to pay Fastly for AI model usage fees to access multiple top-tier models.

1 Like
3

Hey @Dandelionss I’ll pass this on to the team. It’s an interesting idea, but I can’t comment on whether this is something we’ll pursue or not.