Hey everyone 
If you’ve ever been told to “do more with AI” but felt uneasy about the risks that come with it, like untracked provider keys shared across codebases, concern over what a rogue prompt or runaway agent could do to your app, or maybe it’s that quiet dread that someone’s going to blow through the entire AI budget before anyone notices. Well, AI Runtime Control can help. Here’s what it does:
ARC is a control plane that sits between your applications and the model providers they call (either LLMs or your own self-hosted models). Instead of every app and developer holding provider keys and calling providers directly, everything routes through one Fastly-managed endpoint.
This might sound like just a gateway, but it’s much more than that. Here are some of the features we offer today - more to come soon.
-
Flexible accounting via virtual keys. Virtual keys let you split usage and controls across apps, teams, or developers instead of sharing one raw provider key.
-
Usage visibility and controls. Token and request usage is visible across the summary, virtual key, and provider views, with controls to manage it.
-
Detailed logging. ARC captures full request and response logs, including prompts and completions, for debugging and analysis.
-
BYOK model usage. Bring their own provider keys and route that usage through ARC.
-
Self-hosted and custom LLMs. ARC supports routing to self-hosted endpoints and custom models, not just the major hosted providers.
-
Multi-provider failover. A virtual key can point at an ordered list of targets, and when the primary errors out ARC automatically retries the next one, transparent to the caller.
-
Model retries. ARC retries a failed request against the same target, shipping alongside failover as the related resilience capability.
-
Budget limits. A monthly dollar budget can be set on a virtual key that either hard-blocks or keeps serving once spend is exhausted, your choice per key.
-
Budget alerts. Threshold alerts (80% & 100%) fire through Fastly’s Notification Service before and when the limit is hit.
-
Rate limiting per virtual key. Independent requests-per-minute and tokens-per-minute caps per key protect provider quota and help protect your customer-facing AI products and features from obvious attack vectors.
-
Passthrough SSO. ARC can connect outbound to a provider using your SSO or federated identity (IAM, Entra ID, Google identity) instead of forcing a static API key.
-
Prompt injection protection (SmartParse). This is an optional, add-on security capability providing first-line protection against prompt injection and other LLM-specific attacks.
Where to look when something happens
Everything ARC handles shows up in two views in the control panel:
-
Summary: request and token volume over time, broken down by virtual key and model, so you can see at a glance what’s actually driving usage.
-
Logging: individual request records, filterable by key, provider, model, or date range, down to a full JSON session log with the exact message sent, the response returned, and token counts.
Why this is worth a look now
-
Stop passing raw provider keys around Slack, instead issue, rotate, and revoke access per app or developer instead.
-
Get a real answer when someone asks “what’s our AI spend by team,” instead of stitching together separate provider invoices.
-
Add production-grade resilience (failover, rate limiting) to AI features without writing retry logic yourselves.
-
One integration path means new capabilities Fastly ships later can be made available without you having to re-architect anything.
Visit our documentation to learn more and get started. We also have a few cool tutorials to check out too.
As always, if you have any questions, please drop them below. We’re actively building on this and we want to hear what’s missing from your setup.
We are not currently considering doing that. Rather we feel we can provide the most value by giving you control, cost management, governance, and security. Let me know if that helps or if you have any additional questions.