Hi,
We are hosting a large file download around 150MB. An end-user reports they run CI jobs on Azure Pipelines which apparently kicks off 11 simultaneous jobs, each one downloading the file.
4 out of the 11 jobs experience 404 errors.
Could Fastly’s servers perceive this as a bot attack coming from Azure and prevent some of the downloads? That might even be expected, and not a bad idea.
I am not seeing 404 errors, or any errors, on the origin servers. The file is almost certainly cached in the CDN. Which means the origins are not sending 404.
BTW - the logs also say:
WARNING: Error: HTTP 404 NOT FOUND for url (...)
Elapsed: 00:00.125658
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
I have advised the user to implement retries (which are suggested there).
If you can work with your end user to include the Fastly-Debug header in their download requests, that will provide more details in the response when a 404 occurs.
Is the Fastly service serving this file in Delivery or Compute? If it’s in Delivery, do have ‘segmented caching’ enabled?
- Fastly-Debug. I have sent the suggestion to the user. Not sure if they will be able to include it.
- Delivery or Compute: I believe we aren’t using Compute currently so it would be Delivery.
- Yes, segmented caching is enabled on the download. It’s a large file, that was needed.
In general the download is working great.
The console says 2% error rate, and that would include mistaken URLs. It is a curiosity what’s happening with Azure. Possibly the issue is entirely on their side, somehow.
1 Like
Here is a hypothesis about what is occurring.
The general TTL is quite long. I configured a shorter TTL for 404’s, based on this article: A Cache Recipe for Success | Fastly
And yet apparently that is not taking effect. Examining the raw VCL, the Cache Setting for the 404 rule (Priority 10) is showing up after VCL Snippets.
There is a VCL Snippet (priority 100) that is setting a TTL. The rules are competing. Both the Cache Setting and the VCL Snippet run
return(deliver);
That will return and short-circuit other rules.
A confusion is that a “Cache Setting” at priority 10 appears in the VCL after a “VCL Snippet” with priority 100.
The “VCL Snippet” with priority 100 happens first.
In this case, the VCL snippet is setting the TTL to a very long number and then return, and it never arrives at the “Cache Setting”.
Wouldn’t a “priority 10” (cache setting) appear before a “priority 100” (vcl snippet)?
1 Like
I’ve chatted with one of our VCL experts, and it appears you’ve discovered some unexpected behavior (violation of the Principle of Least Surprise, at a minimum!). Please create a support ticket by contacting support@fastly.com and our support team will help you reconfigure the service to get the behavior you want.
1 Like
Thank you!
I have manually taken the code from “Cache Settings” and placed it at the top of the existing “VCL Snippet”. That way it happens first, as if it had the higher priority. Then removed the “Cache Setting”. So everything is contained in the snippet only. I believe this ought to solve it.
At least superficially it does seem to be surprising. Both sections add code in vcl_fetch. Both sections have a priority. I imagined the purpose of priority: 10 was to cause higher priority. But it appears to treat “Cache Settings” like an entirely different section of the VCL, which is at a lower priority.
I wonder if the “Recipe for Success” (which is great in that it addresses an important problem about 404’s) should include text about this. Or maybe the topic is already in documentation. This is possible - there are many docs.
You’ve implemented the suggestion that my colleague had 
If you don’t mind, please do open a support ticket so that this unexpected behavior can be tracked in our internal systems.