we are running an Ibexa DXP project with Fastly as the HTTP caching proxy and are seeing some unexpected cache behavior.
On the Symfony/Ibexa side, we set a cache TTL of 86400 seconds (24 hours). We can confirm that Fastly receives this value correctly, as beresp.ttl is also set to approximately 86400 seconds.
However, some cached pages seem to disappear from Fastly much earlier, sometimes after only 30 minutes, even though their TTL has not expired.
I understand that Fastly may evict objects from its cache before their TTL expires, for example due to resource constraints and its cache eviction strategy. This would make sense for rarely accessed pages, such as old press articles that might only receive one or two requests per day.
What surprises us is that this also happens to more frequently accessed pages, such as our industry/landing pages. We would expect those objects to remain in the cache much longer.
We are also using Fastly shielding, so if an object is no longer available on an edge node, we would expect the shield to provide another cache layer before the request reaches the Ibexa backend.
As far as we can tell, these objects are not being explicitly purged by Ibexa or through a content update.
My questions are:
Is there a way to determine why a particular object was evicted or removed from Fastly’s cache before its TTL expired?
Can Fastly provide any debugging information that distinguishes between TTL expiration, explicit purge/invalidation, and cache eviction?
With shielding enabled, is there a way to determine whether a MISS at the edge resulted in a shield HIT or an actual request to the origin?
Are there any Fastly debugging headers or logging options that would help trace this behavior?
Any ideas on how to investigate this would be greatly appreciated.
We won’t be able to fully pursue these questions here, some of them will likely have to be handled via a support ticket, but there will be some basic information required to make progress.
Which POP(s) are you using for shielding?
What is the is RPS of the objects which you are expecting to stay in the cache for longer than 30 minutes?
That name you have provided is a specific cache node in the Fastly network, it’s not the name of a POP. I presume based on the name that you have chosen MUC as your shield POP though.
1 or 2 requests per hour will not be sufficient to keep content alive in the Fastly cache, I believe.
Requests that result in 404 responses from the origin will not have any effect on previously-cached repsonses with 200 (or similar) status codes.
Do you have any recommendations for keeping cached content in Fastly for longer and reducing the likelihood of early eviction?
Also, is there a way to determine how frequently an object needs to be requested in order to have a good chance of remaining in the cache? Is there any guidance regarding the relationship between request frequency and cache eviction?
Hey @kisse91 – There are multiple variables that factor into an LRU (least recently used) eviction on our network. It depends on an object’s request frequency, as well as the physical capacity of the specific POP in our network where the object is stored, and that POP’s popularity with global traffic. It’s not something where there’s a clear-cut solution, beyond increasing your object’s request frequency.
Broadly speaking, I would recommend shifting your shield to a major metro area, potentially Frankfurt if you’re already using MUC, where there’s more physical. But again, there’s no clear guarantee that this may change the experience.
is there any way to determine whether the MUC POP we are currently using as our shield has sufficient cache capacity for our traffic, or whether cache pressure could be causing the early evictions we are seeing?
The following table shows the request metrics from our production environment. Could you give us your assessment of the cache miss rate? From our perspective, it seems unusually high, even if we exclude the 404 requests, which mostly result in cache misses.
The statistics are aggregated per day, based on data with a one-minute resolution:
Metrics
Total
Average
Min
Max
95th Percentile
Bandwidth (B)
58 GB
0 GB
0 GB
2 GB
0 GB
Requests
787 701
548
46
6 083
1 103
Hits
494 475
344
33
2 797
707
Pass
88 988
62
0
758
203
Miss
150 637
105
1
3 202
289
Errors
18 983
13
0
217
36
Origin Offload (%)
100 %
99,88 %
100 %
100 %
Hit Ratio (%)
77,91 %
27,91 %
97,78 %
92,71 %
Cache Coverage (%)
87,65 %
20,22 %
100 %
98,78 %
All Status 1xx
0
0
0
0
0
All Status 2xx
635 180
442
34
6 035
901
All Status 3xx
65 289
45
4
434
113
All Status 4xx
72 619
51
0
873
198
All Status 5xx
14 613
10
0
114
29
Status 406
49 184
34
0
750
160
Status 404
18 959
13
0
358
38
Status 429
3 427
2
0
209
0
Miss Time (ms)
912,55 ms
0,64 ms
0 ms
24,52 ms
1,59 ms
Also, would it make sense to periodically request a set of important pages, for example every five minutes, to keep those objects warm in the cache if you compare it with the total requests we get? Or would you recommend a different approach to prevent frequently used content from being evicted prematurely?
Can you explain what goal you are trying to achieve that would be helped by the content staying in the Fastly cache? The two primary reasons for most customers are origin offload and latency (time-to-first-byte) reduction. In your case the origin fetch rate is so low that origin offload doesn’t seem to be of much benefit.
Latency reduction could be a benefit but only if the client requests are originating from locations near the MUC POP and the origin is not near the MUC POP. If the requests are originating from locations which are not near the MUC POP then the bulk of the apparent latency for those requests is going to be edge-to-shield requests, which won’t be eliminated by the content being present in the cache in the MUC POP.
As Austin’s reply noted, there will never be enough information available to customers to be able to evaluate whether their content is likely to stay present in the cache in any given POP or not when the traffic level is this low. Traffic patterns change dramatically on a daily and weekly basis, plus they also change dramatically when major events occur on the Internet (or in the real world). For example a World Cup football match which happened to include Bayern Munich would likely push most ‘low activity’ content out of the cache in the MUC POP for the duration of the match, because all the streaming services carrying the match would be pushing a large amount of content through the POP, and the remaining cache space would be retained by sites that have a regular high-traffic pattern.
Our main goal is to increase the cache hit rate in Fastly and keep pages cached for as long as possible. Since our application does not have a particularly large number of URLs or a high amount of traffic overall, I would expect our cache footprint to be relatively small. So, in theory, there should be enough cache capacity to keep our pages cached for longer. Maybe I just still do not know, how fastly works.
Yes, you’ve stated that, but what will that achieve for you? The fact that your traffic volume is low is actually working against you, not for you: the cache system does not allocate chunks of space per customer (or per service), it’s a shared cache across all services which receive traffic in the POP, so a service with low traffic volume will have a relatively low cache-hit ratio. There is no mechanism available to ‘pin’ the content in the cache, at least not any which would be relevant in this situation.
If you can describe what benefits you expect to achieve by keeping the content in the cache, we may be able to suggest alternative approaches.