Unexpected TTL, fastly-debug-ttl

General
1

Hi,

I am getting an inconsistent result for TTL on a webpage.

The file is at path /static/css/styles.css

The server is sending a header “cache-control:max-age=604800”

At that point, you would expect to see a TTL based on 604800 which is a week. It would count down from a week.

In a VCL snippet, fetch, priority 100:

if ( req.url ~ "^/$" ) {
  set beresp.ttl = 300s;
  return (deliver);
} else if ( req.url ~ "^/static/css/" ) {
  set beresp.ttl = 300s;
  return (deliver);
...

The TTL is being overridden to 300s.

Which will take precedence? 604800s or 300s? The intention is to use 300s.

On the command-line, running this command:

curl -svo /dev/null -H "Fastly-Debug:1" https://www.example.com/static/css/styles.css 2>& 1 | grep fastly-debug-ttl 2>&1

result:

< fastly-debug-ttl: (H cache-den8262-DEN 602565.735 0.000 2234)

fastly-debug-ttl shows the TTL is based on the 604800. It’s not using 300 seconds.

Then completely purge the cache from the Fastly console.

Run the command again a few times.

curl -svo /dev/null -H "Fastly-Debug:1" https://www.example.com/static/css/styles.css 2>& 1 | grep fastly-debug-ttl 2>&1

result:

< fastly-debug-ttl: (H cache-den8265-DEN 289.977 0.000 10)

Now it’s 300 seconds. Great! That was the goal. Although it’s mysterious since the VCL didn’t change.

I had done a full purge from the Fastly console, before that. Maybe the purge fixed it.

Wait another 5 minutes.

curl -svo /dev/null -H "Fastly-Debug:1" https://www.example.com/static/css/styles.css 2>& 1 | grep fastly-debug-ttl 2>&1

result:

< fastly-debug-ttl: (H cache-den8263-DEN 604793.347 0.000 7)

Back to using 604800 seconds as the TTL.

Any ideas about what’s happening?

2

Hey @sdarwin! I’m a Solutions Engineer at Fastly. In Fastly, beresp.ttl will take precedence over Cache-Control. The order of precedence is as follows:

  • beresp.ttl
  • Surrogate-Control
  • Cache-Control
  • Expires

If no caching directive is available that the fallback TTL is used. By default that’s set to 1hr and you can change this in the UI settings.

Do you still see the issue? From the code snippet you’ve shared I would suggest that you remove the return(deliver); statements in your code snippet. It is causing your VCL code to exit the vcl_fetch subroutine early and skipping any other important default VCL code. The easiest way to see what code is being skipped is to click on ‘Show VCL’ on the UI and navigate to your code snippet. Any code that comes after this snippet in the vcl_fetch subroutine is not being executed.

3

Hi Arisa,

I would suggest that you remove the return(deliver);

Most Fastly documentation includes return(deliver)

Google this query : fastly set ttl set beresp.ttl

Result 1: beresp.ttl | Fastly Documentation

 set beresp.ttl = 321s;
  return (deliver);
}

Result 2: Cache freshness and TTLs | Fastly Documentation

set beresp.ttl = 3600s; # Cache in Fastly
set beresp.ttl -= std.atoi(beresp.http.Age);
return(deliver);

and so on. There are more results.

Why is the documentation recommending that?

If I examine “Show VCL”, then the fetch routine ends with something like this:

else {
    # apply the default ttl    
    set beresp.ttl = 300s;
      }

It will apply the default ttl if you permit execution of the entire subroutine.

Do you still see the issue?

This morning, if I run the exact same test:

< fastly-debug-ttl: (H cache-den8278-DEN 3097.294 0.000 503)

Good.

That is without any modification of the VCL.

So it’s not certain… if purge takes much longer than expected (more than 5 minutes), or if the issue is still present but it’s sporadic, or another explanation.

4

Hi Arisa,
I spoke too soon. :slight_smile: In the above post, notice that it’s using 3600 seconds (one hour). We have three environments “production” “staging” and “testing”, with the same VCL. STAGE is using 300 seconds. PRODUCTION is using 3600 seconds. The above 3600s was from production.

Switching back to staging, it was successful also when I posted earlier.

$ curl -svo /dev/null -H "Fastly-Debug:1" https://www.example.com/static/css/styles.css 2>& 1 | grep fastly-debug-ttl 2>&1

300 seconds:

< fastly-debug-ttl: (H cache-den8233-DEN 287.606 0.000 12)

However, now just an hour later, on the same staging website:

< fastly-debug-ttl: (H cache-den8258-DEN 603430.562 0.000 1369)

The issue reappears again.

Here is a wild guess… Is 300 seconds too short? Maybe the time is so brief that it is confusing the system in some way. But I think last week I was also testing PRODUCTION where the setting is 3600 seconds, and even then 604800 seconds appeared.

5

Hey @sdarwin,

For the first link you shared (Result 1), that’s an example from a Fiddle, our VCL sandbox environment. It has a quirk where you do need to call return(deliver); early to see the TTL overwrite take effect. Here’s a slightly altered example to demonstrate what I mean: Fiddle - Fastly. For the Result 2, I think what the example code is supposed to show that eventually return(deliver); should be called. In your case, since you’re using VCL snippets, you can see in ‘Show VCL’ that Fastly automatically takes care of this for you by calling it at the end of the subroutine. This means that you don’t have to call it.

As for why your issue appears again, 300s isn’t too short. Theoretically the lowest TTL you can set is 1s. I would recommend that you get in touch with our support team at support@fastly.com to help you troubleshoot your specific issue. They will be able to review your service and determine if anything else could be causing this issue.

6

In the slightly altered example it says:

  1. set beresp.ttl = 67s;

at first apparently setting the TTL to 67s.

And then next:

  1. “return(deliver) is uncommented here to show that default TTL of 1hr is applied //return (deliver);”

I think it means “commented”.

It is “commented out” with two slashes, which means essentially “removing”.

The result of removing “return(deliver)” is that the “default TTL of 1hr is applied”.

That agrees with what I was saying. Right?

If “return(deliver)” is absent, the “default TTL of 1hr is applied”.

So “return(deliver)” ought to be present to override the TTL.

7

Hi @sdarwin,

Sorry let me a bit more clear about what I meant. The Fiddle quirk I mentioned is only specific to Fiddle and not the way the usual Fastly service works.

Recommendation for TTL overwrite for:

Fiddle - Add a return(deliver) statement, exactly as what you observed above. If you remove the return(deliver); (in the example it’s commented out with //) then the default TTL of 1 hr is applied.

Usual VCL service - You do not need to add return(deliver) yourself when adding a snippet. This is because Fastly automatically adds this at the end of the vcl_fetch subroutine. The reason why we want this to be at the end is so that we can run through the entire subroutine. When the code executes as usual, Fastly will search whether you’ve set beresp.ttl or any other caching directive (e.g. cache-control) in this order of precedence (highest to lowest priority):
beresp.ttl > surrogate-control > cache-control > expires.
If it doesn’t find any of them, then the default TTL will take effect. In your case this is 300s.

Hope that this helps clear things up!

8

About the original problem: for the moment I am exploring the solution of removing the Cache-Control header from the server-side which will hopefully remove the conflict between the different TTLs, and maybe that should be done anyway if we are trying to override the Cache-Control all the time.

Continuing the discussions of setting the TTL in VCL, our websites are behaving the same way as the fiddles.

Observe the code. The last section of vcl_fetch is this, and it’s provided automatically by Fastly. The TTL here is the fallback TTL from the GUI.

  if (beresp.http.Expires || beresp.http.Surrogate-Control ~ "max-age" || beresp.http.Cache-Control ~"(?:s-maxage|max-age)") {
    # keep the ttl here
  } else {
        # apply the default ttl
    set beresp.ttl = 300s;
  }
  return(deliver);
}

What does this accomplish? Unless we have something set in terms of “beresp.http.Expires” or “beresp.http.Surrogate-Control” or “beresp.http.Cache-Control”, it’s going to put the TTL to 300s.
Right? That will cancel any overrides we attempted. It will override the overrides. And set the TTL to 300s most of the time.

Testing confirms that. Consider this snippet below.

Priority 100 vcl_fetch snippet

It will set the TTL to 500s. This works.

...
} else if ( req.url ~ "^/static/css/" ) {
    set beresp.ttl = 500s;
    return (deliver);
}
...

However this fails to set the TTL:

...
} else if ( req.url ~ "^/static/css/" ) {
    set beresp.ttl = 500s;
    # return (deliver);
}
...

Why? Probably because of the code section above where it says set beresp.ttl = 300s; , which is the fallback ttl.

9

Hi @sdarwin,

I think your specific problem would be more suited for our support team. They’ll be able to best advise you on how to resolve your issue by looking at your entire service instead of just a code snippet. You can email them at support@fastly.com.

10

Hi @ArisaRoong,

Thanks. I have contacted the support team about the original issue in this thread, the unexpected ttl, switching between 300s and 604800s.

There is a second issue that has appeared in our discussion. Something new. And that is “how can you set beresp.ttl in a snippet in the Fastly web console?” Approximately the last 8 lines of code in vcl_fetch (provided in the VCL automatically) will almost always overwrite beresp.ttl unless you find a way to skip that section.

11

Hi @sdarwin,

I see what you mean now. You’re referring to this piece of default code:

# If no TTL has been provided in the response headers, set a default
  if (!beresp.http.Expires && !beresp.http.Surrogate-Control ~ "max-age" && !beresp.http.Cache-Control ~ "(?:s-maxage|max-age)") {
    set beresp.ttl = 3600s;

    # Apply a longer default TTL for images processed using Image Optimizer
    if (req.http.X-Fastly-Imageopto-Api) {
      set beresp.ttl = 2592000s; # 30 days
      set beresp.http.Cache-Control = "max-age=2592000, public";
    }
  }

In this case I would recommend that you use custom VCL. I would recommend that you read this documentation here which talks about Fastly macros and this section on custom VCL Custom VCL allows you to place your TTL overwrite code after the default TTL code. Also when using custom VCL make sure that you copy the entire VCL boilerplate before adding your own code. Give this a go and in the same support ticket, you can ask them to check whether you have implemented it correctly.

12

Thanks for the information.
Custom VCL… It seems like a big step. maybe that will be needed, eventually.
Noticed this in the boilerplate.

 # If the response is setting a cookie, make sure it is not cached
  if (beresp.http.Set-Cookie) {
    return(pass);
  }

By calling return (deliver); earlier that was skipped. Which then permits the usage of low-variability cookies (0 or 1), rather than disabling caching every time in such a case. So calling return (deliver);, and skipping the section, may actually help us, and allow our cookie strategy to work.