Inconsistent cache lookup


#1

Hi all,
We have a fairly complicated setup running with multiple backends and URL rewrites and fallback in case one of our backends returns an error. The gist of it is that when a request to our main backend fails with a 5xx error, we rewrite the request url and set the req.host and backend to another backend which serves up some static backup content, so things can still run smoothly. Another thing to note is that we have shielding enabled. The last thing is that to enable more consistent PURGE behavior, we’ve rewritten the hash function to use the host and url from the initial request (we save these as headers on the request and use them later in vcl_hash. Our hash function looks like:

sub vcl_hash {

  call log_hash;

  set req.hash += req.http.Appcues-Original-Url;
  set req.hash += req.http.Appcues-Original-Host;
#FASTLY hash
  return (hash);
}

I’m seeing inconsistent behavior with caching when we do this fallback. Because we have shielding enabled, I’m expecting that the content should be cached on the shield after we make our first request. This seems to happen sometimes, but only occasionally - normally we get a MISS. This isn’t the behavior when we’re not in the fallback case, which seems very odd. I’m listing a sequence of HTTP requests/responses (irrelevant headers are removed) that illustrate what’s going on:

This is a sequence of reqests for the fallback case (doesn’t work as expected):

GET /timeout/timeout.js HTTP/1.1
Connection: keep-alive
Fastly-Debug: True

HTTP/1.1 200 OK
Age: 0
Cache-Control: max-age=10
Date: Wed, 11 Mar 2015 17:25:20 GMT
ETag: "b9f7f844934f40cdc6e66918a32052ac"
Fastly-Debug-Path: (D cache-atl6231-ATL 1426094720) (F cache-atl6225-ATL 1426094720) (D cache-jfk1023-JFK 1426094720) (F cache-jfk1023-JFK 1426094720)
Fastly-Debug-TTL: (M cache-atl6231-ATL - - 0) (M cache-jfk1023-JFK - - 0)
Fastly-Restarts: 1
Surrogate-Control: max-age=2592000
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Served-By: cache-jfk1023-JFK, cache-atl6231-ATL
X-Timer: S1426094720.491773,VS0,VE101


GET /timeout/timeout.js HTTP/1.1
Connection: keep-alive
Fastly-Debug: True

HTTP/1.1 200 OK
Age: 0
Cache-Control: max-age=10
Date: Wed, 11 Mar 2015 17:25:23 GMT
ETag: "b9f7f844934f40cdc6e66918a32052ac"
Fastly-Debug-Path: (D cache-jfk1030-JFK 1426094723) (F cache-jfk1030-JFK 1426094723) 
Fastly-Debug-TTL: (M cache-jfk1030-JFK - - 0) 
Fastly-Restarts: 1
Surrogate-Control: max-age=2592000
X-Cache: MISS
X-Cache-Hits: 0
X-Served-By: cache-jfk1030-JFK
X-Timer: S1426094723.587333,VS0,VE70

This is a sequence of reqests for the normal case (works as expected):

GET /12.js HTTP/1.1
Connection: keep-alive
Fastly-Debug: True

HTTP/1.1 200 OK
Age: 0
Date: Wed, 11 Mar 2015 17:38:03 GMT
Etag: W/"ude9ClBZbGhd+qyfhXZaNQ=="
Fastly-Debug-Path: (D cache-atl6223-ATL 1426095483) (F cache-atl6230-ATL 1426095483) (D cache-jfk1024-JFK 1426095483) (F cache-jfk1030-JFK 1426095483)
Fastly-Debug-TTL: (M cache-atl6223-ATL - - 0) (M cache-jfk1024-JFK - - 0)
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Served-By: cache-jfk1024-JFK, cache-atl6223-ATL
X-Timer: S1426095482.819359,VS0,VE261


GET /12.js HTTP/1.1
Connection: keep-alive
Fastly-Debug: True

HTTP/1.1 200 OK
Age: 0
Date: Wed, 11 Mar 2015 17:38:06 GMT
Etag: W/"ude9ClBZbGhd+qyfhXZaNQ=="
Fastly-Debug-Path: (D cache-iad2133-IAD 1426095486) (F cache-iad2140-IAD 1426095486) (D cache-jfk1031-JFK 1426095486) (F cache-jfk1030-JFK 1426095483)
Fastly-Debug-TTL: (M cache-iad2133-IAD - - 0) (H cache-jfk1031-JFK - - 3)
X-Cache: HIT, MISS
X-Cache-Hits: 1, 0
X-Served-By: cache-jfk1031-JFK, cache-iad2133-IAD
X-Timer: S1426095486.263026,VS0,VE20

EDIT: A bit of further debugging – I’m logging the fastly_info.state variable and in the broken case, it shows MISS as the state for any node that lives in my shield cluster. On the edge nodes, it shows MISS-CLUSTER. In the case that does work properly, it shows MISS-CLUSTER as the state for the shield cluster nodes, however. This definitely seem to be indicative of the problem but I don’t know what’s causing it. Could the fact that req.restarts > 0 be causing this somehow?

I can’t seem to determine why the caching works inconsistently in the fallback case. Any thoughts?

Thanks!


#2

Hi-
If you look at your generated VCL from within the app, you will see the expanded #FASTLY hash macro shows the additional code of:

#-FASTLY HASH CODE WITH GENERATION FOR PURGE ALL
  #
    
  
  #if unspecified fall back to normal
  {
    set req.hash += req.url;
    set req.hash += req.http.host;
    set req.hash += "#####GENERATION#####";
    return (hash);
  }
  #--FASTLY END HASH CODE

So the below edits are most likely adding to the hash which already has req.http.host and `req.url``

Do you think this might be explaining the behavior you’re seeing?


#3

Unfortunately I don’t think that’s it. Because I’m overwriting the entire vcl_hash function in my custom VCL and just including that #FASTLY hash macro (which just adds the generation part), I get this in my generated VCL:

sub vcl_hash {

  call log_hash;

  set req.hash += req.http.Appcues-Original-Url;
  set req.hash += req.http.Appcues-Original-Host;

#--FASTLY HASH start
  # support purge all
  set req.hash += "#####GENERATION#####";
#--FASTLY HASH end
  return (hash);
}

I really think it’s somehow related to the req.restarts > 0 but it doesn’t seem like that should affect things. Unless the restart causes the hash to occur twice? Is that what you’re suggesting?


#4

OK. I don’t think it would have anything to do with restarts. Do you have shielding enabled?

This one might be hard to debug without going through the rest of your VCL.

edit - yeah…sorry…see the shielding. would still like to see the vcl :smile:


#5

and following up a bit…did you get a chance to look at this article?


#6

Yea, I’ve read through that one (and I think pretty much every other doc on here haha :slight_smile: ).
I’ll post the VCL but it’s pretty confusing to follow! (Also I blocked out some server-specifc stuff for privacy/security).

Our “main” VCL:

# Backends are defined in Fastly UI as F_bundler_service and F_s3_backend. We can safely make these references.

include "Deployment Specific Config";

# Set headers to store request info across restarts of the request in case it fails
sub set_bundle_headers {
  set req.http.Appcues-Requested-Bundle = "True";
}

sub vcl_recv {


  if (req.http.Appcues-Debug == "True") {
    call log_recv;
  }

  # Only rewrite if a GET or a PURGE and if it's not a reroute triggered from vcl_fetch
  if (req.request == "GET" || req.request == "FASTLYPURGE") {

    if (!req.http.Appcues-Original-Url && !req.http.Appcues-Original-Host) {
      # Save the original URL and host for creating the cache hash
      set req.http.Appcues-Original-Url = req.url;
      set req.http.Appcues-Original-Host = req.http.host;
    }


    # Rewriting /22.js -> /bundle/accounts/22/versions/stable
    if (req.url ~ "^/([0-9]+|test).js$") {
      # Set a bunch of headers for later
      set req.http.Appcues-Script-Account = regsub(req.url, "^/([0-9]+|test).js$", "\1");
      set req.http.Appcues-Script-Version = "stable";
      set req.http.Appcues-Script-Type = "min";
      call set_bundle_headers;
      set req.url = regsub(req.url, "^/([0-9]+|test).js$", "/bundle/accounts/\1/versions/stable");
    }
    # Rewriting /22.min.js -> /bundle/accounts/22/versions/stable?type=min
    else if (req.url ~ "^/([0-9]+|test).(([a-zA-Z]+)\.)?js$") {
      # Set a bunch of headers for later
      set req.http.Appcues-Script-Version = "stable";
      set req.http.Appcues-Script-Account = regsub(req.url, "^/([0-9]+|test).(([a-zA-Z]+)\.)?js$", "\1");
      set req.http.Appcues-Script-Type = regsub(req.url, "^/([0-9]+|test).(([a-zA-Z]+)\.)?js$", "\3");
      call set_bundle_headers;
      set req.url = regsub(req.url, "^/([0-9]+|test).(([a-zA-Z]+)\.)?js$", "/bundle/accounts/\1/versions/stable?type=\3");
    }
    # Rewriting /development/22.js -> /bundle/accounts/22/versions/development
    # Rewriting /1.9.1/22.js -> /bundle/accounts/22/versions/1.9.1
    else if (req.url ~ "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).js$") {
      # Set a bunch of headers for later
      set req.http.Appcues-Script-Version = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).js$", "\1");
      set req.http.Appcues-Script-Account = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).js$", "\2");
      set req.http.Appcues-Script-Type = "min";
      call set_bundle_headers;
      set req.url = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).js$", "/bundle/accounts/\2/versions/\1");
    }
    # Rewriting /development/22.debug.js -> /bundle/accounts/22/versions/development?type=debug
    # Rewriting /1.9.1/22.min.js -> /bundle/accounts/22/versions/1.9.1?type=min
    else if (req.url ~ "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).(([a-zA-Z]+)\.)?js$") {
      # Set a bunch of headers for later
      set req.http.Appcues-Script-Version = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).(([a-zA-Z]+)\.)?js$", "\1");
      set req.http.Appcues-Script-Account = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).(([a-zA-Z]+)\.)?js$", "\2");
      set req.http.Appcues-Script-Type = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).(([a-zA-Z]+)\.)?js$", "\4");
      call set_bundle_headers;
      set req.url = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([0-9]+|test).(([a-zA-Z]+)\.)?js$", "/bundle/accounts/\2/versions/\1?type=\4");
    }
    # Testing/debug route for timeouts
    else if (req.url == "/timeout/timeout.js") {
      set req.http.Appcues-Script-Version = "timeout";
      set req.http.Appcues-Script-Account = "timeout";
      set req.http.Appcues-Script-Type = "min";
      call set_bundle_headers;
    }
  }

#FASTLY recv

  # If this is a restart (i.e. the previous attempt failed) and we were requesting a bundle before we should reroute to S3
  # OR if this a bundle request but the backend is unhealthy then we should also reroute to S3
  if ((req.request == "GET" || req.request == "FASTLYPURGE") && req.http.Appcues-Requested-Bundle == "True" &&
      (req.restarts > 0 || !req.backend.healthy )) {
    # Reroute /22.js to /js/appcues.min.js or <version>/22.js to /js/<version>/appcues.min.js
    if (req.http.Appcues-Original-Url ~ "([0-9]+|test).js$") {
      set req.url = "/js" regsub(req.http.Appcues-Original-Url, "([0-9]+|test).js$", "appcues.min.js");
      # Set static resource headers since we're now requesting the files from S3
      set req.http.Appcues-Static-Resource-Version = req.http.Appcues-Script-Version;
      set req.http.Appcues-Static-Resource-Name = "appcues.min.js";
      call log_fallback;
      call route_to_s3;
    }
    # Reroute /22.min.js to /js/appcues.min.js or <version>/22.min.js to /js/<version>/appcues.min.js
    else if (req.http.Appcues-Original-Url ~ "([0-9]+|test)\.min\.js$") {
      set req.url = "/js" regsub(req.http.Appcues-Original-Url, "([0-9]+|test)\.min\.js$", "appcues.min.js");
      # Set static resource headers since we're now requesting the files from S3
      set req.http.Appcues-Static-Resource-Version = req.http.Appcues-Script-Version;
      set req.http.Appcues-Static-Resource-Name = "appcues.min.js";
      call log_fallback;
      call route_to_s3;
    }
    # Reroute /22.debug.js to /js/appcues.js or <version>/22.debug.js to /js/<version>/appcues.js
    else if (req.http.Appcues-Original-Url ~ "([0-9]+|test)\.debug\.js$") {
      set req.url = "/js" regsub(req.http.Appcues-Original-Url, "([0-9]+|test)\.debug\.js$", "appcues.js");
      # Set static resource headers since we're now requesting the files from S3
      set req.http.Appcues-Static-Resource-Version = req.http.Appcues-Script-Version;
      set req.http.Appcues-Static-Resource-Name = "appcues.js";
      call log_fallback;
      call route_to_s3;
    }
    # Testing/debug route for timeouts
    else if (req.url == "/timeout/timeout.js") {
      # Set static resource headers since we're now requesting the files from S3
      set req.http.Appcues-Static-Resource-Version = req.http.Appcues-Script-Version;
      set req.http.Appcues-Static-Resource-Name = "appcues.min.js";
      set req.url = "/js/appcues.min.js";
      call log_fallback;
      call route_to_s3;
    }
  }

  # When req.url is like /development/appcues.css or just /appcues.css
  # When req.url is like /development/widget-bundle.js or just /widget-bundle.js
  # When req.url is like /development/widget-bundle.debug.js or just /widget-bundle.debug.js
  # When req.url is like /development/widget-bundle.min.js or just /widget-bundle.min.js
  # Or if this request is a reroute/fallback to S3 trigger in vcl_fetch
  if ((req.request == "GET" || req.request == "FASTLYPURGE") && (req.url ~ "^/([-0-9a-zA-Z\.]+)/([-0-9a-zA-Z\.]+\.css|widget-bundle\.js|widget-bundle\.min\.js|widget-bundle\.debug\.js)" ||
        req.url ~ "^/[-0-9a-zA-Z\.]+\.css|widget-bundle\.js|widget-bundle\.min\.js|widget-bundle\.debug\.js")) {
    # prepend all these requests with /js
    if (req.backend != shield_jfk_ny_us) {

      if (req.url ~ "^/([-0-9a-zA-Z\.]+)/([-0-9a-zA-Z\.]+\.css|widget-bundle\.js|widget-bundle\.min\.js|widget-bundle\.debug\.js)") {
        set req.http.Appcues-Static-Resource-Version = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([-0-9a-zA-Z\.]+\.css|widget-bundle\.js|widget-bundle\.min\.js|widget-bundle\.debug\.js)", "\1");
        set req.http.Appcues-Static-Resource-Name = regsub(req.url, "^/([-0-9a-zA-Z\.]+)/([-0-9a-zA-Z\.]+\.css|widget-bundle\.js|widget-bundle\.min\.js|widget-bundle\.debug\.js)", "\2");
      }
      else {
        set req.http.Appcues-Static-Resource-Version = "stable";
        set req.http.Appcues-Static-Resource-Name = regsub(req.url, "^/([-0-9a-zA-Z\.]+\.css|widget-bundle\.js|widget-bundle\.min\.js|widget-bundle\.debug\.js)$", "\1");
      }

      # Send .debug. versions to unminified
      if (req.url ~ "\.debug\.(js|css)$") {
        set req.url = "/js" regsub(req.url, "\.debug\.", ".");
        call log_static_resource_reroute;
        call route_to_s3;
      }
      # Send .min. versions to minified
      else if (req.url ~ "\.min\.(js|css)$") {
        set req.url = "/js" req.url;
        call log_static_resource_reroute;
        call route_to_s3;
      }
      # Send anything else to minified
      else {
        set req.url = "/js" regsub(req.url, "\.(css|js)$", ".min.\1");
        call log_static_resource_reroute;
        call route_to_s3;
      }
    }
  }

  # When req.url is like /development/appcues.js or just /appcues.js
  # When req.url is like /development/appcues.min.js or just /appcues.min.js
  # When req.url is like /development/appcues.debug.js or just /appcues.debug.js
  # Don't do this when we have requested a bundle
  if ((req.request == "GET" || req.request == "FASTLYPURGE") && req.http.Appcues-Requested-Bundle != "True") {
    if (req.backend != shield_jfk_ny_us) {
      # Go to non-user-specific, minified code
      if (req.url ~ "^(/[-0-9a-zA-Z\.]+)?/(appcues\.min\.js|appcues\.js)$") {
        set req.http.Appcues-Static-Resource-Version = regsub(req.url, "^/?([-0-9a-zA-Z\.]+)?/(appcues\.min\.js|appcues\.js)$", "\1");
        set req.http.Appcues-Static-Resource-Name = "appcues.min.js";
        set req.url = "/js" regsub(req.url, "appcues.*\.js$", "appcues.min.js");
        call log_static_js_request;
        call route_to_s3;
      }
      # If .debug go to unminified code
      else if (req.url ~ "^(/[-0-9a-zA-Z\.]+)?/appcues\.debug\.js$") {
        set req.http.Appcues-Static-Resource-Version = regsub(req.url, "^/?([-0-9a-zA-Z\.]+)?/appcues\.debug\.js$", "\1");
        set req.http.Appcues-Static-Resource-Name = "appcues.debug.js";
        set req.url = "/js" regsub(req.url, "appcues\.debug\.js$", "appcues.js");
        call log_static_js_request;
        call route_to_s3;
      }
    }
  }

  if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
    return(pass);
  }

  if (req.http.Appcues-Debug == "True") {
    call log_recv_end;
  }

  return(lookup);
}

sub vcl_fetch {
#FASTLY fetch

  # In the case where the response is coming back with the static version of the script
  # when we were actually requesting a bundle, set the Surrogate-Key to the account-specific values.
  if (req.http.Appcues-Requested-Bundle == "True" && (req.http.Appcues-Routed-To-S3 || beresp.http.Appcues-Routed-To-S3 == "True")) {
    if (req.http.Appcues-Debug == "True") {
      call log_rewrite_keys;
    }
    set beresp.http.Surrogate-Key = "embedscript account/" req.http.Appcues-Script-Account " version/" req.http.Appcues-Script-Version " type/" req.http.Appcues-Script-Type " version/" req.http.Appcues-Script-Version "/type/" req.http.Appcues-Script-Type;
  }

  # Set Surrogate-Key if there's no value
  if (!beresp.http.Surrogate-Key && req.http.Appcues-Routed-To-S3 == "True" && !req.http.Appcues-Requested-Bundle &&
      req.http.Appcues-Static-Resource-Name && req.http.Appcues-Static-Resource-Version) {
    set beresp.http.Surrogate-Key = "static-resource static-resource-name/" req.http.Appcues-Static-Resource-Name " static-resource-version/" req.http.Appcues-Static-Resource-Version " static-resource-version/" req.http.Appcues-Static-Resource-Version "/static-resource-name/" req.http.Appcues-Static-Resource-Name;
  }

  # If we had routed this request to S3 then we we want to overwrite the cache headers
  if (req.http.Appcues-Routed-To-S3 == "True") {
    call log_cache_headers;
    set beresp.http.Surrogate-Control = "max-age=2592000";
    set beresp.http.Cache-Control = "max-age=10";
    unset beresp.http.Expires;
  }

  if ((beresp.status == 500 || beresp.status == 503) && req.restarts < 1 && (req.request == "GET" || req.request == "HEAD")) {
    restart;
  }

  if(req.restarts > 0 ) {
    set beresp.http.Fastly-Restarts = req.restarts;
  }

  if (beresp.http.Set-Cookie) {
    set req.http.Fastly-Cachetype = "SETCOOKIE";
    return (pass);
  }

  if (beresp.http.Cache-Control ~ "private") {
    set req.http.Fastly-Cachetype = "PRIVATE";
    return (pass);
  }

  if (beresp.status == 500 || beresp.status == 503) {
    set req.http.Fastly-Cachetype = "ERROR";
    set beresp.ttl = 1s;
    set beresp.grace = 5s;
    return (deliver);
  }

  if (beresp.http.Expires || beresp.http.Surrogate-Control ~ "max-age" || beresp.http.Cache-Control ~"(s-maxage|max-age)") {
    # keep the ttl here
  } else {
    # apply the default ttl
    set beresp.ttl = 3600s;
  }

  return(deliver);
}

sub vcl_hit {
#FASTLY hit
  if (!obj.cacheable) {
    return(pass);
  }

  if (req.http.Appcues-Debug == "True") {
    call log_cache_hit;
  }

  return(deliver);
}

sub vcl_miss {
#FASTLY miss
  if (req.http.Appcues-Debug == "True") {
    call log_cache_miss;
  }

  return(fetch);
}

sub vcl_deliver {
#FASTLY deliver

  # Set CORS headers
  set resp.http.Access-Control-Allow-Origin = "*";
  set resp.http.Access-Control-Allow-Headers = "X-Requested-With,Authorization";
  set resp.http.Access-Control-Allow-Methods = "GET,PUT,POST,DELETE,OPTIONS";

  # If we're replying to an edge server. This will only run on the shield when it's acting as the shield.
  if (req.backend != shield_jfk_ny_us) {
    # We want to notify the edge server that we had to fallback to S3.
    if (req.http.Appcues-Routed-To-S3 == "True") {
      set resp.http.Appcues-Routed-To-S3 = "True";
    }
  }
  # If this is an edge server then unset the header so it doesn't go to the client
  else {
    if (resp.http.Appcues-Routed-To-S3 == "True") {
      unset resp.http.Appcues-Routed-To-S3;
    }
  }

  if (req.http.Appcues-Debug == "True") {
    call log_delivery;
  }

  return(deliver);
}

sub vcl_error {
#FASTLY error
}

sub vcl_pass {
  call log_pass;
#FASTLY pass
}

sub vcl_hash {

  call log_hash;

  set req.hash += req.http.Appcues-Original-Url;
  set req.hash += req.http.Appcues-Original-Host;
#FASTLY hash
  return (hash);
}

Our included VCL (at the top):

sub route_to_s3 {
  set req.backend = F_s3_backend;
  set req.http.host = "#######################";
  set req.http.Appcues-Routed-To-S3 = "True";
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Rerouting to S3: " req.http.host req.url;
}

sub log_static_resource_reroute {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Requesting a static resource, so rerouting to S3 -- current host: " req.http.host " url: " req.url " backend: " req.backend " restarts: " req.restarts;
}

sub log_fallback {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Previous request failed, attempting to fallback to S3 -- current host: " req.http.host " url: " req.url " backend: " req.backend " restarts: " req.restarts;
}

sub log_static_js_request {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Requesting a static appcues.js file: " req.http.host req.url " backend: " req.backend " restarts: " req.restarts;
}

sub log_cache_hit {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Got a cache HIT -- restarts: " req.restarts " object: " req.http.Appcues-Original-Host req.http.Appcues-Original-Url;
}

sub log_cache_miss {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Got a cache MISS -- restarts: " req.restarts " object: " req.http.Appcues-Original-Host req.http.Appcues-Original-Url;
}

sub log_delivery {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Cache state -- restarts: " req.restarts " object: " req.http.Appcues-Original-Host req.http.Appcues-Original-Url " state: " fastly_info.state;
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Delivery -- Version: " req.http.Appcues-Script-Version " Account: " req.http.Appcues-Script-Account " Type: " req.http.Appcues-Script-Type " ResourceVersion: " req.http.Appcues-Static-Resource-Version " Resource: " req.http.Appcues-Static-Resource-Name;
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Surrogate-Key values: " req.http.Surrogate-Key;
}

sub log_hash {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Hashing object (custom) -- url: " req.http.Appcues-Original-Url " host: " req.http.Appcues-Original-Host;
}

sub log_rewrite_keys {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Rewriting Surrogate-Key value because this is being cached as a bundled script. path=" req.url;
}

sub log_recv {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Received request -- url: " req.http.host req.url " server: " server.identity;
}

sub log_recv_end {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Processed request -- url: " req.http.host req.url " server: " server.identity;
}

sub log_cache_headers {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Writing Cache-Control and Surrogate-Control headers on resp -- server: " server.identity;
}

sub log_pass {
  log {"syslog ################ QA Papertrail :: "} time.start.usec " " time.elapsed.usec " Got a PASS on server: " server.identity;
}

#7

Hi Ryad,

What I think is getting in your way is that we disable clustering on restarts. Clustering is where one machine in a datacenter (cluster) is primary and one is secondary for any specific object and all the other nodes go there instead of your origin.

Because restarts are often because of errors, and the error might have been in the clustering (we are reevaluating this), the clustering is disabled. You can, however, force the clustering if you create the Fastly-Force-Shield header. That should take care of making sure it is the primary or secondary that do the requests to origin, and make sure that your objects are cached for the whole cluster.

Let us know how this goes.

Cheers,

Doc


#8

Hi Doc,

We ended up changing our VCL completely and this ended up being a non-issue at that point (We decided not to cache the fallback content at all). But your explanation certainly seems to explain the behavior I was seeing. I think this would have fixed it. So for anyone else in a similar situation to me, take a look at Doc’s answer above!

Thanks a lot for your help!