Ways to Test Your Caching with cURL


#1

Add the Fastly-Debug: True header to a request

  • Example: curl -svo /dev/null [url] -H "Fastly-Debug: true"

  • Purpose: provides more information about how the object is cached and handled by Fastly caches.

  • The following three headers are included in the object response:
    Fastly-Debug-Path: which cache servers serve the request
    Fastly-Debug-TTL: the TTL of the object in the queried server(s)
    Fastly-Debug-Digest: the hash digest of the object.


Run a request through a specific cache node

  • Example: curl -svo /dev/null <url> -x <cache-node>.hosts.fastly.net:80
    In the place of <cache-node>, you can insert any cache node that you usually see in the “X-Served-By” header on a response (ie cache-jfk1020, cache-lax1427, cache-fra1246, etc) running through Fastly.

  • Purpose: allows you to target your testing and make multiple calls to the same cache server. This can be useful for testing HIT/MISS, TTL values, or to check if the cache is correctly serving different versions of an object (for example if GZIP is enabled).


Add the Fastly-No-Shield: true header to a request

  • Example: curl -svo /dev/null [url] -H "Fastly-No-Shield: true"

  • Purpose: Disables clustering on the request. This may help you see what the request/response looks like when one cache server communicates directly with your origin.


Test a URL with a loop:

  • Examples:
    for ((i=1;i<=10;i++)); do curl -svo /dev/null www.example.com; sleep 2; done
    while true; do curl -svo /dev/null www.example.com; sleep 2; done

  • Purpose: this is convenient for seeing the results from numerous url calls and can also be helpful for testing logging output/performance.

    To monitor for the presence of certain headers (Etag, Content-Type, etc) in a sample of responses:

    for ((i=1;i<=1000;i++)); do curl -svo /dev/null www.example.com 2>&1 | grep Etag; sleep 1; done

    while true; do curl -svo /dev/null www.example.com 2>&1 | grep Content-Type; sleep 1; done


#2

Thanks for all those useful hints, they saved me a ton of headache and time already.

One question: If I want a request to be handled via https, not http, and still want to specify a specific cache node, how would I do this? I tried

curl -svo /dev/null <https-url> -x <cache-node>.hosts.fastly.net:443

which returns a Proxy CONNECT aborted and

curl -svo /dev/null <https-url> -x <cache-node>.hosts.fastly.net:80

which returns a HTTP/1.1 400 Invalid URI.

Do you know what’s the right way to do it? I understand that normally fetching via http or https should yield the same results, but there might be some different rules in place which make the behaviour different.


#3

You will want use --resolve in your curl command:

curl -svo /dev/null --resolve '<customer_hostname>:443:<ip address of cache node>' <https URL>

There are a few things you will need to confirm before running this command:

  • That you have TLS enabled with Fastly via a Shared Certificate or Hosted Cert
  • Determine the IP address of the cache node

For instance:

host cache-lax1420.hosts.fastly.net
cache-lax1420.hosts.fastly.net has address "199.27.79.20"

Note: If you have a hosted certificate, change the last octet to match your dedicated map

Then run your command:

Example:

curl -svo /dev/null --resolve 'www.example.com:443:199.27.79.20' https://www.example,com/some_file

#4

That’s awesome, thanks so much @cassandra!


#5

Another tip is to use --compressed so that curl sends Accept-Encoding: deflate, gzip, to mimic browser behavior a little more closely.

And it’s always a good idea to test both with and without --compressed to make sure that responses are compressed when they should be and aren’t when they shouldn’t. Content-Encoding: gzip in the response headers signifies that the response is compressed.


#6

To test with a loop:
watch -n [number] curl -svo /dev/null [url] -H “Fastly-Debug: true”

Where “number” is an integer.