Diagnosing fastly setup for amazon s3 bucket


#1

Greets! I have a piece of OSS (rails) I’ve inherited and it’s configurations seem a bit odd. And I’m also not sure it’s even doing anything. Let’s start with the Fastly configs for our static assets.

I want to achieve optimal performance from caching. As a reference I pings an image from fastly.com

https://www.fastly.com/sites/default/files/fastly_logo.png -> 136ms

Pretty quick, not bad, right? Presumably the people at Fastly indeed trust their service enough to use it on their home page :stuck_out_tongue:

Now let’s move on to speed testing the OSS image served by Fastly:

https://act.s.eff.org/uploads/0020c606c08d49e3bca109395a08b659/airship-banner.jpg -> 415ms

Oh noez, that’s not as fastly as the Fastly one :frowning:

Let’s test the backend directly:

https://s3-us-west-1.amazonaws.com/actioncenter/uploads/0020c606c08d49e3bca109395a08b659/airship-banner.jpg -> 458ms

That number is ~50ms slower than going through Fastly.

My assumption is that when I request the asset from Fastly, Fastly then immediately requests it from S3, waits for S3 to finally serve it up, and then finally Fastly serves it back to me. How do I test this hypothesis and correct the problem?

Edit: These tests are done in firefox with the ‘disable cache’ checkbox ticked to simulate a user’s first browse of the app. The net panel’s response times are what I used for speed testing.


#2

My assumption is that when I request the asset from Fastly, Fastly then immediately requests it from S3, waits for S3 to finally serve it up, and then finally Fastly serves it back to me.

That’s true for a cache MISS (ie the CDN doesn’t have the file in the POP servicing the request).

How do I test this hypothesis and correct the problem?

What’s the problem? From what you’ve said, the file is slower being requested from origin and faster going through the CDN. That sounds right to me!

I prefer to troubleshoot using curl, as I find it easier.

There are a few things you can look for to see if a file is cached or not:

curl -svo /dev/null https://act.s.eff.org/uploads/0020c606c08d49e3bca109395a08b659/airship-banner.jpg

< Age: 0
< X-Served-By: cache-lcy1126-LCY
< X-Cache: MISS
< X-Cache-Hits: 0

Those headers show:

  1. How long that resource has been in this particular cache node,
  2. Which cache node serviced the request,
  3. Whether it was a HIT or a MISS,
  4. How many HITs this cache node has served.

For a subsequent request I get the following:

< Age: 541
< X-Served-By: cache-lcy1127-LCY
< X-Cache: HIT
< X-Cache-Hits: 1

For requests that are PASSes for some reason, you’ll always see X-Cache: MISS.

In addition there are some timing features you can use with curl to make diagnostics easier.
See point #4 here: Diagnosing Latency Issues

You should make the request several times so you’re sure you’re getting HITs and note the times. You can then compare times going through Fastly and straight to the origin.


#3

Thanks for offering me help Justin!

In FF’s net tab, when I request the Fastly Logo, I see a response timing broken down to 104ms of ‘waiting’ time. Correct me if I’m wrong, but waiting represents time spent where the request is on it’s way to the server, where the response is on it’s way back, and where the server itself is doing something. In reference to this SO http://stackoverflow.com/questions/11587867/firebug-net-waiting-time I’m following up on configurations in fastly to see if that’s configured properly.

So I wondered if it had to do with the size of the image, so I put the fastly image right onto my asset stack and checked to see how that exact fastly logo (3.8kb) did on my configuration. With a warm cache, I get…

https://act.s.eff.org/uploads/31d7ce9239a948278b3c0102a7e55b72/fastly_logo.png -> 104ms

And S3…

https://actioncenter.s3-us-west-1.amazonaws.com/uploads/31d7ce9239a948278b3c0102a7e55b72/fastly_logo.png -> 130ms

Those are really good times. So is the high wait time simply to be expected for large images? (The blimp is about 60kb). I would expect the browser to show me a higher ‘recieving’ time, rather than a higher ‘wait’ time, but perhaps I’m misunderstanding the metrics, or perhaps there’s some niche understanding of how large files are served vs small files that I’m not aware of.


EDIT: It looks like I forgot to mention the breakdown for requests of the large file on my implementation, that was really bad of me, I was working late. I’m testing again, and I’m not getting similar results, is someone from fastly working on my setup? That’s really cool and awesome if so, otherwise my performance issues may have just disappeared or been imagined in a late night devops episode of a webdev struggling against his biological need for sleep:

test subjects:

https://s3-us-west-1.amazonaws.com/actioncenter/uploads/0020c606c08d49e3bca109395a08b659/airship-banner.jpg
https://act.s.eff.org/uploads/0020c606c08d49e3bca109395a08b659/airship-banner.jpg

Direct from S3:

* Wait Time:  115ms
* Recieving:  317ms
* Total:  ~432ms

Via Fastly:

* Wait Time:  155ms
* Recieving:  180ms
* Total:  ~335ms