What's a good way to compare Fastly and my old CDN in a speed test?


#1

If you’re doing a test with Fastly, you’ll first need to set your service up so that it’s caching properly. This is a prerequisite to doing any meaningful benchmarking.

Once you have Fastly set up as you’d like to a production, you can test your whole page using WebPageTest. The application supports side by site testing

You can also test single objects with a quick script we wrote that use apache benchmark and MTR: https://gist.github.com/aspires/8189975

The code:

#!/bin/bash

# Dependencies: ApacheBench, MTR

REQUESTS=100
CONCURRENCY=10
FASTLY='www.example.com.global.prod.fastly.net'
CURRENT='www.example.com'

for url in 'path/to/test' 'path/to/test2' 'path/to/test3'; do
  for host in $CURRENT $FASTLY; do
    ab -n $REQUESTS -c $CONCURRENCY "http://${host}/${url}" >> fastly.ab.log
    #echo "http://${host}/${url}"
  done
done

for host in $CURRENT $FASTLY; do
  mtr -c $REQUESTS -w -r $host >> fastly.mtr.log
done

There are actually a lot of valid testing options available, and each has its tradeoffs.


#3

Also, @vvuksan has open sourced some great tools for inspecting whole requests: https://github.com/vvuksan/fantomTest


#4

A few lessons we’ve learned from doing CDN testing that could help during evaluation:

  • using “backbone” tests is often an inaccurate measure of how well a CDN performs. This is because backbone tests are done by testing nodes sitting in large data centers and not subjected to the normal bandwidth and latency constraints that real users are.
  • if a testing provider has a “last mile” option (as opposed to backbone), it’s generally a much better way of testing CDN performance since it’s closer to what real users experience. Last mile tests should either have testing nodes behind actual home routers, or should be emulating real bandwidth/latency conditions.
  • when drawing conclusions, make sure you have ample data. This may vary based on what is being tested, and how, but usually the more data the better
  • With large amounts of data, looking at medians is generally better than averages since they eliminate egregious outliers.
  • often, CDN testing ignores the performance of cacheable, but long-tail content. Long-tail content is stuff that’s cacheable, but doesn’t get requested as often, so it’s not as “hot.” This could be highly personalized content, things that were popular once but are less so now, or any other number of objects. When testing, try to figure out some way to look at the performance of long-tail content. Just because something is cacheable, it doesn’t mean the CDN has it at the very edge all the time. Testing long-tail content would expose this.

There’s lots more; we’re learning lessons about this stuff all the time, but these should be some good things to keep in mind.