Inconsistent perf to our CDN


#1

I have been looking at perf on https://meta.discourse.org

I have been noticing very uneven performance for the static CDN based assets.

For example:

https://meta-discourse.global.ssl.fastly.net/cdn_asset/meta_discourse_org/application-06deb287fb9f210ef9381e1168a3ecef.js?v=1&origin=https%3A%2F%2Fmeta.discourse.org

I often see timelines like this in chrome:

Even though:

X-Served-By:cache-syd1620-SYD

Which is in Sydney where I am located.

Do we have something misconfigured that is causing uneeded round trips back to our server for this content?

Here are some other traces I am picking:

890ms just to get a 302

When it is all working right stuff seems ultra fast, but in general we don’t seem to be hitting this case.

I am seeing way too much cache misses here:

Accept-Ranges:bytes
Access-Control-Allow-Origin:https://meta.discourse.org
Cache-Control:max-age=31557600, public
Connection:Keep-Alive
Content-Encoding:gzip
Content-Transfer-Encoding:binary
Content-Type:application/x-javascript
Date:Tue, 21 Oct 2014 00:19:33 GMT
Fastly-Restarts:1
Keep-Alive:timeout=10, max=42
Last-Modified:Mon, 20 Oct 2014 21:28:23 GMT
Server:nginx
Status:200 OK
Strict-Transport-Security:max-age=314159
Transfer-Encoding:chunked
Via:1.1 varnish
X-Cache:MISS
X-Cache-Hits:0
X-Content-Type-Options:nosniff
X-Frame-Options:SAMEORIGIN
X-Request-Id:209b9416-cf5c-495f-96b2-795a7024b020
X-Runtime:0.029540
X-Served-By:cache-syd1620-SYD
X-Timer:S1413850772.732731,VS0,VE347
X-UA-Compatible:IE=edge
X-XSS-Protection:1; mode=block

Any idea what is messed up with my config?


#2

Headers look good.

What happens when you make multiple requests? do you see the hit counter increment?

I also see a request that looks like this:
https://meta-discourse.global.ssl.fastly.net/cdn_asset/meta_discourse_org/application-06deb287fb9f210ef9381e1168a3ecef.js?v=1

The object itself is larger (> 1MB), so that can certainly affect things, but the caching should remove a lot of the problem. Do all of the static assets have the potential to be requested with a modifying query string on the end?


#3

Been debugging this all day, I think it is working a bit better now.

I found I was missing the Content-Length , was caching application/x-javascript as opposed to application/javascript and did not have the superflous Expires header.

Now that I added all this stuff it seems to be working as expected. Its hard to pin the culprit though.


#4

To answer this, actually no, that was a temporary hack we needed that I used to expire a bunch of header. Content itself is always the same.


#5

Download speeds still feel a bit slow despite getting hits now, for example from web page test it is taking us 708ms to download this 200kb file

URL: https://meta-discourse.global.ssl.fastly.net/cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1&origin=https%3A%2F%2Fmeta.discourse.org
Loaded By: https://meta.discourse.org/:55
Host: meta-discourse.global.ssl.fastly.net
Location: Fastly
Error/Status Code: 200
Client Port: 0
Start Offset: 0.885 s
DNS Lookup: 47 ms
Initial Connection: 49 ms
SSL Negotiation: 160 ms
Time to First Byte: 68 ms
Content Download: 708 ms
Bytes In (downloaded): 195.0 KB
Bytes Out (uploaded): 0.5 KB


Response Headers:

HTTP/1.1 200 OK
Date: Tue, 21 Oct 2014 05:30:01 GMT
Server: nginx
Content-Type: application/javascript
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Expires: Wed, 21 Oct 2015 05:29:09 GMT
Access-Control-Allow-Origin: https://meta.discourse.org
Last-Modified: Tue, 21 Oct 2014 05:25:57 GMT
Content-Transfer-Encoding: binary
Cache-Control: max-age=31557600, public
X-Request-Id: 187b6f06-d625-444b-a5f9-fcad235ae12c
X-Runtime: 0.032013
X-UA-Compatible: IE=edge
Content-Encoding: gzip
Strict-Transport-Security: max-age=314159
Content-Length: 198831
Accept-Ranges: bytes
Via: 1.1 varnish
Age: 53
X-Served-By: cache-jfk1020-JFK
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1413869401.906995,VS0,VE0
Keep-Alive: timeout=10, max=50
Connection: Keep-Alive


This feels a bit on the slow side for content moving from NY to Dullas, do you have IW set to 10 on all servers? (this should be default on all 3.0+ kernels)


#6

Hey-
one thing I notice in your config that could help:
You don’t have gzip enabled. usually pretty straightforward in the app to set up. your .js files could definitely get reduced in size significantly.

One of the things also to look at with webpagetest (we love them!), is that the default it to set up a restricted bw feed. I think the idea is to emulate home networks.

Here’s sample output from what I’m seeing:

$ curl -w “\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n” -svo /dev/null -HFastly-Debug:1 https://meta-discourse.global.ssl.fastly.net/cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1&origin=https%3A%2F%2Fmeta.discourse.org
[1] 77831

  • Hostname was NOT found in DNS cache
  • Trying 23.235.47.184…
  • Connected to meta-discourse.global.ssl.fastly.net (23.235.47.184) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: /opt/local/share/curl/curl-ca-bundle.crt
    CApath: none
  • SSLv3, TLS handshake, Client hello (1):
    } [data not shown]
  • SSLv3, TLS handshake, Server hello (2):
    { [data not shown]
  • SSLv3, TLS handshake, CERT (11):
    { [data not shown]
  • SSLv3, TLS handshake, Server key exchange (12):
    { [data not shown]
  • SSLv3, TLS handshake, Server finished (14):
    { [data not shown]
  • SSLv3, TLS handshake, Client key exchange (16):
    } [data not shown]
  • SSLv3, TLS change cipher, Client hello (1):
    } [data not shown]
  • SSLv3, TLS handshake, Finished (20):
    } [data not shown]
  • SSLv3, TLS change cipher, Client hello (1):
    { [data not shown]
  • SSLv3, TLS handshake, Finished (20):
    { [data not shown]
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  • Server certificate:
  • subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=*.a.ssl.fastly.net
  • start date: 2014-01-01 00:00:00 GMT
  • expire date: 2015-01-05 12:00:00 GMT
  • subjectAltName: meta-discourse.global.ssl.fastly.net matched
  • issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
  • SSL certificate verify ok.

GET /cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1 HTTP/1.1
User-Agent: curl/7.38.0
Host: meta-discourse.global.ssl.fastly.net
Accept: /

< HTTP/1.1 200 OK
< Date: Tue, 21 Oct 2014 06:16:01 GMT

  • Server nginx is not blacklisted
    < Server: nginx
    < Content-Type: application/javascript
    < Status: 200 OK
    < X-Frame-Options: SAMEORIGIN
    < X-XSS-Protection: 1; mode=block
    < X-Content-Type-Options: nosniff
    < Expires: Wed, 21 Oct 2015 06:15:34 GMT
    < Access-Control-Allow-Origin:
    < Last-Modified: Tue, 21 Oct 2014 05:25:57 GMT
    < Content-Transfer-Encoding: binary
    < Cache-Control: max-age=31557600, public
    < X-Request-Id: c97a7a12-2e8b-4e74-8d46-2aae54a9d1f5
    < X-Runtime: 0.035031
    < X-UA-Compatible: IE=edge
    < Strict-Transport-Security: max-age=314159
    < Content-Length: 651711
    < Accept-Ranges: bytes
    < Via: 1.1 varnish
    < Age: 27
    < X-Served-By: cache-sjc3131-SJC
    < X-Cache: HIT
    < X-Cache-Hits: 1
    < X-Timer: S1413872161.884353,VS0,VE2
    <
    { [data not shown]
  • Connection #0 to host meta-discourse.global.ssl.fastly.net left intact

Lookup time: 0.042
Connect time: 0.085
PreXfer time: 0.228
StartXfer time: 0.284

Total time: 1.335

This object has a content-length of 651711, so gzip would definitely help here.


#7

Thats odd, I am definitely seeing gzip enabled when I am testing. Are you sure you sent curl up “Accept-Encoding:gzip” when you did the test?


#8

Hi-
I’m sending the curl both with --compress and without and I get the same content length.

We do support gzipping at the edge, but it doesn’t seem to be configured for your site.

Here’s the curl with the accept-encoding header and the same content-length as without:

$ curl --compress -w "\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n" -svo /dev/null https://meta-discourse.global.ssl.fastly.net/cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1&origin=https%3A%2F%2Fmeta.discourse.org
[1] 78329
pwohlers@Peters-MacBook-Air:~
$ * Hostname was NOT found in DNS cache
*   Trying 199.27.79.185...
* Connected to meta-discourse.global.ssl.fastly.net (199.27.79.185) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
* 	 subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=*.a.ssl.fastly.net
* 	 start date: 2014-01-01 00:00:00 GMT
* 	 expire date: 2015-01-05 12:00:00 GMT
* 	 subjectAltName: meta-discourse.global.ssl.fastly.net matched
* 	 issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
* 	 SSL certificate verify ok.
> GET /cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1 HTTP/1.1
> User-Agent: curl/7.38.0
> Host: meta-discourse.global.ssl.fastly.net
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< Date: Tue, 21 Oct 2014 15:58:20 GMT
* Server nginx is not blacklisted
< Server: nginx
< Content-Type: application/javascript
< Status: 200 OK
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Expires: Wed, 21 Oct 2015 06:15:21 GMT
< Access-Control-Allow-Origin:
< Last-Modified: Tue, 21 Oct 2014 05:25:09 GMT
< Content-Transfer-Encoding: binary
< Cache-Control: max-age=31557600, public
< X-Request-Id: a32578b6-cd8e-46e7-91db-5290b78d4a5b
< X-Runtime: 0.033988
< X-UA-Compatible: IE=edge
< Strict-Transport-Security: max-age=314159
< Content-Length: 651711
< Accept-Ranges: bytes
< Via: 1.1 varnish
< Age: 34979
< X-Served-By: cache-lax1435-LAX
< X-Cache: HIT
< X-Cache-Hits: 1
< X-Timer: S1413907100.548417,VS0,VE3
<
{ [data not shown]
* Connection #0 to host meta-discourse.global.ssl.fastly.net left intact

Lookup time:	0.004
Connect time:	0.015
PreXfer time:	0.060
StartXfer time:	0.076

Total time:	0.249

[1]+  Done                    curl --compress -w "\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n" -svo /dev/null https://meta-discourse.global.ssl.fastly.net/cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1

#9

This is super strange, I am getting content length of 200k on both the source and origin as opposed to your 650k

MBA:~ sam$ curl --compress -w "\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n" -svo /dev/null https://meta-discourse.global.ssl.fastly.net/cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1&origin=https%3A%2F%2Fmeta.discourse.org
[1] 11728
MBA:~ sam$ * Hostname was NOT found in DNS cache
*   Trying 103.245.222.184...
* Connected to meta-discourse.global.ssl.fastly.net (103.245.222.184) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: *.a.ssl.fastly.net
* Server certificate: DigiCert High Assurance CA-3
* Server certificate: DigiCert High Assurance EV Root CA
> GET /cdn_asset/meta_discourse_org/vendor-134ec4c7a99c947213a44af37e9ce02f.js?v=1 HTTP/1.1
> User-Agent: curl/7.37.1
> Host: meta-discourse.global.ssl.fastly.net
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
< HTTP/1.1 200 OK
< Date: Tue, 21 Oct 2014 21:01:48 GMT
* Server nginx is not blacklisted
< Server: nginx
< Content-Type: application/javascript
< Status: 200 OK
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Expires: Wed, 21 Oct 2015 21:01:18 GMT
< Access-Control-Allow-Origin: 
< Last-Modified: Tue, 21 Oct 2014 20:08:02 GMT
< Content-Transfer-Encoding: binary
< Cache-Control: max-age=31557600, public
< X-Request-Id: 9c001fb8-e352-464a-9869-c7810bd7eac8
< X-Runtime: 0.034662
< X-UA-Compatible: IE=edge
< Content-Encoding: gzip
< Strict-Transport-Security: max-age=314159
< Content-Length: 198831
< Accept-Ranges: bytes
< Via: 1.1 varnish
< Age: 30
< X-Served-By: cache-syd1620-SYD
< X-Cache: HIT
< X-Cache-Hits: 1
< X-Timer: S1413925308.547378,VS0,VE0
< 
{ [data not shown]
* Connection #0 to host meta-discourse.global.ssl.fastly.net left intact

Lookup time:	0.069
Connect time:	0.085
PreXfer time:	0.278
StartXfer time:	0.306

Total time:	0.530

I can definitely confirm this is showing up compressed in dev tools as well.


#10

Because there’s no Vary header on accept-encoding, since I first requested the object uncompressed multiple times, I probably filled the cache-nodes with the uncompressed version.

I would bet that if you enabled gzip in the Fastly app, purged the object, that you would get compressed objects for the requests where you send an accept-encoding header and uncompressed when you don’t.

Right now, the hash key, without the vary header, will only store the first object it gets. subsequent requests will simply retrieve the object that was first requested…compressed or non-compressed.


#11

aha, I just added that header, will deploy shortly. I really don’t care about causing issues with ie6 and below :slight_smile:

Thanks heaps for pointing this out.


#12

well, we killed off SSLv3 a week or two ago, so that IE6 thing is probably pretty broken across the internet anyway. The other thing is that our gzipping implementation is smart and will not send compressed content to clients that don’t support it.