Cache incorrectly serving old content under specific conditions


#1

I’m having a problem with cdn.netbsd.org which is served by fastly. My service (repology.org) fetches pkgsrc index regularly, and I’ve just discovered that the cdn serves a ~month old version to me most of the time, and just ocasionally serves the up to date file. Which is even more strange, I could only reproduce it by fetching with python requests module, and I could only reproduce it from the single host (194.67.206.223/2a02:f680:1:1100::7742, not sure which protocol it uses).

The details. Here’s what correct reply looks like:

% python3 -c "import requests; print(requests.get('https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/INDEX').headers)"
{'Server': 'bozohttpd/20170201', 'Last-Modified': 'Wed, 18 Apr 2018 11:45:52 GMT', 'Content-Type': 'text/plain', 'Cache-Control': 's-maxage: 3777', 'Content-Encoding': 'gzip', 'Via': '1.1 varnish, 1.1 varnish', 'Fastly-Debug-Digest': '7142a0418a63699aa4b1b34304a461b81acab5d8998ff7e16974810fcb47b76a', 'Content-Length': '2146819', 'Accept-Ranges': 'bytes', 'Date': 'Wed, 18 Apr 2018 13:09:32 GMT', 'Age': '897', 'Connection': 'keep-alive', 'X-Served-By': 'cache-sjc3125-SJC, cache-bma7020-BMA', 'X-Cache': 'HIT, HIT', 'X-Cache-Hits': '1, 4', 'X-Timer': 'S1524056972.247413,VS0,VE0', 'Vary': 'Accept-Encoding'}

Note the 15 minute age and 2146819 content length. Here’s what happens from the named host:

% python3 -c "import requests; print(requests.get('https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/INDEX').headers)"                                                                                                              
{'Server': 'bozohttpd/20170201', 'Last-Modified': 'Sun, 18 Mar 2018 11:14:35 GMT', 'Content-Type': 'text/plain', 'Cache-Control': 's-maxage: 3777', 'Content-Encoding': 'gzip', 'Via': '1.1 varnish, 1.1 varnish', 'Fastly-Debug-Digest': '7142a0418a63699aa4b1b34304a461b81acab5d8998ff7e16974810fcb47b76a', 'Content-Length': '2133937', 'Accept-Ranges': 'bytes', 'Date': 'Wed, 18 Apr 2018 13:11:01 GMT', 'Age': '2640410', 'Connection': 'keep-alive', 'X-Served-By': 'cache-sjc3127-SJC, cache-hhn1550-HHN', 'X-Cache': 'MISS, HIT', 'X-Cache-Hits': '0, 1', 'X-Timer': 'S1524057062.858942,VS0,VE3', 'Vary': 'Accept-Encoding'}

Note Age of 2640410 (that’s more than 30 days!) and Content-length of 2133937 (old smaller file).


#2

Hi Dmitry,

Thanks for reaching out.
It seems TTL( beresp.ttl ) is currently set longer than 30 days for this URL.
So it is possible to get the 30 days old object from the edge POP, and you might need to reach out to them to see why they’d like to serve the long TTL cache.

As far as I see, the different edge nodes have the different age objects at this moment and some of them have a different Content-Length. However, I don’t see anything specifically outstanding that hints the different cache nodes are sending different objects in the VCL. If this is not expected and they’d like to submit a ticket for us, we are more than happy to assist to sort this out.

Please let us know if you have any further question.
Thank you,
Hiro | Fastly Support


Cache incorrectly serving old content under specific conditions
#3

The moments when the cdn server the fresh file can be seen on this graph:

https://repology.org/graph/repo/pkgsrc_current/metapackages_total.svg

as spikes (newer file->more data).


#4

@AMDmi3 for the TTL we keep the object in the cache for as long as we can base on the rules a customer set either by using Cache-Control or VCL (Varnish Configuration Language) on a specific service. So to grab a new fresh object, it would require a user to request the object again after it expires. Or the object has some sort of ETAG or if modified since headers.

I recommend opening a ticket with us and/or NetBSD, there might be something in their VCL that could be changed to ensure you get the latest object. I think the communication and tracking over a ticket would be more beneficial for this issue than a community post.

Thanks,
Steven


#5

Thank you, I’ve written NetBSD guys.