Load-testing a new origin stack by sending duplicating HTTP traffic to it


#1

I’d like to load-test a new origin stack, by duplicating all HTTP GET traffic sent to the old-origin stack.

At this point, ideally, the request to the new stack would be fire-and-forget - generate the traffic, forwarding auth cookies, but don’t wait for the response. The old origin stack would still be the one returning the response to the user.

Is it possible to do this with Fastly VCL?!


#2

Hi @rtyley,

Yes, you can do this, albeit without the async aspect. You will need to wait for the response from the new stack before sending the request to the old stack. You could also, for example, write a log line if the reported content size of the new stack’s response is different to the old stack.

Here’s an example of how to do something like what you want:

https://fiddle.fastlydemo.net/fiddle/e1a941d9

You might improve on this by only sending a portion of your traffic to the new stack, using our randomness functions, or ramp up gradually from 0 tom 100 percent.

Andrew


#3

One thing that might make this a little better for the end user, would be to set an ultra low time to first byte timeout on the origin you want to smoke test. That way Fastly can fail quickly, restart and go to the old origin and send the data on more quickly. This is an even better idea if the response and its data is not wanted from the smoke test.

The details of how to set the timeout are at https://docs.fastly.com/guides/debugging/changing-connection-timeouts-to-your-origin


#4

Thanks for your help with this - with your assistance I was able to come up with VCL that looked like this:

https://fiddle.fastlydemo.net/fiddle/6d80110f

…the one thing I didn’t understand was why return (pass); seems to be necessary - ideally, I would like both my backends to be getting the same quantity of requests, but returning pass on the TEST branch means that my TEST backend will receive requests for cached static assets that my PROD backend probably won’t…? I tried removing the return (pass); … but then no traffic was directed to my TEST backend :frowning:

In any case, this is probably good enough for me to be getting on with! Thanks again!


#5

If you remove the return(pass) then you are not terminating the recv function, and you will fall through to code that will set the backend to the default one. You can fix this by returning explictly but using return(lookup) instead of return(pass).

If you do this, then your test backend’s content will be in cache. Since the backend is not part of the cache key, we’ll match both test and prod content on the lookup. That’s why I thought it was easier not to cache test. But you’re right, and if you want the behaviour to be similar, you’ll need to add a request header to record which backend you are targeting:

set req.http.Backend-Mode = "test";

And then add a Vary header on the response, either from the backend, or in fetch:

set besp.http.Vary = "Backend-Mode";