Fastly configuration for JFrog artifactory

Network Services
21

@noguxun that makes complete sense! thanks for sending that over.

My final question is about the first request for an object (e.g one that doesn’t exist in cache). It’s significantly slower than the same request made directly to the resource. Do you know why this might be the case?

E.g request directly to the resource:

time wget --server-response -O /dev/null http://packages.adoptium.net/artifactory/rpm/rhel/9/x86_64/Packages/temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm
URL transformed to HTTPS due to an HSTS policy
--2023-09-04 10:16:03--  https://packages.adoptium.net/artifactory/rpm/rhel/9/x86_64/Packages/temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm
Resolving packages.adoptium.net (packages.adoptium.net)... 52.10.249.110, 35.160.231.47, 52.32.129.26
Connecting to packages.adoptium.net (packages.adoptium.net)|52.10.249.110|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 
  Date: Mon, 04 Sep 2023 09:16:04 GMT
  Transfer-Encoding: chunked
  Connection: keep-alive
  X-JFrog-Version: Artifactory/7.64.7 76407900
  X-Artifactory-Id: a89daeb8e3f1ab4ce0fc8c3abdab48f88990193c
  X-Artifactory-Node-Id: adoptium-artifactory-primary-0
  Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=4d7391c565df5b57&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGkaCXVzLXdlc3QtMiJGMEQCIHsa%2BcXnvwsiz2tdpcn2FMwDgymJKfOUIzKHxnLWEusAAiAUFCv5Eo96mGSoXwlwWGfisOhUXeXeVyvvg1mhgMzBxSqGBQhCEAIaDDk5OTkzNDc5MDQxNCIMkmk4ZYdlRt3R6NGJKuMENiM4Fp1R6ZkTHeJ7%2BUIqA86BApj34NWN6Tc9gIW4hQVcucx%2BXnFPrPhRQ502TU%2BmcgUq1kGZfB17BW32bFwxoJOBcYDfFdS9%2FqFe9tOZ1LM17qBPEfKuyCXxQl6rezIkwERB%2BcsmS8pH6NFlJAiXM58JY2QtX1zuHM0LoSM6s971fkX7Q%2BQGoq43DnqdLJyNRZC%2BBFwe%2BTCS9v8QM12KF3cjKRhcltYkB%2Fb5hzkUGBQRko5iAiEU%2FLqJeOWuNQl430rpVLQbAP3I2hpvsPVL5mmCBimnXRQsnXhvFNroIShSBxwNYghSvbgbN%2FJYfTbK11hWMxKUB0X1nwjrzg7RBHqNLXTz7NL60OAfspdzh2DtmqFpsdCpJLbh2Aqt8YFV0z0IRjv5QasoL2ulcaz8Vam7hoVM0xRZbA6MC%2BZOUytkSQzu3G6kNq7Kzg44YRFdDuCzWa9c4GiojstFsuRsjJv6dgM0mU13C9HFLuodLIgHBPlL9FvP276p9lRUGLJAFSL0Aj%2Bb8WR7ZJCRibj9D63ky%2Btq0sikNXYrtyDA9Ck8K3l9RxAU8XgqW%2FOynDvQ2ciPvklFArRNVHnk6DBraxa4pUcAUijuLuRVeYeg2eZyHQQKxNfMfjpFr%2BRucLmXI0DoNIpAQaKl%2B13JSl2k%2BgVJnPdhzEty2ey9Y1huHqq4UQu8o%2F3X%2Bk7hvte7sxPfRfA6iggs%2B%2F4cA4Ch6%2FLFrSScio6ga%2FuyF7QP4sC68oYOBsDpwjfOxEsVXA36VkVrOe0IICIxysn2GzimeZMTlZ00GLxFLDwSQuAv7xScGyckTl4woLjWpwY6mwEEszGQ0U0o%2BKKQmBdMTRBPTWDVQAgZWJx0HR0FBr2bmLwBnRg4KUjYfDOiaQRWurSgVdaCGZIYsDaRzR%2FtxUPQchlFh0l4i7GXt1rrPwFMhFSRljlAC%2FkMHwtyDXnVxftERri21pAz8DhbTTwPsCtLWkQibwAI2IS3Bi7KB%2Bd553B3vyesJ5B320fxf3S6Awe3IqDOLrjaU1MJOQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230904T091604Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHPR5LHPJQ%2F20230904%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=dd40fed83b76316f6bcedc32c7bcbe72b5ef4647e650393ac5ac0df26cc161f8
  Strict-Transport-Security: max-age=31536000; includeSubDomains
  X-Request-ID: bd0645bbce2f68b18320ff185c8dffe8
Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=4d7391c565df5b57&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGkaCXVzLXdlc3QtMiJGMEQCIHsa%2BcXnvwsiz2tdpcn2FMwDgymJKfOUIzKHxnLWEusAAiAUFCv5Eo96mGSoXwlwWGfisOhUXeXeVyvvg1mhgMzBxSqGBQhCEAIaDDk5OTkzNDc5MDQxNCIMkmk4ZYdlRt3R6NGJKuMENiM4Fp1R6ZkTHeJ7%2BUIqA86BApj34NWN6Tc9gIW4hQVcucx%2BXnFPrPhRQ502TU%2BmcgUq1kGZfB17BW32bFwxoJOBcYDfFdS9%2FqFe9tOZ1LM17qBPEfKuyCXxQl6rezIkwERB%2BcsmS8pH6NFlJAiXM58JY2QtX1zuHM0LoSM6s971fkX7Q%2BQGoq43DnqdLJyNRZC%2BBFwe%2BTCS9v8QM12KF3cjKRhcltYkB%2Fb5hzkUGBQRko5iAiEU%2FLqJeOWuNQl430rpVLQbAP3I2hpvsPVL5mmCBimnXRQsnXhvFNroIShSBxwNYghSvbgbN%2FJYfTbK11hWMxKUB0X1nwjrzg7RBHqNLXTz7NL60OAfspdzh2DtmqFpsdCpJLbh2Aqt8YFV0z0IRjv5QasoL2ulcaz8Vam7hoVM0xRZbA6MC%2BZOUytkSQzu3G6kNq7Kzg44YRFdDuCzWa9c4GiojstFsuRsjJv6dgM0mU13C9HFLuodLIgHBPlL9FvP276p9lRUGLJAFSL0Aj%2Bb8WR7ZJCRibj9D63ky%2Btq0sikNXYrtyDA9Ck8K3l9RxAU8XgqW%2FOynDvQ2ciPvklFArRNVHnk6DBraxa4pUcAUijuLuRVeYeg2eZyHQQKxNfMfjpFr%2BRucLmXI0DoNIpAQaKl%2B13JSl2k%2BgVJnPdhzEty2ey9Y1huHqq4UQu8o%2F3X%2Bk7hvte7sxPfRfA6iggs%2B%2F4cA4Ch6%2FLFrSScio6ga%2FuyF7QP4sC68oYOBsDpwjfOxEsVXA36VkVrOe0IICIxysn2GzimeZMTlZ00GLxFLDwSQuAv7xScGyckTl4woLjWpwY6mwEEszGQ0U0o%2BKKQmBdMTRBPTWDVQAgZWJx0HR0FBr2bmLwBnRg4KUjYfDOiaQRWurSgVdaCGZIYsDaRzR%2FtxUPQchlFh0l4i7GXt1rrPwFMhFSRljlAC%2FkMHwtyDXnVxftERri21pAz8DhbTTwPsCtLWkQibwAI2IS3Bi7KB%2Bd553B3vyesJ5B320fxf3S6Awe3IqDOLrjaU1MJOQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230904T091604Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHPR5LHPJQ%2F20230904%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=dd40fed83b76316f6bcedc32c7bcbe72b5ef4647e650393ac5ac0df26cc161f8 [following]
--2023-09-04 10:16:04--  https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=4d7391c565df5b57&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGkaCXVzLXdlc3QtMiJGMEQCIHsa%2BcXnvwsiz2tdpcn2FMwDgymJKfOUIzKHxnLWEusAAiAUFCv5Eo96mGSoXwlwWGfisOhUXeXeVyvvg1mhgMzBxSqGBQhCEAIaDDk5OTkzNDc5MDQxNCIMkmk4ZYdlRt3R6NGJKuMENiM4Fp1R6ZkTHeJ7%2BUIqA86BApj34NWN6Tc9gIW4hQVcucx%2BXnFPrPhRQ502TU%2BmcgUq1kGZfB17BW32bFwxoJOBcYDfFdS9%2FqFe9tOZ1LM17qBPEfKuyCXxQl6rezIkwERB%2BcsmS8pH6NFlJAiXM58JY2QtX1zuHM0LoSM6s971fkX7Q%2BQGoq43DnqdLJyNRZC%2BBFwe%2BTCS9v8QM12KF3cjKRhcltYkB%2Fb5hzkUGBQRko5iAiEU%2FLqJeOWuNQl430rpVLQbAP3I2hpvsPVL5mmCBimnXRQsnXhvFNroIShSBxwNYghSvbgbN%2FJYfTbK11hWMxKUB0X1nwjrzg7RBHqNLXTz7NL60OAfspdzh2DtmqFpsdCpJLbh2Aqt8YFV0z0IRjv5QasoL2ulcaz8Vam7hoVM0xRZbA6MC%2BZOUytkSQzu3G6kNq7Kzg44YRFdDuCzWa9c4GiojstFsuRsjJv6dgM0mU13C9HFLuodLIgHBPlL9FvP276p9lRUGLJAFSL0Aj%2Bb8WR7ZJCRibj9D63ky%2Btq0sikNXYrtyDA9Ck8K3l9RxAU8XgqW%2FOynDvQ2ciPvklFArRNVHnk6DBraxa4pUcAUijuLuRVeYeg2eZyHQQKxNfMfjpFr%2BRucLmXI0DoNIpAQaKl%2B13JSl2k%2BgVJnPdhzEty2ey9Y1huHqq4UQu8o%2F3X%2Bk7hvte7sxPfRfA6iggs%2B%2F4cA4Ch6%2FLFrSScio6ga%2FuyF7QP4sC68oYOBsDpwjfOxEsVXA36VkVrOe0IICIxysn2GzimeZMTlZ00GLxFLDwSQuAv7xScGyckTl4woLjWpwY6mwEEszGQ0U0o%2BKKQmBdMTRBPTWDVQAgZWJx0HR0FBr2bmLwBnRg4KUjYfDOiaQRWurSgVdaCGZIYsDaRzR%2FtxUPQchlFh0l4i7GXt1rrPwFMhFSRljlAC%2FkMHwtyDXnVxftERri21pAz8DhbTTwPsCtLWkQibwAI2IS3Bi7KB%2Bd553B3vyesJ5B320fxf3S6Awe3IqDOLrjaU1MJOQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230904T091604Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHPR5LHPJQ%2F20230904%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=dd40fed83b76316f6bcedc32c7bcbe72b5ef4647e650393ac5ac0df26cc161f8
Resolving jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)... 52.92.148.33, 52.92.132.25, 52.218.220.51, ...
Connecting to jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)|52.92.148.33|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  x-amz-id-2: EvMS1bjdsN6Zee5Rpxh6A1p8MltZS1IB/RIoRa+jouFwmAQYYfk2Y6XxVKijw/S0UPHFNpDAUBc=
  x-amz-request-id: XD2MN0JWGHY911MH
  Date: Mon, 04 Sep 2023 09:16:06 GMT
  x-amz-replication-status: COMPLETED
  Last-Modified: Fri, 29 Jul 2022 13:41:13 GMT
  ETag: "73b91cbf2f960e8567324c42668d8de7"
  x-amz-server-side-encryption: AES256
  x-amz-version-id: 1Tt.ErWWJlbhN583fT8a.YVtm1awUeIB
  Content-Disposition: attachment;filename="temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm"
  Accept-Ranges: bytes
  Content-Type: application/x-rpm
  Server: AmazonS3
  Content-Length: 165558717
Length: 165558717 (158M) [application/x-rpm]
Saving to: ‘/dev/null’

/dev/null                                                              100%[=========================================================================================================================================================================>] 157.89M  1.43MB/s    in 69s     

2023-09-04 10:17:14 (2.28 MB/s) - ‘/dev/null’ saved [165558717/165558717]

wget --server-response -O /dev/null   0.34s user 0.57s system 1% cpu 1:10.59 total

Vs via Fastly (no cache):

time wget --server-response -O /dev/null http://packages.adoptium.net.global.prod.fastly.net/artifactory/rpm/rhel/9/x86_64/Packages/temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm
--2023-09-04 10:10:20--  http://packages.adoptium.net.global.prod.fastly.net/artifactory/rpm/rhel/9/x86_64/Packages/temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm
Resolving packages.adoptium.net.global.prod.fastly.net (packages.adoptium.net.global.prod.fastly.net)... 151.101.0.249, 151.101.64.249, 151.101.128.249, ...
Connecting to packages.adoptium.net.global.prod.fastly.net (packages.adoptium.net.global.prod.fastly.net)|151.101.0.249|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Connection: keep-alive
  Content-Length: 165558717
  x-amz-id-2: NzWLplll4gCNJJsXXG5zQ+5eonzaFuRmrBoat2eQfBUZ2fiUkvuOHiQoz1ljwl0YOQgUjdVuiH4=
  x-amz-request-id: K7K3CB6TFM0S2Q6V
  x-amz-replication-status: COMPLETED
  Last-Modified: Fri, 29 Jul 2022 13:41:13 GMT
  ETag: "73b91cbf2f960e8567324c42668d8de7"
  x-amz-server-side-encryption: AES256
  x-amz-version-id: 1Tt.ErWWJlbhN583fT8a.YVtm1awUeIB
  Content-Disposition: attachment;filename="temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm"
  Content-Type: application/x-rpm
  Server: AmazonS3
  Cache-Control: public, max-age=15780096.000
  Fastly-Restarts: 1
  Accept-Ranges: bytes
  Date: Mon, 04 Sep 2023 09:10:21 GMT
  Via: 1.1 varnish
  Age: 0
  X-Served-By: cache-lcy-eglc8600078-LCY
  X-Cache: MISS
  X-Cache-Hits: 0
  X-Timer: S1693818620.145221,VS0,VE1271
Length: 165558717 (158M) [application/x-rpm]
Saving to: ‘/dev/null’

/dev/null                                                              100%[=========================================================================================================================================================================>] 157.89M   642KB/s    in 4m 52s  

2023-09-04 10:15:13 (554 KB/s) - ‘/dev/null’ saved [165558717/165558717]

wget --server-response -O /dev/null   0.10s user 0.59s system 0% cpu 4:53.10 total
22

@gdams When segmented caching is enabled, Fastly grabs content from the origin block by block (using HTTP range request) in sequence.
The default block size is only 1M.
So, for a 100M file, there could be some non-neglectable overhead.
One solution to mitigate the overhead is to set a larger block size, like below

set req.enable_segmented_caching = true;
# set the block size to 4M
set segmented_caching.block_size = 4194304;

I do not have a precise number for you, but 4M or 8M is commonly seen.

If block size changes, the cache key will also change, which means the existing cached object will be invalidated.

23

@noguxun that’s perfect! I’ve set it to 8M and it’s now comparable to a direct fetch. Thanks so much for your help getting this all setup! You’ve been most useful!

24

Going back to the skipping cache for certain paths. I’ve added the following block to the bottom of my vcl_fetch file:

if (beresp.http.Cache-Control ~ "no-store") {
    set beresp.cacheable = false;
    return(deliver);
}

The idea being to skip the cache on anything that has Cache-Control no-store. But after activating these changes I still see Cache hits in the headers. Any ideas?

An example URL is http://packages.adoptium.net.global.prod.fastly.net/artifactory/deb/dists/jammy/Release

25

Actually I take this back, I now seem to be getting consistent cache misses so I think we’re all good

26

@gdams , Glad to hear it helps and see you are making good progress!

Apology that I forgot to mention that we had better return (pass) instead of (deliver) .
The code should look like below:

if (beresp.http.Cache-Control ~ "no-store") {
    set beresp.cacheable = false;
    return(pass);
}

return (pass) will make concurrent requests to the object not waiting on Origin’s response. See more at
https://developer.fastly.com/learning/concepts/edge-state/cache/request-collapsing/

A similar piece of code could also be found in the generated VCL

  if (beresp.http.Cache-Control ~ "private") {
    set req.http.Fastly-Cachetype = "PRIVATE";
    return (pass);
  }

And Fastly recommended Boilerplate custom VCL
https://developer.fastly.com/learning/vcl/using/#custom-vcl

  # By default we set a TTL based on the `Cache-Control` header but we don't parse additional directives
  # like `private` and `no-store`. Private in particular should be respected at the edge:
  if (beresp.http.Cache-Control ~ "(?:private|no-store)") {
    return(pass);
  }
1 Like
27

@gdams , sorry, I made a mistake, to make the service more performant, we need to set the code below.

if (beresp.http.Cache-Control ~ "no-store") {
    return(pass);
}

This is the only way to create a cacheable hit-for-pass object that will not make requests to wait.

To learn more, see:

https://developer.fastly.com/learning/concepts/edge-state/cache/request-collapsing/

especially

https://developer.fastly.com/learning/concepts/edge-state/cache/request-collapsing/#hit-for-pass

our previous code is exactly what the above page are trying to warn us

WARNING:: If a request is subject to request collapsing, and the origin response is not cacheable, then the response cannot be used to satisfy queued requests, AND we also cannot create a hit-for-pass marker. In this situation the next request in the queue will be sent to origin and the remaining requests will form a new queue, resulting in the requests being sent consecutively, not concurrently. In some cases this can create extreme response times of several minutes.

To avoid this bottleneck behavior, ensure that wherever possible, the request skips the cache entirely, and otherwise, that responses are always cacheable. When you don't want to actually cache them, pass on response (in VCL services return(pass) in vcl_fetch).
28

here is my vcl_fetch function:

# Tag the response so that we can track whether it came from a
# customer origin (and not a Fastly shield POP)
set beresp.http.redirectchase_isorigin = req.backend.is_origin;
set beresp.do_stream = true;

if (beresp.http.Cache-Control ~ "(?:private|no-store)") {
  return(pass);
}

if (beresp.http.Content-Type ~ "^(application/x-rpm|application/x-debian-package|application/vnd.android.package-archive)") {
        set beresp.ttl = 15780096s;  # 6 months
        set beresp.http.Cache-Control = "public, max-age=" + beresp.ttl;
}
29

All looks good to me :star_struck:

1 Like
30

hey @noguxun,

we’ve been running our instance in production for 2 weeks now and mostly it seems to be okay although I’m getting a few people reporting issues where a failed download gets cached in fastly and then the CDN continues to serve up a partial binary or a broken webpage. Is there something missing in our config that says “do not cache the result if the request failed?”

31

E.g here is an example of a request serving HTML rather than a binary. The LCY server was working at this time but the LHR one wasn’t. It’s not clear to me what is happening here.

 wget --server-response -O /dev/null https://packages.adoptium.net/artifactory/rpm/rhel/7/x86_64/Packages/temurin-17-jdk-17.0.6.0.0.10-3.x86_64.rpm
--2023-10-13 08:44:11--  https://packages.adoptium.net/artifactory/rpm/rhel/7/x86_64/Packages/temurin-17-jdk-17.0.6.0.0.10-3.x86_64.rpm
Resolving packages.adoptium.net (packages.adoptium.net)... 151.101.3.42, 151.101.67.42, 151.101.131.42, ...
Connecting to packages.adoptium.net (packages.adoptium.net)|151.101.3.42|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Connection: keep-alive
  Content-Length: 3029
  Content-Type: text/html; charset=UTF-8
  Cache-Control: public, max-age=0
  Content-Security-Policy: img-src 'self' data: getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com heapanalytics.com; script-src 'self' 'unsafe-eval' cdn.heapanalytics.com heapanalytics.com www.google-analytics.com youtube.com www.youtube.com https://js.driftt.com https://widget.drift.com fast.appcues.com api.appcues.net https://static.hotjar.com https://script.hotjar.com 'unsafe-inline' api.appcues.net producttourtool.jfrog.io getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; frame-src youtube.com www.youtube.com js.driftt.com widget.drift.com fast.appcues.com producttourtool.jfrog.io https://www.youtube-nocookie.com https://player.vimeo.com https://vars.hotjar.com https://player.vimeo.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; media-src js.driftt.com; font-src 'self' data: https://heapanalytics.com https://fonts.gstatic.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com https://script.hotjar.com; base-uri 'self'; style-src 'self' 'unsafe-inline' https://heapanalytics.com fast.appcues.com producttourtool.jfrog.io api.appcues.net https://fonts.googleapis.com https://fonts.google.com https://static.hotjar.com https://script.hotjar.com https://fonts.google.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; form-action 'self'; frame-ancestors 'self'
  Etag: W/"bd5-18a80f098b0"
  Feature-Policy: geolocation 'none';microphone 'none';camera 'none';payment 'none'
  Last-Modified: Sun, 10 Sep 2023 21:12:46 GMT
  Referrer-Policy: no-referrer
  X-Content-Security-Policy: img-src 'self' data: getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com heapanalytics.com; script-src 'self' 'unsafe-eval' cdn.heapanalytics.com heapanalytics.com www.google-analytics.com youtube.com www.youtube.com https://js.driftt.com https://widget.drift.com fast.appcues.com api.appcues.net https://static.hotjar.com https://script.hotjar.com 'unsafe-inline' api.appcues.net producttourtool.jfrog.io getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; frame-src youtube.com www.youtube.com js.driftt.com widget.drift.com fast.appcues.com producttourtool.jfrog.io https://www.youtube-nocookie.com https://player.vimeo.com https://vars.hotjar.com https://player.vimeo.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; media-src js.driftt.com; font-src 'self' data: https://heapanalytics.com https://fonts.gstatic.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com https://script.hotjar.com; base-uri 'self'; style-src 'self' 'unsafe-inline' https://heapanalytics.com fast.appcues.com producttourtool.jfrog.io api.appcues.net https://fonts.googleapis.com https://fonts.google.com https://static.hotjar.com https://script.hotjar.com https://fonts.google.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; form-action 'self'; frame-ancestors 'self'
  X-Content-Type-Options: nosniff
  X-Frame-Options: SAMEORIGIN
  X-Ratelimit-Remaining: 49
  X-Webkit-Csp: img-src 'self' data: getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com heapanalytics.com; script-src 'self' 'unsafe-eval' cdn.heapanalytics.com heapanalytics.com www.google-analytics.com youtube.com www.youtube.com https://js.driftt.com https://widget.drift.com fast.appcues.com api.appcues.net https://static.hotjar.com https://script.hotjar.com 'unsafe-inline' api.appcues.net producttourtool.jfrog.io getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; frame-src youtube.com www.youtube.com js.driftt.com widget.drift.com fast.appcues.com producttourtool.jfrog.io https://www.youtube-nocookie.com https://player.vimeo.com https://vars.hotjar.com https://player.vimeo.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; media-src js.driftt.com; font-src 'self' data: https://heapanalytics.com https://fonts.gstatic.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com https://script.hotjar.com; base-uri 'self'; style-src 'self' 'unsafe-inline' https://heapanalytics.com fast.appcues.com producttourtool.jfrog.io api.appcues.net https://fonts.googleapis.com https://fonts.google.com https://static.hotjar.com https://script.hotjar.com https://fonts.google.com getbeamer.com static.getbeamer.com app.getbeamer.com functions.getbeamer.com backend.getbeamer.com push.getbeamer.com; form-action 'self'; frame-ancestors 'self'
  X-Xss-Protection: 1; mode=block
  X-Request-ID: 33cd806d10cce2910393ba221eac13d3
  Fastly-Restarts: 1
  Accept-Ranges: bytes
  Date: Fri, 13 Oct 2023 07:44:12 GMT
  Via: 1.1 varnish
  Age: 0
  X-Served-By: cache-lhr7328-LHR
  X-Cache: MISS
  X-Cache-Hits: 0
  X-Timer: S1697183052.641681,VS0,VE641
  Vary: Accept-Encoding
  Strict-Transport-Security: max-age=31557600
Length: 3029 (3.0K) [text/html]
Saving to: ‘/dev/null’

/dev/null                        100%[==========================================================>]   2.96K  --.-KB/s    in 0s      

2023-10-13 08:44:12 (12.9 MB/s) - ‘/dev/null’ saved [3029/3029]
32

Navigating to the URL in a browser seems to suggest that it’s throwing a 501 error rather than dropping the cache and trying to get the binary from S3.

Screenshot 2023-10-13 at 08.49.08

33

We reproduced similar issue, I am asking internal team about it.

34

501 is likely to be generated by the origin, your service’s configuration does not generate 501 error.

35

@noguxun is there a chance we could setup a support call at some point to work through this problem?

36

Hi @gdams

Absolutely! Please reach out to your account manager (or contact support@fastly.com) and pass them this thread link :+1:

37

I’m working on exactly the same topic now. @noguxun you mentioned this configuration:

if (req.url.path == "/some_path" && beresp.status == 302) {
  set beresp.cacheable = false;
}

All large files from JFrog are returning 302. Conceptually I am trying to understand if any caching would be occurring.

Step 1: The first reply from jfrog is a 302 redirect, and it has many query parameters: ?X-Amz-Date=20240105T202841Z&X-Amz-SignedHeaders=host&X-Amz-Expires=30&X-Amz-Credential=ASIA2QE4OP45I%2F20240105%2Fus-west-2%2Fs3%2Faws4_request&X-Amz Signature=6d76891827b9ce7a6195c1a5719357a9472aa8e578f645

It would not be cached because it’s a 302 and we set beresp.cacheable = false; in the VCL, correct?

Step 2: The redirect to AWS S3 is followed. The large file is retrieved from S3. Because the cache key includes a long query string with tokens and expiration timeouts, and the query string changes very frequently, wouldn’t that also prevent caching?

Does beresp.cacheable = false; apply to both steps 1 and 2 ?

Due to these multiple reasons, it seems like “caching” would not happen, and requests would just be passed through to the origin. Is that the right interpretation?

Update: This is with “redirect chasing” in place. A concern is, even with “redirect chasing” the query string on the file is very long, and constantly changing.

38

I suspect your analysis is correct; that redirect URL is time-limited and signed because Artifactory has to ensure that users cannot go around it to access content that they would not otherwise have permission to access.

Effectively caching that content will be challenging, I believe.

1 Like