Fastly configuration for JFrog artifactory

Network Services
1

Hi there,

I’m trying to setup Fastly to wrap my Artifactory service (adoptium.jfrog.io).

Initially, the setup seemed quite straight forward (I setup the domain name and I setup the host as the URL of the jfrog instance. All was working very well until I realised that Fastly is caching the S3 redirect which expires after 15 minutes. I had a quick read through this guide: Amazon S3, but I couldn’t work out how to get this working.

Any help is most appreciated!

Thanks

2

how to get this working.

What is the expected behavior?

If you would like to have the redirect (status 301) not cached, you can add code in vcl_fetch to mark it non-cachable, like in Fiddle - Fastly

if (beresp.status == 301) {
  log "redirect";
  set beresp.cacheable = false;
}
3

If you would like to follow the redirect and get the content from S3 again. There is an example here https://developer.fastly.com/solutions/examples/follow-redirects-at-the-edge/

4

I would like to have Fastly cache the actual artefact and prevent it from being fetched again from the expired S3 presigned URL. How would I configure it this way?

5

Then https://developer.fastly.com/solutions/examples/follow-redirects-at-the-edge/ is the example to refer to.

the code is easier to read in this fiddle.
https://fiddle.fastly.dev/fiddle/4eb38392

6

Thanks fro providing that fiddle! I’ve made those changes but I’m still not convinced that Fastly is serving it up using a cache? Here is a wget showing the location:

➜  ~ wget --server-response -O /dev/null http://packages.adoptium.net.global.prod.fastly.net/artifactory/rpm/rhel/9/x86_64/Packages/temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm
--2023-08-31 09:18:28--  http://packages.adoptium.net.global.prod.fastly.net/artifactory/rpm/rhel/9/x86_64/Packages/temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm
Resolving packages.adoptium.net.global.prod.fastly.net (packages.adoptium.net.global.prod.fastly.net)... 151.101.0.249, 151.101.64.249, 151.101.128.249, ...
Connecting to packages.adoptium.net.global.prod.fastly.net (packages.adoptium.net.global.prod.fastly.net)|151.101.0.249|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 Found
  Connection: keep-alive
  Content-Length: 0
  X-JFrog-Version: Artifactory/7.64.7 76407900
  X-Artifactory-Id: a89daeb8e3f1ab4ce0fc8c3abdab48f88990193c
  X-Artifactory-Node-Id: adoptium-artifactory-primary-0
  Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=b24d58f3833522ce&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAgaCXVzLXdlc3QtMiJHMEUCIQDQLLJF9rU075%2B%2FLOguynzdDrPqMwu1JERwgQyrTZzLHAIgFKdJxwzuRz8lIGWq1m8NdjsInLuvsPgVs4ASIBJkvhAqjwUI0f%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw5OTk5MzQ3OTA0MTQiDN13GplsgH1IfJGbqirjBIlAS%2FfXXkIN33YiiwvlUM5mePtRu5nwWyTZ86QBHh8FYH4Dh3D6j8qOFFKWy10N83UavTCMYEhiz%2FwqokfQ6HXhb%2F6H6%2FOT5k6Hn2UEmyFORSzrLtR8MsrC%2FI4ly6DwzDldhhf7Wk8Xa1Y05npwkMGgqoyYMhJ1%2F9idr5EI8bWXf80mnlEiI4j1vilEu7u7kI%2BTIHaJNIGAt%2Fhm5R0OhbaLQNukTreGrdLaeH2SXmEKdZbM%2Fev57oNiHIEzXiuWZnlFj7udrMkGghfv0IYcsZl5Bv65qA3FhC6stWdRs6kRG6kSZJibyM28lHn8gHgoWdRa71xbwMenb4Fsn%2FnNp0RD%2BXxJhvS%2FklDkFP6t28aE0Cu2VprXnO%2FAIM2oQl7CS4xtCx5JwjzwvsuA6oX3s8%2FAAUg5ViSmRgbsmphzXgTNIeMZ3vm8FVla7Xa25UIq5U2mvnwT2vFSwFf0i5ZuTzpZH3wWKmXZcrLKHOYyX2DyV%2FsSvj8qFt1Md1glGSaqhfG0gqev1LczPYyBqFnykyYophVo4G7IYjuuRoyh5FpMWl0pTqUPkLEZDJZJyWAEwETatPtFRwWA8GI1VStGGvLcnd4eDGnBmUg1jq7PfN4F4Lyp%2BJ0gld166BeXTX5gfhL0zLtzUNRdfeKNNIlTLEVbaQqbQMHOM9y%2BCwDsyIuJU%2B2VO4yDOKf7GViO09GeWPiI3tX8KNUtGa59YZlrq8Ggm5%2FegtA4yQhWSG4RlHqDIfutCaHAbrZfBE%2FPIHEkF99%2BlaIeraQ1CDpzJheHCMVc83iGafLwPdLx%2FCiB%2BGme1r9IMJOSwacGOpoB774xgnK5YeLk%2Fhu7nVdQ7Y6dXdRITSGqQZmnMfDZnZU9FNMVDMUlVkPIXiTHujijtyyNL7SDKkpXH7rs1pBr5WBZ%2FexqzRM3c0rSN0dGSVhqTt1eAQp1BRzC5TT1iwtblspC8r1AoTbTZlO%2BZ10w5b%2BlxBFSdiK2qAMTcU9omzziBwRhTtzBu%2FpRbQHTIZHv2GGGSZflKlTCdg%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230831T081608Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHIN2XLSUQ%2F20230831%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=15008cc2e1362cc957948780d86d15cbfcf8beaf69e854965059b12b015abdf6
  Strict-Transport-Security: max-age=31536000; includeSubDomains
  X-Request-ID: 31c18ba5942f746b2612a5d97917f6cd
  Accept-Ranges: bytes
  Date: Thu, 31 Aug 2023 08:18:28 GMT
  Via: 1.1 varnish
  Age: 140
  X-Served-By: cache-lcy-eglc8600060-LCY
  X-Cache: HIT
  X-Cache-Hits: 1
  X-Timer: S1693469908.152477,VS0,VE1
Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=b24d58f3833522ce&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAgaCXVzLXdlc3QtMiJHMEUCIQDQLLJF9rU075%2B%2FLOguynzdDrPqMwu1JERwgQyrTZzLHAIgFKdJxwzuRz8lIGWq1m8NdjsInLuvsPgVs4ASIBJkvhAqjwUI0f%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw5OTk5MzQ3OTA0MTQiDN13GplsgH1IfJGbqirjBIlAS%2FfXXkIN33YiiwvlUM5mePtRu5nwWyTZ86QBHh8FYH4Dh3D6j8qOFFKWy10N83UavTCMYEhiz%2FwqokfQ6HXhb%2F6H6%2FOT5k6Hn2UEmyFORSzrLtR8MsrC%2FI4ly6DwzDldhhf7Wk8Xa1Y05npwkMGgqoyYMhJ1%2F9idr5EI8bWXf80mnlEiI4j1vilEu7u7kI%2BTIHaJNIGAt%2Fhm5R0OhbaLQNukTreGrdLaeH2SXmEKdZbM%2Fev57oNiHIEzXiuWZnlFj7udrMkGghfv0IYcsZl5Bv65qA3FhC6stWdRs6kRG6kSZJibyM28lHn8gHgoWdRa71xbwMenb4Fsn%2FnNp0RD%2BXxJhvS%2FklDkFP6t28aE0Cu2VprXnO%2FAIM2oQl7CS4xtCx5JwjzwvsuA6oX3s8%2FAAUg5ViSmRgbsmphzXgTNIeMZ3vm8FVla7Xa25UIq5U2mvnwT2vFSwFf0i5ZuTzpZH3wWKmXZcrLKHOYyX2DyV%2FsSvj8qFt1Md1glGSaqhfG0gqev1LczPYyBqFnykyYophVo4G7IYjuuRoyh5FpMWl0pTqUPkLEZDJZJyWAEwETatPtFRwWA8GI1VStGGvLcnd4eDGnBmUg1jq7PfN4F4Lyp%2BJ0gld166BeXTX5gfhL0zLtzUNRdfeKNNIlTLEVbaQqbQMHOM9y%2BCwDsyIuJU%2B2VO4yDOKf7GViO09GeWPiI3tX8KNUtGa59YZlrq8Ggm5%2FegtA4yQhWSG4RlHqDIfutCaHAbrZfBE%2FPIHEkF99%2BlaIeraQ1CDpzJheHCMVc83iGafLwPdLx%2FCiB%2BGme1r9IMJOSwacGOpoB774xgnK5YeLk%2Fhu7nVdQ7Y6dXdRITSGqQZmnMfDZnZU9FNMVDMUlVkPIXiTHujijtyyNL7SDKkpXH7rs1pBr5WBZ%2FexqzRM3c0rSN0dGSVhqTt1eAQp1BRzC5TT1iwtblspC8r1AoTbTZlO%2BZ10w5b%2BlxBFSdiK2qAMTcU9omzziBwRhTtzBu%2FpRbQHTIZHv2GGGSZflKlTCdg%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230831T081608Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHIN2XLSUQ%2F20230831%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=15008cc2e1362cc957948780d86d15cbfcf8beaf69e854965059b12b015abdf6 [following]
--2023-08-31 09:18:28--  https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=b24d58f3833522ce&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAgaCXVzLXdlc3QtMiJHMEUCIQDQLLJF9rU075%2B%2FLOguynzdDrPqMwu1JERwgQyrTZzLHAIgFKdJxwzuRz8lIGWq1m8NdjsInLuvsPgVs4ASIBJkvhAqjwUI0f%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw5OTk5MzQ3OTA0MTQiDN13GplsgH1IfJGbqirjBIlAS%2FfXXkIN33YiiwvlUM5mePtRu5nwWyTZ86QBHh8FYH4Dh3D6j8qOFFKWy10N83UavTCMYEhiz%2FwqokfQ6HXhb%2F6H6%2FOT5k6Hn2UEmyFORSzrLtR8MsrC%2FI4ly6DwzDldhhf7Wk8Xa1Y05npwkMGgqoyYMhJ1%2F9idr5EI8bWXf80mnlEiI4j1vilEu7u7kI%2BTIHaJNIGAt%2Fhm5R0OhbaLQNukTreGrdLaeH2SXmEKdZbM%2Fev57oNiHIEzXiuWZnlFj7udrMkGghfv0IYcsZl5Bv65qA3FhC6stWdRs6kRG6kSZJibyM28lHn8gHgoWdRa71xbwMenb4Fsn%2FnNp0RD%2BXxJhvS%2FklDkFP6t28aE0Cu2VprXnO%2FAIM2oQl7CS4xtCx5JwjzwvsuA6oX3s8%2FAAUg5ViSmRgbsmphzXgTNIeMZ3vm8FVla7Xa25UIq5U2mvnwT2vFSwFf0i5ZuTzpZH3wWKmXZcrLKHOYyX2DyV%2FsSvj8qFt1Md1glGSaqhfG0gqev1LczPYyBqFnykyYophVo4G7IYjuuRoyh5FpMWl0pTqUPkLEZDJZJyWAEwETatPtFRwWA8GI1VStGGvLcnd4eDGnBmUg1jq7PfN4F4Lyp%2BJ0gld166BeXTX5gfhL0zLtzUNRdfeKNNIlTLEVbaQqbQMHOM9y%2BCwDsyIuJU%2B2VO4yDOKf7GViO09GeWPiI3tX8KNUtGa59YZlrq8Ggm5%2FegtA4yQhWSG4RlHqDIfutCaHAbrZfBE%2FPIHEkF99%2BlaIeraQ1CDpzJheHCMVc83iGafLwPdLx%2FCiB%2BGme1r9IMJOSwacGOpoB774xgnK5YeLk%2Fhu7nVdQ7Y6dXdRITSGqQZmnMfDZnZU9FNMVDMUlVkPIXiTHujijtyyNL7SDKkpXH7rs1pBr5WBZ%2FexqzRM3c0rSN0dGSVhqTt1eAQp1BRzC5TT1iwtblspC8r1AoTbTZlO%2BZ10w5b%2BlxBFSdiK2qAMTcU9omzziBwRhTtzBu%2FpRbQHTIZHv2GGGSZflKlTCdg%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230831T081608Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHIN2XLSUQ%2F20230831%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=15008cc2e1362cc957948780d86d15cbfcf8beaf69e854965059b12b015abdf6
Resolving jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)... 52.218.200.42, 52.92.211.177, 52.218.160.78, ...
Connecting to jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)|52.218.200.42|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  x-amz-id-2: 7rJHSkw+qGKBJFttian4zIq/Wg6d2K3X3sEp4RfM62Sz4/vjV6ExLGDElCVqol5gup6AMxJB48o=
  x-amz-request-id: GTZVHASSKG61RXPT
  Date: Thu, 31 Aug 2023 08:18:29 GMT
  x-amz-replication-status: COMPLETED
  Last-Modified: Fri, 29 Jul 2022 13:41:13 GMT
  ETag: "73b91cbf2f960e8567324c42668d8de7"
  x-amz-server-side-encryption: AES256
  x-amz-version-id: 1Tt.ErWWJlbhN583fT8a.YVtm1awUeIB
  Content-Disposition: attachment;filename="temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm"
  Accept-Ranges: bytes
  Content-Type: application/x-rpm
  Server: AmazonS3
  Content-Length: 165558717
Length: 165558717 (158M) [application/x-rpm]
Saving to: ‘/dev/null’

/dev/null                                            100%[===================================================================================================================>] 157.89M  3.21MB/s    in 51s     

2023-08-31 09:19:20 (3.08 MB/s) - ‘/dev/null’ saved [165558717/165558717]

As you can see it does hit the cache but the location still seems to be coming from s3.amazonaws. Perhaps I’m missing something?

7

In fact I know it’s not caching because after 15 minutes I’m now getting a 403 forbidden:

Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=b24d58f3833522ce&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAgaCXVzLXdlc3QtMiJHMEUCIQDQLLJF9rU075%2B%2FLOguynzdDrPqMwu1JERwgQyrTZzLHAIgFKdJxwzuRz8lIGWq1m8NdjsInLuvsPgVs4ASIBJkvhAqjwUI0f%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw5OTk5MzQ3OTA0MTQiDN13GplsgH1IfJGbqirjBIlAS%2FfXXkIN33YiiwvlUM5mePtRu5nwWyTZ86QBHh8FYH4Dh3D6j8qOFFKWy10N83UavTCMYEhiz%2FwqokfQ6HXhb%2F6H6%2FOT5k6Hn2UEmyFORSzrLtR8MsrC%2FI4ly6DwzDldhhf7Wk8Xa1Y05npwkMGgqoyYMhJ1%2F9idr5EI8bWXf80mnlEiI4j1vilEu7u7kI%2BTIHaJNIGAt%2Fhm5R0OhbaLQNukTreGrdLaeH2SXmEKdZbM%2Fev57oNiHIEzXiuWZnlFj7udrMkGghfv0IYcsZl5Bv65qA3FhC6stWdRs6kRG6kSZJibyM28lHn8gHgoWdRa71xbwMenb4Fsn%2FnNp0RD%2BXxJhvS%2FklDkFP6t28aE0Cu2VprXnO%2FAIM2oQl7CS4xtCx5JwjzwvsuA6oX3s8%2FAAUg5ViSmRgbsmphzXgTNIeMZ3vm8FVla7Xa25UIq5U2mvnwT2vFSwFf0i5ZuTzpZH3wWKmXZcrLKHOYyX2DyV%2FsSvj8qFt1Md1glGSaqhfG0gqev1LczPYyBqFnykyYophVo4G7IYjuuRoyh5FpMWl0pTqUPkLEZDJZJyWAEwETatPtFRwWA8GI1VStGGvLcnd4eDGnBmUg1jq7PfN4F4Lyp%2BJ0gld166BeXTX5gfhL0zLtzUNRdfeKNNIlTLEVbaQqbQMHOM9y%2BCwDsyIuJU%2B2VO4yDOKf7GViO09GeWPiI3tX8KNUtGa59YZlrq8Ggm5%2FegtA4yQhWSG4RlHqDIfutCaHAbrZfBE%2FPIHEkF99%2BlaIeraQ1CDpzJheHCMVc83iGafLwPdLx%2FCiB%2BGme1r9IMJOSwacGOpoB774xgnK5YeLk%2Fhu7nVdQ7Y6dXdRITSGqQZmnMfDZnZU9FNMVDMUlVkPIXiTHujijtyyNL7SDKkpXH7rs1pBr5WBZ%2FexqzRM3c0rSN0dGSVhqTt1eAQp1BRzC5TT1iwtblspC8r1AoTbTZlO%2BZ10w5b%2BlxBFSdiK2qAMTcU9omzziBwRhTtzBu%2FpRbQHTIZHv2GGGSZflKlTCdg%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230831T081608Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHIN2XLSUQ%2F20230831%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=15008cc2e1362cc957948780d86d15cbfcf8beaf69e854965059b12b015abdf6 [following]
--2023-08-31 09:31:33--  https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adoptium/filestore/38/385548e13f70ca83a28cf62fbb677196a95bfb84?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=rpm&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=rhel%2F9%2Fx86_64%2FPackages%2Ftemurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm&X-Artifactory-projectKey=temurin&x-jf-traceId=b24d58f3833522ce&response-content-disposition=attachment%3Bfilename%3D%22temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAgaCXVzLXdlc3QtMiJHMEUCIQDQLLJF9rU075%2B%2FLOguynzdDrPqMwu1JERwgQyrTZzLHAIgFKdJxwzuRz8lIGWq1m8NdjsInLuvsPgVs4ASIBJkvhAqjwUI0f%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw5OTk5MzQ3OTA0MTQiDN13GplsgH1IfJGbqirjBIlAS%2FfXXkIN33YiiwvlUM5mePtRu5nwWyTZ86QBHh8FYH4Dh3D6j8qOFFKWy10N83UavTCMYEhiz%2FwqokfQ6HXhb%2F6H6%2FOT5k6Hn2UEmyFORSzrLtR8MsrC%2FI4ly6DwzDldhhf7Wk8Xa1Y05npwkMGgqoyYMhJ1%2F9idr5EI8bWXf80mnlEiI4j1vilEu7u7kI%2BTIHaJNIGAt%2Fhm5R0OhbaLQNukTreGrdLaeH2SXmEKdZbM%2Fev57oNiHIEzXiuWZnlFj7udrMkGghfv0IYcsZl5Bv65qA3FhC6stWdRs6kRG6kSZJibyM28lHn8gHgoWdRa71xbwMenb4Fsn%2FnNp0RD%2BXxJhvS%2FklDkFP6t28aE0Cu2VprXnO%2FAIM2oQl7CS4xtCx5JwjzwvsuA6oX3s8%2FAAUg5ViSmRgbsmphzXgTNIeMZ3vm8FVla7Xa25UIq5U2mvnwT2vFSwFf0i5ZuTzpZH3wWKmXZcrLKHOYyX2DyV%2FsSvj8qFt1Md1glGSaqhfG0gqev1LczPYyBqFnykyYophVo4G7IYjuuRoyh5FpMWl0pTqUPkLEZDJZJyWAEwETatPtFRwWA8GI1VStGGvLcnd4eDGnBmUg1jq7PfN4F4Lyp%2BJ0gld166BeXTX5gfhL0zLtzUNRdfeKNNIlTLEVbaQqbQMHOM9y%2BCwDsyIuJU%2B2VO4yDOKf7GViO09GeWPiI3tX8KNUtGa59YZlrq8Ggm5%2FegtA4yQhWSG4RlHqDIfutCaHAbrZfBE%2FPIHEkF99%2BlaIeraQ1CDpzJheHCMVc83iGafLwPdLx%2FCiB%2BGme1r9IMJOSwacGOpoB774xgnK5YeLk%2Fhu7nVdQ7Y6dXdRITSGqQZmnMfDZnZU9FNMVDMUlVkPIXiTHujijtyyNL7SDKkpXH7rs1pBr5WBZ%2FexqzRM3c0rSN0dGSVhqTt1eAQp1BRzC5TT1iwtblspC8r1AoTbTZlO%2BZ10w5b%2BlxBFSdiK2qAMTcU9omzziBwRhTtzBu%2FpRbQHTIZHv2GGGSZflKlTCdg%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230831T081608Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-Credential=ASIA6RUGCBMHIN2XLSUQ%2F20230831%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=15008cc2e1362cc957948780d86d15cbfcf8beaf69e854965059b12b015abdf6
Resolving jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)... 52.218.246.99, 52.92.249.129, 52.92.195.33, ...
Connecting to jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)|52.218.246.99|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 403 Forbidden
  x-amz-request-id: 8YA92GS3VMDDWFT6
  x-amz-id-2: Zz2ubcqU2h82YwLjdMFn/NehohbEuCpNuWOyb1tw5aqH8UUGPLuUVcXfV9NY3OW6Qt6dgvTC6pY=
  Content-Type: application/xml
  Transfer-Encoding: chunked
  Date: Thu, 31 Aug 2023 08:31:34 GMT
  Server: AmazonS3
2023-08-31 09:31:34 ERROR 403: Forbidden.
8

I can confirm the service is still returning 302

$ curl -I  http://packages.adoptium.net.global.prod.fastly.net/artifactory/rpm/rhel/9/x86_64/Packages/temurin-11-jdk-11.0.16.0.0.8-1.x86_64.rpm
HTTP/1.1 302 Found
...
Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-adop...

I briefly checked your service (as a Fastly employee).
I noticed that the host (s3) of the new location is different from the existing origin.
All origin must be registered on the services, including the S3 one.

The example fiddle I mentioned previously can only handle redirecting to the same origin, Notice there are some comments in vcl_deliver

# Only do so if the location header does not specify a host, or the host matches the client-side host header, or a whitelist of 'local' domains
...

So what needs to be done here:

  • Create a new S3 origin jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com
  • Add a False condition to it to prevent it from being set as a default backend
  • After the restart of 302, vcl_recv will be called again, set the backend to S3 .

You should also be able to get good support from support@fastly.com, pasting the link of this thread in the email would make the conversation smoother.

9

I have written a fiddle to demonstrate how to chasing 302 to a new origin

https://fiddle.fastly.dev/fiddle/fab5a586

In this fiddle
We are making a request

https://httpbin.org/redirect-to?url=https%3A%2F%2Fhttp-me.glitch.me%2Fstatus%2F200&status_code=302

to httpbin.org , which will return a new location, pointing to a new host http-me.glitch.me

< location: https://http-me.glitch.me/status/200

And our fiddle will restart and continue to grep the ‘/status/200’ from `http-me.glitch.me

Some information about Add a False condition to jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com

When you create a condition, type false into Apply if ... edit box

10

Thanks @noguxun for the updated Fiddle, I’m now permanently getting 404 when I make requests to that URL. Any idea what else might be wrong?

Thanks so much for your help!

11

I see the s3 backend does not have an override host set to jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com, can you add that to your S3 origin and try again?

see more at

(Optional) In the Override host field, enter the hostname of your override Host header based on the origin you’re using. The value in this field will take precedence over anything you've set using the global override host configuration. To see example Host headers for third-party services, refer to our developer documentation on overriding the Host header.

https://developer.fastly.com/learning/integrations/backends/#overriding-the-host-header

12

Thanks @noguxun I’ve made that change, but I’m still getting 404 errors each time I make that request.

13

You need to attach false condition to the JFrog backend too.

Other wise, the generated VCL will generate some backend set code to override the F_S3 setup.

14

Also, to improve the performance, better to enable Streaming Miss

To be able to serve files more than 20M, Segmented Caching is needed.

15

Thanks @noguxun that seems to be the magic piece. I am now able to download directly from the Fastly cache. Two questions:

  1. The download speed seems pretty slow to me, anything I can do to improve that?
  2. There are certain files that I cannot cache. E.g the ones in https://adoptium.jfrog.io/ui/repos/tree/General/rpm/rhel/7/x86_64/repodata which change all the time when new RPM files are published. What can I do to ensure those URLs are always routed to S3 for the latest version?
16

Actually looking at the file it has the Cache-Control: no-store header so perhaps Fastly already knows not to cache it?

curl -I https://adoptium.jfrog.io/artifactory/rpm/rhel/7/x86_64/repodata/repomd.xml
HTTP/1.1 200 
Date: Fri, 01 Sep 2023 12:09:23 GMT
Content-Type: application/xml
Content-Length: 1555
Connection: keep-alive
X-JFrog-Version: Artifactory/7.64.7 76407900
X-Artifactory-Id: a89daeb8e3f1ab4ce0fc8c3abdab48f88990193c
X-Artifactory-Node-Id: adoptium-artifactory-primary-0
Last-Modified: Thu, 31 Aug 2023 14:51:12 GMT
ETag: bdc19d09e9053cdfbf5333e3e230d4d4959f3bce
X-Checksum-Sha1: bdc19d09e9053cdfbf5333e3e230d4d4959f3bce
X-Checksum-Sha256: 0c48298eaf7a2d2ddf06dbc594dbc3383863036e6daf16f5ef463f6d24ff411f
X-Checksum-Md5: 5339c27a3108b1faed24a5a03551c364
Accept-Ranges: bytes
X-Artifactory-Filename: repomd.xml
Cache-Control: no-store
X-Request-ID: 97afa04f1ed6985aad9f2468ed558dfe
17

Following up again, the download speeds are fine now - I’m guessing it was taking time to propagate to all the regions.

18

Not really, See Fastly’s behavior at

You can set 302 response to non-cachable for some path in vcl_fetch()

if (req.url.path == "/some_path" && beresp.status == 302) {
  set beresp.cacheable = false;
}
19

@noguxun I think I’ ve very close to having this fully working. I added the following code to my vcl_fetch:

if (beresp.http.Content-Type ~ "^(application/x-rpm|application/x-debian-package|application/vnd.android.package-archive)") {
        set beresp.ttl = 15780096s;  # 6 months
        set beresp.http.Cache-Control = "public, max-age=" + beresp.ttl;
}

Which I can see sets the headers when I make follow up requests but I still seem to be finding that the files are getting removed from cache within a day. Am I missing anything else in my config?

20

@gdams, It is an expected behavior.
Setting the longer TTL does not guarantee the object to be cached on Fastly POP for TTL time.
If the object is not frequently accessed, it might be evicted by Fastly to make space for objects that are more frequently accessed.

TTL just tells Fastly, If the object can be found on Fastly POP, it ok to serve this object within TTL time.

See Cache freshness and TTLs | Fastly Developer Hub

IMPORTANT: We won’t necessarily store objects for the full TTL requested, and may evict less popular objects earlier, especially if they are large. We also do not automatically evict objects when they reach their TTL. They simply become stale.