Purge CDN path via webhook

General
1

Hi all,

I’m planing to use fastly as CDN to deliver the firmware and package artifacts of a mid sized open source project. We’re running 15-50 buildbots in parallel which upload artifacts to an origin server overwriting the past package indexes and checksum files.

During initial tests this showed issues since the index/checksums could get out of sync with the actual files or vice versa.

To improve the situation I’d like to call a webhook via our buildbots which “flushes” a certain path of the CDN.

For now all I could find is some VCL example which I fail to understand entirely…

Is there a webhook library or wrapper or anything to make this a simple API call authenticated by some token?

Thank you all very much for further information.

2

Hey Paul,

Thanks for considering Fastly for this. You can definitely do what you want. First use the VCL you found (or use one of the compute languages if you want the same functionality in a compute service).

The effect of this code is to tag your objects with all the ancestor paths that make up their URLs, so if you serve an object on the path /binaries/dist/2023/12, we would tag that object in cache with

/binaries/dist/2023/12
/binaries/dist/2023
/binaries/dist
/binaries

Then you can issue a surrogate key purge to delete all objects with a particular key:

Does that work?

Also, you did say a “certain path” and though you linked to the prefix example, I wonder if you actually just need to purge one file? If that’s the case there’s no need to write any code and you can just purge using the single URL purge method, also documented on the page above.

Andrew

1 Like
3

Following on with what Andrew said, there are a number of ways you can do this:

  • The buildbots themselves can issue the purge URL API call after they complete pushing new content to the files in the origin storage.

  • The buildbots can generate a webhook-like event (HTTP GET or POST) to an intermediary service which is able to issue the purge URL API call on their behalf; this method avoids making the API credentials available to all of the buildbots, but requires deployment of that intermediate service. That service could be a very simple program running in Fastly Compute which would get the necessary API credentials from a Config Store. There are other options too: IFTTT or similar cloud-business-logic services could be used for this.

Other options are possible too, but would require polling for changes to the relevant files and thus incur additional purging latency.

4

Hi @triblondon & @kevin.fleming

Both your advises were very helpful and I was able to get it working in no time, excellent support!

I’m planing to do the DNS switch tonight/tomorrow, please brace for impact :wink:

3 Likes