CoSocial using Fastly CDN for Mastodon

Hey all! I’m one of the folks working on setting up CoSocial, a co-op owned social web for Canadians.

We’ve got a really basic tech setup right now: a Mastodon install on Digital Ocean Toronto data center.

Managing files by tuning media caching and cleanup is one of the next things on our list, and in fact file usage is the thing that’s grown the fastest for us.

We’d like to use Fastly as our CDN, and are looking for best practices and guides on setting that up for Mastodon.

There’s the QuickStart page here Start here

But for example many Mastodon installs do use root/apex domains

We recommend that you don’t use apex or second-level domains with Fastly. For example, you should use www.example.com and not example.com .

I assume we’ll want to purge content from Fastly in the same way we purge cached media in tootctl cron jobs. We’re just setting that up.

I’m sure I’ll have lots of other questions as we dig in.

Thanks @jenn for setting up this category!

3 Likes

New user, so posting a link to CoSocial blog here as a follow up https://cosocial.info/

Having configured several sites with Fastly (and some of those also with Cloudflare and/or CloudFront), I have a few general pointers. If you have specific questions or problems, I’m happy to help!

Use a separate Fastly Service for staging

Nothing in Fastly stops you from running staging.example.com and production.example.com through the same Service. But there are some features (like TLS versions) that can only be configured at the Service level. For confidence in rolling out changes to your configuration, put staging in a separate service.

This is especially true if you’re using apex domains and/or terraform.

Use a separate apex domain for your staging service

I’ve tried www.example.com and staging.example.com in the past and it creates a lot of problems. You want staging to look as much like production as possible. Apex domains work pretty well on Fastly, but they have some limitations (both in Fastly and in the web in general) that you want to know about early. Use example.com and example.dev and save yourself a lot of headaches. (Some things that are different between apex domains and others include DNS anycast, CNAME support, and cookie handling.)

Make sure your content is actually cacheable

If your response includes a set-cookie header, Fastly won’t (and shouldn’t) cache it. If you have a resource that is composed of five other pieces of data, it needs to expire at the rate of the fastest of those. Decompose resources and make more, faster HTTP requests. (Sometimes, your data will tell you that you really do need to group things. For that, Edge-Side Includes and other similar compositional techniques can help.)

Set cache-control headers in your origin

Your origin knows the most about the data, its dependencies, and how often it will change. It is the best place to define cache behavior. You can override TTLs in Fastly, but do this only when all other options have failed.

You’ll get very far with stale-while-revalidate

This is specially true for “popular” resources. Every resource is different, but if I don’t know what cache behavior to pick, I’ll start with

cache-control: public, max-age=0, s-maxage=60, stale-while-revalidate=15552000 stale-if-error=36288000

That will cause browsers to never cache the response, but Fastly to cache it for 1 minute, then continue to serve the stale version for an additional 3 days while it attempts to get a fresh version. Each Fastly node queues up all the clients asking for the same resource, so your origin will only get one revalidation request for all the simultaneous clients each time a resource expires.

(If stale-while-revalidate isn’t cutting it for you – for example, if you need to expire resources faster than 60s – you’ll have to switch to long cache times and active purging. Cache tags are the next easiest solution.)

Use cache tiering

If your users are all geographically concentrated, this may not matter much. If they’re mostly concentrated, but with a long global tail, cache tiering will dramatically improve performance for the folks outside of the core area.

Beware cache fragmentation

If you have a resource that varies by language, user-agent, or some other input that has huge variation, HTTP caching won’t buy you much out-of-the-box.

For example, say my browser sends accept-language: en-US,en;q=0.5,fr;q=0.4 and yours sends accept-language: en-CA,en;q=0.6;fr;q=0.1. If the server supports en but not any country-specific variant, those will both resolve to en. But (assuming the response has a proper vary: accept-language header) they’re cached separately.

To improve performance and reduce load, you can have Fastly decompose that accept-language header to a resolved-language: en header and vary on that. Check out Financial-Times/polyfill-service for a sophisticated example of this.

1 Like

Thanks @jamesarosen, that’s super detailed.

But! I really have no idea what I’m doing! How do I go from this to having a CDN in front of media attachments on Mastodon is kind of the template or step by step I’m looking for.

My question on root domains looks to be answered (that it will work), like our own CoSocial.ca. But that I will need to go domain shopping for another one for testing.

We’re on NameCheap right now. Do we need to move DNS to Fastly?

Hey Boris! Welcome. We (Fastly) are working on more detailed, Mastodon-specific content, but ahead of this, are you in touch with the folks running Mastodon.social, or Universeodon? [Cf. the toot I mentioned you all in yesterday to try and get this ball rolling.]

In the mean time we can try and workshop this a bit in this thread, and I’ll let others I know that are more expert in Mastodon specifics, know about this.

Take a look at our core concepts guides over on Developer Hub. These pages should help with some of the early ground.

Beyond that, I think it’ll help to split the problem and architecture up and solve each piece. E.g., we have specific instructions for using a shield in front of a backend object store like S3, if you use such a place for media storage. (The shield pattern is essentially getting a nearby Fastly POP to get objects from the S3 bucket, and then pass these on to the actual edge POPs nearby your users. Getting the objects into the Fastly network could help reduce storage costs, and you’re not paying for traffic within the Fastly network.)

2 Likes

Thanks for this great reply James… and welcome to the forums!

1 Like

Last I checked, Fastly doesn’t offer DNS registrations. I use DNSimple wherever possible. They’re reasonably priced and very kind folks with an easy-to-use product.

1 Like

Great that S3 link is helpful.

I’d love for us to get to Tutorial style content like “how to add Fastly to a single Mastodon server”

The larger installs will of course be more custom and have more tech resources.

1 Like

I’d love for us to get to Tutorial style content like “how to add Fastly to a single Mastodon server”

+1 for that!

Also check out the integrations page and note that some object stores have reduced egress cost, to Fastly. Are you wanting all your infrastructure to be within Canada?

:wave: I’ve done some work setting up Fastly in front of Mastodon instances - both in front of image store individually and in front of the instance as a whole. I’d be happy to answer any specific questions!

3 Likes

Hey Daniel, thanks for joining!

@boris: @fastly-daxtens is one of our experts here. Daniel and I are collaborating on a tutorial as you’re asking for, and we’d love to collaborate with you on that too, and perhaps have you validate that it’s what you’re after, gives the initial results you’re looking for, etc.

1 Like