Brotli Compression Support


#1

Chrome is starting to support Brotli compression via HTTPS. See https://groups.google.com/a/chromium.org/forum/#!searchin/blink-dev/brotli/blink-dev/JufzX024oy0/WEOGbN43AwAJ

Will fastly use Brotli compression when it is better than gzip and the browser supports it?

Thanks!


#2

Will fastly use Brotli compression when it is better than gzip and the browser supports it?

I expect we’ll move in that direction. Rolling a new compression format across our network is something that will take some nontrivial R&D, as with similar changes, but we’re always trying to move things forward.


#3

Just an update, I have brotli enabled on https://meta.discourse.org it works, I can see Vary:Accept-Encoding in the response headers. Is there any reason this is not coming through the CDN?


#4

We currently normalize Accept-Encoding, which doesn’t account for brotli. And we can’t just update this normalization, since it would increase our storage by quite a bit, for no gain. We’re working on it.


#5

Ideally if the normalization accepted path as a param a pretty clean workaround could be introduced that adds minimal impact on your side.

If I request say:

https://cdn.fastly.com/asset.js?brotli=allow

The presence of brotli=allow can flag allowing of brotli in normalization routine.

The trouble I see with brotli is that the place it is used most is for bootstrapped assets, such as common js and css files. For those files it is very complicated to inject custom headers. Another mechanism is needed. (whitelisting specific assets in your dashboard is an option as well)

If you simply turn on brotli in the normalization routines you double your storage, so I assume that will be a non starter.

Projecting long-term, due to the enormous cost of brotli compression I do not anticipate it used on all assets any time soon, but for specific ones that impact page load times they can be used today.


#6

Good news, everyone! (Did you hear Professor Farnsworth’s voice in your head? I did.)

For those wanting to support Brotli right now, with Fastly, there’s a way to work around our normalization of the Accept-Encoding header. Now first a warning, do not skip the normalization step below, and don’t use this workaround if your origin doesn’t actually support Brotli. In either case you would be hurting your hit ratio, and that would actually hurt performance!

The trick is to use the Fastly-Orig-Accept-Encoding header, and normalize that to one of 3 values: gzip, br, or not set at all. (The latter being a value of sorts in this case.)

To achieve this, you can use this VCL in vcl_recv:

    if (req.http.Fastly-Orig-Accept-Encoding) {
      if (req.http.User-Agent ~ "MSIE 6") {
        # For that 0.3% of stubborn users out there
        unset req.http.Accept-Encoding;
      } elsif (req.http.Fastly-Orig-Accept-Encoding ~ "br") {
        set req.http.Accept-Encoding = "br";
      } elsif (req.http.Fastly-Orig-Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
      } else {
        unset req.http.Accept-Encoding;
      }
    }

A couple of notes:

  1. While this code is fine, if any new compression scheme shows up that has br in its identifier, and a browser would support that, but not Brotli, things would break. Now this is pretty unlikely to happen, but if you’re worried about it, you can use (^|[, ])br($|[, ]) as the regular expression instead.
  2. Those of you who have read my Vary blog post might wonder “what about deflate?”. Well, if you’re really worried about the less than 0.02% of user agents out there that support deflate but not gzip, you can add that back in, but since sending them an uncompressed version of the object doesn’t actually cause errors, I wouldn’t worry about it personally, and would prefer the slight increase in hit ratio that you gain from leaving deflate out of things.

To get this effect, you would expand the code to:

  if (req.url ~ "brotli=allow") {
    if (req.http.Fastly-Orig-Accept-Encoding) {
      if (req.http.User-Agent ~ "MSIE 6") {
        # For that 0.3% of stubborn users out there
        unset req.http.Accept-Encoding;
      } elsif (req.http.Fastly-Orig-Accept-Encoding ~ "br") {
        set req.http.Accept-Encoding = "br";
      } elsif (req.http.Fastly-Orig-Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
      } else {
        unset req.http.Accept-Encoding;
      }
    }
  }

#7

So just to clarify, req.http.Fastly-Orig-Accept-Encoding is something you add from your side? not something requesters are expected to add to the HTTP requests?

How do I go about inserting the rule using the Fastly UI? I don’t appear to have the ability just to simply edit the generated VCL?


#8

Yes. Before we normalize Accept-Encoding in our master VCL, we copy its value to Fastly-Orig-Accept-Encoding.

You could add a Header object with the following settings:

Name: Normalize Accept-Encoding for Brotli
Type: Request
Action: Set
Destination: http.Accept-Encoding
Source: if(req.http.User-Agent ~ "MSIE 6", "identity", if(req.http.Fastly-Orig-Accept-Encoding ~ "br", "br", if(req.http.Fastly-Orig-Accept-Encoding ~ "gzip", "gzip", "identity")))

Notes:

  1. Instead of unsetting the header if there’s to be no compression, it uses the value identity. This value means “no compression” as well. See https://tools.ietf.org/html/rfc7231#section-5.3.4

  2. Because of the use of "identity" instead of an empty value, clients unable to do gzip will cause cache misses.

    If this is a concern, instead of using "identity" you can use a non-existent header, like req.http.My-Non-Existent-Header. That will basically unset Accept-Encoding.

    For security purposes, you should add a Header object (type: Request, action: Delete) with a lower priority number, to ensure that that header is indeed non-existent. By setting the header to a value, someone can basically set Accept-Encoding to whatever they want, and when using a unique value for each request, bypass caching.

Alternatively, you can email support to ask for Custom VCL to be enabled for your account. See also https://docs.fastly.com/guides/vcl/uploading-custom-vcl and https://docs.fastly.com/guides/vcl/mixing-and-matching-fastly-vcl-with-custom-vcl.

Let me know how that works, or if you have any further questions!


#9

Mea culpa. I posted that code without testing it, and it turns out there was something I missed in our Master VCL that made this not work.

Good news is that that has been fixed, and the above code now works.

If you’ve had this code in your VCL, you’ll have to purge things that were requested in the timeframe between deploying that code and roughly an hour ago.

Also, thanks to Donald Stufft of PyPI fame for pointing out the code didn’t have the desired effect. (He is running this in production now.)


#10

Thanks very much for this workaround @drwilco, any other news on Brotli entering mainstream support as more of a built-in thing?


#11

Chrome and Firefox have both shipped brotli and Microsoft Edge is in the process of shipping it. Getting mainstream support for this as @codinghorror suggested would be really nice!


#12

Safari 11 on iOS 11 and High Sierra will support brotli as well. Therefore all Mayor Browser will support Brotli in October 2017.

Has there been any progress in a fastly implementation in the last 1 3/4 Years on this matter? And I do not mean the current working hack, with accept-encoding :wink:


#13

It’s October 2017 already. My how time flies.

Is there any progress in a Fastly implementation of brotli, perchance?

Cheers </a>.


#15

Very interested to know more about BROTLI support.
There are no reasons why BROTLI is not available in my opinion on Fastly.


#16

It would be an interesting feature :slight_smile: Hope it’s coming soon.


#17

this vcl is little bad I think.

Client is request to the fastly.

Accept-Encoding:gzip, deflate, br

When origin server does not support the brotli, fastly got the un-compressible files even if supported the gzip both origin server and clients.

In order to avoid this problem, req.http. Fastly-Orig-Accept-Encoding value should be append (not overwrite)but I can not find this option.

when fastly backend request include gzip, deflate, br, the origin server can compress contents using gzip.

Is my understanding of this correct?


#18

Yeah, this VCL is only meant for setups that support Brotli. If your origin does not support Brotli, you don’t have to do anything.


#19

I had adjust VCL.
but I don’t know FASTLY could used vcl_backend_fetch.

vcl_recv 

if (req.restarts == 0) {
  # normalize Accept-Encoding to reduce vary
  if (req.http.Accept-Encoding) {

  #brotli handling
    if (req.http.Accept-Encoding ~ "br") {
        set req.http.TMP-Accept-Encoding = "br";
        set req.http.X-brotli = "true";
    }

    if (req.http.User-Agent ~ "MSIE 6") {
      unset req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
     set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "$", ",gzip");
     set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "^,", " ");
    } elsif (req.http.Accept-Encoding ~ "deflate") {
       set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "$", ",deflate");
       set req.http.TMP-Accept-Encoding = regsub(req.http.TMP-Accept-Encoding, "^,", " ");
    } else {
      unset req.http.Accept-Encoding;
    }
  }
}

vcl hash

##brotli
    if(req.http.X-brotli == "true") {
        hash_data("brotli");
    }

vcl_backend_fetch

unset bereq.http.Accept-Encoding;
unset bereq.http.X-brotli;

set bereq.http.Accept-Encoding = bereq.http.TMP-Accept-Encoding;

#20

What are you trying to do here? Because this VCL will not work with Fastly.

Also, adding something to the hash is a bad idea usually. (As it breaks purging.)


#21

Hey,

We looked at this, and none of these solutions were great for us. We went a different route.

We used VCL Snippets like so:

Name: Normalize Accept-Encoding for Brotli 
Type (placement of the snippet): within subroutine, recv
VCL:
  declare local var.normalized STRING;

  set var.normalized = "";

  if (req.http.Fastly-Orig-Accept-Encoding) {
    # Supported encodings, in order of preference.
    if (req.http.Fastly-Orig-Accept-Encoding ~ "([ ]*,[ ]*)?br[ ]*(,[ ]*|$)") {
      set var.normalized = var.normalized + "br,";
    }
    if (req.http.Fastly-Orig-Accept-Encoding ~ "([ ]*,[ ]*)?gzip[ ]*(,[ ]*|$)") {
      set var.normalized = var.normalized + "gzip,";
    }
    if (req.http.Fastly-Orig-Accept-Encoding ~ "([ ]*,[ ]*)?deflate[ ]*(,[ ]*|$)") {
      set var.normalized = var.normalized + "deflate,";
    }
  }
    
  if (var.normalized == "" ) {
    unset req.http.Accept-Encoding;
  } else {
    set req.http.Accept-Encoding = regsub(var.normalized, ",$","");
  }

This allowed us to:

  • not have to enable custom VCL and carry that burden forward
  • support many different accept encodings, and have our server pick which it supports
  • have a simple place to understand where to change the accept-encoding behaviour

Hope this helps someone else.