How do I enable basic ESI in my VCL?


#1

You can implement basic ESI through Fastly with custom VCL.
Note: If you do not have custom VCL enabled on your account contact support@fastly.com

Edge Side Includes (ESI)

ESI is used to dynamically insert content into pages that are cached. ESI can insert content into pages, but it can also include logic like conditionals. Fastly’s implementation of ESI only includes the <include> tag, which is the same as open source Varnish. Customers with complex ESI logic must transition to using a combination of ESI and VCL in order to achieve the same thing on Fastly.

Include Example

Inside a document that is cached by Varnish, you can use include tags. They look like this:

<esi:include src="/foo/bar"/>

This tag will be replaced by a call from Varnish back to itself with the URL /foo/bar. This can either be a synthetic or a cached object, perhaps even from a different origin.

However, this behavior is not enabled by default. ESI processing must be enabled in vcl_fetch. Basic usage:

sub vcl_fetch {
  esi;
}

Best Practices

Conditionally Enable ESI

You should always enable ESI conditionally. If you don’t, Varnish will have to inspect the response body’s content on every cache miss. Not only is this inefficient, but you risk finding ESI tags in binary data, which could lead to corruption of the file. This is the proper way to enable ESI, in this case only for HTML:

sub vcl_fetch {
	if (req.url.ext == "html") {
		esi;
	}
}
ESI and Shielding

By default, Varnish will replace ESI tags on requests from edge datacenters. Since the edge datacenter will receive the replaced content from the shield, this essentially breaks the ESI functionality. In order to get around this, there is a variable called req.esi. This variable is used in vcl_recv, and allows the user to disable ESI replacement on any request. If you set req.esi = false; Varnish will preserve the tags and send them to the client. req.esi defaults to true. Shielding and ESI can coexist with the following VCL:

sub vcl_recv {
	if (req.http.Fastly-FF) {
		set req.esi = false;
	}
}

sub vcl_fetch {
	if (req.url.ext == "html") {
		esi;
	}
}

Notes

  • Changes made to variables within an ESI request do not persist back to the original request.
  • Changes made to variables in the top level request do not persist to the ESI request.
  • req.topurl is a variable that contains the URL of the original request, and is only available with the ESI request. req.topurl will be NULL if the request was not generated with ESI, so the variable can be checked to test whether a request is generated by ESI.

Useful Examples

Synthetic Includes

This example assumes a document has include tags with the URL /esi/token. The example will insert tokens into the document for later validation.

sub vcl_recv {
	if (req.http.Fastly-FF) {
		set req.esi = false;
	}

	if (req.topurl && req.url == "/esi/token") {
		error 990 "fastly internal";
	}
}

sub vcl_fetch {
	if (req.url.ext == "html") {
		esi;
	}
}

sub vcl_error {
	if (obj.status == 990) {
		set obj.status = 200;
		set obj.response = "OK";
		synthetic urlencode(digest.time_hmac_sha256(req.http.key, 7200, 0));
		return(deliver);
	}
}
Cached Includes
sub vcl_recv {
	if (req.http.Fastly-FF) {
		set req.esi = false;
	}
}

sub vcl_fetch {
	if (req.url.ext == "html") {
		esi;
	}

	if (req.topurl) {
		/* cache includes for 1hr */
		set beresp.ttl = 1h;
	}
}

sub vcl_hash {
	{
		/* cache different versions of includes for each user */
		if (req.topurl && req.http.Cookie:session_id) {
			set req.hash += req.http.Cookie:session_id;
		}

		set req.hash += req.url;
		set req.hash += req.http.host;
		set req.hash += "#####GENERATION#####";
		return (hash);
	}
}

Keep in Mind:

  • Fastly currently doesn’t support the use of ESI with GZIP. Both the container and the ESI URL need to be uncompressed.

  • If you can’t avoid compression, you may want to explore using AJAX6 instead.

  • Fastly supports the use of single quotes and double quotes in your ESI tags.

  • ETags and Last-Modified headers on ESI fragments are not honored when the reassembled page is received by the end-user. The reassembled page will have the response headers of the root object and content replaced for the ESI fragments. To prevent stale content returned when the browser sends If-None-Match or If-Modified-Since unset the following:

    • unset beresp.http.ETag;
    • unset beresp.http.Last-Modified;

Conditionally Disabling GZIP in Order to Serve ESI
#2

Can fastly then compress the output to the client, ie. after the esi has been processed?


#3

@ranguard not at the moment, though we’re actively working on supporting this :smiley:. Depending on the way you’re using ESI, conditionally deactivating gzipping for ESI content may be workable in the interim. But, if you’re gzipping a significant portion of your content, this isn’t ideal.


Examples of ESI Logic in Custom VCL
#4

To see an example of ESI logic already inserted into Fastly’s boilerplate, please visit this public Github repo, which also contains examples of CSRF tags in HTML:

And to check out more examples of how you can use ESI logic, visit our follow-up ESI Community Page: Examples of ESI Logic in Custom VCL


#5

Hi there!

Is this the current documentation for ESI? Any update on ESI support/caveats?

Thanks!

Ari


#6

Hi,

not at the moment, though we’re actively working on supporting this

has there been any progress in allowing ESI and compression? We have moved to fastly last Year and one of the reasons was support for ESI. We had to drop it temporarily from our Websites to allow everything to work, but my developers are still sad - that we can’t use it right now.

Best,
Tim


#7

Hi @TimWeyand

We currently have an implementation that some customers are using, but it’s not been made generally available. There are a few criteria that need to be in place to use it, but to find out more, and if it’ll be available for you to use, send in a support ticket and we’ll let you know.