Brotli is a compression algorithm from Google that has been making waves recently. You may have heard about its average 10-20% filesize savings over gzip, at comparable decompression speeds. This makes it a very irresistable format for serving static content, but it still has immature support from CDNs, which puts it out of reach for general widespread availability. A search run over HTTP Archive data indicates that there are around 25 web properties serving Brotli Content-Encoding URLs from Fastly, the CDN we use to host our static content, as of July 25th.

While Fastly does not automatically convert content to Brotli for users, they removed a critical roadblock in serving Brotli by exposing the client’s Accept-Encoding header back in June 2016. This is all we need in order to know if a user will understand a Brotli response. In this article, we explain how we implemented serving responses with Brotli compression for our existing URLs, progressively enhancing the performance of our website for the clients that support this new format! With a clear solution to upgrade responses to Brotli, the number of websites supporting it should only increase!

Behind the scenes

To send Brotli responses to our users, we must have a Brotli version of our files! At Yelp, when a developer makes a change to a Javascript, CSS, or SVG file, we build a new version of that package and upload it to Amazon S3. That’s the backing datastore and source of truth for all requests through Fastly to yelpcdn.com. So, if the CDN hasn’t received a request for the content with that URL, including version hash, it will be fetched from S3. Since this file doesn’t change once it’s been uploaded, we gzip it before uploading and set the Content-Encoding header in S3 itself.

In order to serve Brotli content, we will now upload a second version of every file to S3 with the same path and suffixed with .br:

if file_ext in TEXT_EXTENSIONS:
    gzip_file(srcfilename, destfilename)
    brotlify_file(srcfilename, destfilename + '.br')

Since this is an article about Brotli, here is that helper’s definition:

def brotlify_file(srcfilename, destfilename):
    with open(srcfilename, 'rb') as srcfile:
        with open(destfilename, 'wb') as destfile:
            destfile.write(brotli.compress(srcfile.read()))

The cache key

Fastly uses Varnish behind the scenes, and this is where we will add logic to cache a Brotli version of a file separately from the gzip version of the same file. Other headers that are part of the response’s Vary header are part of that cache key along with the URL. The Accept-Encoding header is almost always present in the Vary response header, and normalization means you can get a much better cache hit ratio since browsers may have a lot of different combinations of values. In the following sections, you will see how this header is used to determine which backend asset to request, and why it isn’t quite enough on its own.

The approach

We aren’t uploading Brotli compressed versions of 100% of our assets yet, so for a client that accepts Brotli and gzip, we’d like to first try for the Brotli version. If that version does not exist, we’ll fall back to the gzip version.

However, if we get a cache miss on the Brotli version for every request, we will be dramatically slowing down the response time instead of improving it! And we will rack up a larger AWS bill with so many GET requests. But we’re aware of this issue, and in the next section we’ll learn the basics of Varnish needed to avoid making that mistake.

The flow of control

In order to understand the next section, I’d highly recommend referring to this wonderful diagram to understand what logic is executed during each step of a request being processed by Varnish.

We will add a bit of logic in three methods to give us the desired flow of control. Here is the relevant excerpt of the logic that we’re adding:

The relevant Varnish methods we're interested in analyzing further.

The special sauce

Fastly has a boilerplate varnish file posted in their support documentation and provides extensive instructions (and, err… warnings…) on how to use it. They’ve also recently released a new online VCL editor, which might be able to complete the required modifications.

We will start from the boilerplate, and will include a custom file as the first line in each of vcl_miss, vcl_fetch, and vcl_recv. The three custom files contain:

custom_recv.vcl

# Add br to the choices for normalized Accept-Encodings
# Doesn't require a brotli response, but more encodings means multiple cached objects per key
if (req.http.Fastly-Orig-Accept-Encoding) {
    if (req.http.User-Agent ~ "MSIE 6") {
        # For that 0.3% of stubborn users out there
        unset req.http.Accept-Encoding;
    } elsif (req.http.Fastly-Orig-Accept-Encoding ~ "br") {
        set req.http.Accept-Encoding = "br";
    } elsif (req.http.Fastly-Orig-Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
    } else {
        unset req.http.Accept-Encoding;
    }
}

custom_miss.vcl

# Brotli compression: user must support it, and must be text file type
# If so, look for it the first time
if (req.http.Accept-Encoding ~ "br" && req.url ~ "\.(js|css|svg)($|\?)" && req.restarts == 0) {
    set bereq.url = regsub(req.url, "\.(js|css|svg)($|\?)", "\.\1\.br\2");
}

custom_fetch.vcl

# if we looked for a brotli file and didn't find it, restart
if (bereq.url ~ "\.br($|\?)" && beresp.status != 200) {
    restart;
}

Understanding the Changes

We recognize the contents of custom_recv, that’s the Accept-Encoding normalization logic to add Brotli support that was provided by Fastly in the community thread about Brotli support.

From the diagram above, we know that the second step in flow of control that we want to customize is vcl_miss. If there has been a cache miss, and the client can understand Brotli, and there might be a Brotli version of this Content-Type, and we haven’t restarted yet (read on), then we will change the backend URL to a version that appends .br to the end of the file (but before query string parameters if present), and try fetching that.

That is, given a request from the user for: /assets/srv0/svg_icons/asset_version_hash/assets/svg_sprite.js?params

We will change the backend request URL to be: /assets/srv0/svg_icons/asset_version_hash/assets/svg_sprite.js.br?params

Waiting to rewrite the URL until after we have a miss is the trick to avoiding superfluous requests to the backend. If we only change the backend URL, the asset will be cached keyed against the original URL, so we don’t have to know which one was successfully fetched.

If any of the criteria for Brotli are not met, we won’t do anything special at all, continuing on with Varnish’s normal logic flow.

Finally, in custom_fetch, we have gotten a response from the backend. We determine if we tried to find a Brotli file and we failed. If so, restart! That directive tells Varnish to start from vcl_recv all over again. Except the value of req.restarts will be 1 and not 0. We’ll notice that the next time through when we get to vcl_miss, we’ll continue along Varnish’s normal logic flow, and act casual like we didn’t just try to look for a non-existent file.

This does mean the cache will contain a gzip version of the file cached under the Brotli normalized Accept-Encoding. So the next time the same URL is requested, that will yield a cache hit of the gzip version, even if the client does accept Brotli. This is a fine compromise for us because not every file has both versions available, we assume that if there is ever a backend miss, it will always happen. By the time we are fully upgraded to generate Brotli versions of every file, we will let these age out by the version hash in the URL (or purge the assets from cache).

The results

We’ve been running these 6 lines of custom Varnish in production with Fastly since January with no issues, and our users who support Brotli can rest easy knowing that we are continually trying our hardest to keep our site’s static content loading as fast as possible even through multiple asset hash changes each day.

Here is the distribution of encoded body sizes for one of our Brotli-available files as seen by Android browser mobile user agents requesting m.yelp.com:

The relative number of requests by filesize for a file with both gzip and Brotli versions available.

About 80% of them receive the Brotli version of the file, and 20% use gzip. (And though it varies by file type and contents, this file is about 17% smaller).

Well now you know the secret! Please go out and implement this, and help us implement cool tech like Brotli to keep the web modern and fast.

Back to blog