Custom log format to additional logging endpoint


#1

I would like to add a custom logging format and have it sent to a logging destination that is different from my two existing destinations, which were both configured through the web UI. I’m new to VCL and I am not sure the best way to do that.

In my standard generated VCL, inside vcl_deliver there are entries for two log destinations that look something like this:

  #syslog Dest_One
  log {"syslog 6wMdDggKIUzAZy47sjjhW9 Dest One :: "} req.http.Host {" "} req.http.Referer {" "} req.http.User-Agent {" "} req.http.Fastly-Client-IP {" "} {""-""} {" "} {""-""} {" "} now {" "} req.request {" "} req.url {" "} resp.status;
  #syslog Dest_Two
  log {"syslog 6wMdDggKIUzAZy47sjjhW9 Dest Two :: "} now {" "} time.start.sec {" "} req.http.Host {" "} resp.status {" "} resp.http.content-length {" "} req.http.Fastly-Client-IP {" "} req.request {" "} req.url {" "} resp.http.Content-Type {" "} geoip.city {" "} geoip.region {" "} geoip.country_code {" "} geoip.continent_code {" "} geoip.area_code {" "} geoip.postal_code {" "} fastly_info.state;

I assume I need to create a third endpoint in the console, but I want to make sure that it only receives my own custom log format, not a default. How do I do that? There is a community post about vcl_log but I don’t see any reference to that in the generated VCL. Also I don’t want to overwrite my existing logging setup.

Any guidance would be appreciated.


#2

Ok, after some experimentation I figured it out on my own. The community post I linked to above is very helpful, once you understand a few things.

First, create a new logging destination. Set it up exactly how you want it except for the log format, which won’t matter. Then add a condition on that log destination (click the gear, then Conditions). In order to prevent double log lines, you need to add a condition that prevents this handler from ever running. Something like !req.url in the “Apply if” field is good. Now if you look at your generated VCL you’ll see something like:

if( !req.url ) { 
         #syslog Dest_Three
    log {"syslog 6wMdDggKIUzAZy47sjjhW9 Dest Three :: "} req.http.Fastly-Client-IP {" "} {""-""} {" "} {""-""} {" "} now {" "} req.request {" "} req.url {" "} resp.status;
}

The if check there ensures that this handler will not run. Take note of the first part of the log line (the {syslog 6wMdDggKIUzAZy47sjjhW9 Dest Three :: "} part) – you’ll need to copy that part into your custom VCL to ensure that Fastly delivers the logs to the right place.

Now, onto the custom VCL. vcl_log is just another subroutine that gets run. I’m not exactly sure where in the flow it lives because I can’t find much documentation on it, but it gets run somewhere in the Varnish machinery. Fastly’s boilerplate VCL does not define it, which is good, because you don’t end up overloading any of their code.

In it, log however you want, something like:

sub vcl_log {
  log {"syslog 6wMdDggKIUzAZy47sjjhW9 Dest Three ::"}
    {" timestamp="} time.start.sec
    {" host="} req.http.Host
    {" status="} resp.status
    {" bytes="} resp.http.content-length
    {" client_ip="} req.http.Fastly-Client-IP
    {" method="} req.request
    {" url=""} req.url {"""}
    {" content_type=""} resp.http.Content-Type {"""}
    if(geoip.city,{" geoip_city=""} geoip.city {"""},{""})
    if(geoip.region,{" geoip_region=""} geoip.region {"""},{""})
    if(geoip.country_code,{" geoip_country="} geoip.country_code,{""})
    if(geoip.continent_code,{" geoip_continent="} geoip.continent_code,{""})
    {" fastly_region="} server.region
    {" fastly_datacenter="} server.datacenter
    {" fastly_node="} server.identity
    {" fastly_state="} fastly_info.state
    {" duration_ms="} time.elapsed.msec;
}

And then upload that custom VCL and you’re done!

One last thing from that community post that threw me off: it warns you about adding the #FASTLY deliver macro to the vcl_deliver handler. Ignore that. Your custom VCL is not touching the vcl_deliver handler, so you don’t need to worry about that.


#3

Thanks a lot for publishing this info. It really helps to make sense of the original Fastly post on this subject. Really helpful.


#4

This was exactly what I was just trying to do, and it worked like a charm. Thanks!


#6

Worth pointing out that custom VCL might not be necessary anymore with the improved log formatting options available now. See https://docs.fastly.com/guides/streaming-logs/custom-log-formats and https://docs.fastly.com/guides/streaming-logs/changing-log-line-formats for more info.