Log request data to Google BigQuery

DevHub comments
1

Build raw JSON strings matching your BigQuery table schema to send log data to BigQuery.

Use this solution in your VCL service (click RUN below to test this solution or clone it to make changes):

This is a companion discussion topic for the original entry at https://developer.fastly.com/solutions/examples/logging-to-google-bigquery
1 Like
2

It took me some time to find out, hence I want to post it here. If you want to write nice access json formatted logs from expressly, similar to what we can do with VCL, update to the latest expressly (2.3.0) and js-compute (3.8.0) library use expressly’s finish callback as follows:

import { includeBytes } from "fastly:experimental";
import { Router } from "@fastly/expressly";
import {
  getGraphQLParameters,
  processRequest,
  renderGraphiQL,
  shouldRenderGraphiQL,
} from "graphql-helix";
import { schema } from "./schema";
import { Logger } from "fastly:logger";
import { getGeolocationForIpAddress } from "fastly:geolocation"

const router = new Router();

router.use((req, res) => {
  let date = new Date(Date.now()).toISOString();
  let geo;
  try {
    geo = getGeolocationForIpAddress(req.ip);
  } catch (error) {
    console.error(error);
  }

  res.on("finish", (finalResponse) => {         
    //Format log data as JSON object
    let log_data = { 
      "@timestamp": date,
      "ip": req.ip,  
      "geoip": {
        "city_name":  geo.city,
        "country_code2": geo.country_code,
      },
      "request": {
        "method": req.method,
        "url": req.url,   
        "host": {
          "subdomain": req.hostname,
        },
        "headers": {
          "accept_language": req.headers.has("accept-language") ? req.headers.get("accept-language") : "",
          "accept_charset": req.headers.has("accept-language") ? req.headers.get("accept-charset") : ""
        }   
      },
      "response": {
        "headers": {
          "location": finalResponse.headers.has("location") ? finalResponse.headers.get("location") : ""
        }
      },
      "referrer": {
        "url": req.headers.has("referer") ? req.headers.get("referer") : ""
      },
      "user_agent": {
        "value": req.headers.has("user-agent") ? req.headers.get("user-agent") : ""
      },
      "status": {
        "code": finalResponse.status
      }    
    };

    let logger = new Logger("access");
    logger.log(JSON.stringify(log_data));
  });
});

The way via the callback is required if you want to log the final status code and redirect locations properly.

We do structured json logging to elasticSearch via logstash and further enrich the data in our log pipelines.

3 Likes