Loggings not combined to log files as expected when using AWS S3 to receive logs

I have Fastly logging setup to send loggings to AWS S3 bucket. I’m seeing each request being logged to separate log file. I expected that one log file would combine loggings of requests occurring during specified period.

For testing I’ve following config:

I’d have expected to see each file to contain loggings for that 300 second period, but seeing each request being logged to separate file. Is my assumption incorrect how this should work and is there way to get it working as I expected it to work?

Documentation (source: Log streaming: Amazon S3) says about period config:

(Optional) In the Period field, enter an interval (in seconds) to control how frequently your log files are rotated. Rotation entails the finalization of one file object and the start of a new one, never removing any previously created file object.

Heya @RandomDeveloper,

Most likely, the reason for this is that the requests are being processed by different Fastly log aggregators (of which we have several spread across the world). How many requests are you seeing per second, in total? If the answer is very low, then it does make sense that you could experience each request ending up in a separate log file. In practice though, it’s not a file per request, it’s a file per aggregator, per time period.

If you push some more traffic, you’ll likely see more than one log entry in each file.

Understood. Thank you for the clarification. I have been testing with low number of requests and saw that requests send couple seconds apart from each other ended up in own log files. This is not optimal for my use case, but I guess it is doable.

I suppose you could write a Lambda function to combine them and trigger it on a schedule. The only other idea that comes to mind is that if you enable shielding, and log only from the shield POP, it’s likely that you’ll get all entries in one file. However, you won’t log edge hits.