Here is a step-by-step guide to configuring your robots.txt file through the Fastly UI:
Go to the Content section of your service and click ‘New’ next to Responses. Complete the fields in the dialog as you see fit and include your customized robots.txt wording into the response body. Refer to the following example:
Your newly created response should appear in the list of Responses. Next, click the gear menu and select Request Conditions.
Complete the Request Condition fields and include the req.url condition which applies to the location of your robots.txt file, similar to below:
Click “Create” and you’ve completed the steps to creating a custom response for your robots.txt file.
For a more in-depth explanation of creating custom responses, please checkout Fastly’s Responses Tutorial.
###Why can’t I customize my robots.txt file with ‘global.prod.fastly.net’?
Adding the <.global.prod.fastly.net> extension onto your domain - as in www.example.com.global.prod.fastly.net - via the browser or in a cURL command is used to test how your production site will perform with Fastly’s services. In order to prevent google from accidentally crawling the test URLs, we provide an internal robots.txt file.
This file cannot be customized via the Fastly UI if you have not CNAMEd your domain to global.prod.fastly.net.