We would like to add the Fastly CDN in front of a website hosted on GCP Google Cloud Platform. It seems that the most compatible configuration (in certain ways) is to have the same hostname on the backend and frontend. That is, the websites themselves respond to www.example.com. The CDN is also configured to be on the domain www.example.com .
Google load balancers support “dns authorization” certificates. That involves added an _acme-challenge CNAME record in DNS. Because the IP address of the website will be pointing at a CDN instead of the load balancer, it’s not possible to use an IP address-based method.
Next step, configure Fastly. The problem is that Fastly certificates use the exact same method, and they conflict.
Warning!
Conflicting record(s) found at _acme-challenge.www.example.com. Please remove the record(s) and add the following CNAME record: 96ra25r.fastly-validations.com
It appears you may only have one _acme-challenge CNAME.
The next method in Fastly is “ACME HTTP/CNAME challenge”. I tried that, however the same error occurs.
Warning!
Conflicting record(s) found at _acme-challenge.www.example.com. Please remove the record(s) and add the following CNAME record: 96ra25r.fastly-validations.com
The next option is “Using an email challenge to verify domain ownership”. Contact support. Perhaps this is the only way.
Any ideas? Can the _acme-challenge conflict be solved somehow?
It is surprising (to me) this topic hasn’t been discussed in this forum already. There were no search results about _acme-challenge. Aren’t most companies installing SSL certificates on the backend and the frontend? If so, how do they solve the problem of validating the same certificates in two places, and avoid a conflict in the process.
Hey @sdarwin, welcome to the Fastly community forum. Great to have you here.
You’re right that we only support a single _acme-challenge record per subdomain for DNS validations. I’m not sure of the reasoning behind this but as I understand, this is common practice across the industry.
If you’re not a fan of using an email challenge, we do offer the ability to validate domain ownership using an HTTP challenge. Documentation for that is available here: Setting up TLS with certificates Fastly manages
Alternatively you could change the hostname of the backend load balancer to something different, meaning you would not have this clash when validating with DNS. This is what most customers tend to do.
In the case that none of these options are feasible for you, I would recommend that you contact support@fastly.com, who will be able to give you some more specific assistance in this matter.
This is done by adding a CNAME pointing to t.sni.global.fastly.net. What appeared to happen is that after adding that CNAME, but with the _acme-challenge from Google still present, the result is immediately "Warning! Conflicting record(s) found at _acme-challenge " . The DNS challenge takes precedence and prevents the next challenge from succeeding.
Right.
There is a lucky trick which seems like a hack, but may be ok. Google will configure SSL on a base domain + a wildcard. So, example.com and *.example.com. Then set up Fastly at only www.example.com . In that case, the certificate requests between Fastly and Google will be sufficiently different to bypass the error.
CDNs are in a unique position, that they need to provision SSL certificates overlapping with the backend servers and potentially causing a conflict.
10 or 20 years ago, Let’s Encrypt didn’t exist. You purchased certificates every 2-3 years from a vendor. In that scenario, the request process was out-of-band. No problem. However now everything is migrating to automatic DNS based solutions. The latest iteration in Google Cloud is shifting responsibility to their “Certificate Manager” and a recommend method there is “dns authorization” which interferes with Fastly certs. Here are ideas to improve the situation.
Enable Fastly’s HTTP challenge to work: modify the HTTP challenge so it won’t fail in the presence of another vendor’s _acme-challenge CNAME. Then you can use _acme-challenge for the backend, and HTTP challenge for the frontend, simultaneously.
(possibly) Allow a particular certificate to switch from DNS challenge to HTTP challenge in the future, during renewal. When first setting things up, you could juggle those acme-challenge records, adding and deleting them, until the certs are installed successfully. Inconvenient but not impossible. However you wouldn’t want to do that every 3 months. For renewals, it should work with (1) HTTP challenge, even if it had been set up earlier using a DNS challenge.
(the best) Instead an “_acme-challenge” record, create a customized method that supports “_acme-challenge-cdn” or “_acme-challenge-fastly” CNAME records. It removes the collision, and allows the necessary certs to be generated.
You are correct that others in the industry must have this problem as well. Even more complex with customers that use multiple CDNs. The root of the problem is that DNS only supports on CNAME. The industry is addressing the issue however with a new ACME challenge type, DNS-ACCOUNT-01 It’s still in draft, but we expect to implement this solution in the first half of 2024. You can read about it here:
Great! The introduction of that article describes the problem…
The dns-01 challenge specified in section 8.4 of [RFC8555] requires
that ACME clients validate the domain under the _acme-challenge label
for the TXT record. This unique label creates an impediment limiting
the number of other entities domain validation can be delegated to.
Pardon me for jumping in here, but I’ve been using Let’s Encrypt for many years for my own personal stuff and I’ve never left _acme-challenge records ‘laying around’. They get created during the issuance of a new cert, or renewal of a cert, and then removed, all by the tool I use for managing certs via LE and ACME.
Is this not the normal situation? From this topic’s comments it appears that we have customers who leave these records in place for long periods of time, and I thought the contents of the records were relatively short-lived and there wouldn’t be any value in leaving them around…
I had been assuming, but without really investigating this topic, that each time the renewal occurs they re-check the presence of those same TXT or CNAME records, and would expect those challenges to be in-place. Otherwise, you would have these certs in perpetuity, without any more verifications done. Even 20 years later. Permanent certificates. That would be slightly surprising.
On the other hand, if they are not infinite duration, verification reoccurs. then you are back to this problem under discussion. But, not certain.
This is CA-initiated issuance/renewal, where the CA does all the work and stores the resulting key/cert for you (which then means they are responsible for TLS termination for the domain name that uses the cert as well). This makes sense in a CDN-that-is-also-a-CA situation as is the case for Fastly and others, and doesn’t require that the CA have the ability to manipulate records in the DNS zone where that domain name lives.
My systems use the owner-initiated issuance/renewal model, where the end system initiates the process (creating a key, requesting the cert, etc.) and stores the results locally. This model uses the ‘regular’ mode of DNS-01 challenge, where the challenge record is supplied by the CA during the issuance process, the end system pushes it into DNS, the CA checks in 60 seconds (or so) to verify that the record exists, issues the cert, and then the end system removes the challenge record. These systems are not behind a CDN, clearly
I’m also struggling with the same issue, where Google LB set a TXT record, but the CNAME record actually takes presedence, and as such if I register a Fastly _acme-challenge record, Google will not renew certs. Google is actually able to automatically create and remove the TXT records, but this is not the case for Fastly.
Is the only solution to avoid manual labor use a good old “long lived” certificate on one of these services then?
You are correct, at the moment it is difficult to use ACME to issue certificates for the same domain from multiple CAs, as the challenge records are not unique. There is a new proposed challenge type (DNS-ACCOUNT-01, IIRC) which will address that problem, as the challenge record names will be constructed using a hashed account identifier representing the account at each CA. This is still a proposal and hasn’t been implemented in any ACME servers or clients yet, but it’s coming.
In your case “Google is able to automatically create and remove the TXT records” implies that you are using Google’s DNS services for the domain; that certainly makes things easier, but doesn’t help the multiple-CDN+CA situation as only one of the CDN+CAs can also be the DNS provider. If the DNS provider supports dynamic updates using RFC 2136 (DNS UPDATE) then some (or all) of the ACME clients in use by the CDNs could use that to add and remove challenge records as needed, but Fastly does not offer that sort of mechanism and given the introduction of DNS-ACCOUNT-01 to the ACME protocol there wouldn’t be much value in adding RFC 2136 support to the Fastly ACME client.
I should also note that if a customer is using two CDN+CAs, only one of them has to support DNS-ACCOUNT-01 for the problem to be resolved; the other can continue using DNS-01, as the TXT record names will not conflict.
