DNS-over-HTTPS performance

More than 79% of websites use encryption in 2019, up from 26% in 2014. However, whilst the vast majority of web browsing traffic (HTTP) may be encrypted nowadays, DNS traffic is almost never encrypted. This means that DNS is still susceptible to surveillance and manipulation.

DNS-over-HTTPS was proposed at the IETF in 2018 as a remedy to this. As the name suggests, this effectively wraps DNS traffic inside HTTPS traffic, meaning that the DNS requests and responses cannot be intercepted in cleartext or manipulated.

The DNS-over-HTTPS proposal has gained traction amongst browser vendors, with both Chrome and Firefox indicating that they intend to adopt it. Firefox already supports it, but it is disabled by default. Chrome and Firefox account for more than 80% of the desktop browser market worldwide.

Desktop browser market worldwide

The DNS-over-HTTPS (DoH) resolvers that the browsers propose to use are not provided by the ISPs. Chrome would likely use Google's DoH resolver, whilst Firefox would use Cloudflare's.

The potential for DNS-over-HTTPS (DoH) to be switched on by default by web browsers, coupled with the fact that the DNS servers in use will be provided by third parties, is what has the ISPs very concerned.

ISP provided DNS (Do53)

Arguments for DoH

The main arguments in favour of the browsers adopting DNS-over-HTTPS (DoH) are simple and obvious:

1. It improves privacy by preventing interception. Whilst most web traffic is encrypted nowadays, you can still tell a lot from DNS requests alone. This can reveal not only what websites you are looking at, but what devices you have in your home. Of course, whomever operates your DoH resolver would have full visibility of your DNS requests, so you have to decide whether you trust Google and Cloudflare more than your ISP.
2. It prevents hijacking of your DNS traffic, and the manipulation of requests and/or responses.

There is a more subtle benefit to DoH that was discovered in a recent paper titled Analyzing the Costs (and Benefits) of DNS, DoT, and DoH for the Modern Web: DNS-over-HTTPS operates over TCP, which can retransmit data very quickly in the case of packet losses, whereas traditional DNS clients use UDP and wait for a fixed time before retrying. So in lossy networks, DoH may outperform UDP-based DNS. However, DoH-over-TCP has head of line blocking (i.e. if there is significant packet loss then all requests over that TCP connection slow down). DoH-over-HTTP/3 - which uses the QUIC under the hood, which in turn uses UDP - would go a long way to resolve this.

Arguments against DoH

ISPs have been the primary opponents to the current DoH deployment plans. The main objection is that the use of Google/Cloudflare as the default DoH resolvers will mean that the ISP's DNS servers will no longer be in the path of their customer's DNS requests. This in turn means that:

1. The ISP will not be able to see customer DNS traffic. Visibility of DNS traffic is what allows many ISPs to perform lawful interception and content filtering (either to comply with legal requirements, or to offer their own parental control products).
2. Performance may be worse for their customers, as their customer DNS requests will now have to travel further to receive a reply (ISP's DNS servers typically operate inside the ISP network, and are therefore closer to the customer than Google/Cloudflare's DNS servers).
3. It will impact the ISP's CDN relationships. Some CDNs rely on DNS to steer traffic to CDN cache nodes operating inside ISP networks. Akamai is the largest example of a CDN that uses this approach (YouTube and Netflix steer traffic towards their caches using application layer logic rather than relying on DNS, so are unaffected by the DNS provider in use).

It is important to note that the 'default-on' deployment of DoH by Chrome and Firefox has not happened yet. It is still a hotly discussed topic. Both browser vendors have indicated that they will not switch it on by default in its current form at this time, in response to the feedback from ISPs.

Measuring performance

Putting aside the privacy topic, we can measure the impact on performance from using DoH.

We have developed support for measuring DoH in our Whiteboxes and Router Agent. The Whiteboxes are dedicated hardware measurement probes deployed on consumer broadband connections globally.

Using both UDP-based DNS (herein referred to as 'Do53') and DoH, using multiple different DNS providers, we are going to study the following performance topics:

1. DNS resolution time for an Akamai hostname;
2. The ratio of responses that steer users towards an Akamai cache inside the ISP's datacenters and IP space ('on-prem');
3. The round-trip time to the resolved IP address of the Akamai hostname; and
4. The web page loading time, including time-to-first-visual and time-to-visually-complete.

We will make comparisons using the following DNS providers:

1. ISP-provided Do53;
2. Google Do53;
3. Cloudflare Do53;
4. Google DoH; and
5. Cloudflare DoH.

The Akamai hostname used in all DNS measurements is a2.w10.akamai.net.

Akamai operate a very large CDN, with caches operating at many layers of the network, and often work closely with ISPs to cache content as near to the user as possible. In their deepest deployments with ISPs, Akamai have caches installed inside ISP datacenters and share their IP space. These 'on-prem' caches benefit everyone: the customer finds content is delivered faster, the ISP benefits from a reduction in transit/peering usage and Akamai can reduce the demands on their centralised infrastructure.

When querying for the hostname a2.w10.akamai.net you will be returned an IP address of the nearest Akamai cache. This may even be an IP address that is within the ISP's IP address space. This Akamai hostname is the one that underpins many large video-on-demand services, including BBC iPlayer (for example, iPlayer uses vod-dash-uk-live.akamaized.net, which is a CNAME to a2.w10.akamai.net). So if an ISP has Akamai caches locally, we can expect a2.w10.akamai.net to resolve to an IP address belonging to that ISP's IP ranges. Whilst we tested this hostname extensively across Europe, we have not assessed its suitability for use in other parts of the world.

It is important to acknowledge the following limitations in our work here:

1. We intentionally do not include the TLS setup time in the DoH results. Firefox typically uses the same TLS session for multiple DoH requests, so the initial TLS connection overhead is amortised across multiple requests.
2. We categorise 'ISP-provided Do53' as whatever is returned to the Whitebox via DHCP. We acknowledge that some users may have reconfigured their CPE to use a different upstream DNS server; ISPs tell us that this is typically <0.1% of their customer-base.

3. We do not look at the performance of DNS-over-TLS (DoT).
4. We are measuring from a very small sample of devices - typically between 2 and 40 Whiteboxes per ISP represented here. Given that DNS services are centralised and we're not trying to compare ISPs against one another here (instead we are interested in their interactions with DNS providers), this is not a significant concern.
5. The results presented here are not intended to be used to compare ISPs against one another - the sample size is far too small. They are intended to provide some comparison between the DNS providers and protocols.
6. We only look at performance to Akamai's CDN. This has been chosen because it's a very large CDN, with deep ISP penetration, and their traffic distribution mechanism relies heavily on DNS. A fuller study would look at a broader set of CDNs including Level3, Limelight, StackPath and others.

Name resolution performance

Our first test looks at average DNS resolution time for the Akamai hostname, split by DNS provider, protocol, and ISP. Table 1 below summaries the results.

Table 1. DNS resolution times for a2.w10.akamai.net, split by DNS provider. All values in milliseconds. Lower is better. Sample size of 299922 tests across 400 Whiteboxes.

The first thing we observe in the results is that there is no significant performance difference (benefit or harm) in using DoH versus Do53. The difference between Do53 and DoH on the same DNS provider is typically only a few milliseconds - certainly within the error margins afforded by this small sample.

We see much larger differences when comparing DNS providers against one another. It might be surprising to see that Cloudflare delivers DNS resolution time on par with the ISP's own DNS servers in most cases, with Cloudflare returning averages of 23.4ms for Do53 and 25.2ms for DoH, whilst ISP Do53 servers averaged 25.6ms. Google delivers the slowest DNS resolution time in almost all cases (48.8ms for Do53 and 51.4ms for DoH).

One likely reason for Cloudflare's strong performance in this metric is that Cloudflare do not support EDNS Client Subnet. This means that Cloudflare can much more aggressively cache DNS responses as their cache does not need to be scoped per subnet. This is discussed in more detail in Analyzing the Costs (and Benefits) of DNS, DoT, and DoH for the Modern Web.

In Europe, a highly developed Internet market, it should not be surprising that all of the DNS providers have broad coverage and can generally deliver low DNS resolution times.

Use of on-prem Akamai caches

Our second test looks at how frequently the various DNS providers will steer our responses towards an Akamai cache located within the ISP's datacenters and IP space. We call these 'on-prem' caches, and rely on the resolved IP address being within the ISP's IP address space to determine its on-prem status. These results are shown below in Table 2.

Table 2. The percentage of responses for a query for a2.w10.akamai.net that were steered to an on-prem Akamai cache. Higher is better (indicates a higher cache utilisation ratio). Sample size of 299922 tests across 400 Whiteboxes.

Before we compare different DNS providers, we should point out that we have removed rows where it was unclear what deployment model Akamai used with the ISP. Akamai will sometimes deploy servers inside ISP datacenters and use their IP space (which we have termed 'on-prem' for the purposes of this post), and this is very easy for us to identify. In other cases they may deploy servers in ISP datacenters but still use Akamai IP space. In other cases still they may colocate servers themselves in a neutral datacenter and use their own IP space, but still have private peering with the ISPs. Without a clear way to distinguish between these last two scenarios, we have focused our results only on the first scenario (which can be clearly distinguished based upon the resolved IP address).

As can be expected, the vast majority of the ISPs sampled clearly had Akamai caches present on-prem. Deutsche Telekom, Telecom Italia, and Vodafone Italy have been removed as they fell into one of the ambiguous cases discussed above.

For ISPs that do clearly have Akamai caches on-prem, we can observe that users' DNS responses are reliably steered towards these caches when using the ISP's Do53 servers, Google Do53 and Google DoH. This conclusively demonstrates that Google's DoH resolver does honour EDNS Client Subnet. Moreover, there is little difference in on-prem cache hit-rate between ISP resolvers, Google Do53 and Google DoH (with a couple of exceptions).

The downside to Cloudflare's decision to not support EDNS Client Subnet can be seen very clearly in the results above. Cloudflare DoH did not steer a single response to an Akamai on-prem cache, for any ISP. Strangely, Cloudflare Do53 appears to behave differently to Cloudflare DoH for a few ISPs - Sky and Vodafone in the UK, and Vodafone in Spain. These three ISPs report a sizeable number of responses from Cloudflare Do53 being sent to on-prem Akamai caches, despite Cloudflare not supporting EDNS Client Subnet. How is this possible? The cause of this is almost certainly the ISPs' CPE hijacking DNS requests and redirecting them to the ISP's own DNS servers. This is perfectly possible on Do53, precisely because the traffic is unencrypted. This is certainly the cause for Sky in the UK, as documented at ISPReview. Proponents of DoH would happily point to the fact that the same hijacking does not work on DoH, as evidenced in the results above.

We can also observe a few ISP-specific behaviours from the table above that would be of interest to network engineers in those ISPs:

1. BT (UK) - Whilst almost 100% of DNS requests made to BT's own Do53 DNS servers are steered to on-prem Akamai caches, only 80% of DNS requests made to Google's Do53 and DoH servers are steered to an on-prem Akamai cache. There's an extra 20% free cache hit-rate available for BT here!
2. Plusnet (UK) - Plusnet are a subsidiary of BT, and they make use of the Akamai caches within BT's network. However, only 68% of DNS requests to Plusnet's Do53 servers are steered to the on-prem Akamai caches, and only 7% of those made to Google's Do53 and DoH servers use the on-prem caches.
3. Sky (UK), Vodafone (UK and Spain) - These ISPs appear to be hijacking Do53 requests to Google and Cloudflare and are redirecting them to their own ISP Do53 servers.

DoH without EDNS Client Subnet (e.g. Cloudflare)

DoH with EDNS Client Subnet (e.g. Google)

Round-trip latency to resolved IP address

Our third test looks at the round-trip latency to the resolved IP address from each of the DNS providers. We do not perform any filtering for whether the Akamai cache is on-prem or not; the results are simply blended. These results are shown below in Table 3.

Table 3. Round-trip latency to the resolved IP for a2.w10.akamai.net using different DNS providers. All values in milliseconds. Lower is better. Sample size of 271660 tests across 400 Whiteboxes.

There is almost no difference between round-trip latency for IP addresses resolved over Do53 and DoH, for a single DNS provider. As with previous charts, the differences are far more closely tied to the DNS providers themselves, rather than the resolution mechanism.

The impact of Cloudflare's decision to not support EDNS Client Subnet is very visible in this chart too. Cloudflare consistently returned IP addresses with higher round-trip latencies than other DNS providers. Moreover, for 9 of the 20 ISPs measured, there was at least a 50% increase in RTT over the best case when using Cloudflare. The most severe cases were found in Greece, where round-trip latency to the Akamai hostname increased from 17-31ms to 104-110ms when using Cloudflare's resolvers.

Web page loading time

Lastly, we look at web page loading time when using DoH versus ISP-provided Do53. This last test was only performed on UK Whiteboxes, and did not use Do53 from Google or Cloudflare (only DoH was used from these providers). The page https://www.bbc.co.uk/news/ was used as our target web page, which is a very popular destination in the UK.

Whilst the BBC uses Akamai extensively, none of the assets on the BBC News homepage were served from on-prem Akamai caches within ISP networks at the time of testing (the on-prem caches are certainly used for other BBC services though, such as videos on iPlayer).

This test used the new Website Performance Test, which uses a real web browser's rendering engine to assess the web browsing performance from the customer's connection.

Table 4 shows the results of this test.

Table 4. Web page rendering time, split by DNS provider. All values in seconds. Lower is better.

The results here show that there is minimal difference in accessing BBC News when using ISP-provided Do53, Google DoH and Cloudflare DoH. The time-to-first-visual varied by at most 10%, with most ISPs seeing differences of less than 3%. The time-to-visually-complete varied by at most 8%. There are no stand-out issues to see here.

Of course, if we instead tested a web page that experienced ~100ms increases in RTT like we saw in Greece with Cloudflare, then that would likely have a much more noticeable impact on web page loading time.

Conclusion

In this blog post we've looked at a range of performance topics related (some quite loosely!) to DNS-over-HTTPS.

A common thread amongst all of our findings was there was no tangible performance difference between Do53 and DoH for a single DNS provider (e.g. Google Do53 performed extremely similarly to Google DoH in all cases). However, much larger differences were found between the DNS providers themselves (regardless of whether Do53 or DoH were used).

We found that ISP-provided DNS servers and Cloudflare's DNS servers (both DoH and Do53) delivered the fastest DNS resolution times. Google's were noticeably slower. This may be due to the fact that Google honour EDNS Client Subnet, whereas Cloudflare does not, allowing Cloudflare to serve more responses from its cache.

We also studied the percentage of DNS responses for an Akamai hostname that steered clients towards Akamai cache nodes located inside ISP datacenters and IP space ('on-prem'). We found that the majority of ISPs tested had Akamai caches on-prem. Using ISP-provided Do53 servers, Google Do53 servers or Google DoH resulted in a high hit-rate for these on-prem caches. However, using Cloudflare's Do53 or DoH servers resulted in almost no traffic being directed to the on-prem Akamai caches. This is not a fault or limitation of DoH in any way; this is caused by Cloudflare's decision not to support EDNS Client Subnet. We also found cases of some ISPs hijacking DNS queries to redirect them to their own servers, which works for Do53 but not DoH.

Perhaps most importantly, we looked at the round-trip latency to the IP addresses returned by the various DNS providers. Again, we found that there was no performance benefit/harm in using DoH versus Do53 here. We did find significant differences between DNS providers though. In some instances, using Cloudflare's Do53 or DoH resolvers increased round-trip latency to Akamai very significantly (by over 100ms).

Lastly, we looked at the web page loading time of BBC News for ISPs in the UK. We found no meaningful difference between using ISP-provided Do53, Google DoH and Cloudflare DoH resolvers in this scenario.

In summary, we have found no significant performance delta between Do53 and DoH. We have, however, found significant performance differences between the DNS providers themselves, which were visible regardless of the resolution mechanism in use.

Purely from a performance perspective, it might be interesting to consider what would happen if Chrome and Firefox enabled DoH by default tomorrow. If Chrome did this and used Google's DoH servers, then our results demonstrate that there would likely be minimal performance difference to the user, and on-prem Akamai cache utilisation would not be materially affected. However, if Firefox enabled DoH by default tomorrow and used Cloudflare's DoH servers, then some users would experience increased latency to Akamai's CDN and Akamai's on-prem caches would effectively disappear. This would mean that more traffic needed to go to Akamai's off-net caches, thus increasing pressure on ISP peering/transit links.

Deployment of DoH is an evolving topic that is still being actively discussed. The authors of Chrome have recently indicated that they may be willing to use ISP-provided DoH servers (we are unaware of any ISP actually support DoH on their own servers currently).

We intend to expand our DNS-over-HTTPS measurement coverage to other parts of the world soon, and will publish further articles on this topic. Get in touch if you have an idea for a measurement that you'd like to see explored!