Analyzing The Performance Impact of edns-client-subnet on CDN
This post is originally on CDNetworks blog but it’s not accessible after their website update last month. Since there are still some people looking for this article even from my blog, here I decided to post a recovered copy from a backup found in archive.org (part1 part2). Originally there were 2 blog posts but here I merged into one. Also fixed some of broken links and recovered lost image.
Original URL (getting 404 as of 2019/2/4):
- Analyzing The Performance Impact of edns-client-subnet on CDN: Part 1
- Analyzing The Performance Impact of edns-client-subnet on CDN: Part 2
1. Analyzing The Performance Impact of edns-client-subnet on CDN
August 29, 2014 - Junho Choi and Jeehoon Kim (Performance Team, CDNetworks)
CDNetworks has been a member of A Faster Internet and added support of edns-client-subnet extension to our platform. In this post I’ll talk about what we have seen recently as well as the actual impact of global CDN performance.
What is edns-client-subnet?
EDNS is basically an extension mechanism of the DNS protocol to overcome the size limit of DNS UDP packets and to carry additional data for DNS queries. In 2011 Google wrote an IETF draft to send Client IP information using the EDNS0 extension and this is usually called edns-client-subnet. Because it is designed to keep privacy, the sender has freedom to limit client IP information. Instead of sending a full IP address, the DNS server is able to send partial information such as /24 only.
What is the impact of edns-client-subnet to CDN?
CDNs largely depend on DNS for its load balancing. There are 2 big implementations of global CDN systems. The first one is to perform load balancing using DNS load balancing. GLB (Global Load Balancer) or GSLB (Global Server Load Balancer) is acting as a DNS server and it returns the best available edge server IP address to the end user. In this system, GSLB should know where the end users is and it depends on the assumption that the end user is using a DNS resolver (local resolver) which is very close to end user in terms of network performance. For example, Comcast cable users automatically use a DNS resolver provided by Comcast during DHCP configuration. The other one is using anycast and geographic load balancing using BGP.
In this case GSLB is not required but edns-client-subnet information can be used for additional information. GSLB only sees the source IP address of the DNS packet as it comes from DNS resolver. A major drawback of this approach is that when the end user configures the DNS resolver, which is far from the actual end user’s location, it will result in inaccurate load balancing. For example if a Comcast user configured his/her PC to use 4.2.2.2 (Level 3’s DNS resolver IP), GSLB will think this user is connecting from Level 3.
This generally happens when the end user incorrectly configures his/her resolver IP address. This was a problem when Google and OpenDNS started their DNS resolver service. These vendors are using anycast for their resolver IP (e.g. 8.8.8.8) and behind this anycast IP there are multiple DNS POPs. However, the DNS POPs are not in every country or network. For example someone using Google DNS from South Korea, will find that there is no Google DNS POP in South Korea, so the DNS query will be routed to nearby Google DNS POP which is Hong Kong or Chinese Taipei. Then, GSLB thinks that this user is coming from Hong Kong or Chinese Taipei and give a sub-optimal edge server IP address to the end user. edns-client-subnet is used to overcome this problem with CDNs and public DNS resolvers. Now Google DNS and OpenDNS (2 large providers who supports edns-client-subnet) send a query with the client IP address (public IP address, not private) to the CDN’s GSLB if supported. This will give extra information for load balancing and CDNetworks’ GSLB uses this information for better load balancing.
Distribution of Google DNS and OpenDNS users in the world
CDNetworks has been participating in Cedexis Radar community for many years. It gives a detailed view of how CDNetworks’ performs in the world. First thing we will do is to see how many requests from Google DNS and OpenDNS are in the Cedexis raw log. There are many fields in the Cedexis raw log but we only need the following data:
- ( Client IP, Resolver IP )
From Client IP we know which country end user belongs to. Also, from the resolver IP we know which resolver IP they used. When the end user sends the DNS query to 8.8.8.8, CDNetworks’ GSLB will not see 8.8.8.8 as the DNS query source IP. The actual DNS request is coming from one of the DNS POPs and we only see the unicast IP of their DNS POP so we need to know what is the unicast IP block of Google DNS and OpenDNS. This information is published in public, so we are able to know which resolver IP belongs to Google DNS or OpenDNS.
The following table is a list of all countries where Google DNS and OpenDNS are more than 10% of the total requests. This data was collected during 8/09/2014 ~ 8/15/2014 and I removed some of countries which have a low volume of samples.
| Country | Google DNS % | OpenDNS % | Others % |
|---|---|---|---|
| Indonesia | 62.4% | 6.3% | 31.3% |
| Vietnam | 53.9% | 6.4% | 39.7% |
| Algeria | 53.8% | 3.5% | 42.7% |
| Bangladesh | 35.9% | 2.1% | 62.0% |
| Tunisia | 32.6% | 0.6% | 66.8% |
| Costa Rica | 27.6% | 1.1% | 71.3% |
| Guatemala | 27.0% | 0.8% | 72.1% |
| Malaysia | 23.0% | 3.4% | 73.7% |
| Turkey | 18.7% | 7.0% | 74.3% |
| Iraq | 21.8% | 1.7% | 76.5% |
| Albania | 19.8% | 1.8% | 78.3% |
| Macedonia | 19.9% | 0.2% | 79.9% |
| Italy | 16.4% | 2.6% | 80.9% |
| Brazil | 16.5% | 2.0% | 81.4% |
| Ukraine | 17.7% | 0.8% | 81.6% |
| India | 14.0% | 3.5% | 82.5% |
| Puerto Rico | 14.2% | 3.2% | 82.6% |
| Egypt | 14.2% | 2.6% | 83.2% |
| Czech Republic | 16.4% | 0.4% | 83.3% |
| Hong Kong | 14.4% | 1.3% | 84.2% |
| Philippines | 12.6% | 2.4% | 85.0% |
| Poland | 13.1% | 1.0% | 86.0% |
| Slovakia | 12.0% | 1.0% | 87.0% |
| Argentina | 11.7% | 1.1% | 87.2% |
| Thailand | 12.3% | 0.2% | 87.5% |
| Pakistan | 11.0% | 0.9% | 88.0% |
| Bosnia and Herzegovina | 11.5% | 0.2% | 88.3% |
| Ireland | 9.7% | 1.8% | 88.5% |
| Ecuador | 10.0% | 1.2% | 88.8% |
| Serbia | 11.1% | 0.1% | 88.8% |
| Russian Federation | 10.6% | 0.4% | 89.1% |
As you can see Indonesia, Vietnam, and Algeria have more than 50% of Google DNS and OpenDNS requests and this is very interesting. Maybe in those countries people manually configured to use Google DNS or OpenDNS, or the ISPs provides those public DNS providers by default without building their own DNS resolver, or their DNS resolver use Google DNS as a DNS forwarder host. In average, 9.6% of requests is from Google DNS and 1.2% is from OpenDNS. In the next article, we will see how CDN performs differently in those countries.
For more information of Google DNS/OpenDNS statistics/performance, please see the following article:
2. Performance impact of edns-client-subnet in CDNetworks
Now we enabled edns-client-subnet support in CDNetworks’s GSLB for Cedexis domains and compared before and after results.
This will give a clear view of what is changed. Note that Cedexis Radar is RUM (Real User Measurement), so the actual sample (end users) is not same when we compared with 2 different periods. However, when we have a large enough number of samples, it’s good enough for comparison.
How edns-client-subnet information Changes Global Load Balancing
One big impact on supporting edns-client-subnet occurs when coming from Google DNS, the load balancing results will be more accurate. For example, let’s say we have an end user in Indonesia who configured Google DNS as a DNS resolver. When the end user sends a DNS request, it will go through one of Google DNS POPs. In this case, most likely the Google DNS Singapore POP will be used because Singapore is the closest (in terms of network routing) location for Indonesian users.
Without edns-client-subnet support, our GSLB will think that this request is from Singapore and return CDNetworks’ edge POP IP in Singapore. Google DNS has a mechanism to detect if DNS servers accept the edns-client-subnet extension so it will not send the client IP information to other DNS servers if it is not supported. In the case of OpenDNS, you need to contact them if your DNS accepts and processes edns-client-subnet information properly.
When edns-client-subnet is properly supported, Google DNS will send edns-client-subnet extension data to CDNetworks’ GSLB. CDNetworks’ GSLB will use this information to perform load balancing instead of the source IP of the DNS query. In this case CDNetworks’ GSLB will see this user coming from Indonesia, not Singapore. Then, CDNetworks’ GSLB is able to return the IP address of CDNetworks’ edge POP in Indonesia.
One caveat is that, as explained in the previous article, it is the DNS resolver’s freedom of what to send to the other DNS. Google DNS and OpenDNS send only /24 of client public IP so CDNetworks’ GSLB is not able to know last octet of client IP. This may create some problem in location accuracy but in general it’s good enough.
Traffic shift by enabling edns-client-subnet
From above table, there are 3 countries which have more than 50% of Google DNS and OpenDNS users: Indonesia, Vietnam and Algeria. Let’s see a traffic analysis for each country to see how the traffic pattern has changed after we enable edns-client-subnet.
We will compare total traffic served by CDNetworks’ POPs in those countries.
Case 1: Indonesia
| POP | edns-client-subnet OFF | edns-client-subnet ON | Diff (%) |
|---|---|---|---|
| Jarkarta-1, Indonesia | 16.13% | 53.60% | +37.47% |
| Jarkarta-2, Indonesia | 5.52% | 24.78% | +19.26% |
| Singapore | 8.28% | 14.09% | +5.81% |
| Taipei, Chinese Taipei | 66.34% | 0.1% | -66.24% |
| etc | 3.73% | 7.43% |
In Indonesia, there is 62.4% of Google DNS and 6.3% of OpenDNS requests. Before enabling edns-client-subnet, more than 70% of traffic was served by our Singapore and Taipei POPs, but after enabling, almost all Taipei traffic has shifted to 2 CDNetworks’ POPs in Jakarta and Singapore and have seen very little increase.
This is a very good example of the impact of routing change. Google DNS and OpenDNS have a Taipei POP which is closet location for Indonesia and that’s why traffic was served from Taipei for Google DNS and Open DNS users.
Case 2: Vietnam
| POP | edns-client-subnet OFF | edns-client-subnet ON | Diff (%) |
|---|---|---|---|
| Hanoi, Vietnam | 22.25% | 46.41% | +24.16% |
| Saigon, Vietnam | 7.81% | 22.16% | +14.35% |
| Hong Kong | 11.38% | 24.09% | +12.71% |
| Taipei, Chinese Taipei | 53.14% | 0.22% | -52.92% |
| etc | 5.42% | 7.12% |
Next to Indonesia, there are almost 60% of Google DNS and OpenDNS requests in Vietnam. Before enabling, Taipei was serving 53.14% of Vietnam traffic. After enabling, Taipei traffic completely moved to 2 Vietnam POPs (Hanoi and Saigon) and Hong Kong, which now serves more than 90% of Vietnam users.
Same as Indonesia, Google DNS has a POP in Taipei which is closet Google DNS POP for Vietnam users. However, it looks like Taipei is not a best location for Vietnam users because the traffic is almost gone.
Case 3: Algeria
| POP | edns-client-subnet OFF | edns-client-subnet ON | Diff (%) |
|---|---|---|---|
| Paris, France | 23.70% | 35.64% | +11.94% |
| Frankfurt, Germany | 20.16% | 30.84% | +10.68% |
| Palermo, Italy | 0.11% | 5.28% | +5.17% |
| Milan, Italy | 7.95% | 12.65% | +4.70% |
| London, UK | 0.89% | 4.74% | +3.85% |
| Amsterdam, Netherlands | 24.10% | 2.28% | -21.82% |
| Madrid, Spain | 22.48% | 0.95% | -21.53% |
| etc | 0.61% | 7.62% |
In case of Algeria, CDNetworks doesn’t have a POP there so actual traffic is distributed to nearby European POPs depending on the latency. Interesting thing is that more than 40% of the traffic has shifted from Amsterdam and Madrid and distributed to Paris, Frankfurt, Palermo, Milan and London POPs.
Performance Changes
Looking at response time changes, Indonesia and Vietnam have a noticeable performance improvement after we enabled edns-client-subnet. There looks to be almost no improvement in Algeria and that’s because CDNetworks doesn’t have a local POP and end users of Google DNS and OpenDNS users are routed to other countries. Paris and Amsterdam are well connected regions so there will not be much change.
Conclusion
From this article, we see that edns-client-subnet extension is useful for CDNs who are doing DNS based load balancing. When GSLB knew about the end user’s location accurately, it does a better job of load balancing by utilizing the client IP information when calculating edge POP and IP, which GSLB will return to end users.
To maximize the impact of supporting this extension, note that 2 conditions are important to see the changes:
- High ratio of Google DNS and OpenDNS users: otherwise you will see very small changes
- CDN has a local POP in the country: when properly supported, traffic will be shifted to a local POP