What is CloudFlare? CloudFlare makes websites faster and safer using our globally distributed network to deliver essential services to any website: • Performance • Analytics • Content Optimization • Third party services • Security www.cloudflare.com 2
How does CloudFlare Work? CloudFlare works at the network level. • Once a website is part of the CloudFlare community, its web traffic is routed through CloudFlare’s global network of 23 (and growing) data centers. • At each edge node, CloudFlare manages DNS, caching, bot filtering, web content optimization and third party app installations. www.cloudflare.com 3
IPv6 Gateway With the Internet's explosive growth and the number of on-net devices closing in on IPv4's maximum capacity, CloudFlare now offers an automatic IPv6 gateway seamlessly bridging the IPv4 and IPv6 networks. • For most businesses, upgrading to the IPv6 protocol is costly and time consuming. • CloudFlare’s solution requires NO hardware, software, or other infrastructure changes by the site owner or hosting provider. • Enabled via the flip of a switch on the site owner’s CloudFlare dashboard. • Users can choose two options: (FULL) which will enable IPv6 on all subdomains that are CloudFlare Enabled, or (SAFE) which will automatically create specific IPv6-only subdomains (e.g. www.ipv6.yoursite.com). www.cloudflare.com 4
Anycast CDN Anycast prefixes • Same IP Prefixes advertised in each site. • 23 Sites around the world. • No unicast used for content delivery • Unicast used to pull traffic from “origin” and management Traffic Control • How the eyeball ISP routes • ISP A routes to CloudFlare in Hong Kong, traffic will be served for ISP A from Hong Kong. www.cloudflare.com 6
Anycast CDN Traceroute from Singapore:
traceroute to 18.104.22.168 (22.214.171.124), 30 hops max, 40 byte packets 1 202-150-221-169.rev.ne.com.sg (126.96.36.199) 0.351 ms 0.406 ms 0.456 ms 2 s4-6-r10.cyberway.com.sg (188.8.131.52) 0.610 ms 0.652 ms 0.692 ms 3 anutsi10.starhub.net.sg (184.108.40.206) 2.579 ms 2.575 ms 2.562 ms 4 six2utsi1.starhub.net.sg (220.127.116.11) 1.452 ms 1.633 ms 1.768 ms 5 SH.gw5.sin1.asianetcom.net (18.104.22.168) 1.561 ms 1.620 ms 1.610 ms 6 te0-0-2-0.wr1.sin0.asianetcom.net (22.214.171.124) 2.135 ms 1.921 ms 1.950 ms 7 gi4-0-0.gw2.sin3.asianetcom.net (126.96.36.199) 1.909 ms 1.907 ms 1.882 ms 8 CDF-0003.gw2.sin3.asianetcom.net (188.8.131.52) 1.417 ms 1.504 ms 1.493 ms 9 cf-173-245-61-248.cloudflare.com (184.108.40.206) 1.470 ms 1.461 ms 1.520 ms Traceroute Completed.
Traceroute from Hong Kong:
From traceroute.hgc.com.hk to 220.127.116.11
traceroute to 18.104.22.168 (22.214.171.124), 64 hops max, 44 byte packets 1 bbs-1-250-0-210.on-nets.com (126.96.36.199) 0.423 ms 0.329 ms 0.320 ms 2 10.2.193.17 (10.2.193.17) 0.719 ms 0.661 ms 0.682 ms 3 peer (188.8.131.52) 0.569 ms 0.550 ms 0.545 ms 4 cloudflare-RGE.hkix.net (184.108.40.206) 1.893 ms 2.419 ms 1.910 ms 5 cf-173-245-61-248.cloudflare.com (220.127.116.11) 2.101 ms 1.973 ms 1.780 ms www.cloudflare.com 7
Transit Who? • Choice of Transit Provider is VERY important • We’ve chosen a limited number of providers per region: • Two in US/EU • Two in Asia. • Single Provider makes routing easier, but two for full reach • Transit provider should offer good routing controls • You need to be able to keep routes within a region • Prepend to specific peers • Transit Provider should make use of “Hot Potato” routing to their peers. • ie. Peer and exchange traffic in every mutual location.
Transit Routing Controls? • Must be able to keep advertisements within region. • A Customer of your European transit provider is likely to be a peer of your Asian Transit provider • You don’t want to serve traffic from Asia for Europe • Some routing controls listed on: http://www.onesc.net/communities/ • A lot of work should be done in the presales stage to understand the providers network and how they peer. • Looking at AS1299’s (Telia-Sonera) whois entry gives a good idea how they peer. www.cloudflare.com 11
Transit Choices? • Many providers can give you good coverage for common US and EU Locations (San Jose, LA, New York, London, Amsterdam, etc...) • One provider can’t do it all in Asia. • Asian networks are usually somewhat ‘disconnected’ • Few peer with NTT in Asia • NTT, Pacnet, TATA all disconnected from each other. • Transit in the US could be far cheaper for the provider than within Asia. • Supplement this with Peering www.cloudflare.com 12
Peering USA Peering • Is it economic to peer? • Transit is < US$1 • Eyeball networks probably *wont* peer with you • Comcast (not at any exchange) • ATT • South America? • Peering in Miami • Most networks open to peering www.cloudflare.com 14
Peering EU Peering • Same argument as US, might be more costly to peer • Many networks more open to peering however • Major providers / incumbents more difficult, probably wont peer: • DTAG • TeliaSonera • Telecom Italia Sparkle • Telefonica • France Telecom • IX’s have good reach to surrounding regions. • AMS-IX, DE-CIX, netnod, LINX
Peering Asia Peering • Very economical. • Large providers may not peer • HKIX……. (and Hong Kong Equinix) • Local Loop to HKIX can be around US$1,000 for 1G • No IX charges. • HKIX will get you 100% of domestic Hong Kong. • Very Good Vietnam and some Taiwan, Korea, Japan and China routes too • Singapore Equinix • Priced competitively • Great Coverage for South East Asia (Indonesia, Thailand, Malaysia, India) • JPIX and JPNAP much more costly. www.cloudflare.com 16
Peering Asia/Hong Kong Peering Economics • Transit ~US$10/M (HK) • HKIX US$2/M at 50% utilisation • Worthwhile after just 100mbit utilization. • If transit is more expensive, justification comes faster! • Hong Kong users will be upset if you don’t peer at HKIX • Quoting a customer:
“Having a PoP in HKG without announcing prefixes to HKIX is like a dinner without fork.”
Peering Asia/Sydney Peering Economics • Transit ~US$30+/M • Equinix/PIPE ~US$2/M at 50% utilisation • Worthwhile after around 30mbit. • Around half the “Eyeballs” Connect to Equinix or PIPE in Sydney, WAIX in Perth. • No Telstra, Optus. (or AAPT/Verizon)
Challenges Challenges • Routing • Inefficient routing, optimizing. • Turning up peering, causing unexpected routing changes • Russian Network preferred our routes via HKIX instead of in Europe. • Keeping optimal routing to Eyeball Networks • Deployments into new markets • China • South America.
Challenges SOME NETWORKS DO STRANGE THINGS! • An Israeli ISP is doing per-packet load sharing over multiple ISPs • A SYN will connect in Amsterdam • Amsterdam anycast node replies with SYN-ACK • Washington DC receives ACK. • TCP IS BROKEN! • Troubleshooting is not easy. www.cloudflare.com 21
Challenges How to troubleshoot? • Whole new techniques • Ping is wonderful tool. • Ping from Anycast IP, to determine if remote side is reachable from that node. • Ping from Unicast IP, to determine if remote side is reachable.