Content Distribution Networks

Outline
Implementation Techniques
Hashing Schemes
Redirection Strategies

Spring 2002

CS 461

Design Space
Caching
explicit
transparent (hijacking connections)

Replication
server farms
geographically dispersed (CDN)

Spring 2002

CS 461

Story for CDNs

Traditional: Performance
move content closer to the clients
avoid server bottlenecks

New: DDoS Protection

dissipate attack over massive resources
multiplicatively raise level of resources needed to attack

Spring 2002

CS 461

Denial of Service Attacks (DoS)

client

client

attacker
Spring 2002

server
CS 461

client
4

Distributed DoS (DDoS)

zombie
attacker

client
client
server

slave attacker
zombie

client
zombie

Spring 2002

zombie
CS 461

Redirection Overlay
Geographically distributed server clusters
R

R
R

Internet Backbone
R

R
R
clients
Spring 2002

Distributed request-redirectors
CS 461

Techniques
DNS
one name maps onto many addresses
works for both servers and reverse proxies

HTTP
requires an extra round trip

Router
one address, select a server (reverse proxy)
content-based routing (near client)

URL Rewriting
embedded links
Spring 2002

CS 461

Redirection: Which Replica?

Balance Load
Cache Locality
Network Delay

Spring 2002

CS 461

Hashing Schemes: Modulo

svr0

URL (key)

Easy to compute
Evenly distributed
Good for fixed number of
servers
Many mapping changes after a
single server change

svrN

Spring 2002

CS 461

Consistent Hashing (CHash)

url-0

Hash server, then URL

svr0
Closest match
svr1 Only local mapping changes
after adding or removing
servers
Used by State-of-the-art CDNs

svrN

url-1

svr2

Unit circle
Spring 2002

CS 461

10

Highest Random Weight (HRW)

high
URL

weight0
RH

sort

svr0
svr1
svr2
svr
svr
svrN

Spring 2002

Hash(url, svrAddr)
Deterministic order of access set of
servers
Different order for different URLs
Load evenly distributed after server
changes

weight1
weight2

weight0
weightN

low
CS 461

11

Redirection Strategies
Random (Rand)
Requests randomly sent to cooperating servers
Baseline case, no pathological behavior

Replicated Consistent Hashing (R-CHash)

Each URL hashed to a fixed # of server replicas
For each request, randomly select one replica

Replicated Highest Random Weight (R-HRW)

Similar to R-CHash, but use HRW hashing
Less likely two URLs have same set of replicas
Spring 2002

CS 461

12

Redirection Strategies (cont)

Coarse Dynamic Replication (CDR)

Using HRW hashing to generate ordered server list

Walk through server list to find a lightly loaded one
# of replicas for each URL dynamically adjusted
Coarse grained server load information

Fine Dynamic Replication (FDR)

Bookkeeping min # of replicas of URL (popularity)
Let more popular URL use more replicas
Keep less popular URL from extra replication

Spring 2002

CS 461

13

Simulation
Identifying bottlenecks
Server overload, network congestion

End-to-end network simulator prototype

Models network, application, and OS

Built on NS + LARD simulators
100s of servers, 1000s of clients
>60,000 req/s using full-TCP transport
Measure capacity, latency, and scalability

Spring 2002

CS 461

14

Network Topology

WA

PACA

MA

IL

MI

NE
CO
R

R
S

DC
R

SDCA

GA
TX

R
C

R
C

C
C

S Server, C Client, R - Router

Spring 2002
CS 461

15

Simulation Setup
Workload
Static documents from Web Server trace, available at
each cooperative server
Attackers from random places, repeat requesting a
subset of random files

Simulation process
Gradually increase offered request load
End when servers very heavily overloaded

Spring 2002

CS 461

16

Capacity: 64 server case

T h r o u g h p u t r e q /s

Normal Operation
35000
30000
25000
20000
15000
10000
5000
0
Rand
CDR

R-CHash
FDR

R-HRW
FDR-Ideal

A single server can handle ~600 req/s in simulation

Spring 2002

CS 461

17

Capacity: 64 server case

T h r o u g h p u t re q /s

Under Attack (250 zombies, 10 files, avg 6KB)

50000
40000
30000
20000
10000
0
Rand
CDR

R-CHash
FDR

R-HRW
FDR-Ideal

A single server can handle ~600 req/s in simulation

Spring 2002

CS 461

18

Latency: 64 Servers Under Attack

Randoms Max: 11.2k req/s
R-HRW
CDR

R-CHash
Rand

R-CHash Max: 19.8k req/s

R-HRW
FDR

FDR

80

80

60

60

CDFof Response

100

CDFof Response

100

40
20
0

R-CHash
CDR

40
20
0

0.1

1
10
100
Response Time in Logscale (Seconds)

Spring 2002

0.1

1
10
100
Response Time in Logscale (Seconds)

CS 461

19

Latency At CDRs Max: 35.1k req/s

FDR

CDR

CDF of Response

100
80
60
40
20
0
0.1

1
10
100
Response Time in Logscale (Seconds)

Spring 2002

CS 461

20

Capacity Scalability
Under Attack (250 zombies, 10 files)

45000
40000
35000
30000
25000
20000
15000
10000
5000
0

Throughput req/s

Throughput req/s

Normal Operation

16

32

48

64 80

96 112 128

Num of Servers
CDR
R-HRW
Rand

Spring 2002

50000
45000
40000
35000
30000
25000
20000
15000
10000
5000
0
0

16

32

48

64

80

96 112 128

Num of Servers

CDR

CS 461

R-HRW

Rand

21

Various Attacks (32 servers)

1 victim file, 1 KB

10 victim files, avg 6KB

35000

30000
25000

25000

Throughput req/s

Throughput req/s

30000
20000
15000
10000
5000
0

20000
15000
10000
5000
0

100

200

300

400

500

600

700

Num of Zombies (slave attackers)

Rand

R-HRW

Spring 2002

800

100

200

300

400

500

600

700

Num of Zombies (slave attackers)

CDR

Rand

CS 461

R-HRW

CDR

22

800

Deployment Issues
Servers join DDoS protection overlay
Same story as Akamai
Get protection and performance

Clients use DDoS protection service

Same story as proxy caching
Incrementally deployable
Get faster response and help others

Spring 2002

CS 461

23