BIG IP ASM V12

DDOS PROFILE
Global Service Tech Summit, Seattle
Sep, 2015, v3

Lior Rotkovitch, NPI

ASM, L7 DDoS & Analytics

lior@f5.com

ASM DDoS Profile

DDOS - HTTP FLOODS ATTACKS

DDOS - BOTS

From single IP to single URL

Simple bots

From multiple IPs to single fixed URL

Impersonating Bots

From multiple IPs to multiple fixed URLs

Bots with cookies & JS capabilities

From multiple IPs to multiple random URLs

Bots acting as full browser

From multiple IPs from a specific country

Fine Tune your Thresholds & Reporting

F5 Networks, Inc

Reporting

HTTP Floods facts:

Users Or Bots

Legitimate Layer 7 requests

Asking a web page thousands of time instead of one (few) times

Exhausting backend servers resources: memory, CPU, Disk etc

Relatively easy to execute with simple tools

Not easy to detect the offending source nor to prevent it

Wrong identification will prevent valid users from accessing the site (false positive )

User
Source IPs

Unidentified User

Hacktivism

Web Site

Servers

Database

Google Web Bot

Web Bot

F5 Networks, Inc

HTTP Floods types

Users Or Bots

Requests increase from or/and to URLs inside the web site

From single IP to single URL

From multiple IPs to single fixed URL

From multiple IPs to multiple fixed URLs

From multiple IPs to multiple random URLs

From multiple IPs originating from a specific country

User
Source IPs

Unidentified User

Hacktivism

Web Site

Servers

Database

Google Web Bot

Web Bot

F5 Networks, Inc

ASM Detection & Mitigation concept - HTTP Floods

ASM process:
Users Or Bots

1.

Monitoring entities: RPS Latency IPs URLs

2.

Detecting Increase

3.

Activating Mitigation

Source IPs
User
Source IPs

Unidentified User

Hacktivism

RPS

Web Site

Latency

App URLs &

objects

Servers

Database

Google Web Bot

Web Bot

F5 Networks, Inc

ASM Detection & Mitigation concept DoS Profile

Location: Security DoS Protection DoS Profiles dos

F5 Networks, Inc

TPS Based Detection: Transaction Per Second based detection

and mitigation

Client: Hey server, can I get the web page ?

Server

ASM: no, you are sending too many requests.

You will have to

Answer CSID
Answer CAPTCHA
Be Rate Limited / Blocked

F5 Networks, Inc

TPS Based Detection

Monitoring Request Per Second increase form source IP, Geo, URL, Site Wide.
Then apply one of the mitigation policies: CSID, CAPTCHA, Rate limit

F5 Networks, Inc

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

By Source IP: Detection Criteria

Detection: thresholds for determining DDoS attack - by source IP increase
Mitigation: which mitigation will apply on the offending source IP

Detection

Mitigation

F5 Networks, Inc

10

By Source IP: Detection Criteria

Detection Ratio

Ratio thresholds - measuring ratio with two time interval:

Long (History Interval): Measure the last 1 hour RPS averages every 10 seconds
Short (Detection Interval): Measure the last 10 seconds RPS averages every 10 seconds

F5 Networks, Inc

11

By Source IP: Detection Criteria

Detection Ratio

Example:

Long (History Interval): 50 TPS

Short (Detection Interval): 370 TPS
F5 Networks, Inc

TPS increased by: ((370 - 50) /50)*100 = 640%

640% > 500% = True
12

By Source IP: Detection Criteria

At least X Transactions:

Detection Ratio

A minimum condition to
prevent false positive
increase (source IP
starts browsing the site
and goes from 0 to 30
RPS)

TPS increased by %

Example:
F5 Networks, Inc

640%

AND

minimum fix number of transactions

AND

40 =

True
13

By Source IP: Detection Criteria

Detection Ratio
or
Fixed

TPS reached:

Ratio thresholds
ORed with fixed TPS

(TPS increased by %
Example:
F5 Networks, Inc

640%

AND

minimum fix number of transactions) OR TPS reached

AND

40

OR

200

True
14

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

15

Client Side Integrity Defense Concept

Client: Hey server, can I get the web page ?

Server

if a browser:
Yes, Im a browser
User

Web Bot

F5 Networks, Inc

If a bot:
*^lkjdfg@#$

ASM: no, you are sending too many

requests. Are you a browser ?
ASM: ok, you are allowed. Here is the web
page you asked for.
ASM: Bye Bye Blocked

16

Client Side Integrity Defense Concept

Checking JavaScript capabilities
A client is considered legitimate if it meets the following criteria:
The client support JavaScript
The client support HTTP cookies
The client should calculate a challenge inside the JS
If satisfied = legitimate client that can access the site

F5 Networks, Inc

17

Client Side Integrity Defense - Flow

User
First main page access

Browser

DoS Profile

App

HTTP Request (no cookie)

Send JS test

Computational challenge
Solve challenge/
set cookie with time stamp

HTTP Request (cookie)

Reconstruct request
Original HTTP Request
HTTP Response (main page)

HTTP Response (main page)

More object requests (cookie)
Validate cookie: format & time stamp
More object requests
Deliver page

F5 Networks, Inc

More responses

More responses

This is the flow and timeline

of events.
Transparent to the user,
done under the hood
Note that request is held at
the ASM and not arriving the
app until checks are
satisfied
Not all checks are described
here, some are internal IP.

18

Client Side Integrity Defense JavaScript sample

The JS is obfuscated
From user perceptive this
is transparent action.

F5 Networks, Inc

19

Client Side Integrity Defense Mitigation summary

If no reply No problem for us
If didnt solve the challenge but still sending request Block (RST)
If did solve the challenge but:
Cookie is wrong format Block (RST)
Time stamp expired Block (RST)

If client access a resource (image) without getting the cookie first Block (RST)

F5 Networks, Inc

20

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

21

CAPTCHA Challenge - Concept

Client: Hey server, can I get the web page ?

Server

If a user:
OK, I answered
User

If none user:
Ha ? *^lkjdfg@#$
Web Bot

F5 Networks, Inc

ASM: no, you are sending too many

requests. Please answer this CAPTCHA
challenge, show me your human !:

ASM: ok, you are allowed. Here is the web

page you asked for.
ASM: Bye Bye Block him dood !

22

CAPTCHA Challenge

Completely Automated Public Turing test to tell

Computers and Humans Apart

Ultimate solution for identifying human or bot

Send challenge to every IP that reached IP detection criteria thresholds

To CAPTCHA or not to CAPTCHA ?

Some argues that CAPTCHA is not a good usability because an innocent user gets
CAPTCHA and he will not know why. So, remember that a valid user should pass
browser tests. i.e. if a user is blocked (or gets a CAPTCHA) there is a reason and
maybe he is not innocent (infected ?) .
F5 Networks, Inc

23

CAPTCHA customize response

Can be customize to the web site look and feel colors via css
Failure Response page is served if the first attempted fails
F5 Networks, Inc

24

CAPTCHA Challenge - Flow

User
Request mypage.php

Browser

DoS Profile

App

GET /mypage.php (no cookie)

CAPTCHA HTML +JS response
CAPTCHA rendered

Send CAPTCHA

Cookie with time stamp

Solve CAPTCHA
Submit CAPTCHA
solution
GET /mypage.php + CAPTCHA
cookie

mypage.php
rendered

F5 Networks, Inc

HTML of mypage.php

While the system is still in a

state of attack the offending
source will be presented with
another CAPTCHA every 5 min.
Same as CSID, request is held
at the ASM until CAPTCHA is
solved

Verify CAPTCHA solution

Validate cookie
GET /mypage.php
HTML of mypage.php

25

CAPTCHA mitigation summary

If didnt submit the challenge - no request DOSing us
If didnt solve the challenge but still sending us attacks Blocked
If did solve the challenge but:

Cookie is wrong format RST

Time stamp expired 5 min RST

F5 Networks, Inc

26

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

27

Request Blocking / Rate limit

Client: Hey server, can I get the web page ?

ASM: no, you are sending too many requests.

Im limiting your requests sending rate

While CSID and CAPTCHA try to understand who is the offending source (bots or browsers)
request limiting is indifferent to the identity and limits the offending sources.

F5 Networks, Inc

28

Request Blocking

Request Blocking:

Blocking: block all IPs from the offending source if a source IP reached thresholds I dont
want him on my site at this point
Rate Limit: limit the amount of allowed request from the offending source if reached
thresholds I can sustain only some of the traffic at this point
F5 Networks, Inc

29

Request Blocking Mitigation Summary

Block all blocking all traffic from the offending
source (i.e. I dont want to see any more traffic from
this source)
Rate Limit rate limit the offending source

Rate limit will limit to long (history) TPS rate

Example
If long was 50 TPS And increase in short is 150 TPS
Rate limit to 50 TPS

F5 Networks, Inc

30

TPS based: by source IP Summary

Measuring source IP increase

All source IPs that reached the thresholds will be presented with the enabled mitigation
If still increasing , fall back according to the order in the GUI (switching mitigation)

Client Side Integrity Check

CAPTCHA Challenge
Request Blocking

F5 Networks, Inc

31

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

32

HTTP Floods Geolocation detection and Mitigation

Users Or Bots

http floods type: From multiple source

IPs originating from a specific country

User
Source IPs

Unidentified User

Hacktivism

Web Site

Servers

Database

Google Web Bot

Web Bot

F5 Networks, Inc

33

Geolocation - Detection
Geolocation Relative to the whole traffic of the site:
500 % request increase of the whole site from a specific country
AND
At least 10 % of the whole site traffic

F5 Networks, Inc

34

Geolocation Mitigation
All clients requests arriving from the specific country will be presented with mitigation:
Client Side Integrity Check

CAPTCHA Challenge
Request Blocking

(note that blocking will block

all users from this country)
F5 Networks, Inc

35

Geolocation Black n White listing

Specifies the countries that the system always blocks

whenever the system is in a state of DDoS detection.
Done regardless of the thresholds set in the DDoS
profile

F5 Networks, Inc

Allows access to the web site regardless of

geolocation detection criteria thresholds only
i.e. other thresholds still apply

36

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

37

HTTP Floods URL Detection and Mitigation

Measuring requests increase on a URL

Floods types:

Users Or Bots

From multiple IPs to multiple fixed URLs

From multiple IPs to multiple random URLs
Source IPs
User
Source IPs

Unidentified User

RPS

Web Site

App URLs &

objects
Hacktivism

Servers

Database

http://site.com/sell.php
Google Web Bot
Web Bot

F5 Networks, Inc

38

URL Detection Criteria

Collecting RPS on URLs

TPS increase by* AND at least X TPS **

Calculation:
*Ration of long and short
**Minimum TPS thresholds for detection
F5 Networks, Inc

OR TPS reached
39

URL Detection Criteria Mitigation

All clients that access the URL: Client Side Integrity Check
CAPTCHA Challenge

Request Blocking Rate limit (No block all)

F5 Networks, Inc

40

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

41

HTTP Floods Site Wide Detection and Mitigation

Floods types:

Users Or Bots

From multiple IPs to multiple random URLs

Cases where DDoS attack is under the radar

Source IPs
User
Source IPs

Unidentified User

RPS

Web Site

App URLs &

objects
Hacktivism
Servers

Database

Monitoring: all entities

Google Web Bot
Web Bot

F5 Networks, Inc

42

Site-Wide Detection Criteria

Collecting RPS on the entire website (all entities URLs, IPs)
In some cases the floods will avoid thresholds for IP based or URL based.
Site wide provide another layer of detection and prevention

Detection:

Ratio
Fixed

*TPS increase by AND Minimum TPS thresholds for detection

OR TPS reached
F5 Networks, Inc

43

Site-Wide Detection Criteria Mitigation

Prevention polices

All clients that access the site:

F5 Networks, Inc

Client Side Integrity Check

CAPTCHA Challenge
Request Blocking - only rate limit no blocking
44

TPS Based Detection

1.

BY Source IP (Detection & Mitigation Polices)

2.

Mitigation polices:
a)

Client Side Integrity Defense

b)

CAPTCHA challenge

c)

Request Blocking

3.

By Geolocation (Detection & Mitigation Polices)

4.

By URL (Detection & Mitigation Polices)

5.

By Site Wide (Detection & Mitigation Polices)

6.

Prevention Duration

a
2b
c
3

6
F5 Networks, Inc

45

Prevention duration

Escalate top down every 120 second if thresholds are still increasing
Client Side Integrity Check

CAPTCHA Challenge
Request Blocking

F5 Networks, Inc

De escalate - start from the top

Every 7200 seconds

46

Stress Based detection

Predictive Latency predict how long it will take to
serve a new incoming request

ASM: Hey server, how many more requests

can you handle ?
Im the server

Server: Im fine, keep on sending them

F5 Networks, Inc

47

Stress Based Detection and prevention concept

Client: Hey server , can I get the web page ?

Im the server

.
.
After a while

ASM: mmm let me check. The Server can take

additional incoming requests. you are allowed

Client: Hey server, can I get web pages again now ?

ASM: no, my backend latency is now too high
and you are sending too many request.
You will have to:
Answer CSID or
Answer CAPTCHA or
Be rate limit
F5 Networks, Inc

48

Stress Based GUI

Same concept as TPS
based: source IP, Geo, URL,
Site wide and their
mitigation policies.
Addition condition of
backend latency. i.e. only
when the two conditions
reach thresholds, then apply
mitigation policy.

Note: Can work together (operate in

parallel) with TPS based and act as
layers of protection (e.g. TPS based
does only CSID in alert mode and
Stress based does request blocking in
case of latency increase)
F5 Networks, Inc

49

Stress Based Detection & Mitigation

Similar to TPS based,
Quiz yourself, what each item
means ?

1. By Source IP
a) CSID
b) CAPTCHA
c) Request Blocking
2. By Geolocation
3. By URL
4. Site Wide

F5 Networks, Inc

50

Stress Based Detection thresholds condition

TPS
Latency
threshold AND threshold
exceeded?
exceeded?

Mitigation Is activated when

two types of thresholds are
reached :
Latency thresholds
AND

Then:
Activate
Mitigation Policy

F5 Networks, Inc

TPS thresholds

51

Stress Based Detection thresholds condition

TPS thresholds
stress detection
prevention

in order to apply a prevention policy, both TPS and

Latency thresholds must be exceeded, then the enabled
prevention policy is activated.
Latency thresholds are not visible in the GUI, they are part
of automatic detection.
F5 Networks, Inc

Example:
Automatic stress detection enters a
state of exceeding thresholds. This by
itself will not active the prevention.
Only when the TPS thresholds will
exceed then the prevention policy is
activated.
52

TPS based

VS

Quick way to protect against DDoS. Im in trouble

and I want to block now !

Fixed number on the TPS reached is very easy

and useful. Also easy to detect offending sources

Stress based
Allows the option to activate the Mitigation only
when the backed experiencing latency AND RPS
increase (I only want to block when the attack is
causing backend latency)
Provide Layers of defense and notify about backend
issues (not just DDoS)

Conclusion:
TPS based is quick while latency based
allows more granular approach

F5 Networks, Inc

53

Heavy URLs
Not all URLs are equal
Some are more attractive than others

F5 Networks, Inc

54

Heavy URLs
Heavy URLs are URLs that consume more processing resources
from the server
Are good application DoS point - Even few requests can DoS the app
Typical heavy URLs are search box, product IDs
http://site.com/serach.php?q=a
Heavy URL

Servers

Database

Ho wow, this will take a while

Searching
..
hold on
..
Almost there.
F5 Networks, Inc

55

Heavy URLs concept

Automatically measures latency on URLs for 48 hours

and decide who is heavy
When any URL based mitigation is active, the heavy URLs
that was detected as heavy will also get the active
mitigation

F5 Networks, Inc

56

Heavy URLs concept

Heavy URL is another detection capability. Once it is reached the thresholds AND one of the By URL detection
thresholds are reached Then the URLs that consider heavy URLs will be applied with the active mitigation policy
F5 Networks, Inc

57

Heavy URLs configuration

Example: By URL TPS reached 1000 TPS and is currently applying CSID mitigation.
Heavy URL is enabled
F5 Networks, Inc

58

Heavy URLs Configuration

1

1. Automatic Detection - Automatically add URL

that will be considered as heavy

2. Manual Heavy URLs Manually add URL that

will be considered as heavy

3. Ignored URL Exclude those URL from

heaviness

4. Latency Threshold Above this threshold ->

heavy URL

F5 Networks, Inc

59

Heavy URL Reporting

Security Reporting DoS Application URL Latencies

Example:
If search.php is defined as heavy and if index URL is currently being mitigated with CSID because it exceeded the
thresholds of URL reached (or fixed) then every source IP that is accessing search.php will also get the CSID check.
F5 Networks, Inc

60

 Reporting first

Remember, security is a process

Fine Tune your thresholds

Before DDoS Attack
During DDoS Attack

F5 Networks, Inc

61

First rule of detection - AVR Reporting

Know your web site metrics

AVR graphs help you understand the site metrics:

Statistics Analytics Overview
Statistics Analytics Transactions View by

AVR graphs inside ASM tab:

Security Reporting DoS Overview
Security Reporting DoS Application Transaction outcome

F5 Networks, Inc

Sources
IPs
URLs
Site Wide
Geolocation
RPS
TPS
Latency
L7 DDoS measurements

62

Why Fine Tune Thresholds ?

Out of the box thresholds are good for most web sites
Depending on the web site traffic fine tuning thresholds might be needed.
Fine tuning thresholds can be divided into:
Before DDoS Attack
During DDoS Attack

Good for me ??

F5 Networks, Inc

63

Fine Tune Thresholds Before attack

Process:

Pre requisite: Enable DDoS Profile on the desired virtual

1) White list IPs, geolocations countries, URLs (admin) etc

2) Get visibility with transparent mode write down metrics*

3) Test and decide which prevention will apply thresholds exceeds (TPS bases/
Latency based , heavy URL config etc)
4) Fine tune thresholds for fixed and ratio based
5) Switch to blocking When needed

*good list for L7 DDoS metrics

F5 Networks, Inc

64

Fine Tune Thresholds Before DDoS for Source IP

Go to Statistics Analytics HTTP Transaction

View by: Client IP address
List top TPS Avg IPs

F5 Networks, Inc

65

Fine Tune Thresholds Before DDoS for Source IP

By examining the client IP Address you can

conclude the averages of normal traffic you
expect to see from the top source IPs.
Knowing normal averages can help defining
the TPS increase by ratio.
The idea is that you can determine how much
traffic is allowed until assumed a ddos attack.
F5 Networks, Inc

66

Fine Tune Thresholds Before DDoS for Geolocation

Go to Security Reporting dos Application transaction outcome

The same concept works for the

geolocation thresholds graph.

From the drilldown choose Countries on

AVR reports
Which countries you expect to see traffic ?
F5 Networks, Inc

67

Fine Tune Thresholds Before DDoS for URL

The same idea applies to URLs.

Sort graph by URLs
Which URL should have to highest RPS ?

F5 Networks, Inc

68

Fine Tune Thresholds Before DDoS for URL

F5 Networks, Inc

69

Fine Tune Thresholds Before DDoS for Site Wide

On the drilldown choose

Virtual Server

This will help us understand

the over all traffic load that we
have when there is no DDoS
attack.

F5 Networks, Inc

70

Fine Tune Thresholds Before DDoS for Site Wide

Site wide = Virtual server

The overall traffic should be much higher than the other thresholds.
The values reflect the total amount of TPS that the virtual can handle.
F5 Networks, Inc

71

Fine Tune Thresholds During attack

Process:
1) Fine tune white list source if needed
2) Identify sources that exceed thresholds (source IPs, URLs, Geo, SiteWide) by
looking at reporting.

3) Determine the attack type: from fixed/random source IP to fixed/ random URL.
Conclude which of the detection types you need (source IP only ? Source IP and URL
based only ? etc. )
4) Fine tune thresholds according to the exceeding sources (ratio / fixed)
5) Apply mitigation and decide what is working and what is not. Uncheck the
mitigations that are not effective

6) Go to step 1 and repeat

F5 Networks, Inc

72

Fine Tune Thresholds During attack Source IP

Security Reporting dos
Application transaction
outcome

On the drilldown choose

Client IP Address

F5 Networks, Inc

73

Fine Tune Thresholds During attack Geolocation

Security Reporting dos

Application transaction
outcome
On the drilldown choose
Countries

F5 Networks, Inc

74

Fine Tune Thresholds During attack URLs

Security Reporting dos

Application transaction
outcome
On the drilldown choose URLs

F5 Networks, Inc

75

Fine Tune Thresholds During attack - Site Wide

Security Reporting dos

Application transaction
outcome
On the drilldown choose
Virtual Servers

F5 Networks, Inc

76

AVR reports and graphs

Security Event Logs DoS Application Events

Host IP

Time line

F5 Networks, Inc

Attack start / end

Mitigation type can help

understand which of the
mitigation is effective and
when switching mitigation
occurred

Number
of TPS

Attack ID : Clicking
will show graph

77

AVR reports and graphs

Security Reporting DoS Application Transaction Outcomes

High, medium and low

impact allow to filter
high impact attacks
and deal with them
first

impact is the latency

on the backend for all
entities. The higher
the latency the higher
the impact is.
F5 Networks, Inc

78

AVR reports and graphs

Security Reporting DoS Application Transaction Outcomes

Start and End points - red flags indicate the start of an attack and the green flags indicate the end of
an attack. Switching mitigation can occure several time over the DDoS attack.
F5 Networks, Inc

79

AVR reports and graphs

Incomplete Indicates traffic that was dropped by the server because the connection was
incomplete or the server did not respond.

Blocked Indicates traffic that was blocked as a result of the mitigation policy (any of the
prevention policies including bots blocking)

Proactive Mitigation Indicates the amount of time that the proactive bot defense mechanism

was severed
CAPTCHA mitigation Indicates the amount of time that the CAPTCHA challenge was severed
to offending sources

CS integrity mitigation Indicates the amount of time that the client-side integrity defense
challenge was severed to offending sources
BIG IP Response Indicates traffic that is a response to the client from the BIG-IP system.

Cache by BIG IP Indicates traffic that is served from cache configured (WA, RamCache)
Whitelisted Indicates traffic from IP Address that are in the whitelist of the DoS profile
Pass through - Indicates traffic that is pass to the application trough ASM to the server
F5 Networks, Inc

80

AVR reports and graphs

The AVR DoS graph now
shows the thresholds
that are set in the TPS
detection tab.
The Display Thresholds
check box will display
them or clear them from
the graph.

F5 Networks, Inc

81

Fine Tune Thresholds Summary

Before DDoS:
Write the normal thresholds for the web site: (IPs, Geolocation, URLs, Site Wide)
Set the ratio and the fix threshold for each of the above detection criteria (How much the web
site can take 2 times the traffic , 5 times etc)
Test the configuration and the prevention policy, then conclude which one is good for you

During DDoS:

Identify the source IPs, URLs and entire site traffic increase and determine the attack type
Set the fixed TPS number in each of the above criteria and apply mitigation
Verify the results in the Transaction outcome graph

F5 Networks, Inc

82

DDoS Bots - Detection & Mitigation

F5 Networks, Inc

83

Layers of defense against Bots

Simple Bots
Gohogle

Impersonating Bots
Bots with cookies / JS capabilities

This bot section is mostly about

bots that DoS / DDoS. However,
Bots detection and prevention
can be used for various bot
problems the site is experiencing.

Bots acting as full browser

F5 Networks, Inc

84

DDoS Bots
Users Or Bots

Google Web Bot

Bots can be classify in many ways, mostly there are:

1. Simple bots
2. Impersonating Bots
3. Bots with cookies & JS capabilities
4. Bots acting as full browser
Web Site

Web Bot

Unidentified
User

Servers

Database

User

F5 Networks, Inc

85

Enabling Bot signatures protection

F5 Networks, Inc

86

Bots Simple Bot

Simple bot can be any command line tool such as:
curl , wget , ab

Im a simple Bot
Server

ASM: yes, I have your signature. Sorry

mate you are blocked.

F5 Networks, Inc

87

Categorizing Bots
Bad Bots aka Malicious
are well know command
line tools we want
them out

Good Bots aka Benign are

well know search engine and
monitoring tools we want
them in

F5 Networks, Inc

Bad Bots

Good Bots

88

Bot Signatures -

Each categories include:

None ignore
Report report only used for monitoring
Block block

None
Report
Block
None
Report
Block

F5 Networks, Inc

89

Excluding specific bot signatures from category settings

Specific signature can be excluded from the

category setting
Search the signature in Available signature list
and move it to the left pane.
In this example ab tool will not be blocked
even if the category that includes it is in
blocking mode
F5 Networks, Inc

90

First - White list good Bots

3
DNS Server

Google
1

Im a google Bot

Web Server

2
ASM: lets see if you really are. Im doing
Reverse DNS lookup.

Google
Thanks

F5 Networks, Inc

Yes, I see that, please continue.

91

White list good Bots with their domain name

3. The signature includes domain name.

ASM issue Reverse DNS query to verify
the origin of the request
1. Request arrive with User
Agent : Googlebot/2.1

2. ASM search the

google bot signature

4. Once approved ASM will allow the

google bot to access the web site

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

F5 Networks, Inc

DNS Server

Web Server
92

Bot Signature Repository

Security Options DoS Protection Bot Signatures List

Bot Signature repository for

the entire system is under
Options.
Bot signatures repository is
update with the ASM
signature update

F5 Networks, Inc

93

Bot Signature List: general signatures repository

Signatures can be sort by:
Signature Category
Signature Type:
Malicious / Benign
User Define signatures
Yes / No
Partition: signature can
be assign to a specific
partition
Clicking on any of the sorting
will change the order.

F5 Networks, Inc

94

Sorting the Bot Signature Repository

Various filtering

Create new Bot Signature

F5 Networks, Inc

95

Bot Signature Categories

Creating new category

for Malicious or Benign

F5 Networks, Inc

96

Create a new bot signature: simple edit mode

Bot Signature name

Domain name execute

reverser DNS query to
verify origin. Add the
domain if the Bot has one
Category

Simple edit mode: inside

a user agent header or
in a URL.
Create when done
F5 Networks, Inc

97

Create a new bot signature - advanced edit mode

Advanced Edit Mode - rule granularity

For full details consult F5 document

Signature syntax example:

headercontent: "sample_text"; useragentonly;

F5 Networks, Inc

98

Bot signature facts

Signatures associated with a domain name are
validated with reverse DNS lookup.

Blocking and reporting :

Block flag - resets the connection and reports the
action as "bot signature block" with the bot
signature name.
Report flag - Report bot name and categories
(AVR)
Updating bot signature as part of the ASM
signature update

F5 Networks, Inc

99

Bots Impersonating Bot

Gohogle

DNS Server

Im a google Bot
ha ha ha

Gohogle

Bummer

F5 Networks, Inc

Web Server

ASM: lets see if you are. Im doing Reverse

DNS lookup.
ASM: Hey DNS, whos this guy ?
DNS: no one important
ASM: you are not google bot
Bye Bye -> block this creature !

100

Bots Impersonating Bot

3 The real google bot includes domain

name. ASM issue Reverse DNS query to
verify the origin of the request
1 Request arrive with User
Agent : Googlebot/2.1

2 ASM search the

google bot signature

DNS Server

4 If the source IP is not the expected one

according to the DNS query ASM will block
the impersonating bot

Web Server
F5 Networks, Inc

101

Bots with cookies & JS capability

Im a bot that can

understand JS and
support cookies
Ha ?

Web Servers

ASM: prove it, answer my challenges

No you are not, bye bye -> block this bot.

Bummer

F5 Networks, Inc

102

Proactive Bot Defense

PBD is good for:

Bots that can handle JS
Bots that can handle JS and cookies
Bots floods
Under the radar bots
Block any bot accessing the site (humans only web site)

F5 Networks, Inc

103

Proactive Bot Defense and Bot Signature

Proactive Bot
Defense is now
integrated with
the bot
signatures.
When enabling
proactive bot
defense the
bot signature
feature will be
enabled as
well

F5 Networks, Inc

104

Proactive Bot Defense

Send Client side challenges to ALL client and thus mitigate bots all the time

Various challenges are sent and then validate by PBD blocked or allowed

F5 Networks, Inc

105

PBD - Client side integrity defense - flow

User
First main page access

Browser

DoS Profile

App

HTTP Request (no cookie)

Computational challenge
Solve challenge/
set cookie with time stamp
HTTP Request (cookie)

Reconstruct request
Original HTTP Request
HTTP Response (main page)

HTTP Response (main page)

More object requests (cookie)
Validate cookie: format & time stamp
More object requests
Deliver page
F5 Networks, Inc

More responses

More responses
106

Proactive Bot Defense configuration

Always sending CS all the time
During attack only if other component of the dos profile is in dos mode PBD
will send the CS challenge (acting as two layers of mitigation)
This allows second layer of protection (rate limit and PBD)
Grace period - cookie expiration time 300 = 5min
White list exclude PBD on those IPs

F5 Networks, Inc

107

Bots acting as full browser

DNS Server

Im a Bot that
simulate browser

Web Server

ASM: ok, what are your capability ? If you will not

answer right you will have to answer a CAPTCHA
Capability ?
CAPTCHA ?

You are not human, byyyye -> block this unhuman !

Bummer
F5 Networks, Inc

108

PBD Additional bots identification with capabilities script

Bots: Bots acting as full browsers - Browser Simulation

F5 Networks, Inc

109

How bots that simulate

browsers are evaluated ?

Block Suspicious Browsers addition tests are done to understand if this is a bot or a browser. ASM
will evaluate the source and will give it a score:
if the score indicates that the source is a bot it will block it.
If the score indicate uncertainty and if CAPTCHA challenge is checked, then CAPTCHA will be presented
to the source. If answer it is a human if not, blocked.
F5 Networks, Inc

110

Block Suspicious Browsers

If Block Suspicious Browsers is unchecked send CS Challenge

If Block Suspicious Browsers is checked and CAPTCHA is checked send Client
Capabilities challenge and give it a score: If score in doubt send a CAPTCHA for
human verification
If Block Suspicious Browsers is checked but CAPTCHA Challenge is unchecked do
not send CAPTCHA and only block if the score is more than a human
F5 Networks, Inc

111

Client Capabilities -challenge script flow

User
First request GET /sell.php

Browser

DoS Profile

App

GET /sell.php (no cookie)

Client Capabilities Challenge response

Return Client Capabilities
verification

1.
2.
3.

Authenticate and decrypted JS results,

Verify capabilities and set a score
Determine an action based on score

Blank page & Set cookie

Original HTTP Request + cookie

Reconstruct request
HTTP Response (cookie)

HTTP Response
GET /img.png (cookie)

Validate cookie: format

& time stamp
GET /img.png (cookie)

F5 Networks, Inc

112

DoS Bots Reporting

F5 Networks, Inc

113

Bot signatures simulation

Reporting dos Application Transaction outcomes

Transaction outcomes
is very useful for
monitoring traffic and
indicates various
measurements

F5 Networks, Inc

114

Bot signatures simulation

Analytics HTTP throughput request throughput

AVR will provide details on DoS bot signatures (use drill downs )
F5 Networks, Inc

115

Summary
Simple bots can easily be detected and blocked
White listing of bots = visibility to bot access and keep other bots out
Impersonating bots can be monitored / blocked
Bots that support JavaScript and cookies can now be noticed and be blocked
Reporting on the visiting bots to your web site is available via AVR
Custom bots signature is powerful tool to deal with bots
Bots signature is updating via the ASM signatures update

F5 Networks, Inc

116

Resources

Our documentation is free for all. Read and learn more:

BIG-IP Application Security Manager Operations Guide
BIG-IP Application Security Manager: Getting Started
BIG-IP Application Security Manager: Implementations
BIG-IP Application Security Manager: Custom Signature Reference

BIG-IP Analytics: Implementations

F5 Networks, Inc

117