The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search Presentation

Performance Related
Changes and their User

Impact
Eric Schurman
Principal Development Lead
Bing

Jake Brutlag
Decision Support Engineering Analyst
Google

Experiments
Server Delays (Microsoft and Google)
Page Weight Increases
Progressive Rendering

Server-side Delays Experiment
Goal
Determine impact of server delays
Methodology
Delay before sending results
Different experiments with different delays
Small number of users
Monitor negative impact
Server Delays Experiment: Results
Strong negative impacts
Roughly linear changes with increasing delay
Time to Click changed by roughly double the delay
D
i
s
t
i
n
c
t
Q
u
e
r
i
e
s
/
U
s
e
r
Q
u
e
r
y
R
e
f
i
n
e
m
e
n
t
R
e
v
e
n
u
e
/
U
s
e
r
A
n
y

C
l
i
c
k
s

S
a
t
i
s
f
a
c
t
i
o
n
T
i
m
e

t
o

C
l
i
c
k
(
i
n
c
r
e
a
s
e

i
n

m
s
)
50ms - - - - - -
200ms - - - -0.3% -0.4% 500
500ms - -0.6% -1.2% -1.0% -0.9% 1200
1000ms -0.7% -0.9% -2.8% -1.9% -1.6% 1900
2000ms -1.8% -2.1% -4.3% -4.4% -3.8% 3100
- Means no statistically significant change
Google Web Search Delay Experiments
A series of experiments on a
small % search traffic to
measure the impact of
latency on user behavior
Randomly assign users to the
experiment and control
groups (A/B testing)
Server-side delay:
Emulates additional server
processing time
May be partially masked by
network connection
Varied type of delay,
magnitude (in ms), and
duration (number of weeks)
Search Traffic Impact
Type of Delay
Delay
(ms)
Experiment
Duration (weeks)
Impact on Average Daily
Searches Per User
Pre-header 50 4 Not measurable
Pre-header 100 4 -0.20%
Post-header 200 6 -0.29%
Post-header 400 6 -0.59%
Post-ads 200 4 -0.30%
Increase in abandonment heuristic = less satisfaction
Abandonment heuristic measures if a user stops interacting with
search engine before they find what they are looking for
Active users (users that search more often a priori) are more
sensitive
d
a
i
l
y

s
e
a
r
c
h
e
s

p
e
r

u
s
e
r

r
e
l
a
t
i
v
e

t
o

c
o
n
t
r
o
l
wk1 wk2 wk3 wk4 wk5 wk6
-
1
%
-
0
.
8
%
-
0
.
6
%
-
0
.
4
%
-
0
.
2
%
0
%
0
.
2
%
200 ms delay
400 ms delay
actual
trend
Impact of Post-header Delays Over Time
-0.22%
-0.44%
-0.36%
-0.74%
d
a
i
l
y

s
e
a
r
c
h
e
s

p
e
r

u
s
e
r

r
e
l
a
t
i
v
e

t
o

c
o
n
t
r
o
l
wk3 wk4 wk5 wk6 wk7 wk8 wk9 wk10 wk11
-
1
%
-
0
.
8
%
-
0
.
6
%
-
0
.
4
%
-
0
.
2
%
0
%
0
.
2
%
d
e
l
a
y
r
e
m
o
v
e
d
Persistent Impact of Post-header Delay
200 ms delay
400 ms delay
actual
trend
-0.08%
-0.21%
Page Weight Experiment
Goal
Determine impact of a heavier page.
Isolate bytes over the wire cost, not
layout costs, etc.

Methodology
Use incompressible HTML comments
Vary size (from 1.05x to 5x page size)
and location of payload
Experiment with payload in individual
and multiple locations
US-only test mostly good broadband
<html>
<head>
<CSS />

</head>
<body>
<answer>...

</answer>
<results>
<result>...

</result>
<result>...

</result>
<result>...

</result>
</results>
...

<script />

</body>
</html>
Page Weight Experiment: Results
Minimal impact for small payloads
Payload at top of page had stronger effect
Performance suffered slightly would have been
worse if tested in regions with poor connectivity
Click metrics impacted more than Query metrics
Largest experiment (approx 5X control page size)
Any Clicks: -0.55%
No changes to query metrics
Results only apply to one GET not multiple
Progressive Rendering Experiment
Goal
Determine impact sending visual header before results.
Methodology
Build page in phases
Send using HTTP 1.1 Chunked Transfer Encoding
Application design impacts
Visual Header - Fast to compute
Results - Slower to compute
Progressive Rendering Experiment:
Results
Metric Change
Performance
Faster across all latency percentiles
4-18% faster to download all HTML
Roughly halved time to see visible page
change
Time to Click ~9% faster
Query refinement +2.2%
Clicks overall +0.7%
Pagination +2.3%
Satisfaction +0.7%
Conclusion
"Speed matters" is not just lip service
Delays under half a second impact business
metrics
The cost of delay increases over time and persists
Number of bytes in response is less important than
what they are and when they are sent
Use progressive rendering

The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search Presentation

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search Presentation

Încărcat de

Drepturi de autor:

Formate disponibile

Performance Related

Changes and their User

S-ar putea să vă placă și