!

!
!
TO!
Half$day(Tutorials(
1:00(PM(
(
(
(
(
(

Performance!Testing!Services!for!Web!
and!CloudSBased!Architectures!
!
Presented!by:((
(

! Amber!Race!
(
Big!Fish!Games!
!
Brought!to!you!by:!(
!(
!
!

(
!
350(Corporate(Way,(Suite(400,(Orange(Park,(FL(32073((
888$$ 268$$ 8770(FF(904$$ 278$$ 0524($(info@techwell.com($(http://www.starwest.techwell.com/(((
(

((
!

Amber!Race!
!
Amber!Race!is!a!senior!software!development!engineer!in!test!at!Big!Fish!Games.!
After!majoring!in!Asian!studies,!teaching!in!Japan,!and!traveling!the!world,!she!
stumbled!into!software!testing!and!has!been!loving!it!ever!since.!Amber!has!more!
than!fifteen!years!of!testing!experience!at!Big!Fish!and!Microsoft,!doing!everything!
from!manual!application!testing!to!tools!development!to!writing!automation!
frameworks!for!web!services.!She!has!worked!on!a!wide!variety!of!products!and!
written!automation!in!C#,!C++,!Python,!and!Java.!Amber!currently!specializes!in!test!
automation!and!performance!testing!for!high=volume!back=end!services!supporting!
iOS!and!Android!games.!
!
 9/10/18&

GETTING READY FOR THE

BIG TIME

Intro to Load and Performance Testing

Amber Race, Senior SDET, Big Fish Games

INTRODUCTION

Amber Race - @ambertests

1&
 9/10/18&

AGENDA

•  Overview of the general concept and goals

of performance and load testing
•  How to develop a load testing strategy for
*your* application
•  Demonstration of several tools that can help
you analyze and test your applications

Amber Race - @ambertests

BASIC CONCEPTS

Amber Race - @ambertests

2&
 9/10/18&

W H AT I S
PERFORMANCE
A N D L OA D
TESTING?

Testing that covers the

speed and stability of
the application or
service under high-
traffic conditions.

Amber Race - @ambertests

PERFORMANCE VS LOAD TESTING

PERFORMANCE
•  Response time for individual
requests
•  Includes time to render the
page
•  Performance issues on the
client are very visible
•  Lots of variance due to
network conditions
LOAD
•  How the back-end services
perform with a high volume
of requests
•  Includes CPU, memory
usage, and database traffic
•  High load can affect
performance and vice-versa

Amber Race - @ambertests

3&
 9/10/18&

WHICH ONE SHOULD I DO?

Amber Race - @ambertests

TESTING THE CLIENT

Amber Race - @ambertests

4&
 9/10/18&

Visually complete vs actually complete

Image Compression

WHAT TO Caching
LOOK FOR
Content Delivery Network (CDN)

Background Requests

Amber Race - @ambertests

TOOLS

•  Chrome Dev Tools

•  PageSpeed Insights:
https://developers.google.com/speed/pagespeed/insights/
•  WebPagetest https://www.webpagetest.org/
•  Proxies
•  Charlesproxy (MacOS and iOS)
•  Fiddler (Windows)
•  Wireshark
•  Profilers (Android Studio, XCode Instruments)

Amber Race - @ambertests

5&
 9/10/18&

DEMO: CHROME DEV

TOOLS

Amber Race - @ambertests

LOAD TESTING SERVICES

Amber Race - @ambertests

6&
 9/10/18&

Proxies

•  Charlesproxy
•  Fidler

Load Generation Frameworks

TOOLS •  JMeter
•  Gatling

Load Running Services

•  Blazemeter
•  Flood.io

Amber Race - @ambertests

WHAT TO LOOK FOR

•  Slow apis ( > 250)

•  Duplicated calls
•  Call frequency
•  Calls in correct order

Amber Race - @ambertests

7&
 9/10/18&

What APIs are used the most

FIGURING
What APIs are the “heaviest”
OUT WHAT
TO TEST
What APIs are most critical

Amber Race - @ambertests

MONITORING

Monitoring in production is *the best* way to determine how

your application and services are really being used

Amber Race - @ambertests

8&
 9/10/18&

UNDERSTAND THE SYSTEM

•  One monolithic service?

•  Many microservices?
•  Third-party dependencies?
•  Databases?
•  Memcache? Redis?

Amber Race - @ambertests

Session Based
– replicates a Stress – push
part or all of a services to
typical user the limit
session
KINDS OF
TESTS
Benchmark –
Soak tests –
quick tests
run for a long
run in a CI
time
pipeline

Amber Race - @ambertests

9&
 9/10/18&

Location

CLOUD
C O N S I D E R AT I O N S

Scaling VM Type

Amber Race - @ambertests

SERVERLESS TESTING

Amber Race - @ambertests

10&
 9/10/18&

S E RV E R L E S S I S
S E RV E R S , J U S T
IN SMALLER
P O RT I O N S

Amber Race - @ambertests

Throttling: Account vs Function

Cold start issues

KEY
ASPECTS More memory == more $$$

More latency == more $$$

Amber Race - @ambertests

11&
 9/10/18&

JMETER

Amber Race - @ambertests

Java-based, open-source tool

maintained by the Apache
Software Foundation.

Tool for creating multi-

WHAT IS threaded load on web
JMETER? services or applications.

Tool for testing performance

of web services and
applications under varying
conditions.

Amber Race - @ambertests

12&
 9/10/18&

JMeter will not execute javascript

inside HTML

JMETER IS
JMeter can fetch page resources,
NOT A but it will not render images
BROWSER!!!!

Cannot replace tools like Selenium

for client-based UI testing

Amber Race - @ambertests

JMETER SETUP

•  http://jmeter.apache.org/download_jmeter.cgi
•  Requires Java 8 or higher
•  Download zip file and expand anywhere in your PATH.
•  Create shortcut
•  Windows: jmeter/bin/jmeter.bat
•  Mac: jmeter/bin/jmeter
•  Download plugin-manager.jar from
https://jmeter-plugins.org/get/
•  Move jar to the lib/ext folder in your JMeter installation

Amber Race - @ambertests

13&
 9/10/18&

DEMO: BASIC JMETER

TEST PLAN

Amber Race - @ambertests

GATLING

Amber Race - @ambertests

14&
 9/10/18&

WHAT IS Scala-based Uses a scripting

load generation interface rather
GATLING? tool than a GUI

Amber Race - @ambertests

GATLING SETUP

•  https://gatling.io/download/
•  Requires Java 8 or higher
•  Download zip file and expand anywhere in your PATH.
•  Set GATLING_HOME to the main gatling directory
•  Create shortcut
•  Windows: GATLING_HOME/bin/gatling.bat
•  Mac: GATLING_HOME/bin/gatling.bat
•  Recorder is in the same directory as recorder.*

Amber Race - @ambertests

15&
 9/10/18&

DEMO: GATLING TEST

Amber Race - @ambertests

JMETER VS GATLING

JMETER GATLING
•  Java based •  Scala based
•  Graphic Interface •  Scripting interface
•  Mature plugin ecosystem
•  Remote capability

Amber Race - @ambertests

16&
 9/10/18&

RUNNING LOAD TESTS

Amber Race - @ambertests

C ALCULATING LOAD: HOW

MUCH IS ENOUGH?

•  In a normal distribution curve,

your peak hour will have
roughly 20% of total daily
traffic
•  Use the average session length
to calculate the number of
concurrent sessions.
•  Use the number of requests in
a session to calculate the total
number of requests per
second.

http://onlinestatbook.com/2/calculators/normal_dist.html
Amber Race - @ambertests

17&
 9/10/18&

Daily Active Users& 1000000&

Avg Session Length (minutes)& 10&
Concurrent Users! 32900!

Percent Authenticated& 20%&

Percent Anonymous& 80%&
Avg Calls in Auth Session& 30&Calls Per Second - Auth& 0.05&
Avg Calls in Anon Session& 20&Calls Per Second - Anon& 0.03333333&
Transactions per second! 1207!

CALCULATING LOAD

Amber Race - @ambertests

DEMO: HEADLESS
JMETER

Amber Race - @ambertests

18&
 9/10/18&

Remote Distribute within your network

MOAR LOAD Cloud Run tests from the cloud

Blazemeter (supports JMeter and Gatling)

Third Flood.io (supports JMeter and Gatling)
Party
Octoperf (JMeter only)

Amber Race - @ambertests

REPORTING RESULTS

Amber Race - @ambertests

19&
 9/10/18&

CREATING REPORTS IN JMETER

•  JMeter includes a module for generating HTML reports

•  Report properties are stored in jmeter/bin/
reportgenerator.properties file and can be updated in the
user.properties file
•  To create a report from an existing csv file:
jmeter –g <results.csv> -o <output_dir>
•  To include report creation in the test run:
jmeter –n –t <jmx_file> -l <test_log> -e –o
<output_dir>

Amber Race - @ambertests

Average vs Mean vs Median

SO MANY
STATISTICS, SO 3 Nines? 5 Nines?
LITTLE TIME
Sampling frequency

Amber Race - @ambertests

20&
 9/10/18&

AWS CLOUDWATCH

Amber Race - @ambertests

Load data into multiple

listeners to get a better idea
of actual behavior

Beware of coordinated
INT ERPRETING
RESULTS omission, which masks
potential performance issues

Be clear about whether you

are most concerned about
performance or load

Amber Race - @ambertests

21&
 9/10/18&

LET’S PLAY WITH

GRAPHS!

Amber Race - @ambertests

Actual Actual
production production base
hardware load

RUNNING
TESTS IN
PRODUCTION With proper
monitoring, you Be sure to tell
can stop your your ops team
test before it what you are
brings down doing!
production

Amber Race - @ambertests

22&
 9/10/18&

QUESTIONS?

•  Contact me!
•  Email: amber.race@outlook.com
•  Twitter: @ambertests
•  Restful Booker - great for all kinds of API test
practice!
•  Mark Winteringham (@2bittester)
•  https://restful-booker.herokuapp.com

Amber Race - @ambertests

23&