User behaviour analysis

a.k.a. Usability testing

3-dimensional
framework
•Attitudinal vs.
Behavioral
•Qualitative
(formative) vs.
Quantitative
(summative)
•Context of Use
3-dimensional
framework
•Attitudinal vs.
Behavioral
•Qualitative
(formative) vs.
Quantitative
(summative)
•Context of Use
https://www.nngroup.com/articles/wh
ich-ux-research-methods/
HiPPO
• Highest Paid Person’s Opinion

• http://www.cbsnews.com/news/when-highest-paid-persons-opinion-stomps-on-your-project/
RMIT University©2016 CS&IT 4
ZEBRA
• Z.E.B.R.A = Zero Evidence But Really Arrogant

• https://grasshopperherder.com/in-defense-of-the-hi-p-p-o/
RMIT University©2016 CS&IT 5
Data
•Both HiPPO and ZEBRA has the same problem
•Lack of Data
•But usability testing provides the opposite

RMIT University©2016 CS&IT 6

Usability problems – can you spot it?

Stanstead airport toilet, https://flic.kr/p/9bjhd

RMIT University©2016 CS&IT 7

Usability problems – can you spot it?
•Where do I press to select the pump I want to use?

https://www.flickr.com/photos/olorrain/3114327407/in/pool-everyday-usability/

RMIT University©2016 CS&IT 8

How to get evidence of usability
problems in UI?
•Eye tracking
•Interaction logging
•Those are tools in user testing

RMIT University©2016 CS&IT 9

Conducting Usability
Studies
What are Usability Studies?
•Also known as user testing.
•Measured observation of users’ behaviour as they engage
with your application.
•Uses the scientific methods like metrics, data and
measurements to prove assumptions.
What is the goal of a usability study?
•To measure the effectiveness of a feature set within your
application.
•It all really depends on the development model or the
requirements of the clients.
•To do this you establish metrics such as time taken, or
number of errors during a test.
Usability Testing – examples
Usability testing of fruit
https://www.youtube.com/watch?v=3Qg80qTfzgU

RMIT University©2016 CS&IT 13

Components of a test
•Test plan
•Screener
•Script
•Test resources
•Location
•Equipment
•People
•Post test survey
Creating A Test Plan
•Just like how you need to create a design plan, a usability
test requires a solid plan as well.
•So what is required in that plan?
•Well other than the introduction.
Creating A Test Plan

The 1-Page Usability Test Plan by David Travis (Image Source: UserFocus)
A Good Screener
•This means an excellent survey to identify the exact users
you wanted to test.
•Just like how you need to poll the ideal or targeted users
for information during the design phase.
•Make sure to avoid selection bias in the questions.
Screener questions
•Bad
–List your favourite Web sites
–Do you check online news
every time you surf the Web?
–What do you love most about
Wired News?
RMIT University©2016
• Good
–Write a list of the Web addresses
(URLs) of sites that you go to
often, or that you really like. Write
up to 10 Web addresses.
–How often do you check online
news?
–Several times a day
–Once a day
–Which of these features in Wired
News are important to you? (check
all that apply)
–Number of stories on a given topic
–Number of different topics covered
CS&IT 18
A Tight Script
•During the test you will want to keep it as structured as
possible and read/memorize off a readied script.
•This includes introduction, explaining the test, the
equipment and the tasks involved.
•Important because you will want to provide the same test
environment to all the testers for consistency.
Lab-based user testing
Formal usability lab
• Facility design
–Low-traffic low-noise area or good sound
insulation in walls
–Good acoustics and lighting
–Electrical, ventilation & ceiling
requirements
–Double-pane one-way mirror
• System design
–Digital audio/video/screen-activity
monitor/capture
–Event-logging software
–Editing system

RMIT University©2016 CS&IT 21

The Tasks
•The plan should include each tasks that will be given to the
testers. This ideally should be in the form of a scenario.
•In the plan all the metrics and information pertaining to the
task should be provided.
•For example, ideal time taken to complete task, starting
state, finished states, extra information required, possible
deviations.
Observation Charts/Tables
•Included in the plan should be the observation documents
that will be used for note taking.
•A structured approach will make data analysis less painful
and could easily provide connected themes.
Post Test Surveys
•Aside from observations being the main source of
information, testers can also be provided with a
questionnaire at the end of the survey.
•The results will affirm your observations and can provide
you with Likert scale data.
•Open ended questions here can be used to gain further
insights into the testers’ thoughts.
Conducting the test.
•Each task/scenario should be printed on separate pieces
of paper and be provided to the user one at a time.
•Some users tend to get ahead of themselves or be
confused if you give them a thick stack of documents.
•Be polite, friendly but firm.
Conducting the test.
•Encourage the tester to think aloud; silence is a wasted
test. (Think aloud protocol)
•Vocalize their actions, thoughts and feelings.
•Ensure the testers feel comfortable voicing out their hatred
for your interface.
•Just ask them politely why.
Conducting the test.
•Don’t guide the tester through tasks
•unless it is considered a fail.
•It’s not a training session.
•Impress upon the testers that it’s the application being
tested, not them.

•A usability test is not an opportunity to show off your shiny

program, but to help you get your program shiny.
Conducting the test.
•Tester asks you if that is the right way to complete the
task?
•Reply in the form of another question asking why they
think that is so.
•Tester encounters a problem?
•Ask them why.
Conducting the test.
•Could include a short questionnaire after each task.
•Time each attempt at a task, even if you are not focused
on that aspect.
Usability Testing – examples
Usability testing of Windows 8
https://www.youtube.com/watch?v=v4boTbv9_nU

RMIT University©2016 CS&IT 30

Compiling your findings.
By organizing your comments, findings and measurements
you can come up with justifications for your design or a
change in design.
A very important technique is to identify grouped/themed
issues from multiple comments and data.
Sample findings from eye tracking
•Banner blindness

• https://www.nngroup.com/articles/banner-blindness-old-and-new-findings/
RMIT University©2016 CS&IT 32
Sample findings from eye tracking
•F-shaped pattern for reading web content

• https://www.nngroup.com/articles/f-shaped-pattern-reading-web-content/

RMIT University©2016 CS&IT 33

Remote user testing
Web services for user testing
•http://www.usertesting.com/

•Specify the tasks you want users to perform.

–Test on all devices: mobile, tablet, desktop.
•We recruit participants
–Users are available to test 24/7.
•You get results in 1 hour
Web services user testing
•Let’s see those videos http://www.usertesting.com/
•Another similar service: http://www.trymyui.com/

RMIT University©2016 CS&IT 36

Controlled experiment
Controlled experiment: A/B testing
• From Ronny Kohavi, Microsoft’s General Manager of the Analysis and
Experimentation team & Stanford Professor http://ai.stanford.edu/~ronnyk/
• Concept is trivial
–Set your OEC = Overall Evaluation Criterion.
–Randomly split traffic between two (or more)
versions
–A (Control)
–B (Treatment)
–Collect metrics of interest
–Analyze

http://www.exp-platform.com/Pages/KDD2015KeynoteExPKohavi.aspx
RMIT University©2016 CS&IT 38
The First Controlled Experiment
•The earliest controlled experiment was a test for
vegetarianism, suggested in the Old Testament's Book of
Daniel
Test your servants for ten days. Give us nothing but vegetables to eat and water to drink.
Then compare our appearance with that of the young men who eat the royal food, and
treat your servants in accordance with what you see

http://www.exp-platform.com/Pages/KDD2015KeynoteExPKohavi.aspx

Ronny Kohavi 39
 Real Example: MSN Home Page Search Box
• OEC: Clickthrough rate for Search box and popular searches

Differences: A has taller search box (overall size is the same),

has magnifying glass icon, “popular searches”
B has big search button, provides popular searches without calling them out
• Raise your left hand if you think A Wins (top)
• Raise your right hand if you think B Wins (bottom)
• Don’t raise
Ronny Kohaviyour hand if they are the about the same 40
 Real Example: Bing Ads with Site Links
• Should Bing add “site links” to ads, which allow advertisers to offer
several destinations on ads?
• OEC: Revenue, ads constraint to same vertical pixels on avg

A B
• Pro adding: richer ads, users better informed where they land
• Cons: Constraint means on average 4 “A” ads vs. 3 “B” ads
Variant B is 5msc slower (compute + higher page weight)
• Raise your left hand if you think A Wins (left)
• Raise your right hand if you think B Wins (right)
• Don’t raise
Ronny Kohaviyour hand if they are the about the same 41
User testing in product
development phase
Alpha, Beta testing

http://www.guru99.com/alpha-beta-testing-demystified.html
RMIT University©2016 CS&IT 43
Google Analytics – sample interaction logging

RMIT University©2016 CS&IT 44

Google Analytics – Behaviour Flow

• https://blog.crazyegg.com/2016/01/07/5-ways-expose-split-testing-ideas/
RMIT University©2016 CS&IT 45
Quantitative and Qualitative Research
in Product Development Phase
1.STRATEGIZE: In the beginning phase of the
product development, you typically consider new
ideas and opportunities for the future. Research
methods in this phase can vary greatly.
2.EXECUTE: Eventually, you will reach a "go/no-
go" decision point, when you transition into a
period when you are continually improving the
design direction that you have chosen. Research
in this phase is mainly formative and helps you
reduce the risk of execution.
3.ASSESS: At some point, the product or service
will be available for use by enough users so that
you can begin measuring how well you are
doing. This is typically summative in nature, and
might be done against the product’s own
historical data or against its competitors.

https://www.nngroup.com/articles/which-ux-research-methods/

RMIT University©2016 CS&IT 46

Quantitative and Qualitative Data in UX
research
•Both are useful and complementary
•Qualitative data describes (e.g. give insights to nuance)
•Quantitative data counts (measures the significance)
•UX/UCD research does both
•https://www.youtube.com/watch?v=qjsfblXdocg

RMIT University©2016 CS&IT 47