Usability Testing: A Quick, Cheap, and Effective Method

Jerilyn Prescott Matt Crichton

University of Portland University of Portland
5000 N. Willamette Blvd 5000 N. Willamette Blvd
(503) 943-7469 (503) 943-7469
prescott@up.edu mcrichto@up.edu

ABSTRACT Thus, usability testing was initiated.

The University of Portland Office of Computer and The usability testing was based on Rubin’s work. His definition of
Telecommunication Services (CTS) created an Intranet web site usability testing is, “a process that employs participants who are
for the University community during the summer of 1998. In the representative of the target population to evaluate the degree to
summer of 1999, a usability test was developed in an effort to which a product meets specific usability criteria [1].” Rubin
design a more user-centered web page. CTS has a limited budget describes six basic elements of usability testing:
and no usability testing facilities. Also, the usability testing 1. Development of problem statements or test
needed to be performed in a relatively short time so that indicated objectives…
changes could be completed over the summer break. These 2. Use of a representative sample of end users
conditions made it necessary to develop a quick, cheap yet which may or may not be randomly chosen.
effective usability test.
3. Representation of the actual work environment.
Keywords 4. Observation of end users who either use or
Usability Testing, User-Centered Design, Web Pages, Readability review a representation of the product.
Controlled and sometimes extensive
1. INTRODUCTION interrogation and probing of the participants by
In the fall of 1998, the University of Portland Office of Computer the test monitor.
and Telecommunication Services (CTS) implemented an Intranet
5. Collection of quantitative and qualitative
web site. The stated purpose and goals of the Intranet site was to
performance and preferences measures.
provide a place for the faculty, staff, and students to find
information regarding computing and telecommunication 6. Recommendation of improvements to the
technology and services at the University of Portland. design of the product.
Student workers, under the guidance of the CTS staff, created the There were three constraints on our usability test. First, the
CTS Intranet site. They thoroughly researched the design of web usability test had a small budget. Second, the testing would be
sites and drew their final design plans from many highly regarded limited to the equipment on campus. The last constraint was time.
resources. Their effort met the initial goals of the project in that it The research, development and implementation needed to be done
had a great deal of useful information. However, the University in a short period of time so changes could be made during the
community was not utilizing the site for two reasons. One, it was summer break. These three constraints were surmounted in the
not advertised enough. Through a survey, it was found the users design of a usability test.
did not know the CTS web site existed. Second, the users who
visited the site did not find it useful and did not return. Another 2. DEVELOPMENT OF USABILITY TEST
approach was needed to make the site successful. A decision was Using Rubin’s six elements, we created the following usability
made to include the users in designing the CTS web site and use a test.
user-centered design approach.
2.1 Development of Problem Statement
User-centered design requires a change in attitude. Jeffrey Rubin A key element of designing an effective usability test is to have a
[1] defines user-centered design (UCD) as, “not only the concrete purpose and problem statement or objectives. The
techniques, processes, methods, and procedures for designing purpose and objective guide the rest of the testing procedures. Our
usable products and systems, but just as important, the philosophy stated purpose and problem statement/objectives were:
that places the user at the center of the process.” Instead of trying
to guess about the users’ needs and abilities, we asked them. Purpose
The purpose of the usability testing is to determine if the
CTS web site is easy to navigate for the average user. It is
hoped that the data collected within these tests will make
Permission to make digital or hard copies of all or part of this work for personal or
the web site easier to use and the process of finding
classroom use is granted without fee provided that copies are not made or information faster and less frustrating.
distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. To copy otherwise, to republish, to post on servers
Problem Statement/Objective
or to redistribute to lists, requires prior specific permission and/or a fee. 1. Are there potential navigation and organizational
SIGUCCS 99, Denver, Colorado problem areas within the site?
1999 ACM 1-58113-144-5/99/0011..$5.00
2. Is there any confusing terminology within the site?

176
2.2 Representative Sample of Users
It is important to identify the target audience and then recruit
representatives from that audience to participate in the usability
test. It does not take many participants to identify trends. Research
indicates only five to seven participants are enough to show
trends.
The computer skills of the University of Portland’s community
vary greatly. For convenience, we grouped users into three
categories according to skill level:
• Beginner: A user who has never been in front of a computer or
has extremely limited experience.
• Intermediate: A user who is comfortable with the basic
Windows/Mac operations.
• Advanced: A user who has above average knowledge of
computer usage.
A pool of candidates, using personal knowledge of users’
abilities, was created and six participants were recruited (no
students were tested due to summer break). This was an accurate
representation of the University faculty and staff (see table 1). No
incentives were given to participate other than improving the web
site.
Level Faculty/Staff Male/Female
0/1 0/1
Beginner
Intermediate 1/2 1/2
Advanced 1/1 2/0
Total 2/4 3/3
Table 1. Demographics of Participants
Note: Some universities may require nondisclosure
agreements and consent from participants.
2.3 Test Environment
The testing environment will determine if you have a cheap or
expensive test. Many believe that a usability test lab with data
capturing and video equipment is required for the best results.
However, that is not necessary for an effective test as proven by
our usability test.
In most cases, the test was done in the participants’ office, using
their computer. Otherwise, it was done in either a computer lab or
classroom. The only additional equipment was a tape recorder.
This enabled the administrator to review and clarify data after the
test was completed.
2.4 Observations of End Users
2.4.1 Role of the Administrator
In the test, the administrator performed various roles, including
that of administrator, monitor and observer. Thus, it was very
important to state exactly what the administrator should and
should not do. For consistency, one administrator was used for all
the tests. The following defined the role of the administrator:
The person administering the test will be sitting close to
the participant in order to hear any questions the
participant might have. Careful attention will be given to
where the mouse goes on the screen as the participant
navigates within the site. The administrator will not
177
initiate any conversation, unless he or she senses the
participant is having difficulty and is in danger of not
being able to continue the test. The administrator will then
offer encouragement.
The administrator can speak to the participant if the
participant has a question, but the administrator may not
give hints as to where the answers to the questions are
located in the web site.
The administrator will record any and all observations on
a sheet of paper that the tape recorder cannot distinguish.
The administrator will demonstrate any task the
participant cannot complete. After testing, the
administrator will ask the participant some focused
questions designed to extract more information and get
the participant’s opinion of the web site.
2.4.2 The Test
The test consisted of three parts. Before the test was given, the
administrator read the following instructions:
This is not a test of you, but a test of the web pages. You
cannot fail; only the pages can fail. Be verbal in
expressing what you are thinking as you are searching for
the answers. There are a total of ten questions. Explain
what you are thinking as you are searching for each
answer. If you cannot find the answer to a question,
explain where and why in the web site you went in an
attempt to find the answer. Please try to find the answers
to the questions in the web site, even if you already know
the correct answer. Don’t spend too much time on one
problem. Skip it if you find it frustrating. Try to explain
what was frustrating and what you were thinking when
you were trying to find the answer. Please be honest in
your answers even if you have some negative comments.
Negative comments will help us more than only positive
comments. Remember to verbalize your thought processes
as you navigate within the web site.
2.4.2.1 Terminology
The first part was to test the use of terminology. A series of words
used on the Intranet site was given, and the participants were
asked to explain what they thought they would find if they clicked
on the terms. Some of the words included Resnet, Help,
Telecommunications, and Contact Us.
2.4.2.2 Navigation/Organization
The second part had the participant using the web site to answer
specific questions. The questions were designed so that the user
had to navigate throughout the site and down many levels. A few
of the questions were:
• What are the current hours of the help desk?
• How do I set tabs in Mircosoft Word?
• What phone number do I dial with my modem to connect to the
University if I live off campus?
2.4.2.3 Overall Opinion
For the final part of the test, the administrator asked focused
questions. As stated in the role of the administrator, these
questions were designed to get more information regarding the
site and to get the participant’s overall opinion of the site.
Examples of the focused questions included:
• What do you think about the web site?
• Did any link take you to an unexpected place?
• Did you find all the information you were looking for in a
timely manner?
• Did you always know where you were within the site
structure?
• What do you consider the most valuable part of this site?
2.5 Collection of Data
The administrator collected and tabulated all of the data. A set of
evaluation measures and a recording sheet were developed. After
the test was completed a summary of the results was made with
the following information:
• Number of tasks completed incorrectly
• Number of tasks skipped
• Major problems/obstacles associated with each task
• Observations/comments concerning each participant’s actions
2.6 Recommendations
Many recommendations were garnered from the usability test.
Several major trends were identified:
• Participants wanted to get to the answer as quickly as possible
• Participants thought there was too much plain text to scan
through
• Participants wanted all the important information on one
screen with easy to read links, bullets and numbers
• Participants wanted terms that beginning users would
understand
• Participants wanted the ability to search the site
The participants’ recommendations led to many changes on all the
Intranet web pages. For example, figure 1 is the initial CTS home
page and figure 2 is the same page after applying the participants’
recommendations (numbers refer to figure 2):
1. A search engine was developed
2. A pull down menu was inserted at the bottom of every page
3. The table was clearly labeled with ‘Main Categories’ and ‘Sub
Categories’
4. The menu text was enlarged and the font changed
5. Additional categories were added
This allowed for maximum information (links) on the viewable
screen without scrolling, the text was easy to scan for information,
and terms were in the language of everyday users.
178
3. CONCLUSION
This usability test took about two weeks to research, design and
perform. The only costs associated with it were Rubin’s book,
Handbook of Usability Testing, the cassette tapes, and the
administrator’s time. It is estimated, now that we have done the
research and preparation, the time to set up and perform a
usability test in the future would be approximately a week.
In the future, improvements can be made with the regard to the
collection of data. The data was more qualitative than
quantitative. Even so, we are very satisfied with our results for a
first run.
For an effective test, it is important to develop a concrete plan.
The plan must:
• State the test’s objectives
• Identify the participants
• Reflect the actual work environment
• Contain a set of tests to gather observations
• Have performance measurements
• Create a set of recommendations
Input from the users was never sought during the initial creation
of our Intranet site. The creators had the attitude that they knew
what was best for the users. Now, after implementing usability
testing, they have changed their approach to a user-centered
design. Our experiences proved that a usability test can be quick,
cheap and extremely effective.
4. ACKNOWLEDGMENTS
We began much of our research after a presentation by Nancy
Huling, Judith Ramey, and Helene Williams from the University
of Washington at the NWACC/EDUCAUSE/CNI/CUPA meeting
on May 5, 1999 in Portland, Oregon. We wish to thank them for
the resources they provided for their presentation.
5. REFERENCES
[1] Rubin, Jeffrey. The Handbook of Usability Testing. New
York: John Wiley and Sons, Inc., 1994.
6. ADDITIONAL RESOURCES
Dumas, Joseph and Janice Redish. A Practical Guide to Usability
Testing. Norwood, N.J.: Ablex Pub. Corp., 1993.
Web Sites
usableweb.com/new.html
www.useit.com
www.ahref.com/guides/design/199806/0615jef.html
webreview.com/wr/pub/97/04/25/usability
www.ucc.ie/hfrg
www.customerinsites.com
www.zdnet.com/devhead
128.95.202.203/tc517/readings.htm
www.cast.org/bobby
 Figure 1 Initial CTS Home Page

Figure 2 CTS Home Page Using Participants' Recommendations

This paper is also scheduled for the poster session, page 215

179