Running head: USABILITY TESTING OF A WEB AUTHORING TOOL

Usability Testing of a Web Authoring Tool James Call, Kevin Fowler, Duren Thompson, Sharon Trahan University of Tennessee

IT 595

USABILITY TESTING OF A WEB AUTHORING TOOL At the core of many tasks in the field of instructional technology is the ability to share and disseminate resources to a wide variety of users quickly and easily. Currently, common

ways to do this are through blogs and web sites. Thus it is important for practitioners in this field to be familiar with not only a wide range of blogging and web authoring tools but also to evaluate these tools for ease of use. While team members had recently experimented with several different blog and website design tools, we chose to evaluate WordPress.com (hereafter referred to as WordPress) for usability. Team members felt this tool was robust and flexible, but perhaps due to this, challenging for novice users. In this paper, we describe the tool tested, explain the methods we used for testing, and discuss the results of that testing.

Part I - Description of the Web Authoring Tool Our team has heard WordPress.com touted as a versatile online web-authoring tool that is available to the general public for free or at low costs (About Us, n.d.). It began in 2003 as an open source blogging system, WordPress.org, but has evolved into both a hosted blogging service and, for many, an economical system for managing many types of online content as well (About WordPress, n.d.). Our team feels that this authoring tools numerous functions, features, and plug-ins can extend its benefits to computer users of varying abilities from novice to moderate to expert. Based on a survey of the sites about and help pages, we determined that WordPress is intended for current computer users, both male and female, over the age of eighteen years old. It is our opinion that, in order to benefit from the uses of Word Press, users must have at least an average reading level and the ability to understand moderate to difficult instructions and concepts. Possessing basic computer skills is, we feel, another prerequisite for using

USABILITY TESTING OF A WEB AUTHORING TOOL WordPress, but more extensive experience with technology is likely necessary in order to reap the full benefits of this web-authoring tool.

In surveying example sites, we found the range of functions available in WordPress to be enormous. Aside from blogging, WordPress can be utilized to create websites for limitless purposes. Example uses include content management, resource repository, educational, photo management, and business websites. With the use of easily installed, low cost plug-ins that allow the user to use WordPress for almost any intended purpose, we feel that this tools functionality is nearly immeasurable. Web authoring tasks that users will engage in while using this tool will vary based on their computer knowledge and abilities. A novice user, for example, can take advantage of the numerous templates available to create a standard blog or website, enter text, and perhaps upload photos referring to the help option available, as needed, at the click of a button. A more experienced user, however, could customize an existing template using widgits or editing html code in order to create blogs or websites with a more unique look or to meet a more specific purpose. For more advanced users, there are also thousands of plug-ins, as well as the option of low cost customized CSS editing available that extend WordPresss capabilities and tap into the tools full potential.

Part II - Usability Testing via Low Cost Methods Tools Used: As WordPress is an existing web authoring tool, our usability study fits Rubin & Chisnells description of a follow up study: A follow-up study occurs after formal release of the product. The idea is to collect data for the next release, using surveys, interviews, and

USABILITY TESTING OF A WEB AUTHORING TOOL

observations. (Rubin & Chisnell, 2008, p. 20). For our usability test, we chose to focus on using observation and survey methods, although a short interview was added to the process due to midtesting observations by team members. Based on our knowledge of technologies publically available at low or no cost, we felt these methods would provide us with the maximum amount of information with a minimal effort on the part of the usability test participant (user). For the observation portion of the test, we chose to use a remote moderated testing method, asking users to log into a screen sharing tool and attempt to complete certain tasks (Sauro, 2012). The screen sharing tool we chose to use was Adobe Connect. This technology allowed all team members to listen to users, view their mouse actions in real time via desktop sharing, and provided a recording of all activities for later playback and review. Adobe Connect is a relatively low-cost online web conferencing product that features screen-sharing capability. Annual costs for Adobe Connect range from $300 to $150.00 (discounts available to educational users), or there is a pay per use option at a rate of $0.32 per user, per minute. Although a similar tool, Blackboard Collaborate, was also available for use, concerns about its choppy and unreliable performance led the team to a test run of Adobe Connect. Based on this test run, the team decided that Adobe Connects high quality screen-sharing capability and ease of login/entry would provide a much more positive and trouble-free usability testing experience for both the users and the observers. For the survey portion of the test, we chose to use an online survey tool that is also low or no cost to the public - Surveymonkey.com. While there are limits to the number of respondents and survey features available via the tool for free, our planned survey fell well within these limits. It was important to team members that the survey would require little time or effort from

USABILITY TESTING OF A WEB AUTHORING TOOL the users and that their responses would remain anonymous. This would encourage users to be honest in their responses and to complete the entire survey.

User Testing Process A test site, along with a generic user account, was created in Wordpress for the purposes of the test and to aid in team communications. A small group of users were selected by an outside agency (a semi-random grouping), although all were known to the team. A specific user testing appointment was set for each individual user. Approximately 24 hours before the usability testing appointments, users were sent an e-mail containing an overview of the testing process and an attachment with instructions for completing specific tasks during the usability test. In this e-mail, users were encouraged to print these instructions and to talk aloud during the testing process. Following this, a separate e-mail which detailed the process for accessing Adobe Connect was sent to the users. In order to help smooth and speed the connection process for the actual usability testing, the email also included links to software to download and install prior to the appointment time (see Appendices A & B). At the appointed time, the user logged in to Adobe Connect and was greeted by a moderator who ensured that the remote desktop sharing and audio features worked smoothly (a web camera was used by the moderator only during this process). A testing facilitator then greeted the user and guided them through the usability test. Again, users were encouraged to complete tasks independently and to talk aloud as they worked, but they were also invited to ask the facilitator for assistance at any time. Efforts were made to reassure participants that they could not fail or make mistakes and to assure them that their assistance in this process was greatly appreciated. All team members observed each participants mouse and keyboard actions

USABILITY TESTING OF A WEB AUTHORING TOOL via the desktop share and listened to any conversation. Each team member made separate notes and compared those notes aloud immediately after each user test event. Each user test was recorded for later review by team members (sample recording available here: http://www.youtube.com/watch?v=rtk4JJEixtI ). During the test, users were asked to log on, create a new page, title the page, add text (according to a given example), insert an image, and then publish and view their work. Once

users completed the requested tasks in Wordpress, they were directed to a page containing a link to the post-survey in SurveyMonkey. The survey requested a report on their experiences in completing the test and their thoughts on WordPress in general (see Appendix C). After the second user test, team members determined that knowledge of each users technology background and experience with Wordpress would be helpful for testing purposes. For this reason, two interview questions regarding these areas were added to the testing process. This information was gathered after the fact by individual team members.

Part III - Usability Testing Results Team Observations & User Responses Based on responses to interview questions, we determined that our usability testing was conducted on three persons with varying degrees of technology competence ranging from novice to moderate with all stating that they were comfortable with computer technology. All users reported minimal prior experience with WordPress. All users were female and currently enrolled in a University-level educational technology course. Each individual test took approximately twenty minutes. A simple summary of user responses to both survey questions and interview questions is provided in Appendix D.

USABILITY TESTING OF A WEB AUTHORING TOOL The first task we asked our test subjects to do after logging in was to create a new page

within the site. Based on survey responses, two out of the three subjects found this process to be easy while the third found it to be moderate in difficulty. During the test, observers noticed that one of the subjects had trouble locating the New Page button on Word Press. Another subject had issues seeing the New Page icon and other icons that WordPress uses. This user reported that the relative lightness in color of the icons caused her to click on Posts first before clicking on Pages. The next task asked each user to add text to the new page created. This seemed to be an easy task for all subjects, and when they were polled, all three described the task as easy. We noted that one member added a title to her page in a different format from the example provided in the instructions, which observers felt contributed to her later difficulty in locating her completed page. The third task was to insert a picture onto the page of text. Each user had a different opinion of the difficulty of the task with one easy, moderate, and difficult response to the posttest survey. While observing each during the testing, observers felt that this task showed signs of difficulty for two of those tested. One subject had trouble locating the insert picture icon, and after publishing her page was not able to see the picture she inserted. (Observers were not able to determine why this occurred it appeared all necessary steps were completed to successfully insert an image). Another showed some hesitation in beginning this process and almost chose the wrong icon to insert an image, but she was able to easily complete the insertion process thereafter. Finally, we asked users to publish (save) their pages and view them. Observers noted some difficulties during this portion of the testing; however, two-thirds rated the task as easy

USABILITY TESTING OF A WEB AUTHORING TOOL and the other chose moderate. Specifically, we observed hesitations and false starts in determining what to click on in order to view the pages that they created. As stated earlier, one subject chose a strange title format which made finding her page more difficult.

Conclusions According to Rubin & Chisnell (2008), To be useable, a product or service should be useful, efficient, effective, satisfying, learnable and accessible. Below is our analysis of the usability of WordPress.com, based on this definition. We feel that WordPress is useful as a tool for web authoring based on the results of our testing. Usefulness concerns the degree to which the product enables a user to achieve his or her goals, and is an assessment of the users willingness to use the product at all. (Rubin & Chisnell, 2008, p. 4) All three subjects were able to complete the tasks fully, and each either said they would use the product again or suggested changes to the product, implying that they would use it again. We also feel that WordPress is an efficient tool based on the time it took for the goals of each subject to be accomplished accurately and completely. Although there was a noticeable amount of hesitation throughout the test, all three subjects completed the tasks in roughly the same amount of time around twenty minutes. Based on our results, we feel that WordPress may not be a very effective tool for new or novice users. Effectiveness refers to the extent that a product behaves in the way that the user expects it to (Rubin & Chisnell, 2008, p. 4). The only task that 100% of users found to be easy was adding text to a page. The other tasks ranged from easy to difficult with the most difficult being inserting a picture or image, although 66% of users were able to create the

USABILITY TESTING OF A WEB AUTHORING TOOL page, publish it, and view it with moderate ease. The learnability (part of effectiveness) of the tasks seemed, however, to be equally high for all subjects, and one stated that she felt she would be comfortable with WordPress after a few more uses. In comparing the users minimal prior experience, the lack of instruction provided, and the rare assistance requested during the test to the overall success of their performance, we feel that the learnability of Wordpress for these basic tasks is reasonable, although not completely transparent. The satisfaction of the tool seems to have been mixed for our subjects. Aside from finding some tasks to be difficult, each person had a suggestion to improve Word Press. Two users had a suggestion to make publishing and viewing a post easier, and the other suggested modifying the icons used on Word Press. Both areas are related to the accessibility of the tool. Accessibility involves making products usable by people who have disabilities (Rubin & Chisnell, 2008, p. 5). If the icons used on WordPress are difficult to see, then this is an issue for those with visual disabilities. A suggestion to address this problem might be to enlarge the icons used and/or change the colors so that they stand out more from their surroundings. In conclusion, we feel that results of our testing indicate that WordPress has difficulty

passing the usability test for low or novice level users. When a product is truly usable, the user can do what he or she wants to do the way he or she expects to be able to do it, without hindrance, hesitation, or questions. (Rubin & Chisnell, 2008, p. 4) We found that our users, self-described as comfortable with computers, could accomplish all tasks using Word Press, but with hindrances (not being able to see an uploaded image), hesitation (having difficulty finding the right link or button), and some feelings of dissatisfaction. We feel that with some small adjustments to icon color, size, and placement, it would not be difficult to define WordPress as a highly usable web-authoring tool for first-time users.

USABILITY TESTING OF A WEB AUTHORING TOOL Team Reflections on Usability Testing

10

We learned a great deal from the results of the testing, but we feel we also learned much from our own procedures. We discovered a number of ideas for improving our process to provide for a smoother user and observer experience and to ensure more definite results from future testing. First, we noted that if our test subjects did not enter the test with the instruction sheet we provided (printed or open on the desktop), it made testing much more difficult. We would have to delay the test for the subject to retrieve them or read them aloud to the user during the test. In truth, for lower-level computer users, printed instructions would be best. Second, we discovered that our instructions could have been more clear. For example, we did not specifically indicate how to title the newly created page our subjects created, so one chose an odd title, causing confusion in viewing the page. While directions indicated that users should first type text and then insert an image, the given example showed the image first and the text second. This created some small logistical confusion for two users. In another example, we felt that the link to the survey was not sufficiently clear, and one subject clicked in the wrong spot which took her to an advertisement page instead of the survey. Third, we understood after the testing was complete that many of the hesitations discussed earlier might have been based on the subjects anxiety because they were being watched by three extra observers. Hesitations may also have been due to a fear of failure during testing. We reassured each subject before the test that there was no way to do poorly during the testing, but the fear of failure could have been increased due to the intimidation of multiple people watching them work. In the future, we could choose to have only one tester with each user for future usability tests, with other members of the testing team viewing the recording at a later time. In addition, we noted that, although encouraged several times, most users did not talk

USABILITY TESTING OF A WEB AUTHORING TOOL aloud or express their thinking during the test. Again, this may have been due to general discomfort with being observed, unfamiliarity with the talk aloud process, or the optional nature of this additional task.

11

Lastly, we discussed how being organized before a usability test makes the test run very efficiently for the testers and the test subjects. Creation and testing of the test blog and the well-discussed and reviewed specific user instructions contributed greatly to the success we felt we did have in this activity. Our ability to access and test Adobe Connect twice before the usability test event was also integral to our process running smoothly. Shifting user feedback to the online post-survey also made it easier for observers to take notes as each subject completed the tasks assigned.

USABILITY TESTING OF A WEB AUTHORING TOOL References (n.d.) About WordPress. Retrieved from http://wordpress.org/about/ (n.d) About Us. Retrieved from http://en.wordpress.com/about/ Rubin, J., & Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (2nd ed.). [rubin_chisnell_2008_chp1-2.pdf]. Sauro, J. (2012) Comparison of Usability Testing Methods, posted on Measuring Usability: Quantitative Usability, Statistics and & Six Sigma. Retrieved from http://www.measuringusability.com/blog/method-comparison.php

12

USABILITY TESTING OF A WEB AUTHORING TOOL Appendix A Instructions/Information E-mailed to Usability Test Participants E-mail 1: Subject: WordPress Usability Test Information

13

Thank you again for taking the time to participate as our Users for this class assignment. Below is information about the test. Please do not hesitate to contact us if you have any questions. Your usability appointment is at: [Fill in time for each person] Please note that we consider you our guests, and you can do nothing wrong in this process. If there are any technical issues, the fault lies with us, not you, and we thank you for your patience ahead of time. This usability test has the following general flow: - At your test time, Join an Adobe Connect meeting by clicking on an e-mailed link (mailed separately from Kevin) - Follow instructions by the Moderator (Kevin) to check your sound and share your desktop. - Login to Wordpress and complete several simple tasks (see attached instructions). - Complete a short online survey of your thoughts about using Wordpress for these tasks. For this test you will need sound and microphone typical to participating in Blackboard for class, but you will not need a webcam. In preparation for the test, we recommend that you: - review the attached instructions, and have them available during the usability test (we recommend printing them). - determine an image file you would like to use in a profile of yourself. Again, we thank you, and if you have any questions, please do not hesitate to ask! ITC595 Team 2: James, Kevin, Sharon, Duren

USABILITY TESTING OF A WEB AUTHORING TOOL Appendix A Attachment for E-mail 1:

14

USABILITY TESTING OF A WEB AUTHORING TOOL

15

USABILITY TESTING OF A WEB AUTHORING TOOL Appendix A E-mail 2: Subject: InstructionalTechnology59549834FA2012: Usability Test Link and Downloads Classmates/Usability Group:

16

By now you have received an e-mail from Duren Thompson containing directions on completing our usability test. This e-mail contains: * Adobe Connect Plugin: Something that you can download for your computer right now to make things easier for Thursday night. * Usability Test Link: The link to enter the usability test later at the assigned time * My Phone Number/Skype Username: In case you have any problems. [Adobe Connect Plugin] You can download the Adobe Connect Plugin below. Yes, you can do this the night of at your appointed time, but hopefully this will speed things up a bit. If you have any issues downloading and installing, or you aren't sure how to extract a zip file, let me know...I can send it in a different file format. Adobe Connect 9 Meeting Add-in for Win (Zip format, 3.5MB): http://www.adobe.com/go/adobeconnect_9_addin_win Adobe Connect 9 Meeting Add-in for Mac (Zip format, 7.5 MB) http://www.adobe.com/go/adobeconnect_9_addin_mac Note: 1) You will also need Adobe Flash, but your computer probably already has it. 2) As much as I love Google Chrome, it doesn't seem to like Adobe Connect. I would suggest clicking on the above download links in either Internet Explorer, Mozilla Firefox, or Safari if you are on a Mac. [Usability Test Link] Here is the link to the Usability Test: http://nsci.adobeconnect.com/ut Click on it at your scheduled time. Just type in your name and sign in as a guest. I should be on the other side giving you directions on how to turn on your microphone in your new environment. It is crucial that you be completely signed out of Blackboard Collaborate. [My Phone Number/Skype Username] Technical difficulties are bound to happen. If you are unable to login to Connect, your microphone doesn't seem to work, the plugin doesn't install correctly, feel free to give me a call at (XXX) XXX-XXXX and I will do my best to work through the situation. My Skype username is KevinLFowler if you want to get in touch with me that way. We appreciate you helping us out with this, and we appreciate you perhaps going outside your comfort zone in order to make this a success. Thank you!

USABILITY TESTING OF A WEB AUTHORING TOOL Appendix B WordPress Blog Created for Usability Test Participants Available on line via: http://it595usertestgrp4.wordpress.com/

17

USABILITY TESTING OF A WEB AUTHORING TOOL Appendix C WordPress Usability Test Post-Survey Administered via SurveyMonkey

18

USABILITY TESTING OF A WEB AUTHORING TOOL Appendix D Personal Interview Results What do you consider your technology comfort level? (very comfortable, somewhat comfortable, not comfortable) #1 somewhat comfortable #2 - somewhat comfortable #3 very comfortable What do you consider you technology experience level? (Expert, moderate, novice) #1 novice to moderate #2 moderate #3 no answer (verbal was moderate to expert)

19

Survey Question Results 1. Please rate your experience with completing the following tasks in Word Press. Rating Scale Results: Rating Average 1.33 Response Count 3

Easy Creating a new page 66.7% (2) 100.0% (3) 33.3% (1)

Moderate 33.3% (1) 0.0% (0) 33.3% (1)

Difficult 0.0% (0) 0.0% (0) 33.3% (1)

Adding text to a page

1.00

Inserting an image or picture

2.00

Publishing and viewing your post

66.7% (2)

33.3% (1)

0.0% (0)

1.33

USABILITY TESTING OF A WEB AUTHORING TOOL 1. Please rate your experience with completing the following tasks in Word Press: Participant Comments:

20

Creating a new page responses: I was able to locate the create new page option quickly. #1 The instructions really helped make this easier. #2 I find the menu options on WordPressto be confusing and the typeface is so light, it's not noticeable enough. #3 Adding text to a page responses: I was able to type in my text and it came up when I clicked view page. #1 Inserting an image or picture responses: I had difficulty, because I thought the image had downloaded, but when I clicked on view page the image was not there. #1 Again, the icons and labels are not readily viewable. #3 Publishing and viewing your post responses: The publish button was easy to find, but I had trouble finding the view post button again after I had left the page after viewing my post the first time. I sadly had to ask for assistance. #1 2. What recommendations, if any, do you have that would improve your experience with Word Press? I need experience with it. The directions I followed were all something I had done before. It took me a moment to remember where everything was. Your group described everything perfectly. It was a painless process. #1 Make it clear when I have published something. #2 Better labeling for menu options, maybe dynamic buttons so when you scroll over one, you can see your options for each heading. #3