Running head: EVALUATING GOLDSTAR 1

Evaluating Goldstar Phone Etiquette and Procedure Training

Miguel Medina, Eva Morris, Brianne Fitzgerald

California State University Monterey Bay

IST 622 Assessment & Evaluation

Professor Dr. Bude Su

July 25, 2017

EVALUATING GOLDSTAR 2

Table of Contents

I. Introduction ................................................................................................................. 4

II. Methodology ............................................................................................................... 5

Prototype ......................................................................................................................... 5

Tryout Conditions ........................................................................................................... 6

Learners........................................................................................................................... 6

Expected Outcomes ........................................................................................................ 8

Affective Domain........................................................................................................ 8

Cognitive Domain. ...................................................................................................... 8

Process ............................................................................................................................ 9

III. Results ................................................................................................................... 11

Entry Conditions ........................................................................................................... 11

Instruction ..................................................................................................................... 12

Summary of Data .......................................................................................................... 12

Additional Observations ............................................................................................... 18

Recommendations ......................................................................................................... 18

IV. Summary ................................................................................................................... 19

V. Appendices................................................................................................................ 20

Appendix A ................................................................................................................... 20

Appendix B ................................................................................................................... 20
EVALUATING GOLDSTAR 3

Appendix C ................................................................................................................... 21

Appendix D ................................................................................................................... 22

Appendix E ................................................................................................................... 23

Appendix F.................................................................................................................... 26

Appendix G ................................................................................................................... 28

VI. References ............................................................................................................. 29

EVALUATING GOLDSTAR 4

I. Introduction

Goldstar Buick GMC is a full-service auto dealership based in Salinas, California. A

Business Development Center (BDC) has been constructed at the dealership for continuous in-

house employee training. The average workweek of a car salesman consists of over 60 hours.

Sales customers frequently arrive unannounced and sales representatives must stop their

activities and attend to them. Therefore, using a computer as the platform for delivering training

is flexible and allows training to be completed in small increments. Furthermore, if the

employees require, they can complete portions of their training at home or during down times on

their mobile devices.

The Goldstar Phone Etiquette and Procedure Training eLearning module was created in

response to Goldstars desire to create a consistent sales process and enhanced communication

protocols for customer relations. The Goldstar module was designed to be an online,

asynchronous lesson to accommodate sales employees intense and irregular schedule.

Employee turnover is high, and training program costs are considerable. This eLearning

module evaluation is designed to assess learning, regardless of prior knowledge, and to discover

any necessary improvements needed for the lesson content. The usability test audience consists

of ten individuals, who have limited, or no experience in the sales field. Two evaluation types

planned for this training are:

1. Lesson content evaluation, which consists of an analysis of pre- and post-test scores to

assess the effectiveness of instruction.

2. Usability testing of the module, which will uncover any technical issues and design flaws

of the module. Feedback from the learners interacting with the lesson can be used to

further improve the content of the training and the test questions.
EVALUATING GOLDSTAR 5

II. Methodology

Prototype

The sales module is currently at the prototype stage, with the intention of integrating it

into a larger curriculum. The module itself is approximately ten minutes in duration. It runs best

on a computer and can be accessed either at the Goldstar facility during downtime at work, or

remotely at home. Ultimately, the goal is to make the module accessible on handheld devices

such as smartphones and iPads.

The eLearning module is introduced and guided by a pedagogical agent, Bryan. Lesson

content consists of a video tutorial, phone scripts, and embedded practice activities that are used

to engage learners and solidify the newly-learned content. Supportive feedback and guidance is

provided along the way. A real-world sales scenario is provided to help learners conceptualize an

authentic phone call between themselves and a customer. The program's ease of use should be

universally appealing and require minimal technical skills or pre-training. As a result of taking

the course, the learners are to gain a full understanding of the content and be able to perform

professionally (Figure 1). The practice activities are designed to be engaging and fun, yet also

meet the lesson objectives (Figure 2).

Figure 1. Slide 3 (Learning Objectives) from eLearning Module

EVALUATING GOLDSTAR 6

Figure 2. Slide 6 (Practice Activity) from eLearning Module

Tryout Conditions

The assessment instruments and the training module were accessed during the same

observation session. The facilitators provided their laptop computers for the assignment. The

session facilitators followed a Facilitator Instructions script (Appendix A) to ensure consistent

instructions for all test takers. Learners were given Learner Instructions in electronic form on the

computer where they completed the assignment. The document contains links to the surveys and

tests in Google Forms, and a link to the training module, which is hosted on CSUMBs server,

www.itcland.csumb.edu (Appendix B).

Learners

The participants chosen for this evaluation in large represent the target audience, which is

predominantly male and ranges in age from 19 to 60+ years. The target training audience also

comes from various cultural backgrounds, levels of education, and has varying degrees of

computer expertise. As shown in Figures 3 through 5, the learners chosen for this evaluation and

assessment assignment reflect the target audience demographic closely. The learner age chart

(Figure 3) reflects the learner age, which ranges from 19 to 73 years of age. Learner ethnicity
EVALUATING GOLDSTAR 7

was predominantly white, with a third of the learners represented Latino and mixed ethnicity

(Figure 4).

Figure 3. Learner Age Figure 4. Learner Ethnicity

This distribution of ethnicities matches the current salesforce of Goldstar closely.

Customer support experience of the learners ranged from none to over five years of experience

(Figure 5). Overall the learner group represent a diverse sample of age, experience, and ethnical

backgrounds.

Figure 5. Customer Support Experience of the Learner

EVALUATING GOLDSTAR 8

Expected Outcomes

Upon completion of the training module, it is expected that learners have a greater level

of confidence toward conducting sales phone calls, and have an increase in knowledge pertaining

to the proper phone etiquette when conducting a sales call. Learners will use the sales phone

script as support aids when they are exposed to situations that involve customer inquiry.

Eventually, the sales phone scripts should become learned common knowledge for the

salesperson as he/she applies the knowledge into practice. Ultimately, the learner will feel

comfortable engaging with customers during a sales phone call and meeting their expectations

with professionalism. This training focuses on two learning domains: cognitive and affective.

Affective Domain. Two goals of learning fall under the affective domain. Learners must

choose to engage with the lesson fully for the best learning outcomes. They must then choose to

apply their new knowledge on the job with sales phone calls (when delivered to the target

audience). The learning module strives to motivate learners toward knowledge transfer on actual

sales calls by providing phone scripts, which can be downloaded and used as job aids. Providing

practice with these documents within the lesson will build the learners phone sales confidence.

The module is scenario-based, and the pedagogical agent Bryan represents what is considered to

be the most common gender and age category of the target audience. By personalizing the

training, learners will be motivated to engage with the content. Additionally, the look and feel of

the module has been customized for Goldstar with dealership images and logos. The branded

material will generate familiarity with the instruction, therefore further increasing confidence and

engagement.

Cognitive Domain. Learners must acquire the knowledge for making critical decisions

and judgements during a sales call. The training covers two content areas: phone etiquette and
EVALUATING GOLDSTAR 9

the usage of phone scripts. The module begins with an overview of the training and its

objectives, followed by a video presentation of phone etiquette standards. The phone etiquette

standards are further explained in a different format (phone scripts) that the learner can review

and download to keep. Practice activities follow each of the content sections to solidify the

content and to keep the learner engaged. The first practice activity is meant to be engaging and

fun, while the second practice exercise is scenario-based, using a situation commonly faced by

sales personnel.

The module design is intended to reduce cognitive load by segmenting the content,

providing visual clues, and presenting it in different modalities (e.g. video, documents,

interactions). The objective of professional, positive communication is demonstrated throughout

the lesson and serves as reinforcement of the concepts presented.

Process

Assessment instruments were provided in form of Google Forms. The post-course

knowledge assessment was originally built into the prototype. However, in order to make the

assessment for the pre-and post-test consistent (e.g., the matching test item could not be

formatted the same in Google Forms as it was in Adobe Captivate), the quiz was removed from

the online module and constructed independently.

The three authors of this paper served as facilitators and observers of the evaluation

project. The learner instructions (Appendix B) allowed the learners move through each step

independently and in the correct order: 1) pre-course survey, 2) knowledge pre-assessment, 3)

eLearning module, 4) knowledge post-assessment, and 5) post-course survey. All ten users were

observed individually and in-person by the facilitators. Aside from feedback was captured in

Google forms as part of the survey, observational feedback was also collected in the table
EVALUATING GOLDSTAR 10

Learner Observations (Appendix F). It took each user on average 20-30 minutes to complete two

surveys, two assessments, and the training module. The evaluation and assessment duration has a

wider range for two reasons: one learner experienced a technical glitch mid-training and had to

start over again, and two users pressed a button that took them back to an earlier stage of the

training, without the possibility of skipping the already-viewed content.

Pre-course Survey. The purpose of the pre-course survey is to provide demographic

information of the learner including: gender, ethnicity, and age, to ensure that the audience

selected for the evaluation and assessment assignment reflects the target audience. The learners

are also questioned on how much sales experience they have, and if they have any customer

support experience (Appendix C).

Knowledge Pre-assessment. The knowledge pre-assessment determines how much

knowledge the learner has about the topic of phone etiquette and car sales procedure prior to

taking the training. The questions were extracted from the original eLearning module prototype

and converted into Google Form format. Question #5 changed in layout as Google Form cannot

display matching items the same was as Captivate. Unfortunately, the change in item display

resulted in multiple negative reviews by test takers (Appendix F, Learner Observations).

The collected data serves as a baseline for the knowledge post-assessment. One answer

choice was changed in question #2, when the first test taker reported clueing of question #3 after

completing the assessment. As a result, the first test takers collected data of the pre-and post-

assessment was invalidated (Appendix D).

Training Module. The training module is an interactive approach to help the learner

engage and retain long term cognitive information concerning appropriate attitudes and etiquette

while making successful sales calls. It is asynchronous to adapt with the sporadic, demanding
EVALUATING GOLDSTAR 11

schedule of sales employees. The eLearning module was made available online via CSUMBs

ITCDland server to allow test takers to access the course from any computer without requiring a

login. Learners are asked to enter their name to begin the training module to be entered on a

certificate of completion. The certificate feature was disabled for this evaluation project.

Knowledge Post-assessment. The knowledge post-assessment is a measure of learning,

which may have taken place due to lesson participation. The questions in the pre- and post-

assessment are the same (Appendix D). The data gathered in the post-assessment is analyzed

against the pre-assessment to determine whether results improved after completing the course.

The first usability tester pointed out a clueing item in the knowledge assessment (same in pre-

and post). We fixed the test item after this instance, and therefore did not include the learners

data in the knowledge assessment. However, we retained the users data for the pre- and post-

course survey.

Post-Course Survey. The post-course survey focused on usability of the training module

(Part A), as well as learner experience (Part B) and lesson feedback (Part C). Response options to

questions are provided in form of Yes/No answers, constructed responses, and the Likert scale

with five response options (Strongly Agree Strongly Disagree). Learners are queried on layout,

ease of navigation, training agent, difficulty of assessment, etc. (Appendix E).

III. Results

Entry Conditions

As expected, the learners were able to follow the instructions easily, and successfully

accessed all of the forms and the training module using the links provided in the Learner

Instructions document. The session was designed to be no more than 30 minutes which

alleviated concerns from those learners who participated during the work day or shortly after.
EVALUATING GOLDSTAR 12

Instruction

The facilitator provided initial instructions before the start of the assignment. The learners

participated with the instruction as expected. While taking the course, most learners found the

pedagogical agent helpful when providing navigation instructions. Learners followed the

instructions, engaged with the activities, and read the documents as intended. Both practice

activities revealed areas of improvement for the navigation (Appendix E). In Practice Activity 1,

although instructed properly by the agent, learners gravitated toward clicking the Next button

rather than the Submit button, which would have provided the learner with feedback on the

activity. The other area that received the most common negative feedback was Practice Activity 2

Learners were unsure of how which button to select to get where they wanted to. One user

experienced a technical glitch mid-course with the disappearance of a Next button and had to

restart the course. This was recorded as an anomaly, as this event could not be recreated at a

later time.

Summary of Data

The expected outcomes of engaging with the learning module were that upon completion,

the learners will have gained the knowledge to conduct a more professional sales call with

customers. Therefore, a one-tailed t-test was conducted using a directional research hypothesis

(H1 : P1>P2 ) that has been selected with an accompanying null hypothesis (H0 : P1<=P2) and an

= 0.05.

The charted results suggested that most of the learners scored significantly higher on

answering test questions in the assessment after taking the course, than they did on the pre-

knowledge assessment (Figure 6).

EVALUATING GOLDSTAR 13

Figure 6. Comparison of Pre- and Post-Knowledge Tests. Learners 1 through 8 experienced an

increase in knowledge (learner 4 scored zero points in the pre-knowledge test). Only learner 9

had a decrease in knowledge after taking the course.

Only nine out of the ten learners are represented in this pre- and post-knowledge tests

results, as the first learner was exposed to a clueing question. Her results were omitted for. The

observer reported the discrepancy to the team, and one of the questions was modified, which

resulted in excluding the assessment results of the first learner. That learners pre- and post-

survey answers remained in the overall evaluation.

A paired t-test was conducted between the before and after results of the knowledge tests.

Because the p-value (0.002721, DF=8) < (0.05), we rejected the null hypothesis. Furthermore,

the t stat (3.772) > t critical one-tail (1.859), which also suggested that we reject the null

hypothesis. There was a significant difference in knowledge after taking the course (Table 1).
EVALUATING GOLDSTAR 14

Table 1. t-test Pre- and Post-Assessment

More males, than females, were available to take the course. Two-thirds of the learners

were male, while only one-third of the learners where female (Figure 7). The correct average

answers per male were significantly higher compared to the correct average answers of their

female counterparts.

Figure 7. Male vs. Female Comparison

Analysis was also conducted to determine if there was a statistically significant difference

between the means of the two independent samples (males and females) using the following

hypothesis (= 0.05):
EVALUATING GOLDSTAR 15

H0 : 1 = 2 There is no significant difference between male and female learning

H1: 1 2 There is a significant difference between male and female learning

The two-sample assuming unequal variances t-test was conducted using the male and

female knowledge test score variables (Table 2). Because the p-value (0.283, DF=3) > (0.05),

we failed to reject the null hypothesis. Furthermore, the t Stat (1.305) < t Critical two-tail

(3.182), which also suggested that we should fail to reject the null hypothesis. There was no

significant difference in learning between males and female learners.

Table 2. t-test Male vs. Female

Additionally, Correlation Coefficient r Measures were conducted on several self-

reporting answers in the pre- and post-surveys. The outcomes of some of the more significant r

measure values along with possible explanations are provided in the proceeding table.

Table 3. Correlation Coefficient r Measures

Enjoyed Course & Possible explanation is that the Agent Bryan made the course
Agent Bryan easier to follow and more enjoyable.
Enjoyed Course & Because the course was enjoyable the learners were motivated
Increased Confidence in their ability to make a sales phone call.
Age & Increased Age suggest more general experience and maturity,
Increased Confidence therefore a baseline of confidence may already exist.
Enjoyed Course & Because they enjoyed the course they were motivated to learn
Increased Knowledge more.
EVALUATING GOLDSTAR 16

Pace appropriate & Lesson
contributed to Knowledge
Navigation easy Lesson
contributed to knowledge
Customer Support Experience
& Lesson Contributed to
Knowledge
Because the lesson pace was appropriate, it permitted the to
absorb information.
Because the navigation was easy to follow, they focused their
cognitive attention on the information.
The negative correlation might have been a result of older
learners already knowing the materials presented.

Note. This table highlights strong correlations and possible explanations. For the full table with

results, see Appendix G.

Four specific questions were asked in the post-survey concerning the usability of the

course (Figure 8). The questions addressed the ease of comprehension in completing the course:

whether the instructions were easy to follow, the ease of use and navigation throughout the

course, whether the pace of the course was conducive to learning, and whether the learners

experienced any technical issues with the course. The answers were quantified by adding a value

of 1 for a Strongly Disagree answer and a value of 5 for a Strongly Agree answer. As the

chart demonstrates, most questions ranked at least a value of 3 (Neutral) or higher, suggesting

that the course was user-friendly and easy to use.

Figure 8. Self-reported Usability Feedback

EVALUATING GOLDSTAR 17

Self-reporting questions were also used to discover whether the learners felt that they had

gained knowledge from the course (Figure 9). The same ranking values were used for the

answers (1 = Strongly Disagree, 5 = Strongly Agree). The results suggest a high level of approval

for the overall questions: the learners believed the lesson contributed to their understanding of

the sales etiquette and attitude, the course increased their confidence when making a sales call,

and taking the course increased their knowledge.

Figure 9. Self-reported Knowledge Retention

Although usability analysis results appear positive, there were a few issues observed

during the course participation as described in the observation list. Many of the issues can be

resolved with professional production, clarification of questions, and navigation button

positioning. One question that particularly stood out in both the pre- and post-knowledge tests

was the multiple-choice question #4 (Appendix D). This was meant to be a matching question,

but unfortunately, the developers were limited to the construct of the design because they were

bound to the limitations of the Google Forms.

EVALUATING GOLDSTAR 18

Additional Observations

Some of the feedback collected (positive as well as negative), as either part of the

observation notes or in form of comments in the post-course survey, is highlighted below. The

entire list of comments can be viewed in Appendix F, Learner Observations.

Learners enjoyed the design and interactivity of the practice activity #1 (truck bed)

exercise, but some got confused because of the button arrangement on the slide.

Phone script: learners appreciated the simple format, content of the phone scripts, and the

availability to download the PDF files. They also acknowledged how helpful the

documents would be when downloaded as job aids.

Practice Activity #2: Learners go lost in the navigation, as many started reading the

on-screen text while the pedagogical agent still gave instructions how to proceed next.

Recommendations

The issues and difficulties discovered during the evaluations proved critical to updating

the course in design, usability, and user instructions. The results also included recommendations

for improving navigation. The course can also be further enhanced by simplifying the questions,

creating a more professional video, and using a more flexible survey platform. Future tests could

include a larger sample group that is more evenly dispersed across ethnicity, gender, age, and

amount of customer support experience. Other recommendations include:

1. Improvement of pedagogical agent (Bryan): most learners enjoyed the pedagogical

agent, but several responses indicated that there was too much movement and one learner

recommended removing the agent from the video screen. Solution: keep the pedagogical

agent as part of the training, but reduce the number of movements in the module.
EVALUATING GOLDSTAR 19

2. Improved Navigation: Inclusion of a Table of Contents in the sidebar, or the ability to

navigate back to a location if they selected the wrong button (add learner navigation

control) without having to wait for the buttons to appear after a timed period.

3. Assessment: change of answer choice(s) in assessment Question #5 (Appendix D) to

more closely and accurately represent the content in the training module.

IV. Summary

Analysis of the knowledge pre-and post-tests supports the effectiveness of the Goldstar

Phone Etiquette and Procedure Training. The research hypothesis, that learners gain knowledge

by completing the course, was substantiated by the results of the paired t-tests conducted. A

significant difference in knowledge after taking the course was discovered, which suggests that

learning indeed took place. The odd ratio of males to females presented a bias notion that males

answered more correctly than females. A two-sample t-test assuming unequal variances was

conducted which resulted in failure to reject the null hypothesis that there is no significant

difference in learning between males and females. This finding suggests that males and females

were equally successful.

Self-reported user information further supported that the learners acquired new

customer phone sales skills by taking the course. The higher ratings reported for increased

confidence of conducting a professional sales call is a positive indicator that learning occurred.

Furthermore, the self-reported usability information confirmed that the overall course was user-

friendly and enjoyable. The self-reported data suggests that the learners comprehended the

importance of attitude when making a sales call, which is an affective component of the course.
EVALUATING GOLDSTAR 20

V. Appendices

Appendix A
Facilitator Instructions
Open up the learner instructions on the test takers computer. The document contains all the links
for steps 1-5, listed in the correct order.

Say: Please follow the steps in the order provided. While going through each step, describe your
progress out loud.

Give example, see also information below observation checklist.

Let the learners know that during the evaluation and training, you are only there to observe. You
are not to answer any questions, only to help when they are completely stuck (e.g., experience
technical difficulties).

Appendix B
Learner Instructions
Thank you for participating in todays training. Please know that your responses will be
beneficial in evaluating the course. Your feedback will be very valuable, and will help in making
this training better.

Please start the evaluation and assessment process by accessing the content in the order shown
below, starting with Step 1.

Step 1: Pre-course survey

https://goo.gl/forms/FNMNsI1s9YjXWx5E3
Step 2: Knowledge Pre-assessment
https://goo.gl/forms/8SUgW57eFyDI9b2t1
Step 3: eLearning training
http://itcdland.csumb.edu/~evmorris/training/622
Step 4: Knowledge Post-assessment
https://goo.gl/forms/QNwpcOOYFe09GlQR2
Step 5: Post-course survey
https://goo.gl/forms/ASBM4dmmxcosAnSS2
EVALUATING GOLDSTAR 21

Appendix C
Pre-course survey

1. What is your age?

________________________________________________

2. What is your gender identification?

________________________________________________

3. What is your ethnicity?

________________________________________________

4. Are you currently employed in a customer service role?

Yes
No

5. Have you ever been employed in a customer service role?

Yes
No

6. If you answered the previous question with Yes, state how long.
Less than 6 months
6 months - 1 year
1-2 years
2-5 years
5+ years
7. Do you have sales experience?
Yes
No
EVALUATING GOLDSTAR 22

Appendix D
Knowledge Pre- and Post-Assessment

1. What is the BEST thing you can do during an entire sales phone call? (select one)
Get customer information
Smile throughout
Push the sale
Take notes

2. When providing sales information to a customer, you should never (select one)
Talk about the weather
Talk about the service department
Invite the customer to lunch
Lie to the customer

(Note: Last answer choice changed from Provide unsolicited information to Lie to the
customer, after the first test taker commented that this response was clueing next test
question. This cause the assessment data from the first test taker to become invalid.)

3. What are the downsides of offering unsolicited information to a customer?

(select all that apply)
You will be violating state law that requires you to reveal all information.
There are no downsides, extra information is helpful to the customer.
Information may be important to you, but not to the customer.
The customer may change their mind.

4. Connect the sales concept with its definition. Put a checkmark under the best definition
(only one selection per row).
EVALUATING GOLDSTAR 23

Note: The original formatting of the item was different in the eLearning module (shown
below). The matching item could not be setup up in Google forms the same way.

a) Rudeness __ Effective listening skills build rapport

b) Vocal tone __ You never know the limits of using offensive language
c) Professionalism __ Interrupting customers makes for bad customer service
d) Attitude __ The volume of your voice affects the perception of you
e) Attention __ Sets the mood for the conversation

5. A customer calls and inquires about a specific vehicle. You are new and are not sure what
to say. What is the BEST course of action? (select one)
Place the customer on hold and get your manager
Place the customer on hold and check the dealership lot for the car
Pull up the phone script and use it in the call with the customer
Tell the customer you dont know anything about the specific vehicle

Appendix E
Post-Course Survey (survey questions are correct, but formatting of section is TK)

Part A: Usability

1. Did you have any trouble accessing the course?

Yes
No

2. The instructions in the course were easy to follow.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

3. The course was easy to navigate.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
EVALUATING GOLDSTAR 24

4. The pace of the course was appropriate.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

5. The course was free from technical problems.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

6. Please add additional comments concerning the preceding questions.

________________________________________________

Part B: Course Feedback

7. The course was enjoyable.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

8. The quiz had an appropriate level of difficulty.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

9. The agent Bryan enhanced the learning experience.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
EVALUATING GOLDSTAR 25

10. What did you like MOST about the course?

________________________________________________

11. What did you like LEAST about the course?

________________________________________________

12. If you could change anything in the course, what would it be?
________________________________________________

13. Please add any additional comments concerning the preceding questions.
________________________________________________

Part C: Lesson Feedback

14. The lesson contributed to my ability to conduct a proper sales call.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

15. I feel confident about making a proper sales phone call.

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
EVALUATING GOLDSTAR 26

Appendix F

Learner Observations

# of
Instructional
Observations/Feedback Learners
Element
commented
Are you currently employed/Have you previously been
employed in customer service?
1
Learner found both questions confusing/ contradicting (in
combination).
Pre-Survey Unusual to type age, usually there is an age range to select
from.
Usually, there is a radio button to select male or female. 1
I am lazy, I prefer the radio buttons instead of typing in
text.
Pedagogical agent distracting during video segment 1
Pedagogical Pedagogical agent aided in guiding the learner with
Agent 3
commands. Enjoyed him.
Agents motions distracting in introduction. 3
Lighting in video could be improved. It was darker when
1
speaker wore white shirt than when he wore a dark shirt.
Video
For most the video, speaker did not look directly into the
1
camera by slightly underneath it. Annoying.
Overall
Speaker sounded tinny 1
Narration/Audio
Fun.
5
Entertaining.
Practice Activity Advanced to next slide before finished 2
#1: Truck Bed Clicked Next button instead of Submit, therefore did
Activity 3
not receive feedback on activity.
Feedback unclear-didnt know if answer was incorrect or if
3
was doing the dragging incorrectly
Scrolled down to bottom of script for next button 1
Had to figure out on own how to close script window to
5
return to lesson
Next and Back button disappeared while reviewing
Phone Scripts phone script. Appears to be anomaly, as this issue was not 1
repeatable.
Instructions say when you are done reading, click the Next
button, adds confusion as they look for a Next button on 4
the phone script documents.
Liked the format of the phone scripts as job aids. 1
EVALUATING GOLDSTAR 27

Not sure which button they were supposed to select. 5

Selected a button and had unintended consequences then
3
could not return to where they wanted.
Wording was ambiguous. Non-specific and unsolicited
1
information
Bryan is talking while text is available to read, trying to
Practice Activity read text while listening to Bryan.
#2: Irate
customer Read text on screen, missed Bryans instruction to click on 3
Practice item when ready for the question. Instead,
learner clicked on video button and did a whole detour of
the module again (not set up to jump back and forth).
Correct Answer: YEAH!
Bryan narrated his way through the module, learner 1
expected him to say something in addition to YEAH!
- Question with rows and columns confusing
- Took a long time to figure out what was being asked
- The red background when more than one bullet was
selected in row/column was alarming to the learner.
- Contradicting instructions (one selection per row,
one response per column)

Knowledge
Pre-/Post- 9
Assessment

Went back and forth on answer choices, was not clear. 1

Last question: A customer calls

Learner was not sure whether the item reflected the

Question #1 eLearning module correctly.
1
The video called for getting information and calling
customer back. The correct answer in the item was pull
up the phone script.
Some font remnant appeared on some slides that made it
Other Technical 2
look like a checkbox
EVALUATING GOLDSTAR 28

Additional learner quotes:

Did he change outfits? (and chuckling)- during video
Im caught up looking at his clothes
Whats this checkbox for?- artifact on screen looks like a check box to click on
Am I supposed to read all of these?- referencing the scripts
I dont know what Im supposed to do here - referencing slide 8 instructions
That was fun, I like that, cute - truck bed sorting activity
I would rather have had the video open in full screen and remove Bryan from the screen
Its distracting.
I feel like I am being punished for choosing the wrong button- in response to selecting
the Etiquette Video when what was wanted was to do the practice. Result of not having
clear directions, making the wrong selection and then having to watch it all over again to
get to the practice.
The speaker in the video was very personable.

Appendix G
EVALUATING GOLDSTAR 29

VI. References

Russ-Eft, D. F., & Preskill, H. S. (2009). Evaluation in organizations: a systematic

approach to enhancing learning, performance, and change. New York, NY: Basic Books.