Developing

Questionnaire Practice

8/03/2020 Riesanti
 Data collection methods

Interviewing

Questionnaires

Observational studies

Projective tests

8/03/2020 Riesanti
 OBJECTIVES IN WRITING A QUESTIONNAIRE

to elicit the information that is

required to enable the researcher to
answer the objectives of the survey.

8/03/2020 Riesanti
 Strengths
They can be administered to large numbers of people using a variety of
methods

The use of questionnaires is more anonymous than face-to-face interviews.

Standardized questionnaires allow you to compare responses across various

groups and products.

Results can be used as input to other methods such as interviews, focus

groups, and usability testing.

Modern online tools make basic data analysis easy and fast (though
interpretation still requires the human touch).
[Wilson, 2013: 34-35]
8/03/2020 Riesanti
 Weaknesses

Many user-centered design (UCD) practitioners have little

formal training

Questionnaire design requires a wide range of knowledge

and training.

Questionnaires, like other self-report methods, suffer from

problems of recall, order effects, context effects, and
estimation bias.

8/03/2020 Riesanti
 Weaknesses
Questionnaires with many open-ended questions can generate large amounts
of data that take significant effort to analyze and more time to interpret.

Return rates can be low with online and mail surveys, and the nature of the
sample is not always clear.

Online and mail questionnaires are not as flexible as interviews or focus

groups where you can ask follow-up questions.

Survey fatigue, the overuse of online surveys, can be a serious problem, even
for normally dedicated respondents.

Privacy laws have become a barrier to online and mailed surveys

Wilson, 2013: 35-36]

Riesanti
 questionnaire assumptions
The questionnaire is tapping the concepts of
interest.
The potential respondents are competent sources
of the information you need.
The respondents are willing to commit the time
to answer your questionnaire
All of the respondents understand your questions
in the same way.
People will follow the rules of the survey and
provide honest data.

Wilson, 2013: 32
8/03/2020 Riesanti
 Sequencing the sections

Exclusion question

Screening questions

Main questionnaire

8/03/2020 Riesanti
 Exclusion/ Security question
• A common, although not universal, practice is
to exclude respondents from research surveys
• The security question is usually asked as a
prompted question, with respondents shown
a list of industries and professions.

8/03/2020 Riesanti
 Exclusion/ Security question (2)

8/03/2020
Riesanti
 Screening questions
• With all data collection methods other than
face-to-face interviewing these questions
must be asked at the beginning to ensure
eligibility

8/03/2020 Riesanti
 Screening questions (2)

8/03/2020
Riesanti
 Main questionnaire
• Once into the main questionnaire, the writer must
consider the order in which the various topics are
presented to the respondents.
• As a rule, it is better to work from the most general
topics through to the most specific.
• It is generally advisable to start any section of the
interview with behavioural questions before going on
to ask attitudes and images
• If the interview is to include questions of a sensitive
nature, then
• they should not be asked right at the beginning of the
interview

8/03/2020 Riesanti
 Flow chart to plan questionnaire

8/03/2020 Riesanti
{Brace, 2008: 42]
 Type of Question

Personal Factual Factual questions informant factual

questions about others questions

Questions about
Questions about Questions about normative
attitudes beliefs standards and
values

Questions about
knowledge

[Bryman, 2012: 253]

 Personal factual questions
• These are questions that ask the respondent to
provide personal information, such as age,
education, occupation, marital status, income,
and so on.
• This kind of question also includes questions
about behaviour
• Such factual questions may have to rely on the
respondents’ memories, as when they are asked
about such things as frequency of church
attendance, how often they visit the cinema, or
when they last ate out in a restaurant.

8/03/2020 Riesanti
 Factual questions about others.

• An example of such a question would be one

about household income, which would require
respondents to consider their own incomes in
conjunction with those of their partners.
• Like personal factual questions, an element of
reliance on memory recall is likely to be
present

8/03/2020 Riesanti
 Informant factual questions.
• This kind of question can also be found in
certain contexts,as when people are asked
about such things as the size of the firm for
which they work, who owns it,whether it
employs certain technologies, and whether it
has certain specialist functions.
• Such questions are essentially about
characteristics of an entity of which they have
knowledge, in this case, a fi rm.

8/03/2020 Riesanti
 Questions about attitudes.
• Questions about attitudes are very common in
both structured interview and self-completion
questionnaire research.
• The Likert scale is one of the most frequently
encountered formats for measuring attitudes.

8/03/2020 Riesanti
 Questions about beliefs
• Respondents are frequently asked about their
beliefs, possibly religious and political beliefs.
• Another form of asking questions about beliefs is
when respondents are asked whether they
believe that certain matters are true or false
– for example, a question asking whether the
respondent believes the UK is better off as a result of
being a member of the European Union.
• Alternatively, in a survey about crime,
respondents might be asked to indicate whether
they believe that the incidence of certain crimes
is increasing.
8/03/2020 Riesanti
Questions about normative standards and
values
• Respondents may be asked to indicate what
principles of behaviour influence them or they
hold dear.
• The elicitation of such norms of behaviour is
likely to have considerable overlap with
questions about attitudes and beliefs, since
norms and values can be construed as having
elements of both.

8/03/2020 Riesanti
 Questions about knowledge
• Questions can sometimes be employed to
‘test’ respondents’ knowledge in an area.
• For example, as part of their study of the role
of the mass media in the public understanding
of science, Hargreaves et al. (n.d.) asked
survey respondents to answer a large number
of knowledge questions relating to scientifi c
issues.

8/03/2020 Riesanti
 The Social Context of Question Asking

• The precise wording of questions plays a vital

role in determining the answers given by
respondents.
• This fact is not appreciated as fully as it should
be, even in ordinary conversation

8/03/2020 Riesanti
 Two priests, a Dominican and a Jesuit, are discussing whether
it is a sin to smoke and pray at the same time. After failing to
reach a conclusion, each goes off to consult his respective
superior. The nextweek they meet again. The Dominican says,
“Well, what did your superior say?”

The Jesuit responds, “He said it was all right.”

“That’s funny,” the Dominican replies. “My superior said it was

a sin.”
The Jesuit says, “What did you ask him?”

The Dominican replies, “I asked him if it was all right to smoke

while praying.”

“Oh,” says the Jesuit. “I asked my superior if it was all right to

pray while smoking.”
8/03/2020 Riesanti
 Types of question and data

Open-ended

Closed-Ended

8/03/2020 Riesanti
 Types of question and data-
Advantages
Useful when you do not know much
about a particular topic and thus
cannot generate credible response
categories.
Useful when the list of known
responses is very long.
Good for exploratory studies at the
beginning of projects.
Helpful as a follow-up to a closed
question. For example, if you ask a
person to rate the usability of a
Useful for getting at strong opinions closed
or topics that may have been missed questions.
by the questionnaire designer.
Efficient when you are asking
questions that can be easily recalled
without a list
8/03/2020
open or closed
Disadvantages
Can be demanding for respondents,
especially if you ask too broad a
question.
Typically produces many responses
but only a few on each topic.
Takes significant coding effort.
Is sometimes difficult to compare
the results of open-ended questions
across
Higherthe sample. Some
nonresponse rate than
Requires more time to answer.
Wilson, 2013:45
Riesanti
 Types of question and data-
Advantages
Easier for respondents to answer than
open questions.
Easy to code and analyze.
Appropriate when you are certain that you have
covered the list of possible responses.
8/03/2020
open or closed
Disadvantages
Respondents may feel that they have to
choose an alternative that isn’t what they
view as the “best” answer.
Some closed questions require research to
identify the appropriate response categories
Riesanti
Wilson, 2013:45
 Likert Scales and Guttman Scales

[Brace, 2008:66]

8/03/2020 Riesanti
 Likert Scales and Guttman Scales

[Bradburn dkk, 2004: 128]

8/03/2020 Riesanti
 Planning and Developing
a Questionnaire
Determine the Pilot test the
What general questions What type of question Devise an explicit data
purpose of your should I ask? structure should I use? analysis plan. questionnaire and
data collection tools

Consider how to
Determine the sampling Evaluate your draft establish trust, increase
Decide appropriateness
requirements questions. rewards, and reduce
costs

Gather requirements
Consider your Review the questions
and general questions Create and review
capabilities for relevance
from stakeholders

[Wilson, 2013]
8/03/2020 Riesanti
 1. Determine the purpose of your data
collection
a. Understanding user needs
b. Gathering information about particular attributes of your
users, their tasks, and environments
c. Validating design decisions
d. Understanding user attitudes or opinions
e. Comparing the attitudes of different groups
f. Gathering facts
g. Assessing product usability or satisfaction
h. Gathering information comparing competitive products
i. Eliciting knowledge from experts
j. Convincing management to do something

8/03/2020 Riesanti
 2. Decide appropriateness
a. Can questionnaire data provide useful information
that will answer your design or business questions?
b. Do you have enough information on the topic to
design a useful questionnaire?
c. Do the benefits of designing a questionnaire and
survey outweigh the costs involved?
d. Do you have the resources necessary to design and
implement a questionnaire study?
e. Can you gather valid and reliable data using a
questionnaire?
f. Do you have access to a reasonable sample of
respondents who match your target audience?

8/03/2020 Riesanti
3. Gather requirements & general questions

a. Interview key stakeholders about what they

know and do not know about users and how
they use a product.
b. Brainstorm with the product team about what
they want to learn from a questionnaire study.
c. Conduct a short brainwriting session
d. Conduct a focus group to find out what issues
are important to key stakeholders and target
user groups.

8/03/2020 Riesanti
 4. Consider your capabilities

experience and training to design, implement,

and analyze the data from a questionnaire
study.
The design of questionnaires requires:
• background experience in many areas, such as
scale development, psychometrics (theories
and methods of psychological measurement),
• sampling, and
• content analysis.
8/03/2020 Riesanti
 5. Determine the sampling requirements for
your questionnaire study
1. Specify or at least acknowledge the sampling
process.
– Probability samples,).
– Haphazard samples
– Convenience sample.
– Purposive samples
– Snowball sampling
2. Compile a list of people in the target populations
from which you can choose a sample, or decide how
and where you are going to publicize the survey to
recruit your respondents.
3. Select the sample.

8/03/2020 Riesanti
 6. What general questions should I ask?

a.Motivating to the respondent

b.Interpretable by multiple respondents in the
same way
c.Answerable accurately (e.g., whether
respondents can recall and report on past
behaviors)
d.Relevant to all respondents

8/03/2020 Riesanti
7. What type of question structure should
I use?

8/03/2020 Riesanti
 Mutually Exclusive Responses

8/03/2020 Riesanti
 8. Evaluate your draft questions.

Conduct an expert review

Ask a small focus

Conduct brief interviews

8/03/2020 Riesanti
 9. Create a prototype and review it.

• Design a prototype questionnaire & review

the prototype against principles of
questionnaire design
• Interview a few people not closely associated
with the project
• If you have survey experts, ask them to review
the prototype

8/03/2020 Riesanti
10. Review the questions for relevance

Relevance is an important attribute of

questionnaires

8/03/2020 Riesanti
 11. Consider how to establish trust

8/03/2020 Riesanti
12. Devise an explicit data analysis plan
– Whether you need specific answer categories
– How you will code missing data.
– How you will code unusual answers.
– What method you will use for coding open-ended data
– What analyses you will do on single questions and sets of questions.
Consider the following types of analyses:
• Content analysis,
• Measures of central tendency (mean, median, mode) and variability
(standard deviation, variance).
• Outlier analysis.
• Exploratory analyses (box and whisker plots, histograms).
• Confidence intervals.
– Any hypotheses that you may have and what questions will be used to
test those hypotheses.
– Whether you want to cross-tabulate data from different questions.
8/03/2020 Riesanti
 13. Pilot test the questionnaire

• Pretesting questionnaires is essential for discovering

flaws and usability issues with cover letters, the
questionnaire itself, and the method of administration
• Respondents should be encouraged to comment on
any aspect of the questionnaire, including unclear or
ambiguous questions, the completeness and clarity of
the response categories, biased questions,terminology,
legibility (Is the text size large enough for older
respondents?), sentence structure, and threatening
questions

8/03/2020 Riesanti
 Types of Pilot Survey

 informal pilots carried out with a small number of

colleagues;
 cognitive interviewing in which the questionnaire is
tested amongst respondents;
 accompanied interviewing which may be used
principally to test for interviewer and routeing errors;
 large-scale pilot studies where a larger number of
interviews can be used to test for completeness of
brand lists or incidence of sub-groups;
 dynamic pilots, where question wording is changed
between interviews to test alternatives based on
responses received
[Brace, 2008]
8/03/2020 Riesanti
 Reliable & Valid questionnaire

1. Conduct an item analysis to determine

which questions are “good” and which are
“bad.”
2. Test the reliability of the questionnaire
3. Assess the validity of the questionnaire
4. Develop standards and norms.

[Dumas (2003) in Wilson (2013])

8/03/2020 Riesanti
 Test the reliability of the questionnaire

• Ambiguous questions or response categories that

lead respondents to interpret the questions
differently
• Use of terms that are not understood by all
respondents.
• Double questions where it is not clear what part
of the double question participants are
responding to.
• Differences in the administration of the
questionnaire
[Dumas (2003) in Wilson (2013])
8/03/2020 Riesanti
 Assess the validity of the questionnaire
a. Content validity: Does the questionnaire include content that is
relevant to the purpose of the questionnaire?
b. Criterion validity: Do the results of the questionnaire correlate
with subsequent outcomes such as sales figures or website hits?
c. Convergent validity: Do the results for one questionnaire correlate
with the results of other questionnaires or methods that attempt
to measure the same thing?
d. Internal validity : Internal validity deals with the confidence you
place in the causal statements you make from a study.
e. External validity : External validity refers to the generalizability of
causal relationships. In a somewhat looser sense, external validity
refers to the relevance of the findings in a wider context (“the real
world”).

[Dumas (2003) in Wilson (2013])

8/03/2020 Riesanti
 Develop standards and norms

A final step in the development of psychometric

questionnaires is to create norms regarding
what is good and bad.

[Dumas (2003) in Wilson (2013])

8/03/2020 Riesanti
 Words and Questions to Avoid in
Your Questionnaire
1. Avoid double-barreled questions
2. Avoid double negative questions
3. Avoid unnecessary repetitiveness
4. Avoid ‘‘leading’’ or ‘‘loaded’’ questions
5. Avoid biased questions

[Gideon, 2012]

8/03/2020 Riesanti
 Double-Barreled Questions
Johnson and Christensen (2010) define ‘‘double
barreled’ ’questions as those that ‘‘combine
two or more issues or attitudes in
a single item’’ (p.175).

‘Do you think young children should have

access to cell phones and credit cards?’

8/03/2020 Riesanti
 Double Negative Questions

Please tell me whether you agree or disagree

with the following statement about teachers
in public schools:
Teachers should not be required to supervise
students in the halls, the lunchroom, and the
school parking lot (Converse and Presser
(1986: p. 13).

8/03/2020 Riesanti
 Avoid unnecessary repetitiveness

• it is also time-consuming, as it prolongs the

questionnaire.
• Repetitiveness may also be demoralizing.
• Respondents are more likely to develop
fatigue and frustration, and as a result will skip
questions or stop taking the survey all
together.
• As a result, non-response error will increase,
and survey reliability will decrease
8/03/2020 Riesanti
 Okpechi, S. O., & Belmasrour, R. (2016). Women
accountants in practicing accounting firms: Their
status, investments and returns. In Handbook on
Well-Being of Working Women (pp. 317-343).
Springer, Dordrecht.

8/03/2020 Riesanti
 Research Methodology
A structured questionnaire consisting of 43 questions.

Women practicing accountants

about their perceptions on their education, training, status,

investments and retention,

There are also questions on work environment, family, sex

and gender issues that appear to be the challenging
constraints that women face when making career decisions.

8/03/2020 Riesanti
 Research Methodology

The questionnaire was designed, pilot-tested and

revised after comments were received from four
professors at University of New Orleans and
Southern University at New Orleans.

A small sample of five women practicing

accountants in New Orleans metropolis was
requested to complete it and to provide their
comments.

8/03/2020 Riesanti
 Research Methodology

A follow- up cognitive interview was conducted in order to

learn about their thought process, appropriateness or
inappropriateness of the questions.

This pre-test process is critical for a judgmental sampling

technique that was used. Willis ( 2004 ) used this technique to
ensure that “he, the researcher and respondent share
identical mental representation of the task.”

The comments from the pilot study resulted in the re-writing

and reduction of the number of questions from 55 to 43 and
the actual time for completing the questionnaire reduced
from 12 to 9 min.

8/03/2020 Riesanti
 Sampling Technique

Purposive or judgmental sampling technique was justifiably used because

not all practicing women accountants were surveyed.

To be included in the sample, a respondent must be a qualifi ed certified

public accountant and working in a practicing accounting fi rm.

Because of the special nature of the sample, a wide range of methods

were used that include identification of respondents through:
• professors, students and individuals; and, members of Association of Women Accountants
(ASWA) in New Orleans. At their continuing professional education training seminar,

over 40 questionnaires were distributed to their members.

8/03/2020 Riesanti
 Sources
1. Sekaran, U. (2016). Bougie. M,” Research Methods for Business: A
Skill Building Approach”. UK: John Wiley & Sons.
2. Blumberg, B. F., Cooper, D. R., & Schindler, P. S. (2014). Business
research methods. McGraw-hill education.
3. Wilson, C. (2013). Credible checklists and quality questionnaires: A
user-centered design method. Newnes.
4. Gideon, L. (2012). Handbook of survey methodology for the social
sciences. New York, NY: Springer.
5. Brace, I. (2008). Questionnaire design: How to plan, structure and
write survey material for effective market research. Kogan Page
Publishers.
6. Bryman, A. (2012). Social research methods. Oxford university
press.
7. Bradburn, N. M., Sudman, S., & Wansink, B. (2004). Asking
questions: the definitive guide to questionnaire design--for market
research, political polls, and social and health questionnaires. John
Wiley & Sons.

8/03/2020 Riesanti