Questionnaire Design & Sampling Issues

Prof. Rushen Chahal

Role of a Questionnaire
Survey Purpose & Objectives
The Questionnaire Approach Level of Data Analysis

Customer Needs

Research Findings Conclusions and Recommendations Managerial Action Researcher Follow-up

Survey Timetable

Issues continually addressed throughout the process.

Respondent Information

Prof. Rushen Chahal

Criteria for a Good Questionnaire

=
Provide decision-making information; Consider the respondent; Meet editing and coding requirements.
Prof. Rushen Chahal

Key Questionnaire Mechanics

Going through each questionnaire to ensure that skip patterns were followed and the required questions were filled out.

Sequence in which questions are asked, based on a respondents answer.

The process of grouping and assigning numeric codes to the various responses to a question.
Prof. Rushen Chahal

The Questionnaire Design Process

Determine Survey Objectives, Resources, and Constraints Determine the Data Collection Method Determine the Question Response Format Decide on the Question Wording Establish Questionnaire Flow and Layout Evaluate the Questionnaire Obtain Approval of all Relevant Parties Pretest and Revise the Questionnaire Prepare the Final Copy Execute the Survey
Prof. Rushen Chahal

Key Questionnaire Mechanics

As directed by management

Determine Survey Objectives, Resources, and Constraints

Shaped by time & budget

Determine the Data Collection Method

Knowledge of respondent key

Determine the Question Response Format

Remember dos and donts

Decide on the Question Wording

Questions should flow logically

Establish Questionnaire Flow and Layout

Prof. Rushen Chahal

Key Questionnaire Mechanics

For length, missing & unnecessary questions, etc

Evaluate the Questionnaire

Ensure Mgmt. buy-in

Obtain Approval of all Relevant Parties

Survey Says

Test & revise questions

Pretest and Revise the Questionnaire

Decide on format/layout

Prepare the Final Copy

Mail, telephone, etc. Prof. Rushen Chahal

Execute the Survey

The Response Format

Open and Closed Questions

Questions to which the respondent replies in his or her own words.

Probed Vs. Un-probed.

Questions requiring respondents to choose from a list of answers.

Dichotomous: Choice is between two answers. Multiple Choice: Choice is among three or more options. Scaled Responses: Designed to capture the intensity of respondents feelings.
Prof. Rushen Chahal

Questionnaire Dos

Be as brief as is appropriate for your audience; Be grammatically simple; Be focused on a single issue or topic; Use the respondents core vocabulary; Use plenty of white space between the questions; Number the questions; Use consistence scales; State instructions clearly; Questions should be interpreted equally by respondents.
Prof. Rushen Chahal

Questionnaire Don'ts

Biasing the respondent; Using loaded or leading phrasing; Using words overstating the condition; Assuming criteria that are not obvious; Using specific example for a general case; Being beyond the respondent's ability to answer; Requiring the respondent to guess at a generalization; Asking for specifics when only generalities will be remembered.
Prof. Rushen Chahal

Establishing Questionnaire Flow

Screeners - Qualifying Questions Ask general questions first - establish respondent buy-in Basic questions to lay the groundwork for upcoming questions Example: Have you shopped at Macys in the past month? Warm-ups - First Few Questions: Gets the respondent thinking about the topic at hand Establishes parameters about the respondents attitudes, behavior, etc. Example: How often do you go shopping? Transitions - First Third of Questions: Questions that set the tone for the more difficult questions to come Example: Now Im going to ask you some more difficult questions Complicated - Second Third of Questions: Use of rating scales for attributes, attitudes, beliefs, opinion, etc Tackling controversial issues Example: How likely are you to go to the movies? (scale 1 to X) Classification - Last Third of Questions: Personal & demographic type questions Example: What is your religion? Prof. Rushen Chahal

Build them up during the survey process with increasing difficult/thought provoking questions and conclude with more probing questions.

Questionnaire Considerations
Address these issues when designing and conducting a survey

Incidence rate; Time and budget issues; Purpose of the information; Quality of information desired; Getting a representative sample; Willingness of respondents to participate; Availability of respondents to participate.
Prof. Rushen Chahal

Questionnaire Design Issues

Start with opening letter indicating: Who you are, why you are doing the the survey, how long it will take, how they were selected, the surveys purpose, whether its confidential & anonymous, thank them for participating, etc. Use plenty of white space between the questions; Ensure the format, font, layout, and appearance is consistent; State the instructions clearly; Clarify questions as they are asked if necessary such as: Clarify one, pick two, etc. Allow enough space for open-ended questions; Ensure questions are interrelated - not only stand alone questions Include a closing remark - Thank You , etc.
Prof. Rushen Chahal

Questionnaire Design Issues

Use screening questions as appropriate; Begin with interesting questions to nab respondent; Ask general questions first harder/more invasive ones last; Put instructions in capital letters; Use proper transitions throughout the questionnaire; Ensure skip patterns are in place as needed.

Prof. Rushen Chahal

Chapter Nine

The Internet Impact

The questionnaire appearance consistency is easier to achieve; The questionnaire can be checked for typos easily; The survey can be created quickly; Skip patterns can be efficiently established; The survey can be distributed quickly for expert review & input.

Over reliance on electronic survey construction can lead to the researchers getting sloppy as he might think the software will do the work and correct any errors; The researcher might feel less connected to the process; Multiple versions of the survey might get circulated / distributed.
Prof. Rushen Chahal

The Internet Impact

The Internet Impact On Questionnaire Development

Email Surveys Cost & Profitability Software

Internet Surveys

Prof. Rushen Chahal

Global Research Issues

Cultural differences - gender, body language, behavioral; Traditions, religion, ways of conducting business, beliefs; Word usage differences - phrases, expressions, idioms; Acceptable & unacceptable types of questions; Receptive level of audience for given question types; Information The best way to execute the survey telephone,mail, etc.; Which issues are sensitive & how to approach them; Whether various dialects are present; What issues are most important to your audience.
Prof. Rushen Chahal

Basic Sampling Issues

Prof. Rushen Chahal

Prof. Rushen Chahal

The Concept of Sampling

The process of obtaining information from a subset of a larger group.

The entire group of people about whom information is needed; Also called the universe or population of interest.
Prof. Rushen Chahal

The Concept of Sampling

Census: Data collection from or about every member of the population of interest. Also called canvassing the population by asking everyone a set of questions. Sample: A subset of all the members of a population of interest.
Prof. Rushen Chahal

Defining the Population of Interest

Some Basis to Consider Geographic Area: City, county, state(s), MSA, country, etc. Demographics: Age, income, ethnicity / race, religion, occupation, etc. Usage: Frequency, purpose, rate, etc. Awareness: How did they become aware, why are they not aware, what are their perceptions, attitudes, beliefs, etc.
Prof. Rushen Chahal

Developing a Sampling Plan

1. 2. 3. 4. 5. 6. 7. Define the Population of Interest Choose the Data Collection Method Identify the Sampling Frame Select a Sampling Method Determine Sample Size Develop Operational Procedures Execute the Operational Sampling Plan

Prof. Rushen Chahal

Developing a Sampling Plan

Define the Population of Interest: Determine the characteristics of those you are interested in studying. Determine which group of people or entities about which you want to learn more.

1. Define the Population of Interest 2. Choose the Data Collection Method

Choose the Data Collection Method: Determine how you collect the sample - such as mail, Internet, telephone, mall intercept, etc.

3. Identify the Sampling Frame 4. Select a Sampling Method 5. Determine Sample Size 6. Develop Operational Procedures 7. Execute the Operational Sampling Plan

Identify the Sampling Frame: A list of population elements from which units to be sampled can be selected or a specified procedure for generating such a list.
Prof. Rushen Chahal

Developing a Sampling Plan

Selecting a Sampling Method: Determine how you will get the sample list through probability or non-probability methods.

1. Define the Population of Interest 2. Choose the Data Collection Method 3. Identify the Sampling Frame

Probability Sampling: Samples in which every element of the population has a known, nonzero, likelihood of selection.

4. Select a Sampling Method 5. Determine Sample Size 6. Develop Operational Procedures 7. Execute the Operational Sampling Plan

Non-probability Sampling: Samples in which specific elements from the population have been selected on a nonrandom manner.
Prof. Rushen Chahal

Probability Sampling
For More Information

A sample selected by assigning a number to every element of the population and then using some method for randomly selecting elements to be in the sample such as random digit dialing.

A sample in which the entire population is numbered and elements are selected using a skip interval every nth name is selected.

Prof. Rushen Chahal

Probability Sampling

A sample that is forced to be more representative through simple random sampling of mutually exclusive and exhaustive subsets either proportionally or disproportionally. Good for data that are not normally distributed.

A sample in which the sampling units are selecting from a number of small geographic areas to reduce data collection costs.
Prof. Rushen Chahal

Non-Probability Sampling

A sample based on using people who are easily accessible - such as mall intercepts or other high traffic locations.

A sample in which the selection criteria are based on the researchers personal judgment about representativeness of the population under study. The researcher selects who should be in the study.
Prof. Rushen Chahal

Non-Probability Sampling

A sample in which additional respondents are selected based on referrals from initial respondents.

A sample in which quotas, based on demographic or classification factors selected by the researcher, are established for population subgroups.
Prof. Rushen Chahal

Developing a Sampling Plan

Determine the Sample Size: Determining the sample size will be based on factors such as the level of accuracy you want to achieve, the time and money you have to do the survey, and on the sampling collection method.

1. Define the Population of Interest 2. Choose the Data Collection Method 3. Identify the Sampling Frame 4. Select a Sampling Method 5. Determine Sample Size

Develop Operational Procedures: Your operational plan to conduct the probability or non-probability sampling. Determine the phases of the sample selection process. Multi-stage sampling involves combining sampling methods.
Prof. Rushen Chahal

6. Develop Operational Procedures 7. Execute the Operational Sampling Plan

Developing a Sampling Plan

Execute the Operational Sampling Plan:

1. Define the Population of Interest

The execution phase of the research. Administering the questionnaire - sending the mailers, making the phone calls, conducting the mall intercepts, etc.

2. Choose the Data Collection Method 3. Identify the Sampling Frame 4. Select a Sampling Method 5. Determine Sample Size 6. Develop Operational Procedures 7. Execute the Operational Sampling Plan

Prof. Rushen Chahal

Sampling and Non-Sampling Errors

Sampling Error:

Error that occurs because the sample selected is not perfectly representative of the population.

Non-Sampling Error:

All error other than sampling error - also called measurement error.
Prof. Rushen Chahal

Sampling and Non-Sampling Errors

X

+/-

ES

+/-

ENS

Where:
= sample mean = true population mean

SN

ES = sampling error E
= non-sampling, or measurement, error
Prof. Rushen Chahal

Internet Sampling

Targeted respondents can complete the survey at their convenience; The data collection is relatively inexpensive; Survey software can facilitate the data collection process; The survey can be completed quickly;

The sample might not be representative of the population; You can not always be sure who is completing the survey; Maintaining respondent confidentially can be a challenge; Data security issues can be difficult to manage.

Prof. Rushen Chahal

Sample Size Determination

Prof. Rushen Chahal

Chapter Eleven Objectives

To learn the financial and statistical issues in the determination of the sample size. To discover methods for determining the sample size. To gain an appreciation of a normal distribution of data. To understand population, sample, and sampling distributions. To distinguish between point and interval estimates. To recognize problems involving sampling means and proportions.

Prof. Rushen Chahal

Sample Size for Probability Samples

Census: Population canvas - not really a sample Asking the entire population Judgment: Best guess of experts Draw on your experience to determine sample size Conventional: What have others done? See what the sample size has been for similar studies
Prof. Rushen Chahal

Sample Size for Probability Samples

Arbitrary / Rule of Thumb: Applies some industry accepted rule of thumb Generally better for smaller populations Picking x percent of the population to be in the sample Budget Available: What can we afford? How much do we want to spend? How much time are we allotting for each respondent Statistical: Variance, SD, confidence interval play a key role
Prof. Rushen Chahal

The Normal Distribution

Central Limit Theorem:
The idea that a distribution of a large number of sample means or sample proportions will approximate a normal distribution - regardless of the distribution of the population from which they were drawn.

Normal Distribution:
The continuous distribution that is bell shaped and symmetrical about the mean. The mean, median, and mode are equal. About 68% of the observations are within +/- one standard deviation, 96% are within two standard deviations, and 99+% are within three standard deviations of the mean.
Prof. Rushen Chahal

The Normal Distribution

Proportionate Properties:
A feature that the number of observations falling between the mean and a given number of standard deviations from the mean is the same for all normal distributions.

Standard Normal Distribution:

Normal distribution with a mean of zero and a standard deviation of one.
Prof. Rushen Chahal

The Normal Distribution

Standard Deviation:
The measure of dispersion calculated by subtracting the mean of the series from each value in a series, squaring each result, summing the results, dividing the sum by the number of observations minus 1, and taking the square root of this value.

sum

Standard Deviation

(X1- X) (N-1)

Prof. Rushen Chahal

Population and Sampling Distributions

Population Distributions:
The frequency distribution of all the elements of a population.

Sampling Distributions:
The frequency distribution of all the elements of an individual sample.
Prof. Rushen Chahal

Sampling Distribution of the Mean

Sampling Distribution of the Mean:
The theoretical frequency distribution of the means of all possible samples of a given size drawn from a particular population; it is normally distributed.

Standard Error of the Mean:

Standard deviation of a distribution of sample means.
Prof. Rushen Chahal

Point and Interval Estimates

Point Estimate:
The particular estimate of a population value.

Interval Estimate:
The interval or range of values within which the true population value is estimated to fall.
Prof. Rushen Chahal

Point and Interval Estimates

Confidence Level:
The probability that a particular interval will include the true population value - also called the confidence coefficient.

Confidence Interval:
The interval that, at the specified confidence level, includes the true population value.
Prof. Rushen Chahal

Sampling Distribution of the Proportion

Sampling Distribution of the Proportion:

The relative frequency distribution of the sample proportions of many random samples of a given size drawn from a particular population; It is normally distributed.

Prof. Rushen Chahal

Determining Sample Size

Problems Involving Means:

N=

Z *o

N = Sample Size Z = level of confidence desired in the results. A 95% confidence interval would make Z=1.96. In other words, if we conduct this survey 100 times, at least 95 of those time the true population average would fall within out interval estimate. = Population standard deviation. e = error rate - a management decision (ex. plus or minus 3%)

Prof. Rushen Chahal

Determining Sample Size

Problems Involving Proportions:
2

N = Sample Size Z = level of confidence desired in the results. A 95% confidence interval would make Z=1.96. In other words, we would by 95% confident that the average results in the whole population (were we to survey the whole population) would be within 1.96 standard deviations from the mean. p = variance (how different you predict the population is), q = 100-p e = error rate - a management decision (ex. plus or minus 3%)

Prof. Rushen Chahal

Determining Sample Size

To revise down your original sample size while maintaining the same level of accuracy. This technique is good for small populations.

RSS
(revised sample size)

N
(original sample size)

Population - Original Sample Size Population - 1

Prof. Rushen Chahal

Determining Sample Size

You need to over sample since its unlikely that everyone you contact will agree to answer the questionnaire. If you determine that you need 800 respondents in the survey (RSS), and a typical response rate for your type of survey is 30%, then use the calculations to below to get the number of people you would actually have to contact to get 800 completed responses.
O = RSS/.30 O = RSS/.30 or or O = 800/.3 O = 800/.3 O = 2,667 O = 2,667 Thus your new sample size is 2,667 - with 30% Thus your new sample size is 2,667 - with 30% response you can expect 800 people to respond. response you can expect 800 people to respond.
Prof. Rushen Chahal