QUANTITATIVE RESEARCH METHODS

Irina Shklovski

Quantitative Research Methods

Include a wide variety of laboratory and nonlaboratory procedures Involve measurement

Quantitative Research Methods

Measurement
Populations

and Sampling Random Assignment Generalizability

Quantitative Research Methods

Measurement
Populations and Sampling Random Assignment Generalizability

Time
Cross-sectional studies & single experiments Longitudinal studies & repeated measures

Quantitative Research Methods

Method
Experiments

& Quasi-experiments Behavioral Measures Questionnaires & Surveys Social Network Analysis Archival and Meta-Analysis

What we will talk about today

Measurement
Population

& Sampling Random Assignment Generalizability

Method
Experiments

& Quasi-experiments Questionnaires & Surveys

Measurement Sampling

Specify your population of concern Sampling

Selecting

respondents from population of concern Random sampling Systematic selection Stratified sampling Convenience sampling Snowball sampling

Sampling Biases

Non-response bias
Be

persistent Offer incentives and rewards Make it look important

Volunteer bias
Some

people volunteer reliably more than others

for a variety of tasks

Random assignment

Different from random sampling Mostly used for experiments or quazi-experiments Protects against unsuspected sources of bias Does NOT guarantee to balance out the differences between participants Chance is LUMPY

Generalizability

How do you know that what you found in your research study is, in fact, a general trend? Does A really, always cause B? If A happens, is B really as likely to happen as you claim? Always? Under certain conditions?

Association vs. Causality

Thanks to Sara Kiesler for these graphs!

Experiments & Quasiexperiments

experiment
Pronunciation: \ik-sper--mnt also -spir-\ Function: noun Etymology: Middle English, from Anglo-French esperiment, from Latin experimentum, from experiri Date: 14th century

An operation or procedure carried out under controlled conditions to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law

Experiments & Quasiexperiments

Key feature common to all experiments:

To

deliberately vary something in order to discover what happens to something else later To seek the effects of presumed causes

An Experiment is

A controlled empirical test of a hypothesis. Hypotheses include:

A causes B A is bigger, faster, better than B A changes more than B when we do X

Two requirements:
Independent variable that can be manipulated Dependent variable that can be measured

Experiments in Research

Comparing one design or process to another Deciding on the importance of a particular feature in a user interface Evaluating a technology or a social intervention in a controlled environment Finding out what really causes an effect Finding out if an effect really exists

Remember

Experiments explore the effects of things that can be MANIPULATED (but there is a caveat)

Types of Experiments

Randomized units/participants assigned to receive treatment or alternative condition randomly Quazi no random assignment Natural contrasting a naturally occurring event (i.e. disaster) with a comparison condition

If your study involves experiments

Experimental design:
Shadish W.R., Cook T.D. & Campbell P.T. (2002) Experimental and Quasi-Experimental Design for Generalized Causal Inference. Boston, Mass: Houghton Mifflin

Experimental data analysis:

Bruning, J. L. & Kintz, B. L. (1997). Computational handbook of statistics (4th ed.). New York: Longman.

Questionnaires & Surveys

Self-report measures
Questionnaires
Interviews Diaries

& surveys

Types
Structured Open-ended

Questionnaires & Surveys

Advantages
Sample

large populations (cheap on materials &

effort) Efficiently ask a lot of questions

Disadvantages
Self-report

is fallible Response biases are unavoidable

Response biases

Relying on peoples memory of events & behaviors

Emotional

states can prime memory Recency effects Routines are deceiving

Social desirability
Solution:

none that are simple

Yea-saying
Solution:

vary the direction of response alternatives

General Survey Biases

Sampling are respondents representative of population of interest? How were they selected? Coverage do all persons in the population have an equal change of getting selected? Measurement question wording & ordering can obstruct interpretation Non-response people who respond differ from those that do not

Design is KEY

Format booklet, printed vertical, one-sided Question ordering earlier questions can prime answers to later questions Page layout group similar items & use consistent fonts and response categories Pre-testing conduct think-alouds with volunteers demographically similar to expected participants

Common Problems

Avoid complicated & double-barrel questions

Complexity

increases errors & non-response

Navigation is paramount make sure the survey is EASY to follow Open-ended questions
The

size of the field allotted will determine the number of words amount differences have little impact

Incentive is key
BUT

If your study involves surveys

Designing surveys:
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys : the tailored design method (3rd ed.). Hoboken, N.J.: Wiley & Sons. Fowler, F. J. (1995). Improving survey questions : design and evaluation. Thousand Oaks: Sage Publications.

Analyzing data:
Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

So what?

Difference between quantitative methods is in the questions they can answer There are a LOT of methods and even more statistical techniques Regardless of the method, if its not an experiment, you CAN NOT prove causation

Things we did NOT talk about

Reliability assessments Validity assessments Statistical analysis of data Interpretation of results