Evaluating Research Methods

Evaluating the validity of a piece of psychological research requires you to do a number of things. If you are evaluating a study, you need to: Identify the features of the study that threaten its validity (ie, poor sample) Explain the impact these features might have had (ie, limiting generalizability) Assess the seriousness and/or likelihood of this impact. There are two main types of validity that you look for in a study: internal and external. Internal validity is a measure of how well the study demonstrates a causal relationship between variables. Things that may limit internal validity include: 1. poor operationalization of variables (ie, were the variables measured in a consistent manner?) 2. lack of standardization (ie, were all subjects treated identically?) 3. presence of confounding variables (ie, are their other, uncontrolled differences between the subjects that may account for changes in the dependent variable?) 4. expectancy effects (ie, did the subjects or researchers knowledge about aspects of the study, such as the purpose or which group they were in, influence the results?) 5. confirmation bias (ie, did the researchers build in methods to collect data in an unbiased manner?) External validity is a measure of how well the study can be generalized to the population at large. Things that may limit external validity include: 1. a non-representative sample 2. a setting or task that is artificial or too constricted to mimic a realworld scenario Part I: Assessing Validity Read the two scenarios below and try to identify at least one feature of each that potentially reduces its validity. Make note of (a) what the feature is; (b) the type of validity it threatens and the effect it might have, and (c) how serious a flaw you judge it to be. 1. Jeanette is conducting an experimental study of memory in high school students. She has recruited a sample of AP students and given them a word-learning task. They first do the task while music is playing and then in silence. She has operationalized memory as the number of words recalled from the learning task. 2. Peter is doing a study of the relationship between Facebook usage and learning. The university IT department provides him with data on 1) the amount of time a student spent logged onto Facebook in the previous semester, and 2) their GPA for the same semester. He is given (confidential) data on every fifth student enrolled from an alphabetical list. Part II: Analyzing Research 1. A researcher wants to find out if the characteristics of an audience

Evaluating Research Methods

affected performance of a skilled behavior. Twenty student participants were recruited from a college of music. All were learning to play the violin. Before the experiment began, the participants were independently rated on their musical ability and formed into pair of similar ability. Each participant was given the task of sight-reading an unfamiliar piece of music in front of an audience of five people. One of each pair was told that the audience consisted of members of the public. The other participants were told that the audience consisted of members of the city orchestra. The participants performances were recorded and then analyzed by a professional violinist who was unaware of the aims of the experiment. The number of errors made by each participant was recorded. The researcher presented her results as follows: Sight-reading errors in front of an audience perceived to be expert or non-expert musicians.

Expert Mean 3.2 Standard 1.8 deviation

Non-expert 1.6 1.5

a. State a suitable hypothesis for this experiment. b. Identify an extraneous variable the researcher would need to control in this experiment and explain the effect it might have had on the results if it had been left uncontrolled. c. Identify the dependent variable and describe how it was operationalized. d. Why was it necessary that the judge who counted the participants errors was unaware of the aims of the experiment? e. Suggest what the researcher might have concluded from the results of this experiment. A researcher wanted to find out if recent experiences with an authority figure had an effect on a persons willingness to obey authority. Participants were recruited from a university Psychology department. They were sent to a room in the university where they were asked to complete a screening questionnaire. The questionnaire was deliberately designed to be confusing. Once they had completed the questionnaire, a confederate wearing a badge reading Professor Jones: Senior Researcher checked their answers. For half the participants, he thanked them for their responses. For the other half, the confederate criticized the participants for filling in the questionnaire incorrectly. The participant was then taken to a laboratory where a different researcher gave them a series of meaningless, boring tasks to perform. The number of tasks the participant carried out before quitting was recorded. a. Explain how the IV and the DV in this experiment were operationally defined. b. State a suitable hypothesis for this experiment.

Evaluating Research Methods

c. Explain the problem raised by using two different groups of participants for the two conditions of this experiment. d. Suggest a suitable method for resolving (or at least minimizing) this problem. e. Outline one feature that might have affected the external validity of this experiment. f. Suggest why a different researcher was used for the second phase of this experiment.

Evaluating Research Methods

Answers
Part I: Assessing Validity 1. a) her students are not representative of the population (they are in AP classes); also, they do the work with music first every time making practice effects a possible confounding variable; b) external (for nonrepresentative) or internal (for confounding variable); c) both problems are pretty big.

2. a) presence of confounding variables. Maybe those who spend a lot of time on facebook are younger; b) internal; c) moderate
Part II: Analyzing Research 3. a. Performing in front of experts impairs performance OR If subjects believe they are being observed by professional musicians, then they will not play as well. b. Time of day, difficulty of piece of music, conditions in the room c. Quality of performance, defined as number of mistakes d. To eliminate confirmation bias (the tendency to look for evidence that supports your beliefs) f. Performing in front of experts impairs performance. 4. a. IV = whether the participant was politely thanked or criticized immediately prior to performing the task; DV = number of tasks the participant completed before quitting. b. Students who are insulted will quit sooner OR Students are less likely to comply when they have been recently treated poorly by an authority figure. c. The two different groups may differ in some way (maybe in how they respond to authority or ability to manage anger or self-esteem or degree of time they have) e. Make sure you have a large sample size and randomly assign them to groups. f. Psychology students may not be representative of the population at large and performing a meaningless, boring task is not some that people are typically asked to do in real life. g. Because they didnt want personal animosity or vengeance to confound their study.