Designing Scientific

Experiments
Dr. Gail P. Taylor
MBRS-RISE Coordinator
UT San Antonio
08/2006

References

CRITICAL THINKING, THE SCIENTIFIC METHOD, AND PAGE

25* OF GILBERT Dany S. Adams, Department of Biology,
Smith College, Northampton, MA 01063
http://www.sdbonline.org/SDBEduca/dany_adams/critical_thinkin
g.html#blurb

Validity: http://carbon.cudenver.edu/~lsherry/rem/validity.html

At the Bench, A Laboratory Navigator; updated edition. Kathy

Barker, Cold Spring Harbor Press, 2005

Scientific Method

Observe phenomenon & conceive ideas

Make predictions/develop a hypothesis
Devise a test/formulate experiment
Carry out experiments
Draw conclusions from results
Reject or support hypothesis

Types of Experiments
Science does not generally deal with
facts, but rather with evidence
Each experiment weakens or
strengthens a hypothesis
All evidence is not equal
Try to discern cause and effect!

Planning Experiments I
What ideas have you come up with?
Why is your idea important?
Have other people tested this idea
before?
http://www.pubmed.org

What type of background information is

available?
Define question/Develop hypothesis
(and null hypothesis)

Determining Causality
Causality can be difficult to prove
Different experimental designs impact differently
Correlative Evidence (weak evidence)
Found together in time or space
Loss of Function (stronger evidence)
Blocked a phenomenon

Gain of Function (strongest evidence)

Initiation of event leads to second event (additional function)

Example Protein X may be

involved in Cellular Aggregation
Show it
Correlative evidence (time and space):
Antibody used to detect:
Found in particular microorganism when aggregating
(and not when free living)
Found in area where cells are contacting one another
during aggregation

No causality; nothing beyond inference about

function
Clumping could cause the protein expression
Clumping and protein expression could be induced by
same causative agent
Could be completely coincidental

Example Protein X may be

Necessary for Cell Aggregation
Block it
Loss of Function - What does its loss do to
clumping?
Antibody to protein used to block it from
functioning.
Or knock out gene
Clumping no longer takes place
Need controls Clumping specifically and only being inhibited
cells not dying
May support real clumping agent to function

Therefore it is necessary for clumping

Example Protein X may be

Sufficient for Cellular Aggregation
Move it
Gain of Function
In organism that does not normally clump
Artificially introduce required protein
Or artificially turn it on at all times (constituitively
express)

Aggregation now takes place

Therefore is sufficient to induce clumping

Progression to Necessary and

Sufficient
Often you will see this progression
through Biological scientific papers
What is it?
Yes, its there
Yes, its in the right place
Its loss produces this response
Its addition produces this response

Planning Experiments II
Consider statistical methodologies during
planning stages
Look in prior papers for ideas about statistics.
Statistical analysis will generally discern that
likelihood that a result occurred by chance
Consult mentor or statistician for confirmation
Compare 1 treatment and control: t-test
Decide on p (Probability value) p < 0.05 or 0.01

Compare many treatment groups: ANOVA

Many more

Planning Experiments III

Variables
Independent (manipulated)
Dependent (outcome)

# of samples (minimum 2, 3 better)

# repetitions (minimum 2x)

Internal Validity
Cause and Effect- Did the experimental treatment,
and only the experimental treatment, cause the
effect!
Controls (Be Careful!!!)
Prevent additional variables from sneaking into your
experiment
Must control for:
Selection: Anything that makes treatment and control groups
different at beginning (random assignment)
History: What different things may happen between expt. And
control groups between initial treatment and measurement
Maturation: Natural changes in subjects (aging)
Instrumentation: All tests/equipment/reagents must stay
same throughout experiment
Testing: incoming may teach the subject
Mortality: Subjects may leave or die (contamination)
Regression: If initial test scores were high, on average, will
naturally move towards mean

External Validity

The extent to which the findings of the study can be applied,

reproduced, or generalized to another setting or systems.
i.e., techniques to ensure that these groups correspond to
general population

Unrepresentative Sample: Sample members not representative of

general population.
Clear Description of the Treatment or Protocol (replicability)
Hawthorne Effect: Subjects know that they are being studied and it
influences behavior
Novelty Effect: Particularly in humansenjoy experiment, then
possibly dont.
Pretest Sensitization: If the pretest is part of the treatment, it will
obviously affect the results or findings.
History and Treatment Interaction: something else happened which
influenced results, for all participants
Measurement of the Dependent Variable: Treatment and data
collection must be the same every time!

Types of Controls
Experimental
Standards/calibration
Animal/Cell selection/care
Positive controls
See what a positive response looks like and that it can be
obtained. (positively expressing cells)

Negative controls
Shows what a zero response looks like

Treatment controls
All groups treated identically except for indep.
Variable
If two treatments combined, show individual
All time points must be covered
Multiple samples

Keeping it Simple
Your mentor wants to look at the time course
effects of a possible cancer suppressor on
proliferation and mRNA expression in six breast
cancer cell lines. Wants to look at 0, 12, 24, 36,
48, 72, 96h

The beauty of Small

experiments.
Mega Experiment

Ex: 6 types of cells, 7 time points, treated

and untreated (2), in triplicate (3).
6x7x2x3 = 252 plates

Plan Strategically and Break it down

1 cell line, treated and untreated, duplicate,

7 time points = 28. Or postpone
duplicates.

Results from Small

Experiments

Low possibility for confusion

Reasonable workload
Reasonable use of resources
Ability to assess as progress
Easy to interpret
Can change directions on fly

Easy to create discrete graphs

Chasing the Big Problem

For a Publication
Need a Big Picture of what you are
pursuing; tell a good story
Start with correlation
Get additional information
Knock it out/Add it back/Overexpress

Slight modifications, depending on field

How to do Experiment Obtain

Protocol
Instructions for carrying out a particular technique
If followed, will produce desired results
Best if its a proven protocol
Designing your own is time-consuming
Obtain from another investigator
In lab, best

A book of protocols, from web, from kit

Will need fine-tuning for your local circumstances

Methods section from published papers (least reliable)

Review Protocol
Read and do dry run-through
May find logic gaps
May find references to common
procedures you do not know

Personalize Protocol
Rewrite (keeping same steps, etc) to
make more sense to you.
Add notes about own equipment
required

Fully Prepare before Experiment

Buy all required materials
Radioisotopes

Make all solutions and buffers

Reserve machine time if needed

Follow Protocol exactly, first time

through
If it doesnt work, you can assume its
you.
Do again. Not work?
Can get help from person who provided

Modify Protocol

Once protocol is working, modify.

Make notations of changes
Rewrite for next run though
Good if type into computercan record
changes and re-print.

During Experiment
Record which media, temperature or
date-sensitive agents you used
Record any procedural deviation
Dropped something
Delayed
Calibration questions

Put in lab notebook in a timely fashion

Interpreting Results I
Did the Experiment work?
Examine procedural (markers, cells lived)
Examine positive control (yes, antibody
working)
Examine negative control (No, did not have
everything come up positive)

Interpreting Results II
What were the results?
Compared to controls, did you see effect?
Graph your data
How big was effect?
Did effect vary over time?

Interpreting Results III

What does the experiment mean?
Do the results make sense?
Was the result what you expected?
Can you explain spurious results?
What additional controls may you need?

Interpreting Results IV
What do other investigators think?
Talk to lab members
Discuss results with someone versed in
technique
Run through background papers again
Repeat results

Interpreting Results V
Are the results repeatable?
Do experiment again
Add any additional controls

Agh! It didnt work!

1. Check notes.
2. Redo the experiment
3. Focus on individual parts of expt.
1. + and controls

4. Do partial expt. to insure its fixed

5. When youve done alltry again several
times
6. If external protocol, may want to switch

Switching Projects

Never can reproduce data

Project has little support from PI
Direction of project has changed
Not technically possible to do
experiments well
Project too difficult or involved