[Downloaded free from http://www.ehpjournal.com on Thursday, October 4, 2018, IP: 10.232.74.

23]

Original Article

How Common Are Experimental Designs in Medical Education?

Findings from a Bibliometric Analysis of Recent Dissertations
and Theses
Kenneth D. Royal, Jason C. B. Rinaldo1
Department of Clinical Sciences, North Carolina State University, Raleigh, NC, 1Office of Assessment, Rawls College of Business, Texas Tech University, Lubbock, TX, USA

Abstract
Background: There has been a recent influx of researchers in the field of medical education coming from medical and health science
backgrounds. Researchers from health fields often misunderstand that studies involving experimental designs are relatively rare throughout
educational research. Experts in education research note that experimental designs largely are incompatible with educational studies due to
various contextual, legal, and ethical issues. Purpose: We sought to investigate the frequency with which experimental designs have been
utilized in recent medical education dissertations and theses. Methods: A bibliometric analysis of dissertations and theses completed in the
field of medical education between 2011 and 2016. Results: Fewer than 10% of doctoral dissertations and master’s theses involved some type
of experimental design. Only 6.12% of all dissertation and master’s projects involved randomized experiments. Conclusions: Randomized
experiments occur only slightly more frequently in medical education than other educational fields.

Keywords: Bibliometrics, dissertations and theses, education research, experiments, medical education, research design, research quality

Introduction hegemonistic views about research designs typically are

In recent years, there has been a significant influx in the number
of individuals conducting research in the field of medical
education. The vast majority of these researchers come from
medical and health science backgrounds and have little to no
formal training in education research. These researchers tend
to be accustomed to experimental designs, as these research
designs are frequently used in drug trials and health‑related
research. This design is considered to be the most rigorous,
least biased, and of the highest quality.[1] Naturally, many
of these individuals have called for an increased use of
experimental designs and randomized controlled trials in
medical education studies. Unfortunately, educational studies
often do not allow for the same randomization as can be found
in drug or treatment conditions.
Concomitantly, numerous medical education researchers
have questioned the quality of medical education research
due to what they perceive to be a lack of rigorous research
designs. Many of these individuals immediately dismiss any
work that does not involve an experimental design. These
Access this article online
Quick Response Code:
Website: is given and the new creations are licensed under the identical terms.
www.ehpjournal.com
perceived as insulting by researchers with formal training in
the field of education. Such attacks on nonexperimental designs
are reminiscent of the “Paradigm Wars” in which researchers
in the social and behavioral sciences debated the superiority
of quantitative versus qualitative methods for decades before
eventually agreeing that each type of inquiry has its own merits
and appropriateness for use.
Unquestionably, there is a significant difference between
research in the medical and health sciences and research in the
education sciences.[2] For instance, what is entirely appropriate
in medicine may be entirely inappropriate, unethical,
impractical, and/or illegal in education (and vice versa).[2]
Further, the nature of the variables studied between the sciences
and the methods used to handle those challenges often are
Address for correspondence: Dr. Kenneth D. Royal,
Department of Clinical Sciences, North Carolina State University,
1060 William Moore Dr. Raleigh, NC 27607, USA.
E-mail: kdroyal2@ncsu.edu
This is an open access journal, and articles are distributed under the terms of the Creative
Commons Attribution‑NonCommercial‑ShareAlike 4.0 License, which allows others to
remix, tweak, and build upon the work non‑commercially, as long as appropriate credit
For reprints contact: reprints@medknow.com

DOI: How to cite this article: Royal KD, Rinaldo JC. How common are
10.4103/EHP.EHP_5_18 experimental designs in medical education? Findings from a bibliometric
analysis of recent dissertations and theses. Educ Health Prof 2018;1:28-30.

28 © 2018 Education in the Health Professions | Published by Wolters Kluwer - Medknow

[Downloaded free from http://www.ehpjournal.com on Thursday, October 4, 2018, IP: 10.232.74.23]

Royal and Rinaldo: Experimental designs in medical education

unique to one’s discipline. In fact, experimental designs theses. Search parameters included the most recent 5 years,
largely are incompatible with most educational contexts. with a specific date range of August 1, 2011 to August 1, 2016.
To illustrate this point, the National Science Foundation’s Each dissertation and thesis was reviewed to determine whether
2015–2016 Survey of Earned Doctorates reported there are an experimental or quasi‑experimental design was employed.
40 subdisciplines within the field of education.[3] In a review
of all dissertations completed across these many education Results
subdisciplines, research has noted that  <1% involved
randomized experiments.[4] Of the 147 dissertations and theses identified, 14  (9.52%)
utilized either an experimental or quasi‑experimental
Interestingly, the disconnect between medical/health science design. Of these 14 studies, 11 (78.57%) were dissertations
researchers and education researchers whose training typically and 3  (21.42%) were theses. With respect to experimental
is rooted in the social and behavioral sciences appears to come design type, 9  (64.29%) involved randomized experiments,
down to a single, but significant, difference in perspective. That and 5  (35.71%) were quasi‑experimental in nature. Of
is, medical and health science researchers typically begin their the 9 randomized experimental studies, 7  (77.78%) were
research with an experimental design in mind and then craft a dissertations, and 2 (22.22%) were theses.
research question that can be answered within an experimental
context. Education and other social and behavioral researchers, With respect to degree type, of the 14 experimental
on the other hand, begin with a particular research question or quasi‑experimental designs used, 6  (42.9%) were
in mind and then identify an appropriate research design to Doctorates in Education (EdD), 4 (28.6%) were Doctorates of
best answer the question(s). These fundamental differences Philosophy (PhD), 1 (7.14%) was a Doctorate of Public Health,
in research norms based on one’s disciplinary training tends and 3 (21.42%) were Masters of Science degrees. Finally, with
to create confusion about what constitutes “good” research in respect to the colleges granting these degrees, 7 (50.00%) were
interdisciplinary fields such as medical education where the from colleges of education, whereas 7 (50.00%) came from
biomedical and social/behavioral sciences converge. other colleges (e.g., medicine, public health, engineering, and
arts and sciences). A full breakdown of results is presented in
Any ongoing confusion about what constitutes “good” Table 1.
medical education research is harmful for the medical
education community and need resolution. For example,
many medical education researchers with formal training in Discussion
education research have expressed frustration with reviewers Overall, 9.52% of doctoral dissertations and master’s theses in
and editors that insinuate any study that does not utilize some the subject area of medical education over the 5‑year period
form of experimental design is “weak,” “flawed,” or otherwise involved some type of experimental design. Only 6.12% of
incapable of yielding valid results.[2] On the other hand, many these projects involved randomized experiments, a rate that
researchers with little or no formal training in education is only slightly higher than a typical dissertation or thesis in
research are frustrated by article submissions that do not meet the field of education.[4] These results support the notion that
their expectations for a high‑quality research design.[2] Thus, research in medical education has far more in common with
some resolution, or dialog toward resolution, is necessary.
To that end, the purpose of this study was to investigate the Table 1: Medical education dissertations and theses
research designs utilized in recent dissertations and theses in utilizing experimental or quasi‑experimental designs
the subject area of medical education. The rationale is that
Study Design Dissertation Degree College
dissertations and theses provide unique insights about research or thesis type
in the field of medical education. For example, dissertations and
Emmert[5] E D EdD Education
theses may originate from a variety of academic disciplines, Gucev[6] E D EdD Education
tend to be rather exhaustive works often exceeding a hundred Miller Juve[7] E D EdD Education
pages, typically involve rigorous research designs, usually are Best[8] Q D PhD Engineering
led by a committee chair/mentor who is a content expert in Hess[9] Q D DPH Public health
the subject area, and almost always are evaluated by a team of McCoy[10] E D EdD Education
3–5 (or more) faculty experts, many of whom also are subject Buck[11] Q D PhD Psychology
matter experts and specialists in research methodology. Cheung[12] E T MS Medicine
Wee[13] E D PhD Medicine
Methods Latham[14] E D EdD Education
Carney[15] Q T MS Medicine
A bibliometric analysis was conducted using ProQuest Ford[16] Q D EdD Education
Dissertations/Theses, the largest repository for dissertations Varthis[17] E D PhD Education
and theses in the world with  >3.8 million works. A  search Hill[18] E T MS Medicine
was conducted using the keyword “medical education” in the E: Experimental design, Q: Quasi‑experimental design, D: Dissertation,
subject field. This resulted in a total of 147 dissertations and T: Thesis, DPH: Doctorate of public health

Education in the Health Professions  ¦  Volume 1 ¦ Issue 1 ¦ January-June 2018 29

[Downloaded free from http://www.ehpjournal.com on Thursday, October 4, 2018, IP: 10.232.74.23]
Royal and Rinaldo: Experimental designs in medical education
the methodological conventions of education and the social
and behavioral sciences than those of the medical and health
sciences.
Of course, some may argue dissertations and theses are
subpar in quality when compared to peer‑reviewed, published
literature. Although there certainly will be some variability in
the quality of graduate projects (much like there is variability
in quality of peer‑reviewed papers), there also is much
reason to believe these graduate projects may actually be
of superior quality to many peer‑reviewed papers published
in the medical education literature (e.g., incredibly rigorous
requirements, expert faculty oversight and mentoring,
and subject matter expert evaluators). Thus, it would be a
mistake to dismiss the quality of dissertations and theses
because they have yet to undergo peer‑review by an academic
journal, especially given many of the long‑standing concerns
relating to academic peer‑review  (e.g., reviewer shortage
and questionable quality reviews).[19,20] In fact, many fine
peer‑reviewed publications stem directly from a researcher’s
graduate project.
Conclusions
Medical education research greatly benefits from
interdisciplinary perspectives, which includes methodological
practice and research design. Certainly, researchers are
encouraged to pursue experimental designs when ethical,
legal, and operational constraints permit. However, results
of this study further illustrate that conducting randomized
experiments in education is an ambitious, but generally
unattainable goal. Thus, it should serve as a reminder that it
is important to reassess assumptions about research designs
when conducting research in the field of medical education.
Financial support and sponsorship
Nil.
Conflicts of interest
Both authors are editors for Education in the Health
Professions. Thus, peer‑review was initiated and performed
by an independent editor associated with the journal.
References
1. Zlowodzki M, Jonsson A, Bhandari M. Common pitfalls in the conduct
of clinical research. Med Princ Pract 2006;15:1‑8.
2. Royal KD, Rinaldo J. There’s education, and then there’s education in
medicine. J Adv Med Educ Prof 2016;4:150‑4.
30 Education in the Health Professions  ¦  Volume 1  ¦  Issue 1  ¦  January-June 2018
3. Survey of Earned Doctorates. National Science Foundation. Available
from: http://www.nsf.gov/statistics/srvydoctorates.  [Last retrieved on
2017 Sep 15].
4. Nave  B, Miech  EJ, Mosteller  F. The role of field trials in evaluating
school practices: A  rare design. In: Stufflebeam  DL, Madaus  GF,
Kellaghan  T, editors. Evaluation Models: Viewpoints on Educational
and Human Services Evaluation. Boston: Kluwer Academic Publishers;
2000. p. 145‑61.
5. Emmert  MC. Pilot Test of an Innovative Interprofessional Education
Assessment Strategy [dissertation]. USA: University of California – Los
Angeles; 2011.
6. Gucev GV. Cognitive Task Analysis for Instruction in Single‑Injection
Ultrasound Guided‑Regional Anesthesia [dissertation]. USA: University
of Southern California; 2012.
7. Miller Juve  AK. Reflective Practice and Readiness for Self‑directed
Learning in Anesthesiology Residents Training in the United
States [dissertation]. USA: Portland State University; 2012.
8. Best  AC. Online Academic Support Peer Groups for Medical
Undergraduates  [dissertation]. USA: Nova Southeastern University;
2012.
9. Hess  JS. Residency Education in Preparing Adolescents and
Young Adults for Transition to Adult Care: A  Mixed Methods Pilot
Study [dissertation]. USA: University of South Florida; 2014.
10. McCoy L. Virtual Patient Simulations for Medical Education: Increasing
Clinical Reasoning Skills through Deliberate Practice  [dissertation].
USA: Arizona State University; 2014.
11. Buck  KS. Impact of Educational Messages on Patient Acceptance of
Male Medical Students in OB‑GYN Encounters  [dissertation]. USA:
East Carolina University; 2014.
12. Cheung  JJ. Preparing for Simulation‑Based Education and Training
Through Web‑Based Learning: The Role of Observational Practice and
Educational Networking [thesis]. Canada: University of Toronto; 2014.
13. Wee AG. Promoting Oral Cancer Examination to Primary Care Providers
at Medical Clinics in Nebraska  [dissertation]. USA: University of
Nebraska Medical Center; 2014.
14. Latham  PS. The Impact of Instructional Design in a Case‑Based,
Computer‑Assisted Instruction Module on Learning Liver Pathology
in a Medical School Pathology Course  [dissertation]. USA: George
Washington University; 2015.
15. Carney BL. Improving Safe Opioid Prescribing among Internal Medicine
Residents using an Observed Structured Clinical Exam  (OSCE)
Education Tool [thesis]. USA: Boston University; 2015.
16. Ford CL. An Evaluation Study of the Biomedical Careers Program at
Robert Wood Johnson Medical School  [dissertation]. USA: Rutgers
University; 2015.
17. Varthis  S. Students’ Perceptions of Blended Learning and its
Effectiveness as a Part of Second Year Dental Curriculum [dissertation].
Columbia University; 2016.
18. Hill JA. Examining the Time Course of Memory Retention for Medical
Gross Anatomy in First Year Medical Students  [thesis]. USA: Boston
University; 2016.
19. Albert AY, Gow JL, Cobra A, Vines TH. Is it becoming harder to secure
reviewers for peer review? A test with data from five ecology journals.
Res Integr Peer Rev 2016;1:14.
20. Hames  I. Peer Review and Manuscript Management in Scientific
Journals: Guidelines for Good Practice. United  Kingdom: Wiley,
Chicester; 2007.