Essay

Forsch Komplementrmed Klass Naturheilkd 2005;12:159167

DOI: 10.1159/000085412

Published online: May 17, 2005

Building an Evidence House: Challenges and Solutions to

Research in Complementary and Alternative Medicine
Wayne B. Jonas
Samueli Institute for Information Biology, Alexandria, VA, USA

Key Words
Complementary medicine Research methods
Epistemology Standards of research

Schlsselwrter
Komplementrmedizin Forschungsmethoden
Epistemologie Forschungsstandards

Summary
Conventional biomedicine is having a revolution in scientific input from genomics to imaging to information
and systems biology. Biomedicine is also struggling to
find a balance between rigor and relevance such that
public values and health care costs can be properly managed. At the same time complementary and alternative
medicine (CAM) is becoming increasingly popular. Can
rigorous research in CAM be developed? Can it be held
to the same standards of evidence as conventional medicine? Should it be held to those standards? Are there additional standards and better integration strategies for
CAM that are of value to all medicine, complementary or
conventional? In this article, I address some of the major
challenges faced by investigators when conducting research in CAM. These challenges include: quality standards of research; the evolving nature of science; accommodating pluralism; addressing underlying assumptions; and, managing controversial topics in CAM research. These challenges are formidable and will require
that CAM attain a sufficient level of science to move it
out of the margins of health care and a more careful approach to research integration that can keep its focus on
public benefit and the publics health. I suggest a framework of an Evidence House for addressing many of
these challenges.

Zusammenfassung
In der konventionellen Biomedizin findet derzeit eine Revolution des wissenschaftlichen Inputs statt, von der Genomik ber die Bildgebung hin zur Informations- und
Systembiologie. Ausserdem kmpft die Biomedizin um
ein Gleichgewicht zwischen wissenschaftlicher Strenge
und Relevanz, damit ffentliche Mittel und Gesundheitsausgaben sinnvoll investiert werden. Zur gleichen Zeit
gewinnen Methoden der komplementren und alternativen Medizin (CAM) an Beliebtheit. Ist es mglich, wissenschaftlich hochwertige Forschung auch im Bereich
CAM zu entwickeln? Knnen dafr dieselben Evidenzstandards angelegt werden wie fr die konventionelle
Medizin? Sollen dieselben Standards angelegt werden?
Gibt es zustzliche Standards oder bessere Integrationsstrategien fr CAM, die fr die gesamte Medizin, die
komplementre wie die konventionelle, gelten? Im vorliegenden Artikel werden einige Hauptprobleme errtert,
mit denen sich Wissenschaftler im Bereich der CAM-Forschung konfrontiert sehen. Dazu gehren: Qualittsstandards der Forschung, der Entwicklungscharakter von
Wissenschaft, der Umgang mit Pluralismus; der Umgang
mit zugrunde liegenden Annahmen und der Umgang mit
kontroversen Themen in der CAM-Forschung. Diese Probleme sind betrchtlich und verlangen, dass CAM einen
ausreichenden Grad von Wissenschaftlichkeit erreicht,
um nicht lnger ein Schattendasein im Gesundheitswesen zu fhren, und dass ein strengerer Ansatz der Forschungsintegration erreicht wird, der den Nutzen fr die
ffentliche Gesundheit im Auge behlt. Es wird ein Rahmen in Form eines Evidence House vorgeschlagen,
um diesen Problemen zu begegnen.

2005 S. Karger GmbH, Freiburg

Fax +49 761 4 52 07 14
E-mail Information@Karger.de
www.karger.com

Accessible online at:

www.karger.com/fkm

Wayne B. Jonas, MD
Director, Samueli Institute
1700 Diagonal Road, Suite 400
Alexandria, VA 22314, USA
Tel. +1 703 299-4800, Fax 535-6752
E-mail wjonas@siib.org

Introduction
The recurring question for investigators attempting to do high
quality, scientific investigation in the fields of complementary
and alternative medicine (CAM) is: Can research in CAM be
held to the same standards of evidence as conventional medicine? In my opinion, not only can CAM be investigated using
quality standards of evidence, it must use high research standards and it must develop improved integration strategies for
research if it is to advance with both rigor and relevance. It is
better to have no information than faulty and misleading information that subsequently requires we unlearn false facts
(claims that are believed to be true, but are not) before we
can investigate those claims properly.
The repeated painful lesson in conventional medicine is that
the potential for harm inflicted by modern practices is great
when implemented before rigorous evaluation. Thus, we have
learned there is no substitute for appropriately conducted science, and that science is not a perfect discerner of truth. As
William James said a century ago We live forward, but we understand backward.
The term evidence-based medicine (EBM) has become a
synonym for good or scientific; it has the capability of both
supporting and denying the value of CAM practices. EBM is
commonly presented in the context of the hierarchy of evidence that has been so elegantly presented by Sackett and
others [1]. In this arrangement, information from systematic
reviews (SR) of randomized controlled trials is judged the
best evidence, followed by individual randomized controlled
trials (RCT), then by non-randomized trials, observational
studies and case-series, in that order [1]. In such a hierarchy,
all efforts are focused on approximating the type of evidence
at the top of the hierarchy and lower levels are considered inferior. Replicable clinical experiments that isolate additive, casual links between selected aspects of an intervention and selected outcomes of an illness become the gold standard when
this hierarchy model is used.
It is my opinion that this strategy for investigation used in conventional medicine is not sufficient for CAM research. Rather,
research standards that go beyond the evidence hierarchy of
conventional medicine are needed. They must address the diversity and complexity of CAM models, the assumptions of
CAM world views and the politically controversial nature of
many CAM practices while also addressing the paucity of
thoroughly trained investigators needed to probe these questions. My recommendation is that we build an evidence
house for CAM research constructed of high quality rooms
that reflect additional strategic standards to those currently
accepted in conventional medicines evidence hierarchy [2].
Here, I describe some of the challenges to building that house
presented by CAM research. Once built, however, the evidence house strategy offers improved application of research
in medicine, complementary or conventional.
As can be true for conventional medicine, CAM is a diverse

160

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

and heterogeneous area, and specific challenges face each individual category of CAM. The primary challenges for research in CAM fall into 5 categories:
1. quality standards of research
2. the evolving nature of science
3. accommodating pluralism
4. addressing underlying assumptions
5. managing the controversial topics
I will address each of these in turn and discuss how the concept of an Evidence House can assist in approaching these
challenges.
1. Quality Standards of Research
Many CAM practices are delivered by practitioners with variable competence. Some fields have no training standards, thus
making it difficult to assure that competent and well-trained
practitioners are available to implement research or practice.
A variety of training, education and licensing standards exist
[3, 4]. Without assurance of adequate competence in the delivery of CAM interventions, collaboration between practitioners and researchers makes it difficult to implement quality research and apply proven therapies.
The inconsistent quality of CAM products, especially herbs
and dietary supplements, is a major barrier to rigorous research on these products. Recently, one of the most promising
herbal mixtures for prostate cancer, PC-SPES [5, 6], was
found to be adulterated with conventional drugs [7]. The quality and content of many herbal and dietary supplements can
vary considerably from label to label and batch to batch [8].
This makes replication and generalizability unlikely. At the
same time, excessive standardization and the purposeful isolation of what are considered to be the active components, risks
eliminating the synergistic effect of mixtures thought to be so
valuable in herbalism. Similarly, it is rare that homeopathic
studies on ultra-low doses of chemicals include purposeful
verification of purity with atomic absorption spectroscopy or
other methods. Thus, a balance of experimental and observational data are needed to get a full picture of the effects of a
practice or product.
Lack of theory development and testing is another challenge to
research on many CAM practices. The idea of a vital energy
flowing through and between bodies is integral to many CAM
systems such as qi in Chinese medicine. But this is considered
an untenable theory in biomedicine. Discovery of the production of endogenous opioids with acupuncture offered a more
testable theory for Western science and opened up research
approaches to acupuncture and pain. Still, the opioid theory
does not explain many of the effects seen from acupuncture
and more theory development and descriptive information is
needed [9].
There are four theories offered to explain the observed effects
of homeopathy. These are (1) placebo [10], (2) water structuring [11], (3) generalized entanglement [12, 13], and (4) the sil-

Jonas

ica hypothesis. Groups espousing these theories rarely exchange information, and very few investigators are developing
and testing these theories in a systematic matter to determine
which, if any, of these may be true [14].
Arguably, the most complete and accurate theory to explain
the effects of healing intention is called the Data Augmentation Theory [15]. Yet, this theory is practically unknown
among investigators in mind-body medicine, spirituality and
placebo research. More discussion of theory and its ability to
approximate data in CAM systems is needed. Theory development, like qualitative clinical research should be considered a
room of information that lends meaning to CAM practices
and is as essential to medicine as laboratory studies or RCTs.
The meaning and theories given to CAM practices lead to the
comparisons and controls selected in CAM research. The development of proper controls is a significant challenge to advancing research in CAM. Acupuncture and chiropractic have
developed a variety of control comparisons. However, questions such as these remain: Should we use needle placement
in the wrong points, superficial needle placement, non-needle
standard-of-care treatments or retractable needles? How can
adequate therapy and adequate blinding be achieved in manipulative medicine? What is the proper way to adjust for expectation effects in therapies that require training such as
meditation and biofeedback? [16]
Each control selected provides specific and different information from another control. But specific information is often
over-generalized in study conclusions postulating general effects of the therapy. This confuses rather than advances our
understanding of CAM. For example, current data indicates
that acupuncture is effective for low back pain compared to
no treatment, but is no more effective than sham needle control treatments [17, 18]. Proponents of acupuncture use this
information to say that acupuncture is effective for low back
pain, and detractors use the same information to say that
acupuncture is not effective for low back pain. Both are right
and both are misleading. These kind of statements confuse the
public, policy makers and a good percentage of practitioners.
Use of this information has more to do with preferences and
values than science, since both statements are technically correct but not specific enough. Specificity in reporting is important to prevent such confusion and classification of results as
to the goals of different controls described in the discussion
and conclusions.
Lack of basic research is another challenge for CAM. Biomedicine is undergoing a revolution in understanding and application of molecular and cellular biology in medicine [19, 20]. Increasingly in conventional research, new therapies arise from
theory to laboratory testing to phase I, II and III clinical trials.
Most information in CAM comes from clinical observation
and may be left behind this evolving science if high quality
basic research is not pursued. For example, homeopathic Arnica is said to be useful for ischemic stroke. We tested this first
in cellular and then an animal model of stroke [21, 22]. We

found that Arnica had no effect on brain injury from stroke in

24 hours, but that ultra-low doses of the excitatory amino acid
glutamate reduced brain damage by nearly 50% when given
during the reperfusion phase and measured 24 hours later.
These two effects could not have been investigated or discovered using clinical research.
We often hear that CAM should be tested to see if it is effective before mechanism research is done. Such an oversimplified agenda will drive funding for large-scale definitive tests
of popular CAM therapies. However, bypassing the normal,
incremental approach to biomedical science is risky and often
results in failure of the study or the therapy [23]. Laetrile, vitamin C, and perhaps chelation therapy are examples of such
therapies.
The controversy surrounding vitamin C is instructive. Linus
Pauling, one of the most respected scientists of the 20th century, claimed that vitamin C was effective against cancer. He
convinced the government to support large, phase III clinical
studies of oral vitamin C against cancer. None of them showed
vitamin C was effective against cancer [24]. Decades later, laboratory research on absorption of vitamin C demonstrated
that high doses are not absorbed from the gastrointestinal
tract. Intravenous vitamin C, however, can produce very high
serum levels that appear to have oxidant effects (as opposed
to anti-oxidant effects at lower oral doses) and are toxic to
cancer cells [25]. Lack of laboratory and pilot clinical work
with both oral and intravenous vitamin C may have set back a
potentially useful therapy for decades. The research room of
basic science cannot be neglected in CAM without eventually
reaching a dead end.
The above examples illustrate a problem in both complementary and conventional research. Most investigators specialize
in one type of research. Few appreciate the need and relationship of multiple types of research design. In other words, they
do not learn about or apply strategies in research integration.
Laboratory, qualitative, observational, randomized controlled
trials, health services research, and meta-analysis all have their
goals, designs and criteria for quality. Each fills an important
niche in the realms of scientific information. Maintaining a
balanced and logical research portfolio in CAM is necessary
[26]. This is done by matching the publics informational needs
with proper research designs and methods and then applying
quality criteria that are customized to each method. Research
methodologies are tools that we use and not tools that use us
[27]. A balanced research integration strategy is key to the
proper use of those methodologies.

Challenges to Research in CAM

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

2. The Evolving Nature of Science

Medicine and Evidence
The application of rigorous science to medicine is a relatively
new phenomenon. Controversy and new findings about re-

161

search methods emerge regularly. The first RCT was conducted about 50 years ago, yet quality standards for conducting
RCTs have only recently emerged through groups such as
CONSORT and the Cochrane Collaboration [28, 29]. A review of 159 Cochrane reviews of commonly used therapies revealed that about 23% had evidence of positive effectiveness,
21% had insufficient evidence of any effect, 20% had evidence of no effect and 7% had evidence of harm [30]. A summary of studies on the percent of practices from around the
world that are based on RCTs or non-RCT data found that
the RCT evidence base varies widely from 11% in some primary care settings to over 70% at an inpatient hematology
center. The average RCT base for clinical decisions was 37%.
Science is still making its way into conventional medicine and
CAM lags in this regard [31].
Emerging Concepts in Research
250 years ago, blinding was first applied in a research study
when the orthodox scientific community was trying to debunk
mesmerism [32]. From this effort, the importance of blinding
in all clinical research was appreciated and eventually adopted. New concepts continue to emerge in clinical research during its application in CAM areas. For example, a recently published comprehensive overview of the clinical evidence on
homeopathy found evidence from quality SRs that favors
homeopathy over placebo, but an important twist was discovered when looking at this data [33]. The data were reexamined
with a focus on how much of the effect attributed to homeopathy might be the result of a new concept in clinical research called small study bias. Small study bias is the inherent
instability of effect rates in studies with small samples (N <
200) combined with publication bias normally extant in clinical research [34]. The dataset of clinical trials in homeopathy
showed evidence of an effect of small study bias much larger
than previously thought calling the observed effect of homeopathy over placebo into question. If true, this finding has implications not just for bias in homeopathy, but for estimating
power and doing meta-analysis in all clinical research [35]. Recent evidence of unexpected findings from very large studies
on hormone replacement therapy and digitalis support this
idea [36]. In CAM studies where modest effect sizes are expected (herbal studies, for example) the sample sizes required
to prove a therapy effective or ineffective may be woefully
underestimated and impossible to achieve in the current regulatory market [37]. Observational research with proper analysis approaches may be a more practical alternative.
Low Assay Sensitivity
The initial design of the first large scale National Institutes of
Healths (NIH) study on St. Johns Wort (hypericum) for the
treatment of depression contained two control conditions, a
placebo and a proven antidepressant. Only in retrospect was it
apparent that the trial had low assay sensitivity. That is, the
study was not sensitive enough to answer the hypothesis. This

162

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

did not simply mean low power for the primary outcome,
however. Events that occurred during the process of executing
the study itself meant that the postulated question could not
be answered. This was likely due to small study bias factors
and variables that enhance the context or placebo effects in
the study [37]. For example, blinding was not complete because the antidepressant used, sertraline, produced more
physiological effects than placebo or the herb. When physiological effects of treatment are experienced by some patients
on an active treatment in depression studies, effect sizes are
falsely elevated unless an active placebo that mimics these
effects is used [38]. An active placebo was not used in this
study. Other factors which may also have contributed to low
assay sensitivity in the study include the excellent individual
attention given to patients, the enthusiasm for hypericum that
existed in the public at the time the study was initiated, the increasing rates of placebo response in depression over time in
general [39] and the marked variation in both placebo and active drug response rates across cultures [37]. For example,
placebo response rates of gastric and duodenal ulcers are high
in Germany, but low in Belgium [40]. The difference is so
great that studies were unable to confirm the effectiveness of
H2 blockers on ulcers in Germany, but easily did so in Belgium. Context effects are a great challenge to research in all of
medicine, but may be more so in traditional therapies that are
inherently part of and accepted in a culture.
While low assay sensitivity is a problem in conventional medicine it is more problematic in complementary medicine. For
example, because herbs are derived directly from plants and
are not single chemical agents like drugs, their effect is more
likely due to multiple active ingredients each at low concentrations. Finding optimal doses and mechanisms of action
under these circumstances is difficult and even if found, the individual ingredients are not likely to work well when separated for standardization as required in research studies. Thus,
testing these compounds may be very expensive, requiring
more and larger studies even than conventional drugs [37].

Observational vs Randomized Trials

The value of observational compared to controlled trial data is
a challenging area for CAM research where the importance of
observational and outcomes research is often espoused. Two
therapies may have very different specific and non-specific effects. Depending on the proportion of these effects, a RCT
may favor one therapy while the greatest benefit is actually
derived from the other [41]. The relationship of B-carotene
intake to cardiovascular disease mortality is one such example. Observational studies show clear and statistically significant benefit. RCTs show the opposite. Which is right? It appears that B-carotene does make a difference, but supplementing with it does not [42]. Should large observational studies be believed over small RCTs given the instability of small

Jonas

TS

Goals

IN

Regulators

ST

TE

TE

ST

Resear ch Methods

SE

IN

Audience Pr efer ences

EF

FE

Public
Health
Health
Reviews
Meta-analysis Proof General Services
Clinical
Use
Research
VI
V
Researchers
Practitioners
Randomized
Epidemiology
Controlled
Attribution
Association Outcomes
Trials
Basic
III
IV
Scientists
Patients
Qualitative
Laboratory
Mechanism
Meaning II
Case Reports
I

Fig. 1. The Evidence House. Permission to

use from BMJ Publishing Group, West J Med
2001;175:80.

VALIDITY TESTING
VALUES

study results just discussed? The conventional evidence hierarchy would choose RCT data. However, a series of articles
published in the New England Journal of Medicine showed
that many quality observational studies found no significant
difference in outcome compared to randomized trials [43].
The evidence house gives each design its due for their respective purposes. Clearly, all domains of research in the evidence
house have important places in science. A major challenge to
CAM is accommodating the evolving nature of biomedical research by clearly aligning research goals with appropriate
methods and not trying to bend the evolution of CAM to predefined methodological conceptions.

3. Accommodating Pluralism
The evidence hierarchy is too simplistic for much of medicine
and for the complex interventions in many CAM practices. As
every clinician knows, patients recover for complex and interacting reasons, many of which are not additive and cannot be
isolated in controlled environments. The best evidence under
these circumstances may be observational data from clinical
practice that can estimate the probability of a patients recovery in a realistic context [41]. In addition, patients illnesses
are complex physical, psychological and social experiences
that cannot be reduced to single, objective measures [44]. In
some cases, the most valuable information for a clinical decision is a highly subjective judgment about life quality. This
personal experience of an illness can only be captured with
qualitative research, not questionnaires or blood tests [45].
The best evidence under these circumstances is the meaning
a patient has of their illness and recovery. At other times the
best evidence comes from laboratory studies. The discovery
that St. Johns Wort can accelerate the metabolism of immunosuppressive drugs, for example, is the most crucial infor-

Challenges to Research in CAM

mation for the management of patients on such medications

taking herbs [46, 47]. Controlled trials may not discover such
interactions. By arranging types of evidence into a hierarchy,
we obscure the fact that sometimes the best evidence is not
causal, not objective, not additive and not clinical. Pluralism
has a role in research in CAM. A recent publication from the
Hastings Center contains an excellent collection of thoughtful
articles on this subject [48]. The need for pluralism is a challenge to both CAM and conventional research, where uniformity is sometimes mistaken for quality.
The Evidence House
The main research domains involved in the scientific investigation of either complementary or conventional medicine
can be arranged in a semi-hierarchical manner forming functional rooms in an Evidence House (fig. 1). On the left side
of the Evidence House are types of information that seek out
causal attributions and mechanisms of action. Laboratory research (room I) forms a foundation of understanding for
mechanisms and causal links on which controlled clinical research can build. In the case of homeopathy or prayer, for
example, RCT data seem to point us in an illogical direction
that can only be resolved with basic science research. RCTs
(room III) and SRs of RCTs (room V) attempt to verify suspected causal links of specific interventions in clinical settings. However, if we confine ourselves to the left side of the
Evidence House, we will never obtain information about the
relevance of medicine for patients from qualitative research
(room II) or what happens in the actual world of clinical
practice from observational data (room IV) or the generalizability of an intervention when placed in our health care delivery system from health services research (room VI). The
rooms on the right side of the Evidence House provide information about relevance and utility of practices, whether
proven or unproven.

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

163

Evidence and Ethics

Whether we construct an evidence hierarchy or an evidence
house also has ethical implications for medicine. This is
because different groups prefer different types of evidence.
Regulatory authorities and those interested in the approval of
new treatments are most interested in RCT (room III) or SR
(room V) data rather than observational research. Health care
practitioners frequently want to know the probability of effects in populations in clinical practice (room VI). Those paying for care are generally more interested in health services research (room VI). Patients are intensely interested in stories
and detailed descriptions of patients similar to themselves
(room II). Rationalists want to know how things work and
some will accept only practices with conceptual plausibility
(room I). If resources are disproportionately invested into one
room of the house (say for the regulatory goals of rooms III
and V) to the neglect of others, science will not obtain the
evidence needed for full public participation in clinical decisions and so violate the ethical principles of autonomy and
beneficence. A livable house should not have an elaborate
kitchen and no bathroom. Each has different functions and all
need to be high quality.
Validity Criteria
For clinical research, one must add additional validity criteria
to those normally used in research on conventional medicine
to assess the CAM practice and model being tested. The evaluation of research quality in CAM would use the same approach as that in conventional areas, with additional items relevant for specific CAM areas. The challenge is to evaluate
clinical research in CAM by internal, external and model validity standards. The validity of RCTs, for example, uses a set
of quality criteria that test internal validity or the likelihood
that observed effects are biased. These criteria emphasize the
importance of allocation concealment, randomization method,
blinding, proper statistical methods, attention to dropouts and
other factors.
When evaluating the applicability of information from clinical
research, including RCTs and observational trials, quality criteria for external validity or the likelihood that the observed
effects will occur in varied situations are used. Numerous
quality rating systems have been developed for the evaluation of clinical research. Notably, the CONSORT group has
produced a widely adopted set of reporting guidelines for
RCTs [29]. Other guidelines have been published for reporting meta-analyses, observational trials and diagnostic tests
[49]. These guidelines cover basic quality items for all clinical
research, including those on CAM.
Model Validity
In addition to issues of internal and external validity, CAM
research requires special attention to model validity or the
likelihood that the research has adequately addressed the
unique diagnostic taxonomy, treatment process, and therapeu-

164

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

tic context of the system under investigation [27]. Since many

CAM systems originate outside of Western medicine, research
requires adequate expertise in the application of the CAM
system under investigation. Many CAM practices are investigated in populations for which the practice is traditional and
integral to the culture. Given the marked variation in response
produced even by standardized treatments in different cultures, the results produced in one culture may not always
translate to another [10].
Finally, individual expectation and the nature of the information given to patients (including the informed consent procedure) can impact outcomes significantly [50]. The quality dimension of model validity in the design and evaluation of
CAM research evaluates these factors. Table 1 lists the main
quality criteria for internal, external and model validity in clinical research on CAM. These criteria have been applied in the
evaluation of several different types of CAM clinical research
[27].

4. Addressing Underlying Assumptions

Assumptions in Western Science
Another major challenge to CAM research comes from the
underlying assumptions of many CAM practices. In conventional research, basic assumptions about the operating rules
of reality are not questioned. Western science assumes that
physical forces predominate, objective measurement is achievable and preferred and separation of causal elements without
significant modification of the phenomenon being measured is
the ideal strategy [51]. Reductionistic approaches allow us to
develop the best therapies with specific causation that does
not cross system levels. These are considered unchangeable
principles of nature. By refusing to question these assumptions, the rules of inquiry can be fixed into an evidence hierarchy that holds internal validity criteria and specific causation
as the gold standard for research [52].
Alternative Medicine Assumptions
Many traditional and CAM healing systems, however, do not
share these assumptions. Here are a few examples:
Consciousness. The primary basis for Ayurvedic medicine is
universal or primordial consciousness. Universal consciousness assumes fundamental, non-material connectivity between
all observables that result in mental, emotional, spiritual and
physical change. All therapy, including dietary and herbal
treatments, focus on achieving deeper contact with this consciousness for healing [53].
Energy. Traditional Chinese Medicine is based on the principle of qi or vital energy, a concept not accepted in conventional Western science. Health and healing occur when a balanced flow of qi is achieved in the person [54].
Synergy. Traditional herbal medicine holds as a fundamental
principle that synergy between herbal mixtures is superior to

Jonas

Table 1. The likelihood of validity evaluation guidelines [27]

Dimension

Main criteria

Internal Validity:
How likely is it that the effects
reported are due to the independent
variable (the treatment)?

Randomization: Was subject assignment to treatment groups done randomly and in a concealed manner?
Baseline Comparability: Were gender, age, and prognostic factors balanced?
Change of Intervention: Was there loss to followup, contamination, poor compliance?
Blinding: Did the patients, practitioners, evaluators, analysts know who got the treatment?
Outcomes: Was the objectivity, reliability, and sensitivity of the outcome assessed?
Analysis: Was the number treated large? Were p values significant? Were there multiple outcomes
measured and analyzed?

External Validity:
How likely is it that the observed
effects would occur outside the study
and in different settings?

Generalizability: Was there a range of patients seen in practice or were there multiple or narrow inclusions
and exclusions? Was the study done at several sites with similar results?
Reproducibility: Was the treatment clear? Were confidence intervals reported? Was the treatment
transferable to other practitioners?
Clinical Significance: Was the effect size big enough to make a difference? Is the condition in need of this
type of treatment? Were any preferences determined? Was adherence good?
Therapeutic Interference: Was there flexibility in varying the treatment? Was feedback on the outcomes
available? Is the treatment feasible in your setting?
Outcomes: Were the outcomes clinically relevant? Were the outcomes checked for importance with the
patients? Were any important outcomes missing?

Model Validity:
How likely is it that the study
accurately reflects the system under
investigation?

Representativeness/Accuracy: Were the therapists well trained and experienced? Was the treatment
strategy adequate? Was the treatment clearly described?
Informed Consent: Was the informed consent comprehensive? Was it effective did patients understand it?
Did it generate expectations different from practice?
Methodology Matching: Were the goals of the study clear and limited? Did the investigators select the
correct research method to achieve the goals?
Model Congruity: Were the patients classified, was the treatment determined and were the outcomes
assessed according to the system of the practice being assessed?
Context: How well was the intervention adapted to the culture, family, meaning of the patient?

Reporting Quality:
How likely is it that the report
accurately reflects what was found
in the study? How clear and accurate
is the information presented?

Comprehensiveness: Can you address the above criteria?

Clarity: Could you reproduce this study?
Conclusions: Were the conclusions and reporting format (e.g., relative vs absolute improvement rates,
strength of wording) appropriate to the data collected?

isolated herbs or their components. Isolation of plant components is foreign to most herbal use around the world and is felt
to reduce the healing capacity of the approach [55].
Holism. Homeopathy rests on the idea of stimulating the entire organism with low-dose and precisely targeted signals.
Homeopathy targets the complete set of symptoms of a person physical, mental and general and looks for precise patterns in the healing response of the person as a whole. The
idea of holism in homeopathy is not the same as being comprehensive, the most common interpretation of holism in conventional medicine [56].
Beauty. Much of Native American Medicine rests on the assumption that illness reflects loss of spiritual harmony. Healing occurs with the re-establishment of right-relationship with
ones environment and community. This is called in some
tribes the restoration of beauty and requires a community,
not an individual treatment approach [57].
Spirituality. Many traditional healing systems assume that the
use of symbolic interventions can directly influence a persons

health and affect cure. The recent increase in research on spirituality, prayer and healing intention are examples of attempts
at scientific evaluation of these concepts that some argue go
beyond the boundaries of scientific inquiry [58].

Challenges to Research in CAM

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

Limits of Scientific Inquiry

In Western science we reject many of the assumptions just listed outright, or we reject that they can be investigated with science. Certainly, it is axiomatic that science requires measurement, either directly or indirectly of the associations of interest in research. To what extent, however, can the effects of
symbolism, synergy and acausality be tested scientifically?
There are areas that we can investigate now that in the past
were considered beyond the scientific microscope. Will this
happen again in areas considered pseudoscientific and metaphysical? Can we set prior probabilities on the limits of scientific inquiry? One of the greatest challenges to research in
CAM is to learn how and when and when not to bring these
assumptions under the scrutiny of high quality science [59].

165

5. Managing the Controversial Topics

Distribution of CAM Research Resources
New things are usually resisted by the status quo. The Office
of Alternative Medicine (OAM) and the National Center for
Complementary and Alternative Medicine (NCCAM) were
born in the midst of political and social controversy and
against considerable resistance from leadership at the NIH
and in the scientific community. There are two ways to avoid
controversy in CAM. One is to ignore the more controversial
areas, and do research only in those topics that are close to
conventional medicine. The second is to set up systems that
explicitly manage the controversy as the science gets done.
The usual approach in science is the first.
A CRISP search and analysis of CAM practices funded by the
NCCAM shows that over 80% of research money goes to
areas where practices are largely already accepted and formulated in conventional medicine. Examples are single component nutritional supplements, accepted theoretical models of
behavioral medicine, chemical extracts of single herbal components, and so forth. Another 12% are in areas that have become generally accepted as needing research because we have
found physical or chemical explanations for them that fit
Western assumptions. Examples include acupuncture, manipulation, biofeedback and imagery. Less that 2% of NCCAM
funding goes to areas considered core concepts in most CAM
practices. There are taboo areas in CAM such as spirituality,
energy, non-locality, bioelectromagnetics and herbal combinations that are not addressed in publicly funded research. Ironically, these very concepts are often the ones used to sell CAM
to the public and formulate therapeutic plans by CAM professionals.
Conflict Resolution in CAM Research
An attempt by NIH in 1995 to study a cancer therapy called
anti-neoplastons on brain cancer provides important data on
this topic. After an elaborate protocol and start of enrollment,
the study failed to be completed because of controversy between the investigators and the founder of the therapy. An investigation launched from a social conflict resolution perspective clearly showed that failure of the study was because of inadequate management of the social controversy around the
therapy, not the scientific and methodological issues involved
in the trial [60]. Subsequently, a meeting was held with national experts in conflict management and models for the man-

agement of social processes in controversial areas of CAM

science were developed. One of these models was recently
successfully tested on a controversial claim in digital biology
for the Department of Defense [61]. In controversial areas, it
should be assumed that conflict will arise and management
systems for these controversies should be arranged before the
conduct of the research [62].

Conclusions
I have not discussed all the challenges to research in CAM,
only those that I perceive as the most difficult for developing
innovative, relevant and high quality CAM information. The
challenges to CAM research are many and complex. Most
claims in CAM arise out of faith and beliefs, not data. The
goal of CAM research is to unlock those mysteries by being
more explicit and using high quality science. In addition, new
and balanced strategies for the integration of research methods are needed. With appropriate methodology we can discern what works, what doesnt, and why. In addition, research
should tell us what is useful and meaningful in patients lives.
Science is in many ways a sacred act, based ultimately on our
faith in reason and discernment. However, the more one
delves deeply into science, the more life remains a mystery.
As Daniel Boorstin, author of The Discoverers said after
completing his massive work on the lives of great explorers,
The greatest obstacle to discovery is not ignorance, but the illusion of knowledge [63]. With humility we can use the powerful methods of science we have, develop better methods in
the future and still stand before the mystery and the miracle of
life [64].

Acknowledgements
The author would like to thank Ronald A. Chez, M.D., for his review and
comments on this manuscript and Cindy Crawford, B.A., for her assistance in manuscript preparation. This research was funded by the Samueli
Institute for Information Biology.

Competing Interests
None declared.

References
1 Sackett DL, Rosenberg WM, Gray JA, Haynes RB,
Richardson WS: Evidence based medicine: What it
is and what it isnt. BMJ 1996;312(7023):7172.
2 Jonas W: The evidence house. West J Med 2001;
175:7980.
3 Leake R, Broderick J: Current licensure for
acupuncture in the United States. Altern Ther
Health Med 1999;5(4):9496.

166

4 Mishra L: Status of ayurvedic education and licensing in the United States. Altern Ther Health Med
2001;7(1):20.
5 Pfeifer B, Pirani J, Hamann S, Klippel K: PC-SPES,
a dietary supplement for the treatment of hormone-refractory prostate cancer. Br J Urol Int
2000;85:481485.

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

6 Small E, Frohlich M, Bok R: Prospective trial of

the herbal supplement PC-SPES in patients with
progressive prostate cancer. J Clin Oncol 2000;18
(21):35953603.
7 White J: PC-SPES: A lesson for future dietary supplement research. J Natl Cancer Inst 2002;94(17):
12611263.

Jonas

8 American Botanical Council: The nature of Ginseng: Traditional use, modern research, and the
question of dosage. Herbal Gram 2002;54:3451.
9 Han J: Opioid and antiopioid peptides: A model of
Yin-Yang balance in acupuncture mechanisms of
pain modulation; in Hammerschlag R, Stux G,
(eds): Clinical Acupuncture: Scientific basis. Berlin,
Springer, 2001, pp 5169.
10 Moerman D, Jonas W: Deconstructing the placebo
effect and finding the meaning response. Ann Intern Med 2002;136:471476.
11 Torres J: On the physical basis of succussion.
Homeopathy 2002;91(4):221224.
12 Milgrom L: Patient-practitioner-remedy (PPR)
entanglement. Part 3: Refining the quantum
metaphor for homeopathy. Homeopathy 2003;92
(3):152160.
13 Walach H: Entanglement model of homeopathy as
an example of generalized entanglement predicted
by weak quantum theory. Forsch Komplementaermed Klass Naturheilkd 2003;10:192200.
14 Walach H, Schneider R: Proceedings: Future directions and current issues of research in homeopathy.
Freiburg, Samueli Institute, 2002.
15 May E: Challenges for healing and intentionality
research: Causation and information; in Jonas W,
Crawford C (eds): Healing, Intention and Energy
Medicine: Science, Research Methods and Clinical
Implications. London, Churchill Livingston, 2003,
pp 283292.
16 Integration of behavioral and relaxation approaches into the treatment of chronic pain and insomnia.
NIH Technology Assessment Panel on Integration
of Behavioral and Relaxation Approaches into the
Treatment of Chronic Pain and Insomnia. JAMA
1996;276(4):313318.
17 Schneider C, Jonas W: Are alternative treatments
effective? Issues and methods involved in measuring effectiveness of alternative treatments. Subtle
Energies Energy Med 1995;5(1):6992.
18 Tulder: Acupuncture for low back pain. The
Cochrane Review 1999(2).
19 Jasny B, Kennedy D: The human genome. Science
2001;291(5507):1153.
20 Dennis C, Gallagher R, Campbell P: Everyones
genome. Nature 2001;409:813.
21 Jonas W, Lin Y, Williams A, Tortella F, Tuma R:
Treatment of experimental stroke with low-dose
glutamate and homeopathic Arnica montana. Perfusion 1999;12:452462.
22 Marotta D, Marini A, Banaudha K, Maharaj S,
Jonas W: Nonlinear effects of glutamate and KCl
on glutamate toxicity in cultured rat cerebellar neurons. Int J Neurosci 2003;113:491502.
23 Jonas W: Policy, the public and priorities in alternative medicine research. Ann Am Acad Polit Soc Sci
2002;583:2943.
24 Henson DE, Block G, Levine M: Ascorbic acid: biologic functions and relation to cancer. J Natl Cancer Inst 1991;83(8):547550.
25 Nagy B, Mucsi I, Molnar J, Varga A, Thurzo L:
Chemosensitizing effect of vitamin C in combination with 5-fluorouracil in vitro. In Vivo 2003;17
(3):289292.
26 Lewith G, Walach H, Jonas W: Balanced research
strategies for complementary and alternative medicine; in Lewith G, Jonas W, Walach H (eds): Clinical Research in Complementary Therapies: Principles, Problems and Solutions. London, Churchill
Livingston, 2002, pp 328.

Challenges to Research in CAM

27 Jonas W, Linde K: Conducting and evaluating clinical research on complementary and alternative
medicine; in Gallin J (ed): Principles and Practice
of Clinical Research. San Diego, Academic Press,
2002, pp 401426.
28 Jadad A: Randomized Controlled Trials: A Users
Guide. London, BMJ Books, 1998.
29 Moher D, Schulz K, Altman D: The Consort statement: revised recommendations for improving the
quality of reports of parallel-group randomized trials. Ann Intern Med 2001;134(8):651662.
30 Ezzo J, Bausell B, Moerman D, Berman B, Hadhazy V: Reviewing the reviews: How strong is the
evidence? How clear are the conclusions? Int J
Technol Assess Health Care 2001;17:457466.
31 Pelletier K: Conventional and integrative medicine:
evidence based? Sorting fact from fiction. FACT
2003;8(1):36.
32 Kaptchuk TJ: Intentional ignorance: the history of
blind assessment and placebo controls in medicine.
Bull Hist Med 1998;72:389433.
33 Jonas W, Kaptchuk T, Linde K: Critical overview of
homeopathy. Ann Intern Med 2003;138:393399.
34 Egger M, Juni P, Holenstein F, Sterne J: Are the
Clinical Effects of Homeopathy Bias Effects? Bristol, UK, Department of Social Medicine, University of Bristol, 2001.
35 Vickers A: Underpowering in randomized trials reporting a sample size calculation. J Clin Epidemiol
2003;56:717720.
36 Rossouw J, Anderson G, Prentice R: Risks and
benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the
Womens Health Initiative randomized controlled
trial. JAMA 2002;288(3):321333.
37 Jonas W: St. Johns Wort and depression. JAMA
2002;288(4):446.
38 Kirsch I, Spirstein G: Listening to Prozac but hearing placebo: a meta-analysis of antidepressant medication. Prev Treat 1998;1:0002a.
39 Walsh B, Seidman S, Sysko R, Gould M: Placebo
response in studies of major depression: variable,
substantial, and growing. JAMA 2002;287(14):
18401847.
40 Moerman D: Cultural variations in the placebo
effect: ulcers, anxiety and blood pressure. Med
Anthropol Q 2000;14:5172.
41 Walach H, Jonas W, Lewith G: The role of outcomes research in evaluating complementary and
alternative medicine, in Lewith G, Jonas W, Walach
H (eds): Clinical Research in Complementary
Therapies. London, Churchill Livingston, 2002, pp
2945.
42 Egger M, Smith GD: Misleading meta-analysis.
BMJ 1995;310:752754.
43 Benson K, Hartz A: A comparison of observational
studies and randomized controlled trials. New Engl
J Med 2000;342(25):18781886.
44 Cassel E: The Nature of Suffering, ed 2. Oxford
University Press, 2004.
45 Aldridge D: A qualitative research perspective on
healing; in Jonas W, Crawford C (eds): Healing, Intention and Energy Medicine: Science, Research
Methods and Clinical Implications. London,
Churchill Livingston, 2003, pp 225238.
46 Ruschitzka F, Meier P, Turina M: Acute heart transplant rejection due to St. Johns Wort. Lancet
2000;355:548549.

47 Piscitelli S, Burstein A, Chaitt D: Indinavir concentrations and St. Johns Wort. Lancet 2000;355:
547548.
48 Callahan D (ed): The Role of Complementary and
Alternative Medicine: Accommodating Pluralism.
Washington, DC, Georgetown University Press,
2002.
49 Stroup D, Berlin J, Morton S: Meta-analysis of observational studies in epidemiology: a proposal for
reporting. JAMA 2000;283(15):20082012.
50 Bergmann J, Chassany O, Gandiol J, Deblois P,
Kanis J: A randomised clinical trial of the effect
of informed consent on the analgesic activity of
placebo and naproxen in cancer pain. Clin Trials
Metaanal 1994;29:4147.
51 Clouser K, Hufford D: Nonorthodox healing systems and their knowledge claims. J Med Philosophy 1993;18(2):101106.
52 Jonas W, Levin J: Introduction: Models of medicine
and healing; in Jonas W, Levin J (eds): Essentials
of Complementary and Alternative Medicine.
Philadelphia, Lippincott Williams and Wilkins,
1999, pp 116.
53 Lad D: Ayurvedic medicine; in Jonas W, Levin J
(eds): Essentials of Complementary and Alternative Medicine. Philadelphia, Lippincott Williams
and Wilkins, 1999, pp 200215.
54 Lao L: Traditional Chinese Medicine; in Jonas W,
Levin J (eds): Essentials of Complementary and
Alternative Medicine. Philadelphia, Lippincott
Williams and Wilkins, 1999, pp 216232.
55 De Smet P: The safety of herbal products; in Jonas
W, Levin J (eds): Essentials of Complementary and
Alternative Medicine. Philadelphia, Lippincott
Williams and Wilkins, 1999, pp 108147.
56 Jonas W, Ernst E: The safety of homeopathy; in
Jonas W, Levin J (eds): Essentials of Complementary and Alternative Medicine. Philadelphia, Lippincott Williams and Wilkins, 1999, pp 167171.
57 Cohen K: Native American medicine; in Jonas W,
Levin J (eds): Essentials of Complementary and
Alternative Medicine. Philadelphia, Lippincott
Williams and Wilkins, 1999, pp 233251.
58 Benor D: Spiritual healing; in Jonas W, Levin J
(eds): Essentials of Complementary and Alternative Medicine. Philadelphia, Lippincott Williams
and Wilkins, 1999, pp 369382.
59 Callahan D: Introduction; in Callahan D (ed): The
Role of Complementary and Alternative Medicine.
Washington, DC, Georgetown University Press,
2002, pp viix.
60 Hammer M: Burzynski antineoplaston case study:
Conflict issues and recommendations. Bethesda,
MD, Office of Alternative Medicine Report, NIH,
1996.
61 Ives J: Evaluating unusual claims and devices using
a team approach: A case study. Subtle Energies Energy Med 2003;13(1):3959.
62 Hufford D: Evaluating complementary and alternative medicine: the limits of science and of scientists. J Law Med Ethics 2003;31(2):198212.
63 Boorstin DJ: The Discoverers. New York, Random
House, 1983.
64 Berry W: Life is a Miracle: An Essay against Modern Superstition. Washington, DC, Counterpoint,
2000.

Forsch Komplementrmed Klass Naturheilkd

2005;12:159167

167