Sunteți pe pagina 1din 17

Contemporary Educational Psychology 39 (2014) 1228

Contents lists available at ScienceDirect

Contemporary Educational Psychology

journal homepage:

Modeling and measuring epistemic cognition: A qualitative

Jeffrey A. Greene , Seung B. Yu
University of North Carolina at Chapel Hill, CB#3500, Chapel Hill, NC 27599, United States

a r t i c l e i n f o a b s t r a c t

Article history: Since Perry rst proposed that students beliefs about knowledge and knowing were an important aspect
Available online 22 October 2013 of learning, there has been a proliferation of models of epistemic cognition, and empirical studies of how
epistemic cognition relates to learning. Unfortunately, the dominant means of measuring epistemic cog-
Keywords: nition, self-report instruments, have numerous psychometric problems. These problems prompted us to
History return to interview methods used by Perry and other seminal researchers, to investigate the degree to
Social studies which current epistemic cognition models aligned with novices and experts cognition. Using an explor-
atory, multiple case qualitative design, we interviewed middle school students and university professors
Epistemic cognition
from two domains, biology and history. We found numerous ways in which the current conceptualiza-
tions and measures of beliefs about knowledge and knowing may need to be altered. Our recommenda-
tions range from the revision of item wordings to a complete rethinking of the very idea of domain-
specicity in epistemic cognition research.
2013 Elsevier Inc. All rights reserved.

1. A qualitative investigation of expert and novice conceptions reliability and factorial validity of inferences from scores from
of epistemic cognition in biology and history self-report instruments of EC cast doubt upon the ndings of any
study that uses those instruments, including most of the ones that
Research into epistemic cognition (EC; Chinn, Buckland, & have used quantitative analyses to relate students beliefs about
Samarapungavan, 2011; Greene, Azevedo, & Torney-Purta, 2008; knowledge and knowing to academic outcomes and covariates
Hofer & Pintrich, 1997, 2002; Kitchener, 2002; Sandoval & am, (Schraw & Olafson, 2008). Critically, if an instrument is not mea-
2010) has increased exponentially since Perrys (1968/1999) sem- suring what researchers believe it is measuring (i.e., if psychomet-
inal phenomenological interview research on students beliefs ric analyses result in poor validity evidence), then relations
about knowledge and knowing (Hofer & Bendixen, 2012). between scores on that instrument and any other outcome, such
Researchers have asserted that differences in students EC inu- as academic performance, become moot.
ence a multitude of educational outcomes including academic It is important to note that some researchers have avoided the
achievement (Buehl & Alexander, 2005; Greene, Muis, & Pieschl, psychometric problems of self-report EC instruments by using
2010; Hofer, 2000; Schommer, Crouse, & Rhodes, 1992), self-regu- qualitative data collection and analysis techniques (e.g., Feucht &
lated learning (Brten & Stromso, 2005; Muis, 2007, 2008), web Bendixen, 2010), or conducting quantitative investigations that
searching (Tu, Shih, & Tsai, 2008) and scientic understanding have used measures other than self-report such as think-aloud
(Conley, Pintrich, Vekiri, & Harrison, 2004), among many others. protocols (e.g., Mason, Boldrin, & Ariasi, 2010) interviews (e.g.,
Unfortunately, concern about the psychometric adequacy of the Sandoval & am, 2010) and observation (e.g., Rosenberg, Hammer,
most frequently used measures of EC (i.e., self-report instruments) & Phelan, 2006). While these studies are exempt from the psycho-
is another common thread running through the relatively brief metric critiques described here, it is nonetheless the case that the
history of this area of research (e.g., Buehl, 2008; Clarebout, Elen, majority of work in the eld of EC over the last ten years has in-
Luyten, & Bamps, 2001; DeBacker, Crowson, Beesley, Thoma, & volved quantitative analysis of scores from seemingly problematic
Hestevold, 2008; Greene et al., 2008; Hofer & Pintrich, 1997; Wood self-report instruments (DeBacker et al., 2008). Therefore, the eld
& Kardash, 2002). The consistently poor evidence regarding the of EC research nds itself at a difcult crossroads: the number of
studies showing relations among epistemic cognition, learning
phenomena, and academic outcomes continues to grow, but poor
Corresponding author. Address: Learning Sciences and Psychological Studies
Program, School of Education, University of North Carolina at Chapel Hill, 113
psychometric evidence for the adequacy of measures used in a
Peabody Hall, CB#3500, Chapel Hill, NC 27599, United States. Fax: +1 919 843 2614. majority of these studies casts a dark cloud of doubt over their
E-mail address: (J.A. Greene). ndings.

0361-476X/$ - see front matter 2013 Elsevier Inc. All rights reserved.
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 13

While there are many possible explanations for the current underlying construct, they had in common representations of
state of affairs in EC research, three plausible reasons are that (a) meaning making through hierarchical sequences of stage develop-
instrument developers need to rene the foci or wording of the ment. Kuhns work is used extensively today, and is a good repre-
self-report items used to measure EC; (b) instrument developers sentative of the developmental class of EC models. Her model
need to consider different ways of measuring EC (e.g., discontinue suggested movement from positions of realist, to absolutist, then
using Likert-type items, capture student behaviors; Greene et al., multiplist, and nally to evaluativist, describing a progression in
2010; Hofer, 2004; Hofer & Sinatra, 2010; Muis, Bendixen, & Hae- the engagement of knowledge that was roughly analogous to Perrys
rle, 2006); or (c) the poor psychometric qualities of instruments model. The transitions from dualism to multiplism and from abso-
derived from conceptual models of EC indicate problems with lutist to multiplist in the two models both represented a change
those models themselves. Researchers continue to explore the rst from purely objective to purely subjective views of knowledge. Rel-
two possible explanations (e.g., Greene, 2009; Greene et al., 2010; ativism and evaluativist positions were characterized by a reconcil-
Mason et al., 2010; Muis, 2008), and we applaud that work. This iation of the two extremes. These developmental models suggested,
study was conceived to investigate the third option. The purpose among other things, that knowledge becomes increasingly contex-
of this study was to explore novices and experts EC, and assess tualized with progress along the developmental continuum.
the ways in which current models did and did not align with our The rst multi-dimensional model of EC was put forth by
participants EC. To do this, we employed interview methods sim- Schommer (1990). It expanded the research eld by advocating
ilar to those utilized by the progenitors of EC research, including for a system of ve independent but related dimensions: xed abil-
Perry (1968), King and Kitchener (1994), Belenky, Clinchy, Gold- ity, quick learning, simple knowledge, certain knowledge, and
berger, and Tarule (1986), and Baxter Magolda (1992). We felt that omniscient authority (i.e., source of knowledge). Some researchers
the open-ended, inductive, and exploratory nature of qualitative (DeBacker et al., 2008; Hofer & Pintrich, 1997; Muis et al., 2006)
research was necessary to investigate the conceptual foundations have claimed that the rst two dimensions extend beyond the
and assumptions upon which researchers have built their quantita- scope of epistemic concerns. The other three constructs cohere
tive instruments. Such an investigation precludes generalization with the dimensions proposed by Hofer and Pintrich (1997) who
and proclamation of trends, but does allow for the uncovering of categorized simple knowledge and certain knowledge as nature
evidence that might otherwise be missed in quantitative analyses of knowledge beliefs. Source of knowledge was categorized as a
where assessments are necessarily constructed from conceptual nature of knowing belief. To this category, they added justica-
models, and the assumptions underlying those models (Lewis & tion as a fourth aspect of the construct.
Grimes, 1999). We sought to explore, at a deep, foundational level, Simple knowledge refers to the view of knowledge as discrete
the models used to conceptualize EC, with the hope of nding po- and unrelated facts. Learners along one end of the simple knowl-
tential explanations for the poor psychometric results that plague edge dimension view facts as unconnected with each other, while
the eld. learners closer to the other end view facts as highly connected and
related. The certain knowledge dimension characterizes the views
2. Literature review of learners who consider knowledge as unchanging to be nave
while those who consider knowledge to be tentative and evolv-
Within the paradigm of qualitative research, the literature re- ing (Hofer & Pintrich, 1997, p. 120) as sophisticated. At the nave
view plays a less signicant role in the design of empirical work than end of the source of knowledge dimension, knowledge is viewed as
it does in the quantitative paradigm (Creswell, 2008). While quali- originating outside of the self. Learners more advanced along this
tative researchers are certainly informed as to the literature in their dimension believe themselves to be progenitors of knowledge. Jus-
respective elds, their focus on participant meaning making, open- tication for knowing refers to the process of evaluating sources
ended inquiry, and allowing themes to emerge from the data often and means of knowledge including observation, authority, rules,
includes an intentional de-emphasis of prior theory, so that their and what feels right (Hofer, 2004, p. 46) at the time.
participants voices can be heard clearly. Given our wish to question Hammer and Elby (2002, 2003) championed the situated re-
the underlying conceptual foundations of EC research, we used prior sources model and criticized the developmental and multidimen-
literature to contextualize and motivate our work, but not circum- sional models inclusion of domain-general, static beliefs.
scribe it. Therefore, here we provide an overarching summary of According to Hammer and Elby, learners epistemologies are actu-
EC models, the critiques that problematize those models and inspire ally ne-grained cognitive resources that exist at a level of specic-
our work, and a discussion of how we used that literature to frame ity beyond that of beliefs or theories. Learners display different
our exploratory multiple-case study. Per the qualitative research epistemic behavior based upon the context; in some situations a
paradigm, this brief literature review will be supplemented in the particular set of cognitive resources may be activated, whereas in
results section, where we will incorporate additional literature to a different context a very different set could be activated to inu-
broaden, deepen, and contextualize our ndings (Mertens, 2010). ence learning. This view of EC differs greatly from many other mod-
els of EC particularly as it regards degree situatedness. Recently,
2.1. Models of epistemic cognition Chinn, Buckland, and Samarapungavan (2011) have proposed a dra-
matic expansion of the dimensions of EC to include components as
While models of epistemic cognition are numerous and diverse, diverse as epistemic aims (i.e., peoples goals for inquiry such as
three of the most prominent kinds are: developmental, multi- knowledge or truth), epistemic values (i.e., peoples beliefs about
dimensional, and situated resources-based. Developmental models the worth of different epistemic aims), and epistemic virtues (i.e.,
began with the work of Perry (1968/1999) who, after extensive dispositions that are benecial in the pursuit of epistemic aims,
interviews with numerous male Harvard students, proposed that such as open-mindedness). They have also advocated for a consider-
college students progress along a continuum of positions that he ation of the situated nature of these EC components.
named dualism, multiplism, relativism, and commitment with rela-
tivism. Following Perrys work, other researchers produced their 2.2. Recent critiques of models of epistemic cognition
own models of EC, each building their models off of data from inter-
views with students (e.g., Baxter Magolda, 1992; Belenky et al., 2.2.1. The nature of knowledge
1986; King & Kitchener, 1994; Kuhn, Cheney, & Weinstock, 2000). A majority of the dimensional models of EC include two specic
While all of these models differed according to the framing of the beliefs about the nature of knowledge: the degree to which it is
14 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

simple and the degree to which it is certain. Murphy and col- composed of substantively identical items. For example, Buehl,
leagues (Murphy, 2005; Murphy & Alexander, 2004; Murphy, Alex- Alexander, and Murphy (2002) followed the advice of two experts
ander, Greene, & Edwards, 2007) argued that these dimensions in educational assessment evaluation when constructing domain-
were not epistemological, in the philosophical sense of the term. specic versions of their EC measure:
Rather, they argued that the dimensions were better aligned with
These individuals recommended keeping the core wording
the philosophical area of ontology (i.e., the study of reality and
of items the same across domains to enhance comparability
the basic categories of being). Greene, Torney-Purta, & Azevedo
and to reduce error variance. Thus, all items were rewritten
(2010) expanded upon this idea, arguing that Chis (1992) concep-
so that each item could apply to either domain with the
tualization of ontological categories and attributes was a more
change of a single word. For example, the item, Mathematics
appropriate way of thinking about learners beliefs about the nat-
relates to day to day life, was rewritten as History relates
ure of knowledge than the terms simple and certain. In short,
to day to day life. This process resulted in the elimination of
Chi argued that individuals possess an oftentimes non-conscious
several items that could not easily be applied to both domains.
ontology of the various different categories of knowledge within
(p. 423)
an academic domain. Unique attributes differentiate these catego-
ries. For example, within a science ontology there may be different Greene et al. (2010) followed a similar procedure for their
categories for material substances and processes. The former instrument, focusing on the domains of mathematics and history.
may have attributes such as has weight and occupies space Other researchers (Hofer, 2000; Schommer & Walker, 1995) have
whereas the latter would have a different set of attributes such used the exact same items to measure EC in mathematics, the so-
as leads to change (cf., Chi, 1997). According to Greene and col- cial sciences, and the hard sciences, by asking participants to think
leagues, EC researchers who are interested in the degree to which about one academic area when lling out the instrument the rst
individuals see knowledge as simple are really inquiring about time, and to think about a different academic area when lling
the complexity and accuracy of individuals ontologies; whereas out the same set of items a second time.
investigations into certain knowledge are more accurately de- While these researchers have found interesting intraindivid-
scribed as focusing upon the degree to which individuals are will- ual item response patterns that sometimes differed across
ing to amend those categories and attributes in the light of new domains, the underlying presumption in all of these studies
evidence. Chinn, Buckland, and Samarapungavan (2011), on the was that the underlying dimensions (e.g., simplicity of knowl-
other hand, argued that these dimensions are ontological only for edge, certainty of knowledge) were equally relevant in each
realists, and that in all other cases are more appropriately de- domain, and that there were no dimensions that were unique
scribed as epistemological. to a particular academic domain. In contrast, we wondered
We were not interested in either proving or disproving whether whether the actual dimensions of EC differed across
the nature of knowledge beliefs were best described as epistemo- academic domains. For example, an individuals views on the
logical or ontological. Rather, what struck us about these argu- simplicity of knowledge may be very predictive of learning
ments was that it was possible that individuals beliefs about the in a science domain, but less so in history. Or, it may be that
nature of knowledge might be far more complex than what could simplicity means different things in different domains.
be captured using just two dimensions: simple and certain (Chi, Perhaps additional, discipline-unique dimensions are needed.
1997; Chinn et al., 2011; Greene et al., 2008; Hammer & Elby, Therefore, one of the things we wished to explore in our case
2002, 2003). Therefore, we focused a portion of our interview pro- study was the nature of domain-specicity in EC, and whether
tocol upon how participants thought about domain knowledge, prior conceptualizations (e.g., identical dimensions across
and endeavored to explore whether additional dimensions beyond domains) were adequately capturing individuals views about
simple and certain were needed to describe their EC. To facili- knowledge and knowing across various academic domains. This
tate this discussion, we included a gure from Chis (1997) chapter required that we conduct separate interviews with individuals
that graphically depicted an example ontology with categories and in two different academic contexts, which for this study were
attributes. biology and history.

2.2.2. Domain-specicity 2.2.3. Authorities as sources of knowledge

Perrys (1968/1999) initial model of EC has been described as Numerous models of EC include the idea that nave individu-
domain-general, although he did allow for the possibility of small als justify knowledge claims based upon what they learn from
differences in beliefs across academic areas. Many researchers authority gures, whereas sophisticated individuals rely on their
working at the end of the 20th century posited similarly domain- own means of justication such as rationality (Hofer & Pintrich,
general models (e.g., King & Kitchener, 1994; Schommer, 1990). 1997; King & Kitchener, 1994; Schommer, 1990). Yet, within philo-
Within the last ten years, many EC researchers have begun advo- sophical epistemology there is a strong tradition of using authority
cating for examining multiple levels of EC, from domain-general gures as justication (cf., Murphy et al., 2007). Chinn, Buckland,
to domain-specic or even task-specic beliefs (Buehl & Alexander, and Samarapungavan (2011) pointed out that philosophers often
2005; Greene et al., 2008, 2010; Hofer, 2006; Muis et al., 2006; refer to the idea of testimony, rather than authority, to make clear
Sandoval, 2009). At this point in the elds development, it is fair that experts may be acceptable sources of justication provided
to say that there has been a shift in the overall tenor of EC research that those experts have been properly vetted. Sandoval and am
toward multiple-level conceptualizations, to the degree that Hofer (2010) showed that elementary school children differentiated be-
(2006) has called the domain-generality versus domain-specicity tween various sources of justication, privileging rsthand experi-
debate pass (p. 67). Hammer and Elbys (2002, 2003) work rep- ences with collecting data above arguments from authority.
resents the most explicit move away from a domain-general EC, Research ndings such as these caused us to wonder whether EC
where epistemological resources are activated based upon context, researchers positioning of justication by authority solely as a
and may vary from task to task, and setting to setting, within the nave belief (e.g., Schommer, 1990) needed to be reexamined.
same discipline or domain. We also wondered whether individuals differed in the degree to
What we found interesting about the elds move toward the which they privileged different kinds of authorities, such as do-
inclusion of domain-specic EC was that many researchers created main experts, professors, teachers, and textbooks. As such, we in-
multiple, domain-specic versions of their instruments that were cluded questions in our interview protocol related to
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 15

participants beliefs about authority, and the justication of knowl- so student participants would be familiar enough with the content
edge claims using testimony. to respond. Second, we chose disciplines that have distinct per-
spectives on knowledge and knowing so that we could examine
2.2.4. When development begins the ways in which EC across disciplines differed in degree, kind,
Many of the models of EC reviewed by Hofer and Pintrich or both. Biology has traditionally been considered a hard (Biglan,
(1997) were based upon ndings that suggested that development 1973) or well-structured (Frederiksen, 1984) discipline, whereas
beyond the initial positions of EC does not begin until learners history has traditionally been considered a soft or ill-struc-
reach college age. Chandler, Hallett, and Sokol (2002) presented tured discipline. Through our choices of participants and disci-
both a theoretical and an empirical argument against the idea that plines, we hoped to uncover evidence that could be used to
EC remains xed until college, citing evidence of adolescents problematize current models of EC, and perhaps lead to benecial
engaging in relativist thinking (Boyes & Chandler, 1992; Chandler, changes to those models, from minor tweaks to major renements.
Boyes, & Ball, 1990). Likewise, Burr and Hofer (2002) have shown Ultimately, it was our hope that these rened conceptual models of
that very young children display elements of EC, and that the pres- EC could lead to development of quantitative instruments with
ence of EC seemed to be related to childrens development of a the- strong psychometric qualities that would better examine how EC
ory of mind (Flavell, 2004). Kuhn, Cheney, and Weinstock (2000), relates to learning phenomena and outcomes.
as well as Mason and Scirica (2006), have also presented evidence
that students younger than college age are capable of engaging in
advanced EC. Sandoval (2005), among others, has argued for more 3. Methods
research into the development of EC. Therefore, these conceptual
and empirical arguments informed our choice of sample. Rather 3.1. Participants
than expecting college students to serve as sources of relatively
undeveloped EC, we worked with middle school students, who For this exploratory multiple-case study, intense case sampling
we felt were old enough to have experienced academic domain dif- (Creswell, 2008; Yin, 2006) was utilized. The cases for this study,
ferences in school, but young enough to be likely to display rela- middle-school students and university faculty members, were
tively undeveloped EC. solicited to exhibit a wide variety of EC so that the phenomenon
could be better understood than if our sample included partici-
2.3. Framing our study as an exploratory case study pants that were likely to share similar EC. Multiple cases were re-
quired to do this type of sampling. In addition, we sought
Each of the criticisms we described, as well as the concerns participants from two different academic domains, biology and
about the psychometric qualities of EC self-report instruments, history, so that we could examine domain differences in EC. We
led us to the conclusion that a thorough reexamination of EC, conducted 16 interviews: two with university faculty in biology,
and the conceptual models of these phenomena, was warranted. two with university faculty in history, six with middle-school stu-
These critiques informed how we constructed our investigation dents of biology, and six with middle-school students of history.
of epistemic cognition, but we felt it important to move beyond Our sample included four faculty members, two from a biology
these critiques as well, and open up our investigation to every as- department, and two from a history department, at a large, public
pect of these models. The desire to query the foundations of episte- university in the Southeast. The faculty members included two wo-
mic cognition research, the need to be able to move beyond a priori men and two men. We sought faculty members with little to no
assumptions about EC in our questioning, and the aforementioned background in education or philosophy, with the hope of capturing
difculties with self-report instruments, all led us to employ inter- a set of relatively unbiased responses to questions about epistemic
view methodologies similar to those used by the scholars who ini- cognition.
tiated the eld of epistemic cognition (i.e., Baxter Magolda, 1992; We recruited middle school students in the same county.
Belenky, Clinchy, Goldberger, & Tarule, 1986; King & Kitchener, Approximately 70% of the county population identied as White,
1994; Perry, 1968). The qualitative, multiple-case study interview non-Hispanic, 12% identied as Black, 8% identied as Hispanic
approach (Madill & Gough, 2008; Yin, 2006) of our investigation or Latino, and 7% identied as Asian. Median household income
enabled us to access student cognition in ways that mirrored the was approximately $56,000, with roughly 17% of the population
exploratory, model-building methods used by these early EC reporting as below the poverty level. The school itself was located
researchers (Hofer & Bendixen, 2012). Indeed, exploratory case near a mid-sized town in this predominantly rural county, and was
study designs are well-suited for questioning and possibly altering one of three middle-schools serving that county. Interview data
established theories (e.g., Hofer, 2004). In our study, we used our were collected in the second, Spring semester of the school year.
knowledge of current models of EC not as a prescription for under- The researchers, upon entering the school and after working within
standing our participants beliefs about the nature of knowledge it frequently over the course of several weeks, saw it as a welcom-
and knowing, but rather as a permeable boundary that shaped ing, well-run environment where students seemed comfortable yet
our initial questioning. We were particularly interested in partici- focused. Teachers appeared to have little difculty managing their
pant responses that extended beyond this boundary, or called into classrooms, and there were no disruptions or other egregious inci-
question the legitimacy of the boundary itself. dents at any point during data collection. Interviews took place in
To ensure that we captured EC in all its forms, we explored both an extra classroom in the school, which was being used to store
university faculty members and middle-school students beliefs SmartBoards that had been recently purchased for each classroom
about the nature of knowledge and knowing, and worked with par- in the building.
ticipants from two different academic disciplines, biology and his- Middle school student participants were recruited through one
tory. To examine the ways in which EC varied as a function of prior history and one biology teacher. Each teacher nominated 10 typi-
knowledge and experience, or not, we chose to work with two cally-functioning students who they believed would be able to
groups of participants who greatly differed in their level of experi- articulate their beliefs about knowledge and knowing in the do-
ence, or presumed expertise, in the domain (i.e., middle-school stu- main. We asked teachers to nominate a group of students whose
dents and university faculty members). Our choice of disciplines aggregate demographic prole would approximate that of the
(i.e., biology and history) was made based upon several factors. school as a whole. There was no coordination between teachers
First, both disciplines are included in most middle school curricula, regarding student nomination. These students were from multiple
16 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

sections of each teachers course, but these sections did not inten- domains. We also had domain-specic questions, which could
tionally differ in terms of average student performance (i.e., they have lead to other probes at a similar level of specicity. Each mid-
were not tracked). The teachers happened to teach different stu- dle-school student participant was interviewed regarding a single
dents, thus there was no overlap in terms of student nominations discipline, based upon the class from which the participant was re-
across classes. Given our interest in understanding students EC cruited. Therefore, participants recruited from a biology class were
as it was displayed through our dialog with them, we did not re- interviewed about biology only, and the same for participants from
quest that teachers select for, or provide us, information regarding history classes.
academic performance or any other demographic information. For The interview protocols for the faculty members (see Appendix
each teacher, six of the 10 students returned signed consent forms B) included questions regarding what participants believed stu-
and agreed to participate in the study. dents needed to know to be knowledgeable in the domain, their
The 8th grade participants consisted of eight girls and four boys beliefs about the nature of knowledge in the domain, how knowl-
(i.e., 5 girls from the history course, 3 girls from the biology edge in the domain benetted society, and how they went about
course). As mentioned previously, we chose 8th grade students be- justifying knowledge claims. We also had domain-specic ques-
cause we felt they were far enough along in their schooling to have tions for faculty members. At the conclusion of each interview
experienced disciplinary differences, but not so experienced that we asked the participant whether there were any other important
explicit instruction or discussion regarding epistemic cognition points that we had not covered.
within the discipline would have been likely. Discussions with
their teachers indicated that while the teachers included disciplin- 3.3. Procedure
ary practices (e.g., experimentation and replication in biology;
explanatory power in history) in their instruction, students were Middle-school student participants were recruited in person.
not exposed to terminology from, or models of, epistemic cognition The rst author went to the teachers classes, explained the study,
research. and provided students with parental consent forms to take home.
Students who returned signed consent forms were also given a
3.2. Protocols personal assent form to read and sign. Participants were inter-
viewed during their class time, in a separate room. Either the rst
We used semi-structured interviews to investigate our partici- author or a trained graduate student interviewed each participant.
pants EC. These interviews occurred over a three-month period Participants were told they could end the study at any time, for any
to accommodate the teachers schedules and school holidays. The reason; none chose to do so.
interviews themselves ranged from 18 to 27 min for the students Faculty participants were recruited via email from a large public
and 49 to 53 min for the professors. Each interview was conducted university in the southeastern United States. The rst author re-
individually with either one or two researchers and the participant. viewed academic department websites for a list of potential partic-
Semi-structured interview protocols were used to minimize inter- ipants. Faculty members who consented to participate were
viewer effects (Patton, 2002) but researchers were encouraged to interviewed either in their campus ofce or in the rst authors of-
deviate from the protocol as needed, and probe participant re- ce. Although none chose to do so, all participants were informed
sponses for relevant information (see Appendices A and B). Each that they could end the study at any time, for any reason.
participant was asked each of the questions listed in the appendi- Interviews were audio-taped so that the researchers could focus
ces, however, interviewers were encouraged to ask follow-up ques- on the participants responses, rather than having to take notes.
tions (i.e., probes) to clarify or expand participants responses These audio-tapes also ensured that participant data were cap-
when appropriate (Patton, 2002). What makes the interview tured accurately for later analysis. During all of the interviews,
semi-structured was the ability to alter the wording of the ques- researchers were free to take reective eld notes regarding their
tions, and add probes, to ensure that participants EC was surfaced observations. There was also time in between interviews and after
and explored as thoroughly as possible. There were times where a each interview session to write down additional eld notes. These
participants response to one question or probe resulted in a suf- eld notes were included as data sources during analysis.
ciently clear answer to another question, in which case the Identifying information for each participant, aside from infor-
researchers may have asked an amended version of the other ques- mation about their domain (i.e., history or biology) and experience
tion that was appropriate given the dialog. Our goal was to explore level (i.e., middle-school or faculty), were securely stored sepa-
participants EC, therefore while we ensured that each participant rately from the rest of the data. All interviews were transcribed
was asked about each underlying construct of interest, we followed by a trained graduate student. The rst and second author listened
qualitative interview techniques and allowed for probes into par- to each audio-tape while reviewing the transcriptions, to ensure
ticularly interesting or important statements that arose from the accuracy.
interview (Patton, 2002).
The interview protocols differed by domain of knowledge (i.e., 3.4. Data analysis and authenticity
biology versus history) and level of experience within domain of
knowledge (i.e., middle-school student versus faculty member). Interview data were analyzed independently by the rst and
However, the protocols were designed to address the same core is- second authors. Data analysis occurred during data collection,
sues. The middle-school protocol (see Appendix A) included ques- informing probes for future interviews, and after data collection
tions about the participants impressions of their classes, their had ended. The authors independently hand-sorted the data, read-
beliefs about what their teachers wanted them to learn, their be- ing and rereading the transcriptions to identify relevant participant
liefs about the nature of knowledge in the domain, and how they responses and eld notes. Relevant text segments were categorized
went about justifying knowledge claims. However, the actual ques- using in vivo coding or low-inference indicators, with the goal of
tions included terminology that we thought would be more famil- keeping the codes as representative of the participants meaning
iar for middle-school students. Though we have questioned data as possible. As the authors engaged in coding the data, they kept
collection procedures that merely interchange domain names (Sec- separate lists of important ndings from the data, as well as what
tion 2.2.2), we replicated a similar protocol in this study because the data did not elucidate (e.g., see Section 4.2). The researchers
we considered it important for helping to unearth possible differ- also endeavored to be reexive about their analysis, and let the
ences between responses from students referring to different data speak rather than forcing their own ideas onto the data
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 17

(Patton, 2002). Segments of data were grouped under common 4.1. The nature of knowledge
codes, and then codes were combined into overarching themes
(Creswell, 2008). The data collection portion of the study ended 4.1.1. Simple knowledge is not so simple
when the authors felt that the data were saturated and that One of the rst questions we asked each of our professor partic-
redundancy had been reached. ipants was what they believed to be the important kinds of knowl-
The rst and second authors met frequently to discuss codes edge in their domain:
and themes derived from the data. As themes were developed,
Biology Professor #1: The dates, the names of things being sort
each author went back to the data to clarify, deepen, and question
of a given, acknowledging that that is a kind of knowledge, a
codes and themes. This iterative process allowed both researchers
sort of naming kind of knowledge. But when I think of what
to question their understanding of EC in general, as well as the data
makes biology biology, you have to know the connections
collected from the participants. As interviews progressed, the
between those things. So thats to me the most important kind
authors began seeing redundancy. Within the context of the initial
of knowledge.
set of interviews, we felt that saturation was reached and conse-
quent data analysis did not result in a sense that additional data Our history professor participants shared similar views. Declar-
were required. The themes reported here were agreed upon by ative knowledge (Schraw, 2006) was mentioned: I dont think
both authors as relevant and accurate depictions of the partici- anybody who takes a U.S. history course should be able to pass a
pants EC. However, it is important to note that there were numer- course if they dont know what the 14th amendment is. So thats
ous interindividual differences among participants, as expected. that core bedrock knowledge (History Professor #1). However,
The authors engaged in triangulation among sources and analysis both history professors stressed that while factual knowledge
in an attempt to ensure the authenticity of their inferences and was necessary, it was also the least interesting or important kind
data presentation. The second author, having not collected the data of knowledge in terms of historical expertise:
personally, acted as a peer debriefer for the rst author. Therefore,
we utilized numerous techniques for establishing the authenticity History Professor #1: And so, with that example of the 14th
of our data representation and analyses, including semi-structured amendment, I try to tell them, I want you to know what the
interviewing with probes, in vivo coding, triangulation of data, and 14th amendment is about, but I want you to know whats
peer debrieng. The authors also disciplined their subjectivity by important about the 14th amendment. So, just saying when it
critically examining their own positionality throughout the data was written, when it was adopted, and whats in the 14th
analysis process (Creswell, 2008; Patton, 2002). amendment, thats three quarters of the way. Then youve gotta
come with why it matters.

3.5. Positionality History Professor #2: History students, let me say, undergradu-
ate students tend to think of history very much as the facts, they
The rst author is a Caucasian male faculty member at a state consider it a list of information, of very uninteresting sorts of
university who has studied EC. He has published both conceptual information that they need to memorize. . . and historians tend
and empirical articles, as well as book chapters, on EC, and has to think. . . facts are kind of a low priority.
been involved in numerous conference presentations on the topic.
These professors clearly ascribed epistemic value, albeit a rela-
As such, he was careful to be reective about his own beliefs about
tively low value, to declarative knowledge. Yet, each professor
EC, and to endeavor to hear participants voices, rather than impos-
mentioned procedural knowledge as another, perhaps more impor-
ing his own ideas about what he thought they were saying. The
tant kind of knowledge in the domain. The rst biology professor
rst author also thought it was important that he triangulate his
referred to skill knowledge and mentioned basic procedural
inferences with those of another researcher, a version of peer
knowledge like pipetting as well as more advanced problem solv-
debrieng. Therefore, he recruited and trained the second author,
ing skills. The history professors stressed the importance of stu-
who did not engage in any of the interviewing, to analyze the data
dents learning various interpretative and argumentation skills,
from his own perspective. In addition, the rst author was aware of
along with historical empathy and perspective taking (Hartmann
his status as an adult faculty member, which might have affected
& Hasselhorn, 2008). Our history professors distinctions among
how middle school participants responded to him. Efforts were
various types of knowledge aligned with work in history education,
made to establish rapport and help the participant feel comfortable
including rst-order substantive knowledge, second-order sub-
when responding (Creswell, 2008).
stantive knowledge, and procedural knowledge (VanSledright &
The second author is an Asian-American male graduate student
Limn, 2006). First-order substantive knowledge includes the
at a large university whose primary research interest is EC. Being
what of history whereas second-order substantive knowledge in-
relatively new to the eld, his own conceptions have been much
cludes concepts and metatheories that historians use to under-
less inuenced by the EC literature than the conceptions of the rst
stand history.
author. He came to the study with a previously developed interest
In both domains, our faculty most emphasized knowledge deal-
in philosophy, signicant experience teaching language, and many
ing with context and relationships over and above independent
years residing in multi-cultural settings. This background was
facts. An appropriate collection and systematization of these var-
brought to bear on his interpretations of the interview data against
ious kinds of knowledge privies experts to what some have called
pre-established constructs and theories.
principled knowledge (i.e., well-integrated domain knowledge;
Alexander, 2003; Gelman & Greeno, 1989; Lampert, 1986). Princi-
4. Results pled knowledge allows experts to navigate the constraints of a do-
main, engage primary-source historical documents or new
Here we present a review of the themes we inferred from our scientic discoveries through the lens of their respective disci-
data analyses. These themes include empirical ndings related to plines, and create new and meaningful connections between ideas.
the nature of knowledge, ontological categories, the nature of Among our middle school participants, there were a number
knowing, domain commonalities and differences, and EC develop- whose beliefs matched what our professors had grown accustomed
ment. Each of these themes are described using quotations from to seeing in novice learners: a focus on declarative knowledge. For
our data, and elaborations from our codes and notes. example, the most common types of knowledge mentioned by our
18 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

middle school history participants were historical events, dates, 4.1.2. Certain knowledge is also not so simple
and people: Most of the time you dont really have to know the Our expert participants had strong reactions to questions about
persons perspective, you just have to know what happened to whether knowledge in their domain was certain:
really know the history (History Student #4). However, there
History Professor #1: Oh, not at all. Utterly uncertain. [laughter]
were also quite a few middle school students who expressed be-
And you could say that the level of certainty thats achievable
liefs at a more advanced level than what might be predicted from
for any particular historical fact. . . exists in inverse proportion
models of EC:
to its importance. The more important something is, the less
History Student #2: And I think with the general facts and acceptable it is to be actually known.
stuff a lot of the facts, you know, some of the basic facts of
who did this and why they did that, arent as important to me The other professors in our sample shared similar views, with
as you know, the broad, like the World War I and why that the biologists focusing upon defeasibility as a key aspect of scien-
happened. tic inquiry. Perhaps predictably, some of our middle school stu-
dents saw knowledge as xed:
This focus upon conceptual knowledge, and in particular histor- Interviewer: Ok, so, think about your history book that you have
ical themes, was certainly not expected based upon EC models and now and one that might be written twenty years from now. Do
our interviews with history professors. Some of our middle school you think that theyre going to say the same things, like in terms
students also privileged procedural knowledge like historical of the American Revolution or the World War I?
empathy skills:
History Student #1: If you dont know how [historical gures] History Student #4: Um, yeah. I think theyll say the same thing
were thinking back then and how they were feeling then. . . you in terms of those. Maybe not the exact words but theyll
wont be as good in history, because you kinda, you gotta know have the general information, the dates, the events. . . I think
how theyre thinking and know what theyre feeling to really theyll have those the same. Theyll add new stuff, yeah, but I
understand them. Like, I mean. . . like some of the former presi- think the events and stuffll be pretty much the same because,
dents, you might think theyre a bad president because of some its not still going on. Its not really much information you can
stuff they did, but if you think about maybe how they were feel- add.
ing or what their situation was, you might kinda sway the other On the other hand, some of our student participants believed
way, think theyre pretty good for what they did. that particular types of knowledge changed over time, while other
kinds did not:
In sum, all of our faculty, and even some of our middle-school
students, interpreted the word knowledge as referring to much Biology Student #1: The laws never really change or they
more than declarative knowledge. havent yet, I dont really think they ever will. . . sometimes
We believe these responses highlight a potential gap in facts change too, like over time. . . I know that there are certain
models of EC, and in particular how the nature of knowledge is animal facts that have changed. Like theyve said that theres a
conceptualized. Our respondents discussion of multiple types of certain amount of animals, but theyve discovered more.
knowledge, including declarative, procedural, and principled
Biology Student #3: Certain things. . . I think were really sure
knowledge, supported Chinn et al. (2011) assertion that research-
about. And certain things were not. Like were not really sure
ers should examine how individuals EC may vary across these
about evolution. . . Were sure about the processes that we
types of knowledge. It may be that researchers presumptions
know, that make sense, theyre logical, and we have evidence
about how sophisticated it is to agree that knowledge is
to back it up. We dont really have evidence of we dont have
simple need to be moderated depending upon whether the
multiple evidences of how things can happen for evolution that
knowledge in question is declarative, procedural, or conceptual.
we dont really know. So the things that we have a lot of evi-
Our faculty seemed to suggest that declarative knowledge in their
dence and not really a lot of conicting things on like people
domains was relatively simple, meaning that this is a fairly
dont have a lot of different sort of opinions about photosynthe-
sophisticated belief, or at least not correctly characterized as
sis. People dont think it occurs differently. We dont debate
a nave one. Faculty members did not characterize procedural
photosynthesis in science class, because it kind of just explains
or principled knowledge as simple, however, whereas some
things and it makes sense.
student participants either did see these kinds of knowledge
claims as simple (i.e., individual knowledge claims relatively What intrigued us about these statements, and what is particu-
unconnected from one another), or did not mention them at all. larly relevant to models of EC, is that our faculty, and a subset of
Given this, it may be that beliefs about the simplicity of proce- our middle school students, did acknowledge that different kinds
dural or principled knowledge develop over time, but beliefs of knowledge have varying degrees of certainty. Similar to the
about declarative knowledge do not. argument we made previously about simple knowledge dimen-
In terms of measurement, we are aware of no EC instrument sions in EC models, our participants responses suggested that
with specic items addressing different classications of knowl- believing knowledge, as a general concept, is certain is not a
edge (e.g., declarative, procedural, and principled). Items such as reliable indicator of naivety. Again, researchers who wish to evalu-
When I study I look for specic facts (Schommer, 1990, p. 500) ate participants belief in certain knowledge should carefully con-
may lead individuals with advanced levels of EC to respond in a sider clarifying the kind of knowledge claim (i.e., declarative,
nave manner (e.g., strongly agree with the item) because it is procedural, principled) they use to tap this belief. Items such as
possible that they do indeed, look for facts (i.e., our History profes- In [math/history], what is a fact today will be a fact tomorrow
sors bedrock knowledge), but only as one kind of knowledge (Greene et al., 2010, p. 251) are unlikely to be valid indicators of
among many. Further, items such as To know [math/history] well, epistemic sophistication given that respondents interpretations
you need to memorize what you are taught (Greene et al., 2010, p. of the word fact could vary between declarative, procedural,
251) seem problematically vague, because participants may re- and principled knowledge. Alternative words for fact like the
spond differently depending upon whether they infer that the item word truth may be too general to yield accurate, consistent
refers to declarative, procedural, or principled knowledge. responses as well, for example: Truth is unchanging in this
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 19

subject (Hofer, 2000, p. 390). Such variance likely adds signicant, important than structures, themes and metasystems, whereas
and systematic, measurement error to scores from these EC sophisticated beliefs were just the opposite. Our ndings regard-
instruments. ing differences in terms of what kinds of knowledge that middle-
school students and faculty valued also align with the work of
4.1.3. Epistemic values Chinn, Buckland, and Samarapungavan (2011), who identied
Our expert participants clearly had both implicit and explicit epistemic value as one of many ideas from philosophical episte-
values regarding different kinds of knowledge in their domains: mology that might be protably transferred to psychological inves-
tigations of EC. It may be that epistemic values regarding
History Professor #2: So, I try to emphasize to [students] that
declarative, procedural, and principled knowledge act as strong
there are these orders of, of knowledge, a hierarchy. . . thats
discriminators between novices and experts.
the biggest challenge. . . you can see it in the way they underline
the textbook. Theyre essentially virtually underlining without
4.2. Ontological categories
any obvious sense of hierarchy. So, theyre trying to learn things
that they dont need to know.
Interviews with our biology area participants incorporated a
Many of the quotes presented in this article illustrate that our diagram of science domain ontology (Chi, 1992). To our surprise,
professors were frustrated that their students were focused upon it did not help clarify our discussions or the middle-school stu-
the wrong kinds of knowledge, i.e., declarative facts rather dents understanding. Our biology facultys responses were mixed.
than conceptual or principled knowledge. They gave positive feedback on the idea of ontological categories
and the explicit teaching of their attributes. The Chi diagram was
History Professor #2: [The students] idea of what it is that acknowledged as being potentially useful to help students orga-
theyre going to be tested on is so different from what it is that nize and see common threads (Biology Professor #2) and to help
Im going to be asking that I have to gradually nudge them, pull them memorize less because of greater understanding (Biology
them from what it is that they think history is about and what it Professor #1). On the other hand, our faculty admitted to not being
is that we do. aware of ontologies in general or the scheme laid out in the dia-
gram. One even suggested re-arrangements that challenged the
Our biology faculty wanted their students to be able to see con-
ontologys accuracy:
nections between declarative knowledge concepts, to put things
together in [their] mind and engage in internal problem solving Interviewer: But, in terms of these broad categories, are there
(Biology Professor #1). The history professors stressed that stu- categories missing?
dents needed to learn to think analytically and develop cogent Biology Expert #2: (Viewing the diagram) Um, substances, pro-
analysis of the relationship between whatever set of historical cesses... now of course I would say that this would be a process,
events they are interested in (History Professor #2). Rather than but. . .
memorizing large quantities of disconnected information, our fac- Interviewer: The mental state.
ulty members hoped for their students to participate in conceptual Biology Expert #2: The mental states would be processes. Um.
thinking that utilized an understanding of the principles underly-
Interviewer: Why would, why would you say thats a process?
ing and permeating their disciplines. One history professor empha-
Biology Expert #2: I guess because its, its a. . . Its a, um, prod-
sized that historians were most interested in themes like
uct of a chemical, you know, a chemical in a certain region of
colonialism that had both specic applicability to particular histor-
the brain. . .. Um, substances, processes, I guess if you had some-
ical events as well as general utility in terms of providing plausible
thing like a, um. . . something like a, uh, maybe a category of
hypotheses regarding why other historical events occurred. The
forces, if that would be the right word, that is involved in, um,
epistemic value of well-argued interpretations, above the discov-
prompting or required for, you know, the material things to
ery of historical facts, was clear:
do certain processes.
History Professor #2: Some students tend to think that theres
Among the middle school students, even after some explicit
going to be the killer fact, that if you just know that fact thats
verbal explanation by the interviewer, none of their reactions sug-
going to explain this thing or the classic example of conspiracy
gested an accurate understanding of the ontology or its purpose.
theory. . . (what) I love about history is its playful. . . There are
The following exchange is an example of the students confusion:
certain things that are beyond the boundaries. You cannot argue
that the holocaust didnt happen. Thats arguing factual evi- Biology Student #3: I think that a mental state would be a the-
dence and the only reason someone would make that argument ory, because you cant really know what state youre in.
is for, obviously, political ends. Um, but within the bounds of Interviewer: Mm-hmm.
why the holocaust happened, my word! Biology Student #3: I think that that would be a fact, and that
would be a law. (Pointing at parts of the diagram.)
The issue of what kinds of knowledge were valued was also re-
Interviewer: So, processes are facts.
lated to the certainty of that knowledge.
Biology Student #3: Yes.
Interviewer: So in terms of the value of knowledge, it sounds Interviewer: And, substances are laws.
like students tend to value things that they feel they can be cer- Biology Student #3: Yes.
tain about, whereas people who are more advanced at history
tend to value interpretations, argumentative structure, context, From interviews with both faculty and students, we were left
is that. . . accurate? with the impression that the ontology diagram was not a helpful
History Professor #1: Yeah. tool for these particular participants in their respective contexts.
We expected quite the opposite: that participants would gravitate
Clearly, our professors valued different kinds of knowledge than toward what we perceived to be an effective way of thinking about
their students, as well as some of the middle school students in our how knowledge in a discipline is structured. Nonetheless, our pur-
sample. This difference between our faculty and novices aligns pose with this study was to explore participants EC, and disci-
with Leinhardt et al. (1994) ndings that nave views of knowl- plined subjectivity requires that we report both data that we
edge in history included the belief that events were more expected to nd, as well as unexpected data, including data that
20 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

contradicts our past work (Mertens, 2010). In terms of these nd- you want them to learn to do an equation, and in order to do
ings, it may well be that while ontological categories underlies an equation, theyll have to use lots of numbers, but whatever
conscious conceptualizations, it is implicit and deeply engrained its not important, remembering the numbers has nothing to
to the point that even an explicit taxonomy or systemization do with whats important about equations. Knowing how to
may not be immediately recognizable unless it is packaged by a work the equations. And thats something that I think that
formatting scheme and terminology that is familiar to experts or many, many students never quite get.
On the other hand, the ways that the experts in each domain de-
4.3. Domain commonalities and differences scribed high-level knowledge predictably varied. Our biology
professors talked about connections between ideas, and the pro-
Across domains, the language that participants used to talk cedural knowledge needed to conduct science (e.g., experimenta-
about knowledge was both similar and different. Three of our four tion skills, the mechanics of working in a lab setting, etc.). When
professors specically mentioned the word fact when describing asked what kinds of knowledge were important in history, our pro-
what they considered to be the basic, low-level and somewhat fessors mentioned words like interpretations and themes that
uninteresting knowledge in their eld. The biology professors helped create plausible explanations for societal behavior that had
examples of facts focused upon denitions, whereas the historians both specicity and elasticity (History Professor #2). The profes-
referred to historical events, people, and places. The following sors also discussed the importance of having certain kinds of pro-
quotes reect the perceived nature and larger purpose of facts that cedural knowledge that were specic to history such as historical
was consistent among all of our experts: a fact is usually one perspective taking or historical empathy skills (Hartmann &
[unrelated] item (Biology Professor #1); I tend to assign things, Hasselhorn, 2008).
secondary readings, which are not really about facts that they When comparing the language used by our experts with the
should know, but opportunities for them to try out their interpre- language of models of EC, and their associated instruments, we
tive skills (History Professor #1). found that nave views of the nature of knowledge were well de-
scribed. It seemed likely that all of the experts would recoil at
In addition, the distinctions between low-level and high- the characterization of knowledge in their domain as an accumu-
level knowledge were described similarly across domains: lation of facts (Hofer & Pintrich, 1997, p. 120). What seems to be
History Professor #1: Someone might say that the discovery of missing in models of EC, however, are specic descriptions of ad-
Brazil and the planting of sugarcane in Brazil were equally fac- vanced beliefs about the nature of knowledge within domains.
tually part of the European colonization of the world. . . So, its While there is clearly a web of connections (Biology Professor
not so much a difference in the status as regarding whether #1) among building-block ideas in the respective domains, concep-
its fact or not, but as regarding the power and the reach, tual and principled knowledge seemed to be constructed differ-
explanatory potential of the idea, so that when you get to the ently in biology and history. Likewise, if the nature of knowledge
idea that sugarcane was. . . grown in Brazil and, you might even factors are broadened to include procedural knowledge, then do-
add that it was very important to the Brazilian economy, thats main differences become even more salient. It is difcult to imag-
certainly helpful in understanding Brazil. You know it really ine a domain-general denition of the nature of knowledge factors
does get you somewhere and it explains among other things. . . - that could capture advanced experimentation skills in biology as
why so many people in Brazil are of African descent, because well as historical empathy skills in history. This difference in the
slaves and sugarcane went together basically. . . So there is cer- specic procedural knowledge relevant to these domains was
tainly something, there is explanatory power that is interesting. acknowledged by one of our historians:
But then if you move further out and say. . . the discovery of Bra-
zil and planting sugarcane are both things that help us under- History Professor #2: I have not gured out how to help stu-
stand colonialism as a process, then the complexity of that dents who may be extremely adept at working through a model
idea and the interpretive reach of that idea, both of those things of knowledge. . . grounded in the scientic approach. . . history is
are much greater. So theres a lot more that you can say about very different.
colonialism not involving sugarcane at all.
Overall, our comparisons of how experts in biology and history
Biology Professor #1: You might know what DNA is and you
talked about the nature of knowledge in their domains revealed
might know what nucleotides are and you might know what
strong similarities in terms of nave beliefs. There was also some
genes are, in terms of being able to dene each of those. . . but
congruence across domains regarding the purpose of high-level
you have to be able to know how they work together, that
knowledge: to connect and explain relations among low-level
DNA is part of the chromosome and it is, a length of it is a gene,
facts. However, when comparing our facultys responses to the
and how those t together, those connections. . . Because again,
denitions in models of EC, it was clear that nave beliefs were suf-
you could say DNA carries the genetic code that is something it
ciently described, but sophisticated beliefs about procedural and
does, but to me thats a fact. . . Thats not an explanation. The
conceptual knowledge were underspecied.
real relationship between DNA and the code, theres sort of a
Our results cohere to some degree with the work of Samara-
richer part that would again in my mind at least involve con-
pungavan, Westby, and Bodner (2006), who found that chemis-
necting it with mutation.
try researchers epistemic beliefs, disciplinary knowledge, and
Our historians saw the use of themes as a way of connecting inquiry practices were all chemistry-specic, rather than sci-
and explaining historical actions and events, and our biologists ence-specic (i.e., common epistemic practices across physics,
saw the use of theories (e.g., mutation) as a way of connecting chemistry, and biology, among other hard sciences) or domain-
and explaining the relationships between biological phenomena. general. Those authors argued that it is unlikely that there are
Each of our experts saw facts, the things that middle-school stu- domain-general or even science-specic epistemic beliefs, as
dents tend to focus upon, as necessary but not nearly sufcient posited by some educational psychologists (e.g., Schommer &
materials that were used to build high-level knowledge: Walker, 1995). Rather, the work of Samarapungavan and col-
leagues supports the work of Hammer and Elby (2002), who ar-
History Professor #1: The facts are simply. . . something that
gued that students rely upon ne-grained epistemic resources
allowed [students] to work with these ideas, as if, for example
that exist at the setting or task level. While our ndings suggest
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 21

that much more attention needs to be paid to domain-specicity, This professor discussed how historians seek themes that not
including, perhaps, the introduction of some number of task-spe- only help people understand why a particular historical event oc-
cic dimensions of EC, it is not yet clear whether there are abso- curred (i.e., specicity) but also why other historical events might
lutely no EC dimensions that generalize across domains. What is have occurred (i.e., elasticity). Each history professor mentioned
clear is that EC researchers must take seriously the possibility that themes must align with the accepted details of historical
that intraindividual differences on common EC dimensions events (e.g., the chronology of events) but that the true test of
across domains (e.g., participants scoring high on simple knowl- whether a claim would be considered valuable in history was the
edge in science but low on similar items in history) may not be degree to which it was helpful in understanding human activity:
capturing all of the relevant variance regarding how individuals
History Professor #2: Well, the two classic [themes] of the dis-
EC differs across domains, tasks, and settings. Clearly, more re-
cipline are cause and effect and change over time. And I encour-
search is needed in this area.
age students to always think about whether during whatever
particular historical event or moment in time, they should ask
4.4. The nature of knowing themselves Why is change occurring? or Why isnt change
4.4.1. Expert perspectives on justication and authority
When talking about sophisticated and valued knowledge in What struck us about each of our interviews with faculty was
their domains, our faculty acknowledged the tentative, contingent, that multiple sources of justication (e.g., coherence with other
and defeasible nature of these claims: claims, predictability of other events) were required before these
individuals were willing to anoint a claim with epistemic status.
Biology Professor #1: Theories are the big overarching things
This focus on multiple sources of justication aligns with several
that we hold as fundamentally true always acknowledging the
recent conceptual pieces on EC (Chinn et al., 2011; Greene et al.,
tentativeness of science, that theres a chance theyre not true
2008; Murphy et al., 2007) and runs contrary to models of EC that
but its where were going today and its what were basing
use only one dimension to characterize individuals beliefs about
our understanding on today.
justication (e.g., Hofer & Pintrich, 1997).
As Greene et al. (2008) have argued, an acknowledgment of Given that many EC models have the implicit assumption that it
defeasibility increases the importance of establishing the means is important to develop students ability to justify knowledge, we
of justication in a eld. Therefore, we asked specic questions were particularly struck by this statement:
about how these faculty validated knowledge claims qua
History Professor #1: In my classes I authorize the textbook as a
source of true, true facts [laughter] right? And, uh, which is,
The biology professors in our sample mentioned peer-reviewed which is uh you know, provisional and heuristic. . . But, provi-
journal publications as a salient, but not entirely sufcient, sionally for courses in the way that students sort of are
means of justifying of knowledge claims: approaching learning history, [I] give the text book that episte-
Biology Professor #2: I mean I think authority is a big, is a big mological status.
determiner of what we come to say is knowledge, um, only
because its such a, its such a peer-reviewed system. But I dont In essence, this professor alleviated the need for students in his
think theres a sense of authority where [a] Nobel person can class to justify declarative knowledge qua knowledge. However,
say something and everybody will just believe it; theyll say, his intention when doing that was to sufciently justify the bed-
well show me the evidence. rock knowledge that would be used to develop students historical
understanding and argumentation skills. Giving the textbook
These faculty seemed to view the peer-review process as a
authority status allowed students to focus upon developing other
proxy for doing their own verication of the legitimacy of a knowl-
EC skills, such as evaluating knowledge claims based upon the
edge claim. Given the impracticality of conducting their own inves-
specicity and elasticity of themes.
tigations of every study published, these faculty relied on their
peers to review the submitted studies for rigor and only accept
4.4.2. Middle school student perspectives on authority
those that were worthy of achieving knowledge status.
Across both the biology and history domains, our middle school
Both of our biology professors listed a number of other kinds of
participants frequently mentioned textbooks and teachers as suf-
justication for a knowledge claim beyond publication in peer-re-
cient justication for knowledge claims. However, when asked
viewed journals, including replication studies, and predictability
what they would do if a textbook and a teacher endorsed different,
(Biology Professor #1). The latter term was described as the degree
conicting knowledge claims, many participants listed scientists or
to which a knowledge claim cohered with other knowledge claims,
the Internet as the nal arbiter of truth. Views of the Internet as a
and proved helpful in predicting additional phenomena. These per-
reliable source of knowledge were varied, with two history stu-
spectives aligned well with the work of Murphy, Alexander,
dents specically mentioning it as a viable source, while another
Greene, and Edwards (2007), who found similar ideas (i.e., coher-
four were more skeptical, particularly of the veracity of websites
ence and reliabilism) among the kinds of justication discussed
like Wikipedia.
within philosophical epistemology. Overall, our biology professors
Two of our middle school biology participants preferred doing
appeared to take a reasonably skeptical stance toward knowledge
experiments to evaluate knowledge claims, rather than relying
claims, requiring a preponderance of evidence (e.g., peer-review,
on other authority gures. These students responses aligned with
replication, coherence, predictive utility) before accepting a knowl-
Sandoval and ams (2010) ndings that elementary school chil-
edge claim qua knowledge.
dren privileged rst-hand experience with data over other means
The history professors also had criteria for establishing knowl-
of justication. Another of our student participants distrusted text-
edge qua knowledge:
books because they contained spelling errors that led to questions
History Professor #2: Thats probably the hardest thing of all about whether the authors were sufciently knowledgeable to
to explain. I think I see it in good historical arguments. . . one present authoritative knowledge claims. Further, some of our stu-
that combines. . . a certain kind of specicity to it as well as dents were unsure how to reconcile between various sources of
elasticity. knowledge:
22 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

History Student #2: Ive always imagined people that write the knowledge, and therefore their views of justication were much
textbook you know, as the guy has big glasses on and who like more limited than the facultys views. In biology, the means of jus-
studied history for like forever and ever. But Ive never really tication within the eld were evident to both middle-school stu-
been positive that the textbooks right [be]cause in a lot of clas- dents and faculty, whereas in history the means of justication
ses, not just history, the textbook says a lot of different things were much more oblique to the middle-school students, primarily
than, you know, what the present date person is saying or what because they were focused on what the historians would call the
our teacher is saying. But, Im not really sure how, or to trust wrong kinds of knowledge.
4.4.4. Implications for the nature of knowing
Overall, there seemed to be a great deal of variance across mid- Each of our participants, including the middle-school students
dle school students regarding the relative reliability of various and the faculty, mentioned means of justication that have been
authorities in biology and history, with particular skepticism called authorities in the EC literature (e.g., peer-reviewed jour-
regarding the Internet. Learned others, including teachers and pro- nals, scientists, teachers). Experts in both disciplines, and middle-
fessors or scientists, were the most frequently mentioned authori- school students in biology, discussed the importance of nding
tative sources. Participants were much more variable in their faith multiple means of justication. Our history middle-school students
in textbooks as an authoritative source, a nding that somewhat had a much more nave faith in authorities. Therefore, measure-
contradicts past research that found that middle school students ment items like I am most condent that I know something
viewed textbooks as sacred texts that would not lie to them (Gold- when I know what the experts think (Hofer, 2000, p. 251) may
man, 2004; VanSledright, 2002). These students confusion regard- elicit a variety of responses from participants, depending upon
ing what sources to privilege may explain why our professors felt whether they infer that the item is asking whether authorities on
the need to anoint certain sources, like particular textbooks, as their own can be sufcient justication, or whether authorities
authoritative. Given the professors desire for students to develop are one piece of evidence among many. Chinn, Buckland, and
advanced thinking skills, which across domains involved justifying Samarapungavan (2011) have pointed out that while psychological
principled and not declarative knowledge, it seems reasonable to EC research has positioned unquestioned belief in authorities as an
avoid the cognitive load that comes with having to justify bedrock indicator of nave beliefs, in philosophy arguments from authority,
knowledge as knowledge. or what they call testimony (p. 22), are common and often
accepted by experts, sometimes with little questioning, depending
4.4.3. Middle school student perspectives on means of justication upon the source.
Our faculty in biology mentioned experimentation, replication, Our ndings suggest that EC researchers need to not only mea-
and coherence as the primary ways of justifying knowledge claims, sure learner beliefs about a more diverse set of sources of justica-
along with publication in peer-reviewed journals. We were curious tion for knowledge claims than has been conducted in the past
as to what our middle-school students would say about justica- (Greene et al., 2008; Murphy et al., 2007); they also need to nd
tion in biology: ways to determine whether individuals are questioning and using
Interviewer: How do you know that what youre learning is testimony in ways that are accepted within particular academic
true? How do you know that its accurate? domains (e.g., evaluating the credentials of the authority, deter-
Biology Student #4: You dont really ever know I guess if its mining the coherence of the claim with other knowledge claims).
true, but I guess its been around for long enough that what It also seems important to identify the kinds of knowledge that
were learning has been established by a lot of people that know respondents think about when answering such items, as that could
what theyre talking about. Theres maybe some specics may inuence the degree to which testimony would be seen as suf-
not be right, but. . . its been tested and proven. . . Well, we think cient justication.
that it happened. It could not really happen, but its been proven Beyond testimony as a source of justication, our ndings indi-
and, or supposedly proven, I mean it could not be really happen- cated that EC researchers should build on the work of Sandoval and
ing. Its just. . . it makes sense that it happens because it ts with am (2010) to examine how individuals, particularly middle school
other things that happen. students, rank various authority sources (e.g., teachers, textbooks,
scientists, Internet, etc.). It may be that certain rankings are predic-
Overall, our biology students had what we considered to be tive of academic success, whereas others are not. In particular, if a
very sophisticated views of justication in biology. Most under- teacher has one ranking, and the student another, this may lead to
stood that a single experiment could provide evidence for a claim, different conclusions about controversial knowledge claims qua
but that replications and theoretical coherence were also needed knowledge. Finally, it seems unlikely that survey items that group
before the claim could be tentatively adopted as knowledge. In all of these potential sources into a single authority will yield
our sample, the middle-school students and faculty shared a com- reliable and useful data about the sophistication of individuals EC.
mon view of how justication occurred in biology.
In history, our middle-school students had very different views 4.5. When development begins
about justication than our faculty, primarily because the majority
of our middle-school students believed that historical knowledge Going into the interview portion of this project, we were not
was solely comprised of facts such as historical events and peo- sure whether the middle school participants would be able to con-
ple. Therefore, when asked how they knew something to be verse with us about EC issues, given their abstract nature and the
accurate in history, most referred to various authority gures fact that many EC researchers have found little to no EC sophistica-
who could substantiate those facts, as described previously, identi- tion in participants younger than college age (e.g., King & Kitchen-
fying the locus of justication outside of the self. This sole focus er, 1994). Many of the quotations in previous sections of this paper
upon outside authority, or testimony, among the middle-school have illustrated that some, but not all, of our participants were able
students was in stark contrast to the means of justication used to engage in EC beyond the rudimentary levels. Additional evi-
by our faculty, which included evaluations of the quality of histor- dence of sophisticated EC included one science student who, when
ical interpretations and arguments. Compared to the history pro- asked how she evaluates knowledge claims, said that she picked
fessors, our middle school history students had very different one that ts best to everything else, that makes the most sense.
interpretations of what we meant when we asked about historical Such a focus on what Murphy, Alexander, Greene, and Edwards
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 23

(2007) would call coherence and rationality does not align with the of EC models was to start from an open-ended, exploratory stance,
developmental assumptions of many models of EC. as opposed to a theoretically-driven one which typies quantita-
Another student participant, when asked about reconciling tive research. We believe that this decision led to interview data,
competing knowledge claims in history, referred to ideas about and subsequent inferences about participants EC, that would have
historical perspective taking that were similar to those of our fac- otherwise been difcult, if not impossible, to obtain through meth-
ulty: I kind of just put myself in the shoes of each side and see ods derived directly from EC models. Such ndings complement
which one might make more sense. This kind of historical per- current quantitative research, and conceptual work, in the eld
spective taking would seem unlikely for a person who had sim- (e.g., Chinn et al., 2011; Mason, Ariasi, & Boldrin, 2011).
ple views of knowledge and knowing. Additionally, it does not Thus, our interviews were informed by the various conceptual
seem that middle school students are necessarily devoid of the epi- models of EC that dominate the literature, but not restricted by
stemic values held by experts: them. Rather, we used our participants responses as a tool to chal-
lenge theory, and broaden our understanding. In the end, we found
History Student #2: . . .when teachers tell me facts, I just like ok,
that research into beliefs about the nature of knowledge and know-
Ill just study that and you know take a test on it and Ill be done
ing may benet from investigation into the natures of particular
with it. But when they talk about you know, why they did this
knowledge domains. Through our exploratory multiple case study
and, it put- it further explains it so, I mean, just general facts
design and data analysis, we identied a number of ways that
that we need to know dont really, I mean, dont really have
models of EC might be protably altered to better match novice
much impact on me as when teachers actually explain you
and expert beliefs about the natures of knowledge and knowing.
know, why this person did that, and you know, what it did
We have also identied ways that the measurement validity of
and the reason why they did it and stuff like that.
EC self-report instruments might be improved. However, our nd-
ings also revealed concerns with directions that we had initially
Other researchers have found evidence of sophisticated EC in
believed to be likely to be fruitful (i.e., ontological categories).
students younger than college age (Greene et al., 2010; Kuhn
et al., 2000) and our ndings cohere with those studies.
5.1. Nature of knowledge
4.6. Summary of results
Both our faculty and our middle-school student respondents
Overall, our interviews revealed a number of ways in which cur- interpreted the term knowledge broadly, including what we
rent models of EC do align with our participants thinking (e.g., ex- would call declarative, procedural, and principled types of knowl-
perts beliefs about the certainty of knowledge) as well as a edge. After comparing these responses to our review of conceptual
number of ways in which they do not (e.g., experts distinctions be- denitions and specic quantitative items from measures of EC, it
tween different kinds of knowledge; middle school students seems that the current literature has addressed declarative knowl-
understanding of justication in biology). Our ndings, some of edge beliefs well, but is underspecied in terms of advanced con-
which are anomalous in terms of current EC models, have direct ceptual or principled knowledge beliefs, also called second-order
implications for how EC is measured, and may explain why EC substantive knowledge in history (VanSledright & Limn, 2006),
instruments have displayed poor psychometric qualities. Table 1 and completely bereft of attention to procedural knowledge be-
illustrates a number of these important ndings. liefs. Models of EC include the assumption that beliefs about the
nature of knowledge change over time, but that assumption might
5. Discussion not accurately describe how participants, and particularly experts,
view declarative knowledge. While our faculty acknowledged that,
By conducting a qualitative investigation of faculty and middle- in actuality, all knowledge claims are defeasible, declarative
school students EC across two different domains, we sought to ex- knowledge claims (e.g., the Declaration of Independence was
plore the ways in which the conceptual literature did and did not signed in 1776) seemed to be, for practical purposes, relatively
seem to be capturing participants EC, with the hope of discovering simple and certain. Therefore, models that do not account for dif-
potentially protable new directions for EC model and measure ferences in the simple and certain nature of knowledge beliefs
development. It was our hope that elucidating such gaps might ex- across types of knowledge, and instruments that include items
plain, among other things, why many self-report instruments have about knowledge in general, are unlikely to adequately discrim-
shown such poor measurement validity (e.g., DeBacker et al., inate between novices and experts, leading to poor measurement
2008). Critically, we felt that the best way to question the viability validity.

Table 1
Findings relevant to the re-conceptualization of current approaches to EC models and measurement.

Simple knowledge Certain knowledge Epistemic value

 Experts believed declarative knowledge matters, but is of low epistemic  Experts believed knowledge is uncertain and  Experts strongly valued
value defeasible principled and conceptual
 Experts and some novices believed connections between declarative  History experts believed the importance of knowledge
knowledge (e.g. principled, conceptual knowledge) are of high epistemic knowledge usually has an inverse relationship  Some novices strongly val-
value to its certainty ued declarative knowledge
 Experts believed that declarative knowledge is simple  Novice students varied in their beliefs about the
certainty of knowledge
Justication Experts language
 Experts believed defeasibility exists, justication matters, and multiple  Terminology used to describe low-level knowl-
sources of justication are best edge was remarkably similar across disciplines
 Novices displayed variability in which sources they viewed as trustworthy  Terminology used to describe high-level knowl-
 Biology novices displayed views of justication resembling those of edge was notably different across disciplines
experts. History novices displayed views of justication very different  Descriptions of procedural knowledge differed
from experts views across disciplines
24 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

Further, our faculty emphasized that a key difference between to determine the degree to which individuals utilize one or more
their understanding of knowledge in their elds and that of their means of justication to establish knowledge. It may be the case
students was the kinds of knowledge they valued (Chinn et al., that novices warrant new knowledge through only a single or a
2011). Particularly in history, our students and faculty seemed to few kinds of justication, such as our history students, whereas ex-
differ on what they thought we meant when we asked about perts routinely examine knowledge using a broader spectrum of
knowledge. Much of the knowledge that our middle-school stu- evidence.
dents valued in history was what our faculty viewed as the least As Chinn, Buckland, and Samarapungavan (2011) have sur-
interesting or important aspects of historical understanding. As mised, all of our participants, including the faculty, depended upon
Chinn, Buckland, and Samarapungavan (2011) have stated, models various authorities to some degree when evaluating knowledge
of EC might be improved by the addition of epistemic value, and claims. This suggests that researchers need to reconsider models
the absence of items measuring differences in epistemic value of EC that position justication by authority as a solely nave view
likely diminishes EC instruments utility in differentiating between (e.g., Schommer, 1990). When attempting to identify adaptive EC,
experts and novices. it may be more accurate to examine the degree to which individu-
Greene, Azevedo, and Torney-Purta (2008) have argued that EC als use of authorities as justication matches that of experts. We
researchers interest in peoples beliefs about the nature of knowl- also found that students had varied opinions regarding which
edge can be better described by incorporating Chis (1992, 1997) authorities were more reliable than others. These ndings suggest
ideas about ontological categories. While it is possible that we that it is not sufcient to simply ask individuals to indicate their
were not asking the right questions, it is nonetheless the case that belief in a general authority or expert.
our faculty participants did not seem to recognize or resonate with Finally, perhaps it was not surprising that we found evidence of
the knowledge ontology we presented them. If Greene and col- fairly sophisticated EC among our middle school students, given re-
leagues are correct that Chis ontological categories are how ex- cent research (e.g., Greene et al., 2010; Mason et al., 2010). None-
perts think about knowledge, then these ontologies are tacit, at theless, we felt it important to demonstrate our own form of
least for our participants. It may also be the case that more do- replication here by highlighting these ndings, adding further evi-
main-specic language is needed for these ontologies to resonate dence that refutes the idea that EC development does not begin un-
with experts (Hammer & Elby, 2002), or that the idea of ontological til the college years. Clearly, EC plays a powerful role in learning
categories, as conceptualized by Greene and colleagues, does not throughout adolescence, and likely earlier as well.
match how novices and experts think about knowledge.
Finally, while both biology and history faculty agreed that there 5.3. Limitations
were low-level and high-level kinds of knowledge in their do-
mains, and that these levels were related, there were clear differ- Our study, and the associated ndings, were derived from a
ences in the specic ways in which the professors talked about small sample of faculty and middle school students from a partic-
conceptual and principled knowledge in their domain. Biologists ular area of the United States. Our purpose was not to generalize,
referred to connections, while historians focused upon interpreta- but rather to investigate whether the data led to reasonable con-
tions. We have a difcult time imagining how domain-specic EC cerns regarding models of EC and their measurement, and indeed
items could be fashioned such that beliefs about the nature of con- this was the case. Whether these specic ndings are representa-
ceptual knowledge could be captured in both domains by simply tive of larger populations remains open to investigation, but clearly
substituting the word biology for the word history, as EC we have proof-of-concept that our participants responses do not
researchers have done in the past (Buehl et al., 2002; Greene t well into most current models of EC. Another limitation was our
et al., 2010). Our data call into question the very meaning of do- reliance on interviews as the sole measure of participants EC.
main-specic EC. It may be that it is not just that individuals be- Numerous researchers (e.g., Chinn & Brewer, 1993) have shown
liefs about the nature of knowledge can vary across domains, but that while scientists may espouse certain beliefs (e.g., that anoma-
that the actual factors themselves (e.g., simple knowledge, certain lous data should lead to a reconsideration of underlying theory),
knowledge) may differ across domains. These ndings align with they often do not act in accordance with these beliefs (e.g., ignoring
the work of Samarapungavan et al. (2006) who found epistemic be- anomalous data). These ndings call into question whether inter-
liefs and practices that were specic to chemistry, and they argued views are sufcient means of truly understanding experts do-
that these specic beliefs and practices would not transfer effec- main-specic beliefs and behaviors. In addition, our sample was
tively to other hard sciences. small and convenient, and we were not able to access key academic
performance or school demographic data. This limited our ability
5.2. Nature of knowing to select a sample based upon these factors. Future research should
incorporate more diverse and systematically gathered samples.
Our interviews revealed a great deal of nuance regarding how While we attempted to engage in disciplined subjectivity through
our experts justied knowledge qua knowledge. Faculty in biology critical reection, it may be the case that our inferences were un-
relied upon a coherentist (Murphy et al., 2007) strategy, evaluating duly inuenced by our particular experiences with EC. Future re-
the cumulative weight of evidence across numerous sources (e.g., search is needed to determine whether the proposed
peer-review journal articles, replications, predictive utility). Pro- modications resulting from our study are accurate depictions of
fessors in history focused on the quality of historical arguments novice and expert EC, and whether they prove useful in relating
in terms of multiple factors including (a) their utility at explaining EC to academic outcomes.
specic human and societal behavior, (b) their general applicability For this study, we adopted a decidedly qualitative approach and
to a wide variety of historical phenomena, and (c) their alignment therefore did not attempt to leverage the aspects of controlled de-
with bedrock knowledge about historical people, places, and sign that are typical when supporting claims made via a quantita-
events. These ndings align with the work of Greene, Azevedo, tive paradigm. The strength of our qualitative approach was to
and Torney-Purta (2008) who argued that EC researchers needed facilitate an illumination of EC phenomena that was less con-
to measure individuals beliefs about multiple sources of justica- stricted by the assumptions and models of past work. Our goal
tion, and further substantiate Hammer and Elbys (2002, 2003) was not to comprehensively compare learners along all of the EC
work regarding how epistemic cognition is specic to domain. aspects suggested in the protocol for every participant. Rather,
Our studys ndings suggest that methods need to be developed we hoped to nd clear evidence of EC phenomena, inasmuch as
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 25

it was possible for the scale of our study, by operating outside of to not only include intraindividual variation in EC across do-
the a priori EC assumptions inherent and necessary in quantitative mains, but also the possibility that certain relevant beliefs about
research tools. The semi-structured protocol and interview design the nature of knowledge and knowing themselves are domain-
was meant to simulate ideas and provide opportunities through specic (Hammer & Elby, 2002, 2003).
which participants could articulate their thoughts as freely as pos- Likewise, many EC instruments measure expertise in a do-
sible. Necessary to our interests was allowing and even encourag- main based upon the degree to which individuals disagree with
ing our participants to pursue their own lines of thought and for nave statements about the nature of knowledge and knowing.
the interviewer to follow participant lead in terms of specic top- Unfortunately, merely disagreeing with a nave statement is not
ical interests. Such an approach is in line with qualitative research a reliable indicator of expertise, particularly when considering
ideals that value maximization of participant voice and its ability that many EC researchers posit in their models that beliefs about
to elucidate the complexity of worldly phenomena (Patton, knowledge and knowing exist on a continuum from nave to
2002). Unfortunately, this also means that participants interview sophisticated, and that these beliefs are not dichotomous choices
experiences necessarily varied in some ways, precluding exacting between navet and sophistication (Greene et al., 2008; Hofer &
comparisons across participants. Therefore, ndings from this Pintrich, 1997; Schommer, 1990). Non-experts may disagree
study should be considered empirically supported suggestions with claims such as To know [math/history] well, you need to
for the re-consideration of theory and measurement rather than memorize what you are taught (Greene et al., 2010, p. 251),
an attempt to prescribe a specic set of recommendations but that does not mean that they hold beliefs about justication
regarding individual differences in EC across populations. Such that are similar to those of experts. It seems to us that more
work, likely incorporating rigorous quantitative methods, is attention must be paid to writing items that capture sophisti-
certainly needed. cated beliefs about knowledge and knowing. Given our concerns
Finally, this study did not investigate the larger contextual about how faculty might be interpreting the items on quantita-
inuence of instructional practice upon EC development. It is likely tive measures of EC, it seems that cognitive interviewing studies
that differing instructional practices and curricular goals in biology (Karabenick et al., 2007) with this population are warranted.
and history strongly contribute to particularities in EC develop- There were also important nuances in how our middle
ment. Our interviews revealed multiple instances of expert-like school students thought about authorities in biology and
EC from some of our biology novices along the justication dimen- history. While there was variance, in general our participants
sion, but not from the history novices. We also saw multiple in- seemed to privilege certain authorities over others. Therefore,
stances of expert-like EC from some of our history novices along a potentially useful way to measure EC would be to determine
the simple knowledge dimension, but not so much from the biol- whether students have ranked authorities in ways similar to
ogy novices. If EC models begin to account for such differences in experts, or in ways that reliably lead them to accepted
their developmental trajectories, research into the degree to which knowledge claims. Such rankings might predict how students
they are determined by characteristics of the domain or by instruc- engage in open-ended searches of multiple information sources
tional practice would be an important line of inquiry. (e.g., Internet) when attempting to answer ill-structured problems
(Brten, Britt, Stromso, & Rouet, 2011; Hofer, 2004; Mason
5.4. Implications for measuring epistemic cognition et al., 2010).
Finally, we would be remiss if we failed to mention that our
One implication that we hope is clear from this study is that EC ndings may add support to the argument that EC cannot be mea-
research would benet from additional multiple and mixed meth- sured using survey-type items at all (Hofer, 2006; Muis et al.,
ods investigations of the phenomenon (Lewis & Grimes, 1999). In 2006). It may be that other types of investigations, such as think-
terms of our study, we did not feel that quantitative analyses of aloud protocols (Mason et al., 2010) are required to surface and
our relatively small samples, particularly when subdivided into understand EC. It may also be the case that what learners do is
groups, would yield interpretable data. However, the growing use far more important than what they report or say regarding their
of think-aloud protocol data analysis techniques (e.g., Mason EC (Greene et al., 2010). Therefore, analyses of learner actions
et al., 2011) presents an example of how quantitative inferences and interactions may be the necessary next frontier in EC research
can be made from qualitative data. Likewise, we believe that the (e.g., Greene et al., 2010).
unique ability to question theory with qualitative data nicely com-
plements the model testing and generalization that can result from 6. Conclusion
well-designed quantitative research. In the future, we plan to iter-
ate between quantitative and qualitative techniques to advance to- Until researchers are able to accurately capture intra- and inter-
ward a deeper and more comprehensive conceptualization of what individual EC variation, claims of relations between EC and aca-
EC is, and how it affects learning and performance among novices demic outcomes will warrant at best tentative status as
and experts. knowledge. The eld simply must do a better job of measuring
In our review of our ndings we outlined a number of ways EC. Ultimately, the poor measurement validity evidence for self-re-
that EC models could be expanded or altered. Our hope is that port measures of EC may be due to any of the three reasons we
such changes will lead to measures of EC with strong evidence listed previously, or another reason altogether. Our study explored
of measurement validity. However, if self-report instruments one particular possibility, that the models of EC themselves were in
are to be useful measures of EC, they need both strong theory need of modication due to a lack of alignment with novices and
and item design. Writing items that tap beliefs about the value experts stated EC and a lack of attention to the inherent differ-
of sophisticated ideas in biology (e.g., theories) and history ences between domains of knowledge. We believe our ndings
(e.g., themes) may prove to be a challenge, particularly if the suggest numerous potentially benecial avenues for exploration
items need to be parallel, as asserted by other researchers and potential modication of EC models. In particular, we believe
(e.g., Buehl et al., 2002; Greene et al., 2010; Hofer, 2000). It much more nuance is needed regarding how models of EC address
may be the case that the items needed to truly capture expertise the various kinds of knowledge, the degree of similarity and differ-
in one domain are necessarily and fundamentally different than ence in beliefs across and within domains, and the ways in which
those needed to capture expertise in another (Samarapungavan individuals utilize multiple means of justication to evaluate
et al., 2006). This would expand the idea of domain-specicity knowledge claims.
26 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

Acknowledgments Biology-specic question:

In terms of debatable things in science like evolution or global
The authors would like to thank Kristin Dellinger and Stacy warming, how do you decide what to believe and what not?
Ruse for their assistance in collecting and transcribing data. History-specic question:
This research was supported by a grant from the University Think about your history book now and one written 20 years
of North Carolina at Chapel Hill School of Education (Guy B. from now, will they say the same things in terms of the American
Phillips Research Award for Grant Initiation) awarded to the Revolution or World War I?
rst author.

Appendix B.
Appendix A.
Faculty member semi-structured interview protocol
Middle school student semi-structured interview protocol:
(1) To be knowledgeable in [biology/history], what are the
(1) Can you tell me how you feel about [biology/history] in kinds of things students need to know?
general? (2) In [biology/history] there is a compendium of knowledge.
(2) Can you tell me how you feel about [biology/history] class? Within that compendium there are different types of knowl-
(3) What do you think your teachers want you to learn in [biol- edge. What are those types?
ogy/history] class? (3) How would you classify different types of knowledge in
(4) What are the types of things you need to know in [biology/ [biology/history], how would you label them?
history] to do well? (4) Are there different kinds of classications that are important
(5) How would you classify different types of knowledge in or useful?
[biology/history], how would you label them? (5) Id like to know more about how you think about knowledge
(5a) Are there different kinds of classications that are in [biology/history]. For example, one kind (or category) of
important or useful? knowledge in [biology/history] is dates when important
(6) Id like to know more about how you think about knowledge events happened. Another kind (category) of knowledge
in [biology/history]. For example, one kind (or category) of is [for history: the people involved in those events for biol-
knowledge in [biology/history] is dates when important ogy: substances, the physical things you might study].
events happened. Another kind (category) of knowledge What other kinds (or categories) of knowledge can you
is [for history: the people involved in those events for biol- think of?
ogy: substances, the physical things you might study]. (6) Now that you have listed many different kinds of knowl-
What other kinds (or categories) of knowledge can you edge, what are the important ways those kinds of knowledge
think of? differ? What makes these categories distinct from one
(7) Now that you have listed many different kinds of knowledge, another?
what are the important ways those kinds of knowledge dif- (7) How does knowledge of [biology/history] help us as a
fer? What makes these categories distinct from one society?
another? (8) Would you say that some kinds of knowledge are simpler or
(8) Would you say that some kinds of knowledge are simpler or more complex than others, or are they all about the same?
more complex than others, or are they all about the same? (9) Would you say that some kinds of knowledge are more or
(9) Would you say that some kinds of knowledge are more or less certain than others, or are they all about the same in
less certain than others, or are they all about the same in terms of certainty?
terms of certainty? (10) How likely do you think it is that each of these kinds of
(10) How likely do you think it is that each of these kinds of knowledge might change over time as humans learn more
knowledge might change over time as humans learn more about them?
about them? (11) In [biology/history] how do we know if something is true or
(11) In [biology/history] how do we know if something is true or is a fact?
is a fact? (12) How do you know that these kinds of knowledge are really
(12) In terms of debatable things in science like evolution or glo- knowledge and not something different like a belief or
bal warming, how do you decide what to believe and what guess?
not? (13) How can you justify a knowledge claim as knowledge?
(13) How do you know that these kinds of knowledge are really (14) In thinking about [biology/history] textbooks now versus
knowledge and not something different like a belief or those that will be used 20 years from now, what differences,
guess? if any, will there be?
(14) When do you accept something as knowledge? (How many (15) Is there anything else that you think is important about what
sources do you use in order to justify something as weve been discussing that you havent mentioned yet?
knowledge?) (16) In terms of helping students be successful in history, are
(15) Are there certain people or other sources of knowledge that there other things that you think are important, that stu-
you listen to more than another? (Internet, teacher, parents, dents should know?
books, etc.)
(16) If a scientist, a textbook, a teacher, and an internet site all Biology-specic questions:
say something different about a topic, how do you decide What are the characteristics of a law or theory that differentiate
what is true? it from a fact?
(17) How do you know something versus being unsure of It has been proposed that there are three broad categories in
something? biology: substances, processes, and mental states, and that one
(18) Is there anything else that you think is important about what reason why students confuse an idea like heat is because they
weve been discussing that you havent mentioned yet? mis-categorize. By doing so they apply characteristics of sub-
J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228 27

stances to heat which is actually a process. Do you think that instruction: Essays in honor of Robert Glaser (pp. 125186). Hillsdale, NJ:
teaching students about science ideas in terms of these categories
Goldman, S. R. (2004). Cognitive aspects of constructing meaning through and
and their characteristics might be helpful? across multiple texts. In N. Shuart-Ferris & D. M. Bloome (Eds.), Uses of
intertextuality in classroom and educational research (pp. 313347). Greenwich,
History-specic questions: CT: Information Age.
Greene, J. A. (2009). Collegiate faculty expectations regarding students epistemic
How important is it to understand how people were thinking in and ontological cognition and the likelihood of academic success. Contemporary
the past? Educational Psychology, 34, 230239.
How are the whys of history similar or not similar to the 05.003.
Greene, J. A., Azevedo, R., & Torney-Purta, J. (2008). Modeling epistemic and
facts of history? ontological cognition: Philosophical perspectives and methodological
directions. Educational Psychologist, 43(3), 142160.
References Greene, J. A., Muis, K. R., & Pieschl, S. (2010). The role of epistemic beliefs
in students self-regulated learning with computer-based learning environ-
ments: Conceptual and methodological issues. Educational Psychologist, 45(4),
Alexander, P. A. (2003). The development of expertise: The journey from
acclimation to prociency. Educational Researcher, 32(8), 1014.
Greene, J. A., Torney-Purta, J., & Azevedo, R. (2010). Empirical evidence regarding
Baxter Magolda, M. B. (1992). Knowing and reasoning in college: Gender-related
relations among a model of epistemic and ontological cognition, academic
patterns in students intellectual development. San Francisco: Jossey-Bass.
performance, and educational level. Journal of Educational Psychology, 102(1),
Belenky, M. F., Clinchy, B. M., Goldberger, N. R., & Tarule, J. M. (1986). Womens ways
of knowing. New York: Basic Books.
Hammer, D., & Elby, A. (2003). Tapping epistemological resources for learning
Biglan, A. (1973). The characteristics of subject matter in different academic areas.
physics. The Journal of the Learning Sciences, 12(1), 5390.
Journal of Applied Psychology, 57(3), 195203.
Hammer, D., & Elby, A. (2002). On the form of a personal epistemology. In B. K. Hofer
Boyes, M., & Chandler, M. J. (1992). Cognitive development, epistemic doubt, and
& P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about
identity formation in adolescence. Journal of Youth and Adolescence, 21,
knowledge and knowing (pp. 169190). Mahwah, NJ: Erlbaum.
Hartmann, U., & Hasselhorn, M. (2008). Historical perspective taking: A
Brten, I., Britt, M. A., Stromso, H. I., & Rouet, J.-F. (2011). The role of epistemic
standardized measure for an aspect of students historical thinking. Learning
beliefs in the comprehension of multiple expository texts: Toward an integrated
and Individual Differences, 18, 264270.
model. Educational Psychologist, 46(1), 4870.
Hofer, B. K. (2000). Dimensionality and disciplinary differences in personal
Brten, I., & Stromso, H. I. (2005). The relationship between epistemological beliefs,
epistemology. Contemporary Educational Psychology, 25, 378405.
implicit theories of intelligence, and self-regulated learning among Norwegian
Hofer, B. K. (2004). Epistemological understanding as a metacognitive process:
postsecondary students. British Journal of Educational Psychology, 75, 539565.
Thinking aloud during online searching. Educational Psychologist, 39, 4355.
Buehl, M. M., & Alexander, P. A. (2005). Motivation and performance differences in
Hofer, B. K. (2006). Beliefs about knowledge and knowing: Integrating domain
students domain-specic epistemological belief proles. American Education
specicity and domain generality: A response to Muis, Bendixen, and Haerle
Research Journal, 42, 697726.
(2006). Educational Psychology Review, 18(1), 6776.
Buehl, M. M., Alexander, P. A., & Murphy, P. K. (2002). Beliefs about schooled
Hofer, B. K., & Bendixen, L. D. (2012). Personal epistemology: Theory,
knowledge: Domain specic or domain general? Contemporary Educational
research, and future directions. In K. R. Harris, S. Graham, & T. Urdan
Psychology, 27, 415449.
(Eds.), Educational psychology handbook. Theories, constructs, and critical
Buehl, M. M. (2008). Assessing the multidimensionality of students epistemic
issues (1, pp. 225254). Washington DC: American Psychological
beliefs across diverse cultures. In M. S. Khine (Ed.), Knowing, knowledge and
beliefs: Epistemological studies across diverse cultures (pp. 65112). New York:
Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories:
Beliefs about knowledge and knowing and their relation to learning. Review of
Burr, J. E., & Hofer, B. K. (2002). Personal epistemology and theory of mind:
Educational Research, 67, 88140.
Deciphering young childrens beliefs about knowledge and knowing. New Ideas
Hofer, T., & Pintrich, P. R. (2002). Personal epistemology: The psychology of beliefs
in Psychology, 20, 199224.
about knowledge and knowing. Mahwah, New Jersey: Lawrence Erlbaum
Chandler, M. G., Boyes, M., & Ball, L. (1990). Relativism and stations of epistemic
doubt. Journal of Experimental Child Psychology, 50, 370395.
Hofer, B. K., & Sinatra, G. M. (2010). Epistemology, metacognition, and self-
Chandler, M. J., Hallett, D., & Sokol, B. W. (2002). Competing claims about competing
regulation: musings on an emerging eld. Metacognition & Learning, 5, 113120.
knowledge claims. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology:
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Rhee
The psychology of beliefs about knowledge and knowing (pp. 145168). Mahwah,
Bonney, C., et al. (2007). Cognitive processing of self-report items in educational
New Jersey: Lawrence Erlbaum Associates.
research: Do they think what we mean? Educational Psychologist, 42(3),
Chi, M. T. H. (1992). Conceptual change within and across ontological categories:
Examples from learning and discovery in science. In R. Giere (Ed.), Cognitive
King, P. M., & Kitchener, K. S. (1994). Developing reective judgment: Understanding
models of science: Minnesota studies in the philosophy of science (pp. 129160).
and promoting intellectual growth and critical thinking in adolescents and adults.
Minneapolis: University of Minnesota Press.
San Francisco: Jossey-Bass.
Chi, M. T. H. (1997). Creativity: Shifting across ontological categories exibly. In T. B.
Kitchener, R. (2002). Folk epistemology: An introduction. New Ideas in Psychology,
Ward, S. M. Smith, & J. Vaid (Eds.), Creative thought: An investigation of
20, 89105.
conceptual structures and processes (pp. 209234). Washington, DC: American
Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological
Psychological Association.
understanding. Cognitive Development, 15, 309328.
Chinn, C. A., & Brewer, W. F. (1993). The role of anomalous data in knowledge
Lampert, M. (1986). Knowing, doing, and teaching multiplication. Cognition and
acquisition: A theoretical framework and implications for science instruction.
Instruction, 3, 305342.
Review of Educational Research, 63, 149.
Leinhardt, G., Beck, I., & Stainton, C. (1994). Teaching and learning in history.
Chinn, C. A., Buckland, L. A., & Samarapungavan, A. (2011). Expanding the
Hillsdale, NJ: Lawrence Erlbaum.
dimensions of epistemic cognition: Arguments from philosophy and
Lewis, M. W., & Grimes, A. J. (1999). Metatriangulation: Building theory from
psychology. Educational Psychologist, 46(3), 141167.
multiple paradigms. Academy of Management Review, 24(4), 672690.
Clarebout, G., Elen, J., Luyten, L., & Bamps, H. (2001). Assessing epistemological
Madill, A., & Gough, B. (2008). Qualitative research and its place in psychological
beliefs: Schommers questionnaire revisited. Educational Research and
science. Psychological Methods, 13(3), 254271.
Evaluation, 7, 5377.
Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action:
Conley, A. M., Pintrich, P. R., Vekiri, I., & Harrison, D. (2004). Changes in
Spontaneous reections about knowledge and knowing during online
epistemological beliefs in elementary science students. Contemporary
information searching and their inuence on learning. Learning and
Educational Psychology, 29, 186204.
Instruction, 21(1), 137151.
Creswell, J. W. (2008). Educational research: Planning, conducting, and evaluating
Mason, L., Boldrin, A., & Ariasi, N. (2010). Searching the web to learn about a
quantitative and qualitative research (3rd Ed.). Columbus, OH: Merrill Prentice
controversial topic: Are students epistemically active? Instructional Science, 38,
DeBacker, T. K., Crowson, H. M., Beesley, A. D., Thoma, S. J., & Hestevold, N. L. (2008).
Mason, L., & Scirica, F. (2006). Prediction of students argumentation skills about
The challenge of measuring epistemic beliefs: An analysis of three self-report
controversial topics by epistemological understanding. Learning and Instruction,
instruments. The Journal of Experimental Education, 76(3), 281312.
16, 492509.
Feucht, F. C., & Bendixen, L. D. (2010). Exploring similarities and differences in
Mertens, D. M. (2010). Research and evaluation in education and psychology:
personal epistemologies of U.S. and German elementary school teachers.
Integrating diversity with quantitative, qualitative, and mixed methods.
Cognition and Instruction, 28(1), 3969.
Thousand Oaks, CA: Sage Publications.
Flavell, J. H. (2004). Theory-of-mind development: Retrospect and prospect. Merill-
Muis, K. R. (2007). The role of epistemic beliefs in self-regulated learning.
Palmer Quarterly, 50, 274290.
Educational Psychologist, 42(3), 173190.
Frederiksen, N. (1984). Implications of cognitive theory for instruction in problem
Muis, K. R. (2008). Epistemic proles and self-regulated learning: Examining
solving. Review of Educational Research, 54, 407636.
relations in the context of mathematics problem solving. Contemporary
Gelman, R., & Greeno, J. G. (1989). On the nature of competence: Principles for
Educational Psychology, 33, 177208.
understanding in a domain. In L. B. Resnick (Ed.), Knowing, learning, and
28 J.A. Greene, S.B. Yu / Contemporary Educational Psychology 39 (2014) 1228

Muis, K. R., Bendixen, L. D., & Haerle, F. C. (2006). Domain-generality and domain- Schommer, M. (1990). Effects of beliefs about the nature of knowledge on
specicity in personal epistemology research: Philosophical and empirical comprehension. Journal of Educational Psychology, 82, 398504.
reections in the development of a theoretical framework. Educational Schommer, M., Crouse, A., & Rhodes, N. (1992). Epistemological beliefs and
Psychology Review, 18, 354. mathematical text comprehension: Believing it is simple does not make it so.
Murphy, P. K. (2005, November). Toward a model of topic knowledge and belief Journal of Educational Psychology, 84, 435443.
change. Paper presented at the annual conference of the Southwest Consortium for Schommer, M., & Walker, K. (1995). Are epistemological beliefs similar across
Innovative Psychology, Las Vegas, Nevada. domains? Journal of Educational Psychology, 87, 424432.
Murphy, P. K., Alexander, P. A., Greene, J. A., & Edwards, M. N. (2007). Schraw, G. (2006). Knowledge: Structures and processes. In P. Alexander & P. Winne
Epistemological threads in the fabric of conceptual change. In S. Vosniadou (Eds.), Handbook of educational psychology (2nd ed., pp. 245264). San Diego:
(Ed.), Reframing the conceptual change approach in learning and instruction Academic.
(pp. 105122). New York: Elsevier Press. Schraw, G. J., & Olafson, L. J. (2008). Assessing teachers epistemological and
Murphy, P. K., & Alexander, P. A. (2004, May). Epistemological threads in the fabric ontological worldviews. In M. S. Khine (Ed.), Knowing, knowledge and beliefs:
of conceptual change. Invited paper presented at the 4th European Symposium on Epistemological studies across diverse cultures (pp. 2544). New York: Springer.
Conceptual Change, Delphi, Greece. Tu, Y., Shih, M., & Tsai, C. (2008). Eighth graders web searching strategies and
Patton, M. Q. (2002). Qualitative evaluation and research methods (3rd ed.). Thousand outcomes: The role of task types, web experiences, and epistemological beliefs.
Oaks, CA: Sage Publications. Computers & Education, 51, 11421153.
Perry, W. G., Jr. (1968/1999). Forms of ethical and intellectual development in the VanSledright, B. (2002). Confronting historys interpretive paradox while teaching
college years: A scheme. New York: Holt, Rinehart and Winston. fth graders to investigate the past. American Educational Research Journal, 39,
Rosenberg, S., Hammer, D., & Phelan, J. (2006). Multiple epistemological coherences 10891115.
in an eighth-grade discussion of the rock cycle. The Journal of the Learning VanSledright, B. A., & Limn, M. (2006). Learning and teaching social studies: A
Sciences, 15(2), 261292. review of cognitive research in history and geography. In P. A. Alexander & P. H.
Samarapungavan, A., Westby, E. L., & Bodner, G. M. (2006). Contextual epistemic Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 545570).
development in science: A comparison of chemistry students and research Mahwah, NJ: Lawrence Erlbaum Associates.
chemists. Science Education, 90, 468495. Wood, P., & Kardash, C. (2002). Critical elements in the design and analysis of
Sandoval, W. A. (2005). Understanding students practical epistemologies and their studies of epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal
inuence on learning through inquiry. Science Education, 89(4), 634656. epistemology: The psychology of beliefs about knowledge and knowing
Sandoval, W. A. (2009). In defense of clarity in the study of personal epistemology. (pp. 231260). Mahwah, New Jersey: Lawrence Erlbaum Associates.
The Journal of the Learning Sciences, 18(1), 150161. Yin, R. K. (2006). Case study methods. In J. L. Green, G. Camili, & P. B. Elmore (Eds.),
Sandoval, W. A., & am, A. (2010). Elementary childrens judgments of the epistemic Handbook of complementary methods in education research (pp. 111122).
status of sources of justication. Science Education, 95(3), 383408. Mahwah, NJ: Lawrence Erlbaum Associates.