Out

ProQuest Number: 3724851
All rights reserved
INFORMATION TO ALL USERS

The quality of this reproduction is dependent upon the quality of the copy submitted.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if material had to be removed,
a note will indicate the deletion.
ProQuest 3724851
Published by ProQuest LLC (2015). Copyright of the Dissertation is held by the Author.
All rights reserved.

This work is protected against unauthorized copying under Title 17, United States Code
Microform Edition ProQuest LLC.
ProQuest LLC.
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, MI 48106 - 1346
Massive Open Online Courses: Evaluation and Usage Patterns of Residential Students in Higher
Education
By
Kristin Palmer
Abstract
This study examined UVa student behaviors and perceptions of MOOCs that were
developed by the University and partially used in residential courses from 2012-2014. Student
behaviors such as lecture viewing were examined by looking at the clickstream data. Student
perceptions such as anticipated versus actual value of watching course videos were collected
through an online survey. Comparisons were then made such as if UVa students indicated high
actual value of watching videos, did they actually watch course videos?
Based on the data examined for this study, students consistently showed an increase from
anticipated to actual value for watching course videos, taking quizzes, and posting in discussion
forums in a MOOC if the MOOC was related to their class work. Students who took quizzes in
the MOOC consistently submitted quizzes more than once. Students in courses that required
participating in the MOOC watched some lectures and completed tasks that were required for the
final grade for the residential course. None of the UVa students watched all of the lecture videos.
Students stated a desire to have convenient, ease to access materials outside of class with
increased faculty interaction during residential class time.
! ii!! !
Keywords: MOOCs, massive open online course, flipped classroom, blended learning, open
educational resources, University of Virginia, Coursera, residential student experience.
! iii!
! !
Copyright by
Kristin Palmer
2015
! iv!
! !
ACKNOWLEDGEMENTS
Thank you to the University of Virginia for providing me the opportunity to work with a wealth
of talented professors and gifted students. Thank you to my peer researchers at Coursera and
Oxford. Thank you to my committee for persevering and believing in me. Thank you to Dr. Lee
Mahon for being a wonderful person whose unfailing support and wisdom provided the
backbone to my experience with my dissertation study. Thank you to my family for keeping it
real. Thank you to my friends for being supportive. Thanks to the universe for allowing me to
be a mom to the three coolest kids in the galaxy!
! v!! !
DEDICATION
To my sons and the life we choose to lead, the paths we create and the doors we open. Work
hard, be honest and be brave.
! vi!
! !
TABLE OF CONTENTS
CHAPTER(ONE:(INTRODUCTION(..............................................................................................(1!
Introduction(.....................................................................................................................................(1!
Statement(of(the(Problem(................................................................................................................(3!
Purpose(of(the(Study(........................................................................................................................(6!
Research(Question(and(Methodology(..............................................................................................(6!
Significance(of(the(Study(..................................................................................................................(7!
CHAPTER(TWO:(REVIEW(OF(THE(LITERATURE(..........................................................................(8!
Introduction(.....................................................................................................................................(8!
Background(......................................................................................................................................(9!
Categories(of(Research(Literature(..................................................................................................(17!
Summary(.......................................................................................................................................(23!
CHAPTER(THREE:(METHODOLOGY(........................................................................................(25!
Introduction(...................................................................................................................................(25!
Theoretical(Framework(..................................................................................................................(26!
Participants(....................................................................................................................................(31!
Setting(...........................................................................................................................................(31!
Limitations(.....................................................................................................................................(34!
Summary(.......................................................................................................................................(34!
CHAPTER(FOUR:(FINDINGS(....................................................................................................(35!
Introduction(...................................................................................................................................(35!
Data(Set(1:(Quantitative(Data(from(Clickstream(Data(on(UVa(Student(Course(Behaviors(................(35!
Data(Set(2:(Online(Survey(Collecting(UVa(Student(Perception(........................................................(54!
Summary(.......................................................................................................................................(73!
CHAPTER(FIVE:(DISCUSSION(AND(RECOMMENDATIONS(.......................................................(76!
Discussion(......................................................................................................................................(76!
Limitations(.....................................................................................................................................(85!
Summary(.......................................................................................................................................(87!
Recommendations(.........................................................................................................................(89!
REFERENCES(.........................................................................................................................(91!
(
! vii!
! !
LIST OF FIGURES AND TABLES
Figure 1: MindWires illustration showing the two branches of MOOCs cMOOCs and
xMOOCs ......................................................................................................................................... 5!
Figure 2: Illustration by Ravi Chandran illustrating evolving trends in education circa
June 2012 ...................................................................................................................................... 12!
Table 1: Different Perceptions of MOOCs as Reported in The Maturing of the MOOC, Literature
Review of Massive Open Online Courses and Other Forms of Online Distance Learning
(Department for Business Innovation and Skills, 2013) ............................................................... 13!
Table 2: Table of Different MOOC Platforms, the Partners for Each of the Platforms as of April
2014............................................................................................................................................... 15!
Figure 3: Venn diagram illustrating the intersection of higher education, technology, and
society ........................................................................................................................................... 28!
Table 3: Illustration of Timeline for Process Steps for Research Approach. ............................... 29!
Table 4: Table of Clickstream Data for Internal UVa Students in The Modern World MOOC... 38!
Table 5: Table of Clickstream Data for The Kennedy Half Century MOOC ............................... 40!
Table 6: Table of Clickstream Data for Effective Classroom Interactions: Supporting Young
Childrens Development MOOC................................................................................................... 42!
Table 7: Table of Clickstream Data for The Modern World Session 1 MOOC ........................ 44!
Table 8: Table of Clickstream Data for The Modern World Session 2 MOOC ........................ 46!
Table 9: Table of Clickstream Data for Plagues, Witches, and War: The Worlds of Historical
Fiction MOOC .............................................................................................................................. 48!
Table 10: Table of Clickstream Data for Design Thinking for Business Innovation MOOC....... 50!
Table 11: Table of Clickstream Data for Foundations of Business Strategy MOOC .................. 52!
! viii!
! !
Table 12: UVa Behaviors for Video Viewing, Video Downloading, Quiz Taking, and Discussion
Forum Posting in MOOCs that were Not Required...................................................................... 53!
Table 13: UVa Behaviors for Video Viewing, Video Downloading, Quiz Taking, and Discussion
Forum Posting in MOOCs that were Not Required as Compared to Non-UVa Student
Learners ........................................................................................................................................ 53!
Table 14: Responses from Open-Ended Questions in the Online Survey Completed by UVa
Students ......................................................................................................................................... 56!
Table 15: UVa Student Responses to Question of Interest in Flipped Classroom Approach where
MOOC was Optional (n=13) ........................................................................................................ 57!
Table 16: UVa Student Responses to Question of Anticipated Value and Actual Value of Videos
where MOOC was Optional (n=13) ............................................................................................. 58!
Table 17: UVa Student Responses to Question of Anticipated Value and Actual Value of Quizzes
where MOOC was Optional (n=13) ............................................................................................. 59!
Table 18: UVa Student Responses to Question of Anticipated Value and Actual Value of
Discussion Forums where MOOC was Optional (n=13) ............................................................. 60!
Table 19: UVa Student Responses to Question of Interest in Flipped Classroom Approach where
MOOC was Required (n=22) ....................................................................................................... 62!
where MOOC was Required (n=22) ............................................................................................. 63!
where MOOC was Required (n=22) ............................................................................................. 64!
Discussion Forums where MOOC was Required ......................................................................... 65!
! ix!
! !
Table 23: UVa Student Responses to Question of Interest in Flipped Classroom Approach when
MOOC was Not Related to Student Course Work (n=14) ............................................................ 69!
when MOOC was Not Related to Student Course Work (n=14) .................................................. 69!
when MOOC was Not Related to Student Course Work (n=11) .................................................. 70!
Discussion Forums when MOOC was Not Related to Student Course Work (n=14) .................. 71!
! x!! !
LIST OF APPENDICES
Appendix A: Approval from Archie Holmes to use UVa Coursera data for study...98
Appendix B: Online Survey for all UVa students that participated in a UVa MOOC..99
Appendix C: Email request to students to collect feedback via online survey ...103
! xi!
! !
CHAPTER ONE
Introduction
Many say higher education is in crisis. This crisis revolves around cost, quality of
the education received, and access. Some factors influence this crisis: The first
generation of digital natives now completing university (Johnson, 2014); the Internet
providing a wealth of high quality open educational resources (UNESCO, 2015); smart
devices are ubiquitous with over 4.1 billion mobile device users in 2014 (Guglielmo,
2014); and employees continuously learn new skills (Hess, 2014). Seventy-five (75%)
percent of the higher education landscape today is comprised of non-traditional students
(Hansen, 2014). It is a fact that there is a physical gap in the ability to provide
educational resources such as teachers, buildings, and books to a planet of over 7 billion
inhabitants (UNICEF, 2015; Worldometers, 2015).
The cost, quality, and access argument is reported in the education industry periodicals
on a regular basis, where the cost of education is rising faster than inflation. Bowen (2012) put it
succinctly stating, the cost per student rose appreciably faster than an economy-wide index of
costs in general (p. 3). This cost increase leads to increased student loan debt upon graduation
from a 4-year college, a debt that is currently between $25,000-$40,000 on average depending on
institution (Institute for College Access & Success, 2014).
One of the reasons cost is so important is the shift in income potential for a college
graduate versus a high school graduate. Among individuals 25 to 32 years old, the college-
degree holders make an average of $17,500 more than the median annual earnings of individuals
with only high school diplomas (Kurtzleben, 2014). In order for students to have the higher
!
2
earning potential, students need to graduate.
Not only do students need to graduate with the degree, they need to graduate in 4 years.
The 2012 graduation rate indicated that full-time undergraduate students who began their pursuit
of a bachelors degree at a 4-year degree-granting institution in fall 2006 amounted to 59% (U.S.
Department of Education, 2014). Students not only do not graduate in 4 years, they also appear
to not have the job skills employers require. In 2013, approximately 1.5 million (53.6%)
individuals under 25 years of age with a bachelors degree were jobless or underemployed
(United States Census Bureau, 2013). Other studies found that students just do not have the
skills required by employers (Weiner, 2013). Some argue the lack of quality is related to a
fundamental mismatch of students to institutions, pointing out that if students chose institutions
more aligned with their interests the 4-year graduation rate would increase (Allen & Van Der
Velden, 2001). Regardless, underemployed or unemployed students are likely to default on
student loans, thus the quality issue escalates the cost issue creating a potential financial disaster
threatening our society at large. We witnessed this in the $1.2 trillion in student loan debt in
2012 (Denhart, 2013).
The last component of the crisis is access. The ability to graduate requires
students to be able to take the classes necessary to graduate. There is a severe capacity
crunch when budgets are cut and staff is reduced resulting in students unable to register
for gateway courses (Fain, 2013). Outside of the United States, there are larger issues
around access as there are not enough physical resources to build and staff schools.
Students in Africa have died fighting their peers to register for the few open slots at
university (Koller, 2012).
!
3
Statement of the Problem
The crisis in education around cost, quality, and access has led administrators,
faculty, and researchers to investigate alternative options in instructional content and
delivery. Online learning is one possible solution to this crisis. It provides a scalable
virtual classroom driving down cost per student while providing the possibility of a
robust learning environment with the equivalent quality as face to face (Ni, 2013).
Massive open online courses (MOOCs) might be a potential entry solution in online
learning for institutions. MOOCs require little institutional investment or resources. To
date, there has been little research on residential student experiences with MOOCs.
Institutions have been investigating the infrastructure, process, quality controls,
and pedagogy of online learning. MOOCs were introduced as a third-party online
learning solution where institutions could produce online educational content without a
large investment. Since MOOCs were free and open to the world, they did not have the
same limitations or expectations of online educational content provided to for-credit, for-
cost, residential students. MOOCs were a low barrier to first-step entry into the world of
online education with the added benefit of increasing brand recognition and being
innovative (Downes, 2008).
MOOCs were first introduced in 2008 and exploded into media attention in 2012
when dozens of high profile universities joined MOOC platform providers such as
Coursera, edX, and NovoEd. MOOCs reached hundreds of thousands of participants and
provided educational content for free. The MOOCs created in 2008 by Downes and
Cromier became known as connectivist MOOCs or cMOOCs and typically enrolled less
than 200 students. The focus in cMOOCs was students producing content and
!
4
encouraging community interaction facilitated by an instructor. This content then
became the content of the course with the process of creating the content being the
method of instruction. Figure 1 shows the development of different MOOC providers,
types of MOOCs (cMOOCs and xMOOCs) over time starting in 2008.
MOOCs created in 2012 were mostly large enrollment with the first Artificial
Intelligence course offered by Sebastian Thrun, educator, programmer, robotics
developer, and computer scientist at Stanford University. His first online course on
Artificial Intelligence drew 160,000 students from around the world, including those
already taking the course in-person (Morrison, 2013).
These large enrollment MOOCs were differentiated from cMOOCs in the
academic community by naming them xMOOCs. These xMOOCs initially had the sage
on the stage videotaped for streaming the lecture content to the masses. The focus of the
MOOCs was the instructors producing the content and the participant passively
consuming it. There is a difference in pedagogy between cMOOCs (connectivist where
learners create the content within their classroom networks) and xMOOCs (sage on stage
delivers content; Morrison, 2013). Despite different approaches to pedagogy, the fact is
that millions of participants around the world were engaged at some level with MOOCs.
Institutions producing MOOCs were getting experience in online learning.
!
5
Figure 1. MindWires illustration showing the two branches of MOOCs cMOOCs and
xMOOCs.
Since 2012, there has been a frenzy of media and research trying to understand
the pedagogical effectiveness of MOOCs (xMOOCs and cMOOCs), participant types,
participation trends, student motivation, and student retention. There has not been
research to date on residential student experiences with MOOCs. Research of behavior
trends and perceived value by residential students can inform if, how, why, when, and
where institutions might gain value in utilizing MOOCs on campus to decrease costs,
increase quality, and/or improve access. Having research data on the experience and
behavioral trends of residential students can inform institutional strategies for online
learning. Unfortunately, there is little if any research to date on the behavioral patterns in
MOOCs and/or the perceived value of MOOCs for residential higher education students.
Research data that will define the behavioral interactions and perceived value of MOOCs
!
6
can inform online learning strategies. Data on residential students in online learning
programs may inform solutions for the cost, quality, and access in higher education.
Purpose of the Study
The purpose of this study was to research and gather data on behavior patterns and
perceived value of MOOCs to residential students in higher education at a public university.
MOOCs, course materials, and students in this study are at the University of Virginia (UVa).
MOOCs in this study are hosted on the Coursera platform. Data provided included course data,
behavioral data, and perceived value. There are approximately 1,500 (name)@virginia.edu
participants in UVa MOOCs from 2012-2014. Based on data from the Office of Institutional
Assessment and Studies at UVa, 445 of these email addresses are known to be current residential
students. Archival data from January 2013 through December 2014 were collected for this
research.
Research Question and Methodology
My research question is, What are the usage patterns and perceived value of MOOCs for
residential higher education students at UVa? This was a mixed methods process using
quantitative methods.
Data from this study were gathered to assess students using MOOCs to flip the classroom
at UVa. The concept of flipping the classroom is changing the instructional approach so that
instruction that was only available in class is now accessed at home before class. For this study,
MOOC lecture videos are viewed before class and the classroom becomes the place to work
through problems, advance concepts, and engage in collaborative learning (Tucker, 2012).
Quantitative research methods were used to analyze archival data for each UVa MOOC.
These data included the clickstream and behavioral data of participants on the Coursera platform.
!
7
The data were filtered and examined as to the UVa student behaviors in UVa MOOCs. I
analyzed the data by looking at data for participants with a (name)@virginia.edu email address.
My study included archival data from 2012-2014 classes representing 13 humanities, education,
and business courses.
Quantitative and qualitative research methods were used to gather data on student
perceptions of the MOOC. The data source was an online survey to all (name)@virginia.edu
participants in UVa MOOCs. The online survey asked students to rate their experience with the
course materials and included open-ended questions around interest in participating in a flipped
classroom including the effectiveness of MOOCs materials.
Significance of the Study
Online learning is seen as a potential solution to many issues in higher education around
cost, quality, and access. MOOCs are a method of producing online educational content. My
study presents data on behavioral patterns and perceptions as to the value of MOOCs for
residential higher education students at UVa. This study is the first to collect behavioral and
perception data from residential students in MOOCs. It is intended to inform future use of
MOOCs and online learning solutions for residential higher education students and to understand
the current student perception of flipped classrooms for instruction. It also may be of use in
understanding how to influence cost, quality, and/or access to higher education.
!
8
CHAPTER TWO
Review of the Literature
Introduction
Online learning is not new with a long history of research in pedagogy for online
learning, distance education, and correspondence courses to deliver educational materials to
students (Casey, 2008). All of the research materials on these technology interventions have
been around for over fifty years. Technologies have developed since correspondence courses
ranging from radio, projection systems, television, home computers, the Internet, and mobile
smart devices. The Internet has been known to create disruption in other industries such as
publishing and media and has positioned itself to disrupt education (Christensen, Johnson, &
Horn, 2008). Pundits range from claiming that the world will only have 10 universities (Leckart,
2012) to saying that MOOCs are a failure and that they will have no effect on residential
education in America (Lucas, 2013).
One of the technology disruptions in education is known as a massive open online
course, commonly referred to as a MOOC. The term MOOC was coined in 2008 to describe an
open online course at the University of Manitoba in Canada. A range of both topics and
platforms has since emerged, and the term was described as the educational buzzword of 2012
by Daniel (2012). Reflecting widespread interest in the concept, 2013 was deemed the year of
the MOOC. MOOCs are discussed across a range of media, including journals, newspapers,
social media, blogs, and industry reports (Pappano, 2012).
!
9
Background
A cornerstone of academia is creating new knowledge. This knowledge is then shared in
what is increasingly an open environment through the Internet (Lane & McAndrew, 2010).
Creating open educational resources (OER) is an emerging trend that is evolving as academics
realize that they have the ability to communicate freely around the world and provide new
educational content to a worldwide audience. This differs from the paradigm of the 18th-20th
centuries of using a publishing company to disseminate new knowledge and scholarly
publications. OER has become an important resource for educators and students.
In 2001, the Massachusetts Institute of Technology (MIT) launched Open Course Ware
(OCW) to publish materials from courses in the World Wide Web, with license to use, modify,
or redistribute the materials. Since 2001, there have been other notable efforts to generate OER
from institutions, including the OpenLearn project at the Open University in the UK (2008-
current), the Open Learning Initiative by Carnegie Mellon University (2001-current), and the
Fathom Archive out of Columbia University (2000-2003). These projects generally have been
considered a failure due to the lack of content use and reuse and the large investments made with
questionable return on investment (Kortemeyer, 2013). Some view a subset of these programs,
such as MITs engineering program, as successful because of the value of establishing brand and
dominance in domain knowledge in the global community.
OER was created in the traditional paradigm of courses, during semesters, within a
university. Although OER has been around for over a decade, no standards for categorizing and
archiving materials in discreet chunks or segments have been developed. This discreet segment
approach has been referred to as textbooks 2.0 (Rawlinson, 2007). This view of the OER as
!
10
discreet chunks is appealing for its ability to become part of a larger course as supplemental
course materials. The concept is that educators can use these OER as part of a traditional course
or as an independent learner focusing on gaining one skill or competency. The concept of open
access to learning was taken in different directions with the introduction of the massive open
online courses or MOOCs in 2008 (Fini, 2009).
MOOCs bring together participants anywhere in the world that have an Internet
connection with an expert or experts who seek to provide an educational experience. In
cMOOCs, connectivity is provided through social networking with a set of freely accessible
online resources providing the main lesson materials. There tend to be no prerequisites, fees,
formal accreditation, or predefined required level of participation. Participation in a MOOC is
completely voluntary, with participants entering and exiting the MOOC at will throughout the
duration of its availability. Some universities keep OER such as MOOC content available at all
times, while others run MOOC sessions for a limited time and might offer future sessions where
the content can be modified.
Connectivism and Connective Knowledge, an online course offered through the
Learning Technologies Centre and Extended Education at the University of Manitoba and
facilitated by George Siemens and Stephen Downes (Liyanagunawardena, Adams & Williams,
2013), was considered the first MOOC. This online course had 25 paid enrollments (for credit),
with around 2,200 non-credit, non-fee-paying students. It used the principles of connectivism
(Siemens, 2005), unlike the traditional form of online learning. According to the CCK11 class
website (http://cck11.mooc.ca/how.htm), the course did not consist of a body of content one is
!
11
supposed to remember. Rather, the learning in the course results from the activities undertaken,
will be different for each person.
In 2012, professors at Stanford University created several for-profit MOOC companies:
Coursera, NovoEd, and Udacity. Also in 2012, professors at Harvard and Massachusetts
Institute of Technology teamed up to launch the not-for-profit MOOC provider edX. Although
most of media attention in 2012-2013 was on the development of MOOC platforms in North
America, several localized platforms were being developed internationally: Open2Study, Schoo,
Veduca, FutureLearn, iVersity, Udemy, and JooMoo. Some of these platforms, namely Schoo
from Japan and JooMoo from Spain, have already gone out of business.
In 2012-2013, MOOCs were a popular topic in media. Stories ranged from interviews
with MOOC professors and students, articles reporting usage and providers, to articles
speculating on the effect and future of higher education. Academic papers first appeared in 2008,
with more papers being produced annually. This trend is similar to what has happened with
other technology developments, such as Twitter, within the academic community. MOOCs also
started appearing in industry reports, such as the Horizon Report and the Gallup Poll in 2012.
It is important to contextualize the MOOC technology with the overall evolution of
society. Figure 2 by Ravi Chandran from June 2012 captures many societal trends; mobile
devices, learning analytics, crowd sourcing eBooks, rich interfaces, personal learning
environments, flipped classrooms, and free educational resources. MOOCs are considered to be
free or open educational resources. OER content can be delivered through mobile devices,
personalized based on engagement, crowd sourced for content, and leveraged for providing
learning and teaching analytics.
!
12
Figure 2. Illustration by Ravi Chandran illustrating evolving trends in education circa June
2012
MOOCs in Literature and Media.
For tracking the media on MOOCs, I used several databases and services that were
available through Fielding Graduate University: Elsevier Social & Behavioral Sciences College
Edition (ScienceDirect.com), ERIC, Google Scholar, and ProQuest. I used the advanced search
feature and searched abstracts, titles, and keywords for MOOC and massive open online
course. Tables included in the appendix show a clear increase in the occurrence of these
phrases across all search engines starting in 2012. The density of article types appeared in trade
and scholarly journals with the mass media outlets of newspapers and magazines close behind.
Another piece of research, The Maturing of MOOCs by the Haggard, Brown, Mills, Tait,
Warburton, Lawton, & Angelo (2013), captures some of the key messages from the leading
industry reports in a succinct and humorous way (see Table 1).
!
13
Table 1
Different Perceptions of MOOCs as Reported in The Maturing of the MOOC, Literature Review
of Massive Open Online Courses and Other Forms of Online Distance Learning (Haggard et al.,
2013)
Report In a sentence: MOOCs.
mean learners will take control, a victory for

EIU
democracy.
are interesting, but most people wont rush into

Sloan Consortium 2012
them.
will disrupt international demand for national

Austrade: More than MOOCs
education product.
Canadas SSRC (The McAuley can transform the national digital economy, but
Report/The MOOC Model) may exclude many learners
JISC-CETIS: Implications for HE are just the next step in online learning technology.
IPPR/Pearson: An Avalanche is are already disrupting and transforming HE: Stand

Coming by for mayhem!
Education Advisory Board are forcing universities to use new pedagogies.
UniversitiesUK: The MOOC may help HE reform, but think hard before
Moment producing one.
Opinion of Provosts is that MOOCs threaten HE

Survey of Provosts
institutions and arent what universities need most.
Opinion of Presidents is that MOOCs might help to

Gallup Survey of U.S. College
improve pedagogy and reputation, but not much
Presidents
else.
Platforms and Institutional Partnerships
Table 2 shows several platforms, including when they were founded, number of courses,
number of registered users, partners (if applicable), and general notes. The information in this
table is current as of April 1, 2014. From the table, patterns can be seen regarding the business
!
14
strategy of different platforms. For example, you can see that Coursera has the most partners.
EdX has many partners, most of which are international. Udacity partners both with institutions
and individual instructors. NovoEd has some partners but is focusing more on running small,
closed course instances of the content for credit and/or cost. In most of these partners, the
content is hosted on the platform but owned by the instructor or institution. Additionally in most
of these platforms, the relationship with the student is owned by the platform. For example,
participants on Coursera frequently state they love the Coursera course, not that they love the
University of Virginia course on Coursera. Listed below are the platforms, the partners, and
notes about the offerings:
!
15
Table 2
Table of Different MOOC Platforms, the Partners for Each of the Platforms as of April 2014
Platform Partners
Coursera American Museum of Natural History, Berklee College of Music, Brown,

Founded 2012 California Institute of Technology, California Institute of the Arts, Case
California, USA Western Reserve University, Columbia, Commonwealth Education Trust,
623 courses Copenhagen Business School, Curtis Institute of Music, Duke, Ecole Centrale
7.1+ million users Paris, Ecole Normale Superieure, Ecole Polytechnique, Ecole Polytechnique
Federale de Lausanne, Eindhoven University of Technology, Emory
University, Exploratorium, Fudan University, Georgia Tech, Hebrew
University of Jerusalem, HEC Paris, Higher School of Economics, Icahn
School of Medicine at Mount Sinai, IE Business School, IESE Business
School, John Hopkins, Koc University, Korea Advanced Institute of Science
and Technology, Ludwig-Maximilians-Universitat Muchen, Match Teacher
Residency, Moscow Institute of Physics and Technology, Nanyang
Technological University in Singapore, National Geographic Society, National
Taiwan University, National University of Singapore, New Teacher Center,
Northwestern, Peking University, Princeton, Relay Graduate School of
Education, Rice, Rutgers, Saint Petersburg State University, Sapienza
University of Rome, Shanghai Jiao Tong University, Stanford, Technical
University of Denmark, Technion Israel Institute of Technology, Technische
Universitat Muchen, Tecnologico de Monterrey, Tel Aviv University, Chinese
University of Hong Kong, Hong Kong University of Science and Technology,
Museum of Modern Art, Ohio State, Pennsylvania State University, University
of British Columbia, University of Chicago, University of Edinburgh,
University of Melbourne, University of North Carolina at Chapel Hill,
University of Tokyo, The World Bank, Universidad Nacional Autonoma de
Mexico, Universita Bocconi, Universitat Autonoma de Barcelona, Universiteit
Leiden, University of Alberta, University of Amsterdam, UC Irvine, UC San
Diego, UC Santa Cruz, University of Colorado Boulder, University of
Copenhagen, University of Florida, University of Geneva, University of
Illinois at Urbana-Champaign, University of Lausanne, University of London,
University of Manchester, University of Maryland in College Park, University
of Michigan, University of Minnesota, University of Pennsylvania, University
of Pittsburgh, University of Rochester, University of Toronto, University of
Virginia, University of Washington, University of Western Australia,
University of Wisconsin-Madison, University of Zurich, UNSW Australia,
Vanderbilt, Wesleyan, Yale, University of Kentucky, University of Nebraska,
!
16
Platform Partners
University of New Mexico, University of Tennessee, University of Georgia,

West Virginia University, State University of New York, Tennessee Board of
Regents, University of Colorado System, and University of Houston
NovoEd Carnegie Foundation for the Advancement of Teaching, Babson Global Inc,
Founded 2012 Acumen, SRI International, Kauffman Fellows Academy, Decision Education
California, USA Foundation, Entrepreneurship Center at UCSF, University of Virginia Darden
32 courses School of Business, Princeton, Strategic Decisions Group, Pontificia
Universidad Catolica de Chile, Wharton at University of Pennsylvania, and
University of Maryland
Udacity San Jose State University, Georgia Tech

Founded 2011 Udacity partners with individual instructors for course creation vs. institutions.
California, USA
37 courses
1.6+ million users
edX MIT, Harvard, Berkeley, University of Texas, Boston University, TU Delft,

Founded 2012 University of Queensland, Berklee College of Music, Caltech, Columbia
Massachusetts, University, Cornell, Dartmouth, Davidson, Ecole Polytechnique Federale de
USA Lausanne, ETH Zurich, Australian National University, Georgetown, Hong
175 courses Kong University of Science and Technology, IIT Bombay, Karolinska
2.1 million users Institutet, Kyoto University, McGill, Peking Univesrity, Rice, Seoul National
University, Technische Universitat Muchen, Tsinghua University, Universite
Catholique de Louvain, University of Chicago, University of Hong Kong,
University of Tokyo, University of Toronto, University of Washington, and
Wellesley
Udemy Udemy partners with individuals vs. institutions.

Founded 2010
California, USA
60 courses
1+ million users
Open2Study Massey University of New Zealand, Griffith, Macquarie, RMIT, and

Founded 2013 Australian University partners in the Open Learning Consortium
Australia
49 courses Note: 154,528 students from 217 countries, with 304,279 enrollments
watching 1,025,085 videos over 4,130,916 minutes across 49 courses as per
!
17
Platform Partners
Opn2Study community dashboard on April 10, 2014
Veduca Berkeley, Columbia, Google, Harvard, University of Michigan, MIT, NYU,

Founded 2011 `Oxford, Princeton, Stanford, TED, UCLA, Unicamp,Unesp, UnB, UFSC,
Brazil UNSW, USP, and Yale
32,000 users
FutureLearn University of Auckland in New Zealand, University of Bath, University of

Birmingham, University of Bristol, British Council, British Library, University
Founded 2012 of Edinburgh, University of Exeter, University of Glasgow, King's College
England London, Lancaster University, University of Leeds, University of Leicester,
University of Liverpool, Loughborough University, Monash University,
36 courses Newcastle University, University of Nottingham, The Open University,
Queen's University Belfast, University of Reading, University of Sheffield,
200,000+ users
University of Southampton, University of Strathclyde Glasgow, Trinity
College Dublin, and University of Warwick
iVersity iVersity partners with individuals vs. institutions.

Founded 2013
Germany
31 courses
500,000+ users
Categories of Research Literature
The research in MOOCs largely falls into five categories: pedagogy and learning
outcomes, learner types, retention, best practices, and rationale for MOOCs. The initial literature
at the advent of MOOCs focused on pedagogy with the introduction of connectivism as a
framework for learning by George Siemens (2005). David Cromier introduced the concept of
the massive open online course or MOOC in 2008 (Morrison, 2013). As MOOCs became
popular, researchers began writing about learning outcomes related to pedagogical approach.
How did the learning outcomes compare between cMOOCs, xMOOCs, and the face-to-face
!
18
classroom. Researchers such as Glance, Forsey, and Riley (2013) argued that MOOCs can be
just as effective for learning as face-to-face in some cases and more effective if using formative
quizzes, retrieval practice, short videos, and discussion forums for teacher-student and student-
student interactions (Glance et al., 2013).
Other researchers such as Kop, Fournier, and Mak (2011) wrote about the pedagogy of
abundance and moving from data scarcity to abundance (Weller, 2011). Kop, Fournier, and Mak
looked at the behavior changes from receiving information to just-in-time learning from
networks (Boyd, 2010) and the role of MOOCs for creating that networked knowledge. They
outlined the four main types of activity in connective MOOCs: aggregation, remixing,
repurposing, and feeding forward and the value of creating shared knowledge (Kop et al., 2011).
Some of the research on pedagogy worked with specific student populations. Saadatmand and
Kumpulainen (2014) looked at the participant perceptions in cMOOCs. In this research, the
theoretical background of MOOCs was reviewed (connectivist versus sage on the sage),
students usage of tools documented, an online survey completed by 65 learners, and 12
interviews were conducted with virtual chat tools. The results from this research identified that
participants need to develop self-organization, self-motivation, and have a reasonable amount of
technical proficiency to manage the abundance of resources and the open format.
Colvin, Champaign, Liu, Zhou, Fredericks, and Pritchard (2014) wrote an article
comparing learning on campus with learning in the MOOC. This study was done with the MIT
8.M Rev Mechanics Review class, which is required for first-year MIT students. The study used
pre and post testing with item response theory (IRT) to evaluate what learning was actually
happening in a MOOC. In the study, they had a cohort of MIT students, making it one of the
first published studies including residential college student behavior in MOOCs. The authors
!
19
concluded that in spite of extra instruction the on-campus students had there was no evidence of
positive, weekly relative improvement of on-campus students compared with the online students
(Colvin et al., p. 9).
A study published in 2013 by Breslow, Pritchard, DeBoer, Stump, Ho, and Seaton looked
at students at MIT, behavior patterns, and learner types to understand how MOOCs contribute to
on-campus teaching and learning. In this study, there are some startling statistics. For example,
in the MIT MOOC on edX, only 3% of the learners participated in discussion forums but 52% of
certificate earners used the discussion forum. Fully 88% of learners were male. Other large
contributions from this research included looking at the usage and completion data by type of
learner. For this study, they used all registrants, all students who clicked on the course website,
and students who demonstrated different levels of engagement with the course and certificate
earners. They found a correlation between students previous course experience and math and
the achievement in the MOOC. While the study included data from 200 students at MIT,
findings were not reported out using the higher education residential student as a category.
Other studies work to establish who is participating in MOOCs, defining learner types.
The University of Pennsylvania (PENN) team from the Provost office published an influential
article looking at a survey across 32 PENN MOOCs in the fall of 2012. The survey showed that
the population was young, well educated, employed, and from developed countries (Christensen,
Steinmetz, Alcorn, Bennett, Woods, & Emanuel, 2013). Of the 34,779 respondents, most were
male and primarily interested in advancing in their current job. Eighty-three percent had a post-
secondary degree, 40% were under 30 years old, and they found that reasons for enrolling varied
by country and course type.
!
20
Anderson, Huttenlocher, Kleinberg, and Leskovec (2014) conducted a study to identify
who were taking MOOCs and why. They outlined engagement taxonomies and looked at click
stream data from Stanford MOOCs. They categorized learners with the intent of trying to
improve online learning by understanding who was learning online. The categories included
viewers, solvers, all-rounders, collectors and bystanders. Viewers watched lectures, handing in
few assignments. Solvers handed in assignments but viewed few lectures. All-rounders
balanced watching of lectures with handing in assignments. Collectors downloaded lectures but
did not turn in assignments. Bystanders registered for the course but did not engage with the
content. The paper concluded that high achievers typically consumed many lectures. The study
included using badges to incentivize participants to view content, posting in forums, and to vote
in forums. The researchers found that subtle differences such as introducing badges or
displaying student progress improved learner engagement (Anderson et al., 2014).
Another category of MOOC research examines retention. There is abundant media
calling MOOCs failures (Lewin, 2013). This is hard to refute coming from a paradigm of higher
education where student retention and graduation rates are viewed as critical to success. In a
paradigm of for-credit, for-cost, we want all students to finish their courses completely and
graduate. MOOCs in 2012-2103 found roughly 3% of enrolled students successfully completed
the course.
In an article published by Kizilcec, Piech, and Schneider (2013), the authors looked at
different learner types to understand how to design MOOCs to improve retention. In this article,
the authors created the following learner types: completing, auditing, disengaging, and sampling.
Completing learners completed the majority of assessments. Auditors did assessments
infrequently but watched the video content. Disengaging leaners completed assessments at the
!
21
beginning of the course and then disappeared. Sampling learners watched the videos and did not
complete any assessments. The study found that most participants were male, most learners
were participating as lifelong learners, and completing learners had a higher overall experience
than other groups. The authors proposed designing simple cognitive tools such as integrated
note-taking, task lists, calendars, and concept mapping tools to help support the learners.
A different approach to understanding retention in MOOCs was taken by Clow in 2013.
Clow presented a funnel of participation in MOOCs. He describes the history of open resources
and the comparison of cMOOCs with xMOOCs as described in the background section of this
study. He discussed the use of the funnel of participation as a tool for understanding drop out in
MOOCs. The proposed funnel includes the stages of awareness, registration, activity, and
progress and is used widespread in marketing and sales domains.
One of the last main areas of research to date has been around best practices and value of
MOOCs. This category includes research around things you should know about MOOCS
(Newman & Oh, 2014), MOOC expectations and reality (Hollands & Tirthali, 2014), patterns
that are emerging in MOOCS (Macleod, Haywood, Woodgate, & Alkhatnai, 2015), and rules for
engagement (Trumbore, 2014). Kolowich (2014) identified five things that researchers have
discovered about MOOCS. These factors included at-risk students are less likely to complete
MOOCs, discussion forums are rarely used, and it is not conclusive if MOOCs will help
underprivileged learners succeed in college. Papers and studies in this category present a range
of findings, sometimes contradictory. Kolowich found that learners in the Stanford MOOCs had
a strong growth mindset, those at the University of Michigan were focused on increasing their
economic mobility, learners at Duke University did better in learning gains if they contributed
!
22
on the discussion forums, and at the University of California, Irvine underprepared students
responded to incentives such as early transfer into majors.
Newman and Oh (2014) examined the data released by Harvard and MIT in the spring of
2014. These data showed that most learners were male, had degrees, were in their mid-20s and
were mostly located in North America. Newman and Oh found that learners with a doctorate
viewed the most content and that serial students and students without a high school degree were
the most engaged. He proposed to look into different motivations and goals of learners for future
research.
Trumbore (2014) looked at engagement and motivation. The focus was on developing
opportunities for student collaboration, providing cohesive open-ended assignments, and
building peer learning communities. Her study found open-ended projects, such as strategic
analysis, decision trees, design solutions, detailed business plans, and visual arguments based on
data analysis with course work, led to a synthesis of course content and improved learner
engagement. The data also showed the value of formal and informal feedback loops and
working on teams. For learners working on teams, 68% rated them as valuable. Learners cited
that peer grading provided an opportunity to act like experts in a topic and led to deeper
understanding of the material. The culminating capstone project that was synthesized from
multiple assignments in class and built something meaningful outside of class may have helped
learners persist even in the absence of traditional modes of motivation.
Hollands and Tirthali (2014) interviewed different organizations to understand the value
they were getting from MOOCs compared to the associated costs. In their 159-page report, some
of the findings related to setting goals for MOOCs at an institution. The reported goals included
(a) extending the reach of the institution and access to education, (b) building and maintaining
!
23
brand, (c) improving economics by lower costs or increasing revenues, (d) improving
educational outcomes for both MOOC participants and on-campus students, (e) innovation in
teaching and learning, and (f) conducting research on teaching and learning. In the study, the
reported cost for MOOCs ranged from $39,000 to $325,300. The cost associated with MOOC
development included video production, instructor, instructional design support, technical
support for participants, programming and special features, and analysis of platform data.
Hollands and Tirthali (2014) work looked at the future of higher education. They
discussed the unbundling of educational services, more choice for students in how their content
is delivered, greater price competition, greater use of technology in education, more just-in-time
training, competency-based credentials, and emphasis on demonstrating capabilities needed by
employers. The authors recommended experiments with blended and hybrid delivery models on
campus, to help struggling students find low-risk options to build skills and work towards the
development of an adaptive learning environment. For administrators, the authors recommend
creating a strategy for engagement and standard processes for production and adoption of
MOOCs. They recommend measuring and tracking metrics and checking these data with student
recruitment, faculty recruitment and retention, grant revenues, and generosity of donors. For
faculty, the authors recommend using evidence-based best practices in online learning, using pre
and post assessment of skills, and developing metrics to assess gain in cognitive skills.
Summary
The advent of massive open online courses has led to a new field of study for researchers.
Popular themes in the research include who is taking MOOCs, why are they taking MOOCs,
how can MOOCs be used to make classroom teaching more effective, and where are MOOCs
best utilized. The expectation of many is that the research over the next couple of years will
!
24
focus on learning analytics, designs for improving course design based on learning analytics, and
adaptive learning methods in MOOCs. Few of the studies to date have included research related
to residential students in higher education. The studies that have included higher education
students have focused on learning outcomes and comparing the learner performance with and
without the extra resources provided to residential students (class time, teaching assistants, office
hours, etc.). Although there are differing points of view, it does appear by the abundance and
depth of research that MOOCs have had an impact on higher education and are a part of a
portfolio of options in online learning.
!
25
CHAPTER THREE
Methodology
Introduction
There have been numerous studies of MOOCs using quantitative data measures to
understand who is using MOOCs, patterns of behavior, and effective ways to enhance retention
and interaction among participants. My study helps to understand the impact of MOOCs to
residential higher education students at a public university through a mixed method approach. I
examined two data sources: (a) private archived quantitative data on participant behavior on the
Coursera platform and (b) online survey data of UVa students engaged in UVa MOOCs from
2012 to 2014.
The archived quantitative data provided behavioral data on the activities students
engaged in when in the MOOC. Examples of data collected with this source were videos
watched, quiz participation, quiz attempts, and discussion forum participation. Permissions to
access this data had been provided by the Provosts Office at the University of Virginia
(Appendix A).
The online survey (Appendix B) was distributed via email (Appendix C) to UVa students
in spring 2015. This survey included questions that verified the role of students in the MOOC,
perceptions on the usefulness of different components (video, quizzes, and forums) to learning
outcomes at the beginning versus the end of the semester, and open-ended questions. Some
questions on this form used a Likert scale to quantify perceptions, while other questions were
free text.
!
26
Participants
The population in this research was UVa students that participated in UVa MOOCs in
2012-2014. There were approximately 1,500 (name)@virginia.edu participants in UVa MOOCs.
Based on data from the Office of Institutional Assessment and Studies at UVa, 445 of these
email addresses were known to be current residential students. In my research, I
assessed overall student behaviors such as time accessing materials, course videos
watched, performance on quizzes and discussion forum participation across all UVa
MOOCs.
gathered qualitative feedback from students across all UVa MOOCs to identify their
perceptions on the effectiveness of MOOCs for creating an engaging residential
student learning experience.
Providing the engagement patterns and course behaviors of the residential students spoke
to some of the specific design opportunities and challenges for MOOC content in higher
education. It was feasible for me to conduct my study now, this academic year, due to my
position/role as the Director of Online Learning Programs working out of the Office of the
Provost at the University of Virginia. I am the site administrator and data coordinator for the
Coursera platform. In my role, I promote research into large data sets from MOOCs, thereby
establishing an example of research that can be conducted that is valuable with MOOC data at
UVa and perhaps higher education at large.
Theoretical Framework
I approached my research project with a constructivist worldview. I believe that
participants in MOOCs create their learning based on the culture in which they live. I live in the
!
27
United States and have a perspective on this study positioning higher education and technology
in the society of the United States. I considered a systems thinking perspective as one that
believes in a web of interdependence (Senge, 1990). I looked at the overlap between the society,
technology, and higher education to note the following:
Society and technology lead to abundant information on demand delivered to
individual devices;
Society and education have a crisis point of high cost/high debt, low quality and lack
of accessibility; and
Technology and higher education provide the ability to offer personalized, adaptive
content to students through the use of learning analytics and behavior pattern logging.
The audience of MOOCs and the role that MOOCs may play for higher education
residential students is worth investigating to see if MOOCs can increase the level of engagement,
increase student satisfaction, increase learning outcomes, and/or decrease costs in delivery of
educational content. Figure 3 illustrates the intersection of higher education, technology, and
society.
!
28
Figure 3. Venn diagram illustrating the intersection of higher education, technology, and
society.
For this mixed-methods research study, I worked in parallel on the data collection. The
overall timeline is listed below (see Table 3). All research was approved by the Fielding
University IRB process as well as the UVa IRB process.
!
29
Table 3
Illustration of Timeline for Process Steps for Research Approach
Data Source 1: Quantitative Data Study on Behavior Patterns
Following my review of the literature I connected to research teams at the University of
Michigan, MIT, Oxford, and University of Pennsylvania to discuss their research findings,
opportunities for collaborating on future research, and I offered to share the data from UVa
MOOCs. The result of this first step was the literature review and a community of inquiry on this
topic.
The next step was to leverage the approaches for understanding the learner analytics to
illustrate usage patterns. I worked with peer researchers at Oxford and Coursera to create
algorithms to extract the relevant UVa student data from the archival data. I aggregated the UVa
data to illustrate usage patterns of the residential students.
Data Source 2: Survey Data of Residential Students
The online survey was administered to all UVa students that participated in UVa
MOOCs. There were approximately 1,500 (name)@virginia.edu participants in UVa MOOCs.
!
30
Based on data from the Office of Institutional Assessment and Studies at UVa, 445 of these
email addresses were known to be current residential students. I anticipated a response rate
around 10% with an n value of roughly 45. The actual n value was 52.
The first question in this online survey had students identify their role: UVa student
where MOOC was required, UVa student where MOOC was voluntary, UVa student not
enrolled in corresponding residential course, UVa alumni, and other. The other category was
intended for faculty members and others that received the survey that were affiliated with the
university but not in the role of student or alumni.
Questions in the online survey gathered input on the perceived effectiveness of the
MOOC. Questions solicited students to provide their expectations at the beginning of the
semester and experiences at the end of the semester for features on the MOOC such as video
lectures, discussion forums, and quizzes. Additional questions asked students if they had taken
more than one MOOC and if they had any thoughts or feedback about their experience or online
learning at the university. There were open-ended questions asking what they liked most/least
about the learning experience, were there any surprises, and if they would like to have more
classes in the flipped classroom approach with lectures viewed outside of class.
!
31
Participants
Students that were eligible to participate in the study were students that had a MOOC as
part of their course work at UVa between 2012 and 2014. They were identified based on their
enrollment in MOOCs with a (name)@virginia.edu email address. Students were able to
participate regardless of their level of participation in the MOOC. It is possible that some
students had been active in discussion forums and taking exams, while others may have created
an account but never accessed or engaged with any of the content. Some professors required
participation in the MOOC, while others created the MOOC content to be supplemental and
voluntary for the UVa students.
All UVa MOOC participants with a (name)@virginia.edu address received an email
communication sent directly from my Virginia email account asking them to participate in an
online survey related to the MOOC that would take approximately 5 minutes. The email stated
that the purpose of the survey was to understand the impact of MOOCs for residential students
and that the survey would be anonymous and in no way would affect their grades. All students
participating were over 18 years of age. The possible student population for this study could have
been as great as 1,500 UVa students. Fifty-two students responded to the online survey. I
conducted this research during the spring semester 2015.
Setting
This study was conducted through the Office of Online Learning Programs at the
University of Virginia. I am the Director and sole employee of this office. I am also the site
administrator and data coordinator for the Coursera platform at UVa, and a site administrator and
contribute to the UVa iTunes U platform. I have full access to all data and course materials for
every MOOC produced by UVa. I report to the Provost Office and am a member of the 2012-
!
32
2013 Presidents Strategic Planning Committee. The Cornerstone Plan was the result of this
committee and was approved by the Board of Visitors in October 2013. It included the
following high-level strategic pillars for the university coming in to the second century of the
university: (a) extend and strengthen the distinctive residential culture, (b) strengthen the
capacity to advance knowledge and serve the public through research, scholarship, creative arts,
and innovation, (c) provide educational experiences that deliver new levels of student
engagement, (d) assemble and support a distinguishing faculty, and (e) steward resources to
promote academic excellence and affordable access.
UVa is perceived as an elite public university and consistently ranks second for all public
universities in the US News and World Report rankings of United States 4-year institutions.
UVa is rather unique since our President Terry Sullivan was asked to resign in the summer of
2012. This act was widely publicized in higher education media. The Board of Visitors for the
university had stated that the university was not being progressive enough in utilizing online
learning opportunities. There was a strong backlash by the faculty and numerous protests that
filled the central lawn of the UVa grounds with students, faculty, and staff in support of the
ousted President Sullivan. President Sullivan was reinstated shortly after the protests. Since this
event, the university has had high turnover in leadership positions with vacant CFO and VP of IT
roles, seven new deans, and new leadership for the university hospital. UVa has an operating
budget of approximately three billion dollars annually.
The institutional contract with Coursera was executed within 2 weeks of the forced
resignation of President Sullivan. There were no costs associated with the contract. I was tasked
to run this program. Since that time, UVa has produced 14 MOOCs and run 42 sessions of these
MOOCs. A session is equivalent to a semester and usually sessions run during fall, spring, or
!
33
summer semesters. MOOCs have been produced by three of the eleven schools at UVa: Curry
School of Education, Darden Graduate School of Business, and the College of Arts and Sciences.
Course domains include education, business, philosophy, history, literature, and physics. Over
two million learners from over 200 countries have participated in these MOOCs with over
50,000 receiving Statements of Accomplishment (SoAs). SoAs are widely perceived to be
representative of participants successfully completing a MOOC. MOOCs have been published
mainly on the Coursera platform but also on Apples iTunes U platform. It has been said that
UVa has a strong book culture and recalcitrant views towards online education. It has an
established reputation as a Southern gentlemans university where students come to sporting
events dressed formally, frequently wearing bow ties.
In my role at the university, I am very fortunate to work with brilliant professors,
wonderful staff across all of the schools, and research partners at elite institutions. I have
autonomy to work on research as long as faculty are happy and MOOCs run successfully. I
telecommute from California, typically making 6-12 work trips to Charlottesville, Virginia
during the year.
I have a good working relationship with the IRB at UVa and partner closely with them
for both the project of analyzing data from unidentified global participants and survey
instruments for all UVa students. Since I work for the Provost Office, I have a high level of
visibility and perceived respect. This might have helped to influence participant or student
response rates or responses.
!
34
Limitations
My database and data mining analytic skills are developing so I partnered with
researchers at UVa and peer institutions to understand and represent data collected on
the behavior patterns of UVa students in MOOCs.
Students might have been influenced by my role at the university in their responses.
UVa is an elite residential university where students do not represent the typical
higher education student, especially those students that are facing a lack of access to
high quality education without cost constraints.
Students responded about their experiences in the past so the responses might not
have been as clear or objective as they would have been if the intervention had been
conducted during or immediately after the UVa course that included the MOOC
content.
Summary
I have partnered with peer researchers from Oxford and Coursera to analyze the
behavioral data in eight MOOCs to identify residential UVa student participation in these
MOOCs. I implemented an online survey to a study population of 445 students to gather
feedback on perceptions on MOOCs. Fifty-two students responded to the online survey. I
conducted this research in the spring of 2015 for student involvement in MOOCs from 2012-
2014.
!
35
CHAPTER FOUR
Findings
Introduction
There are two data sets presented in this section. The first data set was collected from
clickstream data for each MOOC. The intent of this behavioral data collection was to identify
behavior trends of UVa students when interacting with MOOC materials such as videos, quizzes,
and discussion forums. The second data set was collected from an online survey administered to
UVa students. The intent of this survey was to collect UVa student perceptions related to the
MOOC and online learning.
Data Set 1: Quantitative Data from Clickstream Data on UVa Student Course Behaviors
For this data set, students were divided into two groups: (a) students where the MOOC
was required as part of their residential course work, and (b) students where UVa student
participation was optional. The first group included students from the session of The Modern
World that was open for UVa students only and The Kennedy Half Century where UVa students
were designated as Community Teaching Assistants and required to post on the discussion
forums as part of their grade. The second group included the other sessions of The Modern
World that were open to global learners. The second group also included the business MOOCs
Foundations of Business Strategy and Design Thinking. Finally, the second group included the
education MOOC Effective Childhood Interaction and the literature MOOC, Plagues, Witches,
and War: The Worlds of Historical Fiction.
For each group, I reported on the behaviors of UVa students and the overall learner
population. I included course video views, course video downloads, quiz submissions, and
discussion forum postings. The word active is used to indicate learners that have had some
!
36
interaction with the MOOC contents other than solely registering for the MOOC. Average posts
per learner or any data citing average computes the average number of posts for those that did
post at least once. The data provided insight to the behaviors of UVa students compared to non-
UVa learners in the MOOC. In the conclusion, I compared the perceptions of UVa students from
the online survey to the actual behavior as seen through the clickstream data.
Questions for the behavioral clickstream data:
Are UVa students watching the course videos?
Of the UVa students that do watch course videos, how many videos on average do the
UVa students watch?
Are UVa students downloading course videos?
Of the UVa students that do download the course videos, how many videos on average do
the UVa students download?
Are UVa students taking the quizzes?
Of the UVa students that do take the quizzes, how many quiz attempts on average do
UVa students make?
Do UVa students post to the discussion forums?
Of the students that post to the discussion forums, how many postings do the UVa
students make?
How do these behaviors compare to non-UVa learners in the MOOC?
Group 1: MOOCs where UVa student participation was required
The Modern World
This MOOC was developed by the College of Arts and Sciences, and consisted of 15
weeks of content that was launched on January 1, 2013. It was hosted on http://uva.coursera.org
!
37
and was only open to learners that had UVa credentials for authenticating and accessing the
content. The total enrollment was 100 with 77 learners active in the first week and 95 learners
active in the last week. There were no non-UVa students participating in this MOOC.
Ninety-two percent of UVa students viewed a course video. There were 92 videos in the
course. The average number of videos viewed by UVa students was 124. Thirty-four percent of
the UVa students downloaded a video. Zero percent participated in the discussion forums.
Ninety-five percent of UVa students submitted a quiz and of the students that submitted a quiz
100% submitted a quiz more than once (Table 4).
Further analysis of the clickstream data provides detail on how many UVa students
watched how many of the MOOC videos:
12% of UVa students watched >0% and < 10% of MOOC videos
10% of UVa students watched > 20% and < 30% of MOOC videos
!
38
Table 4
Table of Clickstream Data for Internal UVa Students in The Modern World MOOC
The Modern World UVa Internal Number Percentage

UVa students active in MOOC 73
UVa students that viewed a video 67 92%
Number of video views by UVa students that viewed a video 8313
Average number of videos viewed by UVa students 124
Number of video viewers that were not UVa students 38
Number of videos viewed by non-UVa learners 2678
Average number of views for non-UVa learners 70
Number of video downloaders from UVa 25 34%
Number of videos downloaded by UVa students 155
Average video downloads per UVa student 6
Number of video downloaders that were not UVa students 6
Number of videos downloaded by non-UVa student learners 119
Average downloads per non-UVa student learners 20
Number of forum posters from UVa students active in the 0 0%
MOOC
Number of forum posts by UVa students 0
Average posts per UVa student Not Applicable
Number of forum posters that were not UVa students 0
Number of forum posts by non-UVa student learners 0
Average number of posts per non-UVa student learner Not Applicable
Number of UVa students who submitted a quiz 69 95%
Number of UVa students with more than one quiz submission 69 100%
Number of quiz submissions from UVa students 6044
Average number of quiz submissions for each UVa student 88
Number of quiz submitters that were non-UVa students 37
Number of non-UVa student quiz submitters with >1 36 97 %
submission
Number of quiz submissions by non-UVa student learners 1823
Average number of quiz submissions from non-UVa student 49
learners
!
39
The Kennedy Half Century
This MOOC was developed by the Presidents Office at UVa, consisted of 4 weeks of
content, and launched on October 21, 2013. The total enrollment was 15,974 with 7,422 learners
active in the first week, 3,660 learners active in the last week, and 3,682 learners receiving a
Statement of Accomplishment from Coursera for successfully completing the MOOC. UVa
students registered in the residential course were assigned the role of Community Teaching
Assistant and required to answer questions on the discussion forums.
Seventy-eight percent of UVa students viewed a course video. Nine percent of the UVa
students downloaded a video. Ninety-seven percent participated in the discussion forums.
Sixty-five percent of UVa students submitted a quiz. Of the 65% that submitted a quiz, 100%
submitted at least one quiz more than once. Average number of video views for UVa students
was 14 and for non-UVa student learners was 29. Average number of video downloads for UVa
students was 24 and for non-UVa student learners was 31. Average number of forum postings
for UVa students was 14 and for non-UVa student learners was 5. Average number of quiz
submissions for UVa students was 17 and for non-UVa student learners was 41 (Table 5).
There were 80 videos in the MOOC. Further analysis of the clickstream data provides
detail on how many UVa students watched how many of the MOOC videos:
!
40
Table 5
Table of Clickstream Data for The Kennedy Half Century MOOC
Kennedy Number Percentage

Average number of videos viewed by UVa students that viewed 14
a video
MOOC
Average posts per UVa student 14
Average number of posts per non-UVa student learner 5
Average number of quiz submissions per UVa student 17
Number of quiz submitters that were non-UVa student learners 7350
Number of non-UVa student learner quiz submitters with >1 6769 99.977 %
submission
learners
!
41
Group 2: MOOCs where UVa student participation was optional
Effective Classroom Interactions: Supporting Young Childrens Development
This MOOC was developed by the Curry School of Education, consisted of 4 weeks of
content, and launched on October 22, 2013. The total enrollment was 26,811 with 7,270 learners
active in the first week, 2,843 learners active in the last week, and 829 learners receiving a
Statement of Accomplishment from Coursera for successfully completing the MOOC.
Sixty-three percent of UVa students viewed a course video. Zero percent of the UVa
students downloaded a video. Fifty percent participated in the discussion forums. Sixty-three
percent of UVa students submitted a quiz. Of the 63% that submitted a quiz, 100% submitted at
least one quiz more than once. Average number of video views for UVa students was 13 and for
non-UVa student learners was 33. Average number of video downloads for UVa students was 0
and for non-UVa student learners was 25. Average number of forum postings for UVa students
was 7 and for non-UVa student learners was 9. Average number of quiz submissions for UVa
students was 15 and for non-UVa student learners was 31 (Table 6).
!
42
Table 6
Table of Clickstream Data for Effective Classroom Interactions: Supporting Young Childrens
Development MOOC
Effective Classroom Interactions: Supporting Young Number Percentage

Childrens Development
Average number of videos viewed by UVa students that 13
viewed a video
Average video downloads per UVa student Not Applicable
Number of forum posters from UVa students 4 50%
Number of quiz submitters that were non-UVa student 7837
learners
Number of non-UVa student learner quiz submitters with >1 7503 96%
submission
learners
!
43
The Modern World Session 1
This MOOC was developed by the College of Arts and Sciences, consisted of 15 weeks
of content, and launched on January 1, 2013. The total enrollment was 66,281 with 28,089
learners active in the first week, 5,404 learners active in the last week, and 4,939 learners
receiving a Statement of Accomplishment from Coursera for successfully completing the MOOC.
Ninety-two percent of UVa students viewed a course video. Thirty-one percent of the
UVa students downloaded a video. Five percent participated in the forums. Eighty-nine percent
of UVa students submitted a quiz. Of the 89% of students that submitted a quiz, 100% submitted
a quiz more than once. Average number of video views for UVa students was 144 and for non-
UVa student learners was 41. Average number of video downloads for UVa students was 10 and
for non-UVa student learners was 52. Average number of forum postings for UVa students was
1 and for non-UVa student learners was 6. Average number of quiz submissions for UVa
students was 106 and for non-UVa student learners was 48 (Table 7).
!
44
Table 7
Table of Clickstream Data for The Modern World Session 1 MOOC
Modern World Session 1 Number Percentage

viewed a video
MOOC
learners
submission
learners
!
45
The Modern World Session 2
This MOOC was developed by the College of Arts and Sciences, consisted of 15 weeks
of content, and launched on January 13, 2014. The total enrollment was 35,008 with 10,200
learners active in the first week, 1,536 learners active in the last week, and 2,071 learners
Fifty percent of UVa students viewed a course video. Six percent of the UVa students
downloaded a video. Zero percent participated in the forums. Sixty-seven percent of UVa
students submitted a quiz. Of the 67% of UVa students that submitted a quiz, 100% submitted a
quiz more than once. Average number of video views for UVa students was 11 and for non-UVa
student learners was 10. Average number of video downloads for UVa students was 1 and for
non-UVa student learners was 11. Average number of forum postings for UVa students was 0
and for non-UVa student learners was 2. Average number of quiz submissions for UVa students
was 7 and for non-UVa student learners was 9 (Table 8).
!
46
Table 8
Table of Clickstream Data for The Modern World Session 2 MOOC
Modern World Session 2 Number Percentage

viewed a video
MOOC
learners
Number of non-UVa student learner quiz submitters with >1 13097 99.977 %
submission
learners
!
47
Plagues, Witches, and War: The Worlds of Historical Fiction
This MOOC was developed by the College of Arts and Sciences, consisted of 8 weeks of
content, and launched October 15, 2013. The total enrollment was 22,111 with 7,568 learners
active in the first week, 2,618 learners active in the last week, and 829 learners receiving a
Statement of Accomplishment from Coursera for successfully completing the MOOC.
Forty-two percent of UVa students viewed a course video. Thirteen percent of the UVa
students downloaded a video. Eleven percent participated in the forums. Sixteen percent of
UVa students submitted a quiz. Of the 16% of UVa students that submitted a quiz, 67%
submitted a quiz more than once. Average number of video views for UVa students was 9 and
for non-UVa student learners was 14. Average number of video downloads for UVa students
was 1 and for non-UVa student learners was 32. Average number of forum postings for UVa
students was 4 and for non-UVa student learners was 6. Average number of quiz submissions
for UVa students was 5 and for non-UVa student learners was 4 (Table 9).
There were 86 videos in the MOOC. A portion of the videos in this MOOC were
recorded sessions of the residential students in class with visiting authors. Further analysis of the
clickstream data provides detail on how many UVa students watched how many of the MOOC
videos:
!
48
Table 9
Table of Clickstream Data for Plagues, Witches, and War: The Worlds of Historical Fiction
MOOC
Plagues, Witches, and War: The Worlds of Historical Number Percentage

Fiction
viewed a video
MOOC
learners
submission
learners
!
49
Design Thinking for Business Innovation
This MOOC was developed by the Darden Graduate School of Business, consisted of 5
weeks of content, and launched on November 5, 2013. The total enrollment was 74,891 with
18,137 learners active in the first week, 6,011 learners active in the last week, and 761 learners
Thirty-eight percent of UVa students viewed a course video. Zero percent of the UVa
students downloaded a video. Zero percent participated in the discussion forums. Thirty-eight
percent of UVa students submitted a quiz. Of the 38% of UVa students that submitted a quiz,
60% submitted a quiz more than once. Average number of video views for UVa students was 9
and for non-UVa student learners was 13. Average number of video downloads for UVa
students was 0 and for non-UVa student learners was 28. Average number of forum postings for
UVa students was 0 and for non-UVa student learners was 6. Average number of quiz
submissions for UVa students was 2 and for non-UVa student learners was 2 (Table 10).
!
50
Table 10
Table of Clickstream Data for Design Thinking for Business Innovation MOOC
Design Thinking for Business Innovation Number Percentage

viewed a video
Average video downloads per UVa student Not Applicable
MOOC
Number of forum posts by UVa students 0.0
Number of UVa students with more than one quiz 3 60%
submission out of 12 UVa students that submitted a quiz
learners
submission
learners
!
51
Foundations of Business Strategy
This MOOC was developed by the Darden Graduate School of Business, consisted of 6
weeks of content, and launched June 16, 2014. The total enrollment was 49,135 with 25,313
learners active in the first week, 4,895 learners active in the last week, and 730 learners receiving
a Statement of Accomplishment from Coursera for successfully completing the MOOC.
Fifty percent of UVa students viewed a course video. Seven percent of the UVa students
downloaded a video. Zero percent participated in the discussion forums. Thirty-six percent of
UVa students submitted a quiz. Of the 36% of UVa students that submitted a quiz, 60%
submitted a quiz more than once. Average number of video views for UVa students was19 and
for non-UVa student learners was 12. Average number of video downloads for UVa students
was 10 and for non-UVa student learners was 27. Average number of forum postings for UVa
students was 0 and for non-UVa student learners was 4. Average number of quiz submissions
for UVa students was 3 and for non-UVa student learners was 5 (Table 11).
!
52
Table 11
Table of Clickstream Data for Foundations of Business Strategy MOOC
Foundations of Business Strategy Number Percentage

viewed a video
Number of forum posters from UVa students active in the 0
MOOC
learners
submission
learners
!
53
Data from this source also show video viewing, quiz taking, and discussion forum
posting across UVa MOOCs that were not required for residential course work (Table 12).
Table 12
UVa Behaviors for Video Viewing, Quiz Taking, and Discussion Forum Posting in MOOCs that
were Not Required
Data comparing UVa learners to non-UVa learners for video viewing, video downloading,
quiz taking, and discussion forum posting in MOOCs that were not required are illustrated in
Table 13.
Table 13
UVa Behaviors for Video Viewing, Video Downloading, Quiz Taking, and Discussion Forum
Posting in MOOCs that were Not Required as Compared to Non-UVa Student Learners
!
54
Data Set 2: Online Survey Collecting UVa Student Perception
The email with the link to the survey was sent to 445 emails associated with a current
undergraduate or graduate student at UVa. All 445 received multiple communications for the
yearly experience survey in February 2015 for undergraduates and December 2014 for graduate
students. In addition, 247 students had been a part of at least one other requested survey sample
in spring 2015. Based on these prior survey requests, it was assumed this survey would have few
respondents due to survey fatigue. An n value of 45, 10% of the survey population, was
targeted.
The online MOOC survey had a total of 106 respondents; 52 were students with the other
respondents being mostly alumni. The students fell into three distinct groups: (a) students
enrolled in a residential course with the MOOC as optional, (b) students enrolled in a residential
course with the MOOC as required, and (c) students that were not enrolled in the residential
course associated with the MOOC. A Likert Scale was used for six questions. This Likert Scale
was from 1 (Extremely Valuable) to 5 (Not At All Valuable). All data in the findings for the
online survey were self-reported by students through the survey. Findings for each group are
presented after the table summarizing feedback across all student respondents to the open-ended
survey questions (Table 14).
There were four open-ended questions in the online survey. Table 12 represents the
responses from the open-ended questions distributed by student group: MOOCs as optional,
MOOCs required, and MOOCs taken outside of any residential course. Responses that were
stated more than once in any group, have the number of responses marked after the response. For
example, if four students stated the materials were convenient, the table documents convenient
x 4.
!
55
The four open-ended questions were
What did you like most about having the MOOC as part of your learning experience?
What did you like least about having the MOOC as part of your learning experience?
Did anything surprise you about this experience?
Any thoughts/feedback you'd like to share?
Across all groups, UVa students noted the ease of use and convenience of the MOOC
materials. For MOOCs related to residential courses in which the UVa student was enrolled,
students indicated a more in-depth learning in the residential classroom although disliked the
time commitment outside of class.
!
Table 14
Responses from Open-Ended Questions in the Online Survey Completed by UVa students
56
57
Group 1: Student evaluation where the MOOC was optional
Of the 13 students who submitted the online survey in this group, 6 (46%) chose to take
additional MOOCs outside of the residential course. Seven (54%) had taken only one MOOC,
the one associated with the course.
Question: How interested are you in taking another course taught in this way, with materials in a
MOOC and class time focused on discussion? (Table 15)
1 = Extremely Interested, 5 = Not At All Interested
Table 15
UVa Student Responses to Question of Interest in Flipped Classroom Approach where MOOC
was Optional (n=13)
MOOCs Optional Responses Percent

1 (Extremely Interested) 3 23%
2 3 23%
3 (Neutral) 6 46%
4 1 8%
5 (Not At All Interested) 0 0
Question: How valuable did you expect the MOOC videos to be in helping you learn? (Table
16)
1 = Extremely Valuable, 5 = Not At All Valuable
!
58
Table 16
UVa Student Responses to Question of Anticipated Value and Actual Value of Videos where
MOOC was Optional (n=13)
Anticipated Value of Videos Responses Percent

1 (Extremely Valuable) 0 0%
2 2 15%
3 (Neutral) 9 70%
4 2 15%
5 (Not At All Valuable) 0 0%
Actual Value of Video Responses Percent

2 6 46%
3 (Neutral) 3 23%
4 1 8%
Question: How valuable did you expect the MOOC quizzes to be in helping you learn? (Table
17)
!
59
Table 17
UVa Student Responses to Question of Anticipated Value and Actual Value of Quizzes where
MOOC was Optional (n=13)
Anticipated Value of Quiz Responses Percent

2 1 8%
3 (Neutral) 7 54%
4 5 38%
Actual Value of Quiz Responses Percent
2 4 31%
3 (Neutral) 5 38%
4 0 0%
Question: How valuable did you expect the MOOC forums to be in helping you learn? (Table
18)
!
60
Table 18
UVa Student Responses to Question of Anticipated Value and Actual Value of Discussion
Forums where MOOC was Optional (n=13)
Anticipated Value of Forums Responses Percent

2 0 0%
3 (Neutral) 6 46%
4 6 46%
Actual Value of Forums Responses Percent

2 2 15%
3 (Neutral) 5 38%
4 3 23%
Questions: What did you like most about having the MOOC as part of your learning experience?
Ease of use.
I could do the work whenever it was most convenient for me.
I liked being able to do the videos on my own time, with the ability to pause or go back to
a part of the lecture I missed.
I was able to replay the information if I did not understand it the first time, or if I wanted
to review the material.
I usually ended up not paying attention to the MOOC and I usually take it via Coursera.
Question: What did you like least about having the MOOC as part of your learning experience?
I enjoy the typical classroom experience.
!
61
Not having the live experience and Q&A that usually occurs in a live lecture setting.
I did not like that watching the material for the course took up a long period of time.
A lot of the things are peer graded in Coursera. How is the person in my class qualified
to grade?
Did anything surprise you about this experience?
I learned more than I expected to learn.
I was surprised that I enjoyed the course. I was able to participate more in discussion
because I fully understood the material before coming to class.
Any thoughts/feedback you'd like to share?
There were no responses to this question.
Any thoughts/feedback about online learning at UVa?
Need more courses for those nurses working full time.
I really prefer not to take courses online if I can help it. I do not like online courses.
!
62
Group 2: Student evaluation where the MOOC was required
Of the 22 students who submitted the online survey, only 3 chose to take additional
MOOCs outside of the residential course. Nineteen had taken only one MOOC, the one
associated with the course.
Table 19
UVa Student Responses to Question of Interest in Flipped Classroom Approach where MOOC
was Required (n=22)
MOOCs Required Responses Percent

2 4 18%
3 (Neutral) 3 14%
4 3 14%
5 (Not At All Interested) 5 23%
20)
!
63
Table 20
MOOC was Required (n=22)

2 6 27%
3 (Neutral) 12 55%
4 4 18%

2 3 14%
3 (Neutral) 5 23%
4 4 18%
21)
!
64
Table 21
UVa Student Responses to Question of Anticipated Value and Actual Value of Quizzes where
MOOC was Required (n=22)

2 5 23%
3 (Neutral) 10 45%
4 6 27%

2 5 23%
3 (Neutral) 8 36%
4 4 18%
22)
!
65
Table 22
Forums where MOOC was Required (n=22)

2 4 18%
3 (Neutral) 6 27%
4 3 14%

2 4 18%
3 (Neutral) 4 18%
4 6 27%
Question: What did you like most about having the MOOC as part of your learning experience?
Sixteen students responded to this question. Responses fell into two categories:
accessibility of resources and learning.
Accessibility of Resources
Nine comments referred to the accessibility and convenience of materials and resources.
One participant said, I liked having the main lectures for the course in video format, as I could
pace the course to fit the rest of my academic schedule. Another stated, I also get restless
easily, so it was good to be able to pause, walk around, and go back in a video.
Learning
Six comments referred to deeper learning, engagement, and critical thinking on the
subject matter. One participant said, I got to ask more questions and learn about the subject in a
more in-depth manner. Another stated, The discussion in class made sure we truly understood
!
66
the lecture material in the videos. It helped me engage with and think critically about the
material. Another stated, Using the MOOC format allowed the professor to have discussions
with the student and what would've been normal discussion sections were used for research
projects. This gave us both better access to information (through the professor), and allowed us a
greater depth of knowledge on the subject.
Sixteen students responded to this question. Responses fell into three categories: time
commitment, quizzes, and discussion forums.
Time Commitment
Four responses referred to the significant time commitment outside of class required to
watch all of the MOOC video lectures. One student stated, Having the MOOC took away from
my time outside the classroom as I had to sit through the hours of videos that the professor
posted. Another stated, Often I'd realize it was the night before class and I had up to 3 hours
of videos to watch, which could be exhausting. As a result, I did not always watch every video.
Quizzes
Four responses referred to due dates and the difficulty of the quizzes. One student stated,
The quizzes were always tricky. Another stated, Due dates of assignments. Maybe it should
be set up during a time that is more convenient for student.
Discussion Forums
Two responses referred to the discussion forums. One student stated, I really did not
feel the need to engage with the non-UVA students who were online; the forums were not really
!
67
necessary for my learning experience. The other student stated, The discussion forums were
difficult to navigate. My classmates would tell me about these people they talked to in the
MOOC but I could not find the thread.
Question: Did anything surprise you about this experience?
Eleven students responded to this question. Responses fell into two categories: video
usefulness and flipped classroom effectiveness.
Video Usefulness
Four responses referred to the enjoyment and usefulness of the videos. One student
stated, The videos were surprisingly very informational and beneficial to learning the material.
Another stated, I was surprised by how helpful the videos were.
Flipped Classroom Effectiveness
Two responses referred to the effectiveness of having materials available outside of
classroom and the flipped classroom approach. One student stated, I was surprised how much
work I was able to do -- having video lectures and class time for discussion is a lot of time.
Another stated, I had taken online classes before but never in conjunction with an in-class
portion. I was surprised at how well the online and the in-class portions of the course were able
to complement each other.
Question: Any thoughts/feedback you'd like to share?
Eight students responded to this question. The majority of responses related to the
balance of in-class discussions and out-of-class lecture viewing. One student stated, I think the
course worked best as a combination of MOOC and traditional classroom techniques. I don't
!
68
believe I would have enjoyed the course as much as I did if it had been strictly MOOC. Another
stated, Structure class time better to learn from the professor's vast experience.
Question: Any thoughts/feedback about online learning at UVa?
Eight students responded to this question. The majority of responses related to
maintaining the traditional classroom experience. One student stated, There should always be
some in-person component in order to make the class seem more substantial/credible. Another
stated, I think online can be very helpful, but think it should never replace classroom learning. I
definitely missed the interaction of a classroom.
Group 3: Student evaluation where the MOOC was not related to any course work they were
enrolled in
Of the 14 students who submitted the online survey, 8 were taking more than one
MOOC.
1 = Extremely Interested, 5 = Not At All Interested
!
69
Table 23
UVa Student Responses to Question of Interest in Flipped Classroom Approach when MOOC
was Not Related to Student Course Work (n=14)
MOOCs Outside Class Responses Percent

2 3 21%
3 (Neutral) 3 21%
4 2 14%
5 (Not At All Interested) 4 29%
24)
Table 24
MOOC was Not Related to Student Course Work (n=14)

2 4 28%
3 (Neutral) 6 42%
4 1 7%

2 3 21%
3 (Neutral) 5 35%
4 0 0%
!
70
25)
Table 25
UVa Student Responses to Question of Anticipated Value and Actual Value of Quizzes when
MOOC was Not Related to Student Course Work (n=11)

2 5 35%
3 (Neutral) 2 14%
4 4 28%

2 4 28%
3 (Neutral) 3 21%
4 2 14%
26)
!
71
Table 26
Forums when MOOC was Not Related to Student Course Work (n=14)

2 0 0%
3 (Neutral) 5 35%
4 4 28%

2 3 21%
3 (Neutral) 2 14%
4 2 14%
Question: What did you like most about having the MOOC as part of your learning experience?
Nine students responded to this question. Responses fell into two categories:
accessibility of resources and subject matter.
Accessibility of Resources
Five comments referred to the accessibility, flexibility, and convenience of materials and
resources. One participant said, Convenient way to receive well-curated information. Another
stated, The flexibility to learn on my own time and the challenge of committing myself to
something that wasn't required.
Subject Matter
Five comments referred to the excellent instructors and high quality materials. One
participant said, It was great to be able to watch one of UVa's best professors in action; I hadn't
ever had the chance to take a course with Peter Onuf on grounds. Another stated, I enjoyed the
!
72
interaction and the instructor was incredibly engaged with the class as a whole as well as
individuals. The questions and activities were practical and the readings were informative and
challenging.
Eight students responded to this question. Responses fell into two categories: lack of
interaction and time commitment.
Lack of Interaction
Five responses referred to the lack of interaction with others in the MOOC. One student
stated, It was very sterile. You aren't in a real course, interacting with the professor. I prefer in-
person classes. Another stated, No direct interaction with the instructor or any official source
was a big let down.
Time Commitment
Two responses referred to the significant time commitment. One student stated, I spent
more time on the computer than I would have in class and got far less from the experience.
Another stated, This learning experience involved an incredible amount of time more so than a
residential classroom experience. Keeping up with the forums, readings, and projects were
challenging.
Question: Did anything surprise you about this experience?
Four students responded to this question. One response stated, The courses broad
appeal to so many people. Another stated, It was far more difficult and less satisfying than I
expected. Another stated, How useful it was. Another stated, Since this course was open to
!
73
anyone, there were lots of non-UVa students participating and I was unpleasantly surprised by
the low level of discussion and the partisan screeds that some people went on.
Question: Any thoughts/feedback you'd like to share?
Four students responded to this question. One student stated, Stop overscripting the
student experience. Serious students want to learn and don't need to be led by the hand to engage
in a topic. Another stated, I know there has been a lot of fawning over MOOCs as the future of
education, but I really hope that's not true. They're really boring and it's hard to stay motivated to
finish them. Face-to-face classes are just so much more stimulating. I mean, I like that we have
this as a record of Onuf's teaching and all, but other than that, meh. I can take it or leave it.
Another stated, Real-time group meet or office hour will help.
Question: Any thoughts/feedback about online learning at UVa?
Three students responded to this question. One student stated, Not a great way to teach
in my opinion. Another stated, I felt like it has been a quality experience that was substantive
and valuable. Another stated, UVa should provide more MOOCs.
Summary
Data presented in this chapter are clickstream data and online survey data related to UVa
MOOCs. Clickstream data from 8 MOOCs are included with behavioral data related to UVa
student video views, video downloads, quiz taking, quiz attempts, and discussion forum
participation. Clickstream data include comparisons between the UVa students and non-UVa
student learners. Two of the 8 MOOCs were required for UVa students to participate in for their
residential class work. One MOOC was open to only UVa students and had no non-UVa
learners in the session.
!
74
Clickstream data from the required MOOCs indicated that UVa students will participate
in a MOOC based on what is required for the residential course. If a professor required
participation in a discussion forum, the UVa students participated in a discussion forum. If the
professor required taking quizzes in the MOOC, the UVa students took quizzes in the MOOC.
Data also indicated that students watched videos, but no students watched all MOOC videos.
Clickstream data from the other MOOCs indicated that UVa students viewed videos and
took quizzes. If the UVa student took a quiz, he or she will most likely take at least one quiz
more than once. Compared to the non-UVa learners, the UVa students downloaded fewer videos
and participated less in discussion forums.
Data from the online survey are organized in three groups: students who were required to
take the MOOC as part of class work, students where the MOOC was optional but related to
class work, and students where the MOOC was optional and not related to their class work.
Online survey data included anticipated and actual perceived value of videos, quizzes, and
discussion forums. In addition, open-ended questions were included in the online survey to
gather data on what UVa students liked most, liked least, and were most surprised about when
taking the MOOC.
Survey data indicated that the actual value of videos, quizzes, and discussion forums was
higher than the anticipated value for the students where the MOOC was required or the MOOC
was optional but related to class work. Responses in the survey tended to shift from neutral
anticipated value to either high or low actual value. This trend is more evident with students
where the MOOC was not required for their course work. UVa students in courses where the
!
75
MOOC was required or optional but related to course materials rated the actual value higher than
the students who took the MOOCs completely outside of course work.
UVa students stated in the open-ended survey questions an appreciation for the
convenience of materials outside of the classroom. UVa students indicated an interest in a
flipped classroom approach to learning. UVa students stated they valued the residential course
experience. UVa students stated there was a large time commitment outside of class to view all
MOOC materials. The perceptions stated in the online survey are consistent with the behaviors
captured in the clickstream data. UVa students view videos, take quizzes, and if required for
course work, will participate in the discussion forums.
!
76
CHAPTER FIVE
Discussion and Recommendations
Discussion
Based on the data from the findings, UVa students divided into two distinct groups for
their course behavior based on clickstream data and three distinct groups for their course
perceptions measured by the online survey. The two groups that appeared in the course
behaviors through the clickstream data were (a) UVa students required to take the MOOC as part
of the residential course they were currently enrolled in for credit and (b) MOOCs where UVa
student participation was optional which might have materials related to a course they were
enrolled in or it might be completely outside of their course work.
The three groups that emerged for student perception of MOOCs through the course
survey were (a) UVa students who were taking the MOOC that was optional and provided
supplemental course materials for the course they were currently enrolled in for credit, (b) UVa
students required to take the MOOC as part of the residential course they were currently enrolled
in for credit, and (c) UVa students who took a UVa MOOC on their own which was completely
outside of their course work.
Course Behaviors Measured Through Clickstream Data
The intent for the findings from this data source were to see what the residential students
were actually doing in the MOOC. Clickstream data was aggregated for the MOOCs to look for
patterns in UVa student behavior. Attention was focused on what UVa students were doing with
the lecture videos, discussion forums, and quizzes.
The two groups that appeared in the course behaviors through the clickstream data were
!
77
(a) UVa students required to take the MOOC as part of the residential course they were currently
enrolled in for credit and (b) MOOCs where UVa student participation was optional. These
optional MOOCs might have been materials related to a course UVa students were enrolled in or
the MOOC might have been completely outside of the students course work.
Questions addressed with the clickstream data included
Are UVa students watching the course videos?
Of the UVa students that do watch course videos, how many videos on average do the
UVa students watch?
Are UVa students downloading course videos?
Of the UVa students that do download the course videos, how many videos on average do
the UVa students download?
Are UVa students taking the quizzes?
Of the UVa students that do take the quizzes, how many quiz attempts on average do
UVa students make?
Do UVa students post to the discussion forums?
Of the students that post to the discussion forums, how many postings do the UVa
students make?
How do these behaviors compare to non-UVa learners in the MOOC?
Group 1: Students where the MOOC was required as part of their residential course work
For MOOCs that were required as part of residential course work, the data indicated that
students completed whatever components were specifically required as part of the course grade.
If the professor required residential students to take the quizzes, the UVa students took the
quizzes. If the professor required residential students to post in the discussion forums, the UVa
!
78
students posted in the discussion forums. A majority of the students also watched some of the
course videos though few downloaded them.
There were two residential courses that required UVa student participation within a
MOOC. The first was The Modern World MOOC. This MOOC was restricted to persons within
UVa. Students in this residential version of the course were required to complete the online
quizzes as part of their course grade. Ninety-five percent of students participated in the quizzes
and 100% of students submitted a quiz more than once. Presumably students took the quizzes
more than once to try and improve their score. Ninety-two percent of UVa students viewed a
course video, with the average number of videos viewed by students at 124. Thirty-seven
percent of students viewed between 50-60% of course videos. Thirty-four percent of the UVa
students downloaded a video. Zero percent participated in the forums.
The second residential course that required UVa student participation in a MOOC was
the Kennedy Half Century. For this MOOC, the professor had his UVa students designated as
Community Teaching Assistants (TAs). The residential students were required to answer
questions by global learners in the discussion forum as part of their course grade. Ninety-seven
percent of UVa students participated in the discussion forums in this MOOC. Seventy-eight
percent of UVa students viewed a course video. Twenty-eight percent of students viewed
between 10-20% of course videos. Nine percent of the UVa students downloaded a video. Sixty-
five percent of UVa students submitted a quiz. Of the 65%, 100% submitted a quiz more than
once.
The Kennedy Half Century MOOC was open to global learners. How did the behavior of
UVa students compare to that of non-UVa student learners? Average number of video views for
UVa students was 14 and for non-UVa student learners was 29. Average number of video
!
79
downloads for UVa students was 24 and for non-UVa student learners was 31. Average number
of forum postings for UVa students was 14 and for non-UVa student learners was 5. Average
number of quiz submissions for UVa students was 17 and for non-UVa student learners was 41.
Based on the data from the Kennedy MOOC, we can say that UVa students watched less of the
course videos, posted more in the discussion forums, and took fewer quizzes than the non-UVa
student learners.
Group 2: MOOCs where UVa student participation was optional
For MOOCs that were part of the data set for this study, the data indicated students tend
to watch some video lectures and took some quizzes although rarely download videos or post to
discussion forums. If students took a quiz, chances are high that they took quizzes multiple
times. This finding relates to the theory of mastery learning. In this theory, students perform
better if they are provided opportunities to revisit materials until they are fully understood before
moving on to the next concept (Bloom, 1984)
How did UVa students in these optional MOOCs compare to non-UVa learners? With
the data set examined across six MOOCs, the data are not conclusive. The video views by
course varied greatly. It is possible to say the trend of UVa student behavior based on these data
indicated that UVa students download fewer video lectures and post less frequently to discussion
forums than non-UVa learners.
Course Perceptions Measured Through Online Survey
The online survey was administered in April 2015. UVa students self-selected to
voluntarily complete this survey. There were a total of 445 emails associated with current
undergraduate or graduate students at UVa. All 445 had received multiple communications for
the yearly experience survey in February 2015 for undergraduates and December 2014 for
!
80
graduate students. In addition, 247 students had been a part of at least one other requested
survey sample in spring 2015. Based on these prior survey requests, it was assumed this survey
would have few respondents due to survey fatigue.
The online MOOC survey had a total of 106 respondents; 52 were students with the other
respondents mostly alumni. The students fell into three distinct groups: (a) students enrolled in a
residential course with the MOOC as optional, (b) students enrolled in a residential course with
the MOOC as required, and (c) students that were not enrolled in the residential course
associated with the MOOC. Students that took the MOOC in conjunction with a UVa residential
course overwhelmingly indicated that the MOOC videos, quizzes, and forums were of high
actual value to their course experience. Students who were taking the MOOC outside of a
residential course had more distributed responses.
Group 1: Students who took the MOOC as optional within the residential course
Thirteen students were in the group where the MOOC was optional. Students indicated
they were neutral in response to taking another class in the flipped classroom approach where
lecture materials were provided outside of designated class time. About half of the students who
completed the survey indicated that they had taken more than one MOOC. Students consistently
reported an increased actual value compared to anticipated value for videos, quizzes, and forums.
In the open-ended survey questions, students indicated the ease of use and convenience as
the attributes they liked most with having the MOOCs as part of the course work. Students
reported learning more and being more prepared for class although the time commitment outside
of class was cited as what they liked least. Students reported they would like more faculty
interaction during class, more online courses, and not offering only online courses.
!
81
The anticipated value of the videos was neutral, with 70% of respondents selecting the
Neutral response. The actual value of the video as assessed at the end of the semester course
shows an increase in perceived value. There was a 23% increase in students selecting
Extremely Valuable and a 31% increase in students selecting Very Valuable. There was a
7% decrease from the anticipated value to the actual value of the videos being Not of Much
Value.
The anticipated value of the quizzes was neutral, with 54% of respondents selecting the
Neutral response. The actual value of the quizzes as assessed at the end of the semester course
shows an increase in perceived value. There was a 31% increase in students selecting
Extremely Valuable and a 23% increase in students selecting Very Valuable. There were
no students that indicated actual value was negative. There was a 38% decrease from the
anticipated value to the actual value of the quizzes being Not of Much Value. Zero percent of
the students indicated that the actual value of the quizzes was Not of Much Value or Not At
All Valuable.
The anticipated value of the forums was neutral to negative, with 46% of respondents
selecting the Neutral response and 46% selecting the Not of Much Value response. Zero
percent of the students indicated that they anticipated the values would be Extremely Valuable
or Valuable. The actual value of the forums shows an increase in perceived value but 38%, the
majority of respondents, indicated Neutral as the actual value of discussion forums. There was
a 15% increase in students selecting Extremely Valuable and a 15% increase in students
selecting Very Valuable. There was a 23% decrease from the anticipated value to the actual
value of the forums being Not of Much Value.
Group 2: Students with the MOOC as required within the residential course
!
82
There were 22 students where the MOOC was required. Students indicated they were
interested in taking another class in the flipped classroom approach where lecture materials were
provided outside of designated class time. Fifty percent of students were Extremely Interested
or Very Interested to take another flipped class. Fourteen percent were Neutral, 14% Not
Interested, and 23% Not at all Interested. Almost none of the students who completed the
survey indicated that they had taken more than one MOOC. Students consistently reported an
increased actual value compared to anticipated value for videos, quizzes, and forums.
In the open-ended survey questions, students indicated the more in depth in class
discussions, increased faculty interaction in class, learning content more in depth, ease of use,
convenience, and array of perspectives as the attributes they liked most with having the MOOCs
as part of the course work. A startling 36% of the students wrote in free form that the discussion
in class was more personal and more in depth with 32% indicating they liked the increased
faculty interaction during class. Students reported being surprised by how beneficial and
informative the course videos were and learning more than they anticipated. Students reported
the time commitment outside of class, discussion forums, quizzes, due dates for assignments, and
lack of timely feedback on questions as the attributes they liked least. Students reported that they
wanted more faculty interaction and less in-class technology, such as iClicker tests, during
residential course instruction when MOOCs were required. Students reported they would like
better transcripts, consistent quality, inviting and entertaining lectures, and structured class time
in a required course that includes MOOC materials. Student feedback on online learning at UVa
included to always have some in-person component and to keep it up.
!
83
shows a substantial increase in perceived value. There was a 45% increase in students selecting
Extremely Valuable.
The anticipated value of the quizzes was neutral, with 45% of respondents selecting the
Neutral response. The actual value of the quizzes as assessed at the end of the semester course
shows a substantial increase in perceived value. There was a 23% increase in students selecting
Extremely Valuable. There were no students that indicated actual value was Not At All
Valuable. There was a 9% decrease from the anticipated value to the actual value of the quizzes
being Not of Much Value.
selecting Not At All Valuable. Zero percent of the students indicated that they anticipated the
values would be Extremely Valuable. The actual value of the forums shows responses
balanced across all of the choices but in general more neutral than the anticipated value. There
was a 14% increase in Extremely Valuable and a decrease of 18% in Not At All Valuable.
Group 3: Students who took UVa MOOC(s) outside of any residential course
There were 14 students where the MOOC was required. Student response from this
group tended to be more distributed and less conclusive. Student response to the flipped
classroom was a range but 29% stated, Not At All Interested. Fifty-seven percent of students
who completed the survey indicated that they had taken more than one MOOC.
In the open-ended survey questions, students indicated the convenience, access to
resources, access to professors, interesting subject matter, and lack of requirement to buy a text
as the attributes they liked most. Students reported the lack of direct faculty interaction, passive
learning, time commitment, quizzes, and lack of depth as the attributes they liked least. Students
were surprised at the difficulty, lack of usefulness, global audience, anonymity, and lack of
!
84
satisfaction. Students in this group recommended that the discussion forums should be optional,
journaling might be a good activity within the MOOC, keeping all the course logistics in one
place, adding real time group meetings or office hours, and keeping face-to-face classes. Student
feedback on online learning at UVa included providing more MOOCs, continuing creating
MOOCs that are substantive and valuable, and looking at the Old Dominion University (ODU)
teletechnet lectures as a model.
shows a larger distribution in perceived value. There was a 7% increase in students selecting
Extremely Valuable and a 14% increase in Not At All Valuable.
The anticipated value of the quizzes was distributed with 35% expecting the quizzes to be
Valuable and 21% expecting the quizzes to be Not At All Valuable. The actual value of the
quizzes shows a 7% increase in both the Extremely Valuable and Not At All Valuable
responses.
selecting Not Valuable or Not At All Valuable. The actual value of the forums shows
responses distributed with a 21% increase in the Valuable response and a solid 42% of
students indicating forums were Not At All Valuable.
Conclusions looking across perceptions and behaviors
Comparing the student perceptions of value against the behavioral data from the
clickstream indicates that there is a correlation in two areas. Students state videos are of actual
value and then demonstrate the behavior watching at least some videos. Students also indicate
high actual value of quizzes and behavior indicates some students take quizzes and almost all of
!
85
those students take the quizzes multiple times. It is more difficult to draw any conclusions
comparing quizzes and discussion forums between perceptions and behaviors.
Limitations
The primary limitations were the timeframe of this study and number of respondents.
Behavioral data examined in some cases were 2 years old while the perception data were recent.
The disparity of asking students perceptions for a MOOC they had taken over a year ago affects
the legitimacy of the perceptional data submitted.
A secondary limitation of this study was the number of respondents. There was survey
fatigue of the population group. UVa students had been administered several online surveys in
close proximity to this request. The assumption is that fewer students responded to this survey
due to this survey fatigue. Creating groups of students based on the role of the MOOC in their
course work also led to smaller population size. More student survey data would make the
perceptional survey results more representative of student experiences.
A third limitation to this study is who is identified as a UVa student. Students that are
currently registered and enrolled in classes at UVa in the spring 2015 semester are included in
the study group for the behavioral clickstream analysis. Any student that enrolled in a UVa
MOOC at a time between 2012-2014 is included in the population group for the perception data.
Therefore, UVa students that took the MOOC at UVa and then graduated, may have completed
the online survey and self-selected their role as student; however, this same former UVa student
would not be included in the behavioral clickstream population since he or she is not currently
registered as a student at the time of the study.
Another limitation is the limiting of this studys required MOOCs to the MOOCs where
only enrolled UVa students could be included in the data. There were only two classes that
!
86
could be included in this population. There were other MOOCs that may have required
participation as part of a residential course, such as Plagues, Witches and War: The Worlds of
Historical Fiction; however, other UVa students may also have registered for this class as
optional or supplemental to their course work since these MOOCs were open to any and all
learners. The only two MOOCs that were either open only to UVa students enrolled in the class
or UVa students who were enrolled in the class as a specific role in the MOOC could be included
in this data set. In addition, the assignment of a specific role, in this case as Community
Teaching Assistant compared to student, may have also influenced UVa student behaviors.
Therefore, these behaviors might not be consistent with behaviors that could be anticipated for
students where the MOOC is required as part of the course work when their role in the MOOC is
as a student.
Another limitation of this study is that it only includes MOOCs on Coursera rather than
the broader open educational resources (OER) that faculty may use with students in the
classroom. For example, UVa publishes content to Apples iTunes U, but this content was not
included in the scope of this research study. Behaviors and perceptions of OER from other
platforms may be different than those in this study.
Data in this study may have been more effectively presented in unique learner stories or
case studies. If the study had not been anonymous and confidential, it would have been possible
to create a story of the UVa student experience matching specific individual student behaviors to
their perceptions and then tying that data to their learning outcomes in the residential classroom.
Data in this study do not correlate with the faculty member that taught the MOOC or the
residential course. There are substantial differences in styles and expectations for faculty. This
study may have been more robust if data were examined across one professors MOOCs in
!
87
multiple iterations; for example, using only one MOOC but comparing data year over year with
one year the MOOC being required and another year the MOOC being optional for students.
Then the study could compare the student behaviors and perceptions year over year controlling
for the professor style and expectations.
My role reporting to the Provost Office at the University of Virginia and as Director of
Online Learning Programs may have influenced student answers within the online survey.
This study in the end does not address the critical issues brought up in the introduction
related to the cost, quality, and access of higher education. Some of the data presented in this
study may be useful for providing input on the quality and access to higher education materials.
At the time of this research study, there are no other research projects looking solely at
residential student experiences in MOOCs. Therefore, it is not possible to correlate the findings
in this study with higher education residential students at other institutions to compare perception
or behavior data or conclusions.
Summary
Based on the data examined on behaviors for this study, one can anticipate that UVa
students will complete course work in a MOOC that is part of what is required for their
residential course final grade. One can anticipate UVa students will watch some of the videos,
some students will take quizzes, and almost all students that take a quiz, will take a quiz multiple
times. One can anticipate that none of the UVa students will watch all of the videos. UVa
students who are taking the MOOC as optional show similar behavior patterns with the non-UVa
learner. There is some indication that they download fewer videos and post to discussion forums
less than non-UVa learners.
Based on data examined on perceptions for this study, UVa students appreciated the
!
88
convenience and ease of having course materials available outside of class. This unbundling of
class materials and experimenting with blended and hybrid learning methods aligns with the
research of Hollands et al. (2014). Students were surprised by the out-of-class time commitment
but stated they would like to participate in more flipped classrooms, with more materials
available outside of class, and more in-depth direct faculty interaction during the residential
classroom time. Providing MOOC materials outside of class as a textbook 2.0 aligns with the
research of Rawlinson (2007).
Students in a class with the MOOC required or optional but related to class work ranked
the actual value of videos, quizzes, and discussion forums higher than the anticipated value.
Responses tended to shift from neutral in the anticipated value to either low or high actual value.
Across the behavioral and perception data, students are consistent: Students rate videos and
quizzes of high value and their behavior follows with most students watching some videos and
most students taking quizzes taking the quiz multiple times.
Technology does appear to be poised to disrupt education (Christensen et al., 2008) to
some extent. Students involved with this study stated more in-depth learning and an appreciation
for the convenience and ease of use of the MOOC materials. Future studies incorporating the
recommendations of Glance et al. (2013) to include formative quizzes, retrieval practice, short
videos, and discussion forums for teacher-student and student-student interactions would be
valuable.
Does this research study answer the research question poised: What are the perceptions
and behaviors of higher education residential students in a MOOC? Yes. It does look at
multiple data sources to gain insight into behaviors and perceptions of MOOCs with higher
education residential students at UVa during the 2012-2014 academic years. Does this research
!
89
conclude ways to integrate MOOCs into the higher education landscape? No. There are factors
which exist: Technology is ubiquitous on the UVa campus, information is available on demand
at any time on the Internet, and students are expecting technology to provide adaptive,
personalized recommendations. The data in this study encourage further experimentation with
providing online resources to higher education residential students and further study on the
learning outcomes associated with blended and flipped classroom approaches to teaching and
learning.
Recommendations
For research related to MOOCs with residential higher education students, there are
several recommendations. A valuable immediate study would be to examine the learning
outcomes of students related to their behaviors (such as mastery learning through multiple quiz
attempts) and perceptions in MOOCs. For example, if a student watches more MOOC videos,
do he or she achieve higher scores in the residential classroom? Researching data to see if there
are inferences between these perceptions and behaviors to learning outcomes would be extremely
valuable for universities trying to define their strategy for flipped classrooms, blended learning,
and online learning.
Another valuable study would be controlling for the professor style and tracking students
behaviors, perceptions, and learning outcomes year over year in MOOCs that are required and
optional. Examining the flipped classroom which integrates MOOC materials and the perception
and engagement of students would provide insight to how to create a more engaging residential
student experience. A similar study to this one with gathering data in conjunction with the actual
residential course session would be valuable.
Another valuable study would examine the learning outcomes in a flipped classroom that
!
90
utilizes MOOC materials compared to a flipped classroom utilizing low production cost videos.
There is research that suggests the more authentic (less production cost) video materials leads to
improved learning outcomes. A study that would compare the two approaches to supplemental
residential course materials would be valuable.
There still remain the larger social issues of cost, quality, and access in higher education.
MOOCs are just one method of trying to teach in an online environment. How can we improve
the efficiency of students being able to graduate from a 4-year institution in 4 years? How can
we leverage open or low-cost resources to scale the higher education classroom effectively?
How do we cultivate students to have the life skills and practical skills to be employable after
they complete a course of study? How do we create educational environments for more students
to be successful? How do we scaffold learning environments that rely on scale and peer-to-peer
interaction to be effective?
Many are listening to the siren call of ubiquitous technology, abundant online educational
resources, the capability of adaptive learning, and the power of learner analytics. Where will
these new emerging trends lead us in an educational landscape that is rapidly changing? How
will millennial and digital native students learn and what will be their expectations of the 21st
century higher education experience? What do we need to do as an institution to meet the
educational needs of students, employers, and society?
!
91
REFERENCES
Allen, J., & Van der Velden, R. (2001). Educational mismatches versus skill mismatches:
effects on wages, job satisfaction, and on-the-job search. Oxford Economic Papers, 53(3), 434-
452.
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2014, April). Engaging
with massive online courses. In Proceedings of the 23rd international conference on World wide
web (pp. 687-698). International World Wide Web Conferences Steering Committee.
Bloom, B. (1984). The 2 Sigma Problem: The search for methods of group instruction as
effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16
Boyd, D. (2010). Streams of content, limited attention: The flow of information through
social media. Educause Review, 45(5), 26.
Bowen, W. G. (2012). The cost disease in higher education: Is technology the answer.
The Tanner Lectures Stanford University, Stanford, CA.
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T.
(2013). Studying learning in the worldwide classroom: Research into edXs first MOOC.
Research & Practice in Assessment, 8(1), 13-25.
Casey, D. M. (2008). The historical development of distance education through
technology. TechTrends, 52(2), 45.
Christensen, C. M., Horn, M. B., & Johnson, C. W. (2008). Disrupting class: How
disruptive innovation will change the way the world learns (Vol. 98). New York, NY: McGraw-
Hill.
!
92
Christensen, G., Steinmetz, A., Alcorn, B., Bennett, A., Woods, D., & Emanuel, E. J.
(2013). The MOOC phenomenon: Who takes massive open online courses and why?. Available
at SSRN 2350964.
Clow, D. (2013, April). MOOCs and the funnel of participation. In Proceedings of the
Third International Conference on Learning Analytics and Knowledge (pp. 185-189). ACM.
Colvin, K. F., Champaign, J., Liu, A., Zhou, Q., Fredericks, C., & Pritchard, D. E.
(2014). Learning in an introductory physics MOOC: All cohorts learn equally, including an on-
campus class. The International Review of Research in Open and Distributed Learning, 15(4).
Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and
possibility. Journal of Interactive Media in Education, 2012(3), Art-18.
Denhart, C., & Staff, F. (2013). How the $1.2 trillion college debt crisis is crippling
students, parents and the economy. Forbes Magazine.
Downes, S. (2008). Places to go: Connectivism & connective knowledge. Innovate:
Journal of Online Education, 5(1), 6.
Fain, P. (2013, January 16) As California goes? Inside Higher Education. Retrieved
from https://www.insidehighered.com/news/2013/01/16/california-looks-moocs-online-push
Fini, A. (2009). The technological dimension of a massive open online course: The case
of the CCK08 course tools. The International Review of Research in Open and Distributed
Learning, 10(5).
Glance, D. G., Forsey, M., & Riley, M. (2013). The pedagogical foundations of massive
open online courses. First Monday, 18(5).
Guglielmo, C. (2014, February). Mobile traffic will continue to rise, rise, rise as smart
devices take over the world. Forbes Magazine, February 2014. Retrieved from
!
93
http://www.forbes.com/sites/connieguglielmo/2014/02/05/mobile-traffic-will-continue-to-rise-
rise-rise-as-smart-devices-take-over-the-world/
Haggard, S., Brown, S., Mills, R., Tait, A., Warburton, S., Lawton, W., & Angulo, T.
(2013). The maturing of the MOOC: Literature review of massive open online courses and other
forms of online distance learning. Department for Business, Innovation and Skills, UK
Government.
Hansen, R. (2014). The architects of online learning: A strategic partnership for the
sustainability of higher education. Educause Review Online, November 2014. Retrieved from
https://www.educause.edu/ero/article/architects-online-learning-strategic-partnership-
sustainability-higher-
education?utm_source=Informz&utm_medium=Email+marketing&utm_campaign=EDUCAUS
E)
Hess, E. D. (2014). Learn or die: Using science to build a leading-edge learning
organization. New York, NY: Columbia University Press.
Hollands, F. M., & Tirthali, D. (2014). MOOCs: Expectations and reality. Center for
Benefit-Cost Studies of Education, Teachers College, Columbia University, New York, NY.
Institute for College Access & Success. (2014) Quick facts about student debt. Retrieved
from http://projectonstudentdebt.org/files/pub/Debt_Facts_and_Sources.pdf
Johnson, F. (2014, September). Teaching the Digital Native. National Journal. Retrieved
from
http://www.nationaljournal.com/policy/insiders/education/teaching-the-digital-native-20140915
!
94
Kizilcec, R. F., Piech, C., & Schneider, E. (2013, April). Deconstructing disengagement:
Analyzing learner subpopulations in massive open online courses. In Proceedings of the third
international conference on learning analytics and knowledge (pp. 170-179). ACM.
Koller, D. (2012, June).What were learning from online education. TED. Retrieved
from
http://www.ted.com/talks/daphne_koller_what_we_re_learning_from_online_education?languag
e=en
Kolowich, S. (June 27, 2014). 5 things researchers have discovered about MOOCs. The
Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/wiredcampus/5-
things-researchers-have-discovered-about-moocs/53585
Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to
support human beings? Participant support on massive open online courses. The International
Review of Research in Open and Distributed Learning, 12(7), 74-93.
Kortemeyer, G. (2013, February). Ten years later: Why open educational resources have
not noticeably affected higher education, and why we should care. EDUCAUSE Review.
Kurtzleben, D. (2014, February). Study: Income gap between young college and high
school grads widens. US News and World Report. Retrieved from
http://www.usnews.com/news/articles/2014/02/11/study-income-gap-between-young-college-
and-high-school-grads-widens
!
95
Lane, A., & McAndrew, P. (2010). Are open educational resources systematic or
systemic change agents for teaching practice?. British Journal of Educational Technology, 41(6),
952-962.
Leckart, S. (2012). The Stanford education experiment could change higher learning
forever. Wired Magazine, 20.
Lewin, T. (2013). After setbacks, online courses are rethought. The New York Times, 10.
Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A
systematic study of the published literature 2008-2012. The International Review of Research in
Open and Distributed Learning, 14(3), 202-227.
Lucas, H. C., Jr. (2013). Can the current model of higher education survive MOOCs and
online learning?. Educause Review, 48(5), 54.
Macleod, H., Haywood, J., Woodgate, A., & Alkhatnai, M. (2015). Emerging patterns in
MOOCs: Learners, course designs and directions. TechTrends, 59(1), 56-63.
Morrison, D. (2013). The ultimate student guide to xMOOCs and CMOOCs. MOOC
News and Reviews.
Newman, J., & Oh, S. (2014). Things you should know about MOOCs. The Chronicle of
Higher Education.
Ni, A. Y. (2013, Spring). Comparing the effectiveness of classroom and online learning:
Teaching research methods. Journal of Public Affairs Education, 199-215.
Pappano, L. (2012). The year of the MOOC. The New York Times, 2(12), 2012.
!
96
Rawlinson, L. (2007, November 8). Throw away your school books: Here comes
textbook 2.0. CNN International. Retrieved from
http://edition.cnn.com/2007/TECH/11/08/connexions.learning/
Saadatmand, M., & Kumpulainen, K. (2014). Participants' perceptions of learning and
networking in connectivist MOOCs. MERLOT Journal of Online Learning and Teaching, 10(1),
16-30.
Senge, P. M. (1990). The fifth discipline: The art and practice of the learning
organization. New York, NY: Doubleday/Currency.
Siemens, G. (2005). Connectivism: A learning theory for the digital age. International
Journal of Instructional Technology and Distance Learning, 2(1), 3-10.
Trumbore, A. (2014) Rules of engagement: Strategies to increase online engagement at
scale. Change: The Magazine of Higher Learning 46(4); 38-45.
Tucker, B. (2012, Winter). The flipped classroom . EdNext Journal, 12(1).
United Nations Childrens Fund (UNICEF). (2015). India education. Retrieved from
http://www.unicef.org/india/children_2359.htm
United Nations Educational, Scientific and Cultural Organization (UNESCO). (2015).
Open Educational Resources, 2015. Retrieved from
http://www.unesco.org/new/en/communication-and-information/access-to-knowledge/open-
educational-resources/
United States Census Bureau. (2013). Educational attainment. Retrieved from
http://www.census.gov/hhes/socdemo/education/
!
97
Weiner, J. (2013, September 5). Why Sally cant get a good job with her college degree.
Washington Post. Retrieved from http://www.washingtonpost.com/blogs/she-the-
people/wp/2014/09/05/why-sally-cant-get-a-good-job-with-her-college-degree/
Weller, M. (2011). A pedagogy of abundance. Revista Espaola de Pedagoga, 223-235.
Worldometers. (2015). Current world population. Retreived from
http://www.worldometers.info/world-population/
!
98
Appendix A
Approval from Archie Holmes to use UVa Coursera data for study
Kristin,
I think having a better understanding of the interactions that UVa students have with MOOC
materials is important as we work to implement Pillar 3 of the Cornerstone plan. As such, I am
granting you permission to use Coursera data from UVa courses, subject to IRB approval of your
protocol.
I look forward to hearing about what you learn in your dissertation research.
Archie
Archie Holmes
Vice Provost for Educational Innovation & Interdisciplinary Studies
Professor of Electrical and Computer Engineering
University of Virginia
!
99
Appendix B
Online Survey for all UVa students that participated in a UVa MOOC
!
100
!
101
!
102
!
103
Appendix C
Email request to students to collect feedback via online survey.
Email Subject Line/Title: Invitation and Consent for Research Participation in the study of:
Massive Open Online Courses (MOOCs) Evaluation and Usage Patterns of Residential
Students in Higher Education
Dear (Student Name),
You have been asked to participate in a research study conducted by Kristin Palmer, a doctoral
student in the School of Educational Leadership for Change at Fielding Graduate University,
Santa Barbara, CA. This study is supervised by Dr. Lee Mahon. This research involves
understanding the usage and perceived value of MOOCs. It is part of Kristins Fielding
Doctorate of Education dissertation. You are being invited to participate because you are a
student at UVa and have participated in a MOOC course.
The study involves participating an online survey. The total time involved in participation will be
approximately 5 minutes. Responses are being collected for this survey through March 1, 2015
UVa Student Experience in MOOCs Online Survey link:
https://docs.google.com/forms/d/1RoGwHGIWAYU96XfIIXsyE4AYsVUm2sUZE-
9ntNakJCM/viewform
Information you provide will be kept strictly anonymous. No names or identifying information
will be included in any documents. Any records that would identify you as a participant in this
study are not publicly accessible. A pseudonym will be assigned for any quotes that might be
included in the final research report. The results of this research will be published in Kristin
Palmers dissertation and possibly in subsequent journals, books, and other publications and
!
104
presentations.
The University of Virginia may use information provided in this study to help develop
curriculum to better serve and engage residential students. The risks to you are considered
minimal to none. You may withdraw from this study by choosing not to participate. If you do
submit the online survey, you will not be able to withdraw from this study since there are no
names associated with the data and therefore we are not able to delete your submitted responses.
You may benefit from this study through insights gained during reflections of your experience.
You may request a copy of the summary of the final results by emailing Kristin Palmer at
kristin@virginia.edu with the word STUDY in the email title. Results will be available after
June 2015.
If you have any questions about any aspect of this study or your involvement, please tell the
Researcher before completing the online survey. You may also contact the supervising faculty if
you have questions or concerns about your participation in this study. The supervising faculty
has provided contact information at the bottom of this form.
If you have questions or concerns about your rights as a research participant, contact the Fielding
Graduate University IRB by email at irb@fielding.edu or by telephone at 805-898-4033. The
Institutional Review Board of Fielding Graduate University retains the right to access the signed
informed consent forms and other study documents.
To proceed, please either Agree or Do Not Agree to participating in this study on the online
survey.
Thank You,
Kristin Palmer
1247 Chesterton Avenue
!
105
Redwood City, CA 94061
434-249-6659
kristin@virginia.edu
and
Dr. Lee Mahon
Mentor and Advisor
Fielding Graduate University
2020 De la Vina Street
Santa Barbara, California 93105-3814
800-340-1099

Out

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Out

Încărcat de

Drepturi de autor:

Formate disponibile

ProQuest Number: 3724851

All rights reserved

INFORMATION TO ALL USERS

All rights reserved.

increased faculty interaction during residential class time.

be a mom to the three coolest kids in the galaxy!

hard, be honest and be brave.

Figure 2: Illustration by Ravi Chandran illustrating evolving trends in education circa

June 2012 ...................................................................................................................................... 12!

(Department for Business Innovation and Skills, 2013) ............................................................... 13!

society ........................................................................................................................................... 28!

Childrens Development MOOC................................................................................................... 42!

Fiction MOOC .............................................................................................................................. 48!

Forum Posting in MOOCs that were Not Required...................................................................... 53!

Learners ........................................................................................................................................ 53!

Students ......................................................................................................................................... 56!

MOOC was Optional (n=13) ........................................................................................................ 57!

where MOOC was Optional (n=13) ............................................................................................. 58!

where MOOC was Optional (n=13) ............................................................................................. 59!

Discussion Forums where MOOC was Optional (n=13) ............................................................. 60!

MOOC was Required (n=22) ....................................................................................................... 62!

where MOOC was Required (n=22) ............................................................................................. 63!

where MOOC was Required (n=22) ............................................................................................. 64!

Discussion Forums where MOOC was Required ......................................................................... 65!

percent of the higher education landscape today is comprised of non-traditional students

inhabitants (UNICEF, 2015; Worldometers, 2015).

institution (Institute for College Access & Success, 2014).

earning potential, students need to graduate.

Velden, 2001). Regardless, underemployed or unemployed students are likely to default on

2012 (Denhart, 2013).

university (Koller, 2012).

Statement of the Problem

faculty, and researchers to investigate alternative options in instructional content and

learning for institutions. MOOCs require little institutional investment or resources. To

Institutions have been investigating the infrastructure, process, quality controls,

and pedagogy of online learning. MOOCs were introduced as a third-party online

same limitations or expectations of online educational content provided to for-credit, for-

innovative (Downes, 2008).

encouraging community interaction facilitated by an instructor. This content then

method of instruction. Figure 1 shows the development of different MOOC providers,

types of MOOCs (cMOOCs and xMOOCs) over time starting in 2008.

Intelligence course offered by Sebastian Thrun, educator, programmer, robotics

already taking the course in-person (Morrison, 2013).

These large enrollment MOOCs were differentiated from cMOOCs in the

consuming it. There is a difference in pedagogy between cMOOCs (connectivist where

Institutions producing MOOCs were getting experience in online learning.

the pedagogical effectiveness of MOOCs (xMOOCs and cMOOCs), participant types,

research to date on residential student experiences with MOOCs. Research of behavior

Purpose of the Study

perceived value of MOOCs to residential students in higher education at a public university.

Research Question and Methodology

and business courses.

classroom including the effectiveness of MOOCs materials.

Significance of the Study

understanding how to influence cost, quality, and/or access to higher education.

Review of the Literature

learning, distance education, and correspondence courses to deliver educational materials to

education in America (Lucas, 2013).

One of the technology disruptions in education is known as a massive open online

social media, blogs, and industry reports (Pappano, 2012).

A cornerstone of academia is creating new knowledge. This knowledge is then shared in

centuries of using a publishing company to disseminate new knowledge and scholarly

dominance in domain knowledge in the global community.

online courses or MOOCs in 2008 (Fini, 2009).