1

Use of web-based research materials in education: Is uptake occurring? Original proposal title: The use of web-based research materials: Using web analytics in conjunction with survey data to better understand how online research materials are used

Paper presented at CSSE, Montreal, 2010

Amanda Cooper, Hilary Edelstein, Ben Levin, Joannie Leung Ontario Institute for Studies in Education University of Toronto Theory and Policy Studies in Education Department 252 Bloor Street West Toronto, Ontario M5S 1V6 Canada E-mail: amanda.cooper@utoronto.ca hilary.edelstein@utoronto.ca ben.levin@utoronto.ca joannie.leung@utoronto.ca

Use of web-based research materials in education: Is uptake occurring? Introduction The internet age The rise of the internet offers new possibilities for research dissemination globally. The emergence of information technology has enabled people to access information at an ease and rate like never before (Dede, 2000). With nearly 2 billion internet users worldwide, the level of online activity is staggering and increasing exponentially: between 2000 and 2009 alone, internet usage worldwide increased by 400%! (http://www.internetworldstats.com/stats.htm). Every sector has been transformed by the internet age, and the education sector is no exception. Knowledge mobilization (KM) is about using research more to improve policy and practice in education. There is growing interest globally in understanding KM processes (Cooper, Levin & Campbell, 2009; Levin, 2010; Nutley, Walter & Davies, 2007). Researchers and educational organizations are increasingly using websites and the internet as the primary vehicle for the dissemination of research findings in the form of reports, knowledge sharing events, and the creation of interpersonal networks to support KM efforts (Hartley & Bendixen, 2001; Greenhow, Robelia & Hughes, 2009); consequently, investigating web-based dissemination and collaboration strategies (websites, social media, virtual networks, and so on) might better inform our understanding of KM in the current technological societal context. We have begun exploring online KM strategies as part of the Research Supporting Practice in Education (RSPE) program (www.oise.utoronto.ca/rspe). RSPE, housed at OISE and funded through Dr. Ben Levins Canada Research Chair, is a program of research and related activities that investigates KM across different areas of the education system including perspectives of research producers, research users, and in relation to the emerging technological landscape. The increasing importance of the internet draws our attention to use of web-based research materials as an important area for additional research. Currently, we have two studies underway which attempt to explore and evaluate the internets growing role in KM in education. One study is an analysis of KM strategies in educational and other organizations based on analysis of their websites (Qi & Levin, 2010). In this work we have developed inductively a common metric for assessing KM work as revealed on websites in terms of strategies (products, events and networks) and indicators as they relate to strategies (different types, ease of use, accessibility, focus of audience and so on). This analysis helps us understand the range of KM strategies being employed by different kinds of organizations, including research producers, research users and intermediaries. Our data show that few organizations display a wide range of practices related to KM, and many organizations have virtually no KM activity (Qi & Levin, 2010). Many organizations focus on posting reports and research-related products online, with far attention to building interaction through events or networks. However, as we looked at the range of different research products in a variety of formats for a variety of audiences, we also wondered whether and how much people are actually using these web-based research resources, a subject on which there appears to be very little empirical evidence. While a great deal of effort goes into developing websites for sharing of research materials and resources, there is little or no empirical evidence on the value or impact of these strategies. In fact, we could find no studies of how people actually use web-based research material in education. This study was conceived to explore the use of online research dissemination. The research question guiding this work is: How much and by whom are web-based research findings and analyses being used? The study uses two data sources to determine the extent and nature of use of web-based research products First, it uses web analytics to track website use. Second, survey data extend the web analytics data by asking users directly questions about research use that cannot be answered from usage data While our data are only beginning to accumulate from our various partner organizations, we hope this paper will stimulate discussion about this research design or other ways of studying the use of web resources. We expect that data from this study will shed light on what forms of online dissemination strategies are effective. An overview of the paper This paper is organized into four parts. We use a review of related literature to develop a conceptual framework for studying the use of web-based research materials. We describe the challenges and opportunities in studying research use online in relation to our approach of Google Analytics (GA) in conjunction with online surveys. Fourth, we provide some initial web metrics data from one of

3
our partner organizations in order to begin a discussion about the possibilities and limitations of these data to gauge research use and to answer KM questions. Literature Review This literature review is organized into three sections. The first outlines some key findings about knowledge mobilization generally that set the context for this study. The second examines the sparse literature that discusses KM in relation to the internet. The third section provides some introduction to the literature on web analytics and web metrics related to the study of KM. What we know about KM Research use is a multifaceted, nonlinear process that takes place within and between diverse organizations in the education system (Lemieux-Charles & Champagne, 2004; Levin, 2004; McLauglin, 2008; Nutley et al., 2007). Factors affecting KM also arise at multiple levels including individual, organizational and structural, as well as environmental and contextual (Berta & Baker, 2004). From a cross-disciplinary review of the literature, Levin (2004) outlines that KM is a function of the interaction among three main areas: research producing contexts; research using contexts; organizations and processes which mediate between these two contexts. All of this takes place over time within a larger societal context. Multiple iterations of use Understanding of KM has been growing in the past decade due to increasing interest in the topic as a way to improve public services (Cooper et al., 2009; Davies, Nutley & Smith, 2000). However many important issues remain unexplored, particularly in education (Cooper & Levin, 2010; Nutley et al., 2007). The empirical evidence suggests that research use remains modest across sectors, especially in education (Behrstock, Drill & Miller, 2009; Biddle & Saha, 2002; Cordingley, 2008; Hemsley-Brown, 2004; HemsleyBrown & Sharp, 2003; Lemieux-Charles & Champagne, 2004; Levin, S, Cooper & Mascarenhas, 2009; Pfeffer & Sutton, 2000). The use and impact of research are difficult to measure (Amara, Ouimet & Landry, 2004). One reason is that research may inform our thinking in ways that are not overtly visible in behaviour, sometimes referred to as conceptual use. Research can be used in direct and observable ways (instrumental use) though this is typically less frequent (Amara et al., 2004; Landry, Amara & Lamari, 2001). So it can be hard to know whether or to what extent research has actually informed the thinking or actions of people or organizations. The discussion of multiple kinds of research use is at least 30 years old, and still relies on Weiss (1979) foundational work on the many meanings of research utilization. Knott and Wildavsky (1980) also proposed seven levels of research utilization that remain relevant today: Reception Cognition ReferenceEffortAdoption Implementation Impact These sequential stages attempt to trace the different components involved from the time that someone actually receives a research related product to the point of impact resulting from that use. The literature on KM indicates that research use happens over time. Incorporating research into policy and daily practice is not an instantaneous process; rather, a multitude of factors - from quality of the evidence to the credibility of the messenger, to the effort it takes on the part of practice organizations to implement evidence-based changes - all affect how quickly (if at all) KM occurs (Levin, 2004; Nutley et al., 2007; McLaughlin, 2008). Timperley (2010) proposes that behaviour change takes at least three years to be fully incorporated. Her work involves intense and sustained interaction with teachers in order to have them use evidence (predominantly student assessment data disaggregated into different areas) to guide their practice. Others contend that incorporating research substantively takes much longer: Studies in healthcare show that it can take a decade or more before research evidence on the effectiveness of interventions percolates through the system to become part of established practice. The abandonment of ineffective treatments in the light of damning evidence can be equally slow (Davies, Nutley & Smith, 2000, p. 10). Many examples from the health sector and education sector reinforce this point such as the long road to increasing hand washing among health practitioners or the amount of time it took to end corporal punishment in schools. These studies suggest that in order to understand how much research use is actually going on in the education system, studies need to attend to the issue longitudinally. This study includes a longitudinal element. Audience and the format of research matters Many studies have reported that tailoring research products for groups of stakeholders increases the likelihood of use (Cordingley, 2008; Biddle & Saha, 2002; Levin, S, Cooper & Mascarenhas, 2009). Our team found similar results of research use by principals in school districts (Levin, S, Cooper & Mascarenhas, 2009). On the other hand, Belkhodja et al. (2007) found that interaction and contextual considerations of production and practice environments were much more influential than format of products.

4
Practitioners in the field have time and time again insisted that the format of the research influences whether or not they actually use it (Cordingley, 2008; Behrstock, Drill & Miller, 2009; Biddle & Saha, 2002; Levin, Sa, Cooper & Mascarenhas, 2009); however, this claim does not appear to have been tested. There is simply not enough empirical evidence yet to know whether adaptation of products or interaction and recognition of context are most important to research use. Our study will explore this issue to some extent by assessing which products are actually accessed and downloaded. The importance of active mobilization of research A considerable amount of research suggests that passive dissemination of research products has limited effectiveness (Armstrong, Waters, Crockett & Keleher, 2007; Grimshaw et al., 2006; Lavis, Robertson, Woodside, McLeod, & Ableson, 2003). If this is so, investing time and resources in passive online dissemination mechanisms also seems a doubtful strategy, yet one that is common. One cannot assume that research is being used just because it is freely available online. Research also provides growing evidence that successful dissemination efforts need to consider the audience and have dedicated staff and resources (Levin, 2008; Cooper et al., 2009). Dede (2000) similarly cautions that the internet, if utilized in the same way that traditional research dissemination has occurred (for example simply transferring large quantities of data to practice settings), will not yield different results. Hence, he suggests that reconceptualising the historic role of information technology in knowledge mobilization and use is central to its future effectiveness (p. 3). Linking KM and technological literature The literature on KM in relation to technology is sparse. Although many contend that the internet and various websites can facilitate this work, we found only a few studies in the health sector that explicitly addressed the use of the internet to mobilize research knowledge. Ho et al. (2004), in a conceptual paper, explore the potential synergy of research knowledge transfer and information technology, which they refer to as technology-enabled knowledge translation (TEKT). They provide evaluation dimensions and methodologies for TEKT including structural, subjective, cognitive, behavioural and systemic elements in order to help researchers compare successful models and characterize best practices of TEKT. However they do not provide any empirical data on these practices or ideas. Dede (2000) discusses the role of emerging technologies explicitly in relation to knowledge mobilization, dissemination and use in education. He elaborates on three ideas to use the internet to spread best practice across educational organizations. First, emerging information technologies enable a shift from the transfer and assimilation of information to the creation, sharing and mastery of knowledge (p. 2). Here, active collaboration among stakeholders, facilitated through the internet, is seen as a way to co-construct knowledge in a more meaningful way, because it takes into account contextual factors and, as a result, increases uptake. Second, Dede highlights that dissemination efforts must include all the information necessary for successful implementation of an exemplary practice, imparting a set of related innovations that mutually reinforce overall systemic change (p. 2). He argues that interactive media can facilitate this process, but must include detailed plans along a number of important areas leadership, professional development, and so on. Third, a major challenge in generalizing and scaling up an educational innovation is helping practitioners unlearn the beliefs, values, assumptions, and culture underlying their organizations standard operating practices (p. 3). He argues that professional rituals are deeply entrenched and that changing practitioners behaviours can be supported through virtual communities that provide social support for this difficult and sometimes threatening process. Jaded (1999) argues that the internet provides opportunities for networking and partnerships in the health sector. But he also lists a number of conditions that are necessary in order for online KM to be effective: a better understanding of the way service users and practitioners use the internet; systems that are easy to access and use; rapid transmission systems (bandwidth he argues is still too slow in many parts of the world); and information that is relevant and in a format that is ready to use. Different strategies are needed to integrate the large volumes of available information in a meaningful way; virtual interaction might still need to facilitate face-to-face meetings; and global access to technology is still needed to ensure global equity. While these are interesting ideas, they provide little evidence on the actual use of web-based research materials. Conceptual Framework For purposes of this study we conceptualize use of web-based research material in terms of the interaction between three elements (Figure 1): 1) Research evidence: Various aspects of the research products influence use. Type of resource (idea, product, contact, link) Format (long or short print version, video, language)

5
2) User: Role (parent, teacher, student, researcher, district administrator, journalist, interested citizen) Purpose of visit to website (work, study, personal reasons) Relevance (how tailored to particular users)

3) Actual use over time: Comparing original intention to actual use. Use over time (no use, undetermined usefulness, immediately useful, intended future use, actually used) Sharing of materials (formally and informally; internally or externally to their workplace) Type of use (conceptual, symbolic, instrumental)

RESEARCH EVIDENCE
Idea, product, link or contact Format of resource Relevance (Tailored for audience)
EXTENT &TYPE OF USE Intensity of Use Conceptual Instrumental Symbolic

TIME

CONTEXT
Stakeholder Individual/ Org Linkage Formal/ informal Internal/ external

No use Undetermined Immediate use Intended future use Actual Use

Figure 1. Conceptual framework: Online research use as the interplay between evidence, audience and use over time. Method This study involves our team partnering with educational organizations in Canada and abroad to investigate use of web-based research in education. The organizations vary in form and function; for example, one partner is a unit within a school district, while others are intermediary research organizations or have websites designed to be databases of relevant research. The study uses two data sources to assess the extent and nature of use of research products found on the websites of participating organizations. First, web analytics track website usage in various ways. Second, we developed two surveys, administered at two different points in time, that ask visitors directly about their use of these web-based resources. Using web analytics Web analytics software provides useful data on the use of research materials from websites (Wikipedia, http://en.wikipedia.org/wiki/Web_analytics). Tracking and understanding web analytics allows us to understand the specific activity on a website, translated into metrics (Ledford and Tyler, 2007). Types of metrics include: hits, page views, visits, unique visitors, referrers, search engines, keywords, time spent on site, exit pages, entrance pages, bounce rate, repeat visits, subscribers and conversion rate (Table 1). These data exist for each page on a website and for each product on the site, allowing comparisons over time and across sites. For this study we chose Google Analytics (GA) software because it is widely used already, including by most of our partner organizations, and because it offers a range of useful tools to analyze the data it provides. GA also allows our partner organizations to give us access directly to their data, facilitating our analysis.

6
Table 1 Google Analytics web metrics (Clifton, 2008; Ledford & Tyler, 2007; Page, 2008) Definition The general point for all analytics information. Clicking on any clickable point on the dashboard will take you to the in-depth analytics section of that point. The map overlay is a visual cue to see how many visitors from which countries have visited the site. Additionally, the map overlay can be broken down by region, province/state, and city thereby comparing specific sections of the world with each other. Shows a segmentation of visitors: what language/s they speak, where their network is located and which browser/operating system they use. There is also a section (with a pie graph) demonstrating percentage of visitors who are new versus who are returning visitors. Although this section is helpful on its own, it is more helpful to use as a comparison tool comparing between months the amounts of visitors using it as a comparison tool could tell us if the new visitor in one month has become a returning visitor by looking at increases and by looking at the visitor loyalty breakdown. Shows where visitors are being referred from to the website such as search engine link, another website, and direct traffic to the site. The specific information relating to content for each page and/or document of the website. Includes how many people visited the page and the percent of page views. Often includes the unique views of each page. This is a summary of visits, page views, pages per visit viewed, bounce rate, average time on the site, new visits. Clicking on any one of the headers will bring you to a further analysis of that point. Line graph plotting how many visits a day, spread out by once a week points. Scrolling over the line graph on each of the points, the number of visitors per day pops up. Within this portion all the analytics for visitors can be found. Provides an average time that each visitor spent on the site per day. From this number we could presume or infer how many pages the visitor read, if they downloaded something, or through breaking down the number, how many visitors bounced in and out of the site within a few seconds. Provides a percentage of how many visitors on that day bounced on/off the site within a few seconds of coming to the site. From this statistic, we could presume that the visitor did not find what they were looking for, or that it was the wrong site. The percentage of people and the number (in brackets after the percent), per day of completely new IP addresses tracked coming to the site. The approximate percent and number of pages visitors viewed when coming to the site. The number of unique page views represents the number of individual visitors who have reviewed your pages

Web Metric

Dashboard

Map overlay

Visitor overview

Traffic source overview

Content overview

Site usage

Visits

Visitor trending

Time on site

Bounce rate

Absolute unique visitors

Average page views

Unique page views

The first image that one sees on logging in to GA is the dashboard (Figure 2).

Figure 2. CEA Dashboard from Google Analytics September 1, 2009- April 26, 2010. From the dashboard, users can view different reports to understand what pages visitors view, where visitors come from, and what products visitors access. Table 1 describes the definitions of different web metrics. In this paper, we specifically report on nine metrics: Content, site usage, visits, time on site, bounce rate, absolute unique visitors, page views, average page views and unique page views. There are, however, limits to what Google analytics can tell us. While the analytics tell us about frequency of downloads of different formats of products (for instance full reports versus executive summaries) they do not provide information about who visits the site or, more importantly, about what people do with the research information after their visit. Since actual use of resources is our fundamental interest, we developed a two survey model to use in conjunction with web analytics to deepen our understanding of the use of web-based materials. A two-part survey We are using a two part survey. When people visit one of our partner sites, they are invited to take part in a short survey (Appendix A) that asks them about whether they found useful information on this visit to the site and about their plans for using any such information. They are also invited to take part in a second survey (Appendix B), to be sent to them at a later date, that asks about their actual use of the materials or resources since their initial visit. The second survey is being circulated to those who volunteer either 30, 60 or 90 days after their initial visit. Both surveys (Table 2) focus on whether the research-related products or resources are used at all and, if so whether this use is conceptual (informs thinking on future issues, and so on) or instrumental (affects the users thinking on research, work, or practice; impacts how the user does work in their context, and whether or not the participant shares the information with others formally or informally inside or outside their organization). Table 2 Survey questions in relation to type of use and time Intention /Use over No use Undetermined Use at this time Immediate usefulness Intended future use Actual Use (as determined by

8
Time Type of Use No use Conceptual Use Instrumental Use Q8 Q10 Q7, Q11 Q10 Q9, Q13, Q14, Q15, Q16 follow-up survey)

Symbolic Use Level of Impact Q12

Partner organizations We currently have either have in place or are about to have in place eight partner organizations; two in Canada and six in England. Each partner organization is involved in attempting to share research information in education through making it available on their website. We hope to recruit additional partners in the coming year; there is in principle no limit to how many organizations could take part. Partners have very little work to do; they have to provide us with access to their GA data, to embed some tracking codes on particular pages and products, and to embed our initial survey on their site. The benefit for the partner organizations is the data analysis and reporting we provide on the use of their web-based research related materials. We also provide each partner with comparative data on the other study participants (anonymously). This will allow organizations to see how the take-up of research resources on their site compares with other educational organizations and should help them improve their sharing of research-related products. Since each educational organization has different goals and, as a result, different content and layouts of their websites, we work with each partner to identify and track some particular targets on their website. Targets can refer to a number of different things depending on the website a particular web page, a product, an initiative that is linked to multiple products, and so on. Our analysis then focuses on these targets. We use ratios to compare different targets in order to gauge intensity of uptake of research materials in relation to other kinds of information within and between organizations (Figure 3).

Figure 3. Metrics analysis framework examining research-based targets within and between educational organizations.

9
Tracking different targets within a single website allows an organization to compare uptake of different initiatives or products. Tracking several sites over time provides the opportunity to compare them in terms of their power to generate visitors to and downloads of material related to research findings in education. Looking at these data across sites and times will allow us to understand more about how, in general, web-based products are used and which kinds of approaches seem to have the greatest impact.

A preliminary example from piloting the project with CEA The research partner whose data we report in this paper is the Canadian Education Association (CEA). The mission statement of the CEA, an organization founded more than a century ago, is to initiate and sustain dialogue throughout Canada influencing public policy issues in education for the ongoing development of a robust, democratic society and a prosperous and sustainable economy. The CEA relies on good theory and research evidence as the foundation on which to build shared understanding and commitment with organizations that share their values and purposes (http://cea-ace.ca/abo.cfm, 2009). Because it is a national organization with a small staff it relies heavily on dissemination strategies including its website. We focus our exploratory findings on the Google Analytics data for CEA on three targets: a. b. c. Comparing CEAs research and policy page to other pages in regard to page views, average time on page and bounce rate Comparing which products (PDFs) are accessed the most, with a focus on comparing full reports versus executive summaries Comparing the uptake of two research-based initiatives: What Did You Do in School Today (WDYDIST) and the CEAs study of the Ontario Primary Class Size initiative.

What Did You Do in School Today (WDYDIST) is a research project that gathers survey data from middle and secondary students in schools across Canada to explore their social, academic and intellectual engagement. We tracked five research-related products from this project: National report (52 pages) Summary report (4 pages) Two supporting document reports that included a report on student engagement (26 pages) and a teaching effectiveness framework and rubric (18 pages) FAQ document (5 pages).

Each of these documents is available as a PDF in English and in French in several parts of the CEA website, including on the homepage, the main research page, and a specific WDYDIST page. The New & Noteworthy page also includes various announcements pertaining to the project in June 2009, August 2009, and September 2009 as the media picked up on the project. The Class Size Reduction Initiative is a research project which evaluates the Ontario governments implementation of a class size reduction policy that reduced class size in the 90% of Ontario primary classrooms to 20 or fewer students as of 2008. We tracked six research-related products from this initiative: National Report (22 pages) Executive Summary (2 pages) Evaluation Report (140 pages) Question and Answer document (1 page) Literature review on class size reduction (36 pages) A paper that was in the CEA quarterly magazine in the fall of 2008 (4 pages).

As with WDYDIST, these documents are available as PDF files in both official languages in multiple locations on the CEA website, including the homepage, the main research page in two locations, and from the New & Noteworthy page with announcements pertaining to the release of the full report in February 2010.

10

We have analytics data for the overall site usage, with a focus on comparing research-related pages to non-research pages from September 2009 through April 2010 (page views, unique page views, average time spent on page, and bounce rate). For the research initiatives, we report on data for the product specific targets from February, 2010 (when the appropriate tracking code was inserted) through April 2010. From these three targets we noticed: 1. 2. Visitors tend to view non-research related pages more but to spend more time on pages that have research-related content Non-research pages and resources were more visited and used than research-related pages and resources. Visitors accessed longer versions of reports more than they did short versions where both were available

Visitors tend to spend the most time on pages that have research-related content but view non-research related pages more From September 2009 through April 2010 the CEA website was visited more than 200,000 times. The pages with the most views are shown in Table 3, and are not research related. On these non-research pages, visitors spent an average of 30-50 seconds on the page. In contrast, visitors spent the most time on average on pages that had research related content. On the WDYDIST page visitors spent an average of 2:33 on the page. Similarly, on the Focus on Literacy page, visitors spent 3:49 on the page. Although visitors spent more time on these pages with research-related content, the bounce rate (see Table 1) was also highest on these pages and lowest on the pages that had general information about what CEA does. Table 3 All page views September 2009-April 2010 Rank 1 2 3 4 5 6 7 8 9 10 Page Home page Education Canada publication page About CEA CEA publication page Research and policy main page WDYDIST page FAQ Focus on Literacy page Education Canada Spring 2010 page Focus On main page Page views 25,070 6,663 6,537 6,090 6,089 5,702 5,545 4,467 3,693 3,484 Unique Page Views 18,608 5,007 4,565 4,383 4,169 4,012 4,932 3,752 2,521 2,478 Average time on page (minutes) 1:04 0:47 0:38 0:27 0:45 2:33 1:50 3:49 1:18 0:29 Bounce rate 41.60% 51.55% 39.46% 20.34% 43.90% 68.68% 68.60% 78.76% 52.70% 41.09%

Not surprisingly, the CEA home page had substantially more views than the target research pages across the eight months of tracking.

11

Figure 4. Comparison of home page views to research page targets for CEA. We also compared time spent on the target pages (Table 4). Table 4 Comparison of average time spent on home page, research and policy page and WDYDIST page per month Month Average time spent on page (minutes) Home page September October November December January February March April 1:04 1:01 0:59 0:58 1:18 1:06 1:03 0:56 Research and Policy 0:39 0:48 0:51 0:54 0:49 0:44 0:42 0:34 WDYDIST 2:17 3:02 2:54 3:07 3:45 2:24 1:23 1:49

12
The most time is spent by visitors on the WDYDIST; it should be noted that this page has a series of 2-3 minute videos embedded in it. While we cannot track access to the videos (because they are embedded on the page) this might account for the additional time spent watching the videos from the initiative. In another attempt to compare these data, Figure 5 shows views of Research and Policy and WDYDIST as a percentage of the homepage views.

Figure 5. Comparison of CEA homepage, Research and policy page and WDYDIST page. While access to both research page targets are low comparison to the homepage, in the WDYDIST activity peaked in November 2009 with 1,046 page views that month. This peak corresponded to a media release and additional media attention surrounding the initiative. We explored the ten products on the whole CEA site to see how many would be research related (Table 5). Table 5 Top 10 accessed PDFs from the CEA website February through April 2010 Rank 1 2 3 Page 2009-2010 School Calendar WDYDIST National Report Public Education in Canada: Facts, trends and attitudes (2007) Page views 777 284 176 Unique Page Views 686 270 169 Average time on page (minutes) 2:22 3:35 3:53

13
4 5 6 7 8 9 10 Beyond doing school: From stressedout to engaged in learning WDYDIST Teaching effectiveness framework and rubric Democracy at Risk article WDYDIST Student engagement report KI-ES-KI contact handbook order form Class size National Report A vision for early childhood education and care article 137 134 119 119 98 68 62 119 123 106 106 86 58 61 2:11 2:52 2:09 2:22 2:04 3:20 2:08

The PDF with school calendar information from across Canada was by far the most frequently accessed document. Also consistent among the top ten were the WDYDIST National Report; WDYDIST teacher effectiveness report; the WDYDIST student engagement report and the KI-ES-KI contact order form. In addition to being the top accessed product, the school calendar had an average view time of 2:22, which is similar to the view times of the research-related products. The KI-ES-KI order form had the shortest average time on page, at 2:04. Comparing the uptake of two research based initiatives We were interested in comparing the uptake (measured as frequency of access to the research-related products) of the CEA target initiatives: WDYDIST and the Class size reduction project. We found that the WDYDIST initiative had a greater uptake than the Class size project (Tables 6 and 7). Table 6 Top content: PDFs relating to the What Did You Do In School Today Project February 2010-April 2010 Rank 5 7 19 52 72 Page WDYDIST Teaching effectiveness report WDYDIST Student engagement report WDYDIST National Report Summary WDYDIST FAQ document WDYIST National Report French Page views 134 119 39 16 11 Unique Page Views 123 106 38 15 9 Average time (minutes) 2:52 2:09 2:59 1:36 3:41

Table 7 Top content: PDFs relating to the Class Size Reduction Project February 2010-April 2010

14
Rank 9 15 18 27 61 62 Page External view of the Class Size National Report External view of the Class Size Evaluation Report Literature review of Class Size Reduction Class Size Executive Summary Q and A document Evaluation Report Page views 68 51 41 28 13 13 Unique Page Views 58 44 35 24 13 12 Average time (minutes) 3:20 2:03 1:29 3:06 0:38 0:55

WDYDIST has been up for longer on the CEA website (launched May 2009). It also has its own webpage as well as more diverse products such as videos on the website and document downloads. Hence, WDYDIST applies more strategies in terms of both products and media attention, with frequent news releases at key times within the two year research project. In contrast, the Class Size Reduction project was only released in February 2010 and, so far, there has not been as much space devoted to this initiative on the CEA website or attention by media (there has been only one news release on the project). Visitors accessed longer versions of reports more than they did short versions where both were available We were interested in exploring the frequent claim that readers prefer to access shorter versions of research such as executive summaries. For both these initiatives on the CEA site, in fact, the longer reports were viewed more often than the shorter versions. The WDYDIST student engagement report (26 pages) was viewed 134 times and the teaching effectiveness framework (18 pages) 119 times while the summary report was only viewed 38 times in the reported time frame. For the Class size project, visitors viewed the National Report 68 times (22 pages) and the Evaluation Report (140 pages) 51 times externally and 13 times internal to the organization website whereas the summary report (1 page) was viewed 28 times in the reported time frame. In addition, visitors spent more time on the longer reports than on the summary reports. In the case of WDYDIST, visitors spent an average of 2:59 on summary version of the national report and 3:35 on the long version For the Class size project, visitors spent an average of 3:06 on the summary but 3:20 on the National Report. Survey findings At the time of writing this paper, we do not have enough responses to our online surveys to report any data. Currently about 1% of visitors are responding to this survey while the second, follow-up survey is too new to be able to report take up or results. As we add more partners we will have more data from both of these instruments.

Conclusion Dissemination of research materials through the internet is a ubiquitous practice but although it takes considerable resources, we have virtually no knowledge about its impact. This paper outlines a study currently underway that seeks to fill some of that gap, including outlining its conceptual basis and giving examples of the kind of data it will provide. Web analytics applied over time and across organizations will increase our understanding of the kinds of strategies and products that are most effective in creating attention. Our two part survey, if effective, will start to provide information on how and how much people actually use the materials they obtain from various websites. Both approaches will add to the base of empirical knowledge on effective mobilization of research in education.

15

References Amara, N., Ouimet, M., & Landry, R. (2004). New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Science Communication, 26(1), 75-106. Armstrong, R., Water, E., Crockett, B., & Keleher, H. (2007). The nature of evidence resources and knowledge translation for health promotion of practitioners. Health Promotion International, 22, 254-260. Behrstock, E., Drill, K. & Miller, S. (2009). Is the supply in demand? Exploring how, when and why teachers use research. Learning Point Associates. Paper presented at the Annual Meeting for American Education Research Association, Denver, Colorado. Berta, W. B., & Baker, R. (2004). Factors that impact the transfer and retention of best practices for reducing error in hospitals. Health Care Management Review, 29(2), 90-97. Belkhodja, O., Amara, N., Landry, R., & Ouimet, M. (2007). The extent and organizational determinants of research utilization in Canadian health services organizations. Science Communication, 28(3), 377-417. Biddle, B., & Saha, L. (2002). The untested accusation: Principals, research knowledge, and policy making in schools. Westport, CT: Ablex. Clifton, B. (2008). Advanced web metrics with Google analytics. Indianapolis: Wiley Publishing Inc. Cooper, A., Levin, B., & Campbell, C. (2009). The growing (but still limited) importance of evidence in education policy and practice. Journal of Educational Change, 10(2-3), 159-171. Cooper, A. & Levin, B. (in press, accepted January 2010). Some Canadian contributions to understanding knowledge mobilization. Evidence and Policy.

16
Cordingley, P. (2008). Research and evidence-informed practice: focusing on practice and practitioners. Cambridge Journal of Education, 38(1), 37-52. Davies, H., Nutley, S., & Smith, P (2000). What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Dede, C. (2000). The role of emerging technologies for knowledge mobilization, dissemination, and use in education, paper commissioned by the Office of Educational Research and Improvement, US Department of education. Retrieved February 2010 from http://www.virtual.gmu.edu/ss_pdf/knowlmob.pdf Greenhow, C., Robelia, B., & Hughes, J. (2009). Web 2.0 and classroom research: What path should we take now? Educational Researcher, 38(4), 246259. Grimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C., Fraser, C., & Vale, L. (2006). Toward evidence-based quality improvement: Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. Journal of General Internal Medicine, 21, S14-20. Hartley, K. & Bendixen, L. (2001). Educational Research in the Internet Age: Examining the Role of Individual Characteristics. Educational Researcher, 30(9): 22 - 26. Hemsley-Brown, J., & Sharp, C. (2003). The use of research to improve professional practice: a systematic review of the literature. Oxford Review of Education, 29(4), 449-470. Hemsley-Brown, J. (2004). Facilitating research utilization: A cross-sector review of research evidence. The International Journal of Public Sector Management, 17(6), 534-552. Ho, K., Bloch, Gondocz, T., Laprise, R., Perrier, L., Ryan, D., Thivierge, R., Wenghofer, E. (2004). Technology-enabled knowledge translation : Frameworks to promote research and practice, Journal of Continuing Education in the Health Professions, 24, 90-99. Jadad, A. (1999). Promoting partnerships : challenges for the internet age, BMJ, 319, 761-764. Knott, J., & Wildavsky, A. (1980). If dissemination is the solution, what is the problem? Knowledge: Creation, Diffusion, Utilization, 1(4), 537-578. Landry, R., Amara, N., & Lamari, M. (2001). Utilization of social science research knowledge in Canada. Res Policy, 30, 333-349. Lavis, J., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? The Milbank Quarterly, 81(2), 221-48. Ledford, J., and Tyler, M. (2007). Google analytics 2.0. Indianapolis: Wiley Publishing Inc. Lemieux-Charles, L., & Champagne, F. (2004). Using knowledge and evidence in health care: Multidisciplinary perspectives. Toronto: University of Toronto Press. Levin, B. (2004). Making research matter more. Education Policy Analysis Archives, 12(56). Retrieved November 15, 2008, from http://epaa.asu.edu/epaa/v12n56/ Levin, B. (2008, May). Thinking About Knowledge Mobilization. (Paper prepared for an invitational symposium sponsored by the Canadian Council on Learning and the Social Sciences and Humanities research Council of Canada, Vancouver).

17

Levin, B., S, C., Cooper, A., & Mascarenhas, S. (2009). Research use and its impact in secondary schools. CEA/OISE Collaborative Mixed Methods Research Project Interim Report. Levin, B. (2010). Theory, research and practice in mobilizing research knowledge in education. Paper presented at the 39th Annual Canadian Society for the Study of Education Conference, Montreal, Quebec. McLaughlin, M. (2008). Beyond misery research. In C. Sugrue, (Ed.). The future of educational change: International perspectives (pp.176-185). London and New York: Routledge. Nutley, S., Walter, I., & Davies, H. (2007). Using evidence: How research can inform public services. Bristol: Policy Press. Page, R. (2008). Web metrics 101: What do all these terms mean? Retrieved on November 3, 2009 from www.makeuseof.com/tag/web-metrics-101 Pfeffer, J., & Sutton, R. (2000). The knowing- doing gap: How smart compaies turn knowledge into action. Boston: Harvard Business School Press. Qi, J. & Levin, B. (2010). Strategies for mobilizing research knowledge: A conceptual model and its application. Paper presented at the 39th Annual Canadian Society for the Study of Education Conference, Montreal, Quebec. Timperley, H. (2010). Using evidence in the classroom for professional learning. Paper presented at the Ontario Education Research Symposium, Toronto, Ontario, Canada. Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39(5), 426-431.

Appendix A: Initial CEA Survey

18

19

20

21

22

Appendix B: Follow-up Survey

23

24

25

26