___________________________________________ _____________

Final Evaluation Report

Evaluation of the Online Teaching Tool Elluminate

___________________________________________ _____________

Norma Jordan
EDTECH 505 Section 4172

July, 27 2011

Table of Contents
Section 1 2 3 4 5 6 7 8 9 10 Description Cover page Table of Contents Learning Reflection Executive Summary Purpose(s) of the Evaluation Background Information Description of Evaluation Design Results & Discussion Conclusions & Recommendations Appendices (includes EPD & Timeline) Page

3 4 5 6 7 9 15 16

Learning Reflection Before entering this class I knew very little about evaluation. It has been a challenge to grasp all of the information over the past eight weeks, but empowering at the same time. I feel like I am better prepared in my current career as an educator, but the knowledge I have gained also gives me leverage in any career that I choose to pursue from here. I have learned so much about evaluation this semester. Telling you everything I have learned would take far more than a page, so I have chosen a few aspects that really stand out to me. First, I have gained an understanding of why we evaluate. It is important and necessary to understand how programs are working and if they are meeting the needs they were designed to meet. We evaluate to bring about a better product in the end. Second, I gained an understanding of the difference between evaluation and research. The line between the two is not incredibly large, but now I can see the difference. I now understand that we do research to collect data, create a hypothesis, and create new knowledge. We evaluate to gain a better understanding of what is already in place. I also gained an understanding of the differences between qualitative and quantitative data, and how to go about collecting each. Years ago, right after my bachelors I took a research and writing class. I left that class with an A, but I can honestly say that I was confused about the differences between the two data forms. I think that two factors made the difference in my new understanding, one being the fact that I am more mature and have a little life experience behind me, and two that the information was presented in such a way that the difference made sense to me. Although I have learned many things, the last thing that I want to address here is that Evaluation has different models to follow. I dont know what I thought before this class, I think I believed that evaluation was mostly about opinion, but reading about and learning more about the models to follow has been a great learning experience. By doing the final project, I learned that even when you think you have covered everything, there will always be something that was missed, and data collection needs to be from several devices to get the most accurate information possible. I also learned that evaluators can not control all of the factors in an evaluation, nor is it their job to attempt to control these factors, but simply to state what was found from the data gathered. The information that I have gained here will be useful and something I can refer to often in my future, both as an educator and an educational 3

technologist. I enjoy writing grants, and I think that now I know what I do about evaluation my skills will be improved. I plan to revisit this information often in the hopes of building my skills and becoming better at it.

Executive Summary Synchronous Teaching and learning on the Internet is quickly becoming a standard means of educating K-12 students online. This is a way of offering younger students an alternative to education in their local school. Schools that hold such synchronous teaching sessions include K-12 Inc, Desire2Learn, and various other private and public schools. One tool that is available to host these synchronous sessions is Elluminate Live, now known as Blackboard Collaborate. Elluminate is a collaborative web conferencing tool that enables moderators (teachers) and participants the ability to share voice, video, and text with each other in real time. The interface contains a whiteboard, chat, and desktop applications for teaching and learning. Currently 29 of 50 states have online schools that use K12 curriculum and host synchronous teaching sessions for their K-8 Students using elluminate. The large volume of students nationally that are being schooled in this manner continues to grow. These circumstances encourage an evaluation of such software to determine if students are able to get the same education that they would in a regular classroom with a teacher looking over their shoulder. Does elluminate offer the right tools to keep student attention, and are teachers pleased with the software. What could be changed to improve its use? It was determined that data would be collected in two ways for this evaluation. A survey would be sent to teachers and administrators who currently use elluminate as a primary way of teaching their students. The survey would determine if teachers felt elluminate live was able to meet student needs and if elluminate provides the right kinds of tools for teaching students in the K-8 age range. The other evaluation tool will be a checklist of observations made while watching old class recordings. The checklist and observations will be used to determine which tools are used most frequently and to observe if the tools appear to be meeting the needs of the students. Class recordings and data collected from the teachers survey indicate that some tools meet the needs of students. It was also determined that many tools are engaging to students and enable teachers to teach from a distance and still offer a personal touch. Teachers were very happy with the overall elluminate software, but would make small changes if possible. It was determined that elluminate does offer a great collaboration tool for K-8 students, but that more research needs to be done in this are to determine if elluminate meets all of the needs of K-8 students. 4

Purpose(s) of the Evaluation Online teaching of younger students is becoming more and more popular in the United States. Elluminate is currently used by over seven million teachers K-16 and continues to grow at a rapid pace. It was developed to be used to extend the physical boundaries of a traditional classroom to students who are being schooled at home. Elluminate is said to give students live interaction with a teacher, promote active learning, improve student performance, facilitate online communities, and improve comprehension for younger students who choose this method of education. Teachers are given the use of elluminate to host their live online classrooms. Most schools use live asynchronous sessions for younger age students. The purpose of this evaluation is to see if elluminate provides the training and tools needed by online elementary school teachers to effectively teach students online. The evaluation will determine if elluminate provides tools that help elementary teachers teach k-6 students effectively. Does elluminate provide the tools needed to engage and enhance distance learning for younger students? What tools could be changed or enhanced to make the program better. Central Questions: 1) What kind of training do teachers receive before hosting a live classroom on elluminate? 2) How often do teachers use certain tools within their live classes? 3) Which tools do teachers perceive as helping them be a successful teacher? 4) Which tools meet the needs of the class members? 5) Which tools keep students engaged? 6) What technical difficulties does elluminate need to address in the software to make it better? Who is impacted by the results? Teachers and administrators of schools that use elluminate to teach online classes to K-6 grade students would be the most impacted by this information. Followed by parents of students that are schooled through elluminate. Lastly K12 Inc. may be interested in the findings because they are currently partners with many schools that use elluminate. What was the origin of the program or product? 5

Elluminate Inc. was founded in Calgary, Alberta in 2000. It is a web based conferencing program that allows users to synchronously meet online. Elluminate was designed with a focus on education. It was developed to meet the needs of online schools and companies that need to meet and collaborate in a real-time setting. Elluminate offers both synchronous sessions, but the user may also create a recording to be accessed at anytime, anyplace. Background Information: What are the standards and/or goals of the program or product? Elluminate is a leading provider of online collaboration products and tools for teachers. In 2010 elluminate was serving seven million teachers and students in 170 countries. The goal of the product is to provide a real time synchronous teaching platform for teachers to use to educate their students. What did previous programs or products look like? If others existed, what was learned from them? As of today, elluminate live has been bought by blackboard and is now called blackboard collaborate. The interface of blackboard collaborate has not changed from elluminate, it looks the same once inside the classroom. WebEx was the first online collaboration environment developed in 1996, but is more focused towards business. Elluminate followed in 2000, but has more of a focus on education. Macromedia Breeze came about in 2003, but in 2006 became known Adobe Connect. It too has a business focus. There are quite a few different platforms available now, but it seems that Elluminate, Adobe Connect, and WebEx are the most popular. I am unsure of what WebEx looked like in the beginning, other than it had a focus in business collaboration. Elluminate is the first online teaching collaboration environment. Who are the people involved with the program: Elluminate was founded by two men in Alberta Canada, Nashir Samanani and Mike Mabey. Nashir graduated with a Bachelor of Computer Science Degree from the University of Calgary. He has founded several technology companies including being co-founder of elluminate. Nashiers current role is the chief executive officer of elluminate. Mike Maybe is co-founder of elluminate. He is responsible for the companies technology vision and implementation. He is the Chief technology officer and founder. Program Characteristics: Elluminate was designed with education in mind. The program includes a variety of ways to hold class sessions online, monitor student activity during class sessions, and enable communication between all participants. Elluminate provides tools that allow group work, class recordings, and the ability to publish recordings for playback at a later time. Elluminate allows the moderator the ability to bring in PowerPoint 6

presentations and control what the participant sees. The moderator can also give the participants the ability to navigate on their own. The elluminate classroom can be controlled by the moderator. Class tools can be given immediately upon entering the class, or handed out by the teacher at different intervals. Tools can also be taken away if necessary. Elluminate will also allow participants to work in groups on the same screen. The participants can have discussions related to their project using the microphone tool. Elluminate's virtual meeting room has a shared whiteboard to write and draw on if given teacher permission. The participants can mark up the presentation with answers or notes. The options that elluminate provides goes on and on with the use of more tools like: raise hand button, happy face button, confused face button, polling tools, leave class button, and many other options. Moderators can also give students the flexibility to navigate through the presentation on their own, they can review a presentation at their own pace with the option of going back and forth between slides and the option of taking notes if needed without feeling rushed. Evaluation Design The model that the evaluator used for this project was the goal-free model. The objective was to observe and measure actual data related to elluminate and its current use with K-8 grade students. The evaluator wanted to determine what the program currently provided for K-8 students, and tried to carefully distinguish between what the program is actually doing vs. what the program was intended to do. The evaluator wanted to determine if elluminate met the needs of the current K-8 population, and wanted to provide information to the users so that they could look at the data collected and arrive at a judgment of their own based on the findings. The evaluator has taught using elluminate, so chose the goal free model trying to avoid bias of any kind. It seems that this model offers fewer opportunities for this, so it was the model the evaluator felt best met the needs. The goal free model was also used because it may be used to identify unintended pros and cons. The objectives of this project are as follows: Objective 1: Teacher Experience Determine how many years the interviewees have had teaching online using elluminate. Objective 2: Training Determine what kind of training the k-8 teachers are currently receiving on elluminate. Objective 3: Use of tools Determine which tools in the elluminate suite are used by K-8 teachers. 7

Objective 4: Successful Teaching Determine which tools teachers perceive as the best tools for meeting the needs of their K-8 classes. Objective 5: Meeting Needs Determine which tools meet the needs of the students in k-8 classes. Objective 6: Student Engagement Determine which tools teachers perceive as being able to keep their k-8 students engaged. Objective 7: Determine changes that teachers would like to see to help them better meet the needs of their k-8 students. Instrumentation In order to collect information to be used in the evaluation, the evaluator designed and sent out an online survey to teachers and administrators who currently teach K-8 students online using elluminate. The purpose of the survey is to collect information about teachers and their perceptions of their online classes. The results of the questionnaire were collected and used to form an overall picture of how teachers use elluminate and if they feel it is meeting their current needs. The evaluator also watched class recordings to determine if teacher perceptions were accurate, or if there was a discrepancy between tools teachers believed that they used verses tools that they actually used. Sample: The evaluator sent the survey to teachers that work for a public school in Idaho that uses K-12 curriculum. All teachers within this school use elluminate to host their online collaboration teaching sessions. Going into the evaluation the evaluator did not know the years of experience that the teachers had. It was found that the majority of teacher polled had taught for an online public school using elluminate for between two and six years. Data Collection Procedures: The evaluator developed the survey and sent it to online teachers using two different methods. The evaluator posted the survey on facebook, and also sent the survey by e-mail to colleagues. The evaluator asked the participants to forward the link to any additional teachers that might be willing to participate in the survey. The survey was left open for a two-week time frame to collect data. 8

During this time the evaluator also collected class recordings from the teachers. The evaluator watched class recordings and collected data on which tools were used in k-8 classes, and which tools engaged the students. Data Analysis The evaluator collected all data into an Excel spreadsheet. From there the data was turned into percents based on the number of responses. The evaluator then took the data and shifted it so that all tools that fell within a use are were together so that the graphs would be easier to read. At this point the evaluator chose the specific car graph that would be used to show the data, and turned the graphs and data into charts. The evaluator followed this same procedure for all of the graphs in the results section. Results & Discussion of the Results The results of the survey and class observations were compiled and reviewed to form the following conclusions about elluminate as an online teaching tool of K-8 grade students. Objective 1:Teacher Experience The majority of participants for this survey reported that they had taught online for 4-6 years, followed by participants that had taught for 2-3 years online as seen in chart 1 below. The details of this survey were not surprising because the increase of K-8 students learning online is fairly new. Chart 1

Objective 2: Training The majority of participants reported that their training was a face-to-face, hands on training either offered by K12 inc. or another teacher at the school that they are employed with. Two other types of training that were reported were online training, and being self-taught (Chart 2). Chart 2

10

Objective 3: Use of tools The survey question asked how often do you use the following tools in your K-8 Classes? Teachers reported five different tools that they use in 100% of their classes taught. Those tools include: whiteboard, smiley icons, hand raise, chat and microphone tools. Other tools that were used by the majority of teachers most of the time include the clap/thumbs down tool, clip art, class recordings and time clock. It was determined that three tools were used seldom if ever by k-8 teachers, those tools are shout outs, invite new students, and the notes tool. Other tools in the elluminate suite were used some of the time as seen in the chart below(Chart 3).

11

Chart 3- How often do you use the following tools in your K-8 classes?

Objective 4: Successful Teaching The survey question and class observations were used to determine which tools helped teachers be successful at their job. Both forms of data showed that the whiteboard tool was the most effective, closely followed by the microphone tool and the chat tool. This is a significant piece of information because these three tools are they ways to communicate within the online class. Other tools that were also considered to help the teacher be successful were the hand raise, record class, smiley, and thumbs down icons. These too are also forms of communicating within the online class. Other tools that were seen as somewhat helpful are the tools that allow for interaction with other computer 12

programs, like the web tour, multimedia, and file transfer. Also coming in this category were the tools that allow students the ability to interact with each other, like the breakout rooms. Again we see the following tools as teachers reported as not useful: the notes, shout outs, and invite new student tools.

Objective 5: Meet Needs The data for meeting student needs shows that the whiteboard tool, microphone tools, and the chat tool that are most likely to meet student needs. Sixteen of the nineteen tools addressed in the survey and evaluators observation are said to meet the need of students to some degree. Again we see the three tools that are not used are the notes, shout outs, and invite new student tools. Which tools do you feel help you best meet the needs of your class?

13

Objective 6: Student Engagement The results show that the tools that keep students most engaged are the whiteboard tool, chat tool, and microphone tool, smiley icons, and thumbs up thumbs down tool. Again we see that online teachers reported that the majority of the tools were at least somewhat helpful at keeping their students engaged. Teachers feel that the shoutouts actually move out of the bottom category on this section, and leave the least helpful tools as the note tool and the invite new students tool.

14

Which tools keep your students engaged.

Objective 7: Changes All teachers did address something that they would like to see changed. The majority of teachers had complains about the speed of application sharing, and multimedia tools. They reported that these were difficult to use because the users had big delays in when the teacher derived the content and when the users actually received the content. Another complaint was that when the microphone tool has more than one user there is a horrible echo and they would like to see that improved. Although teachers had things they would like to see changed, they felt that

15

elluminate was a good tool and they were very pleased with its use. Conclusions & Recommendations Immediate Conclusions: It was determined based upon the data collected that the most useful tools for the k-8 online teacher are the whiteboard tools, the chat tool, and the microphone tool. This data can be seen in every chart in the result section. Most of the tools were seen as somewhat helpful to the teacher, and a few tools were seen as not helpful at all. The following are conclusions based on the data: The tools show a bell curve, three tools are very useful, three of the tools were seen as not useful, but the majority of the tools were seen as somewhat useful by all of the teachers. The evaluator does not feel that elluminate is meeting all the needs of online students at this time. The evaluator still feels elluminate is one of the best programs currently available to educators, but according to the survey and data collected it needs to be improved upon so that more of the tools fall into the Always category. Teachers may need additional training on the tools to make them more effective in class. The evaluator determined that maybe some of the tools are not familiar to the teachers and that is why they are not used as often. Long Range Planning It was determined based on the data collected for long range planning and developments that: Teachers may need more training on elluminate. Elluminate may need to communicate more with teachers to determine how to make better use of the tools that are available. Elluminate may need to conduct more research and user tests for younger students to determine their needs. Evaluation Insights If I could do this evaluation differently I would ask more questions on the survey, and get more specific data about certain tools. I would ask questions about teacher comfort level with some of the tools to see if the tools are not used because they are not useful, or because teachers do not know how to use the tools. I think some of the data is skewed in this manner. I was not able to collect as much data as I would have liked. Only 25 teachers took the online survey, I was hoping to have a minimum of 50. I also would have spent more time having users give feedback on the collection tools to make sure I had really thought of all the pros and cons before sending out the survey for data collection. I would have also had more teachers edit the survey and come up

16

with ideas for it. Appendices Time Line 6/27/11 Monday 6/28/11 Tuesday Brainstorm ideas/make plans for the overall project Send project description to Ross Perkins Gather information and collect e-mail addresses and 6/29/11 Wednesday phone numbers of sample population. Gather information and collect e-mail addresses and 6/30/11 Thursday phone numbers of sample population. Compile and write Evaluation's Program Description 7/1/11 Friday (EPD) work on instruments: survey questions, interview 7/2/11 Saturday questions, etc. work on instruments: survey questions, interview 7/3/11 Sunday questions, etc. send drafts of the instruments: survey questions, 7/4/11 Monday interview questions, to peers for evaluation. 7/5/11 Tuesday 7/6/11 Wednesday 7/8/11 7/9/11 7/10/11 7/11/11 7/12/11 7/13/11 7/14/11 7/15/11 7/16/11 7/17/11 7/18/11 7/19/11 7/20/11 7/21/11 7/22/11 7/23/11 7/24/11 7/25/11 7/26/11 7/27/11 7/29/11 Friday Saturday Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sunday Monday Tuesday Wednesday Friday Conduct teacher interviews/ collect surveys Conduct teacher interviews/ collect surveys Conduct teacher interviews/ collect surveys Watch teacher class recordings Watch teacher class recordings Watch teacher class recordings Analyze Data that was collected Analyze Data that was collected Analyze Data that was collected Begin to write final report- draft outline Write Write Write Write Write Purpose and Background Methodology Section Results Section Discussion and Conclusion Sections the Summary section

Work on revising and working on the final report Work on revising and working on the final report Wrap-up Final Project Due

17

Evaluators Program Description

Link to EPD

Data Collection Tools: Online Survey

18

19

20

21

22

Observation Checklist 23

Tools whiteboard tools smiley icons hand raise tool clap/thumbs down tool clip art chat microphone record class shout outs web tours time clock explore screens quiz tool multimedia (video,MP3, etc...) file transfer notes invite new students web cam break out rooms

Frequency of Use

Other tools used by teachers that are not on the survey:

24

25