Evaluation

Planning Document

Jeff Toorongian EDTECH512, Fall 2011 Prof. K. Diane Hall

Evaluation Criteria for WBI: An Introduction to the Canvas Learning Management System Evaluation Criteria Effectiveness: Mastery of WBI learning goals as indicated through completion of Module lessons and activities Appeal: Self-paced WBI is used and well received. Learners work through all training modules. Content is engaging and well received. Video tutorials well received. Explanation WBI modules will have video tutorials and demonstrations. Effectiveness will be demonstrated through successful completion of module activities As self-paced instruction, the WBI will need to attract and retain learner interest to keep them engaged through the entire course. Instruction will need to establish itself as a valued resource for all instructors using the new LMS for their teaching activities. Modules will feature video tutorials reviewing important material in each module. These resources will keep learners engaged and provide important visual learning opportunities. Data Sources -Web-based assessments -Completed module learning objects and products -Learner comments and feedback. -Course facilitator comments and feedback. -Usage data on participant progress through course content (did learners complete all modules, or start and quit?) -General feedback from learners, facilitators, and SME, regarding course content, delivery methods, and levels of learner engagement and overall impressions with course content, structure, and delivery within Canvas LMS. -Feedback from course evaluation will provide details on time invested by learners. Openended comments in evaluation survey will allow learners to comment further on delivery methods, course structure, and content.

Efficiency: WBI provided a short, engaging, and content rich experience for busy instructors

Participants are busy. The goal of the WBI is to deliver self-paced, efficiently presented information that they need to get up and running with a new LMS product. By delivering the instruction within the LMS they receive additional hands-on, practical experience that will further support their successful completion of learning goals.

Evaluation Matrix for Formative Evaluation Questions for WBI: An Introduction to the Canvas Learning Management System

Evaluation Criteria and Categories Effectiveness/Goals

Questions Are the learning goals aligned with instructional materials? Are the learning goals aligned with instructional activities and assessments? Are the activities and assessments achievable in a time that is reasonable for a self-paced course? Do the learners have feedback mechanisms available to them if needed?

Methods and Tools End-user survey, Instructor SME review, Designer expert review.

Effectiveness/Content

Do the video tutorials provide adequate content to achieve module learning objectives? Are the instructional materials accurate and comprehensive enough to support learning goals? Are the instructional materials aligned with learning activities and goals?

End-user survey, Instructor SME review, Designer expert review, Extant data.

Effectiveness/Technology Are learners able to access all instructional materials needed? Does the course navigation function properly? Do the video tutorials open and play properly? Do other learning resources (links) work when accessed?

End-user survey, Instructor SME review, Designer expert review.

Does downloadable content open properly in the target application? Effectiveness/Message Design Is the course structure and navigation clear, intuitive, and functional within the LMS? Are the instructional video tutorials well designed and readable? Is the audio quality of the tutorials conducive to learning without major distractions or technical problems? Is the video quality of the tutorials conducive to learning without major distractions or technical problems? Are visuals clear and easy to read? Are the instructional materials clearly and consistently labeled and linked? Are the lesson activities and assignments clearly described and easily comprehended? Appeal/Goals Do the goals encourage participation from the learners? Are the learning objectives appropriate for the most important skills needed by learners? Appeal/Content Is the web-based instruction engaging to the learner? Do the learning goals provide for active learning opportunities for participants? Does the instruction encourage End-user survey, Instructor SME review, Designer expert review, Extant data. End-user survey, Instructor SME review, Designer expert review. End-user survey, Instructor SME review, Designer expert review.

learners interact with the LMS as they develop new skills? Appeal/Technology Is the instructional content easy and intuitive to navigate? Are there any grammatical or typographic errors that distract from the learning objectives? Do media files and external links work properly? Appeal/Message Design Is the instructional message aligned to the intended learner group (university instructors)? Is the instructional web environment intuitive, clear, and conducive to learning? Is navigation through course content easy and intuitive? Are graphical elements within the web environment attractive, effective and not distracting to the learner. Efficiency/Goals Are learning goals clearly and simply stated? Are the learning activities in alignment with course objectives? Efficiency/Content Is the instruction presented clearly End-user survey, Instructor and without distractions? SME review, Designer expert review, Extant data. Does each learning module represent unique learning objectives? Does the web-based instruction function effectively and without distraction or complications to the learner? End-user survey, Instructor SME review, Designer expert review. End-user survey, Instructor SME review, Designer expert review. End-user survey, Designer expert review. End-user survey, Designer expert review.

Efficiency/Technology

Do the video tutorials cover important and necessary information to help learners achieve goals? Is the audio in instructional media of high quality and support the learner and facilitate a transfer of information? Efficiency/Message Design Are the learning modules unified in design and overall learning goals? Are the learning objectives presented logically and in a manner appropriate for learner success? Formative Evaluation Plan End-user survey, Designer expert review.

Who are the Stakeholders?

Primary Stakeholders: - Course trainer/facilitator will have responsibility for developing course content in instructional modules, designing course layout within the Canvas LMS, and facilitating course participants during course. - eLearning support staff will be responsible for course design (in conjunction with course trainer), support content development and delivery, provide technical support to course participants, and facilitate course administration for course participants. - Course participants are the focus of the WBI. Their success in achieving learning objectives and gaining a strong working knowledge of their new learning management system will be the critical factor that will determine the success of the training effort.

Secondary Stakeholders: - IT staff who support the server-side LMS installation, back end support, and campus application integrations are responsible for keeping the LMS functional. - Senior Administrators, including CIO, Library/eLearning Director, and executive team members are ultimately responsible for the selection and support of the LMS, and providing the necessary resources (staff and funding) to provide training to instructors.

What is being evaluated?

- Course Objectives: Are the objectives that have been developed for the training course appropriate to assure that participants will have sufficient skills with the new LMS to begin developing their course in it? - Course Assignments and Exercises: Do the assignment and exercises developed and presented in the WBI sufficient to assure learner mastery of the skill(s) in question? - Course Assessments: Do the assessments tools adequately measure achievement of the learning objectives? - Design/Navigation: Is the navigation within the course clear, intuitive, and consistent? Are design elements and the graphical display of course content (as experienced within the LMS) appealing, appropriate to the content and learner audience, and convey course information effectively? - Course Content: Are the course materials of high quality, free of errors (grammatical and technical), and robust in breadth and depth of required content? - Delivery Methods: Do the instructional delivery technologies work properly? Does all linked content function? Do all media files play as intended within or external from the LMS, as designed?

Who are the Evaluators and Reviewers?

Evaluator/Designer: An instructional designer, on staff in the eLearning group will provide expert review of the WBI, identify any areas of design weakness, suggestions to improve the design. This instructional designer has more than seven years of practical experience supporting faculty on campus in the design of their course elements within the LMS. They also have extensive technical expertise with troubleshooting and user LMS technical issues. Expert Reviewer/Instructor: An Associate Professor in the university chemistry department will provide technical expertise relevant to the Canvas learning management system. He has piloted several of his online and campus-based courses in the Canvas LMS over the last two semesters. His practical knowledge of Canvas will provide invaluable insight into the WBI content from the viewpoint of an instructor. End-User reviewers: A small group (no more than 10) recent participants in the Canvas WBI course will complete an online user-feedback survey. This reviewer group will be university instructors of various ranks. Their feedback will offer important information about course content, organization, and design from a user perspective. Follow-up interviews with a subset of these reviewers may be conducted if time and resources permit, and are deemed necessary.

What are the Evaluation Methods and Tools?

-Expert reviews from designer and instructor via checklist or written comments. -End-user survey data. -Extant data (see Evaluation Matrix for detailed breakdown each criteria)

When and how should formative evaluation take place? The instructional designer and instructor reviewer will conduct initial review of WBI design plans. At this early stage their input will likely involve observations and suggestions related to course content, organization, and identified learning objectives. An analysis of instructional activities and learner motivational strategies may also be reviewed. As course development nears completion, the instructional designer and instructor reviewer will review course objectives, content, and message design issues to find areas of weakness and identify errors that need addressing. Due to a very small e-Learning support staff and the relatively short format of this web-based course, a limited try-out cycle will be conducted, consisting of a small-group evaluation. This group of up to ten university instructors or other individuals with teaching experience and responsibilities, such a graduate students, will be asked to engage the course content as it is designed and delivered within the Canvas learning management system. The try-out cycle with end-user reviewers will also provide important feedback on the technical quality of the instructional content and the approximate time it takes to complete all course materials. What decisions need to be made as the WBI design plan and prototype are developed and revised? Results from formative evaluation will primarily be qualitative data and will be compiled from survey results and evaluation comments. These findings will guide the design team in their decision-making as they identify the areas of concern and resultant changes to the course materials. Instances where there are divergent opinions regarding needed changes, as identified in the evaluation results, may require consultation with stakeholders for guidance. Comments and completed course assignments from end-user reviewers in the small group try-out cycle will provide important feedback regarding attitudes and newly developed skills. This information should prove valuable in helping to determine needed changes to course assignments, course instructional content, and course design and navigation. Preliminary Plan for Summative Evaluation This is a new web-based training course, so no preexisting data from an original course is available. Results from participant end-user surveys accompanied by their completion of course activities will be the primary methods of confirming success of the instruction in preparing participants to use the Canvas LMS. Preliminary summative evaluation plan criteria, questions, and related data sources are provided in the table below. Results of the summative evaluation findings will be presented in a written report that will be submitted to the Director of the Center for Teaching and Learning and the designated primary course instructor/facilitator. Information on the structure of the evaluation process, data sources, results and any recommendations will be included in the report.

Evaluation Criteria Effectiveness

Main Questions - Did course help learners to achieve learning objectives? - Were the activities and assessments achievable in a time allocated for the selfpaced course? - Do learner perceptions indicate they are prepared to use the Canvas LMS upon completion of the training course?

Data Sources Course assessment activities End-user survey results, and LMS student tracking data.

End-user survey results

Efficiency

- Did each learning module provide unique learning objectives as part of the training course? - Were course instructional materials effective and engaging to the users?

Expert reviewer responses, end-user survey results

Appeal

End-user survey results

- Did the web course design Expert reviewer responses, and navigation support end-user survey results participant learning objectives? - Do course objectives encourage learning participation in WBI? Expert reviewer responses, end-user survey results

2011 Jeff Toorongian