Leveraging Technology to Gather, Analyze, and Use Assessment Data for Program Improvement

WASC Conference April 12, 2013 Dr. Kathy Theuer, Associate Dean and Director of Accreditation Dr. Carla Piper, Assessment Coordinator and Instructional Designer

Brandman University
Part of the Chapman University System Private/Non-Profit School of Education, 1 of 5 Schools 26 campuses across California, Washington & Online Offer Fully Online & Blended/Hybrid Courses
2

Credential Programs
Multiple Subject Teaching Credential Single Subject Teaching Credential Special Education Teaching Credential Administrative Services Credential Pupil Personnel Services Credentials
School Counseling School Psychology

Degree Programs
Bachelor of Arts in Early Childhood Education Master of Arts in Teaching (MAT) Master of Arts in Education (MAE) Master of Arts in Educational Leadership and Administration Master of Arts in Counseling Ed.S in School Psychology Master of Arts in Special Education Doctor of Education in Organizational Leadership
4

Accreditation

The Challenge
Gather and store data from multiple sources Aggregate data by program Disaggregate data by: Campus Delivery model (blended/fully online) Term/Academic Year Analyze data and generate reports for multiple accrediting bodies
6

Creating an Assessment Infrastructure

Platforms

People

Processes

The Platforms
Assessment Management System

Paired with our Learning Management System

Processes
What data is collected? Who gathers the data? When is data collected? Where is data stored? Who analyzes data? Who provides technology training? Who addresses technology issues? Who monitors participation in the assessment process?

Assessment Team
Director of Accreditation- Oversees assessment process, coordinates data collection, analysis and report writing, collaborates with assessment team members Assessment Coordinator- Builds and copies rubrics each term, generates reports, oversees calibration of instructors Accreditation Technical Coordinator- Manages fieldwork and TPA sites, addresses technical issues with LiveText
10

Assessment Team
Accreditation Team- Reviews data with curriculum teams, shares data with stakeholders and prepares accreditation reports Center for Instructional Innovation- Provides training for students and faculty on the use of LiveText

11

Data Sources
Course Embedded Signature Assignments linked to PLOs Fieldwork Evaluations/Field Placement Data Student Disposition Assessments Demonstration of Mastery Portfolios Teacher Performance Assessments
12

Course Embedded Assessments

Example: Signature Assignment EDUU 609 Curriculum Comparative Analysis Paper Student View- Blackboard Instructor View in LiveText Generating Reports
Instructor Level Program Level
13

Student View in Blackboard

14

Instructor View in LiveText

15

Instructor/Student View in LiveText

16

Instructor Report

17

Aggregated Results

18

Disaggregated Results

19

Inter-Rater Reliability

20

Disposition Assessments
Excerpt from Brandmans School of Education Professional Code of Conduct
The National Council for the Accreditation of Teacher Education (NCATE) defines dispositions as professional attitudes, values a nd beliefs demonstrated through both verbal and nonverbal behaviors as educators interact with students, families, colleagues and communities. These positive behaviors support student learning and development. The School of Education at Brandman University has adopted a set of professional dispositions that we feel are essential for prospective educators to possess and demonstrate throughout the credential/masters program. The dispositions adopted by Brandman Universitys School of Education are listed below. The behaviors listed are examples of the kind of conduct expected of students in the School of Education, but are not intended to be inclusive.

Professional Demeanor & Responsibility - punctual and prepared for classes and field experiences, dresses appropriately in professional
settings, responsible for meeting deadlines and following instructions, accepts responsibility for own actions, capable of dealing with disappointment, flexible, adapts to change, reacts appropriately under stress, exhibits a positive attitude toward schools, teaching, students and parents

Commitment to Learning for ALL Students - shows sensitivity to all students, respects individual differences, responds to the needs of
students and their families and communities, embraces diversity to improve student learning, demonstrates compassion, patience and fairness

Communication - has command of written language (mechanics, usage, spelling) and oral English language, uses standard English in writing and speaking,
demonstrates responsive listening skills, able to communicate with a variety of audiences

Collaboration - is skilled in expressing self in groups, works effectively with professional colleagues and other adults, able to accept and respect different points of
view

Self Reflection - engages in problem solving and self-evaluation, accepts and acts upon constructive criticism and is receptive to new ideas, uses feedback to make
improvements, strives for personal and professional growth

Ethics - demonstrates truthfulness and honesty, maintains ethical and legal behaviors in interactions with others, maintains confidentiality, respects the intellectual property
of others by giving credit and avoiding plagiarism/cheating

21

Aggregated Results

22

Disaggregated Results

23

Fieldwork Placement Data

- Create a database of partnerships statewide (over 500 signed partnerships) - Assign Master Teachers and University Supervisors - Provide a shared workspace for student teacher/master teacher/supervisor - Evaluate student performance
24

25

Fieldwork Shared Workspace

26

Fieldwork Assessment

27

Demonstration of Mastery Portfolios

1. Student completes online application form in LiveText 2. Student builds and submits portfolio in LiveText 3. Technical Coordinator matches student with with assessors 4. Portfolio is assessed by multiple reviewers 5. Technical Coordinator lets student know if they have passed 6. Student records are updated accordingly
28

Portfolio Application Form

29

Portfolio Template

30

Portfolio Template

31

Assigning Assessors

32

Scoring Portfolios

33

Teacher Performance Assessments

1. Student completes online application form in LiveText 2. Student submits TPA 3. Technical Coordinator matches student with assessors 4. Technical Coordinator lets student know if they have passed 5. Student records are updated accordingly

34

TPA Application Form

35

Teacher Performance Assessments

36

Assigning Assessors

37

TPA Rubric

38

Benefits of Technology
LiveText has allowed us to: Gather and store assessment data easily Establish consistent expectations of instructors across multiple campuses Assess inter-rater reliability Monitor instructor and student participation in assessment activities

39

Benefits of Technology
Develop a database of fieldwork sites and placements Utilize multiple assessors for demonstration of mastery portfolios and teacher performance assessments Conduct blind reviews for portfolios and teacher performance assessments

40

Challenges of Technology
Training for students & faculty Getting students registered for LiveText Monitoring instructor use of LiveText Revising rubrics

41

Questions???
For more information contact:

Kathy Theuer theuer@brandman.edu Carla Piper piper@brandman.edu

42