School of Medicine

STUDENT EVALUATION FOR C21

Dr Brian Jenkins
Sub-Dean Quality, Governance and Compliance, Institute of Medical Education, Cardiff University

OBJECTIVES
! " To promote consistency of student evaluation in C21 ! " To allow benchmarking with other programmes ! " To develop horizontal and longitudinal comparators of

CHANGES IN C21
Core Questions
Both Cardiff University (CU) and the GMC have piloted core evaluation question sets that explore the students' perspective on issues such as quality of teaching, resources, assessment and feedback. These standardised question sets are being used in C21. The CU core questions are being used in the end of year evaluations, and the GMC core questions are used for clinical blocks of teaching, usually within NHS organisations. The reporting for the GMC core questions will use a different 'traffic light' format to what was previously used. Both numbers and colours are used to add detail to the reports whilst preserving readability. They have been introduced in years 1 and 5 for 2013-14, and will be rolled out to other years in 2014-15. Extra questions that target Welsh language issues will also be rolled out in 2014-15. The use of standard core questions will allow benchmarking of student evaluations with other programmes in CU and with other GMC pilot sites. This adds elements of externality and transparency to C21 quality governance arrangements.

student experience
! " To explore better methods of qualitative feedback

BACKGROUND
Learning is of course the main purpose of any university programme, and this is judged by student assessments managed by programme staff. However, it is the responsibility of the staff that manage the curriculum to maintain a learning environment for the students that is fit for purpose. Student evaluations are a measure of student satisfaction with the curriculum, and provide evidence of educational quality and governance for stakeholders and regulators. Many evaluations take the form of a questionnaire which typically include questions about resources (such as supplied learning materials, library facilities), support mechanisms (such as academic mentors, remediation) and the quality of teaching.

Qualitative Questions
Online free text boxes will continue to be used in C21 to collect qualitative data in C21, but there are some advantages to using representative groups of students that meet with educational staff to collect qualitative data. Not only can problems be identified, but there can also be a helpful dialogue between staff and students to agree a plan of action. This avoids delays in the quality action cycle and encourages students to engage with curriculum management. Focus groups are being trialled for the second part of the Year 1 in 2013-14 during the new case based learning sessions. The Improving Student Experience (ISE) method of meeting with representative groups of students will be continued in C21. This is a valuable method of obtaining direct student feedback from representative groups of students, and the popular Dean's Question Time will also continue in C21.

Quantitative Questions
The use of Likert scales may be used to produce a series of student responses to posed statements. These are scaled from negative to positive in terms of student experience. In order to aid interpretation of evaluations that often produce large amounts of data, results may be usefully summarised in 'traffic light' format, ranging from 'green' positive responses to 'red' negative responses, with amber as intermediate (Fig. 1). Fig. 1: Traffic Light report

How could the new student handbook be improved? Do you feel that you had adequate learning resources at the hub?

If you had problems, were they resolved to your satisfaction?

What did you enjoy the most?

This shows an example of the use of core questions for benchmarking. Students in different locations throughout Wales were asked the same (core) questions, and the results used to compare student satisfaction between locations. In this case the student evaluations of (the fictitious) St. Iolo's were compared with the All-Wales average for 15 core questions. The percentages quantify the agreement of the student population with the positive statements listed in the first column. It can be seen that St. Iolo's has fewer positive student evaluations than the all-Wales average, indicating a potential educational quality problem that may need investigation.

Actions
Action plans resulting from negative student evaluations will be more tightly managed than in past years. A list of outstanding actions will be kept and only signed off when there is evidence of resolution. This will form part of the evidence for educational governance. Examples of good practice will be more widely disseminated as part of a programme of enhanced engagement with educational partners in 2014-15.

Qualitative Questions
These are open-ended questions used to identify issues that are important to the students but have not been addressed by the quantitative questions, or to provide more specific feedback on new curriculum developments. A typical method is to use text boxes in a online survey, with the student responses summarised by thematic analysis. For end of year evaluations we usually categorise qualitative responses into Top 5 (positive responses) and Bottom 5 (negative) responses to give a focus to the action planning that results from evaluations.

OTHER CONSIDERATIONS
Student evaluation requires good communication with staff and students and the new changes will need more administrative support and organisation. Evaluation scores can be influenced by many factors other than what it is desired to evaluate. Poor quality of teaching may be compensated for by helpfulness of front line staff. There may be misconceptions by students about what should be taught. There are both halo and negative halo effects. Some aspects such as assessments will never by evaluated as well as say, small group teaching for obvious reasons. Student evaluations and their management provide important evidence of quality, but interpretation is needed and potential confounding factors need to be taken into account.

Actions
Negative evaluations should generate corrective actions, and further evaluations should demonstrate that corrective actions were successful. The whole educational management cycle of evaluation, planning, action and re-evaluation provides evidence of good educational governance required by stakeholders such as the Welsh Government and regulators such as the General Medical Council (GMC) and the Quality Assurance Agency.