Students Evaluation of the OSCE experience in Cardiff Medical School

Within Cardiff Medical School, despite several years experience of conducting OSCEs, no such study has been reported so far to investigate students perception of the quality, acceptability and usefulness of OSCE exams. Whilst timely feedback to students from all assessments has been emphasized in schools policy; this study could therefore make a useful contribution to the research related to students evaluation of the OSCEs and the information obtained from diverse groups of students could be used as the basis for improving and refining the quality of clinical exams and to help teachers and clinicians better understand students perception towards clinical exam. Students as major consumer of the assessment play a key role in determining the quality, acceptability, and usefulness of the clinical exams. Appropriate feedback from students could be used for improvement and refinement of the design, implementation, and teaching of clinical skills and competencies assessment. Unfortunately, research literature on students evaluation of clinical assessment experience is still sparse and only a few studies have been reported to date in the published literature on students perception of OSCE experience.

Institute of Medical Education, School of Medicine, Cardiff University

Authors: Dr Saadia Tayyaba, Professor Kamila Hawthorne and Professor Phil Smith

Introduction

Table 1: Students rating of the OSCE station competencies

Stations/Tasks Level of task difficulty (%) Easy 4 6 9 Clinical Skills Stations Average Difficult 67 72 84 30 21 7

Objective

Evaluation of students perception of the quality, fairness, and usefulness of the clinical exams and their general attitude towards OSCEs To investigate students perception regarding assessment of various station specific tasks and competencies To investigate variation in students perception of quality, fairness, and usefulness of clinical exams across diverse students groups

Students Evaluation Questionnaire

Communication skills stations

Investigation and management questions

Clinical interpretation of the symptoms

Explanation to the patients

Identifying correct physical signs; summarizing salient findings to the patients

Differential diagnosis with justification

Examination (fluency, technique)

Not Enough Average Enough 15 13 12 55 59 53 29 28 35

Adequacy of the time (%)

14 5 5

74

74

73

21

20

12

13

11

18

62

62

57

25

27

26

A 28- items structured, self administered questionnaire was developed to gather information related to students perception of following four domains of the OSCE experience: Quality: OSCE scenarios reflected real life simulation and Time allowed to complete a task was adequate at each station. Some additional ratings on station specific tasks and competencies were also obtained in terms of the difficulty, adequacy of time allocated, and perceived degree of learning gained and needed. The scale reliability was estimated to be 0.75. Data were collected from 335 final years (Y04-Y05) students with the help of OSCE facilitators at the end of each OSCE using a selfadministered questionnaire. Ethical approval for the study was obtained beforehand, participation was voluntary and students were informed about the objectives and confidentiality of the information obtained. The sample included 18.3% ethnic minority students, 14% Welsh speakers, 9% mature, and 8 % students with some known disability. Attitude: OSCE makes me feel like a doctor today than a student and OSCE was less stressful than the written exams. Fairness: Overall, OSCE was fair in assessing students knowledge and skills and OSCE examiners were objective. Process of OSCE: example question OSCE process was known beforehand.

Building rapport, non-verbal behaviour

Listening attentively /summarizing to the patients

Background information (past medical, social, drug history)

History/ information gathering on presenting symptoms

Introduction (greeting, explanation of the role)

48 17 22 30

49 78 73 63

3 4 4 6

27 21 20 20

45 55 59 58

28 24 21 22

34

60

21

52

26

Sample and Procedure

Figure 1: Students perception of the OSCE process

Results

The findings clearly indicate a clear majority of the diverse groups of students (including over 80% home and ethnic minority students, students with some disability, and language groups) had a higher overall agreement with the process, fairness, and quality of the OSCEs, and felt that the OSCEs accurately assessed their skills and competencies in core areas of curriculum. A small majority of students also reported some stations required a higher degree of learning than they had achieved and some open-ended comments identified a need for more clarity of instructions and higher time for completion of tasks at these stations.

What did the student say about the OSCE process?

Over 70% home and ethnic minority students agreed time allowed to complete tasks was adequate; however there were some open-ended comments from students where they pointed out acute station as having insufficient time for scenario and skill examination. Over 75% students with some disability were also happy with the amount of time allocated to complete station specific tasks.

A small majority of students (22%) reported the space for the OSCE was inadequate and not noise free. However, there was no relationship between students performance on a station and some complaints about the adequacy of space.

A vast majority of home and ethnic minority students, (over 90%) agreed patients were generally cooperative and OSCE facilitators were friendly.

Figure 2: Students' perception of the OSCE Fairness

A small majority (less than 5%) home and ethnic minority students and about 9% students with some disability disagreed on clarity of instructions to complete the task, A cross tabulation showed these students also found clinical interpretation of symptoms, investigation and management tasks to be more difficult compared to students who found instructions to be clearer. There were some concerns over the adequacy of the space, 23% overall, 13% ethnic minority, and 29% students with some disability reported the space was not adequate and noise free, some open ended comments specially highlighted acute station.

How fair were the OSCEs in assessing students skills and competencies

In terms of diversity, although around 66% ethnic minority students strongly agreed that OSCE was fair in assessing students knowledge and skills and examiners to be objective (67.4%), a small majority (26%) found examiners to be intimidating. Over 80% students with disability perceived OSCEs to be fair, agreed examiner were objective (67%) and gender of the patient did not affect their performance (95%).

The overall perception on fairness of OSCE was higher among diverse student groups and there was no significant variation in the perception of the OSCE fairness across these groups. Overall, majority of the students (over 60%) agreed OSCE examiners were objective, not intimidating (62%), personality, gender, and ethnic attributes of the patient did not affect their performance (70%), they got enough information beforehand on what would be covered in the OSCE (78%).

Figure 3: Students' perception of the Quality of the OSCEs

Were the students happy with the quality of the OSCEs

Content coverage: Majority of the students (88%) appreciated quality of the OSCE exams in terms of content coverage of the clinical skills and competence Order of the Stations: over 80% students including ethnic minority, Welsh speakers, mature students, and students with some disability agreed the Order/ sequence of the stations was logical and comfortable. Consistency of the tasks: Around 86% students strongly agreed with the taught skills and instructions to complete the station tasks.

Discussion and Conclusion

The findings clearly indicate there are no significant differences in perceptions of the diverse groups of final years students (including home, ethnic minority students, students with some disability, and language groups) on the process, fairness, and quality of the OSCES.

Vast majority of the students believed that OSCEs were fair in assessing their skills and competencies, covered broad range of skills, they gained a higher degree of knowledge in all the stations and the gender, personality or ethnic attributes of students did not have an impact on their performance. There were some concerns on time allowed to complete station tasks was not adequate and recommended in open-ended comments to increase time for future OSCEs, particularly for the acute station. Over two third of all the students indicated the OSCE facilitators were friendly and cooperative.

Figure 4: Students attitude towards the OSCEs

Although substantial proportion of the students agreed they were given enough information and resources on what would be covered in the OSCE and knew the process beforehand, over 80% students reported OSCE caused them to be nervous and they found it more stressful than the written assessments. It would be interesting to include intermediate MB students in future studies to find out if there is a variation in the perception of the OSCE fairness and quality across senior and junior students.

References

Allen, R., Heard, J., Savidge, M., Bittengle, J., Cantrell, M & Huffmaster, T. (1998). Surveying students attitudes during the OSCE. Advances in Health Science Education. 3, 197-206.

Contact information: tayyabas@cardiff.ac.uk hawthornek@cardiff.ac.uk smithpe@cardiff.ac.uk

Shitu, B., Girma, T. (2008). Objective Structured Clinical Examination (OSCE): Examinees perception of department of paediatrics and child health, Jimma University. Ethiopian journal of Health Sciences. 18,2, 47-52