Establishing the reliability and validity of a Virtual Reality Upper Gastrointestinal simulator using a novel video-endoscopic assessment technique.

Moorthy K, Munz Y,Jiwanji M, Bann S, Chang A, Darzi A Department of Surgical Oncology and Technology, Imperial College, London, UK

INTRODUCTION:

RESULTS

DISCUSSION:
There were significant differences across the groups for all parameters. The percentage of mucosa visualized is an indicator of the thoroughness of examination. Even though the intermediate group carried out a more thorough procedure, they were only slightly faster than the novices, as a result of which the efficiency of performance remained unchanged. The fact that there was no difference in between Groups 1 and 2 in terms of the percentage of pathologies identified was of some concern. Playback of the procedure revealed that this was a result of both groups failing to identify an ulcer in the posterior wall of the first part of the duodenum. A majority of the trainees in the intermediate group failed to scan the first part of the duodenum in a systematic and thorough manner. In fact the stimulus for the development of the video-endoscopic assessment came from this finding. The systems reliability of assessment in terms of nearly all the parameters was greater than 0.80. Such a high level of reliability between independent observers is considered to be suitable for even high stakes assessment in surgery. The video-endoscopic assessment is able to discriminate endoscopists with varying levels of experience.

A few studies have established the construct validity and reliability of Groups- performance parameters Face validity questionnaire (5 point Likert scale) The reliability coefficient between the averaged performance on case 4 and 5 was other virtual reality (VR) simulators as tools for assessment. Paramet Novices the procedure and 0.88 for time taken forIntermedi Experts for the percentage of mucosa Media visualized. Construct validity is the ability of the simulator to differentiate Question er (mean+/- ate (mean+/endoscopists with varying levels of expertise. Reliability is the ability The reliability of assessment across the 4 attempts (both cases 4 and 5) was 0.90 for n Graphics 4 SD) (mean+/- of mucosa visualized and 0.86 for efficiency of time taken, 0.89 for percentage SD) of the simulator to assess skills reliably over 3-4 procedures. Face Time 242.7 236.4 183.2 SD) validity is the extent to which the simulation is representative of real performance. (sec) (107.6) (85.3) (49.5) Simulation of ease/ 4 Video-endoscopic score: Significant difference in the score across the groups procedures. % 78.8 (9.1) 86.1 (5.7)85 (5.5) complexity (p<0.001). The level of agreement between the two Force feedback0.90. 3 observers was Demonstration of construct validity is still a crude way of correlating mucosa Correlation between metrics of the case rated by the observers and the simulators Efficienc 0.37 0.40 0.53 performance on the simulator with performance in real life. The assessment(0.17) of performance: Btween the global score Training percentage 5 mucosa and the tool of (0.11) y (0.17) demonstration of this correlation is crucial to the acceptance of VR visualized (Spearmans rho= 0.60, p<0.001), the percentage of pathologies identified simulators as tools for assessment. Assessment tool 4 %age p=0.05) and whether98.7 (7.9) 82.8 89.7 (rho=0.34, or not retroflexion was performed (rho=0.65, It has been difficult to demonstrate transfer of skills from VR to real patholog (22.3) (19.5) procedures primarily because it is difficult to assess performance in p<0.001). y real procedure. An objective assessment tool could correlate performance in real procedures with performance in virtual reality.
KW, p<0.001

AIM:

Establish the validity and reliability of the VR upper gastrointestinal (UGI) simulator Develop a global rating scale for the assessment of skills Correlate the global scale with the simulators metrics
KW, p<0.001 KW, p<0.001

MATERIAL AND METHODS:

CONCLUSIONS:
This study has demonstrated the construct validity and reliability of the simulators metrics. Significant correlation between the video-endoscopic score and the simulators metrics. Video-endoscopic score: novel method for the assessment of skills in UGI endoscopy

The VR system (Simbionix, Israel):

The Subjects: Group 3- experts (>200 diagnostic procedures) Group 2- intermediate (10-50 procedures) Group 1- novices (never performed) The modules: Diagnostic upper GI endoscopy (gastroscopy) module. The participants were permitted to perform case 1 on the simulator for an unlimited period of time to get acquainted with the system. Following which they all undertook 2 attempts each on cases 4 and 5 in a random manner. Case 4: Hiatus hernia with significant reflux esophagitis, ulcer in the posterior wall of D 1 Case 5: Esophageal diverticulum, a polyp in the fundus of the stomach, malignant lesion in the antrum. Study data: Construct validity- simulators metrics, Face validityQuestionnaire on a Likert scale, Video-endoscopic (global) score (Fig 1) Video-endoscopic score: 8 criteria- Examination of esophagus, body, antrum, fundus, duodenum, pyloric intubation, knowledge of the procedure and flow of the procedure- all rated on a 5 point Likert scale. One case of every participant played back to two independent blinded observers. Data analysis: Kruskal Wallis, Mann-Whitney for inter-group differences; ReliabilityCronbachs alpha; Correlation between simulators metrics and global score- Spearmans rho

FUTURE WORK:
Validate the video-endoscopic score in real procedures Use the video-endoscopic to show transfer of skills from VR to real procedures.
KW, p<0.001 KW, p<0.001

REFERENCES: Datta V, Mandalia M, Mackay S, Darzi A. The Pre-Op flexible sigmoidoscopy trainer. Validation and early evaluation of a virtual reality based system. Surg Endosc. 2002; 16: 1459-63 Ost D, DeRosiers A, Britt JE, Fein AM, Lesser ML, Mehta AC. Assessment of a bronchoscopy simulator. Am J Respir Crit Care Med. 2001; 164: 2248-55. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skills via an innovative bench station examination. Am J Surg. 1996; 172: 226-230. Neumann M, Firedl S, Meining A et al. A score card for Upper GI endoscopy: evaluation of Interobserver variability in examiners with various levels of experience. Z Gastroenterol. 2002; 40 (10): 857-62.