problem-based learning

The inuence of tutoring competencies on problems, group functioning and student achievement in problem-based learning
Henk J M Van Berkel & Diana H J M Dolmans

CONTEXT Prominent factors in problem-based learning (PBL) are the problems to be solved, tutorial group functioning and tutors competencies. These factors mutually affect one another and largely determine whether a powerful learning environment will be created. It is a tutors task to stimulate active, self-directed, contextual and collaborative learning and display interpersonal behaviour that is conducive to students learning. We investigated the effects of tutors competencies on students learning and on other variables, such as group functioning and student achievement. OBJECTIVES We investigated whether tutors who stimulate active, self-directed, contextual and collaborative learning make better use of problems and meaningful contexts in PBL and also enhance group functioning. We also investigated whether the quality of problems has a positive impact on group functioning and whether group functioning advanced student achievements. METHODS Questionnaires were used to collect data from students at the end of 11 modules in Years 1 and 2 of a PBL undergraduate medical curriculum. We used structural equation modelling to test the t of a theoretical model representing the factors of interest and their relationships. RESULTS Stimulation of active and constructive learning, self-directed learning and collaborative learning by tutors enhanced the quality of the problems and group functioning. The quality of the
Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands Correspondence: Henk J M Van Berkel, Department of Educational Development and Research, Maastricht University, PO Box 616, 6200 MD Maastricht, The Netherlands. Tel: 00 31 43 388 5750; Fax: 00 31 43 388 5779; E-mail: h.vanberkel@educ.unmaas.nl

problems promoted group functioning, which was found to have a positive effect on student achievement. CONCLUSIONS Tutors competencies had a positive effect on the learning of students. This suggests that it would be worthwhile including these competencies in staff development. KEYWORDS education, medical, undergraduate *standards; problem-based learning *standards; professional competence *standards; group processes; students, medical *psychology; motivation; questionnaires. Medical Education 2006; 40: 730736
doi:10.1111/j.1365-2929.2006.02530.x

INTRODUCTION
Problem-based learning (PBL) can be regarded as a constructivist approach to learning. Constructivism emphasises learners active engagement in their learning and thus in constructing cognitive networks. Self-directed learning is a key component of constructivism, as is learning in meaningful contexts. The latter is assumed to facilitate transfer of what has been learned. Finally, learning is expected to benet from collaboration among learners.1,2 These modern insights into learning constitute the foundations of PBL.3 In PBL, students learn by working to resolve problems which have been specially designed for this purpose. Problems can represent authentic patient cases, but they may also involve descriptions of phenomena that students are asked to explain. It is the function of problems in PBL to provide a

730

Blackwell Publishing Ltd 2006. MEDICAL EDUCATION 2006; 40: 730736

731

Overview
What is already known on this subject Prominent factors in problem-based learning are the problems to be solved, tutorial group functioning and tutors competencies. These factors mutually affect one another. What this study adds Tutors competencies had a positive effect on student learning. Tutors competencies also enhanced the quality of the problems and group functioning. The quality of the problems promoted group functioning. Suggestions for further research The model should be tested against other datasets. Studies that are aimed at obtaining further insights should be qualitative.

meaningful context for student learning. Students tackle the problems by discussing them in small groups. In this way they are actively involved in their own learning processes. This learning context, where different issues are raised by students within group discussions, is assumed to foster more indepth understanding of topics. A prerequisite for effective group functioning is collaboration among students. Tutors facilitate the group sessions. Students discuss the problem in hand and identify issues or topics that are essential to resolving the problem, but about which they nd their knowledge to be decient. These issues are described in terms of learning objectives to be pursued in selfdirected learning activities. During self-study, students consult sources that can clarify these issues. Subsequently, the outcomes of these efforts are discussed in the group. In summary, PBL is a constructivist approach to learning that stimulates active, self-directed, contextual and collaborative learning. The above description shows that PBL is a complex learning environment in which different variables interact. This paper investigates the effect of tutor competencies on PBL characteristics. Several process-oriented studies have examined the relation-

ships between the key aspects of the learning environment in PBL. Following the ideas of Carroll,4 Gijselaers and Schmidt developed a causal model comprising 3 input variables, 2 process variables and 2 output variables.5 The input variables were assumed to inuence the process variables, which, in turn, were expected to affect the output variables. What the model hypothesised was that prior knowledge, quality of problems and tutor performance (input) inuenced the amount of time spent on self-study activities and group functioning (process), which nally impacted on student achievement and interest in subject matter (output). When tested against empirical data, the model was found to demonstrate a good t, as demonstrated by the non-signicant chi-square statistic (0.70; d.f. 9). Van den Hurk et al. used a similar causal model in their study into the relationship between the quality of learning issues and depth and breadth of reporting and student achievement.6 They found a positive relationship between depth of reporting and achievement. Van Berkel and Schmidt tested a model in which they introduced a new process variable ) student commitment to studying ) which was operationalised as attendance at group sessions.7 They found this variable to be strongly related to student achievement.7 In a subsequent study, the same authors tested the effect of the quality of lectures as another new input variable.8 This variable appeared not to contribute to the t of the model, however. These process-oriented studies demonstrated that the outcomes of PBL are very much determined by the quality of PBL problems and by small-group functioning.5 This appears to support the constructivist underpinnings of PBL, which imply that student learning benets from problems that offer a meaningful context and from tutorial groups where active and collaborative learning is stimulated. Apart from the quality of problems and group functioning, the learning environment in PBL is also highly dependent on what tutors contribute to the learning process. The causal models we described reect how performance of the tutor role has a direct effect on group functioning. The role of tutor encompasses the following distinct practices:9,10 stimulating active learning by asking students to summarise in their own words what they have learned and by encouraging students to nd connections between the issues they have studied;

Blackwell Publishing Ltd 2006. MEDICAL EDUCATION 2006; 40: 730736

732

problem-based learning
stimulating self-directed learning by encouraging students to generate well dened learning objectives and to use a variety of resources in their search for information; facilitating contextual learning by encouraging students to apply their prior knowledge to the problem to be resolved; facilitating collaborative learning by encouraging students to regularly evaluate collaboration in the group and give constructive feedback about group work, and demonstrating interpersonal behaviour that reects a positive motivation to full the tutor role and awareness ones own strengths and weaknesses as a tutor.

METHODS
Subjects We conducted this study in the context of the PBL undergraduate medical curriculum at Maastricht Medical School, the Netherlands. In the academic year 200304 we collected data from questionnaires that are administered to all students as part of the regular curriculum evaluation process. We used data collected at the end of every module in Year 1 (6 modules) and Year 2 (5 modules). Modules are built around themes. Students meet for 2 weekly sessions in groups of 9)10 students, facilitated by a tutor. During these sessions the students discuss problems and generate learning objectives. These objectives are pursued in self-directed study activities and students report their ndings in the following group session. In each of the 352 groups at least 6 students completed the questionnaires. Instrument We evaluated tutor performance using a questionnaire that was validated in an earlier study and which has since been incorporated into the regular evaluation cycle at Maastricht Medical School.9,10 The questionnaire comprises 11 items, derived from contemporary constructivist approaches to learning and instruction, which form the basis of PBL. The items consist of statements and students are asked to indicate agreement on a 5-point scale (1 strongly disagree, 5 strongly agree). In previous research,9 a factor analytic study demonstrated that 5 factors underlie the 11 items of the questionnaire very well. Therefore, these 5 factors were assumed to represent the 11 items: active learning (items 1, 2 and 3), selfdirected learning (items 4 and 5), contextual learning (items 6 and 7), collaborative learning (items 8 and 9), and tutors interpersonal behaviour (items 10 and 11). A 5-point scale was used for the items in this questionnaire. Another item, with a 10-point scale, asked students about group functioning (Table 1). We used the results on the regular end-of-module tests as the measure of student achievements. The end-of-module tests are summative tests consisting of some 150 true false knowledge items relating to the topics covered in the module concerned. The percentage of correct answers is transformed to a 10point scale with 2 decimals, so that 64.3% correct answers yields a score of 6.43. Students have to pass the tests in order to obtain the required credits. The reliabilities of these tests (Cronbachs alpha) range

It is assumed that tutors in a PBL environment who have the competencies that enable them to demonstrate these behaviours will help their students to achieve better results. In order to investigate the effects of these tutor competencies, we tested the hypothesis that tutors ability to stimulate active, self-directed, contextual and collaborative learning has a positive effect on the use of problems and the meaningful context they provide and on small-group functioning. We hypothesised that the quality of PBL problems positively affects group functioning, which, in turn, has a positive impact on student achievement. In order to test these hypotheses, we incorporated them into a model and tested it against data collected within a PBL curriculum (Fig. 1).

Active/constructive learning Quality of PBL problems Self-directed learning

Collaborative learning

Group functioning Interpersonal behaviour

Contextual learning Achievement

Figure 1 Theoretical model.

Blackwell Publishing Ltd 2006. MEDICAL EDUCATION 2006; 40: 730736

733

Table 1 Variables and underlying items and rating scales included in the model Type of learning Active constructive learning 1 2 3 Self-directed learning 4 5 Contextual learning 6 7 Collaborative learning 8 9 Interpersonal behaviour 10 11 Quality of problem-based learning 12 13 Group functioning 14 Achievement 15 Rating scale

The tutor stimulated us to summarise in our own words what we had learned The tutor stimulated us to look for connections between issues discussed in the group The tutor stimulated us to try and understand the underlying mechanisms theories The tutor stimulated us to generate clearly dened learning objective by ourselves The tutor stimulated us to try and understand underlying mechanisms The tutor stimulated us to apply knowledge to the problem The tutor stimulated us to apply knowledge to other situations problems The tutor stimulated us to give constructive feedback about our group work The tutor stimulated us to evaluate group collaboration regularly The tutor was clearly motivated to full the tutor role The tutor had a clear picture about his strengths weaknesses as a tutor problems The problems sufciently stimulated group discussion The problems encouraged self-study Give a mark between 1 and 10 for tutorial group productivity Percentage of correct items on true-false items

15* 15 15 15 15 15 15 15 15 15 15 15 15 110 110

* Scale of 15, where 1 fully disagree, 5 fully agree. Scale of 110, where 6 sufcient, 10 excellent. A percentage of 64.3 is recalculated as 6.43 on a scale of 110.

from 0.61 to 0.87. All data were analysed at the group level. Statistical analysis Because we wanted to test the hypothesised model against empirical data, we used model-tting techniques derived from structural equating theories. The model-t procedure yields several statistics that allow assessment of the t of empirical data to a proposed theoretical model.11 Unfortunately, there is no single best statistic that gives decisive insight into the t of a model. In addition, there are no criteria to evaluate the statistic, such as a test of signicance,12 although some authors give rules of thumb. The best approach is to compute several statistics, which reect the t of the model. The coherence of the statistics is taken to be indicative of the t of the model. Browne and Mels recommend restricting the statistics to the following indices:13 CMIN (based on minimum value of the discrepancy), P (probability level), FMIN (also based on the minimum value of the discrepancy), FO (estimated population discrepancy), PCLOSE (probability of close t), and RMSEA (root mean square of approximation). Whilst following this approach,13 we added indices recommended by Marsh and Yeung:14 the TLI

(Tucker)Lewis index) and CFI (comparative t index).8 All these indices are discussed in length in van Berkel and Schmidt.8 This paper limits itself to nominating the rules of thumbs which indicate signicant results. As mentioned, these indices are applied to evaluate the t between the empirical data and the theoretical model. In addition, these indices can also be used as indicators to modify the model. For example, when it turns out that some variables do not add anything to the explanation of the model, we can omit these variables from the model because a model with less variables is preferable. The statistical package lations.12
AMOS

was used for the calcu-

RESULTS
Structural equation modelling starts by computing the intercorrelations of the variables in the study. These intercorrelations form the basis of the empirical model. The t of this model with the theoretical model is investigated. The strengths of the intercorrelations between the variables give a rst impression

Blackwell Publishing Ltd 2006. MEDICAL EDUCATION 2006; 40: 730736

734

problem-based learning
of the model t. For example, the correlations between the 5 dependent variables with the independent variable Group function are relatively high compared to independent variable Quality of PBL problems. Table 2 shows these intercorrelations, means and standard deviations of the variables. Correlations, based on averaged group scores, varied between 0.05 and 0.80. The values of the t indices resulting from testing the theoretical model are presented in Table 3. The beta weights and the squared multiple correlations of the model are displayed in Fig. 2. (Also shown are results from a simplied model to be described later.) A beta weight indicates a causal path between the 2 variables, whereas the squared multiple correlation is the percent variance explained in that variable. The beta weights are the standardised partial regression coefcients of each endogenous (dependent) variable on its priors, the independent variables. That is, the beta for any path (i.e. the path coefcient) is a partial weight controlling for other priors for the given dependent variables. Given the strengths of the beta weights and the values of the t indices, reported in Table 3 and Fig. 2, the model offers a reasonable description of the data, but not more than that. Although the indices in Table 3 are close to the values that indicate model t, most do not reach the limit. There is room for improvement. Some beta weights are relatively low. However, this raises the question of whether the model might not be simplied. A more parsimonious model with fewer variables, with an equally good t to the empirical data would be preferable.12 One way of simplifying a model is to examine the signicance of the beta weights. The beta weights of the variables Interpersonal behaviour and Contextual learning were 0.06

Table 3 Fit indices of the 2 models Value indicating model t Chi-square d.f. P CMIN DF RMSEA PCLOSE FMIN FO TLI CFI Theoretical model 38.371 10 0.000 3.937 0.091 0.011 0.112 0.084 0.944 0.980 Simplied model 12.840 6 0.046 2.140 0.057 0.339 0.037 0.019 0.976 0.990

0.050 3.000 0.050 0.050 0.050 0.050 0.950 0.950

(P 0.390) and 0.05 (P 0.302), respectively. As the signicance of these values indicated that these variables did not contribute substantially to the t of the model, we removed them from the model. Next, the t of the new simplied model was tested (Table 3, right column and Fig. 2, right side). Figure 2 shows the totals of R2 for the dependent variables (top right). The path coefcients (the digits connected with arrows) symbolise the effects of the independent variable on the dependent variable. For example, 47% of the variance of the variable Group functioning is explained by the 4 variables mentioned. In general, there were no, or only very small, differences between the coefcients derived with the 2 models. This implies that the removal of the 2 variables changed neither the squared multiple correlations nor the beta weights. Table 3 shows the comparison between the t indices of the 2 models. As can be deduced from the RMSEA and the corresponding P-value PCLOSE, both models yielded

Table 2 Intercorrelations, means and standard deviations of the variables (n 352) 1 1 Quality of PBL problems 2 Group functioning 3 Achievement 4 Active constructive learning 5 Self-directed learning 6 Contextual learning 7 Collaborative learning 8 Interpersonal behaviour Means Standard deviations Number of items Minimum maximum 2 3 4 5 6 7 8

0.57* 0.14* 0.40* 0.40* 0.25* 0.28* 0.29* 3.70 0.39 2 15

0.32* 0.51* 0.53* 0.37* 0.48* 0.46* 7.30 0.76 1 1 10

0.19* 0.24* 0.05 0.21* 0.09 6.89 0.45 150 0 10

0.63* 0.80* 0.54* 0.72* 3.73 0.49 3 15

0.51* 0.62* 0.67* 3.39 0.43 2 15

0.43* 0.66* 3.71 0.48 2 15

0.67* 3.39 0.59 2 15

3.88 0.52 2 15

* Signicant at the 0.01 level (2-tailed).

Blackwell Publishing Ltd 2006. MEDICAL EDUCATION 2006; 40: 730736

735

Active/ constructive learning Self-directed learning

0.25 0.25

0.20 Quality of PBL problems 0.41

Active/ constructive learning Self-directed learning

0.25 0.25

0.20 Quality of PBL problems 0.41

0.18 Collaborative learning 0.19 Interpersonal behaviour 0.06 0.05 Contextual learning Group functioning 0.32 0.10 Achievement 0.47

0.22 Collaborative learning 0.23 Group functioning 0.32 0.10 Achievement 0.48

Figure 2 Theoretical model and simplied theoretical model including beta weights and squared multiple correlations.

reasonably good ts of the models to the data. Based on the t indices, however, the simplied model appeared preferable.

DISCUSSION AND CONCLUSIONS

The aim of this study was to test a theoretical model that represents the assumed inuence of tutor competencies on several key variables in PBL. The model is based on a positive relationship between the extent to which tutors stimulate active, constructive, self-directed, contextual and collaborative learning and model positive interpersonal behaviour, and the quality of PBL problems, small-group functioning and student achievement. When we tested this model against empirical data, it appeared that the model could be simplied by deleting the input variables Interpersonal behaviour and Contextual learning. The results demonstrated a strong direct effect of the quality of PBL problems on group performance, which directly inuenced student achievement. This is in line with ndings from earlier studies into the complex interrelationships between different variables in PBL.5 What our study adds is that tutors efforts to stimulate active learning impact positively on the effectiveness of PBL problems as perceived by students. In other words, if tutors use techniques to stimulate active learning, for instance, by asking students to summarise in their own words what they have learned and encouraging them to nd links between the issues discussed in the group, students are stimulated to make optimal use of the meaningful context provided by the problems. Another nding

was that group functioning was positively affected by tutors stimulating collaborative learning through encouragement of regular evaluation by the students of the group process and through constructive feedback about group functioning. Finally, the ndings demonstrate that when tutors stimulate selfdirected learning, for instance, by encouraging students to generate well dened learning objectives, both the perceived effectiveness of PBL problems and small-group functioning appear to benet. These results can be interpreted as indications of ways in which different tutor competencies can further student learning in a PBL context. The results also demonstrate that, contrary to the initial hypothesis, tutors interpersonal behaviour and stimulation of contextual learning did not signicantly contribute to better group functioning. These variables were found to make no substantial contribution to the t of the theoretical model. A possible explanation for this nding is that, within the setting of our study, all tutors were highly motivated, as was reected in the high average ratings on this variable they received from students. In other words, most tutors scored highly on interpersonal behaviour. We have not been able to nd a satisfactory explanation for the nding that stimulation of contextual learning had no signicant effect. The ndings of this study suggest that it would be worthwhile for medical schools to give high priority to tutor training in the competencies that proved to stimulate student learning. Tutors are of crucial importance to the functioning of a group. This tutor behaviour can be instilled, by, for example, providing

Blackwell Publishing Ltd 2006. MEDICAL EDUCATION 2006; 40: 730736

736

problem-based learning
tutors with feedback after observation or by longitudinal training. In general, the success of PBL appears to depend strongly on tutors competencies, the quality of PBL problems and group functioning. The impact of the quality of problems suggests that a protable approach to staff development might be to offer courses on designing the type of problem that is likely to optimally promote active and self-directed learning. Such problems should be neither too tightly nor too loosely structured. Nor should they be too narrowly or too broadly dened. One of the limitations of this study is that parts of the hypothesised model were tested against only 1 dataset, although the ndings were found to be consistent with the results of studies by Gijselaers & Schmidt, Van Berkel & Schmidt and Van den Hurk et al.58 In further research the model should be tested against other datasets. In addition, there may be a possible bias in this study because we did not investigate the inuence of the (un)reliability of the variables on the outcomes. This effect can be measured by computing the disattenuated correlation coefcients. In addition, we used test scores as an outcome variable. However, these test scores were obtained from different tests. This does not guarantee that these scores are statistically equivalent. A second limitation is that, although the model does give an indication as to which variables inuence each other, it does not provide us with deeper insight into how these variables are actually interrelated. Studies aimed at obtaining further insights would have to be qualitative and should address the perceptions not only of students, but also of teachers.

REFERENCES
1 Ertmer PA, Newby TJ. Behaviourism, cognitivism, constructivism: comparing critical features from an instructional design perspective. Performance Improvement Q 1993;6:5072. Ertmer PA, Newby TJ. The expert learner: strategic, self-regulated and reflective. Instructional Sci 1996;24: 124. Dolmans DHJM, De Grave W, Wolfhagen HAP, Van der Vleuten CPM. Problem-based learning: future challenges for educational practice and research. Med Educ 2006;39:732741. Carroll JB. A model of school learning. Teachers College Record 1963;64:72333. Gijselaers WH, Schmidt HG. Development and evaluation of a causal model of PBL. In: Nooman ZM, Schmidt HG, Ezzat ES, eds. Innovation in Medical Education. An Evaluation of its Present Status. New York: Springer 1990;95113. Van den Hurk MM, Dolmans DHJM, Wolfhagen HAP, Van der Vleuten CPM. Testing a causal model for learning in a problem-based curriculum. Adv Health Sci Educ Theory Pract 2001;6:1419. Van Berkel HJM, Schmidt HG. Motivation to commit oneself as a determinant of achievement in problembased learning. Higher Educ 2000;40:23142. Van Berkel HJM, Schmidt HG. On the additional values of lectures in a problem-based curriculum. Educ Health 2005;18:4561. Dolmans DHJM, Wolfhagen HAP, Scherpbier AJJA, Van der Vleuten CPM. Development of an instrument to evaluate the effectiveness of teachers in guiding small groups. Higher Educ 2003;46:43146. Dolmans DHJM, Ginns P. A short questionnaire to evaluate the effectiveness of tutors in PBL: validity and reliability. Med Teacher 2005;27:5348. Bollen KA. Structural Equations with Latent Variables. New York: Wiley 1989;151176. Arbuckle JL, Wothke W. AMOS Users Guide, Version 4.0. Chicago: SmallWaters Corporation 1999;397. Browne MW, Mels G. RAMONA Users Guide. Columbus, Ohio: Ohio State University 1992;78. Marsh HW, Yeung AS. Longitudinal structural equation models of academic self-concept and achievement. Gender differences in the development of math and English constructs. Am Educ Res J 1998;35:70538.

4 5

10

11

Contributors: HJMVB was responsible for commenting on drafts of the manuscript, wrote the Methods and Results sections, analysed the data and edited the nal version of the paper. DHJD wrote the rst and nal drafts of the paper, excluding the Methods and Results sections. Acknowledgements: the authors thank Mascha Verheggen for collecting the data on achievement for this study, and Diana Riksen and Guus Smeets for setting up the initial datasets used in this study. Funding: this research was funded by Maastricht University. Conicts of interest: none. Ethical approval: the authors state that ethical approval was not required.

12 13 14

Received 19 May 2005; editorial comments to authors 24 August 2005, 25 October 2005; accepted for publication 30 January 2006

Blackwell Publishing Ltd 2006. MEDICAL EDUCATION 2006; 40: 730736