Experience on Outcome-Based Teaching and Learning

Oliver Au1 and Reggie Kwan2

1
Department of Computer Science, Loughborough University, UK
o.t.s.au@www.lboro.ac.uk
2
Caritas Francis Hsu College, Hong Kong, China
rkwan@cfhc.caritas.edu.hk

Abstract. Outcome-based teaching and learning emphasizes the explicit

declaration of learning outcomes which identify the tasks students are expected
to be able to perform after completing the course, and to what standard. OBTL
also requires the teaching, learning and assessment activities to align with the
stated learning outcomes. We interviewed fifteen university instructors about
their experience of teaching outcome-based computer science courses and how
OBTL has affected students and themselves. Academic departments had not
adjusted their teaching load to account for the added effort required for the
migration to OBTL. Most instructors continue to use time-saving norm-
referenced assessment. Students did not seem to perform better under OBTL.
With the way OBTL is being implemented, it is doubtful that the full benefits of
OBTL will ever be realized.

Keywords: Outcome-based teaching and learning (OBTL), outcome-based

education (OBE), frequent summative assessment.

1 Introduction
OBTL has been a popular acronym in the higher education in Hong Kong over the
past few years because of its endorsement by the funding agency University Grants
Committee (UGC). In OBTL, learning outcomes are stated explicitly and must be
supported by suitable teaching, learning and assessment activities. Not entirely new,
OBTL corresponds closely to outcome-based education (OBE) that has been around
from the early 80’s [1]. OBE in turn has its roots in the competency-based movement
introduced in the late 60’s [2]. Advocates claimed that OBTL benefits student
learning [3]. This paper examines this claim based on the experience of fifteen
academic staffs who teach outcome-based computing courses at the university level in
Hong Kong.
Successful experience of using OBTL has been reported but it was largely
perceptual, anecdotal, and small scale [4] [5] [6]. There were also reports of withdrawal
due to unsuccessful implementations [7]. A notable and recent failure took place in
Western Australia where the government abandoned most of its OBE system for upper
school (year 11 and 12) in response to massive objection from teachers and parents [8].
Literature that demonstrates the effects of OBTL at the university level is especially
lacking [9]. There is a need for more empirical study of how OBTL affects learning at
the university level.

F.L. Wang et al. (Eds.): ICHL 2009, LNCS 5685, pp. 133–139, 2009.
© Springer-Verlag Berlin Heidelberg 2009
134 O. Au and R. Kwan

Some readers may prefer to see large scale quantitative research where objective
and precise relationships between variables are tested. Our chosen research approach
however is qualitative for the following reason. University instructors do not like to
be monitored and they have a fair amount of freedom in teaching and assessment. It is
beyond our power to enforce learning and assessment activities be done in specific
ways needed for a quantitative research. Instead we interviewed the academic staffs to
find out the actual effects of their OBTL implementations at the course level. This
qualitative research could pave the way for a quantitative research in the future.
In Section 2, we describe the interviews with the instructors. In Section 3, we
describe their OBTL implementations. Sections 4 and 5 respectively report the effects
of OBTL on students and instructors from the instructors’ perspective. We summarize
and reflect on our findings in Section 6.

2 The Interviews
We shall refer to academic staffs as instructors regardless of their title being instructor,
lecturer, assistant professor, associate professor or professor. We interviewed fifteen
instructors about their use of OBTL in eighteen computing courses. Among these
courses, thirteen were introductory courses open to students from any disciplines. The
other five courses were advanced courses taken mostly by computer science majors.
The instructors were interviewed individually for about 30 minutes on basically three
questions:
1. How did they implement OBTL in their courses?
2. How has OBTL affected their students?
3. How has OBTL affected them?

The interviews were not taped in order to encourage instructors to share freely.
They were assured that their experience would only be used collectively without the
risk of their identities being unveiled. We scripted each interview and forwarded it to
the corresponding instructor for possible corrections. The instructors were pleased
with the accuracy and completeness of the scripts. Thirteen of the fifteen scripts were
accepted by the interviewed instructors without modifications. One instructor asked a
statement be removed from the script because more observations were necessary
before a generalized statement of the observed student performance could be made.
Another instructor asked the record of a negative OBTL experience be removed. It is
uncertain that the correction request was due to an error in the original script or the
instructor not wanting to be associated with a true but negative experience. From what
we can tell, the interviewed instructors, by and large, were candid to discuss their
experience openly.

3 OBTL Implementations
Arbitrarily bad implementations leading to failures gave no proof that OBTL could
not be successfully implemented. Hence we asked the instructors how they
implemented OBTL in their courses.
 Experience on Outcome-Based Teaching and Learning 135

3.1 Course Learning Outcomes

First, a clarification of terminology is in order. In the U.K., undergraduate students

enrol in a number of modules in an academic term to complete a course in three years.
In the U.S. and Canada, undergraduates enrol in a number of courses in a term to
complete a program in four years. A course in North American is a module in the
U.K. We use the American terminology in this paper. Most of our courses have five
to six course learning outcomes. Two instructors painstakingly broke down the main
outcomes into subtasks or multiple levels of performance standard.

3.2 In the Classroom

All instructors raised students’ attention to the course learning outcomes in the first
class. Some instructors even reminded students of the relevant learning outcomes
every class. About half of the instructors reported that OBTL had no other impacts to
their lectures.
Some instructors posted questions relating to the learning outcomes at the
beginning of lectures to guide students’ attention. The questions were found effective
in motivating the students and keeping them focused throughout the class.

3.3 Assessment Approach

In its purest form, OBTL assessment is criterion-based. Students are given multiple
opportunities to demonstrate their abilities to perform the tasks specified in the
learning outcomes. If a student failed to perform the task in the first two attempts but
succeeded in the third attempt, he or she still has achieved the learning outcome. In
the spirit of OBTL, the final grade should just be the same whether the student only
succeeded in the last attempt or in all three attempts.
A different assessment approach is norm-referenced. Each question in a quiz,
assignment or examination is assigned a weight. Student performance is reflected by a
final score, usually out of a hundred, accumulated from all the assessment activities.
Instructors are free to decide whether a grade C requires a final score of 55 or 60
based on the performance of the whole class. OBTL advocates consider the use of
norm-referenced assessment harmful.
Thirteen of our instructors used norm-referenced assessment in their courses that
OBTL advocates advise against. Two remaining instructors used criterion-based
assessment. However they had reverted back to norm-referenced assessment because
criterion-based assessment took up too much of their time without significant
benefits. One of them said, “Criterion-based assessment is problematic. First, judging
the level of student performance on an outcome is subjective. Second, a student might
do better in the mid-term than in the exam. Was it because the student was too
nervous in the exam? Or had the student lost the ability over the few weeks between
mid-term and exam? I really don’t know how to interpret this.”

3.4 Assignments

At the introductory level, computing courses lend themselves well to the use of
frequent assignments. Since most courses in our study were introductory, students had
136 O. Au and R. Kwan

always been given numerous assignments. Under OBTL, our instructors had further
increased the number of assignments, for example from biweekly to weekly. Of
course, a weekly assignment will be shorter than a bi-weekly assignment. However
students still saw a net increase in the total amount of time spent on the more frequent
assignments under OBTL.

4 OBTL Influence on Students

4.1 Motivation

Thanks to the increased frequency, the smaller assignments took less time for
instructors to grade and give feedback. The students received feedback of their work
sooner and typically within a week. They were happy to know their current progress.
For the few courses that used criterion-based assessments, students were also happy to
have multiple opportunities to demonstrate their abilities on the learning outcomes. If
a student had already achieved a learning outcome in a quiz, he or she had one less
item to worry about in the final examination.
Some instructors reported an increase in class participation. We can think of two
possible explanations, either the students were better motivated or they became more
capable. The net increase in assignment workload did not cause students to complain
except in one course where insufficient marks were allocated to the assignments.
Three instructors reported that their students were happier in outcome-based courses.
But one instructor put it bluntly, “Students do not seem to care if a course is OBTL or
not. They only care about how OBTL will affect their grades.”

4.2 Performance

Only one instructor reported a noticeable improvement in students’ performance after

switching to OBTL. But it is worth noting that this instructor received a very poor
evaluation from students in the previous delivery of the same course in non-OBTL
fashion. A swing from an extreme back to the norm may be natural and unrelated to
the approach used.
One other instructor kept track of the rate of learning outcomes achieved in
quizzes, mid-term and final examinations. A trend of progression was observed. In
other words, more students could achieve a learning outcome in subsequent tests. But
this is an improvement in performance from one assessment to the next. It did not
indicate overall improvements due to the use of OBTL.
The other thirteen instructors reported no observed improvement in the overall
student performance. This stagnant student performance in the final examination is
especially disturbing amid the moderate increase in time spent on assignments and a
likely improvement in motivation. The lack of observed improvement may not be a
true picture. Papers and pencils used in a conventional setting of examinations could
be unsuitable in the assessment of some learning outcomes. Even with known
limitations in in-class quizzes and examinations, we cannot rely heavily on take-home
assessment due to the concern of possible plagiarism. The unrealistically good
performance on the assignments suggests that students are receiving too much help
from fellow students.
 Experience on Outcome-Based Teaching and Learning 137

Higher frequency of assignments is usually found to improve student performance

[10]. The increased frequency in our sample has not found to improve student
performance. It may suggest that the increase from bi-weekly to weekly is
insignificant to university students.

5 OBTL Influence on Instructors

5.1 Assessment Design
The creation of practical artifacts is fundamental in computing. Students are almost
always required to create something useful, for example working programs, diagrams,
calculations, proofs and test data. It is rare in computing courses that students do not
need to produce concrete artifacts. With or without OBTL, computing students are
always asked to produce some artifacts. Under OBTL, learning outcomes are
essentially the abilities to produce these artifacts. Therefore we were not surprised
when the majority of our instructors did not find learning outcomes useful in their
question preparation. However learning outcomes may help assessment design in
other disciplines.

5.2 Instructor Workload

Instructors had taken the time to learn OBTL from seminars, workshops, printed and
online materials. They spent a significant amount of time to prepare questions in the
weekly assignments, grading them and giving comments. They could manage the
additional effort in teaching and learning preparation.
The biggest problem for them is criterion-based assessment. A simple spreadsheet
can represent a student with a row and a learning outcome with a column. But a
learning outcome can be tested multiple times: in a quiz, a mid-term examination and
again in a final examination. The spreadsheet has just gotten a little more
complicated. In fact, a question can test multiple outcomes and vice versa an outcome
can be demonstrated in multiple questions. Finally, the student’s achievements in
various outcomes need to be combined into a final grade. Our instructors found the
effort excessive without proportionate benefits in student learning. They concluded
that, given their current teaching load, criterion-based assessment endorsed by OBTL
advocates is infeasible for courses with over fifty students. The two instructors, who
had faithfully attempted the criterion-based assessment, decided to revert back to
norm-referenced assessment.
The academic department tried to alleviate the problem of added instructor
workload by approving instructors’ requests to hire student helpers to mark the
assignments. But this solution has its problems. One instructor reported the difficulty
in hiring qualified helpers. Another instructor had to restrain the assignment questions
so that they were easier to mark by the helpers.

6 Summary
Universities in Hong Kong have invested in OBTL: hiring external consultants,
sending instructors to workshops, and creating printed and online OBTL resources.
OBTL advocates recognize the extra time required to go OBTL [3]. However our
138 O. Au and R. Kwan

interviewed instructors had to teach the same number of courses under OBTL. With
the typically large university classes in Hong Kong, instructors have taken the
shortcut of norm-referenced assessment instead of the criterion-based assessment
endorsed by OBTL advocates [11]. Fourteen of the fifteen instructors reported that
student performance is unaffected by OBTL. We have made the assumption here that
the instructors know how to accurately assess the students’ abilities. Increasing
assignment frequency from biweekly to weekly has not helped student performance
either. Our findings are biased towards introductory courses and may not apply to
non-computing courses.
The current grading system used in the universities, that represent a student’s
performance by a percentage or a letter grade, does not make OBTL any easier.
Combining a student’s abilities in various outcomes into a single final grade is tedious
and somewhat arbitrary [12]. An overhaul of this grading system may be in order
before criterion-based assessment can be practically applied.
A key reason for the current push towards OBTL in Hong Kong is accountability.
However no mechanisms are in place to check that the teaching, learning and
assessment activities are indeed constructively aligned with the learning outcomes.
Instructors are our main reference of student performance. When instructor
performance is affixed to student performance, instructors would be tempted to paint
a rosy picture that is not necessary accurate and complete. The use of OBTL in this
context gives a false sense of accountability.
To prevent OBTL from being reduced to a paper exercise, instructors’ teaching
load must be lowered to account for the additional effort in assessment. The effects of
OBTL are not fully understood. Researchers should continue to look for empirical
evidence. Missing in our paper is the perspective from students. Hopefully in the
future, we will be better at tweaking OBTL for various disciplines and course levels.
For your food for thought, we shall conclude the paper with the following quotation
from an instructor.

My courses teach computer programming. I do not see the benefits of applying

OBTL. I have always been a good teacher. I earned top evaluation from
students. Even the weak students can learn in my courses. I received requests
from around the world to use my online teaching materials. I don’t want to
change the non-OBTL approach that I have been using successfully for years.

References
1. Spady, W.: Outcome-Based Instructional Management. The Australian Journal of
Education 26(2), 123–143 (1982)
2. Malan, S.P.T.: The ‘New Paradigm’ of Outcomes-Based Education in Perspective. The
Journal of Family Ecology and Consumer Sciences 28 (2000)
3. Biggs, J., Tang, C.: Outcome-Based Teaching and Learning – What is it, Why is it, How
do we make it work? – a workshop document used in the City University of Hong Kong,
Hong Kong Baptist University and University of Hong Kong (2006)
4. Evan, K.M., King, J.A.: The Outcomes of Outcome-Based Education: Research and
Implications. In: Annual Meeting of the American Education Research Association, San
Francisco, California (1992)
 Experience on Outcome-Based Teaching and Learning 139

5. Glatthorn, A.A.: Outcome Based Education: Reform and the Curriculum Process. Journal
of Curriculum and Supervision 8(4), 354–363 (1993)
6. Evan, K.M., King, J.A.: Research on OBE: What We Know and Don’t Know. Educational
Leadership 51(6) (1994)
7. Manno, B.V.: Outcome-Based Education: Has It Become More Affliction Than Cure?
Center of the American Experiment (August 1994)
8. Australian Associated Press. WA dumps Outcomes Based Education (OBE), December 12
(2007)
9. Berlach, R.G.: Outcome-Based Education and the Death of Knowledge. In: Conference of
the Australian Association for Research in Education, Melbourne (2004)
10. Wallace, A.: Promoting Frequent Assessment to Improve Student Learning. In: The SoTL
Commons – A Conference for the Scholarship of Teaching and Learning. Georgia
Southern University, Statesboro, Georgia, March 11-13 (2009)
11. Biggs, J.: Teaching for Quality Learning at University, 3rd edn. Open University Press
(2007)
12. Au, O.T.S.: A Tabular Approach to Outcome-Based Course Planning. In: 1st International
Conference on Hybrid Learning – ICHL Selected Papers, Hong Kong, August 13-15
(2008)