EQANIE-Conference 2011, Vienna

Learning Outcomes and Quality Management in Informatics Education

Distractor Quality Analyze In Multiple Choice Questions Based On

Information Retrieval Model
Artur Mkrtchyan
Information Security and Software Development Department,
State Engineering University of Armenia, Yerevan, Armenia

Overview Information Retrieval Model Match Criterion in Knowledge

In spite of the appealing characteristics of multiple-
choice questions (MCQ) - assessment of knowledge
using traditional MCQ is not objective and effective.
Multiple choice questions are frequently used but
there are no easy ways to investigate non-functioning
distractors in developed tests.
This presentation mentions advantages and
disadvantages of traditional multiple choice questions
Issues related to analysis and evaluation of non-
functional distractors can be solved by the theory of
artificial intelligence. However, this theory has
incredible complexity and enormous number of
objects to research.
The main reason for limiting the power of
computer based solution in the knowledge assessment
sphere is ambiguity (multi-variant) in the
Assessment Systems
Fig. 1 and 2 show the scheme of automatic search of
documentation, and automatic matching of functioning
distractors, respectively.
Consider Fig. 1. It shows the basic operations as a
result of which the subscriber receives the information
he/she needs.
User

and introduces a new way to investigate non- representation of test items (distractors). The reasons
functioning distractors in developed tests based on of ambiguity are synonyms, homonyms and Information Query

information retrieval model. polysemous vocabulary which are characteristics of

Search Query Image (SQI)
Unlike traditional statistical based analyzes above- any natural language.
mentioned approach allows test developers to analyze A similar problem exists in the development of Criterion for
Comparing SQI with DI
test quality before knowledge assessment process. information retrieval systems, which have to solve issuance of the
document:
Boolean function
essentially the same issues as in the case of
.
Document Image (DI)
Introduction investigating and analyzing distractors in test items. Issuing Document
However, information retrieval systems have a longer Document
Multiple-choice questions are now a strongly road of development compared with the e-learning
preferred testing instrument across many higher systems. So it makes sense to investigate systems of Fig. 1. Scheme of automated search of documents

education disciplines and business students in information retrieval in order to explore the possibility Test Administrator

particular. Ultimately, there is no one ideal assessment of reusing concepts of information retrieval model and
process and so too, MCQ testing has its advantages use it as a basis for analyzing test items’ distractors. Test Item Distractor

and disadvantages.
Search Query Image (SQI)
Document Image and Descriptor dictionaries Based On Distractor

Multiple Choice Question

In the descriptor based Information Retrieval Comparing SQI with DI Criterion for matching
A traditional multiple choice question (or MCQ) is Systems (IRS) there is a concept of Document Image distractor with document
content: Boolean function
one in which a student chooses one answer from a (DI) which is a list of keywords or descriptors, and Document Image (DI)
number of choices supplied. A multiple choice other information: numbers, names, product names, Issuing Boolean
question consists of and others, reflecting the basic semantic content of the Document
True or Some Score

• Stem - the text of the question

• Options - the choices provided after the stem
• Key - the correct answer in the list of options
• Distracters - the incorrect answers in the list of options.
Advantages of MCQ
• Compared with traditional assessment questions,
MCQs offer faster marking, less necessity for
cross-marking.
• MCQs are useful to educational institutions under
pressure to reduce unit costs.
• Multiple choice tests often require less time to
administer for a given amount of material than
would tests requiring written responses.
Disadvantages of MCQ
• There are no tools or approaches how to investigate
non-functioning distractors in developed tests
beside statistical analyzes.
• With MCQs there is a possibility of guessing the
correct answer.
• They assess only trivial recognition of facts, rather
than high-level thinking, such as exercising
judgment and synthesis
• They offer choices for answers, rather than ask the
candidate to construct the answer.
Test Quality
Distractors play a vital role for the process of
multiple-choice testing, in that good quality distractors
ensure that the outcome of the tests provides more
credible and objective picture of the knowledge of
the testees involved. On the other hand, poor
distractors would not contribute much to the
accuracy of the assessment because obvious or too
easy distractors will pose no challenge to the students
and as a result, we will not be able to distinguish high
performing learners from low performing.
document where the document refers to any unit of
information: article, a technical description of a
device, a monograph, patent, scientific report, etc.
The process of generating document image (DI)
from document is called Indexing. We used well
known algorithms and solutions for keyword
extractions such as KEA (Keyphrase extraction
algorithm), Maui (Multi-purpose automatic topic
indexing) and etc.
Keyword - a term which refers to the words of the document, many
of which reflect its core semantic content.
Descriptor - does the same role as the keyword. For all keywords
taken from the indexed text, with the help of Descriptor Dictionary
corresponding descriptor codes are included to DI.
With the help of descriptors all keywords of
document are assigned to an artificial semantic
uniqueness. Using Descriptor Dictionaries we are
reducing the size of Document Image.
Match Criterion in Information Retrieval
System
By analogy with the Document Image in this case we
speak about Search Query Image (SQI). Having
keywords and descriptors is not enough to perform
search. We need to specify matching criterion based
on which the computer decides to provide the
document as responding or not.
The natural way is to define Boolean model. In
construction of Boolean model, we introduce the
following interpretation of the logical variables of
Boolean functions. Let’s denote attributes/keywords
as A1, A2, A3, ...An
Boolean function, which will be called as a
search/retrieval function, assigning to the retrieval of
the document. We assume that the document responds
to the request if f (ÀÀ  À  Àn 
Fig. 2. Scheme of automated matching of distractors with documents
Scheme of automated matching of distractors with
documents presented in Fig. 2, is a complete analog of
the automated search of documents.
Finding non-functioning distractors in test items will
improve whole assessment process. In our case we will
find non-functioning distractors by finding functioning
distractors, all remaining distractors will be considered
as non-functioning.
Conclusion
As mentioned earlier, Information Retrieval Systems
have a longer road of development compared with
artificial intelligence and e-Learning systems.
Using an information retrieval model, where the basic
tenets of the theory apply to the IRS on the
construction of systems for automated assessment of
knowledge offers the possibility of simulation of
Boolean Functions of any criterion for evaluating the
distractors and does not require their semantic
analysis.
References
1. Isaacs, G. “About multiple choice questions in Multiple Choice Testing”; Green Guide, Vol
2.
1994.
Chan C. “Assessment: Multiple Choice Questions”; Assessment Resource Centre, 2008.
3. Schuwirth LWT, Vleuten CPM van der. “Different written assessment methods: what can
be said about their strengths and weaknesses?”; Med Educ 2004, 38(9):974-979.
4. Farley JK. “The multiple-choice test: writing the questions”; Nurse Educ 1989, 14(6):10-12.
Guha, R. V., McCool, R., Miller, E. “Semantic search. In Proc. of the 12th Intl”; Budapest,
5.
Hungary, (2003) 700-709.
6. Cizek GJ, O'Day DM “Further investigations of nonfunctioning options in multiple-choice
test items”; Educ Psychol Meas 1994, 54(4):861-872.
7. Osterlind SJ. “Constructing test items: Multiple-choice, constructed- response, performance,
and other formats”; 2nd edition. Boston: Kluwer Academic Publishers; 1998.
8. Medelyan, O., Witten I. H. “Thesaurus Based Automatic Keyphrase Indexing”; In Proc. of
the Joint Conference on Digital Libraries 2006, Chapel Hill, NC, USA, pp. 296-297.
9. R. Wilkinson and P. Hingston. “Using the Cosine Measure in a Neural Network for
document retrieval” ACM SIGIR Conference on Research and Development in Information
Retrieval, p. 202-210, Chicago, IL USA, October 1991.
10. Järvelin, K., Kekäläinen, J., and Niemi, T. “ExpansionTool: Concept-based query
expansion and construction” Information Retrieval 4, Issue 3-4 (2001) 231-255.

Phone: +37477572708, Email: mkrtchyan.artur@gmail.com