Dr.

Plaban Kumar Bhowmick

Assistant Professor
Centre for Educational Technology
About E61002
 Course Description
 Use of basic NLP tools and techniques to address
challenges in text-based eLearning systems or to
develop text-based e-learning applications
 Course Objectives
 To identify challenges in text mediated e-learning
systems
 To identify NLP techniques to deal with the
challenges
 To apply NLP tools to address the challenges
 End semester exam
 40 marks
 Mid semester exam
 30 marks
 TA evaluation
 Project work
▪ 20 marks (mid term and end term evaluation)
 Assignments
▪ 10 marks
 Books
 Speech and Natural Language Processing,
Jurafsky and Martin
 Handbook of Automated Essay Evaluation,
Shermis and Burstein
 Automated grammatical error detection for
language learners, Leacock et al.
 Research articles
 ACL, COLING, Computational Linguistics and
others
 Educational Natural Language Processing,
 Tutorial in COLING, 2008.
 Natural Language Processing and
eLearning
 Prof. Dr. Iryna Gurevych, Dr. Delphine Bernhard
 Ubiquitous Knowledge Processing, Technische
Universitat Darmstad
 Natural Language Processing for
Educational Applications
 Rada Mihalcea, University of North Texas
http://10.35.32.20/moodle/
 How different computational techniques and
systems can automate different processes in
teaching and learning?
 E-learning (or eLearning)
 the use of electronic media and information and
communication technologies (ICT) in education
 Synonymous with
 multimedia learning,
 technology-enhanced learning (TEL),
 computer-based instruction(CBI),
 computer-based training (CBT),
 computer-assisted instruction or computer-aided
instruction (CAI),
 And many others
 Pre-packaged multi-media course materials
with restricted type and set of assessment
items.
 Not learner centric
 Mostly focus towards cognitive aspects of
learning ignoring social and affective
dimensions altogether.
 Web 2.0 is not a new web
 Web 2.0: a new way of designing
participation, hosting services, and web-
based communities, promoting creativity and
information sharing.
 specific technologies like wikis and blogs, a
new way of creating web pages like mash-
ups, and a massive use of descriptors or tags
in what has been defined as a folksonomies.
 Individual production and User Generated
Content
 Harness the power of the crowd
 Data on an epic scale
 Architecture of Participation
 Openness
 A wiki is essentially a website constructed in such a
way as to allow users to change content on the site.
 Key features:
▪ Hypertextual structure
▪ Social authoring - collaborative production
▪ Dynamic document - always under construction
 Educational use:
▪ To support collaborative work, substituting old .doc or .pdf
documents.
▪ To produce a course or study corpus in cooperation with all
academic stakeholders: lecturers, students, …
▪ To distribute information to students, in order to facilitate the
updating of materials by the professor.
 A blog is a way of distributing news
 Key features:
▪ There are one or several authors that produce entries
▪ Visitors can add comments
▪ New entries and comments do not substitute older ones
▪ It is possible to subscribe in order to receive news via email or
through RSS readers.
 Educational use:
▪ Teachers have used blogs as an easy way to produce dynamic
learning environments without previous knowledge of html.
▪ Students have used blogs as an alternative digital portfolio or as a
learning log.
▪ Ultimately, blogs have been used as support for collaborative work.
 Online office: Google drive, MS Sharepoint
etc
 Social bookmarking
 Video repositories and online videos
 Social networks
 Study at any place, any time
 Several devices may be used for learning: computer,
iPod, PDA, etc.

 Authority in educational systems is distributed:

collective intelligence and wisdom of the
crowds
 Learn not only from teachers and instructors, but also
from peers

 New forms of knowledge organization: tags and

folksonomies
 Rise of the MOOCS

Massive - enrolment numbers

Open - no mandatory qualifications some learners
are not students
Online - fully of universities

Course - structured, temporal

 Course designed in short (~10min) modules

 Low study hours per week - modules not degree programmes
 Certificates of completion rather than credit…
The MOOC Platform

Content Assessment Communication

Video lecture Multiple Choice Quiz (MCQ) Threaded discussion forum

Video group discussion Peer Assessment

‘Robot’ grading

• Live webcasts or Hangouts

• Twitter
 We know it’s important even in this digital
age.
 Textual discourses in education
 Learning materials in web
 Wiki, blog
 Assessment items
 Formative feedback
 Use of Natural Language Processing (NLP)
techniques to assist current e-learning and
MOOC-based platforms through automated
 text analysis
 assessment item generation
 automated grading and feedback generation
 text adaptation
 tutoring dialog processing
 content metadata extraction
 etc.
 Educational Natural Language Processing

eLearning NLP

Computer aided Computational analysis

learning/Instruction of language
 Definition of CALL from Levy 1997:
 “Computer-Assisted Language Learning (CALL) may
be defined as the search for and the study of
applications of the computer in language teaching
and learning”

 Intelligent CALL (ICALL), or more appropriately

parser-based CALL from Holland et al 1993:
 ICALL relies on parsing, “a technique that enables the
computer to encode complex grammatical knowledge
such as humans use to assemble sentences, recognize
errors and make corrections”
From Markus Dickinson
slides
Research since the 90s has shown that awareness
of language forms and rules is important for an
adult learner to successfully acquire a foreign
language.
 The time a student can spend with an
instructor/tutor typically is very limited.
 work on form and grammar is often
deemphasized and confined to homework
 The downside
 learner has relatively few opportunities to gain
awareness of forms and rules and receive
individual feedback on errors.
 Excellent opportunity for developing Computer-
Aided Language Learning (CALL) tools to
 provide individual feedback on learner errors and
 foster learner awareness of relevant language forms
and categories.
 But CALL systems which offer exercises
 typically are limited to multiple choice, or simple form
filling
 feedback usually is limited to yes/no or letter-by-letter
matching of the string with a pre-stored answer.
 Linguistic modeling is needed to improve on this
situation:
 tokenization: identify words
 morphological analysis: identify/interpret morphemes
 syntactic analysis: identify selection, government and
agreement relations and word order requirements
 formal pragmatic analysis: identify coreference
relations, information structure partitioning, . . .
 Computational tools identifying such linguistic
properties need to be integrated into CALL
systems to obtain language-aware “Intelligent”
CALL (ICALL).
 Intelligent CALL (ICALL) focuses on using
linguistics and natural language processing to
make CALL better.
 When building a full-blown ICALL system,
several issues arise:
 Diagnosing and accounting for user errors
Parser-based CALL
 Modeling the system on particular (kinds of) users
 Presenting useful feedback to the user
 We want to use parsers to:
 Assign an analysis to a learner’s input
 Identify erroneous spots in the input
 Provide an analysis of what the error signifies

 But parsers are designed for well-formed

constructs
 When learners type in incorrect sentences, parsers
have to be modified to correctly analyze the
incorrectness.
 Use mal-rules = rules which are added to the
grammar to handle error cases.
 e.g., A singular noun and a plural verb are allowed to
combine, but it is marked as an error.
 Constraint relaxation: a parser can be reworked
to handle ill-formed input.
 Parsers normally just “die” when handling bad input.
 So, we allow some constraints (e.g., that a subject and
verb must match in number) to be relaxed while
parsing
 Given multiple possible analyses for a sentence,
which one is most likely, based on:
 the stage of acquisition of the learner
 the first language of the learner
 the focus of the exercise
 What kinds of errors are we interested in and do
we expect?
 We would like an error typology = a classification of
errors into different groups.
 People are starting to look at error-tagged corpora to
find the most common errors
 Feedback = response to the learner based on
their input. Purpose of feedback:
 Reinforcement: feedback can act as a reinforcer to
learn a particular concept (behaviorism)
 Learning processes need feedback to know right from
wrong (cognitivism)
 Things to keep in mind when designing a
system:
 Feedback needs to be accurate.
 Displaying more than one error message at a time is
not helpful (Heift 2001).
 Explanations should be short.
 Learner Error Corpus
 Error taxonomy
 L1 specific corpora collection and annotation
 Automatic detection of grammatical error
 Parser-based methods
 Statistical methods
 Grammatical error correction
 Correcting individual errors
 Correcting whole sentence
 Evaluation of error detection and correction
 Readability
 The feature of language that makes it easy to read
is called readability.
 Greater readability increases
 Comprehension (Understanding)
 Retention (Memory)
 Reading speed
 Persistence (reading more of the text)
 Readers feel frustrated.
 Most often, they stop reading without
even thinking about it.
 They may seek help or call support.
 They go to some other task.
The reader The text
 Prior knowledge  Content
 Reading skill  Style
 Interest  Design
 Motivation  Organization
 Look at
 Words and their formations
 Average sentence length
 Is the text coherent?
 Readability formula
 They make some hard assumptions
 Do not consider semantic features
 Reading Ease = 65.9 Reading Ease = 65.4

Word’s readability scores for these two texts are

almost identical. Yet they are clearly different in
nature and effect on the reader.
 What is constructed when we comprehend a
sentence?
 Propositional representation
 Do words have role in comprehension?
 Mental lexicon
 How do we resolve different ambiguities?
 A thief shot a cop in the park
 How does comprehension relate to reading
time?
 Reading time experiments
 Eye tracking experiments
 What if a particular text segment is not
readable?
 Sources of difficulty
▪ Lexical: Meaning of technical terms, entities
▪ Syntactic difficulty
▪ Discourse level difficulty
 Entity linking in text
 linking name mentions in text with their
referent entities in a knowledge base

 Wikify
 “I notice that you use plain, simple
language, short words, and brief sentences.
That is the way to write English—it is the
modern way and the best way. Stick to it;
and don’t let the fluff and flowers and
verbosity creep in.

“When you catch an adjective, kill it. No, I

don’t mean utterly, but kill most of them—
then the rest will be valuable. They weaken
when close together. They give strength
when they are wide apart.”

—Mark Twain, in a letter to a 12-year-old

boy.
 A subtask of text simplification
 Replacing words or short phrases by simpler
variants in a context aware fashion
 Motivation
 To reach out to wider range of readers having
limited vocabulary
▪ Children
▪ People with low literacy level or cognitive disability
▪ Second language learners
 Technical Medical Language
 Hypertension risk factors include obesity,...
 High blood pressure risk factors include excessive
weight,...
 Legal Language
 The Products transacted through the Service are...
 The Products managed through the Service are...
 Low Literacy Readers
 Hitler committed terrible atrocities during the second
World
 Hitler committed terrible cruelties during the second
World War
 The readability of a text can be improved by
transforming it into a simpler text
 Characteristics of manually simplified texts
 shorter sentences
 fewer and shorter phrases
 fewer adjectives, adverbs and coordinating
conjunctions
 nouns are less often replaced with pronouns
Original text: Congress gave Yosemite the money to repair damage from the 1997
flood.
Abridged text: Congress gave the money after the 1997 flood
 Writing test questions is an extremely
difficult and time consuming task for
teachers
 Use of NLP to automatically generate test
items for
 language learning
 comprehension test
 Multiple-Choice-Questions
stem

Who was voted the best international footballer for 2004?

(a) Henry (b) Beckham (c) Ronaldinho (d) Ronaldo

distractors key

 Fill-in-the-blanks cloze questions

 Challenges
 Identifying candidate sentence
 Identifying key and stem
 Identifying distractors
 Evaluation of item generation systems
 Why question generation?
 Idealistic vision: learners ask questions to deal
with knowledge deficit
 Reality:
▪ Trouble in identifying knowledge gap
▪ Shyness
▪ typical student asks less than 0.2 questions per hour in a
classroom (Graesser and Person’s, 1994)
▪ Train the learner’s to ask deep questions (such as why,
why not, how, what-if, what-if-not)
 Text-to-Question Generation
Chinese words are made up of many little marks instead of letters. These words are
called characters. There is no alphabet, so children do not need to learn to spell.
However, they have to learn at least 3,500 different characters before they can read a
simple book. Hundred of years ago, when the Chinese began to write, each word was
a picture. The word for umbrella still looks like an open umbrella.

1. What are the Chinese words are made up of?

2. Are the Chinese words made up of letters?
3. What are Chinese words called?
4. Why do the Chinese children not need to learn to spell?
5. Do Chinese children need to learn to spell?
6. How does the word for umbrella look like in Chinese?
 Lexical
 Question depends on the semantics of the answer
▪ Herman Melville wrote Moby Dick  Who wrote Moby Dick? What
wrote Moby Dick?
 Non-compositionality of words or phrases
▪ Multiword expressions, phrasal verbs
▪ When Russia invaded Finland in 1808, Helsinki was again burned to
the ground  What was Helsinki again burned to?
 Syntactic
 Imprecision of syntactic parsers
 Constraints on WH-movements
▪ John thought Mary said that James wanted Susan to like Peter
▪ Who thought Mary said that James wanted Susan to like Peter? (answer: John)
▪ Who did John think Mary said that wanted Susan to like Peter? (answer: James)
 Discourse Challenges
 Pronoun anaphora
▪ Abraham Lincoln was the 16th president. He was assassinated
by John Wilkes Booth.  Who was he assassinated by?
 Vague noun phrases
▪ . . . The show boosted the studio to the top of the TV cartoon
field . . . .  What boosted the studio to the top of the TV
cartoon field?
 Implicit discourse relations and world knowledge
▪ Booth jumped into the box and aimed a single-shot, round-slug
.44 caliber Henry Deringer at his head, firing at point-blank
range  Who killed Abraham Lincoln?
 Automated essay evaluation
 “The process of evaluating and scoring written prose
via computer programs” (Shermis and Burstein)
 Use NLP technique to provide
▪ Holistic scoring
▪ Formative feedback
 Why automatic essay scoring?
 to reduce laborious human effort
▪ Software systems do the task fully automatically
▪ Computer generated scores match human accuracy
 Ellis Page, 1966
 Grading burden, a significant impediment to the
improvement of overall writing capacity
 Project Essay Grade (PEG), 1973
 Evolution of commercial AEE
 Intelligent Essay Assessor (Pearson Education)
▪ Latent Semantic Analysis
 E-rater® (ETS)
▪ NLP to extract linguistic information and text
characteristics (grammatical errors, discourse analysis)
 C-rater (ETS)
▪ NLP to extract content information for short-answer
scoring
 Grammatical errors
 Subject-verb agreement
 Verb-form
 Pronoun errors
 Discourse structure
 Development of thesis statements, main points,
support, conclusion
 Topic relevant word usage
 Sophistication
 Idioms, metaphor, style
 Grammaticality
 Syntactic processing
 Word usage
 Collocation
▪ Powerful tea vs strong computer
 Content coverage
 Computational text similarity
 Text coherence
 Discourse processing
 Intelligent Tutoring System
 present challenging problems and questions
to the learner
 the learner types in or utters answers
 there is a lengthy multiturn dialogue as
complete solutions or answers evolve

Automated Dialog
Processing
 AutoTutor
 Tutoring Research Group (TRG) at the University
of Memphis
 ATLAS/ANDES
 University of Pittsburgh
 BEETLE
 University of Edinburgh
 Dialog and conversational agent
 Architecture of dialog-based ITSs