NATURAL LANGUAGE PROCESSING

RESEARCH SURVEY
Sajin SS
2016/07/17
S7, CSE, CET

What is Natural Language Processing

Natural Language Processing (NLP) refers to AI method of

communicating with an intelligent systems using a natural
language.
# Computers communicating with humans in our own
language is a scientific dream!
# Where does NLP stand?

Where does NLP stand

Components of NLP

The 2 components of NLP are :

# Natural Language Understanding (NLU)

Mapping the given input in natural language into useful

representations.
Analyzing different aspects of the language.

# Natural Language Generation (NLG)

involves text planning, sentence planning and text

realization

NLU is harder than NLG !

Issues in NLU

# Humans inputs often not well-formed (ungrammatical,

typos)
# Each language is different: grammar, vocabularies, etc
# SMS/social media talk?

# Technically speaking Lexical, Syntactical and Referential

Ambiguities
Difficult to deal with all these concerns at the same time!

Steps In NLP

THE NLP RESEARCH - Introduction

# NLP research has evolved from the era of punch cards and
batch processing to the era of Google and the likes of it
which process millions of web pages
# Work in NLP field has concentrated first on one problem
then on another
# Most NLP research carried out in the early days focused on
syntax, partly because syntactic processing was manifestly
necessary
# However,there were some researchers who concentrated
on semantics also at that time

THE NLP RESEARCH - Introduction

# In early semantic based work like Masterman and Ceccato

(Ceccato, 1967) world knowledge was deemed as
requirement for linguistic semantics
# Later works also recognized need for knowledge
representation
Minsky(1968)
Schank (1975)

Knowledge Representation

MINSKY THEORY

# Society of Mind theory (Minsky, 1986) - basis of Network

Representation of Knowledge
# Agents: Minsky theorized that the mind is made of many
little parts called agents.
# These groups of agents, or agencies, are responsible for
performing some type of functions
remembering, comparing, generalizing, exemplifying,
analogizing, simplifying, predicting,

# It has given birth to many attempts to build common-sense

knowledge bases for NLP tasks
The Open Mind Common Sense project

OPEN COMMON SENSE PROJECT

# Knowledge represented in Natural Language

# Not hand-crafted by expert engineers but spontaneously

inserted by online volunteers.
# Today its used for various NLP tasks

Textual Affect Sensing (H. Liu,Lieberman, Selker, 2003)

Casual Conversation Understanding (Eagle, Singh,
Pentland, 2003)
Opinion Mining (Cambria ,Hussain, 2012)
Story Telling (Hayden et al., 2013)

Jumping Curves

# NLP research has not evolved at the same pace as other

technologies in the past 15 years.
# NLP research has made great strides in producing
artificially intelligent behaviors
Googles AlphaGo
IBMs Watson
Apples Siri

# User Generated Content - ever increasing

# Hence the need to - Jump The curve

Jumping Curves

Syntactic Curve

# Syntax centered NLP still most popular way to manage

NLP tasks
Information Retrieval and Extraction
Auto categorization
Topic Modelling

# Classified mainly into 3 categories

Keyword Spotting
Lexical Affinity
Statistical NLP

Syntactic Curve - Keyword Spotting

# Naive Approach

# Text classified into categories based on the presence of

fairly unambiguous words
# Popular Projects

Ortonys Affective Lexicon (Ortony, Clore, Collins, 1988)

Penn Treebank (Marcus, Santorini, Marcinkiewicz, 1994)
PageRank (Page, Brin, Motwani, Winograd, 1999)
LexRank (Gnes, Radev, 2004)

# Issues

Relies on presence of obvious words

Syntactic Curve - Lexical Affinity

# More Sophisticated

# Assigns to arbitrary words a probabilistic affinity for a

particular category
Eg: "Accident" might be assigned a 75% probability of
indicating a negative event, as in "car accident"

# Issues

If based solely on word-level can be tricked , I avoided an

accident
difficult to develop a re-usable, domain-independent model

Syntactic Curve - Statistical NLP

# Mainstream direction since 1990

# Relies on language model based on ML Algorithms

# Capable of KS and LA, but also punctuation and word

co-occurrence frequency
Eg: "Accident" might be assigned a 75% probability of
indicating a negative event, as in "car accident"

# Issues

Semantically Weak
Doesnt work well on smaller text units - sentences or
clauses

Semantic Curve

# Focuses on the intrinsic meaning associated with natural

language text.
# Steps away from the blind usage of keywords and word
co-occurrence count
# Broadly categorized into 2 categories based on use of :
External Knowledge (Taxonomic)
Intrinsic Semantics of Documents (Endogenous NLP)

Semantic Curve - Endogenous NLP

# Uses ML techniques for semantic analysis of a corpus by

building structures that approximate concept.
# Advantages

Effective
Less Expert Manpower
Portable

Semantic Curve - Taxonomic NLP

# Initiative aiming to build universal taxonomies for

grasping hierarchical semantics
# Semantic Knowledge built upon "isA" relationship
# Eg Pablo Picasso - isA - artist
# Issues

Strictly defined - hence less flexible

Eg. A book as a paperweight

# Wiki Taxonomy is a popular Semantic Web Project

Pragmatic Curve

# "Narratives"

# Decoding how narratives are generated and processed by

the human brain
Might lead us to truly understand and explain human
intelligence and consciousness.
Central for reasoning, decision-making, and
"sense-making"

# Computational Model - A powerful way to investigate

narrative understanding.
# Once NLP Research grasps semantics at a level equal to
Human Text Processing, jump to Pragmatic curve can be
made

Pragmatic Curve

# Few Works done in Pragmatic Analysis are

Plan Graphs (Young, 2007)

Common-Sense Reasoning (Mueller, 2007)
Computational Models of Narrative(Patrick Winston, 2009,
2011)

# Computational Models of Narrative(Patrick Winston, 2009,

2011)
Believes that human intelligence stems from our unique
abilities for "storytelling" and "understanding"
lists 5 key hypothesis that tries to emulate story telling
ability

Recent Advances

# Advances in Neural Nets And Deep Learning

Plan Graphs (Young, 2007)

# Speech Processing

Amodei et al. (2015). End-to-End Speech Recognition in

English and Mandarin
Though the 2 languages very different - same system can be
used to recognize speech
Applications : Speech Recognition, Spoken Dialogues

# News Reading from Google DeepMind

A Neural Network that maximises the probability of an

answer given a news article and a question on it

Recent Advances

# Word Prediction for mobile typing by SwiftKey

Based on Neural Language Models (LM)

Given several previous words, whats the most probable
next word
Limitation of Traditional LMs - a word can make sense with
2 previous words but not in context
Neural Language Models - are able to look further back and
make sense of the whole expression

Recent Advances

# Neural Nets And Deep Learning

Artificial Neural Network algorithms are inspired by

biological neural networks
Deep Learning refers to ANNs with higher number of
hidden layers
Many recent advances in some NLP tasks are due to
advances in ANNs and DL

Recent Advances - NLP Meets Deep Learning

# Natural Language Reasoning

One of the early tasks that showed that neural nets can help
machines reason like we do "Analogy Tasks" (Mikolov et
al., 2013)
Eg - man is to woman as king is to ___ ?

Recent Advances - NLP Meets Deep Learning

# Natural Language Reasoning

One of the early tasks that showed that neural nets can help
machines reason like we do "Analogy Tasks" (Mikolov et
al., 2013):
Eg. man is to woman as king is to ___ ?
Think of that as of :
meaning(king) - meaning(man) + meaning(woman) = ?

Recent Advances - NLP Meets Deep Learning

# How Machines can understand the meaning

Imagine all words and their meanings live in a

high-dimensional space called- Semantic space
Each dimension in this space represents some aspect of the
word meaning
Concepts and words that mean similar things should live
close to each other in this space

Recent Advances - NLP Meets Deep Learning

# Represent Words as Vectors by

Learning from a lot of data

Use of ML Algorithms (eg, Deep Learning Algorithms)

# meaning(king) - meaning(man) + meaning(woman) = ?

# vector(king) - vector(man) + vector(woman) = vector(?)

References

Erik Cambria, Bebo White Jumping NLP Curves: A Review of

Natural Language Processing Research. IEEE Computational
Intelligence Magazine, May 2014
Karen Sparck Jones Natural Language Processing: A
Historical Review. Computer Laboratory, University of
Cambridge, October 2001
Tomas Mikolov, Ilya Sutskever, Kai Chen, Jeffrey Dean,
Greg Corrado Distributed Representations of Words and
Phrases and their Compositionality. Google Inc. Mountain
View, 2013
Dario Amodei End-to-End Speech Recognition in English and
Mandarin. Baidu Silicon Valley AI Lab, 2015

References

Moritz Hermann Teaching Machines to Read and Comprehend.

DeepMind, 2015
J. Allen Natural Language Understanding. Redwood City, CA:
Benjamin/Cummings. 1987
Artificial Intelligence: Natural Language Processing,
https://www.tutorialspoint.com/artificialintelligence.
SwiftKey , https://blog.swiftkey.com/neural-networks-ameaningful-leap-for-mobile-typing/. SwiftKey, October
2015