News Article

Summarization
Ryan Endacott and Krit Pattamadit
Project Goal
● To provide a concise and informative
summary of a given news article
● Reduce information overload
● Quickly determine which articles are worth
reading
Corpora
● Brown Corpus news articles
● New York Times articles
● Quora answers
● Can be used on any text
Summarization
● Extraction vs Abstraction
● Steps:
○ Preprocess and tokenize article
○ Transform into intermediate representation for
analysis
○ Generate summary analysis
Sentence Extraction
● Analyze and classify the importance of each
sentence in order to form a summary with
existing sentences
● Author already conveyed main ideas, we just
have to pick them out
● Abstraction is hard and hasn’t given good
results in the past
Preprocessing and Interpretation
● Goal: tokenize input into sentences and
words for evaluation
● NLTK sent_tokenizer and word_tokenizer
How to choose sentences
● Goal: Choose the most important n
sentences to be the summary
● Number of Proper Nouns
● Distance from start of article
● TextRank
TextRank
● Unsupervised graph-based ranking
algorithm for text
● Based on Google PageRank
● Takes into account:
○ Word frequency
○ Keywords
○ Relationship between sentences
TextRank continued
● Connected, undirected graph of the text
● Each sentence is a vertex
● Edges are similarity between sentences
● Weighted by word occurrence and frequency
Results
● Simple summaries of main article ideas
● Large reduction in quantity of text while
conveying main points and ideas
● Examples…
Beer Running
I ran and drank beer every ● Tim Cigelske, Beer
day for 3 years. It started
Runner
the streak on my 29th
birthday and my goal was to ● http://qr.ae/Gk4bO
run at least one mile and ● 415 words -> 52
drink at least one beer per words
day for a year. I wanted to
do something big for the
final year of my 20s.
Being CEO of a Startup
The exponential productivity from great
people will always amaze you. You will ● Paul DeJoe,
not settle for things anymore because you
will see what is possible when you hold
founder of Ecquire.
out for the best and push to find people com
that are the best. You'll become addicted
to finding the hardest challenges because ● http://qr.ae/GkvvP
there's a direct relationship between how
difficult something is and the euphoria of a
● 1198 words -> 71
feeling when you do the impossible. words
Biden on China Air Dispute
BEIJING --- Chinese leaders pushed back at
visiting Vice President Joseph R. Biden Jr. on ● New York Times
Wednesday over what they assert is their right
to control a wide swath of airspace in the bitterly
● http://www.nytimes.
contested East China Sea. But the Chinese also com/2013/12/05/world/a
indicated that they had not decided how
aggressively to enforce their so-called air sia/biden-arrives-in-
defense identification zone, which has ignited
tensions with Japan. Shuttling from one feuding
china-seeking-restraint-
neighbor to the other, Mr. Biden arrived here from-beijing.html?
from Tokyo to urge China’s president, Xi
Jinping, to show restraint in the zone, which Mr. smid=pl-share
Biden said the United States regarded as
illegitimate and a provocation. ● 1090 words -> 102
words
Problems & Further Development
● Didn’t complete web crawler, currently just
takes raw text
● Works best on articles with one central idea
● Possible improvements include:
○ Adding title to the graph (with extra weight?)
○ Word stemming or lemmatization
○ Sentence position
○ Sentence length
Live Demo!
http://nlpsummarize.herokuapp.com/

Source
https://github.com/ryan-endacott/nlp
Thanks!
Questions? Comments? Ideas for
improvement?