Text Classification

and Nave Bayes

1he 1ask of 1exL
Classlcauon
uan !urafsky
Is th|s spam?
uan !urafsky
Who wrote wh|ch Iedera||st papers?
1787-8: anonymous essays Lry Lo convlnce new ?ork
Lo raufy u.S ConsuLuuon: !ay, Madlson, PamllLon.
AuLhorshlp of 12 of Lhe leuers ln dlspuLe
1963: solved by MosLeller and Wallace uslng
8ayeslan meLhods
!ames Madlson
Alexander PamllLon
uan !urafsky
Ma|e or fema|e author?
1. 8y 1923 presenL-day vleLnam was dlvlded lnLo Lhree parLs
under lrench colonlal rule. 1he souLhern reglon embraclng
Salgon and Lhe Mekong delLa was Lhe colony of Cochln-Chlna,
Lhe cenLral area wlLh lLs lmperlal caplLal aL Pue was Lhe
proLecLoraLe of Annam.
2. Clara never falled Lo be asLonlshed by Lhe exLraordlnary fellclLy
of her own name. She found lL hard Lo LrusL herself Lo Lhe
mercy of faLe, whlch had managed over Lhe years Lo converL
her greaLesL shame lnLo one of her greaLesL asseLs.
S. Argamon, M. koppel, !. llne, A. 8. Shlmonl, 2003. Cender, Cenre, and Wrlung SLyle ln lormal Wrluen 1exLs," 1exL, volume 23, number 3, pp.
321-346
uan !urafsky
os|nve or neganve mov|e rev|ew?
unbellevably dlsappolnung
lull of zany characLers and rlchly applled saure, and some
greaL ploL LwlsLs
Lhls ls Lhe greaLesL screwball comedy ever lmed
lL was paLheuc. 1he worsL parL abouL lL was Lhe boxlng
scenes.
3
uan !urafsky
What |s the sub[ect of th|s arnc|e?
AnLogonlsLs and lnhlblLors
8lood Supply
ChemlsLry
urug 1herapy
Lmbryology
Lpldemlology
.
6
MeSn Sub[ect Category n|erarchy
?
MEDLINE Article

uan !urafsky
1ext C|ass|hcanon
Asslgnlng sub[ecL caLegorles, Loplcs, or genres
Spam deLecuon
AuLhorshlp ldenucauon
Age/gender ldenucauon
Language ldenucauon
SenumenL analysls
.
uan !urafsky
1ext C|ass|hcanon: dehn|non
!"#$%:
a documenL &
a xed seL of classes ( = [)
1
, )
2
,., )
*
}
+$%#$%: a predlcLed class ) ! (

uan !urafsky
C|ass|hcanon Methods:
nand-coded ru|es
8ules based on comblnauons of words or oLher feaLures
spam: black-llsL-address C8 (dollars" Anuhave been selecLed")
Accuracy can be hlgh
lf rules carefully rened by experL
8uL bulldlng and malnLalnlng Lhese rules ls expenslve
uan !urafsky
C|ass|hcanon Methods:
Superv|sed Mach|ne Learn|ng
!"#$%,
a documenL &
a xed seL of classes ( = [)
1
, )
2
,., )
*
}
A Lralnlng seL of - hand-labeled documenLs .&
/
0)
/
1022220.&
-
0)
-
1
+$%#$%,
a learned classler 3,& ! )
10
uan !urafsky
C|ass|hcanon Methods:
Superv|sed Mach|ne Learn|ng
Any klnd of classler
naive 8ayes
Loglsuc regresslon
SupporL-vecLor machlnes
k-nearesL nelghbors
.
Text Classification
and Nave Bayes
1he 1ask of 1exL
Classlcauon
Text Classification
and Nave Bayes
naive 8ayes (l)
uan !urafsky
Nave 8ayes Intu|non
Slmple (naive") classlcauon meLhod based on
8ayes rule
8elles on very slmple represenLauon of documenL
8ag of words
uan !urafsky
1he bag of words representanon
I love this movie! It's sweet,
but with satirical humor. The
dialogue is great and the
adventure scenes are fun It
manages to be whimsical and
romantic while laughing at the
conventions of the fairy tale
genre. I would recommend it to
just about anyone. I've seen
it several times, and I'm
always happy to see it again
whenever I have a friend who
hasn't seen it yet.
!(
)=c
uan !urafsky
1he bag of words representanon
I love this movie! It's sweet,
but with satirical humor. The
dialogue is great and the
adventure scenes are fun It
manages to be whimsical and
romantic while laughing at the
conventions of the fairy tale
genre. I would recommend it to
just about anyone. I've seen
it several times, and I'm
always happy to see it again
whenever I have a friend who
hasn't seen it yet.
!(
)=c
uan !urafsky
1he bag of words representanon:
us|ng a subset of words
x love xxxxxxxxxxxxxxxx sweet
xxxxxxx satirical xxxxxxxxxx
xxxxxxxxxxx great xxxxxxx
xxxxxxxxxxxxxxxxxxx fun xxxx
xxxxxxxxxxxxx whimsical xxxx
romantic xxxx laughing
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxx recommend xxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xx several xxxxxxxxxxxxxxxxx
xxxxx happy xxxxxxxxx again
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxx
!(
)=c
uan !urafsky
1he bag of words representanon
!(
)=c
great 2
love 2
recommend 1
laugh 1
happy 1
... ...
uan !urafsky
Planning GUI
Garbage
Collection
Machine
Learning
NLP
parser
tag
training
translation
language...
learning
training
algorithm
shrinkage
network...
garbage
collection
memory
optimization
region...


Test
document

parser
language
label
translation

8ag of words for document c|ass|hcanon
...
planning
temporal
reasoning
plan
language...
?
Text Classification
and Nave Bayes
naive 8ayes (l)
Text Classification
and Nave Bayes
lormallzlng Lhe
naive 8ayes
Classler
uan !urafsky
8ayes' ku|e App||ed to Documents and
C|asses
P(c | d) =
P(d | c)P(c)
P(d)
- lor a documenL & and a class )
uan !urafsky
Nave 8ayes C|ass|her (I)
c
MAP
= argmax
c!C
P(c | d)
= argmax
c!C
P(d | c)P(c)
P(d)
= argmax
c!C
P(d | c)P(c)
MAP is maximum a
posteriori = most
likely class
Bayes Rule
Dropping the
denominator
uan !urafsky
Nave 8ayes C|ass|her (II)
c
MAP
= argmax
c!C
P(d | c)P(c)
Document d
represented as
features
x1..xn
= argmax
c!C
P(x
1
, x
2
,, x
n
| c)P(c)
uan !urafsky
Nave 8ayes C|ass|her (IV)
How often does this
class occur?
c
MAP
= argmax
c!C
P(x
1
, x
2
,, x
n
| c)P(c)
C(|4|
"
-|(|) parameLers
We can just count the
relative frequencies in
a corpus
Could only be esumaLed lf a
very, very large number of
Lralnlng examples was
avallable.
uan !urafsky
Mu|nnom|a| Nave 8ayes Independence
Assumpnons
P(x
1
, x
2
,, x
n
| c)
8ag of Words assumpnon: Assume posluon doesn'L
mauer
Cond|nona| Independence: Assume Lhe feaLure
probablllues 5(6
7
|)
8
) are lndependenL glven Lhe class )2
P(x
1
,, x
n
| c) = P(x
1
| c)P(x
2
| c)P(x
3
| c)... P(x
n
| c)
uan !urafsky
Mu|nnom|a| Nave 8ayes C|ass|her
c
MAP
= argmax
c!C
P(x
1
, x
2
,, x
n
| c)P(c)
c
NB
= argmax
c!C
P(c
j
) P(x | c)
x!X
"
uan !urafsky
App|y|ng Mu|nnom|a| Na|ve 8ayes
C|ass|hers to 1ext C|ass|hcanon
c
NB
= argmax
c
j
!C
P(c
j
) P(x
i
| c
j
)
i!positions
"
positions " all word posluons ln LesL documenL

Text Classification
and Nave Bayes
lormallzlng Lhe
naive 8ayes
Classler
Text Classification
and Nave Bayes
naive 8ayes:
Learnlng
uan !urafsky
Learn|ng the Mu|nnom|a| Nave 8ayes Mode|
llrsL auempL: maxlmum llkellhood esumaLes
slmply use Lhe frequencles ln Lhe daLa
Sec.13.3

P(w
i
| c
j
) =
count(w
i
, c
j
)
count(w, c
j
)
w!V
"

P(c
j
) =
doccount(C = c
j
)
N
doc
uan !urafsky
CreaLe mega-documenL for Loplc 8 by concaLenaung all docs ln
Lhls Loplc
use frequency of 9 ln mega-documenL
arameter esnmanon
fracuon of umes word 9
7
appears
among all words ln documenLs of Loplc )
8

P(w
i
| c
j
) =
count(w
i
, c
j
)
count(w, c
j
)
w!V
"
uan !urafsky
rob|em w|th Max|mum L|ke||hood
WhaL lf we have seen no Lralnlng documenLs wlLh Lhe word
!"#$"%&' and classled ln Lhe Loplc pos|nve ($()*+%,)-.?

Zero probablllues cannoL be condluoned away, no mauer
Lhe oLher evldence!

P("fantastic" positive) =
count("fantastic", positive)
count(w, positive
w!V
"
)
= 0
c
MAP
= argmax
c

P(c)

P(x
i
| c)
i
!
Sec.13.3
uan !urafsky
Lap|ace (add-1) smooth|ng for Nave 8ayes

P(w
i
| c) =
count(w
i
, c) +1
count(w, c) +1
( )
w!V
"
=
count(w
i
, c) +1
count(w, c
w!V
"
)
#
$
%
%
&
'
(
(
+ V

P(w
i
| c) =
count(w
i
, c)
count(w, c)
( )
w!V
"
uan !urafsky
Mu|nnom|a| Nave 8ayes: Learn|ng
CalculaLe 5()
8
) Lerms
lor each )
8
ln ( do
&:);
8
" all docs wlLh class =)
8
P(w
k
| c
j
) !
n
k
+!
n +! | Vocabulary |
P(c
j
) !
| docs
j
|
| total # documents|
CalculaLe 5(9
<
| )
8
) Lerms
=>6%
8
" slngle doc conLalnlng all &:);
8
lor

each word 9
<
ln ?:)@A$B@CD
"
<
" # of occurrences of 9
<
ln =>6%
8
lrom Lralnlng corpus, exLracL Vocabulary
Text Classification
and Nave Bayes
naive 8ayes:
Learnlng
Text Classification
and Nave Bayes
naive 8ayes:
8elauonshlp Lo
Language Modellng
uan !urafsky
Generanve Mode| for Mu|nnom|a| Nave 8ayes
38
c=China
X
1
=Shanghai

X
2
=and

X
3
=Shenzhen

X
4
=issue

X
5
=bonds

uan !urafsky
Nave 8ayes and Language Mode||ng
naive bayes classlers can use any sorL of feaLure
u8L, emall address, dlcuonarles, neLwork feaLures
8uL lf, as ln Lhe prevlous slldes
We use on|y word feaLures
we use a|| of Lhe words ln Lhe LexL (noL a subseL)
1hen
naive bayes has an lmporLanL slmllarlLy Lo language
modellng.
39
uan !urafsky
Lach c|ass = a un|gram |anguage mode|
Asslgnlng each word: (word | c)
Asslgnlng each senLence: (s|c)=# (word|c)
0.1 l
0.1 love
0.01 Lhls
0.03 fun
0.1 lm
.
l love Lhls fun lm
0.1 0.1 .03 0.01 0.1
Class #:;
(s | pos) = 0.0000003
Sec.13.2.1
uan !urafsky
Nave 8ayes as a Language Mode|
Whlch class asslgns Lhe hlgher probablllLy Lo s?
0.1 l
0.1 love
0.01 Lhls
0.03 fun
0.1 lm
Model pos Model neg
lm love Lhls fun l
0.1 0.1 0.01 0.03 0.1
0.1 0.001 0.01 0.003 0.2
(s|pos) > (s|neg)
0.2 l
0.001 love
0.01 Lhls
0.003 fun
0.1 lm
Sec.13.2.1
Text Classification
and Nave Bayes
naive 8ayes:
8elauonshlp Lo
Language Modellng
Text Classification
and Nave Bayes
Mulunomlal naive
8ayes: A Worked
Lxample
uan !urafsky
Choos|ng a c|ass:
(c|d3)



([|d3)



1/4 * (2/9)
3
* 2/9 * 2/9
= 0.0001
Doc Words C|ass
1ralnlng 1 Chlnese 8el[lng Chlnese c
2 Chlnese Chlnese Shanghal c
3 Chlnese Macao c
4 1okyo !apan Chlnese [
1esL 3 Chlnese Chlnese Chlnese 1okyo !apan ?
44
Cond|nona| robab|||nes:
(Chlnese|)) =
(1okyo|)) =
(!apan|)) =
(Chlnese|8) =
(1okyo|8) =
(!apan|8) =
r|ors:
5())=
5(8)=
3

4
1

4

P(w| c) =
count(w, c) +1
count(c)+| V |

P(c) =
N
c
N
(3+1) / (8+6) = 6/14 = 3/7
(0+1) / (8+6) = 1/14
(1+1) / (3+6) = 2/9
(0+1) / (8+6) = 1/14
(1+1) / (3+6) = 2/9
(1+1) / (3+6) = 2/9
3/4 * (3/7)
3
* 1/14 * 1/14
= 0.0003
!
!
uan !urafsky
Nave 8ayes |n Spam I||ter|ng
SpamAssassln leaLures:
Menuons Cenerlc vlagra
Cnllne harmacy
Menuons mllllons of (dollar) ((dollar) nn,nnn,nnn.nn)
hrase: lmpress ... glrl
lrom: sLarLs wlLh many numbers
Sub[ecL ls all caplLals
P1ML has a low rauo of LexL Lo lmage area
Cne hundred percenL guaranLeed
Clalms you can be removed from Lhe llsL
'resuglous non-AccredlLed unlverslues'
hup://spamassassln.apache.org/LesLs_3_3_x.hLml
uan !urafsky
Summary: Na|ve 8ayes |s Not So Na|ve
very lasL, low sLorage requlremenLs
8obusL Lo lrrelevanL leaLures
lrrelevanL leaLures cancel each oLher wlLhouL aecung resulLs
very good ln domalns wlLh many equally lmporLanL feaLures
ueclslon 1rees suer from EC@F->"%@G:" ln such cases - especlally lf llule daLa
Cpumal lf Lhe lndependence assumpuons hold: lf assumed
lndependence ls correcL, Lhen lL ls Lhe 8ayes Cpumal Classler for problem
A good dependable basellne for LexL classlcauon
8ut we w||| see other c|ass|hers that g|ve beuer accuracy
Text Classification
and Nave Bayes
Mulunomlal naive
8ayes: A Worked
Lxample
Text Classification
and Nave Bayes
reclslon, 8ecall, and
Lhe l measure
uan !urafsky
1he 2-by-2 connngency tab|e
correcL noL correcL
selecLed Lp fp
noL selecLed fn Ln
uan !urafsky
rec|s|on and reca||
rec|s|on: of selecLed lLems LhaL are correcL
keca||: of correcL lLems LhaL are selecLed
correcL noL correcL
selecLed Lp fp
noL selecLed fn Ln
uan !urafsky
A comb|ned measure: I
A comblned measure LhaL assesses Lhe /8 Lradeo ls l measure
(welghLed harmonlc mean):
1he harmonlc mean ls a very conservauve average, see !!H 8.3
eople usually use balanced l1 measure
l.e., wlLh $ = 1 (LhaL ls, % = ): I = 25H/(5+H)
R P
PR
R P
F
+
+
=
! +
=
2
2
) 1 (
1
) 1 (
1
1
"
"
# #
Text Classification
and Nave Bayes
reclslon, 8ecall, and
Lhe l measure
Text Classification
and Nave Bayes
1exL Classlcauon:
Lvaluauon
uan !urafsky
34
More 1han 1wo C|asses:
Sets of b|nary c|ass|hers
ueallng wlLh any-of or muluvalue classlcauon
A documenL can belong Lo 0, 1, or >1 classes.
lor each class cC
8ulld a classler
c
Lo dlsungulsh c from all oLher classes c' C
Clven LesL doc d,
LvaluaLe lL for membershlp ln each class uslng each
c

d belongs Lo any class for whlch
c
reLurns Lrue
Sec.14.5
uan !urafsky
33
More 1han 1wo C|asses:
Sets of b|nary c|ass|hers
Cne-of or mulunomlal classlcauon
Classes are muLually excluslve: each documenL ln exacLly one class
lor each class cC
8ulld a classler
c
Lo dlsungulsh c from all oLher classes c' C
Clven LesL doc d,
LvaluaLe lL for membershlp ln each class uslng each
c

d belongs Lo Lhe one class wlLh maxlmum score
Sec.14.5
uan !urafsky
36
MosL (over)used daLa seL, 21,378 docs (each 90 Lypes, 200 Loknens)
9603 Lralnlng, 3299 LesL arucles (ModApLe/Lewls spllL)
118 caLegorles
An arucle can be ln more Lhan one caLegory
Learn 118 blnary caLegory dlsuncuons
Average documenL (wlLh aL leasL one caLegory) has 1.24 classes
Cnly abouL 10 ouL of 118 caLegorles are large

Common categories
(#train, #test)
Lva|uanon:
C|ass|c keuters-21S78 Data Set
Earn (2877, 1087)
Acquisitions (1650, 179)
Money-fx (538, 179)
Grain (433, 149)
Crude (389, 189)
Trade (369,119)
Interest (347, 131)
Ship (197, 89)
Wheat (212, 71)
Corn (182, 56)
Sec. 15.2.4
uan !urafsky
37
keuters 1ext Categor|zanon data set
(keuters-21S78) document
<REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="12981"
NEWID="798">
<DATE> 2-MAR-1987 16:51:43.42</DATE>
<TOPICS><D>livestock</D><D>hog</D></TOPICS>
<TITLE>AMERICAN PORK CONGRESS KICKS OFF TOMORROW</TITLE>
<DATELINE> CHICAGO, March 2 - </DATELINE><BODY>The American Pork Congress kicks off tomorrow,
March 3, in Indianapolis with 160 of the nations pork producers from 44 member states determining industry positions
on a number of issues, according to the National Pork Producers Council, NPPC.
Delegates to the three day Congress will be considering 26 resolutions concerning various issues, including the future
direction of farm policy and the tax law as it applies to the agriculture sector. The delegates will also debate whether to
endorse concepts of a national PRV (pseudorabies virus) control and eradication program, the NPPC said.
A large trade show, in conjunction with the congress, will feature the latest in technology in all areas of the industry,
the NPPC added. Reuter
&#3;</BODY></TEXT></REUTERS>
Sec. 15.2.4
uan !urafsky
Confus|on matr|x c
lor each palr of classes c
1
,c
2
> how many documenLs from c
1

were lncorrecLly asslgned Lo c
2
?
c
3,2
: 90 wheaL documenLs lncorrecLly asslgned Lo poulLry
38
Docs |n test set Ass|gned
Uk
Ass|gned
pou|try
Ass|gned
wheat
Ass|gned
coee
Ass|gned
|nterest
Ass|gned
trade
1rue Uk 93 1 13 0 1 0
1rue pou|try 0 1 0 0 0 0
1rue wheat 10 90 0 1 0 0
1rue coee 0 0 0 34 3 7
1rue |nterest - 1 2 13 26 3
1rue trade 0 0 2 14 3 10
uan !urafsky
39
er c|ass eva|uanon measures
keca||:
lracuon of docs ln class 7 classled correcLly:
rec|s|on:
lracuon of docs asslgned class 7 LhaL are
acLually abouL class 7:

Accuracy: (1 - error raLe)
lracuon of docs classled correcLly:
c
ii
i
!
c
ij
i
!
j
!
c
ii
c
ji
j
!
c
ii
c
ij
j
!
Sec. 15.2.4
uan !urafsky
60
M|cro- vs. Macro-Averag|ng
lf we have more Lhan one class, how do we comblne
muluple performance measures lnLo one quanuLy?
Macroaverag|ng: CompuLe performance for each class,
Lhen average.
M|croaverag|ng: CollecL declslons for all classes,
compuLe conungency Lable, evaluaLe.
Sec. 15.2.4
uan !urafsky
61
M|cro- vs. Macro-Averag|ng: Lxamp|e
1ruLh:
yes
1ruLh:
no
Classler: yes 10 10
Classler: no 10 970
1ruLh:
yes
1ruLh:
no
Classler: yes 90 10
Classler: no 10 890
1ruLh:
yes
1ruLh:
no
Classler: yes 100 20
Classler: no 20 1860
Class 1 Class 2 Mlcro Ave. 1able
Sec. 13.2.4
Macroaveraged preclslon: (0.3 + 0.9)/2 = 0.7
Mlcroaveraged preclslon: 100/120 = .83
Mlcroaveraged score ls domlnaLed by score on common classes
uan !urafsky
Deve|opment 1est Sets and Cross-va||danon
MeLrlc: /8/l1 or Accuracy
unseen LesL seL
avold overng (Lunlng Lo Lhe LesL seL')
more conservauve esumaLe of performance
Cross-valldauon over muluple spllLs
Pandle sampllng errors from dlerenL daLaseLs
ool resulLs over each spllL
CompuLe pooled dev seL performance
1ralnlng seL uevelopmenL 1esL SeL 1esL SeL
1esL SeL
1ralnlng SeL
1ralnlng SeL uev 1esL
1ralnlng SeL
uev 1esL
uev 1esL
Text Classification
and Nave Bayes
1exL Classlcauon:
Lvaluauon
Text Classification
and Nave Bayes
1exL Classlcauon:
racucal lssues
uan !urafsky
63
The Real World
Cee, l'm bulldlng a LexL classler for real, now!
WhaL should l do?
Sec. 15.3.1
uan !urafsky
66
No training data?
Manually written rules
lf (wheaL or graln) and noL (whole or bread) Lhen
CaLegorlze as graln
need careful cralng
Puman Lunlng on developmenL daLa
1lme-consumlng: 2 days per class
Sec. 15.3.1
uan !urafsky
67
Very little data?
use naive 8ayes
naive 8ayes ls a hlgh-blas" algorlLhm (ng and !ordan 2002 nlS)
CeL more labeled daLa
llnd clever ways Lo geL humans Lo label daLa for you
1ry seml-supervlsed Lralnlng meLhods:
8ooLsLrapplng, LM over unlabeled documenLs, .
Sec. 15.3.1
uan !urafsky
68
A reasonable amount of data?
erfecL for all Lhe clever classlers
SvM
8egularlzed Loglsuc 8egresslon
?ou can even use user-lnLerpreLable declslon Lrees
users llke Lo hack
ManagemenL llkes qulck xes
Sec. 15.3.1
uan !urafsky
69
A huge amount of data?
Can achleve hlgh accuracy!
AL a cosL:
SvMs (Lraln ume) or knn (LesL ume) can be Loo slow
8egularlzed loglsuc regresslon can be somewhaL beuer
So naive 8ayes can come back lnLo lLs own agaln!
Sec. 15.3.1
uan !urafsky
70
Accuracy as a function of data size
WlLh enough daLa
Classler may noL mauer
Sec. 15.3.1
8rlll and 8anko on spelllng correcuon
uan !urafsky
kea|-wor|d systems genera||y comb|ne:
AuLomauc classlcauon
Manual revlew of uncerLaln/dlculL/new" cases
71
uan !urafsky
Underow revennon: |og space
Muluplylng loLs of probablllues can resulL ln oaung-polnL underow.
Slnce log(6D) = log(6) + log(D)
8euer Lo sum logs of probablllues lnsLead of muluplylng probablllues.
Class wlLh hlghesL un-normallzed log probablllLy score ls sull mosL probable.
Model ls now [usL max of sum of welghLs
c
NB
= argmax
c
j
!C
logP(c
j
) + logP(x
i
| c
j
)
i!positions
"
uan !urafsky
73
How to tweak performance
uomaln-speclc feaLures and welghLs: J>CD lmporLanL ln real
performance
Someumes need Lo collapse Lerms:
arL numbers, chemlcal formulas, .
8uL sLemmlng generally doesn'L help
upwelghung: Counung a word as lf lL occurred Lwlce:
uLle words (Cohen Slnger 1996)
rsL senLence of each paragraph (MuraLa, 1999)
ln senLences LhaL conLaln uLle words (ko >% @B0 2002)
Sec. 15.3.2
Text Classification
and Nave Bayes
1exL Classlcauon:
racucal lssues