Song Genre Classification through Quantitative Analysis of Lyrics
Doran Walsten
Johns Hopkins University
dwalste1@jhu.edu
Abstract
The goal of this project was to implement a
machine learning algorithm to the challenging
problem of song genre identification. Specifi-
cally, we were interested in learning how well
a standard clustering algorithm, K-means,
could cluster similar songs into genres. In ad-
dition, we attempted to find which features of
the text we calculated were most valuable in
clustering.
1 Introduction
An important concept in machine learning is clas-
sification. Classification is loosely defined as la-
beling an input piece of data with a particular label
that identifies that specific input with a subset of
all data received. Machine learning is the training
of a computer or algorithm to successfully label
input with minimal error.
For this project, our inspirations were mobile
song identification applications such as Sound-
Hound and Shazam. These applications are able to
listen to audio and accurately identify songs. How-
ever, we were interested in whether the lyrics of
song contain enough information to classify a song
in a particular genre.
Mayer, et al. (2008) wrote an article detailing
their method for identifying the genre of a song.
The three types of features that they analyzed in
the text were rhyme scheme, part of speech, and
general statistics. These general statistics included
average word length, average length of line, words
per minute, etc. The novel approach of their paper
was to analyze the rhyming scheme to assist with
classification. This helped improve classification a
Daivik Orth
Johns Hopkins Univeristy
dorth4@jhu.edu
marginal amount. However, we wanted to try a
different feature of the text instead: sentiment.
Sentiment analysis in natural language pro-
cessing is a complex problem. The goal of senti-
ment analysis is to determine the positivity or
negativity of a string based on the words given and
the relationships between those words. For a hu-
man, it is relatively simple to detect this sentiment.
However, for machines its a bit more challenging.
Just as in real life, sentiment depends on context.
For example, although the word funny is consid-
ered positive, not funny is considered negative.
Thus, an algorithm cannot simply look at the
words present in the string, it must be able to de-
termine which words are related to each other first.
2 Machine Learning Methods
In this project, we utilized two main machine
learning concepts in order to generate our clusters
of genres. These two concepts were K-means clus-
tering and Sentiment analysis.
2.1 K-means
K-means is an algorithm that aims to take n-
observations and cluster them into k clusters,
where every observation belongs to the cluster
with the nearest mean. The problem is computa-
tionally NP-hard, we implemented an efficient
heuristic algorithm that converges to to a local op-
timal solution. Given a set of observations x1,
x2,, xn, each of which is an m-dimensional vec-
tor, the goal of K-Means is to group these vectors
into sets S1, S2, , Sk, where k is the number of
clusters. The objective function to minimize is the
Euclidean distance between each point and the
centroid of the cluster it is assigned to. This outline
of K-means as well as the pseudocode for the actu-
al implementation were found at the Stanford Ar-
tifical Intelligence course webpage (Piech, 2013).
The first step of the algorithm is to initialize a
starting set of clusters by choosing k observations
in the data set. K-means works best when the ini-
tial clusters are as far apart as possible. To do this,
the first cluster is chosen uniformly at random.
Through k-1 iterations, the distance to the nearest
cluster is computed for each observation. Then,
another cluster is chosen using a weighted proba-
bility distribution such that the probability of a par-
ticular observation being chosen as the next center
is proportional to the distance to the point squared.
Next, we repeat the following assignment and
update steps until convergence.
Assign every observation to the nearest
cluster mean based on Euclidean distance
Update the new means to be centroids of
the observations in the new clusters.
We chose to use K-Means primarily because we
wanted to determine whether or not artists could be
grouped into genres by clustering the artists songs
without genre labels. We also would have had to
generate our own labels, which is both time con-
suming and restricting on the potential genres a
song could be a member of. We felt that K-Means
would be a good starting point for unsupervised
learning/classification for a few reasons. Firstly, K-
Means is relatively sensitive to outliers and noise
in the data, which helped in determining what set
of features and artists allowed for the best training
and test accuracies. If we were to continue this pro-
ject, we would take our K-Means implementation
and used it to initialize Expectation Maximization
with a Gaussian Mixture.
2.2 Sentiment Analysis
For Sentiment Analysis, we used a package de-
veloped by the Stanford NLP Group1. The back-
bone of the sentiment analysis package is the
Stanford Sentiment Treebank and Recursive Neu-
ral Tensor Network.
The Treebank is composed of hundreds of thou-
sands of phases labeled by human judges with cer-
tain levels of positivity or negativity (Socher,
2013). The Stanford parser generated these
1 2
See http://nlp.stanford.edu/software/corenlp.shtml for more
info about the Stanford CoreNLP package
phrases, which is an algorithm that identifies the
groups of words that make up phrases in a line of
text. These phrases can then be organized in a tree
that shows the relations between different parts of
a sentence or paragraph (The Stanford NLP
Group).
The Recursive Neural Tensor Network is used
to predict the sentiment of a phrase by recursively
iterating through the nodes of the phrase tree, and
computing the output of a composition function for
the two children of a node (Socher, 2013). In this
way, the network integrates the features of individ-
ual phrases on increasing scales to predict the
overall sentiment at the top of the tree.
3 Work Completed
The first step completed in this project was data
collection. Copyright laws make it difficult to col-
lect data about songs, including their lyrics. We
wanted to find a database that had this information
readily accessible. The first database we looked
into was one entitled The Million Song Dataset.
This was a database created by some faculty mem-
bers at Columbia University to make more of this
data available to researchers in academia. They
proposed a bag-of-words approach to storing lyric
information. They generated a set of lemmas of
common words then determined the frequency of
these words in a song. In addition, this dataset in-
cluded other important features of songs including
duration. While we were working on the proposal,
this seemed like a great resource. However, we
realized that this database would only be helpful
for a couple of features, and the dataset had actual-
ly been poorly maintained and was missing the
majority of the dataset. We had no choice but to
abandon this database. This was frustrating be-
cause we lost a lot of time trying to see if this data-
base could work for our project.
We turned to LyricWikia for help. This site ac-
tually stores lyrics online for almost every song
imaginable. Some very friendly people on the In-
ternet posted an outline of Javascript necessary to
extract lyrics from this website. The approach was
to first perform an http request of the site to get
URL information of our song of interest. Then, we
used a website parsing package, Jsoup2, to actually
See http://jsoup.org for additional info about the Jsoup pack-
age
extract the lyrics from the web page. This approach
worked very well, and was performed in the Lyr-
icsExtractor class we wrote.
Another challenge in data collection was storing
song names to be analyzed. Unfortunately, Lyr-
icWikia did not have the best interface for making
requests, and we were forced to store songs in in-
dividual text files for individual artists. LyricWikia
does not store songs by genre, only by artist. Thus,
we had to associate each song with its artist. A fur-
ther problem was getting a list of songs for said
artist in a quick fashion. Luckily, the website
www.songfacts.com had a list of songs for the ma-
jority of artists we were interested in. We used the-
se list to generate our text files.
Next, we computed all of the features for a par-
ticular song in java. Every feature was computed
by iterating over strings derived from the original
song text. This was done through basic string oper-
ations in StringParameterExtrac-
tor.java. This class also contained calls to
methods in the Stanford NLP package to compute
the sentiment. Here is a more detailed description
of the features we extracted.
Type/Token Ratio This is the ratio of
unique words to the total number of words
in the song
Unique words per line
Mean word length Average number of
characters in a word
Words per line
Sentiment Percentages This was a 3-
dimensional vector with each index repre-
senting the percentage of lines that are
negative, neutral, and positive in the song
Total number of words in the song (Basic
indicator of song duration)
We implemented the K-means with initializa-
tion algorithm described in section 2. This was a
good choice. Because of this initialization step, our
algorithm converges within 10 iterations, check to
be sure. However, we learned that getting K-
means to work was the easy part. Understanding
the output and deciding how to run the algorithm
on our data was another challenge entirely.
We wrote the kmeans.java to perform all
operations associated with K-means on input data.
Then, we constructed a kmeans object in the
GenreClustering.java to actually run K-
means on our data. We attempted to cluster our
data in many different ways. Unfortunately, none
were very successful. Early on in the project, we
did not include the total number of words feature,
so we were working with 7 features instead of 8.
To get our training set, we chose two artists from
each genre in our collection of artist text files. Our
genres were Classic Rock, Country, Emo, Grunge,
Modern Rock, Pop, R&B, and Rap. For example,
we chose Nirvana and Soundgarden for Grunge
and The Beatles and Led Zeppelin for Classic
Rock. These files contained between 50 and 100
songs, and we combined all of the feature vectors
from all of the songs into one giant ArrayList of
ArrayLists in Java. We then ran K-means over this
data, assuming 8 clusters.
We immediately began to notice problems when
we used this approach. We were getting horrible
training accuracy. In fact, this plagued us through-
out the project. We found this accuracy by deter-
mining which cluster center each of our input
songs was assigned to. We had hoped that a major-
ity of the songs would share similar cluster centers,
but they were all over the board. In most cases,
there were 3 to 4 cluster centers included in each
genres training data. The one exception was
Eminem and some of the rap artists. They were
consistently isolated. We believe this was due to
their high words per line feature.
However, we quite enjoyed seeing that country
songs were included with rap songs because they
too had high words per line. We decided that we
wanted to try a different approach to clustering that
would hopefully lead to better results.
We noticed that the variance of the features be-
tween songs for one artist was very high. For ex-
ample, see Table 1 in Appendix A for statistics on
a subset of Beyonces songs. There is significant
variability between songs, and we thought this was
leading to some of the problems with our cluster-
ing. Artists like Beyonce are diverse and write
about a variety of material. Our proposed solution
was to take the average feature vector of 20 songs
of a particular artist, collect 7 to 10 of these aver-
aged vectors for artists in a particular genre, then
cluster the resulting vectors. Our argument for this
approach is that artists on a whole can be consid-
ered to be a part of a genre, so we should consider
an averaged sample of their work.
Unfortunately, this approach still had some is-
sues. Rappers and Country artists were still clus-
tered together, and our training vectors within a
predicted genre still assigned to a variety of differ-
ent clusters. However, it was better than the first
go-around of K-means. Because the variance in our
initial predicted clusters has decreased, K-means
doesnt have to deal with as noisy of data while it
runs. See Table 2 for a summary of statistics in the
Pop training data.
The next step we took proved to be a great ex-
ample of what not to do with K-means. We were
hoping to add one more feature to our current 7.
We decided to try out total number of words. This
was hopefully going to be a good indicator of song
length. When we added this feature, we started to
get very consistent results for our training data.
Thus, we decided to actually try out some test cas-
es. However, we soon realized that there was a
problem. When we started to remove features to
determine which ones have more importance, the
output cluster of K-means did not change until we
removed the total words feature. In the rush to fin-
ish the project, we had forgotten the fundamentals
of Euclidean distance.
Insofar, we had not normalized any of the fea-
tures. However, the total number of words was on
the scale of hundreds while every other feature was
either less than one or between 0 and 10. As a re-
sult, by not normalizing the total number of words,
we placed a large bias on that feature. While this
led to clustering that was not terrible, it was effec-
tively clustering along a number line. This was not
our original goal.
To compensate for this, we normalized the data
using rescaling and reran K-means. This produced
the results described below.
4 Results
One of the first tests we completed was to check
that our K-means algorithm was initiated properly
and working. To do so, we created some artificial
data. These vectors were in 3-space and were clus-
ters of 20 points surrounding (0,0,0), (10, 10, 10)
and (20, 20, 20). We added a small amount of ran-
dom noise to the points to finish our preparation.
When we ran our method over this data, it con-
verged in two iterations, and the cluster centroids
were accurate. The three cluster centroids returned
were (0.5296, 0.4936, 0.5499), (10.5488,
10.52336, 10.6080), and (20.4972, 20.4777
20.6394). With these results, we were confident
that our method was working and decided to try it
out on our lyrics data.
To complete testing of our K-means algorithm,
we focused on the results produced by the last
round of data where we rescaled the data between
0 and 1. We first trained a K-means object with the
centroids for the number of clusters that were in-
terested in. After running K-means with a wide
variety of number of clusters and seeing the re-
sults, we settled on 10 clusters. Many artists or
songs cannot be defined specifically as just one of
the genres we included in our project. Its highly
likely that there will be songs that share features
with multiple genres. Having additional clusters
will take this possibility into account.
Even with rescaling, there was still a lot of vari-
ance in the assignment of training samples within a
particular genre. We believe that rescaling brought
the points too close to one another and finding 10
clusters among nearby points is difficult. Our test-
ing protocol was as follows:
We found which two clusters were most
frequently assigned to training samples in
a particular genre
Find the cluster assignments for songs in
test files for each genre
Determine the percentage of these test
songs labeled with either cluster
After running once with all features given
to K-means, remove 1 feature and repeat
the above steps again.
This final step allowed us to get a rough idea of
the importance of different features for the cluster-
ing of a specific genre. We chose to not mix multi-
ple genres together in the test file for a couple
reasons. First, labeling every song would have
been a challenge for us. Second, we designed our
K-means to account for the fact that a genre might
have songs that are included in more than one clus-
ter. Third, we were more interested in seeing how
well new samples in what we consider the same
genre agree with the training samples. Although
we did not use the traditional method to determine
the effectiveness of our algorithm, we still uncov-
ered some interesting results. See Table 3 below
for a summary of the accuracies we observed for
various genres and exclusions of features.
 All Features Feature Removed
Included Type/Token Unique Mean Words % Negative % Neutral % Positive Total #
WPL WL Per Line Words
Classic 13.33 26.67 12 12 14.67 12 21.33 21.33 17.33
Rock
Country 11.27 32.39 12.68 23.94 11.27 12.68 12.68 35.21 14.08
Emo 9.84 26.23 24.59 16.39 22.95 8.2 18.03 13.11 8.2
Grunge 15.48 28.57 25 42.86 35.71 40.48 23.81 26.19 13.1
Modern 25.76 19.7 19.7 27.27 21.21 7.58 25.76 12.12 13.64
Rock
Pop 11.94 8.96 13.43 28.36 29.85 22.39 17.91 16.42 23.88
R&B 19.15 12.77 21.28 21.28 21.28 17.02 14.89 21.28 4.26
Rap 16.07 23.21 58.93 23.21 17.86 23.21 60.71 21.43 25

Table 3: Percent inclusion for test songs in each genre of interest in all test cases

the length of the longest 10 words in each song.

Although mean word length seemed to be a

Neutral sentiment seems to be a generally bad fea-
ture for genre clustering. For every genre except
R&B, there was a relatively significant increase in
accuracy when the neutral sentiment feature was
removed. Based upon further inspection of the
cluster means, this makes sense because there is
relatively little variance in neutral sentiment be-
tween genre type. This behavior is also explained
by the methods used by Stanford to develop the
package. In their article, they noticed that for
strings of shorter length, the majority of the super-
vised labels that judges gave were neutral (Socher,
2013). It is difficult to determine the sentiment of
only a couple of words. For the majority of genres
tested, the average number of words per line is be-
tween 5 and 6 words. On the other hand, some
songs such as rap songs have close to 10 words per
line. This means that there is more variance in the
potential positive or negative labels for the lines in
a rap song than others. This is probably the reason
that both country and rap have the lowest percent-
age of neutral lines. We could have made this a
more global trend by determining the sentiment of
stanzas instead of individual lines. This would
have further improved the sentiment features as an
informative feature.
Mean word length also proved to be a feature,
which only seemed to add noise. While we thought
it was plausible that different genres would use
varying levels of vocabulary, mean word length is
simply not a good measure of this. Because of the
abundance of words like the, as, and, etc.
this feature did not vary significantly between dif-
ferent songs. A more intelligently designed feature
that would more accurately reflect the difference in
vocabulary between genres would be to average
universally useless feature, there were many fea-
tures that were highly informative for certain gen-
res but not at all in others. For example, negative
sentiment between grunge and modern rock gener-
ated different results. When negative sentiment
was removed for grunge, accuracy was improved
from 15% to 40%. In Modern rock, when negative
sentiment was removed, accuracy was decreased
from 25% to 7.58%, a dramatic decrease. The fact
that certain features improved the accuracy for cer-
tain genres but reduced the accuracy in other gen-
res may indicate that binary or ternary
classification would be a better starting point for
genre classification. One way tod o this would be
to run an SVM or logistic regression and find a
separating hyper-plane between songs in and not in
a particular genre. Of course, this would require
labeling our training examples.
There are some pretty interesting trends in the
accuracy data when you focus on the changes in
accuracy as sentiment parameters are removed. For
example, the accuracy of Rap increases up to 60%.
This change can be understood in the context of
the average sentiment values for Rap compared to
the other genres. See Table 4 for values. Rap has
the largest gap between negative and positive per-
centages at 21. This means that when the neutral
feature is taken away, feature vectors in this form
stand out. Another case that is interesting is when
the negative sentiment is removed. The percent
inclusion for Emo and Modern decrease a lot while
Grunge increases to 40%. As shown in Table 4,
these three genres are very similar when in regard
to neutral and positive sentiment. It is highly pos-
sible that the clustering algorithm found a centroid
 that is unable to discern between Grunge and these As suggested in our proposal, one way that we
other two. could dramatically improve the output of our clus-
ter is to couple K-means with an Expectation-
Maximization with Gaussian Mixtures. In K-
%Negative %Neutral %Positive
Classic
Means we make the implicit assumption that the
Rock 0.28 0.54 0.18 clusters are spherically shaped. Of course this is a
Country 0.29 0.49 0.22 naive assumption. In 2-D space, there might be two
Emo 0.31 0.53 0.16 very thin oval-shaped clusters oriented in such a
Grunge 0.23 0.59 0.18 way that the outer edges of one cluster will be sig-
Modern nificantly closer to the mean of the adjacent clus-
Rock 0.26 0.57 0.17 ter. By applying E-M to a GMM, or another
Pop 0.3 0.5 0.2 probability distribution, one can find clusters of
R&B 0.27 0.51 0.22 varying shapes and distributions.
Rap 0.37 0.48 0.16 One of the last goals in our proposal was to run
K-Nearest Neighbors and compare it to K-means.
Table 4: Mean Sentiment percentage values At that time, we believed that using an unsuper-
for each genre of interest vised approach might work better. After the chal-
lenges we had with K-means, we are convinced
Sentiment plays a heightened role in the now that using a supervised approach to genre identifi-
rescaled version of the feature vectors because eve- cation would be better. In the Mayer article, they
ry feature is now found between 0 and 1. Before, did use both K-Nearest Neighbors as well as SVMs
many of the other features were much larger than to attempt to classify songs by genre. However,
sentiment, and this meant that sentiment did not even with supervised learning and sometimes thou
have as much weight in the Euclidean distance cal- sands of features, their test accuracy only occa-
culations. sionally broke 25%. In our proposal, we said we
were hoping for 70% test accuracy. This was based
5 Proposal Commentary on experience that we had in class instead of evi-
dence demonstrated in papers on the subject.
The goals and milestones described in our pro-
posal ended up being a little too lofty for us to References
achieve. However, we are still proud of our efforts.
We were able to complete our first four milestones, Antonio Castro and Brian Lindauer. 2012. Author Iden-
which included: tification on Twitter. Retrieved from
http://cs229.stanford.edu/proj2012/CastroLindauer-
Extract the data we needed from the internet
AuthorIdentificationOnTwitter.pdf
Choose features based on the Mayer article Rudolf Mayer, Robert Neumayer, and Andreas Rauber.
Use ArrayLists of ArrayLists as our data Rhyme and Style Features for Musical Genre Classi-
structure of choice fication by Song Lyrics. 2008. Retrieved from
Implement and test our K-means algorithm http://www.ismir2008.ismir.net/papers/ISMIR2008_
Analyze our results in such a way that we 235.pdf
have clear next steps if we were to continue Chris Piech. 2013. K Means. Retrieved from
http://stanford.edu/~cpiech/cs221/handouts/kmeans.h
the project.
tml
However, there were a few bumps along the way Richard Socher, et al. 2013. Recursive Deep Models for
that inhibited us. First and foremost was data col- Semantic Compositionality Over a Sentiment Tree-
lection. Because we were building our data from bank. Retrieved from
scratch, this process took a long time. In addition, http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN
every time that we wanted to make an adjustment .pdf
in our features, we had to re-compute features over The Stanford NLP Group. The Stanford Parser: A sta-
every song. Doing a project where the data has tistical parser. Retrieved from
already been accumulated would have definitely http://nlp.stanford.edu/software/lex-parser.shtml
allowed us to focus more time on improving and
learning more about our algorithm.
Appendix A: Feature Data

Words
Unique Mean Per % nega- Total #
Type/Token WPL WL Line tive %Neutral %Positive Words

Mean 0.31 2 3.7 6.81 0.35 0.44 0.22 393.2

Variance 0.065 0.519 0.17 1.777 0.162 0.158 0.139 178.276

Min 0.18 1.23 3.44 3.48 0.13 0.19 0.06 161

Max 0.4 3.16 4.06 9.66 0.72 0.76 0.47 835

Range 0.22 1.93 0.62 6.18 0.59 0.57 0.41 674

Table 1: Statistics on Sample of Beyonce Feature vectors, note variance and range

Words
Unique Mean Per % nega- Total #
Type/Token WPL WL Line tive %Neutral %Positive Words

Mean 0.32 1.92 3.72 6.06 0.3 0.5 0.2 356.45

Variance 0.027 0.243 0.098 0.372 0.027 0.056 0.038 39.66

Min 0.28 1.67 3.59 5.52 0.25 0.38 0.16 297.22

Max 0.37 2.34 3.9 6.58 0.34 0.55 0.28 409.35

Range 0.09 0.67 0.31 1.06 0.09 0.17 0.12 112.13

Table 2: Statistics on collection of averaged Pop artist feature vectors