Text Data Mining with Twitter and R

Twitter is a favorite source of text data for analysis: it’s popular (there is a huge volume of variety on all topics) and easily accessible using Twitter’s free, open APIs which are easily consumable in JSON and ATOM formats.

Some people have used Twitter for sophisticated analysis such as predicting flu outbreaks and the stock market, but let’s start with something simpler and less ambitious: an introduction to text data mining using Twitter and R. We’ll download live data using the Twitter APIs, parse it, build a corpus, demonstrate some basic text processing, and plot a hierarchical agglomerative cluster—because everyone likes pictures. I query for a controversial topic, abortion, in hopes of visualizing the two sides of the debate.

There is a specialized package for R called twitteR, but it isn’t available for Windows, but it’s easy to substitute the generic XML package and the Twitter search API documentation for our needs.

###
### Read tweets from Twitter using ATOM (XML) format
###

# installation is required only required once and is rememberd across sessions
install.packages('XML') 

# loading the package is required once each session
require(XML)

# initialize a storage variable for Twitter tweets
mydata.vectors <- character(0)

# paginate to get more tweets
for (page in c(1:15))
{
	# search parameter
	twitter_q <- URLencode('#prolife OR #prochoice')
	# construct a URL
	twitter_url = paste('http://search.twitter.com/search.atom?q=',twitter_q,'&rpp=100&page=', page, sep='')
	# fetch remote URL and parse
	mydata.xml <- xmlParseDoc(twitter_url, asText=F)
	# extract the titles
	mydata.vector <- xpathSApply(mydata.xml, '//s:entry/s:title', xmlValue, namespaces =c('s'='http://www.w3.org/2005/Atom'))
	# aggregate new tweets with previous tweets
	mydata.vectors <- c(mydata.vector, mydata.vectors)
}

# how many tweets did we get?
length(mydata.vectors)

Based on the limits of the Twitter API, you should now have 1500 vectors representing 1500 tweets.

###
### Use tm (text mining) package
###

install.packages('tm')
require(tm)

# build a corpus
mydata.corpus <- Corpus(VectorSource(mydata.vectors))

# make each letter lowercase
mydata.corpus <- tm_map(mydata.corpus, tolower) 

# remove punctuation 
mydata.corpus <- tm_map(mydata.corpus, removePunctuation)

# remove generic and custom stopwords
my_stopwords <- c(stopwords('english'), 'prolife', 'prochoice')
mydata.corpus <- tm_map(mydata.corpus, removeWords, my_stopwords)

# build a term-document matrix
mydata.dtm <- TermDocumentMatrix(mydata.corpus)

# inspect the document-term matrix
mydata.dtm

# inspect most popular words
findFreqTerms(mydata.dtm, lowfreq=30)

The most popular terms include abortion, dont, funding, gop, parenthood, planned, prochoice prolife, tcot, women. Though not explicitly stated here, the words “planned” and “parenthood” are a collocation (words which occur together more often than by chance). Let’s see which words are associated with a term.

> findAssocs(mydata.dtm, 'fetus', 0.20) 
         fetus          child            247     believeing    brainwashed          cared 
          1.00           0.31           0.30           0.30           0.30           0.30 

The number under each word is an association score, so the search term always occurs with the search term. The next most-associated term is “child,” etc. In some applications, a stemmer or spell checker could help with the misspelled word “believeing.”

To make a Hierarchical Agglomerative cluster plot, we need to reduce the number of terms (which otherwise wouldn’t fit on a page or the screen) and build a data frame.

# remove sparse terms to simplify the cluster plot
# Note: tweak the sparse parameter to determine the number of words.
# About 10-30 words is good.
mydata.dtm2 <- removeSparseTerms(mydata.dtm, sparse=0.95)

# convert the sparse term-document matrix to a standard data frame
mydata.df <- as.data.frame(inspect(mydata.dtm2))

# inspect dimensions of the data frame
nrow(mydata.df)
ncol(mydata.df)

Now the data frame (a standard data structure in R) contains a bag of words (specifically, 1-grams) which are simple frequency counts. Though the structure is lost, it retains much information and is simple to use. The data frame is ready for cluster analysis using a cluster analysis function available in R core. The following code is basically copied from Robert I. Kabacof’s “Cluster Analysis” page.

mydata.df.scale <- scale(mydata.df)
d <- dist(mydata.df.scale, method = "euclidean") # distance matrix
fit <- hclust(d, method="ward")
plot(fit) # display dendogram?

groups <- cutree(fit, k=5) # cut tree into 5 clusters
# draw dendogram with red borders around the 5 clusters
rect.hclust(fit, k=5, border="red") 

Adjust the quantity five to best fit the data, and now we have the plot:

The terms higher in the plot are more popular, and terms close to each other are more associated. For example, today there is a fear the US government will shut down, so the terms “budget,” “funding,” and “shutdown” appear together, but these are not associated with the term “woman. The term “periodpiece” is a Twitter account (remember, punctuation was removed including @ which designates accounts), and the cluster with “periodpiece” and “life” is a semantic argument (example).

Some possible next steps include:

• K-means cluster
• Remove hyperlinks from tweets
• Basic word association plots (built in to the tm package but requires Rgraphviz which can be tricky to install)
• Word association fans
• Sentiment analysis: which hashtag has more positive mood?
• Classification: to which side of the debate does a new tweet (without a hashtag) belong?
• Find a happier topic