Paper, Tags: #linguistics, #sarcasm-detection
We present a computational system that harnesses context incongruity as a basis for sarcasm detection. We incorporate two kinds of incongruity features: explicit and implicit, and two text forms: tweets and discussion forum posts. We show that our features can capture interesential incongruity.
Sarcasm is defined as: a cutting, often ironic remark intended to express contempt or ridicule (source: the free dictionary). Sarcasm detection is the task of predicting a text as sarcastic or non-sarcastic.
Past work involves rule-based and statistical approaches using unigrams and pragmatic features, extraction of common patterns such as hashtag-based sentiment, a positive verb being followed by a negative situation, or discriminative n-grams.
None of these are based on linguistic theories, so in this paper we use one such linguistic theory: context incongruity. Our system outperforms two state-of-the-art sarcasm detection systems (Riloff et al., 2013; Maynard and Greenwood, 2014). Our system shows an improvement for short tweets as well as long discussion forum posts.
Campbell and Katz (2012) state that sarcasm occurs along different dimensions, namely failed expectation, pragmatic insincerity, negative tension, presence of a victim and along stylistic components such as emotion words.
Eisterhold et al., 2006, observe that sarcasm can be identified based on the statement preceding and following the sarcastic statement. This happens when the incongruity isn't expressed within the sarcastic text itself.
History of sarcasm detection:
- 2006: sarcasm recognition in speech using average pitch, pitch slope and laughter cues
- 2009: simple linguistic features like interjection, changed names
- 2010: syntactic and pattern-based features
- 2011: study on the role of unigrams and emoticons
- 2013: sarcasm detection based on hashtags
- 2013: rule-based system using phrases of positive verb phrases and negative situations from tweets.
Riloff et al., 2013, state that sarcasm is a contrast between positive sentiment word and a negative situation.
It's defined as the state of being not in agreement, as with principles. Context incongruity is a necessary condition for sarcasm. Ivanko and Pexman (2003) state that the sarcasm processing time depends on the degree of context incongruity between the statement and the context.
We consider two cases of incongruity, explicit and implicit (demands a higher processing time).
It's overtly expressed through sentiment words of both polarities (I love being ignored), where there's a positive and a negative word.
It's overtly expressed through phrases of implied sentiment, as opposed to opposing polar words (I love this paper so much that I made a doggy bag out of it). There's no explicit incongruity, but the second clause has an implied sentiment that's incongruous with the polar word 'love'.
We conduct a naïve, automatic evaluation on a dataset of 18k sarcastic tweets. ~11% of all tweets have 1+ explicit incongruity. We also manually evaluate 50 sarcastic tweets and observe that 10 have explicit incongruity, while others have implicit incongruity.
Our system augments the feature vector of a tweet with features based on the two types of incongruity. We use 4 features:
- lexical: unigrams obtained using feature selection techniques
- pragmatic: emoticons, laughter expressions, punctuation marks, capital letters
- implicit incongruity: related to implicit phrases
- explicit incongruity: numeric, qualitative features
An explicit incongruity giving rise to sarcasm bears resemblance to thwarted expectations, "My tooth hurts! Yay!", the negative word is incongruous with the positive Yay. Our explicit incongruity feature are a relevant subset of features from a past system to detect thwarting:
- number of sentiment incongruities: number of times a positive word is followed by a negative word, and vice versa
- Largest positive/negative sequence: the length of the longest series of contiguous positive/negative words
- Number of positive and negative words
- Lexical polarity: polarity based purely on the basis of lexical features.
We use phrases with implicit sentiment as the implicit incongruity features. The sentences are sentiment bearing verb and noun phrases, the latter being situations with implied sentiment. We extract both polarities, we include them as extra features.
We use three datasets:
- Tweet-A (5208, 4170 sarcastic): we label tweets as sarcastic if they have the hashtags #sarcasm or #sarcastic, and non-sarcastic if they have the hashtags #notsarcasm and #notsarcastic.
- Tweet-B (2278 tweets, 506 sarcastic): manually labeled
- Discussion-A (1502 discussion forum posts, 752 sarcastic): created from the Internet Argument Corpus that contains manual annotations for sarcasm. We randomly select 752 sarcastic and 752 non-sarcastic discussion forum posts.
We run the algorithm by Riloff et al., 2013, on a dataset of 4k tweets, half sarcastic. The algorithm results in 79 verb phrases and 202 noun phrases. We train our classifiers for different feature combinations, using LibSVM with RBF kernel and report average 5-fold cross-validation values.
We compare precision, recall and F1 score results for Riloff et al.'s algorithm and ours. All statistical classifiers surpass the rule-based algorithms. The best F-score, 0.88, obtained when all four kinds of features are used is an improvement of ~5% over the baseline and 40% over the algorithm by Riloff et al. Even in the case of the Discussion dataset, our features result in an improved performance.
- Subjective polarity: sarcasm is subjective
- No incongruity within text
- Incongruity due to numbers: going to work for 2h is totally worth the 45min drive
- Dataset granularity: sentences marked as sarcastic but containing non-sarcastic portions lead to irrelevant features
- Politeness: post all your inside jokes on facebook, I really want to hear about them.