A comparative study of the effectiveness of sentiment tools and human coding in sarcasm detection
| Date | 13 August 2018 |
| DOI | https://doi.org/10.1108/JSIT-12-2017-0120 |
| Published date | 13 August 2018 |
| Pages | 358-374 |
| Author | Phoey Lee Teh,Pei Boon Ooi,Nee Nee Chan,Yee Kang Chuah |
| Subject Matter | Information & knowledge management,Information systems,Information & communications technology |
A comparative study of
the effectiveness of sentiment
tools and human coding
in sarcasm detection
Phoey Lee Teh
Department of Computing and Information Systems,
Sunway University, Bandar Sunway, Malaysia
Pei Boon Ooi
School of Healthcare and Medical Sciences, Sunway University,
Bandar Sunway, Malaysia, and
Nee Nee Chan and Yee Kang Chuah
Sunway University, Bandar Sunway, Malaysia
Abstract
Purpose –Sarcasm is often used in everyday speech and writing and is prevalent in online contexts. The purpose
of this paper is to investigate the analogy between sarcasm comments from sentiment tools and the human coder.
Design/methodology/approach –Using the VerbalIrony Procedure, eight human coders were engaged
to analyse comments collected from an online commercialpage, and a dissimilarity analysis was conducted
with sentiment tools. Threeconstants were tested, namely, polarity from sentimenttools, polarity rating by
human coders;and sarcasm-level ratings by human coders.
Findings –Results found an inconsistentratio between these three constants. Sentiment toolsused did not
have the capability or reliability to detect the subtle, contextualized meanings of sarcasm statements that
human coders could detect. Further research is required to refine the sentiment tools to enhance their
sensitivityand capability.
Practical implications –With these findings, it is recommended that further research and
commercialization efforts be directed at improving current sentiment tools –for example, to incorporate
sophisticated human sarcasm texts in theiranalytical systems. Sarcasm exists frequently in media, politics
and human forms of communications in society.Therefore, more highly sophisticated sentiment tools with
the abilitiesto detect human sarcasm would be vital in research and industry.
Social implications –The findings suggest that presently, of the sentimenttools investigated, most are
still unable to pick up subtle contextswithin the text which can reverse or change the message that the writer
intends to send to his/her receiver. Hence, the use of the relevant hashtags (e.g. #sarcasm; #irony) are of
fundamental importance in detection tools. This would aid the evaluation of product reviews online for
commercialusage.
Originality/value –The value of this studylies in its original, empirical findings on the inconsistencies
between sentiment tools and human coders in sarcasm detection. The current study proves these
inconsistencies are detected between human and sentiment tools in social media texts and points to the
inadequacies of current sentiment tools. With these findings, it is recommended that further research and
commercialization efforts be directed at improving current sentiment tools –to incorporate sophisticated
human sarcasmtexts in their analyticalsystems. The system canthen be used as a reference for psychologists,
mediaanalysts, researchers and speechwriters to detect cues in the inconsistenciesin behaviourand language.
Keywords Detection, Social media, Linguistic, Sarcasm, Verbal irony
Paper type Research paper
JSIT
20,3
358
Received6 December 2017
Revised26 March 2018
4 June2018
Accepted3 September 2018
Journalof Systems and
InformationTechnology
Vol.20 No. 3, 2018
pp. 358-374
© Emerald Publishing Limited
1328-7265
DOI 10.1108/JSIT-12-2017-0120
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/1328-7265.htm
1. Introduction
Sarcasm is derived from the Latin word that means “to tear flesh”, and it has been called
“hostility disguised as humour”and is usually used by some people to make fun of others
(Pazzanese, 2015). It has been defined in dictionaries as a low form of wit intended to insult
someone (Merriam-Webster, 2017). Sarcasm is often confused with irony. Dynel (2014) defines
irony as a way of using words that are opposite in meaning. Sarcasm uses irony, i.e. the words
that are opposite in meaning to be unpleasant or to wound someone. Being sarcastic, hence, is
sometimes a cultural trait in some societies that support the intention to tease, humour or joke
with words (Reyes et al., 2012). Pazzanese (2015) argues that sarcasm is the lingua franca of the
internet, as it is so prevalent among internet users who use it to show contempt or derision.
There is a lack of agreement among researchers (computer scientists, linguists and
psychologists) on the formal definition of sarcasm and its structure. Among the many
theories proposed to explain thisphenomenon, there is an agreement on the impossibility of
a formal definition as the term is dynamic, as it has experienced changes and has regional
variations (Filatova, 2012). To produce or interpret sarcasm, both the expressers and
recipients of sarcasm must surmount the conflict (i.e. psychological distance) between the
literal and actual meanings of the sarcastic expressions (Huanget al.,2015). If the decoding
of sarcasm is difficult for expressersand recipients, it is even more difficult for computers to
detect because it does not functionin the same way that literal language does. Thus, opinion
mining and sentiment analysis systems can improve their performance given the correct
identification of sarcastic utterances in their appropriate contexts (Filatova, 2012;Hu and
Liu, 2004;Pang and Lee, 2008). Hence, various cases of sarcastic texts expressions can be
comprehended onlywhen situated within a certain context or within a broadertext setting.
A review of the extant literature (Burgers et al.,2011;Joshi et al.,2012;Rajadesingan
et al., 2015;Schifanellaet al., 2016;Teh et al.,2016;Agrawal and Awekar, 2018) showsa gap:
to date there is no comparative study of the effectiveness of sentiment tools and human
coding in sarcasm detection. This study aims to fill this gap with the following three
research objectives:
(1) to investigate the similarity of sentiment analysis between sentiment tools and
human coders in sarcasm texts;
(2) to review sarcastic comments and to identify the respective polarity and sarcasm
level; and
(3) to compare the effectiveness of sentiment tools and human coding in sarcasm
detection.
The theoretical underpinning of this study is based mainly on Burgers, van Mulken and
Schellens’s(2011)theoretical framework (see Table I) which has identified sarcasm under
four markers: Tropes, Schematic, Morphosyntactic and Typographic irony markers. This
study is also referred to other theoretical studies on sarcasm identification,such as from the
Table I.
Sarcasm markers
Markers Tropes Schematic Morphosyntactic Typographic
Categories Metaphor
Hyperbole
Understatement
Rhetorical
Ironic repetition
Ironic echo
Change of register
Exclamation
Tag question
Focus topicalization
Interjections
Diminutives
Capitalization
Quotation marks
Punctuation marks
Emoticons
Sentiment
tools and
human coding
359
To continue reading
Request your trial