Uncertainty in emotion recognition
| Published date | 12 August 2019 |
| DOI | https://doi.org/10.1108/JICES-03-2019-0034 |
| Date | 12 August 2019 |
| Pages | 273-291 |
| Author | Agnieszka Landowska |
| Subject Matter | Information & knowledge management,Information management & governance,Information & communications technology |
Uncertainty in emotion recognition
Agnieszka Landowska
Faculty of Electronics, Telecommunications and Informatics,
Gdansk University of Technology, Gdansk, Poland
Abstract
Purpose –The purpose of thispaper is to explore uncertainty inherent in emotionrecognition technologies
and the consequencesresulting from that phenomenon.
Design/methodology/approach –The paper is a general overviewof the concept; however, it is based
on a meta-analysisof multiple experimentaland observational studiesperformed over the past coupleof years.
Findings –The main finding of the papermight be summarized as follows: there is uncertaintyinherent in
emotion recognition technologies,and the phenomenon is not expressed enough, not addressed enough and
unknown bythe users of the technology.
Practical implications –Practical implications of the study are formulated as postulates for the
developers,users and researchers dealing with the technologiesof automatic emotion recognition.
Social implications –As technologies that recognize emotions are becoming more and more common,
and perhaps more decisions influencingpeople lives are to come in the next decades, the trustworthiness of
the technologyis important from a scientific, practical and ethical point of view.
Originality/value –Studying uncertainty of emotionrecognition technologies is a novel approach and is
not exploredfrom such a broad perspectivebefore.
Keywords Uncertainty, Confidence, Accuracy, Affective computing, Reliability, Sentiment analysis,
Trustworthiness, Emotion recognition
Paper type General review
Introduction
One of the modern approaches to capturing the human experience is the automatic
recognition and processing of human emotions, called affective computing. The term was
proposed by Rosalind Picard in 1995 as “computing that relates to, arises from, and
influences emotion”(Picard,1995).
The majority of research on affective computing is focused on the construction of
algorithmsthat recognize emotionson the basis of single- or multi-modalhuman observation.
The symptoms of emotions on which the recognition algorithms are based include facial
expressions (observable in the video channel), body posture (identified from the video
channel or by the motion and pressure sensors), behavioral patterns (e.g. the patterns of
keyboard and mouse usage), sentiment analysis of text, physiological characteristics
(observable by means of sensors or a thermalimaging cameras) and speech prosody(sound
channel) (Kolakowska et al.,2013). All these observation channels are characterized by
certain susceptibility to manipulation and disturbances, and thelevel of these disturbancesis
additionallydependent on the context and the task beingperformed, as well as on individual
characteristics of the person they concern. During multiple experiments performed over the
This work was supported in part by the Polish-Norwegian Financial Mechanism Small Grant Scheme
under Contract Pol-Nor/209260/108/2015 and in part by the DS Funds of ETI Faculty, Gda
nsk
University of Technology.
Emotion
recognition
273
Received24 March 2019
Revised24 March 2019
Accepted27 March 2019
Journalof Information,
Communicationand Ethics in
Society
Vol.17 No. 3, 2019
pp. 273-291
© Emerald Publishing Limited
1477-996X
DOI 10.1108/JICES-03-2019-0034
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/1477-996X.htm
past years, diverse factors for false identification ofemotions were observed, and this paper
summarizesour lessons learned fromthe perspective of uncertaintythey bring.
The uncertainty of measuringinput data has a significant impact on the effectiveness of
the algorithms that are constructed based on these data. Similarly, the uncertainty
associated with the measurement of specific symptoms of emotions, as well as the
inaccuracy of individual classifiers, may result in limited credibility of the methods that
recognize human emotions(Brodny, Landowska, 2018).
Affective computing researchers are well familiar with the limitations of the domain. The
methods provided so far are, as all artificial intelligence algorithms, susceptible to noise,
mislabeled data, changing contextual circumstances. Emotional expressions, which the
algorithms are basing on, are highly individual and even might change depending on a mood.
In-the-wild conditions make the results even less reliable. All mentioned factors lead to
uncertainty in the analysis of human affect. But still, most research studies in affective
computing concentrate on providing accuracies rather than expressing the confidence of the
results. Commercially available emotion recognition solutions provide limited or no information
on the confidence of the result so far. The quantification and characterization of the resulting
output uncertainty is an important matter when results are used to guide decision-making.
Simple and everyday use of emotion recognition starts with automatic smile detection
algorithms, applied in digital cameras and mobile devices. Moreover, products, films and
advertisements are evaluated using these affect recognition techniques, usually involving
large user groups and statistical analysis. However, the methods are also applied in areas
where decisions regarding individuals are made. Examples include the use of algorithms
that recognize stress and unusual behavior at airports for anti-terrorist purposes or the
analysis of facial expressions during a remote job interview, both being already the case.
Because technologies that recognize emotions are becoming more and more common, the
research on accuracy and trustworthiness of the methods is important from a scientific,
practical and ethical point of view. Perhaps more decisions influencing people lives are to
come in the next decades. Can people trust affective computing? Are people aware of the
uncertainty relatedto automatic emotion recognition?
Commercial software currently available in the market, e.g. recognizing emotions based
on facial expression analysis, providesan estimate of an emotional state (if only the face is
detected), usually without providing any information about the degree of confidence upon
given hypothesis (Brodny et al.,2016). During past observational studies, sometimes it
happened that the facial expression analysis software returned the emotion recognized on
the room equipment elements or detected frames of the glassesas eyebrows, which affected
the results of the analysis.
The purpose of this paper is to explore the uncertainty sources of emotion recognition
and to provide some postulates regarding uncertainty representation. The paper does not
explore uncertaintyframeworks in detail and does not provide a solution, butrather aims at
gaining attentionto the problem of trustworthy emotion recognition technologies.
Background
Uncertainty can be describedas a state of the analyst that cannot foresee a phenomenon due
to intrinsic variabilityof the phenomenon itself, or to lack of knowledge and information.
Every measurement is associated with uncertainty (JCGM, 2008). The reliability of
measurement depends on multiple factors. The reliability can be influenced by choosing
appropriate methods and tools, but it is not possible to completely eliminate uncertainty.
Automatic recognition ofemotions is also a kind of measurement procedure; however, what
is worth emphasizing, it measures a complex, internal phenomenon, which is human
JICES
17,3
274
To continue reading
Request your trial