Does attention mechanism possess the feature of human reading? A perspective of sentiment classification task
| DOI | https://doi.org/10.1108/AJIM-12-2021-0385 |
| Published date | 09 May 2022 |
| Date | 09 May 2022 |
| Pages | 20-43 |
| Subject Matter | Library & information science,Information behaviour & retrieval,Information & knowledge management,Information management & governance,Information management |
| Author | Lei Zhao,Yingyi Zhang,Chengzhi Zhang |
Does attention mechanism possess
the feature of human reading?
A perspective of sentiment
classification task
Lei Zhao, Yingyi Zhang and Chengzhi Zhang
School of Economics and Management,
Nanjing University of Science and Technology, Nanjing, China
Abstract
Purpose –To understand the meaning of a sentence, humans can focus on important words in the sentence,
which reflects our eyes staying on each word in different gaze time or times. Thus, some studies utilize
eye-tracking values to optimize the attention mechanism in deep learning models. But these studies lack to
explain the rationality of this approach. Whether the attention mechanism possesses this feature of human
reading needs to be explored.
Design/methodology/approach –The authors conducted experiments on a sentiment classification task.
Firstly, they obtained eye-tracking values from two open-source eye-tracking corpora to describe the feature of
human reading. Then, the machine attention values of each sentence were learned from a sentiment
classification model. Finally, a comparison was conducted to analyze machine attention values and
eye-tracking values.
Findings –Through experiments, the authors found the attention mechanism can focus on important words,
such as adjectives, adverbs and sentiment words, which are valuable for judging the sentiment of sentences on
the sentiment classification task. It possesses the feature of human reading, focusing on important words in
sentences when reading. Due to the insufficient learning of the attention mechanism, some words are wrongly
focused. The eye-tracking values can help the attention mechanism correct this error and improve the model
performance.
Originality/value –Our research not only provides a reasonable explanation for the study of using
eye-tracking values to optimize the attention mechanism but also provides new inspiration for the
interpretability of attention mechanism.
Keywords Eye-tracking values, Attention mechanism, Attention, Human reading
Paper type Research paper
1. Introduction
The rise of artificial intelligence (AI) has profoundly changed the way humans understand
the world. Machines can make a series of responses similar to humans, becoming more
intelligent and even far superior to humans. The AlphaGo defeating the world champion of
Go is the best example. However, in the field of natural language processing (NLP), due to the
complexity and diversity of human languages, machines cannot fully understand human
expressions in some tasks, e.g. machine translation (L€
aubli et al., 2020), summary generation
(Sheela and Janet, 2021), keyphrase extraction (Zhang et al., 2022a) and so on. Whether
machines really possess a human-like way of thinking is worthy of in-depth exploration in
this background.
When we are reading a sentence, we can focus on some words that are useful for
understanding the sentence. In other words, we do not pay the same attention to all words,
which reflects our eyes staying on each word in different gaze time or times. The eye-tracking
AJIM
75,1
20
This work is supported by Open Fund Project of Fujian Provincial Key Laboratory of Information
Processing and Intelligent Control (Minjiang University) (No. MJUKF–IPIC201903).
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/2050-3806.htm
Received 22 December 2021
Revised 4 March 2022
Accepted 23 April 2022
Aslib Journal of Information
Management
Vol. 75 No. 1, 2023
pp. 20-43
© Emerald Publishing Limited
2050-3806
DOI 10.1108/AJIM-12-2021-0385
corpus (ETC) is a record of this feature of human reading. It uses eye-trackers to capture the
time or fixation times of eye gaze on each word (Rayner et al., 2012). ETCs have been applied
to various NLP tasks, e.g. part-of-speech tagging (Barrett et al., 2016), sentiment analysis
(Mishra et al., 2016b), multiword expression (Rohanian et al., 2017), keyphrase extraction
(Zhang and Zhang, 2021), etc. These studies used eye-tracking values to optimize the
attention mechanism, but they lack to explain the rationality of this approach. Therefore, it is
essential to compare the machine attention values obtained by the model based on attention
mechanism and the eye-tracking values, providing theoretical support for these studies.
The study is in the line of interpretability research of the attention mechanism. Previous
research had explored the relationship between machine attention and human attention, in
which human attention is represented by the important degree of words manually annotated.
Generally, the important degree is in a binary distribution, represented by 0–1(Sen et al.,
2020), different from the continuous distribution of machine attention values. Since the binary
distribution can only indicate which words are important in a sentence, but cannot compare
the important degree between words, the analysis based on this cannot fully describe the
relationship between machine attention and human attention. In contrast, eye-tracking
values are a continuous distribution, and a larger eye-tracking value indicates that the word
is more important in a sentence. Figure 1 shows the attention distribution of machine
attention values and eye-tracking values on a movie review randomly selected from ZuCo. In
this example, both attention mechanism and human focus on the word “mesmerizing”in the
sentence, which greatly affects judging the sentiment of this review as positive. This example
shows a certain relationship between the machine attention values and the eye-tracking
values.
The purpose of our study is to analyze whether the attention mechanism possesses the
feature of human reading, focusing on the important words in sentences when reading. We
conducted experiments on a sentiment classification task. The eye-tracking values were used
to describe the feature of human reading, which are collected from two open-source ETCs, i.e.
ZuCo (Hollenstein et al., 2018) and Eye-tracking and Sentiment Analysis-II (Mishra et al.,
2016a). They are sentiment classification task-driven corpora. The machine attention values
for the two ETCs were learned from a sentiment classification model based on the attention
mechanism. To compare the machine attention values and the eye-tracking values, we used
Spearman correlation coefficient (Maritz, 1995), Jensen Shannon divergence (Lin, 1991)to
analyze the correlation between them. Combining the characteristics of the sentiment
classification task, we explored whether the attention mechanism and humans focus on the
same words, and calculated the attention rate of specific part-of-speech words and sentiment
words. Through experiments, we found the attention mechanism can focus on important
words, such as adjectives, adverbs and sentiment words, which are valuable for judging the
sentiment of sentences on the sentiment classification task. It possesses the feature of human
reading, focusing on important words in a sentence when reading. Due to the insufficient
learning of the attention mechanism, some words are wrongly focused. The eye-tracking
Note(s): There are four types of eye-tracking values, i.e., nFix, FFD, TRT, and RRT, introduced in Section 3.1.
The darker the color, the higher attention to the word
Figure 1.
An example of the
attention distribution
of machine attention
values and eye-
tracking values on a
movie review
The feature of
human reading
21
To continue reading
Request your trial