Evaluating hotels rating prediction based on sentiment analysis services

Date20 July 2015
Publication Date20 July 2015
AuthorRutilio Rodolfo López Barbosa,Salvador Sánchez-Alonso,Miguel Angel Sicilia-Urban
SubjectLibrary & information science,Information behaviour & retrieval
Evaluating hotels rating
prediction based on sentiment
analysis services
Rutilio Rodolfo López Barbosa, Salvador Sánchez-Alonso and
Miguel Angel Sicilia-Urban
Department of Computer Science, University of Alcala,
Alcala de Henares, Spain
Purpose The purpose ofthis paper is to assess the reliabilityof numerical ratings of hotelscalculated
by three sentiment analysisalgorithms.
Design/methodology/approach More than one millionreviews and numerical ratings of hotelsin
seven cities in four countries were extracted from TripAdvisor web site. Reviews were classified as
positive ornegative using three sentiment analysistools. The percentage of positivereviews was used to
predict numerical ratings that were then compared with actual ratings.
Findings All tools classifiedreviews as positive or negative in a way that correlated positively with
numerical ratings. More complex algorithms worked better, yet predicted ratings showed reasonable
agreementwith actual ratings for most cities.Predictions for hotels were lessreliable if based on less than
50-60 percent of available reviews.
Practical implications These results validate that sentiment analysis can be used to transform
unstructured qualitative data on user opinion into quantitativeratings. Current tools may be useful for
summarizingopinions of user reviews of products and serviceson web sites that do not require usersto
post numerical ratingssuch as traveler forums. This summarizingmay be valuable not just to potential
users, but also to the serviceand product providers and offers validation and benchmarking for future
improvement of opinion mining and prediction techniques.
Originality/value This work assessesthe correlation between sentiment analysis of hotelsreviews
and their actual ratings. The authors also evaluated the reliability of results of sentiment analysis
calculated by three different algorithms.
Keywords Sentiment analysis, Consumer-generated content, Intra-class correlation,
Opinion mining, TripAdvisor reviews
Paper type Research paper
1. Introduction
Developing an automated method for identifying a customers feelings or attitude toward
people, services or products based on customer-generated content holds tremendous value
for researchers, corporations and customers themselves. This is the goal of sentiment
analysis, which aims to identify customer opinions or attitudes on the basis of their
spoken or written comments. Converting these opinions and attitudes into numbers is a
way to rapidly synthesize and analyze customer experiences, allowing customers t o make
decisions about buying the product or contracting the service, and allowing companies to
make decisions about launching new products or redesigning products and services.
The significant impact of online reviews on electronic word of mouth (eWOM)
communication is now well established in the literature (Trusov et al., 2009; Zhu and
Zhang, 2006). The growing reliance of consumers on online reviews of products
and services has led to such an abundance of user-generated content that no potential
customer can hope to sift through it all. Sentiment analysis tools, which can condense
Aslib Journal of Information
Vol. 67 No. 4, 2015
pp. 392-407
©Emerald Group Publishing Limited
DOI 10.1108/AJIM-01-2015-0004
Received 13 January 2015
Revised 12 May 2015
Accepted 12 May 2015
The current issue and full text archive of this journal is available on Emerald Insight at:
large amounts of text comments into a few easily digestible numbers, can provide a
powerful way to aggregate and summarize the full range of opinions.
The tourism and hospitality sector is an excellent example of an industry in which
the success of products and services increasingly depends on large amounts of user-
generated content posted on social media sites. Sites such as like.com, Booking.com and
HolidayCheck.com allow users not just to rate hotels and their amenities on numerical
scales, but also to write theiropinions or attitudes astext. However, there are manyother
web sites, especially traveler forums, such as travelerspoint.com, lonelyplanet.com and
losviajeros.com that allow users to share their experiences by writing comment without
issuingany numeric rate. Thus,these web sites would improve the user searchby posting
automatically calculated numeric ratings. The text comments are a powerful form of
eWoM and are no less important than the numerical ratings (Gretzel and Yoo, 2008).
Testament to the potential of sentiment analysis for harnessing the power of eWoM
is the fact that dozens of commercial and public tools have been developed for this
purpose. OpinionFinder was developed by teams of researchers at the University of
Pittsburg, Cornell University and the University of Utah comparisons (Wils on et al.,
2005a). This algorithm uses a lexicon to identify sentiment expressions based on
context (Wilson et al., 2005b). The Recursive Neural Tensor Network (RNTN) tool was
developed by researchers at Stanford University; it works by labeling phrases in parse
trees of sentences using a data set called Sentiment Treebank, and it functions as a
sentiment analysis annotator in the Stanford CoreNLP (Socher et al., 2013). CoreNLP
is an integrated suite of natural language processing (NLP) tools for English (Manning
et al., 2014). Our group has developed an unsupervised lexicon-induced sentiment
analysis tool called SentUAH, which uses the tokenizer, sentence splitter and part-of-
speech (POS) tagger from CoreNLP. The tool works in combination with SentiWordNet
(Esuli and Sebastiani, 2006; Baccianella et al., 2010) and a naïve Bayesian approach
to data mining. The aim of developing this tool was not to improve efficiency but to
confront the results of a simple algorithm (naïve Bayes which is well known as efficient
for specific cases) with more complex algorithms and assess the reliability of all of them
to predict numerical ratings with a large amount of data. We are unaware of studies
benchmarking these software tools against a large experimental data set or against one
another. Such a study is important for validating sentiment analysis tools in the field,
and for guiding the future improvement of these programs.
Therefore we compared the three software tools for their potential ability to predict
numerical hotel ratings based on text comments. We examined more than 1 million
reviews of 3,535 hotels in seven cities posted on TripAdvisor.com, this was all the
English comments, more than 75 percent of the total amount. We used sentiment
analysis to classify the comments as positive or negative and thereby generate
predicted ratings from those comments. Then we compared the model-generated
ratings with the actual ratings. This is the first study to our knowledge that assesses
the ability of sentiment analysis to provide a bridge between comments per hotel
(qualitative) and ratings (quantitative) from the same source doing this at the review
(comment) level instead of at the sentence level.
2. Background
2.1 Sentiment analysis
Sentiment analysis is a subfield of NLP that draws on approaches from information
retrieval and computational linguistics to identify opinions expressed in text. It is
considered a specific type of text mining (Han et al., 2011), and it has been called opinion
hotels rating

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT