Improving the design of a recommendation system using evaluation criteria and metrics as a guide
| Date | 12 August 2019 |
| DOI | https://doi.org/10.1108/JSIT-01-2019-0019 |
| Pages | 304-324 |
| Published date | 12 August 2019 |
| Author | Adekunle Oluseyi Afolabi,Pekka Toivanen |
| Subject Matter | Information & knowledge management,Information systems,Information & communications technology |
Improving the design of a
recommendation system using
evaluation criteria and
metrics as a guide
Adekunle Oluseyi Afolabi and Pekka Toivanen
School of Computing, University of Eastern Finland, Kuopio, Finland
Abstract
Purpose –The roles recommendation systems play in health care have become crucial in achieving
effective care and in meeting the needs of modern caregiving.Asaresult,effortshavebeengeared
toward using recommendation systems in the management of chronic diseases. Effectiveness of these
systems is determined by evaluation following implementation and before deployment, using certain
metrics and criteria. The purpose of this study is to ascertain whether consideration of criteria during
the design of a recommendation system can increase acceptance and usefulness of the recommendation
system.
Design/methodology/approach –Using survey-style requirements gathering method, the specific
health and technology needsof people living with chronic diseases were gathered. Theresult was analyzed
using quantitative method. Sets of harmonized criteria and metrics were used along with requirements
gathered from stakeholders to establish relationship among the criteria and the requirements. A matching
matrix was used to isolate requirements for prioritization. Theserequirements were used in the design of a
mobile app.
Findings –Matching criteria against requirements highlights three possible matches, namely, exact,
inferential and zero matches. In any of these matches, no requirement was discarded. This allows priority
features of the system to be isolatedand accorded high priority during the design. This study highlightsthe
possibilityof increasing the acceptance rate and usefulnessof a recommendation system by using metrics and
criteria as a guide during the design process of recommendation systems in health care. This approach was
applied in the design of a mobile app calledRecommendations Sharing Community for Aged and Chronically
Ill People. The result has shown that with thismethod, it is possible to increase acceptance rate, robustness
and usefulnessof the product.
Research limitations/implications –Inability to know the evaluationcriteria beforehand, inability to
do functional analysis of requirements, lack of well-defined requirements and often poor cooperation from
people living with chronic diseasesduring requirements gathering for fear of stigmatization, confidentiality
and privacybreaches are possible limitations to this study.
Practical implications –The result has shown that with this method, it is possible to isolate more
important featuresof the system and use them during the design process,thereby speeding up the design and
increasing acceptancerate, robustness and usefulness of the system. It also helps to see in advance the likely
features of the system that will enhance its usefulnessand acceptance, thereby increasing the confidence of
the developers in their ability to deliver a system that will meet users’needs.As a result, developers know
beforehand where to concentrate their efforts during system development to ascertain the possibility of
increasing usefulnessand acceptance rate of a recommendation system.In addition, it will also save time and
cost.
Compliance with Ethical Standards.
Funding: This study was not funded.
Conflict of interest: The authors declare no conflict of interest.
Informed consent: Informed consent was obtained from individuals who participated in this study.
JSIT
21,3
304
Received28 January 2019
Revised5 April 2019
28May 2019
Accepted13 August 2019
Journalof Systems and
InformationTechnology
Vol.21 No. 3, 2019
pp. 304-324
© Emerald Publishing Limited
1328-7265
DOI 10.1108/JSIT-01-2019-0019
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/1328-7265.htm
Originality/value –This paper demonstrates originality by highlighting and testing the possibility of
using evaluation criteria and metricsduring the design of a recommender system with a view to increasing
acceptance and enhancing usefulness. It also shows the possibility of using the metrics and criteria in
system’sdevelopment process for an exercise other thanevaluation.
Keywords Evaluation, Health care, Metrics, Design, Criteria, Recommender systems, ReSCAP
Paper type Research paper
1. Introduction
Recommendation systems in healthcare play crucial roles in modern care giving and in the
management of chronic diseases.As a result, they have become a focus of research in recent
years. Efforts in thisdirection are seen in issues relating to general health-careinterventions,
nutrition and well-being (Ali et al., 2018a,2018b;Arens-Volland et al.,2018).
Recommendation systems have also been applied in physical and mental health (Tene and
Tangedipalli, 2018;Yang et al., 2017). This is an indication that recommendation systems
can be used for effective care delivery to achieve or improve mental health. More
importantly, focus is shifting to practical implementation of some of these systems unlike
what obtained in the past decades (Afolabi et al.,2015). People living with chronic diseases
can benefit from personalized recommendations in the management of their various
diseases to promote healthy living and good management of their conditions (Lafta et al.,
2016;Oliva-Felipe et al., 2018). An instance of this is use of a recommendation system for
linking the electronic patientrecords to beneficial clinical information for use by physicians
(Gil et al., 2019). However, deriving more benefitswill require producing systems that meet
users’health, technology and functional needs. This will involve providing personalized
recommendations to cater to their needs. Meeting these needs is crucial to delivering
effective health interventions.In addition, ascertaining that a system has met these relevant
user needs is another important exercise. Evaluating the system is a practice designed for
achieving this. Evaluation can reveal how effective and useful these systems are and can
indicate whether they have met the expected needs of users and are usefulto them, thereby
increasing their rate of acceptance. The number of health-care recommendation systems
evaluated has increasedconsiderably in recent years. Evaluation is usually carried out using
metrics and criteria. However,there has been no set of metrics or criteria collectively agreed
to by the body of researchers for health recommender systems evaluation, although some
researchers have used certain metrics because other respected researchers have used them
(Zhong and Li, 2016). In the past, accuracy has been about the only metric that was in the
focus as far as evaluation of recommendation systems is concerned. However, many
researchers have advocated the need to look beyond accuracy and consider other metrics
(Chai et al.,2018;McNee et al., 2006b) suchas diversity (Kunaver and Požrl, 2017;Zhang and
Hurley, 2008), usefulness (Murakami et al., 2007), coverage and serendipity (Ge et al.,2010;
Herlocker et al.,2004), novelty (Vargas and Castells, 2011) and if possible multiple metrics
(Gurini et al.,2018).This advocacy notwithstanding, recent review has shown that greater
focus is still on accuracy.
Evaluation criteriahave not been given publicity as metrics have been.While reasons for
this have not been advanced in literature, it is not unconnected with the ease with which
metrics can be measured. In addition, many researchers have used the two terms
interchangeably; therefore,it is seems pretty difficultto delineate betweenthe two. Another
possibility may be the possibilityof using one to determine the other. However, criteria play
a crucial and unique role in defining the acceptability and usefulness of a recommendation
system. Criteria determinewhether a recommender system has distinct qualities that define
Evaluation
criteria and
metrics as a
guide
305
To continue reading
Request your trial