Extracting and applying evaluation criteria for ontology quality assessment
| Pages | 338-354 |
| DOI | https://doi.org/10.1108/LHT-01-2019-0012 |
| Published date | 16 September 2019 |
| Date | 16 September 2019 |
| Author | Seonghun Kim,Sam G. Oh |
Extracting and applying
evaluation criteria for ontology
quality assessment
Seonghun Kim and Sam G. Oh
iSchool –Library and Information Science and Data Science,
Sungkyunkwan University (SKKU), Seoul, The Republic of Korea
Abstract
Purpose –The purpose of this paper is to formulate apposite criteria for ontology evaluation and test them
through assessments of existing ontologies.
Design/methodology/approach –A literature review provided the basis from which to extract the
categories relevant to an evaluation of internal ontology components. According to the ontology evaluation
categories, a panel of experts provided the evaluation criteria for each category via Delphi survey. Reliability
was gauged by applying the criteria to assessments of existing smartphone ontologies.
Findings –Existing research tends to approach ontology evaluation through comparison with well-
engineered ontologies, implementation in target applications and appropriateness/interconnection appraisals
in relation to raw data, but such methodologies fall short of shedding light on the internal workings of
ontologies, such as structure, semantic representation and interoperability. This study adopts its evaluation
categories from previous research while also collecting concrete evaluation criteria from an expert panel and
verifying the reliability of the resulting 53 criteria.
Originality/value –This isthe first published studyto extract ontology evaluationcriteria interms of syntax,
semantics and pragmatics. The results can be used as an evaluation index following ontology construction.
Keywords Ontology, Knowledge management, Delphi method
Paper type Research paper
1. Introduction
Ontologies provide the tools needed to overcome barriers when integrating data and
knowledge from heterogeneous data sets, facilitating knowledge discovery in the big data
era (Amith et al., 2018). The term “ontology”–from the Greek “on”(being) and “logos”
(theory) and the Latin “ontologia”–traditionally refers to a branch of philosophy on the
study of being. The expression of that which exists presupposes conceptualization, a
process rooted in concepts and objects as well as the relationship thereof (Genesereth
and Nilsson, 1987). Gruber (1993) describes ontologies as an “explicit and formalized
specification of conceptualization,”an interpretation carried over into the field of
information management to yield a framework which “clearly and explicitly defines and
demarcates the internal thoughts of the human mind or shared concepts concerning the
phenomena of the external world”(Ko and Seo, 2005).
Ontologies have proved a powerful tool across a broad spectrum of areas,
including terminology, data semantics management, Semantic Web and artificial
intelligence (Ko and Seo, 2005). In terminology and data semantics management,
ontologies serve as a method of expressing the definitions and relationships of terms or
resources, while they provide the foundation for more meaningful Web development in
Semantic Web studies and the knowledge base for diverse implementations in artificial
intelligence research (Hahm et al., 2014). Ontologies also demonstrate considerable
promise in heightening analytic effectiveness when dealing with big data gleaned from
social media (Kim, 2015).
The sheer range of such potential has drawn greater attention to issues of quality
assessment and selection. Quality assessments for ontologies rely heavily on comparison
with well-engineered existing ontologies, utilization in target applications, appropriateness/
Library Hi Tech
Vol. 37 No. 3, 2019
pp. 338-354
© Emerald PublishingLimited
0737-8831
DOI 10.1108/LHT-01-2019-0012
Received 15 January 2019
Accepted 12 April 2019
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/0737-8831.htm
338
LHT
37,3
interconnection appraisals for raw data, and the adoption of pre-established evaluation
criteria. However, these methods concentrate on either applying ontologies to retrieval and
inference tasks as the basis of function assessment or using ontologies to enhance
applications (Noh, 2011; Kim and Ahn, 2007; Lee and Kim, 2007; Lee et al., 2007). They
accordingly fail to deliver concrete reinforcement when evaluating the internal aspects of an
ontology, such as structure, semantic representation and interoperability (Park et al., 2008).
Addressing these shortcomings, this study gathered quality assessment parameters
from ontology experts and formulated its final set of criteria with professional feedback.
Reliability was verified by applying the criteria in each category to ontology evaluation in a
specific domain. The results are expected to contribute relevant criteria in various
assessments of ontology quality and to formulate a checklist for ontology construction.
2. Ontology evaluation methodology
Ontology evaluation methodologies fall into four categories (Brank et al., 2006): first,
comparing the new ontology to gold standard ontologies of substantiated quality; second,
utilizing the new ontology in its intended application and ascertaining functionality; third,
assessing the appropriateness and interconnection between the new ontology and its source
data sets; and, fourth, conducting an expert evaluation based on pre-defined criteria,
standards and requirements.
Maedche and Staab (2002) employ the first method, introducing a standard framework
by which an ontology may be expanded, modified or compared to a different ontology in a
larger domain. Their research centers on an empirical study of lexical, taxonomic and
relational comparisons.
One example of the second method is Porzel and Malaka (2004), which begins with the
assumption that an effective ontology will prove its worth by demonstrating competence in
task environments. The ontology used in the study is subjected to task-based evaluation
and measured for concept vocabulary, hierarchy/granularity and semantic relations.
Brewster et al. (2004) conduct a data-driven evaluation using the third method. The first
phase of the research considers the requirements of the specific application and subject area
to which the ontology has been tailored. Following the extraction of terms from the domain,
the results are sifted to determine those best suited to existing applications. The second
phase carries out automatic mapping tests between the extracted terms and both the studied
ontology and the ontology used as a point of comparison. Simple or vector comparisons are
executed on the basis of terms discoverable in the extraction but not in the included
ontologies or terms. The third phase involves a more sophisticated three-tier process of
keyword identification, query expansion and ontology mapping intended to enhance the
appropriateness of the evaluation, while the final phase exhibits a probability-based
approach to ontology assessment.
Another case relevant to the third method, Lee and Kim (2013), examines 376 instances of
ontology source data on the Saemaeul Movement provided by the National Archives of
Korea. Instances determined to be topic- or query-appropriate are applied to a comparative
analysis of existing search engines vs ontology-based search engines.
The fourth method is relevant to Cho and Kang (2013), Lozano-Tello and Gómez-Pérez (2004)
and Park et al. (2008). Verifying an ontology intended to facilitate the comprehension of
geographical concepts, Cho and Kang interviews four ontology experts to conduct an
assessment of taxonomy and instance dimensions in eight subcategories. Through the analytic
hierarchy process method, Lozano-Tello and Gómez-Pérez develops ONTOMETRIC in response
to the upsurge in ontology-based applications. This tool enables the user to select appropriate
ontologies through an evaluation process complete with comprehensive characteristics and
factors within the dimensions of tools, language, content, methodology and cost. Park et al.
339
Extracting and
applying
evaluation
criteria
To continue reading
Request your trial