Evaluating knowledge management systems efficacy and effectiveness in a design science context
| Date | 11 November 2013 |
| Published date | 11 November 2013 |
| DOI | https://doi.org/10.1108/JSIT-08-2013-0041 |
| Pages | 324-346 |
| Author | Sittimont Kanjanabootra,Brian Corbitt,Miles Nicholls |
| Subject Matter | Information & knowledge management,Information systems |
Evaluating knowledge
management systems efficacy
and effectiveness in a design
science context
Sittimont Kanjanabootra
School of Architecture and Built Environment,
The University of Newcastle, Newcastle, Australia
Brian Corbitt
College of Business, RMIT University, Melbourne, Australia, and
Miles Nicholls
Graduate School of Business and Law, RMIT University,
Melbourne, Australia
Abstract
Purpose – This paper aims to propose a framework for the evaluation of artefacts in Design Science
and test it using an exemplar case of a knowledge management system (KMS) developed for an
Australian refrigeration manufacturing company.
Design/methodology/approach – The research uses Design Science research methodology in a
specific case study context. The artefact studied was developed using an ontology based on an
engineering design conceptualisation and created using an ontology generator, Prote
´ge
´. Research data
for the evaluation of the framework were collected using a combination of document analysis,
interviews, shadowing and observations.
Findings – The evaluation framework developed for the research and applied to the KMS specifically
built for the company was shown to be useful in determining the efficacy and effectiveness of the
research outcomes in terms of usefulness to the company engineers in the technical analysis of their
work, and for the CEO and COO as part of their strategic planning for the company. The evaluation
framework helped the researcher and the engineers as collaborators to demonstrate the extent of
improvement in the design and build processes in the company.
Originality/value – Prior research in both Information System and Design Science has not provided
a specific, generalizable, evaluation framework for system developers to use as a guide during the
systems development process. This research proposes an evaluation framework which covers all
broad aspects of evaluation and efficacy, accepting that evaluation frameworks must be flexible in
enabling changes to accommodate variations in the types and purposes of artefacts developed.
Keywords Design Science,Evaluation framework, Knowledge management system
Paper type Research paper
Introduction
This paper describes the process of evaluating the efficacy and effectiveness of a
knowledge management system (KMS) built as part of research for an Australian
refrigeration manufacturing company, designed to enable the company to deal with
identified strategic business problems, and proposes an evaluation framework for
Design Science research. The artefact studied, here a KMS, is a key element in Design
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1328-7265.htm
Received 19 August 2013
Revised 19 August 2013
Accepted 3 September 2013
Journal of Systems and Information
Technology
Vol. 15 No. 4, 2013
pp. 324-346
qEmerald Group Publishing Limited
1328-7265
DOI 10.1108/JSIT-08-2013-0041
JSIT
15,4
324
Science research. It is important to understand the evaluation process as this is central
to determining the quality of Design Science research. Whilst there has been significant
attention paid to the principles and enablement of Design Science research (Geerts,
2011; Hevner and Chatterjee, 2010a, b; Wang et al., 2011) there is significantly less
attention given to attenuating evaluation (Venable, 2010), albeit that such evaluation is
an integral part of Design Science research. This paper focuses on the process of
evaluation of a specific KMS, but provides the context for more generalizability of the
evaluation process with applications across other Design Science research.
The research context
The research was nested in resolving a strategic business problem for a refrigeration
company, through creating and using a KMS. Knowledge management (KM) helps
experts in organizations to pass their knowledge to new employees to be able to work in
the organization (Alavi and Leidner, 2001; Bartholomew, 2008; Hackett, 2000; Schwartz,
2006) and has been used to focus on organizational improvement (Davenport and
Prusak, 1998). Business today operates in a knowledge-based business framework. New
technologies and methods have been studied to facilitate KM in organizations, not just
for day-to-day uses but also about strategically using knowledge to improve business
(Davenport and Prusak, 1998; Ichijo and Kohlbacher, 2008; Nonaka and Takeuchi, 1995).
This research has focused on knowledge that employees in companies already have but
which has not been used to its full potential (Quintas et al., 1997). Capturing this
knowledge, it is argued, can improve quality of products and services reduce costs and
improve organizational use of time. Kamara et al. (2002) argue that KM is the way that
organizations make value out of their intellectual assets via methods, tools and
techniques to improve the business.
From an engineering perspective, KM includes the way to use knowledge to extract
information from an information overloaded environment and to re-use that engineering
and manufacturing knowledge to achieve design requirements and reduce overtime
delivery of products by better time utilization (Quintas et al., 1997). Implementing KMS
alone though is not the key to success (Alavi and Leidner, 2001; Davenport and Prusak,
1998; Kamara et al., 2002). Organizations that have implemented a KMS have still
struggled in achieving business improvement. There are other problems related to
people and organizations that need to be addressed. In engineering this can be
professional knowledge and experience and can relate to specific design processes built
up over long periods of design and manufacture. IT systems can certainly store explicit
knowledge as artefacts and documents and, developed properly, can store captured
knowledge. However, there are problems with knowledge capture which often means
that attempts at the use of KM for strategic purposes fail (Kanjanabootra et al., 2010; Silvi
and Cuganesan, 2009). By following this approach it can be argued that the strategic
benefits of the use of KM can be realised. This is the role of evaluation and is the
focus here.
The research setting
The company involved in this research is a customizing refrigerated display cabinet
manufacturer. The company manufactures customized refrigerators in various forms
such as food product display units in supermarkets, wine cellars, fresh produce
displays, dairy cabinets, etc. These customized refrigerators differ between units as
Knowledge
management
systems
325
To continue reading
Request your trial