Information technology evaluation: issues and challenges
| Pages | 37-55 |
| DOI | https://doi.org/10.1108/13287261011032643 |
| Published date | 02 February 2010 |
| Date | 02 February 2010 |
| Author | Govindan Marthandan,Chun Meng Tang |
| Subject Matter | Information & knowledge management |
Information
technology
evaluation
37
Journal of Systems and Information
Technology
Vol. 12 No. 1, 2010
pp. 37-55
#Emerald Group Publishing Limited
1328-7265
DOI 10.1108/13287261011032643
Information technology
evaluation: issues and challenges
Govindan Marthandan and Chun Meng Tang
Faculty of Management, Multimedia University, Cyberjaya, Malaysia
Abstract
Purpose – To justify an increase in information technology (IT) spending and to understand
utilization of limited organizational resources on IT, the correlation between IT and business
performance has been of great interest to business managers. However, business managers face
issues and challenges in finding out how and to what extent IT is able to deliver the intended
benefits. The purpose of this paper is to examine IT evaluation issues and challenges faced by
information systems (IS) researchers, IS specialists, and business managers.
Design/methodology/approach – This paper begins by reviewing the disparate discussions in
past literature on IT evaluation issues and challenges. It then provides a synthesis of the disparate
discussions by identifying eight issues and challenges in IT evaluation.
Findings – The eight issues and challenges identified are: evaluation scope, evaluation timing, unit
of analysis, level of analysis, different perspectives, different dimensions, different measures, and
underpinning theoretical frameworks. It conc ludes with some suggestions on ways to improve IT
evaluation practices.
Originality/value – This paper posits that before a pragmatic IT evaluation approach can be
developed, it is necessary to first understand the issues and challenges faced by IS researchers, IS
specialists, and businessmanagers in IT evaluation. Havingidentified the eight issues and challenges,
this paper provides pointers on what needs to be considered when conducting ITevaluation.
Keywords Communication technologies, Decision making, Value analysis
Paper type General review
1. Introduction
As information technology (IT) advances dramatically with new features and
capabilities, moving away from the data processing era to a strategic information
systems (IS) era, efficiency measures are no longer sufficient to illustrate the true
business value of IT. Over the years, expectations of business managers on IT value
have gradually shifted from operational-centered to strategic-focused. How fast a
processor runs or how many pages a printer prints might not interest business
managers. Instead, business managers are interested in the strategic advantages that
IT brings. As highlighted by Seddon et al. (2002), IT evaluation is slowly shifting from
the technical or financial perspective to a business-o riented one. Thus, it is many times
more difficult to evaluate IT now than in the past as we are looking at not only tangible
but also intangible benefits.
Traditionally,business managers justify their ITsp ending with the help of financial
techniques, e.g. payback period, net present value, and internal rate of retur n, that are
common for evaluation of capital investments. However, the effectiveness of financial
techniques in evaluating IT investments is debatable as IT investments are different
from other capital investments – especially when we cannot estimate hidden,
intangible, and non-financial benefits (Ballantine and Stray, 1998; Irani and Love, 2000/
2001). For example, return on investment (ROI) calculation can be negative for a
threshold IT investment that is critical for the survival of an organization. Strategic IT
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1328-7265.htm
The authors would like to thank the anonymous reviewers and the Editor for the valuable
comments which have helped us to improve the quality of the paper.
JSIT
12,1
38
investments cannot be evaluated using tangible criteria alone; there can be other
intangible benefits (Weill and Olson, 1989).
Without considering intangible benefits, the use of traditional evaluation techniques
can lead to wrong investment decisions (Anandarajan and Wen, 1999; Irani, 2002; Irani
et al., 1997; Law and Ngai, 2005; Lubbe and Remenyi, 1999). Irani et al. (2002) explain
that as organizations are searching for long-term strategic benefits rather than short-
term operational benefits, traditional evaluation techniques can be inappropriate. Irani
et al. (2006) reason that if the investment objective of an IT project is to reduce
operational costs, then traditional evaluation techniques are sufficient. However, for a
project that aims to exploit IT for strategic benefits, a newer approach like balanced
scorecard might be a better choice for evaluation purposes.
In strategic IT investment evaluation, identifying qualitative, intangible, or hidden
value poses a major challenge to business managers. Also, a strategic IT implementation
requires complementary organizational and business changes, e.g. business processes
redesign, which in turn makes it difficult to segregate IT value from the total value
(Brynjolfsson and Hitt, 2000). The differentobjectives of IT investments further complicate
IT evaluation. For example, in a study of IT performance in 33 small-to-medium-sized
valve manufacturing firms, Weill (1992) categorized IT investments in terms of
management objectives, i.e. strategic, informational, and transactional. He reported that
transactional IT contributed significantly to better firm performance, but not in the case of
informational IT. However, strategic IT did not prove its worth over a longer term. This
could be because strategic advantages of strategic IT would be eroded once competitors
have emulated it. He concluded that different types of IT investment served different
management objectives and thus exhibited different effects on firm performance.
Although organizations are keen to determine the relationship betwe en IT
investment and organizational performance, the assessment of IT investment returns
is difficult and organizations do not have adequate evaluation frameworks for that
purpose. Leem and Kim (2004) claim that past IS evaluation studies suffer from several
limitations. First, IS evaluation should be conducted in all stages of IS development life
cycle, i.e. from initiation to post implementation. However, past studies only assessed
a particular stage. Second, although past studies have be en working hard on
performance measures and IS-perfo rmance relationship, they have not been successful
in identifying the best evaluation approach. Third, past studi es in general did not
discuss business implications of IS evaluation.
Berghout and Remenyi (2005), having reviewed some 298 articles presented at the
European Conference on IT Evaluation, conclude that future research should try to
build a theoretical groundwork of IT evaluation with strong core concepts and better
evaluation methodologies. Stressing the imp ortance of ITevaluation, they suggest that
the reasons behind lukewarm support among IS practitioners for IS evalua tion should
be explored further. Although a daunting challenge, there is a strong need to develop
underpinning theoretical frameworks for better evaluation of IT investment.
The difficulties, problems, issues, and challenges faced by IS researchers, IS
specialists, and business managers have been repo rted in past studies. However, the
discussions are disparate, scattered, and in most cases, lack depth. This paper is
motivated to fill in this knowledge gap. This paper posits that before a pragmati c IT
evaluation framework can be developed, it is necessary to first understand the issues
and challenges faced by IS researchers, IS specialists, and business managers in IT
evaluation. In order to meet this objective, this paper follows the p ointers provided by
Webster and Watson (2002) on writing a review paper.
To continue reading
Request your trial