Defining Systems to Evaluate System Efficiency and Effectiveness
| Author | Jessica Renger,Ralph Renger,Jirina Foltysova,Wayne Booze |
| DOI | 10.1177/1035719X1701700302 |
| Published date | 01 September 2017 |
| Date | 01 September 2017 |
| Subject Matter | Academic Article |
4Evaluation Journal of Australasia Vol 17 | No 3 | 2017
ACADEMIC ARTICLE Evaluation Journal of Australasia Vol 17 | No 3 | 2017 | pp. 4–13
RALPH RENGER | JIRINA FOLTYSOVA | JESSICA RENGER | WAYNE BOOZE
Dening systems to evaluate system
eciency and eectiveness
This paper focuses on the application
of systems thinking, systems theory,
and systems evaluation theory (SET ) in
evaluating modern day systems. SET consists
of three steps purposively sequenced with
each being a prerequisite for the success of
the next step. The rst foundational step is to
dene the system. Systems thinking provides
theoretical rationale for dening the system
boundaries, components, and relationships.
However, there is no literature describing
how to dene these system elements. Using
an example from the evaluation of several
United States cardiac care systems, the
paper shares a number of methods used to
dene the system boundaries, components,
and relationships. The paper describes
how each of these elements informs the
evaluation of step two of SET—evaluating
system eciency. The discussion shares
lessons learned, and notes the relationship
between methods used in system and
program evaluation.
Dening systems to evaluate system
eciency and eectiveness
The interest in the potential of systems thinking to
improve evaluation quality continues to grow (Adams,
Hester, Bradley, Meyers & Keating, 2014; Renger, 2015;
Renger, 2016; Renger, Foltysova, Ienuso, Renger & Booze,
2017; Renger, McPherson, Kontz-Bartels & Becker, 2016;
Renger, Wood, Williamson & Krapp, 2012; Rogers,
2011; Wehipeihana, 2011; Williams, 2015; Williams &
Hummelbrunner, 2010). One reason for the interest in
systems thinking is its potential to address limitations
associated with theory-driven program evaluation
(Renger, 2015; Renger, 2016). Critics of theory-driven
program evaluation argue approaches, like logic modeling,
are artificial because program assumptions tend to be
linear. As such, the program assumptions do not reflect
the reality in which programs operate, ignoring many
other contextual factors influencing program outcomes
(Morrel, 2010). Thus, findings from such evaluations are
sometimes dicult to interpret, often meaningless, and
therefore, are of limited utility (Lee, 2017; Patton, 2008).
Systems thinking represents a way forward to address
these criticisms and produce more usable evaluations
(Patton, 2008; Williams, 2015).
Within the evaluation literature, three core systems
principles continually emerge: boundaries, components,
and relationships (Hargreaves & Podems, 2012; Williams
& Hummelbrunner, 2010). It is reasoned the application
of these core principles in program evaluation should lead
to better understanding of the context in which a program
operates and thus more realistic and useful evaluations
(Patton, 2008). Williams and Hummelbrunner (2010)
published numerous research methods to assist evaluators
in applying systems thinking principles to improve
program evaluations.
More recently, Renger (2015) examined the utility of
systems thinking to evaluate modern day systems, not
programs. Ericson (2011) defines a modern day system as
an integrated composite of components that provide
function and capability to satisfy a stated need or
objective. A system is a holistic unit that is greater than
the sum of its parts. It has structure, function, behavior,
characteristics, and interconnectivity. Modern day
systems are typically composed of people, products, and
environments that together generate complexity and
capability (p. 402).
To continue reading
Request your trial