Utilizing a realist evaluative research approach to investigate complex technology implementations. An e-learning lecture capture exemplar
| Pages | 22-41 |
| Date | 13 March 2017 |
| Published date | 13 March 2017 |
| DOI | https://doi.org/10.1108/JSIT-04-2017-0027 |
| Author | Melanie Rose Nova King,Ray J. Dawson,Steve J. Rothberg,Firat Batmaz |
| Subject Matter | Information & knowledge management,Information systems,Information & communications technology |
Utilizing a realist evaluative
research approach to investigate
complex technology
implementations
An e-learning lecture capture exemplar
Melanie Rose Nova King and Ray J. Dawson
Department of Computer Science, Loughborough University,
Loughborough, UK
Steve J. Rothberg
School of Mechanical and Manufacturing Engineering, Loughborough University,
Loughborough, UK, and
Firat Batmaz
Department of Computer Science, Loughborough University,
Loughborough, UK
Abstract
Purpose –This study aims to investigate the effectiveness of a theory-driven realist evaluative research
approachto better understand complex technology implementationsin organizations.
Design/methodology/approach –An institution wide e-learning implementation of lecture capture
(LC), within a UK University, was chosen, and a realist evaluation framework was used, tailored for
educational technology. The research was conducted over four, increasingly focused, evaluation cycles
combining engagement analytics, user interviews and theory to refine what works (or does not work), for
whom, in which contextsand why.
Findings –Despite explicit demand and corresponding investment, overall student engagement is
lower than expected. Increased student use appears linked to particular staff attitudes and
behaviours and not to specific disciplines or course content. The main benefits of LC are providing
reassurance to the majority, aiding revision and understanding for the many and enabling catch-up
for the few. Recommendations for future research are based on some unexpected outcomes
uncovered, including evolving detrimental student behaviours, policy development based on
technological determinism and future learner-centred system development for next-generation LC
technologies.
Practical implications –The realist approach taken, and evaluation framework used, can be adopted
(and adapted) for future evaluative research. Domain specific reference models, categorizing people and
technology,supported analysis across multiple contexts.
Originality/value –This study responds to a call for more theory-based research in the field of
educational technology. The authorsdemonstrate that a theory-driven approach provides real and practical
recommendations for institutions and allows for greater insight into the political, economic and social
complexityof technology implementation.
Keywords Educational technology, Engagement analytics, Evaluation methodology,
Lecture capture, Realist evaluation
Paper type Research paper
JSIT
19,1/2
22
Received9 April 2017
Revised25 July 2017
Accepted1 August 2017
Journalof Systems and
InformationTechnology
Vol.19 No. 1/2, 2017
pp. 22-41
© Emerald Publishing Limited
1328-7265
DOI 10.1108/JSIT-04-2017-0027
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/1328-7265.htm
1. Introduction
There is a growing scholarly critique admonishing the lack of theory-based research in the
field of educational technology,resulting in a general mistrust of such technology (Gunn and
Steel, 2012;Selwyn, 2014b). Consequently, there is a call for researchers to take a more
critical perspective on the use of technology in education (Bulfinet al.,2015). Selwyn and
Facer’s (2013) research has observedthe tension between rhetoric and reality in educational
technology scholarship, that is “well able to discuss how educational technology could and
should be used, but less competent and confident in discussing how and why educational
technologies are actuallybeing used”. They state that the research is “ill equippedto support
the building of an achievable political or institutional project to realize desirable change”
(Selwyn and Facer, 2013, p. 3). The prominence of digital systems in all aspects of higher
education (HE) also produces an increasingly complex and problematic landscape of data
structures and work processes across all boundaries of operations, teaching and research,
leading to “[deep] rooted concerns over the social, political and cultural roles of these
systems”(Selwyn,2014a, p. 44).
To address the political and cultural complexity of technologies’use in education, as
well as the lack of theory in research, we draw upon realist evaluative research (Pawson
and Tilley, 1997) as a catalyst for a new direction of investigative technique. The theory-
driven approach has been adapted and refined into an evaluation framework specifically
tailored for socio-technical initiatives in education (King, et al., 2016). This research tests
the application of the framework to see if it can bring a more critical perspective on the
use of technology in education, provides insights into the complex political and cultural
landscapes and address the five factors previously identified as barriers to effective
evaluation (King et al., 2014). These are:
(1) Summative evaluations (of products, projects or process) are premature and can
never provide a full understanding of the potential influence and future impact on
an initiative.
(2) Existing software evaluation techniques and technology acceptance models are
inappropriate for dealing with complex contextual factors.
(3) HE is in a rapid state of political and corporate change often requiring quantitative
evidence of efficiencies made.
(4) Complex implementation chains or the iterative nature of agile development means
what is being evaluated is always in a state of change.
(5) The use of inconsistent terminology within HE, often locally adapted or country
specific, is a barrier to synthesis across studies.
1.1 Research objectives and significance of this study
This paper outlines an experiment that tests the evaluation framework to answer the
following researchquestions:
RQ1. Does a realist approach(and the tailored framework) address the factors identified
as barriers to effectiveevaluation of technology in education?
RQ2. Can a theory-driven approach provide real and practical recommendations for
organizations?
RQ3. Does this approach provide a greater insight into the political, economic, cultural
and social complexityof complex technology implementations in education?
An e-learning
lecture capture
exemplar
23
To continue reading
Request your trial