Decision-making framework with double-loop learning through interpretable black-box machine learning models
| Pages | 1389-1406 |
| Date | 14 August 2017 |
| Published date | 14 August 2017 |
| DOI | https://doi.org/10.1108/IMDS-09-2016-0409 |
| Author | Marko Bohanec,Marko Robnik-Šikonja,Mirjana Kljajić Borštnar |
| Subject Matter | Information & knowledge management,Information systems,Data management systems,Knowledge management,Knowledge sharing,Management science & operations,Supply chain management,Supply chain information systems,Logistics,Quality management/systems |
Decision-making framework
with double-loop learning
through interpretable black-box
machine learning models
Marko Bohanec
Salvirt Ltd, Ljubljana, Slovenia
Marko Robnik-Šikonja
Faculty of Computer and Information Science, University of Ljubljana,
Ljubljana, Slovenia, and
Mirjana KljajićBorštnar
Faculty of Organizational Sciences, University of Maribor, Kranj, Slovenia
Abstract
Purpose –The purpose of this paper is to address the problem of weak acceptance of machine learning (ML)
models in business. The proposed framework of top-performing ML models coupled with general explanation
methods provides additional information to the decision-making process. This builds a foundation for
sustainable organizational learning.
Design/methodology/approach –To address user acceptance, participatory approach of action design
research(ADR) was chosen. The proposedframework is demonstrated ona B2B salesforecasting process in an
organizationalsetting, following cross-industrystandard process for data mining (CRISP-DM) methodology.
Findings –The provided ML model explanations efficiently support business decision makers, reduce
forecasting error for new sales opportunities, and facilitate discussion about the context of opportunities in
the sales team.
Research limitations/implications –The quality and quantity of available data affect the performance of
models and explanations.
Practical implications –The application in the real-world company demonstrates the utility of the
approach and provides evidence that transparent explanations of ML models contribute to individual and
organizational learning.
Social implications –All used methods are available as an open-source software and can improve the
acceptance of ML in data-driven decision making.
Originality/value –The proposed framework incorporates existing ML models and general explanation
methodology into a decision-making process. To the authors’knowledge, this is the first attempt to support
organizational learning with a framework combining ML explanations, ADR, and data mining methodology
based on the CRISP-DM industry standard.
Keywords Machine learning, Double-loop learning, B2B sales forecasting, Explanation of black-box models
Paper type Research paper
1. Introduction
Research has shown that the more companies characterized themselves as data driven, the
better they performed on the objective measures of financial and operational results.
In particular, companies in the top third of their industry in the use of data-driven decision
making were, on average, 5 percent more productive and 6 percent more profitable than their
competitors (McAfee et al., 2012; Brynjolfsson et al., 2011). This is a clear motivation for
decision makers to become involved with data-driven decision making, the practice of basing Industrial Management & Data
Systems
Vol. 117 No. 7, 2017
pp. 1389-1406
© Emerald PublishingLimited
0263-5577
DOI 10.1108/IMDS-09-2016-0409
Received 30 September 2016
Revised 27 December 2016
30 March 2017
Accepted 5 April 2017
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/0263-5577.htm
The authors are grateful to the company Salvirt, Ltd, for funding the research and development
presented in this paper. Mirjana KljajićBorštnar and Marko Robnik-Šikonja were supported by the
Slovenian Research Agency, ARRS, through research programs P5-0018 and P2-0209, respectively.
1389
Decision-
making
framework
decisions on data analysis, rather than on intuition (Provost and Fawcett, 2013). However,
business transformation can only be accomplished by people, not by hardware (the computer,
in our context) (Deming, 1986). Therefore, an ability to learn from data is needed to build the
capacity to adapt to changes in an environment in order to achieve organizational goals and
vision. Such learning is characterized by the change of behavior because of an individual/group
exposure to experience (KljajićBorštnar et al., 2011). Two types of learning are distinguished:
single-loop and double-loop learning (Argyris, 1992; DiBella and Nevis, 1998). The single-loop
refers to changing behavior in order to achieve the stated goal. The double-loop learning
supports changing mental models, vision, and beliefs, and therefore building a foundation for
organizational knowledge and sustainable decision making.
The progress in the application of data mining to the broader field of customer
relationship management (CRM) solutions (Ngai et al., 2009) motivates the research question
in this paper:
RQ1. How can interpretable recommendations of knowledge discovery technologies be
integrated into a complex business decision process and continuously support
organizational learning?
Participants in a decision-making process turn to such a solution for an unbiased and
transparent explanation of past outcomes and future predictions and, by doing so, evaluate
options in the context of their business environment. Research has shown that the users of
knowledge-based systems are more likely to adhere to recommendations when, besides the
predictive performance of models, explanations are also available (Arnold et al., 2006) and
user acceptance is secured (Gönül et al., 2006). In order to apply prediction models, users
have to trust them first, and the models’transparency is a major factor in ensuring the trust.
However, current solutions provide their predictions and recommendations in different
formats (Alvarado-Valencia and Barrero, 2014).
Top-performing machine learning (ML) models are complex black boxes; for example,
random forests (RF), support vector machines (SVM) or neural networks achieve
significantly better predictive performance than simple, interpretable models such as
decision trees (DT), naïve Bayes (NB), or decision rules do (Caruana and Niculescu-Mizil,
2006). This is one of the reasons for low usage and acceptance of ML models in areas where
transparency and comprehensibility of decisions are required.
This paper introduces a novel approach to support a knowledge-based decision-making
process with an application of general explanation methodology for ML models. We propose
a framework, which formalizes the process of using company’s historical business data
within a decision-making process and upgrades it with explanations and people in the loop.
The framework utilizes transparent explanations of arbitrary predictive ML models,
including the top-performing black-box models. To encourage organizational acceptance,
we developed the proposed framework using an action design research (ADR) approach,
combining design science research and action research principles in designing an artifact
(in our case the framework with the ML model in its core) engaging participating
organization. The proposed framework follows cross-industry standard process for data
mining (CRISP-DM) recommendations (Wirth and Hipp, 2000). It represents a
comprehensive process model for carrying out data mining projects, independent of both
the industry sector and the technology used. Explanations of ML models are based on two
general methods for explaining classification models and their predictions (Robnik-Šikonja
and Kononenko, 2008; Štrumbelj and Kononenko, 2010). With the proposed framework,
we build a foundation for double-loop learning as a basis to establish new premises
(i.e. paradigms, schemes, mental models, or perspectives), with the potential to override
existing ones (Nonaka and Takeuchi, 1995). To demonstrate the utility of the proposed
framework, we apply it to a challenging real-world B2B judgmental sales forecasting as an
1390
IMDS
117,7
To continue reading
Request your trial