Machine learning facilitated business intelligence (Part II). Neural networks optimization techniques and applications
| Pages | 128-163 |
| Date | 27 November 2019 |
| Published date | 27 November 2019 |
| DOI | https://doi.org/10.1108/IMDS-06-2019-0351 |
| Author | Waqar Ahmed Khan,S.H. Chung,Muhammad Usman Awan,Xin Wen |
| Subject Matter | Information & knowledge management,Information systems,Data management systems,Knowledge management,Knowledge sharing,Management science & operations,Supply chain management,Supply chain information systems,Logistics,Quality management/systems |
Machine learning facilitated
business intelligence (Part II)
Neural networks optimization
techniques and applications
Waqar Ahmed Khan and S.H. Chung
Department of Industrial and Systems Engineering,
The Hong Kong Polytechnic University, Kowloon, Hong Kong
Muhammad Usman Awan
Institute of Quality and Technology Management,
University of the Punjab, Lahore, Pakistan, and
Xin Wen
Department of Industrial and Systems Engineering,
The Hong Kong Polytechnic University, Kowloon, Hong Kong
Abstract
Purpose –The purpose of this paper is three-fold: to review t he categories explaining mainly o ptimization
algorithms (techniqu es) in that needed to imp rove the generalizati on performance and learn ing speed
of the Feedforward Neura l Network (FNN); t o discover the change in res earch trends by analyzin g
all six categories (i.e . gradient learning algor ithms for network traini ng, gradient free learni ng
algorithms, optimiz ation algorithms for lea rning rate, bias and varia nce (underfitting and ove rfitting)
minimization algori thms, constructive top ology neural networks, me taheuristic search alg orithms)
collectively; and rec ommend new research dire ctions for researcher s and facilitate users to un derstand
algorithms real-world applications in solving complex management, engineering and health
sciences problems.
Design/methodology/approach –The FNN has gained much attention from researchers to make a more
informed decision in the last few decades. The literature survey is focused on the learning algorithms and the
optimization techniques proposed in the last three decades. This paper (Part II) is an extension of Part I.
For the sake of simplicity, the paper entitled “Machine learning facilitated business intelligence (Part I): Neural
networks learning algorithms and applications”is referred to as Part I. To make the study consistent with
Part I, the approach and survey methodology in this paper are kept similar to those in Part I.
Findings –Combining the work performed in Part I, the authors studied a total of 80 articles through
popular keywords searching. The FNN learning algorithms and optimization techniques identified in the
selected literature are classified into six categories based on their problem identification, mathematical model,
technical reasoning and proposed solution. Previously, in Part I, the two categories focusing on the learning
algorithms (i.e. gradient learning algorithms for network training, gradient free learning algorithms) are
reviewed with their real-world applications in management, engineering, and health sciences. Therefore, in
the current paper, Part II, the remaining four categories, exploring optimization techniques (i.e. optimization
algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms,
constructive topology neural networks, metaheuristic search algorithms) are studied in detail. The algorithm
explanation is made enriched by discussing their technical merits, limitations, and applications in their
respective categories. Finally, the authors recommend future new research directions which can contribute to
strengthening the literature.
Research limitations/implications –The FNN contributions are rapidly increasing because of its ability
to make reliably informed decisions. Like learning algorithms, reviewed in Part I, the focus is to enrich the
comprehensive study by reviewing remaining categories focusing on the optimization techniques. However,
future efforts may be needed to incorporate other algorithms into identified six categories or suggest new
category to continuously monitor the shift in the research trends.
Industrial Management & Data
Systems
Vol. 120 No. 1, 2020
pp. 128-163
© Emerald PublishingLimited
0263-5577
DOI 10.1108/IMDS-06-2019-0351
Received 11 July 2019
Revised 11 October 2019
Accepted 28 October 2019
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/0263-5577.htm
This work was supported by a grant from the Research Committee of The Hong Kong Polytechnic
University under the account code RLKA, and supported by RGC (Hong Kong) –GRF, with the Project
Number: PolyU 152131/17E.
128
IMDS
120,1
Practical implications –The authors studied the shift in research trend for three decades by collectively
analyzing the learning algorithms and optimization techniques with their applications. This may help
researchers to identify future research gaps to improve the generalization performance and learning speed,
and user to understand the applications areas of the FNN. For instance, research contribution in FNN in the
last three decades has changed from complex gradient-based algorithms to gradient free algorithms, trial and
error hidden units fixed topology approach to cascade topology, hyperparameters initial guess to analytically
calculation and converging algorithms at a global minimum rather than the local minimum.
Originality/value –The existing literature surveys include comparative study of the algorithms,
identifying algorithms application areas and focusing on specific techniques in that it may not be able to
identify algorithms categories, a shift in research trends over time, application area frequently analyzed,
common research gaps and collective future directions. Part I and II attempts to overcome the existing
literature surveys limitations by classifying articles into six categories covering a wide range of algorithm
proposed to improve the FNN generalization performance and convergence rate. The classification of
algorithms into six categories helps to analyze the shift in research trend which makes the classification
scheme significant and innovative.
Keywords Optimization techniques, Data analytics, Machine learning, Feedforward neural network,
Industrial management
Paper type Research paper
1. Introduction
The Feedforward Neural Network (FNN) has gained much attention from researchers in the
last few decades (Abdel-Hamid et al., 2014; Babaee et al., 2018; Chen et al., 2018; Chung et al.,
2017; Deng et al., 2019; Dong et al., 2016; Ijjina and Chalavadi, 2016; Kastrati et al., 2019;
Kummong and Supratid, 2016; Mohamed Shakeel et al., 2019; Nasir et al., 2019; Teo et al.,
2015; Yin and Liu, 2018; Zaghloul et al., 2009) because of its ability to extract useful patterns
and make a more informed decision from high dimensional data (Kumar et al., 1995;
Tkáčand Verner, 2016; Tu, 1996). With modern information technology advancement,
the challenging issue of high dimensional, non-linear, noisy and unbalanced data are
continuously growing and varying at a rapid rate so that it demands efficient learning
algorithms and optimization techniques (Shen, Choi and Chan, 2019; Shen and Chan, 2017).
The data may become a costly resource if not analyzed properly in the process of business
intelligence. Machine learning is gaining significant interest in facilitating business
intelligence in the process of data gathering, analyses and extracting knowledge to help
users in making better informed decisions (Bottani et al., 2019; Hayashi et al., 2010; Kim et al.,
2019; Lam et al., 2014; Li et al., 2018; Mori et al., 2012; Wang et al., 2005; Wong et al., 2018).
Efforts are being made to overcome the challenges by building optimal machine learning
FNNs that may extract useful patterns from the data and generate information in real-time
for better-informed decision making. Extensive knowledge and theoretical information are
required to build FNNs having the characteristics of better generalization performance and
learning speed. The generalization performance and learning speed are the two criteria that
play an essential role in deciding on the use of learning algorithms and optimization
techniques to build optimal FNNs. Depending upon the application and data structure, the
user might prefer either better generalization performance or faster learning speed, or a
combination of both. Some of the drawbacks that may affect the generalization performance
and learning speed of FNNs include local minima, saddle points, plateau surfaces,
hyperparameters adjustment, trial and error experimental work, tuning connection weights,
deciding hidden units and layers and many others. The drawbacks that limit FNN
applicability may become worse with inappropriate user expertise and insufficient
theoretical information. Several questions were identified in Part I of the study that may be
the causes of the above drawbacks. For instance, how to define network size, hidden units,
hidden layers, connection weights, learning rate, topology and many others.
Previously, in Part I of the study, we made an effort to answer the key questions by
reviewing two categories explaining the learning algorithms. The categories were named as
129
Neural
networks
optimization
techniques
gradient learning algorithms for network training and gradient free learning algorithms.
In the current paper, Part II, we made an effort to review the remaining four categories
explaining optimization techniques (i.e. optimization algorithms for learning rate, bias and
variance (underfitting and overfitting) minimization algorithms, constructive topology
neural networks, metaheuristic search algorithms). Part II is an extension of Part I. For each
category, researcher efforts to demonstrate the effectiveness of their proposed optimization
techniques in solving real-world management, engineering, and health sciences problems
are also explained to enrich the content and make users familiar with FNN applications
areas. Moreover, all categories are collectively analyzed in the current paper in order to
discover the shift in research trends. Based on the review of the existing literature, the
authors recommended future research directions for strengthening the literature. In-depth
knowledge from the survey will help researchers to design a new, simple and compact
algorithm having better generalization performance characteristics, and in generating
results in the shortest possible time. Similarly, users may be able to decide and select the
algorithm that best suits their application area.
The paper is organized as follow: Section 2 is about survey methodology. Section 3
briefly overviews Part I of the study. In Section 4, four categories that focus on optimization
techniques are reviewed with a detailed description of each algorithm in terms of its merits,
limitations, real-world management, engineering, and health sciences applications. Section 5
is about future directions to improve FNN generalization performance and learning speed.
Section 6 concludes the paper.
2. Survey methodology
2.1 Source of literature and philosophy of review work
The sources of the literature and philosophy of review work are identical to those in Part I.
Combining the review articles of Part I, the authors studied in total 80 articles, in which
63 (78.75 percent) were journal papers, 10 (12.50 percent) conference papers, 3 (3.75 percent)
online arXiv archives, 2 (2.50 percent) books, 1 (1.25 percent) technical report and
1 (1.25 percent) online academic lecture. Previously, in Part I, only 38 articles were reviewed,
mainly in learning algorithms categories. In the current paper, the remaining articles are
reviewed in Section 4, which explains the optimization techniques needed to improve the
generalization performance and learning speed of FNN. In this section, all 80 articles (Part I
and Part II) are collectively analyzed to discover the shift in research trends.
2.2 Classification schemes
This paper classification is an extension of Part I and focuses on optimization techniques
recommended in the last three decades for improving the generalization performance and
learning speed of FNN. In this subsection, all categories are collectively analyzed to discover the
shift in research trends. Combining the categories explained in Part I, in total six categories are:
(1) gradient learning algorithms for network training;
(2) gradient free learning algorithms;
(3) optimization algorithms for learning rate;
(4) bias and variance (underfitting and overfitting) minimization algorithms;
(5) constructive topology FNN; and
(6) metaheuristic search algorithms.
Categories one and two are extracted from Part I, whereas, categories three to six are
reviewed in the current paper. In Part I, the first category considered gradient learning
130
IMDS
120,1
To continue reading
Request your trial