Review of the Delphi method in library and information science research
| DOI | https://doi.org/10.1108/JD-09-2019-0178 |
| Pages | 929-960 |
| Published date | 28 February 2020 |
| Date | 28 February 2020 |
| Author | Brady D. Lund |
| Subject Matter | Library & information science,Records management & preservation,Document management,Classification & cataloguing,Information behaviour & retrieval,Collection building & management,Scholarly communications/publishing,Information & knowledge management,Information management & governance,Information management,Information & communications technology,Internet |
Review of the Delphi method in
library and information
science research
Brady D. Lund
Emporia State University, Emporia, Kansas, USA
Abstract
Purpose–This article presents an introduction to the Delphi method and review of Delphi studies published in
the literature of library and information science (LIS).
Design/methodology/approach –A review of Delphi studies published between the years of 1971 and 2019
is performed, using studies retrieved from the Library and Information Science Source database. A total of 122
articles were retrieved and evaluated based on the population studied, means of identifying experts, number of
participants for each study round, type of Delphi, and type of findings.
Findings –General librarians (any type), academic librarians, and information science researchers are the
most common populations in LIS Delphi studies. On average (middle 50 percent of studies), 14–36 experts are
used in the first round of LIS Delphi studies (median n523). Employment in a specific role and publications in
scholarly journals are the most common means of identifying experts. Variants of the e-Delphi(online survey/
email) method are increasingly common, particularly in LIS Delphi studies that focus on general information
science, rather than library, topics. Though LIS Delphi studies are relatively few in number, they have a
consistent record of being published in some of the most prestigious LIS journals.
Originality/value –This paper provides an introduction to the Delphi method for LIS research and presents
an overview of existing literature in LIS that utilizes the research method. No overview of this extent exists in
the LIS literature, and, thus, this paper may serve as an important information source about the method for LIS
researchers.
Keywords Review, Research methods, Delphi, Delphi method, Information studies, Information research,
Library and information science
Paper type General review
Though the Delphi method has existed as a legitimate research method for over five decades,
there is limited knowledge of this method among library and information science (LIS)
researchers and few LIS scholarly publications that employ the method. Little is known of
how the method has historically been used in LIS publications and how this knowledge may
guide future LIS Delphi studies. The purpose of the following review article is to present an
overview of the major features of LIS Delphi studies –populations used, types of Delphi used,
types of findings retrieved –in order to both encourage and provide precedent and guidance
for future LIS Delphi research.
Literature review
History of the Delphi method
The Delphi method was developed by the RAND Corporation in the 1950s and 1960s as a
means to prepare for national security threats during the Cold War (Woudenberg, 1991).
RAND gathered several top foreign policy and national security experts across the United
States and utilized a new methodology of consensus-gathering, which they believed would
mitigate the challenges presented from a single member of a focus group dominating the
Review of the
Delphi method
929
Author would like to acknowledge the support of the Research Faculty at Emporia State University's
School of Library and Information Management.
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/0022-0418.htm
Received 12 September 2019
Revised 8 November 2019
21 January 2020
26 January 2020
Accepted 30 January 2020
Journal of Documentation
Vol. 76 No. 4, 2020
pp. 929-960
© Emerald Publishing Limited
0022-0418
DOI 10.1108/JD-09-2019-0178
conversation and the vast disparity in responses received during individual interviews
(Dalkey and Helmer, 1963;Helmer, 1967). The researchers quickly recognized that this
method held potential value for the study of consensus among other social groups, like
improving the diffusion of educational innovations (Helmer, 1966), students (Dalkey, 1969),
and marketing (Jolson and Rossow, 1971). At this point in time, however, the method was still
only used within the RAND Corporation; few academics had adopted it as a legitimate
research method (Pill, 1971).
Adoption of the Delphi method as a legitimate research method in academia began during
the early 1970s, primarily in the fields of management and marketing, where collecting the
focused opinions of a large group of respondents was shown to have potential benefit for
developing advertising campaigns (Jolson and Rossow, 1971). Other areas where researchers
began to experiment with the method include medical science (Milholland et al., 1973;Travis,
1973), information systems (Turoff, 1971;Grabbe and Pyke, 1972), and computer and
information science (Firschein et al., 1973;Nanus et al., 1973). One interesting aspect of the
Delphi method –being a method that is often used to forecast the future –is that its usefulness
can, in a sense, be examined by reevaluating these early studies conducted over four decades
ago. Generally, these studies show mixed results. For instance, in Firschein et al.’s study of
artificial intelligence, there were correct predictions about the development of search engines,
industrial robotics, and diagnostics for machinery during the 1980s and 1990s, but overly
optimistic responses in regard to when robot tutor (1988), automatic language translator
(1995), computer psychiatrist (2000), robot chauffeur (2000), and animal symbionts (2010)
would be commercially available.
A quintessential methodological book for Delphi is Linstone and Turoff’s (1975) The
Delphi method: Techniques and applications. This book was a major step in the legitimization
of Delphi as a scholarly research method. It outlined the philosophy, aim, and steps in the
Delphi method. Following the publication of this book (which has been cited nearly 10,000
times, according to the Web of Science database), research involving the Delphi method has
continued to grow substantially (Rowe and Wright, 2011). Today, the method is used in
virtually all disciplines, and is considered to have an acceptable level of validity for
performing scholarly research (Landeta, 2006;Lilja et al., 2011;Tomasik, 2010;Worrell et al.,
2013). The basic elements of the Delphi method, as described by Dalkey and Helmer during its
early development (1963, p. 458), include (1) repeated individual questioning of a panel of
experts, (2) anonymity among the panel of experts, and (3) interspersed opinion feedback. As
the method emerged from the RAND Corporation and was adopted by an increasingly large
group of researchers, the method became more formalized, and variations began to emerge.
Description and philosophy of the Delphi method
As noted by Aichholzer (2009, p. 253), “a major justification for the validity of the Delphi
method derives from the ‘theory of errors,’according to which the aggregated group
responses can be expected to represent a statement that is superior to the majority of the
individual experts’ones.”Another way to state this theory is Aristotle’s adage, “the whole is
greater than the sum of its parts.”Even experts may misjudge what aspects of a topic are
important, or have ideas that are not “the best,”but when considered as a collective, the best
ideas tend to be elevated.
According to Linstone and Turoff (1975), the Delphi method can be viewed as a structured
communication process from which the intended outcome is a consensus to a complex (even
ambiguous) problem (p. 3). Participants in a Delphi study are traditionally selected based on
some level of expertise relevant to the research questions of the study. The Delphi method
itself is “an iterative process to collect and distill the anonymous judgements of experts using
a series of data collection and analysis techniques interspersed with feedback”(Skulmoski
JD
76,4
930
et al., 2007, p. 1). In the ideal administration of the Delphi method, this process results in a
consensus among the respondents, whereby in the last iteration, only very precise and
popular feedback is collected, giving a clear, expert response to the research questions. A full
depiction of the Delphi process is shown in Figure 1.
As noted by Hasson and Keeney (2011), there are at least ten commonly used types of
Delphi designs:classical, used to forecast and gatheropinion; modified, similar to classical,but
generally fewer rounds; decision, used to inform immediate decision-making; policy, used to
guide policydevelopment; real-time, similar to focusgroup, but with tabulation of results after
each round; e-Delphi, a Delphi study facilitated by emailor online survey; technological, real-
time onlineDelphi, similar to real-time; online,Delphi performed in the setting of a chat roomor
discussion board; argument, where participants are purposely selected to represent opposite
sides of an issue; and disaggregative policy, where participants are asked to speculate on
future scenarios. Many other less-common types and variations of the Delphi method exist.
Selection of experts
Careful selection of experts for the Delphi study is immensely important. These individuals
should not be selected based on relationships to the researcher (Avella, 2016;Hasson et al.,2000).
Nor should they necessarily be the most visible members of a particular field (e.g. in a study
about library instruction, the former presidents of ALA or CILIP are not necessarily experts,
even though they are visible figures within the field of librarianship). Rather, the experts in a
Delphi study should be those individuals with the most intimate knowledge and experience with
the topic (Baker et al.,2006;Welty, 1972). In a study of library instruction, this likely means
actual library instructors. These individuals, even though they may not conduct substantial
research or publicly speak about or represent library instruction, work as instructional
librarians every day, which gives them an insider’s perspective on the topic. Often, these
individuals are identified based on their contributions to the literature within a field (Gordon,
1994). For a study of instructional librarianship topics, the researcher may conduct a literature
search for contributions on this topic and refer to top journals in this field (e.g. College and
Research Libraries,Journal of Academic Librarianship), and identify a pool of potential
participants to contact based on the authors making significant contributions to the field.
Assumptions for Delphi research
Assumptions made by researchers who use the Delphi method include that the ideas
generated by experts are necessarily the best ideas, that the researcher can conduct this
research without introducing problematic bias into the iterative process, and that sufficient
response rate can be attained for this type of study (Hsu and Sandford, 2007). There is some
reason to believe experts may not necessarily produce the best new ideas (Welty, 1972). As
found in industry and entertainment, often an outsider’s perspective can actually produce the
most useful new ideas. However, identifying potential outsiders that would be capable of
1. Identify Research Problem
2. Develop Research Questions
3. Select the “Experts” (Participants in the Study)
4. Administer the First Round of the Study (Questionnaire)
5. Synthesize Responses from First Round of the Study
6. Present Respondents with the Synthesized Responses
7. Administer the Second Round of the Study
8. Synthesize Responses from Second Round of the Study
9. Present Respondents with the Synthesized Responses
10. Continue this Process Until a Pre-Prescribed Number of Rounds Has
Been Reached, or the Respondents Appear to Have Reached a Consensus
11. Analyze the Responses from Each Round; Disseminate Findings!
Figure 1.
Steps in the classical
Delphi method
Review of the
Delphi method
931
To continue reading
Request your trial