Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies
| Date | 19 December 2022 |
| Pages | 104-131 |
| DOI | https://doi.org/10.1108/AJIM-05-2022-0253 |
| Published date | 19 December 2022 |
| Subject Matter | Library & information science,Information behaviour & retrieval,Information & knowledge management,Information management & governance,Information management |
| Author | Tianji Jiang,Jiqun Liu |
Reflection on future directions:
a systematic review of reported
limitations and solutions
in interactive information
retrieval user studies
Tianji Jiang
Department of Information Studies, School of Education and Information Studies,
University of California Los Angeles, Los Angeles, California, USA, and
Jiqun Liu
School of Library and Information Studies, University of Oklahoma, Norman,
Oklahoma, USA
Abstract
Purpose –The purpose of this paper is to understand how users behave and evaluate how systems with users
are essential for interactive information retrieval (IIR) research. User study methodology serves as a primary
approach to answering IIR research questions. In additionto designing user study procedures, understanding
the limitations of varyingstudy designs and discussing solutions to the limitations is also critical for improving
the methods and advancing the knowledge in IIR.
Design/methodology/approach–Given this unresolved gap, we apply the faceted framework developed by
Liu and Shah (2019) in systematically reviewing 131 IIR user studies recently published (2016–2021) on
multiple IR and information science venues.
Findings –Our study achieve three goals: (1) extracting and synthesizing the reported limitations on multiple
aspects of user study (e.g. recruitment, tasks, study procedures, system interfaces, data analysis methods)
under associated facets; (2) summarizing the reported solutions to the limitations; (3) clarifying the connections
between types of limitations and types of solutions.
Practical implications –The bibliography of user studies can be used by students and junior researchers
who are new to user-centered IR studies as references for study design. Our results can facilitate the reflection
and improvement on IR research methodology and serve as a checklist for evaluating customized IIR user
studies in varying problem spaces.
Originality/value –To our knowledge, this work is the first study that systematically reviews the study
limitations and solutions reported by IIR researchers in articlesand empirically examines their connections to
different study components.
Keywords Meta-analysis, Systematic review, User study, Interactive information retrieval, Study limitation,
Users and interactive retrieval
Paper type Research paper
1. Introduction
Understanding people’s information seeking behaviors is a central theme to interactive
information retrieval (IIR) studies. In spite of some widely acknowledged limitations, such as
substandard design components, inadequate sample size and uncontrolled research settings,
user study is still one of the most effective methods for observing users and measuring search
activities and experiences. User study, as a method in information retrieval studies, refers to
the study that involves human participants to understand human information seeking
AJIM
76,1
104
This work is partially supported by a Faculty Investment Program grant from the Research Council of
the University of Oklahoma Norman Campus. The authors would like to thank Julie Torgerson and
Shuyu Lu for their work on the initial-round paper annotation.
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/2050-3806.htm
Received 18 May 2022
Revised 4 August 2022
5 October 2022
Accepted 22 October 2022
Aslib Journal of Information
Management
Vol. 76 No. 1, 2024
pp. 104-131
© Emerald Publishing Limited
2050-3806
DOI 10.1108/AJIM-05-2022-0253
behaviors or user-oriented s earch systems. It focuses on rec ording, analyzing and
understanding participants’motivations, information needs, information seeking
behaviors and their interactions with search systems through observation techniques, task
analysis and other feedback methodologies. According to the cognitive viewpoint of
information retrieval (IR) research, information search and the process of retrieval is
essentially interactive, thus users should not be abstracted out of IR system evaluation
(Belkin, 1990;Ingwersen, 1996). For that reason, as an approach focusing on users and user-
system interactions, user study is a critical part of IIR research. Indeed, it’s the only way to
discover exactly what these users expect, think and need, addressing a variety of research
questions that are directly or indirectly related to search system design and evaluation
(Liu and Shah, 2019) and bridge the gap between the kind of information services needed and
the kind of services in existence (Satyanarayana, 2017). Moreover, it can remove bias by
learning about the users from their experiences or perspectives and help to ground design
decisions based on a good grip on users’needs (Yang, 2019). Since the emergence of IIR
research, user study has been widely adopted in the research community. Surveys, user
experiments and observations are conducted to frame information retrieval behaviors,
formulate people’s interactions with IR systems or evaluate IR systems.
Although user study plays a critical role in IIR studies, the limitations should be
acknowledged. User study requires strict restrictions on research setting and process and
needs to be well planned from the formulation of research questions to the presentation of
results. It can induce a deficiency in the research design, data analysis and conclusions if the
method is not conducted in the proper process, within the appropriate participants, or under
proper circumstances. Even though it has been well designed and carefully conducted, some
of these studies still proved to be inadequate to comprehensively reveal the real needs of
information users, draw a complete picture of information retrieval behaviors, or sort out the
whole process of user-system interactions (Satyanarayana, 2017). The lack of reflection and
knowledge on user study limitations may also negatively affect the reusability of the research
resources (e.g. data, experimental design and systems, study procedure, interview protocols)
generated through customized, expensive data collection processes (Liu, 2022).
A considerable number of studies in IIR have been conducted to elaborate the advantages
of user study and identify the issues that have to be concerned in research design (e.g. Kelly,
2009;Liu and Shah, 2019). There are also a few studies that investigate the process, steps and
techniques through reviewing well-prepared and clearly articulated user study articles,
which assist inexperienced scholars to grasp the method. However, there is little scholarship
that critically examines the previous literature to see the limitations related to user study and
how the authors address the limitations, which is essential for better understanding the value
of user study and exploring the generalizability and reusability of outcomes found by the
method. As we already have a large amount of induvial user studies, researchers are
expecting an overview to summarize the reported limitations and solutions in the studies.
Moreover, revealing, understanding and learning the limitations and solutions can benefit the
future researchers in this field and enhance their research design and evaluation.
This study aims to identify the reported limitations of user studies based on a systematic
review and meta-analysis on recently-published IIR user studies. Specifically, our work seeks
to: (1) identify and summarize the limitations/issues that are frequently reported in published
user studies, especially those which may significantly alter the results of the study;
(2) identify and summarize the authors’responses to the reported limitations, including both
solutions to the limitations or explanations of why they believe that the limitations are not big
issues; (3) explore the correlations between the reported limitations and solutions in published
user studies (4) explore the correlations between research limitations and any study features,
which may be research topic, research settings, participants or task; and (5) synthesize
findings from varying study scenarios to find out what the future IIR researchers should
Interactive
information
retrieval user
studies
105
Get this document and AI-powered insights with a free trial of vLex and Vincent AI
Get Started for FreeStart Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting