Estimating student ability and problem difficulty using item response theory (IRT) and TrueSkill
| Published date | 20 May 2019 |
| Pages | 67-75 |
| Date | 20 May 2019 |
| DOI | https://doi.org/10.1108/IDD-08-2018-0030 |
| Author | Youngjin Lee |
| Subject Matter | Library & information science,Library & information services,Lending,Document delivery,Collection building & management,Stock revision,Consortia |
Estimating student ability and problem
difficulty using item response theory (IRT)
and TrueSkill
Youngjin Lee
University of Kansas, Lawrence, Kansas, USA and University of North Texas, Denton, Texas, USA
Abstract
Purpose –The purpose of this paper is to investigate an efficient means of estimating the ability of students solving problems in the computer-
based learning environment.
Design/methodology/approach –Item response theory (IRT) and TrueSkill were applied to simulated and real problem solving data to estimate
the ability of students solving homework problems in the massive open online course (MOOC). Based on the estimated ability, data mining models
predicting whether students can correctly solve homework and quiz problems in the MOOC were developed. The predictive power of IRT- and
TrueSkill-based data mining models was compared in terms of Area Under the receiver operating characteristic Curve.
Findings –The correlation between students’ability estimated from IRT and TrueSkill was strong. In addition, IRT- and TrueSkill-based da ta mining
models showed a comparable predictive power when the data included a large number of students. While IRT failed t o estimate students’ability and
could not predict their problem solving performance when the data included a small number of students, TrueSkill did not experience such problems.
Originality/value –Estimating students’ability is critical to determine the most appropriate time for providing instructional scaffolding in the
computer-based learning environment. The findings of this study suggest that TrueSkill can be an efficient means for estimating the ability of
students solving problems in the computer-based learning environment regardless of the number of students.
Keywords Problem solving, User modeling, Prediction model, Educational data mining (EDM), Log file analysis, Learning analytics (LA)
Paper type Research paper
Introduction
Recently, US Department of Education emphasized the
importance of developing computer-based learning
environments that can provide customized learning contents
tailored to the ability of students (Bienkowski et al., 2012). It
is anticipated that such adaptive learning environments can
maximize the learning outcome of students because students
can be engaged in personalized learning activities matching
their level of understanding (Tanenbaum et al., 2013). To
develop an adaptive learning environment, it is essential to
accurately estimate the ability of students as they are engaged
in various learning activities. Typically, computer-based
learning environments estimate the ability of students, which
is changing over time as a result of their learning, by having
students solve a series of problems. The estimated ability of
students can then be used to provide differentiated learning
experiences.
The simplest way to estimate the ability of students solvinga
series of problems is to count the numberof correct answers or
to compute the fraction of correct answers submitted by
students. Because of its simplicity, this approach is frequently
used in many computer-based learning environments such as
massive open online courses (MOOCs); students receive
instructional supports and guidance when they submit an
incorrect answer a certain number of times. However, the
heuristics like this are unlikely to maximize the learning
outcome of students because theydo not take into account the
difficulty of problems and the ability of students. When the
problem is difficult, it makes sense to allowmore opportunities
before providing instructional supports. Likewise, we do not
want to postpone providing help to academically weaker
students because they are likely to get frustrated, fail the
learning task and may giveup their learning entirely unless they
receive instructionalsupports and guidance in time. Moreover,
the effectiveness of such heuristics has not been thoroughly
investigatedin the computer-based learning environment.
Item response theory (IRT) is an approach that can address
the shortcomings of simple count or fraction of correctanswers
in estimating the ability of students solving problems in the
computer-based learning environment (Baylari and Montazer,
2009;Chen et al., 2005). IRT assumes that the ability of
students does not change while taking a test and each problem
is independent of other problems in the same test. Under these
assumptions, IRT can estimate the ability of students and the
difficulty of problems that are invariant to students and
problems being used in estimation(Ayala, 2009). As IRT takes
into account both the ability of students and the difficulty of
problems, solving more difficult problems is treated differently
The current issue and full text archive of this journal is available on
Emerald Insight at: www.emeraldinsight.com/2398-6247.htm
Information Discovery and Delivery
47/2 (2019) 67–75
© Emerald Publishing Limited [ISSN 2398-6247]
[DOI 10.1108/IDD-08-2018-0030]
Received 16 August 2018
Revised 20 November 2018
22 December 2018
Accepted 28 December 2018
67
To continue reading
Request your trial