Virtually No Effect? Different Uses of Classroom Computers and their Effect on Student Achievement
| Author | Oliver Falck,Ludger Woessmann,Constantin Mang |
| Date | 01 February 2018 |
| Published date | 01 February 2018 |
| DOI | http://doi.org/10.1111/obes.12192 |
1
©2017 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd.
OXFORD BULLETIN OF ECONOMICSAND STATISTICS, 80, 1 (2018) 0305–9049
doi: 10.1111/obes.12192
Virtually No Effect? Different Uses of Classroom
Computers and their Effect on Student
Achievement*
Oliver Falck, Constantin Mang and Ludger Woessmann
Ifo Institute at the University of Munich, (e-mails: falck@ifo.de, constantin@mangmail.de
and woessmann@ifo.de.)
Abstract
Most studies find little to no effect of classroom computers on student achievement. We
suggest that this null effect may combine positive effects of computer uses without equiv-
alently effective alternative traditional teaching practices and negative effects of uses that
substitute more effective teaching practices. Our correlated random effects models exploit
within-student between-subject variation in different computer uses in the international
TIMSS test. We find positive effects of using computers to look up information and neg-
ative effects of using computers to practice skills, resulting in overall null effects. Effects
are larger for students with high socioeconomic status and mostly confined to developed
countries.
I. Introduction
The use of computer-based teaching methods and virtual learning technologies in the class-
room has raised high expectations to improveeducational achievement (e.g. Peterson, 2010;
Economist, 2013). These methods are often seen as the biggest technologyshift in decades,
if not in centuries, set to revolutionize the traditional teacher-centric lecturing style and
to unleash the potential for improvements in teaching quality and efficiency. However,
the empirical evidence on the effects of computers on student achievement has been dis-
appointing, mostly finding no effects (Bulman and Fairlie, 2016).This paper suggests that
such null effects may be the result of a combination of using computers for activities that
are more productive than traditional teaching methods, thus improving student outcomes,
and using computers in ways that substitute more effective traditional practices, thus low-
ering student outcomes. Our evidence shows that using computers to look up ideas and
JEL Classification numbers: I21, I28.
*For helpful comments, we wouldlike to thank two anonymous referees, Eric Bettinger, Mat Chingos, Tom Dee,
Rob Fairlie, David Figlio, and seminar participants at Stanford University, the London School of Economics, the
Universityof Zurich, Humboldt University Berlin, the University of Passau, IZA Bonn, the Ifo Institute, the European
Economic Association, the German Economic Association, the CESifo area conference in economics of education,
the IIIrd ICT Conference Munich, and the Digital Economy Workshop, Collegio CarloAlber to,Turin. Woessmannis
grateful to the Hoover Institution at Stanford University for its hospitality during workon this paper.
2Bulletin
information indeed improves student achievement, but using computers to practice skills
reduces student achievement.
The central point in our reasoning is that there are opportunity costs of time. Every
classroom minute can be used for one activity or another. Thus, if the time spent on com-
puters is increased,it substitutes different alternative time uses. On the one hand, computers
can be used for specific applications, such as exploring new ideas and information on the
Internet, that do not have comparably effective alternatives in the traditional world. If these
computer uses substitute less effective uses of classroom time, student learning will in-
crease. On the other hand, computers can be used for more traditional applications, such
as practicing skills, that have potentially more effective conventional teaching alternatives.
If these are crowded out, student learning will decrease. Thus, the net effect of computer
use depends on the specific activities that they are used for and the relative effectiveness
of the activities that they crowd out.An overall null effect of computer use may be the sum
of positive and negative effects.
We test this hypothesis using information on the specific uses of computers in the
classroom in the Trends in International Mathematics and Science Study (TIMSS). Our
sample of the 2011 TIMSS test coversthe maths and science achievement of over 150,000
students in 30 countries in 8th grade and nearly 250,000 students in 53 countries in 4th
grade. In detailed background questionnaires, TIMSS surveys how often teachers in each
subject have their students use computers in three distinct activities: look up ideas and
information; practice skills and procedures; and (only in 8th grade) process and analyse
data. Apart from enabling an analysis of different types of computer use, the international
character of the TIMSS data allows us to test whether any effect is context-specific or
generalizes across different settings.
Our identification strategy exploits the two-subject structure of the TIMSS data. It is
hard to imagine a field experiment that would assign different types of computer use ran-
domly across classrooms, not least because of teacher resistance. But in observational data,
it is not random which students and classrooms use computers. Forexample, the availability
of computers in a school is likely related to the socioeconomic status of the neighbourhood,
and teachers may choose to use computers based on students’achievement levels. To avoid
bias from non-random selection of students into specific schools or classrooms, our em-
pirical model identifies from variation in computer use across subjects within individual
students. This between-subject variationallows us to estimate within-student effects, hold-
ing subject-invariantunobser vedschool and student characteristics constant. We generalize
between-subject models with student fixed effectsthat assume the same effect of computer
use on student achievement in both subjects (e.g. Dee, 2005, 2007; Lavy, 2015) to corre-
lated random effects models with subject-specific effects (Metzler and Woessmann, 2012),
which prove empirically relevant in our setting. To address non-random computer choices
by different teachers, we draw on the rich TIMSS background information on teachers and
their teaching methods. To further rule out bias from unobserved teacher characteristics or
non-random selection of teachers into computer use, we also identify from between-subject
variation within the same teacher when restricting our 4th-grade analysis to a sample of
students taught by the same teacher in both subjects.
In line with most of the literature, on average we do not find a significant effect of
computer use on student achievement. But this null effect is the combination of positive
©2017 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd
Virtually no effect? 3
and negative effects of specific computer uses: Using computers to look up ideas and
information has a positive effect, whereasusing computers to practice skills and procedures
has a negative effect(and using computers to process and analyse data has no effect). In 8th
grade – which is the main focus of our analysis as computer use should be mature by this
stage – this pattern is evident in science but not in maths. Interestingly, we find the same
pattern of opposing use-specific effects in 4th grade, but there it is strongest in maths. This
might indicate that the positive effect of using computers to look up ideas and information
is particularly pertinent in the explorative stages of a subject matter.
In terms of effect sizes, going from no to daily computer use for looking up ideas and
information increases 8th-grade science achievement by 10–13% (depending on teaching
methods controls) of a standard deviation, but it reduces achievement by 7–11% of a
standard deviation when used to practice skills and procedures. Such effect sizes may be
viewedas modest to large. For example, as a general rule of thumb, average student learning
in a year is about one-quarter to one-third of a standard deviation (Woessmann, 2016). In
terms of other inputs, Lavy (2015) finds an effect size of 6% of a standard deviation for a
1-hour increase in instruction time per week and Metzler and Woessmann (2012) find that
a one standard deviation increase in subject-specific teacher achievement increases student
maths achievement by about 9% of a standard deviation. Examples of effect sizes of other
educational interventions easily range from zero for class-size effects in many countries
(e.g. Woessmann and West, 2006; Altinok and Kingdon, 2012) to, for example, 40% of a
standard deviation for specific charter school interventions (Abdulkadiro˘glu et al., 2011).
Looking across countries, results are strongest among OECD countries and mostly
insignificant in less developed countries. There are no systematic differences along other
country dimensions such as broadband access or size of the country, indicating that general
Internet familiarity and the size of the software market do not seem to be crucial. Results
also do not differ systematically by gender or by individual levels of achievement or com-
puter acquaintance, indicating that effects do not depend on individual competencies.
However, effects are less pronounced for students from low socioeconomic background.
The patterns suggest that results are mostly a general feature of specific computer uses.
Results are also robust in the within-teacher specification in 4th grade.
Our results can help reconcile some of the diverging findings in the literature. Most
studies of computer use in school find little to no effect of classroom computers on stu-
dent achievement, in particular when looking at investment in computer technologies in
general.1But there are exceptions of studies finding significant positive effects of specific
computer-assisted instruction programmes,2and in all these cases, there are indications
that computers are being put to more effective uses in the sense of our framework (see
section ‘Conceptualizing the mixed existing evidence’ for details). Our result that effects
of classroom computers differ by their specific use also relate to the recent literature on
computers at home which emphasizes that home computers can be put to conducive uses
such as schoolwork as well as detrimental uses such as gaming or entertainment (Fairlie
and London, 2012; Fairlie and Robinson, 2013; Faber, Sanchis-Guarner and Weinhardt,
1For example, Angrist and Lavy (2002), Rouse and Krueger (2004), Goolsbee and Guryan (2006) and Leuven
et al. (2007); see Bulman and Fairlie (2016) for a review.
2See Machin, McNally and Silva (2007), Banerjee et al. (2007) and Barrow, Markman and Rouse (2009).
©2017 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd
To continue reading
Request your trial