The role of cognitive validity testing in the development of CALL, the comprehensive assessment of leadership for learning
| Published date | 29 April 2014 |
| Date | 29 April 2014 |
| Pages | 358-378 |
| DOI | https://doi.org/10.1108/JEA-01-2013-0008 |
| Author | Mark H. Blitz,Jason Salisbury,Carolyn Kelley |
| Subject Matter | Education,Administration & policy in education,School administration/policy |
The role of cognitive validity
testing in the development of
CALL, the comprehensive
assessment of leadership
for learning
Mark H. Blitz, Jason Salisbury and Carolyn Kelley
Educational Leadership and Policy Analysis,
University of Wisconsin-Madison, Madison, Wisconsin, USA
Abstract
Purpose – The Comprehensive Assessment of Leadership for Learning (CALL) is an online
task-based assessment of distributed instructional leadership. In developing CALL, researchers faced
the challenge of structuring survey items that would measure leadership practice rather than
individual traits. Critical in this work was developing items that accurately reflected current
leadership practice. Therefore, the purpose of this paper is to pilot the CALL instrument and
conducted cognitive validity testing on the instrument.
Design/methodology/approach – CALL researchers piloted the survey in six schools in Wisconsin
in order to test and refine the survey instrument. Researchers conducted cognitive walk-through
interviews with five participants from each school: principals, associate principals, teachers,
department chairs, guidance counselors, and activities directors. The interviews focussed on specific
items in order to observe the users’ thought processes and rationale in choosing a response for each
item. The researchers focussed on relevancy, clarity, and accuracy of survey items in collecting and
analyzing the resulting data.
Findings – Three specific survey items were identified that exemplify these challenges and
opportunities such as: accessible language, extended leadership, socially desirable responding,
360-degree perspectives, applying appropriate terminology, and identifying appropriate practices.
These findings provided insight into survey development work and implications of distributed
leadership. The authors discuss the challenges of creating a task-based leadership assessment.
Originality/value – Developing a formative assessment of school leadership is valuable in
supporting school leaders’ work. The process of utilizing a qualitative approach to develop a
quantitative instrument has proven critical in measuring task-based distributed leadership.
Keywords Leadership, Assessment, Organizational development, Methodology, Cognitive validity
Paper type Research paper
Educators operate in an age of assessment; consequently, researchers and policy
makers work to develop assessments that accurately measu re school success. A critical
charge for developers of these assessments is to create tools that prove useful for the
practitioners being assessed. Cognitive validity testing is one process to ensure a
particular assessment is relevant to school-w ide practice. This paper discusses the
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/0957-8234.htm
Received 15 January 2013
Revised 22 July 2013
28 August 2013
Accepted 8 September 2013
Journal of Educational
Administration
Vol. 52 No. 3, 2014
pp. 358-378
rEmeraldGroup PublishingLimited
0957-8234
DOI 10.1108/JEA-01-2013-0008
The research reported was supported by the US Department of Education Institute of Education
Sciences (Award R305A090265) and by the Wisconsin Center for Education Research, School of
Education, University of Wisconsin-Madison. Any opinions, findings, or conclusions expressed
in this paper are those of the author and do not necessarily reflect the views of the funding
agencies, WCER, or cooperating institutions.
358
JEA
52,3
importance of cognitive validity testing in the development of Comprehensive
Assessment of Leadership for Learning (CALL). Funded by the Institute for
Educational Sciences (IES), CALL specifically focusses on using technolo gy to support
data-driven instructional leadership. Data-driven instructional le adership has become
commonplace for school leaders, which has led researchers to examine the type of data
leaders use. Student test scores and trait-based surveys illuminate areas of strength
and weakness in schools, but school le aders do not have information on the work
behind making improvements in those areas (Anderson et al., 2010). CALL is an
online formative assessment of school leadership designed to examine school-wide
leadership tasks, and it uses a task-based distributed leadership framewo rk to examine
leadership practice across a school. The CALL assessment tool measures leadership
practice in five core domains:
(1) focus on learning;
(2) monitoring teaching and learning;
(3) building nested learning communities;
(4) acquiring and allocating resources; and
(5) maintaining a safe and effective learning environment.
Within each of these core domains exist four or five sub-d omains. The CALL
survey items are situated within these sub-domains and inquire about specific
leadership practices.
While the intent of developing a formative feedback assessment on distributed
instructional leadership has been a noble one, the challenges of developing such a
tool move beyond correlating items and identifying standards. Most leadership
evaluation tools assess the performance of an individual principal by asking
respondents to rate the principal in general areas of leadership practice (Condon and
Clifford, 2010). For a prime example of an effective principal evaluation tool, one
could look to the Vanderbilt Assessment of Leadership in Education (VAL-ED),
which is comprised of a 360-degree survey in which a school’s teachers, principal,
and district supervisor respond to questions concerning VAL-ED’s core components
and key processes (Goldring et al., 2009; Murphy et al., 2007). VAL-ED’s solid
research base and high reliability (Porter et al., 2010) have contributed to the tool’s
sound reputation. To be sure, there is value in assessing an individual principal
by focussing on that person’s effectiveness and the traits that contribute to that
effectiveness. CALL utilizes an alternative approach to assessingleadership. The CALL
instrument measures research-based leadership tasks or practices, thereby shifting the
focus from the individual to the work of leaders across the school (Kelley et al., 2012;
Blitz, 2012). In CALL, respondents rate the effectiveness of leadership practices and
rarely respond to questions about the individual principal.
Since CALL is task-based rather than individual- or trait-based, the primary work
of developing this tool involved creating survey items that capture leadership practices
that are relevant to current school contexts as well as present constr ucts that
accurately reflect the work done in schools. Rather than inquire about general domains
of practice such as budgeting, teacher evaluation, or student learning, the CALL
constructs delve more deeply into leadership practice, often inquiring about the wo rk
of teachers in the classroom. Therefore, the challenge for the CALL researchers and
developers was to create survey items designed to capture specific practices while at
359
Cognitive
validity
testing
To continue reading
Request your trial