Design and development of rubrics to improve assessment outcomes. A pilot study in a Master's level business program in India
Date | 01 February 2011 |
Published date | 01 February 2011 |
Pages | 84-104 |
DOI | https://doi.org/10.1108/09684881111107771 |
Author | Malini Y. Reddy |
Subject Matter | Education |
Design and development of
rubrics to improve assessment
outcomes
A pilot study in a Master’s level business
program in India
Malini Y. Reddy
Department of Marketing and Strategy, IBS Hyderabad,
Icfai Foundation for Higher Education, Andhra Pradesh, India
Abstract
Purpose – This paper seeks to discuss the characteristics that describe a rubric. It aims to propose a
systematic method for developing curriculum wide rubrics and to discuss their potential utility for
program quality assessment.
Design/methodology/approach – Implementation of rubrics is a recent phenomenon in higher
education. Prior research and theoretical issues related to rubric design and development are
discussed. The proposed method for rubric development is illustrated by deriving generic analytic
scoring rubrics for two assessment methods, namely projects and cases in a Master’s level business
program in India. Aspects related to the validity of the rubrics developed are investigated and results
of reliability study conducted using intraclass correlation coefficient (ICC) are reported.
Findings – Upon testing, the rubrics were found to be reliable and valid grading tools. Results of
inter-rater and intra-rater reliability analyses illustrated that the rubrics supported standardization of
the manner in which cases and projects could be evaluated in different business courses.
Practical implications – Whereas previous studies on rubric construction have largely
concentrated on task specific rubrics, this study focuses on development of curriculum wide rubrics
that can be employed for assessment of students’ learning at both course and program level.
Originality/value – To date there has not been any published work on issues of assessment of
student learning through project and case analysis rubrics within diverse courses in a business
program. The method detailed in the study can guide the development of generic rubrics for
alternative assessment methods employed in business programs as well as in other disciplines.
Keywords Assessment, Learning,Higher education, Business schools,Students, India
Paper type Research paper
Introduction
A “rubric” in education literature is commonly understood as an assessment tool that is
used to describe and score observable qualitative differences in performances. It
captures the essence of performance in academic tasks by “listing the criteria, of what
counts, and describing levels of quality from excellent to poor” (Andrade and Du, 2005,
p. 1). In assessment methods such as projects, case analysis, essays, portfolio, where
the constructed responses given by students cannot be evaluated with complete
objectivity, rubrics are considered an effective approach for achieving reliable
(consistent) and valid (accurate) professional judgment of students’ performances
(Pellegrino et al., 1999).
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/0968-4883.htm
QAE
19,1
84
Received January 2010
Revised July 2010
Accepted November 2010
Quality Assurance in Education
Vol. 19 No. 1, 2011
pp. 84-104
qEmerald Group Publishing Limited
0968-4883
DOI 10.1108/09684881111107771
The development and use of rubrics for assessment of learning outcomes has
become a popular and recognizable trend since the early 1990s. While they have been
extensively used at the school level in the USA, their application in higher education, as
well as their usage in other countries has gathered momentum only in the last one
decade (Montgomery, 2002). Several business schools in the USA are currently using
rubrics for assessment of students’ learning at course and program level[1]. This
interest in use of rubrics in business programs is growing rapidly in other countries as
well (Martell and Calderon, 2005). The departments of education and quality assurance
agencies in several countries (e.g. the USA, the United Kingdom, Australia, France,
Turkey) as well as globally recognized accreditation bodies (e.g. AACSB, EFMD,
AMBA) which have adopted assessment of learning outcomes as the cornerstone of
their assurance and accreditation process, have contributed to the growing awa reness
and importance of rubrics in business education. They have been proposed as a viable
means for scrutinizing actual samples of student work produced in the programs for
assessing and assuring the attainment of the specified learning outcomes.
Having said this, it is puzzling to note the miniscule literature on the development and
use of rubrics in business programs. A majority of the limited samples of rubrics designed
for use in post secondary level business courses are task specific. There are no published
studies on development and use of a rubric for an assessment method such that it could be
easily implemented for different assignments/tasks in a variety of business courses (hence
forth, referred as curriculum wide generic rubrics). This paper, attempts to fill this gap by
demonstrating a systematic process for developing rubrics for two assessment methods,
namely, written analysis of cases and course based projects. The process illustrated can
be used for developing rubrics that are generic enough to suitably serve a variety of
assignment objectives in a course; can be implemented in different courses across
academic terms; and can possibly be applicable at multiple institutions.
Literature on rubric design and development
Several educational researchers have studied aspects of rubric design and have made
recommendations for their appropriate construction. Literature presents two equally
relevant approaches:
(1) development from the perspective of describing the essentia l features of rubrics
(e.g. Huba and Freed, 2000; Arter and McTighe, 2001); and
(2) development by outlining a serie s of discrete steps to follow one after the other
(e.g. Moskal, 2000; Mertler, 2001).
Studying and synthesizing these approaches is essential for a thorough understanding
of the process of rubric construction.
A rubric has three essential features namely, evaluation criteria, quality definitions
and scoring strategy (Popham, 1997). Evaluation criteria are the factors that an
assessor considers when determining the quality of a student’s work. These bring in
the much needed clarity in understanding the requirements of performance especially
in areas which are difficult to define, such as critical thinking, problem solving, team
skill. These also permit criterion-referenced discrimination of performances and enable
monitoring of students learning against each criterion. The criteria thereby pro vide the
students with opportunities to reflect on both the content of their performance as well
as process of learning (Parke, 2001).
Describing the development of a rubric to assess student learning as a multi-step
approach, the Council for Higher Education Accreditation (CHEA) recommends
Design and
development of
rubrics
85
To continue reading
Request your trial