Context and Culture in Evaluation: A Case Study of Evaluation Anthropology
| Date | 01 March 2017 |
| Published date | 01 March 2017 |
| Author | Elise Howard |
| DOI | 10.1177/1035719X1701700105 |
| Subject Matter | Academic Article |
A C A D E M I C A R T I C L E Evaluation Journal of Australasia Vol 17 | No 1 | 2017 | p p . 3 0 – 3 8
eLiSe howaRd
Context and culture in evaluation:
a case study of evaluation anthropology
Designing programs to address poverty and inequality
for Australian Aboriginal communities over recent
decades has proved problematic. There is a need for
greater consideration of different cultural perspectives.
A culturally appropriate evaluation framework can
provide a range of strategies to embrace cultural
difference. Evaluation anthropology, one of many
culturally appropriate approaches, emphasises
understanding of socio-cultural environments and
contexts, and reflective practice to draw attention
to cultural bias. This paper will define evaluation
anthropology and then reflect on its usefulness in
establishing an evaluation framework for a preliteracy
program located in a remote Aboriginal community
in Australia. The aims of the program are to improve
school readiness through developing preliteracy
Elise Howard is a Research Project Manager,
(English language) skills in children aged 0–3 years.
College of Arts, Society and Education at James
Developing an evaluation framework for the program
Cook University, Townsville, Queensland.
required an approach that accounted for the socio-
Email: elise.howard@jcu.edu.au
cultural aspects of literacy development. The lessons
from this case study demonstrate the need for
place-specific theory to inform program design and
evaluation practice.
introduction
Efforts to reduce inequity between Indigenous and non-Indigenous
Australia in health, education and employment outcomes continue
to show little progress. The most recent reports Closing the
gap (Department of the Prime Minister and Cabinet, 2016) and
Overcoming Indigenous disadvantage (Productivity Commission, 2016)
both acknowledge that while some improvements in wellbeing measures
have occurred, meeting overall targets ‘remains a significant challenge’
30
E v a l u a t i o n J o u r n a l o f A u s t r a l a s i a Vo l 1 7 | N o 1 | 2 0 1 7
A C A D E M I C A R T I C L E
(Department of the Prime Minister and Cabinet, 2016,
practices between evaluators and community members
p. 5). Further to this, Aboriginal and Torres Strait
(Chourinard & Cousins, 2009; Mertens & Wilson, 2012;
Islander peoples located in remote areas are more likely
Rossingh & Ynunpingu, 2016). Approaches informed
to experience greater disadvantage than their regional or
by Indigenist research philosophy and methodology
urban counterparts (Department of the Prime Minister
emphasise the importance of constructing knowledge
and Cabinet, 2016; Productivity Commission, 2016).
from participants’ perspectives and data collection
Productivity Commission Deputy Chair, Karen Hester,
techniques that are more suited to community value
recently made a call for greater attention to strength based
systems and practices, such as storytelling, oral histories
approaches and the importance of evaluation in assisting
and pathway mapping (Chilisia, 2012; Hopson, Kirkhart
to identify programs that work, ‘but the overwhelming
& Bledsloe, 2012; Pipi, 2010; Dart & Davies, 2003).
lack of robust, public evaluation of programs highlights
Within the field of culturally appropriate evaluation,
the imperative for Indigenous policy evaluation’
evaluation anthropology aims to bring the techniques
(Productivity Commission, 2016). In addition, increased
and theoretical frameworks of anthropology to
evaluation efforts need to be culturally appropriate to
inform evaluations and enable the worth or merit of
account for context, the impact of colonisation and
programs to be determined based on multiple values and
difference in values or conceptualisations of wellbeing.
perspectives. Programs are developed and defined by the
Pre-packaged solutions designed on national scales,
culture within which they operate, therefore judgements
based on Western or even urban assumptions often fail to
about the value or worth of the program depends on
be appropriate in remote contexts. Complex social issues in
what is culturally desirable (OdellButler, 2005). This
diverse contexts do not fit neatly into linear cause and effect
requires consideration of:
models, and are instead grounded in the reality of the daily
■ socio-cultural systems
and historical lives of community members (Grey, Baxter &
Sutton, 2016; Markiewicz, 2012; Rossingh & Yunupingu,
■ the impact of programs on social structures and vice
2016). In response, evaluators need to adopt reflective
versa
approaches, identify Western assumptions underpinning
■ the lived experience of workers and participants
program designs and explore more locally and culturally
appropriate evaluation approaches.
■ the meanings attributed to the program
In this case study, evaluation anthropology was
■ stakeholder ideas of what success looks like, and
selected as one evaluation approach that could account
for culture and context. This practice paper will define
■ reflective and reflexive practice (Copeland-Carson,
evaluation anthropology and then reflect on its usefulness
2005).
in establishing an evaluation framework for a pre-literacy
In Australia, a number of factors need to be
program located in a remote Aboriginal community in
considered in shaping culturally appropriate evaluation
Australia. Developing an evaluation framework for the
frameworks. Most importantly, evaluations need to
program required an exploration of culturally appropriate
account for differences between Indigenous and non-
evaluation methods and application of evaluation
Indigenous life experiences. This includes considering the
anthropology in practice. Some findings are shared and
impact of history, access to services and infrastructure,
pseudonyms have been used for participant names to
rates of youth incarceration, early school leaving, life
protect confidentiality. This case study demonstrates the
expectancy and different concepts of family relationships
need for place-specific theory to inform program design
and structure (Eversole, 2003; Markiewicz, 2012;
and evaluation practice.
Scougall, 2006). Community members’ prior experiences
with evaluation may also affect participation, particularly
Culturally appropriate evaluation in theory
if past evaluations may have brought little benefit
or even led to program cuts or closures (Kelly, 2006;
Culturally appropriate evaluation can find its origins
Scougall, 2006; Wehipeihana, 2008). Finally, evaluators
in the work of evaluators such as Guba and Lincoln
need to avoid the idea of one type of Aboriginality,
(1989) who aimed to overcome ‘management bias’
based on notions of traditional culture, and recognise
and to accommodate ‘value-pluralism’ by encouraging
contemporary constructions that are unique to each time,
greater dialogue of all stakeholders in shaping evaluation
place and space (Martin, 2008). In particular, remote
criteria. They asserted the need to move beyond
living has unique challenges to achieving equivalent
approaches that focus on measurement, testing and
employment and education outcomes in comparison
performance assessment. In more recent times, culturally
to urban Indigenous and non-Indigenous populations
appropriate evaluations are characterised by a respect
(Department of the Prime Minister and Cabinet, 2016;
and understanding of socio-cultural factors, diverse value
Productivity Commission, 2016).
systems and ways of knowing, and through collaborative
Howard—Context and culture in evaluation: a case study of evaluation anthropology
31
A C A D E M I C A R T I C L E
Context
ethnographies to gain some understanding of context,
The program evaluated for this case study aims to
communication with program staff prior to the visit
develop pre-literacy (English language) skills for
and maintaining flexible ideas around data collection
Aboriginal children aged 0–3 years and to equip families
approaches. Participant involvement in the evaluation
to improve school readiness for their children. Designed
was negotiated through program staff who had strong
by a non-government organisation, the program
community relationships.
commenced in 2012 in response to the requests of
Ethics approval for this research was received
parents, carers and Law Women (elders) for help with
through the James Cook University Human Ethics
getting children to school:
Sub-Committee and was required to address the James
Cook University Indigenous Research Protocols (2016).
the Elders had recognised that the old ways had been
In addition, as pointed out by Smith, ‘researchers are in
interrupted and they now wanted their grandchildren
receipt of privileged information... they have the power
to take their place in contemporary society, go away
to distort, to make invisible, to overlook, to exaggerate
to University come back and help us… to have an
and to draw conclusions, based not on factual data, but
opportunity for a future in mainstream life. (Jessica)
on assumptions, hidden value judgements, and often
The program employs three local staff who visit
downright misunderstandings’ (2005, p. 176). Reflexivity
camps surrounding a remote town on a weekly basis, in
is an important part of decolonising practice (Denzin
a van loaded with donated books. Staff lay out a mat, the
& Lincoln, 2011;...
To continue reading
Request your trial