‘I’m sorry but I can’t take a photo of someone’s capacity being built’: Reflections on evaluation of Indigenous policy and programmes
| Author | Ruth McCausland |
| DOI | 10.1177/1035719X19848529 |
| Date | 01 June 2019 |
| Published date | 01 June 2019 |
| Subject Matter | Academic Article |
research-article2019
Academic Article
Evaluation Journal of Australasia
2019, Vol. 19(2) 64 –78
‘I’m sorry but I can’t take a
© The Author(s) 2019
Article reuse guidelines:
photo of someone’s capacity
sagepub.com/journals-permissions
https://doi.org/10.1177/1035719X19848529
DOI: 10.1177/1035719X19848529
journals.sagepub.com/home/evj
being built’: Reflections on
evaluation of Indigenous
policy and programmes
Ruth McCausland
UNSW Sydney, Australia
Abstract
The Australian Government has recently increased resourcing for evaluation of
Indigenous programmes following critical reports by the Australian National Audit
Office and Productivity Commission around their failure to significantly reduce
Indigenous disadvantage. Evaluation in Indigenous affairs has a long history, although
not a consistent or coordinated one. While there is significant knowledge held by
those with experience in commissioning and conducting evaluations for Indigenous
programmes over a number of decades that could usefully inform current efforts,
there has been little research focused on this area. This article outlines the findings of
qualitative research about evaluation in Indigenous policy conducted with policymakers,
senior public servants, programme managers, researchers and independent evaluation
consultants that sought to privilege the voices and perspectives of Aboriginal and
Torres Strait Islander people. It outlines key themes derived from those interviews
relating to the methods, parameters, politics and accountability around government-
commissioned evaluation in Indigenous policy and programmes and concludes by
canvassing ways that evaluation could better serve the interests of Aboriginal and
Torres Strait Islander peoples.
Keywords
Aboriginal and Torres Strait Islander peoples, evaluation methods, Indigenous
policy, programme evaluation, qualitative research
Corresponding author:
Ruth McCausland, School of Social Sciences, UNSW Sydney, Sydney, NSW 2052, Australia.
Email: ruth.mccausland@unsw.edu.au
McCausland
65
Introduction
In its 2017–2018 budget, the then Turnbull Government announced it was allocating
AUD $40 million to evaluate the Australian Government’s AUD $30 billion spending
on Indigenous programmes. It emphasised that Aboriginal and Torres Strait Islander
people would benefit from this investment in ‘building the evidence to better direct
funding on the ground and deliver better outcomes’ (Department of Prime Minister
and Cabinet, 2017). This followed reports by the Australian National Audit Office
(2017) and Productivity Commission (2016) that were critical of the Commonwealth
Government’s failure to significantly reduce Indigenous disadvantage and recom-
mended more rigorous evaluation of Indigenous programmes. In February 2018, the
Department of Prime Minister and Cabinet released an overarching Evaluation
Framework to guide the evaluation of programmes and activities under the Indigenous
Advancement Strategy (Department of Prime Minister and Cabinet, 2018a). It has also
established an Indigenous Evaluation Committee to support transparency and ensure
that the conduct and prioritisation of evaluations is independent and impartial
(Department of Prime Minister and Cabinet, 2018b), and in December 2018, the
Treasurer announced the appointment of the first Indigenous Policy Evaluation
Commissioner of the Productivity Commission (Frydenberg, 2018). The objective of
the Commonwealth Government’s efforts in this area is ‘to strengthen the quality,
credibility, and influence of the evaluations of policies and programs’ (Department of
Prime Minister and Cabinet, 2018a).
The first section of this article reviews the literature on evaluation in Indigenous
policy, including significant critiques of the failure of evaluation in this area to
serve the interests of Aboriginal and Torres Strait Islander communities. The article
then sets out the methodology and findings of qualitative research conducted with
policymakers, senior public servants, programme managers, researchers and evalu-
ation consultants about evaluation in Indigenous policy that sought to privilege the
voices and perspectives of Aboriginal and Torres Strait Islander people given such
critiques. The title, ‘I’m sorry but I can’t take a photo of someone’s capacity being
built,’ is drawn from an interview with a senior Indigenous public servant quoting
a Minister who had funded a programme in an Aboriginal community and wanted
positive media coverage in light of an upcoming election. This tension between
measuring the success of community-based, culturally led programmes using met-
rics that are meaningful and relevant to Aboriginal communities, and those consid-
ered robust and objective to government, was one theme consistent in many of
those interviews. This article outlines this and other key themes derived from those
interviews relating to the methods, parameters, politics and accountability around
government-commissioned evaluation in Indigenous policy and concludes by can-
vassing ways that evaluation could better serve the interests of Aboriginal and
Torres Strait Islander peoples. The great expertise held by many people who have
worked in commissioning and conducting evaluations in Indigenous policy and
programmes over a number of decades can usefully inform current efforts in this
area, and this article seeks to make a contribution to this end.
66
Evaluation Journal of Australasia 19(2)
Evaluation and Indigenous policy: background literature
Evaluation in Indigenous affairs at a Commonwealth Government level has a long his-
tory, although not necessarily a consistent or coordinated one. The Aboriginal and
Torres Strait Islander Commission (ATSIC) had an Office of Evaluation and Audit
from its inception in the 1980s, though government emphasis on evaluation in ATSIC
was primarily on the auditing of Indigenous organisations, using it as the basis for
‘punishment and defunding’ rather than learning or capacity building (Chaney in
Productivity Commission, 2013, p. 58). It has been argued that Indigenous pro-
grammes have been subject to greater scrutiny than any other sector (Altman &
Russell, 2012); however, this scrutiny has not necessarily involved widespread rigor-
ous evaluations that have informed or improved Indigenous policy (Hudson, 2016;
Productivity Commission, 2013, 2016). The perspectives and interests of Aboriginal
and Torres Strait Islander people have not been generally well served by government
approaches to evaluation in Australia to date.
In a complex and highly contested policy area such as Indigenous affairs, notions
of success, failure and evaluations themselves have been described as politically
charged (Altman & Russell, 2012). The systemic disadvantage faced by Aboriginal
and Torres Strait Islander peoples is often responded to by government with multi-
faceted policy initiatives with multiple objectives that have been observed to be ill-
defined, unarticulated and contradictory (Cobb-Clark, 2013). This makes the
evaluation of programmes associated with such policy initiatives often difficult and
contentious. Indigenous affairs has been described as characterised in particular by a
lack of effective links between data, theory and policy development (Pholi, Black, &
Richards, 2009) with a history marked by pilot programmes and trials that were dis-
continued, with or without evaluation, following a change in government or policy
priorities (Walden, 2016). Significant amounts of funding characterised as ‘Indigenous’
has been allocated to non-Indigenous organisations and employees and administrative
costs (Altman & Russell, 2012). Despite much emphasis on ‘evidence-based policy’ in
Indigenous affairs by political leaders, there remains a lack of critical analysis of what
constitutes evidence and to what end in evaluation of Indigenous policy (McCausland,
2015). How to measure positive social change and attribute its cause is one of the
greatest challenges in evaluation, and the multi-departmental investment and control
at a Federal and State/Territory level that characterises Indigenous affairs make this
particularly complicated and contested (Cobb-Clark, 2013). Indigenous organisations
are required to account for funding to many different agencies and regularly required
to demonstrate the effectiveness of relatively small programmes through broad out-
comes such as increased school attendance, reduced crime rates as well as more nebu-
lous measures such as increased community capacity (McCausland, 2005).
The annual Closing the Gap report, tabled in Parliament by the Prime Minister each
year for the past decade (Department of the Prime Minister and Cabinet, 2019), is the
government’s most high-profile measurement of progress against a range of targets
aimed to reduce disadvantage faced by Indigenous people in the areas of life expec-
tancy, child mortality, education and employment. While not formally framed as an
McCausland
67
evaluation of Indigenous policy and programmes, critiques of the Closing the Gap
targets point to key tensions in this area. Closing the Gap targets have been argued to
render a ‘deeply entrenched development problem into a hyper technical monitoring
exercise’, an approach which filters down to community level (Altman & Russell,
2012, p. 17). A focus on the plethora of statistics generated by monitoring tools has
meant there has been...
To continue reading
Request your trial