Social indicators –promises and problems: A critical review

R E F E R E E D A R T I C L E
E v a l u a t i o n J o u r n a l o f A u s t r a l a s i a , V o l . 3 ( n e w s e r i e s ) , N o . 1 , A u g u s t 2 0 0 3 , p p . 1 7 – 2 6
Social indicators –
promises and problems:
a critical review
Anona Armstrong
Victoria University in conjunction with Crime Prevention Victoria
(CPV) received an ARC grant to investigate the relationships
Ronald Francis
between crime prevention and community governance. The
first task of the project was to develop a framework linking
community needs, community capacity, wellbeing and CPV
interventions, to guide selection of social indicators and then
compile a database of data from various sources. Among the
difficulties inherent in developing social indicators are:
selecting a framework to guide the development and analysis
of the indicators; the difficulty of obtaining a reliable across-
government comprehensive database that would be
continuously updated; the different contexts, policy goals and
programs that indicators could serve; the significance of
different definitions and contexts; applying appropriate criteria
to guide the selection of the indicators; and the diversity of
views about how indicators should or could be used. The
purpose of this paper is to describe how these issues are
addressed in this project, the theoretical framework that
guides the selection of data from the database and how some
of these difficulties are addressed.
Introduction
Social indicators are statistics which, similar to the economic statistics in the National
Anona Armstrong is a
Accounts, are intended to provide a basis for making concise, comprehensive and balanced
Professor in the Faculty of
judgements about the conditions in a specific society.
Business and Law at Victoria
The term ‘social indicator’ is attributed to Bauer, who in 1966 edited Social Indicators,
University, Melbourne,
a collection of essays instituted by the American Academy of Arts and Sciences to study the
Victoria. She is also Deputy
impact of the space program on American society. A social indicator was defined by the
Director of the Centre for
Organisation for Economic Cooperation and Development (OECD) as a ‘direct and valid
International Corporate
statistical measure which monitors levels and changes over time in a fundamental social
Governance Research.1
concern’ (OECD 1976, p. 14). A social concern was an identifiable and definable
Ronald Francis is a Professor
aspiration or concern of fundamental and direct importance to human wellbeing. The first
in the Faculty of Business and
24 concerns to be described by the OECD addressed eight ‘primary goal areas’: health,
Law at Victoria University,
individual development through learning, employment and the quality of working life,
Melbourne, Victoria. He is
time and leisure, command over goods and services, the physical environment, personal
also a Professorial Fellow at
safety and the administration of justice, and social opportunity and participation. The
the Centre for International
concerns were selected because they had policy relevance and were quantifiable.
Corporate Governance
Research.2
A r m s t r o n g & F r a n c i s – S o c i a l i n d i c a t o r s – p r o m i s e s a n d p r o b l e m s
17

---

R E F E R E E D A R T I C L E
OECD social indicators
Another problem is ‘ownership’ of the
The OECD uses social indicators for two purposes:
information. If a coordinated database is
to describe social developments in member
established, who owns the data and what safeguards
countries, and to determine how effective society
its future use? In the case of CPV’s database, raw
and government are in altering social outcomes.
data are managed within the department under a
According to the OECD (2001), indicators of social
Memorandum of Understanding with other
development require ‘a broad coverage of social
departments that safeguards confidentiality and
issues. Insofar as social life requires health,
restricts their use.
education, freedom to develop, resources and a
stable basis of social interactions, so must the
Different definitions
indicators reflect these various dimensions’ (p. 9). In
The use of different definitions of constructs can
their 2001 publication the OECD produced
produce conflicting results. Some of the difficulties
comparative trend data for 49 social indicators of
in definition are illustrated by reference to terms
self-sufficiency, equity, health and social cohesion
such as social capital, community, community
for 29 OECD countries.
capacity and poverty. Their varied definitions are
not discussed here. However, the problems are
Social indicators for government
illustrated in the OECD (2001) discussion of the
programs
definition and measurement of relative poverty
which pointed out that because what is meant by ‘a
In general, early attempts at using social indicators
decent standard of living’ varies across countries
to evaluate the impact of government programs
and over time, there is no widely agreed measure of
were of limited value because the indicators were
poverty across countries. The OECD approach was
too far removed from the activities of the programs
to look at relative poverty, defined as existing when
they were intended to evaluate. Social indicators of
a family income was less than half of the median
crime, for example,
income. ‘This … does capture well the idea that
were of limited value
Most of the indicator reports in
what really matters is not just subsistence but also
in assessing the results
the 70s and 80s were mainly
the ability to participate in mainstream society’
of the activities of
descriptive and produced
(2001, p. 62).
social programs
according to the major functional
A problem for our project is the meaning given
providing services,
service provision roles of
to constructs such as ‘community’. Does community
such as youth
refer to the locale covered by a local neighbourhood
government departments, such
counselling, even
watch association or does it refer to the network of
as health, housing and education.
though their longer
associations formed by the members of a particular
term aim was to
neighbour interest group? In our mobile society a
reduce crime. Most of the indicator reports in the
school community can be drawn from areas across
70s and 80s were mainly descriptive and produced
Melbourne. In crime prevention, an initiative such
according to the major functional service provision
as improving access lighting in a shopping centre
roles of government departments, such as health,
may have a very local impact, or it may address the
housing and education (Australian Bureau of
concerns of a multitude of shoppers from many
Statistics 1978; Armstrong & Wearing 1979, 1981).
different localities. In our study we restricted our
definition of community to local government areas
Difficulties in obtaining reliable data
(LGAs) as defined by the ABS because the data are
across government databases
most readily available or convertible to this level.
The early indicator reports reflected the
individualistic approaches of government
A comment on social indicators and
departments to addressing community problems.
performance measures
Government departments operated in isolation (the
Although performance indicators are used to
silos) and were reluctant to share their data which
measure program activity or results, social
are regarded (rightly) as their intellectual property
indicators according to the OECD ‘cannot be used
base. Most social data collected by government
to evaluate whether a particular social program is
departments are confidential and there is always a
effective. Rather, social indicators can be used to
danger of confidential information that is shared
assess whether and how the broad thrust of policy is
inadvertently becoming public. Most departmental
addressing important social issues’ (2001, p. 9).
data are now linked directly to strategic planning
When indicators are used at a society level they are
and performance (and often to interstate if not
usually termed ‘social’ rather than ‘performance’
international benchmarks, for example student
indicators.
standards in education). Departments are often
Some of the differences between performance
sensitive about their performance being made public
and social indicators are shown in Table 1. In
without adequate explanations. In other cases,
general, performance indicators are applied at the
private sector service partners can request that
level of specific projects and programs whereas
information be withheld for ‘commercial in
social indicators are focused on larger social goals.
confidence’ reasons.
18
E v a l u a t i o n J o u r n a l o f A u s t r a l a s i a , V o l . 3 ( n e w s e r i e s ) , N o . 1 , A u g u s t 2 0 0 3

---

R E F E R E E D A R T I C L E
TABLE 1: DIFFERENT USES OF PERFORMANCE AND SOCIAL INDICATORS
EVALUATION TARGET
SOCIAL INDICATORS
PERFORMANCE INDICATORS
Establishing social goals
Social indicators
Whole-of-government approaches
Social indicators
to common problems
Identifying community needs
Social indicators
Identifying emerging social problems
Social indicators
Achievement of program outcomes
Social indicators
Performance indicators
Organisational/strategic performance
Social indicators
Performance indicators
Performance of specific programs
Performance indicators
Performance monitoring of program
Performance indicators
inputs, processes and activities
Achievement of program outputs
Performance indicators
However, they are not exclusive and both...