Tải bản đầy đủ - 0 (trang)
Chapter 13. Evaluation and Third-sector Programmes

Chapter 13. Evaluation and Third-sector Programmes

Tải bản đầy đủ - 0trang

13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



Introduction and policy implications

The third-sector is a crucial element of strategies for local economic

development. Not only do the activities of third-sector organisations often

contribute to a range of positive regeneration impacts, but they are also

increasingly seen to be important players in economic development partnerships

and in the delivery of local public services. third-sector organisations argue for a

particular set of “added value” impacts such as social capital and community

involvement which can be vital elements of economic development. They are

also credited with creating innovations which can inform new policies and

models of economic development. The intermediate labour market model, for

example, was a third-sector response to long-term unemployment.

Assessing the impacts and outcomes of third-sector activity is therefore

necessary to find out just how such organisations contribute to regeneration;

the extent of their impacts; and how their particular “value added” can be

further supported and developed. The results of good evaluation can be used

to support ongoing development and improvement in the sector, aid

replicability, refine or change government policy, and support appropriate and

effective funding and financing of their activities.

However, good evaluation of the third-sector in local economic

development currently appears quite limited. This is due to a variety of factors

arising from internal capacity constraints; the requirements of different

funders which may conflict, be limited or create overload; and the lack of

appropriate processes and measurement systems which are able to capture

the particular outcomes, often intangible, which different kinds of thirdsector organisation may create. Evaluation is also difficult because thirdsector organisations often have a range of goals and activities. Their impacts

are also hard to untangle from the local context of which they are a part and

from the actions of public, private and other third-sector players.

This paper summarises a range of approaches to third-sector evaluation

and particularly stresses the need to develop measurement systems which

start from the point of view of the objectives of the organisation itself and its

needs rather than just the requirements of outside funders and policymakers. This not only enables more comprehensive evaluations but also

provides information which can directly inform further organisational and

policy development.



334



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



These approaches cover both process (formative) and ex post evidence of

impact (summative) evaluation, arguing that underpinning both is a need for

systematic data collection and new methods of capturing outcomes that are

hard to quantify. It is also important to recognise the importance of

incorporating the community and users in the evaluation process in order to

create improved and relevant measures of outcomes, identify unintended

consequences, and better understand cause and effect. Participative

evaluation can also create more robust and effective programmes since it

contributes to the regeneration process itself by creating buy-in from the local

community as well as involving beneficiaries in finding their own ways to

achieve outcomes and targets.

There are a variety of implications for policy makers:





A greater understanding of how third-sector organisations contribute to

local economic development can lead to changes in policy, better

partnership arrangements and targeted support for organisations with

strong impact on key development objectives.







There is a need to combine methodologies in order to understand “how

something works” as well as “whether it works” and “how it can be

improved”.







There is a need to focus on outcomes and not outputs in funding schemes

for third-sector organisations.







third-sector organisations require resources and capacity to be able to

evaluate their activities appropriately. This may involve allowing funding

streams to incorporate finance for appropriate evaluation or the provision

of advice on the most effective methods through support agencies.







Evaluations that are able to capture social and environmental impacts and

relate these to public expenditure and public targets could support the

creation of new and innovative forms of funding based on social investment

or on social payment or incentives schemes.







If evaluations are able to capture high social returns on investments but low

economic returns, this helps create arguments for the need for subsidy if those

outcomes are valuable contributions to local development and policy goals.







The need to recognise that third-sector organisations face a range of

evaluation requirements and that it is more beneficial to support the

creation of internal data collection systems which can be used to underpin

a range of evaluations both for internal and external use.







Understanding third-sector impacts may enable a “market making” role for

the public sector where support can be given to the development of

organisations which are able to address multiple policy priorities in a

joined-up way.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



335



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES







Good evaluation can help the public sector reconsider its procurement

strategies and contracts in order to capture the multiple outcomes of thirdsector providers, for example, by delivering a required service whilst also

reducing unemployment. This means that government contractors should

be able to deliver and to design contracts which incorporate the kinds of

added-value which the third-sector may be able to provide. They may also

wish to allocate more resources to supporting the third-sector in its ability

to create and access appropriate tenders and for increasing its capacity to

deliver.1







A greater understanding of third-sector organisations and the communitybased strategies can give more weight to recognizing and capturing the

value of approaches to economic development which are less reliant on topdown programmes. Bottom-up and locally appropriate models which

engage the community and relevant contributions from the third-sector,

private and public sector need to be appropriately evaluated and assessed

in order to determine their relative impacts.



The last point raises an important issue which is not tackled in this paper.

It is critical to recognise the multiplicity of actors and outcomes in

development partnerships and in disadvantaged areas and to measure the

synergy between them and their relationship to overall outcomes. More work

needs to be done to develop a systems approach to evaluation which can

assess the overall impacts of a range of different activities and organisations,

but also identify unique contributions from individual players and the

interactions between them.



Understanding the third-sector

The third-sector is a contested concept. It may, for example, be equated with

the idea of the “social economy”, seen by the European Union as comprising

CMAF – co-operatives, mutuals, associations and foundations. Others see this

classification as limited, ignoring the many examples of not-for-profits or social

enterprises that do not fall into easily defined categories.

In effect, the third-sector describes a space which includes a range of

more recognised terms such as not-for-profits, voluntary sector, mutuals,

social enterprises or community enterprises. These organisations tend to have

in common a focus on social or environmental objectives rather than profitseeking. However, some do create a profit but reinvest this in their activities or

distribute to relevant stakeholders rather than external shareholders or the

owner-manager.

In reality, this third-sector space is very diverse and has no clearly

defined boundaries. It overlaps with the business sector and with the public

sector. There are, for example, some very interesting examples of mission-



336



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



driven shareholder businesses and organisations that are part community

and part public-owned which break down easy distinctions and often provide

new and innovative ways of delivering public services.2

The third-sector can range from organisations that are fully-grant

dependent to those that are fully self-financing and from those that compete

in mainstream markets to those that provide non-marketable goods and

services. The range of social and environmental goals is extremely diverse as

are their governance structures and the different groups of stakeholders who

are engaged with each organisation.

This diversity and overlap with other sectors therefore means that it is

difficult to create evaluative frameworks that are conceptually different from

those used for other purposes. However, they may require some modification

to deal with the distinctiveness of third-sector approaches and outcomes.



How does the third-sector support local economic development?

The importance of the third-sector arises from its role in contributing to

regeneration in ways which go beyond a reliance on physical renewal and

inward investment. It is widely recognised that regeneration and local

economic development is a multi-faceted process involving issues of social

capital, health and welfare and local culture as well as creating jobs and new

businesses. Indeed, regeneration only benefits residents if ways can be found

of ensuring that jobs are created for local people and that any wealth creation

does not just “leak” out of the area.

The third-sector can have a role in, for example:





Increasing employment or employability.







Supporting the creation of new enterprises, for example, through managed

workspaces or organisations supporting the creation of new businesses.







Providing new inclusive ways of doing business, for example, co-operative

models.







Addressing undermet or unmet needs for goods and services, for example,

in social housing, food, and finance.







Addressing social issues such as poor health or the needs of refugees.







Building local infrastructure – for example, arts, sport facilities, transport

solutions.







Creating social capital and increased community engagement.







Recycling and developing responses to other environmental needs.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



337



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



Why measure impact?

The process of measuring outcomes and impacts is a way of showing

whether or not organisations achieve the goals that they set themselves or the

requirements of funding arrangements. It also enables organisations to

understand “how” they are achieving their aims which can support future

organisational development, funding applications and aid in replication of

models or in contributing to policy change. Innovations in the third-sector

often showcase new ways in which social, environmental or economic policy

may be affected. Evaluation can also unearth the unexpected outcomes that

can arise from particular programmes of work or interventions.

Another reason for evaluation is to create accountability to, and

engagement of, key stakeholders. This is necessary not only for legitimacy but

also to create a sense of shared ownership and enable beneficiaries and other

knowledgeable stakeholders to contribute expertise towards understanding

“how” something is working and how it might be improved in particular local

circumstances. This means that participative approaches to evaluation can

contribute to the effectiveness of the economic development process itself by

creating new ways of addressing local needs (often involving the local people

themselves) and responding to specific local opportunities for economic and

social change. Changes may seem small and simple but can have large

impacts. For example, an evaluation of crime levels in an area, both actual and

perceived, found that local people believed that the trees adjacent to their

houses in a park helped burglars scale their walls undetected. By removing

those trees, feelings of safety were increased. (It was not recorded if actual

crime levels went down).3

Evaluation is also a necessary part of organisational change and

management, supporting staff and directors in the development and revision

of programmes and in designing future strategies and more effective ways of

working.

We are therefore talking here about evaluation as both a formative process

approach, with implications for the ongoing development of a programme, as

well as its use in a more summative way to demonstrate final impacts. There is

no reason, however, why these two approaches should not be combined. Often,

data collected for ongoing process evaluation is the same as that required for

summative evaluations.

There are, though, certain third-sector managers who are somewhat

dismissive of evaluation. For some, it is a luxury that would be a nice thing to

do, were it not that scarce resources need to be targeted on delivery and often

on day-to-day survival.



338



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



It has also been argued that performance measurement may cause

organisations to strive for continuous improvement when this may not be

appropriate to the third-sector and current practice should be “good enough”.4



Similarities and differences of third-sector evaluation methods

Measuring the impact of the third-sector is in some ways no different

from evaluating any other local economic development programme or activity.

It is primarily about the most appropriate tools and techniques. However,

evaluation of the third-sector is often required, or driven, by the need to show

particular impacts – to show that a certain way of doing things adds value and

to explain why this is the case. It is also important to evaluate all the impacts

of an organisation, not just specific activities. There may well be synergies

between the projects conducted by an organisation, or the way the

organisation works may itself create added-value. One example could be that

of tenant-owned housing which, in addition to its services, has been shown to

create other benefits including increased self-confidence in tenants and

involvement in civic activities.5 Through considering the overall mission of

the organisation and its effectiveness, rather than the impacts of individual

projects, whole-organisation evaluation can also show gaps in provision or

assess the general capacity and effectiveness of staff and processes. Standard

programme evaluation frameworks may not be able to capture all the

processes at work, or the specific intangibles created by the organisation.

The kinds of specific outcomes we are talking about have been suggested

to include:





Closeness to the community.







Active engagement of users and beneficiaries.







Access to and understanding of disadvantaged groups.







Trust by users in the quality of a service. This can be particularly important

for services where it is difficult for the users or relatives of users to judge

quality, for example, in elderly care or childcare.







Providing nonmarketable goods and services.



These different attributes or activities are often assumed or anecdotal

rather than shown. They are also not necessarily confined to the third-sector.

It is therefore important that they are demonstrated and that organisations

have the ability to measure their impacts and improve their performance on

the basis of the information that they gather.



Issues in impact measurement and the third-sector

We need first to distinguish between evaluating the impact of a thirdsector organisation in particular programmes and evaluating the impact of the



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



339



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



organisation as a whole. There is also the challenge of disentangling the

inputs of different organisations within multi-actor partnerships.

We also need to distinguish between processes that the organisation

uses, such as engaging stakeholders, and particular outcomes. This is

necessary because it could be argued that engaging stakeholders is time

consuming, using resources that may better be focused on direct delivery.

However, the processes may improve the delivery or be a valuable end in

themselves, creating for example, increased social capital, greater community

engagement, buy-in to the programme, or increased self-esteem.

Unfortunately, third-sector organisations are not always good at showing

their impacts or their particular added value. One of the main reasons for this

is the many evaluation requirements arising from programme funders or

other finance providers which can include different measures and different

timescales for delivery. The result can be management overload, high costs

and partial evaluations of an organisation’s activities. Funder evaluation

requirements are in reality fairly simply based on outputs rather than

outcomes and more qualitative analysis.

There is a tendency, therefore, for many third-sector organisations to

see funder monitoring and evaluation as more of a burden than a useful

process. They feel that external evaluations generally have little use as

learning tools and that they do not really communicate the essence of what

organisations are doing and how they make that difference.

Other barriers to quality evaluation include:



340







Constraints of time, resources, skills and knowledge.







The complexity of many evaluative approaches.







The plethora of different available models which means that organisations

are unsure as to the quality of different evaluation techniques, or how to go

about choosing the most appropriate approaches for their needs.

Additionally, multiple evaluation approaches can make comparability and

bench-marking difficult between similar organisations.







Unavailability of appropriate techniques, particularly those which can

evaluate intangible outcomes and impacts. Social firms for example are

o rg anisations t hat op erate in m ainstream markets but em ploy

predominantly disabled people or those with a mental health problem.

They are currently looking to develop a measure of their ability to increase

positive health outcomes.







The multiple nature of objectives and outcomes which can require a whole

range of different evaluative approaches.







The different evaluation requirements of different audiences from funders

to beneficiaries to management.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES







Funders feeling concerned that some of their money is being used for

evaluation and not for service delivery.







Evaluation being seen as a test rather than a tool that can be used to

increase effectiveness.



In addition, there are also general problems which arise in measuring

regeneration impacts including:





The length of time for different outcomes or impacts to become apparent

following interventions or initiatives.







The multiplicity of inter-related factors which influence success or failure

within a particular context or with different groups of people.







The difficulty of disentangling impact from broader causal influences in the

local area or in the broader macro-environment.



Types of evaluation techniques used

The kinds of evaluative techniques used by third-sector organisations

range from qualitative to quantitative and from external or “extractive”

evaluation to evaluation which is participatory and may involve users in

defining their own evaluative measures. The more participatory the approach,

the more likely it is that the measurement itself becomes an integral part of

the regeneration process – whether helping to define or refine the programme

and changing the nature of involvement or commitment by stakeholder

participants.



Participative evaluation – Participatory Monitoring and Evaluation (PMandE)

began in the 1970s in the international development field as a result of the

recognition that external evaluation with preset criteria was unable to capture

how change happens or to create a process which is inclusive and works with

the views and aspirations of those most directly affected.*

*



More information on Participatory Monitoring and Evaluation can be

obtained, for example, in a Policy Briefing from the Institute for Development

Studies Website: Policy Briefing, Issue 12, November 1998. www.ids.ac.uk.



This approach involves local people in choosing and designing

appropriate indicators to measures local outcomes and also in the data

collection process. It can therefore contribute to the renewal process through

identifying locally important criteria as well as involving local people in

identifying solutions. PMandE challenges some of the established ideas of

rigorous data collection and analysis. In some cases data collected may be

more reliable. For instance, in Merthyr Tydfil in the UK, information collected



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



341



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



by children on crime in a study in 1996 was better than that collected by the

police since people were more likely to tell children the truth.6

The process can also be made externally valid through choice of relevant

indicators and appropriate sampling.

Many third-sector organisations make use of certain evaluation

techniques but often in a fairly basic and ad hoc way. There is a whole range of

evaluative approaches that could be used but these are not necessarily easy to

access or tailored to specific needs.



Strengths

● Feeds back into the regeneration process.

● Engages stakeholders and can lead to a greater sense of ownership of and



engagement in the process which can lead to increased trust, awareness

and community capacity.

● Builds on local knowledge and experience of what works and how it does



so which increases knowledge of causality and therefore learning.

● Can lead to better quality information.

● Engages people who are often left out of formal surveys.

● Stimulates action and civic engagement.



Weaknesses

● Can be time and resource intensive.

● Problems related to comparability and robustness.

● Must balance the need for particular indicators which are comparable



across areas in order to assess relative impacts with local choice of

indicators that are specific to the particular context. The latter can often be

invaluable in understanding how something has happened and how it

could be improved. An example of a comparable question would be: “Do

you feel safe in your area?” and a more specific question would be “Do you

feel safe walking through [named] park?” The first question taps general

levels of community feelings of safety and the latter a more specific space

where interventions might be made, if appropriate or possible.

● Participation is not a given and requires creativity.

● Participation may be biased through involvement of pressure groups and



more “powerful” or vociferous residents. More broadly, subjective

assessments can be problematic since they can change according to the

timing of the evaluation or the context in which it is made.



342



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



There are several overviews of evaluation in the third-sector, for example,

the Independent Sector’s Measure Project which is a survey of outcome

measurement techniques used in the US.7 There is also a Canadian review of

evaluation resources by Bozzo and Hall.8

There has been a significant amount of work focused on the third-sector

and regeneration. A range of approaches and evaluation frameworks have

been developed. These have focused on measuring the particular added value

of third-sector organisations around issues such as changes in community

involvement or social capital.

Kendall and Knapp (1999) set out an approach to performance

measurement which includes some of the possible added value of the

voluntary sector – choice, participation, advocacy and innovation – alongside

criteria often used to evaluate the public sector – economy, effectiveness,

efficiency and equity.9 Whilst theoretical, this work was used by Community

Evaluation Northern Ireland to develop a framework for measuring social

capital which could be used to select appropriate indicators.10

Another example of a third-sector evaluation framework is that developed

by the Scottish Community Development Centre – ABCD – Achieving Better

Community Development.11 This model sees evaluation as an integral part of

community development, and community involvement as a vital part of

evaluation. They developed a pyramid of outcomes from personal

empowerment and participation through to the creation of sustainable

communities.

The New Economics Foundation has also developed a framework – Prove It!

– for measuring social capital which allows the use of both locally defined

measures as well as others which can be compared across areas and projects.

The information can therefore contribute to both process and summative

evaluations.



Prove It!

Prove It! is an approach to evaluation developed in partnership with

Groundwork – an environmental not-for-profit organisation in the UK – and

Barclays Bank in order to assess the change in “social capital” in an area as a

result of regeneration projects. Whilst social capital is a contentious concept,

this project was specifically designed to find out whether projects to improve

the local physical environment were also creating change in the community

through, for example, increasing the number of relationships between people,

or levels of trust. The model is easily accessible to local people as well as the

public sector and uses the language of measurement and learning rather than

evaluation, which can be discouraging to some.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



343



13.



EVALUATION AND THIRD-SECTOR PROGRAMMES



Prove It! measures the often invisible effects of regeneration activities on

local people who are actively involved in choosing indicators and collecting

the data. Some indicators are readily available and others are newly designed

to capture changes in attitudes and action, for example, in levels of graffiti.

Qualitative and quantitative techniques are used, including surveys and

existing local data. The full process and its outputs are set out in an easy-touse handbook.12

Unintended impacts of the projects assessed included greater community

engagement, ideas for new projects, commitment to the regeneration activities,

and enjoyment and learning from the process itself. There was also increased

understanding between older and younger generations as a result of the latter

being involved in data collection.

Prove It! is being further developed to make the approach more rigorous

and to permit benchmarking across activities and areas whilst retaining the

participative methodology.



Using indicators

A core element of all the approaches mentioned above is that of

evaluating processes and outcomes which are often hard to measure.

Examples of such outcomes include increased trust and decreased fear of

crime. In order to prevent reinvention of the wheel and to aid comparison,

robust and tested indicators are useful. Some of these have already been

developed, whilst others are being designed or still need to be created.

A good example of an innovative and useful indicator is that of the

measure of “distance travelled” used by many intermediate labour market

organisations and employment projects.13 This concept refers to the progress

that a beneficiary makes towards employability, or outcomes that are more

easily measured, as a result of the project intervention. It acts as a measure of

the soft outcomes achieved such as interpersonal skills, organisational skills,

time management, or confidence rather than just looking at whether or not

someone has found a job. For many people who have been in long-term

unemployment, obtaining a job will be an immense step and considerable

effort and time will be needed for them to become employable. Increased

“employability” therefore involves a range of skills and attitude changes and,

for some people, the distance to be travelled towards achieving employment

will be greater than for others.

Such a measure is therefore more appropriate as an indicator of the

success of an intervention than just whether or not someone obtains a job. It

is also vital to ensuring that funders recognise the real added-value and do not

create easy output targets which could bias behaviour so that organisations

are pushed towards only dealing with people who are the easiest to employ.



344



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Chapter 13. Evaluation and Third-sector Programmes

Tải bản đầy đủ ngay(0 tr)

×