Tải bản đầy đủ - 0 (trang)
Chapter 9. A Commentary on Brian Rubson’s Paper and the Workshop “Area-based Policy Evaluation”

Chapter 9. A Commentary on Brian Rubson’s Paper and the Workshop “Area-based Policy Evaluation”

Tải bản đầy đủ - 0trang

9.



A COMMENTARY ON BRIAN RUBSON’S PAPER AND THE WORKSHOP “AREA-BASED POLICY EVALUATION”



T



his commentary focuses on three main points from Professor Robson’s

paper and the associated workshop discussion on the United Kingdom

experience of area-based policy evaluation. They reflect issues that are also

important for evaluation practice in other countries. Firstly, the quality and

usefulness of area-based policy evaluation is often less than it could be

because certain basic evaluation concepts are not being systematically

addressed. Secondly, we should recognise that there exist a number of barriers

to good quality area-based policy evaluation at local level, which limit the

degree to which key evaluation concepts are employed. Thirdly, there are a

number of ways in which local governments and development agencies can

help to build evaluation capacity at local level, thus helping to overcome

barriers to more rigorous area-based policy evaluation.



Key concepts for good quality area-based policy evaluation

As Professor Robson’s paper points out, the past three decades have seen

much area-based policy evaluation in the UK. Additionally, the arrival of the

Labour government in 1997 brought a significant expansion in area-based

policy evaluation work, associated with an expanded range of area-based

programmes and a commitment to more evidence-based policy making across

government as a whole. However, the evaluation that has been carried out has

been of variable quality. Whilst Robson signals some major good practice

evaluations, he also argues that many smaller scale evaluations have failed to

grapple with certain conceptual issues that should be included in any rigorous

area-based policy evaluation. Some key issues needing more attention in

current evaluation practice are outlined below, drawing on and in parts

extending the arguments in Robson’s paper:





222



Programme rationale and objectives. An evaluation should start by

understanding the rationale and objectives originally set for a programme,

i.e. what problems the programme was expected to address and how it was

expected to address them. The underlying rationale may be articulated in

policy documents but often it must be teased out by the evaluator from

interviews with local policy makers. The evaluator should assess whether

the rationale and objectives were appropriate and are still appropriate, and

also compare programme results with the rationale and objectives to assess

programme effectiveness in its own terms. A problem that often arises is

that the rationale behind policy turns out to be misplaced to a greater or



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



9.



A COMMENTARY ON BRIAN RUBSON’S PAPER AND THE WORKSHOP “AREA-BASED POLICY EVALUATION”



lesser degree. Thus there has been much discussion of the tendency for

policy makers to take solutions “off the shelf” or to follow fads, without

properly thinking through whether the approach is really appropriate to the

needs and opportunities of their area. A classic example is of localities

seeking to attract high technology industry because of its associations with

high wages and growth potential, even if the local conditions do not exist

for sustainable development of the sector (university links, skilled labour,

good communications, residential attractiveness, etc.). A related issue is

consideration of alternatives, i.e. could there be alternative ways of

achieving the same objectives more effectively? A rigorous evaluation

should assess the programme rationale, considering whether it is

appropriate and is the best way of meeting the objectives.





Deadweight. Too many evaluations fail to consider the counterfactual, i.e.

what would have happened in the absence of a programme. In particular, it

is important to consider how far policy-supported actions would have taken

place anyway (deadweight). Without such an assessment, it is easy to

overestimate programme effects by attributing all of the positive outcomes

in the area to the programme being evaluated. Where the counterfactual

has been taken into account in UK evaluations, evaluators have tended to

rely on self-assessments made by surveyed policy makers, programme

managers, beneficiaries and other stakeholders. It is difficult to have full

confidence in such evidence because respondents find it very difficult to

answer hypothetical “what if” questions and because self-interest may lead

to biased responses. Control group or control area comparisons can

complement self-assessment exercises and provide the evaluator with

greater confidence in results.







Displacement and substitution. Displacement occurs where the start-up or

growth of policy-supported enterprises/organisations leads to a loss of

activity in other enterprises/organisations. In the context of labour market

policy, substitution occurs where policy leads to supported individuals

taking employment at the expense of non-supported individuals. Often

these effects are not fully dealt with by evaluators. As with deadweight,

evaluators often rely on self-assessment exercises about the degree to

which other firms, individuals or areas are likely to have been adversely

affected. Control group or control area comparisons should be used more

often to complement this information. There has also been relatively little

consideration to date of whether displacement and substitution should be

discounted as entirely negative effects or whether they can be considered in

part as a positive stimulus to dynamic adjustment and innovation in the

local economy. The latter could be true if policy results in supported

organisations and individuals developing competitive advantages over

others, leading to higher productivity overall. Furthermore, it has been



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



223



9.



A COMMENTARY ON BRIAN RUBSON’S PAPER AND THE WORKSHOP “AREA-BASED POLICY EVALUATION”



argued that the mere creation of “churn” in the labour market could be

beneficial if it protected individuals from the harm of being inactive for long

periods of time. Thus the challenge for evaluators is firstly to measure the

degree of displacement and substitution and secondly to decide how far

this should be considered as a positive or negative phenomenon.



224







Longitudinal evaluations. Area-based policy evaluations have often been

limited to ex post assessments on completion of the programme. But it is

difficult to track how well a programme has functioned if evaluation only

takes place after the event, because of changes that take place during the

programme, such as openings and closures of firms, movement of people

and changes in the nature of supported activities. Some UK evaluations

have attempted to develop a longitudinal approach, generally by making

cross-sectional analyses of activity within the area at two or more points in

time. For example, annual surveys of land use and enterprise activity on UK

Urban Development Corporation zones helped to show the processes of

change underway during the life of these initiatives. It is important to track

programmes over time in this way.







Household mobility. When examining how far policy is helping people living

in marginalised neighbourhoods, evaluations must also confront the

complex issue of the effects of household mobility into and out of the target

area on evaluation results. The difficulty arises because assisted people

may move out of the area and benefits accruing to them would not be

assigned to the initiative unless the evaluation has some way of tracking

the people assisted. Alternatively, a regeneration initiative may encourage

people to move into the area, improving the population profile of the target

area and giving the impression that policy has helped local people if

incomers are relatively well off, although policy may not have directly

helped the people originally targeted. In the UK, area-based policy

evaluations have only rarely tried to track the movement of people that

policy aims to help, but this should be included in the design of rigorous

area-based policy evaluations.







Outcomes versus outputs. Many evaluations stop at quantifying the outputs of

a programme, for example the number of people obtaining a qualification,

the number of people finding employment or the number of hectares of

derelict land brought back into use. However, what is really of interest is the

resulting outcome in terms of improving the quality of people’s lives and

stabilising or improving the economic and social vitality of the area.

Assessing outcomes should therefore be the final target of area-based policy

evaluation, concentrating on measures such as incomes, poverty, health,

migration and land prices.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



9.



A COMMENTARY ON BRIAN RUBSON’S PAPER AND THE WORKSHOP “AREA-BASED POLICY EVALUATION”







Disentangling causality. Establishing causality is especially important for

area-based policies because they tend to put in place multiple interventions

addressing various aspects of economic, social and physical regeneration.

The complication for evaluation is that actions aimed at one activity may

have impacts on others. For example, measures for employment generation

may reduce crime whilst measures that reduce crime may encourage new

business activity. In order to judge the right policy mix for an area,

policymakers needs to know the relative contribution of different actions to

overall outputs and outcomes. There is still much progress to be made in

measuring the extent and nature of synergies between actions. Sometimes

matrix approaches are applied, indicating whether or not given actions

contribute or are likely to contribute to improvements in more than one

output field. This provides a framework for identifying possible synergies

but does not go far in measuring their extent or how they occur. The need

to measure synergies is increasingly recognised, but research is needed to

develop more sophisticated techniques for doing so.







Small area data. Because area-based policy tends to target small geographical

areas whilst official data tends to be available only for larger administrative

units, evaluators often face problems obtaining satisfactory data to measure

the baseline situation and subsequent changes in the area compared with

neighbours and the wider economy. As Robson notes, this issue has been

recognised in the UK and considerable effort is going into improving the

situation.







Scale of intervention. Area-based policy evaluations tend to make their

central reference the comparison of programme outputs or outcomes with

programme costs. There are two main problems with this way of thinking.

Firstly, there tends to be no assessment of whether the policy is being

applied at sufficient scale to address the problems in a satisfactory

timescale. A key question for evaluators should be, “if this policy is applied

at this rate, how long would it take to resolve the problems of the area?”

Secondly, there tends to be no assessment of whether, if resources were

increased, an effective and efficient small-scale initiative could be scaledup to deal with larger areas and client groups. Apart from the issue of

whether any programme would work in the same way in a different context,

the failure to consider scale can lead to misleading policy prescriptions

suggesting that successful programmes naturally should be expanded.

Small-scale schemes often appear very promising because they can deal

with the areas and groups where policy is likely to make the most

difference, but it might be difficult to achieve the same success with larger

areas and groups.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



225



9.



A COMMENTARY ON BRIAN RUBSON’S PAPER AND THE WORKSHOP “AREA-BASED POLICY EVALUATION”



Why is good quality area-based policy evaluation relatively rare?

Workshop participants sought to establish why good quality area-based

policy evaluations, which take into account the sorts of conceptual issues referred

to above, are relatively rare. The following possible barriers were identified:

1. Evaluation involves many positive externalities that are difficult for a local

government or development agency to capture. For example, results are

often made public, so it is easy for other areas to “free-ride” by referring to

evaluations carried out by others.

2. There are economies of scale in evaluation that suggest they may best be

undertaken at central level or by groups of local areas co-operating together

rather than by individual local governments or agencies.

3. Good quality evaluation can be extremely expensive and can therefore

become a significant proportion of total programme expenditure, especially

for small programmes. In economic terms, this can be thought of as a

problem of non-divisibility in that the costs of rigorous evaluation are

difficult to reduce even for small programmes. Local agencies therefore

have to balance the relative merits of expanding a programme or fully

evaluating it. It is probably not wise to spend large amounts of money on

evaluation of a very small programme.

4. In the United States, although less so in Europe and many other OECD

countries, most economic development programmes are funded through

forgone tax receipts rather than budget expenditure. The costs of economic

development initiatives are thus less visible and often are not perceived as real

costs by local officials and the electorate. There appears to be less pressure to

evaluate programmes funded through forgone tax receipts, although pressure

groups in the USA are beginning to draw attention to the issue.

5. Following a similar logic, often economic development carried out at the

local level is not funded locally but from national, regional or other funds.

Where little local money is being spent, local governments and agencies are

less likely to wish to evaluate a local programme, although other funders

may step in.

6. There is a fundamental mismatch between political and economic

development timeframes. Thus, the political timeframe tends to be an

election cycle, whereas economic development can take one to two

decades. Local politicians are likely to be reluctant to sanction major longterm evaluations if they will provide little in the way of supporting evidence

for their current activities.

7. Many local development professionals seem to be content to refer to case

study examples of best practice rather than commissioning and using their

own evaluations. This may reflect the lower cost of best practice exchange



226



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



9.



A COMMENTARY ON BRIAN RUBSON’S PAPER AND THE WORKSHOP “AREA-BASED POLICY EVALUATION”



and the attractiveness of keeping up with new trends presented in

conferences and publications. But, whilst best practice exchange can be

useful, especially when it is based on information from fully evaluated

programmes, it is not a substitute for direct evaluations of local

programmes. The problem here appears to be a scarcity of informed

consumers to drive forward good quality area-based policy evaluation at

local level.



Capacity building in area-based policy evaluation

In response to the above hypotheses, workshop participants suggested

certain actions that might increase the quantity and quality of evaluation of

area-based policies and their use by local policy makers and politicians in

future programme design and implementation.





Encourage mechanisms that lead to local funding of programmes, such as

central government block grants that allow local governments and agencies

flexibility in the choice and design of programmes for their areas or

programmes that enable local governments to use retained locally-raised

taxes for regeneration purposes. Local funding of programmes will create

local pressure to evaluate that spending and lead to a greater sense of local

ownership of the evaluation findings and commitment to using them.







Encourage evaluations undertaken in collaboration by groups of local

governments or development agencies or supported by a higher-level

agency in order to address problems of externalities, economies of scale and

high evaluation cost as a percentage of programme budgets.







Address differences between the timescales of evaluators and politicians and

policy makers to secure greater demand for and use of evaluation. This means

stressing to politicians and policy makers that they should engage in economic

development as a long-term exercise, for example by setting up arm’s length

development agencies with long-term goals, and that they cannot expect

evaluators to demonstrate immediate success. At the same time, it should be

stressed to evaluators that politicians and policy makers need interim results

to help justify the continuation of programmes that appear to be working or to

adjust programmes that appear not to be running well.







Similarly, address differences of language between evaluators and

politicians and policy makers, again to increase the demand for and use of

evaluation, by increasing the understanding of politicians and their

advising officials on evaluation techniques and implications and

simplifying the way evaluators present their findings.







Improve the capacities of local policy makers to commission and use areabased policy evaluations, including developing expertise on how to judge

the quality of evaluation work and how to feed it into the next phase of



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



227



9.



A COMMENTARY ON BRIAN RUBSON’S PAPER AND THE WORKSHOP “AREA-BASED POLICY EVALUATION”



policy and practice improvement. Such client capacity building could work

through education and information exchange to local development

professionals. The following specific steps may be suggested:

– Forums could be developed in which experience could be exchanged

between people involved in taking forward local area-based policies and

using evaluations locally.

– Centres of excellence for training in area-based policy methods might be

established, where the techniques and uses of evaluation could be an

important component.

– Public dissemination of examples of good quality evaluations could be

encouraged, for example using internet sites or magazines and

publications aimed at local regeneration practitioners.



228



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



ISBN 92-64-01708-9

Evaluating Local Economic and Employment Development

How to Assess What Works among Programmes and Policies

© OECD 2004



Chapter 10



Evaluating Business Assistance Programs

by

Eric Oldsman,

Nexus Associates Inc.

and

Kris Hallerg*

Operations evaluation department, World Bank,

Washington, USA



* The views expressed here are those of the authors and do not represent the policies

of the World Bank Group.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



229



10.



EVALUATING BUSINESS ASSISTANCE PROGRAMS



Foreword

Governments around the world are supporting a wide range of business

assistance programs that aim to promote the development of private firms,

particularly small and medium-sized enterprises (SMEs). Despite the level of

resources committed to these programs, there has been relatively little effort

devoted to determining whether these programs have indeed been successful

in achieving intended outcomes. On the whole, evaluations have tended to

rely on inherently flawed before-and-after studies or potentially biased

testimonials from gratified customers.

However, there are better alternatives available to governments. Surveys

of potential beneficiaries clearly have a place in evaluations, enabling

evaluators to glean useful information on the perceptions of participants. But

care needs to be given to ensure that surveys address critical aspects of

program design, are worded in a way that takes the counterfactual directly

into account, and are based on representative samples.

That said, under certain circumstances, participant judgment may not

provide sufficient evidence of program impacts. Here, governments may want

to employ more rigorous experimental or quasi-experimental designs to

provide valid estimates of the impact of particular programs, controlling for

extraneous factors that may influence observed changes in performance.

Statistical techniques can be used to test various hypotheses concerning the

impact of key variables. Moreover, to the extent possible, evaluations should

include case studies based on rich narratives to explain causal mechanisms and

identify elements of the program design that need to be modified. Finally,

regardless of the particular approach, all evaluations should be based on clear

statements detailing the target population, intended outcomes, and

assumptions concerning the links between program activities and stated goals.

This paper seeks to provide government officials with a better

understanding of the critical issues involved in program evaluation and the

various tools that can be used in carrying out such studies. It focuses

specifically on quantitative methods that can be used in summative

evaluations of business assistance programs targeted to SMEs.



230



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Chapter 9. A Commentary on Brian Rubson’s Paper and the Workshop “Area-based Policy Evaluation”

Tải bản đầy đủ ngay(0 tr)

×