Tải bản đầy đủ - 0 (trang)
Chapter 1. Introduction and Summary Evaluating Programmes for Local Economic and Employment Development: an Overview with Policy Recommendations

Chapter 1. Introduction and Summary Evaluating Programmes for Local Economic and Employment Development: an Overview with Policy Recommendations

Tải bản đầy đủ - 0trang



he papers brought together in this volume were first presented at the

conference “Evaluating Local Economic and Employment Development”, held

in Vienna in November 2002. This conference was organised by the OECD’s

Local Economic and Employment Development (LEED) Programme, with

financial and logistical support from the European Commission (DG

Employment) and Austria’s Ministry of Economic Affairs and Labour. The

holding of the conference was motivated by the widespread perception that

there is a deficit in many OECD member countries with respect to the volume

and quality of evaluative research on policies and programmes used to

enhance local development. Why, for instance, is the evaluation literature on

local development so relatively thin? Is this a result of inadequate public

commitment to and practice of evaluation in this field, or perhaps a symptom

of conceptual and methodological difficulties particular to local development?

These and other issues were explored in the conference papers and


The conference attracted leading international figures in the field and

sought to do three things: to consider how governments use evaluative

research; to examine best-practice in evaluating the schemes most frequently

used for local economic and employment development, and to consider

whether rigorous evaluation methods can be used to assess the impacts on

entire localities of multi-instrument strategies and programmes.

Use and misuse of evaluation

It is always the government’s responsibility to ensure that public money

is well spent, as alternative uses of funds constantly compete for policy

spending priorities. The objective of evaluation is to improve decision-making

at all levels – in management, policy and budget allocations. Evaluation is

receiving renewed attention in OECD countries and is recognized as

“important in a result-oriented environment because it provides feedback on

the efficiency, effectiveness and performance of public policies and can be

critical to policy improvement and innovation” (OECD, 1999).

Evaluation is essentially about determining programme effectiveness or

incrementality, specifically the value-added of an operating programme or a

potential public initiative. This primary purpose has become somewhat obscured

by the fact that the work of evaluation has been largely focused on so-called

formative evaluation activities, which provide information in improving




programme design and operations. Accurate information at this level is

important but insufficient in a citizen-focused management regime that requires

judgements of worth or merit. In this context, there is a growing demand for

impact (sometimes termed “summative”) evaluations (Canada, 2000). These are

systematic attempts to measure the effects, both intended and unintended, of

some government intervention, or mix of interventions, on desired outcomes.

Such evaluative practices range widely in their complexity and rigour, often using

comparative analysis across time, across participants and non-participants, or

across detailed case studies. They typically rely on pre-post programme analysis

(“single-system methods”), experimental or quasi-experimental designs, or

detailed analyses of single cases that may be more feasible to apply in practice

settings than in control-group settings. Design choices notwithstanding, such

evaluations also require reliable and valid measures, as well as statistical

procedures to determine the significance of an intervention on the outcome of

interest. By establishing the links between stated policy, ensuing decisions, and

impacts, evaluation provides an important learning and feedback mechanism,

regardless of the specific moment of the policy process (State Services

Commission, 1999). Building evaluation criteria into policy proposals forces an ex

ante focus on desired outcomes. And ex post evaluation is an important tool for

identifying policy successes and failures. Taken together, ex ante and ex post

evaluations provide the critical evidence in support of results-based


Yet there is a low perceived demand for good evaluation of public policy

in general and of local development in particular, depending upon the country

and time in question. Numerous explanations for this have been offered,

relating to both the production and uses of evaluation. On the production side,

one is reminded of Henry Kissinger’s reference to the heat of politics, in which

the urgent steals attention from the important. Evaluation gets crowded out by

other, immediate demands from ministers, especially against the background

of fluctuating policy settings, the long timeframes needed for results to be

realized, and the need to allocate funding to develop and sustain the

necessary evaluation resources and technical staff capabilities (whether as

evaluators or intelligent customers of evaluations). Different evaluation

techniques also carry different price tags, with the gold standard of long-term

random assignment experiments at one end of the spectrum, and process

evaluations at the other. Where choices are forced, they are often in favour of

the least expensive approach. Methodological issues also factor into policy

managers’ reticence towards evaluation. Evaluation against outcomes is

considered just too hard:

“… evaluation never provides uncontroversial answers. All social policy

evaluation is plagued by the problem of the counter-factual – you never




have a control. All experience suggests it is expensive, difficult, and

controversial.” (State Services Commission, 1999).

Some also note the limitation of evaluation estimates of underlying

population parameters or typical results. Viewed from the practical lens of

public administration and public policy, one major issue with contemporary

analysis is the overemphasis on average cases. While such analysis provides a

great deal of useful baseline policy information, it is often not the behaviour of

the typical and undistinguished that concerns us the most. More often, it is

the exception to the statistical norm – the agencies that represent good or

excellent practice, the identification of high risk programmes, or programmes

that can meet multiple goals simultaneously such as efficiency and equity –

that demands recognition or remedial attention (Meier and Gill, 2000). The

focus on the high-performing or high-risk cases may highlight the reasons

that separate them from the typical. The current state of evaluation practice

does not handle such information demands well.

It is a long-standing observation that evaluation can be a double-edged

sword in its uses. The very nature of a detailed scrutiny is a bottle half-full or

half-empty, as even exemplary programmes have warts that may present

politically-sensitive communication challenges to government. Concerns about

how the results of evaluation might be used figure most prominently where

self-interested stakeholders prevail – ministers and public servants might be

equally attached to certain policies and reluctant to see them scrutinized.

Leadership courage at the highest level is often needed to resolve such issues

and to protect the evaluation function and its proponents. Thus in a number of

jurisdictions there is the dilemma of nurturing an environment of transparency

that avoids self-interest and capture while, at the same time, risk managing the

evaluation function. Integrity is at the core and its observance or not will

determine the level of public confidence in the evaluation function and its uses

to improve accountability, management practices and budgeting.

Since the mid-1990s, sustained efforts to modernize comptrollership by a

number of OECD countries have created some necessary conditions – and

suggested mechanisms for government ministries – to develop a capacity for

outcome evaluation. New performance reporting to government treasury

departments increasingly demand ex ante information on how programme

activities contribute to the achievement of outcomes, as well as ex post

information on progress in working towards those outcomes. Budget

processes today reflect incentives to encourage ministers to focus on

continual priority-setting between low- and high-value policy results. At the

same time, the field of evaluation is rapidly advancing in terms of the creation

of rich panel data, new techniques and computing technology. It is on such a

note of promise that we now turn our attention to the state of the art in

evaluating local and regional development.




Evaluating local development

Throughout OECD Member countries significant resources are dedicated to

programmes and policies to foster local and regional development. Bartik, in this

volume, describes the magnitude of expenditures on subnational development in

the United States. He cites an estimate that US$20-30 billion is assigned annually

by local and state governments just to programmes of targeted business support.

An additional amount of around US$6 billion is spent each year by the Federal

government. These allocations take the form of direct spending on programmes

and, overwhelmingly, tax incentives. Furthermore, such figures would be

considerably enlarged – even doubled – if more general state and local

government spending and tax incentives for business were included. In England

and Wales, a 1999 study of local councils concluded that these spend “322 million

pounds on economic development each year, and also manage billions of pounds

of domestic and European regeneration funds” (Audit Commission, 1999). Public

outlays on such a scale clearly merit a major investment in efforts to evaluate

their effectiveness and efficiency.

Especially in poorer places, programmes and policies to promote local

economic development encompass interventions in a wide range of sectors.

Initiatives include actions in markets for property, labour, business

information and advice, financial, health, education, and social services,

policing, infrastructure, taxation, and institution building of different sorts.

Many programmes have a long track record, in some cases stretching over a

number of decades. The key features of such interventions are the subject of

an abundant literature. The papers in this volume focus on an array of

programmes that have their primary impact on local labour markets and/or

business development.

By acknowledging the links and interactions between different

interventions that have local development as their focal point, this volume

addresses a departure from traditional programme evaluation. The

conventional approach is to evaluate individual policy instruments and

programmes against their explicitly stated objectives. In this way, programme

evaluations tend to produce isolated and often disappointing findings,

without due regard to the interaction and cumulative impact of policies that,

by design or not, work in a “target-oriented” way (Schmid, O’Reilly and

Schömann, 1996). This broader perspective recognizes that policies designed

to influence area-based development do not exist in isolation and that an

integrated approach is warranted.




There are too few high-quality assessments of local development

policies and programmes

Given the magnitude of resources used for local economic and

employment development, few countries have made a commensurate

investment in generating rigorous evaluative evidence on which policies work

and which do not. It is not simply that, frequently, there is limited funding of

evaluation. In addition, as discussed below, there is considerable variation in

the usefulness of many studies that purport to be evaluations. For instance,

the local development literature is replete with case studies of local areas and

their development programmes. However, many such studies simply describe

a particular locality at a given point in time. They lack the longitudinal data on

the area and/or its residents that might help to trace the causes of changes in

economic or social circumstances. Consequently, policymakers are frequently

unsure about which policy choices are best suited to which circumstances,

and which policies should not be considered at all. 1 Indeed, the

aforementioned study on councils in England and Wales found that many “are

uncertain whether their efforts result in real opportunities for local people”.

At least for locally-oriented programmes, evaluations often use methods

and criteria that are insufficiently stringent to serve as a guide to policy. For

example, there are around 1 000 business incubators in the United States.

Most receive some public funding and many have local or regional

development goals. There are numerous detailed studies of incubation

schemes. However, to the knowledge of the authors, there has not yet been an

assessment of business incubators in the United States that has used a control

group methodology. In the absence of control-group assessments, ascribing

changes in enterprise performance to the effects of incubation may be

mistaken. Similarly, Boarnet (2001) shows that despite the popularity of

“enterprise zone” policies, and the existence of a large number of evaluative

studies, little of this research has been directly relevant to policy. This is

because studies have been unable to identify which economic developments

in the target areas would likely have occurred in the absence of the

programmes. Indeed, across a range of programmes commonly used for local

economic and employment development, too little is known about the net

outcomes that can reasonably be ascribed to interventions.

Other difficulties also hinder the evaluation challenge. For instance, there

is a tendency in many studies to examine (the costs of) output delivery rather

than the achievement of net outcomes. That is, the focus of reporting is often

on such variables as the numbers of training places provided, firms assisted,

incubator units established, volume of credit disbursed, area of derelict land

reclaimed, etc. The more complex question of how the supply of these outputs

has affected the status of target beneficiaries usually receives less attention.




When relevant data are gathered, they are often collected at the end of a

programme’s life, with no baseline or ex ante information. Furthermore, it is

not uncommon for evaluations to be produced by the very sponsors of the

programmes being assessed.

Job creation is the most common yardstick of local development policy.

However, studies often use opaque or non-standard measures of job creation.

This makes cost-per-job-created claims unreliable and even misleading. For

example, if a company has been created in, or moves to, a given location, and

hires ten employees, this is invariably publicised as the creation of ten jobs.

However, if the ten recruits simply moved from existing positions there may

have been no net job creation (such redeployment might occur if jobs have

been displaced from existing firms following competition from the new

enterprise. Recruits might also have left old jobs voluntarily on account of

more attractive conditions in the new openings). Typically, only a part of new

hiring involves net job creation. But it is also possible that local welfare could

rise even if all hiring involved redeployment. For instance, by comparison with

the displaced positions the new posts might offer income or other gains to

hirees. Conversely, recruitment could be associated with a decline in welfare if

persons who are redeployed from displaced positions experience income or

other losses. Reports of job creation do not always take such considerations

into account.

Some studies also evaluate on criteria that are inappropriate. For instance,

job creation measures can be unsuitable to assessments of the business

development schemes that are a staple of local development strategies. The

effects of such enterprise support schemes can be had on a range of business

practices. They may impact, for instance, on the ability of entrepreneurs to

adopt advanced management practices, to manage a company’s inventory and

cash flow, to raise product quality and lower process waste, to enter overseas

markets, etc. Job creation can be a secondary effect of these outcomes, but need

not arise automatically. More generally, many business development

programmes enhance firm-level productivity. This can create pressure for

labour shedding if demand for firms’ output is static. Such considerations

underscore the need to properly align evaluation parameters with the nature of

the programmes being assessed. In the case of enterprise support, programmes

often need to be evaluated on how specific business development practices in

target firms have changed, rather than short-term impacts on job creation.

The overall paucity of high-quality evaluation is doubly regrettable in as

much as local development initiatives are often intended to serve pilot or

experimental functions. Policy piloting is, indeed, one of the claimed

justifications for local development approaches per se. But this experimental

function is squandered when programmes are not well evaluated. Indeed, in

this connection, it is worth noting that some fundamental propositions about




local development and the policies that promote it are only poorly understood.

For example:

There is little quantitative evidence for the purported efficiencies of local

development approaches per se. There are plausible generic reasons for

thinking that local design and implementation should be superior for some

types of policy and not others (for instance, superiority is unlikely in the

case of policies that involve significant scale economies). But it is rare to

find quantitative evidence of improvements in economic efficiency

stemming from an increased role for the local level.

There is limited understanding of the net effects of a range of business

support schemes. For instance, as already mentioned, business incubators,

despite their proliferation, have hardly been subject to systematic economic

assessment anywhere. Similarly, in OECD Member countries, micro-credit

programmes have rarely been evaluated using control-group techniques.

And for this report only one systematic study was found that examined the

local impact of self-employment support [see Cowling and Hayward (2000)].

There is minimal information available that quantifies the costs, benefits

and additionality associated with local partnerships, a frequent mode of

programme design and (sometimes) delivery.2

As Bartik notes in this volume, there is no direct empirical evidence for the

notion that local employment benefits will be superior when there is a

closer match between newly created jobs and the job skills of the local

unemployed. Similarly, there is little empirical support for the contention

that local employment benefits will be greater when local labor market

institutions are more efficient in job training and matching.3

Furthermore – and not particular to the local development arena – there

often exists a communication gap between policymakers and evaluation

professionals. For instance, Eisinger (1995) showed that 34 out of 38 US states

had conducted evaluations of at least parts of their economic development

programmes in the early 1990s. However, only 8 states had made changes to

programmes in the light of evaluation recommendations. Policy evaluation

appeared to have little incidence on policy formulation.

All of the above does not imply that good studies are unavailable, or that

development practitioners and policymakers are unaware of the issues at

stake. The various chapters in this volume document valuable examples of a

diverse range of high-quality evaluations. An early review, Foley (1992), points

to a variety of careful studies that have wrestled with complex issues of

programme deadweight, displacement, substitution and multiplier effects.4

Particularly in the United States, numerous thorough assessments have been

made of regionally-based science and technology extension services [see

Shapira (2003) for a review]. Enterprise zones have been assessed with




increasing sophistication [see for example, Bondonio and Engberg (2000) and

Boarnet (2001)]. Smith, in this volume, cites sophisticated evaluations that

have addressed such varied issues as the effects of casinos on employment,

and the growth impacts of public sponsorship of sports and sports facilities.

Various national and subnational governments have also created guidelines

and institutional capacities for evaluating local development programmes (see

for instance, HM Treasury, 1995).5 Among the counterpart organisations

working with the OECD, including regional and local development agencies,

ministries of labour and other public institutions, there is an intense interest

in evidence about what works and why. Nevertheless, the OECD-wide picture

is one of deficit with respect to the quantity and quality of policy-relevant

evaluation. And the evidence base is weak even with regard to a number of the

basic tenets of local development practice.

Why is there too little of the needed evaluative evidence?

There are a number of possible explanations for why the evidence base

for policy is weak. These include the following:

Possible objections to evaluation among programme managers and implementing

agencies. This might stem from fear that support will be withdrawn if

programmes receive a negative assessment. This is a problem for public

policy evaluation per se. Objections might also reflect the fact that the more

statistically sophisticated evaluations have often been useful in deciding

whether a policy has worked, but have been weak in describing how the

policy might be improved (Bartik and Bingham, 1997). Among programme

administrators, this may have reduced the perceived usefulness of evaluation.

Practical and methodological challenges of rigorous evaluation. Measuring such

general equilibrium effects as deadweight, displacement and substitution is

notoriously difficult. But evaluation of local development policies can

involve additional complexity given that effects on a geographic area are

being assessed in conjunction with effects on target groups (persons or

firms). Effects on target groups need not translate into effects on the local

area. This is the case, for instance, when created job vacancies are filled by

in-migrants, or when an improvement in skills among residents facilitates

their relocation, or when increased business activity leads to higher levels

of out-of-area input procurement. Such difficulties are further compounded

if evaluators have to consider how a number of policies interact across a

geographic area that might contain multiple target groups. In addition,

some local development outcomes can be difficult to quantify (such as

reduced fear of crime, in the case of neighbourhood policing initiatives, or

aspects of community capacity building).




The direct and indirect costs of evaluation. Direct costs can be particularly

significant for experimental or quasi-experimental forms of evaluation.

Programme managers also sometimes view evaluation and monitoring as

an intrusive source of administrative burden. In addition, evaluation can

itself involve economies of scale (with certain fixed costs, for instance, in

the collection and organisation of data) and of scope (as insights from one

evaluation might be applied to others). Such economies imply that

evaluations are often best sponsored and/or undertaken by higher levels of

government. It is therefore unlikely that local authorities will produce a

record of systematic evaluation that matches the extent of local policy


Incentives for local authorities to under-invest in evaluative knowledge. Because

the benefits from evaluation findings can accrue to agencies other than

those that sponsor the studies, local bodies may under-invest in evaluation.

Policy and programme overload. Evaluation can appear to be an unrewarding

investment when, as is often the case, government initiatives are numerous

and the population of active programmes changes constantly (a problem

made worse when programmes have multiple objectives).6 The extended

time horizon over which some programmes yield measurable effects might

also discourage policymakers from investing in evaluation.7

A weak understanding of evaluation techniques and principles, and a lack of

suitably trained evaluators, in many local authorities (a lack of in-house

evaluation capacities can also place local authorities in a disadvantageous

position when subcontracting evaluation studies).

In many countries, a lack of appropriate small-area data. In some cases, the

geographic units over which data is collected – in say health or education – do

not coincide with the units across which the local development programmes

act. In some contexts small-area data is simply unavailable.

Despite these disincentives, some OECD countries recognize the critical

importance of systematic, rigorous evaluations in public decision-making.

Canada, for instance, has made evaluation evidence obligatory for the renewal

or reauthorization of all federal programmes. This comptrollership

requirement is a powerful inducement to invest in and build an evaluation

culture across the federal government. Such a national commitment has not

typically been matched by subnational governments, which operate the

majority of local development programmes. Yet the possible benefits from a

greater quantity and quality of local development evaluations could be

considerable. Most obviously, improved evaluation could help local and

central authorities to allocate sizeable resources in an economically efficient

manner. “Best-practice” in different programme types could be gauged in a

meaningful way, such that different implementation modalities in given




programme types could be properly chosen. In addition, it is often observed

that evaluation can be a learning exercise for evaluators and policymakers.

Given that local development is a multi-sectoral endeavour, and that

programme goals are sometimes vague or even inappropriate, a potentially

important benefit of enhanced evaluation could result from encouraging

implementing agencies to clearly specify what the goals of a programme are.

Careful programme monitoring is also critical and complex

Monitoring is a tool that can furnish information essential to programme

management and superior programme outcomes. Monitoring can also ensure

close contact with beneficiaries. However, monitoring is generally not

equivalent to evaluation, being rarely concerned with issues of net

programme outcomes. Taking the example of labour market programmes,

Smith, in this volume, illustrates how performance standards need not (and

generally do not) serve as a good substitute for impact estimates unless there

is a systematic relationship between the two.

In addition to yielding information on programme implementation (and,

possibly, impacts), performance indicators can define subcontract

relationships with service providers and serve as an instrument of

accountability. Furthermore, the choice of performance measures for

programmes creates incentives that shape the ways in which services are

provided. When the continued funding of programmes depends on the

achievement of pre-specified performance targets, inappropriate performance

measures can have serious and sometimes difficult-to-foresee effects on

programme implementation and effectiveness. Accordingly, care is needed in

the selection of performance indicators.

For example, an incentive exists to increase client throughput if the

performance of a micro-enterprise scheme is assessed against the number of

persons who enter the programme. Clients may be encouraged into the scheme

– or be accepted when they should be dissuaded – regardless of probable

business outcomes. In a similar fashion, when loan repayment rates have been

used as the principal performance measure in micro-credit projects, staff have

sometimes used legitimate accounting procedures to turn in high published

rates of repayment, while programmes have been designed in ways that would

preclude the reporting of low repayment rates (Woolcock, 1999). Similarly, if

programmes are assessed against the number of enterprise start-ups they

bring about, then support might shift away from services that are likely to

enhance business survival. And funding which is based on output measures –

while having the virtue of administrative simplicity – involves making payments

after providers have incurred expenditures. This can cause support to be

directed towards types of services that require little initial spending (Metcalf

et al., 2001).




In general, single-variable performance measures invite distortions in the

running of programmes that have complex effects. Performance measures

should be sought that reflect the complexity of the programmes and outcomes

being monitored. For instance, using again the example of enterprise support

programmes, performance measures could combine data on start-up rates

– if enterprise creation is an underlying goal – with indicators of business

survival – because simply creating firms is not enough. Higher weightings

could be given to projects involving enterprises in which the size of capital

invested and expected business incomes are comparatively high (it is in such

firms that displacement is likely to be low). Similarly, to avoid the common

bias towards working with entrepreneurs who would be successful even

without the programme, higher weightings could be afforded to the

establishment and successful management of firms by individuals who face

barriers to enterprise (such as persons whose loan applications have been

rejected by a bank).

Metcalf et al. (2001) note that service providers may face different

performance measurement requirements from a range of programme

sponsors. In this regard, governments can act to ensure consistency of

performance measures across similar programmes. Bringing about a degree of

standardisation in requirements for performance data can reduce

administrative burden, especially for service providers that receive funds from

more than one source. Such standardisation could also improve comparability

across government-funded programmes, which would facilitate the

generalisation of best-practice.

It is also important that monitoring not be perceived principally as a

means of control. Service providers should be convinced of the utility of

measuring performance. In practice, monitoring is sometimes performed in a

perfunctory way. Service providers are often unconvinced that the data they

are asked to assemble are useful.

Overview of the papers in this volume

Evaluation research and policy learning

Ging Wong’s paper – Policy Learning Through Evaluation: Challenges and

Opportunities – brings together a rich variety of insights from the former

Director of the Canadian government’s largest evaluation service. The paper

situates the evaluation function within the broader context and processes of

policy formulation, as well as providing a careful exposition of what

evaluation is and is not. Clear distinctions are drawn between evaluation and

financial audits, and between evaluation and various performance-based

management and accountability systems. Wong concludes that the three

fundamental tasks of evaluation are: to facilitate public accountability;



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Chapter 1. Introduction and Summary Evaluating Programmes for Local Economic and Employment Development: an Overview with Policy Recommendations

Tải bản đầy đủ ngay(0 tr)