Tải bản đầy đủ - 0 (trang)
Chapter 5. Four Directions to Improve Evaluation Practices in the European Union: A Commentary on Timothy Bartik’s Paper “Evaluating the Impacts of Local Economic Development Policies on Local Economic Outcomes: What has been done and what is doable?”

Chapter 5. Four Directions to Improve Evaluation Practices in the European Union: A Commentary on Timothy Bartik’s Paper “Evaluating the Impacts of Local Economic Development Policies on Local Economic Outcomes: What has been done and what is doable?”

Tải bản đầy đủ - 0trang

5. FOUR DIRECTIONS TO IMPROVE EVALUATION PRACTICES IN THE EUROPEAN UNION



Introduction

The aim of this commentary is to place the techniques and methods

reviewed and recommended in Timothy Bartik’s paper “Evaluating the

Impacts of Local Economic Development Policies on Local Economic

Outcomes: What Has Been Done and What is Doable?” into the context of the

European Union (EU) (looking in particular at the local/regional economic

development programs co-funded by EU Structural Funds). Can we say, on this

side of the Atlantic, that EU-sponsored local/regional economic development

programs have been subject to rigorous evaluations, as Timothy Bartik can

c lai m is the cas e for a numb er of US prog rams ? Us ing the s am e

methodological framework contained in Timothy Bartik’s paper I argue that on

this side of the Atlantic we could improve evaluation practices in four

different directions:

1. being clearer on what rigorous evaluation is;

2. recording better data;

3. incorporating evaluation needs into policy design;

4. exploiting the heterogeneity of regional implementation designs across the

EU.

It is appropriate that, as the focus of Timothy Bartik’s paper is

predominantly on evaluating business incentive programs, this commentary

is largely based on my on-going experience in evaluating EU-sponsored

investment incentives to small and medium enterprises (SMEs) in Objective 2

(Ob. 2) areas1 and comparing these incentives with US Federal “Empowerment

Zones”. To my knowledge, however, current and past evaluations of incentive

programs for SMEs in Ob. 2 areas are somehow representative of general

evaluation practices adopted for other local economic development policies

implemented in the EU.



Being clearer on what rigorous evaluation is

This section of the commentary needs the statement of a short premise.

Prevailing evaluation terminology is slightly different between the US and the

EU for local economic development programs (particularly as regards

evaluations of programs co-funded by the EU Structural Funds). Terms largely

used in the EU such as “ex ante”, “in itinire” and “ex post evaluation” are not so

common in the US evaluation literature. The reason for this is that what in the



144



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



5. FOUR DIRECTIONS TO IMPROVE EVALUATION PRACTICES IN THE EUROPEAN UNION



EU is referred to as “ex ante evaluation” is often in the US referred to as

“program feasibility assessment” and is strictly considered as a part of the set

of broad programming activities needed to effectively design policy

interventions. “In itinere evaluation” is very often labelled as “monitoring

program activities”. Readers should be clear that Timothy Bartik’s paper is

mostly concerned with what in the EU would be referred to as “ex post

evaluation” (perhaps with some interview and focus group methods that in

the EU would also be referred to as “in itininere evaluation”). In agreement with

Timothy Bartik’s terminology I will in this discussion adopt the term

“evaluation” in place of “ex post evaluation”.

As is clearly presented in Timothy Bartik’s paper, three complementary

approaches are most needed for rigorous evaluation of local economic

development policies that provide business development incentives:

1. outcome impact evaluation on proximate dimensions of business activity;

2. estimating fiscal and employment benefits on the overall local economy

through regional model (such as REMI2 or Implan3);

3. surveys of clients and client focus groups to improve effective management

of the program.

Outcome impact evaluation on proximate dimensions of business

activity (such as employment or investment expenditures) [approach a)] is a

core component of rigorous evaluation. It yields crucial evidence on the

proximate effectiveness of the program by estimating how “what happened”

differs from “what would have happened but for” the policies. Such impact

estimates can then serve as a basis for estimating the fiscal and employment

benefits of the program on the overall local economy through regional macroeconometric models that may allow an assessment of the cost-effectiveness

of the intervention [(approach b)]. Surveys of clients and client focus groups,

instead, can be run quite independently from both approaches a) and b) and

constitute what would be referred to as “in itinere evaluation” in the prevailing

terminology of EU Structural Fund evaluations.

In place of rigorous outcome impact evaluation, current evaluation

practice for programs co-sponsored through the EU structural funds often

attempt to measure only “what happened” in the target areas (or firms)

instead of estimating differences between “what happened” and “what would

have happened but for” the policies. If precisely measured, knowing “what

happened” can be useful as it yields important information on the program

activity that could help to effectively manage the program. However, as we are

strongly reminded in Timothy Bartik’s paper, it has to be clear that measures

of “what happened” are not enough for rigorous evaluation.

Examples of such practice can be found in the thematic evaluation report

on the Structural Fund impacts on SMEs commissioned by the European



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



145



5. FOUR DIRECTIONS TO IMPROVE EVALUATION PRACTICES IN THE EUROPEAN UNION



Commission (Ernst and Young 1999) and two reports summarizing the

European Commission’s evaluation of the 1989-93 Ob. 2 programs (Malan 1998,

Bachtler and Taylor 1999). In Ernst andYoung (1999) and Malan (1998),

judgments on the impact of EU Ob. 2 programs on target areas are formed by

comparing the economic growth of the Ob. 2 regions with growth in the group

of non-Ob. 2 regions in the EU. However, such comparison would identify the

actual impact of the Ob. 2 programs only if assisted and non-assisted areas

would have performed in the same way without the intervention. Assigning

all credit for any performance difference between assisted and non-assisted

areas to the program does not constitute rigorous evaluation practice unless

sound statistical/econometric methods (e.g. Bartik 1991, Bartik and Bigham

1995, Bondonio 2000, Bondonio 2002, Manski 1995, Moffit 1991, Smith 2000) are

used to test and separate performance differences due to different preintervention characteristics between assisted and non-assisted regions.

In some evaluation reports commissioned by Italian reg ional

administrations to evaluate the business incentive measures implemented in

Italian Ob. 2 areas [e.g. Ecoter “Docup Ob. 2: Rapporto di valutazione finale”

(Docup Ob. 2: Final Evaluation Report), prepared for the Piedmont Region,

1999] impact estimates of the program intervention are retrieved by summing

the number of jobs that assisted entrepreneurs reported in their application

packages would soon be created following the completion of the assisted

investment.

Other evaluation practices have produced impact estimates of cash

assistance programs for SMEs in Ob. 2 areas by counting the total number of

jobs that interviewed entrepreneurs indicated as jobs that would have not be

created absent the program assistance (e.g. Malan 1998 and Ernst and Young

1999). As argued by Timothy Bartik, such an approach might be biased by the

tendency of assisted entrepreneurs to claim that cash benefits had a large

impact, so as to keep the cash coming in the future.

It has to be noted, finally, that applying REMI, Inplan and other regional

macroeconomic models to estimate fiscal and employment benefits on the

local economy does not constitute rigorous evaluation if such models are

estimated based on inputs that are unreliable measures of the program

outcome impact on proximate dimensions of business activity. Lacking

reliable evidence from rigorous outcome impact evaluation, impact estimates

on the overall local economy would be inaccurate if they were based on the

wrong inputs given to the macroeconomic regional models. For example, in

the final evaluation report prepared by Ecoter for the Piedmont Region in 1999,

the overall employment impact on the local economy of Ob. 2 areas is

estimated by applying a set of multipliers from a macroeconomic regional

model directly to the total figure for all approved investments in the area (a



146



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



5. FOUR DIRECTIONS TO IMPROVE EVALUATION PRACTICES IN THE EUROPEAN UNION



procedure which implicitly assumes that no investments would have occurred

in the area in the absence of the program intervention).



Recording better data

Almost every existing evaluation report of geographically-targeted

business incentive programs sponsored by the EU structural funds stresses

the need to obtain better data on program activity to improve the quality of

evaluation. Improving the program monitoring systems is typically the

suggested remedy in order to yield more precise and reliable data on the

program beneficiaries and the amounts of the cash assistance and/or services

offered to them (see for example Ernst and Young 1999).

However, appropriate statistical methods often exploit performance

differences between assisted and non-assisted firms (or areas) and before and

after the program intervention to retrieve reliable net outcome impact

estimates of the program intervention (e.g. Bartik 1991, Bartik and Bigham

1995, Moffit 1991, Smith 2000). Thus, it has to be clear that having the best data

solely on assisted businesses or target areas is not enough for rigorous

evaluation. To properly use such statistical methods would instead require

good data on both assisted and non-assisted businesses (or target and nontarget areas). For the case of spatially targeted business incentive programs,

moreover, rigorous evaluation would greatly benefit from data recorded at the

plant level by national/European statistical systems that match employment

information from employer records with socio-economic data on residents of

small statistical geographic units.

In the EU, NUTS_3s4 are the smallest current geographical units at which

data from the official statistical systems are currently easily available with

good reliability for comparisons across time. However, NUTS_3s are not small

enough to allow the boundaries of important assisted areas (such as the Ob. 2

areas) to be precisely reconstructed.

In Greenbaum and Bondonio (2003), characteristics of assisted areas for

the US Federal “Empowerment Zone” (EZ) programs and EU Ob. 2 areas were

analyzed and compared. In the US, data that precisely matched all EZ

boundaries could be retrieved by combining sets of census tracts (standard

geographic units used by the US Census Bureau). However, for the EU no exact

measure of Ob. 2 area characteristics could be easily retrieved. Only the

availability of reliable data for small standardized geographic units (such as

the NUTS_5s) would have allowed an acceptable reconstruction of Ob. 2 area

characteristics. At present, however, NUTS 5 data are difficult to obtain and

very unreliable for comparisons across time, as they are based on city

administrative boundaries that frequently change over time.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



147



5. FOUR DIRECTIONS TO IMPROVE EVALUATION PRACTICES IN THE EUROPEAN UNION



As reliable panel data are important for rigorous analysis, evaluation

practices would greatly improve in the EU as a result of building integrated

statistical systems that yield easily accessible data sorted by small geographic

units that remain stable over time (or for which changes are limited and easily

traceable).

Integrated EU data systems should also include registries of assisted

firms from all sources of public assistance sorted by EU nations and/or

regions. Creating such registries is very much needed in order to ensure that

assisted firms are compared to non-assisted firms and not to firms receiving

public subsidies from sources other than EU sponsored programs. “De Minimis”

rules that impose caps on the total amount of public assistance receivable by

EU firms are slowly inducing administrations of individual EU countries to

create registries of all subsidized firms. Integrating such regional registries in

a unified easily accessible European archive would be of great help to enable

more rigorous evaluation to be performed throughout the EU.



Incorporating evaluation needs into policy design

As reported in Timothy Bartik’s paper, reliable data for rigorous

evaluation can also be obtained by incorporating some evaluation needs into

the policy design of local economic development programs. This option

should be given proper consideration in the EU as the implementation of

changes in European statistical systems may be drawn-out and expensive.

One way to obtain reliable data for rigorous evaluation would be to designate

assisted areas those boundaries exactly overlap the existing geographical

units (or groups of geographical units) of EU statistical systems.

Implementing policies with experimental protocols would be another way

to obtain data for rigorous evaluation. Very often, however, strong political

reluctance to exclude needy areas and/or firms from public assistance

undermines large-scale implementation of experimental protocols for local

economic development programs. Nevertheless, some form of experiment

could be acceptable and should be given proper consideration in the EU, in

particular using the procedure that Timothy Bartik describes of random

selection of firms for targeted marketing of the program. As suggested in the

paper, such an experimental protocol can produce a significant difference in the

program’s usage rate between the group of treated firms (those receiving the

marketing efforts) and the control group of firms not receiving the marketing

efforts. Differences in the program’s usage rate between the treated and the

control group of eligible firms could be exploited to retrieve reliable impact

estimates of the program intervention. As it does not cause any eligible firm to

be arbitrarily excluded from program assistance, the implementation of such an

experimental protocol might face minimal political resistance.



148



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



5. FOUR DIRECTIONS TO IMPROVE EVALUATION PRACTICES IN THE EUROPEAN UNION



Exploiting heterogeneity of regional implementation designs

across the EU

As stated in Timothy Bartik’s paper, statistical analysis using control or

comparison groups can give insights into why and how a program works,

provided that sufficient variation in program designs is observed and

accurately measured.

For EU-sponsored programs, plenty of variation in policy implementation

designs exists across the different regions where the Ob. 2 programs are

implemented. Such heterogeneity in policy design (in these and other EUsponsored programs) should not be considered a threat to the validity of the

analysis because it limits the comparability of evaluation results across the

EU. Rather, it should be viewed as a great opportunity for testing the

effectiveness of a variety of policy designs and differences in the generosity of

the cash incentives and/or services offered to assisted firms.

To take advantage of such heterogeneity, appropriate statistical methods

have to be implemented so that across-region variation in policy features is

adequately operationalised and region or country-specific independent

economic trends are controlled for and kept separate from the impact

estimates of the region-specific policy features (e.g. Bondonio, 2002). Moreover,

if plant-level data are available, it is important to note that the analysis can be

implemented by separating the observed business outcomes (e.g. employment

growth) into three components:





change attributed to new firms attracted to assisted areas;







change in incumbent firms;







change from firms that cease to trade.



Sorting business outcomes in such a way can be very useful as it would

allow investigation of whether certain policy features are more effective in

attracting new firms rather than countering decline in existing production (or

vice versa ). Incentives that appear to be appropriate for attracting new firms

could be recommended for target sites such as newly developed industrial

parks, rather than sites where the main targets of the intervention are firms

already operating in the assisted area.



Notes

1. Regions with declining industrial production eligible for EU-sponsored assistance.

2. Regional Economic Models, Inc. (REMI®), www.remi.com.

3. IMPLAN Group, www.implan.com.

4. NUTS stands for Nomenclature of Units for Territorial Statistics which is the fivetier hierarchical regional structure used to standardize the economic territories of



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



149



5. FOUR DIRECTIONS TO IMPROVE EVALUATION PRACTICES IN THE EUROPEAN UNION



the EU. NUTS_3 areas are in the middle of the hierarchical structure and are

formed by the set of geographic units composed of single second-tier sub-national

jurisdictions (comparable in many aspects to US counties). NUTS_1 areas (which

are the largest units of the hierarchical structure composed as groups of

contiguous regions or states corresponding to the largest sub-national

jurisdictions for each EU nation) and NUTS_5 areas (composed as the set of city or

town jurisdictions of EU nations) complete the hierarchy.



References

BACHTLER, J. and TAYLOR S. (1999), “Objective 2: Experiences, Lessons, and Policy

Implications”, Final Report, European Policies Research Centre, http://europa.eu.int/

comm/regional_policy/sources/docgener/evaluation/pdf/finalrep_full.pdf.

BARTIK, T.J. (1991), Who Benefits from State and Local Economic Development Policies?

Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.

BARTIK, T.J., BINGHAM R. (1995), Can Economic Development Programs be Evaluated,

W.E. Upjohn Institute for Employment Research, Kalamazoo, MI: Staff Working

Paper 95-29.

BONDONIO, D. (2000), “Statistical Methods to Evaluate Geographically-Targeted

Economic Development Programs”, Heinz School Working Papers No. 2000-5,

Carnegie Mellon University, www.heinz.cmu.edu/wpapers.

BONDONIO, D. (2002), “Evaluating Decentralized Policies: A Method to Compare the

Performance of Economic Development Programmes across Different Regions or

States”, Evaluation, Vol. 8, No. 1, pp. 101-124, 2002.

GREENBAUM, R. and BONDONIO, D. (2003), “A Comparative Evaluation of Spatially

Targeted Economic Revitalization Programs in the European Union and the United

States”, ICER-International Center for Economic Research Working Paper Series

No. 3/03, January 2003.

ERNST and YOUNG (1999), “Thematic Evaluation of Structural Fund Impacts on SMEs”,

Synthesis Report, European Commission.

MALAN, J. (1998), “Translating Theory into Practice: Lessons from the Ex Post

Evaluation of the 1989-93 Objective 2 Programmes”, paper presented at the 1997

Seville Conference on Evaluation Practice in the Field of Structural Policies, http://

europa.eu.int/comm/regional_policy/sources/docconf/seville/sevil_en.htm.

MANSKI, C.F. (1995), Identification Problems in the Social Sciences. Cambridge, MA and

London, UK: Harvard University Press.

MOFFIT, R. (1991), “Program Evaluation with Nonexperimental Data”, Evaluation Review

15, 3:291-314.

SMITH, J. (2000), “A Critical Survey of Empirical Methods for Evaluating Active Labor

Market Policies”, Swiss Journal of Economics and Statistics 136(3): 1-22.



150



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



ISBN 92-64-01708-9

Evaluating Local Economic and Employment Development

How to Assess What Works among Programmes and Policies

© OECD 2004



Chapter 6



The Evaluation of Programs aimed

at Local and Regional Development:

Methodology and Twenty Years of Experience

using REMI Policy Insight

by

Frederick Treyz,

Ph.D.

and

George I. Treyz,

Ph.D.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



151



6. THE EVALUATION OF PROGRAMS AIMED AT LOCAL AND REGIONAL DEVELOPMENT



Foreword

Policy makers need to evaluate the total effect of local and regional

programs in order to make informed decisions. Development proposals have

economic, social, and demographic implications that go well beyond their

direct effects. To understand these effects, analysts need to use a

comprehensive economic forecasting and simulation model. Local and

regional policy analysis models show the full effects of policy changes on the

local economy, including socioeconomic consequences that may otherwise be

unforeseen or unrecognized.

This paper describes the REMI Policy Insight model, the leading regional

economic forecasting and policy analysis model. Over one hundred institutes,

universities, government agencies and other organisations use custom-built

REMI models specified to states, counties, cities, and other regions. These

model users are located primarily in the US, but also include organisations

using or planning to use models for regions in Belgium, France, Germany, Italy,

the Netherlands, Spain, and the United Kingdom. The EU Commission has

recently contracted REMI to develop REMI models for evaluation of structural

fund investments.

Analysts use the REMI model to evaluate the economic effects of

economic development programs, transportation infrastructure investments,

environmental and energy regulations, and other policies that have an effect

on the regional or local economy. REMI studies include evaluations of highspeed rail, new highways, business tax incentive programs, water resources

issues, air pollution controls, electric utility deregulation, and hundreds of

other applications. Often, users incorporate a REMI analysis into their process;

for example, evaluating all potential business relocation proposals for a given

state or city.

REMI provides a comprehensive modeling framework that shows total

policy effects, even those that are not anticipated. For example, a policy to

reduce air pollution may have cost consequences for businesses that will

reduce competitiveness, but the policy may also have the non-pecuniary

effect of making the area a more pleasant place to live. In this case, the loss of

competitiveness will reduce output but the cleaner air will lead to inward

migration that will increase the labor force. These increases will in turn

increase labor productivity and reduce wages, thereby cutting costs and

increasing competitiveness, which will increase economic activity. The net



152



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Chapter 5. Four Directions to Improve Evaluation Practices in the European Union: A Commentary on Timothy Bartik’s Paper “Evaluating the Impacts of Local Economic Development Policies on Local Economic Outcomes: What has been done and what is doable?”

Tải bản đầy đủ ngay(0 tr)

×