Tải bản đầy đủ - 0 (trang)
Chapter 16. A Commentary on the Workshop “Evaluating Territorial Employment Pacts”

Chapter 16. A Commentary on the Workshop “Evaluating Territorial Employment Pacts”

Tải bản đầy đủ - 0trang

16.



A COMMENTARY ON THE WORKSHOP “EVALUATING TERRITORIAL EMPLOYMENT PACTS”



“T



erritorial Employment Pacts” (TEPs) are encompassing, area-based

networks for the co-ordination and implementation of employment and

economic development policies in integrated projects. In both Austria and

Italy this approach was first introduced as an EU pilot program and

subsequently continued, in somewhat modified form, as an element of

national policy. It is important to note that the TEPs are not organisations in a

formal sense with, for example, their own budgets and employees but (more

or less) voluntary partnerships in which the constituent organisations retain

their individual identity. Only a relatively small amount of funding is provided

to cover the costs of the development and co-ordination of the TEP programs,

which leverages a disproportionately large amount of joint activity by the

member organisations.



The TEP networks themselves are extremely varied within and between

countries: They differ, for example, in the areas covered [entire provinces

(Austria) or local networks (Italy)], in the constellation of public and private

actors involved, whether their principal function is co-ordination and

planning (Austria) or operating local programs (Italy), and whether their focus

is more on employment (Austria) or economic development (Italy).

There are two special rationales for the TEPs that need to be considered

in designing an evaluation: 1) it is assumed that they yield synergy effects by

creating an encompassing formalised local coalition with shared goals and

co-ordinated projects. If no additional program resources are available, this

can only be achieved by increasing efficiency and effectiveness through

bundling of existing resources, for example, through better co-ordination of

local employment strategies. Focusing limited resources on a smaller number

of priority targets may also enhance policy outcomes. 2) Another major

rationale is to foster the development of social capital in the form of local

policy networks, which are assumed to have a favorable long-term impact on

local employment and economic development.

The papers presented by Paola Casavola and by Peter Huber on evaluating

territorial employment pacts in Italy and Austria, respectively, are both based

on ongoing evaluations of national programs. Both emphasize the objective of

promoting local networks for implementing economic and employment

policies and seek to develop an appropriate evaluation strategy different from

that of mainstream program evaluation. As Peter Huber argues: “Evaluating the

efficiency of institutions in implementing programs implies a different



382



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



16.



A COMMENTARY ON THE WORKSHOP “EVALUATING TERRITORIAL EMPLOYMENT PACTS”



counterfactual from that researched in much of the active labor market policy

literature.” Rather than asking what would have happened if an individual had

not participated, “the counterfactual here should answer the question: what

would have happened if the institution designing the measure had not existed.”

In her study of territorial pacts in Italy Paola Casavola provides a

quantitative assessment of the determinants of pact formation based on data

from 61 Italian districts. The general assumptions of the Italian program of

incentives for the formation of territorial pacts is that local employment and

economic development is fostered 1) by promoting dense networks of social

and economic co-operation and that 2) there is a positive “agglomeration”

effect to combining individual programs within a broader integrated policy

project.1 These interesting assumptions are not, however, the subject of her

paper. Instead her evaluation for the Italian Ministry of Economics and

Finance focuses on the intermediate goals that are the actual focus of the

Italian program: the progress of the integrated projects and the characteristics

of the local coalitions. Combining monitoring data on pact performance with

social and economic indicators and survey data on the development of the

pact coalitions, she investigates a number of key issues for 61 Italian territorial

pacts: 1) To what extent is successful pact formation affected by “initial

conditions”, i.e. social and economic conditions in the pact localities? 2) How

does the quality of the start-up process affect pact outcomes? In both cases

she uses “realization speed” measured in terms of the stepwise take-up of

authorized pact funds as the dependent variable. Briefly stated, she concludes

that the realization of pacts was not adversely affected by initial conditions,

i.e. this form of policy governance is also feasible in disadvantaged localities,

and that “pacts” in which all relevant actors were involved from the beginning

proceeded faster, which suggest that investment in the start-up phase is

particularly important for the success of territorial pacts.

Peter Huber’s paper reports on the evaluation design and initial results

from an evaluation of the Austrian territorial employment pacts as part of a

broader evaluation of European Social Fund Objective III programs in Austria

being carried out by the Austrian Institute for Economic Research (WIFO). His

focus is likewise processes rather than individual or labor market outcomes.

His overriding research question is an institutional counterfactual: How would

policies of the participating organisations have been different in the absence

of the TEP organisation?

The evaluation design focuses on three principal types of questions:

1) Strategy formulation including issues related to the quality of the pact

coalition and its success in achieving a strategic integration of the partners;

2) Analysis of implementation with particular attention to conflicts within

coalitions and the impact of the TEP on the participating institutional actors

as well as the possible long-term “multiplier effects” of “social capital” created



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



383



16.



A COMMENTARY ON THE WORKSHOP “EVALUATING TERRITORIAL EMPLOYMENT PACTS”



by the TEP experience. He shows the relevance of traditional program

evaluation categories (e.g. deadweight, displacement) even in a more

qualitative, case study of the impact of governance institutions. 3) Finally,

under policy-uptake he advocates giving particular attention to the incentives

for regional actors to engage in local policy coalitions, which depend on the

perceived costs and benefits for the potential coalition partners.

The discussion on the papers at the Vienna conference was largely

devoted to informational questions about the Italian and Austrian programs

(e.g. differences between pacts in Italy and Austria and the types of labor

market programs implemented) and technical aspects of the analysis (e.g. how

success is measured). More general themes related to the distinctive focus of

both papers on “governance issues” rather than individual program outcomes

and the difficulty of defining and measuring social capital. Both the Austrian

and Italian programs are based on assumptions about the importance of

encompassing networks for local employment and economic development,

which are not directly addressed in either of the papers.

Both papers illustrate the need for a different evaluation strategy when

assessing the impact of local governance structures. Standard evaluation

methodology in labor market research has focused largely on individual

program effects. Based on an explanatory model borrowed from medical

research, this type of evaluation has sought to estimate the impact of program

“treatments” on individual participants. In this methodological paradigm the

employment and earnings of program participants are compared with that of

an experimental or quasi-experimental control group of non-participants (the

counterfactual) in order to estimate net effects.2

This tradition of impact analysis has greatly increased our sophistication

in assessing the net effects of labor market programs. However this standard

model has two major shortcomings from the point of evaluation of local

employment and economic development policies:3

1. It tends to assume a simplified model of the policy process in which program

“treatments” are highly standardized, underestimating the importance of

local program variation. The actual “treatment” that program participants

receive may vary considerably for participants throughout the country. In

fact labor market programs provide complex services that are, at local

implementation, seldom uniform.4 Moreover, labor market programs in

many countries merely provide framework regulations that by design

permit a great deal of local variation in program content. This is the case,

for example, for many European Social Fund programs, of which the

Austrian territorial employment pacts are an example, as well as for major

national training and job creation programs in Germany and in other

countries with relatively decentralized public employment services.5 In



384



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



16.



A COMMENTARY ON THE WORKSHOP “EVALUATING TERRITORIAL EMPLOYMENT PACTS”



sum, program “treatments” are seldom so standardized as the analogy to

medical research suggests (Scheirer 1994). Local effects in program

evaluation can in principal be examined if the evaluation design provides

valid findings at the relevant regional and local levels.6

2. More importantly, the focus on individual program effects neglects a range

of other evaluation questions that are central to policy-analysis

(management and process evaluations, efficiency studies, cost-benefit

analyses, etc.) for which other methodologies may be more suitable (see de

Koning and Mosley 2001, Palumbo and Calista 1990, Scheirer 1994). For

example, structural issues related to the impact of changes in the

organisation of public employment service, including financing,

governance, and management structures, cannot be addressed in an

experimental or quasi-experimental framework. Similarly, evaluation

issues related the impact of local governance structures, like the territorial

pacts under consideration here, require a different evaluation design.

What’s different about evaluating territorial pacts?

1. Although in both countries the territorial employment pacts originate in an

EU initiative later incorporated into national policy, the “local” dimension is

in the first instance the territorial or area focus of policy, in contrast to

individual- or firm-oriented policies. Improvement in individual programs

and aggregate outcomes is sought by influencing collective behavior in

formulating and implementing policies at the local level. The objective is to

transform local implementation structures by promoting broad coalitions

in formulating and implementing policies. The central evaluation question

here is thus whether and to what extent this local effect is achieved and

not, in the first instance, the impact of program “treatments” on individual

participants.7

2. A second evaluation question is the impact of the programs run by the

territorial pacts on local labor market programs. Has the new form of

governance (TEPs) had an impact on the level and mix of local (labor

market) policy at the local level? Do the actors involved alter their priorities

and policy portfolios as a result of their involvement in the TEP coalition? A

major rationale for this type of implementation strategy is to increase the

efficiency and effectiveness of policy by improving co-ordination among the

different agencies and between public and private actors involved in local

policy implementation. For example, in most countries different

organisations are responsible for labor market policy and local economic

development policies. This seems to be a principal function of the Austrian

TEP model, which in most cases represent agreements between the regional

public employment service offices and the provincial authorities

responsible for economic development.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



385



16.



A COMMENTARY ON THE WORKSHOP “EVALUATING TERRITORIAL EMPLOYMENT PACTS”



3. Another distinctive evaluation question is the impact of program-induced

changes in governance on aggregate policy outcomes in the regions. In

order to answer this question a methodology different from the control

group method to assess the impact of programs on participants is required.

A weakness of this method even in the case of program evaluation is that it

does not take into consideration displacement, substitution and other

indirect effects at the aggregate level in the labor market. A regional-level

aggregate impact analysis can be used to assess effects at the local level

measured in terms of indicators of local employment and economic

development. It might utilize a time series or cross section framework (or

both), or a matched comparison of regional units.8 An aggregate impact

analysis would, for example, also be the appropriate framework for testing

the common assumption of these programs that enhancing “social capital”

in the form of local policy coalitions engenders better local economic and

labor market performance.

4. A final evaluation question is, last but not least, the classical question of

program impact on individual participants. Given the heterogeneity of the

territorial pacts and the assumption that local implementation matters, a

locally-focused evaluation is required, based ideally on data on local labor

market outcomes for participants and a control group. In principle

evaluation of local programs can be carried out using the same

experimental or quasi-experimental methods applied to national programs,

although this appears to be relatively infrequent. For example Leitner et al.

(2002) examine the impact of labor market policy measures of the Vienna

Territorial Employment Pact using a quasi-experimental method based on a

propensity score matching procedure. Data on program participants and a

control group of non-participants were drawn from social insurance records

and used to assess the net impact of the Vienna programs on participants’

employment and earnings.

Why was an impact evaluation of participation in local employment

programs using a control group method possible in this case? 1) The European

Social Fund mandates evaluation of its programs, although the actual quality of

the evaluations may vary greatly; 2) The economies of scale that make

evaluation of smaller programs in some localities too costly were not an

obstacle in Vienna; 3) Evaluation expertise was available (Institute for Advance

Studies [HIS]), which should be the case in most European countries; 4) The key

element appears to have been the ready availability of data from social security

records on the work careers of local participants and non-participants, which in

most European countries is not the case; 5) noteworthy also is the fact that the

final evaluation report of the 1999 local Vienna programs was completed in the

Spring of 2002, which is indicative of the trade-off between evaluation quality

and the need of program administrators for real-time feedback.



386



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



16.



A COMMENTARY ON THE WORKSHOP “EVALUATING TERRITORIAL EMPLOYMENT PACTS”



Notes

1. The implicit model of the first assumption is the success story of the Italian

industrial districts, which is attributed to their special “social capital”

endowment.

2. For an overview see, for example, the paper by Smith in this volume.

3. See also Mosley, H. and E. Sol (2001).

4. Mosley and Degen (1994) discuss the actual variation in services provided even

within highly structured UK training programs.

5. Schierer (1994) distinguishes between “aggregate” and “targeted” programs.

6. The variance in net program effects at the local level could serve as a measure of

(unobserved) variation in implementation.

7. In fact, the papers by Peter Huber (Austria) and Paola Casavola (Italy) raise the

question of what we mean by “local” evaluation, which refers not just to

evaluation of local programs but specific local effects.

8. See de Koning (2001) for a general discussion of approaches to aggregate impact

analysis.



References

DE KONING, Jaap and Hugh MOSLEY (eds) (2001), Active Labor Market Policy and

Unemployment, Cheltenham, UK: Edward Elgar.

DE KONING, Jaap (2001), “Models for Aggregate Impact Analysis of Active Labor Market

Policy” in de Koning and Mosley 2001.

LEITNER, Andrea, Angela WROBLEWSKI, Peter PRENNERM, Helmut HOFER, Andreas

SCHUH, Helmut MAHRINGER (2002), Evaluation der arbeitsmarktpolitischen

Mnnahmen des TBP Wien 1999, Vienna: Institut für Hưhere Studien.

MOSLEY, Hugh G. and Christel DEGEN (1994), “The Re organisation of Labor Market

Policy: Further Training for the Unemployed in the United Kingdom”. WZB

Discussion Paper FS I 94-205, Wissenschafts-zentrum Berlin für Sozialforschung.

MOSLEY, Hugh and ELS SOL (2001), “Process Evaluation of Active Labor Market Policies

and Trends in Implementation Regimes” in de Koning and Mosley 2001.

PALUMBO, D.J. and D.J. CALISTA (1990), “Opening up the Black Box: Implementation

and the Policy Process”, in Palumbo and Calista (eds.) Implementation and the Policy

Process: Opening up the Black Box, Greenwood Press, New York, pp 3-16.

PAWSON, Ray and Nick TILLEY (1997), Realistic Evaluation, London: Sage.

SCHEIRER, M.A. (1994), “Designing and Using Process Evaluation”, in J. Wholey,

H. Hatry, K. Newcomer (eds), Handbook of Practical Program Evaluation, San

Francisco: Jossey-Bass Publishers, pp. 40-68.



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



387



ISBN 92-64-01708-9

Evaluating Local Economic and Employment Development

How to Assess What Works among Programmes and Policies

© OECD 2004



Chapter 17



A Review of Impact Assessment

Methodologies for Microenterprise

Development Programmes

by

Gary Woller,

Associate Professor Romney Institute of Public Management,

Brigham Young University Utah, USA



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



389



17.



A REVIEW OF IMPACT ASSESSMENT METHODOLOGIES FOR MICROENTERPRISE DEVELOPMENT



Introduction

Over the last several years, OECD governments have invested millions of

dollars in microenterprise development programmes in OECD and lesser

developed countries (LDCs). Microenterprise development is based on a

couple of underlying premises: 1) self-employment is a key component in

creating economic opportunities for low-income persons with otherwise

limited employment or earning options, and 2) the primary constraints to

productive self-employment among low-income persons are access to

capital (loans) and training. For OECD governments, self-employment

expands the range of policy options to combat poverty in its many

manifestations. For the poor, self-employment expands the range of

livelihood and coping options. Self-employment policies appear to offer

particular benefits in economies characterised by chronic under or

unemployment or by high levels of informal economy participation by the poor.1

In theory, programme participants leverage loans and training to start

and expand micro and small enterprises thereby generating higher levels of

enterprise returns; higher enterprise returns translate into higher household

income; and higher household income is in turn invested in improved

household socioeconomic well-being. In practice, the specific impacts of

microenterprise development programmes are hard to pin down and harder

still to measure. Impact assessments require adoption of research

methodologies capable of isolating specific impacts out of a complicated web

of causal and mediating factors and high decibels of random environmental

“noise”, as well as attaching specific units of measurement to tangible and

intangible impacts that may or may not lend themselves to precise definition

or measurement. It is not an easy task.

Nonetheless, microenterprise development competes with other

development-employment policies for scarce public funds, and it is

reasonable that policymakers should want to know whether microenterprise

development is a good investment relative to other policy options. Fortunately,

methodologies to assess programme impact do exist. Drawing on principles

and experience in the natural and social sciences, impact assessment (IA)

methodologies are well-developed and are well-known to researchers. The

same methodologies, however, are much less well-known to policymakers,

with important implications for policy analysis.



390



EVALUATING LOCAL ECONOMIC AND EMPLOYMENT DEVELOPMENT – ISBN 92-64-01708-9 – © OECD 2004



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Chapter 16. A Commentary on the Workshop “Evaluating Territorial Employment Pacts”

Tải bản đầy đủ ngay(0 tr)

×