Tải bản đầy đủ
Appendix 14.2 Lisrel Input for Final Analysis of the Effect of “Interest in Political Issues in the Media” on “Political Interest in General”

Appendix 14.2 Lisrel Input for Final Analysis of the Effect of “Interest in Political Issues in the Media” on “Political Interest in General”

Tải bản đầy đủ

286

The Quality of Measures for Concepts-by-Postulation

select
1 2 3 4 5/
model ny = 2 nx = 3 ne = 2 nk = 3 ly = fu,fi te = di,fr lx = fu,fi td = sy,fi ga = fu,fi be = fu,fi
ps = sy,fi ph = sy,fi
value 1 ly 1 2
free ly 2 2
free be 2 1 ps 2 2
value .52 lx 1 1
value .73 lx 2 2
value .48 lx 3 3
value .64 td 1 1
value .38 td 2 2
value .69 td 3 3
value .09 td 2 1 td 3 1 td 3 2
value 1 ga 1 1
free ga 1 3 ga 1 2
value 1 ph 1 1 ph 2 2 ph 3 3
free ph 2 1
free ph 3 1 ph 3 2
start .5 all
out mi sc adm = of ns

15
Correction for Measurement
Errors

Measurement errors will remain, no matter how much we do our best to improve
the questions. That means that the estimates of the relationships between the
­variables will be affected by these errors. Therefore, it is necessary to correct for
these errors. Thus, the most important application of our research on the quality of
questions and composite scores is the use of the quality estimates for correction
for  measurement errors in the analysis between variables. In the past, complex
­procedures have been discussed for the correction of measurement errors using
multiple indicators. In this chapter, we will present a very simple procedure for the
correction of measurement errors that is not much more difficult than regression
analysis or causal modeling without latent variables.
15.1  Correction for Measurement Errors in Models
with Only Concepts-by-Intuition
In this section, we want to show by a simple example how this can be done. The
example we want to use is a model to explain opinions about immigration. Variables
to explain this opinion have been collected in the third round of the ESS. Some of
the questions have already been discussed. For this example, we suggest the model
­presented in Figure 15.1.

Design, Evaluation, and Analysis of Questionnaires for Survey Research, Second Edition.
Willem E. Saris and Irmtraud N. Gallhofer.
© 2014 John Wiley & Sons, Inc. Published 2014 by John Wiley & Sons, Inc.


287

288

Correction for Measurement Errors

Economic threat

Allowing more people
from outside Europe

Better life
Culture threat

Figure 15.1  A simple model for explaining opinions about immigration.

The concepts presented in this model can be seen as concepts-by-intuition.
These concepts can therefore be measured by direct questions. The English version
of the question used for the dependent variable, which we will call “allow,” was used
in round 3 of the ESS and was measured by the second question mentioned in the
following text:
B35 CARD 14  Now, using this card, to what extent do you think [country] should
allow people of the same race or ethnic group as most [country’s] people to come and
live here?
Allow many to come and live here
1
Allow some
2
Allow a few
3
Allow none
4
(Don’t know)
8
B37 STILL CARD 14  How about people from the poorer countries outside Europe?
Use the same card.
Allow many to come and live here
1
Allow some
2
Allow a few
3
Allow none
4
(Don’t know)
8
The questions used for the other variables are mentioned in the following text with
the name we have given to the different variables.
“Economic threat”
B38 CARD 15  Would you say it is generally bad or good for [country]’s economy
that people come to live here from other countries? Please use this card.
Bad for the
Good for the
(Don’t
economy
economy
know)
   00 01 02 03 04 05 06 07 08 09 10   88

MEASUREMENT ERRORs IN MODELS with only concepts-by-intution

289

“Cultural threat”
B39 CARD 16 And, using this card, would you say that [country]’s cultural life is
­generally undermined or enriched by people coming to live here from other countries?
Cultural life
Cultural life
(Don’t
undermined
enriched
know)
    00 01 02 03 04 05 06 07 08 09 10    88
“Better life”
B40 CARD 17 Is [country] made a worse or a better place to live by people coming
to live here from other countries? Please use this card.
Worse place
Better place
(Don’t
to live
to live
know)
    00 01 02 03 04 05 06 07 08 09 10    88
Note that the explanatory variables are measured using the same scale. That
may cause common method variance (cmv). The dependent variable is measured in
a rather different way, and therefore, we do not expect cmv between this variable and
the other three variables.
The correlations between these variables obtained in Ireland in round 3 of the
ESS were as indicated in Table 15.1.
The quality of the questions has been determined in two different ways. First
of all, these questions were involved in an MTMM experiment. The quality of the
­questions was therefore determined by an experiment. Secondly, these questions
were also coded in order to be included in the meta-analysis over all evaluated
­questions. In that case, the SQP 2.0 program can be used to predict the quality of
these questions. Here, we will illustrate the procedure using the SQP predictions
because this is the more general procedure. Using the MTMM estimates, the
procedure is of course the same.
In Table 15.2, we present the quality predictions obtained using the authorized
codes of the questions presented previously.
On the basis of the data in Table 15.1 and the quality information in Table 15.2,
the effects presented in Figure 15.1 can be estimated with and without correction
for  measurement errors. Normally, the analysis is done without correction for
measurement errors. In that case, the estimation is done on the basis of the correlation
matrix of Table 15.1 without any adjustment.

Table 15.1  The correlations between these variables
obtained in Ireland (n = 1700)
Allow
Better life
Economic threat
Culture threat

 1.00
−.470
−.423
−.447

1.00
.662
.718

1.00
.704

1.00

290

Correction for Measurement Errors

Table 15.2  The quality of the questions predicted by SQP
Variable

Method

r2

v2

m2

q2

Allow
Economic threat
Culture threat
Better life

SQP2.0
SQP2.0
SQP2.0
SQP2.0

.826
.770
.761
.748

.906
.780
.705
.725

.094
.220
.295
.275

.747
.601
.537
.543

Table 15.3  The correlations adjusted for cmv below the diagonal with the quality
estimates on the diagonal and the cmv’s above the diagonal
Allow
Better life
Economic threat
Culture threat

.747
−.470
−.423
−.447

.00
.543
.476
.503

.00
.186
.601
.509

.00
.215
.195
.537

If one wants to correct for measurement error, one has to correct the correlations
in the matrix in the following way:



 observed rij − cmv 
Corrected rij = 
 (15.1)

q i .q j





where the cmv = ri m i m j rj (15.2)

This result follows directly from Equation (9.1). The observed correlations are obtained
from the collected data (Table 15.1), and the cmv and the quality estimates can in g­ eneral
be obtained from SQP 2.0 (here Table 15.2). In general, then, the corrected correlations
can be obtained in this way. The simplest way is to calculate the cmv first and subtract
these values from the observed correlations. After that, one can substitute the 1 s on the
diagonal of the correlation matrix by the quality of each question. If one then asks a
SEM program to analyze the correlation matrix, the program first transforms the
provided matrix into a correlation matrix that automatically contains the corrected correlations for all variables. This is so because this transformation is done by dividing all
cells by the product of the quality coefficients as suggested in Equation 15.1.
First, the correlation matrix is adjusted by subtracting the cmv for each cell. In
this case, we do not expect cmv for the “allow” variable and the other variables
because their measurement procedure is different. For the other cells, we have calculated the cmv using Equation 15.2.
The adjusted matrix for this example is presented in Table 15.3. The cmv’s for the
different cells are presented above the diagonal. We see that quite a large part of the
correlations is due to cmv. As a consequence, the cells below the diagonal indicating
the correlations corrected for the cmv are much smaller than the observed correlations presented in Table 15.1.
On the diagonal, the qualities of the different variables have been presented as
­predicted by SQP. Note that the cmv’s are calculated on the basis of the reliability
­coefficients and the method effects while on the diagonal the qualities are presented.

MEASUREMENT ERRORs IN MODELS with only concepts-by-intution

291

Table 15.4  The correlations corrected for measurement error
Allow
Better life
Economic threat
Culture threat

 1.00
−.738
−.631
−.706

1.00
.833
.931

1.00
.896

1.00

Table 15.5  The estimates of the effects of the explanatory variables on the variables
allow and better life

Effects on allow from
Better life
Economic threat
Cultural threat
Total explained (R2)
Effects on better life from
Economic threat
Cultural threat
Total explained (R2)

Without correction

With correction for errors

−.265a
−.133a
−.154a

−.609a
.001
−.140a

.254

.547

−.310a
.500a

−.007
.938a

.564

.868

Means significantly different from 0.

a 

The lower triangular matrix is not a correlation matrix anymore; however, by
transforming this covariance matrix into a correlation matrix, one will get the correlations between these variables corrected for measurement error. The result is
­presented in Table 15.4.
In the first column, the correlations have only been corrected for random measurement
errors. The other cells were also corrected for systematic errors. Comparing this table
with Table 15.1, we see that all correlations have been increased by this correction for
measurement error even though the systematic errors (cmv) were quite large. Given the
changes in the correlations, we should also expect that the estimates of the effects will
be different. The effects have been estimated with the ML estimator of LISREL. The
inputs for these analyses are presented in Appendix 15.1. The results without and with
correction for measurement error are presented in Table 15.5.
The results change considerably by correction for measurement errors. A
systematic effect is that the effect of the variable economic threat is reduced
and the effect of the better life and cultural threat have considerably increased. In
particular, the conclusions with respect to the effects on the variable “allow” change
considerably. The effect of the variable “better life” gets much larger. While without
correction for measurement error, the effects of the economic and cultural threat
are  approximately equal. After the correction, the effect of the cultural threat is
approximately the same, while the effect of the economic threat is only minimally
different from 0. A similar effect can be seen for the other equation.
Besides that, we see that in both equations, the unexplained variance is reduced
considerably. After correction for measurement errors, it is clear that there is still
reason for looking for other explanatory variables because the lack of explanatory

292

Correction for Measurement Errors

power cannot come from measurement errors. This conclusion would not have been
possible if one had not corrected for measurement errors.
We give this example in order to show that taking into account the quality of the
questions (i.e., correction for measurement errors) can have a considerable effect
on the results of the analysis of the relationships between variables. Therefore, we
are of the opinion that the information about the quality of questions is essential for
the analysis of survey data and even more so in comparative research.
We hope that we have demonstrated that the procedure for taking into account
the quality of questions is very simple. There is therefore no reason not to correct
for measurement errors.
15.2  Correction for Measurement Errors in Models
with Concepts-by-Postulation
In the estimation of models with concepts-by-postulation, one normally computes
composite scores for these concepts and evaluates the quality of the composite
scores using Cronbach’s α, and after that, one continues with the analysis as if there
were no measurement errors, while a Cronbach’s α below 1 indicates that there are
measurement errors. In doing so, one will get biased estimates of the effects one is
interested in. This is not necessary. Therefore, we will illustrate here how one can
get unbiased estimates of effects of models containing concepts-by-postulation. We
will use for this purpose a topic that has been very popular recently.
During the last 15 years, a lot of attention has been given to the theory of “social
capital” (Coleman 1988; Putnam 2000; Newton 1999, 2001; Halpern 2005). This
theory suggests that investment in “social contact” functions for ­people as an asset
that results in trust in other people and in the political system. We take these
hypotheses as the starting point for our model and add more variables to it because
we think that not only “social contact” influences “social trust” and “political trust.”
We enrich the model by adding the variables “experience of discrimination” and
“political interest” for explanation of “social trust” and “political trust.” To explain
“political trust,” the variables “political efficacy” and “political interest” are added.
Figure 15.2 incorporates these variables into a simple substantive model.
In this model, it is assumed that the variables “social contact,” “experience of
discrimination,” and “political interest” cause a spurious correlation between
“social trust” and “political trust” and that these two latter variables, possibly, also
have a reciprocal causal relationship. The reciprocal effect is included because it is
plausible and to date has not been falsified.
15.2.1  Operationalization of the Concepts
In Table 15.6, we give an overview of the operationalization of the different concepts
defining the chosen approach of the ESS in the first round. Most concepts are
­concepts-by-postulation with several reflective indicators.
“Social contact” is a concept-by-postulation with two formative indicators as has
been discussed in Chapter 15. “Social trust,” “political trust,” and “political efficacy”

MEASUREMENT ERRORs IN MODELS with concepts-by-postulation

293

Experience of
discrimination
Social trust

ζ1

Political trust

ζ2

Social contact

Political interest

Political efficacy
Figure 15.2  A structural model of a simple theory about effects of “social contact” and
other variables on “social trust” and “political trust.”
Table 15.6  The operationalization of the concepts in Figure 15.2
Concept name

Concept type

Observed indicator

Characterization of Indicators

Social contact

Postulation

Social trust

Postulation

Political trust

Postulation

Political efficacy

Postulation

Discrimination
Political interest

Intuition
Intuition

Informal contact
Formal contact
Can be trusted
Fair
Helpful
Parliament
Legal system
Police
Complex
Active role
Understand
Discriminated
Interested

Formative
Formative
Reflective
Reflective
Reflective
Reflective
Reflective
Reflective
Reflective
Reflective
Reflective
Direct question
Direct question

are concepts-by-postulation with reflective indicators. “Experience of discrimination”
is a concept-by-intuition measured by a direct question. “Political interest” could
have been measured in different ways, but we opted for a direct question as a measure
for the concept-by-intuition.
Part III of this book demonstrated how to estimate the size of the errors or the
quality of a single question by using MTMM experiments; at least three forms of
the same request for an answer are needed. In Chapter 13, we showed that an estimate
of the size of the errors can also be obtained through the SQP program. It reduces the
number of concepts to be measured to one for each indicator, which is more efficient
than the MTMM approach.
Furthermore, in Chapter 14, we have already seen that the quality of a measure
for a concept-by-postulation can be derived if the qualities of the measures for

294

Correction for Measurement Errors

Table 15.7  Possible designs of a study with respect to the number of observed
variables included in the model
Number of observed variables
Composite scores

Indicators for
each concept

Social contact
Social trust
Political trust
Political efficacy
Discrimination
Political interest

1
1
1
1
1
1

2
3
3
3
1
1

6
9
9
9
3
3

2
9
9
9
1
1

No. of observed variables

6

13

39

31

Concept name

MTMM

In the ESS

the  concepts-by-intuition are known. Therefore, the number of observed variables
can be reduced to 1 for each variable in the model. Our overview of the different
­possibilities to evaluate the quality of the measures in a study leads to designs that
differ with respect to the number of observed variables and complexity of the model.
Table 15.7 summarizes the possibilities.
This table shows that only 6 observed variables are needed if composite scores for
all concepts mentioned in the model are calculated while 13 variables are used if one
form of each of the indicators for these concepts is employed. The option to evaluate
the data quality through MTMM analysis for this substantive research corresponds
to the need for 39 observed variables. Finally, Table 15.7 informs us that there are 31
observed variables in the ESS out of the 39 mentioned.
Our advice is to avoid making models with 31 or 39 variables, because it increases
the risk of serious errors in the design and analysis. It calls for a complex model of a
combination of MTMM models for each concept and the corresponding substantive
model of Figure 15.2. Therefore, in the following discussion, we will concentrate on
the use of composite scores (six observed variables).
The following two steps are needed to reduce the design of the analysis while correcting for measurement error:
1. An evaluation of the quality of the measurement instruments
2. An analysis of the substantive model correcting for the detected errors
In the next section, we will give an overview of the data quality of the possible
observed variables.
15.2.2  The Quality of the Measures
It is beyond the scope of this chapter to describe in detail how all the questions were
evaluated. Some of the results of the studies of the quality of the measurement instruments have been presented previously. The results of these evaluations have been summarized in Table 15.8. The indicators for “social trust,” “political trust,” and “political

295

MEASUREMENT ERRORs IN MODELS with concepts-by-postulation

Table 15.8  Quality estimates of the 13 indicators from the Dutch study in the ESS
Coefficient for
Method
used

Concept name

Indicator

Reliability

Validity

Quality

Consistency

Social contact

Informal
Formal

.79
.68

1.0
1.0

.79
.68




SQP
SQP

Social trust

Be trusted
Fair
Helpful

.87
.83
.84

1.0
1.0
1.0

.87
.83
.84

.84
.94
.66

MTMM
MTMM
MTMM

Political trust

Parliament
Legal system
Police

.85
.90
.94

.95
.96
.96

.81
.86
.90

.66
.99
.66

MTMM
MTMM
MTMM

Political efficacy

Complex
Active role
Understand

.88
.94
.86

.96
.97
.97

.85
.91
.83

.89
−.57
−.78

MTMM
MTMM
MTMM

Discrimination

Direct request

.72

.72

.52



SQP

Political interest

Direct request

.96

.80

.77



SQP

efficacy” were evaluated by MTMM experiments,1 while the other ­indicators have been
evaluated by SQP. In the table, we see that the quality of the indicators2 evaluated by an
MTMM experiment is much better than the quality of the indicators evaluated by the
SQP program. Given that the SQP program is based on the MTMM experiments, there
is no reason to think that this difference is due to the evaluation method used. The real
reason is that MTMM experiments were done in the pilot study, and the best method
was selected for the main questionnaire in the definitive research. The results from the
study confirm that this procedure is s­ uccessful. The questions evaluated with the SQP
program were not developed in the same way. They were not involved in an MTMM
study in the pilot study and therefore were not improved upon.
This table also shows that for “social trust” and “social contact,” the method
effects are 0 so that the validity coefficient, which is the complement, is equal to 1.
For the concepts “political trust” and “political efficacy,” this is not true; there, the
validity coefficients are not 1.
The low value of the quality of the “experience of discrimination” variable is of
­concern. The quality of this indicator is low because the explained variance in the
1

In this chapter, the Dutch data of the first official round of the ESS are analyzed and not the data from the
pilot study. As a consequence, the coefficients are slightly different from those presented in Chapter 14.
As far as predictions with SQP are made, they are made with SQP 1.0. The predictions are not sufficiently
different to repeat the analysis with the new numbers because the procedures are exactly the same and the
result will be only minimally different.
2
The reader is reminded that the quality coefficient is the product of the reliability and the validity coefficient and the quality itself is the quality coefficient squared, which can be interpreted as the percentage
explained variance in the observed variable by the concept-by-intuition.

296

Correction for Measurement Errors

observed score is only 27%. This is due partially to the lack of precision of the scale
used, which is a yes/no response scale. Here, a scale with gradation would result in a
better quality measure. However, in the context of our illustration, this lack of quality
will serve to show just how large the effect of correcting for measurement error can be.
Table  15.8 also shows the size of the consistency coefficients of the different
reflective indicators for the concept-by-postulation that they are supposed to measure. We have included these coefficients because they play a role in calculating the
measures of the composite scores (Chapter 14). Such relationships do not exist for
concepts with formative indicators or concepts-by-intuition.
Finally, we have to mention that we did not specify the method effects because
they are the complement of validity (1—validity coefficient squared). These effects
are important because the method factors cause correlations between the observed
variables, which have nothing to do with the substantial correlations. In this study,
such method effects can be found within sets of variables for the same concept, but
not across the different concepts of the model, since the methods are too different
for the different substantive variables.
Now that we have discussed the quality of the indicators, we can turn to the quality
of the composite scores for the different concepts-by-postulation that have been
included in Figure 15.2. Chapter 14 discussed the procedures to estimate the quality
of the composite scores for the “social contact” and “political efficacy” concepts.
The measures for “social trust” and “political trust” are calculated using regression
weights, followed by evaluation of the quality of these composite scores, using
Equation (14.3). The results for these four concepts-by-postulation have been
­summarized in Table 15.9.
Table 15.9  The quality of the measures for the concepts-by-postulation
Construct
Variable
name

Composite score

Indicator

Validity
coefficient

Regression
weights

Informal
Formal

.79
.68

.14
.92

Can be trusted
Fair
Helpful

.73
.81
.55

.35
.50
.10

Parliament
Legal system
Police

.53
.86
.59

.09
.74
.13

.76
−.52
−.66

.53
−.20
−.34

Social contact

Social trust

Political trust

Political efficacy
Complex
Active role
Understand

Quality
coefficient

Method
effect

.74

.00

.81

.00

.87

.31

.86

.22