Tải bản đầy đủ
2 The Quality of Non- MTMM Questions in the Database

2 The Quality of Non- MTMM Questions in the Database

Tải bản đầy đủ

THE QUALITY OF NON-MTMM QUESTIONS IN THE DATABASE

253

Figure 13.9  Overview of the questions of round 1 from Ireland.

there are two possibilities: either the questions have already been coded or they have
not. Looking at Figure 13.9, we see both examples; question A3, for example, has
been coded, while question A4 has not been coded. A1 was already coded, but A2
was not coded. In reality, the latter is now coded because we used this question as an
example.
If we select question A1, we once again get the pop-up screen for this question as
before, presenting the quality prediction by SQP based on the approved coding. We
can also find the details of the quality estimates and the specification of the codes. So
far, it goes the same as before.1
If we select A2, however, the process is different because A2 has not been coded
so far. In selecting this question, we get the screen presented in Figure 13.10.
In order to get a prediction of the quality of this question, the first step is to code
the question. If you click on “Code question to create my own prediction,” the screen
appears that is presented in Figure 13.11.
Selecting “Begin coding” leads one to the screen presented in Figure 13.12.
On the lower left-hand side, the question and answer categories are presented. On
the top left side, the first characteristic that we should code is presented. This is the
domain of the question. The possible categories have been indicated. At the side,
some information about this characteristic is indicated. If one selects a category, the
choice is presented on the right side of the screen, and the next characteristics to be
coded appear at the left side. This characteristic is coded in the same way, and this
process goes on until all characteristics are coded.
1

Except that in this case, one coder did not finish the task and therefore no prediction was generated (−99).

254

The SQP 2.0 Program for Prediction of Quality and Improvement

Figure 13.10  Screenshot of question A2 of round 1 in Ireland.

Figure 13.11  The screen with the question and the option to begin coding.

For some codes, you will need to refer to the questionnaire, given that in the
SQP program, only the basic information is provided about the introduction, the
request, and the answer categories. Information whether instructions were given,
a show card was available, or a don’t know answer was possible or registered are
not provided in SQP.

THE QUALITY OF NON-MTMM QUESTIONS IN THE DATABASE

255

Figure 13.12  The first screen of the coding procedure.

Figure 13.13  The screen after the coding has been completed.

Sometimes, the program makes a suggestion for a possible answer. For example,
it suggests how many sentences and words there are in the questions. In that case,
you can accept the suggestion by clicking on “next,” or you can correct the number
and click on “next” to go to the next characteristic. When the coding is done for
all characteristics, the screen captured in Figure 13.13 appears.

256

The SQP 2.0 Program for Prediction of Quality and Improvement

Figure 13.14  The quality prediction for question A2 in Ireland.

At this stage, you can ask for a prediction of the quality of the question by clicking
on the text “Get Quality Prediction.” If predictions are requested, the screen captured
in Figure 13.14 will appear.
In this case, only the prediction of SQP is presented because no quality estimate
was obtained for this question in an MTMM experiment. If one would like the predictions of the quality coefficients that are the square root of the quality predictions,
one has to click on the button at the right saying “View quality coefficients.” In that
case, one also gets the prediction intervals.
It will be clear that by coding this question, the first screen presented in Figure 13.9
is no longer the same because question A2 has now also been coded. In order to
follow this process, the user can repeat the coding for the same question or, even
better, code another question that has not been coded so far.
13.3  Predicting the Quality of New Questions
For the third option of the program, we have to go back to the home page of SQP and
select the option “Create a new question.” If one chooses to introduce a new question,
a screen appears that asks for information about the particular study and question. In
this case, let us specify that we are doing a study called “immigration” in English.
The name in the questionnaire is A1 and the concept is “equality.” This information
is also presented in the screen in Figure 13.15.
The next step is to introduce the question itself. In this case, we have chosen to
introduce the question about the value of equal opportunities in the Schwartz

PREDICTING THE QUALITY OF NEW QUESTIONS

257

Figure 13.15  Input of the basic information about the question.

Human Value scale. This question has an introduction, a question with stimuli, and
six response categories. Figure 13.16 is the continuation of the page captured in
Figure 13.15.
After having saved the question, the screen presented in Figure 13.17 appears.
The next thing one has to do is code the question. This process begins with c­ licking
on “Begin coding.” The coding process was already described previously. If this
­process is finished, one should ask the program to perform the quality prediction. In
this case, we obtained the result presented in Figure 13.18.
It will be clear that this question is not very good. The quality is .55, and so 45%
of the variance in the observed scores is error. We can therefore ask the program for
suggestions for improvements. In this case, the results after evaluation of all characteristics are the suggestions presented in Figure 13.19.
This analysis shows that several improvements can be made. We see that
choosing another country would help. Of course, this is an impossible option.
Possible alternatives are presented by the characteristics “avgwrd_total,” “stimulus,” “visual,” etc. One should realize that this table gives the improvement for
one question characteristic keeping all the other characteristics the same as
they are. This means that by combining several of these characteristics, one may
be able to improve the question even more. The program gives suggestions for this,
but one would have to test the new version again. We cannot just add the different

258

The SQP 2.0 Program for Prediction of Quality and Improvement

Figure 13.16  The form in which to specify the question.

Figure 13.17  The question with the button “Begin coding.”

improvements together. We have seen in Figure  13.19 that SQP suggests several
improvements, especially with respect to the length of the text, the use of stimuli,
the data collection, the concept, etc. Let us start with the latter issue. The question
asked about the similarity of the respondent to the person described in the stimulus, while there was a mixture of two concepts presented in the stimulus: a value
statement and a norm. This is what Saris and Gallhofer (2007a) have called a

PREDICTING THE QUALITY OF NEW QUESTIONS

259

Figure 13.18  The prediction of SQP 2.0 of the quality of the question “equality.”

Figure 13.19  Several suggestions for improvement of the question “equality.”

complex concept because the item asks a similarity about other concepts. Besides
that, two different concepts have been combined in the stimulus. This could lead
to a great deal of confusion for the respondent. We have also seen that the use of
batteries of statements has a negative effect; therefore, following the suggestions

260

The SQP 2.0 Program for Prediction of Quality and Improvement

of Saris and Gallhofer (2007b), we would advise to measure the value with an
item-specific question like:
How important or unimportant is it for you that all people be treated equally?
1. Completely unimportant
2. Unimportant
3. Neither unimportant nor important
4. Important
5. Extremely important
This question is much shorter, it has a bipolar item-specific scale, and no statement
is used. Note that the change of some aspects of the request also changes a lot of
other aspects. Therefore, it makes sense to evaluate once again how good the quality
of this question is according to the SQP program. In order to check this, we introduce
this new question in the program, code the question, and ask for the quality prediction. The result is presented in Figure 13.20.
The Schwartz question had a quality of .55, while the new question has a
quality of .64. This would mean that the explained variance of the observed variable by the variable of interest—the value equality—has increased by nearly 10%.
One can also look at further possible improvements, but the explained variance
will never be ­perfect. This means that measurement errors will still remain.
Therefore, correction for measurement error is also important as we will see in the
next chapters.
Note that the improvement in quality was mainly obtained by the increase in the
validity. This means that by using this formulation, the systematic effect of the

Figure 13.20  The quality of the reformulated question.

SUMMARY

261

Figure 13.21  The quality coefficients, interquartile range, and standard error.

method, that is, the complement of the validity, has been reduced. This can also be
seen in the reduction in the CMV, which is now rather small.
A more detailed picture of the quality can be obtained by clicking on the text at
the right “View quality coefficients.” If we do so, we arrive at the screen presented in
Figure 13.21.
The quality coefficients are the square root of the quality indicators themselves.
These are the coefficients that are estimated in the MTMM experiments. In this
screenshot, we can see the uncertainty that exists in the estimates presented in the
interquartile range and the standard error. It will be clear that a considerable range of
uncertainty remains.
Nevertheless, the attractiveness of this approach lies in the fact that we can get these
estimates before the data have even been collected. While the MTMM experiments are
time consuming and expensive, the quality estimates using SQP 2.0 are obtained with
minimal efforts and allow researchers to improve their data collection before they
spend a lot of money on it. It is not possible to take into account 60 question characteristics while formulating a question. SQP makes it possible to evaluate the questions
and suggests improvements. This is the major advantage of this procedure.
13.4  Summary
In this chapter, we have shown that the SQP 2.0 program can be used to obtain (1)
the quality estimates that were obtained by MTMM experiments, (2) the quality
predictions of questions that are in our database but not part of an MTMM

262

The SQP 2.0 Program for Prediction of Quality and Improvement

experiment, and (3) the quality predictions of new questions that a researcher
would like to evaluate. We have also shown that the program provides suggestions
for improving questions in a simple way.
The most important advantage of the new SQP 2.0 program above version 1.0 is
that it provides predictions of the quality of questions in more than 22 countries
based on a database of more than 3000 extra questions that have been evaluated in
MTMM experiments to determine the quality of the questions.
Another very important advantage of the new program compared to the earlier
version is that the earlier program had to be downloaded and used on one’s own PC.
The new version is an Internet program with a connected database of survey questions, which now contains all questions used in both the old and new experiments, as
well as all questions asked so far in the ESS. This means that there are already more
than 60,000 questions in all languages used in the ESS that appear in the SQP database. The number of questions will grow in three ways: first, by way of the new
studies carried out by the ESS, which, in each round, adds another 280 questions
across all languages used; second, through the new studies that are added to the database by other large-scale cross-national surveys; and, third, thanks to the introduction
of new questions on the part of researchers, using the program to evaluate the quality
of their questions.
Therefore, the SQP program is a continuously growing database of survey questions in most European languages with information about the quality of the questions
and the possibility to evaluate the quality of the questions that have not been evaluated so far. In this way, the program will be a permanently growing source of
information about survey questions and their quality. To date, there is no other
program in the world that offers such possibilities.
Exercises
1. As we have said, the relationships between variables cannot be compared across
countries if the quality of the questions is not comparable. Check with SQP
whether relations of questions measuring political trust can be compared with
each other. For which countries can comparisons be made, and for which
­countries can they not be compared?
2. Social trust is measured with three indicators:
a.  Check which indicator in your language is the best for social trust.
b.  Are the MTMM estimates and the predictions of SQP comparable?
c.  Is the best indicator good enough, or does this question need improvement?
d.  How could you improve this question?
e.  Specify the new question.
f.  Test the new question by introducing the question in SQP in the study called
“Test” and evaluate if the new question is better than the ESS question.

14
The Quality of Measures for
Concepts-by-Postulation

In this chapter, we pick up the discussion of the first chapter about concepts-­
by-postulation (CP) and concepts-by-intuition. This is important because often,
the concepts people want to study are not so simple that they can be operationalized by concepts-by-intuition. Several concepts-by-intuition are also combined
into one CP in order to obtain a measure of the concept of interest with better
reliability and/or validity. To date, we have become familiar with the quality of
measures for concepts-by-intuition. In this chapter, we want to show how this
information can be used to say something about the quality of measures for CP.
This is possible because a measure of a CP is an aggregate of several measures of
concepts-by-intuition.
First, we will introduce the possible structures of CP. The logic is that the
measures of CP are based on concepts-by-intuition. In order to determine the
score of the CP, the relationships between the concepts-by-intuition and the CP need
to be known. In some cases, one can control whether the expected ­relationships
indeed exist. Depending on the hypothesized relationships, different tests for the
structure of the measures are performed. In this chapter, we will discuss these
different structures and indicate how the quality of the measures for the CP can
be determined on the basis of the estimated quality of the measures for the
concepts-by-intuition.

Design, Evaluation, and Analysis of Questionnaires for Survey Research, Second Edition.
Willem E. Saris and Irmtraud N. Gallhofer.
© 2014 John Wiley & Sons, Inc. Published 2014 by John Wiley & Sons, Inc.


263