Tải bản đầy đủ
Instrument 8. A: Writing Sensitive Questions

Instrument 8. A: Writing Sensitive Questions

Tải bản đầy đủ

Guidelines for Writing Selection Items

201

the survey, so as not to incriminate themselves. Anonymity is an approach often
used to obtain a high response rate when a questionnaire attempts to measure
behaviors in sensitive areas. However, it is also important to consider if and how
the demographic information might compromise that level of confidentiality. For
example, with some effort, the information provided in the first ten demographic
items on the YRBS could be used to identify individuals fitting a specific profile,
particularly where the sample size is small.
When developing instruments to produce information that will be used by
individuals other than yourself, it is important to be clear about who will “own”
these data (we will examine this issue in more detail in Chapter Fifteen). This
should be documented in writing, as part of a legal contract, to ensure that the
data and data sources (that is, the completed questionnaires) are not made available to entities that could misuse them. In the case of the YRBS, parents were
ensured that the “survey procedures have been designed to protect your child’s
privacy and to allow for anonymous participation.” However, the cover letter did
not describe how the data would be handled or released. Ultimately, your integrity
as researcher and instrument designer will be compromised if information you
are responsible for is misused.
The ten YRBS questions on drug use attempt to measure behaviors and not
attitudes; these are selection items that ask about frequency of use and that make
use of alternative response sets (Chapter Nine) and not rating scales. Each item is
written clearly and concisely. Items specifically address usage by drug type, including cocaine, heroin, steroids, and methamphetamines. The instrument designers
cannot be faulted for their directness, given the sensitive nature of the topic.
The Youth Risk Behavior Survey also conveys a message about the importance of defining the purpose of your study. On the one hand, many items are
related to a healthy life style. For example, the nutrition questions seek information about the frequency of eating vegetables, fruits, and dairy products. On the
other hand, a number of items specifically address at-risk and illegal behaviors.
The duality of the instrument tends to obscure its purpose and may raise questions about who will use the results and how that information will be applied.

Instrument 8.B: Biased Language
George Lakoff is a linguist who is interested in how politicians frame issues and
use language. For example, the term tax relief connotes that taxation is onerous
and thus something we need relief from. The term can be used to support one side of
an argument, such as support for tax cuts, without presenting the other side, such as
paying for national defense, education, and public health (Lakoff, 2004).

c08.indd 201

7/9/07 11:50:25 PM

202

Designing and Constructing Instruments for Social Research and Evaluation

The following questionnaire was developed for a member of the U.S. House of
Representatives (Goodlatte, 1998) and suggests the sort of framing that seems
likely to bias respondents’ answers. (The response rate as a percentage is provided
for each option.) For example, the question about affirmative action (item 5) produced the following results:
Should the federal government grant hiring, contracting, or educational preferences
or quotas on the basis of race or gender?
(a) Yes (5%)

(b) No (95%)

This item illustrates the use of biased language. The meaning of the term
quotas is quite different from the meaning of preferences, as quotas infers assigned
numbers. For example, organizations may implement policies that give preferences to various groups, such as veterans, minorities, the handicapped, or people
with special, work-related skills. This is quite different from a government regulation that requires organizations to hire a predetermined number of individuals.
Because the terms have different meanings, it is difficult to know which of the
terms each respondent used to rate the item.
As we will examine in Chapter Thirteen, the method for collecting data can
also create a form of bias. Because bias has been introduced here through the
framing of items and by the process of administration, the results of this survey
cannot be considered reliable or representative of voters in the congressional
district where it was used.
INSTRUMENT 8.A: YOUTH RISK BEHAVIOR SURVEY
(SAMPLE ITEMS).
This survey is about health behavior. It has been developed so you can tell us what
you do that may affect your health. The information you give will be used to develop
better health education for young people like yourself.
DO NOT write your name on this survey. The answers you give will be private. No one
will know what you write. Answer the questions based on what you really do.
Completing the survey is voluntary. Whether or not you answer the questions will not
affect your grade in this class. If you are not comfortable answering a question, just
leave it blank.
The questions that ask about your background will be used only to describe the types
of students completing this survey. The information will not be used to find out your
name. No names will ever be reported.
Make sure to read every question. Fill in the ovals completely. When you are finished,
follow the instructions of the person giving you the survey.
Thank you very much for your help.

c08.indd 202

7/9/07 11:50:25 PM

Guidelines for Writing Selection Items

203

DEMOGRAPHICS:
1. How old are you?
1. 12 years old or younger
2. 13 years old
3. 14 years old
4. 15 years old
5. 16 years old
6. 17 years old
7. 18 years old

6. In the home where you spend the
most time, who are the adults who
are the most responsible for you?
1. Mother and Father
2. Mother
3. Father
4. Grandparents
5. Foster Parents
6. Step-Parents

2. What is your sex?
1. Female
2. Male
3. In what grade are you?

7. Other Adult Relatives
8. Other
7. How tall are you without your shoes
on?

1. 9th grade

1. Under 4’8”

2. 10th grade

2. 4’8” to 4’11”

3. 11th grade

3. 5’ to 5’2”

4. 12th grade

4. 5’3” to 5’5”

5. Ungraded or other grade

5. 5’6” to 5’8”

4. How do you describe yourself?
(Select one or more responses.)
1. American Indian or Alaskan Native
2. Asian

6. 5’9” to 6’
7. Over 6’1”
8. How much do you weigh without
your shoes on?

3. Black or African American

1. Less than 65 pounds

4. Hispanic or Latino

2. 66–90 pounds

5. Native Hawaiian or Other Pacific
Islander

3. 91–115 pounds

6. White

5. 141–165 pounds

5. Do you receive free or reduced
lunch at school?
1. Yes

4. 116–140 pounds
6. 166–180 pounds
7. 181–205 pounds
8. Greater than 206 pounds

2. No

c08.indd 203

7/9/07 11:50:25 PM

204

Designing and Constructing Instruments for Social Research and Evaluation

9. Who/what most influences your
decisions and behavior?

3. 3 to 9 times
4. 10 to 19 times

1. Family

5. 20 to 39 times

2. Friends and peers

6. 40 or more times

3. School teachers/counselors/
nurses
4. Religious leaders
5. TV/newspaper/magazine
6. Other
7. Not sure
10. How often do you attend religious
services?

56. During your life, how many times
have you sniffed glue, breathed the
contents of aerosol spray cans, or
inhaled any paints or sprays to get
high?
1. 0 times
2. 1 or 2 times
3. 3 to 9 times

1. Never

4. 10 to 19 times

2. Rarely

5. 20 to 39 times

3. Once or twice a month

6. 40 or more times

4. About once a week or more
The Next 10 Questions Ask about
Cocaine and other Drugs:
54. During your life, how many times
have you used any form of cocaine,
including powder, crack, or freebase?
1. 0 times
2. 1 or 2 times
3. 3 to 9 times
4. 10 to 19 times
5. 20 to 39 times
6. 40 or more times
55. During the past 30 days, how many
times did you use any form of
cocaine, including powder, crack
or freebase?
1. 0 times
2. 1 or 2 times

57. During the past 30 days, how
many times have you sniffed glue,
breathed the contents of aerosol
spray cans, or inhaled any paints or
sprays to get high?
1. 0 times
2. 1 or 2 times
3. 3 to 9 times
4. 10 to 19 times
5. 20 to 39 times
6. 40 or more times
58. During your life, how many times
have you used heroin (also called
smack, junk, or China White)?
1. 0 times
2. 1 or 2 times
3. 3 to 9 times
4. 10 to 19 times
5. 20 to 39 times
6. 40 or more times

c08.indd 204

7/9/07 11:50:25 PM

Guidelines for Writing Selection Items

205

59. During your life, how many times
have you used methamphetamines
(also called speed, crystal, crank, or
ice)?

61. During your life, how many times
have you used a needle to inject any
illegal drug into your body?
1. 0 times

1. 0 times

2. 1 time

2. 1 or 2 times

3. 2 or more times

3. 3 to 9 times

62. During the past 12 months, has
anyone offered, sold or given you an
illegal drug on school property?

4. 10 to 19 times
5. 20 to 39 times
6. 40 or more times

1. Yes

60. During your life, how many times
have you taken steroid pills or
shots without a doctor’s prescription?
1. 0 times

2. No
63. How old were you when you tried
any form of illegal drug (excluding
tobacco and alcohol) for the first
time?

2. 1 or 2 times

1. I have never tried an illegal drug

3. 3 to 9 times

2. 8 years old or younger

4. 10 to 19 times

3. 9 or 10 years old

5. 20 to 39 times

4. 11 or 12 years old

6. 40 or more times

5. 13 or 14 years old
6. 15 or 16 years old
7. 17 years old or older

INSTRUMENT 8.B: RESULTS OF THE 1998 CONGRESSIONAL
QUESTIONNAIRE.
Earlier this year, I included a survey with my Congressional Report to learn your views
on several important issues facing our country. I am very pleased with the response,
as nearly 20,000 folks took the time to complete the survey and return it to my office.
Below are the survey results. Next to each answer, in parentheses, is the percentage of
folks who indicated that response. For example, for Question 1, 44 percent of those
responding believe that moral decline is the number one problem facing the United
States. Thank you to all those who participated.
1. What do you consider to be the number one problem facing the United States?
(a) Education (12%)

(b) Economy (5%)

(c) Crime (11%)

(d) IRS/Tax System (17%)

(e) Moral Decline (44%)

(e) Other (12%)
(Continued)

c08.indd 205

7/9/07 11:50:25 PM

206

Designing and Constructing Instruments for Social Research and Evaluation

2. Regarding federal spending, since 1995, the Congress has cut:
(a) Too much (6%)

(b) Not enough (78%)

(c) The right amount (16%)

3. Regarding changing federal income taxes, which of the following is closest to your views?
(a) Replace the current federal income tax with a flat tax (40%)
(b) Replace the current federal income tax with a national sales tax (20%)
(c) Modify the current federal income tax (35%)
(d) Leave the federal income tax as is (5%)
4. Should the IRS be:
(a) Abolished (27%)

(b) Dramatically changed (66%)

(c) Left as is (6%)

5. Should the government grant hiring, contracting, or educational preferences or quotas
on the basis of race or gender?
(a) Yes (5%)

(b) No (95%)

6. Should the federal government have unrestricted access to information and communications transmitted by individuals and businesses over the Internet?
(a) Yes (15%)

(b) No (85%)

7. Should Congress continue investigating the current presidential campaign and administration’s fundraising activities?
(a) Yes (59%)

(b) No (41%)

8. U.S. troops have been in Bosnia for two years. What do you think should be done?
(a) Pull the troops out now and let European powers work it out (55%)
(b) Continue U.S. involvement indefinitely (15%)
(c) Allow the troops to stay up to a year longer (30%)
9. Which statement more closely reflects your views on education?
(a) The federal government should be given more control over our children’s
education (5%)
(b) Educational decisions should be kept at the state and local level (95%)
10. Do you consider your personal beliefs to be:
(a) Conservative (50%)

(b) Moderate (47%)

(c) Liberal (3%)

Endnotes
1. Respondents may also provide socially undesirable responses: for example, a substance-abusing
client who does not want to participate in treatment might exaggerate behaviors that are
not tolerated by a rehabilitation program.

c08.indd 206

7/9/07 11:50:26 PM

Guidelines for Writing Selection Items

207

2. The higher the number the greater the intensity, whether the intensity is positive (from
worst to best) or negative (from pain free to intense pain). For example, on the Symptoms
Distress Scale the value of 5 is given to the most stressful health problems. In the following
example, also notice that the more intense symptomatology is placed on the left-hand side
of the response scale (Hinds, Schum, & Srivastava, 2004).
Please put a circle around the number that most closely measures how well you sleep last
night.
Couldn’t have been worse

5

4

3

2

1

A perfect night

Key Concepts and Terms

c08.indd 207

appropriate language

primacy effect

sensitive question

biased item

readability

socially desirable response

bipolar scale

recency effect

unidimensionality

middle option

response set

unipolar scale

neutral option

selection item

7/9/07 11:50:26 PM

Y
CHAPTER NINE

SELECTION ITEMS
Alternative Formats

In this chapter we will
• Present guidelines for creating selection items using alternative response sets.
• Describe the process of rank ordering and guidelines for creating items that
ask respondents to rank their responses.
In this chapter we continue to present information about several approaches
to constructing items where you, as instrument developer, provide the
response sets. These approaches are scales that use alternative response sets
and response alternatives that are ranked rather than rated. An example of
rating using an alternative response scale is asking respondents to select the year’s
best picture out of a list of five motion pictures. Alternatively, you could use
the process of ranking to determine the best picture; instead of asking respondents to select just one response, you would ask respondents to rank their preferences from one to five and determine the “winner” as the movie given the highest
rank by the majority of respondents. The advantage of this approach is that
it also provides information about the second most-liked movie, the third,
and so on.
As in the previous chapter, we will define and describe each type of item
format, provide guidelines for constructing each item, and explain how to analyze
and report findings produced by these alternative item formats. Making use of
208

c09.indd 208

7/9/07 11:50:55 PM

Selection Items: Alternative Formats

209

these additional formats can support effective decision making by helping you
obtain the exact information you seek.

Alternative Response Scales
In this section we use the term alternative response scales, or alternative response sets, to
describe scales that offer alternatives to the more familiar scales of agreement,
frequency, or intensity. For example, demographic items, such as questions about
age, gender, educational level, cultural background, and socioeconomic status,
often make use of alternative response scales.
Alternative response scales may consist of items that are clearly related, thus
forming a continuum, or they may consist of a number of diverse factors, related
to the stem of the question but not to each other (see the following examples).
Alternative response scales may also take the form of dichotomous items—such
as questions eliciting a yes or no response or asking the respondent’s gender, female
or male—and check-all-that-apply items.

EXAMPLES
Responses Related to Each Other
and to the Stem

Responses Unrelated to Each Other
But All Related to the Stem

What is your professional rank?

Which of the following hobbies do
you currently pursue?















Full Professor
Associate Professor
Assistant Professor
Instructor
Adjunct Faculty
Other

Bird watching
Camping or hiking
Cars or car repairs
Flying
Model trains/planes/cars
Photography

Alternative Response Scale Used to Obtain Factual Information
How many hours a week do you spend reading—including newspapers, magazines,
and books? (Check one)






c09.indd 209

Less than 1 hour
More than 1 hour, but less than 3 hours
More than 3 hours, but less than 5 hours
More than 5 hours, but less than 7 hours
More than 7 hours

7/9/07 11:50:55 PM

210

Designing and Constructing Instruments for Social Research and Evaluation

Alternative Response Scale Used to Measure Attitudes and Beliefs
Regarding the federal income tax, what actions do you believe Congress should take?
(Check one)






Replace the current income tax with a flat tax.
Replace the current income tax with a national sales tax.
Reduce all tax brackets by 10%.
Leave the system as it is.
No opinion.

Before considering specific criteria for writing this type of item, it is important to understand some general points. First, the purpose of these alternative
formats for selection items is to elicit information from the respondents through a
predetermined response set. As the instrument designer, you will have considered
beforehand whether to use an open-ended question or a response set. For example,
you might ask for a respondent’s job classification with an open-ended question,
or you might create a list of job classifications. An open-ended question might
obtain a broader range of responses; the disadvantage is that this broad range
could be difficult to categorize. One advantage of using an alternative response
set that limits choices can be that the results can be easily analyzed. If you decide
to limit respondents’ choices, your next task is to determine what the choices
in the response set will be. In the previous example using academic ranks, the
respondents are limited to six alternatives. As with all items in which you provide the choices, this narrowed focus can also turn to a disadvantage. Pretesting
is essential to ensure that you have included all the likely response choices for
each item.
An alternative response set may be relatively cumbersome if the question you
are asking can be fully answered with one number. For example, you could ask
respondents to indicate the number of children they have by circling a number on
a response set (1, 2, 3, 4, 5 or more), or you could obtain the information perhaps
more easily by asking an open-ended question such as, “Indicate the number of
children currently living in this household ———.” Consequently, it is important
to weigh both the benefits and limitations of using an alternative response set
rather than an open-ended item.
Second, alternative response scales are an effective approach when differences between choices are qualitative and not just quantitative, as in the previous example asking about hobbies. And these qualitatively different choices are
particularly useful when all or most of the response categories are known. This
provides the added advantage of ensuring that all respondents can select from the
same set of response categories.
Now let’s consider the guidelines for developing alternative response sets.

c09.indd 210

7/9/07 11:50:56 PM

Selection Items: Alternative Formats

211

State Items Clearly
The stem and responses should be stated clearly and unambiguously. It is important to
check that each item is not worded in such a way that different respondents
are likely to interpret it differently. One way to accomplish this is to read the
item aloud, because even when you do not see a problem, you may hear that a
word has more than one meaning in the context of the statement. Another way
to check for lack of clarity is to have several different people read each item and
then explain to you what they think the item or specific words mean. If their
understanding is different from what you intended, then you should rewrite
the item.

Ensure Language Is Appropriate
Item language should be appropriate to the respondents. As we discussed in Chapter Eight,
it is important to check your statements for readability, unnecessarily technical language, and unintentionally sensitive wording. In particular, if a response
set is used to collect demographic data, such as race, ethnicity, marital status,
or income, be sure that gathering this information is absolutely essential and that
items are carefully worded. Individuals may not answer questions when they are
uncomfortable with the terminology used: for example, researchers asking about
race need to be aware that during the 1960s and 1970s the term black replaced the
term Negro in common usage and that for the past twenty years the term African
American has been widely used in addition to black.

Write Stems Unidimensionally
The item stem should be stated unidimensionally. As noted in the previous chapter, this
means that only one attribute or trait should be described in the stem. Consider
the following example of a double-barreled item, which may be almost impossible
to answer as a respondent’s answer to the two-part stem may not match any
of the paired answers shown in the response set. The way to correct this, of
course, is to create two separate items.
EXAMPLE
Original: Identify your position and salary:
❑ Administrator ($50,000–$100,000 annually)
❑ Teacher ($25,000–$50,000 annually)
❑ Teacher’s aide ($10,000–$20,000 annually)

c09.indd 211

7/9/07 11:50:56 PM