Tải bản đầy đủ
Instrument 1. C: Example of a Checklist

Instrument 1. C: Example of a Checklist

Tải bản đầy đủ

Introduction

25

INSTRUMENT 1.C: RESEARCH EVALUATION CHECKLIST.
Significance of the Study
❑ What makes the study useful, important or of interest to you?
❑ What do you want to know from the research?
❑ Did the researchers clearly explain why they did the study?
❑ Have the researchers convinced you of the importance of the study?
❑ Does the study address a gap in knowledge or provide new information about the
topic or issue?
❑ Were constructs, variables and terms clearly defined?
❑ Was a clear rationale presented for the constructs or variables examined in the
study?
❑ Were consumers or family members involved in helping to design the study?
❑ Were board members, administrators or agency staff involved in helping to design
the study?
Research Questions and Hypotheses
❑ Were the research questions and hypotheses clearly stated?
❑ Did the research questions and hypotheses accurately forecast what would take
place in the study?
Methodology
❑ Were the procedures clearly described?
❑ Could someone repeat the study after reading the methods section?
❑ Were appropriate criteria used to select the sample?
❑ Were methods used to prevent bias in the study?
❑ Were the procedures and instruments reliable and valid?
❑ Were the procedures and instruments free of potential bias (for example, age,
gender, racial or ethnic)?
❑ Was a control group needed to address the main question of the study?
❑ Was an appropriate control or comparison group used in the study?
❑ What steps did the researchers take to prevent harm or distress to the research
participants?
❑ Was the study approved by an institutional review board (IRB) or a similar committee on ethics?
Sample, Representativeness, and Generalizability
❑ Did the researcher justify the size of the sample?
❑ Did the researcher describe the methods used to determine sample size?
❑ Was the sample size sufficient to find significant results?
❑ Do the characteristics of the sample match those of the population of interest?
❑ Did the study setting match the location where the results will be applied?

c01.indd 25

7/9/07 11:45:59 PM

26

Designing and Constructing Instruments for Social Research and Evaluation

Statistical Methods and Results
❑ Did the researchers use appropriate statistical tests to evaluate hypotheses or
answer the research questions?
❑ Did the researchers use the appropriate statistical tests for small sample sizes?
❑ Did the researchers clearly identify the major findings of the study?
Discussion, Limitations, and Implications

❑ Did the researchers identify the limitations and biases of the study?
❑ Did the researchers discuss how the limitations and biases influence the
results?

❑ Did the researchers discuss recommendations from or practical implications of the
findings?

❑ Did the researchers discuss what future studies could be done on this topic or
issue?

❑ Did the researchers state how they will use the findings?
❑ Did the researchers recommend how others can use the findings? For example, did
they indicate that the findings might be used to:
❑ improve a current practice or service?
❑ develop or revise a policy?
❑ change a mental health law?
❑ support a request for funding?
❑ support implementation of a new program or service?
Source: Adapted from Heacock, Koehoorn, & Tan, 1997; Kazdin, 1998; Krathwolh,
1988; Centers for Disease Control and Prevention, 2006; National Alliance on Mental
Illness, 2006; Lutz, 2006. Reprinted with permission.

Endnotes
1. Despite our best efforts, we humans “filter” what we observe, based on prior experience.
Consequently, two people may see the same event, but interpret it differently, and if asked
to measure the event using some form of observational instrument, they may come up
with decidedly different ratings; consider, for example, the scores different judges give to
an ice-skating performance.
2. Scriven (2000, p. 1) defines checklist as, “a list of factors, properties, aspects, components,
criteria, tasks, or dimensions, the presence or amount of which is to be separately considered, in order to perform a task.”

c01.indd 26

7/9/07 11:45:59 PM

Introduction

27

Key Concepts and Terms

c01.indd 27

attitude scale

objective

selection item

behavior rating

observation

self-report

checklist

poll

stem

instrument

performance rating

subjective

inventory

psychometric instrument

supply item

item

questionnaire

survey

Likert response scale

rating scale

test

mode of administration

response set

7/9/07 11:45:59 PM

Y
CHAPTER TWO

INSTRUMENTS AND SOCIAL INQUIRY

In this chapter we will
• Describe approaches to social inquiry.
• Examine how methodology influences the selection and construction of
instruments.
As with any human endeavor, there is a history helping to explain how and
why we do what we do when we engage in social inquiry. The purpose of this
chapter is to provide a conceptual basis for the use of social science instruments
in social inquiry. We explain what makes the social sciences a science, describe
approaches to systematic inquiry, and explain the role of instruments in measurement and information gathering.

Instruments and Questionnaires in the Context of
Social Science Research
Social science refers to the application of a systematic approach to understanding
human actions and interactions. Social scientists refer to this process as sense
making, an attempt to give meaning to our existence by asking questions that lead us
to try to explain such very basic concepts as, What is truth? What is reality? How
28

c02.indd 28

7/9/07 11:46:37 PM

Instruments and Social Inquiry

29

do we make sense of the world? The approaches used in social science inquiry
are the same across the social science disciplines: political science, sociology,
anthropology, education, psychology, and management and business administration.
Ultimately, the purpose of social science and of the methodologies used to conduct
social inquiry is to better understand, explain, and influence human behavior.
From an historical perspective, the social sciences were not always scientific.
Early attempts to explain human behavior were often based solely on observation,
description, and conjecture, and for much of human history, events were explained
as being the result of divine intervention, luck, or chance. Such basic attempts to
comprehend situations are referred to as ordinary knowing, and for thousands of
years this was the primary approach to sense making ( Judd, Smith, & Kidder,
1991). Ordinary knowing is characterized by intuition, hunches, observation, inference, and past experience. Historically, understandings of beliefs, social relationships, and human behavior were most often formulated on political and religious
philosophy (for example, the concept that kings obtained their authority to rule
through divine right), rather than on systematic inquiry to define underlying causal
relationships. This began to change in the mid- to late nineteenth century, when
it was demonstrated that the methodologies employed in the natural and physical
sciences, including measurement, experimentation, systematic observation, and
the use of statistical analysis, could be adapted to questions of social relevance.
In the late nineteenth century, research in biology and medicine increased
understanding of human physiology, and the work of Darwin helped to build a
conceptual basis for understanding human development. If human physiological
mechanisms could be explained, perhaps the cognitive and behavioral attributes
of humans, individually (psychology) and collectively (sociology), could be as well.
And if human biology could be examined by application of scientific methods,
why couldn’t these same methods be used to understand human thought and
behavior? Consequently, in the late nineteenth and early twentieth centuries, the
methods used in the physical sciences were applied to the study of human behavior
and relationships, resulting in social science as a systematic discipline of inquiry.
One of the factors that helped to “legitimize” the social sciences as a science
was the use of measurement and statistical analysis. Measurement provides a means
for the systematic classification and coding of information. Measurement is a critical factor in the development of instruments because we want to construct items
that will produce data that can be organized in a coherent, logical, and replicable
fashion. In turn this supports a methodical approach for analysis, such as the use
of statistics to examine trends, patterns, and associations or to test hypotheses.
The social sciences also gained legitimacy by employing probability theory
and statistics in describing and explaining human behavior. Probability had
long been used in biology and the other natural sciences for understanding and

c02.indd 29

7/9/07 11:46:38 PM

30

Designing and Constructing Instruments for Social Research and Evaluation

predicting variation among observations. By the early nineteenth century, probability theory was being applied as a way of understanding the effects of population growth. Statistics were originally referred to as political arithmetic. The term
statistic is derived from the German word statistik, which meant information important to matters of the state. By the late 1800s and early 1900s, statistical analysis
had become the primary means of analyzing data in the social sciences. For
example, Frederick Taylor, referred to as the “father of scientific management,”
carried out his work at the turn of the twentieth century and was among the first
to use statistics to measure and predict work output (Albanese, 1981). It was also
during this period that British statistician Karl Pearson derived his formula for
calculating correlations (the strength of a relationship between variables), which is
now foundational for many forms of social research. With measures and statistics
as analytical tools, social scientists could finally lay claim to methods as rigorous
as those employed in the natural and biological sciences (Porter, 1986).
The adaptation of scientific methods, measurement, and statistical analysis to
social inquiry created a systematic approach, overcoming many of the shortcomings present in methodologies based solely on observation and description. First,
social science research is grounded in theory. In general use theory often refers to
an unverified assumption, but a scientific theory is a logical explanation that has
been tested repeatedly and is well supported by evidence. Researchers develop
hypotheses (assumptions) about these theories and then conduct empirical
research, or systematic observation and measurement, in an attempt to examine and
produce evidence to support or refute these hypotheses. Scientific theories are
never fully proven; however, they are confirmed as hypothesis testing develops
a body of supportive evidence. The strength of this evidence is often increased
when a study replicates and builds upon prior research.
Second, systematic inquiry requires that there be rules and standards to guide
research. Such guidelines ensure that other researchers can replicate a study in
an attempt to support or refute earlier findings. They also ensure that the sources
of information used in a study are credible and appropriate for the research.
A popular belief or explanation may sound and appear plausible, but to be proven
it must withstand the scrutiny of systematic inquiry and replication. For example,
a study published in a prestigious medical journal found that infertile women
who were prayed for by prayer groups became pregnant twice as often as those who
did not have people praying for them. However, upon investigation it was found
that the research methods were unsystematic, the results could not be reproduced,
and questions were raised about the integrity of the researchers (Flamm, 2004).
Third, social science research employs accepted methodologies for conducting research, as well as accepted methodologies for collecting, analyzing, and
interpreting data. There are a number of these methodologies, and a research

c02.indd 30

7/9/07 11:46:38 PM

Instruments and Social Inquiry

31

question may suggest a specific method for collecting and analyzing the data, or
it may suggest a variety of methods and the researcher must choose the one that
best fits his or her skills, situation, and resources. Indeed, in the real world, inquiry
methods and data are never completely tidy:
A scientist is characterized neither by a willingness to believe or a willingness
to disbelieve, nor yet a desire to prove or disprove anything, but by the desire to
discover what is, and to do so by observation, experiment, verification, and
falsification. So doing, the scientist expects that others will take the trouble to
check his findings, for it is only by such independent testing that his finding can
be verified. Scientists do not believe in fundamental and absolute certainties.
For the scientist, certainty is never an end, but a search; not the ordering of
certainty, but its exploration. For the scientist, certainty represents the highest
degree of probability which attaches to a particular judgment at a particular
time level, a judgment or conclusion that has been arrived at by experiment,
inference, or observation. . . . Scientists lack a superstitious regard for the catchwords of science, and believe that all knowledge is infinitely perfectible
[Montagu, 1984, pp. 7–8].

Until the mid-twentieth century, scientific research was positivist in outlook.
The philosophy of positivism holds that only that which can be observed can be
studied, for every effect there is a cause, and it is possible to understand the world
well enough that phenomena can be predicted and controlled (Trochim, 2001).
The natural and physical sciences set the stage for positivism because, for the
most part, these sciences were working with constants, such as the speed of light
or the number of electrons in an element. These constants hold true even though
individuals may experience them differently. Additionally, physicists, astronomers,
biologists, and chemists could directly observe many of the things they wanted to
measure. For example, Galileo was the first to use a telescope to observe Jupiter’s
moons and calculate the motion of their orbits.
In contrast, social science phenomena behave in a very different manner.
Unlike physical phenomena, the activities that social scientists study are situational: that is, they may differ within and between situations or settings, and they
may change over time. For example, people’s attitudes toward government and
political leaders may change in response to these leaders’ statements and actions,
a change that may appear as a rise or fall in a presidential approval rating. Additionally, the social sciences acknowledge the existence of multiple realities: Social and
cultural phenomena are experienced differently by individuals, and social science
attempts to understand the world as experienced by individuals or groups of
individuals through their shared realities. Finally, social and behavioral scientists are
often interested in attributes that cannot be directly observed, such as psychological

c02.indd 31

7/9/07 11:46:38 PM

32

Designing and Constructing Instruments for Social Research and Evaluation

and social states, including depression, anxiety, self-esteem, locus of control, and
the like. At best, social science instruments attempt to measure characteristics
associated with these states (referred to as constructs ). If a social scientist is
interested in knowledge and cognitive abilities, the instrument might attempt to
measure some observable aspect of knowledge, such as the ability to recall information or the breadth of an individual’s vocabulary. Consequently, social science
instruments may or may not obtain information that represents a direct measure
of the subject of interest.
Today, most physical and social scientists have adopted a philosophical
approach to inquiry referred to as postpositivism. This means that although the
methods used to examine phenomena must be rigorous and systematic, it is
understood that not all phenomena can be easily understood, explained, or
predicted. Contemporary astronomers and physicists, for example, now find
that the things they need to measure often cannot be directly observed, such as
atomic particles. Unable to account for all of the matter and energy that their
calculations demonstrate must exist in the universe, they have postulated the
existence of dark matter and dark energy, which can be inferred from gravitational effects but are undetectable using current methods of measurement
(Freedman & Turner, 2003).
Lincoln and Guba (1985) make this distinction between positivism and postpositivism (which they refer to as the new paradigm): “Where positivism is concerned
with surface events or appearances, the new paradigm takes a deeper look. Where
positivism is atomistic, the new paradigm is structural. Where positivism establishes meaning operationally, the new paradigm establishes meaning inferentially.
Where positivism sees its central purpose to be prediction, the new paradigm is
concerned with understanding. Finally, where positivism is deterministic and bent
on certainty, the new paradigm is probabilistic and speculative” (p. 30).
In addition to trying to answer why questions, social scientists are often
interested in what, how, and if questions. For example, they may be interested in how
a particular program works, if clients are receiving the intended outcome of services, or if employees are carrying out their job duties as assigned. The results of
such inquiry are often used to make value judgments about programs, processes,
people, and products, such as whether a program should continue to be funded,
whether services need to be added or reduced, or whether an employee will
receive a raise in pay. For social scientists investigating such questions, the use of
instruments allows them to measure program and individual efforts and effects
consistently and systematically.
It is sometimes difficult to believe that many of the approaches we currently
use to collect and analyze data were developed just a few generations ago, and that
others have been developed in our lifetimes. For example, systematic measurement,

c02.indd 32

7/9/07 11:46:39 PM

Instruments and Social Inquiry

33

even in the physical sciences, is relatively new. In eighteenth century France, there
were multiple measures for a foot, which varied between 10.6 and 13.4 inches,
and different measures for a pound, bushel, and liter. During that period, physical
measures differed not only between countries, but between cities within a country
and even between neighborhoods within a city (Strauss, 1995). One of the first psychometric instruments, the Stanford-Binet Test of Intelligence, was developed by
Alfred Binet in 1905 and adopted for systematic use in the United States by Lewis
Terman in 1916 (Kamin, 1995). Development of tests and measures of academic
achievement flowered in the early twentieth century, as reflected in the publication
of E. L. Thorndike’s pioneering text on measurement in 1904 (Worthen, Borg, &
White, 1993). In regard to survey questionnaires, Babbie notes that even though
one of the first surveys of political attitudes was conducted in 1880, for the most
part “contemporary survey research is a product of American researchers in this
century” (1990, p. 37). For example, it was during the 1924 and 1928 presidential
campaigns that the Hearst newspaper chain ran preelection polls in almost every
state of the union. George Gallup started his polling organization in 1935, and
in 1936, using a random sample of potential voters, successfully predicted that
Franklin Roosevelt would be reelected president (Newport, 2004).
It was also during this period that the format for questionnaire items was
being honed. In 1929, L. L. Thurstone explored the theory of attitude measurement and developed a schema for categorizing responses along a continuum
that allowed measurement on both sides of the continuum; that is, responses to
a series of items could be used to measure both agreement and lack of agreement. In 1932, Rensis Likert developed a format for item construction that asked
respondents to indicate their level of agreement with an item (strongly agree, agree,
disagree, and strongly disagree). This has probably become the most frequently used
format for constructing items to measure opinions and beliefs ( Judd et al., 1991).
Additionally, the values associated with the items could be added together to create a score reflecting the strength of the relationship between the items and the
underlying concept that the instrument attempts to measure.
Over time the methods developed for social science research have been
adapted to virtually all activities requiring measurement and analysis. The following section describes the principal approaches to social inquiry. Although there are
many ways to gather information, almost all require some form of instrument.

Methods of Inquiry
Instrument construction is just one facet of the larger process of discovery and
decision making. To ensure that you are obtaining the information you want

c02.indd 33

7/9/07 11:46:39 PM

34

Designing and Constructing Instruments for Social Research and Evaluation

and need, it is important to clarify the purpose of your study, define the questions
that need to be addressed, identify potential sources of information, construct
data-gathering instruments, and apply a systematic approach for collecting and
analyzing the data. In this section we will outline a process for focusing your study
or project and describe the relationship between instruments and approaches to
data collection and analysis (illustrated in Table 2.1).
Table 2.1 illustrates the process of defining a project and the relationship
between instruments and other steps in the process. The first phase is to state
the purpose of your project; this can be stated as a question, such as, “What
is the relationship between standardized testing and curriculum development?”
or a declarative statement (hypothesis), such as, “Standardized tests lead to more
effective curriculum.” Not all instruments are developed for the purpose of social
inquiry; they may also be designed for such more mundane but pertinent activities as evaluating people, products, processes, and programs in conjunction with
problem solving and decision making. For example, you may want to develop
an observation instrument to provide teachers with information about student
behaviors to improve the teachers’ classroom management skills, or you may
need a screening instrument to determine the level and type of services to provide to clients referred to a mental health clinic. Regardless of the underlying
rationale—social science research or evaluation for decision making—it is important to start the project by focusing and clarifying the process.
The next step is to formulate the questions that need to be answered in order
to understand and, possibly, resolve the problem. For example: What are “good”
and “bad” classroom behaviors, and can I distinguish between them? Is it important to know how often these behaviors occur? Are there student characteristics
that I need to be aware of ? Are there teacher characteristics that affect student
behavior? And so forth. These questions bring into focus what will be studied and
what data will need to be generated in order to answer the questions.
Methodology refers to how we will go about understanding the phenomenon
or question of interest and addresses the approaches used to collect, analyze,
and interpret information. As we will see in the next section, methodology suggests informational needs and sources and thus the type of instrument needed,
which in turn will suggest whether you can use an existing instrument (off the
shelf or adapted to your situation) or will need to construct an instrument to meet
your unique circumstances and informational needs.
There are two broad categories of methodologies: qualitative and quantitative. Qualitative methods involve “open-ended explorations of people’s words,
thoughts, actions, and intentions” as a means of obtaining information ( Judd,
et al., 1991, p. 299). An interview that results in a written transcript, field notes
reflecting direct observation of participants, and video-recordings are examples

c02.indd 34

7/9/07 11:46:39 PM

c02.indd 35

7/9/07 11:46:39 PM

Identify the
mechanism(s) for
obtaining information
to support decision
making and
understanding. May
be off the shelf or
developed to meet the
particular situation and
informational needs.

Formulate the
questions that help to
focus the study by
pointing out and
clarifying the areas
of concern. Can also
identify resources
needed to implement
the study.

Source: Adapted from Worthen and Sanders (1987, p. 244).

Instrument

Questions
Describe the specifics
of the information
collection procedures.
For example, how,
when, to whom, and
under what conditions
will the instrument(s)
be administered?

Information
Collection
Strategies

Describe the process
that will be carried out
to facilitate answering
the questions. To what
extent do the data and
how they were
obtained influence
how they should be
analyzed?

Information
Analysis &
Interpretation

Reporting
Consider the reporting
decisions: Who should
receive the
information? What
type of information
should each of the
individuals receive?
When should people
receive it? In what
format should it be
presented?

Methodology
State the conceptual and technical basis for collecting, analyzing, and interpreting the data
(quantitative, qualitative, or mixed method).

Purpose Statement
State the major consideration, concern, or question to be addressed.

TABLE 2.1: STUDY PLANNING GRID.