Tải bản đầy đủ
Instrument 12. C: Confl ict Resolution Skills Assessment

Instrument 12. C: Confl ict Resolution Skills Assessment

Tải bản đầy đủ

310

Designing and Constructing Instruments for Social Research and Evaluation

INSTRUMENT 12.C: CONFLICT RESOLUTION
SKILLS ASSESSMENT.
Conflict Resolution Needs Assessment
Answer each question by providing the response that most accurately reflects your personal
view of your school.
1. I am a:

❏ student

❏ staff member

❏ parent

❏ other

2. Conflicts interfere with the teaching and learning process:
❏ often

❏ sometimes

❏ rarely

3. Problems between people at this school are caused by:
a.
b.
c.
d.
e.
f.
g.

expectation to be competitive
intolerance between adults and students
intolerance between students
poor communication
anger and/or frustration
rumors
problems brought to school from somewhere else

often








sometimes








rarely








4. Without exceeding 100% as the total, estimate the percentage of problems referred for
disciplinary action by the following categories:
a.
b.
c.
d.
e.

between students
between student and classroom teacher
between student and other staff members
between student and school rules
other

_______%
_______%
_______%
_______%
_______%
Total
100%

5. Indicate the types and frequency of conflicts experienced by students in this school:
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.

c12.indd 310

put-downs/insults/teasing
threats
intolerance of differences
loss of property
access to groups
rumors
physical fighting
verbal fighting
school work
other: _____________________________________

often











sometimes











rarely











7/9/07 11:52:58 PM

Organizing the Instrument

311

6. Indicate the effectiveness of each of the following actions in causing a student to change
a problem behavior:

a.
b.
c.
d.
e.
f.

very
effective







time out
detention
conference with an adult
suspension
contacting parent(s)
expulsion

somewhat
effective







not
effective







7. Without exceeding 100% as the total, what percentage of influence do the following
groups have in the way the school operates?
a.
b.
c.
d.
e.
f.
g.

students
teachers
parents
principals and school administrators
superintendents and district administrators
board of education
other

_______%
_______%
_______%
_______%
_______%
_______%
_______%
Total
100%

8. In this school, I am generally:

a.
b.
c.
d.
e.

treated fairly
treated with respect
given equal opportunity
treated with compassion
accepted

most of the
time






about one-half
of the time






not very often






9. I am allowed to solve problems that affect me:
❏ nearly always

❏ sometimes

❏ hardly ever

10. This school should do a better job teaching students to:
a.
b.
c.
d.
e.
f.

c12.indd 311

tell another person how I feel
disagree without making the person angry
respect authority
control anger
ignore someone who is bothering me
solve problems with other students

definitely yes







maybe







definitely no







7/9/07 11:52:59 PM

312

Designing and Constructing Instruments for Social Research and Evaluation

11. When I need help, I ask for it:
❏ nearly always

❏ sometimes

❏ hardly ever

12. If I needed help, I think I could get it from:
a.
b.
c.
d.
e.
f.
g.
h.

definitely yes









a parent
a brother or sister
another family member
a teacher
a counselor
another school staff member
another adult
another student

maybe









definitely no









13. I think this school has:
❏ more problems than most other schools
❏ about the same amount of problems as most other schools
❏ fewer problems than most other schools

Endnotes
1. Of course the danger here is that respondents might retaliate and not provide honest
responses.
2. Surveys administered over the Internet may include page counters, so that the respondent knows how many screens or pages of questions have been completed and how many
are left.

Key Concepts and Terms

c12.indd 312

anonymity

font

statement of purpose

branching question

introductory statement

title

confidentiality

questionnaire organization

typography

demographic section

order effect

typestyle

7/9/07 11:52:59 PM

Y
CHAPTER THIRTEEN

ADMINISTERING THE INSTRUMENT

In this chapter we will
• Provide guidance for administering an instrument that is completed by a
rater/observer.
• Provide guidance for administering self-report instruments.
Ultimately a piece of artwork is taken out of the studio and shared with
others, perhaps in an art gallery or as an illustration for a book or magazine.
Likewise, having constructed and pretested your instrument, you will take it out
of the “laboratory” and implement it in the real world.
Although this chapter focusing on administration is near the end of the
book, issues related to administration can and do influence the design and structure of the instrument. Therefore you should be thinking about administration
issues beginning with the earliest stages of the instrument construction process.
For example, knowing how you plan to deliver your questionnaire to a specific
target audience can influence your choice of wording, item type, and instrument format. At best we are providing an introduction to the process, and in fact
administration of instruments is a field unto itself. Therefore we encourage readers
to seek out additional resources, such as Kazdin (1982) for a thorough discussion
of using observer, or rater, instruments and Dillman (2000) and Babbie (1990) for
excellent descriptions of the processes of survey research.
313

c13.indd 313

7/9/07 11:53:31 PM

314

Designing and Constructing Instruments for Social Research and Evaluation

Administering an instrument is primarily a technical process and can be
divided into two broad categories: instruments completed by a rater, or observer,
and self-report instruments. For each approach you must consider certain issues
to ensure that the data obtained are reliable, usable, and meaningful.

Administering Instruments Completed by a Rater
Many instruments are designed to be used by an external rater, or observer, such
as performance evaluations conducted by managers and supervisors, developmental inventories administered by a therapist; interview guidelines used by program
evaluators; checklists completed by auditors or accreditation and licensure teams;
and behavior checklists used to support research in the social sciences. Regardless
of your instrument’s intended purpose, you should address the following considerations to successfully implement it:
• Select the setting for observation.
• Select a sampling strategy.
• Train observers.

Site Selection
Although we usually think of an instrument administration setting as a physical
environment, it also involves the time of day in which observations will be made,
the individuals involved in the process—both raters and those being rated—and
relationships between the rater and the object of measurement, be it an organization, individual, or inanimate object. To a large extent, issues related to site
selection should have been taken into consideration as you developed your instrument, so that you could tailor it to the particular situation.
Site selection may require obtaining entrance, such as approval to enter
an organization to interview employees or to conduct an observational study.
It is important to make contact with the organization, to obtain approval and
to learn what you might need to do to comply with the organization’s policies
and procedures. For example, you might want to study student-teacher interactions in a school system where you are not an employee. Initial discussions with
school administrators would be followed up in writing, in the form of a letter of
agreement or a contract, outlining the scope of the study and agreements about
confidentiality; selection of schools, teachers, and students to be observed; data
ownership; and how the results of the study will be shared with the school system.

c13.indd 314

7/9/07 11:53:31 PM

Administering the Instrument

315

Although entrance is not a problem with an instrument used internally, such as
a job performance evaluation, it is still important to administer the instrument in
conformance with organization policies, procedures, and objectives. Additionally,
if your instrument is to be used in a department other than your own, it may need
to be approved by the department manager.
In some cases preexisting agreements may facilitate entrance. Many educational, health, and human service organizations, including school systems, colleges
and universities, hospitals, and nursing homes, are licensed and accredited. If you
are a member of a survey team administering a checklist as part of an accreditation or licensing process, the organization is typically open to inspection, with
(and sometimes without) advance notice.
Studies based on observation or participation of individuals or review of confidential information in records require the informed consent of participants (from
parents or legal guardians when studying young children). The individuals who
gives informed consent agree to participate in your study and are fully informed
about procedures and interventions that will be used, their responsibilities as participants, the ways data will be collected and secured, and any risks that might be
associated with the process or intervention. If you are going to conduct a study
involving people, you should become familiar with participants’ rights and the
guidelines and regulations that protect participants, such as organizational procedures and state and federal regulations. For example, medical researchers should
comply with federal regulations for the protection of human subjects (Protection
of Human Subjects, 2001). Additionally, your study will require review and
approval by an institutional review board (IRB) if you are administering an instrument as part of a research project.
Additional issues when considering setting involve raters’ well-being and ability to do the work properly, time of day, methods of data collection, and the
effect of observers on the observed. Unless you are doing clandestine surveillance,
there is no reason why raters should be placed in environments that are physically
uncomfortable or that hinder their ability to make observations. For example,
individuals conducting record audits should have adequate lighting and a place
to sit with adequate desk space to open records for inspection. Given current
technology the instrument could be created as a word processing or database template so that the auditors can document their observations on a laptop or palmtop
computer. It is also important to consider the length of time over which observations will be made, so the opportunities for observers to take care of their physical
needs are incorporated into the observation plan. Safety is yet another concern,
both for the rater and the people being evaluated. For example, if the task is to
complete an inventory of books in a large metropolitan library, it might not be
safe to send an individual employee to a dimly lit storage area in the basement

c13.indd 315

7/9/07 11:53:32 PM

316

Designing and Constructing Instruments for Social Research and Evaluation

of the library without first ensuring that access to the space is controlled. To
further enhance staff safety you might plan to have the inventory conducted by
two or more employees working as a team. At no time should raters and clients
be placed in situations that compromise their physical safety in order to make
observation more efficient.
You might also need to think about time of day: for example, do observers
need to adjust their personal schedules in order to conduct observations at night?
Will observers need adequate daylight to videotape observations? The subject
of the observation might be available only at certain times or even seasons: for
example, you might be studying student behavior during spring break, or “beach
week.”
You need to determine whether the data collection and recording will be
done directly by people or by automated recording devices. Although we focus
here on the observation and collection of data by human beings, there are a
number of devices, such as video cameras and traffic counters, that can record
and store data, and many of the issues that matter for human observers matter for
automated recording devices as well. For example, electronic equipment may fail
if the temperature is too hot or cold, if the humidity is too high, or if the lighting
is inadequate to capture visual images clearly.
There is an extensive body of literature regarding the influence observers
have on those being observed. This relationship was first noticed during studies conducted between 1924 and 1933 at Western Electric’s Hawthorne plant
in Chicago. Led by Harvard researcher Elton Mayo, the researchers examined
how productivity and efficiency were influenced by physical changes in the work
environment, assembly processes, and relationships between workers. In perhaps
the best-known experiment the researchers created a carefully controlled environment for women who were assembling telephone components. No matter how
they modified the working conditions, the assembly workers’ output increased.
This ultimately led the researchers to the conclusion that their presence had as
much influence on the worker’s output as the environmental and situational variables that they manipulated (an outcome often referred to as the Hawthorne
Effect) (Albanese, 1981).
During the past seventy years the findings coming out of that study have been
heatedly debated. For example, did the researchers’ presence really influence work
output or were the workers reacting to other factors, such as the Great Depression
and the fear of losing their jobs? The importance of this to our discussion is that
researchers must be aware that people may change their behaviors in response
to being observed. Several actions can be taken to mitigate this problem. In most
cases it is important to inform people that they are being observed and to explain
the purpose of your observations. You can even show them the instrument and

c13.indd 316

7/9/07 11:53:32 PM

Administering the Instrument

317

explain how the observation will be conducted. If you believe that observers’
presence could influence behaviors, then you can try to have observers spend as
much time as possible with those being observed so that they become acclimated
to the observers’ presence and become less guarded.
An alternative approach for reducing observers’ possible effects is to make
observations in the field. Fieldwork is conducted in the environment where the
behavior to be observed normally occurs, such as a client’s home or a work
setting rather than a laboratory setting. Although the laboratory setting may be
more convenient for the researcher, anxiety and reactivity (behavior in response
to being observed) might be lessened when the observations occur in a setting that
participants find familiar or comfortable.
Unobtrusive observations occur when individuals are unaware they are being
observed, such as when observations are made through a two-way mirror. Unobtrusive observation may present ethical problems, and it is important to consider
its impact. Typically, it is conducted as part of a research protocol that has been
approved by an oversight body, such as a human subjects or institutional review
board. At a more pragmatic level, unobtrusive observation is not appropriate
when the findings will affect decisions about the individuals or programs being
observed. For example, a principal should not base a teacher’s performance evaluation on clandestine observations, as this is likely to create an atmosphere of
distrust.
The goal of identifying issues related to setting is to conduct your measurements efficiently and effectively, with minimal disruption by the rater and the
object of measurement. Some forethought and planning in light of the factors
just discussed should facilitate this process.
Sampling
Sampling involves making decisions about selecting who and what will be measured. For example, if you have developed a checklist to audit medical charts, you
may decide to review every chart, every fifth chart, or every fiftieth chart. Because
sampling is often associated with surveys, we will examine this aspect of sampling
in more detail in the next section, covering self-report instruments.
When applied to instruments administered by an external rater, sampling also
refers to such factors as the frequency and duration of observations and the number of clients to be observed. When we use rater instruments we often want to use
the findings to measure a single entity, such as an employee’s performance or a
specific behavior in a child. Or consider the researcher who wants to study classroom interactions. An instrument is developed that will measure verbal and nonverbal interventions, such as eye contact, physical touch, and verbal redirection

c13.indd 317

7/9/07 11:53:32 PM

318

Designing and Constructing Instruments for Social Research and Evaluation

of student behavior. The researcher has trained four graduate students who will
make classroom observations. The study will be conducted at two middle schools,
in one seventh-grade history class and one eighth-grade English class at each
school. Consent to participate has been obtained from the students and the students’ parents or guardians. To acclimate the students and teachers to their presence, the raters plan to spend at least two weeks in the classroom before using the
instrument to formally document their observations.
However, the researcher must still resolve several questions about the observation process. For example, will the observers rate only interactions between the
teacher and individual students or will they also rate interactions between
the teacher and groups of students? If group interactions are to be counted,
both the instrument and observation process must be designed to take this into
consideration. For one thing, in situations where multiple individuals in a group
are being observed, the sheer number of behaviors may limit observations to
whether the behavior did or did not occur, and behavior frequency and duration will not be measured. Another decision regards scheduling the observations.
Kazdin (1982) has identified four ways to schedule observations: frequency measures, discrete categorization, interval recording, and duration.
Frequency measures count the number of times an action or behavior occurs
within a given time period; for example, an instrument could be created to count
the number of times a teacher uses verbal praise to maintain classroom discipline
or to tally how frequently a child displays age-appropriate social skills. The behavior must be clearly defined to reduce ambiguity and enhance observer reliability.
Additionally, the definition should describe a distinct beginning and ending for the
behavior. In other words, the instrument should make it possible to distinguish
teacher verbal interventions to maintain classroom discipline from verbal interactions that are a part of classroom instruction, or to tell when children’s shouting
is an aspect of age-appropriate play rather than of anger.
To provide structure to the process, frequency measures are taken during a
predetermined time frame. The observation period could be the duration of the
class, such as fifty minutes, or at fixed intervals, such as the first twenty minutes of
each hour. An observer counting social behaviors might observe for ten minutes,
break for ten minutes, and then observe for ten minutes over the period of one
hour. “The rate of response each day can be obtained by dividing the frequency
of responses by the number of minutes observed each day. This measure will
yield frequency per minute or rate of response, which is comparable for different
durations of observation” (Kazdin, 1982, p. 27).
Discrete categorization differs from using frequency measures in that the
instrument is designed to measure both occurrences and nonoccurrences, that
is, to document whether the behavior of interest was present or not present, or

c13.indd 318

7/9/07 11:53:32 PM

Administering the Instrument

319

performed correctly or incorrectly, within a given time period. If you were
interested in measuring verbal praise as a means of maintaining classroom discipline, you would identify, during pretesting, a number of words, phrases, or
statements that convey positive recognition. You would then measure the presence and the absence of this verbal praise in response to certain behaviors and
situations during fixed intervals, such as fifteen-minute intervals. For example,
if a teacher verbally redirects a student by saying, “Please take your seat,” does
the teacher follow up with “thank you” or some other positive statement when the
student displays the desired behavior? Similarly, a checklist for a developmentally
delayed child who is being taught to perform activities of daily living such as
tying her shoe, toileting, and bathing might measure whether or not a behavior
occurred and whether it was executed appropriately.
When you use interval recording the focus is on time intervals rather than discrete behaviors. You are interested in whether the behavior did or did not occur
during a fixed period of time rather than its frequency. You might design your
instrument for observations at five-minute intervals during a fifty-minute class.
Regardless of the actual number of times the teacher used verbal praise during
each five-minute block of time, it would be recorded as having either occurred or
not occurred during that time. A variation of interval recording is time sampling.
Rather than making your five-minute observations during one class period, you
might spread them out over a longer period, such as the entire school day.
Finally, you might want to base your observations on duration. For example,
you might be interested in how much time the teacher spends in reinforcing positive classroom behaviors during a fixed interval, as compared to the amount of
time spent in classroom instruction.
Given these choices, you can understand why it is important to consider the
design of an observer instrument, as the structure and format of the instrument
should support the measurement process and will influence how it is administered.
Observer Training
In the fourth chapter we defined interrater reliability and explained how to compute the percentage of agreement (the level of agreement between the ratings of
two independent observers). Having constructed and pretested your instrument,
you will still need to consider how observers will be prepared and trained to
administer the instrument consistently. You should consider this an integral part
of the instrument construction process, as improper administration can compromise the validity and reliability of your data and findings and may ultimately
adversely affect decisions about individuals, such as academic placement,
entrance into a program, and evaluation of work performance.

c13.indd 319

7/9/07 11:53:32 PM

320

Designing and Constructing Instruments for Social Research and Evaluation

The most frequently used method of training observers involves vignettes,
either written or videotaped. Using vignettes is preferable to training observers in
the field, where their activities may be disruptive to programs or clients.
A written vignette might take the form of a brief case history. Each rater individually reads the vignette and then uses that information to score the instrument
that is to be administered. The raters then compare their scores and discuss the
reasons for any different interpretations of the vignette and their individual ratings. Questions about the meanings of words or items can also be answered at this
time, as well as questions about the observation process. This scoring and comparing process is repeated a number of times, until the raters can demonstrate a fairly
high level of consistency. Kazdin (1982, p. 73) notes that “traditionally, agreement
was regarded as acceptable if it met or surpassed .80 or 80 percent, computed by
frequency or point-by-point agreement ratios. Research has shown that many factors contribute to any particular estimate of agreement. Hence, it is not only the
quantitative estimate that needs to be evaluated, but also how that estimate was
obtained and under what conditions.” In other words, even though a high level
of agreement is desirable, if you are unable to obtain 80 percent agreement or
more, it is important to look at other factors that might be influencing the process,
such as problems with the instrument, the observation method, or the training
process.
The training program for scoring the Child and Adolescent Assessment Scale
(CAFAS) offers an example of using vignettes. The CAFAS assists mental health
professionals to evaluate children and adolescents with emotional, behavioral,
and substance abuse problems, and the CAFAS Self-Training Manual (Hodges, 1996)
uses two sets of vignettes to help raters develop their skills. In the first set, six case
histories are presented along with completed and scored instruments. Explanations describe how the ratings were obtained, and raters can compare their scores
with the scores that the instrument designer has indicated are appropriate for the
case. The second set of vignettes contains ten case histories. Raters complete each
vignette and then compare and contrast their scores with each other’s scores and
with an “answer key” of recommended ratings. Consequently, raters have sixteen
opportunities to hone their skills in completing the instrument.
Videotaped vignettes are another medium for observer training. Observers
view a video of the behaviors or situation that they will rate—it may show a
simulated or a real-life situation—and have a chance to practice and sharpen
their scoring. The advantage of video vignettes is that observers can go back over
a vignette repeatedly to see and discuss the behaviors they are evaluating.
In some situations you may need to provide training on an ongoing basis. For
example, if your organization routinely hires or promotes employees into supervisory positions, then you should regularly schedule performance-rating training

c13.indd 320

7/9/07 11:53:33 PM