Tải bản đầy đủ
Instrument 14. A: Example of a Web Survey

Instrument 14. A: Example of a Web Survey

Tải bản đầy đủ

348

Designing and Constructing Instruments for Social Research and Evaluation

INSTRUMENT 14.A: WEB QUESTIONNAIRE.
Parent Survey of Stonewall Jackson Middle School (SJMS)

Please click on the button to record your selection. This questionnaire contains 10 items and
should take no more than five minutes to complete. Thank you for your participation. Please go
to www.stonewalljackson.edu/survey to view the result.
Strongly Disagree

Strongly Agree

I feel that my child receives a good education
at SJMS.
The faculty of SJMS maintains good
classroom discipline.
The school building is well maintained.

My child’s teacher communicates his or her
progress to me regularly.
Homework assignments appear to be
meaningful.
SJMS faculty do a good job of preparing my
child for the state’s standardized test.
Classrooms are decorated attractively
(posters, pictures, etc.) to invite learning.
Number of children attending SJMS:

1

No

I am a member of the SJMS PTO.

Yes

My child/children get to school by:

walking
taking the school bus
are driven to school by car

Additional Comments:

Click here to enter your data and exit
this survey:
Exit

c14.indd 348

7/10/07 12:43:57 AM

Computers and Instrument Construction

349

Key Concepts and Terms

c14.indd 349

coverage error

e-mail survey

pull-down box

data analysis

instrument design

survey software

data entry

instrument organization

Web survey

database

item construction

7/10/07 12:43:57 AM

Y
CHAPTER FIFTEEN

MANAGING THE DATA AND REPORTING
THE RESULTS

In this chapter we will
• Provide guidance for managing the data produced by an instrument.
• Explore who owns the instrument and the data it produces.
• Describe ways to report survey results to different audiences.
Throughout the United States artists find opportunities to display and sell
their work. As we noted earlier, ultimately, an artist must take his or her creation
out of the studio and place it before the public, and the time will come when
you, the instrument designer, will need to not only administer your instrument
but share the results of that endeavor as well.
In this chapter we address managing the data obtained by administering an
instrument and also reporting the results. We use the term data management broadly
here to include such processes as organizing and cleaning the data for analysis as
well as securing the rights to the data and the instrument itself. Reporting covers
how information from an analysis can be presented, to whom, and under what
conditions. For example, the results from a psychometric instrument might be
used by a clinician or treatment team, in conjunction with the client, to develop
a treatment plan. The results from administering a job evaluation might be used
to determine who receives a pay raise or gets promoted. The results from an
organization survey might be used to improve the service delivery or product
350

c15.indd 350

7/10/07 12:00:36 AM

Managing the Data and Reporting the Results

351

quality. Although different situations necessitate different types of instruments,
which produce quite different information, if you have followed the advice given
in these chapters you will have created a good instrument that provides you and
other users with information that can be used with confidence in a variety of
decision-making situations.

Data Management
Let’s say that you have created a questionnaire containing thirty Likert type rating
items that you plan to administer to 150 police officers in the local law enforcement office. Now is the time to consider how you are going to manage the 4,500
pieces of data the survey will produce. Data management is important because it
is another step in the instrument construction process that can affect the accuracy
and trustworthiness of your results. Considerations include entering the data,
organizing data for analysis, coding, handling errors, securing the database, and
determining who owns the data. Additionally, qualitative and quantitative data
will be managed differently in some respects.
The first phase of data management involves organizing the data for analysis.
Fortunately, you can use computer programs dedicated to data management or
analysis, such as database, statistical, and spreadsheet software. Database software, such as Microsoft Access and IBM’s Lotus Approach, have a user-friendly
interface and excel at sorting and tabulation, making them suitable for examining
qualitative information generated from open-ended items. Data are entered in a
table. The column variables, referred to as fields, allow you to organize the data
into categories, such as text (names, dates, addresses), and numerical values, such
as coded responses. Row variables make up a record; a record contains all the data
about one person, place, or thing. Certain database software, such as Ethnograph,
is intended specifically for analyzing narrative information, including interview or
focus group transcripts, field notes, diaries, and meeting minutes. These programs
can store, categorize, and sort (based on codes that you develop) words, phrases,
sentences, and paragraphs, as well as photographs and audio and video files.
Statistical software is dedicated to handling large data sets and conducting
quantitative analysis. Database software supports basic calculations, but statistical software is specifically designed to carry out all manner of quantitative data
analysis. There are many statistical software applications on the market, of varying prices and complexity. When you purchase the Statistical Package for the
Social Sciences (SPSS), Statistical Analysis Software (SAS), or Minitab, you buy a
license to use the software for a limited period, and the price may vary depending
on the number of modules you purchase or on whether you can buy a student

c15.indd 351

7/10/07 12:00:37 AM

352

Designing and Constructing Instruments for Social Research and Evaluation

version at reduced cost. Some free statistical software can be downloaded from the
Internet (when this book went to print, links to statistical freeware were available
at http://statpages.org).
Spreadsheet software lies somewhere in between database and statistical
applications, as it is effective for sorting and tabulating and current versions also
support both basic and advanced statistical calculations. Early versions were
limited in the amount of data that could be entered. Today one of the more
commonly used applications, Microsoft Excel, supplies 256 columns and 65,536
rows, for a total of 16,777,216 cells, in one spreadsheet; this should be more
than enough data storage space for most users. Most software applications allow
you to transfer data; for example, even if you originally entered your data into a
spreadsheet, you may be able to import them into database or statistical software
for another level of analysis.
Typically, the choice of software is dependent on the user’s knowledge and
experience, as learning how to use new software applications adds to the time
needed to complete data analysis. For studies where you need to generate frequencies, compute descriptive statistics such as means and standard deviations, and
plot graphs, spreadsheet software will be more than adequate. If you are interested in exploring the relationship between variables, such as demographic data
and ratings, then statistical software may facilitate that level of analysis. If you
have a lot of narrative responses, then database software, particularly applications
designed for qualitative analysis, should be considered. Regardless of the type of
software you use to organize and analyze your data, there are a number of steps
you should take to prepare those data for analysis: (1) develop a coding guide, or
structure; (2) enter the data; (3) check and clean the data for accuracy; and (4)
create a process to safeguard the data.
Develop a Coding Guide
Coding means using a systematic process for entering data for analysis. Depending
on your software you may be able to record text directly; the application may be
able to treat terms like male and female as if they were numerical data, counting the
frequency of occurrences as well as using them in quantitative analysis. In many
cases the item format will facilitate coding; it may be logical to use 1 and 2 to
identify dichotomous variables like gender or yes or no responses. Back in Chapter
Seven we noted that you could label response scale alternatives with numbers such
as 1, 2, 3, . . . to represent, for example, strongly disagree, disagree, undecided, and so
forth. These numbers may also be used to code the responses. Or you may have
written responses, such as the name of a political party, that you can code with a
number, such as 1 = Democrat, 2 = Republican, 3 = Independent. Although you could use

c15.indd 352

7/10/07 12:00:38 AM

Managing the Data and Reporting the Results

353

an alphabetical coding scheme (A, B, C, and so on), numerical coding facilitates
computer analysis of the data, particularly for multi-item scales that produce a
true numeric value.
By convention, the number 9 is used in coding to indicate missing data. If
an item has more than nine response choices, or values (as a list of job titles
might, for example), then the numbers 99 or 999 can be used for this purpose.
The number 9 is used, rather than 0, because a questionnaire may use 0 to
represent a null value, such as number of children or number of previous arrests.
Similarly, the number 7 is used to code a don’t know response (Welch & Comer,
2001).
You should record your coding system in a codebook, along with any rules
or standards you have created for entering data. Judd, Smith, and Kidder (1991)
observe that “all too often, researchers fail to maintain an adequate and detailed
codebook. As a result, when they attempt to return to a set of data after some interval, perhaps as short as a few days or weeks, they have a difficult time reconstructing what the numbers in the data matrix really mean. Codebooks should therefore
be complete and detailed. Further, multiple copies should be made and they
should be stored in safe places” (p. 358). Coding can also help you to maintain
confidentiality: if all items are coded, including identifying information such as
names, e-mail addresses, and telephone numbers, it may be impossible for someone who does not have access to the codebook to identify a participant from the
information in the database.
Enter the Data and Check for Accuracy
As little as fifty years ago databases were paper-and-pencil documents and the
data were entered and tabulated by hand. Researchers had to wait until the early
1970s before handheld calculators with basic statistical functions were marketed
(and believe it or not these early models sold for $100 to $150). You now have a
number of ways to enter data into a computer database, including typing data in
manually, using optical character recognition (OCR) software to scan the instrument (or its separate answer sheet), and having respondents enter data directly, as
they do in completing an electronic survey on the Internet. All three approaches
are open to error as you, the respondent, or the optical scanner might enter data
incorrectly. And an error rate of only 2 percent for the instrument producing
4,500 bits of data would result in nearly 90 incorrect database entries.
There are several methods available to improve data coding quality. You
can spot-check by selecting items at random and comparing the data entered
in the database to the data recorded on the paper-and-pencil instrument.
You should always visually inspect the data to identify blatant errors, such as a

c15.indd 353

7/10/07 12:00:38 AM

354

Designing and Constructing Instruments for Social Research and Evaluation

450-year-old respondent, a rating of 9 on a 5-point Likert scale, or a net income
of $19,000 a year. Just as you have someone proofread the instrument for spelling
and grammatical errors during the pretesting phase, it is helpful to have someone
proofread the database for possible errors. If you have guaranteed confidentiality
or anonymity, you must first delete all uncoded data that would give a reviewer
sufficient information to identify respondents. Aside from names, you might need
to delete birthdates, job titles, and addresses, as well as unique identifiers, such as
a driver’s license number, that taken singularly or in combination might point to a
particular individual.
Some computer programs require you to make each entry twice and will
not allow you to save the data if the two entries do not coincide. In some software applications you can (or must) first determine validation rules that govern
how data can be entered. For example, you can specify that only text, numbers,
dates and times, or monetary values can be entered; i9,000 will be rejected if
the field requires a numerical value. You can also specify the number of characters, so that you cannot inadvertently enter two numbers in a field that requires
only one.
In an ideal world your pretesting will have been so effective that respondent
or raters will not make mistakes as they complete the instrument. In the real
world you should anticipate such errors as marking multiple items, scratching
out responses, and misaligning responses (Figure 15.1). Multiple marking occurs
when the respondent or rater makes a selection, changes his or her mind, and
then makes another selection but does not cross out the first choice. Scratch outs
appear when a respondent crosses out a first choice and marks another; however,
the scanner cannot differentiate between the markings and so may not record the
response. Misalignment can occur in two ways. At the item level, the response
choices may be grouped together so closely that the respondent inadvertently
marks more than one alternative. This is more likely to occur when you ask individuals to circle their choice. Misalignments also occur when the respondent or
rater does not align an item and his or her selection for that item, a problem most
people have experienced when using a separate answer sheet. Once this happens,
all subsequent responses will be misaligned (Macey, 1996). This might be evident
on a test, where once an answer is out of position on the answer sheet all the
others will be misaligned as well. A teacher seeing a pattern of incorrect answers
will probably deduce that it is due to misalignment. However, this would not be as
obvious in instruments where individuals are providing subjective responses. To a
large extent these problems can be minimized during the design and formatting
stage of instrument construction. For example, respondents are less likely to misalign responses when items are separated by lines or shading and when item stems
and their corresponding response sets are presented on the same page, rather

c15.indd 354

7/10/07 12:00:38 AM

Managing the Data and Reporting the Results

355

FIGURE 15.1: EXAMPLES OF DATA ENTRY ERRORS BY
RESPONDENTS.
Marking Multiple Items
About when was this building first built?
1999 to 2000
1995 to 1998
1990 to 1994
1980 to 1989
1970 to 1979
Prior to 1970
Rewritten Items/Unsolicited Data
How many rooms do you have in this house?
5 rooms
1 room
2 rooms
6 rooms
3 rooms
7 rooms
4 rooms
8 or more rooms
Which best describes this building?
A mobile home
A one-family attached house
A one-family detached house
A building with 2 apartments
A building with 3 or 4 apartments
A building with 5 to 9 apartments
A building with 10 or more apartments
Misalignment
Apartments in my neighborhood are affordable (circle one):
Strongly Disagree Agree Strongly
Disagree
Agree
Scratch-out
Write in today’s date and then fill in the corresponding ovals.
Day
Month
Year
0
3
1
0
0
8
1
2
3
4
5
6
7
8
9
0
Source: These items, but not the errors shown, are taken from the U.S. Census,
2000.

c15.indd 355

7/10/07 12:00:39 AM

356

Designing and Constructing Instruments for Social Research and Evaluation

than having the answers on a separate sheet. You can minimize the circling of
more than one response alternative by providing sufficient space between the
choices.
A related problem is unsolicited data—respondents or raters may cross out your
item stem and rewrite it to suit their perceptions, or they may write in a response
alternative that you have not provided. We discussed this in relation to Instrument
6.A, where the yes or no options did not fit some of the checklist items, so raters
wrote in NA, not sure, or ? Even unsolicited responses, however, may have value.
In the case of Instrument 6.A, unsolicited data pointed out flaws in the checklist design that should be corrected. Or consider an organization survey where
unsolicited responses reveal problems that might not have surfaced otherwise
given the range of questions. Ignoring these responses would result in the loss of
meaningful information. One way to handle this situation is to provide a code for
unsolicited data when you design the database. For example, if you were using a
5-point response scale ranging from unsure to definitely, you could code unsolicited
responses as 7 and missing data as 9.
Create a Process to Safeguard the Data
After you have administered your instrument and cleaned, checked, and
entered the data, you have a responsibility to safeguard the data. Security measures
should ensure that only authorized individuals have access to the data and
that backup copies exist in case you lose the original data, in a hard drive crash,
for example. Your first step should be to ensure that your computer or the
database, or both, are password protected. This will limit unauthorized access
to all but the most dedicated of computer hackers. Encryption provides another
layer of security; encrypted files are transformed into random codes nearly
impossible for hackers to break. In addition to encrypting document files, it is also
possible to encrypt computer hardware, including the hard drive, and portable
media such as a DVD or memory card. Encryption software can be purchased,
and some basic versions of encryption software are available free of charge on
the Internet.
You should also plan to back up your database file—on removable media
(CD, DVD, or flash drive) if the original file is on your computer hard drive, or
on your hard drive if the original file is stored on a network server. The advantage of saving data on a network server is that the files are routinely backed up.
However, even with electronic safeguards in place, network servers are often the
target of hackers. The bottom line when it comes to security is to have two or
more copies of your documentation, including data files, on different platforms:
hard drive, removable media, and if possible a server. Additionally, files should

c15.indd 356

7/10/07 12:00:39 AM

Managing the Data and Reporting the Results

357

be encrypted to reduce the possibility that data will be inadvertently shared or
even stolen.

Who Owns and Has Access to the Instrument and the Data?
A painting is an original artwork. As long as the artist maintains possession, it
belongs solely to him or her; once it has been sold the artist loses all rights to his
or her creation and cannot profit from its resale. If you have been the driving
force in creating an instrument—the sole developer, perhaps, or the leader of
a team of developers—you are probably assuming that the instrument (and the
data it produces), like the artist’s painting, belongs to you or your team. However,
depending on the circumstances under which it was created, this may not be the
case. Issues surrounding ownership of an instrument are complex and dependent
on a number of factors. We can provide background information; however, you
will still need to consider your unique situation.
If you develop an instrument as part of your job duties as an employee of a
federal agency, that instrument belongs to the U.S. government. This is typically
true for state and local government agencies as well, but rules do vary from state
to state and municipality to municipality and must always be checked. When the
government agency owns the instrument, your work may be placed in the public
domain and shared openly, and no one is required to get your permission to copy
and use it. Agencies may do this to meet various objectives, including supporting
research and evaluation in the study area and encouraging other researchers to
assess instrument reliability and validity.
Conversely, government agencies may limit access to work you have done
for them; in which case people who wish to view the study and its instrument
must make a request under the Freedom of Information Act. If you develop
an instrument using government funding, such as a grant, ownership rights to the
instrument and the data will typically be articulated in the agreement between
you and the government agency.
If you develop an instrument for your employer, then under current copyright law it will probably be considered a work product and as such it belongs to
the organization. Like government agencies, some nonprofit organizations place
instruments in the public domain to support research and information dissemination. They may also be required to place an instrument in the public domain
when government funds were used to support its development.
Faculty of colleges and universities are encouraged to engage in research
as part of their academic development, and publications generated from these
activities are often considered during the process of granting tenure. Therefore,

c15.indd 357

7/10/07 12:00:39 AM

358

Designing and Constructing Instruments for Social Research and Evaluation

faculty have traditionally assumed that because a survey instrument is a
product of their original research, it belongs to them and not to the institution.
The American Association of University Professors has taken the following
position: “It is the prevailing academic practice to treat the faculty member as
the copyright owner of works that are created independently and at the faculty
member’s own initiative for traditional academic purposes” (Springer, 2004).
In recent years, however, universities have profited by retaining ownership of
faculty work products. In particular, inventions and discoveries arising from faculty research, such as medications and vaccines, can become a source of revenue
for the university. For that reason faculty should learn their institution’s policy
regarding ownership prior to engaging in research. The picture can become even
more cloudy when the research depends on external funding, such as a government agency or private foundation grant, in which case faculty need to develop
an agreement that specifies who owns what. Some faculty have established private corporations to manage outside funding, instead of channeling these funds
through the university. In this way they can retain certain rights over their work
product that they might otherwise have to relinquish to the university.
The following excerpt from the Copyright Policy of the University of Virginia
(2004, pp. 1–2), with which the authors are affiliated, shows how one institution of
higher learning views ownership of employee work. However, because standards
do vary, it is important to learn and comply with the policies and procedures of
your particular institution.
Work-for-Hire Rule: The “work-for-hire” rule, defined by the Copyright
Act, provides that when an employee produces copyrightable work within the
scope of employment, the copyright to that work belongs to the employer and
not the author.
Employee Ownership: The employee owns the rights to any work created
at his or her own initiative and outside the scope, time, and place of employment.
University Ownership: By operation of the copyright law, the University
owns all rights, title and interest in copyrightable works created by
University employees while acting within the scope of their employment.
The University cedes copyright ownership to the author(s) of scholarly and
academic works (such as journal articles, books and papers) created by
academic and research faculty who use generally available University resources.
However, the University asserts its right of copyright ownership if significant
resources (including sponsor-provided funds) are used in creation of such works
and (a) the work generates royalty payments; or (b) the work is of commercial
value that can be realized by University marketing efforts.

c15.indd 358

7/10/07 12:00:40 AM