|
Items in cart [1] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 215
APPENDIX
D
Laboratory Methods and
Document Redesign
Leda Kydoniefs and Marie van Melis-Wright
Office of Survey Methods Research
Bureau of Labor Statistics
BACKGROUND
In response to the 1983 Advanced Seminar on Cognitive Aspects of
Survey Methodology (CASM), interdisciplinary survey research centers and
laboratories have been established at a number of governmental organiza-
tions (Jabine et al., 1984~. In keeping with the CASM mission, such centers
actively promote collaboration among behavioral scientists from various
fields, statisticians, and survey researchers in order to improve data quality
and the data collection process (Dippo and Norwood, 1992~.
One such research center is the Behavioral Science Research Center
(BSRC) at the Bureau of Labor Statistics (BLS). Founded in 1988, its
professional staff includes psychologists from various specialties (e.g., so-
cial, cognitive, educational psychology, artificial intelligence), statisticians,
sociologists, a psychological anthropologist, and research assistants. Cur-
rently, primary areas of investigation include questionnaire development
and design, form redesign, interviewer training, interviewer-participant in-
teractions, interview administration methodologies, and basic research in
the areas of memory, proxy reporting, confidentiality, and privacy.
Since 1989, BSRC has undertaken two major form redesign studies: an
effort sponsored by the Internal Revenue Service (IRS) to redesign selected
tax forms and the redesign of the BLS Survey of Occupational Illnesses and
Injuries. Among the goals of the redesign efforts are to reduce respondent
burden, improve data quality, reduce cost and processing time, and develop
formats suited for use with different technologies (e.g., optical scanners,
administration by computer).
215
OCR for page 216
216
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
A distinct feature of the BLS form-redesign project is the application of
behavioral science methods to increase understanding of the cognitive, so-
cial, and organizational factors in the information-gathering process. Re-
searchers have many data collection methods from which to choose, for
example, expert analysis, interactional behavior coding, laboratory and field
observational techniques, focus groups, vignette classifications, rating tech-
niques, free and dimensional sorting, ethnographic and cognitive interviews,
and concurrent and retrospective think-alouds (Forsythe and Lessler, 1990~.
This paper illustrates the utility of some of these methods using examples
from the two redesign projects mentioned above.
IRS SCHEDULE C STUDY
Schedule C (Profit or Loss from Business) is an attachment to IRS
Form 1040. It is used by taxpayers who classify themselves as "sole
proprietorships." In 1987, taxpayers filed 14.5 million Schedule C forn;~s.
The data collected from Schedule C are used in aggregate form by the IRS
and other governmental agencies to, for example, help determine estimates
of personal income at the local level.
One of the goals of the redesign effort was to simplify the taxpayer's
task, which included locating the description of his or her business or pro-
fessional activity in the listing on the back of the form, finding the corre-
sponding 4-digit code, and writing the description and the 4-digit code on
the front of the form. To meet this goal, an interdisciplinary research team
was created, which consisted of IRS representatives and behavioral scien-
tists from the American Institutes for Research (AIR) and the BSRC labora-
tory. The research team used a variety of behavioral science methods to
obtain different perspectives on taxpayers' problems with the form and the
potential solutions. The methods they used included in-depth telephone
interviews with IRS editors of Schedule C, observations and in-depth debriefings
of taxpayers, concurrent and retrospective think-alouds with taxpayers, and
an experiment evaluating taxpayers' performance under the old and revised
versions of Schedule C.
Rather than focus only on the taxpayer, the research team decided to
collect input from other users of Schedule C, specifically IRS editors of
Schedule C and primary and secondary data users (e.g., Statistics of Income
Division, IRS; and Bureau of the Census and Bureau of Economic Analysis,
U.S. Department of Commerce). These groups regularly handle the com-
pleted forms, analyze or disseminate the data, or serve as gatekeepers and
trackers/monitors of the flow of forms. Thus, they are in an excellent
position to identify typical errors made by the taxpayer.
The BSRC team conducted a number of in-depth, semistructured tele-
phone interviews with a subsample of editors of Schedule C from among
OCR for page 217
APPENDIX D
217
the 26 national IRS Service Centers; the AIR team conducted observations
and used concurrent think-alouds with a small sample of taxpayers; and IRS
representatives used an open-ended written questionnaire to interview end
users from the Census Bureau and the Bureau of Economic Analysis. All
the methods combined yielded a wealth of qualitative and quantitative data
pinpointing problems and errors, including lack of or unclear instructions,
errors occurring during transfer of information from one side of the form to
the other, small print and cluttered physical layout, missing or hard-to-
understand principle business and professional code descriptions, use of
invalid or outdated codes, and a tendency of taxpayers to favor the "Other"
or "Unable to Classify" categories over more specific descriptions. Before
making revisions, the research team also collected information from IRS
representatives to make sure the revisions would not interfere with impor-
tant organizational constraints (e.g., policies with regard to mandated state-
ments, paper size, number of pages, format of instructions, and so on).
Based on the information gathered during the first stage of the research,
the AIR and IRS focus of the revisions crystallized into:
alphabetizing the main- and subcategories of the principal business
codes on the back of Schedule C,
· increasing the size of the print,
improving the clarity of and completeness of instructions printed on
the form,
moving the four-digit codes originally located to the left of the cat-
egory description to the right, and
removing visual "clutter."
The rationale for these changes was to facilitate searching, reduce scanning
time and errors, and facilitate the taxpayer's task.
The final phase of the project involved two experiments. The first
compared the existing Schedule C with a revised version on which the 4-
digit code was listed to the right of the description of the activity. In
experiment 2, three revised versions were compared. In both experiments,
subjects in each condition were given a list of 48 scenarios (business or
professional activities) and asked to complete a Schedule C for each of
them. The majority of scenarios were descriptions of activities taken di-
rectly from the current form, synonyms of descriptions listed on the form,
and ambiguous descriptions. The last group were descriptions for which
several reasonable responses, including "Other" and "Unable to Classify,"
were options each subject might consider. Outcome measures included
accuracy, response time, and a measure of preference by taxpayers for the
various versions. In addition, behavioral observations, concurrent and ret-
rospective think-alouds, and in-depth debriefings were conducted to investi-
gate taxpayers' rationale for their performance.
OCR for page 218
218
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
The scope of this paper does not allow for an in-depth discussion of the
results. Briefly, however, the findings from experiment 1 did not reveal any
significant differences. In experiment 2, alphabetization and format changes
yielded higher accuracy and shorter completion times than for the old ver-
sion. Findings also showed that respondents did not read the directions,
even when they were printed directly above the activity codes, and moving
the 4-di~it codes from the left to the right of the description caused more,
Not surprisingly, the accuracy levels for the various
scenarios varied—from 85 percent correct for exact descriptions, 78 percent
for synonyms, and 42 percent for ambiguous descriptions. An in-depth
error analysis was conducted to determine the type of errors made by the
taxpayer.
On the basis of the results of the experiments, a set of recommendations
for the redesign of Schedule C codes was developed, as well as for large-
scale testing on a representative sample of taxpayers.
Feedback from key participants in the research was very favorable re-
garding the methods and techniques used. In addition to the information
necessary for the development of revised forms, the close collaboration
between key players resulted in a firm commitment to the task at hand and
future form development.
-
not fewer, errors."
SURVEY OF OCCUPATIONAL ILLNESSES AND INJURIES
A major redesign of the Occupational Safety and Health Statistics (OSHS)
system, the nation's source of information on job-related illnesses and inju-
ries, was begun in 1987. The OSHS Survey of Occupational Illnesses and
Injuries is a mandatory survey and is conducted by mail. The redesign
effort included a review of all aspects of the survey:
forms design, collection, and processing.
. .
content, sampling,
(, , , ~ ~
The forms redesign portion of the OSHS project was begun in 1989 and
consisted of several pilot/feasibility studies investigating alternative survey
formats. Once the decision was made in 1991 to pursue the design of a
booklet format, the forms redesign efforts intensified. The goal was fourfold:
· Develop a booklet survey form with accompanying instructions that
was "user friendly."
.
Design a questionnaire format with a built-in flexibility that would
permit certain minor variations in item content. (These variations are re-
quired in order to maximize the survey's utility across the nation.)
1 This likely happened because the description of some activities took up two lines of text, in
which case, the 4-digit code was put at the end of the second line. However, as it turned out,
participants did not take that code, but rather selected the one below it (i.e., the code for the
next category).
OCR for page 219
APPENDIX D
219
· Test and evaluate old (i.e., rephrased) questionnaire content as well
as new survey items.
· Test and evaluate the redesigned OSHS survey that was developed.
To achieve these goals, a variety of behavioral science and test devel-
opment methods were used, each of which added a different perspective to
the knowledge base and forms redesign efforts. Focus groups, consisting of
volunteers from nearby establishments, were used to test the newly devel-
oped items and instructions for clarity, comprehension, and respondent bur-
den. Some respondents were videotaped in their establishments while com-
pleting the questionnaire, and retrospective think-aloud interviews were later
conducted with them; others were videotaped using the concurrent think-
aloud technique.2 These "think alouds" permitted the researchers to view
the respondent's interaction with the questionnaire and the respondent within
the respondent's working environment.
After the focus group data were analyzed and the appropriate changes
were made in the survey booklet, it was decided to evaluate the data collec-
tion and processing aspect of this newly developed survey format. The
questionnaire was mailed to approximately 1,500 respondents located in
four states. The data were then processed and analyzed within each state,
using the state's "standard operating procedures." This method revealed
certain deficiencies in the sampling portion of the survey document that
were not evident when each respondent was observed individually. Once
again, the test results were analyzed and the survey was revised accord-
ingly.
Once the focus groups, concurrent and retrospective think-alouds, and
test mailings were completed and all appropriate changes implemented, the
forms redesign process was approaching completion. To ensure that the
changes reflected "true" improvement and had not caused or created other
errors or difficulties, the survey was mailed to approximately 200 respon-
dents located in five states. The focus was again on evaluating the survey
contents (i.e., comprehension of items and response options, clarity of in-
structions, and respondent burden). Establishments were sampled in such a
way as to permit assessment of differences in data quality between large
and small establishments and rural and urban establishments. In addition to
the OSHS questionnaire, respondents were asked to answer a number of
questions designed to collect information on certain complex items (e.g.,
employment average, total hours worked by all employees in a year), time
required to complete the OSHS survey, and the "readability" of the survey
booklet.
As the completed forms were received, debriefing interviews were con-
ducted by telephone with approximately 50 percent of the respondents. The
2 For further information on this and other techniques, see Mullin et al (1991).
OCR for page 220
220
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
focus of the interviews was to investigate the respondents' feelings regard-
ing the new survey format and the additional items, achieve a better under-
standing of respondent burden, and obtain an estimate of the validity of the
data received. The data obtained were analyzed and the survey item content
and/or the design of the booklet was revised accordingly.
SUMMARY
The BSRC laboratory used behavioral science methods to investigate
errors and to test revisions of the IRS Schedule C and the BLS OSHS
survey. The methods used included focus groups, behavioral observations,
concurrent and retrospective think-alouds, in-depth debriefings, semistructured
telephone interviews, open-ended written interviews, and expert analysis.
All of the methods yielded important information and enhanced the overall
redesign efforts. While each has its own strengths and weaknesses in terms
of psychometric properties, cost and time requirements, and so on, they all
share the characteristic of providing information from perspectives that have
often been ignored in the past.
REFERENCES
Dippo, C. and J. Norwood
1992 A review of research at the Bureau of Labor Statistics. Pp. 271-290 in Questions
about Questions: Inquiries into the Cognitive Bases of Surveys. New York:
Russell Sage.
Forsythe, B. H. and J. T. Lessler
1990 Cognitive Laboratory Methods: A Taxonomy. Paper presented at the Interna-
tional Conference on Measurement Errors in Surveys, Tucson, Arizona.
Jabine, T. B., M. L. Straf, J. M. Tanur, and R. Tourangeau, eds.
1984 Cognitive Aspects of Survey Methodology: Building a Bridge Between Disci-
plines. Washington, D.C.: National Academy Press.
Mullin, P. A., L. A. Miller, M. van Melis-Wright, D. B. Stone
1991 Laboratory Research Methods for Improving Surveys. Paper presented at the
GAO Technical Conference, University of Maryland, College Park. Available
from the Behavioral Science Research Center, Bureau of Labor Statistics.
Representative terms from entire chapter:
oshs survey
