|
Items in cart [0] |
Questions? Call 888-624-8373 |
| ||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 25
CHAPTER 2
AFTER THE SEMINAR
Chapter ~ summarized the views expressed by the CASM project participants
at the St. Michaels and Baltimore meetings, with emphasis on their
suggestions for cro~s-disciplinary research by cognitive scientists and
survey researchers. This chapter describes some of the outcomes of the
project, including research plans and activities developed by partici-
pants after the St. Michaels meeting and dissemination of project results
through the publication or presentation of papers and other means.
Each of the first six sections of this chapter describes a research
program or activity initiated by CASM participants working as individuals
or in small groups. The first four sections describe plans for rather
substantial research efforts. The first section describes a multiyear
collaborative research program involving cognitive scientists and survey
researchers . The plan for this program, which is already under way, was
developed by CASM participants Sirken and Fuch~berg for the National
Center for Health Statistics (NCHS ~ . The program, described in the second
section was developed by CASM participants Tourangeau, Salter, D'Andrade,
and Bradburn, along with other cognitive scientists. The program, whose
objective is to study the cognitive underpinnings of the survey interview
process, is also funded and under way.
The third and fourth sections contain prospectuses for survey
collections of data that would be of considerable interest to cognitive
scientists. Converse and Schuman propose to investigate personal
interpretation of recent historical events for a sample of the U.S.
population; funding for this project is expected soon. Tulving and Press
present a proposal for a national memory inventory in which memory
capabilities and other cognitive abilities would be tested for a large
probability sample of the U.S. population; although the authors are not
now in a position to pursue their proposal, they welcome and would
cooperate with efforts by others to undertake the proposed research.
The fifth and sixth sections describe research done by students under
the direction of Loftus and Ross, two of the cognitive scientists who
participated in the CASM project.
The last section of. this chapter describes outreach activities:
steps taken by the CASM participants to share the ideas developed during
and after the seminar with others and to recruit new members of the
interdisciplinary network that has been established.
25
OCR for page 26
LABORATORY-BASED RESEARCH ON THE COGNITIVE ASPECTS
OF SURVEY METHODOLOGY
National Center for Health Statistics
(Monroe Sirken and Robert Fuchaberg)
The research project outlined in this plan uses the National Health
Interview Survey (NHIS) as a test bed for research and experimentation of
the sort discussed at the CASH seminar.
Purpose and Objectives
Questionnaire design and data collection procedures are among the weakest
links in the survey measurement process, and past efforts to improve the
quality of survey instruments and procedures have posed serious and
difficult methodological problems that are unlikely to be recolored by
traditional survey research methods. Therefore, it is essential to test
nontraditional modes for conducting research on survey methods. The
objective of this project is to investigate the cognitive laboratory as
the setting for conducting research on the cognitive aspects of. survey
methodology. It will tackle three of the most important questions to
emerge from CASH. Namely, under what conditions are laboratory methods
1 ikely to:
(~) produce results similar to or different from traditional field
methods?
succeed where traditional methods have failed?
enhance the results obtained by traditional methods?
Although survey researchers and cognitive scientists are both
concerned with the manner in which individuals handle information, their
approaches to the problem and the methods used to study the problem are
quite different, and there has been very little communication between
them. Surrey researchers are concerned about the survey measurement
process and use field experiments to test response effects in terms of
the wording, response categories, and orderings of. questions. They make
very little, it any, use of controlled laboratory experiments to
investigate the ma=er in which the respondents and interviewers process
the information presented by the survey instrument. The traditions
method of developing, testing, and evaluating survey instruments involves
sizeable field pretests and pilot studies of questionnaires that are
developed by survey statisticians and tested under Formals survey
conditions by trained interviewers. Cognitive scientists, on the other
hand, are concerned about the system individuals use in processing
information. Cognitive psychologists conduct controlled experiments in
a laboratory setting involving direct and intensive interaction with
relatively small samples of subjects to investigate the mental procedures
by which information is processed. A mayor objective of this project is
26
OCR for page 27
27
to contribute to the advancement of both disciplines, and to effect
communication between them, including collaboration in research studies.
The demonstration will conduct laboratory-based research on the
cognitive aspects of survey design using the combined methods of the
cognitive and statistical sciences. Cognitive knowledge and techniques
will be used to gain a better understanding of the effects of cognitive
factors in the survey measurement process. From these laboratory
findings statistical models will be developed for controlling survey
measurement errors.
Benefits
The laboratory is the ideal setting for conducting interdisciplinary
research in which the combined technologies of the cogniti Ire, social,
biological, and computer sciences can be applied in researching the
cognitive aspects of survey methodology. Participation of NCHS staff in
the interdisciplinary laboratory, as described later in this plan when
discussing collaborative arrangements, will help to bridge the gap that
currently exists between government agency survey methodologists and
university survey researchers and social scientists. There will be
potential benefits for all disciplines. The project will provide
additional methodologies for researching cognitive issues in surveys, new
phenomena to examine in basic research in the cognitive and related
sciences, and tested strategies for producing improvements in the methods
and statistics of federal statistical surveys in general, and in NHIS, in
particular.
A note of caution is in order about the potential benefits of this
project. It is not expected that the project will produce definitive
substantive findings with respect to any cognitive issues, although it
may provide important leads for subsequent research. The major emphasis
will be methodological rather than substantive. Even so, it is
recognized that the methodological findings obtained in thin or any
single study will not be conclusive until verified by other researchers
in subsequent trials.
Collaborative Arrangements
This demonstration project will be conducted in a collaborative mode.
Since NCHS has neither a cognitive research laboratory nor a staff of
cognitive scientists, it will make contractual arrangments with
universities to have the experiments conducted in their laboratories and
with their scientists. Not only will this arrangement be cost-effective
for this project, but it will, as noted earlier, have the major long-term
benefit of establishing closer research ties between the federal
statistical establishment and universities.
The NCHS and university laboratory staffs will collaborate in all
research aspects of this project, and in the preparation of research
reports, many of which will be suitable for publication in scholarly
OCR for page 28
28
journals. The NCHS will be primarily responsible for the -surrey and
statistical methods and the contractor for the cognitive methods.
Work Plan
Experiments will be conducted to tent the application of laboratory-based
methods for two broad types of questionnaire design problems:
development of survey instruments
(2) investigation of specific cognitive issues
The NHIS questionnaire will be used by the laboratory as the survey
instrument for both types of experiments. The supplement to the NHIS
questionnaire will be used to test the development of survey
instruments. Thin part of the questionnaire collects information on
specific health topics (child care, health promotion, prescribed
medicine, etc.) and changes annually. The specific cognitive issues will
be generic to surveys and could arise in either the NHIS supplement or
the core of the NHIS questionnaire. The latter collects basic
information about the nation's health (health status, utilization of
health services, eSc.) and undergoes virtually no change from year to
year.
The workplans for developing and pretesting a supplement to the NHIS
questionnaire and for conducting laboratory research on specific
cognitive issues relating to the NHIS questionnaire, respectively, are
discussed in the next two sections. These plans were developed within
the context of the collaborative mode in which the project will be
conducted. On the one hand, these plans are intended not to overly
restrict the Center 'a or the subcontractor' creativity as the research
progresses. This is a concession to the nature of this research project,
and also a major advantage of having the laboratory research conducted
outride NCHS. On the other hand, the plans were developed with the view
to pursuing certain objectives and producing particular products within
a specified time frame. This is a requirement necessary to ensure
project accountability, and also a major advantage of having the project
administered by NCHS.
Survey Instrument Development
The Center's schedule of activities to develop and tent the 1987 NHIS
supplement and a proposed schedule of the laboratory's activities are
presented below.
OCR for page 29
29
Date ~.~e~
1785 Develop analysis plan
6/85 Complete first draft of
supplement
10/85 Prepare pretest version of
supplement
12/85 Start 0~3 pretest clearance
3/86
4/86 Conduct field pretest
Participate in field pretest
6/86 Prepare pilot study version
of supplement
7/86 Start of OMB pilot study
clearance
10/86 Conduct pilot study
10/86 Design NHIS supplement
1/87 Start 1987 NHIS
Complete first draft of
supplement
Complete testing of first draft
of supplement
Complete tenting of pretest
version of the supplement
~ contingent on a three-
month extension of the
laboratory subcontract
In accordance with the Center's established timetable for
constructing its annual supplements to the NHIS questionnaire, the topic
for the 1987 NHIS supplement will be selected during 1984 and the
literature search will be completed by January ~ 985, exact' y two years
before the NHIS commences. During the two-year period, January
1985-1 987, the Center staff will be engaged in the tightly scheduled set
of activities as noted above. These activities exemplify the traditional
method of constructing survey instruments. The sine qua non of this
method is that the instruments are field tented under conditions that
simulate the actual survey conditions as closely as possible. This
approach is in sharp contrast to the proposed laboratory activities which
would be conducted under controlled laboratory conditions. The schedule
of laboratory activities is linked to the NHIS schedule of established
activities so as to maximize the laboratory 's potential contributions in
developing and testing the ~ 987 NHIS supplement, subject to the condition
that these activities should not interfere with nor Jeopardize the basic
integrity of the NHIS established testing practices.
This phase of the prospect will delineate the potential role of.
laboratory-based research in developing and testing survey instruments.
The project could result in the development of improved NHIS pretesting
protocols, including improved field pretesting methods for training and
debriefing interviewers, and ninnovative" laboratory-based methods for
conducting unstructured interviews and group interviews.
Developing the First Draft of the NHIS Supplement
The laboratory will devote the five-month period, January-May 1985, to
developing the first draft of the NHIS supplement. During this same
period the NCHS staff will be independently developing its own first
draft of the NHIS supplement.
OCR for page 30
So
In early January 1985, NCHS will provide the laboratory with the
items of information that will be collected in the 1987 NHIS supplement,
and the contractor will transform these items into survey questions and
procedures. Using cognitive techniques, such an protocol analysis, the
laboratory will investigate the manner in which respondents process the
required information, and on the basin of these findings, draft questions
and design its own first draft of the NHIS supplement.
Comparing the first drafts of the NHIS supplements that are developed
separately by the NCHS and the laboratory will indicate the extent to
which the laboratory method is a surrogate for the traditional NHIS
method and to what extent it produces different results. Merging what
appear to be the best features of both versions of the questionnaire and
comparing the combined result with the questionnaire that was developed
entirely by traditional methods will provide a basis for assessing the
enhancement value, if any, of developing questionnaires in the laboratory
as an adjunct to, or in place of, traditional developmental methods.
Testing the First Draft of the NHIS Supplement
During the six-month period from June 1985 to November 1985, the first
drafts of the NHIS supplement will be pretested in the laboratory.
Possibly three different versions will be laboratory tested: one that
was developed by the NHIS staff, another developed by the laboratory
staff, and possibly a third which incorporated what are Judged to be the
best features of the other two versions.
The laboratory will assess whether the drafts of the NHIS supplement
are eliciting the kinds of information they are supposed to elicit.
Laboratory testing will be performed on a variety of subjects who will be
selected because they are expected to experience different types of
cognitive problems with the questionnaire. The criteria for selecting
sub jects will depend somewhat on the topic covered by the THIS
supplement, but certainly they will reflect demographic, ether and
economic differences in the population.
The laboratory pretest findings will be discussed with NCH~ staff
during November 1985, so that they can be incorporated into an improved
draft of the questionnaire that the NHIS staff would be preparing to
accompany its request [or OMB clearance in December 1985 to conduct a
field pretest in 1986.
Testing the Field Pretest Draft of the NHIS Supplement
During the four-month period, December 1985 until the subcontract ends in
March 1986, the field pretest versions of the NHIS supplements will be
pretested in the laboratory and then independently field pretested during
April 1986. Pretesting identical drafts of the NHIS supplement by both
laboratory and field methods will make it possible to compare and
evaluate how well each method assessed whether the NHIS supplement was
doing what it is supposed to be doing and, if not, what revisions were
needed. Relative costs and turnaround times of conducting pretests by
each method would also be compared.
OCR for page 31
31
Specific Cognitive Issues
As noted earlier, the manner in which people handle information is of
common interest to cognitive and survey scientists, but objectives and
methods of the two sciences are quite different. Survey scientists
conduct field experiments to evaluate the quality of responses elicited
by survey instruments. Cognitive scientists, on the other hand,
establish generalizations about the mental systems people use for
processing information by conducting laboratory experiments. The mission
of this project is to demonstrate the enhancement value to both
scientific fields of conducting laboratory research on particular
cognitive issues that have been implicated by survey scientists as
adversely affecting the quality of survey responses.
Cross-fertilization of the two scientific fields is the keynote of
this pro ject. Cognitive issues that arise in surveys are representative
of. wider classed of cognitive phenomena that are being studied in
cognitive science, but under restricted and unnatural laboratory
conditions. Therefore, it is believed that bringing the survey cognitive
issues and the survey experience with there issues into the cognitive
laboratory will generate ideas for basic and applied research in
cognitive science. And feeding the laboratory research findings on these
cognitive issues back to survey scientists will, in turn, --simulate the
development of improved statistical models of survey measurement errors
and improved methods of constructing survey instruments. For cognitive
science, the ultimate benefit will be a better understanding of the way
people process information, and for survey science, it will be improved
control over the cognitive component of survey measurement.
Three well-known, but largely unresolved, survey problems are
presented as possible candidates for laboratory research. They are:
telescoping, conditioning, and the respondents ' perceptions of the
confidentiality of their responses. Each problem involves cognitive
issues that are poorly understood and, as will be explained later, seem
to present interesting material for laboratory research. For example,
for unknown reasons the effects of conditioning and telescoping are
asymmetric. The conditioning effects of ordering questions or response
categories are often more pronounced when ordered one, rather than
another, way. Similarly, the telescoping effects of allocating events
either to earlier or later periods than those in which they actually
occurred usually results in inaccurately allocating fewer events to lens
recent than to more recent periods.
Telescoping
Failure of respondents to recall events and to recall correctly when the
events occurred are mayor sources of error in the collection of survey
data. The errors associated with these two cognitive sources are often
confounded in surveys, which may help to explain why the classical
negative accelerated forgetting curve predicted by cognitive science does
not necessarily hold in surveys.
OCR for page 32
32
The tendency of survey respondents to allocate events either to
earlier or later periods than those in which the events actually occurred
is called telescoping. Typically, survey respondents report
retrospectively about events that occurred during a reference period,
which is a calendar period of specified length that precedes the
interview date. The telescoping phenomenon has been observed both for
reference periods that are bounded by prior interviews and for unbounded
reference periods.
The findings of survey research indicate that unbounded recall has a
net forward telescoping effect, that is, more events are shifted forward
in time and erroneously reported in the reference period than are shifted
backward and erroneously not reported in the reference period. In
addition, events that are correctly placed within the reference period
tend to be reported as having occurred more recently than they actually
did. The importance of the event, the length of the reference period,
and the characteristics of respondents all appear to have an effect on
the telescoping phenomenon. With bounded recall there appears to be
telescoping within the reference period itself, with the net forward
effect being greatest for the most recent part of the reference period.
The telescoping phenomenon is representative of a wider class of
cognitive phenomena involving temporal judgments. There is no doubt that
cognitive scientists appreciate that the process of making temporal
judgments is an important component of event memory; however, their
understanding of the event-dating process is based primarily on
laboratory experiments which involve neither naturally occurring events
nor long-term memory. Consequently, it is unclear how well existing
cognitive theory on temporal judgment applies to the real world of
personal events such as those respondents are asked to recall in
surveys. Apparently, the telescoping phenomenon, per se, has not been
investigated in the cognitive laboratory, and it is proposed that the
survey experience with this phenomenon may offer interest)" leads for
designing innovative laboratory experiments.
Conditioning
All scientific investigations are subject to the risk that the measuring
instruments will disturb the phenomenon under observation and thereby
affect the accuracy of its measurement. In this broad sense, the
conditioning concept in survey science is analogous to Heidelberg 's
uncertainty principle in physics, but without the latter ' ~ specificity.
Conditioning in survey research usually refers to the distorting
effect of the total survey measurement process on survey responses, but
in the narrower sense used here it refers more specifically to the
response effects of collecting an item or ret of items of information on
another item or set of information items. For example, it refers to the
response effects of adding one set of questions to another set of
questions, such as the effects of NHIS core questions on the questions in
the NHIS supplement or vice versa. It also refers to the effects of.
ordering a particular set of questions or response categories, or
reinterviewing the same respondents, an in panel and quality check
OCR for page 33
33
surveys. The narrower definition is adopted here because it makes the
survey conditioning phenomenon more amenable to laboratory
experimentation.
Although there are many examples of conditioning effects in THIS and
other scientific surveys, problems often arise unexpectedly since the
phenomenon is not well understood by survey scientists. For example, in
his CASH paper, Bradburn refers to a mysterious asymmetric effect of
question and response ordering. He notes that a different effect is
observed when questions are ordered in one way than when they are ordered
in another way. It is proposed that this curious effect of conditioning
in surveys may offer leads for designing critical laboratory experiments
on the conditioning effects of rotating the order in which the material
is presented.
Perceptions of Confidentiality
The response effects of asking for information about sensitive topics is
a major survey concern because (1) policy makers and other users of
health survey data often require this type of information, and (2)
respondents are usually reluctant to provide this information and the
quality of the information reported is often suspect. Examples of
sensitive topics are: illicit behavior such as drug use, drunk driving,
low-esteem behavior such as excessive drinking, overeating, and diseases
with social stigma such as cancer, venereal diseases, tuberculosi a,
mental illness, etc .
Although scientific surveys subscribe to a strict policy of
protecting the confidentiality of the reported information, assurances of
this policy are often insufficient to overcome the suspicions of
respondents that their responses may be disclosed in an identifiable form
to third parties or their reluctance to report sock ally undesirable
behavior to an interviewer. Survey scientists try to reassure
respondents by using data collection techniques that seek to preserve the
anonymity of the persons for whom sensitive information is reported in
household surveys. Although it seems obvious that the success of these
techniques would be greatly affected by the respondents' perceptions of
the confidentiality protection afforded by these techniques, their
perceptions have not been subjected to in-depth research and hence they
are largely unknown.
There are three survey techniques often used for preserving
respondent anonymity:
(1) Randomized response--a respondent is simultaneously presented
the sensitive question and another non-sensitive question, each of which
can be answered yes or no. He/she answers only one question and he/she
alone knows which one, because he/she selected the question to be
answered by a random process such as flipping a coin.
(2) Network sampling--the respondent serves as an informant for
other persons to whom he/she is linked by virtue of kinship, friendship,
or some other designated relationship, but who are otherwise
unidentified.
OCR for page 34
34
(3) Self-enumeration--the respondent writes the answers to the
sensitive questions on a blank sheet of paper that he/she seals in a
elf-addressed envelope and mails to the survey organization.
The problem of respondent compliance in surveys on sensitive topics
is representative of a wider class of cognitive phenomena in which people
are faced with the task and the rink of making decisions on the basis of
information that they may neither fully comprehend nor believe. Numerous
applications of the anonymity techniques in surveys on sensitive topics
have produced mixed results that traditional survey research methods have
been unable to explain satisfactorily. It is expected that laboratory
research on respondents' perceptions of these techniques under varying
conditions may improve the design of surveys on sensitive topics and may
lead to an improved understanding of the cognitive processes by which
people assess risks on the basis of incomplete information.
OCR for page 35
COGNITIVE PROCESSES IN SURVEY RESPONDING: PR W ECT SUMMARIES
Roger Tourangeau, Villiam Salter, Roy D'Andrade,
Norman Bradburn, and associates
Researchers at the National Opinion Research Center (NORC), Yale
University, and the University of Chicago have proposed three
interrelated research programs to carry out a series of studies on the
cognitive underpinnings of the survey interview process. All three
projects share a common framework, which is described here briefly. The
framework assumes that respondents in surveys proceed through three mayor
stages in answering survey questions: they interpret the question,
retrieve the relevant information, and formulate a response. It is our
shared belief that response effects in surveys can bent be understood by
examining these processes in detail. Since different types of questions
make different demands at each stage, we have also adopted a simple
scheme for classifying survey quest ~ ons. We distinguish three broad
classes: questions that elicit attitudes or opinions; questions that ask
about behaviors; and questions that concern the causes or reasons for
behavior. The framework thus suggests nine areas of. investigation
def ined by the three stages of survey responding and the three types of
questions.
The NORC research program, developed by Roger Tourangeau, Roy
D'Andrade, and Norman Bradburn, is entitled Recognitive Processed in
Survey Responding: Attitudes and Explanations. ~ It concerns two types
of survey questions--those concerning attitudes and reasons--and includes
studies on all three stages of survey responding--interpretation,
retrieval, and Judgment. The Yale program, developed by Robert Abelson,
is entitled "Cognitive Processed in Survey Responding: Multiple Schemas
and the Role of Affect. It deals with the same two classes of survey
questions as the NORC research program and offers a complementary
perspective on some of the same issues explored there. It dovetails with
the NORC work in other ways--it extends the analysis of attitudes in
terms of cognitive schemata and incorporates studies on the role of
affect in survey responses.
The University of Chicago program, developed by William Salter,
Steven Shevell, Lance Rips, and Norman Bradburn, in entitled Recognitive
Processes in Survey Responding: Time and Frequency Estimation.8 It
focuses on the remaining class of survey questions, those that concern
behavior. It includes studies on the retrieval and Judgment processes
and on how these processes interact when respondents must Judge the
timing or frequency of events.
All three research programs share the interdisciplinary perspective
of the Advanced Research Seminar on Cognitive Aspects of Surrey
Methodology. The research teams for each project include researchers who
have done cognitive research or survey research or work in both fields.
In addit ion, all three include a commitment to the replication and
extension of laboratory findings to the field setting. Each project
incorporates plans for split-ballot studies to be conducted within the
context of a national surrey.
35
OCR for page 60
60
References
Cralk, P. I. M.
1977 Age differences in human memory. In J.E. Birren and R.W.
Schaie, eds., i.
New York: Van Nostrand Reinhold.
Craik, F.I.M, and Levy, B.A.
1976 The concept of primary memory. In W.~. Estes, ea., Hand~ok
~ , Vol 4. Hillsdale,
N.~.: Lawrence Erlbaum Associates.
Kelley, H. P.
1964 Memory abilities: a factor analysis. PA o
~aDhs ~ 1.
Tulving, E.
1972 Epinodic and semantic memory. In E. Tul~ring and W.
Donaldson. eds.. A. New York: Academic
1 983
Press.
New York: Oxford Oni~rersity
Press.
Tulving, E., Schacter, D. L., and Stark, H. A.
1982 Priming effects in word-fragment completion are independent
of recognition memory.
~ ~ 8: 336-342.
Underwood, B.~., Boruch, R.F., and Mali, R.A.
1978 Composition of episodic memory. ~;_~
~ 107: 393-419.
.
Waugh, N. C. ~ and Norman, D. A.
1965 Primary memory. ~ Rim 72:89-104.
Wechaler, D.
1945 A standardized memory -scale for clinical use.
Psychos on ~ 9 : 87-95.
OCR for page 61
PROTOCOL JlNALYSIS OF RESPONSES TO SURVEY Rl3CILL QUESTIONS
Elizabeth Loflus
One idea that was received enthusiastically at the St. Michaels seminar
wan the suggestion for using protocol analysis to study how survey
respondents retrieve information from memory to respond to questions
about pant events. The first section of this item, prepared a few weeks
after the seminar, expands on this idea and presents some results from
five pilot interviews conducted to explore the potential utility of this
technique.
Subsequently, David Fathi, a student of the author, has conducted
research in this Dame area for an honors thesis. The second section of
this item describes some results from 23 protocol analyses of responses
to two questions; one on use of. health care facilities and one on
deposits to a credit account. One of the questions arising in this
research was how to relate different methods of retrieval to the validity
of responses. Since it proved difficult to obtain verification data for
the health and credit deposit questions, some subsequent work by Fathi
has related to questions asking students in an undergraduate psychology
course to recall the exact dates of examinations given in the course.
Experimental ProJect: Protocol Analysis
In many national surveys, respondents are asked to recall personal events
Prom their lives. For example, in the National Health Interview Survey,
respondents are asked, "During the past 12 months, about how many times
did (you) see or talk to a medical doctored In the National Crime
Surrey, respondents are asked, non the last six months, did anyone beat
you up, attack you, or hit you with something, such as a rock or bottle?.
Very little is known about the precise strategies for retrieving personal
information of thin sort.
One method for learning about cognitive strategies is through the use
of protocols (Ericsson and Simon, 1980~. In the protocol technique,
people are asked to think aloud as they answer specific questions. The
verbalizations produced are called protocols, and they can subsequently
be transcribed and analyzed. This method has an advantage over the
similar technique of asking people after the fact to describe how they
arrived at a particular answer or estimate. The ~atter-the-fact.
technique has the disadvantage that people often provide reasons or
rationalizations for their bobavior that are not the true reasons but
rather are strategies that subjects believe should have been
appropriate (Nisbett and Ross, 1980~.
To explore the feasibility of a protocol analysis approach to the
problem of how people retrieve personal experiences of the type required
on, say, the National Health Interview Survey, we asked five pilot
subjects to think aloud while answering specific questions. We first
gave subjects some practice questions so they could gain experience in
Verbalizing their thought processes. Those were questions such an, .~n
61
OCR for page 62
62
the last 12 months, have you eaten lobster?. Then we asked some
health-related questions and Rome crime victimization questions. For
example, we asked subjects, .In the last 12 months, how many times have
you gone to a doctor, or a dentist, or a hospital, or utilized any health
care specialist or facility?. and, sin the last 12 months, have you been
the victim of a crime?
Many specific questions can be answered by examining the protocols
produced by subjects. For example, one specific question Ad this: Do
people answer the health question by starting from the beginning of the
12-month period and moving toward the present (the past to present
approach), or do they start from the most recent event and move backward
(the present to past approach)? One might predict that respondents would
begin with the most recent events, since these might be more "available
in memory (Tversk~r and Rahneman, 1973~. While our results must be
considered preliminary, they indicate that, to the contrary, the past to
present approach is the favored one. For example, one female respondent
answered the health question by saying, Let's see . . . six . . . six
months ago ~ went to the dentist. Last month I went to the doctor. I
think that' s it . ~
It this tendency to pref er the past to present retrieval sequence for
the health question were to be documented in a full study, it would
suggest that people might be most efficient at retrieving information if
prompted to do so by cues that allowed them to start in the past and work
toward the present. Of course, this hypothesis would need to be
explicitly tested mince we know that simply because most people perform
acts in a particular way does not necessarily mean this is the most
efficient way to do so.
Another specific issue that could be addressed by analysis of the
protocols is the extent to which respondents produce new information when
asked further questions that relate to ones that were asked earlier. For
example, the response of the female quoted above indicated two health
related contacts. However, later this respondent was asked, nIn the last
2 months have you been to a dentistry Her answer:
Let's see . . . I had my teeth cleaned six months ago, and no . . .
and then I had them checked three months ago, and I had a tooth . . .
yeah, I had a toothache about Harch . . . yeah. So, yeah, I have. n
(Interview conducted in July 1983.) This protocol again indicates a
preference for the past-to-present retrieval sequence, but also indicates
the production of two additional dentist visits that were not provided
earlier to the more general question.
Although it is well suspected that additional questions will produce
additional instances, it is not known why. Protocols could shed light on
this issue. Furthermore, it is not known whether beginning an interview
with a general question (e.g., shave you been to a specialist? is the
optimal technique. It is possible that after having said, anon to the
general questions, subjects may be less likely to search memory in an
eSfort to answer the specific question than they might have had they not
been asked the general question to begin with. He simply do not know
whether this is the case. However, an examination of protocols given to
specie to quest ions that either are or are not preceded by general ones
would be more informative.
OCR for page 63
63
One interesting observation from the five pilot protocols is the
large number of instances in which people change their answer as they are
in the midst of speaking. For example, one female subject who was asked
the crone question answered: "No, not that I can think of, unless . . .
Oh, I had two dollars stolen at work, but that's it. n Another said:
"No, I haven't, that I can remember . . . Yes, ~ was--T was thinking
about my car, and I had come tapes stolen from my car, in Montlake, about
six months ago."
We could speculate that if subjects had been responding using a more
formal checklist technique in which they simply had to Ray eyed or "no"
that these two instances might never have been reported. Under the more
leisurely approach provided by the protocol technique, the instances
emerged from memory. One question that naturally comes to mind is
whether we can improve on current interviewing techniques to take
advantage of this possible discovery. For example, if respondents were
asked to think for a minute, and then answer the question, would we be
able to accomplish the same benefits within the context of the more
typical interviewing procedures?
In short, many interesting issues can be explored through the use of
protocols. Specific hypotheses can be tested concerning how personal
information is retrieved by people. Moreover, methods for improving the
interview process can be tested in this fashion.
Order of Retrieval in Free Recall of Autobiographical Material
Six subjects were asked, "In the last 12 months, how many times have you
gone to a doctor, or a dentist, or a hospital, or utilized any health
care specialist or facility? Three subjects were asked, "In the last 12
months, I'd like you to try and recall all the times that you deposited
money in your A La Card account., and for each time, try and give me the
date as accurately as possible, n or a slight variation on this question.
Seventeen subjects were asked both there questions, with the A La Card
question coming first. Subjects were instructed to Think out loud" an
they responded, and their remarks were taped and transcribed verbatim,
yielding 23 protocols in response to the health care question and 20 in
response to the A La Card question.
Sixteen of the 23 health care protocols and ~ of the 20 A La Card
protocols contained fewer than two instances of the behavior in question,
thus yielding no information about order of retrieval. For those
protocols containing 2 or more instances of the behavior in question, a
"+~ was assigned for each time a subject went from a temporally more
distant instance to one more recent, and a In for each tome the subject
moved from a more recent instance to one in the more distant past. This
was regarded as a rough way to quantify the degree to which subjects
tended spontaneously to retrieve these autobiographical memories in one
direction or another.
MA system under which costs of meals eaten by students are charged
against an account to which periodic deposits are made.
OCR for page 64
64
When all the + and - signs were counted up across all the protocols,
the results were an follows:
Health Care + 7
3
A La Card ~ 23
6
-
-
When the protocols were clas~ified according to whether the direction
of recall was consistently forward (i.e., all +'s), consistently backward
(all -'s), not clearly in one direction or another (both +'s and -'s), or
there was no information about direction of recall available (fewer than
2 instances of the behavior in question produced), the results were as
follows:
Health Care:
A La Card:
all +
all -
both ~ and -
no information
4
2
16
23
all +
all -
both + and - 3
no information ~
20
7
2
These results were taken as evidence that, at least in response to
these questions, subjects tend to retrieve autobiographical memories in
a predominantly past-to-present, or forward, direction.
References
Ericsson, E.A., and Simon, H.A.
1980 Verbal reports an data.
Nisbett, R., and Ross, L.
1980
~ud~-e~t. Englewood Cliffs, N.J.: Prentice-Hall.
T~ersky, A., and Rahneman, D.
1973 Availability: a heuristic for Judging frequency and
probability. 0~;~· ~10~ 5:207-232.
P~xrholu~al B~x 87:215-251.
OCR for page 65
TROUGHS AND RESEARCH ON ESTIMATES "OUT PAST IND FUTURE BEHAVIOR
Lee Ross
The Strategy and Tactics of Survey Methodology
as They Pertain to Determining Past Actions and Out comes
basic issue raised at the St. Michaels coherence concerned the special
status of surveys, such as the National Hearth Interview Survey, which
focuses primarily on relatively ob~t~ past actions and outcomes
rather than opinions, preferences, fears, intentions, or other largely
~ b ~ ive responses. A great deal of the traditional science and art of
the survey methodologist has focused on the problem of potential
instrument or interviewer bias. Many strategic decisions about
instructions, wording of items, and the role of the interviewer are
designed to minimize such potential for bias by minimizing the role of
both the instrument and interviewer in defining terms, suggesting
response strategies, and especially in providing feedback about the
responses themselves. While such precautions may be entirely appropriate
in the context of political surveying or other attempts to ascertain
attitudes, beliefs, or other subjective states of the respondent, they
may be less necessary when the responses in question deal with specific
concrete past actions and outcomes by the respondent. Furthermore, in
attempting to determine the respondent's past actions and outcomes
through measurement of his or her recollections or estimates, a rather
different set of potential sources of error or bias come into play--i.e.,
the types of factors with which cognitive psychologists have long been
concerned in their study of human memory and Judgment.
Cognitive psychologists have identified many factors that impair
performance or introduce error in recall or Judgment and, perhaps even
more pertinent to present concerns, they have identified factors or
strategies that lead to improvement in performance. If such
psychologists were confronted with the problem of designing instruments
and interviewer protocols for facilitating accurate recur and estimation
of past actions and outcomes, ~ suspect that they would worry relatively
little about traditional instrument or interviewer effects and a great
deal about how to help the respondent remember the events in question or
estimate the relevant magnitudes or frequencies associated with such
events. They would worry about the impact of the respondents' general
theories, scbemas, or expectations on their recall or estimates. They
would worry about the effect of the respondents' concerns with
self-presentation (and perhaps selt-perception and evaluation as well).
Most importantly, perhaps, they would design instruments and procedures
to overcome such obstacles through trial and error testing which measured
accuracy of recall or esti~^tion--i.~., compared recollections and
estimations to direct measures of the actions and outcomes in question.
~ could offer many Radicals suggestions about techniques that could
enhance accuracy of recollections or estimates. One might make heavy use
of models--i.e., letting the respondent see someone doing a good Job of
being systematic and complete, or using specific memory aids or mnemonics
65
OCR for page 66
66
in recalling their medical history, consumer behavior, crime
victimization experiences, or whatever topic is the focus of the survey.
One might encourage the interviewer to offer suggestions about how to
remember or estimate the responses or outcomes in question (and train
that interviewer in how to prompt such recalls One might explicitly
warn the respondent about common biases or errors. One might encourage
the respondent to describe events in his or her own words before
attempting to answer specific questionnaire items. Perhaps the most
extreme possibility would be furnishing the respondent with informative
nanchors~--e. g., mean Judgments, typical responses, or ~base-rates. for
people in general or people who are like them in terms of pertinent
demographic charact ecliptics .
Research on Apart ~ and futures Behavioral Estimates
for Self and Other
One preliminary piece of research has been undertaken in my laboratory
that was prompted by the foregoing comments, even though its direct
relevance may not be immediately apparent. It was our thesis that
recollection of past events, at leant in cases where Episodic recalls is
likely to be imperfect at best, or even nonexistent, is closely akin to
other related Judgment tacks. Specifically, we sought to compare
estimates of specific past performances to parallel predictions about
future response and to compare estimates and predictions about one's .°XP
responses with parallel estimates and predictions about the responses of
a peer whom one knows rather well and has ample opportunity to observe on
a day-to-day basis.
Ultimately, our concern will be with relative accuracy, and with the
relationship between accuracy and confidence, in these four different
domains (i.e., set[/other x past/future). Pursuing this concern,
however, will demand that we choose responses for which we can
independently assess actual behavior to which the relevant estimates or
predictions can be compared. This will pose significant methodological
hurdles and tax our ingenuity. Undoubtedly, it will also restrict our
domain of. inquiry to responses that are normally recorded ~ e. g., checks
written, purchases made, books checked out of the library, time logged on
computer, long distance telephone calls ~ or at least recordable by an
observer (behavior in contrived experimental nettings, perhaps television
watching, study the, class attendance, etc.~--domains which may or may
not be representative of those that figure in survey concerns and domains
and which may or may not be typical in terms of difficulty of. recall or
estimation.
For now, we have chosen to ignore accuracy per se, and to focus on
confidence intervals--i. e., to compare subjects' certainty (or, to be
more precise, the range of their uncertainty) about the frequency and
magnitude of their own past behaviors or outcomes with their certainty
about frequencies for parallel future responses. Furthermore, we compare
confidence intermurals regarding self-estimates and predictions with
parallel estimates and predictions for other people. In other words, we
are comparing estimates or recollections about one's past behavior, about
OCR for page 67
67
which one might have a basis for considerable certainty, with three types
of estimate that one would expect to be highly uncertain and heavily
Theory basely rather than data based.
In our single research effort, Stanford students were asked to make
estimates about a variety of pant and future responses either for
themselves (self-estimate) or for their roommates (roommate estimates).
Items included number of checks written (or to be written) in a 30-day
period, money spent in restaurants, hours spent watching television,
number of long distance telephone calls, and so forth. For each item
they made a best guess and then bracketed that best guess with an upper
and lower confidence limit. In [act, they furnished two different
confidence limits: 50 percent limits (such that they thought the
probability of the upper limit being too low was .25 (or 25 percent) and
that the probability of the lower limit being too high was also .25 (or
25 percent) and 80 percent limits ~ such that the probabilities of the
limits being too low or too high were each NO percent). They also rated
the ease or difficulty of making each estimate, and indicated how
"surprisedn they would be if the "right answers was not contained within
the confidence limits that had been specified (although these items shall
not be dealt with in thin brief report). The research design made use of
both within- and between-~ubJect comparisons--with self versus other a
between-subject factor and past versus future a within-subJect factor
(order of past vs. future was counterbalanced , as was the order of
specific items).
The results of this pilot effort (see Table 1) can be summarized
succinctly. Confidence limits for predictions of the future were only
TABLE ~ Relative Width of 50 Percent Confidence Intervals
for Estimates of Past Behavior and Predictions of Future
Behavior
Estimate of Prediction of
Past Behavior Future Behavior Combined
Judgment
about self ~ 00 ~ 3 ~ ~ ~ 5
Judgment about
roommate 1 1 1 124 117
Combined ~ 05 ~ 28
Note: All intervals were transformed to reflect magnitude
relative to interval for past behavior of self. Means
reported summarize results for 1-2 behavioral estimates and
predictions. A total of 25 subjects offered confidence
intervals for self only (both pant and future behavior) and
an equal number offered confidence intervals for roommate
only (both past and future behavior).
OCR for page 68
68
modestly wider than confidence limits for estimates about the past
regardiens of whether it was the self or ones roommate who was the
target. Furthermore, confidence internals about one's roommate were no
wider, overall, than confidence levels about oneself; for estimates about
the past, the confidence interval was slightly wider for roommate than
self; for predictions about the future, the reverse wan true. None of
theme main effects or interaction effec~c^, moreover, appear to be
statistically significant (although this may change somewhat when data
from a second cohort of subjects in added to our analysis. It is
particularly noteworthy that the confidence intervals for ostensively
data-based estimates about one's pant behavior were only marginally
narrower than for the ostensively theory-based estimates about the future
responses of one's roommates. Such data certainly prompt one to wonder
exactly how data-based estimates about ones past really are! They also
encourage the type of speculation offered earlier--i. e., that the
accuracy of any inferences we might want to make about that behavior
would be facilitated by procedures that facilitated recall or encouraged
more accurate estimation strategies on the part of the respondent.
Finally, the data comparing self-est imates and roommate estimates are
interesting in their own right, beyond any relevance to concerns of
survey methodology. People apparently believe that they can mare as
accurate and confident estimates about other people' ~ responses as they
can about their own, particularly when it is future rather than past
responses that are the subject of such estimates. It is obviously
tempting to find out whether such relative immodesty regarding one's
social predictions, and modesty regarding self-prediction, is ''ustified
~ Just as it is tempting to find out whether one's ability to predict the
future is really as good, and one's estimates about the pant are really
as bad, as suggested by the confidence limits offered in our study). The
need for follow-up research is e~rident--research in which accuracy, and
therefore calibration of such confidence intervals, can be assessed
directly.
OCR for page 69
69
OUTREACH ACTIVITIES
In the months that have elapsed since the January 1984 meeting in
Baltimore, several of the seminar participants have been active in
publicizing the activities and outputs of CASH and in seeking to
encourage others to work in this exciting cross-disciplinary field. The
principal method of outreach has been the presentation and publication of
papers, but other means are also being used.
On January 26, 1984, Norman Bradburn attended a seminar, Problems of
Measuring Behavior,. sponsored by the Economic and Social Research
Council in London, England, and presented a paper, Potential
contributions of cognitive sciences to survey questionnaire design. ~ In
February Bradburn made presentations on the same subject to three
organizations in West Germany: the Zentrum fur Umtragen und Methodische
Analyze (Z0M4, in Mannheim, the Institut fur Demo-~kopie in Allenabach,
and the Max Planck Society in Munich. In July 1984 Roger Tourangeau
attended a ZUMA seminar entitled Asocial Information Processing and
Surrey Methodology and presented a paper, "Question order and context
effects. n
At the 39th Annual Conference of. the American Association for Public
Opinion Research held on May 17-20, 1984, at Lake Lawn Lodge, Dele~ran,
Wisconsin, Judith Tanur chaired a session on contributions of cognitive
psychology to survey research that included a paper entitled Ran
information processing approach to recall in surveys. by Norman Bradburn
and a paper entitled Attitude measurement: a cognitive perspectives by
Roger Tourangeau. The discussant was Elizabeth Martin of the Bureau of
Social Science Research.
At the annual meeting of the American Statistical Association held on
August ~ 3- ~ 6, ~ 984, in Philadelphia, Pennsylvania, a session on cognitive
aspects of survey methodology was sponsored by the Section on Survey
Research Methods and cosponsored by the Social Statistics and Statistical
Education Sect ions . Organized and chaired by Judith Tanur, the session
included a paper by Roger Tourangeau entitled Interchanges between
cognitive science and survey methodology and a paper authored by David
C. Fathi, Jonathan W. Schooler, and Elizabeth F. Lortus, presented by
Elizabeth Loftus, entitled Moving survey problems into the cognitive
psychology laboratory. The discussants were two of the CASH guests at
St. Michael~, Dr. Jacob J. Feldman of the National Center for Health
Statistics and Professor Phillip J. Stone of. Harvard University.
A paper entitled Recognitive psychology meets the national survey. by
Elizabeth F. Lofts, Stephen E. Fienberg, and Judith M. Tanur has been
prepared in response to an invitation from the ~rtr~ and
is expected to appear ~ n December 1984.
The editor of the ~ _~ has invited members
of the CASH group to prepare papers. A paper entitled nCognitive aspects
of health surveys for public information and policy. is being prepared by
Stephen E. Fienberg, Elizabeth F. Loftus, and Judith M. Tanur. A
companion piece being prepared by Monroe Sirkin and Judith Lessler stems
from their laboratory-based research project at the National Center for
Health Statistics. The editor of the ~v in inviting several
OCR for page 70
JO
Health Statistics. The editor of the ~a~y is inviting several
discussants for these papers.
In response to a preliminary proposal presented at its board meeting
in June 1984, the Social Science Research Council (SSRC) is organizing a
working group on cognition and survey research in order to prepare a
detailed plan for the activities of a possible SSRC committee that would
bear thesame name. Cochaired by Robert Abelson and Judith Tanur, the
working group includes Roy D'Andrade, Stephen Fienberg, Robert Groves,
Robin Hogarth, Don Kinder, and Elizabeth LofLus. The staff person
responsible for this effort at SSRC is Robert Pearson, a guest at CASM's
Balt imore meet ing.
Theme are the outreach activities that the editors have been able to
identify as this report goes to press; there may well be others that have
escaped their attention. It seems reasonable to predict, on the basis of.
this record, that we can look forward to a continuing round of relevant
reports and discussions an further results emerge from the
cross-disciplinary research programs and activities genera~ced by the CASH
project .
Representative terms from entire chapter:
cognitive issues
