|
Items in cart [4] |
Questions? Call 888-624-8373 |
PAPERBACK Free Resources |
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
| ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
PART I.
Issues of Theory and Methodology
OCR for page 2
Human Performance Research: An Overview
Monica J. Harris and Robert Rosenthal
Harvard University
OCR for page 3
Table of Contents
Interpersonal Expectancy Effects eeeeeeeeee.~ee.~1
Definition. eeeeeee.~eeeeeeeeeeeeaeee.eeeeeeee.eeeee.eeeeeeee.~.e.1
Evidence for Expectancy Effects eee.eee.~eeeeee..eee.~.eeaeeeae.~.e2
Methodological Implications of Expectancy Effects ee.~.ee.~.e.~3
Mediation of Interpersonal Expectancy Effects e.~.eeeee.~.eeeee5
Ran: i ~ I .C$:tl-.C ~
The Four-Factor Theory e.~eeaee.~eeeee.~ee..eee.~.
Meta-Analysis of Expectancy Mediation eeee..eeeee
Human Performance Technologies and Expectancy Effects ee e.
Research on Accelerated Learning...
Neurolinguistic Programming... ee
Imagery and Mental Practiced...
Biofeedback.........
Parapsychology eeeeeeeeeseeeeae.~41
Situational Taxonomy of Human Performance Technologies 52
Sugges t ions for Future Research . e.~. 57
Expectancy Con tro 1 Des i gns . . . . . e.~.e. 57
Central ~ for Expectancy Effects .. ~ ~ ......
Expectancies ant the Enhancement of Human Performance....
Conclusion................................
Ref erence se~~eea~~ ~~e
^~.ea.~62
ooe.65
OCR for page 4
4
interpretation of results: Increasing ransom noise merely makes it more
difficult to obtain significant results, but increasing systematic bias can
result in completely erroneous cone fusions .
Experimenter expectancy effects are a potential source of problems for
any research area, but they may be especially influential in more recent
research areas lacking well-established findings. This is because the first
studies on a given treatment or technique are typically carries out by
creators or proponents of the technique who tend to hold very positive
expectations for the efficacy of the technique. It is not until later that the
technique may be investigated by more impartial or skeptical researchers, who
may be less prone to expectancy effects operating to favor the technique. Many
of the human performance technologies of interest in the present paper are
relatively recent innovations, and thus may be especially susceptible to
expectancy effects.
In principle, expectancy effects could be investigated by introducing
expectations as a manipulation in addition to the independent variable of
theoretical interest. This method, which will be described in detail later,
allows the direct comparison of the magnitudes of the effects due to the
phenomenon and effects due to expectancies. Another approach, perhaps even
richer theoretically, is to examine directly the processes underlying
expectancy effects as they occur in various areas. In same areas, such as the
area of teacher expectancy effects, a considerable amount of research has been
conducted in this manner, and there is now a good general understanding of
what variables are important in mediating teacher expectancies. However, in
other areas, such as the human technologies of interest here, this background
research is lacking. The best that can be done in such cases is: (a) to
OCR for page 5
Interpersonal Expectancy Effects and Human Performance Research
Monica J. Harris and Robert Rosenthal
Humans have long tried to surmount their traditional limitations and to
increase their performance. Long ago such efforts were aided by social
institutions of religion, proto-medicine, and magic. More recently, Such
efforts have been aides by social institutions of science ant its associated
technologies. Systematic programs have been developed with such aims as
improving communication, accelerating learning, ant increasing conscious
control over physiological processes. Because the promise of enhancing human
performance is so appealing, considerable resources, both in terms of time ant
money, are being invested in these programs. The time has come to step back
ant evaluate human performance technologies so that resources may be directed
more appropriately. The purpose of this paper is to aid in such an evaluation.
We will focus specifically on the possible influence of interpersonal
expectancy effects on several human performance technologies. The paper
advances in three steps: First, we describe the methodological, theoretical,
ant empirical issues relevant to the study of expectancy effects, including
how expectancy effects are mediated. Second, we describe each of several types
of human performance research and speculate on the extent to which expectancy
effects may be responsible for the experimental results. Finally, we discuss
more generally how the literature on expectancy effects can be applied to the
development and evaluation of human performance technologies.
Interpersonal Expectancy Effects
Def inition
An interpersonal expectancy effect occurs when a person (A), acting in
accordance with a set of expectations, treats another person (B.) in such a
OCR for page 6
2
manner as to elicit behavior that tends to confirm the original expectations
(Rosenthal, 1966, 1976~. For example, a teacher who believes that certain
pupils are especially bright may act more warmly toward them, teach them more
material, and spend more time with them. Over time, such a process could
result in greater gains in achievement for those students than would have
occurred otherwise.
The concept of an expectancy effect was first introduced by Merton (1948)
in his discussion of the self-fulfilling prophecy, which he defined as "a
false definition of the situation evoking a new behavior which makes the
originally false conception come true" (p. 195~. The first systematic
application of the concepts of expectancy effects in the field of psychology
came in the 1960s with a program of research on experimenter expectancy
effects (e.g., Rosenthal, 1963~. This research demonstrated that the
experimenter's hypothesis may act as an unintended determinant of experimental
results. In other words, experimenters may obtain the results they predicted
not because the relationship exists as predicted in the real world but because
the experimenters expected the sub jec ts to behave as they did .
Evidence for Interpersonal Expectancy Effects
Although originally fraught with controversy, the existence of
interpersonal expectancy effects is no longer in serious doubt. In 197B,
Rosenthal ant Rubin reported the results of a meta-analysis of 345 studies of
expectancy effects. A meta-analysis is the quantitative combination of the
results of a group of studies on a given topic. This meta-analysis showed that
the probability that there is no relationship between experimenters'
expectations and their subjects' subsequent behavior is less than .0000001.
The practical importance of expectancy effects was also substantial; the mean
OCR for page 7
effect size of expectancy effects across the 345 studies was equivalent to a
correlation coefficient of .33.
This meta-analysis also investigated the importance of expectancy effects
within a wide variety of research domains. There were eight categories of
expectancy studies: reaction time experiments, inkblot tests, animal learning,
laboratory interviews, psychophysical judgments, learning and ability, person
perception, and everyday situations or field studies. Although effect sizes
varied across categories, the importance of expectancy effects within each
category was firmly established. These results suggest that expectancy effects
may occur in many different areas of behavioral research and emphasize the
importance of taking into account the possibility of expectancy effects when
designing and conducting studies.
Although initially focused on the psychological experiment as the domain
of interest, research on expectancy effects turned quickly to other domains
where expectancy effects might be operating, domains such as teacher-student,
employer-employee, and therapist-client interactions. Over the years, research
interest has also turned from merely documenting the existence of expectancy
effects to delineating the processes underlying expectancy effects.
Methodological Implications of Expectancy Effects
Experimenter expectancy effects are a source of rival hypotheses in
accounting for experimental results. In other words, a given result could be
causes not by the independent variable under investigation but rather by the
experimenter' a expectation that such a result would be obtained. As rival
hypotheses, expectancy effects can be considered a threat to the internal
validity of a study; they are a source of systematic bias rather than random
error. Consequently, expectancy effects present a serious danger to the
OCR for page 8
4
interpretation of results: Increasing random noise merely makes it more
difficult to obtain significant results, but increasing systematic bias can
result in completely erroneous cone fusions .
Experimenter expectancy effects are a potential source of problems for
any research area, but they may be especially influential in more recent
research areas lacking well-established findings. This is because the first
studies on a given treatment or technique are typically carried out by
creators or proponents of the technique who tend to hold very positive .
expectations for the efficacy of the technique. It is not until later that the
technique may be investigated by more impartial or skeptical researchers, who
may be less prone to expectancy effects operating to favor the technique. Many
of the human performance technologies of interest in the present paper are
relatively recent innovations, and thus may be especially susceptible to
expectancy effects.
~ .
In principle, expectancy effects could be investigated by introducing
expectations as a manipulation in addition to the independent variable of
theoretical interest. This method, which will be described in detail later,
allows the direct comparison of the magnitudes of the effects due to the
phenomenon and effects due to expectancies. Another approach, perhaps even
richer theoretically, is to examine directly the processes underlying
expectancy effects as they occur in various areas. In Some areas, such as the
area of teacher expectancy effects, a considerable amount of research has been
conducted in this manner, ant there is now a good general understanding of
what variables are important in mediating teacher expectancies. However, in
other areas, such as the human technologies of interest here, this background
research is lacking. The best that can be done in such cases is: (a) to
OCR for page 9
Us
analyze the situations of interest, (b) to determine whether mediating
mechanisms shown to be important in traditional research areas are likely to
be present in the new areas, and (c) to estimate the extent to which
expectancy effects could be influential in the new area. The present paper
undertakes such an analysis.
Mediation of Interpersonal Expectancy Effects
Basic Issues
-
A primary question of interest with respect to expectancy effects is the
question of mediation: How are one person's expectations communicated to
another person so as to create a self-fulfilling prophecy? This question in
turn can be broken down into two components. The first component is the
differential behaviors that are displayed by the expecter as a result of
holding differential expectancies (the expecter-behavior link). For example,
in what ways do teachers treat their high expectancy students differently? The
second component is the differential behaviors that are associated with actual
change in expectee behavior and self-concept (the behavior-outcome link). For
example, what teacher behaviors result in better academic performance by the
students? Both these aspects are critical in understanding expectancy
mediation, for even if we could show an enormous effect of expectancy on
expecter behavior (e.g., teachers smile more at high expectancy students),
that behavior would not be important in expectancy mediation unless it
actually impacted on the expectee to create better outcomes (e.g., being
smiled at leads to better grates).
The Four-Factor "Theory"
~ ,
Rosenthal (1973a, 1973b) proposed a four-factor "theory" of the mediation
of teacher expectancy effects. In this view, four broad groupings of teacher
OCR for page 10
6
behaviors are hypothesized to be involved in teacher expectancy effects. The
first factor is climate, referring to the warmer 60cioemotional climate that
teachers may create for their high expectancy students. This factor includes
warmth communicated in both verbal and nonverbal channels. The second factor,
feedback, refers to teachers' tendency to give more differentiated feedback to
high expectancy students. The third factor, input, refers to the tendency to
teach more material and more difficult material to high expectancy students.
The fourth factor is output, or the tendency for teachers to spend more time
with high expectancy students and provide them with greater opportunities for
responding. Although the four factor theory was originally proposed to account
for the mediation of teacher expectancy effects, it seems reasonable to think
that these factors may also operate in other domains where expectancy effects
may be operating.
Meta-analysis _ Expectancy Mediation
The question of how expectancy effects are mediated is ultimately an
empirical one. Luckily, many studies address the mediation of expectancy
effects, and we have conducted a meta-analysis of this literature (Harris &
Rosenthal, 1985). Essentially, we read all the studies we could find that
examined expectancy mediation (resulting in an initial pool of 180 studies)
and classified them according to the mediating variables that were
investigated. This resulted in 31 mediating behaviors each of which was
examined in at least four studies. We then computed an overall significance
level and effect size for each of the 31 categories, separately for the
expectancy-behavior effects and the behavior-outcome effects.
The results of this meta-analysis pointed to the practical importance of
16 behaviors in mediation: negative climate, physical distance, input,
OCR for page 11
7
positive climate, off-task behavior, duration of interactions, frequency of
interactions, asking questions, encouragement, eye contact, smiles, praise,
accepting students' ideas, corrective feedback, nods, and wait-time for
responses. Table 1 summarizes the results of the meta-analysis for these 16
behaviors, presenting the effect sizes for the expectancy-behavior links and
the behavior-outcome links separately. An intuitive way of understanding these
effect sizes is given by the Binomial Effect Size Display (BESD; Rosenthal &
Rubin, 1982). The BE SD expresses correlations in terms of percent increase in
"success" rates due to a given "treatment," with the treatment group success
rate computed as .50~(r/2) and the control group success rate computed as
.50-(r/2). So, for example, the correlation of .21 for Positive Climate can be
interpreted using the BE SD as meaning that the percentage of teachers
exhibiting above average amounts of Positive Climate will increase from 39.5Z
[.50-(.21/2)] for low expectancy students to 60.5: [.50+(.21/2)] for high
expectancy students. The other effect sizes can be similarly interpreted.
Note that in Table 1 the effect sizes for behavior-outcome relations tend
to be larger than the effect sizes for expectancy-behavior relations. One
possible reason for this is that expectancies are manifested in myriad ways,
meaning that the relationship between expectations and any particular behavior
is not likely to be very strong. However, we can more accurately predict a
person's response to a particular behavior once we know that a particular
behavior has occurred. In other words, if we can condition on the behaviors
emitted, we are in a better position to make more accurate predictions.
We also presented a summary analysis evaluating the four factor theory.
The ten behavior categories with the most studies (and therefore providing the
most stable estimates) were reclassified into the four factors of climate,
feedback, input, and output. We then computed an overall significance level
OCR for page 142
38
belief in the new ~anscenderd physics remains.
Some recent examples of the "vividness" criteria in media reports are Me press coverage given
~ me~-bend~ng children (e.g. Defty, Washington Post' March 2, 1980) and the tremendous attention
given the Columbus, Ohio, poltergeist (Safran, Reader's Digest, December 1984; San Francisco Chron-
icle, March 7, 1984, from Associated Press). Both stones developed Croup extremely unreliable per-
sonal experience (Rand), 1983; Kurtz, 1984b) and demonstrate the way Mat personal reports fit me
requirements of the media better than caution or rigor. Expenmental analysis is rarely as dramatic or
newsworthy as personal reports, especially since rigorous analysis emphasizes a cautious conservative
approach. FoDow-up stones on the "debunking" of these phenomena rarely receive comparable atten-
non to the first excited reports.
The public television program Nova is regarded as one of the best popular freemen of
scientific affairs in any communicator medium. Yet itS program on ESP has been vilified by skeptics
of paranonnal phenomena (Lutz, 1984b). It tried to show both sides of the issue-- it included dramatic
"recreations" of the most famous ESP experiments and interviews with civics of ESP who proposed
altemadve explanations of these experiments. The mcreated stones were more exciting and vividly
memorable than the interviews. The enthusiasm and hopefulness of the believed was. more gripping
than the skeptics' "accenn~abon of Me negative". What were the producers of Nova to do about the fact
that what made a good story also was memorable and persuasive-- even though these elements were
irrelevant to what was ~e? In this case, Hey went for the good story.
OCR for page 143
39
Perceptual biases and mediated information
People with strong pm-exisiing beliefs are rarely affected by any presentation of evidence.
Instead, Hey manage to find some confim~adon in aU presentations. The "biased assimilator" of evi-
dence relevant to our beliefs is a phenomenon that seems obviously true of omen, but sometimes
difficult to believe In ourselves. Consider a classic social psychological study of students' perceptions
of the annual Pr~nceton-Darunouth football game. (Hastorf and Cant, 19541. Students from the
opposing schools watched a movie of the rough 1951 football game and were asked to carefully record
an infractions. The Ho groups ended up with different scorecards based on the same game. Of coupe,
this is not remarkable at an. We see this in sports enthusiasts and portico pow even day. But
what is worth noting is that the students used objective trial by trim recording techniques and they sod
saw different games if Hey were on different sides.
This is a clue to the reason that people cannot understand why others continue to disagree wad
them, even after they have been shown the "truth". We construct our perceived wodd on the basis of
expectations and theones, and then we fall to take this constructed nature of the world into account.
When we talk about the same "facts" we may not be arguing on the basis of the same construed evi-
dence. This is especially important when we are faced with ~nte~prei~ng mixed evidence. IN almost an
real-world cases, evidence does not come neatly packaged as "pro" or "con", and we have to interpret
how each piece of evidence supports each side.
IN a more recent extension of this idea, social psychologists at Stanford University presented
proponents and opponents of capital punishment with some studies that purported to show Hat deter-
rence worked, and some studies apparently showing that capital punishment had no deterrence effect
Cord, Ross & Lepper, 1979~. They reasoned that common sense must dictate that mixed evidence
should lead to a decrease in certainty In the beliefs of both partisan groups. But if partisans accept
OCR for page 144
40
supportive evidence at face value, critically scrutinize contradictory evidence, and construe ambiguous
evidence according to their theones, both sides might actuary strengthen their beliefs on the basis of
the mixed evidence.
'1he answer was clear in our subjects assessment of the pertinent deterrence studies.
Both groups believed that the methodology that had yielded evidence supportive of
Weir view had been clearly superior, both in its relevance and freedom from artifact, lo
the methodology that had yielded non-suppor~ve evidence. ~ fact however, the sum
jects were evaluating exactly the same designs and procedures, wad only me purported
results vaned....To put the matter more bluntly, the two opposing groups had each con-
strued the "box-score" vis a vis empirical evidence as tone good study supporting my
view, and one lousy study supporting the opposite view'-- a state of affairs that seem-
ingly justified the maintenance and even the strengthening of their paracular viewpoint"
(Ross, 1986, p. 14).
This result leads to a sense of pessimism for those of us who Mink that "truth" comes from the
objective scientific collection of data, and from a solid replicable base of research. Giving the same
mixed evidence to two opposing groups may drive the partisans farther apart. How is intellectual and
emotional rapprochement possible?
One possible source of optimism comes from related work by Ross and his colleagues (Ross,
Lepper & Hubbard, 1975) in which the experimenters gave subjects false information about their abil-
ity on some task. After subjects built up a theory to explain this ability, the experimenters discredited
the original information, but the subjects retained a weaker form of the theory they had built up. The
only form of debriefing that effectively abolished the (inappropriate) theory involved telling the sub-
jects about the perseverance phenomenon itself. This debriefing about the actual psychological process
involved finally aDowed the subjects to remove the effect of the false information. Biased assimilation
may be weakened in a similar way: when we understand that our most "objective" evaluations of evi-
dence involves such bias, we may be more able to understand that our opponents truly are reasonable
people.
OCR for page 145
41
Another reaction to processed evidence is the perception of hostile media bias. Why should
politicians from both ends of the spectrum believe that the media is particularly hostile to their side?
At first glance, this widespread phenomenon seems to contradict assimilative biases-- often, we don't
react to stories in the press by selectively choosing supportive evidence; instead we perceive that the
news story is deliberately slanted in favor of evidence against our side. Ross and colleagues speculated
that the same biasing construal processes are at work. A partisan has a rigid construction of the truth
that lines up with his or her beliefs, and when "evenhanded" evaluations am presented, they seem to
stress the questionable evidence for the opposition.
Support for these speculations came from studies on the news coverage of both the 1980 and
1984 presidential election and the 1982 "Beirut Massacre" (Vallone, 1986; Vallone, Ross ~ Lepper,
1985). These issues were chosen because there were actively involved partisans on both sides avail-
able. The opposing parties watched clips of television news coverage. Not only did they disagree about
the validity of the facts presented, and about the likely beliefs of the producers of the program, but
they acted as if they saw different news clips. "Viewem of the the same 30-minute videotapes reported
that the other side had enjoyed a greater proportion of favorable facts and references, and a smaller
proportion of negative ones, than their own side" (Ross, 1986, p. 18). However, objective viewers
tended to rate the broadcasts as relatively unbiased.
These "objective" viewers were defined by the experimenters as those without personal
involvement or strong opinions about the issues. But the partisans themselves-- if they are involved in
college football, the capital punishment debate, party politics or the Arab-Israeli conflict-- claim to be
evaluating the evidence on its own merits. And in a sense they are: They evaluate the quality of the
evidence as they have constructed it in their mind. It is the illusion of "direct perception" that is the
fatal barrier to understanding why others disagree with us. To the extent that we "fill in" ambiguities
OCR for page 146
42
in me infom~a~aon given we can find inte~pretatons mat make the evidence fit our model. Because
scientific practice demands public definition of concepts, measures and phenomena, personal constn~c-
dons are minimized and meanings debate can take place. But when we rely on casual observation,
personal experience and entenain~ng narratives as sources of evidence, we have too much room to
create our own persuasive consnual of the evidence.
Problems in Evaluanng Evidence [V: The Elect of Formal Research
FonTIal research structure and quantitative analysis may not be me only, or best, route to
"understanding" problems. Often, an in~epth qualitative familianty with a subject area is necessary ~
truly grasp the nature of a problem. But in all public policy programs, a private understanding must be
followed by a public demonstration of the efficacy of the program.
Only quantitative analysis leads to such a demonstration, and only quantitative evidence will
force partisans to take the other side seriously. The effect of the acceptance of this argument can be
seen in different ways in two domains: parapsycholog~cal research, and medicine. The effect of the
rejec~acn of this argument can be seen In the development of the human potential movement.
Modem parapsychology is almost entirely an experimental science, as any CUmOIy look
Hugh its influential journals vAI1 demonstrate. Articles published in He Journal of Parapsychology or
the Journal of the Sociery for Psychical Research explicitly discuss the statistical assumptions and con-
trolled research design used in their studies. Most active parapsychological researchers believe that the
path to scientific acceptance lies Hugh me adoption of rigorous experimental method.
Robert Jahn, formerly dean of engineering and applied sciences at Princeton University and an
active experimenter in this field, argues that "further careful study of this formidable field seems
justified, but only within Be context of very well conceived and technically impeccable experiments of
OCR for page 147
43
large data-base capability, with disciplined attention to the pertinent aesthetic factors, and with more
constructive involvement of the critical community" (Jahn, 1982, quoted in Hyman, 1985, p. 4). This
attitude has not caused the traditional scientific institutions to embrace parapsychology, so what have
parapsychologists gained from it?
Parapsycholog~sts have now amassed a large literature of experiments, and this compendium of
studies and results can now be assessed using the language of science. Discussions of the status of
parapsychological theories can be argued on the evidence: quantified, explicit evidence. As it stands,
the evidence for psychic phenomena is not convincing to most traditional scientists (Hymen, 1981).
But critical discussions of the evidence can take place on the basis of specifiable problems, and not
only on the basis of beliefs and attitudes (e.g. the exchange between Hyman and HonoIton on the qual-
ity of the design and analysis of the psi ganzfeld experiments, starting with Hym an, 1977; and Honor-
mn, 1979).
In direct contrast to this progression is the attitude of the human potential movement towards
evaluation and measurement. Kurt Back (1972) titled his personal history of the human potential
movement "Beyond Words" but it could have been just as accurately called "Beyond Measurement".
He begins his book and his history with an examination of the roots of the movement in the post-war
enthusiasm for applied psychology. Academic psychologists and sociologists were anxious to measure
the increase in efficiency that would result from group educational activities. They examined group
productivity, the solidarity and cohesion of the groups themselves, as wed as the weD-being of the
group members.
Few measurable changes were found, and this led the research-oriented scientists to either lose
interest in these group phenomena or to lose interest in quantitative measurement. Many of those
involved in the group experiments-- even some of the scientist who began with clearly experimental
OCR for page 148
outlooks-- were caught up in the phenomenology, the experience of the group processes.
44
Back describes many influendal workers in this movement who started out with keen beliefs
that controBed experiments wad groups processes would reveal significant observable effects. When
these were not forthcoming, Me believers made two claims: We effects of group processes were too
subtle, diffuse and holistic to be measure by reductionist science, and the only evidence that really
mattered was subjective experience-- the individual case was We only level of interest, and this level
could never be cape by extemal "objective" measurements.
"Believing the language of the movement, one might look for msearch, proof, and the
acceptability, of disproof. In fact, me followers of me movement are quote immune to
rational argument or persuasion. The experience they are seeking exists, and me believ-
ers are happy in their closed system which shows Hem mat Hey alone have mue
Lights and emotional beliefs....Seen in this light, He history of sensitivity mining is a
struggle to get beyond science" (Back, p. 204).
The dangem in trying to get beyond science In an important policy area am best described by
an example from surgical medicine. This example is often used in introductory statistics' classes
because it demonstrates that good research really makers In the world. It shows how opt bask
on personal experience or even unconsoled research can cause the adoption or condnuanon of
dangerous policies.
One treannent for severe bleeding caused by cinhosis of me liver is to send the blood Trough
a portacaval she. This operation is time-consuming and risky. Many studies (at least 50y, of varying
sophistication' have been undertaken to determine if the benefits outweigh the risks. These studies are
reviewed in Grace, Muench, and Chahners, 1966; He stadsucal meaning is discussed in Freedman'
Pisan~ & Pukes, 1978~.
The message of the studies is clear: the poorer studies exaggerate He benefits of the surgery.
Seventy-five percent of the studies without control groups (24 out of 32) were very enthusiastic about
OCR for page 149
45
tile benefits of the shunt. In the studies which had control groups which were not randomly assigned,
67% (IO out of IS) were very en~usiasuc about Me benefits. But none of the studies widi random
assignment to contm} and experimental groups had results that led to a high degree of enthusiasm.
He of these studies showed Be shunt to have no value whatsoever.
on the experiments without controls, me physicians wem accidentally biasing the outcome by
including only the most healthy patients in the study. In the e~cpenments with non~domued controls,
me physicians were accidentally biasing the outcome by assigning Be poorest padents to the control
group that did not receive the shunt. Ordy when the confound of patient heath was removed by mn-
domizanon was it clear that the risky operation was of lime or no vague.
Good research does matter. Even physicians, highly selected for intelligence and highly
trained in intuitive assessment, were misled by their daily expenence. Because He Amoral studies were
publicly available, and because the quality of the studies could be evaluated on the basis of their exper-
unental method, the overall conclusions were decisive. Until the human potential movement agrees on
the importance of quantitative evaluation, it win remain spUt into factions based on ideologies main-
tained by personal experience.
Focal research methods are not He orgy or necessarily best way to team about the true state
of nature. But good research is He ordy way to ensure Hat real phenomena win drive out illusions.
The story of the "discovery" of N-rays in France in 1903 reveals how even physics, the hardest of the
hard sciences, could be led astray by subjective evaluation (Broad ~ Wade, 1982, p. Ilk. This "new"
form of X-rays made sparks brighten when viewed by the naked eye. The best physical scientists in
France accepted this breakOuough because they wanted to believe in iL It took considerable logical
and experiment effort to convince the scientific establishment that He actual phenomenon was self-
decepi~on. Good research can disconfirm theones, subjective judgment rawly does.
OCR for page 150
46
In his clique of the use of poor research practices, Pitfalls of Hwnan Research, Barber (1976)
points out that many Saws of natural inference can creep into scientific research. "The validity and
generalizability of experiments can be significantly improved by malting more explicit the pitfalls Mat
are integral to their planning...and by keeping the pitfalls in full view of researchers who conduct
experimental studies" (pp. 90-91). While scientists and scientific methods are not immune to the flaws
of subjective judgment, good research is designed to minimize the impact of these problems.
We proper use of science in public policy involves replacing a "person-onent~" approach
win a "me~od-onented" approach (Hammond, 1978~. When cndcs or supporters focus on the person
who is setting policy cntena, the debate involves the bias and mobvanons of me people involved. But
attempts to precisely define the variables of interest and to gather data that relate to these variables
focus the adversanal debate on He quality of He nietho~s used. This "is sc~entificaBy defensible not
because it is flawless (it isn't), but because it is readily subject to scientific cnucism" (Hammond,
1978, p. 135~.
Intuitive Judgment and the evaluation of evidence: A summary
Personal experience seems a compeding source of evidence because it involves He most basic
processing of information: perception, attention, and memory storage and retrieval. Yet while we have
great confidence in the accuracy of our subjective impressions, we do not have conscious access to the
actual processes unsolved. Psychological expenment~ion has revealed that we have too much
confidence in our own accuracy and objectivity. Humans are designed for quick thinking rather than
accurate thinking. Quick, confident assessment of evidence is adaptive when hesitation, uncertainty and
self-doubt have high costs. But natural shortcut methods are subject to systematic errors and our in~s-
peci~ve feelings of accuracy are misleading.
OCR for page 151
47
These ergo of intuitive judgment lead people to search OUt confirming evidence, to interpret
mixed evidence in ways that confirm their expectations, and to see meaning in chance phenomena.
This same biased processing of information makes it very difficult to change our beliefs and tO under-
stand the point of view of those with opposing beliefs. These errors and biases are now well-
documented by psychologists and decision theorists, and the improvement of human judgment is of
central concern in current research. Me long-tenn response to this knowledge requites broad educa-
tional programs in basic statistical inference, and formal decision-m~king, such as those proposed and
examined by various authors in Kahneman et al (1982). Already, business schools include "de-
biasing" procedures in their programs of formal decision-making. But with the complex technological
nature of our society, most researchers believe that some instruction on how ~ be a better consumer of
infonnadon should start In public schools.
The immediate response should be a renewed commitment to formal structures In deciding
important policy, and a new realization that personal experience cannot be decisive in forming such
policy. As Gilbert, Light and Mosteller (1978) post Out ~ their review of me efficacy of social ~nno-
vations, only true experiment teals can yield knowledge that is reliable and cumulate. While for-
mal research is slow and expensive, and scientific knowledge increases by Any increments, Me final
result is impressively useful. Perhaps most important, explicit public evidence is our best hope for
moving toward a consensus on appropriate public policy.
OCR for page 152
