|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
|
| ||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 13
2
Basic Research on Incentives
A
broad interpretation of “incentive” could encompass all determi-
nants of behavior and require a literature review that includes all
fields in the social and behavioral sciences. As explained in Chap -
ter 1, the committee focused on research related to incentives in which an
explicit consequence—either positive or negative—is attached to a mea -
sure of performance and on two areas that together provide a complemen-
tary picture of what we know about their effects: theoretical research from
economics about using performance-based incentives and experimental
research from psychology on motivation and external rewards.1
The work from economics provides a framework for understanding
how the effect of incentives can vary from context to context and from
person to person. The work from psychology provides empirical results
showing how the behavior caused by incentives can vary from context to
context and from person to person. Together, these two literatures provide
a picture of the complexity of the structure of incentives and an under-
standing of the subtle differences in their design that can be crucial in
determining their effects. Although we use these two research literatures
to structure our analysis, we also discuss some empirical results from eco-
nomics, sociology, and personnel psychology where they are applicable.
1 Although the committee focused in particular on theoretical work from economics and
empirical work from psychology, we recognize that this division is artificial since the re -
search in both fields includes complementary theoretical and empirical work. Where ap -
propriate, the chapter notes related empirical work in economics and theoretical work in
psychology.
13
OCR for page 14
14 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
ECONOMIC THEORY AND ISSUES
Economics has a well-developed body of theoretical research that
looks at how organizational incentives should be designed and uses the
results of that work to understand why different organizations use dif-
ferent incentives. This body of research applies the general economic
approach of explaining human behavior as resulting from individuals’
trying to get the best outcomes for themselves within the constraints of
their environments. This general framework for understanding human
behavior has proven to be quite powerful, although there are critiques
that it misses important aspects of human psychology that limit the abil-
ity to determine the best outcome in the idealized way that economists
assume (see, e.g., Ariely, 2008; Rabin, 1998).
The research on the use of incentives in organizations extends the
general economic framework by analyzing differences in the objectives
of the individuals who make up an organization. In particular, the work
contrasts the objective of an organization as a whole—as defined by the
owner or “principal” of that organization—with the objective of an indi -
vidual worker or “agent.” As a very basic example, an owner probably
cares about the organization’s overall profit while the workers care about
their own pay, hours of work, and levels of effort. Because of the differ-
ence in these objectives, a worker in an organization may not behave in
ways that will best achieve the owner’s goals for the organization—which
can make the organization less productive and thereby make things worse
for the workers indirectly by reducing employment or pay in the long run.
To help correct such a situation, incentives can be used to encourage the
workers to work toward the owner’s goals for the organization.
The classic example of the effect of incentive structures is to contrast
the effect of paying workers by the hour with the effect of paying them by
the amount of work they perform measured by some quantity of output.
The latter is often known as a “piece rate,” derived from a manufacturing
context in which a worker is paid for each piece produced. The owner of
the company will want the workers to produce more per hour in order to
increase profits: switching to a piece rate gives the worker an incentive to
do so; paying by the hour may not do so. Sales commissions are one of the
well-known ways in which piece rates are currently used in many indus-
tries. Empirical research has shown many situations in which simple piece
rate incentives operate as the economic theory predicts (Prendergast,
1999), although the efficiency of incentives depends on the precise social
relations that tend to grow up around piece work (see Burawoy, 1979;
Sallaz, 2009).
Beyond the basic difference between paying by the hour and by the
piece, there are important and subtle complexities that affect the way
incentives operate. A number of contrasts in incentive structures provide
OCR for page 15
15
BASIC RESEARCH ON INCENTIVES
some understanding about the ways that incentives work in different
settings or for different people. In the rest of this section, we discuss five
different types of complexity that have been analyzed and the important
considerations they raise for the design of incentives in education:
1. finding performance measures,
2. the different effects of incentives on different people,
3. the effects of uncertainty and control,
4. the effects of working in groups, and
5. weighing the benefits of incentives against their costs.
Finding Performance Measures to Use with Incentives
In most jobs, the value of the work performed by each worker is dif-
ficult to assess. For example, for many jobs, it is hard to measure what
workers produce because their output cannot be counted in any meaning-
ful way. The qualitative aspects of that work—the relationship with the
client, the clarity of the report, the accuracy of the numbers—are more
important in determining its value than such countable outcomes as the
number of meetings held, pages written, or spreadsheets produced.
The difficulty in measuring the true results of what workers do is an
important constraint in providing incentives—and the difference between
the available measures of workers’ output and the true value of that out-
put has consequences for the way incentives operate. In an attempt to
provide appropriate incentives, organizations often look for performance
measures to use in objectively quantifying what each worker is produc-
ing. The problem is that these performance measures necessarily focus on
the aspects of the job that can be easily quantified and neglect the qualita -
tive aspects of the job that cannot be easily quantified. When incentives
are attached to these performance measures, the predictable result is that
workers often focus on the readily quantifiable aspects of the job that
affect the performance measures and neglect the quantitative and qualita-
tive aspects of the job that do not factor into the performance measures.
There are numerous examples of the distortion that results from the
use of incentives with performance measures that do not adequately
reflect the true value of the work that is being done. These examples con-
firm the findings in the theoretical analyses about the problems that can
result when incentives are attached to performance measures that are not
closely aligned with the true value of the work. For example, computer
programmers rewarded by the length of their programs write longer pro-
grams, surgeons penalized for high mortality rates take less risky cases,
and chief executive officers (CEOs) rewarded for their company’s earn -
OCR for page 16
16 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
ing performance manipulate those earnings reports (Prendergast, 1999;
Rothstein, 2008).
A good example of this kind of result in education occurs when incen-
tives are attached to the number of “proficient” students: the result is that
extra attention is given to the students who are just below the threshold
of proficiency, while teachers and schools may compete for the proficient
students who do not bring the threat of negative consequences. Another
example can be seen when college rankings reward a more selective
admissions policy: the result is that college recruiters encourage appli -
cants from unqualified students because they will effectively get credit
for rejecting them (Stevens, 2007).
In these examples, the incentives placed on the performance mea -
sures lead workers to perform actions that increase the performance
measures but not the underlying value of their work. It is typical for
performance measures to become distorted when they are used for
incentive purposes. This is a version of the phenomenon sometimes
referred to as Campbell’s law (Campbell, 1975, p. 49):
The more any quantitative social indicator is used for social decision
making, the more subject it will be to corruption pressures and the more
apt it will be to distort and corrupt the social processes it is intended to
monitor.
In organizations seeking to find an appropriate incentives scheme, the
performance measures used may evolve over time in the search for mea -
sures that are well aligned with the true value that the workers produce.
For example, the Job Training Partnership Act of 1982 initially provided
incentives for local employment and training centers that were based on
job placement rates and the wages at the time of placement. These incen -
tives led the centers to focus on people with stronger work histories who
were more likely to find work and to be paid more. The program then
added performance measures focused on people with weaker work his-
tories and on changes in earnings. The successor to this act, the Workforce
Investment Act, currently uses a combination of 17 performance measures
to provide incentives to local employment and training centers (Heinrich
and Marschke, 2010). Similar evolution in performance measures has
occurred in other areas in which performance incentives have been used,
such as in health care (see Rothstein, 2008).
In education, many incentives are currently focused on a narrow
set of measures derived from annual test results in grades 3-8 in read -
ing and mathematics. This focus falls far short of a complete measure of
desired educational outcomes. Most notably, it omits entirely such things
as advanced levels of performance in the two tested subjects; areas of per-
formance in those subjects that are hard to assess with standardized tests;
OCR for page 17
17
BASIC RESEARCH ON INCENTIVES
performance in other subjects and other grades; growth in such important
characteristics as creativity, curiosity, persistence, values, collaboration,
and socialization; and the eventual success of students in graduating,
obtaining postsecondary education, finding productive and satisfying
work, and contributing as members of their communities.
The challenge of finding appropriate performance measures to use
with incentives is often made more difficult by the challenge of defining
the underlying goals that one wants the performance measures to reflect.
For many organizations, it can be difficult or impossible to specify the
organization’s goals in a way that would satisfy all stakeholders. This
can be true not only for such institutions as not-for-profit organizations,
government agencies, and schools, but also for groups and individuals in
for-profit firms. For example, different stakeholders in a for-profit energy
corporation may disagree about whether to focus on fossil fuels or on the
development of renewable energy sources.
In education, schools are responsible for educating students in many
ways: fostering cognitive skills, emotional and physical development,
readiness for work and civic participation, as well as students’ health and
safety. In addition, schools are charged with ensuring that all students
meet some minimal standards and that some of them are able to meet
very high standards. Although these goals are not inconsistent, they all
compete for the limited education resources that are available and, ulti -
mately, require schools to make difficult tradeoffs among them (Dixit,
2002). These trade-offs affect the design of the accountability system. Ide -
ally, one would like to have at least one performance measure linked to
each goal, but this ideal is often not practical to carry out. Consequently,
further trade-offs in the selection of performance measures are generally
called for. Finally, within the set of performance measures that will be
used, it is necessary to decide how heavily to weight each measure in the
overall incentives system. This added challenge of reaching a consensus
about an organization’s objective that can be captured in a set of feasible
performance measures compounds the difficulty of finding appropriate
measures that are aligned with that objective.
A theoretical analysis shows that an optimal incentives scheme will
place less weight on performance measures that are less aligned with the
true value of what the workers produce (Baker, 2002). What is critical
is not whether there is an overall correlation between the performance
measures and the workers’ true productivity, but whether they are corre-
lated “at the margin”—that is, correlated for additional changes from the
status quo—so that the actions that improve the measures also improve
the workers’ underlying productivity.
The distinction between an overall correlation and a correlation at
the margin is especially important because of the distortion in perfor-
OCR for page 18
18 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
mance measures that occurs when incentives are attached to them. A
performance measure may be generally correlated with the full range of
outcomes without incentives—so that high levels of the measure are asso-
ciated with overall good performance—but when incentives are attached
to the performance measure, the actions taken to increase the performance
measure on the margin may not increase overall performance at all. This
common outcome is referred to as “gaming” the system or the test. Test
preparation classes are an example of this phenomenon: knowing when
to guess at the answer and when to skip the question will improve a score
without increasing learning in the domain of the test (Koretz, 2008b).
In education, it is not clear how strong the current incentives are.
Objectively they may seem small, because they rarely involve serious
consequences, like substantial bonuses or decertification in the case of
teachers. However, studies of cheating by teachers suggest that some of
them react very strongly to the seemingly small incentives in the current
system (see Chapter 4). It is also important to keep in mind, as noted
below, that any given set of incentives will have different effects in differ-
ent settings and for different people, causing some people to work harder
or more effectively and others to give up or to work in ways that thwart
an organization’s goals.
Distortion is virtually unavoidable in an incentives system that uses
performance measures that do not reflect the full value of workers’ pro -
ductivity. And as noted above, few jobs lend themselves to comprehensive
measurement, so one should usually expect some distortion and take
steps to minimize it. In education, one example of distortion occurs when
teachers and students focus narrowly on tested material and ignore topics
that are not covered on the tests.
In evaluating an incentives system, it is important to evaluate, not
whether distortion exists, but whether the incentives result in a sufficient
increase in the output desired to justify the costs of running the incentives
system, including the costs of monitoring the performance measures,
providing the incentives, and addressing the unintended, negative effects.
Because of the distortion in performance measures that results from plac-
ing incentives on those measures, the true change in the output that
results from the incentives system cannot be determined by looking at
changes in the performance measures being used in the system, but must
be determined by looking at other indicators of performance. In an educa-
tional setting using test-based incentives, this means that it is necessary to
look at other tests besides the tests attached to the consequences—other
tests that are not themselves designed to mimic the high-stakes test—in
order to determine how the incentives are affecting achievement.
As a result of the difficulty in measuring results, most organizations
base their incentives on subjective rather than objective measures or on
OCR for page 19
19
BASIC RESEARCH ON INCENTIVES
some combination of the two (Prendergast, 1999; Rothstein, 2008). Sub-
jective measures have the potential to provide a more complete assess-
ment of the contribution of each worker, with the ability to appropriately
take into account special circumstances and to discount the value of
quantitative measures that may be influenced by gaming behavior. Of
course, there are problems with subjective measures, including that their
reliability and validity are affected by such things as the reluctance of
supervisors to differentiate workers in their performance assessments,
the information that is privately held by workers about their own effort
and performance, and the attempts of workers to game the measures by
spending time to influence their supervisors’ assessments. These difficul -
ties will be compounded in settings, such as schools, that do not face
strong pressure to produce good results and may have personnel policies
that discourage differentiation of workers on the basis of their perfor-
mance. Systems that rely on subjective performance measures must have
or must create incentives for the relevant authorities (e.g., principals) to
act on their subjective assessments, while protecting the workers from
arbitrary—or even capricious—evaluations.
The Different Effects of Incentives on Different People
One of the important results in the economic theory about incentives
is that the effect of a particular incentives structure is likely to be differ-
ent for different people. Although incentives are often structured so that
everyone is given the same target, the target will often be easy for some
people to meet but hard for others (Lazear, 2000). As a result, the effect
of the incentive is likely to differ, encouraging greater performance for
those people who are able to reach the target with some extra effort but
discouraging performance for those people who believe they are unlikely
to reach the target at all.2
This differential effect can lead in turn to differential turnover across a
group of people receiving incentives: over time, an organization that uses
performance incentives is likely to attract and retain workers who can
achieve the targets that are rewarded by the incentives, while workers who
are unlikely to be successful will become discouraged and leave. Research
shows this differential effect of incentives on people. For example, Lazear
(2000) studied a change from hourly to piece-rate pay for workers who
install windshields in cars: he found that productivity improved by 35
percent, one-third of which was produced by lower productivity work-
ers leaving the firm and being replaced by higher productivity workers.
2 See the discussion in the section “Psychological Results and Issues” below about the ef -
fects of low and high targets.
OCR for page 20
20 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
The knowledge that incentives will have different effects on different
people depending on their ability to achieve the targets can be readily
applied to examples within education. Lazear (2006) applies the theory to
the case of incentives given to teachers—in a model in which teachers dif-
fer in their effectiveness in raising student test scores—and produces the
result that incentives will cause some teachers to increase their effort and
others to change occupations. In Lazear’s model, this differential reaction
would lead to increasing effectiveness in the pool of teachers over time—
as measured by the ability of the teachers to raise test scores—because the
ones who leave are those who are less able to respond effectively to the
incentives.3 Similarly, economic theory suggests that incentives given to
students—such as high school exit exams—will cause the students who
have greater ability to pass the test, but can only pass it with increased
effort, to increase their effort while causing the students who have less
ability to pass the test to drop out (Betts and Costrell, 2001). If exit exams
are introduced without making other adjustments to provide remedia-
tion and support to students who will have difficulty passing the test,
the differential reaction could lead to increasing achievement in students
who graduate and increasing numbers of students who do not graduate.
(Chapter 4 looks at the literature related to these responses by students
and teachers in more detail.)
In the teacher and student examples just mentioned, the economic
models assume that the actions available to the teachers and students
are either to increase effort or quit. A similar model of the different reac-
tions to test-based incentives might consider instead that the two actions
available are different versions of “increasing effort”—one involving
greater focus on the full curriculum and the other involving extra time
in test preparation. A model of teacher and student reactions to test-
based incentives—in which different teachers and students have different
abilities to be successful on the tests by focusing on the full curriculum or
different beliefs about what instructional strategy would be successful—
would show that the same incentives structure could lead to an increased
attention to the full curriculum for some teachers and students while also
leading to increased attention to test preparation by others. Teachers and
students who believed they could be successful on tests by focusing on
the full curriculum might choose to do so in such a model, while others
might choose instead to focus on test preparation.
3 Research related to teacher turnover has shown that the teachers who leave teaching
before their second year tend to be worse than the average teacher, as measured by changes
in student test scores (Boyd et al., 2009). This research was conducted under the general
approach to school-based accountability under the No Child Left Behind Act; we are not
aware of any research comparing types of teacher turnover occurring with stronger and
weaker teacher incentives.
OCR for page 21
21
BASIC RESEARCH ON INCENTIVES
As noted in the previous section (and discussed in more detail in
Chapter 3), a focus on test preparation is likely to distort the test scores,
resulting in an increase in the scores that is inflated as a measure of the
true learning in the domain. So in this alternative model the same incen -
tive might lead to test score increases for both groups of teachers and
students, but the actions producing those score increases and the true
learning involved would be dramatically different for the two groups.
Importantly, these differences would be invisible without gathering addi -
tional information beyond the test score data.
Effects of Uncertainty and Control in Providing Incentives
In most jobs, both the value of what workers produce and the mea-
sures of that value can be strongly affected by many factors that the work-
ers themselves do not control. For example, a client might be very moti-
vated or not, or budget constraints may limit options for improvements
that are needed. As a result, if an employer uses incentives, it is likely
that the payoffs will vary according to those other factors, in addition to
varying because of the workers’ own efforts. However, people generally
dislike uncertainty, and if their pay is going to be influenced by factors
they cannot control, they will want a higher level of pay on average to
compensate for that uncertainty.
A theoretical analysis shows that an optimal incentives scheme will
place less weight on performance measures that are subject to greater
uncertainty because they are subject to factors that the workers do not
control (Baker, 2002). The use of such performance measures will require
firms to pay their workers a higher average level of pay because the work-
ers will need to be compensated for the greater uncertainty in their pay
in comparison with what they would receive at another job. Although
the firm may benefit on average from the response of the workers to the
incentives, the higher level of average pay that workers need to compen -
sate them for the uncertainty will reduce the extent to which the firm
uses incentives. If the workers are adverse to the uncertainty associated
with such performance measures, it may not be worthwhile for the firm
to use incentives because the benefit from the increased productivity of
the workers due to the incentives may be less than the cost of the higher
average pay needed to compensate them for the uncertainty.
In education, many factors affect student learning that teachers and
schools do not directly control, including, in particular, many aspects of
students’ home environments. As a result, the learning that occurs in the
classroom of an individual teacher can vary widely from student to stu-
dent and from year to year. As a result of this uncertainty and variability
in student outcomes, many teachers dislike incentives based on student
OCR for page 22
22 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
outcomes and will need to be compensated at a higher average level to
make up for this uncertainty. Although there are things schools can do to
affect or work around aspects of students’ home environments—such as
working with parents or providing breakfast or study time at school—
such interventions are not likely to be sufficient to counteract the variabil-
ity in home environments across students. There is also strong evidence
of random year-to-year fluctuations in student performance even at the
school level, perhaps because one year the test happens to ask more ques-
tions that were covered in the school’s curriculum or because of common
environmental factors, such as whether there was an important school
basketball game the night before the exam (Kane and Staiger, 2002).
In many jobs, workers are compared with each other as a way of
reducing the effect of factors that the workers themselves do not control.
The argument is that workers in similar jobs will be subject to similar
uncertainties that are beyond their control. So, for example, CEOs may be
judged by the performance of their company’s stock price compared with
other companies in the same industry as a way of controlling for changes
in the industry that are outside the control of each CEO. The technique of
comparing workers with each other rather than to an objective standard is
often used in promotions, which is one of the most common ways of pro-
viding incentives in firms (Prendergast, 1999). In education, the approach
of comparing teachers or schools with each other could address common
year-to-year changes, like fluctuations in test difficulty, but it would not
account for the most important year-to-year changes, which occur at the
student level and so do not affect every teacher in the same way. Many
researchers are currently working on “value-added” techniques, which
statistically adjust for differences at the student level to make it possible to
compare the results of different teachers. However, as noted in Chapter 3,
it is not yet clear how fully these models can account for student differ-
ences to provide accurate measures of teacher effectiveness.
Effects of Groups in Providing Incentives
Economic theory has also looked at some of the issues in designing
incentives for groups of people rather than for individuals. In many jobs,
workers need to work together in a team (or group), and the results of
their work depend on the contributions of all the members. There are inev-
itable tradeoffs in the available measures of the workers’ contributions.
On the one hand, any measures of the work done by individual workers
will miss their contributions to the work of the other team members and
so will give an inaccurate assessment of that worker’s total productivity.
On the other hand, performance measures based on the productivity for
the entire team will be very uncertain indicators of the performance of
OCR for page 23
23
BASIC RESEARCH ON INCENTIVES
any single worker because they will depend on the performance of all
members of the team. In this situation, there is a tension between using
inaccurate individual performance measures that ignore each worker’s
contributions to the team and using team performance measures that vary
because of the performance of all the team members and therefore pro -
vide only weak incentives to each worker. Whether it is better to provide
incentives at the individual or team level in this situation depends on the
relative importance of cooperation by the team members and the degree
of uncertainty added by using a team performance measure (Baker, 2002).
In education, student learning is affected by many other people besides
the designated teacher for a class, including other teachers, students’ par-
ents, and students’ peers. In addition, there are important opportunities
for teachers to contribute to the teaching skills of their colleagues, thereby
affecting the learning of their colleagues’ students indirectly. (Chapter 4
discusses results of studies providing incentives to teachers, including
experiments that compared the effects of incentives provided to individual
teachers and to all teachers as a group in a school.)
Research outside economics raises issues about the functioning of
organizations that go beyond the issues addressed by economic theory.
For example, sociological research deals with the structure of organiza-
tions and the formation of occupational norms. Even in schools in which
teachers do not appear to be working together or working with each
other’s students, there are still important group processes that influence
how any external incentives are interpreted and communicated among all
the teachers. Organizational theory describes schools as “loosely coupled”
organizations that buffer classroom practice from change and outside
scrutiny and therefore respond to outside pressures by making largely
symbolic changes (Firestone, 1985; Meyer and Rowan, 1977, 1978; Weick,
1976).
The standards-based accountability movement recognized these ten-
dencies and sought to counter them. A call for systemic reform by Smith
and O’Day (1990) argued that a “fundamental barrier to developing and
sustaining successful schools in the USA is the fragmented, complex
multi-layered educational policy system in which they are embedded”
(p. 237). The systemic reform strategy aimed to overcome loosely coupled
organizational structures through state-led education reform that empha -
sized unified goals, a coherent system of instructional guidance, and
restructured educational governance. Some of the concrete manifestations
of this approach have included school-level efforts to coordinate, support,
and monitor instruction by changes to emphasize principals’ roles as
instructional leaders, promote mentoring relationships among teachers,
and institute coaching models for teacher improvement.
Organizational theories predicted that the shift toward systemic
OCR for page 26
26 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
PSYCHOLOGICAL RESULTS AND ISSUES
As in economics, psychology has a long-standing appreciation of the
importance of incentives in motivating behavior—going back to the begin-
ning of the discipline—with research over the past few decades showing
the complexity of the relationship between incentives and behavior. This
research has led to the counterintuitive finding that under some circum -
stances incentives actually reduce the behavior that is being rewarded
rather than increase it.
The counterintuitive result has shown up in experiments that provide
an explicit incentive that takes the place of preexisting internal motivation
by rewarding people for behavior they would have engaged in anyway
without the incentives. For instance, Deci (1971) found that when college
students were paid to perform interesting cube puzzles, they were less
likely to perform the puzzle on their own during a free-choice period.
Similarly, when nursery school children were offered a “good player
award” for drawing a picture, they were less likely to draw when they
were back in their regular classrooms (Lepper, Greene, and Nisbett, 1973).
Once explicitly rewarded for a particular behavior, people tend to stop
that behavior when the reward is discontinued. A number of other early
studies showed that use of an external reward to motivate people to do
something they would have done anyway can have detrimental effects
on the quality and creativity of performance, as well as on subsequent
motivation to perform the activity (Lepper and Greene, 1978).
The finding that external rewards can undermine internal motiva-
tion was initially very controversial, seeming to contradict both conven-
tional wisdom and a wide body of experimental research in psychology.
Over a decade, a succession of meta-analyses both supported (Rummel
and Feinberg, 1988; Tang and Hall, 1995; Wiersma, 1992) and contested
(Cameron and Pierce, 1994) that finding. These were followed by a new
meta-analysis that provided a more complete and nuanced review of the
contrasting conditions in the literature (Deci, Koestner, and Ryan, 1999).
The new meta-analysis considered 128 studies published from 1971 to
1999, including each of the studies addressed by Cameron and Pierce
(1994); this study showed clearly that tangible rewards do significantly
and substantially undermine internal motivation.
Other research at the intersection of psychology and economics has
shown that the way people perceive consequences and the way they
decide between options with different consequences can be strongly
affected by the way the different options are framed (see, e.g., Ariely,
2008; Rabin, 1998). For example, options framed as losses are perceived
differently than the same options framed as gains. Similarly, people may
reject options that are objectively better if the options are framed in a way
that makes them seem unfair. A number of researchers have attempted to
OCR for page 27
27
BASIC RESEARCH ON INCENTIVES
reconcile these psychological findings with the more standard view from
economics that people choose according to the objective benefits of the
different options, without reference to how those benefits are described
(e.g., Fehr and Falk, 2002; Frey and Jegen, 2001).
In the rest of this section, we look in more detail at the specific circum-
stances that produce the negative effect of rewards on internal motivation,
and on the research that has focused on learning and educational settings.
We do so in three areas:
1. internal and external motivation,
2. the motivation to learn, and
3. incentives and public service work.
Internal and External Motivation
Deci and Ryan (1985) synthesized the large body of experimental
work on human motivation in a theory that provides a framework for
understanding the varying effects of external rewards. In this theory, inter-
nal motivation derives from a basic human need for self-determination
that involves being able to make choices and manage the interaction
between oneself and one’s environment. When self-determined, a person
will “engage in an activity with a full sense of wanting, choosing, and per-
sonal endorsement” (Deci, 1992, p. 44). The need for self-determination
involves needs for autonomy, competence, and relatedness, each of which
can be affected by external rewards.
Autonomy refers to the extent that people do something of their own
choosing, both in and out of the context of external pressures. For exam-
ple, one student may do homework simply to avoid punishment from his
parents. Another student may do homework because she believes, despite
a lack of interest in the topic, that it may be useful to her career. Both
students are doing things that they would not do out of interest, so both
are externally motivated. Yet the behavior of the second student entails
more of an element of choice rather than simple compliance, and therefore
she is exercising a certain degree of autonomy. The student has identified
and understood the importance of the behavior and has internalized and
assimilated it. In this respect, the student’s behavior shares many charac -
teristics with behavior that is internally motivated.
It is creating this type of “buy-in” that is such a challenge for edu-
cators and employers. It can be fostered by giving a student a sense
of relatedness, which is a sense of belonging with the school (or other
institution, person, or family) and sharing and accepting its mission or
goal. Competence is another key factor—that is, the feeling on the part
of a student that she understands the goal and has the skills to succeed.
OCR for page 28
28 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
This contrasts with the first student, who is simply complying with his
parents but likely feels controlled, with little autonomy, which has a nega -
tive effect on internal motivation. The needs for autonomy, competence,
and relatedness help to make sense of the effects of external rewards in
different contexts.
One particularly interesting finding on the effects of external rewards
relates to rewards for doing well or meeting a specified standard. The
effects of such rewards have been shown to be mixed (Deci, Koestner,
and Ryan, 1999): sometimes receiving a reward that signifies compe-
tence appears to enhance subsequent motivation, sometimes it seems
to decrease subsequent motivation, and sometimes it seems to have no
effect. The key seems to be that when rewards signifying competence are
used in a way that seems very controlling to the person—so that it limits
autonomy—the result tends to be negative. However, if a reward does
not seem to be pressuring but instead simply signifies competence, it can
have the intended positive effect. When people are told their performance
is being evaluated, it is often experienced as controlling, and internal
motivation may decrease even when positive evaluative information is
subsequently provided (Harackiewicz, Abrahams, and Wageman, 1987).
The information provided by the reward can also be important. In
situations in which higher performers receive higher rewards while lower
performers receive lower rewards, the less-rewarded performers tend to
interpret the shortfall as a signal not only that they did poorly this time,
but also that they are unlikely to do well in the future. People who feel
incompetent stop trying and become “demotivated.” This effect has been
found to be quite large (Deci, Koestner, and Ryan, 1999).
Rewards need not be tangible. The effects of verbal praise are also
complex (Henderlong and Lepper, 2002). Verbal rewards are generally
predicted to enhance internal motivation because they are informational,
and they also feed a person’s feelings of competence. However, not all
praise has a positive effect. Several studies have shown that controlling
positive feedback that seems to pressure the recipient (“good, you did just
as you should”) leads to less internal motivation than positive feedback
that is purely informational (“you did well on that task”) (Pittman et al.,
1980; Ryan, 1982). One study found that verbal feedback that empha -
sizes performance relative to others tends to reduce internal motiva-
tion, whereas feedback centered on whether one has reached a certain
level of performance on the task tends to increase interest in the task
(Harackiewicz Abrahams, and Wageman, 1987).
The kinds of goals that are set also affect motivation. The highest
level of effort occurs when the task is moderately difficult, and the lowest
levels occur when the task is either very easy or very hard (Abramson et
al., 1978; Atkinson, 1964; Csikszentmihalyi, 1990; Deci and Ryan, 1985).
OCR for page 29
29
BASIC RESEARCH ON INCENTIVES
When goals are set so high that people do not believe they can achieve
them, the goals are demotivating and set the stage for feelings of helpless-
ness, reduced effort, withdrawal, and lower self-esteem. Another factor is
the specificity of the goals that are set: identifying specific, difficult goals
leads to higher performance than does simply urging people to do their
best (Locke and Latham, 2002). This occurs because do-your-best goals
have no external referent and allow for a wide range of acceptable perfor-
mance levels. The ideal goals provide optimal challenge: they encourage
people to stretch themselves and are attainable with effort. Incentives
can in turn affect the goals that people set for themselves: when people
are given a choice of tasks that differ in terms of difficulty, they tend to
choose relatively easy tasks if there is an external reward for successful
completion, but they choose more challenging tasks in the absence of an
external reward (Shapira, 1976).
The Motivation to Learn
Rewards-based incentive systems are commonly used in educational
settings. For example, many teachers try to improve student performance
by systematically rewarding students who follow classroom rules with
praise, gold and silver stars, or tokens exchangeable for prizes. 4 Grades
and honor rolls are intended to recognize excellence in achievement, but
they may come to serve as external rewards to motivate students to work
hard.
Incentives can backfire if they reduce the motivation that students
have to learn. Young children have a natural propensity to learn that
prompts exploration, curiosity, and a readiness to engage new material
that frequently results in learning without the application of any external
incentives. Given this propensity, why is the use of grades a standard
part of the educational system? Part of the answer is that grades provide
information to others (such as parents and college admissions officers)
about how students have done, as well as information to the students
themselves that they are learning the right things. Another part of the
answer, of course, is that students are not always motivated to learn
the things that may be useful or important for them to learn, so that the
external signaling and motivation provided by grades can encourage stu -
dents to learn skills and topics they might otherwise ignore. Sometimes
the initial learning produced by external motivation will lead students to
discover an interest in a new area that can lead to internal motivation for
later learning.
4 See, for example, the National Education Association’s resources on classroom manage -
ment at http://www.nea.org/classmanagement/archive.html [December 2010].
OCR for page 30
30 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
The key to using rewards in the classroom is to do so in a way that
fosters autonomous motivation. As discussed above, autonomous moti -
vation involves engaging students in a learning activity by helping them
identify with and fully accept its importance for their own personal goals
and values, even though the activity is not inherently interesting to them
(at least initially) and therefore not internally motivating. An example
would be a student who studies biology very hard in order to get excel-
lent grades so she can go to medical school, because that is her personal
goal. Autonomous motivation has been found to be a strong predictor
of both conceptual learning and psychological well-being (Benware and
Deci, 1984; Deci, Ryan and Williams, 1996; Grolnick and Ryan, 1987;
Grolnick, Ryan, and Deci, 1991). Autonomous motivation has also been
found to be associated with greater creativity on art activities at the
elementary level (Koestner et al., 1984), less likelihood of dropping out
for high school students (Vallerand et al., 1997), and a preference for chal -
lenges (Shapira, 1976).
Contemporary theories of motivation predict that tests and other
forms of evaluation will best foster learning when they have informa -
tional significance. Evaluations would be expected to be most motivating
when they provide relevant feedback in a noncontrolling way—that is,
by providing individuals with specific feedback that points the way to
becoming more effective or more competent, but without pressure or
control. Motivation research suggests that when evaluations are experi-
enced as controlling, they may produce temporary compliance, but they
ultimately undermine internal motivation and commitment to the activity.
And when evaluations convey incompetence to the individual, or when
they are based on overly challenging standards that are perceived to be
beyond the reach of the individuals, they are likely to reduce motivation
and lead to a withdrawal of effort.
Both experimental and field studies have supported these predictions
concerning the effects of evaluations on motivation (Ryan and Brown,
2005). For example, in one experiment at an elementary school (Grolnick
and Ryan, 1987), students were given a reading comprehension task
under three conditions: (1) they were told they would not be tested at all;
(2) they were told they would be tested, but only to determine what kinds
of ideas were learned, with no consequences for failure or success; and (3)
they were told they would be tested and graded, and the grade would be
delivered to their classroom teacher. The results showed that under the
third condition—which represented a controlling use of evaluations—the
students demonstrated short-term, rote memory but produced a signifi -
cantly lower level of conceptual learning than the two noncontrolling
conditions.
Classroom studies have shown that when teachers are oriented toward
OCR for page 31
31
BASIC RESEARCH ON INCENTIVES
being controlling (for example, using evaluations and rewards), students
are less internally motivated, less desirous of challenges in schools, and
less confident in their skills (Deci et al., 1981; Ryan and Grolnick, 1986).
What leads teachers to be controlling? They may become controlling
when they themselves are pressured to get students to perform. In one
study (Deci et al., 1982), participants were given the task of helping stu -
dents learn a cognitive-perceptual task. The teachers all had the same set
of problems to work with and were given the same preparation. However,
one group of teachers was explicitly told that it was their job to make sure
their students performed “up to high standards,” while the other group
received no such instruction. The participants who were pressured to pro-
duce high student achievement were more controlling and less support-
ive of students’ autonomy. Specifically, they engaged in more lecturing,
criticizing, praising, and directing—all techniques that have been shown
to have a negative impact on students’ interest in learning and their will-
ingness to undertake difficult academic challenges. A subsequent study
(Flink, Boggiano, and Barrett, 1990) examined this effect in the context of a
newly introduced school-based curriculum for elementary students across
several schools. As would be predicted by the prior work, the results
showed that teachers pressed toward higher standards were more likely
to engage in controlling instructional behaviors, and the more they did
so, the more poorly the students actually performed on the objective tests.
Finally, there is evidence to suggest that focusing parents’ concerns
on performance outcomes will lead them, like teachers, to use pressuring
motivational strategies that may backfire, leading to lower achievement
over the long term (Grolnick, 2003; Grolnick et al., 2002). In contrast, par-
ents who are supportive of their children’s autonomy can help them to be
more internally motivated (Grolnick, 2009).
Incentives and Public Service Work
Although most jobs include an external financial reward, many peo-
ple enjoy work and do not do it only because they are paid. Organiza-
tional psychologists have found that people are most motivated in jobs
that involve a wide variety of tasks, give them autonomy, provide good
feedback, and encourage identification with the organization’s mission
(Griffin, 1991). The organization’s mission may be particularly important
to teachers and other people in public service who are motivated by the
opportunity to contribute to society by such goals as educating children,
caring for the elderly, or helping the unemployed (Burgess and Ratto,
2003; Heinrich and Marschke, 2010; Prendergast, 1999). One example of
the commitment of teachers to the mission of educating children is the
finding, from two surveys (Ingersoll, 2003, p. 179), that most teachers
OCR for page 32
32 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
spend a substantial amount of their own money on curriculum materials
and classroom supplies.
People in public service professions often view themselves as organi -
zational stewards and may be motivated primarily by internal rewards,
such as trust, autonomy, and job satisfaction, and the goals of their organi-
zation. The feeling of contributing to the public good may satisfy their per-
sonal needs and goals. In the field of public administration, research has
found support for the importance of public service motivation (Heinrich
and Marschke, 2010).
This motivation potentially has implications for the effectiveness of
external incentives in public service occupations. Because of their under-
lying motivation, giving public service workers a financial incentive may
be counterproductive, in that it signals that the relationship between the
organization and the employee more closely resembles a market one, thus
diluting effort and motivation (Burgess and Rato, 2003). Although there
have been successful public-sector incentive programs—such as a Brazil -
ian program that tied the pay of tax collectors to the number of tax evad-
ers they apprehended (Kahn et al., 2001)—it is important that the incen-
tives structure be aligned with the mission of the organization. Chapter 4
discusses the evidence related to providing performance pay for teachers.
CONCLUSIONS
The overarching message about incentives from research in econom-
ics, psychology, and related fields is that the details of incentive systems
are critical to their success. Although there are many situations in which
incentives work in a straightforward way, increasing the targeted behav-
ior as intended, there are also many situations in which incentives fail
to produce their desired effect because important details have not been
taken into account.
Although the research in economics and psychology shares a high-
level conclusion about the complexity of the link between incentives and
behavior, the two fields point toward different considerations when one
considers the research on the use of incentives in education. Work in eco -
nomic theory analyzes the likely effect of incentives that are structured in
different ways. These differences tend to be defined in concrete terms—
who receives what consequence under what conditions. In contrast, much
of the recent work in psychology about incentives analyzes subtleties in
the different ways that incentives are framed and communicated.
When the committee looked at the ways that incentives have been
used in education, this difference in the types of features considered by
economics and psychology affected our ability to identify the relevant
contrasts in existing educational research. In the large-scale applications
OCR for page 33
33
BASIC RESEARCH ON INCENTIVES
of incentives that we review, we are able to note interesting and poten -
tially important contrasts in the types of features suggested by economic
theory—who gets what, when—but we are not able to note interesting
contrasts in how the incentives are framed and communicated to the
people they seek to influence. As a result, our synthesis of the basic
research provides more detail about the findings from economic theory
because we are able to use that detail to guide our review of the research
on applications of incentives in education (see Chapter 4).
As the research in economic theory discussed in this chapter shows,
at least four key elements need to be carefully considered in designing
incentive systems: who is targeted by the incentives, what performance
measures are used, what consequences are used, and what support is
provided.
Target In complex organizations, incentives can be designed for peo-
ple in different positions who can affect outcomes in different ways.
Although the effects of incentives will be transmitted informally through
the organization, that transmission will likely be imperfect. It may be
important to target direct incentives to the people who can make the
changes needed to improve outcomes.
Performance Measures The performance measures used with incen-
tives have to be aligned with the desired outcomes for the incentives to
have their desired effect. In particular, the measures need to be chosen
so that behavior that increases the measures also increases the desired
outcomes.
Consequences The size and structure of the consequences provided
by the incentives will affect how the incentives operate and should be
designed to be appropriate to the situation.
Support Incentives will typically have different effects for different
people who have different abilities to successfully reach the target that
will provide a reward or avoid a sanction. Without resources in support
of an organization’s objectives, incentives can be discouraging to the very
people they are often intended to help, particularly if those people do not
have the capacity to be successful.
The literature in psychology is not inconsistent with the four ele-
ments above—and much early psychological research focused precisely
on such structural features of incentives—but much recent work in psy-
chology about incentives has focused on a different element that has
OCR for page 34
34 INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION
proven to be important in many settings: how incentives are framed and
communicated.
Framing and Communication In most organizations, the commit-
ment people have to the organization’s mission is a critical part of their
motivation. Incentives need to be framed and communicated in ways that
reinforce that commitment—rather than erode it—by emphasizing the
information they provide on progress toward shared goals.
In the chapters that follow we look at the use of test-based incentives
in education through the lens of these key elements, with the caveat that
we are able to say nothing about interesting contrasts in how incentives
have been framed and communicated.
The research in economic theory discussed in this chapter also raises
two important points related to evaluating the success of incentive sys -
tems, nonincentivized performance measures for evaluation and weigh -
ing costs and benefits.
• Nonincentivized performance measures for evaluation: Incentives will
often lead people to find ways to increase measured performance
that do not also improve the desired outcomes. Because of the
resulting distortion in the performance measures used with the
incentives, it is usually necessary to find different performance
measures—that are not being used in the incentives system—to
use when evaluating how the incentive system is working.
• Weighing costs and benefits: Incentive systems will typically gener-
ate a mix of costs and benefits that must be weighed against each
other to determine the net value of the system. The costs will
include monetary costs associated with running the system itself,
as well as nonmonetary costs, such as the negative reactions of
people who are left out of rewards or sanctioned under the incen-
tive rules.
Again, the literature in psychology is not inconsistent with these two
points from economics about how incentive systems should be evaluated,
but the focus of recent work on psychology related to incentives speaks
more directly to another point, changes in disposition.
• Changes in dispositions: In addition to evaluating the changes in
a set of defined objective outcomes, it is important to consider
the way incentive systems affect people’s dispositions. No mat-
ter how broadly an incentives system is designed, in most situ-
OCR for page 35
35
BASIC RESEARCH ON INCENTIVES
ations we are likely to care about a broader range of outcomes
than could be measured by available performance measures and
rewarded by feasible incentive systems. With respect to these
broader outcomes, it is important to know how incentives change
the way people are disposed to act when they are not being
directly affected by the incentives.
We consider these points further in Chapter 4 in the context of our discus-
sion about the use of test-based incentives in education, with the caveat
that we are able to say little about how these incentives have changed
people’s dispositions to act when they are not being directly affected by
the incentives.
OCR for page 36
