|
Items in cart [0] |
Questions? Call 888-624-8373 |
HARDBACK + PDF HARDBACK PDF BOOK PDF CHAPTERS Free Resources |
||||||||||||||||
|
||||||||||||||||
|
|
|||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 127
Determining Optimal
Types of Incentives
ver the years of the All-Volunteer Force, various incentives have
been developed and offered to help strengthen and shape mili-
~_ tary enlistments. The effectiveness of these incentives has been
addressed and demonstrated, using a variety of evaluation approaches.
This chapter considers methodological issues in determining the optimal
types of enlistment incentives for specific recruiting goals. We begin by
reviewing different types of enlistment options. Next, we consider the
various types of effects the incentives may produce, both intended and
unintended, and the related methodological issues in assessing their im-
pact. We continue with a brief review of the analysis and methods dis-
cussed in the earlier chapters, which is followed by a longer consideration
of analytical issues applicable to each of the evaluation approaches. Last,
drawing on each of these areas, we conclude with a discussion of match-
ing potential incentives and their effects to the appropriate assessment
goals and evaluation methods.
TYPES OF ENLISTMENT INCENTIVES
Enlistment options may be grouped in various ways. For example,
they can be distinguished in economic terms by whether they provide a
cash incentive, cash in kind, or some other feature valued by potential
enlisters, such as conditions of service. An alternative is to group incen-
tives according to the segment of the recruiting market they are intended
to attract, such as youth interested in attending college or those looking
127
OCR for page 128
128
EVALUATING MILITARY ADVERTISING AND RECRUITING
for job training. In the discussion that follows we use a mix of these two
approaches. We do so because that classification scheme lends itself directly
to our discussion of the possible effects of these incentives, appropriate
assessment goals, and evaluation methods.
Youth also enlist for intrinsic reasons, such as patriotism. Matching
recruiting messages and strategies to such motivations was discussed
earlier. In developing or assessing the effectiveness of such strategies,
considerations analogous to those discussed for the options considered in
this chapter apply. More generally, enlistment options can be categorized
based on social-psychological or economic theory according to their intended
effects on the determinants of enlistment behavior, and their success in
achieving these effects can be assessed as part of the evaluation process
for the options.
One group of incentives consists of cash enlistment bonuses. These
bonuses are offered for various purposes: increasing enlistments; channel-
ing recruits into hard-to-fill occupational specialties; encouraging recruits
to report for active duty quickly; encouraging enlistments or accessions
during the seasons of the year, such as late winter and early spring, that
pose chronic challenges; encouraging enlistments among persons who
served previously but left the military; and so forth.
Military pay increases also are used to increase enlistments. Such
increases are less targeted or flexible than bonuses and are intended to
provide an overall enlistment stimulus. Compensation increases also can
be provided through changes in other benefits, such as housing or sub-
sistence allowances.
Another group of options consists of incentives provided to encour-
age youth interested in postsecondary education to see military service as
a facilitator of their educational aspirations, rather than as an alternative
career option. (As discussed in Chapter 2, this amounts to changing their
beliefs about the consequences of enlisting compared with options in the
civilian sector, and it can also change the analogous views of their key
influencers.) Such education incentives include credit for college courses
completed prior to accession (such as credit toward meeting military
training requirements); near-term financial incentives, such as entry at an
advanced pay grade, student loan repayment, or a bonus; funding for
college after service, for example, through the Montgomery GI Bill or the
Service's college fund; money for attending college before service, through
options such as the Army's College First program (currently being pilot
tested); or in-service continuing education programs, in which classes are
provided on site or through distance learning.
A fourth type of incentive is directed specifically at improving the
recruit's job opportunities. Included in this group are such options as the
OCR for page 129
DETERMINING OPTIMAL TYPES OF INCENTIVES
129
Army's Partnership for Youth Success (PaYS) program and the GED Plus
pilot program.
Another group of incentives involves choices concerning the condi-
tions of one's military service. Included in this group are such options as
the term of enlistment, location of assignment, and combination of active
and reserve forces duty, among others.
Of course, the Services can and do use specific enlistment options in
combination with others. One example is the combination of enlistment
bonuses with postservice educational benefits. Another is the combina-
tion of assistance for attending college classes with financial compensa-
tion or training credit for having done so.
POTENTIAL EFFECTS OF INCENTIVE
To structure the analysis of the effects of a current or prospective
enlistment option, it is important to first address several key issues. These
issues include, but are not limited to, the types of questions highlighted
below.
· Who is the target of the option? For example, is the option intended
to increase overall enlistments or specifically those among highly qualified
youth (high-aptitude high school graduates)? Similarly, are there key sub-
groups of interest in the target population? If the incentive is intended to
attract college-bound youth into the military (by encouraging them to see
military service as a facilitator of their educational aspirations, rather than
as an alternative career option), is it aimed at college graduates, current
students, college stopouts (those who attended some college, have not yet
graduated, but are not currently enrolled), high school graduates who
have not yet attended college, high school seniors interested in college, or
some combination of these subgroups?
· What are the desired effects of the option? These can include such
outcomes as increased supply through increased enlistments or market
expansion (or improved recruit performance through increased supply
among high-aptitude youth), skill channeling into hard-to-fill or new spe-
~As is true for postsecondary educational incentives, the PaYS program is intended to
change youths' beliefs about the consequences of enlisting compared with options in the
civilian sector in this case by providing job training that can be applied to the civilian
sector and helping the youth to be considered for employment in a related job after service-
and it can also change the analogous views of their key influencers. Among its other goals,
such as increasing training opportunities, GED Plus is designed to facilitate enlistment
among high school dropouts with positive propensity while they achieve the qualifications
needed to serve on active duty.
OCR for page 130
130
EVALUATING MILITARY ADVERTISING AND RECRUITING
cialties, increased man-years, increased affiliation with the reserve forces
following active duty, cost savings, and so forth, singly or in combination.
· What are possible unintended or undesirable effects of the option
(for example, on such matters as the quality of enlistments, term of service
distribution, attrition, or retention)?
· What are the potential timing and magnitudes of the desired and
unintended effects?
· What are the option's potential costs? This includes such consider-
ations as the direct cost for new recruits, overall cost including economic
rent, (discounted) future costs, training costs, costs or savings to the
reserve forces?
· How much do we know about the types and specific features of
incentives that might generate effects analogous to those desired from the
option? Does the option differ in important ways from past incentives for
which we have enlistment data?
We consider each of these issues in turn.
Who is the target of the option? Enlistment incentives are typically
targeted according to the qualifications of the recruit and the occupa-
tional specialty in which he or she enlists. Qualifications often refer to
whether or not the recruit has a traditional high school diploma and the
level of his or her (written) aptitude score on the Armed Forces Qualifica-
tion Test (AFQT). Youth with diplomas who score in the upper 50 percen-
tile on the AFQT are considered to be highly qualified and are eligible for
the largest incentives (subject to meeting the incentives' other require-
ments). Alternatively, an option such as GED Plus may be targeted to the
non-high school graduate market.
Within the target population as a whole, there may be subgroups of
special interest. This heightened interest may reflect a Service's desire to
improve its recruiting performance for a particular youth subgroup. For
example, in the case of GED Plus, one such subgroup consists of Hispanic
youth. Alternatively, there may be heightened interest in specific sub-
groups because the subgroups require different marketing or recruiting
activities to effectively employ the option. In the case of incentives designed
to appeal to college-bound youth, as noted, such subgroups may consist
of high school seniors, high school graduates without college, college
stopouts, youth currently enrolled in college, and college degree holders.
Another relevant consideration concerns the option's potential appeal to
the targeted youths' key influencers or referents (see Chapter 2~.
What are the desired effects of the option? The incentive can be
intended to affect a number of domains. These include, but are not limited
to, increasing the enlistment rate among subgroups of the youth popula-
OCR for page 131
DETERMINING OPTIMAL TYPES OF INCENTIVES
131
lion who currently are enlisting; expanding enlistments to other sub-
groups; channeling enlisters into specific military occupational specialties;
increasing the number of years served by the typical enlistee; increasing
affiliation with the reserve forces following active duty; or reducing
recruiting costs. Defining the desired effects of the option is central to
defining the outcome measures needed to assess its effectiveness.
What are possible unintended or undesirable effects of the option
(on such matters as the quality of enlistments, the term-of-service dis-
tribution, attrition, or retention)? In addition to its desired effects, the
option may have unintended consequences, which may be significant and
thus require assessment. For example, an incentive designed to increase
enlistments among high-aptitude GED holders should not do so at the
expense of enlistments among high-aptitude high school graduates, which
is the primary recruiting market. Alternatively, youth joining the military
under an incentive designed to expand the recruiting market by offering
shorter terms of service should not be those who would have enlisted for
longer terms under existing programs. When incentives have potentially
significant adverse outcomes, the unintended effects should be assessed
in the evaluation together with those desired. Options may also have
more subtle unintended consequences that nonetheless are very impor-
tant to address in assessing the options' efficacy. For example, to the
extent that the incentive increases youth-propensity to enlist for a particu-
lar market or subgroup, recruiters may decrease the effort they expend
for that market or for other subgroups (increase environmental con-
straints). This is because recruiters may not need to leverage the full
potential of the incentive in order to meet their recruiting goals and thus
can target their efforts. This can lead to underestimating the option's
actual potential to increase enlistments or expand the market, and it needs
to be addressed specifically in the analysis.
What are the potential timing and magnitudes of the desired and
unintended effects? The timing of the option's effects has a direct bearing
on the period of the evaluation. Options designed to increase accessions
into the military in the near term may require a relatively short assess-
ment period, whereas options designed to increase man-years or reserve
forces affiliation may require that enlisters under the option be followed
up through their first term of service (or even longer). Still other options
designed to increase accessions may do so only over a longer period of
time. Incentives provided to allow youth to attend college classes follow-
ing enlistment but prior to entering active duty fall into this category and
thus require a longer evaluation period to cover the increased time between
the enlistment and accession points. A longer period between enlistment
and accession would be expected to lead to higher loss rates among
enlisters before their reporting for active duty. As a result, capturing such
OCR for page 132
132
EVALUATING MILITARY ADVERTISING AND RECRUITING
loss rates becomes an important component of the analysis. Options that
increase enlistments in the near term may also have longer term conse-
quences. Postservice educational benefits might be expected to increase
enlistments but also to increase separations if the recipients leave the
military at a higher than average rate in order to attend college.
The anticipated magnitude of the incentive's effects has a direct bear-
ing on the statistical power of the analysis. Consequently, it affects the
number of observations required to make a reasonable assessment of the
viability of the option. This, in turn, can affect the number of persons to
whom the option is offered as well as the required duration of the
assessment.
What are the option's potential costs (such as the direct cost for new
recruits, overall cost including rent, discounted future costs, participa-
tion rates, training costs, reserve forces costs or savings)? Enlistment
options that have relatively small elasticities in increasing enlistments can
nonetheless be very cost-effective recruiting tools depending on their cost.
Conversely, the utility of incentives with high elasticities can be limited
by their cost. To the extent that the costs of an enlistment incentive are of
interest, they need to be captured in the evaluation. This can include the
costs (and effects) of alternative values of the incentive or of variations in
the number of specialties for which it is offered. The cost of providing the
option to qualifying enlisters typically also includes costs in the form of
"economic rent." This applies to the payment of the incentive to recruits
who would have enlisted even in its absence (a cost we seek to minimize
relative to the cost incurred to increase enlisted supply).2 A related cost
concerns the impact the option may have on enlistments under other
incentives and the resulting overall net cost increase or savings.
Incentives that are paid out only after a considerable period of time,
such as postservice educational benefits, have true costs well below their
nominal value, unlike enlistment bonuses. Thus, their cost needs to be
discounted in light of the delay in providing them, and the eventual usage
rates for the incentive need to be considered. This can include both the
proportion of eligible recipients who use any part of the incentive as well
as the proportion of their entitlement they eventually use. Enlistment
incentives can also increase or reduce other types of military costs. For
example, an enlistment option combining active and reserve forces duty
can reduce reserve forces recruiting costs by increasing affiliation rates
2The benefits considered in the rent calculation can include more general benefits to
society provided by the incentive, in addition to increased enlisted supply. An incentive
that increased college attendance or provided civilian-related job training could be expected
to improve the employment opportunities and earnings of the recipients (and the taxes
paid by them) over the longer term.
OCR for page 133
DETERMINING OPTIMAL TYPES OF INCENTIVES
133
from the active force into the reserve forces and training costs by encour-
aging affiliation in the occupational specialty the soldier held while on
active duty. Further, enlistment options can affect attrition and retention
rates, which have implications for recruiting and training costs.
How much is known about the types and specific features of incen-
tives that may generate effects analogous to those desired from the
option? Does the option differ in important ways from past incentives
for which we have enlistment data? The extent to which a prospective
option differs from current and past options has a direct bearing on the
requirements for new data collection to assess its effectiveness. At one
extreme, consider a new enlistment bonus whose value lies between the
values of enlistment bonuses already offered. In such a case, it is very
likely that existing data could be used to estimate the effects of offering
the new incentive. Alternatively, consider the case in which a prospective
enlistment option differs in important ways from those currently avail-
able in its basic features or in the portion of the youth market to which it
is targeted. An example would be a program with unique features, such
as PaYS, or a program designed to attract youth who would not otherwise
have joined the military, for example, by providing programs to help
them attend college between the enlistment and accession points.
EVALUATION METHODS AND ISSUES IN THEIR USE
Four major types of evaluation approaches are commonly used in the
development and assessment of recruiting incentives:
· Focus groups,
· Surveys,
· Econometric (regression) analyses of existing data, and
· Experimental or quasi-experimental assessments.
Each of these approaches has advantages and limitations. Focus
groups lend themselves especially well to exploring in depth the attrac-
tive and unattractive features of a limited number of prospective or cur-
rent incentives (or to helping develop new types of options), whereas
surveys can be especially useful in more general assessments of the rela-
tive attractiveness of a set of specific alternative options to the recruiting
population as a whole or among particular market segments. Regression
analyses are especially helpful in assessing enlistment rates under current
(or past) options and in understanding the possible effects of trade-offs
among the options or of limited adjustments to the options. When we
need to estimate enlistment rates under prospective options that differ
significantly in their features or market segment from current and past
OCR for page 134
134
EVALUATING MILITARY ADVERTISING AND RECRUITING
options, then experimental or quasi-experimental designs can be particu-
larly useful.
We next consider the strengths and weaknesses of each approach in
detail. We note here that each approach can have both unique advantages
and limitations and this often leads to the use of multiple assessment
methods in the evaluation of recruiting options.
Focus Groups
The use of focus groups is especially helpful for shaping prospective
recruiting programs by enabling the researcher and policy maker to
explore the appeal of a range of incentives or enlistment options to youth
or their influencers, including programs currently available to recruits as
well as prospective options. Moreover, the focus groups can provide
insight into the particular strengths and weaknesses of such programs,
including the usefulness and appeal of specific features of the options and
the extent to which that appeal varies across key subgroups of the recruit-
ing market. In so doing, the focus groups can facilitate the discovery and
exploration of new option features or effects that may be helpful in achiev-
ing military recruiting goals. A further advantage of this approach is that
the information can be gathered relatively quickly and inexpensively.
The use of focus groups is not without limitations. The groups must
be carefully constructed to represent the population of interest and to
facilitate open discussion. Groups can be too large to generate open dis-
cussion, can be affected by social desirability concerns or other response
biases, or can be dominated by a small number of especially vocal partici-
pants (or by the group leader). Even if these potential obstacles are over-
come, the groups still have only a limited ability to provide information
that is generalizable to the full population of interest, because of the
limited number of participants and the semistructured nature of the dis-
cussions. This is particularly true when the purpose of the evaluation is to
estimate effects of prospective alternative incentives or options such as
the preference rates for one option over another or actual enlistment rates.
Surveys
Surveys also can provide rapid and reasonably inexpensive assess-
ments of existing or prospective enlistment options among youth or their
influencers. Because they are administered individually and, moreover,
the questions can vary across individual respondents, they can be con-
structed to cover more ground than what can normally be addressed
using focus groups. This is particularly true for surveys administered
using computers, such as computer-assisted telephone interviews or web-
OCR for page 135
DETERMINING OPTIMAL TYPES OF INCENTIVES
135
based surveys; in such surveys, an individual respondent's answers to
specific questions as he or she proceeds through the survey can be used to
determine which succeeding questions he or she is asked. For example,
surveys are ideal to assess potential interest of youth in military service;
can be designed to facilitate comparisons of the attractiveness and effects
(among youth or their influencers) of a substantial number of specified,
alternative recruiting options; can be designed to readily assess the varia-
tion in propensity and in the effects of such incentives for key youth
population subgroups of interest; and can be designed to explore the
covariation of enlistment interest and the appeal of various incentives (or
their specific features) with a wide variety of other factors, such as demo-
graphic factors, educational aspirations, job and career goals, reasons for
enlisting and barriers to military service, discussions with influencers,
contacts with the military, and awareness of current military benefits.
Questions dealing with such matters as media preferences, Internet use,
and recreational habits can also be included to provide insights into the
most effective ways to market enlistment incentives, both overall and to
specific subgroups.
As is true for focus groups, there are important issues and limitations
in the use of surveys to assess the effects of alternative enlistment incen-
tives on military recruiting. Drawing firm conclusions regarding the effi-
cacy of alternative enlistment options requires that the sample must be
representative of the target population, that the sample size must be ade-
quate, and that a sufficient proportion of the surveys are completed and
returned. Determining the appropriate sample size involves decisions
about the acceptable confidence intervals surrounding the key point esti-
mates and differences (in responses to specific items) to be provided by
the survey; the statistical variances of the corresponding items; the frac-
tion of respondents that will be asked to answer the key questions; the
estimated completion rate for the survey; and the acceptable level for the
statistical power of the analysis to detect true differences.
Even when the sample is adequate and representative, care still must
be taken in interpreting survey results concerning the prospective effects
of alternative enlistment options. Although statistically significant rela-
tionships between one's stated likelihood of enlisting in such surveys as
the Youth Attitude Tracking Study (YATS) and Monitoring the Future
(MTF) and one's actual enlistment decision have been demonstrated, it
also is true that respondents' misestimate their true enlistment probability
(Bachman, Segal, Freedman-Doan, and O'Malley, 1998; Orvis, Gahart,
and Ludwig, 1992~. Youth seeing themselves as very likely to enlist are, in
fact, less likely to actually join the military than they believe, whereas
youth seeing themselves as very unlikely to enlist are more likely to do so
than they believe. The accuracy of a respondent's stated enlistment inten-
OCR for page 136
136
EVALUATING MILITARY ADVERTISING AND RECRUITING
lions varies with a number of factors, such as the temporal proximity of
the enlistment decision point and the underlying rate of enlistment in the
respondent's demographic group (normative support). When there is a
considerable time period between the intention assessment and actual
enlistment decision point, for example, changes in the respondent's life,
in world events, in the economy, or in the information obtained through
discussions with key influencers can alter the youth's enlistment inten-
tions. This also limits the ability of surveys of the youth population to
capture the possible longer term costs and effects of alternative enlist-
ment policies, such as the incentive's effect on attrition and retention,
even if the survey includes questions concerning such effects.
Past research also suggests that youth overestimate their true respon-
siveness to changes in enlistment options. While surveys provide an
excellent methodology to assess the relative effects of alternative enlist-
ment incentives, obtaining good estimates of the true enlistment rates
under the options is likely to require additional data. Moreover, the spe-
cific details of the information provided to respondents concerning cur-
rent or prospective enlistment options are important and can influence
their answers; this involves matters both of question wording and order,
as well as social desirability concerns and other response biases. Finally,
surveys of enlistment interest in the youth population (the supply side)
do not capture the potential behavior of recruiters and recruiting man-
agers as the recruiting market changes or new incentives are introduced
(environmental facilitators or constraints the demand side); this again
limits the ability of survey data to provide accurate estimates of the true
enlistment rates that would be obtained under alternative options.
Econometric Models
As discussed in some detail in Chapter 5, econometric approaches
rely on existing data and thus can provide rapid and inexpensive assess-
ments and comparisons of policy alternatives. They also can provide
statistical controls for confounding factors such as historical differences
in enlistments across recruiting areas or the effects of changes in the
economy and can allow assessment of incentive effects for subgroups of
the youth population. Econometric models can assess the effects and costs
of enlistment options over time and in different recruiting environments.
They provide data on actual enlistment rates and can directly provide
data on the magnitudes of unintended effects; such effects can include
market substitution the substitution of enlisters brought in under a pro-
gram for other types of enlisters or for enlistments under other options
that would have occurred in the absence of the program and can be
used to correct for changes in demand-side behaviors (environmental
OCR for page 137
DETERMINING OPTIMAL TYPES OF INCENTIVES
137
constraints or facilitators) that may mask the true potential of an option to
increase enlistments or expand the recruiting market (Buddin, 1991;
Polich, Dertouzos, and Press, 1986~.
Although their reliance on existing data offers advantages of speed
and cost in assessing the efficacy of alternative recruiting options, it limits
the usefulness of econometric models in shaping and assessing prospec-
tive options. Because the models are limited to assessment of previous or
current recruiting incentives, the possible effects of new options that differ
in important ways from such incentives in their features or the market
segments to which they are intended to appeal cannot readily be
assessed. Moreover, the accuracy of econometric models in forecasting
enlistment rates can be limited by a variety of other factors that merit
careful attention by the analyst. These include, but are not restricted to,
the often limited amount of enlistment variation they may explain, espe-
cially at the individual (versus cross-sectional) level or over time; the
possible misspecification of the model (for example, due to the inclusion
of an insufficient set of predictors or use of an inappropriate functional
form); failure to capture important recruiter mission-setting or manage-
ment practices that affect enlistments; failure to capture major shifts in
propensity that may occur over time; and the possible correlation between
supply- and demand-side variables in the model, such as changes in
resources in response to changes in enlistment propensity. If recruiting
resources are increased in response to a decline in youth's propensity to
enlist and it then takes a while for the resources to improve the recruiting
environment, one could underestimate the efficacy of the resources or, in
the extreme, obtain incorrect signs for their regression coefficients. Simi-
larly, if recruiters reduce their effort in response to the efficacy of an
incentive in increasing enlistment propensity, the potential effects of the
option on increasing enlistments or creating market expansion can be
underestimated.
Experimental and Quasi-Experimental Designs
Use of experimental or quasi-experimental designs to evaluate the
potential effects of enlistment options offers several advantages. The
recruiting effects of prospective enlistment incentives can be quantified at
low risk compared with full implementation (e.g., by avoiding unneces-
sary costs or the difficulties of subsequently withdrawing an established,
national incentive). Moreover, the effects of alternative prospective poli-
cies or of alternative versions of a prospective policy can be directly
assessed and compared. The effects of potential confounding factors in
assessing the efficacy of recruiting programs which need to be statisti-
cally controlled after the fact in econometric models can be directly and
OCR for page 138
138
EVALUATING MILITARY ADVERTISING AND RECRUITING
greatly reduced through balancing of the test areas offering new pro-
grams and the control areas not offering the programs on factors that are
known to affect enlistment rates. These include, for example, geographical
location, local economic conditions, demographic characteristics, past
recruiting production, local recruiting missions, such recruiting resources
as the number of recruiters in the area, and the alternative enlistment
options that may be available locally, particularly to the extent that other
Services or reserve components offer related programs (for example, alter-
native financial assistance programs for college attendance).
By employing an experimental design that compares (1) changes in
enlistments in the test areas under the new option relative to enlistments
in the same areas during a pretest baseline period with (2) the trend in
enlistments in the control areas not offering the new program over the
analogous time periods, the approach provides a direct control for tempo-
ral and fixed differences in enlistment rates. Experimental and quasi-
experimental designs can also directly provide data on the magnitudes of
unintended effects; as noted, such effects can include market substitution
or changes in recruiter effort or time allocation in response to the change
in enlisted supply created by the new incentive (such as reduction of
effort as supply increases).
Given a test period of sufficient duration, the experiment's interim
results, concerning both desired and unintended effects, can be used to
recommend adoption of the test program or to provide the basis for
changes in the incentive or in recruiter mission for the incentive that are
designed to enhance its efficacy. The remainder of the test period can then
be used to evaluate the performance of the modified program.
Use of experimental or quasi-experimental designs to test the poten-
tial effects of alternative options is not without potential problems. Cer-
tain recruiting incentives, such as military pay increases, simply do not
lend themselves to pilot testing; it is not practicable to limit them to cer-
tain areas of the country or to undo them if their effect is not as desired. In
cases in which legal authority is required to test prospective options,
obtaining such authority may require considerable time, and the specifics
of the enabling legislation can compromise the test's results. Similarly, the
specific details of the incentive's test implementation such as the pro-
portion of occupational specialties eligible for the incentive have a direct
bearing on the estimates of the enlistment effects of the incentive that will
be derived from the experiment.
Such issues require careful attention in the analysis, particularly if
comparisons with other incentives that differ in such characteristics are
desired (see, for example, Buddin and Roan, 1994~. Additional resources
for the test programs may also be needed, particularly if the experiment is
to continue over an extended time period. A related issue concerns the
.
.
OCR for page 139
DETERMINING OPTIMAL TYPES OF INCENTIVES
139
feasibility of conducting a stable test in the operational environment, par-
ticularly if recruiting conditions improve or worsen significantly during
the test period or if the experiment suggests the usefulness of the pilot
program and other recruiting areas (or Services) decide they want it right
away.
Moreover, there are certain inherent limits when experimental or
quasi-experimental designs are used to assess the effects of prospective
enlistment options. These include the fact that the program attributes that
can be evaluated often are more aggregated or limited in number than is
the case for some of the other evaluation methods; this can occur due to
resource constraints, policy-based limits on enlistments under the pilot
program, or statistical requirements to support the reliability of the test
results, among other factors. Also, when new programs are tested in an
experiment as compared with econometric analyses of alternative
options using existing data, for example their overall cost-effectiveness
may be unknown for years; this is because such a determination requires
assessment not only of enlistments which can be measured in the rela-
tive near term experimentally but also of patterns of attrition, retention,
and job performance, which take much longer to unfold. Finally, no matter
how well balanced the test and control cells may be at the outset of an
experimental assessment, the test does not control for differing inter-
temporal changes across the cells, such as those that could occur due to
changes in local economic conditions or changes in demand-related
factors, such as recruiting behavior or resourcing.
Combinations of Evaluation Methods
Sometimes a combination of evaluation methods is used to assess the
efficacy of alternative enlistment options. One approach that has been
used in the past is to combine econometric analysis with the experimental
approach in order to help preserve the balance of the test and control cells
by adjusting statistically for unexpected, differing intertemporal changes
across the cells.
The other methodologies also can be used in combination. For exam-
ple, focus groups may be used to help explore and define prospective
enlistment options that are then tested in large youth market surveys or in
pilot tests employing experimental designs (or, conversely, focus groups
or surveys are used to explore the effects of policy experiments). Another
example is the application of econometric methods or other analytical
techniques to existing datasets in order to help interpret survey results.
The other approaches could be used to derive a metric to help translate
stated enlistment intentions in the survey into estimated enlistment rates.
OCR for page 140
140
EVALUATING MILITARY ADVERTISING AND RECRUITING
ASSESSMENT OF INCENTIVE EFFECTS
In the final section of this chapter, we provide examples of designing
appropriate research to help determine the optimum typed of enlistment
incentives for a given policy goal by matching the desired effects of the
potential incentives, the corresponding assessment goals, and evalua-
tion methods.
Consider the case in which the Office of the Secretary of Defense
(OSD) or an individual Service would like to expand its recruiting efforts
into a new or largely untapped market and is interested in developing
enlistment incentives to do so. To the extent that the market is new or
exploratory research on new options is desired, econometric modeling
using existing enlistment data will not be especially helpful in choosing
optimal incentives for recruiting this market. Experimental or quasi-
experimental approaches would limit the number of options that could be
tested, due to analytical, cost, and legislative considerations, among other
factors, and thus are premature. The policy maker also lacks sufficient
information at this point to design a survey to gather detailed information
on the areas of interest. Focus groups, however, are well suited for this
purpose.
The Army's interest in materially improving its recruiting perfor-
mance among youth who are committed to attending college after high
school but would consider enlistment subsequently provides an example.
Over the past 20 years, the proportion of youth attending college within a
year of high school graduation has risen dramatically, from about half to
approximately two-thirds (U.S. Bureau of the Census, 2000, 2002~. How-
ever, the military has lagged in adjusting its recruiting strategy and still
recruits the vast majority of its enlisted force from youth who have not
attended college. In recognition of this mismatch, over the past several
years the Army has employed policies that give greater emphasis to
recruiting youth from the college market (which includes students,
stopouts, and high school seniors). As part of that effort, it has conducted
focus groups with youth in the college market to better understand their
reasons for or against enlisting and the types of enlistment options they
would find appealing. Similarly, OSD recently sponsored a study of the
college-bound youth population concerning their enlistment interest and
the types of incentives that would stimulate that interest (Kilburn and
Asch, 2003~. The initial step in the research consisted of a series of focus
groups with each of the subgroups listed above; this helped OSD define
and shape incentives that appeared to be promising for further investigation.
Next, consider the case in which a policy maker wishes to understand
in some detail the relative appeal of an array of alternative recruiting
options and wants to be able to generalize the information to the youth
OCR for page 141
DETERMINING OPTIMAL TYPES OF INCENTIVES
141
population. The options may include current incentives, prospective
enlistment incentives, or both. The details of interest, for example, may
include the usefulness of the incentives for key youth subgroups, such as
high aptitude youth and college stopouts; those potentially interested in
military service for various reasons, such as job training, assistance for
attending school or taking a break from work or school; or those with
specific barriers to military service. If the policy maker wants to compare
current and prospective incentives, it may also be helpful to assess aware-
ness levels in the subgroups of interest for the significant current options.
To the extent that research on new options is desired or the youth
population characteristics of interest are not captured in the military's
enlistment databases for example, starting and then stopping out of col-
lege before completing a year, recruits' reasons for enlisting, the level of
support for enlisting among their influencers, or youths' awareness levels
of specific options existing enlistment data will not be especially helpful
in choosing optimal incentives. Experimental or quasi-experimental
approaches would again limit the number of options that could be tested,
and they might not identify key subgroups of interest without additional
data collection procedures. Given the policy maker's interest in system-
atic information that can be generalized to the youth population, focus
groups lack the required coverage and rigor. In such instances, youth
surveys can be used very effectively to address the issues of concern. In
the OSD study just discussed, the incentives developed using focus groups
were included in a subsequent survey of college-bound youth that pro-
vided a variety of detailed, generalizable enlistment-related information,
including the incentives' potential effects on enlisted supply. Similar con-
siderations apply to the use of influencer focus groups and surveys.
As discussed earlier in this chapter, surveys must sample and obtain
data from a sufficient number of respondents representative of the popu-
lation and subgroups of interest. Care also must be taken in the wording
and sequencing of the questions included to represent the enlistment
options. The results will be best suited to understanding the relative effects
of the alternative incentives. Using propensity measures for which data
exist to link responses to enlistments will help to meaningfully interpret
the responses to the options in the survey by providing an enlistment
index value for each option and subgroup. Even so, past research has
shown that respondents misestimate their true enlistment probability to
some extent, and there also is the possibility of changes in demand-side
factors, such as recruiter behavior in response to new markets or options.
Thus, the index values do not equate to the actual enlistment rates that
will be obtained under the options. There are various analytical approaches
that can be used to construct the samples, survey questions, and enlist-
ment indexes (see, for example, the Youth Attitude Tracking Study, 1976-
OCR for page 142
142
EVALUATING MILITARY ADVERTISING AND RECRUITING
1999 Wilson et al., 2000; Kilburn and Asch, 2003; Orvis and McDonald,
forthcoming).3
A policy maker's interest in determining the optimal types of enlist-
ment incentives may involve wanting to optimize the cost-effectiveness
of the mix of current options or to get the greatest return on investment
for additional recruiting resources that can be made available in the near
term. By analogy, the issue could involve wanting to reduce a particular
type of resource for example, recruiters and wanting to know how to
best offset that reduction with alternative resources. Answering these
types of questions quickly and with confidence requires estimates of
actual enlistment rates, costs, and elasticities, based on known past results.
Econometric modeling is ideal in such cases (see, for example, Warner,
Simon, and Payne, 2001; Buddin and Roan, 1994; Asch and Orvis, 1994~.
Using econometric modeling to address such questions also has the
advantage of enabling the policy maker to examine in advance the prob-
able longer term consequences of making such allocations and trade-offs
on such factors as attrition and retention.
3The purpose of an enlistment index is to allow the prospective recruiting effects of alter-
native options to be compared more meaningfully than would be possible simply by com-
paring the distributions across an intention measure generated by the options. There are
various approaches to constructing such enlistment indexes. They all include linking enlist-
ment records to stated enlistment intention levels normally, from earlier surveys in order
to establish enlistment rates for the intention levels. For example, this can be done for the
individual levels (response categories) of the intention measure (e.g., definitely, probably,
probably not, definitely not or 0 to 10 likelihood of enlisting); for combinations of the
categories (e.g., positive versus negative propensity); or for composite measures that com-
bine more than one intention measure (e.g., "definitely" on any of several individual Service
measures becomes "definitely" on the composite measure, or an unaided mention of plans
to enlist forms the top category, followed by positive propensity without an unaided men-
tion, and so forth). Various follow-up periods can be used to determine the enlistment
rates such as one year, lifetime, etc. depending on one's purpose (and the ages or school
year of the respondents). It is important to match the intention measures to be used in the
new survey, the sample composition (e.g., market segments, rules on inclusion of youth in
the delayed entry pool), and the enlistment period of interest to those used in the earlier
follow-up work to generate the index. The enlistment index value can simply be the enlist-
ment rate found in the enlistment records check for respondents stating the given intention
level, or it can be transformed or made conditional on other factors (e.g., through inclusion
in a regression analysis of enlistments) to help control for the effects of other factors on a
person's enlistment decision, which may not be captured fully by the intention measure. It
may also be helpful to use the index values to compute a measure of the options' appeal
relative to each other, by scaling the index values relative to the most (or least) appealing
option (i.e., that which most (or least) increases stated propensity to enlist in the survey).
Finally, the index values may be calculated independently or relative to that for the
respondent's (baseline) likelihood of enlisting under current options.
OCR for page 143
DETERMINING OPTIMAL TYPES OF INCENTIVES
143
The data collected from focus groups is not sufficiently generalizable
or precise to address these types of questions. Although surveys can be
useful in comparing the relative appeal of alternative options, they do not
provide absolute enlistment (or loss) rates, and they do not directly cap-
ture the effects of changes in such factors as the number of recruiters or
advertising levels. Finally, given the focus of these questions on trade-offs
among existing incentives and resources, using an experimental approach
would not provide results in a timely manner and would be unnecessarily
restrictive.
When the types of incentives to be evaluated are reasonably well
defined but they are new in total or vary significantly in their features
from current options, assessing their effects requires new data collection.
Collecting that information with reasonable precision beyond that
afforded by surveys or focus groups but with lower risk than through
national implementation can be accomplished using experimental or
quasi-experimental pilot tests of the programs. The general features of
such tests, as well as their strengths and limitations, were described ear-
lier. There are numerous examples of national recruiting tests, and they
cover a variety of effects of potential interest to policy makers.
For example, the Educational Assistance Test Program compared
recruiting effects for the Army, the Navy, and the Air Force of different
types and amounts of postservice educational benefits (Fernandez, 1982~.
Issues of concern included the overall effect on highly qualified enlist-
ments of the incentives; the effects of the type and value of the incentive;
and the effect of differences in the value of the incentive across the Ser-
vices. The Enlistment Bonus Experiment compared the results of bonuses
differing in dollar value and term of enlistment obligations on highly
qualified enlistments and the channeling of recruits into hard-to-fill occu-
pational specialties (Polich et al., 1986~. The 2+2+4 Recruiting Experiment
addressed the effects on active and reserve forces component enlistments
among highly qualified youth of a program allowing two years of active
duty plus training time followed by two years in the Selective (drilling)
reserve forces and four years in the Individual Ready Reserve in return
for enhanced postservice educational benefits (Buddin, 1991~. The key
effects of interest included active enlistments, changes in job choice and
term-of-service obligation, affiliation rates into the Selective Reserve, affili-
ation rates into the same occupation held while on active duty, and the
overall impact on military man-years. The 2+2+4 test used a true experi-
mental design that included random assignment of qualifying youth to
eligibility for the program.
Another example of employing a quasi-experimental design is pro-
vided by the College First/GED Plus National Recruiting Test. These
pilot programs are currently in progress (Sellman, 1999; Orvis and
OCR for page 144
144
EVALUATING MILITARY ADVERTISING AND RECRUITING
McDonald, forthcoming). The College First program's main purpose is to
assess the effect on highly qualified enlistments and market expansion of
an incentive allowing recruits to attend college for up to two years after
enlisting but before reporting for active duty. The analysis examines such
effects both overall and for key subgroups of the college-bound youth
market. Based on these analyses, it identifies useful changes in the pro-
gram and in recruiter missioning to support it. GED Plus was designed to
increase opportunities for and enlistments among qualifying youth with-
out traditional high school diplomas. To qualify, youth have to score in
the upper half of the AFQT distribution and pass strict screening criteria.
Youth without GED certificates have to obtain them through an attendance-
based course prior to going onto active duty. Key outcome measures
include the program's effects on high-aptitude enlistments among youth
without traditional high school diplomas and, in particular, among His-
panic youth, who are disproportionately represented in this market; the
impact, if any, on highly qualified enlistments (market substitution); first-
term attrition rates under the program; and the cost-effectiveness of eligi-
bility for alternative enlistment incentive levels among GED Plus recruits,
including the extent to which each level is being leveraged by recruiters.
Helping policy makers choose the optimal incentives to accomplish
their objectives and obtaining reasonable estimates of the incentives' prob-
able effects often involves the use of multiple evaluation methodologies.
We have noted such combinations in the preceding discussion. For exam-
ple, focus groups can provide a highly useful initial step in designing new
enlistment options. The main results of the groups can then be used for
follow-up assessments that provide more rigorous and generalizable
results through surveys or recruiting experiments. We also noted that
quantitative assessments linking prior survey results to actual enlistment
data can be very helpful in interpreting enlistment propensity informa-
tion collected in new surveys and in quantifying the likely enlistment
outcomes associated with alternative incentives.
As discussed earlier in this volume, econometric modeling can be
used in conjunction with experimental or quasi-experimental designs to
provide improved estimates of the potential recruiting benefits of new
options. The Enlistment Bonus Experiment, for example, used econometric
analysis to support a quasi-experimental design. The regressions con-
trolled for changes over time in economic and other conditions that could
have acted to unbalance the test's experimental and control cells. Based
on Dertouzos' approach, they also attempted to control for unobserved
reductions in recruiters' effort in response to the improvement in recruit-
ing conditions under the test options and the ease of substitution of less-
well-qualified versus highly qualified recruits.
OCR for page 145
DETERMINING OPTIMAL TYPES OF INCENTIVES
145
For illustrative purposes, we conclude with an example of using the
four evaluation methodologies in combination. The Army is currently
assessing the benefits of eArmyU, an online distance learning program
allowing soldiers to pursue courses and degrees from more than 20 col-
leges and universities and approximately 100 programs, and of soldiers'
interest in modified versions of the program. The current program pro-
vides a free laptop computer to participants, up to 100 percent tuition
assistance, and a number of other benefits, such as an Internet service
provider and free books and delivery. In return, the soldier must agree to
serve for at least three years from the eArmyU enrollment point and must
complete 12 semester hours within two years of enrolling.
The evaluation is assessing participation rates in the current program
across subgroups (using econometric modeling); retention effects of the
current program (econometric modeling and focus groups with partici-
pants in the program); and likely enrollment rates and retention effects
under alternative versions of the program some with reduced incen-
tives matched by reduced service and course completion requirements, in
order to allow more soldiers to participate within the resources available
for the program (quasi-experimental pilot test, survey of enlisted personnel,
and focus groups with participants, supervisors, and education adminis-
trators of the alternative versions of the program). The evaluation also is
assessing quality of life benefits of the current program to the soldier and
his or her family (focus groups) as well as the effects of the current pro-
gram on readiness and duty performance (focus groups).
Representative terms from entire chapter:
enlistment rates
