|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 34
3
Accuracy and Sources of Errors
Estimates of the number of people eligible and likely to participate in
WIC are useful for both budgetary and evaluative needs only to the extent
to which they are reasonably accurate representations of their "true" levels
and trends in eligibility and participation. It is not possible to observe ac-
tual eligibility for WIC in the population because eligibility is only ob-
served when an individual applies for WIC benefits. However, administra-
tive records can be utilized to construct a benchmark for examining the
accuracy of the estimates of the number of participants.
This chapter begins by accessing the accuracy of the USDA method
used to predict the number of women, infants, and children who partici-
pate in the program. For the overall population, USDA's methodology has
led to relatively accurate forecasts of the number of participants. However,
for the various subgroups of WIC eligible participants women, infants
and children the accuracy of their forecasts of the total number has been
poor. The chapter goes on to identify potential sources of error.
ACCURACY OF THE USDA METHODOLOGY
In preparing a budget request for the WIC program, USDA employs
the most current survey data from the March Current Population Survey
(CPS). As discussed in the previous chapter, the survey data for a given year
could be used to predict eligibility and participation at least four years into
the future. For example, when USDA was preparing their budget request
34
OCR for page 35
ACCURACY AND SOURCES OF ERR ORS
35
for 2002, they would have employed data covering calendar year 1998. The
accuracy of this forecast of the number of participants in 2002 depends on
two factors. The first is how well the methodology predicts the number of
participants in the year for which there are data (in this example, 1997~.
The second is the validity of the assumption that participation will be un-
changed over the forecast period.
To assess the level of prediction error in the estimates the error re-
sulting from the use of currently available data to predicti~ture eligibility
and participation Table 3-1 compares the estimated number of partici-
pants by category with actual administrative counts of participants. This
ratio was computed using data from 1992-2000. Estimates are first totaled
over all categories and then given separately by each eligibility category.]
The first line of each category contains the ratio of the estimated num-
ber of WIC participants in a given year, t, to the actual number of WIC
participants in the same year, t (labeled concurrent year ratio). Examining
the first set of ratios, which are totaled over all categories, we see that the
ratio of the estimated to actual number of participants ranges from 1.37 in
1992 to 0.86 in 2000. According to this measure of accuracy, the estimates
of the number of participants were overstated by 37 percent in 1992 but
understated by 14 percent in 2000. Patterns within eligibility categories
vary greatly, however. The ratios for infants and breastEeeding postpartum
women show a consistent underestimate of the number of WIC partici-
pants compared with administrative counts of actual participants. This un-
derestimate is getting worse over time. For infants, the ratios range from
about 0.80 in 1992 to about 0.61 in 2000. For breastEeeding postpartum
women, the ratios range from 0.99 in 1992 to about 0.49 in 2000. Ratios
for children show that the estimates initially overestimate the number of
participants but are very close to the actual number of participants from
1998 onward. Estimates for pregnant women were overstated in early years
but in subsequent years have been slightly understated. For nonbreast-
feeding postpartum women, estimates of the number of participants are
significantly understated for every year except 1992 and 1993.
These ratios do not really reflect the task USDA confronts each year
in trying to predict eligibility and participation for the year the proposed
budget is to cover. In actuality, estimates of the numbers eligible and likely
Administrative records on the number of participants may include ineligible partici-
pants. Estimates of eligibility may differ from administrative records because of this, but we
do not know how big these differences may be.
OCR for page 36
36 ESTIMATING ELIGIBILI>~D PAR TICIPATION FOR THE ~CPROG~
TABLE 3-1 Accuracy of Current Methods Used to Predict WIC
Participants: Ratios of Estimated Participants to Actual Participants,
1992-2000
1992 1993
1994
All Eligibility Categories
Concurrent year ratios:a 1.365 1.300 1.111
Forecasted ratios:6
Infants
Concurrent year ratios: 0.795 0.790 0.693
Forecasted ratios:
Children
Concurrent year ratios: 1.756 1.642 1.385
Forecasted ratios:
Pregnant Women
Concurrent year ratios: 1.270 1.115
Forecasted ratios:
NonbreastSeeding Women
Concurrent year ratios: 1.351 1.003
Forecasted ratios:
BreastSeeding Women
Concurrent year ratios: 0.993 0.770
Forecasted ratios:
All Postpartum Women
Concurrent year ratios: 1.218 0.919
Forecasted ratios:
aConcurrent ratios = Estimated Participants in year t / Actual Participants in year t.
Forecasted Ratios = Estimated Participants for year (t + 4) / Actual Participants year t.
Note: To compute the number of eligible persons who will participate, an 80 percent
. . . .
participation rate IS assumec .
to participate for a given year, t, are used to predict the number of partici-
pants forecast for year t + 4 (e.g., data from 1992 are used to predict FY
1996 participants). To take the forecasting component of the estimation
process into account, the estimated number of participants in year t are
compared with the actual number of participants from administrative
records from year t + 4 for each eligibility category and overall categories
(labeled forecasted ratio). These ratios are given in the last line of each
eligibility category.
OCR for page 37
ACCURACY AND SOURCES OF ERR ORS
37
1995 1996 1997 1998 1999 2000
1.020
0.698
1.197
0.861
0.961
1.026
0.674
0.732
1.117
1.180
1.070
1.164
0.845
0.918
0.664
0.722
0.780
0.848
0.908
1.040
0.630
0.738
1.069
1.205
0.998
1.170
0.796
0.933
0.560
0.656
0.705
0.827
0.865
0.976
0.602
0.657
1.023
1.179
0.953
1.039
0.774
0.844
0.516
0.563
0.672
0.733
0.842
0.962
0.589
0.668
0.996
1.141
0.953
1.082
0.755
0.857
0.485
0.550
0.646
0.733
0.860
0.959
0.609
0.650
1.014
1.167
0.987
1.054
0.781
0.834
0.487
0.520
0.660
0.705
Actual number of participants come from ENS administrative records. Estimated
numbers of participants are the USDA estimations given to the panel by USDA.
The forecasted ratios assume that estimates of participation are used to predict the
actual level of participants four years into the future.
Based on the forecasted ratios for total numbers of participants, it ap-
pears that the current USDA method of estimating the number of partici-
pants for a future budget cycle is quite accurate. The total number of pre-
dicted participants matches the number of actual participants quite closely,
ranging from a slight overestimate in 1996 (2.6 percent) to a slight under-
estimate in 2000 (4.1 percent), indicating very small levels of errors. These
smaller error levels are really just a coincidence, because the ratios for the
total number of participants mask substantial over- and underestimation
OCR for page 38
38 ESTIMATING ELIGIBILI~YANDPARTICIPATIONFOR THE WICPROGRAM
among the different eligibility categories. Numbers of infant participants
are substantially underestimated across all years, as are estimates of the num-
ber of breastSeeding postpartum women. In contrast, numbers of partici-
pating children are overestimated by 14 to 21 percent compared with ad-
ministrative records. Numbers of participating pregnant women are
overestimated also, but the degree is not so serious in later years. Numbers
of nonbreastSeeding postpartum women are consistently underestimated
and become more substantially underestimated in later years, so that by
2000, estimates are understated by about 17 percent.
Although USDA's current methods to estimate eligibility and partici-
pation seem to accurately forecast the overall number of participants, these
methods do not accurately forecast specific eligibility categories. Accurately
estimating the number of participants in each category is important be-
cause food package costs differ across each eligibility category. Furthermore,
changes in program rules or program administration could affect eligibility
and participation for each category differently. For example, increases in
the Medicaid income thresholds in states for infants and children would
affect eligibility of those groups but not that of pregnant and postpartum
women.
SOURCES OF ERROR
The true number of eligible persons who are likely to participate is
unknown. In making budgetary and programmatic decisions, USDA's goal
is to come as close to the true number of eligibles and participants as pos-
sible. There are two possible sources of error in making these estimates: (1)
errors that cause a systematic bias in the estimated number of persons eli-
gible or likely to participate and (2) prediction errors.
Errors Causing Systematic Biases
Errors may arise because data or methods used to make the estimates
are not able to fully capture all the programmatic features or the realities of
individuals' economic and family situations, leading to inaccurate estimates
of eligibility and participation. For example, the March CPS, the data set
used currently to make eligibility estimates, collects annual income instead
of monthly income. As Chapter 2 explains, local WIC agencies can use
weekly and monthly income, rather than annual income, to determine
whether an applicant is eligible for WIC. Variation in monthly income
OCR for page 39
ACCURACY AND SOURCES OF ERR ORS
39
could mean that some people gain eligibility for WIC in some months but
have annual incomes that would otherwise make them ineligible for WIC.
The following possible sources of systematic error are identified and dis-
cussed in this report:
· The undercount of infants and the overcount of children in the
CPS.
· The use of annual rather than monthly income in estimating eligi-
bility for WIC.
Not fully accounting for adjunctive eligibility through means-tested
programs, particularly Medicaid.
· The inaccuracy of adjustments to account for nutritional risk among
income-eligible persons.
The inaccuracy of adjustments to account for breastEeeding status
.
.
among postpartum women.
· The inaccuracy of adjustments for the percentage of eligible persons
who will participate in WIC.
Prediction Errors
The second source of errors arises because eligibility and participation
must be predicted for future years from data that are probably four years
old. For example, the following changes could lead to prediction errors:
.
Demographic changes (e.g., lower birth rates or increased immigra-
tion).
· Changes in family structure (e.g., if the proportion of infants and
children living in single-parent rather than two-parent families de-
clines, then, since single-parent families are, on average, poorer than
two-parent families, the proportion of those income-eligible infants
and children should decline).
Changes in the income distribution due to changes in wages or
unemployment.
Changes in program rules, including WIC, but also in other means-
tested programs for which adjunctive eligibility is granted. Medic-
aid rule changes in recent years are a primary example.
Changes in WIC program administrative practices.
Changes in participation rates of other means-tested programs.
· The length of time between the year that data are available for esti-
OCR for page 40
40 ESTIMATING ELIGIBILI>~D PAR TICIPATION FOR THE ~CPROGR~
mation and the year for which the estimate is being made. All else
equal, a longer time period should lead to larger errors.
These changes could, of course, have opposite effects on estimates of eligi-
bility and participation. For example, the increase in Medicaid income eli-
gibility thresholds would tend to increase the number of eligible persons.
However, the economic expansion of the late 1990s should have led to a
decrease in the number of persons income eligible for WIC and for Medic-
aid. It is difficult to infer the extent to which these changes offset each
other or not.
In considering current and alternative approaches to estimating eligi-
bility and participation for WIC, the panel has attempted to address these
r
two sources or errors.
EVALUATING CURRENT AND ALTERNATIVE METHODS FOR
E S T I M A T I N G E L I G I B I L I T Y A N D P A R T I C I P A T I O N
In this report we evaluate methods currently used to estimate eligibil-
ity and participation for WIC and examine alternatives to current meth-
ods. We outline different methodological options and include recommen-
dations for new approaches to the various components of the estimation
methodology. The panel considered several factors for evaluating alterna-
tives:
The accuracy of the estimates.
The feasibility of implementing an approach (e.g., the expense and
burden of implementing an approach).
The quality, availability, and timeliness of data used by an approach.
· Correspondence of the method with current WIC rules and their
application at the local level. For example, adjunctive eligibility is
part of the rules of the program, and methods for estimating eligi-
bility should account for it.
For some components of the estimation process, it is not clear how to
assess the accuracy of an approach. For example, the minor adjustment
currently made to account for adjunctive eligibility appears to be inad-
equate (National Research Council, 20011. However, data limitations cre-
ate problems in determining the true number of people who gain eligibility
for WIC only through adjunctive eligibility. Estimates based on survey data
OCR for page 41
ACCURACY AND SOURCES OF ERR ORS
41
are limited because respondents underreport Medicaid and other program
participation (although to what degree is not known). Administrative data
on Medicaid enrollees do not identify the age of children nor the income of
enrollees; therefore, no administrative data on the number of people who
would gain eligibility for WIC solely through adjunctive eligibility are avail-
able. The panel also based its assessments of methodologies on the premise
that the methodology should reflect current rules and practices of the pro-
gram.
Finally, the panel recognizes that the USDA estimates serve different
purposes and that different methods of estimation may be appropriate for
different purposes. Evaluation of the effectiveness of the WIC program
requires estimates of eligibility and participation. Because eligibility is not
directly observed, estimates must be based on information reported from
survey data. To make budgetary forecasts, reasonably accurate estimates
might be forecasted based on administrative records from past years. Chap-
ters 4-7 focus on the estimation of numbers of people eligible for WIC.
Chapter 8 focuses on estimates of participation among eligible people and
methods for estimating participation among them. In Chapter 9, we syn-
thesize all our findings and provide different options for estimating eligibil-
. . . .
lty ant ~ participation.
SUMMARY
This chapter began with an assessment of the accuracy of the USDA
estimates to predict the number of WIC participants. Using available data
to examine the accuracy of participation estimates, the panel found that the
total number of participants matches the reported number quite closely.
However, the numbers of participating infants and breastEeeding postpar-
tum women are seriously underestimated, while the numbers of participat-
ing children and pregnant women are overestimated. The chapter also ex-
plains the two types of errors the panel investigated in its review of the
current methodology errors that cause systematic bias and prediction er-
rors. Finally, in evaluating estimation methods, the panel considered accu-
racy, feasibility, characteristics of data sources, and correspondence of the
method with current rules and their application at the local level.
Representative terms from entire chapter:
administrative records
