National Academies Press: OpenBook
« Previous: 5 Technical Approach to Estimation
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

6
Evaluation of Estimates

By eliminating the need to collect applications, conduct verifications, and assess the eligibility status of students taking meals, a new Provision 4 would ease administrative burden in exchange for providing free meals to all students. Even schools operating under Provision 2 or Provision 3 might find Provision 4 attractive because it eliminates the “base-year” requirements to collect applications, conduct verifications, and count meals served by category. Reimbursements under Provision 4 would be based on claiming percentages estimated from the American Community Survey (ACS) in combination with data from other sources and information on participation as described in Chapter 5.

This chapter focuses on the suitability of such estimates from the perspective of their fitness for use. The panel will examine their quality from three perspectives: (1) by exploring sources of possible discrepancies between the estimated claiming percentages and the concept behind the authorizing legislation and regulations of the school meals programs, (2) by evaluating the quality of the estimates in the context of the error associated with existing practices and provisions, and (3) by assessing the estimates in the context of the decision processes to be affected by the estimates. In taking this approach, we recognize that no system for determining claiming percentages is perfect. We seek to identify the best method possible, not only from an error perspective, but also from the viewpoint of reducing costs and burden associated with administering the school meals programs and of improving access to the programs by the nation’s school children relative to current practices.

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

The quality of an estimate has many determinants, including the data sources that are inputs to the estimate and the underlying models used to generate the estimate. Survey estimates, for example, are subject to errors that arise in the survey process of sampling a population, obtaining data from the sampled households, and processing the collected data to create an analysis data set. Errors in administrative databases arise from the fact that most of these databases were not created to be analyzed as a whole, but rather to manage individual cases. Attention has seldom been given to editing the data in a unified way, so there may be many data entry or other errors. A survey or administrative database will record information on variables to measure concepts that are developed for specific applications, and these variables may not match the programmatic intent of the school meals programs. Another part of the process will involve identifying which records in a survey or administrative database are associated with the school district or school based on some geographic domain, and this will also be subject to error. Finally, when estimates for small populations, such as small school districts or individual schools, are needed, the estimation method will almost certainly involve some form of statistical model that specifies a structure to approximate—with error—the observed relationships in the population.

While this list may seem extensive, the current procedures for certification and meal counting in the school meals program, for example, are subject to their own errors associated with administrative processes that involve parents, students, lunch room staff, and office staff. The Access, Participation, Eligibility, and Certification (APEC) study (U.S. Department of Agriculture, Food and Nutrition Service, 2007b) has shown that the error rates and costs associated with these processes can be large. Thus, it is our intent to identify a method for implementing a new special provision that improves on the current approach in a cost-effective manner.

The chapter next describes the policy and decision-making context in which Provision 4 will be considered and summarizes the challenges associated with the current approaches to determining the proper reimbursements to school districts. The remainder of the chapter discusses various dimensions of quality that will be considered and outlines proposed approaches for evaluating the quality of the potential methods for estimating claiming percentages for Provision 4.

POLICY AND DECISION-MAKING CONTEXT

To promote improved learning and nutrition among school-age children, the National School Lunch Program (NSLP) and the School Breakfast Program (SBP) provide free and reduced-price meals to needy children. Under the traditional rules and procedures for operating the school meals programs, school districts serve as local administrators of the program

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

by determining the eligibility status of students (free, reduced price, full price), devising menu plans to meet the nutritional requirements set by the federal government, serving meals, and collecting money from students expected to contribute to the cost of the meals. The federal government reimburses states for the subsidized cost of meals, and states reimburse school districts. It is in the interest of all parties that the mechanism by which the schools are reimbursed is both accurate and predictable. From the national perspective, accurate and predictable reimbursements provide incentives for schools to foster program access and meet the nutritional needs of children while providing adequate oversight to control the program’s overall budget. Local schools desire accurate and predictable reimbursements to protect local resources from being inadvertently diverted from other educational priorities to the school meals programs.

What is meant by an accurate and predictable reimbursement to a school district? Federal legislation requires that for every meal meeting nutritional requirements, the district shall receive an amount of reimbursement based on the eligibility status of the student to whom the meal was served. In school year 2009-2010, the district receives $2.68, $2.28, and $0.25 for lunches served to students approved for free, reduced-price, and full-price meals, respectively. Schools that served more than 60 percent free and reduced-price lunches during 2007-2008 are eligible for 2 cents more per category.1

In the traditional approach, the accuracy of the reimbursement depends on five factors:

  1. the correct certification of students as approved for free or reduced-price meals,

  2. the ability of the cashier to determine whether a student’s meal meets the federal nutrition requirements,

  3. the correct classification of each student taking a meal by approval category (free, reduced price, or full price),

  4. the counting of meals served to students by their approval status, and

  5. the transmission of the school’s determinations to the school district and state to the federal government for reimbursement.

Under Provision 4, reimbursement would be based on claiming percentages estimated from the ACS and total meal counts that involve Factors 2, 4, and 5.2 Hence, the errors that will be eliminated with Provi-

1

Additional details on nutritional requirements of meals and reimbursements are provided in Chapter 2.

2

For Factor 4, errors in counting the total number of meals served will remain, but any errors in counting the number of meals by category will be eliminated.

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

sion 4 will be those related to application, certification, and verification (Factor 1), those associated with identifying the eligibility category of a student who is being served a meal (Factor 3), and those associated with counting meals by category (part of Factor 4).

As discussed in Chapter 5 and later in this chapter, however, the distribution of eligible students (distribution E in Figure 2-1) will be estimated with error using the ACS and other data. Furthermore, because Provision 4 requires no meal counting by category of eligibility, the distribution of meals served (distribution MU in Figure 2-1) will not be measured for the schools operating under Provision 4. The relationship between eligibility and meal participation by category will need to be estimated using the methods discussed in Chapter 5. These methods rely on observational data from districts that are not from a nationally representative sample, which may affect the accuracy of reimbursement based on the estimated claiming percentages. The panel will consider these issues as it carries out its analysis.

Although the panel will identify potential sources of error in estimates that may lead to errors in the reimbursements the school should be receiving, it is important to recognize that the correct amount of reimbursement will never be known. However, it is desirable that the level of error and uncertainty surrounding estimates used for reimbursement be limited to acceptable levels. Therefore, the panel will seek to develop methods that improve on the consistency and accuracy of the traditional method for determining reimbursements.

For management and planning purposes at both the local and the federal levels, it is desirable to minimize the level of intertemporal variation in the reimbursements due to random fluctuations in annual estimates that are unrelated to real changes in conditions. This is not to imply that reimbursements should remain constant over time. On the contrary, the variation in reimbursements over time should reflect changes in the correct reimbursements owed to the school as a result, for example, of an economic downturn (or boom) that results in more (or fewer) students being eligible for free and reduced-priced meals.

ERRORS IN CURRENT METHODS TO DETERMINE REIMBURSEMENTS

Traditional Method

While the panel will focus on evaluating the accuracy and reliability of a new method (Provision 4) to determine reimbursements, it is important to place this evaluation in the context of errors associated with the procedures in use today. Currently, the majority of school districts use

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

what we call the “traditional” method of operating the school meals programs. As described earlier, at the beginning of the school year, the district initiates a process in which parents are asked to apply to the school meals programs by supplying their income and the number of household members or the information required to established categorical eligibility (e.g., a Supplemental Nutrition Assistance Program [SNAP] case number).3

In this process, parents of students who are not directly certified need to apply in order for their children to receive the benefits of free or reduced-price meals. If an eligible family does not apply and is not identified by direct certification, a needy student has been denied access to free or reduced-price meals to which she or he is entitled.4

Even if parents submit the application form for their children, they must correctly complete it. This requires that parents have a correct understanding of the program definitions of income and membership in the household. For example, when parents are asked to report the number of household members, they need to know that foster children living in their household should not be counted, but that relatives such as aunts or grandparents should be counted. Parents need to know which forms of income should and should not be included. The application process further requires that parents accurately apply these concepts to their individual family situation.

After an application is submitted, school or district officials must review the application and determine whether the student is eligible for free meals or reduced-price meals (or has to pay full price). Even if the application is completely accurate, errors can be made at this stage in the certification process. Although the required annual verification of a sample of applications might reduce errors in the completion and review of applications, substantial certification errors still remain, as discussed below.

Once a student is approved for a specific eligibility status (free, reduced price, or full price), the school must retain daily records of the number of meals served for each eligibility status by linking a meal served to a student and then linking that student to his or her certified eligibility status. The daily records are compiled and then submitted to the school district. The school district submits them to the state. The state completes Form FNS-10, providing the information the Food and Nutrition Service (FNS) uses to determine reimbursements. At each stage of this process, errors may occur.

The APEC study (U.S. Department of Agriculture, Food and Nutrition Service, 2007b) described in Chapter 2 was an effort by FNS to obtain

3

An application does not need to be submitted if a student has been directly certified for free meals. Chapter 2 provides details about direct certification.

4

This is not counted as a certification error in official statistics, however.

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

national estimates of the amounts and rates of erroneous payments in the NSLP and the SBP. The APEC study found that the certification process is especially prone to error, with approximately 9 percent of total reimbursements for both the NSLP and the SBP considered erroneous due to certification errors. The study reported on two sources of certification error: (1) household reporting errors and (2) administrative errors made by districts in processing applications. It established that 23.2 percent of all certified students and denied applicants had household reporting errors on their forms, while 8.3 percent were subject to administrative error. (The two sources of error could occur on the same application and may have been offsetting.) Administrative error led to overcertification for 6.2 percent of applications and undercertification for 2.1 percent of applications, while household reporting error led to overcertification for 13.5 percent of applications and undercertification for 9.7 percent of applications.5 The most common type of household reporting error was misreporting of total income. This error affected 20 percent of certified or denied students. And 8 percent of certified or denied students had errors in the number of household members listed on the form. The most common administrative error was certification of the student as eligible for free or reduced-price meals when the application was incomplete.

According to the APEC study (U.S. Department of Agriculture, Food and Nutrition Service, 2007b:vol. 1, p. 53), roughly 14 percent of those approved as eligible for free meals should have been approved for a status with fewer benefits (8 percent for reduced price and 6 percent for full price). At the other end of the distribution, 36 percent of the students whose applications were denied, and thus were deemed as full-price students, should have been approved as free or reduced price (19 and 17 percent, respectively). Given the limited range of incomes for which a student would qualify for reduced-price meals, students who were approved with reduced-price status had the greatest amount of error. Roughly one-third of the approved reduced-price students should have been approved as free, and 25 percent should have been approved as full price.

The APEC study demonstrates the level of error in the traditional approach. It also demonstrates that the net effect of certification error tends to result in overreimbursement to districts, as illustrated next.

To quantify the effect that certification errors can have on the distribution of students by eligibility status when using the traditional method, the APEC study compared the distribution of students based on the categories for which they had been approved with the distribution based

5

Overcertification occurs when a student is certified for more benefits than those to which she or he is entitled. For example, a student approved for free meals is overcertified if she or he should have been approved for reduced-price or full-price meals.

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

on their true eligibility status, using the sample of students who went through the certification process and either had been certified for free or reduced-price meals or had their applications denied. The distribution based on approval status was 78 percent free, 17 percent reduced price, and 5 percent full price (U.S. Department of Agriculture, Food and Nutrition Service, 2007b:51). The distribution based on true eligibility status was 74 percent free, 14 percent reduced price, and 12 percent full price (U.S. Department of Agriculture, Food and Nutrition Service, 2007b:53).

The impact of certification errors on the amount of reimbursement is reflected in what can be called a “blended reimbursement rate,” or the average reimbursement per meal served (recall that a district is reimbursed $2.68, $2.28, and $0.25 for a meal served to a free, reduced-price, and full-price student, respectively). The blended rate (for those students who go through the certification process) based on the distribution of true eligibility status is $2.33, but it is $2.49 when based on the distribution of students as approved by the certification process—a 16 cent per meal difference.6

For the purpose of reimbursements, however, the relevant distribution is of meals served by category, not the distribution of students by category. As shown in Table 2-4, the fiscal year 2009 participation rates—meals served divided by the number of approved students adjusted for absenteeism—were 80, 72, and 46 percent for the free, reduced-price, and full-price categories, respectively. If we assume that the participation rate is based solely on the approval status and if eligibility has been correctly assessed, the blended rate for students going though the certification process and taking participation into account would be $2.45 per meal. Based on the approval status determined by the certification process, the blended rate for students going through the certification process and taking participation into account is $2.55.7

The APEC study also evaluated noncertification errors, classified as cashier error or aggregation error. The study found that the process by which cashiers assess and record whether a meal is reimbursable is a substantial source of erroneous payments, particularly in the SBP. However, most schools had fairly low levels of cashier error. The high

6

The blended rates above exclude the 25 cents per meal reimbursed for participants from the full-price category whose families did not submit an application and were not directly certified, but the amount reimbursed for this group is the same regardless of the classification of the other students. Hence the difference, 16 cents per meal, accurately reflects the impact of incorrect classification on reimbursement. It does not address the fact that some of the full-price students may have been eligible for a free or reduced-price meal.

7

The smaller implied overpayment based on the distributions of meals served rather than the distributions of students arises because participation rates are much lower for students paying full price than for students approved for free or reduced-price meals.

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

aggregate levels of cashier error arose from a few large schools having very high levels of this type of noncertification error. The study did not quantify cashier error due to Factor 3, incorrectly identifying the approval category of a student taking a reimbursable meal. However, it was hypothesized that automated point of sale technology in place in most schools would minimize this type of error.

Provision 2

From the perspective of expanding program access and providing nutritional meals to students, the adoption of Provision 2 by a district is clearly beneficial. Since the certification process no longer affects what the students will pay for their meals, participation in the program should increase. Students who truly were needy but were misclassified as full-price students will no longer have to pay for meals. The cost of this approach is that students who should be asked to pay will no longer be required to do so. This cost will fall on the school districts.

In general, the sources of error in reimbursements noted for the traditional method will also apply to Provision 2. Given that the claiming percentages remain fixed over a 3-year period, claims for reimbursement under Provision 2 will not reflect changes in income or demographics that would be reflected in the traditional method, which creates an additional source of error. Provision 2 reimbursements rely on less frequent use of application and certification procedures than under the traditional method—only once every 4 years. Infrequent certification may result in less accurate estimates of the claiming percentages, since parents may be less likely to return the application forms, and, even if they do, their responses may be less accurate due to their lack of familiarity with the forms. In addition, office staff may no longer be as skilled in performing the certification process, which may add another source of error. For these reasons, the accuracy of estimated claiming percentages might worsen when applications and certifications are done only once every 4 years.

The APEC study found evidence of this. It observed that erroneous payments due to certification error are more common in Provision 2 or Provision 3 schools (in their base years) than in schools not using these provisions (erroneous payments were approximately 1.75 percent larger for the Provision 2 or Provision 3 schools for the NSLP). A large proportion of students certified for free meals in the base year of Provision 2 or Provision 3 schools were overcertified. Hence, the claiming percentages for free or reduced-price meals tend to be overstated. The significance of the finding is that because the claiming percentages in these schools are fixed for at least 3 years, the Department of Agriculture has no mecha-

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

nism for correcting the erroneous claiming percentages unless the schools reestablish them in a new base year.

Provision 3

Federal reimbursement under Provision 3 is based on estimating the number of meals served in a category as the product of the number of meals served in that category during the same month in the base year times a factor that reflects changes in enrollment, the number of operating days, and inflation. Errors associated with certification during the base year for Provision 3 schools are as described under the traditional method and Provision 2 above.

One feature of Provision 3 is that schools do not have to count the number of qualified meals served each day. If the number of meals served by category remains unchanged (so that the participation rates of students remains unchanged), Provision 3 and Provision 2 reimbursements should be the same, but Provision 3 relieves the school of determining the daily count of meals served and thereby lowers the school’s administrative costs. However, any change in participation when enrollment stays steady would result in erroneous payments.

POTENTIAL ERRORS IN PROPOSED DATA SOURCES AND ESTIMATION METHODS

The primary data source for estimating eligibility percentages under Provision 4 is the ACS. Estimates from other national surveys, such as the Survey of Income Program Participation (SIPP), may also be employed to adjust ACS-based estimates for systematic bias. Although we focus on the ACS, the discussion applies to most national surveys more broadly.

Probability surveys are evaluated using a framework called total survey error, which identifies the types of errors that occur at various points in the development of a survey estimate. Components of total survey error include sampling (reflecting the fact that data are collected on a portion, rather than all, of the population), coverage (the degree to which the frame used to draw the sample includes the entire target population), nonresponse (failure to obtain responses for the entire sample), specification (the degree to which a question asked matches the concept about which information is desired), measurement (unintentional or intentional errors in a respondent’s answer), and processing (errors in applying coding, statistical processing, and estimation methods). In the context of estimating eligibility for free and reduced-price meals, the most problematic error components for the ACS are likely to be sampling error, specification

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

error, and measurement error. Although we will consider other sources of error in our evaluation, the ACS has a relatively high coverage rate and response rate, and processing errors in an ongoing survey tend to be small due to the repeated use of systems developed for the survey.

Sampling Error

In the context of a probability sample, sampling error is the error from observing only a portion of the population rather than the entire population. It recognizes that an estimated percentage (or other parameter estimate) calculated from the sample responses is very unlikely to be the true percentage associated with the population. Metrics of sampling error provide a measure of how much an estimate would vary if the sample were redrawn under the same design and the survey conducted in exactly the same way. Sampling error is typically quantified via the standard error of the estimate or related measures, such as a confidence interval or margin of error. The sampling error is a function of the underlying sample design and the population variability of the characteristic used to calculate the estimate. Sample designs are typically configured to obtain the most precise estimates of specific domains given cost and operational constraints.

When estimates of small domains—geographic areas or population groups—are of interest, sample sizes are generally small and sampling error is correspondingly large. For this study, it is likely that the reliability of direct estimates of the fraction of students eligible in each school meals category for individual schools, and even some entire school districts, will not be acceptable. In this situation, it is common, as discussed in Chapter 5, to use a model-based or model-assisted estimation approach that uses auxiliary data and modeled relationships to improve the precision of estimates for the small domains. A potential issue is the bias that can occur if the model approximation is incorrect. Methods to evaluate lack of fit can be used to assess this issue. As noted in Chapter 5, the Small Area Income and Poverty Estimates program has experience in developing estimates of the percentage of children in families with income less than 100 percent of poverty, and one way to mitigate the effect of small sample sizes is to extend these models to provide estimates of children in households with incomes under 130 percent and between 130 and 185 percent of the poverty level.

Specification and Measurement Error

When conducting a survey, one is generally interested in collecting data on a specific concept, even if one cannot always directly observe the

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

concept. Specification error arises when the question or measurement method does not match the target concept. In our application, we will interpret specification error somewhat differently. That is, we will look at specific questions in the ACS with respect to the concepts associated with school meals eligibility criteria (e.g., income, reporting unit), rather than the original target concept that the survey question was designed to measure. A concept related to specification error is measurement error, which arises in the response process. There are many potential sources of measurement error, depending on the type of question. For example, a respondent may have difficulty understanding or be inattentive to the correct meaning of the question; have trouble recalling past events or estimating income in accordance with the question’s definition; or provide erroneous answers due to social desirability pressures, perceived stigma, or privacy concerns in answering sensitive questions, such as questions about income and program participation.

In considering specification and measurement errors, the panel will focus on variables used to estimate eligibility: income, relationships within the household, program participation (SNAP, other welfare assistance), school status, grade, and age. For the ACS, the annual income measure appears to be comparable to the Current Population Survey measure used to determine official poverty rates (Czajka and Denmead, 2008). However, the annual figure averages over monthly income fluctuations and, as noted in Chapter 5, is likely to indicate as ineligible some students who would be eligible for free or reduced-price meals based on monthly income values (U.S. Census Bureau, 1998). Moreover, relative to program eligibility criteria, household relationships are not completely ascertained in the ACS, and in some situations, such as with multiple family units living in a housing unit, the identification of a household for purposes of eligibility determination may be incomplete. Although the ACS has a question that obtains information on SNAP participation during the past year, cash and other welfare assistance programs are lumped into a single question, and only some of those programs confer categorical eligibility for free meals. There is also evidence that program participation is underreported in the ACS.8 Finally, the geographic detail needed for some objectives in this study may require estimates based on the 5-year ACS, and thus will be insensitive to recent changes in the economy. As noted in Chapter 5, these issues suggest

8

Czajka and Denmead (2008:170) report, “As a rule surveys underreport numbers of participants in means tested programs, so in comparing estimates of participation across surveys, more is generally better.” Of the surveys they examined, SIPP has the highest number, 31.4 million people (or 11.2 percent of the population), in families receiving welfare or food stamps at any time during 2002. The ACS is second, with 24.5 million people or 8.8 percent of the population.

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

that adjustment factors may be needed to reduce the potential for bias in ACS eligibility estimates.

In addition to the factors noted above, the accuracy of survey estimates for small geographic areas depends on how well the geographic definition of the areas (school attendance boundaries) corresponds to the geographic areas in the survey (census blocks or block groups). The panel will use the school attendance areas in the case study districts to assess potential errors in digitized boundaries and use simple models to determine the impacts of such errors on the estimates that are of primary interest in this study.

PROPOSED EVALUATION PLAN

The previous sections have raised several potential issues associated with the inputs to estimates or the estimates themselves and the contextual backdrop against which we will assess these sources of error. We expect some error components to be quantifiable from the basic principles of statistics applied to the data sources or estimators themselves, while others will require further information gathering and analyses in order to evaluate assumptions and potential errors. It is possible that some aspects of the potential error in estimates will not be resolvable within the constraints of this study, and further studies may be suggested.

We anticipate using the following tools to describe the quality of estimates for Provision 4 and the potential impact of major issues on the accuracy of the estimates themselves, the costs and efficacy of operating the school meals programs under Provision 4 (relative to the traditional approach), and on decisions that are made by policy makers and program administrators.

First, we will analyze the results of the estimation processes outlined in Chapter 5 and the potential impact of errors in the data sources on the estimates. This will include not only an evaluation of sampling error, but also, to the extent possible, an evaluation of the potential biases that may exist in the estimates relative to errors in the current methods under traditional operating procedures.

Second, we will use the six proposed case study school districts to obtain information for estimating error in the context of existing practices. For example, we will examine and measure the uncertainties introduced at the school district and school attendance-area levels by boundary mapping and translate these into potential uncertainties regarding estimates. We will also use data from the schools in the case study school districts to assess the potential magnitude of discrepancies between reimbursements based on ACS-based claiming percentages under Provision 4 and

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×

reimbursements based on current data and methods under traditional procedures, which the case study districts are now using.

Third, we will conduct a workshop with school district representatives to gain a better understanding of the issues of interest to schools and school districts relative to Provision 4. We will encourage school district representatives to discuss potential errors and costs associated with both the current approaches and Provision 4. Case study district representatives will be key participants in the workshop.

Fourth, where direct information is not available, we will attempt to perform calculations to identify theoretical minima and maxima for potential errors and construct simulations to identify specific conditions that are more likely to be sensitive to errors and to determine the range of possible errors under these conditions relative to the uncertainties that exist in current methods.

Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 87
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 88
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 89
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 90
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 91
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 92
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 93
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 94
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 95
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 96
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 97
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 98
Suggested Citation:"6 Evaluation of Estimates." National Research Council. 2010. Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/12917.
×
Page 99
Next: 7 Operational Feasibility »
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report Get This Book
×
Buy Paperback | $46.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National School Lunch Program and the School Breakfast Program, administered by the Food and Nutrition Service of the U.S. Department of Agriculture (USDA), are key components of the nation's food security safety net, providing free or low-cost meals to millions of school-age children each day.

Under the most commonly adopted provisions, USDA reimburses districts for meals served on the basis of data collected in a "base year," during which applications are taken. After 3 or 4 years, applications must be taken again to establish new base-year data, unless the district provides evidence that local conditions have not changed.

A special provision that does not require applications to be taken every few years would reduce burden, be more attractive to school districts, and potentially increase student participation by expanding access to free meals. To support the development of such a provision, the Food and Nutrition Service asked the National Academies to study the technical and operational issues that arise in using data from the American Community Survey (ACS)--a new continuous survey replacing the long-form survey of the decennial census--to obtain estimates of students who are eligible for free and reduced-price meals for schools and school districts. Such estimates would be used to develop "claiming percentages" that, if sufficiently accurate, would determine federal reimbursements to districts for the schools that provide free meals to all students under a new special provision that eliminates the base-year requirements of current provisions.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!