Small-Area Estimates of School-Age Children in Poverty Interim Report 2, Evaluation of Revised 1993 County Estimates for Title I Allocations (1998) / Chapter Skim
Currently Skimming:
4. EVALUATIONS
4. EVALUATIONS
Pages 33-77
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 33... ...
Such comparisons may indicate changes that would be helpful for a model; they may also suggest that an alternative model is preferable. The Census Bureau's county estimates of poor school-age children are produced by using a county regression model, a state regression model, and county population estimates developed with demographic analysis techniques (see Chapter 2~.
|
From page 34... ...
evaluation of the state model, including examination of regression output, external evaluation in comparison with 1990 census estimates, and consideration of the state raking factors by which county model estimates are adjusted to make them consistent with the state model estimates; and (6) evaluation of county population estimates for children aged 5-17 (see also Appendix B)
|
From page 35... ...
The panel and the Census Bureau examined the underlying assumptions of the four candidate models through evaluation of the regression model output for 1989 and 1993.i Although such an evaluation is not likely to provide conclusive evidence with which to rank the performance of alternative models, particularly when they use different transformations of the dependent variable, examination of the regression output is helpful to determine which models perform reasonably well. iThe evaluation of the county regression output pertains to the regression models themselves, that is, before the predictions are combined with the direct CPS estimates in a "shrinkage" procedure or raked to the estimates from the state model (see Chapter 2)
|
From page 36... ...
However, since the census equations only affected the weights for the weighted least squares regression and the extent of "shrinkage" in combining model estimates and direct estimates for counties with households in the CPS sample, analyses of the 1990 census regressions are not discussed here. 3The standardization of the residuals involved estimating the predicted standard errors of the residuals, given the predictor variables, and dividing the observed residuals by the predicted standard errors.
|
From page 37... ...
, the coefficients for the three "poverty rate" predictor variables ratio of child exemptions reported by families in poverty on tax returns to total child exemptions (column 1) , ratio of food stamp recipients to the total population (column 2)
|
From page 38... ...
bPredictor variables: (1) ratio of child exemptions reported by families in poverty on tax returns to total child exemptions; (2)
|
From page 39... ...
Normality The normality of the standardized residuals was evaluated through use of QQ plots, which match the observed distribution of the residuals with the theoretical distribution, and other displays of the distribution. All four models exhibit some skewness in their standardized residuals, with the log rate models (c, d)
|
From page 40... ...
The effect on estimates of poor school-age children would stem from: a shift in the weights assigned to each county in fitting the regression model, which would very likely result in only a modest change in the estimated regression coefficients; and a change in the weight given to the direct estimates, which could have an appreciable effect only on the estimates for counties with large CPS sample sizes. Outliers The existence of outliers was evaluated through examination of plots of the distributions of the standardized residuals and plots of standardized residuals against the predictor variables and through analysis of patterns in the distribution of the 30 largest absolute standardized residuals for the various categories of counties.
|
From page 41... ...
Reliance on a single 8The county estimates reflect the effects of the state model and the county population estimates as well as the county regression model, but the differences in model performance vis-a-vis the census in the evaluation are due to the particular form of the county model. The models for which the 1990 census comparisons were performed were estimated with the method of moments.
|
From page 42... ...
The census comparisons were done for the following procedures: (i) Stable shares procedure, in which the county estimates of poor schoolage children for 1989 are the 1980 census estimates for 1979 after ratio adjustment to make the 1980 census national estimate equal the CPS national estimate for 1989.
|
From page 43... ...
, in which the county estimates of poor school-age children for 1989 are developed by converting 1980 census estimates of the proportions of poor school-age children for 1979 to estimated numbers by use of 1990 county population estimates of total schoolage children 5-17 and then raking the estimated numbers to the Census Bureau's state model estimates for 1989.
|
From page 44... ...
between the model estimate and the 1990 census estimate for each county, divided by the total number of counties. Column 2 of Table 4-2 is the average proportional absolute difference for county estimates of the number of poor school-age children, measured as the sum for all counties of the absolute difference between the model estimate and the 1990 census estimate as a proportion of the census estimate for each county, divided by the total number of counties and expressed as a percentage.
|
From page 45... ...
NOTES: The census estimates are controlled to the CPS national estimate for 1989. See text for definitions of models and measures; N.A.: not available.
|
From page 46... ...
It was expected for the same reason that the log number models would have higher average absolute differences for estimates of proportions of poor school-age children than would the log rate models because population estimates must be used to convert the estimated numbers from the log number models to estimated proportions. However, model (a)
|
From page 47... ...
47 a' be ¢ o to VO o o o so a' o Cq a' .~ ~ C)
|
From page 48... ...
48 .= ¢ ca s° o to o ~ ca ·4= o z V o ~ ~ ca So ca ¢ V =~.^ ~ Do o ° z ° so tm z ~ o 1 ~ ~ _4 ~ ~ ;^ so o 4= v CM ~ o o CM ~ ~ o ~ ~ ~o ~ ~ ~ o Do Do CM ~ ~ o ~ ~ CM Do CM ~CM .
|
From page 49... ...
49 oc ~ ~ ~ oo o M M ~ cM .
|
From page 50... ...
so ca in c)
|
From page 51... ...
51 be R ~R = C)
|
From page 52... ...
For example, the model may over~under~predict, on average, the number of poor school-age children in larger counties relative to smaller counties.l4 If the census estimates are a reasonably accurate standard for comparison, sizable category differences between model and census estimates that are not explained by differences in the CPS and census measurement of poverty or another reason would be disturbing. They would indicate that the errors in the model estimates are not random errors (which occur in any set of estimates)
|
From page 53... ...
exhibits a strong monotonic pattern in which the number of poor school-age children is overpredicted for counties with higher percentages of group quarters residents relative to counties with lower percentages. Also, the magnitude of the category differences for counties classified by percent group quarters residents is small for model (b)
|
From page 54... ...
, which simply ratio adjusts the 1980 census estimates to the CPS national estimate for 1989, performs substantially worse than all of the models and other procedures on almost every characteristic. Category Differences for Specific Characteristics Category differences from the 1990 census estimates are discussed below for characteristics for which Table 4-3 shows that some or all four candidate models exhibit poor performance in comparison with the census in estimating the number of poor school-age children: percent change from 1980 to 1990 in the poverty rate for school-age children; percent population growth from 1980 to 1990; 1990 population size; percent Hispanic population in 1990; percent group quarters residents in 1990; and census division.
|
From page 55... ...
In contrast to the four candidate models, it overpredicts the number of poor school-age children in counties that experienced declines or smaller increases in population from 1980 to 1990 relative to counties that experienced larger population increases. The spread between the largest positive and negative category differences for the stable shares procedure is 32 percentage points.
|
From page 56... ...
0 ~ O ~ O ~ O ~ 0 ~ 0 0 ~ 0 _ 0 ~ ~ Cal ~ c`) FIGURE 4-1 Change in poverty rate for school-age children, 1980-1990: Category differences from the 1990 census.
|
From page 57... ...
~0 a) FIGURE 4-2 Population growth, 1980-1990: Category differences from the 1990 cen sus.
|
From page 58... ...
for 1989 performs reasonably well in predicting numbers of poor school-age children for county population size categories (see Figure 4-3~. Percent Hispanic Population in 1990 All four candidate models tend to overpredict the number of poor school-age children in counties with larger percentages of Hispanics relative to counties with smaller percentages, but the spread between the largest positive and negative differences is small.
|
From page 59... ...
a) O O O O O O O O 0 ~0 FIGURE 4-3 Population size, 1990: Category differences from the 1990 census.
|
From page 60... ...
(iv) | : ~ en- \T ~ ~0 ~ o l o o oo l o Cal FIGURE 4-4 Percent Hispanic population, 1990: Category differences from the 1990 census.
|
From page 61... ...
The spread between the largest positive and negative differences is 11 percentage points. Because the county estimates from the four candidate models are raked to the state estimates from the Census Bureau's state model, category differences on this characteristic must be attributable to the state model.l6 The state model 16The category differences are the same for all four candidate models because they are raked to the same set of state estimates; see Table 4-3.
|
From page 62... ...
~0 u' u' a a) l of of FIGURE 4-5 Percent group quarters residents, 1990: Category differences from the 1990 census.
|
From page 63... ...
This reversal is expected because the use of population estimates for children aged 5-17, which themselves contain errors, to convert estimated numbers to estimated proportions from the log number models puts these models at a disadvantage for comparisons of proportions. Conversely, the use of population estimates for children aged 5-17 to convert estimated proportions to estimated numbers from the log rate models puts these models at a disadvantage for comparisons of numbers (see below, "Use of Postcensal Population Estimates".
|
From page 64... ...
o En U] FIGURE 4-6 Census division: Category differences from the 1990 census.
|
From page 65... ...
o o vO o cq o to o so ,~o.
|
From page 66... ...
CPS national total. Using the state model to rake the 1980 census county estimates for consistency with updated estimates of poor school-age children in each state, as is done in procedures (ii)
|
From page 67... ...
. The Census Bureau performed chi-square tests to determine if there were significant differences between estimates from the March 1990 CPS and the 1990 census of the number of school-age children and the number and proportion poor in this age group in 1989 for county groupings (Fay, 19971.17 More specifically, the tests determined if the ratios of the CPS and census estimates for categories of a characteristic, such as county population size, were significantly different from each other.
|
From page 68... ...
Small implied increases were found in the Central Corn Belt, the Southern Appalachian Coal Region, the Coastal Plain Cotton Region, the Northern Great Plains, 19The discussion refers to "implied" trends because the Census Bureau's county model is not designed to directly estimate change over time.
|
From page 69... ...
In some states, the original 1993 county estimates released by the Census Bureau had not been examined, and there appeared to be little interest in discussing them. In other states, the estimates had been looked at, but the general admonitions about standard errors that accompanied their release had dampened interest in studying them in detail.
|
From page 70... ...
STATE MODEL The state model plays an important role in the production of county estimates of poor school-age children. Evaluations conducted of the state model include an internal evaluation of the regression output for 1989 and 1993 and an external evaluation through comparing 1989 estimates from the model with 1990 census estimates of the proportion of poor school-age children by state.
|
From page 71... ...
. Linearity Plots of standardized residuals against the four predictor variables in the state model the proportion of child exemptions reported by families in poverty on tax returns, the proportion of people receiving food stamps, the proportion of people under age 65 who did not file a tax return, and a residual from the analogous regression equation using the previous census as the dependent variable support the assumption of linearity.
|
From page 72... ...
Also, although there is less information available for the state model than for the county regression models, the residual plots and the box plots of the distributions of the standardized residuals against the categories of states show little evidence of any heterogenous variance. Finally, there is no evidence of outliers from examination of the residual plots or displays of the distributions of the standardized residuals from the state regression model.
|
From page 73... ...
found a smaller overall average absolute difference from the 1990 census when the county model estimates were raked to the state model estimates for 1989 than when the county model was used without raking (National Research Council, 1997:31~. On the assumption that a county model is performing well, one would expect the state raking factors to be tightly distributed around 1.0 that is, one would expect relatively minor differences between the estimates for states formed by summing the county estimates before raking and the estimates from the state model.
|
From page 74... ...
The investigation should include consideration of whether there is any feature of the state model that might explain the variation in the raking factors. USE OF POSTCENSAL POPULATION ESTIMATES The process for producing updated estimates of school-age children in poverty at the county level and the use of those estimates in the Title I allocation formulas require population totals by age in noncensus years for two purposes: as a variable in the county regression equation (population under age 18 or under age 21, depending on the model)
|
From page 75... ...
The overall average proportional absolute difference in the 1990 county estimates of the population aged 5-17 was 6.3 percent, unweighted by county population size, and 4.9 percent, weighted by size. By comparison, the overall average absolute difference in the 1990 county estimates of the total population was 3.6 percent unweighted and 2.3 percent weighted.
|
From page 76... ...
. In the aggregate, the use of population estimates to convert estimated numbers from log number models to estimated proportions adds about 1 percentage point to the overall average proportional absolute difference between the model estimates for 1989 and the 1990 census estimates (compare column 3 with column 2 of Table 4-2 for the two log number models)
|
From page 77... ...
were developed on the basis of 1990 census data. Because of the 4-year instead of 10-year period for updating, it is likely that errors in the 1994 population estimates are smaller than errors in the 1990 population estimates and that they have even smaller effects on the estimates of the number and proportion of poor school-age children.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.