Small-Area Estimates of School-Age Children in Poverty Evaluation of Current Methodology (2000) / Chapter Skim
Currently Skimming:
Evaluations of County Estimates
Evaluations of County Estimates
Pages 57-108
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 57... ...
or whether they are used to develop estimates for school districts. The Census Bureau's county estimates of poor school-age children are produced by using a county regression model and a state regression model (see Chapter 4~.1 A comprehensive evaluation of these two components of the estimation procedure should include both "internal" and "external" evaluations.
|
From page 58... ...
The focus of the evaluation effort was on alternative county models, particularly the assumptions underlying the regression equations and how the estimates of poor school-age children in 1989 from each model compared with 1990 census estimates. The state model was examined as well, both directly and as it contributed to the county estimates of poor school-age children.
|
From page 59... ...
examined residuals and model differences from the census, respectively, for categories of counties. The following characteristics were used for categorizing counties: census geographic division; metropolitan status of county; population size in 1990; population growth from 1980 to 1990; percentage of poor school-age children in 1980; percentage of Hispanic population in 1990; percentage of black population in 1990; persistent poverty from 1960 to 1990 for rural counties; economic type for rural counties; percentage of group quarters residents in 1990; number of households in the CPS sample in 1988-1991 (or whether the county had sampled households)
|
From page 60... ...
evaluation of the state model, including examination of regression output for 1996, 1995, 1993, 1992, 1991, 1990, and 1989 and consideration of the state raking factors by which county model estimates are adjusted to make them consistent with the state model estimates. COUNTY MODEL INTERNAL EVALUATION 1993 Evaluations The panel and the Census Bureau examined the underlying assumptions and other features of the four models, (a)
|
From page 61... ...
Linearity of the relationships between the dependent variable and the predictor variables was assessed graphically, by observing whether there was evidence of curvature in the plots of standardized residuals against the predictor variables in the model. In addition, plots of standardized residuals against CPS sample size and against the predicted values from the regression model were also examined for curvature.
|
From page 62... ...
bPredictor variables: (1) ratio of child exemptions reported by families in poverty on tax returns to total child exemptions; (2)
|
From page 63... ...
There are substantial differences in the estimated coefficients for the ratio of total child tax exemptions to the population in the log rate models across time periods and some differences between the coefficients in the two models. Inclusion or Exclusion of Predictor Variables The possibility that one or more predictor variables should be excluded from a model was assessed by looking for insignificant t-statistics for the estimated values of individual regression coefficients.6 The need to include a predictor variable, or possibly to model some categories of counties separately, was assessed by looking for nonrandom patterns, indicative of possible model bias, in the distributions of standardized residuals displayed for the various categories of counties.7 The only predictor variables with nonsignificant t-statistics are the population under age 21 (column 3 in Table 6-1)
|
From page 64... ...
The effect on estimates of poor schoolage children would stem from two factors: a shift in the weights assigned to each county in fitting the regression model, which would very likely result in only a modest change in the estimated regression coefficients; and a change in the weight given to the direct estimates, which could have an appreciable effect on the estimates only for the few counties with large CPS sample sizes. Outliers The existence of outliers was evaluated through examination of plots of the distributions of the standardized residuals and plots of standardized residuals against the predictor variables and through analysis of patterns in the distribution of the 30 largest absolute standardized residuals for the various categories of counties.
|
From page 65... ...
. Constancy of the Regression Coefficients Because the county model is refitted for each prediction year, constancy of the regression coefficients for the predictor variables over time is not as important as it would be if the estimated regression coefficients from the model were used for predictions for subsequent
|
From page 66... ...
Nonetheless, it is desirable for the coefficients to be in the same direction and not fluctuate wildly in size over time. Table 6-2 shows the regression coefficients for the predictor variables for the 1995 county model estimated for 1995 and 1993 and for 1989 with both the original and revised IRS data (see Chapter 4~.8 The coefficients for the three "poverty level" predictor variables child exemptions reported by families in poverty on tax returns (column 1)
|
From page 67... ...
Investigation of the standardized residuals for categories of counties for the county model estimated for 1995, 1993, and 1989 reveals little evidence of persistent bias. However, there is some suggestion that the model tends to consistently overpredict the number of poor school-age children in smaller size counties (i.e., the model estimates are somewhat higher than the CPS direct estimates for smaller counties)
|
From page 68... ...
Improvements in estimating the model error and sampling error variances should be sought to reduce or eliminate this problem. COUNTY MODEL EXTERNAL EVALUATION Comparisons with 1990 Census Estimates For external evaluation of alternative models that were considered for 1993 estimates, the panel and the Census Bureau compared the estimated number and proportion of poor school-age children for 1989 for the four candidate models with 1990 census estimates.9 The evaluation examined the overall difference 9The county estimates reflect the effects of the state model and the county population estimates as well as the county regression model, but the differences in model performance vis-a-vis the census in the evaluation are due to the particular form of the county model.
|
From page 69... ...
The evaluation does not address the issue that model-based estimates for a given year are used for Title I allocations about 3 years later. The 1990 census estimates that are used in the comparisons are ratio adjusted by a constant factor to make the census national estimate of poor school-age children equal the 1989 CPS national estimate.
|
From page 70... ...
, in which the county estimates of poor school-age children for 1989 are developed by converting 1980 census estimates of the proportions of poor school-age children for 1979 to estimated numbers by use of 1990 county population estimates of total schoolage children 5-17 and then raking the estimated numbers to the Census Bureau's state model estimates for 1989.
|
From page 71... ...
Column 2 of Table 6-3 is the average proportional absolute difference for county estimates of the number of poor school-age children, measured as the sum for all counties of the absolute difference between the model estimate and the 1990 census estimate as a proportion of the census estimate for each county, divided by the total number of counties and expressed as a percentage. Column 3 is the average proportional absolute difference for county estimates of the proportion of poor school-age children.
|
From page 72... ...
NOTES: The census estimates are controlled to the CPS national estimate for 1989. See text for definitions of models and measures; N.A.: not available.
|
From page 73... ...
shows lower and model (b) shows not appreciably higher average proportional absolute differences for estimates of poverty rates compared with the better log rate model (c)
|
From page 74... ...
difference between the model estimate of poor school-age children and the 1990 census estimate for each county, divided by the sum of the census estimates for all counties.l3 Counties are grouped into five or six categories for each of 11 characteristics those that were considered in the assessment of the county model regression output discussed above.l4 The measure in Table 6-4 expresses model-census differences for groups of counties in terms of numbers of poor children, similar to the overall average absolute difference in column 1 of Table 6-3. However, the category difference is expressed as an algebraic measure in which positive differences (overpredictions)
|
From page 75... ...
75 oft ¢ blob o ~ vO o ~ O ~ ca ca ¢ V 0> o 3 o o -o o Cq an C Co ~ ~ o 3 ~ it ~ ~ To ¢ AX .= ¢ (~ ~ oo o ° so t' z o tm z ° o ~ .
|
From page 76... ...
76 an o q ¢ Em ca C)
|
From page 77... ...
77 oo o oo .
|
From page 78... ...
78 ca so C)
|
From page 79... ...
79 ~ ;^ so .
|
From page 80... ...
In contrast, the stable shares procedure (i) , which simply ratio adjusts the 1980 census estimates to the CPS national estimate for 1989, performs .
|
From page 81... ...
Percentage Change from 1980 to 1990 in Poverty Rate for School-Age Children All four candidate models show a pronounced pattern of overpredicting the number of poor school-age children in counties that experienced the greatest decline in the poverty rate for school-age children from 1980 to 1990 and, conversely, underpredicting the number of poor school-age children in counties that experienced the greatest increase in the poverty rate for school-age children in that period. The category differences are smaller for the log number models (a, b)
|
From page 82... ...
(iv) 1 1 ~\T\ i_ 1 0 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ 0 c, ° ~ ~ co ~ 0 ~ co ~ co ~ cO ~ ~ 0 — 0 ~ ~ ~ c'' ~ co FIGURE 6-1 Change in poverty rate for school-age children, 1980-1990: Category differences from the 1990 census.
|
From page 83... ...
In contrast to the four candidate models, it overpredicts the number of poor school-age children in counties that experienced declines or smaller increases in population from 1980 to 1990 relative to counties that experienced larger population increases. The spread between the largest positive and negative category differences for the stable shares procedure is 32 percentage points.
|
From page 84... ...
(iv) O ~ 1 ~ ~ _ ~ \ T ~ a' ~ ~ ~ ~ 0 o o o ~ ~ ~ o a' ° ~ ° 0 0 it 0 a' ~ ° a' ~ '0 ~ s a, O° at - (O at ~ c~ a' a' — FIGURE 6-2 Population growth, 1980-1990: Category differences from the 1990 census.
|
From page 85... ...
al al o o o o ~ ° i o o o ° o Lr) of o ° C\l LO o C~ FIGURE 6-3 Population size, 1990: Category differences from the 1990 census.
|
From page 86... ...
, which relies solely on the 1980 census estimates, performs poorly on this characteristic. However, the averaging procedure (iv)
|
From page 87... ...
. O ~ ~ lo lo u - 0 co l lo Cal o FIGURE 6-4 Percent Hispanic population, 1990: Category differences from the 1990 census.
|
From page 88... ...
The spread between the largest positive and negative differences is 11 percentage points. Because the county estimates from the four candidate models are raked to the state estimates from the Census Bureau's state model and census divisions are combinations of states, category differences on this characteristic must be attributable to the state model.
|
From page 89... ...
a) 1 o o l o FIGURE 6-5 Percent group quarters residents, 1990: Category differences from the 1990 census.
|
From page 90... ...
/ 1 1` it/ I ~ / lo./ .O In ~ A, ~ ~6 Al Al ~ Eli ~ ~ ~ TIC 0 0 a) u' o Oh u' o oh u' FIGURE 6-6 Census division: Category differences from the 1990 census.
|
From page 91... ...
The Census Bureau performed chi-square tests to determine if there were significant differences between estimates from the March 1990 CPS and the 1990 census of the number of school-age children and the number and proportion poor in this age group in 1989 for county groupings (Fay, 1997~.17 More specifically, the tests determined if the ratios of the CPS and census estimates for categories of a characteristic, such as county population size, were significantly different from each other. The characteristics tested were those examined in the 1990 census comparisons.
|
From page 92... ...
92 o to VO o o to o so Jo Cq an .= Cq o Cq Cq to Do Cq at ·~ ad o ~ an an Do 3 ~ an a, .= at an O A ~ be ~ ¢ ·= 1 of ~ an ¢ EM ¢ so ¢ · ~ C)
|
From page 93... ...
Using the state model to rake the 1980 census county estimates for consistency with updated estimates of poor school-age children in each state, as is done in procedures (ii)
|
From page 94... ...
O 10This analysis is not the same as the analysis of regression output described above, in which the standardized residuals from the model for counties with sampled households in the CPS representing the standardized differences between the model estimates and the direct estimates on the log scale were examined for categories of counties.
|
From page 95... ...
(5) Census Regiond Northeast 217 -2.87 0.81 -4.36 10,708 Midwest 1,055 -0.49 0.61 -4.31 11,393 South 1,425 4.05 -0.13 4.48 15,440 West 444 -4.16 -0.95 -0.43 12,141 Census Divisiond New England 67 -13.51 1.87 27.07 3,696 Middle Atlantic 150 0.05 0.54 -9.79 7,012 East North Central 437 -6.10 -0.64 -3.04 6,841 West North Central 618 18.31 4.25 -7.44 4,552 South Atlantic 591 1.82 0.83 4.12 8,150 East South Central 364 -5.53 -5.85 9.32 2,529 West South Central 470 12.00 1.90 2.44 4,761 Mountain 281 -3.91 19.87 0.84 5,543 Pacific 163 -4.24 -6.48 -0.92 6,598 Metropolitan Status Central county of metropolitan area 493 -2.75 -0.91 -3.53 34,343 Other metropolitan 254 53.75 -3.64 8.44 2,801 Nonmetropolitan 2,394 1.24 3.50 8.32 12,538 1990 Population Size Under 7,500 525 -17.21 57.03 0.74 933 7,500-14,999 630 19.82 -23.67 -0.19 1,550 15,000-24,999 524 2.94 6.24 17.02 2,289 25,000-49,999 620 30.46 -0.23 -4.46 4,204 50,000-99,999 384 -2.52 4.99 22.47 5,979 100,000-249,999 259 17.27 12.12 -3.88 8,263 250,000 or more 199 -7.24 -2.49 -3.10 26,464 1980 to 1990 Population Growth Decrease of more than 10.0% 444 -2.71 -22.03 -4.29 2,170 Decrease of 0.1-10.0% 972 -4.31 2.44 -1.32 10,655 0.0-4.9% 547 6.04 3.41 3.18 8,015 5.0-14.9% 620 1.12 5.97 4.61 11,590 15.0-24.9% 260 -0.07 -4.11 -10.44 9,305 25.0% or more 292 -0.52 -2.27 10.31 7,947 continued on next page
|
From page 96... ...
(5) Percentage of Poor School-Age Children, 1980 Less than 9.4% 516 2.74 7.22 -1.07 14,980 9.4-11.6% 524 1.39 5.28 4.35 12,291 11.7- 14.1 % 530 -10.01 -6.49 -6.72 9,837 14.2-17.2% 523 1.28 -5.82 0.44 5,217 17.3-22.3% 519 9.32 17.41 0.23 4,623 22.4-53.0% 523 1.05 -14.81 4.11 2,734 Percentage Hispanic, 1990 0.0-0.9% 1,770 1.26 -0.75 3.13 12,848 1.0-4.9% 847 9.33 1.45 4.32 16,966 5.0-9.9% 193 -2.81 17.24 6.38 6,999 10.0-24.9% 181 -4.02 -5.14 -8.29 7,236 25.0-98.0% 150 -7.90 -3.29 -5.26 5,633 Percentage Black, 1990 0.0-0.9% 1,446 8.32 8.02 5.09 10,929 1.0-4.9% 615 7.41 1.04 -1.83 10,630 5.0-9.9% 294 5.41 -2.07 0.95 8,646 10.0-24.9% 381 -4.89 -0.75 3.51 13,437 25.0-87.0% 405 -6.85 -2.82 -6.30 6,040 Persistent Rural Poverty, 1960- l 99oe Rural, not poor 1,740 -2.62 1.53 5.47 9,734 Rural, poor 535 22.45 -0.15 14.81 1,698 Not classified 866 -1.28 -0.28 -2.68 38,250 Economic Type, Rural Countiese Farming 556 -24.56 -29.31 -12.41 1,634 Mining 146 46.97 27.59 40.67 901 Manufacturing 506 -7.10 -3.58 -1.51 2,369 Government 243 120.13 27.59 59.39 1,661 Services 323 -12.18 -12.42 -11.86 2,760 Nonspecialized 484 6.99 18.35 23.89 2,018 Not classified 883 -1.18 -0.20 -2.59 38,339 Percentage of Group Quarters Residents, 1990 Less than 1.0% 545 3.32 22.03 16.60 3,494 1.0-4.9% 2,187 -1.58 -1.27 -1.84 41,648 5.0-9.9% 299 11.90 -1.22 4.51 3,980 10.0-41.0% 110 49.44 -6.28 17.02 560
|
From page 97... ...
, Ymode1 is the estimated number of poor school-age children from the county model, and YCps is the estimated number of poor schoolage children from a 3-year weighted average of the CPS, is Ei (Ymode!
|
From page 98... ...
The model shows a tendency to underpredict the number of poor schoolage children in the largest counties, those with 250,000 or more population. This finding is consistent with the results from analyzing the distribution of the standardized residuals from the regression output.
|
From page 99... ...
As was noted above, this finding is consistent with the knowledge that any regression model can only partially predict which cases will have the most extreme values of the outcome variable. Local Assessment of 1993 County Estimates The panel performed another type of external evaluation of the original 1993 county estimates of poor school-age children the use of local knowledge.20 Using the original 1993 model estimates for all 3,143 counties in the United States, the analysis first sought to identify groups of counties for which the 1993 estimates seemed unusually high or low in relation to prior levels and trends (e.g., from 1980 to 1990)
|
From page 100... ...
Some of these implied changes are apparently related to the general effect of population size, discussed above. However, the findings in this regional analysis, in particular, suggested which states and counties to follow up in discussions with local officials.
|
From page 101... ...
Although the trends for a few counties were not accepted locally, the analysis found no strong indicators of potential bias for groups of counties sharing common characteristics in the county model. Summary Considering the external evaluations of alternative models that were conducted by comparison with 1990 census estimates, the external evaluations of 3
|
From page 102... ...
STATE MODEL EVALUATION The state model plays an important role in the production of county estimates of poor school-age children. Evaluations conducted of the state model for the assessment of the revised 1993 county estimates included an internal evaluation of the regression output for 1989 and 1993 and an external evaluation that compared 1989 estimates from the model with 1990 census estimates of proportions of poor school-age children.
|
From page 103... ...
. Linearity Plots of standardized residuals against the four predictor variables in the state model the proportion of child exemptions reported by families in poverty on tax returns, the proportion of people receiving food stamps, the proportion of people under age 65 who were not included on a tax return, and a residual from the analogous regression equation using the previous census estimate as the dependent variable support the assumption of linearity.
|
From page 104... ...
ratio of child exemptions reported by families in poverty on tax returns to total child exemptions; (2) ratio of people receiving food stamps to total population; (3)
|
From page 105... ...
Therefore, if a non-zero estimate for model error variance is produced, it might have important consequences for the state estimates of poor school-age children. Second, while there are some appreciable differences, the model estimates were within two standard errors of the direct estimates for almost all states in each year.
|
From page 106... ...
. Mlsslsslpp Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina CPS Direct Estimate (1)
|
From page 107... ...
These results suggest that the state model is performing reasonably well: differences between model and direct estimates are neither unusually large nor strongly persistent. However, more work should be conducted to evaluate the current procedures for estimating the sampling error variance of the state model and the effects on the model estimates.
|
From page 108... ...
Also, some variation in the raking factors is expected given the form of the county model and the need to transform the predicted log values of poor school-age children to estimated numbers before the raking is performed. Other sources of this variability could include the use of 3year averages of CPS estimates as the dependent variable in the county model versus single-year estimates in the state model, sampling variability, and, possibly, individual state effects that are not captured in the county model (see Chapter 9 and National Research Council, 2000:Ch.3~.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.