3
Panel Findings and Conclusions: Data Quality and Reliability

In this section the panel selectively addresses several major aspects of the Bureau of Transportation Statistics (BTS) report, focusing on the analysis of the requirements for data quality and reliability associated with the purposes of allocation formulas. Section 4 of this report discusses potential (new) sources of data, the strengths and weaknesses of the models used in the BTS report, and the possible role of emerging technologies in obtaining needed data.

The panel concurs with the BTS report that available freight data are not of sufficient coverage and quality to permit precise determination of international trade traffic by state. The panel’s review of the availability of validated data to support estimates suggests needed improvements in data collection programs to enhance the accuracy and reliability of international trade traffic measures for formula apportionment purposes.

The panel concurs with the central finding of the BTS study (Bureau of Transportation Statistics, 2003) that the present state of available freight data does not permit precise determination of international trade traffic by state.

The panel was limited in its ability to evaluate the accuracy and reliability of the data for use as formula factors because no allocation formula that depends on ton-miles or value-miles of international trade traffic has been developed or proposed. Without knowing the formula and its purpose,



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 15
Measuring International Trade on U.S. Highways 3 Panel Findings and Conclusions: Data Quality and Reliability In this section the panel selectively addresses several major aspects of the Bureau of Transportation Statistics (BTS) report, focusing on the analysis of the requirements for data quality and reliability associated with the purposes of allocation formulas. Section 4 of this report discusses potential (new) sources of data, the strengths and weaknesses of the models used in the BTS report, and the possible role of emerging technologies in obtaining needed data. The panel concurs with the BTS report that available freight data are not of sufficient coverage and quality to permit precise determination of international trade traffic by state. The panel’s review of the availability of validated data to support estimates suggests needed improvements in data collection programs to enhance the accuracy and reliability of international trade traffic measures for formula apportionment purposes. The panel concurs with the central finding of the BTS study (Bureau of Transportation Statistics, 2003) that the present state of available freight data does not permit precise determination of international trade traffic by state. The panel was limited in its ability to evaluate the accuracy and reliability of the data for use as formula factors because no allocation formula that depends on ton-miles or value-miles of international trade traffic has been developed or proposed. Without knowing the formula and its purpose,

OCR for page 15
Measuring International Trade on U.S. Highways it is not possible to assess how accurate estimates of international trade traffic must be. If, for example, the central purpose of the formula is to compensate states for wear and tear on their highway infrastructure, then even highly precise measures of value-miles will bear little relationship to the objective of the formula, and ton-miles will only roughly reflect the goal of the formula. Less accuracy is needed if the purpose of the formula is insensitive to variations in ton-miles than if the formula is highly sensitive. More generally, the accuracy that is needed for formula allocation may depend on whether the objective is constructing, maintaining, and operating infrastructure; reducing congestion; minimizing the impact of the highways on the environment; improving the safety of the network; or any of a number of other worthwhile objectives. When evaluating the quality and reliability of data, such factors as error, bias, and transparency would be weighed differently, depending on the objective of the apportionment formula. It is instructive to compare the quality of international trade traffic data with that of the data used in existing federal highway apportionment formulas. Each major federal highway program has a legislated formula that relies on data obtained from the joint Federal Highway Administration Highway Performance Monitoring System. The primary measures used in the current allocation formulas include lane miles, vehicle miles traveled, diesel fuel data, state population, urbanized area population, and nonhighway recreational fuel use. Each of these data elements has its own error measures, as well as an established validation procedure that is documented in guidelines that were carefully developed in a federal-state cooperative venture. It is important to note that each of the data elements used in the existing formulas is directly and independently measured. The measures are also transparent: their sources are well known, and they use carefully monitored procedures for verification. The current formula allocation factors do not rely on model-based estimation procedures; each has a directly measurable error structure in which bias and variance can be specified. The characteristics of verifiability and transparency are extremely important for estimates used as allocation factors. Unfortunately, the estimates computed in the BTS report are neither verifiable nor transparent. The BTS model-based estimates of ton-miles and value-miles of inter-

OCR for page 15
Measuring International Trade on U.S. Highways national trade traffic have their genesis in a multiplicity of data sources and methodologies. As the BTS report appropriately notes, “limitations in the data have a negative effect on their accuracy and reliability.” Even when it is possible to specify the error associated with one of the data inputs, such as the sampling variance for the Commodity Flow Survey (CFS), it is not possible to specify the quality and reliability of the overall estimates, which reflect the accumulation of the errors1 of several data sources. It is also not possible to compute the uncertainty that is introduced into the estimates by several of the key methodologies, such as use of the gravity model, the substate allocation procedure, and the highway network assignment procedure. The BTS report appropriately addresses the need for separately identifying international trade traffic data for use in an allocation formula. However, the panel is concerned that in the long run, a focus on estimating ton-miles and value-miles of international trade traffic by state may distract attention from the larger issues of characterizing sources of highway wear and tear and congestion for purposes of funds allocation. Wear and tear, congestion, and other concerns are not a function of whether the traffic is international or domestic; rather, they relate to such measures as traffic volumes, axle load factors, vehicle types and categories, and weight. It was observed in the panel’s workshop that pavement doesn’t care whether traffic is international or domestic. Moreover, even total ton-miles are only roughly related to wear and tear and other concerns. It is likely that total ton-miles of all types of freight traffic moving by state highway can be estimated more accurately than the disaggregated measures of domestic and international trade traffic.2 Part of the problem of accurately estimating international trade traffic stems from two issues of disaggregation. We offer two examples, on disaggregating diesel fuel use and on computing substate estimates. 1   By errors due to data sources here, we mean all sources by which the estimate fails to equal the true value, which includes various types of bias and sample variance. 2   The panel notes that the Federal Highway Administration’s Office of Freight Management and Operations uses a variety of data sources in the development of the Freight Analysis Framework (FAF), which estimates (as of 1998) and forecasts (to 2010 and 2020) total freight tonnage flows on U.S. highways, as well as U.S.-Canadian truck traffic. The base year of the FAF is the 1993 economic census year, for which CFS data are also available. However, the FAF is assembled from existing data sources. Therefore, it is subject to the same problems of low frequency of acquisition and questionable accuracy as the data sources.

OCR for page 15
Measuring International Trade on U.S. Highways Disaggregating Diesel Fuel Use. The BTS report assesses the accuracy and reliability of diesel fuel data as a measure of ton-miles of international trade traffic on state highways. Diesel fuel data is already a factor in the apportionment of funds for the National Highway System program, but the measure has several weaknesses when pressed to serve as data for allocation purposes. It is difficult to estimate the number of gallons consumed for road use in each state because fuel may be purchased in one state and used in another, and there are differences in state reporting of motor fuel sales. Procedures used to adjust sales data to use data (by means of reports submitted by carriers under the International Fuel Tax Agreement) add variability to the estimates. Furthermore, not all diesel fuel is used for on-the-road vehicles: for example, diesel fuel is also used to run construction equipment and railroad locomotives, and therefore any heterogeneity in these uses across states would further limit the utility of diesel sales for this purpose. Other factors further reduce the reliability of these estimates. Nonetheless, the diesel fuel data are currently used as a factor in the apportionment formula. The BTS report states that these estimates “inherently capture international trade traffic.” Several computations are necessary to separate international from domestic diesel fuel use. The BTS report uses a regression analysis of the relationship between the estimate of total ton-miles by truck by state and diesel fuel sales by state to produce a formula for predicting ton-miles based on diesel fuel use. The report finds that the regression relationships for many states are weak, suggesting that “diesel fuel usage estimates do not provide good predictors of total trade ton-miles on a state-by-state basis.” The panel is concerned that the next step—to break down the weak total relationships into domestic and international components—would require comparing estimates of international trade traffic ton-miles from the BTS study with estimates of total ton-miles by state from other sources. The panel concurs with the BTS report that the process of disaggregating diesel fuel use into its domestic and international trade traffic components would compound variability in that it would apply a ratio of two estimates with significant variability with state diesel fuel use data that have known quality problems. Computing Substate Estimates of Import Trade Traffic Flows. Existing data sources provide estimates of imports flowing to each state, but not of the distribution of imports within states. The distribution

OCR for page 15
Measuring International Trade on U.S. Highways of imports within a state is estimated by assuming that imports to each county are proportional to annual payrolls computed from data in the 1997 County Business Patterns. The accuracy of this assumption is unknown. For discussion of this point, see National Research Council (2004). The issue of the quality of data used to support fund allocation formula programs is a complicated one; for a broad review of the considerations involved and much more general issues concerning inputs to and use of fund allocation programs, see National Research Council (2003b).