Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 Mode' Uncertainly and Evaluation MOBILE IS A TOOL for estimating current and forecasting future mobile- source emissions, including quantifying the effects of control measures. These results form key elements of many air-quality regulatory compli- ance programs and directly affect transportation planning and the selec- tion of control strategies. Thus, there is a need for a high degree of accu- racy from MOBILE. One of the specific charges to this committee is to assess, to the extent practical, the overall accuracy of the current version of the MOBILE model in predicting atmospheric emissions. Such information is derived from model evaluation. However, the necessary information to quantitatively evaluate the current or upcoming versions of MOBILE with great confi- dence does not exist. This lack of information is one of the most serious concerns with MOBILE and its use. Assessing accuracy in the context of MOBILE involves model evalua- tion, typically by comparing real-world emissions with model predictions. It also involves considering uncertainty and bias that arise *om the wide variety of observations, assumptions, and mathematical relationships that underlie MOBILE model algorithms. This chapter discusses uncertainty and model evaluation, and reviews previous studies on these topics. iThis report uses model evaluation in reference to assessing the ability of a model to accurately represent the real world, for example by being able to estimate the emissions from mobile sources with little error. This differs from model validation that refers to assessing the correctness of the form of the model. MOBILE is based 735
7 36 MODELING MOB/[E-SOURCE EMISSIONS DEFINITION OF TERMS Accuracy, Uncertainty, and Bias Accuracy in MOBILE refers to its ability to correctly estimate the true value of emissions. Bias is the tendency for estimates to be consistently higher or lower than the true values. Uncertainty is the variability (or scatter) in MOBILE's prediction about the actual emissions. MOBILE is accurate, for example, if it predicts the correct emissions factors for the fleet of on-road vehicles, and if it predicts the actual changes that result from mobile-source emissions control programs such as inspection and maintenance (I/M). (MOBILE, as typically applied, provides only point estimates without any statistical confidence intervals around those esti- mates). MOBILE predictions can be assessed by comparing accurate in- use measurements of vehicle emissions and air quality to model predic- tions. This is termed model evaluation and is the subject of the second half of this chapter. Uncertainty and bias in MOBILE arise from many sources, primarily from the data used to construct the model, and from errors in analyses and assumptions leading to model formulations (discussed further below). Un- certainty and bias in MOBILE are difficult to assess because of the com- plexity of the model, the uncertainty in the underlying emissions data and model formulation, the uncertainty in the input data, and the difficulty in obtaining accurate measures of real-world emissions (e.g., from analyzing ambient data). Uncertainty, as used here, should not be confused with repeatability. If MOBILE is run several times with the same set of inputs, the model will always generate exactly the same output. The model is repeatable simply because it has no stochastic or random component; this does not by any means imply that the model is accurate. Figure 4-1 shows the difference between bias and uncertainty. The top box in Figure 4-1 shows a case in which the model provides accurate re- sults. The bottom box shows estimates that are uncertain (in that they are scattered about a one-to-one correspondence) but unbiased. The mid- dle box shows estimates that are biased; on average the predictions are below the actual emissions. For MOBILE's many uses, it is important that the model's estimates be accurate with a high degree of certainty and on statistical analyses of data, and the form of MOBILE is a set of statistical rela- tionships. Thus, model validation would correspond to determining whether the statistical relationships were derived in a valid manner and implemented correctly. Model evaluation determines whether those relationships provide accurate results.
M ODES UNCERTAINTY AND EVALUATION 7 3 7 0.50 0.40 - ~e 0.30 o - LO E 0.20 0.10 0.00 0.50 - 0.40- ._ c a ._ In ._ - - ~ 0.10- / /. 0.00 - 0.00 0.30 - 0.20 - Accurate / - / / ,,' / . . . .. . . . .. ... / ,. ./, , . . 1 0.00 0.10 0.20 0.30 0.40 0.50 Predicted Emissions (g/mi) Biased 7 ·- ~ ... , . ~ 1 0.10 0.20 0.30 0.40 0.50 Predicted Emissions (g/mi) Uncertain 0.50 0.40- E 0.30- o ._ u' E 0.20 I as 0.10 0.00 - / 0.00 0.10 0.20 0.30 0.40 0.50 Pre d icte d Eon is s ions (g/m i) · Series1 Series1 Series1 FIGURE 4-1 Representation of bias and uncertainty (hypothetical data).
~ 38 M ODEL/NG MOB!LE-SOURCE EMISSIONS a low bias. If not, transportation and air-quality planners could be led to implement costly, unnecessary control programs. Sensitivity Model sensitivity refers to the variation in model output in response to changes in model inputs such as average speed, ambient temperature, fuel volatility, and I/M program parameters. It is important that air-quality planners understand the effects of changes in these model inputs. The U.S. Environmental Protection Agency (EPA) has provided sensitivity analyses for some earlier versions of MOBILE (see discussion below), but not extensive analyses for MOBILES. A comprehensive sensitivity analy- ses should be performed for all model inputs and provided as part of user guides for all future versions of MOBILE. TYPES AND SOURCES OF UNCERTAINTY AND ERROR Many kinds of uncertainty plague the emissions estimates provided by MOBILE. A major fraction of those uncertainties arise from limitations in the scientific and technical basis of MOBILE as well as the data inputs, including sampling and measurement errors. As described above, the pri- mary source of uncertainty in MOBILE output is in the underlying emis- sions data used to generate the model formulations. There can also be limitations in MOBILE's structure. Input uncertainties, including data on vehicle characteristics and usage, are propagated through the model and contribute substantially to uncertainty. Another source of uncertainty that should be analyzed is the true variability in the system being mod- eled. It is clear that it is not practical to eliminate all uncertainty from mobile-source emissions models. MOBILE is a statistically based model, and its accuracy depends on valid and comprehensive samples as the foun- dation of the statistical relationships within. Some uncertainty comes from the random variation in the relatively small samples. No sample short of 100% (a complete census) can be large enough to completely elimi- nate randomness from affecting parameter error, but in all practical cir- cumstances, the influence of random variation on an estimate of a parame- ter decreases as the sample size gets larger. Even if 100% of the on-road vehicles were tested, uncertainty would still arise from the incompleteness and unrepresentativeness of the tests and errors in the testing procedure. Analysts must also consider true variability and its impact on uncer- tainty. Within defined fleets, manufacturers do not produce exactly the same kinds of vehicles. Vehicles from a single manufacturer are not iden-
MODEL UNCERTAINTY AND EVA[UAT!ON 7 39 tical. Also, emissions from any one vehicle vary from time to time and place to place in ways that are correlated with age, mode of driving, and many other things. Fleet characteristics vary spatially, as do topography and driving habits. The contribution of this true variability to the overall uncertainty of MOBILE is unknown, but could be partially characterized in an advanced mobile-source emissions model if the data (both input and model formulation) were available. Uncertainties further arise from limitations inherent in the model's structure. For example, no fleet of vehicles is well characterized. Road- way networks and associated driving patterns are not perfectly repre- sented. MOBILE does not capture all the factors leading to emissions (particularly high emissions, such as those induced by steep road grades). These and other model limitations discussed in Chapter 3 influence uncer- tainty. Following are descriptions of the general types of uncertainties that arise in the MOBILE model. Several examples of each type are provided, for both MOBILES and MOBILES. For additional information, see the study by Wenzel et al. (In press), which discusses emissions variability in more detail, and also describes other issues that complicate the statistical analysis of vehicle-emissions test data. Nonrepresentative Vehicle Samples MOBILE algorithms and emissions factors are based largely on test data from in-use vehicles that are solicited through the mail or by recruit- ment at I/M test stations. Typically, the owner is provided with a small payment and a rental vehicle until testing is completed. Free vehicle re- pairs are sometimes provided as an incentive. Response rates for such recruitment efforts are very low, typically less than 25% and sometimes as low as 5%. Recruited vehicles have serious bias issues because high emitters and tampered vehicles as well as expensive luxury vehicles are less likely to be voluntarily submitted for testing. As discussed in Chapter 3, very high emitting vehicles are a relatively small fraction of the on-road vehicle fleet, but they contribute a very large fraction of total vehicle emissions. Emis- sions from high-emitting vehicles are much more variable than emissions from normal emitters (Knepper et al. 1993), and thus require a large sam- pling fraction to obtain reasonably accurate estimates of their emissions (and to estimate the effects of control programs for their emissions). It is thus critically important that such vehicles be appropriately represented in emissions testing programs. If emissions from high emitters are not properly characterized, then MOBILE emissions factors can be seriously biased.
740 MODELING MOBILE-SOURCE EMISSIONS EPA recognizes that underrepresentation of high-emitting vehicles in the Federal Test Procedure (FTP) and IM240 databases underlying MOBlLE5's basic emissions rates is a serious shortcoming that produces biased Jow) emissions estimates. For MOBILES, EPA is proposing to ad- just the basic emissions rates based on data from the Dayton, Ohio, IM240 program. Although this is a step in the right direction, there are still bi- ases in the Dayton IM240 data. One type of bias arises from noncompli- ance with the program - the database does not include emissions from ve- hicles that are registered in the I/M area but do not obtain the required inspection (and repair if needed). Remote sensing studies have shown that these noncomplying vehicles have higher than average emissions (e.g., Stedman et al. 1997; Stedman et al. 1998; Wenzel 1999~. In fact, the I/M program compliance rate is one of the inputs to MOBILE, and the model assumes that noncomplying vehicles have emissions about twice as high as complying vehicles. A second problem is that when the Dayton program was first implemented, there is strong evidence that owners of vehicles that had been registered in the I/M area changed their registration to sur- rounding counties that were not subject to the Dayton I/M program ~IcClintock 1999~. Again, these vehicles are more likely to be higher emitting vehicles. Although EPA recognizes the inherent biases in their testing data, use of the Dayton IM240 data to adjust for those biases still likely results in biased emissions estimates. Some of the MOBILE algorithms and correction factors are based on very small samples, which may be nonrepresentative of the population. An example for MOBILES is the test data used to derive air-conditioning correction factors for light-duty vehicles (LDVs). As described in Chapter 3, the effects of full air-conditioning operation on vehicle emissions were determined from a sample (of 37 vehicles) 23 passenger cars and 14 light-duty trucks, al] from model years 1990 to 1996. Only five of the vehi- cles were high emitters for at least one pollutant. Similarly, the activity data for air-conditioning operation were obtained from a fleet of only 20 vehicles operating in Phoenix, Arizona, from August to October 1994. And, although the actual air-conditioning operation depends on the torque generated by the compressor, no data were available for this variable. Variability in Vehicle Emissions Vehicle emissions are highly variable, for a variety of reasons. Two ve- hicles of the same make and manufacturer, model year, technology, and accumulated mileage can have very different emissions measured on the same test or drive cycle. Such variation can be caused by factors such as how the vehicle has been driven and maintained, prior tampering with emissions control system components, and repeated excessive driving
MODEL UNCERTAINTY AND EVALUATION 7 4 7 loads. Studies have also shown relationships between socioeconomic fac- tors and vehicle emissions, with vehicles in lower than median income households exhibiting higher than average emissions (Singer and Harley 2000). Vehicles of the same age and technology also exhibit differences in emis- sions because of manufacturing and emissions-control design differences. Analyses of I/M data have shown that specific vehicle models have much lower or much higher emissions than average (Wenzel 1997~. In general, vehicles with higher emissions exhibit much more test-to-test variability than lower-emitting vehicles (Bishop and Stedman 1996~. Many factors can contribute to variability in repeated emissions tests of the same vehicle. For example, failures of some emissions control system components (such as a partially degraded catalyst) can be intermittent and therefore result in higher emissions some of the time. Other sources of differences in the vehicle emissions test data underlying the MOBILE model arise from the testing process. For example, back-to-back emissions tests will vary because of differences in the measurement equipment, cali- brations, and personnel (e.g., driving styles in tracking a target speed-time trace on a dynamometer). Those and other factors create uncertainties in the statistical models fitted to the data that are the basis of MOBILE emissions factors. An ex- ample of the enormous scatter in the emissions test data underlying MOBILE is shown in Figure 4-2. The figure shows the test data used to estimate NOX emissions for Tier 1 LDVs in MOBILE6. The solid line in the middle of the figure corresponds to 2 times the FTP standard; vehicles with emissions above this level are considered high emitters. Partly be- cause of the lack of sufficient data, and partly because EPA assumes that high emitters are "broken" vehicles whose emissions are always high no matter how old the vehicles are, emissions for the high emitters are mod- eled as a constant (the upper dashed line). This emissions estimate is then adjusted with the Dayton IM240 data as described above. Below the solid line, the vehicles are considered to be normal emitters. For these vehicles, the basic emissions rate is modeled as a linear function of accumulated mileage (the lower dashed line). Clearly there is large uncertainty in the emissions data and consequently in the basic emissions rates for normal- and high-emitting vehicles estimated from these data.2 These vehicle-to- vehicle differences are critical for some uses of MOBILE, but the scatter 2EPA has updated their original analysis of these data for Tier 1 NOx emissions using additional data sets. The additional data reduces the confidence limits (al- though not the scatter) for the normal emission regression equation. However, there is still significant uncertainty in the estimated mean value for high emitters.
742 - U] ._ I: o . . ~ . 1 ~ 1 ~ , 1;: 1 ,o .!2 1 ~ 1 I: 1 ~ , ~ 1 ~ , ~ 1 At is ! it 1 . . . . ·. . ~ , ~ Cal O ~ 10 ~ ~ O ~ ~ ~ ~~ ~ · · O 2 1 ~ ~ 1 l ~ ~ 41 t ~ 1 ~ · i l . . ~ , . . · t . $: LU It it l. ; ~ t=.- . .' . ~ :~N ·.~ L lo N EN 004 it_ ._' A A o . - A ·_' A C) . _~ ~5 O _ _ do lo lo lo ._ o of ~ · . O O IwI6 ~ N O O O O CO o y Fl a C~ a a) ~D a) Ce C~ CQ ¢ ~Q ~ O .. :m ~ ~ O a, v C) - .o a 1
M ODEL UNCERTAINTY AND EVALUATION 7 43 tends to get smoothed out in aggregated data, such as estimates of area- wide burdens. What does not get smoothed out is consistent bias, such as systematic under-estimation of the emissions from high-emitters. Incorrect Model Formulation The emissions factors, emissions-factor adjustments, and estimates of emission control program effects in MOBILES and its predecessors are estimated from statistical analyses of available test data. The statistical models are chosen using both engineering considerations (to represent the physical process) and statistical considerations (such as selecting the model that produces the best statistical correlation). Uncertainties can arise from incorrect or inappropriate engineering and statistical models. Examples of incorrect physical-engineering models include the following: Neither MOBILES nor MOBILES includes road-grade effects, as these are not incorporated into the test data underlying the models. Road- grade influences vehicle emissions; higher emissions occur under vehicle load conditions such as steep road grades. Light-duty truck emissions are sometimes estimated from automobile test data, because of the lack of sufficient data for trucks. Although some light-duty trucks are used (and may emit like) passenger vehicles, other light-duty trucks are used regularly as working vehicles and frequently carry heavy loads; emissions from these working trucks are likely to be higher than automobiles of the same model and age. In MOBILE, the lack of sufficient data to estimate the effects of high emitters is sometimes filled in by assuming that high emitters behave as normal emitters do. In EPA's multistep adjustment of basic emissions rates for high emit- ters using Dayton IM240 data, one step is to estimate full IM240 emis- sions from fast-pass IM240 data (see Glossary for definition of fast-pass). This is done using second-by-second IM240 data from Wisconsin, so simu- lated fast-pass emissions can be compared to full IM240 emissions (EPA l999g). The regression equation developed by EPA has the time (in sec- onds) of the fast-pass as one of the predictors of the full IM240 emissions; full IM240 emissions are assumed to be linearly related to the logarithm of fast-pass emissions. Given the variation in the speed-time trace of the IM240 data, there is no physical reason why fast-pass time should be lin- early related to full IM240 (or to the log of full IM240) emissions. How- ever, the coefficient for the fast-pass time is statistically significant in the regressions for all three pollutants (CO, VOCs, and NOX). This appears to be because newer vehicles are much more likely to pass early in the test than older vehicles (Pollack et al. 1999a).
744 MODELING MOBILE-SOURCE EM`SS/ONS Inaccurate physical-engineering models can also arise from the failure to obtain correct data in a test program. Examples of that include the follow- ~ng: MOBILES speed-correction factors (SCFs) were determined from emissions of vehicles driving over a series of test driving cycles. The driv- ing cycles were characterized only by average speed, although, as dis- cussed in Chapter 3, there are many other factors in a driving pattern that also affect vehicle emissions. The SCFs used in MOBILE6 are facility-spe- cific. For each facility, driving cycles were developed from analyses of real- world (instrumented vehicles) driving-pattern data. Although there is more aggressive driving in these cycles, and they are facility-specific, the cycles are still characterized for a given facility, by a single parameter- average speed. The development of modal (or second-by-second) emissions models (discussed in Chapter 5) will go a long way towards resolving these Issues. The FTP cycle and the facility-specific speed correction factor cycles of MOBILE6 represent samples of a universe of driver and vehicle behav- ior. That universe may have low- frequency events with extremely high emissions. If these low-frequency high-emission events are not properly represented in the cycles used for MOBILE, an accurate picture of exhaust emissions will not be obtained. Some of the data critical to estimation of air-conditioning effects on emissions were not available for use in MOBILE6. These include the ef- fect of vehicle speed, the time the compressor is on, the validity of the heat index as a measure of air-conditioner use, car occupant behavior in air- conditioner use, or the effect of the actual compressor torque. Examples of incorrect statistical models include the following: In some analyses, the intercept of the statistical model is forced through zero, to match physical processes; this represents a trade-off be- tween the correct statistical model and the correct physical model. A1- though such model alteration makes sense from an engineering point of view, it introduces bias into the resulting statistical model. Two examples of this practice in MOBILE6 are the following: The determination of hot-running emissions from FTP test data; the test data are first transformed using logarithms, then the intercept is forced to be zero (EPA l999k). MOBILE6 uses Dayton IM240 data to adjust FTP test data to ac- count for the absence of high emitters in the FTP sample. The exhaust-
M ODEE UNCERTAINTY AND EVALUATION 7 45 emissions rate adjustment is treated as an additive function of mileage, with zero increase in the adjustment at zero mileage (EPA l999g). Logarithmic transformations are frequently applied to emissions data for fitting statistical models to bring the emissions data closer to the nor- mal (Gausian) distribution, for which most statistical methods are devel- oped. The transformed data may still be non-normal, and the translation of results back into the original scale may be hard to interpret. In addition, the log transformation has the effect of reducing the influ- ence of high values, which may be of greatest concern. Distributions of vehicle emissions typically show the majority of the emissions from normal emitters at relatively low levels, and a small fraction of the emissions at very high levels. These distributions with long right tails are much closer to lognormal than normal, although other distributions such as gamma have been fit as well (Zhang et al. 1994~. Another reason that logarithmic models are commonly fit is that variability in vehicle emissions is higher at higher emissions levels; logarithmic transformations stabilize the vari- ance. However, logarithmic transformations are usually inappropriate in analyses of data sets with both normal and high emitters. With Togarith- mic transformations, the effects of high emitters are minimized relative to normal emitters, whereas in the atmosphere the effects of high emitters are, in fact, much greater than normal emitters. In summary, logarithmic transformations offer some convenience in analysis and the opportunity to use simple statistical models, but at a cost of introducing potentially seri- ous errors. Figure 4-3 shows an example from MOBILES of an inappropriate use of a logarithmic model. The data in the figure show hydrocarbon (HC) emis- sions from model-year 1993 fuel-injected cars from a data set of Wisconsin IM240 second-by-second emissions; these data were used as part of the multistep process to adjust basic emissions rates using Dayton IM240 data (EPA l999m). The Dayton data contain many fast-pass emissions tests, and the Wisconsin data were used to develop regression equations to pre- dict fast-pass emissions (FHC on the horizontal axis) to full IM240 emis- sions (HC on the vertical axis). The graph on the left is a scatterplot of the full IM240 HC emissions against the fast-pass HC emissions; the top and right sides of the figure provide histograms of each of these variables indi- vidually. Clearly these emissions measurements are not normally distrib- uted. The right graph shows the same data, but with a logarithmic trans- formation applied to both the full IM240 (LHCnz on the vertical axis) and fast-pass HC (LFHC on the horizontal axis) measurements. Both the scat- terplot and the marginal histograms show that the logarithmic transfor- mation is an over-transformation it diminishes the effects of the high-
746 ~ _ 1 1-- . 1 1 1 1 1 1 'i ~ ~ 1 1 ~ Do ~ CO U. ~ CO ~ ~ o _ o o o o o o o o o o ~ OH Fo. _0 ~ _ _ _° o · ~ ~ · ~ ~ ~ ~ ~ _ sol ~ ~ ~ Jo, ~ ~ \] . . o V ~ ~ ~ V a ~ ~ ~ ~ H Ct ~ ~ ~ ~ ~ Cal O ~ ~ ~ -a ~ ~ .-. - Cd O ~1 ~ _ d 5- Ct.= ~ O ~ O C) O ·e ~ O Ct~ ~ ° a, a) 0 ~ 5, ~ ~ ~ 0 0 Cd ~ V 0 ~ ~ ~ ~ ~ a) =- = ° ° Ct . ~ ~ · - 0 ~ ~ ~ ·e ~ CO o. ~ , ~ 5- . - a) ·5-~ C<S =2 (V ~~ 5- Ct
M ODES UNCERTAINTY AND EVALUATION 7 4 7 emitting vehicles and overemphasizes the effects of the lower-emitting ve- hicles. Uncertainties in Vehicle-Activity Data The MOBILE model estimates emissions factors in grams per mile (g/mi). Some of the uncertainties in these emissions factors have been de- scribed above. Local transportation and air-quaTity planners then obtain estimates of vehicle miles traveled (VMT). VMT estimates are multiplied by the MOBILE emissions factors to derive an estimate of total on-road vehicle emissions. Although VMT estimates are not part of the MOBILE model itself, they are essential to the use of the model, and mobile source emissions inventory results can be substantially affected by uncertainties in the VMT estimates. In addition, uncertainties in other inputs to MOBILE, such as average speed, vehicle age distributions, and vehicle mix, also contribute to uncertainties in MOBILE emissions factors and emissions inventories estimates with these factors. Reviewing the uncer- tainty in transportation inputs to the model is beyond the scope of this review, but work has been done in this area (e.g., Chatterjee et al. 19971. PREVIOUS MOBILE SENSITIVITY AND UNCERTAINTY STUDIES Complete characterization of MOBILES model sensitivity and uncer- tainties on emission-factors has not been conducted, although it is critical to accomplishing one of the specific tasks assigned to the committee. No comprehensive study has attempted to estimate the relative importance of a wide range of input and model uncertainties on output uncertainties. However, limited sensitivity and uncertainty analyses have been per- formed for both the current and earlier versions of MOBILE. These are described in this section. However, none of these estimates are truly com- prehensive in their scope. Sensitivity Analyses EPA's Office of Transportation and Air Quality (OTAQ) and others have performed sensitivity analyses of recent versions of MOBILE. Although EPA's sensitivity analyses have been focused exclusively on model re- sponse to changes in input parameters only, other authors have reviewed model sensitivity to some parameters within the model. EPA first pre- pared a collection of graphs and tables showing sensitivity of MOBILE4 inputs to speed, temperature, and fuel Reid vapor pressure (RVP) (Shih
748 MODE[/NG MOBILE-SOURCE EMISSIONS 19901. For MOBILES, EPA prepared a large collection of tables showing MOBILES emissions factors every 5 years from 1995 to 2020 by vehicle class and emissions mode as a function of input altitude, speed, operating- mode fractions, air-conditioning usage, and fraction of the fleet carrying extra load or towing a trailer (EPA 1998j). In the first major outside review of MOBILE, PoBack et al. (1991) evalu- ated the sensitivity of MOBILE4 (and also EMFAC7E) to changes in both user inputs and model parameters and functions. Sensitivity of predicted emissions factors was evaluated in response to alternative fleet character- ization (mileage accumulation, registration distributions, and diesel pene- tration rates), exhaust-emissions rates (emitter category proportions and deterioration rates), exhaust-correction factors (speed, temperature, and fuel volatility), I/M program parameters and tampering rates, and evapo- rative emissions-factor parameters. Several studies have performed MOBILES sensitivity simulations. Heiken et al. (1994) assessed the sensitivity of model output to alternative formulations for exhaust-emissions rates, evaporative system pressure- purge test pass-fail rates, and evaporative basic emissions rates. Fox (1996), using Monte Carlo simulations, evaluated the sensitivity of MOBlLE5 to uncertainties in seven key model inputs minimum and maximum temperature, fuel RVP, average speed, fraction of VMT from catalyst and noncatalyst vehicles in cold-start operation, and fraction of VMT from catalyst vehicles in hot-start mode. Chatterjee et al. (1997) fo- cused on key travel-related inputs to MOBILES (speed, VMT, vehicle clas- sification, and operating-mode fractions) and assessed MOBILES sensitiv- ity to these variables for conformity analyses. Uncertainty Analyses Several studies have been performed to assess uncertainties in specific MOBILE model components. Heiken et al. (1994) calculated confidence intervals for model components of particular interest to the American Pe- troleum Institute, the study's sponsor. These included IM240 to FTP cor- relations, running losses, and evaporative system pressure-purge test pass-fail rates. MOBILES SCFs have been the focus of two studies. Guensler (1993) and GuensTer and Leonard (1997) assessed uncertainty in speed-correction factors (SCFs) using Monte Carlo analyses based on SCF regression errors. Chatterjee et al. (1997) performed statistical analyses of SCF regressions; confidence intervals for SCFs as a function of speed were then used to estimate uncertainty in emissions at typical roadway types and speeds (congested and uncongested freeways and congested ar- terials). Kini and Frey (1997) derived estimates of uncertainties in basic emis-
M ODEL UNCERTAINTY AND EVALUATION 7 49 signs rates and speed corrections combined for one specific MOBILES LDV technology group (port-fuel and throttle-body injection vehicles). Exhaust emissions errors for these vehicles only were estimated using a bootstrap technique applied to regression errors. Uncertainties for the emissions factors were found to be 20% to 40% for volatile organic compounds (VOCs), and 25% to 55% for NOX; biases in the emissions factors were also found. Uncertainties in all exhaust-emissions correction factors combined in the EMFAC7G model were determined by Pollack et al. (1999b). These correction factors (speed, fuel, off-cycle emissions, high emitter corrections, and variable start times), analogous to those in MOBILE, collectively in- troduce 20% to 40% uncertainty in the model's exhaust emissions rates. It is important to note, however, that in this and all uncertainty studies de- scribed here, the uncertainty estimates were derived considering only un- certainties in the statistical models fit to the available test data. Total MOBILE (and EMFAC) model uncertainty includes many other factors, and hence is likely to be substantially higher. WHY UNCERTAINTY ANALYSES ARE NEEDED A comprehensive uncertainty analysis for MOBILE could assess the relative contributions to model uncertainty of specific model parameters, data gaps, and model statistical and engineering formulations. Uncer- tainty attribution studies are used in other aspects of air-quality assess- ments, and have been effective in targeting areas for more detailed study. An uncertainty assessment of MOBILE is critical to provide model users with important information and to target future research. At a minimum, the assessment should provide estimates of uncertainty for each of the model's output emissions factors, including contribution of each emissions factor to the overall uncertainty in the on-road inventory. For example, it might be the case that the emissions factors for motorcycle emissions have the greatest variability, but that their small contributions to the on-road inventory would not justify the cost of obtaining more test data. An uncertainty profile is critically needed for guiding future mobile source modeling development. Specifically, an uncertainty profile should guide future emissions testing programs in generating data that would be most effective in improving the model. Uncertainty analysis has other important uses. It provides information for interpreting model results and the appropriate use of those results, and can provide confidence to policy-makers making decisions based on associ- ated analyses. Air-quality planners might wish to design their control pro- grams to compensate for some margin of error, to have more confidence
750 MODELING MOBJ[E-SOURCE EMISSIONS that air-quality goals for the future wiB be achieved. Further, an uncer- tainty analysis will provide information relevant to model accuracy, al- though it does not provide the actual accuracy of the model. As noted ear- lier in this chapter, the model can be very accurate even if the results are highly uncertain. Likewise, an uncertainty analysis might miss a key er- ror, or the actual uncertainties in the model application may be greater than those tested ("uncertainties in the uncertainties"), leading to a larger error. The problem, however, is that doing an assessment of uncertainty in the MOBILE model is difficult and resource-intensive. Although some meth- ods have been developed and applied to estimate some of the components of uncertainty in MOBILE and related emissions models (described above under previous studies), there are intractable problems in estimating some of the components of the model's uncertainty. For example, it is difficult to assess the uncertainty arising from non-representative samples, or from incorrect model formulation. That a perfect assessment of uncertainty cannot be done, however, should not stop researchers from estimating the uncertainties that can be addressed quantitatively. To do so, EPA must archive all data used in the analyses leading to model formulation, and must carefully document all analysis approaches. As noted in this report, EPA has made substantial improvements in documenting modeling ap- proaches, but is not archiving all data sets used in MOBILES develop- ment. A critical unanswered question at the heart of the issues related to MOBILE's uncertainty and evaluation is how accurate the model needs to be to serve its various uses described in Chapter 2. This committee has not been able to identify any authoritative publication that says how close the model should be to reality. It is clear that EPA, working with the user community, should identify the level of accuracy desired for a particular application. For example, if it were determined that the current version of MOBILE is accurate enough to fulfill all of its roles (which the committee deems it is not), little further work in that direction is necessary. On the other hand, if a significant reduction in model uncertainty is warranted, then considerable work is suggested, and possibly a new design if the cur- rent approach is too limited (as is concluded here). There will not be a sin- gle answer, as various applications demand different levels of accuracy. Designing future emissions models should take into account the accuracy required for different applications. INTRODUCTION TO EVALUATION Considering the important applications of MOBILE, it is critical that MOBILE reflects actual emissions of the fleet of on-road vehicles. Much of
MODEL UNCERTAINTY AND EVALUATION 7 5 7 the knowledge and information used in creating MOBILE, have come from FTP-based testing of small numbers of vehicles. This information is weighted to reflect the amount of cold and hot starts and stabilized run- ning conditions estimated to be occurring in the real world, as described in Chapter 3. This approach may not produce estimates representative of the in-use fleet because of such factors as vehicle-recruitment bias (typically well over 75% of the selected vehicle owners refuse to participate in FTP testing), inaccurate and outdated characterization of the average real- world driving cycle, and small databases. For instance, the FTP driving cycle is based on driving conditions in Los Angeles in the 1960s. EPA has done little to test the accuracy of MOBILE other than conduct a recent comparative analysis and adjustment to state I/M test data, which is subject to inaccuracy in reflecting average real-world fleet emis- sions (EPA l999g; EPA l999m). Many different independent techniques could be used to help evaluate the accuracy of MOBILE. A wide range of studies, using a variety of techniques, can already provide more direct in- sight into the accuracy of MOBILE. Some of the results of these studies are presented in the following sections. In general, these studies indicate that MOBILE substantially underestimates VOC emissions, while NOX emissions appear to be more accurately predicted. This results in a dis- crepancy in the VOC to NOX ratio when comparing ambient observations to model predictions. MOBILE'S SENSITIVITY TO VARIATION IN DRIVING AND STARTS The New York State Department of Environmental Conservation (NYDEC) in 1994 and 1995 developed mobile-source inventories using microscale driving-cycle data (speeds, ambient temperature, engine oper- ating mode, and VMT by hour) for a wide range of counties in the state (Keenan and Escarpeta 1994~. This study can be used to help assess the sensitivity of MOBILE's vehicle emissions rates to changes in the driving cycle and ambient temperature. The results in Figure 4-4 illustrate MOBILE's VOC and NOX emissions ratio sensitivity to changes in traffic conditions. On the left, measured average speeds are shown for 17 differ- ent counties. On the right, MOBILE-predicted ratio of VOC to NOX corre- sponding to the local driving-cycle data on two different days (at two differ- ent temperatures) are shown. VOC to NOX ratios are indicated to rapidly rise as the VMT-weighted average vehicle speeds fall for counties ranging from very rural to highly urbanized. The figure shows that model esti- mated emissions are very sensitive to details of local driving conditions. The NYDEC study also found that MOBILE's driving-cycle default value for the engine thermal state operating mode (cold- and hot-start por- tion of VMT) was not representative of any of the wide range of traffic con-
7 52 M ODEIING M OB`[E-SOURCE EMISSIONS MAN HATTAN BROOKLYN QUEENs u, NASSAU A, STATEN IS.- BRONX- SUFFOLK - ROCKLAND - WESTCHESTER - PUTNAh ~ DUTCHES' ct ORANGE SULLIVAr COLUMBI, ULSTE' DELAWAR E i . ,, . ,,~. : r: . ~ . : ~ : :r : ·? : : : ~ : GREENE ~ 50 40 30 Speed (mph) 20 10 0 1 2 3 4 5 VOC / NOx o June 18 July 11 FIGURE 4-4 Emissions ratios derived from NYDEC study. The left panel shows the effects of increased population density on average driving speed for 17 counties. The counties start with the lowest population density at the bottom (Greene) and end with the highest at the top (Manhattan). The right panel shows the effect of applying the average speed data to MOBILE to predict the VOC/NOx mass ratio. These emissions show a large sensitivity to the details of local urban driving. Source: Keenan and Escarpeta 1994. ditions studied (very rural to highly urbanized counties in New York State), as indicated in Table 4-1. Because cold starts account for a signifi- cant portion of the VOC and carbon monoxide (CO) emissions of the total driving cycle, underestimating the frequency of cold starts can signifi- cantly contribute to the underprediction of VOC and CO emissions by MOBILE. In summary, the NYDEC study illustrates that slight inaccuracies in the assumptions, default values, or specific input of driving-cycle parame- ters to MOBILE (particularly speed input and the hot- and cold-start mode split) can lead to significant underprediction of real-world fleet VOC emis- sions. This indicates the need for input parameters to be based on obser- vations of local conditions. While the NYDEC study was based on MOBILES, the conclusions continue to be applicable to MOBILES, al- though MOBILES has been improved by separating cold start emissions from running emissions. A fundamental change to modal (instantaneous)
M ODEL UNCERTAINTY AND EVALUATION 7 53 TABLE 4-1 The Percent of Total VMT Allocated to Cold Starts and Hot Starts for Real-World Measurements Compared With MOBILE Predictions VMT Weighted Daily Average Area Cold Starts Hot Starts New York State (average) 25.6% 10.4% Upstate New York (rural areas) 27.1% 12.2% Downstate New York (metropolitan areas) 22.8% 9.5% MOBILE Default 20.6% 27.3% Source: Keenan and Escarpeta 1994. emissions modeling would be necessary to ensure that effects of local driv- ing conditions are accurately reflected on emissions. STATE EMISSIONS INSPECTION TESTING State I/M testing programs provide vehicle emissions test data for a large population of the real-world fleet. EPA has recently used data from selected state IM2403 tests to correct deterioration rates for running emis- sions in the upcoming MOBILES model (EPA l999m). EPA recognizes some limitations of IM240 data, such as the absence of cold starts and shortcomings in the driving cycle. But the agency argues that this tech- nique represents a census of all vehicles and, thus, is relatively unbiased. Figure 4-5 shows EPA's formula fitted adjustment to FTP-based testing results using the IM240 data. The revised model, based on IM240 data, results in a large change in predicted emissions of about +150% for VOCs, + 50% for CO, and +30% for NOX for 1988-1993 port-fuel injection cars, as illustrated in this figure. EPA indicates that existing IM240 data still suffer from lack of uniform preconditioning. If every vehicle were preconditioned with a full IM240 cycle prior to the IM240 test cycle, the results could be useful in evaluating the predicted MOBILES emissions rates. However, there are other serious limitations to state IM240 data that bias it away from true real-world 3The IM240 test differs from an idle or tailpipe test in that the emissions are measured while the vehicle is driving on a dynamometer. The test driving cycle that includes accelerations, cruise, idle and deceleration, operated is intended to assess emissions under conditions more similar to actual driving conditions that a test of an idle vehicle.
754 MODELING MOB!LE-SOURCE EMISSIONS HO Wow 12( ......... of CO thumb ,~ _ In - [~QX Urns 1~ _ 1~ : M' ~ ~100~ ~0 FrP~E:D O O O WIT .~ ~0 FIGURE 4-5 EPA adjustment to MOBILES FTP data based on the anal- ysis of IM240 data. Source: EPA l999g. emissions of the overall fleet. Thus, even a preconditioned set of l/M data is not ideal for testing MOBILE. Among limitations of the IM240 test as a measure of overall fleet emissions are that the test does not
M ODEL UNCERTAINTY AND EVALUATION 7 55 account for new vehicles because they are exempt from IM testing; account for unregistered vehicles; reflect the level of emissions between test cycles; reflect tampering after the test; reflect emissions from vehicles operating in the "prior to precondition- ing" state; reflect vehicles that "disappear" from the I/M program after failing the test but continue to operate in the urban area; reflect vehicles that register outside the I/M boundary to escape test- ing requirements but continue to be operated in the urban area; and . account for cold-start effects. Additionally a fully preconditioned IM240 data set would not provide any measure of cold-start emissions. Two elements are necessary for the accuracy of the MOBILE model regarding cold starts. The first is a reli- able measurement of the magnitude of cold-start emissions, including the dependence on mileage and age for both normal and high emitters, and the effect of soak time. The second is an accurate measure of the population of vehicles with high-start emissions. In MOBILES, EPA has assumed that the fraction of high emitters for start emissions is the same as the fraction of high emitters for running emissions. High emitters also might have cat- alyst malfunctions, including increases in light-off temperature. An in- crease in light-off temperature would have more of an effect on start emis- sions than on running emissions. The IM240 adjustment that EPA made for MOBlLE6 was one step toward improving estimates of deterioration rates and providing a more accurate reflection of real-worId conditions. However, this improved estimate directly applies only to running emis- sions. EPA should consider methods for improving its estimates of real- world start emissions in future versions of MOBILE. REMOTE SENSING Remote-sensing measurement of emissions of vehicles while they are in use provides another approach to generating a large database of real-worId emissions that can be used to test the accuracy of MOBILE. Remote sens- ing has the advantage of being able to capture a snapshot of a very large real-world fleet without the disadvantages of using IM240 data, which have been previously mentioned. Its main disadvantages are that it is does not provide an accurate measurement at low emissions levels and it reflects vehicles typically in a single mode of operation. A recent comprehensive analysis of Arizona IM240 data using a very large remote-sensing database confirms some of the limitations of using IM240 data to reflect real-world fleet emissions reductions from I/M pro-
756 MODEL/NO MOB/LE-SOURCE EM!SS/ONS grams (Wenzel 1999~. The study included 4 million readings of 1.2 million vehicles. The study compared the MOBl:LE and IM240s estimate of the emission reductions resulting from an I/M program versus estimates made from the remote-sensing data for vehicles subject to IM240 and remote sensing tests. The results, shown in Table 4-2, indicate emissions reduc- tions from I/M programs calculated from IM240 data are less than the re- ductions predicted by MOBIILE. Using remote sensing to estimate emis- sions reductions from I/M produces the smallest estimated emissions re- ductions. The discrepancies, particularly between results from IM240 and remote sensing data are likely due to about 33% of vehicles failing I/M testing (high emitters) do not return for retest (disappear), and 60% of those that fail continue to operate in the airshed after 6 months; and repaired vehicles have a high deterioration rate. ROADSIDE INSPECTION Roadside pullover studies in which a random sample of vehicles are pulled off the road and subject to a loaded-mode emissions tests could per- haps offers the best direct measure of the overall real-world fleet emis- sions rates if a sufficient sample size is collected. Because on-road vehicles are selected at random and emissions are measured under actual real- world conditions, it does not have most of the disadvantages of IM240 test- ing and remote sensing previously mentioned. TABLE 4-2 Comparison of Emission Reduction Estimates for an I/M Program Based on MOBILE, IM240, and Remote Sensing Data Emissions Reduction Method CO VOC NOX MOBILES Prediction 16.2% 16.9% 16.7% Arizona IM240 Data Analysis 14.5% 14.0% 7.1% Arizona Remote-Sensing Data 7% 11% Not measured Analysis Source: Wenzel 1999. 4A loaded-mode test is one that puts a car through a simulated driving cycle on a dynamometer.
M ODES UNCERTAINTY AND EVALUATION 7 57 The California Bureau of Automotive Repair (BAR) conducted random road-side pullover inspections of over 27,000 vehicles in 1997-1999. The objectives of the study are to help characterize fleet emissions and estab- lish a baseline for evaluating California's I/M program effectiveness. BAR recognized the potential inaccuracies of directly using I/M program data to assess real-world effectiveness of I/M. BAR is also conducting a similar series of tests after implementation of an enhanced I/M program in Cali- fornia to accurately assess the effectiveness of this program. When the results and analysis of this test program become available, they might pro- vide the best insight into the accuracy of emissions models such as MOBILE. AMBIENT AIR-QUALITY MONITORING AND MODELING Using MOBILE-generated emissions data in airshed modeling and com- paring the results with measured air-quaTity data offers yet another ap- proach to testing the accuracy of MOBILE emissions predictions. An ex- tensive ozone-modeling and emissions-inventory study comparison was made for the South Coast Air Basin in the Los Angeles area in 1987 (Chico et al 1993; Harley et al 1993b; Wagner and Wheeler 1993~. This study used California's EMFAC mobile-source emissions-inventory model. EMFAC has produced lower estimated emissions than previous versions of MOBILE, but the emerging MOBILES is expected to compare relatively closely with previous versions of EMFAC. Thus, the California study has relevance to evaluating the accuracy of MOBILE. This study found that airshed modeling substantially underpredicted ozone levels. Additionally, it was determined that when the on-road mobile-source VOC emissions, predicted by EMFAC, were multiplied by a factor of 2.5 and the airshed model rerun, the airshed model predictions of ozone closely matched ambi- ent measurements. Figure 4-6 shows the comparison between ambient observations and the airshed model run for both levels of VOC emissions. The South Coast study also compared CO to NOX ratios and VOC to NOX ratios derived from the EMFAC-airshed model predictions with ambient measurements in the Sherman Way tunnel in Van Nuys. Table 4-3 shows that the two measured values are similar, further indicating that the emissions model underpredicts CO and VOC emissions from motor vehi- cles by about a factor of 2. A more recent Desert Research Institute (DRI) study of ambient air- quality measurements versus emissions model estimates for Los Angeles (Zielinska et al 1999) indicates a deviation in the VOC to NOX ratios simi- lar to the findings in thel987 South Coast study. The results of this study found that the EMFAC model underestimates the VOC to NOX ratio of
7 58 M ODE[ING M OBILE-SOURCE EMISSIONS 40 20 _ E Cat 10 of o . ,, , ,., , ,., ,.,.,,,, , ,,,,,I,,,,, ,,., - 00 06 12 18 00 Ob 12 18 00 06 12 1B Aug 26 Aug 27 Aug 28 ,, ·, .,, ·,,,,, ·,,,, ·, ·, ·, ., · ~ ~ I · ~ ~ ~ ~ I .~'rr''' l, · I, ·,,,, · i I, I, I . ., . ~ . . ~ CLAR + OBSERVED BASE SENSITIVITY + ~1 Jar t I ~ ) I ) ~ I+ / 1 ~ 1 + J I Hi. i+ L ,+ LIT ~ , , 5 ~ , ,' ~ AL / ~ _ I A ~ _ FIGURE 4-6 Comparison of airshed model predictions of diurnal ozone con- centrations with observations of ambient ozone concentrations for two dif- ferent VOC emissions levels. Model-predicted VOC emissions are multi- plied by a factor of 2.5 in the sensitivity case to improve the fit to the am- bient ozone concentration profile. Observations are at Claremont (CLAR), California. Source: Fujita 1999. emissions by a factor of 2.15 compared with ambient data collected in Los Angeles in 1995. MOBILE6 will possibly make these ambient and emis- sions inventory discrepancies even worse. This conclusion is based on emissions data in EPA's paper (1998k) on possible regulatory action with respect to Tier 2 vehicle emissions and gasoline sulfur standards, in which EPA made adjustments to MOBILE5b to simulate the expected MOBILE6. MOBILE6 was unavailable to the committee to confirm if this conclusion is correct. The South Coast and DRI studies are thus consistent with other inde- pendent techniques (IM240 and remote sensing) in demonstrating that EMFAC (and, likely MOBILE as well) underestimates VOC emissions and, in particular, that it results in a significantly inaccurate estimate of the VOC to NOX emissions ratio of the real-world fleet.
M ODES UNCERTAINTY AND EVALUATION 7 59 TABLE 4-3 Ratio of Measured to Modeled Emissions, Using EMFAC7E Model Results, for Overall Ambient Measurements and Tunnel Studies R1_ I ~ ]~ LC° 1 LNOX measured 1.5 LC° NOX voc _ Modeled NOX Measured 2.5 CIVIC] LNOX UMOdeJed 2.1 2.2 Source: Fujita 1999. TUNNEL STUDIES The analysis of air samples in highway tunnels has been used for sev- eral years as a means of measuring vehicle emissions and testing model predictions (e.g., Pierson et al. 1990~. Tunnel studies capture the emis- sions from a large number (typically thousands) of in-use vehicles, thus providing measurements of the fleet's average emissions. Generally the vehicles are operating in a hot-stabilized cruise mode with average speeds from around 25 to 70 miles per hour (mph). Some tunnels have a signifi- cant grade. A report published by the Coordinating Research Council (CRC) (Gertler et al. 1997) summarizes results from a 1995 study in five different tunnels in Boston, New York City, Phoenix, and Los Angeles as wed as *om several previous urban tunnel studies. The CRC tunnel study results are summarized in Table 4-4, where the average emissions factors for CO, VOCs, and NOx are given, as well as the ratios of several emissions fac- tors. The sampled fleets were largely light-duty gasoline fueled vehicles. The data for the Fort McHenry and Tuscarora tunnels were obtained for a mixture of light-duty and heavy-duty vehicles and were analyzed to ex- tract the light-duty component reported in the table. The third column of the table shows a range of about 2-fold in the aver- age speed of the vehicles in the different tunnels. Emissions of CO, VOCs, and NOX differed substantially among the tunnels, typically about a factor
760 A ho CQ A · ~ a) ~ ox- Cal o m ~ V ·e o o a) . - ~ V ~ a) ED o AL > ~ tD Cal o ¢ V2 Ct I En Cal o V_~` V ~ V X ~ so ox I, Cal CO o ~ o Cal o ~ ~ ~ ~ ~ ~ ~4 ~ ~ o o o o o ~ ~ o o o o o o o ~ ~ o o Ct ~ Cal Go ~ ~ ~ o ~ -co ~ Do ~ 00 Dad ~ ~ ~ ~ ~ . . . . . . . . o o o o o ~ o o o ~ ~ ~ 0 Rae ~ cat 0 0 ~ co c~ co °° ~ o °° C~ ~ 00 ~ CO ~ ~ ~ C~ L~ ~ O C~ ~ C~ O ~ ~ ~ 00 · · · · ~7 O O ~ ~ ~ 00 C~ 0 00 00 ~ C9 ~ ~ CO . . O O O O ~ ~ O O O C~ ~ ~ ~ ~ ~ O O 1 - t~ ~ ~ C~ O O 00 O CO . . . . ~ 00 CD . . . . . c~ L~ ~ ~ ~D ~ ~ ~ ~ C~ 5- L~ ~ ~ ~ L~ ~ d4 ~ ~ Ct ~ ~ ~ ~ ~ 00 a a) ~n u2 ~ ~ ~ ~ ~ > ~, ~ ~ ~ O ~ O a, ~ · - ~ ~ ~ 0 ~ u2 ~ ~ V ~ V ~ E~ bD C~ ·,. 5- C~ Ct 5- C) 5- Ct C? q) ~Q E~ ._, C~ C~ C~ :^ ,Q o · - , CO .G d o 5- Ct a 3 ~ ~o ~ ~, U) {~5 >. C~ U) 5- J3 ~ ._, - Ct s~ s~ . . C) o U:
M ODEL UNCERTAINTY AND EVALUATION 7 6 7 of 4. These differences can be accounted for by the differences in the ages and modes of operation of the vehicles being tested. However, there is much less variation in the ratios of some of the pollutants, notably the VOC to NOx and CO to CO2 ratios. The CRC report discusses some incon- sistencies between these data and both the MOBILE and California's EMFAC emissions models. Although tunnel data provide only a snapshot of vehicle emissions, they can be valuable and should continue to be used in testing the accuracy of MOBILE and examining the effects of fuels, operating mode, and fleet composition. However, vehicle operation in tunnels tends to significantly deviate from average real-worId conditions. Tunnel traffic tends to have higher speeds, less stop-and-go driving, and less loaded-mode operation than average real-worId urban conditions. CHEMICAL-MASS BALANCE Chemical-mass balance (CMB) is another approach for evaluating emis- sions model estimates with ambient observations. CMB uses a sophisti- cated set of chemical fingerprints derived from speciation of source emis- sions, which are apportioned mathematically to ambient air samples. An oxidant assessment study in southeast Texas (Fujita et al. 1995) included extensive comparison CMB estimates versus MOBILE-estimated VOC and NOx emissions. This study distinguished contributions from liquid gasoline and gasoline vapor from vehicle exhaust in the ambient atmospheric measurements. The findings of this study and another CMB study (Korc et al. 1995) coun- ter arguments that it is evaporative emissions that account for MOBILE's underprediction of VOCs. The major conclusions of these studies are The sum of ambient liquid gasoline, gasoline vapors, industrial and compressed natural gas contributions agrees reasonably well with and val- idates the corresponding MOBILE emissions inventory estimates. The discrepancies between CMB ambient- and emissions-derived VOC to NOx ratios and ambient- and emissions-derived acetylene (a major tracer of fingerprint of motor vehicle exhaust) at the Clinton site suggest that the absolute amount of on-road mobile-source exhaust VOC emissions were substantially underestimated by MOBILE. The average ratio of CMB-derived ambient VOC emissions from mo- bile sources compared with those estimated from MOBILE at the Clinton site was 2.3. The latest state-of-the-art CMB study was conducted in the Denver area
7 62 M ODELING M OBI[E-SOURCE EMISSIONS in 1996 and 1997 under the Northern Front Range Air Quality Study (NFRAQS). The study was directed toward evaluating particulate matter (PM) emissions and sources; it indicated that PARTS greatly underesti- mates the contribution of gasoline vehicles compared with diesel vehicles. Tables 3-10 and 3-11 show this underestimation of emissions rates for LDVs by PARTS. The study also found a very large contribution of start emissions and high emitters to the total emissions from gasoline vehicles. Table 4-5 provides a summary of pertinent NFRAQS results. Underesti- mating emissions from starts and high emitters may be a significant cause of MOBILE's underprediction of other emissions (CO and VOCs) found in various studies. Watson et al. (in press) contains a recent summary of CMB studies. These studies tend to show that the relative contributions of mobile source VOC emissions to the total inventory determined by CMB are two to three times higher than those estimated using mobile source emissions factor models such as MOBILE and EMFAC. FUEL-BASED APPROACH TO EMISSIONS ANALYSIS Another approach to evaluating mobile-source emissions estimates from MOBILE is to compare MOBILE's results with those estimated through a fuel-based approach. The remote-sensing and tunnel-study methods de- scribed above measure exhaust emissions under operating conditions, but, unlike a dynamometer test, they do not measure the emissions on a gram per mile basis. The remote-sensing and tunnel studies measure the emis- sions concentrations of VOCs, CO, and NOx relative to the concentration of the combustion product carbon dioxide (COD. Carbon dioxide is the major carbon-containing product of fuel combustion, and thus the CO2 emission provides a measure of the amount of fuel burned. For improved accuracy, a correction is applied for the carbon in the VOCs and CO emissions. The determination of the concentrations of VOCs, CO, and NOx relative to CO2 provide measurements of these emissions relative to the amount of fuel consumed. These fuel-based emissions factors, measured in grams per gallon (g/gal), vary much less with changes in the vehicle mode of opera- tion than do the travel-based (g/mi) emissions factors (Singer and Harley 1996; Singer et al. 1999~. This might give the fuel-based emissions method an advantage over the travel-based method because the latter might not accurately represent the variations in the driving cycles of urban areas. Because the fuel-based emissions approach is less sensitive to the details of the vehicles' operation (speed and acceleration), this method is less sus- ceptible to inaccuracies derived from the MOBILE model's failure to repre- sent realistic urban vehicle operation.
MODEL UNCERTAINTY AND EVALUATION 7 63 TABLE 4-5 Comparison of CMB and PARTS Particulate Emissions from NFRAQS Study Vehicle Category Cold Starts CMB PARTS 32.5% 3/4 Non-Smokers 7.5% 29.0% High PM Emitter 31.3% 3/4 Diesel 28.8% 71.0% - Source: Fujita et al. 1998; Watson et al. 1998. The fuel-based emissions inventory is a concept developed from mea- surements of vehicle emissions on a gram per gallon of fuel. If the emis- sions amounts in terms of g/gal for a representative on-road fleet are known, then the fleet-wide emissions rate using the fuel consumption rate in gallons per unit of time can be calculated. Fuel consumption is deter- mined from fuel sales records. Knowledge of the fleet composition and the fuel economy of the different types of vehicles is also required to estimate the emissions inventory. As described in previous sections, the MOBILE model employs a travel- based method to develop emissions inventories. This requires knowledge of the emissions levels (g/mi) for different modes of driving or for represen- tative driving trips, vehicle activity or use, and the fleet composition. Both the travel-based and fuel-based methods require information on the in-use fleet composition, ambient temperature, and other factors that affect emis- sions and vary with geographical region. Singer and Harley (1996) developed a fuel-based CO inventory for the South Coast Air Basin in California and compared the results to Califor- nia's MVEl7F model. The fuel-based CO inventory estimate was of a fac- tor 2.2 times larger for cars and 2.6 times larger for trucks than the travel- based model. In a second study used more than 60,000 remote-sensing measurements made at 38 sites in Los Angeles between May and October 1997, to develop CO and VOC inventories using the fuel-based method (Singer and Harley 2000~. Their estimates for the on-road, fleet-stabilized exhaust emissions from cars and light- and medium-duty trucks are larger than the California MVEl7G model by factors of 2.4 ~ 0.2 for CO and 3.5 0.6 for VOCs. Similar tests of the MOBILE model emissions inventories should be made to test the model's consistency and accuracy. The fuel- based approach to emissions inventories is a promising method that can be used to reduce the uncertainties in emissions predictions.
64 MODEL/NO MOB/[E-SOURCE EMISSIONS SUMMARY OF FINDINGS AND RECOMMENDATIONS In this chapter, quantitative estimates of MOBILE's overall accuracy and uncertainty were not provided. The data are not available to do so with any confidence. Some studies have identified significant discrepancies in MOBILE predictions, but a complete assessment is not possible. Spe- cific findings and recommendations follow. Findings 1. EPA has done very little to use existing independent techniques to test the overall ability of MOBILE to accurately estimate real-world emis- sions of the overall on-road fleet. Others have applied many different tech- niques that test the accuracy of MOBILE. Most studies have found that MOBILES and earlier versions are substantially underpredicting VOC emissions (approximately by a factor of 2) and, to some extent, underpre- dicting CO and the PM contributions of gasoline-powered vehicles to the overall emissions. This is in contrast to NOx emissions, which appear to be more accurately estimated. 2. At present there is an inadequate understanding and quantification of the sources of uncertainties in MOBILE. These uncertainties arise from small and nonrepresentative emissions data, statistical analyses of these data, and assumptions that underlie and define the MOBILE model's aigo- rithms and predictions. Quantification of uncertainties is critical for un- derstanding the weaknesses in the model, and identifying the most critical needs for further emissions test data. 3. A critical unanswered question at the heart of issues related to MO- BILE's uncertainty and evaluation is how accurate the model needs to be to serve its various uses described in Chapter 2. It appears that EPA has not determined a desirable level of accuracy for MOBILE. It is clear that EPA, working with the user community, should determine the level of ac- curacy needed, and plan accordingly. For example, if it was determined that the current version of MOBILE is accurate enough to fulfill all of its roles (which it is not), little further work in that direction is necessary. On the other hand, if a significant improvement in model accuracy is de- manded, then considerable work is suggested, and possibly a new design if the current approach is too limited (as is concluded here). There will not be a single answer, as various applications demand different levels of accu- racy. Designing future emissions models should take required accuracy for different applications into account. 4. EPA has provided only limited sensitivity analyses for earlier ver- sions of MOBILE. Although the primary source of uncertainty in MOBILE
M ODEE UNCERTAINTY AND EVALUATION 7 65 output is in the underlying emissions data and data analyses used to gen- erate the model formulations, uncertainty in model inputs can also create uncertainty in model outputs. It is important that air-quality planners understand the sensitivity of model outputs to uncertainty in model inputs such as average speed, ambient temperature, fuel volatility, and I/M pro- gram parameters. 5. A dominant cause of MOBILE's underprediction of real-world emis- sions appears to be related to the driving-cycle testing protocol (because the FTP-based driving cycle is not representative of current driving pat- terns) and associated adjustment factors and default values forming the basis for MOBILE. In the real world, slower speeds, heavier-Ioaded oper- ating conditions associated with more congested stop-and-go driving condi- tions, and a more dominant role of cold starts, aD of which produce en- riched engine-operating conditions, appear to be the key factors in explain- ing the discrepancy of the real-world driving cycle compared with the aver- age driving cycle reflected by MOBILE. This observation indicates the es- sential need to move toward a true modal modeling approach. 6. Other important factors that appear to be significant sources of error in MOBILE are recruitment bias in vehicles that are FTP-tested and small databases for sensitive parameters. 7. Although corrections have been made to many parts of the emerging MOBILES model to improve its accuracy, it appears the end result might deviate even more than past versions of the model with respect to the VOC to NOx ratios. This will create, among other things, greater uncertainty in ozone modeling. Recommendations 1. EPA should assess the levels of accuracy needed to fulfill its regula- tory responsibilities and required for specific applications of the MOBILE model. EPA should compare the needed accuracy to the accuracy of the MOBILE model, and identify specific elements of the model that contrib- ute most to its inaccuracy. EPA should use the results of such an assess- ment to help guide the development of the next generation of models that would have improved accuracy in critical model components. 2. Enhanced model evaluation studies should begin immediately and continue throughout the long-term evolution and development of mobile- source emissions models. These studies need to be conducted to reduce gaps between model-predicted emissions and the resulting air quality, and also to reduce gaps between model-predicted emissions reductions from control programs, such as vehicle I/M programs, and those that actually occur in implementation. The evaluation should include (but not be lim-
7 66 M ODELING M OBI1E-SOURCE EMISSIONS ited to) field observations, including tunnel studies; remote-sensing mea- surements; source-receptor modeling; roadside pullovers; and air-quality monitoring and modeling; vehicle emissions testing data from vehicle I/M programs; and other vehicle emissions tests. Evaluation studies should be done with oversight and guidance from an independent body, including technical experts, and should be undertaken in tandem with the sensitiv- ity and uncertainty studies suggested in the next recommendations. 3. Rigorous sensitivity analyses should be performed for all model in- puts and provided as part of user guides for MOBILES and all future ver- sions of MOBILE. From these sensitivity analyses, EPA should provide guidance to transportation and air-quality planners on the most critical model inputs affecting model results. 4. EPA, along with other agencies and industries, should undertake the necessary measures to conduct quantitative uncertainty analyses of the mobile-source emissions models in the modeling toolkit (discussed in Chapter 6), especially the MOBILE model. Future versions of the MOBILE model and other models in the toolkit should be developed to fa- cilitate uncertainty analyses. Results of the uncertainty analyses should be used to guide research plans for obtaining additional test data that would increase the accuracy of the model.
