|
Items in cart [4] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 135
4
Mode' Uncertainly and Evaluation
MOBILE IS A TOOL for estimating current and forecasting future mobile-
source emissions, including quantifying the effects of control measures.
These results form key elements of many air-quality regulatory compli-
ance programs and directly affect transportation planning and the selec-
tion of control strategies. Thus, there is a need for a high degree of accu-
racy from MOBILE.
One of the specific charges to this committee is to assess, to the extent
practical, the overall accuracy of the current version of the MOBILE model
in predicting atmospheric emissions. Such information is derived from
model evaluation. However, the necessary information to quantitatively
evaluate the current or upcoming versions of MOBILE with great confi-
dence does not exist. This lack of information is one of the most serious
concerns with MOBILE and its use.
Assessing accuracy in the context of MOBILE involves model evalua-
tion, typically by comparing real-world emissions with model predictions.
It also involves considering uncertainty and bias that arise *om the wide
variety of observations, assumptions, and mathematical relationships that
underlie MOBILE model algorithms. This chapter discusses uncertainty
and model evaluation, and reviews previous studies on these topics.
iThis report uses model evaluation in reference to assessing the ability of a model
to accurately represent the real world, for example by being able to estimate the
emissions from mobile sources with little error. This differs from model validation
that refers to assessing the correctness of the form of the model. MOBILE is based
735
OCR for page 136
7 36 MODELING MOB/[E-SOURCE EMISSIONS
DEFINITION OF TERMS
Accuracy, Uncertainty, and Bias
Accuracy in MOBILE refers to its ability to correctly estimate the true
value of emissions. Bias is the tendency for estimates to be consistently
higher or lower than the true values. Uncertainty is the variability (or
scatter) in MOBILE's prediction about the actual emissions. MOBILE is
accurate, for example, if it predicts the correct emissions factors for the
fleet of on-road vehicles, and if it predicts the actual changes that result
from mobile-source emissions control programs such as inspection and
maintenance (I/M). (MOBILE, as typically applied, provides only point
estimates without any statistical confidence intervals around those esti-
mates). MOBILE predictions can be assessed by comparing accurate in-
use measurements of vehicle emissions and air quality to model predic-
tions. This is termed model evaluation and is the subject of the second
half of this chapter.
Uncertainty and bias in MOBILE arise from many sources, primarily
from the data used to construct the model, and from errors in analyses and
assumptions leading to model formulations (discussed further below). Un-
certainty and bias in MOBILE are difficult to assess because of the com-
plexity of the model, the uncertainty in the underlying emissions data and
model formulation, the uncertainty in the input data, and the difficulty in
obtaining accurate measures of real-world emissions (e.g., from analyzing
ambient data).
Uncertainty, as used here, should not be confused with repeatability. If
MOBILE is run several times with the same set of inputs, the model will
always generate exactly the same output. The model is repeatable simply
because it has no stochastic or random component; this does not by any
means imply that the model is accurate.
Figure 4-1 shows the difference between bias and uncertainty. The top
box in Figure 4-1 shows a case in which the model provides accurate re-
sults. The bottom box shows estimates that are uncertain (in that they
are scattered about a one-to-one correspondence) but unbiased. The mid-
dle box shows estimates that are biased; on average the predictions are
below the actual emissions. For MOBILE's many uses, it is important
that the model's estimates be accurate with a high degree of certainty and
on statistical analyses of data, and the form of MOBILE is a set of statistical rela-
tionships. Thus, model validation would correspond to determining whether the
statistical relationships were derived in a valid manner and implemented correctly.
Model evaluation determines whether those relationships provide accurate results.
OCR for page 137
M ODES UNCERTAINTY AND EVALUATION 7 3 7
0.50
0.40
-
~e 0.30
o
-
LO
E 0.20
0.10
0.00
0.50 -
0.40-
._
c
a
._
In
._
-
-
~ 0.10- /
/.
0.00 -
0.00
0.30 -
0.20 -
Accurate
/
-
/
/
,,'
/
. . . .. . . . .. ... /
,. ./, , . . 1
0.00 0.10 0.20 0.30 0.40 0.50
Predicted Emissions (g/mi)
Biased
7
·-
~ ...
, . ~
1
0.10 0.20 0.30 0.40 0.50
Predicted Emissions (g/mi)
Uncertain
0.50
0.40-
E
0.30-
o
._
u'
E 0.20
I
as 0.10
0.00 -
/
0.00 0.10 0.20 0.30 0.40 0.50
Pre d icte d Eon is s ions (g/m i)
· Series1
Series1
Series1
FIGURE 4-1 Representation of bias and uncertainty (hypothetical data).
OCR for page 138
~ 38 M ODEL/NG MOB!LE-SOURCE EMISSIONS
a low bias. If not, transportation and air-quality planners could be led to
implement costly, unnecessary control programs.
Sensitivity
Model sensitivity refers to the variation in model output in response to
changes in model inputs such as average speed, ambient temperature, fuel
volatility, and I/M program parameters. It is important that air-quality
planners understand the effects of changes in these model inputs. The
U.S. Environmental Protection Agency (EPA) has provided sensitivity
analyses for some earlier versions of MOBILE (see discussion below), but
not extensive analyses for MOBILES. A comprehensive sensitivity analy-
ses should be performed for all model inputs and provided as part of user
guides for all future versions of MOBILE.
TYPES AND SOURCES OF UNCERTAINTY AND ERROR
Many kinds of uncertainty plague the emissions estimates provided by
MOBILE. A major fraction of those uncertainties arise from limitations in
the scientific and technical basis of MOBILE as well as the data inputs,
including sampling and measurement errors. As described above, the pri-
mary source of uncertainty in MOBILE output is in the underlying emis-
sions data used to generate the model formulations. There can also be
limitations in MOBILE's structure. Input uncertainties, including data on
vehicle characteristics and usage, are propagated through the model and
contribute substantially to uncertainty. Another source of uncertainty
that should be analyzed is the true variability in the system being mod-
eled.
It is clear that it is not practical to eliminate all uncertainty from
mobile-source emissions models. MOBILE is a statistically based model,
and its accuracy depends on valid and comprehensive samples as the foun-
dation of the statistical relationships within. Some uncertainty comes
from the random variation in the relatively small samples. No sample
short of 100% (a complete census) can be large enough to completely elimi-
nate randomness from affecting parameter error, but in all practical cir-
cumstances, the influence of random variation on an estimate of a parame-
ter decreases as the sample size gets larger. Even if 100% of the on-road
vehicles were tested, uncertainty would still arise from the incompleteness
and unrepresentativeness of the tests and errors in the testing procedure.
Analysts must also consider true variability and its impact on uncer-
tainty. Within defined fleets, manufacturers do not produce exactly the
same kinds of vehicles. Vehicles from a single manufacturer are not iden-
OCR for page 139
MODEL UNCERTAINTY AND EVA[UAT!ON 7 39
tical. Also, emissions from any one vehicle vary from time to time and
place to place in ways that are correlated with age, mode of driving, and
many other things. Fleet characteristics vary spatially, as do topography
and driving habits. The contribution of this true variability to the overall
uncertainty of MOBILE is unknown, but could be partially characterized
in an advanced mobile-source emissions model if the data (both input and
model formulation) were available.
Uncertainties further arise from limitations inherent in the model's
structure. For example, no fleet of vehicles is well characterized. Road-
way networks and associated driving patterns are not perfectly repre-
sented. MOBILE does not capture all the factors leading to emissions
(particularly high emissions, such as those induced by steep road grades).
These and other model limitations discussed in Chapter 3 influence uncer-
tainty.
Following are descriptions of the general types of uncertainties that
arise in the MOBILE model. Several examples of each type are provided,
for both MOBILES and MOBILES. For additional information, see the
study by Wenzel et al. (In press), which discusses emissions variability in
more detail, and also describes other issues that complicate the statistical
analysis of vehicle-emissions test data.
Nonrepresentative Vehicle Samples
MOBILE algorithms and emissions factors are based largely on test
data from in-use vehicles that are solicited through the mail or by recruit-
ment at I/M test stations. Typically, the owner is provided with a small
payment and a rental vehicle until testing is completed. Free vehicle re-
pairs are sometimes provided as an incentive. Response rates for such
recruitment efforts are very low, typically less than 25% and sometimes as
low as 5%.
Recruited vehicles have serious bias issues because high emitters and
tampered vehicles as well as expensive luxury vehicles are less likely to be
voluntarily submitted for testing. As discussed in Chapter 3, very high
emitting vehicles are a relatively small fraction of the on-road vehicle fleet,
but they contribute a very large fraction of total vehicle emissions. Emis-
sions from high-emitting vehicles are much more variable than emissions
from normal emitters (Knepper et al. 1993), and thus require a large sam-
pling fraction to obtain reasonably accurate estimates of their emissions
(and to estimate the effects of control programs for their emissions). It is
thus critically important that such vehicles be appropriately represented
in emissions testing programs. If emissions from high emitters are not
properly characterized, then MOBILE emissions factors can be seriously
biased.
OCR for page 140
740 MODELING MOBILE-SOURCE EMISSIONS
EPA recognizes that underrepresentation of high-emitting vehicles in
the Federal Test Procedure (FTP) and IM240 databases underlying
MOBlLE5's basic emissions rates is a serious shortcoming that produces
biased Jow) emissions estimates. For MOBILES, EPA is proposing to ad-
just the basic emissions rates based on data from the Dayton, Ohio, IM240
program. Although this is a step in the right direction, there are still bi-
ases in the Dayton IM240 data. One type of bias arises from noncompli-
ance with the program - the database does not include emissions from ve-
hicles that are registered in the I/M area but do not obtain the required
inspection (and repair if needed). Remote sensing studies have shown that
these noncomplying vehicles have higher than average emissions (e.g.,
Stedman et al. 1997; Stedman et al. 1998; Wenzel 1999~. In fact, the I/M
program compliance rate is one of the inputs to MOBILE, and the model
assumes that noncomplying vehicles have emissions about twice as high as
complying vehicles. A second problem is that when the Dayton program
was first implemented, there is strong evidence that owners of vehicles
that had been registered in the I/M area changed their registration to sur-
rounding counties that were not subject to the Dayton I/M program
~IcClintock 1999~. Again, these vehicles are more likely to be higher
emitting vehicles. Although EPA recognizes the inherent biases in their
testing data, use of the Dayton IM240 data to adjust for those biases still
likely results in biased emissions estimates.
Some of the MOBILE algorithms and correction factors are based on
very small samples, which may be nonrepresentative of the population.
An example for MOBILES is the test data used to derive air-conditioning
correction factors for light-duty vehicles (LDVs). As described in Chapter
3, the effects of full air-conditioning operation on vehicle emissions were
determined from a sample (of 37 vehicles) 23 passenger cars and 14
light-duty trucks, al] from model years 1990 to 1996. Only five of the vehi-
cles were high emitters for at least one pollutant. Similarly, the activity
data for air-conditioning operation were obtained from a fleet of only 20
vehicles operating in Phoenix, Arizona, from August to October 1994.
And, although the actual air-conditioning operation depends on the torque
generated by the compressor, no data were available for this variable.
Variability in Vehicle Emissions
Vehicle emissions are highly variable, for a variety of reasons. Two ve-
hicles of the same make and manufacturer, model year, technology, and
accumulated mileage can have very different emissions measured on the
same test or drive cycle. Such variation can be caused by factors such as
how the vehicle has been driven and maintained, prior tampering with
emissions control system components, and repeated excessive driving
OCR for page 141
MODEL UNCERTAINTY AND EVALUATION 7 4 7
loads. Studies have also shown relationships between socioeconomic fac-
tors and vehicle emissions, with vehicles in lower than median income
households exhibiting higher than average emissions (Singer and Harley
2000).
Vehicles of the same age and technology also exhibit differences in emis-
sions because of manufacturing and emissions-control design differences.
Analyses of I/M data have shown that specific vehicle models have much
lower or much higher emissions than average (Wenzel 1997~. In general,
vehicles with higher emissions exhibit much more test-to-test variability
than lower-emitting vehicles (Bishop and Stedman 1996~.
Many factors can contribute to variability in repeated emissions tests of
the same vehicle. For example, failures of some emissions control system
components (such as a partially degraded catalyst) can be intermittent
and therefore result in higher emissions some of the time. Other sources
of differences in the vehicle emissions test data underlying the MOBILE
model arise from the testing process. For example, back-to-back emissions
tests will vary because of differences in the measurement equipment, cali-
brations, and personnel (e.g., driving styles in tracking a target speed-time
trace on a dynamometer).
Those and other factors create uncertainties in the statistical models
fitted to the data that are the basis of MOBILE emissions factors. An ex-
ample of the enormous scatter in the emissions test data underlying
MOBILE is shown in Figure 4-2. The figure shows the test data used to
estimate NOX emissions for Tier 1 LDVs in MOBILE6. The solid line in
the middle of the figure corresponds to 2 times the FTP standard; vehicles
with emissions above this level are considered high emitters. Partly be-
cause of the lack of sufficient data, and partly because EPA assumes that
high emitters are "broken" vehicles whose emissions are always high no
matter how old the vehicles are, emissions for the high emitters are mod-
eled as a constant (the upper dashed line). This emissions estimate is then
adjusted with the Dayton IM240 data as described above. Below the solid
line, the vehicles are considered to be normal emitters. For these vehicles,
the basic emissions rate is modeled as a linear function of accumulated
mileage (the lower dashed line). Clearly there is large uncertainty in the
emissions data and consequently in the basic emissions rates for normal-
and high-emitting vehicles estimated from these data.2 These vehicle-to-
vehicle differences are critical for some uses of MOBILE, but the scatter
2EPA has updated their original analysis of these data for Tier 1 NOx emissions
using additional data sets. The additional data reduces the confidence limits (al-
though not the scatter) for the normal emission regression equation. However, there
is still significant uncertainty in the estimated mean value for high emitters.
OCR for page 142
742
-
U]
._
I:
o
.
.
~ .
— 1
~ 1
~ ,
1;: 1
,o
.!2 1
~ 1
I: 1
~ ,
~ 1
~ ,
~ 1
At
is !
it
1
.
.
.
.
·.
.
~ , ~
Cal O ~ 10 ~ ~ O
~ ~ ~ ~~ ~ ·
· O
2
1 ~ ~
1
l ~
~ 41
t ~
1 ~
· i
l
.
. ~
, .
.
· t
.
$:
LU
It
it
l.
;
~ t=.-
. .' . ~
:~N
·.~ L
lo
N
EN
004
it_
._'
A
A
o
. -
A
·_'
A
C)
. _~
~5
O
_ _
do
lo
lo
lo
._
o
of ~
· .
O O
IwI6
~ N O
O O O
CO
o
y
Fl
a
C~
a
a)
~D
a)
Ce
C~
CQ
¢
~Q ~
O ..
:m ~
~ O
a, v
C) -
.o
a
1
OCR for page 143
M ODEL UNCERTAINTY AND EVALUATION 7 43
tends to get smoothed out in aggregated data, such as estimates of area-
wide burdens. What does not get smoothed out is consistent bias, such as
systematic under-estimation of the emissions from high-emitters.
Incorrect Model Formulation
The emissions factors, emissions-factor adjustments, and estimates of
emission control program effects in MOBILES and its predecessors are
estimated from statistical analyses of available test data. The statistical
models are chosen using both engineering considerations (to represent the
physical process) and statistical considerations (such as selecting the
model that produces the best statistical correlation). Uncertainties can
arise from incorrect or inappropriate engineering and statistical models.
Examples of incorrect physical-engineering models include the following:
Neither MOBILES nor MOBILES includes road-grade effects, as
these are not incorporated into the test data underlying the models. Road-
grade influences vehicle emissions; higher emissions occur under vehicle
load conditions such as steep road grades.
Light-duty truck emissions are sometimes estimated from automobile
test data, because of the lack of sufficient data for trucks. Although some
light-duty trucks are used (and may emit like) passenger vehicles, other
light-duty trucks are used regularly as working vehicles and frequently
carry heavy loads; emissions from these working trucks are likely to be
higher than automobiles of the same model and age.
In MOBILE, the lack of sufficient data to estimate the effects of high
emitters is sometimes filled in by assuming that high emitters behave as
normal emitters do.
In EPA's multistep adjustment of basic emissions rates for high emit-
ters using Dayton IM240 data, one step is to estimate full IM240 emis-
sions from fast-pass IM240 data (see Glossary for definition of fast-pass).
This is done using second-by-second IM240 data from Wisconsin, so simu-
lated fast-pass emissions can be compared to full IM240 emissions (EPA
l999g). The regression equation developed by EPA has the time (in sec-
onds) of the fast-pass as one of the predictors of the full IM240 emissions;
full IM240 emissions are assumed to be linearly related to the logarithm of
fast-pass emissions. Given the variation in the speed-time trace of the
IM240 data, there is no physical reason why fast-pass time should be lin-
early related to full IM240 (or to the log of full IM240) emissions. How-
ever, the coefficient for the fast-pass time is statistically significant in the
regressions for all three pollutants (CO, VOCs, and NOX). This appears to
be because newer vehicles are much more likely to pass early in the test
than older vehicles (Pollack et al. 1999a).
OCR for page 144
744 MODELING MOBILE-SOURCE EM`SS/ONS
Inaccurate physical-engineering models can also arise from the failure to
obtain correct data in a test program. Examples of that include the follow-
~ng:
MOBILES speed-correction factors (SCFs) were determined from
emissions of vehicles driving over a series of test driving cycles. The driv-
ing cycles were characterized only by average speed, although, as dis-
cussed in Chapter 3, there are many other factors in a driving pattern that
also affect vehicle emissions. The SCFs used in MOBILE6 are facility-spe-
cific. For each facility, driving cycles were developed from analyses of real-
world (instrumented vehicles) driving-pattern data. Although there is
more aggressive driving in these cycles, and they are facility-specific, the
cycles are still characterized for a given facility, by a single parameter-
average speed. The development of modal (or second-by-second) emissions
models (discussed in Chapter 5) will go a long way towards resolving these
Issues.
The FTP cycle and the facility-specific speed correction factor cycles
of MOBILE6 represent samples of a universe of driver and vehicle behav-
ior. That universe may have low- frequency events with extremely high
emissions. If these low-frequency high-emission events are not properly
represented in the cycles used for MOBILE, an accurate picture of exhaust
emissions will not be obtained.
Some of the data critical to estimation of air-conditioning effects on
emissions were not available for use in MOBILE6. These include the ef-
fect of vehicle speed, the time the compressor is on, the validity of the heat
index as a measure of air-conditioner use, car occupant behavior in air-
conditioner use, or the effect of the actual compressor torque.
Examples of incorrect statistical models include the following:
In some analyses, the intercept of the statistical model is forced
through zero, to match physical processes; this represents a trade-off be-
tween the correct statistical model and the correct physical model. A1-
though such model alteration makes sense from an engineering point of
view, it introduces bias into the resulting statistical model.
Two examples of this practice in MOBILE6 are the following:
The determination of hot-running emissions from FTP test data; the
test data are first transformed using logarithms, then the intercept is
forced to be zero (EPA l999k).
MOBILE6 uses Dayton IM240 data to adjust FTP test data to ac-
count for the absence of high emitters in the FTP sample. The exhaust-
OCR for page 145
M ODEE UNCERTAINTY AND EVALUATION 7 45
emissions rate adjustment is treated as an additive function of mileage,
with zero increase in the adjustment at zero mileage (EPA l999g).
Logarithmic transformations are frequently applied to emissions data
for fitting statistical models to bring the emissions data closer to the nor-
mal (Gausian) distribution, for which most statistical methods are devel-
oped. The transformed data may still be non-normal, and the translation
of results back into the original scale may be hard to interpret.
In addition, the log transformation has the effect of reducing the influ-
ence of high values, which may be of greatest concern. Distributions of
vehicle emissions typically show the majority of the emissions from normal
emitters at relatively low levels, and a small fraction of the emissions at
very high levels. These distributions with long right tails are much closer
to lognormal than normal, although other distributions such as gamma
have been fit as well (Zhang et al. 1994~. Another reason that logarithmic
models are commonly fit is that variability in vehicle emissions is higher
at higher emissions levels; logarithmic transformations stabilize the vari-
ance. However, logarithmic transformations are usually inappropriate in
analyses of data sets with both normal and high emitters. With Togarith-
mic transformations, the effects of high emitters are minimized relative to
normal emitters, whereas in the atmosphere the effects of high emitters
are, in fact, much greater than normal emitters. In summary, logarithmic
transformations offer some convenience in analysis and the opportunity to
use simple statistical models, but at a cost of introducing potentially seri-
ous errors.
Figure 4-3 shows an example from MOBILES of an inappropriate use of
a logarithmic model. The data in the figure show hydrocarbon (HC) emis-
sions from model-year 1993 fuel-injected cars from a data set of Wisconsin
IM240 second-by-second emissions; these data were used as part of the
multistep process to adjust basic emissions rates using Dayton IM240 data
(EPA l999m). The Dayton data contain many fast-pass emissions tests,
and the Wisconsin data were used to develop regression equations to pre-
dict fast-pass emissions (FHC on the horizontal axis) to full IM240 emis-
sions (HC on the vertical axis). The graph on the left is a scatterplot of the
full IM240 HC emissions against the fast-pass HC emissions; the top and
right sides of the figure provide histograms of each of these variables indi-
vidually. Clearly these emissions measurements are not normally distrib-
uted. The right graph shows the same data, but with a logarithmic trans-
formation applied to both the full IM240 (LHCnz on the vertical axis) and
fast-pass HC (LFHC on the horizontal axis) measurements. Both the scat-
terplot and the marginal histograms show that the logarithmic transfor-
mation is an over-transformation it diminishes the effects of the high-
OCR for page 156
756 MODEL/NO MOB/LE-SOURCE EM!SS/ONS
grams (Wenzel 1999~. The study included 4 million readings of 1.2 million
vehicles. The study compared the MOBl:LE and IM240s estimate of the
emission reductions resulting from an I/M program versus estimates made
from the remote-sensing data for vehicles subject to IM240 and remote
sensing tests. The results, shown in Table 4-2, indicate emissions reduc-
tions from I/M programs calculated from IM240 data are less than the re-
ductions predicted by MOBIILE. Using remote sensing to estimate emis-
sions reductions from I/M produces the smallest estimated emissions re-
ductions. The discrepancies, particularly between results from IM240 and
remote sensing data are likely due to
about 33% of vehicles failing I/M testing (high emitters) do not return
for retest (disappear), and 60% of those that fail continue to operate in the
airshed after 6 months; and
repaired vehicles have a high deterioration rate.
ROADSIDE INSPECTION
Roadside pullover studies in which a random sample of vehicles are
pulled off the road and subject to a loaded-mode emissions tests could per-
haps offers the best direct measure of the overall real-world fleet emis-
sions rates if a sufficient sample size is collected. Because on-road vehicles
are selected at random and emissions are measured under actual real-
world conditions, it does not have most of the disadvantages of IM240 test-
ing and remote sensing previously mentioned.
TABLE 4-2 Comparison of Emission Reduction Estimates for an I/M
Program Based on MOBILE, IM240, and Remote Sensing Data
Emissions Reduction Method CO VOC NOX
MOBILES Prediction 16.2% 16.9% 16.7%
Arizona IM240 Data Analysis 14.5% 14.0% 7.1%
Arizona Remote-Sensing Data 7% 11% Not measured
Analysis
Source: Wenzel 1999.
4A loaded-mode test is one that puts a car through a simulated driving cycle on
a dynamometer.
OCR for page 157
M ODES UNCERTAINTY AND EVALUATION 7 57
The California Bureau of Automotive Repair (BAR) conducted random
road-side pullover inspections of over 27,000 vehicles in 1997-1999. The
objectives of the study are to help characterize fleet emissions and estab-
lish a baseline for evaluating California's I/M program effectiveness. BAR
recognized the potential inaccuracies of directly using I/M program data to
assess real-world effectiveness of I/M. BAR is also conducting a similar
series of tests after implementation of an enhanced I/M program in Cali-
fornia to accurately assess the effectiveness of this program. When the
results and analysis of this test program become available, they might pro-
vide the best insight into the accuracy of emissions models such as
MOBILE.
AMBIENT AIR-QUALITY MONITORING AND MODELING
Using MOBILE-generated emissions data in airshed modeling and com-
paring the results with measured air-quaTity data offers yet another ap-
proach to testing the accuracy of MOBILE emissions predictions. An ex-
tensive ozone-modeling and emissions-inventory study comparison was
made for the South Coast Air Basin in the Los Angeles area in 1987 (Chico
et al 1993; Harley et al 1993b; Wagner and Wheeler 1993~. This study
used California's EMFAC mobile-source emissions-inventory model.
EMFAC has produced lower estimated emissions than previous versions of
MOBILE, but the emerging MOBILES is expected to compare relatively
closely with previous versions of EMFAC. Thus, the California study has
relevance to evaluating the accuracy of MOBILE. This study found that
airshed modeling substantially underpredicted ozone levels. Additionally,
it was determined that when the on-road mobile-source VOC emissions,
predicted by EMFAC, were multiplied by a factor of 2.5 and the airshed
model rerun, the airshed model predictions of ozone closely matched ambi-
ent measurements. Figure 4-6 shows the comparison between ambient
observations and the airshed model run for both levels of VOC emissions.
The South Coast study also compared CO to NOX ratios and VOC to NOX
ratios derived from the EMFAC-airshed model predictions with ambient
measurements in the Sherman Way tunnel in Van Nuys. Table 4-3 shows
that the two measured values are similar, further indicating that the
emissions model underpredicts CO and VOC emissions from motor vehi-
cles by about a factor of 2.
A more recent Desert Research Institute (DRI) study of ambient air-
quality measurements versus emissions model estimates for Los Angeles
(Zielinska et al 1999) indicates a deviation in the VOC to NOX ratios simi-
lar to the findings in thel987 South Coast study. The results of this study
found that the EMFAC model underestimates the VOC to NOX ratio of
OCR for page 158
7 58 M ODE[ING M OBILE-SOURCE EMISSIONS
40
20 _
E
Cat
10
of
o
. ,, , ,., , ,., ,.,.,,,, , ,,,,,I,,,,, ,,., -
00 06 12 18 00 Ob 12 18 00 06 12 1B
Aug 26 Aug 27 Aug 28
,, ·, .,, ·,,,,, ·,,,, ·, ·, ·, ., · ~ ~ I · ~ ~ ~ ~ I .~'rr''' l, · I, ·,,,, · i I, I, I . ., . ~ . . ~
CLAR
+ OBSERVED
BASE
SENSITIVITY
+
~1
Jar
t I
~ )
I )
~ I+
/ 1
~ 1 +
J I
Hi.
i+
L
,+
LIT ~ , , 5 ~ , ,' ~ AL
/ ~ _
I A ~ _
FIGURE 4-6 Comparison of airshed model predictions of diurnal ozone con-
centrations with observations of ambient ozone concentrations for two dif-
ferent VOC emissions levels. Model-predicted VOC emissions are multi-
plied by a factor of 2.5 in the sensitivity case to improve the fit to the am-
bient ozone concentration profile. Observations are at Claremont (CLAR),
California. Source: Fujita 1999.
emissions by a factor of 2.15 compared with ambient data collected in Los
Angeles in 1995. MOBILE6 will possibly make these ambient and emis-
sions inventory discrepancies even worse. This conclusion is based on
emissions data in EPA's paper (1998k) on possible regulatory action with
respect to Tier 2 vehicle emissions and gasoline sulfur standards, in which
EPA made adjustments to MOBILE5b to simulate the expected
MOBILE6. MOBILE6 was unavailable to the committee to confirm if this
conclusion is correct.
The South Coast and DRI studies are thus consistent with other inde-
pendent techniques (IM240 and remote sensing) in demonstrating that
EMFAC (and, likely MOBILE as well) underestimates VOC emissions
and, in particular, that it results in a significantly inaccurate estimate of
the VOC to NOX emissions ratio of the real-world fleet.
OCR for page 159
M ODES UNCERTAINTY AND EVALUATION 7 59
TABLE 4-3 Ratio of Measured to Modeled Emissions, Using EMFAC7E
Model Results, for Overall Ambient Measurements and Tunnel Studies
R1_ I ~ — ]~
LC° 1
LNOX measured 1.5
LC°
NOX
voc
_
Modeled
NOX Measured 2.5
CIVIC]
LNOX UMOdeJed
2.1
2.2
Source: Fujita 1999.
TUNNEL STUDIES
The analysis of air samples in highway tunnels has been used for sev-
eral years as a means of measuring vehicle emissions and testing model
predictions (e.g., Pierson et al. 1990~. Tunnel studies capture the emis-
sions from a large number (typically thousands) of in-use vehicles, thus
providing measurements of the fleet's average emissions. Generally the
vehicles are operating in a hot-stabilized cruise mode with average speeds
from around 25 to 70 miles per hour (mph). Some tunnels have a signifi-
cant grade.
A report published by the Coordinating Research Council (CRC)
(Gertler et al. 1997) summarizes results from a 1995 study in five different
tunnels in Boston, New York City, Phoenix, and Los Angeles as wed as
*om several previous urban tunnel studies. The CRC tunnel study results
are summarized in Table 4-4, where the average emissions factors for CO,
VOCs, and NOx are given, as well as the ratios of several emissions fac-
tors. The sampled fleets were largely light-duty gasoline fueled vehicles.
The data for the Fort McHenry and Tuscarora tunnels were obtained for a
mixture of light-duty and heavy-duty vehicles and were analyzed to ex-
tract the light-duty component reported in the table.
The third column of the table shows a range of about 2-fold in the aver-
age speed of the vehicles in the different tunnels. Emissions of CO, VOCs,
and NOX differed substantially among the tunnels, typically about a factor
OCR for page 160
760
A
ho
CQ
A
· ~
a)
~ ox-
Cal
o
m
~ V ·e
o
o
a)
. -
~ V ~
a)
ED
o
AL >
~ tD
Cal
o ¢ V2
Ct
I
En
Cal
o
V_~`
V ~
V X
~ so
ox
I,
Cal CO o ~ o Cal o
~ ~ ~ ~ ~ ~ ~4 ~ ~
o o o o o ~ ~ o o
o o o o o ~ ~ o o
Ct ~
Cal Go ~ ~ ~ o ~ -co ~
Do ~ 00 Dad ~ ~ ~ ~ ~
. . . . . . . . —
o o o o o ~ o o o
~ ~ ~ 0 Rae ~ cat 0 0
~ co c~ co °° ~ o °° C~
~ 00 ~ CO ~ ~ ~ C~
L~ ~ O
C~ ~ C~ O ~ ~ ~ 00
· · · · ~7
O O
~ ~ ~ 00 C~
0 00 00 ~ C9 ~ ~ CO
. .
O O O O ~ ~ O O
O
C~ ~
~ ~ ~ ~ O O
1 - t~ ~ ~ C~ O O
00
O CO
. . . .
~ 00
CD
. . . . .
c~ L~ ~ ~ ~D
~ ~ ~ ~ C~
5- L~ ~ ~ ~ L~ ~ d4 ~ ~
Ct ~ ~ ~ ~ ~ 00
a
a)
~n
u2 ~ ~ ~ ~ ~
> ~, ~ ~ ~ O ~ O
a, ~ · - ~ ~ ~ 0 ~
u2 ~ ~ V ~ V ~ E~
bD
C~
·,.
5-
C~
Ct
5-
C)
5-
Ct
C?
q)
~Q
E~
._,
C~
C~
C~
:^
,Q
o
· - ,
CO
.G
d
o
5-
Ct
a
3 ~
~o ~
~,
U) {~5
>.
C~
U)
5-
J3 ~
._,
-
Ct
s~
s~
. .
C)
o
U:
OCR for page 161
M ODEL UNCERTAINTY AND EVALUATION 7 6 7
of 4. These differences can be accounted for by the differences in the ages
and modes of operation of the vehicles being tested. However, there is
much less variation in the ratios of some of the pollutants, notably the
VOC to NOx and CO to CO2 ratios. The CRC report discusses some incon-
sistencies between these data and both the MOBILE and California's
EMFAC emissions models.
Although tunnel data provide only a snapshot of vehicle emissions, they
can be valuable and should continue to be used in testing the accuracy of
MOBILE and examining the effects of fuels, operating mode, and fleet
composition. However, vehicle operation in tunnels tends to significantly
deviate from average real-worId conditions. Tunnel traffic tends to have
higher speeds, less stop-and-go driving, and less loaded-mode operation
than average real-worId urban conditions.
CHEMICAL-MASS BALANCE
Chemical-mass balance (CMB) is another approach for evaluating emis-
sions model estimates with ambient observations. CMB uses a sophisti-
cated set of chemical fingerprints derived from speciation of source emis-
sions, which are apportioned mathematically to ambient air samples. An
oxidant assessment study in southeast Texas (Fujita et al. 1995) included
extensive comparison CMB estimates versus MOBILE-estimated VOC and
NOx emissions.
This study distinguished contributions from liquid gasoline and gasoline
vapor from vehicle exhaust in the ambient atmospheric measurements.
The findings of this study and another CMB study (Korc et al. 1995) coun-
ter arguments that it is evaporative emissions that account for MOBILE's
underprediction of VOCs. The major conclusions of these studies are
The sum of ambient liquid gasoline, gasoline vapors, industrial and
compressed natural gas contributions agrees reasonably well with and val-
idates the corresponding MOBILE emissions inventory estimates.
The discrepancies between CMB ambient- and emissions-derived
VOC to NOx ratios and ambient- and emissions-derived acetylene (a major
tracer of fingerprint of motor vehicle exhaust) at the Clinton site suggest
that the absolute amount of on-road mobile-source exhaust VOC emissions
were substantially underestimated by MOBILE.
The average ratio of CMB-derived ambient VOC emissions from mo-
bile sources compared with those estimated from MOBILE at the Clinton
site was 2.3.
The latest state-of-the-art CMB study was conducted in the Denver area
OCR for page 162
7 62 M ODELING M OBI[E-SOURCE EMISSIONS
in 1996 and 1997 under the Northern Front Range Air Quality Study
(NFRAQS). The study was directed toward evaluating particulate matter
(PM) emissions and sources; it indicated that PARTS greatly underesti-
mates the contribution of gasoline vehicles compared with diesel vehicles.
Tables 3-10 and 3-11 show this underestimation of emissions rates for
LDVs by PARTS. The study also found a very large contribution of start
emissions and high emitters to the total emissions from gasoline vehicles.
Table 4-5 provides a summary of pertinent NFRAQS results. Underesti-
mating emissions from starts and high emitters may be a significant cause
of MOBILE's underprediction of other emissions (CO and VOCs) found in
various studies.
Watson et al. (in press) contains a recent summary of CMB studies.
These studies tend to show that the relative contributions of mobile source
VOC emissions to the total inventory determined by CMB are two to three
times higher than those estimated using mobile source emissions factor
models such as MOBILE and EMFAC.
FUEL-BASED APPROACH TO EMISSIONS ANALYSIS
Another approach to evaluating mobile-source emissions estimates from
MOBILE is to compare MOBILE's results with those estimated through a
fuel-based approach. The remote-sensing and tunnel-study methods de-
scribed above measure exhaust emissions under operating conditions, but,
unlike a dynamometer test, they do not measure the emissions on a gram
per mile basis. The remote-sensing and tunnel studies measure the emis-
sions concentrations of VOCs, CO, and NOx relative to the concentration of
the combustion product carbon dioxide (COD. Carbon dioxide is the major
carbon-containing product of fuel combustion, and thus the CO2 emission
provides a measure of the amount of fuel burned. For improved accuracy,
a correction is applied for the carbon in the VOCs and CO emissions. The
determination of the concentrations of VOCs, CO, and NOx relative to CO2
provide measurements of these emissions relative to the amount of fuel
consumed. These fuel-based emissions factors, measured in grams per
gallon (g/gal), vary much less with changes in the vehicle mode of opera-
tion than do the travel-based (g/mi) emissions factors (Singer and Harley
1996; Singer et al. 1999~. This might give the fuel-based emissions method
an advantage over the travel-based method because the latter might not
accurately represent the variations in the driving cycles of urban areas.
Because the fuel-based emissions approach is less sensitive to the details
of the vehicles' operation (speed and acceleration), this method is less sus-
ceptible to inaccuracies derived from the MOBILE model's failure to repre-
sent realistic urban vehicle operation.
OCR for page 163
MODEL UNCERTAINTY AND EVALUATION 7 63
TABLE 4-5 Comparison of CMB and PARTS
Particulate Emissions from NFRAQS Study
Vehicle Category
Cold Starts
CMB PARTS
32.5%
3/4
Non-Smokers 7.5% 29.0%
High PM Emitter 31.3% 3/4
Diesel 28.8% 71.0%
-
Source: Fujita et al. 1998; Watson et al. 1998.
The fuel-based emissions inventory is a concept developed from mea-
surements of vehicle emissions on a gram per gallon of fuel. If the emis-
sions amounts in terms of g/gal for a representative on-road fleet are
known, then the fleet-wide emissions rate using the fuel consumption rate
in gallons per unit of time can be calculated. Fuel consumption is deter-
mined from fuel sales records. Knowledge of the fleet composition and the
fuel economy of the different types of vehicles is also required to estimate
the emissions inventory.
As described in previous sections, the MOBILE model employs a travel-
based method to develop emissions inventories. This requires knowledge
of the emissions levels (g/mi) for different modes of driving or for represen-
tative driving trips, vehicle activity or use, and the fleet composition. Both
the travel-based and fuel-based methods require information on the in-use
fleet composition, ambient temperature, and other factors that affect emis-
sions and vary with geographical region.
Singer and Harley (1996) developed a fuel-based CO inventory for the
South Coast Air Basin in California and compared the results to Califor-
nia's MVEl7F model. The fuel-based CO inventory estimate was of a fac-
tor 2.2 times larger for cars and 2.6 times larger for trucks than the travel-
based model. In a second study used more than 60,000 remote-sensing
measurements made at 38 sites in Los Angeles between May and October
1997, to develop CO and VOC inventories using the fuel-based method
(Singer and Harley 2000~. Their estimates for the on-road, fleet-stabilized
exhaust emissions from cars and light- and medium-duty trucks are larger
than the California MVEl7G model by factors of 2.4 ~ 0.2 for CO and 3.5
0.6 for VOCs. Similar tests of the MOBILE model emissions inventories
should be made to test the model's consistency and accuracy. The fuel-
based approach to emissions inventories is a promising method that can be
used to reduce the uncertainties in emissions predictions.
OCR for page 164
64 MODEL/NO MOB/[E-SOURCE EMISSIONS
SUMMARY OF FINDINGS AND RECOMMENDATIONS
In this chapter, quantitative estimates of MOBILE's overall accuracy
and uncertainty were not provided. The data are not available to do so
with any confidence. Some studies have identified significant discrepancies
in MOBILE predictions, but a complete assessment is not possible. Spe-
cific findings and recommendations follow.
Findings
1. EPA has done very little to use existing independent techniques to
test the overall ability of MOBILE to accurately estimate real-world emis-
sions of the overall on-road fleet. Others have applied many different tech-
niques that test the accuracy of MOBILE. Most studies have found that
MOBILES and earlier versions are substantially underpredicting VOC
emissions (approximately by a factor of 2) and, to some extent, underpre-
dicting CO and the PM contributions of gasoline-powered vehicles to the
overall emissions. This is in contrast to NOx emissions, which appear to be
more accurately estimated.
2. At present there is an inadequate understanding and quantification
of the sources of uncertainties in MOBILE. These uncertainties arise from
small and nonrepresentative emissions data, statistical analyses of these
data, and assumptions that underlie and define the MOBILE model's aigo-
rithms and predictions. Quantification of uncertainties is critical for un-
derstanding the weaknesses in the model, and identifying the most critical
needs for further emissions test data.
3. A critical unanswered question at the heart of issues related to MO-
BILE's uncertainty and evaluation is how accurate the model needs to be
to serve its various uses described in Chapter 2. It appears that EPA has
not determined a desirable level of accuracy for MOBILE. It is clear that
EPA, working with the user community, should determine the level of ac-
curacy needed, and plan accordingly. For example, if it was determined
that the current version of MOBILE is accurate enough to fulfill all of its
roles (which it is not), little further work in that direction is necessary. On
the other hand, if a significant improvement in model accuracy is de-
manded, then considerable work is suggested, and possibly a new design if
the current approach is too limited (as is concluded here). There will not
be a single answer, as various applications demand different levels of accu-
racy. Designing future emissions models should take required accuracy
for different applications into account.
4. EPA has provided only limited sensitivity analyses for earlier ver-
sions of MOBILE. Although the primary source of uncertainty in MOBILE
OCR for page 165
M ODEE UNCERTAINTY AND EVALUATION 7 65
output is in the underlying emissions data and data analyses used to gen-
erate the model formulations, uncertainty in model inputs can also create
uncertainty in model outputs. It is important that air-quality planners
understand the sensitivity of model outputs to uncertainty in model inputs
such as average speed, ambient temperature, fuel volatility, and I/M pro-
gram parameters.
5. A dominant cause of MOBILE's underprediction of real-world emis-
sions appears to be related to the driving-cycle testing protocol (because
the FTP-based driving cycle is not representative of current driving pat-
terns) and associated adjustment factors and default values forming the
basis for MOBILE. In the real world, slower speeds, heavier-Ioaded oper-
ating conditions associated with more congested stop-and-go driving condi-
tions, and a more dominant role of cold starts, aD of which produce en-
riched engine-operating conditions, appear to be the key factors in explain-
ing the discrepancy of the real-world driving cycle compared with the aver-
age driving cycle reflected by MOBILE. This observation indicates the es-
sential need to move toward a true modal modeling approach.
6. Other important factors that appear to be significant sources of error
in MOBILE are recruitment bias in vehicles that are FTP-tested and small
databases for sensitive parameters.
7. Although corrections have been made to many parts of the emerging
MOBILES model to improve its accuracy, it appears the end result might
deviate even more than past versions of the model with respect to the VOC
to NOx ratios. This will create, among other things, greater uncertainty in
ozone modeling.
Recommendations
1. EPA should assess the levels of accuracy needed to fulfill its regula-
tory responsibilities and required for specific applications of the MOBILE
model. EPA should compare the needed accuracy to the accuracy of the
MOBILE model, and identify specific elements of the model that contrib-
ute most to its inaccuracy. EPA should use the results of such an assess-
ment to help guide the development of the next generation of models that
would have improved accuracy in critical model components.
2. Enhanced model evaluation studies should begin immediately and
continue throughout the long-term evolution and development of mobile-
source emissions models. These studies need to be conducted to reduce
gaps between model-predicted emissions and the resulting air quality, and
also to reduce gaps between model-predicted emissions reductions from
control programs, such as vehicle I/M programs, and those that actually
occur in implementation. The evaluation should include (but not be lim-
OCR for page 166
7 66 M ODELING M OBI1E-SOURCE EMISSIONS
ited to) field observations, including tunnel studies; remote-sensing mea-
surements; source-receptor modeling; roadside pullovers; and air-quality
monitoring and modeling; vehicle emissions testing data from vehicle I/M
programs; and other vehicle emissions tests. Evaluation studies should be
done with oversight and guidance from an independent body, including
technical experts, and should be undertaken in tandem with the sensitiv-
ity and uncertainty studies suggested in the next recommendations.
3. Rigorous sensitivity analyses should be performed for all model in-
puts and provided as part of user guides for MOBILES and all future ver-
sions of MOBILE. From these sensitivity analyses, EPA should provide
guidance to transportation and air-quality planners on the most critical
model inputs affecting model results.
4. EPA, along with other agencies and industries, should undertake the
necessary measures to conduct quantitative uncertainty analyses of the
mobile-source emissions models in the modeling toolkit (discussed in
Chapter 6), especially the MOBILE model. Future versions of the
MOBILE model and other models in the toolkit should be developed to fa-
cilitate uncertainty analyses. Results of the uncertainty analyses should
be used to guide research plans for obtaining additional test data that
would increase the accuracy of the model.
Representative terms from entire chapter:
test data
