|
Items in cart [1] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 234
Appendix C
Some Statistical Issues in
Inspection and Maintenance Evaluations
VEHICLE EMISSIONS DISTRIBUTIONS
A few broken vehicles contribute a disproportionate share of the total emis-
sions made up of exhaust hydrocarbon (HC), carbon monoxide (CO), and
nitrogen oxide (NOX) or evaporative HC emissions (including diurnal, running
loss, liquid leaks, or hot-soak emissions). A vehicle can deteriorate in many
ways, and different types of deterioration affect emissions differently. As
discussedin Chapter ~ ofthis report, thereis considerable correlation between
exhaust HC and CO emissions in high-emitting vehicles, but little relationship
between these and high NOX emissions. High evaporative emissions may or
may not be correlated with high exhaust emissions (Pierson et al. 19999. High
diurnal evaporative emitting vehicles are usually different from those with high
hot-soak emissions.
Since most vehicles do not have high emissions, the distribution of emis-
sions among vehicles for any single emission type are highly skewed and are
best characterized by a log or gamma distribution (Zhang et al. ~ 994; Wenze]
et al. 2000~. Emissions distributions are more skewed for newer model-year
vehicles than for their older counterparts, because only a few newer vehicles
have broken fuel delivery or emissions-control equipment. HC and CO emis-
sions distributions are more skewed than those of NOX.
234
OCR for page 235
Appendix C 235
SAMPLING METHODS AND BIAS
Special care mustbe taken to avoid selectionbias. Since a small percent-
age of vehicles will emit a large percentage of the emissions, any sampling
technique that decreases or enhances the percentage ofthese higher emitters
may cause questionable conclusions to be drawn from the data. Every report
describing an analysis of an inspection and maintenance (~/M) program that
uses a sample of the vehicle fleet to estimate the benefits of the program
should estimate the degree of selection bias in the sample. The methods for
selecting the vehicles in the sample and the tests to determine if the sample is
representative ofthe fleet should tee described. An estimate ofthe magnitude
of selection bias should also be made for any vehicle sample used to derive a
correlation between an I/M test and the Federal Test Procedure (FTP) for the
purpose of estimating the tons of pollutant per day reduced by the I/M pro-
gram.
Mail solicitation of vehicles for laboratory testing has been practiced by the
California Air Resources Board (CARB), the U. S. Environmental Protection
Agency (EPA), and by auto manufacturers. Responses to mail solicitations
have shown low voluntary acceptance by vehicle owners; CARB and EPA
have typically experienced acceptance rates on the order of 10°/0. In the
California I/M pilot study, when CARB threatened vehicle owners solicited by
mad! with loss of registration if they did not come in for testing, only a 60%
acceptance rate was obtained. The selection bias in mall solicitation sampling
will be influenced by the rewards, penalties, and risks perceived by the recipi-
ent ofthe solicitation. If vehicle owners think that they might be penalized by
the result ofthe tests they are asked to volunteer for, they will be less likely to
agree to do so. If free inspections and repairs are offered for the vehicles to
be tested, a disproportionate number of dirty vehicles might be included in th
sample.
- ~ ~ r--r
'The CARB acceptance rate of 60% is not 60% response rate of letters that were
mailed out. Fifteen hundred solicitation letters were mailed out. Of the 1,500, 444
vehicles were dismissed, usually because (1) that the vehicle had previously received
a Smog Check test, (2) that the addressee no longer owned the vehicle, or (3) the
solicitation letter was resumed as undeliverable. The " 60% response rate" is 60% of
the (1,500-444) vehicles. Of the 1,500 letters mailed out, only 43% resulted in a re-
cruited vehicle.
OCR for page 236
23 6 Evaluating Vehicle Emissions I/M Programs
Higher acceptance rates have been obtained when vehicles were solicited
directly rather than by mail. Recruitment of vehicles to be tested for multiday
evaporative emissions testing solicited directly at an I/M lane resulted in a 90°/0
acceptance in the Coordinating Research Council's (CRC) study of evapora-
tive emissions. High acceptance rates were also obtained in the 1997-1999
California roadside tests, where vehicles were pulled over by police of dicers,
asked if they would agree to be tested, and then tested at roadside. In a sam-
ple of the California roadside-test program, where vehicles were also mea-
sured by remote sensing as they left the roadside test area, a 92% acceptance
rate was obtained. The remote-sensing results in this case showed no differ-
ences in the average HC and CO emissions for a mode! year between the
vehicles' drivers who agreed to be tested and those who refused. An earlier
roadside testing program, however, showed evidence of considerable bias
because of higher emissions from vehicles whose owners refused to have
them tested.2
Samples of vehicles taken from I/M lanes will not include vehicles that
avoid being tested. These include both unregistered vehicles and vehicles
registered in an area where I/M is not required but garaged and driven in the
I/M area. If recruitment at I/M lanes is limited to vehicles arriving for their
initial tests, not having had pre-test repairs, selection bias may exist, since
vehicles that received pre-test repairs probably had higher emissions . Vehicles
that are unsafe to test on a dynamometer would not be tested in an I/M pro-
gram using loaded-mode testing.3
FACTORS INFLUENCING VEHICLE EMISSIONS
Vehicle emissions are influenced by numerous factors other than I/M.
When evaluating the effect of an I/M program, other factors that may be
2In an earlier Roadside Testing program in California, remote sensing showed that
vehicles whose owners refused to allow testing had emissions more than twice those
whose owners agreed to the roadside tests (Stedman et al. 1994~.
3There is a group of vehicles (a relatively small fraction of the fleet) that cannot be
tested on a two-wheel drive (2WD) dynamometer due to all-wheel drive (AWD) or non-
switchable traction control. However, many centralized I/M programs are either using
or considering the use of AWD dynamometers and therefore will be able to test these
vehicles. The number of vehicles that are unsafe to test on a dynamometer is small
enough so that it can be considered insignificant. Many of these are older (pre-1981)
vehicles that are not subject to loade-mode testing.
OCR for page 237
Appendix C 237
partly responsible for emissions changes (either reducing or increasing them)
need to be considered. To obtain the influence ofthe I/M program itself, these
other factors should be shown either to have a low impact on the results, to be
controlled, or to be randomized through sample selection.
Vehicle age and mode} year are important factors in vehicle emissions, but
the most important factor is vehicle maintenance. Most analyses group vehicle
data by mode! year, which is closely associated with vehicle technology. For
example, the shift from carbureted to fuel-injected vehicles occurred over a
period of a few years.
Since emissions deterioration rates have been decreasing due to improve-
ments in vehicle design, the amount of emissions reduction attributable to an
I/M program will be a function of the year during which the evaluation was
made. When comparing evaluations of I/M programs in different years,
change in vehicle technology must tee taken into account. This includes vehicle
design (engine design, fuel delivery system, and emission controls) and vehicle
type (i.e., passenger car, light-duty buck), especially if different vehicle mode!
years and different vehicle types within a model year were built to different
regulatory emission standards.
The amount of vehicle use (miles driven per year) can also influence
vehicle emissions. High-use vehicles (e.g., taxis) can be expected to have
faster rates of deterioration than similar vehicles driven fewer miles.
Another important factor influencing vehicle emissions is driving mode
(i.e., cold start, warm start, low load, high load, high acceleration or decelera-
tion). In a cold start, the emissions control systems will not tee operating at full
capability. Under high load, many vehicles are designed to run fuel rich, caus-
ing very high CO emissions. During high acceleration or deceleration, some
vehicles have high HC emissions.
Fuel quality also effects emissions. Laboratory tests using fuel different
from that blended for local conditions can introduce bias into the data. Fuel
composition parameters influencing emissions include volatility (especially for
evaporative emissions), sulfurievel, the presence of oxygenates (especially for
older, carbureted vehicles), and other fuel reformulation, such as federal and
California reformulated gasolines. Where there is a seasonal change in fuel
composition (e. g., higher oxygenates and volatility in winter), comparisons of
year-to-year changes in vehicle emissions should sample vehicles taken the
same time of the year. In the wintertime, areas using oxygenated fuels may
have a lower I/M test failure rate, because CO emissions are reduced as much
as ~ 0°/0. In this situation, the use of oxygenated fuels allows vehicles that
normally would have failed to "pass the test" without being repaired.
OCR for page 238
238 Evaluating Vehicle Emissions I/M Programs
Additionally, ambient conditions (temperature, altitude, end humidify) can
have an impact on vehicle emissions as evidenced by the strong seasonal
variation in the emissions from I/M and remote-sensing data.
Finally, the motorist socioeconomics can have an effect on vehicle emis-
sions. Less affluent areas will have, on average, older vehicles. But age
corrected vehicle emissions from a less effluent area still showedhigheremis-
sions (Stedman et al. ~ 994~. Correlations have been found between average
vehicle emissions and the zip code where the vehicle was registered (Singer
and Harley 2000~. Vehicles ofthe same mode] year and vehicle type regis-
tered in more affluent areas have lower emissions (Wenzel 1997~.
HUMAN BEHAVIORAL FACTORS
The introduction of a more severe I/M program can lead to the re-registra-
tion of vehicles into areas not requiring the new I/M program (Stedman et al.
~ 997; McClintock ~ 9984. Some ofthese re-reg~stered vehicles still drive in the
I/M program area but are not subject to inspection.
The degree of enforcement affects the human behavior of avoiding the
test. An evaluation of an I/M program should help in determining where addi-
tional enforcement and/or additional economic incentives would improve pro-
gram benefits.
Fraud, such as cleanpiping, distorts program assessmentbased solely on
I/M program records. Clean piping is a type oftest fraud in which a technician
tests a clean vehicle and attributes the result to the vehicle that is supposed to
be tested to ensure that the vehicle passes. The inspection-lane data would
indicate that the program is more effective than it actually is. The use of
roadside pullover testing and/or remote-sensing measurements is not sensitive
to test fraud and could help to identify testing stations that should be subject to
covert audits to detect fraudulent behavior.
NUMBER OF VEHICLES
I/M programs are designed to minimize the percentage of high-emitting
vehicles. To tell whether such vehicles are becoming less prevalent in the
fleet, sufficient numbers of vehicles need to be in the sample. Comparing two
OCR for page 239
Appendix C 239
populations requires enough vehicles in each to have a valid comparison. If
further subdivision ofthe population is necessary, additional vehicle emissions
data are required.
A small sample size may produce too much uncertainty to adequately
describe the average emissions reductions due to the I/M program. The size
of the sample to use depends on the amount of confidence one wants in the
results.4
The number of vehicles needed to characterize fleet emissions is also
influenced by vehicle test-to-test variability. Some vehicles, especially some
high-emithug ones, show considerable variation in emissions in repeated tests
under the same conditions. The shape ofthe vehicle emissions distribution has
consequences for how to choose vehicle samples for analysis.
Stratified sampling is a method of reducing the total number of vehicles
sampled, while obtaining sufficient numbers of hi~-emitting vehicles. For this
purpose more vehicles are selected from segments ofthe population that are
expected to have more high-emitting vehicles. To keep the frequency ofthese
vehicles in perspective, aparalle] record ofthe frequency ofthe fleet segment
in the total fleet has to be obtained.
A variety of stratified sampling methodologies have been used. EPA and
CARE have used sampling strategies based on vehicle technology groupings
and vehicle model year. California has used Radian/Eastern Research Group's
high-emitter index to select vehicles for sending to test-only inspection stations.
The CRC E-35 evaporative emissions study selected equal numbers of vehi-
cles in three age groups corresponding to their evaporative emissions-control
technologies.
4When Stedman et al. (1997) applied the step method in Denver, five daily aver-
ages were used to get an estimate of the effect of the enhanced I/M Program being
introduced. With about 4,000 remote-sensing measurements per day (2,000 each for
"enhanced I/M tested" and "not enhanced I/M tested"), the study found about a 7%
2-3% emissions reduction benefit, with 95% confidence. Because Stedman et al. ran
the experiment halfway through the new biennial program, almost all other factors were
randomized. The appropriate sample sizes in any specific test will depend on whether
the significance of other factors (such as vehicle type, fuel, and socioeconomics)
needs to be understood. Guidance should be sought from a statistician familiar with
handling non-normal distributions.
OCR for page 240
240 Evaluating Vehicle Emissions I/M Programs
AVERAGE VALUES AND LOG TRANSFORMS
Some researchers take the log of (sometimes binned) vehicle emissions
data to obtain a near normal distribution and then take the mean and the error
limits ofthe log-transformed distribution. This reduces the weight (importance)
of the high-emitting vehicles in the relationships (Pollack et al. 1999~. The
mean of a log-transformed sample is the geometric mean ofthe sample rather
than the arithmetic mean. The total vehicle emissions introduced into the
atmosphere are the sum of all the individual vehicle emissions or the arithmetic
average of emissions per vehicle times the number of vehicles. The mean of
a set of log-normally distributed vehicle emissions will always be less than the
arithmetic average.
TESTING NULL HYPOTHESES
If the sample is sufficiently large, it canbe randomly divided into two sets,
and the difference between the (model-year weighted) averages of the two
sets of data should be zero plus or minus some value within an uncertainty
range. Assumptions about the lack of influence of certain factors should be
checked with a null hypothesis.
CONFIDENCE LIMITS
Confidence limits of vehicle emissions in log or gamma distributions are
asymmetric and can be generated using bootstrap analysis. A bootstrap ap-
proach is a Monte Carlo-style simulation technique used to estimate the confi-
dence interval when errors are non-normally distributed (Chatterjee et al.
1997).
Normal statistics can be applied to the arithmetic averages of sufficiently
large subsamples of non-normal distributions. The confidence limits, in this
case symmetrical, apply to the means of the samples.
CORRELATIONS
Pearson product correlation is not appropriate for log or gamma distribu-
tions because the R2 values are overly influenced by the small, high emitting
OCR for page 241
Appendix C 241
fraction. Scatter plots should be presented in linear space so the reader can
visually assess the degree of correlation. Spearman rank correlation can be
performed, however, to minimize the effect that high emitters have on vehicle
emission distributions.
ISSUES IN EMISSIONS OF I/M TEST DATA
TO ESTIMATE I/M BENEFITS
The difference between initial fad] and final pass results on the same vehi-
cle tested in one cycle of the I/M program overstates the benefit of the I/M
program because of regression to the mean.s Also, the deterioration of emis-
sions until the next required test (the next test cycle) is not taken into account.
COMPARISON OF TEST RESULTS AMONG PROGRAMS
I/M evaluation methods depend on comparing emissions from one vehicle
fleet with another, or comparing emissions from one vehicle fleet at different
times. Fleet emissions are dependent on the load that the vehicles are under
during the test. In order to compare a test fleet with a reference fleet mea-
sured using different tests, a correlation equation is necessary. This equation
is created from a third fleet (a "correlation fleet") that has experienced both
tests. The test and reference fleets need to be free from selection bias and
representative ofthe same population. The correlation fleet needs to tee free
from selection bias end representative ofthe same vehicle population, unless
it can be shown that the correlation equation is not sensitive to potential differ-
ences between the correlation fleet and the test and reference fleets. The
correlation equations should be derived from fleets subject to the same kind of
test procedure. Significant differences between fleets may be caused by
differences in any and all ofthe following: vehicle emission control and fuel
system technologies, vehicle ages, vehicle types, inspection maintenance histo-
ries, socioeconomic owner histories. In addition, similar filet and environmental
conditions may be required (altitude, temperature, etc.) for measurement con-
sA description of the statistical concept of regression to the mean by W.M.
Trochim, a professor in the Department of Policy Analysis and Management at Comell
University, is available on the web at http://trochim.human.comell.edu/kb/regrmean.
him.
OCR for page 242
242 Evaluating Vehicle Emissions I/M Programs
ditions for the test, reference, and environmental fleet emission measurements.
However, fuel and environmental conditions are not taken into account in the
pass/fai! cutpoints when the scheduled I/M test is performed.
1
Representative terms from entire chapter:
selection bias
