|
Items in cart [1] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 146
6
"valuating Inspection and Maintenance
Programs: Methods for Estimating
Emissions Reductions
Inspection and maintenance (~/M) program evaluation helps address whether
the investment of human and capital resources required for I/M programs is
beneficial. Although a judgment about whether I/M is beneficial is beyond the
scope of this report, the committee charge does include a call for identifying
evaluation criteria end methods. We will describe a set of criteria and discuss
methods of evaluation using these criteria in this chapter and the next.
Ideally, the primary criterion should tee the effect ofanI/M program on air
quality and the associated effects on human health and welfare. However, it
is very difficult to separate the relatively modest emissions impacts of I/M
programs from other policies designed to reduce emissions and from other
anthropogenic and natural changes that influence air quality. Because ofthis
difficulty, attempts to assess the impacts of I/M programs on ambient air emis-
sions have been difficult—even for carbon monoxide (CO), a pollutant that is
generated almost solely by light-duty vehicles (I DVs) (Scherrer and Kittelson
1994; ENVIRON 1998).
Thus, the criteria and methods of evaluation discussed in this chapter focus
on the reduction in emissions brought about by I/M programs. Difficulties in
estimating emissions reductions arise because vehicle emissions are variable
over time and with driving method, emission tests themselves are variable and
imperfect, and perhaps most important, the behavior of motorists, technicians,
146
OCR for page 147
Methods for Estimating Emissions Reductions 147
and even state enforcement authorities play a key role in the emissions reduc-
tions achieved. This chapter focuses on defining the emissions reductions to
be measured, describes some ofthe obstacles to measuring them, and outlines
possible methods for measuring them. Other discussions on emissions reduc-
tions are found in Chapter 3, which describes prior studies that have estimated
emissions-reduction benefits from I/M programs, and in Chapter 5, which
describes how emissions reductions are estimated in MOB TEE.
A full evaluation of I/M requires that the criteria be defined more broadly
thanjust the reduction in emissions. At a minimum, additional criteria include
cost and cost-effectiveness of program designs, enforcement requirements,
and such factors as public acceptance and political feasibility. These additional
criteria are discussed in Chapter 7.
METHODS FOR MEASURING EMISSIONS REDUCTIONS
There are several inherent difficulties in evaluating the emissions reduc-
tions from an I/M program. One is defining the baseline, the condition against
which the I/M program is compared. Attempting to discern the benefits by
comparing an area with an I/M program with a reference area (either an area
with a reference I/M program or a non-I/M fleet) is confounded by differences
between the area and its reference in climate, socioeconomic conditions, and
other characteristics. Additionally, vehicle technologies are also continuing to
improve, so the emissions benefits of a program depend on when they are
being measured. Sorting out vehicle repairs or scrappage that occurs because
of an I/M program versus what would occur even in its absence is also diffi-
cult. Finally, there are numerous statistical issues associated with evaluating
I/M, some of which are summarized in Appendix C.
An I/M program has the potential to reduce emissions in a number of
ways. Motorists might be persuaded to better maintain their vehicles as a
result ofthe program. Emissions might tee reduced as a result of repairs made
in anticipation of an I/M inspection (referred to as pre-inspection repairs) or
as a result of failing the inspection test. Finally, some vehicles may be scrapped
or sold outside the I/M area because, given the age or condition of the vehicle,
the owner did not think the repair was worth the cost. We summarize these
sources of emissions reduction resulting from an I/M program in Table 6-] .
It is important to contrast these sources of emissions reductions with the
OCR for page 148
148 Evaluating Vehicle Emissions I/M Programs
TABLE 6-l Sources of Emissions Reduction from I/M
Type of Emissions Reduction
1. Improved maintenance,which
leads to lower emissions
Data Requirements
2. Repairs to emissions equipment
made before an emissions test in
anticipation of the test
Roadside tests or remote sensing
3. (a) Repair of a vehicle's emissions In-program test data, comprehensive remote-
systems as a result of failing a test sensing data, or roadside-pullover data
(b) Length of time repairs last for a In-program test data, change-of-ownership
vehicle repaired as a result of fail- test data, comprehensive remote-sensing
ing an I/M test data, or roadside-pullover data
4. Early scrapping or transfer of high- In-program test data together with remote-
emitting vehicles outside the I/M sensing data or vehicle-registration data
region (fleet effects)
MOBILE modeling approach to I/M described in Chapter 5, which attributes
most emissions benefits to the instantaneous repair of failed vehicles.
We emphasize et the outact that the components of emissions reductions
arising from an I/M program, as described in Table 6-l, are very difficult to
estimate. Human behavior and lack of complete evidence confound the esti-
mation of emissions reductions at every turn. For example, emissions reduc-
tions as measured by in-program data on individual vehicles might not repre-
sent real emissions reductions on some vehicles but might result from partial
repair or even retesting with no repair. Repair in anticipation of the I/M pro-
gram might represent real and long-lasting emissions reduction from some
vehicles but would not be accounted for if only I/M test observations were
used for evaluation. Also important is the amount of emissions-reduction-
related repairs that would be done without any I/M program.
Because of these issues, the approaches for evaluation described in this
chapter include data needs and methods that attempt to account for all the
factors that influence emissions reductions technical, behavioral, and others.
All the categories of emissions reductions listed in Table 6-1 must be evaluated
to determine the effectiveness of a program. We first discuss sources of data
for measuring emissions changes from I/M.
OCR for page 149
Methods for Estimating Emissions Reductions ~ 49
Data to Estimate I/M Emissions Reductions
I/M programs have the potential to reduce tailpipe emissions of nitrogen
oxides (NOX) andCO end both tailpipe end evaporative (includingliquidleaks)
emissions of hydrocarbons (HC). Tailpipe emissions are easier to measure but
depend on many factors related to the condition and operation ofthe vehicle;
measurements can come from in-program (test results from centralized or
decentralized I/M programs) or on-road (remote sensing, roadside pullovers)
sources. Attributes of each are discussedbelow. Non-tailpipe HC emissions
are very difficult and expensive to measure; they require special equipment,
invasivetestmethods,andiongtesttimes. As aresult,although many tailpipe
emissions data are available, there are no evaporative emissions measurements
that directly measure I/M effectiveness to reduce all sources of non-tailpipe
emissions. As discussed in Chapter I, evaporative emissions represent a
significant but unquantified source of overall vehicle HC emissions.
Data from I/M Programs
Tailpipe data from I/M programs (in-program data) can come from the
program itself or from separate tests run for the purpose of evaluating the
program. Inspection lane data cover tailpipe, visual, and some functional tests,
such as a test of the gas cap. Data can be from idle tests, steady-state loaded-
mode tests (e.g., the acceleration simulation mode (ASM) test), or transient
loaded-mode tests (e.g., the IM240~. Chapter 3 contains descriptions ofthese
tests. Idle tests measure concentrations of CO and HC; steady-state loaded-
mode tests measure concentrations of CO, HC, andNOx; andtransientIoaded-
mode tests measure mass emissions of CO, HC, and NOX. Data can also be
gathered from visual and functional tests. Using in-program data is appealing
because this information can be collected at little or no extra cost.
Because ofthe very large amount of data collected as part of ongoing I/M
programs, detailed analysis can reveal information about the vehicle fleet and
the I/M program. All tested vehicles are identified, and they can be followed
Some examples of the separate in-program tests include running two consecutive
tests on vehicles or, for a program that uses a fast-pass system, running full IM240
tests for the purpose of gathering unambiguous in-program data.
OCR for page 150
~ 50 Evaluating Vehicle Emissions I/M Programs
from cycle to cycle, giving estimates of emissions deterioration rates and repair
effectiveness. These data also can be used to develop vehicle profiles (dis-
cussed in Chapter 4) and information about the performance of test stations.
Several factors might contribute to underestimating actual emissions reduc-
tions using in-program data. Several sources of emissions reduction listed in
Table 6- ~ cannot be measured with in-program data, such as the effect of an
I/M program on improving vehicle maintenance before the I/M test or on
emissions reduction gained by causing vehicles to leave the area or be
scrapped early. Measuring the impact of I/M programs on these parameters
is made difficult by the inherent turnover of the fleet and maintenance that
would occur in the absence of a program.
Conversely, there are reasons why in-program data might overestimate
actual emissions reductions. In-program data do not include emissions from
vehicles that avoid testing. Avoidance can result in exaggerated emissions
reductions in a number of ways. Owners might not bring their vehicles to be
tested at all. That is a problem if it is assumed that all vehicles are tested and
that failing vehicles receive repairs yielding some average emissions reduction.
Owners might collude with technicians running the test program to falsify the
emissions level of a failing vehicle so that it passes. Or owners might tempo-
rariTy fix a vehicle to pass the test without fully repairing it. If vehicles are
prepared for the test, they are not typical of vehicles on the road. in this case,
again, in-program data overstates emissions reductions if it is assumed that all
I/M repairs last some average length of time.
Finally, because ofthe statistical problem referred to as "regression to the
mean," emissions of a failing vehicle likely will be Tower on retesting, even in
the absence of repairs.2 Since only vehicles that fail are retested, this group
has higher then average emissions. Thus, even in the absence of repairs, their
emissions would tend to move closer to the mean ofthe fleet upon retesting.
Not accounting for this phenomenon would tend to overstate actual emissions
reductions from in-program data.
There are other more general problems with in-program data. The gold-
standard tests to measure tailpipe emissions are the Federal Test Procedure
Regression to the mean is a statistical phenomenon where the initial scores of a
selected group within a normal distribution will tend to move toward the population
mean in a follow-up test. Although the movement of an individual score cannot be
predicted based on this phenomenon, the group average will likely move toward the
population mean during follow-up tests.
OCR for page 151
Methods for Estimating Emissions Reductions 151
(FTP) and the supplemental FTP (SFTP) dynamometer tests. These tests are
used to certify that emissions from new vehicles do not exceed federal emis-
sions standards. However, these tests are far too costly end lime-consuming
to measure a large sample of vehicles. Most I/M tests suffer from a lack of
consistent preconditioning and an inability to represent all the driving modes
represented in the FTP and SFTP.4 Environmental conditions also have been
shown to affect emissions test results (Anderson and Wilkes 1998; EPA
2000b). Additionally, vehicle emissions vary from test to test, especially for
many high-emitting vehicles (Knepper et al. ~ 993; Bishop and Stedman ~ 996;
Coninx 2000~. Many factors can contribute to variability in repeated emissions
tests of the same vehicle. These factors, which are also summarized in
Wenze! et al. (2000), include the presence of intermittent failures of emissions-
contro] system components (such as a malfunctioning oxygen sensor) or fluctu-
ations in back-to-back emissions test results because of differences in the
measurement equipment, calibrations, or test personnel (e.g., different driving
styles in tracking a target speed-time trace on a dynamometer).
Some test results, such as those for idle and ASM tests, also must be
converted from concentration measurements to mass emission rates. The
correlation ofthe idle test with the FTP is illustratedin Figure 6-l for the same
car tested in the same laboratory (Haskew et al. ~ 987~. It includes 604 obser-
vations on mode] year ~ 98 ~ and ~ 982 vehicles and is plotted on a logarithmic
scale to reduce data scatter.
Data from Roadside Testing
Exhaust emissions test data can also be obtained from vehicles subject to
roadside tests. Under such a program, vehicles are randomly purled over and
given an emissions test similar to the vehicle inspection test. Visual and func-
tional tests can also be done to determine tampering rates (see the discussion
3A typical estimate for an FTP test is $800 to $1,000 per test for a vehicle delivered
to the laboratory. The additional charge for an SFTP is $750.
4The FTP and SFTP have extensive protocols regarding fuel specifications and
environmental conditions. Their driving cycles include driving modes not included in
transient loaded-mode I/M tests, including cold-start and high accelerations. They are
also done under laboratory conditions, with better-calibrated instruments and better-
trained technicians.
OCR for page 152
152 Evaluating Vehicle Emissions I/M Programs
604 Observations
60. .
~0
'IFTP }IC
AV6- 0.77
~7
,.;
$+ +
~ . A+ ~
A+
. . ~ 1
. . . ~ ~ Be
~0
100 it'
I~ ~ ~ - ~ ~
1000
FIGURE 6-1 EPA emissions-factor data from 1981-1982 industry closed-Loop cars. The
correlation of the idle concentration test with the FTP test is illustrated in the plot. The
horizontal scale is the idle test concentration measurement in parts per million (ppm);
the vertical scale gives the FTP mass test result for the same vehicle tested in the same
laboratory. Logarithmic scales are used to keep the data on the plot. Source: Haskew
et al. 1987. Reprinted by permission from SAE paper 871 103; copyright 1987, Society
of Automotive Engineers, Warrendale, PA.
ofthe results from the U.S. Environmental Protection Agency (EPA) national
tampering survey in Chapter 3), and to check components ofthe evaporative
emissions-control system. Roadside testing is fairly expensive because it re-
quires portable testing equipment, technicians to do the testing, and officers to
pull over vehicles. California's roadside test program measured emissions from
about ~ 0,000 vehicles per year from ~ 997 to the present. EPA' s national tam-
pering surveys, conducted from the late ~ 970s through ~ 992, measured about
7,500 vehicles per year between lL985 and 1992.
Because ofthe costs and time required to perform roadside testing and for
political reasons, emissions on far fewer vehicles are measured in this manner
than are measured in the inspection lanes. This implies that samples need to be
carefully defined to avoid selection bias and to obtain a representative sample
ofthe fleet. However, roadside testing represents an independent source of in-
OCR for page 153
Methods for Estimating Emissions Reductions 153
use evaluation data. Many of the shortcomings of in-program data, such as
motorists avoiding testing or making repairs solely to pass the test, are not
present in roadside pullover data. Roadside testing coupled with a vehicle's
I/M history can demonstrate the effectiveness of an I/M program or changes
to a program, the duration of repairs, the occurrence of pretest repairs, the
level of compliance with the program, and whether there is a need to increase
enforcement.
Data from Remote Sensing
Remote sensing is a third source of tailpipe emissions data (see Chapter
4 for a Ascription of remote sensing). Remote sensing allows a large number
of in-use vehicles to be measured. As with roadside testing, remote sensing
can help assess the effectiveness of an I/M program or changes to the pro-
gram, the duration of repairs, the occurrence of pretest repairs, the level of
compliance with the program, and whether there is a need to increase enforce-
ment. Remote-sensing measures emissions for about one-half second of
Unving for each vehicle. Care to select proper measurement sites is needed
for remote-sensing evaluations of I/M programs so that cold-start and off-
cycle emissions are avoided. Other issues discussed in Chapter 4 relate to
quality control and quality assurance, site selection, and coverage. Measure-
ments at a single site represent only one sample and one range of operating
conditions. The correlation of results from the vehicles measured by remote
sensing to vehicles and operating conditions (such as the load on the vehicle)
of the entire fleet or to results measured From a standard test procedure re-
mains problematic.S As with any measurement that estimates only concentra-
tion of pollutants, there is also the issue of converting measurements from
concentration units to mass emissions.
Remote sensing provides the only way to estimate pretest repairs effec-
sThe vehicle specific load (VSP) distribution (percentage of vehicles at different
loads) can be calculated for a fleet of vehicles measured by remote sensing. In an
IM240 test, a single vehicle is subjected to different loads for different amounts oftime.
The load distribution in the IM240 will be different in almost all cases than the load
distribution of a fleet of vehicles measured in a remote-sensing campaign. Jimenez et
al. (1999) have recommended that the load distribution in the remote-sensing fleet be
adjusted to that of the IM240 test by weighting the remote-sensing measurements by
load so that the weighted load distribution is the same as that in the IM240 test.
OCR for page 154
154 Evaluating Vehicle Em
issior~s I/M Programs
lively. Extensive remote sensing can identify trends in emissions by mode]
year before the I/M test to determine whether owners are taking actions to
reduce emissions before theirtests. As with roadside testing, remote sensing
offers a way to determine the extent of noncompliance with an I/M program,
including whether vehicles that fait I/M tests are still driven in the I/M area.
Remote sensing can also determine the percentage of vehicles exempt from
testing but Diving in the I/M area, together with an estimate oftheir contnbu-
tion to overall vehicle emissions.
Data on Evaporative Emissions
The evaporative emissions reductions possible from periodic inspection and
repair programs are difficult, if no/impossible, to estimate. One needs to know
the frequencies and impacts of evaporative system failures, the ability to detect
failure, and the ability of the service industry to make effective and durable
repairs. The only current data with which to estimate evaporative emissions
from I/M tests on pre-on-board diagnostic (OBD) vehicles are limited to visual
observations and gas-cap and iill-line pressure tests. There is no method for
directly measuring evaporative emissions short of subjecting vehicles to a
laboratory SHED (sealed housing for evaporative determination) test.6
Duration of Vehicle Repairs Issues
The length oftime that repairs remain effective is a central issue for evalu-
ating I/M program benefits. If emissions reductions from vehicle repairs last
on average 2 years, then emissions reductions from I/M are double what they
would tee if reductions lasted only ~ year. There has been some analysis ofthis
issue, but it has certainly not been resolved, and further work must be done.
The evidence to date is mixed. A number of studies find that at least some
vehicles have reductions that last a very short time. Lawson ~ ~ 993) observed
that many high-emitting vehicles stopped in random roadside pullovers had
either passed their emissions test in the previous 90 days or went on to pass
their emissions test in the following 90 days. This finding suggested that some
6The SHED test involves placing the vehicle in a sealed enclosure and monitoring
HC concentrations over time.
OCR for page 155
Methods for Estimating Emissions Reductions 155
repairs were of short duration done simply to pass the test. The IMRC study
(2000) found from change-of-ownership data that about 20% of vehicles that
failed and later passed would have failed again immediately after the test, and
McCTintock ~ ~ 999a) found in Colorado using remote-sensing data that a sign~fi-
cant amount ofthe emission reduction estimated from lane data was lost within
~ month of the final IM240 test. Wenze} (in press) found in Arizona that half
of on-road benefits disappear after ~ year and that 40°/O ofthe vehicles that fad]
in one I/M cycle, fait again in the next cycle. Rajan ~ ~ 996) found in a Califor-
nia study that on average vehicle emissions ofthe repaired vehicles returned
to their original level after 2 years. There is also evidence that some emission
reductions lastformuchiongerperiods. Again, the IMRC study (2000) found
that the 80°/O of vehicles that appeared to receive lasting repairs had emissions
reductions for NOX that extended at least a year, and for HC, deterioration in
those benefits began after 9 months. This study was not able to look beyond
a year after repair.
in summary, there appears to be a distribution of repair duration of the
vehicles that do have emissions reductions as a result of I/M. That distribution
ranges from a matter of days to several years or beyond. There needs to be
more study of the factors that influence how long repairs last, such as how
different I/M program configurations might affect repair durability.
Methods for Estimating EM Tailpipe Emissions Reflections
The data sources described above can be used to estimate vehicle tailpipe
emissions for model-year-specific emissions by a number of possible methods.
We summarize three methods and describe the advantages and disadvantages
of each. The reader is referred to Chapter 3, which discusses previous evalu-
ations using some of the methods described below.
Reference Method
This method compares the emissions by model year measured by the test
program with those of a reference program. The reference program may be
a null program (non-~/M) or a benchmark I/M program. The EPA has recom-
mended comparison with a benchmark I/M program. Before companug vehi-
cle fleets in the test and reference program, they are adjusted for model year
OCR for page 156
156 Evaluating Vehicle Emissions I/M Programs
and, if possible, odometer reading. Other factors influence fleet emissions
besides I/M. These parameters include vehicle types, vehicle models, socio-
economic characteristics of vehicle owners, altitude, climate, and fuel. Adjust-
ments between the two fleets need to be made for these factors in order to
estimate the effect ofthe I/M program. The effects ofthese other factors may
be seen by comparing fleets from different areas having the same I/M pro-
gram status. EPA released guidance on using the reference method for AM
evaluation where the Arizona I/M program is used as the benchmark (Sierra
Research ~ 997; EPA ~ 9986~. Critiques of this method are discussed in the
following section.
An advantage ofthe reference method is thatit canbe applied at any point
during the program' s lifetime as opposed to only when there is an incremental
change in the program (as with the step method of evaluation described be-
low). It can account for the effects of multiple I/M cycles, pre-inspection
maintenance and repairs, and deterioration of emissions repairs. If remote
sensing is used to collect the data, deterioration of emissions for vehicles not
inspected in the I/M program can also be estimated.
However, it is difficult to find two areas that are similar in all respects
except for their I/M programs. To the extent that the test area and the baseline
non-~/M area are different outside of I/M program status and those differences
have an impact on vehicle emissions, the difference must be fully accounted
for in the analysis. Unexplained differences would be incorrectly attributable
to the I/M program.
Critiques of the EPA-Approved Reference Method
The EPA guidance on the use of a benchmark program in Phoenix was
critiqued by Wenzel and Sawyer (1998), and by Rothman (1998~. The EPA
guidance document, the Sierra Research report describing this method, end the
critiques may be found at the EPA web site (ht~://www.epa.gov/oms/epg/
progeval.htm). Comments in the critiques were not incorporated into the guid-
ance document. Wenze] and Sawyer's concerns included possible sample bias
in recruitment options; errors in the conversion of concentrations to mass
emissions rates; and the fact that the method of model-year stratification, in
which vehicles are grouped by mode! year, does not accurately group vehicles
according to the technologies used in a vehicle's fuel delivery and computer
control systems. Rothman's concerns included recruitment bias end use ofthe
OCR for page 158
158 Evaluating Vehicle Emissions I/M Programs
nia Bureau of Automotive Repair, nosed that the California evaluation of emis-
sions reductions using roadside pullover data (described in Chapter 3 and in
CARB 2000b) showed different emissions reductions in Sacramento and Los
Angeles. Depending on which city was used as the benchmark, the other
city's I/M program could have been considered to be exemplary or insufficient.
Socioeconomic differences, vehicle mode] differences, and other factors might
be responsible for the differences in vehicle emissions reductions. Such fac-
tors are not accounted for or estimated in the current EPA guidance for evalu-
ating I/M using the EPA-approved method.
Step Method
When an I/M program is initiated or significantly mode led, at some point
about half the vehicles will fall under the new program and half will still be
tested under the old program. in the step method, emissions of a random
sample of vehicles tested under the new program are compared with those of
vehicles yet to be tested to determine the effect ofthe change.7 The method
was applied to roadside pullover emissions data by Lawson (1993) and the
California Air Resources Board (CARB 2000b) to evaluate different versions
of the Smog Check program. The CARB evaluation is described in Chapter
3 . Using emissions data collected with remote sensing, Stedman et al. ~~ 997)
also used the step method to estimate the incremental effect ofthe enhanced
Colorado I/M program.
A critical advantage of the step method is that the tested and untested
cohorts come from the same vehicle fleet.8 Thus, there is no need to correct
for differences in climate, fuels, or socioeconomic factors required in applica-
tion of the reference method. Further, the method possibly can be able to
detect an impact of the program on fleet composition. Motorists might re-
7It should be noted, however, that the application of the reference method to an
ongoing benchmark I/M program, the step method, or the comprehensive methods is
complicated by the possible residual effects of prior I/M cycles, which might affect the
emissions reductions occurring during subsequent cycles.
tin the case of a step change in a biennial program where, for example, all even
model-year vehicles have gone through the test and all odd model-year vehicles have
not (or vice versa), corrections must be made for subfleet differences in vehicle age,
mileage accumulation, or emissions-control technologies.
OCR for page 159
Methods for Estimating Emissions Reductions 159
.
register their vehicles outside the I/M area to avoid being subject to the pro-
gram, as was observed by Stedman et al. (1997, 1998) and McClintock
(l 999b). The step method would tee able to observe that there are more such
vehicles in the tested cohort than in the untested group. The step method can
be applied to estimate the impact of a change to an I/M program or to estimate
the deterioration in vehicle emissions from one test cycle to the next.
Comprehensive Method
In this method, vehicles are split into groups according to their test results:
initial pass, faiVpass, faiVwaiver, and faiVno-pass (see Table 6-2~. Average
vehicle emissions by test group are followed over time using remote sensing
(Wenze] 1999a; IMRC 2000) or using change of ownership I/M test data
(IMRC 2000~. Periodic test-cycle-to-test-cycle in-program data can also be
used to estimate initial emissions reductions and repair deterioration for
fail/pass cars. To get estimates of repair deterioration with better resolution
(months rasher then annual or biennia!) from in-program data, initial test data
from inspections made at intermediate times are necessary, such as emissions
tests that can occur with change of ownership. The level and change of emis-
sions over time give information about the emissions reduction seen as a result
of the test and pretest repairs together with the emissions deterioration be-
tween one test and the next. The comprehensive method can be used over a
number of cycles. As described in Chapter 3, this method was used by the
California Inspection and Maintenance Review Committee (IMRC 2000) to
help estimate the emissions reduction for the Smog Check II program.
A simplified version ofthis method involves calculating emissions reduc-
tions for vehicles that fait and are repaired. Initial inspection data for each
failing vehicle are compared with observations of emissions at final retest
(whether the vehicle passed or not). Ando et al. (2000) used this method to
evaluate the emissions reductions from the Arizona program. It has the advan-
tage of being relatively simple because it requires only data collected by the
I/M program. It also follows individual vehicles instead of examining changes
in average emissions by model years. However, it does not account for all
sources of emissions reductions listed in Table 6-1 . It is likely to overestimate
emissions reductions due to I/M because of the regression to the mean prob-
lem and because it does not provide any estimate of the amount of cheating
(fixing to pass the test). It will understate emissions reductions because it does
OCR for page 160
160 Evaluating Vehicle Emissions I/M Programs
TABLE 6-2 Vehicle Categones with Respect to I/M Status
Pass
Fail
· Passing emissions test, and low on-road emissions
· Passing emissions test, but high on-road emissions
· Passing within X months
· Obtaining a waiver after repairs over some minimum expenditure
· No final pass within X months
Legal: (1) Scrappeda
(2) Sold outside the I/M region and no longer driven in it
Illegal: (1) Dnven in the region, but without passing the I/M test
(2) Sold outside the region but still driven in it
(In some places this is against I/M rules.)
aThere is some natural scrappage rate, which is about 5% per year in California for
vehicles 10 years old.
not provide an assessment of how much legitimate repair is occurring in antici-
pation ofthe I/M test. Although the use in some states of fast-pass or fast-faiT
algorithms may make analyzing test data more difficult, Ando et al. (1999)
suggested one method for estimating full IM240 emissions from partial test
results.
Methods for Estimating Emissions Reductions
From Induced Fleet Change
In some cases, the methods descnbed above will provide estimates of
emissions changes resulting from both vehicle repair and changes in the make-
up of the fleet. The step method can evaluate both of these effects. Other
methods have provided only estimates of emissions reductions by model year.
To aggregate these estimates to emissions reductions from the fleet, estimates
of any changes in the fleet makeup resulting from the I/M program must be
included. Because ofthe I/M program, some motorists might decide it is either
too expensive or too onerous to get their vehicles through the inspection pro-
cess. The latter group of vehicles can be of several types. If a vehicle fails
and receives some repair, it might qualify for a waiver in the region. In other
cases, motorists might decide to scrap a vehicle earlier than they otherwise
OCR for page 161
Methods for Estimating Emissions Reductions 161
would, instead oftryingto get through the inspection process; or they might sell
the vehicle in another market that is not subject to the same I/M requirements.9
Table 6-2 summarizes all the possibilities for the fates of failing vehicles.
Induced Scrapping
Motorists might fail the test, obtain an estimate of repair costs, and elect
to scrap the vehicle instead of repairing it. Even if the potential repair bill
exceeds the waiver amount, motorists might decide to scrap rather than face
future repair cost uncertainty (most waivers are one time only). To evaluate
this effect, we must compare the underlying "natural rate" of scrappage of
vehicles with the scrappage rate with the I/M program.
If motorists do scrap their vehicles early because of I/M, the resulting
emissions reductions depend on what the vehicles' remaining lifetimes would
have been without I/M and the difference in emissions between the scrapped
vehicles and the replacement vehicles or alternative transportation modes. If
a scrapped vehicle is marginal rather than a high emitter, the emissions reduc-
tion induced is not large.
Although there is reasonably good statistical information about the ex-
pected remaining lifetimes of vehicles of different vintages, very little is Mown
about the remaining lifetimes of vehicles having trouble passing the I/M test.
These vehicles are more likely to be in worse overall condition and have Tower
economic value compared with vehicles of similar age that do not have trouble
passing the I/M test. It is clear that their expected remaining lifetimes would
be lower with an I/M program, but how much lower? Alberini et al. (1996)
found that vehicles scrapped under the voluntary Delaware Vehicle Retire-
ment Program were more polluting than the average older vehicle and had
about half the expected remaining lifetime of the average older vehicle, or
about ~ .7 years. Dill (2000) reported that vehicles scrapped under a CARB
buy-back program and under the Bay Area Air Quality Management Distnct' s
vehicle buy-beck program had a lifetime expectancy of about 3 years. These
vehicles also had higher emissions than other vehicles ofthe same model year.
Finally, replacement~ansportation for scrapped vehicles is complicatedby
the fact that the purchase of another vehicle, whether new or used, starts a
9Many regions have no I/M requirements because air-quality measurements do
not exceed state or federal standards.
OCR for page 162
~ 62 Evaluating Vehicle Emissions I/M Programs
series oftransactions through the market that are virtually impossible to iden-
tify. The usual assumption is that scrapped vehicles are replaced by vehicles
that represent the average of all vehicles in the fleet.
Despite these difficulties, simplifying assumptions canbe made to estimate
fleet effects from I/M. The induced scrappage rate canbe determined jointly
with the induced relocation rate (see below) in the following way. From I/M
program data, the number of vehicles or share ofthe inspected fleet that fails
end never passes can be calculated. Some fraction ofthese vehicles will still
be driven in the region, either with lapsed registrations or through some other
illegal means. An estimate ofthis number can be determined by remote sens-
ing or automatic license-plate readers together with I/M lane data (e.g., see
Colorado Air Quality Control Commission ~ 999 and TMRC 2000~. Because
these vehicles have not been repaired and might still be driven in the area, they
should not tee included in any estimate of emissions reductions. The remaining
vehicles can be considered either scrapped or relocated to another region; in
either case, emissions in the region fall. In a biennial I/M program, a reason-
able assumption is that the vehicles scrapped because of the program are
scrapped 2 years earlier than they otherwise would (see discussion of the
Alberini et al. ~ ~ 996) and Dill (2000) studies above) and that the emissions of
the replacement vehicles are the same as the fleet average. Further research
is needed to better understand how well such assumptions reflect reality.
Induced Relocation
Motorists or dealers might sell vehicles outside the region rather than pay
large repair bills. These vehicles can be treated just like the scrapped vehicles
described above forthe purposes ofthis analysis, if itcanbe demonstrated that
they do indeed remain outside the program area.
SUMMARY OF RECOMMENDATIONS FOR
EVALUATION OF EMISSIONS REDUCTIONS
On the basis of our review of methods for evaluation here in Chapter 6
end results of previous evaluations of emissions reductions in Chapter 3, the
committee has a number of findings and recommendations. First, we summa-
rize some key areas of uncertainty that future evaluations and studies need to
address.
OCR for page 163
Methods for Estimating Emissions Reductions ~ 63
· Little is known about the durability of repairs in I/M programs.
There is some evidence that a share of repaired vehicles go back to their
failing emission levels within a few months, but other vehicles retain low emis-
sions for longer periods (Rajan 1996; ENVIRON 1998; McClintock 1999a;
Wenzel ~999a; IMRC2000; RegionalAir Quality Council2000~. Understand-
ing how long repairs last is critical because the effectiveness of repairs has a
large impact on the total emissions reductions achieved by an I/M program and
the required frequency of retesting faiVpass vehicles.
· A related issue is that of vehicles with widely variable emissions
(Knepper et al. ~ 993; Bishop and Stedman ~ 996; Coninx 2000~. Are there
vehicles with intermittent problems that produce emissions levels that vary
between high and low for no apparent reason, so that they fait an initial test,
pass the next without repair, and then appear again as high emitters on the
road a short time later? How many vehicles pass a retest simplybecause their
emissions were Tow that time rather than because they had received effective
repairs? How many ofthese vehicles are there, and what are the implications
for traditional I/M testing?
· There is evidence that there are significant numbers of vehicles whose
emissions decrease in the weeks before their I/M test (IMRC 2000~. These
vehicles were earlier referred to as having had pretest repairs. The number
of such vehicles and the extent and duration of their repairs need to be in-
cluded in the evaluation of an I/M program. Additionally, it is necessary to
consider how many of these vehicles would have received repairs without an
I/M program.
· Many vehicles that fail in I/M tests never get repaired to a passing
level. These vehicles need to be tracked to determine whether they are
scrapped or still driven in the I/M region.
· The benefit of I/M programs in reducing non-tailpipe HC emissions is
unlmo~vn. The potential for benefits from reducing evaporative emissions and
liquid gasoline leaks should be evaluated.
In addition to shedding light on these questions, evaluations must quantify
all emissions reductions attributable to I/M programs. Based on the issues
discussed in Chapters 3 and 5 related to modeling, it is clear that the MOBILE
i°Here we are referring to vehicles that obtain repairs after having initially failed
the I/M test. We distinguished between these fail/pass cars and the fail/pass cars that
do not get repairs but rely on improved preconditioning or the variable nature of
vehicle emissions to pass an emissions test after initially failing.
OCR for page 164
~ 64 Evaluating Vehicle Emissions I/M Programs
and EMFAC models should not be used as evaluation tools. in the past, both
have greatly overestimated the benefits of vehicle emissions-control programs,
such as I/M programs. Evaluations of emissions benefits must be based on
I/M-test and on-road vehicle measurements associated with the program being
evaluated. Ideally, emissions benefits would be estimated with multiple sources
o f e m i s s i o n s ~ a t a a n ~ m u ~ t i p ~ e e v a ~ u a t i o n m e t h o ~ s . E v a ~ u a t i o n s w o u ~ ~ b e p e r -
formed periodically at the same location and compared with results from other
program evaluations. Such a comprehensive evaluation would quantify all
sources of emissions reductions listed in Table 6-~.
The committee also recognizes that comprehensive evaluations can require
a commitment of money and time not available to all state programs. The
committee recommends that EPA ensure that at least some comprehensive
evaluations are done that address the full array of emissions impacts incorpo-
ratingmultiple data sources and evaluation methods. The committee realizes
that there mustbe incentives for some states to do such comprehensive evalu-
ations on a long-term basis, possibly by spreading the costs across all states.
Some states, such as California and Colorado, already have incentives in their
state requirements for evaluating I/M programs. As described in Chapter 3,
these states have already performed multiple evaluations by a variety of meth-
ods.
Any state undertaking such a comprehensive evaluation also needs to have
a well-established procedure for collecting and analyzing vehicle emissions
data from in-program and on-road sources. The guidance for data collection
and evaluation should be peer reviewed, and comments gathered during the
review should be addressed.
The committee believes that selecting three to five states with different
program types and from different regions in the country would provide a suffi-
cientrange offull evaluations. Besides providing for an estimate ofthe emis-
sions reductions, comprehensive evaluations could tee usedby all programs to
improve forecasts of I/M benefits from the MOBILE and EMFAC models and
to assess the potential emissions impacts from changes in program design. Full
evaluations could also be used to help less comprehensive program evaluations
estimate allpotential sources of emissions impacts due to I/M. Some ofthe
evaluations should tee conducted independently ofthe agencies that menage the
I/M program or have a role in setting I/M policy and all evaluations should be
peerreviewedbyindependentscientists,economists,andstatisticians. Results
of evaluations should tee made public so that all states canbeneiit from whet
is learned. EPA also should pursue publishing some aspects of these evalua-
OCR for page 165
Methods for Estimating Emissions Reductions ~ 65
lions inprofessiona~jou~nals so that they may tee further reviewed and dissemi-
nated.
States Conducting Comprehensive Evaluations
Comprehensive evaluations must account for all the emissions reductions
in Table 6-~: emissions impacts from pretest repairs; the initial emissions re-
ductions from repairs and the length oftime repairs last; the amount of fraud
in the program (including vehicles that fraudulently register outside an I/M area
but still operate within the area); and the number of vehicles that are scrapped
or sold because ofthe I/M program. One difiFicultyis the need to account for
repairs that would have been done without an I/M program. There is no per-
fect method for evaluating all emissions impacts from I/M the reference,
step, and comprehensive methods all have their own inherent limitations. The
ideal evaluation is a reference method comparing an I/M area with a non-~/M
area, which would take into account normal repair (from the non-~/M area),
pre-inspection repair, test fraud, and I/M benefit from repair (from the I/M
area). However, there are a number of challenges to this approach, as dis-
cussed earlier in this chapter. The most appropriate evaluation method will
vary with the type of I/M program in place and the availability of comparison
sites; the best method must include a non-~/M fleet for baseline comparisons.
The most appropriate method might also depend on the timing of changes in
the I/M program. However, evaluation methods and data sources are comple-
mentary. Each adds information to reduce the uncertainty ofthe estimate and
to better understand the effectiveness of the I/M program.
For a comprehensive evaluation, in-program data canbe used to determine
how many vehicles take the test. Program populations can be compared with
actual vehicle registrations to evaluate how many vehicles are actually partici-
pating in the program. These data can also show the number of vehicles that
pass the test after initially failing in each I/M cycle. However, a key issue for
determining emissions reductions is to know how many ofthese vehicles were
repaired and how long repairs last. Because repairs could be temporary or
simply made for "passing the test," remote sensing and random roadside pull-
overs can be used to assess how well the repaired vehicles are staying re-
paired. Roadside testing and in-program testing are ways to discover tamper-
ing with emissions-control components. Remote-sensing or roadside-pullover
data can be used to estimate the percentage of emissions reduction due to pre-
OCR for page 166
~ 66 Evaluating Vehicle Emissions I/M Programs
inspection repairs. However, these must be distinguished from repairs that
would have occurred without an I/M program. Program data can also identify
how many vehicles fait the test and never pass. License-plate data collected
as part of a remote-sensing database are necessary to determine how many
ofthose vehicles are still driven in the region and to indicate whether changes
to the makeup of the vehicle fleet are due to an I/M program.
States Conducting Shortened Evaluations
For the states that cannot do a comprehensive evaluation, a shortened
evaluation should tee developed. The short evaluation should do the following:
Not be based on the MOBILE model.
· Include all components of emissions reduction (from Table 6-~.
· Use the best evidence from data collected in full evaluations for the
value of urdmown emissions-reduction components.
· Use some on-road data from the local area if possible.
in a shortened evaluation, in-program data, registration data, and any local
on-road data that can be collected will be used along with evidence from more
thorough evaluations to estimate the emissions-reduction components in Table
6- ~ . To the extent data are not available to estimate some aspect of emissions
reduction from a program' s own data sources, data and assumptions based on
the best comprehensive evaluations from other sources must be used. Infor-
mation and assumptions should be updated over time as more evidence be-
comes available. The use of assumptions based on evaluations of otherpro-
grams for certain key variables is a reasonable approach if there are not great
differences among programs in these variables. For example, if the amount
of pretest repair is reasonably consistent across states that have measured this
parameter, then use of the average amount of such repair is a good proxy in
areas that do not measure it directly. If large variations are found in aspects
of I/M performance among states conducting thorough evaluations, then this
shortened evaluation method would have to be reevaluated.
EPA will need to develop guidance for a shortened evaluation method. A
review committee should be established to advise EPA in the selection of
shortened evaluation methods and in selection of what information can be
drawn from comprehensive evaluations to inform the shortened evaluation.
OCR for page 167
Methods for Estimating Emissions Reductions ~ 67
Assumptions used in the shortened evaluation can then tee continually improved
over time as more evidence becomes available.
The shortened evaluation method must be linked with results from compre-
hensive evaluations. The shortened evaluation method should not go forward
unless comprehensive evaluations are also being performed. Any shortened
evaluation method needs to be validated. States conducting comprehensive
evaluations oftheirI/M programs also should do a shortened evaluation. EPA
will review the differences between the results ofthe comprehensive evalua-
tion and the shortened evaluation and will modify the shortened evaluation
method so that its results are more similar to the comprehensive evaluation"
Validation of the shortened evaluation should be done at least once every 3
years.
One way of structuring the shortened method would involve the following
steps:
· In-program data are collected over a test cycle and an estimate is
made of emissions reductions for all failing vehicles: initial test results minus
final test results (even for vehicles that still do notpass). In some cases, these
results need to be adjusted to a standard emissions test (e.g., IM240 and FTP)
to estimate the emissions benefit using correlation equations.' ~
· Adjustments are made to account for regression to the mean in test
data. The magnitude of the adjustment can be made based on other more
comprehensive evaluations.
· The adjusted in-program data are aggregated across all failing vehicles
and an assumption is made about the expected average length of repair dura-
tion, not including fraud. (This assumption about repair duration would be
taken from on-road evidence from comprehensive evaluations.)
· A further adjustment to the results would be made based on an esti-
mate of fraudulent emissions reductions or emissions reductions that are made
only for the period of the test. This estimate could be based on the audits of
the testing stations run by the state, remote-sensing or roadside-survey data if
any are available for the particular region, or some average estimate of fraud
in other programs.
Sphere are inherent difficulties with correlating other test results with an IM240
test that evaluates emissions over only 30-240 seconds of a 2-year period and only
when the vehicle is on its "best behavior" to 'pass the test." However, such correla-
tions must be done to estimate mass emissions reductions.
OCR for page 168
68 Evaluating Vehicle Emissions I/M Programs
· Further adjustment should be made to reflect pretest repairs and re-
pairs that would take place without an I/M program. This adjustment could be
based on estimates from other more comprehensive evaluations.
~ Retirement or relocation of vehicles as a result of I/M can be esti-
mated based on the evidence from more comprehensive studies and from local
vehicle-registration data and remote-sensing data.
Performance Indicators
In addition, both types of program evaluations should compile performance
indicators. Although they do not directly estimate emissions reductions, perfor-
mance indicators are relatively easy to measure, supplement the evaluations
described above, end provide relatively concise indicators of a program 's suc-
cess. EPA, and the states themselves, should use these performance indicators
to rate states' I/M programs and to help direct improvements nationally.
These performance indicators could include the following:
· An estimate ofthe total number of vehicles driven in the I/M region,
the share of those vehicles that are eligible for inspection, and the share of
those that actually are inspected.
· Failure rates by mode} year at the program cutpoints.
· Estimates ofthe average emissions of passing vehicles and average
emissions of failing vehicles.
Share of failing vehicles that actually get repaired to below program
cutpoints and their average emission rates before and after repair.
Share of failing vehicles that do not ever pass the I/M test, their aver-
age emissions, and an estimate ofthe number ofthose still driven in the area.
The rate of repeat failures from one I/M cycle to the next.
Estimates of the actual number of high emitters on the road.
These indicators could also be checked against assumptions used in model-
ing to make the models more realistic and improve the forecast of emissions
reductions for the state implementation plan.
Over time, alternative programs to reduce in-use vehicle emissions might
be developed by states. Systems that rely exclusively on remote sensing or
OBD may be used as technology improves in the future. These programs
would also need to be evaluated. Some ofthe issues associated with evaluat-
ing these emerging testing technologies are discussed in the following chapter.
Representative terms from entire chapter:
vehicle emissions
