|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 313
12 Determining Outcome
We can no longer afford to provide health care without Mowing more about
its successes aM failures. The Era of Assessment and Accountability is
dawning at last; it Is the . . . Iatest~ut probate) not the last~hase of our
efforts to achieve an equitable health care system, of satisfactory quality, at a
price we can afford
Arnold Relman
The Rationale for Outcome Determination
This report focuses primarily on ways to improve the outcome of treatment for
alcohol problems. In previous chapters it was suggested that the clarification of basic
concepts, comprehensive pretreatment assessment, more precise characterizations of the
treatments provided, and careful matching of individuals to treatment by means of explicitly
stated and modifiable guidelines would enhance the proportion of positive outcomes. In
this chapter the committee examines issues involved in understanding the target of all of
this activity, the outcome of treatment.
For purposes of orientation, a distinction may be drawn between the short-term
goals of treatment and its outcome. The goals of treatment include detoxification (where
required); the reduction or elimination of alcohol use; the concomitant reduction of the
signs and symptoms and of the consequences of alcohol use; the resolution of intercurrent
medical, psychiatric, and social problems; and a modification in attitude toward drinking
behavior leading to a commitment to its amelioration in future. The attainment of these
goals is a major therapeutic achievement, a fact that should not be obscured by what
follows. Goals will not be extensively discussed in this chapter, although it should be
emphasized that, as with outcomes, there is a great need for their more frequent and
detailed documentation.
The outcome of treatment has to do with the maintenance of these goals over
the longer term; that is, with whether the commitment to the amelioration of drinking
behavior has been realized. Although achievement of the short-term goals of treatment
is laudable, it does not ensure a favorable outcome. And as the epigraph suggests,
outcome is the bottom line. A health economist put it with characteristic directness in
discussing the issue of quality of care: Quality means: did the patients get better?"
(McClure, 1985a:43~. Organizations that accredit treatment facilities have increasingly
stressed the evaluation of treatment outcome as a prerequisite for accreditation (Schroeder,
1987~. One such organization recently advised as follows:
The Commission now has an entire subsection of standards specifically
focused on program evaluation and the importance of program results.
This reflects a blending of technical requirements and the Commission's
long-term emphasis on utilization of outcome measurements . . .
organizations should keep in mind that an important focus of accreditation
site surveys will be on the extent that evaluation reports are actually
assisting them to accomplish their goals. (Commission on Accreditation
of Rehabilitation Facilities, 1988:10-11)
3~13
OCR for page 314
314 BROADENING IlIE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
Current interest in treatment outcome is fueled in large measure by concern over
the growth of health care expenditures. There is an understandable desire to be certain
that a return on such expenditures is being realized.
concerned about treatment outcomes-and for other reasons.
Yet clinicians have long been
The provision of treatment in the absence of knowledge of results has been likened
to playing golf in the fog (Ziskin, 1970~. One can stand at the tee, drive balls into the
distance with impeccable form, fantasize about what good drives they were, and
congratulate oneself on being a surpassingly good golfer. Yet what does this avail if one
does not know where the balls are landing? Golfing under these circumstances becomes
an exercise in unreality; no reasonable feedback can be provided that will be useful in
improving one's game.
In addition to its potentially improving effect on clinical practice, there are ethical
and legal reasons for promoting a systematic knowledge of treatment outcome. Because
an effective treatment may nevertheless prove harmful to some individuals (see Chapter 6),
an ethical obligation exists to monitor outcome so that deleterious effects can be detected
and countered. Such an obligation is consistent with the principle of primum non
nocere the first duty of the treater is to do no harm. Should the detection of adverse
effects of treatment be overlooked or ignored by treaters, remedies may be sought at law.
Wishing to practice in the most effective and ethical manner possible, clinicians
commonly take steps to assure positive outcomes. In the regular follow-up of individuals
after treatment, they observe them closely and often modify their therapeutic approaches
accordingly. They are also at increased pains to utilize those treatments that sophisticated
research studies have indicated are efficacious.
Unfortunately, accumulated experience now suggests that such strategies, while
laudable, are not sufficient.
~ ~ _ .
Following patients clinically is an appropriate and useful
practice, but one that is directed primarily at a determination of whether the goals of
treatment were achieved rather than at treatment outcome. It is likely to provide an
incomplete picture of outcome results: attrition is considerable, and those who fail to
improve are less likely to return for follow-up appointments. If a follow-up visit is not
carefully structured, it may not systematically and quantitatively explore the multiple aspects
of outcome that are now felt to be important. If follow-up is carried out by those who
administered treatment, there is a tendency to perceive more favorable outcomes than may
actually exist, and treatment recipients will be reluctant to bring forth evidence that the
treatment they have been given has not been effective.
Being guided by research studies in the provision of treatment may enhance the
probability of positive results, but it does not guarantee them. The results of a treatment
outcome study are considered positive if, in the aggregate, the outcomes are significantly
better than those following either no treatment or a comparison treatment. But it is rare
that all treated subjects have positive outcomes, and the relevant clinical question is
whether a particular individual had a positive outcome.
Moreover, there are factors that may constrain the more general applicability of
research findings to clinical practice. The subjects of research studies frequently differ in
important ways from persons seen in clinical settings (Seiden, 1961; Hlatky et al., 1984;
Longabaugh and Lewis, 1988~. Although a treatment may be effective for research subjects,
it is not necessarily effective for a very different clinical population.
Apart from subject differences, there may also be differences in the treatment as
delivered. It used to be common for discussions of treatment outcome to be graced by a
modest qualifying phrase: "Treatment X had the following results in our hands." Although
now used only infrequently, the phrase is still highly meaningful, especially in the case of
complex nonbiological interventions, such as those that are often utilized in the treatment
of alcohol problems. Even with adequate quality assurance mechanisms in place (see
Chapter 5) there are likely to be differences between a treatment as studied in a research
OCR for page 315
DETERMINING OUTCOME
315
setting and the same treatment as delivered in a treatment setting. This state of affairs is
not necessarily unfortunate-variations on a basic treatment theme may be advantageous in
dealing with an extremely variable target population- but it does mean that results obtained
in a research setting cannot automatically be extrapolated beyond it.
Thus, although the carefully controlled research study may be an important method
for exploring treatment efficacy, its clinical applicability is limited by numerous factors.
From the standpoint of an individual seeking treatment, controlled studies have commonly
been carried out in other programs, at other times, by different staff. What the
prospective consumer of treatment services wants to know, however, is whether, given his
problem, his characteristics, and his circumstances, he can reasonably expect to achieve a
positive result from this program, at this time, and with these staff (cf. Paul, 1967; Pattison
et al., 1977~. The desire for such knowledge is shared by prospective third-party
reimbursers. Knowledge of outcome results is a cornerstone of the "buy right" strategy
advocated for all purchasers of health care (McClure, 1985a,b). According to this strategy,
third party reimbursers can improve the quality of treatment by systematically shifting their
economic support to providers who produce the best results at the most reasonable cost.
In the past the limited number of treatment providers constrained such a strategy, but the
growth of the treatment enterprise has now made it feasible.
What may be concluded from the foregoing considerations of clinical practice,
research design, accreditation, ethics, legal considerations, marketing, and financing is
relatively straightforward. Knowledge of the achievement of treatment goals is important,
and very much to be encouraged, but it is not sufficient. Individual treatment programs
must develop systematic and detailed knowledge of the outcomes experienced by those to
whom they deliver services. Because of constant flux in critical dimensions of the treatment
scene over time (changes in the patient population, in the treatment staff, in the available
alternatives, in funding policy, etc.), such knowledge cannot be occasional but must be
ongoing.
Although information regarding the achievement of short-term treatment goals is
sometimes sought, the development of a comprehensive understanding of outcome is not
common in the treatment of alcohol problems. Why this should be the case when multiple
considerations favor knowledge of outcome is uncertain, and (as will be seen in the next
section) there are some important exceptions. The reasons for the relative neglect of
outcome determination may not be specific to the treatment of alcohol problems, since it
is also a feature of the treatment of medical problems (Schroeder, 1987; Bunker, 1988;
Relman, 1988; Lohr et al., 1988; Wennberg, 1988~.
Some Examples of Systematic Outcome Determination
In some instances, concerted attempts have been made to determine the outcome
of treatment for alcohol problems. The state of Oklahoma operates a mental health
information system that generates data on the performance of all psychosocial treatment
programs. An addition to this system has been made by the state agency responsible for
the treatment of alcohol problems:
[W]e require the alcoholism programs to submit follow-up data on a
random sample of all the clients taken in. This information is collected
in a standardized form, and the service is reimbursed [for the collection
of this data] in the amount designated by the schedule of payments. The
random sample is generated every month by the computer from the pool
of patients intaken jsic] by the program 6 months earlier. The level of
completion of the follow-up quota is an important criterion in the decision
OCR for page 316
316 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
of our department to renew or deny renewal funding by each program.
(Paredes et al., 1981:384~.
More recently (as of January 1, 1988), the state of Minnesota has instituted a
requirement that all alcohol treatment programs receiving public monies must participate
in a similar sort of management information system that includes regular follow-up of
treated individuals. The development of systems of outcome monitoring is not a feature
of the public sector of service delivery exclusively, however. With the rise of utilization
management in the treatment of alcohol problems (Korcok, 1988; Lewis, 1988) a concern
with outcome has surfaced in the private sector. Given increasing competition among pro-
viders of treatment for alcohol problems, documentation of positive outcomes may become
a critical element in marketing. Commercial organizations that retail such documentation
in the alcohol field, for example, the Chemical Abuse/Addiction Treatment Outcome
Registry (CATOR), already include this inducement in their promotional materials.
Some details of the CATOR operation may be cited to illustrate how such an
organization operates. A private corporation located in Saint Paul, Minnesota, CATOR
contracts with individual treatment programs on a per capita basis (currently $98 per client
per year) to provide treatment outcome data. Program staff are trained by CATOR staff
working onsite to administer client data forms and forms documenting the details of
treatment. These forms are then submitted to CATOR, which conducts posttreatment
follow-up of individual clients by telephone from its main office.
The data that have been gathered are analyzed by CATOR staff, and feedback is
provided to the individual programs that subscribe to the service, not only in terms of
aggregate outcome for the program itself but in terms of how the program is doing in
comparison with similar programs in the same registry. Such comparisons are possible
because uniform data are obtained from all subscribing programs. The CATOR registry
is quite large; its most recent version contains individual data on approximately 31,000
persons, with completed follow-up data on approximately 17,000 (N. G. Hoffmann,
CATOR, personal communication, March 2, 1989~. Although it is used primarily to
prepare reports for its subscribers, the registry can also be employed to explore general
issues in the treatment of alcohol and drug problems.
Thus reports have been prepared on 1,776 women in treatment (Harrison and
Belille, 1987), on 1,824 adolescents in treatment (Harrison and Hoffmann, 1987), on 569
adults who completed outpatient treatment and were successfully followed for four
consecutive six-month intervals after treatment (Harrison and Hoffmann, 1988), and on
2,303 adults discharged from inpatient treatment who were successfully followed in the
same manner (Hoffmann and Harrison, 1988~.
In summary, there are multiple reasons for favoring the systematic monitoring of
the outcome of treatment, both in health care generally and in treatment for alcohol
problems specifically. Many treatment programs attempt to develop information on the
achievement of short-term treatment goals; although this is praiseworthy, it is not sufficient.
The systematic determination of the status of individuals at the end of treatment is not
common, and determination of outcome during the postdischarge period is even less
common. Several examples can be cited in which monitoring of the outcome of treatment
for alcohol problems has been and is currently being carried out. In view of the
fundamental importance of knowledge of outcome, however, much more needs to be done.
OCR for page 317
DETERMINING OUTCOME
2 ~ ~
A Major Caveat
For all of the reasons discussed above, the monitoring of outcome results is of
central importance to the treatment of persons with alcohol problems. In addition to its
many advantages, however, outcome monitoring has an important limitation. It does not
prove that the outcomes observed following treatment are the result of the treatment
provided.
From the manner in which outcome information is often used in marketing
treatment programs, this limitation seems not to be well understood. Imputations of
treatment efficacy based on outcome monitoring are a regular feature of mass media
advertising. The committee believes that outcome monitoring should be used only with
an understanding of its limitations; it also considers an examination of its limitations to
be a necessary counterbalance to the foregoing discussion.
If a given result occurs following a given event, there is an understandable
tendency to believe that the event caused the result. The belief is particularly compelling
if the event was intended to cause the result. Treatment is intended to produce positive
results; when a positive outcome is observed after treatment, the tendency is to conclude
that it is due to the treatment provided. Certainly, that is one possibility. Yet a little
reflection will indicate that there are alternative possibilities and that this
sequence-treatment followed by a positive result-does not necessarily mean that the positive
result was caused by the treatment.
As an example, let us take the familiar sequence of the alarm clock going off,
followed by the rising of the sun. That this sequence occurs is readily and repeatedly
verifiable. But the alarm clock does not cause the sun to rise, as may also be readily
verified by not setting the clock. The two events happen to occur in a time sequence
identical to the one that would be observed if the second event in the sequence were the
result of the first, that is, if there were a causal relationship between them. But there is
not. The temporal sequence is consistent with causality but does not demonstrate causality.
So frequent and ancient is the logical fallacy of the assumption that a temporal sequence
demonstrates causality that there is a Latin tag for it: post hoc ergo propter hoc, or (roughly)
After this, therefore because of this.
It follows that improvement subsequent to treatment is not necessarily the result
of the treatment provided. Of what might it be the result, then, if not of the treatment
itself? There are many possible alternative explanations. Alcohol problems, like other
problems, have a natural course~r, more accurately, several natural courses (see Chapter
2~. They may come and go. Treatment may have happened to occur in the middle of this
process but may not have affected it (much as the alarm clock happened to ring while the
sun was in the process of rising but did not cause it to rise). Life goes on while people
are in treatment, and such life events as the threatened loss of a job or an important
relationship may be the crucial determinants of outcome rather than treatment. That
alcohol problems can improve in the absence of any formal treatment is well known (see
Chapter 6~.
None of these considerations negates the value or importance of treatment. There
is ample evidence (see Chapter 5) that treatment can be crucial to positive outcome in
many persons. Moreover, treatment can be carried out with certainty; nontreatment events
or processes that might favor positive outcomes may, indeed, exist, but their occurrence is
far from certain. Some alcohol problems come and go, but others do not. Even if
improvement were eventually to occur without formal treatment, formal treatment may still
be indicated because it might accelerate the process. Thus, treatment is very important; but
the point here is that, because positive outcomes can occur that are related to factors other
than treatment, proof that treatment has produced a positive outcome is not a simple
matter.
OCR for page 318
318 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
To establish with certainty in a given instance that there was a causal relationship
between treatment and improvement requires a more elaborate procedure. Basically, the
results obtained in a group of persons who receive treatment must be compared with the
results obtained in an essentially similar group who do not receive treatment. If there are
differences in outcome between these two groups beyond what can be expected on the basis
of chance, the results can be attributed to the treatment with confidence.
The randomized controlled trial ARCH is this kind of procedure (see the discussion
in Chapter 5~. Randomization is the method commonly used to ensure the essential
comparability of the treated and untreated groups on factors affecting outcome. The term
controlled, used in this context, means that the presence of treatment has been controlled
for by having a comparison group that was not treated (or, in a common variant, received
a different kind of treatment). Outcome studies characteristically look only at people who
have been treated; they do not look at randomly selected, untreated controls drawn from
the same admission population. Hence the RCI, aided by a suitably selected comparison
group, is better able to test the effect of treatment.
Yet the RCI has its disadvantages (see Chapter 5), and outcome monitoring has
corresponding (and in many respects complementary) advantages. In a recent editorial in
the New England Journal of Medicine, the disadvantages of the RCI in medicine generally
were summarized as being of a practical nature: "We cannot afford to conduct randomized
controlled trials for every test, procedure, or medication in use. To do so would require
far too many research resources and would not produce results soon enough" (Greenffeld,
1989:1142~. In contrast,
The scope of observational studies can be expanded much more easily
than that of randomized controlled trials to include large numbers of
patients and providers, maximizing the opportunity to gauge the
effectiveness of routine medical care practices in various clinical settings,
by various clinicians, and for various patient groups. Answers to questions
about the effectiveness of care for important subgroups of patients
(including those with specific coexisting morbid conditions), which would
not be supplied by randomized controlled trials, can therefore be provided
.... Longitudinal observational studies permit both the examination of a
complex set of decisions, including the decision not to perform a
procedure, and the assessment of the supportive care that follows.
(Greenfield, 1989:1142)
The editorial goes on to caution that "care must be taken in interpreting the
results of longitudinal observational studies, even though they are intuitively appealing and
offer a quick solution to the information needs of policy makers." (p. 1143) It speaks of
the capability of such studies to "reduce our dependence on randomized controlled trials"
(p. 1143) rather than eliminating the need for such trials. A similar note is struck in a
recent Institute of Medicine background paper on the assessment of technological
innovation in medical practice (Gelijns, 1989~. The virtues of observational methods, which
are felt to have been enhanced by recent methodological developments, are enumerated
(pp. iii, 51-52), but the recommendation is for a mixture of methods: "Evaluation of the
risks and benefits of new technologies during their development will have to rely not only
on experimental methods (including randomized controlled clinical trials) but also on
improved observational methods of clinical conditions" (p. 51~.
As these sources suggest, there is an important and highly complementary
relationship between the RCE and the treatment outcome study. The RCI furnishes
evidence that improvements in outcome are due to the treatment provided but only under
the conditions of the experiment; it does not demonstrate, as indicated above, that the
OCR for page 319
DETERMINING OUTCOME
319
same results will consistently be produced in The real world. On the other hand, the
treatment outcome study does not prove that positive outcomes are due to the treatment
provided. But it does demonstrate that improvements have occurred sin the real worlds
following the delivery of the treatment.
Let us suppose that there has been (a) ample demonstration from controlled trials
that a particular treatment can be efficacious and (b) ample demonstration from treatment
outcome monitoring that the same treatment, concretely embodied in a specific treatment
program, is consistently associated with positive results. If these two circumstances coexist,
one can entertain a reasonable certainty that effective treatment is being provided. If
either element (a) or (b) is absent, one cannot be as certain. Thus, an important goal for
the immediate future is the increased implementation of both randomized controlled trials
and routine monitoring of treatment outcome.
Implementing Outcome Monitoring
Setting the Stage for Outcome Monitoring
Imagine a situation in which the following statement is made: "Sixty percent of
the people who pass through our treatment program achieve a positive outcome. Let us
assume that the statement is valid. How meaningful is it?
HA cautious response to the presentation of any treatment outcome rate, it has
been noted, Requires that we ask several questions" (Emrick and Hansen, 1983:1086~.
One question that immediately comes to mind is "What sorts of alcohol problems do the
people who come to your program exhibit?" A second question might be "What are the
characteristics of the people that are treated in your program?" A third question might be
"What sort of treatment is provided by your program?" In the absence of answers to these
and other questions, a simple statement about the proportion of positive outcomes is not
highly meaningful.
Specifying the Problem arid the Individual A 60 percent positive outcome rate in
a program seeing persons with mild alcohol problems of brief duration has a different
meaning than the same rate in a program seeing persons with severe problems of long
duration. A 60 percent positive outcome rate in a program seeing socially stable
individuals has a different meaning than the same rate in a program seeing socially unstable
individuals. The specification of problems and of individuals being considered for
treatment has been discussed earlier in Chapter 10. In a very real sense the determination
of outcome is a process of reassessment, in which the individual's status following
treatment is compared with his status prior to treatment. As has been noted, "pretreatment
functioning needs to be assessed for comparison with posttreatment adjustment, using
parallel pre-post data-gathering procedures" (Emrick and Hansen, 1983:1082~. Assessment
thus helps to set the stage for outcome determination.
Specifying the Treatment A 60 percent positive outcome rate in an elaborate,
time-consuming, and expensive form of treatment has a different meaning than the same
rate in a simple, brief, and inexpensive form of treatment. The relevant question here is
Outcome of what?" The specification of treatment is also important for the classification
of treatment programs and for matching to particular treatments (see Chapters 3 and 11~.
Briefly, there is a need to specify the treatments that are provided along multiple
dimensions including treatment orientation or philosophy, the stage of the problem at
which the treatment is directed, the setting of the program, the treatment modality utilized,
its goals or objectives, the criteria for selecting individuals for treatment, its length, its
intensity, its content, and other factors.
OCR for page 320
320 BROADENING lilE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
For purposes of outcome determination it is not enough to know what treatment
was provided; one should also know what treatment the individual received. Many
treatment programs offer several treatment options simultaneously (e.g. group therapy,
individual counselling, and educational lectures), and it is assumed that all persons in the
program receive all of them in similar proportions; in fact, this may not be the case (W.
J. Filstead, Parkside Medical Services, personal communication, March, 1988~. If a
medication such as disulfiram (Antabuse) is dispensed, it is important to know whether it
is being taken. Experience suggests that compliance with a prescribed regime of medication
does not always occur, and a monitoring mechanism may be advisable to ensure that the
medication is actually being used (cf. Peachey and Annis, 1984; Peachey and Kapur, 1986~.
Perhaps it is only stating the obvious to emphasize that, in order to understand
treatment outcome data, one must be clear about the nature of the problems being treated,
the nature of the individuals undergoing treatment, and the nature of the treatment.
Unfortunately, clarity about these matters is too frequently lacking at the present time. It
has long been a goal of research on the treatment of alcohol problems that all of these
critical dimensions should be explicitly accounted for (but the goal is not always realized;
see Sobell et al., 1987~. What is being urged, in a sense, is that the distinction between
treatment and research practice be reduced to the point that they approximate each other.
The Content of Outcome Monitoring
Knowledge of an individual's status after completing treatment is much more
meaningful if it can be compared closely with his or her status prior to treatment.
Accordingly, there should be a parallelism between the coment of assessment and the content of
outcome determination. A comparison of the "core indices [of outcome] to be used for all
treatment-evaluation studies" noted in a recent review (Emrick and Hansen,
1983:1084-1085) and the content of assessment suggested in Table 10-2 of this report
demonstrates such parallelism. Among the shared content domains are physical health,
including morbidity and mortality; drinking behavior; other substance use; legal problems;
vocational functioning; family and social functioning; emotional functioning; cognitive
functioning; and the operation of situational variables and life events.
Some aspects of a comprehensive pretreatment assessment might not be directly
relevant posttreatment for example, a specification of the individual's treatment goals.
On the other hand, it would be of great importance to determine at the follow-up point
or points whether the goals initially viewed as appropriate had been achieved, or whether
there had been a shift of goals during the course of treatment. A thorough posttreatment
assessment of customer satisfaction might be valuable in several ways. Guidance about
modifications in program content and process could be obtained, but a more positive and
participatory feeling toward the program and toward treatment generally might also be
engendered by the fact of soliciting such feedback. In addition, regular reports of customer
satisfaction may serve as an incentive and reward to the staff.
Prior treatment history would not be reassessed in outcome monitoring, but it
would be essential to document any further treatment received in the interim between
treatment and follow-up. Certain individual characteristics that might remain relatively
unchanged (e.g., intelligence, personality, and family history) might not be reassessed,
although specific reasons for doing so in individual cases come to mind. Because full
testing of cognitive functioning is an expensive and time-consuming procedure, it might
be repeated only if initial impairment had been detected. However, it is a common
experience that marked improvement in cognitive functioning within the normal range can
be characteristic of a positive treatment outcome, and the documentation of this
improvement may have a salutary effect on the individual's outlook.
OCR for page 321
DETERMINING OUTCOME
321
There has been a tendency in the field to rely primarily on changes in the level
of use of alcohol to determine the outcome of treatment. Certainly, level of use is a
critical focus for both treatment and outcome, and its status will bear a strong relationship
to other outcome parameters (Babor et al., 1988~. But the relationship is not invariable
and may be quite discordant in some individuals, particularly those with lower levels of
alcohol use (cf. Edwards, 1974~. A case can therefore be made, as with assessment, for
the use of multiple dimensions in the determination of outcome. "Because of the non-
orthogonal relationships among treatment outcome domains," one authoritative paper states
succinctly, Use of multiple outcome measures is essential" (Emrick and Hansen, 1983:1084~.
Brief mention should be made here of a problem in selecting individuals for
treatment that was touched upon earlier in this chapter. It has long been clear that certain
population characteristics are associated in general with favorable outcomes. Those with
less severe problems accompanied by fewer symptoms and consequences and of shorter
duration, as well as such personal and situational characteristics as affluence, high
intelligence, high educational levels, social stability, a high level of verbal skills and
personal attractiveness are likely to do well in treatment and are more attractive to work
with (cf. the TRAVIS syndromes of Schofield [19643~. A program could substantially
improve its proportion of positive outcomes, and the ease of its staff, by choosing to deal
only with problems and individuals of this kind. Although it is true that some such
individuals do not do well and that persons in this group may have as profound a need for
treatment as any others, a high proportion of positive outcome results in a population of
this kind carries a rather different meaning than would be the case in other populations
(e.g., a population of skid row inhabitants). Less substantial gains in a more impaired and
less advantaged group may represent an equal or superior therapeutic achievement. Thus,
in the understanding of outcome results, relative rather than absolute standards are best
employed.
At present, there is great variability in the measures that are used when outcomes
are examined (Emrick and Hansen, 1983; Sobell et al., 1987~. The use of a uniform set of
treatment outcome criteria would be an enormous advantage because it would permit the
aggregation of data and the comparison of treatment outcomes across programs. For
example, with uniform outcome criteria it would be possible to compare the outcomes of
individuals treated with pharmacotherapy and those treated with counseling. At present,
data cannot be aggregated in this way because different and often incompatible outcome
measures are used. The argument that particular programs need to present their data in
particular ways to satisfy various data and funding requirements does not preclude the use
of uniform criteria as well. If unique information is required, it can be collected in addi-
tion to shared uniform measures.
As with the development of a uniform assessment process, agreement on uniform
outcome criteria needs to be forged through a series of consensus exercises involving all
segments of the treatment community. Indeed, if the determination of outcome is
conceptualized in large measure as reassessment, consensus on the content of both as-
sessment and outcome determination could be sought simultaneously. The increment in
useful information that would result from uniformity in assessment and outcome data
would be considerable and would contribute importantly to the further enhancement of
. .
positive treatment outcomes.
The Process of Outcome Monitoring
Training A quite practical aspect of viewing outcome monitoring as reassessment
is that only one set of staff, rather than two, is required. The measures being parallel,
training to use them for assessment is also training to use them for follow-up. In many
OCR for page 322
322 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
instances, staff will be reassessing individuals that they assessed initially; such familiarity
may contribute to a greater likelihood that reassessment will be accepted and provides
valuable feedback directly to relevant staff.
Timing How long after treatment should outcome be determined? In practice the
range has been very wide indeed, ranging from a few weeks to as many as five years
(Ogborne, 1984). What period might be optimal is a knotty question. Outcome results
tend to be unstable-that is, individuals who are evaluated at time A tend to have differ-
ent outcomes than when evaluated at time B (Annie and Ogborne, 1980)-and it is
questionable whether time A or time B represents the Cereals outcome.
There is also the issue that "although a lengthy period of follow-up may cast
important light on the course of patients' drinking and related problems posttreatment,
as the period of evaluation is lengthened less of what is observed can be attributed directly
to treatment" (Emrick and Hansen, 1983:1082). What one may be observing over lengthy
periods of time is the effect of nontreatment factors (e.g., loss of jobs, promotions,
marriages, divorces, etc.) rather than the effect of treatment (Cronkite and Moos, 1980).
Accordingly, there may be a progressive decline in treatment effect from the point of
treatment termination. Such a progressive decay curve is often seen in medical treatments,
in which the active ingredient in therapy is what the treater does. It has also been seen
in treatment for alcohol problems, as well as for similar problems (Hunt et al., 1971~.
Individuals pursuing such a posttreatment course have been termed "faders" (Moos et al.,
1982).
But there are other patterns of response over time. For example, in some
treatments for alcohol problems, individuals are taught skills that they then apply to their
life situations. With increased practice, they may apply the skills more effectively. In these
circumstances the treatment effect may increase rather than decrease from the point of
formal treatment termination. Such a pattern has been observed (Annie and Ogborne,
1980); in this case the individuals have been called ~sleepers" (Moos et al., 1982). If
different forms of treatment have variable effectiveness curves over time after treatment,
how can one specific point in time be chosen as the "gold standard" for determination of
outcome?
The answer probably is that it cannot be. A compromise would be to assess
outcome at regular intervals following treatment and to continue to do so for a reasonable
period of time. A schedule of reassessment, then, at 6 months, 12 months, and 18 months
seems defensible, especially if the reassessment serves clinical as well as outcome
determination purposes. For clinical reasons alone one would wish to evaluate individuals
posttreatment on a schedule of at least this stringency. It would be possible, with such a
pattern, to identify both Faders" and nsleepers.n
Sampling Outcome monitoring can be carried out for all persons who enter
treatment. As an alternative the determination can be made on selected samples (e.g.,
every third admission) as in the Oklahoma system. Using a sample is an effective
procedure if a generalized knowledge of outcome is what is required. However, if the
purposes of outcome monitoring are also in part clinical, it is more reasonable to opt for
the monitoring of outcome for all persons who have entered treatment. For clinical
purposes, one must know the outcome of treatment in every case, not excluding those who
have failed to complete the prescribed course of treatment. The effectiveness of sampling
in meeting the ethical obligations of prisms non nocere and in preventing the potential
legal consequences of failure to identify harmful effects of treatment upon particular
individuals is unclear.
Setting In research studies, follow-up interviews are commonly carried out by
means of a face-to-face interview, often in the patient's home. As an alternative, patients
may be asked to return to the scene of their treatment for follow-up interviews. Because
either of these options involves direct personal contact, additional direct observations can
OCR for page 323
DETERMINING OUTCOME
323
be made, and such procedures as breathalyzer testing or the use of biological markers can
be deployed.
CATOR, as noted earlier, has conducted its follow-up interviews by telephone;
others have used mailed questionnaires. These techniques, although less costly and
time-consuming, do not permit the observations and methods available with direct contact
and are thought to be less likely to produce valid responses, particularly from those who
have not been satisfied with the treatment they have received. There is little in the way
of empirical data to substantiate this impression. It may be that, to achieve a satisfactory
response level-the higher, the better-a combination of methods will need to be used.
Corroboration The issue of the reliability of self-reports was discussed in Chapter
10. Self-reports seem inherently to be neither reliable nor unreliable; rather, they are
markedly affected by the circumstances in which they are elicited. Under favorable
circumstances, self-reports achieve a satisfactory level of reliability.
On the other hand, it is undoubtedly useful to obtain corroborating data from
wari^~c f`~rtPrn~1 cn'1r~Pc. Anti at least in research studies. there has been an increasing
tendency to do so (Sobell et al., 1987~. The usual sources include family and other
informants, public records, and the use of various biochemical tests to measure alcohol
consumption either directly (e.g., breathalyzers, urinalyses) or indirectly (e.g., such markers
as gamma glutamyl transpeptidase and high-density lipoprotein). These measures increase
the validity of outcome information but at a cost of time, money, and effort.
Utility Outcome information may be principally useful in persuading third-party
payers to purchase services from a particular treatment program, in persuading prospective
clients to enter the program, and in qualifying for accreditation. In a larger sense,
however, outcome data provide an ethical justification for purveying treatment and a means
of improving its effectiveness. From an ethical standpoint the provision of treatment in the
absence of knowledge of results is a questionable procedure. (Of course, if the results are
not favorable, the ethical problems in continuing to provide treatment may be insu-
perable.) In most instances, outcome information is likely to indicate that admission to a
given treatment program is followed by positive outcomes in a significant proportion of the
people who seek its services; at the same time, however, it is followed by no significant
change or by a worsening of problems in another but also significant proportion. A
number of responses are possible in the face of such information: (a) improvements to the
treatment being currently provided may be introduced, after which outcome may be
examined again; (b) additional treatments that seem likely to help those who are not being
helped may be introduced, and their outcomes may be examined; and (c) those who are not
being helped by the treatment provided may be preferentially referred to programs whose
outcome information indicates a higher probability of a positive outcome for individuals
with their characteristics and types of problems. These alternatives are more extensively
discussed in Chapters 11 and 13. All of them may substantially improve treatment
outcomes, both for the individual programs and for the persons seeking treatment.
, in, V %*" arm._, arm. ~ _. __ , _ _, _
The Locus of Responsibility for Outcome Monitoring
The monitoring of treatment outcomes can be carried out under differing auspices.
Responsibility for determining outcome can be taken (a) by individuals external to the
program, (b) by the program itself, or (c) by some combination of the two. Theoretical
and practical advantages may be advanced for all three approaches.
The principal advantage of external evaluation is greater objectivity. Programs
themselves may be presumed to have an interest in demonstrating that their treatment is
associated with a high level of positive outcomes; external evaluators presumably would
not share such an interest. The principal disadvantages of external evaluation include lack
OCR for page 324
324 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
of understanding of the program being evaluated, lack of availability, and cost. In Chapter
10, some of the advantages of an assessment process that was functionally independent of
treatment were discussed. Similar advantages would accrue to functionally independent
determination of treatment outcome; and, because outcome determination and reassessment
are substantially the same process, both could be performed by the same functional unit.
Because this option has already been discussed, the focus here will be on options (b) and
(c).
Programs themselves may conduct treatment outcome monitoring. Indeed, when
clinical as opposed to research treatment programs are being evaluated, this method is
probably the most common. Program personnel are at least geographically available and
presumably sympathetic to and knowledgeable about their own programs. In this case,
information on treatment outcomes would be directly useful to the very personnel who
carried out the evaluation and hence would tend to ensure its pertinence and perhaps also
narrow the notorious treatment-research gap (cf. Garfield, 1978; Miller, 1987~. Mounting
research projects within treatment programs may also foster a more observing and objective
attitude among staff toward their program than would otherwise be the case.
As an example of a within-program evaluation, half of a group of 100 patients
who were seen for initial assessment of alcohol problems were randomly sent "a personal
letter expressing concern for the patient's well-being and repeating our invitation for
further assistance." Fifty percent of those sent the letter returned for additional contact, as
compared with 31 percent of those not sent the letter; 76 percent returned the same day
the letter was received, as compared with 12.5 percent; and 80 percent returned sober, as
compared with 31 percent. All results were significantly beyond what would have been
expected on the basis of chance (Koumans and Muller, 1965~. A second and similar study
demonstrated the effectiveness of a telephone call versus no call (Koumans et al., 1967~.
A large residential treatment program visited by staff and committee members of
this study conducted an evaluation of the effects of their program on persons who had
alcohol or drug problems (or both) and eating disorders. The evaluation was carried out
by an internal research group, which designed the study in close collaboration with clinical
staff. Records were reviewed to establish the prevalence of eating disorders, and special
questionnaires to be used both at admission and at follow-up were devised. The key
finding was that individuals with eating disorders were found to have outcome results with
respect to alcohol or drug problems that were similar to those of individuals without eating
disorders. After much discussion, the program decided to continue to admit individuals
with both alcohol or drug problems and eating disorders (J. Spicer, Hazelden Foundation,
personal communication, 1988~.
If programs conduct their own evaluations, problems arise in terms of validity,
allocation of resources, and the uniformity of the data gathered. How objective program
personnel would be in evaluating the results of their own work is an important issue. If
the research were in the hands of program but nonclinical personnel, as in the example
immediately above, the effects of bias might be somewhat attenuated. Even if the results
were in fact not biased, their persuasiveness to others might be less than with an external
evaluation. Finally, when individual programs gather data on their own clients they tend
to gather them in an idiosyncratic manner. This lack of uniformity makes comparisons
across programs, even on such seemingly straightforward items as demographic variables,
difficult or impossible. Consensus on outcome measures, as recommended above, could be
helpful in reducing this problem.
At present, most programs do not possess staff devoted to outcome monitoring.
One approach would be to reallocate staff assignments so that some proportion of clinical
time (and thus some proportion of the program budget, depending upon the intensity of
follow-up to be done and the methods to be employed) was used for this purpose.
OCR for page 325
DETERMINING OUTCOME
325
Alternatively, individual programs could be enriched by providing them with additional
personnel to design and carry out outcome monitoring.
Programs may collaborate with external organizations to monitor their outcomes.
CATOR is an example of this approach. It relies upon program personnel (after training)
to produce descriptive data on patients and treatments but does its own follow-up and data
analysis. In its promotional material, CATOR stresses the value of its externality: "Perhaps
most importantly, potential patients and referral sources know you care about treatment
outcomes by your willingness to have them audited by an objective, respected outside
source" (Belille, n.d. [ca. 1987~:4~.
The state systems in Oklahoma and Minnesota that were described earlier are
similar in their approach, in that they involve collaborations between programs and external
agencies. However, they differ in that treatment program personnel, rather than personnel
of the collaborating agency, are responsible for conducting the outcome determinations.
This practice also changes the financing pattern; in Oklahoma, for example, programs are
paid for conducting outcome determinations on a prearranged per capita basis. Also, of
course, the basis of participation differs; participation in CATOR is optional, but partic-
ipation in the state systems is obligatory for programs receiving public funds.
It would appear from this brief consideration of the locus of responsibility for
outcome monitoring that all of the options described have both advantages and
disadvantages. If validity is the bottom line with respect to outcome determination (and
it is for most of the purposes that outcome information is asked to serve) then the
maximum feasible degree of externality in outcome determination is desirable. If outcome
determination for practical reasons cannot be wholly external, some combination of internal
and external loci of responsibility would be the next most desirable option. Examples
would be a commercial organization such as CATOR in the private sector or the state
systems in Oklahoma and Minnesota in the public sector. Providing that steps were taken
to ensure functional independence, the use of an assessment or research group within a
program might be yet another option.
Wholly internalized outcome monitoring may be useful for various local purposes
but will tend not to be persuasive beyond the confines of the program. Nevertheless, as
indicated above, such internal program research can be of great value in making sound
decisions about the future course of the program. In addition, a program that has become
accustomed to such internal research may well develop a capacity for objective self-scrutiny
that will eventually result in an openness to external evaluation. Although not a fully
satisfactory method of determining treatment outcome, internal monitoring is much to be
preferred to an absence of effort toward this end.
The Funding of Outcome Determination
It is generally assumed that the introduction of widespread, comprehensive, and
ongoing outcome monitoring, whether in the alcohol treatment field or in the medical
treatment field generally, will raise the costs of providing treatment. At the least it is
assumed that outcome monitoring will require an initial investment until a payoff of
improved treatment efficiency is realized. Thus a medical periodical has editorialized that
To achieve these objectives will require much new financial support and unprecedented
cooperation among physicians, government, private insurers, and employers" (Relman,
1988:1222).
Some are uncertain whether a financial payoff will in fact be realized. In calling
for a national system of "outcomes management," Ellwood (1988) says that such a program
Will not automatically favor a decrease or increase in health care expenditures (p. 1556~.
OCR for page 326
326 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
This is especially the case if any increase in the efficiency of treatment makes it more
attractive so that demand for treatment is also increased.
As will be discussed in Chapter 21, the cost of treatment for alcohol problems is
great, but is dwarfed by the cost of the consequences of alcohol problems. Systematic
knowledge of outcome and its application to treatment may increase effectiveness and
efficiency by improving treatment matching. The committee considers it likely that the
overall costs of treatment may rise as a result of its recommendations because savings
owing to improved effectiveness and efficiency will be more than offset by additional costs
arising from the treatment of greater numbers of persons who are newly attracted to an
expanded and improved system. Even though the costs increase, however, they will
continue to represent only a fraction of the cost of the consequences of alcohol problems.
The effort involved in determining treatment outcome and utilizing that knowledge to
improve treatment is thus likely to prove worthwhile. More people will be treated, and
they will be treated more appropriately; these constitute important benefits.
Conclusions and Recommendations
There is much to be said for determining whether the short-term goals of
treatment for alcohol problems, such as reduction in the level of use of alcohol, have been
achieved. More needs to be done in this regard. But a more pressing (although related)
need is the determination of the longer-term outcome of treatment. Not a substitute for
more rigorous controlled techniques that can demonstrate treatment efficacy, outcome
monitoring nevertheless offers benefits in that it addresses many important issues, is more
readily implemented on a broad basis, and complements significantly what can be learned
in other ways.
Ideally, the outcome of treatment for all individuals entering specialized treatment
programs for alcohol problems should be determined. There are a multiplicity of reasons
for such a course, including ethical reasons, but the principal purpose would be to improve
the ability to provide the most effective treatment to each individual by serving as a guide
to matching. To reach this goal, consensus must be achieved or' the need for outcome
determi~ifor', on. the parameters to be used in determining outcome, and on the optimal waylays)
to go about making outcome determinations. Many variant approaches to all of these matters
are possible. The committee believes, however, that a quantum increment ire alteration to
outcome determir~wr' Is crucu~l to the future of the effort to treat alcohol problems.
REFERENCES
Annis, H. M., and A. C. Ogborne. 1980. The temporal stability of alcoholism treatment outcome results. Addiction
Research Foundation, Toronto.
Babor, T. F., Z. Dolinsky, B. Rounsaville, and J. H. Jaffe. 1988. Unitary versus multidimensional models of
alcoholism treatment outcome: An empirical study. Journal of Studies on Alcohol 49:167-177.
Belille, C. n.d. (ca. 1987~. Bringing definition to a soft science. Saint Paul, Minn.: Chemical Abuse/Addiction
Treatment Outcome Registry (CATOR), Ramsey Clinic.
Bunker, J. P. 1988. Is efficacy the gold standard for quality assessment? Inquiry 25:51-58.
Commission on Accreditation of Rehabilitation Facilities (CARFV). 198$. Program Evaluation in Alcoholism and
Drug Abuse Treatment Programs. Tucson, Arizona: Commission on Accreditation of Rehabilitation Facilities.
Cronkite, R. C., and R. H. Moos. 1980. Determinants of the posttreatment functioning of alcoholic patients: A
conceptual framework. Journal of Consulting and Clinical Psychology 48:305-316.
OCR for page 327
DETERMINING OUTCOME
327
Edwards, G. 1974. Drugs, drug dependence, and the concept of plasticity. Quarterly Journal of Studies on Alcohol
35:176-195.
Ellwood, P. M. 1988. Outcomes management: A technology of patient experience. New England Journal of
Medicine 318:1549-1556.
Emrick, C. D., and J. Hansen. 1983. Assertions regarding effectiveness of treatment for alcoholism: Fact or
fantasy? American Psychologist 38:1078-1088.
Garfield, S. L. 1978. Research on client variables in psychotherapy. Pp. 191-232 in Handbook of Psychotherapy
and Behavior Change: An Empirical Analysis, 2nd ea., S. L. Garfield and A. E. Bergin, eds. New York: John
Wiley and Sons.
Gelijns, A. C. 1989. Technological Innovation: Comparing Development of Drugs, Devices, and Procedures in
Medicine. Washington, D.C.: National Academy Press.
Greenfield, S. 1989. The state of outcome research: Are we on target? New England Journal of Medicine
320:1142-1143.
Harrison, P. A., and C. A. Belille. 1987. Women in treatment: Beyond the stereotype. Journal of Studies on
Alcohol 48:574-78.
Harrison, P. A., and N. G. Hoffmann. 1987. CATOR 1987 Report: Adolescent Residential Treatment: Intake
and Follow-Up Findings. Saint Paul, Minn.: Chemical Abuse/Addiction Treatment Outcome Registry (CATOR),
Ramsey Clinic.
Harrison, P. A., and N. G. Hoffmann. 1988. Adult Outpatient Treatment: Perspectives on Admission and Outcome.
Saint Paul, Minn.: Chemical Abuse/Addiction Treatment Outcome Registry (CATOR), Ramsey Clinic.
Hlatky, M. A., K L. Lee, F. E. Harrell, Jr., R. M. Califf, D. B. Ptyor, D. B. Mark, and R. A. Rosati. 1984. Tying
clinical research to patient care by use of an observational data base. Statistics in Medicine 3:375-384.
Hoffmann, N. G., and P. A. Harrison. 1988. Treatment Outcome: Adult Inpatients Two Years Later. Saint Paul,
Minn.: Chemical Abuse/Addiction Treatment Outcome Registry (CATOR), Ramsey Clinic.
Hunt, W. A., L. W. Barnett, and L. G. Branch. 1971. Relapse rates in addiction programs. Journal of Clinical
Psychology 27:455-456.
Korcok, M. 1988. Managed Care and Chemical Dependency: A Troubled Relationship. Providence, R.I.: Manisses
Communications Group, Inc.
Koumans, Al J. R., and J. J. Muller. 1965. Use of letters to increase motivation for treatment in alcoholics.
Psychological Reports 16:1152.
Koumans, A. J. R., J. J. Muller, and C. F. Miller. 1967. Use of telephone calls to increase motivation for treatment
in alcoholics. Psychological Reports 21:327-328.
Lewis, J. S. 1988. Growth in managed care forcing providers to adjust. Alcoholism Report 16~24~:1.
Lohr, K N., K D. Yordy, and S. O. Thier. 1988. Current issues in quality of care. Health Affairs 7:5-18.
Longabaugh, R., and D. C. Lewis. 1988. Key issues in treatment outcome studies. Alcohol Health and Research
World 12:168-175.
McClure, W. 1985a. Buying right: The consequences of glut. Business and Health 2(9):43-46.
McClure, W. 1985b. Buying right: How to do it. Business and Health 2(10):41-44.
Miller, W. R. 1987. Behavioral alcohol treatment research advances: Barriers to utilization. Advances in Behavior
Research and Therapy 9:145-164.
Moos, R. H., R. C. Cronkite, and J. W. Finney. 1982. A conceptual framework for alcoholism treatment evaluation.
Pp. 1120-1139 in Encyclopedic Handbook of Alcoholism, E. M. Pattison and E. Kaufman, eds. New York: Gardner
Press.
OCR for page 328
328 BROADENING THE BASE OF TREATMENT FOR ALCOHOL PROBLEMS
Ogborne, ~ G 1984. Issues in follow-up. Pp. 173-215 in A System of Health Care Delivery, vol. 3, F. B. Glaser,
H. M. Annis, H. A. Skinner, S. Pearlman, R. L. Segal, B. Sisson, A. C. Ogborne, E. Bohnen, P. Gazda, and T.
Zimmerman. Toronto: Addiction Research Foundation.
Paredes, A, D. Gregory, and O. H. Rundell. 1981. Empirical analysis of the alcoholism services delivery system.
Pp. 371-404 in Research Advances in Alcohol and Drug Problems, vol. 6, Y. Israel, F. B. Glaser, H. Kalant, R.
E. Popham, W. Schmidt, and R. G. Smart, eds. New York: Plenum Press.
Pattison, E. M., M. B. Sobell, and L. C. Sobell. 1977. Emerging Concepts of Alcohol Dependence. New York:
Springer Publishing Company.
Paul, G. L. 1967. Strategy of outcome research in psychotherapy. Journal of Consulting Psychology 31:109-118.
Peachey, J. E., and H. M. Annis. 1984. Pharmacologic treatment of chronic alcoholism. Psychiatric Clinics of
North America 7:745-756.
Peachey, J. E., and B. M. Kapur. 1986. Monitoring drinking behavior with the alcohol dipstick during treatment.
Alcoholism: Clinical and Experimental Research 10:663 666.
Relman, ~ S. 1988. Assessment and accountability: The third revolution in medical care. New England Journal
of Medicine 319:1220-1222.
Schofield, W. 1964. Psychotherapy: the Purchase of Friendship. Englewood Cliffs, NJ.: Prentice-Hall.
Schroeder, S. A. 1987. Outcome assessment 70 years later. Are we ready? New England Journal of Medicine
316:160-162.
Seiden, R. H. 1961. The use of Alcoholics Anonymous members in research on alcoholism. Quarterly Journal of
Studies on Alcohol 21:506-509.
Sobell, M. B., S. Brochu, L. C. Sobell, J. Roy, and J. A. Stevens. 1987. Alcohol treatment outcome evaluation
methodology: State of the art 1980 84. Addictive Behaviors 12:113-128.
Wennberg, J. E. 1988. Improving the medical decision-making process. Health Affairs 7:99-106.
Ziskin, J. 1970. Coping with Psychiatric and Psychological Testimony. Beverly Hills, Calif.: The Law and Psychiatry
Press.
Representative terms from entire chapter:
outcome monitoring
