Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 55
2
Assessment of the QIO Program:
Findings and Conclusions
CHAPTER SUMMARY
This chapter describes the study's evaluation approach, summa-
rizes the key findings discussed in detail in Part II, and presents
the committee's conclusions about the impact of the Quality Im-
provement Organization program. The committee concludes that
although the evidence regarding the program's effects on quality
of care is limited and inconclusive, the program has the potential
to be a valuable asset as providers become more involved with
public reporting and incentive payments from Medicare and other
insurers.
The historical overview of the Quality Improvement Organization
(QIO) program in Chapter 1 describes a public program that has changed
direction, functions, and methods frequently over its 35-year history. Given
the absence of earlier evaluations of the program, uncertainties exist with
regard to its potential roles in the future. As a base for the recommenda-
tions for the future of the program presented in Chapters 4 and 5, this
chapter details the committee's findings and conclusions concerning the
program's impacts. Specifically, the chapter addresses whether:
· There has been improvement in the quality of health care services
provided to Medicare beneficiaries.
· The QIO program has contributed to that improvement.
· Certain components of the QIO program should be eliminated or
strengthened.
· There should be a continuing federal role in technical assistance for
quality improvement and, if so, whether that role should be stronger.
It becomes clear from the descriptive details presented in Part II that the
55
OCR for page 56
56 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
QIO program is richly varied. The organizations holding the QIO contracts
for each state often differ dramatically in structure, in the services they
provide, and in their quality intervention programs, perhaps reflecting dif-
ferences in the health care services in their communities. This variation
makes it difficult to generalize about all QIOs and the program at the na-
tional level, as well as to determine what changes in structure might be of
most benefit to the program and yield the greatest improvements in the
quality of care.
The historical overview presented in the previous chapter shows how
the program has changed over time. Nonetheless, many of the organiza-
tions holding state contracts and some of their key leadership have persisted
with the program through much of its evolution and have demonstrated
remarkable flexibility in adapting to each new statement of work. Through
that evolution, the QIOs have built partnerships with the providers and
other key stakeholders in their states, such as state departments of health
and local chapters of the American Association for Retired Persons. The
committee gathered anecdotal evidence of many favorable and some less
favorable relationships between QIOs and providers, brought to the com-
mittee's attention at national conferences, site visits, and informal discus-
sions with providers and consumers. In addition, results of large national
surveys of physicians, hospitals, nursing homes, and home health agencies
reveal generally favorable attitudes toward their local QIO. (The provider
surveys are discussed in detail in Chapter 10.) The committee finds these
relationships and the QIOs' capacity for adaptability to be valuable assets.
The remainder of this chapter begins with a discussion of the first two
questions raised above: (1) whether care for Medicare beneficiaries has im-
proved, and (2) whether the improvements, if any, can be attributed to the
QIO program. Alternatives to the current QIO program, along with their
advantages and disadvantages, are considered next. The chapter then ad-
dresses program infrastructure at both the national and state levels. Next is
a discussion of the main functions of the QIOs, followed by a review of
structural issues, such as funding and board composition. Finally, the chap-
ter addresses the oversight responsibilities of the Centers for Medicare and
Medicaid Services (CMS). Detailed data and analyses supporting the com-
mittee's findings and conclusions are presented mainly in Part II and the
appendixes, relevant portions of which are referenced throughout the dis-
cussion here.
EVIDENCE OF QUALITY IMPROVEMENT IN MEDICARE
The Medicare program has carefully tracked the growth in the numbers
of beneficiaries, the expenditures made on their behalf, and the increasing
variety and use of covered services over the history of the program; how-
OCR for page 57
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 57
ever, it has produced much less information on the quality of those services.
The former director of the QIO program in CMS examined Medicare ser-
vices by comparing data for 22 clinical inpatient and outpatient measures
against state baseline performance reported through the QIO program from
19981999 to 20002001. It was found that the weighted national aver-
age, as well as the state with the median performance, had improved on 20
of the 22 indicators (Jencks et al., 2003). A recent national study employing
9 of the same hospital quality indicators used by Jencks and colleagues
(2003), along with publicly reported data on Medicare and all other pa-
tients, found that the level of hospital performance had improved on all but
1 measure (Jha et al., 2005). Although these studies were less than ideal--in
particular, the number of measures used in each study was limited, and the
measures focused primarily on care provided within hospitals--their results
indicate that health care for Medicare beneficiaries has improved. There are
signs of improvement as well in clinical activities targeted as national pri-
orities on which QIOs and other organizations have focused, such as
mammograms and care for heart attacks and diabetes (Leatherman and
McCarthy, 2005). Improvements in care for beneficiaries in managed care
were also noted from 2000 to 2004 (NCQA, 2005). On the other hand,
much evidence, cited in Chapter 1 of this report and in Performance Mea-
surement: Accelerating Improvement (IOM, 2006), indicates that the qual-
ity of care varies greatly from provider to provider by geographic location,
race or ethnicity, and income of the beneficiary and that many people do
not receive all the services they need, particularly appropriate preventive
care. There is substantial room for further improvement.
Can the gradual improvements in care that have been accomplished be
attributed to the QIO program? Because of the nature of their evaluation
designs, the studies mentioned above cannot be used to determine the cause
of the improvements documented or attribute them to the QIO program.
During site visits and focus group discussions conducted for this study, the
committee heard frequent anecdotal evidence from providers and the QIOs
regarding positive impacts on quality resulting from the multiple forms of
technical assistance provided by the QIOs (see Chapter 8). Some studies of
specific, limited QIO interventions or collaboratives have also documented
improvements over time, but likewise cannot be used to conclude that the
improvements were due to the QIOs' efforts (Marciniak et al., 1998; Kiefe
et al., 2001; Gould et al., 2002; Chu et al., 2003; Daniel et al., 2004;
Dellinger et al., 2005). In addition, preliminary data on several measures of
the quality of care under the 7th scope of work (SOW) suggest that the
QIOs may have had a positive impact on the care received from nursing
home, home health agency, and hospital providers that participated inten-
sively with their QIOs ("identified participants") in comparison with that
received from all such providers statewide. The data also suggest that those
OCR for page 58
58 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
identified participants achieved greater improvements on measures related
to the clinical area of their particular intervention relative to improvements
achieved by participants who collaborated with QIOs on interventions un-
related to those measures (personal communication, W. C. Rollow, July 8,
2005; personal communication, J. Kelly, September 8, 2005). The analyses
were incomplete as of this writing, however, and it is unclear whether these
apparent differences will be significant.
Given the lack of consistent and conclusive evidence in scientific jour-
nals and the lack of strong findings from the committee's analyses, it is not
possible to determine definitively the extent of the impact of the QIOs and
the national QIO infrastructure on the quality of health care received by
beneficiaries. Many confounding factors make it difficult to attribute the
results obtained thus far. Also, the literature does not address the QIO
program as a whole; rather, it merely addresses the impacts of specific qual-
ity improvement activities of individual QIOs or quality measures aggre-
gated at the state or national level. Other aspects of the program, such as
the impact of QIO case reviews on quality and the value of QIO Support
Centers (QIOSCs), have received little or no scrutiny from evaluators. (See
Chapter 8 for a discussion of QIOSCs.)
One challenge to evaluating the QIO program is that QIOs recruit
voluntary participants and generally partner with multiple stakeholder or-
ganizations to conduct quality improvement interventions (see Chapter 8).
It would be difficult to identify true control groups; random assignment
has not been tried because of practical and political implications; studies
often rely only on changes that have been observed compared with perfor-
mance at the baseline; it is difficult to measure impacts within the time
frame of the study; and it is nearly impossible to distinguish the impacts of
a QIO from those of its partners and other environmental factors. Also,
the nature of the interventions varies depending on the provider type and
the QIO, as well as from one provider to another, because the intervention
methods used are determined largely by each QIO. The voluntary nature
of provider participation with QIOs introduces the possibility of bias in
the self-selection of participants, which in turn limits the value of compari-
sons of the rates of change for identified participant groups with the rates
for the entire state. Also, because the QIO has the responsibility to raise
the level of quality statewide as well as for identified participants, a large
difference between the two rates of quality improvement may indicate a
particularly successful program for identified participants but a very weak
statewide program. These research limitations and inconclusive findings in
the literature are not limited to studies of the QIO program, but also ham-
per other studies of improvements in health care quality.
In addition to studies related to the QIOs, the literature review per-
formed for this study focused on more generic quality improvement studies
OCR for page 59
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 59
in health care. In addition to 23 studies focused specifically on the QIO
program and various QIO-led quality improvement interventions, the com-
mittee examined 9 articles, including 2 Cochrane reviews (the latter system-
atically assessed a wide range of studies according to explicit, high stan-
dards). (This literature review is discussed further in Chapter 9, and selected
articles are summarized in Tables A.1 and A.2 in Appendix A.) The studies
paint an inconclusive picture of the effectiveness of quality improvement
programs, whether the interventions are conducted by QIOs or other orga-
nizations, for both Medicare and non-Medicare services. In part, this lack
of conclusive data results from the research challenges mentioned above.
Few of the studies used randomized controlled trials or control groups.
An examination of additional literature on the transfer of knowledge
about various quality improvement methods also provided inconclusive
findings on the effectiveness of these methods; no specific method has been
identified as best. Studies do show the importance of commitment on the
part of an organization's leadership, staff empowerment, and the develop-
ment of a plan to promote quality improvement (Bradley et al., 2005; CMS,
2005a). Analysis of the management literature on effecting change to im-
prove patient safety in health care organizations reveals that several human
resource management practices are helpful in that process. In particular, to
promote successful change and the adoption of new practices, it is impor-
tant for management to promote "ongoing communication; training; use of
mechanisms for measurement, feedback, and redesign; sustained attention;
and worker involvement" (IOM, 2004:118).
Knowledge transfer is considered an important part of the QIO pro-
gram, although evidence that it occurs is meager. A significant portion of
the functions of the QIOSCs relates to knowledge transfer among the QIOs:
training the trainer, the establishment and maintenance of communities of
practice, and the identification and promotion of best practices.
During the site visits, telephone interviews, and focus group discussions
conducted by the committee, the chief executive officers (CEOs) of the QIOs
emphasized the need to include early adopters, leaders, and champions, in
addition to middle and late adopters, in quality interventions to promote
their uptake by practitioners and institutions beyond those involved in the
interventions. On the other hand, focusing solely on providers that are lead-
ers and champions and may be relatively easy to recruit to a QIO technical
assistance program could limit the participation of providers that may need
the QIO's assistance the most.
The QIO program considers sharing among QIOs to be important and
has established mechanisms for this purpose. For example, the shift in the
7th SOW from a relative failure rate definition for QIOs that must re-
compete to an absolute target for automatic renewal meant that QIOs were
more willing to share and help each other. They were no longer competing
OCR for page 60
60 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
to avoid the penalty associated with being in the bottom quartile; they were
all trying to meet absolute targets and could cooperate without jeopardy.
The 8th SOW maintains similar incentives.
While the QIO contracts and QIO collaboratives are based on sharing
and knowledge transfer, other forces in health care are pushing providers in
the opposite direction, toward more competition. Under the pressures of
public reporting, no providers will want to see their name on the bottom
half of the quality-of-care list, which would indicate that the care they pro-
vide is of below-average quality. Pay-for-performance schemes are likely to
reinforce that competition, depending on the reward structure. Unintended
consequences of public reporting and pay for performance should be exam-
ined, but no evidence yet exists on what impacts, if any, these programs will
have on competition, and it is unclear how QIO collaboratives will work as
the environment becomes more competitive for providers. In some commu-
nities, providers may welcome the use of public reporting and use it to
benchmark and identify other providers from which they could learn. In
any case, the program should be considering new options for QIO interven-
tions and mechanisms, such as web-based training and self-guided study
tools, to extend the QIOs' reach to more providers.
Because of the limitations of the scientific literature, the committee at-
tempted to use program-generated data, including the performance scores
used by CMS to evaluate each QIO's contract performance, to identify high-
and low-performing QIOs (see Chapter 10). CMS conducts separate per-
formance assessments for each SOW task and subtask. The scores related to
Task 1 (technical assistance for quality improvement) define "quality" for
the program. The scores used by the committee are based on specified clini-
cal measures for hospitals, nursing homes, home health care agencies, and
physician offices during the 7th SOW through December 2004. To retain
its QIO contract during the next SOW without competing, a QIO must
attain a passing score for each task; no total or average score based on all
tasks is assigned to each QIO. The committee's analyses revealed the
following:
· A QIO's score in one provider setting did not correlate with that in
any other setting.
· No correlation existed between spending per beneficiary in a state
on the QIO's technical assistance in a particular setting and the QIO's per-
formance measure for that setting.
· No correlation existed between the state performance score in a par-
ticular setting and measures of the provider's satisfaction with the QIO.
· Some regional variations in the performance scores were noted, as
were variations in contract rounds for the home health care provider scores
(see Chapter 13 for a discussion of contract rounds).
OCR for page 61
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 61
The committee was unable to ascertain why one QIO might have a
higher performance score than another in a particular setting. One might
expect that many skills needed to carry out a quality improvement interven-
tion would be transferable from one provider setting to another. They may
in fact be transferable, but perhaps some QIO staffs are not well organized
to effect that transfer of knowledge, or the provider communities and their
stakeholder organizations may differ in ways that affect the success of the
QIOs' interventions. Also, the role played by providers' voluntary partici-
pation in the impacts of the QIO program is unclear. One study of hospital
and ambulatory care measures showed that absolute improvement on 21 of
the 22 measures tracked was greater for those states starting at a lower-
quality baseline than for those starting at a higher-quality baseline (Jencks
et al., 2003). While many inferences can be drawn about why a trend can-
not be identified to link improvement and setting of care, it must be noted
that some skills and expertise required to provide assistance in each setting
are not necessarily the same. Given limited resources and the need to com-
plete other tasks, such as communications and beneficiary protection, QIOs
must choose how best to allocate their resources.
The committee heard consistent reports of extensive variations among
the QIOs: some are outstanding, while others are mediocre. These judg-
ments appear to reflect an overall perspective on the organizations rather
than the QIOs' performance on specific tasks. Yet there are no objective
global measures, based on contract performance scores for quality improve-
ment, that could indicate which QIOs belong in which category or even
whether there are in fact significant differences in overall performance. For
example, some QIOs score high on improving care in one setting but not in
others. The complex and detailed formulas used by CMS to assess each
state's performance on the core tasks of the SOW can be used to determine
the QIOs' performance only on each task separately. For example, 6 of the
36 QIOs in the first two contract rounds failed on at least one assessment
standard (task or subtask) and had to recompete for an 8th SOW contract.
Those 6 QIOs included some that would not generally be viewed as belong-
ing in the bottom tier on overall performance, some that had received sub-
stantial contracts from CMS to conduct special studies or to serve as a
QIOSC, and some that were named "best" at particular tasks by other
CEOs according to the committee's web-based data collection tool.
ALTERNATIVES TO THE CURRENT QIO PROGRAM
In considering alternatives to the current QIO program, the committee
decided it was necessary to step back and ask some fundamental questions.
First, should the federal government exercise a stronger role in the pro-
vision of technical assistance to health care providers to promote more rapid
OCR for page 62
62 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
improvements in health care quality? The evidence base for the committee's
Performance Measurement report (IOM, 2006) and for the present study
(see the introduction to this report) indicates that the health care received
by Americans is seriously inadequate, and that the health care system needs
to produce better-quality care that meets standards supported by the best
scientific evidence. While these shortcomings have been in the headlines
and discussed extensively in the provider community for many years,
progress toward improved quality continues to be painfully slow. The com-
mittee concludes that the public sector needs to play a substantial role in
improving the quality of care for all Americans. This is especially so for
those who depend upon federal programs, such as Medicare, Medicaid, the
State Children's Health Insurance Program, the Department of Defense's
TRICARE, and the programs of the Veterans Health Administration and
the Indian Health Service, which provide coverage and care to roughly 100
million people (IOM, 2002). The creation of a performance measurement
and reporting system and the implementation of payment systems that re-
ward quality care are important steps in accelerating improvement, but they
are insufficient to achieve the six quality aims outlined by the IOM (IOM,
2000, 2001). The committee believes the federal government is well suited
to promote better-quality care for all Americans because it spends more
than $513 billion annually on their care. The magnitude of this investment
can generate positive changes in quality throughout the health care system
(IOM, 2002). As the IOM concluded previously, "the federal government
must assume a stronger leadership role to address quality concerns" (IOM,
2002:x).
Second, given a role for the federal government in providing technical
assistance for quality improvement, should that assistance be offered at no
cost to providers, as is now the case through the QIOs, or should providers
be expected to pay for the assistance? The following findings can help an-
swer this question:
· Many providers have resources that allow them to purchase techni-
cal assistance from private organizations, such as those discussed in Chap-
ter 3 and listed in Table B.1 in Appendix B. Also, many providers spend
considerable sums to hire quality improvement experts and to conduct im-
provement programs internally. While the extent and distribution of such
expenditures are unknown, they are generally considered to be more com-
mon among hospitals, large physician groups, and managed care organiza-
tions than among medium and small physician practices and other provider
settings. The committee does not wish to discourage such behavior.
· At the same time, particular attention needs to be paid to disparities
in access to and use of technical assistance resources within the provider
community. Some providers are ready and willing to undertake the internal
OCR for page 63
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 63
changes needed to improve quality but lack the resources to purchase the
technical assistance they need to achieve these reforms.
· Continuing to achieve improvement at the current rate would be
costly in lives and dollars and is therefore unacceptable.
· Regardless of the current level of expenditures by providers for tech-
nical assistance to improve quality, the end result is clear--improvements in
quality nationally are progressing too slowly. While many providers may be
willing to expend their own resources to improve quality when payment
becomes based on such improvements, others are likely to be left behind.
Some providers simply do not have sufficient resources to purchase the
necessary services readily. The expansion of pay-for-performance systems
to cover a greater portion of a provider's revenues is likely to worsen this
situation. Providers with low levels of quality according to the given metrics
may receive no increase or perhaps even a decrease in reimbursements.
· In some areas, poor-quality providers may be the only ones available
to a portion of the population, and their failure would further exacerbate
disparities in health care access and quality.
The committee concludes that it is important to improve the level of
care offered by all providers, even if they cannot afford to purchase techni-
cal assistance privately, and that some level of technical assistance should
be available through the federal government as a public good. As providers
improve their care processes for Medicare beneficiaries, it is likely that other
patients will benefit as well.
PROGRAM INFRASTRUCTURE
Given the committee's limited ability to attribute quality improvements
in Medicare directly to the efforts of the QIO program, it is necessary to
consider whether the current program should continue. Although it may
appear obvious, the committee believes the existence of 41 separate organi-
zations holding QIO contracts dedicated to providing quality improvement
services in every state, the District of Columbia, Puerto Rico, and the Virgin
Islands is a significant asset. The cadre of trained experts in QIOs (see Chap-
ter 7) is a potentially valuable resource for offering technical assistance in
quality improvement and for helping hospitals, nursing homes, home health
agencies, and outpatient physician practices collect data on their perfor-
mance and aggregate and analyze those data to improve the care they
deliver.
The QIO program has created an infrastructure across the United States
with staff trained and experienced in various quality improvement tech-
niques. The QIO program serves as a focal point within CMS for assisting
health care service providers in improving the care they offer to Medicare
OCR for page 64
64 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
beneficiaries and others. It is also the only program within CMS that ac-
tively addresses health disparities (Jost, 2005). The following are examples
of the QIOs' expertise and experience drawn from the committee's QIO
and Regional Office site visits, telephone interviews, and web-based data
collection tool:
· All but seven QIO contracts are staffed with at least one employee
who is a Certified Professional in Healthcare Quality (through the Health-
care Certification Board, the National Association for Healthcare Quality,
or a similarly recognized professional accreditation in quality improvement).
Eleven QIOs have from 10 to 34 such certified staff.
· Eighty-eight percent of QIO CEOs believe their leadership staff have
substantial competencies for such functions as collaboration, relationship
building, team development, and performance measurement. Many CEOs
stated that their QIOs have individuals with other leadership skills avail-
able to carry out the tasks of the 8th SOW.
· A national survey of the QIOs by the Best Practices QIOSC (Qualis
Health, 2004) showed that the quality improvement management staff of
many QIOs were familiar at a minimum with various quality improvement
techniques and programs, such as the Baldrige criteria, a collaborative meth-
odology, human factors, International Standards Organization (ISO) 9000
criteria, Lean principles, and the Six Sigma program (see Chapter 7 for a
discussion of QIO staff training and Chapter 9 for a description of each
method). Some QIOs indicated having used some of those tools; only a
small number of QIOs reported having staff certification for specific pro-
grams, such as Six Sigma. Collaboratives were a major method used in the
6th and 7th SOWs, and many QIO staff were trained in its use. This is
reflected in the fact that 98 percent of the QIOs reported being familiar
with the method, and 95 percent of the QIOs reported having used it (per-
sonal communication, J. Kelly, June 29, 2005). Almost half of the respon-
dents were familiar with even the least common methods--those based on
ISO 9000 criteria and Lean principles (Qualis Health, 2004).
· The QIOs had a separate task in the 7th SOW to reduce disparities
between an identified group, such as a particular underserved or rural popu-
lation, and a reference group. In the 8th SOW, the disparities task was
folded into all the quality improvement technical assistance tasks rather
than remaining separate (see Chapter 8).
· All QIOs gained experience in assisting hospitals with reporting mea-
sures during the 7th SOW, when nearly all hospitals decided to participate
voluntarily in public reporting to avoid a payment reduction. QIOs were
also involved with the public rollout of comparative nursing home mea-
sures (see Chapter 11).
OCR for page 65
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 65
Not only does each QIO have technical expertise in quality improve-
ment, data management, and data analysis, but significant expertise in these
areas also exists within CMS at the national and regional levels (see Chap-
ter 13). The Central and Regional Office program staff may appear to be
small (132 full-time equivalents) relative to the size of the program they
oversee, the variety of tasks they perform, and their responsibilities for im-
proving the quality of care for Medicare beneficiaries. Among the staff,
however, are people with advanced training and experience in medicine,
nursing, quality improvement, epidemiology, scientific research, and data
systems (CMS, 2004; personal communication, J. Kelly, September 8, 2005).
Although frustration and complaints with regard to the staff at CMS
were voiced frequently during the committee's site visits to QIOs, telephone
interviews, and other informal meetings, 85 percent of the QIOs rated their
Project Officer "good" or "excellent" on expertise; the Scientific Officers
received similarly high ratings on their performance (see Chapter 13). The
QIO program's administrator reported that the program has sufficient staff
and expertise to conduct the work required under the 8th SOW (personal
communication, W. C. Rollow, July 8, 2005). In addition, the program has
established potentially useful national communications networks and a data
repository that currently serve both the QIOs and the public reporting of
hospital data (CMS, 2004) (see Chapter 13 for further discussion of the
data systems).
The committee notes that at both the state and the national levels, the
QIOs and CMS have established important working relationships with pro-
viders, their professional associations, and various other stakeholder groups
and convened parties around specific issues (CMS, 2004; Westat, 2005;
National Health Policy Forum, 2004). The QIOs and CMS have also been
major participants in such collaborative efforts as the Hospital Quality Al-
liance and the Ambulatory Care Quality Alliance. Both the convening abil-
ity of the QIOs and the clout of CMS to bring national organizations to the
table are key ingredients for promoting widespread, coordinated quality
improvement.
The committee recognizes the expertise available within the QIO pro-
gram and the enthusiasm, commitment, and dedication to quality improve-
ment exhibited by staff and leadership from the Central Office at CMS, the
Regional Offices, and all the organizations holding QIO contracts. The com-
mittee concludes that the potential exists for the QIO program to have a
measurable positive impact on improving the quality of care for Medicare
beneficiaries, serving useful and important functions in the rapidly chang-
ing world of Medicare.
OCR for page 71
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 71
· Broader representation of various disciplines and expertise--The
QIOs have diverse constituents, including hospitals, physician practices,
nursing homes, and home health care agencies. However, the QIO boards
appear to have relatively few members from health professions other than
physicians. The committee found that physicians dominated nearly all the
QIO boards, that two-thirds of all board members listed in the IOM's web-
based data collection tool were physicians, and that all responding QIO
CEOs (39 of 41) listed representatives of office-based practices on their
boards. In particular, nurses (especially those in home health care), physi-
cian assistants, pharmacists, and physical and occupational therapists ap-
pear to be in short supply on QIO boards and are nonexistent on most
boards, even though these types of health care professionals could play
critical roles in improving the quality of care. Boards need to become more
inclusive, encouraging participation from a variety of stakeholders and rel-
evant experts. The growing responsibilities of QIOs during the 7th SOW
included assisting stakeholders with data measurement, collection, aggrega-
tion, analysis, and reporting as part of their provision of technical assis-
tance for quality improvement. The committee expects those functions to
increase in importance, and the minimal representation of QIO board mem-
bers with expertise and experience in information technology management
and oversight is a concern. Board members from outside the health care
sector with experience in quality improvement, process measurement, and
performance accountability could also contribute to the governance of
QIOs. At the same time, when considering the addition of consumers, health
professionals, and others to its board, a QIO must balance the need to keep
the board's membership to a workable number. The creation of alternative
mechanisms, such as advisory committees, may be necessary to obtain the
varied input needed while keeping the board to a manageable size.
· Board member development and assessment--Studies have shown
that organizational leadership can benefit from the implementation of a
systematic plan for individual board member development, an annual as-
sessment of each board member's performance, and an evaluation of the
board as a whole (Orlikoff and Totten, 2005). Such assessments are par-
ticularly important in light of recent scandals involving both for-profit and
not-for-profit health care entities; passage of the Sarbanes-Oxley Act of
2002 (P.L. 107-204), which brings greater accountability and transparency
to the for-profit corporate world; and efforts in the not-for-profit sector by
Board Source, Independent Sector, the Aspen Institute, and the Center for
Healthcare Governance, which have recommended new requirements for
board accountability. Just under a quarter of the 41 organizations holding
contracts under the 7th SOW reported that they had formal mechanisms in
place for evaluating the performance of individual board members; the same
proportion of QIOs evaluate the overall performance of their boards.
OCR for page 72
72 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
· Financial oversight and strategic guidance--Only 23 of the 41 orga-
nizations with QIO contracts have a board finance committee, only 15 re-
ported that they have an audit committee, and only 8 reported that they
have a strategic planning committee. Thus there appears to be a need for
increased attention to these important areas of board responsibility to pro-
vide adequate oversight and strategic direction to QIO management.
· Overall responsibility--In addition to financial oversight and strate-
gic guidance, the QIO board should be responsible for assessing the overall
performance of the CEO and senior staff, as well as the accomplishment of
defined goals and priorities.
Physician-Access or Physician-Sponsored QIOs
Outmoded Requirement
The legislative requirement that QIOs be physician-access or physician-
sponsored organizations was intended to assure physicians that their clini-
cal work would be reviewed by their peers on the basis of local practice
standards. Although certain clinical assessments and case reviews still re-
quire medical expertise, the physicians need not be local because quality of
care is now defined according to evidence-based national guidelines and
standards rather than local patterns of practice. The committee concludes
that this requirement is now outmoded and unnecessarily limits the compe-
tition for QIO contracts from other entities.
Limited Competition
The committee finds that few entities other than out-of-state QIOs have
been serious competitors for QIO core contracts in the past or are expected
to compete for the 8th SOW, given the wide and complex assortment
of required tasks and structural requirements. Other entities that are not
physician-access or physician-sponsored organizations might have the ca-
pacity to perform all or portions of that contract (see Chapter 3 for further
discussion of "other entities"). Yet there has been a history of very limited
competition for QIO contracts, and there have been few opportunities
to replace QIOs that did relatively poorly or that failed on their previous
contract:
· All but two of the QIOs in the 7th SOW held their state's contract
under the 6th SOW.
· Fully 70 percent of the 7th SOW contractors were also contractors
under the 1st SOW.
OCR for page 73
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 73
· No outside entities bid against the previous QIO for three of the
seven state contracts opened for competition for the 7th SOW.
· As of the end of the 7th SOW, five contracts had been recompeted
and one was to be; there were no competitors for three of five contracts
(personal communication, S. Pazinski, November 14, 2005).
Conflicts of Interest
Federal regulations for QIOs are designed to foster impartiality and to
prevent conflicts of interest between members of the QIO board and the
health care providers whose care is the focus of the QIO assessment through
case review (see Chapter 7). These regulations contribute to a perception of
neutrality and respect for the QIO, as well as acceptance of its quality im-
provement and case review tasks, among the provider community and ben-
eficiaries. They also mean that the QIO boards have derived limited benefit
from the involvement of executives from the provider community, such as
hospital CEOs whose hospitals might come under the review of a QIO.
Under the 8th SOW, organizations that hold QIO contracts are not
permitted to contract with providers in their state(s) for technical assistance
or to review services for Medicare beneficiaries that are similar to the assis-
tance or services they provide under the QIO contract. Although some pro-
viders and insurers in some states may wish to purchase additional QIO
services beyond those supported by Medicare funds, the QIO is not allowed
to negotiate such a contract (personal communication, D. Schulke and T.
Ketch, American Health Quality Association, June 30, 2005). This prohibi-
tion appears to be based on a perception that lucrative contracts with the
QIO could influence relations with certain providers and the review of their
cases. The potential for such conflicts appears less likely now, however,
given the decreased emphasis on case review and sanctions since the early
days of the Professional Standards Review Organization program. There is
clearly a market demand for the additional services of some QIOs, and
under its current level of funding, the Medicare QIO program is unlikely to
meet fully the future need for technical assistance with quality improve-
ment. The committee finds that many QIOs have demonstrated a capacity
to develop external revenues (not from CMS) and extensive relations with
providers and provider organizations within their own states. At least 15
QIOs receive more than half of their revenues from sources other than the
core contract. The committee concludes that if the case review functions
were no longer maintained within each QIO, the need for this contracting
prohibition might be eliminated. Reasonable controls, however, would be
necessary to prevent favoritism in the QIOs' selection of recipients of their
technical assistance with Medicare quality improvement.
OCR for page 74
74 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
Confidentiality Restrictions
The attitudes of the early predecessors to the QIOs and physician pres-
sure have generated strict confidentiality restrictions on the QIOs' treat-
ment of clinical data. The committee concludes that these restrictions pro-
tect the privacy of providers and are not well suited to the current need for
transparency, public reporting, and consumer access to information (Gaul,
2005). The restrictions also constrain the use of data on quality measures to
support the quality improvement process. These restrictions were estab-
lished mainly through regulations, and the secretary of DHHS could make
policy changes consistent with Health Insurance Portability and Account-
ability Act (HIPAA) regulations without the need for new legislation (per-
sonal communication, T. Jost, Washington and Lee University School of
Law, December 21, 2004).
Consistent with CMS's concerns regarding the use of data and protec-
tion of providers, CMS required that for this study, the committee use mea-
surement data on the performance of each QIO only in the aggregate for
the nation as a whole, or deidentify the data to protect the QIOs. The IOM
had never requested access to physician or patient data, which would have
required protections. The committee finds that the program's lack of trans-
parency concerning its key contractors is incompatible with the broader
trends within Medicare and the health care system to disclose providers'
quality measures through public reporting.
OVERSIGHT OF THE QIO PROGRAM3
CMS has the challenge of managing the QIO program in the field, as
well as integrating it into the operational responsibilities of the Medicare
program. The main oversight functions required of the CMS Quality Im-
provement Group, which runs the QIO program, include the following:
· Operation of complex data processing and communications systems
for the program
· Management and evaluation of QIO core contracts and contracts
for QIOSCs, special studies, and support contracts
· Strategic planning needed to ensure the continued usefulness of the
QIO program to the Medicare program
Chapter 13 describes the various communications and management mecha-
nisms used by the QIO program and its data systems.
3Most of the information in this section was drawn from the committee's site visits, tele-
phone interviews, and web-based data collection tool.
OCR for page 75
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 75
Data Processing and Communications
Communications among the QIO Central Office, the Regional Offices,
and the QIOs are maintained through e-mails, list serves, restricted and
public websites, and electronic reporting tools. To some extent, the QIOs
are expected to operate in the virtual world, and they are encouraging pro-
viders to move in that direction as well. Training for CMS staff and pro-
grams, including QIO staff, is conducted over the Internet; communities of
practice stay in touch with monthly conference calls; and meetings are video
broadcast across the country to minimize travel. Some of these mechanisms
do not work as well as the QIOs would like: CEOs claim that the frequency
of conference calls makes it impossible to participate every time; that they
do not receive policy memoranda and other necessary guidance promptly;
and that reporting systems, such as those for case review and program ac-
tivities, are difficult to use.
Management communications among CMS Project Officers, QIOs, and
CMS Contract Officers are particularly problematic. A QIO can be caught
between two different interpretations of an issue, and the interpretation of
the Contracts Office has priority. The CMS Project Officer provides routine
guidance to the QIO and often serves as a communications link when policy
questions arise. While the Project Officer may request an activity outside
the scope of the QIO contract and beyond the budget, however, the Con-
tracts Office may subsequently refuse to consider a contract modification.
The Standard Data Processing System supports the communications
tools mentioned above, as well as the flow and processing of data from
medical records. It is key to the QIO program, and 63 percent of the QIOs
rated the value of the system as excellent or good, although some
QIOs mentioned the need to update the system and integrate it with their
own equipment. It will be important for QIO and CMS staff to be closely
involved with national and regional initiatives concerning data exchange,
to understand strategic policy issues related to health information technol-
ogy, and to keep their software and hardware current. The main concern
QIOs expressed is that the data reported through the Dashboard section of
the QIONet internal CMS website and used to monitor the progress of the
QIOs' quality improvement efforts often are not timely (see Chapter 13).
Although there is some conflicting evidence about the value of rapid data
feedback (Beck et al., 2005), the QIOs and providers frequently stressed the
usefulness of real-time data for quality improvement and for the care of
specific patients.
As CMS increases the measures required for public reporting by hospi-
tals and physicians, the challenges to the Standard Data Processing System
will increase. (CMS conducts much of the processing of the data for nurs-
ing home and home health care measures.) The committee concludes that
OCR for page 76
76 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
the growing volumes of data, the expanded numbers of data sources, the
increased need for the timeliness of data processing, and the interest in
user-friendly data reports for different stakeholders will exert pressures on
the current system. At the same time, the transition to electronic health
records may reduce the demand for data abstraction, increase the need for
data-auditing services, and improve the timeliness of the data. Long-range
planning for these changes is essential to smooth operations and should
include an array of people with detailed operational expertise early in the
process. The implementation of a national system for quality measurement
and reporting will have an additional impact on the demand for data pro-
cessing and other activities (IOM, 2006). The committee recognizes, how-
ever, that full implementation of both electronic health records and the
new national public reporting system will take time. Therefore, paper-based
tools that can be used prospectively, as opposed to chart abstraction, will
also be important. With DHHS fully supporting providers' move to the use
of electronic health records, as well as electronic communications, Medi-
care could serve as a guide to the future if its data systems keep pace with
the transition.
QIO Contract Management and Evaluation
CMS's day-to-day management of each QIO contract is a major re-
sponsibility of staff in the Regional Offices, who perform detailed on-site
assessments at 9 and 18 months into the contract period and in the interim
keep in touch with the QIOs to which they are assigned. However, most of
a QIO's performance rating for contract renewal is based on data on pro-
viders' performance on quality measures and complex formulas, with sepa-
rate calculations for each task. The increased complexity of tasks in the 8th
SOW led to a significantly more complicated set of formulas and incentive
awards than those in the 7th SOW (see Chapter 10). The absence of an
overall evaluation plan, guidance, and program priorities has created a situ-
ation in which each of CMS's Government Task Leaders designs the evalu-
ation formulas for his or her own tasks in the SOW. The formulas do not
draw upon consistent time frames, goals, or definitions. The resulting com-
plexity makes it difficult to construct a coherent overall assessment of a
QIO. It is unclear, therefore, whether these formulas adequately assess "im-
provement" and "quality," and the committee finds that the program's sys-
tem for evaluating the QIOs requires further development to enhance its
effectiveness. As recommended in a 1994 IOM letter report, staff with ex-
pertise and experience in program evaluation, as well as outside assistance,
are needed to design formative and summative evaluations of the QIOs and
to contribute to the design of contract performance reviews (IOM, 1994).
The complex method for evaluating the contract performance of QIOs
OCR for page 77
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 77
under their core contract is in sharp contrast to the limited, relatively simple
methods used in the 7th SOW to assess the performance of the QIOSCs,
QIOs conducting special studies, and organizations carrying out support
contracts. Although these contracts are assessed for timeliness and com-
pleteness of deliverables and spending, CMS relies primarily on the judg-
ments of the Project Officers. It may be noted that contracts for QIOSCs,
special studies, and program support accounted for almost one-third of the
total apportionment for the 7th SOW. The value of these expenditures is
diminished by the lack of a comprehensive listing of these various contracts
and summaries of their contents and results, which are thus not diffused
throughout the quality improvement community. Although many of the
support contracts are related tangentially to the QIO program, QIOs may
be unaware of those contracts or of a special study that is under way and
would be of interest to them. CMS is in the process of developing a manage-
ment and accountability plan to better administer all of its contracts. Few
details were available as of this writing, however, and implementation of
the plan remains to be seen.
One factor that may have contributed to the previously discussed lim-
ited competition from other entities for the 8th SOW was the timing of
submission of proposals for the core contract. The request for proposal was
released on April 1, 2005, and the deadline for submission of proposals was
April 15, 2005, before the SOW was final and the full amount of funds
available was known. An official version of the 8th SOW was issued on
May 20, 2005, but some elements were still under discussion within DHHS
and the Office of Management and Budget that could have required future
and unknown modifications to the SOW. In the first contract round, the
QIOs were required to submit their bids before the release of the final SOW,
and the bids had to be based on a draft SOW and spending estimates. Ma-
jor revisions to the 8th SOW were made on November 4, 2005, well past
the beginning of the new contract period. Because late revisions can affect
QIO staffing and work schedules, QIOs reported that the timing has been
frustrating and has hampered their ability to prepare for the start of the
new contract (personal communication, D. Schulke and T. Ketch, Ameri-
can Health Quality Association, June 30, 2005). Additionally, delayed re-
leases may deter organizations with no previous experience in holding a
QIO contract from bidding because of the inadequate amount of time
to prepare.
During the committee's site visits and telephone interviews, several
QIOs indicated that the 3-year length of the contract is also a problem.
They believe this period of time is insufficient to obtain data describing the
initial impact of their quality improvement projects and to allow them to
adjust their interventions and later be remeasured for their contract perfor-
mance evaluations. Given the experience with the preparation for the 8th
OCR for page 78
78 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
SOW, as well as the increased expectations of this SOW, it appears that
CMS would also benefit from a longer contract period.
Strategic Planning
Two positive signs concerning policy planning for the QIO program
suggest to the committee that CMS is moving in the right direction. First,
the Quality Improvement Group has initiated a long-range planning pro-
cess to better position the program 12 years into the future. Although some
strategic planning discussions have recognized the importance of coordi-
nated care and crosscutting issues in improving the overall safety and qual-
ity of care, representatives of hospitals, home health care agencies, physi-
cian practices, and nursing homes have not had opportunities to meet to
discuss appropriate measures and mechanisms for tracking a patient being
released from one facility and placed in another. Much of the internal stra-
tegic planning process has focused on individual tasks and separate Gov-
ernment Task Leaders; the external planning process with stakeholders has
followed the same pattern of separating each of the four main health care
settings. The committee concludes that implementation of the new Part D
prescription drug benefit and therapeutic pharmacy strategies offers impor-
tant opportunities for QIOs to focus on the coordination of care across
provider settings. Strategic planning separately for different providers and
care settings is inconsistent with future measures of quality and perfor-
mance that will be encouraged by the national performance measurement
and reporting system recommended by this committee (IOM, 2006).
Another positive sign concerning strategic planning is the rejuvenation
of the CMS Administrator's Quality Council and its staffing by the Quality
Team. This management structure could give the QIO program an opportu-
nity to integrate its activities with those of other CMS functions, such as
public reporting, in the early planning stages. The council has adopted a
vision of the QIO program--"the right care for every person every time"--
and has made it the vision of the whole agency. The Quality Council is
coordinating the development of a number of quality-related projects and
presented a Quality Improvement Roadmap to guide the agency's ongoing
quality-related work. The current roadmap is more a listing of projects and
strategy options than a guide with priorities. Nonetheless, the QIOs are
identified as having a major role in one of five major strategies: "assisting
practitioners and providers in making care more effective, particularly in-
cluding the use of effective electronic health systems" (CMS, 2005b:13).
Given the commitment throughout CMS to improving the safety and qual-
ity of care nationally, the QIO program could play an important part in
carrying out this strategy.
OCR for page 79
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 79
The committee concludes that thorough evaluations of the impact of
the QIO program as a whole and of the effectiveness of specific quality
improvement interventions, based on the articulation of clear and well-
defined goals and priorities, would provide useful input to the long-range
strategic planning process. Evaluations of QIO efforts in physician offices
will be particularly important since the QIOs have had relatively little expe-
rience in that provider setting, and there is little evidence of what approaches
are most successful in stimulating major system changes.
SUMMARY
While there is evidence that the quality of care for Medicare beneficia-
ries has been improving, the evidence concerning specific quality improve-
ment efforts, the overall impact of individual QIOs, and the impact of the
QIO program in the aggregate is limited and inconclusive. Nevertheless, the
committee considers the QIO program to be a potentially rich resource as
the health care system moves toward the increased use of performance mea-
surement and pay for performance. The QIO program has shown consider-
able adaptability over its 35-year history. It currently has a strong founda-
tion of quality improvement experts in each state; a network of collaborative
relationships with providers and stakeholders; and a national infrastructure
to support data collection, reporting, aggregation, and auditing, as well as
research to further the development and the use of quality measures. To
make more effective use of these valuable resources, however, a major re-
structuring of the program is needed. In Chapters 4 and 5, the committee
presents its recommendations and rationales for these changes. First, how-
ever, Chapter 3 presents a discussion of the changing health care environ-
ment and new functions that will affect QIOs and other organizations con-
ducting similar activities. It reviews the committee's first report and how
implementation of that report's recommendations concerning the establish-
ment of a national system for performance measurement and reporting will
likely affect the QIO program. The chapter also examines other entities that
might compete to perform the needed functions.
REFERENCES
Beck C, Richard H, Tu J, Pilote L. 2005. Administrative data feedback for effective cardiac
treatment: AFFECT, a cluster randomized trial. Journal of the American Medical Asso-
ciation 294(3):309317.
Bradley EH, Carlson MDA, Gallo WT, Scinto J, Campbell MK, Krumholz HM. 2005. From
adversary to partner: Have quality improvement organizations made the transition?
Health Services Research 40(2):459476.
OCR for page 80
80 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
Chu LA, Bratzler DW, Lewis RJ, Murray C, Moore L, Shook C, Weingarten SR. 2003. Im-
proving the quality of care for patients with pneumonia in very small hospitals. Archives
of Internal Medicine 163(3):326332.
CMS (Centers for Medicare and Medicaid Services). 2002. 7th Statement of Work (SOW).
[Online]. Available: http://www.cms.hhs.gov/qio [accessed April 9, 2005].
CMS. 2004. The Quality Improvement Organization Program: CMS Briefing for IOM Staff.
[Online]. Available: http://www.medqic.org/dcs/ContentServer?cid=1105558772835&
pagename=Medqic%2FMQGeneralPage%2FGeneralPageTemplate&c=MQGeneral
Page [accessed December 26, 2005].
CMS. 2005a. Best Practice Methods Special Study: Report of First Year Scan of QIO Inpatient
Practice. Unpublished. Baltimore, MD: Centers for Medicare and Medicaid Services.
CMS. 2005b. Quality Improvement Roadmap. [Online]. Available: http://www.
medicaldevices.org/public/issues/documents/CMSMedicareroadmap.pdf [accessed De-
cember 26, 2005].
Daniel DM, Norman J, Davis C, Lee H, Hindmarsh MF, McCulloch DK, Wagner EH,
Sugarman JR. 2004. A state-level application of the chronic illness breakthrough series:
Results from two collaboratives on diabetes in Washington state. Joint Commission Jour-
nal on Quality and Safety 30(2):6979.
Dellinger EP, Hausmann SM, Bratzler DW, Johnson RM, Daniel DM, Bunt KM, Baumgardner
GA, Sugarman JR. 2005. Hospitals collaborate to decrease surgical site infections. The
American Journal of Surgery 190(1):915.
Gaul GM. 2005, July 26. Once health regulators, now partners. Washington Post. p. A1.
Gould BE, Grey MR, Huntington CG, Gruman C, Rosen JH, Storey E, Abrahamson L, Conaty
AM, Curry L, Ferreira M, Harrington KL, Paturzo D, Van Hoof TJ. 2002. Improving
patient care outcomes by teaching quality improvement to medical students in commu-
nity-based practices. Academic Medicine 77(10):10111018.
IOM (Institute of Medicine). 1994. An Assessment of the HCFA Evaluation Plan for the
Medicare Peer Review Organization. Washington, DC: National Academy Press.
IOM. 2000. To Err Is Human: Building a Safer Health System. Kohn LT, Corrigan JM,
Donaldson MS, eds. Washington, DC: National Academy Press.
IOM. 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Wash-
ington, DC: National Academy Press.
IOM. 2002. Leadership by Example: Coordinating Government Roles in Improving Health
Care Quality. Corrigan JM, Eden J, Smith BM, eds. Washington, DC: National Academy
Press.
IOM. 2004. Keeping Patients Safe: Transforming the Work Environment of Nurses. Page A,
eds. Washington, DC: The National Academies Press.
IOM. 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The
National Academies Press.
Jencks SF, Huff ED, Cuerdon T. 2003. Change in the quality of care delivered to Medicare
beneficiaries, 19981999 to 20002001. Journal of the American Medical Association
289(3):305312.
Jha AK, Li Z, Orav EJ, Epstein AM. 2005. Care in U.S. hospitals--the hospital quality alliance
program. New England Journal of Medicine 353(3):265274.
Jost TS. 2005. Racial and Ethnic Disparities in Medicare: What the Department of Health and
Human Services and the Centers for Medicare and Medicaid Services Can, and Should
Do. National Academy of Social Insurance Study Panel on Sharpening Medicare's Tools
to Reduce Racial and Ethnic Disparities. Washington, DC: National Academy of Social
Insurance.
Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. 2001. Improving
quality improvement using achievable benchmarks for physician feedback: A randomized
controlled trial. Journal of the American Medical Association 285(22):28712879.
OCR for page 81
ASSESSMENT OF THE QIO PROGRAM: FINDINGS AND CONCLUSIONS 81
Leatherman S, McCarthy, D. 2005. Quality of Health Care for Medicare Beneficiaries: A
Chartbook. New York: The Commonwealth Fund.
Marciniak TA, Ellerbeck EF, Radford MJ, Kresowik TF, Gold JA, Krumholz HM, Kiefe CI,
Allman RM, Vogel RA, Jencks SF. 1998. Improving the quality of care for Medicare
patients with acute myocardial infarction: Results from the Cooperative Cardiovascular
Project. Journal of the American Medical Association 279(17):13511357.
National Health Policy Forum. 2004. Quality Improvement in Maryland: Partnerships and
Progress. Washington, DC: National Health Policy Forum.
NCQA (National Committee for Quality Assurance). 2005. The State of Health Care Quality
Report: Industry Trends and Analysis. Washington, DC: National Committee for Qual-
ity Assurance.
NORC (National Organization for Research at the University of Chicago). 2004. Final Re-
port: Physician Meetings on Take-Up of Electronic Health Records. Unpublished. Wash-
ington, DC: National Organization for Research at the University of Chicago.
Orlikoff JE, Totten MK. 2005. The pros and cons of board compensation. Does the promise
justify the means? Healthcare Executive 20(1):4648.
Qualis Health. 2004. HCQIP Improvement Methodologies Survey Results. Seattle, WA: Qualis
Health.
Rollow W. 2005. QIO Program: Update and Policy Considerations. PowerPoint Presentation
to the Committee on Redesigning Health Insurance, June 13, Washington, DC.
Westat. 2005. Survey for Provider Satisfaction with Quality Improvement Organizations.
Unpublished. July 1. Rockville, MD: Westat.
Representative terms from entire chapter:
qio program
