2
Description of the Vaccine Safety Datalink
ROLE OF FDA AND CDC IN ASSESSING VACCINE SAFETY
Vaccine Development and Licensure
The Food and Drug Administration (FDA) is responsible for monitoring the safety of candidate vaccines from preclinical studies through prelicensure clinical trials. When a biologics license application (BLA) is submitted for a specific vaccine, FDA, with advice from the Vaccines and Related Biological Products Advisory Committee (VRBPAC), makes the decision of whether to license the product and its manufacturing establishment. FDA also plays a critical role in the postlicensure surveillance of the safety of the vaccine after approval for its administration to the general public (Ball et al., 2004; Baylor et al., 2004).
When sponsors believe that they have sufficient data to initiate the first Phase 1 clinical trial of the vaccine in humans, they submit an investigational new drug (IND) application to FDA’s Center for Biologics Evaluation and Research (CBER). FDA has 30 days to review the IND, which includes information on the rationale for the vaccine, the methods of manufacture, characterization, and the results of relevant preclinical tests of the vaccine’s safety, immunogenicity, and (if possible) efficacy in animal models and in vitro models. An important part of the IND is the clinical protocol, which must undergo an ethics review and approval by an Institutional Review Board (IRB). As the vaccine candidate advances stepwise through the clinical-development path of Phase 2 and Phase 3 trials, FDA, through the IND process, monitors its safety profile in con-
junction with the IRB(s) that initially approved the trial and (for larger studies) a study-specific data safety monitoring board associated with the trial. If questions or concerns of safety arise during a clinical trial, FDA can put the study on “clinical hold” until the questions are satisfactorily addressed.
In parallel with the clinical-development path, the manufacturers develop processes to move from preparation of pilot vaccine lots to large-scale manufacture of lots that are consistent in characterization and immunogenicity. The large-scale manufacturing process is generally in final form before Phase 3 trials begin. Defined manufacturing methods and well-described tests to control the vaccine at critical steps in the manufacturing process are fundamental to ensure the safety and purity of vaccines and to achieve consistency in manufacture. FDA regulations related to the manufacture, product quality, and clinical testing of vaccines are found in Title 21 of the Code of Federal Regulations (CFR).1 To supplement information contained in the CFR, FDA periodically makes available guidance documents that address various aspects of and issues related to vaccine safety.2
If the results of the Phase 1, 2, and 3 clinical trials support the safety, immunogenicity, and efficacy of the vaccine, the manufacturing facility is adequate, and the product of manufacture is consistent, the sponsor can submit a BLA to FDA. After consideration of the data, FDA, with advice from VRBPAC, can license the vaccine. VRBPAC includes public members and non-FDA scientists, clinicians, biostatisticians, and epidemiologists (FDA, 2002).
Postmarket Surveillance of Vaccine Safety
FDA continues to play an important role after licensure by making periodic inspections of the manufacturing facility. It can request results of
the manufacturer’s tests for controlling the quality (purity, potency, and safety) of different lots of the vaccine, and it can request that samples be submitted for testing by CBER (FDA, 2002).
One mechanism for continued FDA surveillance of safety is active and passive Phase 4 studies (including studies sometimes agreed to by the sponsor as a condition of FDA licensure). In addition, FDA and the Centers for Disease Control and Prevention (CDC) jointly manage the Vaccine Adverse Events Reporting System (VAERS), a passive pharmacovigilance (that is, vaccinovigilance) system that receives 10,000-15,000 reports of adverse events a year on standardized reporting forms (CDC and FDA, 2003). About 15% of VAERS reports reflect serious adverse events involving life-threatening conditions, hospitalization, permanent disability, or death (CDC and FDA, 2005). VAERS relies on ad hoc reporting from patients (or parents of pediatric patients) and from health care providers. The adverse events reported may or may not be causally associated with the immunization.
VAERS is a valuable tool for detecting signals of potential vaccine adverse reactions, but it cannot be used to make definitive vaccine safety determinations, because reporting is voluntary and thus subject to under-reporting and bias. Because the design of VAERS does not allow the definition of a study population (that is, it lacks “denominator data”), adverse event rates cannot be calculated with any precision (CDC and FDA, 2004). VAERS cannot be used for statistically valid analyses of direct links between vaccinations and health outcomes. The U.S. system of medical recordkeeping also does not allow direct links between vaccination and health outcomes, because medical record data are not maintained electronically in a standard format. The Vaccine Safety Datalink (VSD) was created and designed to overcome many of those and other limitations of the data available in 1991.
Other CDC Vaccine Safety Activities
In addition to its joint oversight of VAERS with FDA, CDC plays a critical role in vaccine safety by maintaining the VSD and by performing case-control and other epidemiologic studies in the VSD population and in various other U.S. populations when epidemiologic signals suggest that there may be a problem. CDC has also been a pioneer in helping to develop the (informal) subspecialty of immunization safety by funding seven Clinical Immunization Safety Assessment (CISA) centers throughout the United States. One task of the CISA centers is the standardized assessment of persons who experienced acute vaccine adverse reactions (such as anaphylaxis) to enhance knowledge of the biologic basis of and host risk factors for rare but severe reactions (Pless et al., 2004). CDC also
is responsible for monitoring the incidence of vaccine-preventable diseases at the national level (Chen, 2004).
DEVELOPMENT OF THE VACCINE SAFETY DATALINK
In 1991, the Institute of Medicine (IOM) Committee to Review the Adverse Consequences of Pertussis and Rubella Vaccines recognized that there were serious gaps and limitations in the current knowledge of and research capacity for assessing vaccine safety (IOM, 1991). That committee concluded that the infrastructure for vaccine safety surveillance had weaknesses and that research capacity must be improved to facilitate reviews of vaccine safety (IOM, 1991). To address some of those concerns, the National Immunization Program (NIP) promptly moved to support the creation of the VSD, a collaboration with a consortium of several managed care organizations (MCOs) to allow timely investigations of vaccine safety concerns and to allow retrospective vaccine safety studies (Davis, 2004).
The VSD is a unique national resource for evaluating vaccine safety. It began with four MCOs and now includes data on more than 7 million members of eight MCOs (Chen et al., 1997; Davis, 2004), about 3.2% of the U.S. population under 18 years old, and 1.7% of the population 18 years old and older (CDC, 2004d). The data included in the VSD are compiled by computer from the participating MCOs’ routine administration of health services; there is no chart review or other verification that the data meet reasonable standards for health research. For example, clinical detail is sparse, and care received outside the MCO is not noted. The VSD database links data on patient characteristics, health outcomes (according to data resulting from inpatient, outpatient, and emergency-room records), and vaccination history (vaccine type, date of vaccination, manufacturer, lot number, and injection site) (Davis, 2004). The VSD can be a valuable tool for the retrospective assessment of vaccine safety because the number of people included is large, they generally receive most of their health services at the MCOs, and demographic, health outcome, and vaccination data are maintained electronically. The VSD includes “denominator” data for the study population (in contrast with the data contained in VAERS), so event rates can be calculated.
The NIP contributes substantial resources to the support of the VSD database, to targeted studies that use VSD data, and to support of the VSD data sharing program. In fiscal year 2004, the NIP supported VSD-related activities with about $13 million and about 8.5 full-time-equivalents personnel within the NIP (CDC, 2004d).
Changes in the VSD Contract Provisions
The VSD was established originally by contracts between CDC and four MCOs. The VSD has been expanded to eight MCOs, which the NIP currently supports through a single comprehensive contract with America’s Health Insurance Plans. The contract supports the establishment and maintenance of an infrastructure across the MCOs participating in the VSD that allows scientifically rigorous and efficient monitoring of vaccine safety; creation and compilation of combined electronic files from each of the participating MCOs that link vaccination data, medical outcomes, and other relevant data; and evaluation of selected vaccine safety questions by analysis of the combined data provided by the participating MCOs (CDC, 2004d).
The current VSD contract began in September 2002 and has a performance period of 10 years (CDC, 2004d). The new contract provisions, which introduce several major changes, apply to data from the year 2001 and later. The considerations that prompted the change in contract provisions are unclear. Before September 2002, the automated data files that contained VSD data before 2001 were contract deliverables from the MCOs (CDC, 2004d). Those data files were maintained at CDC and considered a database owned by CDC. With the contract that was renegotiated in 2002, ownership of VSD data generated after December 31, 2000, remains with the MCOs (CDC, 2004d).
COMPLEXITY AND LIMITATIONS OF THE VACCINE SAFETY DATALINK DATABASE
The VSD database is updated annually. The participating MCOs each extract relevant data from multiple administrative databases and merge them to create a generally consolidated picture of the members’ vaccination and health histories. The completeness and accuracy of the separate administrative databases—which contain vaccination, outpatient, inpatient, emergency-room, pharmacy, and enrollment-status data at each MCO—vary with the type of data, calendar time, and the MCO. Although VSD investigators have worked to standardize data elements across sites, a working knowledge of the complexities of the VSD database is required to analyze the data properly. Issues of defining people “at risk” for vaccination or health events is especially important in the creation of valid scientific data, and the accuracy of VSD analyses requires careful tracking of the enrollment status of each person on the basis of enrollment files. A strong working knowledge of statistics and epidemiology is critical to ensure correct use of stratification, modeling, adjustment (such as for age), and other means to control the effects of confounders and biases. Further-
more, the size of the database and the multiple factors related to vaccination and confounding health outcomes suggest that skilled statistical programming is needed for this resource to be used appropriately.
The limitations of the administrative datasets for data on vaccinations and health outcomes have led NIP-affiliated and MCO-affiliated VSD investigators to conclude that the VSD database is useful primarily to identify and set priorities among subjects that warrant targeted case-control studies. The case-control studies are intended to provide more reliable data because to conduct the study, charts for individual enrollees are reviewed for completeness, consistency, and accuracy. In the case-control studies, variable definitions and coding conventions are standardized, and problems of missing data or misclassification with respect to exposure and outcomes are substantially reduced through one or more methods of data collection (such as on-site medical-record review, patient interviews or examinations, and surveys). The datasets used in the case-control studies have considerably improved validity and reliability. The data are compiled only with substantial effort at the MCOs and only with access to highly confidential personal information. It is the smaller case-control datasets from published studies that are in principle available to outside researchers for reanalysis studies done after 2000. The chart-reviewed data are not available to external researchers for new studies through the VSD data sharing program. These are datasets that arose specifically from retrospective studies, however, so many assumptions inherent in the study designs cannot be examined without repeating the data collection.
Ability to Do Surveillance with the VSD
The VSD has been described as a tool for timely surveillance of vaccine safety concerns (Davis, 2004). However, the manner in which the VSD data files are constructed restricts its utility for surveillance. The committee finds that the VSD should not be characterized as a resource for active surveillance of vaccine safety matters, because data files from each MCO are added to the VSD only each year (CDC, 2004d), thereby limiting the capacity for active, timely surveillance of vaccine safety matters. The VSD has many advantages over other databases (such as comprehensive data on study and control populations), but support of surveillance, as normally conceived in public health practice and research, is not one of its identifying characteristics. The VSD is a robust database for large retrospective studies so it is a valuable resource for a variety of studies.
Proprietary Concerns About VSD Data
A unique aspect of the VSD compared with other databases supported by CDC is that it is constituted from the administrative databases of
several MCOs, privately run organizations operating in a competitive field of health care that may not want their competitors to know the details of their subscriber base or business practices. Most other databases supported by CDC are from CDC-sponsored surveys. One consequence of the use of MCO administrative data is the need for protection of proprietary information. MCOs may want to protect different types of information. The current VSD contract includes provisions to protect proprietary interests in relation to the VSD data sharing program (CDC, 2004e).
The MCOs, the NIP, and the National Center for Health Statistics (NCHS) have acknowledged that MCOs’ proprietary concerns affected the design of the VSD data sharing program (Bernier, 2004a; Davis, 2004). The committee was not charged with examining whether proprietary concerns justify some of the limitations but it finds that it is important to recognize the effect of the limitations on the ability of the VSD data sharing program to share data with external researchers.
The MCOs, the NIP, and NCHS sometimes cited the confidentiality of proprietary information in their descriptions of the VSD data sharing program. Throughout this report, when the committee discusses confidentiality, it refers to the confidentiality of individually identifiable information in the VSD; despite its importance, the protection of proprietary information is not implied by the committee’s use of confidentiality in this report.
THE SHELBY AMENDMENT AND THE INFORMATION QUALITY ACT
Whenever there are questions about public access to and the quality of data collected or supported by the federal government, the applicability of the Shelby Amendment and the Information Quality Act (IQA) must be considered. The Shelby Amendment and the IQA are often viewed as compatible and mutually enforcing in that both promote public access to government information (Copeland and Simpson, 2004), the Shelby Amendment focusing on issues of access and the IQA focusing on issues of quality.
The Shelby Amendment, enacted in October 1998 as part of the Treasury and Postal Section of the Omnibus Consolidated and Emergency Supplemental Appropriations Act for fiscal year 1999 (Pub. L. No. 105-277 [1998]), directed the Office of Management and Budget (OMB) to amend its Circular A-110 to ensure that all data produced under a federal award be made available to the public through the procedures established under the Freedom of Information Act (FOIA) 5 U.S.C. § 552 (GAO, 2004). The revised Circular A-110 says that a FOIA request can be submitted “for research data relating to published research findings produced un-
der an award that were used by the Federal Government in developing an agency action that has the force and effect of law” (OMB, 1999). In that context, research data are defined as “the recorded factual material commonly accepted in the scientific community as necessary to validate research findings, but not any of the following: preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, or communications with colleagues”; findings are considered published if they “are published in a peer-reviewed scientific or technical journal” or if a federal agency “publicly and officially cites the research findings in support of an agency action that has the force and effect of law” (OMB, 1999). The committee recognizes that determining the vaccine safety actions that have the “force and effect of law” (if any) could have implications for access to VSD data if the Shelby Amendment is found to be applicable.
The IQA, enacted in December 2000 as Section 515 of the Treasury and General Government Appropriations Act for fiscal year 2001 (Pub. L. No. 106-554 [2000]), required OMB to issue guidelines that “provide policy and procedural guidance to Federal agencies for ensuring the quality, objectivity, utility, and integrity of information (including statistical information)” disseminated to the public by federal agencies (Information Quality Act 44 U.S.C. § 35904(d)(1) [2001]; Information Quality Act 44 U.S.C. § 3516 [2001]). The IQA also required federal agencies to develop their own information quality guidelines and to establish administrative procedures to allow people to seek correction of information that does not comply with the OMB guidance. The committee recognizes that the IQA could have implications for the ability of members of the public to dispute the quality of VSD studies if the IQA is found to be applicable to such studies.
The committee recognized that the Shelby Amendment and the IQA could have important implications for access to VSD data and preliminary findings from the VSD. The applicability of those laws to the VSD should be explored further. If the Shelby Amendment and the IQA are found to be applicable, they could affect the procedures that are used by external researchers to gain access to VSD data and by members of the public to question the quality of VSD studies.
Recommendation 2.1: The committee recommends that the NIP and NCHS seek legal advice to clarify the applicability of the Shelby Amendment and the Information Quality Act to VSD data and VSD preliminary findings.