Executive Summary
The goal of improving health through the use of new medicines and medical devices cannot be achieved without public confidence and participation in the clinical trials process. During clinical development, investigational therapies are tested with human subjects, yielding essential information for assessments of their safety and effectiveness. The importance of informed consent and human subject protection to the integrity of the clinical trials process has been widely discussed over the last 50 years. It is well accepted that human subjects should not be needlessly exposed to risks in trials that fail to yield valid data. Assuring the quality and therefore the usefulness of the data from human clinical trials, however, has received less attention, but it is also a crucial topic because public confidence in the value of clinical trials is ensured when the public knows that the data are of high quality and useful. In addition, regulatory and medical decisions about approval or use of new or novel therapies are dependent upon the reliability of data from clinical trials. For these reasons, there is widespread agreement that data from such trials should be of high quality.
Despite this fundamental agreement, many challenges remain. The current processes for assuring data quality were developed individually in response to various problems or crises rather than in a comprehensive quality management framework. Although the current system is successful, it is relatively expensive and time-intensive, may limit the overall investment in clinical trials, and may not provide the best-attainable quality for the degree of investment. Additionally, there is no consensus definition of ''quality'' as it applies to data from clinical trials. Finally, many changes that have the potential to affect data quality are occurring in the areas of clinical practice and clinical trials, including widespread computerization of data entry and handling, use of contract research organizations (CROs) to perform or organize clinical trials, the increased frequency of multinational trials, and changes in the health care delivery system.
The Roundtable on Research and Development of Drugs, Biologics, and Medical Devices convened a workshop on April 14 and 15, 1998, to discuss these and related topics. Representatives of major parties with vested interests in new medical product development were included. The workshop was successful in broadening the dialogue among the Food and Drug Administration (FDA), industry, and the public on the subject of data quality and validity in clinical trials for the regulatory decision-making process. The workshop participants described the components of the current system, and debated challenges and opportunities for improvement.
Assuring Clinical Trial Data Validity: The Current Process
Data quality efforts begin during the planning stages of a clinical trial and continue throughout FDA review of a marketing application. Product developers are responsible for planning and conducting trials, assembling and analyzing data, and preparing accurate regulatory submissions. FDA recently published the Guidance for Industry on Good Clinical Practices. This document, which represents a harmonized position among the regulatory bodies of the United States, the European Union, and Japan, provides advice on record keeping and procedures for many aspects of the conduct of clinical trials. After receiving an application, FDA evaluates the quality of the data that have been submitted, and also performs audits at clinical trial sites to further assess data quality. The following sections describe the major components of these processes.
Design of Protocol, Case Report Forms, and Data Collection Systems
The industry, government sponsor, or a CRO prepares the clinical protocol, including the forms used for collection of clinical data (often with input from FDA). The organization also develops computer systems for creation of a database and analysis of the information. The protocol and the case report form design, particularly the complexity of the design and the amount of data collected, have important influences on data quality. Some companies are now using remote data entry systems, whereby some or all trial data are entered directly into a computer or a centralized database. The designs of such systems present challenges to regulators.
Clinical Investigator and Study Personnel Training
The sponsor or a CRO trains physicians who will be conducting the study and trains study coordinators and other personnel who organize the study at each
site. This training is critical to ensuring that the protocol is followed correctly and that case report forms are properly completed.
Clinical Site Monitoring
The sponsor or a CRO periodically sends trained personnel (monitors) to each study site to check on the progress and quality of performance of the study. Monitors review case report forms and other study records to ensure that the documentation is complete. The pharmaceutical industry estimates that the monitoring of drug trials can consume 15 to 30 percent of overall trial costs.
Industry Data Quality Assurance Procedures
This step involves assembly of all the data from the trial, entry of the information into databases, and evaluation of the data for quality. If case report forms are paper-based, double data entry into the database is usually performed to minimize transcription errors. The data are then subjected to extensive quality assurance procedures involving follow-up activities on missing or potentially inaccurate datum points. Often, audits of clinical sites will be performed by the sponsor or a CRO as part of the industry quality assurance program.
FDA Data Analysis
FDA clinical reviewers and statisticians evaluate the data submitted in the application. FDA data analysis often includes checking and verification of data from important analyses submitted by the sponsor, as well as performance of exploratory analyses to answer questions that emerge from the review.
FDA Data Quality Assurance Evaluation
FDA clinical reviewers evaluate the quality of the data in the application using techniques such as auditing of case report forms to verify the accuracy of tabulated data, evaluation of follow-ups on reported adverse events, and verification of the primary outcome measure at the case report form level. An overall assessment of data quality is developed. The overall assessment is a factor in determining whether the application merits approval. If serious questions regarding overall data integrity are not resolved, FDA will not approve the application.
FDA Clinical Study Audit Program
During application review, FDA selects clinical sites that generated data for the application for auditing purposes. A thorough on-site review of these sites is conducted by trained FDA inspectors. Record keeping, adherence to the protocol, informed-consent procedures, and other aspects of the study are assessed. If objectionable conditions are found, a report (referred to as FDA Form 483) is provided to the principal investigator at the conclusion of the audit.
FDA Enforcement Activities
If an investigator is found to have serious or repeated problems in performing clinical studies, FDA will take steps to debar the individual from performing trials for regulatory purposes. In cases of fraud, criminal prosecution may be pursued.
Opportunities for Improving the Process
During the workshop participants identified a number of opportunities for significant improvements in the overall process. Some of the goals included the need for an increase in the dialogue among the involved parties, maintenance or improvement of quality with a concomitant lowering of costs, increases in the levels of public knowledge and confidence in the process, a means of dealing with emerging changes such as widespread computerization and delivery of new therapies to the market, and protection of the public from risk. Overall, participants identified a need for systemic collaborative improvements, such as improving processes in the context of the entire system with the input of all parties involved in the system. The following sections describe some of the targeted and overall strategies for improving the process.
Targeted Strategies
Broader Involvement of Patient and Consumer Groups
- Representatives of patient and consumer groups indicated that extensive involvement of their constituencies was essential to ensure that their needs and concerns are accounted for and adequately addressed. The constituencies' confidence and participation in the clinical trials process is essential to successful product development.
- Additionally, particular concerns of patient representatives included issues of conflict of interest and investigator bias. Reimbursement of investigators for recruitment and retention of trial subjects was an issue; this practice could be
- a disincentive to maintaining high quality in clinical trials. The interrelated aspects of financial reimbursement and incentives, as well as scientific conduct in clinical trials, deserve further discussion.
Greater Emphasis on Building Quality into the Process
In the current system, both industry and FDA expend considerable effort auditing and correcting data once the data are collected. Many participants suggested that higher quality could be achieved by designing quality during the planning stages of clinical programs.
Overall Strategies
The following sections describe some of the opportunities for improving the overall process of assuring data quality and validity.
More Extensive Training of Clinical Investigators and Study Personnel
Since many quality problems result from failure to follow study designs, protocols, or collect data properly, development of a larger cadre of well-trained and experienced investigators and study personnel has the potential to improve data quality across studies. Many organizations are now interested in providing or participating in such training.
Data Standardization
Standardization has been successfully used in many industries to improve quality. In the clinical trial arena, terminologies, forms or computer screens used to collect data, and the tools and systems used to analyze the data are candidates for standardization. Entries for medical events and for concomitant medications were noted as areas of particular complexity. FDA is evaluating ways to further standardize the presentation of safety data.
Data Simplification
Although millions of data points are collected during the average clinical trial, some of these data are not used in the decision-making process. One important strategy for improving quality is to simplify data collection by distinguishing critical data prospectively. Further work is needed to provide a better understanding of what data sets are being submitted and how they might be simplified.
Improvements In Quality Control and Quality Assurance Methods
Benchmark standards are developed for data quality in the clinical trial setting. A significant amount of effort goes into detecting and correcting problems, such as missing data points and transcription errors, which are very different from uncertainty issues. Participants agreed that there is a "hierarchy of errors." Because a clinical trial often generates millions of data points, ensuring 100 percent completeness, for example, is often not possible; however, it is also not necessary. High-quality data may be defined as data strong enough to support conclusions and interpretations equivalent to those derived from error-free data. Certain data points are more important to interpreting the outcome of a study than others, and these should receive the greatest effort and focus. Implementation of this definition would require agreement on data standards.
Evaluations of monitoring and auditing techniques are also needed. Industry uses a variety of monitoring and auditing techniques to ensure high-quality data. These have not been compared or tested to see which are the most effective. Additional discussions and evaluation of monitoring and auditing techniques should be undertaken.