Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 163
6
Lessons from the Case Studies
Omics research is a continually evolving field, with rapidly advanc-
ing scientific techniques that are highly dependent on bioinformatics and
rigorous statistical methods to effectively interpret high-dimensional data.
Despite the challenges inherent in a new field and some setbacks (Dupuy
and Simon, 2007; Ransohoff, 2005; Ransohoff and Gourlay, 2010; Simon
et al., 2003), a number of omics-based tests derived from this research
have reached the market and are being used to manage patient care across
a broad range of medical specialties and subspecialties.
The purpose of the case studies was to examine the test discovery
and development processes used in omics research and omics-based test
development to help determine the criteria needed to effectively guide
development of omics-based tests, and to consider the roles of the various
responsible parties in test development (discussed in Chapter 5). These
examples have influenced the emerging field of omics-based tests and will
continue to guide the field as more information about these tests accrues.
The case studies focus on factors such as:
• The discovery and confirmation of omics-based tests;
• Analytical validation;
• Statistical and bioinformatics validation;
• Clinical/biological validation; and
• Clinical utility and clinical use.
The most extensive case study centers on several omics-based tests
developed by a Duke University laboratory to predict sensitivity to chemo-
163
OCR for page 164
164 EVOLUTION OF TRANSLATIONAL OMICS
therapeutic agents (see Appendix B). These tests were used to select patient
therapy in the three clinical trials identified in the IOM committee’s state-
ment of task. In addition, the committee examined six tests that are cur-
rently commercially available (Table 6-1): Oncotype DX (Genomic Health),
MammaPrint (Agendia), Tissue of Origin (Pathwork Diagnostics), OVA1
(Vermillion), AlloMap (XDx), and Corus CAD (CardioDx), as well as
one test that did not advance to clinical use (OvaCheck). These cases
reflect a number of diseases and types of omics research. The committee
also reviewed the development and use of human epidermal growth fac-
tor receptor 2 (HER2) testing, one of the earliest single-biomarker tests
to guide the choice of cancer therapy. This last case study illustrates the
challenges involved in the development of a single analyte test and sug-
gests how the complexity could be magnified for multianalyte, omics-based
tests. Despite many years of research and development, challenges remain
in defining the optimal test method and interpretation for HER2 testing.
METHODS
Examination of what transpired in the development of the omics-based
tests at Duke University was based on presentations to the committee, a
panel discussion with Duke University researchers and administrators, doc-
uments provided by the National Cancer Institute and Duke University, and
the peer reviewed literature. The Duke case study is illustrative of the more
systemic challenges involved in the development of omics-based tests and
the need for rigorous criteria for test development, and is discussed in more
detail in Appendix B. For the other case studies, the committee reviewed
publicly available materials, including peer reviewed publications and Food
and Drug Administration (FDA) 510(k) clearance decision summaries. The
committee also invited presentations on the development of Oncotype DX
and MammaPrint (Shak, 2011; van ‘t Veer, 2011). Each of the six summa-
ries for the commercially available tests was sent to the respective company
for review of factual accuracy and completeness.
Detailed case studies were prepared for Oncotype DX, MammaPrint,
Tissue of Origin, and AlloMap. The organization of the detailed case
studies reflects the test development process that the committee recom-
mends in Chapters 2-4, including candidate omics-based test discovery
and confirmation, development into a defined and validated omics-based
test, and evaluation for clinical use. Shorter case studies were assembled
for HER2, OVA1, OvaCheck, and Corus CAD, and include details on dis-
covery, development, and clinical use. Case studies appear in Appendix A.
OCR for page 165
165
LESSONS FROM THE CASE STUDIES
TABLE 6-1 Overview of Commercially Available Omics-Based Tests
Test Indication
Company (diagnostic, prognostic, FDA
effect modifier)a
Test Name and Test Website Clearance?
Oncotype DX Genomic Health Prognostic No
www.oncotypedx.com/ “a 21-gene assay that provides
en-US/Breast.aspx an individualized prediction of
chemotherapy benefit and
10-year distant recurrence to
inform adjuvant treatment
decisions in certain women with
early-stage breast cancer”
(Genomic Health, 2011).
ASCO guidelines state:
“Oncotype DX may be used to
identify patients who are
predicted to obtain the most
therapeutic benefit from adjuvant
tamoxifen and may not require
adjuvant chemotherapy. In
addition, patients with high
recurrence scores appear to
achieve relatively more benefit
from adjuvant chemotherapy
than from tamoxifen” (Harris et
al., 2007).
MammaPrint Agendia Prognostic Yes
www.agendia.com/pages/ “a qualitative in vitro diagnostic
mammaprint/21.php test service, performed in a
central laboratory, using the gene
expression profile of fresh breast
cancer tissue samples to assess a
patient’s risk for distant
metastasis (up to 10 years for
patients less than 61 years old,
up to 5 years for patients ≥ 61
years) . . . for breast cancer
patients, with Stage I or Stage II
disease, with tumor size ≤ 5.0 cm
and lymph node negative. [Test]
result is indicated for use as a
prognostic marker only” (FDA,
2011b).
continued
OCR for page 166
166 EVOLUTION OF TRANSLATIONAL OMICS
TABLE 6-1 Continued
Test Indication
Company (diagnostic, prognostic, FDA
effect modifier)a
Test Name and Test Website Clearance?
Tissue of Pathwork Diagnostics Diagnostic Yes
“The Pathwork® Tissue of
Origin www.pathworkdx.com/
TissueOfOriginTest/ Origin Test is an in vitro
diagnostic intended to measure
the degree of similarity between
the RNA expression patterns in
a patient’s formalin-fixed,
paraffin-embedded (FFPE) tumor
and the RNA expression patterns
in a database of fifteen tumor
types (poorly differentiated,
undifferentiated, and metastatic
cases) that were diagnosed
according to the current clinical
and pathological practice” (FDA,
2010).
OVA1 Vermillion Guides referral to a gynecologic Yes
http://ova-1.com/ oncologist
“a qualitative serum test that
combines the results of five
immunoassays into a single
numerical score for women with
an ovarian adnexal mass present
for which surgery is planned as
an aid to further assess the
likelihood that malignancy is
present when the physician’s
independent clinical and
radiological evaluation does not
indicate malignancy. The test is
not intended as a screening or
stand-alone diagnostic assay”
(FDA, 2011c).
OCR for page 167
167
LESSONS FROM THE CASE STUDIES
TABLE 6-1 Continued
Test Indication
Company (diagnostic, prognostic, FDA
effect modifier)a
Test Name and Test Website Clearance?
AlloMap XDx Prognostic Yes
www.allomap.com/ “an In Vitro Diagnostic
Multivariate Index assay
(IVDMIA) test service,
performed in a single laboratory,
assessing the gene expression
profile of RNA isolated from
peripheral blood mononuclear
cells (PBMC[s]). [It] is intended
to aid in the identification of
heart transplant recipients with
stable allograft function [at least
2 months (≥ 55 days)
posttransplant] with a low
probability of moderate/severe
acute cellular rejection (ACR) at
the time of testing” (FDA,
2008).
Corus CAD Cardio Dx Diagnostic No
www.cardiodx.com/ “blood test that can quickly and
corus-cad/ safely assess whether or not [a]
product-overview/ patient’s symptoms are due to
obstructive coronary artery
disease (CAD) . . . a decision-
making tool that can help
identify patients unlikely to have
obstructive CAD” (Cardio Dx,
2011).
aThe designation here reflects the intended use described in the FDA 510(k) decision summary,
if there is one, or if not, the company-proposed use and/or guideline recommendation.
NOTE: ASCO = American Society of Clinical Oncology, FDA = Food and Drug Administration.
OCR for page 168
168 EVOLUTION OF TRANSLATIONAL OMICS
LESSONS LEARNED FROM THE CASE STUDIES
The review of the case studies highlighted several key concepts for
omics-based test discovery, development, and validation. These include
• importance of a well-designed development plan;
• data and code availability;
• avoidance of overlap between discovery and validation specimens;
• locking down all aspects of the test prior to evaluation for clinical
utility and use;
• interaction with FDA;
• clinical/biological validation characteristics;
• assessment of clinical utility; and
• role of the investigators and institutions in scientific oversight.
An important point to keep in mind is that these case studies are rep-
resentative of an early period in the new field of omics-based test develop-
ment, prior to broad agreement on standard processes and criteria for the
various stages of test development. As the scientific community gains expe-
rience with omics research, more information about ideal test development
processes will continue to emerge. Likewise, evidence on clinical utility for
these tests will also become richer as more information accrues over time.
Recommendations from this IOM committee on the processes for omics-
based test development aim to provide needed clarity to the field.
Importance of a Well-Designed Development Plan
As described in Chapter 2, an important aspect of test development
is a well-designed development plan. Components of a test development
plan include starting with a clinically meaningful question, developing a
candidate test on a training set of specimens, locking down the candidate
test, and employing rigorous test validation procedures. The availability
of appropriate archival tissue for clinical/biological validation can greatly
facilitate rigorous development procedures. If appropriate archival samples
are not available for assessment of clinical utility, then the development plan
should include a prospective clinical trial design.
Tests developed in universities (or where the developmental process is
started in universities) are likely to grow out of basic research on the bio-
logical meaning of the omics elements that comprise the test and gradually
evolve into more formal test development. This is illustrated by the studies of
HER2 and MammaPrint, although the development of MammaPrint crossed
into a more formalized approach after Agendia was formed. In contrast,
companies are likely to start with a focus on a specific test that will eventu-
OCR for page 169
169
LESSONS FROM THE CASE STUDIES
ally have commercial value and thus generally have a clear development
plan early on, as seems to be the case with Genomic Health. According to a
presentation to the IOM committee by Steven Shak, chief medical officer of
Genomic Health, Oncotype DX had a clearly articulated development plan
from the start. Investigators defined the purpose of the test and subsequently
created and implemented a multistep, multistudy approach to develop their
omics-based test “to provide the evidence regarding analytic performance,
clinical[/biological] validity, and clinical utility to meet the needs of patients,
physicians, payors, and regulators” (Shak, 2011).
Data and Code Availability
As described in Chapter 2, the committee recommends that data and
metadata used for identification of an omics-based test should be made
available in an independently managed database in standard format, and
that code and fully specified computational procedures for omics-based
tests should be made sustainably available. Chapters 2 and 5 discuss the
importance of sharing data within the scientific community, especially in
the setting of omics research, where data and analyses are particularly
complex. Sharing data can enable external verification of the results and
allow other investigators to generate additional insights from the data. In
Chapter 5, the committee recommends that journals and funders require
the public availability of data, metadata, prespecified analysis plans, code,
and fully specified computational models.
The importance of data availability was highlighted in the OvaCheck
case study, in which publicly available datasets enabled external researchers
to uncover serious problems in experimental design (Baggerly et al., 2004)
that ultimately demonstrated that the published computational model
(Petricoin et al., 2002) was based on artifacts and not on relevant biologi-
cal signals.
In the Duke case, external investigators did not have full access to the
data and code, and this limited the ability to independently evaluate the tests
to determine whether they were valid (Baggerly and Coombes, 2009; Baron
et al., 2010). Conflicting and unclear information in the papers and cited
references regarding the data and statistical methods contributed to the
inability of colleagues in the scientific community to understand and repli-
cate the generation of the computational models (Baggerly, 2011; McShane,
2010; Review of Genomic Predictors for Clinical Trials from Nevins, Potti,
and Barry, 2009). When Baggerly and Coombes attempted to assess the
validity of the tests, they determined that there was insufficient information
to reproduce the published results using the available data and the methods
published in the Nature Medicine paper (Baggerly, 2011; Potti et al., 2006).
A review of the six commercially available tests (Table 6-2) illustrates
OCR for page 170
170 EVOLUTION OF TRANSLATIONAL OMICS
TABLE 6-2 Data Availability
Computational
Model
Test Published? Raw Data from Discovery Publicly Available?
Oncotype DX Yes Discovery RT-PCR data not available
(Paik et al.,
2004)
MammaPrint No Discovery microarray data available
Tissue of No Raw data from discovery includes both publicly
Origin available (GEO accession number GSE2109) and
private sources
OVA1 No Initial raw data used to find a preliminary panel of
biomarkers is available; raw data used to finalize the
panel of biomarkers in the OVA1 test is unavailable;
data on samples used to train the computational model
are available in the 510(k) decision summary, but are
not available in peer-reviewed literature
AlloMap Yes The microarray data used to initially identify
(Deng et al., biomarkers during the discovery phase of product
2006) development are available in GEO under accession
number GSE2445. Raw RT-PCR training data were
provided to FDA but not reported in Deng et al. (2006)
Corus CAD Yes Discovery microarray data available (GEO accession
(Rosenberg et number GSE20686)
al., 2010)
NOTE: FDA = Food and Drug Administration, GEO = Gene Expression Omnibus, RT-PCR
= reverse-transcriptase polymerase chain reaction.
aPersonal communication, Laura van ‘t Veer, Agendia, November 1, 2011.
bPersonal communication, Steve Rosenberg, CardioDX, December 12, 2011.
OCR for page 171
171
LESSONS FROM THE CASE STUDIES
Is the Methodology to Derive the
Went Through Computational Model Available, and
FDA Clearance in Sufficient Detail to Fully Independent Clinical Research
Processes? Reproduce? Entity Involved?
No Methodology used to derive and train Yes
the computational model is not National Surgical Adjuvant
available in sufficient detail to fully Breast and Bowel Project
reproduce
Yes Some of the details needed for Yes
independent replication are unclear TRANSBIG, a Consortium of
from the supplementary materials, but the Breast International Group
inquiries about the computational
model have always been answereda
Yes Methodology used to derive and train No
the computational model is not
available
Yes Methodology used to derive the Yes (PrecisionMed International
computational model is not available housed specimens, Quest
Diagnostics performed
biomarker measurements,
Applied Clinical Intelligence
performed data analysis)
Yes Unknown. A description of the Yes
methods for biomarker discovery, test Cardiac Allograft Rejection
development, and computational Gene Expression Observational
model are detailed in the Deng et al. investigators
(2006) supplement
No Some of the details needed for Yes
independent replication, such as “The analysis was
penalization tuning parameters for independently performed under
Ridge regression, are not provided. the supervision of Dr. Schork at
RT-PCR data used to derive the Scripps Translational Science
computational model were not Institute”
published but are available upon
request to qualified investigatorsb
OCR for page 172
172 EVOLUTION OF TRANSLATIONAL OMICS
that public availability of all omics-based test data and code has not been
the standard of practice. The field of omics is early in its development and
the standards for data sharing have been unclear and only slowly evolving
toward more transparency. Commercial interests and protection of propri-
etary information may also limit the public availability of some data and
information.
The cases highlight several examples in which test developers explicitly
note the availability of data. For example, Paik et al. (2004), Deng et al.
(2006), and Rosenberg et al. (2010) report the computational model for
Oncotype DX, AlloMap, and Corus CAD, respectively. Both tests devel-
oped as laboratory-developed tests (LDTs) had published computational
models (Oncotype DX and Corus CAD); only one FDA-cleared test has a
published computational model (AlloMap). Discovery microarray data are
available for MammaPrint, AlloMap, and Corus CAD (Deng et al., 2006;
van ‘t Veer et al., 2002).1 Buyse et al. (2006) report that raw microarray
data and clinical data for the MammaPrint clinical validation study were
deposited with the European Bioinformatics Institute ArrayExpress data-
base. Although there are examples of developers reporting the availability
of a test’s computational model or data used in discovery or validation, as
Table 6-2 shows, there often is not enough information publicly available
for external investigators to fully reproduce a test.
The IOM committee recognized that it might not always be possible
to make this information publicly available due to the protection of intel-
lectual property. For publicly funded research, the committee recommends
that code and fully specified computational procedures should be made
available at the time of publication or at the end of funding. For commer-
cially developed tests, code and fully specified computational procedures
would be submitted for FDA review if seeking approval or clearance, or
would be described in a publication in the case of an LDT (see Chapter
2). Companies that seek FDA clearance or approval for their tests would
have had to submit data to FDA as part of the 510(k) clearance processes
or premarket approval (PMA) processes, respectively, but only the infor-
mation reported in the FDA decision summary is made publicly available.
Avoidance of Overlap of Discovery and Validation Specimens
One of the challenges of omics research is the lack of available
specimens on which to conduct correlative science and exploration of
candidate omics-based tests (reviewed in IOM, 2010). Partly due to this
challenge, a number of tests have been developed with overlapping train-
1 Microarray data from Corus CAD are available, but PCR data used in test development
are unavailable. Personal communication, Steve Rosenberg, October 21, 2011.
OCR for page 173
173
LESSONS FROM THE CASE STUDIES
ing and validation datasets. In the cases that the committee reviewed, two
commercial omics-based tests used overlapping training and development
datasets at some point in their development processes: MammaPrint and
AlloMap (Table 6-3). Of the samples used in the development of the
MammaPrint computational model, 78 percent were reused in the van de
Vivjer et al. (2002) study, leading to criticism about the overlap between
training and validation datasets (Kim and Paik, 2010; Ransohoff, 2003,
2004). A subsequent validation study (Buyse et al., 2006) was performed,
and suggested that overfitting due to the use of training samples was likely
a problem in the 2002 study. In the AlloMap case, the first validation
study used 63 specimens that had not been used in previous development
of the test. This second validation included the 63 primary validation sam-
ples plus samples from patients who had contributed samples to the gene
discovery and/or diagnostic development phases of the study. Overlap of
training and validation datasets can lead to a number of problems in test
discovery and development, including overstatement of the accuracy of an
omics-based test and incorrect error estimation (Leek et al., 2010). The
OvaCheck case study illustrates the importance of independent validation
datasets (Diamandis, 2004). If test samples are obtained independently,
preferably drawn from another institution at a different point in time, and
prepared and analyzed separately, then the risk of batch effects is greatly
reduced. As described in Chapter 2, test developers should clearly separate
training and validation datasets in order to avert these problems, which
are among the most common causes of a test ultimately failing clinical
validation.
TABLE 6-3 Statistical and Bioinformatics Validation Considerations
Overlap of Discovery
and Validation Datasets
at Some Point in Test
Test Lock-Down Reported? Development?
Oncotype DX Yes (Paik et al., 2004) No
Yes (Personal communicationa)
MammaPrint Yes
Tissue of Origin Yes (Monzon et al., 2009; Pillai et al., 2011; No
personal communicationb )
Yes (Personal communicationc)
OVA1 No
Yes (Personal communicationd)
AlloMap Yes
Corus CAD Yes (Rosenberg et al., 2010) No
aPersonal communication, Laura van ‘t Veer, Agendia, November 28, 2011.
bPersonal communication, Ed Stevens, Pathwork Diagnostics, October 18, 2011.
cPersonal communication, Scott Henderson, Vermillion, December 12, 2011.
dPersonal communication, Mitch Nelles, XDx, October 12, 2011.
OCR for page 174
174 EVOLUTION OF TRANSLATIONAL OMICS
Locking Down All Aspects of the Test Prior to Evaluation for Clinical Use
In Chapter 4, the committee recommends that a validated omics-based
test should not be changed during a clinical trial. The omics-based test
should be locked down prior to this stage of development (more informa-
tion on the importance on locking down an omics-based test can be found
in Chapters 2, 3, and 4). As noted by Baggerly and colleagues and by
McShane, the computational models developed by a Duke University lab
were not locked down prior to use in the clinical trials (Baggerly, 2011;
McShane, 2010). This was a serious shortcoming in the development of
the Duke omics-based tests. For three other cases, the developers explicitly
stated in the study publication that their test was locked down before clini-
cal validation: Oncotype DX (Paik et al., 2004), Tissue of Origin (Monzon
et al., 2009; Pillai et al., 2011), and Corus CAD (Rosenberg et al., 2010).
Communication with test developers provided additional confirmation that
AlloMap, OVA1, and MammaPrint were locked down (Table 6-3).
Interaction with FDA
In Chapter 4, the committee recommends that FDA clarify the regu-
lation of omics-based tests by developing and finalizing a guidance or
regulation defining which omics-based tests require FDA review, the type
of review required, and when this review should occur. A similar guidance
should be developed and finalized for oversight of LDTs that are currently
not reviewed by FDA.
Review of the committee’s case studies demonstrates that companies
have pursued both LDT and FDA pathways for developing an omics-based
test. The use of multiple pathways indicates a lack of clarity and consistency
on the regulatory requirements for omics-based tests. Five of the commer-
cially available tests that the committee examined are performed exclusively
by each company’s proprietary Clinical Laboratory Improvement Amend-
ments of 1988 (CLIA)-certified laboratory.2 Two companies did not seek
FDA clearance and market their tests as LDTs: Genomic Health (Oncotype
DX) and CardioDx (Corus CAD). Four companies received FDA 510(k)
clearance of their tests: Agendia (MammaPrint), Pathwork Diagnostics
(Tissue of Origin), Vermillion (OVA1), and XDx (AlloMap). The publicly
available 510(k) clearance decision documents summarize the analytical
and clinical validation results that the company/sponsor submitted to FDA.
However, FDA clearance does not mean that a test has clinical utility, and
2 OVA1 is performed exclusively by Quest Diagnostics, which is subject to CLIA certification
(Quest Diagnostics, 2011). Currently Pathwork Diagnostics offers Tissue of Origin exclusively
through its CLIA-certified laboratory, but is developing an in vitro diagnostic test kit for other
laboratories (Pathwork Diagnostics, 2010).
OCR for page 175
175
LESSONS FROM THE CASE STUDIES
lack of FDA clearance does not mean that a test does not have clinical util-
ity. This is an important distinction between the FDA approval process for
drugs versus devices (which includes omics-based tests).
In Chapter 5, the committee also recommends that FDA communicate
the investigational device exemption (IDE) requirements for omics-based
tests used in clinical trials. Commercial developers, such as those examined
in the case studies, may be more familiar with IDE requirements than aca-
demic institutions. In several of the case studies, companies and FDA held
a pre-IDE meeting to determine whether an IDE would be required for the
company’s test development process. FDA determined that an IDE was not
needed for both the AlloMap and Tissue of Origin tests because the test
was not directing patient therapy in the studies proposed to assess the test.3
Physicians can now use the test for that purpose, however. Agendia reported
that it received an IDE for MammaPrint that helped clarify the process
and requirements for the de novo 510(k),4 and Vermillion reported that
it received an IDE for OVA1.5 Two ongoing prospective studies (the TAI-
LORx and RxPONDER trials) direct patient management on the basis of
the Oncotype DX Recurrence Score. For both trials, information required
for approval of investigational use of Oncotype DX in the trial was sub-
mitted as part of an investigational new drug application to FDA.6 In the
Duke case study, the investigators did consult FDA regarding the need for
an IDE (FDA, 2011a). In 2009, FDA sent a letter to Duke stating that the
omics-based tests being studied in the three clinical trials named in the IOM
statement of task needed to go through the IDE process (Chan, 2009). In
response, the investigators made some changes to the protocol of the stud-
ies, and Duke contacted FDA for further clarification about whether an
IDE was still required (FDA, 2011a; Potti, 2009). The Duke Institutional
Review Board (IRB) determined that an IDE was not needed when it did
not receive a reply from FDA7 (FDA, 2011a). However, in retrospect, the
Duke IRB recognized that an IDE should have been obtained for the omics-
based tests because the tests were used to direct patient management in the
clinical trials (FDA, 2011a).
Regardless of which pathway is taken to market, consultation with FDA
3 Personal communication, Mitch Nelles, XDx, October 12, 2011; Personal communication,
Ed Stevens, Pathwork Diagnostics, October 18, 2011.
4 Personal communication, Laura van ‘t Veer, Agendia, November 28, 2011.
5 Personal communication, Scott Henderson, Vermillion, November 1, 2011.
6 Personal communication, Lisa McShane, National Cancer Institute, February 9, 2012.
7 According to the FDA website, the Center for Drug Evaluation and Research has no record
of receiving the December 2009 letter from Dr. Anil Potti discussing an exemption for the
trial that received pre-IDE review. The letter was brought to FDA’s attention during its 2011
inspection of the Duke IRB and clinical investigators (see http://www.fda.gov/MedicalDevices/
ProductsandMedicalProcedures/ InVitroDiagnostics/ucm289100.htm).
OCR for page 176
176 EVOLUTION OF TRANSLATIONAL OMICS
can be beneficial and is recommended. For example, the developers of the
OVA1 test sought FDA input, and this early dialogue with FDA prompted
Vermillion to include two different cut-off values for the test, depending
on a patient’s menopausal status (Fung, 2010). Although Genomic Health
did not meet with FDA, the company indicated that it benefited from past
experience working with FDA and from the extensive background material
FDA provides on its website about assay validation.8
Clinical/Biological Validation Characteristics
The choice of study design for the clinical/biological validation studies of
the tests varied widely among the case studies (Table 6-4). Designs included
retrospective studies, a prospective–retrospective study (using archived speci-
mens from previously conducted, formal clinical trials that evaluated treat-
ment options that might be affected by the test’s use), and prospective trials
in which the test was not directing therapy. In the Duke case, the investiga-
tors attempted to validate some of the tests using cell lines, or in one case,
clinical samples from patients with breast cancer (Bonnefoi et al., 2007).
However, prospective clinical trials in which the omics-based tests were
used to direct patient therapy were initiated prematurely, before clinical/
biological validity had been established.
As described in Chapter 3, the committee recommended that the iden-
tity of the specimens used for clinical/biological validation of the omics-
based test should be blinded to the individuals performing and interpreting
the test results during clinical/biological validation of the test when fea-
sible. In some instances, this may entail working with another independent
group to conduct validation studies. In the case of Oncotype DX, Genomic
Health completed reverse-transcriptase polymerase chain reaction (RT-
PCR) analysis of the specimens used for clinical validation and supplied
the results to the National Surgical Adjuvant Breast and Bowel Project,
who conducted the analyses evaluating the association between recurrence
score and clinical outcome. In the clinical validation for MammaPrint by
Buyse et al. (2006), the data were housed at TRANSBIG,9 and the statistical
analyses were conducted by the International Drug Development Institute.
For Corus CAD, the publication describing the clinical/biological validation
noted that an investigator at the Scripps Translational Science Institute was
responsible for the data analyses, while the laboratory work was performed
at CardioDX.
8 Personal communication, Steven Shak, Genomic Health, December 13, 2011.
9A consortium launched by the Breast International Group (BIG) to promote international
collaboration in translational research. (See www.breastinternationalgroup.org/Research/
TRANSBIG.aspx.)
OCR for page 177
177
LESSONS FROM THE CASE STUDIES
TABLE 6-4 Choice of Trial Designs for Clinical/Biological Validation
Test Clinical Validation Designs
Oncotype DX Prospective–retrospective (Paik et al., 2004)
Retrospective (Habel et al., 2006)
MammaPrint Retrospective
Tissue of Origin Retrospective
OVA1 Prospective (not directing therapy—trial was for development and
validation)
AlloMap Prospective (not directing therapy—trial was for development and
validation)
Corus CAD Prospective (not directing therapy—trial was for development and
validation)
The development and validation pathways for Oncotype DX and Mam-
maPrint offer some interesting comparisons. Kim and Paik (2010) note
that both MammaPrint and Oncotype DX went through their respective
development steps either “by design or by demand from the community.”
Investigators largely chose different strategies for test discovery and devel-
opment, including the type of tissue used in discovery, the type of risk score
(dichotomous variable versus low-, intermediate-, and high-risk categories);
different regulatory approaches; and a different intended use population.
Oncotype DX was designed for patients with estrogen-receptor-positive,
lymph-node-negative, early-stage breast cancer. MammaPrint has a poten-
tially larger indication, including patients with estrogen-receptor-negative
tumors. Despite these differences, MammaPrint and Oncotype DX have
shown around 80 percent agreement in outcome classification (Fan et al.,
2006), although they only have one gene in common.
Assessment of Clinical Utility
Clinical utility is defined as “evidence of improved measurable clinical
outcomes, and [a test’s] usefulness and added value to patient management
decision-making compared with current management without [the] test”
(Teutsch et al., 2009, p. 11). Assessment of clinical utility is not part of the
FDA’s evaluation of a test, and generally, clinical utility is determined after
a test or device is on the market, sometimes decades later. Clinical utility
is different from the intrinsic attributes of a test’s performance characteris-
tics, such as sensitivity and specificity. Clinical utility is not concerned with
how a test performs, but rather how its use influences health outcomes. As
described in Chapter 4, the ideal way to assess clinical utility is through
OCR for page 178
178 EVOLUTION OF TRANSLATIONAL OMICS
prospective randomized controlled trials addressing, for example, whether
the use of the new test results in an increase in the length or quality of life,
a significant increase in progression-free survival, or avoidance of unneces-
sary treatment (especially toxic treatment) by patients who are unlikely to
benefit. Clinical trials assessing clinical utility are important because they
can inform how a test is used in practice.
As described in Chapter 4, prospective trial designs in which bio-
markers direct patient management (as in Figures 4-4 and 4-5) are useful
approaches to assess the clinical utility of the biomarker. However, they
are also the study designs that pose the highest risk to trial participants,
because treatment decisions are determined by the test result. The justi-
fication for using these designs depends substantially on the amount of
information known about the test. For example there were no reported
validation attempts using clinical tumor samples from patients with lung
cancer for the cisplatin test developed by the Duke University lab, even
though the first trial in which the cisplatin test was used to guide therapy
was the NCT00509366 trial for advanced lung cancer. This type of trial
design may have been premature, given the lack of tumor-specific validation
of the cisplatin test.
Evidence on clinical utility evolves as new information about a test
emerges, and the absence of data on clinical utility should not be inter-
preted as a lack of utility. A defining feature of both the MammaPrint and
Oncotype DX development stories is the need for a large, prospective trial
to provide more information on each test’s utility in clinical practice. In
both cases, the prospective trial was not initiated until after the test was
on the market and widely available for clinical use as a prognostic factor.
The results of the prospective studies TAILORx and MINDACT will pro-
vide prospective information on the clinical utility of Oncotype DX and
MammaPrint, respectively, for the first time.
The Role of Investigators and Institutions in Scientific Oversight
Chapter 5 comprehensively discusses the roles of responsible parties in
the conduct of omics research and omics-based test development. Here, the
roles of the investigators and institutions are highlighted for their central
role in ensuring rigorous omics research and test development. As illus-
trated in the Duke case study (Appendix B), transparency and open com-
munication are integral to the conduct of science, whether it be reporting
of data and code, disclosure of conflicts of interest, or reporting of potential
breaches in scientific procedures.
Investigators are responsible for the accuracy of their data, the fairness
of their conclusions, and responding appropriately to criticism. Investiga-
tors ensure that clinical research is conducted with the engagement of
OCR for page 179
179
LESSONS FROM THE CASE STUDIES
appropriate scientific expertise, including the involvement of individuals
with proper biostatistics and bioinformatics expertise, and that the research
has the approval of relevant review bodies. It also is important for all
members of a research team to understand the aims and intricacies of col-
laborative studies and for coauthors of a publication to keep each other
informed about constructive criticism of the work and ways to improve
ongoing research.
Institutions play an important role in establishing a culture of scien-
tific integrity and transparency, including setting expectations of behavior,
achievement, and integrity, and providing safe environments for reporting
irregularities to prevent lapses in scientific integrity. Institutions are directly
charged with being the “oversight” bodies when specific scientific questions
or challenges arise, including investigating questions of misconduct, or sim-
ply in investigating “soundness of science.” Oversight processes that will
maintain integrity even in the presence of institutional conflicts of interest,
both financial and non-financial (such as factors that impact an institution’s
reputation) may be especially important in addressing this charge. Closer
attention to such conflicts may have been helpful in avoiding the events that
occurred in the Duke case (see Appendix B).
CONCLUDING REMARKS
The case studies illustrate the multitude of considerations that must
be taken into account to move omics research into test development for
clinical use. The involvement of investigators, institutions, funders, and
journals is essential for ensuring good research practices and oversight of
omics-based test discovery and development. A well-designed test develop-
ment plan addresses a clinically meaningful question and employs rigorous
test discovery, development, and validation procedures. This includes lock-
ing down all aspects of an omics-based test prior to evaluation for clinical
utility and use, and avoiding overlap between discovery and validation
specimens. Choosing an appropriate clinical/biological validation strategy
and interacting with FDA prior to initiation of validation studies also reflect
a well-designed test development plan. Making data and code available are
critical aspects of test development because it enables external verification
of the results and generation of additional insights that can advance science
and patient care.
REFERENCES
Baggerly, K. A. 2011. Forensics Bioinformatics. Presentation at the Workshop of the IOM
Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in
Clinical Trials, Washington, DC, March 30-31.
OCR for page 180
180 EVOLUTION OF TRANSLATIONAL OMICS
Baggerly, K. A., and K. R. Coombes. 2009. Deriving chemosensitivity from cell lines: Forensic
bioinformatics and reproducible research in high-throughput biology. Annals of Applied
Statistics 3(4):1309-1334.
Baggerly, K. A., J. S. Morris, and K. R. Coombes. 2004. Reproducibility of SELDI-TOF pro-
tein patterns in serum: Comparing datasets from different experiments. Bioinformatics
20(5):777-785.
Baron, A. E., K. Bandeen-Roche, D. A. Berry, J. Bryan, V. J. Carey, K. Chaloner, M. Delorenzi,
B. Efron, R. C. Elston, D. Ghosh, J. D. Goldberg, S. Goodman, F. E. Harrell, S. Galloway
Hilsenbeck, W. Huber, R. A. Irizarry, C. Kendziorski, M. R. Kosorok, T. A. Louis, J. S.
Marron, M. Newton, M. Ochs, J. Quackenbush, G. L. Rosner, I. Ruczinski, S. Skates, T. P.
Speed, J. D. Storey, Z. Szallasi, R. Tibshirani, and S. Zeger. 2010. Letter to Harold Varmus:
Concerns about Prediction Models Used in Duke Clinical Trials. Bethesda, MD, July 19,
2010. http://www.cancerletter.com/categories/documents (accessed January 18, 2012).
Buyse, M., S. Loi, L. J. van ‘t Veer, G. Viale, M. Delorenzi, A. M. Glas, M. S. d’Assignies, J.
Bergh, R. Lidereau, P. Ellis, A. Harris, J. Bogaerts, P. Therasse, A. Floore, M. Amakrane,
F. Piette, E. T. Rutgers, C. Sortiriou, F. Cardoso, and M. J. Piccart. 2006. Validation and
clinical utility of a 70-gene prognostic signature for women with node-negative breast
cancer. Journal of the National Cancer Institute 98(17):1183-1192.
Cardio Dx. 2011. What Is Corus CAD? http://www.cardiodx.com/corus-cad/product-overview/
(accessed November 21, 2011).
Chan, M. M. 2009. Letter to Division of Medical Oncology, Duke University Medical
Center. http://www.fda.gov/downloads/MedicalDevices/ProductsandMedicalProcedures/
InVitroDiagnostics/UCM289102.pdf (accessed February 9, 2012).
Deng, M. C., H. J. Eisen, M. R. Mehra, M. Billingham, C. C. Marboe, G. Berry, J. Kobashigawa,
F. L. Johnson, R. C. Starling, S. Murali, D. F. Pauly, H. Baron, J. G. Wohlgemuth, R. N.
Woodward, T. M. Klingler, D. Walther, P. G. Lal, S. Rosenberg, S. Hunt, and for the
CARGO Investigators. 2006. Noninvasive discrimination of rejection in cardiac allo-
graft recipients using gene expression profiling. American Journal of Transplantation
6(1):150-160.
Diamandis, E. 2004. Mass spectrometry as a diagnostic and a cancer biomarker discov-
ery tool: Opportunities and potential limitations. Molecular and Cellular Proteomics
3(4):367-378.
Dupuy, A., and R. M. Simon. 2007. Critical review of published microarray studies for cancer
outcome and guidelines on statistical analysis and reporting. Journal of the National
Cancer Institute 99(2):147-157.
Fan, C., D. S. Oh, L. Wessels, B. Weigelt, D. S. A. Nuyten, A. B. Nobel, L. J. van ‘t Veer, and
C. M. Perou. 2006. Concordance among gene-expression-based predictors for breast
cancer. New England Journal of Medicine 355(6):560-569.
FDA (Food and Drug Administration). 2008. 510(k) Substantial Equivalence Determination
Decision Summary Assay and Instrument Combination Template. http://www.accessdata.
fda.gov/cdrh_docs/reviews/K073482.pdf (accessed November 21, 2011).
FDA. 2010. 510(k) Substantial Equivalence Determination Decision Summary (K092967).
http://www.accessdata.fda.gov/cdrh_docs/reviews/K092967.pdf (accessed November 16,
2011).
FDA. 2011a. FDA Establishment Inspection Report, Duke University Medical Center .
h ttp://www.fda.gov/downloads/MedicalDevices/ProductsandMedicalProcedures/
InVitroDiagnostics/UCM289106.pdf (accessed February 9, 2012).
FDA. 2011b. 510(k) Substantial Equivalence Determination Decision Summary (K101454).
http://www.accessdata.fda.gov/cdrh_docs/reviews/K101454.pdf (accessed September 19,
2011).
OCR for page 181
181
LESSONS FROM THE CASE STUDIES
FDA. 2011c. Substantial Equivalence Determination Decision Summary (K081754). http://
www.accessdata.fda.gov/cdrh_docs/reviews/K081754.pdf (accessed October 11, 2011).
Fung, E. T. 2010. A recipe for proteomics diagnostic test development: The OVA1 test, from
biomarker discovery to FDA clearance. Clinical Chemistry 56(2):327-329.
Genomic Health. 2011. Overview: What Is the Oncotype DX Assay? http://www.oncotypedx.
com/en-US/Breast/HealthcareProfessional/Overview.aspx (accessed September 14, 2011).
Habel, L. A., S. Shak, M. Jacobs, A. Capra, C. Alexander, M. Pho, J. Baker, M. Walker, D.
Watson, J. Hackett, N. T. Blick, D. Greenberg, L. Fehrenbacher, B. Langholz, and C.
P. Quesenberry. 2006. A population-based study of tumor gene expression and risk
of breast cancer death among lymph node-negative patients. Breast Cancer Research
8(3):R25.
Harris, L., H. Fritsche, R. Mennel, L. Norton, P. Ravdin, S. E. Taube, M. R. Somerfield, D. F.
Hayes, and R. C. Bast Jr. 2007. American Society of Clinical Oncology 2007 update
of recommendations for the use of tumor markers in breast cancer. Journal of Clinical
Oncology 25(33):5287-5312.
IOM (Institute of Medicine). 2010. A National Clinical Trials System for the 21st Century:
Reinvigorating the Cooperative Group Program. Washington, DC: The National Acad-
emies Press.
Kim, C., and S. Paik. 2010. Gene-expression-based prognostic assays for breast cancer. Nature
Reviews 7(6):340-347.
Leek, J. T., R. B. Scharpf, H. C. Bravo, D. Simcha, B. Langmead, W. E. Johnson, D. Geman, K.
Baggerly, and R. A. Irizarry. 2010. Tackling the widespread and critical impact of batch
effects in high-throughput data. Nature Reviews Genetics 11(10):733-739.
McShane, L. M. 2010a. NCI Address to the Institute of Medicine Committee on the Review
of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials. Presented at
Meeting 1. Washington, DC, December 20.
Monzon, F. A., M. Lyons-Weiler, L. J. Buturovic, C. T. Rigl, W. D. Henner, C. Sciulli, C. I.
Dumur, F. Medeiros, and G. G. Anderson. 2009. Multicenter validation of a 1,550-gene
expression profile for identification of tumor tissue of origin. Journal of Clinical Oncol-
ogy 27(15):2503-2508.
Paik, S., S. Shak, G. Tang, C. Kim, J. Baker, M. Cronin, F. L. Baehner, M. G. Walker, D.
Watson, T. Park, W. Hiller, E. R. Fisher, D. L. Wickerham, J. Bryant, and N. Wolmark.
2004. A multigene assay to predict recurrence of tamoxifen-treated, node-negative breast
cancer. New England Journal of Medicine 351(27):2817-2826.
Pathwork Diagnostics. 2010. Pathwork Tissue of Origin test for FFPE cleared by U.S. Food and
Drug Administration. http://www.pathworkdx.com/News/M129_FDA_Clearance_Final.
pdf (accessed November 17, 2011).
Petricoin, E. F., A. M. Ardekani, B. A. Hitt, P. J. Levine, V. A. Fusaro, S. M. Steinberg, G. B.
Mills, C. Simone, D. A. Fishman, E. C. Kohn, and L. A. Liotta. 2002. Use of proteomic
patterns in serum to identify ovarian cancer. Lancet 359(9306):572-577.
Pillai, R., R. Deeter, C. T. Rigl, J. S. Nystrom, M. H. Miller, L. Buturovic, and W. D.
Henner. 2011. Validation and reproducibility of a microarray-based gene expression
test for tumor identification in formalin-fixed, paraffin-embedded specimens. Journal of
Molecular Diagnostics 13(1):48-56.
Potti, A. 2009. Letter to FDA’s CDER from Division of Medical Oncology, Duke University Medi-
cal Center. http://www.fda.gov/downloads/MedicalDevices/ProductsandMedicalProcedures/
InVitroDiagnostics/UCM289103.pdf (accessed February 9, 2012).
Potti, A., H. K. Dressman, A. Bild, R. F. Riedel, G. Chan, R. Sayer, J. Cragun, H. Cottrill,
M. J. Kelley, R. Petersen, D. Harpole, J. Marks, A. Berchuck, G. S. Ginsburg, P. Febbo,
J. Lancaster, and J. R. Nevins. 2006. Genomic signatures to guide the use of chemothera-
peutics. Nature Medicine 12(11):1294-1300.
OCR for page 182
182 EVOLUTION OF TRANSLATIONAL OMICS
Quest Diagnostics. 2011. Licenses and Accreditation. http://www.questdiagnostics.com/brand/
company/b_comp_licenses.html (accessed November 21, 2011).
Ransohoff, D. F. 2003. Gene-expression signatures in breast cancer. New England Journal of
Medicine 348(17):1716.
Ransohoff, D. F. 2004. Rules of evidence for cancer molecular-marker discovery and valida-
tion. Nature Reviews Cancer 4(4):309-314.
Ransohoff, D. F. 2005. Bias as a threat to the validity of cancer molecular-marker research.
Nature Reviews Cancer 5:142-148.
Ransohoff, D. F., and M. L. Gourlay. 2010. Sources of bias in specimens for research about
molecular markers for cancer. Journal of Clinical Oncology 28(4):698-704.
Review of Genomic Predictors for Clinical Trials from Nevins, Potti, and Barry. 2009.
Durham, NC: Duke University.
Rosenberg, S., M. R. Elashoff, P. Beineke, S. E. Daniels, J. A. Wingrove, W. G. Tingley, P. T.
Sager, A. J. Sehnert, M. Yau, W. E. Kraus, K. Newby, R. S. Schwartz, S. Voros, S. G.
Ellis, N. Tahirkhelli, R. Waksman, J. McPherson, A. Lansky, M. E. Winn, N. J. Schork,
E. J. Topol, and for the PREDICT (Personalized Risk Evaluation and Diagnosis In the
Conorary Tree) Investigators. 2010. Multicenter validation of the diagnostic accuracy of
a blood-based gene expression test for assessing obstructive coronary artery disease in
nondiabetic patients. Annals of Internal Medicine 153(7):425-434.
Shak, S. 2011. Case Study: Oncotype DX Breast Cancer Assay. Presentation at the Workshop
of the IOM Committee on the Review of Omics-Based Tests for Predicting Patient Out-
comes in Clinical Trials, Washington, DC, March 30.
Simon, R., M. D. Radmacher, K. Dobbin, and L. M. McShane. 2003. Pitfalls in the use
of DNA microarray data for diagnostic and prognostic classification. Journal of the
National Cancer Institute 95(1):14-18.
Teutsch, S. M., L. A. Bradley, G. E. Palomaki, J. E. Haddow, M. Piper, N. Calonge, D.
Dotson, M. P. Douglas, and A. O. Berg. 2009. The Evaluation of Genomic Applications
in Practice and Prevention (EGAPP) Initiative: Methods of the EGAPP Working Group.
Genetics in Medicine 11(1):3-14.
van de Vijver, M. J., Y. D. He, L. J. van ‘t Veer, H. Dai, A. A. M. Hart, D. W. Voskuil, G. J.
Schreiber, J. L. Peterse, C. Roberts, M. J. Marton, M. Parrish, D. Atsma, A. Witteven,
A. M. Glas, L. Delahaye, T. van der Velde, H. Bartelink, S. Rodenhuis, E. T. Rutgers, S. F.
Friend, and R. Bernards. 2002. A gene-expression signature as a predictor of survival in
breast cancer. New England Journal of Medicine 347(25):1999-2009.
van ‘t Veer, L. J. 2011. Case study—MammaPrint. Presented at Meeting 2 of the Commit-
tee on Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials,
Washington, DC, March 30.
van ‘t Veer, L. J., H. Dai, M. J. van de Vijver, Y. D. He, A. A. M. Hart, M. Mao, H. L. Peterse,
K. van der Kooy, M. J. Marton, A. T. Wittereveen, G. J. Schreiber, R. M. Kerkoven, C.
Roberts, P. S. Linsley, R. Bernards, and S. F. Friend. 2002. Gene expression profiling
predicts clinical outcome of breast cancer. Nature 415(31):530-536.
