Review of a Research Protocol Prepared by the University of Utah: Letter Report (2002)

Advisers to the Nation on Science, Engineering, and Medicine

National Academy of Sciences

National Academy of Engineering

Institute of Medicine

National Research Council

On October 25, 2001, the National Research Council’s Committee to Review a Research Protocol Prepared by the University of Utah met at the Beckman Center of the National Academies in Irvine, California. The task before the committee was to review critically the proposed methods and analyses and to assess whether they were appropriate and complete. This task differs somewhat from the usual ones the committee addresses which are to review draft reports issued at the end of a study. However, in this instance, the very limited information provided the committee precluded a detailed evaluation of the study and its likelihood of success. Accordingly, the committee focused its attention on identifying improvements rather than commenting on the basic proposal.

The committee notes, however, that a study of thyroid disease in the Utah-Nevada-Arizona population has the promise to produce important information. The Chernobyl studies have clearly demonstrated an excess of thyroid cancer following much higher ¹³¹I exposures, while the Hanford fallout study, with a lower average ¹³¹I dose, did not show any thyroid effects. The Utah study has a higher dose distribution than the Hanford study, so that it could potentially have greater statistical power and be more informative in the low dose range. It may also have less uncertainty in the dosimetry than Hanford because of the substantial number of external exposure measurements made in the study areas at the time of the Nevada Test Site nuclear detonations. The past performance of these investigators in locating and enlisting the participation of the study subjects in the previous cycle of this study was very good, which is important for minimizing selection bias. The potential value of this study thus provides a context in which to evaluate the proposed study design, the dosimetric, epidemiologic and clinical procedures, the analytic methods and the statistical power. However, an appropriate statistical analysis is needed to demonstrate whether this potential could be realized. The committee has attempted to be critical in a constructive manner, even to the extent of providing a technical framework for the estimation of statistical power, given the sources of uncertainty in the study data (see Appendix B).

2001 Wisconsin Ave, NW, Washington, DC 20007 USA

202-334-2232 (telephone) 202-334-1639 (fax) E-mail: national-academies.org

The Radiation Studies Branch of the Centers for Disease Control and Prevention (CDC) charged the committee to address the following:

1. Are the study objectives attainable?

2. Are the proposed study design and the methods for data collection and statistical analyses unambiguous, and do they adequately address the study objectives? In particular, we would like the Committee to comment on the following:

   1. The proposed expansion of the study cohort beyond the previous follow-up study of 1985-1986.

   2. The adequacy of the proposed design in dealing with possible detection, selection, and information biases.

   3. The appropriateness of the proposed outcome measures.

   4. The approach for accounting for uncertainty and variability in individualized dose estimates and the dose-response analysis.

3. Has the statistical power of the study been appropriately addressed?

Present at the October 25, 2001 meeting, in addition to the members of the committee and the BRER staff, were representatives of the investigative team at the University of Utah, Joseph L. Lyon, the principal investigator; Wayne Meikle, the coinvestigator; Mary Bishop Stone, the program manager; Stephen Alder, the project’s statistician, and Owen Hoffman, who was recently hired as a consultant to the project, and Judy Qualters and Felix Rogers of CDC. Dr. Lyon and his colleagues described the proposed study and responded to the committee’s questions. The committee’s comments and recommendations set out in the paragraphs that follow stem from the written, albeit incomplete, documentation of the proposed research, the presentations, and responses to the committee’s queries. Our comments and recommendations are organized around the questions previously listed.

Question 1. Are the study objectives attainable?

The protocol is not sufficiently developed nor rigorous enough to permit a judgment of whether all the study objectives are attainable. Four objectives are stated:

“a. to test whether a healthy group of children, inadvertently exposed to radioactive iodine will have an increased number of thyroid neoplasms related to this exposure while controlling for other causes of thyroid cancer including medical x-rays and occupational radiation.”

Insufficient and incomplete information is presented in the protocol, and too few analyses have apparently been carried out, for the committee to judge the attainability of this objective in this cohort. In particular two elements necessary to ensure that the objective will be met were not adequately described in the protocol: safeguards in the study’s design, data collection and analysis to minimize the likelihood and magnitude of biases, and a correctly conceptualized method for determining the statistical power of the protocol and its application in the study design. Some examples of such issues are discussed in the committee's answer to question number 2 and in Appendix A.

“b. to determine if exposure to low doses of radiation was associated with non-neoplastic forms of thyroid disease detected and/or occurring up to 44 years after the exposure.”

Again, the evidence presented in the protocol pertinent to this objective is too limited for the committee to judge whether it is attainable; further specification of safeguards against bias and evaluation of the statistical power attainable in this cohort are required (Some examples of such issues are discussed in the committee's answer to question number 2 and in Appendix A). In addition, inadequate attention was paid to ways of detecting and controlling for geographic variation in the prevalence of these thyroid conditions.

“c. to identify, locate, and enroll individuals who were exposed as children to the fallout in the 1950s in Washington and Lincoln counties, but who moved from these counties before the study group was assembled in 1965.”

The protocol lacks detailed plans for locating the people in question (especially women), having them screened for thyroid disease wherever they reside, and ensuring their screening rates will be high enough to impart confidence in the validity of this part of the study. One would have expected to see a detailed statement of methods and some preliminary data to predict how successfully this objective can be attained.

“d. to update the Phase II dose assignment model with information that has become available during the past 15 years, and to improve the treatment of uncertainty in the dose assignment model.”

Too little information is given in the protocol as to the dose-assignment model that has been or will be used, the information that has become available, and the methods proposed for treatment of uncertainty for the committee to determine attainability of this objective. Substantial information on the thyroid-dose method used in Phase II has been published (see, for example, Stevens et al., 1992, and Simon et al., 1990). It would have been highly desirable to summarize that information in the protocol, to indicate the similarities and differences with the proposed dose assignment model, and to indicate what new information and improvements were to be incorporated.

Question 2. Are the proposed study design and methods for data collection and statistical analyses unambiguous, and do they adequately address the study objectives? In particular, we would like the Committee to comment on the following:

1. The proposed expansion of the study cohort beyond the previous follow-up study of 1985-1986.

2. The adequacy of the proposed design in dealing with possible detection, selection, and information biases.

3. The appropriateness of the proposed outcome measures.

4. The approach for accounting for uncertainty and variability in individualized dose estimates and the dose-response analysis.

The statements of the objective of this research—as in the task statement, the study

statement, the questionnaires, and the consent form—are inconsistent and inaccurate. For example, the sole statement of purpose on the exposure questionnaire “this survey is part of a study of the effects of lifestyle and the environment on health” is overly vague, so that it is questionable whether or not this meets the usual standards of informed consent; and a further example of an incorrect statement of purpose is given in Appendix A. Provision of misleading or inaccurate statements of purpose to study subjects is unacceptable. The task and study statements, questionnaires, consent forms, and any other study subject contact documents or scripts should be carefully reviewed and revised to make them consistent and accurate. Similarly, the definitions of the statistical outcomes need to be sharpened and made more specific. For instance, our meeting with the investigators showed that little, if any, thought had been given to which persons should be excluded (because of prior disease) in the piecewise temporal analyses. In addition, the protocol should contain information on the expected number of cases of (rather than just rates of) thyroid cancer in the study in the absence of exposure to fallout radiation from the nuclear detonations at the Nevada Test Site and on the expected number of excess cases under the several scenarios used in the statistical power calculations. This would aid in interpreting the statistical power results. The same applies to thyroid nodules and other disease outcomes.

2a: The proposed expansion of the study cohort beyond the previous follow-up study of 1985-1986. The estimates of the probable attained study size seem optimistic and perhaps unrealistic. First, they do not take into account the study losses that will occur if parents are not available for dosimetry interviews (for example, with respect to milk consumption). At the meeting, these losses were estimated to be about 20%. That estimate seems to be low in light of the current age distribution of the parents; many will be deceased. There is no explanation or protocol to cover the case in which no parents would be available for interviews. A carefully reasoned estimate or the result of a pilot study is needed rather than an extemporaneous number.

Second, the investigators assumed that they would be able to locate and enlist the participation of 90% of the new augmented-population members. Tracing such people who reside outside of the three-state area, especially women, after about 50 years will be difficult. Furthermore, their participation in going to their local thyroid physicians in diverse locations might lead to greater attrition than that in the Utah-Nevada-Arizona in-state screening program and may thereby be subject to greater self-selection bias, as well as to lack of uniformity in the thyroid screening procedures by the local physicians. More attention to the rates of study losses is warranted. There was no pilot-program information on and virtually no plan for obtaining screening participation from those who reside outside the three-state area. That weakness applies to the subgroups that have moved out of state both before and after 1965.

We recommend that before deciding upon whether to augment the cohort with out-of-staters, a pilot study on a random sample is needed to determine the success rate in locating these subjects, in obtaining thyroid screening participation, and in obtaining milk consumption information from mothers. A separate reliability study should be conducted to determine the reliability between current and (preferably) 1965 or 1986 reports of milk consumption/source information; this is needed to determine the degree of uncertainty in current milk-exposure estimates and its effect on the amount of statistical power gain afforded by augmenting the study cohort.

Third, milk-consumption data will have to be obtained now for virtually all the new additions to the cohort. The milk-consumption data will be obtained about 50 years after the exposure time, and a substantial fraction will have to be obtained from surrogates of the mothers (because of the death or infirmity of the mothers). Studies show that the reliability of food-consumption data obtained several decades after the fact is quite poor (Dwyer et al., 1989). There is likely to be less gain in statistical power than one might expect from adding numbers if the added exposure data are unreliable. That raises a question about the value of study-size augmentation. At a minimum, we recommend that simulations be performed to estimate the effect of unreliable milk-consumption data on the exposure estimates and on the consequent degree of gain in study power afforded by adding subjects with relatively unreliable exposure information to the study.

2b: The adequacy of the proposed design in dealing with possible detection, selection, and information biases. Several elements in the epidemiologic and statistical methods require further thought and specification to reduce bias and achieve validity. A number of safeguards to minimize information bias were missing from the protocol. Safeguards should be fully developed to maximize study-staff and subject blinding. For instance, the parent dosimetry interviews should be conducted before subjects’ thyroid screening. At the screening, a subject should complete the medical-history and other questionnaires first. The examiners (palpation and sonography) should be blinded as to each subject's medical-history report. While it can be argued that the person performing the palpation should be blinded to the sonographic results and vice versa, here it would seem that a consensus of the examiners on the interpretation of their findings would improve the study design. The recommendation for fine-needle aspiration (FNA) should be made after blinded review of sonograms by the three radiologists on the review panel. The cytologic reading should also be performed in a blinded fashion, and surgical specimens should have a blinded pathologic review. A control should be built into the protocol to ensure that all examiners conduct the same proportions of examinations in low- and high-exposure geographic areas; otherwise, examiner differences could produce a bias.

Several additional specifications of criteria must be instituted to ensure objectivity. For example, the biopsy criterion for a "prominent nodule" in a multinodular gland was not specified (for instance, must it be greater than 1 cm in diameter?). Criteria should be given for performance of FNA (for instance, under what circumstances will one, two or more aspirates be obtained?).

The protocol did not indicate which "other potential confounders" will be evaluated for possible inclusion in the analyses. When asked, the investigators did not indicate one of the more important potential confounders: that a subject had participated in the 1965 and 1985 screenings (screening will detect thyroid disease in addition to or earlier than that found in routine medical care).

The investigators propose to augment the more highly exposed portion of the cohort from the targeted Utah and Nevada counties by searching state birth records to identify persons who were probably in the targeted Utah and Nevada counties during the peak fallout period but who emigrated from those states before the initial cohort definition in 1965. However, they do not propose the same augmentation for the "low-exposure" Arizona group. The Utah-Nevada group

would therefore contain early out-migrants, but the Arizona group would not, so the subject-selection procedures would not be comparable. Whether that would bias thyroid-disease rates is not known, but the fact that it is not known means that it should be appropriately guarded against by proper study design.

There has been no evaluation of the reproducibility or accuracy of the milk-consumption questionnaire or of the effect of obtaining the answers at substantially different times after the potential exposure, and no evaluation was built into the study. At a minimum, the investigators could reinterview a subsample of perhaps 200 parents or parent surrogates who were interviewed in 1986 and conduct a detailed comparison of responses on the two occasions. Better yet would be comparing the 1986 and current questionnaire data related to milk consumption with the 1965 milk-consumption estimates if they are still available. In fact, if there are enough 1965 data, it would be valuable to analyze thyroid disease in relation to the dose estimates based on the 1965 data as an alternative check on results. Thought could also be given to whether statistical and other methods could be developed to permit using the 1965 data for the corresponding 1986-2002 milk-consumption variables when the former are available.

2c: The appropriateness of the proposed outcome measures. A more complete description of the clinical procedures is necessary. It should include

• Methods used to minimize examiner bias based on history or medical findings.

• Quality-assurance procedures for laboratory, ultrasound, and examination procedures.

• Pertinent specification of the ultrasonic equipment and how it will be used. The choice of ultrasound equipment needs to consider the method of recording data for subsequent image analysis and display.

• FNA procedure—discussion of the rationale for single vs multiple biopsies of nodules to minimize sampling errors.

We recommend that the investigators lay out a protocol in detail that describes how they will handle persons residing outside the three-state area and that they explicitly consider the maintenance of consistency between those inside and those outside the three-state area, provide information on how the screening will be handled for those residing outside the three-state area, and consider the financial aspects of this protocol.

2d: The approach for accounting for uncertainty and variability in individualized dose estimates and the dose-response analysis. The approach to uncertainty and variability in individual dose estimates and the dose-response analysis is poorly described and appears contradictory. For example, the choice of the regression method (linear or logistic) for the primary analysis of incidence data differs throughout the protocol. Appendix D of the proposal describes a method for quantifying uncertainty in the risk estimates while allowing for uncertainty in dose, but the committee is not convinced that the approach is appropriate (and we were confused by the material presented to us). In any event, the description of the approach is not transparent; it is important to revise the description by using mathematical notations that define more precisely the concepts presented in this section.

The calculations for the primary source term, ¹³¹I in milk, are not obvious. The questionnaires have a number of items pertaining to the milk source, but it is not clear how any of the data, other than decay time, will be used. There is discussion of cow-to-cow variation among breeds of cows in home milk vs dairy milk, cows vs goats, and so on, but the equations presented seem to have only a single term for milk and to use a single value for fallout-to-feed and feed-to-milk factors. Where did those factors come from? Are they national averages, or are they adjusted for local conditions? Are home conditions the same as dairy conditions? This shows again that the extensive information on Phase II dosimetry that is included in Stevens et al. (1992) should have been summarized in the protocol, if, indeed, the methodology of Stevens et al. (1992) is to be used in the proposed study. The protocol should state how the questionnaire data are to be used, or if they are not to be used, why they are being collected. It should also evaluate whether the uncertainties discussed would significantly affect the power of the study, or could introduce significant biases.

To complete the computation of estimated dose for people in the study, it is necessary to estimate the deposition of ¹³¹I on the ground from the airborne plume generated by the weapons tests. That was apparently done in Phase II (the proposal gives no information on this point) by assuming that the deposition of ¹³¹I was spatially constant in each of the counties. However, as has been shown in a number of studies (Beck and Anspaugh, 1991, Thompson, 1990), deposition concentrations are not spatially constant, and therefore location effects should be examined for their potential effect on the power of the study or in introducing bias, and perhaps taken into account. Weather is at least one of the factors in this spatial differentiation. The proposal claims (page 18) that areal deposition will be updated, taking weather into account, but there is no information on how this will be done or on the sources and nature of the weather information that will be used.

Question 3. Has the statistical power of the study been appropriately addressed?

The protocol developed some expected rates of thyroid cancer for various dose groups based on the National Council on Radiation Protection and Measurements (NCRP) (1985) risk estimates and cancer-registry data but did not use them in the calculations. The protocol spoke of tripling the rates of thyroid cancer observed in cancer registries, but it was unclear if this was done. Similarly, it spoke of using a relative biological effectiveness (RBE) of three (that is, an effectiveness factor of 1/3) but apparently did not do so. The statistical-power projection ended up using the central estimate of the risk coefficient from the previous Utah study to project risk. That estimate is uncertain (being based on only eight cases). The investigators might better have used consensus estimates by the committee on NCRP (1985), Biological Effects of Ionizing Radiation (NRC, 1990), or others. Furthermore, it is strongly recommended that the investigators not use a single estimate for statistical-power calculations, but vary the input parameters (screening-effect magnitude, RBE, risk coefficient, and input parameters for the dosimetry uncertainties) and produce a range of statistical-power estimates. We also recommend that power calculations be performed for selected nonneoplastic thyroid diseases for which the background prevalence rates are much higher, e.g., autoimmune diseases. This would give a more realistic idea of the likely statistical power of the study.

Some power calculations are presented in the protocol, but they are based on a simulation

with several puzzling aspects. For example, in the simulation study to estimate power, the sample size was taken to be random and to range in value from 1000-5000 subjects. In power calculations it is not customary to have the sample size be a random quantity (although often a range of power calculations corresponding to plausible sample sizes are presented). Here, however, the range of sample size was very extreme, and did not correspond to the estimates of sample size given in the protocol or in the presentations by other investigators. Formula 4 on page 40 is confused in that σ² is described there as the variance of true dose given the estimated dose, whereas it should be the variance of estimated dose or, more precisely, the variance of the expected value of true dose given the input data available for an individual (see NRC, 2000). Even if that is corrected, the formula is valid only when errors are not shared from subject to subject. Because of those logical difficulties, we conclude that the investigators have not made the statistical power of the study clear. The power calculations as described in the protocol do not take into account the fact that the primary analysis will include the previously reported cases of cancer and neoplasia, which have already shown a marginally significant dose-response. The main issue, then, to be addressed is the amount of new information that will be obtained in the update of the study. Formal power calculations should be done separately for the updated study, neglecting all the previously obtained data (so that follow-up of each subject only starting from the time of the previous examination up to the present exam, is considered). Then in a combined analysis, the investigators could consider the likely reduction of the length of the confidence interval, for the dose response already obtained, when the new study information is added to the old. In other words the importance of the new information in adding to the old is of paramount importance.

In summary, it is important that the investigators attempt to include in their statistical power simulation model all the sources of uncertainty, and that the simulation model treat the “shared” and “unshared” sources of uncertainty appropriately. A technical report prepared by one of our members, given as Appendix B, outlines a statistical approach to do so.

The committee’s recommendation is that the decision on whether the study is conducted should be based on a scientifically defensible protocol in which an improved design is advanced and the proposed methods and analyses are justified in detail. Specifically:

• The thyroid dose assignment should be compared with the methodology used in 1986, and the improvements should be clearly identified. A preliminary reliability/validity study should be conducted to compare milk consumption/source data obtained in 2002 with that obtained in 1965, if possible, or at least in 1986. This would help determine the value of the proposed augmentation of subjects and of the study in general.

• The power of the study should be more carefully investigated and summarized under a variety of conditions reflecting the uncertainties in the size of the study population, completeness of data, and uncertainties in exposure assessments.

• The proposed augmentation of study size using out-of-staters merits consideration if it meets several criteria: the study design for the augmentation guards against bias; current recall of milk consumption/sources is sufficiently reliable and accurate to assure that the added subjects would significantly improve study power; and pilot data demonstrate the ability to locate, screen and obtain parental milk consumption reports from a high proportion of these subjects.

• The letters, questionnaires, and consent forms should not be used until they are appropriately revised to be consistent with the study objectives and to reduce potential bias.

• The information provided on the number of people to be interviewed, the length of interview time, and the number of interview teams appears inconsistent with the project schedule. The study schedule should be reviewed carefully.

• Adequate safeguards need to be built into the protocol to minimize biases in the study design, data collection and analysis. Thorough safeguards should be developed to ensure appropriate blinding of both study-staff and subjects. The study details should be reviewed and revised as appropriate to ensure to the maximum extent possible that "blind" is indeed blind. For instance, coded identification numbers on documents for purposes of person identification should be replaced with random numbers. This needs to be corrected before interviews begin.

If you desire elaboration on the comments above or the accompanying appendixes, please do not hesitate to call or write to Dr. Isaf Al-Nabulsi or me.

Sincerely yours,

William J. Schull

Chairman

Beck, H. L. and Anspaugh, L. R. Development of the County Data Base: Estimates of Exposure Rates and Times of Arrival of Fallout in the ORERP Phase-II Area. Comparison of Cumulative Deposition Density Estimates Based on Analyses of Retrospective and Historical Soil Samples. United States Department of Energy Report DOE/NV-32, 1991.

Dwyer, J. T., Gardner, J., Halvorsen, K., Krall, E. A., Cohen, A., Valadian, I. Memory of Food Intake in the Distant Past. Am J Epidemiol. 130: 1033-1046, 1989.

NCRP (National Council on Radiation Protection and Measurements). Induction of Thyroid Cancer by Ionizing Radiation. Report No. 80. Recommendations of the National Council on Radiation Protection and Measurements. Bethesda, MD. 1985.

NRC (National Research Council). Review of the Hanford Thyroid Disease Study Draft Final Report. Washington, DC: National Academy Press, 2000.

NRC (National Research Council). Health Effects of Exposure to Low Levels of Ionizing Radiation. The Committee on the Biological Effects of Ionizing Radiation (BEIR V). Washington, DC: National Academy Press, 1990.

Simon, S. L., Lloyd, R. D., Till, J. E., Hawthorne, H. A., Gren, D. C., Rallison, M., and Stevens, W. Development of a Method to Estimate Dose from Fallout Radioiodine to Persons in a Thyroid Cohort Study. Health Physics 59(5): 669-691, 1990.

Stevens, W., Till, J. E., Thomas, D. C., Lyon, J. L., Kerber, R. A., Preston-Martin, S., Simon, S. L., Rallison, M. L., and Lloyd, R. D. Assessment of Leukemia and Thyroid Disease in Relation to Fallout in Utah: Report of a Cohort Study of Thyroid Disease and Radioactive Fallout from the Nevada Test Site. Supported by the National Cancer Institute (Contract #N01-CO-23917), U. S. Public Health Service, Department of Health and Human Services. Salt Lake City, UT: University of Utah, 1992.

Thompson C. B. Estimates of Exposure Rates and Fallout Arrival Times Near the Nevada Test Site. Health Physics 59:555-563, 1990.