Page 91
4
General Guidelines
This report has surveyed a large number of experimental design and analysis strategies that are useful, at least to some degree, in studies with small numbers of participants. Throughout the report the committee has pointed out that, whenever possible, large and adequately powered randomized clinical trials are the method of choice. The committee has also noted that in some cases such studies are impractical or impossible to conduct and that one must derive inferences from less rigorously controlled studies with less statistical power. To this end, the committee has presented several different designs that can be used in a variety of different circumstances in which full randomized clinical trials are not possible. In addition, the committee has presented several different analytical strategies, some common and others somewhat novel, that form a basic toolkit for small clinical studies. Here the committee provides some basic guidance on types of trial designs and analysis strategies that should be used and the circumstances in which they should be used. The reader should note that this guidance is limited, and different approaches or combinations of these approaches may be more useful in a specific setting.
The committee has discussed a variety of analysis issues, including sequential analysis, hierarchical models, Bayesian analysis, decision analysis, statistical prediction, meta-analysis, and risk-based allocation. When an investigator is attempting to better understand a dose-response relation or a
Page 92
response surface and has limited resources, sequential analysis is an ideal technique. It allows an adaptive approach to the allocation of resources in the most efficient way possible, such that one may identify an optimal dosage or set of conditions with the smallest number of participants.
When the data are collected in clusters, for example, from space missions or through collaboration with several clinics, hierarchical models, meta-analysis, or statistical prediction strategies may be useful. When choosing among these, consider the following. Meta-analysis can be used if different outcome or assessment measures are used for the different clusters such that each cluster is a separate study. The advantage of meta-analysis is that it allows one to combine information from studies or cluster samples that do not share a common endpoint. By contrast, a hierarchical model can be used if the studies or clusters contain both experimental and control conditions and they all use the same endpoint. The hierarchical model will adjust the standard errors of the estimated parameters for the within-cluster correlation that is produced by sharing a common environment. Hierarchical models are also the method of choice when there are repeated measurements for the same individuals, either over time or in a crossover study in which each participant is subjected to two or more treatment conditions. In this case the individual is the cluster and the hierarchical model is a method that can be used to put together what are essentially a series of n-of-1 experiments with a sample size of 1 (n-of 1 experiments). In some cases, however, each cluster may contain only experimental participants and one wishes to compare the clusters sequentially with a historical or parallel control group. Typically, the latter groups are larger. Since control and experimental conditions are not nested within clusters, hierarchical models do not apply and in fact would confound the difference between experimental measurements and the control measurements within the random cluster effect. This case is treated, however, as a problem of statistical prediction, in which the control measurements are used to derive an interval that will contain a proportion of the experimental measurements (e.g., 50 percent or the median) in each cluster with a reasonable level of confidence.
By contrast, decision analysis, Bayesian approaches, and ranking and selection are generally used to arrive at a decision about whether a particular intervention is useful or better than some alternative. These approaches are generally more subjective and may call for expert opinions to reach a final decision. Bayesian methods underlie many of the methods described here, including prediction, meta-analysis, and hierarchical models. Among these, ranking and selection allow one to arrive at a statistically optimal solution with a modicum of subjective inputs; however, decision analysis often allows
Page 93
a more complete characterization of a problem. Coupled with Bayesian methods, decision analysis has the additional benefit of being able to assess the sensitivity of the decision rule to various inputs or assumptions that went into constructing the decision rule.
Finally, the committee has also presented risk-based allocation as a useful tool for research with small numbers of participants. This method is quite different from the others but can be useful for those cases in which it may be unethical to withhold treatment from a high-risk population by randomly assigning them to a control group.