Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 20
A Strategy for Assessing Science: Behavioral and Social Research on Aging 2 The NIA Behavioral and Social Research Program This chapter describes behavioral and social science research in the National Institute on Aging (NIA), the research investments made by the Behavioral and Social Research (BSR) Program in NIA, and the ways in which research investments are currently evaluated, prospectively and retrospectively, in the National Institutes of Health (NIH) generally and BSR specifically. STRATEGIC GOALS OF NIA NIA is one of 24 grant-making entities among the 27 institutes and centers that make up the National Institutes of Health. In each institute or center, scientific programs are organized into programmatic areas, such as the BSR Program in NIA. NIA’s mission, as described in its strategic plan (National Institute on Aging, 2001; http://www.nia.nih.gov/AboutNIA/StrategicPlan/), is “to improve the health and well-being of older Americans through research.” NIA supports “research on aging processes, age-related diseases, and special problems and needs of the aged,” the training of researchers for work in these areas, resources for accelerating research progress, and dissemination of information on research advances and directions to the public and interested groups. Pursuant to its mission, NIA supports research in a variety of biomedical and social science areas. Box 2-1 lists the institute’s major research goals, which are described in greater detail in the strategic plan. Even this abbreviated listing makes clear the breadth of the institute’s mandate. NIA implements its mission by supporting both biological and social science research; both problem-focused, applied research
OCR for page 21
A Strategy for Assessing Science: Behavioral and Social Research on Aging BOX 2-1 Research Goals and Subgoals of the National Institute on Aging Goal A: Improve health and quality of life of older people Prevent or reduce age-related diseases, disorders, and disability Maintain health and function Enhance older adults’ societal roles and interpersonal support and reduce social isolation Goal B: Understand healthy aging processes Unlock the secrets of aging, health, and longevity Maintain and enhance brain function, cognition, and other behaviors Goal C: Reduce health disparities among older persons and populations Increase active life expectancy and improve health status for older minority individuals Understand health differences associated with race, ethnicity, gender, environment, socioeconomic status, geography, and culture Monitor health, economic status, and life quality of elders and inform policy Goal D: Enhance resources to support high-quality research Train and attract a diverse workforce of new, mid-career, and senior researchers necessary for research on aging Develop and sustain a diverse NIA workforce and a professional environment that supports and encourages excellence Disseminate accurate and compelling information to the public, scientific community, and health care professionals Develop and distribute research resources SOURCE: National Institute on Aging (2001). and research on basic processes related to health, illness, and well-being; and research at all levels of analysis from the molecular to the societal. Reflecting the range of scientific research areas supported by NIH, NIA organizes its research support within four programs: Biology of Aging, Geriatrics and Clinical Gerontology, BSR, and Neuroscience and Neuro-psychology of Aging. Because of the variety of types of research that NIA supports, its decisions about research portfolios can be quite challenging. Among these challenges, especially in apportioning funds, are those of comparing and setting priorities among research fields. Moreover, the institute’s mission requires it to judge research fields both on scientific grounds and in
OCR for page 22
A Strategy for Assessing Science: Behavioral and Social Research on Aging terms of their potential to improve the health and well-being of older Americans. Thus, its priority setting and budget planning unavoidably involve comparisons of research investments both among program areas and within programs. The questions BSR has posed for this study are integral not only to its mission, but also to those of NIA and NIH. THE DIVERSE BSR RESEARCH PORTFOLIO As stated on its web site (http://www.nia.nih.gov/ResearchInformation/ExtramuralPrograms/BehavioralAndSocialResearch/), the BSR Program “supports basic social and behavioral research and research training on the processes of aging at both the individual and societal level.” It focuses on the following: How people change during the adult lifespan Interrelationships between older people and social institutions The societal impact of the changing age composition of the population Emphasis is placed on: (1) the dynamic interplay between individuals' aging; (2) their changing biomedical, social, and physical environments; and (3) multilevel interactions among psychological, physiological, social, and cultural levels. BSR supports research, training, and the development of research resources and methodologies to produce a scientific knowledge base for maximizing active life and health expectancy. This knowledge base is required for informed and effective public policy, professional practice, and everyday life. BSR also encourages the translation of behavioral and social research into practical applications. In the 2005 fiscal year, the BSR Program awarded $159.5 million for research—about one-fifth of all awards in NIA and a doubling from the level of 1997 in current dollars. This included approximately $75 million in support for investigator-initiated projects (funding categories R01 and R03), $28 million in support for larger program projects (P01), $14 million in cooperative agreements (U01), and smaller amounts for other categories of research support (see Table 2-1). Tables 2-2A and 2-2B show the amounts awarded for research in BSR’s nine topical areas between 1997-2000 and 2001-2005 respectively, highlighting the breadth of the BSR research portfolio (two tables are needed because the organizational structure of BSR changed after FY 2000). It also shows the shifts in levels of support for research overall and in the different areas. Overall research support doubled between 1997 and 2003, but it has been essentially unchanged since then. As Tables 2-2A and 2-2B show, research support for demographic research in BSR increased every year from
OCR for page 23
A Strategy for Assessing Science: Behavioral and Social Research on Aging TABLE 2-1 BSR Grants Awarded in Millions of Dollars (numbers of awards in parentheses) by Mechanism in FY 1997-2005 Category 1997 1999 2001 2003 2005 Research program $54 (228) $69 (243) $ 97 (308) $122 (311) $125 (315) Center 10 (28) 10 (22) 10 (22) 12 (44) 13 (39) Small business innovation research 8 (35) 7 (29) 8 (32) 9 (39) 8 (33) Career 2 (19) 3 (28) 5 (42) 6 (51) 6 (50) Training 4 (25) 5 (28) 6 (31) 7 (33) 7 (35) Interagency agreements 2 (11) 5 (31) 4 (31) 4 (26) 4 (20) TOTAL $79 $98 $130 $161 $164 NOTE: Totals do not add up to exact amount due to rounding off of figures. SOURCE: Behavioral and Social Research Program, National Institute on Aging. TABLE 2-2A Distribution of BSR Program Funds (in millions of current dollars) by Research Topic, FY 1997-2000 Topic 1997 1998 1999 2000 Cognitive functioning 13.4 14.9 16.6 16.8 Personality and social psychology 5.2 4.9 5.2 8.4 Old people in society 5.3 7.5 7.6 7.7 Psychosocial geriatrics 11.5 14.8 17.1 19.5 Health care organizations 11.6 9.5 8.8 8.4 Demography 12.3 14.4 15.2 20.4 Population epidemiology 5.9 6.1 7.6 9.7 Health and retirement economics 13.1 13.8 9.1 11.3 Databases (e.g., Health and Retirement Survey) 0.6 0.6 7.7 9.3 TOTAL 79.0 86.6 94.9 111.3 TABLE 2-2B Distribution of BSR Program Funds (in millions of current dollars) by Research Topic, FY 2001-2005 Topic 2001 2002 2003 2004 2005 Behavioral medicine 31.8 44.9 44.0 44.9 42.1 Cognitive aging 20.4 21.7 20.7 22.5 19.6 Psychological development 12.9 15.8 14.6 14.9 20.9 Demography 22.1 30.6 36.3 34.7 28.2 Epidemiology 10.9 7.2 13.0 14.3 15.1 Health and retirement economics 9.5 10.2 11.4 14.7 17.1 Health and social institutions 11.8 2.6 3.6 4.1 3.4 Behavior genetics 0.3 0.6 1.3 1.8 1.8 Databases (e.g., Health and Retirement Survey) 9.6 12.1 12.9 10.3 11.3 TOTAL 129.4 145.7 157.8 162.3 159.5 NOTE: The organizational structure of BSR changed after FY 2000, so that comparability across the full-time period is only modest in some areas. Also, some projects, particularly the larger program projects, have become more interdisciplinary over time and harder to fit into these categories. NOTE: Totals do not add up to exact amount due to rounding off of figures. SOURCE: Behavioral and Social Research Program, National Institute on Aging.
OCR for page 24
A Strategy for Assessing Science: Behavioral and Social Research on Aging 1997 to 2003, from about $12 million in 1997 to about $36 million in 2003, but has since decreased, to about $28 million in 2005. Support for research in health and retirement economics fluctuated between $9 and $14 million per year between 1997 and 2003, and then increased to about $17 million in 2005. Support in other fields has had other historical records. Not shown in the table is the trend of funded research, particularly the larger program projects, toward becoming more interdisciplinary over time. This makes the classification of research support into fields somewhat problematic and implies that the time trends for particular research areas should be interpreted as approximations. The portfolio of research supported by BSR is diverse, in terms of both substantive focus and the ways the research is expected to generate scientific progress and improve the health and well-being of older people in America. This diversity reflects the inherent complexity of understanding behavioral, social, and economic processes associated with aging; the multiple ways in which the dynamics of the aging population of the United States affects individuals, families, communities, and public policies; and continuing debates within and across scientific fields about the causes and impacts of processes of aging and about public policy choices that BSR-sponsored science can inform. As noted in a recent compendium of studies on aging, health, and public policy, “We cannot plan for population change or design appropriate and effective responses without understanding, for example, the processes that underlie increases in longevity, the mechanisms that accelerate or delay the onset of disability, the incentives that affect retirement decisions, including employment and saving for retirement, and the role of public programs and policies in all of these factors” (Waite, 2004:4). A few examples taken from recent BSR reports of “scientific advances” illustrate the diversity of the topics on which BSR supports research as well as the variety of ways in which BSR support advances the mission of NIA “to improve the health and well-being of older Americans through research.” Understanding causes of longevity: Studies showing that late childbearing has positive effects on survival of the oldest old shed more light on the likely longevity of populations now in middle age (Zeng and Vaupel, 2003). Other studies are continuing to explore genetic factors responsible for longevity, using nonhuman models (e.g., Spencer and Promislow, 2005). Health care expenditures and health outcomes: Research documented the relationship of health status at age 70 to future health care expenditures and raised issues about the future of Medicare (Lubitz et al., 2003; Cutler, 2003). Research from a repeated survey was used to forecast the future nurs-
OCR for page 25
A Strategy for Assessing Science: Behavioral and Social Research on Aging ing home population (Lakdawalla et al., 2003). Other research documented the relationships of health care expenditures to health outcomes for older Americans and compared them in different regions (Fisher et al., 2003a, 2003b). The course and consequences of cognitive changes in aging: Studies have demonstrated that social activities, such as playing games and participating in volunteer work, slowed the rate of cognitive decline in people over age 70 (Glei et al., 2005). One study has shown that normal aging-related cognitive declines leave people susceptible to believing claims, such as those in advertisements, even if they are told the claims are false (Skurnick et al., 2005). Health disparities among social groups: Studies have explored possible explanations for the association between socioeconomic status and mortality and considered the policy implications (e.g., Wong et al., 2002; Adams et al., 2003; Adda et al., 2003; Deaton, 2002). Studies have quantified health disparities among the aging by gender, race, and ethnicity (e.g., McKenna et al., 2005), demonstrated effects of living in poor neighborhoods (Wen et al., 2005), shown that a higher educational level of children is associated with better outcomes of illnesses in their aging parents (Zimmer et al., 2002), and shown differences in primary care physicians’ treatment of aging women and men reporting symptoms of cardiovascular disease (Arber et al., 2006). Health effects of emotions: Studies have demonstrated and begun to explain the adverse health effects of negative affect and negative perceptions of aging (Rosenkranz et al., 2003; Levy, 2003). Other studies have begun to demonstrate explanatory physiological mechanisms, such as a strong association between loneliness and higher blood pressure (Hawkley et al., 2006; Boomsma, 2005) and an association of involvement by men in social networks and plasma fibrinogen concentrations (Loucks et al., 2005). New theories of the aging process: Research has developed and tested a new evolutionary theory of aging that offers an explanation of the factors that have contributed to low fertility rates and to the consequent aging of human populations (Lee, 2003). Declining rates of disability among older Americans: Studies demonstrating a declining rate of disability among older populations were analyzed for their implications for the future of Medicare (Freedman et al., 2002). Further studies have suggested that the increasing prevalence of obesity may reverse the decline in disability at about the time when a rapid increase is expected in the Medicare-eligible population (Chernew et al., 2005; Goldman et al., 2005; Lakdawalla et al., 2005; Olshansky et al., 2005; Preston, 2005; Reynolds et al., 2005). Personality changes in aging: Research has shown that age cohort
OCR for page 26
A Strategy for Assessing Science: Behavioral and Social Research on Aging is a major factor accounting for change during aging for some personality traits, such as neuroticism, which can affect health outcomes (Mroczek and Spiro, 2003). Effects of early-life stress on longevity: Research on a human cohort showed that health in early life, including exposure to serious infectious diseases, affects mortality in later life (Costa, 2003). Experiments with worms demonstrated that the expression of a specific gene mediated a positive effect of early-life stress on longevity (Rea et al., 2005). BSR-funded research focuses on many kinds of outcomes that are important to older Americans: longevity, health and disability, economic security, caregiving, cognitive functioning, and access to health care, among others. Findings from these studies are designed to help older Americans in various ways, from changing national policy to suggesting ways that everyday behaviors can improve well-being in later life. Methodologically, BSR supports research that varies from applied analyses of secondary data (e.g., modeling future nursing home populations from past data) to theoretical development of an evolutionary theory of aging. The applied research is often directed at near-term, targeted policy and program questions; the theoretical research contributes first to fuller understanding of the dynamics of aging and then to new or reformulated approaches to societal responses to aging. For this study, an important aspect of BSR-supported research is the great diversity of its disciplinary content. As is evident from Table 2-2, BSR supports research in a number of disciplines and fields, including psychology, demography, economics, sociology, and population epidemiology, among others. Increasingly, its support has gone to interdisciplinary research. One of many such examples is research drawing on concepts from sociology, social psychology, and physiology that demonstrated how the stress of low-status jobs could trigger metabolic mechanisms that predispose people to cardiovascular disease (Brunner et al., 2002). In this respect, BSR research support mirrors current trends in NIA and NIH, which increasingly reflect appreciation of the interdisciplinary structure of scientific inquiry and the contribution that an interdisciplinary orientation can make to the relevance and application of research findings to practice and policy (National Research Council, 2000a, b). BSR-sponsored activities have also led to the development of new interdisciplinary areas of research by catalyzing direct interaction among researchers working on similar issues from different disciplinary perspectives. For example, an April 1996 workshop that brought together demographers, evolutionary theorists, biologists, anthropologists, and others to share their understandings of human longevity led to a report (National Research Council, 1997) that named and help create the new interdisciplinary field of
OCR for page 27
A Strategy for Assessing Science: Behavioral and Social Research on Aging biodemography of aging (see Box 2-2 for more on this field). Developments in the new field led to calls for collecting social and biological data in the same survey instruments. A workshop in February 2000 brought together a broad interdisciplinary group, this time including economists, ethicists, pathologists, and others, to consider the potential and the risks of gathering biological data along with social survey data and to discuss what kinds of biological data would be most useful to social scientists studying processes of aging. The report of this workshop (National Research Council, 2001d) helped further advance interdisciplinary research linking the biological and social influences on processes of aging. In addition to supporting research that produces scientific results such as the above, BSR invests in the development of databases and measurement techniques that make these and other studies possible and in workshops and conferences that bring together researchers from different fields. These investments are designed to stimulate research and discussions in order to sharpen research questions in existing disciplinary fields, consider the applicability of new analytical methods and databases, explore and refine research questions at the interstices and overlaps among disciplines, and open up new fields for research. BSR supports the development and archiving of multidisciplinary databases, such as the Health and Retirement Study, the National Long Term Care Survey, the Wisconsin Longitudinal Study, and many others (for a list, see Behavioral and Social Research Program, 2004), that make it possible to conduct research that crosses disciplines. These databases allow exploration of important issues germane to NIA’s mission, such as the effects of socioeconomic status on health and the causes of declining disability among older Americans. BSR-funded research also provides useful knowledge for informing public policy decisions. For example, continuing research on declining physical disability and cognitive impairment among older Americans is relevant to policy decisions about the future of Social Security and of Medicaid, which pays for almost half of all nursing home costs. Estimates of the magnitude of decline are directly useful for anticipating the future costs to governments of caring for disabled older adults and thus to financial planning for Medicaid. Also, to the extent that disability decline is caused by events in people’s life experiences, research that identifies the causal factors may suggest small investments in well-being that will have large future payoffs in declining financial costs for health care and improved well-being of older adults and caregivers. BSR values research that bears on any of a broad range of aspects of health and well-being. Thus, the program applies disjunctive outcome criteria in making retrospective assessments of the value of the research it supports. It values both intellectual and practical results, but it does not expect that any particular study or area of research must produce both to
OCR for page 28
A Strategy for Assessing Science: Behavioral and Social Research on Aging BOX 2-2 Biodemography of Aging BSR currently supports work in the field of biodemography of aging through two program projects: one at the University of California, Davis, on the “Biodemography of Life Span” (e.g., Carey, 2003; Carey and Tuljapurkar, 2003), and one at Duke University on “Oldest Old Mortality” (Vaupel et al., 1998). It also supports a number of smaller research projects in this area. Biodemography, the confluence of biological and demographic studies of aging, has made important contributions in several areas. A significant contribution of BSR-supported work in this area is that it provides the central link between demographic and epidemiological studies on humans and detailed biological studies of other organisms. BSR has supported comparative empirical work on several species (fruit flies, Medflies, yeast, nematodes, and humans) that has demonstrated characteristic patterns of old-age mortality across species in which the mortality hazard becomes roughly constant at high ages. This is an important finding and influences much current work on mortality at the oldest ages (100 +) in humans. This finding has also focused attention on the biological and evolutionary determinants of mortality trajectories (Vaupel et al., 1998). BSR support has stimulated new and important work on the evolutionary basis of mortality patterns. Classical theories of senescence view mortality patterns as a result of a balance between deleterious mutations and selection that eliminates such mutation. Work by Lee (2003) shows that evolution of mortality in social species can be rather different—sociality in many species results in intergenerational transfers of time, food, and care (e.g., from women to their grandchildren), and these transfers can alter the strength of selection on old-age mortality by making older individuals more important contributors to fitness. Other recent work has examined the relationship between mutations and selection in shaping the age trajectory of mortality and has greatly be valuable. It also values efforts such as databases, workshops, and conferences, which provide lines of communication or infrastructure that can result indirectly in improved theory or applied results. In making prospective judgments about research, BSR is similarly eclectic and open to the many kinds of benefits that research can provide. DECISION-MAKING PROCESSES AT BSR With so many objectives and so many ways that social and behavioral research contributes to the understanding of processes of aging and the
OCR for page 29
A Strategy for Assessing Science: Behavioral and Social Research on Aging extended population genetic theory in this area (Steinsaltz et al., 2005). All these lines of research are shaping the search to understand the specific genetic and environmental determinants of mortality in humans. BSR has stimulated research on the heritability of longevity using twin register data. Researchers in the Duke project (Manton and Yashin, 2000; Christensen et al., 2001a, 2001b; Tan et al., 2004) have developed valuable methods for integrating genetic and demographic data to distinguish genetic from environmental influences on mortality. They have found that variance in longevity has a significant, but not a predominant, genetic component—not much larger than is typical for other quantified traits. This finding suggests that the key to understanding longevity does not lie in a few genes and implies that the analysis of mutation-selection dynamics in determining longevity is likely to have a high payoff. Curtsinger and colleagues (Pletcher and Curtsinger, 2000a, 2000b; Khazaeli and Curtsinger, 2001) have produced similar findings in Drosophila species and are now making progress at identifying the quantitative trait loci that are responsible for shaping differences in longevity within and between populations. BSR has supported detailed studies of how proximate factors, such as physiology of metabolic and reproductive energy use, dietary variability, and timing of reproductive and sexual activity, affect mortality in several species (Cargill et al., 2003; Carey, 2003; Carey et al., 2005; Vaupel et al., 2003). These studies have produced useful methods of analysis that are applicable to human studies, and the substantive results show how physiology and diet interact with reproductive metabolism to shape mortality. At this stage, these studies have raised as many questions as they answer, but they are key in designing future experiments and in providing mechanistic ways of studying such phenomena as the Barker effect—namely, that early life (fetal and infant) conditions can have significant effects on mortality in later life (Barker et al., 2002). well-being of older Americans, BSR is routinely faced with the tasks of allocating money efficiently within and across different research fields and among research applications. Changes in the national conditions of older Americans, government priorities, and science also require that BSR continually reassess these allocations. The tasks become more challenging when research funds tighten in relation to the number of proposals being received and the number of researchers working in the fields in which BSR supports research. This has been the case in BSR and NIA more generally since 2003, as it has been across NIH (Mandel and Vessel, 2006). The success rate of applications in BSR, that is, the percentage of applications reviewed that are
OCR for page 30
A Strategy for Assessing Science: Behavioral and Social Research on Aging actually funded, has declined from 30.7 percent in FY 2001 to 17.1 percent in FY 2005. There have been similar declines in other units in NIA (National Institute on Aging, 2005). Concern with Relative Quality Ferment exists in several federal science agencies about the adequacy of existing priority setting and proposal selection procedures for selecting high-payoff, transformative, and high-quality research proposals. Expressed with varying degrees of explicitness, concerns about undue conservatism, resistance to interdisciplinarity, and the existence of unacceptably low field-specific or study group–specific standards of quality inherently reflect criticisms of the discipline-based peer review system. This report responds to the specific concerns of the director of BSR regarding the adequacy of the program’s current decision-making processes for assessing research proposals across fields and for assessing its overall research portfolio. One concern affects the assessment of research proposals, specifically the possibility that different standards of research quality are being applied to proposals in different fields, some of them “higher” than others. Imagine a scale of height in which some people measure in inches and others in centimeters, without anyone knowing what units others are using, and in which items are compared by the numbers without regard to the units. If that is an apt analogy for the rating processes used at BSR (or elsewhere in NIH), in a community-dominated decision process, proposals of high quality in high-standard (inch) fields would be denied support while lower quality proposals in low-standard (centimeter) fields would be funded. To put this idea in the language of ranking, the concern is that proposals judged to be in the top 10 percent in quality in one field might have much lower ranking according to the standards of another field.1 A related concern operates at the level of portfolio decisions. To the extent that scientific advisory groups contain people who apply different standards to research, group consensus processes may be counterproductive, perhaps by leading to support for the least objectionable proposals or to schisms within advisory groups that can have unpredictable effects on their advice. If advisory group members with different standards trust each other in their narrower areas of expertise, the group’s advice would be to support more research in the weakest areas than would be warranted on an objective quality standard. If members do not trust each other, they would have no way to reach consensus advice on the recommendations to offer to program managers (Brenneis, 1994). One result might be that a group offers consensus advice based on tacit agreement to trust each expert’s judgment in his or her own area, while individual advisory group members grumble in private that the group has endorsed supporting weak research.
OCR for page 31
A Strategy for Assessing Science: Behavioral and Social Research on Aging Some researchers, including leading researchers, have expressed discontent in this regard about existing study sections and review panels in both NIH and the National Science Foundation (NSF). The challenge for science assessment is to determine whether such expressions derive from the kind of problem described here, or instead reflect the idiosyncratic, parochial, or selective disciplinary interest. As expressed here, these concerns presume that there is a single quality metric that can be applied to research across scientific fields: that differences in standards are differences in the ways a single underlying dimension of quality is being assessed, not differences in the kinds of things being assessed or in judgments of the relative importance of different objectives (such as scientific quality and potential societal benefit) that are supposed to be combined in making assessments. The above discussion of BSR-funded research suggests that the program does not in fact apply a single metric in assessing the results of the research it supports. At least in terms of the ways that research contributes to the “health and well-being of older Americans,” several modes of contribution seem almost equally important. In this respect, BSR appears to apply disjunctive criteria of value, such that a research activity may be judged as having resulted in an advance or a discovery if it satisfies any one or two of a relatively long list of implicit criteria. Our collective experience with BSR, coupled with the expressed concerns of program managers and leading researchers, lead us to take seriously the possibility that existing priority setting and proposal selection processes may not be doing an adequate job of ensuring that the program supports research of a uniformly high “quality” across the fields the program supports. We take these concerns seriously even though it seems clear that quality is not being judged and should not be judged on a single dimension. Addressing concerns about relative quality requires developing a more nuanced idea of the dimensions of quality that might appropriately be used to evaluate research portfolios. Here we suggest a few dimensions of research quality that might apply in BSR and consider which of them might provide cause for concern. In Chapters 3 and 4, we address knowledge about the progress of science and ways of assessing the outputs and potential value of science when criteria are multidimensional. Dimensions of Quality For a research sponsor such as NIA, various dimensions of research quality might be used for prospective or retrospective assessment. We briefly note a few. Conceptual and methodological quality: Scientific research is often
OCR for page 32
A Strategy for Assessing Science: Behavioral and Social Research on Aging judged by how clearly its concepts are defined, how reliably they can be measured, and how well the operational measures of variables and concepts correspond to the underlying concepts themselves. Scientific progress sometimes consists of developing clearer concepts when they had been ambiguous, better measures when they had been unreliable, or better evidence that a measure is a valid indicator of a construct of interest. Thus, one appropriate way of judging the quality of research is methodologically in the terms of the research itself: research passes a quality test if it uses clear concepts and reliable and valid measures. It passes another quality test if it moves scientific discussions in its field toward using clearer concepts or better measures. Advancing thinking: Scientific progress is often defined in terms of efforts to develop and validate theories or conceptual models that attempt to make sense of whole classes of phenomena and thus provide explanations of some generality. If a research activity develops a new theory to explain phenomena that were previously thought to be unrelated, or develops a new way to test a theory, or produces results that call an existing theory into question, its quality may appropriately be judged higher than a research activity that simply replicates a previous finding or is presented in a way that is unrelated by theory to any wider set of phenomena. Of course, the fact that a piece of research calls previous findings into question does not imply that it meets other quality criteria. Generating widely applicable research findings: Scientific research is also judged by the extent to which its results stimulate researchers, particularly outside its immediate field, to carry out new research. For example, the development of game theory as an analytic method has led to a variety of applications to problems in economics, social psychology, international relations, and other fields. BSR-sponsored research linking social and emotional factors to illness and longevity (e.g., Brunner et al., 2002; Adams et al., 2003; Rosenkranz et al., 2003) may stimulate various lines of research to elucidate the mechanisms explaining these relationships, with possible applications to more effective health promotion. Research can generate widespread interest in various ways, including developing new lines of theory, raising new research questions, calling into question widely accepted societal beliefs, and developing new ways to study understudied phenomena. Even research that seems at first to be of interest to only a few academics may later become the foundation stones of far-reaching theoretical and applied advances. Generally, though, the more widely cited research findings are, not only within but across disparate fields of investigation, the more important those findings tend to be. Practical application: Science that is supported for its potential benefits for health and well-being, as at NIH, can also be judged on the basis of its actual or potential practical application—its “broader impacts,” in
OCR for page 33
A Strategy for Assessing Science: Behavioral and Social Research on Aging the language of the NSF merit review criteria (available: http://www.nsf.gov/pubs/1999/nsf99172/nsf99172.htm). Research can be appropriately judged to be more important on this dimension from an NIH perspective if it produces (or can reasonably be expected to produce) results, methods, techniques, or other outputs that advance the sponsor’s ultimate societal mission. Research that is unlikely to produce such results can be considered to be less important on this dimension. As already noted, a wide variety of possible practical applications are of interest to NIA. Research may be rated highly in terms of practical application if it can be applied to any important aspect of health or well-being and if the applications are likely to affect the health or well-being of large numbers of people. Considering the variety of dimensions on which research quality may be judged, it is reasonable to ask which dimensions are of greatest concern to BSR in making comparisons of research in different fields. This is, of course, a policy question for BSR and NIA. Comparative assessments of fields will be more responsive to the sponsor’s needs if BSR can be specific about the objectives it deems most important to consider in assessing the relative contributions of research in different fields. With clarification of the objectives, it becomes more possible for BSR to identify possible metrics for the objectives or, when there are no generally agreed-on measures, to move expert deliberations toward making more explicit comparisons along those dimensions. Comparisons across research fields should consider the quality and importance of research that has been supported in relation to the levels of support that have been made available for research in the fields. As can be seen from Table 2-2, a considerable range exists in funding levels among BSR’s branches, from less than $2 million in 2005 for research in behavior genetics to about $42 million for research in behavioral medicine. RESEARCH REVIEW IN NIH NIH research review practices set a context for any effort to reconsider the practices for setting research priorities. They do this in at least two ways: they affect the pace at which review and advisory panels and their decision rules can change, and they provide opportunities and constraints for the discretion of research managers. Funding and Peer Review Extramural funding at NIH is made primarily through grants and contracts. Contracts are used when the agency defines the work to be done and intends to use the product of the contract. Grants are based on the
OCR for page 34
A Strategy for Assessing Science: Behavioral and Social Research on Aging premise that there is a shared interest in the outcomes of the research. A third mode, cooperative agreements, involves grants in which there is an understanding that the federal sponsor will be involved in a predetermined way in the work of the grant. Review of research proposals at NIH follows different procedures depending on the funding instrument and on whether the proposals are solicited or unsolicited. The largest component of most programs, including BSR, consists of projects funded through grants as a result of unsolicited, investigator-initiated submissions. Researchers submit applications to NIH, and they are assigned by the Center for Scientific Review (CSR) to a relevant institute or to multiple institutes for administration. Within the institute(s) they are assigned to programmatic areas, such as BSR. These assignments are based on referral guidelines, updated periodically, in which institutes and programs have outlined their scope of interests. These grant applications are also assigned to review groups, almost always established study sections, for merit review. Study sections cover a defined range of substance that may fall within the purview of a single institute or center or cut across many. Members of review groups are proposed by the scientific review administrator in CSR and approved by NIH. Membership is typically for four years, and therefore the composition changes by about 25 percent each year. In addition to regular members, individuals may serve on an ad hoc basis for a meeting or two. This is done to ensure there is a specific skill needed for the review of some application, or a way to try out an individual before nomination for regular membership. Members whose terms are expiring may be asked to suggest possible members to fill their roles, a practice that could lend considerable stability to study sections. However, review groups do change over time as they respond to the changes in the nature of the applications being referred to NIH: if there is growth in a field or area, more reviewers are added to address that need; if applications decline, reviewers in that area become less prevalent on the committee. In its selection of members of peer review groups, NIH acknowledges the person’s stature in a field and its confidence that the individual can provide useful assessments of scientific merit. The members of a group may be called on to review a variety of projects in different fields, so groups are constituted to bring relevant skills to bear on all applications. Applicants do not know which reviewers were specifically assigned to their applications. Panels make two types of recommendations: first, whether a proposal meets threshold levels of scientific quality and conforms to other eligibility criteria; second, assessments of scientific merit, expressed in terms of a numerical “priority score.”2 Proposals are placed in a rank order based on their priority scores and are normally funded in that order until all or a predetermined percentage of the program’s requested budget is committed. The recommendations of peer review are presented to the relevant
OCR for page 35
A Strategy for Assessing Science: Behavioral and Social Research on Aging institute’s advisory council. The councils have the right to submit their own recommendations. They can recommend against funding projects with higher priority scores and in favor of funding projects that have received less favorable scores on the basis of judgments about program relevance. The institute in turn acts, informed by those recommendations. If there is a question about error in a particular review, the applicant can ask that the program manager consider the applicant’s concerns. If the program manager cannot resolve concerns, they can be presented to the advisory council. Although program management has discretionary authority to forward recommendations that differ from those of the study sections, NIH has built a reputation for adherence to recommendations of scientific peer review. By reputation, program managers infrequently seek to make decisions that deviate from the assessments of the study section. It is not unusual to find that the first 80 percent of the available funds are allocated according to the score, but that for the remaining 20 percent, program manager judgment is factored into the decision whether or not to fund. The typical practice in NIH institutes is to require a strong rationale for any decision by a program manager to fund a project that did not score well or not to fund a project with a high review score. In the latter case, for example, a conclusion that a project duplicates one already being supported provides a sufficiently strong rationale. NIH program managers are generally seen to exercise less discretionary authority relative to peer reviewers than their counterparts at NSF, and considerably less than their counterparts at the Department of Defense’s Defense Advanced Research Projects Agency (DARPA), where program managers, even though they consult widely with experts in a field, are generally viewed as having wide latitude to select from among the proposals they receive. NIH periodically reviews the structure of review groups and adjusts their scope and the expertise needed. The panel on scientific boundaries for CSR began in 1998, and updates are posted on the CSR web site (http://www.csr.nih.gov/review/irgdesc.htm). The one-quarter annual turnover of review groups could possibly act as a brake on change. Review panels might not quickly reflect changing interests and priorities in the scientific community, and they might continue to provide a favorable home to areas of science after research productivity has peaked. Also, a review group is likely to have few reviewers in any new field. Thus, as is implicit in NIH’s articulated rationale for establishing its new Roadmap and for forming new panels to review proposals submitted under this initiative, investigators in a new field may find they are being reviewed by people who do not know the field well. Changes in the review process can be achieved by revising criteria for scientific merit. For example, in 1997, the criterion of “innovation” was
OCR for page 36
A Strategy for Assessing Science: Behavioral and Social Research on Aging added to help respond to the concern that review groups had become too conservative (see Notice 97-010 and NIH OD-05-002). Reviewers are required to address how a proposal is or is not innovative, although it is recognized that some meritorious projects may not be innovative. Specific review criteria may be issued for a Request for Applications or Program Announcement solicitation, and special criteria may also be established for certain types of research, such as that involving the preparation of natural products (see Center for Scientific Review notice, 23 July, 2003) or the use of human subjects, children, animals, etc. Discretion of Research Managers As already noted, mainstream practice in NIH is for program managers to adhere closely to the recommendations of study sections or ad hoc peer review groups regarding the funding of unsolicited proposals that have been received. Doing otherwise risks raising objections that such actions constitute an erosion of support for peer review. Program managers, however, have more discretion outside the realm of unsolicited proposals. Acting as scientific entrepreneurs, program managers can encourage research in new fields by soliciting proposals both from existing cadres of researchers and by seeking to enlist the interest of researchers who traditionally have not sought support from NIH. Building a new field of science and building a new community of researchers thus becomes a joint, mutually reinforcing process. An institute has the greatest discretion in contracting for work that meets very specific needs. It may issue a Request for Proposals (RFP), in which it can specify in advance what it wants to have done and whether the product of the contract is for the institute’s use. For example, an institute or program can take the product of a contract, such as a data set, and make it available to others to analyze. Contract proposals are typically reviewed by review groups specially created for the purpose and managed by the institute or center rather than by CSR. Program managers may also shape their research portfolios through Requests for Applications (RFAs) that define a specific problem, goals for the research, and sometimes specifics of methods to be used, such as reanalysis of available data or collection of new data, while allowing latitude in how researchers address the problem. RFAs may include specific review criteria and may also include a set-aside of funds, making clear the institute’s commitment to the area. Most RFAs are reviewed by a special review group that encompasses just the applications submitted in response to the RFA. The review may be managed by CSR or by the institute’s review office. Plans for RFAs must be reviewed and approved by an institute’s advisory council as part of the institute’s program plan. They often result from specific consultation with the scientific community. In some cases, an external review
OCR for page 37
A Strategy for Assessing Science: Behavioral and Social Research on Aging of a field that identifies research needs may be the basis for an RFA (e.g., a National Research Council report). Nevertheless, program managers have latitude in drafting the RFA and in contracting for reviews of particular areas that might lead to subsequent RFAs. Institutes and centers may also issue Program Announcement (PAs) that remind the scientific community of continuing interest in an area. PAs must be reviewed after three years and either dropped or reissued. PAs may involve a set-aside of funds and may involve a special review, but such special treatment is not necessary. In many cases the responses to a PA are reviewed by the standing review group to which that application would have been reviewed had there been no PA, and funded according to the overall funding plan. If there is a special review, it can be managed by CSR or the center or institute that made the announcement. Thus, the PA and the RFA are two powerful and efficient ways in addition to contract research that NIH communicates about priorities and interests with the scientific community. All solicitations are published on the NIH web site, and there is a weekly email listing of all announcements issued that week. RFAs and PAs identify the funding mechanisms to which they apply. They may be restricted to small grants to individual investigators. They may use a mechanism such as the R21 (exploratory/development grants) to show interest in eliciting innovative projects. R21 awards are usually given to researchers who have little or no preliminary data. They have upper limits on length of award and funding, but they provide sufficient funds to get an investigator started on a new line of inquiry. BSR/NIA has announced that it will accept R21 applications in all seven of its major emphasis areas, identifying 18 specific topics of interest (http://www.nia.nih.gov/ResearchInformation/ExtramuralPrograms/BehavioralAndSocialResearch/R21Grants.htm). This form of communication with the scientific community of the office’s interest in supporting innovative work on certain topics is less formal than the issuance of an RFA or a PA. An RFA may solicit proposals for large grants such as program projects (P01) or centers (P30, P50), which provide core support or help institutions bring together elaborate teams of people to address a problem. Large grants serve to focus an institution’s activity around a research question; the downside to such awards is the possibility that they create vested interests in continuing the support in a recipient institution, sometimes backed by a member of Congress. Thus, it may be difficult or painful for an institute to terminate support for such projects. (Procedures and criteria for evaluating centers and related large awards are reviewed in Institute of Medicine, 2004). The above discussion illustrates the various means that programs and program managers in NIH can use to grow specific program areas, to shape
OCR for page 38
A Strategy for Assessing Science: Behavioral and Social Research on Aging a research portfolio, and to provide support to areas that are seen as needing additional attention. In addition, research managers may use formal announcements as a way to bring relevant work together for direct competition rather than having projects come in at different times. BSR has used the full range of options available for identifying new and promising research areas and for encouraging scientists to develop those lines of research. NIH does not offer many tools to its research managers to exercise discretion in identifying research areas that are not, or are no longer, programmatic priorities. In a tight budgetary context, an institute may take “negotiated” reductions in grants and, in doing so, may make greater reductions in grants in substantive areas in which there is less interest or perceived payoff. Some institutes and centers give program managers latitude in making these judgments. For example, an institute or center might declare that, unless there are specific programmatic or policy reasons, applications will be paid up to 85-90 percent of the funds available. The remaining funds may be allocated to meet programmatic goals without special action of the advisory council. This strategy is typically used when the grants involved are close to indistinguishable in quality according to reviewers’ judgments. Another discretionary tool is that program managers can elect to decline to have a project assigned to the institute or center if it is over $500,000 in direct costs and does not fit their program priorities. Institutes, of course, can discourage proposals in an area by simply reducing the amount of funding they provide to it and by publicizing the low absolute and relative success rates of proposals in the area. An institute may also communicate a desire to spend less in a given area by omission: it can publish areas of interest and leave some areas off the list. Expressions of interest (or disinterest) tend to circulate quickly in the scientific community and may influence decisions about research to propose. However, these processes may produce mixed signals and misunderstandings between program officers and researchers about whether it is fields of study that are being deemphasized or specific proposals. In general, though, it is not easy or popular to declare an area of inquiry to be of low interest, so a research manager needs a solid basis and organizational support for such a declaration. NOTES 1. This concern may play out differently in different review processes. In NIH study sections, members normally come from several disciplines, even if they have in common concern and expertise in a particular research area or field. If researchers in different disciplines have different standards in these terms, the differences in standards are unlikely to perturb the overall decision process unless panel members habitually defer to the judgments of panel members rating proposals from their own disciplines.
OCR for page 39
A Strategy for Assessing Science: Behavioral and Social Research on Aging 2. Reviewers in NIH study sections rate proposals that are of acceptable quality on a scale from 1.0 (outstanding) to 5.0 (acceptable). The average rating is multiplied by 100 and is called the priority score. Priority scores are ranked within the review group to allow the proposals to be prioritized. Based on its available funding, an institute decides what percentage of proposals can be supported and establishes a “payline,” which is a percentile number such that proposals ranked at that percentile or higher among all those reviewed are normally funded. For continuing review groups, percentiles are normally calculated on the basis of the past three rounds of ratings. Funds are normally granted in percentile order, with funding going to proposals scoring above the payline regardless of which study section conducted the review. One effect of this procedure is that study sections are normalized to account for any systematic differences among them in how members rate proposals.
Representative terms from entire chapter: