Decadal Science Strategy Surveys: Report of a Workshop (2007)

The opening afternoon, Session 1, of the workshop featured three panels moderated by Space Studies Board (SSB) chair Lennard Fisk. Panelists and others discussed five recent National Research Council (NRC) studies from three perspectives: those of the science community, federal agencies, and the Congress. In the Survey Committee Panel, the chairs of the committees that wrote Astronomy and Astrophysics in the New Millennium; Connecting Quarks with the Cosmos: Eleven Science Questions for the New Century; Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond; New Frontiers in the Solar System: An Integrated Exploration Strategy; and The Sun to the Earth—and Beyond: A Decadal Research Strategy in Solar and Space Physics summarized their studies from the perspective of approaches employed, impacts, problems, and recommendations for future surveys. An agency panel and a congressional panel provided opportunities for speakers from the National Aeronautics and Space Administration (NASA), the National Oceanic and Atmospheric Administration (NOAA), the National Science Foundation (NSF), the Office of Science and Technology Policy (OSTP), and Congress to address the utility, impacts, limitations, and problems of past surveys from the perspectives of the key government audiences they represented.

The first member of the Survey Committee Panel, Christopher McKee, of the University of California at Berkeley, provided an overview of the most recent

decadal survey for astronomy and astrophysics.¹ As co-chair of the survey committee that wrote the report, he said that the 15 members had been chosen to maximize disciplinary expertise and to reflect institutional perspectives and noted that the survey had drawn on the work of nine expert panels and four cross-panel working groups that involved an additional 96 participants. To obtain community-wide input, the survey committee held two town-hall sessions at American Astronomical Society (AAS) meetings plus more than 20 meetings at individual institutions, hosted a Web site soliciting input, and devoted a significant portion of one committee meeting to gathering views from international astronomers.

Dr. McKee said that one notable aspect of the study, which was identified as an important aspect in later workshop discussions, was that although it had had interest and funding support from two of the three major federal sponsors for astronomy in the United States—NASA and NSF—the Department of Energy (DOE) did not participate. He said that the impact of the survey report on federal actions was mixed. At NSF, three major recommended telescope facilities were all making good design progress, and NSF had acted positively on some of the survey’s key policy recommendations. On the other hand, the prospects for funding construction of recommended facilities at the NSF remained uncertain. Similarly, some highly recommended NASA space missions were making excellent progress, but a larger number of other highly ranked initiatives were on hold. For a complete assessment of the impacts of the deferrals of recommended missions, see A Performance Assessment of NASA’s Astrophysics Program.²

Dr. McKee indicated that unforeseen developments, including postsurvey budget constraints, cost growth in recommended missions, and the Columbia disaster, which delayed the refurbishment of the Hubble Space Telescope by several years and incurred many hundreds of millions of dollars in additional costs, impaired NASA’s ability to act on the survey report recommendations.

In looking back at the astronomy and astrophysics survey, Dr. McKee saw three important lessons. First, he suggested that future surveys should be broadened even more than in the past. For example, the interfaces between fundamental physics and cosmology (see discussion of Connecting Quarks with the Cosmos, below) should be integrated into the survey more fully. He also recommended expanded outreach efforts to include interaction with the American Physical Society as well as the AAS and even more town meetings. Second, Dr. McKee called for improved cost estimates for high-priority initiatives if at all possible, along with a uniform approach to estimating costs for all mission and facility candidates. He noted that current estimates for some major projects had reached 1.5 to

3 times the survey estimates.³ Third, he emphasized that survey committees need to consider full life-cycle costs, including costs for both facility development and operations, so that useful comparisons could be made between candidates for new initiatives and initiatives recommended in prior surveys.

Dr. McKee concluded by offering the following recommendations for future assessments of astronomy and astrophysics projects:

• Get broad expertise on the survey committee.

• Continue to make the hard decisions in setting priorities.

• Do better on highlighting the science—Astronomy and Astrophysics for the New Millennium was science-based.

• Secure DOE support and add panels related to DOE astrophysics.

• Reprioritize previously recommended projects that have not started.

• Study operating costs of ground-based facilities, which are very long-lived. Operating costs far exceed construction costs over the operational lifetime of a facility.

• Consider prioritizing the Explorer program, provided that launch vehicles are available.

• Be prepared to follow up for a decade. Rearrange the membership of the NRC Committee on Astronomy and Astrophysics to include a significant fraction of the survey committee. Doing so will allow changing priorities if costs increase greatly, but only as a last resort since this could undermine the process.

Michael Belton, of Belton Space Exploration Initiatives, LLC, astronomer emeritus at the National Optical Astronomy Observatories and chair of the steering committee for the solar system exploration survey, discussed experiences for that study. The survey report⁴ was the product of five expert panels, which prepared the scientific material, and a 15-person steering committee, which drew on the panel inputs to create a set of integrated priorities. In addition to addressing NASA explicitly, the survey indirectly had recommendations for NSF and DOE. Community input and involvement had been ensured by means of multiple town-hall-style meetings that involved committee and panel members, contacts with relevant professional societies, and a collection of white papers solicited at the initiative of the AAS Division of Planetary Sciences.

In discussing the impacts of the survey on solar system exploration, Dr. Belton noted that members of the community seemed to be very satisfied with the outcome and more unified as a community. He added that both NASA and the Office

of Management and Budget (OMB) seemed satisfied with the results, which provided a scientific basis for a number of programmatic and budgetary decisions. One exception to NASA’s positive reaction was the report’s recommendation that NASA contribute to NSF’s support of a major ground-based telescope, the Large Synoptic Survey Telescope.

Dr. Belton indicated that the survey had faced a number of problems, including the short time available to complete the study; uncertainty surrounding the costs of the candidate missions and their readiness to be implemented; rapid, ongoing changes in NASA’s senior staff, organization, and goals; and the difficulty of coping with new political initiatives that had uncertain lifetimes (e.g., the introduction of lunar exploration objectives into the 2004 Vision for Space Exploration). However, he noted that the most damaging problem for the survey had been the tendency for the community, and especially NASA’s follow-up advisory committees, to diverge from the recommended survey goals and to recreate strategies in response to new scientific discoveries and inevitable new start problems, budgetary crises, and new political initiatives.

Dr. Belton offered the following recommendations for future surveys:

• Start survey preparations early, including early identification of the steering committee and preliminary mission cost assessment work at NASA laboratories. At least 2 years should be allocated to conducting a survey.

• Assure adequate funds for the survey itself and also provide support for peripheral technical and mission studies thought necessary by the steering committee for public outreach and for advocacy.

• Because the political climate does matter, finalize the strategy of a survey in the first year of a new or incumbent administration. This suggests that decadal surveys should take place every 8 or 12 years to match the political cycles.

• Initiate a one-solar-system exploration survey that includes the science of Mars and the Moon with the rest of solar system science. This will ensure that resources applied to solar system exploration are used most effectively.

• Establish a formal follow-up connection between NASA (and possibly other agencies) and the NRC to ensure that proper focus is maintained on contemporary survey goals in light of new discoveries and personnel, budgetary, and political changes. Perhaps NRC might consider utilizing the survey steering committee on an ad hoc basis throughout the decade following a survey instead of disbanding it. Given the substantial community input to surveys, agencies should not reinvent or redefine the survey.

• Create explicit interfaces to the international community, the human spaceflight program, and the extrasolar-planet community. This is not difficult in principle but will require time and money.

Louis J. Lanzerotti, chair of the Solar and Space Physics Decadal Survey Committee and research professor at the New Jersey Institute of Technology,

discussed his committee’s study,⁵ which was organized under the auspices of the Committee on Solar and Space Physics (CSSP) of the SSB and which had sponsorship from NASA, NOAA, NSF, the Air Force Office of Scientific Research, and the Office of Naval Research. Like its counterpart surveys summarized above, Dr. Lanzerotti’s survey committee held numerous town meetings and then integrated input from six expert panels to prepare overall recommendations for program goals and priorities. It differed in one important way, however, in that its charge included addressing relevant operational programs of NOAA, the Department of Defense (DOD), and DOE, as well as the basic scientific research programs of NASA and NSF. To facilitate attention to this aspect of the survey, the committee had held a special transition-to-operations workshop on space weather. In prioritizing new initiatives, the committee applied criteria that included scientific merit, contribution to national goals, technological readiness, adequacy of the theoretical foundation, temporal synergies with other missions or facilities, and cost.

Dr. Lanzerotti summarized lessons learned from The Sun to the Earth—and Beyond as follows:

• A strong chair of the survey’s oversight committee (in this case, the CSSP) is essential.

• NRC staff involvement and knowledge are essential.

• Because the importance of applications in this field was recognized from the outset, relevant agencies and outside communities (e.g., industry) were invited to participate.

• Panel reports were subjected to NRC standard review procedures to provide extra credibility.

• Continued follow-up by the survey’s oversight committee after survey delivery is critical; otherwise participants who are involved in the survey do not know how or if their recommendations are being acted on.

• Without follow-up, it will not be known for how long an individual agency will pay attention to the survey.

• Early buy-in by stakeholder agencies and their continued involvement are critical if the survey is to provide the most credible advice to the nation.

• Perhaps the greatest weakness is credible cost estimation, even when it is recognized as a weakness and addressed from the outset.

• One legacy program from Astronomy and Astrophysics in the New Millennium was treated, incorrectly as it turns out, by assuming it would be carried out.

• The next survey of solar and space physics will probably need to place even more emphasis on addressing applications.

Berrien Moore of the University of New Hampshire discussed another survey that has a strong applications element as well as fundamental science aspects—Earth Science and Applications from Space. The survey committee, for which Dr. Moore served as co-chair, completed its interim report in April 2005,⁶ but the final report was not ready in time for the workshop.⁷ The survey, which had sponsorship from NASA, NOAA, and the U.S. Geological Survey, was organized around seven thematic panels and an executive committee made up of 11 at-large members plus the chairs of the seven panels.

Dr. Moore noted that an interim report had not been part of the original study plan but was added when it was seen to be critical, because, to quote the report (p. 2), “Today, this system of environmental satellites is at risk of collapse.” He added that since the interim report, the situation confronting the survey committee had deteriorated further owing to the major problems of cost, schedule, and scope affecting the DOD/NOAA/NASA National Polar Orbiting Environmental Satellite System and the termination or delay of still other satellite programs at NASA. Other problems facing the survey committee had included the overly broad and open-ended charge, which was to review the status of the field; overlapping or ambiguous agency roles; and obtaining and verifying mission cost estimates. Dr. Moore also noted that the way the expert panels had been set up posed a challenge for the survey. By using a thematic approach (e.g., there were panels on weather, climate, and human health and security) instead of a more traditional disciplinary approach (e.g., atmospheric science, oceanography, and glaciology), the study risked appearing to disenfranchise some disciplines and favor others even if that had not been the intent of the organizers.

Even though the survey on Earth science and applications from space was not yet final, Dr. Moore drew on the experience of his survey committee and panels to offer a slate of lessons learned:

• “Building trust” in the community was a nontrivial task; this may not be an issue in the future, but it taught the survey committee a painful lesson.

• “Reviewing the status of the field(s)” is too open-ended an assignment; that aspect of the charge needs to be more carefully framed for Earth science.

• Agency roles need to be handled better, but how this can be done is problematic.

• Cost estimates need to come from outside the survey committee. NASA and the Aerospace Corporation are candidates for doing this, but whether they can be really independent is uncertain.

• Given that an interim report was added to the list of tasks, the schedule was unrealistic.

The fifth and final speaker on behalf of recent survey studies was Michael S. Turner, chair of the Committee on the Physics of the Universe, professor at the University of Chicago, and former NSF assistant director for mathematical and physical sciences. The committee prepared an interim report in January 2001 and a full report, Connecting Quarks with the Cosmos: Eleven Science Questions for the New Century,⁸ in April 2002. Neither was strictly a decadal survey, but they had similar impacts. Dr. Turner explained that the study dealt with science at the boundaries between physics and astronomy and therefore cut across both disciplines and across multiple agencies (DOE, NASA, and NSF). For that reason it was a study about exciting fundamental science (dark matter, solar neutrinos, etc.) that could all too easily fall through the cracks. Dr. Turner said that a key aspect of the study was that it framed the key science questions first and then formulated programmatic recommendations that were always tied to the science.

Unlike the other classical decadal surveys, Connecting Quarks with the Cosmos did not produce a strictly ordered list of priorities; rather, it presented a strategic package of projects to achieve the science goals. The study resulted in a federal plan of action put together by OSTP.⁹ Dr. Turner indicated that many of the report’s recommendations were receiving positive treatment and making progress, and that all of the recommendations were at least in the planning stages at all three agencies. Unfortunately, however, constrained science budgets have limited progress in implementing the recommendations of this study as well as all the other studies. He noted that the committee’s science-first approach had helped create an atmosphere of excitement and that the effort to foster interdisciplinary and interagency cooperation was bearing fruit. While Connecting Quarks with the Cosmos served to reinforce the astronomy and astrophysics decadal survey, it did add a bit of confusion because it created, in effect, a separate set of priorities. Nevertheless, because the recommendations did not contradict or change the astronomy and astrophysics survey priorities and because it added a new dimension of scientific excitement, the net effect of the survey was positive.

Dr. Turner concluded with three lessons learned:

• The science-first format is well suited for discovery science.

  • It captures the excitement of the enterprise,

  • It is the metric by which OSTP and OMB are now judging the discovery science fields,

• It provides a reference point for planning, measuring progress, and responding to a changing fiscal climate, and

• It is an effective mechanism for bridging disciplines and agencies.

• Interdisciplinary and interagency approaches characterize today’s science, and since they are not going away, they must be accommodated.

• Texture and a small number of recommendations are as important as ordered lists in a report.

The Panel on Agencies provided an opportunity for representatives from four executive branch agencies to comment on how their agencies had received recent survey reports. Mary Cleave, NASA’s associate administrator for science, opened the discussion. She began by noting that the surveys had played a number of roles: They had facilitated NASA–community communications and consensus development, securing broad expertise and independence, authoritatively documenting recommendations, and maintaining unity among the scientific community and other stakeholders. She cited NASA impressions about particular strengths and weaknesses of the four recent reports that had been discussed by Drs. McKee, Belton, Lanzerotti, and Moore and also shared some NASA recommendations for the future. Among the strengths of the surveys were the inclusion of explicit, clear priorities (or, in the case of the Earth science and applications interim report, issues rather than priorities) and the comprehensive coverage provided by the studies. Weaknesses included the surveys’ tendency to gloss over where to rank initiatives recommended in earlier surveys, the cost implications of (some) recommended missions, and relative priorities for small missions (e.g., Explorers) and the core research programs.

In looking toward the future, Dr. Cleave offered the following recommendations for the next decadal surveys:

• Focus on science questions and objectives, not specific missions. If missions are identified, NASA and the NRC should work together on approaches to find mission cost boundaries that fit within the budget.

• Consider how objectives fit together to answer science questions.

• Consider how to know when an objective has been met.

• Provide clearer definitions of mission classes and of which missions fit within those classes.

In discussing the NASA presentation, workshop participants called attention to the fundamental conflict between having the surveys provide prioritized sets of key scientific objectives (which participants seemed to agree was important) and being able to estimate the costs of pursuing those objectives. As one participant put it, “How does one [put a] price [on] an objective?” The challenge of translat-

ing a science-based survey into useful and actionable recommendations that are credible and realistic from the standpoint of a budget became one of the recurring themes of the workshop.

Judith Sunley, the NSF executive officer for mathematical and physical sciences, provided comments from the perspective of the NSF. She said that the decadal surveys, especially their treatment of priorities, had been very valuable for the NSF. They help the NSF articulate the scientific arguments and gain community support for the programs and projects it is considering and help make trade-off choices between small and large programs. She noted her agreement with the science-first approach to the surveys, which had been mentioned earlier by Dr. Turner and others. In looking forward to future decadal surveys Dr. Sunley offered the following recommendations:

• The costing for proposed initiatives needs to be done in a reasonably consistent way so that the costs of current and recommended future programs or facilities can be compared.

• Cost estimates need to include full life-cycle costs, including those of predevelopment definition and design, construction, operations, and decommissioning.

• In addition to recommending construction of new facilities, surveys should be prepared to consider what existing facilities might be phased out to ensure robust support for new initiatives.

• Surveys should look for opportunities for international partnerships, which can help make development, construction, and operations costs more affordable to the United States. Workshop participants noted, and Dr. Sunley agreed, that the impact of international cooperation on such costs needs to be understood.

• Explicit considerations of programmatic roles and relationships between agencies and opportunities for interagency collaboration, as emphasized earlier by Dr. Turner, are also important.

• Survey committees need to be mindful of the breadth of the scientific communities that they are charged to represent and of inadvertently disenfranchising some segments of a community.

Mary Kicza, the NOAA assistant administrator for satellite and information services, provided a NOAA perspective. She began by calling attention to important differences between NASA and NOAA, which survey committees need to recognize. The differences include the fact that NOAA’s budget is only about one quarter that of NASA and is increasingly earmarked, and that NOAA’s civil servant population is two thirds that of NASA. NOAA has a special need to ensure long-term continuity in its observing programs, and in a constrained budget environment there are fewer opportunities and less flexibility to introduce new directions to the program. Furthermore, NOAA’s external operational user

community is very broad, and interfacing with it presents a more difficult challenge than with NASA’s research missions.

Ms. Kicza said it is important to understand the difference between managing cost growth during mission development and relying on premature cost estimates, as both have presented problems for NOAA and have resulted in having to delay and/or downsize programs. She said that putting realistic bounds on missions being considered for the distant future remains a difficult challenge. Finally, she noted that Earth observations have to be examined in the context of a global system of observing systems; international roles, responsibilities, and partnerships provide an important contextual overlay. One aspect of this is the need to have international standards for instrument calibration and data validation.

The final speaker from the federal executive branch panel (Panel on Agencies) was Jon Morse, senior policy analyst in the Science Division of the White House Office of Science and Technology Policy. Dr. Morse noted that his comments would draw also on views from colleagues in the Executive Office of the President, especially the Office of Management and Budget. He commented on the beneficial aspects of decadal surveys; particular issues and concerns about the most recent survey reports; approaches for facilitating decision making and other suggestions for improvements; and considerations for making surveys realistic. (Key excerpts from Dr. Morse’s presentation are reproduced in Appendix C.)

Dr. Morse supported an approach that begins by framing key science questions (“science first”) for a field and then explaining what measurements and capabilities are needed to answer those questions. He argued that the surveys should recommend priorities associated with the current program and the future program and should define what the survey committee looks for in terms of program balance and the mix between small, medium, and large initiatives and core research. He indicated that budget analysts and decision makers at OMB preferred to see cost estimates in terms of rough, rounded-off life-cycle costs rather than more detailed (and often underestimated) construction costs. Dr. Morse said it would be especially helpful for survey reports to discuss how unforeseen programmatic, policy, or scientific circumstances might alter the priorities and what science could be accomplished under several different budget scenarios.

The third and final panel of Session 1 drew views about the decadal survey process from one current and two former congressional staffers. William Adkins, president of Adkins Strategies LLC and former staff director of the House Subcommittee on Space and Aeronautics, called the decadal surveys the gold standard for scientific advice. He said that while surveys should focus on the science and steer clear of political considerations, their timing should be sensitive to political milestones or changes in the organization of Congress. He also recommended that future surveys should characterize the level of uncertainty (“error bars”) in cost

estimates and technology risk, comment on whether high-priority initiatives need to go in a particular order, and propose priorities for different budget scenarios.

Johannes Loschnigg, the current staff director of the House Subcommittee on Space and Aeronautics, noted that survey reports are read closely on Capitol Hill and their recommendations are compared carefully with agency program plans. He said that the surveys are viewed as being a firm expression of community preferences and that a well-done survey is the best antidote for having Congress yield to special pleading or making arbitrary decisions about agency programs. He noted, however, that future surveys need to address a problem with the recommendations of recent surveys—that is, their incompatibility with the resources that are likely to be available. Survey committees also need to state explicitly what they see in terms of mission size trade-offs in a balanced program.

Robert Palmer, a former staff director of the House Committee on Science, agreed with his colleagues that members of Congress think very highly of the decadal surveys because they help them do their jobs of advocating a strong U.S. science program. He sampled the views of other congressional staff and learned that they concurred on two points about decadal surveys. First, future surveys need to describe a healthy core program—that is, they need to state a level of activity and content below which an agency program should never fall. Second, new surveys should start afresh and not assume that legacy priorities automatically apply when setting priorities for new programs.

Discussions among workshop participants in response to the presentations of the three panels drew out several recurring themes. First, all participants seemed to share the view that the decadal surveys are important and that they have positive impacts on federal agency planning and decision making and on science community unity. The efforts by survey committees to seek broad community participation and consensus building and to set explicit, science-based priorities were cited again and again. The survey chairs and others also often agreed about problem areas that need to be understood and addressed. These include obtaining credible cost estimates; addressing and engaging all relevant government agencies rather than only one or two sponsors of major programs; reevaluating recommendations from prior surveys; not overlooking small programs such as NASA Explorers; taking into account the portfolio mix of mission sizes in an agency’s program; and following up after survey reports are completed and delivered.

Various speakers also highlighted other ends that effective surveys should attempt to attain:

• Breadth in committee membership and in community input;

• Synchrony with agency planning processes, and maybe even with election cycles;

• Balance between identifying broad science objectives versus providing specific advice about new program initiatives;

• Healthy core programs and attention to how they fit into the context of recommended new projects;

• Appreciation of budget realities;

• Full life-cycle costs for projects;

• Clarity about potential biases that might be hidden in cost estimates;

• Advice on phasing of projects;

• Principles for deciding how to respond to major budgetary, programmatic, or political changes;

• Attention to the challenges intrinsic to international or interagency cooperative programs;

• Integration with complementary or overlapping surveys for other disciplines;

• Attention to applications as well as fundamental science; and

• Mechanisms to monitor government attention and responsiveness.

All of these points reinforce the concept of a decadal survey as a strategic package. That is, decadal studies need to provide the best balance of scientific priorities and prioritized missions (a strategic package), work with the agencies to understand the budgetary and political realities, and give the agencies advice and decision rules to help them respond to major budget, programmatic, or political changes, while understanding that the agencies need flexibility in implementing specific programs and missions. A report needs texture and creative recommendations centered around key scientific questions and goals as much as it needs ordered lists.