2
Assessment Methodology
FUNDAMENTAL SCIENCE AND ENGINEERING RESEARCH
For this study it is important to have a clear definition and understanding of “fundamental science and engineering research.” The committee chose to adopt the definitions used by the Office of Management and Budget (OMB) in tracking and reporting federal spending for research and development (R&D).1 The OMB requires all federal agencies to report R&D spending in four categories:
-
Basic research. Systematic study directed toward a fuller knowledge or understanding of the fundamental aspects of phenomena and of observable facts without specific applications toward processes or products in mind.
-
Applied research. Systematic study to gain knowledge or understanding necessary to determine the means by which a recognized and specific need can be met.
-
Development. Systematic application of knowledge or understanding directed toward the production of useful materials, devices, and systems or methods, including design, development, and improvements of prototypes and new processes to meet specific requirements.
-
Facilities and equipment.
-
R&D equipment. Such equipment includes the acquisition or design and production of movable equipment, such as spectrometers, research satellites, detectors, and other instruments.
-
R&D facilities. Such facilities include the acquisition, design, and construction of, or major repairs or alterations to, all physical facilities for use in R&D activities. Facilities include land, buildings, and fixed capital equipment, regardless of whether the facilities are to be used by the government or by a private organization, and regardless of where title to the property may rest. This category includes such fixed facilities as reactors, wind tunnels, and particle accelerators.
-
The OMB has used these or similar categories in the collection of R&D data since 1949. The use of the OMB categories and the financial data reported allowed the committee to determine trends in expenditures for fundamental science and engineering research and for the facilities and equipment that support that research.
NASA uses the terminology of technology readiness levels (TRL levels 1 through 9)2 to define the technological maturity of a hardware or software system under development. TRL 1 refers to scientific knowledge generated as an underpinning of hardware or software concepts and applications. TRL 2 applies when a practical application has been identified but is speculative and without experimental proof or detailed analysis to support the conjecture. TRL 3 applies when analytical studies place the technology in an appropriate context and laboratory demonstrations, modeling, simulation, and nonintegrated software components have validated the analytical prediction. The definitions of the TRL system are provided in Appendix B.
The committee decided that the NASA TRL 1 through 3 (TRL 1-3) categories best matched the study requirement to “focus on an appraisal of equipment, facilities and support services used for fundamental science and engineering research” and directed NASA to limit presentations on facilities, equipment, support services, and science/engineer interviews to these categories.
STUDY PROCESS AND LIMITATIONS
The committee invited NASA Headquarters personnel and representatives of each NASA center that sponsored fundamental research to make a series of presentations at the first committee meeting held at the National Academy of Sciences building in Washington, D.C., on September 8 and 9, 2009. Before the meeting, the committee had requested that NASA Headquarters and each NASA center provide the following information at this first meeting and limit all presentations and material to activities encompassing TRLs 1-3:
-
Expenditures by year for new laboratory equipment over the past 5 years;
-
Expenditures for new facilities and major upgrades to existing facilities over the past 5 years;
-
Planned expenditures for laboratory equipment and facility improvements over the next 3 years;
-
The age distribution of existing laboratory equipment and the maintenance, repair, and upgrades of older equipment;
-
Facility maintenance budgets by year over the past 5 years; and
-
The uniqueness and importance of specific facilities to the NASA scientific and technology missions.
Also, before that first meeting, the committee decided that the expertise of its members matched the two main disciplines into which fundamental science and engineering research at NASA—namely, aeronautics research and space/Earth science research—were classified. Accordingly, two subcommittees were formed; their members are listed in Appendix C.
At the first meeting on September 8, 2009, the committee received presentations from NASA Headquarters representatives of the Aeronautics Research Mission Directorate (ARMD), the Science Mission Directorate (SMD), and the Exploration Systems Mission Directorate (ESMD) an overview of how NASA assesses facility conditions and details of the NASA budgets over the past 5 years. Presentations from the Ames Research Center (ARC), Jet Propulsion Laboratory (JPL), Marshall Space Flight Center (MSFC), Glenn Research Center (GRC), and Johnson Space Center (JSC) were made on the second day of the meeting.
It is important to mention the limitations on this study that resulted from the narrowly focused statement of task (SOT), the constrained study and travel budgets, the limited time available to committee members, and the inability to command as much time as they would like from NASA center personnel. Another limitation was the reliance on NASA to let the committee know which NASA facilities are or were engaged in TRL 1-3 work. Not only was the committee unable to verify independently that the list of facilities presented by NASA was comprehensive, but it also became apparent to the committee during site visits that each center had defined the committee’s requests somewhat differently—that is, some centers only presented facilities that are currently engaged in low-TRL work, while others included facilities that have been or could have been so engaged.
The focus on fundamental science and engineering research eliminated several NASA centers that do not conduct a significant amount of, or any, TRL 1-3 research, such as JSC, the Stennis Space Center (SSC), and the Kennedy Space Center (KSC). JSC does conduct a small amount of TRL 1-3 research, but the committee decided, based on the material presented at the first meeting, that it did not warrant a visit.
Although the NASA Authorization Act of 2005, Section 507, designated the International Space Station (ISS) as a national laboratory, the committee chose not to include that site in this study because of its difficulty of access, its uniqueness, and the potential decommissioning in 2015. The committee also chose not to include fundamental medical research conducted at NASA in support of the manned flight program because several reports had been issued on human space research, some by the NRC,3 over the past few years. Following a query to NASA Headquarters, it was determined that supercomputers used for the study of computational fluid dynamics, among other areas of application, would not be examined by this committee.
The SOT (Appendix A) asked for an appraisal of equipment, facilities, and support services used for fundamental science and engineering research, as well as on their adequacy for supporting NASA goals. The SOT did not call for a detailed study of the individual programs that constitute the NASA fundamental research enterprise or an assessment of the efficacy and quality of the research performed, or the adequacy of those programs to support the NASA goals. As required by the SOT, the committee focused on assessing the state of laboratory capabilities and on whether they are equipped and maintained to support the NASA research activities. Further, the SOT required the committee to rely on the experience and expertise of its members to make comparisons of NASA’s laboratory capabilities with other entities.
In 2004, a task force of the NASA Advisory Council, the NASA Federal Laboratory Review (NFLR) Task Force, was formed to provide an independent evaluation of NASA’s R&D. The NFLR defined “laboratory” as all the activities and facilities at a center and subordinate organizational units that perform or support the performance of R&D. JPL, although a federally funded research and development center operated by the California Institute of Technology, was included as a NASA center since NASA is the sponsor and owns the property and equipment. The committee adopted that definition of a laboratory for this study.
PREPARATIONS FOR CENTER VISITS
Before visiting a NASA center, the committee presented the center management with a list of questions. The questions ranged from specifics on the types of research funding used by the center to how their acquisition, maintenance, and upgrading of equipment and support services were managed. The list of questions sent to the NASA centers appears in Appendix G. The committee requested that a significant portion of the time for a visit be devoted to touring research laboratories and interacting with research scientists and engineers. Center management responded with proposed agendas that aimed to satisfy committee requirements. The agendas were then mutually adjusted to optimize committee goals and needs during the visit.
For those centers that focus on aeronautics or space and Earth science research, the committee members who visited were experts in that particular discipline. For example, the aeronautics group identified earlier and by name in Appendix C, visited GRC, Langley Research Center (LaRC), and ARC, while the space and Earth science group visited GSFC and JPL. Both groups visited GSFC in connection with the first committee meeting in nearby Washington, D.C. A subgroup of the space and Earth science group joined with the aeronautics group to visit GRC to assess some of the space science activities at that center. Likewise, a subgroup of the space and Earth science group visited ARC but at a different time from the aeronautics group because of schedule conflicts, and a separate subgroup visited MSFC. At each visit, subgroups were assigned to collect the committee inputs and prepare a draft report on the visit.
A typical center visit consisted of initial presentations by center senior management to acquaint the committee with the operations of the center and to address earlier questions and information requirements. Walking tours of R&D facilities and laboratories followed. During these tours committee members had adequate time to interact with the research scientists and engineers and to ask questions. After the tours the committee members had the opportunity to meet and interact with working research scientists and engineers. Typically, around 15 people attended these hour-long sessions, and the committee developed a keen sense of the issues facing the research scientists. Before departure the committee members met to record their initial findings.
At the second committee meeting, in Irvine, California, on November 11 and 12, 2009, the committee was briefed on the fundamental research, equipment, and facilities at Dryden Flight Research Center (DFRC). This briefing and the resulting discussions form the basis of the assessment of DFRC in Chapter 4.