NASA's Elementary and Secondary Education Program: Review and Critique (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Executive Summary T he federal role in precollege science, technology, engineering, and mathematics (STEM) education is receiving increasing attention in light of the need to support public understanding of science and to develop a strong scientific and technical workforce in a competitive global economy. Federal science agencies, such as the National Aeronautics and Space Administration (NASA), are being looked to as a resource for enhancing precollege STEM education and bringing more young people to scientific and technical careers. For NASA and other federal science agencies, concerns about work- force and public understanding of science also have an immediate local dimension. The agency faces an aerospace workforce skewed toward those close to retirement and job recruitment competition for those with science and engineering degrees. In addition, public support for the agency’s mis- sions stems in part from public understanding of the importance of the agency’s contributions in science, engineering, and space exploration. COMMITTEE TASK In the NASA authorization act of 2005 (P.L. 109-555 Subtitle B‑­Education, Sec. 614) Congress directed the agency to support a review and evaluation of its precollege education program to be carried out by the National Research Council (NRC). The legislation mandated that the review include recommendations to improve the effectiveness of the pro- gram and address four tasks:  

 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM 1. an evaluation of the effectiveness of the overall program in meeting its defined goals and objectives; 2. an assessment of the quality and educational effectiveness of the major components of the program, including an evaluation of the adequacy of assessment metrics and data collection requirements available for determining the effectiveness of individual projects; 3. an evaluation of the funding priorities in the program, including a review of the funding level and trend for each major component of the program and an assessment of whether the resources made available are consistent with meeting identified goals and priorities; and 4. a determination of the extent and effectiveness of coordination and collaboration between NASA and other federal agencies that spon- sor science, technology, and mathematics education activities. NASA, in consultation with the NRC, interpreted the charge to mean a focus on the Elementary and Secondary Program managed by the Office of Education. This program includes seven projects: 1. the Aerospace Education Services Project (AESP) 2. the Science, Engineering, Mathematics and Aerospace Academy (SEMAA) 3. NASA Explorer Schools (NES) 4. the Digital Learning Network (DLN) 5. Education Flight Projects (EFP) 6. the Educator Astronaut Project (EAP) 7. the Interdisciplinary National Science Project Incorporating Research and Education Experience (INSPIRE) The study committee reviewed a wide range of documents related to NASA’s programs in precollege STEM education, heard testimony from NASA staff, and commissioned three papers. As is the case with many fed- eral science agencies involved in education outreach, only a limited number of external evaluations of NASA education projects have been conducted. As a consequence, the committee also relied on relevant research evidence and committee members’ collective expertise when drawing conclusions about how projects could be improved. The committee developed specific recommendations for only three of the seven projects—NES, AESP, and SEMAA—because the other four projects had been in place too short a time or lacked sufficient documentation of project performance. The report provides a summary of the committee’s findings regard- ing the recent history of NASA’s education program and K‑12 projects (Chapter 2) and the federal context for NASA’s role in K‑12 education, 

EXECUTIVE SUMMARY  including discussion of other science agencies (Chapter 3). It also discusses each of the seven projects in depth with specific suggestions for improve- ment (Chapter 4). Finally, it reviews NASA’s current approach to project review and evaluation and offers suggestions for improving the process (Chapter 5). Chapter 6 of the report details the committee’s conclusions and recommendations. CONTEXT OF K‑12 EDUCATION AT NASA Education and contributing to public understanding of science have been important components of NASA’s mission since its creation by the 1958 Space Act. NASA does not, however, have the lead federal role in pre- college STEM education, which is the responsibility of the National Science Foundation and the U.S. Department of Education. Rather, as a discoverer of new science and a creator of new technology, NASA, like other federal science agencies, has an important complementary role in STEM education. That role is closely linked to and guided by the core scientific, engineering, and exploration missions of the agency. The bulk of the K‑12 STEM education activities in the agency are in the Office of Education and the Science Mission Directorate (SMD), which each account for about 50 percent of the agency’s total funding for K‑12 STEM education. Thus, the seven projects that make up the Elementary and Secondary Program in the headquarters Office of Education on which this review was to focus represent only about one-half of the activities in K‑12 education undertaken by the agency. Traditionally, the Office of Education and SMD have had different approaches to developing and implementing K‑12 education projects. SMD devotes a percentage of funds connected with each major science mission to education activities. Proposed education activities connected to each mis- sion are described as part of the proposal for science funding and undergo competitive expert review. In contrast, the Office of Education is supported by a line item in NASA’s budget. Projects are developed by Office of Educa- tion staff or originate in NASA field centers and are then expanded. This history has resulted in a broad and diverse portfolio of projects that vary in scope, target audiences, and objectives. In 2006 the agency adopted a new strategic coordination framework that is designed to bring coherence to the education activities across the agency; it was in the early stages of implementation as the committee’s study was done. CONCLUSIONS The committee’s conclusions regarding the effectiveness of NASA’s K‑12 education program and areas for improvement are summarized here 

 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM as responses to the four purposes stated by congress. Given the charge to the committee from Congress and from NASA, the committee focused on the seven specified core programs. The committee also took a wider view of NASA’s entire portfolio in K‑12 STEM education in drawing its conclusions. Effectiveness of the Elementary and Secondary Program The committee was limited in its ability to draw conclusions about the overall effectiveness of the headquarters Office of Education’s Elementary and Secondary Program because of instability in the program and lack of rigorous evaluation. NASA’s education portfolio has experienced rapidly shifting priorities, fluctuations in budget, and changes in management struc- ture that have undermined the stability of programs and made evaluation of effectiveness challenging. NASA does not have a coherent overall plan for evaluation and for how results of evaluation should inform program and project design and implementation. Few of NASA’s projects have been formally evaluated, and none has been evaluated rigorously. Consequently, there are little data across projects on which to base conclusions about effectiveness. Effectiveness of Individual Projects and Adequacy of Assessment Metrics The Elementary and Secondary Program overall is to be commended for its effort to reach underrepresented groups. The committee concludes that the seven specified core projects are somewhat effective at raising awareness of the science and engineering of NASA’s missions and generating students’ and teachers’ interest in STEM subjects. As currently configured, however, the projects cannot be shown to be effective at enhancing learning of STEM content or providing in-depth experiences with the science and engineering of the missions. Evaluation of individual projects is complicated by the fact that individ- ual projects have taken on the broad goals of the Elementary and Secondary Program rather than developing project-specific, focused goals and objec- tives that are appropriate to the design and scope of individual projects. Currently, data collection efforts common to all projects chiefly consist of counts of sessions offered, numbers of participants, and immediate feed- back from them. Such data are insufficient to evaluate the effectiveness of projects or of the program as a whole. The current data collection system, the NASA Education Evaluation Information System (NEEIS), is inadequate for supporting effective evaluation and has technical shortcomings. 

EXECUTIVE SUMMARY  Funding Priorities NASA has demonstrated a strong commitment to funding STEM edu- cation activities. However, because K‑12 activities originate in different administrative units in the agency, it is difficult to track all of the funding for K‑12 education. Funding for education through the Office of Educa- tion has declined from $230 million in 2003 to $153 million in 2007 and has been significantly affected by an increasing number of congressionally directed appropriations (CDAs, also known as earmarks). Such fluctuations in budget have undermined the program’s stability and coherence. NASA does not appear to have budgeted sufficient funds for a thor- ough evaluation of projects; however, because budgets for evaluation are reported as project costs, information on total funds targeted specifically for evaluations is unavailable. The committee questions whether the agency has sufficient resources and expertise to adequately support the school-level cur- ricular reform efforts of the NASA Explorer Schools project. NASA should also consider whether current information and communications technology could be used to improve the cost-effectiveness of some projects. Coordination and Collaboration Between NASA and Other Federal Agencies NASA has participated in federally coordinated activities, but NASA does not systematically coordinate with other federal agencies involved in STEM education nor interact with other federal agencies to draw on exper- tise related to the design of STEM education projects. There have been a limited number of cross-agency projects in which NASA has had good col- laboration with other federal agencies such as the GLOBE program, which is sponsored by NASA, the National Science Foundation, and the Depart- ment of State with cooperation from many organizations and government agencies including the National Oceanic and Atmospheric Administration and the Department of Education. RECOMMENDATIONS The committee identified four broad areas that are important for improving NASA’s efforts in K‑12 STEM education: (1) the nature of NASA’s role in K‑12 STEM education, (2) continuous improvement of projects, (3) partnerships and expertise in education, and (4) information and communications technology. Additional, detailed recommendations for individual projects are included in Chapter 6 (the numbering here follows that used in the chapter). 

 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM NASA’s Role in K‑12 STEM Education Recommendation 1  NASA should continue to engage in education activities at the K‑12 level, designing its K‑12 education activities so that they capitalize on NASA’s primary strengths and resources, which are found in the mission directorates. These strengths and resources are the agency’s scientific discoveries; its technology and aeronautical developments; its space exploration activities; the scientists, engineers, and other technical staff (both internal and external) who carry out NASA’s work; and the unique excitement generated by space flight and space exploration. Recommendation 2  The exciting nature of NASA’s mission gives par- ticular value to projects whose primary goal is to inspire and engage students’ interest in science and engineering, and NASA’s education portfolio should include projects with these goals. Because engineering and technology development are subjects that are not well covered in K‑12 curricula, projects aimed at inspiring and engaging students in these areas are particularly important. Recommendation 3  NASA should provide opportunities for teachers and students to deepen their knowledge about NASA‑supported areas of science and the nature of science and engineering through educa- tional activities that engage them with the science and engineering carried out by the mission directorates. Recommendation 4  NASA should strive to support stability in its education programs, in terms of funding, management structure, and priorities. Recommendation 8  The NASA headquarters Office of Education should focus on leadership and advocacy for inclusion of education activities in the programs of NASA’s four operating directorates, quality assurance, internal coordination, and coordination with other agencies and organizations. In the development of new education projects, the office should partner closely with the directorates or centers and con- sult with external education experts. 

EXECUTIVE SUMMARY  Continuous Project Improvement NASA has not adopted mechanisms to ensure continuous improvement of projects within the Elementary and Secondary Program. For example, goals and objectives for individual projects reflect very closely the overall goals for the entire elementary and secondary program and are not well calibrated to the scope and target audience of individual projects. Effective program design and management requires that a project’s goals, desired outcomes, and evaluation metrics be aligned. This alignment is not gener- ally the case for the seven Office of Education precollege projects that this committee was asked to examine. NASA also lacks an overall plan for evaluation of its precollege port­ folio and projects. Such a plan should include definition of measurable project goals and objectives, framing of the purposes of evaluations and key questions, and a plan for how information from the evaluation will inform the design and implementation of projects. NASA’s new strategic coordina- tion framework for education is designed to address these issues of review and evaluation; however, it is still in initial stages of implementation. Recommendation 5  NASA should take a more intentional approach to portfolio development than it has to date so that individual projects are well defined and have clear and realistic goals and objectives given their target audiences. Management of the resulting portfolio should include periodic review of the balance of investment across projects. Recommendation 17  NASA should develop an overall evaluation plan for its K‑12 education program and projects and allocate the resources needed to implement the plan. Recommendation 18  For portfolio management, the NASA evalu- ation plan should include some cross-project evaluations as well as project-specific evaluations. Recommendation 19  NASA should plan the scale, design, and fre- quency of each project evaluation so that it aligns to the scale and goals of the project, and to the nature of the decisions that need to be made. Recommendation 20  NASA should use evaluation findings to inform project design as well as project improvement. To do so, NASA should establish mechanisms to connect evaluations to program and project decisions. 

 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM Partnerships and Expertise in Education Given NASA’s primary focus on science, engineering, and technology, the agency employs a large staff with expertise in these areas. The number of agency staff who have primary expertise in education is limited. The technical staff in the agency cannot be expected to have sufficient expertise in K‑12 STEM education to allow them to develop effective education p ­ rojects on their own. Thus, the scientists and engineers in the agency need to work in concert with experts in education, often from outside the agency, in order to achieve the appropriate mix of expertise in science, engineering and education to design and implement effective education projects. Recommendation 6  NASA program and project planning and execu- tion should make better and more consistent use of opportunities to involve education stakeholders, to partner with individuals and orga- nizations that can provide expertise in education, and to connect to the existing infrastructure for K‑12 STEM education. Recommendation 7  NASA’s partnerships in education should be designed in light of the specific objectives of each project. NASA can play a lead role in projects intended to inspire and engage students and should use strategic partnerships to leverage the impact of such projects. For projects designed to affect schools through work with students, teachers, or curriculum materials, NASA should work in partnerships with organizations that complement NASA’s science and engineering expertise with education-specific expertise and avenues of dissemination. All partnerships should begin during the early stages of project design. Information and Communications Technology The agency’s K‑12 education projects do not appear to be using infor- mation and communications technology effectively. Projects tend to use technology that was modern at the time of inception and do not make efforts to periodically update it. Continued use of the outdated informa- tion and communications technology can lead to inefficiencies in the use of project funds. Recommendation 9  NASA should make better use of current and emerging information and communications technology to provide broader and more user-friendly access to NASA materials, to support 

EXECUTIVE SUMMARY  NASA’s K‑12 STEM education activities, to extend the reach of NASA’s education activities, and to maintain a centralized data system. Recommendation 10  NASA should periodically review each project to determine whether its components are the most cost-effective uses of resources, given current information and communications technology alternatives. Overall, NASA makes significant contributions to K‑12 education by providing access to its expertise in science, engineering, technology, and space exploration. It is uniquely positioned to inspire and engage students in STEM subjects and to expose teachers and students to the nature of sci- ence and engineering through exposure to the agency’s missions. However, the Elementary and Secondary Program is not realizing NASA’s potential as a resource for education as effectively as could be hoped. Developing a culture of ongoing improvement, cultivating sustained partnerships that bring in expertise in education, and using information and communication technology more effectively are promising strategies for improving NASA’s programs in K‑12 education. When these are linked to a coherent and well- funded plan for evaluation, the agency stands poised to have a positive and demonstrable impact on learning and teaching in STEM subjects.