NASA's Elementary and Secondary Education Program: Review and Critique (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction P ublic outreach and science education have been important compo- nents of the mission of the National Aeronautics and Space Admin- istration (NASA) since its creation in 1958. NASA’s strategy for promoting these components has evolved during the life of the agency, and it has undergone considerable change in the last 10 years. Most recently, as part of a restructuring of the entire agency, agencywide education programs at NASA were reorganized and subjected to an internal review guided by a new, detailed strategic plan for education (National Aeronautics and Space Administration, 2006a). The reorganization and new education plan pro- vide a unique opportunity for a review and evaluation of NASA’s past and ongoing activities in education. This report focuses on NASA’s K‑12 education activities, as mandated by congressional language in the 2005 reauthorizing legislation for the agency. The review comes at a time when science, technology, engineering, and mathematics (STEM) education is a subject of increasing national con- cern. Focus on STEM education primarily reflects a concern that national competitiveness, both economic and security related, requires that a high percentage of students leaving high school are capable and motivated to pursue careers in science and technology. It also reflects concern that there is a lack of public understanding of science and scientific inquiry. On the first point, if the United States is to remain scientifically innovative and competi- tive in an increasingly globalized economy, preparing students for science and engineering careers is imperative. On the second point, a democratic society needs all citizens to be scientifically literate in order to participate in national debates on such scientific issues as climate change and alternative 11 

12 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM fuels. For NASA, there is also a local issue as the agency faces an aerospace workforce that is skewed toward employees who are nearing retirement, as well as competition in recruiting job candidates with science and engineer- ing degrees. One response to these concerns has been to reexamine the role of federal science agencies in supporting and advancing STEM education for kinder­garten through grade 12 (K‑12). This study of NASA’s K‑12 education portfolio provides an opportunity not only to examine NASA’s activities in grades K‑12, but also to examine the larger issue of defining the appropriate role for science agencies in supporting improved K‑12 STEM education. THE COMMITTEE’S CHARGE AND APPROACH The Committee to Review and Evaluate NASA’s Precollege Education Portfolio was established by the National Research Council (NRC) to undertake this study. The committee included 12 members with expertise in the history and structure of NASA education programs; program evaluation for a range of program types (specifically targeted to the kinds of projects in the NASA portfolio); science and mathematics instruction at both the ele- mentary and secondary levels, with particular knowledge of earth and space sciences; teacher professional development; education policy and practice in science and mathematics at the state and local levels; and measurement. Special emphasis was given to identifying individuals for this committee who have a working knowledge of NASA as an organization, as well as knowledge of NASA’s Elementary and Secondary Education Program (see Appendix A for biographical sketches). The study focused on the purposes identified by Congress in its charge to the study committee to “conduct a review and evaluation of NASA’s precollege science, technology and mathematics education program. The review and evaluation shall include such recommendations as the NRC determines will improve the effectiveness of the program and include 1. an evaluation of the effectiveness of the overall program in meeting its defined goals and objectives; 2. an assessment of the quality and educational effectiveness of the major components of the program, including an evaluation of the adequacy of assessment metrics and data collection requirements available for determining the effectiveness of individual projects; 3. an evaluation of the funding priorities in the program, including a review of the funding level and trend for each major component of the program and an assessment of whether the resources made avail- able are consistent with meeting identified goals and priorities; and 

INTRODUCTION 13 4. a determination of the extent and effectiveness of coordination and collaboration between NASA and other federal agencies that spon- sor science, technology, and mathematics education activities. The NASA headquarters Office of Education and the NRC agreed to focus the review on the seven projects in the Elementary and Secondary Education Program (see Box 1‑1 for a definition of programs and projects). Those projects are referred to in this report as the seven core projects: 1. The Aerospace Education Services Project (AESP) provides training for teachers to use NASA STEM curricula and new and evolving education pedagogy and supports student STEM education through student projects, classroom visits, and inquiry-based ­ activities. AESP employs former teachers who travel nationwide to work with teachers, students, and schools to improve STEM education. The majority of AESP activities are in NASA Explorer Schools (see below). 2. The Science, Engineering, Mathematics and Aerospace Academy (SEMAA) is conducted during and after school for K‑12 students to expose historically underrepresented youth to activities in the fields of science, engineering, mathematics, and technology. SEMAA includes three components: curricular support materials for use during and after school, interactive family activities, and access to NASA technology at Aerospace Education Laboratories. 3. The NASA Explorer Schools (NES) project immerses selected high- minority and high-poverty urban and rural middle schools in NASA mission content by providing them access to NASA resources, peo- ple, and products. It is implemented through 3-year partnerships between NASA and the selected school teams, which are identified by the NASA centers. 4. The Digital Learning Network (DLN) makes NASA’s educational resources and its scientists and engineers available to students and teachers through video conferencing. 5. Education Flight Projects (EFP) provides a way for students and teachers to capitalize on the data and images provided by NASA’s   rojects aimed at museums and science centers fall within the Office of Education’s Informal­ P Education Program and were not included in the review.   The term “centers” in this report refers to the nine NASA field centers and the Jet P ­ ropulsion Laboratory.   Although the DLN was an activity within the NES when the committee began its work, there were discussions in NASA’s Office of Education about making it an independent project. Therefore, NASA staff requested that we treat it as such for the purposes of our study. 

14 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM BOX 1-1 Programs and Projects: Definitions Program is a group of projects that are guided by a common set of overarching goals and share similar target audiences. The NASA Office of Education has five programs: elementary and secondary education, higher education, e-education, informal education, and the Minority University Research Education Program (MUREP). Projects are the component parts of programs and include a set of activities that address the same specific measurable goals aimed at a specific audience. The seven projects that are the primary focus of this study make up the Elementary and Secondary Program in the Office of Education. In some cases, the formal name of a project includes the word “program”: for example, the Aerospace Education Services Program. We have chosen to refer to these as projects for the sake of clarity because they are part of the Elementary and Secondary Program. scientific and exploration missions and interact with astronauts on the International Space Station. 6. The Educator Astronaut Project (EAP) includes the educator astro- naut recruitment and selection activities that guide the recruitment of a small number of qualified educators to join the Astronaut Corps. These teachers develop educational material related to their work as astronauts. A subset of teachers chosen through the selec- tion process, who do not join the Astronaut Corps, are selected to form the Network of Educator Astronaut Teachers. These teachers serve as NASA education advocates by engaging their schools and communities in NASA education activities. 7. The Interdisciplinary National Science Project Incorporating Research and Education Experience (INSPIRE), which is under development, is a three-tiered project designed to maximize student participation and involvement in STEM and to enhance the STEM pipeline from middle school through high school to the under- graduate college level. Recognizing that there are education activities related to K-12 educa- tion located outside of the Elementary and Secondary Program, the com- mittee initially considered including all NASA projects related to K-12. 

INTRODUCTION 15 However, the preliminary information we collected confirmed that an all- inclusive and detailed review was impossible given time and budget con- straints. Thus, the committee carried out its charge to focus mainly on the seven core projects with recognition that they do not capture the full range of the agency’s K‑12 education activities. For comparison purposes the committee included some examination of K-12 education activities that are based in the Science Mission Directorate (SMD) and are not directly managed by the Elementary and Secondary Pro- gram However, due to the resources and timeline of the study, the exami- nation of these activities was necessarily more limited. The SMD has been especially active in developing education projects and materials. Over the course of this review, the committee discovered that in fiscal 2006, the SMD spent about the same amount of money on K‑12 STEM education projects as the headquarters Office of Education. The SMD’s work is largely sepa- rate from the agencywide programs managed by the headquarters Office of Education. Thus, the committee considers the seven core projects in the context of the broader portfolio of K‑12 education activities in NASA. The committee carried out its work through an iterative process of gathering information, analyzing and deliberating it, identifying gaps and questions, gathering additional information to fill these gaps, and carrying out further analysis and deliberations. The contractually determined time and resources for the study constrained the scope of the committee’s review to existing documentation and discussions with NASA program and project staff. The committee did not carry out extensive original data collection. Because of these constraints, the study is best thought of as an expert review rather than a formal program evaluation. In its search for relevant information, the committee held three public fact-finding meetings; reviewed documents related to NASA’s K‑12 educa- tion portfolio, such as budget requests, project evaluations, project plans, and other technical reports; and commissioned background papers. Over the first three meetings, the committee heard presentations and engaged in discussions with staff of the NASA Office of Education who are involved with K‑12 education projects, as well as directors of educa- tion and outreach projects based in NASA’s SMD. The committee members were also briefed by people who had conducted evaluations of some specific NASA education projects. At the second meeting, in addition to presenta- tions about NASA’s projects, the committee explored the larger question of how federal science agencies can best engage in K‑12 education activities, through a panel discussion among representatives from the Department of Energy, the National Oceanic and Atmospheric Administration, the National Institutes of Health, and the National Science Foundation. At the third meeting, the committee heard only from NASA staff affiliated with the Elementary and Secondary Program. 

16 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM The committee also commissioned three papers to provide background and in-depth analysis. One paper provided a critique of existing external evaluations of NASA’s K‑12 education projects. Another paper provided an analysis of the Explorer Schools Project in the context of what is known about successful models for comprehensive and subject specific school reform. The authors of these two papers presented early drafts of their work at the committee’s third meeting. The third paper, commissioned after the third meeting, compared the proposed model of INSPIRE with successful models from multiyear projects focused on engaging students in science and engineering. These three papers were valuable resources for the committee in developing our conclusions and recommendations and writing this report. OVERVIEW OF NASA’S EDUCATION PROGRAMS NASA’s K‑12 STEM education projects are in the headquarters Office of Education, the mission directorates, and the centers. Some of the ­projects are deeply embedded in the research and exploration activities of the agency; others are more general, agency-supported projects that draw broadly on NASA’s science, technology, and engineering resources. These latter ­projects are managed primarily in the headquarters Office of Education, which houses the Elementary and Secondary Education Program. The projects that are closely tied to the research exploration activities of the agency are managed entirely in individual mission directorates. The mission-­embedded projects have a particular responsibility to inform the public about the sci- ence and engineering of each mission and to make resources available for educators who want to include this content in their teaching. Most of these projects are carried out by non-NASA employees in universities or research institutions that report to and work under the guidance of NASA mission directorate staff. The headquarters Office of Education and mission K‑12 projects tend to operate independently, although both may have staff housed in the same center that work together and share some resources and information. The educational efforts within the mission directorates and in the NASA ­centers collectively contribute to the agency’s education goals, objectives, and out- comes. Recently, to support the agency’s strategic education coordination framework, the Office of Education developed an education portfolio that aligns with the agency’s strategic plan, provides a governance structure, and creates an agencywide strategic planning, implementation, and evalu- ation framework for education. The education portfolio is described in Chapter 2. 

INTRODUCTION 17 PREVIOUS REVIEWS OF NASA’S EDUCATION PROGRAMS There have been two previous major efforts to review and evaluate NASA’s education activities. Both reviewed the entire portfolio, including not only K‑12 activities, but also higher education, informal science, and public outreach. In the early 1990s, NASA asked the NRC for advice and assistance in how to manage and monitor an expanding portfolio of educa- tion activities. The resulting NASA Education Programs Outcomes Com- mittee was charged with defining appropriate goals for NASA’s education projects and recommending data collection procedures and indicators that would show whether the projects were effectively meeting their goals. The first committee’s report (National Research Council, 1994) con- tained a set of recommended goals and indicators for assessing the ­quality of NASA’s education projects, including those at the K‑12 level. The com- mittee further recommended that NASA gradually and deliberately under- take implementation of the indicator system described in its report; that NASA dedicate a fixed percentage of its education budget (5–10%) to indicators and evaluation; and that NASA continually review the agency’s collection of programs. One of the core projects for the current review, AESP, existed at the time of the 1994 review and was included in the analysis. However, its focus has changed considerably in recent years. The other projects that this com- mittee was asked to review did not exist when the previous NRC committee conducted its review. However, the goals and indicators developed by that committee in the 1994 report may still be relevant to the current portfolio and were taken into consideration by the current committee. In 2001, at the direction of the Office of Management and Budget, NASA contracted for an external evaluation of its education program. The purpose was to determine the extent to which the NASA education program provides an important contribution to the federal education portfolio, as well as to provide an assessment of the program’s strengths and opportuni- ties for improvement. The review focused on five questions: 1. Is there an appropriate role in education for NASA that is unique from other federal agencies such as the U.S. Department of Educa- tion and the National Science Foundation? 2. What is the appropriate role for NASA in education? 3. Has NASA established appropriate goals and objectives for its education program? 4. Is the NASA Implementation Plan a document that can effectively guide the education program to achieve the identified goals and objectives? 

18 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM 5. Is NASA’s education program effective at achieving its established goals and objectives for the appropriate balance of recipients? A seven-member expert panel was appointed to carry out the review. They reviewed evaluation data and other materials and participated in a 3-day session to examine NASA plans and projects. They reviewed the pro- fessional literature, existing data, strategic plans, and testimony by selected project administrators and program participants. They produced a report of their findings and recommendations for future direction (Westat, 2001). The expert panel concluded that NASA has a unique opportunity to use its facilities and personnel to enrich science education from the K‑12 level through the Ph.D. degree level. They emphasized that part of NASA’s role is to transfer and infuse the results of NASA research, development, and technology into the nation’s STEM education efforts. They also concluded that NASA had established appropriate goals, had an appropriate imple- mentation plan, and had been effective in reaching its goals. PREVIOUS REVIEWS OF OTHER FEDERAL STEM EDUCATION PROGRAMS In order to review and evaluate NASA’s K‑12 education portfolio, the committee determined that it needed to identify and understand the various ways that other federal science agencies are or could be involved in K‑12 education. Thus, this study connects to early and ongoing efforts to assess the role of federal science agencies in STEM education, including several reports and the ongoing efforts of the Academic Competitiveness Council. In 1993, recognizing the need to enhance the coordination of federal STEM programs, the Committee on Education and Human Resources of the Federal Coordinating Council for Science, Engineering and Technology (FCCSET) formulated a 5-year agenda. The first step was to appoint an expert panel charged with conducting a broad review of federal programs in STEM education and assessing federal program evaluation efforts. The panel developed a report that recommended improved management and coordination of programs, a more balanced distribution of existing funds, and comprehensive program evaluation (Federal Coordinating Council for Science, Engineering and Technology, 1993). The panel’s findings confirmed that coordination of federal programs across agencies and governmental levels, and the private sector, was minimal. In addition, it concluded that core federal programs in STEM lack balance and coherence. The panel found that federal spending on STEM was not guided by assessments of national need, that few federal programs had been thoroughly evaluated to determine their effectiveness, and that funding for evaluation and evalu- ation personnel was extremely limited. Furthermore, evaluation practices 

INTRODUCTION 19 were often inadequate for the purpose of improving programs, making informed decisions about program retention or expansion, and providing for accountability. More recently, the Academic Competitiveness Council (ACC), estab- lished through the Deficit Reduction Act of 2005, was charged with review- ing all federal programs with a focus on mathematics and science education, and reporting its findings to Congress in February 2007. The ACC’s goal was to ensure the greatest return from the government’s investment in STEM education. As a result, the ACC’s effort focused most closely on program effectiveness, overlap, and duplication. In its report (U.S. Department of Education, 2007a), the ACC states that K‑12 STEM education programs should focus on student learning, teacher quality, and student engagement. As did the FCCSET’s expert panel report, the ACC report concluded that nearly all of the reviewed federal programs lacked rigorous metrics and methods for evaluation. The ACC recommended [T]he ACC program inventory, goals, and metrics should be a living r ­ esource updated regularly; agencies and the federal government should foster knowledge of effective practices through improved evaluation and/or implementation of proven-effective, research based instructional ­materials and methods; Federal agencies should improve their coordination of K‑12 STEM education programs with state and local school systems; Federal agencies should adjust program designs and operation so that programs can be assessed and measurable results can be achieved; funding for STEM education programs should not increase unless a plan for appropriately rigorous, independent evaluation is in place; Agencies with STEM educa- tion programs should collaborate on implementing the ACC recommenda- tions under the auspices of the National Science and Technology Council. (U.S. Department of Education, 2007a, p. 34) The ACC recommendations demonstrate that the situation today is not very different from the portrait of federal investments in STEM educa- tion painted by the FCCSET expert panel in 1993 (Federal Coordinating Council for Science, Engineering and Technology, 1993). This context of little coordination and limited rigorous evaluation presents a challenge to the present committee for addressing the first and fourth major items in its charge, “[to make] a determination of the effectiveness of the overall pro- gram in meeting its defined goals and objectives; [to make] a determination of the extent and effectiveness of coordination and collaboration between NASA and other Federal agencies that sponsor science, technology, and mathematics education activities.” Thus, the committee determined that a critical step in assessing NASA’s K‑12 activities was to identify the appropriate roles for a federal science 

20 NASA’S ELEMENTARY AND SECONDARY EDUCATION PROGRAM and technology agency in STEM education and then to articulate the unique contributions that NASA can and should make. ORGANIZATION OF THE REPORT This report reviews NASA’s K‑12 STEM education projects. It specifi- cally focuses on the purposes identified by Congress in its mandate to the study committee. Furthermore, it provides guidance to NASA’s continued efforts to support K‑12 STEM education. This chapter introduces the goals and scope of the study, and previous efforts to review NASA’s and all fed- eral agencies’ K‑12 STEM education projects. Chapter 2 provides an overview of NASA’s K‑12 STEM education program, a historical account of NASA’s involvement in this area, and a description of NASA’s K‑12 framework for education projects within the headquarters Office of Education, the mission directorates, and the centers. Chapter 3 illustrates the role of the federal government and federal agencies in K‑12 STEM education. It specifically outlines the role of federal science agencies, and NASA in particular. Chapter 4 evaluates the NASA portfolio in K‑12 STEM education based on briefings from NASA staff, administrative documents, annual reports, recent external evaluations, and research in K‑12 education regarding best practices in professional development, curriculum, instruction, and school reform. Particular attention was paid to program design and effectiveness in regard to the seven core Office of Education projects. Chapter 5 critiques NASA’s previous project evaluations and provides a framework for guiding future project evaluations. Chapter 6 presents conclusions and recommendations. Based on these conclusions and recommendations, the report specifically answers the four congressionally mandated questions described earlier in this chapter.