Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary T he National Oceanic and Atmospheric Administration (NOAA) is responsible for understanding and predicting changes in the Earth’s environment and conserving and managing coastal and marine resources to meet the nation’s economic, social, and environmental needs. Since it was created in 1970, the agency has supported education projects that cover a range of topics related to the agency’s scientific and steward- ship mission, including oceanic, atmospheric, climate, and environmental sciences. Given human dependence on the Earth for health, well-being, and economic growth, the importance of these interconnected fields and environmental stewardship cannot be overstated. Education efforts at NOAA are distributed across a range of internal offices. Some of them have long had mandates to engage in education activi- ties, but it was not until 2007 that NOAA received an agencywide mandate for education through the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Act. The act calls for NOAA to support and coordinate formal and infor- mal educational activities to enhance public awareness and understanding of issues related to its mission. The act also requires that NOAA develop a 20-year education plan, to be reevaluated and updated every 5 years. The Committee for the Review of the NOAA Education Program was established by the National Research Council (NRC) to take stock of the existing education portfolio and review the education strategic plan man- dated by the America COMPETES Act. The committee was specifically asked to comment on: 

OCR for page 1
 NOAA’S EDUCATION PROGRAM 1. NOAA’s role in education, 2. its education goals and outcomes, 3. the composition and management of its education portfolio, 4. its education evaluation practice, and 5. the impact of its education efforts. The committee followed an iterative process of gathering information, analyzing and deliberating on it, identifying gaps and questions, gathering additional information to fill these gaps, and carrying out further analysis. A contract between NOAA and NRC determined time and resources avail- able for the study and constrained the scope of the committee’s review to existing documentation, site visits, testimony from NOAA staff, and com- missioned papers. The resulting report provides a summary of the national education context for NOAA’s role in education (Chapter 2) and of the education strategic plan and its strengths and weaknesses (Chapter 3). It also describes the individual education projects (Chapter 4). In Chapter 5, the education evaluation approach of the agency is described, and sugges- tions for improving the process are provided. The final chapter presents the committee’s conclusions and recommendations. NOAA’S ROLE IN EDUCATION The national need to educate the public about the ocean, coastal resources, atmosphere, and climate and to support workforce development in related fields is well established. The federal government role in address- ing these needs as part of the national effort is also widely accepted. NOAA’s role in education has been recognized for approximately 30 years, as evidenced by the mandates to engage in education activities given to individual operating branches and programs, and more recently by the America COMPETES Act. The agency has a broad mandate to engage in and coordinate education and stewardship initiatives related to ocean, Great Lakes, climate, and atmospheric science, as well as other fields related to its mission. NOAA must fulfill these responsibilities in the context of a national effort, implemented at state and local levels. The agency must use formal and informal learning environments to improve learning and under- standing of science, technology, engineering, and mathematics (STEM) and to advance environmental education. Although NOAA is unique among federal agencies in its focus on stewardship and on ocean, coastal, Great Lakes, atmospheric, and climate science, its mission overlaps with and complements the missions of other federal agencies. Many federal agencies, institutions of higher education, and private and nonprofit organizations have additional resources that help improve the nation’s understanding and interest in the relevant sciences

OCR for page 1
 SUMMARY and that help develop strategies to care for the environment. However, coordination of these activities in a cohesive way that leverages the unique assets of each federal agency, as well as the formidable infrastructure and capabilities outside the federal government, has proven to be a challenge. NOAA can contribute to national education efforts through a variety of programs and assets, including modern and groundbreaking technologies and discoveries; research equipment; data sets; technical staff, including sci- entists, engineers, and researchers; stewardship and management of natural resources; specialized education expertise; partnerships; and connections to local, regional, national, and international stakeholders and natural resource managers. In addition, NOAA is one of the key federal agencies engaged in management and stewardship of the coasts and oceans. These natural environments can support important educational opportunities and provide the agency with connections to the surrounding communities and organizations concerned with environmental issues. NOAA’s role in education is shaped by the distributed nature of its education efforts across the five line offices and the Office of Education, the small number of agency staff involved in education, and its small education budget. Because of their diverse missions, the line offices (some of which have individual education mandates) and the Office of Education can act independently and sometimes even in competition with each other. The majority of education programs are usually implemented by an individual or a small team at a particular location. And NOAA’s education budget is relatively small in comparison to that of other federal agencies engaged in STEM education, such as the U.S. Department of Education, the National Science Foundation, the National Aeronautics and Space Administration, and the U.S. Department of Energy. Limited education resources and the inherently global nature of NOAA’s mission make strategic partnerships necessary in order for the agency to accomplish its ambitious goals. Clear education goals, planning, and stra- tegic use of resources are critical aspects for effective partnerships. NOAA can play a supporting role in state and local education systems and a leadership role in federal STEM education endeavors specific to oce- anic, coastal, Great Lakes, atmospheric, and climate sciences. Such efforts will be most productive if they align with local education needs and national education standards, because education activities and products that do not consider the needs of the potential audiences are less likely to be successful. Recommendations Regarding NOAA’s Role in Education Recommendation I.1: NOAA should fulfill its role in education through the use of:

OCR for page 1
 NOAA’S EDUCATION PROGRAM • agency and external expertise in science, engineering, technology, and education; cutting-edge scientific research and exploration activities; internationally collected datasets; and advances in tech- nology and engineering; • place-based assets that directly connect local issues to national and global science and stewardship issues: marine sanctuaries, estuarine research reserves, fisheries activities, and other natural resources protected and managed by federal, state, and local entities; • partnerships with local and state education infrastructure, aca- demic institutions, government agencies, business and industry, and private-sector and nonprofit organizations; and • the agency’s global science and international partnerships. Recommendation I.2: In order to adequately address the mismatch between its available resources and its ambitious education agenda, NOAA should better align and deploy its resources. This may require the termination of certain activities and programs that, based on appropriate evaluation, do not directly and effectively contribute to its education and stewardship goals. Recommendation I.3: Within the constraints of NOAA’s mandates in edu- cation, the agency should continually evaluate where it leads, collabo- rates, follows, or declines to participate in partnerships with others. These decisions should be guided by consideration of the agency’s role, assets, resources, and priorities in education and the strengths and missions of other agencies, institutions, and organizations engaged in education. EDUCATION GOALS AND OUTCOMES The NOAA education strategic plan for 2009-2029, developed by its recently formed Education Council, provides goals, outcomes, and a framework to organize a large set of individual education activities into a coherent portfolio. The plan outlines two goals: (1) to advance the envi- ronmental literacy of the nation, and (2) to promote a diverse workforce in oceanic, coastal, Great Lakes, atmospheric, and climate sciences. At this time, NOAA is developing a strategic implementation plan to specify how it will accomplish these goals. The strategic plan lists six outcomes under the goal to improve environmental literacy and three outcomes under the goal to promote a diverse workforce. The plan has multiple strengths. It includes appropriate goals of sup- porting environmental literacy and workforce development and stresses the need for partnerships with appropriate agencies, institutions, and organi- zations. The plan also illustrates a commitment to developing education programs informed by evidence about effective practices and contributing

OCR for page 1
 SUMMARY to this body of knowledge. In addition, there is an emphasis on the use of ocean, coastal, and other place-based resources as unique and valuable assets for learning. Overall, the 2009-2029 education strategic plan is a step forward from the previous education strategic plan. The plan is not without weaknesses. In the evaluation literature, out- comes are typically thought of as measurable changes or absolute levels of performance that can be expected as a result of efforts to reach a goal. How- ever, only three of the six environmental literacy outcomes in the education strategic plan and two of the three workforce development outcomes align with these expectations. The other outcomes describe strategies or processes that might contribute to reaching the goals. In addition, although diversity is a focus of the workforce goal, there is no mention of diversity or broaden- ing participation in the environmental literacy goal. It is also unclear how NOAA can accomplish its goal of supporting the creation of a “world-class” workforce without a clear understanding of its own and the nation’s work- force needs in the relevant areas. Although the importance of partnerships is stressed in the plan, there is no specific guidance about how or with whom to partner to connect to the national STEM infrastructure and human capacity. Finally, the use of the term “NOAA science” in the strategic plan is confusing. It is unclear whether this term is meant to refer to the science conducted by NOAA scientists, the research or the results of research funded by NOAA, or any science conducted on topics related to NOAA’s mission. Recommendations Regarding Education Goals and Outcomes Recommendation II.1: NOAA education programs should formally address broadening participation of underrepresented groups as an important out- come through all phases, from the initial stages of planning through imple- mentation and evaluation. The environmental literacy goal, in particular, should include outcomes related to reaching out to underserved and under- represented communities. Recommendation II.2: To reach NOAA’s environmental literacy goal, the Education Council should develop its implementation plan and future revi- sions of the education strategic plan to: • clarify how it will capitalize on scientific findings, engineering advances, and stewardship activities that relate broad national priorities to local concerns to engage individuals of all ages in education; • articulate how NOAA education programs will draw on the sci- entific, engineering, research, and other expertise accessible within the agency as well as in the broader community;

OCR for page 1
 NOAA’S EDUCATION PROGRAM • address the mismatch between the lack of an outcome related to stew- ardship and the focus on stewardship outcomes in local programs; • consistently define outcomes as measurable concepts that allow an assessment of whether a goal is being reached, to clearly distinguish outcomes on audiences (impact) from outputs of activities; and • provide more opportunities for local and regional education staff from all education programs to share effective practices and lessons learned. Recommendation II.3: To achieve the workforce development goal, the education strategic plan, the education implementation plan, or both should call for periodic assessment of the current and anticipated needs in fields critical to NOAA’s mission to guide investment in appropriate workforce development activities. Recommendation II.4: NOAA education programs should draw from cur- rent and relevant scientific and engineering advances regardless of what agency, institution, or organization they are originated or funded by. COMPOSITION AND MANAGEMENT OF THE EDUCATION PORTFOLIO NOAA supports a wide range of education programs for varied audiences that include K-12, postsecondary, graduate, and informal education activities with local, regional, national, and international scope. NOAA has developed professional development programs, classroom materials, curricula, museum exhibits, place-based learning experiences, literacy documents, and other products. The audiences of the agency’s education programs include teach- ers, students, scientists, and the public. A coherent, coordinated education portfolio is needed for achieving goals effectively and efficiently, for sharing successful strategies to engage and teach different audiences, for the pooling of resources to support synergistic activities, for developing cross-discipline activities, and for sustaining consistent education strategies. Management of a federal education portfolio is complicated, and NOAA has characteristics that make it particularly challenging. Individual education programs may have separate mandates and often have local components with local control. Education programs are managed differ- ently across the line offices and the Office of Education as a result of avail- able resources for education (staff and funding), separate missions, and individual education mandates. The differences in management structures, missions, and education mandates are obstacles to creating a cohesive and coordinated education portfolio.

OCR for page 1
 SUMMARY The Education Council is the primary means for NOAA to manage its education portfolio. Although relatively new, the Education Council, led by the Office of Education, serves an essential, high-level internal coordinat- ing function. The Education Council led the development of the education strategic plan and is developing the collaborative working relationships necessary to implement it. However, the Education Council does not have budgetary or institutional control over the education efforts of the line and program offices, which limits its effectiveness in carrying out the agency’s education mandate. Since the education portfolio has developed in the absence of an over- arching strategic direction and without a system to monitor or catalogue activities, it is difficult to assess its composition, balance, and impact. Such a system is needed to make informed decisions about the balance of the portfolio. What is clear, though, is that to date NOAA’s education programs have been focused more on ocean or coastal concepts and issues and less on climate and atmospheric ones. Efforts are emerging to bring greater atten- tion to climate and atmospheric issues and concepts across the agency’s education activities. Recommendation Regarding Composition and Management of the Education Portfolio Recommendation III.1: NOAA should develop and implement a system to monitor and catalogue its education portfolio and guide decisions regard- ing what programs should be developed, continued, modified, or ended. In balancing the portfolio, the Education Council should • increase attention to climate and atmospheric science education programs to complement the current focus on ocean science. These programs should emphasize the strong connections and interac- tions among the ocean, the atmosphere, the land, and human and nonhuman species; • provide purposeful attention to both STEM learning and steward- ship goals so as to enable synergies; and • make decisions based on national education needs, the education pri- orities of the agency, and a clear picture of its education portfolio. EDUCATION EVALUATION PRACTICES The challenges of carrying out appropriate evaluations of education ini- tiatives are large. Most federal science agencies are struggling to meet these challenges. NOAA is giving increasing attention to evaluating its education

OCR for page 1
 NOAA’S EDUCATION PROGRAM initiatives. The strategic education plan 2009-2029 highlights the need for more comprehensive evaluation of NOAA education initiatives. NOAA has conducted evaluations of a small number of its educational activities, and these evaluations are limited in scope and quality. Summative evaluations have been carried out on a very small proportion of education activities, and there has been little consideration of evaluation that would enable it to recalibrate the entire portfolio to effectively meet its goals. The evaluations that have been carried out tend to focus on short-term and intermediate rather than long-term outcomes; rely on participant opinion, feedback, beliefs, and knowledge; and usually do not address outcomes related to attitudes or behavior. Outcome-based evaluations generally lack control or comparison groups or other ways to attribute potential changes solely to the education efforts themselves. The Education Council is increasing its emphasis on evaluation and mov- ing toward comprehensive program evaluation through the adoption of the Bennett Targeting Outcomes of Programs (TOP) model. The adoption of a uniform model as a framework to guide evaluation strategies and practices across all education programs is a useful step and may help implement a more strategic, coherent approach. However, as is the case with most evaluation models, the TOP model does not include specific guidance regarding the implementation of evaluations or how to design high-quality ones. Data are needed for several purposes, including project monitoring, fiscal due diligence, and program evaluation. NOAA needs a systematic way of collecting data for each purpose to ensure that data are comparable across programs and initiatives and provide useful information for balanc- ing the portfolio and assessing the strategic alignment of programs in it to the agency’s overall goals. Recommendations Regarding Education Evaluation Practices Recommendation IV.1: The Education Council should continue to improve the evaluation expertise of its education program managers, contract with external evaluators for summative evaluation, and require the incorpora- tion of the most appropriate and rigorous evaluation strategies during program development to guide design, continual improvement, and delivery of its education programs. Recommendation IV.2: The Education Council should increase the empha- sis on high-quality evaluations. Summative evaluations should focus on the program outcomes related to learning and stewardship, not only satisfac- tion with education experiences, and should use the most appropriate and rigorous evaluation designs.

OCR for page 1
 SUMMARY Recommendation IV.3: The Education Council should consider develop- ing a number of approaches to inform strategic portfolio management and how evaluation findings can be used to inform decisions about portfolio balance. Recommendation IV.4: Education programs should evaluate internal col- laboration among line offices and between education and operational and scientific staff, as well as the quality of external partnerships with other agencies, institutions, organizations, and the broader STEM communities. EVIDENCE OF IMPACT Although NOAA has created a large number of education initiatives with its limited education budget, there is evidence of impact for only a small proportion of them. The majority of initiatives that have collected information assessed only their scope and reach, which are not sufficient to judge impact. On the basis of the available evidence, all that can be said about the impact of such programs is that they are positively perceived by the participants. However, as yet, NOAA education programs serve a relatively small proportion of the nation’s population. There is a growing body of literature regarding effective practices in formal and informal science education, behavior change, reaching under- served populations, and workforce preparation. This literature can be used to support the development of science education programs that are likely to be successful. Recommendation Regarding Evidence of Impact Recommendation V.1: NOAA education staff should draw on evidence from education research, evaluations of NOAA programs, and external education expertise to identify and implement effective practices for sup- porting education activities. Overall, NOAA’s education staff is dedicated and passionate about addressing areas related to the agency’s mission. Among NOAA’s most valuable assets, they have developed diverse education activities for a wide range of audiences and regions. The agency is to be commended for its historic commitment to education, which precedes the agencywide congressional mandate on education. The agency’s current education strategic plan is a significant improvement over the previous one. We hope that our recommendations continue to help NOAA improve its education efforts.

OCR for page 1