Click for next page ( 74


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 73
7 Evaluation of Professional-Development Programs Large amounts of federal and private money are being spent on innovative programs for the professional development of K-12 science teachers, and many people are devoting tremendous amounts of time and energy to this critical com- ponent of excellent science education. How do we know whether these resources are productive whether a program's objectives have been accomplished? We need to be able to determine which programs are working best and which are not, and we need to know whether a program has provided participants with new insights, knowledge, or skills and whether it has led to improved achievement among students. We have few data on the long-term impact of the programs; obtaining objective data on such programs is daunting and expensive. The prob- lem is not new. If we can learn from the past, we should be able to do better in the future. Much of what we know about the effectiveness of professional-development programs is based on anecdotes and on reports from teachers, principal investiga- tors, and program directors involved with the programs themselves. In our re- view of professional-development programs, we heard repeatedly that teachers felt empowered by their participation and gained an enhanced sense of profes- sionalism. We also heard that many teachers particularly elementary-school teachers felt that they had increased their content knowledge and were more comfortable in using inquiry-based methods of instruction in their classrooms. That kind of subjective information is important and useful, but the overwhelm- ing majority of programs we examined had no formal mechanisms for determin- ing the effectiveness of programs by assessing how students fared after partici- pating in programs in which their teachers participated. 73

OCR for page 73
74 PROFESSIONAL DEVELOPMENT OF SCIENCE TEACHERS We are concerned about that situation because of the lessons learned in the 1980s. In 1984, the General Accounting Office concluded that although the NSF-sponsored teacher institutes of the 1960s and 1970s had favorable effects on teachers, as measured by attitudinal surveys, there was no empirical evidence to support the contention that the institutes had favorable effects on students. At the time, the costs of evaluation were not allowed in education grants and there might have been a dearth of instruments available to measure effects on students. A decade later, there are still an inadequate number of techniques to measure and evaluate the effects. At a time of federal budget cutbacks, the lack of evidence linking spending to student achievement was used as one excuse for the elimina- tion of science-education programs. Federal appropriations for science educa- tion practically disappeared from the political landscape until another "crisis in education" had begun again, this time driven by consensus about science literacy and economic competitiveness rather than national security. We need to know the effects of professional-development activities on the classroom behavior of teachers, such as the extent to which they incorporate the content and process elements of their training into their classroom teaching. Obviously, we must evaluate student performance what they know and are able to do as a result of their teachers' professional-development activities. We need to develop measures of the cost effectiveness of various programs on classroom behavior. Ultimately, we need to design ways to collect longitudinal data to measure the effects of professional-development programs for teachers on their students, including how they learn and make decisions beyond high school. Obtaining such data will require perseverance to collect and analyze comparable data over periods of 5-10 years. Funding to support those long-term efforts must also be obtained. But if the barriers can be surmounted, we will gain better insight into how professional development affects teacher and student outcomes over time. This chapter presents ideas and recommendations for the development of effective evaluation tools that will yield the data we need. The primary question to be answered is, "Has the professional-development program being evaluated helped teachers to create a high-quality learning environment so that the students are doing better in science?" To develop evaluation strategies that answer that primary question, we must know what forms of evaluation are required, who should conduct evaluations, what tools and programs we need to conduct useful evaluations, what levels of funding are needed to conduct useful evaluations, and how the administrative structure supports program evaluation. FORMS OF PROGRAM EVALUATION Program evaluation can take many forms. Not all professional-develop- ment programs need to be evaluated in the same way. For example, a lecture

OCR for page 73
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS 75 series does not require as extensive an evaluation as a program designed to foster systemic reform. In its most general form, however, we believe that evaluation will be most effective if it is designed in the initial planning stages of a program, if it measures the success of a program against its stated goals, and if it continues throughout the life of the program and, for students, beyond. Concurrent Design and Evaluation of Professional-Development Programs It- program planners include evaluation in program design, they will find that it will help them to Define specific, realistic, important, and measurable program goals. Identify scientific content and science-process skills that are appropriate for teachers and their students. Choose instructional strategies and followup activities that are consistent with the objectives of the program and reinforce core concepts. Establish mechanisms for receiving continuing participant feedback. Establish, before the program begins, procedures and instruments for col- lecting overall program-evaluation data. . Examine a program's cost effectiveness or efficiency. The last issue is perplexing because it addresses the age-old problem of "comparing apples and oranges": how can one compare the relatively high cost of a high-school biotechnology program, with its expensive equipment, to the relatively low cost of an elementary-school science program that serves hundreds of teachers? Is the elementary-school science program more cost-efficient sim- ply because it has a lower per-teacher cost? Table 7.1 shows how the planning of a professional-development program and its evaluation are conceptualized as a seamless process. The sought-after goals of professional-development programs are listed in the first column and are based on the characteristics of effective programs identified in Chapter 2. The remaining columns show the steps toward the goals and what might be evaluated to assess the progress of the program. Continuing Evaluation Continuing evaluation (as contrasted with summary evaluation on comple- tion of a program) can include both informal and formal mechanisms to help program facilitators to identify problems (and successes) during various stages of program implementation. Informal evaluation can be conducted by program staff. Continuing evaluation often uses questionnaires, interviews with partici- pants, or self-reports in the form of journal excerpts; these types of evaluation

OCR for page 73
76 Cq be o a' o a' a' o .= Cq a' o o o .= o 3 a' a' be .Y EM o sit : - o. o v m 4= o 4= o Ed ca V, o .^ ca 0 ~ ca ~ 0 t,.4 ca 0 ca =.4 ~ 0 ~ 4= O o C) ~ ~ ca 0 lo ~ ca ca hi ~ Ct 8 ..= ~ ~ ~ ~ ~ ~ ~ ~) ca 0~e an R O R ~ a R R .- R O ~ o ~ R ca ~ ~d O S~ O ~ ~; ca ca ~,4 ~; ~ SO~ ~ ca ~ ~ O 0 ._4 S~ 4._, S-. ~; ~ O 0 ;^ ~ C) ca ~ U. S~ 4._, C) ._4 0 S-. ;^ ~ C) P C) ~ ca S~ O O C) ~; U:~ -~ ca 4= ca _ ca 4= O - 0 cd d R V, .~ ~ C) d ~ .~ 4= ._4 ~ C) t,.! ~ .~ ~ 0 3 d ca ca S~ d O .~ , 4= C) C) ca ca _4 4= 4= ca 1 - ~; ca _4 S~ ca .> i, 4= ~ d 1 ~ tM4 ~ . 1 Cd O S~ d 4= O S~ C) 4= 0 O _ ~ ca t~4 , ~ ~ _4 _4 ~ _ _ _ V, ~ 0 O _ ~ ~ s~ O ~ca 4= ~ s~ .0 O ~ - . .O V, .^ O ~ca 4= 4= p~ ~ ~ s- ~ ~0 ca4= O ~ ~ ~ ~ 4= ~ ~P ~ ~ O ~ ~ ~ o U E ~ ~ O d 4= . ~ v~ t-~ ~ o t~o cr' R V~ R R o P~ ~ ~ ~ ~ ~ R ~ t.~ ~ R ,~ tD Z ~ d o R C _ ~ ~ C R ~ ~ ~ d ~ R ~ 't ~ _ ~ ~ R R - ~ ~ O V, ,4 ~ ~ _ ~ 4= ~C) ~ ca P4 =) . ~4 ~ = O C) C) ca _ 4= Cd |4 V, 14 0 c,, 3 ca ~ _4 ~ i~ ~ C) 14 O d O . ~,4 -4 V, ~ - 4 {t {t O P44 4 ~P4 O C) ~ ~4 ~ Ct Cd "R~ O cr, .^ ca 3 8 _ ~4 C C ~ ca ~O ~,4 ,4 ~ 4 ~ V, P4 ,4 4,) 4= ~ ~4 4= ~ O C) P4 . ~ o] tM4 C) ca ~I ~ 4 ~ V, ~ _4 {t ~,4 ,O4 ~ ~ .~4 ~ ,= O O P4 Ct ~ ~ 4,) .,4 -4 _ _4 ~O =4 V, ~ca 4 0 O P4 ce ,4 =) -t O ~ O - 4 V, o .4= R Ce ~ >4 4~ O O d P4 ~4 ~ O E~ ~ _ ,4 C,) ca S ~4 5 ,4 4= ~ d 4= C' ~ 4= ~4 {t X ~ 4 o 54 ~ 4 ct {t P4 ~ V, ,O4 P4 ~ ~ ~d ~ P4 ~ t~ 0 -4 ~ O O O p O ~ O V, O 4-) ;^ ,4 ~ O 0 14 d 4= ~ ~4 0 14 ~4 .,4 ~ 4 4= _4 ~ .,4 0 ~ O _ ,4 - '~ c~ ca 4 ~ O O .4~4 ,4 _ ca - 4 C,) ,4 O Ct 4.) 4= ~ 4') .' 4 d _ ~ 4 4= "4 4= 0 4= ca ,4 ,4 ~4 4= ca ^C {t ~ 4 ~ ~0 4 {t =4 p4 4= ~ 4= .,4 ~ C) ~ .r _4 P4 ,4

OCR for page 73
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS 77 should rely heavily on participants' comments so that appropriate changes can be incorporated into the program. Often, continuing evaluation leads both to better ways to achieve the initial goals and to changes in the goals themselves as perceived needs change. The evolution and improvements in programs that result from observations made during continuing evaluation are desirable, and all the most-effective programs that we studied used it. However, the changes and improvements in programs that result from continuing evaluation complicate long-term evaluation of program effectiveness because it is aiming at a moving target. Long-Term Evaluation Formal evaluation of the impact of an overall program requires long-term strategies for data-gathering and analysis that begin with the program's design and continue throughout the life of the program. Most evaluation stops when a program ends, although it can take years for the impact on students to become apparent. Usually, long-term data are not collected, although their collection might be as simple as tallying the number of science electives taken by students of a teacher in a middle-school program. Such data provide a quick indicator of students' interest in science, which might or might not reflect good science teach- ing in earlier grades. We found a lack of overall program evaluations linking teacher participation in professional development with enhancement of teaching skills or student per- formance. To determine the ultimate impact of a program, long-term evaluation is needed to keep track of program participants and how they incorporate new information and techniques into their classroom activities. Whether evaluation is intended to be continuing or summary, fundamental questions must be addressed: What are the teachers learning? Is effective peda- gogy being modeled in the professional-development sessions? Does the pro- gram address "real needs" of teachers? Does the program hold promise of favor- ably affecting student learning in science? Other questions that one should consider in the evaluation are the following: Is the program built on models of other well-tested programs? Answers to this question can reveal whether program planners have educated themselves about other professional-development programs around the country rather than inventing new programs from scratch. Does the program engage participants in open discussion about educa- tional practices, reform, curriculum, and policies? Over the years, educational reforms have been piled on other educational reforms. Veteran teachers have seen them come and go. For strategic reasons or out of skepticism, many teachers wait in silence for each fad to pass so that they can go back to what has worked best for them. Open discussion is a sign of active participation and can reflect a

OCR for page 73
78 PROFESSIONAL DEVELOPMENT OF SCIENCE TEACHERS deep structural change in how schools work (that is, adjustment of a structure to accommodate participating teachers' input). Does the program have or plan to develop a relationship with systemwide initiatives in the region? Answers to this question will provide evidence about the extent to which a program is complementary to or coordinated with systemwide efforts. Are participants particularly teachers being intellectually stimulated and inspired by the program? Answers to this question might shed light on whether teachers are motivated to participate in other professional-development activities or whether, in the long run, they choose to stay in the field. Who is participating in the program and why? Answers to this question will reveal whether the program is reaching all teachers or only those who are already motivated. Evaluators should also examine the diversity of the partici- pant pool with respect to the age, race, and ethnicity of students taught and teachers' race, social class, work habits, age, teaching experience, and stage of career. Answers to this question might lead to the development of strategies to involve less-motivated teachers in professional-development activities. Does the program result in changes in teachers' teaching practices and in student learning? Answers to this question require that data be collected that could show changes in teachers' behavior and student performance. Evaluators need to seek creative indicators of these changes. For example, the adoption of hands-on or innovative laboratories could be monitored through records of the purchase of necessary supplies and equipment assuming that these are being used in the classroom. Changes in lesson plans and laboratory activities might also indicate that teachers have altered their practices, as might videotapes of classroom activities when "before and after" tapes exist. . EVALUATORS As noted above, informal, continuing evaluation can be conducted by pro- gram staff using questionnaires, interviews with participants, or self-reports in the form of journal excerpts. We found that formal evaluation is usually con- ducted by principal investigators or program directors. For science-based pro- grams, the principal investigator, hence the evaluator, will probably be a scientist. Special problems face him or her. Scientists are trained to plan careful experi- ments that generate data to support or refute a hypothesis. In education, objective data are much harder to generate because the variables are harder to identify and control. Scientists are accustomed to having responsibility for the implementa- tion of a program but not for its evaluation. They are unlikely to have training in the evaluation of educational programs. Unlike scientific research, whose prod- uct is a peer-reviewed paper, the "product" we are dealing with an education program involves human interactions and is not as readily subject to peer re- view. Scientists therefore must be aware of the complexities inherent in analyz

OCR for page 73
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS 79 ing educational programs. Finally, it is difficult for any principal investigator, whether scientist or science educator, to oversee an objective analysis of his or her own program. Evaluation does not have to be only a means to determine the "success" or "failure" of a program or individuals. If education researchers who are trained in evaluation are involved from the beginning of a program, evaluation can also be a means of conducting research on professional-development programs. Educa- tion researchers who serve as evaluators can help to bridge the gap between education research and practice. They can help to design programs that have measurable components. And they can produce overall program reports that highlight difficulties faced during program implementation as a means of letting other program directors know how to improve their programs or avoid mistakes. When evaluation is seen as a means for educating others in the field, program staff might feel that they can be honest in their evaluation of their own programs. Having an evaluator involved in the planning of the program can help to ensure that program objectives are clear and focused, that the evaluator will begin to think about evaluation strategies and instruments before the program begins, and that the program will include appropriate points for the evaluation of progress and midcourse correction. One warning: As always, balance is necessary. As one increases one's focus on program evaluation, one needs to be careful not to contrive neat evaluations by looking for easily measured outcomes or easily administered tests at the expense of effective program design and implementa- tion. EVALUATION TOOLS In theory, well-designed evaluations of professional-development programs for science-education reform should be based on testable hypotheses. In reality, most current evaluation strategies fall into the category of using "How did you like the course?" questionnaires before teachers have had the chance to try new techniques in the classroom. Instead, long-term followup evaluations of teachers to determine how they are implementing their new knowledge and techniques in the classroom are needed, as are new evaluation instruments that can measure the long-term effects of programs on both teachers and students. Much research on ways to conduct useful evaluations is still needed. Re- search could be conducted on existing programs to produce a comprehensive guide to evaluating professional development (including a computer database for ease of access and updating) that would provide program directors and other interested parties with specific examples of effective evaluation techniques and tools. The document could address such issues as measurement of teacher and student conceptual understanding, application of inquiry-based learning, effects of professional-development programs on students (with samples of assessment tools for measuring student outcomes), and change in teaching practices or school

OCR for page 73
80 PROFESSIONAL DEVELOPMENT OF SCIENCE TEACHERS reform. The document should include evaluation research being done interna- tionally. FUNDING AND ADMINISTRATIVE ISSUES Most funding agencies and private funders request information about pro- grams' effectiveness, but few provide funds or guidance to conduct full-scale evaluations. Effective evaluation procedures are expensive, and available fund- ing is inadequate for the kinds of evaluation that are necessary to assess the linkages between teacher preparation and student performance. Budgets for programs normally do not include a line item for evaluation. Evaluation of a program's effects will be most valid when time has passed since the program ended. Only then can the impact on students be noted. However, most funding agencies require a grant-program report before an appropriate evaluation time. Funding agencies should allocate more of their resources to support both evalua- tion of selected existing programs and evaluation research. NSF previously has placed the responsibility for evaluating programs on principal investigators, most of whom do not have a background in survey or social-science research. NSF's program officers, overwhelmed with large num- bers of grants, have little time to provide feedback to grantees about their grants or to follow up on programs. The average tenure of a program officer in the Education and Human Resources Directorate is 2 years, but most grants run for 3 years. Considering that a grant is in preparation, review, and approval for about 9-12 months, it is possible to be assigned to three different program officers over a single grant period. Rarely does an NSF program officer stay long enough to see a round of grants through to completion, much less the next round of proposal submissions as those programs are submitted for renewal. Newer NSF policies recognize the need for outside evaluation. RECOMMENDATIONS Those involved in professional-development programs should recognize the importance of establishing an evaluation plan from the onset of program design. That process forces a clarity of thinking about program objectives and activities in much the same way that careful planning of one's scientific research forces clarity of thinking about experimental design. Every professional-development program should include evaluation mechanisms from the beginning. The evaluation should continue for the life of the program and seek feedback from all participants. Because formal long-term evaluation of an entire program requires more extensive analysis than informal evaluation, it is not cost-effective for every program. Instead, we recommend that clusters of similar kinds of programs, such as biotechnology or systemic programs, or all programs in a geographic region be

OCR for page 73
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS 81 reviewed as a group. The goal is to learn about the net effect of various kinds of programs on teachers and students and to identify strengths, weaknesses, and gaps in content, pedagogy, or geographic distribution. Program directors and principal investigators should not necessarily have primary responsibility for long-term evaluation of their own programs; these persons often do not have sufficient background in the methods of social-science and education research and might not be objective about their own programs. Ideally, professional-development programs include a budget item for a qualified evaluator, who should be involved from the onset of program. We concur in previous recommendations to NSF that continuing evaluations (and evaluation specialists) be considered "legitimate uses of project funds" (SRI, 1988~. Scientists and other program developers should work with colleagues in the social sciences and education to identify experienced program evaluators who can help to plan and participate in overall program evaluation. . Funding agencies should support more research about effective evalua- tion methods. NSF should fund the development of a comprehensive guide to evaluating professional development that includes a compendium of educational research and examples of effective evaluation. The task is complex and requires evaluators who have formal training in program evaluation. A recent publication by the Council on Foundations, which includes nine extensive case studies of evaluation projects, is a start in this direction (Johnson, 1993~.