6
Design Principles for Fostering Science in a Federal Education Research Agency

The federal government has an important and legitimate role in supporting research as a public good, including research in education (e.g., National Research Council, 1999d; President’s Committee of Advisors on Science and Technology, 1997; Geweke and Straf, 1999). The federal government’s role in education research dates back to the middle of the nineteenth century, when the U.S. Department of Education was established1 to collect statistics and provide exemplary models for the nation’s schools. Then as now, the nation recognized the value of centrally generated education research that should be made available to all states, districts, and schools. In the absence of a federal leadership role, knowledge gained by one state or district that might be relevant to others would not likely be widely distributed, as individual states tend to undervalue the benefits that would accrue to others. Moreover, many scientific studies contrast alternative education approaches or models, and important comparisons are frequently made across states, districts, and schools. The federal government is also the natural place to collect and make data widely available on education performance, costs, processes, inputs, and their interrelationships.

1  

The U.S. Department of Education was first formed in 1867; its name was changed to the Bureau of Education shortly thereafter and later to the Office of Education. The modern U.S. Department of Education was established in 1979.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 127
Scientific Research in Education 6 Design Principles for Fostering Science in a Federal Education Research Agency The federal government has an important and legitimate role in supporting research as a public good, including research in education (e.g., National Research Council, 1999d; President’s Committee of Advisors on Science and Technology, 1997; Geweke and Straf, 1999). The federal government’s role in education research dates back to the middle of the nineteenth century, when the U.S. Department of Education was established1 to collect statistics and provide exemplary models for the nation’s schools. Then as now, the nation recognized the value of centrally generated education research that should be made available to all states, districts, and schools. In the absence of a federal leadership role, knowledge gained by one state or district that might be relevant to others would not likely be widely distributed, as individual states tend to undervalue the benefits that would accrue to others. Moreover, many scientific studies contrast alternative education approaches or models, and important comparisons are frequently made across states, districts, and schools. The federal government is also the natural place to collect and make data widely available on education performance, costs, processes, inputs, and their interrelationships. 1   The U.S. Department of Education was first formed in 1867; its name was changed to the Bureau of Education shortly thereafter and later to the Office of Education. The modern U.S. Department of Education was established in 1979.

OCR for page 127
Scientific Research in Education Assuming a legitimate federal role in education research, this chapter addresses the question: How should a federal education research agency be designed to foster scientific research in education, given the complexities of the practice of education, the stringencies of the scientific principles, and the wide range of legitimate research designs? While our focus is on design principles for a single agency, we point out that education research of national interest has historically been supported by several offices in the U.S. Department of Education, by other agencies of the federal government, and by private organizations (e.g., foundations). A federal agency is only one part of this larger enterprise, but it occupies a central place within it. Indeed, while the committee makes a number of suggestions for one agency to lead the scientific enterprise, we recognize that some of the tasks might best be conducted in partnership with other agencies or nongovernmental organizations, and we encourage the exploration of such options. Within this broader context of scientific research in education, this chapter takes up the specific issue of how a federal research agency might be designed to best fulfill its role in the scientific enterprise. Our approach in this chapter is forward looking. Throughout, we speak of a generic agency because the committee wanted to free its deliberations from exclusive consideration of the current incumbent, the U.S. Department of Education’s Office of Educational Research and Improvement (OERI). Although this report is in part intended to help policy makers think about the pending reauthorization of OERI, the committee was not charged with, nor did it conduct, an evaluation of OERI. Rather, we relied on data we have collected from a sampling of federal social science research agencies and programs—including OERI—about how they support their science missions.2 In short, while we reiterate that we did not evaluate OERI, we clearly could not avoid learning about it or discussing it, especially in a comparative way, to address our charge effectively. Thus, 2   Specifically, we collected data from OERI, the Social, Behavioral, and Economic Sciences and Education and Human Resources Directorates at the National Science Foundation, the Child Development and Behavior Branch at the National Institute on Child Health and Human Development, and the Social and Behavioral Research Program at the National Institute on Aging.

OCR for page 127
Scientific Research in Education throughout this chapter we refer to OERI and other agencies, most often comparing various aspects of funding and operations among them. We also relied on information the committee gathered at a workshop it sponsored in March 2001 that featured panels of senior officials from these and other agencies as well as knowledgeable experts about the federal role. The participants discussed the federal role in education research and related social sciences across several agencies with an eye toward the future of a federal education research agency (again, OERI was one of several agencies represented and discussed). This event is summarized in a workshop report (see National Research Council, 2001d). Based on the information gathered at the workshop and through subsequent data collection, our guiding principles of science, the features of education that influence the conduct of research, and the nature of scientific progression, we develop six design principles around the notion of creating a scientific culture. We argue throughout this report that science itself is supported through the norms and mores of the scientific community, and we believe that cultivating these values within a research agency is the key to its success. We also note that decades of organizational fixes at the current agency have arguably not done much to improve its culture and, consequently, its reputation. Our focus on a scientific culture within an agency stems from the recognition that an agency in many ways reflects the field it supports, and vice versa. An agency’s success requires a strong group of scholars, and the broader community depends in part on a vibrant federal presence. Thus, our design principles emphasize the role of researchers to lead and staff the agency, to serve on advisory boards, to help synthesize the current state of knowledge, and to act as peer reviewers of proposals and programs of research. The principles also recognize the role of the agency in building the professional capacity of the field. Other themes in this report are embedded in the design principles as well. For example, we take up the issue of research ethics—an influential aspect of the education research enterprise (see Chapter 4)—from the perspective of the federal regulations that govern them. We also argue for flexible decision-making authority in the agency to accommodate the dynamic nature of scientific progress and opportunity (see Chapters 2 and 3). And we suggest the agency attempt to enhance part of its research port-

OCR for page 127
Scientific Research in Education folio by finding ways to bring it closer to the complexities of educational practice (see Chapter 4). It is important to recognize the difference between this focus on developing a scientific culture and the focus on research methods in H.R. 48753—the bill that at least in part led to this report—and several related debates about the future of OERI. The language in the bill contains many of the key concepts we treat in this report, including systematic data collection, experimentation, rigorous reasoning, replication, and peer review. However, attempting to boost the scientific basis of federally funded education research by mandating a list of “valid” scientific methods is a problematic strategy. The inclusion of a list of methods—regardless of how they are applied in particular situations—erroneously assumes that science is mechanistic and thus can be prescribed. We have shown that science adheres to a set of common principles but its application depends greatly on the particulars of a given situation and the objects of inquiry. The definitions also make clear distinctions between quantitative and qualitative methods, implying that these two types of research approaches are fundamentally different; we argue the opposite. Furthermore, the use of definitions of methods as a tool for improvement fails to recognize the crucial role of theory and, as we emphasize, a strong, self-regulated, skeptical community of researchers that pushes the boundaries of knowledge. It is in this spirit that we focus on scientific culture in approaching the design of a federal education research agency. The committee recognizes an inherent dilemma in designing an agency to support scientific research in education. Scientific education research is often grounded in the practical problems of teaching, learning, and schooling and their varied contexts. Therefore, it is important to engage practitioners in the functions of the agency and to link research to the design and management of federal education programs. However, as we describe below, history has shown that a close bureaucratic relationship between research and educational programming in a federal agency can overwhelm the research function. Thus, we attempt to clarify the proper roles of researchers, practitioners, and politicians to ensure the needs of these com- 3   To view the text of the bill, go to http://thomas.loc.gov/ and search for H.R. 4875 in the 106th Congress.

OCR for page 127
Scientific Research in Education munities are met and their strengths maximized. We believe strongly that the responsibility for the success of the agency and the broader research effort it supports lies not solely with federal policy makers, but is shared among all those who have a stake in education and education research. Another dilemma has to do with the composition of the education research community itself. As we argue earlier in this report, we believe that the vast diversity that characterizes the field of education research is both a great strength and a troubling weakness. The variation in epistemological paradigms, methodological tools, and professional training lends the enterprise intellectual vitality. This same variation, however, is a source of cultural divisions among subfields that fosters isolation and impedes scientific consensus building and progress. In short, the “community” of scientists in education is really an eclectic mix of scholars with different norms and different standards of evidence. While we talk about the scientific community and its role in a federal agency as if it were a unified, easily identifiable group, the reality is more complex. The talent pool in education research is shaped by a number of structural, historical, and cultural variables, and parsing them out requires careful analysis. Thus, in the discussion that follows, we attempt to highlight issues that may be relevant in the implementation of the design principles vis-à-vis the field. Our vision is that the fundamental mission of a federal education research agency would be to promote and protect the integrity of scientific research in education with the goal of generating knowledge that can inform policy and practice decisions.4 To achieve this mission, the agency needs to develop and nurture a scientific culture, and to do so, it must have an infrastructure, supported by sufficient resources, that enables a cadre of experienced staff to make decisions flexibly and to interact continuously 4   Although our focus is on scientific research, we believe that the federal government should also fund related activities, such as development and demonstration work, a statistics function, a national library, other forms of educational scholarship (e.g., history and philosophy), and a research dissemination and implementation structure. Scientific research is related to, and often depends on, these functions. Indeed, we believe that it is the integration of scientific knowledge with insights from the humanities and other scholarly pursuits that will ultimately yield the most powerful understanding of education. However, we do not address them explicitly in this report because they are outside the scope of the committee’s charge to focus on scientific research in education.

OCR for page 127
Scientific Research in Education with the field it supports. We develop six design principles from these core ideas: Staff the agency with people skilled in science, leadership, and management. Create structures to guide agenda, inform funding decisions, and monitor work. Insulate the agency from inappropriate political interference. Develop a focused and balanced portfolio of research that addresses short-, medium-, and long-term issues of importance to policy and practice. Adequately fund the agency. Invest in research infrastructure. The rest of this chapter elaborates these principles and provides suggestions for specific mechanisms that could be implemented to support them. We stress that these suggestions do not reflect a view that there is one “model” that dictates the specific design features of any federal research agency. Indeed, the U.S. federal research enterprise is characterized by a range of structures and processes that is effective in respected agencies across the federal government (National Research Council, 2001d; Mathtech, 1996). DESIGN PRINCIPLE 1 Staff the Agency with People Skilled in Science, Leadership, and Management We begin with leadership and staffing deliberately: a scientific culture begins (and ends) with competent people. Attracting and retaining an adequate number of qualified leaders and staff is so critical to a healthy federal education research agency that we believe without it, little else matters. There is no substitute for leadership and human capacity. The leaders of the agency are of paramount importance. All federal agency leaders need leadership and management skills and, for this agency, the leaders—political appointees and career officials alike—must be respected educational researchers with strong scientific credentials. The culture of an organization emanates from its leaders; without research

OCR for page 127
Scientific Research in Education experience at the top levels, the norms and mores of science will likely not take hold. Similarly, the agency’s research staff should have extensive education research experience. They must have knowledge of relevant content as well as be able to recognize scientifically rigorous design, theory, data collection strategies, and analysis techniques. The agency should employ a mix of research staff, where promising junior scholars work alongside senior research staff to infuse new ideas into the agency’s work and to develop future senior staff of the agency. Providing ongoing professional development opportunities for research staff is also critical to allow continuing and sustained interaction with the broader research community. How can a federal education research agency attract and retain such human resources to develop and maintain a scientific culture that fosters and protects the integrity of scientifically rigorous research in education? This is a difficult question. In keeping with the range of hiring strategies of several existing federal research agencies,5 a federal education research agency should have the authority to pursue multiple approaches to develop its leadership and staff. Developing a core of permanent staff offers the benefit of increasing institutional knowledge and long-term relationships within the government and the research field. Short-term assignments can serve the dual purpose of updating the agency with new ideas from the field and acquainting university faculty and other researchers with the operations, needs, and 5   While all federal personnel actions are governed by Title V of the United States Code, many research agencies have exemptions from certain provisions regarding hiring practices to be able to retain the services of temporary scientific or technical employees. Within these parameters, agencies staff their organizations in very different ways. In the Social, Behavioral, and Economic Sciences Directorate at the U.S. National Science Foundation (NSF), for example, 40 percent of the current research staff is comprised of several types of temporary “rotators” who work in the directorate for a short time. These temporary appointments have historically been used to bring the expertise and perspective of active researchers into the operations of the agency. The Child Development and Behavior Branch of the National Institute for Child Health and Human Development (NICHD) at the National Institutes of Health (NIH), in contrast, has had only three short-term staff appointments since 1995; instead, it depends on permanent staff to convene workshops with those in the field for the purpose of ensuring it is in touch with the perspective of active researchers.

OCR for page 127
Scientific Research in Education accomplishments of the agency. For similar reasons, the appointment of postdoctoral fellows can be beneficial. These appointments also help build the capacity of future leaders in the research field. Another way for a federal education research agency to cultivate scientific norms in its staff is to engage in collaborative research efforts with other agencies to encourage interaction with staff who have traditions of supporting related scientific work. Such collaborations can enrich the breadth and depth of staff knowledge, and they also offer the benefit of developing cutting-edge, interdisciplinary research programs across federal agencies (National Research Council, 1999d). There are several current interagency efforts in education research from which lessons could be learned: examples include the Interagency Education Research Initiative, a partnership of OERI, NICHD, and NSF aimed at understanding how to scale up promising education practices, and a joint OERI-NICHD initiative focused on understanding how best to help bilingual students learn to read in English. Although these policy tools can help attract and retain top staff, staffing will depend heavily on related issues, such as funding, reputation, and leadership. For example, to the extent that education research is underfunded relative to other opportunities available to researchers (see Design Principle 5 below), top talent will likely go elsewhere, both in the field and in the agency. The early 1980s provides a lesson. With significant federal budget cuts in education and social science research, researchers migrated, especially to the health and national defense fields, many never to return. These funding deficiencies have affected OERI’s ability to attract and retain a cadre of capable staff despite having hiring authority similar to NSF and NIH. Staff levels were reduced drastically in the 1980s as a result of deep federal budget cuts, but even as its funding began to climb in the 1990s, the agency again lost 25 percent of its staff, including some of the most capable and experienced individuals (Vinovskis, 2000). The reputation of an agency and its leadership will also affect staffing. Developing a good reputation, of course, is not a simple matter of policy fixes. Adequate funding will help, but if the agency suffers from a poor reputation, its leaders will have to be creative about staffing possibilities and may need to convince a critical mass of researchers to serve at the same time. In this vein, the development of a scientific culture is critical; initial

OCR for page 127
Scientific Research in Education appointments have to be highly talented researchers who, in addition to being offered very attractive positions, should be encouraged to view such federal service as an important way to strengthen their profession. DESIGN PRINCIPLE 2 Create Structures to Guide Agenda, Inform Funding Decisions, and Monitor Work To accomplish its core tasks, a federal education research agency must be supported by a coherent system of governance. While we do not offer a comprehensive plan, we do believe that two essential elements of such a structure have the highest probability of cultivating scientific norms both inside and outside the agency: a high-level governing board and standing peer review panels of top-flight scientists. Governing Board Governing boards are common management and oversight tools for federal research agencies. We believe that particular attention to aspects of the board’s composition and responsibilities can further the development of a scientific culture within the agency as well as foster interactions with other stakeholders. We suggest that the agency operate under the general direction of a high-level governing board, drawn from leaders in education policy and practice, education research, business, and both political parties. The diversity of the governing board will allow the many different cultures associated with each representative group to learn from one another as they work toward common goals. Many research agencies currently have some kind of a governing or advisory board (e.g., the National Science Board of the NSF and OERI’s National Educational Research Policy and Priorities Board [NERPPB]). Such a board should provide advice to senior leadership, recommend research directions, help to safeguard the independence of the agency, provide critical links to practice and policy communities, and reinforce scientific norms inside the agency. A key task of this board would be to develop the research agenda. No matter how strong its science, if the agency does not carefully develop and maintain its agenda in close collaboration with researchers, policy makers, and practitioners and in alignment with available resources, it will fail to

OCR for page 127
Scientific Research in Education meet its mission. The challenges facing American education—low achievement levels, cost, growing numbers of second-language learners—are very real and demand serious investment. For education research to play a role, it is imperative that a federal education research agency have clear, longterm priorities for generating knowledge and promoting its transfer and utilization to engage the field in a collaborative effort toward understanding core issues in education. The agency might also include an agenda-setting committee, chaired by a distinguished practitioner, which would work with the board on the research agenda. Representatives from the scientific community should serve to help identify areas that warrant further research on the basis of the state of scientific development; this task may involve identifying areas of research that are ripe for immediate testing, or that require more basic descriptive research to generate scientific hypotheses or assertions. Representatives from the practice communities should serve to articulate the high-priority issues for educational improvement from a practical perspective. And representatives from the policy communities should serve to articulate short and enduring policy issues, as well as the feasibility of moving in new directions as recommended by researchers and practitioners. Another role for a governing board would be to report to Congress and the nation on the agency’s progress toward clearly stated and commonly shared goals. In the spirit of the 1993 Government Performance and Results Act (GPRA), the agency should be accountable for research results: knowledge generation and dissemination. Since research results are difficult to quantify, federal research agencies have struggled to comply with this law. A recent report (National Research Council, 1999b) provides some guidance on how to assess the outcomes of research for GPRA reporting. Plans and measures should be developed according to the character of the research program and acknowledge (as we do in this report) that the progression of science is jagged and often unexpected. Standing Peer Review Panels Peer review is the single most typically used mechanism for nurturing a scientific culture within and outside federal research agencies, and one that should play a feature role in a federal education research agency. In

OCR for page 127
Scientific Research in Education the ideal, peer review is both a process by which scientific work is assessed and funded and a product in that it provides a venue for the scientific culture of self-regulation (Chubin and Hackett, 1990) we describe throughout this report. The process works on several levels. First, by involving a group of active researchers, the current state of the art is introduced and used in judging proposed research. Second, especially when used as a feedback mechanism for the field (National Research Council, 2001d), the review process itself encourages the development of an active community of scientists working together on education problems: the process of reviewing proposals and communicating feedback fosters the development of common standards of quality and other scientific norms in the field over time. Third, the peer-review process acts as a buffer against outside political pressures to choose certain proposals or fund certain researchers regardless of scientific merit. A wide variety of peer review structures—ad hoc review committees, standing panels, mixture of outside and panel evaluations, and the like— can work. Indeed, the current federal system is characterized by this diversity of approaches (U.S. Government Accounting Office, 1999). For example, NIH uses both standing “study sections” and ad hoc review groups, while NSF and OERI only use ad hoc panels of reviewers for each competition. In contrast, the Office of Naval Research does not use panels of peer reviewers to fund proposals but rather regards its staff as peers able to make such decisions internally (National Research Council, 2001d). We believe that a federal education research agency ought to use standing review panels akin to the NIH study sections as its primary peer review vehicle. We envision these standing panels as providing continuity in overseeing research programs (see Design Principle 4). This suggestion reinforces the recommendations of several other groups (that are studying OERI in particular), including the RAND panels (see http://www.rand.org/multi/achievementforall/) and the NERPPB in its policy statements (National Educational Research Policy and Priorities Board, 2000). When researchers join peer review panels that have a life over a period of years, panel members strengthen their knowledge and the panel as a whole develops an integrated, communal expertise. Members then communicate this knowledge to their colleagues through their review of proposals and interaction with colleagues, and they also demand that, in the proposals

OCR for page 127
Scientific Research in Education funding in this way, we find a stark contrast between OERI and other federal social science research outfits across the federal government. The substantial and long-term investment made by NICHD in early reading research, for example, has reaped a significant return for the agency and the nation. NICHD has invested a total of $100 million over 30 years specifically to better understand phonological awareness and related early reading competencies. It was only through this substantial, sustained investment that, in conjunction with significant funding and intellectual contributions by other federal agencies (e.g., the Office of Special Education Programs in the U.S. Department of Education) and other countries, the research in this relatively focused area has been able to grow. By contrast, the scope of OERI’s research mandate is sweeping and its funding level modest. Its 1994 reauthorization established five institutes within the agency that roughly sketched its agenda. These five institutes include such broad categories as student achievement, students at risk of educational failure, education policy, early childhood, and postsecondary and life-long learning, and fund research through a range of mechanisms (e.g., centers, field-initiated studies). Each of these institutes spans academic subject areas (e.g., reading, mathematics, science, history) and in many cases educational levels (e.g., student achievement, policy, and at risk all span pre-K through adult learning). This categorization is a reasonable way to parse the field, and we would expect any federal education research agency to cover a similar breadth of content. However, it is unreasonable to expect that robust, research-based knowledge could grow out of them given the fact that roughly $130 million per year (fiscal 2000 level) must cover this broad scope. Over the course of its history, the primary research agency has had roughly the same agenda but large differences in funding levels. A 1992 National Research Council report charted the precipitous drop in funding for the National Institute of Education (NIE, predecessor agency to OERI) and OERI between 1973 and 1991. In 1973, NIE’s total budget was $136 million ($527.5 million in 2000 constant dollars). By 1991, only $78.4 million ($99.1 million in 2000 constant dollars) of OERI’s budget was allocated to research. This substantial drop in funding occurred with no commensurate change in the scope of its agenda. The report argued that OERI’s limited resources had been spread “so thinly that mediocrity was almost assured. Only a few lines of research have been sustained for

OCR for page 127
Scientific Research in Education the time they needed to bring them to fruition” (National Research Council, 1992, p. 3). To put this mismatch in dollars into perspective, the Tennessee STAR study (see Box 3-3), a single investigation in a single state that spanned 4 years, cost $10 million over its lifetime. Although total funding at OERI (adjusted for inflation) has risen substantially, nearly all of the increase has funded service-oriented programs with only tenuous connections to research. Since 1990, there has been a slight rebound in total education research funding, with the fiscal 2000 level at approximately $130 million (including nonresearch activities, OERI’s fiscal budget exceeds $800 million). We view this trend as positive, but believe that given the current breadth of the education research agenda, future increases will be necessary. In sum, we believe that if a federal education research agency is to have an agenda at least as ambitious in scope as its predecessors’, its funding must be higher than these agencies have had in the past. At the risk of overstating the obvious, we wish to make clear that this is not a call to simply “throw more money at research.” Money alone will not ensure the creation and accumulation of high-quality science-based knowledge in education— or any other field. Increases in funding must be targeted to important problems and attract the best researchers in the country to work on them. Thus, funding should increase as other design principles are institutionalized in a federal education research agency. In particular, steady growth in funding for research should occur as parallel investments are made in human resources in the agency and in the field of education research more globally, the topic we take up next. DESIGN PRINCIPLE 6 Invest in Research Infrastructure The infrastructure of any organization is the basic underlying system that determines how it functions to meet its mission. Research infrastructure includes a wide range of supports, but most commonly refers to scientists in the field (people), the tools those scientists have to conduct their work (instrumentation and methods), and the resources those scientists need (time, money, and access to research participants). We believe it is essential for a federal education research agency to consistently invest part of its annual appropriations in infrastructure-building programs.

OCR for page 127
Scientific Research in Education Specifically, we believe funding is particularly critical in three areas: the education research community; data development, information sharing, and access; and links with the practice and policy communities. Community of Education Researchers A federal agency must play a role in nurturing the community of education researchers. The greater the field’s capacity to conduct high-quality scientific research in education and to monitor and maintain high scientific standards, the greater is the likelihood the agency will succeed in its mission. Our focus, consistent with the theme of developing a scientific culture in the agency, is on nurturing scientific norms in the field as a whole. Historians tracing the field of education research have noted its failure “…to develop a strong, self-regulating professional community” (Lagemann, 2000, p. ix) over a long period of time. We argue throughout this report that the role of the community of scientists in enforcing scientific principles and engaging in professional, skeptical debate about a reasonably well-defined corpus of scientific work is paramount to the success of the enterprise. The complexity of education, and the attendant scope of the research effort, has to date hindered the ability of groups of scholars to form such a community with common intellectual focus. The organization of programs of research within a federal education research agency (see Design Principle 4 above) would provide a natural springboard for the development of such communities (see http://www.rand.org/multi/achievementforall). The strategic focus of such programs and the standing panels that guide them can provide a common language and set of goals to coalesce groups of peer investigators. In addition, the agency should create incentives for those whom it funds to publish their research and syntheses in peer-reviewed journals.8 Such incentives might include a requirement for progress reports to include evidence of peer-reviewed journal publications, final reports to be in the form of a series of journal articles, and evaluations of new proposals that take into 8   Such journals typically demand, for example, justification of research questions based on balanced, critical reviews of prior research, use of rigorous methods and analyses, and a careful chain of logic in interpretations of findings.

OCR for page 127
Scientific Research in Education consideration the publication record of the principal investigator and other key personnel. A federal education research agency cannot develop and maintain these communities alone. It can leverage its investment in human resources through partnerships with other federal agencies, scholarly professional associations, colleges and universities (especially schools of education), journal publishers, and others. These partnerships could lay the foundation for broad-based efforts aimed at various parts of the system that interact with the education research profession. For example, partnerships with journal publishers and professional associations could lead to the development and monitoring of standards for journal publications and professional meetings. Collaborations with professional associations might feature training and fellowship programs for young scholars (e.g., the Statistics Institute at the annual meeting of the American Educational Research Association [AERA], funded jointly by OERI and the National Center for Education Statistics [NCES], or the AERA Research Grants Program funded by NSF, OERI, NCES, and AERA [Shavelson, 1991] to support dissertation and field-initiated research studies and to place research fellows at NSF and NCES). The agency could also forge links with schools of education, schools of arts and sciences, and other university departments to develop strategies for training and supporting future scientists in education research. The training of education researchers is a long-term undertaking. As we discuss in Chapter 1, current scholarship in education is generated by investigators trained in schools of education as well as in, for example, psychology, history, economics, sociology, mathematics, biology, and public policy departments. In schools of education, students often pursue nonresearch-oriented goals (e.g., school administration) and may therefore reach the graduate level without any research training. In a related vein, publication standards and peer review also vary considerably in education journals. These complex structural issues will require careful study and innovative approaches to address them effectively. Data Development, Sharing, and Access The advancement of scientific knowledge is facilitated when investigators work with the same set of variables and theoretical constructs. Ideally,

OCR for page 127
Scientific Research in Education the field uses a common set of constructs across research studies to enable replication in different contexts and to better understand the extent to which findings from one study can be extended in other situations. This common core would facilitate understanding of how variables and relationships between variables change over time; if the construct changes, there is no basis for comparison from one time to another. In education, this base has been difficult to establish. As we argue in Chapter 4, there is little consensus about the goals of education, which has presented the community with the challenge of making sense of findings from multiple studies on similar topics but based on different measures. Weak theoretical understanding (see Chapter 5) is another reason why such constructs have not yet been fully developed in education. A federal education research agency is a logical central place to develop and maintain databases that house these common variables. With the emergence of new technologies for data collection, management, and analysis, such an agency, perhaps in collaboration with an education statistics agency (like the current NCES) and as theory is strengthened, could develop the capacity to maintain data systems on major issues that provide rich information about educational achievement, processes, costs, institutions, policies, and services on an ongoing basis. The system could draw on the extensive resources already available through NCES, the Organisation for Economic Co-operation and Development (OECD), and NSF and develop a system based on a common conceptual frame that links these data in a coherent way.9 For similar reasons, the agency should encourage and facilitate data sharing among its grantees while ensuring privacy and other ethical standards are met. A key role for a federal education research agency in developing the data infrastructure for scientific education research is by facilitating access to research participants (e.g., students, teachers, administrators, policy makers) and sites (e.g., classrooms, schools, state legislatures). This access is essential to the viability of education research and its potential as a tool for improv- 9   The committee is aware that similar efforts in the past have failed. The development of a conceptual framework will be difficult and contentious, but the lack of coherence across existing indicator systems is a serious problem in education research and should continue to be pursued as a long-term goal.

OCR for page 127
Scientific Research in Education ing education, but researchers have difficulty gaining access to these sources of data for at least two reasons. First, educational practitioners (especially teachers) typically do not see education research as useful to their day-to-day work (Weiss, 1995). This indifference often means that school officials are unwilling to commit the resources (which is usually a substantial amount of time) required to engage in research efforts. A second reason arises out of federal rules and regulations regarding research ethics. Data access for education research involves legitimate concerns about protecting research participants—particularly young students—from inappropriate actions in the name of research. Protections for human research participants, including participants in education research, have been in effect in the United States since 1974 (now codified in Title 45 Part 46 of the U.S. Code of Federal Regulations). The primary protective mechanism outlined in these federal regulations are institutional review boards (IRBs), oversight groups that review all federally funded research involving human participants to ensure their ethical treatment. It is important to recognize that education research, including evaluation studies, rarely presents any true risk to the participant so long as care is taken to protect identities and that researchers understand and are responsive to the needs of individual participants. Explicit exemptions outlined in the U.S. code (see Box 6-1) make this clear. Tom Puglisi, the former Director of Human Subject Protections in the federal Office for Human Research Protections, summed up the intent of current law most succinctly by stating that “much social and behavioral research is exempt from the regulations governing research” (Puglisi, 2001, p. 34). In addition to this core statute, there are at least two other laws (the Family Education Rights and Privacy Act and the Protection of Pupil Rights Amendment) and U.S. Department of Education policy (developed by the agency’s Family Policy Compliance Office) that govern access to education data. In combination, these rules have been variously interpreted and implemented, often creating confusion and erecting unnecessary barriers to conducting scientific research that typically poses “minimal risk” to students. To add to this already maze-like array of statutes, regulations, and policies, a recently passed amendment (Parental Freedom of Information Amendment) to the U.S. House of Representatives version of the pending “No Child Left Behind” legislation, would compound the situa-

OCR for page 127
Scientific Research in Education tion further. Ironically, this amendment would undermine the $30 million evaluation program proposed in the same bill. Research ethics is a complex area that the committee did not have the time nor the expertise to consider fully.10 The committee believes that the basic principles that underlie these regulations and govern the ethical conduct of research involving human participants must be upheld; however, we do see bureaucratic problems and inconsistencies in the way these principles have been implemented. A federal education research agency will need to address these issues as a vital part of its investment in building infrastructure. If ethical access to data on students cannot be achieved, scientific progress will be seriously hindered. We suggest that the agency, in collaboration with other federal agencies conducting science-based education research and other interested groups (e.g., social science research associations, research ethicists) invest some of its resources to work toward the dual goals of scientific access to data and protection of individuals. Without ethical access to research participants and sites, the mission of the agency cannot be met. Links to Practice and Policy Communities We argue above that the practice and policy communities must be engaged in the work of the agency to develop its research agenda. We also call for regular syntheses of research findings to inform practitioners and policy makers about the cumulative knowledge that scientific education research has generated. Here, we suggest a third connection to practice and policy communities, based on the premise that field-based education research that adheres to scientific principles (see Chapter 3) and attends to the features of education (see Chapter 4) will be significantly strengthened by an infrastructure that bridges the gap between researchers and practitioners. We wish to be clear that we are not calling for the agency to develop a dissemination network to “translate” research into practice. The transla- 10   The National Research Council’s Panel on Institutional Review Boards, Surveys, and Social Science Research is reviewing current and proposed methods of human subjects’ protection in social science data collection. It is focusing on the structure, function, and performance of the institutional review board system.

OCR for page 127
Scientific Research in Education BOX 6-1 Exemptions from U.S. Regulations Governing Research Ethics Requirements Research activities in which the only involvement of human subjects will be in one or more of the following categories are exempt from this policy: Research conducted in established or commonly accepted education settings, involving normal education practices, such as: (i) research on regular and special education instructional strategies, or (ii) research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods. Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior, unless: (i) information obtained is recorded in such a manner that human subjects can be identified, directly or through identifiers linked to the subjects; and (ii) any disclosure of the human subjects’ responses outside the research could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects’ financial standing, employability, or reputation. Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures, or observation of public behavior that is not tion of research findings into practice is not a straightforward affair, and indeed, many have rejected this common metaphor outright (see e.g., Willinsky, 2001). The effect of social science on practice is typically indirect, affecting change incrementally through “knowledge creep” (Weiss, 1980, 1991a, 1999). The scholarly literature on research utilization also suggests that local application of knowledge is a long-term process that involves changes in practitioners’ beliefs, as well as in their procedural skill for implementing the knowledge (Weiss, 1991b, 1999). And how to spark

OCR for page 127
Scientific Research in Education exempt under paragraph (b)(2) of this section, if: (i) the human subjects are elected or appointed public officials or candidates for public office; or (ii) federal statute(s) require(s) without exception that the confidentiality of the personally identifiable information will be maintained throughout the research and thereafter. Research, involving the collection or study of existing data, documents, records, pathological specimens, or diagnostic specimens, if these sources are publicly available or if the information is recorded by the investigator in such a manner that subjects cannot be identified, directly or through identifiers linked to the subjects. Research and demonstration projects which are conducted by or subject to the approval of department or agency heads, and which are designed to study, evaluate, or otherwise examine: (i) public benefit or service programs; (ii) procedures for obtaining benefits or services under those programs; (iii) possible changes in or alternatives to those programs or procedures; or (iv) possible changes in methods or levels of payment for benefits or services under those programs. SOURCE: Code of Federal Regulations. Title 45-Public Welfare, Part 46-Protection of Human Subjects, pp. 107-108. Washington, DC: U.S. Government Printing Office. large-scale change in the U.S. education system—research-based or otherwise—is not well understood (Elmore, 1996). Two recent reports have drawn on these and related literatures to suggest fundamentally new ways of organizing the education research enterprise. The first, Improving Student Learning: A Strategic Plan for Education Research and Its Utilization (National Research Council, 1999d), makes the case that education research would have a stronger impact on practice if it were supported by an infrastructure that promoted ongoing collaborations

OCR for page 127
Scientific Research in Education among researchers, practitioners, and policy makers. A second phase of this Strategic Education Research Partnership is currently focused on how to take this idea and build a place—and the enabling strategies, incentives, and infrastructure—to allow these partnerships to flourish. The second, a report of the National Academy of Education (1999), made a similar argument that the prevailing model of research implementation—moving from basic research to development to large-scale implementation of programs—is based on simplistic assumptions about the nature of education and education research. The report concluded that a more productive perspective would view research production and research understanding as part of the same process, also suggesting the need for better partnerships between researchers and educators. Both reports, therefore, simultaneously urge the supply of, and the demand for, education research. Although the critical issue of research utilization is beyond the scope of the committee’s charge (although we do believe that more research on the topic is very much needed), we focus here on the benefits to scientific inquiry that these collaborative models envision. We suggest that a federal education research agency invest in an infrastructure that builds connections between researchers and practitioners because we see the potential to enhance the research itself. Sustained collaborations between researchers and practitioners could strengthen field-based scientific education research by incrementally infusing a deeper knowledge of the complexities of educational practice into theory building, empirical testing, and methods development in a number of ways. First, situating the research in the messiness of day-to-day educational environments would enable closer attention to context, which we argue is essential to recognize and treat in scientific research. This infrastructure would also establish mutual trust and working relationships that could offer long-term, facilitated access to research participants and sites, and so protect against research being abandoned (as we describe in Chapter 4) when the dynamic conditions surrounding education inevitably shift (e.g., changes in school leadership). Furthermore, strategically and appropriately engaging the knowledge of practitioners’ craft throughout the research process can provide relevant insights that otherwise might be missed. There are a few examples of such models in practice (e.g., Consortium on Chicago School Reform, http://www.consortium-chicago.org), but this

OCR for page 127
Scientific Research in Education kind of infrastructure building is fundamentally new. We suggest that an agency support such partnerships carefully and incrementally. There are not only significant structural and cultural barriers to forging these partnerships, but there is also the potential for them to be unproductive. The nature of their work requires practitioners to be driven by immediate crises of the day. These needs could skew the research to be too short-term and tactical in nature to contribute substantially to science-based knowledge. Similarly, there may also be tradeoffs between traditional views of scientific quality and the utility of the work for practice (National Research Council, 2001d). Thus, we urge that the development of these collaborations should include explicit plans for studying their effectiveness and improving them over time. CONCLUSION We believe that clear and consistent focus on translating these design principles into action will promote a strong scientific culture within an agency and strengthen the federal role in education research. For those who know the history of NIE or OERI, many of the principles will strike a familiar chord. For those who don’t, many of them will seem self-evident. However hackneyed or intuitive, we believe they are the crux of the matter. Too often “reform” efforts of the past have focused on changing the existing agency’s organizational structure without adequately grappling with the core issues related to building an infrastructure that supports a scientific community and fosters scientific norms within the agency. Arguably, not since the early days of NIE has the primary agency in the federal government charged with education research had the basic tools to develop a scientific culture and to achieve its mission. Although the details may shift, the principles we propose are intended to stand as guideposts for a federal agency charged with support of scientific education research regardless of the particular situation of the existing federal infrastructure at any given point in time.