Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Assessing Evaluation Studies: The Case of Bilingual Education Strategies 1 Bilingual Education in the United States This report can be read on at least two different levels. On a broad level, the report can be viewed as a discussion of statistical methods appropriate for education, particularly bilingual education studies, using two specific studies as case examples. On a more narrow level, the report discusses specific details of two U.S. Department of Education studies about bilingual education. (The report does not explicitly consider organization issues; for these we refer the reader to Atkinson and Jackson (1992)). This chapter provides the context for the studies under review, some initial words about assessment of studies, and an overview of the report. RESEARCH CONTEXT The research reviewed in this report exists in the context of a broad and rich array of research traditions on bilingualism and bilingual education. From an academic perspective, the hybrid fields of psycholinguistics, sociolinguistics, and education have viewed bilingualism both as a topic intrinsically worthy of investigation and as a phenomenon that offers interesting perspectives on, or even critical tests of, theories about linguistic, cognitive, and social processes. These theoretical views of bilingualism—such as the cognitive effects of bilingualism, the social correlates of knowing two or more languages, the nature of the relationship among languages, and the relationship between language and its family, school, community and societal contexts—provide a background for understanding of bilingual education practice and policy; these issues are discussed in Fishman (1977) and Hakuta (1986).
OCR for page 2
Assessing Evaluation Studies: The Case of Bilingual Education Strategies From an applied perspective, bilingual education has generated research under two distinct sociological conditions: one in which the students are native speakers of the predominant language of the society and the other in which they are immigrants or are members of minority-language groups. For example, Canadian bilingual education programs are noted for their success in promoting the learning of French by English speakers through a method called “immersion,” in which English monolingual children begin their schooling experience exclusively in French, with the gradual phasing in of instruction in the native language. A wealth of research shows two key results: these students learn French at levels far beyond what they would have achieved in traditional French language instruction programs, and these students perform in English at a level comparable to their peers who are in monolingual English instruction. This situation has been characterized as “additive bilingualism” (Lambert, 1980), in which the second language is added without detriment to the native language. Useful overviews of the Canadian research are Lambert (1992) and Swain (1992) and the references contained in those articles. In contrast to the Canadian context, bilingual education in the United States exists primarily as a means to aid the transition of immigrant and linguistic minority children into English. Program evaluations seek information on the efficiency by which English is taught, with little or no interest in maintenance of the native language. The situation in which the native language of immigrants and minorities has low social prestige value has been called “subtractive bilingualism.” It typically results in monolingualism in the majority language within two to three generations. The United States has been home to speakers of practically every modern language, yet these linguistic resources have not been captured, and Americans have a well-deserved reputation as incompetent in foreign languages. Linguistic issues in education—both in additive and subtractive bilingual settings—are of increasing importance in this era of global interdependence and cross-national mobility of workers in all industrialized nations. The Canadian research effort in bilingual education in an additive setting has been exemplary for its blend of theory, empirical tests, and sustained effort. The evaluation efforts in the United States—the subject of this report—provide an important barometer not just of the effectiveness of bilingual education, but perhaps more importantly of U.S. research capacity and policy in this area. POLICY CONTEXT President Johnson signed the Bilingual Education act as Title VII of the Elementary and Secondary Education Act (ESEA) on January 2, 1968. The act was reauthorized in 1974, 1978, 1984 and 1988, and it is scheduled for reauthorization in 1992. This section offers a brief sketch of the bilingual education policy context from 1968 to the late 1980s, when the the studies reviewed in this report were completed, with primary focus on the early 1980s, when the requests for proposals for these studies were developed.
OCR for page 3
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Probably the most important issue in understanding the context of bilingual education is the specification of the role of the native languages of students. The original legislation provided for the training of teachers and aides to work with students with limited English skills, as well as for the development of materials and activities to involve parents in the schools. It was limited to students from poor backgrounds and did not prescribe use of the native language or culture in instruction. By 1974 the act was expanded to include students regardless of poverty, and more importantly, it required inclusion of the child's native language and culture “to the extent necessary to allow a child to progress effectively through the educational system” (quoted in Crawford, 1989). Fueling the issue of native language use was the Lau v. Nichols decision by the Supreme Court (414 U.S. 563, 1974) and its interpretation by the Office for Civil Rights (OCR) in the Office of Education. The court decision was based on Title VI of the Civil Rights Act and ruled that students with limited English proficiency, in the absence of treatment, were “effectively foreclosed from any meaningful education.” The court eschewed specific remedies, noting: “Teaching English to the students of Chinese ancestry who do not speak the language is one choice. Giving instructions to this group in Chinese is another. There may be others.” The OCR response, officially titled Task-Force Findings Specifying Remedies Available for Eliminating Past Educational Practices Ruled Unlawful under Lau v. Nichols, was issued on August 11, 1975. The remedies went beyond the Lau decision: they required that bilingual education of some form be provided at the elementary school level in cases where injustice was found; an English as a Second Language (ESL) program was deemed acceptable at the middle school level. Although the remedies did not have the status of federal regulations, they were effectively used as such in disputes with school districts. On the legislative front, questions were raised about the effectiveness of bilingual education programs as early as the late 1970s. One study, conducted by the American Institutes for Research (AIR) under contract from the Office of Planning, Budget and Evaluation (OPBE) (Dannoff, 1978), compared students in 38 ESEA Title VII programs with those students in ESL classes; it failed to find positive impact. However, it did find evidence for students' being kept in bilingual programs even after they had attained proficiency in English and a sizable proportion of program personnel who supported a philosophy of maintenance bilingualism. The AIR study had many shortcomings. However, the panel wishes to emphasize that the failure to find effects in poorly designed evaluative studies should not be taken as evidence that such effects do not exist. The 1978 reauthorization of the Bilingual Education Act required that the programs it supported involve the use of the native language and was quite directive in requesting research on a variety of questions. Section 742 of the act instructed the Assistant Secretary of Education to “coordinate research activities of the National Institute of Education (NIE) with the Office of Bilingual Education (OBE), the National Center for Education Statistics (NCES) and other appropriate agencies in order to develop a national research program for bilingual education.”
OCR for page 4
Assessing Evaluation Studies: The Case of Bilingual Education Strategies As a result, in the spring of 1978, an Education Division Coordinating Committee was established to implement the broad mandate. This committee came to be called the “Part C Committee,” named after the part of the legislation requiring its establishment. The committee was chaired by the Deputy Assistant Secretary for Education (Policy Development) and included representatives from the National Institute of Education, the National Center for Education Statistics, the Office of Bilingual Education (later the Office of Bilingual Education and Minority Language Affairs (OBEMLA)) and the Office of Evaluation and Dissemination (later the Office of Planning, Budget, and Evaluation), and ad hoc representatives from the Office of the Health Education and Welfare Assistant Secretary for Planning and Evaluation. The Office of Education identified three categories in which research might be directed (Education Division, U.S. Department of Health, Education, and Welfare, Proposed Research Plan for Bilingual Education, July, 1979): Investigation of various national needs for bilingual education; Research to improve the effectiveness of services for students; and Research and evaluation to improve the management and operation of the Title VII programs. Under the rubric of efforts to improve services (B), the following research studies were specified: studies to determine and evaluate effective models of bilingual-bicultural programs; studies to determine language acquisition characteristics and the most effective method of teaching English (in a bilingual-bicultural program); a 5-year longitudinal study [on the effectiveness of this title]; studies [on] … methods of [identifying children needing services]; studies [on] … teaching reading to Limited English Proficient (LEP) children and adults; studies of … teaching about culture. As the research agenda was being formulated by the Part C Committee, the proposed federal regulations regarding remedies to Lau violations were finally published in the Federal Register on August 5, 1980, during the final months of the Carter administration. The proposed Lau regulations went even further than the earlier remedies, mandating bilingual education in schools with more than 25 LEP students from the same language group. However, the change in administration brought an immediate change in the government's approach. The new Secretary of Education in the Reagan administration, Terrel Bell, withdrew the proposed regulations on February 2, 1981, calling them “harsh, inflexible, burdensome, unworkable, and incredibly costly” and criticizing federal mandates for native language instruction as “an intrusion on state and local responsibility” (quoted in Crawford, 1989, page 42). The Reagan administration was hostile to bilingual education. President Reagan, for example, early in his first term, declared: “It is absolutely wrong and
OCR for page 5
Assessing Evaluation Studies: The Case of Bilingual Education Strategies against American concept to have a bilingual education program that is now openly, admittedly, dedicated to preserving their native language and never getting them adequate in English so they can go out into the job market” (New York Times, March 3, 1981). This statement by President Reagan suggests that the preservation and use of native language and the neglect of teaching English occur together in bilingual education programs. This is by no means the case. An influential document in the early years of the administration was an internal staff study written at OPBE and later published (Baker and de Kanter, 1983). The study had been undertaken at the request of the Carter administration's White House Regulatory Analysis and Review Group in relation to its proposed regulations. The study, completed in September 1981, reviewed evaluation reports from 39 transitional bilingual education programs. The authors believed these programs offered valid comparisons with alternative programs (ESL and structured immersion), and concluded that there was insufficient evidence to warrant the prescription of bilingual education. The authors called for flexibility and better designed studies (Baker and de Kanter, 1983, page 52): The low quality of the methodology found throughout the literature is a serious problem. Studies have evidenced a lack of random assignment between treatment and control groups, the use of study designs that cannot show a treatment effect in the absence of random assignment, and a failure to apply appropriate statistical tests to demonstrate program effects. Neither the Baker and de Kanter study nor the earlier AIR study were subjected to independent review prior to dissemination, and many serious questions have been raised about them (see, e.g., Willig, 1987). By the time the Bilingual Education Act was up for reauthorization in 1984, the questions raised by the AIR study and the Baker and de Kanter document, in conjunction with a decidedly different political climate in Washington, paved the way toward allowing flexibility in how funds were awarded. Specifically, a compromise was struck in which 4 to 10 percent of the total bilingual education funding could be used towards Special Alternative Instructional Programs (SAIP)—programs that do not use the native language. William Bennett, the new Secretary of Education, focused national attention on bilingual education on September 26, 1985, in a well-publicized address, at which he said (cited in Crawford, 1989, page 71): “After $1.7 billion of Federal funding, we have no evidence that the children whom we sought to help—that the children who deserve our help—have benefited.” He called for an initiative to remove the cap on SAIP and advocated greater flexibility and local control. The proposed changes were published in the November 22, 1985, issue of the Federal Register. The political battle over program flexibility in funding continued into the 1988 reauthorization, in which up to 25 percent of funds for programs were made available for SAIP. Intervening in this conflict was a controversial General Accounting Office report (U.S. General Accounting Office, 1987) requesting expert evaluation of the validity of claims made by the Department of Education regarding the lack
OCR for page 6
Assessing Evaluation Studies: The Case of Bilingual Education Strategies of evidence on the effectiveness of bilingual education. The report concluded that there was research evidence to support bilingual education and questioned the basis of claims made by Secretary Bennett and other department officials. The report served as a counterweight against the strong claims made by the critics of bilingual education. Also relevant in the funding formula debate was the leak of an interim report on the first-year scores from the Immersion Study in June, 1986, in which unadjusted scores showed the English immersion group scoring lower than the other two groups in most of the measures. It was evident that there were high stakes involved in the results from this study. It is critically important for the design and implementation of any intervention study (more generally, any intervention) that the objectives (goals) of the treatments be clearly delineated. At least two possibly conflicting objectives of bilingual education are to (1) minimize the time required to learn English and to (2) maximally facilitate the learning of basic skills, of which English proficiency is only one component. Neither these, nor any other clearly stated objectives, are listed in the legislation for bilingual education, the legal cases associated with it, or in the studies under review. The lack of clearly stated objectives makes any claims of effectiveness, or lack thereof, particularly of teaching approaches, exceedingly difficult to interpret, regardless of the specific data being discussed or the variables used in the claimed assessment of effectiveness. In Appendix A we provide a brief history of bilingual education in the United States. That material is excerpted from the report The Condition of Bilingual Education in the Nation: A Report to Congress and the President (U.S. Department of Education, 1991). We also present a chronology of important events in bilingual education policy. THE PRESENT STUDIES The two studies under review in this report were both initiated within the Department of Education through requests for proposals (RFPs) issued by the Part C Coordinating Committee in 1982. It is notable that this occurred shortly after the collapse of the Lau regulations as proposed by the Carter administration, as well as the circulation of the Baker and de Kanter report, and there was increasing uneasiness about the prescription of the use of the native language. Because it was an interagency policy body, the actions of the Part C Committee might be seen partly as the result of compromises between competing interests. For example, the National Institute of Education (NIE) valued basic research on learning and bilingualism, as well as classroom-based research. Most of the field-initiated research on bilingual education during this period was monitored by this agency: during the 1979–1981 fiscal years, NIE project officers monitored a set of seven studies under the title of “Significant Bilingual Instructional Features” that were funded for a total of over $4 million (National Clearinghouse for Bilingual Education Forum, 1982). The purpose of these studies was to identify instructional
OCR for page 7
Assessing Evaluation Studies: The Case of Bilingual Education Strategies FIGURE 1-1 Part C Funds, 1979–1983 features in classrooms with minority-language students and to understand the linguistic, cognitive, and social processes involved. An analysis of the shift of Part C funds over time is seen in Figure 1-1. Between 1981 and 1982, research priorities shifted from NIE-funded basic research studies to evaluation studies funded OPBE. The two studies under review in this report were funded principally by OPBE, and accounted for a substantial portion of the funds available in 1982 and 1983. The Longitudinal Study It is noteworthy that the RFP for the National Longitudinal Study begins with apparent reference to the AIR and Baker and de Kanter documents: Previous evaluations of the effectiveness of services provided to language minority limited English proficient students have focused narrowly and by design on selected subpopulations of students and selected services provided. One example of such an evaluation was the 2-year longitudinal study of ESEA Title VII-funded Spanish/English bilingual education projects, completed in 1978 and funded by the U.S. Department of Education. More recently, an attempt to synthesize results from diverse research studies demonstrated again the importance of complete documentation of service, school, child, and home characteristics for drawing conclusions about service effectiveness. Although in the past such evaluations may have been appropriate for the Department, this is not the case now. For the Department to formulate federal education policy regarding services provided to language minority limited English proficient students in a time of decreasing federal monies and regulation, a comprehensive information base is required—a base containing information about the broad range of services being provided to such students.
OCR for page 8
Assessing Evaluation Studies: The Case of Bilingual Education Strategies The RFP also makes note of the basic research studies, including the studies funded by NIE on significant instructional features: Collectively, these studies are considered by the Department to constitute part of a comprehensive approach to better understanding problems and potential solutions to providing appropriate educational services to language minority limited English proficient students in the United States. Additionally and importantly, the results of these studies provide a significant part of the foundation which the Department has considered essential before a major evaluation of the effectiveness of a broad range of services provided such students could be successfully undertaken. The Department now believes that the prerequisite foundation is in place and is therefore initiating a national longitudinal evaluation of the effectiveness of services provided language minority limited English proficient students in the United States. In fact the study did not result in an evaluation of a set of distinct program alternatives, about which much was known as a result of basic research. As noted later in this report, the study suffered from a key failing: the programs that it was intended to evaluate did not generally exist in well-defined form, making their evaluation very problematic. The analyses that were eventually conducted were not a definitive evaluation, but, rather, an attempt to find characteristics of programs that showed evidence of being effective. The Immersion Study The RFP for the Immersion Study draws text, at times verbatim, from the Baker and de Kanter report. Notably absent are any references to other research funded by the Part C Committee. The tensions between the different agencies participating in the Part C Committee were known to many researchers in the field who received their funds from it. Because of difficulties encountered by NIE during this period, the competition over priorities for projects increasingly focused on OBEMLA rather than OPBE. As witnessed by the distribution of funds in 1983, OPBE had clearly become dominant in the process. Administrative Arrangements for the Studies Administratively, OPBE started and maintained control of the Immersion Study. For the Longitudinal Study, however, control shifted between OPBE and OBEMLA. The panel reviewed a number of documents and memoranda written by the contractors for the Longitudinal Study that referred to organizational disputes regarding which agency was actually in charge and how this affected the work of the study. OVERVIEW OF THE PANEL'S REPORT The Panel to Review Evaluation Studies of Bilingual Education was convened to report on the two particular studies: The National Longitudinal Study
OCR for page 9
Assessing Evaluation Studies: The Case of Bilingual Education Strategies of the Evaluation of the Effectiveness of Services for Language-Minority Limited-English-Proficient Students (Burkheimer et al., 1989) and The Longitudinal Study of Immersion Strategy, Early-exit and Late-exit Transitional Bilingual Education Programs for Language-Minority Children (Ramirez et al., 1991a, 1991b). A major charge to the panel was to suggest alternative ways of conducting evaluation studies. The issues of appraising the studies at hand and providing advice for future studies are inexorably intertwined. Throughout this report the panel mixes advice and questions about future studies with a critical evaluation of the Longitudinal and Immersion Studies. When there was a choice between specific comments on one of the studies or general comments about evaluation studies, the panel has chosen the latter path, using specific comments as examples. Chapter 2 sets the theoretical stage for a careful statistical discussion of the two studies and introduces definitions and examples of statistical and experimental terminology. The heart of this report consists of Chapters 3 and 4, which present the panel's appraisals of the Longitudinal Study and Immersion Study, respectively. Both of the chapters also present quotations that show the Department of Education's official view of these studies. Much of the substance in these chapters speaks directly to the two studies, but each chapter also offers general comments on conducting evaluation studies. Both chapters offer specific suggestions for future analyses of the reported data. Although the panel has attempted to make these chapters self-contained, there are inevitable references to the final reports of the two studies. Readers who have no knowledge of the appraised reports should still be able to understand the discussion in these chapters. As the two studies were very different in intent and execution, so the two chapters reflect very different statistical and substantive concerns, but the general methodology and statistical concerns in the two chapters are complementary. Chapter 5 continues the discussion of the design of bilingual education studies that begins in Chapter 2 but focuses on the need for underlying theories. The panel believes that explicit theories of education, including explicit objectives, are required to both motivate and structure the sensible statistical design of studies to evaluate alternative forms of education. For example, the primary objectives of bilingual education in the United States remain controversial. It is imperative that there be clear and agreed upon statements of the goals of bilingual (or other) education programs before one attempts to assess the impact of different programs. Once there is an explicit set of goals there also need to be theories that suggest how to achieve those goals. The theories may be based on exploratory studies, examination of existing studies, or other methods. Only after a theory is articulated can one begin to define interventions, measurement tools, and design strategies. Just as important is the concern about how to measure the postulated effects, who the subjects will be, and how the interventions might differentially effect different subject populations. There is a tension between the need for broad studies that are applicable to
OCR for page 10
Assessing Evaluation Studies: The Case of Bilingual Education Strategies a general population and narrowly focused studies that are likely to find specific effects. The panel strongly believes that it is much better to find out what works somewhere than to fail to find what works anywhere. Without proper statistical designs, most intervention studies are likely to fail. A reasonable strategy is to attempt to sequentially accumulate knowledge about programs through focused, designed, and controlled experiments; when possible, it is better to perform controlled experiments than observational studies. The panel's conclusions and recommendations are presented in Chapter 6. This chapter is partitioned into a section of conclusions about each study and a general set of conclusions and recommendations. The panel highlights the major weaknesses in the studies and provide specific recommendations on how to avoid similar weaknesses in future studies. The appendices provide some background material on bilingual education in the United States, a technical summary of a statistical methodology used in the Immersion Study, a list of acronyms used in the report, and biographical sketches of the panel and staff. REFERENCES Atkinson, R., and Jackson, G., eds. (1992) Research and Education Reform: Roles for the Office of Educational Research and Improvement. Committee on the Federal Role in Education Research, Commission on Behavioral and Social Sciences and Education, National Research Council. Washington, D.C.: National Academy Press. Baker, K. A., and de Kanter, A. A., eds. (1983) Bilingual Education: A Reappraisal of Federal Policy. Lexington, Mass.: Lexington Books. Burkheimer, Jr., G. J., Conger, A. J., Dunteman, G. H., Elliott, B. G., and Mowbray, K. A. (1989) Effectiveness of services for language-minority limited-english-proficient students (2 vols). Technical report, Research Triangle Institute, Research Triangle Park, N.C. Crawford, J. (1989) Bilingual Education: History, Politics, Theory, and Practice. Trenton, N.J.: Crane Publishing Co. Dannoff, M. N. (1978) Evaluation of the impact of ESEA Title VII Spanish/English bilingual education programs. Technical report, American Institutes for Research, Washington, D.C. Fishman, J. A. (1977) The social science perspective. In Bilingual Education: Current Perspectives. Vol. 1: Social Science, pp. 1–49. Rosslyn, Va.: Center for Applied Linguistics. Hakuta, K. (1986) Mirror of Language: The Debate on Bilingualism. New York: Basic Books. Lambert, W. E. (1980) Two faces of bilingualism. In Focus, No.3. Rosslyn, Va.: National Clearinghouse for Bilingual Education. Lambert, W. E. (1992) Pros, cons, and limits to quantitative approaches in foreign language research. In B. F. Freed, ed., Foreign Language Acquisition Research and the Classroom, chapter 19, pp. 321–337. Lexington, Mass.: D. C. Heath and Co. National Clearinghouse for Bilingual Education Forum (1982) Update: Part C bilingual education research. National Clearinghouse for Bilingual Education, Rosslyn, Va.
OCR for page 11
Assessing Evaluation Studies: The Case of Bilingual Education Strategies Ramirez, D. J., Yuen, S. D., Ramey, D. R., and Pasta, D. J. (1991a) Final report: Longitudinal study of structured-english immersion strategy, early-exit and late-exit transitional bilingual education programs for language-minority children, Volume I. Technical report, Aquirre International, San Mateo, Calif. Ramirez, D. J., Pasta, D. J., Yuen, S. D., Billings, D. K., and Ramey, D. R. (1991b) Final report: Longitudinal study of structured-english immersion strategy, early-exit and late-exit transitional bilingual education programs for language-minority children, Volume II. Technical report, Aquirre International, San Mateo, Calif. Swain, M. (1992) French immersion and its offshoots: Getting two for one. In B. F. Freed, ed., Foreign Language Acquisition Research and the Classroom, chapter 19, pp. 321–337. Lexington, Mass.: D. C. Heath and Co. U.S. Department of Education (1991) The Condition of Bilingual Education in the Nation: A Report to the Congress and the President. Office of the Secretary. Washington, D.C.: U.S. Department of Education. U.S. General Accounting Office (1987) Bilingual education: A new look at the research evidence. Briefing Report to the Chairman, Committee on Education, Labor, House of Representatives, GAO/PEMD-87-12BR. Willig, A. (1987) Meta-analysis of selected studies in the effectiveness of bilingual education. Review of Educational Research, 57(3), 351–362.
Representative terms from entire chapter: