PART II
Applying What Is Known: Strategies for Evaluating Teaching Effectiveness



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 69
PART II Applying What Is Known: Strategies for Evaluating Teaching Effectiveness

OCR for page 69
This page in the original is blank.

OCR for page 69
5 Evaluation Methodologies Part I of this report describes recent research on ways to rethink and restructure teaching and learning, coupled with new approaches to evaluation and professional development for faculty. Those findings have the potential to reshape undergraduate education in science, technology, engineering, and mathematics (STEM) for a much larger number of undergraduates. However, developing strategies for implementing and sustaining such changes requires the commitment of all members of a college or university community. In a teaching and learning community, the most effective evaluation is that which encourages and rewards effective teaching practices on the basis of student learning outcomes (Doherty et al., 2002; Shapiro and Levine, 1999). Assessment of student learning at its best enables students to identify their own strengths and weaknesses and to determine the kinds of information they need to correct their learning deficiencies and misconceptions. When such evaluation is properly employed, students learn that they can engage in self-assessment and continuous improvement of performance throughout their lives. Accordingly, this chapter offers practical guidance to postsecondary faculty and administrators on ways to institute a system of both evaluation and professional development that can contribute to significant gains in teaching effectiveness for faculty who teach undergraduates. The chapter describes how input from students (undergraduates and graduate teaching assistants), colleagues, and faculty self-evaluation can be used for evaluating individual instructors. It also describes the advantages and disadvantages of these various approaches. As stated in Chapter 1, ongoing formative assessment of student learn-

OCR for page 69
ing can have powerful benefits both in improving learning and in helping faculty improve their teaching on the basis of the feedback they receive from a variety of sources. The information gathered during such assessments also can serve as a basis for more formal, summative evaluations that have an impact on important personnel decisions. The technique of outcomes assessment as a means of measuring student learning and the use of that information to improve teaching are considered first. Additional strategies and methods for formative evaluation follow. The chapter concludes with a series of suggestions for improving summative evaluation of faculty. The committee emphasizes that the approaches described in this chapter are but a sampling of the techniques that appear in the research literature on improving the evaluation of teaching and student learning. They are Assessment Is More Than Grades To many, the word “assessment” simply means the process by which we assign students grades. Assessment is much more than this, however. Assessment is a mechanism for providing instructors with data for improving their teaching methods and for guiding and motivating students to be actively involved in their own learning. As such, assessment provides important feedback to both instructors and students. Assessment Is Feedback for Both Instructors and Students Assessment gives us essential information about what our students are learning and about the extent to which we are meeting our teaching goals. But the true power of assessment comes in also using it to give feedback to our students. Improving the quality of learning in our courses involves not just determining to what extent students have mastered course content at the end of the course; improving the quality of learning also involves determining to what extent students are mastering content throughout the course. SOURCE: Excerpted from National Institute for Science Education (2001b).

OCR for page 69
included here on the basis of the committee’s analysis of the research literature and the expertise of individual committee members, and with the expectation that each institution will adapt or modify these approaches according to its individual needs. IMPROVING TEACHING BY EXAMINING STUDENT LEARNING: OUTCOME ASSESSMENT One approach to improving student learning is outcome assessment—the process of providing credible evidence that an instructor’s objectives have been obtained. Outcome assessment enables faculty to determine what students know and can do as a result of instruction in a course module, an entire course, or a sequence of courses. This information can be used to indicate to students how successfully they have mastered the course content they are expected to assimilate. It can also be used to provide faculty and academic departments with guidance for improving instruction, course content, and curricular structure. Moreover, faculty and institutions can use secondary analysis of individual outcome assessments to demonstrate to prospective students, parents, college administrators, employers, accreditation bodies, and legislators that a program of study produces competent graduates (Banta, 2000). Outcome Assessment Activities Faculty members, both individually and as colleagues examining their department’s education programs, have found the following activities helpful when undertaking outcome assessment: Developing expected student learning outcomes for an individual course of study, including laboratory skills. Determining the point in a student’s education (e.g., courses, laboratories, and internships) at which he/she should develop the specified knowledge and skills. Incorporating the specified learning outcomes in statements of objectives for the appropriate courses and experiences. Selecting or developing appropriate assessment strategies to test student learning of the specified knowledge and skills. Using the results from assessment to provide formative feedback to individual students and to improve curriculum and instruction. Adjusting expected learning outcomes if appropriate and assessing learning again. Such a process can lead to continual improvement of curriculum and instruction.

OCR for page 69
Faculty in STEM are challenged in their teaching by a set of circumstances that most faculty in other disciplines do not encounter, such as designing laboratory and field components of courses, incorporating modern technology into courses, or supervising students involved with original research (see Chapter 2 for additional detail). However, faculty in these disciplines also have an array of assessment methodologies from which to choose that address particular learning outcomes (e.g., see Doherty et al., 2002). Student responses in each of the following formats can first be studied for the information they provide about individual student learning and performance, and then compared across students and classes for clues about the strengths and weaknesses of curriculum and instruction: Classroom quizzes and exams Projects Poster presentations of library or laboratory research Cooperative experiences Portfolios (collections of work) Standardized tests both within and across disciplines Student journals Questionnaires Interviews Focus groups Scoring of Outcome Assessments: Primary Trait Analysis Increasingly, primary trait analysis (Lloyd-Jones, 1977) is being used as a scoring mechanism in outcome assessment (Walvoord and Anderson, 1998). Primary trait analysis is a technique whereby faculty members consider an assignment or test and decide what traits or characteristics of student performance are most important in the exercise. They then develop a scoring rubric (Freedman, 1994) for these traits and use it to score each student’s performance. For example, Emert and Parish (1996) developed multiple-choice and short-answer tests for undergraduate students enrolled in courses in algebra, discrete mathematics, and statistics. Students were asked to submit supporting work to provide additional insight into their thought processes and the extent to which they had developed an understanding of mathematical concepts. Emert and Parish developed the following scoring rubric to assess performance on each item their students provided: Score Criterion 3 Conceptual understanding apparent; consistent notation, with only an occasional error; logical formulation; complete

OCR for page 69
  or near-complete solution/ response 2   Conceptual understanding only adequate; careless mathematical errors present (for example, algebra, arithmetic); some logical steps lacking; incomplete solution/response 1 Conceptual understanding not adequate; procedural errors; logical or relational steps missing; poor or no response to the question posed 0 Does not attempt problem, or conceptual understanding totally lacking By studying the aggregate scores for each item, Emert and Parish and their colleagues discovered that students missed most items because they lacked the conceptual understanding to address the problem appropriately (as opposed to making careless errors). By inspecting the items missed by large numbers of students, faculty discovered which concepts needed to be addressed through instruction again, perhaps in alternative ways. Understanding such misconceptions by students can provide instructors with valuable insights into how they might adjust their teaching techniques or emphases to address these kinds of problems (see, e.g., National Research Council [NRC], 1997a, 1999b). Benefits of Outcome Assessment It can be difficult and time-consuming for faculty to redesign course objectives to focus on student learning outcomes, to agree with colleagues on comprehensive learning outcomes for the entire curriculum, and to select or develop appropriate assessment tools. It can be equally or more difficult for faculty to adopt a routine of systematically collecting and studying assessment data and then making improvements based on that feedback. However, some examples of positive, multidimensional change have been documented from departments that have taken assessment seriously. These departments update curricula continuously. They develop new courses and phase out others as needs change. And they can document improvement in student learning (Wergin, 1995; Wergin and Swingen, 2000). Other changes that have been prompted by outcome assessment include faculty employing more active learning strategies that enable students to practice the concepts they are learning in class. Alumni and employers are being asked to comment on curriculum and instruction and even to serve as evaluators of teaching and learning. For example, at Virginia Polytechnic Institute and State University, the Department of Civil Engineering created an alumni advisory board and asked its

OCR for page 69
members to debrief a group of juniors and seniors regarding the department’s curriculum. The students discussed such issues as overcrowding due to space limitations. In response, the soil mechanics laboratory was expanded through privately sponsored renovation. In addition, students’ concerns about opportunities to learn to use the latest software led to the development of a new computer laboratory. And a perceived need for improved communication skills encouraged faculty to develop new writing-intensive courses and introduce them into the civil engineering curriculum (Banta et al., 1996). Outcome assessment can be difficult to implement because it requires that faculty reorient their course and curriculum objectives to focus on what students learn rather than what faculty teach. Nonetheless, the committee has concluded that outcome assessment can be an important approach to emphasizing and focusing on what and how students learn. OTHER STRATEGIES AND METHODS FOR FORMATIVE EVALUATION Formative Evaluation by Undergraduate Students Research has shown that the best way to improve teaching is to provide individual faculty members, particularly in their first years of teaching, with ongoing individualized formative feedback from students and colleagues (Brinko, 1993; Cambridge, 1996; Centra, 1993; Hutchings, 1996). Instructors are best served by informal evaluation activities that take place throughout a course, especially when coupled with consultations with learning experts.1 Such informal activities can help instructors identify what is working and what needs to be improved while the course is still in progress. For example, helpful and regular feedback from students allows midcourse corrections in such areas as organization, methods of teaching, and the introduction or modification of activities designed to enhance learning. Many institutions have already recognized the benefits of such midcourse corrections and offer faculty guidance and appropriate forms for conducting various levels of student surveys (see Appendix B). The National Institute for Science Education (NISE) provides a “Student Assessment of Learning Gains” website where faculty can use and modify questionnaires designed to 1   In contrast, Marsh and Roche (1993) report that feedback gathered at the end of a course had significantly greater long-term impact on the improvement of teaching than midcourse evaluations.

OCR for page 69
offer both formative and summative feedback from their students about how various elements of their courses are helping the students learn. This innovative website also allows students to complete the survey form on line and provides instructors with a statistical analysis of the students’ responses.2 The results of studies on formative evaluations of student learning indicate that the techniques described below require modest effort, are easy to carry out, and consume very little class time. In addition, faculty can obtain regular feedback from their students through the use of course listservs, electronic mail, or a website for student feedback connected to a course’s website. Repeated Measurements of Student Learning and Teaching Effectiveness The typical end-of-course student evaluation form is an indirect assessment tool that can help an instructor understand what worked to assist learning in a course and what did not. Instructors may feel that students’ scores on final examinations in their courses provide a valid measure of student learning and that this measure can also be used to assess their effectiveness as a teacher summatively. However, many factors other than the instructor’s teaching competence can affect examination results, including prior knowledge; students’ preconceptions; and their ability, interest, and skills in the subject area (Centra, 1993). Another factor is student effort. Even the most effective teachers can do only so much to motivate students. Although most college teachers try to motivate students to learn, in the end students must take responsibility for their own learning and academic achievement. For the past three years, the Carnegie Foundation for the Advancement of Teaching and Pew Forum on Undergraduate Learning (2002) have published annually the National Survey of Student Engagement: The College Student Report. Each of these reports is compiled from responses to a questionnaire whose respondents consist of thousands of first-year and senior undergraduates at 4-year colleges and universities.3 The students are asked about the extent to which they participate in classroom and campus activities shown by research studies to be important to learning. Questions from the 2   Additional information and links to the survey forms are available at <http://www.wcer.wisc.edu/salgains/instructor/>. 3   The list of institutions that participated in this project is available at <http://www.indiana.edu/~nsse/>.

OCR for page 69
2001 survey instrument are provided in Appendix B.4 This instrument and its parent, the College Student Experiences Questionnaire (Indiana University, 2000), can provide important information about the quality of effort students are committing to their work. If a teaching evaluation form is distributed only at the end of a course, it cannot help the instructor make useful modifications for students who are currently enrolled. A better way to assess student learning and teaching effectiveness is to test students at the beginning and then again at the end of a course and inspect the “gain scores.” An instructor’s willingness and ability to use gain scores to improve a course may be considered favorably during a summative evaluation of teaching. At the same time, gain scores are easily misinterpreted and manipulated and may not be statistically reliable (both pre- and post-tests are characterized by unreliability that is compounded when the two are used together). Therefore, they should not be used exclusively to examine student learning for purposes of summative evaluation. Another indirect measure of student learning that some faculty have found particularly useful is a questionnaire that lists the learning outcomes for a course or series of courses. Students may be asked to indicate how much the course or the entire curriculum increased their knowledge and skills in the specified areas. For maximum usefulness, teachers may want to add their own course-related items to student evaluation forms, as well as encourage written or oral communication from students, including computer-assisted feedback. Evaluations of laboratory, field, and extra clinical or discussion sections require special questions, as do evaluations of student advising (NISE, 2001a). Direct Questioning of Students The easiest way to find out whether students understand what is being said is to ask them directly. But unless instructors have developed sufficient rapport and mutual respect among the students in their class, they should avoid questions or situations that could make it awkward for students to respond (“Who is lost?”) or are so generic as to lead to nonresponses (“Are there any questions?”). Instead, instructors should pose questions that encourage more specific responses, (e.g., “How many of you are understanding what we are talking about?”). Various forms of information technology, such as in-class response keypads, can facilitate asking such questions, allowing students to 4   The survey instruments for both 2000 and 2001 are also available at <http://www.indiana.edu/~nsse/>.

OCR for page 69
answer without fearing that they will be singled out or ridiculed by their peers if they indicate their lack of understanding. Even better, instructors can ask students to paraphrase briefly the key points or essence of a discussion or lecture. At the end of a class session, students can be asked individually or in pairs to write a brief summary of the main ideas presented and submit it to the instructor (anonymously). If this method is used, students should clearly understand that the written summary is not a quiz and will not be graded. Minute Papers and Just-in-Time Teaching At the end of a class, instructors can ask students to write for a minute or two on one of the following kinds of questions: “What is the most significant thing you’ve learned today?” “What points are still not clear?” or “What question is uppermost in your mind at the end of today’s class?” Responses can help instructors evaluate how well students are learning the material. Student responses to the second and third questions also can help instructors select and structure topics for the next class meeting. Large numbers of such short papers can be read quickly, and a review of unclear concepts can take place at the next class meeting (Angelo and Cross, 1993; Schwartz, 1983). A similar approach, developed by the physics education community, is “just-in-time” teaching (Dougherty, 1999). Students are asked to respond to one or two short questions posed by the instructor the day before a subject is to be taught. They submit their responses via e-mail or to a website. These responses give the instructor a good idea of what the students do and do not understand about the concepts to be considered. The instructor can then adjust the amount of time spent on explaining the concepts, working through problems, or providing examples that will help the students learn and understand the concepts. Student Teams Another documented approach involves asking a team of students to work throughout the term on continuous course evaluation (Baugher, 1992; Greene, 2000; Wright et al., 1998). The team members are encouraged to administer questionnaires and interview their peers about how the instructor is or is not promoting learning. For larger classes, a liaison committee of two to four students can be established that meets periodically with the instructor to discuss difficulties or dissatisfactions. Membership on the committee can be rotated from a list of volunteers as long as the entire class knows who the liaisons are at any given

OCR for page 69
Institutions That Practice the Scholarship of Teaching • Encourage student evaluations of teaching. • Support a mentoring program for teachers. • Sponsor seminars, workshops, or conferences on teaching and learning. • Require/ encourage faculty to prepare teaching portfolios or detailed reports on teaching. • Periodically review teaching. • Publish results of learning outcome and teaching environment surveys. • Weigh teaching performance heavily in hiring and promotion decisions. • Encourage a peer review program. • Have active programs or centers to support teaching and learning. • Have training program for teaching assistants. • Provide grants to support research on teaching and learning. • Have a plan for assessing student-learning outcomes. • Survey students and graduates on learning experiences. • Use evidence of student learning in hiring and promotion decisions. • Reward the use and development of effective teaching practices. • Reward teachers/ departments that promote the use of means by which discipline knowledge can be related to students. • Have staff development programs that emphasize diverse teaching practices.   SOURCE: Centra (2001, pp. 8–9).

OCR for page 69
External support obtained for such purposes as improving teaching or purchasing instrumentation for teaching laboratories Videotaping Videotaping is a useful strategy that enables instructors to see what they do well and what needs to be improved. In consultation with an expert from the campus’s teaching and learning center, instructors can determine whether they exhibit such classroom behaviors as dominating a discussion, allowing students enough time to think through questions, or encouraging all students to participate in discussions. Faculty who have been videotaped find the experience extremely helpful, especially if they discuss the analysis with someone having expertise in classroom behavior. Videotaping is best used for formative evaluation. Before-and-After Self-Assessment Faculty members can use before-and-after self-assessment to determine whether course outcomes meet their expectations. Before a course begins, the instructor writes brief comments about the types of students for whom the course is intended. Given that audience, the instructor lists the most important course and learning goals and the teaching strategies she or he will design to achieve them. Once the semester has been completed, the instructor prepares a similar brief description of the types of students who actually enrolled, the instructional methods that were used, and how the students’ achievement of major goals was measured. The evaluation should address (1) goals the instructor believes were met and evidence of student learning and academic achievement, (2) goals that were not realized, (3) the nature of and possible reasons for discrepancies between the instructor’s original intentions and actual outcomes, and (4) how the instructor might modify the course in the future to achieve more of the intended goals. These self-assessments can become part of a teaching portfolio that can later be used for more summative types of evaluation. Another form of before-and-after assessment may help instructors who are interested in examining their teaching behaviors and effectiveness rather than course outcomes. For this technique, instructors use the end-of-course evaluation form, but complete the questionnaire before their course begins (predicting how they think they will do) and again at the end of the semester (how they believe they did). They also may wish to fill out a questionnaire at the end of the term based on what they expect, on average, their students will say about their teaching. In most cases, such self-evaluations are

OCR for page 69
likely to be more positive than student ratings (Centra, 1973; Feldman, 1989). In looking at the results, instructors may wish to focus on any deficiencies noted in the self-evaluation or on discrepancies between their own evaluations and those of their students. SUMMATIVE EVALUATION OF TEACHING Evaluations from Undergraduate Students Questionnaires are most commonly used for summative student evaluations of teaching. The questionnaires can be machine-scored and fall into two categories: those developed locally by campus teaching and learning centers by consulting the literature or adapting forms used elsewhere, and those developed by other institutions or organizations and made available for a fee. Questionnaires vary somewhat in the characteristics of teachers and courses covered, as well as in the quality and usefulness of the scores generated for the instructor. Typically, student evaluation instruments have attempted to identify strengths and weaknesses of instructors in the following areas: organization or planning; teacher-student interactions; clarity and communication skills; workload assigned and perceived difficulty of a course; quality and fairness of grading, assignments, and examinations; students’ ratings of their own learning and progress; and students’ ratings of their level of effort, attendance, and participation in the course, completion of assignments, and motivation. Questionnaires used for student evaluations sometimes address aspects of a faculty member’s teaching style that may or may not contribute to student learning. For example, they may ask whether the faculty member makes eye contact with students during discussions, how many questions the instructor poses during class (as compared with the nature of the questions), or how often students may be assigned to work in groups rather than work alone. Such questions are appropriate only if they are explicitly intended to provide formative feedback for the instructor, but should not be used for summative purposes. Each instructor has a unique personality, persona, and approach to teaching. The primary concern when developing or analyzing questions on student questionnaires for purposes of summative evaluation should be whether the students are actually learning at the desired level and in ways that are consistent with the course goals (Rosenthal, 1976).

OCR for page 69
Global ratings of the course overall or the teacher’s instructional effectiveness also are common to most student questionnaires. For courses in science and engineering, special questions about the efficacy of laboratories, fieldwork, and research experiences also can be included as part of the standardized form or posed in a separate questionnaire. For example, the University of Washington provides separate evaluation forms for laboratories, as well as for clinics and seminars (e.g., University of Washington Education Office of Educational Assessment7). Appendix B provides more specific information about and several examples of student questionnaires for evaluating undergraduate teaching. See also Davis (1988) for compilation of questions that can be used on an end-of-course questionnaire. It is important to note that questionnaires usually do not permit students to assess such characteristics as an instructor’s level of knowledge of subject matter. Students cannot and should not evaluate instructors in this regard. Instead, faculty peers and colleagues should assess these characteristics of an instructor’s teaching. Additional detail about the use of student evaluations for summative purposes is provided in Appendix A. Summative Evaluation by Graduate Teaching Assistants If a department wishes to involve teaching assistants in performing summative evaluations of faculty or improving a department’s educational offerings and approaches to teaching and learning, both the teaching assistants and faculty must feel confident that the procedures for gathering information will preserve the assistants’ anonymity. Teaching assistants need to know before participating how the information will be used and who will see the data. When evaluations from teaching assistants are to be used for personnel decisions, the department might consider asking for written assessments. Alternatively, a system might be established whereby teaching assistants would be interviewed informally by a member of the evaluation committee and their comments recorded and submitted collectively. In either case, teaching assistants should be asked to indicate the basis for their assessment. Such information might include the number of courses they have taught with the instructor, descriptions of their training and supervisory activities, the nature and amount of their contact with undergraduate students, whether they 7   Additional information is available at <http://www.washington.edu/oea/>.

OCR for page 69
were allowed to obtain informal student opinions about the course, and the extent to which they observed each major aspect of the course (e.g., lecture, laboratory). Teacher assistants can be asked for the following kinds of information: An overall judgment of the effectiveness of the faculty member’s teaching. An analysis of the particular strengths and weaknesses of the teaching as reflected in the design, preparation, and conduct of the course. If the department wants specific comments on particular aspects of teaching, the instructions to the teaching assistants should emphasize the need for supporting evidence. The extent to which working with the instructor contributed to the teaching assistant’s own professional development in teaching. The appropriateness of the instructor’s assignments and expectations of the teaching assistants. For each question posed, the teaching assistants should be encouraged to supply specific examples. If their responses are summarized for personnel decisions, the summary must indicate the number of teaching assistants who worked with the faculty member and the number from whom information was obtained. Summative Evaluation by Faculty Colleagues The following approaches might help some institutions obtain more systematic and complete information on teaching performance for purposes of summative evaluation. When these approaches could also be useful for formative evaluation, this is noted. Ad Hoc Committees on Teaching Effectiveness The department might appoint an ad hoc committee on teaching to evaluate each faculty member who is being considered for tenure or promotion. At smaller institutions, where final decisions for promotion and tenure may rest with an institution-wide committee rather than individual departmental committees, a similar panel separate from the committee on tenure and promotion could be established regularly to review the institution’s policies with regard to the process and use of summative evaluations for teaching. The only responsibility of such ad hoc committees would be to evaluate teaching performance. The committee could consist of senior faculty members, one or two junior faculty members, and one or more graduate or senior-level undergraduate students. One or more of these ad hoc committee members should be from outside the candidate’s department.

OCR for page 69
The materials to be considered by the committee could include a variety of teaching-related materials, all of which would be supplied by the candidate: course syllabi and examinations, teaching and learning aids, and evidence of the impact of the candidate’s teaching on students’ learning and intellectual growth. The faculty member also could be asked to submit documentation for the following: currency of course content, participation in the design of courses, contributions to curriculum and instruction, supervision of student research, advising duties, preparation of teaching assistants (if appropriate), and individual and collaborative efforts to improve teaching effectiveness. Candidates should also prepare and submit a self-assessment of their teaching effectiveness. The self-assessment could address questions such as the following: What are the goals of your teaching? Why were these goals selected? How did you know whether students were gaining competence and learning the material? How well did the courses meet your learning goals for your students, and how do you know? What problems, if any, did you encounter in attempting to meet these goals? How did you conduct the course and challenge and engage students? How did your methods take into account the levels and abilities of students? How satisfied were you with the course? What were the strong and weak points of your teaching? What would you change or do differently the next time you teach the course? What did you find most interesting and most frustrating about the course? The candidate’s department chair also could provide the committee with student evaluations from courses taught previously, names and addresses of student advisees, dissertation advisees, enrollees in past and current courses, and the candidate’s cumulative teaching portfolio if one has been prepared. The candidate should see the list of materials submitted to the committee and be given the opportunity to supplement it. Through brief interviews, telephone calls, letters, or brief survey questionnaires issued to the candidate’s current and former students from a variety of courses, the committee could compile a picture of students’ views of the teacher that would supplement the written evaluation reports from past courses. In addition, each committee member could observe and evaluate at least two of the candidate’s classes. Studies of such ad hoc committees revealed that members met several times to discuss their individual findings, used a rating form, and prepared a report, which was then submitted to a departmental tenure and promotion committee (see Centra, 1993, pp. 129– 131 for details). Given the highly

OCR for page 69
positive reliability coefficients reported by Root (1987) for colleague evaluations when the colleagues are properly prepared, one can conclude that the assessments of a faculty member’s teaching effectiveness thus provided are reliable. Colleagues’ Evaluation Questionnaires Several questionnaires have been designed to elicit colleagues’ evaluation of a candidate’s teaching effectiveness for summative evaluation purposes, although they may also be used for formative evaluation. Two forms developed at Syracuse University and the University of Texas at Austin provide scaled-response items and open-ended questions that faculty colleagues and department chairs can use to guide their analysis of a candidate’s chosen instructional materials, as well as teaching behaviors they observe during classroom visits. These forms, printed in their entirety in Appendix C, cover questions grouped under the following five characteristics of good teaching: organization of subject matter and course, effective communication, knowledge of and enthusiasm for subject matter and teaching, fairness in examinations and grading, and flexibility in approaches to teaching. A form designed by French-Lazovik (1981) is also provided in Appendix C. This form offers five broad questions with which faculty peers can evaluate such dimensions as the quality of materials used in teaching. The form also lists which portfolio materials should be reviewed and suggests a focus for colleagues when examining these materials. Other institutions have developed more extensive guides to help candidates prepare for peer evaluation and to assist faculty colleagues in conducting such evaluations effectively (e.g., the University of Texas’s Preparing for Peer Evaluation;8 see also the many resources available through the websites of college and university teaching and learning centers throughout the United States and in other countries).9 While the kinds of forms included in Appendix C have proven helpful to faculty in identifying what materials and characteristics of a candidate’s teaching to assess, the reliability and validity of their evaluations depend on the use of 8   Additional information is available at <http://www.utexas.edu/academic/cte/PeerObserve.html>. 9   A list of websites for teaching and learning centers of colleges and universities in Asia, Australia and New Zealand, Europe, and North America is available at <http://eagle.cc.ukans.edu/~cte/resources/websites.html>.

OCR for page 69
appropriate procedures. For example, as noted above, Root’s (1987) study indicated that a minimum of three departmental colleague evaluators should use the form. They should discuss the evaluation criteria before reviewing materials and making classroom visits. Evaluators also should be provided with examples of evaluations from other candidates, both internal and external, that illustrate high and low ratings. EFFECTIVE IMPLEMENTATION OF EVALUATION METHODOLOGIES Before revising and implementing policies and procedures for evaluating teaching, especially for summative evaluation, stakeholders should proceed in ways that will confer maximum credibility on the results of their efforts. Depending on the institution in question, administrators, the academic senate or committee on tenure and promotion, and faculty must accept that the results of evaluation efforts will be helpful both in personnel decisions and in improving the teaching effectiveness of faculty. Policies and procedures that could assist in the process include the following: Closely involving the institution’s faculty in selecting evaluation methods, drafting the policies and procedures to be implemented at the departmental and institutional levels, and determining the procedures to be used for analyzing and reviewing the results of summative evaluations of teaching. Recognizing and addressing as part of the system of evaluation the full range of teaching styles and activities, both in and out of class. Effective evaluation systems should be able to assess a broad range of teaching styles and approaches. Making evaluation forms and supporting documents freely available to faculty so they understand what information will be considered legitimate and relevant in the evaluation of their teaching performance. Establishing uniform procedures for collecting and using information from students. For example, institution-wide procedures should be defined that protect the anonymity of respondents and ensure that instructors do not see end-of-semester student evaluations until after they have submitted their grade reports. Establishing a uniform and equitable system for the analysis and review of evaluation data, including appropriate response rates for end-of-course student questionnaires. Making clear which letters and surveys will be kept confidential; which can be seen by the faculty under review;

OCR for page 69
and which information, if any, will be shared with students for purposes of selecting future courses. In addition, the following procedures could make any evaluation process more equitable and more easily accepted. Regular Meetings Between New Faculty Members and the Department Chair The department chair should meet with each new faculty member and make clear the department’s and the institution’s general expectations and policies regarding teaching. Norms of grading for assigned courses should be described. The chair also should encourage the new faculty member to consult with other department colleagues who teach the same or related courses to develop policies and procedures for establishing desired learning outcomes, pedagogical approaches, and methods for assessing learning (see, e.g., Annex Box 1-2 in Chapter 1). New faculty members should be encouraged from the beginning of their employment to contribute actively to such discussions. The chair also should encourage and assist new faculty members to work with faculty colleagues both within and outside the department on improving their teaching, and possibly assign a senior mentor to assist them. Formative Discussions Between the Department Chair and Individual Faculty Members Optimally, department chairs should meet at least annually with each member of the department to discuss teaching accomplishments and issues. Such meetings are especially critical for any faculty member whose teaching evaluations are substantially below the department’s expectations or those of other departmental colleagues. These meetings should occur well before summative decisions are to be made so that candidates have ample opportunities to develop a plan for improving their teaching. Additional meetings at regular intervals should be scheduled to assess progress in addressing concerns. Sharing of Faculty-Generated Teaching Portfolios The department’s academic personnel files could include a teaching portfolio for each faculty member. Faculty members could place in the portfolio copies of their course materials (including learning objectives and expected outcomes), syllabi, reading lists, assignments, examinations, and instructional materials. A website also could be established for this purpose. Depending on institutional policy, student evaluation forms or summaries of students’ course evaluations also could be included in the portfolio. It should be

OCR for page 69
assumed that a faculty member would continue to have access to all materials in his or her portfolio, unless letters solicited or submitted in confidence were protected under rules of the university. Feedback from Graduating Seniors and Alumni As part of the department’s regular academic program review, graduating seniors and alumni could be surveyed. Relevant survey information about an individual instructor’s teaching effectiveness would be placed in his or her teaching portfolio. Instructors should be made aware that such information will be included in their portfolios and be allowed to provide written comments or responses, where permissible. Departmental Panel on Teaching Effectiveness and Expectations In addition to an ad hoc department committee to monitor candidates’ progress in teaching, as discussed above, the department as a whole could establish a faculty panel that would summarize the department’s policies and procedures regarding expectations for teaching effectiveness, the methods and criteria used to judge that effectiveness, and the role of evaluation in academic personnel decisions. The panel would remind faculty of the resources available to them through the institution for improving their teaching. Members of such a panel might include a former recipient of the campus teaching award, a respected senior faculty member who teaches introductory and lower division courses, and a newly tenured associate professor. Oversight Committee to Monitor Departmental Curriculum and Instruction The department chair could establish a permanent faculty committee to monitor the quality and effectiveness of instruction by all members of the department. This committee would also oversee all evaluations of curriculum, teaching, and student learning and, where appropriate, nominate faculty for the campus’s or college’s teaching awards. Legal Considerations All stakeholders who are involved with the evaluation of teaching must act in accordance with institutional policies that have been designed to ensure legally equitable and fair treatment of all involved parties. Such policies might require, for example, that: The faculty be involved in the design of an evaluation system, as well as in evaluations of colleagues. The institution complies with all procedures specified in contracts or

OCR for page 69
handbooks, as both are legal documents. The evidence that is used for personnel decisions be job-related and nondiscriminatory. The faculty members be allowed to respond to individual evaluation reports or to clarify information in their dossiers or portfolios. The procedures used in internal review of decisions be clearly elucidated and made available to all faculty.