Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 72
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct 4 Institutional Approaches to Fostering Integrity in Research Research organizations currently rely on a variety of methods for promoting integrity in research. They establish organizational components to comply with regulations imposed by an external environment; they offer educational programs to teach the elements of the responsible conduct of research;1 they implement policies and procedures that delineate the normative practices of responsible research and establish criteria for rewards and recognition; and they develop processes to evaluate and enforce institutional behavior.2 In addition, organizations engage in activities that help establish an internal climate and organizational culture that are either supportive of or ambivalent toward the responsible conduct of research.3 These various approaches are not mutually exclusive, however, nor should they be. A number of programs and activities, integrated across the various levels of an organization, should be in place in order to maximize the impact on the research environment and to support the responsible conduct of research. 1 Chapter 5 provides a detailed discussion of promoting integrity in research through education. 2 Chapters 2 and 3 provide discussions of the types of policies and procedures that should be implemented in a research organization to foster integrity in research. 3 Chapter 3 provides a discussion of the elements of an organization’s structure and processes that have an impact on how members of an organization perceive the ethical climate and culture.
OCR for page 73
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct To establish a basis for organizational learning and continuous quality improvement, organizations should simultaneously implement processes for evaluating their efforts to foster responsible conduct of research. Evaluation can be approached in a variety of ways. One way is to rely on external evaluators to determine compliance with regulatory controls. Another is to rely on a system of performance-based assessments that are initiated and implemented internally. Such assessments can also be used to meet the accountability requirements of outside funding or government sources. In addition, peer reviewers may be used in institutional self-assessment processes; assessments done by peer reviewers may or may not be associated with accreditation by external organizations. This chapter provides a discussion of the strengths and limitations of these various approaches and establishes a rationale for the use of the institutional self-assessment approach to evaluation that is recommended in this report (see Chapter 6). REGULATORY COMPLIANCE The U.S. Congress has consistently affirmed the importance of integrity in research, and the federal government has established a framework for regulating misconduct in science. As described in Chapter 1, regulatory language includes a definition of research misconduct and spells out the requirements for institutional policies and practices to handle reports of misconduct or other types of wrongdoing in the research environment. A regulatory framework requires a rule-making process that may be governed by legislative or administrative actions. The framework may be decentralized (i.e., each institution may develop its own approach), or it may require compliance with a common set of criteria, policies, and practices. The safety standards promulgated by the Occupational Safety and Health Administration (OSHA) are an example of the latter. Regulatory models commonly involve data collection and the generation of reports that document each institution’s compliance with governmental policies. OSHA, for example, very clearly specifies requirements for recording and reporting on-the-job injuries and illnesses, and is authorized to conduct workplace inspections to check reporting and insure compliance. Assessment Strategy Common components of regulatory frameworks include the specification of certain procedures and reporting requirements, the collection of data, and the preparation of reports of compliance practices. The regulatory approach also involves a governmental unit that maintains oversight of the compliance and reporting procedures, investigates complaints
OCR for page 74
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct about rule violations, and offers technical assistance in rule-making and implementation of regulations. In university settings, evaluations for mandatory compliance with government regulatory standards are common. In this framework, full-time government employees frequently (but not always) conduct inspections to evaluate compliance with a well-defined set of standards, followed by some mechanism for feedback (usually an exit interview or a written report). Individual inspectors do not necessarily require professional credentials in the field or activity being evaluated, although many individuals in such positions have such qualifications (i.e., they are not necessarily “peers” of the individuals whose conduct is being evaluated). In some highly regulated research environments, such as those in which new drugs are developed, iterative partnerships can evolve between regulators (e.g., the Food and Drug Administration) and the regulated entities (e.g., pharmaceutical companies). Strengths Several models of regulatory frameworks for research that could be adapted to the oversight of integrity in research already exist. These models include the regulatory frameworks for the oversight of the protection of human research subjects (The Common Rule), the evaluation of misconduct in science, the use of animals in research, and the handling of toxic or radioactive research materials. A regulatory approach to fostering integrity in research is consistent with other governmental efforts to encourage the use of commonly accepted practices and to discourage irresponsible behavior in the research environment. Researchers and institutional officials are familiar with compliance requirements and often participate in the preparation of rulemaking procedures. Individual research centers frequently have some latitude and discretion in adapting government requirements to their own needs, and the centers are responsible for designating specific officials who will ensure that faculty, trainees, and administrative staff understand the importance of the regulations. A regulatory framework fosters a collective and consistent response to social concerns by a broad array of research institutions, and it highlights best practices. The framework also can reduce the opportunity for idiosyncratic and irresponsible policies or practices to be implemented. Limitations A regulatory approach fostering integrity in research also has some limitations (Nature, 2001; Burman, et al., 2001; Sugar, 2002). Such an approach increases the bureaucratization of science and requires documen-
OCR for page 75
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct tation that institutions may find burdensome. A 1999 report commissioned by the National Institutes of Health (NIH) at the request of Congress reviewed the burden of regulations across five key areas, including research integrity. The report identified four problems that were present in all the areas: rigid regulations that dictate process often limit flexibility without enhancing results; rules imposed by multiple agencies have inconsistent requirements; regulation of science by nonscience agencies often leads to additional and nonproductive regulatory burden; and poor communication exists among federal agencies and research institutions (NIH, 1999). The report also notes that regulatory systems tend to impose the same requirements on all research institutions regardless of their characteristics. That is, requirements are aimed at the lowest performer, and hence provide little incentive for superior performance. Regulations often emphasize the areas of common agreement and can reduce important concerns to rules and procedures. It is difficult or impossible for regulations alone to foster an understanding of the critical issues involved, and the required procedures are not always related to the desired outcomes. The adoption of new regulations and the creation of institutional and governmental oversight offices increase the cost of doing science and add to the administrative costs of research centers without necessarily creating a commensurate benefit. In addition, once regulations are adopted, they are difficult to change. Finally, in some instances regulators are forced to focus more on process than on results. That is, the regulators seek to evaluate the extent to which observable processes that are hypothesized to promote some desired outcome are in place; examples of such processes might include protocols to protect human or animal research subjects, measures to ensure laboratory safety, or desirable data-handling practices. Although a focus on process can facilitate intended outcomes, it can also be directed to matters that are relatively unimportant or that are poorly connected with the desired result. A PERFORMANCE-BASED APPROACH A performance-based model for the evaluation of organizational efforts to foster integrity in the research environment offers selected goals and benchmarks4 that can be used as criteria to assess the success of 4 A benchmark is a standard or point of reference used in measuring and/or judging quality or value. Benchmarking is the process of continuously comparing and measuring an organization’s performance, practices, policies, and philosophies against leading, high-performing organizations anywhere in the world to gain information that will help the organization take action to improve its performance (NPR, 1997).
OCR for page 76
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct efforts. These goals and benchmarks are generally linked to rewards, incentives, and, at times, penalties for specific types of behavior. A performance-based approach provides a direct role for research institutions and research team leaders in fostering norms for faculty, research staff, trainees, and students within diverse research settings. The development of a performance-based model would require institutions to formulate a coherent statement of goals that describes the principles of integrity in research that they wish to encourage. The model would also require institutions to implement these goals through a series of actions and assessment strategies. Such actions could include the following: posting the statement (including selected criteria related to personnel actions, such as recruitment offers and hiring and promotion policies and practices) in public places throughout the research institution; creating a bonus plan or award system to reward exceptional behavior; providing mentorship opportunities for senior and junior faculty and investigators that emphasize the importance of learning about the responsible conduct of research; and publicizing and possibly sanctioning actions that are inconsistent with the institution’s research mission. Performance-based systems require that benchmarks be set and that a database be generated in order to measure faculty and trainee compliance with the specified standards to identify areas that need improvement. Such benchmarks can be formal or informal, but they require a broad consensus that what is being measured is important and relevant to integrity in research. Performance benchmarks must be communicated clearly so that all members of a research institution understand what is being measured and why. Performance-based approaches should be implemented incrementally, because data must be collected and analyzed before major institutional changes can be implemented. Such implementation is in keeping with the spirit of continuous quality improvement. Research on organizational behavior found that institutions have ethical climates that differ according to the values, standards, and interests of their members (Victor, 2001). Performance incentives are tools that institutions use to align their members’ behaviors and practices with the values that are expressed as the institutions’ missions and goals. These tools represent one part of a broader set of normative control systems for organizations, and they are shaped not only by aspirational goals but also by societal norms, the form of the organization, the behavior of the leadership, and unique features of research groups.
OCR for page 77
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct Assessment Strategy Assessment strategies may focus on the development of mission statements and benchmark tools alone, or they may include an analysis of the ways that institutional officials use such tools to influence faculty adherence to responsible research practices. Assessment efforts can also be used to review compliance strategies (including the compliance of faculty and research staff), student surveys, and sponsor evaluations, as well as to analyze rewards, incentives, and penalties. In one application of a performance-based assessment, for example, Gilmer (1995) has the students in her research ethics course at Florida State University develop individual portfolios. The students must define their goals for learning and then present several pieces of their work that demonstrate that such learning has been accomplished. Portfolio assessment has been used primarily in educational settings to document the progress and achievements of individual students or teachers, but it has the potential to be a valuable tool for program assessment as well. Another example is the Quality Improvement Program instituted by the Office of Human Research Protections at the Department of Health and Human Services (DHHS) (OHRP, 2002). This voluntary program is intended to help institutions prepare for and successfully achieve accreditation of their human research protection programs by private accrediting agencies. One feature of this program is a Quality Assurance Self-Assessment Tool, which helps the institution determine its level of compliance with federal regulations. In a related effort, the Health Improvement Institute (a private foundation), under a contract with DHHS, has established a national Award for Excellence in Human Research Protection, and the institute currently is developing performance-based evaluation criteria for the award (HII, 2002). On a more general basis, a classic example of performance-based assessment is the Government Performance and Results Act of 1993 (Pub. L. No: 103–62), which requires federal agencies to develop strategic plans for delivering high-quality services or products to the public. Each agency has to (1) establish top-level agency goals and objectives, as well as annual program goals; (2) define how it intends to achieve those goals; and (3) demonstrate how it will measure agency and program performance in achieving those goals (NPR, 1997). As part of this initiative, the National Performance Review (NPR) was created to sponsor and organize benchmarking studies aimed at making government work better and cost less. Performance-based assessment and benchmarking also are the basis for the Malcolm Baldrige Quality Award. Established by the Malcolm Baldrige National Quality Improvement Act of 1987 (Pub. L. No. 100– 107), the award is administered by the National Institute of Standards and
OCR for page 78
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct Technology (NIST) and recognizes U.S. organizations for their achievements in quality and performance. The awards are given to businesses (manufacturing, service, small business) and to education and health care organizations (Seymour, 1995). Judging is based on seven criteria: leadership, strategic planning, customer and market focus, information and analysis, human resource focus, process management, and business results (NIST, 2001). The Baldrige criteria are used by thousands of diverse organizations for self-assessment and training and as a means to develop performance and business processes. Several Institute of Medicine (IOM) reports describe performance-based approaches to improving community health (IOM, 1996a, b, 1997). Strengths Performance-based systems are increasingly common in diverse institutional settings, including health care (with the new emphasis on quality), the transportation sector, and various sectors of the manufacturing and service industries. Conceptual frameworks, measurement tools, and institutional case studies exist that can provide the foundation for the development of such a system in the area of integrity in research (see Appendix B). The setting of benchmarks represents a visible and tangible public commitment to integrity in research, and at the same time recognizes the need for oversight and assessment of questionable conduct in the research environment. Such goals can be flexible and consistent with the diverse institutional cultures of different research centers. Rewards, incentives, and penalties offer valuable educational resources in that they can be used to demonstrate the types of conduct that research institutions do and do not encourage. Limitations Performance-based systems also have limitations. They require a considerable amount of institutional commitment and involvement. Institutional officers need to exercise leadership and authority in the development of a mission statement and performance goals, as well as in the selection of benchmarks that will be used to guide and evaluate behavior. The adoption of performance-based goals can be divisive and controversial if faculty do not share common norms and aspirations, or if such goals lead to restrictions on the types of research that can be conducted. Institution-directed performance goals may run counter to the autonomy and innovative spirit of the research environment in most academic centers. Current limitations, which can be addressed through further research, include the need to identify the benchmarks and criteria most
OCR for page 79
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct relevant to integrity in research, and the need to develop and validate the instruments and measures for assessment. AN INSTITUTIONAL SELF-ASSESSMENT AND PEER REVIEW APPROACH The complexities of balancing formal and informal approaches to fostering integrity in research have led to efforts to have research institutions assess their own performances, including the performances of their managements, faculty and research staff, in terms of complying with stated standards, goals, and practices. Such self-assessments may include evaluations of aspects of certification or institutional assurance of compliance with professional standards within a broader organizational context; this practice is frequently used in the accreditation of professional schools and departments, as well as of educational institutions. This approach is described in detail in Chapter 6. It is presented here only briefly to provide a basis for comparison with the other approaches discussed above. Assessment Strategy The strategy used in the self-assessment and assurance framework has multiple distinct features. The self-assessment process may be voluntary or mandatory. The process of institutional assurance of compliance with professional standards may be linked to an informal peer-review process or may be part of a formal accreditation procedure. Different institutions will draw on one or more of these features, according to their own goals and resources. One such approach is presented for purposes of illustration. The Middle States Commission on Higher Education (MSCHE, 2002) has recently developed proposals for new accreditation processes for its member institutions. The proposals include several distinct features of the internal assessment and accreditation process, including the following: institutional self-study; a team visit; types of accreditation actions; periodic review reports; institutional profile (annual) reports; candidacy and initial accreditation procedures; public information; use of technology (e.g., electronic submission of report materials); and training of evaluators and the institutions’ departmental chairs.
OCR for page 80
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct The accreditation proposals are intended to address the commission’s desire to balance the need to be careful and thorough with competing concerns about the slow and time-consuming nature of self-assessment and accreditation procedures (personal communication, J. Morse, Middle State Commission on Higher Education, [January 4, 2002]; see also www.msache.org/chx02.pdf). Additional examples of assessment strategies are described in Appendix B. Strengths The mission statements of accreditation bodies frequently assert that their role is to promote academic quality through formal recognition of the importance of compliance with professional standards and to advance the process of self-regulation (see, for example, the mission statement of the Council for Higher Education Accreditation [CHEA, 2001]). Institutional self-assessment and accreditation procedures can be powerful influences in the shaping of professional behavior within a self-regulatory system. In the mid-1990s, for example, the National Council of Examinations for Engineers and Surveyors introduced engineering ethics questions in its examination used for granting professional engineering licenses. Every state and U.S. territory now uses these examinations. The inclusion of ethics questions in the engineering examination has been a practical force for introducing these issues into the classroom as well as into the review sessions that are popular with applicants for professional engineering licenses (Rabins, 1998). If self-assessment and assurance requirements are implemented through standard practices used by faculty, then they can become important components in the overall climates of research organizations. In this way, they will have the ability to change institutional cultures over time. The process of self-regulation has the advantage of focusing on those areas that the members of the profession believe are essential to quality and integrity. The introduction of elements of integrity in research and the responsible conduct of research into ongoing processes of institutional self-assessment and assurance helps accustom researchers to addressing these issues as part of the normal accreditation process. It thus diminishes the growth of disparate units within research organizations and strengthens the alignment of concerns about integrity in research with research practices. Limitations As noted above in the example of MSCHE practices, self-assessment and accreditation practices are frequently slow and time-consuming. Re-
OCR for page 81
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct sources for the training of faculty evaluators and the operation of review committees are required at multiple levels within research institutions, including individual departments, research centers, and general administrations. Turnovers in personnel, a lack of familiarity with review and assessment procedures, and changes in the standards that are used as the basis for self-assessment and evaluation can further complicate the cumbersome nature of the assessment process. At this time, an obvious limitation to the self-assessment approach as a means to foster integrity in research is the need to develop and validate evaluation instruments and measures. Further research on methods and measure and elements of the research environment (as described in Chapter 7) should eliminate this limitation. Examples of relevant assessment tools that might be adapted to the research environment are described in Appendix B. SUMMARY Evaluations of activities within research institutions occur in diverse forms and are influenced by different approaches that may consist of voluntary or mandatory elements and that may rely upon professional or volunteer reviewers. The committee has not found research evidence that suggests that any particular approach produces significant differences in measurable outcomes. Each approach has certain strengths as well as limitations. The regulatory compliance approach may be more desirable when professional standards are clear and when measurable indicators can be developed to assess levels of compliance with selected processes and rules in different research settings. However, this approach often fosters attention to minimal standards (the lowest-common-denominator approach) rather than encouraging institutions to determine what is right for their situation and to invest in the efforts necessary to foster more desirable outcomes. Although benchmarks are not yet available to support performance monitoring in the area of research integrity, they could be developed through educational programs and consensus building efforts. The adoption of a performance-based approach enables institutions to move beyond procedural compliance mechanisms as a self-regulatory device (e.g., certifying that they have adopted certain policies, procedures, and educational training efforts) in favor of a framework that fosters greater individuality but still adheres to certain performance standards that reflect basic peer and community values. The institutional self-assessment approach is frequently overlooked in policy debates about research oversight. This approach is commonly associated with highly specialized efforts, such as the accreditation of
OCR for page 82
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct academic institutions or procedures for the handling of laboratory animals. Recent concerns about the treatment of human subjects in research studies have brought new attention to the strategies involved in institutional self-assessment and assurance procedures (IOM, 2001; NBAC, 2001). The committee believes that much of this analysis is relevant to discussions of integrity in research, and therefore the committee suggests that more work is needed to determine how self-assessment coupled with peer review, particularly in the context of institutional accreditation, can be adapted to efforts to foster integrity in research. Chapter 6 provides a fuller discussion of this method of assessment. REFERENCES Burman WJ, Reves RR, Cohn DL, Schooley RT. 2001. Breaking the camel’s back: Multicenter clinical trials and local institutional review boards. Annals of Internal Medicine 134:152– 157. CHEA (Council for Higher Education Accreditation). 2001. Directory of CHEA Participating and Recognized Organizations, 2000–2001. [Online]. Available: http://www.chea.org/Directories/index.cfm [Accessed February 25, 2002]. Gilmer PJ. 1995. Teaching science at the university level: What about the ethics? Science and Engineering Ethics 1:173–180. HII (Health Improvement Institute). 2002. Award for Excellence in Human Research Protections. [Online]. Available: http://www.hii.org/ [Accessed May 14, 2002]. IOM (Institute of Medicine). 1996a. Using Performance Monitoring to Improve Community Health: Conceptual Framework and Community Experience. Washington, DC: National Academy Press. IOM. 1996b. Using Performance Monitoring to Improve Community Health: Exploring the Issues. Washington, DC: National Academy Press. IOM. 1997. Improving Health in the Community: A Role for Performance Monitoring. Washington, DC: National Academy Press. IOM. 2001. Preserving Public Trust. Washington, DC: National Academy Press. MSCHE (Middle States Commission on Higher Education). 2002. Characteristics of Excellence in Education: Eligibility Requirements and Standards for Accreditations. [Online]. Available www.msache.org/chx02.pdf [Accessed March 18, 2002]. NBAC (National Bioethics Advisory Commission). 2001. Ethical and Policy Issues in Research Involving Human Participants. Rockville, MD: U.S. Government Printing Office. NIH (National Institutes of Health). 1999. NIH Initiative to Reduce Regulatory Burden. [Online]. Available: http://grants1.nih.gov/grants/policy/regulatoryburden/ [Accessed May 6, 2002]. NIST (National Institute of Standards and Technology). 2001. Getting Started with the Baldrige National Quality Program: Criteria for Performance Excellence. [Online]. Available: http://www.quality.nist.gov/ [Accessed May 15, 2002]. NPR (National Performance Review). 1997. Serving the American Public: Best Practices in Performance Management. [Online]. Available: http://govinfo.library.unt.edu/npr/library/papers/benchmrk/nprbook.html [Accessed May 15, 2002]. Nature. 2001. Time to cut regulations that protect only regulators. Nature 414:379.
OCR for page 83
Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct OHRP (Office of Human Research Protections). 2002. Quality Improvement Program. [Online]. Available: http://ohrp.osophs.dhhs.gov/humansubjects/qip/qip.htm [Accessed May 14, 2002]. Rabins MJ. 1998. Teaching engineering ethics to undergraduates: Why? what? how? Science and Engineering Ethics 4:291–302 Seymour D. 1995. The AQC Baldrige Report: Lessons Learned by Nine Colleges and Universities Undertaking Self-Study with the Malcolm Baldrige National Quality Award Criteria. Washington, DC: American Association for Higher Education. Sugar AM. 2002. Letters: The crisis in local institutional review boards. Annals of Internal Medicine 136:410–411. Victor, B. 2001. Integrity in the business environment. Presentation at the June 28, 2001, meeting of the Institute of Medicine Committee on Assessing Integrity in Research Environments, Washington, DC.
Representative terms from entire chapter: