Click for next page ( 131


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 130
5 Organization and Management of Large-Scale Biomedical Research Projects ' n the fields of biology and biomedical research, formal management of projects and staff traditionally has not been a major topic of con- ~ cern, nor has it been widely studied. Training in management prac- tices has been quite rare for Ph.D. candidates, and direct assessment of an investigator's managerial skills has played little or no role in promotion decisions or in the review of grant applications. The traditional structure of academic research laboratories, consisting of a single, independent principal investigator who oversees a small number of trainees (graduate students and postdoctoral fellows) and technicians, has been thought to present little need for hierarchical or formalized management methods. Furthermore, managerial oversight of investigator-initiated research by funding organizations has been minimal or nonexistent. With the advent of larger-scale projects that involve more scientists and larger budgets, however, effective management, both scientific and administrative, has become more important. This is especially true when multiple principal investigators and multiple institutions join forces to pursue a common mission or goal. In such collaborative efforts, it can be quite challenging to ensure that all the components of a project fit together and work effec- tively toward a collective goal. Project leaders must strive for a common vision and cultural integration among the various participants, who may include scientists and managers from different disciplines or different sectors, such as academia, industry, and government agencies. Unfortunately, there is little information to guide the establishment of good managerial practices in such cases. This is due in part to the variabil- 130

OCR for page 130
ORGANIZATION AND MANAGEMENT 131 ity of research programs and goals, which makes it difficult to set uniform guidelines. In addition, the management of science, even in large-scale projects, has not been widely studied or assessed. Indeed, even in such fields as high-energy physics, in which large-scale, multi-institutional col- laborations have been the norm for decades, the issue of research man- agement has garnered little attention from scholars and remains a con- cern for scientists (National Research Council, 2001b). According to the American Institute of Physics (1992) "without a dedicated effort to under- stand [these complex] collaborations, policy makers and administrators will continue to have only hearsay and their own memories to guide their management. . . " (page 3~. The issue of research management is now coming to the fore because of increased demand by the government to account for the way federal funds are being used (see Chapter 4~. The current Bush Administration is adamant about applying performance standards to assess the manage- ment and productivity of both large and small research endeavors. Ac- cording to John Marburger, Director of the White House Office of Science and Technology Policy, performance measurement is "an inevitable as well as an essential aspect" of the relationship between the government and scientific research. In particular, he notes that "individuals depen- dent on large facilities bear the heavy responsibility of making judicious choices, ensuring prudent management and optimizing the quotient of discovery versus dollars" (Hafner, 2002: page 1~. EXAMPLES OF MANAGEMENT ASSESSMENT FOR LARGE-SCALE PROJECTS Assessment of Federally Funded Laboratories A recent study of federally funded U.S. research and development (R&D) laboratories examined their structure, management, and output through surveys and case studies, and revealed a great variety of organiza- tional designs (Crow and Bozeman, 2001~. Although a large portion of the laboratories examined were not performing biomedical research, there may be some applicable lessons to be learned from the study results. The au- thors of the study found that a proliferation of large research centers after the 1980s had resulted in new institutional and organizational designs. For example, there are now more collaborative research facilities and multi- sector centers, such as university-industry partnerships. In addition, there are more core user facilities and equipment- or process-driven centers. Tech- nology development and technology transfer are also more common. These newer types of research facilities and collaborations required a concomitant change in the approach to management. Before the emer-

OCR for page 130
32 LARGE-SCALE BIOMEDICAL SCIENCE gence of research centers, most federally funded research was undertaken on a smaller scale by individual investigators conducting discipline- oriented projects. As a result, scientists who became center directors often lacked management experience, and some large-scale organizational needs suffered as a result. Furthermore, there appeared to be few mecha- nisms for diffusing managerial knowledge, and almost no incentive to do so. Rather, the authors of the study concluded that successful manage- ment approaches emerged from blind variation and selective retention (Crow and Bozeman, 2001~. The study found that the partition of center management and scientific leadership (analogous to the chief scientist in industry) was quite effective. It also revealed that selecting research direc- tions through a combination of traditional peer review and a nontradi- tional emphasis on building research capacity worked well. With regard to funding, the authors concluded that the stability of funding is often more important than the actual amount of funding, as it provides a core for long-term planning and the development of support systems. Stable funding also facilitates "capacity evaluation" rather than "output evaluation." The distinction is important because government funding managers generally need to think in terms of projects and grants, but research managers often think in terms of resources and work activ- ity. This divergence leads to a conundrum of trying to maintain the effec- tiveness of competitive peer review without stifling the productivity of research centers. Furthermore, determining a program's value within the environment of the Government Performance and Results Act (GPRA) is difficult when there are so many diverse contributions. Evaluation of the National Science Foundation's Science and Technology Centers Program Lessons may also be learned from an evaluation of the National Sci- ence Foundation's (NSF) Science and Technology Centers (STC) program. As described in Chapter 3, the STC program funds large-scale collabora- tive research that often is multidisciplinary and has broad, long-term goals. An extensive evaluation, including an assessment of organization and management, was conducted about 10 years after the program was initiated (National Academy of Public Administration, 1995, ABT Associ- ates, 1996; National Research Council, 1996~. A study panel appointed by the National Academies concluded that the success of the Centers is highly dependent on both their scientific and administrative management.] Be- cause the Centers vary widely in their scope, objectives, research foci, 1 The evaluation was based on site-visit reports, a survey of and interviews with the Center directors, and a previous report from the National Academy of Public Administration.

OCR for page 130
ORGANIZATION AND MANAGEMENT 133 appropriate institutional linkages, and other characteristics, their man- agement and organizational structures are also quite varied. However, effective oversight of the research programs present several common chal- lenges for both Center directors and NSF program managers. The National Academies panel found that a major challenge for the Center directors is to ensure that their Centers embody real collaboration and are not just groups of independent scientists working in a related area. A challenge for program leaders is to maintain focus over time. In a rapidly evolving field, for example, it can be difficult even for a successful Center that is meeting its initial goals to shift its focus in response to the field's natural evolution (e.g., from a basic to a more applied orientation or from one scientific emphasis to another). Moreover, any given Center may not be well constituted to make such large changes and remain suc- cessful. One of the greatest difficulties for NSF managers is ensuring that review and monitoring processes are effective. The panel concluded that site review by committees that include expert peers is very important, particularly in the first few years of a new program. The periodic site- review process was deemed very helpful in several cases when manage- ment problems occurred, as it assisted program leaders in identifying the problems and developing solutions (National Research Council, 1996~. SPECIAL CONSIDERATIONS FOR THE MANAGEMENT OF LARGE-SCALE BIOMEDICAL RESEARCH PROJECTS Large-scale science clearly requires good management schemes and good managers. But what makes for a good manager, and what defines good management? Of course, there is no single response to this question, as the answer will vary depending on the project goal and the methods used to achieve that goal, both of which can be highly diverse. For ex- ample, the managerial needs of a large-scale project designed purely for the purpose of collecting data and creating a database to be used as a research resource may be quite different from those of a large-scale col- laborative project addressing a complex research question. In general, project management entails four basic components: Setting goals and objectives Establishing a timeframe Planning, orchestrating, and coordinating activities to achieve the goals within that timeframe Evaluating progress toward the stated goals However, the size, cost, complexity, and visibility of large-scale proj- ects generate unique or heightened concerns and therefore demand greater stewardship and accountability than are characteristic of tradi-

OCR for page 130
34 LARGE-SCALE BIOMEDICAL SCIENCE tional small-scale projects.2 Potential management problems tend to be proportional to the size of the project, and even apparently minor deci- sions could have major, precedent-setting implications for a large-scale project. As a result, planning and oversight become more labor- and time-intensive for the grant recipient as well as the funding agency, and thus the skills and commitment of both scientific and administrative managers for such projects are critical. For example, the funding agency must develop succinct and unambiguous terms for the award. Other- wise, it could be difficult to suspend or terminate a large-scale project once it has been launched, because of the visibility, politics, and sheer complexity of the undertaking. Agency staff must also define clearly the plans for monitoring and evaluating progress toward short-term mile- stones and long-term objectives, including potential actions to take when adequate progress is not being made, while still allowing enough flex- ibility to adapt to change as the work progresses. Oversight of many of the models described in Chapter 3 involves steering committees and advisory groups that include scientists who are well-respected peers in the field but are not directly involved in the projects. The extra respon- sibilities of the grant applicants include developing detailed, long-range plans to justify the large budget and the commitment of the funding agency. In fact, large-scale projects may require planning beyond a 5- year timeframe. Such long-range planning is extremely difficult in rap- idly changing fields, and such timeframes are essentially unheard of even in the corporate world, where strategic planning is commonly un- dertaken (National Research Council, 1998~. When a large-scale project is carried out at multiple institutions or is funded by multiple sources, the complexities and difficulties associated with planning, coordination, monitoring, and assessment are exacerbated. Federal, industrial, academic, and nonprofit participants may each have their own priorities and ideas for how best to achieve their goals. Each funding source may also have different requirements for oversight or different stipulations for how to handle data release and intellectual prop- erty issues. Even when funding comes from multiple federal agencies, or perhaps even multiple Institutes within NIH, there can be disagreements over the roles and contributions of the various funders. This was certainly the case in the early efforts to launch the Human Genome Project, when the Department of Energy (DOE) and NIH were competing for funds and control of the project (reviewed by Davies, 2001; Cook-Deegan, 1994; Kevles and Hood, 1992~. The leaders of a project must be able to commu- 2 From STEP Administrative Strategies Forum: Big Science: Big Challenges, March 1, 2002, Bethesda, MD, National Institutes of Health.

OCR for page 130
ORGANIZATION AND MANAGEMENT 135 nicate its vision in order to foster teamwork within the project and accep- tance by the field as a whole. Encouraging and maintaining open and effective communication among a project's various team members is also inherently challenging for large-scale, multi-institutional projects, especially when more than one discipline is involved. Participants must be able to speak the same language and need to trust each other enough to discuss ideas and work in progress. Several of the models described in Chapter 3 include regu- lar meetings of the scientists working within collaborative projects, and this approach appears to be quite useful for facilitating good communi- cation.3 Such forums might be useful for collective decision making within large-scale projects as well. Advances in information technology (the Semantic Web, for example) may also facilitate communication within collaborative or multidisciplinary projects (Hendler, 2003~. Because the time commitment for principal investigators leading large- scale initiatives is likely to be much greater than is the case for more tradi- tional projects (Mervis, 2002; Sulston and Ferry, 2002), it is often necessary to hire managers to oversee the day-to-day work of such a project. In academia, there appears to be a preference for managers with strong re- search credentials rather than strong management experience. However, there is no correlation between a person's abilities as a scientist and as a manager. Scientists are rarely trained to be managers, but it can also be argued that traditional business management training programs are not very applicable to the management of science because they are not adaptive enough (Austin, 2002~. Conventional project management methods work best when the chances are good that a project will progress as expected. In contrast, science projects entail discovery and thus are more likely to re- quire cyclical or iterative planning. Effective scientist managers must have both technical and concep- tual knowledge of the science involved in a project, in some cases in multiple disciplines, as well as good people skills, good judgment, and flexibility. The same is true of program managers within funding agen- cies. However, finding qualified individuals to take such positions can be difficult for a variety of reasons. Within academia, credit for a suc- cessful project may be given primarily to the principal investigator, even if the project manager has assumed significant responsibility. Further- more, project managers, both in government agencies and in academia, do not necessarily have a sense of ownership of the data or other prod- ucts of a project. Thus, taking on such a position could be a risky career 3 Carol Dahl, former director of NCI's Unconventional Innovations Program, in a presen- tation to the National Cancer Policy Board, July 16, 2002.

OCR for page 130
136 LARGE-SCALE BIOMEDICAL SCIENCE move. Similarly, project management staff within federal agencies may not be viewed as scientific peers and may not be willing to assume the risks associated with managing a large-scale endeavor. Indeed, people with the right skills and qualifications for management positions are likely to find industry more appealing because of the career structure, incentives, and rewards for successful completion of a projec* (see the section below for more detail). To address these issues, the National Laboratories use an alternative model encompassing dual career lad- ders that recognize and reward the achievements of managers who may not have the scientific credentials of top-tier researchers (Crow and Bozeman, 2001, see previous section). The National Laboratories also have a long history of managing large-scale projects for academic inves- tigators and for rewarding scientists for their participation in team- oriented research. THE INDUSTRY MODEL OF PROlECT MANAGEMENT: COMPARISON WITH ACADEMIA The details of management approaches vary greatly depending on many factors, such as the environment in which the work is being done and the nature of the desired outcome. Historically, the approaches used most commonly in industry and academic settings are quite different. In fact, the quintessential academic research project involves rela- tively little formal management beyond the individuals doing the work. The laboratory head or principal investigator is responsible for obtaining research funding through proposals that outline the objectives, methods, and expected timeframes of the project. He or she then oversees the work of one or a few graduate students, postdoctoral scientists, or technicians who perform the experiments. There is little or no oversight of the project by department or university officials or by officials of the funding agency. The overall work and productivity of the individual principal investiga- tor are reviewed by university or department officials infrequently, such as when decisions regarding tenure or promotion are made. There is great variation across institutions in how the work of graduate students is evaluated; in the case of postdoctoral scientists, a recent survey indicates that most academic institutions do not require written performance evalu- ations or progress reviews (National Resource Council, 2000~. In contrast, most research undertaken in an industry setting involves 4 Carol Dahl, former director of the NCI's Unconventional Innovations Program, in a presentation to the National Cancer Policy Board, July 16, 2002.

OCR for page 130
ORGANIZATION AND MANAGEMENT 137 a team effort in which many investigators share similar levels of responsi- bility for bringing a single project to successful completion. There are many more layers of oversight and supervision, and everyone must work together toward a common goal. Progress is measured against written goals, a practice that promotes good planning and keeps everyone in- formed about what is expected of them. Generally, all members of the team are formally reviewed on an annual basis using a numerical ranking system that determines pay scales and advancement and is designed to elicit improvements from staff.5 Such review may entail traditional top- down assessment of employees by their immediate supervisors. More recently companies have also been using another form of staff review known as "360 review" (see Box 5-1) a method for assessing teamwork in which an employee's performance is evaluated by everyone in the circle that surrounds him or her, including peers, supervisors, and those who work for the employee (Edwards, 1996~. A significant obstacle to undertaking large-scale, collaborative proj- ects within academia may be the inability of the current academic system to assess the work and productivity of individual team members and to reward those who make a significant contribution to a large-scale effort. For such work to be valued and respected, the criteria used for tenure, promotion, and hiring within academia would need to be changed or expanded to include a wider range of scientific achievements. A shift in emphasis away from measuring a scientist's success in obtaining tradi- tional ROl-type grants and toward an evaluation of research output and research capacity in the form of collaboration networks could facilitate such change. Perhaps the greatest obstacle to implementing this concept would be changing the mind-set of the reviewers who make decisions about promotions and tenure. However, there is at least some precedent for this approach in academia in the review of program grants (e.g., POls), where the total research effort is expected to be greater than the sum of its individual components (see Box 4-7~. Nonetheless, industry is still likely to have many more options at its disposal for recognizing and rewarding the work and contributions of team scientists through bonuses, pay in- creases, and opportunities for advancement within the company. Such career issues are discussed in greater detail in the following chapter. 5 In some cases, a minimum or maximum number of employees must be assigned to the highest and lowest rankings to ensure that the ranking system is meaningful. For example, if a scale of 1-5 is used, with 1 being low and 5 being high, a group may be limited to no more than 15 percent of staff ranked as 5s while being expected to have a minimum of 5 percent of staff designated as Is or 2s.

OCR for page 130
138 LARGE-SCALE BIOMEDICAL SCIENCE SUMMARY The capacity of large-scale biomedical research projects to make innovative and novel contributions to the field depends on their organi- zational structure and oversight. Because of their organizational complex- ity, cost, and visibility, large-scale projects have greater needs for man- agement and oversight than is the case for more traditional biomedical research projects. Effective administrative management and scientific leadership are crucial for meeting expected milestones on schedule and within budget; thus the success of a large-scale project is greatly depen- dent upon the skills and knowledge of the scientists and administrators managing the project. Scientific managers must be well versed in the technical and conceptual aspects of the project, which may be multidiscipl- inary, and must also have exceptional organizational and communication skills to facilitate collaboration. However, it may be quite difficult to re- cruit scientists with the needed skill set into managerial positions because of the unusual status of such positions within the scientific career struc- ture, and because scientists rarely undergo formal training in manage- ment. Furthermore, there is little information available on how to struc- ture such management and oversight, and there are few precedents to follow in biomedical research.

OCR for page 130
ORGANIZATION AND MANAGEMENT 139 To pursue large-scale endeavors in biomedical research effectively and efficiently, then, both universities and government agencies will need to develop incentives to encourage qualified scientists to take on the risks and responsibilities of managerial positions. Doing so could entail new approaches for assessing teamwork and management, as well as novel ways of recognizing and rewarding accomplishment in such positions. Both industry and the National Laboratories may serve as instructive models in achieving these goals, as they have a history of rewarding scientists for their participation in team-oriented research. Universities may need to define new faculty and staff categories that are consistent with this type of research, along with appropriate criteria for performance evaluation and promotion. One attempt on the part of NIH to facilitate interdisciplinary team- work is being undertaken by the Bioengineering Consortium (BECON), one of the few organizational units at NIH that crosses all Institutes and Centers. BECON is currently organizing a symposium called "Catalyzing Team Science," aimed at producing a set of guidelines for NIH on how to stimulate, facilitate, and reward collaborative efforts. The workshop will also include a discussion of academic institutions' assessment and reward procedures. It would also be expedient for NIH to formally assess the organization and management of its ongoing large-scale projects, as the National Science Foundation has done in the past. Such an exercise could perhaps lead to the formulation of guidelines for organizing and manag- ing future large-scale projects more effectively or for assessing the man- agement structure of proposed projects.