National Academies Press: OpenBook

Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories (2013)

Chapter: E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering

« Previous: D--The Structure of the Management Organizations that Govern the NNSA National Security Laboratories
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

E

What the Laboratories’ Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering

For each laboratory management corporation, the contract is the primary document governing its relationship with the government. Each contract includes a statement of work and an annual Performance Evaluation Plan (PEP), which is included in the contract as a contract modification. Other modifications are also included as agreed between the government and the contractor. Other important documents that affect the conduct of science and engineering are the Contract Management Plan, the Parent Organization Oversight Plan, and the annual Performance Evaluation Review (PER), which is based on the PEP.

Although they have important specific differences, the contracts governing SNL, LANL, and LLNL are generally similar in form and content. In general, each specifies: (1) what the laboratory is and does; (2) what the management corporation is committed to do; (3) what work is to be done at the laboratory (the “statement of work,” or SOW); and (4) how that work is to be reviewed, evaluated, and rewarded. The contracts also specify certain rights and responsibilities of specific government offices to task and/or oversee the work. The SOWs, while lengthy, are basically general. The contracts do not assign specific program tasks. The following—from the LLNL contract—is representative:1

Inasmuch as the assigned missions of the Lawrence Livermore National Laboratory (Laboratory) are dynamic, this Statement of Work (SOW) is not intended to be all-inclusive or restrictive, but is intended to provide a broad framework and general scope of the work to be performed at the Laboratory. This SOW does not represent a commitment to, or imply funding for, specific projects or programs. The National Nuclear Security Administration (NNSA) and Department of Energy (DOE) work requirements are developed through strategic planning and program plans.

While this quoted paragraph refers primarily to programmatic work, the contracts make clear and specific that the laboratory management is committed to other duties, including operational and administrative tasks. Indeed, these management tasks constitute a large part of the work against which the management corporations are evaluated.

The performance evaluation process determines how successful the management has been for the period being evaluated (usually a year), and the contract specifies how success is rewarded. The rewards consist of an award fee (incentive fee) and an award term (i.e., an extension of the temporal term of the contract without need for competition). Independent of the review, the management corporation also receives an annual fixed fee (specified in the contract) and payment for allowable expenses associated with running the management corporation (much of which is associated with paying officials of the various boards and advisory groups).

The incentive fees and fixed fees vary across the laboratories and by year. Table E.1 displays approximate values for comparison.

_______________________________

1 See Modification No. M003 Supplemental Agreement to Contract No. DE-AC52-07NA27344, titled “Part III, Section J, Appendix B-Statement of Work” from the LLNL contract for the Statement of Work in full, available at http://www.llnsllc.com/contract/docs/AppendB_mod53_012209.pdf.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

TABLE E.1 Approximate Annual Fee Structures (in millions of dollars)

Laboratory Fixed Fee Maximum Incentive Fee
SNL 16 9-10
LANL 22 52
LLNL 12.5 29.5

Most (≥90 percent) of the incentive fee is awarded when a laboratory successfully executes tasks specified in its annual PEP so that the amount of incentive fee that is really “in play” in any evaluation process is in reality a few million dollars (out of a total annual laboratory budget in excess of $1 billion).

All three contracts state that science and engineering are major parts of what the laboratories do. The LLNL contract is perhaps most explicit in this regard, the SNL contract least so. The following excerpt from the LLNL contract also appears verbatim in the LANL contract (but with a different identifying number):

H-36 INTELLECTUAL AND SCIENTIFIC FREEDOM2

(a) The Parties recognize the importance of fostering an atmosphere at the Laboratory conducive to scientific inquiry and the development of new knowledge and creative and innovative ideas related to important national interests.

(b) The Parties further recognize that the free exchange of ideas among scientists and engineers at the Laboratory and colleagues at universities, colleges, and other laboratories or scientific facilities is vital to the success of the scientific, engineering, and technical work performed by Laboratory personnel.

(c) In order to further the goals of the Laboratory and the national interest, it is agreed by the Parties that the scientific and engineering personnel at the Laboratory shall be accorded the rights of publication or other dissemination of research, and participation in open debate and in scientific, educational, or professional meetings or conferences, subject to the limitations included in technology transfer agreements and such other limitations as may be required by the terms of this Contract. Nothing in this clause is intended to alter the obligations of the Parties to protect classified or unclassified controlled nuclear information as provided by law.

(d) Nothing in the Section I clause entitled “DEAR 952.204-75 Public Affairs” is intended to limit the rights of the Contractor or its employees to publicize and to accurately state the results of its scientific research.

The LLNL SOW states (emphasis added):

2.0 Laboratory Mission and Scope of Work.

The Contractor shall manage, operate, protect, sustain and enhance the Laboratory’s ability to function as a NNSA Multi-Program Laboratory, while assuring accomplishment of the Laboratory’s primary mission - strengthening the United States’ security through development and application of world-class science and technology to enhance the nation’s defense and to reduce the global threat from terrorism and weapons of mass destruction.

To some degree, these, and similar statements are “boilerplate” in that they have no specific obligations attached to them. However, they are indicators of what the orientation of the laboratory is expected to be. Indeed, Section H-13 of the LLNL contract says more pragmatically:

_______________________________

2 See Contract No. DE-AC52-07NA27344 Modification No. 241 for Sections B-H of the LLNL contract, available at http://www.llnsllc.com/contract/docs/Part_I_Section_B-H-LLNL_992011mod241.pdf.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

This Contract is a management and operating contract, which holds the Contractor accountable for performance. This Contract uses clearly defined standards of performance consisting of performance objectives and performance incentives

Identical language is found in H-12 of the LANL contract and H-10 of the SNL contract.

Section 2.0 of the LLNL SOW lists 17 major points that the scope of work is to include. Five of these 17 deal explicitly with science and technology, as indicated by bold underline below:

•   Conducting major NNSA research and development programs including using an earnedvalue management system;

•   Fostering an environment of scientific skepticism and peer review of research programs;

•   Assuring the safety, security, reliability, and performance of the national nuclear weapons stockpile pursuant to national security policy and presidential and congressional directives;

•   Demonstrating design and development capabilities to support a Reliable Replacement Warhead strategy, and stockpile and complex transformation;

•   Providing scientific, engineering, and computational capabilities that support assessment, dismantlement, manufacturing, and refurbishment of the enduring stockpile at a number of sites;

•   Operating major facilities including the National Ignition Facility and the Device Assembly Facility that support broad national interests and users.

•   Ensuring the secure handling and safe disposition of plutonium, highly enriched uranium, and tritium;

•   Helping to deter, detect, and respond to the proliferation of weapons of mass destruction;

•   Conducting fundamental science research, nuclear energy development, and nuclear waste management technology in support of other DOE programs;

•   Contributing to civilian and industrial needs and non-NNSA defense activities through a work for others program by using the scientific and technical expertise that derives from carrying out the Laboratory mission;

•   Providing access to the capabilities of the laboratory to further Department of Homeland Security mission objectives;

•   Advancing of science, mathematics, and engineering education;

•   Advancing science through technological innovation, public and private sector collaboration, and technology transfer to enhance U.S. economic competitiveness and national security;

•   Managing and operating the Laboratory facilities and infrastructure in an efficient, cost effective, and innovative manner;

•   Remediating and restoring the Lawrence Livermore National Laboratory sites;

•   Managing waste minimization, treatment, storage, and disposal of all wastes; and

•   Assisting the nuclear weapons complex in waste stabilization, storage and disposition technologies.

The lists for the other two laboratories are similar, but with different specific elements. These lists appear to be something more than general principles and something less than specific contractual obligations.

Regarding expectations of performance, all three contracts say in Section H-2:

H-2 PERFORMANCE DIRECTION

(a) The Contractor is responsible for the management and operation of the site in accordance with the Terms and Conditions of the Contract, duly issued Work Authorizations (WAs), and written direction and guidance provided by the Contracting Officer and the Contracting Officer’s Representative (COR). NNSA is responsible for establishing the work to be accomplished, the applicable requirements to be met, and overseeing the performance of work of the Contractor. The Contractor will use its expertise and ingenuity in Contract performance and in making choices among acceptable alternatives to most effectively, efficiently and safely accomplish the work called for by this Contract

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

NNSA is responsible for telling the contractor what to do, and the contractor is responsible for figuring out how to do it.

EVALUATION OF THE QUALITY OF SCIENCE AND ENGINEERING

The contracts make clear that the primary responsibility for monitoring contract performance rests with the site offices: Los Alamos Site Office (LASO); Livermore Site Office (LSO); and Sandia Site Office (SSO). From Section G of each of the contracts:

The NNSA Manager, Livermore Site Office (LSO), is the Contractor’s primary point of contact for all technical and administrative matters, except as identified in (b) below, regarding this Contract. The LSO Administrative Contracting Officers are the Contractor’s primary point of contact for all contractual matters for this Contract.

The NNSA Manager, Los Alamos Site Office is the Contractor’s primary point of contact for all technical and administrative matters, except as identified in (b) below, regarding performance of this contact. The LASO Administrative Contracting Officer is the Contractor’s primary point of contact for all contractual matters.

The NNSA Manager, Sandia Site Office (SSO), is the Contracting Officer responsible for this Contract. The SSO is the Contractor’s focal point of contact for all matters, except as identified in (b) below, regarding this Contract.

Clause (b) cited in each of these states that “The Patent Counsel, Office of Chief Counsel, NNSA Service Center, is the Contractor’s focal point for items concerning patent, intellectual property, technology transfer, copyright, open source, licenses and technical data issues.”

However, all three contracts make clear that there are major authorities that are reserved to the contracting officer, and not the site manager:

(b) Clarifying the Contract Relationship
NNSA will establish the work to be accomplished by the Contractor, set applicable requirements to be met by the Contractor and provide performance direction to the Contractor regarding what NNSA wants in each of its programs. NNSA will issue performance direction to the Contractor only through a warranted Contracting Officer or a designated Contracting Officer’s Representative. All other Federal staff and oversight components are therefore precluded from tasking contractor personnel.

H-2 PERFORMANCE DIRECTION

(a) The Contractor is responsible for the management and operation of the site in accordance with the Terms and Conditions of the Contract, duly issued Work Authorizations (WAs), and written direction and guidance provided by the Contracting Officer and the Contracting Officer’s Representative (COR). NNSA is responsible for establishing the work to be accomplished, the applicable requirements to be met, and overseeing the performance of work of the Contractor. The Contractor will use its expertise and ingenuity in Contract performance and in making choices among acceptable alternatives to most effectively, efficiently and safely accomplish the work called for by this Contract

(b) Only the Contracting Officer may issue, modify, and priority rank WAs.

(c) (1) The Contracting Officer and the NNSA Administrator will appoint, in writing, specific NNSA employees as CORs with the authority to issue Performance Direction to the Contractor. CORs are authorized to act within the limits of their delegation letter … COR functions include technical monitoring, inspection, and other functions of a technical nature not involving a change in the scope, cost, or terms and conditions of the Contract. The COR is authorized to review and approve technical reports, drawings, specifications, and technical information delivered by the Contractor.

(2) The Contractor must comply with written Performance Directions that are signed by the COR

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

The following from the LLNL contract is representative (emphasis added):

(b) Performance Appraisal Process.

(1) Performance Evaluation Plan.

(i)A Performance Evaluation Plan shall be developed and finalized by the Contracting Officer , with Contractor input … The NNSA Livermore Site Office Manager reserves the unilateral right to make the final decision on all performance objectives and performance incentives (including the associated measures and targets) used to evaluate Contractor performance. The NNSA Administrator reserves the unilateral right to make the final decision on all award term incentives (including the associated measures and targets) used to evaluate Contractor performance.

(ii)Only the Contracting Officer may revise the Performance Evaluation Plan, consistent with the Contract’s Statement of Work, during the appraisal period of performance …

(2) Contractor Self-Assessment. The Contractor shall prepare an annual self-assessment of its performance against each of the performance objectives and incentives contained in the Performance Evaluation Plan … The Contracting Officer will identify the structure and medium to be used by the Contractor in delivering its annual self-assessment.

Matters are complicated by the fact that the three site offices are organized differently.

Recent Contract Management Plans (CMP) for the three laboratories3,4,5 make it clear that in the case of each location, the contract manager has most of the responsibility. The CMPs for LANL and LLNL are very similar. Note, however, that in section 5.3 (which describes the Site Manager), there is a significant difference.

The LANL contract states: “The Site Office Manager is a senior NNSA manager that provides an on-site, day-to-day presence at the laboratory. The LASO Site Manager is responsible for effective contract administration at LANL to ensure the successful implementation of NNSA programs. The Site Manager relies on the Site CO … to administer contracts based upon demonstrated individual qualifications and Site Office needs, as well as to handle most day to-day administrative contract duties. The LASO manager currently does not possess CO authority. As such, he must rely on his COs and jointly issue directionto LASO which is or may be considered outside the current scope of the Prime Contract.”

The LLNL contract states: “The Site Office Manager is a senior NNSA manager that provides an on-site, day-to-day presence at the laboratory. The LSO Site Manager is responsible for effective contract administration at LLNL to ensure the successful implementation of NNSA programs. The Site Manager is also an Administrative Contracting Officer with authority to administer contracts based upon demonstrated individual qualifications and Site Office needs.

Although the LSO Site Manager is a warranted CO, she relies largely on the Site CO … to handle most day-to-day administrative contract duties.

The SNL contract states: (section 5.1.1)

The SSO Site Manager is appointed as a Contracting Officer. . .. The SSO Manager is a senior NNSA manager that provides an on-site, day-to-day presence at SNL. The SSO Manager is responsible for effective contract management and oversight at SNL to ensure the successful implementation of NNSA programs. The SSO Manager is also a CO with authority to administer the Sandia Contract based upon demonstrated individual qualifications and SSO needs. Although the SSO Manager is a warranted CO, she relies largely on the SSO CO to handle most day-to-day

_______________________________

3 Contract Management Plan for Los Alamos National Laboratory, Contract No. DE-AC52-06NA25396, Los Alamos CMP-9-3-08, 2008.

4 Livermore Site Office, Contract Management Plan for Lawrence Livermore National Laboratory, Contract No. DE-AC52-07NA27344, LSO_CMP_6-10-088, 2008.

5 Contract Management Plan for Sandia Corporation, 2007.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

administrative contract duties. “5.5 Contracting Officer (CO): The CO has sole authority to enter into, administer, or terminate Federal contracts. The CO, through properly written modifications to the contract, is the only person authorized to make changes to cost, scope, and schedule. The CO must ensure that all requirements of law, executive orders, regulations, and all other applicable procedures, including clearances and approvals, have been met. The CO is also responsible for ensuring performance of all necessary actions for effective contracting, ensuring compliance with the terms of the contract, and safeguarding the interests of the United States in its contractual relationships. The FAR allows the CO wide latitude to exercise business judgment. This duty includes the balanced objective of safeguarding the interests of the United States in its contractual relationships and ensuring that contractors receive impartial, fair, and equitable treatment.”

In a formal sense, the site manager is responsible for the evaluation process, but most of the authority rests with the CO. The evaluation begins with the preparation of the PEP, which is agreed (and signed) among the laboratory director, the site manager, and the CO. The laboratory then conducts a self evaluation, which the CO (with the help of others at the site office) uses in the preparation of a PER (performance evaluation report). Final sign-off on the results of the evaluation rests with the Principal Deputy (NNSA) Administrator. Typically, none of these individuals is a scientist or engineer.

These three contract management plans say little or nothing about scientific quality, and nothing about the process for evaluating scientific quality. On the other hand, there is a lot of emphasis on deliverables, both substantive (i.e., related to the research and program work) and procedural/managerial/legal.

PERFORMANCE EVALUATION PLAN

NRC staff reviewed PEPs for LLNL6 and for LANL.7 For each laboratory, the annual PEP is the official basis for the evaluation of performance under the contract, which in turn is the basis for the award (incentive) fee and the award term (i.e., award of continuation of the contract). The PEP is a modification to the contract; it becomes Part III, Section J, Appendix F [of the M&O contract]. Each PEP is the product of the specific site office, and they are significantly different in form.

LLNL PEP

This plan lists eleven Strategic Performance Objectives:

1. Complete essential activities for core weapons program requirements.

2. Strengthen the foundation of deterrence through stockpile science, technology, and engineering.

3. Propose and implement strategies for sustaining a strong deterrent at low numbers compatible with START, NPR and CTBT goals.

4. Execute Inertial Confinement Fusion Ignition and High Yield Campaign in support of stockpile stewardship.

5. Support nonproliferation and threat reduction.

6. Provide science, technology, and engineering excellence.

7. Support current and evolving mission performance by providing effective and efficient facilities and infrastructure.

8. Maintain safe and environmentally sound operations in an efficient and effective manner in support of mission objectives.

_______________________________

6 LANL Performance Evaluation Plan for FY2011.

7 LLNL Performance Evaluation Plan, dated April 28, 2010, is for the evaluation of performance for FY2010 (beginning October 1, 2009).

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

9. Maintain secure operations in an efficient and effective manner in support of mission objectives.

10. Manage business operations in an effective and efficient manner while safeguarding public assets and supporting mission objectives.

11. Governance assures performance and creates long-term sustainable value for the institution.

The PEP organizes the Strategic Performance Objectives as six program objectives (1-6), three operations objectives (7-9), and two institutional management objectives (10 and 11). Objective 6 deals explicitly with excellence in science and engineering and is the only objective to do so.

Attachment 1 breaks the 11 objectives into more detail. The part of that table dealing with Objective 6 is shown in Table E.2.

In calculating the performance objective award fee, all six of the program objectives are considered together. Roughly 35 percent of this part of the fee depends on the six program objectives; 45 percent depend on the three operations objectives, and 21 percent depend on the two institutional management objectives. Although it is not explicitly stated, it appears that Objective 6 (excellence in science and engineering) accounts for perhaps 6 percent of the incentive fee.

The award term is tied to five objectives, none of which is science and technology per se. These are: stockpile stewardship mission; site transformation activities; sustainable management; safety management system; and contractor assurance system. The first of these five contains explicit sciencebased milestones and objectives, but these objectives are not “excellence in science.” These five “ATI”s (award term incentives) are generally combinations of near-term (i.e., current year) and near-to-midterm (i.e., 1-3) objectives.

TABLE E.2 LLNL PEP Strategic Performance Objective 6

Performance Category Number Performance Objective/Measure/Target Type DOE Strategic Plan Reference NNSA Program Office Crosswalk
Programs 6 Provide science, technology, and engineering excellence Objective 3. Scientific Discovery & Innovation NA-121.4
Programs 6.1 Assure the quality of the core science, technology and engineering (ST&E) competencies Measure 3.2 Foundations of Science NA-121.4
Programs 6.1.1 Demonstrate R&D excellence in program execution through external peer reviews, evidence of awards and recognition, and benchmarked S&T metrics. Target(s) - Essential only 3.2 Foundations of Science NA-121.4
Programs 6.1.2: Recruit and retain a workforce, including students and postdocs, that provides the technical engineering, and scientific staff for mission-relevant national needs. Target(s) - Essential only 3.2 Foundations of Science NA-121.4
Programs 6.2 Develop, implement, and periodically update an integrated and balanced ST&E strategy. Measure 3.2 Foundations of Science NA-121.4
Programs 6.2.1 As a measure of successful implementation of the FY 2009 5-year ST&E Roadmap, demonstrate that, internal investments (e.g., LDRD) are aligned with thrust areas identified in the strategic plan. Target(s) - Essential only 3.2 Foundations of Science NA-121.4
Programs 6.2.2 Monitor the long-term impact that LDRD investments have made on the scientific, programmatic, and intellectual property position of LLNL for 2011. Target(s) - Stretch only 3.2 Foundations of Science NA-121.4
Programs 6.3 Develop Strategic Collaborations and an Open Campus Measure 3.2 Foundations of Science NA-121.4
Programs 6.3.1 Implement a strategy for seeking new collaborative research with commercial, governmental, and academic entities. Execute design and construction activities that deliver a facility with open access with beneficial occupancy in FY11. Target(s) - Essential only 3.2 Foundations of Science NA-121.4

SOURCE: LANL Performance Evaluation Plan, dated April 28, 2010, for the evaluation of performance for FY2010 (beginning October 1, 2009), Attachment 1.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

LANL PEP

The plan lists and explains 19 performance based incentives (PBIs). PBI 18 is concerned with the award term incentives while the others are related to award (incentive) fee. PBI 12 is focused on science and engineering quality:

PBI No. 12 Objective: Excellence in Science, Technology, and Engineering Objective Statement: Science, technology, and engineering underpin and enable Los Alamos to provide knowledge and technologies to execute its national security missions. State of the art equipment and facilities enable science to push the frontiers of knowledge; however capabilities rely on the appropriate mix of people and resources. Leadership, competence and insight drive science and position the institution for success.

This is a termed a “stretch” PBI with a maximum value of $5,500,000 (or 10.7 percent of the maximum available). It is evaluated subjectively according to the guidance shown in Table E.3.

PBI 18 lists five performance measures for the award term. One of these five, Measure 18.3 Demonstrate Leadership in Pu Science, is explicitly related to science. The detailed description of measure 18.3 is shown below in Figure B.

TABLE E.3 Performance Measures for LANL PBI No. 12

SECTION 4
PERFORMANCE MEASURES

List associated performance measures, expectations, targets, and fee schedules for FY 2010.

Measure 12 Excellence in Science, Technology, and Engineering (Subjective/Stretch)

Expectation Statement:

The NNSA will subjectively evaluate the contractor’s performance in

Completion Target:

Criteria Target

12.1   Leadership in Scientific, Technology and Engineering Challenges of National Importance

The list below represents subject areas to be considered for scoring Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion.

•   Pursue national security missions In the national interest that are mutually beneficial to NNSA and other customers.

•   Science, technology and engineering strategic plans.

•   Willingness to pursue novel approaches and/or demonstration of innovative solutions to problems that are important to the Nation and Laboratory’s customers.

•   Effectiveness in driving the direction and setting the priorities of the community in a research field.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

12.2   Quality of Science, Technology & Engineering

The list below represents subject areas to be considered for scoring. Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion

•   Impact and contribution to the scientific community as measured by:

•   Peer reviewed publications generated.

•   Significant awards (R&D 100, FIX, society fellows, etc.).

•   Voice of the customer obtained through the “Modified Office of Science” approach.

•   Assessments by 2010 LANL capability reviews.

•   Invited talks, citations, making high-quality data available to the scientific community.

•   Development of tools and techniques that become standards or widely-used in the scientific community.

•   Staff members visible in leadership positions in the scientific community.

12.3   Capability Based Science Program

The list below represents subject areas to be considered for scoring. Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion.

•   Alignment of skills and capabilities with NNSA mission using workforce capabilities tool.

•   Institutional investment in science infrastructure.

•   Effectiveness in leveraging (synergy with) other areas of research.

•   Provide planning and acquire facilities and infrastructure required to support the continuation of the Laboratory’s mission and programs.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

12.4   Stewardship of LANL Scientific Resources

The list below represents subject areas to be considered for scoring. Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion.

•   Increase the effectiveness of the entire scientific enterprise.

•   Scientific user facility utilization.

•   Balance with Laboratory missions by maintaining STE capabilities and facilities in a mutually beneficial manner for the multiple customers.

•   Accountability to institution and institutional practices.

•   Provide efficient and effective communications and responsiveness to customer needs by responding to customer requests with accurate and timely information.

12.5   Energy Frontiers & Challenges

The list below represents subject areas to be considered for scoring Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion.

•   Address recommendations from 2009 review of Laboratory’s Energy Security Science Strategy.

•   Expand the Energy Security Science Strategy to include components for staffing and infrastructure resources. Plan elements for staffing and infrastructure will be delivered to LASO by September 1,2010.

•   Implement the Laboratory’s Energy Security Science Strategy.

•   Effectiveness of joint planning (e.g., workshops) with outside community.

•   Development of capabilities, ideas for new facilities and research programs.

•   Diversify funding base to maintain core science, technology and engineering base.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

12.6   Work for Others Management and Technology Transfer Programs

The list below represents subject areas to be considered for scoring Less than expected performance in a given area does not necessarily equate to a failed score Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion.

•   Effectiveness and efficiency of the management of work for others (WFO) and technology transfer and in the benefits to the Laboratory’s mission performance.

•   Leveraging CRADAs and WFO to maintain critical NNSA capabilities

•   Implementation of corrective actions from FY 2009 management assessments, audits, and capability reviews.

•   Leverage intellectual property for cooperative research projects.

12.7   Science ARRA Management Stimulus

The list below represents subject areas to be considered for scoring Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion

•   Implementation of stimulus capability applicability process for all stimulus competitive grants that are issued.

•   Share lessons learned on stimulus capability applicability process with applicable S&T organizations.

•   Demonstration of effective communication and teaming with outside entities associated with stimulus-related work (State of New Mexico, other laboratones, municipalities, etc.)

•   Oversight of execution of stimulus-funded S&T work to ensure baselines (scope, cost, and schedule) are achievable

•   Oversight of execution of stimulus-funded S&T work to ensure appropriate reporting is submitted to sponsors.

•   Other related items, concerns, and missions.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

12.8   Science, Technology, and Lngineenng Leadership and Management

The list below represents subject areas to be considered for scoring. Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion

•   Integration, alignment, and balancing of institutional STE resources to deliver on mission commitments.

•   Integration in and proactive resolution of performance and management concerns.

•   Integration and synergy across the institution.

•   Management of institutional resources to position Laboratory for FY 2011 and beyond

•   Risk to DOE/NNSA and Laboratory continuity by LANS action or inaction

•   LANS management and mitigation of site risks

•   Consideration of any increases or decreases in site risks

•   Effective leveraging of WFO to offset and/or leverage landlord costs.

12.9   Management of Emergent Science, Technology, and Engineering Issues

The list below represents subject areas to be considered for scoring Less than expected performance in a given area does not necessarily equate to a failed score. Similarly, this list is not considered all inclusive and other topics, issues, or area of concern may be commented on under this criterion.

•   Timely identification and effective action to resolve emerging performance concerns and issues including non-conformances and non-compliances.

•   Response to and implementation of new NNSA, DOE, Presidential and/or Congressional initiatives and requirements

•   Other related items, concerns, and missions.

SECTION 5
ASSUMPTIONS / TECHNICAL BOUNDARY CONDITIONS

List foreseeable impacts to performance.

•   The NNSA and LANS, LLC will agree within 30 days of receipt of final appropriations, or by mutually agreed to date if under a long-term Continuing Resolution, that the funding is sufficient to accomplish these measures. If by this date, Continuing Resolution funding or final appropriation is less than the President’s Budget Request, LANS shall identify any impacts to the FY 2010 PEP measures within 30 calendar days. If interim Continuing Resolution funding is less than the President’s Budget Request, LANS shall identify any impacts to the FY 2010 PEP measures within 15 business days following the approval of the Continuing Resolution.

•   If no specific due date is referenced with any of the PBI completion elements, the due date of that element is to be September 30, 2010.

•   If LANS cannot meet/complete a PBI because of conditions or events that are outside of LANS’ ability to control, the PBI will be renegotiated.

•   Performance Objectives (POs) 11,12,13, and 14 may each address elements of concern and/or success

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×

image

FIGURE E.1 Evaluation measures for LANL 2010 PBI No. 18.

Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 57
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 58
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 59
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 60
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 61
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 62
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 63
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 64
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 65
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 66
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 67
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 68
Suggested Citation:"E--What the Laboratories' Management and Operations (M&O) Contracts Say About the Conduct and Evaluation of Science and Engineering." National Research Council. 2013. Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/13367.
×
Page 69
Next: F--The Investment/Value Returned Framework for Management of S&E »
Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories Get This Book
×
Buy Paperback | $39.00 Buy Ebook | $31.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The three National Security Laboratories--Los Alamos National Laboratory (LANL), Sandia National Laboratories (SNL), and Lawrence Livermore National Laboratory (LLNL)--are managed by private-sector entities under contract to the National Nuclear Security Administration (NNSA). The FY2010 Defense Authorization Act mandated that NNSA task the National Research Council (NRC) to study the quality and management of Science and Engineering (S&E) at these Laboratories.

This study (addressing a total of 5 tasks) is being conducted in two phases. This report covers the first phase, which addresses the relationship between the quality of the science and engineering at the Laboratory and the contract for managing and operating the Laboratory (task 4), and also addresses the management of work conducted by the Laboratory for entities other than the Department of Energy (task 5). The study's second phase will evaluate the actual quality of S&E in key subject areas.

Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories presents assessments of the evolution of the mission of the NNSA Labs and the management and performance of research in support of the missions, and the relationship between the Laboratory Directed Research and Development (LDRD) program and the ability of the Labs to fulfill their mission. The report examines the framework for managing science and engineering research at the Labs and provides an analysis of the relationships among the several players in the management of the Labs--the NNSA, the site offices, the contractors, and the Lab managers--and the effect of that relationship on the Laboratories' ability to carry out science and engineering research.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!