Cover Image

PAPERBACK
$35.00



View/Hide Left Panel

Appendix E
Agency and Industry Efficiency Measures

The table in this appendix (Table E-1) includes efficiency measures developed by agencies and industry. Most of the measures for the agencies were excerpted from the Office of Management and Budget (OMB) Program Assessment Rating Tool (PART) Web site (OMB 2007) and are related to programs of the “research and development” type. These measures either appear in the “Program Performance Measures” section or are cited as agency responses to questions 3.4 or 4.3 on the PART Web site. Question 3.4 is “Does the program have procedures (for example, competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?” Question 4.3 is “Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?” Some of the agency measures listed have been approved for use by OMB as of July 26, 2007.

The table is not an exhaustive list of efficiency measures used by the federal government, but it includes efficiency measures from a variety of agencies, including the Environmental Protection Agency (EPA), the Department of Energy (DOE), the U.S. Department of Agriculture (USDA), the Department of Commerce (DOC), the Department of Defense (DOD), the Department of Health and Human Services (DDHHS), the Department of the Interior (DOI), the Department of Labor (DOL), the Department of Transportation (DOT), the National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA). The industry efficiency measures (and a few agency efficiency measures) were gleaned from presentations during the April 2007 committee meeting.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 95
Appendix E Agency and Industry Efficiency Measures The table in this appendix (Table E-1) includes efficiency measures devel- oped by agencies and industry. Most of the measures for the agencies were ex- cerpted from the Office of Management and Budget (OMB) Program Assess- ment Rating Tool (PART) Web site (OMB 2007) and are related to programs of the “research and development” type. These measures either appear in the “Pro- gram Performance Measures” section or are cited as agency responses to ques- tions 3.4 or 4.3 on the PART Web site. Question 3.4 is “Does the program have procedures (for example, competitive sourcing/cost comparisons, IT improve- ments, appropriate incentives) to measure and achieve efficiencies and cost ef- fectiveness in program execution?” Question 4.3 is “Does the program demon- strate improved efficiencies or cost effectiveness in achieving program goals each year?” Some of the agency measures listed have been approved for use by OMB as of July 26, 2007. The table is not an exhaustive list of efficiency measures used by the fed- eral government, but it includes efficiency measures from a variety of agencies, including the Environmental Protection Agency (EPA), the Department of En- ergy (DOE), the U.S. Department of Agriculture (USDA), the Department of Commerce (DOC), the Department of Defense (DOD), the Department of Health and Human Services (DDHHS), the Department of the Interior (DOI), the Department of Labor (DOL), the Department of Transportation (DOT), the National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA). The industry efficiency measures (and a few agency efficiency measures) were gleaned from presentations during the April 2007 committee meeting. 95

OCR for page 95
96 TABLE E-1 Agency and Industry Efficiency Measures Agency or Organization Program Year Efficiency Measure EPA Endocrine Disruptors 2004 (OPPTS) Cost per labor hour of contracted validation studies (EPA, (combined EPA PART) unpublished material, April 23, 2007) EPA EPA Human Health Research 2005 Average time (in days) to process research-grant proposals from RFA closure to submittal to EPA's Grants Administration Division while maintaining a credible and efficient competitive merit-review system (as evaluated by external expert review) (EPA, unpublished material, April 23, 2007) EPA Land Protection and 2006 Average time (in days) for technical support centers to process and respond to Restoration Research requests for technical document review, statistical analysis, and evaluation of characterization and treatability study plans (EPA, unpublished material, April 23, 2007) EPA Water Quality Research 2006 Number of peer reviewed publications per FTE (EPA, unpublished material, April 23, 2007) EPA Human Health Risk 2006 Average cost to produce Air Quality Criteria/Science Assessment documents Assessment Program (EPA, unpublished material, April 23, 2007) EPA EPA Ecological Research 2007 Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007) EPA Drinking Water Research 2007 Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007) EPA PM Research 2007 Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007) EPA Global Change Research 2007 Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007) EPA Pollution Prevention Research 2007 Percentage variance from planned cost and schedule (approved 3/13/07) (EPA, unpublished material, April 23, 2007)

OCR for page 95
DOD Defense Basic Research 2002 Long-term measure: portion of funded research chosen on basis of merit review; reduce non-merit-reviewed and determined projects by half in 2 years (from 6.0% to 3.0%) (OMB 2007) DOE Advanced Simulation and 2002 Annual average cost per teraflops of delivering, operating, and managing all Computing Stockpile Stewardship Program (SSP) production systems in given fiscal year (OMB 2007) DOE Coal Energy Technology 2005 Administrative costs as percentage of total program costs (OMB 2007) DOE Advanced Fuel Cycle Initiative 2003 Program direction as percentage of total R&D program funding (OMB 2007) DOE Generation IV Nuclear Energy 2003 Program direction as percentage of total R&D program funding (OMB 2007) Systems Initiative DOE National Nuclear Security 2005 Cumulative percentage of active research projects for which independent R&D Administration: Nonprolifera-tion peer assessment of project's scientific quality and mission relevance has been and Verification Research and completed during second year of effort (and again in each later 3-year period Development for projects found to be of merit) (OMB 2007) DOE Nuclear Power 2010 2003 Program direction as percentage of total R&D program funding (OMB 2007) DOE Basic Energy Sciences/ Biological 2006 Average achieved operation time of scientific user facilities as percentage of and Environmental Research total scheduled annual operation time; cost-weighted mean percentage variance from established cost and schedule baselines for major construction, upgrade, or equipment procurement projects (cost variance listed first) (OMB 2007) DOE Hydrogen Program 2003 In 2003, EERE Hydrogen Program had about 130 fuel-cell and hydrogen production research projects that were subject to in-progress peer review by independent experts (Continued) 97

OCR for page 95
98 TABLE E-1 Continued Agency or Organization Program Year Efficiency Measure For all reviewed projects, reviewers provided written comments and numerical ratings – on a scale of 1-4, with 4 being highest – with resulting scores ranging of 2.2-3.9 Program used review results to make important decisions to continue or discontinue projects Research efficiency = 1- [(no. of projects discontinued/(total no. of projects reviewed - no. of projects judged as completed - earmark projects)] (Beschen 2007) DOI U.S. Geological Survey – 2005 Average cost per gigabyte of data available through servers under program control Biological Information (EPA, unpublished material, 2006) Management and Delivery DOI U.S. Geological Survey – 2005 Average cost per sample for selected high-priority environmentally available Biological Research & chemical analyses (EPA, unpublished material, 2006) Monitoring DOI U.S. Geological Survey – 2007 Average cost of systematic analysis or investigation (dollars in millions) (EPA, Energy Resource Assessments unpublished material, 2006) DOI U.S. Geological Survey – 2003 Average cost of systematic analysis or investigation; average cost per analysis Mineral Resource Assessment allows comparisons among projects to determine how efficiencies can be achieved (EPA, unpublished material, 2006) DOI U.S. Geological Survey – 2004 Average cost per analytic result, adjusted for inflation, is stable or declining over Water Resources Research 5-year period (EPA, unpublished material, 2006) DOI U.S. Geological Survey – 2004 Percentage of daily streamflow measurement sites with data that are converted Water Information Collection from provisional to final status within 4 months of day of collection (EPA, and Dissemination unpublished material, 2006)

OCR for page 95
DOI U.S. Geological Survey – 2005 Percentage improvement in detectability limits for selected high-priority Biological Research & environmentally available chemical analytes (EPA, unpublished material, 2006) Monitoring DOI U.S. Geological Survey – 2003 Percentage of total cost saved through partnering for data collection of high- Geographic Research, resolution imagery (EPA, unpublished material, 2006) Investigations, and Remote Sensing DOI Bureau of Reclamation – 2003 Each year, increase in R&D cost-sharing per reclamation R&D program dollar Science and Technology will contribute toward achieving long-term goal of 34% cumulative increase over Program 6-year period (OMB 2007) DOT Highway Research and 2004 Annual percentage of all research projects completed within budget (OMB 2007) Development/ Intelligent Transportation Systems DOT Highway Research and 2004 Annual percentage of research-project deliverables completed on time (OMB Development/ Intelligent 2007) Transportation Systems DOT Railroad Research and 2004 Organizational Excellence: Percentage of projects completed on time (OMB 2007) Development Department of National Assessment for 2003 Timeliness of NAEP data for Reading and Mathematics Assessment in support of Education Educational Progress President's No Child Left Behind initiative (time from end of data collection to initial public release of results for reading and mathematics assessments) (EPA, unpublished material, 2006) Department of National Center for 2003 NCES will release information from surveys within specified times; NCES Education Education Statistics collected baseline information in 2005, examining time-to-release for 31 recent surveys (National Assessment of Educational Progress releases not included in these figures) (EPA, unpublished material, 2006) (Continued) 99

OCR for page 95
TABLE E-1 Continued 100 Agency or Organization Program Year Efficiency Measure DHHS National Center for Health 2005 Number of months for release of data as measured by time from end of data Statistics collection to data release on Internet (OMB 2007) DHHS NIH Extramural Research By 2013, provide greater functionality and more streamlined processes in grant Programs administration by continuing to develop NIH Electronic Research Administration System (eRA) (FY 2004) Develop plan to integrate OPDIVs into eRA (FY 2005) Integrate DHHS 50% of eligible DHHS OPDIVs as eRA users for administration of research grants (FY 2006) Integrate DHHS 100% of eligible DHHS OPDIVs as eRA users for administration of research grants Conversion of business processes – (FY 2005) 25% of business processes done electronically – (FY 2006) 40% – (FY 2007) 55% – (FY 2008) 80% (Duran 2007) DHHS NIH Intramural Research 2005 Reallocation of laboratory resources based on extramural reviews by Boards of Program Scientific Counselors (OMB 2007) DHHS Bioterrorism: CDC Intramural 2006 Decrease annual costs for personnel and materials development with Research development and continuous improvement of budget and performance integration information system tools (OMB 2007) DHHS NIOSH 2004 Percentage of grant award or funding decisions made available to applicants within 9 months of application receipt or deadline date while maintaining credible and efficient two-level peer-review system (OMB 2007) DHHS NIOSH Not used Determine future human capital resources needed to support programmatic currently strategic goals, focusing on workforce development or training and succession planning (Sinclair 2007)

OCR for page 95
DHHS NIOSH 2007 Percentage of grant award or funding decisions made available to applicants within 9 months of application receipt or deadline date while maintaining credible and efficient two-level peer-review system (Sinclair 2007) DHHS Extramural Construction By 2010, achieve average annual cost savings of managing construction grants by expanding use of electronic project-management tools that enhance oversight and 20-year use monitoring (Each FY) Achieve average annual cost of managing construction grants (Duran 2007) DHHS HIV/AIDS Research By 2010, use enhanced AIDS Research Information System (ARIS) database to more efficiently conduct portfolio analysis to invest in priority AIDS research (FY 2005) Improve existing ARIS by converting its mainframe system into Web-based system designed by OAR and IC representatives in consultation with a contractor (FY 2006, FY 2007, FY 2008) Track, monitor, and budget for trans-NIH AIDS research, using enhanced ARIS database, to more efficiently conduct portfolio analysis of 100% of expiring grants to determine reallocation of resources for priority research (Duran 2007) DHHS Research Training Program 2006 By 2012, ensure that 100% of trainee appointment forms are processed electronically, to enhance program management (OMB 2007) NASA Human Systems Research and 2005 Time between solicitation and selection in NASA Research Announcements Technology (OMB 2007) NASA Solar System Exploration 2006 Percentage of budget for research projects allocated through open peer-reviewed competition (OMB 2007) NASA Solar System Exploration 2006 Number of days within which NASA Research Announcement research grants for program are awarded, from proposal due date to selection, with goal of 130 days (OMB 2007) (Continued) 101

OCR for page 95
TABLE E-1 Continued 102 Agency or Organization Program Year Efficiency Measure NASA Original Uniform Measures Complete all development projects within 110% of cost and schedule baseline Peer-review and competitively award at least 80%, by budget, of research projects Reduce time within which 80% of NRA research grants are awarded, from proposal due date to selection, by 5% per year, with goal of 130 days Deliver at least 90% of scheduled operating hours for all operations and research facilities (Pollitt 2007) NASA 2007 Year-to-year reduction in Space Shuttle sustaining engineering workforce for flight hardware and software while maintaining safe flight Reduction in ground operations cost (through 2012) of Constellation Systems based on comparison with Space Shuttle Program Number of financial processing steps and time to perform year-end closing Number of hours required for NASA personnel to collect, combine, and reconcile data of contract-management type for external agency reporting purposes (Pollitt 2007) NASA 2007 On-time availability and operation of Aeronautics Test Program ground test facilities in support of research, development, test, and engineering milestones of NASA and DOD programs from both schedule and cost perspectives Operational cost per minute of Space Network support of missions Ratio of Launch Services Program cost per mission to total spacecraft cost Number of people reached via e-education technologies per dollar invested (Pollitt 2007)

OCR for page 95
NOAA Climate Program 2004 Volume of data taken in annually and placed into archive (terabytes) (EPA, unpublished material, 2006) NOAA Ecosystem Research 2005 Cost per site characterization (OMB 2007) NOAA Ecosystem Research 2005 Percentage of grants awarded on time (OMB 2007) NSF Fundamental Science and 2005 Percentage of award decisions made available to applicants within 6 months of Engineering Research proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007) NSF Research on Biocomplexity 2004 Percentage of award decisions made available to applicants within 6 months of in the Environment proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007) NSF Construction and Operations 2003 Percentage of construction acquisition and upgrade projects with negative cost of Research Facilities and schedule variances of less than 10% of approved project plan (EPA, unpublished material, 2006) NSF Polar Research Tools, 2004 Percentage of construction cost and schedule variances of major projects as Facilities and Logistics monitored by earned-value management (OMB 2007) NSF Support for Research 2004 Percentage of award decisions made available to applicants within 6 months of Institutions proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007) NSF Support for Small Research 2004 Percentage of award decisions made available to applicants within 6 months of Collaborations proposal receipt or deadline date while maintaining credible and efficient competitive merit-review system as evaluated by external experts (OMB 2007) NSF Construction and Operations 2003 Percentage of operational facilities that keep scheduled operating time lost to less of Research Facilities than 10% (OMB 2007) NSF Federally Funded Research 2005 Percentage of operational facilities that keep scheduled operating time lost to less and Development Centers than 10% (OMB 2007) (Continued) 103

OCR for page 95
TABLE E-1 Continued 104 Agency or Organization Program Year Efficiency Measure NSF Information Technology Qualitative assessment by external experts that there have been significant Research research contributions to software design and quality, scalable information infrastructure, high-end computing, workforce, and socioeconomic impacts of IT (EPA, unpublished material, 2006) NSF Polar Research Tools, Percentage of person-days planned for Antarctic research for which program is Facilities and Logistics able to provide necessary research support (EPA, unpublished material, 2006) NSF Polar Research Facilities Research facilities: keep construction cost and schedule variances of major polar and Support facilities projects as monitored by earned-value management at 8% or less Research support: provide necessary research support for Antarctic researchers at least 90% of time (OMB 2007) NSF Support for Individual External validation of "significant achievement" in attracting and preparing U.S. Researchers students to be highly qualified members of global S&E workforce (EPA, unpublished material, 2006) NSF Science and Engineering 2006 Percentage of decisions on preproposals that are merit-reviewed and available to Centers Program Centers Program applicants within 5 months of preproposal receipt or deadline date (OMB 2007) NSF Time to decision for proposals: for 70% of proposals submitted to National Science Foundation, inform applicants about funding decisions within 6 months of proposal receipt or deadline date or target date, whichever is later (Tsuchitani 2007) NSF Facilities cost, schedule, and operations: keep negative cost and schedule variances at less than 10% of approved project plan for 90% of facilities; keep loss of operating time due to unscheduled downtime to less than 10% of total scheduled operating time for 90% of operational facilities (Tsuchitani 2007)

OCR for page 95
USDA USDA Research: Economic Percentage of construction acquisition and upgrade projects with negative cost Opportunities for Producers variance of less than 10% of approved project plan (EPA, unpublished material, 2006) USDA Economic Opportunities for 2004 Cumulative dollars saved for grant review (OMB 2007) Producers USDA Economic Opportunities for 2004 Proposal review time in days (OMB 2007) Producers USDA Research on Protection and 2005 Additional research funds leveraged from external sources (OMB 2007) Safety of Agricultural Food Supply USDA Economic Research Service 2005 Index of ERS product releases per staff year (OMB 2007) USDA Grants for Economic 2006 Cumulative dollars saved for grant review: dollars saved reflect average salary Opportunities and Quality of saved by calculating number of calendar days saved annually between receipt of Life for Rural America proposal and date funding awarded for competitively reviewed proposals, then multiplied by average daily salary for CSREES employees (OMB 2007) USDA In-House Research for 2006 Relative increase in peer-reviewed publications (OMB 2007) Natural Resource Base and Environment USDA In-House Research for 2006 Relative increase in peer-reviewed publications (OMB 2007) Nutrition and Health (Continued) 105

OCR for page 95
TABLE E-1 Continued 106 Agency or Organization Program Year Efficiency Measure Alcoa Return-on-investment calculation: (FY 2005) Improve existing ARIS by converting its mainframe system into Web-based system designed by OAR and IC representatives in consultation with contractor Variable cost improvement Margin impact from organic growth Capital avoidance Cost avoidance Annual impact of these four metrics over 5-year period becomes numerator; denominator is total R&D budget Metric is used most often to evaluate overall value of R&D program and current budget focus (Atkins 2007) Alcoa Time (Atkins 2007) Alcoa Cost (Atkins 2007) Alcoa Customer demand (Atkins 2007) Alcoa Risk (Atkins 2007) Alcoa Impact on business (Atkins 2007) Alcoa Impact on customers (Atkins 2007) Alcoa Location (Atkins 2007) Alcoa Intellectual property (Atkins 2007) Alcoa Aggregate R&D expenditures by laboratory group or by identifiable programs and publish value capture or “success rate” for each on annual basis (Atkins 2007) Alcoa ROI on R&D spending; success rate of launched products (Atkins 2007)

OCR for page 95
Dow Chemical Publications; participation and leadership in scientific community (collaborative research efforts; trade associations; ILSI-HESI; external workshops; adjunct faculty positions, journal or book editors, professional societies) (Bus 2007) IBM ROI on Summer Internship Program and Graduate Fellowship Program: what percentage return as regular IBM research employees? IBM “Bureaucracy Busters” Initiative to reduce bureaucracy in laboratory support, information-technology support, HR processes, and business processes (Kenney 2007) IBM Tracking of patent-evaluation process (Kenney 2007) IBM Customer-satisfaction surveys for support functions to evaluate effect of service reductions (Kenney 2007) IBM Measurement of response time and turnaround for external contracts (Kenney 2007) IBM Measurement of span of responsibility for secretarial support (Kenney 2007) Procter & Time saved in product development (Daston 2007) Gamble Procter & Increased confidence about safety (Daston 2007) Gamble Procter & External relations benefits (although not quantifiable) (Daston 2007) Gamble 107

OCR for page 95
108 Evaluating Research Efficiency in EPA REFERENCES Atkins, P. 2007. Alcoa Research and Development. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environ- mental Protection Agency, April 24, 2007, Washington, DC. Beschen, D. 2007. Federal Research Agencies' Overview. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Envi- ronmental Protection Agency, April 24, 2007, Washington, DC. Bus, J.S. 2007. Research Efficiency: Industry Perspective. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Envi- ronmental Protection Agency, April 24, 2007, Washington, DC. Daston, G. 2007. Evaluating Efficiency of P&G Central Product Safety Research. Presen- tation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC. Duran, D. 2007. Measuring Efficiency in Science. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environ- mental Protection Agency, April 24, 2007, Washington, DC. Kenney, J. 2007. Evaluating R&D Efficiency IBM’s Perspective-Within and Without IBM. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC. OMB (Office of Management and Budget). 2007. ExpectMore.gov. Office of Manage- ment and Budget [online]. Available: http://www.whitehouse.gov/omb/expect more/ [accessed April 5, 2007]. Pollitt, J.A. 2007. Evaluating the Efficiency of NASA’s R&D Programs. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Pro- grams at the Environmental Protection Agency, April 24, 2007, Washington, DC. Sinclair, R. 2007. PART Efficiency Measures at NIOSH. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Envi- ronmental Protection Agency, April 24, 2007, Washington, DC. Tsuchitani, P. 2007. PART Efficiency Measures at NIOSH. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Envi- ronmental Protection Agency, April 24, 2007, Washington, DC.